# HELP - static nat not handling arp correctly - SOLVED

## Moriah

I have a classic bastion firewall gateway feeding into a dmz with a choke firewall between the dmz and the secured lan.  I am using iptables.  I am masquerading the lan, and that works properly.  I also have several servers on the dmz that need to be internet facing.  I am using static nat for them, as the dmz uses non-routable ip addressing.  The internet side of the gateway connects to a cable modem.  The cable modm also provides the gateway router address as one greater than the ip address of the cidr block as a whole.

My problem is that none of the servers are working properly, and after studying the traffic on the intenet interface of the gateway, I see that none of the static natted servers is replying to requests from the gateway for arp packets.

Does anyone have any idea what might be causing this problem?    :Question: 

I realize there will likely be questions for more detail, but this can at least start the conversation.

Thanks!    :Very Happy: 

----------

## krinn

 *Moriah wrote:*   

> Does anyone have any idea what might be causing this problem?    

 

Could be you misunderstand dmz concept?

The rule is simple: you have one computer in dmz per internet entry: if you have one connection, one computer only could be in dmz.

And you've said many things that are strange when speaking about dmz.

So the most probable cause is  I also have several servers on the dmz that need to be internet facing because they cannot.

To make the story short about masquerade and dmz:

When a packet is coming to your internet IP, the fragment hold the IP of the destination host, so if computer is surfing the web, a website will sent to your internet IP a packet that have the IP address of the host as destination. The router than will received that packet because it is behind that IP, see it is for a local IP host, and forward it to that host.

That's masquerading: why an host is able to contact someone in internet, while that someone is able to answer to it ; but also why someone from internet is unable to directly contact a masquerade host.

The dmz is a case when someone is contacting your IP address, but the destination host is not set or unknown, the router doesn't know to whom the packet should goes, and it then route it to 1/ an host that is setup to answer to packet made for a special port (port forwarding) 2/ if none are set, the host that is set as dmz ; the idea of demilitarized zone is kindof strange name, it is not a reference as an host that have no protection or military tools, but as an host that is expose to public while "militarized" area keep hosts away from public eyes, i think canon fodder should represent the dmz host better.

So finally the dmz host is (normally) the strongest host, it is the one everyone will reach trying to reach you, and if you can use only one guard for your security, you won't put a pussy, but the Rambo guy of your company for the task (yeah with some "Captain america is a pussy" MOTD set).

And when a little virus is knocking at your internet IP looking to kill someone, it always then face Rambo.

That's why you cannot really have many host as dmz, because there's only one door to knock to (your internet IP address), and one guard behind that door, and the masquerade hosts are behind doors that are behind Rambo, and you could only enter them after Rambo as check you.

If you want many hosts answering to one port : you must make rambo a load balancer, it is still him that get the query, and him that will balance to what hosts the packet should end.

If you want many servers answer to dedicated usage, you will use port forwarding, and the idea is someone knock at door port 80, rambo is still behind door 80, but he also knows, if someone use that door, he will sent that to another host.

So when an unknown packet (one with no local IP host destination set) is coming to you, the packet is sent to the dmz host, if you put x hosts in dmz, only host one will received it, and if that host is unable to answer it, the packet will time out, if Rambo is not sleeping, he will do the job with that packet, in both case, your secondary dmz host will never seen that packet.

And this explains your none of the servers are working properly ; because they "could" be working properly, just that they never seen a packet because Rambo is doing the job.

Basically, i think your iptables rules are all mess up with plenty entries to reach someone that couldn't be reach as long as he isn't the guard itself.

----------

## Moriah

You *TOTALLY* misunderstood me!

This is a network setup that has worked well for 14 years.  Lately, I have had to make some changes in the way it connects to the internet.  My cable modem provider supplies me with a CIDR block of ip addresses, *NOT JUST ONE* !

My gateway firewall does static nat to each of the dmz based servers, one server per static publicly routable ip address in my CIDR block.

It also does masquarade to hide the choke firewall, which handles the lan full of workstations.  The choke masquerades the workstations, and the gateway hides the choke by means of masquerading the choke itself, so 2 levels of nat.

The servers on the dmz each have their own non-routable ip address, each of which  gets separately static natted by the gateway to the public static ip address assigned to that server.

My problem is the static natted servers seem to not be getting ARP replies back to the cable modem's gateway ip address.  Consequently, they cannot communicate with the outside world.  They communicate with each other and with the lan workstations just fine.  The workstations communicate with the internet just fine.

Now do you understand the problem better?

----------

## Ant P.

First question: are the routing tables on the servers and gateway sane? That they're not replying to ARP can mean one of two things, either they're set to ignore that via sysctl or they believe there's no way to reply (i.e. link-local route).

Also how are you determining the ARPs aren't working? It might be useful to know where exactly the communication breaks down.

Since you haven't mentioned it I assume the LAN can reach the servers OK, just not the internet side?

(Before we waste too much time on this, make sure it's not just a loose cable. It happens...)

----------

## krinn

Sorry, i read it once and twice, really hard to get you. You make it hard with things like "non-routable ip address", i don't get it.

At least i get something: you are mistaking ARP, arp is use to translate IP->mac address, for internet you will use dns but not arp.

And not having arp messages going thru internet is a good thing.

If they can communicate with each other, it's a proof arp is working fine ; because arp will be use by one of your local device to identify and locate one of this natted server to speak with it.

If your problem is then inability to translate name into IP, that's dns problem, something you can test with a natted server, as ping -c1 8.8.8.8 should work then.

If your problem is inability to ping internet (just with the upper example), it's more a routing problem. While it could be firewall, generally you don't care what is getting out, more about what is getting in.

From what i get of your problem, it seems you have a routing problem

----------

## Moriah

@Ant P.:   *Quote:*   

> are the routing tables on the servers and gateway sane?

 

A: Here is the gateway:

```

jacob /etc/rc.d # route -n

Kernel IP routing table

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface

0.0.0.0         96.11.110.97    0.0.0.0         UG    0      0        0 eth1

96.11.110.96    0.0.0.0         255.255.255.248 U     0      0        0 eth1

127.0.0.0       127.0.0.1       255.0.0.0       UG    0      0        0 lo

192.168.2.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0

jacob /etc/rc.d # 

```

And a server:

```

eli ~ # route -n

Kernel IP routing table

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface

0.0.0.0         192.168.2.1     0.0.0.0         UG    3      0        0 eth0

127.0.0.0       127.0.0.1       255.0.0.0       UG    0      0        0 lo

192.168.2.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0

eli ~ # 

```

The ip address 96.11.110.97 is on the cable modem itself, managed by the cable provider.

The provided CIDR block is 96.11.110.96/29, and the gateway itself is at 96.11.110.98.

Regarding the failure to reply to ARP rquests, running tcpdump on the gateway, where eth1 is the cable modem facing side, and eth0 is the dmz facing side (the dmz is addressed by 192.168.2.0/24):

```

jacob /etc/rc.d # tcpdump -i eth1 -n 'net 96.11.110.96/29' | grep ARP

dropped privs to tcpdump

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes

00:27:15.138306 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

00:27:16.138233 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

00:27:17.459055 ARP, Request who-has 96.11.110.99 tell 96.11.110.97, length 46

00:27:18.128536 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

00:27:18.458500 ARP, Request who-has 96.11.110.99 tell 96.11.110.97, length 46

00:27:19.128388 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

00:27:19.458233 ARP, Request who-has 96.11.110.99 tell 96.11.110.97, length 46

00:27:20.128313 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

00:27:21.138417 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

00:27:22.139218 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

00:27:22.389568 ARP, Request who-has 96.11.110.100 tell 96.11.110.97, length 46

00:27:23.138746 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

00:27:23.388325 ARP, Request who-has 96.11.110.100 tell 96.11.110.97, length 46

00:27:24.388372 ARP, Request who-has 96.11.110.100 tell 96.11.110.97, length 46

00:27:25.138873 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

00:27:25.708491 ARP, Request who-has 96.11.110.100 tell 96.11.110.97, length 46

00:27:26.138674 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

00:27:26.708693 ARP, Request who-has 96.11.110.100 tell 96.11.110.97, length 46

00:27:27.138503 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

00:27:27.258625 ARP, Request who-has 96.11.110.99 tell 96.11.110.97, length 46

00:27:27.716343 ARP, Request who-has 96.11.110.100 tell 96.11.110.97, length 46

00:27:28.258528 ARP, Request who-has 96.11.110.99 tell 96.11.110.97, length 46

00:27:29.128480 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

00:27:29.258510 ARP, Request who-has 96.11.110.99 tell 96.11.110.97, length 46

00:27:30.128508 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

00:27:31.128582 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

00:27:32.148763 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

00:27:33.148614 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

00:27:33.158842 ARP, Request who-has 96.11.110.99 tell 96.11.110.97, length 46

00:27:33.159016 ARP, Request who-has 96.11.110.100 tell 96.11.110.97, length 46

00:27:34.148667 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

00:27:34.163545 ARP, Request who-has 96.11.110.99 tell 96.11.110.97, length 46

00:27:34.163769 ARP, Request who-has 96.11.110.100 tell 96.11.110.97, length 46

00:27:35.158646 ARP, Request who-has 96.11.110.99 tell 96.11.110.97, length 46

00:27:35.158795 ARP, Request who-has 96.11.110.100 tell 96.11.110.97, length 46

00:27:35.298626 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

00:27:36.298725 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

00:27:37.298902 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

00:27:39.128597 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

00:27:39.898605 ARP, Request who-has 96.11.110.98 tell 96.11.110.97, length 46

00:27:39.898614 ARP, Reply 96.11.110.98 is-at 00:0e:e8:f9:15:67, length 28

00:27:40.128672 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

00:27:41.128750 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

00:27:43.119001 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

00:27:44.118704 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

00:27:45.118631 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

00:27:46.138887 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

00:27:47.138937 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

00:27:48.138868 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

00:27:50.128916 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

00:27:51.129043 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

00:27:52.128918 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

00:27:54.118973 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

00:27:55.119297 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

00:27:56.118926 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

00:27:57.129229 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

00:27:58.129055 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

00:27:59.129056 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

^C165 packets captured

165 packets received by filter

0 packets dropped by kernel

jacob /etc/rc.d # 

```

You will note that 96.11.110.98, the gateway, did replay to an arp request, but the other addresses are silent.  While tcpdump was running, I was attempting to ping 8.8.8.8 from the 192.168.2.13 dmz server, which should be static natted to 96.11.110.101; the idea was to generate some traffic.

Here is the ethernet port setup on the gateway:

```

jacob /etc/rc.d # ifconfig 

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

        inet 192.168.2.1  netmask 255.255.255.0  broadcast 192.168.2.255

        ether 00:13:20:81:0e:19  txqueuelen 1000  (Ethernet)

        RX packets 8106153  bytes 1496224835 (1.3 GiB)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 10518974  bytes 545842512 (520.5 MiB)

        TX errors 412  dropped 0 overruns 0  carrier 412  collisions 900913

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

        inet 96.11.110.98  netmask 255.255.255.248  broadcast 96.11.110.103

        ether 00:0e:e8:f9:15:67  txqueuelen 1000  (Ethernet)

        RX packets 9739383  bytes 7954058876 (7.4 GiB)

        RX errors 0  dropped 561928  overruns 0  frame 0

        TX packets 6734132  bytes 1359484700 (1.2 GiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 411312

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 16436

        inet 127.0.0.1  netmask 255.0.0.0

        loop  txqueuelen 0  (Local Loopback)

        RX packets 1413731  bytes 325416289 (310.3 MiB)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 1413731  bytes 325416289 (310.3 MiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

jacob /etc/rc.d # 

```

And on the server:

```

eli ~ # ifconfig 

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1400

        inet 192.168.2.13  netmask 255.255.255.0  broadcast 192.168.2.255

        ether 50:e5:49:c8:11:fa  txqueuelen 1000  (Ethernet)

        RX packets 1826169353  bytes 159055033975 (148.1 GiB)

        RX errors 2966  dropped 702  overruns 0  frame 2966

        TX packets 3759557363  bytes 4985540410873 (4.5 TiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536

        inet 127.0.0.1  netmask 255.0.0.0

        loop  txqueuelen 0  (Local Loopback)

        RX packets 669517525  bytes 4611321416923 (4.1 TiB)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 669517525  bytes 4611321416923 (4.1 TiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eli ~ # 

```

Does any of this help?  I can provide more information if needed.

[/quote]

----------

## Ant P.

I think I understand what's happening here...

ARP works directly atop the link-layer protocol, not IP, and doesn't pass through IP routers at all, so what matters to it is what each end of the wire thinks its own address is. The servers may be accessible at those public addresses from the outside world, but as far as they're concerned they only live on a 192.168.2/24 network segment.

Thus it's correct that they don't respond to ARP pings at those public addresses - but a regular ICMP `ping` directed at them should work by traversing the NAT (moreover, it should work the same either side of the NAT). If that doesn't, your problem is very likely in the gateway's iptables rules somewhere.

(side note: that's a huge amount of collisions on the gateway's ethernet, something seems off about that...)

----------

## krinn

the one that always wish to know who is who thru arp request is the router itself, it's the one everyone should contact to ask him then.

if you want server to get arp answer, set its route to 96.11.110.97, always aim at final destination

* set 192.168.2.13 route to 96.11.110.97

* bridge 192.168.2.1 eth0 and eth1 to allow packet from network 192.168.2.* to jump to network 96.11.110.97

----------

## Moriah

@Ant P.  

Those interfaces on the gateway have been up for weeks, so the number of collisions is not as bad as you might think.

Yes, I see now why the cable-modem gateway is not getting replies to its arp requests for the static natted servers, but if it doesn't get a reply, will it not send packates to the ip address it did not get a reply from?  Should they get a reply from the gateway router when it asks for those addresses?  I am thinking not, so I am still confused.

Would you rather see an excerpt of the shell script that set the iptables rules, or an iptables -L dump, or something else.  This has been making me crazy for weeks.  This script used to work using an openvpn tunnel to get to the internet, but not that it has a direct connection, its broken.    :Evil or Very Mad: 

----------

## Ant P.

ARP not working doesn't necessarily mean IP won't, they're two separate protocols at the same layer and have different functions. You shouldn't be able to get ARP replies from those addresses at all, since they're not bound to interfaces on the network.

iptables -L would be very useful, yes (the mangle table will have the interesting stuff, but you can include the filter table too if you want).

----------

## Moriah

I am not using the mangle table, only filter and nat,

```

------------ filter Table ----------------

Chain INPUT (policy DROP 6221 packets, 364806 bytes)

num      pkts      bytes target     prot opt in     out     source               destination         

1      225226 75715903 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           

2           0        0 DROP       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x3F/0x00

3           0        0 DROP       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x03/0x03

4           0        0 DROP       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x06

5           0        0 DROP       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x05/0x05

6           0        0 DROP       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x11/0x01

7           0        0 DROP       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x18/0x08

8           0        0 DROP       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x30/0x20

9           0        0 DROP       all  --  eth1   *       10.0.0.0/8           0.0.0.0/0           

10          0        0 DROP       all  --  eth1   *       172.16.0.0/12        0.0.0.0/0           

11          0        0 DROP       all  --  eth0   *       10.0.0.0/8           0.0.0.0/0           

12          0        0 DROP       all  --  eth0   *       172.16.0.0/12        0.0.0.0/0           

13       1345    48540 DROP       all  --  eth1   *       192.168.0.0/15       0.0.0.0/0           

14          0        0 LOG        all  --  eth0   *      !192.168.2.0/24       0.0.0.0/0            LOG flags 0 level 6 prefix "Bad DMZ input: "

15          0        0 DROP       all  --  eth0   *      !192.168.2.0/24       0.0.0.0/0           

16          0        0 DROP       all  --  *      *       0.0.0.0/8            0.0.0.0/0           

17          0        0 DROP       all  --  *      *       169.254.0.0/16       0.0.0.0/0           

18          0        0 DROP       all  --  *      *       192.0.2.0/24         0.0.0.0/0           

19          0        0 DROP       all  --  *      *       127.0.0.0/8          0.0.0.0/0           

20          0        0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0              LOG flags 0 level 6

21          0        0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0             

22          0        0 LOG        all  --  *      *       255.255.255.255      0.0.0.0/0            LOG flags 0 level 6

23          0        0 DROP       all  --  *      *       255.255.255.255      0.0.0.0/0           

24          0        0 DROP       all  --  *      *       224.0.0.0/4          0.0.0.0/0           

25          0        0 DROP      !udp  --  *      *       0.0.0.0/0            224.0.0.0/4         

26          0        0 ACCEPT     udp  --  *      *       0.0.0.0/0            224.0.0.0/4         

27          0        0 DROP       all  --  *      *       240.0.0.0/4          0.0.0.0/0           

28          0        0 LOG        icmp -f  eth1   *       0.0.0.0/0            0.0.0.0/0            LOG flags 0 level 4 prefix "Fragged ICMP: "

29          0        0 DROP       icmp -f  eth1   *       0.0.0.0/0            0.0.0.0/0           

30          0        0 ACCEPT     icmp --  eth1   *       0.0.0.0/0            96.11.110.98         icmptype 4

31          0        0 ACCEPT     icmp --  eth1   *       0.0.0.0/0            96.11.110.98         icmptype 12

32         13     1034 ACCEPT     icmp --  eth1   *       0.0.0.0/0            96.11.110.98         icmptype 3

33          1       96 ACCEPT     icmp --  eth1   *       0.0.0.0/0            96.11.110.98         icmptype 11

34          0        0 ACCEPT     icmp --  eth1   *       0.0.0.0/0            96.11.110.98         icmptype 0

35         25     1212 ACCEPT     icmp --  eth1   *       0.0.0.0/0            96.11.110.98         icmptype 8

36          0        0 ACCEPT     udp  --  eth1   *       209.18.47.61         96.11.110.98         udp spt:53 dpts:1024:65535

37          0        0 ACCEPT     tcp  --  eth1   *       209.18.47.61         96.11.110.98         tcp spt:53 dpts:1024:65535

38          0        0 ACCEPT     udp  --  eth1   *       209.18.47.62         96.11.110.98         udp spt:53 dpts:1024:65535

39          0        0 ACCEPT     tcp  --  eth1   *       209.18.47.62         96.11.110.98         tcp spt:53 dpts:1024:65535

40          0        0 ACCEPT     tcp  --  eth1   *       0.0.0.0/0            96.11.110.98         tcp spt:873 dpts:1024:65535

41          0        0 ACCEPT     tcp  --  eth1   *       0.0.0.0/0            96.11.110.98         tcp spt:22 dpts:1024:65535 flags:!0x17/0x02

42        704    97480 ACCEPT     tcp  --  eth0   *       0.0.0.0/0            192.168.2.1          tcp spt:22 dpts:1024:65535 flags:!0x17/0x02

43     104363  8831296 ACCEPT     tcp  --  eth1   *       0.0.0.0/0            96.11.110.98         tcp spts:1024:65535 dpt:22

44     155185 71577872 ACCEPT     tcp  --  eth0   *       0.0.0.0/0            192.168.2.1          tcp spts:1024:65535 dpt:22

45          0        0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp spt:37 dpts:1024:65535

46          0        0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp spt:37 dpts:1024:65535

Chain FORWARD (policy DROP 69 packets, 3938 bytes)

num      pkts      bytes target     prot opt in     out     source               destination         

1           0        0 DROP       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x3F/0x00

2           0        0 DROP       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x03/0x03

3           0        0 DROP       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x06

4           0        0 DROP       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x05/0x05

5           0        0 DROP       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x11/0x01

6           0        0 DROP       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x18/0x08

7           0        0 DROP       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x30/0x20

8           0        0 ACCEPT     all  --  eth1   eth0    0.0.0.0/0            192.168.2.13         state NEW,RELATED,ESTABLISHED

9           0        0 ACCEPT     all  --  eth0   eth1    192.168.2.13         0.0.0.0/0            state RELATED,ESTABLISHED

10          0        0 ACCEPT     all  --  eth1   eth0    192.168.2.13         0.0.0.0/0            state RELATED,ESTABLISHED

11          0        0 ACCEPT     all  --  eth0   eth1    0.0.0.0/0            192.168.2.13         state NEW,RELATED,ESTABLISHED

12          0        0 ACCEPT     all  --  eth1   eth0    0.0.0.0/0            192.168.2.12         state NEW,RELATED,ESTABLISHED

13          0        0 ACCEPT     all  --  eth0   eth1    192.168.2.12         0.0.0.0/0            state RELATED,ESTABLISHED

14          0        0 ACCEPT     all  --  eth1   eth0    192.168.2.12         0.0.0.0/0            state RELATED,ESTABLISHED

15          0        0 ACCEPT     all  --  eth0   eth1    0.0.0.0/0            192.168.2.12         state NEW,RELATED,ESTABLISHED

16          0        0 ACCEPT     all  --  eth1   eth0    0.0.0.0/0            192.168.2.11         state NEW,RELATED,ESTABLISHED

17          0        0 ACCEPT     all  --  eth0   eth1    192.168.2.11         0.0.0.0/0            state RELATED,ESTABLISHED

18          0        0 ACCEPT     all  --  eth1   eth0    192.168.2.11         0.0.0.0/0            state RELATED,ESTABLISHED

19          0        0 ACCEPT     all  --  eth0   eth1    0.0.0.0/0            192.168.2.11         state NEW,RELATED,ESTABLISHED

20     742319 599869039 ACCEPT     all  --  eth1   *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED

21     639762 116835056 ACCEPT     all  --  eth0   *       0.0.0.0/0            0.0.0.0/0            state NEW,RELATED,ESTABLISHED

22          0        0 DROP       all  --  eth1   *       10.0.0.0/8           0.0.0.0/0           

23          0        0 DROP       all  --  eth1   *       172.16.0.0/12        0.0.0.0/0           

24          0        0 DROP       all  --  eth0   *       10.0.0.0/8           0.0.0.0/0           

25          0        0 DROP       all  --  eth0   *       172.16.0.0/12        0.0.0.0/0           

26          0        0 DROP       all  --  eth1   *       192.168.0.0/15       0.0.0.0/0           

27          0        0 LOG        all  --  eth0   *      !192.168.2.0/24       0.0.0.0/0            LOG flags 0 level 6 prefix "Bad DMZ input: "

28          0        0 DROP       all  --  eth0   *      !192.168.2.0/24       0.0.0.0/0           

29          0        0 LOG        all  --  *      eth0    0.0.0.0/0           !192.168.2.0/24       LOG flags 0 level 6 prefix "Bad DMZ output: "

30          0        0 DROP       all  --  *      eth0    0.0.0.0/0           !192.168.2.0/24      

31          0        0 DROP       all  --  *      *       0.0.0.0/8            0.0.0.0/0           

32          0        0 DROP       all  --  *      *       169.254.0.0/16       0.0.0.0/0           

33          0        0 DROP       all  --  *      *       192.0.2.0/24         0.0.0.0/0           

34          0        0 DROP       all  --  *      *       127.0.0.0/8          0.0.0.0/0           

35          0        0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0              LOG flags 0 level 6

36          0        0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0             

37          0        0 LOG        all  --  *      *       255.255.255.255      0.0.0.0/0            LOG flags 0 level 6

38          0        0 DROP       all  --  *      *       255.255.255.255      0.0.0.0/0           

39          0        0 DROP       all  --  *      *       224.0.0.0/4          0.0.0.0/0           

40          0        0 DROP      !udp  --  *      *       0.0.0.0/0            224.0.0.0/4         

41          0        0 ACCEPT     udp  --  *      *       0.0.0.0/0            224.0.0.0/4         

42          0        0 DROP       all  --  *      *       240.0.0.0/4          0.0.0.0/0           

43          0        0 LOG        icmp -f  eth1   *       0.0.0.0/0            0.0.0.0/0            LOG flags 0 level 4 prefix "Fragged ICMP: "

44          0        0 DROP       icmp -f  eth1   *       0.0.0.0/0            0.0.0.0/0           

45          0        0 ACCEPT     icmp --  eth1   *       0.0.0.0/0            192.168.2.0/24       icmptype 4

46          0        0 ACCEPT     icmp --  *      eth1    192.168.2.0/24       0.0.0.0/0            icmptype 4

47          0        0 ACCEPT     icmp --  eth1   *       0.0.0.0/0            192.168.2.0/24       icmptype 12

48          0        0 ACCEPT     icmp --  *      eth1    192.168.2.0/24       0.0.0.0/0            icmptype 12

49          0        0 ACCEPT     icmp --  eth1   *       0.0.0.0/0            192.168.2.0/24       icmptype 3

50          0        0 DROP       icmp --  *      eth1    192.168.2.0/24       0.0.0.0/0            icmptype 3

51          0        0 ACCEPT     icmp --  eth1   *       0.0.0.0/0            192.168.2.0/24       icmptype 11

52          0        0 ACCEPT     icmp --  *      eth1    192.168.2.0/24       0.0.0.0/0            icmptype 8

53          0        0 ACCEPT     icmp --  eth1   *       0.0.0.0/0            192.168.2.0/24       icmptype 0

54          0        0 ACCEPT     icmp --  eth1   *       0.0.0.0/0            192.168.2.0/24       icmptype 8

55          0        0 ACCEPT     icmp --  *      eth1    192.168.2.0/24       0.0.0.0/0            icmptype 0

56          0        0 ACCEPT     udp  --  *      eth1    192.168.2.0/24       209.18.47.61         udp spts:1024:65535 dpt:53

57          0        0 ACCEPT     udp  --  eth1   *       209.18.47.61         192.168.2.0/24       udp spt:53 dpts:1024:65535

58          0        0 ACCEPT     tcp  --  *      eth1    192.168.2.0/24       209.18.47.61         tcp spts:1024:65535 dpt:53

59          0        0 ACCEPT     tcp  --  eth1   *       209.18.47.61         192.168.2.0/24       tcp spt:53 dpts:1024:65535

60          0        0 ACCEPT     udp  --  *      eth1    192.168.2.0/24       209.18.47.62         udp spts:1024:65535 dpt:53

61          0        0 ACCEPT     udp  --  eth1   *       209.18.47.62         192.168.2.0/24       udp spt:53 dpts:1024:65535

62          0        0 ACCEPT     tcp  --  *      eth1    192.168.2.0/24       209.18.47.62         tcp spts:1024:65535 dpt:53

63          0        0 ACCEPT     tcp  --  eth1   *       209.18.47.62         192.168.2.0/24       tcp spt:53 dpts:1024:65535

64          0        0 ACCEPT     tcp  --  *      *       0.0.0.0/0            96.11.110.101        tcp spts:1024:65535 dpt:25

65          0        0 ACCEPT     tcp  --  *      *       96.11.110.101        0.0.0.0/0            tcp spt:25 dpts:1024:65535 flags:!0x17/0x02

Chain OUTPUT (policy DROP 0 packets, 0 bytes)

num      pkts      bytes target     prot opt in     out     source               destination         

1      225226 75715903 ACCEPT     all  --  *      lo      0.0.0.0/0            0.0.0.0/0           

2           0        0 LOG        all  --  *      eth0    0.0.0.0/0           !192.168.2.0/24       LOG flags 0 level 6 prefix "Bad DMZ output: "

3           0        0 DROP       all  --  *      eth0    0.0.0.0/0           !192.168.2.0/24      

4           0        0 ACCEPT     icmp --  *      eth1    96.11.110.98         0.0.0.0/0            icmptype 4

5           0        0 ACCEPT     icmp --  *      eth1    96.11.110.98         0.0.0.0/0            icmptype 12

6           0        0 ACCEPT     icmp --  *      eth1    96.11.110.98         0.0.0.0/0            icmptype 3 code 4

7           0        0 ACCEPT     icmp --  *      eth1    192.168.2.0/24       0.0.0.0/0            icmptype 3 code 4

8           0        0 DROP       icmp --  *      eth1    96.11.110.98         0.0.0.0/0            icmptype 3

9           0        0 ACCEPT     icmp --  *      eth1    96.11.110.98         0.0.0.0/0            icmptype 8

10         25     1212 ACCEPT     icmp --  *      eth1    96.11.110.98         0.0.0.0/0            icmptype 0

11          0        0 ACCEPT     udp  --  *      eth1    96.11.110.98         209.18.47.61         udp spts:1024:65535 dpt:53

12          0        0 ACCEPT     tcp  --  *      eth1    96.11.110.98         209.18.47.61         tcp spts:1024:65535 dpt:53

13          0        0 ACCEPT     udp  --  *      eth1    96.11.110.98         209.18.47.62         udp spts:1024:65535 dpt:53

14          0        0 ACCEPT     tcp  --  *      eth1    96.11.110.98         209.18.47.62         tcp spts:1024:65535 dpt:53

15          0        0 ACCEPT     tcp  --  *      eth1    96.11.110.98         0.0.0.0/0            tcp spts:1024:65535 dpt:873

16          0        0 ACCEPT     tcp  --  *      eth1    96.11.110.98         0.0.0.0/0            tcp spts:1024:65535 dpt:22

17        663   101444 ACCEPT     tcp  --  *      eth0    192.168.2.1          0.0.0.0/0            tcp spts:1024:65535 dpt:22

18     109810 70322684 ACCEPT     tcp  --  *      eth1    96.11.110.98         0.0.0.0/0            tcp spt:22 dpts:1024:65535 flags:!0x17/0x02

19     141620 86033116 ACCEPT     tcp  --  *      eth0    192.168.2.1          0.0.0.0/0            tcp spt:22 dpts:1024:65535 flags:!0x17/0x02

20          0        0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp spts:1024:65535 dpt:37

21          0        0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp spts:1024:65535 dpt:37

------------ nat Table ----------------

Chain PREROUTING (policy ACCEPT 83159 packets, 4912179 bytes)

num      pkts      bytes target     prot opt in     out     source               destination         

1           0        0 DNAT       all  --  eth1   *       0.0.0.0/0            96.11.110.101        to:192.168.2.13

2           0        0 DNAT       all  --  eth1   *       0.0.0.0/0            96.11.110.100        to:192.168.2.12

3           0        0 DNAT       all  --  eth1   *       0.0.0.0/0            96.11.110.99         to:192.168.2.11

Chain INPUT (policy ACCEPT 1790 packets, 104552 bytes)

num      pkts      bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 28 packets, 1680 bytes)

num      pkts      bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 28 packets, 1680 bytes)

num      pkts      bytes target     prot opt in     out     source               destination         

1        2851   199584 SNAT       all  --  *      eth1    192.168.2.13         0.0.0.0/0            to:96.11.110.101

2           0        0 SNAT       all  --  *      eth1    192.168.2.12         0.0.0.0/0            to:96.11.110.100

3       10573   316792 SNAT       all  --  *      eth1    192.168.2.11         0.0.0.0/0            to:96.11.110.99

4       61155  3856619 MASQUERADE  all  --  *      eth1    192.168.2.2          0.0.0.0/0           

```

----------

## Ant P.

Right, nat table, that's what I meant to say... (it's been years since I last used iptables).

That lack of matches for the DNAT rules is a pretty big hint. I thought it'd work without this but... can you try adding all of those public addresses to the gateway's eth1?

----------

## Moriah

I have already tried:

```

# iptables eth1 96.11.110.96/29

```

and also:

```

# iptables eth1 96.11.110.96 netmask 255.255.255.248

```

both failed to work at all; I even lost the masqueraded lan's connection to the internet.

So how should I execute your request?  What command line?

----------

## Ant P.

```
ip addr add 96.11.110.96/29 dev eth1
```

Try that (repeat for each address)? I don't think ifconfig understands multiple addresses.

----------

## Moriah

Oh!  Using the newer "ip" command.  I've never used that before, only ifconfig.  OK, I'll give the ip man page a read, then try your suggestion.  I suspected that somehow the other 96.11.110.* addresses were not making it thru the interface, but I had no good way to test it, and I did not know how to the the eth1 interface to pass them.

Thanks for your help.  I will not likely be able to try this out until tomorrow evening, so don't think I am ignoring you...

----------

## Moriah

Apparently, it makes no difference.  After a fresh boot, which runs the firewall script, I get:

```

jacob ~ # ifconfig 

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

        inet 192.168.2.1  netmask 255.255.255.0  broadcast 192.168.2.255

        ether 00:13:20:81:0e:19  txqueuelen 1000  (Ethernet)

        RX packets 227  bytes 25277 (24.6 KiB)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 246  bytes 228158 (222.8 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 11

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

        inet 96.11.110.98  netmask 255.255.255.248  broadcast 96.11.110.103

        ether 00:0e:e8:f9:15:67  txqueuelen 1000  (Ethernet)

        RX packets 636  bytes 261885 (255.7 KiB)

        RX errors 0  dropped 106  overruns 0  frame 0

        TX packets 396  bytes 54449 (53.1 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 15

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 16436

        inet 127.0.0.1  netmask 255.0.0.0

        loop  txqueuelen 0  (Local Loopback)

        RX packets 0  bytes 0 (0.0 B)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 0  bytes 0 (0.0 B)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

jacob ~ # ip addr show

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN group default 

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo

2: dummy0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default 

    link/ether c2:c2:df:15:a8:40 brd ff:ff:ff:ff:ff:ff

3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000

    link/ether 00:13:20:81:0e:19 brd ff:ff:ff:ff:ff:ff

    inet 192.168.2.1/24 brd 192.168.2.255 scope global eth0

4: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000

    link/ether 00:0e:e8:f9:15:67 brd ff:ff:ff:ff:ff:ff

    inet 96.11.110.98/29 brd 96.11.110.103 scope global eth1

jacob ~ # 

```

The script has the following code early in the script:

```

# Here is the interface, IP address, netmask, broadcast address,

# network address, and gateway address assigned to the WILD interface.

# Starting 2018_02_01 time-warner cable ne "spectrum" is providing a

# static CIDR block of:

#

# 96.11.110.96/29

# gw 96.11.110.97

# mask 255.255.255.248

# bcast 96.11.110.103

WILD_IF=eth1

WILD_IP=96.11.110.98

WILD_MASK=255.255.255.248

WILD_BC=96.11.110.103

WILD_NW=96.11.110.96/29

WILD_GW=96.11.110.97

# There is no tunnel as of 2018_02_01; time warner cable provides

# statis ip's directly.  

TUNNEL_IF=$WILD_IF

#----------------------------------------------------------------

# NOTE: This must be changed in:

#   /etc/rc.d/firewall

#   /etc/rc.d/fw_emerger

#   /etc/ppp/options

#----------------------------------------------------------------

TUNNEL_IP=$WILD_IP

# ----------------

# The static CIDR block assigned to elilabs.com by iglou.com is:

# 

# Starting 2018_02_01 time-warner cable ne "spectrum" is providing a

# static CIDR block of:

#

# 96.11.110.96/29

# gw 96.11.110.97

# mask 255.255.255.248

# bcast 96.11.110.103

ELILABS_NW=96.11.110.96/29

ELILABS_MASK=255.255.255.248

ELILABS_BC=96.11.110.103

# ----------------

# Here is the interface, IP address, netmask, broadcast address, and

# network address assigned to the DMZ interface.  This interface

# communicates between the gateway firewall and the DMZ.

DMZ_IF=eth0

DMZ_IP=192.168.2.1

DMZ_MASK=255.255.255.0

DMZ_BC=192.168.2.255

DMZ_NW=192.168.2.0/24

# ----------------

# fire up the interfaces

ifconfig $DMZ_IF $DMZ_IP netmask $DMZ_MASK broadcast $DMZ_BC

ifconfig $WILD_IF $WILD_IP netmask $WILD_MASK broadcast $WILD_BC

#ip addr add $WILD_NW dev $WILD_IF

#ip addr add $ELI_ELILABS_COM_IP dev $WILD_IF

#ip addr add $IPEPS_ELILABS_COM_IP dev $WILD_IF

#ip addr add $DATAPROBE_ELILABS_COM_IP dev $WILD_IF

```

Note that the ip commands are commented out.  After running those ip commands, we see:

```

jacob /etc/rc.d # ifconfig

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

        inet 192.168.2.1  netmask 255.255.255.0  broadcast 192.168.2.255

        ether 00:13:20:81:0e:19  txqueuelen 1000  (Ethernet)

        RX packets 4661  bytes 573473 (560.0 KiB)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 5103  bytes 1279246 (1.2 MiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 265

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

        inet 96.11.110.98  netmask 255.255.255.248  broadcast 96.11.110.103

        ether 00:0e:e8:f9:15:67  txqueuelen 1000  (Ethernet)

        RX packets 1553  bytes 365947 (357.3 KiB)

        RX errors 0  dropped 283  overruns 0  frame 0

        TX packets 1021  bytes 130803 (127.7 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 15

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 16436

        inet 127.0.0.1  netmask 255.0.0.0

        loop  txqueuelen 0  (Local Loopback)

        RX packets 7419  bytes 1050972 (1.0 MiB)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 7419  bytes 1050972 (1.0 MiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

jacob /etc/rc.d # ip addr show

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN group default 

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo

2: dummy0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default 

    link/ether c2:c2:df:15:a8:40 brd ff:ff:ff:ff:ff:ff

3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000

    link/ether 00:13:20:81:0e:19 brd ff:ff:ff:ff:ff:ff

    inet 192.168.2.1/24 brd 192.168.2.255 scope global eth0

4: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000

    link/ether 00:0e:e8:f9:15:67 brd ff:ff:ff:ff:ff:ff

    inet 96.11.110.98/29 brd 96.11.110.103 scope global eth1

    inet 96.11.110.96/29 scope global secondary eth1

jacob /etc/rc.d # 

```

So although we see eth1 now has a secondary scope, it is basicly the same as the primary scope.  Also note that eth0 has the overall scope of a class C network, so ifconfig can specify a CIDR range.

The behavior of the network with respect to static natted servers on the dmz is unchanged.   :Evil or Very Mad: 

Do you have any other ideas what might be wrong?

Perhaps we should try some additional experiments using tcpdump.  I can try to access the 96.11.110.96/29 addresses from outside my own network by connecting to a different isp (I have both cable and DSL, from 2 different providers), although my access to the DSL connection is via a natted wireless connection.  If absolutely necessary, I can move some wires around and get to an un-natted DHCP address on the DSL circuit.  So far, I have mainly been trying to ping 8.8.8.8 (google dns) to test whether I can get out, since it does not require any address resolution at this end.

----------

## Ant P.

Yes, knowing what this looks like from outside might help.

Run nmap on it too, I forgot that existed... very useful for debugging connectivity issues like this. Recommend you use the GUI (USE=zenmap) if possible because it adds a "radar" view that shows the distance each host is at.

----------

## Moriah

I will try to run some experiments from an outside vantage point.  I will connect a laptop to the DSL line, which used DHCP to give the wireless router an ip address, and the router nats the wireless connections, so the ip address of the laptop will be double-natted, but I have a web browser on the wireless router's web page, so I can determine the address the router faces the internet with.  That would be the address we need to look for using tcpdump on the cable modem gateway router.

Hopefully, I will have some time tomorrow (2018_04_21) to try these experiments.  I will post the results here.

Did you have any time to look at the iptables -L displays I posted?

----------

## Ant P.

I've looked at them, but I can't see any obvious mistakes. The DNAT rules aren't being matched, but that looks like a symptom, not a problem.

----------

## Moriah

I'm thinking they aren't being matched because the packet never makes it that far.  I'm thinking we need to look closer at the input chain.

----------

## Moriah

I did some tcpdump-ing and now I can see that the snat-ting is working ok.  When I dump packets containing the web server's ip address, then ping 8.8.8.8 from the web server, I can see the pings going out, but the replies do not come back in:

```

 jacob /etc/rc.d # tcpdump -n -i eth1 | grep 96.11.110.101

dropped privs to tcpdump

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes

10:39:30.830785 IP 96.11.110.101 > 8.8.8.8: ICMP echo request, id 21354, seq 1, length 64

10:39:30.865780 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

10:39:31.829958 IP 96.11.110.101 > 8.8.8.8: ICMP echo request, id 21354, seq 2, length 64

10:39:31.865607 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

10:39:32.830138 IP 96.11.110.101 > 8.8.8.8: ICMP echo request, id 21354, seq 3, length 64

10:39:32.865758 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

10:39:33.830072 IP 96.11.110.101 > 8.8.8.8: ICMP echo request, id 21354, seq 4, length 64

10:39:33.896067 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

10:39:34.830003 IP 96.11.110.101 > 8.8.8.8: ICMP echo request, id 21354, seq 5, length 64

10:39:34.895719 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

10:39:35.829945 IP 96.11.110.101 > 8.8.8.8: ICMP echo request, id 21354, seq 6, length 64

10:39:35.895796 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

10:39:36.829872 IP 96.11.110.101 > 8.8.8.8: ICMP echo request, id 21354, seq 7, length 64

10:39:37.830061 IP 96.11.110.101 > 8.8.8.8: ICMP echo request, id 21354, seq 8, length 64

10:39:37.875922 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

10:39:38.829988 IP 96.11.110.101 > 8.8.8.8: ICMP echo request, id 21354, seq 9, length 64

10:39:38.875799 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

10:39:39.829925 IP 96.11.110.101 > 8.8.8.8: ICMP echo request, id 21354, seq 10, length 64

10:39:39.875827 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

10:39:40.829855 IP 96.11.110.101 > 8.8.8.8: ICMP echo request, id 21354, seq 11, length 64

10:39:40.895908 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

10:39:41.895919 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

10:39:42.895838 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

10:40:03.176243 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

10:40:04.176118 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

10:40:05.176449 ARP, Request who-has 96.11.110.101 tell 96.11.110.97, length 46

^C138 packets captured

138 packets received by filter

0 packets dropped by kernel

jacob /etc/rc.d # 

```

```

jacob /etc/rc.d # tcpdump -n -i eth1 | grep 8.8.8.8      

dropped privs to tcpdump

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes

10:45:55.080897 IP 96.11.110.101 > 8.8.8.8: ICMP echo request, id 24215, seq 1, length 64

10:45:56.090068 IP 96.11.110.101 > 8.8.8.8: ICMP echo request, id 24215, seq 2, length 64

10:45:57.090003 IP 96.11.110.101 > 8.8.8.8: ICMP echo request, id 24215, seq 3, length 64

10:45:58.089939 IP 96.11.110.101 > 8.8.8.8: ICMP echo request, id 24215, seq 4, length 64

10:45:59.089868 IP 96.11.110.101 > 8.8.8.8: ICMP echo request, id 24215, seq 5, length 64

10:46:00.089802 IP 96.11.110.101 > 8.8.8.8: ICMP echo request, id 24215, seq 6, length 64

10:46:01.089740 IP 96.11.110.101 > 8.8.8.8: ICMP echo request, id 24215, seq 7, length 64

10:46:02.089921 IP 96.11.110.101 > 8.8.8.8: ICMP echo request, id 24215, seq 8, length 64

10:46:03.089851 IP 96.11.110.101 > 8.8.8.8: ICMP echo request, id 24215, seq 9, length 64

^C161 packets captured

163 packets received by filter

0 packets dropped by kernel

jacob /etc/rc.d # 

```

But when I ping 8.8.8.8 from a masqueraded lan machine, I see:

```

jacob /etc/rc.d # tcpdump -n -i eth1 | grep 8.8.8.8

dropped privs to tcpdump

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes

10:49:00.994605 IP 96.11.110.98 > 8.8.8.8: ICMP echo request, id 15935, seq 1, length 64

10:49:01.032787 IP 8.8.8.8 > 96.11.110.98: ICMP echo reply, id 15935, seq 1, length 64

10:49:01.996032 IP 96.11.110.98 > 8.8.8.8: ICMP echo request, id 15935, seq 2, length 64

10:49:02.045842 IP 8.8.8.8 > 96.11.110.98: ICMP echo reply, id 15935, seq 2, length 64

10:49:02.997967 IP 96.11.110.98 > 8.8.8.8: ICMP echo request, id 15935, seq 3, length 64

10:49:03.052996 IP 8.8.8.8 > 96.11.110.98: ICMP echo reply, id 15935, seq 3, length 64

10:49:03.998895 IP 96.11.110.98 > 8.8.8.8: ICMP echo request, id 15935, seq 4, length 64

10:49:04.035919 IP 8.8.8.8 > 96.11.110.98: ICMP echo reply, id 15935, seq 4, length 64

10:49:04.999838 IP 96.11.110.98 > 8.8.8.8: ICMP echo request, id 15935, seq 5, length 64

10:49:05.045823 IP 8.8.8.8 > 96.11.110.98: ICMP echo reply, id 15935, seq 5, length 64

10:49:06.001765 IP 96.11.110.98 > 8.8.8.8: ICMP echo request, id 15935, seq 6, length 64

10:49:06.045124 IP 8.8.8.8 > 96.11.110.98: ICMP echo reply, id 15935, seq 6, length 64

10:49:07.003190 IP 96.11.110.98 > 8.8.8.8: ICMP echo request, id 15935, seq 7, length 64

10:49:07.046026 IP 8.8.8.8 > 96.11.110.98: ICMP echo reply, id 15935, seq 7, length 64

^C115 packets captured

115 packets received by filter

0 packets dropped by kernel

jacob /etc/rc.d # 

```

So there is something messed up with the dnat prerouting stuff.

----------

## Hu

First, apologies for neglecting your original private message.  Your configuration is too complex for me to reason about in the small bits of time I can easily spend diagnosing network issues.  You started this thread before I was able to clear enough time to dig through it, and seemed to be getting good advice.

DNAT should only matter for new incoming traffic.  Responses to previous outbound traffic are handled separately, because remapping entries should already exist.

tcpdump sees traffic before netfilter can filter it, so even packets destined to be DROPped can be seen in tcpdump (but only on the host which performs the DROP; hosts deeper in never get the traffic, so they never have a chance to drop it).  If a tcpdump on your public interface shows no pongs arriving, then your problem is that the peer's pongs never arrive, not that they are mishandled on arrival.  Since the same peer pongs for one of your other tests, we can assume that it usually answers ping requests.  I suggest you cease using grep to filter this output interactively, and instead rely on some combination of tcpdump filters and post-processing of a raw capture.  As is, you may be hiding important clues because they fail to match your grep pattern.  If I were to guess, I would say that the source address used on the unanswered pings causes upstream to be able to route the pongs back to you.

----------

## Moriah

Thank you for the response.  I will look into your idea this evening, as I am busy until then.  I just used grep because I wa too lazy to look  up the pcap filter command I really needed.   :Embarassed: 

Also, if I go outside my own 96.11.110.96/29 nework and try to connect to 96.11.110.101 (my web server on my dmz) I fail to see anything with tcpdump.

So outbound originating packets use -m state --state ESTABLISHED,RELATED to get their reply, but dnat still has problems too.

----------

## Moriah

This was finally solved.  See:

https://forums.gentoo.org/viewtopic-p-8219104.html#8219104

----------

