# I could use some help with this masquerading issue [SOLVED]

## drumgod

I am replacing my main router/desktop machine.  It is on a cable modem. The original machine masquerades my outbound traffic, has an ipsec tunnel to my office network, runs bind, ntp, dhcpd for my internal network and works fine albeit slowly.  I have a new machine to replace it.  Configs are all the same.  For testing I have dropped the ipsec and reduced my iptables script to:

```
# Disable forwarding

echo 0 >/proc/sys/net/ipv4/ip_forward

# Flush tables

iptables -F

iptables -F -t nat

iptables -X

# Set defaults

iptables -P INPUT ACCEPT

iptables -P FORWARD ACCEPT

iptables -P OUTPUT ACCEPT

# Masquerade traffic leaving eth0

iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

# Enable forwarding

echo "1" > /proc/sys/net/ipv4/ip_forward

```

From machines on the internal network I can ping external hosts and I can view *extremely* short web pages. (like "hello world" short)  Any traffic of substance does not work correctly.  I put trace rulesets on icmp and tcp and outbound traffic looks okay... inbound traffic is somewhat lacking.

I logged into a web server on the Internet and started tcpdump.  I also ran tcpdump on the external interface of the new router/desktop and then made an http request from an internal machine to the webserver on the Internetnet...  The web server shows a complete transaction with the get request followed by the page src being sent.  The external interface on the router/desktop however shows the outbound get request but only one data packet containing a small (middle) part of the page source coming back in.

The desktop/router itself works just fine... It's only masqueraded traffic going through it that seems to have it's reply traffic lost. (on the external side)  If I drop the old machine back in everything works fine.

Things I have tried:

Factory reset the cable modem.

Swapped all cables.

Replaced internal switch.

Set external interface MAC of new desktop/router to that of the old desktop/router.

Thing I will try:

Set up old machine with a webserver and new ip range as a test network.  Have it serve an address to the "external" side of the new desktop/router then see if my internal network can communicate to the test network masq-ed through the new box.

I'm stumped and could really use some suggestions... TIALast edited by drumgod on Sun Oct 21, 2012 12:07 am; edited 1 time in total

----------

## Hu

This sounds like an MTU/MSS issue.  Usually, that happens when fragmentation is needed and the ICMP Fragmentation Needed message is lost.  However, there could be an upstream device blocking the ICMP-FN message and your old machine happened to have a low enough MTU that things worked anyway.  For both the old and new machines, what is the output of /sbin/ip addr show, when plugged in and serving LAN requests to the Internet?

----------

## drumgod

Howdy.  Thanks for the quick reply.

Here's the old machine in place and working:

```
green etc # /sbin/ip addr show

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

2: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 00:50:2c:08:a7:1d brd ff:ff:ff:ff:ff:ff

    inet 172.30.30.1/24 brd 172.30.30.255 scope global eth1

3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 576 qdisc pfifo_fast state UP qlen 1000

    link/ether 00:a0:c9:99:c3:c8 brd ff:ff:ff:ff:ff:ff

    inet 173.21.252.16/24 brd 255.255.255.255 scope global eth0
```

and from dhcpcd in the logs:

```
Oct 20 13:55:14 green dhcpcd[2087]: eth0: broadcasting for a lease

Oct 20 13:55:14 green dhcpcd[2087]: eth0: offered 173.21.252.16 from 97.64.187.77

Oct 20 13:55:14 green dhcpcd[2087]: eth0: acknowledged 173.21.252.16 from 97.64.187.77

Oct 20 13:55:14 green dhcpcd[2087]: eth0: checking for 173.21.252.16

Oct 20 13:55:19 green dhcpcd[2087]: eth0: leased 173.21.252.16 for 341982 seconds

Oct 20 13:55:19 green dhcpcd[2101]: eth0: eth0: MTU set to 576
```

It will take me a couple minutes to swap in the new box...

----------

## drumgod

Okay, here is from the new machine:

```
grey drumgod # /sbin/ip addr show 

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 576 qdisc mq state UP qlen 1000

    link/ether 00:18:8b:25:38:77 brd ff:ff:ff:ff:ff:ff

    inet 173.23.131.146/24 brd 255.255.255.255 scope global eth0

3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 00:14:c1:2b:1e:91 brd ff:ff:ff:ff:ff:ff

    inet 172.30.30.1/24 brd 172.30.30.255 scope global eth1
```

and from the log:

```
Oct 20 16:15:58 grey dhcpcd[1845]: eth0: broadcasting for a lease

Oct 20 16:16:10 grey dhcpcd[1845]: eth0: offered 173.23.131.146 from 97.64.187.77

Oct 20 16:16:10 grey dhcpcd[1845]: eth0: acknowledged 173.23.131.146 from 97.64.187.77

Oct 20 16:16:10 grey dhcpcd[1845]: eth0: checking for 173.23.131.146

Oct 20 16:16:15 grey dhcpcd[1845]: eth0: leased 173.23.131.146 for 3600 seconds

Oct 20 16:16:15 grey dhcpcd[1871]: eth0: eth0: MTU set to 576

```

----------

## Hu

An MTU of 576 is obnoxiously low, but at least the old and new machines are consistent.  When you tested with the old machine, did you use the same set of rules as shown in the first post?  A network with low MTU may need to clamp the MSS of internal machines, if they expect a full MTU of 1500.

----------

## drumgod

Well shoot... It's working now.

Yea, the configs and addressing were the same on both boxes while testing. The low MTU comes from this option in my dhcpcd.conf:

```
# Respect the network MTU.

option interface_mtu
```

As you can see from the log snippets in the above posts dhcpcd set the MTU to 576.  Remove the above option and the MTU stays the normal 1500... and masquerading works with the new machine.  I have no idea why it would work with the low mtu on the old machine but not the new one.

Anyway, thank you Hu for the insight.  It was quite helpful.  I'll bring my FW rule sets back in and do some more testing, but it looks like it's working now...

----------

## Hu

Your dhcpcd is setting an MTU of 576 because the upstream DHCP server tells it to do that.  The interesting question is whether the upstream server should be doing that.  I have seen some DHCP servers, typically from consumer level ISPs, advertise an MTU of 576 when they should not do so.  Unfortunately, advertising a wrong MTU, whether too low or too high, causes undesirable behavior.  Similarly, overriding the advertised MTU to a value that is too low or too high can cause that same undesirable behavior.  In both cases, if the interface MTU is lower than the link MTU, then you fragment packets unnecessarily, which hurts performance by generating needless copies of the control headers.  On the other hand, if the interface MTU is higher than the link MTU, then you will generate packets which get rejected, forcing you to fragment them down to the right size.  This also hurts performance, since it means you send a packet, get it rejected, then resend it in proper form.  If the interface MTU were set properly, you would generate them with the right size on the first pass.

In the really bad case, using too high an MTU can cause connections to hang, exactly as you originally saw, because the generated packet is rejected, but some node along the path drops the notification that the sender needs to fragment and retry.  The sender is then left hanging.  If it retries at all (as TCP does), it will likely retry with the exact same segment size, causing another drop.  Linux provides some mechanisms to compensate for this when doing NAT, but the proper solution for this situation is to find the device that is dropping ICMP-FN packets and replace it with one that works properly.

----------

