# recent iptables changes broke firewall

## Moriah

I am running an iptables firewall with a fairly complicated firewall script.  The box had not been rebooted since about May 2007.  I had kept up to date with security updates.  A power failure that exceeded the capacity of the UPS batteries caused an unplanned reboot.  The firewall no longer works properly: it is not passing packets to its downstream mate over an openvpn tunnel.  If I turn off the firewall, the packets will pass.  I noticed the date on iptables is Jan 22, 2008, so this version of iptables never ran before the reboot.

Does anybody know what all changed with recent updates to iptables?  I really need to get this machine working, as it is the gateway to an entire network.

----------

## Hu

I suspect the firewall rules were changed and not saved, so the saved rules are wrong.  I suggest you begin the standard procedure for diagnosing NAT/VPN/firewall problems.  Show us what you have, explain your network topology, and show us traffic which is being blocked incorrectly.

----------

## schachti

 *Moriah wrote:*   

> The box had not been rebooted since about May 2007.  I had kept up to date with security updates.

 

How did you keep the kernel up-to-date (especially concerning the recent vmsplice exploit) without rebooting?

 *Moriah wrote:*   

> The firewall no longer works properly: it is not passing packets to its downstream mate over an openvpn tunnel.

 

Did you compile something as a module which is missing now, or did you forget to load the module? Is some error message given by the iptables script (or some error message in the syslog)? You can also add some rule to the tail of each iptables chain that logs everything that is not allowed to pass to hunt down this error.

----------

## noobstate

 *Moriah wrote:*   

> 
> 
> How did you keep the kernel up-to-date (especially concerning the recent vmsplice exploit) without rebooting?
> 
> 

 

was wondering that also .

ps (for original poster) if u updated the kernel u have to re-emerge iptables i think. and if u were only only saving firewall rules within iptables daemon yes it WILL over write it . 

its best to just make a stand alone script, then add it to init on boot that way nothing can touch it .

----------

## Moriah

This is not your plain vanilla setup.  The firewall script is 1286 lines long.  It has worked fine since May until the recent reboot.  I have been mostly out of town that whole time managing the network remotely, so I have been fearful to do a reboot lest I loose it and not be there to recover it.  This firewall script runs explicitly at every boot, so this is not a problem of a simple firewall hand typed directly into the console and then saved.

There is a machine at my isp running gentoo with a firewall (protection only, no nat, etc.) and openvpn.  The machine I am having trouble with gets a dynamic ip from my dsl provider (different isp from where the other machine is).  It then opens a tunnel via openvpn to the machine at the isp.  It is over this tunnel that my static ip's get routed, as they are pprovided by the isp where the tunnel server is located.

The firewall provides protection for the connection that the dhcp address comes from, so only the dhcp server and a couple of name servers and the gateway are able to communicate with it. 

Once the tunnel is up, the firewall also provides the usual protections for that circuit, as that is the main packet path.  

In addition, the firewall provides static nat for the dmz, and masquerades the choke firewall which provides final protection for the lan, which the choke also masquerades a second time.

Neither of the openvpn boxes had been rebooted since May or so until a couple of weeks ago, when storms caused an unscheduled power outage that exceeded the capacity of the UPS batteries.  I had been spending most of my time the past 2 weeks getting the box at the isp working again, as it has different problems after the reboot.  That box is now completely up to date, including kernel (2.6.23-gentoo-r8 ), and has had hardware remote controls put on it so that I can power cycle it and work the console remotely.

The box that is the gateway firewall and tunnel client at my network is the one giving me trouble now.  Its kernel is still 2.6.19-gentoo-r5, but its iptables is ver 1.3.8, dated 22-Jan-2008 (same version as on the box at the isp), which would have been the last time I did the emerge --sync ; emerge --update system ; emerge --update world ; thing.

Obviously, without a network connection, applying an update to this machine is nearly impossible.

Sorry for the long background story, but people give answers assuming I have a 5 or 10 line firewall that is not even in a script file, so I guess I have to explain.

I will now repeat my question: What has changed in recent updates to iptables?  Is the Jan-22 iptables incompatible with the packet filter in my old kernel?

Obviously, when I said I kept everything up to date, but did not reboot, then that meant everything except the kernel.    :Rolling Eyes: 

Everything is compiled into the kernel; there are no modules to worry about.

There are no error messages, either at boot time, or while it is running.

If I turn off the iptables on the client side of the tunnel, it works, but without any static or masquerading nat going on, it is rather worthless to the rest of my network.  This at least proves that the server is working, its openvpn is working, and its firewall is working.  At the client side, it proves that the dhcpcd is working and the openvpn is working, so it must be iptables or the packet filter itself.

So what has changed in either or both of those components?  I need to know so I can fix this.

Thanks!   :Very Happy: 

----------

## Moriah

Well, I built a new kernel, 2.6.23-gentoo-r3, and still no joy.    :Sad: 

However, I did notice that although the man page for iptables still has a description of the mangle table, when I tried to flush the mangle table with iptables -t mangle --flush, I got a complaint saying the table does not exist.

So, is the man page out of date for the code for iptables and the packet filter?  I am not actually using the mangle table in my firewall rules, but WHAT ELSE HAS CHANGED ???

Where are up to date docs on iptables and the packet filter?

----------

## Hu

 *noobstate wrote:*   

> ps (for original poster) if u updated the kernel u have to re-emerge iptables i think. and if u were only only saving firewall rules within iptables daemon yes it WILL over write it . 
> 
> its best to just make a stand alone script, then add it to init on boot that way nothing can touch it .

 

No, on all counts.  It is not necessary to rebuild iptables just because the kernel changed.  iptables is a userspace program that converts our requests into a form the kernel understands.  The saved rules are not part of any package, so they are not overwritten on upgrade.  It is not a good idea to have a freestanding script, because then changes are made in a non-atomic manner.  You should use the provided iptables initscripts to save and load the rules.

Moriah: as far as I know, nothing significant has been removed.  The mangle table is still in the kernel source, as are nat, filter, and raw.  However, it is possible to compile a kernel which does not have all four tables.  It appears that your new kernel has been compiled without the mangle table.  That is a change you caused, either by explicitly disabling the mangle table in the configuration or by not enabling it when it was moved to a new name.  For what it is worth, many people had temporary breaks in their NAT configuration when the options moved around.  Several features did change their configuration names between 2.6.19 and now, so if you did not adjust your kernel configuration accordingly, you may have lost some functionality.  You can regain that functionality by identifying the new names of the features and enabling them.

I mentioned the possibility of unsaved changes because I have seen many people have saved rules, either in a script or saved in /var/lib/iptables/rules-save, where the rules were 95% right, and then someone with root privilege used iptables commands interactively to cover that last 5%.  In such a situation, a power failure would erase that last 5% and leave you with rules that do not quite work.

I do not know if the man page for iptables is completely up-to-date, but it is close enough that I have not found any mistakes.

If you will not show us the saved rules and an example of blocked traffic, we can only offer general guidance about how your configuration may be wrong.  The kernel as a whole has changed quite a bit since 2.6.19, so it is unlikely you can easily find the change that broke your setup, assuming it is a kernel code change at all.  As you already noticed, the mangle table vanished on you due to configuration changes.  Your problem may be as simple as a missing feature that is not causing your rule load to fail.  I have seen this happen where someone had rules of the form:

```
iptables -A INPUT -m obscure-feature -j ACCEPT

# Drop everything else

iptables -A INPUT -j DROP
```

When support for the fictitious obscure-feature match was dropped, the script proceeded to create a one line firewall that dropped all traffic.  The same principle may apply here.  Without seeing your kernel configuration and firewall script, I can only speculate.

The easiest way for us to help you would be if you can post the following items:

- Output of zgrep -E '^[^#]' /proc/config.gz

- Output of iptables-save -c with the firewall running.  That is, run this when the network is not passing traffic correctly.

- Basic network topology.  In particular, we need the names of the interfaces and what network (LAN, WAN, VPN, etc.) is attached to each interface.  It would also be helpful to know what subnets are used by each of the non-public interfaces.  For example, do you use 192.168.0.0/16, 172.16.0.0/12, or 10.0.0.0/8 for your internal networks?

- Contents of your script that loads the firewall

----------

## Moriah

Thank you, Hu, for the enlightening information.  So NAT is different.  That would explain a lot!  This firewall static NATS or masquerades every packet that it forwards.  

 *Quote:*   

> For what it is worth, many people had temporary breaks in their NAT configuration when the options moved around. Several features did change their configuration names between 2.6.19 and now, so if you did not adjust your kernel configuration accordingly, you may have lost some functionality. You can regain that functionality by identifying the new names of the features and enabling them. 

 

Unfortunately, I am now 400 miles away from that machine and have no way to communicate with it until next weekend.  Meanwhile my entire network is still down.  Therefore, I cannot run any tests or experiments until then.

----------

## mmoufid

 *Hu wrote:*   

> It is not a good idea to have a freestanding script, because then changes are made in a non-atomic manner.  You should use the provided iptables initscripts to save and load the rules.

 

I've also been using this same technique, that is, using BASH scripting to assign IPs to network interfaces, for some time now. Like so:

```
IFC="/sbin/ifconfig"

AWK="/usr/bin/awk"

EXTIF="eth0"

EXTIP="`$IFC $EXTIF|$AWK /$EXTIF/'{next}//{split($0,a,":");split(a[2],a," ");print a[1];exit}'`"

```

I do this because my ISP assigns dynamic IP addresses and I don't want to have to edit anything at every boot. If I understand correctly, it's not possible to do this with the provided Gentoo IPtables init scripts?

Others may use this method for reasons of backwards-compatibility--it's simply how things have been done in the past, and adapting a 1286 line script seems counter-productive to me.

I'm assuming that by "atomic" you mean  *Quote:*   

> [...] which is guaranteed to complete successfully or not at all.

 

Does a properly written script along with a custom init script, like mine below, not also provide "atomicity"?

/etc/fw:

```
#!/bin/bash

# fw

# custom firewall script

# Distributed under the terms of the GNU General Public License v3

# 

# Usage: fw [IFACE]

EXTIF=$1

[...]

```

/etc/init.d/fw.eth0:

```
#!/sbin/runscript

IPT="/sbin/iptables"

IFACE="eth0"

depend() {

  need net.$IFACE

}

start() {

  ebegin "Starting firewall"

  einfo "Applying rules ..."

  /etc/fw $IFACE

  einfo "Done"

  eend $?

}

[...]

```

```
rc-update add fw.eth0 default
```

 *Quote:*   

> You should use the provided iptables initscripts to save and load the rules.

 Personally, I like the idea of Gentoo, as a metadistribution, standing for choice. I don't think it's proper to direct users away from a method that's well-established and just as good--if not better considering the reason I've described above. Using an IPtables script, or using the provided init scripts, is a matter of personal preference--a decision which should be left up to the user to make, not for anyone to impose on them.   :Wink: Last edited by mmoufid on Mon Feb 25, 2008 10:50 pm; edited 2 times in total

----------

## mmoufid

 *Moriah wrote:*   

> However, I did notice that although the man page for iptables still has a description of the mangle table, when I tried to flush the mangle table with iptables -t mangle --flush, I got a complaint saying the table does not exist.

 

To verify that you have a "mangle" table:

```
cat /proc/net/ip_tables_names
```

The default IPtables table is "filter". Others like "nat" or "mangle" must be explicitly stated. Perhaps you forgot to do this in your script?

Or, as Hu mentioned, the problem may be in your kernel. To check:

```
grep CONFIG_IP_NF_MANGLE /usr/src/linux/.config

```

----------

## Hu

 *mmoufid wrote:*   

>  *Hu wrote:*   It is not a good idea to have a freestanding script, because then changes are made in a non-atomic manner.  You should use the provided iptables initscripts to save and load the rules. 
> 
> I've also been using this same technique, that is, using BASH scripting to assign IPs to network interfaces, for some time now. Like so:
> 
> ```
> ...

 

Correct, the iptables init script does not support this.  For systems where you need to adjust the rules based on the IP address of one or more interfaces, I suggest using multiple stages:

- First, let the iptables init script load a very basic set of rules that does not need to know the IP address of the interface.  Generally, this set of rules is a "fail secure" configuration.  For systems which are directly exposed to the Internet, the secure failure path filters all traffic not associated with the next step.

- Let the DHCP client assign an address.

- Use a postup hook to replace the basic rules with rules derived from your IP address.  This hook can use iptables-restore to load a secondary set of rules from a file or a pipe.  In either case, these rules can be generated by a script from a firewall template.  In my case, the template has constructs like -A INPUT -d $MY_ADDRESS_HERE -j ACCEPT, where the $MY_ADDRESS_HERE is actually recorded in the template file.  The script looks up my real address and generates a working set of firewall rules from the template, then feeds that to iptables-restore.

 *mmoufid wrote:*   

> Others may use this method for reasons of backwards-compatibility--it's simply how things have been done in the past, and adapting a 1286 line script seems counter-productive to me.

 

I agree.  Converting an existing script by hand is not a good use of time.  However, once the script produces a working set of firewall rules, it should be straightforward to use iptables-save to take a snapshot of those rules so they can be reapplied without further use of the script.  I am a bit dubious as to whether all 1286 lines are needed for a properly functioning firewall, but that is a debate for another time.

 *mmoufid wrote:*   

> I'm assuming that by "atomic" you mean  *Quote:*   [...] which is guaranteed to complete successfully or not at all. 
> 
> Does a properly written script along with a custom init script, like mine below, not also provide "atomicity"?

 

Yes, that is what I meant.  It is possible that your script will be atomic, but I cannot say without seeing more of it.  The key issue, which I neglected to clarify in previous posts, is that iptables-restore loads all the rules in one step, so any errors cause the entire load to fail.  In every case I have seen, people advocating use of a handwritten script have a script which runs iptables commands one at a time, which is not atomic with respect to switching the firewall from the old rules to the new rules.  I deprecate the practice of using such scripts because a failure can leave the system with a firewall that has some rules loaded and other rules missing.  According to Murphy's Law, the rules which will fail to load are the ones that are most important to having the system work correctly.  For a NAT device, that is the address translation rules.  For a firewall, those are the DROP targets that keep malicious traffic from reaching protected services.

 *mmoufid wrote:*   

>  *Quote:*   You should use the provided iptables initscripts to save and load the rules. Personally, I like the idea of Gentoo, as a metadistribution, standing for choice. I don't think it's proper to direct users away from a method that's well-established and just as good--if not better considering the reason I've described above. Using an IPtables script, or using the provided init scripts, is a matter of personal preference--a decision which should be left up to the user to make, not for anyone to impose on them.  

 

Yes, users are welcome to choose their own path.  I disagree that having a separate script is inherently better, though it is certainly true that the provided init script is not sufficient for all users who have a dynamically assigned IP address.  However, I think it is also good to actively warn users when they are pursuing a path which can cause unnecessary trouble later on.  I never refuse to help someone solely on the grounds that they chose a path which I consider to be a poor choice, but I do try to warn them that they are making what I consider to be a bad decision.

----------

## Moriah

Just for the record, I will describe my configuration.

I have a gateway firewall that connects via dsl and a dhcp provided dynamic address.  I have to reconfigure the firewll rules if the dhcp assigned address changes, so I use the hook provided by dhcpcd to run a script that re-runs my firewall script whenever the dhcp assigned address changes.

Once the internet facing interface is up, openvpn builds a tunnel to another machine that I have colocated at a different isp from the dsl provider. (This messy arrangement is due to the tarrifs and costs of using the dsl provider to do everything; they are quite a rip off, but they own the copper pair, so I must use them to get connected to the internet via dsl.)

This second machine colocated at another isp tunnels my cidr block of static ip addresses so I can run my servers.  A bonus of using openvpn this way is I can use the compression provided by openvpn to about double my effective bandwidth for most traffic (email, web, etc.).

Once the tunnel is up, my gateway firewall static nats the static ip addresses to the servers on the dmz.  It also masquerades the choke firewall that separates the dmz from the lan.  This way, the choke is invisible to the internet.

The choke firewall -- another machine -- masquerades the lan and static nats any addresses on the dmz so tht the lan can address the servers.

The overall architecture of the firewall is deny-all and then accept selectively, which accounts for the large number of lines.  Allowing for comment lines, and continuation lines, together with lines of shell sripting code, there is maybe half as many firewall rules as line in the firewall script, maybe even less, although some rules are instantiated in for loops when the same rule applies to a number of ip addresses or ports.

I have been running this script since May of 2007, and its predecessor, which used L2TP for the tunnel instead of openvpn, since October of 2003.  I know the approach is solid.

An additional nice feature of this approach is that I can move to a different location that uses a different isp to connect to the internet -- via dsl, cable modem, or whatever -- and all I need to do is get a connection and a dhcp provided ip address, and the rest works the same way.  I have operated from 2 different states without having to change my servers' ip addresses.  One state was a cable modem connection, the other is the present dsl setup.

----------

## Moriah

I was doing routine maintenence on the gateway firewall and tunnel server I described above 2 nights ago, and I shut down the firewall for about a minute to run a test.  Much to my suprise, my blackberry started dinging, indicating it was receiving multiple emails.  I looked, and they were coming from my email server that is connected via that gateway/tunnel-server.  Wow!  That meant the packets were flowing!  Apparently, the firewall at the other end of the tunnel is working OK after all, and the real problem is at the tunnel-server end.  I checked, and I could get to my web server also.  Next I had a person at the location where the protected network is check and verify they they could get out to the internet.  Yes, everything was connected and working properly -- as long as the firewall at the tunnel server end was off.  After a couple of minutes of this fun, I turned the firewall back on at the tunnel-server, and as expected, the packets stopped flowing.

So now I know the problem is specifically with the iptables firewall at the tunnel-server.  I turn it off, and packets flow; I turn it on, and the flow stops.

This firewall is considerably simpler than the one at the tunnel-client end, so I was happy.  It should be easy to troubleshoot.

But not so.  This firewall was originally generated by a script written by Robert Ziegler, author of the New Riders book "Linux Firewalls".  I have made a few minor modifications to it over the years to accomodate configuration changes, but it is basicly the same as when it was new.  It has been in service since 2004.

So I went over the entire firewall script and could find nothing suspicious looking.  I manually flushed all the rules with iptables --flush and iptables -t nat --flush, and the firewall still blocked the flow.  When I shut it off with /etc/init.d/iptables stop, the flow resumed.  I even checked that the flush worked with iptables -v -n -L and iptables -v -t nat -n -L, and indeed, there were no rules listed.

Now I was beginning to suspect my kernel, so I built 2.6.23-gentoo-r9 and tried it.  I had been running 2.6.23-gentoo-r8, and a considerably older kernel before that.  The new r9 kernel did no better than the r8, so I reverted back to my old kernel from last spring, but it did not work either.

Now the only thing I have left to suspect is the userland iptables command itself.

Does anyone have any ideas or sugestions?

----------

