# iptables blocklist advice...

## Sadako

I'm playing around with using a blocklist from www.bluetack.co.uk, namely the level1 list which includes over 220,000 ip ranges, and dropping packets from them via iptables.

First of all, how bad an idea is it to add so many rules to iptables in the first place, and what kind of overhead will it incur?

Two ideas I'm playing around with to make it more efficient are;

a) only have it match NEW packets, ie with the following:

```
iptables -A INPUT -m state --state NEW -j BLACKLIST
```

The problem I see with this is that I'll need to add an equivalent rule and chain for OUTPUT, because NEW outbound packets to blocked a blocked ip will allowed further ESTABLISHED packets on INPUT, and this will mean adding another 220,000 odd rules...

(On the other hand, blocking packets on OUTPUT may be desireable anyways...)

b) splitting the BLOCKLIST chain into a series of subchains.

By this I mean the BLOCKLIST chain will contain lets say 16 different rules like 

```
iptables -A BLACKLIST -m iprange --src-range 16.0.0.0-31.255.255.255 -j BLACKLIST_16-31
```

one such rule for 16 different ranges of the first octet, and then have all the specific drop rules for each range in the blocklist within that range in the chain referred to (in the above example the BLACKLIST_16-31 chain).

The point of this being it should greatly reduce the number of actual rules compared per each NEW packet, especially since each NEW packet would have to go through all the +220,000 rules prior to being accepted otherwise.

It would probably need more than 16 subchains though, and perhaps having the rules in those subchians similarly splitted two may be wise...

Comments, ideas, anyone?

Is this viable at all?

Would iptables be problematic or degraded with so many rules?

----------

## mgrela

Read about the -m set matching module for iptables. The ebuild is called 'ipset'. It provides a fast hash table/tree of IP addresses you can match against in iptables rules. It can be useful in your situation.

----------

## Sadako

 *mgrela wrote:*   

> Read about the -m set matching module for iptables. The ebuild is called 'ipset'. It provides a fast hash table/tree of IP addresses you can match against in iptables rules. It can be useful in your situation.

 Thanks, I'm having a look at it now.

In fact, it looks kinda perfect, it should be a lot "faster", should make my iptables rules more managable (currently have almost half a million rules while playing around with this), and from  ipset.netfilter.org; *Quote:*   

> In order to drop traffic to-from banned networks or IP addresses, use IP sets in the raw table of netfilter.

 Perfect for what I'm doing!

Thanks again.

----------

## Sadako

Okay, I've been playing with this a bit, and thought I should report back, on the off chance anyone else is interested.

ipsets work wonderfully, but there were a couple of minor obstacles towards implementing them.

For one thing, each "set" type can only have 65536 elements, and in order to store each range as one element you need to use the nethash type with a cidr netmask.

This means converting form the lists like you'll find on bluetack.co.uk, which are of the type "description:192.168.2.128-192.168.148.235".

Before that though, you may need to convert from dos line terminators to unix, with dos2unix.

Then emerge the wonderful net-misc/aggregate-flim, and run the following, INFILE being the blocklist in unix format;

```
sed s/.*:// INFILE | aggregate-flim -i range > INFILE.cidr
```

This of course will have to generate more ranges, in my case from 222729 in the original blacklist to 224527 in cidr notation.

Next you'll have to split out the single IP addresses, ie those with a /32 netmask, as the nethash can only accept up to a /31.

We have to use those single addresses in a separate ipset, of a type other than nethash.

```
grep /32 INFILE.cidr | sed s:/32:: > INFILE.single

grep -v /32 INFILE.cidr > INFILE.netmask
```

To create the ipset of the single addresses, run the following;

```
/sbin/ipset -N SINGLE iphash

while read CIDR; do

        /sbin/ipset -A SINGLE ${CIDR}

done < INFILE.single
```

It's basically the same thing for creating the nethash ipsets, only "nethash" instead of "iphash" in the ipset -N command.

However, because of the 65536 element limit I had to split the 220,000-odd ranges up to more managable sizes.

I won't go into that in detail here, I've whipped up a quick and dirty shell script to do it, but I need to clean it up somewhat.

I'll post the cleaned up version when finished if anyone's interested, but the main thing to remember is to split it up into simple ranges for iptables to use, for example 48.0.0.0-63.255.255.255 for 48.0.0.0/4.

Anyways, for the iptables rules, here's a slightly simplified version of what I'm using (with very strict rules);

```
*filter

:INPUT DROP [0:0]

:FORWARD DROP [0:0]

:OUTPUT DROP [0:0]

:BLACKLIST_IN - [0:0]

:BLACKLIST_OUT - [0:0]

:TCP - [0:0]

:eth0_IN - [0:0]

:eth0_OUT - [0:0]

-A INPUT -m state --state INVALID -j DROP 

-A INPUT -i lo -j ACCEPT 

-A INPUT -p tcp -j TCP 

-A INPUT -i eth0 -j eth0_IN 

-A OUTPUT -m state --state INVALID -j DROP 

-A OUTPUT -o lo -j ACCEPT 

-A OUTPUT -o eth0 -j eth0_OUT 

-A OUTPUT -j REJECT --reject-with icmp-net-prohibited

-A BLACKLIST_IN -s 0.0.0.0/3 -m set --set 0-31 src -j DROP 

-A BLACKLIST_IN -s 32.0.0.0/3 -m set --set 32-63 src -j DROP 

-A BLACKLIST_IN -s 64.0.0.0/6 -m set --set 64-67 src -j DROP 

-A BLACKLIST_IN -s 68.0.0.0/6 -m set --set 68-71 src -j DROP 

-A BLACKLIST_IN -s 72.0.0.0/5 -m set --set 72-79 src -j DROP 

-A BLACKLIST_IN -s 80.0.0.0/4 -m set --set 80-95 src -j DROP 

-A BLACKLIST_IN -s 96.0.0.0/3 -m set --set 96-127 src -j DROP 

-A BLACKLIST_IN -s 128.0.0.0/2 -m set --set 128-191 src -j DROP 

-A BLACKLIST_IN -s 192.0.0.0/5 -m set --set 192-199 src -j DROP 

-A BLACKLIST_IN -s 200.0.0.0/5 -m set --set 200-207 src -j DROP 

-A BLACKLIST_IN -s 208.0.0.0/5 -m set --set 208-215 src -j DROP 

-A BLACKLIST_IN -s 216.0.0.0/5 -m set --set 216-223 src -j DROP

-A BLACKLIST_IN -m set --set SINGLE src -j DROP

-A BLACKLIST_IN -j ACCEPT

-A BLACKLIST_OUT -d 0.0.0.0/3 -m set --set 0-31 dst -j REJECT --reject-with icmp-host-prohibited 

-A BLACKLIST_OUT -d 32.0.0.0/3 -m set --set 32-63 dst -j REJECT --reject-with icmp-host-prohibited 

-A BLACKLIST_OUT -d 64.0.0.0/6 -m set --set 64-67 dst -j REJECT --reject-with icmp-host-prohibited 

-A BLACKLIST_OUT -d 68.0.0.0/6 -m set --set 68-71 dst -j REJECT --reject-with icmp-host-prohibited 

-A BLACKLIST_OUT -d 72.0.0.0/5 -m set --set 72-79 dst -j REJECT --reject-with icmp-host-prohibited 

-A BLACKLIST_OUT -d 80.0.0.0/4 -m set --set 80-95 dst -j REJECT --reject-with icmp-host-prohibited 

-A BLACKLIST_OUT -d 96.0.0.0/3 -m set --set 96-127 dst -j REJECT --reject-with icmp-host-prohibited 

-A BLACKLIST_OUT -d 128.0.0.0/2 -m set --set 128-191 dst -j REJECT --reject-with icmp-host-prohibited 

-A BLACKLIST_OUT -d 192.0.0.0/5 -m set --set 192-199 dst -j REJECT --reject-with icmp-host-prohibited 

-A BLACKLIST_OUT -d 200.0.0.0/5 -m set --set 200-207 dst -j REJECT --reject-with icmp-host-prohibited 

-A BLACKLIST_OUT -d 208.0.0.0/5 -m set --set 208-215 dst -j REJECT --reject-with icmp-host-prohibited 

-A BLACKLIST_OUT -d 216.0.0.0/5 -m set --set 216-223 dst -j REJECT --reject-with icmp-host-prohibited

-A BLACKLIST_OUT -m set --set SINGLE dst -j REJECT --reject-with icmp-host-prohibited

-A BLACKLIST_OUT -j ACCEPT

-A eth0_IN -p icmp -m state --state ESTABLISHED -j ACCEPT 

-A eth0_IN -s 192.168.1.254/32 -p udp -m udp --sport 53 -m state --state ESTABLISHED -j ACCEPT 

-A eth0_IN -p tcp -m tcp --sport 80 -m state --state ESTABLISHED -j ACCEPT 

-A eth0_IN -p tcp -m tcp --sport 443 -m state --state ESTABLISHED -j ACCEPT 

-A eth0_IN -p tcp -m tcp --sport 22 -m state --state ESTABLISHED -j ACCEPT 

-A eth0_IN -p tcp -m tcp --sport 6667 -m state --state ESTABLISHED -j ACCEPT

-A eth0_OUT -p icmp -m state --state NEW -j BLACKLIST_OUT 

-A eth0_OUT -d 192.168.1.254/32 -p udp -m udp --dport 53 -m state --state ESTABLISHED -j ACCEPT 

-A eth0_OUT -d 192.168.1.254/32 -p udp -m udp --dport 53 -m state --state NEW -j ACCEPT 

-A eth0_OUT -p tcp -m tcp --dport 80 -m state --state ESTABLISHED -j ACCEPT 

-A eth0_OUT -p tcp -m tcp --dport 80 -m state --state NEW -j BLACKLIST_OUT 

-A eth0_OUT -p tcp -m tcp --dport 443 -m state --state ESTABLISHED -j ACCEPT 

-A eth0_OUT -p tcp -m tcp --dport 443 -m state --state NEW -j BLACKLIST_OUT 

-A eth0_OUT -p tcp -m tcp --dport 22 -m state --state ESTABLISHED -j ACCEPT 

-A eth0_OUT -p tcp -m tcp --dport 22 -m state --state NEW -j BLACKLIST_OUT 

-A eth0_OUT -p tcp -m tcp --dport 6667 -m state --state ESTABLISHED -j ACCEPT 

-A eth0_OUT -p tcp -m tcp --dport 6667 -m state --state NEW -j BLACKLIST_OUT
```

If you need to omit certain addresses from the blocklists, the easiest way would probably be to create another iphash set of those you want accepted, and have a rule checking for and -j ACCEPT all those matching at the start of the blacklist chain.

Two things to remember with the ipset rules, for the sake of efficiency;

a) the "SINGLE" ipset should be the last one checked, as those with ranges will probably match far more addresses, even if the set has the same number of elements in the iphash set.

In other words, it's far more likely a packet will match the nethash set of ranges rather than the iphash set of single addresses, so that should be checked first.

b) the BLACKLIST check should only be used where you'd usually use a "-m state --state NEW -j ACCEPT", essentially -j BLACKLIST should replace all instances of -j ACCEPT for new packets.

Anyhow, this is just the method I came up with, but it works like a charm and appears to keep extra processing to a minimum.

Hope this is useful to someone else.

Also, thx again to mgrela for pointing me in the direction of ipset.

I'm going to shut up now.

----------

## think4urs11

 *Hopeless wrote:*   

> but the main thing to remember is to split it up into simple ranges for iptables to use, for example 48.0.0.0-63.255.255.255 for 48.0.0.0/4.

 

Did you use http://www.bluetack.co.uk/converter/ for that task?

----------

## Sadako

 *Think4UrS11 wrote:*   

>  *Hopeless wrote:*   but the main thing to remember is to split it up into simple ranges for iptables to use, for example 48.0.0.0-63.255.255.255 for 48.0.0.0/4. 
> 
> Did you use http://www.bluetack.co.uk/converter/ for that task?

 No, I didn't, once you have the blocklist `sed s/.*:// | aggregate-flim -i range` is a lot faster and more convenient.

I played with the converter when I started out on this, but the uncompressed blocklist is 12 mb, and the cidr output is ~6 mb.

I'd hate to see firefox attempting to cope with that.  :Wink: 

Probably would be useful for smaller lists, though.

----------

## mgrela

 *Hopeless wrote:*   

> I'm playing around with using a blocklist from www.bluetack.co.uk, namely the level1 list which includes over 220,000 ip ranges, and dropping packets from them via iptables.
> 
> First of all, how bad an idea is it to add so many rules to iptables in the first place, and what kind of overhead will it incur?
> 
> <SNIP>
> ...

 

If you consider your question answered please add the keyword [closed] to this topic to make it easier to filter out. You can do this by editing the subject of your first post in this topic. Thank you.

Best regards,

----------

## taipan67

Got any further refinements to this, Hopeless? Thanks for highlighting the existence of 'aggregate-flim', i'd never have found it otherwise.  :Cool: 

I'm not sure if i'm getting an error from the app, but when i process a 223,218-line level1.p2p file from http://iblocklist.com/lists.php (bluetack.co.uk seem to be moving their download links around...), the resultant level1.cidr is only 13,959-lines long, of which 222-lines are single ip's - which would fit nicely into just 2 sets:- L1-net.set (nethashed) and L1-ip.set (iphashed). In your experience, do these lists vary much in converted size from one update to another?

Additionally, i've been playing with sorting the list by descending range-size (sort -n -t '/' -k 2), as i read somewhere that it improves performance of set-matching, but this sort-method slightly jumbles up the initial ip-addresses in each sub-group of ranges (try it to see what i mean) - can you offer any clarification of just how iptables traverses sets in order to find matches? Would the type of sorting i've been able to achieve be better or worse than the original order? And if sorting is desirable, can you suggest any better methods?

Thanks in advance if you (or anyone else) can help with this.  :Wink: 

----------

## devilheart

i think that blocking any outgoing connection to blacklisted addresses and blocking any incoming connection with state NEW should be enough

----------

## taipan67

 *devilheart wrote:*   

> i think that blocking any outgoing connection to blacklisted addresses and blocking any incoming connection with state NEW should be enough

 

But then nobody would be able to connect to any servers you run, like a web-server, or an email-server, or an ssh-server to allow you to access your own box remotely, or a bittorrent-server, which is effectively what every bittorrent-client is.

My thinking is to pass all outbound traffic through blacklisting firewall-rules (using ipsets), and also have a rule for incoming traffic after a rule accepting 'established,related' traffic that would reject traffic from blacklisted addresses that failed the previous criteria, ie:- unacceptable new connections - acceptable ones could then traverse on to port-specific rules related to web-server traffic, ssh-traffic, bittorrent-traffic, etc.

The outbound-blacklist is desirable to stop unwanted incoming traffic being accepted by the 'established,related' rule. To just block it as incoming traffic before it goes through that rule would be a waste of outbound bandwidth if you tried to connect to an ip on the blacklist, coz it wouldn't be able to return  :Shocked:  .

----------

## devilheart

 *taipan67 wrote:*   

> But then nobody would be able to connect to any servers you run, like a web-server, or an email-server, or an ssh-server to allow you to access your own box remotely, or a bittorrent-server, which is effectively what every bittorrent-client is.

 you can always add some exceptions for those services

----------

## taipan67

 *devilheart wrote:*   

>  *taipan67 wrote:*   But then nobody would be able to connect to any servers you run, like a web-server, or an email-server, or an ssh-server to allow you to access your own box remotely, or a bittorrent-server, which is effectively what every bittorrent-client is. 
> 
> you can always add some exceptions for those services

 

That's precisely my point - exceptions have to come before the main rule (which is effectively the table-policy), but those exceptions still need a degree of protection from undesirable-ip's...

...The degree of protection afforded by blocklists can be varied by how many and which lists you use. The tricky part is ordering the rules to get the most efficient traffic-flow, which is why i want to accept 'established,related' traffic before it has to traverse the input-blocklists, which is hence why i need to filter output so i don't establish any connections to ip's i wouldn't want to accept new-connections from otherwise...  :Shocked: 

...Was that as convoluted to read as it was to write? Sorry not to be clearer.  :Laughing:   :Embarassed: 

----------

## taipan67

 *taipan67 wrote:*   

> ...i've been playing with sorting the list by descending range-size (sort -n -t '/' -k 2), as i read somewhere that it improves performance of set-matching...

 

Based on the output of...

```
ipset -S 'listname' > 'listname-recovery-file'
```

...i'd say that prior sorting makes no difference, as the output appears to have no discernable order that i can recognise - it's certainly nothing like the order used during set-creation. Does anybody know if 'ipset' is self-optimising in some way when it orders it's sets internally?  :Confused: 

----------

## devilheart

 *taipan67 wrote:*   

> That's precisely my point - exceptions have to come before the main rule (which is effectively the table-policy), but those exceptions still need a degree of protection from undesirable-ip's...
> 
> ...The degree of protection afforded by blocklists can be varied by how many and which lists you use. The tricky part is ordering the rules to get the most efficient traffic-flow, which is why i want to accept 'established,related' traffic before it has to traverse the input-blocklists, which is hence why i need to filter output so i don't establish any connections to ip's i wouldn't want to accept new-connections from otherwise... 
> 
> ...Was that as convoluted to read as it was to write? Sorry not to be clearer.  

 well, you can accept everything outgoing, set input default policy to drop, add rules for dropping packets from blacklistes addresses and then add a rule for related,established connections. in this way even if you are able to start a connection to a blacklisted address, any reply will be dropped

----------

