# How Do I Block Tens of Thousands of IPs efficiently?

## amiatrome

First off, I must confess that the tens of thousands of ips I wish to block belong to the RIAA and gang.   :Embarassed: 

My aim is to find a firewall that when I receive a packet from an ip within the blocklist, it would stealth me. And when my system attempts to send out packets whose destination ip is within the blocklist, it would drop the packet. A simple way to import the ips(would be doing this pretty often) and view packets blocked would be a bonus. Thats where iptables come in I guess.

I have done quite a bit of browsing, but probably due to my relative inexperience in Linux, I didnt get very far. So I decided to seek a bit of pointers here again!

I had been considering Shorewall but its documentation on its blacklist function states very clearly that it filters source addresses only and is not suitable for blocking thousands of ips. Would someone pls point out what would be the optimum method of implementing my firewall?  Or maybe care to discuss which iptables frontend programs genertate the most efficient rules?  :Rolling Eyes: 

Really hoping to get some light on this as it was pretty easy to do it in Windows. I just switched over to linux but didnt expect myself to be stumped so quickly. Thanks!   :Smile: 

----------

## OdinsDream

I was googling for an answer to your question, and I had typed in "iptables blacklist" thinking I'd come up with some generic solution, but boy was I surprised to come across someone just as fed up with RIAA!

http://techfocus.org/comments.php?id=3662&replyid=194&catid=1

It's really amusing, because these people, the RIAA, are really digging themselves a hole, alienating all potential customers. Anyway, it's not exactly the same, since it's an htaccess solution, but it might be helpful as a starting point for your IP range, if you don't already have one.

You may want to look at the source code for the nocatauth project, used for community WIFI projects. This does some type of automatic iptables scripting. If it sees a MAC address it's unfamiliar with, all DNS queries are forced to resolve to your Terms of Service page, and after you agree to it (not exactly sure of the method they're using in this step), iptables allows your traffic through.

So, maybe you'd be able to pick up some tips on automatic iptables updating with that code? Good luck! Please post your progress, it's such a shame to see threads like this end with something like "ok got it, thanks," with no tips on what to do.

----------

## Chris W

The thousands of IP addresses most likely fall into a series of small, contiguous blocks with a few isolated addresses scattered about.   This is certainly the case in the list referenced above.  The blocks can each be dealt with using a suitable netmask in an Netfilter rule, e.g. : 

```
iptables -A INPUT -i eth0 --source 208.050.066.224/27 -j DROP

iptables -A OUTPUT -i eth0 --destination 208.050.066.224/27 -j REJECT
```

  The example drops everything from 208.050.066.224 to 208.050.066.255.  Incidentally, the DROP target is the equivalent of the marketing term "stealth", and REJECT is a more friendly version used to block your internal clients (it returns an error to the sender).  Then a rule for each straggler address and you're done.  The article referenced above also has a comment showing how this could be done.

You can use the LOG target to log traffic matching the rules.

 *amiatrome wrote:*   

> Really hoping to get some light on this as it was pretty easy to do it in Windows. 

 

 :Question:   How is this easily done in Windows?

----------

## Esben

I should be easy to set up in IPtables, and logging, too. In what medium do you have your IPs? It would probably be easy to write a small script that converted an text-input file to iptable rules...

----------

## amiatrome

Hi Odinsdream, I really must agree with you that the Riaa is alienating its customers. But in my country, they had succeeded the moment they launched a lawsuit against the 12 yr old girl.

I have got friends who are avid file sharers in campus, but have been woken up in the middle of the night by campus officers, who want to check their computers.because they are acting on complaints from Warner Bros. Most simple users do not even know what a firewall is for, they only install it because someone said so. In the end, the scare tactics worked and they are herded like sheep back to buying overpriced CDs and DVDs. Don't worry, if i find an easy way to implement a huge blocklist, I will sharre it wirh the entire linux p2p community!   :Very Happy: 

Hi Chris. guess I made a mistake in my first post, I actually have to blocl tens of thousands blocks of ips. I get my blocklist from Bluetack using their Blocklist Manager http://bluetack.co.uk/blm.html and I implemented it in a flash with my outpost firewall and its Blockpost plugin, using a alternative kernel driver to handle the huge ip range. Do take a look here.   :Wink:  http://www.outpostfirewall.com/forum/showthread.php?s=&threadid=7229

Helllo Esben! I will most probably using one of the formats available in Blocklist Convertor http://bluetack.co.uk/convert.html

I am really hoping to find an easy way to implement a huge blocklist. I thought I had it nailed with Shorewall but it was not to be. I have tried out Guarddog etc but I did not find an easy way to import the huge blocklist(either that or it could be that I just didnt dig deep enough  :Sad:  ). As a result, I have not beed able to test the efficiency of the rules they create. Is there a program or a method that you would all know off that could help me? Pls do take into account simple importing and rules efficiency. Thanks!

----------

## Esben

What I would do is simply create a (perl)script that takes an input file in as simple format as available and create iptable rules from that. I think you could place all these rules in a separate chain o r something for easy access. Then setup the retrieval, conversion import up in cron, and your firewall is always RIAA safe.

Though it would be easier to stop stealing --- but so much less fun   :Smile: 

----------

## amiatrome

I agree!   :Very Happy: 

Guess there really isnt any easier way out than for me to get down and dirty with iptables and cron. I will start doing my homework on them rightaway!

But relishing a glimmer of hope, if anyone knows of a easier way, pls do let me in on your great idea!

----------

## berarul

Hello.

I have a sort of thingie that you could cut and paste. It parses files for stuff ... lol ... sorry I'm a bit tired. The thingie is a gentoo initscript which integrates dhcp, squid and iptables. The stuff is usually IP.IP.IP.IP_PORT or IP.IP.IP.IP_MAC_HOST you get the idea. You can easily adapt it to your needs.

http://www.digifin.ro/nexus/nexus.tar.bz2

You could also look around the forums for it. There are comments and is based on the IP MASQ how to, though few traces of it remain.

----------

## naitram

You could also check out peerguardian

found out about it because i have a server that's blacklisted...though i'm not sure why.  It looks like they have an alpha release for linux.  Note that i haven't looked into it much, but it sounds like something that could do what you want.[/url]

----------

## amiatrome

Peerguardian! I was wondering when their site would go online again. So it finally has.  Thanks naitram. I will go check 'em out again.

Thanks berarul, I will go check it out too. I am so new. So many things to check out!!

----------

## berarul

But still ... I am still curious ... how was easy to do in windows?   :Question:   :Question:   :Question:   :Question: 

----------

## amiatrome

Berarul!  :Very Happy:   I agree I didnt give a very good explanation previously. I shall try again.

Ok, In windows, I would install Outpost.

http://www.agnitum.com/download/outpost1.html#(At the right side of the page, there is a link explaining the diff between the "completely free" vs. "pro" version). 

Outpost is considered a solid firewall like Sygate by the Windows hacking community as opposed to the likes of ZoneAlarm. So I would urge everyone running a secondary Windows box to switch over to Outpost, Pro preferably.   :Smile: 

Then I would add a Outpost plugin, called Blockpost(crack required  :Wink:  ), and follow simple instructions to use the alternative kernel driver.

Outpost plugin page: http://www.agnitum.com/products/outpost/plugins.html

Blockpost kermel driver for huge blocklist: http://www.outpostfirewall.com/forum/showthread.php?s=&threadid=9908

I would then import the blocklist.

Blockpost's blocklist format is fully supported by Bluetack http://bluetack.co.uk/convert.html so its very easy to import the blocklist too.

If my memory serves me, I took less than 10mins to get my firewall and p2psafeguards up! Complete with logs, alerts and all. And Outpost has some useful free plugins too, so thats another bonus!

Hope I cleared some of your doubts, Berarul.   :Smile: 

----------

## berarul

there is such a thing as a windows hacking comunity?! Maybe you mean cracking.  :Smile: 

Anyway ... you could do that with iptables easy ... suppose you have a formatted list and a shell script just run the script on the list. No cracks and illegal software involved  :Smile: 

Suppose you would have iptables ... kernel functionality in linux and the shell script (this would be your blackpost plugin) and the list it would probably take you less in linux.

----------

## amiatrome

No. I really mean hacking! You know, the those who do exploits, trojans and viruses that linux users laugh at.   :Wink: 

I did read up on using scripts and iptables to implement my needs, but I did some scouring and the results werent encouraging. They cited very long import times.Do take a look at Pg 3 of the following thread.

http://www.methlabs.org/forums/showthread.php?t=648

Of course, this problem is currently being dealt with by JFM of methlabs.org. Thanks to naitram for pointing it out.

http://www.methlabs.org/forums/showthread.php?t=1369

I forgot to add my apparent lack of scripting skill. But I will go do my homework on it.

----------

## andrew_j_w

No, you mean crackers.

From the Hackers Dictionary.

 *Quote:*   

> 
> 
> hacker: [originally, someone who makes furniture with an axe] n.
> 
>    1. A person who enjoys exploring the details of programmable
> ...

 

 *Quote:*   

> 
> 
> cracker: n. One who breaks security on a system.  Coined ca. 1985
> 
>    by hackers in defense against journalistic misuse of {hacker}
> ...

 

Andrew

----------

## berarul

we have the stupid (IT) media and windows developers for this. How many times haven't you heared (in my country at least and in interviews taken to antivirus developers who say "to protect you from hackers", "a hacker broke into the ...", bla bla ... never once heared the word cracker).

----------

## amiatrome

I stand corrected but let not go off-topic.  :Smile: 

berarul, is it really as simple as you mentioned? Would it take a lot of work to create a script to import the rules into the kernel quickly?

What do you think could be casusing the long import times mentioned in the metlab forums?

----------

## berarul

No ... just take shit from the link I gave you ... as for high importing times, just try it. I'm not sure I understand what you mean by high importing times? If by that you mean time required to put them into the firewall but not to operate them, I wouldn't see a problem with high importing times. you don't change them hourly, do you? And you do have 7days+ uptime usually? Anyways to answer your question:

1) No it would not take a lot of time since you should adapt the scripts from the link I gave you

2) I have no idea. Maybe after you try it you can share the results with us so we will know.

P.S. What is long importing time? 1 hour?

----------

## amiatrome

Previous methods with iptables took about an hour as described by JFM. I did add 2 links in two previous posts ago which describe the situation I am worried about.

 *Quote:*   

> The converter may then go on for a minute or two. If you have automatic mode on the program will run for about 5 minutes.
> 
> Trust me, this is as fast as we can get it, most other methods would take about 1 hour!

 

http://www.methlabs.org/forums/showthread.php?p=6326#post6326

 He is currently trying to implement PeerGuardian in Linux using iptables and has come up with a faster solution.   :Very Happy:   Again, the link to the details is already in two previous posts ago.

----------

## berarul

You never really know untill you try it

----------

## amiatrome

Don't you mean "You never really know until you try my script"?

I wouldnt try the old scripts in the methlabs forums since a few of them have already did and reported bad times. Bad algorithms run bad on any machine.

----------

## OdinsDream

Ok, there seem to be some bad feelings going back and forth, but really, this thread isn't about feelings, it's about a very simple iptables question, and personally, I think it's a pretty interesting idea.

Having said that, and looking at the links provided, I don't think one needs an entire PeerGuardian system for linux. Iptables works very nicely on its own, as far as I've seen.

Any solution you work out is not going to be nearly the same as one in Windows. For one reason, linux tools, especially iptables, and especially in respect to batch operations, works primarily on the command-line. This is where you get actual work done. Nobody is going to provide you a link to any graphical tool that imports this stuff for you. People are going to provide you links to iptables manuals, bash scripting resources, and other very powerful tools.

And they're right to. These are solutions to your question, you're just failing to see it. I strongly suggest stopping by Borders and reading a little about iptables. Once you get to know how to block a single IP address, you can then easily script the rest in Bash, Perl, or any other language.

As for how much time it takes, this isn't about algorithm implementation, it's about linear insertion of iptables rules. This should not take an extreme amount of time, it's simply forming some iptables rules, line-by-line, and having the iptables system interpret them. O(n). If you're that concerned about speed, you're going to need to get into the actual iptables code and work something else out.

Really, it's not daunting, it's very fun to do this kind of stuff, and to get something working that you wrote in a night. I know, I know, it's not windows. Nobody claimed it was, but it is a lot more interesting, a lot more enriching, and a lot more fun.

----------

## amiatrome

thanks for stepping in OdinsDream. I get what you mean, I was prepared for lots of scripting the moment I totally removed Windows.  :Smile: 

I wanted to implement my blocklist with iptables right from the beginning. But I was worried about the O(n) iptables commands. Since I would be updating my p2p blocklist often, I wanted to seek others views for better alternative iptables frontends that did something different. PG uses iptables and seems to have addressed some of my doubts.

 *Quote:*   

> it hooks into iptables directly, and doesn't simply run several thousand "iptables" commands.

 

Again, thanks for your understanding, Odinsdream.

----------

## Chris W

You probably want to look at the iptables-save and iptables-restore commands.   As their names suggest, they save and restore netfilter rules en masse.   The save file format is straight text, so it is not inconceivable that you script a conversion from your blocklist to the save file format and then use iptables-restore to load it.   This is probably about as fast  as you will get--it's essentially the same as thousands of iptables commands without the overhead of thousands of process invocations.

The time taken is not large in any case.  I wrote a quick test script to add 1024 DROP rules on my XP 2400+ machine; it took 3.05 seconds elapsed to run.  I saved the rules and reloaded them with iptables-restore in 0.27 seconds.

----------

## amiatrome

Yup. I saw them earlier and they would definitely be a good timesaver once my rulesets are entered.  :Smile: 

Just for discussion, suppose the  problem isnt O(n) anymore.

 *Quote:*   

> 
> 
> The main problem with running a shell script that contains iptables rules is that each invocation of iptables within the script will first extract the whole rule-set from the Netfilter kernel space, and after this, it will insert or append rules, or do whatever change to the rule-set that is needed by this specific command. Finally, it will insert the new rule-set from its own memory into kernel space. Using a shell script, this is done for each and every rule that we want to insert, and for each time we do this, it takes more time to extract and insert the rule-set.

 

Making a very very rough linear estimate, 

( 10000 (src) DROP + 10000 (dest) DROP ) = ~60s

SInce I know people who block up to 57K ranges, and if linearity cannot be aproximated to anymore, this may affect performance?   :Question: 

----------

## JetAce44

This thread certainly caught my eye, as I know its much more efficient to use a true firewall (such as iptables) in order to stop inbound traffic.

I too did some googling, and came up with a very interesting perl script:

Home Site: http://dessent.net/linblock/

Man Page:  http://www.dessent.net/linblock/linblock.html

I think this is the solution to all of our problems, as it can be added as a cron job to silently update the list whenever you want. The nice thing is that it also uses iptables to block everything. I figure on trying it out in a few minutes, I'll post back on my results.

/Edit: Updated link, first mentioned was a man page.

/Edit #2:

Everything installed and in working order. Took about 60 seconds to convert the list, and had to manually download it because for some reason the script would not grab the list needed. I recomend all of you guys grab it  :Wink: 

----------

## amiatrome

Thanks! Now that is exactly what I was looking for! It was added to Freshmeat right after my last post! I almost gave up when no one could understand my concerns about the rules insertion time.

 *Quote:*   

> Posted: Mon Jul 05, 2004 3:55 am

 

 *Quote:*   

> Added: Mon, Jul 5th 2004 04:48 PDT

 

It still took you 60s? It's almost the same as my very rough and optimistic linear estimate for the bash scripts method(it probably takes longer). Wow, how many ips are u blocking?

----------

## silentbob

Just been reading through this thread, interesting. I've had a look at the linblock perl script and it looks good, but then I found out that the PeerGuardian people have released a llinux version "PeerGuardian for Linux version 0.21 Alpha" (http://methlabs.org/forums/showthread.php?t=1370).

I'm not sure which one to use, both of these have been recently developed (in the past month or so), I guess it's a trial and error see which one works best?

----------

## silentbob

I have my iptables setup using the linblock to block quite a few thousand lines from a guarding.p2p file. If I do an "iptables -nL" there are over 14,000 lines which looks great but it means that restoring the rules takes one or two minutes using 100% CPU usage every time I boot the system. Could be better I think!!

----------

## amiatrome

Maybe you could try using the iptables-save and iptables-restore. Linblock or PG would insert the rules fast. And iptables-save and iptables-restore to save and restore your rules fast.

----------

## wjholden

This Linblock stuff looks like really fantastic software.  I just installed it, but it has no ebuild.  Compared to most installations, I call http://dessent.net/linblock/linblock.html a fairly difficult one since it had many dependencies I had to fulfill on the fly.  I have no experience building ebuilds, so I am asking all those who have written ebuilds whether they think I should spend time working on this or not.

This software is really exciting because it is said on this thread to consume less system resources than PeerGuardian and that's an issue for all of us, plus it automatically updates itself (if you set it to do so) without user interference.  Looks like a great Protowall equivalent for Linux.

----------

## Bob P

I just found this thread and I find it quite interesting.  I have been working on blocking an entire range of IP addresses using Shorewall, the front-end for IP tables.  I've run into some problems implementing the feature to ban ranges of IP addresses and I'm hoping that someone can help.

First of all, I'd like to preface my post with verification that my kernel is properly configured to support banning by ranges of IP addresses:

```
# shorewall check

Loading /usr/share/shorewall/functions...

Processing /etc/shorewall/params ...

Processing /etc/shorewall/shorewall.conf...

Loading Modules...

Shorewall has detected the following iptables/netfilter capabilities:

   NAT: Available

   Packet Mangling: Available

   Multi-port Match: Available

   Extended Multi-port Match: Not available

   Connection Tracking Match: Available

   Packet Type Match: Available

   Policy Match: Not available

   Physdev Match: Not available

 * IP range Match: Available *

```

and i also see that my DROP rules for the offending IP have been tested and appear to be good:

```
# shorewall check

...

Validating rules file...

   Rule "DROP net:64.233.160.0/19 fw all" checked.

   Rule "DROP net:66.246.0.0/16 fw all" checked.

   Rule "DROP net:66.249.0.0/16 fw all" checked.

   Rule "DROP net:80.81.16.106 fw all" checked.

   Rule "DROP net:212.27.41.35 fw all" checked.

   Rule "DROP net:66.249.71.3 fw all" checked.

   Rule "DROP net:66.249.64.0-66.249.64.255 fw all" checked.

   Rule "DROP net:66.249.66.0-66.249.66.255 fw all" checked.

   Rule "DROP net:66.249.71.0-66.249.71.255 fw all" checked.

...

Configuration Validated

Notice:  The 'check' command is provided to catch

         obvious errors in a Shorewall configuration.

         It is not designed to catch all possible errors

         so please don't submit problem reports about

         error conditions that 'check' doesn't find
```

the problem is that the offending IP continues to be logged by my http server:

```
66.249.71.3 - - [07/Sep/2005:05:48:16 +0000] "GET /robots.txt HTTP/1.0" 200 127 "" "Googlebot/2.1 (+http://www.google.com/bot.html)"

66.249.71.3 - - [07/Sep/2005:05:48:16 +0000] "GET / HTTP/1.0" 200 5447 "" "Googlebot/2.1 (+http://www.google.com/bot.html)"

66.249.64.52 - - [07/Sep/2005:06:46:13 +0000] "GET / HTTP/1.0" 200 5447 "" "Googlebot/2.1 (+http://www.google.com/bot.html)"

66.249.64.13 - - [07/Sep/2005:07:07:22 +0000] "GET /robots.txt HTTP/1.0" 200 127 "" "Googlebot/2.1 (+http://www.google.com/bot.html)"

66.249.64.13 - - [07/Sep/2005:07:07:22 +0000] "GET / HTTP/1.0" 200 5447 "" "Googlebot/2.1 (+http://www.google.com/bot.html)"
```

can anyone explain why the packets aren't being dropped by the firewall?

----------

## Taladar

If you block lots and lots of IP ranges you might want to sort them so the big ranges come first. This should reduce the average time that is spent with the rules per packet. Perhaps you can also use a combination of accept and drop to further reduce the number of rules. 

Another idead would be using user defined rules to implement some sort of binary search for the IP range a packet is in. That should reduce the time for each packet from O(n) (1/2 n average) to O(log(n)). But before optimizing like that you should test wether latency gets worse by a significant amount due to the firewall.

I believe that time is much more important than the time needed for "telling" iptables what the rules are. Though the combination of accept and drop rules to replace lots of drop rules might help there too.

----------

## Bob P

can you explain why shorewall is failing to drop the observed packets in spite of what appears to be an adequate set of rules?  i am hoping that someone can point out a simple configuration error on my part.  unfortuantely, it looks like the rules are all valid and shorewall is failing to do its job.    :Confused: 

----------

## quanttrom

Hey guys,

I have a related issue here. I installed linblock and everything. Got the set of rules and applied them.

Now the problem is....it runs slow. Downloading anything is nearly impossible and surfing is kind of annoying since I get 100% CPU usage for every page and it slows down opening up the page and everything.

Am I including the wrong iptables module here or..what am I doing wrong...?

is it really that slow..

Oh Yeah

```
Linux lucky 2.6.14-gentoo-r2 i686 AMD Athlon(TM) XP 2100+ AuthenticAMD GNU/Linux

quanttrom@lucky ~ $ free -m

total       used       free     shared    buffers     cached

Mem:           758        688         69          0         64        317

-/+ buffers/cache:        306        451

Swap:          251          0        250

```

----------

## Bob P

in my case, the problem is [SOLVED].  i had one bad rule high in my shorewall rules table that was allowing the unwanted IP address to pass through.  fixing the rules table solved the problem.

on the subject of speed, inefficient firewall rules are notorious for adding time to the amount of time it takes for the firewall to do its job.  with a firewall, you really want to be as efficient as possible so that throughput is not slowed down.  if you're going to be adding a large set of complex rules, it will take time for the firewall to run though the table, and the difference could be perceptible by the user.

in cases like that, it may be a good idea to:

1.  run a separate pass-through firewall on a dedicated machine, rather than putting a single-ended firewall on your server

2.  use a faster PC for the firewall

3.  use IP tables instead of a perl script.  i could be wrong on this, but i think IP tables would be faster.  maybe someone else can chime in on this.

hth

----------

## quanttrom

First I would like to say that the firewall was running that slow due to a..not a mistake but rather negligance on my part.

quoting from linblock homepage:

 *Quote:*   

> 
> 
> By default, filtering occurs against every packet sent and received. This is the "cautious default" because some people are not using iptables in a "stateful" mode. However, most people do load the ip_conntrack module which maintains state of all the connections. If this applies in your situation, you can reduce the filtering load significantly by only filtering packets in the NEW and RELATED states. The following command can be used after running the script to do this.
> 
> iptables -R INPUT 1 -m state --state NEW,RELATED -j antip2p
> ...

 

I ignored to used that and therefore when trying to use the network I got quite a bit CPU usage and noticable delay. After running that my machine runts MUCH MUCH MUCH faster. There is a giant VERY noticable difference.

I'm a newbie but I decided to play with the iptables and see how things are working and how I can do add rules by hand so I blocked a single IP and found out that in my case having that IP only in the INPUT chain for iptables was not enough to protect me from accessing it. In the case of BitTorrent I can be supplied with this IP by the tracker and my comptuer will connect to it and effectively bypass my blocked IPs. If the IP tries to connect to me I'll be fine but I want to be protected against both. So I took the linblock script and modified it a bit. Remember I'm a newbie so it's not a great modification. I basically added the state matching into the perl script so a user would now have to type in 

```
iptables -R INPUT 1 -m state --state NEW,RELATED -j antip2p
```

 after and I made it generate a second chain with a new name that has the sources /destination field switched. Then I added a line that will add this set of rules to my OUPUT chain. Now I think I'm blocking both cases, me attempting to access them and them attempting to access me. I don't know if this modified version of the script will be of any use to anybody but if anybody wants to have a look or use it I'll be more then happy to provide you with it.

Thanks

P.S. Is the requirement of a second set of rules for OUTPUT caused by a bad rule or is what I'm doing making sense?

----------

