# Efficiently use a live blacklist feed in iptables (w/ ipset)

## Bones McCracker

[Edit: Nov. 2011: Updated the 3 main scripts to ipset-6 syntax]

[Edit: Nov. 2012: Update to incorporate code improvements suggested by Truc]

Background on ipset (if you're familiar with it, skip this):

I'm not affiliated with the netfilter team.  I'm writing this simply because there's not much instructive documentation (e.g. tutorials, how-tos) available pertaining to the use of ipsets, so I'm sharing what I've learned by playing with it, hopefully it will save people some time.

ipset is an extremely useful plugin to iptables, particularly if you want to have a firewall rule that matches against a large set of addresses and/or ports, or if you want to dynamically change the addresses and/or ports that a rule matches against.

It lets you create huge lists of ip addresses and/or ports (with tens of thousands of entries or more) which are stored in a tiny piece of ram with extreme efficiency.  In your iptables rules, you can then simply refer to the lists by name, and the entire list is checked with remarkable speed and in a single netfilter rule.  Also, you can change the contents of the list while the firewall is running.

iptables is actually just the user interface to netfilter.  When you write an iptables rule that lists multiple addresses or ports to be blocked (other than "from-to" ranges or netmasks), each item in the list is actually translated into its own netfilter rule, each of which must be processed.  Unnecessary processing of this type can really slow down your traffic, since every new connection (or in some cases, every packet) has to be processed through these rules.

Although it possible to add and remove iptables rules while the firewall is running, it is not very efficient to do so.  This is how most firewalls (i.e. iptables "front-ends") add and remove entries to their "dynamic blacklist".  Blacklist a new address?  That's a new rule that must be processed for every connection.  You can't "edit" an iptables rule in a running firewall to change the addresses or ports that it matches.  Ipset is, in my opinion, by far the best way to run blacklists, whether static, periodically updated from the Internet, or dynamically managed by your firewall or intrusion detection rules.

Also, while iptables provides for "address ranges" and port ranges (in a from-to, netmask, or CIDR format), it cannot efficiently handle long lists of non-contiguous addresses.  Try writing iptables rules to block 3,000 specific addresses that have nothing in common.  With ipset, this is trivial.  It's also efficient, because you can check a packet against the ipset in a single netfilter action (as opposed to 3,000) much, much faster.

Ipset is made by the same netfilter team that makes iptables.  It is actually a collection of additional netfilter-related kernel modules (additional "matches" and "targets").  It is part of the Xtables add-on package.  In Gentoo, you can get it simiply by emerging ipset (after your "make modules_install") and enabling the appropriate kernel modules.

There are different types of ipsets for storing different types of information (e.g., for random ip addresses, for addresses that are from the same netblock, for random blocks of addresses, for same-sized blocks of addresses, for random ports, etc.).

You can group multiple ipsets of different types into a single "setlist" (an ipset of type list:set) which is still treated as a single ipset by iptables (therefore, you only need one rule to match packets against multiple ipsets).  You can bind ipsets together (e.g. a list of addresses and a list of ports).  The man page explains it all.

Here is some basic information, and I will provide a couple of practical examples of managing a basic ipset for use in a firewall.

http://ipset.netfilter.org/ipset.man.html#index

An example of updating an ipset in an automated fashion, from an internet source:

There are many ways to use ipsets.  This script is just an example.  This script is intended to periodically (hourly) update an ipset used as a "blacklist" in a firewall (while the firewall is running and actively processing).  The list in this case is a list of "Class C"-sized networks (i.e. CIDR /24 blocks of addresses), published hourly by DShield.org, listing the top blocks of networks from which port-scanning activity has been coming in the last 3 days.  It's a tiny list as ipsets go, but it serves the purpose of this example (ipsets can contain many thousands of entries).

That's all incidental.  What's important is that it's a list of /24 netblocks, and we want to blacklist them.  There is a type of ipset called an "iphash" (hash:ip) that is very efficient and handling lists of same-sized networks (i.e., the ipset efficiently contains a list of networks that have the same netmask, in this case /24), so we'll use that hash:ip type of ipset.  We'll use wget to retrieve the block list only if it's been updated.  Then, if we've got a new list, we'll load it up to the firewall.  Since we can instantaneously swap the contents of one ipset for another, that's how we'll update the live firewall -- we'll parse each address out of the downloaded block list and add it to a temporary ipset, then swap the contents of the temporary ipset into the live ipset in the running firewall, instantly updating the firewall's blacklist en masse.

In later posts, there are other examples that are variations on the theme, demonstrating the use of different types of ipsets and other related concepts.  This, however, is good place to start.

This script runs hourly by cron on my system, about five minutes after DShield publishes its hourly update (presently, between HH+15m - HH:20m).  For logging purposes it uses "logger", which you can install, or you can substitute what you want or modify the logging lines to simply echo to your log file.

```

#! /bin/bash

# /usr/local/sbin/block

# BoneKracker

# Rev. 11 October 2012

# Tested with ipset 6.13

# Purpose: Load DShield.org Recommended Block List into an ipset in a running

# firewall.  That list contains the networks from which the most malicious

# traffic is being reported by DShield participants.

# Notes: Call this from crontab. Feed updated every 15 minutes.

# netmask=24: dshield's list is all class C networks

# hashsize=64: default is 1024 but 64 is more than needed here

target="http://feeds.dshield.org/block.txt"

ipset_params="hash:ip --netmask 24 --hashsize 64"

filename=$(basename ${target})

firewall_ipset=${filename%.*}           # ipset will be filename minus ext

data_dir="/var/tmp/${firewall_ipset}"   # data directory will be same

data_file="${data_dir}/${filename}"

# if data directory does not exist, create it

mkdir -pm 0750 ${data_dir}

# function to get modification time of the file in log-friendly format

# stderr redirected in case file is not present

get_timestamp() {

    date -r $1 +%m/%d' '%R

}

# file modification time on server is preserved during wget download

[ -w $data_file ] && old_timestamp=$(get_timestamp ${data_file})

# fetch file only if newer than the version we already have

wget -qNP ${data_dir} ${target}

if [ "$?" -ne "0" ]; then

    logger -p cron.err "IPSet: ${firewall_ipset} wget failed."

    exit 1

fi

timestamp=$(get_timestamp ${data_file})

# compare timestamps because wget returns success even if no newer file

if [ "${timestamp}" != "${old_timestamp}" ]; then

    temp_ipset="${firewall_ipset}_temp"

    ipset create ${temp_ipset} ${ipset_params}

    networks=$(sed -rn 's/(^([0-9]{1,3}\.){3}[0-9]{1,3}).*$/\1/p' ${data_file})

    for net in $networks; do

        ipset add ${temp_ipset} ${net}

    done

    # if ipset does not exist, create it

    ipset create -exist ${firewall_ipset} ${ipset_params}

    # swap the temp ipset for the live one

    ipset swap ${temp_ipset} ${firewall_ipset}

    ipset destroy ${temp_ipset}

    # log the file modification time for use in minimizing lag in cron schedule

    logger -p cron.notice "IPSet: ${firewall_ipset} updated (as of: ${timestamp})."

fi

```

I run it using cron (I use vixie-cron, so I put this in /etc/cron.d):

```
# /etc/cron.d/update_block

# BoneKracker

# 31 March 2011

# Global variables

SHELL=/bin/bash

PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin

MAILTO=root

HOME=/

# Every 15 minutes, poll for update to dshield

# block list, and update firewall blacklist.

# To check timing: grep "block updated" /var/log/crond

# Last I checked it was being published at:

# H+:08                                  

# H+:23

# H+:38

# H+:53

09 *  * * * root    /usr/local/sbin/block

24 *  * * * root    /usr/local/sbin/block

39 *  * * * root    /usr/local/sbin/block

54 *  * * * root    /usr/local/sbin/block
```

Last edited by Bones McCracker on Fri Oct 12, 2012 3:21 am; edited 19 times in total

----------

## ssteinberg

This is actually really interesting, well done. I'd be interested in more feeds.

----------

## Bones McCracker

Here's another.  This one is used to accurately match (block or allow) traffic from or to entire countries.

This script takes a single command-line argument: the 2-letter IANA country code (the top-level domain, such as "us" or "cn").  Based on that, it downloads a list of all the networks of various sizes that are registered to that country code with the regional registrar.  Last I checked, these lists are published twice daily by the ipdeny.com project.  These are larger lists than the previous example.

Like the list in the previous example, this is a list of networks.  However, unlike the previous example, they are networks which vary in size (i.e., have various netmasks).  For that reason, we will use the nethash type of ipset (formally now called hash:net, although referring to it the old way still works).

Selecting the appropriate hashsize: In the example above, where the list is always a specific size and very small, I specifically set the hashsize to something smaller than the default.  We will not do that here, because the default hashsize (1024) is appropriate for all but the smallest of the country codes, and the hashsize is just a starting point anyway (as an ipset gets larger, the hashsize is adjusted upward dynamically).  Setting the hashsize manually only makes sense when you know the approximate size of the ipset in advance, and the point of doing so is to: (a) save a few KiB of ram in the rare case where you are using a list you know is smaller than the default; or (b) to save a bit of rehashing (which occurs as the ipset grows) in the cases where you know the list is larger than the default.   It is also possible to optimize this dynamic resizing process in various ways using the "--probes" and "--resize" options, but the defaults are fine, and we don't need to go into that now.

This script also demonstrates the use of of the setlist type of ipset.  All of the country-specific ipsets are added to a single, combined setlist, so we can refer to the whole group as a single ipset (and matching against it is a single efficient netfilter action).  In the script you'll see where the required setlist is created (if the setlist does not already exist) and where the ipsets are added to the setlist (if they are not already a member of the setlist).

Other than taking a command-line paramater, using a different type of ipset that is of more of a normal size, and employing a hierarchy of ipsets (i.e., using a setlist), this script is pretty much the same as the one above.  I actually run this script twice a day from cron, about an hour after ipdeny.com updates the lists.  I run it multiple times, fetching the lists for several countries and creating a corresponding ipset for each (these are just example countries here; no offense intended to anyone):

```
# /etc/cron.d/update_ipdeny

# BoneKracker

# 4 November 2011

# Twice per day, poll for update to ipdeny

# block lists, and update firewall blacklist.

# To check timing: grep "IPSet: .. updated" /var/log/crond.log

# Last I checked by running hourly, they were being published as follows:

# 05:05 - 05:07

# 15:05 - 15:07

countries="cn ru ir ng"

09  5 * * * root    for c in $countries; do /usr/local/sbin/ipdeny $c; done

09 15 * * * root    for c in $countries; do /usr/local/sbin/ipdeny $c; done
```

Those four ipsets could be used directly in iptables (for example, by using "+cn" anywhere an ip address would normally go), but as noted above, I have combined them into a single "setlist".  That way, I can simply refer to all the networks in all the countries as "+ipdeny"  (there are over 7,000 networks in the example I give here).  To check if a packet matches any of those, Netfilter needs only execute a single action: one call to check the setlist, which responds with a match or no-match almost instantaneously.

In fact, using ipset in general is trivial, with the exception of initially gaining an understanding the different set types and what they are for (that takes a thorough reading of the man page).

So here is that second script:

```

#! /bin/bash

# /usr/local/sbin/ipdeny

# BoneKracker

# Rev. 11 October 2012

# Tested with ipset 6.13

# Purpose: Load ip networks registered in a country into an ipset and load that

# ipset into a setlist containing several such ipsets, while this setlist is

# being used in a running firewall.

#

# Notes: Call this from crontab. Feed updated about 05:07 and 15:07 daily.

#

# Usage: 'ipdeny <TLD>' (where TLD is top-level national domain, such as "us")

[ -n "$1" ] && firewall_ipset="$1" || exit 1

ipset_params="hash:net"

filename="${firewall_ipset}.zone"       # on server, files are "us.zone" etc.

target="http://www.ipdeny.com/ipblocks/data/countries/${filename}"

data_dir="/var/tmp/ipdeny"

data_file="${data_dir}/${filename}"

# if data directory does not exist, create it

mkdir -pm 0750 ${data_dir}

# function to get modification time of the file in log-friendly format

get_timestamp() {

    date -r $1 +%m/%d' '%R

}

# file modification time on server is preserved during wget download

[ -w ${data_file} ] && old_timestamp=$(get_timestamp ${data_file})

# fetch file only if newer than the version we already have

wget -qNP ${data_dir} ${target}

if [ "$?" -ne "0" ]; then

    logger -p cron.err "IPSet: ${firewall_ipset} wget failed."

    exit 1

fi

timestamp=$(get_timestamp ${data_file})

# compare timestamps because wget returns success even if no newer file

if [ "${timestamp}" != "${old_timestamp}" ]; then

    temp_ipset="${firewall_ipset}_temp"

    ipset create ${temp_ipset} ${ipset_params}

    while read network; do

        ipset add ${temp_ipset} ${network}

    done < ${data_file}

    # if ipset does not exist, create it

    ipset create ${firewall_ipset} ${ipset_params} 2>/dev/null

    # swap the temp ipset for the live one

    ipset swap ${temp_ipset} ${firewall_ipset}

    ipset destroy ${temp_ipset}

    # if the setlist does not exit, create it

    ipset create -exist ipdeny list:set

    # if the ipset is not already in the setlist, add it

    ipset add -exist ipdeny ${firewall_ipset}

    # log the file modification time for use in minimizing lag in cron schedule

    logger -p cron.notice "IPSet: ${firewall_ipset} updated (as of: ${timestamp})."

fi

```

Last edited by Bones McCracker on Fri Oct 12, 2012 3:59 am; edited 12 times in total

----------

## ssteinberg

Thanks. I was aware of filtering based on country. Doesn't appeal to me personally as much as a dynamic block list, it is just too general, but people should find it useful. Well done with the scripts.

----------

## Bones McCracker

 *ssteinberg wrote:*   

> Thanks. I was aware of filtering based on country. Doesn't appeal to me personally as much as a dynamic block list, it is just too general, but people should find it useful. Well done with the scripts.

 

You are right.  It serves to demonstrate the basics, though.

Beyond that, it's also easy to dynamically blacklist attackers, etc., by writing iptables rules that use the "SET" target (as opposed to the "set" match).  The basic iptables options are:

--match-set (compare a packet to an ipset)

--add-set (add address to an ipset)

--del-set (remove address from ipset)

Gentoo provides a rudimentary initscript that saves ipsets upon shutdown and restores them on startup.  Some firewall tools, such as Shorewall, have built-in facilities for loading and saving ipsets when the firewall is started or stopped.  You can use those, or manage them yourself (just make sure they are loaded up before iptables).

This inistscript is specific to the init system used by Gentoo Linux, and other Linux distributions provide their own startup scripts.  The latest version (as of this edit) accommodates setlist type ipsets (which much be destroyed before the sets they contain can be destroyed).  If you are installing ipset manually, you can use this as a model.

```
#!/sbin/runscript

extra_commands="save"

IPSET_SAVE=${IPSET_SAVE:-/var/lib/ipset/rules-save}

depend() {

    before iptables ip6tables

    use logger

}

checkconfig() {

    if [ ! -f "${IPSET_SAVE}" ] ; then

        eerror "Not starting ${SVCNAME}. First create some rules then run:"

        eerror "/etc/init.d/${SVCNAME} save"

        return 1

    fi

    return 0

}

start() {

    checkconfig || return 1

    ebegin "Loading ipset session"

    ipset restore < "${IPSET_SAVE}"

    eend $?

}

stop() {

    # check if there are any references to current sets

    if ! ipset list | gawk '

        ($1 == "References:") { refcnt += $2 }

        ($1 == "Type:" && $2 == "list:set") { set = 1 }

        (scan) { if ($0 != "") setcnt++; else { scan = 0; set = 0 } }

        (set && $1 == "Members:") {scan = 1}

        END { if ((refcnt - setcnt) > 0) exit 1 }

    '; then

        eerror "ipset is in use, can't stop"

        return 1

    fi

    if [ "${SAVE_ON_STOP}" = "yes" ] ; then

        save || return 1

    fi

    ebegin "Removing kernel IP sets"

    ipset flush

    ipset destroy

    eend $?

}

save() {

    ebegin "Saving ipset session"

    touch "${IPSET_SAVE}"

    chmod 0600 "${IPSET_SAVE}"

    ipset save > "${IPSET_SAVE}"

    eend $?

}
```

Last edited by Bones McCracker on Sat Jan 07, 2012 1:22 am; edited 3 times in total

----------

## Bones McCracker

Here is another that's more practically useful.  This script creates an ipset that can be used to block all bogons (not just rfc1918 private IP addresses, but every network block that is unassignable or has not yet been assigned by the regional authorities).  These addresses are often used by botnets, spammers, and so on.  It creates a fairly large ipset of a complex type, so it takes tens of seconds to initially load up the temporary ipset (the swapout operation is still virtually instantaneous, as are queries).

```

#! /bin/bash

# /usr/local/sbin/fullbogons-ipv4

# BoneKracker

# Rev. 11 October 2012

# Tested with ipset 6.13

# Purpose: Periodically update an ipset used in a running firewall to block

# bogons. Bogons are addresses that nobody should be using on the public

# Internet because they are either private, not to be assigned, or have

# not yet been assigned.

#

# Notes: Call this from crontab. Feed updated every 4 hours.

target="http://www.team-cymru.org/Services/Bogons/fullbogons-ipv4.txt"

ipset_params="hash:net"

filename=$(basename ${target})

firewall_ipset=${filename%.*}           # ipset will be filename minus ext

data_dir="/var/tmp/${firewall_ipset}"   # data directory will be same

data_file="${data_dir}/${filename}"

# if data directory does not exist, create it

mkdir -pm 0750 ${data_dir}

# function to get modification time of the file in log-friendly format

get_timestamp() {

    date -r $1 +%m/%d' '%R

}

# file modification time on server is preserved during wget download

[ -w ${data_file} ] && old_timestamp=$(get_timestamp ${data_file})

# fetch file only if newer than the version we already have

wget -qNP ${data_dir} ${target}

if [ "$?" -ne "0" ]; then

    logger -p cron.err "IPSet: ${firewall_ipset} wget failed."

    exit 1

fi

timestamp=$(get_timestamp ${data_file})

# compare timestamps because wget returns success even if no newer file

if [ "${timestamp}" != "${old_timestamp}" ]; then

    temp_ipset="${firewall_ipset}_temp"

    ipset create ${temp_ipset} ${ipset_params}

    #sed -i '/^#/d' ${data_file}            # strip comments

    sed -ri '/^[#< \t]|^$/d' ${data_file}   # occasionally the file has been xhtml

    while read network; do

        ipset add ${temp_ipset} ${network}

    done < ${data_file}

    # if ipset does not exist, create it

    ipset create -exist ${firewall_ipset} ${ipset_params}

    # swap the temp ipset for the live one

    ipset swap ${temp_ipset} ${firewall_ipset}

    ipset destroy ${temp_ipset}

    # log the file modification time for use in minimizing lag in cron schedule

    logger -p cron.notice "IPSet: ${firewall_ipset} updated (as of: ${timestamp})."

fi

```

This is the crontab I use:

```
# /etc/cron.d/update_bogons

# BoneKracker

# 16 May 2011

# Every four hours, poll for update to ipv4-fullbogons

# block list, and update firewall blacklist.

# Last I checked by running hourly, it was being published as follows:

# 00:48 - 00:50

# 04:48 - 04:50

# 08:48 - 08:50

# 12:48 - 12:50

# 16:48 - 16:50

# 20:48 - 20:50

#

# To check timing:

# zgrep "fullbogons-ipv4 updated" /var/log/old_logs/cron*

# Global variables

SHELL=/bin/bash

PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin

MAILTO=root

HOME=/

52 0  * * * root    fullbogons-ipv4

52 4  * * * root    fullbogons-ipv4

52 8  * * * root    fullbogons-ipv4

52 12 * * * root   fullbogons-ipv4

52 16 * * * root   fullbogons-ipv4

52 20 * * * root   fullbogons-ipv4
```

----------

## Bones McCracker

If anyone is wondering how I came up with the ${ipset_params}, the answer is trial-and-error.  In the meantime, I'll share the limited understanding I have gained.  I should also point out that I've been working exclusively with hash-type ipsets and the iptreemap, so my optimization insights do not extend to other types.

I would be appreciative of any tips regarding this by a more knowledgeable person, on how to identify optimal values for of these and similar ipset parameters.

When you build an ipset that is one of the "hash" types, do an 'ipset -L' and look at it's size.  The default size is 1024 (which I assume to be bytes).  If your list is small and you guess that a smaller hash could store it, or if the hashsize has been dynamically "grown" to be substantially larger (e.g. well into the multiple megabyte range), then you may want to try to optimize it (i.e., cause the process of loading the ipset to create a more efficient hash).

This is not necessary, since ipset will dynamically grow and rehash an ipset as entries are added (I haven't tested to see if it will dynamically shrink them as entries are removed, but I doubt it).  When all is said and done, pursuing this optimization process might cut the size of an ipset in half, and since they typically stay resident in RAM, this can free up some tens of megabytes of RAM.  Whether that's worth your time is up to you.

There may be a better way to do this, but I basically run the process of loading the ipset multiple times, with varying parameters, trying to get a smaller hash size.  Living by the rule "default is good", and noting that there is typically a point of diminishing returns on varying from them, one can bracket one's way into a reasonable value without too much effort.  Basically the process would be:

```
# ipset -X <your_ipset>

# ipset -N <your_ipset> <ipset_type> --hashsize <hash_size> --resize <resize_percent> --probes <probes>
```

See the man page for an explanation of the parameters.  I have found that all three can have an impact of the resulting size of the hash.

Then you'd run your loop that loads up the ipset.  It may be useful to "time" this, to test if varying parameters makes loading up the ipset take more or less time.  For example:

```
time while [ $((--i)) -ge 0 ]; do /sbin/ipset --add temporary_ipset ${networks[i]}; done
```

Then look at the resulting ipset's hash size:

```
# ipset -L <your_ipset> | head
```

----------

## mimosinnet

I find these scripts really useful. I am learning some perl and I have made a perl version of them. I started with perl on Christmas, as a hobby, so do not expect too much of this version. In the the exercise, I have learned some perl, some bash, and really enjoyed deciphering the regex expressions, still a mystery to me! 

Thanks for this great scripts and the introduction to ipsets!

----------

## Bones McCracker

Glad you found them useful, and thanks for the credit on your page and in your scripts.  I'd be interested to know how they compare in terms of performance.

----------

## mimosinnet

 *BoneKracker wrote:*   

> I'd be interested to know how they compare in terms of performance.

 

Thanks for the suggestion! I have created a test.sh script that calls the three bash scripts:

```
# cat test.sh 

#!/bin/bash

./Ipset_Dshield.sh

echo "Dshiedl"

./Ipset_bogons.sh

echo "Bogons"

./Ipset_Regions.sh cn

./Ipset_Regions.sh vn

echo "Regions"

/etc/init.d/ipset save
```

and executed both the test.sh and Ipset.pl scripts with the command 'time'. These are the results:

 *Quote:*   

> ./test.sh  0,65s user 0,87s system 16% cpu 9,023 total
> 
> ./test.sh  0,76s user 0,93s system 17% cpu 9,876 total
> 
> ./test.sh  0,71s user 0,90s system 17% cpu 9,295 total
> ...

 

The bash script tends to be faster and uses less resources than the perl script. ( Bash 1, Perl 0!   :Wink:  )

This could also be because I am practicing Object Oriented Perl, and loading Moose library for this. 

Cheers!

Update: I have rewritten the script without using Moose. These are the results:

```

perl Set_ip.pl  0,40s user 2,20s system 15% cpu 16,770 total

perl Set_ip.pl  0,50s user 2,62s system 12% cpu 24,782 total

perl Set_ip.pl  0,51s user 2,44s system  8% cpu 36,539 total

perl Set_ip.pl  0,43s user 2,07s system  5% cpu 44,494 total

perl Set_ip.pl  0,35s user 2,00s system 11% cpu 20,693 total

perl Set_ip.pl  0,49s user 2,30s system  8% cpu 31,933 total

perl Set_ip.pl  0,51s user 2,19s system  4% cpu 59,530 total

perl Set_ip.pl  0,48s user 2,43s system  8% cpu 33,630 total
```

Moose is the guilty one!  :Wink: 

----------

## Bones McCracker

Interesting.

----------

## truc

 *BoneKracker wrote:*   

> (snip...)

 

Intersting lecture thanks, I've been meaning to try ipset for a while now, guess it's the time:)

Anyway, a few comments on your script though.

```
# if data directory does not exist, create it

/bin/mkdir -m 0750 ${data_dir} 2>/dev/null

```

You should probably use

```
/bin/mkdir -m 0750 -p "${data_dir}"
```

 instead(note the -p and the absence of stderr redirection)

```
# function to get modification time of the file in log-friendly format

get_timestamp() {

    timestamp=$(/bin/date -r ${data_file} +%m/%d' '%R 2>/dev/null)

}

# file modification time on server is preserved during wget download

get_timestamp

old_timestamp=${timestamp}

```

This is not really important here, but it is counter intuitive that get_timestamp actually defines the timestamp variable.

You could probably use something like this instead(and again, why hidding stderr? it's there for a reason, it helps debugging when something goes wrong)

```
get_timestamp() {

   /bin/date -r "$1" +'%m/%d %R'

}

# file modification time on server is preserved during wget download

old_timestamp=$(get_timestamp "$data_file")
```

Last comment, in the code below, there is no need to use bash arrays (and if you don't, your script is no longer pure bash but pure sh which is somewhat better since you share it)

```

   networks=( $(/bin/sed -rn 's/(^([0-9]{1,3}.){3}[0-9]{1,3}).*$/\1/p' ${data_file}) )

   i=${#networks[*]}

   while [ $((--i)) -ge 0 ]; do

      /usr/sbin/ipset add ${temp_ipset} ${networks[i]}

   done

```

```

   networks=$(/bin/sed -rn 's/(^([0-9]{1,3}.){3}[0-9]{1,3}).*$/\1/p' "${data_file}")

   for net in $networks; do

      /usr/sbin/ipset add ${temp_ipset} ${net}

   done

```

Two more things about it:

	=> your regex is wrong you really want to match a . (dot) with \. and not any character with .

```
networks=$(/bin/sed -rn 's/(^([0-9]{1,3}\.){3}[0-9]{1,3}).*$/\1/p' "${data_file}")
```

And here is how I would say it, I find it easier to understand, but that's personnal opinion;)

```
networks=$(/bin/sed -rn '/^([0-9]{1,3}\.){3}[0-9]{1,3})/ { s/[^0-9.].*//; p }' "${data_file}")
```

	=> And, in the code above you don't really need to use the variable networks:

```

   for net in $(/bin/sed -rn 's/(^([0-9]{1,3}\.){3}[0-9]{1,3}).*$/\1/p' "${data_file}"); do

      /usr/sbin/ipset add ${temp_ipset} ${net}

   done

```

or even

```

   /bin/sed -rn 's/(^([0-9]{1,3}\.){3}[0-9]{1,3}).*$/\1/p' "${data_file}" | while read net; do

      /usr/sbin/ipset add ${temp_ipset} ${net}

   done

```

----------

## Bones McCracker

Truc, it took me a long time to get back to this, but thank you for the excellent corrections and suggestions.  I am incorporating all of them (with one exception, and one in modified form, as below).

Exception: I got rid of the BASH array as you suggested (don't know what I was thinking there), but I left the $networks variable in place because I think it's more apparent what's going on that way.

Modified: The stderr redirection in the get_timestamp function was there to allow processing to continue if the file is not present (which is a normal condition on first run and when the user might purge the data file from /var/tmp).  I took it out per your suggestion, so I made the assignment of "old_timestamp" (which calls the function) conditional on the presence of the data file.

Thank you.

----------

## elmar283

Thank you for the script. When I tried it I got the next error:

```

elmarotter@masterserver /etc/cron.d $ sudo /usr/local/sbin/block 

ipset v6.8: Kernel error received: Invalid argument

ipset v6.8: Kernel error received: Invalid argument

ipset v6.8: Kernel error received: Invalid argument

ipset v6.8: Kernel error received: Invalid argument

ipset v6.8: Kernel error received: Invalid argument

ipset v6.8: Kernel error received: Invalid argument

ipset v6.8: Kernel error received: Invalid argument

ipset v6.8: Kernel error received: Invalid argument

ipset v6.8: Kernel error received: Invalid argument

ipset v6.8: Kernel error received: Invalid argument

ipset v6.8: Kernel error received: Invalid argument

ipset v6.8: Kernel error received: Invalid argument

ipset v6.8: Kernel error received: Invalid argument

ipset v6.8: Kernel error received: Invalid argument

ipset v6.8: Kernel error received: Invalid argument

ipset v6.8: Kernel error received: Invalid argument

ipset v6.8: Kernel error received: Invalid argument

ipset v6.8: Kernel error received: Invalid argument

ipset v6.8: Kernel error received: Invalid argument

ipset v6.8: Kernel error received: Invalid argument

ipset v6.8: Kernel error received: Invalid argument

ipset v6.8: Kernel error received: Invalid argument

ipset v6.8: Kernel error received: Invalid argument

ipset v6.8: Kernel error received: Invalid argument

```

I might have forgotten some kernel function. My iptables  works fine, so that can't be it.

This is my .config: http://eotter1979.xs4all.nl/bestanden/forums.gentoo.org/config

----------

## khayyam

 *elmar283 wrote:*   

> I might have forgotten some kernel function. My iptables  works fine, so that can't be it.
> 
> This is my .config: http://eotter1979.xs4all.nl/bestanden/forums.gentoo.org/config

 

elmar283 ... you need ipset enabled and your .config shows: CONFIG_IP_SET is not set

best ... khay

----------

## elmar283

Thank you. It works fine now!  :Smile: 

----------

## Rain91

Could you post the updated script?

----------

## elmar283

I changed the script so I could put it in my cron width 'crontab -e'.

This are my working scripts:

```

elmarotter@masterserver ~ $ cat /usr/local/sbin/block 

#! /bin/bash 

SHELL=/bin/bash

PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin

MAILTO=root

HOME=/

# /usr/local/sbin/block 

# BoneKracker 

# Rev. 11 October 2012 

# Tested with ipset 6.13 

# Purpose: Load DShield.org Recommended Block List into an ipset in a running 

# firewall.  That list contains the networks from which the most malicious 

# traffic is being reported by DShield participants. 

# Notes: Call this from crontab. Feed updated every 15 minutes. 

# netmask=24: dshield's list is all class C networks 

# hashsize=64: default is 1024 but 64 is more than needed here 

target="http://feeds.dshield.org/block.txt" 

ipset_params="hash:ip --netmask 24 --hashsize 64" 

filename=$(basename ${target}) 

firewall_ipset=${filename%.*}           # ipset will be filename minus ext 

data_dir="/var/tmp/${firewall_ipset}"   # data directory will be same 

data_file="${data_dir}/${filename}" 

# if data directory does not exist, create it 

mkdir -pm 0750 ${data_dir} 

# function to get modification time of the file in log-friendly format 

# stderr redirected in case file is not present 

get_timestamp() { 

    date -r $1 +%m/%d' '%R 

} 

# file modification time on server is preserved during wget download 

[ -w $data_file ] && old_timestamp=$(get_timestamp ${data_file}) 

# fetch file only if newer than the version we already have 

wget -qNP ${data_dir} ${target} 

if [ "$?" -ne "0" ]; then 

    logger -p cron.err "IPSet: ${firewall_ipset} wget failed." 

    exit 1 

fi 

timestamp=$(get_timestamp ${data_file}) 

# compare timestamps because wget returns success even if no newer file 

if [ "${timestamp}" != "${old_timestamp}" ]; then 

    temp_ipset="${firewall_ipset}_temp" 

    ipset create ${temp_ipset} ${ipset_params} 

    networks=$(sed -rn 's/(^([0-9]{1,3}\.){3}[0-9]{1,3}).*$/\1/p' ${data_file}) 

    for net in $networks; do 

        ipset add ${temp_ipset} ${net} 

    done 

    # if ipset does not exist, create it 

    ipset create -exist ${firewall_ipset} ${ipset_params} 

    # swap the temp ipset for the live one 

    ipset swap ${temp_ipset} ${firewall_ipset} 

    ipset destroy ${temp_ipset} 

    # log the file modification time for use in minimizing lag in cron schedule 

    logger -p cron.notice "IPSet: ${firewall_ipset} updated (as of: ${timestamp})." 

fi 
```

part of crontab:

```

09 *  * * *  /usr/local/sbin/block

24 *  * * *  /usr/local/sbin/block

39 *  * * *  /usr/local/sbin/block

54 *  * * *  /usr/local/sbin/block

```

----------

## Bones McCracker

 *Rain91 wrote:*   

> Could you post the updated script?

 

It wasn't the script that was messed up; it was his kernel configuration.

Also, maybe he's using a different version of cron, or a different method of employing it, but I don't believe most people should need to change the scripts in order to use them with cron.  As I see it, those variables belong in the crontab, not in the scripts.

----------

## upengan78

hello,

Can I use this http://www.wizcrafts.net/exploited-servers-iptables-blocklist.html in addition to http://feeds.dshield.org/block.txt ? Thanks for the steps by the way, I got it working on my box but I just felt the list on block.txt isn't very comprehensive. 

ipset list

```
Name: block

Type: hash:ip

Header: family inet hashsize 64 maxelem 65536 netmask 24 

Size in memory: 1368

References: 0

Members:

82.221.99.0

150.164.168.0

61.147.112.0

125.64.12.0

70.88.227.0

186.202.164.0

91.143.199.0

120.86.151.0

61.236.64.0

222.211.95.0

61.150.76.0

222.173.120.0

199.192.241.0

223.78.153.0

216.144.247.0

182.213.176.0

122.154.101.0

74.63.224.0

114.84.107.0

199.241.186.0
```

----------

## Bones McCracker

Sure.  That looks fine to me.   That's what I intended, that these examples would help people create and maintain their own, and in general make use of ipsets.

----------

## upengan78

Sorry to post into this old thread. I noticed this morning target="http://feeds.dshield.org/block.txt" is not working for me since this morning. It's redirected to something.  I am using target="http://www.dshield.org/block.txt" . Hope this helps some people.

----------

## Bones McCracker

Thank you.  I suggest changing it to:

"https://www.dshield.org/block.txt"

(Note https.)

----------

## OverrideZ

hi,

i have made a python version, except that do not check timestamp, just update new entry and delete old one. i am not a expert dev. thanks for block list source  :Very Happy: 

```

#!/bin/env python3.3

"""

---------------------------------------------------------------

 OverrideZ

 Get remote bock list and add to ipset

 ipset block list updater v1.0

---------------------------------------------------------------

 Before use this script, check you created ipset and iptables entry like above

 ipset create banned_ipv4_net hash:net family inet

 ipset create banned_ipv6_net hash:net family inet6

 iptables -I INPUT 1 -i eth0 -m set --match-set banned_ipv4_net src -j DROP

 ip6tables -I INPUT 1 -i eth0 -m set --match-set banned_ipv6_net src -j DROP

---------------------------------------------------------------

"""

IPSET_PATH = "/usr/sbin/ipset"

IPV4_URL = ["http://dshield.org/block.txt", "http://www.team-cymru.org/Services/Bogons/fullbogons-ipv4.txt"]

IPV6_URL = ["http://www.team-cymru.org/Services/Bogons/fullbogons-ipv6.txt"]

#---------------------------------------------------------------

import re, subprocess

from urllib.request import Request, urlopen

from urllib.error import URLError, HTTPError

#---------------------------------------------------------------

class Ipset:

    """

        Manage ipset entry Read/Add/Delete

    """

    #---------------------------------------------------------------

    def __init__(self, inet):

        self.setname = "banned_%s_net" % inet       # ipset chain name

        self.inet = inet                            # inet mode

        self.ripset = re.compile(r"^\d")            # ipset regexp

        self.currentstor = set()                    # ipset stor

    #---------------------------------------------------------------

    def process(self, netlist):

        """

            Process the blocklist data downloaded

        """

        self.read()

        deleted = self.currentstor.difference(netlist)

        added = netlist.difference(self.currentstor)

        same = netlist.intersection(self.currentstor)

        for ip in deleted:

            self.del_ip(ip)

        for ip in added:

            self.add_ip(ip)

        print("%s net | Add : %s | Dup : %s | Del : %s" % (self.inet, len(added), len(same), len(deleted)))

    #---------------------------------------------------------------

    def read(self):

        """

            read and parse current ipset list content

        """

        cmd = [IPSET_PATH, "list", self.setname]

        result = subprocess.check_output(cmd)

        data = result.decode("utf-8")

        for item in data.split("\n"):

            if self.ripset.match(item):

                self.currentstor.add(item.strip())

    #---------------------------------------------------------------

    def add_ip(self, ip):

        """

            add ip to ipset

        """

        cmd = [IPSET_PATH, "add", "-q", "-!", self.setname, str(ip)]

        subprocess.call(cmd)

    #---------------------------------------------------------------

    def del_ip(self, ip):

        """

            del ip to ipset

        """

        cmd = [IPSET_PATH, "del", "-q", "-!", self.setname, str(ip)]

        subprocess.call(cmd)

#---------------------------------------------------------------

class Updater:

    """

        Download and Parse files

    """

    #---------------------------------------------------------------

    def __init__(self, url, mode):

        self.urls = url                                                     # download url

        self.oip = Ipset(mode)                                              # ipset object

        self.rcymru = re.compile(r"^#")                                     # cymru bogon regexp

        self.rdshield = re.compile(r"(^([0-9]{1,3}\.){3}[0-9]{1,3}).*$")    # dshield regexp

        self.currentstor = set()                                            # downloaded ip stor

    #---------------------------------------------------------------

    def download(self, url):

        """

            Download Files and launch parser

        """

        try:

            req = Request(url)

            data = urlopen(req)

            code = data.getcode()

            if code == 200:

                urlsplit = re.split("/", url)

                filename = urlsplit[len(urlsplit)-1]

                if filename == "block.txt":

                    self.parse_dshield_txt(data.read())

                elif filename == "fullbogons-ipv4.txt":

                    self.parse_cymru_txt(data.read())

                elif filename == "fullbogons-ipv6.txt":

                    self.parse_cymru_txt(data.read())

        except HTTPError as error:

            print("HTTP Error: %s %s" % (error.code, url))

        except URLError as error:

            print("URL Error: %s %s" % (error.reason, url))

    #---------------------------------------------------------------

    def parse_dshield_txt(self, data):

        """

            Parse Dshield block list

        """

        dec = data.decode("utf-8")

        for line in dec.split("\n"):

            if self.rdshield.match(line):

                detail = line.strip().split("\t")

                self.currentstor.add("%s/%s" % (detail[0], detail[2]))     

    #---------------------------------------------------------------

    def parse_cymru_txt(self, data):

        """

            Parse cymru full bogon

        """

        dec = data.decode("utf-8")

        for line in dec.split("\n"):

            if not self.rcymru.match(line):

                self.currentstor.add(line.strip())

    #---------------------------------------------------------------        

    def run(self):

        """

            main run func

        """

        for url in self.urls:

            self.download(url)

        if len(self.currentstor) != 0:

            self.oip.process(self.currentstor)

        else:

            print("Download fail")

#---------------------------------------------------------------

if __name__ == "__main__":

    Updater(IPV4_URL, "ipv4").run()

    Updater(IPV6_URL, "ipv6").run()

```

----------

## bdpita

Hi Bones, inspired by your script I made one a little more flexible and easy to upgrade. 

I hope to be of your interest and help to others using Shorewall + ipset with a live blacklist feed.

Best regards, Bernardo.

```

#! /bin/bash

# Bernardo

# Rev. 1 at March 2015

# Tested with ipset 6.12.1 

# Purpose: dynamic update a many block ip/network lists into an ipset

# Notes: call this from crontab. Feed updated every 10/15 minutes

# Tip: to add a new source list copy this after # List Sources:

#

#      x=$(($x + 1))

#      target[${x}${tname}]="Dshield"

#      target[${x}${turl}]="http://feeds.dshield.org/block.txt"

#      target[${x}${tawktype}]="0"

#

# and replace turl whit the url of the new source. Verify that the tawktype corresponds the type of awk to convert the line of the list.

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

# Definitions

declare -r tname=0

declare -r turl=1

declare -r tawktype=2

update=false

ipset_setname="black.block"

ipset_params="hash:net"

dir_data="/var/tmp/${ipset_setname}"

dir_temp="/var/tmp/${ipset_setname}/tmp"

file_data="${dir_data}/${ipset_setname}.txt"

file_temp="${dir_data}/${ipset_setname}.temp.txt"

x=0

# List Sources

x=$(($x + 1))

target[${x}${tname}]="Dshield"

target[${x}${turl}]="http://feeds.dshield.org/block.txt"

target[${x}${tawktype}]="0"

x=$(($x + 1))

target[${x}${tname}]="Spamhaus"

target[${x}${turl}]="http://www.spamhaus.org/drop/drop.lasso"

target[${x}${tawktype}]="1"

x=$(($x + 1))

target[${x}${tname}]="Okean chinese and korean spammers"

target[${x}${turl}]="http://www.okean.com/sinokoreacidr.txt"

target[${x}${tawktype}]="1"

x=$(($x + 1))

target[${x}${tname}]="Wizcrafts Russian botnets, attackers and spammers"

target[${x}${turl}]="http://www.wizcrafts.net/russian-iptables-blocklist.html"

target[${x}${tawktype}]="2"

x=$(($x + 1))

target[${x}${tname}]="RBN Russian IPs"

target[${x}${turl}]="http://doc.emergingthreats.net/pub/Main/RussianBusinessNetwork/RussianBusinessNetworkIPs.txt"

target[${x}${tawktype}]="3"

x=$(($x + 1))

target[${x}${tname}]="OpenBL.org 30 day List"

target[${x}${turl}]="http://www.openbl.org/lists/base_30days.txt"

target[${x}${tawktype}]="3"

x=$(($x + 1))

target[${x}${tname}]="Cymru"

target[${x}${turl}]="http://www.team-cymru.org/Services/Bogons/fullbogons-ipv4.txt"

target[${x}${tawktype}]="3"

# Function timestamp

get_timestamp() {

    date +%d/%m' '%R

}

# if data directory does not exist, create it

mkdir -pm 0750 ${dir_data}

mkdir -pm 0750 ${dir_temp}

ttime=$(get_timestamp)

echo "Start srcipt at ${ttime}"

for i in `seq 1 $x`; do

    tfile=$(basename ${target[${i}${turl}]})

    tfile="${dir_temp}/${tfile}"

    tfile_last="${tfile}.last"

    if [ -f ${tfile} ]; then

   cp -p -f ${tfile} ${tfile_last}

    else

   touch ${tfile_last}

    fi

    echo "# File create automatically by a sh script" > ${tfile}

    echo "# Download -> ${target[${i}${tname}]}" >> ${tfile}

    case ${target[${i}${tawktype}]} in

   "0")

       wget -q -O - ${target[${i}${turl}]} | awk 'NF && !/^[[:space:]]*#/' | sed '/Start/,+1 d' | awk '{ print $1 "/24"; }' >> ${tfile}

        ;;

   "1")

       wget -q -O - ${target[${i}${turl}]} | awk 'NF && !/^[[:space:]]*;/' | awk '{ print $1 }' >> ${tfile}

        ;;

   "2")

       wget -q -O - ${target[${i}${turl}]} | awk -F\> '/^pre>/{print $2}' RS=\< | awk 'NF && !/^[[:space:]]*#/' >> ${tfile}

        ;;

   "3")

       wget -q -O - ${target[${i}${turl}]} | tr -d $'\r' | awk 'NF && !/^[[:space:]]*#/' >> ${tfile}

        ;;

   "4")

       wget -q -O - ${target[${i}${turl}]} | awk 'NF && !/^[[:space:]]*#/' >> ${tfile}

        ;;

   *)

       echo "# ERROR at ${target[${i}${tname}]} the awk type is missing" >> ${tfile}

        ;;

    esac

    echo "" >> ${tfile}

    # Verify if exist diff in files

    if diff -q ${tfile} ${tfile_last} >/dev/null; then

   # No differences delete last

   mv -f ${tfile_last} ${tfile}

    else

   rm -f ${tfile_last}

   update=true

    fi

done

if [ ${update} = true ]; then

    echo "Made the update"

    ttime=$(get_timestamp)

    echo "# File create automatically by a sh script at ${ttime}" > ${file_data}

    for i in `seq 1 $x`; do

   tfile=$(basename ${target[${i}${turl}]})

   tfile="${dir_temp}/${tfile}"

   if [ -f ${tfile} ]; then

       cat ${tfile} >> ${file_data}

   fi

    done

    # Sort and delete duplicates & comments

    sort -b -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n ${file_data} | uniq | awk 'NF && !/^[[:space:]]*#/' > "${file_temp}"

    ttime=$(get_timestamp)

    echo "Start ipset configuration at ${ttime}"

    temp_ipset="${ipset_setname}_temp"

    ipset create -exist ${temp_ipset} ${ipset_params}

    ipset flush ${temp_ipset}

    while read network; do

   ipset add ${temp_ipset} ${network}

   if [ "$?" -ne "0" ]; then

       logger -p cron.err "IPSet: ${temp_ipset} error failed at add ip/network into ipset."

       echo "IPSet: ${temp_ipset} error failed."

       echo "ipset add ${temp_ipset} ${network}"

       exit 1

   fi

    done < ${file_temp}

    # If ipset does not exist, create it

    ipset create -exist ${ipset_setname} ${ipset_params}

    # Swap the temp ipset for the live one

    ipset swap ${temp_ipset} ${ipset_setname}

    ipset destroy ${temp_ipset}

    # Log the file modification time for use in minimizing lag in cron schedule

    logger -p cron.notice "IPSet: ${ipset_setname} updated."

fi

ttime=$(get_timestamp)

echo "Finish script at ${ttime}"

```

----------

## Bones McCracker

Thank you for sharing it.

----------

## mimosinnet

I have been happily using the perl version of your scripts for some time. I have noticed that the bogons page has stopped working. I have updated the perl script and changed the bogons page to the one from wizcrafts.

Here it is the new version of the script.

Thanks again for your successful scripts and great introduction to ipset! 

Cheers!

----------

## Bones McCracker

The Team Cymru page appears to be working again.  From my logs, it looks like my script was failing to download updates to that particular block list from July 24 through July 27, but they have been successful since then.

This looks like a useful block list you have added here, though.  Thank you for sharing this.

----------

## mimosinnet

I have been playing with nftables and perl6 to recreate a similar filtering scheme, and posted it in the gentoo forums. 

Thanks again for you inspiring and clear post on iptables + ipset.   :Very Happy:   :Smile:   :Very Happy: 

Cheers!

----------

## Bones McCracker

You are quite welcome.  I'm glad it was useful.

An implementation of the concepts in perl6 should be pretty awesome!

----------

