# ERROR: ipset failed to stop

## jhon987

Hello guys,

Every time I shut down or reboot my PC openRC gives the following:

```
ipset            | * ipset is in use, can't stop

ipset            | * ERROR: ipset failed to stop
```

It doesn't seem to affect anything however I do find these error messages annoying, anyone knows how can I stop them?

P.S. I have ipset compiled in my kernel (not as a module) maybe it's relevant, either way I'd rather keep it this way.

----------

## khayyam

jhon987 ...

do you have a iptables/firewall rule that references ipset? If so then you probably want the following:

```
rc_ipset_before="firewall"
```

... or

```
rc_before="firewall"
```

I had the same issue (some time back) and resolved it with the above.

HTH & best ... khay

----------

## cboldt

Some people may use the iptables service instead of the firewall service, and I gather there are few other options for firewall.  The principle is the same though.  ipsets have to be built before the rule inserting them to the firewall is called (either iptables-restore, or iptables -I or -A a rule); and an ipset can't be erased until the iptables rule using the set is deleted.

----------

## khayyam

 *cboldt wrote:*   

> Some people may use the iptables service instead of the firewall service, and I gather there are few other options for firewall.

 

cboldt ... the service is provided by a number of packages (with 'provide firewall' in the init.d script), I expected iptables would also provide this, but I see it doesn't (go figure). If that is the case then the OP can substitute 'firewall' for 'iptables'.

best ... khay

----------

## cboldt

ipkungfu has its own init too (/etc/init.d/ipkungfu), similarly deficient.

I have an /etc/init.d/iptables, but darned if I know where it came from. I may have created it as a copy of /etc/init.d/iptables, but if not, I've commandeered it, and mine is now a homebrew with a few extra commands.

Edit to note a typo -- what I have "commandeered" is an /etc/init.d/firewallLast edited by cboldt on Mon Feb 13, 2017 11:06 pm; edited 1 time in total

----------

## khayyam

 *cboldt wrote:*   

> ipkungfu has its own init too (/etc/init.d/ipkungfu), similarly deficient.

 

cboldt ... it isn't as prevelant as I'd initially thought:

```
/var/pkg/gentoo/net-firewall/firehol/files/firehol.initrd

/var/pkg/gentoo/net-firewall/sanewall/files/sanewall.initd

/var/pkg/gentoo/net-firewall/shorewall/files/shorewall-lite.initd-r2

/var/pkg/gentoo/net-firewall/shorewall/files/shorewall.initd-r2

/var/pkg/gentoo/net-firewall/shorewall/files/shorewall6-lite.initd-r1

/var/pkg/gentoo/net-firewall/shorewall/files/shorewall6.initd-r1

/var/pkg/gentoo/net-firewall/ufw/files/ufw-2.initd
```

best ... khay

----------

## jhon987

Thanks for the input guys, it seems that ipset fails to stop no matter its place in the shut down sequence.

I have iptables installed and used as a service (as far as I understand), if I recall correctly, when I installed ipset I remember I read that ipset is useless without iptables.

Anyhow so, I have iptables rule that reference ipset (-A INPUT -m set --match-set NAME src -j DROP).

while ipset is configured to start at 'boot' runlevel, iptables is at 'default' runlevel and both start without errors, therefore I gather that closing them in the opposite order should also work fine.

However, that's not the case (or so it seems, maybe I'm doing something wrong)...

I tried it before iptables, I tried after iptables, I also tried in between saving iptables and the stopping of it but every time I get the error message.

BEFORE

```
ipset            | * ipset is in use, can't stop

ipset            | * ERROR: ipset failed to stop

 [ ok ]

iptables         | * Saving iptables state ...

 [ ok ]

netmount         | * Unmounting network filesystems ...

consolekit       | * Stopping consolekit ...

swap             | * Deactivating swap devices ...

cronie           | * Stopping cronie ...

 [ ok ]

 [ ok ]

 [ ok ]

 [ ok ]

iptables         | * Stopping firewall ...

 [ ok ]
```

AFTER

```
iptables         | * Saving iptables state ...

local            | * Stopping local ...

urandom          | * Saving random seed ...

netmount         | * Unmounting network filesystems ...

cronie           | * Stopping cronie ...

ntpd             | * Stopping ntpd ...

 [ ok ]

swap             | * Deactivating swap devices ...

iptables         | * Stopping firewall ...

 [ ok ]

consolekit       | * Stopping consolekit ...

 [ ok ]

 [ ok ]

 [ ok ]

xdm              | * Stopping sddm ...

 [ ok ]

 [ ok ]

 [ ok ]

 [ ok ]

ipset            | * ipset is in use, can't stop

ipset            | * ERROR: ipset failed to stop
```

BETWEEN

```
iptables         | * Saving iptables state ...

ntpd             | * Stopping ntpd ...

consolekit       | * Stopping consolekit ...

ipset            | * ipset is in use, can't stop

ipset            | * ERROR: ipset failed to stop

cronie           | * Stopping cronie ...

 [ ok ]

swap             | * Deactivating swap devices ...

 [ ok ]

iptables         | * Stopping firewall ...

 [ ok ]
```

Edit: I did some experiments and yeah, stopping ipset after iptables works I think the issue is that openrc doesn't close ipset after iptables even though I've added: rc_ipset_after="iptables" in /etc/rc.conf

I'll see how I can solve that

----------

## cboldt

Or just do a homebrew /etc/init.d script that preserves the order.  Pieces of mine (dearth of error checking, like missing saved iptables files and so forth).  Variables from /etc/conf.d/firewall are brought in to /etc/init.d/firewall , and don't need to be "transferred" or reassigned to the lowercase values I've adopted.

The script DOES refer to the correct sources at start and stop, for example, use of ipsets at STOP refers to the active iptables arrangement.  So, if you add ipsets during live configuration of the firewall, that will be saved correctly when the firewall shuts down.

Pardon the inconsistent indenting, I edited here after cut and paste, to get rid of some "overcomplicated for the suggestion" material in START.

```
# /etc/init.d/firewall - a homebrew

# Transfer assignments from /etc/conf.d/firewall

# Transfer assignments from /etc/conf.d/ipset

firewall_build=${IPTABLES_BUILD:=/usr/local/sbin/fw-build}

firewall_saved=${IPTABLES_SAVE:=/var/lib/iptables/iptables.rules}

ipset_saved=${IPSET_SAVE:=/var/lib/ipset/ipset.rules}

ipset_bin=${IPSET:=/usr/sbin/ipset}

iptables_bin=${IPTABLES:=/sbin/iptables}

start() {

using_ipsets=`grep -e "-m set" ${firewall_saved}`

if [ -n "$using_ipsets" ]; then

    ebegin "Loading saved kernel IP sets"

     $ipset_bin restore -exist < ${ipset_saved}

     eend $?

fi

ebegin "Restoring saved ${SVCNAME}"

${iptables_bin}-restore -c ${firewall_saved}

eend $?

}

stop() {

        using_ipsets=`${iptables_bin}-save | grep -e "-m set"`

        if [ -n "$using_ipsets" ]; then

                ebegin "Saving kernel IP sets"

                $ipset_bin save > ${ipset_saved}

                eend $?

        fi

        ebegin "Saving and Flushing ${SVCNAME}"

        ${iptables_bin}-save -c > ${firewall_saved}

        ${iptables_bin} --flush

        ${iptables_bin} --delete-chain

        eend $?

        if [ -n "$using_ipsets" ]; then

                ebegin "Flushing kernel IP sets"

                $ipset_bin flush

                $ipset_bin destroy

                eend $?

        fi

}
```

Not to say that openrc can't do the job, but I had other reasons for a homebrew firewall init, namely that the firewall changes between being on LAN and being away from home.

Edit to correct the location of "close-code" tag.

2nd edit to remark, some variables were transferred from /etc/conf.d/iptables in the first place.  I edited /etc/conf.d/firewall by hand (meaning I probably created it from scratch), and if you don't have an /etc/conf.d/firewall, you could ...

```
cat /etc/conf.d/iptables /etc/conf.d/ipset >> /etc/conf.d/firewall
```

... then just use those variables directly, e.g., use ${IPTABLES_SAVE} instead of $firewall_saved, and skip the mess of variable reassignments at the top of the script.

----------

## khayyam

 *jhon987 wrote:*   

> Edit: I did some experiments and yeah, stopping ipset after iptables works I think the issue is that openrc doesn't close ipset after iptables even though I've added: rc_ipset_after="iptables" in /etc/rc.conf

 

jhon987 ... clear and unambigious explanations help us help you. Did you try 'rc_ipset_before="iptables"'? That should be all that's needed.

Note that as you have 'after' in rc.conf, and ipset is in the 'boot' runlevel, it will start 'iptables' in 'boot' as 'ipset is after iptables' (or in other words: 'iptables is before ipset')

Also your '-A INPUT -m set --match-set NAME src -j DROP', is this literal? It does match the 'NAME' used for '--create'? What does the following show:

```
# ipset --list | grep '^Name'
```

HTH & best ... khay

----------

## cboldt

Very good point, both services should be in the same runlevel, and I think "default" is the right place for that.

And after sharing my homebrew script, I cleaned it up, simplified slightly, and I see no reason that openrc couldn't handle this correctly.  In my STOP routine, one could be misled into thinking that closing ipset necessarily has action both before after closing iptables.  This isn't so.  I created that misimpression by checking for ipsets in the running iptables, but it is just as valid to check for ipsets in the newly saved $IPTABLES_SAVE.

Which is what the script does now.

```
stop() {

        ebegin "Saving and flushing ${SVCNAME}"

        ${IPTABLES}-save -c > ${IPTABLES_SAVE}

        ${IPTABLES} --flush

        ${IPTABLES} --delete-chain

        eend $?

        if grep -q -e "-m set" ${IPTABLES_SAVE}; then

                ebegin "Saving and flushing kernel IP sets"

                ${IPSET} save > ${IPSET_SAVE}

                ${IPSET} flush

                ${IPSET} destroy

                eend $?

        fi

}
```

----------

## jhon987

Once again guys - thank you for the input!

cboldt < I really appreciate the time and effort you've invested in sharing your homebrew, however, personally, I rather use the solution with the least lines of code as possible.

I'm sure other people who'll encounter this thread in the future will put it to good use though - thanks.

khayyam < You're right I need to be clear as possible (I tend to forget that things that are clear in my mind aren't necessarily clear in others')

Yes, I did tried: 'rc_ipset_before="iptables"' I tried all kinds of combinations such as: 'rc_ipset_after="iptables"' and 'rc_iptables_before="ipset"' and so on... checking the log afterwards always showed that the error persisted.

When I issue 'rc-service ipset stop' in my terminal I get the same errors: * ipset is in use, can't stop | * ipset failed to stop.

However if I issue: 'rc-service iptables stop' and only then 'rc-service ipset stop' (in my terminal) then it stops without errors <- that's what I meant when I said "I did some experiments ..."

Maybe the term "after" doesn't mean that service-A is closed only after service-B has been terminated, maybe it merely means that service-A should initiate termination process after services-B initiated his (not necessarily after service-B already finished terminating). Perhaps it's a bug in opeRC, I don't know...

Anyways, I solved it! thanks to a nice fellow named timeBandit from this thread: https://forums.gentoo.org/viewtopic-t-778598-start-0.html

who said: 

 *Quote:*   

> When you stop a service, any services that need it will be stopped first.

 

So, for reference sake, the solution I used is: ' rc_iptables_need="ipset" '

P.S. khayyam you mentioned that you had this issue too and you solved it through: ' rc_ipset_before="firewall" ' <- it seem to me that ipset must be closed after the firewall (if the firewall is using ipset), so you might want to check your rc.log to make sure the error really is gone

Thank you both, you've directed me in the right direction.

PEACE

----------

## cboldt

rc_ipset_before="firewall"  is a reference to the startup sequence.  The stop sequence would be reversed from that.

Edit to add - the only value my homebrew has in this context is to show the relationship between iptables and ipset, outside of openrc.  Plus, by sharing it, I looked at it more carefully and made improvements.

Whatever one is using for a firewall and ipsets can work together, by using the rc_ipset_before="firewall" setting.

----------

## khayyam

 *jhon987 wrote:*   

> khayyam < You're right I need to be clear as possible (I tend to forget that things that are clear in my mind aren't necessarily clear in others'). Yes, I did tried: 'rc_ipset_before="iptables"' I tried all kinds of combinations such as: 'rc_ipset_after="iptables"' and 'rc_iptables_before="ipset"' and so on... checking the log afterwards always showed that the error persisted.

 

jhon987 ... I think the issue here is you're changing the 'rc_ipset_before="iptables"' then {re-,}starting the specific service, so the previous order is in effect (rather than the order you would get from rc.conf). When testing you should be mindful of the (runlevel) dependencies (see: /run/openrc/deptree), rather than the specific service. Please try the following:

```
rc_ipset_before="iptables"
```

```
# rc-update del ipset boot

# rc-update del iptables default

# rc-update del net.eth0 default

# rc-update del <other_network_related_service> default 

# mkdir /etc/runlevels/online

# rc-update -s add default online # "stack" default runlevel onto online runlevel

# for i in net.eth0 ipset iptables <other_network_related_service> ; do rc-update add $i online ; done

# rc boot

# /etc/init.d/ipset status || /etc/init.d/ipset stop

# rc default

# rc online 

# rc default
```

Note you can have the 'online' runlevel started at boot by providing 'softlevel=online' as a kernel parameter.

 *jhon987 wrote:*   

> When I issue 'rc-service ipset stop' in my terminal I get the same errors: * ipset is in use, can't stop | * ipset failed to stop. However if I issue: 'rc-service iptables stop' and only then 'rc-service ipset stop' (in my terminal) then it stops without errors <- that's what I meant when I said "I did some experiments ..."

 

When you issue "rc-service ipset stop" the order is that of when the service was started (ie, in the 'boot' runlevel) this, I assume, is prior to any change to rc.conf. So, unless you change runlevel, or reboot, the changes in rc.conf won't have the desired effect.

 *jhon987 wrote:*   

> Anyways, I solved it! thanks to a nice fellow named timeBandit from this thread: who said: "When you stop a service, any services that need it will be stopped first." So, for reference sake, the solution I used is: 'rc_iptables_need="ipset"'

 

This is the same as saying 'rc_ipset_before="iptables"'.

 *jhon987 wrote:*   

> P.S. khayyam you mentioned that you had this issue too and you solved it through: ' rc_ipset_before="firewall" ' <- it seem to me that ipset must be closed after the firewall (if the firewall is using ipset), so you might want to check your rc.log to make sure the error really is gone

 

I assure you it works here, and the 'before' is the 'start' order, rc will do the reverse on 'stop'.

 *jhon987 wrote:*   

> Thank you both, you've directed me in the right direction.

 

You're welcome & best ... khay

----------

## jhon987

 *Quote:*   

> cboldt: rc_ipset_before="firewall" is a reference to the startup sequence. The stop sequence would be reversed from that. 

 

Oh, I assumed "before" means the service starts before and stops before <- thanks for clarifying that...

To both cboldt & khayyam < here's what I gather from my tests:

Since I have ipset in the boot runlevel and iptables in the default one thus they already behave as if there was a 'rc_ipset_before="iptables"' 

and to add to that, I even added the 'rc_ipset_before="iptables"' to rc.conf as you can see in my second post on this thread which is comprised of sections from my rc.log I copied.

The fact of the matter is (and you can see that) no matter where the position of ipset was in the stop sequence it always popped up the same error.

Now, I have an assumption for why the before parameter doesn't work in my case: I use 'rc_parallel="Yes"' and furthermore I have 8-core Intel i7 processor.

Either one of these factors, or perhaps due to both at the same time, might have caused ipset to terminate even before iptables finished its termination process and despite the fact I used 'rc_ipset_before="iptables"'.

Since, from what I gather, the "before" statement is less strict than the "need" statement - I assume, the before was ineffective in my case whereas need is effective.

Using the "need" parameter has no backlash for me, if I remove ipset from the runlevels completely openRC still boots fine and no error is thrown whatsoever.

Best regards guys

----------

## cboldt

I think the problem on your system is that the two services are in different runlevels!

The way I see it, the "before" and similar directives are meaningless when comparing two services that are on different runlevels.  For example, if you ordered a service that is in the default runlevel to start before a service in the boot runlevel, your directive would (or should) be ignored.  The services in "boot" are started before the services in "default."

Put both services ("ipset" and whatever you are using for a firewall, e.g. "iptables") in the "default" runlevel.  They are meant to run "as a unit," and booting doesn't depend on ipset.   Your network (other than "net.lo") is likely managed at the "default" runlevel too, and the network and the firewall are part of a bigger "unit."

```
rc-update del ipset boot

rc-update add ipset default
```

----------

## cboldt

I see I ws wrong about inter-runlevel barriers.  It appears there are none.

From /usr/share/doc/openrc-*/guide.md.bz2

```
# The Depend Function

This function declares the dependencies for a service script. This

determines the order the service scripts start.

```

depend() {

        need net

        use dns logger netmount

        want coolservice

}

```

`need` declares a hard dependency - net always needs to be started before this 

        service does

`use` is a soft dependency - if dns, logger or netmount is in this runlevel 

        start it before, but we don't care if it's not in this runlevel.

        `want` is between need and use - try to start coolservice if it is

        installed on the system, regardless of whether it is in the

        runlevel, but we don't care if it starts.

`before` declares that we need to be started before another service

`after` declares that we need to be started after another service, without 

        creating a dependency (so on calling stop the two are independent)

`provide` allows multiple implementations to provide one service type, e.g.:

        `provide cron` is set in all cron-daemons, so any one of them started 

        satisfies a cron dependency
```

I'd say "before" is not literally the reverse of "after."  In the relationship between ipset and iptables, using "iptables after" allows at least the user to create an error by running `/etc/init.d/iptables stop` before stopping ipset.

The guide has more on the subject of service dependency.

----------

