# hosts.deny/libwrap not working for sshd/apache

## Zwisel

Hello all,

I'm running a Gentoo Server since ages but by random I found out that my sshd log files are very big, so I figured out that my sshd gets brute force attacked!

Well, for that reason I use denyhost, which worked in the past. Actually, denyhost is still working but my system is ignoring /etc/hosts.allow and /etc/hosts.deny

I did the usual checks:

```
net-misc/openssh

     Available versions:  6.6_p1-r1 ~6.6.1_p1-r4 6.7_p1 ~6.7_p1-r1 ~6.7_p1-r2 ~6.7_p1-r3 {X X509 bindist +hpn kerberos ldap ldns libedit pam +pie sctp selinux skey static tcpd KERNEL="linux"}

     Installed versions:  6.7_p1(12:43:09 08.12.2014)(bindist hpn pam pie selinux -X -X509 -kerberos -ldap -ldns -libedit -sctp -skey -static)

     Homepage:            http://www.openssh.org/

     Description:         Port of OpenBSD's free SSH release
```

I don't understand why eix openssh shows me the tcpd USE flag but the compiled openssh has no tcp wrapper:

```
ldd `which sshd`

        linux-vdso.so.1 (0x00007fffcc8e6000)

        libpam.so.0 => /lib64/libpam.so.0 (0x00007f687f768000)

        libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f687f540000)

        libcrypto.so.1.0.0 => /usr/lib64/libcrypto.so.1.0.0 (0x00007f687f15e000)

        libutil.so.1 => /lib64/libutil.so.1 (0x00007f687ef5b000)

        libz.so.1 => /lib64/libz.so.1 (0x00007f687ed43000)

        libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f687eb0b000)

        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f687e8eb000)

        libc.so.6 => /lib64/libc.so.6 (0x00007f687e533000)

        libdl.so.2 => /lib64/libdl.so.2 (0x00007f687e32f000)

        libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f687e0ed000)

        /lib64/ld-linux-x86-64.so.2 (0x00007f687f977000)
```

So, for me it is clear why tcpd for ssh is not working, it is not compiled into sshd. But I set the USE flag and recompiled, it did not change.

tcp-wrappers are installed:

```
 sys-apps/tcp-wrappers

     Available versions:  7.6-r8 ~7.6.22 7.6.22-r1 {ipv6 netgroups static-libs ABI_MIPS="n32 n64 o32" ABI_PPC="32 64" ABI_S390="32 64" ABI_X86="32 64 x32"}

     Installed versions:  7.6.22-r1(12:22:31 08.12.2014)(-ipv6 -netgroups -static-libs ABI_MIPS="-n32 -n64 -o32" ABI_PPC="-32 -64" ABI_S390="-32 -64" ABI_X86="64 -32 -x32")

     Homepage:            ftp://ftp.porcupine.org/pub/security/index.html

     Description:         TCP Wrappers
```

Because I disabled USE flag bindist a few weeks ago, I recompiled my system with bindist enabled, but did not work.

Because of the heartbeat bug I disabled USE flag tls-heartbeat in SSL a few months ago, so I enabled it:

```
emerge --info openssh openssl

Portage 2.2.14 (python 2.7.7-final-0, hardened/linux/amd64/selinux, gcc-4.8.3, glibc-2.19-r1, 3.15.10-hardened-r1_default_00_ x86_64)

=================================================================

                         System Settings

=================================================================

System uname: Linux-3.15.10-hardened-r1_default_00_-x86_64-AMD_E-450_APU_with_Radeon-tm-_HD_Graphics-with-gentoo-2.2

KiB Mem:     8131064 total,   3534972 free

KiB Swap:    4194300 total,   4194300 free

Timestamp of tree: Fri, 05 Dec 2014 15:15:01 +0000

ld GNU ld (Gentoo 2.24 p1.4) 2.24

app-shells/bash:          4.2_p53

dev-java/java-config:     2.2.0

dev-lang/perl:            5.18.2-r2

dev-lang/python:          2.7.7, 3.3.5-r1, 3.4.1

dev-util/cmake:           2.8.12.2-r1

dev-util/pkgconfig:       0.28-r1

sys-apps/baselayout:      2.2

sys-apps/openrc:          0.12.4

sys-apps/sandbox:         2.6-r1

sys-devel/autoconf:       2.69

sys-devel/automake:       1.13.4

sys-devel/binutils:       2.24-r3

sys-devel/gcc:            4.8.3

sys-devel/gcc-config:     1.7.3

sys-devel/libtool:        2.4.2-r1

sys-devel/make:           4.0-r1

sys-kernel/linux-headers: 3.16 (virtual/os-headers)

sys-libs/glibc:           2.19-r1

Repositories: gentoo

ACCEPT_KEYWORDS="amd64"

ACCEPT_LICENSE="* -@EULA"

CBUILD="x86_64-pc-linux-gnu"

CFLAGS="-march=native -O2 -pipe -fomit-frame-pointer -mcx16 -mpopcnt -msse3 -msse4a -mmmx"

CHOST="x86_64-pc-linux-gnu"

CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt /var/bind"

CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.4/ext-active/ /etc/php/cgi-php5.4/ext-active/ /etc/php/cli-php5.4/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"

CXXFLAGS="-march=native -O2 -pipe -fomit-frame-pointer -mcx16 -mpopcnt -msse3 -msse4a -mmmx"

DISTDIR="/usr/portage/distfiles"

FCFLAGS="-O2 -pipe"

FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox selinux sesandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"

FFLAGS="-O2 -pipe"

GENTOO_MIRRORS="rsync://de-mirror.org/gentoo/ ftp://ftp.wh2.tu-dresden.de/pub/mirrors/gentoo rsync://mirror.netcologne.de/gentoo/ rsync://mirror.opteamax.de/gentoo/ rsync://ftp-stud.hs-esslingen.de/gentoo/ ftp://gentoo.tiscali.nl/pub/mirror/gentoo/ ftp://mirror.switch.ch/mirror/gentoo/ rsync://mirror.bytemark.co.uk/gentoo/"

LANG="de_CH.utf8"

LC_ALL="de_CH.utf8"

LDFLAGS="-Wl,-O1 -Wl,--as-needed"

MAKEOPTS="-j3"

PKGDIR="/usr/portage/packages"

PORTAGE_CONFIGROOT="/"

PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"

PORTAGE_TMPDIR="/var/tmp"

PORTDIR="/usr/portage"

PORTDIR_OVERLAY=""

SYNC="rsync://rsync.de.gentoo.org/gentoo-portage"

USE="amd64 apache2 berkdb bindist bzip2 cli cracklib crypt cxx dbus declarative dri gnutls gpm gstreamer gudev hardened iconv icu imagemagick intl justify maildir mmx modules multilib mysql ncurses nls nptl nsplugin open_perms opengl openmp openssl pam pax_kernel pcre peer_perms perl python qt3support qt4 readline selinux session spell sse sse2 sse3 sse4a ssl ssse3 svg tcpd threads ubac udev unicode urandom vnc webdav-neon webkit xattr xmlreader xtpax zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="actions alias auth_basic auth_digest authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgid dav dav_fs deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias proxy proxy_http" APACHE2_MPMS="event" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CURL_SSL="gnutls" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="de" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-4" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_3 python3_4" RUBY_TARGETS="ruby19 ruby20" USERLAND="GNU" VIDEO_CARDS="dummy fbdev ati" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"

USE_PYTHON="2.7 3.3 3.4"

Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS

=================================================================

                        Package Settings

=================================================================

net-misc/openssh-6.7_p1 was built with the following:

USE="bindist hpn pam pie (selinux) -X -X509 -kerberos -ldap -ldns -libedit -sctp -skey -static" ABI_X86="64"

dev-libs/openssl-1.0.1j was built with the following:

USE="bindist (selinux) (sse2) tls-heartbeat zlib -gmp -kerberos -rfc3779 -static-libs -test -vanilla" ABI_X86="64 -32 -x32"

CFLAGS="-march=native -O2 -pipe -fomit-frame-pointer -mcx16 -mpopcnt -msse3 -msse4a -mmmx -fno-strict-aliasing -Wa,--noexecstack"

CXXFLAGS="-march=native -O2 -pipe -fomit-frame-pointer -mcx16 -mpopcnt -msse3 -msse4a -mmmx -fno-strict-aliasing -Wa,--noexecstack"
```

To test /etc/hosts.deny I made /etc/hosts.deny look like this:

```
sshd: 127.0.0.1
```

and tried to ssh localhost, which worked.

LibWrap does also exist:

```
ls -l /lib64/libwrap.so.0*

lrwxrwxrwx. 1 root root    16  8. Dez 12:22 /lib64/libwrap.so.0 -> libwrap.so.0.7.6

-rwxr-xr-x. 1 root root 39544  8. Dez 12:22 /lib64/libwrap.so.0.7.6

```

I don't know what to do now. If anyone can help, I would be very happy!  :Smile: 

Thanks and cheers

----------

## krinn

 *Zwisel wrote:*   

> To test /etc/hosts.deny I made /etc/hosts.deny look like this:
> 
> ```
> sshd: 127.0.0.1
> ```
> ...

 

 *man hosts.deny wrote:*   

> ACCESS CONTROL FILES
> 
>        The access control software consults two files. The search stops at the
> 
>        first match:
> ...

 

So if hosts.allow grant 127.0.0.1 access, hosts.deny will not even be read, giving the result you get.

----------

## Zwisel

 *krinn wrote:*   

> 
> 
> So if hosts.allow grant 127.0.0.1 access, hosts.deny will not even be read, giving the result you get.

 

Thanks for the feedback. For the test with localhost I renamed hosts.allow - and in my host.allow is no localhost or something but that, but only one remote host.

----------

## Hu

OpenSSH 6.7 dropped support for tcpwrappers.  This is why Gentoo removed USE=tcpd from the ebuild.

----------

## Zwisel

 *Hu wrote:*   

> OpenSSH 6.7 dropped support for tcpwrappers.  This is why Gentoo removed USE=tcpd from the ebuild.

 

Wow, thanks you so much for this information! I wasted 4h for that.

Now I have to find another solution for holding back potential brute force attacker, and most likely the How Tos and Wiki for denyhost (and other tools) have to be changed.

Does anybody now another solution?  :Smile: 

----------

## Zwisel

A downgrade helped. Obviously this is only a short-time solution. Any gentoo-recommendation?

----------

## araxon

 *Zwisel wrote:*   

> A downgrade helped. Obviously this is only a short-time solution. Any gentoo-recommendation?

 

I have been tinkering with fail2ban all morning, but the results are unsatisfactory at best. The default sshd filter regex-es are not matching the default sshd log messages and the load has increased by +2.0 since fail2ban service started.

I'm considering a downgrade as well.

EDIT: I stand corrected - it does match the log messages, but it takes 45 minutes to process a 300 megabytes of logs.   :Embarassed:   Yes, that is the amount of logs generated since the upgrade to OpenSSH-6.7 and the demise of denyhosts few days ago.

----------

## Zwisel

 *araxon wrote:*   

> EDIT: I stand corrected - it does match the log messages, but it takes 45 minutes to process a 300 megabytes of logs.    Yes, that is the amount of logs generated since the upgrade to OpenSSH-6.7 and the demise of denyhosts few days ago.

 

Don't you have logrotate? I have "only" 5MB/day. But it's a private, unknown server only.

Keep in mind that the log file will be far far smaller after 1 day with fail2ban.

And I don't know how iptables gets configured, by file or by call. Since denyhost is simple writing to hosts.deny, it is fast. fail2ban sets iptables-rules, maybe this takes more time. 

If a lot uf users are updateing ssh they will have the same issue. They can downgrade or switch to fail2ban. But reading this: http://unix.stackexchange.com/questions/65801/hosts-allow-not-required-when-using-iptables it might be better to switch from hosts.* to iptables anyway.

I really would appreciate tutorials, how tos, and expert knowledge on this subject, as I am not a network expert nor a security expert but a simple software developer with a home server!  :Smile: 

----------

## araxon

 *Zwisel wrote:*   

>  *araxon wrote:*   EDIT: I stand corrected - it does match the log messages, but it takes 45 minutes to process a 300 megabytes of logs.    Yes, that is the amount of logs generated since the upgrade to OpenSSH-6.7 and the demise of denyhosts few days ago. 
> 
> Don't you have logrotate? I have "only" 5MB/day. But it's a private, unknown server only.
> 
> 

 

I do logrotate weekly. Must have been an endless stream of hacking attempts lately.

----------

## khayyam

Zwisel, araxon ...

you could run openssh from sys-apps/xinetd as this supports tcpwrappers (USE="tcpd"), though iptables/ipset is probably a more elegant solution. There are various howto's here (and elsewhere) where openssh is setup in such a way that no connection is accepted without a specific packet (forget what method/tools are used) so you might look into this rather than fail2ban.

HTH & best ... khay

----------

## Hu

For small private servers, if you have the luxury of knowing the origin of your users, I would use an iptables-based whitelist, not the blacklist that fail2ban generates.  Add rules to allow incoming connections from IPs used by your users, then drop any ssh requests which do not match the known users.

----------

## Zwisel

 *Hu wrote:*   

> For small private servers, if you have the luxury of knowing the origin of your users, I would use an iptables-based whitelist, not the blacklist that fail2ban generates.  Add rules to allow incoming connections from IPs used by your users, then drop any ssh requests which do not match the known users.

 

I can't do that, because the client IPs change. I mean, I'm off the road, wotrking from different places.

----------

## araxon

 *khayyam wrote:*   

> Zwisel, araxon ...
> 
> you could run openssh from sys-apps/xinetd as this supports tcpwrappers (USE="tcpd"), though iptables/ipset is probably a more elegant solution. There are various howto's here (and elsewhere) where openssh is setup in such a way that no connection is accepted without a specific packet (forget what method/tools are used) so you might look into this rather than fail2ban.
> 
> HTH & best ... khay

 

I have got the fail2ban working on all servers and abandoned the hosts.deny style of blocking. But thank you for the advice - it may come handy for others.

----------

## araxon

 *Zwisel wrote:*   

>  *Hu wrote:*   For small private servers, if you have the luxury of knowing the origin of your users, I would use an iptables-based whitelist, not the blacklist that fail2ban generates.  Add rules to allow incoming connections from IPs used by your users, then drop any ssh requests which do not match the known users. 
> 
> I can't do that, because the client IPs change. I mean, I'm off the road, wotrking from different places.

 

It can be solved by using VPN, but I myself prefer to be able to connect from anywhere without unnecessary layers of complexity. If you did not get the fail2ban working, the easiest solution is to:

```
emerge fail2ban

nano /etc/fail2ban/jail.d/sshd.conf
```

copy-paste the file content:

```
[ssh-iptables]

enabled  = true

filter = sshd

action = iptables[name=SSH, port=ssh, protocol=tcp]

logpath = /var/log/messages

maxretry = 5
```

Then start the daemon:

```
/etc/init.d/fail2ban start

rc-update add fail2ban default
```

----------

## Zwisel

thanks, because I have a hardwarefirewall, no iptables is installed. Do I have to configure iptables in a special way?

----------

## araxon

 *Zwisel wrote:*   

> thanks, because I have a hardwarefirewall, no iptables is installed. Do I have to configure iptables in a special way?

 

```
emerge net-firewall/iptables
```

You can then run

```
iptables -L
```

to show the chains and rules list. If the fail2ban works, it creates new chains called fail2ban-* or f2b-*.

If not, the logfile /var/log/fail2ban.log will come in handy for diagnosing what is wrong.

----------

## Zwisel

Installed everything, the run:

```
/etc/init.d/iptables save
```

and

```
/etc/init.d/iptables start
```

then

```
iptables -L

modprobe: FATAL: Module ip_tables not found.

iptables v1.4.21: can't initialize iptables table `filter': Table does not exist (do you need to insmod?)

Perhaps iptables or your kernel needs to be upgraded.
```

Do I have to reconfigure the kernel? I think so... but couldn't find something in my /usr/src/linux/.config - do you know which option it is?

----------

## Zwisel

OK found out:

http://wiki.gentoo.org/wiki/Iptables

http://wiki.gentoo.org/wiki/Home_Router

 :Smile: 

----------

## mancha

Hello.

The day OpenSSH 6.7 released I posted a patch that restores TCP Wrapper support.

I just added the information to Gentoo bug #531156 as well.

You're welcome to it.

--mancha

----------

## MarkCu

Fsck!!!

I've just stumbled on this thread after an emerge world.

Crap.  Crap.  Crap.  

I don't check my system logs often.  Maybe every 6 months or so....

What's all this failed sshd attempts.  From IP addresses in (whois search...) china.

What's going on here?  My hosts.allow, and hosts.deny aren't working anymore.  

More google searches...

2-3 frustrating hours laters.  Many failed configurations updates on hosts.allow, and hosts.deny.  Why isn't it working anymore?  I'm sure this was working right? (Beginning to doubt myself)...And I finally find this thread.

This REALLY SUCKS.  My beloved gentoo REALLY dropped the ball on this one.

I'm one of those users that knows just enough to (usually) prevent myself from getting in trouble.

I know, we're slave to upstream - it was openssh's decision to drop tcp wrappers.

I see the bug reports https://bugs.gentoo.org/show_bug.cgi?id=531156

Closed as WONTFIX.

Darn it this is a HUGE security whole, with not as much as a message at the end of emerge.

My  whole system security was dependent on hosts.allow, and hosts.deny.

I have host.deny of ALL : ALL, and only open 5 whitelisted IP address for SSHD only in hosts.allow.

Pretty basic, not very flexible, but it's served me very well for 10-15 years.  Now it's gone without

any message at all??  

I quickly shutdown my server until I could deal with this.

First attempt at fixing... Downgrade to openssh pre 6.7.  Think I can do this with a package.mask...

Ok.  Nope.  Emerge doesn't keep around the old ebuilds.  Strike one.

(google) - ok, I should be able to pull the old ebuild from the repository, and recreate my own ebuild.

I've never needed on overlay, but it looks easy enough...

Nope, can't create the manifest.  ebuild's having trouble finding one of the old patches...

(openssh-6.6p1-hpnssh14v4.diff.gz if anyone cares...)

Crap Strike 2. 

Ok, my next attempt - looks like I'll have to figure out how to apply Mancha's patch.  I've no trouble running make, patch, configure etc.

But I've no idea how to make things play nice with portage...

Darn it, I don't have time for this crap...

Shutting down the server again for today until I have time to figure this all out.

Really disappointed in this update.  Can a block be put in place for a portage update based on a non-empty /etc/hosts.allow, and/or /etc/hosts.deny.

Cause one REALLY should.  At least a message at the end of the update...

Very frustratingly yours....

Mark

----------

## Hu

For your specific use case, you can use sshd Match Address blocks to enable the permitted hosts, and have a global configuration that blocks all login for unmatched hosts.  You could also use the suggestion I posted previously, which has the bonus of preventing unauthorized users from even completing a TCP handshake with the sshd.

For change requests to the ebuild, please file a bug.  The developers do not regularly read the forums.

----------

## MarkCu

Thanks for tip, Hu.  

There's not a dearth of alternatives.  It will just take me time to evaluate, implement, and test the security of all of them.

I'm not a full time admin. Heck I'm not a part time admin.  It's something I slog through every so often (like every 3-4 months at least).  Most of the stuff I'll forget between iterations.  Both what I did, and how I did it.  That's ok, I'm quick with man pages and google.

But it does take time, and this need was VERY unexpected.  

I've managed to quickly 1. move to a non-standard sshd port.  2.  Create my own ebuild, with Mancha's path to openssh.  It's working on my virtual machine.  I'm going to turn it back on and emerge it on my real hardware...

Then I'll move on to evaluating all my other options on how to lock down my system again.

I've added a comment to https://bugs.gentoo.org/show_bug.cgi?id=531156  Hopefully the gentoo maintainers will reconsider opening this bug back up, and fixing.

----------

## Ant P.

From memory, this is how to achieve exactly the same security in iptables:

```
iptables -N ssh_whitelist

iptables -A INPUT -p tcp --syn --dport ssh -j ssh_whitelist

iptables -A ssh_whitelist -j DROP

iptables -I ssh_whitelist -s $your_ip -j ACCEPT

iptables -I ssh_whitelist -s $your_ip_2 -j ACCEPT

iptables -I ssh_whitelist -s $your_ip_n -j ACCEPT
```

Though if you're relying on an IP whitelist as your only line of defense, you should seriously consider configuring sshd to not be vulnerable to bruteforce attempts...

----------

## Naib

damn! I didn't know tcp wrapping was dropped... I use fail2ban as I havn't gotten my head around iptables....

guess I need to look into it then,

----------

## ct85711

This is not something I like to see, where because of upstream it makes my entire network open because they don't want to use hosts.*.

I've had in all of my computers for a while, a strict Deny all except local area only (making it so no one can attempt to login unless on my network).  Seems, I am going have to consider dropping openssh completely, because the sshd Match Address applied after they login (I don't want them to even get that chance).  I haven't needed to use a firewall, most specifically because every time I tried, it was broken through with a few days (it's nearly impossible to always get a firewall that won't be broken through).

Edit: Sadly, can't remove openssh from my system, because of dependencies, but I did disable it (including disabling ssh login in pam).

----------

## Hu

If your firewall is easily penetrated, you are configuring it wrong.  A drop-by-default firewall with whitelisting for known good hosts works very well and will not mysteriously break with time.

----------

## ct85711

I'm aware it was configured wrong, but sadly since I can't seem to make a firewall that is correctly configured; I have to figure out a different way to protect my network.  Maybe then I'd consider allowing remote login.

----------

