# Apache2 is running but I don't see it in netstat

## LarryTheNoob

Dear friends,

again I find myself at my vit's end. I used to compile apache by hand and never had an issue like this. This time I have a fresh apache2 from gentoo repos and everything runs swimmingly - at least until I ran "netstat". 

```

xxxx /etc/apache2 # netstat -tulnp

Active Internet connections (only servers)

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    

tcp        0      0 0.0.0.0:3306            0.0.0.0:*               LISTEN      4285/mysqld         

tcp        0      0 0.0.0.0:139             0.0.0.0:*               LISTEN      9051/smbd           

tcp        0      0 0.0.0.0:6379            0.0.0.0:*               LISTEN      5183/redis-server 0 

tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      4201/sshd           

tcp        0      0 0.0.0.0:445             0.0.0.0:*               LISTEN      9051/smbd           

tcp        0      0 0.0.0.0:5666            0.0.0.0:*               LISTEN      7176/nrpe           

udp        0      0 192.168.2.220:123       0.0.0.0:*                           4975/ntpd           

udp        0      0 127.0.0.1:123           0.0.0.0:*                           4975/ntpd           

udp        0      0 0.0.0.0:123             0.0.0.0:*                           4975/ntpd           

udp        0      0 172.17.255.255:137      0.0.0.0:*                           9063/nmbd           

udp        0      0 172.17.0.1:137          0.0.0.0:*                           9063/nmbd           

udp        0      0 192.168.2.255:137       0.0.0.0:*                           9063/nmbd           

udp        0      0 192.168.2.220:137       0.0.0.0:*                           9063/nmbd           

udp        0      0 0.0.0.0:137             0.0.0.0:*                           9063/nmbd           

udp        0      0 172.17.255.255:138      0.0.0.0:*                           9063/nmbd           

udp        0      0 172.17.0.1:138          0.0.0.0:*                           9063/nmbd           

udp        0      0 192.168.2.255:138       0.0.0.0:*                           9063/nmbd           

udp        0      0 192.168.2.220:138       0.0.0.0:*                           9063/nmbd           

udp        0      0 0.0.0.0:138             0.0.0.0:*                           9063/nmbd           

xxxx /etc/apache2 # telnet localhost 80

Trying 127.0.0.1...

Connected to localhost.localdomain.

Escape character is '^]'.

GET

HTTP/1.1 400 Bad Request

Date: Wed, 27 Jun 2018 13:41:44 GMT

Server: Apache

Content-Length: 226

Connection: close

Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

<html><head>

<title>400 Bad Request</title>

</head><body>

<h1>Bad Request</h1>

<p>Your browser sent a request that this server could not understand.<br />

</p>

</body></html>

Connection closed by foreign host.

```

also

```

xxxx /var/log/apache2/error # netstat -tulnp6

Active Internet connections (only servers)

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name  

```

(nothing for ipv6)

Soo - where the hell is my apache? I can use it, it seems to work just fine (both on the external ip and on localhost), I just don't see it in "netstat". Should I be concerned?

```

xxxx /var/log/apache2/error # apache2ctl -v

Usage: grep [OPTION]... PATTERN [FILE]...

Try 'grep --help' for more information.

Server version: Apache/2.4.33 (Unix)

Server built:   Jun 19 2018 09:53:00

```

I should note that I have a mile long httpd.conf file and dozens of vhosts on this machine, I haven't attached just yet because it would have to be heavily redacted first  :Sad: . I will if anyone suggests I should....

----------

## signal

Hy. 

Seems like you're using systemd for your services... which is fine (I guess). 

systemd provides something called "sevice-based activation" for sockets (or rather it's services). While this does provide some benefits (especially with multiple services at startup) it's debatable whether or not a service manager should handle this. That's a completely different story. 

From the looks of it you are using systemd and apache2 with mod_systemd. If so (fairly sure) then this is normal. 

Systemd works with units, one of which happens to be sockets. 

You can check the output of the following:

```
systemctl list-units --type=socket --all
```

on your machine for comparison to your netstat output.

If you already have iproute installed on your system you can use the 

```
ss -l
```

 command (which you should prefer from now on over netstat). 

If not you can install iproute:

```
emerge -av iproute2
```

I don't think that it is required to post your httpd.conf since the service is handling requests (as per your telnet).

If none of the above fit we'll embark on an adventure.

Kind regards, 

az0r

----------

## eccerr0r

You found something quite interesting indeed!

I have apache2 running on both ipv4 and ipv6 on a openrc machine.  Netstat also appears to NOT find its ipv4 listening sockets, but DOES find the ipv6 listening sockets.

I was looking around and found 'ss' which found the ipv4, but NOT ipv6 sockets (!) -- just the reverse, which is baffling!

ss is part of iproute2.  ss appears to use the same options as netstat, but its reporting is different so it will break scrapers.

----------

## Hu

OP: what is the output of emerge --info sys-apps/iproute2 sys-apps/net-tools?

 *signal wrote:*   

> Seems like you're using systemd for your services... which is fine (I guess). 

 Where do you see that? *signal wrote:*   

> systemd provides something called "sevice-based activation" for sockets (or rather it's services).

 As I understand it, that is just systemd assuming a role once held by inetd and later xinetd.  In all cases, the socket should be visible as listening.  The only variance would be in which process owned the socket.

----------

## hdcg

The tcp6 sockets are also act as the IPv4 listening sockets, which is a feature of IPv6 sockets (https://unix.stackexchange.com/questions/237731/why-are-ipv4-tcp-connections-showing-as-tcp6).

On my box (OpenRC based) Apache is listening on IPv4 and IPv6 too and the output is similar:

```
netstat -tulnp | grep apache

tcp6       0      0 :::80                   :::*                    LISTEN      1839/apache2

tcp6       0      0 :::443                  :::*                    LISTEN      1839/apache2

```

I would guess that ss only shows these kinds of socket the other way around.

```
ss -ap | grep http | grep LISTEN

tcp   LISTEN     0       128                                                  *:http                                                            *:*              users:(("apache2",pid=20305,fd=6),("apache2",pid=20304,fd=6),("apache2",pid=1839,fd=6))

tcp   LISTEN     0       128                                                  *:https                                                           *:*              users:(("apache2",pid=20305,fd=4),("apache2",pid=20304,fd=4),("apache2",pid=1839,fd=4))

```

Best Regards

Holger

----------

## LarryTheNoob

Dear friends, 

thank you for your response. 

Signal,

I am pretty sure I am running the default OpenRC. (eg: I am controling services through rc-service & rc-update commands, hope I am not wrong)

Hu,

here is the the result of 

```
emerge -info sys-apps/iproute2 sys-apps/net-tools 
```

```

Portage 2.3.40 (python 3.5.5-final-0, default/linux/amd64/17.0, gcc-6.4.0, glibc-2.26-r7, 4.12.12-gentoo x86_64)

=================================================================

                         System Settings

=================================================================

System uname: Linux-4.12.12-gentoo-x86_64-Intel-R-_Core-TM-_i7_CPU_930_@_2.80GHz-with-gentoo-2.6

KiB Mem:    14344824 total,   5990616 free

KiB Swap:    1999868 total,   1999868 free

Timestamp of repository gentoo: Fri, 22 Jun 2018 08:30:01 +0000

Head commit of repository gentoo: f98bb224f10e90d2d580f5d38406478ceeb336d1

sh bash 4.4_p12

ld GNU ld (Gentoo 2.28.1 p1.0) 2.28.1

app-shells/bash:          4.4_p12::gentoo

dev-java/java-config:     2.2.0-r4::gentoo

dev-lang/perl:            5.24.3-r1::gentoo

dev-lang/python:          2.7.15::gentoo, 3.4.6::gentoo, 3.5.5-r1::gentoo

dev-util/cmake:           3.9.6::gentoo

dev-util/pkgconfig:       0.29.2::gentoo

sys-apps/baselayout:      2.6::gentoo

sys-apps/openrc:          0.34.11::gentoo

sys-apps/sandbox:         2.13::gentoo

sys-devel/autoconf:       2.69-r4::gentoo

sys-devel/automake:       1.15.1-r2::gentoo

sys-devel/binutils:       2.28.1::gentoo, 2.29.1-r1::gentoo, 2.30-r2::gentoo

sys-devel/gcc:            5.4.0-r3::gentoo, 6.4.0-r1::gentoo, 7.3.0-r3::gentoo

sys-devel/gcc-config:     1.8-r1::gentoo

sys-devel/libtool:        2.4.6-r3::gentoo

sys-devel/make:           4.2.1::gentoo

sys-kernel/linux-headers: 4.13::gentoo (virtual/os-headers)

sys-libs/glibc:           2.26-r7::gentoo

Repositories:

gentoo

    location: /usr/portage

    sync-type: rsync

    sync-uri: rsync://rsync.cz.gentoo.org/gentoo-portage

    priority: 1000

    sync-rsync-verify-jobs: 1

    sync-rsync-verify-metamanifest: yes

    sync-rsync-verify-max-age: 24

    sync-rsync-extra-opts: 

ACCEPT_KEYWORDS="amd64"

ACCEPT_LICENSE="* -@EULA"

CBUILD="x86_64-pc-linux-gnu"

CFLAGS="-O2 -pipe -march=nocona"

CHOST="x86_64-pc-linux-gnu"

CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt"

CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.6/ext-active/ /etc/php/apache2-php7.1/ext-active/ /etc/php/cgi-php5.6/ext-active/ /etc/php/cgi-php7.1/ext-active/ /etc/php/cli-php5.6/ext-active/ /etc/php/cli-php7.1/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"

CXXFLAGS="-O2 -pipe -march=nocona"

DISTDIR="/usr/portage/distfiles"

EMERGE_DEFAULT_OPTS="--jobs=9"

ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"

FCFLAGS="-O2 -pipe"

FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync multilib-strict news parallel-fetch parallel-install preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"

FFLAGS="-O2 -pipe"

GENTOO_MIRRORS="http://gentoo.mirror.web4u.cz"

LDFLAGS="-Wl,-O1 -Wl,--as-needed"

MAKEOPTS="-j9"

PKGDIR="/usr/portage/packages"

PORTAGE_CONFIGROOT="/"

PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"

PORTAGE_TMPDIR="/var/tmp"

USE="acl acpi amd64 bash-completion berkdb big-tables bzip2 cli crypt cxx dri fontconfig fortran freetype gcj gd gdbm gif gnutls iconv inno innodb jpeg jpeg2k libtirpc mmx modules multilib ncurses nls nptl openmp opentype pam parse-clocks pcre perl png python readline seccomp sse sse2 ssl tcpd tiff truetype unicode upcall xattr zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon plan sheets stage words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext sse sse2" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="libinput keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-6 php7-0" POSTGRES_TARGETS="postgres9_5 postgres10" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_5" RUBY_TARGETS="ruby23" USERLAND="GNU" VIDEO_CARDS="amdgpu fbdev intel nouveau radeon radeonsi vesa dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"

Unset:  CC, CPPFLAGS, CTARGET, CXX, INSTALL_MASK, LANG, LC_ALL, LINGUAS, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS

=================================================================

                        Package Settings

=================================================================

sys-apps/iproute2-4.14.1-r2::gentoo was built with the following:

USE="berkdb iptables -atm -ipv6 -minimal (-selinux)" ABI_X86="(64)"

sys-apps/net-tools-1.60_p20161110235919::gentoo was built with the following:

USE="arp hostname nls -ipv6 -nis -plipconfig (-selinux) -slattach -static" ABI_X86="(64)"

```

Using the ss tool, I can see what hdcg sees:

```

ss -ap | grep http | grep LISTEN

tcp    LISTEN     0      511     *:http                  *:*                     users:(("apache2",pid=32180,fd=4),("apache2",pid=27629,fd=4),("apache2",pid=27628,fd=4),("apache2",pid=27627,fd=4),("apache2",pid=19750,fd=4))

tcp    LISTEN     0      511     *:https                 *:*                     users:(("apache2",pid=32180,fd=6),("apache2",pid=27629,fd=6),("apache2",pid=27628,fd=6),("apache2",pid=27627,fd=6),("apache2",pid=19750,fd=6))

devel2 /home/petr.kubecka # netstat -tulnp

```

[Moderator edit: changed [quote] tags to [code] tags to preserve output layout. -Hu]

----------

## signal

 *Hu wrote:*   

> OP: what is the output of emerge --info sys-apps/iproute2 sys-apps/net-tools?
> 
>  *signal wrote:*   Seems like you're using systemd for your services... which is fine (I guess).  *Hu wrote:*   Where do you see that?  

 

It was only a guess with the output provided and it's what I personally would look into first (based on ss's output). Other services are listed just fine like mysqld. To my knowledge mysql do not have systemd's socket-based (on-demand) activation support but apache does with mod_systemd and it's handling requests just fine. Thus the guess on my part.

 *Hu wrote:*   

>  *signal wrote:*   systemd provides something called "sevice-based activation" for sockets (or rather it's services). *Hu wrote:*   As I understand it, that is just systemd assuming a role once held by inetd and later xinetd.  In all cases, the socket should be visible as listening.  The only variance would be in which process owned the socket.  

 

Yes and yes. It should be listed either way.

Kinds regards, 

az0r

----------

