# Apache hangs often after kernel upgrade [SOLVED]

## selig

I have recently updated my server. Everything seemed to be running fine but Apache hangs from time to time - more often under higher load. It happened five times today, which is unacceptable. To survive until I solve the problem, I created a script which checks whether the website can be accessed and restarts Apache if it cannot.

My emerge --info:

```

# emerge --info

Portage 2.1.10.11 (hardened/linux/amd64/no-multilib, gcc-4.3.4, glibc-2.11.3-r0, 2.6.39-hr8 x86_64)

=================================================================

System uname: Linux-2.6.39-hr8-x86_64-Intel-R-_Xeon-R-_CPU_E5520_@_2.27GHz-with-gentoo-2.0.3

Timestamp of tree: Fri, 02 Sep 2011 09:15:01 +0000

app-shells/bash:          4.1_p9

dev-lang/python:          2.6.5-r2

dev-util/pkgconfig:       0.23

sys-apps/baselayout:      2.0.3

sys-apps/openrc:          0.8.2-r1

sys-apps/sandbox:         1.6-r2

sys-devel/autoconf:       2.13, 2.65-r1

sys-devel/automake:       1.7.9-r1, 1.9.6-r2::<unknown repository>, 1.10.3, 1.11.1

sys-devel/binutils:       2.21.1-r1

sys-devel/gcc:            4.3.4, 4.4.5

sys-devel/gcc-config:     1.4.1

sys-devel/libtool:        2.2.10

sys-devel/make:           3.82-r1

sys-kernel/linux-headers: 2.6.30-r1 (virtual/os-headers)

sys-libs/glibc:           2.11.3

Repositories: gentoo x-Overlay

ACCEPT_KEYWORDS="amd64"

ACCEPT_LICENSE="* -@EULA"

CBUILD="x86_64-pc-linux-gnu"

CFLAGS="-O2 -mtune=core2 -march=core2 -pipe -fforce-addr"

CHOST="x86_64-pc-linux-gnu"

CONFIG_PROTECT="/etc /var/qmail/alias /var/qmail/control /var/vpopmail/domains /var/vpopmail/etc"

CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.2/ext-active/ /etc/php/apache2-php5.3/ext-active/ /etc/php/cgi-php5.2/ext-active/ /etc/php/cgi-php5.3/ext-active/ /etc/php/cli-php5.2/ext-active/ /etc/php/cli-php5.3/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"

CXXFLAGS="-O2 -mtune=core2 -march=core2 -pipe -fforce-addr"

DISTDIR="/usr/portage/distfiles"

FEATURES="assume-digests binpkg-logs distlocks ebuild-locks fixlafiles fixpackages news parallel-fetch protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch usersandbox"

FFLAGS=""

GENTOO_MIRRORS="ftp://ftp.sh.cvut.cz/MIRRORS/gentoo http://gentoo.supp.name"

LANG="cs_CZ.UTF-8"

LDFLAGS="-Wl,-O1 -Wl,--as-needed"

LINGUAS="cs"

MAKEOPTS="-j8"

PKGDIR="/usr/portage/packages"

PORTAGE_CONFIGROOT="/"

PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"

PORTAGE_TMPDIR="/var/tmp"

PORTDIR="/usr/portage"

SYNC="rsync://rsync.europe.gentoo.org/gentoo-portage"

USE="amd64 apache2 berkdb bzip2 caps clamav cli cracklib crypt cups curl cxx dri expat fam fontconfig ftp gd gdbm geoip hardened iconv imap ithreads jpeg jpeg2k justify lm_sensors logrotate memlimit mmx modules mudflap mysql ncurses nls nptl nptlonly openmp pam pax_kernel pcre perl perlsuid php pic png pppd python readline session slang snmp spell sse sse2 ssl sysfs tcpd threads tiff truetype urandom xml xorg zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="auth_digest cgid dbd ident imagemap actions alias auth_basic authn_alias authn_default authn_file authn_dbd authz_default authz_groupfile authz_host authz_owner authz_user deflate dir env expires ext_filter filter headers include log_config mime mime_magic negotiation rewrite setenvif status vhost_alias" APACHE2_MPMS="worker" CALLIGRA_FEATURES="kexi words flow plan stage tables krita karbon braindump" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="cs" PHP_TARGETS="php5-3" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="fbdev glint intel mach64 mga neomagic nouveau nv r128 radeon savage sis tdfx trident vesa via vmware dummy v4l" XTABLES_ADDONS="geoip"

Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS

```

I am running Apache in a chroot so I thought it was a problem with old libraries etc.. I created a new chroot, with new Apache but the problem persists.

emerge -pv apache:

```

# emerge -pv apache

These are the packages that would be merged, in order:

Calculating dependencies... done!

[ebuild   R    ] www-servers/apache-2.2.17  USE="ssl suexec threads -debug -doc -ldap (-selinux) -static" APACHE2_MODULES="actions alias auth_basic auth_digest authn_alias authn_dbd authn_default authn_file authz_default authz_groupfile authz_host authz_owner authz_user cgid dbd deflate dir env expires ext_filter filter headers ident imagemap include log_config mime mime_magic negotiation rewrite setenvif status vhost_alias -asis -authn_anon -authn_dbm -authz_dbm -autoindex -cache -cern_meta -cgi -charset_lite -dav -dav_fs -dav_lock -disk_cache -dumpio -file_cache -info -log_forensic -logio -mem_cache -proxy -proxy_ajp -proxy_balancer -proxy_connect -proxy_ftp -proxy_http -proxy_scgi -reqtimeout -speling -substitute -unique_id -userdir -usertrack -version" APACHE2_MPMS="worker -event -itk -peruser -prefork" 0 kB

Total: 1 package (1 reinstall), Size of downloads: 0 kB

```

It probably is connected to the segmentation faults I get in logs sometime. It starts to hang a few minutes after the sigsegv.

```

[Mon Sep 05 18:17:49 2011] [notice] child pid 2948 exit signal Segmentation fault (11)

```

It just stops working. Apache processes are there but they are not doing anything. I have to shut down Apache in chroot and start it up again.

```

[Mon Sep 05 18:25:39 2011] [warn] child process 8306 still did not exit, sending a SIGTERM

[Mon Sep 05 18:25:39 2011] [warn] child process 8235 still did not exit, sending a SIGTERM

[Mon Sep 05 18:25:41 2011] [warn] child process 8306 still did not exit, sending a SIGTERM

[Mon Sep 05 18:25:41 2011] [warn] child process 8235 still did not exit, sending a SIGTERM

[Mon Sep 05 18:25:43 2011] [warn] child process 8306 still did not exit, sending a SIGTERM

[Mon Sep 05 18:25:43 2011] [warn] child process 8235 still did not exit, sending a SIGTERM

[Mon Sep 05 18:25:45 2011] [error] child process 8306 still did not exit, sending a SIGKILL

[Mon Sep 05 18:25:45 2011] [error] child process 8235 still did not exit, sending a SIGKILL

[Mon Sep 05 18:25:46 2011] [error] could not make child process 8306 exit, attempting to continue anyway

[Mon Sep 05 18:25:46 2011] [notice] caught SIGTERM, shutting down

```

My kernel config is here: http://pastebin.com/zC0mb45M I am using the latest stable hardened sources, namely hardened-sources-2.6.39-r8

What can I do/try? Any suggestions are appreciated.Last edited by selig on Mon Sep 19, 2011 1:07 pm; edited 1 time in total

----------

## selig

I set up Apache to dump core files and inspected them - the segmentation fault happens in PHP. So I tried to compile PHP with debug symbols. Unfortunately, after I started up Apache with this PHP with USE=debug, it started segfaulting immediately in libpthread when threads were created. I had to restore the half-functional version but I really do not know what I should do about this.

----------

## i92guboj

Did you try discarding some lower level problem (memtest86, fsck your drives, ...)?

Besides that, are you sure the kernel has something to do with it? Does it really work without problems with any lesser version?

----------

## selig

Maybe glibc needs to be recompiled but this system is really bad - it does not have any test environment so I cannot copy it and test it out.

Disks are fine and memory too, all other applications are running fine but PHP is having problems. Maybe it would be safer to use the Prefork MPM - but prior to the kernel upgrade it had been working fine with Worker MPM...

----------

## i92guboj

You can always try an emerge -e world, just to discard some kind of ABI breakage. But before that, I'd save the emerge log. Look into it and see if php has been updated or recompiled recently. Other than all this shoting-in-the-dark, all I can suggest if you don't get this fixed soon enough is to use whatever debug tools you are comfortable with. You could also try to disable the debug USE flag, and use FEATURES=splitdebug instead in your make.conf and see if that allows you to get some more info using gdb or whatever.

For the record, I always have that enabled, since it doesn't interfere with the rest of the system (the debug info is saved separately, not into the object files).

----------

## Mad Merlin

AFAIK, mod_php isn't supported on anything except prefork. I would strongly suggest using that MPM instead of worker.

Barring that, are you using eaccelerator or APC? Did you update Apache/PHP without updating eaccelerator/APC? If so, try recompiling eaccelerator/APC.

----------

## selig

Thanks for the suggestions, I will try recompiling PHP with USE=-threads and enabling the Prefork MPM in Apache. I have read many stories about PHP breaking down with threads enabled. It might be my problem too.

----------

## selig

It was definitely a problem with threading in PHP. First, I lowered the number of threads per process in worker MPM. This helped a bit. On Saturday night, I was able to recompile Apache and PHP without threading and with prefork MPM - and the segmentation faults are gone. The problem with kernel upgrade was probably caused by adding a few hardening options and PHP with its buggy threading implementation was not able to run properly under it.

----------

