# grsec apache2 abort [SOLVED]

## huuan

After doing a massive upgrade on a hardened x86 system kernel 2.6.32-hardened-r9 I'm now getting repeated kern.log entries like this:

```
Jan 17 16:18:26 as kernel: grsec: From 127.0.0.6: Abort occurred at 00005c62 in /usr/sbin/apache2[apache2:23650] uid/euid:81/81 gid/egid:81/81, parent /usr/sbin/apache2[apache2:4604] uid/euid:0/0 gid/egid:0/0
```

I've read the grsec guide and gogled for this but haven't found anything related.

Using default grsec policy and it still happens even after gradm -D

Also confusing to me is the 127.0.0.6 reference as my iptables has a rule to drop src from 127.0.0.1/8 I see similar references in other grsec related posts but none explaining that src IP.

Hope someone here can shed some light on this or point me to the grsec forum if that 's where I should be asking.Last edited by huuan on Sat Jan 28, 2012 7:52 pm; edited 1 time in total

----------

## Hu

This may not be a grsecurity problem.  As part of its general improvements, grsecurity adds logging for certain fatal signals that are not logged in vanilla kernels.  Therefore, switching to a non-hardened kernel might hide the log message without actually fixing the problem.  Try to obtain a core file from the crashed Apache.  Examine the backtrace in the core file.  That will hopefully put you on the path to the root cause.

----------

## huuan

 *Hu wrote:*   

> This may not be a grsecurity problem.  As part of its general improvements, grsecurity adds logging for certain fatal signals that are not logged in vanilla kernels.  Therefore, switching to a non-hardened kernel might hide the log message without actually fixing the problem.  Try to obtain a core file from the crashed Apache.  Examine the backtrace in the core file.  That will hopefully put you on the path to the root cause.

 

Thanks for the very fast response Hu.

I have a backup server that I can mess with to do that. I'm following http://www.gentoo.org/proj/en/qa/backtraces.xml  unless you can point me to a better way. Thanks.

----------

## Hu

That is a good reference.  Beware that some hardening measures may make it more difficult to obtain a core file than on a non-hardened system, but it should still be possible.  Please post back if you encounter any difficulties.

----------

## huuan

Kernel already had core dump enabled. I enabled apache core dump using instructions here:

http://www.cyberciti.biz/tips/configure-apache-web-server-for-core-dump.html

to go to /tmp/apache-core-dump restarted apache and next time the error occurred a 53 meg core dump was written

-rw------- 1 apache apache 53395456 Jan 17 21:30 core

and used this command to look at the dump as below:

 *Quote:*   

>  # gdb  /usr/sbin/apache2 core
> 
> GNU gdb (Gentoo 7.3.1 p2) 7.3.1
> 
> Copyright (C) 2011 Free Software Foundation, Inc.
> ...

 

I  see a stack trace but it means little to me...thanks for any pointers.

----------

## Hu

It means little to me, too.  Are you sure you rebuilt all the relevant packages using the instructions in the guide you cited?  It appears that at least Apache was missed, since you have no symbols for it.

----------

## huuan

You're right. My bad. I will rebuild apache with debug symbols and try over. Might take a day or two as I'm mired in configuring shibboleth-sp to work with the identity provider at my local edu. I will rebuild apache in the next few days and post back. Thanks for your help.

----------

## huuan

Found some time, here's a core dump with symbols and a backtrace:

```

# gdb /usr/sbin/apache2 core 

GNU gdb (Gentoo 7.2 p1) 7.2

Copyright (C) 2010 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.  Type "show copying"

and "show warranty" for details.

This GDB was configured as "i686-pc-linux-gnu".

For bug reporting instructions, please see:

<http://bugs.gentoo.org/>...

Reading symbols from /usr/sbin/apache2...Reading symbols from /usr/lib/debug/usr/sbin/apache2.debug...done.

done.

[New Thread 1484]

warning: Can't read pathname for load map: Input/output error.

Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.

Loaded symbols for /lib/libm.so.6

Reading symbols from /lib/libpcre.so.0...(no debugging symbols found)...done.

Loaded symbols for /lib/libpcre.so.0

Reading symbols from /usr/lib/libaprutil-1.so.0...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/libaprutil-1.so.0

Reading symbols from /usr/lib/libdb-4.8.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/libdb-4.8.so

Reading symbols from /usr/lib/libgdbm.so.3...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/libgdbm.so.3

Reading symbols from /usr/lib/libexpat.so.1...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/libexpat.so.1

Reading symbols from /usr/lib/libapr-1.so.0...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/libapr-1.so.0

Reading symbols from /lib/libuuid.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib/libuuid.so.1

Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib/librt.so.1

Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib/libcrypt.so.1

Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.

Loaded symbols for /lib/libpthread.so.0

Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib/libdl.so.2

Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.

Loaded symbols for /lib/libc.so.6

Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib/ld-linux.so.2

Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib/libresolv.so.2

Reading symbols from /usr/lib/shibboleth/mod_shib_22.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/shibboleth/mod_shib_22.so

Reading symbols from /usr/lib/libshibsp-lite.so.5...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/libshibsp-lite.so.5

Reading symbols from /usr/lib/libxmltooling-lite.so.5...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/libxmltooling-lite.so.5

Reading symbols from /usr/lib/libxerces-c-3.1.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/libxerces-c-3.1.so

Reading symbols from /usr/lib/liblog4shib.so.1...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/liblog4shib.so.1

Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib/libnsl.so.1

Reading symbols from /usr/lib/gcc/i686-pc-linux-gnu/4.5.3/libstdc++.so.6...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/gcc/i686-pc-linux-gnu/4.5.3/libstdc++.so.6

Reading symbols from /usr/lib/gcc/i686-pc-linux-gnu/4.5.3/libgcc_s.so.1...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/gcc/i686-pc-linux-gnu/4.5.3/libgcc_s.so.1

Reading symbols from /lib/libnss_compat.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib/libnss_compat.so.2

Reading symbols from /lib/libnss_nis.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib/libnss_nis.so.2

Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib/libnss_files.so.2

Reading symbols from /lib/libnss_dns.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib/libnss_dns.so.2

Reading symbols from /usr/lib/apache2/modules/mod_actions.so...Reading symbols from /usr/lib/debug/usr/lib/apache2/modules/mod_actions.so.debug...done.

done.

Loaded symbols for /usr/lib/apache2/modules/mod_actions.so

Reading symbols from /usr/lib/apache2/modules/mod_alias.so...Reading symbols from /usr/lib/debug/usr/lib/apache2/modules/mod_alias.so.debug...done.

done.

Loaded symbols for /usr/lib/apache2/modules/mod_alias.so

Reading symbols from /usr/lib/apache2/modules/mod_asis.so...Reading symbols from /usr/lib/debug/usr/lib/apache2/modules/mod_asis.so.debug...done.

done.

Loaded symbols for /usr/lib/apache2/modules/mod_asis.so

Reading symbols from /usr/lib/apache2/modules/mod_auth_basic.so...Reading symbols from /usr/lib/debug/usr/lib/apache2/modules/mod_auth_basic.so.debug...done.

done.

Loaded symbols for /usr/lib/apache2/modules/mod_auth_basic.so

Reading symbols from /usr/lib/apache2/modules/mod_authn_file.so...Reading symbols from /usr/lib/debug/usr/lib/apache2/modules/mod_authn_file.so.debug...done.

done.

Loaded symbols for /usr/lib/apache2/modules/mod_authn_file.so

Reading symbols from /usr/lib/apache2/modules/mod_authz_default.so...Reading symbols from /usr/lib/debug/usr/lib/apache2/modules/mod_authz_default.so.debug...done.

done.

Loaded symbols for /usr/lib/apache2/modules/mod_authz_default.so

Reading symbols from /usr/lib/apache2/modules/mod_authz_host.so...Reading symbols from /usr/lib/debug/usr/lib/apache2/modules/mod_authz_host.so.debug...done.

done.

Loaded symbols for /usr/lib/apache2/modules/mod_authz_host.so

Reading symbols from /usr/lib/apache2/modules/mod_authz_user.so...Reading symbols from /usr/lib/debug/usr/lib/apache2/modules/mod_authz_user.so.debug...done.

done.

Loaded symbols for /usr/lib/apache2/modules/mod_authz_user.so

Reading symbols from /usr/lib/apache2/modules/mod_cgi.so...Reading symbols from /usr/lib/debug/usr/lib/apache2/modules/mod_cgi.so.debug...done.

done.

Loaded symbols for /usr/lib/apache2/modules/mod_cgi.so

Reading symbols from /usr/lib/apache2/modules/mod_cgid.so...Reading symbols from /usr/lib/debug/usr/lib/apache2/modules/mod_cgid.so.debug...done.

done.

Loaded symbols for /usr/lib/apache2/modules/mod_cgid.so

Reading symbols from /usr/lib/apache2/modules/mod_charset_lite.so...Reading symbols from /usr/lib/debug/usr/lib/apache2/modules/mod_charset_lite.so.debug...done.

done.

Loaded symbols for /usr/lib/apache2/modules/mod_charset_lite.so

Reading symbols from /usr/lib/apache2/modules/mod_deflate.so...Reading symbols from /usr/lib/debug/usr/lib/apache2/modules/mod_deflate.so.debug...done.

done.

Loaded symbols for /usr/lib/apache2/modules/mod_deflate.so

Reading symbols from /lib/libz.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib/libz.so.1

Reading symbols from /usr/lib/apache2/modules/mod_dir.so...Reading symbols from /usr/lib/debug/usr/lib/apache2/modules/mod_dir.so.debug...done.

done.

Loaded symbols for /usr/lib/apache2/modules/mod_dir.so

Reading symbols from /usr/lib/apache2/modules/mod_env.so...Reading symbols from /usr/lib/debug/usr/lib/apache2/modules/mod_env.so.debug...done.

done.

Loaded symbols for /usr/lib/apache2/modules/mod_env.so

Reading symbols from /usr/lib/apache2/modules/mod_expires.so...Reading symbols from /usr/lib/debug/usr/lib/apache2/modules/mod_expires.so.debug...done.

done.

Loaded symbols for /usr/lib/apache2/modules/mod_expires.so

Reading symbols from /usr/lib/apache2/modules/mod_ext_filter.so...Reading symbols from /usr/lib/debug/usr/lib/apache2/modules/mod_ext_filter.so.debug...done.

done.

Loaded symbols for /usr/lib/apache2/modules/mod_ext_filter.so

Reading symbols from /usr/lib/apache2/modules/mod_filter.so...Reading symbols from /usr/lib/debug/usr/lib/apache2/modules/mod_filter.so.debug...done.

done.

Loaded symbols for /usr/lib/apache2/modules/mod_filter.so

Reading symbols from /usr/lib/apache2/modules/mod_headers.so...Reading symbols from /usr/lib/debug/usr/lib/apache2/modules/mod_headers.so.debug...done.

done.

Loaded symbols for /usr/lib/apache2/modules/mod_headers.so

Reading symbols from /usr/lib/apache2/modules/mod_include.so...Reading symbols from /usr/lib/debug/usr/lib/apache2/modules/mod_include.so.debug...done.

done.

Loaded symbols for /usr/lib/apache2/modules/mod_include.so

Reading symbols from /usr/lib/apache2/modules/mod_info.so...Reading symbols from /usr/lib/debug/usr/lib/apache2/modules/mod_info.so.debug...done.

done.

Loaded symbols for /usr/lib/apache2/modules/mod_info.so

Reading symbols from /usr/lib/apache2/modules/mod_log_config.so...Reading symbols from /usr/lib/debug/usr/lib/apache2/modules/mod_log_config.so.debug...done.

done.

Loaded symbols for /usr/lib/apache2/modules/mod_log_config.so

Reading symbols from /usr/lib/apache2/modules/mod_logio.so...Reading symbols from /usr/lib/debug/usr/lib/apache2/modules/mod_logio.so.debug...done.

done.

Loaded symbols for /usr/lib/apache2/modules/mod_logio.so

Reading symbols from /usr/lib/apache2/modules/mod_mime.so...Reading symbols from /usr/lib/debug/usr/lib/apache2/modules/mod_mime.so.debug...done.

done.

Loaded symbols for /usr/lib/apache2/modules/mod_mime.so

Reading symbols from /usr/lib/apache2/modules/mod_mime_magic.so...Reading symbols from /usr/lib/debug/usr/lib/apache2/modules/mod_mime_magic.so.debug...done.

done.

Loaded symbols for /usr/lib/apache2/modules/mod_mime_magic.so

Reading symbols from /usr/lib/apache2/modules/mod_negotiation.so...Reading symbols from /usr/lib/debug/usr/lib/apache2/modules/mod_negotiation.so.debug...done.

done.

Loaded symbols for /usr/lib/apache2/modules/mod_negotiation.so

Reading symbols from /usr/lib/apache2/modules/mod_rewrite.so...Reading symbols from /usr/lib/debug/usr/lib/apache2/modules/mod_rewrite.so.debug...done.

done.

Loaded symbols for /usr/lib/apache2/modules/mod_rewrite.so

Reading symbols from /usr/lib/apache2/modules/mod_setenvif.so...Reading symbols from /usr/lib/debug/usr/lib/apache2/modules/mod_setenvif.so.debug...done.

done.

Loaded symbols for /usr/lib/apache2/modules/mod_setenvif.so

Reading symbols from /usr/lib/apache2/modules/mod_speling.so...Reading symbols from /usr/lib/debug/usr/lib/apache2/modules/mod_speling.so.debug...done.

done.

Loaded symbols for /usr/lib/apache2/modules/mod_speling.so

Reading symbols from /usr/lib/apache2/modules/mod_ssl.so...Reading symbols from /usr/lib/debug/usr/lib/apache2/modules/mod_ssl.so.debug...done.

done.

Loaded symbols for /usr/lib/apache2/modules/mod_ssl.so

Reading symbols from /usr/lib/libssl.so.1.0.0...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/libssl.so.1.0.0

Reading symbols from /usr/lib/libcrypto.so.1.0.0...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/libcrypto.so.1.0.0

Reading symbols from /usr/lib/apache2/modules/mod_status.so...Reading symbols from /usr/lib/debug/usr/lib/apache2/modules/mod_status.so.debug...done.

done.

Loaded symbols for /usr/lib/apache2/modules/mod_status.so

Reading symbols from /usr/lib/apache2/modules/mod_unique_id.so...Reading symbols from /usr/lib/debug/usr/lib/apache2/modules/mod_unique_id.so.debug...done.

done.

Loaded symbols for /usr/lib/apache2/modules/mod_unique_id.so

Reading symbols from /usr/lib/apache2/modules/mod_usertrack.so...Reading symbols from /usr/lib/debug/usr/lib/apache2/modules/mod_usertrack.so.debug...done.

done.

Loaded symbols for /usr/lib/apache2/modules/mod_usertrack.so

Reading symbols from /usr/lib/apache2/modules/mod_python.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/apache2/modules/mod_python.so

Reading symbols from /usr/lib/libpython2.7.so.1.0...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/libpython2.7.so.1.0

Reading symbols from /lib/libutil.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib/libutil.so.1

Reading symbols from /usr/lib/apache2/modules/libphp5.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/apache2/modules/libphp5.so

Reading symbols from /usr//lib/libexslt.so.0...(no debugging symbols found)...done.

Loaded symbols for /usr//lib/libexslt.so.0

Reading symbols from /usr//lib/libtidy-0.99.so.0...(no debugging symbols found)...done.

Loaded symbols for /usr//lib/libtidy-0.99.so.0

Reading symbols from /usr//lib/libsybdb.so.5...(no debugging symbols found)...done.

Loaded symbols for /usr//lib/libsybdb.so.5

Reading symbols from /usr//lib/libmcrypt.so.4...(no debugging symbols found)...done.

Loaded symbols for /usr//lib/libmcrypt.so.4

Reading symbols from /usr//lib/libonig.so.2...(no debugging symbols found)...done.

Loaded symbols for /usr//lib/libonig.so.2

Reading symbols from /usr//lib/libldap-2.4.so.2...(no debugging symbols found)...done.

Loaded symbols for /usr//lib/libldap-2.4.so.2

Reading symbols from /usr//lib/liblber-2.4.so.2...(no debugging symbols found)...done.

Loaded symbols for /usr//lib/liblber-2.4.so.2

Reading symbols from /usr//lib/libpng15.so.15...(no debugging symbols found)...done.

Loaded symbols for /usr//lib/libpng15.so.15

Reading symbols from /usr//lib/libjpeg.so.8...(no debugging symbols found)...done.

Loaded symbols for /usr//lib/libjpeg.so.8

Reading symbols from /lib/libbz2.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib/libbz2.so.1

Reading symbols from /usr//lib/libxml2.so.2...(no debugging symbols found)...done.

Loaded symbols for /usr//lib/libxml2.so.2

Reading symbols from /usr/lib/mysql/libmysqlclient.so.16...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/mysql/libmysqlclient.so.16

Reading symbols from /usr//lib/libodbc.so.1...(no debugging symbols found)...done.

Loaded symbols for /usr//lib/libodbc.so.1

Reading symbols from /usr//lib/libxslt.so.1...(no debugging symbols found)...done.

Loaded symbols for /usr//lib/libxslt.so.1

Reading symbols from /usr/lib/libgcrypt.so.11...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/libgcrypt.so.11

Reading symbols from /usr/lib/libgpg-error.so.0...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/libgpg-error.so.0

Reading symbols from /usr/lib/libltdl.so.7...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/libltdl.so.7

Reading symbols from /usr/lib/apache2/modules/mod_wsgi.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/apache2/modules/mod_wsgi.so

Reading symbols from /usr/lib/apache2/modules/mod_perl.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/apache2/modules/mod_perl.so

Reading symbols from /usr/lib/libperl.so.5.12...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/libperl.so.5.12

Reading symbols from /usr/lib/apache2/modules/mod_security2.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/apache2/modules/mod_security2.so

Reading symbols from /usr/lib/perl5/vendor_perl/5.12.2/i686-linux/auto/ModPerl/Util/Util.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/perl5/vendor_perl/5.12.2/i686-linux/auto/ModPerl/Util/Util.so

Reading symbols from /usr/lib/perl5/vendor_perl/5.12.2/i686-linux/auto/Apache2/RequestRec/RequestRec.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/perl5/vendor_perl/5.12.2/i686-linux/auto/Apache2/RequestRec/RequestRec.so

Reading symbols from /usr/lib/perl5/vendor_perl/5.12.2/i686-linux/auto/Apache2/RequestIO/RequestIO.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/perl5/vendor_perl/5.12.2/i686-linux/auto/Apache2/RequestIO/RequestIO.so

Reading symbols from /usr/lib/perl5/vendor_perl/5.12.2/i686-linux/auto/Apache2/RequestUtil/RequestUtil.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/perl5/vendor_perl/5.12.2/i686-linux/auto/Apache2/RequestUtil/RequestUtil.so

Reading symbols from /usr/lib/perl5/vendor_perl/5.12.2/i686-linux/auto/Apache2/Log/Log.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/perl5/vendor_perl/5.12.2/i686-linux/auto/Apache2/Log/Log.so

Reading symbols from /usr/lib/perl5/vendor_perl/5.12.2/i686-linux/auto/Apache2/ServerRec/ServerRec.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/perl5/vendor_perl/5.12.2/i686-linux/auto/Apache2/ServerRec/ServerRec.so

Reading symbols from /usr/lib/perl5/vendor_perl/5.12.2/i686-linux/auto/Apache2/ServerUtil/ServerUtil.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/perl5/vendor_perl/5.12.2/i686-linux/auto/Apache2/ServerUtil/ServerUtil.so

Reading symbols from /usr/lib/perl5/vendor_perl/5.12.2/i686-linux/auto/Apache2/Connection/Connection.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/perl5/vendor_perl/5.12.2/i686-linux/auto/Apache2/Connection/Connection.so

Reading symbols from /usr/lib/perl5/vendor_perl/5.12.2/i686-linux/auto/APR/Table/Table.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/perl5/vendor_perl/5.12.2/i686-linux/auto/APR/Table/Table.so

Reading symbols from /usr/lib/perl5/vendor_perl/5.12.2/i686-linux/auto/Apache2/Response/Response.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/perl5/vendor_perl/5.12.2/i686-linux/auto/Apache2/Response/Response.so

Reading symbols from /usr/lib/perl5/vendor_perl/5.12.2/i686-linux/auto/Apache2/Access/Access.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/perl5/vendor_perl/5.12.2/i686-linux/auto/Apache2/Access/Access.so

Reading symbols from /usr/lib/perl5/vendor_perl/5.12.2/i686-linux/auto/APR/Status/Status.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/perl5/vendor_perl/5.12.2/i686-linux/auto/APR/Status/Status.so

Reading symbols from /usr/lib/perl5/vendor_perl/5.12.2/i686-linux/auto/ModPerl/Global/Global.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/perl5/vendor_perl/5.12.2/i686-linux/auto/ModPerl/Global/Global.so

Reading symbols from /usr/lib/perl5/vendor_perl/5.12.2/i686-linux/auto/Apache2/Const/Const.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/perl5/vendor_perl/5.12.2/i686-linux/auto/Apache2/Const/Const.so

Reading symbols from /usr/lib/perl5/vendor_perl/5.12.2/i686-linux/auto/APR/Const/Const.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/perl5/vendor_perl/5.12.2/i686-linux/auto/APR/Const/Const.so

Reading symbols from /usr/lib/php5.3/lib/extensions/no-debug-non-zts-20090626/ssh2.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/php5.3/lib/extensions/no-debug-non-zts-20090626/ssh2.so

Reading symbols from /usr/lib/libssh2.so.1...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/libssh2.so.1

Reading symbols from /usr/lib/gconv/UTF-16.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/gconv/UTF-16.so

Reading symbols from /usr/lib/gconv/ISO8859-1.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/gconv/ISO8859-1.so

Core was generated by `/usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D STATUS -D LANGUAGE -D SSL -D SSL_'.

Program terminated with signal 6, Aborted.

#0  0x51965422 in __kernel_vsyscall ()

(gdb) bt

#0  0x51965422 in __kernel_vsyscall ()

#1  0x5155892e in raise () from /lib/libc.so.6

#2  0x5169be54 in ?? () from /lib/libc.so.6

#3  0x08090ed4 in child_main (child_num_arg=<value optimized out>) at prefork.c:692

#4  0x08091099 in make_child (s=0x80ba028, slot=8) at prefork.c:768

#5  0x08091d3a in perform_idle_server_maintenance (_pconf=0x80b8130, plog=0x80e61e8, s=0x80ba028)

    at prefork.c:903

#6  ap_mpm_run (_pconf=0x80b8130, plog=0x80e61e8, s=0x80ba028) at prefork.c:1107

#7  0x080671db in main (argc=31, argv=0x5d4b7974) at main.c:739

(gdb) 

```

here's apache config

```
www-servers/apache-2.2.21-r1  USE="ssl -debug -doc -ldap (-selinux) -static -suexec -threads" APACHE2_MODULES="actions alias asis auth_basic auth_digest authn_file authz_default authz_host authz_user cache cgi cgid charset_lite deflate dir disk_cache env expires ext_filter filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id usertrack -authn_alias -authn_anon -authn_dbd -authn_dbm -authn_default -authz_dbm -authz_groupfile -authz_owner -autoindex -cern_meta -dav -dav_fs -dav_lock -dbd -dumpio -file_cache -ident -imagemap -log_forensic -proxy -proxy_ajp -proxy_balancer -proxy_connect -proxy_ftp -proxy_http -proxy_scgi -reqtimeout -substitute -userdir -version -vhost_alias" APACHE2_MPMS="prefork -event -itk -peruser -worker" 0 kB

```

apache options

```

APACHE2_OPTS="-D DEFAULT_VHOST -D INFO -D STATUS -D LANGUAGE -D SSL -D SSL_DEFAULT_VHOST -D CGI -D PHP5 -D PYTHON -D PERL -D SECURITY -D WSGI"
```

and emerge --info

```
# emerge --info

Portage 2.1.10.41 (hardened/linux/x86, gcc-4.5.3-hardenednopie, glibc-2.11.2-r3, 2.6.32-hardened-r9 i686)

=================================================================

System uname: Linux-2.6.32-hardened-r9-i686-Intel-R-_Core-TM-2_Duo_CPU_E4500_@_2.20GHz-with-gentoo-2.0.3

Timestamp of tree: Thu, 19 Jan 2012 10:15:01 +0000

app-shells/bash:          4.1_p9

dev-java/java-config:     2.1.11-r3

dev-lang/python:          2.5.4-r4, 2.6.5-r2, 2.7.2-r3, 3.1.4-r3

dev-util/cmake:           2.8.1-r2

dev-util/pkgconfig:       0.26

sys-apps/baselayout:      2.0.3

sys-apps/openrc:          0.9.4

sys-apps/sandbox:         2.5

sys-devel/autoconf:       2.13::<unknown repository>, 2.68

sys-devel/automake:       1.5::<unknown repository>, 1.7.9-r1, 1.10.1, 1.11.1

sys-devel/binutils:       2.21.1-r1

sys-devel/gcc:            3.4.6-r2, 4.3.4, 4.5.3-r1

sys-devel/gcc-config:     1.4.1-r1

sys-devel/libtool:        2.4-r1

sys-devel/make:           3.82-r1

sys-kernel/linux-headers: 2.6.39 (virtual/os-headers)

sys-libs/glibc:           2.11.2-r3

Repositories: gentoo x-portage

ACCEPT_KEYWORDS="x86"

ACCEPT_LICENSE="*"

CBUILD="i686-pc-linux-gnu"

CFLAGS="-march=prescott -O2 -pipe -ggdb"

CHOST="i686-pc-linux-gnu"

CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt"

CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.3/ext-active/ /etc/php/apache2-php5.4/ext-active/ /etc/php/cgi-php5.3/ext-active/ /etc/php/cgi-php5.4/ext-active/ /etc/php/cli-php5.3/ext-active/ /etc/php/cli-php5.4/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/udev/rules.d"

CXXFLAGS="-march=prescott -O2 -pipe -ggdb"

DISTDIR="/usr/portage/distfiles"

FEATURES="assume-digests binpkg-logs ccache collision-protect distlocks ebuild-locks fixlafiles news parallel-fetch protect-owned sandbox sfperms splitdebug strict unknown-features-warn unmerge-logs unmerge-orphans userfetch"

FFLAGS=""

GENTOO_MIRRORS="http://ftp.ucsb.edu/pub/mirrors/linux/gentoo/  http://distfiles.gentoo.org http://www.biblio.org/pub/Linux/distributions/gentoo"

LDFLAGS="-Wl,-O1 -Wl,--as-needed"

MAKEOPTS="-j3"

PKGDIR="/usr/portage/packages"

PORTAGE_CONFIGROOT="/"

PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"

PORTAGE_TMPDIR="/var/tmp"

PORTDIR="/usr/portage"

PORTDIR_OVERLAY="/usr/local/portage"

SYNC="rsync://mirror.ucsd.edu/gentoo-portage"

USE="acl berkdb bzip2 cli cracklib crypt cups cxx dri gdbm gpm hardened iconv modules mudflap ncurses nls nptl nptlonly openmp pam pax_kernel pcre pic pppd readline session ssl symlink sysfs tcpd urandom x86 xorg zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1 emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions authn_file authz_host authz_user authz_default auth_basic auth_digest alias charset_lite cache disk_cache mem_cache ext_filter deflate filter mime_magic expires headers usertrack unique_id info include cgi cgid speling rewrite log_config logio env setenvif mime status asis negotiation dir so" APACHE2_MPMS="prefork" CALLIGRA_FEATURES="kexi words flow plan stage tables krita karbon braindump" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" PHP_TARGETS="php5-3" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="apm ark chips cirrus cyrix dummy fbdev glint i128 i740 intel mach64 mga neomagic nsc nv r128 radeon rendition s3 s3virge savage siliconmotion sis sisusb tdfx tga trident tseng v4l vesa via vmware nouveau" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"

Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LC_ALL, LINGUAS, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
```

I'm still baffled so any pointers appreciated. Thanks

----------

## Hu

It looks like it may be aborting intentionally.  This can happen if an assertion fails.  However, I lack the background to explain why this process would abort in this way.  I think you need to take this to the Apache maintainers now.

----------

## huuan

Thanks so much for casting your expert eye over the backtrace.

I think maybe it's got caught in a difficult glibc upgrade.

During the many+++ packages upgrade it called for a glibc update as I recall...

said somethings about  *Quote:*   

> Your system still has old SSP __guard symbols. You need to
> 
>  * rebuild all the packages that provide these files first:
> 
>  * //bin/ed 
> ...

 

 that glibc update wanted a whole load of files updated as above before it would be happy, so I masked it. 

Why? well one of the things it wanted to update was shibboleth and there was an unresolved issue with updating xml-securiy-c for that with a bug filed which has just this week been resolved. 

These two bt lines lead me to that conclusion

```

#1  0x5155892e in raise () from /lib/libc.so.6 

#2  0x5169be54 in ?? () from /lib/libc.so.6 

```

but I maybe way off-base as I am a complete neophyte in this area and am clutching at straws converting libc in my head to glibc.

Usually when a gentoo upgrade breaks for me it is because of something stupid that I have done during the process.

Be happy to take this up with apache but I don't want them chasing after some red herring that is a unique creation of my system. Once I confirm that I have done all updates, revdep-rebuilds, etc. and it still lingers I'll pass it upstream.

OK I just now checked and glibc update has become sane again now that I have shibboleth-sp 2.3.4 installed and the old shib un-installed.

I'll try updating glibc and reemerging apache to see if that fixes things.

Your input has helped me learn a lot.

Thanks again for your help.

----------

## Hu

 *huuan wrote:*   

> I think maybe it's got caught in a difficult glibc upgrade.
> 
> These two bt lines lead me to that conclusion
> 
> ```
> ...

 That is a reasonable guess, but unfortunately it is not substantiated by the facts.  raise is a standard C library function used to deliver a signal to the process which calls it.  It is used by, among other things, abort, which causes the process to receive a SIGABRT.  Those lines strongly imply that the SIGABRT was self-induced, but do not, by themselves, tell us why the process decided to abort.

 *huuan wrote:*   

> Be happy to take this up with apache but I don't want them chasing after some red herring that is a unique creation of my system. Once I confirm that I have done all updates, revdep-rebuilds, etc. and it still lingers I'll pass it upstream.

 I understand.  If you want to collect a new backtrace post-upgrade, I will review it before you pass it on.  I am unsure whether I can add any further insight, but it does not take long for me to determine if I can help.

----------

## huuan

I upgraded a bunch more stuff and here is another backtrace

```
# gdb /usr/sbin/apache2 --core core-2

GNU gdb (Gentoo 7.3.1 p2) 7.3.1

Copyright (C) 2011 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.  Type "show copying"

and "show warranty" for details.

This GDB was configured as "i686-pc-linux-gnu".

For bug reporting instructions, please see:

<http://bugs.gentoo.org/>...

Reading symbols from /usr/sbin/apache2...Reading symbols from /usr/lib/debug/usr/sbin/apache2.debug...done.

done.

[New LWP 29745]

warning: Can't read pathname for load map: Input/output error.

[Thread debugging using libthread_db enabled]

Core was generated by `/usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D STATUS -D LANGUAGE -D SSL -D SSL_'.

Program terminated with signal 6, Aborted.

#0  0x55511422 in __kernel_vsyscall ()

(gdb) bt

#0  0x55511422 in __kernel_vsyscall ()

#1  0x550fb968 in raise () from /lib/libc.so.6

#2  0x550fd2c2 in abort () from /lib/libc.so.6

#3  0x54d5b139 in Py_FatalError () from /usr/lib/libpython2.7.so.1.0

#4  0x54da5b0f in ?? () from /usr/lib/libpython2.7.so.1.0

#5  0x08090ec4 in child_main (child_num_arg=<optimized out>) at prefork.c:692

#6  0x08091089 in make_child (s=0x80b7e78, slot=2) at prefork.c:768

#7  0x08091151 in startup_children (number_to_start=3) at prefork.c:786

#8  0x08091cd1 in ap_mpm_run (_pconf=0x80b5f80, plog=0x80e4038, s=0x80b7e78) at prefork.c:1007

#9  0x080671db in main (argc=31, argv=0x5f62f7a4) at main.c:739

```

So I figured maybe I might have missed a python-updater so ran that and it wanted to update a bunch more, did that and it more difficult to get aborts. After reloading a page with the shift key down a bunch got another core, backtrace as below:

```
# gdb /usr/sbin/apache2 --core core-3

GNU gdb (Gentoo 7.3.1 p2) 7.3.1

Copyright (C) 2011 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.  Type "show copying"

and "show warranty" for details.

This GDB was configured as "i686-pc-linux-gnu".

For bug reporting instructions, please see:

<http://bugs.gentoo.org/>...

Reading symbols from /usr/sbin/apache2...Reading symbols from /usr/lib/debug/usr/sbin/apache2.debug...done.

done.

[New LWP 27622]

warning: Can't read pathname for load map: Input/output error.

[Thread debugging using libthread_db enabled]

Core was generated by `/usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D STATUS -D LANGUAGE -D SSL -D SSL_'.

Program terminated with signal 6, Aborted.

#0  0x512ef422 in __kernel_vsyscall ()

(gdb) bt

#0  0x512ef422 in __kernel_vsyscall ()

#1  0x50ed9968 in raise () from /lib/libc.so.6

#2  0x50edb2c2 in abort () from /lib/libc.so.6

#3  0x50b39139 in Py_FatalError () from /usr/lib/libpython2.7.so.1.0

#4  0x50b83b0f in ?? () from /usr/lib/libpython2.7.so.1.0

#5  0x08090ec4 in child_main (child_num_arg=<optimized out>) at prefork.c:692

#6  0x08091089 in make_child (s=0x80ad2c8, slot=7) at prefork.c:768

#7  0x08091d2a in perform_idle_server_maintenance (p=<optimized out>) at prefork.c:903

#8  ap_mpm_run (_pconf=0x80ab3d0, plog=0x80d9488, s=0x80ad2c8) at prefork.c:1107

#9  0x080671db in main (argc=31, argv=0x5cc168f4) at main.c:739

```

Any insight? Thanks.

----------

## Hu

This implicates Python 2.7, but does not give enough detail to understand why.  Could you rebuild Python with debugging symbols and reproduce the problem again?

----------

## huuan

Here's another core after python2.7 rebuild w debug, python-updater and apache rebuilt (w debug)

```
apache-core-dump #  gdb /usr/sbin/apache2 --core core-5

GNU gdb (Gentoo 7.3.1 p2) 7.3.1

Copyright (C) 2011 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.  Type "show copying"

and "show warranty" for details.

This GDB was configured as "i686-pc-linux-gnu".

For bug reporting instructions, please see:

<http://bugs.gentoo.org/>...

Reading symbols from /usr/sbin/apache2...Reading symbols from /usr/lib/debug/usr/sbin/apache2.debug...done.

done.

[New LWP 608]

warning: Can't read pathname for load map: Input/output error.

[Thread debugging using libthread_db enabled]

Core was generated by `/usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D STATUS -D LANGUAGE -D SSL -D SSL_'.

Program terminated with signal 6, Aborted.

#0  0x499c5422 in __kernel_vsyscall ()

(gdb) bt

#0  0x499c5422 in __kernel_vsyscall ()

#1  0x495af968 in raise () from /lib/libc.so.6

#2  0x495b12c2 in abort () from /lib/libc.so.6

#3  0x4920e55e in Py_FatalError (

    msg=0x49253978 "PyEval_AcquireThread: NULL new thread state")

    at Python/pythonrun.c:1670

#4  0x491eb2b2 in PyEval_AcquireThread (tstate=0x0) at Python/ceval.c:272

#5  0x4829caf5 in ?? () from /usr/lib/apache2/modules/mod_wsgi.so

#6  0x08090ec4 in child_main (child_num_arg=<optimized out>) at prefork.c:692

#7  0x08091089 in make_child (s=0x80b7928, slot=6) at prefork.c:768

#8  0x08091d2a in perform_idle_server_maintenance (p=<optimized out>)

    at prefork.c:903

#9  ap_mpm_run (_pconf=0x80b5a30, plog=0x80e3ae8, s=0x80b7928)

    at prefork.c:1107

#10 0x080671db in main (argc=31, argv=0x59965914) at main.c:739
```

tried a gdb on the core w python2.7 but it just gave a couple entries with ? after them

Thanks.

----------

## huuan

Note: Don't know if it is related to the apache issue but I haven't been able to upgrade dev-perl/libwww-perl-6.30.0 [5.836] to the latest due to blocks which I will ask about in another thread

----------

## Hu

Now we have real progress.  The latest backtrace shows that mod_wsgi, an Apache module intended to accelerate the execution of Python scripts, is using the Python interpreter in a way that causes Python to abort the entire process.  Unfortunately, mod_wsgi is not part of the main Apache distribution, but instead comes from www-apache/mod_wsgi.  As such, you do not have symbols for it.  We can iterate this again by rebuilding that and producing yet another core, or you could file a bug with the mod_wsgi maintainers in the hope that they can fix it based on the error message shown in the current stack trace ("PyEval_AcquireThread: NULL new thread state").

As a temporary workaround, you could disable the use of mod_wsgi.  This will let your server run, but may impact the performance of your Python scripts - if they even run at all.  However, a server that successfully offers static content and fails to run Python scripts may be more useful to you than a server which simply does not run.  :Smile: 

----------

## huuan

This is awesome news as I am moving into a season where production downtime is not an easy option and those cores were making me nervous.   

I'm using mod_wsgi only to run trac (AFAIK) so I'm not sure why it is even appearing in the scene when trac is not being called for index files that were causing the cores.

I can uninstall it on the production server for now as it is mostly a LAMP box and continue debug on the backup server. I'll rebuild mod_wsgi with debug and bt for the bug. 

Thanks!   :Very Happy: 

[edit] have now restarted the production server with mod_wsgi disabled and so far no more aborts.

----------

## huuan

after rebuilding mod_wsgi w debug

```
# gdb /usr/sbin/apache2 --core core-6

GNU gdb (Gentoo 7.3.1 p2) 7.3.1

Copyright (C) 2011 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.  Type "show copying"

and "show warranty" for details.

This GDB was configured as "i686-pc-linux-gnu".

For bug reporting instructions, please see:

<http://bugs.gentoo.org/>...

Reading symbols from /usr/sbin/apache2...Reading symbols from /usr/lib/debug/usr/sbin/apache2.debug...done.

done.

[New LWP 16205]

warning: Can't read pathname for load map: Input/output error.

[Thread debugging using libthread_db enabled]

Core was generated by `/usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D STATUS -D LANGUAGE -D SSL -D SSL_'.

Program terminated with signal 6, Aborted.

#0  0x4d15c422 in __kernel_vsyscall ()

(gdb) bt

#0  0x4d15c422 in __kernel_vsyscall ()

#1  0x4cd46968 in raise () from /lib/libc.so.6

#2  0x4cd482c2 in abort () from /lib/libc.so.6

#3  0x4c9a555e in Py_FatalError (msg=0x4c9ea978 "PyEval_AcquireThread: NULL new thread state")

    at Python/pythonrun.c:1670

#4  0x4c9822b2 in PyEval_AcquireThread (tstate=0x0) at Python/ceval.c:272

#5  0x4ba329a9 in wsgi_python_child_cleanup (data=0x0) at mod_wsgi.c:6669

#6  0x4cf13cc8 in apr_pool_destroy () from /usr/lib/libapr-1.so.0

#7  0x59ead7ab in ?? ()

Cannot access memory at address 0x7af44f94

(gdb) 

```

I'll bug it over to the wsgi folks... https://bugs.gentoo.org/show_bug.cgi?id=401225

----------

