# Apache2: Cannot allocate memory

## qQsh

I'm having memory errors with apache. I start up Apache and it's mostly working but 2-3 times a day two child processes are dieing with signal 11 and apache "Cannot allocate memory: fork: Unable to fork new process" for 1-2 seconds. After that, server continues to working until this happens again. From client side it looks like webserver goes away for 1-2 seconds and then everything is fine again.

All other processes (mailserver, for example) are working without any memory problems and i replaced and tested memory with memtest86.

I'm trying to solve this for a weeks but it's still the same. Maybe someone can help with suggestions or workarounds, or help with debugging it.

```
[Sat Jan 08 15:10:21 2011] [notice] Apache/2.2.16 (Unix) proxy_html/3.0.0 PHP/5.2.17-pl0-gentoo mod_ssl/2.2.16 OpenSSL/0.9.8o configured -- resuming normal operations

[Sat Jan 08 19:53:36 2011] [error] child died with signal 11

[Sat Jan 08 19:53:36 2011] [error] child died with signal 11

[Sat Jan 08 19:53:36 2011] [error] (12)Cannot allocate memory: fork: Unable to fork new process

[Sat Jan 08 19:53:36 2011] [error] (12)Cannot allocate memory: fork: Unable to fork new process

[Sat Jan 08 19:53:36 2011] [error] (12)Cannot allocate memory: fork: Unable to fork new process

[Sat Jan 08 19:53:36 2011] [error] (12)Cannot allocate memory: fork: Unable to fork new process

[Sat Jan 08 19:53:36 2011] [error] (12)Cannot allocate memory: fork: Unable to fork new process

[Sat Jan 08 19:53:36 2011] [error] (12)Cannot allocate memory: fork: Unable to fork new process

[Sat Jan 08 19:53:36 2011] [error] (12)Cannot allocate memory: fork: Unable to fork new process

...

```

System info:

```
Gentoo Linux 64bit ~x86, dedicated

# apache2 -l

Compiled in modules:

  core.c

  itk.c

  http_core.c

  mod_so.c

 

APACHE2_OPTS="-D INFO -D PHP5 -D LIMITIPCONN -D STATUS -D SSL -D EVASIVE -D PROXY_HTML -D DEFLATE -D NAGIOS"

 

# free -m

             total       used       free     shared    buffers     cached

Mem:          7991       7901         89          0        419       4563

-/+ buffers/cache:       2919       5072

Swap:          956         45        911

 

# /usr/sbin/apache2 -D INFO -D PHP5 -D LIMITIPCONN -D STATUS -D SSL -D EVASIVE -D PROXY_HTML -D DEFLATE -D NAGIOS -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start -M

Loaded Modules:

 core_module (static)

 mpm_itk_module (static)

 http_module (static)

 so_module (static)

 authz_host_module (shared)

 auth_basic_module (shared)

 authn_file_module (shared)

 authz_owner_module (shared)

 authz_user_module (shared)

 authz_groupfile_module (shared)

 authn_dbm_module (shared)

 authz_dbm_module (shared)

 auth_digest_module (shared)

 log_config_module (shared)

 env_module (shared)

 mime_magic_module (shared)

 expires_module (shared)

 headers_module (shared)

 usertrack_module (shared)

 unique_id_module (shared)

 setenvif_module (shared)

 proxy_module (shared)

 proxy_connect_module (shared)

 proxy_http_module (shared)

 mime_module (shared)

 status_module (shared)

 autoindex_module (shared)

 info_module (shared)

 cgi_module (shared)

 vhost_alias_module (shared)

 negotiation_module (shared)

 dir_module (shared)

 actions_module (shared)

 alias_module (shared)

 rewrite_module (shared)

 file_cache_module (shared)

 cache_module (shared)

 disk_cache_module (shared)

 ext_filter_module (shared)

 deflate_module (shared)

 evasive_module (shared)

 limitipconn_module (shared)

 proxy_html_module (shared)

 php5_module (shared)

 ssl_module (shared)

Syntax OK

# emerge --info

Portage 2.1.9.25 (default/linux/x86/10.0/server, gcc-4.1.2, glibc-2.10.1-r1, 2.6.36-gentoo-r5 x86_64)

=================================================================

System uname: Linux-2.6.36-gentoo-r5-x86_64-Intel-R-_Core-TM-2_Quad_CPU_Q6600_@_2.40GHz-with-gentoo-1.12.11.1

Timestamp of tree: Sat, 08 Jan 2011 11:30:02 +0000

distcc 2.18.3 i686-pc-linux-gnu (protocols 1 and 2) (default port 3632) [disabled]

app-shells/bash:     3.2_p33

dev-java/java-config: 1.3.7-r1, 2.1.10

dev-lang/python:     2.4.4-r13, 2.5.2-r7, 2.6.4

sys-apps/baselayout: 1.12.11.1

sys-apps/sandbox:    1.6-r2

sys-devel/autoconf:  2.13, 2.65

sys-devel/automake:  1.4_p6::<unknown repository>, 1.5::<unknown repository>, 1.6.3::<unknown repository>, 1.7.9-r1, 1.8.5-r3::<unknown repository>, 1.9.6-r1::<unknown repository>, 1.10.1, 1.11.1

sys-devel/binutils:  2.18-r3

sys-devel/gcc:       4.1.2

sys-devel/gcc-config: 1.4.0-r4

sys-devel/libtool:   2.2.6b

sys-devel/make:      3.81

virtual/os-headers:  2.6.23-r3 (sys-kernel/linux-headers)

ACCEPT_KEYWORDS="x86"

ACCEPT_LICENSE="*"

CBUILD="i686-pc-linux-gnu"

CFLAGS="-march=prescott -O2 -pipe -fomit-frame-pointer"

CHOST="i686-pc-linux-gnu"

CONFIG_PROTECT="/etc /var/bind"

CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/eselect/postgresql /etc/fonts/fonts.conf /etc/gconf /etc/php/apache2-php5.2/ext-active/ /etc/php/apache2-php5/ext-active/ /etc/php/cgi-php5.2/ext-active/ /etc/php/cgi-php5/ext-active/ /etc/php/cli-php5.2/ext-active/ /etc/php/cli-php5/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/udev/rules.d"

CXXFLAGS="-march=prescott -O2 -pipe -fomit-frame-pointer"

DISTDIR="/usr/portage/distfiles"

FEATURES="assume-digests binpkg-logs distlocks fixlafiles fixpackages news parallel-fetch protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch"

GENTOO_MIRRORS="http://distfiles.gentoo.org"

LANG="en_GB.UTF-8"

LDFLAGS="-Wl,-O1 -Wl,--as-needed"

MAKEOPTS="-j5"

PKGDIR="/usr/portage/packages"

PORTAGE_CONFIGROOT="/"

PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"

PORTAGE_TMPDIR="/var/tmp"

PORTDIR="/usr/portage"

PORTDIR_OVERLAY="/usr/local/portage"

SYNC="rsync://rsync.gentoo.org/gentoo-portage"

USE="acl apache2 authdaemond bzip2 cgi cli concurrentmodphp cracklib crypt ctype cups cxx dri fortran gd gd2 gdbm gdlib gpm iconv imap json latin1 libgd modules mudflap mysql ncurses nptl nptlonly openmp overload pam pcre php pop3d pppd readline sasl session simplexml snmp softquota ssl sysfs tokenizer truetype unicode vda x86 xml xmlrpc xorg xsl zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1 emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic auth_digest authn_anon authn_dbd authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock dbd deflate dir disk_cache env expires ext_filter file_cache filter headers ident imagemap include info log_config logio mem_cache mime mime_magic negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_http rewrite setenvif so speling status unique_id userdir usertrack vhost_alias cgi cgid" APACHE2_MPMS="itk" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" PHP_TARGETS="php5-2" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="fbdev glint intel mach64 mga neomagic nouveau nv r128 radeon savage sis tdfx trident vesa via vmware dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" 

Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, FFLAGS, INSTALL_MASK, LC_ALL, LINGUAS, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS

www-servers/apache-2.2.16  USE="debug ssl -doc -ldap (-selinux) -static -suexec -threads" APACHE2_MODULES="actions alias auth_basic auth_digest authn_anon authn_dbd authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock dbd deflate dir disk_cache env expires ext_filter file_cache filter headers ident imagemap include info log_config logio mem_cache mime mime_magic negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_http rewrite setenvif speling status unique_id userdir usertrack vhost_alias -asis -authn_alias -cern_meta -charset_lite -dumpio -log_forensic -proxy_ftp -substitute -version" APACHE2_MPMS="itk -event -peruser -prefork -worker" 

```

Thanks for looking into my problem.

----------

## gerdesj

To attack the memory tight thing install atop (you could use top but atop is better) and monitor your memory usage.  Use m to sort on memory usage and see if anything might have a memory leak and using huge amounts of RAM.

Or the problem could be within Apache itself.  Can you correlate the problem with anything in access_log?  ie is the same content being accessed at the same time as the error.

There are lots of other factors which you might need to eliminate/consider.

Is the Apache accessible from the internet

Is Apache up to date

Is all the software on this system up to date

fdisk

Personally I'd first get some more debugging info out of Apache if possible.  There's probably a debug setting somewhere that can increase the amount of log info and then look for a correlation.  Also run atop with its logging switched on to look at the overall system state at the same time.

Cheers

Jon

----------

