# Out of memory problem

## adelante

Does anyone have any idea what could cause this:

 *Quote:*   

> 
> 
> Sep  9 16:51:08 Hyperion kernel: oom-killer: gfp_mask=0xd2
> 
> Sep  9 16:51:09 Hyperion kernel: DMA per-cpu:
> ...

 

EDIT: i am using net-www/apache-2.0.54-r9

----------

## zieloo

This is caused by lack of memory. Linux's policy is to kill processes with the greatest mem usage and new processes when running out of memory. Use 'free' to see how much mem you've got left. If you're running a webserver, maybe it's getting too many requests? Revise your apache confguration.

----------

## adelante

This machine has 2GB of Ram... maybe I should consider upping it, but whats wierd is that this server isnt that busy.

Could it not have been a denial of services attack, where they just keep hold of the connections, since KeepAlive is on in the config file.

----------

## hanj

Hello

This is really strange.. I just encountered this for the first time as well.. and on the same date as you????

```
Sep 12 14:25:59 mycomp.com oom-killer: gfp_mask=0x1d2

Sep 12 14:26:00 mycomp.com DMA per-cpu:

Sep 12 14:26:00 mycomp.com cpu 0 hot: low 2, high 6, batch 1

Sep 12 14:26:00 mycomp.com cpu 0 cold: low 0, high 2, batch 1

Sep 12 14:26:00 mycomp.com Normal per-cpu:

Sep 12 14:26:00 mycomp.com cpu 0 hot: low 32, high 96, batch 16

Sep 12 14:26:00 mycomp.com cpu 0 cold: low 0, high 32, batch 16

Sep 12 14:26:00 mycomp.com HighMem per-cpu: empty

Sep 12 14:26:00 mycomp.com

Sep 12 14:26:00 mycomp.com Free pages:        4852kB (0kB HighMem)

Sep 12 14:26:00 mycomp.com Active:61786 inactive:61489 dirty:0 writeback:4 unstable:0 free:1213 slab:3129 mapped:122763 pagetab

les:715

Sep 12 14:26:00 mycomp.com DMA free:2068kB min:88kB low:108kB high:132kB active:5436kB inactive:5288kB present:16384kB pages_sc

anned:219 all_unreclaimable? no

Sep 12 14:26:00 mycomp.com lowmem_reserve[]: 0 495 495

Sep 12 14:26:00 mycomp.com Normal free:2784kB min:2804kB low:3504kB high:4204kB active:241708kB inactive:240668kB present:50782

4kB pages_scanned:69997 all_unreclaimable? no

Sep 12 14:26:00 mycomp.com lowmem_reserve[]: 0 0 0

Sep 12 14:26:00 mycomp.com HighMem free:0kB min:128kB low:160kB high:192kB active:0kB inactive:0kB present:0kB pages_scanned:0

all_unreclaimable? no

Sep 12 14:26:00 mycomp.com lowmem_reserve[]: 0 0 0

Sep 12 14:26:00 mycomp.com DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 2068kB

Sep 12 14:26:00 mycomp.com Normal: 0*4kB 0*8kB 2*16kB 2*32kB 0*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2784kB

Sep 12 14:26:00 mycomp.com HighMem: empty

Sep 12 14:26:00 mycomp.com Swap cache: add 264861, delete 264537, find 2940/4205, race 0+3

Sep 12 14:26:00 mycomp.com Free swap  = 0kB

Sep 12 14:26:00 mycomp.com Total swap = 1006028kB

Sep 12 14:26:00 mycomp.com Out of Memory: Killed process 25169 (apache2).
```

Nothing has changed on the server as far as web applications or server configurations (PHP, etc). This has happened twice (right after each other ) on the 9th of Sept.. 4 times on the 12th and once today, the 13th.

I upgraded apache to apache-2.0.54-r15 on the 11th, and the only package installed on the 9th was python.

I'm wondering if there is a bug somewhere?? If this is not a bug, how should I start to hunt down the out of control processes??

Here is my vital info:

```
Linux mycomp.com 2.6.11-hardened-r15 #2 Fri Jul 22 21:43:30 MDT 2005 i686 AMD Duron(TM)Processor AuthenticAMD GNU/Linux
```

```
[ebuild   R   ] dev-db/mysql-4.0.25-r2  +berkdb -big-tables -debug -doc -minimal +perl +readline (-selinux) +ssl -static +tcpd 10,943 kB

[ebuild   R   ] net-www/apache-2.0.54-r15  -berkdb -doc -gdbm -ipv6 -ldap (-selinux) +ssl -static -threads 5,436 kB

[ebuild   R   ] dev-php/mod_php-4.4.0  -X +apache2 -berkdb +crypt -curl -debug -doc -fdftk -firebird -flash -freetds +gd -gd-external -gdbm -gmp -hardenedphp -imap -informix -ipv6 -java +jpeg -kerberos -ldap -mcal -memlimit -mssql +mysql -nls -oci8 -odbc -pam +png -postgres +snmp -spell +ssl -tiff -truetype -xml2 -yaz 3,990 kB
```

```
Portage 2.0.51.22-r2 (default-linux/x86/2005.0, gcc-3.3.6, glibc-2.3.5-r1, 2.6.11-hardened-r15 i686)

=================================================================

System uname: 2.6.11-hardened-r15 i686 AMD Duron(TM)Processor

Gentoo Base System version 1.6.13

dev-lang/python:     2.2.3-r1, 2.3.5-r2

sys-apps/sandbox:    1.2.12

sys-devel/autoconf:  2.13, 2.59-r6

sys-devel/automake:  1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6

sys-devel/binutils:  2.15.92.0.2-r10

sys-devel/libtool:   1.5.18-r1

virtual/os-headers:  2.4.19-r1, 2.6.11-r2

ACCEPT_KEYWORDS="x86"

AUTOCLEAN="yes"

CBUILD="i686-pc-linux-gnu"

CFLAGS="-O3 -march=i686 -funroll-loops -pipe"

CHOST="i686-pc-linux-gnu"

CONFIG_PROTECT="/etc /usr/kde/2/share/config /usr/kde/3/share/config /usr/share/config /var/qmail/control"

CONFIG_PROTECT_MASK="/etc/gconf /etc/terminfo /etc/env.d"

CXXFLAGS="-O3 -march=i686 -funroll-loops -pipe"

DISTDIR="/usr/portage/distfiles"

FEATURES="autoconfig distlocks sandbox sfperms strict"

GENTOO_MIRRORS="http://distfiles.gentoo.org http://distro.ibiblio.org/pub/Linux/distributions/gentoo"

MAKEOPTS="-j2"

PKGDIR="/usr/portage/packages"

PORTAGE_TMPDIR="/var/tmp"

PORTDIR="/usr/portage"

PORTDIR_OVERLAY="/usr/local/portage"

SYNC="rsync://rsync.gentoo.org/gentoo-portage"

USE="x86 alsa apache2 apm berkdb crypt eds emboss fortran gd gdbm gif gpm gstreamer imagemagick imlib innodb jpeg libg++ libwww maildir mp3 mysql ncurses ogg openssh pam pdflib perl php png pwdb python readline sasl slang snmp snortsam spell ssl tcpd tiff vorbis xml2 zlib userland_GNU kernel_linux elibc_glibc"

Unset:  ASFLAGS, CTARGET, LANG, LC_ALL, LDFLAGS, LINGUAS
```

Thanks much.

hanji

----------

## DennisBagley

i seem to be having problems with php5 and mysql4.1 - was fine with php5 and mysql4.0... as far as i understood although mysql4.1 is masked this is because of dependacies - and it should be ok in itself ...

so i have 2 machines and on the 4.1 box when i call certain webpages it mysqld cpu usage goes up to 99.9% for a while then it seems to give up

so wondering if this is linked

puge at barse dot freeserve dot co dot uk

----------

## hanj

hmmm

Does the oom-killer nuke the mysql process or apache? I'm currently running php4.4.0 and mysql 4.0.25. oom-killer seems to bite on apache processes only.

thanks

hanji

----------

## pof

I am having exactly the same problem, oom-killer kills my apache2. I have 1,5Gb of RAM and 1 Gb of swap. RAM + SWAP seems to be full when this happens and server "hungs" for a while.

Looking at the apache logs it seems to me that this happens when some "spammers" try to send POST request to my wordpress (blog) to do comment spam, they don't close the connection and apache launches a lot of threads. Bandwidth is not an issue here, it's just a matter of server overload.

Any help would be appreciated.

----------

## bollucks

 *pof wrote:*   

> I am having exactly the same problem, oom-killer kills my apache2. I have 1,5Gb of RAM and 1 Gb of swap. RAM + SWAP seems to be full when this happens and server "hungs" for a while.
> 
> Looking at the apache logs it seems to me that this happens when some "spammers" try to send POST request to my wordpress (blog) to do comment spam, they don't close the connection and apache launches a lot of threads. Bandwidth is not an issue here, it's just a matter of server overload.
> 
> Any help would be appreciated.

 

A carefully administrated apache needs to have an upper limit on the number of processes to prevent this from happening (it's basically a Denial Of Service attack). Check the memory usage of each apache process in top, then divide your total physical (not swap) ram by this value, then put an upper limit on total processes (servers * threads) to be slightly smaller than that.

----------

## pof

I use worker MPM (DEFAULT IF USE=threads), and have changed the default values in httpd.conf like this, but this doesn't solve the problem:

StartServers         1

MaxClients          25

MinSpareThreads      4

MaxSpareThreads     12

ThreadsPerChild      5

MaxRequestsPerChild 10

Applying your rule (servers*threads < total phys mem) with this configuration it should be more than enough memory to handle all requests, but the amount of virtual memory a single apache proccess can consume varies a lot, from 20 to 900Mb so I don't think this is the issue here.

----------

