# OOM Killer Issue

## techmav

I just reinstalled Gentoo on my laptop on Oct 8, 2006.  Prior to this, I could leave my system up and running for months at a time with no problems.  Since then, I've been running into a problem with the OOM Killer.  Whenever I leave the system up and running for more than a few days, it kicks in and usually kills Firefox.  I understand how it's supposed to work, in that it kills processes when the system starts running out of physical and swapspace, but it's kicking in when I am just barely starting to get into the swapspace.  I have managed to also trace down certain other programs that are leaking memory, such as xine.  If I watch a video using xine, OOM Killer kicks in much sooner.  Any ideas or suggestions?  This is getting bad enough to make me consider switching back to windows,and I've been using linux since 1998, and Gentoo since about 2004.

USE flags from /etc/make.conf:

```
USE="-gnome -kde jpeg gif png tiff spell win32codecs xv dvd X mad a52 aac aalib alsa asf esd oss vcd opengl"

```

Software I'm running:

```
mozilla-firefox-1.5.0.7

xorg-x11-7.0-r1

windowmaker-0.92.0-r3

gaim-1.5.0

xchat-2.6.6

```

Hardware Specs:

```
Dell ThinkPad T21

PIII 800MHz Processor

512MB RAM

60GB HDD

512MB swap space configured on /dev/hda2

```

free -m before I went to bed last night:

```
             total       used       free     shared    buffers     cached

Mem:           502        462         40          0          0        140

-/+ buffers/cache:        321        181

Swap:          509          0        509

```

free -m this morning, after OOM kicked in and killed Firefox, with Firefox restarted:

```
             total       used       free     shared    buffers     cached

Mem:           502        497          5          0          0         15

-/+ buffers/cache:        481         21

Swap:          509         54        455

```

uname -a:

```
Linux carter 2.6.17-gentoo-r5 #3 SMP Sun Oct 8 11:04:23 EDT 2006 i686 Pentium III (Coppermine) GNU/Linux

```

dmesg output:

```
oom-killer: gfp_mask=0x200d2, order=0

 <c0130286>   <c0131a8a> 

 <c013d55e>   <c013684d> 

 <c0137ceb>   <c03bacee> 

 <c010f4d3>   <c010f2d4> 

 <c0103243>  

Mem-info:

DMA per-cpu:

cpu 0 hot: high 0, batch 1 used:0

cpu 0 cold: high 0, batch 1 used:0

DMA32 per-cpu: empty

Normal per-cpu:

cpu 0 hot: high 186, batch 31 used:24

cpu 0 cold: high 62, batch 15 used:29

HighMem per-cpu: empty

Free pages:        5460kB (0kB HighMem)

Active:1106 inactive:13192 dirty:0 writeback:168 unstable:0 free:1365 slab:11230

8 mapped:14125 pagetables:199

DMA free:2068kB min:88kB low:108kB high:132kB active:2784kB inactive:0kB present

:16384kB pages_scanned:3564 all_unreclaimable? yes

lowmem_reserve[]: 0 0 495 495

DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB page

s_scanned:0 all_unreclaimable? no

lowmem_reserve[]: 0 0 495 495

Normal free:3392kB min:2804kB low:3504kB high:4204kB active:1640kB inactive:5276

8kB present:507840kB pages_scanned:192 all_unreclaimable? no

lowmem_reserve[]: 0 0 0 0

HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:

0kB pages_scanned:0 all_unreclaimable? no

lowmem_reserve[]: 0 0 0 0

DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 

0*4096kB = 2068kB

DMA32: empty

Normal: 168*4kB 6*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 1*20

48kB 0*4096kB = 3392kB

HighMem: empty

Swap cache: add 214150, delete 201202, find 73390/135752, race 0+2

Free swap  = 401012kB

Total swap = 522104kB

Free swap:       401012kB

131056 pages of RAM

0 pages of HIGHMEM

2361 reserved pages

3185 pages shared

12948 pages swap cached

0 pages dirty

167 pages writeback

14125 pages mapped

112308 pages slab

199 pages pagetables

Out of Memory: Kill process 5788 (mozilla-launche) score 19793 and children.

Out of memory: Killed process 5797 (firefox-bin).

```

----------

## bollucks

Make sure you haven't got a kernel related memory leak. Check the output of /proc/slabinfo when your machine gets oom killer happy and see if something is unusually large. If you do have a kernel memory leak, often the non mainline kernel things are responsible.

----------

## techmav

Tried a new kernel, gentoo-sources 2.6.18--gentoo-r1.  Memory usage seemed to be more stable, until about 8am this morning, after I went to work.  I have free -m running once an hour, dumping to a file, and memory usage just started pegging.  OOM kicked in again, and killed xinit this time.  Going to compile the latest version of X and see if that helps any.  This is really getting frustrating.

----------

## energyman76b

firefox has several memory leaks - and leaks like mad.

Just close it, if you don't use it and only open it, if really needed.

----------

## devsk

 *techmav wrote:*   

> Tried a new kernel, gentoo-sources 2.6.18--gentoo-r1.  Memory usage seemed to be more stable, until about 8am this morning, after I went to work.  I have free -m running once an hour, dumping to a file, and memory usage just started pegging.  OOM kicked in again, and killed xinit this time.  Going to compile the latest version of X and see if that helps any.  This is really getting frustrating.

 

```
$ cat /proc/sys/vm/overcommit_memory 

```

a value of "2" will essentially disable OOM. If you have plenty of ram or swap and are not doing very memory heavy stuff (like requiring memory>ram+swap), "2" is a good value for normal desktop usage. Don't use it on servers where it is a good idea to let kernel kill processes to ease memory pressure instead of getting itself hung or crash.

I had exactly the same problem as yours: firefox getting killed once in a while, and "2" for /proc/sys/vm/overcommit_memory solved my problem. put it in /etc/sysctl.conf (sys.vm.overcommit_memory = 2) if you want it to be used for good.

----------

## techmav

 *energyman76b wrote:*   

> firefox has several memory leaks - and leaks like mad.
> 
> Just close it, if you don't use it and only open it, if really needed.

 

That still doesn't explain why I never had this issue before I reinstalled gentoo last month.  I was running the same version of firefox then as I am now.  Prior to the reinstall, my system would use swap properly.  Now, as soon as I hit swap, OOM killer starts killing processes, including xinit, wmaker, firefox, and X.  It's not just firefox that's getting killed.

----------

## plastix

 *techmav wrote:*   

>  *energyman76b wrote:*   firefox has several memory leaks - and leaks like mad.
> 
> Just close it, if you don't use it and only open it, if really needed. 
> 
> That still doesn't explain why I never had this issue before I reinstalled gentoo last month.  I was running the same version of firefox then as I am now.  Prior to the reinstall, my system would use swap properly.  Now, as soon as I hit swap, OOM killer starts killing processes, including xinit, wmaker, firefox, and X.  It's not just firefox that's getting killed.

 

This is taken from the 2.6.18.2 changelog : 

 *Quote:*   

> 
> 
> commit 252287f4e8e825fbced96f1a8bc7dc1dfead325c
> 
> Author: Martin Bligh <mbligh@mbligh.org>
> ...

 

----------

## ksp7498

when the system is nearly out of memory open up top and look at which process is taking up so much memory.  It should be fairly obvious what program is causing the problem.

----------

## techmav

There was no clear culprit when I would look to see what program was using the most memory.  There was just a massive memory leak.  I could stop all programs and drop out of X, and still have 450MB in use (not cache, in use in buffers).  Plus, it wasn't the same program that was getting OOM killed each time.

I gave up on trying to figure out which package was causing the problem and reloaded the box.  It's been running fine with no problem since.

----------

## techmav

Found the culprit last night.  Running a 2.6.18-r3 gentoo-sources kernel, and the leak is apparently in ACPI.  I was running wmpower to monitor battery state, and while running slabtop, saw that the size-64 slabs were increasing at a steady rate.  When I started killing processes,  the leak stopped as soon as I killed wmpower.  I tried wmbatteries and wmacpi as replacements, but size-64 slabs would increase steadily anytime any of them were running.  

I'm going to emerge a vanilla 2.6.19 kernel today and see if that helps any.

----------

## devsk

wmpower is user level right? If so, it should be a bug which should be solved by wmpower devs. Does the leak reproduce if you use just acpid?

----------

## techmav

I think it's a leak in the kernel, because I tried three different battery monitors, and all three of them leaked in the same fashion.  I'm now running a vanilla 2.6.19 kernel, and it's been up for 5 days with no problem, with wmpower not running.

```
techmav@carter ~ $ free -m

             total       used       free     shared    buffers     cached

Mem:           503        340        162          0         78        101

-/+ buffers/cache:        161        342

Swap:          509          0        509

```

slabtop output

```
 Active / Total Objects (% used)    : 82418 / 114656 (71.9%)

 Active / Total Slabs (% used)      : 4152 / 4152 (100.0%)

 Active / Total Caches (% used)     : 79 / 128 (61.7%)

 Active / Total Size (% used)       : 13886.50K / 15910.42K (87.3%)

 Minimum / Average / Maximum Object : 0.01K / 0.14K / 128.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME                   

 51336  24452  47%    0.05K    713       72      2852K buffer_head

 17608  17608 100%    0.12K    568       31      2272K dentry_cache

 14175  14164  99%    0.40K   1575        9      6300K ext3_inode_cache

  4116   4071  98%    0.04K     49       84       196K sysfs_dir_cache

  3842   3773  98%    0.03K     34      113       136K size-32

  3640   2615  71%    0.27K    260       14      1040K radix_tree_node

  2346   1990  84%    0.08K     51       46       204K vm_area_struct

  1830   1795  98%    0.12K     61       30       244K size-128

  1650   1020  61%    0.12K     55       30       220K filp

  1534   1474  96%    0.06K     26       59       104K size-64

  1485   1483  99%    0.36K    135       11       540K shmem_inode_cache

  1472   1402  95%    0.04K     16       92        64K Acpi-Operand

  1356   1077  79%    0.01K      4      339        16K anon_vma

  1183   1105  93%    0.02K      7      169        28K Acpi-Namespace

  1106   1068  96%    0.27K     79       14       316K inode_cache

   825    825 100%    0.25K     55       15       220K size-256

```

I'm going to turn wmpower back on and monitor the issue.

----------

