# (Kernel?) memory leak [SOLVED]

## jhoos

Hi,

i have a memory problem on a fresh gentoo install. After installing and booting something starts to eat up my ram and after a while the kernel crashes and the system halts.

When I look at /proc/meminfo I can see that SUnreclaim increases all the time at approximately 2-8 thousand kb per second which takes about 10 to 20 minutes until no memory is left.

It is neither the buffers that takes the ram nor any visible process. The largest process is only about 4 MB:

```
 ps aux  | awk '{print $6/1024 " MB\t\t" $11}'  | sort -nr

4.51953 MB              sshd:

3.75781 MB              bash

3.59766 MB              bash

3.53906 MB              -bash

3.52344 MB              -bash

3.28906 MB              sshd:

...

```

Meminfo shows this and one can see at SUnreclaim that the crash will happen soon.

```
 cat /proc/meminfo                                                                                                                       Sun Jun 19 19:26:11 2016

MemTotal:        4041184 kB

MemFree:           31080 kB

MemAvailable:     483988 kB

Buffers:              60 kB

Cached:           429312 kB

SwapCached:            0 kB

Active:           218264 kB

Inactive:         219124 kB

Active(anon):       3684 kB

Inactive(anon):     4820 kB

Active(file):     214580 kB

Inactive(file):   214304 kB

Unevictable:           0 kB

Mlocked:               0 kB

SwapTotal:       4399100 kB

SwapFree:        4399100 kB

Dirty:              4800 kB

Writeback:             0 kB

AnonPages:          8144 kB

Mapped:             7380 kB

Shmem:               360 kB

Slab:            3525980 kB

SReclaimable:      54108 kB

SUnreclaim:      3471872 kB

KernelStack:        2784 kB

PageTables:         1476 kB

NFS_Unstable:          0 kB

Bounce:                0 kB

WritebackTmp:          0 kB

CommitLimit:     6419692 kB

Committed_AS:      15292 kB

VmallocTotal:   34359738367 kB

VmallocUsed:           0 kB

VmallocChunk:          0 kB

HardwareCorrupted:     0 kB

HugePages_Total:       0

HugePages_Free:        0

HugePages_Rsvd:        0

HugePages_Surp:        0

Hugepagesize:       2048 kB

DirectMap4k:       63040 kB

DirectMap2M:     4130816 kB

```

```
 emerge --info

Portage 2.2.28 (python 2.7.10-final-0, default/linux/amd64/13.0, gcc-4.9.3, glibc-2.22-r4, 4.4.6-gentoo x86_64)

=================================================================

System uname: Linux-4.4.6-gentoo-x86_64-Intel-R-_Atom-TM-_CPU_D510_@_1.66GHz-with-gentoo-2.2

KiB Mem:     4041184 total,   1276836 free

KiB Swap:    4399100 total,   4399100 free

Timestamp of repository gentoo: Sun, 19 Jun 2016 00:45:01 +0000

sh bash 4.3_p42-r1

ld GNU ld (Gentoo 2.25.1 p1.1) 2.25.1

app-shells/bash:          4.3_p42-r1::gentoo

dev-lang/perl:            5.20.2::gentoo

dev-lang/python:          2.7.10-r1::gentoo, 3.4.3-r1::gentoo

dev-util/pkgconfig:       0.28-r2::gentoo

sys-apps/baselayout:      2.2::gentoo

sys-apps/openrc:          0.19.1::gentoo

sys-apps/sandbox:         2.10-r1::gentoo

sys-devel/autoconf:       2.69::gentoo

sys-devel/automake:       1.14.1::gentoo, 1.15::gentoo

sys-devel/binutils:       2.25.1-r1::gentoo

sys-devel/gcc:            4.9.3::gentoo

sys-devel/gcc-config:     1.7.3::gentoo

sys-devel/libtool:        2.4.6::gentoo

sys-devel/make:           4.1-r1::gentoo

sys-kernel/linux-headers: 4.3::gentoo (virtual/os-headers)

sys-libs/glibc:           2.22-r4::gentoo

Repositories:

gentoo

    location: /usr/portage

    sync-type: rsync

    sync-uri: rsync://rsync.gentoo.org/gentoo-portage

    priority: -1000

ACCEPT_KEYWORDS="amd64"

ACCEPT_LICENSE="* -@EULA"

CBUILD="x86_64-pc-linux-gnu"

CFLAGS="-march=native -O2 -pipe"

CHOST="x86_64-pc-linux-gnu"

CONFIG_PROTECT="/etc"

CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/gconf /etc/gentoo-release /etc/sandbox.d /etc/terminfo"

CXXFLAGS="-march=native -O2 -pipe"

DISTDIR="/usr/portage/distfiles"

FCFLAGS="-O2 -pipe"

FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"

FFLAGS="-O2 -pipe"

GENTOO_MIRRORS="http://distfiles.gentoo.org"

LANG="de_DE.utf8"

LDFLAGS="-Wl,-O1 -Wl,--as-needed"

MAKEOPTS="-j1"

PKGDIR="/usr/portage/packages"

PORTAGE_CONFIGROOT="/"

PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"

PORTAGE_TMPDIR="/var/tmp"

USE="acl amd64 berkdb bindist bzip2 cli cracklib crypt cxx dri fortran gdbm iconv ipv6 mmx mmxext modules multilib ncurses nls nptl openmp pam pcre readline seccomp session sse sse2 ssl tcpd unicode xattr zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext sse sse2" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-5" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_4" RUBY_TARGETS="ruby20 ruby21" USERLAND="GNU" VIDEO_CARDS="amdgpu fbdev intel nouveau radeon radeonsi vesa dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"

Unset:  CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON

```

Here is my make.conf:

```
cat /etc/portage/make.conf

# These settings were set by the catalyst build script that automatically

# built this stage.

# Please consult /usr/share/portage/config/make.conf.example for a more

# detailed example.

CFLAGS="-march=native -O2 -pipe"

CXXFLAGS="${CFLAGS}"

MAKEOPTS="-j1"

# WARNING: Changing your CHOST is not something that should be done lightly.

# Please consult http://www.gentoo.org/doc/en/change-chost.xml before changing.

CHOST="x86_64-pc-linux-gnu"

# These are the USE flags that were used in addition to what is provided by the

# profile used for building.

USE="bindist mmx sse sse2"

PORTDIR="/usr/portage"

DISTDIR="${PORTDIR}/distfiles"

PKGDIR="${PORTDIR}/packages"

```

Kernel was compiled during install with genkernel.

How can I find out what causes this? I suppose it to be the kernel or a kernel module. But how do I find out which one and why?Last edited by jhoos on Thu Jun 23, 2016 12:36 pm; edited 1 time in total

----------

## fedeliallalinea

 *jhoos wrote:*   

> How can I find out what causes this? I suppose it to be the kernel or a kernel module. But how do I find out which one and why?

 

slabtop command can help?

----------

## jhoos

Hi, 

slabtop points to kmalloc-64:

```
Objekte aktiv / gesamt (% benutzt) : 62003361 / 62030053 (100,0%)

 Slabs aktiv / gesamt (% benutzt)   : 986142 / 986143 (100,0%)

 Caches aktiv / gesamt (% benutzt)  : 63 / 138 (45,7%)

 GrM-CM-6M-C~_e aktiv / gesamt (% benutzt) : 3872772,21K / 3885016,04K (99,7%)

 Minimum / Average / Maximum Object : 0,02K / 0,06K / 4096,00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME

60774966 60774963  29%    0,06K 964682       63   3858728K kmalloc-64

1064168 1064104  99%    0,03K   8582      124     34328K kmalloc-32

 89992  89945  99%    0,07K   1607       56      6428K ext4_io_end

 39312  21384  54%    0,56K   5616        7     22464K radix_tree_node

 18348  18234  99%    0,12K    556       33      2224K kernfs_node_cache

  6944   6887  99%    0,12K    224       31       896K kmalloc-node

  4368   1577  36%    0,19K    208       21       832K dentry

  3780   3627  95%    0,14K    135       28       540K btrfs_path

  3108    672  21%    0,27K    222       14       888K btrfs_extent_buffer

  2904   2904 100%    0,50K    363        8      1452K kmalloc-512

  2583   2583 100%    0,19K    123       21       492K kmalloc-192

  1960   1870  95%    0,07K     35       56       140K Acpi-Operand

  1850    863  46%    0,08K     37       50       148K btrfs_extent_state

  1817   1769  97%    0,17K     79       23       316K vm_area_struct

  1811   1811 100%    4,00K   1811        1      7244K kmalloc-4096

  1616   1263  78%    0,25K    101       16       404K kmalloc-256

  1089   1051  96%    0,04K     11       99        44K Acpi-Namespace

   980    814  83%    0,95K    245        4       980K btrfs_inode

   966    938  97%    0,59K    161        6       644K shmem_inode_cache

   961    868  90%    0,12K     31       31       124K kmalloc-96

   952    824  86%    0,07K     17       56        68K anon_vma

```

----------

## fedeliallalinea

Try with a newer kernel first.

For deepen analysis would need a tool like this or this but my knowledge ends here.

----------

## jhoos

Switched kernel  from 4.4.6 to 4.6.2 with same result. My self compiled kernel eats my ram. Now playing with kedr. Got it installed and running but yet missing a clear strategy on how to make use of it to locate the cause of my problem. To be continued...

----------

## jhoos

Put away kedr for now as it seems to be more for testing a single modul for memory leaks instead of finding which module causes it.

What I did now is to recomplile the kernel with CONFIG_DEBUG_SLAB and CONFIG_DEBUG_SLAB_LEAK enabled which creates a file /proc/slab_allocators that shows who is consuming the slab objects:

```
cat /proc/slab_allocators|sort -g -k2 -r                                                                                                                                                                                                                                                                                                  

btrfs_free_space: 90999 __load_free_space_cache+0x32c/0x700

btrfs_extent_buffer: 57353 __alloc_extent_buffer+0x2a/0xe0

radix_tree_node: 38393 radix_tree_node_alloc+0x5f/0xb0

kernfs_node_cache: 18935 __kernfs_new_node+0x41/0xc0

selinux_inode_security: 17631 selinux_inode_alloc_security+0x3a/0xa0

dentry: 17630 __d_alloc+0x25/0x180

inode_cache: 14536 alloc_inode+0x51/0x90

btrfs_extent_map: 3582 alloc_extent_map+0x1a/0x60

ftrace_event_field: 3156 __trace_define_field+0x36/0xa0

Acpi-Operand: 1858 acpi_ut_allocate_object_desc_dbg+0x3c/0x68

trace_event_file: 1535 trace_create_new_event+0x20/0x70

btrfs_inode: 1390 btrfs_alloc_inode+0x1f/0x210

Acpi-Namespace: 1029 acpi_ns_create_node+0x37/0x46

shmem_inode_cache: 986 shmem_alloc_inode+0x1a/0x30

btrfs_extent_state: 948 alloc_extent_state+0x21/0xd0

proc_inode_cache: 627 proc_alloc_inode+0x1d/0xb0

anon_vma_chain: 527 anon_vma_clone+0x68/0x1b0

vm_area_struct: 509 mmap_region+0x281/0x580

vm_area_struct: 509 copy_process.part.42+0xb59/0x1990

anon_vma: 415 anon_vma_prepare+0xf2/0x150

selinux_file_security: 348 selinux_file_alloc_security+0x37/0x60

anon_vma_chain: 336 anon_vma_prepare+0x48/0x150

vm_area_struct: 328 __split_vma.isra.43+0x5c/0x1b0

anon_vma: 303 anon_vma_fork+0x65/0x120

idr_layer_cache: 280 ida_pre_get+0x64/0xe0

anon_vma_chain: 276 anon_vma_fork+0x97/0x120

task_delay_info: 197 __delayacct_tsk_init+0x1e/0x40

radix_tree_node: 84 __radix_tree_preload+0x35/0x90

vm_area_struct: 58 do_brk+0x23d/0x2e0

vm_area_struct: 40 __install_special_mapping+0x3c/0x110

idr_layer_cache: 36 idr_layer_alloc+0x2b/0x80

file_lock_ctx: 34 locks_get_lock_context+0x9b/0x110

xfs_ioend: 32 mempool_alloc_slab+0x15/0x20

jfs_mp: 32 mempool_alloc_slab+0x15/0x20

idr_layer_cache: 32 idr_preload+0x63/0xa0

cfq_queue: 31 cfq_get_queue+0x184/0x410

request_queue: 30 blk_alloc_queue_node+0x28/0x2a0

inotify_inode_mark: 30 SyS_inotify_add_watch+0x20a/0x310

cfq_io_cq: 27 ioc_create_icq+0x32/0x150

btrfs_free_space: 27 __btrfs_add_free_space+0x31/0x360

avc_node: 26 avc_alloc_node+0x28/0x140

blkdev_requests: 24 alloc_request_struct+0x17/0x20

buffer_head: 22 alloc_buffer_head+0x22/0x70

vm_area_struct: 20 do_execveat_common.isra.39+0x25e/0x6b0

btrfs_delayed_node: 20 btrfs_get_or_create_delayed_node+0x66/0x1a0

blkdev_ioc: 19 create_task_io_context+0x21/0x100

eventpoll_pwq: 13 ep_ptable_queue_proc+0x2d/0xa0

radix_tree_node: 12 radix_tree_node_alloc+0x3c/0xb0

ip_fib_alias: 10 fib_table_insert+0x180/0x460

```

The elements that constantly grow is btrfs_extent_buffer and radix_tree_node. After some while both values drop a lot (e.g. btrfs_extent_buffer from approx. 60.000 to 6.000) but no memory is freed.

```
btrfs_free_space: 90999 __load_free_space_cache+0x32c/0x700

kernfs_node_cache: 18934 __kernfs_new_node+0x41/0xc0

selinux_inode_security: 15572 selinux_inode_alloc_security+0x3a/0xa0

dentry: 15571 __d_alloc+0x25/0x180

inode_cache: 12484 alloc_inode+0x51/0x90

radix_tree_node: 12236 radix_tree_node_alloc+0x5f/0xb0

btrfs_extent_buffer: 6813 __alloc_extent_buffer+0x2a/0xe0

btrfs_extent_map: 3582 alloc_extent_map+0x1a/0x60
```

So I suspect either btrfs_extent_buffer or radix_tree_node to be the cause. I'll now try a complete reinstall but this time on an ext4 partition and omit the btrfs support in the kernel to see how the system behaves without btrfs support.

----------

## jhoos

Think I can confirm it to btrfs related. Install on ext4 and without btrfs support compiled into the kernel shows system is up and running fine. Free mem stays up.

----------

## fedeliallalinea

 *jhoos wrote:*   

> Think I can confirm it to btrfs related. Install on ext4 and without btrfs support compiled into the kernel shows system is up and running fine. Free mem stays up.

 

Good to know.

In this article suggest to add this command in crontab for free memory

```
echo 2 > /proc/sys/vm/drop_caches
```

----------

## jhoos

I analysed this a little bit further: 

I can compile btrfs into the kernel and can create and mount new btrfs partitions without problems. The problem arises only if i try to mount my main and existing btrfs data volume. As soon as i mount that volume my memory runs down. I assume this volume to be somehow defective and the btrfs kernel to have a memory leak in case it shall work on this defective volume.

This gives me some doubts about the stability of BTRFS but i'll give it another try and start with a whole new btrfs partion and restore my data from backup.

----------

