# [Solved, at last!] swapper: page allocation failure...

## cgmd

Hi, all...

I'm not totally sure if this is a hardware or networking problem, but this is what I'm experiencing:

I have a small network, consisting of a couple gentoo machines, a windows xp machine, and an additional gentoo machine which is my "dataserver", for backups and stuff like that.

For about 6 months, I have been backing up my windows xp machine across network onto dataserver, using Retrospect backup software on the windows machine.  The last couple days, the dataserver has started dropping off the network and changing out of user mode to default prompt, during the running of Retrospect, thus killing the uncompleted backup.

My dataserver system log looks like this during the time it is quitting the network and killing the backup:

```

# cat /var/log/messages

Feb  8 17:27:36 dataserver smbd: page allocation failure. order:0, mode:0x20

Feb  8 17:27:37 dataserver [<c014b999>]  [<c015f70e>]  [<c03d023e>]  [<c015f95e>]  [<c03cf0ca>]  [<c03f9f29>]  [<c03f78ea>]  [<c03f8e4f>]  [<c03fe52e>]  [<c03cc0b3>]  [<c03f044e>]  [<c03f2262>]  [<c03cb1f9>]  [<c03c926b>]  [<c0149075>]  [<c016263b>]  [<c0130d56>]  [<c010389a>]  [<c018d4d6>]  [<c0162f07>]  [<c01633b7>]  [<c0102ea9>]  =======================

Feb  8 17:27:37 dataserver Mem-info:

Feb  8 17:27:37 dataserver DMA per-cpu:

Feb  8 17:27:37 dataserver CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0

Feb  8 17:27:37 dataserver Normal per-cpu:

Feb  8 17:27:37 dataserver CPU    0: Hot: hi:  186, btch:  31 usd:  30   Cold: hi:   62, btch:  15 usd:  48

Feb  8 17:27:37 dataserver HighMem per-cpu:

Feb  8 17:27:37 dataserver CPU    0: Hot: hi:   42, btch:   7 usd:   2   Cold: hi:   14, btch:   3 usd:  13

Feb  8 17:27:37 dataserver Active:6436 inactive:44899 dirty:26644 writeback:0 unstable:0 free:1498 slab:204601 mapped:1751 pagetables:96

Feb  8 17:27:37 dataserver DMA free:3520kB min:68kB low:84kB high:100kB active:0kB inactive:3888kB present:16256kB pages_scanned:448 all_unreclaimable? no

Feb  8 17:27:37 dataserver lowmem_reserve[]: 0 873 999

Feb  8 17:27:37 dataserver Normal free:1372kB min:3744kB low:4680kB high:5616kB active:2712kB inactive:70336kB present:894080kB pages_scanned:480 all_unreclaimable? no

Feb  8 17:27:37 dataserver lowmem_reserve[]: 0 0 1013

Feb  8 17:27:37 dataserver HighMem free:1100kB min:128kB low:260kB high:396kB active:23032kB inactive:105372kB present:129732kB pages_scanned:0 all_unreclaimable? no

Feb  8 17:27:37 dataserver lowmem_reserve[]: 0 0 0

Feb  8 17:27:37 dataserver DMA: 0*4kB 0*8kB 0*16kB 0*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3520kB

Feb  8 17:27:37 dataserver Normal: 1*4kB 1*8kB 1*16kB 0*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1372kB

Feb  8 17:27:37 dataserver HighMem: 1*4kB 1*8kB 0*16kB 0*32kB 3*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1100kB

Feb  8 17:27:37 dataserver Swap cache: add 0, delete 0, find 0/0, race 0+0

Feb  8 17:27:37 dataserver Free swap  = 987988kB

Feb  8 17:27:37 dataserver Total swap = 987988kB

Feb  8 17:27:37 dataserver Free swap:       987988kB

Feb  8 17:27:37 dataserver 262064 pages of RAM

Feb  8 17:27:37 dataserver 32688 pages of HIGHMEM

Feb  8 17:27:37 dataserver 3546 reserved pages

Feb  8 17:27:37 dataserver 51387 pages shared

Feb  8 17:27:37 dataserver 0 pages swap cached

Feb  8 17:27:37 dataserver 26644 pages dirty

Feb  8 17:27:37 dataserver 0 pages writeback

Feb  8 17:27:37 dataserver 1751 pages mapped

Feb  8 17:27:37 dataserver 204601 pages slab

Feb  8 17:27:37 dataserver 96 pages pagetables

Feb  8 17:27:37 dataserver smbd: page allocation failure. order:0, mode:0x20

Feb  8 17:27:37 dataserver [<c014b999>]  [<c015f70e>]  [<c015f95e>]  [<c03cf0ca>]  [<c030050a>]  [<c0143cd5>]  [<c0144f64>]  [<c0144edb>]  [<c01057a6>]  [<c010389a>]  [<c015f960>]  [<c03cf0ca>]  [<c03f9f29>]  [<c03f78ea>]  [<c03f8e4f>]  [<c03fe52e>]  [<c03cc0b3>]  [<c03f044e>]  [<c03f2262>]  [<c03cb1f9>]  [<c03c926b>]  [<c0149075>]  [<c016263b>]  [<c0130d56>]  [<c010389a>]  [<c018d4d6>]  [<c0162f07>]  [<c01633b7>]  [<c0102ea9>]  =======================

Feb  8 17:27:37 dataserver Mem-info:

Feb  8 17:27:37 dataserver DMA per-cpu:

Feb  8 17:27:37 dataserver CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0

Feb  8 17:27:37 dataserver Normal per-cpu:

Feb  8 17:27:37 dataserver CPU    0: Hot: hi:  186, btch:  31 usd:  30   Cold: hi:   62, btch:  15 usd:  48

Feb  8 17:27:37 dataserver HighMem per-cpu:

Feb  8 17:27:37 dataserver CPU    0: Hot: hi:   42, btch:   7 usd:   2   Cold: hi:   14, btch:   3 usd:  13

Feb  8 17:27:37 dataserver Active:6436 inactive:44899 dirty:26644 writeback:0 unstable:0 free:1498 slab:204601 mapped:1751 pagetables:96

Feb  8 17:27:37 dataserver DMA free:3520kB min:68kB low:84kB high:100kB active:0kB inactive:3888kB present:16256kB pages_scanned:448 all_unreclaimable? no

Feb  8 17:27:37 dataserver lowmem_reserve[]: 0 873 999

Feb  8 17:27:37 dataserver Normal free:1372kB min:3744kB low:4680kB high:5616kB active:2712kB inactive:70336kB present:894080kB pages_scanned:480 all_unreclaimable? no

Feb  8 17:27:37 dataserver lowmem_reserve[]: 0 0 1013

Feb  8 17:27:37 dataserver HighMem free:1100kB min:128kB low:260kB high:396kB active:23032kB inactive:105372kB present:129732kB pages_scanned:0 all_unreclaimable? no

Feb  8 17:27:37 dataserver lowmem_reserve[]: 0 0 0

Feb  8 17:27:37 dataserver DMA: 0*4kB 0*8kB 0*16kB 0*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3520kB

Feb  8 17:27:37 dataserver Normal: 1*4kB 1*8kB 1*16kB 0*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1372kB

Feb  8 17:27:37 dataserver HighMem: 1*4kB 1*8kB 0*16kB 0*32kB 3*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1100kB

Feb  8 17:27:37 dataserver Swap cache: add 0, delete 0, find 0/0, race 0+0

Feb  8 17:27:37 dataserver Free swap  = 987988kB

Feb  8 17:27:37 dataserver Total swap = 987988kB

Feb  8 17:27:37 dataserver Free swap:       987988kB

Feb  8 17:27:37 dataserver 262064 pages of RAM

Feb  8 17:27:37 dataserver 32688 pages of HIGHMEM

Feb  8 17:27:37 dataserver 3546 reserved pages

Feb  8 17:27:37 dataserver 51387 pages shared

Feb  8 17:27:37 dataserver 0 pages swap cached

Feb  8 17:27:37 dataserver 26644 pages dirty

Feb  8 17:27:37 dataserver 0 pages writeback

Feb  8 17:27:37 dataserver 1751 pages mapped

Feb  8 17:27:37 dataserver 204601 pages slab

Feb  8 17:27:37 dataserver 96 pages pagetables

Feb  8 17:27:37 dataserver smbd: page allocation failure. order:0, mode:0x20

Feb  8 17:27:37 dataserver [<c014b999>]  [<c015f70e>]  [<c015f95e>]  [<c03cf0ca>]  [<c0300622>]  [<c0143cd5>]  [<c0144f64>]  [<c0144edb>]  [<c01057a6>]  [<c010389a>]  [<c015f960>]  [<c03cf0ca>]  [<c03f9f29>]  [<c03f78ea>]  [<c03f8e4f>]  [<c03fe52e>]  [<c03cc0b3>]  [<c03f044e>]  [<c03f2262>]  [<c03cb1f9>]  [<c03c926b>]  [<c0149075>]  [<c016263b>]  [<c0130d56>]  [<c010389a>]  [<c018d4d6>]  [<c0162f07>]  [<c01633b7>]  [<c0102ea9>]  =======================

Feb  8 17:27:37 dataserver Mem-info:

Feb  8 17:27:37 dataserver DMA per-cpu:

Feb  8 17:27:37 dataserver CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0

Feb  8 17:27:37 dataserver Normal per-cpu:

Feb  8 17:27:37 dataserver CPU    0: Hot: hi:  186, btch:  31 usd:  30   Cold: hi:   62, btch:  15 usd:  48

Feb  8 17:27:37 dataserver HighMem per-cpu:

Feb  8 17:27:37 dataserver CPU    0: Hot: hi:   42, btch:   7 usd:   2   Cold: hi:   14, btch:   3 usd:  13

Feb  8 17:27:37 dataserver Active:6436 inactive:44899 dirty:26644 writeback:0 unstable:0 free:1498 slab:204601 mapped:1751 pagetables:96

Feb  8 17:27:37 dataserver DMA free:3520kB min:68kB low:84kB high:100kB active:0kB inactive:3888kB present:16256kB pages_scanned:448 all_unreclaimable? no

Feb  8 17:27:37 dataserver lowmem_reserve[]: 0 873 999

Feb  8 17:27:37 dataserver Normal free:1372kB min:3744kB low:4680kB high:5616kB active:2712kB inactive:70336kB present:894080kB pages_scanned:480 all_unreclaimable? no

Feb  8 17:27:37 dataserver lowmem_reserve[]: 0 0 1013

Feb  8 17:27:37 dataserver HighMem free:1100kB min:128kB low:260kB high:396kB active:23032kB inactive:105372kB present:129732kB pages_scanned:0 all_unreclaimable? no

Feb  8 17:27:37 dataserver lowmem_reserve[]: 0 0 0

Feb  8 17:27:37 dataserver DMA: 0*4kB 0*8kB 0*16kB 0*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3520kB

Feb  8 17:27:37 dataserver Normal: 1*4kB 1*8kB 1*16kB 0*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1372kB

Feb  8 17:27:37 dataserver HighMem: 1*4kB 1*8kB 0*16kB 0*32kB 3*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1100kB

Feb  8 17:27:37 dataserver Swap cache: add 0, delete 0, find 0/0, race 0+0

Feb  8 17:27:37 dataserver Free swap  = 987988kB

Feb  8 17:27:37 dataserver Total swap = 987988kB

Feb  8 17:27:37 dataserver Free swap:       987988kB

Feb  8 17:27:37 dataserver 262064 pages of RAM

Feb  8 17:27:37 dataserver 32688 pages of HIGHMEM

Feb  8 17:27:37 dataserver 3546 reserved pages

Feb  8 17:27:37 dataserver 51387 pages shared

Feb  8 17:27:37 dataserver 0 pages swap cached

Feb  8 17:27:37 dataserver 26644 pages dirty

Feb  8 17:27:37 dataserver 0 pages writeback

Feb  8 17:27:37 dataserver 1751 pages mapped

Feb  8 17:27:37 dataserver 204601 pages slab

Feb  8 17:27:37 dataserver 96 pages pagetables

Feb  8 17:27:37 dataserver eth0: Memory squeeze,deferring packet.

Feb  8 17:27:37 dataserver smbd: page allocation failure. order:0, mode:0x20

Feb  8 17:27:37 dataserver [<c014b999>]  [<c015f70e>]  [<c015f95e>]  [<c03cf0ca>]  [<c03f9f29>]  [<c03f214a>]  [<c03cb1f9>]  [<c03c926b>]  [<c0149075>]  [<c016263b>]  [<c0130d56>]  [<c010389a>]  [<c018d4d6>]  [<c0162f07>]  [<c01633b7>]  [<c0102ea9>]  =======================

Feb  8 17:27:37 dataserver Mem-info:

Feb  8 17:27:37 dataserver DMA per-cpu:

Feb  8 17:27:37 dataserver CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0

Feb  8 17:27:37 dataserver Normal per-cpu:

Feb  8 17:27:37 dataserver CPU    0: Hot: hi:  186, btch:  31 usd:  30   Cold: hi:   62, btch:  15 usd:  48

Feb  8 17:27:37 dataserver HighMem per-cpu:

Feb  8 17:27:37 dataserver CPU    0: Hot: hi:   42, btch:   7 usd:   2   Cold: hi:   14, btch:   3 usd:  13

Feb  8 17:27:37 dataserver Active:6436 inactive:44899 dirty:26644 writeback:0 unstable:0 free:1498 slab:204601 mapped:1751 pagetables:96

Feb  8 17:27:37 dataserver DMA free:3520kB min:68kB low:84kB high:100kB active:0kB inactive:3888kB present:16256kB pages_scanned:448 all_unreclaimable? no

Feb  8 17:27:37 dataserver lowmem_reserve[]: 0 873 999

Feb  8 17:27:37 dataserver Normal free:1372kB min:3744kB low:4680kB high:5616kB active:2712kB inactive:70336kB present:894080kB pages_scanned:480 all_unreclaimable? no

Feb  8 17:27:37 dataserver lowmem_reserve[]: 0 0 1013

Feb  8 17:27:37 dataserver HighMem free:1100kB min:128kB low:260kB high:396kB active:23032kB inactive:105372kB present:129732kB pages_scanned:0 all_unreclaimable? no

Feb  8 17:27:37 dataserver lowmem_reserve[]: 0 0 0

Feb  8 17:27:37 dataserver DMA: 0*4kB 0*8kB 0*16kB 0*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3520kB

Feb  8 17:27:37 dataserver Normal: 1*4kB 1*8kB 1*16kB 0*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1372kB

Feb  8 17:27:37 dataserver HighMem: 1*4kB 1*8kB 0*16kB 0*32kB 3*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1100kB

Feb  8 17:27:37 dataserver Swap cache: add 0, delete 0, find 0/0, race 0+0

Feb  8 17:27:37 dataserver Free swap  = 987988kB

Feb  8 17:27:37 dataserver Total swap = 987988kB

Feb  8 17:27:37 dataserver Free swap:       987988kB

Feb  8 17:27:37 dataserver 262064 pages of RAM

Feb  8 17:27:37 dataserver 32688 pages of HIGHMEM

Feb  8 17:27:37 dataserver 3546 reserved pages

Feb  8 17:27:37 dataserver 51387 pages shared

Feb  8 17:27:37 dataserver 0 pages swap cached

Feb  8 17:27:37 dataserver 26644 pages dirty

Feb  8 17:27:37 dataserver 0 pages writeback

Feb  8 17:27:37 dataserver 1751 pages mapped

Feb  8 17:27:37 dataserver 204601 pages slab

Feb  8 17:27:37 dataserver 96 pages pagetables

Feb  8 17:27:37 dataserver smbd: page allocation failure. order:0, mode:0x20

Feb  8 17:27:37 dataserver [<c014b999>]  [<c015f70e>]  [<c03d023e>]  [<c015f95e>]  [<c03cf0ca>]  [<c03f9f29>]  [<c03f78ea>]  [<c03f8e4f>]  [<c03fe52e>]  [<c03cb9e7>]  [<c03cb8c3>]  [<c03f21f3>]  [<c03cb1f9>]  [<c03c926b>]  [<c0149075>]  [<c016263b>]  [<c0130d56>]  [<c010389a>]  [<c018d4d6>]  [<c0162f07>]  [<c01633b7>]  [<c0102ea9>]  =======================

Feb  8 17:27:37 dataserver Mem-info:

Feb  8 17:27:37 dataserver DMA per-cpu:

Feb  8 17:27:37 dataserver CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0

Feb  8 17:27:37 dataserver Normal per-cpu:

Feb  8 17:27:37 dataserver CPU    0: Hot: hi:  186, btch:  31 usd:  30   Cold: hi:   62, btch:  15 usd:  48

Feb  8 17:27:37 dataserver HighMem per-cpu:

Feb  8 17:27:37 dataserver CPU    0: Hot: hi:   42, btch:   7 usd:   2   Cold: hi:   14, btch:   3 usd:  13

Feb  8 17:27:37 dataserver Active:6436 inactive:44899 dirty:26644 writeback:0 unstable:0 free:1498 slab:204601 mapped:1751 pagetables:96

Feb  8 17:27:37 dataserver DMA free:3520kB min:68kB low:84kB high:100kB active:0kB inactive:3888kB present:16256kB pages_scanned:448 all_unreclaimable? no

Feb  8 17:27:37 dataserver lowmem_reserve[]: 0 873 999

Feb  8 17:27:37 dataserver Normal free:1372kB min:3744kB low:4680kB high:5616kB active:2712kB inactive:70336kB present:894080kB pages_scanned:480 all_unreclaimable? no

Feb  8 17:27:37 dataserver lowmem_reserve[]: 0 0 1013

Feb  8 17:27:37 dataserver HighMem free:1100kB min:128kB low:260kB high:396kB active:23032kB inactive:105372kB present:129732kB pages_scanned:0 all_unreclaimable? no

Feb  8 17:27:37 dataserver lowmem_reserve[]: 0 0 0

Feb  8 17:27:37 dataserver DMA: 0*4kB 0*8kB 0*16kB 0*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3520kB

Feb  8 17:27:37 dataserver Normal: 1*4kB 1*8kB 1*16kB 0*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1372kB

Feb  8 17:27:37 dataserver HighMem: 1*4kB 1*8kB 0*16kB 0*32kB 3*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1100kB

Feb  8 17:27:37 dataserver Swap cache: add 0, delete 0, find 0/0, race 0+0

Feb  8 17:27:37 dataserver Free swap  = 987988kB

Feb  8 17:27:37 dataserver Total swap = 987988kB

Feb  8 17:27:37 dataserver Free swap:       987988kB

Feb  8 17:27:37 dataserver 262064 pages of RAM

Feb  8 17:27:37 dataserver 32688 pages of HIGHMEM

Feb  8 17:27:37 dataserver 3546 reserved pages

Feb  8 17:27:37 dataserver 51387 pages shared

Feb  8 17:27:37 dataserver 0 pages swap cached

Feb  8 17:27:37 dataserver 26644 pages dirty

Feb  8 17:27:37 dataserver 0 pages writeback

Feb  8 17:27:37 dataserver 1751 pages mapped

Feb  8 17:27:37 dataserver 204601 pages slab

Feb  8 17:27:37 dataserver 96 pages pagetables

Feb  8 17:27:37 dataserver smbd: page allocation failure. order:0, mode:0x20

Feb  8 17:27:37 dataserver [<c014b999>]  [<c015f70e>]  [<c03ce969>]  [<c015f95e>]  [<c03cf0ca>]  [<c03f9f29>]  [<c03f2511>]  [<c03cb1f9>]  [<c03c926b>]  [<c0149075>]  [<c016263b>]  [<c0130d56>]  [<c010389a>]  [<c018d4d6>]  [<c0162f07>]  [<c01633b7>]  [<c0102ea9>]  =======================

Feb  8 17:27:37 dataserver Mem-info:

Feb  8 17:27:37 dataserver DMA per-cpu:

Feb  8 17:27:37 dataserver CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0

Feb  8 17:27:37 dataserver Normal per-cpu:

Feb  8 17:27:37 dataserver CPU    0: Hot: hi:  186, btch:  31 usd:  30   Cold: hi:   62, btch:  15 usd:  48

Feb  8 17:27:37 dataserver HighMem per-cpu:

Feb  8 17:27:37 dataserver CPU    0: Hot: hi:   42, btch:   7 usd:   2   Cold: hi:   14, btch:   3 usd:  13

Feb  8 17:27:37 dataserver Active:6436 inactive:44899 dirty:26644 writeback:0 unstable:0 free:1498 slab:204601 mapped:1751 pagetables:96

Feb  8 17:27:37 dataserver DMA free:3520kB min:68kB low:84kB high:100kB active:0kB inactive:3888kB present:16256kB pages_scanned:448 all_unreclaimable? no

Feb  8 17:27:37 dataserver lowmem_reserve[]: 0 873 999

Feb  8 17:27:37 dataserver Normal free:1372kB min:3744kB low:4680kB high:5616kB active:2712kB inactive:70336kB present:894080kB pages_scanned:480 all_unreclaimable? no

Feb  8 17:27:37 dataserver lowmem_reserve[]: 0 0 1013

Feb  8 17:27:37 dataserver HighMem free:1100kB min:128kB low:260kB high:396kB active:23032kB inactive:105372kB present:129732kB pages_scanned:0 all_unreclaimable? no

Feb  8 17:27:37 dataserver lowmem_reserve[]: 0 0 0

Feb  8 17:27:37 dataserver DMA: 0*4kB 0*8kB 0*16kB 0*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3520kB

Feb  8 17:27:37 dataserver Normal: 1*4kB 1*8kB 1*16kB 0*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1372kB

Feb  8 17:27:37 dataserver HighMem: 1*4kB 1*8kB 0*16kB 0*32kB 3*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1100kB

Feb  8 17:27:37 dataserver Swap cache: add 0, delete 0, find 0/0, race 0+0

Feb  8 17:27:37 dataserver Free swap  = 987988kB

Feb  8 17:27:37 dataserver Total swap = 987988kB

Feb  8 17:27:37 dataserver Free swap:       987988kB

Feb  8 17:27:37 dataserver 262064 pages of RAM

Feb  8 17:27:37 dataserver 32688 pages of HIGHMEM

Feb  8 17:27:37 dataserver 3546 reserved pages

Feb  8 17:27:37 dataserver 51387 pages shared

Feb  8 17:27:37 dataserver 0 pages swap cached

Feb  8 17:27:37 dataserver 26644 pages dirty

Feb  8 17:27:37 dataserver 0 pages writeback

Feb  8 17:27:37 dataserver 1751 pages mapped

Feb  8 17:27:37 dataserver 204601 pages slab

Feb  8 17:27:37 dataserver 96 pages pagetables

Feb  8 17:27:37 dataserver smbd: page allocation failure. order:0, mode:0x20

Feb  8 17:27:37 dataserver [<c014b999>]  [<c015f70e>]  [<c0112918>]  [<c015f95e>]  [<c03cf0ca>]  [<c03f9f29>]  [<c03f214a>]  [<c03cb1f9>]  [<c03c926b>]  [<c0149075>]  [<c016263b>]  [<c0130d56>]  [<c04610ca>]  [<c0102f7e>]  [<c018d4d6>]  [<c0162f07>]  [<c01633b7>]  [<c0102ea9>]  =======================

Feb  8 17:27:37 dataserver Mem-info:

Feb  8 17:27:37 dataserver DMA per-cpu:

Feb  8 17:27:37 dataserver CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0

Feb  8 17:27:37 dataserver Normal per-cpu:

Feb  8 17:27:37 dataserver CPU    0: Hot: hi:  186, btch:  31 usd:  30   Cold: hi:   62, btch:  15 usd:  48

Feb  8 17:27:37 dataserver HighMem per-cpu:

Feb  8 17:27:37 dataserver CPU    0: Hot: hi:   42, btch:   7 usd:   2   Cold: hi:   14, btch:   3 usd:  13

Feb  8 17:27:37 dataserver Active:6437 inactive:44898 dirty:26644 writeback:0 unstable:0 free:1498 slab:204601 mapped:1751 pagetables:96

Feb  8 17:27:37 dataserver DMA free:3520kB min:68kB low:84kB high:100kB active:0kB inactive:3888kB present:16256kB pages_scanned:448 all_unreclaimable? no

Feb  8 17:27:37 dataserver lowmem_reserve[]: 0 873 999

Feb  8 17:27:37 dataserver Normal free:1372kB min:3744kB low:4680kB high:5616kB active:2716kB inactive:70332kB present:894080kB pages_scanned:576 all_unreclaimable? no

Feb  8 17:27:37 dataserver lowmem_reserve[]: 0 0 1013

Feb  8 17:27:37 dataserver HighMem free:1100kB min:128kB low:260kB high:396kB active:23032kB inactive:105372kB present:129732kB pages_scanned:0 all_unreclaimable? no

Feb  8 17:27:37 dataserver lowmem_reserve[]: 0 0 0

Feb  8 17:27:37 dataserver DMA: 0*4kB 0*8kB 0*16kB 0*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3520kB

Feb  8 17:27:37 dataserver Normal: 1*4kB 1*8kB 1*16kB 0*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1372kB

Feb  8 17:27:37 dataserver HighMem: 1*4kB 1*8kB 0*16kB 0*32kB 3*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1100kB

Feb  8 17:27:37 dataserver Swap cache: add 0, delete 0, find 0/0, race 0+0

Feb  8 17:27:37 dataserver Free swap  = 987988kB

Feb  8 17:27:37 dataserver Total swap = 987988kB

Feb  8 17:27:37 dataserver Free swap:       987988kB

Feb  8 17:27:37 dataserver 262064 pages of RAM

Feb  8 17:27:37 dataserver 32688 pages of HIGHMEM

Feb  8 17:27:37 dataserver 3546 reserved pages

Feb  8 17:27:37 dataserver 51387 pages shared

Feb  8 17:27:37 dataserver 0 pages swap cached

Feb  8 17:27:37 dataserver 26644 pages dirty

Feb  8 17:27:37 dataserver 0 pages writeback

Feb  8 17:27:37 dataserver 1751 pages mapped

Feb  8 17:27:37 dataserver 204601 pages slab

Feb  8 17:27:37 dataserver 96 pages pagetables

Feb  8 17:27:37 dataserver swapper: page allocation failure. order:0, mode:0x20

Feb  8 17:27:37 dataserver [<c014b999>]  [<c015f70e>]  [<c0112918>]  [<c015f95e>]  [<c03cf0ca>]  [<c030050a>]  [<c0143cd5>]  [<c0144f64>]  [<c0144edb>]  [<c01057a6>]  [<c010389a>]  [<c0101c45>]  [<c0101c7c>]  [<c0101d45>]  [<c054b7b2>]  [<c054b1e0>]  =======================

Feb  8 17:27:37 dataserver Mem-info:

Feb  8 17:27:37 dataserver DMA per-cpu:

Feb  8 17:27:37 dataserver CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0

Feb  8 17:27:37 dataserver Normal per-cpu:

Feb  8 17:27:37 dataserver CPU    0: Hot: hi:  186, btch:  31 usd:  30   Cold: hi:   62, btch:  15 usd:  48

Feb  8 17:27:37 dataserver HighMem per-cpu:

Feb  8 17:27:37 dataserver CPU    0: Hot: hi:   42, btch:   7 usd:  40   Cold: hi:   14, btch:   3 usd:  13

Feb  8 17:27:37 dataserver Active:6271 inactive:44907 dirty:26488 writeback:0 unstable:0 free:1617 slab:204601 mapped:1751 pagetables:96

Feb  8 17:27:37 dataserver DMA free:3520kB min:68kB low:84kB high:100kB active:0kB inactive:3888kB present:16256kB pages_scanned:480 all_unreclaimable? no

Feb  8 17:27:37 dataserver lowmem_reserve[]: 0 873 999

Feb  8 17:27:37 dataserver Normal free:1372kB min:3744kB low:4680kB high:5616kB active:2568kB inactive:70480kB present:894080kB pages_scanned:1253 all_unreclaimable? no

Feb  8 17:27:37 dataserver lowmem_reserve[]: 0 0 1013

Feb  8 17:27:37 dataserver HighMem free:1576kB min:128kB low:260kB high:396kB active:22516kB inactive:105260kB present:129732kB pages_scanned:0 all_unreclaimable? no

Feb  8 17:27:37 dataserver lowmem_reserve[]: 0 0 0

Feb  8 17:27:37 dataserver DMA: 0*4kB 0*8kB 0*16kB 0*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3520kB

Feb  8 17:27:37 dataserver Normal: 1*4kB 1*8kB 1*16kB 0*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1372kB

Feb  8 17:27:37 dataserver HighMem: 80*4kB 15*8kB 3*16kB 0*32kB 3*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1576kB

Feb  8 17:27:37 dataserver Swap cache: add 0, delete 0, find 0/0, race 0+0

Feb  8 17:27:37 dataserver Free swap  = 987988kB

Feb  8 17:27:37 dataserver Total swap = 987988kB

Feb  8 17:27:37 dataserver Free swap:       987988kB

Feb  8 17:27:37 dataserver 262064 pages of RAM

Feb  8 17:27:37 dataserver 32688 pages of HIGHMEM

Feb  8 17:27:37 dataserver 3546 reserved pages

Feb  8 17:27:37 dataserver 51395 pages shared

Feb  8 17:27:37 dataserver 0 pages swap cached

Feb  8 17:27:37 dataserver 26488 pages dirty

Feb  8 17:27:37 dataserver 0 pages writeback

Feb  8 17:27:37 dataserver 1751 pages mapped

Feb  8 17:27:37 dataserver 204601 pages slab

Feb  8 17:27:37 dataserver 96 pages pagetables

Feb  8 17:27:37 dataserver swapper: page allocation failure. order:0, mode:0x20

Feb  8 17:27:37 dataserver [<c014b999>]  [<c015f70e>]  [<c0112918>]  [<c015f95e>]  [<c03cf0ca>]  [<c0300622>]  [<c0143cd5>]  [<c0144f64>]  [<c0144edb>]  [<c01057a6>]  [<c010389a>]  [<c0101c45>]  [<c0101c7c>]  [<c0101d45>]  [<c054b7b2>]  [<c054b1e0>]  =======================

Feb  8 17:27:37 dataserver Mem-info:

Feb  8 17:27:37 dataserver DMA per-cpu:

Feb  8 17:27:37 dataserver CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0

Feb  8 17:27:37 dataserver Normal per-cpu:

Feb  8 17:27:37 dataserver CPU    0: Hot: hi:  186, btch:  31 usd:  30   Cold: hi:   62, btch:  15 usd:  48

Feb  8 17:27:37 dataserver HighMem per-cpu:

Feb  8 17:27:37 dataserver CPU    0: Hot: hi:   42, btch:   7 usd:  40   Cold: hi:   14, btch:   3 usd:  13

Feb  8 17:27:37 dataserver Active:6271 inactive:44907 dirty:26488 writeback:0 unstable:0 free:1617 slab:204601 mapped:1751 pagetables:96

Feb  8 17:27:37 dataserver DMA free:3520kB min:68kB low:84kB high:100kB active:0kB inactive:3888kB present:16256kB pages_scanned:480 all_unreclaimable? no

Feb  8 17:27:37 dataserver lowmem_reserve[]: 0 873 999

Feb  8 17:27:37 dataserver Normal free:1372kB min:3744kB low:4680kB high:5616kB active:2568kB inactive:70480kB present:894080kB pages_scanned:1253 all_unreclaimable? no

Feb  8 17:27:37 dataserver lowmem_reserve[]: 0 0 1013

Feb  8 17:27:37 dataserver HighMem free:1576kB min:128kB low:260kB high:396kB active:22516kB inactive:105260kB present:129732kB pages_scanned:0 all_unreclaimable? no

Feb  8 17:27:37 dataserver lowmem_reserve[]: 0 0 0

Feb  8 17:27:37 dataserver DMA: 0*4kB 0*8kB 0*16kB 0*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3520kB

Feb  8 17:27:37 dataserver Normal: 1*4kB 1*8kB 1*16kB 0*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1372kB

Feb  8 17:27:37 dataserver HighMem: 80*4kB 15*8kB 3*16kB 0*32kB 3*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1576kB

Feb  8 17:27:37 dataserver Swap cache: add 0, delete 0, find 0/0, race 0+0

Feb  8 17:27:37 dataserver Free swap  = 987988kB

Feb  8 17:27:37 dataserver Total swap = 987988kB

Feb  8 17:27:37 dataserver Free swap:       987988kB

Feb  8 17:27:37 dataserver 262064 pages of RAM

Feb  8 17:27:37 dataserver 32688 pages of HIGHMEM

Feb  8 17:27:37 dataserver 3546 reserved pages

Feb  8 17:27:37 dataserver 51395 pages shared

Feb  8 17:27:37 dataserver 0 pages swap cached

Feb  8 17:27:37 dataserver 26488 pages dirty

Feb  8 17:27:37 dataserver 0 pages writeback

Feb  8 17:27:37 dataserver 1751 pages mapped

Feb  8 17:27:37 dataserver 204601 pages slab

Feb  8 17:27:37 dataserver 96 pages pagetables

Feb  8 17:27:37 dataserver eth0: Memory squeeze,deferring packet.

Feb  8 17:27:37 dataserver swapper: page allocation failure. order:0, mode:0x20

Feb  8 17:27:37 dataserver [<c014b999>]  [<c015f70e>]  [<c01290c0>]  [<c015f95e>]  [<c03cf0ca>]  [<c03fcdc7>]  [<c03f9f29>]  [<c03fcf35>]  [<c0127cb4>]  [<c0123c2c>]  [<c01056c5>]  [<c0101d5d>]  [<c0144edb>]  [<c01057b9>]  [<c010389a>]  [<c0101c45>]  [<c0101c7c>]  [<c0101d45>]  [<c054b7b2>]  [<c054b1e0>]  =======================

Feb  8 17:27:37 dataserver Mem-info:

Feb  8 17:27:37 dataserver DMA per-cpu:

Feb  8 17:27:37 dataserver CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0

Feb  8 17:27:37 dataserver Normal per-cpu:

Feb  8 17:27:37 dataserver CPU    0: Hot: hi:  186, btch:  31 usd:  30   Cold: hi:   62, btch:  15 usd:  48

Feb  8 17:27:37 dataserver HighMem per-cpu:

Feb  8 17:27:37 dataserver CPU    0: Hot: hi:   42, btch:   7 usd:  40   Cold: hi:   14, btch:   3 usd:  13

Feb  8 17:27:37 dataserver Active:6271 inactive:44907 dirty:26488 writeback:0 unstable:0 free:1617 slab:204601 mapped:1751 pagetables:96

Feb  8 17:27:37 dataserver DMA free:3520kB min:68kB low:84kB high:100kB active:0kB inactive:3888kB present:16256kB pages_scanned:480 all_unreclaimable? no

Feb  8 17:27:37 dataserver lowmem_reserve[]: 0 873 999

Feb  8 17:27:37 dataserver Normal free:1372kB min:3744kB low:4680kB high:5616kB active:2568kB inactive:70480kB present:894080kB pages_scanned:1253 all_unreclaimable? no

Feb  8 17:27:37 dataserver lowmem_reserve[]: 0 0 1013

Feb  8 17:27:37 dataserver HighMem free:1576kB min:128kB low:260kB high:396kB active:22516kB inactive:105260kB present:129732kB pages_scanned:0 all_unreclaimable? no

Feb  8 17:27:37 dataserver lowmem_reserve[]: 0 0 0

Feb  8 17:27:37 dataserver DMA: 0*4kB 0*8kB 0*16kB 0*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3520kB

Feb  8 17:27:37 dataserver Normal: 1*4kB 1*8kB 1*16kB 0*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1372kB

Feb  8 17:27:37 dataserver HighMem: 80*4kB 15*8kB 3*16kB 0*32kB 3*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1576kB

Feb  8 17:27:37 dataserver Swap cache: add 0, delete 0, find 0/0, race 0+0

Feb  8 17:27:37 dataserver Free swap  = 987988kB

Feb  8 17:27:37 dataserver Total swap = 987988kB

Feb  8 17:27:37 dataserver Free swap:       987988kB

Feb  8 17:27:37 dataserver 262064 pages of RAM

Feb  8 17:27:37 dataserver 32688 pages of HIGHMEM

Feb  8 17:27:37 dataserver 3546 reserved pages

Feb  8 17:27:37 dataserver 51395 pages shared

Feb  8 17:27:37 dataserver 0 pages swap cached

Feb  8 17:27:37 dataserver 26488 pages dirty

Feb  8 17:27:37 dataserver 0 pages writeback

Feb  8 17:27:37 dataserver 1751 pages mapped

Feb  8 17:27:37 dataserver 204601 pages slab

Feb  8 17:27:37 dataserver 96 pages pagetables

Feb  8 17:27:37 dataserver eth0: NULL pointer encountered in Rx ring

Feb  8 17:27:37 dataserver cur_rx:231824, dirty_rx:231824

Feb  8 17:27:37 dataserver eth0: NULL pointer encountered in Rx ring

Feb  8 17:27:37 dataserver cur_rx:231824, dirty_rx:231824

Feb  8 17:27:37 dataserver eth0: NULL pointer encountered in Rx ring

Feb  8 17:27:37 dataserver cur_rx:231824, dirty_rx:231824

Feb  8 17:27:37 dataserver eth0: NULL pointer encountered in Rx ring

Feb  8 17:27:37 dataserver cur_rx:231824, dirty_rx:231824

Feb  8 17:27:37 dataserver eth0: NULL pointer encountered in Rx ring

Feb  8 17:27:37 dataserver cur_rx:231824, dirty_rx:231824

Feb  8 17:27:38 dataserver eth0: NULL pointer encountered in Rx ring

Feb  8 17:27:38 dataserver cur_rx:231824, dirty_rx:231824

Feb  8 17:27:39 dataserver eth0: NULL pointer encountered in Rx ring

Feb  8 17:27:39 dataserver cur_rx:231824, dirty_rx:231824

Feb  8 17:27:39 dataserver eth0: NULL pointer encountered in Rx ring

Feb  8 17:27:39 dataserver cur_rx:231824, dirty_rx:231824

Feb  8 17:27:41 dataserver eth0: NULL pointer encountered in Rx ring

Feb  8 17:27:41 dataserver cur_rx:231824, dirty_rx:231824

Feb  8 17:27:41 dataserver eth0: NULL pointer encountered in Rx ring

Feb  8 17:27:41 dataserver cur_rx:231824, dirty_rx:231824

Feb  8 17:27:41 dataserver eth0: NULL pointer encountered in Rx ring

Feb  8 17:27:41 dataserver cur_rx:231824, dirty_rx:231824

Feb  8 17:27:41 dataserver eth0: NULL pointer encountered in Rx ring

Feb  8 17:27:41 dataserver cur_rx:231824, dirty_rx:231824

Feb  8 17:27:41 dataserver eth0: NULL pointer encountered in Rx ring

Feb  8 17:27:41 dataserver cur_rx:231824, dirty_rx:231824

Feb  8 17:27:42 dataserver eth0: NULL pointer encountered in Rx ring

Feb  8 17:27:42 dataserver cur_rx:231824, dirty_rx:231824

Feb  8 17:27:46 dataserver eth0: NULL pointer encountered in Rx ring

Feb  8 17:27:46 dataserver cur_rx:231824, dirty_rx:231824

Feb  8 17:28:07 dataserver eth0: NULL pointer encountered in Rx ring

Feb  8 17:28:07 dataserver cur_rx:231824, dirty_rx:231824

Feb  8 17:28:09 dataserver eth0: NULL pointer encountered in Rx ring

Feb  8 17:28:09 dataserver cur_rx:231824, dirty_rx:231824

Feb  8 17:28:10 dataserver eth0: NULL pointer encountered in Rx ring

Feb  8 17:28:10 dataserver cur_rx:231824, dirty_rx:231824

Feb  8 17:28:16 dataserver eth0: NULL pointer encountered in Rx ring

Feb  8 17:28:16 dataserver cur_rx:231824, dirty_rx:231824

Feb  8 17:28:16 dataserver eth0: NULL pointer encountered in Rx ring

Feb  8 17:28:16 dataserver cur_rx:231824, dirty_rx:231824

Feb  8 17:28:17 dataserver eth0: NULL pointer encountered in Rx ring

Feb  8 17:28:17 dataserver cur_rx:231824, dirty_rx:231824

Feb  8 17:28:22 dataserver eth0: NULL pointer encountered in Rx ring

Feb  8 17:28:22 dataserver cur_rx:231824, dirty_rx:231824

```

If I monitor dataserver's available free memory during these events, I find that it starts, before running the backup, with:

```

 # free

             total       used       free     shared    buffers     cached

Mem:       1034072      45740     988332          0       3144      22100

-/+ buffers/cache:      20496    1013576

Swap:       987988          0     987988

```

Within a couple minutes, it becomes:

```

# free

             total       used       free     shared    buffers     cached

Mem:       1034072     599064     435008          0       7020     102752

-/+ buffers/cache:     489292     544780

Swap:       987988          0     987988

```

And just before dataserver breaking from the network and dropping out of user mode:

```

 # free

             total       used       free     shared    buffers     cached

Mem:       1034072    1025376       8696          0       4188     194424

-/+ buffers/cache:     826764     207308

Swap:       987988          0     987988

```

So, I conclude that, for some reason, this backup process has started using progressively more memory on dataserver, eventually killing the network connection and terminating the backup.  :Confused: 

Meanwhile, Swap seems to be sitting there doing nothing...  :Question: 

Would someone please help me analyze what's going on, and tell me what might be done about it??

Thanks!

----------

## Dan

"swapper: page allocation failure" definately means the application needs more memory than it can get so yes the question is why isn't it using your swap?

Maybe its an MTU issue and it starts throwing chunks at it that it cant handle?

----------

## cgmd

dcoats wrote: *Quote:*   

> Maybe its an MTU issue and it starts throwing chunks at it that it cant handle?

 

How that would explain many months of no problem prior to the onset of the problem this week?

----------

## Dan

I dont guess it would unless you updated something that is causing it, network driver/kernel anything?

----------

## x22

Your kernel is using way too much memory (slab is kernel memory).

Check /proc/slabinfo (or slabtop command).

----------

## cgmd

slabtop , after a fresh reboot, shows:

```
  Active / Total Objects (% used)    : 26206 / 31788 (82.4%)

 Active / Total Slabs (% used)      : 1481 / 1481 (100.0%)

 Active / Total Caches (% used)     : 88 / 150 (58.7%)

 Active / Total Size (% used)       : 5140.70K / 5771.13K (89.1%)

 Minimum / Average / Maximum Object : 0.01K / 0.18K / 128.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME

  5640   5617  99%    0.12K    188       30       752K dentry_cache

  3444   3374  97%    0.04K     41       84       164K sysfs_dir_cache

  2599   2396  92%    0.03K     23      113        92K size-32

  2360   2113  89%    0.06K     40       59       160K size-64

  2162   1000  46%    0.08K     47       46       188K vm_area_struct

  1716   1711  99%    0.34K    156       11       624K inode_cache

  1120   1119  99%    0.50K    160        7       640K ext3_inode_cache

  1072   1036  96%    0.05K     16       67        64K buffer_head

  1020    396  38%    0.19K     51       20       204K filp

  1012    988  97%    0.04K     11       92        44K Acpi-Operand

   980    973  99%    0.27K     70       14       280K radix_tree_node

   762    419  54%    0.01K      3      254        12K anon_vma

   750    733  97%    0.12K     25       30       100K size-96

   540    539  99%    0.44K     60        9       240K shmem_inode_cache

   507    454  89%    0.02K      3      169        12K Acpi-Namespace

   480    474  98%    0.12K     16       30        64K size-128

   330    229  69%    0.36K     33       10       132K proc_inode_cache

   272    234  86%    0.50K     34        8       136K size-512

   270    263  97%    0.12K      9       30        36K bio

   254      8   3%    0.01K      1      254         4K revoke_table

   240    228  95%    0.19K     12       20        48K size-192

   234     56  23%    0.05K      3       78        12K delayacct_cache

   203     15   7%    0.02K      1      203         4K biovec-1

   203      8   3%    0.02K      1      203         4K tcp_bind_bucket

   202     56  27%    0.04K      2      101         8K pid

   178    173  97%    2.00K     89        2       356K size-2048

   177     25  14%    0.06K      3       59        12K fs_cache

   169      8   4%    0.02K      1      169         4K journal_handle

   156    139  89%    1.00K     39        4       156K size-1024

   150    149  99%    0.12K      5       30        20K kmem_cache

   135    125  92%    0.25K      9       15        36K size-256

   127     16  12%    0.03K      1      127         4K blkdev_ioc

   120     26  21%    0.25K      8       15        32K files_cache

   116     99  85%    0.13K      4       29        16K idr_layer_cache

   113      5   4%    0.03K      1      113         4K fib6_nodes

   113     10   8%    0.03K      1      113         4K ip_fib_hash

   113     10   8%    0.03K      1      113         4K ip_fib_alias

   110     54  49%    0.38K     11       10        44K signal_cache

    92      1   1%    0.04K      1       92         4K inotify_watch_cache

    90     26  28%    0.44K     10        9        40K mm_struct

    90     63  70%    0.25K      6       15        24K ip_dst_cache

    84     80  95%    0.18K      4       21        16K blkdev_requests

    81     55  67%    1.33K     27        3       108K task_struct

    77     77 100%    4.00K     77        1       308K size-4096

    72     54  75%    1.31K     24        3        96K sighand_cache

    72     18  25%    0.05K      1       72         4K journal_head

    60     32  53%    0.12K      2       30         8K sgpool-8

    59      7  11%    0.06K      1       59         4K biovec-4

    59      1   1%    0.06K      1       59         4K inet_peer_cache

    59      1   1%    0.06K      1       59         4K uid_cache

    56     56 100%    0.50K      8        7        32K sock_inode_cache

    45     16  35%    0.25K      3       15        12K skbuff_head_cache

    45     32  71%    0.25K      3       15        12K sgpool-16

    44     44 100%    0.98K     11        4        44K blkdev_queue

                             

```

slabtop shows  (just as a backup run is causeing the network connection to crash) the following:

```
Active / Total Objects (% used)    : 997662 / 1025160 (97.3%)

 Active / Total Slabs (% used)      : 215505 / 215530 (100.0%)

 Active / Total Caches (% used)     : 91 / 150 (60.7%)

 Active / Total Size (% used)       : 847199.18K / 852522.19K (99.4%)

 Minimum / Average / Maximum Object : 0.01K / 0.83K / 128.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME

365955 365955 100%    0.25K  24397       15     97588K skbuff_head_cache

360408 360408 100%    2.00K 180204        2    720816K size-2048

183844 183784  99%    0.06K   3116       59     12464K size-64

 36450  22303  61%    0.12K   1215       30      4860K dentry_cache

 30646  25262  82%    0.50K   4378        7     17512K ext3_inode_cache

 12663  11600  91%    0.05K    189       67       756K buffer_head

  5904   5904 100%    0.50K    738        8      2952K size-512

  4760   4176  87%    0.27K    340       14      1360K radix_tree_node

  3444   3382  98%    0.04K     41       84       164K sysfs_dir_cache

  2599   2515  96%    0.03K     23      113        92K size-32

  2232   1716  76%    0.05K     31       72       124K journal_head

  2162   1223  56%    0.08K     47       46       188K vm_area_struct

  1221   1110  90%    0.34K    111       11       444K inode_cache

  1020    498  48%    0.19K     51       20       204K filp

  1015    571  56%    0.02K      5      203        20K biovec-1

  1012    988  97%    0.04K     11       92        44K Acpi-Operand

   900    804  89%    0.12K     30       30       120K bio

   870    843  96%    0.12K     29       30       116K size-96

   812    192  23%    0.02K      4      203        16K revoke_record

   762    503  66%    0.01K      3      254        12K anon_vma

   549    542  98%    0.44K     61        9       244K shmem_inode_cache

   510    476  93%    0.12K     17       30        68K size-128

   507    454  89%    0.02K      3      169        12K Acpi-Namespace

   320    175  54%    0.36K     32       10       128K proc_inode_cache

   254      8   3%    0.01K      1      254         4K revoke_table

   240    240 100%    0.19K     12       20        48K size-192

   234     74  31%    0.05K      3       78        12K delayacct_cache

   203      8   3%    0.02K      1      203         4K tcp_bind_bucket

   203     16   7%    0.02K      1      203         4K fasync_cache

   202     75  37%    0.04K      2      101         8K pid

   177     43  24%    0.06K      3       59        12K fs_cache

   169      4   2%    0.02K      1      169         4K journal_handle

   168    114  67%    0.18K      8       21        32K blkdev_requests

   165    165 100%    0.25K     11       15        44K size-256

   156    155  99%    1.00K     39        4       156K size-1024

   150    149  99%    0.12K      5       30        20K kmem_cache

   127     34  26%    0.03K      1      127         4K blkdev_ioc

   120     44  36%    0.25K      8       15        32K files_cache

   116     98  84%    0.13K      4       29        16K idr_layer_cache

   113      5   4%    0.03K      1      113         4K fib6_nodes

   113     10   8%    0.03K      1      113         4K ip_fib_hash

   113     10   8%    0.03K      1      113         4K ip_fib_alias

   110     72  65%    0.38K     11       10        44K signal_cache

    92      1   1%    0.04K      1       92         4K inotify_watch_cache

    90     44  48%    0.44K     10        9        40K mm_struct

    88     88 100%    4.00K     88        1       352K size-4096

    81     69  85%    1.33K     27        3       108K task_struct

    72     68  94%    1.31K     24        3        96K sighand_cache

    63     63 100%    0.50K      9        7        36K sock_inode_cache

    60     32  53%    0.12K      2       30         8K sgpool-8

    60     44  73%    0.25K      4       15        16K ip_dst_cache

    59     23  38%    0.06K      1       59         4K biovec-4

    59      1   1%    0.06K      1       59         4K inet_peer_cache

    59      1   1%    0.06K      1       59         4K uid_cache

                                                   
```

Obviously, a tremendous memory drain is occurring. But, why, and how can I correct it??

Thanks!

----------

## cgmd

My saga continues without explanation... 

```
 

Feb 10 10:15:01 dataserver cron[5784]: (root) CMD (/etc/init.d/ntp-client restart)

Feb 10 10:15:21 dataserver swapper: page allocation failure. order:0, mode:0x20

Feb 10 10:15:21 dataserver [<c014b999>]  [<c015f70e>]  [<c015f95e>]  [<c03cf0ca>]  [<c030050a>]  [<c0143cd5>]  [<c0144f64>]  [<c0144edb>]  [<c01057a6>]  [<c010389a>]  [<c0101c45>]  [<c0101c7c>]  [<c0101d45>]  [<c054b7b2>]  [<c054b1e0>]  =======================

Feb 10 10:15:21 dataserver Mem-info:

Feb 10 10:15:21 dataserver DMA per-cpu:

Feb 10 10:15:21 dataserver CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0

Feb 10 10:15:21 dataserver Normal per-cpu:

Feb 10 10:15:21 dataserver CPU    0: Hot: hi:  186, btch:  31 usd:  30   Cold: hi:   62, btch:  15 usd:  51

Feb 10 10:15:21 dataserver HighMem per-cpu:

Feb 10 10:15:21 dataserver CPU    0: Hot: hi:   42, btch:   7 usd:   5   Cold: hi:   14, btch:   3 usd:  12

Feb 10 10:15:21 dataserver Active:6264 inactive:29825 dirty:20777 writeback:0 unstable:0 free:6960 slab:214363 mapped:1779 pagetables:106

Feb 10 10:15:21 dataserver DMA free:3520kB min:68kB low:84kB high:100kB active:0kB inactive:2372kB present:16256kB pages_scanned:0 all_unreclaimable? no

Feb 10 10:15:21 dataserver lowmem_reserve[]: 0 873 999

Feb 10 10:15:21 dataserver Normal free:1384kB min:3744kB low:4680kB high:5616kB active:3020kB inactive:32404kB present:894080kB pages_scanned:0 all_unreclaimable? no

Feb 10 10:15:21 dataserver lowmem_reserve[]: 0 0 1013

Feb 10 10:15:21 dataserver HighMem free:22936kB min:128kB low:260kB high:396kB active:22036kB inactive:84524kB present:129732kB pages_scanned:0 all_unreclaimable? no

Feb 10 10:15:21 dataserver lowmem_reserve[]: 0 0 0

Feb 10 10:15:21 dataserver DMA: 0*4kB 0*8kB 0*16kB 0*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3520kB

Feb 10 10:15:21 dataserver Normal: 0*4kB 1*8kB 0*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1384kB

Feb 10 10:15:21 dataserver HighMem: 0*4kB 1*8kB 1*16kB 0*32kB 0*64kB 1*128kB 5*256kB 4*512kB 7*1024kB 2*2048kB 2*4096kB = 22936kB

Feb 10 10:15:21 dataserver Swap cache: add 0, delete 0, find 0/0, race 0+0

Feb 10 10:15:21 dataserver Free swap  = 987988kB

Feb 10 10:15:21 dataserver Total swap = 987988kB

Feb 10 10:15:21 dataserver Free swap:       987988kB

Feb 10 10:15:21 dataserver 262064 pages of RAM

Feb 10 10:15:21 dataserver 32688 pages of HIGHMEM

Feb 10 10:15:21 dataserver 3546 reserved pages

Feb 10 10:15:21 dataserver 36882 pages shared

Feb 10 10:15:21 dataserver 0 pages swap cached

Feb 10 10:15:21 dataserver 20777 pages dirty

Feb 10 10:15:21 dataserver 0 pages writeback

Feb 10 10:15:21 dataserver 1779 pages mapped

Feb 10 10:15:21 dataserver 214363 pages slab

Feb 10 10:15:21 dataserver 106 pages pagetables

Feb 10 10:15:21 dataserver swapper: page allocation failure. order:0, mode:0x20

Feb 10 10:15:21 dataserver [<c014b999>]  [<c015f70e>]  [<c0112918>]  [<c015f95e>]  [<c03cf0ca>]  [<c0300622>]  [<c0143cd5>]  [<c0144f64>]  [<c0144edb>]  [<c01057a6>]  [<c010389a>]  [<c0101c45>]  [<c0101c7c>]  [<c0101d45>]  [<c054b7b2>]  [<c054b1e0>]  =======================

Feb 10 10:15:21 dataserver Mem-info:

Feb 10 10:15:21 dataserver DMA per-cpu:

Feb 10 10:15:21 dataserver CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0

Feb 10 10:15:21 dataserver Normal per-cpu:

Feb 10 10:15:21 dataserver CPU    0: Hot: hi:  186, btch:  31 usd:  30   Cold: hi:   62, btch:  15 usd:  51

Feb 10 10:15:21 dataserver HighMem per-cpu:

Feb 10 10:15:21 dataserver CPU    0: Hot: hi:   42, btch:   7 usd:   5   Cold: hi:   14, btch:   3 usd:  12

Feb 10 10:15:21 dataserver Active:6264 inactive:29825 dirty:20777 writeback:0 unstable:0 free:6960 slab:214363 mapped:1779 pagetables:106

Feb 10 10:15:21 dataserver DMA free:3520kB min:68kB low:84kB high:100kB active:0kB inactive:2372kB present:16256kB pages_scanned:0 all_unreclaimable? no

Feb 10 10:15:21 dataserver lowmem_reserve[]: 0 873 999

Feb 10 10:15:21 dataserver Normal free:1384kB min:3744kB low:4680kB high:5616kB active:3020kB inactive:32404kB present:894080kB pages_scanned:0 all_unreclaimable? no

Feb 10 10:15:21 dataserver lowmem_reserve[]: 0 0 1013

Feb 10 10:15:21 dataserver HighMem free:22936kB min:128kB low:260kB high:396kB active:22036kB inactive:84524kB present:129732kB pages_scanned:0 all_unreclaimable? no

Feb 10 10:15:21 dataserver lowmem_reserve[]: 0 0 0

Feb 10 10:15:21 dataserver DMA: 0*4kB 0*8kB 0*16kB 0*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3520kB

Feb 10 10:15:21 dataserver Normal: 0*4kB 1*8kB 0*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1384kB

Feb 10 10:15:21 dataserver HighMem: 0*4kB 1*8kB 1*16kB 0*32kB 0*64kB 1*128kB 5*256kB 4*512kB 7*1024kB 2*2048kB 2*4096kB = 22936kB

Feb 10 10:15:21 dataserver Swap cache: add 0, delete 0, find 0/0, race 0+0

Feb 10 10:15:21 dataserver Free swap  = 987988kB

Feb 10 10:15:21 dataserver Total swap = 987988kB

Feb 10 10:15:21 dataserver Free swap:       987988kB

Feb 10 10:15:21 dataserver 262064 pages of RAM

Feb 10 10:15:21 dataserver 32688 pages of HIGHMEM

Feb 10 10:15:21 dataserver 3546 reserved pages

Feb 10 10:15:21 dataserver 36882 pages shared

Feb 10 10:15:21 dataserver 0 pages swap cached

Feb 10 10:15:21 dataserver 20777 pages dirty

Feb 10 10:15:21 dataserver 0 pages writeback

Feb 10 10:15:21 dataserver 1779 pages mapped

Feb 10 10:15:21 dataserver 214363 pages slab

Feb 10 10:15:21 dataserver 106 pages pagetables

Feb 10 10:15:21 dataserver eth0: Memory squeeze,deferring packet.

Feb 10 10:15:21 dataserver swapper: page allocation failure. order:0, mode:0x20

Feb 10 10:15:21 dataserver [<c014b999>]  [<c015f70e>]  [<c0112918>]  [<c015f95e>]  [<c03cf0ca>]  [<c030050a>]  [<c0143cd5>]  [<c0144f64>]  [<c0144edb>]  [<c01057a6>]  [<c010389a>]  [<c0101c45>]  [<c0101c7c>]  [<c0101d45>]  [<c054b7b2>]  [<c054b1e0>]  =======================

Feb 10 10:15:21 dataserver Mem-info:

Feb 10 10:15:21 dataserver DMA per-cpu:

Feb 10 10:15:21 dataserver CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0

Feb 10 10:15:21 dataserver Normal per-cpu:

Feb 10 10:15:21 dataserver CPU    0: Hot: hi:  186, btch:  31 usd:  30   Cold: hi:   62, btch:  15 usd:  51

Feb 10 10:15:21 dataserver HighMem per-cpu:

Feb 10 10:15:21 dataserver CPU    0: Hot: hi:   42, btch:   7 usd:   5   Cold: hi:   14, btch:   3 usd:  12

Feb 10 10:15:21 dataserver Active:6264 inactive:29825 dirty:20777 writeback:0 unstable:0 free:6960 slab:214363 mapped:1779 pagetables:106

Feb 10 10:15:21 dataserver DMA free:3520kB min:68kB low:84kB high:100kB active:0kB inactive:2372kB present:16256kB pages_scanned:0 all_unreclaimable? no

Feb 10 10:15:21 dataserver lowmem_reserve[]: 0 873 999

Feb 10 10:15:21 dataserver Normal free:1384kB min:3744kB low:4680kB high:5616kB active:3020kB inactive:32404kB present:894080kB pages_scanned:0 all_unreclaimable? no

Feb 10 10:15:21 dataserver lowmem_reserve[]: 0 0 1013

Feb 10 10:15:21 dataserver HighMem free:22936kB min:128kB low:260kB high:396kB active:22036kB inactive:84524kB present:129732kB pages_scanned:0 all_unreclaimable? no

Feb 10 10:15:21 dataserver lowmem_reserve[]: 0 0 0

Feb 10 10:15:21 dataserver DMA: 0*4kB 0*8kB 0*16kB 0*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3520kB

Feb 10 10:15:21 dataserver Normal: 0*4kB 1*8kB 0*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1384kB

Feb 10 10:15:21 dataserver HighMem: 0*4kB 1*8kB 1*16kB 0*32kB 0*64kB 1*128kB 5*256kB 4*512kB 7*1024kB 2*2048kB 2*4096kB = 22936kB

Feb 10 10:15:21 dataserver Swap cache: add 0, delete 0, find 0/0, race 0+0

Feb 10 10:15:21 dataserver Free swap  = 987988kB

Feb 10 10:15:21 dataserver Total swap = 987988kB

Feb 10 10:15:21 dataserver Free swap:       987988kB

Feb 10 10:15:21 dataserver 262064 pages of RAM

Feb 10 10:15:21 dataserver 32688 pages of HIGHMEM

Feb 10 10:15:21 dataserver 3546 reserved pages

Feb 10 10:15:21 dataserver 36882 pages shared

Feb 10 10:15:21 dataserver 0 pages swap cached

Feb 10 10:15:21 dataserver 20777 pages dirty

Feb 10 10:15:21 dataserver 0 pages writeback

Feb 10 10:15:21 dataserver 1779 pages mapped

Feb 10 10:15:21 dataserver 214363 pages slab

Feb 10 10:15:21 dataserver 106 pages pagetables

Feb 10 10:15:21 dataserver swapper: page allocation failure. order:0, mode:0x20

Feb 10 10:15:21 dataserver [<c014b999>]  [<c015f70e>]  [<c0112918>]  [<c015f95e>]  [<c03cf0ca>]  [<c0300622>]  [<c0143cd5>]  [<c0144f64>]  [<c0144edb>]  [<c01057a6>]  [<c010389a>]  [<c0101c45>]  [<c0101c7c>]  [<c0101d45>]  [<c054b7b2>]  [<c054b1e0>]  =======================

Feb 10 10:15:21 dataserver Mem-info:

Feb 10 10:15:21 dataserver DMA per-cpu:

Feb 10 10:15:21 dataserver CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0

Feb 10 10:15:21 dataserver Normal per-cpu:

Feb 10 10:15:21 dataserver CPU    0: Hot: hi:  186, btch:  31 usd:  30   Cold: hi:   62, btch:  15 usd:  51

Feb 10 10:15:21 dataserver HighMem per-cpu:

Feb 10 10:15:21 dataserver CPU    0: Hot: hi:   42, btch:   7 usd:   5   Cold: hi:   14, btch:   3 usd:  12

Feb 10 10:15:21 dataserver Active:6264 inactive:29825 dirty:20777 writeback:0 unstable:0 free:6960 slab:214363 mapped:1779 pagetables:106

Feb 10 10:15:21 dataserver DMA free:3520kB min:68kB low:84kB high:100kB active:0kB inactive:2372kB present:16256kB pages_scanned:0 all_unreclaimable? no

Feb 10 10:15:21 dataserver lowmem_reserve[]: 0 873 999

Feb 10 10:15:21 dataserver Normal free:1384kB min:3744kB low:4680kB high:5616kB active:3020kB inactive:32404kB present:894080kB pages_scanned:0 all_unreclaimable? no

Feb 10 10:15:21 dataserver lowmem_reserve[]: 0 0 1013

Feb 10 10:15:21 dataserver HighMem free:22936kB min:128kB low:260kB high:396kB active:22036kB inactive:84524kB present:129732kB pages_scanned:0 all_unreclaimable? no

Feb 10 10:15:21 dataserver lowmem_reserve[]: 0 0 0

Feb 10 10:15:21 dataserver DMA: 0*4kB 0*8kB 0*16kB 0*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3520kB

Feb 10 10:15:21 dataserver Normal: 0*4kB 1*8kB 0*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1384kB

Feb 10 10:15:21 dataserver HighMem: 0*4kB 1*8kB 1*16kB 0*32kB 0*64kB 1*128kB 5*256kB 4*512kB 7*1024kB 2*2048kB 2*4096kB = 22936kB

Feb 10 10:15:21 dataserver Swap cache: add 0, delete 0, find 0/0, race 0+0

Feb 10 10:15:21 dataserver Free swap  = 987988kB

Feb 10 10:15:21 dataserver Total swap = 987988kB

Feb 10 10:15:21 dataserver Free swap:       987988kB

Feb 10 10:15:21 dataserver 262064 pages of RAM

Feb 10 10:15:21 dataserver 32688 pages of HIGHMEM

Feb 10 10:15:21 dataserver 3546 reserved pages

Feb 10 10:15:21 dataserver 36882 pages shared

Feb 10 10:15:21 dataserver 0 pages swap cached

Feb 10 10:15:21 dataserver 20777 pages dirty

Feb 10 10:15:21 dataserver 0 pages writeback

Feb 10 10:15:21 dataserver 1779 pages mapped

Feb 10 10:15:21 dataserver 214363 pages slab

Feb 10 10:15:21 dataserver 106 pages pagetables

Feb 10 10:15:21 dataserver eth0: Memory squeeze,deferring packet.

Feb 10 10:15:21 dataserver eth0: NULL pointer encountered in Rx ring

Feb 10 10:15:21 dataserver cur_rx:235785, dirty_rx:235785

Feb 10 10:15:21 dataserver eth0: NULL pointer encountered in Rx ring

Feb 10 10:15:21 dataserver cur_rx:235785, dirty_rx:235785

Feb 10 10:15:21 dataserver eth0: NULL pointer encountered in Rx ring

Feb 10 10:15:21 dataserver cur_rx:235785, dirty_rx:235785

```

Would someone please show me the light? How can I deal with swapper: page allocation failure:

```

Feb 10 10:15:01 dataserver cron[5784]: (root) CMD (/etc/init.d/ntp-client restart)

Feb 10 10:15:21 dataserver swapper: page allocation failure. order:0, mode:0x20

Feb 10 10:15:21 dataserver [<c014b999>]  [<c015f70e>]  [<c015f95e>]  [<c03cf0ca>]  [<c030050a>]  [<c0143cd5>]  [<c0144f64>]  [<c0144edb>]  [<c01057a6>]  [<c010389a>]  [<c0101c45>]  [<c0101c7c>]  [<c0101d45>]  [<c054b7b2>]  [<c054b1e0>]
```

Thanks!

----------

## NeddySeagoon

cgmd,

Have you updated recently ?

I suspect that one of your applications has a memory leak. This occurs when memory is dynamically allocated then improperly discarded, so the system never recovers it. If it happens with non swappable memory, swap will remain empty (almost) as a few things get squeezed out. Eventually, the leak gets all of memory and maybe swap. Then the kernel begins killing random processes so it can get memory for itself. The only way to reclaim the memory is to reboot.

You log shows that smbd wanted memory and was refused

```
 smbd: page allocation failure. order:0, mode:0x20 
```

Thats samba.

Reboot, then watch top while your backup runs, look for a process that is using more and more memory. I think you can make top sort on the memory fields, so the memory hog will be at the top of the list.

Try upgrading or downgrading whatever that is.

The network issue is a side effect. The network stack can't get memory for incoming packets either. Its the same root cause.

----------

## cgmd

NeddySeagoon...

OK, I observed top over a ssh connection, as memory deteriorated, and this is the final top output at the time the network connection breaks:

```
top - 16:11:19 up 14 min,  2 users,  load average: 0.37, 0.18, 0.06

Tasks:  56 total,   2 running,  54 sleeping,   0 stopped,   0 zombie

Cpu(s):  4.0% us,  7.3% sy,  0.0% ni,  6.6% id, 77.4% wa,  1.7% hi,  3.0% si

Mem:   1034072k total,   951476k used,    82596k free,    11128k buffers

Swap:   987988k total,        0k used,   987988k free,    39848k cached

Connection to 192.168.1.108 closed by remote host.

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND

 5389 root      18   0 13648 7008 2340 D  9.0  0.7   0:16.80 smbd

 4905 root      15   0  1580  232  152 S  3.3  0.0   0:02.70 dhcpcd

  192 root      10  -5     0    0    0 D  0.7  0.0   0:00.06 kswapd0

    1 root      15   0  1564  552  480 S  0.0  0.1   0:00.82 init

    2 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/0

    3 root      34  19     0    0    0 S  0.0  0.0   0:00.01 ksoftirqd/0

    4 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 watchdog/0

    5 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 events/0

    6 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 khelper

    7 root      12  -5     0    0    0 S  0.0  0.0   0:00.00 kthread

   75 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kblockd/0

   76 root      20  -5     0    0    0 S  0.0  0.0   0:00.00 kacpid

  159 root      20  -5     0    0    0 S  0.0  0.0   0:00.00 ksuspend_usbd

  162 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 khubd

  164 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kseriod

  177 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 khpsbpkt

  181 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 knodemgrd_0

  190 root      25   0     0    0    0 S  0.0  0.0   0:00.00 pdflush

  191 root      15   0     0    0    0 S  0.0  0.0   0:00.00 pdflush

  193 root      20  -5     0    0    0 S  0.0  0.0   0:00.00 aio/0

  450 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 kpsmoused

  458 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 ib_cm/0

  459 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 iw_cm_wq

  460 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 ib_addr_wq

  461 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 rdma_cm_wq

  462 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 mthca_catas

  464 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 ipoib

  468 root      16  -5     0    0    0 S  0.0  0.0   0:00.01 kjournald

  650 root      16  -4  2100  916  356 S  0.0  0.1   0:00.79 udevd

 1442 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kjournald

 1445 root      10  -5     0    0    0 S  0.0  0.0   0:00.03 kjournald

 1448 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 kjournald

 4354 root      15   0  1936  604  416 S  0.0  0.1   0:00.01 syslog-ng

 4994 root      18   0 12952 1072  836 S  0.0  0.1   0:00.00 hpiod

 5004 root      15   0  9460 4736 1136 S  0.0  0.5   0:00.01 hpssd.py

 5073 root      15   0  5804 2128 1512 S  0.0  0.2   0:00.03 cupsd

 5264 root      15   0  3928 1252  976 S  0.0  0.1   0:00.02 ntpd

 5326 root      25   0  1888  568  460 S  0.0  0.1   0:00.00 rsync

 5389 root      18   0  9848 2824 1872 S  0.0  0.3   0:00.00 smbd

 5398 root      22   0  9848 1220  284 S  0.0  0.1   0:00.00 smbd

 5399 root      15   0  5500 1360  908 S  0.0  0.1   0:00.01 nmbd

 5468 root      18   0  4220 1080  768 S  0.0  0.1   0:00.00 sshd

 5537 root      18   0  1808  700  576 S  0.0  0.1   0:00.00 cron

 5614 root      16   0  2380 1168  908 S  0.0  0.1   0:00.00 login

 5615 root      18   0  1600  636  552 S  0.0  0.1   0:00.00 agetty

 5616 root      18   0  1600  636  552 S  0.0  0.1   0:00.00 agetty

 5621 root      18   0  1600  636  552 S  0.0  0.1   0:00.00 agetty

 5624 root      18   0  1596  632  552 S  0.0  0.1   0:00.01 agetty

 5625 root      18   0  1600  636  552 S  0.0  0.1   0:00.00 agetty

 5630 cgmd~~    16   0  3044 1584 1264 S  0.0  0.2   0:00.04 bash

 5724 root      25   0  1408  164  112 S  0.0  0.0   0:00.00 vmnet-bridge

 5736 cgmd~~    15   0  2204 1096  852 S  0.0  0.1   0:00.00 top

 5738 root      18   0  7040 2184 1764 S  0.0  0.2   0:00.08 sshd

 5741 cgmd~~    15   0  7192 1488 1032 R  0.0  0.1   0:00.16 sshd

 5742 cgmd~~    15   0  3048 1596 1268 S  0.0  0.2   0:00.11 bash

 5934 cgmd~~    15   0  2204 1100  848 R  0.0  0.1   0:00.01 top

```

It certainly appears that smbd, by far, is the culprit, with dhcpcd also contributing. I am now using net-fs/samba-3.0.24, and I don't know how long ago I did the latest upgrade. I guess I'll try downgrading to net-fs/samba-3.0.22-r3, to try to fix this. I'll report back what I discover.

Thanks, once more!

----------

## NeddySeagoon

cgmd,

That all looks fine. Even smbd is only using 13 MB, some of which is swappable.

You swap usage is still at zero, so even the things that are used only for system start and stop have not been squeezed out.

Its not a memory leak problem.

----------

## cgmd

What's the implication of the syslog reports of swapper: page allocation failure??

----------

## NeddySeagoon

cgmd,

That means that a program asked for some memory and was refused. Normally because the amount of memory requested was not available. Swapping would have been invoked to provide the memory, if required and swappable pages existed.

----------

## cgmd

NeddySeagoon...

Is that not problematic, as indicated by the following snippet from above?

```

Feb 10 10:15:21 dataserver swapper: page allocation failure. order:0, mode:0x20

Feb 10 10:15:21 dataserver [<c014b999>]  [<c015f70e>]  [<c0112918>]  [<c015f95e>]  [<c03cf0ca>]  [<c0300622>]  [<c0143cd5>]  [<c0144f64>]  [<c0144edb>]  [<c01057a6>]  [<c010389a>]  [<c0101c45>]  [<c0101c7c>]  [<c0101d45>]  [<c054b7b2>]  [<c054b1e0>]  =======================

Feb 10 10:15:21 dataserver Mem-info:

Feb 10 10:15:21 dataserver DMA per-cpu:

Feb 10 10:15:21 dataserver CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0

Feb 10 10:15:21 dataserver Normal per-cpu:

Feb 10 10:15:21 dataserver CPU    0: Hot: hi:  186, btch:  31 usd:  30   Cold: hi:   62, btch:  15 usd:  51

Feb 10 10:15:21 dataserver HighMem per-cpu:

Feb 10 10:15:21 dataserver CPU    0: Hot: hi:   42, btch:   7 usd:   5   Cold: hi:   14, btch:   3 usd:  12

Feb 10 10:15:21 dataserver Active:6264 inactive:29825 dirty:20777 writeback:0 unstable:0 free:6960 slab:214363 mapped:1779 pagetables:106

Feb 10 10:15:21 dataserver DMA free:3520kB min:68kB low:84kB high:100kB active:0kB inactive:2372kB present:16256kB pages_scanned:0 all_unreclaimable? no

Feb 10 10:15:21 dataserver lowmem_reserve[]: 0 873 999

Feb 10 10:15:21 dataserver Normal free:1384kB min:3744kB low:4680kB high:5616kB active:3020kB inactive:32404kB present:894080kB pages_scanned:0 all_unreclaimable? no

Feb 10 10:15:21 dataserver lowmem_reserve[]: 0 0 1013

Feb 10 10:15:21 dataserver HighMem free:22936kB min:128kB low:260kB high:396kB active:22036kB inactive:84524kB present:129732kB pages_scanned:0 all_unreclaimable? no

Feb 10 10:15:21 dataserver lowmem_reserve[]: 0 0 0

Feb 10 10:15:21 dataserver DMA: 0*4kB 0*8kB 0*16kB 0*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3520kB

Feb 10 10:15:21 dataserver Normal: 0*4kB 1*8kB 0*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1384kB

Feb 10 10:15:21 dataserver HighMem: 0*4kB 1*8kB 1*16kB 0*32kB 0*64kB 1*128kB 5*256kB 4*512kB 7*1024kB 2*2048kB 2*4096kB = 22936kB

Feb 10 10:15:21 dataserver Swap cache: add 0, delete 0, find 0/0, race 0+0

Feb 10 10:15:21 dataserver Free swap  = 987988kB

Feb 10 10:15:21 dataserver Total swap = 987988kB

Feb 10 10:15:21 dataserver Free swap:       987988kB

Feb 10 10:15:21 dataserver 262064 pages of RAM

Feb 10 10:15:21 dataserver 32688 pages of HIGHMEM

Feb 10 10:15:21 dataserver 3546 reserved pages

Feb 10 10:15:21 dataserver 36882 pages shared

Feb 10 10:15:21 dataserver 0 pages swap cached

Feb 10 10:15:21 dataserver 20777 pages dirty

Feb 10 10:15:21 dataserver 0 pages writeback

Feb 10 10:15:21 dataserver 1779 pages mapped

Feb 10 10:15:21 dataserver 214363 pages slab

Feb 10 10:15:21 dataserver 106 pages pagetables

Feb 10 10:15:21 dataserver eth0: Memory squeeze,deferring packet. 
```

Why is swap not pulled in??

----------

## NeddySeagoon

cgmd,

These numbers

```
Feb 10 10:15:21 dataserver 262064 pages of RAM 

Feb 10 10:15:21 dataserver 214363 pages slab
```

indicate the kernel has swallowed most of RAM (a page is 4kB)

Thats not supposed to happen. Slab is kernel RAM and is probably not swappable.

I don't know why this happens. It may be a kernel issue.

----------

## bombcar

You've got evidence of a kernel memory leak - update to the latest kernel and try again.

----------

## NeddySeagoon

cgmd, 

Do you have any kernel debug options on in your kernel config ?

If so, turn them off and tru a kernel without them.

Look under kernel hacking in make menuconfig

----------

## cgmd

NeddySeagoon,

A look at my "kernel hacking" section did, indeed, show a variety of enabled kernel debug options:

```

 ?                                          [ ] Show timing information on printks                                                          $

  ? ?                                       [ ] Enable __must_check logic                                                                   $

  ? ?                                       [*] Magic SysRq key                                                                             $

  ? ?                                       [ ] Enable unused/obsolete exported symbols                                                     $

  ? ?                                       [*] Kernel debugging                                                                            $

  ? ?                                       (15)  Kernel log buffer size (16 => 64KB, 17 => 128KB)                                          $

  ? ?                                       [*]   Detect Soft Lockups                                                                       $

  ? ?                                       [ ]   Collect scheduler statistics                                                              $

  ? ?                                       [ ]   Debug slab memory allocations                                                             $

  ? ?                                       [ ]   RT Mutex debugging, deadlock detection                                                    $

  ? ?                                       [ ]   Built-in scriptable tester for rt-mutexes                                                 $

  ? ?                                       [ ]   Spinlock and rw-lock debugging: basic checks                                              $

  ? ?                                       [*]   Mutex debugging: basic checks                                                             $

  ? ?                                       [ ]   RW-sem debugging: basic checks                                                            $

  ? ?                                       [ ]   Lock debugging: detect incorrect freeing of live locks                                    $

  ? ?                                       [ ]   Lock debugging: prove locking correctness                                                 $

  ? ?                                       [ ]   Spinlock debugging: sleep-inside-spinlock checking                                        $

  ? ?                                       [ ]   Locking API boot-time self-tests                                                          $

  ? ?                                       [ ]   kobject debugging                                                                         $

  ? ?                                       [ ]   Highmem debugging                                                                         $

  ? ?                                       [ ]   Verbose BUG() reporting (adds 70K)                                                        $

  ? ?                                       [ ]   Compile the kernel with debug info                                                        $

  ? ?                                       [ ] Debug Filesystem                                                                            $

  ? ?                                       [ ] Debug VM                                                                                    $

  ? ?                                       [ ] Debug linked list manipulation                                                              $

  ? ?                                       [ ] Compile the kernel with frame pointers                                                      $

  ? ?                                       [ ] Compile the kernel with frame unwind information                                            $

  ? ?                                       [*] Force gcc to inline functions marked 'inline'                                               $

  ? ?                                       [ ] Run 'make headers_check' when building vmlinux                                              $

  ? ?                                       < > torture tests for RCU                                                                       $

  ? ?                                       [*] Early printk                                                                                $

  ? ?                                       [ ] Check for stack overflows                                                                   $

  ? ?                                       [ ] Stack utilization instrumentation                                                           $

  ? ?                                       [ ] Debug page memory allocations                                                               $

  ? ?                                       [ ] Write protect kernel read-only data structures                                              $

  ? ?                                       [*] Use 4Kb for kernel stacks instead of 8Kb                                                    $

  ? ?                                       [*] Enable doublefault exception handler                                                        $

  

```

With the exception of Enable doublefault exception handler, I unselected all the choices. I would, also, have unselected doublefault exception handler , but the info about it scared me: *Quote:*   

> CONFIG_DOUBLEFAULT:                                                                                                                                     │
> 
>   │                                                                                                                                                         │
> 
>   │ This option allows trapping of rare doublefault exceptions that                                                                                         │
> ...

 

...I need no more grey hair!   :Sad: 

The good news: The windows to dataserver backup, now, runs to completion, without noteworthy errors!   :Very Happy:  ...

```

Feb 11 10:30:01 dataserver cron[5870]: (root) CMD (test -x /usr/sbin/run-crons && /usr/sbin/run-crons )

Feb 11 10:30:01 dataserver cron[5873]: (root) CMD (/etc/init.d/ntp-client restart)

Feb 11 10:30:31 dataserver sSMTP[5947]: Unable to locate mail

Feb 11 10:30:31 dataserver sSMTP[5947]: Cannot open mail:25

Feb 11 10:30:31 dataserver cron[5871]: (root) MAIL (mailed 81 bytes of output but got status 0x0001 )

Feb 11 10:40:01 dataserver cron[5986]: (root) CMD (test -x /usr/sbin/run-crons && /usr/sbin/run-crons )

Feb 11 10:45:01 dataserver cron[5998]: (root) CMD (/etc/init.d/ntp-client restart)

Feb 11 10:45:32 dataserver sSMTP[6063]: Unable to locate mail

Feb 11 10:45:32 dataserver sSMTP[6063]: Cannot open mail:25

Feb 11 10:45:32 dataserver cron[5997]: (root) MAIL (mailed 81 bytes of output but got status 0x0001 )

/var/log/messages-wo_debug lines 1-10/10 (END)                                                                            
```

That is, excepting the Cannot open mail:25 message which is always present, defying my efforts to stifle it!

I have no idea how my kernel config became contaminated with those various selections, but I feel there are likely many additional examples of enabled, but, unnecessary settings, throughout. With each kernel upgrade, I'm guilty of incorporating some "stuff" that sounds good, but is accompanied by very little advice as to what type users should adopt it. Seems like an ongoing dilemma!  :Confused: 

In any event, my system seems to be healthy, once more, thanks to the advice you guys have given, and, in particular, to NeddySegoon, showing me the way!!

Many thanks!!

----------

## NeddySeagoon

cgmd,

If you have the time, it would be good to discover exactly which setting caused the problem.

Theres nothing there that looks like it is supposed to grab more and more memory for saving things in.

I'm glad its fixed.

It looks like some cron jobs are trying to mail output to root, which is normal but you don't have a mail transport agent, so the mails always fail with that error about mail

----------

