# swapper: page allocation failure. order:0, mode:0x20 (NFS)

## KShots

I get a flood of those in my dmesg output when my fileserver gets pegged with heavy NFS traffic from a single host. Usually it knocks the fileserver off the network, which I seem to be able to recover by restarting the network interface (which was luckily reachable via a serial console). The network is running gigabit speeds, but without jumbo frames (I tried it briefly, and the fileserver barfs on the first jumbo frame it receives, requiring a reboot)... so the mtu across the network is all at 1500.

First, the kernel in question:

```
rich@gorgon ~ $ uname -a

Linux gorgon 2.6.32-hardened-r9 #5 Thu Sep 2 11:17:08 EDT 2010 x86_64 VIA Nano processor L2200@1600MHz CentaurHauls GNU/Linux
```

... a snippet of the dmesg output:

```
[530861.382573] swapper: page allocation failure. order:0, mode:0x20

[530861.382580] Pid: 0, comm: swapper Not tainted 2.6.32-hardened-r9 #5

[530861.382584] Call Trace:

[530861.382588]  <IRQ>  [<ffffffff8106552b>] ? __alloc_pages_nodemask+0x536/0x57f

[530861.382607]  [<ffffffff812793cc>] ? ip_queue_xmit+0x2fd/0x347

[530861.382615]  [<ffffffff8124e37d>] ? __alloc_skb+0x36/0x14f

[530861.382623]  [<ffffffff81084a91>] ? __slab_alloc+0x157/0x427

[530861.382629]  [<ffffffff8124e37d>] ? __alloc_skb+0x36/0x14f

[530861.382636]  [<ffffffff8124e37d>] ? __alloc_skb+0x36/0x14f

[530861.382642]  [<ffffffff81084f45>] ? kmem_cache_alloc+0x68/0x92

[530861.382652]  [<ffffffff81284d6c>] ? tcp_try_rmem_schedule+0x43/0x287

[530861.382658]  [<ffffffff8124e37d>] ? __alloc_skb+0x36/0x14f

[530861.382666]  [<ffffffff81289a02>] ? tcp_send_ack+0x23/0xf4

[530861.382673]  [<ffffffff81287ef2>] ? tcp_rcv_established+0x39f/0x503

[530861.382681]  [<ffffffff8128e0ec>] ? tcp_v4_do_rcv+0x2c/0x193

[530861.382688]  [<ffffffff8128e5ef>] ? tcp_v4_rcv+0x39c/0x59e

[530861.382696]  [<ffffffff8135dbc0>] ? tcp_protocol+0x0/0x40

[530861.382703]  [<ffffffff812752f6>] ? ip_local_deliver+0xf6/0x162

[530861.382710]  [<ffffffff81275170>] ? ip_rcv+0x4a0/0x4cd

[530861.382719]  [<ffffffff81255151>] ? process_backlog+0x63/0x8c

[530861.382726]  [<ffffffff81255676>] ? net_rx_action+0x46/0xff

[530861.382735]  [<ffffffff81028b15>] ? __do_softirq+0x71/0xe8

[530861.382743]  [<ffffffff810037fa>] ? call_softirq+0x1a/0x24

[530861.382751]  [<ffffffff81004d13>] ? do_softirq+0x2f/0x64

[530861.382758]  [<ffffffff810288f7>] ? irq_exit+0x34/0x6e

[530861.382764]  [<ffffffff81004943>] ? do_IRQ+0x88/0x9d

[530861.382771]  [<ffffffff81003128>] ? ret_from_intr+0x0/0x9

[530861.382776]  <EOI>  [<ffffffff814085ed>] ? netlbl_cipsov4_genl_policy+0xa5aad/0xd04c0

[530861.382789]  [<ffffffff814085ed>] ? netlbl_cipsov4_genl_policy+0xa5aad/0xd04c0

[530861.382799]  [<ffffffff81008841>] ? mwait_idle+0x4f/0x52

[530861.382805]  [<ffffffff81001407>] ? cpu_idle+0x2e/0x65

[530861.382811]  [<ffffffff81800b0d>] ? 0xffffffff81800b0d

[530861.382817]  [<ffffffff81800391>] ? 0xffffffff81800391

[530861.382822] Mem-Info:

[530861.382825] DMA per-cpu:

[530861.382829] CPU    0: hi:    0, btch:   1 usd:   0

[530861.382833] DMA32 per-cpu:

[530861.382837] CPU    0: hi:  186, btch:  31 usd: 155

[530861.382847] active_anon:97581 inactive_anon:64304 isolated_anon:0

[530861.382851]  active_file:209999 inactive_file:319407 isolated_file:0

[530861.382854]  unevictable:0 dirty:12901 writeback:161043 unstable:0

[530861.382857]  free:3671 slab_reclaimable:26292 slab_unreclaimable:44631

[530861.382861]  mapped:3238 shmem:118570 pagetables:563 bounce:0

[530861.382875] DMA free:12064kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:416kB active_file:384kB inactive_file:2892kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15312kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:148kB slab_unreclaimable:20kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no

[530861.382892] lowmem_reserve[]: 0 3013 3013 3013

[530861.382908] DMA32 free:2620kB min:7004kB low:8752kB high:10504kB active_anon:390324kB inactive_anon:256800kB active_file:839612kB inactive_file:1274736kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3085420kB mlocked:0kB dirty:51604kB writeback:644172kB mapped:12952kB shmem:474280kB slab_reclaimable:105020kB slab_unreclaimable:178504kB kernel_stack:1000kB pagetables:2252kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:96 all_unreclaimable? no

[530861.382926] lowmem_reserve[]: 0 0 0 0

[530861.382932] DMA: 6*4kB 7*8kB 7*16kB 3*32kB 4*64kB 4*128kB 5*256kB 1*512kB 1*1024kB 2*2048kB 1*4096kB = 12064kB

[530861.382949] DMA32: 553*4kB 3*8kB 0*16kB 0*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2620kB

[530861.382966] 647976 total pagecache pages

[530861.382970] 0 pages in swap cache

[530861.382974] Swap cache stats: add 0, delete 0, find 0/0

[530861.382978] Free swap  = 6291448kB

[530861.382982] Total swap = 6291448kB

[530861.383504] 786128 pages RAM

[530861.383504] 14573 pages reserved

[530861.383504] 526497 pages shared

[530861.383504] 246313 pages non-shared

[530861.383504] SLUB: Unable to allocate memory on node -1 (gfp=0x20)

[530861.383504]   cache: skbuff_head_cache, object size: 208, buffer size: 256, default order: 0, min order: 0

[530861.383504]   node 0: slabs: 80, objs: 1280, free: 0
```

... and the output of 'free'

```
rich@gorgon ~ $ free

             total       used       free     shared    buffers     cached

Mem:       3086220    3056988      29232          0      93692    2640392

-/+ buffers/cache:     322904    2763316

Swap:      6291448          0    6291448
```

The relevant 'exports' line

```
/mnt/myth       10.4.12.0/22(rw,async,no_subtree_check,all_squash,anonuid=500)
```

... and the fstab entry on the client pushing the massive amounts of data that seemed to cause the interface failure: 

```
gorgon:/mnt/myth        /mnt/myth       nfs             rw,rsize=8192,wsize=8192,noatime,intr,async,nfsvers=3,actimeo=0,tcp     0 0
```

As far as other information that may be relevant:

The server is running a hardware RAID-6 (via an Areca 1231ML PCI-E x8 card) with GPT partitioning (to enable >2TB partitions) with a cryptfs partition, with houses several LVM2 volumes, which in turn house mostly ext4 filesystems. The cryptfs (via dm-crypt) should be a 'free, no-cost' operation (CPU shouldn't be working any harder) due to the via padlock built into the CPU (pity openssl doesn't know how to use that feature  :Sad: ). In addition, the swap (which to date has never even been touched) is also running on the cryptfs->lvm2 stack.

On an interesting side note, I got lucky that another host, which was running a mythtv backend recording to this fileserver (using an h.264 stream), actually managed to dump its live recording fully intact once communication was restored by resetting the ethernet interface (it was down for more than 15 minutes) - kudos to whoever made that possible  :Smile: .

----------

## cetbaalix

Well I'm not running hardend kernel but I've had the same issue with gentoo-souces < 2.6.32-r11. 2.6.32-r11 (higher versions hopefully too) is working fine and 2.6.34-r1 was also working fine.

----------

## hanj

Since moving to hardened-sources-2.6.36-r6 I'm starting to get frequent page allocation failures (swapper, php, apache and kworker) . Prior to 2.6.36 I never saw these. I just compiled -r9 today and will load that tomorrow off hours to see if this helps. I am running grsec/PAX stuff as well. Any other ideas on where to look at the problem? I've been watching the apache logs closely to see what may be triggering this problem.. but nothing unusual.

hanji

----------

## roarinelk

 *hanj wrote:*   

> Since moving to hardened-sources-2.6.36-r6 I'm starting to get frequent page allocation failures (swapper, php, apache and kworker) . Prior to 2.6.36 I never saw these.

 

Please try and change the page allocator to "SLAB" instead of "SLUB".

On the other hand this could also be a leak introduced with the

hardened patches; certainly something is draining the emergency

pool: order-0 means 1 page, mask 0x20 GFP_ATOMIC (irq safe allocation

from the emergency pool))

----------

## hanj

 *roarinelk wrote:*   

> Please try and change the page allocator to "SLAB" instead of "SLUB".

 

Sounds good. I definitely had SLAB set. I'm recompiling now, but won't be able to reboot until tomorrow.

Here is a page allocation failure message.. might yield some additional information:

```
Feb 13 11:59:26 comp kernel: swapper: page allocation failure. order:1, mode:0x20

Feb 13 11:59:26 comp kernel: Pid: 0, comm: swapper Not tainted 2.6.36-hardened-r6 #1

Feb 13 11:59:26 comp kernel: Call Trace:

Feb 13 11:59:26 comp kernel: [<c11db1e2>] ? printk+0xf/0x15

Feb 13 11:59:26 comp kernel: [<c104847d>] __alloc_pages_nodemask+0x42a/0x456

Feb 13 11:59:26 comp kernel: [<c106065c>] cache_alloc_refill+0x230/0x3e6

Feb 13 11:59:26 comp kernel: [<c10608e8>] kmem_cache_alloc+0x3c/0x62

Feb 13 11:59:26 comp kernel: [<c116756d>] sk_prot_alloc+0x1b/0x51

Feb 13 11:59:26 comp kernel: [<c116760b>] sk_clone+0x18/0x1c7

Feb 13 11:59:26 comp kernel: [<c119c2e0>] inet_csk_clone+0x10/0x78

Feb 13 11:59:26 comp kernel: [<c11ae54c>] tcp_create_openreq_child+0x1b/0x3bb

Feb 13 11:59:26 comp kernel: [<c11ad1f7>] tcp_v4_syn_recv_sock+0x4e/0x15d

Feb 13 11:59:26 comp kernel: [<c11ae44e>] tcp_check_req+0x203/0x2e6

Feb 13 11:59:26 comp kernel: [<c11c050d>] ? ipv4_confirm+0x108/0x119

Feb 13 11:59:26 comp kernel: [<c11aca73>] tcp_v4_do_rcv+0x93/0x165

Feb 13 11:59:26 comp kernel: [<c11acf1f>] tcp_v4_rcv+0x3da/0x664

Feb 13 11:59:26 comp kernel: [<c1194c68>] ip_local_deliver_finish+0xd1/0x170

Feb 13 11:59:26 comp kernel: [<c1194d68>] ip_local_deliver+0x61/0x6a

Feb 13 11:59:26 comp kernel: [<c119494a>] ip_rcv_finish+0x23e/0x254

Feb 13 11:59:26 comp kernel: [<c1194b74>] ip_rcv+0x214/0x237

Feb 13 11:59:26 comp kernel: [<c1030c79>] ? sched_clock_local+0x17/0x105

Feb 13 11:59:26 comp kernel: [<c116fa73>] __netif_receive_skb+0x1e8/0x207

Feb 13 11:59:26 comp kernel: [<c116fba3>] netif_receive_skb+0x32/0x35

Feb 13 11:59:26 comp kernel: [<c114dd3f>] rhine_napipoll+0x2eb/0x41d

Feb 13 11:59:26 comp kernel: [<c116fcbe>] net_rx_action+0x43/0xe5

Feb 13 11:59:26 comp kernel: [<c101e7ef>] __do_softirq+0x56/0xc2

Feb 13 11:59:26 comp kernel: [<c101e880>] do_softirq+0x25/0x2a

Feb 13 11:59:26 comp kernel: [<c101e940>] irq_exit+0x28/0x2a

Feb 13 11:59:26 comp kernel: [<c1003b6c>] do_IRQ+0x60/0x75

Feb 13 11:59:26 comp kernel: [<c11e0000>] ? _etext+0x0/0xab000

Feb 13 11:59:26 comp kernel: [<c1002da9>] common_interrupt+0x29/0x30

Feb 13 11:59:26 comp kernel: [<c11e0000>] ? _etext+0x0/0xab000

Feb 13 11:59:26 comp kernel: [<c1006945>] ? default_idle+0x29/0x3e

Feb 13 11:59:26 comp kernel: [<c10017f5>] cpu_idle+0x1c/0x33

Feb 13 11:59:26 comp kernel: [<c11d6707>] rest_init+0x57/0x59

Feb 13 11:59:26 comp kernel: [<c128b800>] 0xc128b800

Feb 13 11:59:26 comp kernel: [<c128b099>] 0xc128b099

Feb 13 11:59:26 comp kernel: [<c11e0000>] ? _etext+0x0/0xab000

Feb 13 11:59:26 comp kernel: Mem-Info:

Feb 13 11:59:26 comp kernel: DMA per-cpu:

Feb 13 11:59:26 comp kernel: CPU    0: hi:    0, btch:   1 usd:   0

Feb 13 11:59:26 comp kernel: Normal per-cpu:

Feb 13 11:59:26 comp kernel: CPU    0: hi:  186, btch:  31 usd:  69

Feb 13 11:59:26 comp kernel: active_anon:9544 inactive_anon:19489 isolated_anon:0

Feb 13 11:59:26 comp kernel: active_file:33129 inactive_file:53388 isolated_file:0

Feb 13 11:59:26 comp kernel: unevictable:0 dirty:81 writeback:0 unstable:0

Feb 13 11:59:26 comp kernel: free:1566 slab_reclaimable:2754 slab_unreclaimable:1989

Feb 13 11:59:26 comp kernel: mapped:3896 shmem:80 pagetables:1030 bounce:0

Feb 13 11:59:26 comp kernel: DMA free:2012kB min:88kB low:108kB high:132kB active_anon:464kB inactive_anon:804kB active_file:2500kB inactive_file:9696kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15804kB mlocked:0kB dirty:0kB writeback:0kB mapped:48kB shmem:0kB slab_reclaimable:284kB slab_unreclaimable:120kB kernel_stack:48kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no

Feb 13 11:59:26 comp kernel: lowmem_reserve[]: 0 476 476 476

Feb 13 11:59:26 comp kernel: Normal free:4252kB min:2744kB low:3428kB high:4116kB active_anon:37712kB inactive_anon:77152kB active_file:130016kB inactive_file:203856kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:487616kB mlocked:0kB dirty:324kB writeback:0kB mapped:15536kB shmem:320kB slab_reclaimable:10732kB slab_unreclaimable:7836kB kernel_stack:944kB pagetables:4120kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no

Feb 13 11:59:26 comp kernel: lowmem_reserve[]: 0 0 0 0

Feb 13 11:59:26 comp kernel: DMA: 503*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2012kB

Feb 13 11:59:26 comp kernel: Normal: 1051*4kB 6*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 4252kB

Feb 13 11:59:26 comp kernel: 91509 total pagecache pages

Feb 13 11:59:26 comp kernel: 4912 pages in swap cache

Feb 13 11:59:26 comp kernel: Swap cache stats: add 2855972, delete 2851060, find 125202126/125475187

Feb 13 11:59:26 comp kernel: Free swap  = 900556kB

Feb 13 11:59:26 comp kernel: Total swap = 979960kB

Feb 13 11:59:26 comp kernel: 126944 pages RAM

Feb 13 11:59:26 comp kernel: 0 pages HighMem

Feb 13 11:59:26 comp kernel: 1908 pages reserved

Feb 13 11:59:26 comp kernel: 173947 pages shared

Feb 13 11:59:26 comp kernel: 93937 pages non-shared
```

Thanks for the reply!

hanji

----------

## KShots

Odd... I'm running 2.6.36-hardened-r9 now and not seeing these problems any more. I think they stopped when I hit 2.6.34, but I'm not positive.

Also, I'm no longer running NFSv3, I moved over to NFSv4 with kerberos auth/privacy. I'm not sure if that's what fixed the issue or not... I simply haven't seen this since probably a couple weeks after my first posting.

I just started fiddling around with grsec recently, but haven't actually turned it on yet (It's in kernel, userland tools installed).

----------

## guid0

I am still hitting this bug.

May I ask what file systems you have been using? Im currently using a 10G mlx4_en card against a 24port 3ware-LSI RAID6 box and XFS. Tried so many things and the errors i get seem to indicate network but i suspect XFS to be the issue (again).

tnx,

guid0

----------

## nicke#

Had the same problem on my hardened firewall.

Om my system the problem was resolved with the following kernel tweak:

```

echo "8192" > /proc/sys/vm/min_free_kbytes

```

or

```

vm.min_free_kbytes = 8192  

```

for sysctl

Before you start tweaking note your original value, then double it and see if the error goes away. Else increase it a second time.

----------

## guid0

 *nicke# wrote:*   

> Had the same problem on my hardened firewall.
> 
> Om my system the problem was resolved with the following kernel tweak:
> 
> ```
> ...

 

I am already at 

```
echo 524288 > /proc/sys/vm/min_free_kbytes
```

which did improve but did not fix.

----------

## scyld

The same problem here, but with :

```
3.0.4-hardened-r5-kret #1 SMP Thu Dec 8 09:21:45 CET 2011 x86_64 Intel(R) Xeon(R) CPU E5420 @ 2.50GHz GenuineIntel GNU/Linux
```

Moved from 2.6.28-hardened-xx and now from time to time goes that:

```
apache2: page allocation failure: order:1, mode:0x20

Pid: 3292, comm: apache2 Not tainted 3.0.4-hardened-r5-kret #1

Call Trace:

 <IRQ>  [<ffffffff8108a124>] ? 0xffffffff8108a124

 [<ffffffff8108d258>] ? 0xffffffff8108d258

 [<ffffffff8105ff2d>] ? 0xffffffff8105ff2d

 [<ffffffff81312230>] ? 0xffffffff81312230

 [<ffffffff810b78ed>] ? 0xffffffff810b78ed

 [<ffffffff810b7ce1>] ? 0xffffffff810b7ce1

 [<ffffffff812c52ac>] ? 0xffffffff812c52ac

 [<ffffffff812c5450>] ? 0xffffffff812c5450

 [<ffffffff813197bb>] ? 0xffffffff813197bb

 [<ffffffff81332491>] ? 0xffffffff81332491

 [<ffffffff8132f537>] ? 0xffffffff8132f537

 [<ffffffff813321a5>] ? 0xffffffff813321a5

 [<ffffffff8132f3c1>] ? 0xffffffff8132f3c1

 [<ffffffff8130e3e0>] ? 0xffffffff8130e3e0

 [<ffffffff813319dd>] ? 0xffffffff813319dd

 [<ffffffff8130e473>] ? 0xffffffff8130e473

 [<ffffffff812d1543>] ? 0xffffffff812d1543

 [<ffffffff812d5634>] ? 0xffffffff812d5634

 [<ffffffff812d579e>] ? 0xffffffff812d579e

 [<ffffffff811fff64>] ? 0xffffffff811fff64

 [<ffffffff8103c3cd>] ? 0xffffffff8103c3cd

 [<ffffffff8136e80c>] ? 0xffffffff8136e80c

 <EOI>  [<ffffffff81004505>] ? 0xffffffff81004505

 [<ffffffff8103c27c>] ? 0xffffffff8103c27c

 [<ffffffff813409bd>] ? 0xffffffff813409bd

 [<ffffffff8136cd29>] ? 0xffffffff8136cd29

 [<ffffffff810fc770>] ? 0xffffffff810fc770

 [<ffffffff812c18b5>] ? 0xffffffff812c18b5

 [<ffffffff810d3098>] ? 0xffffffff810d3098

 [<ffffffff810bee01>] ? 0xffffffff810bee01

 [<ffffffff8136d779>] ? 0xffffffff8136d779

 [<ffffffff8136d717>] ? 0xffffffff8136d717

Mem-Info:

DMA per-cpu:

CPU    0: hi:    0, btch:   1 usd:   0

CPU    1: hi:    0, btch:   1 usd:   0

CPU    2: hi:    0, btch:   1 usd:   0

CPU    3: hi:    0, btch:   1 usd:   0

CPU    4: hi:    0, btch:   1 usd:   0

CPU    5: hi:    0, btch:   1 usd:   0

CPU    6: hi:    0, btch:   1 usd:   0

CPU    7: hi:    0, btch:   1 usd:   0

DMA32 per-cpu:

CPU    0: hi:  186, btch:  31 usd:  54

CPU    1: hi:  186, btch:  31 usd:  63

CPU    2: hi:  186, btch:  31 usd: 126

CPU    3: hi:  186, btch:  31 usd:  48

CPU    4: hi:  186, btch:  31 usd: 164

CPU    5: hi:  186, btch:  31 usd:  17

CPU    6: hi:  186, btch:  31 usd: 151

CPU    7: hi:  186, btch:  31 usd: 144

Normal per-cpu:

CPU    0: hi:  186, btch:  31 usd:  83

CPU    1: hi:  186, btch:  31 usd:  63

CPU    2: hi:  186, btch:  31 usd: 159

CPU    3: hi:  186, btch:  31 usd: 165

CPU    4: hi:  186, btch:  31 usd: 159

CPU    5: hi:  186, btch:  31 usd: 168

CPU    6: hi:  186, btch:  31 usd:  60

CPU    7: hi:  186, btch:  31 usd: 130

active_anon:1550288 inactive_anon:224178 isolated_anon:0

 active_file:623464 inactive_file:559084 isolated_file:0

 unevictable:5 dirty:901 writeback:0 unstable:0

 free:60043 slab_reclaimable:885054 slab_unreclaimable:53485

 mapped:172746 shmem:271671 pagetables:141049 bounce:0

DMA free:15916kB min:128kB low:160kB high:192kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15692kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes

lowmem_reserve[]: 0 3255 16133 16133

DMA32 free:89756kB min:27248kB low:34060kB high:40872kB active_anon:1272172kB inactive_anon:403816kB active_file:270292kB inactive_file:288388kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3333728kB mlocked:0kB dirty:1208kB writeback:0kB mapped:10980kB shmem:101144kB slab_reclaimable:788400kB slab_unreclaimable:40688kB kernel_stack:3944kB pagetables:144508kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no

lowmem_reserve[]: 0 0 12877 12877

Normal free:134500kB min:107788kB low:134732kB high:161680kB active_anon:4928980kB inactive_anon:492896kB active_file:2223564kB inactive_file:1947948kB unevictable:20kB isolated(anon):0kB isolated(file):0kB present:13186560kB mlocked:20kB dirty:2396kB writeback:0kB mapped:680004kB shmem:985540kB slab_reclaimable:2751816kB slab_unreclaimable:173252kB kernel_stack:1872kB pagetables:419688kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:33 all_unreclaimable? no

lowmem_reserve[]: 0 0 0 0

DMA: 1*4kB 1*8kB 0*16kB 1*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15916kB

DMA32: 22286*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 89144kB

Normal: 33749*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 134996kB

1468945 total pagecache pages

14601 pages in swap cache

Swap cache stats: add 274979, delete 260378, find 706202/718462

Free swap  = 219976kB

Total swap = 996024kB

4194288 pages RAM

83203 pages reserved

4350193 pages shared

3667906 pages non-shared

apache2: page allocation failure: order:1, mode:0x20

Pid: 566, comm: apache2 Not tainted 3.0.4-hardened-r5-kret #1

Call Trace:

 <IRQ>  [<ffffffff8108a124>] ? 0xffffffff8108a124

 [<ffffffff8108d258>] ? 0xffffffff8108d258

 [<ffffffff8105ff2d>] ? 0xffffffff8105ff2d

 [<ffffffff81312230>] ? 0xffffffff81312230

 [<ffffffff810b78ed>] ? 0xffffffff810b78ed

 [<ffffffff810b7ce1>] ? 0xffffffff810b7ce1

 [<ffffffff812c52ac>] ? 0xffffffff812c52ac

 [<ffffffff812c5450>] ? 0xffffffff812c5450

 [<ffffffff813197bb>] ? 0xffffffff813197bb

 [<ffffffff81332491>] ? 0xffffffff81332491

 [<ffffffff8132f537>] ? 0xffffffff8132f537

 [<ffffffff813321a5>] ? 0xffffffff813321a5

 [<ffffffff8132f3c1>] ? 0xffffffff8132f3c1

 [<ffffffff8130e3e0>] ? 0xffffffff8130e3e0

 [<ffffffff813319dd>] ? 0xffffffff813319dd

 [<ffffffff8130e473>] ? 0xffffffff8130e473

 [<ffffffff812d1543>] ? 0xffffffff812d1543

 [<ffffffff81043f3c>] ? 0xffffffff81043f3c

 [<ffffffff812d5634>] ? 0xffffffff812d5634

 [<ffffffff812d579e>] ? 0xffffffff812d579e

 [<ffffffff811fff64>] ? 0xffffffff811fff64

 [<ffffffff8103c3cd>] ? 0xffffffff8103c3cd

 [<ffffffff8136e80c>] ? 0xffffffff8136e80c

 <EOI>  [<ffffffff81004505>] ? 0xffffffff81004505

 [<ffffffff8103c27c>] ? 0xffffffff8103c27c

 [<ffffffff813409bd>] ? 0xffffffff813409bd

 [<ffffffff812c18b5>] ? 0xffffffff812c18b5

 [<ffffffff810d3098>] ? 0xffffffff810d3098

 [<ffffffff8136d779>] ? 0xffffffff8136d779

 [<ffffffff8136d717>] ? 0xffffffff8136d717

Mem-Info:

DMA per-cpu:

CPU    0: hi:    0, btch:   1 usd:   0

CPU    1: hi:    0, btch:   1 usd:   0

CPU    2: hi:    0, btch:   1 usd:   0

CPU    3: hi:    0, btch:   1 usd:   0

CPU    4: hi:    0, btch:   1 usd:   0

CPU    5: hi:    0, btch:   1 usd:   0

CPU    6: hi:    0, btch:   1 usd:   0

CPU    7: hi:    0, btch:   1 usd:   0

DMA32 per-cpu:

CPU    0: hi:  186, btch:  31 usd:   0

CPU    1: hi:  186, btch:  31 usd:   0

CPU    2: hi:  186, btch:  31 usd:   1

CPU    3: hi:  186, btch:  31 usd:   1

CPU    4: hi:  186, btch:  31 usd:   0

CPU    5: hi:  186, btch:  31 usd:  17

CPU    6: hi:  186, btch:  31 usd:   0

CPU    7: hi:  186, btch:  31 usd:   0

Normal per-cpu:

CPU    0: hi:  186, btch:  31 usd:  44

CPU    1: hi:  186, btch:  31 usd:  59

CPU    2: hi:  186, btch:  31 usd:  21

CPU    3: hi:  186, btch:  31 usd:  23

CPU    4: hi:  186, btch:  31 usd:   0

CPU    5: hi:  186, btch:  31 usd: 102

CPU    6: hi:  186, btch:  31 usd:  55

CPU    7: hi:  186, btch:  31 usd:   1

active_anon:1549858 inactive_anon:228883 isolated_anon:0

 active_file:618325 inactive_file:559025 isolated_file:0

 unevictable:5 dirty:962 writeback:0 unstable:0

 free:62559 slab_reclaimable:885143 slab_unreclaimable:53487

 mapped:172812 shmem:276562 pagetables:141053 bounce:0

DMA free:15916kB min:128kB low:160kB high:192kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15692kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes

lowmem_reserve[]: 0 3255 16133 16133

DMA32 free:93772kB min:27248kB low:34060kB high:40872kB active_anon:1269872kB inactive_anon:407824kB active_file:270600kB inactive_file:285980kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3333728kB mlocked:0kB dirty:1292kB writeback:0kB mapped:11520kB shmem:105632kB slab_reclaimable:788708kB slab_unreclaimable:40688kB kernel_stack:3872kB pagetables:144620kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:17 all_unreclaimable? no

lowmem_reserve[]: 0 0 12877 12877

Normal free:140548kB min:107788kB low:134732kB high:161680kB active_anon:4929560kB inactive_anon:507708kB active_file:2202700kB inactive_file:1950120kB unevictable:20kB isolated(anon):0kB isolated(file):0kB present:13186560kB mlocked:20kB dirty:2556kB writeback:0kB mapped:679728kB shmem:1000616kB slab_reclaimable:2751864kB slab_unreclaimable:173260kB kernel_stack:1872kB pagetables:419592kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:212 all_unreclaimable? no

lowmem_reserve[]: 0 0 0 0

DMA: 1*4kB 1*8kB 0*16kB 1*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15916kB

DMA32: 23016*4kB 96*8kB 30*16kB 13*32kB 3*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 94048kB

Normal: 34876*4kB 68*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 140064kB

1468592 total pagecache pages

14698 pages in swap cache

Swap cache stats: add 275231, delete 260533, find 706206/718468

Free swap  = 219040kB

Total swap = 996024kB

4194288 pages RAM

83203 pages reserved

4350425 pages shared

3667161 pages non-shared

apache2: page allocation failure: order:1, mode:0x20

Pid: 566, comm: apache2 Not tainted 3.0.4-hardened-r5-kret #1

Call Trace:

 <IRQ>  [<ffffffff8108a124>] ? 0xffffffff8108a124

 [<ffffffff8108d258>] ? 0xffffffff8108d258

 [<ffffffff810b78ed>] ? 0xffffffff810b78ed

 [<ffffffff810b7ce1>] ? 0xffffffff810b7ce1

 [<ffffffff812c52ac>] ? 0xffffffff812c52ac

 [<ffffffff812c5450>] ? 0xffffffff812c5450

 [<ffffffff813197bb>] ? 0xffffffff813197bb

 [<ffffffff81332491>] ? 0xffffffff81332491

 [<ffffffff8132f537>] ? 0xffffffff8132f537

 [<ffffffff813321a5>] ? 0xffffffff813321a5

 [<ffffffff8132f3c1>] ? 0xffffffff8132f3c1

 [<ffffffff8130e3e0>] ? 0xffffffff8130e3e0

 [<ffffffff813319dd>] ? 0xffffffff813319dd

 [<ffffffff8130e473>] ? 0xffffffff8130e473

 [<ffffffff812d1543>] ? 0xffffffff812d1543

 [<ffffffff812d32c0>] ? 0xffffffff812d32c0

 [<ffffffff812d3aa3>] ? 0xffffffff812d3aa3

 [<ffffffff812d3308>] ? 0xffffffff812d3308

 [<ffffffff81270489>] ? 0xffffffff81270489

 [<ffffffff812716e8>] ? 0xffffffff812716e8

 [<ffffffff812d579e>] ? 0xffffffff812d579e

 [<ffffffff8103c3cd>] ? 0xffffffff8103c3cd

 [<ffffffff8136e80c>] ? 0xffffffff8136e80c

 <EOI>  [<ffffffff81004505>] ? 0xffffffff81004505

 [<ffffffff8103c27c>] ? 0xffffffff8103c27c

 [<ffffffff813409bd>] ? 0xffffffff813409bd

 [<ffffffff812c18b5>] ? 0xffffffff812c18b5

 [<ffffffff810d3098>] ? 0xffffffff810d3098

 [<ffffffff8136d779>] ? 0xffffffff8136d779

 [<ffffffff8136d717>] ? 0xffffffff8136d717

Mem-Info:

DMA per-cpu:

CPU    0: hi:    0, btch:   1 usd:   0

CPU    1: hi:    0, btch:   1 usd:   0

CPU    2: hi:    0, btch:   1 usd:   0

CPU    3: hi:    0, btch:   1 usd:   0

CPU    4: hi:    0, btch:   1 usd:   0

CPU    5: hi:    0, btch:   1 usd:   0

CPU    6: hi:    0, btch:   1 usd:   0

CPU    7: hi:    0, btch:   1 usd:   0

DMA32 per-cpu:

CPU    0: hi:  186, btch:  31 usd:   0

CPU    1: hi:  186, btch:  31 usd:  12

CPU    2: hi:  186, btch:  31 usd:   0

CPU    3: hi:  186, btch:  31 usd:   0

CPU    4: hi:  186, btch:  31 usd:  20

CPU    5: hi:  186, btch:  31 usd:   0

CPU    6: hi:  186, btch:  31 usd:   0

CPU    7: hi:  186, btch:  31 usd:   0

Normal per-cpu:

CPU    0: hi:  186, btch:  31 usd:  23

CPU    1: hi:  186, btch:  31 usd:  29

CPU    2: hi:  186, btch:  31 usd:   0

CPU    3: hi:  186, btch:  31 usd:   0

CPU    4: hi:  186, btch:  31 usd: 149

CPU    5: hi:  186, btch:  31 usd:   0

CPU    6: hi:  186, btch:  31 usd:   0

CPU    7: hi:  186, btch:  31 usd:  30

active_anon:1549858 inactive_anon:228883 isolated_anon:0

 active_file:618183 inactive_file:559025 isolated_file:0

 unevictable:5 dirty:962 writeback:0 unstable:0

 free:62661 slab_reclaimable:885143 slab_unreclaimable:53487

 mapped:172812 shmem:276562 pagetables:141053 bounce:0

DMA free:15916kB min:128kB low:160kB high:192kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15692kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes

lowmem_reserve[]: 0 3255 16133 16133

DMA32 free:93772kB min:27248kB low:34060kB high:40872kB active_anon:1269872kB inactive_anon:407824kB active_file:270600kB inactive_file:285980kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3333728kB mlocked:0kB dirty:1292kB writeback:0kB mapped:11520kB shmem:105632kB slab_reclaimable:788708kB slab_unreclaimable:40688kB kernel_stack:3872kB pagetables:144620kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:20 all_unreclaimable? no

lowmem_reserve[]: 0 0 12877 12877

Normal free:140956kB min:107788kB low:134732kB high:161680kB active_anon:4929560kB inactive_anon:507708kB active_file:2202132kB inactive_file:1950120kB unevictable:20kB isolated(anon):0kB isolated(file):0kB present:13186560kB mlocked:20kB dirty:2556kB writeback:0kB mapped:679728kB shmem:1000616kB slab_reclaimable:2751864kB slab_unreclaimable:173260kB kernel_stack:1872kB pagetables:419592kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:180 all_unreclaimable? no

lowmem_reserve[]: 0 0 0 0

DMA: 1*4kB 1*8kB 0*16kB 1*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15916kB

DMA32: 23022*4kB 97*8kB 38*16kB 13*32kB 3*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 94208kB

Normal: 35026*4kB 88*8kB 7*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 140920kB

1468398 total pagecache pages

14680 pages in swap cache

Swap cache stats: add 275231, delete 260551, find 706206/718468

Free swap  = 219040kB

Total swap = 996024kB

4194288 pages RAM

83203 pages reserved

4350671 pages shared

3666469 pages non-shared

apache2: page allocation failure: order:1, mode:0x20

Pid: 566, comm: apache2 Not tainted 3.0.4-hardened-r5-kret #1

Call Trace:

 <IRQ>  [<ffffffff8108a124>] ? 0xffffffff8108a124

 [<ffffffff8102bcf8>] ? 0xffffffff8102bcf8

 [<ffffffff8108d258>] ? 0xffffffff8108d258

 [<ffffffff810b78ed>] ? 0xffffffff810b78ed

 [<ffffffff810b7ce1>] ? 0xffffffff810b7ce1

 [<ffffffff812c52ac>] ? 0xffffffff812c52ac

 [<ffffffff812c5450>] ? 0xffffffff812c5450

 [<ffffffff813197bb>] ? 0xffffffff813197bb

 [<ffffffff81332491>] ? 0xffffffff81332491

 [<ffffffff8132f537>] ? 0xffffffff8132f537

 [<ffffffff813321a5>] ? 0xffffffff813321a5

 [<ffffffff8132f3c1>] ? 0xffffffff8132f3c1

 [<ffffffff8130e3e0>] ? 0xffffffff8130e3e0

 [<ffffffff813319dd>] ? 0xffffffff813319dd

 [<ffffffff8131ada8>] ? 0xffffffff8131ada8

 [<ffffffff8130e473>] ? 0xffffffff8130e473

 [<ffffffff812d1543>] ? 0xffffffff812d1543

 [<ffffffff812d32c0>] ? 0xffffffff812d32c0

 [<ffffffff812d3aa3>] ? 0xffffffff812d3aa3

 [<ffffffff812d3308>] ? 0xffffffff812d3308

 [<ffffffff81270489>] ? 0xffffffff81270489

 [<ffffffff812716e8>] ? 0xffffffff812716e8

 [<ffffffff812d579e>] ? 0xffffffff812d579e

 [<ffffffff8103c3cd>] ? 0xffffffff8103c3cd

 [<ffffffff8136e80c>] ? 0xffffffff8136e80c

 <EOI>  [<ffffffff81004505>] ? 0xffffffff81004505

 [<ffffffff8103c27c>] ? 0xffffffff8103c27c

 [<ffffffff813409bd>] ? 0xffffffff813409bd

 [<ffffffff812c18b5>] ? 0xffffffff812c18b5

 [<ffffffff810d3098>] ? 0xffffffff810d3098

 [<ffffffff8136d779>] ? 0xffffffff8136d779

 [<ffffffff8136d717>] ? 0xffffffff8136d717

Mem-Info:

DMA per-cpu:

CPU    0: hi:    0, btch:   1 usd:   0

CPU    1: hi:    0, btch:   1 usd:   0

CPU    2: hi:    0, btch:   1 usd:   0

CPU    3: hi:    0, btch:   1 usd:   0

CPU    4: hi:    0, btch:   1 usd:   0

CPU    5: hi:    0, btch:   1 usd:   0

CPU    6: hi:    0, btch:   1 usd:   0

CPU    7: hi:    0, btch:   1 usd:   0

DMA32 per-cpu:

CPU    0: hi:  186, btch:  31 usd:   0

CPU    1: hi:  186, btch:  31 usd:   0

CPU    2: hi:  186, btch:  31 usd:   0

CPU    3: hi:  186, btch:  31 usd:  10

CPU    4: hi:  186, btch:  31 usd:   0

CPU    5: hi:  186, btch:  31 usd:   0

CPU    6: hi:  186, btch:  31 usd:   0

CPU    7: hi:  186, btch:  31 usd:   0

Normal per-cpu:

CPU    0: hi:  186, btch:  31 usd:  26

CPU    1: hi:  186, btch:  31 usd:   0

CPU    2: hi:  186, btch:  31 usd:   0

CPU    3: hi:  186, btch:  31 usd:  97

CPU    4: hi:  186, btch:  31 usd:  32

CPU    5: hi:  186, btch:  31 usd:  19

CPU    6: hi:  186, btch:  31 usd:  38

CPU    7: hi:  186, btch:  31 usd:  30

active_anon:1549858 inactive_anon:228883 isolated_anon:0

 active_file:616974 inactive_file:558970 isolated_file:0

 unevictable:5 dirty:1059 writeback:0 unstable:0

 free:63884 slab_reclaimable:885143 slab_unreclaimable:53487

 mapped:172812 shmem:276562 pagetables:141053 bounce:0

DMA free:15916kB min:128kB low:160kB high:192kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15692kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes

lowmem_reserve[]: 0 3255 16133 16133

DMA32 free:94676kB min:27248kB low:34060kB high:40872kB active_anon:1269872kB inactive_anon:407824kB active_file:270600kB inactive_file:285472kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3333728kB mlocked:0kB dirty:1292kB writeback:0kB mapped:11520kB shmem:105632kB slab_reclaimable:788708kB slab_unreclaimable:40688kB kernel_stack:3872kB pagetables:144620kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:17 all_unreclaimable? no

lowmem_reserve[]: 0 0 12877 12877

Normal free:144944kB min:107788kB low:134732kB high:161680kB active_anon:4929560kB inactive_anon:507708kB active_file:2197296kB inactive_file:1950408kB unevictable:20kB isolated(anon):0kB isolated(file):0kB present:13186560kB mlocked:20kB dirty:2944kB writeback:0kB mapped:679728kB shmem:1000616kB slab_reclaimable:2751864kB slab_unreclaimable:173260kB kernel_stack:1872kB pagetables:419592kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:286 all_unreclaimable? no

lowmem_reserve[]: 0 0 0 0

DMA: 1*4kB 1*8kB 0*16kB 1*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15916kB

DMA32: 22938*4kB 81*8kB 54*16kB 21*32kB 6*64kB 4*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 95088kB

Normal: 35080*4kB 61*8kB 13*16kB 3*32kB 4*64kB 4*128kB 4*256kB 0*512kB 2*1024kB 0*2048kB 0*4096kB = 144952kB

1467143 total pagecache pages

14615 pages in swap cache

Swap cache stats: add 275245, delete 260630, find 706206/718468

Free swap  = 218984kB

Total swap = 996024kB

4194288 pages RAM

83203 pages reserved

4350317 pages shared

3664156 pages non-shared

apache2: page allocation failure: order:1, mode:0x20

Pid: 566, comm: apache2 Not tainted 3.0.4-hardened-r5-kret #1

Call Trace:

 <IRQ>  [<ffffffff8108a124>] ? 0xffffffff8108a124

 [<ffffffff8108d258>] ? 0xffffffff8108d258

 [<ffffffff812fdcc9>] ? 0xffffffff812fdcc9

 [<ffffffff811be630>] ? 0xffffffff811be630

 [<ffffffff810b78ed>] ? 0xffffffff810b78ed

 [<ffffffff810b7ce1>] ? 0xffffffff810b7ce1

 [<ffffffff812c52ac>] ? 0xffffffff812c52ac

 [<ffffffff812c5450>] ? 0xffffffff812c5450

 [<ffffffff813197bb>] ? 0xffffffff813197bb

 [<ffffffff81332491>] ? 0xffffffff81332491

 [<ffffffff8132f537>] ? 0xffffffff8132f537

 [<ffffffff813321a5>] ? 0xffffffff813321a5

 [<ffffffff8132f3c1>] ? 0xffffffff8132f3c1

 [<ffffffff8130e3e0>] ? 0xffffffff8130e3e0

 [<ffffffff813319dd>] ? 0xffffffff813319dd

 [<ffffffff8130e473>] ? 0xffffffff8130e473

 [<ffffffff812d1543>] ? 0xffffffff812d1543

 [<ffffffff812d32c0>] ? 0xffffffff812d32c0

 [<ffffffff812d3aa3>] ? 0xffffffff812d3aa3

 [<ffffffff812d3308>] ? 0xffffffff812d3308

 [<ffffffff81270489>] ? 0xffffffff81270489

 [<ffffffff812716e8>] ? 0xffffffff812716e8

 [<ffffffff812d579e>] ? 0xffffffff812d579e

 [<ffffffff8103c3cd>] ? 0xffffffff8103c3cd

 [<ffffffff8136e80c>] ? 0xffffffff8136e80c

 <EOI>  [<ffffffff81004505>] ? 0xffffffff81004505

 [<ffffffff8103c27c>] ? 0xffffffff8103c27c

 [<ffffffff813409bd>] ? 0xffffffff813409bd

 [<ffffffff812c18b5>] ? 0xffffffff812c18b5

 [<ffffffff810d3098>] ? 0xffffffff810d3098

 [<ffffffff8136d779>] ? 0xffffffff8136d779

 [<ffffffff8136d717>] ? 0xffffffff8136d717

Mem-Info:

DMA per-cpu:

CPU    0: hi:    0, btch:   1 usd:   0

CPU    1: hi:    0, btch:   1 usd:   0

CPU    2: hi:    0, btch:   1 usd:   0

CPU    3: hi:    0, btch:   1 usd:   0

CPU    4: hi:    0, btch:   1 usd:   0

CPU    5: hi:    0, btch:   1 usd:   0

CPU    6: hi:    0, btch:   1 usd:   0

CPU    7: hi:    0, btch:   1 usd:   0

DMA32 per-cpu:

CPU    0: hi:  186, btch:  31 usd:   0

CPU    1: hi:  186, btch:  31 usd:   9

CPU    2: hi:  186, btch:  31 usd:   0

CPU    3: hi:  186, btch:  31 usd:   0

CPU    4: hi:  186, btch:  31 usd:   0

CPU    5: hi:  186, btch:  31 usd:  14

CPU    6: hi:  186, btch:  31 usd:   0

CPU    7: hi:  186, btch:  31 usd:   0

Normal per-cpu:

CPU    0: hi:  186, btch:  31 usd:   0

CPU    1: hi:  186, btch:  31 usd:  19

CPU    2: hi:  186, btch:  31 usd:   0

CPU    3: hi:  186, btch:  31 usd:  26

CPU    4: hi:  186, btch:  31 usd:  93

CPU    5: hi:  186, btch:  31 usd: 125

CPU    6: hi:  186, btch:  31 usd:   4

CPU    7: hi:  186, btch:  31 usd:   0

active_anon:1549858 inactive_anon:228882 isolated_anon:0

 active_file:613433 inactive_file:558897 isolated_file:0

 unevictable:5 dirty:1059 writeback:97 unstable:0

 free:67428 slab_reclaimable:885143 slab_unreclaimable:53487

 mapped:172812 shmem:276562 pagetables:141053 bounce:0

DMA free:15916kB min:128kB low:160kB high:192kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15692kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes

lowmem_reserve[]: 0 3255 16133 16133

DMA32 free:96580kB min:27248kB low:34060kB high:40872kB active_anon:1269872kB inactive_anon:407824kB active_file:270600kB inactive_file:283836kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3333728kB mlocked:0kB dirty:1292kB writeback:0kB mapped:11520kB shmem:105632kB slab_reclaimable:788708kB slab_unreclaimable:40688kB kernel_stack:3872kB pagetables:144620kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:18 all_unreclaimable? no

lowmem_reserve[]: 0 0 12877 12877

Normal free:157216kB min:107788kB low:134732kB high:161680kB active_anon:4929560kB inactive_anon:507704kB active_file:2183132kB inactive_file:1951752kB unevictable:20kB isolated(anon):0kB isolated(file):0kB present:13186560kB mlocked:20kB dirty:2944kB writeback:388kB mapped:679728kB shmem:1000616kB slab_reclaimable:2751864kB slab_unreclaimable:173260kB kernel_stack:1872kB pagetables:419592kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:387 all_unreclaimable? no

lowmem_reserve[]: 0 0 0 0

DMA: 1*4kB 1*8kB 0*16kB 1*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15916kB

DMA32: 23032*4kB 112*8kB 59*16kB 43*32kB 12*64kB 6*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 97136kB

Normal: 36855*4kB 315*8kB 40*16kB 13*32kB 12*64kB 10*128kB 8*256kB 1*512kB 2*1024kB 0*2048kB 0*4096kB = 157652kB

1463663 total pagecache pages

14877 pages in swap cache

Swap cache stats: add 275510, delete 260633, find 706206/718468

Free swap  = 217924kB

Total swap = 996024kB

4194288 pages RAM

83203 pages reserved

4349938 pages shared

3661987 pages non-shared

apache2: page allocation failure: order:1, mode:0x20

Pid: 566, comm: apache2 Not tainted 3.0.4-hardened-r5-kret #1

Call Trace:

 <IRQ>  [<ffffffff8108a124>] ? 0xffffffff8108a124

 [<ffffffff8108d258>] ? 0xffffffff8108d258

 [<ffffffff810b78ed>] ? 0xffffffff810b78ed

 [<ffffffff810b7ce1>] ? 0xffffffff810b7ce1

 [<ffffffff812c52ac>] ? 0xffffffff812c52ac

 [<ffffffff812c5450>] ? 0xffffffff812c5450

 [<ffffffff813197bb>] ? 0xffffffff813197bb

 [<ffffffff81332491>] ? 0xffffffff81332491

 [<ffffffff8132f537>] ? 0xffffffff8132f537

 [<ffffffff813321a5>] ? 0xffffffff813321a5

 [<ffffffff8132f3c1>] ? 0xffffffff8132f3c1

 [<ffffffff8130e3e0>] ? 0xffffffff8130e3e0

 [<ffffffff813319dd>] ? 0xffffffff813319dd

 [<ffffffff8130e473>] ? 0xffffffff8130e473

 [<ffffffff812d1543>] ? 0xffffffff812d1543

 [<ffffffff812d32c0>] ? 0xffffffff812d32c0

 [<ffffffff812d3aa3>] ? 0xffffffff812d3aa3

 [<ffffffff812d3308>] ? 0xffffffff812d3308

 [<ffffffff81270489>] ? 0xffffffff81270489

 [<ffffffff812716e8>] ? 0xffffffff812716e8

 [<ffffffff812d579e>] ? 0xffffffff812d579e

 [<ffffffff8103c3cd>] ? 0xffffffff8103c3cd

 [<ffffffff8136e80c>] ? 0xffffffff8136e80c

 <EOI>  [<ffffffff81004505>] ? 0xffffffff81004505

 [<ffffffff8103c27c>] ? 0xffffffff8103c27c

 [<ffffffff813409bd>] ? 0xffffffff813409bd

 [<ffffffff812c18b5>] ? 0xffffffff812c18b5

 [<ffffffff810d3098>] ? 0xffffffff810d3098

 [<ffffffff8136d779>] ? 0xffffffff8136d779

 [<ffffffff8136d717>] ? 0xffffffff8136d717

Mem-Info:

DMA per-cpu:

CPU    0: hi:    0, btch:   1 usd:   0

CPU    1: hi:    0, btch:   1 usd:   0

CPU    2: hi:    0, btch:   1 usd:   0

CPU    3: hi:    0, btch:   1 usd:   0

CPU    4: hi:    0, btch:   1 usd:   0

CPU    5: hi:    0, btch:   1 usd:   0

CPU    6: hi:    0, btch:   1 usd:   0

CPU    7: hi:    0, btch:   1 usd:   0

DMA32 per-cpu:

CPU    0: hi:  186, btch:  31 usd:  15

CPU    1: hi:  186, btch:  31 usd:   6

CPU    2: hi:  186, btch:  31 usd:   0

CPU    3: hi:  186, btch:  31 usd:   0

CPU    4: hi:  186, btch:  31 usd: 170

CPU    5: hi:  186, btch:  31 usd:  14

CPU    6: hi:  186, btch:  31 usd:   0

CPU    7: hi:  186, btch:  31 usd:   0

Normal per-cpu:

CPU    0: hi:  186, btch:  31 usd: 116

CPU    1: hi:  186, btch:  31 usd:  12

CPU    2: hi:  186, btch:  31 usd:  29

CPU    3: hi:  186, btch:  31 usd:  29

CPU    4: hi:  186, btch:  31 usd: 179

CPU    5: hi:  186, btch:  31 usd:  97

CPU    6: hi:  186, btch:  31 usd:  32

CPU    7: hi:  186, btch:  31 usd:  36

active_anon:1549858 inactive_anon:230052 isolated_anon:0

 active_file:611268 inactive_file:558847 isolated_file:0

 unevictable:5 dirty:1059 writeback:382 unstable:0

 free:68058 slab_reclaimable:885143 slab_unreclaimable:53487

 mapped:172812 shmem:277726 pagetables:141053 bounce:0

DMA free:15916kB min:128kB low:160kB high:192kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15692kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes

lowmem_reserve[]: 0 3255 16133 16133

DMA32 free:96580kB min:27248kB low:34060kB high:40872kB active_anon:1269872kB inactive_anon:407824kB active_file:270600kB inactive_file:282812kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3333728kB mlocked:0kB dirty:1292kB writeback:752kB mapped:11520kB shmem:105632kB slab_reclaimable:788708kB slab_unreclaimable:40688kB kernel_stack:3872kB pagetables:144620kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:37 all_unreclaimable? no

lowmem_reserve[]: 0 0 12877 12877

Normal free:159736kB min:107788kB low:134732kB high:161680kB active_anon:4929560kB inactive_anon:512384kB active_file:2174472kB inactive_file:1952576kB unevictable:20kB isolated(anon):0kB isolated(file):0kB present:13186560kB mlocked:20kB dirty:2944kB writeback:776kB mapped:679728kB shmem:1005272kB slab_reclaimable:2751864kB slab_unreclaimable:173260kB kernel_stack:1872kB pagetables:419592kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:187 all_unreclaimable? no

lowmem_reserve[]: 0 0 0 0

DMA: 1*4kB 1*8kB 0*16kB 1*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15916kB

DMA32: 23101*4kB 130*8kB 61*16kB 43*32kB 12*64kB 6*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 97588kB

Normal: 36522*4kB 531*8kB 85*16kB 24*32kB 16*64kB 10*128kB 8*256kB 1*512kB 2*1024kB 0*2048kB 0*4096kB = 159376kB

1462893 total pagecache pages

15005 pages in swap cache

Swap cache stats: add 275648, delete 260643, find 706206/718468

Free swap  = 217372kB

Total swap = 996024kB

4194288 pages RAM

83203 pages reserved

4349401 pages shared

3658910 pages non-shared
```

VM:

```
vm.compact_memory = 1

vm.overcommit_memory = 0

vm.panic_on_oom = 0

vm.oom_kill_allocating_task = 0

vm.oom_dump_tasks = 1

vm.overcommit_ratio = 50

vm.page-cluster = 3

vm.dirty_background_ratio = 10

vm.dirty_background_bytes = 0

vm.dirty_ratio = 20

vm.dirty_bytes = 0

vm.dirty_writeback_centisecs = 500

vm.dirty_expire_centisecs = 3000

vm.nr_pdflush_threads = 0

vm.swappiness = 80

vm.lowmem_reserve_ratio = 256   256     32

vm.drop_caches = 0

vm.extfrag_threshold = 500

vm.min_free_kbytes = 135168

vm.percpu_pagelist_fraction = 0

vm.max_map_count = 65530

vm.heap_stack_gap = 65536

vm.laptop_mode = 0

vm.block_dump = 0

vm.vfs_cache_pressure = 100

vm.legacy_va_layout = 0

vm.stat_interval = 1

vm.mmap_min_addr = 65535

vm.scan_unevictable_pages = 0

vm.memory_failure_early_kill = 0

vm.memory_failure_recovery = 1
```

On what can this depend on?

----------

## guid0

does your system run nfsd?

guid0

----------

## scyld

 *guid0 wrote:*   

> does your system run nfsd?

 

Nope.

----------

