# Mem-Info Spewing to Syslog

## SDNick484

For some reason, around 2:17 AM this morning, my personal server which has been running fine for the last 60 days, started spewing Mem-Info to the syslog roughly 25-30 times per second.  I have no clue the cause, and am unsure how to stop it.  I'm seeing a message similar to the following constantly repeated:

<<<<<

Mar 22 03:30:28 lisa kernel: Mem-Info:

Mar 22 03:30:28 lisa kernel: Node 0 DMA per-cpu:

Mar 22 03:30:28 lisa kernel: CPU    0: hi:    0, btch:   1 usd:   0

Mar 22 03:30:28 lisa kernel: CPU    1: hi:    0, btch:   1 usd:   0

Mar 22 03:30:28 lisa kernel: Node 0 DMA32 per-cpu:

Mar 22 03:30:28 lisa kernel: CPU    0: hi:  186, btch:  31 usd:  51

Mar 22 03:30:28 lisa kernel: CPU    1: hi:  186, btch:  31 usd:  77

Mar 22 03:30:28 lisa kernel: active_anon:86123 inactive_anon:35140 isolated_anon:0

Mar 22 03:30:28 lisa kernel: active_file:289467 inactive_file:195462 isolated_file:0

Mar 22 03:30:28 lisa kernel: unevictable:0 dirty:111 writeback:0 unstable:0

Mar 22 03:30:28 lisa kernel: free:53451 slab_reclaimable:94581 slab_unreclaimable:3852

Mar 22 03:30:28 lisa kernel: mapped:8640 shmem:5108 pagetables:3189 bounce:0

Mar 22 03:30:28 lisa kernel: Node 0 DMA free:12060kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:1176kB inactive_file:2328kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15364kB mlocked:0kB dirty:0kB writeback:0kB mapped:12kB shmem:0kB slab_reclaimable:368kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no

Mar 22 03:30:28 lisa kernel: lowmem_reserve[]: 0 3005 3005 3005

Mar 22 03:30:28 lisa kernel: Node 0 DMA32 free:201744kB min:6996kB low:8744kB high:10492kB active_anon:344492kB inactive_anon:140560kB active_file:1156692kB inactive_file:779520kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3078036kB mlocked:0kB dirty:444kB writeback:0kB mapped:34548kB shmem:20432kB slab_reclaimable:377956kB slab_unreclaimable:15400kB kernel_stack:2280kB pagetables:12756kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no

Mar 22 03:30:28 lisa kernel: lowmem_reserve[]: 0 0 0 0

Mar 22 03:30:28 lisa kernel: Node 0 DMA: 25*4kB 23*8kB 10*16kB 15*32kB 10*64kB 6*128kB 4*256kB 5*512kB 2*1024kB 0*2048kB 1*4096kB = 12060kB

Mar 22 03:30:28 lisa kernel: Node 0 DMA32: 9848*4kB 14870*8kB 2176*16kB 212*32kB 0*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 201744kB

Mar 22 03:30:28 lisa kernel: 490043 total pagecache pages

Mar 22 03:30:28 lisa kernel: 0 pages in swap cache

Mar 22 03:30:28 lisa kernel: Swap cache stats: add 27951, delete 27951, find 336194/337916

Mar 22 03:30:28 lisa kernel: Free swap  = 3156764kB

Mar 22 03:30:28 lisa kernel: Total swap = 3156764kB

Mar 22 03:30:28 lisa kernel: 784272 pages RAM

Mar 22 03:30:28 lisa kernel: 13084 pages reserved

Mar 22 03:30:28 lisa kernel: 391143 pages shared

Mar 22 03:30:28 lisa kernel: 373831 pages non-shared

>>>>>

I've disabled syslog-ng for the moment to prevent messages from growing too large, but I don't like running in this state.  Today is the first time this has occurred to my knowledge (I have the last 30 days of logs, and today at 02:17:15 is the first appearance of "Mem-Info").  

Anyone have an idea on the cause or fix?  The system is running 2.6.32-gentoo-r2 and app-admin/syslog-ng-3.0.5.  

Below are vmstat and iostat outputs:

<<<<<

# vmstat 5 8

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----

 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa

 0  0      0 221100 507112 1451964    0    0     1     7    2    1  1  1 97  1

 4  0      0 221092 507112 1451964    0    0     0     9  440  389  3  2 96  0

 2  0      0 220968 507112 1451964    0    0     0   121  450  386  2  2 96  0

 1  0      0 220968 507116 1451968    0    0     0    11  451  386  3  2 95  1

 2  0      0 221092 507116 1451968    0    0     0     6  449  389  3  2 95  1

 0  0      0 224108 507116 1451968    0    0     0     2  452  397  3  2 96  0

 4  0      0 224108 507116 1451968    0    0     0     6  456  391  3  2 95  1

 1  0      0 224108 507116 1451968    0    0     0    12  477  397  3  2 95  1

# iostat 5

Linux 2.6.32-gentoo-r2 (lisa) 	03/22/10 	_x86_64_	(2 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle

           1.01    0.01    1.47    0.65    0.00   96.86

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn

sda               1.23         4.25        29.50   22384826  155473891

avg-cpu:  %user   %nice %system %iowait  %steal   %idle

           2.46    0.00    1.84    0.86    0.00   94.84

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn

sda               0.80         3.20        12.80         16         64

avg-cpu:  %user   %nice %system %iowait  %steal   %idle

           2.63    0.00    1.56    0.84    0.00   94.97

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn

sda               1.00         0.00        17.56          0         88

avg-cpu:  %user   %nice %system %iowait  %steal   %idle

           2.78    0.00    1.77    1.14    0.00   94.30

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn

sda               0.80         0.00        14.40          0         72

avg-cpu:  %user   %nice %system %iowait  %steal   %idle

           2.78    0.00    1.69    0.73    0.00   94.80

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn

sda               1.20         0.00        28.80          0        144

>>>>>

----------

## eccerr0r

Looks suspiciously like OOM information but it looks like there's still anonymous memory.  Is this all that's being repeated, and nothing else?  Should have reported the process (or kernel trace stack) that triggered the dump.

However DMA memory looks a little low.  DMA buffers need to remain in memory and never swap out.

What are you running on the machine?  What drivers?

----------

## Ant P.

Could be that the magic sysrq code got stuck somehow... 30 times per second sounds pretty close to default keyboard repeat rate too.

----------

## SDNick484

 *Ant_P wrote:*   

> Could be that the magic sysrq code got stuck somehow... 30 times per second sounds pretty close to default keyboard repeat rate too.

 

Wow, excellent catch, that is exactly what was occurring.  The system does have a keyboard, and it was wedged with some weight on it.  When I looked into it last night, I was ssh'ing in from my iPod touch then a laptop, not on the actual console.  Thanks again!

Eccerr0r, thanks for the suggestion, that's what I was expecting as well.  Glad it was much simpler.

----------

## eccerr0r

Ha... kind of funny, I was thinking about the sysreq key being stuck down but shrugged it off as "nah...he should have typed that on the console..." and "what could have fell on the keyboard at 2AM and pressed sysreq (since it needs a modifier)"... Oh well, some of the most obvious solutions need to be checked...

----------

