# [SOLVED] Kernel crash

## leon_73

Hi all,

Since a couple of days I'm experiencing some system failure... I get the following error and then the box hangs there.  :Sad: 

```
Apr  6 10:51:20 localhost Unable to handle kernel paging request at virtual address 000206cc

Apr  6 10:51:20 localhost printing eip:

Apr  6 10:51:20 localhost c0136fbb

Apr  6 10:51:20 localhost *pde = 00000000

Apr  6 10:51:20 localhost Oops: 0000 [#1]

Apr  6 10:51:20 localhost PREEMPT

Apr  6 10:51:20 localhost Modules linked in: ipt_state iptable_filter iptable_mangle iptable_nat ip_tables snd_pcm_oss snd_mixer_oss snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device ohci_hcd ehci_hcd parport_pc parport floppy yenta_socket rsrc_nonstatic pcmcia_core snd_maestro3 snd_ac97_codec snd_pcm snd_timer snd_page_alloc snd soundcore i2c_i801 i2c_core usbhid uhci_hcd loop usbcore videodev cpufreq_stats acpi_cpufreq freq_table omnibook acx_pci dm_mod intel_agp e100 mii i830 drm i810fb vgastate intelfb agpgart rtc

Apr  6 10:51:20 localhost CPU:    0

Apr  6 10:51:20 localhost EIP:    0060:[<c0136fbb>]    Not tainted VLI

Apr  6 10:51:20 localhost EFLAGS: 00010a16   (2.6.11.6-leoV1)

Apr  6 10:51:20 localhost EIP is at page_waitqueue+0x1b/0x30

Apr  6 10:51:20 localhost eax: 1697a900   ebx: c797a900   ecx: 00000020   edx: 000205c4

Apr  6 10:51:20 localhost esi: 00000002   edi: 00000001   ebp: ffffffff   esp: d74abe40

Apr  6 10:51:20 localhost ds: 007b   es: 007b   ss: 0068

Apr  6 10:51:20 localhost Process kswapd0 (pid: 152, threadinfo=d74aa000 task=d7474580)

Apr  6 10:51:20 localhost Stack: c0137093 c797a900 c797a900 c0141da7 c8f1d944 c1039ce0 00000000 0000000e

Apr  6 10:51:20 localhost 00000001 00000002 00000000 c1039ce0 c797a900 c11ae240 c105a280 c112bd20

Apr  6 10:51:20 localhost c1266300 c11822c0 c11731e0 c10e0ae0 00000021 c03b9304 00000020 00000246

Apr  6 10:51:20 localhost Call Trace:

Apr  6 10:51:20 localhost [<c0137093>] unlock_page+0x23/0x40

Apr  6 10:51:20 localhost [<c0141da7>] invalidate_mapping_pages+0xe7/0x100

Apr  6 10:51:20 localhost [<c0142a67>] shrink_cache+0x1b7/0x360

Apr  6 10:51:20 localhost [<c0141dde>] invalidate_inode_pages+0x1e/0x30

Apr  6 10:51:20 localhost [<c0171376>] prune_icache+0x1f6/0x210

Apr  6 10:51:20 localhost [<c01713af>] shrink_icache_memory+0x1f/0x50

Apr  6 10:51:20 localhost [<c0142204>] shrink_slab+0x154/0x190

Apr  6 10:51:20 localhost [<c014373b>] balance_pgdat+0x2db/0x3b0

Apr  6 10:51:20 localhost [<c01438f9>] kswapd+0xe9/0x110

Apr  6 10:51:20 localhost [<c012cf80>] autoremove_wake_function+0x0/0x60

Apr  6 10:51:20 localhost [<c0103002>] ret_from_fork+0x6/0x14

Apr  6 10:51:20 localhost [<c012cf80>] autoremove_wake_function+0x0/0x60

Apr  6 10:51:20 localhost [<c0143810>] kswapd+0x0/0x110

Apr  6 10:51:20 localhost [<c01012fd>] kernel_thread_helper+0x5/0x18

Apr  6 10:51:21 localhost Code: c4 18 c3 89 f0 e8 06 a1 00 00 eb e9 8d 74 26 00 8b 44 24 04 b9 20 00 00 00 8b 10 69 c0 01 00 37 9e c1 ea 1d 8b 14 95 a8 cc 47 c0 <2b> 8a 08 01 00 00 8b 92 00 01 00 00 d3 e8 8d 04 c2 c3 8d 76 00

```

at the moment I'm runnig a vanilla-2.6.11.6 patched whith reiser4fs, squashfs, and vesafb-tng.

My sysinfo are:

```
~ # emerge info

Portage 2.0.51.19 (default-linux/x86/2004.3, gcc-3.3.5, glibc-2.3.4.20041102-r1, 2.6.11.6-leoV1 i686)

=================================================================

System uname: 2.6.11.6-leoV1 i686 Intel(R) Pentium(R) III Mobile CPU      1066MHz

Gentoo Base System version 1.4.16

Python:              dev-lang/python-2.3.4-r1 [2.3.4 (#1, Feb 18 2005, 08:57:35)]

dev-lang/python:     2.3.4-r1

sys-devel/autoconf:  2.13, 2.59-r6

sys-devel/automake:  1.5, 1.8.5-r3, 1.6.3, 1.7.9-r1, 1.4_p6, 1.9.4

sys-devel/binutils:  2.15.92.0.2-r1

sys-devel/libtool:   1.5.10-r4

virtual/os-headers:  2.6.8.1-r2

ACCEPT_KEYWORDS="x86"

AUTOCLEAN="yes"

CFLAGS="-O2 -pipe -march=pentium3 -fomit-frame-pointer"

CHOST="i686-pc-linux-gnu"

CONFIG_PROTECT="/etc /usr/kde/2/share/config /usr/kde/3.3/env /usr/kde/3.3/share/config /usr/kde/3.3/shutdown /usr/kde/3/share/config /usr/lib/X11/xkb /usr/share/config /var/bind /var/qmail/control"

CONFIG_PROTECT_MASK="/etc/gconf /etc/terminfo /etc/env.d"

CXXFLAGS="-O2 -pipe -march=pentium3 -fomit-frame-pointer"

DISTDIR="/usr/portage/distfiles"

FEATURES="autoaddcvs autoconfig ccache distlocks sandbox sfperms userpriv usersandbox"

GENTOO_MIRRORS="http://pandemonium.tiscali.de/pub/gentoo/ ftp://pandemonium.tiscali.de/pub/gentoo/ http://gentoo.inode.at/ ftp://gentoo.inode.at/source/ http://ftp-stud.fht-esslingen.de/pub/Mirrors/gentoo/"

LANG="it_IT@euro"

MAKEOPTS="-j2"

PKGDIR="/usr/portage/packages"

PORTAGE_TMPDIR="/var/tmp"

PORTDIR="/usr/portage"

PORTDIR_OVERLAY="/usr/local/portage"

SYNC="rsync://rsync.europe.gentoo.org/gentoo-portage"

USE="x86 GAPING_SECURITY_HOLE X X509 acl acpi alsa amd apache2 arts artswrappersuid bash-completion berkdb bind-mysql bitmap-fonts bluetooth bzlib cdparanoia chroot crypt cups curl curlwrappers dbase dbx devmap directfb divx4linux droproot dvd dvdread ecc emboss encode esd ethereal evo evo2 fam fbcon fbdev font-server ftp gdbm gif gmp gnutls gphoto2 gpm gtk gtk2 hal i8x0 icq idea imagemagick imap innodb ipv6 irmc jabber jack java javascript jpeg kdepim kerberos ldap libclamav libg++ lirc lm_sensors lua lzw-tiff maildir mhash mime mmx mp3 mpeg msn mysql mysqli ncurses nls nntp nptl ntlm odbc offensive ogg oggvorbis opengl oscar pam pam-mysql pcmcia pda pdflib perl php plotutils png pnp posix python qt quotas readline rtc ruby samba sasl sdl sftplogging skey slang slp smime spell sqlite sse ssl tcltk tiff tokenizer transcode truetype truetype-fonts type1 type1-fonts unicode usb userlocales wifi winbind xml xml2 xmms xosd xpm xv xvid yahoo zlib video_cards_i810 linguas_it"

Unset:  ASFLAGS, CBUILD, CTARGET, LC_ALL, LDFLAGS
```

Could anybody help me to understand what can I do to save my box.

Tnx

LeoLast edited by leon_73 on Thu Apr 07, 2005 10:36 am; edited 1 time in total

----------

## Shazam

i had this problem, too. i fixed it by using a newer reiser4 patch. actually my kernel compisition right now is the gentoo-2.6.11-r5 kernel with the 2.6.11.6 patch and the reiser4-2.6.11-3 patch over it, should come out almost the same thing you have, but i assume you have used the reiser4-2.6.11-1 patch.

just a guess, maybe you wanna give it a try.

----------

## leon_73

 *Shazam wrote:*   

> i had this problem, too. i fixed it by using a newer reiser4 patch. actually my kernel compisition right now is the gentoo-2.6.11-r5 kernel with the 2.6.11.6 patch and the reiser4-2.6.11-3 patch over it, should come out almost the same thing you have, but i assume you have used the reiser4-2.6.11-1 patch.
> 
> just a guess, maybe you wanna give it a try.

 

I've recompiled the kernel with the new patch... Thanks for for the hint... Now I will test it for a bit, then I will tell you  :Smile: 

Leo

edit 07/04/2005 12:32 +1GMT

I know that 5h are not a big test, but it seems this was the problem... No crash till now!!!!  :Smile: 

tnx a lot!!!!  :Very Happy:   :Very Happy:   :Very Happy: 

----------

## gnac

I've just started having a similar problem with my nightly backups using rsync.  It doesn't happen every day, but every couple of days the system will "mostly" hang, (eg apache continues to work.  The desktop crashes and ssh no longer works).

The output in /var/log/messages is: 

```
May  1 03:01:02 valhalla kernel: c0155533

May  1 03:01:02 valhalla kernel: PREEMPT

May  1 03:01:02 valhalla kernel: Modules linked in: snd_rtctimer snd_seq_midi snd_emu10k1_synth snd_emux_synth snd_seq_virmidi snd_seq_midi_emul snd_pcm_oss snd_mixer_oss snd_seq_oss snd_seq_midi_event snd_seq parport_pc parport forcedeth snd_usb_usx2y snd_usb_lib snd_emu10k1 snd_rawmidi snd_seq_device snd_ac97_codec snd_pcm snd_timer snd_ac97_bus snd_page_alloc snd_util_mem snd_hwdep snd soundcore i2c_nforce2 i2c_core nvidia 3c59x

May  1 03:01:02 valhalla kernel: CPU:    0

May  1 03:01:02 valhalla kernel: EIP:    0060:[<c0155533>]    Tainted: P      VLI

May  1 03:01:02 valhalla kernel: EFLAGS: 00010246   (2.6.14-gentoo-r5)

May  1 03:01:02 valhalla kernel: EIP is at remove_inode_buffers+0x43/0x80

May  1 03:01:02 valhalla kernel: eax: 00000040   ebx: cb6b9624   ecx: ca6b9624   edx: 08165588

May  1 03:01:02 valhalla kernel: esi: cb6b9548   edi: 00000001   ebp: c15be000   esp: c15bfe98

May  1 03:01:02 valhalla kernel: ds: 007b   es: 007b   ss: 0068

May  1 03:01:02 valhalla kernel: Process kswapd0 (pid: 173, threadinfo=c15be000 task=c156d580)

May  1 03:01:02 valhalla kernel: Stack: cb6b9548 cb6b9550 cb6b9548 00000010 c016dcac cb6b9548 c15bfed4 00000000

May  1 03:01:02 valhalla kernel:        00000010 cb6b9860 ceb360d8 00006720 00000000 00000090 dffeeae0 c016dd4f

May  1 03:01:02 valhalla kernel:        00000080 c013f6b9 00000080 000000d0 00018086 0019c800 00000000 00000011

May  1 03:01:02 valhalla kernel: Call Trace:

May  1 03:01:02 valhalla kernel:  [<c016dcac>] prune_icache+0x13c/0x1c0

May  1 03:01:02 valhalla kernel:  [<c016dd4f>] shrink_icache_memory+0x1f/0x50

May  1 03:01:02 valhalla kernel:  [<c013f6b9>] shrink_slab+0x179/0x1c0

May  1 03:01:02 valhalla kernel:  [<c0140b3e>] balance_pgdat+0x2ce/0x3a0

May  1 03:01:02 valhalla kernel:  [<c0140cee>] kswapd+0xde/0x100

May  1 03:01:02 valhalla kernel:  [<c0129e30>] autoremove_wake_function+0x0/0x60

May  1 03:01:02 valhalla kernel:  [<c0102d5e>] ret_from_fork+0x6/0x14

May  1 03:01:02 valhalla kernel:  [<c0129e30>] autoremove_wake_function+0x0/0x60

May  1 03:01:02 valhalla kernel:  [<c0140c10>] kswapd+0x0/0x100

May  1 03:01:02 valhalla kernel:  [<c0101009>] kernel_thread_helper+0x5/0xc

May  1 03:01:02 valhalla kernel: Code: c0 74 4b 8d 9e dc 00 00 00 b8 00 e0 ff ff 21 e0 ff 40 14 8b 86 dc 00 00 00 39 d8 74 20 89 c1 8b 41 d8 a8 02 75 38 8b 01 8b 51 04 <89> 50 04 89 02 89 49 04 89 09 8b 03 39 d8 89 c1 75 e2 b8 00 e0

May  1 03:01:02 valhalla kernel:  <6>note: kswapd0[173] exited with preempt_count 1

```

The backup script is set to run by cron at 3:00.   I am only using 58% of my backup drive.  Here is the script:

```
# rsync scripts derived from information on :

# www.mikerubel.org/computers/rsync_snapshots/

mount -o remount,rw /mnt/backup

# shift old backups

mv /mnt/backup/valhalla.6 /mnt/backup/valhalla.tmp

mv /mnt/backup/valhalla.5 /mnt/backup/valhalla.6

mv /mnt/backup/valhalla.4 /mnt/backup/valhalla.5

mv /mnt/backup/valhalla.3 /mnt/backup/valhalla.4

mv /mnt/backup/valhalla.2 /mnt/backup/valhalla.3

mv /mnt/backup/valhalla.1 /mnt/backup/valhalla.2

mv /mnt/backup/valhalla.0 /mnt/backup/valhalla.1

mv /mnt/backup/valhalla.tmp /mnt/backup/valhalla.0

cp -alf /mnt/backup/valhalla.1/. /mnt/backup/valhalla.0

# perform backup

/usr/bin/rsync -ax --delete / /mnt/backup/valhalla.0/root

/usr/bin/rsync -a --delete /home/ /mnt/backup/valhalla.0/home

/usr/bin/rsync -a --delete /var/log/ /mnt/backup/valhalla.0/var/log

/usr/bin/rsync -a --delete /var/www/ /mnt/backup/valhalla.0/var/www

/usr/bin/rsync -a --delete /var/spool/ /mnt/backup/valhalla.0/var/spool

/usr/bin/rsync -a --delete /var/lib/portage/world /mnt/backup/valhalla.0/var/lib/portage/world

mount -o remount,ro /mnt/backup
```

So my question is, when you say you apply the reiserfs patch, do you simply mean that you compile it into the kernel (via make menuconfig) or do you actually download and apply a special patch?

I'm using gentoo-sources-2.6.14-r5, which I haven't touched in a while.  The current release is 2.6.16-r3, but I didn't use the previous release due to an issue with the nvidia drivers.  This problem just started happening and I haven't made any changes to the kernel or file system tools.

Thanks.

----------

## Shazam

well, do you use reiser4 or reiserfs? reiserfs is already included in the kernel for quite some time and runs stable. reiser4 however isn't in the kernels[yet]. so if you use reiser4, i suggest you should use a newer kernel version, like the one you mentioned and apply a reiser4 patch you get here ftp://ftp.namesys.com/pub/reiser4-for-2.6.

however, if you have sensitive data, i cannot recommend reiser4, as the recovery tools are limited in their functions. i 'downgraded' my filesystem back to reiserfs, which runs as fast as reiser4 [at least for my case] and is more stable.

----------

## gnac

Ahh, I see. I am using "plain" reiserfs. 

Thanks.

----------

## baby_tux

Thanks for the solution! I have currently this problem and didn't what was the cause.

I am going to upgrade now because I had some FS errors recently.Last edited by baby_tux on Wed Oct 24, 2007 1:10 am; edited 1 time in total

----------

## Moriah

I have had problems during my nightly backup runs.  I had assumed it was an LVM2 snapshot problem, but maybe it is an rsync problem.  

I use rsync tunnelled thru ssh to do my nightly backups.  I use LVM2 on the client machines to make a read-only snapshot to freeze the filesystem during the rsync backup.

Sometimes -- maybe once every 1 or 2 weeks -- a machine will die in the middle of the backup operation.  All machines are running 2.6.* kernels, and fairly up to date, but maybe not the very latest kernel.

This problem only started since around April 2006, and the backup system has been running for almost 2 years now.

Any ideas as to the cause   :Question: 

----------

