# Kernel panic: where to go for help?

## nielchiano

Hi,

I just got a kernel panic on one of my test machines. I guess this is not the right forum to ask for kernel-bugs, but can anyone redirect me to the right place?

Just FYI:

I have a system set up to bridge 2 interfaces together, and do firewalling on them. The problem is: when the system is under heavy load, I get a kernel panic:

```
Oops: 0000 [#1]

PREEMPT

CPU:    0

EIP:    0060:[<c039e881>]    Not tainted VLI

EFLAGS: 00010286   (2.6.11-gentoo-r8-netmon1-001)

EIP is at br_nf_pre_routing_finish+0x21/0x2d0

eax: cad80000   ebx: c910f4a0   ecx: 00000003   edx: 00000001

esi: c91a782e   edi: 00000002   ebp: 00000000   esp: c04afca6

ds: 007b   es: 007b   ss: 0068

Process swapper (pid: 0, threadinfo=c04ae000 task=c0421b20)

Stack: 00000484 47244c2c 00001028 c9b1b440 00000000 00000246 c0120f55 cad80000

       c8f4be60 00000000 c8f4be54 c8f4be60 00000070 c011d2d3 c04afdc2 c011d2d3

       c8f4be54 00000003 00000001 00000000 c0383c15 c8f4be54 00000003 00000000

Call Trace:

 [<c0120f55>] __mod_timer+0x135/0x1c0

 [<c011d2d3>] local_bh_enable+0x33/0x90

 [<c011d2d3>] local_bh_enable+0x33/0x90

 [<c0383c15>] ip_nat_fn+0x75/0x1d0

 [<c0383dba>] ip_nat_in+0x4a/0xd0

 [<c039e860>] br_nf_pre_routing_finish+0x0/0x2d0

 [<c039e860>] br_nf_pre_routing_finish+0x0/0x2d0

 [<c032349a>] nf_iterate+0x7a/0xb0

 [<c039e860>] br_nf_pre_routing_finish+0x0/0x2d0

 [<c039e860>] br_nf_pre_routing_finish+0x0/0x2d0

 [<c0323898>] nf_hook_slow+0xf8/0x130

 [<c039e860>] br_nf_pre_routing_finish+0x0/0x2d0

 [<c039e860>] br_nf_pre_routing_finish+0x0/0x2d0

 [<c039f088>] br_nf_pre_routing+0x248/0x420

 [<c039e860>] br_nf_pre_routing_finish+0x0/0x2d0

 [<c039a550>] br_handle_frame_finish+0x0/0x120

 [<c032349a>] nf_iterate+0x7a/0xb0

 [<c039a550>] br_handle_frame_finish+0x0/0x120

 [<c039a550>] br_handle_frame_finish+0x0/0x120

 [<c0323822>] nf_hook_slow+0x82/0x130

 [<c039a550>] br_handle_frame_finish+0x0/0x120

 [<c039a763>] br_handle_frame+0xf3/0x1e0

 [<c039a550>] br_handle_frame_finish+0x0/0x120

 [<c031830d>] netif_receive_skb+0xfd/0x250

 [<c03184df>] process_backlog+0x7f/0x100

 [<c03185d4>] net_rx_action+0x74/0x100

 [<c011d256>] __do_softirq+0x76/0x90

 [<c011d296>] do_softirq+0x26/0x30

 [<c011d365>] irq_exit+0x35/0x40

 [<c0104328>] do_IRQ+0x28/0x40

 [<c010289e>] common_interrupt+0x1a/0x20

Code: 90 00 00 00 83 48 0c 01 eb 8d 55 57 56 53 81 ec c4 00 00 00 8b 9c 24 d8 00 00 00 8b 43 18 89 44 24 1c 8b ab 90 00 00 00 8b 73 28 <8b> 45 0c a8 01 74

 <0>Kernel panic - not syncing: Fatal exception in interrupt

```

----------

## GordSki

It does look like it's the net filter stuff that's causing the panic (from the stack trace anyway).

Try the lists here: http://lists.netfilter.org/

I think that's the project that is responsible for iptables and the assosiated kernel code.

Good luck!

EDIT: Actually, having looked closer, it looks like it's failing in the network driver:

 *Quote:*   

> 
> 
>  [<c03185d4>] net_rx_action+0x74/0x100
> 
>  [<c011d256>] __do_softirq+0x76/0x90
> ...

 

See if you can find the maintainer of the driver in the kernel docs: /usr/src/linux/Documentation/networking

G.

----------

## nielchiano

 *GordSki wrote:*   

> 
> 
> EDIT: Actually, having looked closer, it looks like it's failing in the network driver:
> 
>  *Quote:*   
> ...

 

Hmm, I looked at it too, but I concluded that it's the reception of a packet that will cause a tree of calls to get the packet forwarded/bridged. So, yes, the network driver is the "source" of the problem, but I don't think it caused it... I'll report the bug to bugzilla.netfilter.org and see what happens...

----------

## GordSki

Yeah, sorry. I got confused about the order of the call trace (wasn't sure which end was which  :Shocked: )

G.

----------

## nielchiano

 *GordSki wrote:*   

> Yeah, sorry. I got confused about the order of the call trace (wasn't sure which end was which )
> 
> G.

 

neither am I, but I asumed from:

```
[<c011d365>] irq_exit+0x35/0x40

[<c0104328>] do_IRQ+0x28/0x40
```

that you need to "do_IRQ" before you can "irq_exit"...

----------

## GordSki

I agree, it also make sense to have netif_receive_skb before the calls to the pre-routing code.

G.

----------

## nielchiano

in case anyone wants to follow: https://bugzilla.netfilter.org/bugzilla/show_bug.cgi?id=339

----------

## Mamour

I had the weirdest kinds of crashes earlier this year, all starting with "Oops :PREEMPT:" as soon as my box was under heavy loads.

I ended up having to change the power supply in order to get things running smoothly, as it apparently caused RAM corruption when my CPU was used at 100% for long periods.

Anyway, before doing that, someone here recommended that I compile the kernel without CONFIG_PREEMPT ("Preemptible kernel" in the kernel configuration), which did stabilize my system for a while. As your Panic is apparently caused by PREEMPT, you should give it a try.

----------

## GordSki

This might supply some more info:

```

*  sys-kernel/ksymoops

      Latest version available: 2.4.9

      Latest version installed: [ Not Installed ]

      Size of downloaded files: 64 kB

      Homepage:    http://www.kernel.org/pub/linux/utils/kernel/ksymoops/

      Description: Utility to decode a kernel oops, or other kernel call traces.

      License:     GPL-2

```

G.

----------

## nielchiano

 *Mamour wrote:*   

> I ended up having to change the power supply in order to get things running smoothly, as it apparently caused RAM corruption when my CPU was used at 100% for long periods.

 I don't think that's my problem, since if I don't use the bridge, I can recompile gcc+glibc without any problem at all. (as you know, recompiling gcc+glibc on a P2 400MHz does take a while @ 100%...)

 *Mamour wrote:*   

> Anyway, before doing that, someone here recommended that I compile the kernel without CONFIG_PREEMPT ("Preemptible kernel" in the kernel configuration), which did stabilize my system for a while. As your Panic is apparently caused by PREEMPT, you should give it a try.

 I might try that...

----------

## Mamour

 *nielchiano wrote:*   

> I don't think that's my problem, since if I don't use the bridge, I can recompile gcc+glibc without any problem at all. (as you know, recompiling gcc+glibc on a P2 400MHz does take a while @ 100%...)

 

I know, but the crashes were absolutely random. At times it could go on compiling for hours (I once made a whole emerge -e system without having it crashing), and it would then panic after less than 20 minutes as I tried compiling something later on.

----------

## nielchiano

 *Mamour wrote:*   

>  *nielchiano wrote:*   I don't think that's my problem, since if I don't use the bridge, I can recompile gcc+glibc without any problem at all. (as you know, recompiling gcc+glibc on a P2 400MHz does take a while @ 100%...) 
> 
> I know, but the crashes were absolutely random. At times it could go on compiling for hours (I once made a whole emerge -e system without having it crashing), and it would then panic after less than 20 minutes as I tried compiling something later on.

 

To rule that out, I was able to reproduce the problem on a different computer. The only thing the two machines have in common is the wall-outlet and the switch.

I'm now recompiling without PREEMPT, and see where that brings me

----------

## nielchiano

 *nielchiano wrote:*   

> I'm now recompiling without PREEMPT, and see where that brings me

 

to the start... panic'ed again...

----------

## Mamour

 *nielchiano wrote:*   

>  *nielchiano wrote:*   I'm now recompiling without PREEMPT, and see where that brings me 
> 
> to the start... panic'ed again...

 

Bad news  :Sad: ... I guess your situation boils down to two alternatives. Either you've hit some obscure kernel bug, or you're having a hardware issue. I had the latter myself when my system kept crashing under heavy loads, and it's more probable than the kernel bug theory.

What kind of computer are you using? How powerful is your power supply? Too low wattage really does take its toll on a heavily used system, I can vouch for that.

----------

