# Quad Core: why only 1 core deals with interrupts?

## MiXi-IL

Hi,

I have Intel's Q6600 (Quad Core CPU) and I get the following:

```

 cat /proc/interrupts 

           CPU0       CPU1       CPU2       CPU3       

  0:    1048656          0          0          0   IO-APIC-edge      timer

  1:        706          0          0          0   IO-APIC-edge      i8042

  6:          3          0          0          0   IO-APIC-edge      floppy

  8:         52          0          0          0   IO-APIC-edge      rtc

  9:          0          0          0          0   IO-APIC-fasteoi   acpi

 12:          4          0          0          0   IO-APIC-edge      i8042

 16:          3          0          0          0   IO-APIC-fasteoi   ohci1394

 17:          0          0          0          0   IO-APIC-fasteoi   libata, uhci_hcd:usb3

 18:          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb4

 19:      11550          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb6

 20:          0          0          0          0   IO-APIC-fasteoi   ehci_hcd:usb1, uhci_hcd:usb7

 21:          4          0          0          0   IO-APIC-fasteoi   ehci_hcd:usb2, uhci_hcd:usb5

222:     125651          0          0          0   PCI-MSI-edge      eth0

223:      56436          0          0          0   PCI-MSI-edge      libata

NMI:          0          0          0          0 

LOC:    1041605    1041604    1043077    1043076 

ERR:          0

MIS:          0

```

uname -a :

```

Linux flux91 2.6.20-gentoo-r7 #1 SMP Fri May 4 06:04:25 IDT 2007 i686 Intel(R) Core(TM)2 Quad CPU           @ 2.40GHz GNU/Linux

```

Why aren't the 3 cores 1-3 making any interrupts? Maybe I have my kernel misconfigured for multiple cores?

Thanks.

----------

## i13m

Maybe this site will be useful.

Personally, I just do 

```
 emerge -av irqbalance 
```

----------

## energyman76b

because you have CPU hotplugging enabled in your kernel config.

Deactivate it and be happy.

----------

## MiXi-IL

I'll try irqbalance.

Thanks.

----------

## bunder

 *energyman76b wrote:*   

> because you have CPU hotplugging enabled in your kernel config.
> 
> Deactivate it and be happy.

 

i have it off, and i still get this behaviour on my quad.  irqbalance didn't look like it did anything.   :Confused: 

----------

## energyman76b

[ ] Support for hot-pluggable CPUs (EXPERIMENTAL)

is really off?

strange...

----------

## soccrstar

support for hot plug cpu is disabled on my kernel and I get the same issue

```
~ $ cat /proc/interrupts

           CPU0       CPU1       

  0:   14537802          0   IO-APIC-edge      timer

  1:          2          0   IO-APIC-edge      i8042

  6:          3          0   IO-APIC-edge      floppy

  8:          0          0   IO-APIC-edge      rtc

  9:          0          0   IO-APIC-fasteoi   acpi

 12:       5782          0   IO-APIC-edge      i8042

 14:    2853180          0   IO-APIC-edge      ide0

 15:          0          0   IO-APIC-edge      libata

 16:      83740          0   IO-APIC-fasteoi   libata, uhci_hcd:usb3

 18:        240          0   IO-APIC-fasteoi   ohci1394, libata, ehci_hcd:usb1, uhci_hcd:usb7

 19:          0          0   IO-APIC-fasteoi   libata, libata, uhci_hcd:usb6

 21:          0          0   IO-APIC-fasteoi   uhci_hcd:usb4

 22:      95687          0   IO-APIC-fasteoi   HDA Intel

 23:         10          0   IO-APIC-fasteoi   ehci_hcd:usb2, uhci_hcd:usb5

379:     803053          0   PCI-MSI-edge      eth0

NMI:      36634      39927 

LOC:   15188965   15188963 

ERR:          0

```

I will try the irqbalance program

----------

## vipernicus

 *MiXi-IL wrote:*   

> I'll try irqbalance.
> 
> Thanks.

 

Your kernel isn't patched with suspend2 by any chance is it?

----------

## r3pek

it's supposed to be like that (all irq's in one core) for best performance.

i only one core takes care of all the irq's there will be more cache hit's and so it will take less time to process the interrupt and the other 3 cores (in your case) are free to do what ever your computer is doing without being interrupted. this, in most cases, gives best performance.

----------

## batistuta

 *r3pek wrote:*   

> it's supposed to be like that (all irq's in one core) for best performance.
> 
> i only one core takes care of all the irq's there will be more cache hit's and so it will take less time to process the interrupt and the other 3 cores (in your case) are free to do what ever your computer is doing without being interrupted. this, in most cases, gives best performance.

 

I'm not familiar with the IRQ model in Linux, so please be pedagogical   :Very Happy: 

What I don't understand, is why all IRQs in one CPU will cause most cache hits. I do see that each individual IRQ should be handled always by the same CPU. That is, we don't want it to be handled sometimes by one, and sometimes by another core. But why having *all* IRQs handled by a single CPU improves cache hits? Is it because of some common (generic) kernel IRQ wrapper that is good to have always in the instruction cache? Thanks

----------

## HTS

Q6600 here too.

Seems to work fine out of the box.

Surely a problem in your kernel config.

Here is my output, I'm not using any sort of userspace irq-balancing... (using in-kernel one)

```
           CPU0       CPU1       CPU2       CPU3

  0:     855177     855510     855875     857408   IO-APIC-edge      timer

  1:       2947       2956       2982       2980   IO-APIC-edge      i8042

  8:          0          1          0          0   IO-APIC-edge      rtc

  9:          0          0          0          0   IO-APIC-fasteoi   acpi

 12:          1          0          3          0   IO-APIC-edge      i8042

 16:     274730     274144     274427     273552   IO-APIC-fasteoi   uhci_hcd:usb3, eth0, nvidia

 18:          1          0          1          1   IO-APIC-fasteoi   ehci_hcd:usb1, uhci_hcd:usb5, uhci_hcd:usb8

 19:      27857      27653      27462      27714   IO-APIC-fasteoi   libata, libata, uhci_hcd:usb7

 21:     100922     100624     100057      99962   IO-APIC-fasteoi   uhci_hcd:usb4

 22:     181974     182595     182674     181865   IO-APIC-fasteoi   HDA Intel

 23:          0          0          0          0   IO-APIC-fasteoi   ehci_hcd:usb2, uhci_hcd:usb6

NMI:          0          0          0          0

LOC:    3409065    3409052    3406915    3406887

ERR:          0

```

On a side note, my office workstation is a 2x Xeon-Dual-Core and seems to behave differently. On these, the 1st core in the 1st CPU ends up handling most (85%) of the interrupts... I'd like some clarification on this  :Surprised: 

----------

## mspiegle

I've got the same exact problem with a 16-core AMD box using 10gbit NICs.  I've tried using the irqbalance daemon, running irqbalance one-time, and just manually setting smp_affinity without irqbalance and in all situations the NIC will die after an hour.  If I don't mess with anything, the system runs fine but maxes at at around 3.5gbit/sec (the single core is spending 100% time servicing soft interrupts).

My understanding is that MSI-X is SUPPOSED to be able to balance out among different cores (an advantage over regular MSI).  I have checked the driver code and it indeed makes a call to pci_enable_msix() which returns success.  I'm totally clueless as to what to try next.

I've tried kernels 2.6.16, 2.6.17, 2.6.20, and 2.6.22.  All have the same problem.  I have NUMA and APIC enabled.  I don't have any CPU hotplugging options enabled.  The interrupts just seem to be pegged to core0...

http://pastebin.ca/raw/733856

mod edit by bunder: formatting was out of whack   :Wink: 

----------

## jbcjorge

First, sorry for my poor English...

The cache hits get increased by handling all the interruptions in one core because when the interruption call it's being processed, the CPU makes a request to "know" from the table of interruptions the right information about the interruption itself, so, that information gets stores at the core's cache, and because that single core manages all interruption, it's very probable that the information needed to process the interruption call it was already stored at the cache memory; so, the load of the bus (CPU-RAM) is less, and the speed of response is faster (as the bus CPU-CACHE is faster also)...

Actually, if someone can make the in a multiple core/cpu environment, only one core controls the threads (or process) of greater importance, leaving the remaining cores/cpu to make the trivial thing, the response of the system would be faster and even, most reliable.

I hope to help, and to be right, r3pek....

----------

## vermaden

You can also set them manually, for example timer/eth0 -> core/cpu0 and ata/rest -> core/cpu1

```
# cat /proc/interrupts 

           CPU0       CPU1       

  0: 1891421640      13442    IO-APIC-edge  timer

  1:          8          0    IO-APIC-edge  i8042

  7:          0          0    IO-APIC-edge  parport0

  8:          0          4    IO-APIC-edge  rtc

  9:          0          0   IO-APIC-level  acpi

169:       1327 3054418929   IO-APIC-level  gdth, eth2

177: 3173739569       1384   IO-APIC-level  eth0

185:   32389389         17   IO-APIC-level  eth1

NMI:          0          0 

LOC: 1891729277 1891741262 

ERR:          0

MIS:          0

# cat /proc/irq/177/smp_affinity 

1

# cat /proc/irq/169/smp_affinity 

2

# echo 1 > /proc/irq/169/smp_affinity

# cat /proc/irq/169/smp_affinity 

1

```

----------

## batistuta

 *jbcjorge wrote:*   

> so, that information gets stores at the core's cache, and because that single core manages all interruption, it's very probable that the information needed to process the interruption call it was already stored at the cache memory; so, the load of the bus (CPU-RAM) is less,

 

I agree with that. But for this doesn't answer my question, or at least I don't get it. The way I understand, is that all you need to do assign to CPU 1 a subset of all interrupts, another subset to CPU 2, and so on. As far as I can see (by no means I claim I'm right), as long as each CPU handles the same set of interrupts, then this should guarantee the cache hit. So I'm still confused why a single CPU needs to handle all of them  :Confused: 

I've found a paper on Linux kernel and multicore support, will take a look...

----------

## root_tux_linux

Kerneloption:

noapic or nolapic = only one cpu deals with interrrups

without Kerneloption:

noapic or nolapic = both cpu deals with interrupts

Have you noapic on?

----------

## jbcjorge

Mmm, I guess that reading this could help...

It's from lesswats.org

 *Quote:*   

> Power aware interrupt balancing
> 
> Use the interrupt (IRQ) balancing technique to distribute the various interrupts, from the devices on the system, over the cores/processors in the system.
> 
> IRQ balancing is not new in Linux. It is a mechanism for load balancing the work done by different CPU cores running on a system. For a number of years, there has been a configuration option to compile automatic IRQ balancing into the kernel. It is recommended that you do not use that option. The implementation is rather simple and not optimal for current systems.
> ...

 

----------

## HTS

Indeed. You certainly don't want to use this on 1 multi-core CPU though (because it won't do anything).

----------

## jbcjorge

Actually, irqbalance works in a multi-core CPU, just the way it does in a Multiple-CPU or NUMA system... And I say that, because I have a modest core2 T5500   :Very Happy: 

----------

## Monkeh

 *jbcjorge wrote:*   

> Actually, irqbalance works in a multi-core CPU, just the way it does in a Multiple-CPU or NUMA system... And I say that, because I have a modest core2 T5500  

 

It does very little good with a dual-core CPU. Balancing between cores 0 and 2 with a quad can be useful (I believe), but balancing on a dual core does little.

----------

