# IRQ seems to be using only CPU0.

## SlashBeast

For some unknown to me reason the IRQ seems to use pretty much only CPU0, smp_affinity the '00000f' suppose to use ALL cpus but it does use only the first one, I have googled the http://docs.oracle.com/cd/E24290_01/coh.371/e22838/tune_perftune.htm which let me understand a little about how can I force a specified irq to use specified cpu, I have written a hacky script to balance it between 4 threads (I own core i5-2430M, two cores with ht) and it seems to be working, kind of...

```
#!/bin/sh

# irq-rebalance.sh

for i in /proc/irq/*/smp_affinity; do 

   case "${smp_affinity}" in

      ''|'8')

         smp_affinity=1

      ;;

      '1')

         smp_affinity=2

      ;;

      '2')

         smp_affinity=4

      ;;

      '4')

         smp_affinity=8

      ;;

   esac

   echo "$i: 00000${smp_affinity}"

   echo "00000${smp_affinity}" >"$i"

done
```

Without it, the /proc/interrupts looks like:

```
           CPU0       CPU1       CPU2       CPU3

  0:         24          0          0          0   IO-APIC-edge      timer

  1:      62188          0          0          0   IO-APIC-edge      i8042

  8:        267          0          0          0   IO-APIC-edge      rtc0

  9:     264195          0          0          0   IO-APIC-fasteoi   acpi

 12:     136278          0          0          0   IO-APIC-edge      i8042

 16:      18718          0          0          0   IO-APIC-fasteoi   i915, ehci_hcd:usb1

 17:    7697397          0          0          0   IO-APIC-fasteoi   ehci_hcd:usb2, iwlwifi

 18:    1083624          0          0          0   IO-APIC-fasteoi   ahci, mmc0, eth0

 22:     860678          0          0          0   IO-APIC-fasteoi   snd_hda_intel

NMI:          0          0          0          0   Non-maskable interrupts

LOC:    3362075    3341920    3416035    3924691   Local timer interrupts

SPU:          0          0          0          0   Spurious interrupts

PMI:          0          0          0          0   Performance monitoring interrupts

IWI:          0          0          0          0   IRQ work interrupts

RTR:          3          0          0          0   APIC ICR read retries

RES:     275352     310805     137941      66781   Rescheduling interrupts

CAL: 4294963368 4294964404 4294964540 4294964464   Function call interrupts

TLB:     261929     332582     224693     200127   TLB shootdowns

TRM:          0          0          0          0   Thermal event interrupts

THR:          0          0          0          0   Threshold APIC interrupts

MCE:          0          0          0          0   Machine check exceptions

MCP:        147        147        147        147   Machine check polls

ERR:          0

MIS:          0
```

And yes, I have no idea what I am doing, if anyone have any idea about the IRQ and can tell me why its not balanced I would be thankful.

----------

## aCOSwt

Could be not related at all, just curious to know the status of CONFIG_HOTPLUG_CPU on the kernel your system is running.

----------

## SlashBeast

It is enabled, I think I need it for suspend (s2disk) to work.

----------

## Ant P.

There's little point in balancing IRQs on a non-NUMA system, since most of their work is done in kernel threads already.

----------

## SlashBeast

Well, the thing is, when I have really busy usb ports (like copy a lot of data with avg 28 MBps from one usb device to another) my touchpad is... laggy. I also notice other small issues that could be IRQ fault. Appreantly manual rerouting is the only possible solution. Thats just sad.

----------

## PaulBredbury

 *SlashBeast wrote:*   

> my touchpad is... laggy

 

From my messing with IRQs, trying to optimize USB mouse input, soundcard and Nvidia IRQs (I just created contention and more latency), I would say just let the scheduler do it. I.e.:

```
echo f > /proc/irq/default_smp_affinity

for i in /proc/irq/*/smp_affinity; do

  echo f > $i

done
```

(Is it supposed to be "f" or "0f" or 15?)

I don't know why this isn't automatic for you. Maybe:

 *Quote:*   

> So if on your system CPU0 handles all interrupts by default, this probably means that APIC configured ambiguously.

 

----------

## SlashBeast

 *PaulBredbury wrote:*   

> I don't know why this isn't automatic for you.

 

It is, but because of HOTPLUG_CPU (as I googled) it does force them on CPU0, the great read http://www.alexonlinux.com/smp-affinity-and-proper-interrupt-handling-in-linux

Long story short, when you have f (00000f), instead of using all cpus, it uses CPU0.

For now I adjusted the routes manualy, to put sata, wifi and sound card on different cores/threads.

```
#!/bin/sh

set_irq() {

   irq="$1"

   case "$2" in

      cpu0)

         cpu=1

      ;;

      cpu1)

         cpu=2

      ;;

      cpu2)

         cpu=4

      ;;

      cpu3)

         cpu=8

      ;;

   esac

   echo "${cpu}" >"/proc/irq/${irq}/smp_affinity"

}

# 17: ehci_hcd:usb2, iwlwifi

set_irq 17 cpu3

# 18: ahci, mmc0, eth0

set_irq 18 cpu2

# 22: snd_hda_intel

set_irq 22 cpu1
```

And after few hours I can see that it does make things better, at least on paper.

```
sabre ~ # ./summarize_irq_stats.awk old_interrupts

cpu0: 14022875

cpu1: 3985454

cpu2: 3778816

cpu3: 4191746

sabre ~ # ./summarize_irq_stats.awk /proc/interrupts 

cpu0: 605714

cpu1: 547066

cpu2: 625991

cpu3: 825810
```

----------

## Ant P.

If you want another (slightly crazy) solution you could try booting with "threadirqs=1", which forces all IRQ handlers possible into separate threads. IIRC the realtime kernel project uses this, so it should be safe.

----------

## dmpogo

 *SlashBeast wrote:*   

>  *PaulBredbury wrote:*   I don't know why this isn't automatic for you. 
> 
> It is, but because of HOTPLUG_CPU (as I googled) it does force them on CPU0, the great read http://www.alexonlinux.com/smp-affinity-and-proper-interrupt-handling-in-linux
> 
> Long story short, when you have f (00000f), instead of using all cpus, it uses CPU0.
> ...

 

That cannot be general situation. I have CONFIG_HOTPLUG_CPU=y on all four different machines I manage, and they all show proper IRQ balancing.

Note also that your reference refers to quite old kernels <= 2.6.24

----------

## SlashBeast

 *dmpogo wrote:*   

> That cannot be general situation. I have CONFIG_HOTPLUG_CPU=y on all four different machines I manage, and they all show proper IRQ balancing.
> 
> Note also that your reference refers to quite old kernels <= 2.6.24

 

Then I am absolutly without idea, because it does keep all devices, acpi and rtc on one cpu. What is interesing I just tested on another sandybridge cpu, i7-2600, and I also see that devices' irq occupate single core.

```
# cat /proc/interrupts 

           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       

  0:         25          0          0          0          0          0          0          0   IO-APIC-edge      timer

  1:        249          0          0          0          0          0          0          0   IO-APIC-edge      i8042

  8:          1          0          0          0          0          0          0          0   IO-APIC-edge      rtc0

  9:          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   acpi

 23:         60          0          0          0          0          0          0          0   IO-APIC-fasteoi   ehci_hcd:usb3, ehci_hcd:usb4

 40:          0          0          0          0          0          0          0          0   PCI-MSI-edge      PCIe PME

 41:          0          0          0          0          0          0          0          0   PCI-MSI-edge      PCIe PME

 42:          0          0          0          0          0          0          0          0   PCI-MSI-edge      PCIe PME

 43:          0          0          0          0          0          0          0          0   PCI-MSI-edge      PCIe PME

 44:   98076556          0          0          0          0          0          0          0   PCI-MSI-edge      ahci

 45:   90452375          0          0          0          0          0          0          0   PCI-MSI-edge      eth0

 46:         63          0          0          0          0          0          0          0   PCI-MSI-edge      i915

 47:          0          0          0          0          0          0          0          0   PCI-MSI-edge      xhci_hcd

 48:          0          0          0          0          0          0          0          0   PCI-MSI-edge      xhci_hcd

 49:          0          0          0          0          0          0          0          0   PCI-MSI-edge      xhci_hcd

 50:          0          0          0          0          0          0          0          0   PCI-MSI-edge      xhci_hcd

 51:          0          0          0          0          0          0          0          0   PCI-MSI-edge      xhci_hcd

 52:          0          0          0          0          0          0          0          0   PCI-MSI-edge      xhci_hcd

 53:          0          0          0          0          0          0          0          0   PCI-MSI-edge      xhci_hcd

 54:          0          0          0          0          0          0          0          0   PCI-MSI-edge      xhci_hcd

NMI:     218863     198034     192376     194477      42378      50328      52237      54105   Non-maskable interrupts

LOC: 1401606099 1421191629 1426138094 1432049557  479262595  490584201  490445234  478404349   Local timer interrupts

SPU:          0          0          0          0          0          0          0          0   Spurious interrupts

PMI:     218863     198034     192376     194477      42378      50328      52237      54105   Performance monitoring interrupts

IWI:          0          0          0          0          0          0          0          0   IRQ work interrupts

RTR:          7          0          0          0          0          0          0          0   APIC ICR read retries

RES:   26000157    9166845    2035661    1303693     788125     845999     904785     973690   Rescheduling interrupts

CAL:    7014953    7022613    7070873    7139248     670430     726545     757997     763109   Function call interrupts

TLB:    7950685    7524917    7394655    7322978    5088431    5147306    5228957    5433623   TLB shootdowns

TRM:          0          0          0          0          0          0          0          0   Thermal event interrupts

THR:          0          0          0          0          0          0          0          0   Threshold APIC interrupts

MCE:          0          0          0          0          0          0          0          0   Machine check exceptions

MCP:       8683       8683       8683       8683       8683       8683       8683       8683   Machine check polls

ERR:          0
```

And thats on 3.4.x kernel with debian's config.

 *Ant P. wrote:*   

> If you want another (slightly crazy) solution you could try booting with "threadirqs=1", which forces all IRQ handlers possible into separate threads. IIRC the realtime kernel project uses this, so it should be safe.

 

There is so little info about threadirqs, I wouild like to know about some cons and pros of it.

----------

## _______0

that's so weird, it should be automagically spreaded out over irqs.

I suspect bad kernel config, only explanation. And why are u using a Debian .config??

There shouldn't be done any manual intervention to spread out interrupts on multiple cores.

Also why don't you put i915, iwlwifi, eth0, snd_hda_intel on MSI irq? Those are doable, prolly ahci and mmc0 as well.

----------

## BillWho

SlashBeast,

I've never checked this until I read through this thread. Interestingly enough on a Intel(R) Core(TM) i3-2350M CPU @ 2.30GHz laptop:

```
laptop linux # cat /proc/interrupts 

           CPU0       CPU1       CPU2       CPU3       

  0:        122          0          0          0   IO-APIC-edge      timer

  1:       4959          0          0          0   IO-APIC-edge      i8042

  8:          1          0          0          0   IO-APIC-edge      rtc0

  9:          0          0          0          0   IO-APIC-fasteoi   acpi

 12:     214633          0          0          0   IO-APIC-edge      i8042

 16:      38356          0          0          0   IO-APIC-fasteoi   ehci_hcd:usb1

 19:     242556          0          0          0   IO-APIC-fasteoi   ath9k

 23:      82206          0          0          0   IO-APIC-fasteoi   ehci_hcd:usb2

 41:      78453          0          0          0   PCI-MSI-edge      ahci

 42:          0          0          0          0   PCI-MSI-edge      eth0

 43:        552          0          0          0   PCI-MSI-edge      snd_hda_intel

 44:         88          0          0          0   PCI-MSI-edge      i915

NMI:          0          0          0          0   Non-maskable interrupts

LOC:    1073823     938123     908729     929452   Local timer interrupts

SPU:          0          0          0          0   Spurious interrupts

PMI:          0          0          0          0   Performance monitoring interrupts

IWI:          0          0          0          0   IRQ work interrupts

RTR:          2          0          0          0   APIC ICR read retries

RES:     259554      39511      30216      21274   Rescheduling interrupts

CAL:     257645     252898     233423     233077   Function call interrupts

TLB:          0          0          0          0   TLB shootdowns

TRM:          0          0          0          0   Thermal event interrupts

THR:          0          0          0          0   Threshold APIC interrupts

MCE:          0          0          0          0   Machine check exceptions

MCP:         35         35         35         35   Machine check polls

ERR:          0

MIS:          0

```

On a AMD Phenom(tm) 9150e Quad-Core Processor desktop:

```
stable ~ # cat /proc/interrupts 

           CPU0       CPU1       CPU2       CPU3       

  0:        118          0          0          0   IO-APIC-edge      timer

  1:          0          0          0          2   IO-APIC-edge      i8042

  7:          1          0          0          0   IO-APIC-edge    

  8:          0          0          0          1   IO-APIC-edge      rtc0

  9:          0          0          0          0   IO-APIC-fasteoi   acpi

 12:          0          0          0          4   IO-APIC-edge      i8042

 16:          0          0          6       1898   IO-APIC-fasteoi   ohci_hcd:usb3, ohci_hcd:usb4, snd_hda_intel

 17:          0          0          0          3   IO-APIC-fasteoi   ehci_hcd:usb1

 18:         12         86       1512    1968847   IO-APIC-fasteoi   ohci_hcd:usb5, ohci_hcd:usb6, ohci_hcd:usb7, radeon

 19:          2          8        390      62737   IO-APIC-fasteoi   ehci_hcd:usb2, snd_hda_intel

 22:          0          3        427      48467   IO-APIC-fasteoi   ahci

 42:          0          2        134      23787   PCI-MSI-edge      eth0

NMI:          0          0          0          0   Non-maskable interrupts

LOC:    1578605    2145196    1964563    2957637   Local timer interrupts

SPU:          0          0          0          0   Spurious interrupts

PMI:          0          0          0          0   Performance monitoring interrupts

IWI:          0          0          0          0   IRQ work interrupts

RTR:          0          0          0          0   APIC ICR read retries

RES:    2728849     898339     755100     712052   Rescheduling interrupts

CAL:      52242      85795      65919      59867   Function call interrupts

TLB:          0          0          0          0   TLB shootdowns

TRM:          0          0          0          0   Thermal event interrupts

THR:          0          0          0          0   Threshold APIC interrupts

MCE:          0          0          0          0   Machine check exceptions

MCP:         27         27         27         27   Machine check polls

ERR:         13

MIS:          0

```

Both are running version 3.6.11-gentoo kernel with similar config settings except of course for specific hardware differences. I have absolutely  no clue   :Confused: 

----------

## _______0

all my irqs are evenly spreaded over all the cores and the same number. Let's say the sound card is around 200 accross all the cores. And NOT 2 in one, 400 in another and 502342 in another and 0 in another.

This can't be that hard, double check these sections in menuconfig:

```
General setup -->

Processor type and features -->

```

Boot with systemrescuecd, the latest release and check its behaviour.

----------

## SlashBeast

 *_______0 wrote:*   

> I suspect bad kernel config, only explanation. And why are u using a Debian .config??

 

Because its a Debian.

 *_______0 wrote:*   

> all my irqs are evenly spreaded over all the cores and the same number. Let's say the sound card is around 200 accross all the cores. And NOT 2 in one, 400 in another and 502342 in another and 0 in another.
> 
> This can't be that hard, double check these sections in menuconfig:
> 
> ```
> ...

 

Check for?

----------

## _______0

 *SlashBeast wrote:*   

> 
> 
> Because its a Debian.
> 
> 

 

I fail to see how's that's a gentoo problem then.

All the /proc/interrupts pasted here look plain wrong.

This is how ALL devices should appear like:

```
 16:        257        257        258 IO-APIC-fasteoi   snd_hda_intel
```

----------

## BillWho

SlashBeast.

Here's a dual Intel(R) Xeon(R) CPU 5160  @ 3.00GHz

```
ws490-gentoo ~ # cat /proc/interrupts 

            CPU0       CPU1       CPU2       CPU3       

   0:        163          0          0          0   IO-APIC-edge      timer

   1:          1          1          0          0   IO-APIC-edge      i8042

   8:          5          5          3          2   IO-APIC-edge      rtc0

   9:          0          0          0          0   IO-APIC-fasteoi   acpi

  12:          1          0          1          2   IO-APIC-edge      i8042

  14:        509        520        523        515   IO-APIC-edge      ata_piix

  15:          0          0          0          0   IO-APIC-edge      ata_piix

  16:         78         69         81         76   IO-APIC-fasteoi   nouveau

  18:          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb4

  20:       1480       1480       1475       1476   IO-APIC-fasteoi   ahci

  21:       7954       7958       7949       7960   IO-APIC-fasteoi   ehci_hcd:usb1, uhci_hcd:usb2

  22:          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb3

  23:          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb5

  65:         37         35         36         35   PCI-MSI-edge      snd_hda_intel

  66:          1          0          0          2   PCI-MSI-edge      eth0

 NMI:          0          0          0          0   Non-maskable interrupts

 LOC:      15559      16094      16099      15744   Local timer interrupts

 SPU:          0          0          0          0   Spurious interrupts

 PMI:          0          0          0          0   Performance monitoring interrupts

 IWI:          0          0          0          0   IRQ work interrupts

 RTR:          0          0          0          0   APIC ICR read retries

 RES:       3554       2081       2352       1626   Rescheduling interrupts

 CAL:       1455       1445       2263       2244   Function call interrupts

 TLB:          0          0          0          0   TLB shootdowns

 TRM:          0          0          0          0   Thermal event interrupts

 THR:          0          0          0          0   Threshold APIC interrupts

 MCE:          0          0          0          0   Machine check exceptions

 MCP:          1          1          1          1   Machine check polls

 ERR:          0

 MIS:          0

```

It's a mystery why the I3 and I5 differ so drastically  :Confused: 

----------

## _______0

odd. Can you paste the configs of all systems to narrow down the problem?

Mm.. according to some intarwebz comments could be two things:

Turn these two off:

```
CONFIG_HOTPLUG_CPU=y

CONFIG_ACPI_HOTPLUG_CPU=y

```

I doubt this cuz my IRQs are all distributed with these options on.

A comment on this post http://www.alexonlinux.com/smp-affinity-and-proper-interrupt-handling-in-linux mentions,

```
CONFIG_X86_X2APIC
```

which seems more correct. Try the later one.

And maybe this as well?

```
CONFIG_IRQ_REMAP
```

Last edited by _______0 on Mon Dec 31, 2012 12:05 am; edited 1 time in total

----------

## BillWho

 *_______0 wrote:*   

> odd. Can you paste the configs of all systems to narrow down the problem?

 

_______0,

For all it's worth:

Core(TM) i3-2350M CPU @ 2.30GHz laptop http://bpaste.net/show/67350/

AMD Phenom(tm) 9150e Quad-Core Processor desktop  http://bpaste.net/show/67351/

Dual Xeon(R) CPU 5160 @ 3.00GHz http://bpaste.net/show/67352/

The xeon machine is old and neglected. I refreshed it with a clean install several months ago.

It was just intended to be used to backup files from the AMD and laptop, but over time, it morphed into a crippled desktop - maybe neglect is the secret   :Very Happy: 

----------

## _______0

I think I found it:

```
CONFIG_SMP
```

try that one along with:

```
CONFIG_X86_X2APIC
```

and possibly try this in kernel boot params:

 *Quote:*   

> x2apic_phys     [X86-64,APIC] Use x2apic physical mode instead of
> 
>                         default x2apic cluster mode on platforms
> 
>                         supporting x2apic.
> ...

 

----------

## _______0

try to set this one on:

```
# CONFIG_IRQ_REMAP is not set
```

----------

## SlashBeast

 *_______0 wrote:*   

>  *SlashBeast wrote:*   
> 
> Because its a Debian.
> 
>  
> ...

 .

Uh huh... I said I saw it too on  i7 sandy bridge debian. I do run into this issue regardless of distro on newer intel chips.

 *_______0 wrote:*   

> This is how ALL devices should appear like:
> 
> ```
>  16:        257        257        258 IO-APIC-fasteoi   snd_hda_intel
> ```
> ...

 

You don't say!

And of course every and each of the kernels here have SMP support enabled, without it you can't even see more than CPU0 ...

----------

## BillWho

_______0,

I had high hopes, but:

```
laptop linux # diff .config .config.save

314d313

< CONFIG_X86_X2APIC=y

2442c2441

< CONFIG_IRQ_REMAP=y

---

> # CONFIG_IRQ_REMAP is not set

```

Config changes:

```
Device drivers

   IOMMU Hardware Support

      Support for Interrupt Remapping (EXPERIMENTAL)

Processor type and features

    Support x2apic
```

And the results:

```
bill@laptop ~ $ cat /proc/interrupts 

           CPU0       CPU1       CPU2       CPU3       

  0:        121          0          0          0   IO-APIC-edge      timer

  1:         82          0          0          0   IO-APIC-edge      i8042

  8:          1          0          0          0   IO-APIC-edge      rtc0

  9:          0          0          0          0   IO-APIC-fasteoi   acpi

 12:       1363          0          0          0   IO-APIC-edge      i8042

 16:        540          0          0          0   IO-APIC-fasteoi   ehci_hcd:usb1

 19:        827          0          0          0   IO-APIC-fasteoi   ath9k

 23:        389          0          0          0   IO-APIC-fasteoi   ehci_hcd:usb2

 41:       6889          0          0          0   PCI-MSI-edge      ahci

 42:          0          0          0          0   PCI-MSI-edge      eth0

 43:        551          0          0          0   PCI-MSI-edge      snd_hda_intel

 44:         13          0          0          0   PCI-MSI-edge      i915

NMI:          0          0          0          0   Non-maskable interrupts

LOC:      19254      13568      11128      11235   Local timer interrupts

SPU:          0          0          0          0   Spurious interrupts

PMI:          0          0          0          0   Performance monitoring interrupts

IWI:          0          0          0          0   IRQ work interrupts

RTR:          2          0          0          0   APIC ICR read retries

RES:        954        527        607        284   Rescheduling interrupts

CAL:        906        891        864        787   Function call interrupts

TLB:          0          0          0          0   TLB shootdowns

TRM:          0          0          0          0   Thermal event interrupts

THR:          0          0          0          0   Threshold APIC interrupts

MCE:          0          0          0          0   Machine check exceptions

MCP:          2          2          2          2   Machine check polls

ERR:          0

MIS:          0

```

Well, I'm still  :Crying or Very sad:  Good try though   :Very Happy: 

----------

## _______0

woah! This is getting more interesting by the minute!!

Although the ultimate option might be change to amd proc for proper /proc/interrupt sake, until then we should find out WHY late intels have this odd behaviour.

lol@this:

```
IWI:          0          0          0          0   IRQ work interrupts 
```

On mine there's activity, and evenly spread out (as proper) over cores.

I wonder whether has to do with timer used:

```
cat /sys/bus/clocksource/devices/clocksource0/current_clocksource
```

Could you try booting with these kernel parameters??

```
hpet=force x2apic_phys
```

ps: how did you put ahci on an MSI interrupt??

ps2: what about some un-obvious BIOS/UEFI settings?

This gentoo fella https://forums.gentoo.org/viewtopic-t-937042-start-0.html has it right on his i5, according to his uname but his sig says i7:

 *eccerr0r wrote:*   

> Is it like this on fresh boot?  Single user mode?
> 
> Do you have any special hardware?  Can the drivers be removed one at a time to see if any of the drivers require frequent interrupts by the clock tick timer?
> 
> Do you have HPET enabled on the newer machines? APIC?
> ...

 

----------

## BillWho

_______0,

```
bill@laptop /proc $ cat cmdline

BOOT_IMAGE=/boot/kernel root=/dev/sda8 rootfstype=ext3 hpet=force x2apic_phys ro

bill@laptop ~ $ cat /proc/interrupts 

           CPU0       CPU1       CPU2       CPU3       

  0:        122          0          0          0   IO-APIC-edge      timer

  1:         54          0          0          0   IO-APIC-edge      i8042

  8:          1          0          0          0   IO-APIC-edge      rtc0

  9:          0          0          0          0   IO-APIC-fasteoi   acpi

 12:        487          0          0          0   IO-APIC-edge      i8042

 16:        436          0          0          0   IO-APIC-fasteoi   ehci_hcd:usb1

 19:       1643          0          0          0   IO-APIC-fasteoi   ath9k

 23:        270          0          0          0   IO-APIC-fasteoi   ehci_hcd:usb2

 41:       6538          0          0          0   PCI-MSI-edge      ahci

 42:          0          0          0          0   PCI-MSI-edge      eth0

 43:         12          0          0          0   PCI-MSI-edge      i915

 44:        550          0          0          0   PCI-MSI-edge      snd_hda_intel

NMI:          0          0          0          0   Non-maskable interrupts

LOC:      17806      10814      11136      10214   Local timer interrupts

SPU:          0          0          0          0   Spurious interrupts

PMI:          0          0          0          0   Performance monitoring interrupts

IWI:          0          0          0          0   IRQ work interrupts

RTR:          2          0          0          0   APIC ICR read retries

RES:       1159        518        432        222   Rescheduling interrupts

CAL:        572        554        567        492   Function call interrupts

TLB:          0          0          0          0   TLB shootdowns

TRM:          0          0          0          0   Thermal event interrupts

THR:          0          0          0          0   Threshold APIC interrupts

MCE:          0          0          0          0   Machine check exceptions

MCP:          2          2          2          2   Machine check polls

ERR:          0

MIS:          0

```

No cigar here   :Confused: 

UPDATE: Almost forgot

```
bill@laptop /proc $ cat /sys/bus/clocksource/devices/clocksource0/current_clocksource

tsc

```

----------

## _______0

 *BillWho wrote:*   

> 
> 
> UPDATE: Almost forgot
> 
> ```
> ...

 

oops I forgot this to add in kernel parameters:

```
clocksource=hpet
```

In dmesg you should see this:

```
Switching to clocksource hpet
```

Mmm... what does dmesg say about x2apic?? I am curious.

Also i8042 can be disabled in BIOS/UEFI (serial/parallel port) and in kernel config uncheck keyboard and ps/m mouse, no modern machine use none of those. Safe to get rid of them.

----------

## BillWho

_______0,

```
laptop proc # dmesg|grep hpet

[    0.000000] Command line: BOOT_IMAGE=/boot/kernel root=/dev/sda8 rootfstype=ext3 hpet=force x2apic_phys ro

[    0.000000] Kernel command line: BOOT_IMAGE=/boot/kernel root=/dev/sda8 rootfstype=ext3 hpet=force x2apic_phys ro

[    0.000000] hpet clockevent registered

[    0.251867] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0

[    0.252243] hpet0: 8 comparators, 64-bit 14.318180 MHz counter

[    0.254325] Switching to clocksource hpet

[    0.546737] rtc0: alarms up to one month, y3k, 242 bytes nvram, hpet irqs

```

 *_______0 wrote:*   

> ps: how did you put ahci on an MSI interrupt?? 

 

No idea, I didn't change anything.

 *_______0 wrote:*   

> ps2: what about some un-obvious BIOS/UEFI settings? 

 

I'll double check, but I doubt it.

What is the kernel parameters - is it hpet=force x2apic_phys clocksource=hpet   :Question: 

----------

## _______0

 *BillWho wrote:*   

> 
> 
> UPDATE: Almost forgot
> 
> ```
> ...

 

This:

```
hpet=force x2apic_phys clocksource=hpet
```

to change your clocksource to hpet

----------

## BillWho

_______0,

```
laptop proc # cat /sys/bus/clocksource/devices/clocksource0/current_clocksource

hpet

laptop proc # cat cmdline 

BOOT_IMAGE=/boot/kernel root=/dev/sda8 rootfstype=ext3 hpet=force x2apic_phys clocksource=hpet ro

laptop proc # dmesg|grep x2apic

[    0.000000] Command line: BOOT_IMAGE=/boot/kernel root=/dev/sda8 rootfstype=ext3 hpet=force x2apic_phys clocksource=hpet ro

[    0.000000] Kernel command line: BOOT_IMAGE=/boot/kernel root=/dev/sda8 rootfstype=ext3 hpet=force x2apic_phys clocksource=hpet ro

laptop proc # dmesg|grep hpet

[    0.000000] Command line: BOOT_IMAGE=/boot/kernel root=/dev/sda8 rootfstype=ext3 hpet=force x2apic_phys clocksource=hpet ro

[    0.000000] Kernel command line: BOOT_IMAGE=/boot/kernel root=/dev/sda8 rootfstype=ext3 hpet=force x2apic_phys clocksource=hpet ro

[    0.000000] hpet clockevent registered

[    0.250202] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0

[    0.250576] hpet0: 8 comparators, 64-bit 14.318180 MHz counter

[    0.252657] Switching to clocksource hpet

[    0.544998] rtc0: alarms up to one month, y3k, 242 bytes nvram, hpet irqs

```

No change to interrupts   :Rolling Eyes: 

I'm starting to think this is processor related. I'm going to boot mint/debain and check it   :Confused: 

UPDATE: I booted lmde and there was no difference with /proc/interrupts - all on cpu0   :Confused: 

----------

## kerframil

If you have more than two cores, use sys-apps/irqbalance in conjunction with a >=3.3 kernel. I obtain exceptional results with it for spreading softirq load generated by a busy multiqueue NIC on a 12 core system. If your system is NUMA capable then be sure to set the corresponding USE flag, which is applicable to Opteron CPUs and some Intel CPUs as of the Nehalem microarchitecture. Some distros actually install and enable irqbalance by default, such as Ubuntu.

There is also an alternative going by the name of irqd. I have no experience of using this particular application.

EDIT: This is somewhat tangential but worth mentioning: RPS (receive packet steering) can be useful for increasing network scalability. I manage some Java application servers for a busy online service and, without IRQ balancing and RPS, each server would simply not be able to cope with its workload. However, I would not recommend this for general usage. For those that might benefit, here is a script to enable it. It accepts the name of an ethernet interface as its argument:

```
#!/bin/bash

# https://lwn.net/Articles/361440/

# http://thread.gmane.org/gmane.linux.network/179883/focus=179976

set -e

[[ $IFACE ]] || IFACE="$1"

regex='^eth([[:digit:]]+|\*)$'

[[ $IFACE =~ $regex ]] || { echo "Invalid interface: $IFACE"; exit 1; }

flow=16384

echo $flow > /proc/sys/net/core/rps_sock_flow_entries

queues=(/sys/class/net/$IFACE/queues/rx-*)

for rx in "${queues[@]}"; do

        echo $(sed -e 's/0/f/g' < $rx/rps_cpus) > $rx/rps_cpus

        echo $flow > $rx/rps_flow_cnt

done

logger -t "$(basename $0)" "Enabled RFS/RPS for ${#queues[@]} queues on $IFACE"
```

----------

## roravun

Before I use the knowledge contained in this thread, I would like to ask if messages like 

 *Quote:*   

>  Interrupt took horrible_amount_of_ns_order_of_K 

 

may be caused by unballanced IRQs.

Of course as I discovered, my proc interrupts looks like this:

```
                     CPU0       CPU1       

  0:  107595256          3   IO-APIC-edge      timer

  1:     706460          0   IO-APIC-edge      i8042

  8:          1          0   IO-APIC-edge      rtc0

  9:      46059          0   IO-APIC-fasteoi   acpi

 12:   10466748          0   IO-APIC-edge      i8042

 14:      39423          0   IO-APIC-edge      ata_piix

 15:          0          0   IO-APIC-edge      ata_piix

 16:     589526          0   IO-APIC-fasteoi   uhci_hcd:usb1

 17:          0          0   IO-APIC-fasteoi   uhci_hcd:usb2

 18:      97973          0   IO-APIC-fasteoi   uhci_hcd:usb3

 19:          0          0   IO-APIC-fasteoi   uhci_hcd:usb4

 40:          0          0   PCI-MSI-edge      PCIe PME

 41:          0          0   PCI-MSI-edge      PCIe PME

 42:          0          0   PCI-MSI-edge      PCIe PME

 43:          0          0   PCI-MSI-edge      PCIe PME

 44:          0          0   PCI-MSI-edge      PCIe PME

 45:    2703873          0   PCI-MSI-edge      ahci

 46:         96          0   PCI-MSI-edge      snd_hda_intel

 47:        806          0   PCI-MSI-edge      eth0

 48:   32929731          0   PCI-MSI-edge      iwl3945

 49:   36243940          0   PCI-MSI-edge      radeon

NMI:          0          0   Non-maskable interrupts

LOC:   77904422  134472172   Local timer interrupts

SPU:          0          0   Spurious interrupts

PMI:          0          0   Performance monitoring interrupts

IWI:          0          0   IRQ work interrupts

RTR:          0          0   APIC ICR read retries

RES:   90040347   90498321   Rescheduling interrupts

CAL:       3388      12283   Function call interrupts

TLB:    1964560    2268016   TLB shootdowns

TRM:          0          0   Thermal event interrupts

THR:          0          0   Threshold APIC interrupts

MCE:          0          0   Machine check exceptions

MCP:       1345       1330   Machine check polls

ERR:          0

MIS:          0

```

Also I propose this thread be made sticky. Who knows how many systems with unbalanced irqs are there  :Wink: 

----------

## kerframil

 *roravun wrote:*   

> Before I use the knowledge contained in keframil post, I would like to ask if messages like 
> 
>  *Quote:*    Interrupt took horrible_amount_of_ns_order_of_K  
> 
> may be caused by unballanced IRQs.

 

This is a different matter and related to the use of high-resolution timers. The circumstances under which that message is triggered are described here:

https://git.kernel.org/?p=linux/kernel/git/tip/tip.git;a=commitdiff;h=41d2e49

Regarding my post, I should add that spreading IRQ load with a perfectly even distribution isn't a realistic goal. Nor does interrupt spreading necessarily enhance performance (to the contrary in some cases). Power usage must also be taken into consideration; there's little sense in waking up cores that might otherwise be idle or parked unless there is a clear benefit in doing so. I wouldn't want people looking at /proc/interrupts and thinking that there is something intrinsically 'wrong'.

It's better to think of it as alleviating a potential bottleneck in situations where softirq load is high. When running as a daemon, irqbalance continually adjusts the affinity maps based on the amount of load, among other factors.

----------

## roravun

```
This is a different matter and related to the use of high-resolution timers. The circumstances under which that message is triggered are described here: 
```

I have just discovered I  somehow disabled hpet, so it could be the cause.

As to the interrupts, I will try to move at least sound card to the other CPU. Sometimes when playing games (the CPU0 is 100% busy) sound crackles or lags a bit. When I used pulseaudio the problem was even more evident.

----------

## PaulBredbury

 *roravun wrote:*   

> sound crackles or lags a bit

 

I recommend raising the interrupt priority. Doesn't need a realtime kernel.

----------

## _______0

 *kerframil wrote:*   

> Power usage must also be taken into consideration; there's little sense in waking up cores that might otherwise be idle or parked unless there is a clear benefit in doing so. I wouldn't want people looking at /proc/interrupts and thinking that there is something intrinsically 'wrong'.

 

Some ppl have perfectly even irq balancing among cores. The case to save power would be fine with laptops even then, how do you determine which one is best? balanced or un-balanced?

But the mistery here is why and what makes irqs to be evenly distributed over cores and why some systems don't. I would like to know which piece in the kernel and option in makeconfig is taking this decision and a way to see /proc/interrupts either unbalanced and balanced in a controlled manner. There's gotta be an explanation.

On this system irqs are balanced out of the box, no need for irqbalance nor irqd.

And the fact that irqs are being spread in tandem means that something is doing a pretty damn well job in the kernel.

Sound card example:

```
16:       1985       1972       1990       1992
```

I still want to turn on NMI and PMI

----------

