# Hard drives becoming really slow ... IRQ 19: nobody cared

## anotherguy

UPDATE 4/6: So you don't have to read through it all... IRQ 19 crashes.  Hard drive isn't crippled anymore because I enabled MSI in the kernel.  Other devices on IRQ 19 still crashing.  Please read latest post.

I have this odd problem.  My hard drives suddenly become extremely, painfully, horribly slow!  It's like something somewhere crashes and I'm left with this crippled system where all programs and the X server still work, but any accesses made to the hard drive are much slower.  The system ran fine for a couple days this time before the "crash" occurred.

If I run hdparm after a reboot I get:

hdparm -tT /dev/sda5

```

/dev/sda5:

 Timing cached reads:   2162 MB in  2.00 seconds = 1081.21 MB/sec

 Timing buffered disk reads:    8 MB in  3.19 seconds =   79.68 MB/sec

```

Then, once the "crash" happens (usually after a couple days), I get:

```

hdparm -tT /dev/sda5

/dev/sda5:

 Timing cached reads:   2162 MB in  2.00 seconds = 1081.21 MB/sec

 Timing buffered disk reads:    8 MB in  3.19 seconds =   2.50 MB/sec

```

I know this is a SATA drive, but the numbers reflect the experience I'm having.  

The motherboard is Gigabyte-P35-DS3L.  With a SATA controller and jmicron kernel module installed.  After crash a run of lspci shows me:

```

# lspci -v

00:1f.2 SATA controller: Intel Corporation 82801IB (ICH9) 4 port SATA AHCI Controller (rev 02) (prog-if 01 [AHCI 1.0])

   Subsystem: Giga-byte Technology Device b005

   Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 19

   I/O ports at e600 [size=8]

   I/O ports at e700 [size=4]

   I/O ports at e800 [size=8]

   I/O ports at e900 [size=4]

   I/O ports at ea00 [size=32]

   Memory at fb206000 (32-bit, non-prefetchable) [size=2K]

   Capabilities: [80] Message Signalled Interrupts: Mask- 64bit- Count=1/16 Enable-

   Capabilities: [70] Power Management version 3

   Capabilities: [a8] SATA HBA <?>

   Capabilities: [b0] PCIe advanced features <?>

   Kernel driver in use: ahci

03:00.0 IDE interface: JMicron Technologies, Inc. JMB368 IDE controller (prog-if 85 [Master SecO PriO])

   Subsystem: Giga-byte Technology Device b000

   Flags: bus master, fast devsel, latency 0, IRQ 19

   I/O ports at c000 [size=8]

   I/O ports at c100 [size=4]

   I/O ports at c200 [size=8]

   I/O ports at c300 [size=4]

   I/O ports at c400 [size=16]

   Capabilities: [68] Power Management version 2

   Capabilities: [50] Express Legacy Endpoint, MSI 01

   Kernel driver in use: pata_jmicron

```

```

uname -a

Linux 2.6.28-gentoo-r1 #1 SMP PREEMPT Wed Feb 25 21:46:39 CST 2009 i686 Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz GenuineIntel GNU/Linux

```

emerge --info takes a several minutes to run.  My system is not fast enough to play videos anymore as it cannot transfer fast enough from the hard drive.

```

# emerge --info

Portage 2.1.6.7 (default/linux/x86/2008.0/desktop, gcc-4.1.2, glibc-2.6.1-r0, 2.6.28-gentoo-r1 i686)

=================================================================

System uname: Linux-2.6.28-gentoo-r1-i686-Intel-R-_Core-TM-2_Quad_CPU_Q6600_@_2.40GHz-with-glibc2.0

Timestamp of tree: Sun, 15 Mar 2009 06:45:02 +0000

ccache version 2.4 [enabled]

app-shells/bash:     3.2_p39

dev-java/java-config: 1.3.7-r1, 2.1.6-r1

dev-lang/python:     2.5.2-r7

dev-python/pycrypto: 2.0.1-r8

dev-util/ccache:     2.4-r7

dev-util/cmake:      2.4.8

sys-apps/baselayout: 1.12.11.1

sys-apps/sandbox:    1.2.18.1-r2

sys-devel/autoconf:  2.13, 2.63

sys-devel/automake:  1.4_p6, 1.5, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10.2

sys-devel/binutils:  2.18-r3

sys-devel/gcc-config: 1.4.0-r4

sys-devel/libtool:   1.5.26

virtual/os-headers:  2.6.27-r2

ACCEPT_KEYWORDS="x86"

CBUILD="i686-pc-linux-gnu"

CFLAGS="-march=prescott -02 -pipe -fomit-frame-pointer"

CHOST="i686-pc-linux-gnu"

CONFIG_PROTECT="/etc /usr/kde/3.5/env /usr/kde/3.5/share/config /usr/kde/3.5/shutdown /usr/share/config"

CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/php/apache2-php5/ext-active/ /etc/php/cgi-php5/ext-active/ /etc/php/cli-php5/ext-active/ /etc/revdep-rebuild /etc/splash /etc/terminfo /etc/texmf/web2c /etc/udev/rules.d"

CXXFLAGS="-march=prescott -02 -pipe -fomit-frame-pointer"

DISTDIR="/usr/portage/distfiles"

FEATURES="ccache collision-protect distlocks fixpackages parallel-fetch protect-owned sandbox sfperms strict unmerge-orphans userfetch"

GENTOO_MIRRORS="ftp://gentoo.mirrors.tds.net/gentoo http://gentoo.mirrors.tds.net/gentoo http://mirror.datapipe.net/gentoo ftp://mirror.datapipe.net/gentoo http://194.117.143.71"

LDFLAGS="-Wl,-O1"

MAKEOPTS="-j5"

PKGDIR="/usr/portage/packages"

PORTAGE_CONFIGROOT="/"

PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"

PORTAGE_TMPDIR="/var/tmp"

PORTDIR="/usr/portage"

PORTDIR_OVERLAY="/usr/portage/local/layman/jokey /usr/local/portage /usr/local/sci"

SYNC="rsync://rsync.namerica.gentoo.org/gentoo-portage"

USE="3dnow 3dnowext X acl acpi alsa berkdb bluetooth branding bzip2 cairo cdr cli cracklib crypt cups dbus dri dvd dvdr dvdread eds emboss encode esd evo fam firefox fortran gdbm gif gnome gpm gstreamer gtk hal iconv ipv6 isdnlog jack jpeg kde ldap libnotify mad midi mikmod mmx mmxext mp3 mpeg mudflap ncurses nls nptl nptlonly ogg opengl openmp pam pcre pdf perl png ppds pppd python qt3 qt3support qt4 quicktime readline reflection sdl session spell spl sse ssl startup-notification svg sysfs tcpd tiff truetype unicode usb vorbis win32codecs wxwindows x86 xml xorg xulrunner xv zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1 emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic auth_digest authn_anon authn_dbd authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock dbd deflate dir disk_cache env expires ext_filter file_cache filter headers ident imagemap include info log_config logio mem_cache mime mime_magic negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_http rewrite setenvif so speling status unique_id userdir usertrack vhost_alias" ELIBC="glibc" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" USERLAND="GNU" VIDEO_CARDS="radeon vga vesa"

Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, FFLAGS, INSTALL_MASK, LANG, LC_ALL, LINGUAS, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS

```

When checking my logs (/var/log/mesages) I get the following, which seems to have triggered this problem:

```

Mar 18 12:59:16 hostname irq 19: nobody cared (try booting with the "irqpoll" option)

Mar 18 12:59:16 hostname Pid: 0, comm: swapper Tainted: P           2.6.28-gentoo-r1 #1

Mar 18 12:59:16 hostname Call Trace:

Mar 18 12:59:16 hostname [<c0149924>] __report_bad_irq+0x24/0x90

Mar 18 12:59:16 hostname [<c0149af9>] note_interrupt+0x169/0x1b0

Mar 18 12:59:16 hostname [<c0148735>] handle_IRQ_event+0x25/0x60

Mar 18 12:59:16 hostname[<c014a52b>] handle_fasteoi_irq+0xbb/0xf0

Mar 18 12:59:16 hostname [<c0105b0d>] do_IRQ+0x4d/0x90

Mar 18 12:59:16 hostname [<c0103e13>] common_interrupt+0x23/0x28

Mar 18 12:59:16 hostname [<c0109712>] mwait_idle+0x32/0x40

Mar 18 12:59:16 hostname [<c0101d76>] cpu_idle+0x46/0xa0

Mar 18 12:59:16 hostname handlers:

Mar 18 12:59:16 hostname [<c02f91b0>] (ahci_interrupt+0x0/0x500)

Mar 18 12:59:16 hostname [<c02f44a0>] (ata_sff_interrupt+0x0/0x1b0)

Mar 18 12:59:16 hostname [<c0305900>] (usb_hcd_irq+0x0/0x70)

Mar 18 12:59:16 hostname [<c037f0e0>] (snd_cs46xx_interrupt+0x0/0x1e0)

Mar 18 12:59:16 hostname Disabling IRQ #19

```

Any guidance?

[EDIT 1]

I forgot to mention that I have no boot options in my grub.  I'm googling this irqpoll boot option to see if this could somehow be a solution, but it seems it slows the computer down??  

I also am not messing with the hardware when this happens like inserting a USB drive or anything like that.  I think I was watching a flash video on the internet when it happened.  Any help is appreciated.

[EDIT 2]

Looking in /proc/irq/19:

```

# ls /proc/irq/19

dr-xr-xr-x 2 root root 0 Mar 18 13:59 CS46XX

dr-xr-xr-x 2 root root 0 Mar 18 13:59 ahci

dr-xr-xr-x 2 root root 0 Mar 18 13:59 pata_jmicron

-rw------- 1 root root 0 Mar 18 13:59 smp_affinity

-r--r--r-- 1 root root 0 Mar 18 13:59 spurious

dr-xr-xr-x 2 root root 0 Mar 18 13:59 uhci_hcd:usb7

```

CS46XX is my PCI sound card.

ahci is my SATA controller driver

pata_jmicron is my PATA controller driverLast edited by anotherguy on Mon Apr 06, 2009 11:27 am; edited 2 times in total

----------

## Letharion

Check the drive is well?

```
emerge smartmontools

smartctl -t long /dev/sda5
```

waaait

```
smartctl -l selftest
```

More info:

http://www.cyberciti.biz/tips/linux-find-out-if-harddisk-failing.html

----------

## albright

I had a similar problem. For me, the solution was to enable MSI

(that's Message Signalled Interrupts) in the kernel. The gentoo

handbook says to leave this disabled, but there is some hardware

that requires it. Worth looking into at least.

----------

## anotherguy

albright -

Thanks for the heads up on that.  It easily could have taken me a few hours to stumble across this solution.  I'm giving it a go.  Everything is working well so far.

----------

## anotherguy

UPDATE: I enabled MSI (Message Signalled Interrupts) in the kernel.  After an indiscriminate period of time, and more importantly, it seems like when I'm watching flash videos on the net, the "IRQ crash" happens.  The only difference after enabling MSI is the hard drives don't go down when IRQ 19 crashes.  This at least makes my system usable, which is GOOD.  However my sound is usually sacrificed, which is BAD.  (This also happened before, where all the devices using IRQ 19 crashed, hard drive and sound.) The sound sounds laggy and distorted in Flash videos (as if sound is operating at 1/100th speed).  And is usually barely listenable in XMMS (as if I'm listening to a worn cassette tape).

I then proceeded to enable noirqpoll option to the kernel boot options.  

```
kernel /b root=/dev/sda5 noirqpoll
```

   This didn't work.  

All these changes were using linux-2.6.28-gentoo-r1

I just now changed to linux-2.6.29-gentoo to see if this will help.  I copied .config and 'make oldconfig' accepting all the default options.  Except for CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS: Reroute for broken boot IRQs.  I can't remember the default option for this, but I chose No.

I'll post back if my machine still crashes after upgrading to the newer kernel.

----------

## NeddySeagoon

anotherguy,

You really mean XMMS ?

Thats been dead for years

----------

## anotherguy

NeddySeagoon,

Yup.  I just upgraded my 15 year old car and bought a 6 year old car.  I run XMMS-1.2.10 on my quad-core Q6600 machine.  I use Winamp 2.95 on my dual-core at work.  It does what I need to do.  It's FAST.

Amarok does neat things, and I would use it, but it takes close to 15 MINUTES to load using postgresql as the backend.  It just isn't worth my while to make it work since I have to run other process intensive apps on my computer for work.  I just shuffle through the same 2000 song classic rock library anyway, otherwise I'm listening on my iPhone/Zune.  

Anyway, regardless what I'm running, all sound is a bit crappy.  Can't be this way.

EDIT: I should note, all sound is crappy once the computer crashes.  Been up and running for a several hours now using the 2.6.29 kernel.  ::Crosses fingers::

----------

## Letharion

 *anotherguy wrote:*   

> NeddySeagoon,
> 
> Yup.  I just upgraded my 15 year old car and bought a 6 year old car.  I run XMMS-1.2.10 on my quad-core Q6600 machine.  I use Winamp 2.95 on my dual-core at work.  It does what I need to do.  It's FAST.
> 
> Amarok does neat things, and I would use it, but it takes close to 15 MINUTES to load using postgresql as the backend.  It just isn't worth my while to make it work since I have to run other process intensive apps on my computer for work.  I just shuffle through the same 2000 song classic rock library anyway, otherwise I'm listening on my iPhone/Zune.  
> ...

 

FYI, there's audacious too, which is "supposed to replace XMMS".

(I'm just mentioning it if you would want a media player that's supported by gentoo, but still must more lightweight than amarok)

Btw, where did you get 2.95? I think I'd like that too at work  :Smile:  (I noticed there are several download sites just a google away, do you trust anyone in particular?)

----------

## anotherguy

Letharion,

Thanks for keeping me company until my IRQ 19 crashes again ...

I typically use http://www.oldversion.com/Winamp.html whenever I need to install it on a new machine.  I would still run Avast Antivirus against the .exe.  Just right-click the .exe and hit scan before installing Winamp.

----------

## Letharion

 *anotherguy wrote:*   

> Letharion,
> 
> Thanks for keeping me company until my IRQ 19 crashes again ...
> 
> I typically use http://www.oldversion.com/Winamp.html whenever I need to install it on a new machine.  I would still run Avast Antivirus against the .exe.  Just right-click the .exe and hit scan before installing Winamp.

 

I'd don't mind  :Wink: 

I just wish I could fix your problem.

Ok, wise enough.

----------

## anotherguy

Surprise ...

IRQ 19 just crashed.

```
# tail /var/log/messages

Apr  6 06:19:45 hostname irq 19: nobody cared (try booting with the "irqpoll" option)

Apr  6 06:19:45 hostname Pid: 28319, comm: opera Tainted: P           2.6.29-gentoo #1

Apr  6 06:19:45 hostname Call Trace:

Apr  6 06:19:45 hostname [<c014db24>] __report_bad_irq+0x24/0x90

Apr  6 06:19:45 hostname [<c014dcd5>] note_interrupt+0x145/0x180

Apr  6 06:19:45 hostname [<c014cbb5>] handle_IRQ_event+0x25/0x60

Apr  6 06:19:45 hostname [<c014e66b>] handle_fasteoi_irq+0xbb/0xf0

Apr  6 06:19:45 hostname [<c0105553>] do_IRQ+0x43/0x90

Apr  6 06:19:45 hostname [<c012aa38>] sys_gettimeofday+0x28/0x80

Apr  6 06:19:45 hostname [<c01037e7>] common_interrupt+0x27/0x2c

Apr  6 06:19:45 hostname handlers:

Apr  6 06:19:45 hostname [<c03053c0>] (ata_sff_interrupt+0x0/0x1c0)

Apr  6 06:19:45 hostname [<c0316700>] (usb_hcd_irq+0x0/0x70)

Apr  6 06:19:45 hostname [<c0392c70>] (snd_cs46xx_interrupt+0x0/0x1e0)

Apr  6 06:19:45 hostname Disabling IRQ #19

```

Hard drives aren't slowing down though because I enabled MSI in the kernel.

```

 # hdparm -tT /dev/sda

/dev/sda:

 Timing cached reads:   7194 MB in  2.00 seconds = 3600.35 MB/sec

 Timing buffered disk reads:  218 MB in  3.02 seconds =  72.21 MB/sec

```

----------

## Letharion

I noticed something about the irqpoll thing.

Doesn't it say boot WITH irqpoll, and you tried withOUT, as in NOirqpoll?

----------

## mno

Seems like you're not the only one (https://bugzilla.redhat.com/show_bug.cgi?id=449302), but it seems no one so far has any solution.

----------

## milomak

I think my gentoo install had the same error on boot when I had this problem. Unfortunately I won't have access to that computer till Monday to confirm.

----------

## jburns

You could try enabling ACPI in the kernel.  It should change the routing of the interrupts.

----------

## Cyker

Query:

What other devices are on IRQ19; Is it possible one of them is faulty and is just generating loads of interrupts, or stalling the interrupt line some how?

19 is one of the IRQs the kernel just loves assigning things too; I have 5 other IRQs completely free but the damned kernel will only use 19 and 21!! With the broken NE2000-PCI driver I was using, the kernel would frequently lock up that IRQ line (Which the NIC was using) and anything on it which included USB, SATA2 and 3 and all the other NICs.

Thankfully this system uses PS/2 so I was still able to log in and reboot the machine safely.

One other thing you can try, if a) You're only running a single core and b) If you don't have a crappy modern BIOS, is to disable the IO-APIC - This limits IRQs to the standard set (0-15) - and see if that makes any difference.

----------

