# System freeze related to "rcu_sched detected stalls on CPUs"

## rubik-wuerfel

Hello,

I just installed Gentoo on this new machine, and experience a number of system freezes, which seem to be correlated to this output, as recorded by journalctl:

```
Oct 03 20:36:55 droggelbecher kernel: INFO: rcu_sched detected stalls on CPUs/tasks:

Oct 03 20:36:55 droggelbecher kernel:         3: (1 GPs behind) idle=5af/1/0 softirq=71484/71485 fqs=2098 

Oct 03 20:36:55 droggelbecher kernel:         (detected by 1, t=2102 jiffies, g=61577, c=61576, q=9555)

Oct 03 20:36:55 droggelbecher kernel: Task dump for CPU 3:

Oct 03 20:36:55 droggelbecher kernel: swapper/3       R  running task        0     0      1 0x00000008

Oct 03 20:36:55 droggelbecher kernel:  ffffffffffffff4c ffffffff814dd7d8 0000000000000010 0000000000000212

Oct 03 20:36:55 droggelbecher kernel:  ffff88021d963e88 0000000000000018 ffffffff814dd7d4 000000021d960000

Oct 03 20:36:55 droggelbecher kernel:  ffffffffa089efb0 ffffffffa089efb0 ffff88021278e000 ffffffffa089efb0

Oct 03 20:36:55 droggelbecher kernel: Call Trace:

Oct 03 20:36:55 droggelbecher kernel:  [<ffffffff814dd7d8>] ? cpuidle_enter_state+0x46/0x93

Oct 03 20:36:55 droggelbecher kernel:  [<ffffffff814dd7d4>] ? cpuidle_enter_state+0x42/0x93

Oct 03 20:36:55 droggelbecher kernel:  [<ffffffff814dd847>] ? cpuidle_enter+0x12/0x14

Oct 03 20:36:55 droggelbecher kernel:  [<ffffffff81077f6a>] ? cpu_startup_entry+0x1c0/0x2a9

Oct 03 20:36:55 droggelbecher kernel:  [<ffffffff81024fbe>] ? start_secondary+0x115/0x133

Oct 03 20:37:31 droggelbecher systemd[1]: systemd-logind.service watchdog timeout (limit 1min)!

Oct 03 20:37:49 droggelbecher kernel: sysrq: SysRq : HELP : loglevel(0-9) reboot(b) crash(c) terminate-all-tasks(e) memory-full-oom-kill(f) kill-all-tasks(i) thaw-filesystems(j) sak(k) show-backtrace-all-active-cpus(l) show-memory-usage(m

Oct 03 20:37:53 droggelbecher kernel: sysrq: SysRq : SAK

Oct 03 20:37:53 droggelbecher kernel: SAK: killed process 2125 (X): task_session(p)==tty->session

Oct 03 20:37:53 droggelbecher kernel: SAK: killed process 2125 (X): task_session(p)==tty->session

Oct 03 20:37:55 droggelbecher kernel: sysrq: SysRq : Emergency Sync
```

In this specific case I could not even switch from X11 to the console, fortunately the MagicSysReq keys still worked (on some occasions, even this route for rebooting was blocked).

Any idea what might cause these freezes? (X becomes noticeably slower for a few seconds, then I cannot even move the mouse any more.)

Possibly relevant info:

```
# lspci

00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (external gfx0 port B) (rev 02)

00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD/ATI] RD990 I/O Memory Management Unit (IOMMU)

00:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port B)

00:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port H)

00:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (external gfx1 port A)

00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 SATA Controller [IDE mode] (rev 40)

00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller

00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller

00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller

00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller

00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller (rev 42)

00:14.1 IDE interface: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 IDE Controller (rev 40)

00:14.2 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 Azalia (Intel HDA) (rev 40)

00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 LPC host controller (rev 40)

00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI Bridge (rev 40)

00:14.5 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI2 Controller

00:15.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 0)

00:16.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller

00:16.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller

00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 0

00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 1

00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 2

00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 3

00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 4

00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 5

01:00.0 VGA compatible controller: NVIDIA Corporation GK208 [GeForce GT 720] (rev a1)

01:00.1 Audio device: NVIDIA Corporation GK208 HDMI/DP Audio Controller (rev a1)

02:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02)

03:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02)

05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
```

```
# emerge --info

Portage 2.2.20.1 (python 3.4.3-final-0, default/linux/amd64/13.0/desktop/kde/systemd, gcc-4.8.5, glibc-2.20-r2, 4.0.5-gentoo x86_64)

=================================================================

System uname: Linux-4.0.5-gentoo-x86_64-AMD_FX-tm-4300_Quad-Core_Processor-with-gentoo-2.2

KiB Mem:     8131008 total,   6148164 free

KiB Swap:   40569852 total,  40569852 free

Timestamp of repository gentoo: Thu, 01 Oct 2015 19:45:01 +0000

sh bash 4.3_p39

ld GNU ld (Gentoo 2.25.1 p1.1) 2.25.1

app-shells/bash:          4.3_p39::gentoo

dev-lang/perl:            5.20.2::gentoo

dev-lang/python:          2.7.10::gentoo, 3.4.3::gentoo

dev-util/cmake:           3.2.2::gentoo

dev-util/pkgconfig:       0.28-r2::gentoo

sys-apps/baselayout:      2.2::gentoo

sys-apps/openrc:          0.17::gentoo

sys-apps/sandbox:         2.6-r1::gentoo

sys-devel/autoconf:       2.13::gentoo, 2.69::gentoo

sys-devel/automake:       1.11.6-r1::gentoo, 1.13.4::gentoo, 1.14.1::gentoo, 1.15::gentoo

sys-devel/binutils:       2.25.1-r1::gentoo

sys-devel/gcc:            4.8.5::gentoo, 4.9.3::gentoo

sys-devel/gcc-config:     1.7.3::gentoo

sys-devel/libtool:        2.4.6::gentoo

sys-devel/make:           4.1-r1::gentoo

sys-kernel/linux-headers: 3.18::gentoo (virtual/os-headers)

sys-libs/glibc:           2.20-r2::gentoo

Repositories:

gentoo

    location: /usr/portage

    sync-type: rsync

    sync-uri: rsync://rsync.de.gentoo.org/gentoo-portage

    priority: -1000

sakaki-tools

    location: /usr/local/portage/sakaki-tools

    sync-type: git

    sync-uri: https://github.com/sakaki-/sakaki-tools.git

    masters: gentoo

    priority: 50

ACCEPT_KEYWORDS="amd64"

ACCEPT_LICENSE="* -@EULA"

CBUILD="x86_64-pc-linux-gnu"

CFLAGS="-march=native -O2 -pipe"

CHOST="x86_64-pc-linux-gnu"

CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt"

CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"

CXXFLAGS="-march=native -O2 -pipe"

DISTDIR="/usr/portage/distfiles"

EMERGE_DEFAULT_OPTS="--ask --jobs=5 --load-average=5"

FCFLAGS="-O2 -pipe"

FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms splitdebug strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync"

FFLAGS="-O2 -pipe"

GENTOO_MIRRORS="http://ftp-stud.hs-esslingen.de/pub/Mirrors/gentoo/ http://mirror.leaseweb.com/gentoo/ http://ftp.snt.utwente.nl/pub/os/linux/gentoo http://ftp.halifax.rwth-aachen.de/gentoo/ http://ftp.uni-erlangen.de/pub/mirrors/gentoo"

LANG="en_US.utf8"

LDFLAGS="-Wl,-O1 -Wl,--as-needed"

MAKEOPTS="-j5 -l5"

PKGDIR="/usr/portage/packages"

PORTAGE_CONFIGROOT="/"

PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"

PORTAGE_TMPDIR="/var/tmp"

USE="X a52 aac acl acpi aes alsa amd64 avx bash-completion berkdb bindist bluetooth branding bzip2 cairo cdda cdr cli cracklib crypt cryptsetup cups cxx dbus declarative dri dts dvd dvdr emacs emboss encode exif fam ffmpeg firefox flac fma3 fma4 fortran gdbm gif glamor gpm gtk gtk3 iconv ipv6 jpeg kde kipi lcms ldap libav libnotify lm_sensors mad mmx mmxext mng modules mp3 mp4 mpeg multilib ncurses nls nptl ogg opengl openmp otr pam pango pcre pdf phonon plasma png policykit popcnt ppds python qt3support qt4 qt5 readline sdl seccomp semantic-desktop session sound spell sse sse2 sse3 sse4_1 sse4_2 sse4a ssl ssse3 startup-notification svg systemd tcpd tiff truetype udev udisks unicode upower usb vorbis wxwidgets x264 xcb xcomposite xinerama xml xmpp xop xscreensaver xv xvid zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx fma3 fma4 mmx mmxext popcnt sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3 xop" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="efi-64" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-5" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_4" RUBY_TARGETS="ruby20 ruby21" USERLAND="GNU" VIDEO_CARDS="nvidia nouveau" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"

Unset:  CC, CPPFLAGS, CTARGET, CXX, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON

```

----------

## rubik-wuerfel

Two more remarks:

grepping through the old rcu_sched messages shows that the task dump was always for CPU 3. Could this be relevant or is this behavior to be expected?

I don't know if this is real or just my imagination, but I think that most (but definitely not all) of these freezes happened while browsing the internet, maybe even mostly while entering search terms in the search bar of Firefox.

----------

## rickvernam

Have you continued to experience this issue?

I have a HP Proliant DL380 G7 that just started doing this today.  I do not have X installed, or any such things - just base system + apache, postgres, KMV / libvirt, and a few less interesting things.

The server has been deployed for about 5 months.  Two days ago I rebooted with intel_iommu=on for the first time so that I could make use of VT-d to get some hardware into a KVM instance.

I ended up not using VT-d for other reasons, but left intel_iommu=on in place.

Today the server locked up with

 *Quote:*   

> Oct 13 14:58:38 c2 kernel: INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=8408 jiffies, g=13505, c=13504, q=348)
> 
> Oct 13 14:58:38 c2 kernel: Task dump for CPU 0:
> 
> Oct 13 14:58:38 c2 kernel: swapper/0       R  running task        0     0      0 0x00000008
> ...

 

I rebooted with intel_iommu=on removed from kernel boot, it ran for about 30 minutes then died with the same exact message.

I then rebooted & turned VT-d off in the BIOS...I have now been running for about 40 minutes.

Mine also had Task dump for CPU 0 on all messages.

I'm not sure how IOMMU relates to the RCU system - do you have IOMMU turned on, or VT-d enabled in your BIOS?

The server was running vanilla kernel 4.1.6 since deployment, but I updated to 4.1.10 when I turned intel_iommu=on.  What kernel are you using?

----------

## rubik-wuerfel

This looks clearly as if there are different reasons.

 *rickvernam wrote:*   

> I do not have X installed, or any such things

 

Same here, it started on a fresh install, even before I had started to emerge X.

 *rickvernam wrote:*   

>  - just base system + apache, postgres, KMV / libvirt, and a few less interesting things.
> 
> The server has been deployed for about 5 months.  Two days ago I rebooted with intel_iommu=on for the first time so that I could make use of VT-d to get some hardware into a KVM instance.
> 
> ...
> ...

 

AMD cpu here, problems with both 4.0.5 and 4.2.3, and no virtualization processes in place.

On the other hand, I was able to get rid of the freezes by switching from the open source nvidia gfx drivers (nvidiafb & nouveau) to the proprietary nvidia drivers. Because the problems first appeared without X, I believe that probably nvidiafb is somehow involved.

----------

## rickvernam

Just to follow up for any future readers, I also have not had an occurrence of this since I removed intel_iommu=on line from kernel boot command line and turned VT-d back off in the BIOS.

----------

## Roman_Gruber

microcode is loaded?

----------

## rubik-wuerfel

The microcode module is loaded, sys-kernel/linux-firmware is installed, excerpt from my .config:

```
/usr/src/linux # grep MICROCODE .config

CONFIG_MICROCODE=m

CONFIG_MICROCODE_INTEL=y

CONFIG_MICROCODE_AMD=y

CONFIG_MICROCODE_OLD_INTERFACE=y
```

----------

## rickvernam

 *tw04l124 wrote:*   

> microcode is loaded?

 

I believe so:

```
c2 linux # grep MICROCODE .config

CONFIG_MICROCODE=y

CONFIG_MICROCODE_INTEL=y

# CONFIG_MICROCODE_AMD is not set

CONFIG_MICROCODE_OLD_INTERFACE=y

```

and

```
c2 linux # dmesg| grep microcode

[    1.005308] microcode: CPU0 sig=0x206c2, pf=0x1, revision=0x1a

[    1.005317] microcode: CPU1 sig=0x206c2, pf=0x1, revision=0x1a

[    1.005327] microcode: CPU2 sig=0x206c2, pf=0x1, revision=0x1a

[    1.005336] microcode: CPU3 sig=0x206c2, pf=0x1, revision=0x1a

[    1.005345] microcode: CPU4 sig=0x206c2, pf=0x1, revision=0x1a

[    1.005354] microcode: CPU5 sig=0x206c2, pf=0x1, revision=0x1a

[    1.005364] microcode: CPU6 sig=0x206c2, pf=0x1, revision=0x1a

[    1.005374] microcode: CPU7 sig=0x206c2, pf=0x1, revision=0x1a

[    1.005384] microcode: CPU8 sig=0x206c2, pf=0x1, revision=0x1a

[    1.005393] microcode: CPU9 sig=0x206c2, pf=0x1, revision=0x1a

[    1.005403] microcode: CPU10 sig=0x206c2, pf=0x1, revision=0x1a

[    1.005412] microcode: CPU11 sig=0x206c2, pf=0x1, revision=0x1a

[    1.005421] microcode: CPU12 sig=0x206c2, pf=0x1, revision=0x1a

[    1.005430] microcode: CPU13 sig=0x206c2, pf=0x1, revision=0x1a

[    1.005439] microcode: CPU14 sig=0x206c2, pf=0x1, revision=0x1a

[    1.005447] microcode: CPU15 sig=0x206c2, pf=0x1, revision=0x1a

[    1.005455] microcode: CPU16 sig=0x206c2, pf=0x1, revision=0x1a

[    1.005463] microcode: CPU17 sig=0x206c2, pf=0x1, revision=0x1a

[    1.005472] microcode: CPU18 sig=0x206c2, pf=0x1, revision=0x1a

[    1.005480] microcode: CPU19 sig=0x206c2, pf=0x1, revision=0x1a

[    1.005489] microcode: CPU20 sig=0x206c2, pf=0x1, revision=0x1a

[    1.005496] microcode: CPU21 sig=0x206c2, pf=0x1, revision=0x1a

[    1.005505] microcode: CPU22 sig=0x206c2, pf=0x1, revision=0x1a

[    1.005512] microcode: CPU23 sig=0x206c2, pf=0x1, revision=0x1a

[    1.005567] microcode: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba

```

It appears I can install microcode-ctl & intel-microcode.  This is a production server, so I have to wait until their shift is over before I can do this.

I also had this reoccur today, alternating between

```
Oct 17 04:10:05 c2 kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 12, t=102982 jiffies, g=139, c=138, q=8)

Oct 17 04:10:05 c2 kernel: Task dump for CPU 0:

Oct 17 04:10:05 c2 kernel: swapper/0       R  running task        0     0      0 0x00000808

Oct 17 04:10:05 c2 kernel:  ffffffffffffff10 ffffffff821a829a 0000000000000010 0000000000000202

Oct 17 04:10:05 c2 kernel:  ffffffff82433f28 0000000000000018 ffffffff821a828c ffffffff8249fbb8

Oct 17 04:10:05 c2 kernel:  ffff8811dfc1aa70 ffffffff82476e80 ffffffff82430000 ffffffff82430000

Oct 17 04:10:05 c2 kernel: Call Trace:

Oct 17 04:10:05 c2 kernel:  [<ffffffff821a829a>] ? cpuidle_enter_state+0x9a/0x140

Oct 17 04:10:05 c2 kernel:  [<ffffffff821a828c>] ? cpuidle_enter_state+0x8c/0x140

Oct 17 04:10:05 c2 kernel:  [<ffffffff81ea3daf>] ? cpu_startup_entry+0x23f/0x2d0

Oct 17 04:10:05 c2 kernel:  [<ffffffff824bee7e>] ? start_kernel+0x370/0x37b

Oct 17 04:10:05 c2 kernel:  [<ffffffff824be919>] ? set_init_arg+0x4e/0x4e

```

and

```
Oct 17 04:10:13 c2 kernel: INFO: rcu_sched self-detected stall on CPU { 0}  (t=134463 jiffies g=887585 c=887584 q=12053)

Oct 17 04:10:13 c2 kernel: Task dump for CPU 0:

Oct 17 04:10:13 c2 kernel: swapper/0       R  running task        0     0      0 0x00000808

Oct 17 04:10:13 c2 kernel:  0000000000000001 ffffffff824674c0 ffffffff81eb1b5f 00000000000d8b21

Oct 17 04:10:13 c2 kernel:  ffff8811dfc154c0 ffffffff82467400 ffffffff824674c0 ffffffff8249fbc0

Oct 17 04:10:13 c2 kernel:  ffffffff81eb4bea 0000989680000000 0000000000002f15 0000000000000000

Oct 17 04:10:13 c2 kernel: Call Trace:

Oct 17 04:10:13 c2 kernel:  <IRQ>  [<ffffffff81eb1b5f>] ? rcu_dump_cpu_stacks+0x7f/0xc0

Oct 17 04:10:13 c2 kernel:  [<ffffffff81eb4bea>] ? rcu_check_callbacks+0x3ca/0x6f0

Oct 17 04:10:13 c2 kernel:  [<ffffffff81ebbf51>] ? timekeeping_update.constprop.9+0xe1/0x120

Oct 17 04:10:13 c2 kernel:  [<ffffffff81ebd136>] ? update_wall_time+0x236/0x660

Oct 17 04:10:13 c2 kernel:  [<ffffffff81eb6d01>] ? update_process_times+0x31/0x60

Oct 17 04:10:13 c2 kernel:  [<ffffffff81ec4c23>] ? tick_sched_timer+0x33/0x60

Oct 17 04:10:13 c2 kernel:  [<ffffffff81eb755b>] ? __run_hrtimer.isra.34+0x3b/0xc0

Oct 17 04:10:13 c2 kernel:  [<ffffffff81eb7915>] ? hrtimer_interrupt+0xd5/0x210

Oct 17 04:10:13 c2 kernel:  [<ffffffff81e30da5>] ? smp_apic_timer_interrupt+0x35/0x50

Oct 17 04:10:13 c2 kernel:  [<ffffffff822a8c3b>] ? apic_timer_interrupt+0x6b/0x70

Oct 17 04:10:13 c2 kernel:  [<ffffffff822a7975>] ? _raw_spin_unlock_irqrestore+0x5/0x10

Oct 17 04:10:13 c2 kernel:  [<ffffffff81eb632f>] ? try_to_del_timer_sync+0x3f/0x60

Oct 17 04:10:13 c2 kernel:  [<ffffffff81eb62f0>] ? del_timer+0x50/0x50

Oct 17 04:10:13 c2 kernel:  [<ffffffff81eb6392>] ? del_timer_sync+0x42/0x50

Oct 17 04:10:13 c2 kernel:  [<ffffffff82209e4f>] ? inet_csk_reqsk_queue_drop+0x5f/0x1b0

Oct 17 04:10:13 c2 kernel:  [<ffffffff8220a1f8>] ? reqsk_timer_handler+0x258/0x2c0

Oct 17 04:10:13 c2 kernel:  [<ffffffff82209fa0>] ? inet_csk_reqsk_queue_drop+0x1b0/0x1b0

Oct 17 04:10:13 c2 kernel:  [<ffffffff81eb6495>] ? call_timer_fn.isra.31+0x15/0x70

Oct 17 04:10:13 c2 kernel:  [<ffffffff81eb6692>] ? run_timer_softirq+0x1a2/0x230

Oct 17 04:10:13 c2 kernel:  [<ffffffff81e76d44>] ? __do_softirq+0xd4/0x1e0

Oct 17 04:10:13 c2 kernel:  [<ffffffff81e77045>] ? irq_exit+0x95/0xa0

Oct 17 04:10:13 c2 kernel:  [<ffffffff81e30daa>] ? smp_apic_timer_interrupt+0x3a/0x50

Oct 17 04:10:13 c2 kernel:  [<ffffffff822a8c3b>] ? apic_timer_interrupt+0x6b/0x70

Oct 17 04:10:13 c2 kernel:  <EOI>  [<ffffffff821a829a>] ? cpuidle_enter_state+0x9a/0x140

Oct 17 04:10:13 c2 kernel:  [<ffffffff821a828c>] ? cpuidle_enter_state+0x8c/0x140

Oct 17 04:10:13 c2 kernel:  [<ffffffff81ea3daf>] ? cpu_startup_entry+0x23f/0x2d0

Oct 17 04:10:13 c2 kernel:  [<ffffffff824bee7e>] ? start_kernel+0x370/0x37b

Oct 17 04:10:13 c2 kernel:  [<ffffffff824be919>] ? set_init_arg+0x4e/0x4e

```

for about 20 minutes before it locked hard.  It's always CPU 0, still.

----------

## rickvernam

 *rickvernam wrote:*   

>  *tw04l124 wrote:*   microcode is loaded? 
> 
> ...
> 
> It appears I can install microcode-ctl & intel-microcode.  This is a production server, so I have to wait until their shift is over before I can do this.
> ...

 

Do I understand from this correctly that there is not a microcode update available?

```
c2 ~ # iucode_tool -S --write-earlyfw=/boot/early-ucode.cpio /lib/firmware/intel-ucode/*

iucode_tool: system has processor(s) with signature 0x000206c2

iucode_tool: No valid microcodes were selected, nothing to do...

```

My processor is

```
c2 ~ # head -n10 /proc/cpuinfo

processor       : 0

vendor_id       : GenuineIntel

cpu family      : 6

model           : 44

model name      : Intel(R) Xeon(R) CPU           X5690  @ 3.47GHz

stepping        : 2

microcode       : 0x1a

```

So the microcode file should be 06-2C-02 I suppose, but there is no such a file in /lib/firmware/intel-ucode.

Is there something else I can be doing regarding microcode?

Thanks!

----------

## TigerJr

 *rickvernam wrote:*   

> Have you continued to experience this issue?
> 
> I have a HP Proliant DL380 G7 that just started doing this today.  I do not have X installed, or any such things - just base system + apache, postgres, KMV / libvirt, and a few less interesting things.
> 
> The server has been deployed for about 5 months.  Two days ago I rebooted with intel_iommu=on for the first time so that I could make use of VT-d to get some hardware into a KVM instance.
> ...

 

It's looks like intel X58 errata hardware bug but i didn't find x58!?

http://thr3ads.net/xen-devel/2013/03/2286949-PATCH-0-3-IOMMU-errata-treatment-adjustments

Just look at [PATCH 3/3]

This article contains information about the Intel 55x0 chipset errata - Interrupt remapping issue. Intel 5500/5520/X58 chipset revision 0x13 and 0x22 have an errata (#47 and #53) which makes the IOMMU interrupt remapping unit unreliable. This erratum causes interruptions and the interrupt remapping invalidations become unresponsive.

----------

## rickvernam

 *TigerJr wrote:*   

> 
> 
> It's looks like intel X58 errata hardware bug but i didn't find x58!?
> 
> http://thr3ads.net/xen-devel/2013/03/2286949-PATCH-0-3-IOMMU-errata-treatment-adjustments
> ...

 

The link references stepping levels, but I do not know how to relate the stepping information from that link.  my /proc/cpuinfo says "stepping: 2", but the link says things like stepping B-2 & C-3.  Where does that letter come from, how do I know which I have?

Anyway, I have VT-d turned off in the BIOS, though I do not believe there is an option to turn interrupt remapping on/off.

If I am affected by that errata, would the following be sufficient to avoid hitting the interrupt remapping issue and triggering the lock-up?

```
 --- IOMMU Hardware Support

Generic IOMMU Pagetable Support  --->

[ ]   AMD IOMMU support

[*]   Support for Intel IOMMU using DMA Remapping Devices

[*]     Enable Intel DMA Remapping Devices by default (NEW)

[ ]   Support for Interrupt Remapping
```

I was booted into a kernel that had Support for Interrupt Remapping turned on when I would hit this lock-up...

Thanks for your insight!

----------

## TigerJr

Interupt remmaping not always used by AMD VT-d , but if turn it off you will never use USB\PCI Passthrough. 

 *Quote:*   

> The link references stepping levels, but I do not know how to relate the stepping information from that link. my /proc/cpuinfo says "stepping: 2", but the link says things like stepping B-2 & C-3. Where does that letter come from, how do I know which I have?

 

It's not a cpu stepping, it's revision of southbridge chipset use for that information dmidecode. 

But lspci -k can also give you revision of your devices. 

And than you post your lspci output i didn't see i5500 or i5520 or x58 chipsets those can have revisions(B2 C3). 

About IOMMU, in lspci your mainboard have device:

```
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD/ATI] RD990 I/O Memory Management Unit (IOMMU)
```

If you watch lspci -k output you can see what Kernel driver is in use: amd_iommu

But if your kernel compiled without that driver than I\O Virtualisation would be disabled. It's a part of AMD VT-d. 

Here can hide hardware bug with interrupt remapping or it's just kernel not configured properly.

----------

## rickvernam

Okay, so I do have hardware affected by the errata which you described, rev 0x13.

A section of my lspci:

```
00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 13)

00:02.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 2 (rev 13)

00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 13)

00:04.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 4 (rev 13)

00:05.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 5 (rev 13)

00:06.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 6 (rev 13)

00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 13)

00:08.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 8 (rev 13)

00:09.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 9 (rev 13)

00:0a.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 10 (rev 13)
```

So I will leave Interrupt Remapping turned off in the kernel.  That may disable VT-d, but I can live with that.

```
 --- IOMMU Hardware Support 

Generic IOMMU Pagetable Support  ---> 

[ ]   AMD IOMMU support 

[*]   Support for Intel IOMMU using DMA Remapping Devices 

[*]     Enable Intel DMA Remapping Devices by default (NEW) 

[ ]   Support for Interrupt Remapping
```

Thanks again for your input & direction!

----------

## Polyatomic

Brain damage removed

Sat Oct 31 21:08:42 ACDT 2015Last edited by Polyatomic on Sat Oct 31, 2015 10:40 am; edited 1 time in total

----------

## TigerJr

 *Polyatomic wrote:*   

> 
> 
> OP what was the magic that resolved 
> 
> your issue.

 

I think magic lv3: Mainoard sell and buy new with unaffected chipset, try read magic patches from http://thr3ads.net/xen-devel/2013/03/2286949-PATCH-0-3-IOMMU-errata-treatment-adjustments

 And if it's work, you are lucky.

----------

## Polyatomic

Brain damage removed

Sat Oct 31 21:09:33 ACDT 2015Last edited by Polyatomic on Sat Oct 31, 2015 10:41 am; edited 3 times in total

----------

## TigerJr

Can you post here your lspci output for revision of your chipset?

And i will give you another clue, errata is not a one problem that people have faced using vt-d.

----------

## Polyatomic

Reserved!

----------

## TigerJr

For selling? =) You solve this problem!

----------

## TigerJr

So i have faced with this bug too, on these hardware:

 *Quote:*   

> localhost linux # lspci
> 
> 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
> 
> 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
> ...

 

It's xen guest and dmesg output interesting too:

```
localhost linux # dmesg

[    0.000000] Initializing cgroup subsys cpuset

[    0.000000] Initializing cgroup subsys cpu

[    0.000000] Initializing cgroup subsys cpuacct

[    0.000000] Linux version 4.2.4 (root@next.xenogentoo.org) (gcc version 4.9.3 (Gentoo 4.9.3 p1.3, pie-0.6.3) ) #1 SMP Sun Oct 25 19:04:00 2015

[    0.000000] Command line: root=/dev/xvda1

[    0.000000] tseg: 0000000000

[    0.000000] x86/fpu: Legacy x87 FPU detected.

[    0.000000] x86/fpu: Using 'lazy' FPU context switches.

[    0.000000] e820: BIOS-provided physical RAM map:

[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dfff] usable

[    0.000000] BIOS-e820: [mem 0x000000000009e000-0x000000000009ffff] reserved

[    0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved

[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007f7fffff] usable

[    0.000000] BIOS-e820: [mem 0x00000000fc000000-0x00000000ffffffff] reserved

[    0.000000] NX (Execute Disable) protection: active

[    0.000000] SMBIOS 2.4 present.

[    0.000000] DMI: Xen HVM domU, BIOS 4.6.0 10/18/2015

[    0.000000] Hypervisor detected: Xen

[    0.000000] Xen version 4.6.

[    0.000000] Xen Platform PCI: I/O protocol version 1

[    0.000000] Netfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated NICs.

[    0.000000] Blkfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated disks.

               You might have to change the root device

               from /dev/hd[a-d] to /dev/xvd[a-d]

               in your root= kernel command line option

[    0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved

[    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable

[    0.000000] e820: last_pfn = 0x7f800 max_arch_pfn = 0x400000000

[    0.000000] MTRR default type: write-back

[    0.000000] MTRR fixed ranges enabled:

[    0.000000]   00000-9FFFF write-back

[    0.000000]   A0000-BFFFF write-combining

[    0.000000]   C0000-FFFFF write-back

[    0.000000] MTRR variable ranges enabled:

[    0.000000]   0 base 00F0000000 mask FFF0000000 uncachable

[    0.000000]   1 disabled

[    0.000000]   2 disabled

[    0.000000]   3 disabled

[    0.000000]   4 disabled

[    0.000000]   5 disabled

[    0.000000]   6 disabled

[    0.000000]   7 disabled

[    0.000000] TOM2: 0000000200000000 aka 8192M

[    0.000000] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WC  UC- WT

[    0.000000] found SMP MP-table at [mem 0x000fb710-0x000fb71f] mapped at [ffff8800000fb710]

[    0.000000] Base memory trampoline at [ffff880000098000] 98000 size 24576

[    0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff]

[    0.000000]  [mem 0x00000000-0x000fffff] page 4k

[    0.000000] BRK [0x01b03000, 0x01b03fff] PGTABLE

[    0.000000] BRK [0x01b04000, 0x01b04fff] PGTABLE

[    0.000000] BRK [0x01b05000, 0x01b05fff] PGTABLE

[    0.000000] init_memory_mapping: [mem 0x7f600000-0x7f7fffff]

[    0.000000]  [mem 0x7f600000-0x7f7fffff] page 2M

[    0.000000] BRK [0x01b06000, 0x01b06fff] PGTABLE

[    0.000000] init_memory_mapping: [mem 0x60000000-0x7f5fffff]

[    0.000000]  [mem 0x60000000-0x7f5fffff] page 2M

[    0.000000] init_memory_mapping: [mem 0x40000000-0x5fffffff]

[    0.000000]  [mem 0x40000000-0x5fffffff] page 2M

[    0.000000] init_memory_mapping: [mem 0x00100000-0x3fffffff]

[    0.000000]  [mem 0x00100000-0x001fffff] page 4k

[    0.000000]  [mem 0x00200000-0x3fffffff] page 2M

[    0.000000] ACPI: Early table checksum verification disabled

[    0.000000] ACPI: RSDP 0x00000000000EA020 000024 (v02 Xen   )

[    0.000000] ACPI: XSDT 0x00000000FC00D000 000054 (v01 Xen    HVM      00000000 HVML 00000000)

[    0.000000] ACPI: FACP 0x00000000FC00CCC0 0000F4 (v04 Xen    HVM      00000000 HVML 00000000)

[    0.000000] ACPI: DSDT 0x00000000FC0042E0 008956 (v02 Xen    HVM      00000000 INTL 20140828)

[    0.000000] ACPI: FACS 0x00000000FC0042A0 000040

[    0.000000] ACPI: FACS 0x00000000FC0042A0 000040

[    0.000000] ACPI: APIC 0x00000000FC00CDC0 0000D8 (v02 Xen    HVM      00000000 HVML 00000000)

[    0.000000] ACPI: HPET 0x00000000FC00CF10 000038 (v01 Xen    HVM      00000000 HVML 00000000)

[    0.000000] ACPI: WAET 0x00000000FC00CF50 000028 (v01 Xen    HVM      00000000 HVML 00000000)

[    0.000000] ACPI: SSDT 0x00000000FC00CF80 000031 (v02 Xen    HVM      00000000 INTL 20140828)

[    0.000000] ACPI: SSDT 0x00000000FC00CFC0 000031 (v02 Xen    HVM      00000000 INTL 20140828)

[    0.000000] ACPI: Local APIC address 0xfee00000

[    0.000000] No NUMA configuration found

[    0.000000] Faking a node at [mem 0x0000000000000000-0x000000007f7fffff]

[    0.000000] NODE_DATA(0) allocated [mem 0x7f7fd000-0x7f7fffff]

[    0.000000]  [ffffea0000000000-ffffea0001ffffff] PMD -> [ffff88007ce00000-ffff88007edfffff] on node 0

[    0.000000] Zone ranges:

[    0.000000]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]

[    0.000000]   DMA32    [mem 0x0000000001000000-0x000000007f7fffff]

[    0.000000]   Normal   empty

[    0.000000] Movable zone start for each node

[    0.000000] Early memory node ranges

[    0.000000]   node   0: [mem 0x0000000000001000-0x000000000009dfff]

[    0.000000]   node   0: [mem 0x0000000000100000-0x000000007f7fffff]

[    0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x000000007f7fffff]

[    0.000000] On node 0 totalpages: 522141

[    0.000000]   DMA zone: 64 pages used for memmap

[    0.000000]   DMA zone: 21 pages reserved

[    0.000000]   DMA zone: 3997 pages, LIFO batch:0

[    0.000000]   DMA32 zone: 8096 pages used for memmap

[    0.000000]   DMA32 zone: 518144 pages, LIFO batch:31

[    0.000000] ACPI: PM-Timer IO Port: 0xb008

[    0.000000] ACPI: Local APIC address 0xfee00000

[    0.000000] IOAPIC[0]: apic_id 1, version 17, address 0xfec00000, GSI 0-47

[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)

[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 low level)

[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 low level)

[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 low level)

[    0.000000] ACPI: IRQ0 used by override.

[    0.000000] ACPI: IRQ5 used by override.

[    0.000000] ACPI: IRQ9 used by override.

[    0.000000] ACPI: IRQ10 used by override.

[    0.000000] ACPI: IRQ11 used by override.

[    0.000000] Using ACPI (MADT) for SMP configuration information

[    0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000

[    0.000000] smpboot: 15 Processors exceeds NR_CPUS limit of 8

[    0.000000] smpboot: Allowing 8 CPUs, 6 hotplug CPUs

[    0.000000] e820: [mem 0x7f800000-0xfbffffff] available for PCI devices

[    0.000000] Booting paravirtualized kernel on Xen HVM

[    0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns

[    0.000000] setup_percpu: NR_CPUS:8 nr_cpumask_bits:8 nr_cpu_ids:8 nr_node_ids:1

[    0.000000] PERCPU: Embedded 31 pages/cpu @ffff88007f400000 s89368 r8192 d29416 u262144

[    0.000000] pcpu-alloc: s89368 r8192 d29416 u262144 alloc=1*2097152

[    0.000000] pcpu-alloc: [0] 0 1 2 3 4 5 6 7

[    0.000000] xen: PV spinlocks enabled

[    0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes)

[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 513960

[    0.000000] Policy zone: DMA32

[    0.000000] Kernel command line: root=/dev/xvda1

[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)

[    0.000000] Calgary: detecting Calgary via BIOS EBDA area

[    0.000000] Calgary: Unable to locate Rio Grande table in EBDA - bailing!

[    0.000000] Memory: 2043152K/2088564K available (6759K kernel code, 639K rwdata, 2084K rodata, 1000K init, 740K bss, 45412K reserved, 0K cma-reserved)

[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=8, Nodes=1

[    0.000000] Hierarchical RCU implementation.

[    0.000000]  Build-time adjustment of leaf fanout to 64.

[    0.000000] NR_IRQS:4352 nr_irqs:896 16

[    0.000000] xen:events: Using FIFO-based ABI

[    0.000000] xen:events: Xen HVM callback vector for event delivery is enabled

[    0.000000] Console: colour VGA+ 80x25

[    0.000000] console [tty0] enabled

[    0.000000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 30580167144 ns

[    0.000000] hpet clockevent registered

[    0.000000] tsc: Detected 3000.174 MHz processor

[    0.000000] tsc: Marking TSC unstable due to TSCs unsynchronized

[    0.005000] Calibrating delay loop (skipped), value calculated using timer frequency.. 6000.34 BogoMIPS (lpj=3000174)

[    0.009012] pid_max: default: 32768 minimum: 301

[    0.013029] ACPI: Core revision 20150619

[    0.022288] ACPI: All ACPI Tables successfully acquired

[    0.028909] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)

[    0.034488] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)

[    0.038619] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes)

[    0.042046] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes)

[    0.045479] Initializing cgroup subsys freezer

[    0.049205] CPU: Physical Processor ID: 0

[    0.052011] CPU: Processor Core ID: 0

[    0.055012] process: using AMD E400 aware idle routine

[    0.059015] Last level iTLB entries: 4KB 512, 2MB 8, 4MB 4

[    0.062011] Last level dTLB entries: 4KB 512, 2MB 8, 4MB 4, 1GB 0

[    0.068737] Freeing SMP alternatives memory: 28K (ffffffff81a41000 - ffffffff81a48000)

[    0.085453] x2apic: IRQ remapping doesn't support X2APIC mode

[    0.089000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=0 pin2=0

[    0.101982] clocksource: xen: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns

[    0.105020] Xen: using vcpuop timer interface

[    0.105035] installing Xen timer for CPU 0

[    0.108080] smpboot: CPU0: Dual-Core AMD Opteron(tm) Processor 2222 (fam: 0f, model: 41, stepping: 03)

[    0.110059] cpu 0 spinlock event irq 53

[    0.111102] Performance Events: Broken PMU hardware detected, using software events only.

[    0.112018] Failed to access perfctr msr (MSR c0010007 is 0)

[    0.113868] installing Xen timer for CPU 1

[    0.114093] x86: Booting SMP configuration:

[    0.115018] .... node  #0, CPUs:      #1cpu 1 spinlock event irq 59

[    0.178050] x86: Booted up 1 node, 2 CPUs

[    0.179018] smpboot: Total of 2 processors activated (12083.59 BogoMIPS)

[    0.180544] devtmpfs: initialized

[    0.186158] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns

[    0.190112] RTC time: 20:10:42, date: 10/25/15

[    0.191177] NET: Registered protocol family 16

[    0.197032] cpuidle: using governor ladder

[    0.203021] cpuidle: using governor menu

[    0.205234] ACPI: bus type PCI registered

[    0.207019] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5

[    0.210316] PCI: Using configuration type 1 for base access

[    0.221089] ACPI: Added _OSI(Module Device)

[    0.223059] ACPI: Added _OSI(Processor Device)

[    0.225023] ACPI: Added _OSI(3.0 _SCP Extensions)

[    0.227019] ACPI: Added _OSI(Processor Aggregator Device)

[    0.230569] xen: --> pirq=16 -> irq=9 (gsi=9)

[    0.234681] ACPI: Interpreter enabled

[    0.237023] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S1_] (20150619/hwxface-580)

[    0.241020] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S2_] (20150619/hwxface-580)

[    0.246026] ACPI: (supports S0 S3 S5)

[    0.248017] ACPI: Using IOAPIC for interrupt routing

[    0.250057] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug

[    0.288488] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])

[    0.290037] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI]

[    0.292034] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM

[    0.295125] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.

[    0.299178] acpiphp: Slot [0] registered

[    0.303015] acpiphp: Slot [3] registered

[    0.306487] acpiphp: Slot [4] registered

[    0.310628] acpiphp: Slot [5] registered

[    0.314561] acpiphp: Slot [6] registered

[    0.318564] acpiphp: Slot [7] registered

[    0.322499] acpiphp: Slot [8] registered

[    0.326024] acpiphp: Slot [9] registered

[    0.329496] acpiphp: Slot [10] registered

[    0.333017] acpiphp: Slot [11] registered

[    0.337015] acpiphp: Slot [12] registered

[    0.340491] acpiphp: Slot [13] registered

[    0.344027] acpiphp: Slot [14] registered

[    0.348015] acpiphp: Slot [15] registered

[    0.351492] acpiphp: Slot [16] registered

[    0.355015] acpiphp: Slot [17] registered

[    0.358497] acpiphp: Slot [18] registered

[    0.362500] acpiphp: Slot [19] registered

[    0.366015] acpiphp: Slot [20] registered

[    0.369494] acpiphp: Slot [21] registered

[    0.373015] acpiphp: Slot [22] registered

[    0.377023] acpiphp: Slot [23] registered

[    0.380504] acpiphp: Slot [24] registered

[    0.384502] acpiphp: Slot [25] registered

[    0.388015] acpiphp: Slot [26] registered

[    0.392003] acpiphp: Slot [27] registered

[    0.395518] acpiphp: Slot [28] registered

[    0.399047] acpiphp: Slot [29] registered

[    0.461515] acpiphp: Slot [30] registered

[    0.465065] acpiphp: Slot [31] registered

[    0.468483] PCI host bridge to bus 0000:00

[    0.472024] pci_bus 0000:00: root bus resource [bus 00-ff]

[    0.475020] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]

[    0.478019] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window]

[    0.482021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]

[    0.487019] pci_bus 0000:00: root bus resource [mem 0xf0000000-0xfbffffff window]

[    0.493243] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000

[    0.496014] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100

[    0.500023] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180

[    0.501564] pci 0000:00:01.1: reg 0x20: [io  0xc200-0xc20f]

[    0.502452] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io  0x01f0-0x01f7]

[    0.506019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io  0x03f6]

[    0.509019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io  0x0170-0x0177]

[    0.513019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io  0x0376]

[    0.516808] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000

[    0.516974] * Found PM-Timer Bug on the chipset. Due to workarounds for a bug,

               * this clock source is slow. Consider trying other clock sources

[    0.527134] pci 0000:00:01.3: quirk: [io  0xb000-0xb03f] claimed by PIIX4 ACPI

[    0.534292] pci 0000:00:02.0: [1234:1111] type 00 class 0x030000

[    0.534924] pci 0000:00:02.0: reg 0x10: [mem 0xf1000000-0xf17fffff pref]

[    0.538021] pci 0000:00:03.0: [5853:0001] type 00 class 0xff8000

[    0.538410] pci 0000:00:03.0: reg 0x10: [io  0xc000-0xc0ff]

[    0.538642] pci 0000:00:03.0: reg 0x14: [mem 0xf0000000-0xf0ffffff pref]

[    0.542343] ACPI: PCI Interrupt Link [LNKA] (IRQs *5 10 11)

[    0.546214] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11)

[    0.549207] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11)

[    0.553212] ACPI: PCI Interrupt Link [LNKD] (IRQs *5 10 11)

[    0.565220] ACPI: Enabled 2 GPEs in block 00 to 0F

[    0.568155] xen:balloon: Initialising balloon driver

[    0.574242] vgaarb: setting as boot device: PCI:0000:00:02.0

[    0.575000] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none

[    0.580033] vgaarb: loaded

[    0.583021] vgaarb: bridge control possible 0000:00:02.0

[    0.585172] SCSI subsystem initialized

[    0.587099] libata version 3.00 loaded.

[    0.587133] ACPI: bus type USB registered

[    0.589087] usbcore: registered new interface driver usbfs

[    0.592053] usbcore: registered new interface driver hub

[    0.594071] usbcore: registered new device driver usb

[    0.596080] pps_core: LinuxPPS API ver. 1 registered

[    0.598019] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>

[    0.602201] Advanced Linux Sound Architecture Driver Initialized.

[    0.605088] PCI: Using ACPI for IRQ routing

[    0.607021] PCI: pci_cache_line_size set to 64 bytes

[    0.607394] e820: reserve RAM buffer [mem 0x0009e000-0x0009ffff]

[    0.607400] e820: reserve RAM buffer [mem 0x7f800000-0x7fffffff]

[    0.607681] HPET: 3 timers in total, 0 timers will be used for per-cpu timer

[    0.610039] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0

[    0.613018] hpet0: 3 comparators, 64-bit 62.500000 MHz counter

[    0.618102] clocksource: Switched to clocksource xen

[    0.620401] pnp: PnP ACPI init

[    0.622684] system 00:00: [mem 0x00000000-0x0009ffff] could not be reserved

[    0.625049] system 00:00: Plug and Play ACPI device, IDs PNP0c02 (active)

[    0.625196] system 00:01: [io  0x08a0-0x08a3] has been reserved

[    0.627392] system 00:01: [io  0x0cc0-0x0ccf] has been reserved

[    0.629647] system 00:01: [io  0x04d0-0x04d1] has been reserved

[    0.631868] system 00:01: Plug and Play ACPI device, IDs PNP0c02 (active)

[    0.631905] xen: --> pirq=17 -> irq=8 (gsi=8)

[    0.631948] pnp 00:02: Plug and Play ACPI device, IDs PNP0b00 (active)

[    0.631977] xen: --> pirq=18 -> irq=12 (gsi=12)

[    0.632051] pnp 00:03: Plug and Play ACPI device, IDs PNP0f13 (active)

[    0.632080] xen: --> pirq=19 -> irq=1 (gsi=1)

[    0.632123] pnp 00:04: Plug and Play ACPI device, IDs PNP0303 PNP030b (active)

[    0.632152] xen: --> pirq=20 -> irq=6 (gsi=6)

[    0.632159] pnp 00:05: [dma 2]

[    0.632195] pnp 00:05: Plug and Play ACPI device, IDs PNP0700 (active)

[    0.632228] xen: --> pirq=21 -> irq=4 (gsi=4)

[    0.632270] pnp 00:06: Plug and Play ACPI device, IDs PNP0501 (active)

[    0.632311] xen: --> pirq=22 -> irq=7 (gsi=7)

[    0.632345] pnp 00:07: Plug and Play ACPI device, IDs PNP0400 (active)

[    0.632410] system 00:08: [io  0x10c0-0x1141] has been reserved

[    0.634602] system 00:08: [io  0xb044-0xb047] has been reserved

[    0.636861] system 00:08: Plug and Play ACPI device, IDs PNP0c02 (active)

[    0.645085] pnp: PnP ACPI: found 9 devices

[    0.656479] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns

[    0.660376] pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7 window]

[    0.660381] pci_bus 0000:00: resource 5 [io  0x0d00-0xffff window]

[    0.660386] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]

[    0.660390] pci_bus 0000:00: resource 7 [mem 0xf0000000-0xfbffffff window]

[    0.660439] NET: Registered protocol family 2

[    0.662726] TCP established hash table entries: 16384 (order: 5, 131072 bytes)

[    0.666602] TCP bind hash table entries: 16384 (order: 6, 262144 bytes)

[    0.668924] TCP: Hash tables configured (established 16384 bind 16384)

[    0.671140] UDP hash table entries: 1024 (order: 3, 32768 bytes)

[    0.673319] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes)

[    0.675576] NET: Registered protocol family 1

[    0.677894] RPC: Registered named UNIX socket transport module.

[    0.680052] RPC: Registered udp transport module.

[    0.682163] RPC: Registered tcp transport module.

[    0.684279] RPC: Registered tcp NFSv4.1 backchannel transport module.

[    0.686477] pci 0000:00:00.0: Limiting direct PCI/PCI transfers

[    0.688676] pci 0000:00:01.0: PIIX3: Enabling Passive Release

[    0.690882] pci 0000:00:01.0: Activating ISA DMA hang workarounds

[    0.693165] pci 0000:00:02.0: Video device with shadowed ROM

[    0.693234] PCI: CLS 0 bytes, default 64

[    0.693340] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x2b3ee9f8ea7, max_idle_ns: 440795347344 ns

[    0.698064] futex hash table entries: 2048 (order: 5, 131072 bytes)

[    0.700405] audit: initializing netlink subsys (disabled)

[    0.702627] audit: type=2000 audit(1445803843.255:1): initialized

[    0.705217] HugeTLB registered 2 MB page size, pre-allocated 0 pages

[    0.711079] VFS: Disk quotas dquot_6.6.0

[    0.713353] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)

[    0.716422] NFS: Registering the id_resolver key type

[    0.718566] Key type id_resolver registered

[    0.720690] Key type id_legacy registered

[    0.722912] SGI XFS with ACLs, security attributes, no debug enabled

[    0.732714] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)

[    0.736562] io scheduler noop registered

[    0.738671] io scheduler deadline registered

[    0.740814] io scheduler cfq registered (default)

[    0.743074] pci_hotplug: PCI Hot Plug PCI Core version: 0.5

[    0.745443] uvesafb: failed to execute /sbin/v86d

[    0.747577] uvesafb: make sure that the v86d helper is installed and executable

[    0.751422] uvesafb: Getting VBE info block failed (eax=0x4f00, err=-2)

[    0.753615] uvesafb: vbe_init() failed with -22

[    0.755710] uvesafb: probe of uvesafb.0 failed with error -22

[    0.758641] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0

[    0.764593] ACPI: Power Button [PWRF]

[    0.767911] input: Sleep Button as /devices/LNXSYSTM:00/LNXSLPBN:00/input/input1

[    0.773840] ACPI: Sleep Button [SLPF]

[    0.777655] xen:xen_evtchn: Event-channel device installed

[    0.781629] xen: --> pirq=23 -> irq=28 (gsi=28)

[    0.781783] xen:grant_table: Grant tables using version 1 layout

[    0.786943] Grant table initialized

[    0.792598] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled

[    0.828186] 00:06: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A

[    0.835759] Non-volatile memory driver v1.3

[    0.839144] Linux agpgart interface v0.103

[    0.842568] AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>

[    0.845865] AMD IOMMUv2 functionality not available on this system

[    0.849200] [drm] Initialized drm 1.1.0 20060810

[    0.853115] [drm] Found bochs VGA, ID 0xb0c4.

[    0.856372] [drm] Framebuffer size 8192 kB @ 0xf1000000, ioports @ 0x1ce.

[    0.859788] [TTM] Zone  kernel: Available graphics memory: 1021590 kiB

[    0.863120] [TTM] Initializing pool allocator

[    0.866389] [TTM] Initializing DMA pool allocator

[    6.816112] random: nonblocking pool is initialized

[   21.846327] INFO: rcu_sched self-detected stall on CPU { 1}  (t=23321 jiffies g=-238 c=-239 q=27)

[   21.846327] rcu_sched kthread starved for 18649 jiffies! g18446744073709551378 c18446744073709551377 f0x0

[   21.846327] Task dump for CPU 1:

[   21.846327] swapper/0       R  running task        0     1      0 0x00000008

[   21.846327]  ffffffff818e4740 ffff88007f443c08 ffffffff810756ba 0000000000000001

[   21.846327]  ffffffff818e4740 ffff88007f443c28 ffffffff810778a8 0000000000000083

[   21.846327]  0000000000000002 ffff88007f443c58 ffffffff81094df0 ffff88007f455940

[   21.846327] Call Trace:

[   21.846327]  <IRQ>  [<ffffffff810756ba>] sched_show_task+0xaa/0x110

[   21.846327]  [<ffffffff810778a8>] dump_cpu_task+0x38/0x40

[   21.846327]  [<ffffffff81094df0>] rcu_dump_cpu_stacks+0x80/0xc0

[   21.846327]  [<ffffffff81097cae>] rcu_check_callbacks+0x3ce/0x6a0

[   21.846327]  [<ffffffff810d3937>] ? acct_account_cputime+0x17/0x20

[   21.846327]  [<ffffffff8107832c>] ? account_system_time+0x7c/0x120

[   21.846327]  [<ffffffff81085f21>] ? __raw_callee_save___pv_queued_spin_unlock+0x11/0x20

[   21.846327]  [<ffffffff8109a994>] update_process_times+0x34/0x60

[   21.846327]  [<ffffffff810a8551>] tick_sched_handle.isra.17+0x31/0x40

[   21.846327]  [<ffffffff810a859c>] tick_sched_timer+0x3c/0x80

[   21.846327]  [<ffffffff8109b00b>] __hrtimer_run_queues+0xab/0x130

[   21.846327]  [<ffffffff8109b5a3>] hrtimer_interrupt+0xa3/0x1a0

[   21.846327]  [<ffffffff8100879e>] xen_timer_interrupt+0x2e/0x140

[   21.846327]  [<ffffffff8108d67c>] handle_irq_event_percpu+0x5c/0x100

[   21.846327]  [<ffffffff810908e9>] handle_percpu_irq+0x39/0x50

[   21.846327]  [<ffffffff8108ce47>] generic_handle_irq+0x27/0x40

[   21.846327]  [<ffffffff8137b30b>] evtchn_fifo_handle_events+0x14b/0x150

[   21.846327]  [<ffffffff813783a2>] __xen_evtchn_do_upcall+0x42/0x80

[   21.846327]  [<ffffffff8137a02f>] xen_evtchn_do_upcall+0x2f/0x50

[   21.846327]  [<ffffffff81696ccb>] xen_hvm_callback_vector+0x6b/0x70

[   21.846327]  <EOI>  [<ffffffff812eb964>] ? __memset+0x24/0x30

[   21.846327]  [<ffffffff813fd88a>] ? ttm_bo_move_memcpy+0x5da/0x650

[   21.846327]  [<ffffffff814028b3>] bochs_bo_move+0x13/0x20

[   21.846327]  [<ffffffff813fae63>] ttm_bo_handle_move_mem+0x2a3/0x630

[   21.846327]  [<ffffffff813fb903>] ? ttm_bo_mem_space+0x163/0x340

[   21.846327]  [<ffffffff813fbf82>] ttm_bo_validate+0x1c2/0x1d0

[   21.846327]  [<ffffffff81080001>] ? pull_rt_task+0x111/0x2f0

[   21.846327]  [<ffffffff81402e76>] bochs_bo_pin+0xe6/0x160

[   21.846327]  [<ffffffff81403c4d>] bochsfb_create+0xfd/0x3c0

[   21.846327]  [<ffffffff813cb659>] ? drm_helper_probe_single_connector_modes_merge_bits+0x179/0x4d0

[   21.846327]  [<ffffffff813d695e>] drm_fb_helper_initial_config+0x1de/0x3d0

[   21.846327]  [<ffffffff81403f86>] bochs_fbdev_init+0x76/0xa0

[   21.846327]  [<ffffffff814027c7>] bochs_load+0x87/0x90

[   21.846327]  [<ffffffff813df9aa>] drm_dev_register+0xaa/0x100

[   21.846327]  [<ffffffff813e200e>] drm_get_pci_dev+0xce/0x1e0

[   21.846327]  [<ffffffff8140285e>] bochs_pci_probe+0x8e/0xa0

[   21.846327]  [<ffffffff8130e6f0>] local_pci_probe+0x40/0xa0

[   21.846327]  [<ffffffff8130e65f>] ? pci_match_device+0xef/0x110

[   21.846327]  [<ffffffff8130e812>] pci_device_probe+0xc2/0x110

[   21.846327]  [<ffffffff8140c181>] driver_probe_device+0x131/0x2d0

[   21.846327]  [<ffffffff8140c3ab>] __driver_attach+0x8b/0x90

[   21.846327]  [<ffffffff8140c320>] ? driver_probe_device+0x2d0/0x2d0

[   21.846327]  [<ffffffff8140a3ff>] bus_for_each_dev+0x5f/0x90

[   21.846327]  [<ffffffff8140bd09>] driver_attach+0x19/0x20

[   21.846327]  [<ffffffff8140b95b>] bus_add_driver+0x1ab/0x220

[   21.846327]  [<ffffffff8140cbdb>] driver_register+0x5b/0xe0

[   21.846327]  [<ffffffff8130dea7>] __pci_register_driver+0x47/0x50

[   21.846327]  [<ffffffff813e21f6>] drm_pci_init+0xd6/0x100

[   21.846327]  [<ffffffff81990424>] ? ttm_init+0x6a/0x6a

[   21.846327]  [<ffffffff8199043b>] bochs_init+0x17/0x19

[   21.846327]  [<ffffffff810020f3>] do_one_initcall+0x83/0x1b0

[   21.846327]  [<ffffffff8195e04e>] kernel_init_freeable+0x164/0x1ec

[   21.846327]  [<ffffffff816875b0>] ? rest_init+0x80/0x80

[   21.846327]  [<ffffffff816875b9>] kernel_init+0x9/0xe0

[   21.846327]  [<ffffffff81695a5f>] ret_from_fork+0x3f/0x70

[   21.846327]  [<ffffffff816875b0>] ? rest_init+0x80/0x80

[   84.962072] INFO: rcu_sched self-detected stall on CPU { 1}  (t=86423 jiffies g=-238 c=-239 q=27)

[   84.962072] rcu_sched kthread starved for 81751 jiffies! g18446744073709551378 c18446744073709551377 f0x0

[   84.962072] Task dump for CPU 1:

[   84.962072] swapper/0       R  running task        0     1      0 0x00000008

[   84.962072]  ffffffff818e4740 ffff88007f443c08 ffffffff810756ba 0000000000000001

[   84.962072]  ffffffff818e4740 ffff88007f443c28 ffffffff810778a8 0000000000000083

[   84.962072]  0000000000000002 ffff88007f443c58 ffffffff81094df0 ffff88007f455940

[   84.962072] Call Trace:

[   84.962072]  <IRQ>  [<ffffffff810756ba>] sched_show_task+0xaa/0x110

[   84.962072]  [<ffffffff810778a8>] dump_cpu_task+0x38/0x40

[   84.962072]  [<ffffffff81094df0>] rcu_dump_cpu_stacks+0x80/0xc0

[   84.962072]  [<ffffffff81097cae>] rcu_check_callbacks+0x3ce/0x6a0

[   84.962072]  [<ffffffff810d3937>] ? acct_account_cputime+0x17/0x20

[   84.962072]  [<ffffffff8107832c>] ? account_system_time+0x7c/0x120

[   84.962072]  [<ffffffff81085f21>] ? __raw_callee_save___pv_queued_spin_unlock+0x11/0x20

[   84.962072]  [<ffffffff8109a994>] update_process_times+0x34/0x60

[   84.962072]  [<ffffffff810a8551>] tick_sched_handle.isra.17+0x31/0x40

[   84.962072]  [<ffffffff810a859c>] tick_sched_timer+0x3c/0x80

[   84.962072]  [<ffffffff8109b00b>] __hrtimer_run_queues+0xab/0x130

[   84.962072]  [<ffffffff8109b5a3>] hrtimer_interrupt+0xa3/0x1a0

[   84.962072]  [<ffffffff8100879e>] xen_timer_interrupt+0x2e/0x140

[   84.962072]  [<ffffffff8108d67c>] handle_irq_event_percpu+0x5c/0x100

[   84.962072]  [<ffffffff810908e9>] handle_percpu_irq+0x39/0x50

[   84.962072]  [<ffffffff8108ce47>] generic_handle_irq+0x27/0x40

[   84.962072]  [<ffffffff8137b30b>] evtchn_fifo_handle_events+0x14b/0x150

[   84.962072]  [<ffffffff813783a2>] __xen_evtchn_do_upcall+0x42/0x80

[   84.962072]  [<ffffffff8137a02f>] xen_evtchn_do_upcall+0x2f/0x50

[   84.962072]  [<ffffffff81696ccb>] xen_hvm_callback_vector+0x6b/0x70

[   84.962072]  <EOI>  [<ffffffff812eb964>] ? __memset+0x24/0x30

[   84.962072]  [<ffffffff813fd88a>] ? ttm_bo_move_memcpy+0x5da/0x650

[   84.962072]  [<ffffffff814028b3>] bochs_bo_move+0x13/0x20

[   84.962072]  [<ffffffff813fae63>] ttm_bo_handle_move_mem+0x2a3/0x630

[   84.962072]  [<ffffffff813fb903>] ? ttm_bo_mem_space+0x163/0x340

[   84.962072]  [<ffffffff813fbf82>] ttm_bo_validate+0x1c2/0x1d0

[   84.962072]  [<ffffffff81080001>] ? pull_rt_task+0x111/0x2f0

[   84.962072]  [<ffffffff81402e76>] bochs_bo_pin+0xe6/0x160

[   84.962072]  [<ffffffff81403c4d>] bochsfb_create+0xfd/0x3c0

[   84.962072]  [<ffffffff813cb659>] ? drm_helper_probe_single_connector_modes_merge_bits+0x179/0x4d0

[   84.962072]  [<ffffffff813d695e>] drm_fb_helper_initial_config+0x1de/0x3d0

[   84.962072]  [<ffffffff81403f86>] bochs_fbdev_init+0x76/0xa0

[   84.962072]  [<ffffffff814027c7>] bochs_load+0x87/0x90

[   84.962072]  [<ffffffff813df9aa>] drm_dev_register+0xaa/0x100

[   84.962072]  [<ffffffff813e200e>] drm_get_pci_dev+0xce/0x1e0

[   84.962072]  [<ffffffff8140285e>] bochs_pci_probe+0x8e/0xa0

[   84.962072]  [<ffffffff8130e6f0>] local_pci_probe+0x40/0xa0

[   84.962072]  [<ffffffff8130e65f>] ? pci_match_device+0xef/0x110

[   84.962072]  [<ffffffff8130e812>] pci_device_probe+0xc2/0x110

[   84.962072]  [<ffffffff8140c181>] driver_probe_device+0x131/0x2d0

[   84.962072]  [<ffffffff8140c3ab>] __driver_attach+0x8b/0x90

[   84.962072]  [<ffffffff8140c320>] ? driver_probe_device+0x2d0/0x2d0

[   84.962072]  [<ffffffff8140a3ff>] bus_for_each_dev+0x5f/0x90

[   84.962072]  [<ffffffff8140bd09>] driver_attach+0x19/0x20

[   84.962072]  [<ffffffff8140b95b>] bus_add_driver+0x1ab/0x220

[   84.962072]  [<ffffffff8140cbdb>] driver_register+0x5b/0xe0

[   84.962072]  [<ffffffff8130dea7>] __pci_register_driver+0x47/0x50

[   84.962072]  [<ffffffff813e21f6>] drm_pci_init+0xd6/0x100

[   84.962072]  [<ffffffff81990424>] ? ttm_init+0x6a/0x6a

[   84.962072]  [<ffffffff8199043b>] bochs_init+0x17/0x19

[   84.962072]  [<ffffffff810020f3>] do_one_initcall+0x83/0x1b0

[   84.962072]  [<ffffffff8195e04e>] kernel_init_freeable+0x164/0x1ec

[   84.962072]  [<ffffffff816875b0>] ? rest_init+0x80/0x80

[   84.962072]  [<ffffffff816875b9>] kernel_init+0x9/0xe0

[   84.962072]  [<ffffffff81695a5f>] ret_from_fork+0x3f/0x70

[   84.962072]  [<ffffffff816875b0>] ? rest_init+0x80/0x80

[  150.003525] INFO: rcu_sched self-detected stall on CPU { 1}  (t=151445 jiffies g=-238 c=-239 q=27)

[  150.003525] rcu_sched kthread starved for 146773 jiffies! g18446744073709551378 c18446744073709551377 f0x0

[  150.003525] Task dump for CPU 1:

[  150.003525] swapper/0       R  running task        0     1      0 0x00000008

[  150.003525]  ffffffff818e4740 ffff88007f443c08 ffffffff810756ba 0000000000000001

[  150.003525]  ffffffff818e4740 ffff88007f443c28 ffffffff810778a8 0000000000000083

[  150.003525]  0000000000000002 ffff88007f443c58 ffffffff81094df0 ffff88007f455940

[  150.003525] Call Trace:

[  150.003525]  <IRQ>  [<ffffffff810756ba>] sched_show_task+0xaa/0x110

[  150.003525]  [<ffffffff810778a8>] dump_cpu_task+0x38/0x40

[  150.003525]  [<ffffffff81094df0>] rcu_dump_cpu_stacks+0x80/0xc0

[  150.003525]  [<ffffffff81097cae>] rcu_check_callbacks+0x3ce/0x6a0

[  150.003525]  [<ffffffff810d3937>] ? acct_account_cputime+0x17/0x20

[  150.003525]  [<ffffffff8107832c>] ? account_system_time+0x7c/0x120

[  150.003525]  [<ffffffff81085f21>] ? __raw_callee_save___pv_queued_spin_unlock+0x11/0x20

[  150.003525]  [<ffffffff8109a994>] update_process_times+0x34/0x60

[  150.003525]  [<ffffffff810a8551>] tick_sched_handle.isra.17+0x31/0x40

[  150.003525]  [<ffffffff810a859c>] tick_sched_timer+0x3c/0x80

[  150.003525]  [<ffffffff8109b00b>] __hrtimer_run_queues+0xab/0x130

[  150.003525]  [<ffffffff8109b5a3>] hrtimer_interrupt+0xa3/0x1a0

[  150.003525]  [<ffffffff8100879e>] xen_timer_interrupt+0x2e/0x140

[  150.003525]  [<ffffffff8108d67c>] handle_irq_event_percpu+0x5c/0x100

[  150.003525]  [<ffffffff810908e9>] handle_percpu_irq+0x39/0x50

[  150.003525]  [<ffffffff8108ce47>] generic_handle_irq+0x27/0x40

[  150.003525]  [<ffffffff8137b30b>] evtchn_fifo_handle_events+0x14b/0x150

[  150.003525]  [<ffffffff813783a2>] __xen_evtchn_do_upcall+0x42/0x80

[  150.003525]  [<ffffffff8137a02f>] xen_evtchn_do_upcall+0x2f/0x50

[  150.003525]  [<ffffffff81696ccb>] xen_hvm_callback_vector+0x6b/0x70

[  150.003525]  <EOI>  [<ffffffff812eb964>] ? __memset+0x24/0x30

[  150.003525]  [<ffffffff813fd88a>] ? ttm_bo_move_memcpy+0x5da/0x650

[  150.003525]  [<ffffffff814028b3>] bochs_bo_move+0x13/0x20

[  150.003525]  [<ffffffff813fae63>] ttm_bo_handle_move_mem+0x2a3/0x630

[  150.003525]  [<ffffffff813fb903>] ? ttm_bo_mem_space+0x163/0x340

[  150.003525]  [<ffffffff813fbf82>] ttm_bo_validate+0x1c2/0x1d0

[  150.003525]  [<ffffffff81080001>] ? pull_rt_task+0x111/0x2f0

[  150.003525]  [<ffffffff81402e76>] bochs_bo_pin+0xe6/0x160

[  150.003525]  [<ffffffff81403c4d>] bochsfb_create+0xfd/0x3c0

[  150.003525]  [<ffffffff813cb659>] ? drm_helper_probe_single_connector_modes_merge_bits+0x179/0x4d0

[  150.003525]  [<ffffffff813d695e>] drm_fb_helper_initial_config+0x1de/0x3d0

[  150.003525]  [<ffffffff81403f86>] bochs_fbdev_init+0x76/0xa0

[  150.003525]  [<ffffffff814027c7>] bochs_load+0x87/0x90

[  150.003525]  [<ffffffff813df9aa>] drm_dev_register+0xaa/0x100

[  150.003525]  [<ffffffff813e200e>] drm_get_pci_dev+0xce/0x1e0

[  150.003525]  [<ffffffff8140285e>] bochs_pci_probe+0x8e/0xa0

[  150.003525]  [<ffffffff8130e6f0>] local_pci_probe+0x40/0xa0

[  150.003525]  [<ffffffff8130e65f>] ? pci_match_device+0xef/0x110

[  150.003525]  [<ffffffff8130e812>] pci_device_probe+0xc2/0x110

[  150.003525]  [<ffffffff8140c181>] driver_probe_device+0x131/0x2d0

[  150.003525]  [<ffffffff8140c3ab>] __driver_attach+0x8b/0x90

[  150.003525]  [<ffffffff8140c320>] ? driver_probe_device+0x2d0/0x2d0

[  150.003525]  [<ffffffff8140a3ff>] bus_for_each_dev+0x5f/0x90

[  150.003525]  [<ffffffff8140bd09>] driver_attach+0x19/0x20

[  150.003525]  [<ffffffff8140b95b>] bus_add_driver+0x1ab/0x220

[  150.003525]  [<ffffffff8140cbdb>] driver_register+0x5b/0xe0

[  150.003525]  [<ffffffff8130dea7>] __pci_register_driver+0x47/0x50

[  150.003525]  [<ffffffff813e21f6>] drm_pci_init+0xd6/0x100

[  150.003525]  [<ffffffff81990424>] ? ttm_init+0x6a/0x6a

[  150.003525]  [<ffffffff8199043b>] bochs_init+0x17/0x19

[  150.003525]  [<ffffffff810020f3>] do_one_initcall+0x83/0x1b0

[  150.003525]  [<ffffffff8195e04e>] kernel_init_freeable+0x164/0x1ec

[  150.003525]  [<ffffffff816875b0>] ? rest_init+0x80/0x80

[  150.003525]  [<ffffffff816875b9>] kernel_init+0x9/0xe0

[  150.003525]  [<ffffffff81695a5f>] ret_from_fork+0x3f/0x70

[  150.003525]  [<ffffffff816875b0>] ? rest_init+0x80/0x80

[  215.286173] INFO: rcu_sched self-detected stall on CPU { 1}  (t=216742 jiffies g=-238 c=-239 q=27)

[  215.286173] rcu_sched kthread starved for 212070 jiffies! g18446744073709551378 c18446744073709551377 f0x0

[  215.286173] Task dump for CPU 1:

[  215.286173] swapper/0       R  running task        0     1      0 0x00000008

[  215.286173]  ffffffff818e4740 ffff88007f443c08 ffffffff810756ba 0000000000000001

[  215.286173]  ffffffff818e4740 ffff88007f443c28 ffffffff810778a8 0000000000000083

[  215.286173]  0000000000000002 ffff88007f443c58 ffffffff81094df0 ffff88007f455940

[  215.286173] Call Trace:

[  215.286173]  <IRQ>  [<ffffffff810756ba>] sched_show_task+0xaa/0x110

[  215.286173]  [<ffffffff810778a8>] dump_cpu_task+0x38/0x40

[  215.286173]  [<ffffffff81094df0>] rcu_dump_cpu_stacks+0x80/0xc0

[  215.286173]  [<ffffffff81097cae>] rcu_check_callbacks+0x3ce/0x6a0

[  215.286173]  [<ffffffff810d3937>] ? acct_account_cputime+0x17/0x20

[  215.286173]  [<ffffffff8107832c>] ? account_system_time+0x7c/0x120

[  215.286173]  [<ffffffff81085f21>] ? __raw_callee_save___pv_queued_spin_unlock+0x11/0x20

[  215.286173]  [<ffffffff8109a994>] update_process_times+0x34/0x60

[  215.286173]  [<ffffffff810a8551>] tick_sched_handle.isra.17+0x31/0x40

[  215.286173]  [<ffffffff810a859c>] tick_sched_timer+0x3c/0x80

[  215.286173]  [<ffffffff8109b00b>] __hrtimer_run_queues+0xab/0x130

[  215.286173]  [<ffffffff8109b5a3>] hrtimer_interrupt+0xa3/0x1a0

[  215.286173]  [<ffffffff8100879e>] xen_timer_interrupt+0x2e/0x140

[  215.286173]  [<ffffffff8108d67c>] handle_irq_event_percpu+0x5c/0x100

[  215.286173]  [<ffffffff810908e9>] handle_percpu_irq+0x39/0x50

[  215.286173]  [<ffffffff8108ce47>] generic_handle_irq+0x27/0x40

[  215.286173]  [<ffffffff8137b30b>] evtchn_fifo_handle_events+0x14b/0x150

[  215.286173]  [<ffffffff813783a2>] __xen_evtchn_do_upcall+0x42/0x80

[  215.286173]  [<ffffffff8137a02f>] xen_evtchn_do_upcall+0x2f/0x50

[  215.286173]  [<ffffffff81696ccb>] xen_hvm_callback_vector+0x6b/0x70

[  215.286173]  <EOI>  [<ffffffff812eb964>] ? __memset+0x24/0x30

[  215.286173]  [<ffffffff813fd88a>] ? ttm_bo_move_memcpy+0x5da/0x650

[  215.286173]  [<ffffffff814028b3>] bochs_bo_move+0x13/0x20

[  215.286173]  [<ffffffff813fae63>] ttm_bo_handle_move_mem+0x2a3/0x630

[  215.286173]  [<ffffffff813fb903>] ? ttm_bo_mem_space+0x163/0x340

[  215.286173]  [<ffffffff813fbf82>] ttm_bo_validate+0x1c2/0x1d0

[  215.286173]  [<ffffffff81080001>] ? pull_rt_task+0x111/0x2f0

[  215.286173]  [<ffffffff81402e76>] bochs_bo_pin+0xe6/0x160

[  215.286173]  [<ffffffff81403c4d>] bochsfb_create+0xfd/0x3c0

[  215.286173]  [<ffffffff813cb659>] ? drm_helper_probe_single_connector_modes_merge_bits+0x179/0x4d0

[  215.286173]  [<ffffffff813d695e>] drm_fb_helper_initial_config+0x1de/0x3d0

[  215.286173]  [<ffffffff81403f86>] bochs_fbdev_init+0x76/0xa0

[  215.286173]  [<ffffffff814027c7>] bochs_load+0x87/0x90

[  215.286173]  [<ffffffff813df9aa>] drm_dev_register+0xaa/0x100

[  215.286173]  [<ffffffff813e200e>] drm_get_pci_dev+0xce/0x1e0

[  215.286173]  [<ffffffff8140285e>] bochs_pci_probe+0x8e/0xa0

[  215.286173]  [<ffffffff8130e6f0>] local_pci_probe+0x40/0xa0

[  215.286173]  [<ffffffff8130e65f>] ? pci_match_device+0xef/0x110

[  215.286173]  [<ffffffff8130e812>] pci_device_probe+0xc2/0x110

[  215.286173]  [<ffffffff8140c181>] driver_probe_device+0x131/0x2d0

[  215.286173]  [<ffffffff8140c3ab>] __driver_attach+0x8b/0x90

[  215.286173]  [<ffffffff8140c320>] ? driver_probe_device+0x2d0/0x2d0

[  215.286173]  [<ffffffff8140a3ff>] bus_for_each_dev+0x5f/0x90

[  215.286173]  [<ffffffff8140bd09>] driver_attach+0x19/0x20

[  215.286173]  [<ffffffff8140b95b>] bus_add_driver+0x1ab/0x220

[  215.286173]  [<ffffffff8140cbdb>] driver_register+0x5b/0xe0

[  215.286173]  [<ffffffff8130dea7>] __pci_register_driver+0x47/0x50

[  215.286173]  [<ffffffff813e21f6>] drm_pci_init+0xd6/0x100

[  215.286173]  [<ffffffff81990424>] ? ttm_init+0x6a/0x6a

[  215.286173]  [<ffffffff8199043b>] bochs_init+0x17/0x19

[  215.286173]  [<ffffffff810020f3>] do_one_initcall+0x83/0x1b0

[  215.286173]  [<ffffffff8195e04e>] kernel_init_freeable+0x164/0x1ec

[  215.286173]  [<ffffffff816875b0>] ? rest_init+0x80/0x80

[  215.286173]  [<ffffffff816875b9>] kernel_init+0x9/0xe0

[  215.286173]  [<ffffffff81695a5f>] ret_from_fork+0x3f/0x70

[  215.286173]  [<ffffffff816875b0>] ? rest_init+0x80/0x80

[  225.423476] fbcon: bochsdrmfb (fb0) is primary device

[  225.511319] Console: switching to colour frame buffer device 128x48

[  225.513829] bochs-drm 0000:00:02.0: fb0: bochsdrmfb frame buffer device

[  225.513867] bochs-drm 0000:00:02.0: registered panic notifier

[  225.517061] [drm] Initialized bochs-drm 1.0.0 20130925 for 0000:00:02.0 on minor 0

[  225.519905] loop: module loaded

[  225.521733] ata_piix 0000:00:01.1: version 2.13

[  225.523390] scsi host0: ata_piix

[  225.523836] scsi host1: ata_piix

[  225.523937] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc200 irq 14

[  225.523978] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc208 irq 15

[  225.524238] xen_netfront: Initialising Xen virtual ethernet driver

[  225.526210] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver

[  225.526258] ehci-pci: EHCI PCI platform driver

[  225.526302] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver

[  225.526338] ohci-pci: OHCI PCI platform driver

[  225.526377] uhci_hcd: USB Universal Host Controller Interface driver

[  225.526464] usbcore: registered new interface driver usblp

[  225.526516] usbcore: registered new interface driver usb-storage

[  225.526600] i8042: PNP: PS/2 Controller [PNP0303:PS2K,PNP0f13:PS2M] at 0x60,0x64 irq 1,12

[  225.530314] serio: i8042 KBD port at 0x60,0x64 irq 1

[  225.530453] blkfront: xvda: barrier: enabled; persistent grants: disabled; indirect descriptors: disabled;

[  225.532200] serio: i8042 AUX port at 0x60,0x64 irq 12

[  225.533481] mousedev: PS/2 mouse device common for all mice

[  225.537090] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input2

[  225.537147] input: Xen Virtual Keyboard as /devices/virtual/input/input3

[  225.537236]  xvda: xvda1

[  225.540348] input: Xen Virtual Pointer as /devices/virtual/input/input4

[  225.549960] rtc_cmos 00:02: rtc core: registered rtc_cmos as rtc0

[  225.551740] rtc_cmos 00:02: alarms up to one day, 114 bytes nvram, hpet irqs

[  225.553557] xen_wdt: Xen WatchDog Timer Driver v0.01

[  225.555468] xen_wdt: initialized (timeout=60s, nowayout=1)

[  225.556634] watchdog: Software Watchdog: cannot register miscdev on minor=130 (err=-16).

[  225.557701] watchdog: Software Watchdog: a legacy watchdog module is probably present.

[  225.559965] softdog: Software Watchdog Timer: 0.08 initialized. soft_noboot=0 soft_margin=60 sec soft_panic=0 (nowayout=1)

[  225.562587] device-mapper: ioctl: 4.33.0-ioctl (2015-8-18) initialised: dm-devel@redhat.com

[  225.564468] hidraw: raw HID events driver (C) Jiri Kosina

[  225.566469] usbcore: registered new interface driver usbhid

[  225.567780] usbhid: USB HID core driver

[  225.570207] NET: Registered protocol family 26

[  225.571531] Netfilter messages via NETLINK v0.30.

[  225.572836] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)

[  225.574602] ctnetlink v0.93: registering with nfnetlink.

[  225.576176] ip_tables: (C) 2000-2006 Netfilter Core Team

[  225.577482] geneve_core: Geneve core logic

[  225.578632] Initializing XFRM netlink socket

[  225.579773] NET: Registered protocol family 17

[  225.580967] bridge: automatic filtering via arp/ip/ip6tables has been deprecated. Update your scripts to load br_netfilter if you need this.

[  225.583316] l2tp_core: L2TP core driver, V2.0

[  225.584519] Key type dns_resolver registered

[  225.586190] registered taskstats version 1

[  225.588223]   Magic number: 3:642:195

[  225.589786] acpi device:50: hash matches

[  225.591260] ALSA device list:

[  225.592563]   No soundcards found.

[  226.175106] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input6

[  226.177509] md: Waiting for all devices to be available before autodetect

[  226.178885] md: If you don't use raid, use raid=noautodetect

[  226.180794] md: Autodetecting RAID arrays.

[  226.182167] md: Scanned 0 and added 0 devices.

[  226.183556] md: autorun ...

[  226.184866] md: ... autorun DONE.

[  226.188611] XFS (xvda1): Mounting V4 Filesystem

[  226.758530] XFS (xvda1): Ending clean mount

[  226.759888] VFS: Mounted root (xfs filesystem) readonly on device 202:1.

[  226.761820] devtmpfs: mounted

[  226.764661] Freeing unused kernel memory: 1000K (ffffffff81947000 - ffffffff81a41000)

[  228.118811] systemd-udevd[1442]: starting version 216

[  228.254569] FDC 0 is a S82078B

```

This error delayed kernel boot about 3 minute but after booting system worked

----------

## TigerJr

Found more INFO: rcu_sched self-detected stall on CPU :

https://www.kernel.org/doc/Documentation/RCU/stallwarn.txt

 *Quote:*   

> What Causes RCU CPU Stall Warnings?
> 
> So your kernel printed an RCU CPU stall warning.  The next question is
> 
> "What caused it?"  The following problems can result in RCU CPU stall warnings:
> ...

 

----------

## reith

Very similar crash dump here, But I have a core i3 haswell CPU.  Currently running on 4.0.5 kernel.  I booted in EFI mode and set SMP on.

----------

## Polyatomic

reith, May I?

Could you please paste site service some of the system log. Is your rig on the magnanimous

init factotem?. Say `journalctl -b -1`(-0 is the last boot, -1 the boot before last) You can get a

list with `journalctl  --list-boots` . Of course you prolly new that already, I'm thinking that maybe

a careful observer can analyse the crapout. Prolly not use pastebin.com, its veritable crud. )

----------

## reith

 *Polyatomic wrote:*   

> 
> 
> Could you please paste site service some of the system log. Is your rig on the magnanimous
> 
> init factotem?. Say `journalctl -b -1`(-0 is the last boot, -1 the boot before last) You can get a
> ...

 

I'm not using systemd. Problem happens after system is running for more than 10 hours; so I may see the problem in weekend.

Also I blacklisted nouveau and It's possible I don't see problem again, hopefully.

This is log for a previous crash, but there is no report about possible malfunctionings before it[/code]:

```

Nov 13 19:34:26 localhost kernel: INFO: rcu_sched detected stalls on CPUs/tasks: { 2} (detected by 3, t=60002 jiffies, g=194181, c=194180, q=4450)

Nov 13 19:34:26 localhost kernel: Task dump for CPU 2:

Nov 13 19:34:26 localhost kernel: swapper/2       R  running task    31424     0      1 0x00000008

Nov 13 19:34:26 localhost kernel:  0000000000000010 0000000000000246 ffff88010ac47e68 0000000000000018

Nov 13 19:34:26 localhost kernel:  ffffffff815ec4c1 000009d91b6d951f ffffffff81cb96f8 ffff88010ac40000

Nov 13 19:34:26 localhost kernel:  ffffffff81d15140 ffff88010bd24300 ffffffff81cb9500 ffff88010ac40000

Nov 13 19:34:26 localhost kernel: Call Trace:

Nov 13 19:34:26 localhost kernel:  [<ffffffff815ec4c1>] ? cpuidle_enter_state+0x61/0x160

Nov 13 19:34:26 localhost kernel:  [<ffffffff815ec5e2>] ? cpuidle_enter+0x12/0x20

Nov 13 19:34:26 localhost kernel:  [<ffffffff81086b78>] ? cpu_startup_entry+0x2b8/0x3c0

Nov 13 19:34:26 localhost kernel:  [<ffffffff81030318>] ? start_secondary+0x138/0x140

Nov 13 19:34:29 localhost kernel: sysrq: SysRq : Keyboard mode set to system default

Nov 13 19:34:45 localhost syslog-ng[2309]: syslog-ng shutting down; version='3.7.2'

Nov 13 19:34:45 localhost dhcpcd[3017]: received SIGTERM, stopping

Nov 13 19:34:45 localhost dhcpcd[3017]: eno1: removing interface

Nov 13 19:34:45 localhost dictd[2567]: :I: SIGTERM: c/f = 47/0; 03:01:29r 0.000u 0.000s

Nov 13 19:34:30 localhost kernel: sysrq: SysRq : Terminate All Tasks

Nov 13 19:34:45 localhost vnstatd[2708]: SIGTERM received, exiting.

```

----------

## fredbear5150

I am seeing a similar issue on a new build of an AMD Sempron Triple Core CPU. The PC was previously running Gentoo with no problems, it was in a cupboard for a few months and was then dragged out and rebuilt from new. Ever since that rebuild, it has crashed after 2-3 hours use.

It does seem to be during times of "reasonably" heavy loads, it certainly went down a couple of times when I was running Chrome, for the last couple of days I have been rsync-ing my NAS drive to it preparing that for a rebuild.

Here's what's in my /var/log/messages:

Nov 26 14:53:27 arcturus kernel: Call Trace:

Nov 26 14:53:27 arcturus kernel:  [<ffffffff8100cc2b>] ? default_idle+0x1b/0xc0

Nov 26 14:53:27 arcturus kernel:  [<ffffffff8100d6fa>] ? arch_cpu_idle+0xa/0x10

Nov 26 14:53:27 arcturus kernel:  [<ffffffff8108b0c6>] ? cpu_startup_entry+0x2c6/0x390

Nov 26 14:53:27 arcturus kernel:  [<ffffffff810346f3>] ? start_secondary+0x123/0x130

Nov 26 14:56:27 arcturus kernel: INFO: rcu_sched detected stalls on CPUs/tasks: { 2} (detected by 0, t=780022 jiffies, g=44399, c=44398, q=6563)

Nov 26 14:56:27 arcturus kernel: Task dump for CPU 2:

Nov 26 14:56:27 arcturus kernel: swapper/2       R  running task    15440     0      1 0x00000008

Nov 26 14:56:27 arcturus kernel:  0000000000000000 0000000000000000 0000000000000000 0000000000000000

Nov 26 14:56:27 arcturus kernel:  0000000000000000 0000000000000000 ffffffffffffff2e ffffffff8100cc2b

Nov 26 14:56:27 arcturus kernel:  0000000000000010 0000000000000246 ffff88011ab67ea8 0000000000000018

Sometimes the system locks completely, other times the screen freezes though the mouse pointer still moves, you cannot click anything though.

It also appears to always be on the "last" CPU in my three cores, that maybe a common factor with it being on CPU 3 in the Quad Core CPU.

I'm running kernel 4.1.12, here's my emerge --info:

Portage 2.2.20.1 (python 3.4.3-final-0, default/linux/amd64/13.0/desktop, gcc-4.9.3, glibc-2.21-r1, 4.1.12-gentoo x86_64)

=================================================================

System uname: Linux-4.1.12-gentoo-x86_64-AMD_Phenom-tm-_8400_Triple-Core_Processor-with-gentoo-2.2

KiB Mem:     4049008 total,   2274708 free

KiB Swap:    2621432 total,   2621432 free

Timestamp of repository gentoo: Thu, 26 Nov 2015 09:15:01 +0000

sh bash 4.3_p39

ld GNU ld (Gentoo 2.25.1 p1.1) 2.25.1

app-shells/bash:          4.3_p39::gentoo

dev-java/java-config:     2.2.0::gentoo

dev-lang/perl:            5.20.2::gentoo

dev-lang/python:          2.7.10-r1::gentoo, 3.4.3::gentoo

dev-util/cmake:           3.3.1-r1::gentoo

dev-util/pkgconfig:       0.28-r2::gentoo

sys-apps/baselayout:      2.2::gentoo

sys-apps/openrc:          0.17::gentoo

sys-apps/sandbox:         2.6-r1::gentoo

sys-devel/autoconf:       2.13::gentoo, 2.69::gentoo

sys-devel/automake:       1.10.3-r1::gentoo, 1.11.6-r1::gentoo, 1.13.4::gentoo, 1.14.1::gentoo, 1.15::gentoo

sys-devel/binutils:       2.25.1-r1::gentoo

sys-devel/gcc:            4.9.3::gentoo

sys-devel/gcc-config:     1.7.3::gentoo

sys-devel/libtool:        2.4.6::gentoo

sys-devel/make:           4.1-r1::gentoo

sys-kernel/linux-headers: 3.18::gentoo (virtual/os-headers)

sys-libs/glibc:           2.21-r1::gentoo

Repositories:

gentoo

    location: /usr/portage

    sync-type: rsync

    sync-uri: rsync://rsync.uk.gentoo.org/gentoo-portage

    priority: -1000

steam-overlay

    location: /var/lib/layman/steam-overlay

    masters: gentoo

    priority: 0

sunrise

    location: /var/lib/layman/sunrise

    masters: gentoo

    priority: 1

rion

    location: /var/lib/layman/rion

    masters: gentoo

    priority: 2

bumblebee

    location: /var/lib/layman/bumblebee

    masters: gentoo

    priority: 3

games-overlay

    location: /var/lib/layman/games-overlay

    masters: gentoo

    priority: 4

init6

    location: /var/lib/layman/init6

    masters: gentoo

    priority: 5

mate

    location: /var/lib/layman/mate

    masters: gentoo

    priority: 6

jorgicio

    location: /var/lib/layman/jorgicio

    masters: gentoo

    priority: 7

sakaki-tools

    location: /usr/local/portage/sakaki-tools

    sync-type: git

    sync-uri: https://github.com/sakaki-/sakaki-tools.git

    masters: gentoo

    priority: 50

ACCEPT_KEYWORDS="amd64"

ACCEPT_LICENSE="*"

CBUILD="x86_64-pc-linux-gnu"

CFLAGS="-march=amdfam10 -O2 -pipe"

CHOST="x86_64-pc-linux-gnu"

CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt"

CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"

CXXFLAGS="-march=amdfam10 -O2 -pipe"

DISTDIR="/usr/portage/distfiles"

EMERGE_DEFAULT_OPTS="--jobs=4 --load-average=3"

FCFLAGS="-O2 -pipe"

FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"

FFLAGS="-O2 -pipe"

GENTOO_MIRRORS="ftp://mirror.bytemark.co.uk/gentoo/ http://mirror.bytemark.co.uk/gentoo/ rsync://mirror.bytemark.co.uk/gentoo/ http://mirror.qubenet.net/mirror/gentoo/ ftp://ftp.mirrorservice.org/sites/distfiles.gentoo.org/ http://www.mirrorservice.org/sites/distfiles.gentoo.org/ rsync://rsync.mirrorservice.org/distfiles.gentoo.org/"

LANG="en_GB.UTF-8"

LDFLAGS="-Wl,-O1 -Wl,--as-needed"

MAKEOPTS="-j4 -l3"

PKGDIR="/usr/portage/packages"

PORTAGE_CONFIGROOT="/"

PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"

PORTAGE_TMPDIR="/var/tmp"

USE="3dnow 3dnowext X a52 aac acl acpi alsa amd64 avahi berkdb bindist bluetooth branding bzip2 cairo cdda cddb cdparanoia cdr cli consolekit cracklib crypt css cups cxx dbus djvu dri dts dvd dvdr emboss encode exif fam ffmpeg firefox flac fortran gdbm gif glamor gnome-keyring gphoto2 gpm gtk iconv icu ipv6 java javascript jpeg jpeg2k lame lcms ldap libnotify lm_sensors lock mad mmx mmxext mng modules mp3 mp4 mpeg mtp multilib musepack ncurses nls nptl nsplugin ogg opengl openmp pam pango pcre pdf perl php png policykit popcnt ppds pulseaudio python qt3support readline ruby samba sdl seccomp session speex spell sse sse2 sse3 sse4a ssl startup-notification svg tagwriting tcpd tiff timidity truetype udev udisks unicode upnp upnp-av upower usb vorbis wavpack webkit wmf wxwidgets x264 xattr xcb xml xv xvid zeroconf zlib" ABI_X86="32 64" ALSA_CARDS="cs46xx hda-intel" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="3dnow 3dnowext mmx mmxext popcnt sse sse2 sse3 sse4a" CURL_SSL="nss" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="evdev mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="en en_GB" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-5" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_4" RUBY_TARGETS="ruby20 ruby21" SANE_BACKENDS="plustek canon hp" USERLAND="GNU" VIDEO_CARDS="nouveau" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"

Unset:  CC, CPPFLAGS, CTARGET, CXX, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON

----------

## fredbear5150

Just to add to my previous post, I note the comments about the nouveau driver possibly being the cause.

I should add that with my newly built system, which was running perfectly stable a few months ago under an older kernel, I did replace the radeon card in it then for a nouveau (NVIDIA) card now - so there seems to be a common factor there.

I have just reconfigured the system and put the old ATI card back in, it will be interesting to see if it is now stable. The next couple of hours will tell.

----------

## fredbear5150

Just to confirm that having replaced the NVIDIA card using the nouveau driver with an ATI card using the radeon driver, the system has been up and running and entirely stable for several hours now - it would have crashed out by this time were the original problem still present.

Therefore I think the conclusion that it's a nouveau driver problem is correct. I haven't tried the closed NVIDIA drivers, I don't need to use either those or the closed ATI drivers in my general computing work.

It would be interesting to see what the actual problem is in the nouveau driver that seems to be causing this issue.

----------

## fredbear5150

My final post on the issue but just to confirm that I left the PC doing an "rsync" overnight from a NAS drive to its local hard disk and it was still happily running this morning - with the NVIDIA card it would have definitely have crashed, this is the longest time the PC has been up since the recent rebuild.

----------

## TigerJr

What kernel are you use before 4.1.12 version?

----------

## ElDavo

Finally - Someone with my same issue!

It is so strange. System hangs randomly. Sometimes I can move the mouse on X. This usually happens with Internet browsing.

Intel Pentium N3520 with its integrated Intel GPU Card. Sometimes, the browser hangs and I see the very same dmesg messages you posted. No CPU usage, but high load average.

Something I found really interesting is that with the default Gentoo Linux kernel this bug does not happen, but when I start customizing the configuration, it happens.

I struggled for months to understand what was causing the issue, and I believe I found it, even though I had many false positives in the past.

Please tell us what option did you choose in General Setup > Timers Subsystem > Timer Tick Handling.

The default is tickless idle and I have no problems with that.

I have this problem with Full dynticks system (tickless) ( CONFIG_NO_HZ_FULL ) . 

The strange fact is that this option doesn't change anything by default because you have to put your cpu numbers in the nohz_full= boot parameter, yet changing the option causes a complete kernel recompilation.

----------

