# Kernel just stops?

## The_Great_Sephiroth

I have a strange issue I need help solving. I am installing Gentoo onto an old system with a Pentium D (64bit, dual-core) CPU and 2GB of RAM to function as a printing and scanning server. The thing is, it just freezes when booting. No kernel panic, no logging, nothing. It just plain stops after freeing the unused kernel memory. Below is an image of the location it freezes at.

Screenshot of crash/freeze

Where do I begin? I have checked fstab, my kernel config, everything. Below is what I believe is relevant to this issue. Ask if you need anything else.

Kernel Configuration

Output of "lspci -knn"

Output of "cat /proc/cpuinfo"

----------

## Tony0945

Sounds to me like agetty is borked or the init system never launches it. 

I presume you ran lspci etc. on a CD based Linux. Or are you able to ssh in?

What init system and version?

----------

## The_Great_Sephiroth

I use System Rescue CD 4.90 to do my Gentoo installs. Not sure what init versions and such it has. I can get into the chroot and use it just fine via this method, but the kernel never boots. This is a fresh Gentoo install done last week, so it has whatever versions are out currently unless the init and/or agetty has updated since then.

*UPDATE*

I have a tiny bit of initramfs stuff to detect the BTRFS RAID1 array. This works fine on many other Gentoo boxes.

```

CONFIG_INITRAMFS_SOURCE="/usr/src/initramfs/initramfs_list"

```

/usr/src/initramfs/initramfs_list:

```

# directory structure

dir /proc       755 0 0

dir /usr        755 0 0

dir /bin        755 0 0

dir /sys        755 0 0

dir /var        755 0 0

dir /lib64      755 0 0

dir /sbin       755 0 0

dir /mnt        755 0 0

dir /mnt/root   755 0 0

dir /mnt/boot   755 0 0

dir /etc        755 0 0

dir /root       700 0 0

dir /dev        755 0 0

# busybox

file /bin/busybox               /bin/busybox            755 0 0

#

# fsck deps

#

file /sbin/fsck                 /sbin/fsck              755 0 0

file /lib64/libmount.so.1       /lib64/libmount.so.1    755 0 0

file /lib64/libblkid.so.1       /lib64/libblkid.so.1    755 0 0

file /lib64/libc.so.6           /lib64/libc.so.6        755 0 0

file /lib64/libuuid.so.1        /lib64/libuuid.so.1     755 0 0

file /lib64/ld-linux-x86-64.so.2  /lib64/ld-linux-x86-64.so.2 755 0 0

#

#  fsck.ext3 and added deps

#

file /sbin/fsck.ext3            /sbin/fsck.ext3         755 0 0

file /lib64/libext2fs.so.2      /lib64/libext2fs.so.2   755 0 0

file /lib64/libcom_err.so.2     /lib64/libcom_err.so.2  755 0 0

file /lib64/libe2p.so.2         /lib64/libe2p.so.2      755 0 0

file /lib64/libpthread.so.0     /lib64/libpthread.so.0  755 0 0

#

# btrfs utils and added deps

#

file /sbin/btrfs                /sbin/btrfs             755 0 0

file /sbin/btrfs-convert        /sbin/btrfs-convert     755 0 0

file /sbin/btrfs-debug-tree     /sbin/btrfs-debug-tree  755 0 0

file /sbin/btrfs-find-root      /sbin/btrfs-find-root   755 0 0

file /sbin/btrfs-image          /sbin/btrfs-image       755 0 0

file /sbin/btrfs-map-logical    /sbin/btrfs-map-logical 755 0 0

#file /sbin/btrfs-show-super    /sbin/btrfs-show-super  755 0 0

file /sbin/btrfs-zero-log       /sbin/btrfs-zero-log    755 0 0

file /sbin/btrfsck              /sbin/btrfsck           755 0 0

file /sbin/btrfstune            /sbin/btrfstune         755 0 0

file /sbin/mkfs.btrfs           /sbin/mkfs.btrfs        755 0 0

file /lib64/libz.so.1           /lib64/libz.so.1        755 0 0

file /lib64/liblzo2.so          /usr/lib64/liblzo2.so   755 0 0

#

# Nano editor and added deps

#

file /sbin/nano                 /usr/bin/nano           755 0 0

file /lib64/libncursesw.so.6    /lib64/libncursesw.so.6 755 0 0

file /lib64/libdl.so.2          /lib64/libdl.so.2       755 0 0

#

#  mknod and added deps

#

file /sbin/mknod                /bin/mknod              755 0 0

#

#  more viewer

#

file /sbin/more                 /bin/more               755 0 0

#

#  init script

#

file    /init                   /usr/src/initramfs/init 755 0 0

#

#  fstab

#

file    /etc/fstab              /usr/src/initramfs/fstab        644 0 0

```

/usr/src/initramfs/init:

```

#!/bin/busybox sh

rescue_shell() {

        echo "$@"

        echo "Something went wrong. Dropping you to a shell."

        busybox --install -s

        exec /bin/sh

}

mount_root() {

        echo "scanning for btrfs filesystems.... will take about 5-10 seconds"

        /sbin/btrfs device scan

        echo "mounting /mnt/root"

        mount /mnt/root

}

# temporarily mount proc and sys

mount -t proc none /proc

mount -t sysfs none /sys

mount -t devtmpfs none /dev

# disable kernel messages from popping onto the screen

echo 0 > /proc/sys/kernel/printk

# clear the screen

clear

# mounting rootfs on /mnt/root

mount_root || rescue_shell "Error with uuidlabel_root"

echo "All done. Switching to real root."

# clean up. The init process will remount proc sys and dev later

umount /proc

umount /sys

umount /dev

# switch to the real root and execute init

exec switch_root /mnt/root /sbin/init

```

/usr/src/initramfs/fstab:

```

/dev/sda2       /mnt/root               btrfs   device=/dev/sda2,device=/dev/sdb2,defaults,subvol=@root         0 0

/dev/sda2       /mnt/boot               btrfs   device=/dev/sda2,device=/dev/sdb2,defaults,subvol=@boot         0 0

/dev/sda2       /mnt/root/usr/src       btrfs   device=/dev/sda2,device=/dev/sdb2,defaults,subvol=@source       0 0

/dev/sda2       /mnt/root/home          btrfs   device=/dev/sda2,device=/dev/sdb2,defaults,subvol=@home         0 0

```

----------

## NTU

Is this a custom kernel configuration? What options do you have set, can you post the config? Have you been able to boot that same kernel config on the same system before? You should always be able to get some debug info if you configured the kernel hacking section properly. Early printk, compile with frame pointers, compile with debug info, and make sure you have symbols turned on (KALLSYMS) Perhaps also enable the NMI selftest and locking tests (spinlock and mutex)

I have 3 different kernels installed on my system and bounce between them depending on what I'm doing. I do everything from Coreboot development to RTAI kernel hacking. These are some snippets from my test/debug kernel config:

```
#

# printk and dmesg options

#

CONFIG_MESSAGE_LOGLEVEL_DEFAULT=8

CONFIG_BOOT_PRINTK_DELAY=y

#

# Compile-time checks and compiler options

#

CONFIG_DEBUG_INFO=y

CONFIG_READABLE_ASM=y

CONFIG_UNUSED_SYMBOLS=y

CONFIG_PAGE_OWNER=y

CONFIG_DEBUG_FS=y

CONFIG_HEADERS_CHECK=y

CONFIG_DEBUG_SECTION_MISMATCH=y

CONFIG_SECTION_MISMATCH_WARN_ONLY=y

CONFIG_ARCH_WANT_FRAME_POINTERS=y

CONFIG_FRAME_POINTER=y

CONFIG_STACK_VALIDATION=y

CONFIG_DEBUG_KERNEL=y

#

# Memory Debugging

#

CONFIG_PAGE_EXTENSION=y

CONFIG_DEBUG_PAGEALLOC=y

CONFIG_DEBUG_PAGEALLOC_ENABLE_DEFAULT=y

CONFIG_PAGE_POISONING=y

# CONFIG_PAGE_POISONING_NO_SANITY is not set

CONFIG_PAGE_POISONING_ZERO=y

# CONFIG_DEBUG_OBJECTS is not set

CONFIG_SLUB_DEBUG_ON=y

# CONFIG_SLUB_STATS is not set

CONFIG_HAVE_DEBUG_KMEMLEAK=y

# CONFIG_DEBUG_KMEMLEAK is not set

# CONFIG_DEBUG_STACK_USAGE is not set

CONFIG_DEBUG_VM=y

# CONFIG_DEBUG_VM_VMACACHE is not set

# CONFIG_DEBUG_VM_RB is not set

# CONFIG_DEBUG_VM_PGFLAGS is not set

CONFIG_DEBUG_VIRTUAL=y

CONFIG_DEBUG_MEMORY_INIT=y

CONFIG_DEBUG_PER_CPU_MAPS=y

CONFIG_HAVE_DEBUG_STACKOVERFLOW=y

CONFIG_DEBUG_STACKOVERFLOW=y

CONFIG_DEBUG_SHIRQ=y

#

# Debug Lockups and Hangs

#

# CONFIG_LOCKUP_DETECTOR is not set (for you, probably turn this on)

# CONFIG_DETECT_HUNG_TASK is not set (and turn this on to)

# CONFIG_WQ_WATCHDOG is not set

CONFIG_PANIC_ON_OOPS=y

CONFIG_PANIC_ON_OOPS_VALUE=1

CONFIG_PANIC_TIMEOUT=10

CONFIG_SCHED_DEBUG=y

CONFIG_SCHED_INFO=y

CONFIG_SCHEDSTATS=y

CONFIG_SCHED_STACK_END_CHECK=y

CONFIG_DEBUG_TIMEKEEPING=y

#

# Lock Debugging (spinlocks, mutexes, etc...)

#

CONFIG_DEBUG_RT_MUTEXES=y

CONFIG_DEBUG_SPINLOCK=y

CONFIG_DEBUG_MUTEXES=y

# CONFIG_DEBUG_WW_MUTEX_SLOWPATH is not set

# CONFIG_DEBUG_LOCK_ALLOC is not set 

# CONFIG_PROVE_LOCKING is not set

# CONFIG_LOCK_STAT is not set

# CONFIG_DEBUG_ATOMIC_SLEEP is not set

CONFIG_DEBUG_LOCKING_API_SELFTESTS=y

# CONFIG_LOCK_TORTURE_TEST is not set

CONFIG_STACKTRACE=y

# CONFIG_DEBUG_KOBJECT is not set

CONFIG_DEBUG_BUGVERBOSE=y

CONFIG_DEBUG_LIST=y

CONFIG_DEBUG_PI_LIST=y

CONFIG_DEBUG_SG=y

CONFIG_DEBUG_NOTIFIERS=y

CONFIG_DEBUG_CREDENTIALS=y

#

# RCU Debugging

#

# CONFIG_PROVE_RCU is not set

CONFIG_SPARSE_RCU_POINTER=y

# CONFIG_TORTURE_TEST is not set

# CONFIG_RCU_PERF_TEST is not set

# CONFIG_RCU_TORTURE_TEST is not set

CONFIG_RCU_CPU_STALL_TIMEOUT=21

# CONFIG_RCU_TRACE is not set

CONFIG_RCU_EQS_DEBUG=y

# CONFIG_DEBUG_WQ_FORCE_RR_CPU is not set

# CONFIG_DEBUG_BLOCK_EXT_DEVT is not set

# CONFIG_NOTIFIER_ERROR_INJECTION is not set

# CONFIG_FAULT_INJECTION is not set

# CONFIG_LATENCYTOP is not set

CONFIG_USER_STACKTRACE_SUPPORT=y

CONFIG_HAVE_FUNCTION_TRACER=y

CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y

CONFIG_HAVE_DYNAMIC_FTRACE=y

CONFIG_HAVE_DYNAMIC_FTRACE_WITH_REGS=y

CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y

CONFIG_HAVE_SYSCALL_TRACEPOINTS=y

CONFIG_HAVE_FENTRY=y

CONFIG_HAVE_C_RECORDMCOUNT=y

CONFIG_TRACING_SUPPORT=y

# CONFIG_FTRACE is not set

#

# Runtime Testing

#

CONFIG_TEST_LIST_SORT=y

CONFIG_ATOMIC64_SELFTEST=y

CONFIG_TEST_RHASHTABLE=y

CONFIG_TEST_HASH=y

CONFIG_TEST_USER_COPY=m

CONFIG_TEST_BPF=m

CONFIG_TEST_STATIC_KEYS=m

CONFIG_STRICT_DEVMEM=y

CONFIG_IO_STRICT_DEVMEM=y

CONFIG_X86_VERBOSE_BOOTUP=y

CONFIG_EARLY_PRINTK=y

CONFIG_X86_PTDUMP_CORE=y

# CONFIG_X86_PTDUMP is not set

CONFIG_DEBUG_RODATA_TEST=y

CONFIG_DEBUG_WX=y

CONFIG_DEBUG_SET_MODULE_RONX=y

# CONFIG_DEBUG_NX_TEST is not set

CONFIG_DOUBLEFAULT=y

CONFIG_HAVE_MMIOTRACE_SUPPORT=y

CONFIG_IO_DELAY_TYPE_0X80=0

CONFIG_IO_DELAY_TYPE_0XED=1

CONFIG_IO_DELAY_TYPE_UDELAY=2

CONFIG_IO_DELAY_TYPE_NONE=3

# CONFIG_IO_DELAY_0X80 is not set (set this)

# CONFIG_IO_DELAY_0XED is not set

# CONFIG_IO_DELAY_UDELAY is not set

CONFIG_IO_DELAY_NONE=y (this will change to 0X80)

CONFIG_DEFAULT_IO_DELAY_TYPE=3 (this will change to 0)

# CONFIG_DEBUG_BOOT_PARAMS is not set

CONFIG_CPA_DEBUG=y

CONFIG_OPTIMIZE_INLINING=y

CONFIG_DEBUG_ENTRY=y

CONFIG_DEBUG_NMI_SELFTEST=y

CONFIG_X86_DEBUG_FPU=y
```

Additional options:

```
#

# Executable file formats / Emulations

#

CONFIG_BINFMT_ELF=y

CONFIG_COMPAT_BINFMT_ELF=y

CONFIG_ELFCORE=y

CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS=y

CONFIG_BINFMT_SCRIPT=y

# CONFIG_HAVE_AOUT is not set

# CONFIG_BINFMT_MISC is not set

CONFIG_COREDUMP=y

#

# Generic Driver Options

#

CONFIG_ALLOW_DEV_COREDUMP=y
```

Hope this helps!

----------

## The_Great_Sephiroth

My kernel configuration is linked in the first post. It is the second link. It is the complete kernel configuration, pasted straight from .config in the kernel source directory.

Still no luck getting this to boot. I am supposed to have this in Monday and things are looking bad. If I had a log I'd know what to look for but still no go here. Is there anything else I can provide to help diagnose this issue?

----------

## russK

There seems to be no evidence that the kernel unpacked the initramfs or that the init script mounted your root filesystem.

Did you see any evidence scroll by during the boot?

----------

## The_Great_Sephiroth

I do not use an initramfs. That is to say, I do not build one using something like genkernel.

----------

## Tony0945

I just encountered this on one of my machines.

I changed from the no longer in the tree radeon-ucode to linux-firmware.

Then this happened.  Look at your firmware.

----------

## russK

 *The_Great_Sephiroth wrote:*   

> I do not use an initramfs. That is to say, I do not build one using something like genkernel.

 

You have the internal initramfs from CONFIG_INITRAMFS_SOURCE.

I was trying to determine if it stops before or after your init script.  It looks like it stops before your init script.  Maybe Tony0945 is correct.

----------

## The_Great_Sephiroth

No firmware needed for this system. Intel integrated graphics, no ATI/AMD/nVidia. I have no clue how to tell if my script ran since there is no log. I believe it does stop before, but not sure why.

----------

## The_Great_Sephiroth

I rebuilt the kernel from scratch but will not know if it worked until Monday when I am on-site. I removed the old kernel .config, did a "make menuconfig", and went through the entire process again. The current configuration is linked below.

Current kernel configuration

----------

## russK

I noticed the fstab uses sda2 but the screenshot only shows sda1.  Not sure if that is the problem.

----------

## The_Great_Sephiroth

Let's call this, "bad kernel". I booted System Rescue CD, chrooted, synced, installed new kernel, copied the config, did "make oldconfig", answered a few new questions, built it, and it booted. Copied the configuration back to 4.9.34, built, no boot. No clue what the heck is up with that. I am good now, and actually just upgraded to 4.12.5 on the domain controller. Thanks for your help!

----------

