# Segmentation fault with new kernel after Init process

## eric-gentoo

Hello, I am having trouble booting with kernel 5.4.38.  It does a segmentation fault after the Init process and in the end it says "block device UUID=0081cebf-52b7-4bc4-a396-5d9c8d74014e is not a valid root device.  I don't think the root cause of the issue is the booting process not finding a valid root device.  Something else is causing the kernel to segmentation fault and not finding the root device.  This system is working with kernel 4.19.86 build with genkernel.  Tried also building the kernel manually and it does the same thing at the same place.  I did a genkernel build with the default config to see if it was something with the .config of the old kernel that was causing the issue and I still have the problem.  I also build an initramfs with genkernel.

The UUID of the root partition is good: 

```
Gentoo ~ # lsblk -f /dev/sda3

NAME FSTYPE LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINT

sda3 ext3   ROOT  0081cebf-52b7-4bc4-a396-5d9c8d74014e    8.3G    11% /

Gentoo ~ #
```

When it does the segmentation fault right after the Init process, I see errors like this in the console of the server.

```
traps: uname[198] general protection fault ip:80550bc sp:bfde498c error:0 in busybox[8049000+16c000]

traps: cut[201] general protection fault ip:80550bc sp:bfde498c error:0 in busybox[8049000+16c000]

traps: mknod[205] general protection fault ip:80550bc sp:bfde498c error:0 in busybox[8049000+16c000]

traps: mknod[206] general protection fault ip:80550bc sp:bfde498c error:0 in busybox[8049000+16c000]

```

I re-emerged busybox, coreutils, openrc.

One thing I am wondering about is I have not configured the partitions in the fstab file with UUID.  The partitions are listed with /dev/sdaX.  Is this causing an issue with the booting of newer kernels that I was not facing before? 

I am having the same issue with a physical server and a VM in ESXi.  Any help will be greatly appreciated.  Thank you.

```
Gentoo ~ # emerge --info

Portage 2.3.99 (python 3.6.10-final-0, default/linux/x86/17.0, gcc-9.3.0, glibc-2.30-r8, 4.19.86-gentoo i686)

=================================================================

System uname: Linux-4.19.86-gentoo-i686-Intel-R-_Xeon-R-_CPU_E5504_@_2.00GHz-with-gentoo-2.6

KiB Mem:     3109904 total,   2164172 free

KiB Swap:     530428 total,    530428 free

Head commit of repository mv: ed218d791ccbbf68dafd30c24c737510c4f989fa

Head commit of repository spikyatlinux: 0de940c5f2c4b56b9d8240194f643db896adf606

Head commit of repository vmacs: 47c9f6a71b7ed85c63cb481c624ee7fc4892f565

Timestamp of repository gentoo: Sat, 09 May 2020 19:45:01 +0000

Head commit of repository gentoo: ff847fd1b212359109b301db4e349a433ad604d3

sh bash 4.4_p23-r1

ld GNU ld (Gentoo 2.32 p2) 2.32.0

app-shells/bash:          4.4_p23-r1::gentoo

dev-lang/perl:            5.30.1::gentoo

dev-lang/python:          2.7.18::gentoo, 3.5.7::gentoo, 3.6.10-r2::gentoo, 3.7.7-r2::gentoo, 3.8.2-r2::gentoo

dev-util/cmake:           3.14.6::gentoo

dev-util/pkgconfig:       0.29.2::gentoo

sys-apps/baselayout:      2.6-r1::gentoo

sys-apps/openrc:          0.42.1::gentoo

sys-apps/sandbox:         2.13::gentoo

sys-devel/autoconf:       2.69-r4::gentoo

sys-devel/automake:       1.16.1-r1::gentoo

sys-devel/binutils:       2.32-r1::gentoo, 2.33.1-r1::gentoo

sys-devel/gcc:            9.2.0-r2::gentoo, 9.3.0::gentoo

sys-devel/gcc-config:     2.2.1::gentoo

sys-devel/libtool:        2.4.6-r6::gentoo

sys-devel/make:           4.2.1-r4::gentoo

sys-kernel/linux-headers: 5.4::gentoo (virtual/os-headers)

sys-libs/glibc:           2.30-r8::gentoo

Repositories:

mv

    location: /var/lib/layman/mv

    sync-type: git

    sync-uri: git://anongit.gentoo.org/user/mv.git

    masters: gentoo

    priority: 50

spikyatlinux

    location: /var/lib/layman/spikyatlinux

    sync-type: git

    sync-uri: https://github.com/spikyatlinux/ebuilds_for_gentoo.git

    masters: gentoo

    priority: 50

vmacs

    location: /var/lib/layman/vmacs

    sync-type: git

    sync-uri: git://anongit.gentoo.org/user/vmacs.git

    masters: gentoo

    priority: 50

gentoo

    location: /usr/portage

    sync-type: rsync

    sync-uri: rsync://rsync2.ca.gentoo.org/gentoo-portage

    priority: 1000

    sync-rsync-verify-jobs: 1

    sync-rsync-extra-opts:

    sync-rsync-verify-metamanifest: yes

    sync-rsync-verify-max-age: 24

ACCEPT_KEYWORDS="x86"

ACCEPT_LICENSE="@FREE"

CBUILD="i686-pc-linux-gnu"

CFLAGS="-O2 -march=native -pipe"

CHOST="i686-pc-linux-gnu"

CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt /var/bind /var/spool/munin-async/.ssh"

CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php7.2/ext-active/ /etc/php/apache2-php7.3/ext-active/ /etc/php/apache2-php7.4/ext-active/ /etc/php/cgi-php7.2/ext-active/ /etc/php/cgi-php7.3/ext-active/ /etc/php/cgi-php7.4/ext-active/ /etc/php/cli-php7.2/ext-active/ /etc/php/cli-php7.3/ext-active/ /etc/php/cli-php7.4/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"

CXXFLAGS="-O2 -march=native -pipe"

DISTDIR="/usr/portage/distfiles"

EMERGE_DEFAULT_OPTS="--with-bdeps=y"

ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"

FCFLAGS="-O2 -march=i686 -pipe"

FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"

FFLAGS="-O2 -march=i686 -pipe"

GENTOO_MIRRORS="rsync://gentoo.gossamerhost.com/gentoo-distfiles/ ftp://mirrors.tera-byte.com/pub/gentoo http://gentoo.mirrors.tera-byte.com/ rsync://mirrors.tera-byte.com/gentoo ftp://mirror.csclub.uwaterloo.ca/gentoo-distfiles/ https://mirror.csclub.uwaterloo.ca/gentoo-distfiles/ http://mirror.csclub.uwaterloo.ca/gentoo-distfiles/"

LANG="en_US.utf8"

LDFLAGS="-Wl,-O1 -Wl,--as-needed"

MAKEOPTS="-j3"

PKGDIR="/var/cache/binpkgs"

PORTAGE_CONFIGROOT="/"

PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"

PORTAGE_TMPDIR="/var/tmp"

USE="acl berkdb bzip2 cli crypt dri fortran gdbm iconv libtirpc logrotate ncurses nls nptl openmp pam pcre readline seccomp split-usr ssl tcpd threads unicode x86 xattr zlib" ABI_X86="32" ADA_TARGET="gnat_2018" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1 emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="cgi cgid info rewrite status authz_host authn_core authz_core unixd dir mime alias include negotiation log_config log_forensic logio socache_shmcb setenvif headers" APACHE2_MPMS="event" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="iptables apache bind apcupsd hddtemp df disk md memory mysql network ntpd rrdtool sensors smart cpu interface ethstat dns" CPU_FLAGS_X86="mmx mmxext popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="pc" INPUT_DEVICES="libinput" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" POSTGRES_TARGETS="postgres10 postgres11" PYTHON_SINGLE_TARGET="python3_6" PYTHON_TARGETS="python3_6" RUBY_TARGETS="ruby24 ruby25" USERLAND="GNU" VIDEO_CARDS="amdgpu fbdev intel nouveau radeon radeonsi vesa dummy v4l" XTABLES_ADDONS="account geoip iface ipmark"

Unset:  CC, CPPFLAGS, CTARGET, CXX, INSTALL_MASK, LC_ALL, LINGUAS, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS

Gentoo ~ #

```

----------

## mike155

Hi eric-gentoo,

welcome to the Gentoo forums!

Please post:

The output of 'lspci'

The kernel config of your Linux 4.19.86-gentoo kernel, which works

The kernel config of your Linux 5.4.38 kernel, which doesn't work

Please use wgetpaste to post the kernel configs.

Mike

----------

## eric-gentoo

Hello mike155,

Thank you!

Here is the output of lspci:

http://dpaste.com/2HJADCJ

Here is the output of kernel-config-4.19.86-gentoo-x86:

http://dpaste.com/000N4V7

Here is the output of kernel-config-5.4.38-gentoo-x86:

http://dpaste.com/1B1FYQN

I will take a look myself.  I did try to debug my physical server more than the VM, so I did not take a look as deep on the VM as the physical server.  And the VM is less service impacting if I have to reboot it.  So I will troubleshoot this VM and hope to apply the fix on the physical server.  I have some screenshots of the boot process of the VM.  Is there a way to share those on the forum if you need them?

----------

## mike155

I'm not surprised that your 5.4.38 doesn't work. There are many differences between the two kernel configs!

For example:

one is optimized for M686, the other for CORE2

one has CPU_FREQ enabled, the other not

one has PCI Express enabled, the other not 

 ...

Try the steps below:

Unpack the source code of the latest 5.4 kernel to /usr/src/linux-5.4

Make sure that /usr/src/linux is a link that points to that directory

chdir to /usr/src/linux-5.4

Copy the .config from the 4.19 kernel that works

Run 'make olddefconfig' to migrate the the 4.19 config to version 5.4

compile and install the kernel

See: https://wiki.gentoo.org/wiki/Kernel/Upgrade/en

----------

## eric-gentoo

All those differences are do to the fact that the 5.4.38 kernel was build with the default config of genkernel so there is alot of stuff not present in the customized kernel I am running for the 4.19.86 version.

I wanted to see if the default genkernel config would make my server boot. 

I will rebuild the kernel manually like you said and see how it goes tomorrow.

Thank you.

----------

## eric-gentoo

I forgot that I had 2 kernel 4.19.86 in my server.  One built with genkernel 3.5.x and one built with genkernel 4.0.1.  The kernel built with genkernel 4.0 is showing the same problem as the 5.4 kernel.  So it is not a kernel issue.  I am thinking it is a Initramfs issue.  I have downgraded to genkernel 3.5.3.3 and compiling a new 5.4 kernel manually and generating the initramfs from the 3.5.3.3 genkernel version.

I will post an update as soon as I can.

Thank you

----------

## eric-gentoo

Manually compiling a kernel and generating a initramfs with genkernel 3.5.3.3 is working fine.  I will compile the kernel and generate initramfs with genkernel 3.5.3.3 now and see if it still works.

All this done with kernel version 5.4.38

----------

## eric-gentoo

Fully built kernel and initramfs with genkernel 3.5.3.3 is working fine.

I have rebuilt everything with genkernel 4.0.7-r1 and the problem is back.  So I am thinking there is something wrong in the creation of the initramfs.  Is there a way to build an initramfs without using genkernel?Last edited by eric-gentoo on Tue May 12, 2020 7:44 pm; edited 1 time in total

----------

## eric-gentoo

Is there a way to compare 2 initramfs file?

I copied over the working initramfs (initramfs-genkernel-x86-5.4.38-gentoo) from genkernel 3.5.3.3 to the file name of the genkernel 4.0.7-r1 initramfs (initramfs-5.4.38-gentoo-x86.img) and the system booted correctly.  So the problem is definitely with the initramfs.

----------

## eric-gentoo

I found the lsinitrd command to show the content of the initramfs file.  There are big differences between the initramfs from the 3.5.3.3 genkernel build and the 4.0.7-r1 genkernel initramfs, both with the same version of kernel.

genkernel 3.5.3.3 initramfs:

http://dpaste.com/1X4HK1H

genkernel 4.0.7-r1 initramfs:

http://dpaste.com/2P02755

Maybe lsinitrd is not showing everything in the genkernel 3.5.3.3 initramfs, because it has a size of 9M and it is only showing 7M of files, and no arguments and version information like the 4.0.7 initramfs.

----------

## Hu

initramfs are a particular type of archive.  If you want to be thorough, you could unpack each archive into parallel trees and compare the results to see which files differ and where.

----------

## eric-gentoo

I used lsinitrd --unpack and lsinitrd --unpackearly, to extract the files from the initramfs.  Is that enough?  That did not give me more files than what I see with the lsinitrd /boot/initramfs-genkernel-x86-5.4.38-gentoo for example.

----------

## eric-gentoo

I am back with  a bit more information on the troubleshooting I did.  

I extracted the initramfs file and ran the "uname" symlink in the archive that points to busybox and it segfaults.  

```

Gentoo ~/genkernel4_initramfs/bin # ls -l

total 1904

lrwxrwxrwx 1 root root       7 May 12 20:30 '[' -> busybox

lrwxrwxrwx 1 root root       7 May 12 20:30  ash -> busybox

-rwxr-xr-x 1 root root 1944444 May 12 20:30  busybox

lrwxrwxrwx 1 root root       7 May 12 20:30  cat -> busybox

lrwxrwxrwx 1 root root       7 May 12 20:30  cut -> busybox

lrwxrwxrwx 1 root root       7 May 12 20:30  echo -> busybox

lrwxrwxrwx 1 root root       7 May 12 20:30  mknod -> busybox

lrwxrwxrwx 1 root root       7 May 12 20:30  mount -> busybox

lrwxrwxrwx 1 root root       7 May 12 20:30  sh -> busybox

lrwxrwxrwx 1 root root       7 May 12 20:30  uname -> busybox

Gentoo ~/genkernel4_initramfs/bin # ./uname

Linux

Segmentation fault

Gentoo ~/genkernel4_initramfs/bin # ./uname -a

Linux Gentoo 5.4.38-gentoo #1 SMP Tue May 12 12:24:17 EDT 2020 i686 GNU/Linux

Segmentation fault

Gentoo ~/genkernel4_initramfs/bin #

```

So I guess that is why I get the following error when booting

```

traps: uname[198] general protection fault ip:80550bc sp:bfde498c error:0 in busybox[8049000+16c000] 

```

And I noticed that the file size of busybox is not the same in the initramfs archive than /bin on the server:

```

Gentoo ~/genkernel4_initramfs/bin # ls -l /bin/busybox

-rwxr-xr-x 1 root root 2202492 May 22 11:09 /bin/busybox

Gentoo ~/genkernel4_initramfs/bin # ls -l busybox

-rwxr-xr-x 1 root root 1944444 May 12 20:30 busybox

Gentoo ~/genkernel4_initramfs/bin #

```

If I run the busybox command that is from the /bin directory on the server it works:

```

Gentoo ~/genkernel4_initramfs/bin # /bin/busybox uname

Linux

Gentoo ~/genkernel4_initramfs/bin # /bin/busybox uname -a

Linux Gentoo 5.4.38-gentoo #1 SMP Tue May 12 12:24:17 EDT 2020 i686 i686 i386 GNU/Linux

Gentoo ~/genkernel4_initramfs/bin #

```

I re-emerge busybox, recreated the initramfs and I still have the issue.

Anyone have an idea?

Thank you.

----------

## eric-gentoo

Solved with workaround mentioned in this bug ID.

Basically, use a custom busybox config with genkernel initramfs and add this to the custom config: CONFIG_STACK_OPTIMIZATION_386=n

The generated Busybox from genkernel 4.x no longer segfaults on my system.

----------

