# [SOLVED] unable to mount rootfs after upgrading to 4.19.44

## arcctgx

Hi everyone,

Today I have updated the kernel from 4.19.27-gentoo-r1 to 4.19.44-gentoo without any configuration changes. With the new kernel I have a booting problem if an USB device (pendrive or similar) is plugged in when the system is booting. (If my Kindle wasn't plugged in for charging I'd never have noticed).

It looks as if there is a race condition. Sometimes the pendrive gets detected before the hard drive and gets assigned /dev/sda1, which causes "Kernel panic, unable to mount rootfs" jazz. I tried rebooting many times with a pendrive plugged in using both old and new kernel. With 4.19.44 the pendrive is detected first around 3 out of 4 times. But it never happens with 4.19.27, the hard disk is always detected first.

Initially I thought that the solution would be to change my /etc/fstab to use UUIDs instead of specifying the devices "the old way" as /dev/sdXY. So I did that, but this did not help: the system boots fine with 4.19.44 without any pendrives plugged in, but I'm still getting the behavior described above when the USB device is connected.

I'm probably missing something simple. Perhaps there is some kernel configuration option that I need to enable with the new kernel? Or something else entirely? I would appreciate your help.Last edited by arcctgx on Sat Jun 01, 2019 9:44 pm; edited 1 time in total

----------

## ct85711

The part you need, to use is PARTUUID not UUID.  To use UUID, you need to use a initrd (to read the filesystem), where PARTUUID (partition level) is understood by the kernel without anything, so no initrd is needed.

Included is a link to a previous thread on the same issue (this issue is not dependent on kernel version).

https://forums.gentoo.org/viewtopic-t-1079474-start-0.html

----------

## arcctgx

Thanks for your input, but unfortunaltely using PARTUUIDs instead of UUIDs im my /etc/fstab did not solve the issue I'm having.

I read the thread you were referring to, but I don't see how it applies to my case. In my case kernel correctly detects the rootfs UNLESS there is a USB mass storage device plugged in at boot time.

Some additional info that may be relevant: I'm using grub-2.02-r3, without an initrd. I'm not using disk encryption.

Right now it occured to me that some additional bootloader configuration may be necessary. In my grub.cfg (generated automatically with grub-mkconfig) I can see the following line:

```
linux   /vmlinuz-4.19.44-gentoo root=/dev/sda4 ro printk.time=0
```

For some reason grub-mkconfig is still using /dev/sdXY notation for root. This file is not supposed to be edited manually, so I probably need to change something in /etc/default/grub.

The funny thing is that the manual of grub says:

 *Quote:*   

> ‘GRUB_DISABLE_LINUX_UUID’
> 
>     Normally, grub-mkconfig will generate menu entries that use universally-unique identifiers (UUIDs) to identify the root filesystem to the Linux kernel, using a ‘root=UUID=...’ kernel parameter. This is usually more reliable, but in some cases it may not be appropriate. To disable the use of UUIDs, set this option to ‘true’.
> 
> 

 I had this option commented out by default. Later I set this option explicitly to "false", but grub-mkconfig is still using /dev/sdXY instead of a UUID.

How do I correctly pass the UUID of my root partition to grub?

----------

## Jaglover

 *Quote:*   

> unfortunaltely using PARTUUIDs instead of UUIDs im my /etc/fstab did not solve the issue I'm having.

 

Your fstab is residing in root filesystem, how could your kernel possibly read it if it cannot mount the root filesystem?

----------

## arcctgx

Argh, of course. So editing fstab will not help at all. Now I feel dumb.  :Sad: 

So, what should I do to avoid the problem I described? Any idea? (other that not plugging in USB mass storage devices at boot time?)  :Wink: 

And why does it happen with kernel 4.19.44 but not with 4.19.27?

----------

## Hu

Perhaps the older kernel does not understand that form of USB mass storage, or is configured in a way that it detects it much later, such as after modules load.

For the fix, you should make grub pass an appropriate root=PARTUUID= to your kernel.

----------

## tomtom69

I also had this kind of problem some time ago. It really looks like a race condition which causes the drive enumeration to change dependent on the kernel version if an USB drive is connected in addition to the (internal) SATA drives.

My remedy was to compile the USB storage driver as a module. This causes the USB drives to be always enumerated after rootfs mount which fixed the problem permanently. Might not be "elegant" but did the job.

tom

----------

## arcctgx

OK, so I think I got it right this time. The solution is to pass the correct PARTUUID to the kernel, AND to use PARTUUID in /etc/fstab. That way it doesn't matter if the hard drive is detected as /dev/sda or /dev/sdb or whatever else.

Passing PARTUUID to the kernel is the crucial thing. I did that by adding root=PARTUUID=xxxxxxxx-yyyy-zzzz-aaaa-bbbbbbbbbbbb to GRUB_CMDLINE_LINUX variable in my /etc/defaults/grub. But it seems to be something that grub should figure out by itself, and I have no idea why it's unable to do so. Furthermore, it generates the following grub.cfg content:

```
linux   /vmlinuz-4.19.44-gentoo root=/dev/sda4 ro root=PARTUUID=xxxxxxxx-yyyy-zzzz-aaaa-bbbbbbbbbbbb
```

The ugly thing is that root= is specified twice, but fortunately the latter root= option seems to take precedence over the former.

Thanks everyone, it was a good discussion which pointed me in the right direction.

----------

## NeddySeagoon

arcctgx,

To generalise the advice already provided, you need to avoid using /dev nodes to describe your filesystem to the kernel at all times.

/dev/sda and friends are allocated on a first come first named basis.

When USB storage devices are in the mix, the order can vary from boot to boot never mind kernel to kernel.

In /etc/fstab use UUID, PARTUUID or LABEL. It matters for getting started as the root entry is consulted to know what filesystem type root is for rootfsck.

As has already been said, it cannot be used to locate the root filesystem itself.

LABEL and UUID, need the userspace mount program, so to use then on the kernel command line, an initrd is required as they are filesystem properties.

The kernel is quite happy with PARTUUID its a property of a partition, so use PARTUUID on the kernel command line.

----------

## arcctgx

What you just wrote should be added to https://wiki.gentoo.org/wiki/Fstab.  :Smile: 

Edit:

I just had to update the configuration of hddtemp to use appropriate symlink from /dev/disk/by-id/ instead of /dev/sdXY...

----------

## NeddySeagoon

arcctgx,

Using /dev/disk/by-* symlinks is dangerous. It exposes you to a race condition.

The symlinks are created by udev at boot time.

If you try to use them before udev has created them, you get to keep the pieces.

Its probably ok for hddtemp, since if it misses a reading, you probably won't notice.

----------

## arcctgx

 *Quote:*   

> Using /dev/disk/by-* symlinks is dangerous. It exposes you to a race condition

 I figured there is a possibility of a race condition, though this solution seems good enough for hddtemp. If I'm not supposed to use /dev/sdXY or any of the symlinks created by udev in /dev/disk/, what other option do I have?

One more question if I may. There is an option CONFIG_PM_STD_PARTITION in the kernel configuration. I suppose I should pass resume=PARTUUID=... parameter to the kernel via grub, instead of configuring the resume partition in the kernel?

----------

## Hu

As I read kernel/power/hibernate.c, setting CONFIG_PM_STD_PARTITION assigns an initial value to resume_file.  Passing resume= assigns to resume_file at runtime.  Therefore, setting CONFIG_PM_STD_PARTITION="PARTUUID=" and passing resume=PARTUUID= have the same effect.  You may do whichever you prefer, and can use the runtime value to override one built in to the kernel.

----------

