# [solved] Kernel cannot mount root device on Intel 750 drive

## Jake1055

I successfully installed Gentoo to my laptop, and it is fantastic.  I really like it, and want to replace arch with Gentoo on my desktop.

Here are the specs:

Intel i7-4790k

16GB DDR3 RAM

Intel 750 Series 400GB PCIe ssd

Asus Z97-A motherboard

GTX 980 EVGA

The issue I'm having with the PCIe ssd and the boot process.

Here's some other information about my installation.

```
[root] ~ $ fdisk -l /dev/nvme0n1

Disk /dev/nvme0n1: 372.6 GiB, 400088457216 bytes, 781422768 sectors

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512  bytes / 512 bytes

Disklabel type: gpt

Disk identifier: A75A2F65-EF96-42E8-ABE7-C3778507D9D1

Device             Start       End         Sectors      Size     Type

/dev/nvme0n1p1     2048        1050623     1048576      512M     EFI System

/dev/nvme0n1p2     1050624     7814422591  780371968    372.1G   Linux Filesystem

```

```
[root] ~ $ lspci -k

00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06)

        Subsystem: ASUSTeK Computer Inc. 4th Gen Core Processor DRAM Controller

        Kernel driver in use: hws_uncore

lspci: Unable to load libkmod resources: error -12

00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06)

        Kernel driver in use: pcieport

00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller (rev 06)

        Kernel driver in use: pcieport

00:14.0 USB controller: Intel Corporation 9 Series Chipset Family USB xHCI Controller

        Subsystem: ASUSTeK Computer Inc. 9 Series Chipset Family USB xHCI Controller

        Kernel driver in use: xhci_hcd

00:16.0 Communication controller: Intel Corporation 9 Series Chipset Family ME Interface #1

        Subsystem: ASUSTeK Computer Inc. 9 Series Chipset Family ME Interface

        Kernel driver in use: mei_me

00:19.0 Ethernet controller: Intel Corporation Ethernet Connection (2) I218-V

        Subsystem: ASUSTeK Computer Inc. Ethernet Connection (2) I218-V

        Kernel driver in use: e1000e

00:1a.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2

        Subsystem: ASUSTeK Computer Inc. 9 Series Chipset Family USB EHCI Controller

        Kernel driver in use: ehci-pci

00:1b.0 Audio device: Intel Corporation 9 Series Chipset Family HD Audio Controller

        Subsystem: ASUSTeK Computer Inc. 9 Series Chipset Family HD Audio Controller

        Kernel driver in use: snd_hda_intel

00:1c.0 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 1 (rev d0)

        Kernel driver in use: pcieport

00:1c.3 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d0)

00:1d.0 USB controller: Intel Corporation 9 Series Chipset Camily USB EHCI Controller #1

        Subsystem: ASUSTeK Computer Inc. 9 Series Chipset Family USB EHCI Controller

        Kernel driver in use: ehci-pci

00:1f.0 ISA bridge: Intel Corporation 9 Series Chipset Family Z97 LPC Controller

        Subsystem: ASUSTeK Computer Inc. 9 Series Chipset Family Z97 LPC Controller

        Kernel driver in use: lpc_ich

00:1f.3 SMBus: Intel Corporation 9 Series Chipset Family SMBus Controller

        Subsystem: ASUSTeK Computer Inc. 9 Series Chipset Family SMBus Controller

01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)

        Subsystem: Intel Corporation PCIe Data Center SSD

        Kernel driver in use: nvme

02:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 980] (rev a1)

        Subsystem: eVga.com. Corp. GM204 [GeForce GTX 980]

        Kernel driver in use: nouveau

02:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller (rev a1)

        Subsystem: eVga.com. Corp. GM204 High Definition Audio Controller

        Kernel driver in use: snd_hda_intel

03:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 04)

```

```
[root] ~ $ cat /etc/fstab

# /etc/fstab: static file system information.

/dev/nvme0n1p1   /boot   vfat   defaults   0 2

/dev/nvme0n1p2   /       ext4   defaults   0 1

```

Here are the steps I take to replicate the problem.

1. Remove all contents of ESP(mounted at boot) so I can start fresh.

```
[root] ~ $ rm -r /boot/*

```

2. Remove kernel sources(both symlink and other folder)

```
[root] ~ $ rm -r /usr/src/*

```

3. Unmerge sys-kernel/gentoo-sources

```
[root] ~ $ emerge -c sys-kernel/gentoo-sources

```

4. Emerge sys-kernel/gentoo-sources

```
[root] ~ $ emerge --ask sys-kernel/gentoo-sources

```

5. Generate kernel using genkernel.

```
[root] ~ $ genkernel all

```

6. Copy kernel and initramfs into a seperate file that we can point to from bootloader configuration

```
[root] ~ $ cp /boot/kernel* /boot/kernel.img

[root] ~ $ /boot/initramfs* /boot/initramfs.img

```

7. Install gummiboot to /boot

```
[root] ~ $ gummiboot --path /boot install

Created /boot/EFI.

Created /boot/EFI/gummiboot.

Created /boot/EFI/Boot.

Created /boot/loader.

Created /boot/loader/entries.

Copied /usr/lib/gummiboot/gummibootx64.efi to /boot/EFI/gummiboot/gummibootx64.efi

Copied /usr/lib/gummiboot/gummibootx64.efi to /boot/EFI/Boot/BOOTX64.efi

Created EFI boot entry "Linux Boot Manager".

```

8. Modify /boot/loader/loader.conf

```
[root] ~ $ vim /boot/loader/loader.conf

[root] ~ $ cat /boot/loader/loader.conf

timeout 3

default gentoo

```

9. Create gentoo.conf in /boot/loader/entries/

```
[root] ~ $ vim /boot/loader/entries/gentoo.conf

[root] ~ $ cat /boot/loader/entries/gentoo.conf

title Gentoo Linux

linux /kernel.img

initrd /initramfs.img

options root=/dev/nvme0n1p2 rw

```

10. Exit chroot, reboot, and test.

The gentoo entry boots the initramfs, but fails on the following line.

```
>> Determining root device ..

!! Block device /dev/nvme0n1p2 is not a valid root device...

!! Could not find the root block device in .

!! Please specify another value or:

!! - press Enter for the same

!! - type "shell" for a shell

!! - type "q" to skip...

root block device() :: _
```

I've tried this process by configuring my kernel using make menuconfig and all of that, but I couldn't get it to boot, so I decided to give genkernel a try.

If you need any additional information, please feel free to ask.  I have run into a real wall here, and I can't figure out how to configure my NVME PCIe ssd for Gentoo.

I appreciate any help you may offer.Last edited by Jake1055 on Wed Jun 15, 2016 5:10 pm; edited 2 times in total

----------

## Jake1055

I managed to get the dmesg output into a file and transfer it via a USB drive to my laptop.

Here is the output of dmesg.

If you need any additional information, feel free to ask.

EDIT:  Made the dmesg output a link to pastebin rather than pasting the raw text in between code tags, per request of khayyamLast edited by Jake1055 on Fri Jun 10, 2016 2:10 am; edited 2 times in total

----------

## roarinelk

Stupid question, but just to make sure:  Do you have "CONFIG_BLK_DEV_NVME=y" set in your kernel's .config?

From the looks of the dmesg output, the SSD is found on the PCI bus, but no driver attaches to it.

----------

## khayyam

Jake1055 ...

the issue isn't efi related, you are booting (you might want to change the subject line to reflect this). That means the issue is either the kernel, or (less likely) the initramfs. The lspci output above shows nvme but this is obviously from the install media, please check that CONFIG_BLK_DEV_NVME is enabled. Also, you should be passing 'rootfstype=ext4' as a boot parameter, though I can say if the absence of this would cause the above error (though I'm completely unfamilar with genkernel generated initramfs).

So, if we need to see anything it is the kernel .config (and please, in future, use a pastebin service for large files/output).

best ... khay

----------

## Jake1055

roarinelk:

I let genkernel do all the configuration itself.  Snooping around the kernel configuration by doing the following:

```
[root] linux $ make menuconfig

```

I can see that NVM Express block device is configured as a module.  I assume that genkernel's initramfs loads the modules at boot.  If I recompile the kernel with it built into the kernel not as a module, the same thing occurs(the typing prompt with the 3 options).

----------

## Jake1055

khayyam:

Thanks for the tip.  I changed the title.

Like I said to roarinelk, genkernel built it in as a module(figured out by manually loading the ncurses tool to see the status).  I recompiled as a built in, non-module feature, but the same halt happens on the boot process.

```
>> Determining root device .. 

!! Block device /dev/nvme0n1p2 is not a valid root device... 

!! Could not find the root block device in . 

!! Please specify another value or: 

!! - press Enter for the same 

!! - type "shell" for a shell 

!! - type "q" to skip... 

root block device() :: _

```

I can add "rootfstype=ext4" to the kernel parameters with the following.

```
[root] ~ $ vim /boot/loader/entries/gentoo.conf

[root] ~ $ cat /boot/loader/entries/gentoo.conf

title Gentoo Linux 

linux /kernel.img 

initrd /initramfs.img 

options root=/dev/nvme0n1p2 rw rootfstype=ext4

```

If I do that, I get dropped into the same prompt.  I again went into a shell and harvested a dmesg log.  Then, I used vim to eliminate the first 15 characters of every line on both of my logs(because I expect the timestamps on each to be different).  Then, I ran the following command.

```
[root] ~ $ diff dmesg1.log dmesg2.log

```

Here is the output of that command.

And for good measure, here is my kernel .config

Thank you both for your help.  I was starting to lose hope.  :Smile: 

----------

## khayyam

Jake1055 ...

as I said I'm not at all familiar with genkernel generated initramfs, but I wouldn't have NVME or SATA_ AHCI as a modules (my rule being that if you need it to boot you should select =y). I suspect this is the reason you're not seeing /dev/nvme0n1p2. Please try with 'genkernel --menuconfig all' and set these to =y.

best ... khay

----------

## NeddySeagoon

Jake1055,

When you get the the initrd shell here

```
 >> Determining root device ..

!! Block device /dev/nvme0n1p2 is not a valid root device...

!! Could not find the root block device in .

!! Please specify another value or:

!! - press Enter for the same

!! - type "shell" for a shell 
```

What block devices are in /dev ?

rootfstype does not matter.  For ext4,  the kernel will try ext3 then ext2 and lastly ext4.  The messages about ext3 and ext2 nor working cam be ignored.

----------

## Jake1055

khayyam:

I tried what you suggested, running

```
[root] ~ $ genkernel --menuconfig all
```

and enabling NVM Express Block Device support and AHCI Sata Support(NOT as module, but as built into the kernel).

Then I copied the generated initramfs and kernel into /boot/initramfs.img and /boot/kernel.img so that it would be loaded by gummiboot.

Unfortunately, booting still results in the same prompt.

----------

## Jake1055

NeddySeagoon:

 After I landed at the prompt and got into a shell, I ran the following command:

```
ls /dev/
```

Here is the output of that command.

----------

## Jake1055

Okay, so I never really wanted to go for a genkernel-generated kernel. I like the idea of modifying the kernel to suit my hardware(plus I like a fast boot  :Very Happy: ).  So, using the methodology I outlined in my original post, I removed the existing kernel sources and unmerged them, then re-emerged sys-kernel/gentoo-sources.  Then, I used

```
[root] linux $ make menuconfig
```

to configure the kernel myself.

Here is my *NEW* kernel config.

Now, when I boot a different halt screen happens, I believe because I'm not using an initramfs.  I'm just telling gummiboot to directly load the linux kernel as an efi executable.

Here is a photo on imgur of the screen I'm talking about.

If you don't wish to look at the photo, basically these are the lines that seem important to me:

```
Kernel Offset: disabled

---[ end Kernel panic - not syncing: VFS: unable to mount root fs on unknown-block(0,0)
```

I have encountered this kernel panic before, primarily on my laptop which uses a typical SATA ssd.  I did research at that time and came across this Gentoo wiki page on the subject.

This page tells me that, because the numbers within unknown-block are 0 and 0, then the kernel is unable to "detect the controller for the hard disk (a likely candidate if the message says unknown-block(0,0))".

Now, as can be seen in my kernel config, NVME is built in(configured as =y) the kernel.  I don't know what I need to do in order to get my kernel to be able to recognize this device.

Also, though I feel like its not that important, I have made a few changes to my setup.

1. I changed /etc/fstab to use UUID labels rather than

2. I changed my kernel parameters(set in /boot/loader/entries/gentoo.conf) from using typical device identifiers(/dev/nvme0n1p2) to using UUIDs.

I don't imagine that would be significant, but I figured I'd share in case it was.

----------

## NeddySeagoon

Jake1055,

```
nvme0

nvme0n1

nvme0n1p1

nvme0n1p2
```

  That's your HDD.  

```
!! Block device /dev/nvme0n1p2 is not a valid root device...

!! Could not find the root block device in . 
```

so it looks like that message is in error.  /dev/nvme0n1p2 is clearly present.

```
!! Could not find the root block device in .
```

Is also informative, its just harder to interpret, which is why I missed it the first time.

Between the  !! Could not find the root block device in  and the full stop, the kernel lists all the block devices it can see.  In this case, none at all, not even an optical drive.

That's odd.

If you still have your old kernel and initrd, the next step would be to boot normally, get into the initrd shell, cat the initrd init script and perform the steps starting with mounting root by hand by the copying the script commands one at a time.

--- edit 1 ---

Your lspi shows the complete absence of SATA hardware.  In that really correct?

I would have expected some SATA ports in AHCI mode to be able to connect optical drives.

--- edit 2 ---

The use of filesystem UUIDs in /etc/fstab is fine. It requires the use of the userspace mount command, which is available as soon as root is mounted.

Without an initrd to provide the userspace mount command before root is mounted, you cannot use root=UUID=.. in grub.

However, the kernel understands PARTUUID, so root=PARTUUID= can work.  

Warning:  PARTUUID and UUID are different properties of different block device objects, so the values differ too.

----------

## Jake1055

NeddySeagoon:

I generated an initramfs using genkernel and told gummiboot to use it.

```
[root] /boot $ genkernel initramfs

...

...

...

[root] /boot $ cp initramfs* initramfs.img

[root] /boot $ vim /boot/loader/entries/gentoo.conf

[root] /boot $ cat /boot/loader/entries/gentoo.conf

title     Gentoo Linux

linux /kernel.img

initrd /initramfs.img

options root=PARTUUID=LONGUUIDTHATISCORRECT rw
```

Rebooting, it drops me into the same prompt.  Getting into a shell and listing files in /dev, I see /dev/nvme0n1 still.

You're correct.  I do not have my optical drive plugged into my motherboard.  I have two other SATA ssd's that are not plugged in as well(one drive hosts a windows 10 installation, the other a hackintosh installation).  I unplugged all of these, because I was having issues with my motherboard not booting past the bios splash screen.  Unplugging those SATA devices did the trick.  I also figured it would be good just to make sure I didn't overwrite, accidentily, my other OS partitions.

Wait...  PARTUUID and UUID expect different values?  So the UUID listed from lsblk -f won't work?  What do I need to pass to it, then?

----------

## NeddySeagoon

Jake1055,

You ask blkid 

```
$ /sbin/blkid 

/dev/sda1: UUID="d678d02e-28ab-84e0-c44c-77eb7ee19756" TYPE="linux_raid_member" PARTLABEL="primary" PARTUUID="8626b218-ea7a-4ff7-83f5-03fc2d6ab558"
```

If you do the init steps by hand and it works, you have found a race condition.

You add a delay to fix that.

Maybe adding rootwait to your kernel line would work, as long as you don't use an initrd.

----------

## denix

 *NeddySeagoon wrote:*   

> Jake1055,
> 
> Maybe adding rootwait to your kernel line would work

 

Thanks so much! I've been struggling to boot off of Samsung 950Pro M.2 device for past couple days and all it required was "rootwait"!

----------

## Jake1055

I modified /etc/fstab and /boot/loader/entries/gentoo.conf to use PARTUUID's with the following.  I also added rootwait to my kernel parameters.

```
[root] ~ $ blkid /dev/nvme0n1p2 >> /etc/fstab#my root device

[root] ~ $ vim /etc/fstab

[root] ~ $ cat /etc/fstab

# /etc/fstab: static file system information.

PARTUUID=CORRECTPARTUUID     /boot     vfat     defaults     0 2

PARTUUID=CORRECTPARTUUID     /         ext4     defaults     0 1
```

```
[root] ~ $ blkid /dev/nvme0n1p2 >> /boot/loader/entries/gentoo.conf

[root] ~ $ vim /boot/loader/entries/gentoo.conf

[root] ~ $ cat /boot/loader/entries/gentoo.conf

title Gentoo Linux

linux /kernel.img

options root=PARTUUID=CORRECTPARTUUID rw rootfstype=ext4 rootwait

```

AND IT WORKS... KIND OF...

I turn on my computer, and it boots gummiboot.   The boot seems alright, but it halts in a few places.

The longest halt occurs on this line:

Waiting for uevents to be processed..

It spends a solid 5 seconds on this line.

Later in the boot process, sysklogd tells me a number of directories/files are missing.

Here is a harvested dmesg.

My best guess is that the halt on processing uevents has to do with my Corsair K70 RGB keyboard, which I've always had problems with on Linux.

Following the boot process, it displays "This is desktop."  Then, after another 5 ish seconds, it finishes the sentence with "unkown_hostname[...]".

----------

## NeddySeagoon

Jake1055,

I can explain why it works.  Previously, you had shown that /dev/nvme0n1 was in /dev in the initrd shell after the kernel claimed root wasn't there to be mounted.

Both are true.  From dmesg

```
[    2.154874] clocksource: Switched to clocksource tsc

[    6.352361]  nvme0n1: p1 p2

[    6.393501] EXT4-fs (nvme0n1p2): mounted filesystem with ordered data mode. Opts: (null)
```

That 4 second pause it rootwait doing its thing.  It forces the kernel to wait for nvme0n1: p1 p2 before it tries to mount root.  Without that, the kernel tries to mount root before its available.

A better known example of this is root on USB.  The USB subsystem is not normally started until after root is mounted, that's a problem with root on USB.

Waiting for uevents to be processed is expected and will be several seconds.  Its the kernel telling udev about all the hardware its found at power on, /dev entries being created and udev doing its processing.  Its not specifically your keyboard.

You need to fix 

```
[    6.875736] FAT-fs (nvme0n1p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
```

I can't help with the other things. They will be minor details of your setup.

Some will be systemd specific and I'm not a systemd user.

----------

## Jake1055

Thank you, everyone, for your help.  This community is really welcoming and knowledgeable.  I'll be able to troubleshoot the problems from here on out, now that it actually boots.  :Smile: 

The problem outlined is solved, so I'll change the thread title accordingly.

----------

## Jake1055

Just in service to anyone who might stumble upon this thread and have the same problem, I've discovered something that fixes many of the problems outlined in my original post.

Apparently, the firmware many Intel 750 series NVME drives shipped with is buggy.  It doesn't get recognized by the BIOS as quickly as SATA devices do, or even USB devices.  That's why my kernel couldn't mount the root filesystem.

Here is an intel page describing how to solve this by updating the drive firmware.

Basically, you just go download an .iso file and flash it to a USB flash drive or DVD and boot from it on your system.  It then loads a GUI and you run through a wizard to update the firmware.  Remarkably simpler.  It shaved 6 or 7 seconds off of my boot time.

----------

## NeddySeagoon

Jake1055,

That will be the reduced rootwait time,

Do you need rootwait at all?

----------

