# Grub2 NVraid dmraid

## onevision

So I've had this system setup and running for a couple of years now, and everything has been happy, with the setup being based on http://en.gentoo-wiki.com/wiki/RAID/NVRAID_with_dmraid#Building_the_Kernel.  It's using the nvidia raid built into the mother board, which is running a 4 disk raid0.  So foolishly I upgraded to grub2, without knowing much about it, and now everything has gone... well South.  I tried uninstalling grub2 and reverting back to the normal grub, but regardless of how much I wiped files and reinstalled, I end up with grub2.  While both are currently installed, it would seem that grub2 is being used and grub is sudo (mostly not) functional.

The problem started when I was upgrading my kernel from 3.5.7, which every release after that has been unsuccessful, with the latest being 3.7.10-r1.  I've tried using the config from the previous working kernel with no success.  I ran clean, mrproper, clean and then let the kernel figure things out, which also failed.  I then modified that adding some raid and SATA modules, but all end up with an error that I'm just not understanding.  When I boot, I get the message: /dev/mapper/nvidia_jeaacgah3 is not a valid root device.  Could not find the root block device in .

To me this says the kernel loaded - thus where the error message came from, but it's just not seeing the raid device for some reason.  When I go into the shell and list the /dev directory, sure enough there is no /dev/mapper directory, which I also noticed that I have no /dev/ram0 either.  I have to be missing something obviously that I'm just not seeing, but after days of trying everything under the sun, I'm still coming up empty.  Luckily my old kernel 3.5.7 will boot, but now with other package upgrades I've lost my display for X.  Anyone willing to chuck a dog a bone with some ideas?

Here's some basic info:

lspci

00:10.0 RAID bus controller: NVIDIA Corporation CK804 Serial ATA Controller (rev f3)

00:11.0 RAID bus controller: NVIDIA Corporation CK804 Serial ATA Controller (rev f3)

01:00.0 RAID bus controller: Silicon Image, Inc. SiI 3132 Serial ATA Raid II Controller (rev 01)

00:02.0 PCI bridge: NVIDIA Corporation C51 PCI Express Bridge (rev a1)

00:12.0 PCI bridge: NVIDIA Corporation CK804 PCI Bridge (rev a2)

I know it's odd that there are 2 raid controllers, but this mother board (ASUS) has two raid controllers built into it.  The Nvidia raid can do just about any type of raid between the 4 SATA ports and between the 4 IDE, and can also stack raid configs, although I'm just running a raid0 from the 4 SATA ports.  The Silicon Image raid can be used to mirror the raid array to an external SATA port, which can hotplug a drive in for backup, but I'm not using that raid

fdisk -l

Disk /dev/sda: 320.1 GB, 320072933376 bytes, 625142448 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x6dcac69f

   Device Boot      Start         End      Blocks   Id  System

/dev/sda1              63      273104      136521   83  Linux

/dev/sda2          273105    12868064     6297480   82  Linux swap / Solaris

/dev/sda3        12868065  2500565444  1243848690   83  Linux

Disk /dev/sdb: 320.1 GB, 320072933376 bytes, 625142448 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/sdc: 320.1 GB, 320072933376 bytes, 625142448 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/sdd: 320.1 GB, 320072933376 bytes, 625142448 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/sde: 61.5 GB, 61492838400 bytes, 120103200 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x69205244

   Device Boot      Start         End      Blocks   Id  System

/dev/sde1   *          63   120085874    60042906    7  HPFS/NTFS/exFAT

Disk /dev/mapper/nvidia_jeaacgah: 1280.3 GB, 1280291635200 bytes, 2500569600 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 65536 bytes / 262144 bytes

Disk identifier: 0x6dcac69f

                      Device Boot      Start         End      Blocks   Id  System

/dev/mapper/nvidia_jeaacgah1              63      273104      136521   83  Linux

Partition 1 does not start on physical sector boundary.

/dev/mapper/nvidia_jeaacgah2          273105    12868064     6297480   82  Linux swap / Solaris

Partition 2 does not start on physical sector boundary.

/dev/mapper/nvidia_jeaacgah3        12868065  2500565444  1243848690   83  Linux

Partition 3 does not start on physical sector boundary.

Disk /dev/mapper/nvidia_jeaacgah1: 139 MB, 139797504 bytes, 273042 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 65536 bytes / 262144 bytes

Alignment offset: 33280 bytes

Disk /dev/mapper/nvidia_jeaacgah2: 6448 MB, 6448619520 bytes, 12594960 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 65536 bytes / 262144 bytes

Alignment offset: 24064 bytes

Disk /dev/mapper/nvidia_jeaacgah3: 1273.7 GB, 1273701058560 bytes, 2487697380 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 65536 bytes / 262144 bytes

Alignment offset: 15872 bytes

Disk /dev/sdf: 3000.6 GB, 3000558944256 bytes, 732558336 sectors

Units = sectors of 1 * 4096 = 4096 bytes

Sector size (logical/physical): 4096 bytes / 4096 bytes

I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Disk identifier: 0x000246c6

   Device Boot      Start         End      Blocks   Id  System

/dev/sdf1             256   732558335  2930232320    7  HPFS/NTFS/exFAT

Disk /dev/sdg: 3000.6 GB, 3000592965632 bytes, 732566642 sectors

Units = sectors of 1 * 4096 = 4096 bytes

Sector size (logical/physical): 4096 bytes / 4096 bytes

I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Disk identifier: 0x56882b04

   Device Boot      Start         End      Blocks   Id  System

/dev/sdg1             256   732566641  2930265544   83  Linux

So sda-sdd are the drives making up the raid0, which is then seen as the /dev/mapper/nvidia_jeaacgah.  The sde is just a small drive I try to use to run Windows XP, since it's the only thing that will run Real Flight, only I haven't had much success with that (key expired and I won't pay for Windows).  The sdf and sde are both external USB storage drives.  The raid0 is partitioned with /boot, swap, and / with the root using ext4 and boot ext2

Kernal config:

CONFIG_SATA_AHCI=m

CONFIG_SATA_INIC162X=m

CONFIG_SATA_SIL24=m

CONFIG_ATA_SFF=y

CONFIG_SATA_MV=m

CONFIG_SATA_NV=m

CONFIG_SATA_SIL=m

CONFIG_SATA_SIS=m

CONFIG_PATA_CMD64X=m

CONFIG_PATA_ACPI=m

CONFIG_BLK_DEV_MD=y

CONFIG_MD_AUTODETECT=y

CONFIG_MD_LINEAR=m

CONFIG_MD_RAID0=m

CONFIG_MD_RAID1=m

CONFIG_MD_RAID10=m

CONFIG_MD_RAID456=m

CONFIG_MD_MULTIPATH=m

CONFIG_MD_FAULTY=m

CONFIG_BLK_DEV_DM=m

CONFIG_DM_CRYPT=m

CONFIG_DM_SNAPSHOT=m

CONFIG_DM_MIRROR=m

CONFIG_DM_ZERO=m

CONFIG_DM_MULTIPATH=m

I believe that's all the relevant setting, although would be happy to provide more.  I used the following to compile the initramfs using genkernel:

genkernel --dmraid --install --kernel-config=/usr/src/linux/.config initramfs

My grub/grub.conf config basically looks like this:

title Gentoo 3.7.10-r1

root (hd0,0)

kernel /boot/vmlinuz-3.7.10-gentoo-r1 root=/dev/ram0 init=linuxrc real_root=/dev/mapper/nvidia_jeaacgah3 dodmraid video=vesafb:ywrap,pmipal,1600x1200-75@93 rootfstype=ext4

initrd /boot/initramfs-genkernel-x86_64-3.7.10-gentoo-r1

Although it doesn't seem to be operating from that so much as from the grub2/grub.cfg config:

submenu 'Advanced options for Gentoo GNU/Linux' $menuentry_id_option 'gnulinux-advanced-/dev/ram0' {

    menuentry 'Gentoo GNU/Linux, with Linux 3.7.10-gentoo-r1' --class gentoo --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-3.7.10-gentoo-r1-advanced-/dev/ram0' {

        load_video

        insmod gzio

        echo    'Loading Linux 3.7.10-gentoo-r1 ...'

        linux   /vmlinuz-3.7.10-gentoo-r1 root=/dev/ram0 ro  real_root=/dev/mapper/nvidia_jeaacgah3 dodmraid rootfstype=ext4

        echo    'Loading initial ramdisk ...'

        initrd  /initramfs-genkernel-x86_64-3.7.10-gentoo-r1

    }

One thing worth noting, which I don't know if this is correct or even matters, but my grub/device.map currently looks like:

(fd0)   /dev/fd0

(md0)   /dev/mapper/nvidia_jeaacgah

(hd0)   /dev/sda

(hd1)   /dev/sdb

(hd2)   /dev/sdc

(hd3)   /dev/sdd

(hd4)   /dev/sde

(hd5)   /dev/sdf

(hd6)   /dev/sdh

This is different than I had it before, although it doesn't seem to make a difference how it's set.  When it was working, I didn't have a md0 entry.  Can anyone point me to where all my devices went to on boot?  Device heaven maybe?  I can't seem to get them to be present in /dev.  Any help at all would be greatly appreciated.  Thanks.

-Davey

----------

## NeddySeagoon

onevision,

Save your kernel .config somewhere safe and try 

```
genkernel --dmraid --install all initramfs 
```

See if genkernel knows better than you do.

This will build you a genkernel configured kernel with dmraid support.

----------

## NeddySeagoon

Moved from Unsupported Software to Kernel & Hardware.

dmraid is supported - may get more attention here.

----------

## onevision

Yeah, that was my thoughts as well, and my hell Mary last charge attempt.  The results were... unexpected.  So after having genkernel just handle the compiling of everything, it still failed the same, but at least before it failed it loaded modules.  Apparently among those modules was the raid device, which I was able to check from the shell and then manually specify the device to boot into (/dev/mapper/nvidia_jeaacgah3).  Funny that it would boot from it when manually entered, yet it's the exact same thing configured in grub.

I'm thinking that maybe grub doesn't have the drives and partitions setup correctly.  Grub2 is done a bit differently, and doesn't seem to play very well with raid arrays.  The boot looked okay though, and other than I have zero modules loaded, X still won't start, and SSH service fails regardless of what I do saying "no such device".  During this last batch of updates however, I did upgrade to the new udev, so it's likely related to that, although my NIC interfaces are both up and running.

So I'm sort of fixed.  Hacked to mostly operational anyway.  Thanks for the advice and help.

----------

