# initramfs fails lvm2 root device on md-based raid disk

## g.tomassoni

Dears,

I used genkernel --lvm2 --gensplash initrd to build the initramfs-genkernel-x86-2.6.12-gentoo-r8 file for my kernel-genkernel-x86-2.6.12-gentoo-r8.

My root fs is a logical volume of a vg whose pv is a md-based raid1 of two disk partitions (/dev/sda6 and /dev/sdb6).

I also have the /boot fs on a md-based raid1 of two disk partitions (/dev/sda1 and /dev/sdb1).

The above-mentioned partitions are marked as Linux raid autodetect and autodetection recognized /dev/md0 as composed by /dev/sd[ab]1 and /dev/md1 as composed by /dev/sd[ab]6.

Everything works fine as long as I use a classical initrd file to start the os. However, using a initramfs-style file doesn't work.

The reason I found is that when bootup is controlled by a initramfs file, the kernel init code doesn't invoke the prepare_namespace() function (see line#679 of init/main.c) which is in charge of eventually invoking the autostart_arrays() function in drivers/md/md.c.

This leads to the fact that lvm is unable to discover any md-based vg through vgscan, thereby not activating it. In turn, this leads to the fact that the root device specified in the kernel commandline is not found and the kernel itself asks me for a suitable device.

Assuming correct the behaviour of the linux kernel to not automatically start any md-based raid, the problem seems rooted in the /etc/initrd.scripts' setup_md_device() function, which seems designed to automatically start a raid iff the associated device (/dev/md*) is specified as the root device in the linux commandline. Since my root is a lvm lv, this is not the case and setup_md_device() skips md discovery as a whole.

I also found that issuing the following commands from a shell obtained in the same context of initramfs:

```
mknod /dev/md0 b 9 0

mknod /dev/md1 b 9 1

mdstart /dev/md0

vgscan

vgchange -a y System
```

allows me to successfully boot my system.

Note that /dev/md0 is not the pv on which my root resides: it is the /boot fs. The mdstart seems to start automatically every autostart md-based raid, not just the one specified. Also note that vgscan needs a /dev/md1 block device in order to detect the System vg.

Given all this, I would suggest to modify the setup_md_device() in the /etc/initrd.scripts file in order to:

1) create a set of /dev/md? devices (maybe /dev/md0-9 ?)

2) run mdstart /dev/md0

irregardless of the runstring setting or at least in case we merged our initramfs with a initramfs-lvm2.*.cpio file (i.e.: we did a 'genkernel --lvm2'). This can be easily checked by testing for the /bin/lvm presence.

Regards,

Giampaolo

----------

## johndo

The symptoms you describe match what I am going through.

The only diff

Is there anyway to force genernel to use initrd then?

edit: the only differences are that Im doing lvm2 on raid5Last edited by johndo on Mon Aug 15, 2005 9:17 pm; edited 1 time in total

----------

## g.tomassoni

Hi johndo,

I placed a bug for it, and someone at Gentoo kindly replied to test boot setting which didn't worked before some tuning.

You have to identify the /dev/md? devices which is the pv containing you root's lv, then you have to add the setting:

```
lvmraid=/dev/mdX real_root=/dev/VOLGROUP/LOGICALVOL
```

to your linux boot runstring.

/dev/mdX must be changed into the md device acting as your vg (ie: mine is /dev/md1), while /dev/VOLGROUP/LOGICALVOL is, of course, the path to your root device.

Please note that it seems you can't use root= instead of real_root=. If you actually have a root= setting, just change its name to real_root=

----------

## johndo

I added lvmraid=/dev/md0 to myand my system's functionalty was restored.

my real_root device is actually /dev/hda3. I only use my md0 device for /home, /opt, /tmp, /usr,  and /var.

This is so that I will have a semi-usable system in case the raid ever fails, like it did here (:

on a side note: one of the messages I got was that real_root was detected on a lvmraid array.

This is probably just a cosmetic error and doesn't actually affect anything.

----------

## g.tomassoni

johndo,

this is weird: I guessed that the lvmraid= and real_root= setting were needed only when the root was on a lvm-over-raid device...

Given your configuration, you should be capable to start your system by simply specifing root=/dev/hda3 and omitting any lvmraid= setting...

----------

## johndo

A lot of the bootup programs depend on files that are on the raid array, so I do need it for bootup

I may have been able to start the system and mount the array after it booted by logging in  as root and running the mdstart commands, but things like apache and Xorg won't work until I do.

You also need real_root when root=/dev/ram0, which genkernel specifies by default, iirc.

Very recently (same day I got lvm on raid working again) I had a problem with the volume group on the array: it doesn't exist anymore.  /dev/md0 exists, but I can't get vgscan to find my ''raid5" volume group on it.  My old kernel build can not find it either.  I don't know if this is realted or not, but I don't know how I lost it or how to get it back.

----------

## taskara

I'm trying to get my Gentoo system built on an lvm group on top of raid5, but it won't go.

I am using a genkernel --lvm2 created initrd but when it searches for raid volumes it find nothing! (I did also have --dmraid but I think that's for sudo hardware onboard raid controllers)

However, if I boot to the kernel directly without an initrd, it DOES find the raid devices (but then fails, naturally, because it can't find lvm volumes).

I'm wondering whether this issue is related?

Essentially I have:

sda5, b5, c5, d5 all on a linux software raid 5 array (/dev/md5)

I then have an lvm group called "system" under that, and then volumes "slash" and "home" under that.

The machine is at the office, so I may head down there and try adding "lvmraid=/dev/md5" to grub's kernel line.

Any thoughts?

Thanks,

-c

----------

## taskara

lvmraid option got me booting - but it is strange! The initrd still says that it can't find any software raid devices, but now it CAN find the lvm volumes and it boots all the way. Once I log in I see that I have all my raid devices started, and all the right things are mounted.

There is an issue however, that after it finds the lvm group and start booting, I get "rm - can't remove /sys/blah*" spewing across the screen for about 2 minutes, then it goes and boots just fine.

Has anyone heard of these issues?

I assume lvmraid option is new as I had this working without it previously. Perhaps Gentoo devs have changed the way all this works?

Cheers,

-c

----------

## Wilke

Hope it's ok if I kick an old topic to the top, but I seem to have the same or at least a very similar problem.

My setup is the same as mentioned by the topicstarter. I did run genkernel with all the right flags as described in this topic (--lvm2, set the real_root and lvmraid kernel parameters etc.). However, the kernel cannot mount my LVM-on-RAID1 root partition while booting from the ramfs.

It works fine when I boot from a different root partition (non-LVM/RAID) and then the LVM-on-RAID partitions will also be mounted just fine by the normal Gentoo init-scripts. But when I try to mount the LVM-on-RAID root during boot, the problem is that the RAID is not detected at all, and hence the LVM volumes cannot be found. Even the manual solution described in the topicstart does not work for me: when I run 'mdstart /dev/md0' and then 'cat /proc/mdstat' it will just say there are no RAID devices at all.

Maybe it has to do with genkernel - should I instruct genkernel to run menuconfig and manually change all the RAID and LVM-options to builtin rather than modules? I don't really see how it would help because the initramfs loads the modules just fine. But still it cannot find the RAID. It seems as if it needs the /etc/mdadm.conf in order to detect the RAID volumes or something....(mdadm and /etc/mdadm.conf are not available in the initramfs - as they are both on the root partition itself).

P.S. Yes, the partitions are marked 'fd' - Linux raid autodetect and all that  :Smile: 

----------

## overkll

Wilke,

Came across this post while researching LVM.  Read the whole thing and I think you're the only one who thought of modules vs built in.

If you build the raid driver into the kernel, all md devices will be found and configured automagically, thanks to the raid info stored on the disk (superblock).  /etc/raidtab or /etc/mdadm.conf are not needed.  Don't believe it?  Just pop a fedora core install cd into your system and boot up.  It doesn't have your mdadm.conf file yet it will create the devices correctly.  To bad gentoo cd's don't do the same.   :Embarassed: 

I don't have any real experience yet with LVM2, but I don't see why you shouldn't build that driver into the kernel as well.  Personally, I build everything into the kernel (using menuconfig) unless its only available as an external kernel module ie nvidia, ivtv, etc.  I don't need an initrd or initramfs.  I just boot the kernel directly with grub.

Does lvm *NEED* and initramfs?

Isn't all the necessary LVM info to start up LVM devices already written on the disk?  If so I would think that the built-in kernel driver will start it up like a champ.

----------

## Wilke

Just to confirm, compling LVM + RAID support into the kernel solved the problem - even if you still use it in combination with initramfs (to automatically load any other modules).

I still use genkernel, just make sure you add '--menuconfig' as a parameter and build everything related to LVM/RAID into the kernel.

Command I used to create the kernel: genkernel --lvm --menuconfig all.

So now I wonder, is there a technical reason why the RAID system will not autodetect RAID-partitions when it is loaded as a module from initramfs? Or is this just a "implementation detail" or "bug" in the current kernel? Could this be fixed in the kernel or is there a very good reason why it's like this?

----------

## g.tomassoni

 *overkll wrote:*   

> Does lvm *NEED* and initramfs? 
> 
> Isn't all the necessary LVM info to start up LVM devices already written on the disk? If so I would think that the built-in kernel driver will start it up like a champ.

 

Because you need to issue a 

```
vgscan
```

 followed by a 

```
vgchange -a y VGNAME
```

.

Ok, well. I don't really know why, but I guess that we (or at least me) use very simple LVM configurations, in which the vg is composed by a single phisical volume (doesn't matter if a raid one). Our purpose is often to divide the huge available space in smaller partitions in order to increase performances, still having the chance to enhance the size of a partition when needed. Also, we may use it for "snapshotting" partitions for backup purposes.

LVM may, however, sustain much more complex configurations in which more physical volumes may be involved, and possibly not everyone available at boot time. These situation may demand for some custom device-activation logic which -being custom- can't be embedded into the kernel itself, but needs to be delegated outside. Linux being a general-purpose kernel, an LVM vg is never auto-started, not even if it is "simple".

Please note that this consideration may hold for the raid autostart function too and, infact, this function had been "disabled" from linux 2.6.12 on (but donno if only in the gentoo flavor).

Initramfs (or whatever a system uses to "init" the system itself) must be used both to start the raid units AND the lvm vgs. Think to embedded systems, which may want allow their users to do "strange things" with their raid units or vgs at boot time.

----------

