# After upgrade, LVM volume not detected at boot

## twork

I've just finished an extensive upgrade on my primary desktop, and it's hosed.

My new, "target" setup uses kernel 2.6.34, built by genkernel, using --lvm2.  At boot time, we fail with:

```
[...]

>> Scanning for Volume Groups

  Reading all physical volumes.  This may take a while...

>> Activating Volume Groups

>> Determining root device...

!! Block device /dev/pile/dense-root is not a valid root device...

!! Could not find the root block device in .

   Please specify another value or: press Enter for the same, type "shell" for a shell, or "q" to skip...

root block device() :: 
```

(Hand-transcribing this, typos are possible)

From that point, if I drop out to a shell, my physical devices are present (/dev/sda, /dev/sda1, /dev/sda2, /dev/sdb, and /dev/sdb1), so I think all my hardware support is in order.

The lvm binary is present in the ramdisk, but doesn't detect my volumes, for instance,

```
# lvm pvscan

  No matching physical volumes found

# lvm vgscan

  Reading all physical volumes.  This may take a while...

  No volume groups found

# lvm vgchange -a y

  No volume groups found
```

I can boot my previous kernel (2.6.29, hand-rolled), but my userland now has udev-151, and the old kernel has the deprecated sysfs layout, so I have to hand-create my disk device nodes each time, and I can't make it past single user mode (so no network, so no way to emerge an older udev; no going back).

But, if I boot to the older kernel, my LVM volumes work fine.  Using e.g. fdisk -l, I see that my partitions have type 8e (Linux LVM).  Running "vgdisplay" after my SCSI device nodes are created shows a healthy /dev/pile/dense-root.

I've tried various arguments for the new kernel, with no change in behavior.  The older kernel, with the mostly-working setup, uses the arguments:

```
root=/dev/ram0 dolvm2 lvm2root=/dev/pile/dense-root
```

When I first upgraded I tried copying those arguments verbatim; when they didn't work, I tried following the prompts from my genkernel output; for instance, it says I need "dolvm", not "dolvm2" (no difference); and "real_root" pointing to my root device instead of "lvm2root" (also no change that I can see).

In the config for the new, LVM-busted kernel, I have DM and related code compiled statically, but maybe I'm leaving something out.  From my .config:

```
CONFIG_MD=y

CONFIG_BLK_DEV_MD=y

CONFIG_MD_AUTODETECT=y

CONFIG_MD_LINEAR=y

[...]

CONFIG_BLK_DEV_DM=y

CONFIG_DM_CRYPT=y

CONFIG_DM_SNAPSHOT=y

CONFIG_DM_MIRROR=y

# CONFIG_DM_LOG_USERSPACE is not set

CONFIG_DM_ZERO=y

CONFIG_DM_MULTIPATH=y
```

If there are other kernel settings I need, I'm not sure what they are.  Maybe that's my problem?

Any ideas for someplace else to look for clues?  I'm stumped.

----------

## idella4

Why use a ramdisk?  Just something else to go wrong.  With lvm you would need your boot partition in ect2 or 3, the kernel line with the root=/dev or perhaps UUID= ....  I have two patritions in lvms, and they get there fine

```

genny grub # ls -l /dev/mapper

total 0

lrwxrwxrwx 1 root root       7 Nov 24 16:07 HyperVG01-lenny01--disk1 -> ../dm-0

lrwxrwxrwx 1 root root       7 Nov 24 16:07 HyperVG01-ubuntu01--disk1 -> ../dm-1

crw------- 1 root root 10, 236 Nov 25  2010 control

```

----------

## twork

 *idella4 wrote:*   

> Why use a ramdisk?  Just something else to go wrong.  With lvm you would need your boot partition in ect2 or 3, the kernel line with the root=/dev or perhaps UUID= ....  I have two patritions in lvms, and they get there fine
> 
> 

 

Everything worked fine before I updated my kernel.  Even after kernel update, my ramdisk works fine, my boot partition is its own ext2 filesystem, and when I revert to an older kernel the contents of /dev/mapper are correct for my system.

My problem isn't anything in the mechanics of the boot process.  The trouble is that when it gets to the point of initializing LVM, it fails and can't get any further.

Why use a ramdisk?  Well initially it was because I wanted to try genkernel and that's just the way genkernel does business.  But I'm glad it does use a ramdisk, because when things go wrong (like now) I at least have a miniature OS where I can poke around and run some diagnostics.  I'm not stuck sitting through reboots and trying to watch text scroll by just to get a clue about where things are failing.

----------

## idella4

fair enough, but there are no clues in your post that hints twards what is wrong.

So how did you update from kernel 2.6.29, to 2.6.34?

Save your config, invoke a defconfig & try, it usually gets it right,

If you're desperate, post the whole config and I shall shall run and test it here.

----------

## twork

 *idella4 wrote:*   

> fair enough, but there are no clues in your post that hints twards what is wrong.

 

No kidding!  As far as I can tell it should all work, and the closest I get to diagnostic info is quoted in my post.  I really am looking for clues on where to find clues.

 *Quote:*   

> So how did you update from kernel 2.6.29, to 2.6.34?

 

Copied my old .config to the new kernel directory and ran genkernel with --no-clean.  Since then I've re-run the process with a menuconfig step a couple of times, looking around for things I might have missed, but I haven't changed anything major, just the localversion string, and switched off the deprecated ATA/IDE support just in case that made any difference.  That was before I was sure I had hardware support right; I've been bitten by changes in the SATA drivers on lots of previous upgrades.  But I digress.

 *Quote:*   

> Save your config, invoke a defconfig & try, it usually gets it right,

 

Usually!

Except when something changes.  E.g., see above, SATA.

 *Quote:*   

> If you're desperate, post the whole config and I shall shall run and test it here.

 

Okey dokey.

(I do now have the system back to a state where it boots; turns out I did have a binary package of the previous udev release, so I can at least get it up and running, or I wouldn't be able to get at the new kernel config either.  Anyhow...)

http://evil.uchicago.edu/~mjinks/dense-config.txt

----------

## idella4

twork

while I'm compiling, ever used smartctl??  I'm stumped on how to get readings out of it.

----------

## twork

 *idella4 wrote:*   

> while I'm compiling,

 So kind, thank you. *Quote:*   

>  ever used smartctl??  I'm stumped on how to get readings out of it.

 I don't know what smartctl is, sorry.

----------

## idella4

 *twork wrote:*   

> 
> 
> (I do now have the system back to a state where it boots; 
> 
> 

 

right.

ok, it's on its way, won't be long.  only changed  the processor and that's all I think.  You have some gaps where I would have checks, but we'll see.  This is a 2.6.34-hardened, but with the security stuff unchecked it's a 2.6.34.

----------

## salahx

Actually, I ran into something similar, you may be running into this: https://bugs.gentoo.org/show_bug.cgi?id=292833

----------

## idella4

thanks salahx

oooo, I had to make an adjustment to the firmware entry to get it to complete the compile.  Unexpected.

First run, kernel panic

ok, 2nd run..............

booted to a console, ro, don't know why ro.  Enter the usual mount -o remount,rw,exec rootfs / makes root rw.

It could not detect the lvm partitions, and only having ext3 and reisrfs on board, I couldn't reach the kernel-2.6.34-hardened.

So about the same as you has already. i had to add a couple of entries in firmware to complete the compile, but I suppose that's related to my hardware.

ok, I rebooted into my kernel-2.6.34-r2, and got all partitions as normal, just couldn't get into WM, which is trivial.

Had no initrd, wasn't needed since it got to a console. cmd line was close to 

```

genny bin # cat /proc/cmdline

ro root=/dev/sda6 crashkernel=16M@32M console=tty0 resume=swap:/dev/sda7 fbcon=font:10x18 fbcon=scrollback:1024

```

I'm not quite the guru, so shall take an easier path.

To your config. I added 

 *Quote:*   

> 
> 
> .config - Linux Kernel v2.6.34-hardened-r6 Configuration                                   
> 
>  ────────────────────────────────────────────────────────────────────────────────────────── 
> ...

 

and my processor, and that's it. Assuming you have lvm2 all emerged and bootable on your old kernel, I don't know offhand why it only gets to ro, but I can work it but will take some time.  For now, take this config, my 2.6.34, and cut it down to your system needs. You'll find it has many more components checked.  Try to cut out only the unnecessary, like file systems, and work your way down to a selected minimal state of your preference. It will pick up your lvm2 volumes, the important element.

----------

## KShots

Similar issue here... I run an initramfs from an imanifest file with a custom-written init script. This happened when I updated lvm to 2.02.103. Unfortunately, I did something stupid and updated all of my machines to this version, and now none of them will boot without manual intervention.

I can convince it to boot by issuing a "vgscan --mknodes --ignorelockingfailure", which convinces it to create nodes under /dev/mapper, which I can then mount. I then need to unmount /dev, /proc, and /sys, and do an "exec switch_root /newroot /sbin/init" to start my system up. Oddly enough, my script has _always_ had this line in it (the vgscan line). However, despite having this line and executing it, it apparently always fails the first time. Maybe I should have it execute twice...  :Sad:  For whatever reason, when I updated to the new LVM and recompiled my kernel (my initrd is built into my kernel, so I don't need an initrd line in grub and it compiles with the latest versions of everything every time), the nodes no longer appear in /dev/mapper.

On a side note, I have _always_ received messages on boot (after the initrd) something along the lines of the device nodes not being present (udev should have created them), and falling back to manual creation. Now I no longer receive these messages (and of course the system does not boot).

----------

