# EVMS, grub, kernel upgrade

## fmouse

I have a system that boots from an initrd and brings up EVMS, and mounts the root filesystem from an EVMS volume consisting of a RAID-1 array.  This was a bit tricky to set up, but has worked flawlessly for some time.  The system kernel which runs on the booted system has been Linux kernel 2.6.22-gentoo-r8.  Today I tried to update the kernel to 2.6.25-gentoo-r7 and I'm getting a kernel panic with the dreaded "Unable to mount root fs on unknown-block(0,0)" message.  Usually there's a boot diagnostic preceding this which gives more information, but there's a fair amount of statistical info on the drive system, probably related to the kernel panic, and any error messages which relate to this have already scrolled off-screen when the kernel crashes, and of course there's no scroll-back when this happens.

The system boots using grub, and all stanzas of grub.conf are identical, except that the newest one points to "vmlinuz.new" - a symlink to the problem kernel.  I'm suspecting that something's changed in EVMS support in the kernel, or perhaps I need to modify something somewhere else, I don't know.  The relevant stanzas of grub.conf are:

```

title Current Kernel

  root (hd0,0)

  kernel (hd0,0)/vmlinuz root=/dev/evms/root_lv ramdisk=8192 

  initrd (hd0,0)/evms-2.5.5-initrd.gz

title New Kernel

  root (hd0,0)

  kernel (hd0,0)/vmlinuz.new root=/dev/evms/root_lv ramdisk=8192

  initrd (hd0,0)/evms-2.5.5-initrd.gz

```

Anyone have any ideas?

----------

## cyrillic

If you want the kernel to actually use the initrd when booting, I think the kernel line in grub.conf needs to have root=/dev/ram0 rather than some device node that does not exist until after the initrd is run.

----------

## fmouse

 *cyrillic wrote:*   

> If you want the kernel to actually use the initrd when booting, I think the kernel line in grub.conf needs to have root=/dev/ram0 rather than some device node that does not exist until after the initrd is run.

 

The problem with this analysis is that the first and second configs are identical except for the kernel version, and the first one works just fine, as I noted in my original post.

I actually found the source of the problem, although it's not yet fixed and tested.  Pulling all but a couple of ramdisk buffers out of the kernel config allowed me actually see the error message which produced the problem, which I tracked to a version difference between glibc in the initrd and glibc in the booted system,  EVMS is currently an abandoned project.  Pity!  It's a fine piece of work and very useful.  I understand that efforts are underway to incorporate some of the code and functionality of EVMS into udev.

----------

## fmouse

 *cyrillic wrote:*   

> I think the kernel line in grub.conf needs to have root=/dev/ram0 rather than some device node that does not exist until after the initrd is run.

 

The initrd system runs first.  Although it runs in RAM, it's special in that it automatically runs as a temporary root fs and pre-configures the boot environment for the final system.  One of its first tasks, as specified in the linuxrc file, is to start EVMS, so by the time the filesystems needed for the final system are made available to the kernel, all the EVMS volumes are available.  The kernel root param specifies the "real" root filesystem.  See the discussion under "Grub Users" on http://evms.sourceforge.net/install/root.html.

----------

## kiboko

 *fmouse wrote:*   

>  *cyrillic wrote:*   If you want the kernel to actually use the initrd when booting, I think the kernel line in grub.conf needs to have root=/dev/ram0 rather than some device node that does not exist until after the initrd is run. 
> 
> The problem with this analysis is that the first and second configs are identical except for the kernel version, and the first one works just fine, as I noted in my original post.
> 
> I actually found the source of the problem, although it's not yet fixed and tested.  Pulling all but a couple of ramdisk buffers out of the kernel config allowed me actually see the error message which produced the problem, which I tracked to a version difference between glibc in the initrd and glibc in the booted system,  EVMS is currently an abandoned project.  Pity!  It's a fine piece of work and very useful.  I understand that efforts are underway to incorporate some of the code and functionality of EVMS into udev.

 

I am having a problem that sound suspisously similar to yours. EVMS fails to mount my root volume onto /sysroot when trying to boot a 2.6.29-gentoo kernel. I do not see or notice any other errors.

Could you elaborate on your glibc suspicions and how you got to see the additional error. 

My previous working kernel is 2.6.28-gentoo-r3 which was built with the same glibc as the failing 2.6.29-gentoo one. The evms initrd file is the same for both kernel boots. So why would one mount onto /sysroot fine and the other fail?

Thanks

----------

## fmouse

 *kiboko wrote:*   

> I am having a problem that sound suspisously similar to yours. EVMS fails to mount my root volume onto /sysroot when trying to boot a 2.6.29-gentoo kernel. I do not see or notice any other errors.
> 
> Could you elaborate on your glibc suspicions and how you got to see the additional error. 
> 
> My previous working kernel is 2.6.28-gentoo-r3 which was built with the same glibc as the failing 2.6.29-gentoo one. The evms initrd file is the same for both kernel boots. So why would one mount onto /sysroot fine and the other fail?
> ...

 

It's been a while since I had this problem and I don't remember all the details, however the "unknown-block" error was a report of the result of the error, not a report of the error itself.  The actual error was reported several lines earlier in the kernel panic screen, however once the kernel panics, there's no going back in the terminal scroll-back and the important line was lost.  The report on screen which was visible included a number of lines giving stats on ramdisk buffers, so pulling this down to one or two buffers in the kernel config and rebuilding the kernel allowed me to see the actual error several lines prior to the panic.  Googling this error led me to a discussion on, I think, a Debian forum which implicated a glib version mismatch.

To fix the problem, I mounted evms-2.5.5-initrd.gz as a filesystem and ran ldd on various components of the EVMS package, replacing symlink targets where necessary with the proper current versions of glib.  This solved the problem.  I don't know but what I may have had to recompile one or two pieces from source.  I can send you my current version of evms-2.5.5-initrd.gz if you would like to try it.

Be careful with EVMS!  It's an unsupported system these days (hopefully not forever) and is not bug-free.  I attached a used drive to the system which I wanted to re-use and re-partition from scratch.  The problem was that the used drive contained EVMS data which was pulled in by the running EVMS system at boot time and my RAID-1 synchronization and several other EVMS configurations got thoroughly hozed.  It was late, I was tired, and I ended up losing the entire box - a gateway router/server for a couple of home businesses - and had to re-install everything from bare metal, losing a lot of stored data in the process.  We still haven't fully recovered!

----------

## Eonwe

Check https://bugs.gentoo.org/show_bug.cgi?id=273902 to see if you have the same problem.

----------

## kiboko

 *Eonwe wrote:*   

> Check https://bugs.gentoo.org/show_bug.cgi?id=273902 to see if you have the same problem.

 

Thanks for the pointer. I will track this bug and see if any great answers arrive. 

For the present, I have opted to start moving away from EVMS. I have several Gentoo and Debian systems, so this is a real drag and can only done piecemeal. I have managed to successfully move a Raid-1 Gentoo server from EVMS to LVM2.... the exercise took me 3 days, (most of it leanings and waiting for for raid volumes to rebuild). I do not look forward to the move for the Raid-5 servers and may skip go the LVM2 step and just settle for a MD only world.

----------

