# [solved] Kernel, RAID and mdadm woes (initrd)

## Jimini

Problem:

One RAID1 and one RAID6. The mirrored arrays keep the system (/, /boot, /usr and so on), the RAID6 only keeps shared data. The kernel's autodetection functionality did not work at all with the RAID6, but I could not simply turn it off due to the RAID1, which has to be started before the whole system stuff is mounted.

Solution:

I had to create an initrd which assembles all arrays, before the partitions are mounted.

Step 1 - software installation:

sys-apps/busybox and sys-fs/mdadm need to be installed with activated "static"-useflag:

USE="static" emerge busybox mdadm

Step 2 - preparation of the initrd:

```
mkdir -p /usr/src/initramfs/{bin,dev,etc,proc,sbin,sys}

cd /usr/src/initramfs

cp -a /bin/busybox bin/

cp -a /dev/md* dev/

cp -a /etc/mdadm.conf etc/

cp -a /sbin/mdadm sbin/
```

Create a file /usr/src/initramfs/init with the following content:

```
#!/bin/busybox sh

mount -t proc none /proc

mount -t sysfs none /sys

mount -t devtmpfs none /dev

/sbin/mdadm --assemble /dev/md1 /dev/sdg1 /dev/sdh1

/sbin/mdadm --assemble /dev/md2 /dev/sdg2 /dev/sdh2

/sbin/mdadm --assemble /dev/md3 /dev/sdg3 /dev/sdh3

/sbin/mdadm --assemble /dev/md5 /dev/sdg5 /dev/sdh5

/sbin/mdadm --assemble /dev/md6 /dev/sdg6 /dev/sdh6

/sbin/mdadm --assemble /dev/md7 /dev/sdg7 /dev/sdh7

/sbin/mdadm --assemble /dev/md8 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1

mount -o ro /dev/md1 /mnt/root || rescue_shell

#clean up. The init process will remount proc sys and dev later

umount /dev

umount /sys

umount /proc

exec switch_root /mnt/root /sbin/init

rescue_shell() {

        echo "Something went wrong. Dropping you to a shell."

                busybox --install -s

        exec /bin/sh

}
```

```
chmod +x /usr/src/initramfs/init
```

Step 3 - kernel configuration:

```
cd /usr/src/linux

make menuconfig
```

You have to compile your kernel with the following options:

General Setup --->

[*] Initial RAM filesystem and RAM disk (initramfs/initrd) support

(/usr/src/initramfs/) Initramfs source file(s)  (if you want to embed your initrd into the kernel, set the path to the files here)

Device Drivers --->

Generic Driver Options --->

[*] Maintain a devtmpfs filesystem to mount at /dev

```
make && make modules_install && make install
```

If you set the options to embed the initrd into your kernel, that's it. If you want to keep the initrd seperate, you have to compress it:

```
find . -print0 | cpio --null -ov --format=newc | gzip -9 > /boot/initramfs.cpio.gz
```

That should be the whole thing. I wrote these steps down as I remembered them - and as I found them in this thread or in some guides. If you experience any problems or find mistakes, feel free to post them here!

You should also take a look at the following guides:

http://en.gentoo-wiki.com/wiki/Initramfs

Thanks to NeddySeagoon and cach0rr0 for their support.

Best regards,

Jimini

=================================================================================================================================

Hey there,

a few days ago, I set up a Gentoo box, containing two mirrored disks (RAID1):

Arrays (from /etc.mdadm.conf):

```
ARRAY /dev/md1 devices=/dev/sda1,/dev/sdb1

ARRAY /dev/md2 devices=/dev/sda2,/dev/sdb2

ARRAY /dev/md3 devices=/dev/sda3,/dev/sdb3

ARRAY /dev/md5 devices=/dev/sda5,/dev/sdb5

ARRAY /dev/md6 devices=/dev/sda6,/dev/sdb6

ARRAY /dev/md7 devices=/dev/sda7,/dev/sdb7

ARRAY /dev/md8 devices=/dev/sda8,/dev/sdb8
```

/etc/fstab:

```
/dev/md2        /       ext4    defaults                0 1

/dev/md1        /boot   ext4    defaults                0 1

/dev/md3        none    swap    sw                      0 0

/dev/md5        /tmp    ext4    defaults,nosuid,noexec  0 1

/dev/md6        /var    ext4    defaults,nosuid         0 1

/dev/md7        /usr    ext4    defaults                0 1

/dev/md8        /home   ext4    defaults,nosuid,noexec  0 1
```

Now I have the following problems:

1) Autodetection of fd-flagged partitions by the kernel is activated. All seven arrays can be started without any problem. But during the boot-process, mdadm complains about missing superblocks on the following partitions

```
mdadm: /dev/sda1 has no superblock - assembly aborted

mdadm: /dev/sda2 has no superblock - assembly aborted

mdadm: /dev/sda3 has no superblock - assembly aborted

mdadm: /dev/sda5 has no superblock - assembly aborted

mdadm: /dev/sda6 has no superblock - assembly aborted

mdadm: /dev/sda7 has no superblock - assembly aborted

mdadm: /dev/sda8 has no superblock - assembly aborted
```

Of course these partitions (and also the partitions on /dev/sdb) have no superblock, why should I format them, when I just want to use them in a RAID1?

According to /proc/mdstat and mdadm --detail /dev/md*, all arrays are clean. If I comment out the ARRAY-lines in /etc/mdadm.conf the error disapperas. It seems, as if the arrays are started twice - first by the kernel, then by mdadm, I suppose. But with the lines commented out, mdadm complains about no arrays in its config.

I'm pretty sure that this is not critical at all, but it's kind of annoying.

2) When I shut down the box, /dev/md2 can not be stopped cleanly. My assumption is, that this array has to be stopped after all the other arrays. How can I manipulate the order, in which the arrays are stopped?

Best regards,

Jimini

----------

## NeddySeagoon

Jimini,

There are two sorts of superblock involed.

The raid superblock is written to all of the underlying partitions in a raid set.  The kernel or mdadm uses that to know which partitions belong to which raid set.

Try  

```
mdadm -E /dev/sda1
```

The kernel will not auto assemble raid sets unless the raid superblock version is Version : 0.90 which is no longer the default.

Once the raid set is assembled, you can put a filesystem on it. It then aquires a filesystem superblock, not related to the raid superblock, which the kernel uses to mount the filesystem.

Sight of your  mdadm -E /dev/sda1 outout may shed some light on this.

Oh, I don't have an /etc/mdadm.conf file here at all and mdadm is not run as a part of my boot process.

----------

## Jimini

Thank you for your helpful answer. 

The output of mdadm -E /dev/sda1:

```
/dev/sda1:

          Magic : a92b4efc

        Version : 0.90.00

           UUID : f5bbc70f:b435cc60:b0b17f6f:2d80ee75

  Creation Time : Sat Mar  5 21:20:36 2011

     Raid Level : raid1

  Used Dev Size : 112320 (109.71 MiB 115.02 MB)

     Array Size : 112320 (109.71 MiB 115.02 MB)

   Raid Devices : 2

  Total Devices : 2

Preferred Minor : 1

    Update Time : Wed Mar  9 07:53:07 2011

          State : clean

 Active Devices : 2

Working Devices : 2

 Failed Devices : 0

  Spare Devices : 0

       Checksum : cc3ac745 - correct

         Events : 84

      Number   Major   Minor   RaidDevice State

this     0       8        1        0      active sync   /dev/sda1

   0     0       8        1        0      active sync   /dev/sda1

   1     1       8       17        1      active sync   /dev/sdb1
```

As I understand you, it is still possible to let only the kernel or only mdadm manage the array(s) - of course, that makes sense. But in my case it seems as if I would have to use the kernel to assemble the arrays, because the whole system is stored on them. 

What can I do to fix these error messages? Simply format all partitions on both disks before I create the arrays? Of course, I could also use the kernel only and keep mdadm out of my system (but I liked the possibility of an email-alert, when an array is degraded).

Can you help me with my second problem? I am a bit concerned about / being shut down without being stopped cleanly.

Best regards,

Jimini

----------

## NeddySeagoon

Jimini,

You have several choices ...

Use kernel raid auto assemble and ignore the error mesages

Put mdadm into an initrd and use that to assemble your raid sets.  We will all have to do that one day as raid auto assembly is depreciated.

Set up mdadm to monitor the raid sets but not assemble them. 

You can probably put something in /etc/conf.d/mdadm to achieve that effect.  If not, you can take mdadm out your runlevels and add the appropriate mdadm command to /etc/conf.d/local  in the start section.

----------

## Jimini

I removed /etc/mdadm.conf, now I let the kernel assemble everything. Without mdadm.conf, I do not get any error messages about being unable to stop /dev/md2 anymore. 

It's a bit dissatisfying, but at least it seems to work without difficulties. 

Best regards,

Jimini

----------

## Jimini

I have a big problem to boot my system with an initrd. I have a RAID1 (md[1,2,3,5,6,7], which is mounted as /, /boot/ swap, /usr, /var and /home) and a RAID6. The kernel's autodetection seems to have big problems with my RAID6, so I'd like to assemble all arrays by using an initrd.

According to http://en.gentoo-wiki.com/wiki/Initramfs , I emerged the static built of mdadm and copied the following files into the initramfs' directory:

```
Atlas initramfs # ls *

init

bin:

busybox

dev:

console  md  md0  md1  md2  md3  md5  md6  md7  md8  null  sda  sda1  sdb  sdb1  sdc  sdc1  sdd  sdd1  sde  sde1  sdf  sdf1  sdg  sdg1  sdg2  sdg3  sdg4  sdg5  sdg6  sdg7  sdh  sdh1  sdh2  sdh3  sdh4  sdh5  sdh6  sdh7  tty

etc:

mdadm.conf

lib:

mnt:

root

proc:

root:

sbin:

mdadm

sys:
```

Then I put the following lines into /usr/src/initramfs/init:

```
#!/bin/busybox sh

/sbin/mdadm --assemble /dev/md1 /dev/sdg1 /dev/sdh1

/sbin/mdadm --assemble /dev/md2 /dev/sdg2 /dev/sdh2

/sbin/mdadm --assemble /dev/md3 /dev/sdg3 /dev/sdh3

/sbin/mdadm --assemble /dev/md5 /dev/sdg5 /dev/sdh5

/sbin/mdadm --assemble /dev/md6 /dev/sdg6 /dev/sdh6

/sbin/mdadm --assemble /dev/md7 /dev/sdg7 /dev/sdh7

/sbin/mdadm --assemble /dev/md8 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1
```

Finally, I compressed the folder (find . -print0 | cpio --null -ov --format=newc | gzip -9 > /boot/test.cpio.gz) and edited my grub.conf:

```
title 2.6.36-r8

root (hd0,1)

kernel (hd0,1)/vmlinuz-2.6.36-gentoo-r8 root=/dev/md1

initrd (hd0,1)/test.cpio.gz
```

When I try to boot this entry, I get a kernel panic (Kernel Panic - not syncing: VFS Unable to mount root fs on unknown block (0,0)). I'm sure that I made a mistake with my init, but even after reading lots of postings and guides, I'm still unable to find it.

Best regards,

Jimini

----------

## NeddySeagoon

Jimini,

So close ... 

```
kernel (hd0,1)/vmlinuz-2.6.36-gentoo-r8 root=/dev/ram0 real_root=/dev/md1
```

You may also need an init=/<path/to/init_in initrd> in there too.

You have done everything except told the kernel to use the initrd

----------

## cach0rr0

 *Jimini wrote:*   

> I'm sure that I made a mistake with my init, but even after reading lots of postings and guides, I'm still unable to find it.
> 
> 

 

Possible to post your init on pastebin? 

If you don't care to add in the complexity of trying to parse command-line arguments passed to the kernel, you could do a very small init that does everything you need

If you have devtmpfs support built into your kernel (which also means you don't have to copy those device nodes over, at all):

```

#!/bin/busybox sh

mount -t proc none /proc

mount -t sysfs none /sys

mount -t devtmpfs none /dev 

#assemble raid

/sbin/mdadm --assemble /dev/md1 /dev/sdg1 /dev/sdh1 

/sbin/mdadm --assemble /dev/md2 /dev/sdg2 /dev/sdh2 

/sbin/mdadm --assemble /dev/md3 /dev/sdg3 /dev/sdh3 

/sbin/mdadm --assemble /dev/md5 /dev/sdg5 /dev/sdh5 

/sbin/mdadm --assemble /dev/md6 /dev/sdg6 /dev/sdh6 

/sbin/mdadm --assemble /dev/md7 /dev/sdg7 /dev/sdh7 

/sbin/mdadm --assemble /dev/md8 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1

# mount root read-only. Adjust if md1 is not correct for you

mount -o ro /dev/md1 /mnt/root || rescue_shell

#clean up. The init process will remount proc sys and dev later

umount /dev

umount /sys

umount /proc

exec switch_root /mnt/root /sbin/init

rescue_shell() {

        echo "Something went wrong. Dropping you to a shell."

                busybox --install -s

        exec /bin/sh

}

```

Since devtmpfs will create device nodes automagically, no need to copy device nodes, and the initramfs is fairly empty

```

ricker initramfs # find . |xargs file

.:                          directory

./bin:                      directory

./bin/busybox:              ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, for GNU/Linux 2.6.9, stripped

./dev:                      directory

./etc:                      directory

./etc/mdadm.conf:           ASCII English text

./lib:                      directory

./mnt:                      directory

./mnt/root:                 directory

./sys:                      directory

./usr:                      directory

./init:                     a /bin/busybox sh script text executable

./proc:                     directory

./sbin:                     directory

./sbin/mdadm:               ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, for GNU/Linux 2.6.9, stripped

./sbin/cryptsetup:          ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, for GNU/Linux 2.6.9, stripped

./root:                     directory

```

This type of init setup is not very flexible, as it does not parse /proc/cmdline and check for any parameters you pass to the kernel. For example, if you used this type of init, your grub.conf would just have:

```

title 2.6.36-r8 

root (hd0,1) 

kernel (hd0,1)/vmlinuz-2.6.36-gentoo-r8 

initrd (hd0,1)/test.cpio.gz

```

(the absence of 'root=' is not a typo. Instead of passing root on the command-line, in this case it is hard-coded into the initramfs)

Not perfect. But it may be something to start with? 

This is where I get out of the way and let NeddySeagoon help, as I do not know heaps about mdadm, but the initramfs I am pretty comfortable with now (I think? I hope?)

----------

## Jimini

You guys did a great job, it finally works. I decided to compile the initramfs into the kernel. It took me over one week to get this damn array work the way I wanted it. 

THANKS THANKS THANKS!

Later on, I'll summarize what I did, because helpful guides for this problem are really rare.

Best regards,

Jimini

----------

## Blubbmon

I searched half a day for some howto like this one. Fortunately, I completely read the manpage of genkernel just before I've found this posting. 

If I understand correctly, you just want all your raid stuff to be activated at boot time - no matter if the kernel or init does the job. Then you can built your kernel and initramfs with genkernel (including the command line parameter --mdadm). Afterwards you must use the parameter "domdadm" on grub's kernel line to activate the auto detection. The kernel will still not recognise the raids with metadata version 1.2, but the mdadm included in the initramfs will automagically assemble all raids according to your previously configured /etc/mdadm.conf.

See also: /usr/share/genkernel/defaults/linuxrc and /usr/share/gentoo/defaults/initrd.scripts

However, thanks for the initramfs example!

BTW: Google gives exactly 11 results when searching for "domdadm".

----------

## NeddySeagoon

Blubbmon,

You must also make your /boot non raid or raid1 with version 0.90 raid superblocks or grub1 won't load from it.

----------

## Blubbmon

Right, this is true as long as grub 2 is not "available"  :Smile: 

However, the main problem was, that I created the /boot raid as version 0.9 according to the documentation, but still didn't found the proper option for initramfs.

Maybe someone could update the installation guide: http://www.gentoo.org/doc/en/gentoo-x86+raid+lvm2-quickinstall.xml

THX.

FYI: https://bugs.gentoo.org/show_bug.cgi?id=376691

----------

