# grub setup on software RAID1

## tnt

kernel 2.6.37-gentoo

sys-boot/grub-0.97-r10

I have boot partition set up on 2 members RAID1:

```
md0 : active raid1 sdh2[1] sda2[0]

      488192 blocks [2/2] [UU]

      bitmap: 1/60 pages [4KB], 4KB chunk

```

```
Filesystem            Size  Used Avail Use% Mounted on

/dev/md0              462M  9.7M  429M   3% /boot

```

```
    GNU GRUB  version 0.97  (640K lower / 9216K upper memory)

 [ Minimal BASH-like line editing is supported.  For the first word, TAB

   lists possible command completions.  Anywhere else TAB lists the possible

   completions of a device/filename. ]

grub> find /boot/grub/stage1

 (hd0,1)

 (hd7,1)

```

when I try to install grub on first member of my RAID1, it installs without a problem:

```
grub> root (hd0,1)

 Filesystem type is ext2fs, partition type 0x83

grub> setup (hd0)

 Checking if "/boot/grub/stage1" exists... yes

 Checking if "/boot/grub/stage2" exists... yes

 Checking if "/boot/grub/e2fs_stage1_5" exists... yes

 Running "embed /boot/grub/e2fs_stage1_5 (hd0)"... failed (this is not fatal)

 Running "embed /boot/grub/e2fs_stage1_5 (hd0,1)"... failed (this is not fatal)

 Running "install /boot/grub/stage1 (hd0) /boot/grub/stage2 p /boot/grub/menu.lst "... succeeded

Done.

```

but, when I try to install it the same way on the second RAID1 member, it fails with error 16:

```
grub> root (hd7,1)

 Filesystem type is ext2fs, partition type 0x83

grub> setup (hd7)

 Checking if "/boot/grub/stage1" exists... yes

 Checking if "/boot/grub/stage2" exists... yes

 Checking if "/boot/grub/e2fs_stage1_5" exists... yes

 Running "embed /boot/grub/e2fs_stage1_5 (hd7)"... failed (this is not fatal)

 Running "embed /boot/grub/e2fs_stage1_5 (hd7,1)"... failed (this is not fatal)

 Running "install /boot/grub/stage1 (hd7) /boot/grub/stage2 p /boot/grub/menu.lst "... failed

Error 16: Inconsistent filesystem structure

```

I've tryed to install grub on the second RAID1 member, but point it to the root partition on the first RAID1 member. no luck:

```
grub> root (hd0,1)

 Filesystem type is ext2fs, partition type 0x83

grub> setup (hd7)

 Checking if "/boot/grub/stage1" exists... yes

 Checking if "/boot/grub/stage2" exists... yes

 Checking if "/boot/grub/e2fs_stage1_5" exists... yes

 Running "embed /boot/grub/e2fs_stage1_5 (hd7)"... failed (this is not fatal)

 Running "embed /boot/grub/e2fs_stage1_5 (hd0,1)"... failed (this is not fatal)

 Running "install /boot/grub/stage1 d (hd7) /boot/grub/stage2 p /boot/grub/menu.lst "... failed

Error 16: Inconsistent filesystem structure

```

any ideas how to install grub on both RAID1 members?

----------

## NeddySeagoon

tnt,

You are doing it all correctly.

```
16 : Inconsistent filesystem structure

     This error is returned by the filesystem code to denote an internal

     error caused by the sanity checks of the filesystem structure on

     disk not matching what it expects. This is usually caused by a

     corrupt filesystem or bugs in the code handling it in GRUB.
```

Are you sure you boot raid1 is running ?

What does /proc/mdstat say about it?

Save what you need out of /boot reformat /boot and put the files back.

----------

## tnt

I have the same problem on other 2 boxes, also.

about /boot filesystem/RAID health, I can mount and use:

- md0 as RAID1 (made from sda2 and sdh2) device with ext2 on it

- sda2 as separate device with ext2 on it

- sdh2 as separate device with ext2 on it

```
root@gentoo /tmp # mount /boot

root@gentoo /tmp # df -h /boot

Filesystem            Size  Used Avail Use% Mounted on

/dev/md0              462M  9.7M  429M   3% /boot

root@gentoo /tmp # umount /boot     

root@gentoo /tmp # mdadm -S /dev/md0

mdadm: stopped /dev/md0

root@gentoo /tmp # mount -t ext2 -o ro /dev/sda2 /boot

root@gentoo /tmp # ls -lah /boot/grub/

total 446K

drwxr-xr-x 2 root root 1.0K Nov 19 15:18 .

drwxr-xr-x 4 root root 1.0K Feb 23 14:23 ..

-rw-r--r-- 1 root root  197 Dec 18  2009 default

-rw-r--r-- 1 root root  135 Dec 18  2009 device.map

-rw-r--r-- 1 root root 8.6K Aug  2  2010 e2fs_stage1_5

-rw-r--r-- 1 root root 7.7K Aug  2  2010 fat_stage1_5

-rw-r--r-- 1 root root 7.0K Aug  2  2010 ffs_stage1_5

-rw-r--r-- 1 root root 1.1K Nov 19 15:18 grub.conf

-rw-r--r-- 1 root root 7.0K Aug  2  2010 iso9660_stage1_5

-rw-r--r-- 1 root root 8.5K Aug  2  2010 jfs_stage1_5

lrwxrwxrwx 1 root root    9 Apr  8  2010 menu.lst -> grub.conf

-rw-r--r-- 1 root root 7.2K Aug  2  2010 minix_stage1_5

-rw-r--r-- 1 root root 9.6K Aug  2  2010 reiserfs_stage1_5

-rw-r--r-- 1 root root  34K Aug  2  2010 splash.xpm.gz

-rw-r--r-- 1 root root  512 Aug  2  2010 stage1

-rw-r--r-- 1 root root 106K Aug  2  2010 stage2

-rw-r--r-- 1 root root 106K Dec 18  2009 stage2.old

-rw-r--r-- 1 root root 106K Aug  2  2010 stage2_eltorito

-rw-r--r-- 1 root root 7.2K Aug  2  2010 ufs2_stage1_5

-rw-r--r-- 1 root root 6.6K Aug  2  2010 vstafs_stage1_5

-rw-r--r-- 1 root root 9.4K Aug  2  2010 xfs_stage1_5

root@gentoo /tmp # umount /boot

root@gentoo /tmp # mount -t ext2 -o ro /dev/sdh2 /boot

root@gentoo /tmp # ls -lah /boot/grub/

total 446K

drwxr-xr-x 2 root root 1.0K Nov 19 15:18 .

drwxr-xr-x 4 root root 1.0K Feb 23 14:23 ..

-rw-r--r-- 1 root root  197 Dec 18  2009 default

-rw-r--r-- 1 root root  135 Dec 18  2009 device.map

-rw-r--r-- 1 root root 8.6K Aug  2  2010 e2fs_stage1_5

-rw-r--r-- 1 root root 7.7K Aug  2  2010 fat_stage1_5

-rw-r--r-- 1 root root 7.0K Aug  2  2010 ffs_stage1_5

-rw-r--r-- 1 root root 1.1K Nov 19 15:18 grub.conf

-rw-r--r-- 1 root root 7.0K Aug  2  2010 iso9660_stage1_5

-rw-r--r-- 1 root root 8.5K Aug  2  2010 jfs_stage1_5

lrwxrwxrwx 1 root root    9 Apr  8  2010 menu.lst -> grub.conf

-rw-r--r-- 1 root root 7.2K Aug  2  2010 minix_stage1_5

-rw-r--r-- 1 root root 9.6K Aug  2  2010 reiserfs_stage1_5

-rw-r--r-- 1 root root  34K Aug  2  2010 splash.xpm.gz

-rw-r--r-- 1 root root  512 Aug  2  2010 stage1

-rw-r--r-- 1 root root 106K Aug  2  2010 stage2

-rw-r--r-- 1 root root 106K Dec 18  2009 stage2.old

-rw-r--r-- 1 root root 106K Aug  2  2010 stage2_eltorito

-rw-r--r-- 1 root root 7.2K Aug  2  2010 ufs2_stage1_5

-rw-r--r-- 1 root root 6.6K Aug  2  2010 vstafs_stage1_5

-rw-r--r-- 1 root root 9.4K Aug  2  2010 xfs_stage1_5

root@gentoo /tmp # umount /boot

root@gentoo /tmp # mdadm --assemble /dev/md0 /dev/sda2 /dev/sdh2

mdadm: /dev/md0 has been started with 2 drives.

root@gentoo /tmp # cat /proc/mdstat 

Personalities : [linear] [raid0] [raid1] [raid6] [raid5] [raid4] 

md0 : active raid1 sda2[0] sdh2[1]

      488192 blocks [2/2] [UU]

      bitmap: 0/60 pages [0KB], 4KB chunk

unused devices: <none>

root@gentoo /tmp #

```

so, it seems that both partitions (sda2 and sda7) contain fully accessible filesystem

and, then again, as I showed in previous post, I cannot install grub on hd7 (sdh) even if I use "root hd(0,1)", and installation to hd0 (sda) is possible with the same root.  :Sad: 

----------

## gentoo_ram

That's a lot of hard drives.  The only thing I can think is that there's some funkiness going on with the drive mapping between what Linux sees and what GRUB thinks the BIOS will see.  Of course, the 'find' command output doesn't back up my theory, but it's the only thing I can think.

I have my /boot on a RAID 1 as well.  I did the same thing you did...

root (hd0,0)

setup (hd0)

root (hd1,0)

setup (hd1)

Worked fine for me.

----------

## NeddySeagoon

tnt,

What is in /boot/grub/device.map ?

Does it correspond with your view of the system ?

```
grub> root (hd0,1)

 Filesystem type is ext2fs, partition type 0x83

grub> setup (hd7) 
```

would not produce the desired result, even if it worked.  The grub stage1 and stage 1.5, if its used are changed during the install so that the parts can find one another.

----------

## tnt

device.map looks like:

```
(fd0)   /dev/fd0

(hd0)   /dev/sda

(hd1)   /dev/sdb

(hd2)   /dev/sdc

(hd3)   /dev/sdd

(hd4)   /dev/sde

(hd5)   /dev/sdf

(hd6)   /dev/sdg

(hd7)   /dev/sdh
```

I understand that installing grub on sdh (hd7) MBR and pointing it to the partition on some other hard disk would not produce the desired result - I just tried that to see if there's something wrong with the partition on sdh (hd7,1).

this way, it seems the problem is not partition-related. probably it's MBR-related...

----------

## NeddySeagoon

tnt,

What is 

```
(hd7)   /dev/sdh
```

attached to?

Is it a bootable controller?

----------

## tnt

it should be.

it is secondary jmicron controler and that hard drive is listed in boot order in BIOS.

it is on the second place in boot order (if frist one which is sda fails).

----------

## jormartr

From http://en.gentoo-wiki.com/wiki/RAID/Software#Installing_Grub_onto_both_MBRs

This is what you should use, if you were using raid1 with boot on /dev/sd[ab]1

Install into sda:

```

device (hd0) /dev/sda

root (hd0,0)

setup (hd0)

```

Install into sdb:

```

device (hd0) /dev/sdb

root (hd0,0)

setup (hd0)

```

----------

## tnt

thx a lot, but:

 *Quote:*   

> grub> device (hd0) /dev/sda
> 
> grub> root (hd0,1)
> 
>  Filesystem type is ext2fs, partition type 0x83
> ...

 

----------

## tnt

ok, this seems to do the trick:

1. stop /dev/md0 (RAID1 boot partition made from /dev/sda2 and /dev/sdh2)

```
mdadm -S /dev/md0

mdadm: stopped /dev/md0
```

2. mount (read-only) /dev/sdh2 as /boot

```
mount -o ro -t ext2 /dev/sdh2 /boot
```

3. setup grub:

```
grub> find /boot/grub/stage1

 (hd0,1)

 (hd7,1)

grub> root (hd7,1)

 Filesystem type is ext2fs, partition type 0x83

grub> setup (hd7)

 Checking if "/boot/grub/stage1" exists... yes

 Checking if "/boot/grub/stage2" exists... yes

 Checking if "/boot/grub/e2fs_stage1_5" exists... yes

 Running "embed /boot/grub/e2fs_stage1_5 (hd7)"... failed (this is not fatal)

 Running "embed /boot/grub/e2fs_stage1_5 (hd7,1)"... failed (this is not fatal)

 Running "install /boot/grub/stage1 (hd7) /boot/grub/stage2 p /boot/grub/menu.lst "... succeeded

Done.

grub>
```

I realy don't understand why everything else failed and this succeded, but that's the way it is...

----------

## jormartr

Just for checking that there is no inconsistency between the two members on the raid, execute

```

echo check > /sys/block/md0/md/sync_action

```

then, check /proc/mdstat, and when you see that the check has finished, check the values with

```

cat /sys/block/md0/md/mismatch_cnt

```

that will show you, the number of sectors on the raid1, that differ on members. That obviously should not happen, but sometimes it happens. If the value is 0, it's ok.

----------

## tnt

thx a lot

```
echo check > /sys/block/md0/md/sync_action

cat /sys/block/md0/md/mismatch_cnt 

0

```

and, just in case, what to do (how to sync) array if there's non-zero value ?

----------

## AngelKnight

There's a note in the grub notes somewhere (man page?) to the effect that when you supply (hdX) disk specifiers in grub, you have to be mindful of what the *BIOS* will see, *NOT* what grub-running-on-linux will see.

Disclaimer: This set of steps works out for me and is based on sound principles, but I haven't seen your precise computer.  (This *should* work, but I don't guarantee it won't break your system and require a livecd, misfile your taxes, set your garbage on fire or impregnate your puppy...)

For your setup, what I might recommend is:

1) run grub like this:

```
# grub --device-map=/dev/null

(grub) device (hd0) /dev/sda

(grub) root (hd0,1)

(grub) setup (hd0)

(grub) quit (quit or exit?  I never remember...)
```

2) then, run grub like this:

```
# grub --device-map=/dev/null

(grub) device (hd0) /dev/sdh

(grub) root (hd0,1)

(grub) setup (hd0)

(grub) quit
```

The reason is that whichever disk the BIOS picks, it's likely to be BIOS disk 0x80, which is what grub calls "hd0" under whichever circumstances grub actually loads at the time.

----------

