# root on hardware RAID(scsi) with initrd

## conchyliferous

Hey, this whole scenario might be tricky to explain but I have fought with it some days now so I will try.

I have three disks in my box. One 140GB IDE and two RAIDed 120GB SATA disks.

I have run gentoo for a long time now only on my IDE drive, not even used the RAID disks.

Since the RAID disks are so much faster than the single IDE one, I would like to install gentoo on them and throw away my slow disk.

This should be kind of easy I though. I'm using a RAID/SATA card from Highpoint and I have compiled the driver module from their site and that works. The module is called 'hpt374.ko'.

In my current configuration when i do a 'insmod hpt374.ko', it works like expected. I get some sda devices from it and I can mount them without problems.

When I knew that worked i begun to try get a bootable root on the RAID. First of all I partitioned the RAID into a boot, swap and root partition.

Since BIOS uses interrupt on SCSI (RAID) exactly the same way as it does on ordinary IDE it's no problems to get Grub use the boot partition on the RAID. The tricky part, for me anyway, is to get the Linux kernel to find it's way on the RAID. So the boot partition on the RAID is setup and working but if I try to go ahead and load the kernel I get 'Kernel panic: bla bla' because the RAID driver isn't loaded into kernel yet (since the kernel isn't loaded  :Razz: ). So for this I have to use initrd, a RAM-disk that loads before the kernel.

As I see it, it's when making this initrd image i get problems. I have spent some days now googeling for information how to do this the right way, but there isn't much to go on and most of the information is outdated or unprecise. But I did find how to make and mount the initrd image and some information on what files and directorys that should be in it. Additional to this I want to include and load my 'hpt374.ko' module so that the rest of the kernel can be loaded later. And for the 'hpt374.ko' module I guess I need to have SCSI support and SCSI disk support (RAID=>SCSI). Therefore I need to load 'scsi_mod.ko' and 'sd_mod.ko'. And after that I need to get ext3-support so that I can mount the filesystem, for that I need to load 'jbd.ko' and 'ext3.ko'.

A made the initrd and put the files needed in it. The following is a list of files and directory that I have:

.

./lib

./lib/scsi_mod.ko

./lib/sd_mod.ko

./lib/hpt374.ko

./lib/jbd.ko

./lib/ext3.ko

./bin

./bin/nash

./bin/insmod

./sbin

./sbin/bin

./sbin/modprobe

./etc

./dev

./dev/console

./dev/null

./dev/ram

./dev/systty

./dev/tty1

./dev/tty2

./dev/tty3

./dev/tty4

./loopfs

./proc

./sysroot

./linuxrc

This image gets loaded into the memory before the kernel gets loaded. To make this useful the 'linuxrc' script is run and things should get prepared before the real kernel is loaded. In my case I must have kernel-support for the disks that the kernel is loaded from.

This is my linuxrc script:

```
echo "Loading scsi_mod module"

insmod /lib/scsi_mod.ko

echo "Loading sd_mod module"

insmod /lib/sd_mod.ko

echo "Loading RocketRAID module"

insmod /lib/hpt374.ko

echo "Loading jbd module"

insmod /lib/jbd.ko

echo "Loading ext3 module"

insmod /lib/ext3.ko

mount -t proc /proc /proc

echo Mounting /proc filesystem

echo Creating root device

mkrootdev /dev/root

echo 0x0100 > /proc/sys/kernel/real-root-dev

umount /proc

echo Mounting root filesystem

mount --ro -t ext3 /dev/root /sysroot

pivot_root /sysroot /sysroot/initrd
```

(I know there is a program called mkinitrd that is supposed to make an initrd for you. But I didn't get that to work and I felt I got more control this way)

Now I should boot this up. But before that I need to make changes in my grub.conf and tell grub to load the image.

This is my grub.conf:

```
timeout 43

default 0

title  GNU/Linux

root (hd0,0)

kernel /bzImage ro root=/dev/sda3 noapic

initrd /initrd.img
```

The initrd.img is in place in the boot partition and all that. So let's reboot!

I get trough the bootloader alright and grub reads the initrd image and starts executing the linuxrc script. But already at 'insmod /lib/scsi_mod.ko' I get an error. The error output rushes over the screen very fast and it's hard to read what it's all about, but it's practically alot of hex code and stuff and in the end I get the message: ERROR: /bin/insmod exited abnormally!  :Surprised: 

But the boot continues and I get the same error at every other module except 'hpt374.ko' that, for some reason, seam to load correct and I can see that it gets the devices sda1, sda2 and sda3. And i can see that the total size of the disk is 240GB (and that's correct, it's two RAID 120GB disks). But still I got errors on the other modules so I get a kernel panic when root is going to mount.

So the question is. Why can't I insmod the other modules? This could maybe have something to do with the kernel version the modules are compiled for. But how could that be? They are all compiled for 2.6.11-gentoo-r6 and that's also the version I'm running. Please help me with this or give me other ways of doing this if they exists.

Sorry for the essay, but I wanted this to be useful for as many as possible because it's hard to find information on this.Last edited by conchyliferous on Sun Jun 19, 2005 12:17 pm; edited 1 time in total

----------

## kabage

your initrd commands need libs too (unless you compiled static)  like insmod modprobe

I personally would build all those modules into the kernel not as modules (build-in)  then you just need 

to map the devices                                               

dmsetup (from device-mapper application )  device mapper need to be enabled in kernel also

dmraid    comes to mind  

     (their should be a highpoint module in the kernel if not there some howtos on sticking it in there)

Now your on the right track heres a link to the best initrd (from scratch making I've seen)

https://forums.gentoo.org/viewtopic-t-244837-postdays-0-postorder-asc-highlight-hwsetup-start-0.html

go to creating the initrd section

oh that reminds me in the grub.conf you will need root=/dev/ram0  that is your initial ram disk starting point not the harddisk your harddisk partition would be

real_root=/dev/sda3   or whatever  and dont forget init=/linuxrc

Heres mine (I'm using dmraid thats why the weird device on my harddrive -- If your dualbooting thats the only way to go and I believe its compatible with highpoint *some of them anyway)

```
title=X-Dream Gentoo v7 new init UDEV

root=(hd0,7)

kernel /xdream-v7 root=/dev/ram0 real_root=/dev/mapper/pdc_ciiaceccai7 init=/linuxrc gentoo=udev

initrd /xdreaminit

```

----------

## cummings66

First off I think you're making it too hard on yourself.  You don't need to be so complex.  Things are so much easier if you just say no to genkernel.

Compile a new kernel, and in that kernel make sure you have the driver you need, do not make it a module.

Now once that's done, and I hope you didn't use genkernel for it, do it like this;

make menuconfig and select what you need.

make && make modules_install

mount /boot

cd arch/your type/boot might be something like cd /usr/src/linux/arch/i386/boot/ for example

cp bzImage /boot/kernel-2.6.11-gentoo-r9 (or whatever version you use)

edit the grub.conf as I have it here for example and then run grub

I do it manually per the install manual and keep the old kernel around so that if I screwed up something I can reboot into a working system instead of chrooting into it.

Here's a simple grub.conf that works from my scsi array.  Since you said yours uses scsi device names I think it would work ok for you as is.  You've got to use the hd0,0 bit because that's what grub expects, and later you use the root with the scsi device name that you actually used in fdisk for your root partition.

default 0

timeout 10

splashimage=(hd0,0)/boot/grub/splash.xpm.gz

title=Gentoo Linux (kernel-2.6.11-r9)

root (hd0,0)

kernel (hd0,0)/boot/kernel-2.6.11-gentoo-r9 root=/dev/sda3 splash=verbose

title=Gentoo Linux (kernel-2.6.11-r :Cool: 

root (hd0,0)

kernel (hd0,0)/boot/kernel-2.6.11-gentoo-r8 root=/dev/sda3 splash=verbose

----------

## conchyliferous

I have made some misunderstandings on this that's for sure.

I still don't know exactly in what order everything gets loaded.

First I thought that the initrd image was loaded first of all and if I wanted to load a 3rd part module in it I also needed all dependencies for that module. But that doesn't seam to be the case according to some googling.

The second misunderstanding was about how hardware raid works. The thing is that as soon as I have loaded the Highpoint module and got the sda devices I can fully consider the raid as a scsi disk. My box does not know the difference either. So searching for "root scsi initrd" instead if "root raid initrd" helps a lot. In fact almost everything you find if searching the net for linux and raid is regarding software raid.

I don't think I can make use of cummings66's solution because I need to load my module from Highpoint and I can't compile that into kernel. For that I need the initrd image, but as you say, my initial solution is unnecessary complex.

One more thing, the Highpoint thing you can compile into kernel is not the same module that I need. That is something else.

My second try:

What I did was that I didn't care about loading dependencies for the 'hpt374.ko' module from Highpoint. And the second thing I did was to read the manual for 'mkinitrd'.  :Rolling Eyes: 

So now I got (like before) ext3, scsi_mod and sd_mod compiled into kernel but I will not include and load them in the initrd. (I guess the error I got before when insmoding the modules had to do with that I had the modules compiled in kernel and loaded them with insmod at the same time)

And now I make my image with mkinitrd.

```
mkinitrd --with=hpt374 --fstab etc/fstab --builtin=scsi_mod --builtin=sd_mod --builtin=jbd --builtin=ext3  initrd.img 2.6.11-gentoo-r6
```

Where the fstab contains the following:

```

#SCSI (RAID)

/dev/sda1      /boot   ext3        defaults,noatime    1 2

/dev/sda2      none    swap        sw                  0 0

/dev/sda3      /       ext3        defaults,noatime    0 1

#IDE . doesn't matter anymore. Just put it somewhere. Ignore old swap for now

/dev/hda1       /IDE/boot   ext2        defaults,noatime    1 2

/dev/hda3       /IDE/       ext2        defaults,noatime    0 1

#cd and floppy

/dev/hdc        /mnt/dvdrw  auto        noauto,ro,user  0 0

/dev/hdd        /mnt/dvdr   auto        noauto,ro,user  0 0

/dev/fd0        /mnt/floppy auto        noauto          0 0

none            /proc       proc        defaults        0 0

none            /dev/shm    tmpfs       defaults        0 0

```

My /etc/modules.conf contains, among a lot of other things, the following line:

```
alias scsi_hostadapter hpt374
```

I can manually open the initrd file and check it's doing what it's supposed to do.

The image contains the hpt374.ko file and the linuxrc file is like this:

```
#!/bin/nash

echo "Loading hpt374.ko module"

insmod /lib/hpt374.ko

echo Mounting /proc filesystem

mount -t proc /proc /proc

echo Creating block devices

mkdevices /dev

echo Creating root device

mkrootdev /dev/root

echo 0x0100 > /proc/sys/kernel/real-root-dev

echo Mounting root filesystem

mount -o defaults,noatime --ro -t ext3 /dev/root /sysroot

pivot_root /sysroot /sysroot/initrd

umount /initrd/proc
```

And that looks alright, and since 'mkinitrd' did it and I haven't done it myself it should be right.  :Razz: 

I put this image in my /boot and configure my grub.conf like this:

```
timeout 63

default 0

title  GNU/Linux

root (hd0,0)

kernel /bzImage ro root=/dev/sda3 noapic

initrd /initrd.img
```

When I was going to reboot now I was optimistic, but of course I got an error on boot. Took a picture of the error message and here it comes:

 *Quote:*   

> 
> 
> .
> 
> .
> ...

 

As you see the image gets loaded and the linuxrc script executed. I'm not really sure about what that usb thing is doing there. The hpt374 seam to load alright but the root device cannot open.

Please help!

----------

## irondog

Why don'y you replace:

```
mkrootdev /dev/root

echo 0x0100 > /proc/sys/kernel/real-root-dev

echo Mounting root filesystem

mount -o defaults,noatime --ro -t ext3 /dev/root /sysroot 

```

By:

```
mknod sda3 b 8 3

echo Mounting root filesystem

mount --ro -t ext3 /dev/sda3 /sysroot

```

----------

## conchyliferous

I tried that but got another kernel panic:

 *Quote:*   

> 
> 
> .
> 
> .
> ...

 

The rest of it is like before.

Can't say I understand this. Is this better or worse?  :Razz:  Any other suggestions?

----------

## Unther

It's an improvement, it's the pivot_root command failing not the mount command.  :Very Happy: 

try replacing 

```
pivot_root /sysroot /sysroot/initrd 
```

with

```
cd /sysroot

pivot_root . initrd
```

At one point I did this for an install to a USB device. I just found that this was the way the pivot_root man page recommended, and it wroked for me.

You should probably change your grub config from

```
kernel /bzImage ro root=/dev/sda3 noapic 
```

to

```
kernel /bzImage ro root=/dev/ram0 noapic 
```

----------

## conchyliferous

Well that didn't make it better I'm afraid.

Just changing the 'pivot_root' thing resulted in this:

```
.

.

.

EXT3-fs: mounted filesystem with ordered data mode.

ERROR: failed in exec of cd

pivotroot: pivot_root(.,initrd) failed: 2

umount /initrd/proc failed: 2

ERROR: cd exited abnormally!

pivotroot: pivot_root(.,initrd) failed: 2

umount /initrd/proc failed: 2

VFS: Cannot open root device "sda3" or unknown-block(0,0)

Please append a correct "root=" boot option

Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) 
```

Then changing the kernel root option also had an even worse result:

```
.

.

.

RAMDISK: Compressed image found at block 0

VFS: Mounted root (ext2 filesystem) readonly.

Freeing unused kernel memory: 216k freed

Kernel panic - not syncing: No init found. Try passing init= option to kernel. 
```

Looks like linuxrc isn't read at all!  :Surprised:  Suggestions?

----------

## Unther

My bad, getting careless, and making assumptions from my setup. 

That should have read 

```
kernel /bzImage ro root=/dev/ram0 noapic init=linuxrc
```

 for the grub enetry.

You also need a directory to drop the old root into when you pivot_root in the real root filesystem, I called mine initrd. Create that.

You said you were able to get into the initrd file, what does it contain (Which executables)?

----------

## conchyliferous

That didn't work either. Still got that 'cd' problem but I also got another interesting error.

Check the following output, it's the same till the "mount /proc" thing:

```
.

.

.

RAMDISK: Compressed image found at block 0 

VFS: Mounted root (ext2 filesystem). 

Red Hat nash version 3.5.7 starting 

Loading hpt374.ko module 

hpt374: module license ´Proprietary´ taints kernel. 

RocketRAID 154x/1640 SATA Controller driver 

Version 2.13, Compiled May 9 2005 18:44:05 

usb 4-1: new full speed USB device using uhci_hcd and adress 2 

RAID5 write-back enabled 

ACPI: PCI interrupt 0000:02:03.0[A] -> GSI 11 (level, low) -> IRQ 11 

ACPI: PCI interrupt 0000:02:03.1[A] -> GSI 11 (level, low) -> IRQ 11 

scsi0 : hpt374 

Vendor: HPT3xx Model: RAID 0 Array Rev: 3.0 

Type: Direct-Access ANSI SCSI revision: 00 

SCSI device sda: 468883200 512-byte hdwr sectors (240068 MB) 

sda: asking for cache data failed 

sda: assuming drive cache: write through 

SCSI device sda: 468883200 512-byte hdwr sectors (240068 MB) 

sda: asking for cache data failed 

sda: assuming drive cache: write through 

sda: sda1 sda2 sda3 

Attached scsi disk sda at scsi0, channel 0, id 0, lun 0

Mounting /proc filesystem

Creating block devices

failed to create /dev/sda

failed to create /dev/sda1

failed to create /dev/sda2

failed to create /dev/sda3

Creating root device

mknod: failed to create sda3: 30

Mounting root filesystem

mount: error 2 mounting ext3

ERROR: failed in exec of cd 

pivotroot: pivot_root(.,initrd) failed: 2 

umount /initrd/proc failed: 2 

ERROR: cd exited abnormally! 

pivotroot: pivot_root(.,initrd) failed: 2 

umount /initrd/proc failed: 2 

Kernel panic - not syncing: Attempted to kill init!
```

The output is like before 

It looks like sda gets initialized alrigt but later it fails to load  :Surprised: 

And that cd problem is still there... Maybe that other way works since mkinitrd made it that way?

The little tux in me is about to give up and the big bad windows monster is thinking about how easy this is in windows where you just have to press F4 in the windows installation-program...

Please save me!  :Confused: 

----------

## cummings66

Try this in your grub.

Enter "pci=noacpi" or "acpi=off", in lower case.

----------

## conchyliferous

I'm sorry, can't do that.  :Confused: 

Gave up the whole 'boot from raid just like scsi'-idea, put far to many days on that. Put an old fashion IDE into the box and put / on that while /usr, /var, /opt, /tmp and /home on the scsi(raid). That way I still get the speed where I want it and have no problems booting.

But it would still be great if someone could point out to me what's the problem here. I will check in on this thread now and then and if someone says something that sounds lika a solution I could use that next time. Otherwise it would be good for others with the same problem...

----------

## cummings66

Well, I did a google search and a guy had the exact same problem you did.  He added those lines and it worked for him, thus my passing it along.

Sometimes things like this are tough to setup.  I used a second computer to test things like this and when it's working I backup the system and restore it on the hardware I want.

----------

## micah_death

I don't know if this 'caused a problem for you but when I compiled my raid stuff in my kernel, it reordered my devices according to the lspci list order =(

(all of mine are ide)

/dev/hda became /dev/hde and so on...

I did have a fix for it: (/etc/lilo.conf)

```

image=/boot/bzImage

  label=gentooraid

  read-only

  root=/dev/hdb3

  append="ide4=0x9800,0x9c02,5 ide5=0xa000,0xa402,5 ide6=0xac00,0xb002,11 ide7=0xb400,0xb802,11"

```

Don't know if it will help, but it fixed me =)

----------

## volumen1

Based on this error:

```

pivotroot: pivot_root(.,initrd) failed: 2

umount /initrd/proc failed: 2 

```

I think you should do this: 

```
mkdir /initrd
```

That is, just make an empty /initrd directory off of your root.  I was getting that same error and that fixed me right up.  I got the idea from here, http://gentoo-wiki.com/HOWTO_Install_Gentoo_on_an_LVM2_root_partition#Make_your_initrd

----------

