# KVM, mounting rootfs disk from within /init

## Wizumwalt

I'm running the following code in the /init script of my initramfs.

```

#!/bin/busybox sh

mount -t proc none /proc

mount -t sysfs none /sys

# do stuff

echo "my stuff here"

mount -o ro /dev/sda3 /mnt/root

umount /proc

umount /sys

#exec switch_root /mnt/root /sbin/init

exec /sbin/init < /dev/console > /dev/console 2>&1

```

And I get this error.

```

mount: mounting /dev/sda3 on /mnt/root failed: No such device or address

```

When I built my initramfs dir structure (in /usr/src/initramfs), I ran this line which includes the device to my / partition.

```

cp -a /dev/{null,console,tty,sda*} /usr/src/initramfs/dev/

```

Any ideas about what's wrong here?

Something else I noticed is that if I do ... if [ -f /dev/sda3 ] ... from within the /init just before the mount, it's says it's not there. But I archive it in the cpio which gets passed to initrd. But if I drop into a rescue shell after this error from within /init, I can cd to /dev and see /dev/sda3 is there.Last edited by Wizumwalt on Tue Nov 09, 2010 9:56 pm; edited 2 times in total

----------

## rh1

I'm assuming your using these instructions:

http://en.gentoo-wiki.com/wiki/Initramfs

I had issues until i used mdev, didn't have any luck with devtmpfs either. I thought it was just something i was doing wrong. 

Take a look at  mdev section , basically add this link

```
ln -s ../bin/busybox /usr/src/initramfs/sbin/mdev
```

Then add to /init

```

echo /sbin/mdev > /proc/sys/kernel/hotplug

mdev -s

```

Mine has worked like a charm ever since.

----------

## Wizumwalt

Yeah, I am using that very link.

I'm assuming I run the ln from within my initramfs /sbin. And I added the 'echo /sbin/mdev ...' line after just after I mounted proc and sysfs within the /init, archived the initramfs dir structure, ran it ... but it didn't change anything. I still get the same error.

----------

## rh1

I don't know then. 

 *Quote:*   

> But if I drop into a rescue shell after this error from within /init, I can cd to /dev and see /dev/sda3 is there

 

Can you mount it from the rescue shell or do you get the same error?

----------

## Corona688

Is your disk driver being detected by the kernel?  Just having the sda3 device file isn't enough, the device also has to actually exist  :Very Happy: 

----------

## Wizumwalt

 *Corona688 wrote:*   

> Is your disk driver being detected by the kernel?  Just having the sda3 device file isn't enough, the device also has to actually exist 

 

I'm booting the very same kernel that my host OS is using, and my host OS of course works.

 *rh1 wrote:*   

> 
> 
> Can you mount it from the rescue shell or do you get the same error?

 

I get the very same error when trying to mount from rescue shell. But I cd into /dev and the device is there.

And something else I just noticed, if I use devtmpfs instead of mdev, when I am in the rescue shell, /dev/sda* do not exists like they do w/ mdev. I used tmpfs just like in the gentoo wiki on initramfs.

Something I don't understand about /dev. When do the sdaN devices get created? Is this done by the kernel module for this hd? If so, then why does the gentoo initramfs wiki say to copy over the sda* devices into the directory structure. Something isn't right about these /dev/sda* devices I have and the kernel can't load them because of that. I don't understand this error ...

/dev/sda3 on /mnt/root failed: no such device or address

But if I execute this using devtmpfs instead of mdev in the /init, I get a slightly different error ...

/dev/sda3 on /mnt/root failed: no such file or directory

... and I do have /mnt/root in my initramfs dir struct

----------

## rh1

Just shooting from the hip here but did you remove the /dev/sda* files that you copied to your initramfs before using mdev. I don't have any /dev/sda in my initramfs.

----------

## Wizumwalt

No, I didn't remove them. I copied my host OS's / (/dev/sda3) into /opt/initramfs/dev, then archived it just like the wiki says. So now I'm confused. You say you don't have any, so you didn't follow the copy step of /dev/sda1 in the wiki into your ./initramfs/dev? 

If I use devtmpfs, and even though I've copied /dev/sda1 (/ is sda3 on my box), I don't see it when the rescue shell hits. And you saying you don't have it at all. And the wiki says to copy it, but why if it's not there? I'm a bit confused now.

After it falls into the rescue shell which is just after the error "mount: mounting /dev/sda3 on /mnt/gentoo failed: No such file or directory", here's what's mounted ...

```

/dev # cat /proc/mounts

rootfs / rootfs rw 0 0

none /proc proc rw,relatime 0 0

none /sys sysfs rw,relatime 0 0

none /dev devtmpfs rw,relatime,size=250884k,nr_inodes=62721,mode=755 0 0

```

Here's what I've been using to make this work ...

```

.

|-- bin

|   `-- busybox

|-- dev

|   |-- console

|   |-- null

|   |-- sda3

|   `-- tty

|-- etc

|-- init

|-- lib

|   `-- modules

|-- mk-initramfs.sh

|-- mnt

|   `-- gentoo

|-- proc

|   `-- sys

|       `-- kernel

|-- root

|-- sbin

|   `-- mdev -> ../bin/busybox

`-- sys

    `-- class

```

And here's my /init

```

#!/bin/busybox sh

rescue_shell() {

    echo "Something went wrong. Dropping into shell."

    busybox --install -s

    exec /bin/sh

}

mini_udev() {

    echo /sbin/mdev > /proc/sys/kernel/hotplug

    mdev -s

}

mount -t proc none /proc

mount -t sysfs none /sys

#mini_udev

mount -t devtmpfs none /dev

if [ -f /dev/sda3 ]; then

    echo "/dev/sda3 exists"

else

    echo "not there"

fi

if [ -f /mnt/gentoo ]; then

    echo "/mnt/gentoo exists"

else

    echo "/mnt/gentoo not there"

fi

# mount the root filesystem 

mount -o ro /dev/sda3 /mnt/gentoo || rescue_shell

# clean up

umount /dev

umount /proc

umount /sys

exec switch_root /mnt/root /sbin/init

#exec /sbin/init < /dev/console > /dev/console 2>&1

```

When I run this, the if statements above always print "not there" and "mnt/gentoo not here". But those files do exists as shown in my dir structure.

----------

## frostschutz

What was wrong with the replies to your previous thread on the same matter?

https://forums.gentoo.org/viewtopic-t-848654.html

 *Wizumwalt wrote:*   

> I'm booting the very same kernel that my host OS is using, and my host OS of course works.

 

But the guest has to have access to the drive, the way you stated it in your previous thread it wasn't configured with any drives.

----------

## rh1

 *Quote:*   

> You say you don't have any, so you didn't follow the copy step of /dev/sda1 in the wiki into your ./initramfs/dev? 
> 
> 

 

I did follow those instuctions but when i couldn't get it to work, i switched to mdev. When i switched i removed /dev/sda* from my initramfs. I can't remember why but i think it was causing me an issue. I left the rest of the /dev/* stuff. 

 *Quote:*   

> When I run this, the if statements above always print "not there" and "mnt/gentoo not here". But those files do exists as shown in my dir structure.

 

Well part of the problem is they aren't files. At least /mnt/gentoo definately isn't. "-f" test if it is a file.

 *Quote:*   

> -d filename Returns True if file, filename is a directory. 
> 
> -f filename Returns True if file, filename is an ordinary file. 
> 
> -r filename Returns True if file, filename can be read by the process. 
> ...

 

----------

## Wizumwalt

 *frostschutz wrote:*   

> But the guest has to have access to the drive, the way you stated it in your previous thread it wasn't configured with any drives.

 

Ah, I think I understand what your saying now. I just thought an ide interface was the drive by default, at least that's what I was understanding from the docs. But I think that is wrong.

Can you better recommend what should be on the command line for a serial attached scsi (SAS) drive? Or how to get the info needed for I think ... 

-drive file=/dev/sda3,if=scsi,bus=0,unit=6 

 *rh1 wrote:*   

> 
> 
> Well part of the problem is they aren't files. At least /mnt/gentoo definately isn't. "-f" test if it is a file.
> 
> 

 

Thanks, I missed that too.

----------

## frostschutz

I get the best performance with -drive file=/dev/device,cache=none,if=virtio as suggested by http://www.linux-kvm.org/page/Tuning_KVM

will appear as /dev/vda in the VM

I actually partition that and use grub inside the VM, I have no experience with passing through single partitions and booting kernel directly through KVM.

If I had to do that it'd probably actually leave it that way and just put the filesystem on vda directly, without any partitions. Linux isn't picky about device names.

----------

## Wizumwalt

 *frostschutz wrote:*   

> I get the best performance with -drive file=/dev/device,cache=none,if=virtio as suggested by http://www.linux-kvm.org/page/Tuning_KVM
> 
> 

 

I can't seem to get this to work. Is device your hd (sda), or a partition (sda3) or what? I've tried several things, just keep getting the same error.

Does CONFIG_VIRTIO_PCI need to be enabled in my 2.6.34-r6 kernel for this to work?

Here's the line I've been changing up ...

```

kvm -net nic,macaddr=52:54:00:12:34:56 -net tap,ifname=qtap0,script=no,downscript=no -kernel ./kernel-2.6.34-gentoo-r6 -initrd /opt/gentoo-x86_64-initramfs.cpio.gz -boot d -m 512 -drive file=/dev/sda3,cache=none,if=virtio

```

----------

## frostschutz

In my case it's an LVM volume, but it shouldn't matter. The guest has to support virtio disks, yes. Also the guest won't see it as sda3 but as vda then.

You can leave out the if=virtio, then it'll probably be IDE hda or something...

----------

## Hu

 *Wizumwalt wrote:*   

> 
> 
> ```
> exec /sbin/init < /dev/console > /dev/console 2>&1
> ```
> ...

 This is probably not what you want.  Without a switch root, this will look for a secondary init in the initramfs.

 *Wizumwalt wrote:*   

> 
> 
> ```
> mount -o ro /dev/sda3 /mnt/root
> ```
> ...

 You have attempted to access a device node for which no driver is registered.  This could mean that you statically copied a device without building into the kernel a driver that uses that device number, or that all such drivers built in have elected not to use that device number.

 *Wizumwalt wrote:*   

> When I built my initramfs dir structure (in /usr/src/initramfs), I ran this line which includes the device to my / partition.
> 
> ```
> cp -a /dev/{null,console,tty,sda*} /usr/src/initramfs/dev/
> ```
> ...

 You do not need to build an initramfs like that.  The kernel supports building an initramfs from a manifest file, which has the useful secondary benefit that you can construct the initramfs without needing root privilege on the build system.

 *Wizumwalt wrote:*   

>  *Corona688 wrote:*   Is your disk driver being detected by the kernel?  Just having the sda3 device file isn't enough, the device also has to actually exist  I'm booting the very same kernel that my host OS is using, and my host OS of course works.

 In that case, you most likely do not have the driver you need.  Very few hypervisors have a sufficient diversity of emulated disks that they will present an emulated disk of the same model as your real hardware.  This is one of the reasons that it is often not worthwhile to try to reuse the same kernel on the host and the guest.

 *Wizumwalt wrote:*   

>  *rh1 wrote:*   Can you mount it from the rescue shell or do you get the same error? I get the very same error when trying to mount from rescue shell. But I cd into /dev and the device is there.

 Yes, the device node will be there because you statically placed it there in your initramfs /dev and the initramfs is not equipped to remove inappropriate device nodes.

 *Wizumwalt wrote:*   

> And something else I just noticed, if I use devtmpfs instead of mdev, when I am in the rescue shell, /dev/sda* do not exists like they do w/ mdev.

 When you use devtmpfs, you see the devices that the kernel knows about and can use.  Although you now have mdev in the system, it will not remove inappropriate statically placed device nodes.

 *Wizumwalt wrote:*   

> When do the sdaN devices get created? Is this done by the kernel module for this hd? If so, then why does the gentoo initramfs wiki say to copy over the sda* devices into the directory structure. Something isn't right about these /dev/sda* devices I have and the kernel can't load them because of that. I don't understand this error ...

 The devices get created when a user program creates them (except when using devtmpfs, in which case the kernel handles it for you).  However, the existence of the device node is not synonymous with having a usable device.  In a fully functional system with udev running, udev will create device nodes when it discovers that the kernel has a device.  The kernel will only discover that device if it has a driver for the device.

The Gentoo initramfs wiki tells you to copy over the nodes because that is a quick and dirty way to prime the initramfs that will work, if you configure your kernel with the right device driver.  Using devtmpfs is cleaner, but the wiki may have been written before devtmpfs was popular.

 *Wizumwalt wrote:*   

> But if I execute this using devtmpfs instead of mdev in the /init, I get a slightly different error ...
> 
> /dev/sda3 on /mnt/root failed: no such file or directory
> 
> ... and I do have /mnt/root in my initramfs dir struct

 You also said that with devtmpfs, you do not get /dev/sda3, so it is consistent that you would be unable to open it.

----------

## frostschutz

 *Hu wrote:*   

> Using devtmpfs is cleaner, but the wiki may have been written before devtmpfs was popular.

 

That, and devtmpfs alone doesn't really help. If you expect the device name to change you also have to be able to pass kernel parameters to the initramfs so it knows which device to mount. This is also covered in the Wiki but it makes things way more complicated than they need to be if the device name is known and static, which is usually the case on a Desktop. The Wiki starts out with a minimalistic no frills example on purpose so people have a better chance to understand how it works first, rather than get overwhelmed with an overly complicated script right from the start. I made it this way because most other initramfs guides turned me off by just plastering a huge script there without actually explaining anything.

One thing no one has pointed out so far and which I'd like to do now: I don't see anything in the OPs initramfs script that would actually warrant using initramfs. If you're not doing anything special (such as encrypted root partition, root on LVM, etc.) then there is absolutely no need to use initramfs since the kernel can just mount the root partition directly by itself. If you don't need initramfs, not using it is the best thing you can do, since even the most simple variant of initramfs just adds unnecessary complexity to the matter.

----------

## Wizumwalt

Oh man, I keep asking for advice and get suggestions to go one route, only to find out it's not the way I should be doing things. This is getting so frustrating.

I'm just trying to develop an LKM on my host OS, compile it there, then copy the modules over to my initramfs, archive it, boot my guest OS (should be fine since it's running the same version kernel as the host OS that compiled the module), and test comm between it and another networked application on the host.

 *Hu wrote:*   

> This could mean that you statically copied a device without building into the kernel a driver that uses that device number, or that all such drivers built in have elected not to use that device number.
> 
> 

 

I don't get this part. What other driver do I need, or how do I tell it a device number. I think my kernel already has the support because this kernel already knows how to work w/ this SAS hard drive. I don't understand what I'm missing, or what device number goes where.

It seems like your saying don't worry about having the same kernel version on my host OS as is on the guest OS? Then ... what should I be doing? Getting a guest OS running and doing development there? (Seems like that would suck because I'd be restarting it so much and losing my dev environment settings, not to mention the work in putting in a proper dev environment, which I already have on my host os).

 *hu wrote:*   

> 
> 
> You do not need to build an initramfs like that. 
> 
> 

 

Ok, again, really getting confused here. There's a million ways to do anyone thing in gentoo. Don't tell me it's wrong after so many suggestions this is how it needs to be done, please tell my what and why I need to do it differently. I don't mean to sound like a jerk, and apologies before hand, but this is making me so frustrated.

I'm doing this based on the idea that when developing an LKM, it seems like I'd want the same kernel version I just compiled it with. Compile on the host, move the *.ko's into the initramfs, have /init load the modules when booting the guest OS and see how all the upfront initialization goes, etc... Otherwise I may be chasing issues that don't appear in a different kernel version.

Is this not the way kernel mod developers normally work. I'd be interested in hearing how mods should be properly done. Even, how I should be doing this instead of initramfs now.

 *frostschutz wrote:*   

> One thing no one has pointed out so far and which I'd like to do now: I don't see anything in the OPs initramfs script that would actually warrant using initramfs. If you're not doing anything special (such as encrypted root partition, root on LVM, etc.) then there is absolutely no need to use initramfs since the kernel can just mount the root partition directly by itself. If you don't need initramfs, not using it is the best thing you can do, since even the most simple variant of initramfs just adds unnecessary complexity to the matter.

 

Maybe this is the answer then that I don't need this. But still not sure what I do need. All I can say is what I'm trying to do, mod dev w/ a few user space apps that talk to the host OS, and have them store data onto a file system (doesn't need to be the hd). I should also add a *small* part of my reasoning for doing initramfs was that I wanted to learn it as well.

----------

## frostschutz

 *Wizumwalt wrote:*   

> I think my kernel already has the support because this kernel already knows how to work w/ this SAS hard drive.

 

In a virtualized environment, the guest OS will see a virtual disk controller. It does not matter whether it's a SAS drive or a floppy disk or a file in a filesystem on the host, the guest will see it the way it's emulated, which may either be a virtio disk (gives best performance), a plain old fashioned IDE controller (best compatibility with guest OSes), or something else. So what your guest kernel needs to support is not the SAS hard drive but the virtual disk controller you're using. The device file alone doesn't give you access to any disks, first and foremost it has to be detected by the kernel. The device file is only the interface to user space so to speak...

Also you only need initramfs if the kernel can't get at the root partition by itself. Which is only the case if it's LVM or encrypted or something similar obscure. If your root partition is a plain partition and filesystem, skip the initramfs part entirely, just use the root= kernel parameter instead. Which also will only work if the kernel actually detects the drive in question.

 *Wizumwalt wrote:*   

> I'm doing this based on the idea that when developing an LKM

 

Forgive me for saying so, but if you're developing a kernel module, you shouldn't be having this problem.

If I may ask, what kind of kernel module is it? It can't be hardware related, otherwise you couldn't do your testing in a VM. (What I'm asking here is: does it have to be done in kernel space at all?)

----------

## frostschutz

 *Wizumwalt wrote:*   

> I should also add a *small* part of my reasoning for doing initramfs was that I wanted to learn it as well.

 

Maybe you're trying to do too many things all at once. At my first dib at KVM, I personally was dancing my happy joy dance when I got plain old standard Debian to run.

----------

## Wizumwalt

 *frostschutz wrote:*   

>  *Wizumwalt wrote:*   I should also add a *small* part of my reasoning for doing initramfs was that I wanted to learn it as well. 
> 
> Maybe you're trying to do too many things all at once. At my first dib at KVM, I personally was dancing my happy joy dance when I got plain old standard Debian to run.

 

Well, I already got a gentoo livecd to boot w/ the following. But it took quite a while to boot (a few mins) w/ all the probing. So then I started trying to get a kernel closer to what I was using and not so bloated.

```

kvm -hda ./gentoo-i386.img -cdrom ./livecd-amd64-installer-2008.0.iso -boot d

```

 *frostschutz wrote:*   

> 
> 
> Forgive me for saying so, but if you're developing a kernel module, you shouldn't be having this problem.
> 
> If I may ask, what kind of kernel module is it?

 

It's ok, I'm new to kernel modules as well, one more thing I want to learn. It's not that it has real purpose in the kernel, it's my own playground type protocol that I've been working with. It's just another excuse for me to learn how to do comm inside the kernel.

----------

## Hu

 *Wizumwalt wrote:*   

> It seems like your saying don't worry about having the same kernel version on my host OS as is on the guest OS? Then ... what should I be doing? Getting a guest OS running and doing development there? (Seems like that would suck because I'd be restarting it so much and losing my dev environment settings, not to mention the work in putting in a proper dev environment, which I already have on my host os).

 I said you should not attempt to reuse the same kernel in the guest.  I did not say that it cannot be the same version of the kernel.  The distinction lies in the .config used to build the two kernels.  For example, you and I might both be running 2.6.34 right now, but it is almost certain that we have elected different options in our respective .config files, due to different hardware, different filesystems, network protocols, etc.  In such a case, I would say that we are running different kernels, even though we may have started with exactly the same kernel source code.

 *Wizumwalt wrote:*   

>  *hu wrote:*   You do not need to build an initramfs like that.  Ok, again, really getting confused here. There's a million ways to do anyone thing in gentoo. Don't tell me it's wrong after so many suggestions this is how it needs to be done, please tell my what and why I need to do it differently.

 Please try to get my name right.

I did not say this was wrong, but I do think the method you used to build the initramfs is more trouble than using the kernel's native support for generating an initramfs.  If what you have works, leave it for now.

----------

## Wizumwalt

Thanks for putting the time in to reply. This is very helpful. I feel better now, and kernel versions make sense to me once again. So it looks like my only problem is trying to get virtio installed correctly in my kernel. I had followed this link …

http://www.linux-kvm.org/page/Virtio

… theres a list of CONFIG_ options that I had enabled from that link in my kernel but I guess I still don't have the right ones for my virtual disk controller. So still trying to figure out what needs to go in the kernel for my virtual disk controller.

----------

## Hu

 *Wizumwalt wrote:*   

> So still trying to figure out what needs to go in the kernel for my virtual disk controller.

 If you place your latest attempt in a pastebin, someone may be able to point out any mistakes or missing features.  If you have changed from the kvm command line you mentioned earlier in the thread, it would be good to post what you are using now, so that whoever tunes your .config can test it locally first.

----------

## Wizumwalt

Ok, here's my latest attempt in a pastebin.  If anyone see's anything I'm doing wrong in trying to get the guest OS to boot, any help much appreciated. And if more info is needed, I'll be happy to post.

http://pastebin.com/vdDX9qnv

----------

## Hu

You set the guest to use a virtio interface, but you are still trying to treat the guest hard drive as a SCSI device (/dev/sda3).  If you use a virtio controller, you need to use the virtio driver and the virtio device node.

----------

## Wizumwalt

Ok, I think I am using the correct virtio controller. Here's another latest attempt, added a bit more information.

http://pastebin.com/NGPwhesu

----------

## Hu

You switched to a virtio controller, then commented it out and switched to NFS -- but you never brought up the network!  As a Network File System, NFS requires that the network be up to be able to mount remote filesystems.  Why are you trying to use NFS before you even have a local disk working?  Also, you are now trying to umount /dev, but you no longer mount anything on it, so there is nothing to umount.

----------

## Wizumwalt

I got some other advice that said it was a bad idea to mount the disk in this way and that I should be using NFS instead.

So I've gone back to trying to mount /dev/vda in /init and removed the umount to /dev. I couldn't get that far before, so didn't worry about it. I kind of thought my networking was up, and don't see anything wrong with it yet, though I may try VDE once I can get the guest to boot all the way. But since I'm back to trying to mount the disk, I don't need it for that, and I don't understand why I'm getting the 'No such file or directory' error.

```

...

[0.801975] Freeing unused kernel memory: 1624k freed

[0.899050] busybox used greatest stack depth: 5944 bytes left

[0.899050] exe used greatest stack depth: 5856 bytes left

doing stuff ...

mount: mounting /dev/vda on /mnt/gentoo failed: No such file or directory

Something went wrong. Dropping into shell.

/bin/sh: can't access tty; job control turned off

[0.993766] input: ImExPS/1 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2

...

[0.999232] mdev used greatest stack depth: 5840 bytes left

[1.774058] async/1 used greatest stack depth: 4912 bytes left

```

The /mnt/gentoo does exist in my initramfs dir structure and I can see it from the rescue shell.

KVM cmd looks like this ... 

```

kvm -net nic,macaddr=52:54:00:12:34:56,model=virtio -net tap,ifname=qtap0,script=no,downscript=no -kernel ./kernel-2.6.34-gentoo-r12 -initrd /opt/gentoo-x86_64-initramfs.cpio.gz -append "root=/dev/vda" -m 512 -drive file=./gentoo-i386.img,if=virtio,cache=none

```

----------

## Hu

 *Wizumwalt wrote:*   

> I kind of thought my networking was up, and don't see anything wrong with it yet

 Why would it be up?  Your init script is the first thing to run, and you do not bring it up.

 *Wizumwalt wrote:*   

> I don't understand why I'm getting the 'No such file or directory' error.

 Do you have a /dev/vda device node at the time that mount command runs?  The last init script you showed was not mounting a devtmpfs, and your comments in the post I am quoting suggest that you have not changed that.

----------

## Wizumwalt

Ok, past the mounting issue, turns out there was one more CONFIG_ option needed in the kernel. But I can't really test as I've got a switch_root error. Any insight into this error. I've got /sbin/init in the contents of my initramfs as shown below.

```

[] busybox used greatest stack depth: 5960 bytes left

before mount ...

after mount ...

switch_root: can't execute '/sbin/init': No such file or directory

[] busybox used greatest stack depth: 4360 bytes left

[] Kernel panic - not syncing: Attempted to kill init!

[] Pid: 1, comm: busybox Not tainted 2.6.34-gentoo-r12 #7

[] Call Trace:

[] [<ffffffff81564f3b>] panic+0x9e/0x119

```

Contents of /sbin in initramfs.

```

$ tree sbin

sbin

|-- init -> ../bin/busybox

`-- mdev -> ../bin/busybox

```

Here's my /init.

```

#!/bin/busybox sh

#

#

rescue_shell() {

    echo "dropping into shell."

    busybox --install -s

    exec /bin/sh

}

mini_udev() {

    echo /sbin/mdev > /proc/sys/kernel/hotplug

    mdev -s

}

mount -t proc none /proc

mount -t sysfs none /sys

#mini_udev

mount -t devtmpfs none /dev

# mount the root filesystem

echo "before mount ..." 

mount -o ro /dev/vda /mnt/gentoo || rescue_shell

echo "after mount ..."

# clean up

umount /proc

umount /sys

# boot the real thing

exec switch_root /mnt/gentoo /sbin/init

```

----------

## frostschutz

The sbin/init doesn't have to be in the initramfs, but on the partition you're switch rooting to.

----------

## Wizumwalt

I mounted the fs on my block device to the mount point of my initramfs cpio archive /mnt/gentoo. Then I try to switch_root to that mount point /mnt/gentoo. I have an init on both the initramfs /sbin and on the /sbin of the hd. And /sbin is on the / partition which *I think* /dev/vda is mounting in my /init script. So still not sure what's going on here.

An update ... I just added an 'ls' comand just after the line ...

```
 

mount -o ro /dev/vda /mnt/gentoo || rescue_shell

ls -l /mnt/gentoo

```

... of my /init script, and all I'm seeing is the output listing of a lost+found directory when I try to boot the guest OS. So although something seems like it's being mounted, I'm guessing /dev/vda isn't the correct device. My / fs partition is on /dev/sda3. Also tried /dev/vda1-5 and just ended up w/ "no such file or directory".

And if I add 'ls -l /dev/v*', the only virtio device is vda in /dev of the initramfs.

----------

## Hu

You have gone through several iterations, so it is a bit unclear how all the pieces are put together now.  Please restate, in a single post, the current configuration:KVM command lineContents of the host block devices you give to KVMFull text of the init scriptOutput of find / as run by the init script after it mounts the planned rootfsPutting any of the longer bits in a pastebin is fine.  I just want to get everything restated so we can reproduce your setup in full.

----------

## Wizumwalt

KVM command line ...

```

kvm -net nic,macaddr=52:54:00:12:34:56,model=virtio -net tap,ifname=qtap0,script=no,downscript=no -kernel ./kernel-2.6.34-gentoo-r12 -append "root=/dev/vda" -initrd /opt/gentoo-x86_64-initramfs.cpio.gz -m 512 -drive file=./gentoo-i386.img,if=virtio,cache=none,boot=on

```

/init script ...

```

#!/bin/busybox sh

#

#

rescue_shell() {

    echo "Something went wrong. Dropping into shell."

    busybox --install -s

    exec /bin/sh

}

mini_udev() {

    echo /sbin/mdev > /proc/sys/kernel/hotplug

    mdev -s

}

mount -t proc none /proc

mount -t sysfs none /sys

#mini_udev

mount -t devtmpfs none /dev

echo "before mount ..."

# mount the root filesystem 

mount -o ro /dev/vda /mnt/gentoo || rescue_shell

find /

echo "after mount ..."

# clean up

umount /proc

umount /sys

# boot the real thing

exec switch_root /mnt/gentoo /sbin/init

```

I didn't know how to capture the output of the 'find /' in the above script as it put out quite a lot that I couldn't scroll up and see, and it didn't switch_root properly. But I do know it was the contents of /proc, /sys, and /devices that I could see from the guest OS ... that scrolled by for 15-20 seconds.

What do you mean by the contents of the host block devices?

----------

## Hu

If you run mount -t auto -o ro,loop gentoo-i386.img /mnt/gentoo on the host, do you see the guest's filesystem appear at /mnt/gentoo?

----------

## Wizumwalt

After running your mount command above ...

```

$ mount | grep gentoo

/dev/loop0 on /mnt/gentoo type ext2 (ro)

$ ls -l /mnt/gentoo

total 12

drwx------ 2 root root 12288 Oct 7 07:49 lost+found

```

I can't remember how I had created this img, once I used 'qemu-img create', but I think I've since changed that up to a raw image and it's about 32G in size. It seems basically empty. I read that it's supposed to store the contents of the emulated hard disk. So I'm guessing all my write data to the read-only mounted / fs is written to this disk image? Let me know if I'm using it wrong.

----------

## Hu

You cannot write to a read-only filesystem.  That would defeat the point.

Based on the latest output, it seems everything is working as normal.  Your guest fails to boot because you have given it a hard disk that has no /sbin/init (or anything else) in it to execute.  You will need to make the guest boot a LiveCD image and install something to the virtual block device, or take advantage of the loop mount to have the host populate it with a bootable environment.  Either way, I suggest you switch it to something a little more recent than ext2.  :Smile: 

----------

## Wizumwalt

 *Hu wrote:*   

> You will need to make the guest boot a LiveCD image and install something to the virtual block device, or take advantage of the loop mount to have the host populate it with a bootable environment.  Either way, I suggest you switch it to something a little more recent than ext2. 

 

If I'm mounting a livecd, then what's the purpose of mounting a read-only filesystem ( / ) if it's being replaced for a livecd? I thought mounting the / fs was so that I could at least read stuff like /sbin/init and other executables on / and write to a tmpfs. 

When the docs I read say "install an operating system to the disk image", does that just mean, start the guest OS and pass a disk image file to it on the command line, something like a livecd? 

I've read a bit about creating a loop device, but … what's involved with populating it w/ a bootable environment? Is that like creating an initramfs dir structure and placing anything and everything I plan on using in it?

When I browse the mirror sites, the latest livecd I see is livecd-amd64-installer-2008.0.iso. Is 2008 the latest livecd?

----------

## Hu

Where did I say anything about mounting a read only filesystem and replacing it with a LiveCD?  I told you to use a LiveCD so that you can get a Linux environment in which to install a Gentoo system to the virtual block device - just as you would if you were installing Gentoo to new physical hardware.  You seem to be making this much more complicated than necessary by treating the whole endeavor like it was very different from installing Gentoo on a new machine.

Yes, start a LiveCD in the guest, create partitions (or LVM) on the virtual block device, make filesystems on those partitions, etc.  Follow the handbook in every respect, except that you can skip burning an actual disc since the hypervisor can read the ISO file and make it look like a CD drive.

If you use a raw disk image, you can loop mount it in the host and install files to it.  This can be more efficient than going through the virtualization layer, but it takes you down yet another path.  I suggest sticking with the route that is almost working.

----------

## Wizumwalt

I have an image file called gentoo-i386.img that I mounted to /mnt/gentoo and did a complete gentoo install on. This file has the following partitions in it (loop1 is /boot, loop2 is swap, and /dev/loop3 is the / fs). Like I said, it was basically straight from the handbook and I wasn't sure if I would need some of these.

```

$ sudo losetup -a

/dev/loop0: [0811]:14655814 (/home/dev/projects/kvm/gentoo-i386.img)

/dev/loop1: [0005]:1090 (/dev/loop0), offset 32256

/dev/loop2: [0005]:1090 (/dev/loop0), offset 41126400

/dev/loop3: [0005]:1090 (/dev/loop0), offset 583994880

```

Then I start KVM w/ this image file in the -drive option.

```

kvm -net nic,macaddr=52:54:00:12:34:56,model=virtio -net tap,ifname=qtap0,script=no,downscript=no -kernel ./kernel-2.6.34-gentoo-r12 -append "root=/dev/vda" -initrd /opt/gentoo-x86_64-initramfs.cpio.gz -m 512 -drive file=./gentoo-i386.img,if=virtio,cache=none,boot=on,format=raw

```

But I get the following results ...

```

[...] busybox used greatest stack depth: 5960 bytes left

before mount ...

[...] EXT3-fs (vda): error: no journal found

mount: mounting /dev/vda on /mnt/gentoo failed: Invalid argument

[...] exe used greatest stack depth: 5928 bytes left

Something went wrong. Dropping into shell.

/bin/sh: can't access tty; job control turned off

/ #[...] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input2

[...] async/1 used greatest stack depth: 4912 bytes left

```

And here's my initramfs /init file ...

```

rescue_shell() {

    echo "Something went wrong. Dropping into shell."

    busybox --install -s

    exec /bin/sh

}

mini_udev() {

    echo /sbin/mdev > /proc/sys/kernel/hotplug

    mdev -s

}

mount -t proc none /proc

mount -t sysfs none /sys

#mini_udev

mount -t devtmpfs none /dev

# Mount the root filesystem.

echo "before mount ..."

mount -o ro /dev/vda /mnt/gentoo

init="/sbin/init"

if [ -x "/mnt/gentoo/${init}" ]; then

    echo "after mount ..."

    umount /proc

    umount /sys

    exec switch_root /mnt/gentoo /sbin/init

fi

# Apprently, we failed somewhere's with mount.

rescue_shell

```

Here's some extra info that might help.

```

$ sudo mount /dev/loop3 /mnt/gentoo/

$ ls -l /mnt/gentoo

total 96

drwxr-xr-x  2 root root  4096 Nov  8 16:38 bin

drwxr-xr-x  2 root root  4096 Nov  8 12:04 boot

drwxr-xr-x 10 root root 36864 Oct 20 22:47 dev

drwxr-xr-x 42 root root  4096 Nov  8 16:43 etc

drwxr-xr-x  2 root root  4096 Oct 20 20:38 home

lrwxrwxrwx  1 root root     5 Nov  8 14:22 lib -> lib64

drwxr-xr-x  3 root root  4096 Nov  8 16:37 lib32

drwxr-xr-x  8 root root  4096 Nov  8 16:38 lib64

drwxr-xr-x  4 root root  4096 Oct 20 22:46 mnt

drwxr-xr-x  2 root root  4096 Oct 20 20:38 opt

drwxr-xr-x  2 root root  4096 Oct 20 20:37 proc

drwx------  2 root root  4096 Nov  8 16:45 root

drwxr-xr-x  2 root root  4096 Nov  8 16:38 sbin

drwxr-xr-x  2 root root  4096 Oct 20 20:38 sys

drwxrwxrwt  2 root root  4096 Nov  8 16:39 tmp

drwxr-xr-x 13 root root  4096 Nov  8 14:33 usr

drwxr-xr-x 12 root root  4096 Nov  8 16:34 var

$ cat /proc/mounts | grep loop3

/dev/loop3 /mnt/gentoo ext3 rw,relatime,errors=continue,data=writeback 0 0

```

----------

## Hu

 *Wizumwalt wrote:*   

> I have an image file called gentoo-i386.img that I mounted to /mnt/gentoo and did a complete gentoo install on. This file has the following partitions in it (loop1 is /boot, loop2 is swap, and /dev/loop3 is the / fs).
> 
> Then I start KVM w/ this image file in the -drive option.
> 
> ```
> ...

 You partitioned the virtual drive, but then tried to boot it like it was not partitioned.  You need to change your root= directive to reflect the fact that vda is partitioned.  You probably want root=/dev/vda3. *Wizumwalt wrote:*   

> 
> 
> ```
> 
> init="/sbin/init"
> ...

 You have a redundant set of slashes here.  It is harmless, but not useful.  Also, you should unmount your devtmpfs before proceeding.  It may work without unmounting, but it is good practice to unmount anything you mounted and no longer need. *Wizumwalt wrote:*   

> 
> 
> ```
> $ cat /proc/mounts | grep loop3
> ```
> ...

 This can be written more simply as grep loop3 /proc/mounts.  It is almost never appropriate to use cat to read a file just to pipe it to another program.

----------

## Wizumwalt

Thanks for the other two comments.

 *Hu wrote:*   

> You partitioned the virtual drive, but then tried to boot it like it was not partitioned.  You need to change your root= directive to reflect the fact that vda is partitioned.  You probably want root=/dev/vda3.

 

I've tried /dev/vda3 as well as other partitions, but I get the very same results as before (same output as /dev/vda).

----------

## Hu

You did not mention that before.  What device nodes are created for you in the guest devtmpfs?

----------

## Wizumwalt

I went into my rescue shell after trying to boot the guest OS and I see that they are all there ... /dev/vda, /dev/vda1, /dev/vda2, /dev/vda3

So I also changed /dev/vda to vda3 in the mount line of the /init script of my initramfs as well and it's now able to boot. But this time when I went in to check the device nodes, nothing appeared for /dev/vd* ...  this is still the line I'm using to boot the guest OS with, so I'm assuming it's this kernel in the -kernel param that is being used and not the one on my /dev/vda1 (/boot) partition (which I'm not mounting anyways). Just a side note: when I built my gentoo-i386.img I did it as ...

/dev/loop0p1 = /dev/vda1 = /boot

/dev/loop0p2 = /dev/vda2 = swap

/dev/loop0p3 = /dev/vda3 = /

Exactly as the gentoo handbook says, but it would seem I'm not making use of the first two. And still using this line to start it up.

```

kvm -net nic,macaddr=52:54:00:12:34:56,model=virtio -net tap,ifname=qtap0,script=no,downscript=no -kernel ./kernel-2.6.34-gentoo-r12 -append "root=/dev/vda3" -initrd /opt/gentoo-x86_64-initramfs.cpio.gz -m 512 -drive file=./gentoo-i386.img,if=virtio,cache=none,boot=on

```

So now I see this when booting (not sure how to capture the entire boot output) ...

```

/lib64/udev/write_root_link_rule: line 26: /dev/.udev/rules.d/10-root-link.rules: No such file or directory

/lib64/udev/write_root_link_rule: line 26: /dev/.udev/rules.d/10-root-link.rules: No such file or directory

/lib64/udev/write_root_link_rule: line 26: /dev/.udev/rules.d/10-root-link.rules: No such file or directory

 * Starting udevd ...

udevd[871]: failed to create queue file: No such file or directory

udevd[871]: error creating queue file

No /sbin/udevd found running; none killed.

 * Mounting devpts at /dev/pts ...

 * Checking root filesystem ...

fsck.ext3: No such file or directory while trying to open /dev/loop0p3

/dev/loop0p3:

The superblock could not be read or does not describe a correct ext2 filesystem. If the device is valid and it really contains an ext2 filesystem (and not swap or ufs or something else), then the superblock is corrupt, and you might try running e2fsck with an alternate superblock:

   e2fsck -b 8193 <device>

 * Filesystem couldn't be fixed :(

Give root password for maintenance

(or type Control-D to continue):

```

So I mounted my gentoo-i386.img onto my host and changed /etc/fstab from /dev/loop0p1 to /dev/vda1, and the same for the vda2, vda3 to see if this was it. Still got the same results. So not sure what device should go in /etc/fstab of the guest os image.

If I give the root password, I can log into the / filesystem and it looks ok, although it's read-only which is what I have in my /init. Also, I'm thinking it's ok to mount the / filesystem in the guest OS as rw since it's a seperate image and not the host OS?

Here's my latest /init.

```

#!/bin/busybox sh

#

#

rescue_shell() {

    echo "Something went wrong. Dropping into shell."

    busybox --install -s

    exec /bin/sh

}

mini_udev() {

    echo /sbin/mdev > /proc/sys/kernel/hotplug

    mdev -s

}

mount -t proc none /proc

mount -t sysfs none /sys

#mini_udev

mount -t devtmpfs none /dev

# Mount the root filesystem.

mount -o ro /dev/vda3 /mnt/gentoo

#mount -o ro /dev/vda1 /mnt/gentoo/boot

init="/sbin/init"

echo "init = ${init}"

if [ -x "/mnt/gentoo${init}" ]; then

    umount /proc

    umount /sys

    exec switch_root /mnt/gentoo /sbin/init

fi

rescue_shell

```

And here's the guests /etc/fstab in gentoo-i386.img.

```

# /dev/loop0p1    /boot    ext2    noauto,noatime 1 2

# /dev/loop0p2    none     swap    sw             0 0

# /dev/loop0p3    /        ext3    noatime        0 1         

/dev/vda1    /boot    ext2    noauto,noatime 1 2

/dev/vda2    none     swap    sw             0 0

/dev/vda3    /        ext3    noatime        0 1         

```

Any ideas?

----------

## Hu

 *Wizumwalt wrote:*   

> So I mounted my gentoo-i386.img onto my host and changed /etc/fstab from /dev/loop0p1 to /dev/vda1, and the same for the vda2, vda3 to see if this was it. Still got the same results. So not sure what device should go in /etc/fstab of the guest os image.

 Something is wrong, then.  The output should have changed to reflect the new device names.  When you give the root password, do the vda devices exist in /dev?

----------

## Wizumwalt

The output in the following line (taken from above) does change when I change the device names in /etc/fstab of the guest OS image file.

```

fsck.ext3: No such file or directory while trying to open /dev/loop0p3 

```

And when I give the root passwd, then cd to /dev and get a listing, there is no vda device listed. However, I do see all the /dev/vdaN devices when I enter from the rescue shell before the switch_root happens. Am I treating devtmpfs wrong somehow? Or not passing devtmpfs correctly after the switch_root?

----------

## Hu

It should be fine.  However, I see that you elected not to unmount the devtmpfs before switching to the main filesystem.  Please fix this and try again.  Also, please check whether the system has in fact mounted the devtmpfs before it tries to perform a fsck.  It may be that you need some basic device nodes resident on the underlying static /dev.

Also, it looks like your initramfs is completely pointless.  Everything it does could be done automatically by the kernel, which would save you a great deal of trouble.

----------

## Wizumwalt

I've changed the /init to umount devtmpfs ...

```

if [ -x "/mnt/gentoo${init}" ]; then

    umount /dev

    umount /proc

    umount /sys

    exec switch_root /mnt/gentoo /sbin/init

fi

```

And currently, the initramfs isn't completely pointless. I'm doing this because I want to learn how to use it and I will eventually use it for some module development and load these modules during this phase. It's all more of a learning exercise w/ a vm for now.

I've attached a screenshot of what I'm seeing when it now boots. And before I perform a 'fsck /dev/vda3', I cd into /dev and can confirm that each of the devices ... vda, vda1, vda2, and vda3 are all there. So I really think I'm almost there now. I'm going over my offsets when I partitioned the device, and as far as I can see, they all look correct. Any idea what's going on w/ those msgs?

I have another question as well. Are the /boot and swap partitions in the image I built really needed (just followed the handbook on this install into the image)? I ask because I don't see that my /boot and swap partitions are being used in any way when I start the VM. I only have the KVM append='root=...' option which is mounting the root fs, but that's it. Should I be mounting swap from the /init?

Here are some screenshots of what I'm seeing.

http://imagebin.ca/view/udO__F.html

http://imagebin.ca/view/q2LhwSF.html

http://imagebin.ca/view/KdidJrg.html

----------

## frostschutz

 *Wizumwalt wrote:*   

> Are the /boot and swap partitions in the image I built really needed?

 

Not if you have KVM boot the kernel for you directly. You can then just put the filesystem on /dev/vda directly using no partition whatsoever.

 *Quote:*   

> 
> 
> Should I be mounting swap from the /init?
> 
> 

 

You should do as little as possible in the initramfs, since the real init later on wants to take care of these things by itself. So no, the initramfs does not enable swap, the real init does that for you.

The main purpose of initramfs is getting at the root partition, if it's encrypted or otherwise not reachable by the kernel itself.

----------

## Wizumwalt

Awesome!! I'm able to boot into my own image with KVM using a custom kernel. I created another image with only one partition (left out the other two) and things are working well. Now I get to move onto networking. And I got a lot out of it.

Thanks so much guys for putting up w/ me.

----------

