# NFS4 diskless boot [Solved]

## eackad

I am trying to setup a diskless boot for nodes attached to a gentoo headnode. I followed the diskless gentoo page (diskless nodes). 

I get the following error on the booting node:

```
Please append a correct "root=" boot option; here are the available partitions:

kernel panic - not syncing: vfs: Unable to mount fs on unknown-block(2,0)

```

There is a root= option in my /nodes/pxelinux.cfg/default 

```
DEFAULT diskless

DISPLAY message.txt

PROMPT 10

TIMEOUT 50

LABEL memtest

KERNEL memtest

LABEL diskless

KERNEL bzImage

APPEND ip=dhcp root=/dev/nfs nfsroot=10.0.0.1:/nodes/base raid=noautodetect

LABEL diskless.old

KERNEL bzImage.orig

APPEND ip=dhcp root=/dev/nfs nfsroot=10.0.0.1:/nodes/base

```

The folders are exported (/etc/exports)

```
/nodes/base   10.0.0.0/23(rw,fsid=0,async,no_root_squash,no_all_squash,no_subtree_check,insecure)

/opt   10.0.0.0/23(async,ro,no_root_squash,no_all_squash,no_subtree_check,insecure)

/usr   10.0.0.0/23(async,ro,no_root_squash,no_all_squash,no_subtree_check,insecure)

/home  10.0.0.0/23(async,rw,no_root_squash,no_all_squash,no_subtree_check,insecure)

```

and  exportfs shows them:

```
/nodes/base      10.0.0.0/23

/opt             10.0.0.0/23

/usr             10.0.0.0/23

/home            10.0.0.0/23

```

Further the headnode seems to say it has allowed the node to mount the location (from headnode /var/log/messages):

```
Jul 15 14:21:38 deepthought in.tftpd[3293]: RRQ from 10.0.0.3 filename /pxelinux.cfg/default

Jul 15 14:21:38 deepthought in.tftpd[3294]: RRQ from 10.0.0.3 filename /message.txt

Jul 15 14:21:40 deepthought in.tftpd[3295]: RRQ from 10.0.0.3 filename /bzImage

Jul 15 09:21:53 deepthought dhcpd: DHCPDISCOVER from 00:25:90:2c:ff:06 via enp2s0f0

Jul 15 09:21:53 deepthought dhcpd: DHCPOFFER on 10.0.0.3 to 00:25:90:2c:ff:06 via enp2s0f0

Jul 15 09:21:53 deepthought dhcpd: DHCPREQUEST for 10.0.0.3 (10.0.0.1) from 00:25:90:2c:ff:06 via enp2s0f0

Jul 15 09:21:53 deepthought dhcpd: DHCPACK on 10.0.0.3 to 00:25:90:2c:ff:06 via enp2s0f0

Jul 15 09:21:53 deepthought rpc.mountd[3204]: authenticated mount request from 10.0.0.3:907 for /nodes/base (/nodes/base)

Jul 15 09:21:58 deepthought rpc.mountd[3204]: authenticated mount request from 10.0.0.3:670 for /nodes/base (/nodes/base)

Jul 15 09:22:08 deepthought rpc.mountd[3204]: authenticated mount request from 10.0.0.3:1003 for /nodes/base (/nodes/base)

Jul 15 09:22:28 deepthought rpc.mountd[3204]: authenticated mount request from 10.0.0.3:848 for /nodes/base (/nodes/base)

Jul 15 09:22:58 deepthought rpc.mountd[3204]: authenticated mount request from 10.0.0.3:785 for /nodes/base (/nodes/base)

Jul 15 09:23:28 deepthought rpc.mountd[3204]: authenticated mount request from 10.0.0.3:972 for /nodes/base (/nodes/base)

```

The node's fstab is (/nodes/base/etc/)

```
deepthought:/nodes/base /               nfs4            intr,proto=tcp,wsize=32768,rsize=32768            0 0

#deepthought:/opt       /opt            nfs4            sync,hard,intr,rw,nolock,rsize=8192,wsize=8192    0 0

deepthought:/opt        /opt            nfs4            intr,proto=tcp,wsize=32768,rsize=32768            0 0

#deepthought:/usr       /usr            nfs4            sync,hard,intr,rw,nolock,rsize=8192,wsize=8192    0 0

deepthought:/usr        /usr            nfs4            intr,proto=tcp,wsize=32768,rsize=32768            0 0

#deepthought:/home      /home           nfs4            sync,hard,intr,rw,nolock,rsize=8192,wsize=8192    0 0

deepthought:/home       /home           nfs4            intr,proto=tcp,wsize=32768,rsize=32768            0 0

none                    /proc           proc            defaults                                          0 0

```

Anyone have any ideas? Thanks for the help in advance!

----------

## szatox

Here you go

 *Quote:*   

> LABEL Gentoo
> 
> MENU LABEL Gentoo Live on PXE
> 
> LINUX gentoo/kernel-genkernel-x86-3.7.10-gentoo
> ...

 

 *Quote:*   

> 
> 
> # ls /mnt/linux.images/tftp/gentoo
> 
> System.map-genkernel-x86-3.7.10-gentoo  excludes  gx86.sqfs  initramfs-genkernel-x86-3.7.10-gentoo  kernel-genkernel-x86-3.7.10-gentoo
> ...

 

 *Quote:*   

> # cat /etc/conf.d/atftp 
> 
> # Config file for tftp server
> 
> rc_net_vn0_need="!net"
> ...

 

I hope you can make some sense of bold fragments.

Be carefull with paths, kernel and initramfs are relative to different directory than gx86.sqfs (containing netboot root filesystem)

And answering questions you're likely to ask next:

1) make sure you have ethernet driver compiled in kernel (builtin, not a module)

2) make sure your kernel supports root over NFS

3) you must either make a LiveCD-fashioned FS image or patch your initramfs to let you mount root directly over NFS

4) Unless you know it's a bad idea, use LiveCD-fashioned squashfs image rather than hacking initramfs. Your LAN is going to take it much better.

----------

## eackad

I think I have the kernel compiled with all that is needed:

 *Quote:*   

> CONFIG_NFS_FS=y
> 
> CONFIG_NFS_V2=y
> 
> CONFIG_NFS_V3=y
> ...

 

The ethernet drivers are compiled, not modules and I think working correctly since the headnode gives a dhcp ip address successfully to the node.

However, your third comment is a possible problem. I followed the instructions for diskless gentoo here (http://wiki.gentoo.org/wiki/Diskless_nodes).  There was no mention of constructing a initrd (let alone an initramfs). Will the method given in the wiki not work without a initrd? I want the structure to be as simple as possible.

----------

## szatox

I don't know if it can work without initramfs. Honestly, I doubt it though.

Good news is you can use genkernel to build kernel and initramfs, and since you seem to have already configured kernel you wouldn't have to do anything more than run 'genkernel --oldconfig all' (perhaps adding as well --no-clean to speed up the process). Or you can fairly easily build your own one and tailor it to your wants.

To make liveCD you just have to run 'mksquashfs <source root directory> <output filesystem image.sqfs>' - pretty straight-forward.

You might also try block level access to filesystem. Linux has a nice support for iSCSI, unfortunately I think that would require initramfs too, since you need some user-space tools to configure it, and it is less popular than NFS.

Anyway, don't be afraid of that initramfs thing, it takes a few MB and allows you run "weird" setups  :Wink:  so why not to take advantage of this.

----------

## eackad

Thanks for the help! I built the system files using genkernel (and added kernel support for squashfs) as you suggested and added the commands to the pxelinux as follows

pxlinux.cfg/default

 *Quote:*   

>  DEFAULT diskless
> 
> DISPLAY message.txt
> 
> PROMPT 10
> ...

 

I made the squashfs using: mksquashfs /nodeimage/ nodes.sqfs

The /nodes/base has the following files

 *Quote:*   

>  drwxr-xr-x  2 root root 4.0K Jun 30 11:42 bin
> 
> drwxr-xr-x  2 root root 4.0K Jun 30 07:54 dev
> 
> drwxr-xr-x 52 root root 4.0K Jul  8 10:40 etc
> ...

 

While /nodes/base/ has

 *Quote:*   

>  -rw-r--r-- 1 root root 1.3M Jul 21 09:41 initramfs-genkernel-x86_64-3.12.20-gentoo-nb
> 
> -rw-r--r-- 1 root root 5.4M Jul 21 09:41 kernel-genkernel-x86_64-3.12.20-gentoo-nb
> 
> -rw-r--r-- 1 root root  33M Jul 21 11:06 nodes.sqfs
> ...

 

The node finds the files displaying:

 *Quote:*   

> Loading base/kernel-genkernel-x86_64-3.12.20-gentoo-nb........
> 
> Loading base/initramfs-genkernel-x86_64-3.12.20-gentoo-nb....ready.
> 
> 

 

But it then complains:

 *Quote:*   

>  >> Attempting to mount NFS CD image on 10.0.0.1:/nodes/base with options ro,nolock, risze=1024,wsize=1024
> 
> >> Determining looptype ...
> 
> >> mounting squashfs filesystem
> ...

 

This is from the fstab on the nodes (/nodeimage/etc/fstab) shown in the first post. The /var/log/messgaes on the headnode remains:

 *Quote:*   

> Jul 21 11:17:53 deepthought dhcpd: DHCPACK on 10.0.0.3 to 00:25:90:2c:ff:06 via enp2s0f0
> 
> Jul 21 11:17:55 deepthought rpc.mountd[2713]: authenticated mount request from 10.0.0.3:836 for /nodes/base (/nodes/base)
> 
> 

  the only real difference is that there is only one call to the rpc.mountd (instead of multiple as before)

There are few issues I do not understand about the above. 

1) Should the /nodes/base be mounted ro or should it be rw? 

2) Why does it mount the root system onto /newroot when I want it to be the /?

3) Why is /opt not mounted before /usr since it comes second in the fstab file and why doesn't it show up in /var/log/messages?

Any ideas what I am doing incorrectly?

----------

## szatox

When you boot, initramfs is your root. You provide arguments that tell it what device is supposed to be your root, it mounts it in /newroot, prepairs /newroot to be ran, and when it's done with stuff you need  before you init your system, it triggers pivot_root which rotates your active filesystem. /newroot becomes /, / becomes /oldroot (or something like that) and then /oldroot is umounted and memery freed.

So, if your initramfs mounts /newroot, it's good. Well, actually it's good when it mounts your real root in /newroot.

Now,  bootloader obviously finds kernel and initramfs -> dchp and tftp are configured properly.

params you provided to kernel at boot seem correct.

 *Quote:*   

> > Mounting 10.0.0.1:/usr as /usr: mount -t nfs4 -o intr,proto=tcp,wsize=32760,rsize=32768, ro 10.0.0.1:/usr on /newroot/usr failed: No such file or directory
> 
> !! unable to mount 10.0.0.1:/usr for /usr

 

hold on a minute, why would you mount 10.0.0.1:/usr on /newroot? Don't you have /usr inside your squashfs? This is something that should never happen, /usr is not /, if you have separate /usr, mount it over /nerwoor/usr. Still... why it attempts to mount /usr while you are still on initramfs? It shouldn't even see /etc/fstab yet. Was there and attempt to improve initramfs? (well, you can improve it adding support for persistend changes with aufs, but I suppose getting it running is higher priority)

----------

## eackad

prior to that error it says:

 *Quote:*   

>  >>Making tmpfs for /newroot
> 
> >> determining root device...
> 
> >> mounting /dev/nfs as root...
> ...

 

then it gives the stuff I posted previously. So I thought the error was coming after the pivot_root. Is there at least a way to make the directory for newroot to avoid that error? 

I have the directory for the /usr in the squashfs and I have not modified the initramfs. I simply want the node's /usr to be mounted from the headnode's /usr.

Thanks again for all your help, it seems like it is so close to working, but I have no idea how to change any of the newroot setup.

----------

## szatox

ok, let's take a step back, and look around.

Mount that squashfs image, chroot into it and show output of those 2 things:

```
grep -v '^[[:space:]]*#\|^[[:space:]]*$' /etc/fstab

rc-config --show all

```

----------

## eackad

It seems that error was not the problem, the system was actually hanging on waiting for uevents. I fixed it with a new kernel that included the vga_16 frame buffer and vesa frame buffer since the node has nvidia cards that are not used for display.

Thanks for all your help! [solved]

----------

## szatox

Great, good to know you made it work.

I'm curious what that mounting to /nerwoot was though. It still seems weird the way you described it

----------

## toralf

 *eackad wrote:*   

> It seems that error was not the problem, the system was actually hanging on waiting for uevents. I fixed it with a new kernel that included the vga_16 frame buffer and vesa frame buffer since the node has nvidia cards that are not used for display.
> 
> Thanks for all your help! [solved]

 pls put the [solved] in the topic

----------

## dobbs

 *szatox wrote:*   

> I don't know if it can work without initramfs. Honestly, I doubt it though.

 

It worked very well prior to upgrading to kernel 3.12.x (up to 3.10.25 at least).  I had a nfsroot diskless cluster running for quite some time that suddenly ceased to boot after upgrading the server kernel.  Surprisingly, the systems already running when the server restarted continued to function until they were rebooted, which hid the issue for several days.  I have so far failed to boot any of my diskless nodes (or even a NetBSD nfsroot) in three months.  I can only assume the kernel devs broke nfsroot on the server side somehow.  It's extremely frustrating.

Addendum:

Got nfsroot working again without initrd.  My similar topic is here: https://forums.gentoo.org/viewtopic.php?p=7592928#7592928

----------

