# [SOLVED] Diskless client can't mount NFS root, liveCD can

## dusanc

Hello,

I'm trying to bring up some diskless gentoo clients.

Setup I'm using is: dnsmasq DHCP/TFTP/DNS server and PXELINUX as bootloader.

On boot clients get IP adresses, recieve kernel and kernel options from server and start booting kernel but they stop with:

```
VFS: Unable to mount root fs via NFS.

List of all partitions:

No filesystems couldmount root, tried:

nfs

Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)
```

If I try to mount that NFS share from that same client when it's running from livecd it works.

I used NFSv3, without any firewall rules.

I have no more ideas.

TIA

Dusan

```
krshina3 /home/dusan # tcpdump -vv -i br0 | grep 168.1.101

.....

    192.168.1.101.49164 > 192.168.1.3.41418: [udp sum ok] UDP, length 4

    192.168.1.3.41418 > 192.168.1.101.49164: [bad udp cksum 0x8956 -> 0xfe69!] UDP, length 1412

    192.168.1.101.49164 > 192.168.1.3.41418: [udp sum ok] UDP, length 4

    192.168.1.3.41418 > 192.168.1.101.49164: [bad udp cksum 0x8956 -> 0xfe68!] UDP, length 1412

    192.168.1.101.49164 > 192.168.1.3.41418: [udp sum ok] UDP, length 4

    192.168.1.3.41418 > 192.168.1.101.49164: [bad udp cksum 0x8956 -> 0xfe67!] UDP, length 1412

    192.168.1.101.49164 > 192.168.1.3.41418: [udp sum ok] UDP, length 4

    192.168.1.3.41418 > 192.168.1.101.49164: [bad udp cksum 0x8956 -> 0xfe66!] UDP, length 1412

    192.168.1.101.49164 > 192.168.1.3.41418: [udp sum ok] UDP, length 4

    192.168.1.3.41418 > 192.168.1.101.49164: [bad udp cksum 0x8956 -> 0xfe65!] UDP, length 1412

    192.168.1.101.49164 > 192.168.1.3.41418: [udp sum ok] UDP, length 4

    192.168.1.3.41418 > 192.168.1.101.49164: [bad udp cksum 0x8956 -> 0xfe64!] UDP, length 1412

    192.168.1.101.49164 > 192.168.1.3.41418: [udp sum ok] UDP, length 4

    192.168.1.3.41418 > 192.168.1.101.49164: [bad udp cksum 0x8956 -> 0xfe63!] UDP, length 1412

    192.168.1.101.49164 > 192.168.1.3.41418: [udp sum ok] UDP, length 4

    192.168.1.3.41418 > 192.168.1.101.49164: [bad udp cksum 0x8956 -> 0xfe62!] UDP, length 1412

    192.168.1.101.49164 > 192.168.1.3.41418: [udp sum ok] UDP, length 4

    192.168.1.3.41418 > 192.168.1.101.49164: [bad udp cksum 0x8956 -> 0xfe61!] UDP, length 1412

    192.168.1.101.49164 > 192.168.1.3.41418: [udp sum ok] UDP, length 4

    192.168.1.3.41418 > 192.168.1.101.49164: [bad udp cksum 0x8956 -> 0xfe60!] UDP, length 1412

    192.168.1.101.49164 > 192.168.1.3.41418: [udp sum ok] UDP, length 4

    192.168.1.3.41418 > 192.168.1.101.49164: [bad udp cksum 0x8956 -> 0xfe5f!] UDP, length 1412

    192.168.1.101.49164 > 192.168.1.3.41418: [udp sum ok] UDP, length 4

    192.168.1.3.41418 > 192.168.1.101.49164: [bad udp cksum 0x8956 -> 0xfe5e!] UDP, length 1412

    192.168.1.101.49164 > 192.168.1.3.41418: [udp sum ok] UDP, length 4

    192.168.1.3.41418 > 192.168.1.101.49164: [bad udp cksum 0x8956 -> 0xfe5d!] UDP, length 1412

    192.168.1.101.49164 > 192.168.1.3.41418: [udp sum ok] UDP, length 4

    192.168.1.3.41418 > 192.168.1.101.49164: [bad udp cksum 0x8956 -> 0xfe5c!] UDP, length 1412

    192.168.1.101.49164 > 192.168.1.3.41418: [udp sum ok] UDP, length 4

    192.168.1.3.41418 > 192.168.1.101.49164: [bad udp cksum 0x8956 -> 0xfe5b!] UDP, length 1412

    192.168.1.101.49164 > 192.168.1.3.41418: [udp sum ok] UDP, length 4

    192.168.1.3.41418 > 192.168.1.101.49164: [bad udp cksum 0x8956 -> 0xfe5a!] UDP, length 1412

    192.168.1.101.49164 > 192.168.1.3.41418: [udp sum ok] UDP, length 4

    192.168.1.3.41418 > 192.168.1.101.49164: [bad udp cksum 0x87a6 -> 0x2cf5!] UDP, length 980

    192.168.1.101.49164 > 192.168.1.3.41418: [udp sum ok] UDP, length 4

19:25:35.318485 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.101 tell 192.168.1.3, length 28

19:25:36.342503 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.101 tell 192.168.1.3, length 28

    192.168.1.3.bootps > 192.168.1.101.bootpc: [bad udp cksum 0x84fe -> 0x446f!] BOOTP/DHCP, Reply, length 300, xid 0x1d19cab3, Flags [none] (0x0000)

          Your-IP 192.168.1.101

            Requested-IP Option 50, length 4: 192.168.1.101

    192.168.1.3.bootps > 192.168.1.101.bootpc: [bad udp cksum 0x84fe -> 0x416f!] BOOTP/DHCP, Reply, length 300, xid 0x1d19cab3, Flags [none] (0x0000)

          Your-IP 192.168.1.101

19:25:41.462490 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.101 tell 192.168.1.3, length 28

19:25:41.462698 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.1.101 is-at 00:15:b7:7f:89:6b (oui Unknown), length 46

```

```
krshina3 /home/dusan # rpcdebug -m rpc -s all

krshina3 /home/dusan # rpcdebug -m nfsd -s all

krshina3 /home/dusan # tail -f -n 100 /var/log/messages

Jan  6 18:51:19 krshina3 dnsmasq-dhcp[3896]: DHCPDISCOVER(br0) 00:15:b7:7f:89:6b                                                                                                             

Jan  6 18:51:19 krshina3 dnsmasq-dhcp[3896]: DHCPOFFER(br0) 192.168.1.101 00:15:b7:7f:89:6b

Jan  6 18:51:21 krshina3 dnsmasq-dhcp[3896]: DHCPREQUEST(br0) 192.168.1.101 00:15:b7:7f:89:6b

Jan  6 18:51:21 krshina3 dnsmasq-dhcp[3896]: DHCPACK(br0) 192.168.1.101 00:15:b7:7f:89:6b miner1

Jan  6 18:51:21 krshina3 dnsmasq-tftp[3896]: error 0 TFTP Aborted received from 192.168.1.101

Jan  6 18:51:21 krshina3 dnsmasq-tftp[3896]: failed sending /diskless/shared/pxelinux.0 to 192.168.1.101

Jan  6 18:51:21 krshina3 dnsmasq-tftp[3896]: sent /diskless/shared/pxelinux.0 to 192.168.1.101

Jan  6 18:51:21 krshina3 dnsmasq-tftp[3896]: sent /diskless/shared/ldlinux.c32 to 192.168.1.101

Jan  6 18:51:21 krshina3 dnsmasq-tftp[3896]: file /diskless/shared/pxelinux.cfg/3437d000-5648-16c1-8011-b70257822197 not found

Jan  6 18:51:21 krshina3 dnsmasq-tftp[3896]: file /diskless/shared/pxelinux.cfg/01-00-15-b7-7f-89-6b not found

Jan  6 18:51:21 krshina3 dnsmasq-tftp[3896]: file /diskless/shared/pxelinux.cfg/C0A80165 not found

Jan  6 18:51:21 krshina3 dnsmasq-tftp[3896]: file /diskless/shared/pxelinux.cfg/C0A8016 not found

Jan  6 18:51:21 krshina3 dnsmasq-tftp[3896]: file /diskless/shared/pxelinux.cfg/C0A801 not found

Jan  6 18:51:21 krshina3 dnsmasq-tftp[3896]: file /diskless/shared/pxelinux.cfg/C0A80 not found

Jan  6 18:51:21 krshina3 dnsmasq-tftp[3896]: file /diskless/shared/pxelinux.cfg/C0A8 not found

Jan  6 18:51:21 krshina3 dnsmasq-tftp[3896]: file /diskless/shared/pxelinux.cfg/C0A not found

Jan  6 18:51:21 krshina3 dnsmasq-tftp[3896]: file /diskless/shared/pxelinux.cfg/C0 not found

Jan  6 18:51:21 krshina3 dnsmasq-tftp[3896]: file /diskless/shared/pxelinux.cfg/C not found

Jan  6 18:51:21 krshina3 dnsmasq-tftp[3896]: sent /diskless/shared/pxelinux.cfg/default to 192.168.1.101

Jan  6 18:51:23 krshina3 dnsmasq-tftp[3896]: sent /diskless/shared//bzImage to 192.168.1.101

Jan  6 18:51:27 krshina3 dnsmasq-dhcp[3896]: DHCPDISCOVER(br0) 00:15:b7:7f:89:6b                                                                                                             

Jan  6 18:51:27 krshina3 dnsmasq-dhcp[3896]: DHCPOFFER(br0) 192.168.1.101 00:15:b7:7f:89:6b

Jan  6 18:51:27 krshina3 dnsmasq-dhcp[3896]: DHCPREQUEST(br0) 192.168.1.101 00:15:b7:7f:89:6b

Jan  6 18:51:27 krshina3 dnsmasq-dhcp[3896]: DHCPACK(br0) 192.168.1.101 00:15:b7:7f:89:6b miner1

Jan  6 18:51:58 krshina3 kernel: NFSD: laundromat service - starting

Jan  6 18:51:58 krshina3 kernel: NFSD: laundromat_main - sleeping for 90 seconds

```

```
krshina3 /home/dusan # cat /etc/dnsmasq.conf

# Listen only to this interface

interface=br0

# Assign names based on mac address

dhcp-host=68:07:15:48:5A:AD,192.168.1.10,24h

#Miners

dhcp-host=00:15:b7:7f:89:6b,miner1,192.168.1.101,24h

dhcp-host=54:ee:75:aa:21:74,miner2,192.168.1.102,24h

#dhcp-option=option:root-path,192.168.1.3:/diskless/shared

# Any other DHCP request gets an ip from this range

dhcp-range=192.168.1.201,192.168.1.250,24h

# Enable the TFTP server and set the root directory for files available via TFTP.

enable-tftp

tftp-root=/diskless/shared

dhcp-boot=/pxelinux.0

```

```
krshina3 /home/dusan # cat /diskless/shared/pxelinux.cfg/default

DEFAULT gentoo

LABEL gentoo

LINUX /bzImage

APPEND ip=dhcp root=/dev/nfs rootfstype=nfs nfsroot=192.168.1.3:/diskless/shared loglevel=7 raid=noautodetect noinitrd rw

```

```
krshina3 /home/dusan # cat /etc/conf.d/nfs | grep -v '#'

NFS_NEEDED_SERVICES="rpc.idmapd"

OPTS_RPC_NFSD="8 -V 3 -N 4 -N 4.1"

OPTS_RPC_MOUNTD=""

OPTS_RPC_STATD=""

OPTS_RPC_IDMAPD=""

OPTS_RPC_GSSD=""

OPTS_RPC_SVCGSSD=""

OPTS_RPC_RQUOTAD=""

EXPORTFS_TIMEOUT=30

```

```
krshina3 /home/dusan # cat /etc/exports|grep -v '#'

/diskless/shared *(sync,rw,no_root_squash,no_all_squash,fsid=0,no_subtree_check)

```

```
krshina3 /home/dusan # cat /usr/src/linux/.config | grep NFS

CONFIG_KERNFS=y

CONFIG_NFS_FS=y

# CONFIG_NFS_V2 is not set

CONFIG_NFS_V3=y

# CONFIG_NFS_V3_ACL is not set

# CONFIG_NFS_V4 is not set

# CONFIG_NFS_SWAP is not set

CONFIG_ROOT_NFS=y

CONFIG_NFS_DEBUG=y

CONFIG_NFSD=y

CONFIG_NFSD_V3=y

# CONFIG_NFSD_V3_ACL is not set

# CONFIG_NFSD_V4 is not set

CONFIG_NFS_COMMON=y

```

----------

## krinn

try that:

https://forums.gentoo.org/viewtopic-p-7592928.html#7592928

----------

## dusanc

I recompiled server and client kernel to have only NFSv3 and used nfsvers=3 but still nothing.

Looking at wireshark and tscpdump outputs I think that for some reason client doesn't send any packet to server after getting the IP adress from DHCP.

----------

## NeddySeagoon

dusanc,

Well, the server works, or the liveCD wouldn't work either.

Pastebin the client kernel .config and the init script from the client initrd, if you have a client initrd.

----------

## dusanc

 *NeddySeagoon wrote:*   

> dusanc,
> 
> Well the server works, or the liveCD wouldn't work either.
> 
> Pastebin the client kernel .config and the init script from the client initrd, if you have a client initrd.

 

There's no initrd, kernels are monolithic.

.config is here https://paste.pound-python.org/show/afx0V9fktQXvUOnF4uju/

----------

## krinn

 *dusanc wrote:*   

> I recompiled server and client kernel to have only NFSv3 and used nfsvers=3 but still nothing.
> 
> Looking at wireshark and tscpdump outputs I think that for some reason client doesn't send any packet to server after getting the IP adress from DHCP.

 

you read it bad, the key is vers=3

nfsvers=3 wasn't the key to success  :Smile: 

and the problem is not because the client use an nfsv3 implementation, but because the server try all implementations it knows if the client don't tell him "i'm using a version x", hence the fix with vers=3 -> the client is telling the server, try with nfsv3 on me first.

----------

## dusanc

 *krinn wrote:*   

>  *dusanc wrote:*   I recompiled server and client kernel to have only NFSv3 and used nfsvers=3 but still nothing.
> 
> Looking at wireshark and tscpdump outputs I think that for some reason client doesn't send any packet to server after getting the IP adress from DHCP. 
> 
> you read it bad, the key is vers=3
> ...

 

And we have a winner!

Solution, as said above is to add vers=3:

```
 nfsroot=$SERVERIP:/diskless/shared,vers=3
```

Now clients booted.

2nd question: Is it possible to use NFSv3 just for boot and NFSv4 for remounting / ?

----------

## krinn

 *dusanc wrote:*   

> 2nd question: Is it possible to use NFSv3 just for boot and NFSv4 for remounting / ?

 

the subtle to mount a v4 or v3 share is on the client asking the mount.

Look at (a proper v4 share)

```
/export      192.168.0.0/24(rw,sec=sys,fsid=0,no_root_squash,no_subtree_check,nohide,async,anonuid=250,anongid=250)

/export/distfiles   192.168.0.0/24(rw,no_subtree_check,async,no_root_squash,nohide,secure,anonuid=250,anongid=250)

```

and how client mount this:

mount it as nfsv3: mount server:/export/distfiles ...

mount it as nvsv4: mount server:/distfiles ...

for nfvs4 first directory is the nfs_root, which mean its base, so / because it's /export in my example, export is so / it must have the entry fsid=0 to tag it /

and if you mount /distfiles you then mount it "as an nfsv4" client, because you aim the directory "/" + "distfiles"

and if you mount /export/distfiles you mount it "as nfsv3" client, because there's no "root" concept on v3

keep that in mind, because if you try force nfsv4 with mount /export/distfiles, it will fail because you are saying let me mount "/" + "export" + "distilfes", which mean /export/export/distfiles in the example.

keep also in mind: all directories must be subdir of / for nfsv4.

so you cannot do 

/export ...

/export/distfiles

/var/tmp <- wrong, not subdir of /export.

if you have different directories to mount, you bind them to your nfs_root mount point.

to allow /var/tmp you mount --bind /var/tmp /export/var_tmp

and its entry in the config is set as:

/export/var_tmp ... var_tmp is a subdir of /export, nfsv4 server is now happy

make sure your nfsv4 server is configure correctly, and your nfsv4 or 3 clients will be happy.

but if you mess with the nfsv4 server config, nfsv4 clients will not (while still nfsv3 will remain happy).

----------

