# [SOLVED] NFSv4 problem

## stdPikachu

I did consider putting this under portage because, at the moment, it seems specific to emerging things, but it's more strictly related to NFS 4 so here we are.

I converted my LAN to use NFSv4 last night with the intention to switch everything to secure NFS via Kerberos. The setup seemed to be working OK because I can write files to the rw shares, but when I try and emerge something (portage and distfiles both shared over NFS) it barfs with the following error:

```
banquo ~ # emerge -uvD world

>>> Emerging (1 of 6) x11-wm/fluxbox-1.0_rc3 to /

Traceback (most recent call last):

  File "/usr/bin/emerge", line 5481, in ?

    retval = emerge_main()

  File "/usr/bin/emerge", line 5476, in emerge_main

    myopts, myaction, myfiles, spinner)

  File "/usr/bin/emerge", line 4943, in action_build

    retval = mergetask.merge(pkglist, favorites, mtimedb)

  File "/usr/bin/emerge", line 3123, in merge

    prev_mtimes=ldpath_mtimes)

  File "/usr/lib/portage/pym/portage.py", line 3607, in doebuild

    if need_distfiles and not fetch(

  File "/usr/lib/portage/pym/portage.py", line 2397, in fetch

    if portage_util.ensure_dirs(mydir, gid=portage_gid, mode=dirmode, mask=modemask):

  File "/usr/lib/portage/pym/portage_util.py", line 863, in ensure_dirs

    perms_modified = apply_permissions(dir_path, *args, **kwargs)

  File "/usr/lib/portage/pym/portage_util.py", line 580, in apply_permissions

    os.chown(filename, uid, gid)

OSError: [Errno 22] Invalid argument: '/usr/portage/distfiles/'
```

with the accompanying entry in /var/log/messages:

```
Apr 20 10:04:07 banquo nfs4: couldn't resolve gid 250 to string
```

GID 250 is, naturally, portage and exists on both the server and client. A quick ls shows somethign screwy must be happening somewhere (possibly in idmapd...?) because all permissions are showing UID/GID as some crazy number:

```
banquo ~ # ll /usr/portage/

total 2.9M

drwxr-xr-x 158 4294967294 4294967294 4.0K 2007-04-20 06:03 .

drwxr-xr-x   3 root       root          0 2007-04-20 10:08 ..

drwxr-xr-x  36 4294967294 4294967294 4.0K 2006-11-03 05:05 app-accessibility

drwxr-xr-x 167 4294967294 4294967294 4.0K 2007-04-20 05:06 app-admin

drwxr-xr-x   6 4294967294 4294967294   56 2007-04-20 05:06 app-antivirus

drwxr-xr-x  76 4294967294 4294967294 4.0K 2007-04-20 05:06 app-arch

drwxr-xr-x  35 4294967294 4294967294 4.0K 2007-04-20 05:06 app-backup

drwxr-xr-x  23 4294967294 4294967294 4.0K 2007-04-20 05:06 app-benchmarks
```

I don't have user info being exported through Kerberised LDAP yet so I'm not sure what the solution is to this problem; does anyone know if there's a short-term workaround? Alternatively is there a way to stop portage from trying to force permissions on things, as it has full rwx access to the files it needs...?

If it's any help, here's the exports entries from the server:

```
/exportfs               *(rw,fsid=0,insecure,no_subtree_check)

/exportfs/portage       banquo.snafu.local(rw,nohide,insecure,no_subtree_check,no_root_squash,sync)
```

And the autofs mount entry from the client:

```
banquo ~ # cat /etc/autofs/auto.nfs

portage         -fstype=nfs4,rw,noatime prospero:/portage
```

Any help appreciated!

Edit: OK, problem solved now. I hadn't set up /etc/idmapd.conf correctly on all the clients so ID mapping wasn't working at all. See below for some more verbose ramblings.Last edited by stdPikachu on Sun Apr 22, 2007 6:28 pm; edited 1 time in total

----------

## nielchiano

I'm sorry to piggyback on your problem, but I'm also considering a NFSv4 switchover. Could you provide me with some links to the docs you used?

----------

## stdPikachu

Howdy nielchiano, you'll be pleased to hear that I've solved the problem now, including full Kerberos support. Summary of what I did below if anyone feels like Gentoo Wiki-ing it;

Firstly, you'll want all the NFSv4 options from the kernel (and appropriate module(s) loaded).

For the sake of brevity "prospero" is the NFS server and "banquo" is the NFS client. Both have a KDC installed and prospero is the kadmin server. Not going to go into the nuts'n'bolts ofsetting up a Kerberos realm here, all you need to know is the stuff about the keytab.

In kadmin on the NFS server, do the following (assuming MIT KRB5, Heimdal is different):

addprinc -randkey host/nfs-server-fq-domainname

addprinc -randkey nfs/nfs-server-fq-domainname

addprinc -randkey host/nfs-client-fq-domainname

addprinc -randkey nfs/nfs-client-fq-domainname

ktadd -e des-cbc-crc:normal host/nfs-server-fq-domainname

ktadd -e des-cbc-crc:normal nfs/nfs-server-fq-domainname

And in kadmin on the client (the principals have already been added on the kadmin server so don't do it again!):

ktadd -e des-cbc-crc:normal nfs/nfs-client-fq-domainname

This means the keytab on the server holds the server ticket, and the keytab on the client holds the client ticket. Took me an age to figure that one out because it's rarely explained in plain english.

OK, now the client should be able to auth to the NFS server via an ticket exchange. All well and good.

Now on the NFS server prepare your exports file and export dir. The syntax has changed alot since NFSv3 but the good news is that you can keep your existing NFS exports until you've got the NFSv4 stuff working OK.

Create a root dir along the lines of /export. This will be the export "root" which all exportable filesystems have to live under. To accomplish this you can either move the data there (no thanks), create a symlink to the dir you want to export, or use a --bind mount like so:

```
grep bind /etc/fstab

/usr/portage/distfiles  /exportfs/distfiles     none            bind    0 0

/storage/movies         /exportfs/movies        none            bind    0 0

/storage/music          /exportfs/music         none            bind    0 0

/usr/portage            /exportfs/portage       none            bind    0 0

/storage/tv             /exportfs/tv            none            bind    0 0
```

```
prospero ~ # ll /exportfs/

total 220K

drwxr-xr-x   7 root root      40 2007-04-19 15:07 .

drwxr-xr-x  21 root root    4.0K 2007-03-29 21:16 ..

drwxrwsr-x   4 root portage  16K 2007-04-22 06:29 distfiles

drwxrwxr-x   5 std  users    24K 2007-03-04 19:32 movies

drwxr-xr-x  15 std  users   4.0K 1970-01-01 01:00 music

drwxr-xr-x 158 root root    4.0K 2007-04-22 06:04 portage

drwxr-xr-x  53 std  users   4.0K 2007-04-01 00:14 tv
```

Now we need to open up /etc/exports and define our new export root;

```
# Standard nFSv4 export root

/exportfs               *(ro,fsid=0,insecure,no_subtree_check,no_root_squash,nohide)

# Kerberised NFSv4 export root

/exportfs       gss/krb5(rw,fsid=0,insecure,no_subtree_check,nohide)
```

The nohide option is important, without this option your exported filesystems will appear empty on the client. Now we can prepare individual exports with pretty much the same format;

```
# Standard NFS export

/exportfs/movies        banquo.snafu.local(ro,nohide,insecure,no_subtree_check,sync)

# Kerberised export

/exportfs/movies        gss/krb5(ro,nohide,insecure,no_subtree_check,sync)
```

Note that you can't restrict a kerberised NFS export to a particular set of hosts.

OK, that's a few exports done. Couple more things to do before we fire up NFS; firstly make sure that /etc/imapd.conf on both server and client have the domain name listed (this was the problem I was having originally, since ID's weren't being mapped correctly):

```
[General]

Verbosity = 0

Pipefs-Directory = /var/lib/nfs/rpc_pipefs

Domain = snafu.local

[Mapping]

Nobody-User = nobody

Nobody-Group = nobody
```

Obviously change snafu.local to whatever your DNS domain name is. I'm afraid I have no idea what to do if you don't have one other than to encourage you to give Bind a whirl  :Very Happy: 

OK, now you can /etc/init.d/nfs start. Check that all the required services are running, particularly rpc.idmapd, rpc.gssd and rpc.svcgssd. Note that both the server and client require these NFS daemons to be running in order for things like Kerberos and ID mapping to work properly.

OK, now try mounting something:

```
banquo ~ # mount -t nfs4 prospero:/ /nfs
```

This will attempt to mount the entirity of prospero's /exportfs on /nfs on the client;

```
banquo ~ # ll /nfs/

total 56K

drwxr-xr-x   7 root   root    40 2007-04-19 15:07 .

drwxr-xr-x  22 root   root  4.0K 2007-03-27 16:00 ..

drwxr-xr-x   2 root   root     1 2007-04-19 15:06 distfiles

drwxrwxr-x   5 nobody users  24K 2007-03-04 19:32 movies

drwxr-xr-x  15 nobody users 4.0K 1970-01-01 01:00 music

drwxr-xr-x 158 root   root  4.0K 2007-04-22 06:04 portage

drwxr-xr-x  53 nobody users 4.0K 2007-04-01 00:14 tv
```

Since I don't have a yp service or an LDAP server running (yet) to provide UID and GID mappings across all the computers, some of the ownerships will be a bit b0rked. An easy workaround for this if you only have a few computers is to make sure that UID and GID numbers from /etc/passwd and /etc/group are the same on all systems.

If you fancy giving kerberos auth a whirl, pass the option -o sec=krb5 in the mount command. I highly recommend you get NFS working standalone before you attempt anything with Kerberos.

As a bonus, here's the autofs file from the client to show the new syntax (although please note I haven't tested autofs yet):

```
banquo ~ # cat /etc/autofs/auto.nfs

portage         -fstype=nfs4,sec=krb5,rw,noatime prospero:/portage

movies          -fstype=nfs4,sec=krb5,ro,hard,intr,bg prospero:/movies

music           -fstype=nfs4,sec=krb5,ro,hard,intr,bg prospero:/music

tv              -fstype=nfs4,sec=krb5,ro,hard,intr,bg prospero:/tv
```

N.B. please note this is all done from memory and I don't fancy mucking up my config by repeating it again, so there may be errors. If you spot any please inform me, or add the info to the Wiki! Also please note that I've only just got this working so there will undoubtedly be plenty of un-needed/missing options in my NFS configs above as I'm still learning the new syntax. Once I've got the whole NFS/KRB5/LDAP thing working I'll post something a bit better written  :Very Happy: 

Finally, here's some of the docs I've used (there are many more, but these were the most helpful):

https://help.ubuntu.com/community/NFSv4Howto?highlight=(nfs)

http://www-theorie.physik.unizh.ch/~dpotter/howto/kerberos

http://www.vanemery.com/Linux/NFSv4/NFSv4-no-rpcsec.html

Hope this helps someone else out there  :Smile: 

----------

## nielchiano

 *nielchiano wrote:*   

> some links to the docs you used?

 

Well.... that is definitely more than I'd hoped for. Thanks!!!!!

----------

