# NFS + Kerberos

## cdavidshaffer

I'm having trouble getting NFS + Kerberos to work and, for once, I'm finding that I don't have the tools that I need to diagnose the problem myself.  I have tried to do my homework here (read a lot of what's out there for other distros) but I'm stuck.  Any help would be appreciated!

Here's what I have:

KDC running.  Can kinit from other machines with no problems

Keytab file on NFS client in /etc/krb5.keytab on client with keys for nfs/client.domain@REALM in two default encryptions.  Client running rpc.gssd.

NFS server configured with export:

/u99 *.domain(rw,no_root_squash,no_subtree_check,sec=krb5)

   Server is running nfs services plus RPC stuff needed for GSS + kerberos auth:

```

root     27898  0.0  0.0   2800   632 ?        Ss   Oct15   0:00 /usr/sbin/rpc.idmapd -vvv

root     27933  0.0  0.0   5480  2156 ?        Ss   Oct15   0:00 /usr/sbin/rpc.mountd

root     27979  0.0  0.0   4180   680 ?        Ss   Oct15   0:00 /usr/sbin/rpc.gssd -vvv -rrr

root     28008  0.0  0.0   4008  1508 ?        Ss   Oct15   0:00 /usr/sbin/rpc.svcgssd -vvv -rrr

```

DNS entries so both forward and reverse lookups are consistent

The problem...when I try to mount:

```

nfs-client$ mount -v -t nfs4 -vvv -o sec=krb5 nfs-server.domain:/u99 /mnt/u99

mount.nfs4: timeout set for Wed Oct 19 10:21:04 2016

mount.nfs4: trying text-based options 'sec=krb5,addr=10.71.3.1,clientaddr=10.71.3.6'

mount.nfs4: mount(2): Invalid argument

mount.nfs4: an incorrect mount option was specified

```

with the following message on the CLIENT log:

Oct 19 13:14:47 edmund rpc.idmapd[26139]: New client: d8

Oct 19 13:14:47 edmund rpc.idmapd[26139]: Stale client: d8

Oct 19 13:14:47 edmund rpc.idmapd[26139]: 	-> closed /var/lib/nfs/rpc_pipefs//nfs/clntd8/idmap

Oct 19 13:14:47 edmund rpc.gssd[25512]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clntd8

and no messages on the NFS server log.  There is not enough info in the client log for me to figure out if gssd is failing to authenticate but it doesn't even seem to be contacting the server's rpc.svcgssd (at least, if it is, the server isn't logging anything).  Finally here's my RPC/portmap info for NFS server:

```

   program version netid     address                service    owner

    100000    4    tcp6      ::.0.111               portmapper superuser

    100000    3    tcp6      ::.0.111               portmapper superuser

    100000    4    udp6      ::.0.111               portmapper superuser

    100000    3    udp6      ::.0.111               portmapper superuser

    100000    4    tcp       0.0.0.0.0.111          portmapper superuser

    100000    3    tcp       0.0.0.0.0.111          portmapper superuser

    100000    2    tcp       0.0.0.0.0.111          portmapper superuser

    100000    4    udp       0.0.0.0.0.111          portmapper superuser

    100000    3    udp       0.0.0.0.0.111          portmapper superuser

    100000    2    udp       0.0.0.0.0.111          portmapper superuser

    100000    4    local     /var/run/rpcbind.sock  portmapper superuser

    100000    3    local     /var/run/rpcbind.sock  portmapper superuser

    100024    1    udp       0.0.0.0.183.59         status     superuser

    100024    1    tcp       0.0.0.0.228.212        status     superuser

    100024    1    udp6      ::.141.156             status     superuser

    100024    1    tcp6      ::.205.51              status     superuser

    100005    1    udp       0.0.0.0.179.45         mountd     superuser

    100005    1    tcp       0.0.0.0.192.18         mountd     superuser

    100005    1    udp6      ::.134.100             mountd     superuser

    100005    1    tcp6      ::.142.66              mountd     superuser

    100005    2    udp       0.0.0.0.175.239        mountd     superuser

    100005    2    tcp       0.0.0.0.149.19         mountd     superuser

    100005    2    udp6      ::.161.175             mountd     superuser

    100005    2    tcp6      ::.176.199             mountd     superuser

    100005    3    udp       0.0.0.0.155.86         mountd     superuser

    100005    3    tcp       0.0.0.0.215.99         mountd     superuser

    100005    3    udp6      ::.228.124             mountd     superuser

    100005    3    tcp6      ::.140.95              mountd     superuser

    100003    3    tcp       0.0.0.0.8.1            nfs        superuser

    100003    4    tcp       0.0.0.0.8.1            nfs        superuser

    100227    3    tcp       0.0.0.0.8.1            nfs_acl    superuser

    100003    3    udp       0.0.0.0.8.1            nfs        superuser

    100003    4    udp       0.0.0.0.8.1            nfs        superuser

    100227    3    udp       0.0.0.0.8.1            nfs_acl    superuser

    100003    3    tcp6      ::.8.1                 nfs        superuser

    100003    4    tcp6      ::.8.1                 nfs        superuser

    100227    3    tcp6      ::.8.1                 nfs_acl    superuser

    100003    3    udp6      ::.8.1                 nfs        superuser

    100003    4    udp6      ::.8.1                 nfs        superuser

    100227    3    udp6      ::.8.1                 nfs_acl    superuser

    100021    1    udp       0.0.0.0.197.30         nlockmgr   superuser

    100021    3    udp       0.0.0.0.197.30         nlockmgr   superuser

    100021    4    udp       0.0.0.0.197.30         nlockmgr   superuser

    100021    1    tcp       0.0.0.0.213.149        nlockmgr   superuser

    100021    3    tcp       0.0.0.0.213.149        nlockmgr   superuser

    100021    4    tcp       0.0.0.0.213.149        nlockmgr   superuser

    100021    1    udp6      ::.165.45              nlockmgr   superuser

    100021    3    udp6      ::.165.45              nlockmgr   superuser

    100021    4    udp6      ::.165.45              nlockmgr   superuser

    100021    1    tcp6      ::.158.14              nlockmgr   superuser

    100021    3    tcp6      ::.158.14              nlockmgr   superuser

    100021    4    tcp6      ::.158.14              nlockmgr   superuser

```

I don't have a hosts.deny file on the NFS server and the hosts.allow file has an entry permitted the NFS client to access all services.

Again, any help would be appreciated.

David

----------

## gerdesj

This is the error:  "mount.nfs4: an incorrect mount option was specified".

Break the problem down into bite sized pieces:  Remove the Kerb bit and demonstrate that the volume can be mounted.  Then add on authentication and see if that works. 

Cheers

Jon

----------

## Ant P.

If you're using NFS4 you can get rid of a lot of needless moving parts - put "-N 2 -N 3" in your rpc.{mountd,nfsd} args. I don't know if Kerberos needs idmapd but NFS4 doesn't at all.

One other thing - are your clocks synced?

----------

## cdavidshaffer

 *gerdesj wrote:*   

> This is the error:  "mount.nfs4: an incorrect mount option was specified".
> 
> Break the problem down into bite sized pieces:  Remove the Kerb bit and demonstrate that the volume can be mounted.  Then add on authentication and see if that works. 
> 
> Cheers
> ...

 

The volume mounts just fine if I turn off krb5 on both the client and the server.  I don't know what you mean by "add on authentication" beyond turning on sec=krb5 on both client and server.  Removing it and putting it back had not effect  :Wink: 

Thanks for the suggestions.

David

----------

## cdavidshaffer

 *Ant P. wrote:*   

> If you're using NFS4 you can get rid of a lot of needless moving parts - put "-N 2 -N 3" in your rpc.{mountd,nfsd} args. I don't know if Kerberos needs idmapd but NFS4 doesn't at all.
> 
> One other thing - are your clocks synced?

 

I have shut off all versions of NFS except 4 and 4.1.  No changes.  As far as I can tell, idmapd /is/ required for NFSv4: https://linux.die.net/man/8/idmapd.  OpenRC scripts shutdown NFS if I shut down idmapd.  Killing rpc.idmapd manually has no impact on problem.

Yes, my clocks are all synced to my KDC via NTP.

Thanks for the suggestions though.

David

----------

## SDubb

For what it's worth, I am having the exact same issue. Have you had  any luck getting it to work?

----------

## SDubb

I have made some progress! After littering nfs-utils with print statements trying to track down the problem, I discovered it's failing in the kernel:

```
nfs-client$ mount -v -t nfs4 -vvv -o sec=krb5 nfs-server.domain:/u99 /mnt/u99

mount.nfs4: timeout set for Wed Oct 19 10:21:04 2016

mount.nfs4: trying text-based options 'sec=krb5,addr=10.71.3.1,clientaddr=10.71.3.6'

mount.nfs4: mount(2): Invalid argument <-- emitted by the kernel

mount.nfs4: an incorrect mount option was specified
```

It turns out the key is CONFIG_RPCSEC_GSS_KRB5. It's located at:

```
File systems --->

  [*] Network File Systems --->

        <*>   Secure RPC: Kerberos V mechanism
```

It has some CONFIG_CRYPTO_ dependencies you'll need to satisfy to see it. I initially tried building it as a module; but, that produced the same errors. With it built into the kernel I now get a "Permission denied" error.

Hope that helps!

----------

## msst

Have been playing around with krb as well and finally given up on it. This auth method is about as complicated and badly diagnosable and undocumented as it gets.

Really poor choice of NFS to go for krb authentication.

----------

## depontius

I fiddled with this long ago, several times, and gave up each time.  But I believe I might have something valuable to contribute - a simple question.

Do you have kerberos working correctly, on its own?  Leave NFS entirely out of it, for the moment.

 - Is your server up and running and happy?

 - Can your clients authenticate to the server?

I know it's a bit of a silly question, but I haven't seen yet if anyone is able to klog successfully.

BTW, I was trying to get an integration of LDAP and Kerberos to work - much stickier.  I never got past working on the server, never got to the clients.

----------

