# nfs-utils-2.4.1-r1 on x86 nfs server borks server

## figueroa

Two days ago on doing routine updates, net-fs/nfs-utils-2.4.1-r1 was installed. After rebooting the server and a client due to a kernel upgrade to 4.9.192 last night (kernel upgrade does not appear to be related to the problem), I could no longer mount the server's nfs shares on the client.

The solution was to mask net-fs/nfs-utils-2.4.1-r1 on the server, then downgrading to net-fs/nfs-utils-2.3.3. After restarting nfs and nfsclient on the server, I was again able to mount nfs shares. I did not have to change anything on the client, also a Gentoo machine running amd64. Both machines are running stable.

The error I got on the client was: mount.nfs: access denied by server while mounting

The error in the server log was: [rpc.mountd] can't stat [local directory] exported dir invalid argument

I got hints searching the Internet for the errors at: https://superuser.com/questions/1451826/nfs-cant-stat-exported-dir-invalid-argument

I'm sure others will be bit by this and masking a stable package isn't really a long-term solution, but I trust others will find this posting helpful for the short-term.

----------

## Aiken

For me the relevant bug is https://bugs.gentoo.org/688644

I left all the client machines at the new version. Only the server needed to be downgraded to find a working version.

----------

## figueroa

A commit to fix this bug (re bugzilla) with net-fs/nfs-utils-2.4.1-r2 did not do so.  Routine update to that package on my x86 server crashed the mounts on my 64 bit client and the nfs mounts will not remount with error: mount.nfs: access denied by server while mounting ...

Very disappointing.

Added: Oh, and I can no longer downgrade. 2.4.1-r2 is the ONLY version available of nfs-utils. I'm out of business!

----------

## Aiken

I ended up following the 2nd last comment in this thread https://forums.gentoo.org/viewtopic-t-1070080-start-0.html to get a working ebuild back.

With 2.4.1-r2 I still get the "can't stat exported dir /definitely/a/valid/path: Invalid argument" message on the server. This is all 64 bit. The above got me an ebuild for a working version of nfs-utils. For the time being I have masked it with >=net-fs/nfs-utils-2.4.1-r1 keeping the older version in my local tree.

----------

## figueroa

Thanks, Aiken. That got me started. I've got excellent system backups so I didn't have to go on the hunt for packages. I had the files needed in my backups so I sort of followed this post https://forums.gentoo.org/viewtopic-p-7325834.html to make a local ebuild for nfs-utils-2.3.3 in /usr/local/portage/net-fs/nfs-utils/  Thankfully, I hadn't cleaned out the distfile which I don't keep in my backups. While I was at it, I stashed a copy of the distfile in /usr/local/portage/distfiles/ for insurance.

I don't k now why 2.4.1-r2 didn't work and 2.3.3 does, but I've got my interim fix.

----------

## tld

That sounds like an ugly one. I ran into an issue with the update to 2.4.1-r1 (all cases mounting from x86 to x86 however) where everything got a "Stale file handle". In that case applying the patch that was added in 2.4.1-r2 (nfs-utils-2.4.1-Fix-include-order-between-config.h-and-stat.h.patch that addresses large file support) to the 2.4.1-r1 ebuild corrected this, and was only needed on the server:

https://forums.gentoo.org/viewtopic-t-1102304-highlight-.html

It sounds to me like there's a lot that's shaky in the new version. The above mentioned bug seems to describe a somewhat different issue from mine, and yours as well. That one especially threw me because I'm so used to NFS just plain working that I barely ever think about it.

The premature removal of old ebuilds seems to be getting rampant lately, and this is a pretty good example. I've had several recent cases where I've needed to recover old ebuilds. It took me forever to figure out an easy means of doing so, but here it is:

Recovering deleted ebuilds / files:

1. Go to https://github.com/gentoo/gentoo/ and go to the appropriate package.

2. Click the "History" button.

3. Find the commit immediately before the deletion, and click the link to the right of it labeled "<>".

4. At that point you're browsing the tree as it existed just before the deletion, and you can, again navigate to the appropriate package and all the deleted files will be there.

Tom

----------

## depontius

Many of the references I've seen have simply been to "nfs", but some have been to "nfs v3".  I'm running nfsv4, and I've upgraded the nfs-utils to the bad version, but not rebooted yet.  Do I have a ticking time-bomb here, or is nfsv4 not affected?

----------

## figueroa

 *depontius wrote:*   

> Many of the references I've seen have simply been to "nfs", but some have been to "nfs v3".  I'm running nfsv4, and I've upgraded the nfs-utils to the bad version, but not rebooted yet.  Do I have a ticking time-bomb here, or is nfsv4 not affected?

 

No need to reboot. Just restart nfs and see what happens to your nfs clients. Please let us know.

----------

## depontius

 *figueroa wrote:*   

>  *depontius wrote:*   Many of the references I've seen have simply been to "nfs", but some have been to "nfs v3".  I'm running nfsv4, and I've upgraded the nfs-utils to the bad version, but not rebooted yet.  Do I have a ticking time-bomb here, or is nfsv4 not affected? 
> 
> No need to reboot. Just restart nfs and see what happens to your nfs clients. Please let us know.

 

I've got a new unbooted kernel sitting on the machine, and would just as soon switch to it.  However before I do anything I want to fetch the old ebuild and do an "ebuild install" with it, so I'm an "ebuild merge" away from being back in operation.

----------

## tld

In my case I've yet to use anything other than NFS v3.

Tom

----------

## depontius

Got the files pulled back, "ebuild install" done and a tarball of the results saved aside in case something happens before I get to this.  I'm not sure if /var/tmp gets cleaned on reboots.  Anyway, it's backed up and can be quickly restored.

I'm upgrading nfs-utils from 2.4.1-r1 to 2.4.1-r2, in case that fixes my problem where it didn't for others.  So when it's convenient for the family, I'll give this a try and post the results.

----------

## depontius

Tried nfs-utils-2.4.1-r2 this morning, and it most certainly failed for my nfsv4 setup.  I didn't have a lot of time to inspect the fail signature or anything, since this was a delay in getting to work.  Luckily I had nfs-utils-2.3.3 already built, and just had to merge it, so the home cluster is back up and running.

A giant thanks for seeing this in the forum - getting a heads-up on what I probably would have blamed on a kernel upgrade otherwise.  I'm also about to be gone for a few days, and leaving the family with a mess like this would have been would not have been good.

----------

## Lars

Hi all,

I found a message about nfs-utils 2.4.1 here: https://superuser.com/questions/1451826/nfs-cant-stat-exported-dir-invalid-argument

It seems we need nfs-utils 2.3.3 as long as kernel 4.9.x is supported.

I have the same problem here on one of my servers I need an old 4.9.x Kernel to use an USB Printer. Also I need nfs on this server. And also nfs-utils-2.4.1-r2 did not work for me.

Kind regards.

----------

## depontius

I'm running a 4.19 kernel on my server and 5.2 kernels on my clients, and this problem still hit me.

In any event, it sounds as if the Gentoo power-that-be should bring nfs-utils-2.3.3 back into active portage.  I wonder how many of us have it in local tree, now.

----------

## Aiken

I had previously seen the link Lars posted. My nfs server is stuck on 4.9.159 for a bit longer and all of my client machines are 4.19.72.

Running grep -r statx /usr/src/linux/include on both the 4.9 and 4.19 machines shows statx being found in the 4.19 tree.

----------

## Lars

I'm so stupid!   :Sad: 

I was to fast with updates, now the good old 

```
nfs-utils-2.3.4.ebuild
```

 is gone.

One simple way to get it back is to extract this old ebuild from the current repo.

https://gitweb.gentoo.org/repo/gentoo.git/tree/net-fs/nfs-utils/nfs-utils-2.3.4.ebuild?id=baf42f13f34565100707f3fcba17fa2f0d0a0403  :Very Happy: 

Copy it into an own layout and only use this version.

Works as expected.

----------

## figueroa

Surely this is a big enough deal to deserve a NEWS item?

----------

## krinn

 *figueroa wrote:*   

> Surely this is a big enough deal to deserve a NEWS item?

 

So your news will be?

"If you upgrade to nfs-utils-2.4.1-r2 and if the sun is high, under a rainy day, and if you have never win the lotery, and if you saw a butterfly in your kitchen... maybe you will have problem ; if you have a problem we suggest you to not upgrade"

all this thread is about "it doesn't work if i don't downgrade"...

and all you do is blaming nfs-utils without even trying to figure out what is doing that (because you need nfs-utils AND something else to trigger the bug, for now, the best bet should be a glibc version, a clue given by the reporter of bug 688644)

is any of you at least tried the patch from bug 688644 -> https://688644.bugs.gentoo.org/attachment.cgi?id=580728 ?

because the best info so far is only given by this bug, and if the patch fix your issue, it would confirm the bug and a solve may come fast ; and in case you didn't see, the bug was close RESOLVED FIXED (wrongly mark solve, as -r2 fix the other issue and not this one) ; which mean you will wait for an nfs-utils-2.4.2 fix rather than having an nfs-utils-2.4.1-r3 fix fast

In case you miss, i have myself no issue with nfs-utils-2.4.1-r1, and if i'm not affected by this bug, gentoo devs may not be affected too and won't see it, and none of affected users are helping, the reporter who was affect have provide the patch, now the bug is closed, and reporter still have a patch version that work, and he might think the issue is indeed close (because for him, the issue is close)

----------

## Lars

I will try it, and report. But not before this evening. Thanks for the blow to the neck.

----------

## figueroa

 *krinn wrote:*   

>  *figueroa wrote:*   Surely this is a big enough deal to deserve a NEWS item? 
> 
> So your news will be?
> 
> "If you upgrade to nfs-utils-2.4.1-r2 and if the sun is high, under a rainy day, and if you have never win the lotery, and if you saw a butterfly in your kitchen... maybe you will have problem ; if you have a problem we suggest you to not upgrade"
> ...

 

How does one apply a patch to an ebuild, step-by-step? I'm not a developer or progammer. I'm a user trying to get by and sometimes help others.

Why would stable glibc cause stable nfs-utils to break? It's not supposed to work like that?

----------

## depontius

 *krinn wrote:*   

>  *figueroa wrote:*   Surely this is a big enough deal to deserve a NEWS item? 
> 
> So your news will be?
> 
> "If you upgrade to nfs-utils-2.4.1-r2 and if the sun is high, under a rainy day, and if you have never win the lotery, and if you saw a butterfly in your kitchen... maybe you will have problem ; if you have a problem we suggest you to not upgrade"
> ...

 

I'll just say that on my home cluster, with the server running a 4.19 series stable kernel and clients running newer than that, nfs-utils-2,4,1 killed my mounts as well.  I discovered this on the last evening before leaving on a trip, so I had no time for details, no time for debug.  At the moment I don't even know if it was running nfs-utils-2.4.1-r1 or nfs-utils-2.4.1-r2 - I could fix by downgrading and I did.  I'm far away from home now, and I'll pick this back up when I get home.

----------

## krinn

 *figueroa wrote:*   

> How does one apply a patch to an ebuild, step-by-step? I'm not a developer or progammer. I'm a user trying to get by and sometimes help others.

 

Sorry (i should have tell), here it is https://wiki.gentoo.org/wiki//etc/portage/patches

 *figueroa wrote:*   

> Why would stable glibc cause stable nfs-utils to break? It's not supposed to work like that?

 

But it might be something else than glibc (kernel-headers?), but isn't a "bug" the perfect definition of "It's not supposed to work like that"?  :Smile: 

I'm sorry i cannot help on the issue myself because i don't have the issue, all i could "help" with is showing you my server/client version (i'm not really in mood to upgrade glibc to confirm it is glibc related, i just trust the reporter who seems to have put the finger on the problem perfectly).

server: Portage 2.3.5 (python 2.7.13-final-0, default/linux/x86/13.0, gcc-5.4.0, glibc-2.22-r4, 3.18.47 i686)

client:Portage 2.3.73 (python 3.6.9-final-0, default/linux/amd64/17.1, gcc-9.2.0, glibc-2.28-r6, 4.14.67 x86_64)

client mount one share as nfsv3, other are nfsv4 (reporter say it use only nfsv3, but i think nfsv4 mounts should be affected too)

Confirming this would help you, but help other users too ; so thanks for your testing efforts.

----------

## Lars

Good news, with the given patch attached to https://bugs.gentoo.org/688644 the patched nfs-utils-2.4.1-r2 works on my kernel 4.9.193 Linux box. The filesystem is mountable by other box with kernel 5.3.4.

With current nfs-utils-2.4.1-r2 the filesystem is not mountable from other box with kernel 5.3.4.

I would like to give an ebuild also, but this stupid 'sed' hates me.

```
sed -e "s/if \(errno == ENOSYS\)/if \(errno == EINVAL || errno == ENOSYS\)/" -i support/misc/xstat.c
```

What is wrong with this line?

----------

## Lars

I do not need to quote the round backets.

```
sed -e 's/if (errno == ENOSYS)/if (errno == EINVAL || errno == ENOSYS)/' -i support/misc/xstat.c
```

 :Embarassed: 

----------

## krinn

That's more than one good news, the confirmation of fix and that the dev in the bug is now aware the issue is still there

----------

## figueroa

nfs-utils-2.4.1-r3 has now gone stable:

```
Calculating dependencies  ......... ..... . ...... ..... . ..... .... ... done!

[ebuild     U  ] net-fs/nfs-utils-2.4.1-r3 [2.3.3]
```

Do I need to mask this version too, or are the problems serving nfs3 to a client actually fixed? 2.4.1-r3 is the only ebuild present in /usr/portage/net-fs/nfs-utils/. I have 2.3.3 in my local repository.

----------

## krinn

It looks clear -r3 is made for that, if you are scared, nothing prevent you from quickpkg nfs-utils before upgrading, but your issue should be fixed by -r3

 *Quote:*   

>     net-fs/nfs-utils: Revbump to fix issue with old kernels and statx
> 
>     Bumped straight to stable as this seems to affect many users.
> 
> 

 

----------

## figueroa

 *krinn wrote:*   

> It looks clear -r3 is made for that, if you are scared, nothing prevent you from quickpkg nfs-utils before upgrading, but your issue should be fixed by -r3
> 
>  *Quote:*       net-fs/nfs-utils: Revbump to fix issue with old kernels and statx
> 
>     Bumped straight to stable as this seems to affect many users. 

 

Just a note to confirm that the issue appears to be resolved in current version. I upgraded, and restarted nfs on the server. My client appeared to remained connected; unmounting then remounting was uneventful. Happy.

----------

## krinn

I think thanks should goes to Lars, he was the one testing and confirming the bug, making the solve comes fast

----------

