# /sys and /proc get unmounted during boot

## oleid

Dear *,

I'm trying to boot gentoo/systemd via NFS as rootfs, when I stumble across the following problem: It seems, as if at some point /proc, /sys and /dev get unmounted.

This is best observable when booting to a emergency shell (using the keyword emergency at the kernel command line) and repeatedly executing stuff like:

mount

ls /sys

ls /proc

in the very rescue shell. At some point proc and sys are empty and thus mount doesn't show any mountpoints ( /etc/mtab -> ../proc/self/mounts ).

I'm not sure, if this is in *any* way related to systemd. It could also be some kernel problem, I really don't know.

The system is mostly amd64, the kernel being a notable exception (gentoo-sources-3.18.5). 

I used the default config file from ArchLinux (plus the additional stuff I get asked due to make oldconfig) and thus I used the same kernel version which they are using.

Debugging ( I boot with: debug initcall_debug udev.log_priority=8  ignore_loglevel) sheds no light onto this situation. There is no suspicious output before the problem and no additional output after the unmount.

So my question:

Did *anybody* notice an unmounting of /proc and /sys on your systems? What might cause such an unmount?

Edit:

I should add, that I installed a numbercrunching cluster, whose nodes boot via NFS as rootfs, using gentoo and systemd  about a year ago. So it's definitively possible. But it uses and older kernel and an older systemd version. I might try the older systemd version later on, I just realized it's still in the tree. At the moment, I'm copying the installation from the file server to one of the clients to test local boot.

Edit2:

Okay, booting the copy of the image from a HD works fine.

Edit3:

systemd-215-r3 (the same version as on the working cluster nodes mentioned in the first edit) doesn't do the trick. Strange world.

Best,

Olaf

----------

## DONAHUE

Do you specify a "init=/usr/lib64/systemd/systemd" eqivalent appropriate for your NFS (after the kernel has loaded) in the kernel cmdline built into the kernel or the bootloader? kernel config has? 

```
grep -i systemd /usr/src/linux/.config
```

  *Quote:*   

> CONFIG_GENTOO_LINUX_INIT_SYSTEMD=y

  Supposedly just provided to autoselect systemd support but might affect functionality if unselected.?.

----------

## oleid

 *DONAHUE wrote:*   

> Do you specify a "init=/usr/lib64/systemd/systemd" eqivalent appropriate for your NFS (after the kernel has loaded) in the kernel cmdline built into the kernel or the bootloader? kernel config has? 
> 
> ```
> grep -i systemd /usr/src/linux/.config
> ```
> ...

 

Yes to both. The system boots just fine when not using NFS as rootfs, but e.g. btrfs. Even when mounting btrfs ro.

The initramfs is quite simple. I use archlinux's mkinitcpio which did the job well for me for the other server (in contrast to dracut or genkernel). The code paths are nearly the same for NFS and local mount.

I brought the topic to systemd-devel mailing list[1]. They suggested to strace all the running stuff. I'll do that this afternoon.

[1] http://lists.freedesktop.org/archives/systemd-devel/2015-February/027927.html

----------

## LubosD

I am seeing an identical issue when using nfsroot and OpenRC. I know for *sure* that procfs is mounted at boot time. Minutes later, while using the system, it suddenly disappears (unmounts itself) and applications start complaining.

----------

## oleid

What kernel are you using? For me, the problem appeared on gentoo-sources-3.18, going back to 3.17 worked around the problem.

Oh, by the way, I could roule out systemd as source of the problem, as  PID1 and children didn't yield any syscalls when the unmount disappeared. There is more information on what I tested in the email thread.

----------

## LubosD

 *oleid wrote:*   

> What kernel are you using? For me, the problem appeared on gentoo-sources-3.18, going back to 3.17 worked around the problem.

 

I'm currently running 3.19.1. I was previously running 3.3.5 and it worked just fine.

----------

## LubosD

I have now filed a kernel bug: https://bugzilla.kernel.org/show_bug.cgi?id=104081

I'll also try to find the problematic commit.

----------

## LubosD

It seems that this commit fixes the problem:

```

commit 6c441c254eea2354d686be7f5544bcd79fb6a61f

Author: Trond Myklebust <trond.myklebust@primarydata.com>

Date:   Sun Feb 22 16:35:36 2015 -0500

    NFS: Don't invalidate a submounted dentry in nfs_prime_dcache()

    

    If we're traversing a directory which contains a submounted filesystem,

    or one that has a referral, the NFS server that is processing the READDIR

    request will often return information for the underlying (mounted-on)

    directory. It may, or may not, also return filehandle information.

    

    If this happens, and the lookup in nfs_prime_dcache() returns the

    dentry for the submounted directory, the filehandle comparison will

    fail, and we call d_invalidate(). Post-commit 8ed936b5671bf

    ("vfs: Lazily remove mounts on unlinked files and directories."), this

    means the entire subtree is unmounted.

    

    The following minimal patch addresses this problem by punting on

    the invalidation if there is a submount.

    

    Kudos to Neil Brown <neilb@suse.de> for having tracked down this

    issue (see link).

    

    Reported-by: Nix <nix@esperi.org.uk>

    Link: http://lkml.kernel.org/r/87iofju9ht.fsf@spindle.srvr.nix

    Cc: stable@vger.kernel.org # 3.18+

    Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com>

```

----------

