# Filesystems go missing

## Carnildo

My desktop computer recently developed a habit of losing access to its filesystems (physical, network, and virtual) a few hours after startup.  The root filesystem is still there, but everything else (/dev, /proc, /home, and so on) goes missing: the mountpoints show up as empty directories, or in the case of /dev, the static device nodes on the root filesystem.  The filesystems still show up as mounted, and any terminal with a working directory in one of the filesystems can still see its contents (but only if accessed using relative paths).

There are no relevant entries in dmesg or /var/log/messages, and the disappearance doesn't seem to be related to anything I do.  Nothing changed around the time the problem developed, but since I don't reboot my computer very often, this could be long-delayed fallout from an upgrade as much as three months ago.

I'm running kernel 3.6.11; other version info is available on request.

Any ideas?

----------

## audiodef

How often do you update?

----------

## Carnildo

I usually update once a week, but I'll often put off things like X or running applications.

----------

## wcg

At a guess, some corrupted data structure in the kernel's

dcache (directory cache) or inode cache. You can try to drop

caches when it happens and see if that fixes it.

```

echo 3 > /proc/sys/vm/drop_caches

```

(See /usr/src/linux/Documentation/sysctl/vm.txt.)

If that works, you can change the cache flush to

```

echo 2 > /proc/sys/vm/drop_caches

```

to skip freeing pagecache and only free the dcache and

inode cache and see if that works.

----------

## Carnildo

 *wcg wrote:*   

> At a guess, some corrupted data structure in the kernel's
> 
> dcache (directory cache) or inode cache. You can try to drop
> 
> caches when it happens and see if that fixes it.
> ...

 

Didn't fix it.

----------

## wcg

Well, the shell has a concept of "current working directory", but that

part seems to be working, because listing the contents of directories

relative to the current directory works. You can check it of course:

```

pwd

env | grep `pwd`

```

But I expect that the results will be what you expect them to be.

The problem seems to be absolute paths (beginning with '/').

I do not really see how that could be other than an inode or

dcache problem.

(Is the shell mangling the first character of the pathname as

a command parameter? Then it seems like it would mangle

the first character of relative pathnames, too, and for commands

with more command line options, the shell would not know

which one was the pathname.)

What about something like emacs' dired (directory editing

mode), which will list the contents of a directory in an

editing window? (You may not have emacs. I expect that

vim has some command that will do the same thing, like

```

vim /var

```

) nano apparently does not have that functionality by default.

"nano /tmp" simply reports that /tmp is a directory (meaning

not an editable text file).

Do you see the directory with an editor's directory listing

command when passing both absolute and relative paths

to the editor?

I guess it could be some filesystem bug. What kind of filesystem

is this?

----------

## Carnildo

 *wcg wrote:*   

> I guess it could be some filesystem bug. What kind of filesystem
> 
> is this?

 

All of 'em.  "/" is NFSv2 remounted as NFSv3, /home is NFSv3, "/dev" is a devtmpfs, "/proc" is procfs, "/run" and "/tmp" are tmpfs, "/sys" is a sysfs, "/mnt/cdrom" is an ISO9660 filesystem, "/mnt/memcard" is FAT32, and so on.

----------

## wcg

So probably not a filesystem bug. VFS at most.

This is not a problem that I see reported a lot. I suspect that

having root mounted over nfs has something to do with it,

although I do not know what exactly. Maybe nfs users will

have more insight or be able to suggest diagnostics.

----------

## TomWij

 *Carnildo wrote:*   

> The root filesystem is still there, but everything else (/dev, /proc, /home, and so on) goes missing: the mountpoints show up as empty directories, or in the case of /dev, the static device nodes on the root filesystem.  The filesystems still show up as mounted, and any terminal with a working directory in one of the filesystems can still see its contents (but only if accessed using relative paths).

 

Interesting, haven't heard about this, though it sounds like an udev or kernel bug.

 *Carnildo wrote:*   

> I'm running kernel 3.6.11; other version info is available on request.

 

Could you try a more recent kernel? The stable 3.8.13, the unstable 3.9.7 or the development 3.10-rc6?

----------

## Carnildo

 *TomWij wrote:*   

>  *Carnildo wrote:*   The root filesystem is still there, but everything else (/dev, /proc, /home, and so on) goes missing: the mountpoints show up as empty directories, or in the case of /dev, the static device nodes on the root filesystem.  The filesystems still show up as mounted, and any terminal with a working directory in one of the filesystems can still see its contents (but only if accessed using relative paths). 
> 
> Interesting, haven't heard about this, though it sounds like an udev or kernel bug.

 

That's the conclusion I came to as well, after having the problem with ever-more-minimal systems (I'd narrowed it down to one of udev, bash, init, md5sum, or the kernel, and I'm pretty sure that bash and md5sum running as an unprivileged user can't cause it).

 *Quote:*   

>  *Carnildo wrote:*   I'm running kernel 3.6.11; other version info is available on request. 
> 
> Could you try a more recent kernel? The stable 3.8.13, the unstable 3.9.7 or the development 3.10-rc6?

 

3.8.13 has been running without trouble for a little over 24 hours, which is longer than 3.6.11 ever managed once things started going wrong.

----------

## augury

Your disk may be going bad.  It's a physical thing.  Overheating, dirt.

What sort of spindles are you running?  Any noise? Difficulty starting up?  You can *usually* save the data but the reliability will decrease.  The performance hit isn't to bad.

----------

