# EXT4 Filesystem Full with "Ghost Data"? [SOLVED]

## Dr. Frankenbox

Not sure if this is the right forum for this, but. . .

My root filesystem (ext4) is nearly full, but I can't find the bulk of the data in any file.  Compare the following df and du output:

```
frankenbox / # df -h .

Filesystem      Size  Used Avail Use% Mounted on

/dev/root        90G   83G  2.4G  98% /

frankenbox / # du -sh * --exclude=mnt

8.6M    bin

47M     boot

152K    dev

4.4M    etc

35G     home

0       lib

5.3M    lib32

158M    lib64

16K     lost+found

4.0K    media

172M    opt

du: cannot access `proc/17274/task/17274/fd/4': No such file or directory

du: cannot access `proc/17274/task/17274/fdinfo/4': No such file or directory

du: cannot access `proc/17274/fd/4': No such file or directory

du: cannot access `proc/17274/fdinfo/4': No such file or directory

0       proc

32M     root

11M     sbin

0       sys

68K     tmp

13G     usr

1.5G    var

```

I excluded /mnt because I have other filesystems mounted there, and yes, I verified that the only data under /mnt is in those other filesystems.

The largest thing in that list is /home, at 35 GB.  While that's healthy, there's no way the rest of those numbers add up to 48GB. I've heard that a Linux filesystem can build up "used" space that isn't actually in any file over time, and I've been using this one for some time (probably over a year; I haven't really been keeping track).  Are there any utilities I can run to investigate this further and/or fix the problem if it is what I think?Last edited by Dr. Frankenbox on Sun Apr 29, 2012 3:28 pm; edited 1 time in total

----------

## Hu

I am not aware of any Linux filesystems that leak space.  Such behavior would be a bug that should be detected and fixed by the corresponding fsck.  However, if you have deleted a file and not closed it, it will still consume space.

----------

## toralf

 *Hu wrote:*   

> I am not aware of any Linux filesystems that leak space.  Such behavior would be a bug that should be detected and fixed by the corresponding fsck.  However, if you have deleted a file and not closed it, it will still consume space.

 Right, the command "lsof" should point to such data.

----------

## Dr. Frankenbox

But when I shut down and close the process(es) using those files, the space should be freed up - right?  I certainly haven't deleted anything that large since I last booted (yesterday morning), and while I could believe I have some buggy processes on my system that are deleting files without closing them (or deleting each other's files without proper checking), I seriously doubt they could cause a problem like this in one day.

I will run lsof and see what it says, though.  I just have to merge it first.

edit: Well, that was quick.  There is a huge number of files open on my system (22,303), and many files have been opened by multiple processes, but the largest file in the list is only about 104MB. I seriously doubt it all adds up to 20-30GB, and most of these files probably haven't been deleted.  Sill, is it normal for a running Gentoo desktop system with a GUI and a few applications open (I count 7 windowed apps, visible to the user) to have so many files open at a time?

----------

## Dont Panic

 *Dr. Frankenbox wrote:*   

> I excluded /mnt because I have other filesystems mounted there, and yes, I verified that the only data under /mnt is in those other filesystems.

 

Did you unmount the drives to see if you had any data that was being hidden by the mount point?

If you ever copied data to a directory in /mnt/ with nothing mounted at that point, and then mounted a partition on top of that point, it will hide the contents that were previously copied there unless you mounted with '-o bind'.

----------

## Ant P.

 *Dr. Frankenbox wrote:*   

> edit: Well, that was quick.  There is a huge number of files open on my system (22,303), and many files have been opened by multiple processes, but the largest file in the list is only about 104MB. I seriously doubt it all adds up to 20-30GB, and most of these files probably haven't been deleted.  Sill, is it normal for a running Gentoo desktop system with a GUI and a few applications open (I count 7 windowed apps, visible to the user) to have so many files open at a time?

 

Looks normal to me:

```
# lsof -n | wc -l

28915
```

----------

## Dr. Frankenbox

 *Dont Panic wrote:*   

>  *Dr. Frankenbox wrote:*   I excluded /mnt because I have other filesystems mounted there, and yes, I verified that the only data under /mnt is in those other filesystems. 
> 
> Did you unmount the drives to see if you had any data that was being hidden by the mount point?
> 
> If you ever copied data to a directory in /mnt/ with nothing mounted at that point, and then mounted a partition on top of that point, it will hide the contents that were previously copied there unless you mounted with '-o bind'.

 

There it is!

```
frankenbox mnt # mount

rootfs on / type rootfs (rw)

/dev/root on / type ext4 (rw,noatime,commit=0)

proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)

rc-svcdir on /lib64/rc/init.d type tmpfs (rw,nosuid,nodev,noexec,relatime,size=1024k,mode=755)

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)

debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)

udev on /dev type tmpfs (rw,nosuid,relatime,size=10240k,mode=755)

devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620)

shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime)

/dev/sda6 on /mnt/common type ext3 (rw,noatime,commit=0)

/dev/sdb1 on /mnt/bk type ext3 (rw,noatime,commit=0)

usbfs on /proc/bus/usb type usbfs (rw,noexec,nosuid,devmode=0664,devgid=85)

/dev/sda2 on /mnt/windows type ntfs (rw,dmask=000,fmask=111,nls=utf8)

frankenbox mnt # umount windows/

frankenbox mnt # umount common/

frankenbox mnt # umount bk

frankenbox mnt # ls

bk  cdimage  cdrom  common  floppy  gateway  tmp  windows

frankenbox mnt # du -sh *

32G     bk

4.0K    cdimage

4.0K    cdrom

4.0K    common

4.0K    floppy

4.0K    gateway

4.0K    tmp

4.0K    windows

```

Apparently at some point I had /mnt/bk unmounted while my backup scripts ran and happily saved my backups to my root partition instead of the separate drive I have reserved for them.  Thank you for thinking of that, because I certainly wouldn't have.

So it turns out that I am really the fool here.  My apologies to any ext4 developers I may have offended.

----------

## Hu

As an alternate way to check for this problem, you can use bind mounts to expose directories that are normally hidden.  In your case, you could have skipped the umount commands and instead run: mount --bind / /mnt/floppy && cd /mnt/floppy && du -sh * && umount /mnt/floppy.  When a filesystem is bind-mounted into a new place, mounts beneath the original are not (by default) mounted in their corresponding place beneath the new place.  Thus, the command I gave above would allow /mnt/floppy to show everything in /, even those things normally hidden by your active mounts.  This is of marginal use in your case, but is very handy when the offending file is hiding under a popular mount point, such as the dev or var directories on root.  Unmounting the filesystems that normally hide those directories is quite difficult in a running system, but exposing the contents via a bind mount is easy.

----------

