# Disk space on SSD mysteriously used, [SOLVED]

## peaceful

I've got a perplexing mystery!

My SSD has disk space getting consumed by several megabytes PER MINUTE, and I can't find any files that are actually using the space!  I use "du -h" extensively, and the total used stays the same, but "df" reports my disk space dwindling -- and eventually the hard drive fills up completely and things malfunction, though using du I still can't find any culprit!

So I reboot.  Then all the space used that 'du' couldn't find reappears, but the behaviour continues....so I reboot, and reboot, and reboot every day.  But now it's filling up faster and it's almost full by the end of the day...

???

My setup:

1U server: Used as an Asterisk server (PBX for SIP phones, passes calls to a different asterisk server if they need to go out to the real telephone network)

Linux voip1 2.6.26-gentoo-r4 #3 SMP PREEMPT Sun Sep 20 21:04:05 MDT 2009 x86_64 Intel(R) Xeon(R) CPU E5410 @ 2.33GHz GenuineIntel GNU/Linux

Partition with the mysterious problem:

* /dev/md3 Intel X-25-E SSD (30GB) (x 2 in a mirrored software raid)

Other partitions/disks on the system:

* /dev/md1 mirrored boot partition on same SSD as above

* A ramdisk for recording calls to a file, mounted at /var/spool/asterisk/monitor

* /dev/sdc1 A traditional "500GB" hard drive for moving recorded calls to and storing them long term

Here's what df -h shows right now

```
# df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/md3               29G   20G  7.5G  73% /

udev                   10M  180K  9.9M   2% /dev

/dev/md1               69M  8.5M   57M  14% /boot

none                   24G   11G   14G  45% /var/spool/asterisk/monitor

/dev/sdc1             459G   16G  421G   4% /root/gathered

shm                    16G     0   16G   0% /dev/shm
```

Here's the last bit of what du -h shows right now:

```
# du -h | tail

3.3M   ./bin/panel

3.7M   ./bin

24K   ./.ssh

4.0K   ./.subversion/auth/svn.ssl.server

4.0K   ./.subversion/auth/svn.username

4.0K   ./.subversion/auth/svn.simple

4.0K   ./.subversion/auth/svn.ssl.client-passphrase

20K   ./.subversion/auth

48K   ./.subversion

15G   .

```

Notice the discrepency!!!  I can only find 15G of files on /dev/md3, but df shows 20G used!!!    :Shocked: 

NOTE that I have tried unmounting both the regular hard drive and the ramdisk and the behaviour of df and du does not change at all (except that obviously those disks no longer show up in the df listing).

Any ideas????  Where the heck is the space going???  Why does it magically reappear after a reboot?  Does kernel .26 hate SSDs?  Are Intel's SSDs buggy?  Am I just insane?  Am I missing something obvious?  This is driving me NUTS!  I have those same SSDs in about 6 other servers, a couple of which are also Gentoo, and none of them exhibit this behaviour.Last edited by peaceful on Thu Mar 25, 2010 5:43 pm; edited 1 time in total

----------

## eccerr0r

Might be suffering from the deleted filehandle issue.

If you create a file and keep on appending to it and at the same time delete the file (while the file creator still has a handle on the file) du will no longer report the space consumed but df will continue to track usage of that file.  Rebooting will kill the process and release the handle, and the space is reclaimed.  Killing the process will also have the same effect.

The other possibility is that you have the "missing" files underneath the mountpoint.  Unmounting all partitions and checking may reveal something.  But if it goes away on reboot this isn't likely.

Now I don't know what software you use that does this kind of behavior... no idea about how asterisk works...

----------

## Hu

The explanation offered by eccerr0r seems most likely.  Use lsof to search for files which are open and unlinked.  Given the growth rate, I would suspect either a misconfigured logging daemon or the Asterisk process itself.  Several megabytes a minute for an entire day seems excessive for a server which does only what you have described so far.

----------

## peaceful

Thanks for all the suggestions!

We finally found a tomcat log file that was spewing out debugging information -- the logfile just wasn't showing up under "du" -- probably for one of the reasons you guys mentioned.  So it really was real file usage, that really wasn't showing up in listings.  So weird, but I feel more sane now.

----------

