# [solved] IOError: Errno 28 No space left on device

## midway

Hello All,

On an emerge system I got 55 updates which failed complaining about no disk space left. I am attaching the failed error below. Someone mentioned on net to increase the inode options. Could you describe the steps or point me to a guide at-least? I am in the middle of updating this system. 

```

 * To avoid automounting and auto(un)installing with /boot,

 * just export the DONT_MOUNT_BOOT variable.

 *

 *

 * memtest has been installed in /boot/memtest86plus/

 * You may wish to update your bootloader configs

 * by adding these lines:

 *  - For grub2 just run grub-mkconfig, a configuration file is installed

 *    as /etc/grub/39_memtest86+

 *  - For grub legacy: (replace '?' with correct numbers for your boot partition)

 *     > title=memtest86+

 *     > root (hd?,?)

 *     > kernel /boot/memtest86plus/memtest

 *  - For lilo:

 *     > image  = /boot/memtest86plus/memtest

 *     > label  = memtest86+

 *

 * Messages for package dev-libs/gmp-5.0.5:

 * Old versions of installed libraries were detected on your system.

 * In order to avoid breaking packages that depend on these old libs,

 * the libraries are not being removed.  You need to run revdep-rebuild

 * in order to remove these old dependencies.  If you do not have this

 * helper program, simply emerge the 'gentoolkit' package.

 *   # revdep-rebuild --library '/usr/lib/libgmp.so.3' && rm '/usr/lib/libgmp.so.3'

 * Messages for package sys-fs/squashfs-tools-4.2-r1:

 * This version of mksquashfs requires a 2.6.29 kernel or better

 * XZ support requires a 2.6.38 kernel or better

 * Messages for package dev-libs/mpfr-3.1.1:

 * Old versions of installed libraries were detected on your system.

 * In order to avoid breaking packages that depend on these old libs,

 * the libraries are not being removed.  You need to run revdep-rebuild

 * in order to remove these old dependencies.  If you do not have this

 * helper program, simply emerge the 'gentoolkit' package.

 *   # revdep-rebuild --library '/usr/lib/libmpfr.so.1' && rm '/usr/lib/libmpfr.so.1'

 * Messages for package dev-php/adodb-5.17:

 * ADODB requires some form of SQL or ODBC support in your PHP.

 * Messages for package dev-lang/perl-5.16.1:

 * UPDATE THE PERL MODULES:

 * After updating dev-lang/perl you must reinstall

 * the installed perl modules.

 * Use: perl-cleaner --all

 * You must reinstall perl-core/ExtUtils-ParseXS !

 * The complete build log is located at '/var/tmp/portage/dev-lang/perl-5.16.1/temp/build.log'.

 * The ebuild environment file is located at '/var/tmp/portage/dev-lang/perl-5.16.1/temp/environment'.

 * Working directory: '/var/tmp/portage/dev-lang/perl-5.16.1/work/perl-5.16.1'

 * S: '/var/tmp/portage/dev-lang/perl-5.16.1/work/perl-5.16.1'

 * Regenerating GNU info directory index...

 * Processed 109 info files; 46 errors.

install-info: /usr/share/info/dir: empty file

install-info: /usr/share/info/dir: empty file

[this repeated 46 times]

Traceback (most recent call last):

  File "/usr/bin/emerge", line 50, in <module>

    retval = emerge_main()

  File "/usr/lib/portage/pym/_emerge/main.py", line 2073, in emerge_main

    trees, mtimedb, retval)

  File "/usr/lib/portage/pym/_emerge/main.py", line 402, in post_emerge

    mtimedb.commit()

  File "/usr/lib/portage/pym/portage/util/mtimedb.py", line 122, in commit

    encoding=_encodings['repo.content'], errors='strict'))

  File "/usr/lib/portage/pym/portage/util/__init__.py", line 1185, in write

    return f.write(s)

IOError: [Errno 28] No space left on device

# revdep-rebuild

/usr/bin/revdep-rebuild: line 128: cannot create temp file for here-document: No space left on device

 * Configuring search environment for revdep-rebuild

/usr/bin/revdep-rebuild: line 1080: cannot create temp file for here-document: No space left on device

/usr/bin/revdep-rebuild: line 1089: cannot create temp file for here-document: No space left on device

/usr/bin/revdep-rebuild: line 1090: cannot create temp file for here-document: No space left on device

/usr/bin/revdep-rebuild: line 1091: cannot create temp file for here-document: No space left on device

/usr/bin/revdep-rebuild: line 1100: cannot create temp file for here-document: No space left on device

 * No search defined -- this is a bug.

```

Here are more information:

```

 # cat /etc/fstab

/dev/sda1 /boot xfs defaults,noatime 0 1

/dev/sda2 swap swap sw 0 0

/dev/sda3 / xfs noatime 0 1

/dev/cdrom2 /mnt/cdrom auto defaults,noauto 0 0

shm /dev/shm tmpfs nodev,nosuid,noexec 0 0

# du -h --one-file-system --max-depth=1

0       ./.ICE-unix

0       ./.X11-unix

0       ./.winbindd

0       ./hsperfdata_root

0       .

# df -i

Filesystem     Inodes  IUsed  IFree IUse% Mounted on

rootfs         536720 499207  37513   94% /

/dev/sda3      536720 499207  37513   94% /

tmpfs          129132    168 128964    1% /run

rc-svcdir      129132     76 129056    1% /lib/rc/init.d

cgroup_root    129132      4 129128    1% /sys/fs/cgroup

udev           129132    500 128632    1% /dev

shm            129132      1 129131    1% /dev/shm

/dev/sda1      152576     47 152529    1% /boot

# df -hT

Filesystem     Type    Size  Used Avail Use% Mounted on

rootfs         rootfs   48G   48G   20K 100% /

/dev/sda3      xfs      48G   48G   20K 100% /

tmpfs          tmpfs   505M  140K  505M   1% /run

rc-svcdir      tmpfs   1.0M  100K  924K  10% /lib/rc/init.d

cgroup_root    tmpfs    10M     0   10M   0% /sys/fs/cgroup

udev           tmpfs    10M  4.0K   10M   1% /dev

shm            tmpfs   505M     0  505M   0% /dev/shm

/dev/sda1      xfs     145M   43M  103M  30% /boot

# du -hx --max-depth=1 /

4.0K    /boot

8.8M    /bin

0       /dev

15M     /etc

195M    /lib

0       /mnt

205M    /opt

0       /tmp

0       /sys

40G     /var

6.5G    /usr

0       /proc

64M     /home

9.1M    /sbin

788M    /root

12K     /.gnupg

0       /run

48G     /

```

Last edited by midway on Fri Aug 17, 2012 11:04 am; edited 1 time in total

----------

## aCOSwt

 *nitish.anand wrote:*   

> Someone mentioned on net to increase the inode options.
> 
> ```
> 
> # df -hT
> ...

 

I am afraid that the number of inodes available is not the most significant problem.

Your root filesystem is actually full.

=> Considering its size of 48G, I believe that it should be just easy to find a couple of things to just rm.

 *nitish.anand wrote:*   

> 
> 
> ```
> # du -hx --max-depth=1 /
> 
> ...

 

Start your invertigations from there. 40G in /var is far far far too much.

Unless you get tons of mails / printlogs etc...

----------

## midway

thx aCOSwt. here seems to be the culprits:

```

/var/log:

-rw-rw-r--  1 root    utmp    2723712 Aug 15 12:23 wtmp

-rw-r--r--  1 root    root     292876 Aug 15 12:23 lastlog

-rw-rw----  1 portage portage 1326645 Aug 15 17:00 emerge.log

-rw-rw----  1 portage portage 1226645 Aug 15 17:00 emerge-fetch.log

-rw-r--r--  1 root    root     151054 Feb 10  2012 retropds.log

-rw-------  1 root    root     672128 Aug 15 12:23 tallylog

```

I have rm the lastlog, wtmp and emerge-fetch.log. AFAIK emerge.log is  a good thing to keep? Also the new output looks the same:

```

log # du -hx --max-depth=1 /

4.0K    /boot

8.8M    /bin

0       /dev

15M     /etc

195M    /lib

0       /mnt

205M    /opt

0       /tmp

0       /sys

40G     /var

6.5G    /usr

0       /proc

64M     /home

9.1M    /sbin

788M    /root

12K     /.gnupg

0       /run

48G     /

```

I am puzzled now. Also can the contents of /var/tmp/portage go for good?

```

tmp # ls -al

total 8

drwxrwxrwt  7 root    root      96 Aug 15 15:10 .

drwxr-xr-x 15 root    root    4096 Aug 15 14:57 ..

drwx--x--x  3 root    root      16 Nov  8  2009 Month_9-2

drwxrwxr-x  2 portage portage    6 Mar 27 12:59 binpkgs

drwxr-xr-x  2 root    root     146 Feb  7  2011 genkernel

-rw-------  1 root    root    1977 Aug  8 16:53 nagios-044_0

drwxrwxrwt  2 root    root      38 Jan  4  2011 pear

drwxrwxr-x  6 portage portage   71 Aug 15 17:00 portage

tmp # cd portage

portage # ls -al

total 0

drwxrwxr-x 6 portage portage 71 Aug 15 17:00 .

drwxrwxrwt 7 root    root    96 Aug 15 15:10 ..

drwxrwxr-x 2 portage portage  6 Aug 15 16:53 ._unmerge_

drwxrwxr-x 3 portage portage 24 Aug 15 17:00 dev-lang

drwxrwxr-x 3 portage portage 33 Aug 14 17:16 dev-vcs

drwxrwxr-x 3 portage portage 29 Aug 15 16:42 net-analyzer

```

----------

## aCOSwt

 *nitish.anand wrote:*   

> 
> 
> I have rm the lastlog, wtmp and emerge-fetch.log. AFAIK emerge.log is  a good thing to keep? Also the new output looks the same

 

Of course the output looks the same. You are having 40 gigs in /var and just removed a couple of megs.

Culprits are elsewhere under /var.

 *nitish.anand wrote:*   

> 
> 
> I am puzzled now. Also can the contents of /var/tmp/portage go for good?
> 
> 

 

If you are not currently emerging anything and do not want to keep the tracks of failed builds then yes, you can safely remove the content of /var/tmp/portage.

----------

## midway

Okay, I think I have finally drilled down:

```

# du -a /var | sort -n -r | head -n 10

41627668        /var

40165284        /var/nagios

39616724        /var/nagios/archives

980656  /var/lib

963064  /var/lib/mysql

530932  /var/nagios/spool/checkresults

530932  /var/nagios/spool

242624  /var/log

231104  /var/log/apache2

230512  /var/log/apache2/access_log

nagios # ls -al

total 16288

drwxr-xr-x  6 nagios nagios    4096 Aug 16 14:07 .

drwxr-xr-x 15 root   root      4096 Aug 15 14:57 ..

-rw-r--r--  1 root   root         0 Oct 25  2010 .keep_net-analyzer_nagios-core-0

drwxr-xr-x  2 nagios nagios   28672 Aug 16 00:00 archives

drwxr-xr-x  4 nagios root        69 Aug 14 17:03 home

-rw-r--r--  1 nagios nagios       6 Aug 14 12:20 nagios.lock

-rw-r--r--  1 nagios nagios 5595680 Aug 16 14:07 nagios.log

-rw-------  1 nagios nagios  294912 Nov  9  2009 nagios.tmpQnsGBX

-rw-------  1 nagios nagios       0 Feb 15  2012 nagios.tmpwxgOkC

-rw-r--r--  1 nagios nagios 2444236 Aug 14 12:20 objects.cache

-rw-------  1 nagios nagios 4178715 Aug 16 13:14 retention.dat

drwsr-sr-x  2 nagios apache      62 Aug 14 12:20 rw

drwxr-xr-x  3 nagios nagios      25 Mar 30  2009 spool

-rw-rw-r--  1 nagios nagios 4079069 Aug 16 14:07 status.dat

-rw-r--r--  1 nagios nagios       0 Aug 14 12:20 status.sav

```

Is it safe to delete the whole file from this var directory? AFAICS, the nagios is installed in / 

```

nagios # pwd

/root/nagios

```

----------

## mvaterlaus

i don't know, if you know eclean-dist, but this deletes old installation archives and cleans your portage directory. it is described in [1]. another thing to look at would be your kernel sources under /usr/src. there you can delete every linux-* directory, which you are not using. pay attention when you do this, because you need the sources of the actually running kernel, if you want to rebuild it, and the kernel sources, which you are using for your backup kernel. if you are not sure, which ones you can delete, check the following command, which gives you the actual kernel sources in use:

```

eselect kernel list

```

eselect is part of the package gentoolkit

[1]http://en.gentoo-wiki.com/wiki/Freeing_Up_Disk_Space

----------

## midway

Okay this eclean-dist has freed only 1.6G. I need to delete loads from the /var dir.

```

nagios # pwd

/var/nagios

# du -a /var | sort -n -r | head -n 10 

41627668        /var 

40165284        /var/nagios 

39616724        /var/nagios/archives 

980656  /var/lib 

963064  /var/lib/mysql 

530932  /var/nagios/spool/checkresults 

530932  /var/nagios/spool 

242624  /var/log 

231104  /var/log/apache2 

230512  /var/log/apache2/access_log 

nagios # ls -al 

total 16288 

drwxr-xr-x  6 nagios nagios    4096 Aug 16 14:07 . 

drwxr-xr-x 15 root   root      4096 Aug 15 14:57 .. 

-rw-r--r--  1 root   root         0 Oct 25  2010 .keep_net-analyzer_nagios-core-0 

drwxr-xr-x  2 nagios nagios   28672 Aug 16 00:00 archives 

drwxr-xr-x  4 nagios root        69 Aug 14 17:03 home 

-rw-r--r--  1 nagios nagios       6 Aug 14 12:20 nagios.lock 

-rw-r--r--  1 nagios nagios 5595680 Aug 16 14:07 nagios.log 

-rw-------  1 nagios nagios  294912 Nov  9  2009 nagios.tmpQnsGBX 

-rw-------  1 nagios nagios       0 Feb 15  2012 nagios.tmpwxgOkC 

-rw-r--r--  1 nagios nagios 2444236 Aug 14 12:20 objects.cache 

-rw-------  1 nagios nagios 4178715 Aug 16 13:14 retention.dat 

drwsr-sr-x  2 nagios apache      62 Aug 14 12:20 rw 

drwxr-xr-x  3 nagios nagios      25 Mar 30  2009 spool 

-rw-rw-r--  1 nagios nagios 4079069 Aug 16 14:07 status.dat 

-rw-r--r--  1 nagios nagios       0 Aug 14 12:20 status.sav 

```

Is it safe to delete the whole nagios file (above) from this var directory? 

AFAICS, the nagios is installed in / 

```

nagios # pwd 

/root/nagios 

```

Also, here is my kernel output:

```

src # ls -al

total 8

drwxr-xr-x  3 root root   61 Feb  7  2011 .

drwxr-xr-x 15 root root 4096 Sep 16  2009 ..

-rw-r--r--  1 root root    0 Apr 14  2007 .keep

lrwxrwxrwx  1 root root   24 Feb  7  2011 linux -> linux-2.6.36-hardened-r9

drwxr-xr-x 25 root root 4096 Feb  7  2011 linux-2.6.36-hardened-r9

src # cd linux

linux # ls -al

total 51372

drwxr-xr-x 25 root root    4096 Feb  7  2011 .

drwxr-xr-x  3 root root      61 Feb  7  2011 ..

-rw-r--r--  1 root root     927 Feb  7  2011 ..tmp_kallsyms1.o.cmd

-rw-r--r--  1 root root     927 Feb  7  2011 ..tmp_kallsyms2.o.cmd

-rw-r--r--  1 root root     664 Feb  7  2011 ..tmp_vmlinux1.cmd

-rw-r--r--  1 root root     680 Feb  7  2011 ..tmp_vmlinux2.cmd

-rw-r--r--  1 root root   96525 Feb  7  2011 .config

-rw-------  1 root root   93932 Feb  7  2011 .config.old

-rw-r--r--  1 root root     936 Oct 20  2010 .gitignore

-rw-r--r--  1 root root    4021 Oct 20  2010 .mailmap

-rw-r--r--  1 root root     407 Feb  7  2011 .missing-syscalls.d

-rw-r--r--  1 root root 1489148 Feb  7  2011 .tmp_System.map

-rw-r--r--  1 root root 4441478 Feb  7  2011 .tmp_kallsyms1.S

-rw-r--r--  1 root root 1074792 Feb  7  2011 .tmp_kallsyms1.o

-rw-r--r--  1 root root 4441478 Feb  7  2011 .tmp_kallsyms2.S

-rw-r--r--  1 root root 1074792 Feb  7  2011 .tmp_kallsyms2.o

drwxr-xr-x  2 root root   40960 Feb  7  2011 .tmp_versions

-rwxr-xr-x  1 root root 8726818 Feb  7  2011 .tmp_vmlinux1

-rwxr-xr-x  1 root root 9801754 Feb  7  2011 .tmp_vmlinux2

-rw-r--r--  1 root root       2 Feb  7  2011 .version

-rw-r--r--  1 root root     668 Feb  7  2011 .vmlinux.cmd

-rw-r--r--  1 root root     603 Feb  7  2011 .vmlinux.o.cmd

-rw-r--r--  1 root root   18693 Oct 20  2010 COPYING

-rw-r--r--  1 root root   94019 Oct 20  2010 CREDITS

drwxr-xr-x 86 root root    8192 Feb  7  2011 Documentation

-rw-r--r--  1 root root    2440 Oct 20  2010 Kbuild

-rw-r--r--  1 root root  179296 Oct 20  2010 MAINTAINERS

-rw-r--r--  1 root root   51851 Feb  7  2011 Makefile

-rw-r--r--  1 root root  521347 Feb  7  2011 Module.symvers

-rw-r--r--  1 root root   17459 Oct 20  2010 README

-rw-r--r--  1 root root    3371 Oct 20  2010 REPORTING-BUGS

-rw-r--r--  1 root root 1489148 Feb  7  2011 System.map

drwxr-xr-x 27 root root    4096 Feb  7  2011 arch

drwxr-xr-x  2 root root    4096 Feb  7  2011 block

drwxr-xr-x  3 root root   12288 Feb  7  2011 crypto

drwxr-xr-x 89 root root    4096 Feb  7  2011 drivers

drwxr-xr-x 36 root root    4096 Feb  7  2011 firmware

drwxr-xr-x 72 root root    8192 Feb  7  2011 fs

drwxr-xr-x  2 root root    4096 Feb  7  2011 grsecurity

drwxr-xr-x 22 root root    4096 Feb  7  2011 include

drwxr-xr-x  2 root root    4096 Feb  7  2011 init

drwxr-xr-x  2 root root    4096 Feb  7  2011 ipc

drwxr-xr-x  8 root root   12288 Feb  7  2011 kernel

drwxr-xr-x  7 root root   12288 Feb  7  2011 lib

drwxr-xr-x  2 root root    8192 Feb  7  2011 mm

-rw-r--r--  1 root root    6088 Feb  7  2011 modules.builtin

-rw-r--r--  1 root root   49440 Feb  7  2011 modules.order

drwxr-xr-x 51 root root    4096 Feb  7  2011 net

drwxr-xr-x  8 root root     132 Feb  7  2011 samples

drwxr-xr-x 13 root root    4096 Feb  7  2011 scripts

drwxr-xr-x  8 root root    4096 Feb  7  2011 security

drwxr-xr-x 21 root root    4096 Feb  7  2011 sound

drwxr-xr-x  5 root root      42 Feb  7  2011 tools

drwxr-xr-x  2 root root    4096 Feb  7  2011 usr

drwxr-xr-x  3 root root      16 Feb  7  2011 virt

-rwxr-xr-x  1 root root 9801754 Feb  7  2011 vmlinux

-rw-r--r--  1 root root 8830971 Feb  7  2011 vmlinux.o

```

----------

## midway

okay, I have removed some archived logs of nagios from /var which has freed up almost 21gb.  :Smile: 

thx aCOSwt and mvaterlaus.

Also, any suggestions on the kernels (above)?

----------

## Ant P.

You can run `make distclean` in the kernel dir which deletes all the compiler output, as long as you did `make install` to copy critical files to /boot it won't lose anything important.

----------

## NeddySeagoon

nitish.anand,

Broken emerges build up in /var/tmp/portage, I've had 30G in there once.  When emerge is not running, remove that entire dir.

----------

## midway

thx. Ant P. and NeddySeagoon.

Have freed almost another 10gig. Space shows normal now.

```

~ # du -hx --max-depth=1 /

4.0K    /boot

8.4M    /bin

0       /dev

15M     /etc

195M    /lib

0       /mnt

205M    /opt

0       /tmp

0       /sys

2.0G    /var

3.9G    /usr

0       /proc

64M     /home

9.0M    /sbin

762M    /root

12K     /.gnupg

0       /run

7.1G    /

```

----------

## Circuitsoft

I have the following line in my /etc/fstab which helps a lot.

```

none            /var/tmp/portage        tmpfs   defaults        0 0

```

With 6G RAM on this machine, I've never had a problem.

----------

