# Massive speedup using ccache in disk backed RAMFS

## devsk

It was driven mostly by needs (think speed) and what means I already had (think 12GB RAM). Here is what I needed:

1. ccache gains are good on a fast SSD but it trashes the FS and the disk (SSDs are sensitive to write amplification, resulting in lower life and performance).

2. It should not slow down the first emerge, where there is no hit in ccache. If you use normal HDD, you will see that without ccache, the first compile goes faster. If ccache is not going to give me a lot of hits, this means that I am doing additional work of storing that data and losing performance overall in long term.

Only way to get these two requirements to satisfy was to let go of 1.5GB (size of my ccache) of RAM for good. Put the ccache on the tmpfs and find ways to sync it to disk only on shutdown and load into RAM on startup. Here is what I did:

1. /etc/make.conf

```
# ccache in RAMFS

CCACHE_DIR=/var/tmp/ccache
```

2. /etc/fstab

```

# for faster builds.

none                                    /var/tmp                tmpfs   size=6856M,nr_inodes=1M         0 0
```

3. Restore script

```
$ cat ccache-restore

#!/bin/sh

if [[ ! -d /var/tmp/ccache ]]

then

        mkdir /var/tmp/ccache

fi

rsync -av -H -x /usr/portage/local/ccache/x64/ /var/tmp/ccache/ > /dev/null 2>/var/log/ccache-restore.log

touch /var/tmp/ccache/.restored
```

4. Store script

```
$ cat ccache-store

#!/bin/sh

if [[ -f /var/tmp/ccache/.restored ]]

then

        rsync -av -H -x /var/tmp/ccache/ /usr/portage/local/ccache/x64/ > /dev/null 2>/var/log/ccache-store.log

fi
```

5. Put restore in /etc/conf.d/local.start

```
# restore ccache in a lazy fashion

ccache-restore &

disown %1
```

6. Put store in /etc/conf.d/local.stop

```
# store ccache on disk

ccache-store
```

Results:

```

     Sat Jul 31 00:34:39 2010 >>> net-libs/xulrunner-1.9.2.8

       merge time: 4 minutes and 24 seconds.                                                                                                       

     Sat Jul 31 00:37:24 2010 >>> net-libs/xulrunner-1.9.2.8

       merge time: 1 minute and 17 seconds.
```

```
     Sat Jul 31 00:08:31 2010 >>> mail-client/thunderbird-3.1.1

       merge time: 4 minutes and 21 seconds.                                                                                                       

     Sat Jul 31 00:10:19 2010 >>> mail-client/thunderbird-3.1.1

       merge time: 1 minute and 13 seconds.
```

```
     Sat Jul 31 08:47:32 2010 >>> media-video/ffmpeg-0.6

       merge time: 41 seconds.                                                                                                                                              

     Sat Jul 31 08:47:55 2010 >>> media-video/ffmpeg-0.6

       merge time: 15 seconds.
```

Achieved:

1. No small random writes to SSD until shutdown, which is many days or even weeks for me because I suspend-to-ram.

2. No loss during first run.

3. Massive gain on hits.

Caveats:

Do not try this if you don't have hardware to support it.

You need a modern enough multi-core CPU to make sure that you can calculate hashes orders of magnitude faster than you can compile.

You need at least 6GB of RAM (and that may be small and you may want to reduce the ccache size).

And you need a fast SSD for "/" if you don't want your shutdown to take forever and your startup to slow down because of IO.

I will post the same numbers for openoffice build in a short while.

----------

## devsk

```
     Thu Jul 29 18:36:12 2010 >>> app-office/openoffice-3.2.1

       merge time: 27 minutes and 19 seconds.                                                                                    

     Sat Jul 31 09:50:34 2010 >>> app-office/openoffice-3.2.1

       merge time: 6 minutes and 58 seconds.
```

What remains to be seen is when openoffice-3.2.2 comes out, whether my system will compile it in 27 mins or 7 mins. Likely somewhere in the middle.

----------

## devsk

This came in handy while moving from the overlay based kde to portage based.

```
     Tue Aug 31 18:15:36 2010 >>> kde-base/kdelibs-4.5.1

       merge time: 5 minutes and 25 seconds.

     Fri Sep 10 20:13:16 2010 >>> kde-base/kdelibs-4.5.1

       merge time: 1 minute and 41 seconds.
```

The whole kde 4.5.1 (at least whatever I am interested in) emerged in like 12 minutes.

----------

## neuron

That's... crazy. With speedups that high I wouldn't be surprised if compcache + less ram would also be well worth it.

----------

## devsk

Ok, this just got more interesting with latest feature in compcache or zram as its called now in latest kernel.

Basically, instead of creating /var/tmp as tmpfs, create a compressed block device using zram, create ext4 on it and use that to mount on top of /var/tmp. The advantage is that for 1.3Gig of ccache, it compresses it down to 500MB i.e. only 500MB of real RAM is used. You get a little bit of overhead on CPU but I have a pretty powerful CPU in i7 920. So, that's fine!

Here is what I did:

```
#!/bin/sh

# compcache is not there, get out!

ramz=`find /lib/modules/$(uname -r)/kernel/drivers/ -name zram.ko 2>/dev/null`

[ -z "$ramz" ] && exit 0

# turn it on if nothing passed in as args

if [ -z "$1" ]

then

        # load the zram devices

        modprobe zram num_devices=2

        sleep 1

        # 7GB of /var/tmp

        echo $((7*1024*1024*1024)) > /sys/block/zram0/disksize

        # 1GB of compressed swap

        echo $((1*1024*1024*1024)) > /sys/block/zram1/disksize

        mkswap /dev/zram1

        swapon -p 100 /dev/zram1

        mkfs.ext4 -m0 -N 1000000 -L "zram-vol" /dev/zram0

        mount -o noatime /dev/zram0 /var/tmp

else

        lsmod | grep -q "^zram"

        if [ $? -ne 0 ]

        then

                exit 0

        fi

        # unload the swap devices

        if [ -b /dev/zram1 ]

        then

                swapoff /dev/zram1

                sleep 0.2

                echo 1 > /sys/block/zram1/reset

        fi

        if [ -b /dev/zram0 ]

        then

                umount /dev/zram0

                sleep 0.2

                echo 1 > /sys/block/zram0/reset

        fi

        modprobe -r zram

fi
```

I put this in /usr/bin/compcache, and I invoke compcache from /etc/conf.d/local.start and 'compcache off' from /etc/conf.d/local.stop. Make sure to save the ccache before calling 'compcache off' in local.stop and restore it after calling 'compcache' in local.start.

This is the stats I got after restoring ccache to zram-vol mounted at /var/tmp:

```
# ~/bin/zram_stats 

/sys/block/zram0

disksize:         7516192768 

num_reads:               188 

num_writes:           423258 

invalid_io:                0 

notify_free:               0 

discard:                   0 

zero_pages:            88848 

orig_data_size:   1324761088 

compr_data_size:   471440303 

mem_used_total:    481484800 

avg_compr_ratio:          35 %

mem_overhead:              2 %
```

Check it out:

```
hg clone https://compcache.googlecode.com/hg/ compcache

cd compcache

vi README
```

----------

## vh4x0r

Just stopped by to say that local.start and local.stop are deprecated since openrc-0.6.4. So steps 5 and 6 should be modified to add the corresponding scripts in the local_start and local_stop functions of /etc/conf.d/local.

```
local_start() {

   # This is a good place to load any misc programs

   # on startup (use &>/dev/null to hide output)

   # Restore ccache in a lazy fashion

   /usr/local/bin/ccache-restore &

   disown %1

   # We should always return 0

   return 0

}

local_stop() {

   # This is a good place to unload any misc.

   # programs you started above.

   # Save ccache to disk

   /usr/local/bin/ccache-store

   

   # We should always return 0

   return 0

}

```

----------

