# lightspeed emerging without any tools..

## inbreed

Hi,

i hope nobody else did this post yet, and that my search-lazyness will soon be gone..  :Wink: 

On my way wrtiting some ramdisk-scripts i noticed, that i was able to emerge xorg-x11 within 20 minutes in 1200 mhz only! without any tools or special-USE-Flags.

later on i found ot, that all emerges are flying fast!

how?

all you need is tmpfs support to mount a tmpfs into your /var/tmp/portage.

why?

the way ram -> cpu ->ram -> hdd  seems to be faster than 

hdd -> ram -> cpu -> ram -> hdd -> hdd..

so what, damnt it!

if you got a ot of ram (e.G. 1 GB), try: 

```
mount -t tmpfs tmpfs /var/tmp/portage
```

 to create a RAM/2 ramdisk ( the ram will only be used if necessary.

if you want to emerge xorg, you'D need about 850mb of free ram, so mount like this:

```
mount -t tmpfs tmpfs -o size=850000000 /var/tmp/portage
```

after emerging you can do a manually autoclean  :Wink:  :

```
 umount /var/tmp/portage 
```

please tell me wether it works fine for you or not!

----------

## Vanquirius

I don't have 1Gb of RAM to test large compiles, but I tested emerging smaller programs with 100Mb of RAM for /var/tmp/portage and it worked OK. I haven't timed it, but I don't think it had such a performance increase.

----------

## Forse

I don't think it's such a good idea...tmpfs is shared memory and there for will be also placed in swap...so you're back on hdd -_-

----------

## mudrii

If I have 2G RAM I think it may work   :Rolling Eyes: 

----------

## cheester

HI, that's my first post here  :Wink: 

I also tried that tip and i must say it wasn't faster. I emerged few apps and emerge time was almost the same so there is no or maybe very little performance gain.

 *Forse wrote:*   

> tmpfs is shared memory and there for will be also placed in swap...

 

You can always turn off swap - that's what I did... but I have only 512MB RAM @ 400Mhz  :Sad: 

----------

## inbreed

hm... okies,

thx for testing. on my machine it was incredible faster - i dont use swap anyway, so i was able to do it all in ram..

maybe it also depends in your make-config? hmhm..

----------

## nyda

Or your compiler cache kicked in.

```

root, root/ # genlop -t xorg-x11

 * x11-base/xorg-x11

     Tue Sep  7 16:19:39 2004 --> x11-base/xorg-x11-6.7.99.904

       merge time: 49 minutes and 39 seconds.

     Tue Sep 14 21:29:06 2004 --> x11-base/xorg-x11-6.8.0

       merge time: 38 minutes and 51 seconds.

     Fri Sep 17 15:27:04 2004 --> x11-base/xorg-x11-6.8.0-r1

       merge time: 24 minutes and 20 seconds.

```

----------

## s4kk3

I tested this and not sure if it gives any performance. Havn't done any benchmarks yet.

What is the difference between ramfs and tmpfs? Does ccache affect this? I mean portage uses ccache that is located at hdd, so it would lower performance. This idea sounds great and I will do some bechmarks when I have time.

----------

## Vanquirius

Had forgotten about genlop, I was thinking on using `time` to benchmark it... This makes it easier, having a look at two packages I emerged yesterday using this technique.

```

 * media-video/transcode

     Sun Aug 15 19:19:55 2004 --> media-video/transcode-0.6.11

       merge time: 8 minutes.

     Fri Sep 17 11:22:56 2004 --> media-video/transcode-0.6.11

       merge time: 10 minutes and 9 seconds.

     Sun Nov  7 21:01:21 2004 --> media-video/transcode-0.6.11

       merge time: 10 minutes and 31 seconds.

 * media-video/xine-ui

     Sun Aug 15 20:48:28 2004 --> media-video/xine-ui-0.9.23-r2

       merge time: 2 minutes and 53 seconds.

     Sat Sep 25 20:09:08 2004 --> media-video/xine-ui-0.9.23-r2

       merge time: 3 minutes and 2 seconds.

     Sun Nov  7 21:04:15 2004 --> media-video/xine-ui-0.9.23-r2

       merge time: 2 minutes and 54 seconds.
```

----------

## s4kk3

unmerged ccache and it seems faster. Maybe some bechmark results today

----------

## inbreed

good idea s4kk3  -  ccache must not be used.

well, sorry for all thous killing thier time on reemerging.

```
 root # genlop -t xorg-x11

 * x11-base/xorg-x11

     Sat Nov  6 16:31:53 2004 --> x11-base/xorg-x11-6.8.0-r1

       merge time: 19 minutes and 47 seconds.

 

 merged totally 1 ebuild in 19 minutes and 47 seconds. 

```

on my home machine..

```

cat /proc/cpuinfo | grep MHz

cpu MHz         : 1527.110

```

----------

## s4kk3

Made some bechmarks with mplayer

First without tmpfs or ramfs

```
Mon Nov  8 19:12:10 2004 --> media-video/mplayer-1.0_pre4-r7

       merge time: 8 minutes and 59 seconds.

```

Then with ramfs

```
     Mon Nov  8 19:26:51 2004 --> media-video/mplayer-1.0_pre4-r7

       merge time: 8 minutes and 49 seconds.

```

and finally tmpfs

```
     Mon Nov  8 19:41:39 2004 --> media-video/mplayer-1.0_pre4-r7

       merge time: 8 minutes and 49 seconds.

```

It didn't give so much performance, but i think ramfs or tmpfs is faster with bigger packages. Have to try with xorg this night.

EDIT: I just got an crazy idea. What would happen if ccache is included in ram too? But only thinking if it's possible

----------

## Vanquirius

 *s4kk3 wrote:*   

> EDIT: I just got an crazy idea. What would happen if ccache is included in ram too? But only thinking if it's possible

 

That would be nice, but your ccache size would be heavily limited by the amount of RAM you can give it. In short: your ccache may not be big enough to be useful depending on the range of packages you usually compile. Also, that would mean copying your ccache to and from your hard-disk everytime you reboot for any reason.

----------

## Viha

Yeah, only if the system is rarely rebooted or shutdown. But maybe it would be possible to use something like rsync to keep the copy on hard drive updated, I don't know.

----------

## s4kk3

 *Quote:*   

> That would be nice, but your ccache size would be heavily limited by the amount of RAM you can give it. In short: your ccache may not be big enough to be useful depending on the range of packages you usually compile. Also, that would mean copying your ccache to and from your hard-disk everytime you reboot for any reason.

 

I know that but you can limit ccache to use lesser memory as I have seen it wont use so much space. I had limited ccache to use 100mb space so why that couldn't be included in ram if you have enought ram? And it rarely used that 100mb

----------

## Vanquirius

My experience has been different:

```
# ccache -s

...

cache size                           1.8 Gbytes

max cache size                       2.0 Gbytes
```

----------

## s4kk3

With kdemultimedia I saved about eight minutes with tmpfs, haven't tested with ramfs (with ramfs it looks like it doesn't use ram).

----------

## s4kk3

Damn. How you can compile xorg so fast? With tmpfs it took 52mins to compile it and without 1hour 11mins. Prosessor athlon xp 2100+, 1gb 400 MHz ddr ram. My hdd speed shouldn't affect this cause compilling will be done in ram? Does LDFLAGS increase compilling time?

Here is my make.conf:

```
# These settings were set by the catalyst build script that automatically built this stage

# Please consult /etc/make.conf.example for a more detailed example

CFLAGS="-march=athlon-xp -mtune=athlon-xp -O2 -m3dnow -msse -mmmx -mfpmath=sse387 -ffast-math -fprefetch-loop-arrays -funroll-loops -fforce-addr -pipe ftracer -fomit-frame-pointer"

CHOST="i686-pc-linux-gnu"

CXXFLAGS="-march=athlon-xp -mtune=athlon-xp -O2 -m3dnow -msse -mmmx -mfpmath=sse,387 -ffast-math -fprefetch-loop-arrays -funroll-loops -fforce-addr -pipe -ftracer -fomit-frame-pointer -fvisibility-inlines-hidden -fvisibility=hidden"

MAKEOPTS="-j2"

USE="3dnow 3dnowex 3dnowext amd fPIC fpic mmx mmxex mmxext nptl nvidia ooo-kde openal opengl posix sse video_cards_nvidia videos -arts -man -doc -video_cards_ati -video_cards_radeon -ati -radeon -intel -amd64 -pentium -pentium2 -pentium3 -pentium4 -ia64 -ppc64 -ppc"

PORTDIR_OVERLAY=/usr/local/portage

FETCHCOMMAND="/usr/bin/axel -a -S6 \${URI} -o \${DISTDIR}"

RESUMECOMMAND="/usr/bin/axel -a -S6 \${URI} -o \${DISTDIR}"

LDFLAGS="-Wl,-O1"

LINGUAS="fi"

```

----------

## nyda

 *s4kk3 wrote:*   

> 
> 
> ```
> CFLAGS="-march=athlon-xp -mtune=athlon-xp -O2 -m3dnow -msse -mmmx -mfpmath=sse387 -ffast-math -fprefetch-loop-arrays -funroll-loops -fforce-addr -pipe ftracer -fomit-frame-pointer"
> ```
> ...

 

Ever considered that it might take more time to produce optimized code than normal code? Your depot of CFLAGS might well be a reason why it's slower. You're also missing a - in front of ftracer =)

 *s4kk3 wrote:*   

> I know that but you can limit ccache to use lesser memory as I have seen it wont use so much space. I had limited ccache to use 100mb space so why that couldn't be included in ram if you have enought ram? And it rarely used that 100mb

 

Building openoffice will go through those 100MB at least 8 times. So at the point where your OOo is at 12%, it has already wiped the cache for everything else - rendering your CC somewhat... useless  :Smile: 

OOo is extreme, but so is a 100MB CC.

edit: actually forgot to mention the obligatory "-ffast-math will break your system" comment   :Very Happy: 

----------

## inbreed

uff,

my cflags are something like -O3 -pipe -fomit-frame-pointers -mtune=athlon-xp -march=athlon-xp

thats all.

LDFLAGS="-Wl,-O1 -Wl,--enable-new-dtags -Wl,--sort-common -s" <- maybe they are also dealing with it..

----------

## disturbed

whow , my speed increases a lot... Thx 4 the tip...

grtz

----------

## s4kk3

I have tried with lesser flags. But it didn't make any difference. Maybe it's hardware related. When I installed my first linux (redhat) it decreased my MHz from 1600 to 1300. Maybe it decreased something else at the same time?

----------

## akoning

Hi,

I have been doing the same thing for a while now. 

I found some post in the forum about running /var/log on a tmpfs for power saving on my laptop and thought why not do compilation on a tmpfs dir.

I figured even if the build is bigger than your ram size the overhead due to swapping would not be significant enough to cause the emerge to be slower than when you work on a 'normal' partition.

On my desktop machine (athlon-xp 2800 + 1G) it run's smooth as a baby.

(I did not do any benchmarks but it is definably faster). 

But because of the size of the ram no swap-space is ever used.

On my laptop with 256M it causes freeses.  I think that when there is too much swapping on this machine the kernel locks. 

This machine is tuned to reduce swapping and I have the laptop-mode switch in the proc switched on.

----------

## Vanquirius

 *s4kk3 wrote:*   

> When I installed my first linux (redhat) it decreased my MHz from 1600 to 1300.

 

Any chance you are getting them back?   :Laughing: 

----------

## s4kk3

Switched that back to 1600MHz. But everything else was normal in bios. Maybe my kernel sucks. Is there any guide how to build fast kernel? I mean what should I enable and disable. Haven't found any yet.

Ccache gives little more boost when it is enabled in tmpfs but you can't keep that so big. Well it is better than nothing. Limited ccache to 70mb in ram. Maybe I should make a script that makes tar from ccache dir when shutting down and untars it when booting. Then ccache doens't need to collect all the stuff when it is already there.

----------

## batistuta

I know that this was mentioned 1000 times before, but after I've tried it today I was so impressed, that I thought about mentioning it again for all people new with Gentoo. I even think that the installation manual should say something about it.   :Very Happy: 

tmpfs provides a way dinamically allocate RAM for your /tmp or /var/tmp partition. You can enable it and disable it as you wish on the fly, and it dramatically reduces your compilation time (at least for my system) by minimizing the I/O access.  :Cool: 

Check it out

----------

## irondog

Yep. Tmpfs rules. It doesn't claim free memory if it isn't in use. It's also swappable, which means it won't waste your free memory if tmpfs is populated with garbage.

You might want to add this to /etc/fstab

```
none /var/tmp/portage tmpfs size=800M 0 0
```

There also is no need to clean /var/tmp/portage anymore. Every reboot your /var/tmp/portage is clean.

----------

## pussi

Wow! Thanks for the tip.

Finally I was able to get rid of the ccache ;)

----------

## PaveQ

 *pussi wrote:*   

> Wow! Thanks for the tip.
> 
> Finally I was able to get rid of the ccache 

 

Why would you get rid of ccache? It still speeds up compiling no matter if you got tmpfs. They are completely separate things.

----------

## pussi

 *PaveQ wrote:*   

> Why would you get rid of ccache? It still speeds up compiling no matter if you got tmpfs. They are completely separate things.

 I know, but I actually tried compiling with both tempfs and ccache, and it was faster with tempfs only.

Anyway, I've been thinking of stopping using ccache because it takes too much disk space and tempfs makes compilations faster without that side effect.

----------

## ThamanX

I have 512 MB RAM installed in my System. How many RAM shoulde i use for it ??

Could it be that this Drive gets full ?? How can i prevent me from this ??

----------

## lnxz

You should make a script that mounts a tmpfs on /var/tmp/portage before the emerge and unmounts it afterwards, because some packages will be larger than your ram space, which will cause the compilation to be terminated.

The script should probably empty /var/tmp/portage after each subsequent emerge, too (if you compile several packages at once).

----------

## tomk

Merged from here.

----------

## Sheepdogj15

in case anyone is still curious, tmpfs will swap out if you run out of RAM, whereas ramfs won't. otherwise the theory is the same: dynamically changes in size with how much data is in there, you specify a max size, etc.

----------

## Naib

 *nyda wrote:*   

> Or your compiler cache kicked in.
> 
> ```
> 
> root, root/ # genlop -t xorg-x11
> ...

 

How long

```

Fluid jrb # genlop -t xorg-x11

 * x11-base/xorg-x11

     Mon Apr 11 00:39:30 2005 >>> x11-base/xorg-x11-6.8.2-r1

       merge time: 1 hour, 8 minutes and 19 seconds.

     Sun Jun  5 05:15:37 2005 >>> x11-base/xorg-x11-6.8.2-r1

       merge time: 1 hour, 32 minutes and 40 seconds.

     Sun Jun  5 10:40:30 2005 >>> x11-base/xorg-x11-6.8.2-r1

       merge time: 1 hour, 1 minute and 58 seconds.

     Wed Jun 15 09:35:40 2005 >>> x11-base/xorg-x11-6.8.2-r2

       merge time: 1 hour, 15 minutes and 25 seconds.

     Tue Sep 13 00:38:00 2005 >>> x11-base/xorg-x11-6.8.2-r3

       merge time: 1 hour, 4 minutes and 38 seconds.

     Tue Sep 13 12:19:21 2005 >>> x11-base/xorg-x11-6.8.2-r3

       merge time: 1 hour, 10 minutes and 43 seconds.

     Sun Sep 18 04:34:36 2005 >>> x11-base/xorg-x11-6.8.2-r4

       merge time: 1 hour, 23 minutes and 28 seconds.

     Fri Sep 23 09:09:33 2005 >>> x11-base/xorg-x11-6.8.2-r5

       merge time: 1 hour, 3 minutes and 12 seconds.

     Sun Oct  9 02:21:54 2005 >>> x11-base/xorg-x11-6.8.2-r6

       merge time: 55 minutes and 27 seconds.

     Sat Mar 11 20:49:37 2006 >>> x11-base/xorg-x11-7.0-r1

       [b]merge time: 4 seconds.[/b]

```

only took 4seconds for me   :Wink: 

----------

