# Fast world recompile

## costel78

Sometimes, as every Gentoo user, I have to do a full world recompile. And I have an Intel Core i7 860 CPU.

Looking at it how cores just stand, increasing the time I have to wait until recompilation it's done, make me impatient.

So I read manuals of portage and make.conf I run some test. That was almost two months ago.

Prerequisites:

1. All packages are up-to-date (emerge -auvDN world);

2. No link-inconsistency (revedep-rebuild and emerge @module-rebuild);

3. No other various portage error: (emaint --fix all; eix-update ; eix-test-obsolete);

4. portage-2.2.x - I haven't tested the setup with portage 2.1.x . Anyway, 2.1 version does not support some of vital required feature for this setup.

The way I do a full system and world recompile is:

1. Recompile toolchain in standard way;

2. Obtain a list with all system packages;

3. Obtain a list with all world packages;

4. Compile the two list (one at a time) using --nodeps --oneshoot using aggressive FEATURES to speed up parallelization.

Relevant entries in make.conf:

```
COLLISION_IGNORE="/bin /lib /sbin /usr /var"

EMERGE_DEFAULT_OPTS="--jobs=3 --load-average=8.1 --keep-going --with-bdeps=y --complete-graph"

FEATURES="assume-digests candy collision-protect -ebuild-locks distlocks fail-clean fixpackages news nodoc noinfo parallel-fetch parallel-install -ebuild-locks protect-owned sandbox strict -test unmerge-orphans"
```

Increasing number of parallel package builds over 3 (--jobs=3) are pointless, well, unless you have a hexacore AMD CPU I suppose.

Modify load-average value according to your CPU.

Take a look at fail-clean parallel-fetch parallel-install -ebuild-locks and sandbox portage features. Here is the half of the magic   :Very Happy: 

The other half:

recompile.sh

```
emerge --oneshot portage linux-headers gcc-config mpfr binutils libtool dev-libs/mpc dev-libs/ppl dev-libs/cloog-ppl glibc gcc && env-update && source /etc/profile

for f in `emerge -pve system | grep ebuild | gawk '{print $4;}'` ; do echo -n " ="$f ; done > system_list.txt ; 

for f in `emerge -pve world | grep ebuild | gawk '{print $4;}'` ; do echo -n " ="$f ; done > world_list.txt ; 

emerge --nodeps --oneshot `cat system_list.txt` ; 

emerge --nodeps --oneshot `cat world_list.txt`
```

mpc is for newer gcc, ppl and cloog-ppl are for graphite support. Leave them or remove them, depending of your gcc version/use-flags.

MAKEOPTS it's also at your discretion.

Enjoy fastes world recompile.

----------

## audiodef

Whatever happened to using MAKEOPTS and emerge @system @world?   :Shocked: 

----------

## costel78

Yes, I understand the irony, but it's seem like waste of CPU power to compile only one package at a time, with CPU between 5%-35%, especially when you spent a fortune to buy it.

Regarding MAKEOPTS, it control only one package. Many packages force "-j1" or are two small to benefit from parallel compile. Here is where EMERGE_DEFAULT_OPTS="--jobs=3 --load-average=" appear. 

For comparation:

emerge -e system took 285% as 

```
for f in `emerge -pve system | grep ebuild | gawk '{print $4;}'` ; do echo -n " ="$f ; done > system_list.txt ; emerge --nodeps --oneshot `cat system_list.txt` ; 
```

 because many packages in system are compiling only one at a time therefor time required to install them count on total time.

Emerge -e world took ~ 11 hours. Using the above procedure took only 5:36 hours.

pkgcore developers (maybe sometime in the future this will become portage3) takes a similar approach.

I only want to share the info as pkgcore it's still immature. Someones may find it usefully, someones futile. It's ok    :Smile: 

----------

## NeddySeagoon

costel78,

Its very rare that you need to rebuild your entire install. I think the last time it was actually requied was arounf gcc-3.4 when the ABI changed.

--jobs=  makes a pretty good job of parallel building as it respects dependencies too.  That might be important.

----------

## dwbowyer

All that is really needed is 

```

EMERGE_DEFAULT_OPTS="--jobs=3 --load-average=8.1 --keep-going --with-bdeps=y

```

Just the "--jobs --load-average" parallelizes multiple packages, and caps the load average so as not to swamp they given system. I've been using that for 2 years now on my AMD64x2 (since portage 1.9 I think). Everything else suggested seems to be about build system and world packages in the right order, and there are many solutions for that.

----------

## audiodef

My Phenom II x6 system doesn't take more than a few hours - if that - to emerge world, without any special tweaking. Did you emerge every package in Portage?   :Razz:   :Shocked: 

----------

## dwbowyer

Maybe he is like me... 26GB / is 57% full. and that with /home, /usr/portage, and /usr/share all on separate partitions.

I love my games.

----------

## costel78

 *Quote:*   

> Just the "--jobs --load-average" parallelizes multiple packages, and caps the load average so as not to swamp they given system. I've been using that for 2 years now on my AMD64x2 (since portage 1.9 I think). Everything else suggested seems to be about build system and world packages in the right order, and there are many solutions for that.

 

Well, it's seem that my dependencies list prevent a good parallelization - a long list of only one package at a time...  :Sad: 

I didn't recompile every package in the portage,   :Laughing:    but I use some , hmmm... not widely used.

I'm glad that it's only me with slow emerge -e @system @world. Particular situations, particular solutions   :Smile: 

----------

## Hypnos

 *costel78 wrote:*   

> Sometimes, as every Gentoo user, I have to do a full world recompile.

 

FWIW, I have used Gentoo since 2002, and never done a world recompile.  Even with the gcc-3.4 C++ ABI change, I just recompiled the C++ programs -- not very many, since I used GNOME at the time.

----------

