# How-To Bootstrap and 'emerge system' with distcc

## securiteaze

How-To Bootstrap with distcc

I have successfully bootstrapped a gentoo box from stage1-x86-1.4_rc1.

The following outlines the process used.

Disclaimer: This may break your system! There may be more graceful ways of doing this ... but it worked for me  :Very Happy:  

At the point during an install when you would normally run 

```
scripts/bootstrap.sh
```

Don't.

To use distcc while bootstrapping, do the following: 

Emerge distcc to your system

```
emerge distcc
```

Backup bootstrap.sh

```
cp /usr/portage/scripts/bootstrap.sh /usr/portage/scripts/bootstrap.sh.orig
```

Edit bootstrap.sh to make it use distcc.

```
nano -w /usr/portage/scripts/bootstrap.sh
```

Find the following section in bootstrap.sh.

```
cp /etc/make.conf /etc/make.conf.build

export CFLAGS="`$PYTHON -c 'import portage; print portage.settings["CFLAGS"];'`"

export CHOST="`$PYTHON -c 'import portage; print portage.settings["CHOST"];'`"

export CXXFLAGS="`$PYTHON -c 'import portage; print portage.settings["CXXFLAGS"];'`"

export MAKEOPTS="`$PYTHON -c 'import portage; print portage.settings["MAKEOPTS"];'`"

```

Comment out the line :

```

export MAKEOPTS="`$PYTHON -c 'import portage; print portage.settings["MAKEOPTS"];'`"

```

Should now look like this :

```
cp /etc/make.conf /etc/make.conf.build

export CFLAGS="`$PYTHON -c 'import portage; print portage.settings["CFLAGS"];'`"

export CHOST="`$PYTHON -c 'import portage; print portage.settings["CHOST"];'`"

export CXXFLAGS="`$PYTHON -c 'import portage; print portage.settings["CXXFLAGS"];'`"

#export MAKEOPTS="`$PYTHON -c 'import portage; print portage.settings["MAKEOPTS"];'`"

```

Add a new line above the one you just commented out.

```
export MAKEOPTS="-j12 CC='distcc' CXX='distcc'"
```

The -j12 specifies the number of threads to use; you may want to tweak this. (ie -j8 or -j32).

The rest tells make to use distcc for compiling c and c++.

Should now look like this :

```
cp /etc/make.conf /etc/make.conf.build

export CFLAGS="`$PYTHON -c 'import portage; print portage.settings["CFLAGS"];'`"

export CHOST="`$PYTHON -c 'import portage; print portage.settings["CHOST"];'`"

export CXXFLAGS="`$PYTHON -c 'import portage; print portage.settings["CXXFLAGS"];'`"

export MAKEOPTS="-j12 CC='distcc' CXX='distcc'"

#export MAKEOPTS="`$PYTHON -c 'import portage; print portage.settings["MAKEOPTS"];'`"

```

Add one more line :

```
export DISTCC_HOSTS="localhost <distccd-srv> <distccd-srv> <distccd-srv>"
```

Replace <distccd-srv> with the hostname/IP of the boxes running distccd.

This line tells distcc which hosts to send work and in which order.

On a really slow box you may wish to leave out localhost or move it to last.

Should now look similar to this:

```
cp /etc/make.conf /etc/make.conf.build

export CFLAGS="`$PYTHON -c 'import portage; print portage.settings["CFLAGS"];'`"

export CHOST="`$PYTHON -c 'import portage; print portage.settings["CHOST"];'`"

export CXXFLAGS="`$PYTHON -c 'import portage; print portage.settings["CXXFLAGS"];'`"

export DISTCC_HOSTS="localhost 10.1.1.25 10.1.1.50"

export MAKEOPTS="-j12 CC='distcc' CXX='distcc'"

#export MAKEOPTS="`$PYTHON -c 'import portage; print portage.settings["MAKEOPTS"];'`"

```

Now, scroll the end of the script.*

```

#make.conf has been overwritten, so we explicitly export our original settings

export USE="$ORIGUSE bootstrap"

emerge $myGLIBC $myBASELAYOUT $myTEXINFO $myGETTEXT $myBINUTILS $myGCC || cleanup 1

#restore original make.conf

cleanup 0

```

Find the following line and comment it out.*

```
emerge $myGLIBC $myBASELAYOUT $myTEXINFO $myGETTEXT $myBINUTILS $myGCC || cleanup 1
```

It should now look like

```

#make.conf has been overwritten, so we explicitly export our original settings

export USE="$ORIGUSE bootstrap"

#emerge $myGLIBC $myBASELAYOUT $myTEXINFO $myGETTEXT $myBINUTILS $myGCC || cleanup 1

#restore original make.conf

cleanup 0

```

Now add the following lines above the one you commented out.*

```

export MAKEOPTS="-j12 CC='distcc gcc' CXX='distcc g++'"

emerge $myGLIBC || cleanup 1

export MAKEOPTS="-j12 CC='distcc' CXX='distcc'"

emerge  $myBASELAYOUT $myTEXINFO $myGETTEXT $myBINUTILS $myGCC || cleanup 1

```

Should finally look like

```

#make.conf has been overwritten, so we explicitly export our original settings

export USE="$ORIGUSE bootstrap"

export MAKEOPTS="-j12 CC='distcc gcc' CXX='distcc g++'"

emerge $myGLIBC || cleanup 1

export MAKEOPTS="-j12 CC='distcc' CXX='distcc'"

emerge  $myBASELAYOUT $myTEXINFO $myGETTEXT $myBINUTILS $myGCC || cleanup 1

#emerge $myGLIBC $myBASELAYOUT $myTEXINFO $myGETTEXT $myBINUTILS $myGCC || cleanup 1

#restore original make.conf

cleanup 0

```

Save bootstrap.sh then

```

cd /usr/portage && scripts/bootstrap.sh

```

Enjoy!

*Edited to fix a bug in emerging glibc 10/25

*Edited to change subject 10/27

Comments? Suggestions?

----------

## securiteaze

How-To emerge system with distcc

Disclaimer: This may break your system! There may be more graceful ways of doing this ... but it worked for me   :Very Happy:  

This method allows you to 'emerge system' using distcc.

A few packages have issues, but the workaround is included.

If you are continuing from a distcc bootstrap skip to step 2

Install distcc

```
emerge sync ; emerge distcc
```

Tweak /etc/make.conf for distcc

Backup make.conf

```
cp /etc/make.conf /etc/make.conf.orig
```

Edit /etc/make.conf to use distcc

The following could be the entire content of make.conf

```

#DISTCC_HOSTS tells distcc which hosts to send work

DISTCC_HOSTS="localhost <distccd-srv> <distccd-srv>"

#Make sure to replace <distccd-srv> with the hostname/IP of the boxes running distccd

#MAKEOPTS specifies commandline options when portage call the make command

MAKEOPTS="-j16 CC='distcc' CXX='distcc'"

#the -j16 tell make to spawn 16 threads, you may want to tweak this

#If you don't know what the next lines do then see note below 

CHOST="i686-pc-linux-gnu"

CFLAGS="-march=pentium4 -O3 -pipe"

CXXFLAGS="${CFLAGS}"

```

Note: All about C flags

Save make.conf

Now start the build

```
emerge system
```

There will be some packages which fail to build(util-linux, bin86, procps, psmisc,  netkit-base).

For some reason some packages don't like the CC='distcc', CC='distcc gcc', CXX='distcc', or CXX='distcc g++'.

Other packages (procps) don't like parallel makes (-j2).

Edit /etc/make.conf and comment out the MAKEOPTS line.

It should look like this

```

#MAKEOPTS specifies commandline options when portage call the make command 

#MAKEOPTS="-j16 CC='distcc' CXX='distcc'" 

#the -j16 tell make to spawn 16 threads, you may want to tweak this 

```

Now build the finnicky packages

```
emerge netkit-base procps psmisc
```

Edit /etc/make.conf and uncomment the MAKEOPTS line.

```

#MAKEOPTS specifies commandline options when portage call the make command 

MAKEOPTS="-j16 CC='distcc' CXX='distcc'" 

#the -j16 tell make to spawn 16 threads, you may want to tweak this 

```

Continue the build

```
emerge system
```

When 'emerge system' fails again repeat step 4

Build the last of the finnicky packages.

```
emerge util-linux bin86
```

Repeat step 5 and 6.  There shouldn't be any more build failures.

Enjoy!

Comments? Suggestions?

----------

## gentuse

Good tip.  I would recommend setting CC="distcc gcc" and CXX="distcc g++" to overcome the bug described here:  http://lists.samba.org/pipermail/distcc/2002q4/000308.html

----------

## phong

Thanks for the glibc workaround!  I was going to write up this exact HOWTO but couldn't figure out how to get past glibc so I didn't bother (too bad there are other things that break unless you set CC="distcc" because they don't like a compiler that's two words).  It's also nice to know it's an actual bug (non-recognition of the .S extension) and not just a general shortcoming that can't be fixed.

----------

## BradN

maybe you could make it run a script which then ran "distcc gcc" or something?

----------

## BradB

I followed this guide and had no problems installing.  My workstation is an Athlon800 and the client is a Compaq Armada M300 (P2-300Mhz).

My only tips are

 - to emerge packages without distcc and without changing /etc/make.conf you can 

```
MAKEOPTS="" emerge blah
```

 - the distccd daemon will not run as root - took me ages to figure that one out!!

Cheers

Brad

----------

## zojas

ok, this is fun! but I've run into something strange. I'm bootstrapping on a 90 MHz pentium (this will be my iptables firewall and DNS server; until last night it was running redhat 7.2). my distcc_host contains only "raven", the hostname of my other machine, a 700 MHz athlon tbird.

Started the bootstrap around 9pm last night. At around 9:30 this morning was the last time the distccd on raven was connected to. So I'm thinking it's either done or it failed. when I got home just now I see that not only is it still running but it's invoking gcj to compile java code!! what package is that? this is the bootstrap.sh step still. so apparently it's been putzing with java for 8 hours, which of course all has to compile locally.

----------

## zhenlin

gcj  is java, and is probably from gcc compiling it's own class files. USE="-java"

----------

## Jeld

I have followed this guide, to the letter, and some of the stuff has compiled very fast ( Celeron 533 as client Athlon 1800 XP server ). Gcc though doesn't seem to use distcc. At some point into the compilation it switches to something it calles xgcc and compiles using that.

----------

## phong

Unfortunately, that is a fact of life when compiling gcc.  First it compiles itself using the host compiler (in this case, gcc via. distcc) using very conservative options.  Then using that compiled version, it compiles itself again using optimizations.  This part can't be done with distcc because it needs to use its freshly compiled copy.  Then it compiles itself for a third and final time using the new optimized version.  Again, this can't be done with distcc.  Finally, the two final versions are compared.  If they're identical, it considers itself to be working and installs.  If not, it knows something bad happened along the way.  Unfortunately, compling three times is slow, and more noticably when using distcc since that only works for the first one.

----------

## Jeld

Sorry for my last comment, I already found this information in a different thread. currently I am doing emerge system, and some packages seem to be not distcc friendly. So far I found that bash will use gcc no matter what MAKEOPT says.

Edit: I will start a collection of packages that do not use MAKEOPTS

bash-2.05b-r3

groff-1.18.1

perl-5.8.0-r9

db-3.2.9

expat-1.95.5-r1

ttmkfdir-3.0.4

pam-0.75-r11

XFree is compiling now, so my collection will probably have to stop here ( will let emerge run overnight ). A couple of things worth mentioning abuot speeding up the compile.

1. If you are trying to install on a slower system and have one fast system available you can try to partition the drive, mount partitions in /mnt/gentoo export /mnt/gentoo over nfs and mount on the fast system.  Then you chroot and build. I cuoldn't get the nfsd running off the live CD, but then again, my CD is a bit outdated ( 1.4_rc1 I believe ).

2. On most systems ( even the slower ones ) hard drive access times are much slower then compile times. Therefore setting MAKEOPTS to -j8 or something similar may speed up the compile even on a single CPU system due to the fact that several source files are read at once and then compiled according to the kernel schedule, instead of having to wait for multiple disk accesses.

3. In the same spirit, during compiles gcc writes temp files into /tmp doing 

mount -t tmpfs tmpfs /mnt/gentoo/tmp 

imprives compile speeds dramatically on both my Athlon 1800 XP and Celeron 533.

Good luck.

----------

## AlterEgo

 *Jeld wrote:*   

> 
> 
> 3. In the same spirit, during compiles gcc writes temp files into /tmp doing 
> 
> mount -t tmpfs tmpfs /mnt/gentoo/tmp 
> ...

 

Jeld, please explain this one. I don't understand the gain in this   :Shocked: 

----------

## Jeld

I am not sure, but logicaly speaking, disk writes are time consuming, and since the temp object files are going to be needed by gcc right away teh write  operation have to be completed then and there ( i.e. the write is not really buffered ). So instead of writing temp files to the disk you are writing them to a RAM disk ( tempfs in this case ). Writing to a tempfs disk is much faster then teh hard drive so the compile goes one read/write operation faster. At least that is how I explain this, I might be wrong. It does speed things up quite a bit even on my Celeron 533 128 MB RAM. Also some other operations require temp files such as gzip, groff etc.

----------

## ekoontz

Just to follow up on the suggestion that BradN made above.

(also mentioned by Martin Pool here : http://lists.samba.org/pipermail/distcc/2002q4/000308.html

I tried having : 

```

MAKEOPTS="-j3 CC=distcc CXX=distcc g++"

```

But this did not work for some packages that seemed to want CXX to be one token. So, I wrote a wrapper script which I installed as /usr/bin/distcc-c++ :

```

#!/bin/sh

distcc c++ $*

```

Then I put in my make.conf :

```

MAKEOPTS="-j3 CC=distcc CXX=distcc-c++"

```

Works well now!

----------

## Donovan

 *Jeld wrote:*   

>  Writing to a tempfs disk is much faster then teh hard drive so the compile goes one read/write operation faster.

 

Exactly!  Using tmpfs means it's being held in RAM, and no matter how fast your hard drives are, they'll never be as fast as pure RAM.  Since it's not an actual RAM disk, we don't have to worry about allocating RAM to something that won't be used.  If tmp is blank, no memory is used, as it's allocated on the fly.  Additionally, this frees up the HD from having to handle the temp files.  It's great!  :Very Happy:   (You need TMPFS support in your kernel of course)

The Amigas had a RAM disk like this back in the 80s.  Of course, back then, the GUI and programs ran on a machine with 1/2MB of memory (and still do).  What are the requirements for KDE again?    :Laughing: 

----------

## Normie

I've got a slow Gentoo system (400 mhz p2, 320mb ram), and a fairly speedy XP system (1.4ghz athlon, 512mb ram). I originally wanted to install gentoo on the XP system and boot it with a grub floppy for some good ole' distccd whoring ( :Wink: ), but for really stupid reason's I can't (and won't bother to) explain here, I can't repartition. So I figure I'll just use vmware when I need distccd assistance. Does anyone know how much my performance can expect to be cut (as compared to a "true" gentoo install)? 50%? 60%? What?

(edit) this is OT, I know, but if anyone knows a better way than vmware to install a barebones, made-for-distccd gentoo system without repartitioning, please PM me.(/edit)

----------

## elpierco

I started emerging the system and I see using netstat and top u distcc that it is distributing work to my other machine(an Athlon XP).  I am trying to build the system on a P4 2.8C...the emerge runs for a while then I get this

checking for i686-pc-linux-gnu-g++... gcc

checking whether we are using the GNU C++ compiler... no

checking whether gcc accepts -g... no

checking dependency style of gcc... none

checking how to run the C++ preprocessor... /lib/cpp

configure: error: C++ preprocessor "/lib/cpp" fails sanity check

See `config.log' for more details.

configure: error: /bin/sh './configure' failed for autoconf-lib-link

!!! ERROR: sys-devel/gettext-0.12.1 failed.

!!! Function econf, Line 362, Exitcode 1

!!! econf failed

Should I try to emerge that without distcc?  sys-devel/gettext-0.12.1?

here is what my make.conf looks like

# These settings were set by the catalyst build script that automatically built$

# Please consult /etc/make.conf.example for a more detailed example

CFLAGS="-O3 -march=i686 -pipe -fomit-frame-pointer"

CHOST="i686-pc-linux-gnu"

CXXFLAGS="${CFLAGS}"

USE="-qt -kde"

FEATURES="distcc"

DISTCC_DIR="${PORTAGE_TMPDIR}/.distcc"

MAKEOPTS="-j5"

CC="distcc gcc"

CXX="distcc g++"

----------

## elpierco

So I am going to use the minimal install disk and try this one more time...

Just to have some confirmation on the machine (localhost) that will be building the stage 1 and distributing jobs to others.  Does it matter if CC="distcc" is set on the other boxes? 

Does it matter if this is on a seperate line or should it be included with MAKEOPTS i.e. MAKEOPTS="-j7 CC=distcc" 

Does anyone have any ideas what that error might be?  This is a brand new system components wise.   I will be using a P4 2.8GHz and an Athlon 1800XP.

----------

