# distcc - it works (quick but dirty...)

## de4d

if anybody wondered howto use distcc during an emerge process:

(warning: this method might be ridiculous - better ideas appreciated :)

```

emerge distcc #(if !done)

mv /usr/bin/make /usr/bin/made

```

vi/nano/whatever /usr/bin/make add:

```

#!/bin/bash

JINT=5#or whatever u like

export DISTCC_HOSTS="hostA hostB [localhost]"

export DISTCC_VERBOSE=1 #if u wanna c it working

/usr/bin/made CC=distcc -j$JINT $1 $2 $3 $4 $5 $6 $7 $8 $9#no idea how to pass all (n) args... :P

```

now

```

emerge <whatever>

```

could use distcc

have phun

edit:

ok now, $@ passes all arguments

but theres another problem:

the portage ebuild (and several others) use

make CFLAGS="-march=i386 -O3"

when catching this up to pass it to the real make binary, those double quotes are stripped. this is nothing strange - but make complains about a syntax error (as expected). does anybody know a workaround for that?

----------

## phong

Ok, I've been doing a lot of playing with distcc over the past week or two, and I think I can give some tips that will be slightly quicker and a bit less dirty.  I planed on writing up a whole HOWTO when I determine if/how distcc can be used for the bootstrap (I've already gotten it to work with emerge system, the kernel compile, most of kde and several misc. things).  In the mean time, here's a quickie.

First off, distcc has no built in security mechanisms.  Before running the distcc daemon on any volunteer machines, it's critical that you block off tcp port 4200 incoming at your firewall otherwise anybody can run whatever they want with the permissions of the user running distccd (which need not and should not be root BTW).

Second step, take a quick look at the distcc web page and emerge distcc on all the involved machines.

Add the following lines to your /etc/make.conf

```
CC="distcc"

CXX="distcc g++"        # may not be neccesary, and may serve only to break things - not sure yet

MAKEOPTS="-j3"          # replace 3 with a number slightly higher than the number of CPUs that will be working

DISTCC_HOSTS="volunteer1 volunteer2 ... localhost"

#DISTCC_VERBOSE=1       # if you want distcc to spew extra output

#DISTCC_LOG="whatever"  # to log distcc instead of using stderr, BUT it's limited to the sandbox
```

Now go ahead and run distccd on the volunteer machines.  Do "distccd --help" to get some options for verbose output and whatnot (it'll work fine with no options though).  Before going any further, huff and puff at port 4200 from outside your network to be sure your house is made of bricks.

Now anything you emerge on the original machine should use distcc.  Some packages disable the -j option because their makefiles (or whatever) are written in such a way as they cannot be compiled in a parallel fashion.  Also, some packages just don't compile with distcc for whatever reason.  If one does, just comment out the lines you added to make.conf.  If the primary machine is quite a bit slower than the volunteers, you can just comment out the MAKEOPTS line and it may still be able to compile, just passing off the compile job to the volunteers - just not in parallel, and you may still get a significant speedup.  So far, I've had arts fail (just taking out the MAKEOPTS let it through, and it took advantage of compiling on a faster volunteer machine).  I also had kdelibs fail.  After taking out all the options from make.conf, I tried it again, and it failed right away.  I thought I'd borked it up good, but then I tired it a third time and it went through no problem.  So, the moral is, if a package fails with distcc, it may be a good idea to blow out the corresponding /var/tmp/portage directory manually before trying it again.

Your kernel is one thing that CAN take advantage of distcc for sure, and it gets a really nice boost.  It won't use your make.conf settings though:

```
# make menuconfig

# make dep && make clean

# export DISTCC_HOSTS="....."

# make CC="distcc" -j3 bzImage

# make CC="distcc" -j3 modules

# make modules_install

# [copy your kernel, etc. etc.]

```

Good luck!

----------

## de4d

tanx a lot!

ive been trying similar stuff before i ended up replacing make by a script - just didnt get it to work ...

should have read some more documentation about make ;))

----------

## zen_guerrilla

 *phong wrote:*   

> Now go ahead and run distccd on the volunteer machines. 

 

I followed your instructions on my desktop (volunteer) & my notebook ("emerger"). However when running distccd I get no message and "ps ax | grep distccd" returns null. Also when emerging something on the notebook it returns that connection is not allowed on port 4200 on desktop. 

I tried distccd --verbose --daemon but still no message and no ps. Am I doing something wrong here ?

.:: zen ::.

----------

## phong

Hmmm, does distccd --help give any output?  It should give some output to stderr if run with --verbose though...  Can you try running it with --log-file=test.log or something to see if anything appears in the log file?

----------

## zen_guerrilla

The log simply told me not to run distcc as root, oops  :Cool:  Now everything works gr8. I' m finally emerging gnome2  :Smile: 

phong : Thanx for the great tip on distcc. Now I'm gonna finish my gentoo installation at work.

.:: zen ::.

----------

## verwilst

I have the following in /etc/make.conf:

CC="distcc"

CXX="distcc g++"        

MAKEOPTS="-j2"          

DISTCC_HOSTS="gateway localhost"

DISTCC_VERBOSE=1

and i get this when trying to compile an app with emerge:

checking whether make sets ${MAKE}... yes

checking for i586-pc-linux-gnu-gcc... distcc

checking for C compiler default output... configure: error: C compiler cannot create executables

!!! ERROR: The ebuild did not complete successfully.

!!! Function src_compile, Line 3, Exitcode 77

!!! (no error message)

Any suggestions?

----------

## rac

Anything in config.log that might give us more detail?

----------

## phong

I assume you built distcc before putting that stuff in your make.conf...  Can you try just compiling a single .c file with distcc, i.e. "distcc hello_world.c"?  Before you do that, export DISTCC_VERBOSE="1" so it will give you more info.

----------

## gentuse

I would recommend setting CC="distcc gcc" and CXX="distcc g++" to overcome the bug described here:  http://lists.samba.org/pipermail/distcc/2002q4/000308.html

----------

## kerframil

Thanks so much for this thread, in particular the recommendations for make.conf. It works like a charm, although some builds fail or don't take advantage of distcc in the desired manner (I'm keeping a log of these because I'm setting up a new system).

However, I think I just discovered something rather worrying. I was proceeding to emerge app-misc/endeavour on a pretty clean system. I noticed this message when compilation of the tiff library ebuild dependency began:

```
Reading site-wide parameters from ./config.site.

Oh no, not another i686-pc-linux-gnu system...

Using /usr/bin/distcc for a C compiler (use -with-CC=compilername to override).

Looks like /usr/bin/distcc supports the -g option.

... but not together with the -march=pentium3 -O3 -pipe option, not using it.

Using /usr/bin/make to configure the software.

Defaulting MACHDEPLIBS to -lm.
```

Hmm, a quick check of the gcc manpage explains that the "-g" option produces debugging info in a fashion specific for the host's arch. This message seems to imply that use of that gcc flag prevents distcc from using any optimisations! This is quite distressing, because I have emerged a few things now, and I'm not sure if this has happened on any other builds.

If I've got the above right, then I'm wondering whether it would make sense to report them and whether an ebuild can somehow "override" the use of distcc in this fashion, or whether that would be too kludgy. In any case, I think I'll shunt this along to distcc's author because just maybe this issue can be resolved in a future release (I hope).

PS: I tried going through a few different permutations with the C compiler flags "-march=i686", no "-pipe" but it seems to make no difference

----------

## rac

pahud's question has gone to distcc problem.  Please don't post support questions to threads in Documentation, Tips & Tricks.

----------

## yokem55

Just how scaleable is distcc?  Could you say have a 100 machines on a network all helping make a single build that could then be applied to all the systems?  I mean, if this could cut a KDE compile time down to minutes when distributed so broadly, I wouldn't be suprised to see corporate sysadmins embracing Gentoo since it would be able to produce full binary packages just as fast as they could download them from the net.

----------

## puddpunk

yep, sure!

just on your emerger be sure to set the config up correctly with your 100 slaves, then emerge with the --buildpkg command, and it will build a package which can be distributed over all 100 machines (you'll have to do that yourself though)

----------

## zojas

I use distcc in combination with ccache. that way if the ebuild fails, once you fix it there's not as much work to do.

make sure you have distcc and ccache installed. I put /usr/bin/ccache as the first directory in root's path. 

then in your make.conf file: (almost the same as the code above):

```
CC="ccache distcc gcc"

CXX="ccache distcc g++"

MAKEOPTS="-j3"          # replace 3 with a number slightly higher than the number of CPUs that will be working

DISTCC_HOSTS="volunteer1 localhost"

#DISTCC_VERBOSE=1       # if you want distcc to spew extra output

```

then on all the volunteers you need to change the /etc/init.d/distccd script slightly. just before the "start-stop-daemon" line, add this line:

```
export PATH="/bin:/usr/bin"
```

otherwise the distccd server tries to run ccache too.

With this setup, the machine you run the "emerge" on will cache all the compiling. any compiling that gets done will be distributed.

----------

## mglauche

KDE does not seem to like distcc, so i doubt the 100 machines will help compiling it ..

I think for compiling on such many machines distcc would have to have finer controll over the make process, in order to be still efficient.

I usually get only 30-40% speed improvement by compiling on 3-4 machines, with the distant once being quite a bit faster than the local machine ...

----------

## zojas

I posted some timings using ccache+distcc on emerging mozilla. There are three cases: with distcc+cold ccache, distcc+warm ccache, and with no distcc/no ccache. the times come out as you expect.  :Smile: 

http://desertsol.com/~kevin/ccache_distcc.html

----------

## fyerk

 *phong wrote:*   

> 
> 
> First off, distcc has no built in security mechanisms.  Before running the distcc daemon on any volunteer machines, it's critical that you block off tcp port 4200 incoming at your firewall otherwise anybody can run whatever they want with the permissions of the user running distccd (which need not and should not be root BTW).
> 
> 

 

One thing to note here is that distcc can be run from within [x]inetd. Permissions can then be controlled at the system level via native xinetd access controls or /etc/hosts.(allow|deny).

----------

## proxy

I have noticed an issue which was preventing distcc from working at all:

in the newer ebuilds for gcc, there are two new lines in /etc/env.d/05gcc

they are:

```

CC="gcc"

CXX="g++"

```

these seem top override the values we have been putting in make.conf, therefore making it NOT call "distcc gcc"

the solution is simple, it worked without this before, so comment them out  :Smile: 

then just do a env-update

I wish there were a more elegnant solution, such as it conditionally setting the values, perhaps the script which reads the /etc/env.d/ files should skip environment vairables which have already been set?  or be able to specify ones which will not get overwritten?  I dunno, what do you guys think?

proxy

----------

## nitro322

 *proxy wrote:*   

> I have noticed an issue which was preventing distcc from working at all:
> 
> in the newer ebuilds for gcc, there are two new lines in /etc/env.d/05gcc
> 
> they are:
> ...

 

Thanks, proxy.  I couldn't seem to get distcc to work at all, and this turned out to be my problem.  Works great now.

I do have a question that I'm concerned about, though.  My plan is to run the distcc client on my firewall/router box, a P1 133, and run distccd on my desktop, an Athlon 2100+ MP.  Needless to say, I have quite different CFLAGS on the two machines.  I want my desktop to do as much of the work as possible, so I have my make.conf (on the firewall) set to:

```
DISTCC_HOSTS="desktop"

DISTCC_VERBOSE=1

MAKEOPTS="-j2"

CC="distcc"

CXX="distcc g++"
```

This will compile on my desktop, but I'm not sure what CFLAGS it's using.  I have the desktop set with fairly aggressive optimizations that the firewall definitely can't handle.  So, which CFLAGS options will packages on my firewall be compiled with?  Will this configuration produce stable binaries?  Thanks.

----------

## zojas

the client tells the distccd what flags to use. I use distcc routinely with a pentium II as a client and an athlon as a server and it works fine. I've also used a 90MHz pentium as a client and that worked too.

----------

## mbp

I doubt if distcc would scale to 100 machines, for a few reasons:

- Many Makefiles are written in such a way that they can't schedule 100 jobs at once.  For example, they may not parallelize across directories, and each directory may have <100 files.

- The client may run out of steam trying to run the preprocessor and distcc client for 100 machines.  The client-side overhead is a few percent of the total work, which suggests an upper limit of a few tens of machines.

- The network may saturate trying to pass source and objects around.

Aside from that there is no in-principle reason why distcc couldn't scale up to that number.  It would however be interesting to try.  Unfortunately I don't have 100 Linux machines to try on.   :Sad: 

----------

## Decibels

I am having a bit of confusion on the CC & CXX flages. I see most people include them in their make.conf. 

Then you also see stuff like this:

 *Quote:*   

> Regardless of whether you are using portage or not, do not set CC or CXX; the simple default values provided by gcc-config in /etc/env.d/05gcc are used. Setting these to anything that is not a single C program will cause problems with some builds that use incompatible versions of libtool.

 

from: http://gentoo.superlucidity.net/www/distcc.html

I had just taken them out of make.conf when I was pointed to this page. Only tried to compile the new gcc so far and it failed. Maybe it doesn't like distcc, but still not clear.

Since /etc/env.d/05gcc contains the same info seem like you wouldn't have to  include it anywhere else.

----------

## nephros

I have four machines at home, all running distccd, but not all are up all the time.

I use the following script therefore, like this:

```
[root@scourge]# distcc.sh emerge foo/bar
```

```

CC="distcc gcc"

CXX="distcc c++"

DISTCC_HOSTS="localhost"

MCNT=3 # I have a SMP system, set to 2 for single-cpu

for TRY_HOST in 192.168.0.{1,2,13,23}

do

  echo trying $TRY_HOST

  if ping -q -c 3 -w 5 $TRY_HOST > /dev/null 2>&1 ; then

    echo "$TRY_HOST is up!"

    DISTCC_HOSTS="$DISTCC_HOSTS $TRY_HOST"

    let MCNT++

  fi

done

MAKE="make -j $MCNT"

echo "set: \$CC=\"$CC\" \$CXX=\"$CXX\" \$DISTCC_HOSTS=\"$DISTCC_HOSTS\" \$MAKE=\"$MAKE\""

$@

```

----------

## nacs

After following the instructions in the posts above, I found that distcc wasn't being used. However adding the following line to make.conf fixed it immediately:

```
FEATURES="distcc"
```

----------

## Decibels

Ya, I haven't had any problems lately with distcc and no CC or CXX in the make.conf file. Works fine. Everyonce in a while something like libxml won't compile with it, but that is all.

```
# Distcc 

FEATURES="distcc"

DISTCC_HOSTS="localhost Compaq"

DISTCC_VERBOSE=1

MAKEOPTS="-j4"

##  http://gentoo.superlucidity.net/www/distcc.html said don't do below.

## CC="distcc"

## CXX="distcc g++"
```

----------

## hackerError

mine works with just adding this to make.conf

```

DISTCC_HOSTS="localhost jeremy2"

FEATURES="distcc"

PORTAGE_NICENESS=20

#portage niceness makes portage be nice, so i can play unreal and #emerge openoffice at the same time ;)

```

however, i thought i read that you shouldnt set your CFLAGS to distcc... it seems you guys are doing this for some reason[/code]

----------

## pele_smk

Sorry to beat a dead horse, but I'm following the ooooo sooo simple directions on the gentoo docs:

emerge distcc

# nano -w /etc/make.conf

(Set N to a suitable number for your particular setup)

(A common strategy is setting N as twice the number of total CPUs + 1

available)

MAKEOPTS="-jN"

(Add distcc to your FEATURES)

FEATURES="distcc"

and then: 

/usr/bin/distcc-config --set-hosts "192.168.0.1 192.168.0.2 192.168.0.3"

I only have one host: my desktop which is 192.168.1.105 so I put tha tin there and take out the rest

on both machines I have distcc emerged, I put the --allow 192.168.0.0/24 in the opts just to try and let every ip in...I'll change that after I get it working.

Could someone explain what the --listen means? 

I've done the simple only trying to use distcc to build kernel modules and I've tried merging complete packages....Both give me the "could not distribute, compiling locally" error. When I built my kernel modules, I got the error: "connection refused by peer" I thought that was a good sign, but didn't point me in the right direction:

Thins I know...

Both machines are using distcc version 2.18

I'm using my desktop amd64 to build for my laptop x86. I use the -m32 flag.

The respectable spots in my  make.conf:

MAKEOPTS="-j2"

FEATURES="distcc"

----------

## Kuhndog86

 *pele_smk wrote:*   

> 
> 
> Thins I know...
> 
> Both machines are using distcc version 2.18
> ...

 

From what I have read, if you want an amd64 machine to use distcc with an x86 machine, you need to run distcc in a 32-bit chroot environment on the amd64 box in order for it to work.

see: https://forums.gentoo.org/viewtopic-t-404413-highlight-distcc+amd64+x86.html

----------

