# PAINLESS distcc setup

## ObsidianBlade

As a note to everyone that would like a PAINLESS distcc setup:

PLEASE NOTE: While I'm a relative newcomer to the linux community, I have about ten years of extensive DOS and Windows experience.  Yeah, I know that it dosen't sound like a lot, but think of all the crap that you have to fix on MS platforms in 10 years  :Confused:   As a newcomer to linux, the following setup SEEMS to have the correct versions of distcc and gcc, HOWEVER, I'm not an expert (hell, I'm barely an entry level linux user  :Wink: ) If you have questions about cross-compiling get expert help before you attempt this.

1. install distcc as outlined in the DISTCC Guide:

http://www.gentoo.org/doc/en/distcc.xml

2. Download and burn the KNOPPIX 3.3 (2004 version) LiveCD (yeah, I KNOW it's sounds cheesy but it has distcc already installed)

3. Boot Knoppix on all of the machines that you want in you distcc farm and run the following command line from the 'root shell' on each one:

```
# distccd --user nobody --daemon -jN
```

4. run the following command on each machine in the farm and note the IP addresses

```
# ifconfig
```

5. On the machine that you are running the compile from, likely your Gentoo box (which is actually the client - the rest are servers, go figure), add the IP addresses to the following command where hostx is the IP address of each machine in the farm, starting with the most powerful and working down.  Yes, the local machine is entered as 'localhost'. 

```
# /usr/bin/distcc-config --set-hosts "localhost host1 host2 ....."
```

You now have a distcc farm.  It dosen't get ANY easier than that.  knoppix dosen't seem to have a CPU usage monitor, correct me if I'm wrong, but you can kind of get an idea of workload by watching the 'Memory' section of 'Info Tool'

BTW, if you run this stuff from a window manager, you can set up the distcc gnome monitor and browse the man pages to kill time.   Yeah, read the man pages - you might be surprised what you can learn.

good luck and Enjoy!

EDIT 1:

I don't believe that the DISTCC Guide went over adding these to your make.conf, but the addition of 'distcc' to the features line is needed and apparently adding 'ccache' helps as well.  The other two lines are config lines for ccache and distcc.

```
FEATURES="distcc ccache"

CCACHE_SIZE="2G"

DISTCC_DIR="${PORTAGE_TMPDIR}/.distcc" 

```

***NOTE TO USERS WHO EDITED THE 'make.conf.example' TO GET THEIR 'make.conf' FILE: In the third line you MUST change '{PORTDIR}' from the example present in make.conf.example to  '{PORTAGE_TMP}'.  The line DISTCC_DIR="${PORTDIR}/.distcc WILL cause Access Violations because it is trying to run in a directory that it dosen't have access to (or some such thing.  I saw a note from Gentoo's resident distcc goddess that says why.  If I find it again, I'll stick in  a link.)

EDIT 2:

Some days I just feel like a moron.  RTFM!!!!  I just spent 40 minutes trying to figure out why distcc wasn't working during an emerge system.  Read the FINE PRINT: "Note: It is important to note that during bootstrap and emerge system distcc may not appear to be being used. This is not to be unexpected. Some ebuilds or packages do not work well with distcc, so they intentionally disable distcc.  This also holds true for Mozilla and XFree (according to the DISTCC Guide).

Ouch! (stupidity should be painful!)

----------

## Blue Lightning

Very cool.. thanks!

----------

## christsong84

great tip...I never would have thought of that.  :Very Happy: 

----------

## axxackall

 :Question:   Is there any PAINLESS way to setup distcc to compile cross-platform? I've tried to mix distcc farms of x86 and ppc machines, but they don't like to compile the code of each other.   :Crying or Very sad: 

----------

## endtransmission

thanks man, this was a great help.

----------

## endtransmission

btw, rumor has it that it's important to annotate ccache before distcc in your options.

----------

## pYrania

rumors are wrong.

----------

## Boris27

By the way, if you have a package that doesn't want to be distcc'd, it'll be compiled on the first host in your hosts list. It pays to put your fastest machine in first.

----------

## endtransmission

 *pYrania wrote:*   

> rumors are wrong.

 

ah.  fair 'nuf.  thanks.  good to know.

----------

## sorg

great!

----------

## mOjO_420

just wanted to say thanks!

this is really awesome... my friend came over and we had the following distcc network going:

```

|---------------------------------------------------------------|

|# |       System   | IP Address   |  Combined bogomips   |

|--|--------------------|---------------|-----------------------|

|1.| 2 x Athlon-MP 2400   | 192.168.1.100   |   7982.27    |

|2.| athlon-XP 2500   | 192.168.1.165   |   3630      |

|3.| athlon-XP 1700   | 192.168.1.102   |   2883.58    |

|4.| Duron 1000mhz   | 192.168.1.160   |   1998.84    |

|5.| Duron 950mhz   | 192.168.1.161   |   1900.54    |

|6.| Athlon 700mhz   | 192.168.1.105   |   1395.91    |

|7.| P3 900mhz       | 192.168.1.168   |    655.36    |

|--|--------------------|---------------|-----------------------|

|T.|   11,850      | 192.168.1.0/32|     19,446.5      |

|---------------------------------------------------------------|

```

 (looks like my nice ascii graphs didnt paste very well.. oh well.)

2nd run configuration had the 2 x Athlon-MP system booted to windows

instead of Knoppix but was running two separate copies of vmware

both booted into knoppix 3.3 (02-09-04 release which has GCC 3.3.3).

```

|-----------------------------------------------------------------------|

|# |       System      | IP Address   |   bogomips    |

|--|----------------------------|---------------|-----------------------|

|1.| Athlon-MP 2400 (vmware)   | 192.168.1.159 |   3971.48      |

|2.| Athlon-MP 2400 (vmware)   | 192.168.1.180   |   3984.58    |

|3.| athlon-XP 2500      | 192.168.1.165   |   3630      |

|4.| athlon-XP 1700      | 192.168.1.102   |   2883.58    |

|5.| Duron 1000mhz      | 192.168.1.160   |   1998.84    |

|6.| Duron 950mhz      | 192.168.1.161   |   1900.54    |

|7.| Athlon 700mhz      | 192.168.1.105   |   1395.91    |

|8.| P3 900mhz          | 192.168.1.168   |    655.36    |

|--|----------------------------|---------------|-----------------------|

|T.|   11,850         | 192.168.1.0/32|     19,446.5      |

|-----------------------------------------------------------------------|

```

With the first run config we compiled KDE (only KDE, X was already installed and the KDE packages had been fetched already so download time was not an issue) on the P3 900mhz laptop in less than an hour!!! We spent about as much time setting up the distcc as we did compiling. It was crazy to watch  because the laptop would take its sweet time unpacking the tarballs then running configure then you would see this streak of text scream by in a heartbeat (that was the make processes) then it was back to the slow local operations.    :Cool: 

I should also note that not all of the machines were using knoppix a couple are already running gentoo systems all using GCC 3.3.2

anyway.. just thought i'd share my experience and thanks for the how-to.

edit:   on another thread somewhere about distcc i spotted this little iso... its a VERY minimal bootcd spefically for distCC (only a 38MB iso) so it could be loaded onto one of those USB Flash drives or similar device and has less overhead than the normal knoppix cd. I havent tested it yet but it looks cool.   http://opendoorsoftware.com/cgi/http.pl?cookies=1&p=distccKNOPPIX

----------

## endtransmission

I seem to continue to have problems - or it feels like I do.  

While it looks like the other computer I'm using is doing work, it appears to be only preliminary work while my slow box still sloggs through the compiling - wish the reverse could be true - where my box does the preliminary work and the box I have helping me do the serious crunching.  

I've done everything I can think of to try to work the other machine harder but to no avail.  Its a 1.4Ghz Pentium 4 and it sits on 95% idle for the most part even though the gnome monitor shows it doing prelim work (the purple bars).  

I've even gone so far as to take localhost out of the list of hosts in an effort to see an improvement - nada.  i've tried booting up with different -jN variables (both in the host daemon and my own make.conf file), i've tried adjusting the number of jobs in the host config (ip 192.168.1.101/4 etc...), have tried switching the order of the hosts in the host config file... zilch.

my box is gentoo box running a first generation k7 Athlon 700mhz "Socket A" chip on an old asus k7m board.  The box I have helping me is generously loaned to me by my roomie - booted into knoppix 3.3.  We're both using the same gcc version 3.3.2.

Perhaps this isn't the forum to ask this question, in which case I can post an individual thread.  But in the event it's something silly, I thought I'd ask it here first.

Have been compiling Gnome for the past 14 hours but this post comes after futzing with it for the past 48 trying a series of different items to compile.

Any help would be welcome.

----------

## slonocode

Question:

Will there be problems if the client is running gcc3.2.3 and the server(ie the knoppix booted) are running gcc3.3.1?  Or even if one is 3.3.2-r5 and the other is 3.3.1?

Also if the gcc versions aren't exactly the same but the compile completes is it safe to assume everything is ok or could there be hidden or runtime problems associated with this?

Also are there any other compatablitly issues to worry about?

Thanks

slonocode

----------

## ObsidianBlade

 *endtransmission wrote:*   

> I seem to continue to have problems - or it feels like I do.  
> 
> While it looks like the other computer I'm using is doing work, it appears to be only preliminary work while my slow box still sloggs through the compiling - wish the reverse could be true - where my box does the preliminary work and the box I have helping me do the serious crunching.  
> 
> I've done everything I can think of to try to work the other machine harder but to no avail.  Its a 1.4Ghz Pentium 4 and it sits on 95% idle for the most part even though the gnome monitor shows it doing prelim work (the purple bars).  
> ...

 

I'd ask in the "Portage and programming" forum, you've already done all of the stuff that I would have thought to do....  I assume that you've checked the man pages?

----------

## ObsidianBlade

 *slonocode wrote:*   

> Question:
> 
> Will there be problems if the client is running gcc3.2.3 and the server(ie the knoppix booted) are running gcc3.3.1?  Or even if one is 3.3.2-r5 and the other is 3.3.1?
> 
> Also if the gcc versions aren't exactly the same but the compile completes is it safe to assume everything is ok or could there be hidden or runtime problems associated with this?
> ...

 

Anybody with hard answers feel free to jump in here. I'm not an expert, but through the reading that I've done it SEEMS that as long as the versions of gcc are the same major revsion you should be safe, ie: all are 3.3.x

Once again, I'd drop a line in the 'Portage and Programming' forum and see what you come up with.

----------

## endtransmission

 *ObsidianBlade wrote:*   

>  *slonocode wrote:*   Question:
> 
> Will there be problems if the client is running gcc3.2.3 and the server(ie the knoppix booted) are running gcc3.3.1?  Or even if one is 3.3.2-r5 and the other is 3.3.1?
> 
> Also if the gcc versions aren't exactly the same but the compile completes is it safe to assume everything is ok or could there be hidden or runtime problems associated with this?
> ...

 

Good call.  Think I'll do just that.

----------

## mOjO_420

i'm using the latest Knoppix 3.3 CD (02-09-04) which has gcc 3.3.3 on it and not having too many problems... the previous Knoppix 3.3 CD had gcc 3.3.2 though.. and if i have issues i may switch to using it.  

in response to endtransmission:

i had that problem at first... did a lot of tweaking.. 

my setup is pretty complicated.. my friend took his boxes home so i'm down to an Athlon MP dual 2400 box, an athlon XP 1700 shuttle, a Duron 1ghz, and a Classic Athlon 700 (slot A).  The dual Mp 2400 box is running two instances of Knoppix ver. mentioned above in two different VMWare sessions (booted to runlevel 3 with 'knoppix 3 dma' options and i gave them 128mb of ram each.  To further complicate things I am actually updateing two gentoo machines at once... The athlon XP 1700 shuttle is getting latest KDE and QT and the Classic Athlon 700 is getting newest Gnome and other goodies.  The classic athlon 700 machine can barely handle just managing all the distcc connections so i removed it from the hosts file altogether and its still at 100% cpu all the time. (its a real dog) 

the hosts file on that machine looks like this:

```
192.168.1.180/4 192.168.1.159/4 192.168.1.102/2 192.168.1.160/2
```

the first two machines are the VM's running on my dual MP 2400 machine.

each of the VM's (the knoppix ones) i ran with:

```
 distccd --user nobody --daemon -j8
```

yeah i know i set max to 4 jobs in hosts file BUT i'm compiling on two machines and i wanted the VM's running on the dual machine to absorp most of the work.  btw.. if you wonder why i'm running two its because its a dual cpu machine and vmware in windows wont use over 50% of my cpu's if i dont run two. 

the other computer (athlon xp 1700) has a hosts file like this:

```
192.168.1.159/4 192.168.1.180/4 192.168.1.160/2 localhost
```

and the pertinent parts of the make.conf on the athlon 700 are as follows:

```
MAKEOPTS="-j12"

FEATURES="distcc ccache"

```

i think you're seeing all those cc1plus threads in top on the athlon and thinking it must be doing the compiling but its probably not... its just strained in pushing all that info in and out... i can tell its working as i can watch the cpu go up dramatically on the other machines and also it just compiled firefox in 30 min... what you do also need to keep in mind is that its quick bursts of cpu activity on a fast machine.. i have been watching the VM's eat cpu and the graph looks like a heartrate monitor it spikes quick then drops then spikes then drops... so it seems like its doing nothing most of the time but you're just missing a lot of the peaks and valleys... 

definitely do the following:

1.)  increase the number of jobs on the fast machine to 4 or 6 in the hosts file and when running distccd -j#

2.) take the poor little athlon 700 out of the hosts file altogether or give it one measly thread to chew on.

only make's and gcc instances are going to use distcc anyway.. not the prepping or config stuff all has to be run locally.

hope this helps.  :Wink: 

EDIT:   also you can set a log file for distcc on the athlon with distcc-config then run a tail -f on the log file.. gives more useful information than just watching the compile go... if theres errors its getting when connecting to the other machine it will show them there.

----------

## endtransmission

 *mOjO_420 wrote:*   

> i'm using the latest Knoppix 3.3 CD (02-09-04) which has gcc 3.3.3 on it and not having too many problems... the previous Knoppix 3.3 CD had gcc 3.3.2 though.. and if i have issues i may switch to using it.  
> 
> in response to endtransmission:
> 
> i had that problem at first... did a lot of tweaking.. 
> ...

 

'Preciate it.

Will go back and do some tweaking to see if anything improves but my results are precisely the same as yours.  My poor little classic athlon stays pegged at the 100% cpu mark and the cpu on my host reads like yours - like a heart monitor with just pegs of activity.  I have the Top frequency set to update set every half second so it's updating very quickly.  When I look at the Gnome X monitor (DISTCC_DIR=/var/tmp/portage/.distcc distccmon-gnome) It shows my athlon doing nothing but greenbar-compiling and shows my host doing almost purely purplebar preprocessing work with bursts of compiling thrown in there.

at any rate I cross posted my question here 

https://forums.gentoo.org/viewtopic.php?p=875743&highlight=#875743

in an effort to not hijack this thread.  I'm aware that this isn't a support forum and my questions here seem to be leaning towards the support category.

Thanks again.  Great thread.  Thank you for that.  

Mods: if you'd like to erase my posts - or move them to the thread I posted, I'd understand completely.  Perhaps you can make my thread the foundation for a Duscussion/Response thread about this post so to keep it clean?  Just a thought.  :)

----------

## Clete2

There would not happen to be a Windows distcc? The fastest processer in the house is running XP (not for long, I'll take over soon  :Smile: ).

Not vmware...  :Razz: . I COULD get 2 knoppix CDs but my mom needs to use the computer often...

EDIT: AHA! I have heard of Cygwin, but never tried it. Now is the time for trying it.  :Smile: 

----------

## mOjO_420

 *Clete2 wrote:*   

> There would not happen to be a Windows distcc? The fastest processer in the house is running XP (not for long, I'll take over soon ).
> 
> Not vmware... . I COULD get 2 knoppix CDs but my mom needs to use the computer often...
> 
> EDIT: AHA! I have heard of Cygwin, but never tried it. Now is the time for trying it. 

 

vmware is a program that lets you install any os you want into a virtual environment.. you can install it on WinXP and then boot knoppix inside it, then minimize it and let your mom use the computer...  unfortunattely its commercial software... however there was a thread on how to setup distcc in cygwin somewhere but it sure looked like a lot more hassle than just *cough cough* obtaining a copy of vmware and using it.

----------

## cylgalad

There's a post in this forum about a full distcc for Gentoo that runs under Cygwin and it's working (help me speed up compiling KDE 3.2 on one of my computers) :

https://forums.gentoo.org/viewtopic.php?t=66930

Install Cygwin and download : ftp://ftp.dympna.com/pub/cross-linux-020904-stripped.tar.bz2

----------

## Clete2

 *mOjO_420 wrote:*   

>  *Clete2 wrote:*   There would not happen to be a Windows distcc? The fastest processer in the house is running XP (not for long, I'll take over soon ).
> 
> Not vmware... . I COULD get 2 knoppix CDs but my mom needs to use the computer often...
> 
> EDIT: AHA! I have heard of Cygwin, but never tried it. Now is the time for trying it.  
> ...

 

I am against illegal file-sharing  :Wink: . Thanks for the thread  :Smile: .

EDIT: I followed the guide. I'm emerging kdevelop, but the monitor shows nothing. I am SSHed into the knoppix machine. I hit uptime and it says from 1.00 to 2.00.  (Or it did, now it says 0.50)

EDIT: It's not working. I have a load of 0.00 on the other computer.

bash-2.05b# nano -w /etc/make.conf

bash-2.05b# export PATH="/usr/lib/ccache/bin:/usr/lib/distcc/bin:${PATH}"

bash-2.05b# echo "distcc:x:240:2:distccd:/dev/null:/bin/false" >>/etc/passwd

bash-2.05b# DISTCC_DIR=/var/tmp/portage/.distcc distccmon-gnome

I don't see what I'm doing wrong. I emerged, edited the make.conf to add -j3. I added distcc to FEATURES. I then ran the commands specified in the topic poster's post (except the ccache thing, I'm running as root trying to do this, I recieved no errors). I then did emerge -U world (after starting distcc and setting hosts). I started the monitor:

bash-2.05b# DISTCC_DIR=/var/tmp/portage/.distcc distccmon-gnome

EDIT: YES! I got it, I forgot the trailing / at the end of the monitor thing, it appears to work  :Very Happy: . Finally, OPEN OFFICE HERE I COME  :Very Happy: .

----------

## cptmorgan

the gude and man pages have me a little confused... 

im trying to setup distcc for my laptop so i dont have these huge compile times.  ive read the docs and this guide multiple times and am still having problems...

i already have a box running gentoo and i just want it to help my laptop compile, so do i just install distcc on that box and run the deamon without any other configuring except:

distccd --user nobody --daemon -jN

??

then do i setup distcc on the laptop the same way? but adding hosts to distcc and also features to  make.conf.  this is where im confused... i dont know if all my boxes have to be setup for portage to use distcc or just my laptop that needs help compiling.

ive setup distcc both ways and still get access violation error not being able to make dirs and have tried all the solutions ive found to the problem just for them to change to outcome of the error a differently....   and i dont know if distcc is working cause the gnome distcc monitor im using on my main box isnt showing anything or is it not going to until it starts compiling

----------

## Clete2

try suing to root for a while and then running the commands... just a suggestion.

----------

## endtransmission

 *Clete2 wrote:*   

>  *mOjO_420 wrote:*    *Clete2 wrote:*   There would not happen to be a Windows distcc? The fastest processer in the house is running XP (not for long, I'll take over soon :)).
> 
> Not vmware... :P. I COULD get 2 knoppix CDs but my mom needs to use the computer often...
> 
> EDIT: AHA! I have heard of Cygwin, but never tried it. Now is the time for trying it. :) 
> ...

 

Under the fair-use clause of the copyright act, you are allowed to use it, try it, test it, review it, examine it, inspect it, study it for academic purposes, (etc...) as long as you understand that you do not own it and do not try to make money from it.  There is no set time limit but you are allowed to do this for a "reasonable" amount of time.

If you would like to own a copy for yourself then yes, you must pay for it.    It's important to understand the term ownership conveys a decidedly different set of rights and rules and whatnot.  Posession does not connotate ownership but the term "Evaluation" legally speaking is a relatively broad and sweeping motion.

Incedentally I went the VMware route and strangely enough I'm getting BETTER compile times/speeds running knoppix in a vmware shell then I was just booting Knoppix.  I'm not certain why, but it's something to consider.

Incidentally, it might be good to know - VMware will spawn its own independant IP address when launched so when you boot into knoppix, doublecheck the ip addy with the #ifconfig command.  If you're like me, knowing that will save you alot of frustrating time trying to figure out where you went wrong.  Once you have the VMware IP address it's simply a matter of putting the VMware IP address into your hosts list (and not the normal host IP).

cheers.   :wink:

----------

## axxackall

One of my machines breakes to use distcc with always the same message:

```

--------------------------- ACCESS VIOLATION SUMMARY ---------------------------

LOG FILE = "/tmp/sandbox-dev-libs_-_libmcal-0.7-14164.log"

 

open_wr:   /var/cache/samba/gencache.tdb

--------------------------------------------------------------------------------

```

The package name can be different (in fact it's almost with every package that compiles), but the bottom line is the same.

If samba is stopped  ON EVERY DISTCC HOST then the problem is gone.

Does anyone knows how to fix it WITHOUT STOPPING SAMBA?

----------

## btned

hi all,

I have been trying to get distcc working on my laptop. I was wondering if it is ok to run distcc across subnets. I mean to say that will it work if my laptop is in one subnet and two other machines running distcc are in another subnet. This seems to work sometimes but many times I get the error 

```
failed to compile on localhost
```

 or an error regarding the time it waited for remote process to finish compilation. Is there a way to force the distcc server on the local machine to wait until a response comes from the remote server. 

Thank you

----------

## phranzee

```
USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND

distcc   24594 66.0  0.1  1912  768 ?        RN   16:00   7:45 /usr/bin/distccd --pid-file /var/run/distccd/distccd.pid -N 15 --user distcc
```

hi. i have some trouble with distcc. i start the deamon on two machines and sometimes it fails and one or more processes stay active and take 100% cpu. i even can't kill them. any ideas ? i already player with /etc/distcc/hosts, but it still doesn't work.

```
MAKEOPTS="-j5"                                                                                                   

FEATURES="distcc"                                                                                                

DISTCC_DIR="${PORTAGE_TMPDIR}/.distcc"
```

// distcc v. 2.16 on both

/var/log/syslog:

```
Jul 13 16:24:12 dustpuppy distccd[15943]: (dcc_setup_daemon_path) daemon's PATH is /usr/i686-pc-linux-gnu/gcc-bin/3.3:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/sbin:/usr/lib/distcc/bin:/sbin:/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/scripts:/etc/init.d

Jul 13 16:24:12 dustpuppy distccd[15943]: (dcc_should_be_inetd) stdin is a tty; assuming --daemon mode

Jul 13 16:24:12 dustpuppy distccd[15943]: (dcc_listen_by_addr) ERROR: bind of 0.0.0.0:3632 failed: Address already in use

Jul 13 16:24:12 dustpuppy distccd[15943]: (dcc_exit) exit: code 102; self: 0.000000 user 0.001999 sys; children: 0.000000 user 
```

----------

## roothorick

May I make a suggestion?

http://opendoorsoftware.com/cgi/http.pl?p=distccKNOPPIX

Smaller ISO size (faster download), smaller memory footprint, it starts distcc for you, and even tells you what IP address it managed to pull, if any, right off the bat!

----------

## Gentree

 *Obsidionblade wrote:*   

>  On the machine that you are running the compile from, likely your Gentoo box (which is actually the client - the rest are servers, go figure),

 

"Go figure" is just what you need to do. The term "server" does not  mean the biggest and fastest system on the LAN.

The machine doing the main compilation is a client to the services offered by the other machines running distccd as servers providing ... a service.

It may be that the client is the most lowly thing in the loop and that's why it needs help. 

In any case the terminology does make sense.

HTH  :Cool: 

----------

## iTux

 *axxackall wrote:*   

>   Is there any PAINLESS way to setup distcc to compile cross-platform? I've tried to mix distcc farms of x86 and ppc machines, but they don't like to compile the code of each other.  

 

You mean like using my guide and my experimental ebuilds?

http://www.sable.mcgill.ca/~dbelan2/crossdev/crossdev-powerpc-i686.html

 :Smile: 

iTux

----------

