# Compiling kernels with distcc

## Kream

Compiling kernels with distcc

I test a number of kernels including vanillla, mandrake, lolo-sources, linux-beta and, of course, gentoo-sources so I find myself compiling the kernel lots of times. Even if you use distcc to emerge packages like I do, kernel compilations aren't distributed because the kernel make doesn't use /etc/make.conf. This is how I use distcc to radically speed up my compile times:

 Overview 

Set up distcc

Emerge distcc on the computers

Start distccd on the slave systems

Configure the kernel

Edit Makefile

make the kernel

Set up distcc

Set up distcc on all computers that you want to use, including the host computer. To accomplish this, a simple emerge command suffices:

```
 emerge -u distcc
```

Once done, make sure you start the distccd daemon on all the computers that you want compilation to be distributed to. This is done by running this command on all of them: 

```
/etc/init.d/distccd start
```

Configure the kernel

Change to /usr/src/linux and configure the kernel using make menuconfig or make xconfig (does anyone still use make config ????   :Rolling Eyes:  ). Save your config and and do a make dep.

Edit the Makefile

Fire up your favourite editor and load /usr/src/linux/Makefile. Find this section: 

```
#

# Include the make variables (CC, etc...)

#

```

And change "gcc" in the CC line to 

```

CC              = $(CROSS_COMPILE)distcc

```

Append this line to the section:

```

DISTCC_HOSTS=slave1 slave2 slave3 ... slaveN

```

You can enter the hostnames of the computers if you're sure that they resolve. If they don't, you can use their IP addresses. It's best to enter the computers in descending order of CPU power availability.

If you want the local machine to participate in the compile, remember to start distccd on it as well and add localhost to the list of DISTCC_HOSTS. I don't recommend this, though, as the local machine will already be doing the preprocessing and co-ordinating.

Add DISTCC_HOSTS to the line which says 

```
export  VERSION PATCHLEVEL SUBLEVEL EXTRAVERSION KERNELRELEASE ARCH \
```

 and save and exit.

Make the Kernel

When making the kernel, do a 

```
 make -jQ bzImage 
```

 where Q is the number of parallel makes that you want to run. I recommend that Q be the number of slaves you have + 2. Examine the terminal's output to make sure that distcc is being called instead of gcc. If you're a gkrellm2 user like I am, you'll notice lots of data traffic outbound from your machine (preprocessed code) and a small amount of inbound traffic (compiled code).

I found my kernel being built *much* faster with distcc.

Aniruddha Shankar

Bangalore, India.

Vipassana Works.

----------

## kyron

To add to this wonderful document of yours, here is a script which I use each time I compile a kernel....just to make sue I don't forget anything  :Razz: 

```
make dep && make clean

make -j6 bzImage

make -j6 modules

make modules_install

mount /boot/

#keep a date stamped copy ;-P

cp /boot/bzImage /boot/bzImage.`date +%F_%k-%M`

cp arch/i386/boot/bzImage /boot/bzImage

umount /boot/

#Making sure alsa-driver 

export ALSA_CARDS=ens1370

emerge alsa-driver nvidia-kernel nvidia-glx

depmod -a

update-modules
```

----------

## kyron

One other thing I forgot, I also added ccache to the following line:

```
 CC              = $(CROSS_COMPILE)ccache distcc
```

----------

## matroskin

 *Kream wrote:*   

>  does anyone still use make config ????  
> 
> 

  yes, i do. make config and make oldconfig are pretty good for configuring kernels throu scripts.

----------

## kurifu

For anyone who wishes to use distcc and genkernel to make their kernel, all you have to do is follow your normal distcc setup (emerge -v distcc, visit the distcc howto link provided at the end of the emerge), follow the instruction at the top of this thread to prepare your kernel make file and use the following command with genkernel:

```
genkernel all --menuconfig --makeopts="-j5"
```

Obviously you can make a few changes to the genkernel command line, depending on how you wish to build the kernel. The important command is --makeopts="-j5" which actually sets up the number of parallel makes at once, needed to push some of the processes to your host.

----------

## LordVan

Thanks for the well written instructions  :Smile:  I was looking for just this (my laptop is too slow to compile kernel in reasonable time -- and also tends to overheat  :Sad:   )

----------

## Yamakuzure

 *Kream wrote:*   

> And change "gcc" in the CC line to 
> 
> ```
> 
> CC              = $(CROSS_COMPILE)distcc
> ...

 Hi everybody!

If you came here, because you wanted to have distcc to help to compile a kernel on your Raspberry Pi 3, like me, then please notice, that distcc will fail when used like this, but works exceptionally well with:

```
# Make variables (CC, etc...)

CC              = $(CROSS_COMPILE)armv7a-hardfloat-linux-gnueabi-gcc
```

You should have set up your system like described in the Raspberry Pi/Cross building wiki entry.

----------

## Yord

 *Yamakuzure wrote:*   

>  *Kream wrote:*   And change "gcc" in the CC line to 
> 
> ```
> 
> CC              = $(CROSS_COMPILE)distcc
> ...

 

Hi, 

I came here for the RPI 3, but after much searching (albeit above didn't work for me, I think my distcc must be messed up somehow) I've found a simpler way!    :Smile: 

e.g. make CC="distcc" bzImage

I don't know if you need to add the makeopts -- isn't that sourced through /etc/portage/make.conf?

----------

## NeddySeagoon

Yord,

Kernel builds do not use /etc/portage/make.conf

You need to add -jN to your command, at least, or make will only run one concurrent job, even though it will use distcc.

For readers wanting kernels for arm and the like, cross compiling the kernel is a good introduction co the minefield that can be cross compiling.

The kernel is well behaved under cross compiling.

Do take care with 

```
make modules install
```

though.  Its a really bad idea to install arm kernel modules on your x86_64 build host.

It might be harmless, it might not.

----------

## Yord

 *NeddySeagoon wrote:*   

> Yord,
> 
> Kernel builds do not use /etc/portage/make.conf
> 
> You need to add -jN to your command, at least, or make will only run one concurrent job, even though it will use distcc.
> ...

 

Ah, thanks. I actually have problems with compiling the way I did - distcc won't recognise the slave and it distributes locally instead. 

Just to clarify, I'm trying to compile a kernel from the rpi, using distcc. There wouldn't be a problem with cluttering the build host in this way would there?

----------

## NeddySeagoon

Yord,

That's fine.  You need to ensure that the Pi and helpers all have the same versions of gcc.

distcc will still be slow.  The Pi has all of its IO on a single 400Mbit/sey USB2 root hub.

The Pi3 is slightly different as its WiFi is on a 50MHz 4 bit wide SDIO. That gives a 25MB/sec transfer rate, if it can run the WiFi crypto fast enough.

----------

## Yord

 *NeddySeagoon wrote:*   

> Yord,
> 
> That's fine.  You need to ensure that the Pi and helpers all have the same versions of gcc.
> 
> distcc will still be slow.  The Pi has all of its IO on a single 400Mbit/sey USB2 root hub.
> ...

 

Ah, Thanks!   :Very Happy: 

I'm still having problems, my rpi conf looks like this:

```
# --- /etc/distcc/hosts -----------------------

# See the "Hosts Specification" section of

# "man distcc" for the format of this file.

#

# By default, just test that it works in loopback mode.

192.168.1.90/8                                                                     
```

```
# /etc/conf.d/distccd: config file for /etc/init.d/distccd

DISTCCD_OPTS=""

# this is the distccd executable

DISTCCD_EXEC="/usr/bin/distccd"

# this is where distccd will store its pid file

DISTCCD_PIDFILE="/var/run/distccd/distccd.pid"

# set this option to run distccd with extra parameters

# Default port is 3632.  For most people the default is okay.

DISTCCD_OPTS="${DISTCCD_OPTS} --port 3632"

# Logging

# You can change some logging options here:

# --log-file FILE

# --log-level LEVEL  [critical,error,warning, notice, info, debug]

#

# Leaving --log-file blank will log to syslog

# example: --log-file /dev/null --log-level warning

# example: --log-level critical

DISTCCD_OPTS="${DISTCCD_OPTS} --log-level critical"

# SECURITY NOTICE:

# It is HIGHLY recommended that you use the --listen option

# for increased security. You can specify an IP to permit connections

# from or a CIDR mask

# --listen accepts only a single IP

# --allow is now mandatory as of distcc-2.18.

# example:  --allow 192.168.0.0/24

# example:  --allow 192.168.0.5 --allow 192.168.0.150

# example:  --listen 192.168.0.2

DISTCCD_OPTS="${DISTCCD_OPTS} --allow 192.168.1.0/24"

#DISTCCD_OPTS="${DISTCCD_OPTS} --listen 192.168.0.2"

# set this for niceness

# Default is 15

DISTCCD_OPTS="${DISTCCD_OPTS} -N 15"
```

/usr/lib/distcc/bin looks like this:

```
lrwxrwxrwx 1 root root 15 Aug 20 18:31 armv7a-hardfloat-linux-gnueabi-c++ -> /usr/bin/distcc

lrwxrwxrwx 1 root root 15 Aug 20 18:31 armv7a-hardfloat-linux-gnueabi-c++-4.9.3 -> /usr/bin/distcc

lrwxrwxrwx 1 root root 15 Aug 20 18:31 armv7a-hardfloat-linux-gnueabi-g++ -> /usr/bin/distcc

lrwxrwxrwx 1 root root 15 Aug 20 18:31 armv7a-hardfloat-linux-gnueabi-g++-4.9.3 -> /usr/bin/distcc

lrwxrwxrwx 1 root root 15 Aug 20 18:31 armv7a-hardfloat-linux-gnueabi-gcc -> /usr/bin/distcc

lrwxrwxrwx 1 root root 15 Aug 20 18:31 armv7a-hardfloat-linux-gnueabi-gcc-4.9.3 -> /usr/bin/distcc

-rwxr-xr-x 1 root root 85 Aug 20 18:37 armv7a-hardfloat-linux-gnueabi-wrapper

lrwxrwxrwx 1 root root 38 Aug 20 18:37 c++ -> armv7a-hardfloat-linux-gnueabi-wrapper

lrwxrwxrwx 1 root root 38 Aug 20 18:37 cc -> armv7a-hardfloat-linux-gnueabi-wrapper

lrwxrwxrwx 1 root root 38 Aug 20 18:37 g++ -> armv7a-hardfloat-linux-gnueabi-wrapper

lrwxrwxrwx 1 root root 38 Aug 20 18:37 gcc -> armv7a-hardfloat-linux-gnueabi-wrapper
```

and on the PC doing the loading:

```
# --- /etc/distcc/hosts -----------------------

# See the "Hosts Specification" section of

# "man distcc" for the format of this file.

#

# By default, just test that it works in loopback mode.

# 127.0.0.1                             
```

```
# /etc/conf.d/distccd: config file for /etc/init.d/distccd

DISTCCD_OPTS=""

# this is the distccd executable

DISTCCD_EXEC="/usr/bin/distccd"

# this is where distccd will store its pid file

DISTCCD_PIDFILE="/var/run/distccd/distccd.pid"

# set this option to run distccd with extra parameters

# Default port is 3632.  For most people the default is okay.

DISTCCD_OPTS="${DISTCCD_OPTS} --port 3632"

# Logging

# You can change some logging options here:

# --log-file FILE

# --log-level LEVEL  [critical,error,warning, notice, info, debug]

#

# Leaving --log-file blank will log to syslog

# example: --log-file /dev/null --log-level warning

# example: --log-level critical

DISTCCD_OPTS="${DISTCCD_OPTS} --log-level critical"

# SECURITY NOTICE:

# It is HIGHLY recommended that you use the --listen option

# for increased security. You can specify an IP to permit connections

# from or a CIDR mask

# --listen accepts only a single IP

# --allow is now mandatory as of distcc-2.18.

# example:  --allow 192.168.0.0/24

# example:  --allow 192.168.0.5 --allow 192.168.0.150

# example:  --listen 192.168.0.2

DISTCCD_OPTS="${DISTCCD_OPTS} --allow 192.168.1.0/24"

#DISTCCD_OPTS="${DISTCCD_OPTS} --listen 192.168.0.2"

# set this for niceness

# Default is 15

DISTCCD_OPTS="${DISTCCD_OPTS} -N 15"
```

and again, /usr/lib/distcc/bin looks like this (shouldn't matter?):

```
lrwxrwxrwx 1 root root 15 Aug 21 14:34 armv7a-hardfloat-linux-gnueabi-c++ -> /usr/bin/distcc

lrwxrwxrwx 1 root root 15 Aug 21 14:34 armv7a-hardfloat-linux-gnueabi-c++-4.9.3 -> /usr/bin/distcc

lrwxrwxrwx 1 root root 15 Aug 21 14:34 armv7a-hardfloat-linux-gnueabi-g++ -> /usr/bin/distcc

lrwxrwxrwx 1 root root 15 Aug 21 14:34 armv7a-hardfloat-linux-gnueabi-g++-4.9.3 -> /usr/bin/distcc

lrwxrwxrwx 1 root root 15 Aug 21 14:34 armv7a-hardfloat-linux-gnueabi-gcc -> /usr/bin/distcc

lrwxrwxrwx 1 root root 15 Aug 21 14:34 x86_64-pc-linux-gnu-c++ -> /usr/bin/distcc

lrwxrwxrwx 1 root root 15 Aug 21 14:34 x86_64-pc-linux-gnu-c++-4.9.3 -> /usr/bin/distcc

lrwxrwxrwx 1 root root 15 Aug 21 14:34 x86_64-pc-linux-gnu-g++ -> /usr/bin/distcc

lrwxrwxrwx 1 root root 15 Aug 21 14:34 x86_64-pc-linux-gnu-g++-4.9.3 -> /usr/bin/distcc

lrwxrwxrwx 1 root root 15 Aug 21 14:34 x86_64-pc-linux-gnu-gcc -> /usr/bin/distcc

lrwxrwxrwx 1 root root 15 Aug 21 14:34 x86_64-pc-linux-gnu-gcc-4.9.3 -> /usr/bin/distcc

 
```

I've also rebuild the armv7a-hardfloat toolchain on my pc as a counter-measure. 

So, for some reason distcc works perfectly when I'm using emerge, but it doesn't when I'm trying to compile a kernel (on the rpi). Is this the correct way to make a kernel?

```

make ARCH=arm CC=distcc bcm2709_defconfig

make ARCH=arm CC=distcc  oldconfig

make ARCH=arm CC=distcc menuconfig

make ARCH=arm CC=distcc  -j8 bzImage modules

make ARCH=arm CC=distcc  -j8 modules_install

```

... and obviously copy the kernel img over.

EDIT: So, when I try compiling the kernel, I still get these errors:

```

wonghau linux # make ARCH=arm CC=distcc -j8 bzImage modules

scripts/kconfig/conf  --silentoldconfig Kconfig

  CHK     include/config/kernel.release

  CHK     include/generated/uapi/linux/version.h

  CHK     include/generated/utsrelease.h

make[1]: 'include/generated/mach-types.h' is up to date.

  CHK     include/generated/timeconst.h

  CHK     include/generated/bounds.h

  CHK     include/generated/asm-offsets.h

  CALL    scripts/checksyscalls.sh

  CHK     include/generated/compile.h

  CC      arch/arm/kernel/smp.o

  CC [M]  drivers/bcma/main.o

distcc[6332] ERROR: compile arch/arm/kernel/smp.c on 192.168.1.90/8 failed

distcc[6332] (dcc_build_somewhere) Warning: remote compilation of 'arch/arm/kernel/smp.c' failed, retrying locally

distcc[6332] Warning: failed to distribute arch/arm/kernel/smp.c to 192.168.1.90/8, running locally instead

  CC      block/cfq-iosched.o

distcc[6395] (dcc_build_somewhere) Warning: failed to distribute, running locally instead

  CC [M]  drivers/base/regmap/regmap-i2c.o

distcc[6401] (dcc_build_somewhere) Warning: failed to distribute, running locally instead

  CC [M]  mm/zsmalloc.o

distcc[6411] (dcc_build_somewhere) Warning: failed to distribute, running locally instead

  CC [M]  crypto/seqiv.o

distcc[6412] (dcc_build_somewhere) Warning: failed to distribute, running locally instead

  CC [M]  fs/9p/vfs_super.o

distcc[6414] (dcc_build_somewhere) Warning: failed to distribute, running locally instead

distcc[6368] ERROR: compile drivers/bcma/main.c on 192.168.1.90/8 failed

distcc[6368] (dcc_build_somewhere) Warning: remote compilation of 'drivers/bcma/main.c' failed, retrying locally

distcc[6368] Warning: failed to distribute drivers/bcma/main.c to 192.168.1.90/8, running locally instead

  CC [M]  drivers/base/regmap/regmap-spi.o

distcc[6453] (dcc_build_somewhere) Warning: failed to distribute, running locally instead

distcc[6332] (dcc_please_send_email_after_investigation) Warning: remote compilation of 'arch/arm/kernel/smp.c' failed, retried locally and got a different result.

  LD      arch/arm/kernel/built-in.o

  LD      kernel/trace/libftrace.o

  CC      kernel/trace/trace_events.o

  LD      drivers/block/built-in.o

distcc[6506] (dcc_build_somewhere) Warning: failed to distribute, running locally instead

  CC [M]  drivers/block/pktcdvd.o

distcc[6514] (dcc_build_somewhere) Warning: failed to distribute, running locally instead

  CC [M]  drivers/bcma/scan.o

distcc[6536] (dcc_build_somewhere) Warning: failed to distribute, running locally instead

distcc[6368] (dcc_please_send_email_after_investigation) Warning: remote compilation of 'drivers/bcma/main.c' failed, retried locally and got a different result.

  CC [M]  drivers/bcma/core.o

distcc[6559] (dcc_build_somewhere) Warning: failed to distribute, running locally instead

  CC      kernel/trace/trace_export.o

distcc[6569] (dcc_build_somewhere) Warning: failed to distribute, running locally instead

  CC [M]  fs/9p/vfs_inode.o

distcc[6583] (dcc_build_somewhere) Warning: failed to distribute, running locally instead

  CC [M]  crypto/echainiv.o

distcc[6596] (dcc_build_somewhere) Warning: failed to distribute, running locally instead

  CC [M]  crypto/crypto_user.o

distcc[6608] (dcc_build_somewhere) Warning: failed to distribute, running locally instead

  CC [M]  crypto/cmac.o

distcc[6617] (dcc_build_somewhere) Warning: failed to distribute, running locally instead

```

----------

## laineg

 *Kream wrote:*   

> 
> 
> ```
> 
> CC              = $(CROSS_COMPILE)distcc
> ...

 

That won't work for the for the cross-compiling scenario, because distcc isn't CHOST-prefixed

```

CC              = distcc $(CROSS_COMPILE)gcc

```

Tells distcc to execute gcc for the target, assuming the cross compiler is on your compile servers

And for ccache:

```

CC              = ccache distcc $(CROSS_COMPILE)gcc

```

----------

## ville.aakko

Hi,

Thanks for this guide.

I've just noticed that for some reason distccd parallelizes really poorly (despite using -j5) - only one thread is compiling on the slave. Perhaps there's just too much overhead on the master to distribute compilation effectively / at all? (localhost is not in /etc/distcc/hosts)

I'm compiling a kernel for amd k6-3 box, and obviously it would be quite slow on the box itself.

Also, indeed the original instructions doesn't work if the slave computer is not the same arch. I needed to use:

```
CC              = distcc $(CROSS_COMPILE)i586-pc-linux-gnu-gcc
```

It's a bit confusing, as one can not (or should not) set CROSS_COMPILE in the Makefile, as he/she is not cross-compiling in the Make process, i.e. on the target computer (amd k6-3 in my case) ? 

Maybe this should be added to the distcc cross compilation wiki, as Kernels are quite a large compile. AFAIK it is otherwise outdated, too (the wrapper script is not needed anymore?).

EDIT: The wiki is missing instructions to use "--jobs" on the slave configuration. Now that I added it there, it's starting to give more jobs to the slave  :Smile:  Still, quite some overhead and the master is not distributing processes as fast as it could - maybe 2-3 threads running on the slave (which could handle more, max jobs set to 6)

----------

## NightMonkey

Howdy. It seems that 'CC' has gone away. I see there's "HOSTCC' and 'HOSTCCX' in the Makefile. Anyone in the know on how best to enable Distcc kernel compilation these days? Thanks in advance.

----------

