How to use distcc to preprocess and compile everything remotely only? - gcc

Question

Background:
I have a 128-core server which I would like to use as a build server.
I have a bunch of client machines which work with a not-so-new and not-so-powerful PC. (Can't upgrade! Not in my control.)
What I did:
I followed the distcc documentation.
And installed a virtual machine on the server with exactly the same compiler version, the same distcc version -- basically the same distribution, as on the client-machines.
After configuring the clients and the servers, I can remotely build. I can verify this using the distccmon-text tool. I can see on the server, there are 8 threads started by the distcc daemon and that are awaiting for build-jobs to come. This was good as a first step. You can see the output below to be sure.
Second Step: Since the client machines are dual-core machines while the server offers 128-cores, and not all clients will be compiling at the same time, I wanted to offload as much of the build as possible to the build-server.
Problems:
First Problem: distcc, no matter how I try to configure it, always tries to distribute the build-jobs equally on the client and the server. Even though my configuration file looks as shown below:
# --- /etc/distcc/hosts -----------------------
# See the "Hosts Specification" section of
# "man distcc" for the format of this file.
#
# By default, just test that it works in loopback mode.
# 127.0.0.1
172.24.26.208/8,cpp,lzo
localhost/0
Which as per the distcc documentation should give higher priority to the build-server and lower-priority to the localhost since it comes later in the list. Also, it should give 8 jobs to the build server and 0 jobs to the localhost. But no, that doesn't happen. Upon typing make -j8 what it tries to do is start 4 threads on localhost and 4 on remote. Not good. This you can see from the image below.
Second Problem: What you would also notice is that the pre-processing is being done on the localmachine. For this there is a tool that comes with distcc. It is called the "distcc-pump" or the pump mode and can be used like this.
time pump make CC="distcc gcc" CXX="distcc g++" -j8
Unfortunately, pump mode or not, the pre-processing happens to be happening all on the localhost, as you can see from the above image. Sad.
Note: At no point does the distcc program, with the configurations I listed here, throw any errors nor warnings, neither on the server nor on the clients.
Versions:
gcc 4.4.5
distcc 3.2rc1.2
(Before someone suggests - "upgrade software!", newer versions are most likely not possible for me. Anyways, this version of distcc offers the features that I need. Also, I can upgrade the server virtual-machine but then there would be compiler version mistmatch between clients and the server. The clients I cannot upgrade.)
Any suggestions, feedback on how to improve this setup/(fix the problems) are most welcome.

Answer 1

EDIT : these solutions do not work, I let the answer to avoid someone else to propose again them
Try by
removing the line concerning the localhost in /etc/distcc/hosts c.f. https://superuser.com/questions/568133/force-most-compilation-to-a-remote-host-with-distcc
or may be specifying 127.0.0.1 rather than localhost in /etc/distcc/hosts c.f. an other problem solved with that substitution in https://distcc.github.io/faq.html

Answer 2

distcc actually differentiates between remote and local CPUs. But contrary to your interpretation, in the hosts file the IP address 127.0.0.1 is considered as a remote CPU and a distccd server is expected to be running there. Any number of jobs you define in the hosts file is interpreted only for these server nodes.
According to the man page, "localhost" is interpreted specially. This is what seems to not work for you. An alternative syntax is --localslots=<int>. Have you tested this?
Additionally, distcc runs jobs on the local host (the one where you start the driver program). First, all linking is done there. Second, when you specify a certain parallelism with make -jN, all jobs exceeding the available number of remote jobs are run on your local host, too - in addition to the workload distribution part of distcc. The option --localslots limits these. The man page does not mention localhost explicitly here. And then there are those jobs, which fail on the server and are repeated locally.
For the given 128-core server I would use the number of cores in the hosts file and start only that number of compile jobs:
$ cat ~/.distcc/hosts
172.24.26.208/128,cpp,lzo
$ make -j 128
...
Then I would expect to not see any compile jobs on the local machine.
The man page has some more words regarding recommended job numbers. Search for the section(s) starting with distcc spreads the jobs across both local and remote CPUs.