More efficient multiple file compiles?

[Tom Holly]

I was looking into editing distcc code so that distributed compiles also become simultaneous. This would be nice because instead of one file after another being compiled on a random machine, distcc could distribute  multiple files on different machines at the same time. Is this possible or not worth looking into?

__
distcc mailing list            http://distcc.samba.org/
To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/distcc

On Friday 24 April 2009 06:18:06 am Tom Holly wrote:
> I was looking into editing distcc code so that distributed compiles also
> become simultaneous. This would be nice because instead of one file after
> another being compiled on a random machine, distcc could distribute
> multiple files on different machines at the same time. Is this possible or
> not worth looking into?

Look at the -j option of make.

--
*************************************************
Erik Zeek
zeekec@...
*************************************************
Against stupidity the very gods
Themselves contend in vain.
 - Johann Christoph Friedrich von Schiller (1801)
*************************************************
__
distcc mailing list            http://distcc.samba.org/
To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/distcc

[Benjamin R. Haskell-4]

On Fri, 24 Apr 2009, Erik Zeek wrote:

> On Friday 24 April 2009 06:18:06 am Tom Holly wrote:
> > I was looking into editing distcc code so that distributed compiles
> > also become simultaneous. This would be nice because instead of one
> > file after another being compiled on a random machine, distcc could
> > distribute multiple files on different machines at the same time. Is
> > this possible or not worth looking into?
>
> Look at the -j option of make.
>

Also the '-j' (--jobs) option of distcc, which might be useful in crafting
the '-j' value you use with make.

e.g. I use +zeroconf, and my set of available hosts changes frequently:

$ distcc --jobs
9

$ echo make -j$(distcc -j)
make -j9

# I only use Zsh, but I think $(( math )) stuff works similarly in bash
$ echo make -j$(( $(distcc -j) / 2 ))
make -j4

Best,
Ben
__
distcc mailing list            http://distcc.samba.org/
To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/distcc

[Tom Holly]

make -j12 CC=distcc <target> is not working for me. When exactly do I run make -j? I already have the distcc code built/installed and the pump server running.

__
distcc mailing list            http://distcc.samba.org/
To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/distcc

[Fergus Henderson-4]

The amount of parallelism that you get will depend on the details of how your Makefile is written.

Does your Makefile contain ".NOTPARALLEL"?  From the GNU Make manual:

     If `.NOTPARALLEL' is mentioned as a target, then this invocation of
     `make' will be run serially, even if the `-j' option is given.

Even if your Makefile doesn't contain .NOTPARALLEL, if it is written so that it only invokes distcc once, or if it has serial data dependencies between each distcc invocation, then running it with "-j" won't help.

On Mon, Apr 27, 2009 at 1:46 PM, Tom Holly <hollydt@...> wrote:

make -j12 CC=distcc <target> is not working for me. When exactly do I run make -j? I already have the distcc code built/installed and the pump server running.

 

--
Fergus Henderson <fergus@...>

__
distcc mailing list            http://distcc.samba.org/
To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/distcc

[Tom Holly]

I think mainly I just want to know when in the step by step procedure of building/installing/running distcc, does the make -j come into play?

__
distcc mailing list            http://distcc.samba.org/
To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/distcc

[Fergus Henderson-5]

On Mon, Apr 27, 2009 at 2:20 PM, Tom Holly <hollydt@...> wrote:

I think mainly I just want to know when in the step by step procedure of building/installing/running distcc, does the make -j come into play?

Use "make -j" (or "pump make -j") when running distcc, after you have already built and installed distcc.

--
Fergus Henderson <fergus@...>

__
distcc mailing list            http://distcc.samba.org/
To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/distcc

[Benjamin R. Haskell-4]

On Mon, 27 Apr 2009, Fergus Henderson wrote:

> On Mon, Apr 27, 2009 at 2:20 PM, Tom Holly <hollydt@...> wrote:
>       I think mainly I just want to know when in the step by step procedure of
>       building/installing/running distcc, does the make -j come into play?
>
>
> Use "make -j" (or "pump make -j") when running distcc, after you have
> already built and installed distcc.
>

How was 'make -j12 CC=distcc <target>' failing for you?  That should be
the right invocation, assuming you're not trying to do it before distcc is
installed.

e.g.:

$ cd /usr/src/linux
$ make -j4 CC=distcc
[...]
  CC      kernel/exec_domain.o
  CC      arch/x86/kernel/io_delay.o
  CC      mm/bootmem.o
  CC      arch/x86/kernel/rtc.o
  CC      arch/x86/kernel/trampoline.o
  CC      mm/filemap.o
  CC      kernel/panic.o
[...]

Notice how it's switching between the kernel/ arch/x86/kernel/ and mm/
directories.  That's because the CC steps are being run simultaneously, as
seen by distccmon-text (at a different point in time):

$ distccmon-text
 31766  Preprocess  buffer.c                    192.168.2.23[0]
 31755  Compile     ipcns_notifier.c            192.168.2.23[1]
 31774  Preprocess  clockevents.c               192.168.2.23[1]
 31781  Compile                                    localhost[2]

-- Ben
__
distcc mailing list            http://distcc.samba.org/
To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/distcc

[Tom Holly]

Here's the Makefile I'm using.


__
distcc mailing list            http://distcc.samba.org/
To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/distcc

[ibaker]

On Mon, 27 Apr 2009, Tom Holly wrote:

> I think mainly I just want to know when in the step by step procedure of
> building/installing/running distcc, does the make -j come into play?
>

The distcc man page has a step by step guide that details this.

Ian
__
distcc mailing list            http://distcc.samba.org/
To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/distcc

[Benjamin R. Haskell-4]

On Tue, 28 Apr 2009, Tom Holly wrote:

> Here's the Makefile I'm using.
>
> [...attachment was Makefile for distcc...]

I suppose I'll still take your word for it that distcc is already
installed.  (...and I'll assume you're just compiling distcc as a test.)

That said, distcc compiles fine for me with `make -j4 CC=distcc`.  
Running distccmon-text in another terminal shows several compilations
going at a time.

You still didn't specify what you meant by:

> make -j12 CC=distcc <target> is not working for me.

How is it "not working"?

Best,
Ben
__
distcc mailing list            http://distcc.samba.org/
To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/distcc

[Tom Holly]

Those assumptions are correct. I've had distcc built and installed for a while and it's working wonderfully. Now I'm trying to incorporate make -j and here is the problem:

I ran "pump make -j12 CC=distcc\ gcc all" in the directory where my Makefile and distcc object files reside. Then I did a make install the same way and that all  worked fine.

Now I want to test this on my own C files and that is what I cannot seem to get working because I don't notice the files being compiled simultaneously.

__
distcc mailing list            http://distcc.samba.org/
To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/distcc

[Benjamin R. Haskell-4]

On Tue, 28 Apr 2009, Tom Holly wrote:

> Those assumptions are correct. I've had distcc built and installed for a
> while and it's working wonderfully. Now I'm trying to incorporate make
> -j and here is the problem:
>
> I ran "pump make -j12 CC=distcc\ gcc all" in the directory where my
> Makefile and distcc object files reside. Then I did a make install the
> same way and that all  worked fine.
>
> Now I want to test this on my own C files and that is what I cannot seem
> to get working because I don't notice the files being compiled
> simultaneously.

I'd recommend using environment variables to tell distcc to output more
useful debugging information for your case.  Specifically, I'd look at
using DISTCC_VERBOSE, DISTCC_LOG, and maybe DISTCC_SAVE_TEMPS[1].

For more general 'parallel make' advice, a thread from
help-make@...[2] had some relevant discussion.  Specifically, one of
the suggestions[3] was to augment your targets with timestamp-generating
commands:

e.g.:

AT = @date +"$@: %s" ;

all: foo bar

foo:
        $(AT)sleep 5

bar:
        $(AT)sleep 10


But, maybe that could be expanded upon further, by creating a small
wrapper for distcc that outputs timing information (tested):

==== put this in your path as 'distcc-with-timing' ====
==== (at http://benizi.com/distcc-with-timing ) ====
#!/bin/zsh
_log=${DISTCC_TIMING_LOG=$DISTCC_DIR/timing-log}
_args=($argv)
function _tag () {
    printf "%s %s%s\n" `date +%s` $1 "${DISTCC_TIMING_VERBOSE+ $_args}" >> $_log
}
_tag START-JOB
distcc $argv
_tag FINISH-JOB
==== end of distcc-with-timing ====

$ make -j4 CC=distcc-with-timing
# then inspect $DISTCC_DIR/timing-log

$ DISTCC_TIMING_LOG=/tmp/put-timing-log-here DISTCC_TIMING_VERBOSE=1 make -j4 CC=distcc-with-timing
# then inspect /tmp/put-timing-log-here

If you find more than one START-JOB line in a row, it's running in
parallel.

Best,
Ben

[1] http://distcc.googlecode.com/svn/trunk/doc/web/man/distcc_1.html#TOC_20
[2] http://www.mail-archive.com/help-make@.../msg07297.html
[3] http://www.mail-archive.com/help-make@.../msg07304.html
[4] http://benizi.com/distcc-with-timing
__
distcc mailing list            http://distcc.samba.org/
To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/distcc