[bug #27714] expansion of $(shell) in target forces serialization of targets

[Matt McCutchen-8]

URL:
  <http://savannah.gnu.org/bugs/?27714>

                 Summary: expansion of $(shell) in target forces
serialization of targets
                 Project: make
            Submitted by: vapier
            Submitted on: Fri 16 Oct 2009 07:50:48 AM GMT
                Severity: 3 - Normal
              Item Group: Bug
                  Status: None
                 Privacy: Public
             Assigned to: None
             Open/Closed: Open
         Discussion Lock: Any
       Component Version: 3.81
        Operating System: POSIX-Based
           Fixed Release: None
           Triage Status: None

    _______________________________________________________

Details:

i noticed a package recently that would build serially even when i gave it -j
(on a quad machine).  looking into it, the reason was that $(shell) was
appended to a compiler flag leading to it being expanded when the target was
run:
CFLAGS += $(shell .....)

forcing the $(shell) to be evaluated immediately allowed things to build in
parallel:
SOME_CHECK := $(shell .....)
CFLAGS += $(SOME_CHECK)

behavior is seen with make-3.80 and make-3.81

here is a simple example:
$ cat makefile
t = 0 a b c 1 d e f 2 g h i 3 j k l 4 m n o 5 p q r 6
all: $(t)
        @echo
$(t):
        @printf $@ $(shell :)

$ while make -f makefile -j ; do : ; done
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mno5pqr6

removing the $(shell :) allows for parallel evaluation:
0abc1de2gi3hjkl4mnoqr56p
0abc1de2ghi3jkl4mn5pqor6
0acde21gbhi3jkl4mno5pqr6
0abc1de2ghi3jkl4mn5qr6po
0abc1d2gehi3jkl4o5mpnqr6
0abc1de2ghi3jkl4mno5pq6r
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jl4n5pkqorm6
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mnpqor56




    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?27714>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/



_______________________________________________
Bug-make mailing list
Bug-make@...
http://lists.gnu.org/mailman/listinfo/bug-make

[Matt McCutchen-8]

Follow-up Comment #1, bug #27714 (project make):

My suspicion is that this is just an artifact of your test case.  In your
scenario, make must evaluate the $(shell ...) function (invoking the shell)
while it's expanding the command to be invoked.  What I think is happening
here is that your actual command is so fast that by the time make is ready to
invoke the next command (after it ran the $(shell :) for that target), the
previous command is already completed.  So, it looks like things are happening
serially even though make is doing its best to run them in parallel.

Please try this test again, but have the command you invoke take longer; for
example add a "sleep 1" or something after your printf (not inside the $(shell
...) function!), then print it again maybe.  You should see all the first
prints show up (running in parallel) then all the second prints.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?27714>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/



_______________________________________________
Bug-make mailing list
Bug-make@...
http://lists.gnu.org/mailman/listinfo/bug-make

[Matt McCutchen-8]

Follow-up Comment #2, bug #27714 (project make):

i didnt think it a race condition because the original test case had a fairly
lengthy $(shell) code and the targets always ran in the same order (and
obviously only one cpu at a time -- the other cpus stayed idle)

using sleep shows that it is indeed a make bug.  i see one target show up
every 10 seconds:
$ cat makefile
t = 0 a b c 1 d e f 2 g h i 3 j k l 4 m n o 5 p q r 6
all: $(t) ; @echo
$(t): ; @printf $@ $(shell sleep 10)

$ make -f makefile
<10 seconds>0<10 seconds>a<10 seconds>b............

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?27714>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/



_______________________________________________
Bug-make mailing list
Bug-make@...
http://lists.gnu.org/mailman/listinfo/bug-make

[Matt McCutchen-8]

Follow-up Comment #3, bug #27714 (project make):

Hi Mike; no, sorry, I should have given an explicit example.  Your new test
case is not correct and doesn't prove anything.  I explicitly said do NOT put
the sleep inside the $(shell).  It's important to understand that make itself
is single threaded (it doesn't use pthread() etc.).  Its parallelism is gained
by having the recipes it invokes be run in parallel.  But before it can invoke
the recipe it has to expand all the variables and functions.  That includes
$(shell ...) functions.  Variables and functions (including $(shell ...)
functions) are NOT expanded "in parallel"; they are expanded one at a time as
they are seen in the recipe.

After the entire recipe is expanded, THEN make invokes the command, then it
leaves the command running in the background while it locates the next
possible recipe to start, evaluates it, then runs it in the background, etc.

So, causing the recipe expansion to take a long time (which is what you're
doing by adding the sleep to the $(shell ...) function) actually _decreases_
the amount of parallelism.  Hopefully that's clear.

What you need to do to show the make is working correctly is increase the
time it takes the recipe itself to run: that is, add the sleep OUTSIDE the
$(shell ...).  Something like:


$(t): ; @printf $@ $(shell :); sleep 5; printf $@


You should see the recipes all run in parallel (the first printf's all run,
then a pause, then the second printfs all run).

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?27714>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/



_______________________________________________
Bug-make mailing list
Bug-make@...
http://lists.gnu.org/mailman/listinfo/bug-make

[Philip Guenther-2]

On Fri, Oct 16, 2009 at 4:34 AM, Paul D. Smith <INVALID.NOREPLY@...> wrote:
...
> But before it can invoke the recipe it has to expand all the variables and functions.

I had to ponder that requirement for a bit before it occurred to me
that doing otherwise would either make $(eval) pointless in recipes or
make it an incredible, royal pain to implement.  Ditto for $(error).


Philip Guenther


_______________________________________________
Bug-make mailing list
Bug-make@...
http://lists.gnu.org/mailman/listinfo/bug-make

[Matt McCutchen-8]

Follow-up Comment #4, bug #27714 (project make):

i understand what you're saying, but i think the behavior you describe as
"expected" is actually a bug.  deferred variable expansion for a target
shouldnt affect make's ability to run other targets.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?27714>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/



_______________________________________________
Bug-make mailing list
Bug-make@...
http://lists.gnu.org/mailman/listinfo/bug-make

[Matt McCutchen-8]

Follow-up Comment #5, bug #27714 (project make):

Hrm.  The only way this could be "fixed" would be to make the $(shell ...)
function a true make job, so that it takes a job token, can run in the
background asynchronously, and is basically handled as an anonymous recipe
line that is run before the "real" recipe line.

This would be almost impossible to do, because it would mean we'd have to
introduce an entirely new concept in make: the ability to have "deferred
variable expansion", where we say "please expand this string" and instead of
getting the expanded string back, we'd get back a response saying "I started
it, come back later".   Then later, sometime, we'd have to check to see if it
completed, and somehow restart our expansion at that time.  Allowing expansion
to happen asynchronously would mean completely rewriting a very significant
portion of make.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?27714>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/



_______________________________________________
Bug-make mailing list
Bug-make@...
http://lists.gnu.org/mailman/listinfo/bug-make

[Matt McCutchen-8]

Follow-up Comment #6, bug #27714 (project make):

i dont really know how GNU make is architected, but i dont see why make cant
fork the process to handle the target and then the subprocess does both
deferred expansion and running the commands listed for that target.

i'd also point out that other make implementations (such as BSD makes) handle
this the way i expect -- in parallel.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?27714>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/



_______________________________________________
Bug-make mailing list
Bug-make@...
http://lists.gnu.org/mailman/listinfo/bug-make

[Matt McCutchen-8]

Follow-up Comment #7, bug #27714 (project make):

I'm surprised you say you can do this with BSD make; I'm not aware that BSD
make has any feature like the $(shell ...) function, that can appear inside a
recipe.  What syntax do you use for this in BSD make?

We can't expand the recipe in a forked version of GNU make, because the
expansion of the recipe can have side-effects (for example, if an $(eval ...)
is expanded); if the expansion happens in a forked process then those side
effects are not available in the parent.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?27714>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/



_______________________________________________
Bug-make mailing list
Bug-make@...
http://lists.gnu.org/mailman/listinfo/bug-make

[Matt McCutchen-8]

Follow-up Comment #8, bug #27714 (project make):

# gmake
FOO=   $(echo bar)

# pmake (BSD)
FOO!=   echo bar

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?27714>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/



_______________________________________________
Bug-make mailing list
Bug-make@...
http://lists.gnu.org/mailman/listinfo/bug-make

[Matt McCutchen-8]

Follow-up Comment #9, bug #27714 (project make):

No, that's not equivalent.  First, your GNU make syntax is wrong; you mean
"FOO = $(shell echo bar)".  And second, as far as I can see from the pmake
documentation "FOO != echo bar" is actually equivalent to "FOO := $(shell echo
bar)", so it doesn't apply to the issue discussed in this bug (which is that
the expansion of the shell happens when the recipe is evaluated, not when the
makefile is read in).

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?27714>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/



_______________________________________________
Bug-make mailing list
Bug-make@...
http://lists.gnu.org/mailman/listinfo/bug-make