how to do recursive "subsystem" make properly?

[Mark Galeck (CW)]

Some parts of this message have been removed. Learn more about Nabble's security policy.

Hello,  I see the following recommendation for a “subsystem” recursive make in the gnu make manual:

 

subsystem:

             $(MAKE) -C subdir

 

 

Well, I must not be understanding something, because this does not make (no pun) sense to me… 

 

The problem I have, is that the “subsystem” target as above, will always be remaked, if only to execute make recursively and find out in fact, that everything was up to date in that subdirectory.  Then of course, everything up above, depending on the subsystem target, will also have to be remaked.  In other words, even if you don’t change anything, make will still take time to remake some things.  Clearly this is against the spirit of make. 

 

 

Now, I know what to do to try to prevent that:  for example if the first target in the subdir make is (non-.PHONY):

 

foobar.o: foobar.c

 

then I could put in the upper directory, instead of the above target “subsystem”, the following:

 

subdir/foobar.o: subdir/foobar.c

        $(MAKE) –C subdir foobar.o

 

 

OK, but you see, if foobar.c depended on something else, then again  I would have to include that, prepended with “subdir/” in the upper directory.  And so forth.  Essentially, I would have to include the whole subdir makefile, in the top level makefile, with all targets prepended with “subdir/”.

 

Clearly not the right thing to do. 

 

What am I missing here?    

 

Thank you for any insight. 

 

Mark

 

_______________________________________________
Help-make mailing list
Help-make@...
http://lists.gnu.org/mailman/listinfo/help-make

[John Calcote-2]

Some parts of this message have been removed. Learn more about Nabble's security policy.

Hi Mark,

On 11/3/2009 1:54 PM, Mark Galeck (CW) wrote:

Hello,  I see the following recommendation for a “subsystem” recursive make in the gnu make manual:

 

subsystem:

             $(MAKE) -C subdir

 

 

Well, I must not be understanding something, because this does not make (no pun) sense to me… 

 

The problem I have, is that the “subsystem” target as above, will always be remaked, if only to execute make recursively and find out in fact, that everything was up to date in that subdirectory.  Then of course, everything up above, depending on the subsystem target, will also have to be remaked.  In other words, even if you don’t change anything, make will still take time to remake some things.  Clearly this is against the spirit of make. 

What you seem to be concerned about is that Make will always recurse down into subdir, even if nothing needs to be made in subdir. This is true but unavoidable, because make has to evaluate the state of subdir/Makefile in order to determine if anything has to be made in that directory. This is the nature of recursive make. If you don't want a recursive make, then just put all of your rules in the top-level Makefile.

That said, if nothing needs to be remade, the effect of recursive make is to simply run make several times - one for each sub-make - each of which do nothing more than determine that nothing needs to be done.

Regards,
John

_______________________________________________
Help-make mailing list
Help-make@...
http://lists.gnu.org/mailman/listinfo/help-make

[Mark Galeck (CW)]

Some parts of this message have been removed. Learn more about Nabble's security policy.

>That said, if nothing needs to be remade, the effect of recursive make is to simply run make several times - one for each sub-make - each of which do nothing more than determine that nothing needs to be done.

No, this is my point (I think).  The top-level make, will not just “determine that nothing needs to be done”.  It will run the subdir make, that one will indeed determine that nothing needs to be done.  But then, the top make, will have to remake all the targets, that depended on the “subsystem” target – because the subsystem target was “remade”. 

_______________________________________________
Help-make mailing list
Help-make@...
http://lists.gnu.org/mailman/listinfo/help-make

[Paul Smith-20]

On Tue, 2009-11-03 at 13:06 -0800, Mark Galeck (CW) wrote:

> >That said, if nothing needs to be remade, the effect of recursive
> make is to simply run make several times - one for each sub-make -
> each of which do nothing more than determine that nothing needs to be
> done.
>
> No, this is my point (I think).  The top-level make, will not just
> “determine that nothing needs to be done”.  It will run the subdir
> make, that one will indeed determine that nothing needs to be done.
> But then, the top make, will have to remake all the targets, that
> depended on the “subsystem” target – because the subsystem target was
> “remade”.  

This depends on how you do it.  There are multiple options.

One is to have your top-level makefile not build anything itself, but
merely control all the sub-makes.  Then it doesn't matter.  If you
really want the top-level to do something, you can have it recurse to
the same directory and invoke itself with a special rule.

Another is to use the product of the submake as the target instead of
something like "subdir"; for example if a submake builds a library
libfoo.a, then have that be the target and have the command to build
libfoo.a be the "$(MAKE) -C foosrc" or whatever.  The problem here is if
your submakes build >1 target, it gets hairy.

Another is to use order-only prerequisites and make the subdirectories
be order-only prereqs instead of normal prereqs, so that the targets
don't get rebuilt (see the GNU make manual for more info).

Another is to use sentinel files as the targets; some temporary file
that the submake would only touch if it actually made some change but
wouldn't touch if it didn't: if there's a real file, not a .PHONY
target, and its timestamp doesn't change after make runs the rule to
update it, then make will not treat it as having been modified for
up-to-date computations of targets that depend on it.

There are other possibilities as well, I'm sure.




_______________________________________________
Help-make mailing list
Help-make@...
http://lists.gnu.org/mailman/listinfo/help-make

[Mark Galeck (CW)]

Oh excellent Paul thank you very much!  I don't want to use the product of the submake as a target, because as explained in my first message, I think this would necessitate including all the targets on which that one depends, essentially all of the sub-makefile.  But all the other suggestions I really like, especially the sentinel.  Thank you so much!  

(and with the question I had asked a few days ago, that of interrupting make not working, to which there were no replys, I assume this means "that is the way it is, make does the best possible effort to remove the target".  So my workaround, was when building the targets, to use a temporary name, and then as the last operation, rename it to the correct target name.  )

Mark  

-----Original Message-----
From: Paul Smith [mailto:psmith@...]
Sent: Tuesday, November 03, 2009 2:09 PM
To: Mark Galeck (CW)
Cc: John Calcote; help-make@...
Subject: RE: how to do recursive "subsystem" make properly?

On Tue, 2009-11-03 at 13:06 -0800, Mark Galeck (CW) wrote:

> >That said, if nothing needs to be remade, the effect of recursive
> make is to simply run make several times - one for each sub-make -
> each of which do nothing more than determine that nothing needs to be
> done.
>
> No, this is my point (I think).  The top-level make, will not just
> "determine that nothing needs to be done".  It will run the subdir
> make, that one will indeed determine that nothing needs to be done.
> But then, the top make, will have to remake all the targets, that
> depended on the "subsystem" target - because the subsystem target was
> "remade".  

This depends on how you do it.  There are multiple options.

One is to have your top-level makefile not build anything itself, but
merely control all the sub-makes.  Then it doesn't matter.  If you
really want the top-level to do something, you can have it recurse to
the same directory and invoke itself with a special rule.

Another is to use the product of the submake as the target instead of
something like "subdir"; for example if a submake builds a library
libfoo.a, then have that be the target and have the command to build
libfoo.a be the "$(MAKE) -C foosrc" or whatever.  The problem here is if
your submakes build >1 target, it gets hairy.

Another is to use order-only prerequisites and make the subdirectories
be order-only prereqs instead of normal prereqs, so that the targets
don't get rebuilt (see the GNU make manual for more info).

Another is to use sentinel files as the targets; some temporary file
that the submake would only touch if it actually made some change but
wouldn't touch if it didn't: if there's a real file, not a .PHONY
target, and its timestamp doesn't change after make runs the rule to
update it, then make will not treat it as having been modified for
up-to-date computations of targets that depend on it.

There are other possibilities as well, I'm sure.




_______________________________________________
Help-make mailing list
Help-make@...
http://lists.gnu.org/mailman/listinfo/help-make

[Paul Smith-20]

Hi Mark; we prefer to use inline posting, not top posting, on the list.
Thanks.

On Tue, 2009-11-03 at 14:16 -0800, Mark Galeck (CW) wrote:
> Oh excellent Paul thank you very much!  I don't want to use the
> product of the submake as a target, because as explained in my first
> message, I think this would necessitate including all the targets on
> which that one depends, essentially all of the sub-makefile.

Not at all.  Suppose you have a makefile that builds libfoo.a in a
subdirectory foosrc.  Then in the upper-level makefile you would have
something like this:

        prog: foosrc/libfoo.a
                ...do something...

        foosrc/libfoo.a: FORCE
                $(MAKE) -C foosrc

        FORCE: ;

In this model, libfoo.a will always be considered out of date (because
of the FORCE prerequisite--just be sure you don't create a file "FORCE"
in your directory), so the command to build it will be invoked.  The
command is the submake.

If the submake actually updates libfoo.a, then "prog" will be rebuilt.
If the submake does not change libfoo.a, then "prog" will not be
considered out of date.

As I said, the problem here is that if "foosrc" builds more than one
target you have issues; you don't want both targets to do this or it
will run the make twice (!)

> But all the other suggestions I really like, especially the
> sentinel.  

Actually the sentinel is, IMO, an annoying implementation.  Managing all
those sentinel files is a PITA.  If you're happy to require GNU make,
I'd probably suggest going with order-only prerequisites (or else going
all the way to non-recursive builds) as the most robust solution.

> (and with the question I had asked a few days ago, that of
> interrupting make not working, to which there were no replys, I assume
> this means "that is the way it is, make does the best possible effort
> to remove the target".  So my workaround, was when building the
> targets, to use a temporary name, and then as the last operation,
> rename it to the correct target name.  )

The problem is your issue, as best as I recall, is Windows-specific.
Most of the people reading this list are using make on POSIX systems,
and ^C on POSIX systems does the right thing (at least in the tests I've
conducted).  You might have more luck asking Windows-specific questions
on the make-w32@... mailing list instead: Windows-knowledgeable
folks tend to hang out there.



_______________________________________________
Help-make mailing list
Help-make@...
http://lists.gnu.org/mailman/listinfo/help-make

[Stephan Beal-2]

2009/11/3 Paul Smith <psmith@...>

Another is to use order-only prerequisites and make the subdirectories

be order-only prereqs instead of normal prereqs, so that the targets
don't get rebuilt (see the GNU make manual for more info).

Out of curiosity: doesn't this also assist in making subdir rules friendly with parallel builds? i _seem_ to remember using order-only deps to ensure that certain subdirs are only built after prerequisite subdirs, even in the face of "make -j2".

Or am i mistaken on that point?

--
----- stephan beal
http://wanderinghorse.net/home/stephan/

_______________________________________________
Help-make mailing list
Help-make@...
http://lists.gnu.org/mailman/listinfo/help-make

[Mark Galeck (CW)]

>Actually the sentinel is, IMO, an annoying implementation.  Managing all
those sentinel files is a PITA.  If you're happy to require GNU make,
I'd probably suggest going with order-only prerequisites (or else going
all the way to non-recursive builds) as the most robust solution.


OK, after trying the sentinel file, I can't get it work:

top makefile is


foobar: subdir/sentinel
                echo making foobar
                touch foobar
subdir/sentinel:
                make -C subdir



subdir makefile is

all:
# touch sentinel



Well, what am I doing wrong - because, with this foobar always gets rebuilt.  Probably because make attempted to remake subdir/sentinel as "target", and even though that file did not get updated, make remembers that it remade the target.



Also,  order-only don't work for me either:  

Top makefile is


foobar: | subsystem
                echo making foobar
                touch foobar
subsystem:
                make -C subdir


and subdir makefile is

all: subfile
subfile:
                touch subfile

This time, foobar never gets rebuilt.  

-------------------------

All I need is for this example above, to work.  That is, if I have the "touch subfile" line, then make foobar, rebuilds foobar.  If I comment out the "touch subfile", then make foobar, does not rebuild foobar.  

I am sorry I am such a poor student, I just don't get it from your suggestions, how to do this.  

Mark






> (and with the question I had asked a few days ago, that of
> interrupting make not working, to which there were no replys, I assume
> this means "that is the way it is, make does the best possible effort
> to remove the target".  So my workaround, was when building the
> targets, to use a temporary name, and then as the last operation,
> rename it to the correct target name.  )

The problem is your issue, as best as I recall, is Windows-specific.
Most of the people reading this list are using make on POSIX systems,
and ^C on POSIX systems does the right thing (at least in the tests I've
conducted).  You might have more luck asking Windows-specific questions
on the make-w32@... mailing list instead: Windows-knowledgeable
folks tend to hang out there.



_______________________________________________
Help-make mailing list
Help-make@...
http://lists.gnu.org/mailman/listinfo/help-make

[Mark Galeck (CW)]

>All I need is for this example above, to work.  

OOPS sorry I made a mistake quoting that example.  The bottom makefile is wrong.  Let me try again:

Top makefile is


foobar: | subsystem
                echo making foobar
                touch foobar
subsystem:
                make -C subdir


and subdir makefile is

foobaro:  foobarc
                touch foobaro
                echo making foobaro


Now all I want, is if I touch foobarc, then foobar gets remade.  Then if I don't touch foobarc, foobar does not get remade.  

But right now, with the above, foobar never gets remade.  


_______________________________________________
Help-make mailing list
Help-make@...
http://lists.gnu.org/mailman/listinfo/help-make

[Paul Smith-20]

On Tue, 2009-11-03 at 15:32 -0800, Mark Galeck (CW) wrote:

> top makefile is
>
> foobar: subdir/sentinel
> echo making foobar
> touch foobar
> subdir/sentinel:
> make -C subdir
>
> subdir makefile is
>
> all:
> # touch sentinel

Well, since you have the touch commented out it will never do anything.
Also, be sure to ALWAYS use the variable $(MAKE), never just "make",
when running sub-makes

> Well, what am I doing wrong - because, with this foobar always gets
> rebuilt.  Probably because make attempted to remake subdir/sentinel as
> "target", and even though that file did not get updated, make
> remembers that it remade the target.

The sentinel file has to be created: it has to exist.  A non-existent
file is always considered out of date.  If it exists but then is not
updated, THEN it is not considered out of date.

> Also,  order-only don't work for me either:  
>
> Top makefile is
>
> foobar: | subsystem
> echo making foobar
> touch foobar
> subsystem:
> make -C subdir

Here foobar has to depend on the thing that is getting built
(subsystem/subfile), so the foobar knows when it's out of date.

You want this:

        foobar: subsystem/subfile
                echo making foobar
                touch $@

        subdsystem/subfile: | subsystem

        .PHONY: subsystem
        subsystem:
                $(MAKE) -C $@

Then keep your subsystem makefile the same:

> all: subfile
> subfile:
> touch subfile
>
> This time, foobar never gets rebuilt.  




_______________________________________________
Help-make mailing list
Help-make@...
http://lists.gnu.org/mailman/listinfo/help-make

[Mark Galeck (CW)]

>The sentinel file has to be created: it has to exist.  A non-existent
file is always considered out of date.  If it exists but then is not
updated, THEN it is not considered out of date.


Yes - I did have it created, and still it did not work, just like I explained.  


So, in my last message (before this one) I actually posted a short example of what I am trying to accomplish - so that it is precise.  That example can be solved with the use of FORCE target as Paul described earlier.  But of course, I do have more targets than one in the submakefile.  


So then, I finally got it to work by combining two of Pauls suggestions, the sentinel and the FORCE:

top makefile is


foobar: subdir/sentinel
                echo making foobar
                touch foobar
subdir/sentinel: FORCE
                $(MAKE) -C subdir
FORCE: ;


bottom makefile is

.PHONY: all
all: foobaro foobazo
sentinel: all

foobaro:  foobarc
                touch foobaro
                echo making foobaro
                touch sentinel
               
foobazo:  foobazc
                touch foobazo
                echo making foobazo
                touch sentinel


Now, if I touch foobarc or foobarz, then and only then foobar gets rebuilt!  Great, finally I got it!  


Except...  there is still one little problem.  When foobar does not get rebuilt, we would like some kind of message to the user, indicating that fact, such as "nothing to be done for foobar". And that message has to be after the subdir messages.  How in the world do I do this??

Mark




_______________________________________________
Help-make mailing list
Help-make@...
http://lists.gnu.org/mailman/listinfo/help-make

[Mark Galeck (CW)]

>we would like some kind of message to the user, indicating that fact, such as "nothing to be done for foobar". And that message has to be after the subdir messages.  

Without that message, the user sees this:

D:\tmp>make foobar
make -C subdir
make[1]: Entering directory `D:/tmp/subdir'
make[1]: Nothing to be done for `all'.
make[1]: Leaving directory `D:/tmp/subdir'

which is not a good thing, because they think, we attempted to go into subdirectory, great, but, nothing was attempted in the current directory...  




_______________________________________________
Help-make mailing list
Help-make@...
http://lists.gnu.org/mailman/listinfo/help-make

[Bart Robinson-2]

On Tue, Nov 3, 2009 at 4:16 PM, Mark Galeck (CW) <mgaleck@...> wrote:

>
> So then, I finally got it to work by combining two of Pauls suggestions,
> the sentinel and the FORCE:
>
> top makefile is
>
> foobar: subdir/sentinel
>                echo making foobar
>                touch foobar
> subdir/sentinel: FORCE
>                $(MAKE) -C subdir
> FORCE: ;
>
> bottom makefile is
>
> .PHONY: all
> all: foobaro foobazo
> sentinel: all
>
> foobaro:  foobarc
>                touch foobaro
>                echo making foobaro
>                touch sentinel
>
> foobazo:  foobazc
>                touch foobazo
>                echo making foobazo
>                touch sentinel
>
>
> Now, if I touch foobarc or foobarz, then and only then foobar gets rebuilt!
> Great, finally I got it!

This sentinel stuff seems a little gross and would get in the way of
$^ use, here is an alternate way to achieve what I think you're aiming
for, w/o using the FORCE/sentinel stuff:

-- makefile --
prog: main.o lib/libfoo.a lib/libbar.a
        cc -o $@ $^

%.o: %.c
        cc -c -o $@ $<

# empty commands are magic
lib/libfoo.a: build-libs ;
lib/libbar.a: build-libs ;

.PHONY: build-libs
build-libs:
        $(MAKE) -C lib

clean:
        rm -f *.o prog lib/*.o lib/*.a

-- lib/makefile --
.PHONY: all
all: libfoo.a libbar.a

%.o: %.c
        cc -c -o $@ $<

# have the .c files spring into existence for this test
%.c:
        touch $@

libfoo.a: foo.o
        ar rcs $@ $^
libbar.a: bar.o
        ar rcs $@ $^
-----------------------

The empty commands for the libfoo.a rules are a little magic.  Without
them "make" won't check the mtime of the lib since there are no
commands that could have changed it, and the "prog" target won't get
relinked.  (The previously mentioned order-only suggestions have this
same problem.)

-- bart


_______________________________________________
Help-make mailing list
Help-make@...
http://lists.gnu.org/mailman/listinfo/help-make

[Christophe LYON]

Hi Paul,

> prog: foosrc/libfoo.a
> ...do something...
>
> foosrc/libfoo.a: FORCE
> $(MAKE) -C foosrc
>
> FORCE: ;
>
> In this model, libfoo.a will always be considered out of date (because
> of the FORCE prerequisite--just be sure you don't create a file "FORCE"
> in your directory), so the command to build it will be invoked.  The
> command is the submake.
>
> If the submake actually updates libfoo.a, then "prog" will be rebuilt.
> If the submake does not change libfoo.a, then "prog" will not be
> considered out of date.
>

Sorry if the answer is obvious, but I don't understand why "prog" isn't
simply always rebuilt, because libfooa is always updated. Is it because
the command is $(MAKE)?

(Replacing $(MAKE) -C foosrc by touch $@ actually implies that prog is
always updated)

It tried to find something about this in the documentation, but I failed.

Thanks

Christophe.


_______________________________________________
Help-make mailing list
Help-make@...
http://lists.gnu.org/mailman/listinfo/help-make

[Mark Galeck (CW)]

>Hi Paul,

> prog: foosrc/libfoo.a
> ...do something...
>
> foosrc/libfoo.a: FORCE
> $(MAKE) -C foosrc
>
> FORCE: ;

>Sorry if the answer is obvious, but I don't understand why "prog" isn't
>simply always rebuilt, because libfooa is always updated. Is it because
>the command is $(MAKE)?


Hi Christophe, I am not Paul, I am the one from which this thread originally generated, and this method works for me.  

In fact I should perhaps say, "I am no Paul" :)  but let me comment on why I think it works.  


To decide whether to execute commands for a target, make considers whether any prerequisites are newer than the target.  When does it do that?  Only _after_ they themselves are processed, not before.  I am not sure whether this is mentioned in the manual, but of course it would not make sense to do it before.  

Mark




_______________________________________________
Help-make mailing list
Help-make@...
http://lists.gnu.org/mailman/listinfo/help-make

[Paul Smith-20]

On Mon, 2009-11-09 at 10:35 +0100, Christophe LYON wrote:
> > If the submake actually updates libfoo.a, then "prog" will be rebuilt.
> > If the submake does not change libfoo.a, then "prog" will not be
> > considered out of date.
> >
>
> Sorry if the answer is obvious, but I don't understand why "prog"
> isn't simply always rebuilt, because libfooa is always updated. Is it
> because the command is $(MAKE)?

Because, libfoo.a is NOT always updated, if by "updated" you mean its
time last modified is changed--which is what make cares about.

Make doesn't assume that any time it runs a command, the target for that
command has automatically been updated.  It actually looks at the mod
time for the prerequisites AFTER the commands have run.  If it's newer
than the target, the target is rebuilt.  If not, not.

The "Rule Syntax" section says:

           The criterion for being out of date is specified in terms of the
        PREREQUISITES, which consist of file names separated by spaces.
        (Wildcards and archive members (*note Archives::) are allowed here too.)
        A target is out of date if it does not exist or if it is older than any
        of the prerequisites (by comparison of last-modification times).

This is to be taken quite literally.

When you run the recursive make, it will only change the target
(libfoo.a) if it's out of date and needs to be changed.  If not, then it
won't be touched, and its mod time is left unchanged.



_______________________________________________
Help-make mailing list
Help-make@...
http://lists.gnu.org/mailman/listinfo/help-make

[Christophe LYON]

>
> Because, libfoo.a is NOT always updated, if by "updated" you mean its
> time last modified is changed--which is what make cares about.
>

I thought there was also something about the fact that some commands are
actually executed, or not (ie if some commands are executed, the target
is flagged as updated whether or not it's timestamp has changed). Is
this true?


I have another related question: consider
foo: foo.b
        touch $@

foo.b: foo.d

foo.d:
        touch $@

The first 'make' execution will:
$ make
touch foo.d
touch foo

but the subsequent ones will still:
$ make
touch foo

If I change the 1st rule to "foo: foo.d", then:
$ make
make: `foo' is up to date.

I thought the last change was just a shortcut for the initial sample,
with a useless rule removed. What is different?

Thanks,

Christophe.


_______________________________________________
Help-make mailing list
Help-make@...
http://lists.gnu.org/mailman/listinfo/help-make

[Bart Robinson-2]

On Tue, Nov 10, 2009 at 9:16 AM, Christophe LYON <christophe.lyon@...> wrote:

> I have another related question: consider
> foo: foo.b
> touch $@
>
> foo.b: foo.d
>
> foo.d:
> touch $@
>
> The first 'make' execution will:
> $ make
> touch foo.d
> touch foo
>
> but the subsequent ones will still:
> $ make
> touch foo
>
> If I change the 1st rule to "foo: foo.d", then:
> $ make
> make: `foo' is up to date.
>
> I thought the last change was just a shortcut for the initial sample, with a
> useless rule removed. What is different?

Hi Christophe,
This is because foo.b is never created.  A target (i.e. foo) is always
out of date with respect to a non-existent prerequisite.  You will get the
same behavior with

        foo: foo.b
                touch $@
        foo.b:

or, the more familiar:

        foo: FORCE
                touch $@
        FORCE:

If you wanted the behavior you expect, you can mark foo.b as secondary
in your original example:
        .SECONDARY: foo.b

Try make -r -d with and without that change to see how it is different.

-- bart


_______________________________________________
Help-make mailing list
Help-make@...
http://lists.gnu.org/mailman/listinfo/help-make