problems with node Utilities in Makefiles

[Stepan Kasal]

Hi,
  the node (standards)Utilities in Makefiles
was brought to my attention (by Karel Zak).

I noticed many inconsistencies between the standards and their most
prominent implementation, Autoconf + Automake:

1) we should add "awk" to the list of alowed utils, as current CVS
Autoconf generates configure scripts that use awk to perform the
file instantiation.  (Previously, this was done by sed.)

2) "make dist" in Automake-generated makefiles depends on "find and
chmod" to remove the distdir.

3) If you use the Automake option dist-bzip2, then "make dist" uses
bzip2, not gzip.  Likevise for Automake options dist-zip, dist-tarZ.
If there options are _meant_ to breach GCS, like, e.g. the "foreign"
option, the Automake manual shall note this.

4) Is it still adequate to say about "mkdir -p" that
        most systems don't support it.
What about
        some systems don't support it.
?

IMVHO all these issues should be resolved by changing the standard.
If you agree with that, I can prepare a patch.

Thanks,
        Stepan Kasal

[Ralf Wildenhues]

Hello Stepan, all,

* Stepan Kasal wrote on Thu, Aug 30, 2007 at 04:02:19PM CEST:
>
> I noticed many inconsistencies between the standards and their most
> prominent implementation, Autoconf + Automake:

Yep.  I keep forgetting, awk should be listed.

FWIW, Autoconf promises to be able to make do with dead-ancient Solaris
awk, just in case someone doubts that modern awk is not portable enough.

> 2) "make dist" in Automake-generated makefiles depends on "find and
> chmod" to remove the distdir.

find and chmod are listed in GCS, `Install Command Categories'.

> 3) If you use the Automake option dist-bzip2, then "make dist" uses
> bzip2, not gzip.  Likevise for Automake options dist-zip, dist-tarZ.
> If there options are _meant_ to breach GCS, like, e.g. the "foreign"
> option, the Automake manual shall note this.

I suppose this doesn't have any impact on GCS, no?  I don't think zip
should be added to GCS.  A patch to Automake seems more appropriate
IMVHO.

> 4) Is it still adequate to say about "mkdir -p" that
> most systems don't support it.
> What about
> some systems don't support it.
> ?

What's the difference wrt. GCS?  Except for GNU Coreutils and relatively
recent BSD implementations, one still has to think about concurrency
issues and what happens together with -m, or existing symlinks, see the
Autoconf manual.  But going into detail here seems like the wrong thing
to do, too.  Anyway, I don't have a strong opinion here (kind of a bike
shed color issue).

> IMVHO all these issues should be resolved by changing the standard.
> If you agree with that, I can prepare a patch.

FWIW, I'm not a person to agree here or not, but I suppose you could
prepare a patch to avoid needing two separate agreements (one on the
idea and one on the patch  ;-)

Cheers, and yes, this is all very much IMHO,
Ralf

[Stepan Kasal-2]

Hello,

On Thu, Aug 30, 2007 at 08:49:48PM +0200, Ralf Wildenhues wrote:
> I suppose you could prepare a patch to avoid needing two separate
> agreements (one on the idea and one on the patch  ;-)

well, I'm far from understanding what the patch should actually
contain.

We agree about the awk, you convinced me that the "mkdir -p" proposal
did not make sense, but the remaining two issues puzzle me:

> > 3) If you use the Automake option dist-bzip2, then "make dist" uses

So do I understand correctly that if I use the dist-bzip2 option in a
GNU package, I'm breaching the GCS?
But if I call "make dist-bzip2" to create the second tarball in order
to publish both, it's ok?

IOW, the option would be very handy, but I'm not allowed to use it.

Doesn't that sound weird?

> > 2) "make dist" in Automake-generated makefiles depends on "find and
> > chmod" to remove the distdir.
>
> find and chmod are listed in GCS, `Install Command Categories'.

(Well, I have to admit this node is a surprise for me.  I'll need
time to understand it.)

The node `Utilities in Makefiles' says

: [...] the Makefile rules for [...] installation should not use any
: utilities directly except these:
: [awk] cat cmp cp diff echo egrep expr false grep install-info
: ln ls mkdir mv pwd rm rmdir sed sleep sort tar test touch true


`find' is not allowed, `chmod' is allowed at the end of the node, but
only if ``used via Make variables'' (sic!).

OTOH, node `Install Command Categories' allows a much broader set of
commands for pre- and post-install commands.

My first impression was: ``These statements contradicteach other!''
Then:
``Well, no.  Strictly interpreted, the intersection of these two is
allowed.  `find' is forbidden, `chmod' is allowed only through a
veriable.''
And:
``Automake uses Make variable `am__remove_distdir' for the cleanup.
Does that qualify as >>caling `chmod' via a Make variable<<?

Third interpretation: the node `Utilities in Makefiles' probably
meant normal instalation commands in the Make rules for installation,
not all command in those rules.

I'm totally dazzled.

I will try to read the whole make-stds.texi, perhaps I'll get a clue.

Right now, I can submit only the following one-line patch.

Have a nice day,
        Stepan Kasal

2007-08-30  Stepan Kasal  <kasal@...>

        * make-stds.texi (Utilities in Makefiles): Allow `awk'.

Index: make-stds.texi
===================================================================
RCS file: /cvsroot/gnustandards/gnustandards/make-stds.texi,v
retrieving revision 1.48
diff -u -r1.48 make-stds.texi
--- make-stds.texi 19 Aug 2007 00:01:31 -0000 1.48
+++ make-stds.texi 30 Aug 2007 19:58:32 -0000
@@ -155,7 +155,7 @@
 @c mkfifo mknod tee uname
 
 @example
-cat cmp cp diff echo egrep expr false grep install-info
+awk cat cmp cp diff echo egrep expr false grep install-info
 ln ls mkdir mv pwd rm rmdir sed sleep sort tar test touch true
 @end example
 

[Karl Berry]

    1) we should add "awk" to the list of alowed utils, as current CVS

My recollection is that I am awaiting a patch from Ralf on this issue
and a few others.  Ralf?  I'm not sure if your reply was also saying
that.  I know you sent some patches, but I think they've all been
integrated by now?

    4) Is it still adequate to say about "mkdir -p" that

It makes no difference operationally, as Ralf said, but I can change
"most" to "some" for clarity.  I suppose that is more accurate these
days.

As for dist-bzip2, I'm not clear about your concern.  The coding
standards can't possibly try to specify everything that's permitted or
supported, just set a minimum.

Best,
Karl

[Ralf Wildenhues]

* Karl Berry wrote on Fri, Aug 31, 2007 at 12:32:39AM CEST:
>     1) we should add "awk" to the list of alowed utils, as current CVS
>
> My recollection is that I am awaiting a patch from Ralf on this issue
> and a few others.  Ralf?

Probably best to go ahead with Stepan's awk change.

> I'm not sure if your reply was also saying that.

Yes, I was trying to say that.

> I know you sent some patches, but I think they've all been
> integrated by now?

I think so, yes.

> As for dist-bzip2, I'm not clear about your concern.  The coding
> standards can't possibly try to specify everything that's permitted or
> supported, just set a minimum.

I think this bit is best handled in the Automake manual, if it needs to
be handled at all.  (FWIW, I view Automake as "enabling you to produce a
GCS-compliant package", not as "forcing you to be GCS-compliant" which
maybe helps Stepan's confusion about dist-bzip2 a bit.)

FWIW, I do also think the GCS are very inconsistent between `Utilities
in Makefiles' and `Install Command Categories', but I'm certainly not
the one posting a cleanup patch for that in the near future.

Cheers,
Ralf

[Stepan Kasal]

Hi Karl,

On Thu, Aug 30, 2007 at 05:32:39PM -0500, Karl Berry wrote:
> It makes no difference operationally, as Ralf said, but I can change
> "most" to "some" for clarity.  I suppose that is more accurate these
> days.

OTOH, as Ralf said, only few implementations are known to be
race-free: GNU coreutils and recent BSD's.  That has convinced me
that it's ok to keep the "most" there.

> As for dist-bzip2, I'm not clear about your concern.

Let me put it this way: would you accept the following patch?
I believe it reflects common practice.

Regards,
        Stepan

[Stepan Kasal]

Hi,
  attached please find the patch I promised in my previous mail.
I'm sorry.

Stepan

On Fri, Aug 31, 2007 at 10:20:28AM +0200, Stepan Kasal wrote:
> On Thu, Aug 30, 2007 at 05:32:39PM -0500, Karl Berry wrote:
> > As for dist-bzip2, I'm not clear about your concern.
>
> Let me put it this way: would you accept the following patch?
> I believe it reflects common practice.

2007-08-30  Stepan Kasal  <kasal@...>

        * make-stds.texi (Utilities in Makefiles): Allow `bzip2'.

2007-08-30  Stepan Kasal  <kasal@...>

        * make-stds.texi (Utilities in Makefiles): Allow `awk'.

Index: make-stds.texi
===================================================================
RCS file: /cvsroot/gnustandards/gnustandards/make-stds.texi,v
retrieving revision 1.48
diff -u -r1.48 make-stds.texi
--- make-stds.texi 19 Aug 2007 00:01:31 -0000 1.48
+++ make-stds.texi 31 Aug 2007 08:16:44 -0000
@@ -155,7 +155,7 @@
 @c mkfifo mknod tee uname
 
 @example
-cat cmp cp diff echo egrep expr false grep install-info
+awk bzip2 cat cmp cp diff echo egrep expr false grep install-info
 ln ls mkdir mv pwd rm rmdir sed sleep sort tar test touch true
 @end example
 

[Karl Berry]

awk is one issue -- the argument (as I understand it) is that it is
needed to make autoconf configure scripts more efficient, and it adds
no extra bootstrap burden in practice.  (Right, Ralf?  Anything else I
should mention to rms.)

But bzip2 is another matter.  I don't see the need for it in *building
and installation*, which is what that list is about.  dist-bzip2 is
neither building nor installation, so there are no particular
restrictions on what can be run in that Makefile target.  So what's the
problem?

Thanks,
k

[Ralf Wildenhues]

* Karl Berry wrote on Sat, Sep 01, 2007 at 12:00:34AM CEST:
> awk is one issue -- the argument (as I understand it) is that it is
> needed to make autoconf configure scripts more efficient,

Yes, exactly: it allows config.status to operate with lower work
complexity (more efficiently) given many substituted values; for
large packages this can have substantial impact on configure time.

> and it adds no extra bootstrap burden in practice.

That's what we think: we tried any awk we could find; current gawk is
very portable and cross compilable (and there's no reason why that
should change).

> (Right, Ralf?  Anything else I should mention to rms.)

Well, the fact that old (Solaris) awk will do for Autoconf might help.

> But bzip2 is another matter.  I don't see the need for it in *building
> and installation*, which is what that list is about.  dist-bzip2 is
> neither building nor installation, so there are no particular
> restrictions on what can be run in that Makefile target.  So what's the
> problem?

I don't either think that bzip2 is needed here.  The GCS isn't explicit
in that 'make dist' shouldn't invoke any other tools (many packages do!)
than those listed there, AFAICS.

Cheers,
Ralf

[Stepan Kasal]

Hello,

On Fri, Aug 31, 2007 at 05:00:34PM -0500, Karl Berry wrote:
> But bzip2 is another matter.  I don't see the need for it in *building
> and installation*, which is what that list is about.  dist-bzip2 is
> neither building nor installation, so there are no particular
> restrictions on what can be run in that Makefile target.

you are right, of course.

But the following sentence says:

>>   The compression program `gzip' can be used in the `dist' rule.

My mind interpreted that as that the `dist' rule can use only the
programs allowed for *building and installation* plus `gzip'.

But the text does not in fact state that.

But if the `dist' rule is not in the scope of the regulation, then
the sentence quoted above does not have any effect.  Should it be
deleted?

Have a nice day,
        Stepan

[Karl Berry]

    But if the `dist' rule is not in the scope of the regulation, then
    the sentence quoted above does not have any effect.  Should it be
    deleted?

It would seem so -- seems like nothing but misleading to me, anyway.
Sigh.  More mail for rms's inbox.

k