'tar' in the GNU Coding Standards

[Dr. David Kirkby]

At: http://www.gnu.org/prep/standards/standards.html#Names

it says:
----------------------------------------------------------
The configure script and the Makefile rules for building and installation should
not use any utilities directly except these:

      awk cat cmp cp diff echo egrep expr false grep install-info
      ln ls mkdir mv pwd rm rmdir sed sleep sort tar test touch tr true

----------------------------------------------------------

Sorry, I can't give you a diff against standards.texi or make-stds.texi but here
are my thoughts, and a way to word it.

I believe 'tar' should removed from that list, but allow someone to select a
variable for 'tar'. On Solaris for instance, Sun's tar can not extract all tar
files programs written with GNU tar, though it does manage most of them. (It's
not simply the fact some options just as z and j are not supported, but more
fundamental issues.

Issues with non-GNU versions of tar are common on lots of non-linux platforms
when using GNU tools.

hence it's better the user was able to write:

$ TAR=/usr/local/bin/gtar
$ export TAR



Hence I suggest a revised wording of:

--------------SUGGESTED VERSION---------------------------
The configure script and the Makefile rules for building and installation should
not use any utilities directly except these:

      awk cat cmp cp diff echo egrep expr false grep install-info
      ln ls mkdir mv pwd rm rmdir sed sleep sort test touch tr true

----------------------------------------------------------


Further down, the GNU coding standards say:

------------------------------------------------------
The Makefile rules for building and installation can also use compilers and
related programs, but should do so via make variables so that the user can
substitute alternatives. Here are some of the programs we mean:

      ar bison cc flex install ld ldconfig lex
      make makeinfo ranlib texi2dvi yacc

Use the following make variables to run those programs:

      $(AR) $(BISON) $(CC) $(FLEX) $(INSTALL) $(LD) $(LDCONFIG) $(LEX)
      $(MAKE) $(MAKEINFO) $(RANLIB) $(TEXI2DVI) $(YACC)
----------------------------------------------------------

Given the above, I would suggest tar is added to those lists.

Hence I would suggest that the following

------------------- CURRENT VERSION ------------------------------
The Makefile rules for building and installation can also use compilers and
related programs, but should do so via make variables so that the user can
substitute alternatives. Here are some of the programs we mean:

      ar bison cc flex install ld ldconfig lex
      make makeinfo ranlib texi2dvi yacc

Use the following make variables to run those programs:

      $(AR) $(BISON) $(CC) $(FLEX) $(INSTALL) $(LD) $(LDCONFIG) $(LEX)
      $(MAKE) $(MAKEINFO) $(RANLIB) $(TEXI2DVI) $(YACC)
-------------------------------------------------------------------

is changed to


------------------SUGGESTED VERSION ---------------
The Makefile rules for building and installation can also use compilers and
related programs, but should do so via make variables so that the user can
substitute alternatives. Here are some of the programs we mean:

      ar bison cc flex install ld ldconfig lex
      makeinfo ranlib tar texi2dvi yacc

Use the following make variables to run those programs:

      $(AR) $(BISON) $(CC) $(FLEX) $(INSTALL) $(LD) $(LDCONFIG) $(LEX)
      $(MAKE) $(MAKEINFO) $(RANLIB) $(TAR) $(TEXI2DVI) $(YACC)
-------------------------------------------------------------------

[Karl Berry]

Hi David,

Thanks for writing.

    I believe 'tar' should removed from that list, but allow someone to
    select a variable for 'tar'.

In typical Makefiles tar is already a variable, as are others in that
list.  I'm not sure that we need to standardize the variable names for
them ...

Ralf, wdyt?

Thanks,
Karl

[Dr. David Kirkby]

Karl Berry wrote:

> Hi David,
>
> Thanks for writing.
>
>     I believe 'tar' should removed from that list, but allow someone to
>     select a variable for 'tar'.
>
> In typical Makefiles tar is already a variable, as are others in that
> list.  I'm not sure that we need to standardize the variable names for
> them ...
>
> Ralf, wdyt?
>
> Thanks,
> Karl
>

The piece above this says:


"The configure script and the Makefile rules for building and installation
should not use any utilities directly except these: "

so this is not only Makefiles, but configure scripts too.

I've seen several makefiles and scripts over the years where 'tar' is hard-coded
in, which will cause problems in some cases.

It should also be noted that egrep is not required to be installed on any system
conforming to POSIX 2004.

http://www.opengroup.org/onlinepubs/009695399/

If the intension is to write portable programs, one can't rely on using the
latest standards, but even the 2008 POSIX specification

http://www.opengroup.org/onlinepubs/9699919799/

does not require egrep. I learned that from reading the autoconf manual!

grep -E is supposed to support extended regular expressions (and GNU's grep
does), though according to the autoconf manual, not all greps conform to POSIX
and so therefore they do not all support grep -E.



Dave

[Ralf Wildenhues]

Hello David,

sorry about the delay.

* Dr. David Kirkby wrote on Sat, Oct 24, 2009 at 04:35:36PM CEST:
> ----------------------------------------------------------
> The configure script and the Makefile rules for building and
> installation should not use any utilities directly except these:
>
>      awk cat cmp cp diff echo egrep expr false grep install-info
>      ln ls mkdir mv pwd rm rmdir sed sleep sort tar test touch tr true
>
> ----------------------------------------------------------

> I believe 'tar' should removed from that list, but allow someone to
> select a variable for 'tar'. On Solaris for instance, Sun's tar can
> not extract all tar files programs written with GNU tar, though it
> does manage most of them. (It's not simply the fact some options
> just as z and j are not supported, but more fundamental issues.

I agree that there are problems with non-portable tarballs.  That brings
up a couple of points:

The typical user downloading a tarball from the net and extracting it
does not use the $(TAR) in the makefile inside the tarball; he uses tar
on the command line, before the makefile is used for the first time, and
he doesn't need `make dist' or similar which would use tar again from
within the makefile.

If tar is not sufficient to extract some package tarball, then this
ought to be clear from the installation instructions.  However,
requiring, say, GNU tar in general, would be quite some move; we don't
usually any other GNU tools to be preinstalled, if only for
bootstrapping reasons.

The 'Releases' node of the GCS describes some of the precautions to use
when creating package tarballs.  If possible, I think it would be
prudent to advise the tarball creator to use the most portable archive
format possible.  The GNU tar manual has `Portability' and `Formats'
sections about this, and the GNU Automake manual node `Options' shows
how to apply them to your package, should you use automake.  Automake
uses old v7 archive format by default for `make dist'.

I'm unsure of how wide-spread installation of a more capable tar program
is on systems with a limited vendor tar.

> Issues with non-GNU versions of tar are common on lots of non-linux
> platforms when using GNU tools.
>
> hence it's better the user was able to write:
>
> $ TAR=/usr/local/bin/gtar
> $ export TAR

How would you set $(TAR) if not supplied by the user though?  I haven't
yet seen a good Autoconf test for a capable tar, besides one that
wouldn't select GNU tar only; writing such a macro would probably be a
good thing to do before changing GCS.  Requiring the user to select a
tar program manually seems a bit awkward.  Note that one alternative to
choosing a tar program is to use something like the `missing' program
that Automake selects for some tools that may not be present.  OTOH I
don't see how packages can ignore or replace tar's functionality in that
case, unlike, say, makeinfo, where the info files are typically
distributed in the tarball.


BTW, I agree that egrep and fgrep should be fixed, as you noted
somewhere in this thread.

Thanks,
Ralf

[Dr. David Kirkby]

Ralf Wildenhues wrote:

> Hello David,
>
> sorry about the delay.
>
> * Dr. David Kirkby wrote on Sat, Oct 24, 2009 at 04:35:36PM CEST:
>> ----------------------------------------------------------
>> The configure script and the Makefile rules for building and
>> installation should not use any utilities directly except these:
>>
>>      awk cat cmp cp diff echo egrep expr false grep install-info
>>      ln ls mkdir mv pwd rm rmdir sed sleep sort tar test touch tr true
>>
>> ----------------------------------------------------------
>
>> I believe 'tar' should removed from that list, but allow someone to
>> select a variable for 'tar'. On Solaris for instance, Sun's tar can
>> not extract all tar files programs written with GNU tar, though it
>> does manage most of them. (It's not simply the fact some options
>> just as z and j are not supported, but more fundamental issues.
>
> I agree that there are problems with non-portable tarballs.  That brings
> up a couple of points:

The best way IMHO would be for GNU tar to default to a more portable format
rather than its own 'standard'.

Reading http://www.gnu.org/software/automake/manual/tar/Formats.html

it implies this will happen. The sooner the better I feel. I do not know how
well Sun's tar will handle the tar format of GNU tar then.


> The typical user downloading a tarball from the net and extracting it
> does not use the $(TAR) in the makefile inside the tarball; he uses tar
> on the command line, before the makefile is used for the first time, and
> he doesn't need `make dist' or similar which would use tar again from
> within the makefile.

> If tar is not sufficient to extract some package tarball, then this
> ought to be clear from the installation instructions.  However,
> requiring, say, GNU tar in general, would be quite some move; we don't
> usually any other GNU tools to be preinstalled, if only for
> bootstrapping reasons.

A much easier move would be to get GNU tar to write by default in a more
portable format.

I know there is one package in the Sage project which will not extract with
Sun's tar. But if one uses Sun's tar to create the tarball, GNU tar can extract
it ok.

> The 'Releases' node of the GCS describes some of the precautions to use
> when creating package tarballs.  If possible, I think it would be
> prudent to advise the tarball creator to use the most portable archive
> format possible.  The GNU tar manual has `Portability' and `Formats'
> sections about this, and the GNU Automake manual node `Options' shows
> how to apply them to your package, should you use automake.  Automake
> uses old v7 archive format by default for `make dist'.

That sounds sensible. I'm not sure why GNU tar does not default to something
sensible like that.

> I'm unsure of how wide-spread installation of a more capable tar program
> is on systems with a limited vendor tar.



>> Issues with non-GNU versions of tar are common on lots of non-linux
>> platforms when using GNU tools.
>>
>> hence it's better the user was able to write:
>>
>> $ TAR=/usr/local/bin/gtar
>> $ export TAR
>
> How would you set $(TAR) if not supplied by the user though?

I'm not a wizard on Makefiles, but is it not possible to write them in such a
way that if the environment variable 'TAR' is unset, then $TAR gets set to
'tar', otherwise it is whatever the TAR is set to?

> I haven't
> yet seen a good Autoconf test for a capable tar, besides one that
> wouldn't select GNU tar only; writing such a macro would probably be a
> good thing to do before changing GCS.  Requiring the user to select a
> tar program manually seems a bit awkward.  Note that one alternative to
> choosing a tar program is to use something like the `missing' program
> that Automake selects for some tools that may not be present.  OTOH I
> don't see how packages can ignore or replace tar's functionality in that
> case, unlike, say, makeinfo, where the info files are typically
> distributed in the tarball.

Jöerg Schilling's 'star' is one other implementation I am aware of. He seems to
have the knack of writing extremely portable code.

> BTW, I agree that egrep and fgrep should be fixed, as you noted
> somewhere in this thread.

Can you personally do something about the fgrep and egrep? Or do you know who to
handle the issue to?

> Thanks,
> Ralf
>

Dave

[Ralf Wildenhues]

* Dr. David Kirkby wrote on Sun, Nov 01, 2009 at 10:38:51PM CET:
>
> The best way IMHO would be for GNU tar to default to a more portable
> format rather than its own 'standard'.

I'm not sure whether that will happen: the more portable format has more
limitations, such as the 99 character path length limit.

> Reading http://www.gnu.org/software/automake/manual/tar/Formats.html
>
> it implies this will happen. The sooner the better I feel. I do not
> know how well Sun's tar will handle the tar format of GNU tar then.

I'm not sure what the text is meant to imply will happen, but I find it
weird if the GNU tar manual were to imply what direction GNU Automake
will evolve towards in the future.

> I know there is one package in the Sage project which will not
> extract with Sun's tar. But if one uses Sun's tar to create the
> tarball, GNU tar can extract it ok.

Well, can the tarball be created ok with the v7 format, and no symlinks?

> >The 'Releases' node of the GCS describes some of the precautions to use
> >when creating package tarballs.  If possible, I think it would be
> >prudent to advise the tarball creator to use the most portable archive
> >format possible.  The GNU tar manual has `Portability' and `Formats'
> >sections about this, and the GNU Automake manual node `Options' shows
> >how to apply them to your package, should you use automake.  Automake
> >uses old v7 archive format by default for `make dist'.
>
> That sounds sensible. I'm not sure why GNU tar does not default to
> something sensible like that.

I'm afraid you'll have to ask on a tar mailing list, or read its manual
for a rationale.

> >>hence it's better the user was able to write:
> >>
> >>$ TAR=/usr/local/bin/gtar
> >>$ export TAR
> >
> >How would you set $(TAR) if not supplied by the user though?
>
> I'm not a wizard on Makefiles, but is it not possible to write them
> in such a way that if the environment variable 'TAR' is unset, then
> $TAR gets set to 'tar', otherwise it is whatever the TAR is set to?

Well.  The make part of the answer is: Yes, with GNU make you can just
write
  TAR ?= tar

and with portable make, we can of course still default TAR to tar in
configure.ac:
  : ${TAR=tar}
  AC_SUBST([TAR])

But that's not the real knack.  If plain 'tar' is not sufficient, as you
imply, then the user must choose something different, or we need to have
a macro that automatically selects a better alternative.  So, either we
need to amend installation instructions, which we're fairly reluctant to
do, or we need a good kind of test to distinguish good 'tar' from bad
'tar', which I wouldn't really know how to do well.

> Jöerg Schilling's 'star' is one other implementation I am aware of.
> He seems to have the knack of writing extremely portable code.

That's fairly irrelevant for the issue at hand though; the question is
not whether there exists a portable tar program (GNU tar would be a
candidate as well), but which kind of functionality you can expect to
be *already installed* on your users' systems.

> >BTW, I agree that egrep and fgrep should be fixed, as you noted
> >somewhere in this thread.
>
> Can you personally do something about the fgrep and egrep? Or do you
> know who to handle the issue to?

I'll handle it.

Thanks,
Ralf