GNU tar 1.22

[Sergey Poznyakoff-2]

Hello,

I am pleased to announce the release of GNU tar version 1.22.

GNU tar is available for download from
ftp://ftp.gnu.org/gnu/tar and the mirror sites worldwide (see
http://www.gnu.org/order/ftp.html for the list of those).

Here are the compressed sources:
  ftp://ftp.gnu.org/gnu/tar/tar-1.22.tar.gz    (2.9MB)
  ftp://ftp.gnu.org/gnu/tar/tar-1.22.tar.bz2   (2.1MB)
  ftp://ftp.gnu.org/gnu/tar/tar-1.22.shar.gz   (4.4MB)
  ftp://ftp.gnu.org/gnu/tar/tar-1.22.cpio.gz   (2.9MB)

Here are the GPG detached signatures[*]:
  ftp://ftp.gnu.org/gnu/tar/tar-1.22.tar.gz.sig
  ftp://ftp.gnu.org/gnu/tar/tar-1.22.tar.bz2.sig
  ftp://ftp.gnu.org/gnu/tar/tar-1.22.shar.gz.sig
  ftp://ftp.gnu.org/gnu/tar/tar-1.22.cpio.gz.sig

[*] You can use either of the above signature files to verify that
the corresponding file (without the .sig suffix) is intact.  First,
be sure to download both the .sig file and the corresponding tarball.
Then, run a command like this:

  gpg --verify tar-1.22.tar.gz.sig

If that command fails because you don't have the required public key,
then run this command to import it:

  gpg --keyserver keys.gnupg.net --recv-keys 55D0C732

and rerun the `gpg --verify' command.

This release was bootstrapped with the following tools:
  Autoconf 2.61
  Automake 1.10.1
  Gnulib-tool 0.0.1745-665a-modified

For more information on tar, including links to file downloads,
please see the tar web page: http://www.gnu.org/software/tar
and the project page http://savannah.gnu.org/projects/tar.

Please email bugs or suggestions to <bug-tar@...>.

Changes in this release:

* Support for xz compression

Tar uses xz for compression if one of the following conditions is met:

 1. The option --xz or -J (see below) is used.
 2. The xz binary is set as compressor using --use-compress-program option.
 3. The file name of the archive being created ends in `.xz' and
 auto-compress option (-a) is used.

Xz is used for decompression if one of the following conditions is met:

 1. The option --xz or -J is used.
 2. The xz binary is set as compressor using --use-compress-program option.
 3. The file is recognized as xz compressed stream data.

* Short option -J reassigned as a short equivalent of --xz

* New option -I

The -I option is assigned as a short equivalent for
--use-compress-program.

* The --no-recursive option works in incremental mode.

Regards,
Sergey

[Sebastian Pipping-4]

Sergey Poznyakoff wrote:

> Changes in this release:
>
> * Support for xz compression
>
> Tar uses xz for compression if one of the following conditions is met:
>
>  1. The option --xz or -J (see below) is used.
>  2. The xz binary is set as compressor using --use-compress-program option.
>  3. The file name of the archive being created ends in `.xz' and
>  auto-compress option (-a) is used.
>
> Xz is used for decompression if one of the following conditions is met:
>
>  1. The option --xz or -J is used.
>  2. The xz binary is set as compressor using --use-compress-program option.
>  3. The file is recognized as xz compressed stream data.
>
> * Short option -J reassigned as a short equivalent of --xz

Are you saying that -J no longer creates LZMA-compressed files?
What are the reasons for this decision?



Sebastian

[Sergey Poznyakoff-2]

Sebastian Pipping <webmaster@...> ha escrit:

> Are you saying that -J no longer creates LZMA-compressed files?

Yes, I am.

> What are the reasons for this decision?

We decided to reassign this option, because xz superceedes lzma and
provides a better compression. Therefore, a short option for it is more
useful.

If you still wish to create LZMA compressed files, uze the --lzma
option.

Regards,
Sergey

[Sebastian Pipping-4]

Sergey Poznyakoff wrote:
> We decided to reassign this option, because xz superceedes lzma and
> provides a better compression. Therefore, a short option for it is more
> useful.

Do I understand correctly, that the .lzma and .xz file formats ares
not compatible?  If that's true, why did you still decide to break
the API?  There were no other letters free to use for a new option?



Sebastian

[Sergey Poznyakoff-2]

Sebastian Pipping <webmaster@...> ha escrit:

> Do I understand correctly, that the .lzma and .xz file formats ares
> not compatible?

If I'm not mistaken, they are backward compatible.

> If that's true, why did you still decide to break
> the API?  There were no other letters free to use for a new option?

No, there are not.

Regards,
Sergey

[Sebastian Pipping-4]

Sergey Poznyakoff wrote:

> Sebastian Pipping <webmaster@...> ha escrit:
>
>> Do I understand correctly, that the .lzma and .xz file formats ares
>> not compatible?
>
> If I'm not mistaken, they are backward compatible.
>
>> If that's true, why did you still decide to break
>> the API?  There were no other letters free to use for a new option?
>
> No, there are not.

Are you sure?  It seems to me these letters are theoretically still
available:

  D E Q Y e n q y

I have collected these letters through 1) and 2) from the tar 1.21
man page as follows:

1) == Command ==
        for i in $(echo $(seq 65 90) $(seq 97 122) \
        | awk '{for(i=1;i<NF;i++){printf "%c\n", $i}}'); do \
                bzcat /usr/share/man/man1/tar.1.bz2 \
                | grep -- '-\b'$i'\b' >/dev/null \
                || echo "Unused letter $i" ; \
        done

        == Output (plus comments) ==
        Unused letter D
        Unused letter E
        Unused letter H // Undocumented?
        Unused letter I // Added in 1.22
        Unused letter J // Undocumented?
        Unused letter Q
        Unused letter Y
        Unused letter a // Undocumented?
        Unused letter e
        Unused letter n
        Unused letter q
        Unused letter y

2) == Command ==
        for i in $(echo $(seq 65 90) $(seq 97 122) \
        | awk '{for(i=1;i<NF;i++){printf "%c\n", $i}}'); do \
                bzcat /usr/share/man/man1/tar.1.bz2 \
                | grep -- '-\b'$i'\b' >/dev/null \
                || ( echo "[$i]" ; cd /var/tmp ; \
                tar -$i 2>&1 | sed 's/^/\t/' ) ; \
        done

        == Output ==
        <See attachment>


== Conclusion ==
If

  - .lzma and .xz are not 100% compatible and
  - Other option letters are available

I suggest to

  - revert -J to plain LZMA
  - add a new letter for XZ
  - do a new release asap

As neither Debian nor Gentoo seem to have release-based
xz-utils packages, they will need a few extra hours to package
that before bumping tar, so there might be time left to undo
damage done here in my eyes.

Looking forward to your feedback.




Sebastian


[D]
        tar: invalid option -- D
        Try `tar --help' or `tar --usage' for more information.
[E]
        tar: invalid option -- E
        Try `tar --help' or `tar --usage' for more information.
[H]
        tar: option requires an argument -- H
        Try `tar --help' or `tar --usage' for more information.
[I]
        tar: invalid option -- I
        Try `tar --help' or `tar --usage' for more information.
[J]
        tar: You must specify one of the `-Acdtrux' options
        Try `tar --help' or `tar --usage' for more information.
[Q]
        tar: invalid option -- Q
        Try `tar --help' or `tar --usage' for more information.
[Y]
        tar: invalid option -- Y
        Try `tar --help' or `tar --usage' for more information.
[a]
        tar: You must specify one of the `-Acdtrux' options
        Try `tar --help' or `tar --usage' for more information.
[e]
        tar: invalid option -- e
        Try `tar --help' or `tar --usage' for more information.
[n]
        tar: You must specify one of the `-Acdtrux' options
        Try `tar --help' or `tar --usage' for more information.
[q]
        tar: invalid option -- q
        Try `tar --help' or `tar --usage' for more information.
[y]
        tar: invalid option -- y
        Try `tar --help' or `tar --usage' for more information.

[Mike Frysinger]

On Thursday 05 March 2009 13:29:55 Sergey Poznyakoff wrote:
> Sebastian Pipping <webmaster@...> ha escrit:
> > Do I understand correctly, that the .lzma and .xz file formats ares
> > not compatible?
>
> If I'm not mistaken, they are backward compatible.

well, backwards compatible in the sense that the newer xz-utils package can
handle the older .lzma format just fine ...
-mike

[Sergey Poznyakoff-2]

Sebastian Pipping <webmaster@...> ha escrit:

> Are you sure? It seems to me these letters are theoretically still
> available:
>
>   D E Q Y e n q y

Yes, I am sure.  All these letters are reserved for future use.

> I have collected these letters through 1) and 2) from the tar 1.21
> man page as follows:

You could have saved effort if you'd looked at comment in src/tar.c,
lines 353-365.  The available option letters are listed there, along
with some of planned assignments.

>   - revert -J to plain LZMA
>   - add a new letter for XZ
>   - do a new release asap

No, the decision is taken, and there is no way back.

Regards,
Sergey

[Sergey Poznyakoff-2]

Mike Frysinger <vapier@...> ha escrit:

> well, backwards compatible in the sense that the newer xz-utils package can
> handle the older .lzma format just fine ...

Yes, that's what I meant.

Regards,
Sergey

[Sebastian Pipping-4]

Sergey Poznyakoff wrote:
>> Are you sure? It seems to me these letters are theoretically still
>> available:
>>
>>   D E Q Y e n q y
>
> Yes, I am sure.  All these letters are reserved for future use.

What kind of future use?  Why doesn't this qualify as "future use"?

Please don't forget to add the undocumented letters to
the man page.



Sebastian

Hello Sergey,

Sergey Poznyakoff wrote:
>>Do you plan to auto-compress/recognize lzip files (.lz)?
>
> No problem.  I'll only need some information on how to discern them
> (e.g. by file header signature).  Do you have any pointers?

I see tar-1.22.90 does not yet mention lzip files anywhere. In case my
last message was lost I send the pointers again.

File-5.00 (http://www.darwinsys.com/file/) recognizes lzip files. Here
is the magic entry for lzip:

# lzip
0       string          LZIP            lzip compressed data
!:mime application/x-lzip
 >4      byte            x               \b, version: %d


Also the lzip manual
(http://www.nongnu.org/lzip/manual/lzip_manual.html#File-Format)
describes the file format:

--------------------------------------------------------------------------
A lzip file consists of a series of "members" (compressed data sets).
The members simply appear one after another in the file, with no
additional information before, between, or after them.

Each member has the following structure:

+--+--+--+--+----+----+=============+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ID string | VN | DS | Lzma stream | CRC32 |   Data size   |  Member
size  |
+--+--+--+--+----+----+=============+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

All multibyte values are stored in little endian order.

`ID string'
     A four byte string, identifying the member type, with the value "LZIP".
`VN (version number, 1 byte)'
     Just in case something needs to be modified in the future. Valid
values are 0 and 1. Version 0 files have only one member and lack
`Member size'.
`DS (coded dictionary size, 1 byte)'
     Bits 4-0 contain the base 2 logarithm of the base dictionary size.
     Bits 7-5 contain the number of "wedges" to substract from the base
dictionary size to obtain the dictionary size. The size of a wedge is
(base dictionary size / 16).
     Valid values for dictionary size range from 4KiB to 512MiB.
`Lzma stream'
     The lzma stream, finished by an end of stream marker. Uses default
values for encoder properties.
`CRC32 (4 bytes)'
     CRC of the uncompressed original data.
`Data size (8 bytes)'
     Size of the uncompressed original data.
`Member size (8 bytes)'
     Total size of the member, including header and trailer. This
facilitates safe recovery of undamaged members from multimember files.
--------------------------------------------------------------------------

Thanks,
Antonio.

[Tim Kientzle]

Antonio Diaz Diaz wrote:

>
> Also the lzip manual
> (http://www.nongnu.org/lzip/manual/lzip_manual.html#File-Format)
> describes the file format:
>
> --------------------------------------------------------------------------
> A lzip file consists of a series of "members" (compressed data sets).
> The members simply appear one after another in the file, with no
> additional information before, between, or after them.
>
> Each member has the following structure:
>
> +--+--+--+--+----+----+=============+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>
> | ID string | VN | DS | Lzma stream | CRC32 |   Data size   |  Member
> size  |
> +--+--+--+--+----+----+=============+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Out of curiosity, can the LZMA stream data here be
decompressed/compressed with liblzma from XZ utils?

http://tukaani.org/xz/

The biggest problem I have with LZMA right now is
that there seem to be a lot of incompatible
compression algorithms and file formats all using
the same name.  It's very confusing.

Tim

Tim Kientzle wrote:
>> A lzip file consists of a series of "members" (compressed data sets).
>> Each member has the following structure:
>>
>> +--+--+--+--+----+----+=============+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>> | ID string | VN | DS | Lzma stream | CRC32 |   Data size   |  Member size  |
>> +--+--+--+--+----+----+=============+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>
> Out of curiosity, can the LZMA stream data here be
> decompressed/compressed with liblzma from XZ utils?

I suppose it can. Certainly it can be decompressed/compressed with the
lzma_alone tool from Pavlov's LZMA SDK if you change the header. Lzip's
LZMA algorithm is identical to the original from Pavlov except lzip only
uses default values for LZMA "properties".


> The biggest problem I have with LZMA right now is that there seem to
> be a lot of incompatible compression algorithms and file formats all
> using the same name.  It's very confusing.

AFAIK there are only two "LZMA algorithms", the original LZMA from
Pavlov, used by lzma_alone, lzma-utils, lzip, easylzma and lzmatools,
and LZMA2, used by xz-utils.

There seem to exist three LZMA file formats, the original lzma_alone
from Pavlov, lzip and xz. Both lzip and xz claim to be the successor of
lzma_alone, but given that xz supports several compression algorithms
and filters, and after reading this from the lzma-utils README file:
   "The current LZMA_Alone file format will be replaced with a new .lzma
    format. The new format fixes various problems the LZMA_Alone format
    such as lack of magic bytes and integrity check."
I think lzip is the only possible successor to the "title" of "LZMA file
format".


Regards,
Antonio.