Wrong Native Bit Order on Intel x86_64

_______________________________________________
Tiff mailing list: Tiff@...
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/

On Wed, 7 Oct 2009, Eskandar Ensafi wrote:
>
> I noticed that when I tried to compile tiff-4.0.0-beta4, the native bit order on 64-bit x86_64 systems was
> determined to be FILLORDER_MSB2LSB (not correct as far as I know) whereas on 32-bit x86 systems, it is
> correctly determined to be FILLORDER_LSB2MSB.  I checked my current tiff-3.8.2 installation (the standard
> package provided in Red Hat Enterprise Linux 5.4), and the same discrepancy existed in tiff-3.8.

This is very interesting.  It seems like there should be some
efficient way to determine this at run-time rather than via the
configure script.  I am not sure how to test for native fill order
though.

Bob
--
Bob Friesenhahn
bfriesen@..., http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/
_______________________________________________
Tiff mailing list: Tiff@...
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/

A run-time check would be nice, but I too am not sure how to implement it.
I seem to recall that "bit order" (unlike "byte order") is not something
that you can detect in C.

Hopefully, this tag is not being used by most applications because it
would mean that when you write an image on a 64-bit system and try to view
it on a 32-bit system, or vice versa, then the bits will be reversed and
the image data will become rubbish.  I'm not so sure if "bit order" really
matters, does it?

Eskandar


-----Original Message-----
From: Bob Friesenhahn [mailto:bfriesen@...]
Sent: Wednesday, October 07, 2009 1:00 PM
To: Eskandar Ensafi
Cc: tiff@...
Subject: Re: [Tiff] Wrong Native Bit Order on Intel x86_64

On Wed, 7 Oct 2009, Eskandar Ensafi wrote:
>
> I noticed that when I tried to compile tiff-4.0.0-beta4, the native bit
order on 64-bit x86_64 systems was
> determined to be FILLORDER_MSB2LSB (not correct as far as I know)
whereas on 32-bit x86 systems, it is
> correctly determined to be FILLORDER_LSB2MSB.  I checked my current
tiff-3.8.2 installation (the standard
> package provided in Red Hat Enterprise Linux 5.4), and the same
discrepancy existed in tiff-3.8.

This is very interesting.  It seems like there should be some
efficient way to determine this at run-time rather than via the
configure script.  I am not sure how to test for native fill order
though.

Bob
--
Bob Friesenhahn
bfriesen@..., http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/
_______________________________________________
Tiff mailing list: Tiff@...
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/

>
> In the tiff-4.0 source code, I traced the problem to line 16124 of
> the configure script (or line 327 of configure.ac), where the script
> checks for i*86* instead of i*86*|x86_64.  Unless there is a good
> reason for the native bit order to be MSB-to-LSB on x86_64, I
> believe that this is a bug and needs to be corrected so that all
> Intel and AMD x86 varieties are properly identified as LSB-to-MSB
> architectures.

This issue is now fixed in libtiff CVS.

Hopefully someone will speak up if this was the wrong decision.

Bob
--
Bob Friesenhahn
bfriesen@..., http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/
_______________________________________________
Tiff mailing list: Tiff@...
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/

On 7-Oct-09, at 4:11 PM, Eskandar Ensafi wrote:

> A run-time check would be nice, but I too am not sure how to  
> implement it.
> I seem to recall that "bit order" (unlike "byte order") is not  
> something
> that you can detect in C.

I think you are right.

>
> Hopefully, this tag is not being used by most applications because it
> would mean that when you write an image on a 64-bit system and try  
> to view
> it on a 32-bit system, or vice versa, then the bits will be  
> reversed and
> the image data will become rubbish.  I'm not so sure if "bit order"  
> really
> matters, does it?

IMHO, I agree, native order is irrelevant. You could choose any  
default here.

The same is true of byte order, more or less, but it's easy to sniff  
so easy to write the same order as platform.

--Toby

_______________________________________________
Tiff mailing list: Tiff@...
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/

Eskandar Ensafi wrote:
  > Hopefully, this tag is not being used by most applications because it
> would mean that when you write an image on a 64-bit system and try to view
> it on a 32-bit system, or vice versa, then the bits will be reversed and
> the image data will become rubbish.  I'm not so sure if "bit order" really
> matters, does it?

My (probably incorrect) understanding of HOST_FILLORDER is that it does
not matter. I found this on the net,
        http://www.remotesensing.org/libtiff/internals.html
        "Native CPU byte order is determined on the fly by the library
          and does not need to be specified. The HOST_FILLORDER and
          HOST_BIGENDIAN definitions are not currently used, but may be
          employed by codecs for optimization purposes. "
This seems to supercede somewhat the discussion previously on this
mailing list:
        http://www.asmail.be/msg0054849001.html

I also have an old note though saying that the HOST_FILLORDER just
dictates the order in which the bits get saved into the .tiff file and
that libtiff will correctly interpret the bits when reading the .tiff
file according to the HOST_FILLORDER it was written with.

Regards,
-Edward

_______________________________________________
Tiff mailing list: Tiff@...
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/

Bob Friesenhahn wrote:
I doubt it's the wrong decision since we're just being consistent on
x86_64 with 32-bit x86 on this matter. If we're wrong here, then that
means the original decision on 32-bit x86 was wrong as well. :)

Having said that, it would be nice if we had a regression test of some
sort that would actually test this decision. ie. that libtiff will
correctly read .tiff files generated with both fill orders.

Regards,
-Edward
_______________________________________________
Tiff mailing list: Tiff@...
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/

Hello Bob,

Wednesday, October 7, 2009, 9:59:45 PM, you wrote:

BF> On Wed, 7 Oct 2009, Eskandar Ensafi wrote:
>>
>> I noticed that when I tried to compile tiff-4.0.0-beta4, the native bit order on 64-bit x86_64 systems was
>> determined to be FILLORDER_MSB2LSB (not correct as far as I know) whereas on 32-bit x86 systems, it is
>> correctly determined to be FILLORDER_LSB2MSB.  I checked my current tiff-3.8.2 installation (the standard
>> package provided in Red Hat Enterprise Linux 5.4), and the same discrepancy existed in tiff-3.8.

BF> This is very interesting.  It seems like there should be some
BF> efficient way to determine this at run-time rather than via the
BF> configure script.  I am not sure how to test for native fill order
BF> though.

My guess would be:

if ( (1<<1) == 2 )
  // LSB2MSB
else
  // MSB2LSB

--
WBR,
 Igor                            mailto:skochinsky@...

_______________________________________________
Tiff mailing list: Tiff@...
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/

On 7-Oct-09, at 4:47 PM, Igor Skochinsky wrote:

This cannot work, of course.

As Eskandar suspected, native bit numbering is not visible to C. You  
could expose it using platform specific assembler, but it's probably  
pointless effort.

--Toby

>
> --
> WBR,
>  Igor                            mailto:skochinsky@...
>
> _______________________________________________
> Tiff mailing list: Tiff@...
> http://lists.maptools.org/mailman/listinfo/tiff
> http://www.remotesensing.org/libtiff/

_______________________________________________
Tiff mailing list: Tiff@...
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/

On Wed, 7 Oct 2009, Eskandar Ensafi wrote:
>
> Hopefully, this tag is not being used by most applications because it
> would mean that when you write an image on a 64-bit system and try to view
> it on a 32-bit system, or vice versa, then the bits will be reversed and
> the image data will become rubbish.  I'm not so sure if "bit order" really
> matters, does it?

Of course bit order really does matter. If libtiff encounters a TIFF
file written in a different order then it needs to reverse the order
of the bits while reading.  If TIFF is requested to write output in a
different order than the native order, then it needs to reverse the
order of the bits while writing.  It does this via a TIFFReverseBits()
function in libtiff/tif_swab.c.  This is important for FAX
compression.

Here is where it is used:

   File        Function              Line
1 tif_jbig.c  JBIGDecode             86 TIFFReverseBits(tif->tif_rawdata, tif->tif_rawdatasize);
2 tif_jbig.c  JBIGOutputBie         175 TIFFReverseBits(buffer, len);
3 tif_read.c  TIFFFillStrip         354 TIFFReverseBits(tif->tif_rawdata, bytecount);
4 tif_read.c  TIFFFillTile          577 TIFFReverseBits(tif->tif_rawdata, bytecount);
5 tif_write.c TIFFWriteEncodedStrip 249 TIFFReverseBits(tif->tif_rawdata, tif->tif_rawcc);
6 tif_write.c TIFFWriteEncodedTile  404 TIFFReverseBits((unsigned char *)tif->tif_rawdata, tif->tif_rawcc);
7 tif_write.c TIFFFlushData1        687 TIFFReverseBits((unsigned char *)tif->tif_rawdata,

Some CPUs use a different fill order than others just like some CPUs
are big endian and other CPUs are little endian.

It seems pretty mysterious that it took three years for this issue to
be discovered and reported so probably there is something I am not
understanding.

Bob
--
Bob Friesenhahn
bfriesen@..., http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/
_______________________________________________
Tiff mailing list: Tiff@...
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/

On Wed, 7 Oct 2009, Toby Thain wrote:
>
> As I understand it, the problem is not in the reading and writing code or
> runtime behaviour, but relates to determination of the built default (May not
> match platform's "actual" machine default.)
>
> The 'bug' would be at build-time, but it seems relatively unimportant to get
> right, esp. for a C library.

I hope that you don't mind that I am Cc:ing this response to the list
since it is an important issue to correct properly.

If the default is wrong, then the best-case penalty is that CPU is
wasted executing TIFFReverseBits() when writing and reading files on
the same system and the user explicitly specified the fill-order tag
correctly for the CPU.  The worst-case penalty is that the files can't
be read by some other host.  Since it took three years for this issue
to be reported, the worst-case penalty seems unlikely since
interoperability issues are noticed right away, and because no one has
reported fill-order issues on unusual CPUs.

Bob
--
Bob Friesenhahn
bfriesen@..., http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/
_______________________________________________
Tiff mailing list: Tiff@...
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/