Maybe confused about AMD64 / i386 compatibility

[Joe Greco]

Background (most probably not necessary):

We use rsync snapshots and cheap IDE disks as part of our backup strategy.
Recently, a backup server started getting near full, and with 1.5TB disks
being what they are, we shot to replace a 750GB IDE with a 1.5TB SATA.

The backup server was upgraded to FreeBSD 7.2R/i386, it sports a 4GB flash
which leaves two drive bays in the Intel ISP1100 chassis available.
Adding a SIL3112A gives us the SATA.  Good to go.  Create 1.5TB filesys,
and copy the old 750GB to it.

Option 1, using dd followed by growfs, doesn't work.  Groan.  growfs bombs
out with a negative integer in some array, and it isn't clear to me what
is happening.  So let's do the old slow but reliable dump/restore.

Except that a dump/restore of the filesystem, which is full of hardlinks,
causes massive resource consumption and eventual exhaustion.  Okay, fine.

Option 2, dump with gzip -1 to a file on the new drive (it'll have enough
space to restore in a separate step).  This works great.  Verify file
integrity (~400GB).  Also just fine.

Restore.  Watch restore grow to near 3GB and then bail.  Okay, obviously
an i386 architecture won't handle this restore.  So, think, aha, I'll go
and put it on a FreeBSD 7.1R/amd64 Opteron box with 8GB that is idle at
the moment.

-->> And here's where it all goes wonky.

Using a USB-to-SATA adapter, I plug in the drive to the amd64.  It sees
it.  I mount it.  All looks fine.  Good!

        # mount /dev/da1s1d /mnt
        # df
        Filesystem  1K-blocks       Used     Avail Capacity  Mounted on
        [...]
        /dev/da1s1d 1419039310 432472500 873043666    33%    /mnt

Now I try a restore.  So I do "gunzip < ad3s1e.dmp2 | restore rvf -" ...
and gzip claims the data isn't compressed.

        # gunzip < ad3s1e.dmp2 | restore rvf -
        gunzip: stdin: not in gzip format
        Verify tape and initialize maps
        End-of-tape encountered
        Tape is not a dump tape

Look at the data, looks generally compressed.  But file(1) sees it
as "data", not gzip data.

So, scratch head, wonder if it might be some odd gzip incompatibility,
get latest GNU gzip, still no go.  Rather annoyed, I think, "but I know
I verified this was a good file" - so I take the disk back to the i386,
put it in on the SATA controller, and it works fine.

So suddenly I think the USB-to-SATA is garbage and is corrupting the
data.  I md5 the first portions of the file, and go back, put the disk
on the amd64 with the USB-to-SATA.

        # dd if=ad3s1e.dmp2 bs=2048 count=1 | md5

Returns f86ea62b2c77c58691001371c283c7a0 on i386,
        43ca53ad03650bc8c8fd1279ce19a675 on amd64.

!!!!!!!

This seems really odd.  So just to be sure, I go back to the i386 box,
this time with the USB-to-SATA...  and it returns the correct answer.

Double "!!!!!!!!!"

Now I finally get smart and try working with the raw disk device.

        # dd if=/dev/da1s1 bs=2048 count=1024 | md5
        1024+0 records in
        1024+0 records out
        2097152 bytes transferred in 7.189189 secs (291709 bytes/sec)
        504efcdd93164c496c07603ba9b3774c

In all cases:

7.1amd64/USB: 504efcdd93164c496c07603ba9b3774c
7.2i386/SATA: 504efcdd93164c496c07603ba9b3774c
7.1amd64/USB: 504efcdd93164c496c07603ba9b3774c

So the underlying hardware and driver support "seems" to be fine.

And now this brings me to my question.

In my experience, FreeBSD filesystems are portable across i386-amd64.

Are there circumstances where this isn't the case?

I'm happy to concede that the filesystem in question is rather largish
(1.5TB) and probably stresses things a little bit.

# ls -al
total 432471268
drwxr-xr-x    5 root  wheel              512 Jun 13 11:07 .
drwxr-xr-x   24 root  wheel              512 Feb 18 09:18 ..
drwxrwxr-x    2 root  operator           512 Jun  5 08:20 .snap
-rw-r--r--    1 root  wheel     434044319653 Jun 10 19:25 ad3s1e.dmp2
-rw-r--r--    1 root  wheel               33 Jun 13 11:07 c1
-rw-r--r--    1 root  wheel               33 Jun 13 11:07 c1024
-rw-r--r--    1 root  wheel               33 Jun 13 11:07 c256
-rw-r--r--    1 root  wheel               33 Jun 13 11:07 c4
-rw-r--r--    1 root  wheel               80 Jun 13 10:43 logfile
drwx------  293 root  wheel            18432 Jun  8 04:25 lost+found
-rw-rw-rw-    1 root  wheel       8589934592 Jun  8 00:33 swapfile
drwxrwxrwx    9 root  wheel              512 Jun  8 14:14 x

Oh, and just to add some pain and agony to the puzzle ... it all seems
to work fine on 6.1-RELEASE/amd64 ... and 7.0-RELEASE/amd64.  But on a
different machine.  Which is where I'm doing my extraction.

So I'm not sure what to make of this.

... JG
--
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Peter Jeremy]

On 2009-Jun-13 15:55:29 -0500, Joe Greco <jgreco@...> wrote:
>Adding a SIL3112A gives us the SATA.

These are known to cause data corruption (check the archives).  I
wouldn't trust anything that has passed through a SIL chip without
independent validation.

--
Peter Jeremy

[Joe Greco]

> On 2009-Jun-13 15:55:29 -0500, Joe Greco <jgreco@...> wrote:
> >Adding a SIL3112A gives us the SATA.
>
> These are known to cause data corruption (check the archives).  I
> wouldn't trust anything that has passed through a SIL chip without
> independent validation.

I already said it had been validated.  gunzip|restore tvf was happy
with the entire thing.  A FreeBSD 6.1R/amd64 box is currently happily
RESTORING the SIL-written gzip'ed file using a USB-to-SATA adaptor,
TOO, so all evidence is that the on-disk file data is sane.

I checked for general data corruption at the device level with md5
and posted a brief summary of those results; the results are that
everything appears to be reading the disk blocks sanely.  That was
why I posted such a long summary of what had been done; I felt certain
someone would try to claim dodgy hardware.  The SIL does seem to
spit off lots of spurious interrupts, and does not work at all with
non-native SATA drives; being a backup system, I had already stress
tested various combinations of things, and I'm aware of the various
PC hardware deficiencies.

So, let me re-summarize and simplify the issue to clarify:

I have a large (~400GB) file on a large (~1.5TB) filesystem created
on 7.2R-i386.

7.2R-i386 reads the file correctly (via SIL or via USB-to-SATA).

6.1R-amd64 reads the file correctly (via USB-to-SATA).

7.1R-amd64 does NOT read the file correctly (via USB-to-SATA).

In all of the above cases, the underlying hardware and device drivers
appear to be returning the same data, as evidenced by dd <rawdev>|md5
of random portions of the disk.

In other words, the SIL is not in the equation.

... JG
--
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Joe Greco]

> On 2009-Jun-13 15:55:29 -0500, Joe Greco <jgreco@...> wrote:
> >Adding a SIL3112A gives us the SATA.
>
> These are known to cause data corruption (check the archives).  I
> wouldn't trust anything that has passed through a SIL chip without
> independent validation.

Well, as I originally stated, that wasn't an issue.

The problem has been replicated on other hardware and has been assigned
to freebsd-geom, I guess.

... JG
--
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."