ext2 file system: moving between armel and amd64 etc.

[Osamu Aoki]

Hi,

I realized that mkfs.ext2 creates different superblock depending on which
system it is run under.  The difference can be observed by tune2fs -l:

 ext2 created by Debian lenny on amd64:
  Filesystem flags: signed_directory_hash

 ext2 created by Ubuntu 09.4 armel:
  Filesystem flags: unsigned_directory_hash

It seems it comes from the fact that armel uses unsigned for char.

Since kernel support of this flag is new and lenny or jaunty one seems
to ignore it, I could move file between these 2 system.  (I do fsck -p
to get hash fixed every time I move, first.)  So it works now but this
is sloppy.

What is the right tool to change this "Filesystem flags" setting.  I do
not see in lenny or jaunty.  Can new tune2fs adjust this parameter?
What is the right way to move ext2 filesystem between system?

Osamu

PS: I am playing with Sharp PC-Z1.  
  http://wiki.debian.org/InstallingDebianOn/Sharp/PC-Z1NetWalker


--
To UNSUBSCRIBE, email to debian-arm-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Riku Voipio-2]

Hi Osamu,

On Mon, Oct 26, 2009 at 09:25:36PM +0900, Osamu Aoki wrote:

> I realized that mkfs.ext2 creates different superblock depending on which
> system it is run under.  The difference can be observed by tune2fs -l:
>
>  ext2 created by Debian lenny on amd64:
>   Filesystem flags: signed_directory_hash
>
>  ext2 created by Ubuntu 09.4 armel:
>   Filesystem flags: unsigned_directory_hash
>
> It seems it comes from the fact that armel uses unsigned for char.

Ted, can you check if e2fsprogs expects signed char in the code that
checks for signed directory hash?

> Since kernel support of this flag is new and lenny or jaunty one seems
> to ignore it, I could move file between these 2 system.  (I do fsck -p
> to get hash fixed every time I move, first.)  So it works now but this
> is sloppy.
>
> What is the right tool to change this "Filesystem flags" setting.  I do
> not see in lenny or jaunty.  Can new tune2fs adjust this parameter?
> What is the right way to move ext2 filesystem between system?

Rather, lets try to get it fixed instead. There should be no reason
to run tune2fs when moving a filesystem from armel to x86 machine.

> PS: I am playing with Sharp PC-Z1.  
>   http://wiki.debian.org/InstallingDebianOn/Sharp/PC-Z1NetWalker

Seems like a very interesting device :)


--
To UNSUBSCRIBE, email to debian-arm-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Martin Michlmayr]

* Riku Voipio <riku.voipio@...> [2009-10-26 16:07]:

> >  ext2 created by Debian lenny on amd64:
> >   Filesystem flags: signed_directory_hash
> >
> >  ext2 created by Ubuntu 09.4 armel:
> >   Filesystem flags: unsigned_directory_hash
> >
> > It seems it comes from the fact that armel uses unsigned for char.
>
> Ted, can you check if e2fsprogs expects signed char in the code that
> checks for signed directory hash?

FWIW, such a problem was discussed on this list last year and resulted
in a kernel fix for ext3 and ext4, see
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=5e1f8c9e20a92743eefc9a82c2db835213905e26
The ext3 fix is in the lenny kernel.

I don't think I ever saw a fix for ext2.
--
Martin Michlmayr
http://www.cyrius.com/


--
To UNSUBSCRIBE, email to debian-arm-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Osamu Aoki]

Hi,

On Mon, Oct 26, 2009 at 04:07:58PM +0200, Riku Voipio wrote:

> Hi Osamu,
>
> On Mon, Oct 26, 2009 at 09:25:36PM +0900, Osamu Aoki wrote:
> > I realized that mkfs.ext2 creates different superblock depending on which
> > system it is run under.  The difference can be observed by tune2fs -l:
> >
> >  ext2 created by Debian lenny on amd64:
> >   Filesystem flags: signed_directory_hash
> >
> >  ext2 created by Ubuntu 09.4 armel:
> >   Filesystem flags: unsigned_directory_hash
> >
> > It seems it comes from the fact that armel uses unsigned for char.

I should have put some reference why I said it.  This was because of
Ted's patch.
 
> Ted, can you check if e2fsprogs expects signed char in the code that
> checks for signed directory hash?

I can see http://patchwork.kernel.org/patch/22568/ (2009-05-08 11:58:01) has:

From: Theodore Ts'o <tytso@...>

The original ext3 hash algorithms assumed that variables of type char
were signed, as God and K&R intended.  Unfortunately, this assumption
is not true on some architectures.  Userspace support for marking
filesystems with non-native signed/unsigned chars was added two years
ago, but the kernel-side support was never added (until now).

Signed-off-by: "Theodore Ts'o" <tytso@...>
Cc: akpm@...
Cc: linux-kernel@...

...
 

> > Since kernel support of this flag is new and lenny or jaunty one seems
> > to ignore it, I could move file between these 2 system.  (I do fsck -p
> > to get hash fixed every time I move, first.)  So it works now but this
> > is sloppy.
> >
> > What is the right tool to change this "Filesystem flags" setting.  I do
> > not see in lenny or jaunty.  Can new tune2fs adjust this parameter?
> > What is the right way to move ext2 filesystem between system?
>
> Rather, lets try to get it fixed instead. There should be no reason
> to run tune2fs when moving a filesystem from armel to x86 machine.

With newer patched kernels after 2009-05-08 11:58:01, I expect this is
handled well.  But now we do fsck in one environment without changing
this signed/unsigned flag.  By doind so will make filesystem to be fsck
clean but not matching signed/unsigned flag.

This is why I asked how you have been dealing this situation as porters.

> > PS: I am playing with Sharp PC-Z1.  
> >   http://wiki.debian.org/InstallingDebianOn/Sharp/PC-Z1NetWalker
>
> Seems like a very interesting device :)

Yes.  It is my toy.

Since this is almost like PC with USB and microSD card, I am tempted to
format them in ext2.


--
To UNSUBSCRIBE, email to debian-arm-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Osamu Aoki]

On Mon, Oct 26, 2009 at 11:36:56PM +0800, Martin Michlmayr wrote:

> * Riku Voipio <riku.voipio@...> [2009-10-26 16:07]:
> > >  ext2 created by Debian lenny on amd64:
> > >   Filesystem flags: signed_directory_hash
> > >
> > >  ext2 created by Ubuntu 09.4 armel:
> > >   Filesystem flags: unsigned_directory_hash
> > >
> > > It seems it comes from the fact that armel uses unsigned for char.
> >
> > Ted, can you check if e2fsprogs expects signed char in the code that
> > checks for signed directory hash?
>
> FWIW, such a problem was discussed on this list last year and resulted
> in a kernel fix for ext3 and ext4, see
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=5e1f8c9e20a92743eefc9a82c2db835213905e26
> The ext3 fix is in the lenny kernel.

Yes.
 
> I don't think I ever saw a fix for ext2.

I see fs/ext3/hash.c but do not see fs/ext2/hash.c in kernel.

Let me be specific.  Let me not make judgement if I have ext2 or ext3
without journaling.

I made filesystem using mkfs.ext2. This is fact.  

I did not experience any problem by moving microSD card between armel
and amd64 if I mount it to running system.

But if this microSD card contains bootable system image and I try to
boot it, it haults its booting at /etc/rcS.d/S20checkroot.sh makes fsck
(This boot image was Ubuntu based).

By running "fsck.ext2 -p" from armel system on this microSD card before
booting from it, it became bootable but it kept "signed" flag just like
on amd64.  Since this is now unsigned as I assumed, I wanted to change
this flag with some utility to match what the filesystem is like.

Osamu


--
To UNSUBSCRIBE, email to debian-arm-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[tytso]

On Wed, Oct 28, 2009 at 01:01:25AM +0900, Osamu Aoki wrote:

>
> I see fs/ext3/hash.c but do not see fs/ext2/hash.c in kernel.
>
> Let me be specific.  Let me not make judgement if I have ext2 or ext3
> without journaling.
>
> I made filesystem using mkfs.ext2. This is fact.  
>
> I did not experience any problem by moving microSD card between armel
> and amd64 if I mount it to running system.

I didn't get the first message in this thread, so I'm not sure I
completely understand the context of the question, but it's correct
that the signed vs. unsigned char problem with the hash algorithm only
applies to ext3 and ext4 filesystems, since ext2 doesn't have support
for htree directory indexing, which is where the issue showed up.

As far as I know there are no current signed vs unsigned issues in
ext2/3/4, assuming you are using modern kernels and e2fsprogs.

Regards,

                                        - Ted


--
To UNSUBSCRIBE, email to debian-arm-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Osamu Aoki]

Hi,

On Tue, Oct 27, 2009 at 02:01:28PM -0400, Theodore Tso wrote:

> On Wed, Oct 28, 2009 at 01:01:25AM +0900, Osamu Aoki wrote:
> >
> > I see fs/ext3/hash.c but do not see fs/ext2/hash.c in kernel.
> >
> > Let me be specific.  Let me not make judgement if I have ext2 or ext3
> > without journaling.
> >
> > I made filesystem using mkfs.ext2. This is fact.  
> >
> > I did not experience any problem by moving microSD card between armel
> > and amd64 if I mount it to running system.
>
> I didn't get the first message in this thread, so I'm not sure I
> completely understand the context of the question, but it's correct
> that the signed vs. unsigned char problem with the hash algorithm only
> applies to ext3 and ext4 filesystems, since ext2 doesn't have support
> for htree directory indexing, which is where the issue showed up.
>
> As far as I know there are no current signed vs unsigned issues in
> ext2/3/4, assuming you are using modern kernels and e2fsprogs.

Yes, Debian stable kernels, despite it has non-modern" version number,
have this patch applied.

My questions can be summarized as following 3 questions:

Q1.  Is there some option to fsck.ext2
         to force unsigned under amd64 or
         to force   signed under armel.

Q2.  Is there option to tune2fs to change signed or unsigned flag.

Q3.  Is it normal to have exitcode=1 for fsck under armel if partition
     has been created with mkfs.ext2 under amd64 to be unsigned? (It
     seemed to me that fsck under such case tends to find the
     filesystem not to be cleanly unmounted.)

Basically, I am experiencing some problem as described as Q3 and wanted
to do either Q1 or Q2 actions.

Osamu


--
To UNSUBSCRIBE, email to debian-arm-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[martinwguy]

>  Q1.  Is there some option to fsck.ext2
>          to force unsigned under amd64 or
>          to force   signed under armel.

man fsck

>  Q2.  Is there option to tune2fs to change signed or unsigned flag.

man tune2fs

>  Q3.  Is it normal to have exitcode=1 for fsck under armel if partition
>      has been created with mkfs.ext2 under amd64 to be unsigned? (It
>      seemed to me that fsck under such case tends to find the
>      filesystem not to be cleanly unmounted.)

vi mkfs.c  :)

    M


--
To UNSUBSCRIBE, email to debian-arm-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[tytso]

On Thu, Oct 29, 2009 at 11:04:54PM +0900, Osamu Aoki wrote:
> My questions can be summarized as following 3 questions:
>
> Q1.  Is there some option to fsck.ext2
>          to force unsigned under amd64 or
>          to force   signed under armel.

No.

> Q2.  Is there option to tune2fs to change signed or unsigned flag.

No.

> Q3.  Is it normal to have exitcode=1 for fsck under armel if partition
>      has been created with mkfs.ext2 under amd64 to be unsigned?

Only the first time fsck is run, if the filesystem was created using
an ancient mke2fs.  After that, no, assuming the filesystem was
cleanly unmounted.

>      (It seemed to me that fsck under such case tends to find the
>      filesystem not to be cleanly unmounted.)

I think you're jumping to conclusions here, and since I didn't see
your initial message, I don't know why you think this might be the
case.  How about details about what you are seeing, in as much detail
as you can, with copies of the fsck output as the bare minimum?

The issue with the signed vs unsigned char problem with
htree/dir_index feature --- which does not even exist in ext2, by the
way --- was that the algorithm used for hashing directory names for
htrees was depended on whether or not chars are signed or unsigned.
Given a particular filesystem, it's impossible to know whether it was
originally created and written on a system with a signed char or
unsigned char.  For systems with purely ASCII filenames, where high
bit of each character is zero, it didn't make a difference.  However,
for people who tried to store UTF-8 or ISO-8859-1 chars in their
filenames, those files would not be accessible if the filesystem was
moved between a system with signed chars vs unsigned chars, and a
forced fsck would cause e2fsck to complain and fix up the directory
for the other architecture.

The way we fixed this was with the following hack.  The superblock has
two flags, EXT2_FLAGS_SIGNED_HASH and EXT2_FLAGS_UNSIGNED_HASH.  If
the either flag is set, modern kernels will use the hash algorithm
appropriately tweaked as if chars are signed or unsigned, regardless
of the natural "signed vs unsigned" nature of chars on that platform.
If neither flag is set, the kernel will use the default signed vs
unsigned char version of the algorithm.

If neither flag is set, modern e2fsck binaries will set one of these
two flags, based on the archtecture that it is run on.  Modern mke2fs
binaries will set whichever flag is appropriate with its arguments.
In both cases, the test that is done in C looks like this:

        c = (char) 255;
        if (((int) c) == -1) {
                super->s_flags |= EXT2_FLAGS_SIGNED_HASH;
        } else {
                super->s_flags |= EXT2_FLAGS_UNSIGNED_HASH;
        }

It really doesn't matter whether the signed or unsigned version of the
hash algorithm is used, as long as it's consistent.  So If you happen
to create the filesystem on an unsigned char system, and then use the
filesystem exclusively on a signed char system, things will work fine,
so long as everybody is using modern versions of kernel and e2fsprogs.
You might have problems if occasionally boot into an ancient kernel
that doesn't understand these flags, *AND* you use non-ASCII
characters in filenames.  But that's the only lossage mode you should
see.

If you create the file system using an ancient version of mke2fs, and
then run a modern version of e2fsck, it will print a message to this
fact: "Adding dirhash hint to filesystem", and then since it has
modified the filesystem, it will return with an exit code of 1.  But
it will only do this the first time, since after that point, the
dirhash hint will have been set.

Does this help explain what's going on?

                                                - Ted


--
To UNSUBSCRIBE, email to debian-arm-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Osamu Aoki]

Hi,

Excuse me for my slow response.  

On Thu, Oct 29, 2009 at 03:29:27PM -0400, Theodore Tso wrote:

> On Thu, Oct 29, 2009 at 11:04:54PM +0900, Osamu Aoki wrote:
> > My questions can be summarized as following 3 questions:
> >
> > Q1.  Is there some option to fsck.ext2
> >          to force unsigned under amd64 or
> >          to force   signed under armel.
>
> No.
>
> > Q2.  Is there option to tune2fs to change signed or unsigned flag.
>
> No.

Thanks for confirming that the manpage is the complete one.
 
> > Q3.  Is it normal to have exitcode=1 for fsck under armel if partition
> >      has been created with mkfs.ext2 under amd64 to be unsigned?
>
> Only the first time fsck is run, if the filesystem was created using
> an ancient mke2fs.  After that, no, assuming the filesystem was
> cleanly unmounted.

Thanks.   Debian lenny mke2fs seems current enough as I glanced its source.

As we also know that its kernel properly patched, Debian lenny is in
good shape.

> >      (It seemed to me that fsck under such case tends to find the
> >      filesystem not to be cleanly unmounted.)
>
> I think you're jumping to conclusions here, and since I didn't see
> your initial message, I don't know why you think this might be the
> case.  How about details about what you are seeing, in as much detail
> as you can, with copies of the fsck output as the bare minimum?

Basically, my armel system is not functioning now.  Give me some time.

Here, my armel system on which I had issues was an odd vender provided
system probably created by hardware vender in connection with Canonical.
This device, Sharp PC-Z1, provides us with a system re-initialization
disk on microSD card.  That disk image contains some derivative of
Ubuntu/Debian binary.  The problem seen was when I was running this
re-initialization disk. The fsck code in its init script exits upon
seeing non-arm initialized ext2 disk.

> The issue with the signed vs unsigned char problem with
> htree/dir_index feature --- which does not even exist in ext2, by the
> way --- was that the algorithm used for hashing directory names for
> htrees was depended on whether or not chars are signed or unsigned.
> Given a particular filesystem, it's impossible to know whether it was
> originally created and written on a system with a signed char or
> unsigned char.  For systems with purely ASCII filenames, where high
> bit of each character is zero, it didn't make a difference.  However,
> for people who tried to store UTF-8 or ISO-8859-1 chars in their
> filenames, those files would not be accessible if the filesystem was
> moved between a system with signed chars vs unsigned chars, and a
> forced fsck would cause e2fsck to complain and fix up the directory
> for the other architecture.
>
> The way we fixed this was with the following hack.  The superblock has
> two flags, EXT2_FLAGS_SIGNED_HASH and EXT2_FLAGS_UNSIGNED_HASH.  If
> the either flag is set, modern kernels will use the hash algorithm
> appropriately tweaked as if chars are signed or unsigned, regardless
> of the natural "signed vs unsigned" nature of chars on that platform.
> If neither flag is set, the kernel will use the default signed vs
> unsigned char version of the algorithm.
>
> If neither flag is set, modern e2fsck binaries will set one of these
> two flags, based on the architecture that it is run on.  Modern mke2fs
> binaries will set whichever flag is appropriate with its arguments.
> In both cases, the test that is done in C looks like this:
>
> c = (char) 255;
> if (((int) c) == -1) {
> super->s_flags |= EXT2_FLAGS_SIGNED_HASH;
> } else {
> super->s_flags |= EXT2_FLAGS_UNSIGNED_HASH;
> }
>
> It really doesn't matter whether the signed or unsigned version of the
> hash algorithm is used, as long as it's consistent.  So If you happen
> to create the filesystem on an unsigned char system, and then use the
> filesystem exclusively on a signed char system, things will work fine,
> so long as everybody is using modern versions of kernel and e2fsprogs.
> You might have problems if occasionally boot into an ancient kernel
> that doesn't understand these flags, *AND* you use non-ASCII
> characters in filenames.  

(Kernel was not giving problem.  It was fsck in init code.)

> But that's the only lossage mode you should
> see.

Thanks for this explanation.
 
> If you create the file system using an ancient version of mke2fs, and
> then run a modern version of e2fsck, it will print a message to this
> fact: "Adding dirhash hint to filesystem", and then since it has
> modified the filesystem, it will return with an exit code of 1.  But
> it will only do this the first time, since after that point, the
> dirhash hint will have been set.
>
> Does this help explain what's going on?

Very much so.  Thank you.

Osamu


--
To UNSUBSCRIBE, email to debian-arm-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...