[Bug 14354] New: Bad corruption with 2.6.32-rc1 and upwards

http://bugzilla.kernel.org/show_bug.cgi?id=14354

           Summary: Bad corruption with 2.6.32-rc1 and upwards
           Product: File System
           Version: 2.5
          Platform: All
        OS/Version: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: ext4
        AssignedTo: fs_ext4@...
        ReportedBy: zecke@...
        Regression: No


I don't know how to reproduce but I'm seeing the following issues with
2.6.32-rcX since the "BUG_ON with page buffers has been fixed" on a daily
basis.

- Clean shutdown
- Reboot -> fs turns into ro mode
- Reboot -> fsck. I see blocks assigned to two inodes and many lost+found of
files touched close before the shutdown or during the shutdown. This requires
to go through the 1B, 1C, 1D pass on fsck.


I'm not sure how to reproduce and how to properly report it to be of use for
anyone. I'm seeing ext4 corruption on a daily basis though.

--
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html

http://bugzilla.kernel.org/show_bug.cgi?id=14354


Eric Sandeen <sandeen@...> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |sandeen@...




--- Comment #1 from Eric Sandeen <sandeen@...>  2009-10-09 15:51:36 ---
(In reply to comment #0)
> I don't know how to reproduce but I'm seeing the following issues with
> 2.6.32-rcX since the "BUG_ON with page buffers has been fixed" on a daily
> basis.
>
> - Clean shutdown
> - Reboot -> fs turns into ro mode

When it does this, you should get something in dmesg; can you attach that?

> - Reboot -> fsck. I see blocks assigned to two inodes and many lost+found of
> files touched close before the shutdown or during the shutdown. This requires
> to go through the 1B, 1C, 1D pass on fsck.

please attach the actual e2fsck output.

Thanks,
-Eric

--
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html

http://bugzilla.kernel.org/show_bug.cgi?id=14354





--- Comment #2 from Holger Freyther <zecke@...>  2009-10-09 16:06:46 ---
(In reply to comment #1)
> (In reply to comment #0)
> > I don't know how to reproduce but I'm seeing the following issues with
> > 2.6.32-rcX since the "BUG_ON with page buffers has been fixed" on a daily
> > basis.
> >
> > - Clean shutdown
> > - Reboot -> fs turns into ro mode
>
> When it does this, you should get something in dmesg; can you attach that?

I will try, it is a bit difficult as the distro is not booting up to a login
prompt in this case. I will try hard.

>
> > - Reboot -> fsck. I see blocks assigned to two inodes and many lost+found of
> > files touched close before the shutdown or during the shutdown. This requires
> > to go through the 1B, 1C, 1D pass on fsck.
>
> please attach the actual e2fsck output.

Do you have a better idea than using fsck -y / > /boot/fsck.output for keeping
the log?

--
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html

http://bugzilla.kernel.org/show_bug.cgi?id=14354





--- Comment #3 from Eric Sandeen <sandeen@...>  2009-10-09 16:44:47 ---
(In reply to comment #2)

> > When it does this, you should get something in dmesg; can you attach that?
>
> I will try, it is a bit difficult as the distro is not booting up to a login
> prompt in this case. I will try hard.

if you see it on the screen just a general idea or even a photo is fine :)
(may need to switch consoles)

> > please attach the actual e2fsck output.
>
> Do you have a better idea than using fsck -y / > /boot/fsck.output for keeping
> the log?

hopefully that works fine :)

Thanks,
-eric

--
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html

http://bugzilla.kernel.org/show_bug.cgi?id=14354


Theodore Tso <tytso@...> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |tytso@...




--- Comment #4 from Theodore Tso <tytso@...>  2009-10-09 16:50:35 ---
Can you also attach the output of dmesg, so we can see what kind of device you
are using for your root filesystem, and what sort of boot messages are being
emitted by the kernel?   This could very well be a device driver problem.

Also, what distribution are you using?

--
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html

http://bugzilla.kernel.org/show_bug.cgi?id=14354


Rafael J. Wysocki <rjw@...> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |rjw@...
     Kernel Version|                            |2.6.32-rc1
             Blocks|                            |14230
         Regression|No                          |Yes




--
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html

http://bugzilla.kernel.org/show_bug.cgi?id=14354


Alexey Fisher <bug-track@...> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |bug-track@...




--- Comment #5 from Alexey Fisher <bug-track@...>  2009-10-10 07:32:53 ---
I have same issue three of my systems. This is no hardware issue. I will try to
reproduce it on virtual system. Any tips how to make it easy?

--
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html

http://bugzilla.kernel.org/show_bug.cgi?id=14354





--- Comment #6 from Alexey Fisher <bug-track@...>  2009-10-10 16:48:58 ---
Ok, i think i know now how it's happed. I nneded some time to play to reproduce
it. There is _no_difference_ if you just boot 2.6.32 kernel run fsck and then
boot 2.6.31 kernel and run fsck again - nothing will happen.

here is the way i reproduced it:
1. crash (reset or poweroff) 2.6.32-rc3 kernel,
2. start again in 2.6.32-rc3
3. run fsck it well looks clean.. but some progrums will lost it setting.

4. reboot and start with 2.6.31 kernel
5. run fsck and this will find that ext4 is brocken.
6. after fsck fix it.. many fils will be lost.

--
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html

http://bugzilla.kernel.org/show_bug.cgi?id=14354





--- Comment #7 from Alexey Fisher <bug-track@...>  2009-10-10 16:50:59 ---
the guestion is why fsck think the partition is clean if it use 2.6.32 and if
you force fsck on 2.6.31 it will "fix" it even ther was new files on broken
bloks?

--
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html

http://bugzilla.kernel.org/show_bug.cgi?id=14354





--- Comment #8 from Alexey Fisher <bug-track@...>  2009-10-10 17:00:08 ---
i use e2fsprogs version 1.41.9 -1ubuntu1

--
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html

http://bugzilla.kernel.org/show_bug.cgi?id=14354





--- Comment #9 from Alexey Fisher <bug-track@...>  2009-10-10 17:04:23 ---
Created an attachment (id=23333)
 --> (http://bugzilla.kernel.org/attachment.cgi?id=23333)
fsck log

This log i get after steps described in my message before. This time i lost
just some settings on compiz, evolution and miro ... Thank good i make bakups.

--
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html

http://bugzilla.kernel.org/show_bug.cgi?id=14354





--- Comment #10 from Theodore Tso <tytso@...>  2009-10-10 19:54:58 ---
>1. crash (reset or poweroff) 2.6.32-rc3 kernel,
>2. start again in 2.6.32-rc3
>3. run fsck it well looks clean.. but some progrums will lost it setting.

Are you running fsck with the -f option, or not?  

>4. reboot and start with 2.6.31 kernel

This is a clean shutdown or a crash?

> 5. run fsck and this will find that ext4 is brocken.

Again, is this an fsck -f (forced fsck), or just a normal fsck?  What arguments
are you using to fsck each time?

--
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html

http://bugzilla.kernel.org/show_bug.cgi?id=14354





--- Comment #11 from Theodore Tso <tytso@...>  2009-10-11 01:26:15 ---
One more question; is this a completely stock 2.6.32-rcX kernel, or is this one
with special patches from Ubuntu?  If so, can you give me a pointer to the
Ubuntu git tree and the commit ID that was used?

--
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html

http://bugzilla.kernel.org/show_bug.cgi?id=14354





--- Comment #12 from Theodore Tso <tytso@...>  2009-10-11 02:03:12 ---
When the file system gets remounted read-only, can you please send the output
of the "dmesg" command?   If you don't have an extra file system where you can
save the dmesg output, please do "dmesg | grep -i ext4" and copy down what you
see.  Thanks!!

--
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html

http://bugzilla.kernel.org/show_bug.cgi?id=14354





--- Comment #13 from Alexey Fisher <bug-track@...>  2009-10-11 12:31:21 ---
Now i tryed to make this test clean as possible.

1. start pc and after some time reset it or poweroff
2.after restart start kernel with option break=mount. it will not mount root fs
so use initrd instead.
3. mount root partition manually in reed only mode.
4. chroot
5. fsck -y /dev/root >> fsck.log
6. fsck -y -f /dev/root >> fsck.log

on both kernel after krasch "fsck -y" will return:
=============================================================
fsck from util-linux-ng 2.16
/dev/mapper/zwerg_buntu-root_one: clean, 266498/1220608 files, 2444502/4882432
blocks

dmesg on mount will say this (looks like both kernel ak the same way):
=============================================================
[   32.797149] EXT3-fs: dm-0: couldn't mount because of unsupported optional
features (240).
[   32.808599] EXT4-fs (dm-0): INFO: recovery required on readonly filesystem
[   32.809755] EXT4-fs (dm-0): write access will be enabled during recovery
[   32.823038] EXT4-fs (dm-0): barriers enabled
[   33.014768] kjournald2 starting: pid 1166, dev dm-0:8, commit interval 5
seconds
[   33.014792] EXT4-fs (dm-0): delayed allocation enabled
[   33.014794] EXT4-fs: file extents enabled
[   33.014937] EXT4-fs: mballoc enabled
[   33.014954] EXT4-fs (dm-0): orphan cleanup on readonly fs
[   33.014958] EXT4-fs (dm-0): ext4_orphan_cleanup: deleting unreferenced inode
131262
[   33.014994] EXT4-fs (dm-0): ext4_orphan_cleanup: deleting unreferenced inode
131238
[   33.015004] EXT4-fs (dm-0): ext4_orphan_cleanup: deleting unreferenced inode
131164
[   33.015014] EXT4-fs (dm-0): ext4_orphan_cleanup: deleting unreferenced inode
131161
[   33.015023] EXT4-fs (dm-0): ext4_orphan_cleanup: deleting unreferenced inode
131137
[   33.015032] EXT4-fs (dm-0): ext4_orphan_cleanup: deleting unreferenced inode
131119
[   33.015041] EXT4-fs (dm-0): 6 orphan inodes deleted
[   33.015042] EXT4-fs (dm-0): recovery complete
[   33.397102] EXT4-fs (dm-0): mounted filesystem with ordered data mode

if force fsck "fsck -y -f .." on 2.6.31 kernel:
===========================================================
/dev/mapper/zwerg_buntu-root_one: clean, 266499/1220608 files, 2443934/4882432
blocks
fsck from util-linux-ng 2.16
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/mapper/zwerg_buntu-root_one: 266499/1220608 files (0.7% non-contiguous),
2443934/4882432 blocks

on 2.6.32-rX:
===========================================================
/dev/mapper/zwerg_buntu-root_one: clean, 266474/1220608 files, 2444555/4882432
blocks
fsck from util-linux-ng 2.16
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Entry 'AB08-118C' in /media (32769) has deleted/unused inode 123461.  Clear?
yes

Entry 'mtab' in /var/lib/DeviceKit-disks (57430) has deleted/unused inode 393.
Clear? yes

Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Unattached inode 407
Connect to /lost+found? yes

Inode 407 ref count is 2, should be 1.  Fix? yes

Inode 32769 ref count is 7, should be 6.  Fix? yes

Pass 5: Checking group summary information
Block bitmap differences:  -1606658 -1606772
Fix? yes

Free blocks count wrong for group #0 (9591, counted=9592).
Fix? yes

Free blocks count wrong (2437877, counted=2437878).
Fix? yes

Free inodes count wrong for group #15 (6623, counted=6624).
Fix? yes

Directories count wrong for group #15 (791, counted=790).
Fix? yes

Free inodes count wrong (954134, counted=954135).
Fix? yes


/dev/mapper/zwerg_buntu-root_one: ***** FILE SYSTEM WAS MODIFIED *****
/dev/mapper/zwerg_buntu-root_one: ***** REBOOT LINUX *****
/dev/mapper/zwerg_buntu-root_one: 266473/1220608 files (0.7% non-contiguous),
2444554/4882432 blocks

I i didn't get to reproduce it on kvm-qemu.

--
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html

http://bugzilla.kernel.org/show_bug.cgi?id=14354





--- Comment #14 from Theodore Tso <tytso@...>  2009-10-11 19:07:10 ---
Holger, could you try running same test and see if you get similar results?

Alexey, when you say that the dmesg was the same for both kernels, was it
_identical_?   Was it always the same number of orphaned inodes which were
deleted, and were the inode numbers always the same?  And you're sure you
didn't see anything like:

EXT4-fs (dm-0): error: ext4_fill_super: journal transaction 4641626 is corrupt
EXT4-fs (dm-0: Mounting filesystem read-only

In Holger's report, he mentions that after the first reboot, the file system
turns read-only.   Holger, what do you see in your system logs after each
reboot, in particular before the filesystem gets mounted or remounted
read-only?

--
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html

http://bugzilla.kernel.org/show_bug.cgi?id=14354





--- Comment #15 from Alexey Fisher <bug-track@...>  2009-10-11 21:45:34 ---
Yes, i had read-only problem too, but i can't reproduce it now. I had some
trouble to mount system and fsck it, some important files was corrupt. Suddenly
all i wanted is to make it work again.. so there is not dmesg after it.
On my major PC i used debsums to find broken files and reinstall this. On my
laptop i didn't had so mach luck, the packagebase was corrupted so i
reformatted and reinstall the system. I can try to go back to 2.6.32-rcX and
use it until next crush, but i need to know what to do, to extrakt all
information will need for this bug. Thirst good idea will be probably remote
syslog server.

--
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html

http://bugzilla.kernel.org/show_bug.cgi?id=14354





--- Comment #16 from Theodore Tso <tytso@...>  2009-10-11 23:01:06 ---
I've been trying to recreate this failure mode, and haven't had any luck.   It
looks like you are using LVM, so a really good thing to do is to use the
e2croncheck script right after you reboot and login to the system.  If it
reports a clean filesystem, no worries.  If it doesn't, then it would really be
good to snapshot the output of dmesg to /tmp (I assume you have /tmp mounted as
tmpfs) and e-mail it someplace safe, and e2croncheck can also be configured to
e-mail the e2fsck output someplace safe as well.

I'll note that that what Holger and Alexey are seeing are somewhat different.
Holger seems to be seeing a problem after a clean shutdown after re-installing
glibc from a build directory.  That would imply orphaned inode handling getting
screwed up some how, but I haven't been able to reproduce this on my systems.  
Alexey is seeing a problem after a crash/power failure, which implies some sort
of issue with journal replay.  One change that we did make between 2.6.31 and
2.6.32 is that we enable journal checksums by default.  In theory if the hard
drive is ignoring barriers, or is otherwise screwing up the journal I/O, we
could get a journal checksum error that would abort the journal playback.  That
in theory could explain Alexey's symptoms, but he didn't report a journal
checksum error message.

So I really don't know what's going on right now.

--
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html

http://bugzilla.kernel.org/show_bug.cgi?id=14354





--- Comment #17 from Theodore Tso <tytso@...>  2009-10-12 00:02:59 ---
Created an attachment (id=23350)
 --> (http://bugzilla.kernel.org/attachment.cgi?id=23350)
E2croncheck shell script

--
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html

http://bugzilla.kernel.org/show_bug.cgi?id=14354





--- Comment #18 from Theodore Tso <tytso@...>  2009-10-12 02:18:41 ---
Created an attachment (id=23353)
 --> (http://bugzilla.kernel.org/attachment.cgi?id=23353)
jbd2 debugging patch

The jbd2-checksum-debugging patch adds two options which can be added to the
boot line.   The first, jbd2.xsum_debug=1, will print two additional kernel
messages when replaying a journal that will help debug checksums.   The second,
jbd2.ignore_xsum=1, will cause jbd2 to ignore any checksum errors.

If you can come up with a repeatable test case, it would be useful to apply
this patch and then add "jbd2.xsum_debug=1 jbd2.ignore_xsum=1" to the boot
line.   Then let me know (a) if the problem goes away when you add the boot
command-line options, and (b) give me the dmesg output either way.

Thanks!

--
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html