kernel crash with damaged XFS

[Michael Monnerie-5]

I had that during the last days on my system. It kept on running despite
this error. I xfs_repair'ed the fs again and now it's working.
This is the same server I talked about with Eric last time, where he
improved xfs_repair and that repaired my fs. I had to run
xfs_repair several times again until all errors were solved,
and now nothing is reported anymore.
The server had no crash or whatever since the last repair, so I wonder
where these corruptions come from.

Nov  2 00:01:12 orion.i.zmi.at kernel: Filesystem "dm-0": corrupt inode 3858839593 ((a)extents = 7).  Unmount and run xfs_repair.
Nov  2 00:01:12 orion.i.zmi.at kernel: Pid: 18910, comm: find Tainted: G          2.6.27.29-0.1-xen #1
Nov  2 00:01:12 orion.i.zmi.at kernel:
Nov  2 00:01:12 orion.i.zmi.at kernel: Call Trace:
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff8020c597>] show_trace_log_lvl+0x41/0x58
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff80464a12>] dump_stack+0x69/0x6f
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffffa033cbdb>] xfs_iformat_extents+0xc9/0x1c4 [xfs]
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffffa033d136>] xfs_iformat+0x2b0/0x3f7 [xfs]
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffffa033d364>] xfs_iread+0xe7/0x1eb [xfs]
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffffa0338910>] xfs_iget_core+0x3a5/0x63a [xfs]
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffffa0338c87>] xfs_iget+0xe2/0x187 [xfs]
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffffa0352912>] xfs_lookup+0x79/0xa5 [xfs]
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffffa035b53f>] xfs_vn_lookup+0x3c/0x78 [xfs]
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff802a603b>] real_lookup+0x7e/0x10f
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff802a612f>] do_lookup+0x63/0xb6
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff802a87aa>] __link_path_walk+0x9f4/0xe58
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff802a8dd9>] path_walk+0x5e/0xba
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff802a8f97>] do_path_lookup+0x162/0x1b9
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff802a9930>] user_path_at+0x48/0x79
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff802a1e85>] vfs_lstat_fd+0x15/0x41
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff802a1f88>] sys_newfstatat+0x22/0x43
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff8020b3b8>] system_call_fastpath+0x16/0x1b
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<00007f265cb0f4de>] 0x7f265cb0f4de


mfg zmi
--
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4

_______________________________________________
xfs mailing list
xfs@...
http://oss.sgi.com/mailman/listinfo/xfs

[Eric Sandeen-3]

On Nov 6, 2009, at 3:17 AM, Michael Monnerie <michael.monnerie@...
 > wrote:

> I had that during the last days on my system. It kept on running  
> despite
> this error. I xfs_repair'ed the fs again and now it's working.
> This is the same server I talked about with Eric last time, where he
> improved xfs_repair and that repaired my fs. I had to run
> xfs_repair several times again until all errors were solved,
> and now nothing is reported anymore.
> The server had no crash or whatever since the last repair, so I wonder
> where these corruptions come from.
>
> Nov  2 00:01:12 orion.i.zmi.at kernel: Filesystem "dm-0": corrupt  
> inode 3858839593 ((a)extents = 7).  Unmount and run xfs_repair.

Just FWIW this is not a crash, it is XFS properly handling corrruption  
it encountered.  The corruption could be due to a bug in XFS or other  
code, or a hardware problem.  Does repair fix it this time?

- Eric

> Nov  2 00:01:12 orion.i.zmi.at kernel: Pid: 18910, comm: find  
> Tainted: G          2.6.27.29-0.1-xen #1
> Nov  2 00:01:12 orion.i.zmi.at kernel:
> Nov  2 00:01:12 orion.i.zmi.at kernel: Call Trace:
> Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff8020c597>]  
> show_trace_log_lvl+0x41/0x58
> Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff80464a12>]  
> dump_stack+0x69/0x6f
> Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffffa033cbdb>]  
> xfs_iformat_extents+0xc9/0x1c4 [xfs]
> Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffffa033d136>]  
> xfs_iformat+0x2b0/0x3f7 [xfs]
> Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffffa033d364>]  
> xfs_iread+0xe7/0x1eb [xfs]
> Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffffa0338910>]  
> xfs_iget_core+0x3a5/0x63a [xfs]
> Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffffa0338c87>] xfs_iget
> +0xe2/0x187 [xfs]
> Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffffa0352912>]  
> xfs_lookup+0x79/0xa5 [xfs]
> Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffffa035b53f>]  
> xfs_vn_lookup+0x3c/0x78 [xfs]
> Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff802a603b>]  
> real_lookup+0x7e/0x10f
> Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff802a612f>]  
> do_lookup+0x63/0xb6
> Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff802a87aa>]  
> __link_path_walk+0x9f4/0xe58
> Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff802a8dd9>]  
> path_walk+0x5e/0xba
> Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff802a8f97>]  
> do_path_lookup+0x162/0x1b9
> Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff802a9930>]  
> user_path_at+0x48/0x79
> Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff802a1e85>]  
> vfs_lstat_fd+0x15/0x41
> Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff802a1f88>]  
> sys_newfstatat+0x22/0x43
> Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff8020b3b8>]  
> system_call_fastpath+0x16/0x1b
> Nov  2 00:01:12 orion.i.zmi.at kernel:  [<00007f265cb0f4de>]  
> 0x7f265cb0f4de
>
>
> mfg zmi
> --
> // Michael Monnerie, Ing.BSc    -----      http://it-management.at
> // Tel: 0660 / 415 65 31                      .network.your.ideas.
> // PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
> // Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
> // Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4
>
> _______________________________________________
> xfs mailing list
> xfs@...
> http://oss.sgi.com/mailman/listinfo/xfs
>

_______________________________________________
xfs mailing list
xfs@...
http://oss.sgi.com/mailman/listinfo/xfs

[Michael Monnerie-5]

On Freitag 06 November 2009 Eric Sandeen wrote:

> > I had that during the last days on my system. It kept on running  
> > despite
> > this error. I xfs_repair'ed the fs again and now it's working.
> > This is the same server I talked about with Eric last time, where
> > he improved xfs_repair and that repaired my fs. I had to run
> > xfs_repair several times again until all errors were solved,
> > and now nothing is reported anymore.
> > The server had no crash or whatever since the last repair, so I
> > wonder where these corruptions come from.
> >
> > Nov  2 00:01:12 orion.i.zmi.at kernel: Filesystem "dm-0": corrupt  
> > inode 3858839593 ((a)extents = 7).  Unmount and run xfs_repair.
>
> Just FWIW this is not a crash, it is XFS properly handling
> corrruption   it encountered.  The corruption could be due to a bug
> in XFS or other code, or a hardware problem.  Does repair fix it this
> time?

Yes Eric, that's what I wrote but I see it can be misinterpreted as
belonging to the old case. I meant the actual problem when saying:

I had to run xfs_repair several times again until all errors were
solved, and now nothing is reported anymore. The server had no crash or
whatever since the last repair, so I wonder where these corruptions come
from.

Seems to be time to make a memory check. I have no other idea where the
problem could come from.

mfg zmi
--
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4

_______________________________________________
xfs mailing list
xfs@...
http://oss.sgi.com/mailman/listinfo/xfs

[Justin Piszcz]

On Sat, 7 Nov 2009, Michael Monnerie wrote:

> On Freitag 06 November 2009 Eric Sandeen wrote:
>>> I had that during the last days on my system. It kept on running  
>>> despite
>>> this error. I xfs_repair'ed the fs again and now it's working.
>>> This is the same server I talked about with Eric last time, where
>>> he improved xfs_repair and that repaired my fs. I had to run
>>> xfs_repair several times again until all errors were solved,
>>> and now nothing is reported anymore.
>>> The server had no crash or whatever since the last repair, so I
>>> wonder where these corruptions come from.
>>>
>>> Nov  2 00:01:12 orion.i.zmi.at kernel: Filesystem "dm-0": corrupt  
>>> inode 3858839593 ((a)extents = 7).  Unmount and run xfs_repair.
>>
>> Just FWIW this is not a crash, it is XFS properly handling
>> corrruption   it encountered.  The corruption could be due to a bug
>> in XFS or other code, or a hardware problem.  Does repair fix it this
>> time?
>
> Yes Eric, that's what I wrote but I see it can be misinterpreted as
> belonging to the old case. I meant the actual problem when saying:
>
> I had to run xfs_repair several times again until all errors were
> solved, and now nothing is reported anymore. The server had no crash or
> whatever since the last repair, so I wonder where these corruptions come
> from.
>
> Seems to be time to make a memory check. I have no other idea where the
> problem could come from.
>
> mfg zmi
> --
> // Michael Monnerie, Ing.BSc    -----      http://it-management.at
> // Tel: 0660 / 415 65 31                      .network.your.ideas.
> // PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
> // Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
> // Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4
>
> _______________________________________________
> xfs mailing list
> xfs@...
> http://oss.sgi.com/mailman/listinfo/xfs
>

memtest is a good idea
also run a short & long test on each hdd and show smartctl -a of each
device to see if any may be failing

_______________________________________________
xfs mailing list
xfs@...
http://oss.sgi.com/mailman/listinfo/xfs