GPT boot with ZFS RAIDZ "ZFS: i/o error - all block copies unavailable"

View: New views
20 Messages — Rating Filter:   Alert me  
< Prev | 1 - 2 | Next >

GPT boot with ZFS RAIDZ "ZFS: i/o error - all block copies unavailable"

by Radek Valášek-2 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Hi,

I want to ask if there is something new in adding support to
gptzfsboot/zfsboot for reading gang-blocks?

 From Sun's docs:

Gang blocks

When there is not enough contiguous space to write a complete block, the ZIO
pipeline will break the I/O up into smaller 'gang blocks' which can later be
assembled transparently to appear as complete blocks.

Everything works fine for me, until I rewrite kernel/world after system
upgrade to latest one (releng_8). After this am I no longer able to boot
from zfs raidz1 pool with following messages:

 >/ ZFS: i/o error - all block copies unavailable
/>/ ZFS: can't read MOS
/>/ ZFS: unexpected object set type lld
/>/ ZFS: unexpected object set type lld
/>/
/>/ FreeBSD/i386 boot
/>/ Default: z:/boot/kernel/kernel
/>/ boot:
/>/ ZFS: unexpected object set type lld
/>/
/>/ FreeBSD/i386 boot
/>/ Default: tank:/boot/kernel/kernel
/>/ boot:
//
/I presume it's the same issue as talked in june-2009 current mailing
list
http://lists.freebsd.org/pipermail/freebsd-current/2009-June/008589.html

Any success in that matter?

Thnx for answer.

vaLin
_______________________________________________
freebsd-fs@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-unsubscribe@..."

Re: GPT boot with ZFS RAIDZ "ZFS: i/o error - all block copies unavailable"

by Robert Noland :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Thu, 2009-10-15 at 14:08 +0200, Radek Valášek wrote:
> Hi,
>
> I want to ask if there is something new in adding support to
> gptzfsboot/zfsboot for reading gang-blocks?

I've been thinking of trying to fix this, but haven't really come up
with a repeatable way to test it.  I might be able to come up with at
least a hack to allow booting in the short term, but if you can try this
patch so that we can verify that the issue is indeed gang blocks.  This
doesn't fix anything yet, but it should report when it finds a gang
block.  I know that it is tricky to test when you can't boot, but if you
can apply this patch and reinstall gptzfsboot, it should tell us for
sure that gang blocks are the issue.  I assume that you have a partition
layout something like mine:

balrog% gpart show
=>        34  1953525101  ada0  GPT  (932G)
          34         128     1  freebsd-boot  (64K)
         162     8388608     2  freebsd-swap  (4.0G)
     8388770  1945136365     3  freebsd-zfs  (928G)

If so, all you should need to do is get this built and then:

#gpart bootcode -p /boot/gptzfsboot -i 1 ada0

substituting appropriate partition index and device info obviously.

robert.

>  From Sun's docs:
>
> Gang blocks
>
> When there is not enough contiguous space to write a complete block, the ZIO
> pipeline will break the I/O up into smaller 'gang blocks' which can later be
> assembled transparently to appear as complete blocks.
>
> Everything works fine for me, until I rewrite kernel/world after system
> upgrade to latest one (releng_8). After this am I no longer able to boot
> from zfs raidz1 pool with following messages:
>
>  >/ ZFS: i/o error - all block copies unavailable
> />/ ZFS: can't read MOS
> />/ ZFS: unexpected object set type lld
> />/ ZFS: unexpected object set type lld
> />/
> />/ FreeBSD/i386 boot
> />/ Default: z:/boot/kernel/kernel
> />/ boot:
> />/ ZFS: unexpected object set type lld
> />/
> />/ FreeBSD/i386 boot
> />/ Default: tank:/boot/kernel/kernel
> />/ boot:
> //
> /I presume it's the same issue as talked in june-2009 current mailing
> list
> http://lists.freebsd.org/pipermail/freebsd-current/2009-June/008589.html
>
> Any success in that matter?
>
> Thnx for answer.
>
> vaLin
> _______________________________________________
> freebsd-current@... mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe@..."
--
Robert Noland <rnoland@...>
FreeBSD

[zfs-report-gb.patch]

diff --git a/sys/boot/zfs/zfsimpl.c b/sys/boot/zfs/zfsimpl.c
index ff567a4..a2893bf 100644
--- a/sys/boot/zfs/zfsimpl.c
+++ b/sys/boot/zfs/zfsimpl.c
@@ -920,6 +920,11 @@ zio_read(spa_t *spa, const blkptr_t *bp, void *buf)
  if (!dva->dva_word[0] && !dva->dva_word[1])
  continue;
 
+ if (DVA_GET_GANG(dva)) {
+ printf("ZFS: i/o error - gang block unimplemented!\n");
+ continue;
+ }
+
  vdevid = DVA_GET_VDEV(dva);
  offset = DVA_GET_OFFSET(dva);
  STAILQ_FOREACH(vdev, &spa->spa_vdevs, v_childlink)


_______________________________________________
freebsd-fs@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-unsubscribe@..."

Re: GPT boot with ZFS RAIDZ "ZFS: i/o error - all block copies unavailable"

by Robert Noland :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Thu, 2009-10-15 at 14:08 +0200, Radek Valášek wrote:
> Hi,
>
> I want to ask if there is something new in adding support to
> gptzfsboot/zfsboot for reading gang-blocks?

Ok, I can't figure out any way to test this... beyond the fact that it
builds and doesn't break my currently working setup.  Can you give this
a try?  It should still report if it finds gang blocks, but hopefully
now will read them as well.

robert.

>  From Sun's docs:
>
> Gang blocks
>
> When there is not enough contiguous space to write a complete block, the ZIO
> pipeline will break the I/O up into smaller 'gang blocks' which can later be
> assembled transparently to appear as complete blocks.
>
> Everything works fine for me, until I rewrite kernel/world after system
> upgrade to latest one (releng_8). After this am I no longer able to boot
> from zfs raidz1 pool with following messages:
>
>  >/ ZFS: i/o error - all block copies unavailable
> />/ ZFS: can't read MOS
> />/ ZFS: unexpected object set type lld
> />/ ZFS: unexpected object set type lld
> />/
> />/ FreeBSD/i386 boot
> />/ Default: z:/boot/kernel/kernel
> />/ boot:
> />/ ZFS: unexpected object set type lld
> />/
> />/ FreeBSD/i386 boot
> />/ Default: tank:/boot/kernel/kernel
> />/ boot:
> //
> /I presume it's the same issue as talked in june-2009 current mailing
> list
> http://lists.freebsd.org/pipermail/freebsd-current/2009-June/008589.html
>
> Any success in that matter?
>
> Thnx for answer.
>
> vaLin
> _______________________________________________
> freebsd-current@... mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe@..."
--
Robert Noland <rnoland@...>
FreeBSD

[zfs-gang-block.patch]

diff --git a/sys/boot/zfs/zfsimpl.c b/sys/boot/zfs/zfsimpl.c
index ff567a4..6a18b44 100644
--- a/sys/boot/zfs/zfsimpl.c
+++ b/sys/boot/zfs/zfsimpl.c
@@ -53,6 +53,8 @@ static char *zfs_temp_buf, *zfs_temp_end, *zfs_temp_ptr;
 
 #define TEMP_SIZE (1*SPA_MAXBLOCKSIZE)
 
+static int zio_read(spa_t *spa, const blkptr_t *bp, void *buf);
+
 static void
 zfs_init(void)
 {
@@ -897,6 +899,33 @@ ilog2(int n)
 }
 
 static int
+zio_read_gang(spa_t *spa, const blkptr_t *bp, const dva_t *dva, void *buf)
+{
+ zio_gbh_phys_t zio_gb;
+ vdev_t *vdev;
+ int vdevid;
+ off_t offset;
+ int i;
+
+ vdevid = DVA_GET_VDEV(dva);
+ offset = DVA_GET_OFFSET(dva);
+ STAILQ_FOREACH(vdev, &spa->spa_vdevs, v_childlink)
+ if (vdev->v_id == vdevid)
+ break;
+ if (!vdev || !vdev->v_read)
+ return (EIO);
+ if (vdev->v_read(vdev, bp, &zio_gb, offset, SPA_GANGBLOCKSIZE))
+ return (EIO);
+
+ for (i = 0; i < SPA_GBH_NBLKPTRS; i++) {
+ if (zio_read(spa, &zio_gb.zg_blkptr[i], buf))
+ return (EIO);
+ }
+
+ return (0);
+}
+
+static int
 zio_read(spa_t *spa, const blkptr_t *bp, void *buf)
 {
  int cpfunc = BP_GET_COMPRESS(bp);
@@ -920,20 +949,26 @@ zio_read(spa_t *spa, const blkptr_t *bp, void *buf)
  if (!dva->dva_word[0] && !dva->dva_word[1])
  continue;
 
- vdevid = DVA_GET_VDEV(dva);
- offset = DVA_GET_OFFSET(dva);
- STAILQ_FOREACH(vdev, &spa->spa_vdevs, v_childlink)
- if (vdev->v_id == vdevid)
- break;
- if (!vdev || !vdev->v_read)
- continue;
- if (vdev->v_read(vdev, bp, pbuf, offset, psize))
- continue;
+ if (DVA_GET_GANG(dva)) {
+ printf("ZFS: gang block detected!\n");
+ if (zio_read_gang(spa, bp, dva, buf))
+ return (EIO);
+ } else {
+ vdevid = DVA_GET_VDEV(dva);
+ offset = DVA_GET_OFFSET(dva);
+ STAILQ_FOREACH(vdev, &spa->spa_vdevs, v_childlink)
+ if (vdev->v_id == vdevid)
+ break;
+ if (!vdev || !vdev->v_read)
+ continue;
+ if (vdev->v_read(vdev, bp, pbuf, offset, psize))
+ continue;
 
- if (cpfunc != ZIO_COMPRESS_OFF) {
- if (zio_decompress_data(cpfunc, pbuf, psize,
- buf, lsize))
- return (EIO);
+ if (cpfunc != ZIO_COMPRESS_OFF) {
+ if (zio_decompress_data(cpfunc, pbuf, psize,
+    buf, lsize))
+ return (EIO);
+ }
  }
 
  return (0);
diff --git a/sys/cddl/boot/zfs/zfsimpl.h b/sys/cddl/boot/zfs/zfsimpl.h
index a0b7b72..688bb5c 100644
--- a/sys/cddl/boot/zfs/zfsimpl.h
+++ b/sys/cddl/boot/zfs/zfsimpl.h
@@ -374,6 +374,24 @@ typedef struct vdev_label {
 #define VDEV_LABEL_END_SIZE (2 * sizeof (vdev_label_t))
 #define VDEV_LABELS 4
 
+/*
+ * Gang block headers are self-checksumming and contain an array
+ * of block pointers.
+ */
+#define SPA_GANGBLOCKSIZE SPA_MINBLOCKSIZE
+#define SPA_GBH_NBLKPTRS ((SPA_GANGBLOCKSIZE - \
+ sizeof (zio_block_tail_t)) / sizeof (blkptr_t))
+#define SPA_GBH_FILLER ((SPA_GANGBLOCKSIZE - \
+ sizeof (zio_block_tail_t) - \
+ (SPA_GBH_NBLKPTRS * sizeof (blkptr_t))) /\
+ sizeof (uint64_t))
+
+typedef struct zio_gbh {
+ blkptr_t zg_blkptr[SPA_GBH_NBLKPTRS];
+ uint64_t zg_filler[SPA_GBH_FILLER];
+ zio_block_tail_t zg_tail;
+} zio_gbh_phys_t;
+
 enum zio_checksum {
  ZIO_CHECKSUM_INHERIT = 0,
  ZIO_CHECKSUM_ON,


_______________________________________________
freebsd-fs@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-unsubscribe@..."

Re: GPT boot with ZFS RAIDZ "ZFS: i/o error - all block copies unavailable"

by Radek Valášek-2 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Robert Noland napsal(a):

> On Thu, 2009-10-15 at 14:08 +0200, Radek Valášek wrote:
>  
>> Hi,
>>
>> I want to ask if there is something new in adding support to
>> gptzfsboot/zfsboot for reading gang-blocks?
>>    
>
> Ok, I can't figure out any way to test this... beyond the fact that it
> builds and doesn't break my currently working setup.  Can you give this
> a try?  It should still report if it finds gang blocks, but hopefully
> now will read them as well.
>
> robert.
>
>  
Big thanks for the patches Robert, I will definitely test them as soon
as possible (tomorrow) and report the results immediately to list. I can
repeat this issue probably at any time (up to cca 30 times tested with
the same result), so don't bother about the broken booting, I'm prepared
for it...

vaLin

>>  From Sun's docs:
>>
>> Gang blocks
>>
>> When there is not enough contiguous space to write a complete block, the ZIO
>> pipeline will break the I/O up into smaller 'gang blocks' which can later be
>> assembled transparently to appear as complete blocks.
>>
>> Everything works fine for me, until I rewrite kernel/world after system
>> upgrade to latest one (releng_8). After this am I no longer able to boot
>> from zfs raidz1 pool with following messages:
>>
>>  >/ ZFS: i/o error - all block copies unavailable
>> />/ ZFS: can't read MOS
>> />/ ZFS: unexpected object set type lld
>> />/ ZFS: unexpected object set type lld
>> />/
>> />/ FreeBSD/i386 boot
>> />/ Default: z:/boot/kernel/kernel
>> />/ boot:
>> />/ ZFS: unexpected object set type lld
>> />/
>> />/ FreeBSD/i386 boot
>> />/ Default: tank:/boot/kernel/kernel
>> />/ boot:
>> //
>> /I presume it's the same issue as talked in june-2009 current mailing
>> list
>> http://lists.freebsd.org/pipermail/freebsd-current/2009-June/008589.html
>>
>> Any success in that matter?
>>
>> Thnx for answer.
>>
>> vaLin
>> _______________________________________________
>> freebsd-current@... mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
>> To unsubscribe, send any mail to "freebsd-current-unsubscribe@..."
>>    

_______________________________________________
freebsd-fs@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-unsubscribe@..."

Re: GPT boot with ZFS RAIDZ "ZFS: i/o error - all block copies unavailable"

by Robert Noland :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Thu, 2009-10-15 at 21:37 +0200, Radek Valášek wrote:
> Robert Noland napsal(a):
> > On Thu, 2009-10-15 at 14:08 +0200, Radek Valášek wrote:
> >  
> >> Hi,
> >>
> >> I want to ask if there is something new in adding support to
> >> gptzfsboot/zfsboot for reading gang-blocks?
> >>    

I think that the gang block patch will work, though still haven't gotten
it tested.  However, I'm fairly confident that the issue is not gang
block related.  Right now, I have setup a disk like this:

=>        34  1953525101  ada1  GPT  (932G)
          34         128     1  freebsd-boot  (64K)
         162     8388608     2  freebsd-swap  (4.0G)
     8388770   648019968     3  freebsd-zfs  (309G)
   656408738   648019968     4  freebsd-zfs  (309G)
  1304428706   648019968     5  freebsd-zfs  (309G)
  1952448674     1076461        - free -  (526M)

Note that this is not a raidz pool right now.  It is just 3 toplevel
partitions setup as a single pool.  I finally have this configuration
working reliably.  At least in this case, the issue is due to all of the
partitions not being probed during early boot and so not being added to
the list of vdevs for the pool.  When zio_read finds a dva that points
to a device it doesn't know about, it gives up and whines.

Can you detail for me how you have everything configured, so that I can
try to replicate it.  gpart show, zpool status and zpool get all <pool>
would be good.  I'm not sure that I have enough spare disks lying around
to do this properly, but maybe I can use virtual disks or something.

robert.

> > Ok, I can't figure out any way to test this... beyond the fact that it
> > builds and doesn't break my currently working setup.  Can you give this
> > a try?  It should still report if it finds gang blocks, but hopefully
> > now will read them as well.
> >
> > robert.
> >
> >  
> Big thanks for the patches Robert, I will definitely test them as soon
> as possible (tomorrow) and report the results immediately to list. I can
> repeat this issue probably at any time (up to cca 30 times tested with
> the same result), so don't bother about the broken booting, I'm prepared
> for it...
>
> vaLin
> >>  From Sun's docs:
> >>
> >> Gang blocks
> >>
> >> When there is not enough contiguous space to write a complete block, the ZIO
> >> pipeline will break the I/O up into smaller 'gang blocks' which can later be
> >> assembled transparently to appear as complete blocks.
> >>
> >> Everything works fine for me, until I rewrite kernel/world after system
> >> upgrade to latest one (releng_8). After this am I no longer able to boot
> >> from zfs raidz1 pool with following messages:
> >>
> >>  >/ ZFS: i/o error - all block copies unavailable
> >> />/ ZFS: can't read MOS
> >> />/ ZFS: unexpected object set type lld
> >> />/ ZFS: unexpected object set type lld
> >> />/
> >> />/ FreeBSD/i386 boot
> >> />/ Default: z:/boot/kernel/kernel
> >> />/ boot:
> >> />/ ZFS: unexpected object set type lld
> >> />/
> >> />/ FreeBSD/i386 boot
> >> />/ Default: tank:/boot/kernel/kernel
> >> />/ boot:
> >> //
> >> /I presume it's the same issue as talked in june-2009 current mailing
> >> list
> >> http://lists.freebsd.org/pipermail/freebsd-current/2009-June/008589.html
> >>
> >> Any success in that matter?
> >>
> >> Thnx for answer.
> >>
> >> vaLin
> >> _______________________________________________
> >> freebsd-current@... mailing list
> >> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> >> To unsubscribe, send any mail to "freebsd-current-unsubscribe@..."
> >>    
>
--
Robert Noland <rnoland@...>
FreeBSD

_______________________________________________
freebsd-fs@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-unsubscribe@..."

Re: GPT boot with ZFS RAIDZ "ZFS: i/o error - all block copies unavailable"

by Radek Valášek-2 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Robert Noland napsal(a):

> On Thu, 2009-10-15 at 21:37 +0200, Radek Valášek wrote:
>  
>> Robert Noland napsal(a):
>>    
>>> On Thu, 2009-10-15 at 14:08 +0200, Radek Valášek wrote:
>>>  
>>>      
>>>> Hi,
>>>>
>>>> I want to ask if there is something new in adding support to
>>>> gptzfsboot/zfsboot for reading gang-blocks?
>>>>    
>>>>        
>
> I think that the gang block patch will work, though still haven't gotten
> it tested.  However, I'm fairly confident that the issue is not gang
> block related.  Right now, I have setup a disk like this:
>
> =>        34  1953525101  ada1  GPT  (932G)
>           34         128     1  freebsd-boot  (64K)
>          162     8388608     2  freebsd-swap  (4.0G)
>      8388770   648019968     3  freebsd-zfs  (309G)
>    656408738   648019968     4  freebsd-zfs  (309G)
>   1304428706   648019968     5  freebsd-zfs  (309G)
>   1952448674     1076461        - free -  (526M)
>
> Note that this is not a raidz pool right now.  It is just 3 toplevel
> partitions setup as a single pool.  I finally have this configuration
> working reliably.  At least in this case, the issue is due to all of the
> partitions not being probed during early boot and so not being added to
> the list of vdevs for the pool.  When zio_read finds a dva that points
> to a device it doesn't know about, it gives up and whines.
>
> Can you detail for me how you have everything configured, so that I can
> try to replicate it.  gpart show, zpool status and zpool get all <pool>
> would be good.  I'm not sure that I have enough spare disks lying around
> to do this properly, but maybe I can use virtual disks or something.
>
> robert.
>
>  

Sorry for not responding so long. Here are details you want from me:

# gpart show
=>        34  1953525101  ad6  GPT  (932G)
          34         128    1  freebsd-boot  (64K)
         162  1953524973    2  freebsd-zfs  (932G)

=>        34  1953525101  ad8  GPT  (932G)
          34         128    1  freebsd-boot  (64K)
         162  1953524973    2  freebsd-zfs  (932G)

=>        34  1953525101  ad10  GPT  (932G)
          34         128     1  freebsd-boot  (64K)
         162  1953524973     2  freebsd-zfs  (932G)

=>        34  1953525101  ad12  GPT  (932G)
          34         128     1  freebsd-boot  (64K)
         162  1953524973     2  freebsd-zfs  (932G)

# zpool status
  pool: z
 state: ONLINE
 scrub: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    z           ONLINE       0     0     0
      raidz1    ONLINE       0     0     0
        ad6p2   ONLINE       0     0     0
        ad8p2   ONLINE       0     0     0
        ad10p2  ONLINE       0     0     0
        ad12p2  ONLINE       0     0     0

errors: No known data errors

# zpool get all z
NAME  PROPERTY       VALUE       SOURCE
z     size           3.62T       -
z     used           4.62G       -
z     available      3.62T       -
z     capacity       0%          -
z     altroot        -           default
z     health         ONLINE      -
z     guid           17857007133862981114  -
z     version        13          default
z     bootfs         z/system    local
z     delegation     on          default
z     autoreplace    off         default
z     cachefile      -           default
z     failmode       wait        default
z     listsnapshots  off         default

I've tested your patches but it seems that you're right and it's not
gang related issue. I was able to discover these things on a fully
functional zfs pool (system compiled with your patches):

1, If I overwrite the file /boot/loader.conf (with copy of itself, or
when upgrading kernel/world), next reboot comes with these messages:

BTX loader 1.00  BTX version is 1.02
Consoles: internal video/keyboard
BIOS drive C: is disk0
BIOS drive D: is disk1
BIOS drive E: is disk2
BIOS drive F: is disk3
BIOS 627kB/3405248kB available memory

FreeBSD/i386 bootstrap loader, Revision 1.1
(root@ztest, Thu Oct 22 22:27:22 CEST 2009)
Loading /boot/defaults/loader.conf
ZFS: i/o error - all block copies unavailable
Warning: error reading file /boot/loader.conf

Then I'm still able to boot the system, but I must set the boot
variables included in loader.conf by hand

2, Next I overwrite the file /boot/loader (with copy of itself, or when
upgrading kernel/world) and reboot comes with these messages:

BTX loader 1.00  BTX version is 1.02
Consoles: internal video/keyboard
BIOS drive C: is disk0
BIOS drive D: is disk1
BIOS drive E: is disk2
BIOS drive F: is disk3
BIOS 627kB/3405248kB available memory

FreeBSD/i386 bootstrap loader, Revision 1.1
(root@ztest, Thu Oct 22 22:27:22 CEST 2009)
Loading /boot/defaults/loader.conf
ZFS: i/o error - all block copies unavailable
Warning: error reading file /boot/loader.conf
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
Unable to load a kernel!

After that I'm no longer able to boot the system from zfs pool.

Hope you have some ideas...

vaLin

>>> Ok, I can't figure out any way to test this... beyond the fact that it
>>> builds and doesn't break my currently working setup.  Can you give this
>>> a try?  It should still report if it finds gang blocks, but hopefully
>>> now will read them as well.
>>>
>>> robert.
>>>
>>>  
>>>      
>> Big thanks for the patches Robert, I will definitely test them as soon
>> as possible (tomorrow) and report the results immediately to list. I can
>> repeat this issue probably at any time (up to cca 30 times tested with
>> the same result), so don't bother about the broken booting, I'm prepared
>> for it...
>>
>> vaLin
>>    
>>>>  From Sun's docs:
>>>>
>>>> Gang blocks
>>>>
>>>> When there is not enough contiguous space to write a complete block, the ZIO
>>>> pipeline will break the I/O up into smaller 'gang blocks' which can later be
>>>> assembled transparently to appear as complete blocks.
>>>>
>>>> Everything works fine for me, until I rewrite kernel/world after system
>>>> upgrade to latest one (releng_8). After this am I no longer able to boot
>>>> from zfs raidz1 pool with following messages:
>>>>
>>>>  >/ ZFS: i/o error - all block copies unavailable
>>>> />/ ZFS: can't read MOS
>>>> />/ ZFS: unexpected object set type lld
>>>> />/ ZFS: unexpected object set type lld
>>>> />/
>>>> />/ FreeBSD/i386 boot
>>>> />/ Default: z:/boot/kernel/kernel
>>>> />/ boot:
>>>> />/ ZFS: unexpected object set type lld
>>>> />/
>>>> />/ FreeBSD/i386 boot
>>>> />/ Default: tank:/boot/kernel/kernel
>>>> />/ boot:
>>>> //
>>>> /I presume it's the same issue as talked in june-2009 current mailing
>>>> list
>>>> http://lists.freebsd.org/pipermail/freebsd-current/2009-June/008589.html
>>>>
>>>> Any success in that matter?
>>>>
>>>> Thnx for answer.
>>>>
>>>> vaLin
>>>> _______________________________________________
>>>> freebsd-current@... mailing list
>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
>>>> To unsubscribe, send any mail to "freebsd-current-unsubscribe@..."
>>>>    
>>>>        

_______________________________________________
freebsd-fs@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-unsubscribe@..."

Re: GPT boot with ZFS RAIDZ "ZFS: i/o error - all block copies unavailable"

by Robert Noland :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Sat, 2009-10-24 at 19:44 +0200, Radek Valášek wrote:

> Robert Noland napsal(a):
> > On Thu, 2009-10-15 at 21:37 +0200, Radek Valášek wrote:
> >  
> >> Robert Noland napsal(a):
> >>    
> >>> On Thu, 2009-10-15 at 14:08 +0200, Radek Valášek wrote:
> >>>  
> >>>      
> >>>> Hi,
> >>>>
> >>>> I want to ask if there is something new in adding support to
> >>>> gptzfsboot/zfsboot for reading gang-blocks?
> >>>>    
> >>>>        
> >
> > I think that the gang block patch will work, though still haven't gotten
> > it tested.  However, I'm fairly confident that the issue is not gang
> > block related.  Right now, I have setup a disk like this:
> >
> > =>        34  1953525101  ada1  GPT  (932G)
> >           34         128     1  freebsd-boot  (64K)
> >          162     8388608     2  freebsd-swap  (4.0G)
> >      8388770   648019968     3  freebsd-zfs  (309G)
> >    656408738   648019968     4  freebsd-zfs  (309G)
> >   1304428706   648019968     5  freebsd-zfs  (309G)
> >   1952448674     1076461        - free -  (526M)
> >
> > Note that this is not a raidz pool right now.  It is just 3 toplevel
> > partitions setup as a single pool.  I finally have this configuration
> > working reliably.  At least in this case, the issue is due to all of the
> > partitions not being probed during early boot and so not being added to
> > the list of vdevs for the pool.  When zio_read finds a dva that points
> > to a device it doesn't know about, it gives up and whines.
> >
> > Can you detail for me how you have everything configured, so that I can
> > try to replicate it.  gpart show, zpool status and zpool get all <pool>
> > would be good.  I'm not sure that I have enough spare disks lying around
> > to do this properly, but maybe I can use virtual disks or something.
> >
> > robert.
> >
> >  
>
> Sorry for not responding so long. Here are details you want from me:
>
> # gpart show
> =>        34  1953525101  ad6  GPT  (932G)
>           34         128    1  freebsd-boot  (64K)
>          162  1953524973    2  freebsd-zfs  (932G)
>
> =>        34  1953525101  ad8  GPT  (932G)
>           34         128    1  freebsd-boot  (64K)
>          162  1953524973    2  freebsd-zfs  (932G)
>
> =>        34  1953525101  ad10  GPT  (932G)
>           34         128     1  freebsd-boot  (64K)
>          162  1953524973     2  freebsd-zfs  (932G)
>
> =>        34  1953525101  ad12  GPT  (932G)
>           34         128     1  freebsd-boot  (64K)
>          162  1953524973     2  freebsd-zfs  (932G)
>
> # zpool status
>   pool: z
>  state: ONLINE
>  scrub: none requested
> config:
>
>     NAME        STATE     READ WRITE CKSUM
>     z           ONLINE       0     0     0
>       raidz1    ONLINE       0     0     0
>         ad6p2   ONLINE       0     0     0
>         ad8p2   ONLINE       0     0     0
>         ad10p2  ONLINE       0     0     0
>         ad12p2  ONLINE       0     0     0
>
> errors: No known data errors
>
> # zpool get all z
> NAME  PROPERTY       VALUE       SOURCE
> z     size           3.62T       -
> z     used           4.62G       -
> z     available      3.62T       -
> z     capacity       0%          -
> z     altroot        -           default
> z     health         ONLINE      -
> z     guid           17857007133862981114  -
> z     version        13          default
> z     bootfs         z/system    local
> z     delegation     on          default
> z     autoreplace    off         default
> z     cachefile      -           default
> z     failmode       wait        default
> z     listsnapshots  off         default
>
> I've tested your patches but it seems that you're right and it's not
> gang related issue. I was able to discover these things on a fully
> functional zfs pool (system compiled with your patches):
>
> 1, If I overwrite the file /boot/loader.conf (with copy of itself, or
> when upgrading kernel/world), next reboot comes with these messages:
>
> BTX loader 1.00  BTX version is 1.02
> Consoles: internal video/keyboard
> BIOS drive C: is disk0
> BIOS drive D: is disk1
> BIOS drive E: is disk2
> BIOS drive F: is disk3
> BIOS 627kB/3405248kB available memory
>
> FreeBSD/i386 bootstrap loader, Revision 1.1
> (root@ztest, Thu Oct 22 22:27:22 CEST 2009)
> Loading /boot/defaults/loader.conf
> ZFS: i/o error - all block copies unavailable
> Warning: error reading file /boot/loader.conf
>
> Then I'm still able to boot the system, but I must set the boot
> variables included in loader.conf by hand
>
> 2, Next I overwrite the file /boot/loader (with copy of itself, or when
> upgrading kernel/world) and reboot comes with these messages:
>
> BTX loader 1.00  BTX version is 1.02
> Consoles: internal video/keyboard
> BIOS drive C: is disk0
> BIOS drive D: is disk1
> BIOS drive E: is disk2
> BIOS drive F: is disk3
> BIOS 627kB/3405248kB available memory
>
> FreeBSD/i386 bootstrap loader, Revision 1.1
> (root@ztest, Thu Oct 22 22:27:22 CEST 2009)
> Loading /boot/defaults/loader.conf
> ZFS: i/o error - all block copies unavailable
> Warning: error reading file /boot/loader.conf
> ZFS: i/o error - all block copies unavailable
> ZFS: i/o error - all block copies unavailable
> ZFS: i/o error - all block copies unavailable
> ZFS: i/o error - all block copies unavailable
> ZFS: i/o error - all block copies unavailable
> ZFS: i/o error - all block copies unavailable
> ZFS: i/o error - all block copies unavailable
> ZFS: i/o error - all block copies unavailable
> Unable to load a kernel!
>
> After that I'm no longer able to boot the system from zfs pool.
>
> Hope you have some ideas...

Ok, can you retest with -CURRENT?  I just committed some fixes on
Friday.  I'm having real difficulty in reproducing these issues.  Most
of the problems that I've run into so far had to do with the system not
knowing about all of the vdevs when it wanted to read something.  In
your case, it looks like you are making it to boot3 and it appears to be
seeing all 4 of your disks.  Right now, I've been trying to track down
an issue wher the MOS can't be read, which basically means that we have
screwed up the root block pointer somehow.  I haven't been able to
reproduce that issue in qemu, I have been able to reproduce it with
VirtualBox, but it is really time consuming trying to work in vbox since
I have to reconvert all of the disk images every time I make a change.
I'm actually a bit concerned that it hinges on how many drives are
visible to the bios at various points in time.

robert.

> vaLin
>
> >>> Ok, I can't figure out any way to test this... beyond the fact that it
> >>> builds and doesn't break my currently working setup.  Can you give this
> >>> a try?  It should still report if it finds gang blocks, but hopefully
> >>> now will read them as well.
> >>>
> >>> robert.
> >>>
> >>>  
> >>>      
> >> Big thanks for the patches Robert, I will definitely test them as soon
> >> as possible (tomorrow) and report the results immediately to list. I can
> >> repeat this issue probably at any time (up to cca 30 times tested with
> >> the same result), so don't bother about the broken booting, I'm prepared
> >> for it...
> >>
> >> vaLin
> >>    
> >>>>  From Sun's docs:
> >>>>
> >>>> Gang blocks
> >>>>
> >>>> When there is not enough contiguous space to write a complete block, the ZIO
> >>>> pipeline will break the I/O up into smaller 'gang blocks' which can later be
> >>>> assembled transparently to appear as complete blocks.
> >>>>
> >>>> Everything works fine for me, until I rewrite kernel/world after system
> >>>> upgrade to latest one (releng_8). After this am I no longer able to boot
> >>>> from zfs raidz1 pool with following messages:
> >>>>
> >>>>  >/ ZFS: i/o error - all block copies unavailable
> >>>> />/ ZFS: can't read MOS
> >>>> />/ ZFS: unexpected object set type lld
> >>>> />/ ZFS: unexpected object set type lld
> >>>> />/
> >>>> />/ FreeBSD/i386 boot
> >>>> />/ Default: z:/boot/kernel/kernel
> >>>> />/ boot:
> >>>> />/ ZFS: unexpected object set type lld
> >>>> />/
> >>>> />/ FreeBSD/i386 boot
> >>>> />/ Default: tank:/boot/kernel/kernel
> >>>> />/ boot:
> >>>> //
> >>>> /I presume it's the same issue as talked in june-2009 current mailing
> >>>> list
> >>>> http://lists.freebsd.org/pipermail/freebsd-current/2009-June/008589.html
> >>>>
> >>>> Any success in that matter?
> >>>>
> >>>> Thnx for answer.
> >>>>
> >>>> vaLin
> >>>> _______________________________________________
> >>>> freebsd-current@... mailing list
> >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> >>>> To unsubscribe, send any mail to "freebsd-current-unsubscribe@..."
> >>>>    
> >>>>        
>
--
Robert Noland <rnoland@...>
FreeBSD

_______________________________________________
freebsd-fs@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-unsubscribe@..."

Parent Message unknown Re: GPT boot with ZFS RAIDZ "ZFS: i/o error - all block copies unavailable"

by Radek Valášek-2 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Robert Noland napsal(a):

> On Mon, 2009-10-26 at 10:23 +0100, Merijn Verstraaten wrote:
>  
>> On Mon, 26 Oct 2009 01:31:46 +0100, Robert Noland <rnoland@...>  
>> wrote:
>>    
>>>> After installing 8.0-RC1 (amd64) from USB stick this installation works
>>>> fine. If I csup to RELENG_8 (amd64) and compile + install world and  
>>>> kernel
>>>> booting from the ZFS fails. The initial installation I did just this, on
>>>> another attempt I ran "gpart bootcode -b /boot/pbmr -p /boot/gptzfsboot
>>>> adX" on all disks before rebooting to see if that had any effect. The  
>>>> end
>>>> result is the same. After rebooting the machine I get the following
>>>> prompt(s):
>>>>
>>>>          ZFS: i/o error - all block copies unavailable
>>>>          Invalid format
>>>>
>>>>          FreeBSD/i386 boot
>>>>          Default: tank:/boot/kernel/kernel
>>>>          boot:
>>>>        
>>> Could you type "status" at this point and tell what it shows?
>>>      
>> If I type status at this point I get:
>>
>>    pool: tank
>>    config:
>>        NAME    STATE
>>        tank   ONLINE
>>      raidz1   ONLINE
>>       ad4p3   ONLINE
>>       ad6p3   ONLINE
>>       ad8p3   ONLINE
>>      ad10p3   ONLINE
>>
>> Which seems odd, since that's all the drives there are. So if it finds  
>> these it's already found all drives. My optimistic "Oh! I'll try and boot  
>> again" spirit was however crushed since it just results in the same error.
>>    
>
> Ok, that is both good and frustrating... I haven't produced any boot
> failures with all of the drives visible.  Do, note that I just added
> support for reading gang blocks to the loader. (basically untested,
> since I haven't managed to create them at will)  You will need to update
> your partition boot code for it to be supported during early boot.  i.e.
> gpart bootcode -p /boot/gptzfsboot -i <boot partition> <disk>
>
> The "all block copies unavailable" is a frustrating error, since all it
> means is a failed read, but we don't get a clue what failed or why.
> With the code that is in -CURRENT it will report gang blocks if found,
> even if it fails to read them.
>  
> robert.
>  
So I switched to -CURRENT:

1, overwriting /boot/loader.conf results with:

BTX loader 1.00 BTX version is 1.02
Consoles: internal video/keyboard
BIOS drive C: is disk0
BIOS drive D: is disk1
BIOS drive E: is disk2
BIOS drive F: is disk3
BIOS 627kB/3405248kB available memory

FreeBSD/i386 bootstrap loader, Revision 1.1
(root@ztest, Mon Oct 26 14:01:44 CEST 2009)
Loading /boot/defaults/loader.conf
ZFS: i/o error - all block copies unavailable
Warning: error reading file /boot/loader.conf

so basically the same as in RELENG_8

2, + overwriting /boot/loader results with:

ZFS: i/o error - all block copies unavailable
Invalid format

FreeBSD/i386 boot
Default: z:/boot/kernel/kernel
boot:
\
int=00000001  err=00000000  efl=00000087 eip=0018d27d
eax=0018d2af  ebx=18bf9925  ecx=540d8ef2  edx=00000000
esi=00009401  edi=000919d0  ebp=36571125  esp=80000000
cs=0008  ds=0010  es=0010  fs=0010  gs=0010  ss=0010
cs:eip=1f 68 e2 c6 7d 75 0c 5d-45 58 c7 80 f5 99 bd 9e
       fe 68 2d 3e 3c 35 5e 67-61 12 fe 50 c9 0b e4 70
ss:esp=00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00
       00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00
BTX halted

3, I also try the 'status' as you told to Merijn before BTX halted:

ZFS: i/o error - all block copies unavailable
Invalid format

FreeBSD/i386 boot
Default: z:/boot/kernel/kernel
boot: status  pool: z
config:
NAME  STATE
z  ONLINE
raidz1  ONLINE
ad6p2  ONLINE
ad8p2  ONLINE
ad10p2  ONLINE
ad12p2  ONLINE

radek.

>  
>  
>> Kind regards,
>> Merijn
>> _______________________________________________
>> freebsd-current@... mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
>> To unsubscribe, send any mail to "freebsd-current-unsubscribe@..."
>>    

_______________________________________________
freebsd-fs@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-unsubscribe@..."

Parent Message unknown Re: GPT boot with ZFS RAIDZ "ZFS: i/o error - all block copies unavailable"

by Radek Valášek-2 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Robert Noland napsal(a):

> On Mon, 2009-10-26 at 10:23 +0100, Merijn Verstraaten wrote:
>  
>> On Mon, 26 Oct 2009 01:31:46 +0100, Robert Noland <rnoland@...>  
>> wrote:
>>    
>>>> After installing 8.0-RC1 (amd64) from USB stick this installation works
>>>> fine. If I csup to RELENG_8 (amd64) and compile + install world and  
>>>> kernel
>>>> booting from the ZFS fails. The initial installation I did just this, on
>>>> another attempt I ran "gpart bootcode -b /boot/pbmr -p /boot/gptzfsboot
>>>> adX" on all disks before rebooting to see if that had any effect. The  
>>>> end
>>>> result is the same. After rebooting the machine I get the following
>>>> prompt(s):
>>>>
>>>>          ZFS: i/o error - all block copies unavailable
>>>>          Invalid format
>>>>
>>>>          FreeBSD/i386 boot
>>>>          Default: tank:/boot/kernel/kernel
>>>>          boot:
>>>>        
>>> Could you type "status" at this point and tell what it shows?
>>>      
>> If I type status at this point I get:
>>
>>    pool: tank
>>    config:
>>        NAME    STATE
>>        tank   ONLINE
>>      raidz1   ONLINE
>>       ad4p3   ONLINE
>>       ad6p3   ONLINE
>>       ad8p3   ONLINE
>>      ad10p3   ONLINE
>>
>> Which seems odd, since that's all the drives there are. So if it finds  
>> these it's already found all drives. My optimistic "Oh! I'll try and boot  
>> again" spirit was however crushed since it just results in the same error.
>>    
>
> Ok, that is both good and frustrating... I haven't produced any boot
> failures with all of the drives visible.  Do, note that I just added
> support for reading gang blocks to the loader. (basically untested,
> since I haven't managed to create them at will)  You will need to update
> your partition boot code for it to be supported during early boot.  i.e.
> gpart bootcode -p /boot/gptzfsboot -i <boot partition> <disk>
>
> The "all block copies unavailable" is a frustrating error, since all it
> means is a failed read, but we don't get a clue what failed or why.
> With the code that is in -CURRENT it will report gang blocks if found,
> even if it fails to read them.
>  
> robert.
>  
So I switched to -CURRENT:

1, overwriting /boot/loader.conf results with:

BTX loader 1.00 BTX version is 1.02
Consoles: internal video/keyboard
BIOS drive C: is disk0
BIOS drive D: is disk1
BIOS drive E: is disk2
BIOS drive F: is disk3
BIOS 627kB/3405248kB available memory

FreeBSD/i386 bootstrap loader, Revision 1.1
(root@ztest, Mon Oct 26 14:01:44 CEST 2009)
Loading /boot/defaults/loader.conf
ZFS: i/o error - all block copies unavailable
Warning: error reading file /boot/loader.conf

so basically the same as in RELENG_8

2, + overwriting /boot/loader results with:

ZFS: i/o error - all block copies unavailable
Invalid format

FreeBSD/i386 boot
Default: z:/boot/kernel/kernel
boot:
\
int=00000001  err=00000000  efl=00000087 eip=0018d27d
eax=0018d2af  ebx=18bf9925  ecx=540d8ef2  edx=00000000
esi=00009401  edi=000919d0  ebp=36571125  esp=80000000
cs=0008  ds=0010  es=0010  fs=0010  gs=0010  ss=0010
cs:eip=1f 68 e2 c6 7d 75 0c 5d-45 58 c7 80 f5 99 bd 9e
       fe 68 2d 3e 3c 35 5e 67-61 12 fe 50 c9 0b e4 70
ss:esp=00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00
       00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00
BTX halted

3, I also try the 'status' as you told to Merijn before BTX halted:

ZFS: i/o error - all block copies unavailable
Invalid format

FreeBSD/i386 boot
Default: z:/boot/kernel/kernel
boot: status  pool: z
config:
NAME  STATE
z  ONLINE
raidz1  ONLINE
ad6p2  ONLINE
ad8p2  ONLINE
ad10p2  ONLINE
ad12p2  ONLINE

radek.

>  
>  
>> Kind regards,
>> Merijn
>> _______________________________________________
>> freebsd-current@... mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
>> To unsubscribe, send any mail to "freebsd-current-unsubscribe@..."
>>    


_______________________________________________
freebsd-fs@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-unsubscribe@..."

Re: GPT boot with ZFS RAIDZ "ZFS: i/o error - all block copies unavailable"

by kickbsd kickbsd :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Hi!

I've just tried to follow
http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/RAIDZ2
with RC2
and got the same error as below.

27.10.09, 11:15, "Radek Valášek" <valin@...>:

> Robert Noland napsal(a):
> > On Mon, 2009-10-26 at 10:23 +0100, Merijn Verstraaten wrote:
> >  
> >> On Mon, 26 Oct 2009 01:31:46 +0100, Robert Noland  
> >> wrote:
> >>    
> >>>> After installing 8.0-RC1 (amd64) from USB stick this installation works
> >>>> fine. If I csup to RELENG_8 (amd64) and compile + install world and  
> >>>> kernel
> >>>> booting from the ZFS fails. The initial installation I did just this, on
> >>>> another attempt I ran "gpart bootcode -b /boot/pbmr -p /boot/gptzfsboot
> >>>> adX" on all disks before rebooting to see if that had any effect. The  
> >>>> end
> >>>> result is the same. After rebooting the machine I get the following
> >>>> prompt(s):
> >>>>
> >>>>          ZFS: i/o error - all block copies unavailable
> >>>>          Invalid format
> >>>>
> >>>>          FreeBSD/i386 boot
> >>>>          Default: tank:/boot/kernel/kernel
> >>>>          boot:
> >>>>        
> >>> Could you type "status" at this point and tell what it shows?
> >>>      
> >> If I type status at this point I get:
> >>
> >>    pool: tank
> >>    config:
> >>        NAME    STATE
> >>        tank   ONLINE
> >>      raidz1   ONLINE
> >>       ad4p3   ONLINE
> >>       ad6p3   ONLINE
> >>       ad8p3   ONLINE
> >>      ad10p3   ONLINE
> >>
> >> Which seems odd, since that's all the drives there are. So if it finds  
> >> these it's already found all drives. My optimistic "Oh! I'll try and boot  
> >> again" spirit was however crushed since it just results in the same error.
> >>    
> >
> > Ok, that is both good and frustrating... I haven't produced any boot
> > failures with all of the drives visible.  Do, note that I just added
> > support for reading gang blocks to the loader. (basically untested,
> > since I haven't managed to create them at will)  You will need to update
> > your partition boot code for it to be supported during early boot.  i.e.
> > gpart bootcode -p /boot/gptzfsboot -i  
> >
> > The "all block copies unavailable" is a frustrating error, since all it
> > means is a failed read, but we don't get a clue what failed or why.
> > With the code that is in -CURRENT it will report gang blocks if found,
> > even if it fails to read them.
> >  
> > robert.
> >  
> So I switched to -CURRENT:
> 1, overwriting /boot/loader.conf results with:
> BTX loader 1.00 BTX version is 1.02
> Consoles: internal video/keyboard
> BIOS drive C: is disk0
> BIOS drive D: is disk1
> BIOS drive E: is disk2
> BIOS drive F: is disk3
> BIOS 627kB/3405248kB available memory
> FreeBSD/i386 bootstrap loader, Revision 1.1
> (root@ztest, Mon Oct 26 14:01:44 CEST 2009)
> Loading /boot/defaults/loader.conf
> ZFS: i/o error - all block copies unavailable
> Warning: error reading file /boot/loader.conf
> so basically the same as in RELENG_8
> 2, + overwriting /boot/loader results with:
> ZFS: i/o error - all block copies unavailable
> Invalid format
> FreeBSD/i386 boot
> Default: z:/boot/kernel/kernel
> boot:
> \
> int=00000001  err=00000000  efl=00000087 eip=0018d27d
> eax=0018d2af  ebx=18bf9925  ecx=540d8ef2  edx=00000000
> esi=00009401  edi=000919d0  ebp=36571125  esp=80000000
> cs=0008  ds=0010  es=0010  fs=0010  gs=0010  ss=0010
> cs:eip=1f 68 e2 c6 7d 75 0c 5d-45 58 c7 80 f5 99 bd 9e
>        fe 68 2d 3e 3c 35 5e 67-61 12 fe 50 c9 0b e4 70
> ss:esp=00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00
>        00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00
> BTX halted
> 3, I also try the 'status' as you told to Merijn before BTX halted:
> ZFS: i/o error - all block copies unavailable
> Invalid format
> FreeBSD/i386 boot
> Default: z:/boot/kernel/kernel
> boot: status  pool: z
> config:
> NAME  STATE
> z  ONLINE
> raidz1  ONLINE
> ad6p2  ONLINE
> ad8p2  ONLINE
> ad10p2  ONLINE
> ad12p2  ONLINE
> radek.
> >  
> >  
> >> Kind regards,
> >> Merijn
> >> _______________________________________________
> >> freebsd-current@... mailing list
> >> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> >> To unsubscribe, send any mail to "freebsd-current-unsubscribe@..."
> >>    
> _______________________________________________
> freebsd-fs@... mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@..."

--
Находчивая почта находится здесь: http://mail.yandex.ru/promo/new/search
_______________________________________________
freebsd-fs@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-unsubscribe@..."

Re: GPT boot with ZFS RAIDZ "ZFS: i/o error - all block copies unavailable"

by Matt Reimer :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

2009/10/15 Radek Valášek <valin@...>:

> Hi,
>
> I want to ask if there is something new in adding support to
> gptzfsboot/zfsboot for reading gang-blocks?
>
> From Sun's docs:
>
> Gang blocks
>
> When there is not enough contiguous space to write a complete block, the ZIO
> pipeline will break the I/O up into smaller 'gang blocks' which can later be
> assembled transparently to appear as complete blocks.
>
> Everything works fine for me, until I rewrite kernel/world after system
> upgrade to latest one (releng_8). After this am I no longer able to boot
> from zfs raidz1 pool with following messages:
>
>>/ ZFS: i/o error - all block copies unavailable
> />/ ZFS: can't read MOS
> />/ ZFS: unexpected object set type lld
> />/ ZFS: unexpected object set type lld
> />/
> />/ FreeBSD/i386 boot
> />/ Default: z:/boot/kernel/kernel
> />/ boot:
> />/ ZFS: unexpected object set type lld
> />/
> />/ FreeBSD/i386 boot
> />/ Default: tank:/boot/kernel/kernel
> />/ boot:

Radek,

Try the attached patch (sponsored by VPOP Technologies). I found an
overflow in /sys/cddl/boot/zfs/zfssubr.c:vdev_raidz_read() that was
causing my 6x1TB raidz2 array to fail to boot.

Apply the patch, build everything in /sys/boot, and then make sure you
update both gptzfsboot and /boot/loader.

Robert, I'm guessing you couldn't replicate this because your array
was small enough not to result in block numbers overflowing an int.

The kernel source for the corresponding functionality is in
/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_raidz.c:vdev_raidz_map_alloc().
There all these variables are uint64_t, but I think unnecessarily. I
tried changing the boot loader's vdev_raidz_read() variables to all
uint64_t but then gptzfsboot would reboot itself, likely due to a
stack overflow. The attached patch just changes a few variables that,
after a quick analysis, seemed likely to overflow.

If this looks good, would someone commit it?

Matt
_______________________________________________
freebsd-fs@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-unsubscribe@..."

Re: GPT boot with ZFS RAIDZ "ZFS: i/o error - all block copies unavailable"

by Matt Reimer :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

2009/11/12 Matt Reimer <mattjreimer@...>:
>
> Radek,
>
> Try the attached patch (sponsored by VPOP Technologies). I found an
> overflow in /sys/cddl/boot/zfs/zfssubr.c:vdev_raidz_read() that was
> causing my 6x1TB raidz2 array to fail to boot.
>
> Apply the patch, build everything in /sys/boot, and then make sure you
> update both gptzfsboot and /boot/loader.

Oops, here's the patch.

Matt


_______________________________________________
freebsd-fs@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-unsubscribe@..."

zfssubr.c.patch (570 bytes) Download Attachment

Re: GPT boot with ZFS RAIDZ "ZFS: i/o error - all block copies unavailable"

by Robert Noland :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Thu, 2009-11-12 at 16:54 -0800, Matt Reimer wrote:

> 2009/10/15 Radek Valášek <valin@...>:
> > Hi,
> >
> > I want to ask if there is something new in adding support to
> > gptzfsboot/zfsboot for reading gang-blocks?
> >
> > From Sun's docs:
> >
> > Gang blocks
> >
> > When there is not enough contiguous space to write a complete block, the ZIO
> > pipeline will break the I/O up into smaller 'gang blocks' which can later be
> > assembled transparently to appear as complete blocks.
> >
> > Everything works fine for me, until I rewrite kernel/world after system
> > upgrade to latest one (releng_8). After this am I no longer able to boot
> > from zfs raidz1 pool with following messages:
> >
> >>/ ZFS: i/o error - all block copies unavailable
> > />/ ZFS: can't read MOS
> > />/ ZFS: unexpected object set type lld
> > />/ ZFS: unexpected object set type lld
> > />/
> > />/ FreeBSD/i386 boot
> > />/ Default: z:/boot/kernel/kernel
> > />/ boot:
> > />/ ZFS: unexpected object set type lld
> > />/
> > />/ FreeBSD/i386 boot
> > />/ Default: tank:/boot/kernel/kernel
> > />/ boot:
>
> Radek,
>
> Try the attached patch (sponsored by VPOP Technologies). I found an
> overflow in /sys/cddl/boot/zfs/zfssubr.c:vdev_raidz_read() that was
> causing my 6x1TB raidz2 array to fail to boot.
>
> Apply the patch, build everything in /sys/boot, and then make sure you
> update both gptzfsboot and /boot/loader.
>
> Robert, I'm guessing you couldn't replicate this because your array
> was small enough not to result in block numbers overflowing an int.

This is likely, all of my raidz tests were with vnode backed 1GB memory
disks.  So my largest configuration was a 6 x 1GB raidz2.

> The kernel source for the corresponding functionality is in
> /sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_raidz.c:vdev_raidz_map_alloc().
> There all these variables are uint64_t, but I think unnecessarily. I
> tried changing the boot loader's vdev_raidz_read() variables to all
> uint64_t but then gptzfsboot would reboot itself, likely due to a
> stack overflow. The attached patch just changes a few variables that,
> after a quick analysis, seemed likely to overflow.
>
> If this looks good, would someone commit it?

ps@ grabbed it up already, but I may handle the MFC for him.  I have
some other minor fixups in my tree right now... like teaching printf to
handle %llx.  Thanks for finding this... It's been really frustrating
that I couldn't produce a failing system.

robert.

> Matt
> _______________________________________________
> freebsd-current@... mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe@..."
--
Robert Noland <rnoland@...>
FreeBSD

_______________________________________________
freebsd-fs@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-unsubscribe@..."

Re: GPT boot with ZFS RAIDZ "ZFS: i/o error - all block copies unavailable"

by Matt Reimer :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

2009/11/12 Robert Noland <rnoland@...>:

> On Thu, 2009-11-12 at 16:54 -0800, Matt Reimer wrote:
>> 2009/10/15 Radek Valášek <valin@...>:
>> > Everything works fine for me, until I rewrite kernel/world after system
>> > upgrade to latest one (releng_8). After this am I no longer able to boot
>> > from zfs raidz1 pool with following messages:
>> >
>> >>/ ZFS: i/o error - all block copies unavailable
>> > />/ ZFS: can't read MOS
>> > />/ ZFS: unexpected object set type lld
>> > />/ ZFS: unexpected object set type lld
>> > />/
>> > />/ FreeBSD/i386 boot
>> > />/ Default: z:/boot/kernel/kernel
>> > />/ boot:
>> > />/ ZFS: unexpected object set type lld
>> > />/
>> > />/ FreeBSD/i386 boot
>> > />/ Default: tank:/boot/kernel/kernel
>> > />/ boot:
>>
>> Radek,
>>
>> Try the attached patch (sponsored by VPOP Technologies). I found an
>> overflow in /sys/cddl/boot/zfs/zfssubr.c:vdev_raidz_read() that was
>> causing my 6x1TB raidz2 array to fail to boot.
...

>> The kernel source for the corresponding functionality is in
>> /sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_raidz.c:vdev_raidz_map_alloc().
>> There all these variables are uint64_t, but I think unnecessarily. I
>> tried changing the boot loader's vdev_raidz_read() variables to all
>> uint64_t but then gptzfsboot would reboot itself, likely due to a
>> stack overflow. The attached patch just changes a few variables that,
>> after a quick analysis, seemed likely to overflow.
>>
>> If this looks good, would someone commit it?
>
> ps@ grabbed it up already, but I may handle the MFC for him.  I have
> some other minor fixups in my tree right now... like teaching printf to
> handle %llx.  Thanks for finding this... It's been really frustrating
> that I couldn't produce a failing system.

Is it possible for this patch to get into 8.0-RELEASE, or is it too
late? I suppose it doesn't matter that much since the loader isn't
built with LOADER_ZFS_SUPPORT by default anyway, so folks are going to
have to compile it themselves.

Matt
_______________________________________________
freebsd-fs@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-unsubscribe@..."

Re: GPT boot with ZFS RAIDZ "ZFS: i/o error - all block copies unavailable"

by Robert Noland :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Thu, 2009-11-12 at 22:15 -0800, Matt Reimer wrote:

> 2009/11/12 Robert Noland <rnoland@...>:
> > On Thu, 2009-11-12 at 16:54 -0800, Matt Reimer wrote:
> >> 2009/10/15 Radek Valášek <valin@...>:
> >> > Everything works fine for me, until I rewrite kernel/world after system
> >> > upgrade to latest one (releng_8). After this am I no longer able to boot
> >> > from zfs raidz1 pool with following messages:
> >> >
> >> >>/ ZFS: i/o error - all block copies unavailable
> >> > />/ ZFS: can't read MOS
> >> > />/ ZFS: unexpected object set type lld
> >> > />/ ZFS: unexpected object set type lld
> >> > />/
> >> > />/ FreeBSD/i386 boot
> >> > />/ Default: z:/boot/kernel/kernel
> >> > />/ boot:
> >> > />/ ZFS: unexpected object set type lld
> >> > />/
> >> > />/ FreeBSD/i386 boot
> >> > />/ Default: tank:/boot/kernel/kernel
> >> > />/ boot:
> >>
> >> Radek,
> >>
> >> Try the attached patch (sponsored by VPOP Technologies). I found an
> >> overflow in /sys/cddl/boot/zfs/zfssubr.c:vdev_raidz_read() that was
> >> causing my 6x1TB raidz2 array to fail to boot.
> ...
> >> The kernel source for the corresponding functionality is in
> >> /sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_raidz.c:vdev_raidz_map_alloc().
> >> There all these variables are uint64_t, but I think unnecessarily. I
> >> tried changing the boot loader's vdev_raidz_read() variables to all
> >> uint64_t but then gptzfsboot would reboot itself, likely due to a
> >> stack overflow. The attached patch just changes a few variables that,
> >> after a quick analysis, seemed likely to overflow.
> >>
> >> If this looks good, would someone commit it?
> >
> > ps@ grabbed it up already, but I may handle the MFC for him.  I have
> > some other minor fixups in my tree right now... like teaching printf to
> > handle %llx.  Thanks for finding this... It's been really frustrating
> > that I couldn't produce a failing system.
>
> Is it possible for this patch to get into 8.0-RELEASE, or is it too
> late? I suppose it doesn't matter that much since the loader isn't
> built with LOADER_ZFS_SUPPORT by default anyway, so folks are going to
> have to compile it themselves.

I think we have missed the boat, but I'll talk to re@ and see if we can
get it in.

robert.

> Matt
--
Robert Noland <rnoland@...>
FreeBSD

_______________________________________________
freebsd-fs@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-unsubscribe@..."

Re: GPT boot with ZFS RAIDZ "ZFS: i/o error - all block copies unavailable"

by Stefan Esser-3 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On 13.11.2009 15:25, Robert Noland wrote:
> I think we have missed the boat, but I'll talk to re@ and see if we
> can get it in.

The patch fixed GPT/ZFS booting for me, too. It would be good to have
it in 8.0 (since it is definitely required to boot from ZFS pools with
non-trivial sizes), and does not affect anybody not trying to boot
this way. OTOH, it since you cannot just install FreeBSD on pure ZFS
from sysinstall, it might be sufficient to prominently warn about this
problem and point at the required patch, to prevent foot-shooting.

But having this patch that has been successfully tested by a number
of people that suffered from the GPT/ZFS boot problem looks highly
preferable to me ...

Regards, STefan
_______________________________________________
freebsd-fs@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-unsubscribe@..."

Re: GPT boot with ZFS RAIDZ "ZFS: i/o error - all block copies unavailable"

by Radek Valášek-2 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Stefan Esser napsal(a):

> On 13.11.2009 15:25, Robert Noland wrote:
>  
>> I think we have missed the boat, but I'll talk to re@ and see if we
>> can get it in.
>>    
>
> The patch fixed GPT/ZFS booting for me, too. It would be good to have
> it in 8.0 (since it is definitely required to boot from ZFS pools with
> non-trivial sizes), and does not affect anybody not trying to boot
> this way. OTOH, it since you cannot just install FreeBSD on pure ZFS
> from sysinstall, it might be sufficient to prominently warn about this
> problem and point at the required patch, to prevent foot-shooting.
>
> But having this patch that has been successfully tested by a number
> of people that suffered from the GPT/ZFS boot problem looks highly
> preferable to me ...
>
> Regards, STefan
>  
I can confirm that the patch is working for me too, and I'm able now to
boot from raidz/raidz2 pool after rewriting loader.conf/loader/kernel. I
agree with Stefan, having it in 8.0-RELEASE would be good, catch the boat :)

Big thnx to Matt, great work from all.

_______________________________________________
freebsd-fs@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-unsubscribe@..."

Re: GPT boot with ZFS RAIDZ "ZFS: i/o error - all block copies unavailable"

by Krzysztof Dajka :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Thanks Matt for the patch. I used it with 8.0RC3 release. I installed
FreeBSD under Linux (KVM) my 3x500GB drives were mounted as a scsi
drives. Installation went smoothly but when I rebooted FreeBSD guest
it hang as usual ;) with "ZFS: i/o error - all block copies
unavailable", well it also spit out some LBA errors for the first
time. I was a little disappointed, because I've been trying for three
weeks to replace my Debian system with broken ext3 fs with FreeBSD on
raidz. But I thought to myself I'll give it a try, and run FreeBSD
native. To my suprise it welcomed me with login prompt. Once again
thanks for the patch. It would be good idea to merge it with final
release.

On 11/13/09, Stefan Esser <se@...> wrote:

> On 13.11.2009 15:25, Robert Noland wrote:
>> I think we have missed the boat, but I'll talk to re@ and see if we
>> can get it in.
>
> The patch fixed GPT/ZFS booting for me, too. It would be good to have
> it in 8.0 (since it is definitely required to boot from ZFS pools with
> non-trivial sizes), and does not affect anybody not trying to boot
> this way. OTOH, it since you cannot just install FreeBSD on pure ZFS
> from sysinstall, it might be sufficient to prominently warn about this
> problem and point at the required patch, to prevent foot-shooting.
>
> But having this patch that has been successfully tested by a number
> of people that suffered from the GPT/ZFS boot problem looks highly
> preferable to me ...
>
> Regards, STefan
> _______________________________________________
> freebsd-current@... mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe@..."
>
_______________________________________________
freebsd-fs@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-unsubscribe@..."

Re: GPT boot with ZFS RAIDZ "ZFS: i/o error - all block copies unavailable"

by Robert Noland :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Sat, 2009-11-14 at 10:15 +0100, Krzysztof Dajka wrote:

> Thanks Matt for the patch. I used it with 8.0RC3 release. I installed
> FreeBSD under Linux (KVM) my 3x500GB drives were mounted as a scsi
> drives. Installation went smoothly but when I rebooted FreeBSD guest
> it hang as usual ;) with "ZFS: i/o error - all block copies
> unavailable", well it also spit out some LBA errors for the first
> time. I was a little disappointed, because I've been trying for three
> weeks to replace my Debian system with broken ext3 fs with FreeBSD on
> raidz. But I thought to myself I'll give it a try, and run FreeBSD
> native. To my suprise it welcomed me with login prompt. Once again
> thanks for the patch. It would be good idea to merge it with final
> release.

This was approved by re@ and has been merged to the release branch.  It
should be included in 8.0-RELEASE.

robert.

> On 11/13/09, Stefan Esser <se@...> wrote:
> > On 13.11.2009 15:25, Robert Noland wrote:
> >> I think we have missed the boat, but I'll talk to re@ and see if we
> >> can get it in.
> >
> > The patch fixed GPT/ZFS booting for me, too. It would be good to have
> > it in 8.0 (since it is definitely required to boot from ZFS pools with
> > non-trivial sizes), and does not affect anybody not trying to boot
> > this way. OTOH, it since you cannot just install FreeBSD on pure ZFS
> > from sysinstall, it might be sufficient to prominently warn about this
> > problem and point at the required patch, to prevent foot-shooting.
> >
> > But having this patch that has been successfully tested by a number
> > of people that suffered from the GPT/ZFS boot problem looks highly
> > preferable to me ...
> >
> > Regards, STefan
> > _______________________________________________
> > freebsd-current@... mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-current
> > To unsubscribe, send any mail to "freebsd-current-unsubscribe@..."
> >
--
Robert Noland <rnoland@...>
FreeBSD

_______________________________________________
freebsd-fs@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-unsubscribe@..."

Re: GPT boot with ZFS RAIDZ "ZFS: i/o error - all block copies unavailable"

by Stefan Bethke-2 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Am 13.11.2009 um 01:54 schrieb Matt Reimer:

> Try the attached patch (sponsored by VPOP Technologies). I found an
> overflow in /sys/cddl/boot/zfs/zfssubr.c:vdev_raidz_read() that was
> causing my 6x1TB raidz2 array to fail to boot.

I can confirm as well that the patch (as committed to -current as r199241) makes my loader happy.  Now I just need to figure out why the kernel won't mount root...


Thanks,
Stefan

--
Stefan Bethke <stb@...>   Fon +49 151 14070811




_______________________________________________
freebsd-fs@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-unsubscribe@..."
< Prev | 1 - 2 | Next >