GPT boot with ZFS RAIDZ "ZFS: i/o error - all block copies unavailable"

[Merijn Verstraaten-2]

> Ok, can you retest with -CURRENT?  I just committed some fixes on
> Friday.  I'm having real difficulty in reproducing these issues.  Most
> of the problems that I've run into so far had to do with the system not
> knowing about all of the vdevs when it wanted to read something.  In
> your case, it looks like you are making it to boot3 and it appears to be
> seeing all 4 of your disks.  Right now, I've been trying to track down
> an issue wher the MOS can't be read, which basically means that we have
> screwed up the root block pointer somehow.  I haven't been able to
> reproduce that issue in qemu, I have been able to reproduce it with
> VirtualBox, but it is really time consuming trying to work in vbox since
> I have to reconvert all of the disk images every time I make a change.
> I'm actually a bit concerned that it hinges on how many drives are
> visible to the bios at various points in time.
> robert.

I noticed this thread in the archives and I've been having the same
problems as Radek (see belw) with root on ZFS RAID-Z not booting the
machine. When I install world from the 8.0-RC1 USB image I can boot from
RAID-Z with no problems. But when I csup'ed to RELENG_8 yesterday
(Saturday, so this should be after you committed the new changes) and
build+installed world booting from RAID-Z broke with the exact same
problems as Radek describes below. I just tried again to be absolutely
sure (It just occured to me I perhaps should have done "gpart show",
"zpool status" and "zpool get all z" before breaking the system, but oh
well...).

I have 4 SATA disks ad4, ad6, ad8 and ad10 with three partitions each,
adXp1 is a freebsd-boot partition like Radek has, adXp2 is a 1GB
freebsd-swap partition and adXp3 has the rest of the disk as a freebsd-zfs
partition. These 4 p3 partitions are placed in a single RAID-Z vdev. zpool
status looks pretty much like that of Radek too. I have my root partition
in tank/root and mount things like tank/usr and tank/var inside tank/root.

After installing 8.0-RC1 (amd64) from USB stick this installation works
fine. If I csup to RELENG_8 (amd64) and compile + install world and kernel
booting from the ZFS fails. The initial installation I did just this, on
another attempt I ran "gpart bootcode -b /boot/pbmr -p /boot/gptzfsboot
adX" on all disks before rebooting to see if that had any effect. The end
result is the same. After rebooting the machine I get the following
prompt(s):

         ZFS: i/o error - all block copies unavailable
         Invalid format

         FreeBSD/i386 boot
         Default: tank:/boot/kernel/kernel
         boot:

If I wait a while this prompt repeats itself. I've beent rying to test
inside VirtualBox to see if I could report more detail on what's going
wrong but I ran into the same problem as you with the "MOS can't be read"
error.

This machine and zpool don't (at this time) contain any valuable data, so
if you need someone to test anything for you I'd gladly volunteer my box
if it helps resolve things faster.

Kind regards,
Merijn Verstraaten


On Sat, 2009-10-24 at 19:44 +0200, Radek Valášek wrote:

> Robert Noland napsal(a):
> > On Thu, 2009-10-15 at 21:37 +0200, Radek Valášek wrote:
> >  >> Robert Noland napsal(a):
> >>    >>> On Thu, 2009-10-15 at 14:08 +0200, Radek Valášek wrote:
> >>>  >>>      >>>> Hi,
> >>>>
> >>>> I want to ask if there is something new in adding support to>>>>  
> gptzfsboot/zfsboot for reading gang-blocks?
> >>>>   >>>>        >
> > I think that the gang block patch will work, though still haven't  
> gotten
> > it tested.  However, I'm fairly confident that the issue is not gang
> > block related.  Right now, I have setup a disk like this:
> >
> > =>        34  1953525101  ada1  GPT  (932G)
> >           34         128     1  freebsd-boot  (64K)
> >          162     8388608     2  freebsd-swap  (4.0G)
> >      8388770   648019968     3  freebsd-zfs  (309G)
> >    656408738   648019968     4  freebsd-zfs  (309G)
> >   1304428706   648019968     5  freebsd-zfs  (309G)
> >   1952448674     1076461        - free -  (526M)
> >
> > Note that this is not a raidz pool right now.  It is just 3 toplevel
> > partitions setup as a single pool.  I finally have this configuration
> > working reliably.  At least in this case, the issue is due to all of  
> the
> > partitions not being probed during early boot and so not being added to
> > the list of vdevs for the pool.  When zio_read finds a dva that points
> > to a device it doesn't know about, it gives up and whines.
> >
> > Can you detail for me how you have everything configured, so that I can
> > try to replicate it.  gpart show, zpool status and zpool get all <pool>
> > would be good.  I'm not sure that I have enough spare disks lying  
> around
> > to do this properly, but maybe I can use virtual disks or something.
> >
> > robert.
> >
> >  Sorry for not responding so long. Here are details you want from me:
> # gpart show
> =>        34  1953525101  ad6  GPT  (932G)
>           34         128    1  freebsd-boot  (64K)
>          162  1953524973    2  freebsd-zfs  (932G)
> =>        34  1953525101  ad8  GPT  (932G)
>           34         128    1  freebsd-boot  (64K)
>          162  1953524973    2  freebsd-zfs  (932G)
> =>        34  1953525101  ad10  GPT  (932G)
>           34         128     1  freebsd-boot  (64K)
>          162  1953524973     2  freebsd-zfs  (932G)
> =>        34  1953525101  ad12  GPT  (932G)
>           34         128     1  freebsd-boot  (64K)
>          162  1953524973     2  freebsd-zfs  (932G)
> # zpool status
>   pool: z
>  state: ONLINE
>  scrub: none requested
> config:
>    NAME        STATE     READ WRITE CKSUM
>     z           ONLINE       0     0     0
>       raidz1    ONLINE       0     0     0
>         ad6p2   ONLINE       0     0     0
>         ad8p2   ONLINE       0     0     0
>         ad10p2  ONLINE       0     0     0
>         ad12p2  ONLINE       0     0     0
> errors: No known data errors
> # zpool get all z
> NAME  PROPERTY       VALUE       SOURCE
> z     size           3.62T       -
> z     used           4.62G       -
> z     available      3.62T       -
> z     capacity       0%          -
> z     altroot        -           default
> z     health         ONLINE      -
> z     guid           17857007133862981114  -
> z     version        13          default
> z     bootfs         z/system    local
> z     delegation     on          default
> z     autoreplace    off         default
> z     cachefile      -           default
> z     failmode       wait        default
> z     listsnapshots  off         default
> I've tested your patches but it seems that you're right and it's notgang  
> related issue. I was able to discover these things on a fullyfunctional  
> zfs pool (system compiled with your patches):
> 1, If I overwrite the file /boot/loader.conf (with copy of itself, or
> when upgrading kernel/world), next reboot comes with these messages:
> BTX loader 1.00  BTX version is 1.02
> Consoles: internal video/keyboard
> BIOS drive C: is disk0
> BIOS drive D: is disk1
> BIOS drive E: is disk2
> BIOS drive F: is disk3
> BIOS 627kB/3405248kB available memory
> FreeBSD/i386 bootstrap loader, Revision 1.1
> (root at ztest, Thu Oct 22 22:27:22 CEST 2009)
> Loading /boot/defaults/loader.conf
> ZFS: i/o error - all block copies unavailable
> Warning: error reading file /boot/loader.conf
> Then I'm still able to boot the system, but I must set the bootvariables  
> included in loader.conf by hand
> 2, Next I overwrite the file /boot/loader (with copy of itself, or when
> upgrading kernel/world) and reboot comes with these messages:
> BTX loader 1.00  BTX version is 1.02
> Consoles: internal video/keyboard
> BIOS drive C: is disk0
> BIOS drive D: is disk1
> BIOS drive E: is disk2
> BIOS drive F: is disk3
> BIOS 627kB/3405248kB available memory
> FreeBSD/i386 bootstrap loader, Revision 1.1
> (root at ztest, Thu Oct 22 22:27:22 CEST 2009)
> Loading /boot/defaults/loader.conf
> ZFS: i/o error - all block copies unavailable
> Warning: error reading file /boot/loader.conf
> ZFS: i/o error - all block copies unavailable
> ZFS: i/o error - all block copies unavailable
> ZFS: i/o error - all block copies unavailable
> ZFS: i/o error - all block copies unavailable
> ZFS: i/o error - all block copies unavailable
> ZFS: i/o error - all block copies unavailable
> ZFS: i/o error - all block copies unavailable
> ZFS: i/o error - all block copies unavailable
> Unable to load a kernel!
> After that I'm no longer able to boot the system from zfs pool.
> Hope you have some ideas...

_______________________________________________
freebsd-current@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@..."

[Robert Noland]

On Mon, 2009-10-26 at 00:45 +0100, Merijn Verstraaten wrote:

> > Ok, can you retest with -CURRENT?  I just committed some fixes on
> > Friday.  I'm having real difficulty in reproducing these issues.  Most
> > of the problems that I've run into so far had to do with the system not
> > knowing about all of the vdevs when it wanted to read something.  In
> > your case, it looks like you are making it to boot3 and it appears to be
> > seeing all 4 of your disks.  Right now, I've been trying to track down
> > an issue wher the MOS can't be read, which basically means that we have
> > screwed up the root block pointer somehow.  I haven't been able to
> > reproduce that issue in qemu, I have been able to reproduce it with
> > VirtualBox, but it is really time consuming trying to work in vbox since
> > I have to reconvert all of the disk images every time I make a change.
> > I'm actually a bit concerned that it hinges on how many drives are
> > visible to the bios at various points in time.
> > robert.
>
> I noticed this thread in the archives and I've been having the same
> problems as Radek (see belw) with root on ZFS RAID-Z not booting the
> machine. When I install world from the 8.0-RC1 USB image I can boot from
> RAID-Z with no problems. But when I csup'ed to RELENG_8 yesterday
> (Saturday, so this should be after you committed the new changes) and
> build+installed world booting from RAID-Z broke with the exact same
> problems as Radek describes below. I just tried again to be absolutely
> sure (It just occured to me I perhaps should have done "gpart show",
> "zpool status" and "zpool get all z" before breaking the system, but oh
> well...).
>
> I have 4 SATA disks ad4, ad6, ad8 and ad10 with three partitions each,
> adXp1 is a freebsd-boot partition like Radek has, adXp2 is a 1GB
> freebsd-swap partition and adXp3 has the rest of the disk as a freebsd-zfs
> partition. These 4 p3 partitions are placed in a single RAID-Z vdev. zpool
> status looks pretty much like that of Radek too. I have my root partition
> in tank/root and mount things like tank/usr and tank/var inside tank/root.
>
> After installing 8.0-RC1 (amd64) from USB stick this installation works
> fine. If I csup to RELENG_8 (amd64) and compile + install world and kernel
> booting from the ZFS fails. The initial installation I did just this, on
> another attempt I ran "gpart bootcode -b /boot/pbmr -p /boot/gptzfsboot
> adX" on all disks before rebooting to see if that had any effect. The end
> result is the same. After rebooting the machine I get the following
> prompt(s):
>
>          ZFS: i/o error - all block copies unavailable
>          Invalid format
>
>          FreeBSD/i386 boot
>          Default: tank:/boot/kernel/kernel
>          boot:

Could you type "status" at this point and tell what it shows?

I'm convinced that almost all of these issues have to do with what
drives are visible at any given time.  We actually probe for all the
drives twice, once during the early boot (gptzfsboot) and again in stage
3... They don't find the same set of drives in at least some cases.  In
order to successfully boot, we need to see all of the drives at both
stages.  The more drives that you have the more likely it is that we run
into issues.

> If I wait a while this prompt repeats itself. I've beent rying to test
> inside VirtualBox to see if I could report more detail on what's going
> wrong but I ran into the same problem as you with the "MOS can't be read"
> error.

So, something about the virtual box bios and our loader, don't get along
well.  I finally have it booting from a 6 drive raidz2, but what drives
are detected during the early boot depend on what controllers I've told
vbox to use to host the drives.  Basically in order to get it to work, I
had to tell it that the first two drives are on ide and the remaining 4
drives are on sata/scsi.  Once we get to stage 3, the current code only
finds a single hard drive, so at best you have a 25% chance of
booting...  As the blocks get spread around on the drives, we start
reading block pointers that point to devices that we don't know about.
My current hacked up code is now seeing 30 drives in stage 3, which is
also not correct... but it is booting...

robert.

> This machine and zpool don't (at this time) contain any valuable data, so
> if you need someone to test anything for you I'd gladly volunteer my box
> if it helps resolve things faster.
>
> Kind regards,
> Merijn Verstraaten
>
>
> On Sat, 2009-10-24 at 19:44 +0200, Radek Valášek wrote:
> > Robert Noland napsal(a):
> > > On Thu, 2009-10-15 at 21:37 +0200, Radek Valášek wrote:
> > >  >> Robert Noland napsal(a):
> > >>    >>> On Thu, 2009-10-15 at 14:08 +0200, Radek Valášek wrote:
> > >>>  >>>      >>>> Hi,
> > >>>>
> > >>>> I want to ask if there is something new in adding support to>>>>  
> > gptzfsboot/zfsboot for reading gang-blocks?
> > >>>>   >>>>        >
> > > I think that the gang block patch will work, though still haven't  
> > gotten
> > > it tested.  However, I'm fairly confident that the issue is not gang
> > > block related.  Right now, I have setup a disk like this:
> > >
> > > =>        34  1953525101  ada1  GPT  (932G)
> > >           34         128     1  freebsd-boot  (64K)
> > >          162     8388608     2  freebsd-swap  (4.0G)
> > >      8388770   648019968     3  freebsd-zfs  (309G)
> > >    656408738   648019968     4  freebsd-zfs  (309G)
> > >   1304428706   648019968     5  freebsd-zfs  (309G)
> > >   1952448674     1076461        - free -  (526M)
> > >
> > > Note that this is not a raidz pool right now.  It is just 3 toplevel
> > > partitions setup as a single pool.  I finally have this configuration
> > > working reliably.  At least in this case, the issue is due to all of  
> > the
> > > partitions not being probed during early boot and so not being added to
> > > the list of vdevs for the pool.  When zio_read finds a dva that points
> > > to a device it doesn't know about, it gives up and whines.
> > >
> > > Can you detail for me how you have everything configured, so that I can
> > > try to replicate it.  gpart show, zpool status and zpool get all <pool>
> > > would be good.  I'm not sure that I have enough spare disks lying  
> > around
> > > to do this properly, but maybe I can use virtual disks or something.
> > >
> > > robert.
> > >
> > >  Sorry for not responding so long. Here are details you want from me:
> > # gpart show
> > =>        34  1953525101  ad6  GPT  (932G)
> >           34         128    1  freebsd-boot  (64K)
> >          162  1953524973    2  freebsd-zfs  (932G)
> > =>        34  1953525101  ad8  GPT  (932G)
> >           34         128    1  freebsd-boot  (64K)
> >          162  1953524973    2  freebsd-zfs  (932G)
> > =>        34  1953525101  ad10  GPT  (932G)
> >           34         128     1  freebsd-boot  (64K)
> >          162  1953524973     2  freebsd-zfs  (932G)
> > =>        34  1953525101  ad12  GPT  (932G)
> >           34         128     1  freebsd-boot  (64K)
> >          162  1953524973     2  freebsd-zfs  (932G)
> > # zpool status
> >   pool: z
> >  state: ONLINE
> >  scrub: none requested
> > config:
> >    NAME        STATE     READ WRITE CKSUM
> >     z           ONLINE       0     0     0
> >       raidz1    ONLINE       0     0     0
> >         ad6p2   ONLINE       0     0     0
> >         ad8p2   ONLINE       0     0     0
> >         ad10p2  ONLINE       0     0     0
> >         ad12p2  ONLINE       0     0     0
> > errors: No known data errors
> > # zpool get all z
> > NAME  PROPERTY       VALUE       SOURCE
> > z     size           3.62T       -
> > z     used           4.62G       -
> > z     available      3.62T       -
> > z     capacity       0%          -
> > z     altroot        -           default
> > z     health         ONLINE      -
> > z     guid           17857007133862981114  -
> > z     version        13          default
> > z     bootfs         z/system    local
> > z     delegation     on          default
> > z     autoreplace    off         default
> > z     cachefile      -           default
> > z     failmode       wait        default
> > z     listsnapshots  off         default
> > I've tested your patches but it seems that you're right and it's notgang  
> > related issue. I was able to discover these things on a fullyfunctional  
> > zfs pool (system compiled with your patches):
> > 1, If I overwrite the file /boot/loader.conf (with copy of itself, or
> > when upgrading kernel/world), next reboot comes with these messages:
> > BTX loader 1.00  BTX version is 1.02
> > Consoles: internal video/keyboard
> > BIOS drive C: is disk0
> > BIOS drive D: is disk1
> > BIOS drive E: is disk2
> > BIOS drive F: is disk3
> > BIOS 627kB/3405248kB available memory
> > FreeBSD/i386 bootstrap loader, Revision 1.1
> > (root at ztest, Thu Oct 22 22:27:22 CEST 2009)
> > Loading /boot/defaults/loader.conf
> > ZFS: i/o error - all block copies unavailable
> > Warning: error reading file /boot/loader.conf
> > Then I'm still able to boot the system, but I must set the bootvariables  
> > included in loader.conf by hand
> > 2, Next I overwrite the file /boot/loader (with copy of itself, or when
> > upgrading kernel/world) and reboot comes with these messages:
> > BTX loader 1.00  BTX version is 1.02
> > Consoles: internal video/keyboard
> > BIOS drive C: is disk0
> > BIOS drive D: is disk1
> > BIOS drive E: is disk2
> > BIOS drive F: is disk3
> > BIOS 627kB/3405248kB available memory
> > FreeBSD/i386 bootstrap loader, Revision 1.1
> > (root at ztest, Thu Oct 22 22:27:22 CEST 2009)
> > Loading /boot/defaults/loader.conf
> > ZFS: i/o error - all block copies unavailable
> > Warning: error reading file /boot/loader.conf
> > ZFS: i/o error - all block copies unavailable
> > ZFS: i/o error - all block copies unavailable
> > ZFS: i/o error - all block copies unavailable
> > ZFS: i/o error - all block copies unavailable
> > ZFS: i/o error - all block copies unavailable
> > ZFS: i/o error - all block copies unavailable
> > ZFS: i/o error - all block copies unavailable
> > ZFS: i/o error - all block copies unavailable
> > Unable to load a kernel!
> > After that I'm no longer able to boot the system from zfs pool.
> > Hope you have some ideas...
> _______________________________________________
> freebsd-current@... mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe@..."

--
Robert Noland <rnoland@...>
FreeBSD

_______________________________________________
freebsd-current@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@..."

[Merijn Verstraaten-2]

On Mon, 26 Oct 2009 01:31:46 +0100, Robert Noland <rnoland@...>  
wrote:

>> After installing 8.0-RC1 (amd64) from USB stick this installation works
>> fine. If I csup to RELENG_8 (amd64) and compile + install world and  
>> kernel
>> booting from the ZFS fails. The initial installation I did just this, on
>> another attempt I ran "gpart bootcode -b /boot/pbmr -p /boot/gptzfsboot
>> adX" on all disks before rebooting to see if that had any effect. The  
>> end
>> result is the same. After rebooting the machine I get the following
>> prompt(s):
>>
>>          ZFS: i/o error - all block copies unavailable
>>          Invalid format
>>
>>          FreeBSD/i386 boot
>>          Default: tank:/boot/kernel/kernel
>>          boot:
>
> Could you type "status" at this point and tell what it shows?

If I type status at this point I get:

   pool: tank
   config:
       NAME    STATE
       tank   ONLINE
     raidz1   ONLINE
      ad4p3   ONLINE
      ad6p3   ONLINE
      ad8p3   ONLINE
     ad10p3   ONLINE

Which seems odd, since that's all the drives there are. So if it finds  
these it's already found all drives. My optimistic "Oh! I'll try and boot  
again" spirit was however crushed since it just results in the same error.

Kind regards,
Merijn
_______________________________________________
freebsd-current@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@..."

[Robert Noland]

On Mon, 2009-10-26 at 10:23 +0100, Merijn Verstraaten wrote:

> On Mon, 26 Oct 2009 01:31:46 +0100, Robert Noland <rnoland@...>  
> wrote:
> >> After installing 8.0-RC1 (amd64) from USB stick this installation works
> >> fine. If I csup to RELENG_8 (amd64) and compile + install world and  
> >> kernel
> >> booting from the ZFS fails. The initial installation I did just this, on
> >> another attempt I ran "gpart bootcode -b /boot/pbmr -p /boot/gptzfsboot
> >> adX" on all disks before rebooting to see if that had any effect. The  
> >> end
> >> result is the same. After rebooting the machine I get the following
> >> prompt(s):
> >>
> >>          ZFS: i/o error - all block copies unavailable
> >>          Invalid format
> >>
> >>          FreeBSD/i386 boot
> >>          Default: tank:/boot/kernel/kernel
> >>          boot:
> >
> > Could you type "status" at this point and tell what it shows?
>
> If I type status at this point I get:
>
>    pool: tank
>    config:
>        NAME    STATE
>        tank   ONLINE
>      raidz1   ONLINE
>       ad4p3   ONLINE
>       ad6p3   ONLINE
>       ad8p3   ONLINE
>      ad10p3   ONLINE
>
> Which seems odd, since that's all the drives there are. So if it finds  
> these it's already found all drives. My optimistic "Oh! I'll try and boot  
> again" spirit was however crushed since it just results in the same error.

Ok, that is both good and frustrating... I haven't produced any boot
failures with all of the drives visible.  Do, note that I just added
support for reading gang blocks to the loader. (basically untested,
since I haven't managed to create them at will)  You will need to update
your partition boot code for it to be supported during early boot.  i.e.
gpart bootcode -p /boot/gptzfsboot -i <boot partition> <disk>

The "all block copies unavailable" is a frustrating error, since all it
means is a failed read, but we don't get a clue what failed or why.
With the code that is in -CURRENT it will report gang blocks if found,
even if it fails to read them.
 
robert.
 
> Kind regards,
> Merijn
> _______________________________________________
> freebsd-current@... mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe@..."
--
Robert Noland <rnoland@...>
FreeBSD

_______________________________________________
freebsd-current@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@..."

[Radek Valášek-2]

Robert Noland napsal(a):

> On Mon, 2009-10-26 at 10:23 +0100, Merijn Verstraaten wrote:
>  
>> On Mon, 26 Oct 2009 01:31:46 +0100, Robert Noland <rnoland@...>  
>> wrote:
>>    
>>>> After installing 8.0-RC1 (amd64) from USB stick this installation works
>>>> fine. If I csup to RELENG_8 (amd64) and compile + install world and  
>>>> kernel
>>>> booting from the ZFS fails. The initial installation I did just this, on
>>>> another attempt I ran "gpart bootcode -b /boot/pbmr -p /boot/gptzfsboot
>>>> adX" on all disks before rebooting to see if that had any effect. The  
>>>> end
>>>> result is the same. After rebooting the machine I get the following
>>>> prompt(s):
>>>>
>>>>          ZFS: i/o error - all block copies unavailable
>>>>          Invalid format
>>>>
>>>>          FreeBSD/i386 boot
>>>>          Default: tank:/boot/kernel/kernel
>>>>          boot:
>>>>        
>>> Could you type "status" at this point and tell what it shows?
>>>      
>> If I type status at this point I get:
>>
>>    pool: tank
>>    config:
>>        NAME    STATE
>>        tank   ONLINE
>>      raidz1   ONLINE
>>       ad4p3   ONLINE
>>       ad6p3   ONLINE
>>       ad8p3   ONLINE
>>      ad10p3   ONLINE
>>
>> Which seems odd, since that's all the drives there are. So if it finds  
>> these it's already found all drives. My optimistic "Oh! I'll try and boot  
>> again" spirit was however crushed since it just results in the same error.
>>    
>
> Ok, that is both good and frustrating... I haven't produced any boot
> failures with all of the drives visible.  Do, note that I just added
> support for reading gang blocks to the loader. (basically untested,
> since I haven't managed to create them at will)  You will need to update
> your partition boot code for it to be supported during early boot.  i.e.
> gpart bootcode -p /boot/gptzfsboot -i <boot partition> <disk>
>
> The "all block copies unavailable" is a frustrating error, since all it
> means is a failed read, but we don't get a clue what failed or why.
> With the code that is in -CURRENT it will report gang blocks if found,
> even if it fails to read them.
>  
> robert.
>  

So I switched to -CURRENT:

1, overwriting /boot/loader.conf results with:

BTX loader 1.00 BTX version is 1.02
Consoles: internal video/keyboard
BIOS drive C: is disk0
BIOS drive D: is disk1
BIOS drive E: is disk2
BIOS drive F: is disk3
BIOS 627kB/3405248kB available memory

FreeBSD/i386 bootstrap loader, Revision 1.1
(root@ztest, Mon Oct 26 14:01:44 CEST 2009)
Loading /boot/defaults/loader.conf
ZFS: i/o error - all block copies unavailable
Warning: error reading file /boot/loader.conf

so basically the same as in RELENG_8

2, + overwriting /boot/loader results with:

ZFS: i/o error - all block copies unavailable
Invalid format

FreeBSD/i386 boot
Default: z:/boot/kernel/kernel
boot:
\
int=00000001  err=00000000  efl=00000087 eip=0018d27d
eax=0018d2af  ebx=18bf9925  ecx=540d8ef2  edx=00000000
esi=00009401  edi=000919d0  ebp=36571125  esp=80000000
cs=0008  ds=0010  es=0010  fs=0010  gs=0010  ss=0010
cs:eip=1f 68 e2 c6 7d 75 0c 5d-45 58 c7 80 f5 99 bd 9e
       fe 68 2d 3e 3c 35 5e 67-61 12 fe 50 c9 0b e4 70
ss:esp=00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00
       00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00
BTX halted

3, I also try the 'status' as you told to Merijn before BTX halted:

ZFS: i/o error - all block copies unavailable
Invalid format

FreeBSD/i386 boot
Default: z:/boot/kernel/kernel
boot: status  pool: z
config:
NAME  STATE
z  ONLINE
raidz1  ONLINE
ad6p2  ONLINE
ad8p2  ONLINE
ad10p2  ONLINE
ad12p2  ONLINE

radek.

>  
>  
>> Kind regards,
>> Merijn
>> _______________________________________________
>> freebsd-current@... mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
>> To unsubscribe, send any mail to "freebsd-current-unsubscribe@..."
>>    

_______________________________________________
freebsd-current@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@..."

[Radek Valášek-2]

Robert Noland napsal(a):

> On Mon, 2009-10-26 at 10:23 +0100, Merijn Verstraaten wrote:
>  
>> On Mon, 26 Oct 2009 01:31:46 +0100, Robert Noland <rnoland@...>  
>> wrote:
>>    
>>>> After installing 8.0-RC1 (amd64) from USB stick this installation works
>>>> fine. If I csup to RELENG_8 (amd64) and compile + install world and  
>>>> kernel
>>>> booting from the ZFS fails. The initial installation I did just this, on
>>>> another attempt I ran "gpart bootcode -b /boot/pbmr -p /boot/gptzfsboot
>>>> adX" on all disks before rebooting to see if that had any effect. The  
>>>> end
>>>> result is the same. After rebooting the machine I get the following
>>>> prompt(s):
>>>>
>>>>          ZFS: i/o error - all block copies unavailable
>>>>          Invalid format
>>>>
>>>>          FreeBSD/i386 boot
>>>>          Default: tank:/boot/kernel/kernel
>>>>          boot:
>>>>        
>>> Could you type "status" at this point and tell what it shows?
>>>      
>> If I type status at this point I get:
>>
>>    pool: tank
>>    config:
>>        NAME    STATE
>>        tank   ONLINE
>>      raidz1   ONLINE
>>       ad4p3   ONLINE
>>       ad6p3   ONLINE
>>       ad8p3   ONLINE
>>      ad10p3   ONLINE
>>
>> Which seems odd, since that's all the drives there are. So if it finds  
>> these it's already found all drives. My optimistic "Oh! I'll try and boot  
>> again" spirit was however crushed since it just results in the same error.
>>    
>
> Ok, that is both good and frustrating... I haven't produced any boot
> failures with all of the drives visible.  Do, note that I just added
> support for reading gang blocks to the loader. (basically untested,
> since I haven't managed to create them at will)  You will need to update
> your partition boot code for it to be supported during early boot.  i.e.
> gpart bootcode -p /boot/gptzfsboot -i <boot partition> <disk>
>
> The "all block copies unavailable" is a frustrating error, since all it
> means is a failed read, but we don't get a clue what failed or why.
> With the code that is in -CURRENT it will report gang blocks if found,
> even if it fails to read them.
>  
> robert.
>  

So I switched to -CURRENT:

1, overwriting /boot/loader.conf results with:

BTX loader 1.00 BTX version is 1.02
Consoles: internal video/keyboard
BIOS drive C: is disk0
BIOS drive D: is disk1
BIOS drive E: is disk2
BIOS drive F: is disk3
BIOS 627kB/3405248kB available memory

FreeBSD/i386 bootstrap loader, Revision 1.1
(root@ztest, Mon Oct 26 14:01:44 CEST 2009)
Loading /boot/defaults/loader.conf
ZFS: i/o error - all block copies unavailable
Warning: error reading file /boot/loader.conf

so basically the same as in RELENG_8

2, + overwriting /boot/loader results with:

ZFS: i/o error - all block copies unavailable
Invalid format

FreeBSD/i386 boot
Default: z:/boot/kernel/kernel
boot:
\
int=00000001  err=00000000  efl=00000087 eip=0018d27d
eax=0018d2af  ebx=18bf9925  ecx=540d8ef2  edx=00000000
esi=00009401  edi=000919d0  ebp=36571125  esp=80000000
cs=0008  ds=0010  es=0010  fs=0010  gs=0010  ss=0010
cs:eip=1f 68 e2 c6 7d 75 0c 5d-45 58 c7 80 f5 99 bd 9e
       fe 68 2d 3e 3c 35 5e 67-61 12 fe 50 c9 0b e4 70
ss:esp=00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00
       00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00
BTX halted

3, I also try the 'status' as you told to Merijn before BTX halted:

ZFS: i/o error - all block copies unavailable
Invalid format

FreeBSD/i386 boot
Default: z:/boot/kernel/kernel
boot: status  pool: z
config:
NAME  STATE
z  ONLINE
raidz1  ONLINE
ad6p2  ONLINE
ad8p2  ONLINE
ad10p2  ONLINE
ad12p2  ONLINE

radek.

>  
>  
>> Kind regards,
>> Merijn
>> _______________________________________________
>> freebsd-current@... mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
>> To unsubscribe, send any mail to "freebsd-current-unsubscribe@..."
>>    


_______________________________________________
freebsd-current@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@..."

[Merijn Verstraaten-2]

My apologies to Robert for the double message, mis-addressed the first  
reply to him instead of the list.

On Mon, 26 Oct 2009 16:34:59 +0100, Robert Noland <rnoland@...>
wrote:

>> If I type status at this point I get:
>>
>>    pool: tank
>>    config:
>>        NAME    STATE
>>        tank   ONLINE
>>      raidz1   ONLINE
>>       ad4p3   ONLINE
>>       ad6p3   ONLINE
>>       ad8p3   ONLINE
>>      ad10p3   ONLINE
>>
>> Which seems odd, since that's all the drives there are. So if it finds
>> these it's already found all drives. My optimistic "Oh! I'll try and  
>> boot
>> again" spirit was however crushed since it just results in the same  
>> error.
>
> Ok, that is both good and frustrating... I haven't produced any boot
> failures with all of the drives visible.  Do, note that I just added
> support for reading gang blocks to the loader. (basically untested,
> since I haven't managed to create them at will)  You will need to update
> your partition boot code for it to be supported during early boot.  i.e.
> gpart bootcode -p /boot/gptzfsboot -i <boot partition> <disk>

I tried again yesterday evening by recompiling RELENG_8 and -CURRENT. I
somehow managed to boot into RELENG_8 the first time, but after that this
error comes up:

     ZFS: i/o error - all block copies unavailable
     ZFS: can't read object set for dataset ldd
     Can't find root filesystem - giving up
     ZFS: unexpected object set type ldd
     ZFS: unexpected object set type ldd

     FreeBSD/i386 boot
     Default:/ tank:/boot/kernel/kernel
     boot:
     ZFS: unexpected object set type ldd

     FreeBSD/i386 boot
     Default:/ tank:/boot/kernel/kernel
     boot: status

      pool: tank
      config:
          NAME    STATE
          tank   ONLINE
        raidz1   ONLINE
         ad4p3   ONLINE
         ad6p3   ONLINE
         ad8p3   ONLINE
        ad10p3   ONLINE

After recompiling world/kernel for -CURRENT I get roughly the same error:

     ZFS: i/o error - all block copies unavailable
     ZFS: can't read object set for dataset 30
     Can't find root filesystem - giving up
     ZFS: unexpected object set type 0
     ZFS: unexpected object set type 0

     FreeBSD/i386 boot
     Default:/ tank:/boot/kernel/kernel
     boot:
     ZFS: unexpected object set type ldd

     FreeBSD/i386 boot
     Default:/ tank:/boot/kernel/kernel
     boot: status

      pool: tank
      config:
          NAME    STATE
          tank   ONLINE
        raidz1   ONLINE
         ad4p3   ONLINE
         ad6p3   ONLINE
         ad8p3   ONLINE
        ad10p3   ONLINE

> The "all block copies unavailable" is a frustrating error, since all it
> means is a failed read, but we don't get a clue what failed or why.
> With the code that is in -CURRENT it will report gang blocks if found,
> even if it fails to read them.
> robert.

I've seen no mention of gang blocks in the errors so far.

Kind regards,
Merijn
_______________________________________________
freebsd-current@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@..."

[Robert Noland]

On Tue, 2009-10-27 at 11:17 +0100, Merijn Verstraaten wrote:

> My apologies to Robert for the double message, mis-addressed the first  
> reply to him instead of the list.
>
> On Mon, 26 Oct 2009 16:34:59 +0100, Robert Noland <rnoland@...>
> wrote:
> >> If I type status at this point I get:
> >>
> >>    pool: tank
> >>    config:
> >>        NAME    STATE
> >>        tank   ONLINE
> >>      raidz1   ONLINE
> >>       ad4p3   ONLINE
> >>       ad6p3   ONLINE
> >>       ad8p3   ONLINE
> >>      ad10p3   ONLINE
> >>
> >> Which seems odd, since that's all the drives there are. So if it finds
> >> these it's already found all drives. My optimistic "Oh! I'll try and  
> >> boot
> >> again" spirit was however crushed since it just results in the same  
> >> error.
> >
> > Ok, that is both good and frustrating... I haven't produced any boot
> > failures with all of the drives visible.  Do, note that I just added
> > support for reading gang blocks to the loader. (basically untested,
> > since I haven't managed to create them at will)  You will need to update
> > your partition boot code for it to be supported during early boot.  i.e.
> > gpart bootcode -p /boot/gptzfsboot -i <boot partition> <disk>
>
> I tried again yesterday evening by recompiling RELENG_8 and -CURRENT. I
> somehow managed to boot into RELENG_8 the first time, but after that this
> error comes up:
>
>      ZFS: i/o error - all block copies unavailable
>      ZFS: can't read object set for dataset ldd
>      Can't find root filesystem - giving up
>      ZFS: unexpected object set type ldd
>      ZFS: unexpected object set type ldd
>
>      FreeBSD/i386 boot
>      Default:/ tank:/boot/kernel/kernel
>      boot:
>      ZFS: unexpected object set type ldd
>
>      FreeBSD/i386 boot
>      Default:/ tank:/boot/kernel/kernel
>      boot: status
>
>       pool: tank
>       config:
>           NAME    STATE
>           tank   ONLINE
>         raidz1   ONLINE
>          ad4p3   ONLINE
>          ad6p3   ONLINE
>          ad8p3   ONLINE
>         ad10p3   ONLINE
>
> After recompiling world/kernel for -CURRENT I get roughly the same error:
>
>      ZFS: i/o error - all block copies unavailable
>      ZFS: can't read object set for dataset 30
>      Can't find root filesystem - giving up
>      ZFS: unexpected object set type 0
>      ZFS: unexpected object set type 0
>
>      FreeBSD/i386 boot
>      Default:/ tank:/boot/kernel/kernel

               ^^
This looks strange... Do you have bootfs set to /, or something in
loader.conf?  Does it work if you just type "tank:/boot/kernel/kernel"
at this point?

robert.

>      boot:
>      ZFS: unexpected object set type ldd
>
>      FreeBSD/i386 boot
>      Default:/ tank:/boot/kernel/kernel
>      boot: status
>
>       pool: tank
>       config:
>           NAME    STATE
>           tank   ONLINE
>         raidz1   ONLINE
>          ad4p3   ONLINE
>          ad6p3   ONLINE
>          ad8p3   ONLINE
>         ad10p3   ONLINE
>
> > The "all block copies unavailable" is a frustrating error, since all it
> > means is a failed read, but we don't get a clue what failed or why.
> > With the code that is in -CURRENT it will report gang blocks if found,
> > even if it fails to read them.
> > robert.
>
> I've seen no mention of gang blocks in the errors so far.
>
> Kind regards,
> Merijn
> _______________________________________________
> freebsd-current@... mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe@..."

--
Robert Noland <rnoland@...>
FreeBSD

_______________________________________________
freebsd-current@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@..."

[Merijn Verstraaten-2]

On Tue, 27 Oct 2009 15:49:36 +0100, Robert Noland <rnoland@...>  
wrote:

>> I tried again yesterday evening by recompiling RELENG_8 and -CURRENT. I
>> somehow managed to boot into RELENG_8 the first time, but after that  
>> this
>> error comes up:
>>
>>      ZFS: i/o error - all block copies unavailable
>>      ZFS: can't read object set for dataset ldd
>>      Can't find root filesystem - giving up
>>      ZFS: unexpected object set type ldd
>>      ZFS: unexpected object set type ldd
>>
>>      FreeBSD/i386 boot
>>      Default:/ tank:/boot/kernel/kernel
>>      boot:
>>      ZFS: unexpected object set type ldd
>>
>>      FreeBSD/i386 boot
>>      Default:/ tank:/boot/kernel/kernel
>>      boot: status
>>
>>       pool: tank
>>       config:
>>           NAME    STATE
>>           tank   ONLINE
>>         raidz1   ONLINE
>>          ad4p3   ONLINE
>>          ad6p3   ONLINE
>>          ad8p3   ONLINE
>>         ad10p3   ONLINE
>>
>> After recompiling world/kernel for -CURRENT I get roughly the same  
>> error:
>>
>>      ZFS: i/o error - all block copies unavailable
>>      ZFS: can't read object set for dataset 30
>>      Can't find root filesystem - giving up
>>      ZFS: unexpected object set type 0
>>      ZFS: unexpected object set type 0
>>
>>      FreeBSD/i386 boot
>>      Default:/ tank:/boot/kernel/kernel
>                ^^
> This looks strange... Do you have bootfs set to /, or something in
> loader.conf?  Does it work if you just type "tank:/boot/kernel/kernel"
> at this point?
>
> robert.

I think this might be user error. I just checked and the leading / is  
absent on my screen:

   ZFS: i/o error - all block copies unavailable
   ZFS: can't read object set for dataset 30
   Can't find root filesystem - giving up
   ZFS: unexpected object set type 0
   ZFS: unexpected object set type 0

   FreeBSD/i386 boot
   Default: tank:/boot/kernel/kernel
   boot:

I probably just typo'ed it this morning. As clarification I have  
vfs.root.mountfrom="zfs:tank/root" in loader.conf with my root filesystem  
installed on tank/root and then tank/usr, tank/var etc mounted on  
tank/root. If you need more detail my current setup procedure is:

"gpart create -s GPT <disk>"
"gpart add -b 34 -s 128 -t freebsd-boot <disk>"
"gpart add -b 162 -s 1G -t freebsd-swap <disk>"
"gpart add -t freebsd-zfs <disk>"

"zpool create tank raidz <disk1> <disk2> <diskN>"

"zfs set checksum=fletcher4 tank"

"zfs create -o reserv=512m tank/root"

"zfs create -o mountpoint=/tank/root/usr tank/usr"
"zfs create -o mountpoint=/tank/root/tmp tank/tmp"
"zfs create -o mountpoint=/tank/root/var tank/var"
"zfs create -o mountpoint=/tank/root/home tank/home"
"zfs create tank/usr/obj"
"zfs create tank/usr/src"
"zfs create tank/usr/ports"

export DESTDIR=/tank/root
Run the ./install.sh scripts in the various directories of the dist


"mkdir /boot/zfs"
"zpool export tank && zpool import tank"
"cp /boot/zfs/zpool.cache /tank/root/boot/zfs/"

Set 'LOADER_ZFS_SUPPORT=yes' in /tank/root/etc/src.conf

"chroot /tank/root"
"mount -t devfs devfs /dev"
"unset DESTDIR"
"cd /usr/src/sys/boot/"
"make obj"
"make depend"
"make"
"cd i386/loader"
"make install"
"umount /dev"
"exit"
"export LD_LIBRARY_PATH=/dist/lib"

"gpart bootcode -b /tank/root/boot/pmbr -p /tank/root/boot/gptzfsboot -i 1  
<disk>"

/boot/loader.conf:
     zfs_load="YES"
     vfs.root.mountfrom="zfs:tank/root"

/etc/rc.conf:
     zfs_enable="YES"

"zfs umount -a"
"zfs set mountpoint=legacy tank"
"zfs set mountpoint=/tmp tank/tmp"
"zfs set mountpoint=/var tank/var"
"zfs set mountpoint=/usr tank/usr"
"zfs set mountpoint=/home tank/home"
"zpool set bootfs=tank/root tank"

Then reboot, csup sources to whatever version to test and build  
world/kernel, install and watch things breaks.

Kind regards,
Merijn
_______________________________________________
freebsd-current@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@..."

[Norikatsu Shigemura-3]

Hi rnoland.

On Mon, 26 Oct 2009 10:34:59 -0500
Robert Noland <rnoland@...> wrote:

> > >>          ZFS: i/o error - all block copies unavailable
> > >>          Invalid format
> > >>          FreeBSD/i386 boot
> > >>          Default: tank:/boot/kernel/kernel
> > >>          boot:
> > > Could you type "status" at this point and tell what it shows?
> > If I type status at this point I get:
> >    pool: tank
> >    config:
> >        NAME    STATE
> >        tank   ONLINE
> >      raidz1   ONLINE
> >       ad4p3   ONLINE
> >       ad6p3   ONLINE
> >       ad8p3   ONLINE
> >      ad10p3   ONLINE
> > Which seems odd, since that's all the drives there are. So if it finds  
> > these it's already found all drives. My optimistic "Oh! I'll try and boot  
> > again" spirit was however crushed since it just results in the same error.
> Ok, that is both good and frustrating... I haven't produced any boot
> failures with all of the drives visible.  Do, note that I just added
> support for reading gang blocks to the loader. (basically untested,
> since I haven't managed to create them at will)  You will need to update
> your partition boot code for it to be supported during early boot.  i.e.
> gpart bootcode -p /boot/gptzfsboot -i <boot partition> <disk>
> The "all block copies unavailable" is a frustrating error, since all it
> means is a failed read, but we don't get a clue what failed or why.
> With the code that is in -CURRENT it will report gang blocks if found,
> even if it fails to read them.

        I confirmed reproduce.

        1. zpool list, and get SIZE and CAP.
                $ zpool list
                NAME    SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
                tank   59.5G  48.4G  11.1G    81%  ONLINE  -

        2. reduce AVAIL < 10% with creating dummy file like ...
                $ dd if=/dev/zero of=$HOME/DUMMY.FILE bs=1m count=5632
                5632+0 records in
                5632+0 records out
                5905580032 bytes transferred in 49.822200 secs (118533104 bytes/sec)
                $ zpool list
                NAME    SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
                tank   59.5G  53.9G  5.61G    90%  ONLINE  -

        3. cd /boot/; cp -pr kernel kernel.err
           In this time, if reboot, we can get boot time error.

        4. rm $HOME/DUMMY.FILE, and reboot

        5. boot kernel.err on new-loader.
           I can get "ZFS: gang block detected!" message and overrun:D.
_______________________________________________
freebsd-current@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@..."

[Robert Noland]

On Wed, 2009-10-28 at 07:17 +0900, Norikatsu Shigemura wrote:

> Hi rnoland.
>
> On Mon, 26 Oct 2009 10:34:59 -0500
> Robert Noland <rnoland@...> wrote:
> > > >>          ZFS: i/o error - all block copies unavailable
> > > >>          Invalid format
> > > >>          FreeBSD/i386 boot
> > > >>          Default: tank:/boot/kernel/kernel
> > > >>          boot:
> > > > Could you type "status" at this point and tell what it shows?
> > > If I type status at this point I get:
> > >    pool: tank
> > >    config:
> > >        NAME    STATE
> > >        tank   ONLINE
> > >      raidz1   ONLINE
> > >       ad4p3   ONLINE
> > >       ad6p3   ONLINE
> > >       ad8p3   ONLINE
> > >      ad10p3   ONLINE
> > > Which seems odd, since that's all the drives there are. So if it finds  
> > > these it's already found all drives. My optimistic "Oh! I'll try and boot  
> > > again" spirit was however crushed since it just results in the same error.
> > Ok, that is both good and frustrating... I haven't produced any boot
> > failures with all of the drives visible.  Do, note that I just added
> > support for reading gang blocks to the loader. (basically untested,
> > since I haven't managed to create them at will)  You will need to update
> > your partition boot code for it to be supported during early boot.  i.e.
> > gpart bootcode -p /boot/gptzfsboot -i <boot partition> <disk>
> > The "all block copies unavailable" is a frustrating error, since all it
> > means is a failed read, but we don't get a clue what failed or why.
> > With the code that is in -CURRENT it will report gang blocks if found,
> > even if it fails to read them.
>
> I confirmed reproduce.
>
> 1. zpool list, and get SIZE and CAP.
> $ zpool list
> NAME    SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
> tank   59.5G  48.4G  11.1G    81%  ONLINE  -
>
> 2. reduce AVAIL < 10% with creating dummy file like ...
> $ dd if=/dev/zero of=$HOME/DUMMY.FILE bs=1m count=5632
> 5632+0 records in
> 5632+0 records out
> 5905580032 bytes transferred in 49.822200 secs (118533104 bytes/sec)
> $ zpool list
> NAME    SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
> tank   59.5G  53.9G  5.61G    90%  ONLINE  -
>
> 3. cd /boot/; cp -pr kernel kernel.err
>   In this time, if reboot, we can get boot time error.
>
> 4. rm $HOME/DUMMY.FILE, and reboot
>
> 5. boot kernel.err on new-loader.
>   I can get "ZFS: gang block detected!" message and overrun:D.

Ok, so does it still boot?  Or do you still get an error?

robert.

--
Robert Noland <rnoland@...>
FreeBSD

_______________________________________________
freebsd-current@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@..."

[Norikatsu Shigemura-3]

Hi rnoland.

On Tue, 27 Oct 2009 18:19:15 -0500
Robert Noland <rnoland@...> wrote:

> > 2. reduce AVAIL < 10% with creating dummy file like ...
> > $ dd if=/dev/zero of=$HOME/DUMMY.FILE bs=1m count=5632
> > 5632+0 records in
> > 5632+0 records out
> > 5905580032 bytes transferred in 49.822200 secs (118533104 bytes/sec)
> > $ zpool list
> > NAME    SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
> > tank   59.5G  53.9G  5.61G    90%  ONLINE  -
> > 3. cd /boot/; cp -pr kernel kernel.err
> >   In this time, if reboot, we can get boot time error.
> > 4. rm $HOME/DUMMY.FILE, and reboot
> > 5. boot kernel.err on new-loader.
> >   I can get "ZFS: gang block detected!" message and overrun:D.
> Ok, so does it still boot?  Or do you still get an error?

        After reboot:
        OK: boot kernel
        NG: boot kernel.err

        Sorry, kernel.err is ganged kernel.  I should be called as
        kernel.gang.
_______________________________________________
freebsd-current@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@..."

[Krzysztof Dajka]

I still can't boot. I don't know whether I missed it in first time,
but I'm getting
>ZFS: i/o error - all block copies unavailable
x6 before seeing boot manager. I tried booting with verbose option,
but there was only laconic

>Trying to mount root from zfs:zroot
>ROOT MOUNT ERROR

When I execute lsdev I'm getting
....
disk devices:
       disk0: BIOS drive C:
            disk0p1: FreeBSD boot
            disk0p2: FreeBSD swap
            disk0p3: FreeBSD ZFS
       disk1: BIOS drive D:
            disk1p1: FreeBSD boot
            disk1p2: FreeBSD swap
            disk1p3: FreeBSD ZFS
       disk2: BIOS drive E:
            disk2p1: FreeBSD boot
            disk2p2: FreeBSD swap
            disk2p3: FreeBSD ZFS
pxe devices:
zfs devices:
       zfs0:      zroot

I can also do `ls` and I can browse whole zroot, and everything seems
to be there. At least for the boot loader.
_______________________________________________
freebsd-current@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@..."

[Robert Noland]

On Wed, 2009-10-28 at 21:13 +0100, Krzysztof Dajka wrote:
> I still can't boot. I don't know whether I missed it in first time,
> but I'm getting
> >ZFS: i/o error - all block copies unavailable
> x6 before seeing boot manager. I tried booting with verbose option,
> but there was only laconic

If you hit a key very early then you will land in boot2, where you can
type "status" and it will list the zfs pool.

robert.

> >Trying to mount root from zfs:zroot
> >ROOT MOUNT ERROR
>
> When I execute lsdev I'm getting
> ....
> disk devices:
>        disk0: BIOS drive C:
>             disk0p1: FreeBSD boot
>             disk0p2: FreeBSD swap
>             disk0p3: FreeBSD ZFS
>        disk1: BIOS drive D:
>             disk1p1: FreeBSD boot
>             disk1p2: FreeBSD swap
>             disk1p3: FreeBSD ZFS
>        disk2: BIOS drive E:
>             disk2p1: FreeBSD boot
>             disk2p2: FreeBSD swap
>             disk2p3: FreeBSD ZFS
> pxe devices:
> zfs devices:
>        zfs0:      zroot
>
> I can also do `ls` and I can browse whole zroot, and everything seems
> to be there. At least for the boot loader.
> _______________________________________________
> freebsd-current@... mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe@..."

--
Robert Noland <rnoland@...>
FreeBSD

_______________________________________________
freebsd-current@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@..."

[Krzysztof Dajka]

On Wednesday 28 October 2009 21:43:07 Robert Noland wrote:
> If you hit a key very early then you will land in boot2, where you can
> type "status" and it will list the zfs pool.

I haven't tried it yet. Would it allow me to mount root?
I tried to install from fixit once again, but I've made a script to eliminate
all typo's I could made. I amde script using most of:
http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/RAIDZ1

After rebooting it could even load bootloader:
> ZFS: i/o error - all block copies unavailable
> ZFS: can't read MOS
> ZFS: unexpected object set type lld
> ZFS: unexpected object set type lld

So in my opinion increasing number of sectors for first slice containing
bootcode, helped at least in my case.

I'll try to install FreeBSD in tuesday with new iso, as I suspect that
something went wrong during burning dvd, also I didn't check md5. So maybe
those problems are only related to defuncted dvd, I hope so.
_______________________________________________
freebsd-current@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@..."

[Radek Valášek-2]

alteriks@... napsal(a):

> On Wednesday 28 October 2009 21:43:07 Robert Noland wrote:
>  
>> If you hit a key very early then you will land in boot2, where you can
>> type "status" and it will list the zfs pool.
>>    
>
> I haven't tried it yet. Would it allow me to mount root?
> I tried to install from fixit once again, but I've made a script to eliminate
> all typo's I could made. I amde script using most of:
> http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/RAIDZ1
>
> After rebooting it could even load bootloader:
>  
>> ZFS: i/o error - all block copies unavailable
>> ZFS: can't read MOS
>> ZFS: unexpected object set type lld
>> ZFS: unexpected object set type lld
>>    
>
> So in my opinion increasing number of sectors for first slice containing
> bootcode, helped at least in my case.
>
>  

OK, so what's your current size of the first slice ? Could you paste
here the result of 'gpart show' on your instalation?
> I'll try to install FreeBSD in tuesday with new iso, as I suspect that
> something went wrong during burning dvd, also I didn't check md5. So maybe
> those problems are only related to defuncted dvd, I hope so.
> _______________________________________________
> freebsd-current@... mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe@..."
>  

_______________________________________________
freebsd-current@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@..."

[Robert Noland]

On Thu, 2009-10-29 at 10:44 +0100, alteriks@... wrote:
> On Wednesday 28 October 2009 21:43:07 Robert Noland wrote:
> > If you hit a key very early then you will land in boot2, where you can
> > type "status" and it will list the zfs pool.
>
> I haven't tried it yet. Would it allow me to mount root?
> I tried to install from fixit once again, but I've made a script to eliminate
> all typo's I could made. I amde script using most of:
> http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/RAIDZ1

This seems pretty accurate.  I've never tried to do this with labeled
disks though.  For my current setups, I create the pools using ada0p3,
etc... And once they get imported after boot, they end up using gpt
id's.

balrog% zpool status test
  pool: test
 state: ONLINE
 scrub: none requested
config:

        NAME                                            STATE     READ WRITE CKSUM
        test                                            ONLINE       0     0     0
          raidz2                                        ONLINE       0     0     0
            gptid/c56f3303-be79-11de-813c-002215ea6216  ONLINE       0     0     0
            gptid/c70a07c6-be79-11de-813c-002215ea6216  ONLINE       0     0     0
            gptid/c915d03c-be79-11de-813c-002215ea6216  ONLINE       0     0     0
            gptid/0e4cf049-c0ce-11de-8c99-002215ea6216  ONLINE       0     0     0
            gptid/0ff739a0-c0ce-11de-8c99-002215ea6216  ONLINE       0     0     0
            gptid/10f4e873-c0ce-11de-8c99-002215ea6216  ONLINE       0     0     0

errors: No known data errors

This pool continues to boot fine under qemu, and with my hacks to the
BIOS drive detection, it boots under VBox as well.

> After rebooting it could even load bootloader:
> > ZFS: i/o error - all block copies unavailable
> > ZFS: can't read MOS
> > ZFS: unexpected object set type lld
> > ZFS: unexpected object set type lld

Ok, if you are getting this... It may be helpful if you can run "zdb
-uuu zroot", so I can see what is going on with the root block pointer.
I think you should be able to do that from fixit.  Also note that you
don't have the latest version of the loader if it is printing "lld",
though as long as all of the drives are detected it likely won't make a
difference.  The "can't read MOS" error is the first attempt to read
from the pool after probing all of the devices to sort out the
configuration.  Everything up to this point reads data directly from the
vdev labels on each drive, so this is the first time that it tries to
read from the pool.

> So in my opinion increasing number of sectors for first slice containing
> bootcode, helped at least in my case.

-rw-r--r--  1 rnoland  rnoland  25671 Oct 24 13:56 gptzfsboot

Even with all the debugging that I have enabled at the moment,
gptzfsboot still only requires 51 sectors, so the usual 128 (64k) should
be more than enough.

robert.

> I'll try to install FreeBSD in tuesday with new iso, as I suspect that
> something went wrong during burning dvd, also I didn't check md5. So maybe
> those problems are only related to defuncted dvd, I hope so.
--
Robert Noland <rnoland@...>
FreeBSD

_______________________________________________
freebsd-current@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@..."

[Krzysztof Dajka]

Yesterday I tried installing FreeBSD once more, twice to be frank.
I downloaded amd64 8.0-RC2 checked md5 sum. First time I installed it on
physical machine.

I created script with commands pasted here:
http://pastebin.com/f72813264
After that I executed:

chroot /zroot
mount -t devfs devfs /dev
export DESTDIR=""
cd /usr/src/sys/boot/
make obj
make depend
make
cd i386/loader
make install
umount /dev
passwd
exit
cp /boot/zfs/zpool.cache /zroot/boot/zfs/zpool.cache

Finally I launched this script:
http://pastebin.com/f1449f548

Restart gave me:
>GPT boot with ZFS RAIDZ "ZFS: i/o error - all block copies unavailable

I created virtual machine on linux kvm, and added three drives. After starting
system boot loader failed to launch with same error.

I tried to reinstall FreeBSD under kvm, but it didn't help.

Anyone has succeed installing amd64 8.0-RC2 with raidz1 on gpt drives?
_______________________________________________
freebsd-current@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@..."

[Krzysztof Dajka]

On Thursday 29 October 2009 11:36:38 Robert Noland wrote:

> Ok, if you are getting this... It may be helpful if you can run "zdb
> -uuu zroot", so I can see what is going on with the root block pointer.
> I think you should be able to do that from fixit.  Also note that you
> don't have the latest version of the loader if it is printing "lld",
> though as long as all of the drives are detected it likely won't make a
> difference.  The "can't read MOS" error is the first attempt to read
> from the pool after probing all of the devices to sort out the
> configuration.  Everything up to this point reads data directly from the
> vdev labels on each drive, so this is the first time that it tries to
> read from the pool.
>

Sorry I forgot about this part.
Here is zdb -uuu zroot
Uberblock

        magic = 0000000000bab10c
        version = 13
        txg = 111
        guid_sum = 14036990686815153767
        timestamp = 1257636491 UTC = Sat Nov  7 23:28:11 2009
        rootbp = [L0 DMU objset] 400L/200P DVA[0]=<0:110004a000:400>
DVA[1]=<0:40038400:400> DVA[2]=<0:1980041c00:400> fletcher4 lzjb LE contiguous
birth=111 fill=126 cksum=7e96f92e7:357c99e1eb9:b7d10db3a713:1ac2df05bd147a
_______________________________________________
freebsd-current@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@..."

[Robert Noland]

On Mon, 2009-11-09 at 09:48 +0100, alteriks@... wrote:

> On Thursday 29 October 2009 11:36:38 Robert Noland wrote:
> > Ok, if you are getting this... It may be helpful if you can run "zdb
> > -uuu zroot", so I can see what is going on with the root block pointer.
> > I think you should be able to do that from fixit.  Also note that you
> > don't have the latest version of the loader if it is printing "lld",
> > though as long as all of the drives are detected it likely won't make a
> > difference.  The "can't read MOS" error is the first attempt to read
> > from the pool after probing all of the devices to sort out the
> > configuration.  Everything up to this point reads data directly from the
> > vdev labels on each drive, so this is the first time that it tries to
> > read from the pool.
> >
> Sorry I forgot about this part.
> Here is zdb -uuu zroot
> Uberblock
>
>         magic = 0000000000bab10c
>         version = 13
>         txg = 111
>         guid_sum = 14036990686815153767
>         timestamp = 1257636491 UTC = Sat Nov  7 23:28:11 2009
>         rootbp = [L0 DMU objset] 400L/200P DVA[0]=<0:110004a000:400>
> DVA[1]=<0:40038400:400> DVA[2]=<0:1980041c00:400> fletcher4 lzjb LE contiguous
> birth=111 fill=126 cksum=7e96f92e7:357c99e1eb9:b7d10db3a713:1ac2df05bd147a

Ok, that all looks correct for a raidz1.  The logical size is 1024 bytes
(400L) which is the uncompressed size of the data.  The physical size is
512 bytes (200P) which is one sector on disk and the allocated size is
1024 bytes (512 bytes + 512 bytes of parity).  For a raidz or mirror
pool, the vdev is always 0, which represents the pseudo vdev, but it
doesn't tell us for sure that all of the physical disks have been found.
We should be able to verify that all the disks are present by summing
the GUIDs.  I'll have a look at that, so that we can at least warn if
that is the case.  If all of the physical devices are accounted for,
then something is going wrong in the raidz read function, though I don't
understand why it isn't more prolific if that is the case.  i.e. why my
raidz2 test is working fine.

robert.

--
Robert Noland <rnoland@...>
FreeBSD

_______________________________________________
freebsd-current@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@..."