HAMMER version 4 work in progress (NOT for production systems)

[Matthew Dillon]

    I will be committing bits of work over the next week or so related to
    HAMMER version 4 into master.  This version is marked as WIP (work in
    progress).  It is *NOT* ready for prime time.  Do *NOT* do a forced
    upgrade to version 4 at this time except on scratch filesystems that
    you don't mind losing, for testing purposes only.

    Nearly all of the work is conditionalized on version 4 so running
    the master branch on existing version 1-3 filesystems should still
    work fine.  I don't guarantee it, but I did my best to conditionalize
    it for version 4.  Version 3 filesystems will format the UNDO space a
    bit differently but it should be backwards compatible.

    The first bit of work committed today involves changing the layout
    of the media structures in the UNDO FIFO such that the crash recovery
    code can locate the end of the undo range (and detect any missing
    sectors) without having to rely on index range stored in the volume
    header.  This in turn allows the flush code to remove one of the two
    disk synchronization commands which exist in the flush path.

    The next bit of work will involve adding forward-looking REDO records
    to improve lseek+write+fsync sequences.  File creation will still have
    to go through the normal full flush but once the inode is on-disk my
    hope is I can work out a way to lay down a REDO record for a write()
    such that a fsync() only needs to flush UNDO blocks, and not have to
    flush the volume header or meta-data.  The crash recovery sequence
    would first undo meta-data changes, then re-run higher level REDO
    operations on related inode(s).  Theoretically we can implement REDO
    for anything, including file creation, but realisitically its most
    important use is for small lseek+write+fsync sequences to support
    database operations.

    Needless to say if I can get REDO working then fully-crash-recoverable
    fsync operations will become very, very fast.

    Version 4 is not yet ready for any serious testing.  If you do wish to
    test it the paths being modified here are the crash recovery paths and
    the only real way to test it is by pulling the SATA cable out of the
    drive during heavy disk I/O (do NOT pull the power on a drive), or
    panicing the system on purpose to simulate a crash, and then observe
    whether the crash recovery works on mount after reboot.  Testing inside
    a vkernel might also be useful but you need a fairly large HAMMER virtual
    disk to test realistically.

                                        -Matt
                                        Matthew Dillon
                                        <dillon@...>

[W B Hacker]

Matthew Dillon wrote:

>     the only real way to test it is by pulling the SATA cable out of the
>     drive during heavy disk I/O (do NOT pull the power on a drive), or

Matt,

What thoughts in re pulling the (only) cable on a FW or USB-attached drive?
(as it seesm both easier and 'safer' w/r access and/or other-than-fs damage)

Bill

[Matthew Dillon]

:Matthew Dillon wrote:
:
:>     the only real way to test it is by pulling the SATA cable out of the
:>     drive during heavy disk I/O (do NOT pull the power on a drive), or
:
:Matt,
:
:What thoughts in re pulling the (only) cable on a FW or USB-attached drive?
:(as it seesm both easier and 'safer' w/r access and/or other-than-fs damage)
:
:Bill

    It is fairly safe w/ the AHCI driver or the SILI driver.  It is not
    safe with the NATA driver.  It may or may not be safe w/ the firewire
    driver.  It all depends on whether the driver properly aborts any
    in-progress I/Os or not, and whether the filesystem then does the
    right thing with the error condition.

                                        -Matt
                                        Matthew Dillon
                                        <dillon@...>

[Matthew Dillon]

    Here's an update on the version 4 WIP testing.

    I pulled the SATA cable on the drive a bunch of times and forced HAMMER
    to go into crash recovery and it seemed to do a good job.

    I will probably implement the actual REDO as version 5.  There's already
    plenty in version 4 related to the UNDO FIFO changes to support faster
    flushes and the later REDO to warrent its own version.

    However, I want to do another weeks worth of testing for version 4
    to be sure.  Version 4 is still WIP and is not yet ready for prime
    time.

                                        -Matt
                                        Matthew Dillon
                                        <dillon@...>