3ware hardware raid with battery backup and the impact on barrier and no write cache options.

[William Lewis-2]

Hi,

I am in the process of setting up an XFS file system on underlying hardware consisting of a 3ware 9550SXU (+ battery backup module) and 4 x Seagate ST31500341AS 1.5TB hard drives in Raid 5 configuration.

Reading your FAQ at http://xfs.org/index.php/XFS_FAQ I understand that it is advisable to mount the file system with nobarrier to improve performance. However going on to read about recommended settings for write cache, the advice for 3ware hardware doesn't seem to account for the fact that there are 2 levels of write cache in play, that in the 3ware card itself protected by the battery and the write cache of the disks themselves, which as far as I can understand is also protected by the battery backup (in the correct storage modes - balanced/protection) because the 3ware card uses write journaling to keep track of pending write operations in the disks' cache. Therefore unless I am misunderstanding something the most benefit is to be gained by mounting with nobarrier and having the write cache turned on?

One thing I am not clear about is if nobarrier interacts with the page cache at all and if the lack of barrier leaves you with a situation in which pending writes can be lost from main memory on power failure?

Thanks in advance for any clarification you can provide.

Regards

Will Lewis

_______________________________________________
xfs mailing list
xfs@...
http://oss.sgi.com/mailman/listinfo/xfs

[Justin Piszcz]

On Mon, 2 Nov 2009, William Lewis wrote:

> Hi,
>
> I am in the process of setting up an XFS file system on underlying hardware
> consisting of a 3ware 9550SXU (+ battery backup module) and 4 x Seagate
> ST31500341AS 1.5TB hard drives in Raid 5 configuration.
>
> Reading your FAQ at http://xfs.org/index.php/XFS_FAQ I understand that it is
> advisable to mount the file system with nobarrier to improve performance.
> However going on to read about recommended settings for write cache, the
> advice for 3ware hardware doesn't seem to account for the fact that there
> are 2 levels of write cache in play, that in the 3ware card itself protected
> by the battery and the write cache of the disks themselves, which as far as
> I can understand is also protected by the battery backup (in the correct
> storage modes - balanced/protection) because the 3ware card uses write
> journaling to keep track of pending write operations in the disks' cache.
> Therefore unless I am misunderstanding something the most benefit is to be
> gained by mounting with nobarrier and having the write cache turned on?
>
> One thing I am not clear about is if nobarrier interacts with the page cache
> at all and if the lack of barrier leaves you with a situation in which
> pending writes can be lost from main memory on power failure?
>
> Thanks in advance for any clarification you can provide.
>
> Regards
>
> Will Lewis
>

I am also curious, I have a 16 drive RAID-6 configuration on a
9650SE-16ML and using -o nobarrier or mounting normal the speed/benchmarks
seemed to be the same.  Either barriers are not enabled by default for
3ware RAID arrays or they make no difference in performance?

Justin.

_______________________________________________
xfs mailing list
xfs@...
http://oss.sgi.com/mailman/listinfo/xfs

[Eric Sandeen-3]

William Lewis wrote:

> Hi,
>
> I am in the process of setting up an XFS file system on underlying
> hardware consisting of a 3ware 9550SXU (+ battery backup module) and 4 x
> Seagate ST31500341AS 1.5TB hard drives in Raid 5 configuration.
>
> Reading your FAQ at http://xfs.org/index.php/XFS_FAQ I understand that
> it is advisable to mount the file system with nobarrier to improve
> performance. However going on to read about recommended settings for
> write cache, the advice for 3ware hardware doesn't seem to account for
> the fact that there are 2 levels of write cache in play, that in the
> 3ware card itself protected by the battery and the write cache of the
> disks themselves, which as far as I can understand is also protected by
> the battery backup (in the correct storage modes - balanced/protection)
> because the 3ware card uses write journaling to keep track of pending
> write operations in the disks' cache. Therefore unless I am
> misunderstanding something the most benefit is to be gained by mounting
> with nobarrier and having the write cache turned on?

If the write caches won't go away - or will be fully/gracefully destaged
before they do, then nobarrier should be safe.

> One thing I am not clear about is if nobarrier interacts with the page
> cache at all and if the lack of barrier leaves you with a situation in
> which pending writes can be lost from main memory on power failure?

nobarrier has no interaction with the OS page cache; all the "barrier"
option (the default) does is enforce ordering for journal IO*, and in
practice it does this by flushing the cache at points in time.

-Eric

*well, I think it flushes the drive cache on fsync, too, for data
integrity (vs. the metadata integrity for the journal IO ordering)

> Thanks in advance for any clarification you can provide.
>
> Regards
>
> Will Lewis
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> xfs mailing list
> xfs@...
> http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
xfs@...
http://oss.sgi.com/mailman/listinfo/xfs

[Michael Monnerie-5]

On Montag 02 November 2009 William Lewis wrote:
> Reading your FAQ at http://xfs.org/index.php/XFS_FAQ I understand
> that it is advisable to mount the file system with nobarrier to
> improve performance.

I edited that part, so I'll answer.

> However going on to read about recommended
> settings for write cache, the advice for 3ware hardware doesn't seem
> to account for the fact that there are 2 levels of write cache in
> play, that in the 3ware card itself protected by the battery and the
> write cache of the disks themselves,

There are 2 different caches. One in the controller, on on the disks.

> which as far as I can understand
> is also protected by the battery backup (in the correct storage modes
> - balanced/protection) because the 3ware card uses write journaling
> to keep track of pending write operations in the disks' cache.

I can't believe it's possible for the controller to know when a disk
write is actually on disk while the disk write cache is on. There are
lots of disk who plainly *lie* about that fact. I've seen a web page
once with a listing of disks who lie - quiet long. I don't find the link
ATM, maybe googling helps.

> Therefore unless I am misunderstanding something the most benefit is
> to be gained by mounting with nobarrier and having the write cache
> turned on?

If you care about your data, don't turn on the disk write cache. I
wonder why there's not a single disk vendor building a disk with a small
battery buffer that is able to keep the disk spinning until it can
savely empty disk cache to platters. Would be a real performance benefit
in server environments.

> One thing I am not clear about is if nobarrier interacts with the
> page cache at all and if the lack of barrier leaves you with a
> situation in which pending writes can be lost from main memory on
> power failure?

Writes in main RAM will always be lost on power fail. If you need
protection for that, use fsync(). The thing about barries is to ensure
that metadata is kept consistent on powerfail. It's in fact nothing to
protect your data, only the filesystem metadata. Well, in the end your
data also, as the filesystem survives.

> Thanks in advance for any clarification you can provide.

I hope I could help.

mfg zmi
--
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4

_______________________________________________
xfs mailing list
xfs@...
http://oss.sgi.com/mailman/listinfo/xfs

[Michael Monnerie-5]

On Montag 02 November 2009 Justin Piszcz wrote:
> I am also curious, I have a 16 drive RAID-6 configuration on a
> 9650SE-16ML and using -o nobarrier or mounting normal the
> speed/benchmarks seemed to be the same.  Either barriers are not
> enabled by default for 3ware RAID arrays or they make no difference
> in performance?

I'd say a RAID controller with it's own write cache enabled + BBM will
effectively turn barriers off, even if you can use them as a mount
options. What happens with barriers, is that it writes
1,2,3,barrier,4,5,barrier,6 etc. so, that 123 are sure on disk before 45
happen etc.
The RAID controller will happily tell you it did flush everything,
because as soon as data is in it's cache, it's claimed sure that the
data gets written, and therefore it will tell that the barrier is
already done. And that's why it's a *must* to turn off disk cache
writes, because the filesystem got it's barrier ACK and believes it, the
controller has it's cache written to disk, powerfail.... all gone. That
will do a bigger damage than any single disk could have done.

mfg zmi
--
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4

_______________________________________________
xfs mailing list
xfs@...
http://oss.sgi.com/mailman/listinfo/xfs