filesystem: 12h to delete 32GB of data

[Olivier Mueller-3]

Hello,

$ df -m ; date ; rm -r templates_c ; df -m ; date
Filesystem  1M-blocks   Used Avail Capacity  Mounted on
/dev/da0s1a       989     45   864     5%    /
/dev/da0s1f    128631 102179 16160    86%    /usr
[...]
Wed May  6 00:23:01 CEST 2009

Filesystem  1M-blocks  Used Avail Capacity  Mounted on
/dev/da0s1a       989    45   864     5%    /
/dev/da0s1f    128631 69844 48496    59%    /usr
Wed May  6 12:21:02 CEST 2009


-> it took about 12 hours to delete these 30GB of files and
sub-directories (smarty cache files: many small files in many dirs).
It's a little bit surprising, as it's on a recent HP proliant DL360 g5
with SAS disks (Raid1) running freebsd 6.x
( /dev/da0s1f on /usr (ufs, local, soft-updates) )

Surprisingly, cpu load remained quite low during the operation (apache
stayed responsive).  Is it a known problem on this kind of hardware or
something related to the filesystem? Is there a way to improve this?
Even on my $500 PC with IDE disks this goes quicker... :)

I checked
http://www.freebsd.org/doc/en/books/handbook/configtuning-disk.html but
I'm not sure if this would help in this case. Any suggestion how I can
"fix" that?

Regards,
Olivier


_______________________________________________
freebsd-performance@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..."

[Bill Moran]

In response to Olivier Mueller <om-lists-bsd@...>:

> Hello,
>
> $ df -m ; date ; rm -r templates_c ; df -m ; date
> Filesystem  1M-blocks   Used Avail Capacity  Mounted on
> /dev/da0s1a       989     45   864     5%    /
> /dev/da0s1f    128631 102179 16160    86%    /usr
> [...]
> Wed May  6 00:23:01 CEST 2009
>
> Filesystem  1M-blocks  Used Avail Capacity  Mounted on
> /dev/da0s1a       989    45   864     5%    /
> /dev/da0s1f    128631 69844 48496    59%    /usr
> Wed May  6 12:21:02 CEST 2009
>
>
> -> it took about 12 hours to delete these 30GB of files and
> sub-directories (smarty cache files: many small files in many dirs).
> It's a little bit surprising, as it's on a recent HP proliant DL360 g5
> with SAS disks (Raid1) running freebsd 6.x
> ( /dev/da0s1f on /usr (ufs, local, soft-updates) )
>
> Surprisingly, cpu load remained quite low during the operation (apache
> stayed responsive).  Is it a known problem on this kind of hardware or
> something related to the filesystem? Is there a way to improve this?
> Even on my $500 PC with IDE disks this goes quicker... :)
>
> I checked
> http://www.freebsd.org/doc/en/books/handbook/configtuning-disk.html but
> I'm not sure if this would help in this case. Any suggestion how I can
> "fix" that?

With lots of small files, the time involved is far less dependent on
the size of data, and much more dependent on the number of files, and
the resultant number of directory entries that need to be updated.
"Lots" isn't a particularly accurate count of the # of files, but if
you're talking web cache files, I'll guess they average 5k each, which
means you had 6 million files.  df -i would have been more useful in
the output above.

This brings a number of questions up:
* Are you _sure_ softupdates is enabled on that partition?  That's
  going to make the biggest improvement in speed.
* Are these 7200RPM disks or 15,000?  Again, going to make a big
  difference.
* If apache was still running, is it possible that it was creating
  enough disk activity to slow the activity down?  Running
  top -m io will show you how much disk IO each process is creating.
* When you compared the speed to your laptop, did you delete 6 million
  files from the laptop?  If you deleted a single 30G file, then you're
  comparing apples to atom bombs.

If this is a directory that you blow away on a regular schedule, you'd
do much better to make it a dedicated partition and simply reformat
it.

--
Bill Moran
http://www.potentialtech.com
http://people.collaborativefusion.com/~wmoran/
_______________________________________________
freebsd-performance@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..."

[Arkadi Shishlov-4]

Its probably "dirhash' that is not enabled or its cache is too small for the task.
_______________________________________________
freebsd-performance@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..."

[Olivier Mueller-3]

Thanks for your answer Bill!  (and to Will as well),

Some more infos I gathered a few minutes ago:

[~/templates_c]$ date; du -s -m ; date
Wed May  6 13:35:15 CEST 2009
  2652      .
Wed May  6 13:52:36 CEST 2009

[~/templates_c]$ date ; find . | wc -l ; date
Wed May  6 13:52:56 CEST 2009
  305461
Wed May  6 14:09:39 CEST 2009


So this is on the system after a complete cache cleanup (at 00h00).
300'000 files and 2.6GB.  So this night, there were probably around 3-4
million files to delete.

Deletion may take time, but 20 minutes juste to _count_ all the files
seems pretty long to me...   I think I'll say a word to the developers
to let them tune their caching system a bit :)



On Wed, 2009-05-06 at 08:48 -0400, Bill Moran wrote:
> With lots of small files, the time involved is far less dependent on
> the size of data, and much more dependent on the number of files, and
> the resultant number of directory entries that need to be updated.
> "Lots" isn't a particularly accurate count of the # of files, but if
> you're talking web cache files, I'll guess they average 5k each, which
> means you had 6 million files.  df -i would have been more useful in
> the output above.

Thanks, noted for next time.  Now it looks like that:
Filesystem  1M-blocks  Used Avail Capacity iused    ifree %iused  Mounted on
/dev/da0s1f    128631 70544 47795    60% 1913875 15114219   11%   /usr


> This brings a number of questions up:
> * Are you _sure_ softupdates is enabled on that partition?  That's
>   going to make the biggest improvement in speed.

According to "mount" output, yes.  I found no specific message about
that in the syslog or dmesg.

> * Are these 7200RPM disks or 15,000?  Again, going to make a big
>   difference.

HP 146GB 6G SAS 10K SFF DP ENT HDD  (15k were not available at the time
the servers were ordered)
( http://h18004.www1.hp.com/products/servers/proliantstorage/serial/sas/index.html )

> * If apache was still running, is it possible that it was creating
>   enough disk activity to slow the activity down?  Running
>   top -m io will show you how much disk IO each process is creating.

Yes, apache was still running, but the activity was quite low (it was
during the night, and the webpage doesn't get so many hits before 9 am
local time)

While watching "top -m io", the "du" or "find" takes between 80 and 99%,
so I guess it's not the probleme here:

  PID    UID          VCSW  IVCSW   READ  WRITE  FAULT  TOTAL PERCENT COMMAND
87996   1002            59     56      0      0      0      0   0.00% php
45389   1002            35     25      0      0      2      2   0.84% php
 3964   1002             0      0      0      0      0      0   0.00% httpd
 3822   1002           151     98      0      0      0      0   0.00% httpd
 3005   1002             0      0      0      0      0      0   0.00% httpd
 4129   1002             0      0      0      0      0      0   0.00% httpd
 3971   1002             0      0      0      0      0      0   0.00% httpd
 4231   1002             1      0      0      0      0      0   0.00% httpd
 4132      0           234      5    234      0      0    234  97.91% find
98862   1002             1      0      0      0      0      0   0.00% top
  609      0             0      0      0      0      0      0   0.00% snmpd
[...]


> * When you compared the speed to your laptop, did you delete 6 million
>   files from the laptop?  If you deleted a single 30G file, then you're
>   comparing apples to atom bombs.

Yes sorry, I know :)

> If this is a directory that you blow away on a regular schedule, you'd
> do much better to make it a dedicated partition and simply reformat
> it.

Yes, it is one of the best options. My initial goal was to delete all
files older than N days by cron  (find | xargs | rm, etc.), but if each
cronjob takes 2 hours (and takes so much cpu time), it's probably not
the best way.  

I'll make some more tests on an test-server later this week and speak
with the devs. Thanks again for your very constructive feedback!

Regards,
Olivier

_______________________________________________
freebsd-performance@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..."

[Bill Moran-2]

In response to Arkadi Shishlov <arkadijs.sislovs@...>:

> Its probably "dirhash' that is not enabled or its cache is too small for the task.

I'm no expert, but I thought dirhash only improved read speed.  His
bottleneck would be writes.

--
Bill Moran
Collaborative Fusion Inc.
http://people.collaborativefusion.com/~wmoran/

wmoran@...
Phone: [phone]x4023

****************************************************************
IMPORTANT: This message contains confidential information and is
intended only for the individual named. If the reader of this
message is not an intended recipient (or the individual
responsible for the delivery of this message to an intended
recipient), please be advised that any re-use, dissemination,
distribution or copying of this message is prohibited. Please
notify the sender immediately by e-mail if you have received
this e-mail by mistake and delete this e-mail from your system.
E-mail transmission cannot be guaranteed to be secure or
error-free as information could be intercepted, corrupted, lost,
destroyed, arrive late or incomplete, or contain viruses. The
sender therefore does not accept liability for any errors or
omissions in the contents of this message, which arise as a
result of e-mail transmission.
****************************************************************
_______________________________________________
freebsd-performance@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..."

[Olivier Mueller-3]

On Wed, 2009-05-06 at 16:15 +0300, Arkadi Shishlov wrote:
> Its probably "dirhash' that is not enabled or its cache is too small for the task.

$ sysctl -a |grep dirha
  UFS dirhash  1262   286K       -  9715683  16,32,64,128,256,512,1024,2048,4096
vfs.ufs.dirhash_docheck: 0
vfs.ufs.dirhash_mem: 2087495
vfs.ufs.dirhash_maxmem: 2097152
vfs.ufs.dirhash_minsize: 2560

So it's active, but probably too small as you suggest. Can I update this
value "on the fly" or does it require a reboot (+ settings in
loader.conf) ?

regards,
Olivier


_______________________________________________
freebsd-performance@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..."

[Bill Moran]

In response to Olivier Mueller <om-lists-bsd@...>:
>
> Yes, it is one of the best options. My initial goal was to delete all
> files older than N days by cron  (find | xargs | rm, etc.), but if each
> cronjob takes 2 hours (and takes so much cpu time), it's probably not
> the best way.  
>
> I'll make some more tests on an test-server later this week and speak
> with the devs. Thanks again for your very constructive feedback!

Based on your comments here, it really sounds like your devs need to
implement some sort of cache cleaning algo into their code.  If it's
just deleting the oldest files, then you could probably run it far
more frequently if you simply created a new cache directory each
hour, and deleted the previous one.

Honestly, I'm really confused -- if you can just throw away the cache
each night, then why are you caching to begin with?  If you just need
temp files, why doesn't the app clean up its temp files when it's
done with them?

If you have access to the developers, I think you'll be able to come
up with a much better solution by working with them.

--
Bill Moran
http://www.potentialtech.com
http://people.collaborativefusion.com/~wmoran/
_______________________________________________
freebsd-performance@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..."

[Wojciech Puchar-5]

> -> it took about 12 hours to delete these 30GB of files and
> sub-directories (smarty cache files: many small files in many dirs).
> It's a little bit surprising, as it's on a recent HP proliant DL360 g5
> with SAS disks (Raid1) running freebsd 6.x
> ( /dev/da0s1f on /usr (ufs, local, soft-updates) )
>
if you would use no raid or software raid it will behave normally.

it takes <30 minutes for me to delete 300GB of squid files on
ordinary SATA disk , millions of small files.


_______________________________________________
freebsd-performance@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..."

[Wojciech Puchar-5]

> means you had 6 million files.  df -i would have been more useful in


> the output above.
>
> This brings a number of questions up:
> * Are you _sure_ softupdates is enabled on that partition?  That's

he showed mount output - he has softdeps on.

> * Are these 7200RPM disks or 15,000?  Again, going to make a big
>  difference.

on 7200 RPM ordinary SATA disk i deleted 15 million files taking 300GB
(squid cache) in less than 30 minutes.

for sure it's because of his "hardware raid".

i've NEVER seen "hardware raid" that is actually faster than non-raid
config, or gmirror/gstripe config.

usually it's far much slower
_______________________________________________
freebsd-performance@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..."

[Benjamin Krueger]

Wojciech Puchar wrote:

>> means you had 6 million files.  df -i would have been more useful in
>
>
>> the output above.
>>
>> This brings a number of questions up:
>> * Are you _sure_ softupdates is enabled on that partition?  That's
>
> he showed mount output - he has softdeps on.
>
>> * Are these 7200RPM disks or 15,000?  Again, going to make a big
>>  difference.
>
> on 7200 RPM ordinary SATA disk i deleted 15 million files taking 300GB
> (squid cache) in less than 30 minutes.
>
> for sure it's because of his "hardware raid".
>
> i've NEVER seen "hardware raid" that is actually faster than non-raid
> config, or gmirror/gstripe config.
>
> usually it's far much slower

Sorry, but my experience with that very server using a P400 controller
with 256MB write cache is very different. My benchmarks showed that
controller using Raid5 (with only 4 disks) is significantly faster than
software layouts.

The days when hardware controllers could automatically be considered
slow are long gone. The hardware does get faster over time. Don't make
any assumptions without doing benchmarks.
_______________________________________________
freebsd-performance@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..."

[Gary Gatten]

It could just be me, but I swear Hardware RAID has been faster for many
many years, especially with RAID5 arrays - or anything that requires
parity calcs.  Most of my benchmarking was done on SCO OpenServer and
Novell UnixWare and Netware, but hardware RAID controllers were always
faster and of course required far less host CPU resources.  Raid
0/1/10/0+1/whatever arrays, I recall weren't as drastic, but I can't
imagine the controller making as big a difference as the drives in the
array - unless of course the drive for said controller sux!

<snip>






<font size="1">
<div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
</div>
"This email is intended to be reviewed by only the intended recipient
 and may contain information that is privileged and/or confidential.
 If you are not the intended recipient, you are hereby notified that
 any review, use, dissemination, disclosure or copying of this email
 and its attachments, if any, is strictly prohibited.  If you have
 received this email in error, please immediately notify the sender by
 return email and delete this email from your system."
</font>

_______________________________________________
freebsd-performance@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..."

[Bill Moran]

In response to "Gary Gatten" <Ggatten@...>:

> It could just be me, but I swear Hardware RAID has been faster for many
> many years, especially with RAID5 arrays - or anything that requires
> parity calcs.  Most of my benchmarking was done on SCO OpenServer and
> Novell UnixWare and Netware, but hardware RAID controllers were always
> faster and of course required far less host CPU resources.  Raid
> 0/1/10/0+1/whatever arrays, I recall weren't as drastic, but I can't
> imagine the controller making as big a difference as the drives in the
> array - unless of course the drive for said controller sux!

Keep in mind that there are a LOT of RAID controllers out there, and
yes, some of them suck royally.  Especially the consumer-grade stuff
intended for people to use on their home systems.  I'd be willing
to bet that software RAID is faster than 90% of the consumer grade
RAID cards, and probably more reliable than most of them as well.

Controllers make a huge difference, even in server class RAID (in
my experience).  There is a significant gap in performance between
the good stuff and the good enough stuff.

--
Bill Moran
http://www.potentialtech.com
http://people.collaborativefusion.com/~wmoran/
_______________________________________________
freebsd-performance@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..."

[Gary Gatten]

Sorry, "drive" in last sentence should be "driver"!

----- Original Message -----
From: owner-freebsd-questions@... <owner-freebsd-questions@...>
To: Benjamin Krueger <benjamin@...>; Wojciech Puchar <wojtek@...>
Cc: freebsd-performance@... <freebsd-performance@...>; Olivier Mueller <om-lists-bsd@...>; Bill Moran <wmoran@...>; freebsd-questions@... <freebsd-questions@...>
Sent: Wed May 06 13:08:46 2009
Subject: RE: filesystem: 12h to delete 32GB of data

It could just be me, but I swear Hardware RAID has been faster for many
many years, especially with RAID5 arrays - or anything that requires
parity calcs.  Most of my benchmarking was done on SCO OpenServer and
Novell UnixWare and Netware, but hardware RAID controllers were always
faster and of course required far less host CPU resources.  Raid
0/1/10/0+1/whatever arrays, I recall weren't as drastic, but I can't
imagine the controller making as big a difference as the drives in the
array - unless of course the drive for said controller sux!

<snip>






<font size="1">
<div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
</div>
"This email is intended to be reviewed by only the intended recipient
 and may contain information that is privileged and/or confidential.
 If you are not the intended recipient, you are hereby notified that
 any review, use, dissemination, disclosure or copying of this email
 and its attachments, if any, is strictly prohibited.  If you have
 received this email in error, please immediately notify the sender by
 return email and delete this email from your system."
</font>

_______________________________________________
freebsd-questions@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscribe@..."





<font size="1">
<div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
</div>
"This email is intended to be reviewed by only the intended recipient
 and may contain information that is privileged and/or confidential.
 If you are not the intended recipient, you are hereby notified that
 any review, use, dissemination, disclosure or copying of this email
 and its attachments, if any, is strictly prohibited.  If you have
 received this email in error, please immediately notify the sender by
 return email and delete this email from your system."
</font>


_______________________________________________
freebsd-performance@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..."

[Wojciech Puchar-5]

>> config, or gmirror/gstripe config.
>>
>> usually it's far much slower
>
> Sorry, but my experience with that very server using a P400 controller with
> 256MB write cache is very different. My benchmarks showed that controller
> using Raid5 (with only 4 disks) is significantly faster than software
> layouts.

possibly with RAID5, but for sure slower than single drive

> The days when hardware controllers could automatically be considered slow are
> long gone.

unfortunately not.
_______________________________________________
freebsd-performance@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..."

[Wojciech Puchar-5]

> It could just be me, but I swear Hardware RAID has been faster for many
> many years, especially with RAID5 arrays - or anything that requires

maybe with RAID5, but using RAID5 today (huge disk sizes, little sense to
save on disk space) instead of RAID1/10 doesn't make much sense, as RAID5
is slow on writes by design
_______________________________________________
freebsd-performance@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..."

[Wojciech Puchar-5]

> yes, some of them suck royally.

you should rather say "some of them doesn't suck".
_______________________________________________
freebsd-performance@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..."

[Gary Gatten]

OT now, but in high i/o envs with high concurrency needs, RAID5 is still the way to go, esp if 90% of i/o is reads. Of course it depends on file size / type as well... Anyway, let's sum it up with "a storage subsystem is only as fast as its slowest link"

----- Original Message -----
From: Wojciech Puchar <wojtek@...>
To: Bill Moran <wmoran@...>
Cc: Gary Gatten; Benjamin Krueger <benjamin@...>; freebsd-performance@... <freebsd-performance@...>; Olivier Mueller <om-lists-bsd@...>; freebsd-questions@... <freebsd-questions@...>
Sent: Wed May 06 13:31:53 2009
Subject: Re: filesystem: 12h to delete 32GB of data

> yes, some of them suck royally.

you should rather say "some of them doesn't suck".





<font size="1">
<div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
</div>
"This email is intended to be reviewed by only the intended recipient
 and may contain information that is privileged and/or confidential.
 If you are not the intended recipient, you are hereby notified that
 any review, use, dissemination, disclosure or copying of this email
 and its attachments, if any, is strictly prohibited.  If you have
 received this email in error, please immediately notify the sender by
 return email and delete this email from your system."
</font>


_______________________________________________
freebsd-performance@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..."

[Matthew Seaman-2]

Gary Gatten wrote:
> OT now, but in high i/o envs with high concurrency needs, RAID5 is
> still the way to go, esp if 90% of i/o is reads. Of course it depends
> on file size / type as well... Anyway, let's sum it up with "a
> storage subsystem is only as fast as its slowest link"

It's not just the balance of reads over writes.  It's the size and sequential
location of the IO requests.  RAID5 is good for sequential reads -- eg.
streaming a video -- where the system can read whole blocks from all the
drives involved, calculate parity over the whole lot and then push all that
blob of data up to the CPU.

RAID5 is pretty pessimal if your usage pattern is small reads or writes
randomly scattered over your storage area -- eg. typical RDBMS behaviour
-- which works a great deal better on RAID10.

I'd also contend that the essential difference between a really good fast
hardware raid controller and something disappointingly mundane is a decent
amount of non-volatile cache memory.  For most H/W raid that equates to
using a battery backup unit.  I've been thinking though that a few GB of
fast solid-state hard drive configured as a gjournal for a RAID10 (ie gstripe
+gmirror) might achieve the same effect for rather less outlay...  It
would probably not be too shabby with RAID5 even, but of course you'ld
lose the benefit of offloading parity calculations onto the RAID controller's
CPU. Still, modern multi-core CPUs are probably fast enough nowadays to
make that viable for many purposes.  

        Cheers,

        Matthew

--
Dr Matthew J Seaman MA, D.Phil.                   7 Priory Courtyard
                                                  Flat 3
PGP: http://www.infracaninophile.co.uk/pgpkey     Ramsgate
                                                  Kent, CT11 9PW

[Freddie Cash-8]

On Wed, May 6, 2009 at 12:21 PM, Matthew Seaman
<m.seaman@...> wrote:

> Gary Gatten wrote:
>> OT now, but in high i/o envs with high concurrency needs, RAID5 is
>> still the way to go, esp if 90% of i/o is reads. Of course it depends
>> on file size / type as well... Anyway, let's sum it up with "a
>> storage subsystem is only as fast as its slowest link"
>
> It's not just the balance of reads over writes.  It's the size and
> sequential location of the IO requests.  RAID5 is good for sequential reads -- eg.
> streaming a video -- where the system can read whole blocks from all the
> drives involved, calculate parity over the whole lot and then push all that
> blob of data up to the CPU.
>
> RAID5 is pretty pessimal if your usage pattern is small reads or writes
> randomly scattered over your storage area -- eg. typical RDBMS behaviour
> -- which works a great deal better on RAID10.
>
> I'd also contend that the essential difference between a really good fast
> hardware raid controller and something disappointingly mundane is a decent
> amount of non-volatile cache memory.  For most H/W raid that equates to
> using a battery backup unit.  I've been thinking though that a few GB of
> fast solid-state hard drive configured as a gjournal for a RAID10 (ie
> gstripe +gmirror) might achieve the same effect for rather less outlay...  It
> would probably not be too shabby with RAID5 even, but of course you'ld
> lose the benefit of offloading parity calculations onto the RAID
> controller's CPU. Still, modern multi-core CPUs are probably fast enough nowadays to
> make that viable for many purposes.

Depending on the number of drives you are using, ZFS would also be
worth looking at.  The raidz implementation works quite nicely, and
(in theory) doesn't suffer from the major issues that RAID5/6 does.
It also does implicit striping across all vdevs, so you can make some
very fancy RAID layouts (each vdev can be mirrored, raidz1, raidz2, or
just a bunch of disks).

I don't know if the version of ZFS in FreeBSD 7.x supports hybrid
pools, but the version in FreeBSD 8.0 should, which lets you add SSDs
to the pool to be used automatically as "cache" in-between RAM and
harddrives.

--
Freddie Cash
fjwcash@...
_______________________________________________
freebsd-performance@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..."

[Paul Pathiakis-5]

Sorry.  This statement is incorrect.

If you aren't using ZFS, or even a GEOM volume with mirror/RAID5/softup/etc, you cannot make the statement that hardware RAID is faster.  I learned that 3 years ago.

It takes about 30 minutes to mirror 1.5TB on ZFS.  Try that on hardware RAID.

I did the same with 80 GB SATA drives a couple of years ago.  Gmirror killed hardware mirror by 50%

When your processor on your hardware RAID card is junk and you have a kickass processor and good chunk of memory on your main system and decent controller that isn't getting maxed, the "hardware RAID is always faster" paradigm walked out the door a few years ago.

This does not go for EMC, IBM, Hitachi high-end storage arrays where you write to TBs of RAM Cache.

P.




________________________________
From: Wojciech Puchar <wojtek@...>
To: Gary Gatten <Ggatten@...>
Cc: freebsd-questions@...; Benjamin Krueger <benjamin@...>; Olivier Mueller <om-lists-bsd@...>; freebsd-performance@...; Bill Moran <wmoran@...>
Sent: Wednesday, May 6, 2009 2:31:16 PM
Subject: RE: filesystem: 12h to delete 32GB of data

> It could just be me, but I swear Hardware RAID has been faster for many
> many years, especially with RAID5 arrays - or anything that requires

maybe with RAID5, but using RAID5 today (huge disk sizes, little sense to save on disk space) instead of RAID1/10 doesn't make much sense, as RAID5 is slow on writes by design
_______________________________________________
freebsd-performance@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..."



     
_______________________________________________
freebsd-performance@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..."