RFC: Tuning ext3

[Tod Hagan]

All,

I'm requesting comments from the expert readers of ext3-users on these
notes for tuning ext3 for performance.

Most helpful would be feedback pertinent to RHEL 5; as XFS isn't
supported under Red Hat Enterprise Linux these items are an attempt to
match XFS performance with ext3.

These items were culled from a number of sources. Will they be effective
for achieving the performance goals? Is the risk assessment correct?
Are any too risky for consideration? Has anything been left out?

Thanks.



ext3 Performance Tuning

This document contains notes about optimizing ext3 filesystems for
storage nodes on Linux clusters used for scientific research.

Each of the goals and assumptions has been assigned an identifier in
square brackets which is referenced in the applicable tuning options.

Goals:

[BigSeqIO] Maximize sequential I/O performance with large files (> 1GB),
including simultaneous access to several large files.

[MaxStor] Maximize the amount of usable storage.

[BigFS] Allow large filesystems (> 8 TB).

[GenNFSPerf] Optimize for NFS and general performance.

Assumptions:

[NoHA] High availability isn't needed.

[IntegFS] The integrity of the filesystem is important -- don't lose
existing files.

[Recreate] Newly-created files are unimportant as they can be
re-created.

[UPS] Power to the hardware is guaranteed by a UPS.

[NoBigDir] Directories don't get large.

[NoSysFiles] The filesystems hold data and not system files.

Tuning:

A. Create using -O sparse_super (default) to save space on large
filesystems. [MaxStor]

B. Create using -T largefile4 (one inode per 4 megabytes) to avoid
wasting space on unused inodes. [MaxStor]

C. Create using -m 0 to reserve no blocks for the super-user.
[NoSysFiles][MaxStor]

D. Create using -E stride=N where N matches the underlying RAID.
[GenNFSPerf]

E. Use a kernel >= 2.6.19 (patches for extents and 48-bit support,
requires Ubuntu 7.04 feisty or Fedora Core 7 or custom kernel) to allow
filesystems > 8TB on Intel/AMD chips. [BigFS]

F. Use an external journal on a separate high-RPM drive. [GenNFSPerf]

G. Use a large journal. mkfs -J size=8192 [GenNFSPerf]

H. Mount using -o orlov to use the Orlov block allocator (default,
requires 2.6 kernel). Minimizes seeks by clustering files together. No
risk. [GenNFSPerf]

I. Mount using -o noatime,nodiratime. No risk. [GenNFSPerf]

J. Mount using -o reservation to speed writes to multiple files in the
same directory. No risk. [BigSeqIO][GenNFSPerf]

K. Mount using -o data=writeback. Comparable to XFS and JFS, relaxes all
restrictions on writing cached data. Risky.
[Recreate][UPS][BigSeqIO][GenNFSPerf]

L. Mount using -o commit=Nsec where N > 5 (requires 2.6 kernel, default
is 5 seconds). Reduces sync interval. Risky. [Recreate][UPS][GenNFSPerf]

M. Mount using -o barrier=0 and enable write-back caching on the
controllers and drives. The most(?) risky. [Recreate][UPS][GenNFSPerf]



--
Tod Hagan
Information Technologist
AIRMAP/Climate Change Research Center
Institute for the Study of Earth, Oceans, and Space
University of New Hampshire
Durham, NH 03824
Phone: [phone]


_______________________________________________
Ext3-users mailing list
Ext3-users@...
https://www.redhat.com/mailman/listinfo/ext3-users

[Andreas Dilger]

On May 17, 2007  16:16 -0400, Tod Hagan wrote:
> A. Create using -O sparse_super (default) to save space on large
> filesystems. [MaxStor]

That's the default anyways.

> B. Create using -T largefile4 (one inode per 4 megabytes) to avoid
> wasting space on unused inodes. [MaxStor]

OK if your average file size is > 4MB.

> C. Create using -m 0 to reserve no blocks for the super-user.
> [NoSysFiles][MaxStor]

Can hit fragmentation problems if fs is almost 100% full.  I'd always
leave at least 1% even for large filesystems.

> D. Create using -E stride=N where N matches the underlying RAID.
> [GenNFSPerf]

Yup.

> E. Use a kernel >= 2.6.19 (patches for extents and 48-bit support,
> requires Ubuntu 7.04 feisty or Fedora Core 7 or custom kernel) to allow
> filesystems > 8TB on Intel/AMD chips. [BigFS]

This means ext4.  I'd suggest including the mballoc and delalloc patches
from Alex if performance is the driving factor.  There is some risk
involved in ext4.

> F. Use an external journal on a separate high-RPM drive. [GenNFSPerf]

The journal is mostly sequential writes, so high-RPM is not critical here.

> G. Use a large journal. mkfs -J size=8192 [GenNFSPerf]

This will consume 8GB of RAM on your server, per filesystem.  Watch out.
You probably don't get any benefit beyond 1 or 2 GB.

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.

_______________________________________________
Ext3-users mailing list
Ext3-users@...
https://www.redhat.com/mailman/listinfo/ext3-users

[Ralf Hildebrandt]

* Andreas Dilger <adilger@...>:

> > E. Use a kernel >= 2.6.19 (patches for extents and 48-bit support,
> > requires Ubuntu 7.04 feisty or Fedora Core 7 or custom kernel) to allow
> > filesystems > 8TB on Intel/AMD chips. [BigFS]
>
> This means ext4.  I'd suggest including the mballoc and delalloc patches
> from Alex if performance is the driving factor.  There is some risk
> involved in ext4.

I don't understand this: The whole idea was to stay "supported", and
now the OP want's to use hand-built kernels? Why?

--
Ralf Hildebrandt (i.A. des IT-Zentrums)         Ralf.Hildebrandt@...
Charite - Universitätsmedizin Berlin            Tel.  [phone]
Gemeinsame Einrichtung von FU- und HU-Berlin    Fax.  [phone]
IT-Zentrum Standort CBF                    send no mail to plonk@...

_______________________________________________
Ext3-users mailing list
Ext3-users@...
https://www.redhat.com/mailman/listinfo/ext3-users

[Tod Hagan]

On Thu, 2007-05-17 at 23:25 +0200, Ralf Hildebrandt wrote:

> * Andreas Dilger <adilger@...>:
>
> > > E. Use a kernel >= 2.6.19 (patches for extents and 48-bit support,
> > > requires Ubuntu 7.04 feisty or Fedora Core 7 or custom kernel) to allow
> > > filesystems > 8TB on Intel/AMD chips. [BigFS]
> >
> > This means ext4.  I'd suggest including the mballoc and delalloc patches
> > from Alex if performance is the driving factor.  There is some risk
> > involved in ext4.
>
> I don't understand this: The whole idea was to stay "supported", and
> now the OP want's to use hand-built kernels? Why?

Yes, this item contradicts the constraint of using RHEL-supported ext3
and will be removed from the next revision of the document. In its place
will be a note that ext3 filesystems > 8TB on Intel/AMD chips aren't
possible on vanilla RHEL.

Thanks.

Tod

--
Tod Hagan
Information Technologist
AIRMAP/Climate Change Research Center
Institute for the Study of Earth, Oceans, and Space
University of New Hampshire
Durham, NH 03824
Phone: [phone]


_______________________________________________
Ext3-users mailing list
Ext3-users@...
https://www.redhat.com/mailman/listinfo/ext3-users

[Mingming Caoc]

On Thu, 2007-05-17 at 18:25 -0400, Tod Hagan wrote:

> On Thu, 2007-05-17 at 23:25 +0200, Ralf Hildebrandt wrote:
> > * Andreas Dilger <adilger@...>:
> >
> > > > E. Use a kernel >= 2.6.19 (patches for extents and 48-bit support,
> > > > requires Ubuntu 7.04 feisty or Fedora Core 7 or custom kernel) to allow
> > > > filesystems > 8TB on Intel/AMD chips. [BigFS]
> > >
> > > This means ext4.  I'd suggest including the mballoc and delalloc patches
> > > from Alex if performance is the driving factor.  There is some risk
> > > involved in ext4.
> >
> > I don't understand this: The whole idea was to stay "supported", and
> > now the OP want's to use hand-built kernels? Why?
>
> Yes, this item contradicts the constraint of using RHEL-supported ext3
> and will be removed from the next revision of the document. In its place
> will be a note that ext3 filesystems > 8TB on Intel/AMD chips aren't
> possible on vanilla RHEL.
>

Do you mean >8TB or >16TB?

Linux kernel >2.6.18 fixed ext3 kernel bugs so ext3 can support >8TB
filesystem. The filesystem size is still limited up to 16TB due to 32
bit block numbers (based on 4k default block size). It needs
e2fsprogs-1.39 to able to create a fs >8TB.

To get >16TB filesystem you either move to ext4(avaible >= 2.6.19) or
apply 48bit extents patch though, the later is not in mainline.

Mingming


_______________________________________________
Ext3-users mailing list
Ext3-users@...
https://www.redhat.com/mailman/listinfo/ext3-users

[Mingming Caoc]

On Thu, 2007-05-17 at 16:16 -0400, Tod Hagan wrote:

> All,
>
> I'm requesting comments from the expert readers of ext3-users on these
> notes for tuning ext3 for performance.
>
> Most helpful would be feedback pertinent to RHEL 5; as XFS isn't
> supported under Red Hat Enterprise Linux these items are an attempt to
> match XFS performance with ext3.
>
> These items were culled from a number of sources. Will they be effective
> for achieving the performance goals? Is the risk assessment correct?
> Are any too risky for consideration? Has anything been left out?
>
> Thanks.
>
>
>
> ext3 Performance Tuning
>
> This document contains notes about optimizing ext3 filesystems for
> storage nodes on Linux clusters used for scientific research.
>
> Each of the goals and assumptions has been assigned an identifier in
> square brackets which is referenced in the applicable tuning options.
>
> Goals:
>
> [BigSeqIO] Maximize sequential I/O performance with large files (> 1GB),
> including simultaneous access to several large files.
>
> [MaxStor] Maximize the amount of usable storage.
>
> [BigFS] Allow large filesystems (> 8 TB).
>
> [GenNFSPerf] Optimize for NFS and general performance.
>
> Assumptions:
>
> [NoHA] High availability isn't needed.
>
> [IntegFS] The integrity of the filesystem is important -- don't lose
> existing files.
>
> [Recreate] Newly-created files are unimportant as they can be
> re-created.
>
> [UPS] Power to the hardware is guaranteed by a UPS.
>
> [NoBigDir] Directories don't get large.
>
> [NoSysFiles] The filesystems hold data and not system files.
>
> Tuning:
>
> A. Create using -O sparse_super (default) to save space on large
> filesystems. [MaxStor]
>
> B. Create using -T largefile4 (one inode per 4 megabytes) to avoid
> wasting space on unused inodes. [MaxStor]
>
> C. Create using -m 0 to reserve no blocks for the super-user.
> [NoSysFiles][MaxStor]
>
> D. Create using -E stride=N where N matches the underlying RAID.
> [GenNFSPerf]
>
> E. Use a kernel >= 2.6.19 (patches for extents and 48-bit support,
> requires Ubuntu 7.04 feisty or Fedora Core 7 or custom kernel) to allow
> filesystems > 8TB on Intel/AMD chips. [BigFS]
>
> F. Use an external journal on a separate high-RPM drive. [GenNFSPerf]
>
> G. Use a large journal. mkfs -J size=8192 [GenNFSPerf]
>
> H. Mount using -o orlov to use the Orlov block allocator (default,
> requires 2.6 kernel). Minimizes seeks by clustering files together. No
> risk. [GenNFSPerf]
>
> I. Mount using -o noatime,nodiratime. No risk. [GenNFSPerf]
>
> J. Mount using -o reservation to speed writes to multiple files in the
> same directory. No risk. [BigSeqIO][GenNFSPerf]
>
> K. Mount using -o data=writeback. Comparable to XFS and JFS, relaxes all
> restrictions on writing cached data. Risky.
> [Recreate][UPS][BigSeqIO][GenNFSPerf]
>
> L. Mount using -o commit=Nsec where N > 5 (requires 2.6 kernel, default
> is 5 seconds). Reduces sync interval. Risky. [Recreate][UPS][GenNFSPerf]
>
> M. Mount using -o barrier=0 and enable write-back caching on the
> controllers and drives. The most(?) risky. [Recreate][UPS][GenNFSPerf]
>
>

Two more things:
1. Using "mke2fs -I [inode size]" option at format time to allow ext3
support larger inode sizes.  The inode size can be any power-of-two
(larger than default 128 bytes size, up to the filesystem block size).
With larger inode,  extended attributes can be stored in inode body
directly, thus greatly improve performance on workloads which heavily
uses extended attributes (xattrs).

2. Turn on dir_index feature on ext3 fs with " tue2fs -O dir_index".
This will greatly speed  up lookups in directories with lots of files.
This feature also can be enabled via /etc/mkfs.conf comes with
e2fsprogs-1.39


Mingming

_______________________________________________
Ext3-users mailing list
Ext3-users@...
https://www.redhat.com/mailman/listinfo/ext3-users