block network devices

[Dan Ritter-2]

I've been looking at cluster filesystems over iSCSI and AoE, and
on moderate hardware, performance appears to be execrable.
Anyone around have experience with this sort of thing?

For a simple example, I have:

- a network that I can measure 93 MB/s across
- a disk that I can measure 74 MB/s reading
- a best-case ext3 over AoE of 41 MB/s
- a usual-case ocfs2 over iSCSI of 15MB/s

Anyone have a clue about the disparities between disk speed and
transferring data from that disk over the network?

-dsr-

--
http://tao.merseine.nu/~dsr/eula.html is hereby incorporated by reference.

You can't defend freedom by getting rid of it.
_______________________________________________
Discuss mailing list
Discuss@...
http://lists.blu.org/mailman/listinfo/discuss

[Bill Bogstad]

On Tue, Jun 23, 2009 at 10:35 PM, Dan Ritter<dsr@...> wrote:
>
> I've been looking at cluster filesystems over iSCSI and AoE, and
> on moderate hardware, performance appears to be execrable.
> Anyone around have experience with this sort of thing?
>
> For a simple example, I have:
>
> - a network that I can measure 93 MB/s across

> - a disk that I can measure 74 MB/s reading
> - a best-case ext3 over AoE of 41 MB/s

It seems to me that you are adding both network and filesystem
overhead here.  I just ran some quick 'dd's and I get similar
performance drops between direct reads of /dev/sd* and reads of large
files locally from an ext3 filesystem.  Interestingly,
I have another partition on the same disk formated as a  XFS
filesystem and local reads of a large file from that filesystem
gave me around 70 MB/s.  You might try AoE with XFS rather then ext3
and see what you get.

Bill Bogstad
_______________________________________________
Discuss mailing list
Discuss@...
http://lists.blu.org/mailman/listinfo/discuss

[Dan Ritter-2]

On Tue, Jun 23, 2009 at 11:46:32PM -0400, Bill Bogstad wrote:

> On Tue, Jun 23, 2009 at 10:35 PM, Dan Ritter<dsr@...> wrote:
> >
> > I've been looking at cluster filesystems over iSCSI and AoE, and
> > on moderate hardware, performance appears to be execrable.
> > Anyone around have experience with this sort of thing?
> >
> > For a simple example, I have:
> >
> > - a network that I can measure 93 MB/s across
>
> > - a disk that I can measure 74 MB/s reading
> > - a best-case ext3 over AoE of 41 MB/s
>
> It seems to me that you are adding both network and filesystem
> overhead here.  I just ran some quick 'dd's and I get similar
> performance drops between direct reads of /dev/sd* and reads of large
> files locally from an ext3 filesystem.  Interestingly,
> I have another partition on the same disk formated as a  XFS
> filesystem and local reads of a large file from that filesystem
> gave me around 70 MB/s.  You might try AoE with XFS rather then ext3
> and see what you get.

I can get 73MB/s reading off the ext3 filesystem, so that's not
it. In any case, I don't actually want ext3 or xfs; I'm going to
need a cluster-aware filesystem.

-dsr-

--
http://tao.merseine.nu/~dsr/eula.html is hereby incorporated by reference.

You can't defend freedom by getting rid of it.
_______________________________________________
Discuss mailing list
Discuss@...
http://lists.blu.org/mailman/listinfo/discuss