Getting unexpected results when using small request size in Linux

[Christiaan Willemsen]

We are currently testing our new disk array to see what configuration is
going to give us the best performance for our database. We use
Postgresql, and it (in general) uses requests of 8K to read from disk.
In earlyer real-file test, we could confirm this when looking at iostat
and calculating the request size from transfer rate and iops.

Let's quickly look at the machine:
- dual quad E5430 @2.66 Ghz
- 64 GB of memory
- Adaptec 51645 raid controller, latest firmware
- 16x Seagate 15K.5 SAS

Array setup: 12 disks in (currently raid 6), two in mirror, and two in
mirror

I tested with iometer (both last stable, and latest beta) om Ubuntu
8.04, 8.10, Gentoo with 2.6.27, Centos 5 and SLES10SP, and Windows 2008
64 bit server (eval version).

At first the Linuxes, note that they use various kernels and Adaptec
driver versione (altough I tied to update them all to the latest version):

As in initial test pattern I tested on the raw arrays, with the
following setting:

- 20 workers
- outstanding io's: 32, but also tried 1 and 65 without a lot of difference
- Made an Access specification: transfer size: 8k, 100% read, 100%
sequential, rest default

Then I let it run on the 12 disk array, and on the 2 disk mirror:
12 disk: 140 MB/sec
2 disk: 140 MB/sec

This should not be... So I set to 100% random, 100% read:
12 disk: 1.2 MB/sec
2 disk: 1.2 MB/sec

This should also not be the case. I would expect the 12 disk array to be
much faster... It also does not matter if I use raid 10 or raid 5.
Results are always exactly the same. Also striking was the average IO
response time of more than 100 ms (often even more than 300 ms),
specially when doing random io.

If we get to higher request sizes, performance get better, but response
time is still fairly high.

So I installed Windows, and to my surprise this indeed yielded much
better results, about 14 MB/sec random, and about 600 to 800 MB/sec
sequential. This is what I would expect.

Now is the question: who is to blame for this? So I contacted Adaptec
support, and all they came up with was that the arrays were not
initialized correctly, but after doing as they asked, just as expected
we got the same results. So now they blame the OS. I rather think that
they get more ridiculous after every new answer they give us, and
credibility is falling quickly.

So now I want to make sure that it's not you guys. So I was hoping you
could help me confirming that iometer is right ;) If you want me to do
some test, please tell me so. I really need this issue resolved fast,
because we really need this machine up and running.

Kind regards,

Christiaan Willemsen



-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Iometer-user mailing list
Iometer-user@...
https://lists.sourceforge.net/lists/listinfo/iometer-user

[Christiaan Willemsen]

On Nov 15, 2008, at 12:56 AM, Scott Grumm wrote:

> I have seen iometer on Linux always do IO at Qdepth 1 (number of
> outstanding IOs) regardless of how I have it set to run.  iometer on
> Windows always works as expected.  I'm able to measure the Qdepth on  
> the
> target system.  I'm not sure if it's a kernel issue (2.6.18) or
> something else.

If that is the case, then they should put this in the release notes...  
Since I tested various distros with kernel versions from 2.6.16 to  
2.6.27 I don't think it is a kernel issue. Another pointer would be  
that in a real life test with PostgreSQL the controller also appeared  
to be very slow. I now installed FreeBSD, and it just rocks. To bad I  
can't test with iometer on it :(

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Iometer-user mailing list
Iometer-user@...
https://lists.sourceforge.net/lists/listinfo/iometer-user

[Mark_Lindholm]

Does anyone have a newer WIZARD.mdb , that will work with OFFICE 2007?

The newest one I could find was from 1999.  When I try to graph into
EXCEL it errors.  It says I have EXCEL 12.0 and it wants Version 8.0
(EXCEL 97) or later.

Mark Lindholm
Enterprise HDD Engineering
512 725 3195
mark_lindholm@...

------------------------------------------------------------------------------
Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM)
software. With Adobe AIR, Ajax developers can use existing skills and code to
build responsive, highly engaging applications that combine the power of local
resources and data with the reach of the web. Download the Adobe AIR SDK and
Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com
_______________________________________________
Iometer-user mailing list
Iometer-user@...
https://lists.sourceforge.net/lists/listinfo/iometer-user

[mcRane]

Look at O_DIRECT in the source for IOMeter. I was banging my head against the same types of results for weeks until I got that figured out. Remove the O_DIRECT flag and recompile dynamo and you will see results that more accurately reflect what you can expect to see.