XEN 3.1 on 2.6.18-164.el5xen kernel - Slow IO & Pauses

[Robert Taylor-20]

I've seen tons of posts about slow IO under Xen and others without slow IO
under xen.   Here's what I'm experiencing.   I've got a XEN system on an 2
dual core AMD Opteron 275 processors.

I'm running DRBD for disk, but have also tested without DRBD.
Throughput in Dom-0 seems to be 50-80MB/sec for a file of any size.
Throughput in DomU is 20-30MB/sec for a file under 2GB, and 1-2MB/sec for
a larger file.

I've seen many other posts which are similar.  I've tried diagnosing and
I'm seeing the the IO seems to be throttled somewhere between the DomU and
the Dom0.   (Since they're LVM partitions, I'm assuming Dom0 handles the
LVM, because there is LVM xen module.)

The strange thing I'm finding is that under the DomU, when the IO is being
blocked, its blocking the entire kernel.   In other words, even non-disk
bound processes are pausing.   For instance if you ssh to the box, even
the network IO for SSH will pause for 10-20 seconds before it responds.  
Its very strange.   I was on 3.0 and didn't seem to see it as bad as when
I moved to 3.1.

xm info:

release                : 2.6.18-164.6.1.el5xen
version                : #1 SMP Tue Nov 3 17:53:47 EST 2009
machine                : i686
nr_cpus                : 4
nr_nodes               : 1
sockets_per_node       : 2
cores_per_socket       : 2
threads_per_core       : 1
cpu_mhz                : 2210
hw_caps                :
178bfbff:e3d3fbff:00000000:00000010:00000001:00000000:00000003
total_memory           : 7422
free_memory            : 0
node_to_cpu            : node0:0-3
xen_major              : 3
xen_minor              : 1
xen_extra              : .2-164.6.1.el5
xen_caps               : xen-3.0-x86_32p
xen_pagesize           : 4096
platform_params        : virt_start=0xf5800000
xen_changeset          : unavailable
cc_compiler            : gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)
cc_compile_by          : mockbuild
cc_compile_domain      : centos.org
cc_compile_date        : Tue Nov  3 16:10:34 EST 2009
xend_config_format     : 2


Any help would be appreciated.  This is under CentOS 5.4, 8GB memory, 1TB
SATA nv-sata controller.   Tyan K8SRE S2891 motherboard.   NVidia nForce
4Port SATAlink controller.



_______________________________________________
Xen-users mailing list
Xen-users@...
http://lists.xensource.com/xen-users

[Fajar A. Nugraha-3]

On Sat, Nov 7, 2009 at 2:55 AM, Robert Taylor <rob@...> wrote:
> I've seen tons of posts about slow IO under Xen and others without slow IO
> under xen.   Here's what I'm experiencing.   I've got a XEN system on an 2
> dual core AMD Opteron 275 processors.
>
> I'm running DRBD for disk, but have also tested without DRBD.
> Throughput in Dom-0 seems to be 50-80MB/sec for a file of any size.
> Throughput in DomU is 20-30MB/sec for a file under 2GB, and 1-2MB/sec for
> a larger file.

How do you measure the throughput?

> Any help would be appreciated.  This is under CentOS 5.4, 8GB memory, 1TB
> SATA nv-sata controller.   Tyan K8SRE S2891 motherboard.   NVidia nForce
> 4Port SATAlink controller.

I'm using a similar software setup (RHEL5), different hardware though.
I'd suggest you start with doing "iostat -mx 3" on both dom0 and domU
to see where the bottleneck is. My guess is that some process is using
lots of I/O (possibly in small sizes) so that you're practically IOPS
bound, and the problem is not directly related to Xen.

--
Fajar

_______________________________________________
Xen-users mailing list
Xen-users@...
http://lists.xensource.com/xen-users

[Robert Taylor-20]

Well, for the most part I'm using dd if=/dev/zero of=test bs=4k
count=1000000

This writes a 4gig file to the disk.   I've tried with 100M, 200M, 400M,
800M, 1G, 2G, 4G file sizes.
100M-800M the throughput is quite impressive.
1G the throughput is still 100+M/sec.
2G+ it goes down to 1-2M/sec on DomU.

----

I've done the iostat, and in most all cases the most of the usage is in the
"await" state on the drive I'm testing.
I'll keep digging to see if I can find out more information.  Thanks for the
reply.
 

-----Original Message-----
From: Fajar A. Nugraha [mailto:fajar@...]
Sent: Sunday, November 08, 2009 9:01 PM
To: Robert Taylor
Cc: xen-users@...
Subject: Re: [Xen-users] XEN 3.1 on 2.6.18-164.el5xen kernel - Slow IO &
Pauses

On Sat, Nov 7, 2009 at 2:55 AM, Robert Taylor <rob@...> wrote:
> I've seen tons of posts about slow IO under Xen and others without
> slow IO under xen.   Here's what I'm experiencing.   I've got a XEN
> system on an 2 dual core AMD Opteron 275 processors.
>
> I'm running DRBD for disk, but have also tested without DRBD.
> Throughput in Dom-0 seems to be 50-80MB/sec for a file of any size.
> Throughput in DomU is 20-30MB/sec for a file under 2GB, and 1-2MB/sec
> for a larger file.

How do you measure the throughput?

> Any help would be appreciated.  This is under CentOS 5.4, 8GB memory,
> 1TB SATA nv-sata controller.   Tyan K8SRE S2891 motherboard.   NVidia
> nForce 4Port SATAlink controller.

I'm using a similar software setup (RHEL5), different hardware though.
I'd suggest you start with doing "iostat -mx 3" on both dom0 and domU to see
where the bottleneck is. My guess is that some process is using lots of I/O
(possibly in small sizes) so that you're practically IOPS bound, and the
problem is not directly related to Xen.

--
Fajar


_______________________________________________
Xen-users mailing list
Xen-users@...
http://lists.xensource.com/xen-users

[Fajar A. Nugraha-3]

On Mon, Nov 9, 2009 at 8:17 PM, Rob Taylor <rob@...> wrote:
> Well, for the most part I'm using dd if=/dev/zero of=test bs=4k
> count=1000000
>
> This writes a 4gig file to the disk.   I've tried with 100M, 200M, 400M,
> 800M, 1G, 2G, 4G file sizes.
> 100M-800M the throughput is quite impressive.
> 1G the throughput is still 100+M/sec.
> 2G+ it goes down to 1-2M/sec on DomU.

Next question: what is the allocated memory for domU and dom0? I'm
guessing around 1GB?
To get meaningful results, you should use file size about 2x available
memory. Otherwise it'd just be cached.

--
Fajar

_______________________________________________
Xen-users mailing list
Xen-users@...
http://lists.xensource.com/xen-users