Performance issues with 8.0 ZFS and sendfile/lighttpd

[gnukix]

I can send in more documentation later but I am seeing severe zfs performance issues with lighttpd. Same machine using UFS will push 1gbit or more but same content and traffic load can not hit 200mbit. Ufs does around 3 megabytes/sec IO at 800mbit network but zfs pushes the disks into the ground with 50+ megabytes/sec dusk i/o. No compression no atime no checksums on zfs and still same IO levels. Ufs with soft updates and atime on. Orders of magnitude more disk IO... Like zfs isn't using cache or isn't coalescing disk reads or both.

Has anyone else seen this or have any recommendations? Lighttpd config remains exactly the same as well FYI. Only difference is ufs vs zfs.



Sent from my BlackBerry Smartphone provided by Alltel
_______________________________________________
freebsd-stable@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@..."

[Ivan Voras-7]

gnukix@... wrote:
> I can send in more documentation later but I am seeing severe zfs performance issues with lighttpd. Same machine using UFS will push 1gbit or more but same content and traffic load can not hit 200mbit. Ufs does around 3 megabytes/sec IO at 800mbit network but zfs pushes the disks into the ground with 50+ megabytes/sec dusk i/o. No compression no atime no checksums on zfs and still same IO levels. Ufs with soft updates and atime on. Orders of magnitude more disk IO... Like zfs isn't using cache or isn't coalescing disk reads or both.
>
> Has anyone else seen this or have any recommendations? Lighttpd config remains exactly the same as well FYI. Only difference is ufs vs zfs.

AFAIK, ZFS is incompatible (currently) with some advanced VM operations
(like mmap, and I think sendfile relies on the same mechanism as mmap),
so that could be a cause of the slowdown. Though I'm surprised you can
only get 200 MBit/s - that's 25 MB/s and I think that even with multiple
memcpy-ing data around the kernel you should be able to get hundreds of
MB/s on newer hardware (which normally really can achieve tens of
gigabytes/s of sustained memory access).

What else can you observe from your system? Do you have exceedingly high
sys times and load numbers? I'm also interested in what does 10 seconds
of running 'vmstat 1' looks like on your system. Is it a bare machine or
a virtual machine?


_______________________________________________
freebsd-stable@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@..."

[Miroslav Lachman]

Ivan Voras wrote:

> gnukix@... wrote:
>> I can send in more documentation later but I am seeing severe zfs
>> performance issues with lighttpd. Same machine using UFS will push
>> 1gbit or more but same content and traffic load can not hit 200mbit.
>> Ufs does around 3 megabytes/sec IO at 800mbit network but zfs pushes
>> the disks into the ground with 50+ megabytes/sec dusk i/o. No
>> compression no atime no checksums on zfs and still same IO levels. Ufs
>> with soft updates and atime on. Orders of magnitude more disk IO...
>> Like zfs isn't using cache or isn't coalescing disk reads or both.
>> Has anyone else seen this or have any recommendations? Lighttpd config
>> remains exactly the same as well FYI. Only difference is ufs vs zfs.
>
> AFAIK, ZFS is incompatible (currently) with some advanced VM operations
> (like mmap, and I think sendfile relies on the same mechanism as mmap),
> so that could be a cause of the slowdown. Though I'm surprised you can
> only get 200 MBit/s - that's 25 MB/s and I think that even with multiple
> memcpy-ing data around the kernel you should be able to get hundreds of
> MB/s on newer hardware (which normally really can achieve tens of
> gigabytes/s of sustained memory access).

I have more strange issue with Lighttpd in jail on top of ZFS. Lighttpd
is serving static content (mp3 downloads thru flash player). Is runs
fine for relatively small number of parallel clients with bandwidth
about 30 Mbps, but after some number of clients is reached (about 50-60
parallel clients) the throughput drops down to 6 Mbps.

I can server hundereds of clients on same HW using Lighttpd not in jail
and UFS2 with gjournal instead of ZFS reaching 100 Mbps (maybe more).

I don't know if it is ZFS or Jail issue.

Miroslav Lachman
_______________________________________________
freebsd-stable@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@..."

[Ivan Voras-7]

Miroslav Lachman wrote:

> Ivan Voras wrote:
>> gnukix@... wrote:
>>> I can send in more documentation later but I am seeing severe zfs
>>> performance issues with lighttpd. Same machine using UFS will push
>>> 1gbit or more but same content and traffic load can not hit 200mbit.
>>> Ufs does around 3 megabytes/sec IO at 800mbit network but zfs pushes
>>> the disks into the ground with 50+ megabytes/sec dusk i/o. No
>>> compression no atime no checksums on zfs and still same IO levels. Ufs
>>> with soft updates and atime on. Orders of magnitude more disk IO...
>>> Like zfs isn't using cache or isn't coalescing disk reads or both.
>>> Has anyone else seen this or have any recommendations? Lighttpd config
>>> remains exactly the same as well FYI. Only difference is ufs vs zfs.
>>
>> AFAIK, ZFS is incompatible (currently) with some advanced VM operations
>> (like mmap, and I think sendfile relies on the same mechanism as mmap),
>> so that could be a cause of the slowdown. Though I'm surprised you can
>> only get 200 MBit/s - that's 25 MB/s and I think that even with multiple
>> memcpy-ing data around the kernel you should be able to get hundreds of
>> MB/s on newer hardware (which normally really can achieve tens of
>> gigabytes/s of sustained memory access).
>
> I have more strange issue with Lighttpd in jail on top of ZFS. Lighttpd
> is serving static content (mp3 downloads thru flash player). Is runs
> fine for relatively small number of parallel clients with bandwidth
> about 30 Mbps, but after some number of clients is reached (about 50-60
> parallel clients) the throughput drops down to 6 Mbps.
>
> I can server hundereds of clients on same HW using Lighttpd not in jail
> and UFS2 with gjournal instead of ZFS reaching 100 Mbps (maybe more).
>
> I don't know if it is ZFS or Jail issue.

Do you have actual disk IO or is the vast majority of your data served
from the caches? (actually - the same question to the OP)

_______________________________________________
freebsd-stable@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@..."

[Miroslav Lachman]

Ivan Voras wrote:
> Miroslav Lachman wrote:

[..]

>> I have more strange issue with Lighttpd in jail on top of ZFS.
>> Lighttpd is serving static content (mp3 downloads thru flash player).
>> Is runs fine for relatively small number of parallel clients with
>> bandwidth about 30 Mbps, but after some number of clients is reached
>> (about 50-60 parallel clients) the throughput drops down to 6 Mbps.
>>
>> I can server hundereds of clients on same HW using Lighttpd not in
>> jail and UFS2 with gjournal instead of ZFS reaching 100 Mbps (maybe
>> more).
>>
>> I don't know if it is ZFS or Jail issue.
>
> Do you have actual disk IO or is the vast majority of your data served
> from the caches? (actually - the same question to the OP)

I had ZFS zpool as mirror of two SATA II drives (500GB) and in the peak
iostat (or systat -vm or gstat) shows about 80 tps / 60% busy.

In case of UFS, I am using gmirrored 1TB SATA II drives working nice
with 160 or more tps.

Both setups are using FreeBSD 7.x amd64 with GENERIC kernel, 4GB of RAM.

As the ZFS + Lighttpd in jail was unreliable, I am no longer using it,
but if you want some more info for debuging, I can set it up again.

Miroslav Lachman
_______________________________________________
freebsd-stable@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@..."

[Ivan Voras-7]

Miroslav Lachman wrote:

> Ivan Voras wrote:
>> Miroslav Lachman wrote:
>
> [..]
>
>>> I have more strange issue with Lighttpd in jail on top of ZFS.
>>> Lighttpd is serving static content (mp3 downloads thru flash player).
>>> Is runs fine for relatively small number of parallel clients with
>>> bandwidth about 30 Mbps, but after some number of clients is reached
>>> (about 50-60 parallel clients) the throughput drops down to 6 Mbps.
>>>
>>> I can server hundereds of clients on same HW using Lighttpd not in
>>> jail and UFS2 with gjournal instead of ZFS reaching 100 Mbps (maybe
>>> more).
>>>
>>> I don't know if it is ZFS or Jail issue.
>>
>> Do you have actual disk IO or is the vast majority of your data served
>> from the caches? (actually - the same question to the OP)
>
> I had ZFS zpool as mirror of two SATA II drives (500GB) and in the peak
> iostat (or systat -vm or gstat) shows about 80 tps / 60% busy.
>
> In case of UFS, I am using gmirrored 1TB SATA II drives working nice
> with 160 or more tps.
>
> Both setups are using FreeBSD 7.x amd64 with GENERIC kernel, 4GB of RAM.
>
> As the ZFS + Lighttpd in jail was unreliable, I am no longer using it,
> but if you want some more info for debuging, I can set it up again.

For what it's worth, I have just set up a little test on a production
machine with 3 500 GB SATA drives in RAIDZ, FreeBSD 7.2-RELEASE. The
total data set is some 2 GB in 5000 files but the machine has only 2 GB
RAM total so there is some disk IO - about 40 IOPS per drive. I'm also
using Apache-worker, not lighty, and siege to benchmark with 10
concurrent users.

In this setup, the machine has no problems saturating a 100 Mbit/s link
- it's not on a LAN but the latency is close enough and I get ~~ 11 MB/s.

_______________________________________________
freebsd-stable@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@..."

[Urmas Lett]

gnukix@... wrote:
> I can send in more documentation later but I am seeing severe zfs performance issues with lighttpd. Same machine using UFS will push 1gbit or more but same content and traffic load can not hit 200mbit. Ufs does around 3 megabytes/sec IO at 800mbit network but zfs pushes the disks into the ground with 50+ megabytes/sec dusk i/o. No compression no atime no checksums on zfs and still same IO levels. Ufs with soft updates and atime on. Orders of magnitude more disk IO... Like zfs isn't using cache or isn't coalescing disk reads or both.
>
> Has anyone else seen this or have any recommendations? Lighttpd config remains exactly the same as well FYI. Only difference is ufs vs zfs.

Hi,

I can confirm this on FreeBSD 7.2-STABLE.

first run:
:~# wget -O /dev/null http://192.168.1.1/1000m.bin
--2009-11-04 15:36:15--  http://192.168.1.1/1000m.bin
Connecting to 192.168.1.1:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1048576000 (1000M) [application/octet-stream]
Saving to: `/dev/null'
100%[====================================>] 1,048,576,000 17.0M/s   in
81s
2009-11-04 15:37:36 (12.3 MB/s) - `/dev/null' saved [1048576000/1048576000]

second run is even slower, cannot wait till end:
:~# wget -O /dev/null http://192.168.1.1/1000m.bin
--2009-11-04 15:40:00--  http://192.168.1.1/1000m.bin
Connecting to 192.168.1.1:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1048576000 (1000M) [application/octet-stream]
Saving to: `/dev/null'
71% [==========================>            ] 752,173,056 2.10M/s  eta
2m 0s   ^C


After changing in lighttpd.conf
server.network-backend = "writev"

first run:
:~# wget -O /dev/null http://192.168.1.1/1000m.bin
--2009-11-04 15:47:51--  http://192.168.1.1/1000m.bin
Connecting to 192.168.1.1:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1048576000 (1000M) [application/octet-stream]
Saving to: `/dev/null'
100%[====================================>] 1,048,576,000 44.1M/s   in
27s
2009-11-04 15:48:18 (37.2 MB/s) - `/dev/null' saved [1048576000/1048576000]

second & third run:

:~# wget -O /dev/null http://192.168.1.1/1000m.bin
--2009-11-04 15:48:20--  http://192.168.1.1/1000m.bin
Connecting to 192.168.1.1:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1048576000 (1000M) [application/octet-stream]
Saving to: `/dev/null'
100%[====================================>] 1,048,576,000  788M/s   in
1.3s
2009-11-04 15:48:21 (788 MB/s) - `/dev/null' saved [1048576000/1048576000]

:~# wget -O /dev/null http://192.168.1.1/1000m.bin
--2009-11-04 15:48:24--  http://192.168.1.1/1000m.bin
Connecting to 192.168.1.1:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1048576000 (1000M) [application/octet-stream]
Saving to: `/dev/null'
100%[====================================>] 1,048,576,000  910M/s   in
1.1s
2009-11-04 15:48:25 (910 MB/s) - `/dev/null' saved [1048576000/1048576000]

I have Intel 10GbE PCI-Express directly connected between two servers.

--
Urmas Lett

Tel:    +(372) 7 302 110
Fax:    +(372) 7 302 111
E-Mail: urmas.lett@...
_______________________________________________
freebsd-stable@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@..."

[Miroslav Lachman]

Ivan Voras wrote:

> Miroslav Lachman wrote:
>> Ivan Voras wrote:
>>> Miroslav Lachman wrote:
>>
>> [..]
>>
>>>> I have more strange issue with Lighttpd in jail on top of ZFS.
>>>> Lighttpd is serving static content (mp3 downloads thru flash player).
>>>> Is runs fine for relatively small number of parallel clients with
>>>> bandwidth about 30 Mbps, but after some number of clients is reached
>>>> (about 50-60 parallel clients) the throughput drops down to 6 Mbps.
>>>>
>>>> I can server hundereds of clients on same HW using Lighttpd not in
>>>> jail and UFS2 with gjournal instead of ZFS reaching 100 Mbps (maybe
>>>> more).
>>>>
>>>> I don't know if it is ZFS or Jail issue.
>>>
>>> Do you have actual disk IO or is the vast majority of your data served
>>> from the caches? (actually - the same question to the OP)
>>
>> I had ZFS zpool as mirror of two SATA II drives (500GB) and in the
>> peak iostat (or systat -vm or gstat) shows about 80 tps / 60% busy.
>>
>> In case of UFS, I am using gmirrored 1TB SATA II drives working nice
>> with 160 or more tps.
>>
>> Both setups are using FreeBSD 7.x amd64 with GENERIC kernel, 4GB of RAM.
>>
>> As the ZFS + Lighttpd in jail was unreliable, I am no longer using it,
>> but if you want some more info for debuging, I can set it up again.
>
> For what it's worth, I have just set up a little test on a production
> machine with 3 500 GB SATA drives in RAIDZ, FreeBSD 7.2-RELEASE. The
> total data set is some 2 GB in 5000 files but the machine has only 2 GB
> RAM total so there is some disk IO - about 40 IOPS per drive. I'm also
> using Apache-worker, not lighty, and siege to benchmark with 10
> concurrent users.
>
> In this setup, the machine has no problems saturating a 100 Mbit/s link
> - it's not on a LAN but the latency is close enough and I get ~~ 11 MB/s.

I tried it again to get some system statistics for you, so here it comes.

I do not understand why there are 10MB/s read from disks when network
traffic dropped to around 1MB/s (8Mbps)

root@cage ~/# iostat -w 20
       tty             ad4              ad6             cpu
  tin tout  KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
    0   14 41.66  53  2.17  41.82  53  2.18   0  0  2  0 97
    0   18 50.92  96  4.77  54.82 114  6.12   0  0  3  1 96
    0    6 53.52 101  5.29  54.98 108  5.81   1  0  4  1 94
    0    6 54.82  98  5.26  55.89 108  5.89   0  0  3  1 96


root@cage ~/# ifstat -i bge1 10
        bge1
  KB/s in  KB/s out
    33.32   1174.34
    34.35   1181.33
    33.14   1172.27
    31.64   1118.60


root@cage ~/# zpool iostat 10
                capacity     operations    bandwidth
pool         used  avail   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
tank         382G  62.5G     73     31  3.30M   148K
tank         382G  62.5G    150     38  11.2M   138K
tank         382G  62.5G    148     33  11.3M  99.6K
tank         382G  62.5G    148     29  10.9M  93.2K
tank         382G  62.5G    137     25  10.4M  75.4K
tank         382G  62.5G    149     32  11.3M   122K


root@cage ~/# ~/bin/zfs_get_kernel_mem.sh
TEXT=13245157, 12.6316 MB
DATA=267506688, 255.114 MB
TOTAL=280751845, 267.746 MB


root@cage ~/# ~/bin/arcstat.pl 10
     Time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  arcsz     c
15:34:38  705M   46M      6   46M    6     0    0   29M   18   137061376
   134217728
15:34:48    1K   148     11   148   11     0    0    57   96   137495552
   134217728
15:34:58    1K   151     11   151   11     0    0    59   96   136692736
   134217728
15:35:08    1K   140     10   140   10     0    0    45   76   165005824
   134217728
15:35:18    1K   150      9   150    9     0    0    54   91   141642240
   134217728


root@cage ~/# ~/bin/arc_summary.pl
System Memory:
          Physical RAM:  4083 MB
          Free Memory :  0 MB

ARC Size:
          Current Size:             133 MB (arcsize)
          Target Size (Adaptive):   128 MB (c)
          Min Size (Hard Limit):    16 MB (zfs_arc_min)
          Max Size (Hard Limit):    128 MB (zfs_arc_max)

ARC Size Breakdown:
          Most Recently Used Cache Size:          97%    125 MB (p)
          Most Frequently Used Cache Size:         2%    2 MB (c-p)

ARC Efficency:
          Cache Access Total:             7052224705
          Cache Hit Ratio:      93%       6582803808     [Defined State
for buffer]
          Cache Miss Ratio:      6%       469420897      [Undefined
State for Buffer]
          REAL Hit Ratio:       93%       6582803808     [MRU/MFU Hits Only]

          Data Demand   Efficiency:    96%
          Data Prefetch Efficiency:    DISABLED (zfs_prefetch_disable)

         CACHE HITS BY CACHE LIST:
           Anon:                       --%        Counter Rolled.
           Most Recently Used:         13%        869219380 (mru)
  [ Return Customer ]
           Most Frequently Used:       86%        5713584428 (mfu)
  [ Frequent Customer ]
           Most Recently Used Ghost:    0%        25025402 (mru_ghost)
  [ Return Customer Evicted, Now Back ]
           Most Frequently Used Ghost:  1%        103104325 (mfu_ghost)
  [ Frequent Customer Evicted, Now Back ]
         CACHE HITS BY DATA TYPE:
           Demand Data:                80%        5331503088
           Prefetch Data:               0%        0
           Demand Metadata:            19%        1251300720
           Prefetch Metadata:           0%        0
         CACHE MISSES BY DATA TYPE:
           Demand Data:                38%        179172125
           Prefetch Data:               0%        0
           Demand Metadata:            61%        290248772
           Prefetch Metadata:           0%        0
---------------------------------------------


/boot/loader.conf:

## eLOM support
hw.bge.allow_asf="1"
## gmirror RAID1
geom_mirror_load="YES"
## ZFS tuning
vm.kmem_size="1280M"
vm.kmem_size_max="1280M"
kern.maxvnodes="400000"
vfs.zfs.prefetch_disable="1"
vfs.zfs.arc_min="16M"
vfs.zfs.arc_max="128M"

The network traffic is normally around 30Mbps, but when number of
parallel downloads reaches some level, traffic drops to 6-8Mbps and
number of parallel clients goes up even more.

I can provide network and disk IO graphs if you are interested.

Miroslav Lachman
_______________________________________________
freebsd-stable@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@..."

[Thomas Backman]

On Nov 6, 2009, at 7:36 PM, Miroslav Lachman wrote:

> Ivan Voras wrote:
>> Miroslav Lachman wrote:
>>> Ivan Voras wrote:
>>>> Miroslav Lachman wrote:
>>>
>>> [..]
>>>
>>>>> I have more strange issue with Lighttpd in jail on top of ZFS.
>>>>> Lighttpd is serving static content (mp3 downloads thru flash  
>>>>> player).
>>>>> Is runs fine for relatively small number of parallel clients with
>>>>> bandwidth about 30 Mbps, but after some number of clients is  
>>>>> reached
>>>>> (about 50-60 parallel clients) the throughput drops down to 6  
>>>>> Mbps.
>>>>>
>>>>> I can server hundereds of clients on same HW using Lighttpd not in
>>>>> jail and UFS2 with gjournal instead of ZFS reaching 100 Mbps  
>>>>> (maybe
>>>>> more).
>>>>>
>>>>> I don't know if it is ZFS or Jail issue.
>>>>
>>>> Do you have actual disk IO or is the vast majority of your data  
>>>> served
>>>> from the caches? (actually - the same question to the OP)
>>>
>>> I had ZFS zpool as mirror of two SATA II drives (500GB) and in the
>>> peak iostat (or systat -vm or gstat) shows about 80 tps / 60% busy.
>>>
>>> In case of UFS, I am using gmirrored 1TB SATA II drives working nice
>>> with 160 or more tps.
>>>
>>> Both setups are using FreeBSD 7.x amd64 with GENERIC kernel, 4GB  
>>> of RAM.
>>>
>>> As the ZFS + Lighttpd in jail was unreliable, I am no longer using  
>>> it,
>>> but if you want some more info for debuging, I can set it up again.
>>
>> For what it's worth, I have just set up a little test on a production
>> machine with 3 500 GB SATA drives in RAIDZ, FreeBSD 7.2-RELEASE. The
>> total data set is some 2 GB in 5000 files but the machine has only  
>> 2 GB
>> RAM total so there is some disk IO - about 40 IOPS per drive. I'm  
>> also
>> using Apache-worker, not lighty, and siege to benchmark with 10
>> concurrent users.
>>
>> In this setup, the machine has no problems saturating a 100 Mbit/s  
>> link
>> - it's not on a LAN but the latency is close enough and I get ~~ 11  
>> MB/s.
>
> [...]
> /boot/loader.conf:
>
> ## eLOM support
> hw.bge.allow_asf="1"
> ## gmirror RAID1
> geom_mirror_load="YES"
> ## ZFS tuning
> vm.kmem_size="1280M"
> vm.kmem_size_max="1280M"
> kern.maxvnodes="400000"
> vfs.zfs.prefetch_disable="1"
> vfs.zfs.arc_min="16M"
> vfs.zfs.arc_max="128M"

I won't pretend to know much about this area, but your ZFS values here  
are very low. May I assume that they are remnants of the times when  
the ARC grew insanely large and caused a kernel panic?
You're effectively forcing ZFS to not use more than 128MB cache, which  
doesn't sound like a great idea if you've got 2+ GB of RAM. I've had  
no trouble without any tuning whatsoever on 2GB for a long time now.  
The kmem lines can probably be omitted if you're on amd64, too (the  
default value for kmem_size_max is about 307GB on my machine).

Regards,
Thomas
_______________________________________________
freebsd-stable@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@..."

[Ivan Voras-7]

2009/11/6 Miroslav Lachman <000.fbsd@...>:

> I do not understand why there are 10MB/s read from disks when network
> traffic dropped to around 1MB/s (8Mbps)
>
> root@cage ~/# iostat -w 20
>      tty             ad4              ad6             cpu
>  tin tout  KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
>   0   14 41.66  53  2.17  41.82  53  2.18   0  0  2  0 97
>   0   18 50.92  96  4.77  54.82 114  6.12   0  0  3  1 96
>   0    6 53.52 101  5.29  54.98 108  5.81   1  0  4  1 94
>   0    6 54.82  98  5.26  55.89 108  5.89   0  0  3  1 96

Yes, this could limit your IO if the requests are random enough.
Unfortunately I don't know how would you track down what is really
going on. Maybe some tracing with DTrace?

I'd tell you to use "top -m io" to see if there is a process
responsible, but apparently these statistics are not updated for ZFS,
which in itself may be a bug (which is why I'm crossposting to
freebsd-fs).
_______________________________________________
freebsd-stable@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@..."

[Michael Loftis]

--On Monday, November 02, 2009 10:52 AM +0100 Ivan Voras
<ivoras@...> wrote:

> AFAIK, ZFS is incompatible (currently) with some advanced VM operations
> (like mmap, and I think sendfile relies on the same mechanism as mmap),
> so that could be a cause of the slowdown. Though I'm surprised you can
> only get 200 MBit/s - that's 25 MB/s and I think that even with multiple
> memcpy-ing data around the kernel you should be able to get hundreds of
> MB/s on newer hardware (which normally really can achieve tens of
> gigabytes/s of sustained memory access).
>
> What else can you observe from your system? Do you have exceedingly high
> sys times and load numbers? I'm also interested in what does 10 seconds
> of running 'vmstat 1' looks like on your system. Is it a bare machine or
> a virtual machine?

Real hardware, dual quad core opteron with 64GB memory, a 3Ware 9650SE for
disks.  The rest of the machine is not doing much if anything when the
issue happens.  I've had to remove ZFS from all of the media streaming
servers.  I ca probably get one up for testing again over the next few
weeks.  I've some more hardware coming in.
_______________________________________________
freebsd-stable@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@..."

[Michael Loftis]

--On Monday, November 02, 2009 12:06 PM +0100 Miroslav Lachman
<000.fbsd@...> wrote:

> I have more strange issue with Lighttpd in jail on top of ZFS. Lighttpd
> is serving static content (mp3 downloads thru flash player). Is runs fine
> for relatively small number of parallel clients with bandwidth about 30
> Mbps, but after some number of clients is reached (about 50-60 parallel
> clients) the throughput drops down to 6 Mbps.
>
> I can server hundereds of clients on same HW using Lighttpd not in jail
> and UFS2 with gjournal instead of ZFS reaching 100 Mbps (maybe more).
>
> I don't know if it is ZFS or Jail issue.

Check iostat 5 or zpool iostat 5 -- I bet you're disk thrashing when you
start to slow down.
_______________________________________________
freebsd-stable@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@..."

[Michael Loftis]

--On Monday, November 02, 2009 12:55 PM +0100 Ivan Voras
<ivoras@...> wrote:

> Do you have actual disk IO or is the vast majority of your data served
> from the caches? (actually - the same question to the OP)

That's the problem 64GB of RAM and ZFS doesn't seem to use any cache.  It
also seems to not be realizing when multiple reads are on the same block
(same issue sorta) and dispatches the same I/O request.
_______________________________________________
freebsd-stable@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@..."

[Ben Kelly-3]

On Nov 6, 2009, at 2:53 PM, Michael Loftis wrote:

> --On Monday, November 02, 2009 12:55 PM +0100 Ivan Voras <ivoras@...
> > wrote:
>
>> Do you have actual disk IO or is the vast majority of your data  
>> served
>> from the caches? (actually - the same question to the OP)
>
> That's the problem 64GB of RAM and ZFS doesn't seem to use any  
> cache.  It also seems to not be realizing when multiple reads are on  
> the same block (same issue sorta) and dispatches the same I/O request.

Have you tried adjusting vfs.zfs.arc_meta_limit?  I've noticed the  
default value for this is set poorly when the overall ARC size is  
small.  This happens because various structure's not actually  
allocated from the ARC like dnodes and dbufs are included in the  
metadata usage stats.  When the ARC is large this is somewhat  
negligible, but for small ARCs it overwhelms the calculated default  
metadata limit and you end up not caching any real file system metadata.

You could try increasing the metadata limit or even better increase  
your maximum ARC size to something over 1GB.  (I seem to remember your  
ARC size is 128M from earlier in the thread).

Hope that helps.

- Ben

_______________________________________________
freebsd-stable@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@..."

[Michael Loftis]

--On Friday, November 06, 2009 3:31 PM -0500 Ben Kelly <ben@...>
wrote:

> Have you tried adjusting vfs.zfs.arc_meta_limit?  I've noticed the
> default value for this is set poorly when the overall ARC size is small.
> This happens because various structure's not actually allocated from the
> ARC like dnodes and dbufs are included in the metadata usage stats.  When
> the ARC is large this is somewhat negligible, but for small ARCs it
> overwhelms the calculated default metadata limit and you end up not
> caching any real file system metadata.
>
> You could try increasing the metadata limit or even better increase your
> maximum ARC size to something over 1GB.  (I seem to remember your ARC
> size is 128M from earlier in the thread).

Nope, that was someone else, not setting any ARC limits (or any ZFS
settings at all actually).

>
> Hope that helps.
>
> - Ben
>




_______________________________________________
freebsd-stable@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@..."

[Miroslav Lachman]

Michael Loftis wrote:

>> I have more strange issue with Lighttpd in jail on top of ZFS. Lighttpd
>> is serving static content (mp3 downloads thru flash player). Is runs fine
>> for relatively small number of parallel clients with bandwidth about 30
>> Mbps, but after some number of clients is reached (about 50-60 parallel
>> clients) the throughput drops down to 6 Mbps.
>>
>> I can server hundereds of clients on same HW using Lighttpd not in jail
>> and UFS2 with gjournal instead of ZFS reaching 100 Mbps (maybe more).
>>
>> I don't know if it is ZFS or Jail issue.
>
>
> Check iostat 5 or zpool iostat 5 -- I bet you're disk thrashing when you
> start to slow down.

iostat and zpool was posted in another message.
gstat or systat -vm is showing about 60% busy disk and even if there is
high IO on the disks, lighttpd serving the same content from gmirrored
UFS2 with gjournal and not in jail is serving three times more clients
and bandwidth without this drop down behavior.
Both machines are Sun Fire X2100 M2 with SATA disks.

Miroslav Lachman
_______________________________________________
freebsd-stable@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@..."

[Miroslav Lachman]

Thomas Backman wrote:

> On Nov 6, 2009, at 7:36 PM, Miroslav Lachman wrote:
>
>> Ivan Voras wrote:
>>> Miroslav Lachman wrote:
>>>> Ivan Voras wrote:
>>>>> Miroslav Lachman wrote:
>>>>
>>>> [..]
>>>>
>>>>>> I have more strange issue with Lighttpd in jail on top of ZFS.
>>>>>> Lighttpd is serving static content (mp3 downloads thru flash player).
>>>>>> Is runs fine for relatively small number of parallel clients with
>>>>>> bandwidth about 30 Mbps, but after some number of clients is reached
>>>>>> (about 50-60 parallel clients) the throughput drops down to 6 Mbps.
>>>>>>
>>>>>> I can server hundereds of clients on same HW using Lighttpd not in
>>>>>> jail and UFS2 with gjournal instead of ZFS reaching 100 Mbps (maybe
>>>>>> more).
>>>>>>
>>>>>> I don't know if it is ZFS or Jail issue.
>>>>>
>>>>> Do you have actual disk IO or is the vast majority of your data served
>>>>> from the caches? (actually - the same question to the OP)
>>>>
>>>> I had ZFS zpool as mirror of two SATA II drives (500GB) and in the
>>>> peak iostat (or systat -vm or gstat) shows about 80 tps / 60% busy.
>>>>
>>>> In case of UFS, I am using gmirrored 1TB SATA II drives working nice
>>>> with 160 or more tps.
>>>>
>>>> Both setups are using FreeBSD 7.x amd64 with GENERIC kernel, 4GB of
>>>> RAM.
>>>>
>>>> As the ZFS + Lighttpd in jail was unreliable, I am no longer using it,
>>>> but if you want some more info for debuging, I can set it up again.
>>>
>>> For what it's worth, I have just set up a little test on a production
>>> machine with 3 500 GB SATA drives in RAIDZ, FreeBSD 7.2-RELEASE. The
>>> total data set is some 2 GB in 5000 files but the machine has only 2 GB
>>> RAM total so there is some disk IO - about 40 IOPS per drive. I'm also
>>> using Apache-worker, not lighty, and siege to benchmark with 10
>>> concurrent users.
>>>
>>> In this setup, the machine has no problems saturating a 100 Mbit/s link
>>> - it's not on a LAN but the latency is close enough and I get ~~ 11
>>> MB/s.
>>
>> [...]
>> /boot/loader.conf:
>>
>> ## eLOM support
>> hw.bge.allow_asf="1"
>> ## gmirror RAID1
>> geom_mirror_load="YES"
>> ## ZFS tuning
>> vm.kmem_size="1280M"
>> vm.kmem_size_max="1280M"
>> kern.maxvnodes="400000"
>> vfs.zfs.prefetch_disable="1"
>> vfs.zfs.arc_min="16M"
>> vfs.zfs.arc_max="128M"

> I won't pretend to know much about this area, but your ZFS values here
> are very low. May I assume that they are remnants of the times when the
> ARC grew insanely large and caused a kernel panic?
> You're effectively forcing ZFS to not use more than 128MB cache, which
> doesn't sound like a great idea if you've got 2+ GB of RAM. I've had no
> trouble without any tuning whatsoever on 2GB for a long time now. The
> kmem lines can probably be omitted if you're on amd64, too (the default
> value for kmem_size_max is about 307GB on my machine).

Yes, loader values are one year old when I installed this machine. But I
think auto tuning was commited after 7.2-RELEASE by Kip Macy, so some of
them are still needed or am I wrong? (this is 7.2-RELEASE). I can grow
arc_max but as this machine is running about 6 jails (not CPU or disk IO
consuming), I still need some memory for processes, not just for filesystem.
_______________________________________________
freebsd-stable@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@..."

[Jeremy Chadwick]

On Fri, Nov 06, 2009 at 11:41:12PM +0100, Miroslav Lachman wrote:

> Thomas Backman wrote:
> >On Nov 6, 2009, at 7:36 PM, Miroslav Lachman wrote:
> >
> >>Ivan Voras wrote:
> >>>Miroslav Lachman wrote:
> >>>>Ivan Voras wrote:
> >>>>>Miroslav Lachman wrote:
> >>>>
> >>>>[..]
> >>>>
> >>>>>>I have more strange issue with Lighttpd in jail on top of ZFS.
> >>>>>>Lighttpd is serving static content (mp3 downloads thru flash player).
> >>>>>>Is runs fine for relatively small number of parallel clients with
> >>>>>>bandwidth about 30 Mbps, but after some number of clients is reached
> >>>>>>(about 50-60 parallel clients) the throughput drops down to 6 Mbps.
> >>>>>>
> >>>>>>I can server hundereds of clients on same HW using Lighttpd not in
> >>>>>>jail and UFS2 with gjournal instead of ZFS reaching 100 Mbps (maybe
> >>>>>>more).
> >>>>>>
> >>>>>>I don't know if it is ZFS or Jail issue.
> >>>>>
> >>>>>Do you have actual disk IO or is the vast majority of your data served
> >>>>>from the caches? (actually - the same question to the OP)
> >>>>
> >>>>I had ZFS zpool as mirror of two SATA II drives (500GB) and in the
> >>>>peak iostat (or systat -vm or gstat) shows about 80 tps / 60% busy.
> >>>>
> >>>>In case of UFS, I am using gmirrored 1TB SATA II drives working nice
> >>>>with 160 or more tps.
> >>>>
> >>>>Both setups are using FreeBSD 7.x amd64 with GENERIC kernel, 4GB of
> >>>>RAM.
> >>>>
> >>>>As the ZFS + Lighttpd in jail was unreliable, I am no longer using it,
> >>>>but if you want some more info for debuging, I can set it up again.
> >>>
> >>>For what it's worth, I have just set up a little test on a production
> >>>machine with 3 500 GB SATA drives in RAIDZ, FreeBSD 7.2-RELEASE. The
> >>>total data set is some 2 GB in 5000 files but the machine has only 2 GB
> >>>RAM total so there is some disk IO - about 40 IOPS per drive. I'm also
> >>>using Apache-worker, not lighty, and siege to benchmark with 10
> >>>concurrent users.
> >>>
> >>>In this setup, the machine has no problems saturating a 100 Mbit/s link
> >>>- it's not on a LAN but the latency is close enough and I get ~~ 11
> >>>MB/s.
> >>
> >>[...]
> >>/boot/loader.conf:
> >>
> >>## eLOM support
> >>hw.bge.allow_asf="1"
> >>## gmirror RAID1
> >>geom_mirror_load="YES"
> >>## ZFS tuning
> >>vm.kmem_size="1280M"
> >>vm.kmem_size_max="1280M"
> >>kern.maxvnodes="400000"
> >>vfs.zfs.prefetch_disable="1"
> >>vfs.zfs.arc_min="16M"
> >>vfs.zfs.arc_max="128M"
>
> >I won't pretend to know much about this area, but your ZFS values here
> >are very low. May I assume that they are remnants of the times when the
> >ARC grew insanely large and caused a kernel panic?
> >You're effectively forcing ZFS to not use more than 128MB cache, which
> >doesn't sound like a great idea if you've got 2+ GB of RAM. I've had no
> >trouble without any tuning whatsoever on 2GB for a long time now. The
> >kmem lines can probably be omitted if you're on amd64, too (the default
> >value for kmem_size_max is about 307GB on my machine).
>
> Yes, loader values are one year old when I installed this machine.
> But I think auto tuning was commited after 7.2-RELEASE by Kip Macy,
> so some of them are still needed or am I wrong? (this is
> 7.2-RELEASE). ...

We don't know, because none of the individuals who are maintaining ZFS
at this point in time have actually responded to this question.

http://lists.freebsd.org/pipermail/freebsd-stable/2009-October/052256.html

The community really needs an official answer to this question, and one
from those familiar with the code.

--
| Jeremy Chadwick                                   jdc@... |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |
_______________________________________________
freebsd-stable@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@..."

[Ivan Voras-7]

2009/11/7 Jeremy Chadwick <freebsd@...>:

>> Yes, loader values are one year old when I installed this machine.
>> But I think auto tuning was commited after 7.2-RELEASE by Kip Macy,
>> so some of them are still needed or am I wrong? (this is
>> 7.2-RELEASE). ...
>
> We don't know, because none of the individuals who are maintaining ZFS
> at this point in time have actually responded to this question.
>
> http://lists.freebsd.org/pipermail/freebsd-stable/2009-October/052256.html

Only as a data point and not suggesting anything official: I have
managed to panic 8-STABLE with ZFS kmem exhaustion, so... *shrug* I'm
still scared of using it.
_______________________________________________
freebsd-stable@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@..."

[Miroslav Lachman]

Ivan Voras wrote:

> 2009/11/6 Miroslav Lachman<000.fbsd@...>:
>
>> I do not understand why there are 10MB/s read from disks when network
>> traffic dropped to around 1MB/s (8Mbps)
>>
>> root@cage ~/# iostat -w 20
>>       tty             ad4              ad6             cpu
>>   tin tout  KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
>>    0   14 41.66  53  2.17  41.82  53  2.18   0  0  2  0 97
>>    0   18 50.92  96  4.77  54.82 114  6.12   0  0  3  1 96
>>    0    6 53.52 101  5.29  54.98 108  5.81   1  0  4  1 94
>>    0    6 54.82  98  5.26  55.89 108  5.89   0  0  3  1 96
>
> Yes, this could limit your IO if the requests are random enough.
> Unfortunately I don't know how would you track down what is really
> going on. Maybe some tracing with DTrace?
>
> I'd tell you to use "top -m io" to see if there is a process
> responsible, but apparently these statistics are not updated for ZFS,
> which in itself may be a bug (which is why I'm crossposting to
> freebsd-fs).

DTrace is totally out of my skills ;(

There is otput of top -m io sorted by VCSW displaying JID.

last pid: 17724;  load averages:  0.01,  0.07,  0.08      up 74+20:49:49
  21:03:40
195 processes: 1 running, 193 sleeping, 1 zombie
CPU:  0.0% user,  0.0% nice,  3.6% system,  0.4% interrupt, 96.1% idle
Mem: 462M Active, 2385M Inact, 977M Wired, 21M Cache, 399M Buf, 100M Free
Swap: 6144M Total, 2024K Used, 6142M Free

   PID JID USERNAME   VCSW  IVCSW   READ  WRITE  FAULT  TOTAL PERCENT
COMMAND
17681   8 www         657     64      0      0      0      0   0.00%
lighttpd
17683   8 www         379     41      0      0      0      0   0.00%
lighttpd
17680   8 www         136      5      0      0      0      0   0.00%
lighttpd
17682   8 www          85      0      0      0      0      0   0.00%
lighttpd
  4689   1     90       10      0      0      0      0      0   0.00%
fb_inet_server
  3403   1     90       10      0      0      0      0      0   0.00%
fb_inet_server
  2632   1     90       10      0      0      0      0      0   0.00%
fb_inet_server

All four top consumers is Lighttpd workers.

And as you noted, read, write, fault, total and percent are not updated
on machine with ZFS, so I can't compare it with UFS2 based machine.
Is this bug in top fixed in 8.x? Will you file a PR? (you know more
about FS related things than me :])

Miroslav Lachman
_______________________________________________
freebsd-stable@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@..."