Re: Performance issues with 8.0 ZFS and sendfile/lighttpd

View: New views

4 Messages — Rating Filter: Alert me

	Re: Performance issues with 8.0 ZFS and sendfile/lighttpd by Ivan Voras-7 :: Rate this Message: Reply to Author \| View Threaded \| Show Only this Message 2009/11/6 Miroslav Lachman <000.fbsd@...>: > I do not understand why there are 10MB/s read from disks when network > traffic dropped to around 1MB/s (8Mbps) > > root@cage ~/# iostat -w 20 > tty ad4 ad6 cpu > tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id > 0 14 41.66 53 2.17 41.82 53 2.18 0 0 2 0 97 > 0 18 50.92 96 4.77 54.82 114 6.12 0 0 3 1 96 > 0 6 53.52 101 5.29 54.98 108 5.81 1 0 4 1 94 > 0 6 54.82 98 5.26 55.89 108 5.89 0 0 3 1 96 Yes, this could limit your IO if the requests are random enough. Unfortunately I don't know how would you track down what is really going on. Maybe some tracing with DTrace? I'd tell you to use "top -m io" to see if there is a process responsible, but apparently these statistics are not updated for ZFS, which in itself may be a bug (which is why I'm crossposting to freebsd-fs). _______________________________________________ freebsd-fs@... mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-fs To unsubscribe, send any mail to "freebsd-fs-unsubscribe@..."
	Re: Performance issues with 8.0 ZFS and sendfile/lighttpd by Miroslav Lachman :: Rate this Message: Reply to Author \| View Threaded \| Show Only this Message Ivan Voras wrote: > 2009/11/6 Miroslav Lachman<000.fbsd@...>: > >> I do not understand why there are 10MB/s read from disks when network >> traffic dropped to around 1MB/s (8Mbps) >> >> root@cage ~/# iostat -w 20 >> tty ad4 ad6 cpu >> tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id >> 0 14 41.66 53 2.17 41.82 53 2.18 0 0 2 0 97 >> 0 18 50.92 96 4.77 54.82 114 6.12 0 0 3 1 96 >> 0 6 53.52 101 5.29 54.98 108 5.81 1 0 4 1 94 >> 0 6 54.82 98 5.26 55.89 108 5.89 0 0 3 1 96 > > Yes, this could limit your IO if the requests are random enough. > Unfortunately I don't know how would you track down what is really > going on. Maybe some tracing with DTrace? > > I'd tell you to use "top -m io" to see if there is a process > responsible, but apparently these statistics are not updated for ZFS, > which in itself may be a bug (which is why I'm crossposting to > freebsd-fs). DTrace is totally out of my skills ;( There is otput of top -m io sorted by VCSW displaying JID. last pid: 17724; load averages: 0.01, 0.07, 0.08 up 74+20:49:49 21:03:40 195 processes: 1 running, 193 sleeping, 1 zombie CPU: 0.0% user, 0.0% nice, 3.6% system, 0.4% interrupt, 96.1% idle Mem: 462M Active, 2385M Inact, 977M Wired, 21M Cache, 399M Buf, 100M Free Swap: 6144M Total, 2024K Used, 6142M Free PID JID USERNAME VCSW IVCSW READ WRITE FAULT TOTAL PERCENT COMMAND 17681 8 www 657 64 0 0 0 0 0.00% lighttpd 17683 8 www 379 41 0 0 0 0 0.00% lighttpd 17680 8 www 136 5 0 0 0 0 0.00% lighttpd 17682 8 www 85 0 0 0 0 0 0.00% lighttpd 4689 1 90 10 0 0 0 0 0 0.00% fb_inet_server 3403 1 90 10 0 0 0 0 0 0.00% fb_inet_server 2632 1 90 10 0 0 0 0 0 0.00% fb_inet_server All four top consumers is Lighttpd workers. And as you noted, read, write, fault, total and percent are not updated on machine with ZFS, so I can't compare it with UFS2 based machine. Is this bug in top fixed in 8.x? Will you file a PR? (you know more about FS related things than me :]) Miroslav Lachman _______________________________________________ freebsd-fs@... mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-fs To unsubscribe, send any mail to "freebsd-fs-unsubscribe@..."
	Re: Performance issues with 8.0 ZFS and sendfile/lighttpd by Ivan Voras-7 :: Rate this Message: Reply to Author \| View Threaded \| Show Only this Message 2009/11/7 Miroslav Lachman <000.fbsd@...>: > > And as you noted, read, write, fault, total and percent are not updated on > machine with ZFS, so I can't compare it with UFS2 based machine. > Is this bug in top fixed in 8.x? Will you file a PR? (you know more about FS > related things than me :]) Not much... it depends on from where the stats are collected - there is a fair bit of file system infrastructure that ZFS bypasses and if these stats come from it, they cannot be collected. The stat is apparently updated around sys/kern/vfs_cluster.c: 233 . I'm not very familiar with this layer but since it uses struct buf and the ZFS doesn't use bufcache, this is probably one of the things that is bypassed, though it would be nice if it weren't since this code also defines and uses the vfs.write_behind and vfs.read_max sysctls. Also, since ZFS uses its own threads for IO and the stats are for curthread, it looks like it would maybe need careful work to actually assign the IO stats to the correct thread; otherwise it may be sufficient to add it to vdev_disk.c in vdev_disk_physio(). I don't really know this code and this is mostly mechanical analisys - it might be wrong. At least I'd like to read someone's comment about what is curthread in this code path. _______________________________________________ freebsd-fs@... mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-fs To unsubscribe, send any mail to "freebsd-fs-unsubscribe@..."
	Re: Performance issues with 8.0 ZFS and sendfile/lighttpd by Ollivier Robert :: Rate this Message: Reply to Author \| View Threaded \| Show Only this Message According to Ivan Voras: >I'm not very familiar with this layer but since it uses struct buf and >the ZFS doesn't use bufcache, this is probably one of the things that >is bypassed, though it would be nice if it weren't since this code From what I understand from IRC, Kip is working on fixing ZFS with respect to struct buf (and also help with the ARC vs VM issues). -- Ollivier ROBERT -=- FreeBSD: The Power to Serve! -=- roberto@... In memoriam to Ondine : http://ondine.keltia.net/ _______________________________________________ freebsd-fs@... mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-fs To unsubscribe, send any mail to "freebsd-fs-unsubscribe@..."

	Re: Performance issues with 8.0 ZFS and sendfile/lighttpd by Ivan Voras-7 :: Rate this Message: Reply to Author \| View Threaded \| Show Only this Message 2009/11/6 Miroslav Lachman <000.fbsd@...>: > I do not understand why there are 10MB/s read from disks when network > traffic dropped to around 1MB/s (8Mbps) > > root@cage ~/# iostat -w 20 > tty ad4 ad6 cpu > tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id > 0 14 41.66 53 2.17 41.82 53 2.18 0 0 2 0 97 > 0 18 50.92 96 4.77 54.82 114 6.12 0 0 3 1 96 > 0 6 53.52 101 5.29 54.98 108 5.81 1 0 4 1 94 > 0 6 54.82 98 5.26 55.89 108 5.89 0 0 3 1 96 Yes, this could limit your IO if the requests are random enough. Unfortunately I don't know how would you track down what is really going on. Maybe some tracing with DTrace? I'd tell you to use "top -m io" to see if there is a process responsible, but apparently these statistics are not updated for ZFS, which in itself may be a bug (which is why I'm crossposting to freebsd-fs). _______________________________________________ freebsd-fs@... mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-fs To unsubscribe, send any mail to "freebsd-fs-unsubscribe@..."
	Re: Performance issues with 8.0 ZFS and sendfile/lighttpd by Miroslav Lachman :: Rate this Message: Reply to Author \| View Threaded \| Show Only this Message Ivan Voras wrote: > 2009/11/6 Miroslav Lachman<000.fbsd@...>: > >> I do not understand why there are 10MB/s read from disks when network >> traffic dropped to around 1MB/s (8Mbps) >> >> root@cage ~/# iostat -w 20 >> tty ad4 ad6 cpu >> tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id >> 0 14 41.66 53 2.17 41.82 53 2.18 0 0 2 0 97 >> 0 18 50.92 96 4.77 54.82 114 6.12 0 0 3 1 96 >> 0 6 53.52 101 5.29 54.98 108 5.81 1 0 4 1 94 >> 0 6 54.82 98 5.26 55.89 108 5.89 0 0 3 1 96 > > Yes, this could limit your IO if the requests are random enough. > Unfortunately I don't know how would you track down what is really > going on. Maybe some tracing with DTrace? > > I'd tell you to use "top -m io" to see if there is a process > responsible, but apparently these statistics are not updated for ZFS, > which in itself may be a bug (which is why I'm crossposting to > freebsd-fs). DTrace is totally out of my skills ;( There is otput of top -m io sorted by VCSW displaying JID. last pid: 17724; load averages: 0.01, 0.07, 0.08 up 74+20:49:49 21:03:40 195 processes: 1 running, 193 sleeping, 1 zombie CPU: 0.0% user, 0.0% nice, 3.6% system, 0.4% interrupt, 96.1% idle Mem: 462M Active, 2385M Inact, 977M Wired, 21M Cache, 399M Buf, 100M Free Swap: 6144M Total, 2024K Used, 6142M Free PID JID USERNAME VCSW IVCSW READ WRITE FAULT TOTAL PERCENT COMMAND 17681 8 www 657 64 0 0 0 0 0.00% lighttpd 17683 8 www 379 41 0 0 0 0 0.00% lighttpd 17680 8 www 136 5 0 0 0 0 0.00% lighttpd 17682 8 www 85 0 0 0 0 0 0.00% lighttpd 4689 1 90 10 0 0 0 0 0 0.00% fb_inet_server 3403 1 90 10 0 0 0 0 0 0.00% fb_inet_server 2632 1 90 10 0 0 0 0 0 0.00% fb_inet_server All four top consumers is Lighttpd workers. And as you noted, read, write, fault, total and percent are not updated on machine with ZFS, so I can't compare it with UFS2 based machine. Is this bug in top fixed in 8.x? Will you file a PR? (you know more about FS related things than me :]) Miroslav Lachman _______________________________________________ freebsd-fs@... mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-fs To unsubscribe, send any mail to "freebsd-fs-unsubscribe@..."
	Re: Performance issues with 8.0 ZFS and sendfile/lighttpd by Ivan Voras-7 :: Rate this Message: Reply to Author \| View Threaded \| Show Only this Message 2009/11/7 Miroslav Lachman <000.fbsd@...>: > > And as you noted, read, write, fault, total and percent are not updated on > machine with ZFS, so I can't compare it with UFS2 based machine. > Is this bug in top fixed in 8.x? Will you file a PR? (you know more about FS > related things than me :]) Not much... it depends on from where the stats are collected - there is a fair bit of file system infrastructure that ZFS bypasses and if these stats come from it, they cannot be collected. The stat is apparently updated around sys/kern/vfs_cluster.c: 233 . I'm not very familiar with this layer but since it uses struct buf and the ZFS doesn't use bufcache, this is probably one of the things that is bypassed, though it would be nice if it weren't since this code also defines and uses the vfs.write_behind and vfs.read_max sysctls. Also, since ZFS uses its own threads for IO and the stats are for curthread, it looks like it would maybe need careful work to actually assign the IO stats to the correct thread; otherwise it may be sufficient to add it to vdev_disk.c in vdev_disk_physio(). I don't really know this code and this is mostly mechanical analisys - it might be wrong. At least I'd like to read someone's comment about what is curthread in this code path. _______________________________________________ freebsd-fs@... mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-fs To unsubscribe, send any mail to "freebsd-fs-unsubscribe@..."
	Re: Performance issues with 8.0 ZFS and sendfile/lighttpd by Ollivier Robert :: Rate this Message: Reply to Author \| View Threaded \| Show Only this Message According to Ivan Voras: >I'm not very familiar with this layer but since it uses struct buf and >the ZFS doesn't use bufcache, this is probably one of the things that >is bypassed, though it would be nice if it weren't since this code From what I understand from IRC, Kip is working on fixing ZFS with respect to struct buf (and also help with the ARC vs VM issues). -- Ollivier ROBERT -=- FreeBSD: The Power to Serve! -=- roberto@... In memoriam to Ondine : http://ondine.keltia.net/ _______________________________________________ freebsd-fs@... mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-fs To unsubscribe, send any mail to "freebsd-fs-unsubscribe@..."