Strange (?) hangs with ZFS/rsync.

[Maximillian Dornseif]

I have a ZFS based backup server which rsyncs from a dozen other machines. There is only a single rsync active for any point in time and the system is used for nothing else. It has an amd64 kernel and 4GB RAM. I tried to follow the ZFS tuning guide.

For a few weeks the machine works well. Since monday it rsync always hangs when fetching a big logfile from a remote machine.

The two rsync processes are in state "zfs:&b" and "zfs:lo" and can't be killed.
The other strange thing is that the system is reporting 3367M (!) wired memory.

I have uploaded dmesg output and a some additional information at http://static.23.nu/md/Pictures/freebsd+zfs+issue.txt

Any suggestions on how to debug this issue?

Regards

Maximillian Dornseif

[Maximillian Dornseif]

Maximillian Dornseif wrote:

For a few weeks the machine works well. Since monday it rsync always hangs when fetching a big logfile from a remote machine.

I forget to mention: other processes still work and the ZFS filesystem is still accessible to casual inspection by ls and cat.

The machine does not reboot but hangs during the "syncing" phase of shutdown. It has to be powercycled to restart.

--md

[Maximillian Dornseif]

Maximillian Dornseif wrote:

The machine does not reboot but hangs during the "syncing" phase of shutdown. It has to be powercycled to restart.

after a shutdown -r now the box complains that it couldn't terminate vnlru, bufdaemon and syncer.

[Ivan Voras-7]

Maximillian Dornseif wrote:

> I have a ZFS based backup server which rsyncs from a dozen other machines.
> There is only a single rsync active for any point in time and the system is
> used for nothing else. It has an amd64 kernel and 4GB RAM. I tried to follow
> the ZFS tuning guide.
>
> For a few weeks the machine works well. Since monday it rsync always hangs
> when fetching a big logfile from a remote machine.
>
> The two rsync processes are in state "zfs:&b" and "zfs:lo" and can't be
> killed.
> The other strange thing is that the system is reporting 3367M (!) wired
> memory.

These look like well known problems with ZFS (see
http://wiki.freebsd.org/ZFSKnownProblems). A new version of the ZFS port
  is currently under development, if you want to try it, you need a
8-CURRENT machine and this patch:
http://lists.freebsd.org/pipermail/freebsd-fs/2008-July/004887.html

People have reported that the new patch resolves hangs such as yours.

[George Hartzell-2]

Ivan Voras writes:
 > Maximillian Dornseif wrote:
 > > I have a ZFS based backup server which rsyncs from a dozen other machines.
 > > There is only a single rsync active for any point in time and the system is
 > > used for nothing else. It has an amd64 kernel and 4GB RAM. I tried to follow
 > > the ZFS tuning guide.
 > >
 > > For a few weeks the machine works well. Since monday it rsync always hangs
 > > when fetching a big logfile from a remote machine.
 > >
 > > The two rsync processes are in state "zfs:&b" and "zfs:lo" and can't be
 > > killed.
 > > The other strange thing is that the system is reporting 3367M (!) wired
 > > memory.
 >
 > These look like well known problems with ZFS (see
 > http://wiki.freebsd.org/ZFSKnownProblems). A new version of the ZFS port
 >   is currently under development, if you want to try it, you need a
 > 8-CURRENT machine and this patch:
 > http://lists.freebsd.org/pipermail/freebsd-fs/2008-July/004887.html
 >
 > People have reported that the new patch resolves hangs such as yours.

Not to be a party pooper, but on an arguably underconfigured server
(2GB RAM, dual core AMD, amd64), moving to -CURRENT and applying the
new patch actually required that I tune it with an even lower arc_max
than I had been getting away with on the -STABLE zfs with the same
hardware.

My testing (described on this list a week or two back) is admittedly
unscientific and it may be that the -STABLE configuration was already
skating on thin ice but it's certainly not the case that the subsystem
now autotunes itself on limited hardware.

But, with kernel mem and arc configured as many folks have described,
both versions run nicely in spite of anything I've thrown at them and
the zfs feature set is great.

Thanks again to Pawel and all involved!

g.
_______________________________________________
freebsd-fs@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-unsubscribe@..."

[Maximillian Dornseif]

Ivan Voras-7 wrote:

These look like well known problems with ZFS (see http://wiki.freebsd.org/ZFSKnownProblems).

To be frank, I do not see my problem described there. Rsync from remote mashines does not count as "Heavy IO activity in multithreaded applications", does it?

In the meantime the two ZFS machines both crash whenever I try
zfs delete tank/snapshots@2008

Do I understand it correctly that there is little interest in supporting the 7.x port and most energy is put in hte 8.x port?

--md