8.0-RC2: ZFS deadlock with zfs receive

[Borja Marcos-2]

       
Hello,

I've been sending some alltraces to pjd about this easy to reproduce  
problem. I am using zfs send/zfs receive to replicate a dataset from  
one server to another. At 1 minute intervals, an incremental snapshot  
is sent to update the dataset copy. If there is reading activity on  
the dataset copy, a deadlock can happen rendering ZFS and all the FS  
subsystem unusable. I've tried with 8.0RC2 and it still happens.

How to reproduce:

For example, while doing a make buildworld, replicate the dataset  
containing /usr/obj to the second machine. On the second machine a  
script just runs an endless loop copying all the contents of /usr/obj  
(which, remember, are being copied from the first machine using zfs  
receive) to another location, thus generating quite a lot of read  
activity. The time to deadlock is random, but soon the deadlock appears.

The problem can be triggered by a backup accessing the destination  
dataset or when the security periodic scripts run at night.

I tried to mitigate the problem by cloning a given snapshot at the  
destination and doing the backup from it, but the problem persists.

Any ideas? Any way to mitigate this problem?




Borja.



This is the alltrace after breaking into ddb:

>> PID USERNAME  THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU
>> COMMAND
>> 1914 root        1  62    0 11932K  2564K zfsvfs  0   0:51  0.00%  
>> bsdtar
>> 1093 borjam      1  44    0  8304K  2464K CPU1    1   0:32  0.00% top
>> 1913 root        1  54    0 11932K  2600K rrl->r  0   0:19  0.00%  
>> bsdtar
>> 1019 root        1  44    0 25108K  4812K select  0   0:05  0.00%  
>> sshd
>> 2008 root        1  76    0 13600K  1904K tx->tx  0   0:04  0.00% zfs
>> 1089 borjam      1  44    0 37040K  5216K select  1   0:04  0.00%  
>> sshd
>>  995 root        1  76    0  8252K  2652K pause   0   0:02  0.00% csh
>>  840 root        1  44    0 11044K  3828K select  1   0:02  0.00%

debug.ddb.capture.data: db> alltrace


Tracing command sysctl pid 1373 tid 100183 td 0xffffff00177e5ab0
kdb_enter() at kdb_enter+0x3d
kdb_sysctl_enter() at kdb_sysctl_enter+0x89
sysctl_root() at sysctl_root+0x113
userland_sysctl() at userland_sysctl+0x158
__sysctl() at __sysctl+0xaa
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (202, FreeBSD ELF64, __sysctl), rip = 0x80073d9dc, rsp =  
0x7fffffffe2a8, rbp = 0x7fffffffe37c ---

Tracing command csh pid 1369 tid 100181 td 0xffffff00177e6390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp =  
0x7fffffffe638, rbp = 0x800c59400 ---

Tracing command su pid 1368 tid 100159 td 0xffffff001284aab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3f7
wait4() at wait4+0x35
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x8009e8edc, rsp =  
0x7fffffffe598, rbp = 0x559 ---

Tracing command unlink pid 1363 tid 100162 td 0xffffff001284a000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x79b
ffs_lock() at ffs_lock+0x9c
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
lookup() at lookup+0x5f0
namei() at namei+0x4a9
kern_unlinkat() at kern_unlinkat+0xab
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (10, FreeBSD ELF64, unlink), rip = 0x80072be5c, rsp =  
0x7fffffffec68, rbp = 0 ---

Tracing command sh pid 1361 tid 100100 td 0xffffff000270bab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3f7
wait4() at wait4+0x35
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x800935edc, rsp =  
0x7fffffffe618, rbp = 0x551 ---

Tracing command cron pid 1360 tid 100122 td 0xffffff0002af0390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80095a7ac, rsp =  
0x7fffffffe4b8, rbp = 0x800a9e700 ---

Tracing command unlink pid 1355 tid 100127 td 0xffffff0002aea000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x79b
ffs_lock() at ffs_lock+0x9c
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
lookup() at lookup+0x5f0
namei() at namei+0x4a9
kern_unlinkat() at kern_unlinkat+0xab
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (10, FreeBSD ELF64, unlink), rip = 0x80072be5c, rsp =  
0x7fffffffec68, rbp = 0 ---

Tracing command sh pid 1353 tid 100171 td 0xffffff00177e8ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3f7
wait4() at wait4+0x35
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x800935edc, rsp =  
0x7fffffffe618, rbp = 0x549 ---

Tracing command cron pid 1352 tid 100155 td 0xffffff001284d000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80095a7ac, rsp =  
0x7fffffffe4b8, rbp = 0x800a9e700 ---

Tracing command unlink pid 1347 tid 100157 td 0xffffff001284c720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x79b
ffs_lock() at ffs_lock+0x9c
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
lookup() at lookup+0x5f0
namei() at namei+0x4a9
kern_unlinkat() at kern_unlinkat+0xab
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (10, FreeBSD ELF64, unlink), rip = 0x80072be5c, rsp =  
0x7fffffffec68, rbp = 0 ---

Tracing command sh pid 1345 tid 100108 td 0xffffff0002916ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3f7
wait4() at wait4+0x35
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x800935edc, rsp =  
0x7fffffffe618, rbp = 0x541 ---

Tracing command cron pid 1344 tid 100101 td 0xffffff000270b720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80095a7ac, rsp =  
0x7fffffffe4b8, rbp = 0x800a9e700 ---

Tracing command unlink pid 1337 tid 100172 td 0xffffff00177e8720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x79b
ffs_lock() at ffs_lock+0x9c
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
lookup() at lookup+0x5f0
namei() at namei+0x4a9
kern_unlinkat() at kern_unlinkat+0xab
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (10, FreeBSD ELF64, unlink), rip = 0x80072be5c, rsp =  
0x7fffffffec68, rbp = 0 ---

Tracing command sh pid 1334 tid 100098 td 0xffffff000270e390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3f7
wait4() at wait4+0x35
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x800935edc, rsp =  
0x7fffffffe618, rbp = 0x536 ---

Tracing command newsyslog pid 1333 tid 100065 td 0xffffff00026d6ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
_sx_slock_hard() at _sx_slock_hard+0x1b7
_sx_slock() at _sx_slock+0xc1
zfs_freebsd_reclaim() at zfs_freebsd_reclaim+0x4a
VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0xb5
vgonel() at vgonel+0x119
vnlru_free() at vnlru_free+0x345
getnewvnode() at getnewvnode+0x24f
ffs_vgetf() at ffs_vgetf+0xdf
ufs_lookup_() at ufs_lookup_+0xb46
VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xaf
vfs_cache_lookup() at vfs_cache_lookup+0xf0
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7
lookup() at lookup+0x2eb
namei() at namei+0x4a9
kern_statat_vnhook() at kern_statat_vnhook+0x8f
kern_statat() at kern_statat+0x15
stat() at stat+0x2a
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (188, FreeBSD ELF64, stat), rip = 0x80073251c, rsp =  
0x7fffffffc2b8, rbp = 0x8009075e0 ---

Tracing command cron pid 1331 tid 100147 td 0xffffff0002cb9000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80095a7ac, rsp =  
0x7fffffffe4b8, rbp = 0x800a9e700 ---

Tracing command cron pid 1330 tid 100178 td 0xffffff00177e7000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80095a7ac, rsp =  
0x7fffffffe4b8, rbp = 0x800a9e700 ---

Tracing command unlink pid 1329 tid 100177 td 0xffffff00177e7390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x79b
ffs_lock() at ffs_lock+0x9c
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
lookup() at lookup+0x5f0
namei() at namei+0x4a9
kern_unlinkat() at kern_unlinkat+0xab
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (10, FreeBSD ELF64, unlink), rip = 0x80072be5c, rsp =  
0x7fffffffec68, rbp = 0 ---

Tracing command sh pid 1326 tid 100176 td 0xffffff00177e7720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3f7
wait4() at wait4+0x35
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x800935edc, rsp =  
0x7fffffffe618, rbp = 0x52e ---

Tracing command cron pid 1324 tid 100115 td 0xffffff0002915000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80095a7ac, rsp =  
0x7fffffffe4b8, rbp = 0x800a9e700 ---

Tracing command unlink pid 1319 tid 100059 td 0xffffff000256a720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x79b
ffs_lock() at ffs_lock+0x9c
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
lookup() at lookup+0x5f0
namei() at namei+0x4a9
kern_unlinkat() at kern_unlinkat+0xab
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (10, FreeBSD ELF64, unlink), rip = 0x80072be5c, rsp =  
0x7fffffffec68, rbp = 0 ---

Tracing command sh pid 1317 tid 100058 td 0xffffff0002650000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3f7
wait4() at wait4+0x35
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x800935edc, rsp =  
0x7fffffffe618, rbp = 0x525 ---

Tracing command cron pid 1316 tid 100055 td 0xffffff0002650ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80095a7ac, rsp =  
0x7fffffffe4b8, rbp = 0x800a9e700 ---

Tracing command unlink pid 1309 tid 100112 td 0xffffff0002915ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x79b
ffs_lock() at ffs_lock+0x9c
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
lookup() at lookup+0x5f0
namei() at namei+0x4a9
kern_unlinkat() at kern_unlinkat+0xab
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (10, FreeBSD ELF64, unlink), rip = 0x80072be5c, rsp =  
0x7fffffffec68, rbp = 0 ---

Tracing command sh pid 1307 tid 100167 td 0xffffff0002cb8000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3f7
wait4() at wait4+0x35
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x800935edc, rsp =  
0x7fffffffe618, rbp = 0x51b ---

Tracing command cron pid 1306 tid 100166 td 0xffffff0002af0720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80095a7ac, rsp =  
0x7fffffffe4b8, rbp = 0x800a9e700 ---

Tracing command unlink pid 1301 tid 100164 td 0xffffff00026e7ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x79b
ffs_lock() at ffs_lock+0x9c
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
lookup() at lookup+0x5f0
namei() at namei+0x4a9
kern_unlinkat() at kern_unlinkat+0xab
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (10, FreeBSD ELF64, unlink), rip = 0x80072be5c, rsp =  
0x7fffffffec68, rbp = 0 ---

Tracing command sh pid 1299 tid 100103 td 0xffffff000270a000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3f7
wait4() at wait4+0x35
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x800935edc, rsp =  
0x7fffffffe618, rbp = 0x513 ---

Tracing command cron pid 1298 tid 100160 td 0xffffff001284a720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80095a7ac, rsp =  
0x7fffffffe4b8, rbp = 0x800a9e700 ---

Tracing command sh pid 1289 tid 100161 td 0xffffff001284a390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
_sx_slock_hard() at _sx_slock_hard+0x1b7
_sx_slock() at _sx_slock+0xc1
zfs_freebsd_reclaim() at zfs_freebsd_reclaim+0x4a
VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0xb5
vgonel() at vgonel+0x119
vnlru_free() at vnlru_free+0x345
getnewvnode() at getnewvnode+0x24f
ffs_vgetf() at ffs_vgetf+0xdf
ufs_lookup_() at ufs_lookup_+0xb46
VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xaf
vfs_cache_lookup() at vfs_cache_lookup+0xf0
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7
lookup() at lookup+0x2eb
namei() at namei+0x4a9
kern_statat_vnhook() at kern_statat_vnhook+0x8f
kern_statat() at kern_statat+0x15
stat() at stat+0x2a
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (188, FreeBSD ELF64, stat), rip = 0x8009ae51c, rsp =  
0x7fffffffe798, rbp = 0x800c06b80 ---

Tracing command cron pid 1288 tid 100063 td 0xffffff000264f720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80095a7ac, rsp =  
0x7fffffffe4b8, rbp = 0x800a9e700 ---

Tracing command zfs pid 1285 tid 100143 td 0xffffff0002cbb000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
_cv_wait() at _cv_wait+0x17a
txg_wait_synced() at txg_wait_synced+0x7c
zfsvfs_teardown() at zfsvfs_teardown+0x1b2
zfs_suspend_fs() at zfs_suspend_fs+0x2b
zfs_ioc_recv() at zfs_ioc_recv+0x28b
zfsdev_ioctl() at zfsdev_ioctl+0x8d
devfs_ioctl_f() at devfs_ioctl_f+0x76
kern_ioctl() at kern_ioctl+0xc5
ioctl() at ioctl+0xfd
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (54, FreeBSD ELF64, ioctl), rip = 0x800fe874c, rsp =  
0x7fffffffbbf8, rbp = 0x7fffffffc930 ---

Tracing command csh pid 1283 tid 100163 td 0xffffff0009979ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp =  
0x7fffffffe598, rbp = 0x800c1d300 ---

Tracing command sshd pid 1281 tid 100124 td 0xffffff0002aeaab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
seltdwait() at seltdwait+0xac
kern_select() at kern_select+0x620
select() at select+0x5d
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x8013d772c, rsp =  
0x7fffffffdcb8, rbp = 0x7fffffffdd40 ---

Tracing command zpool pid 1256 tid 100109 td 0xffffff0002916720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_sleep() at _sleep+0x23f
kern_nanosleep() at kern_nanosleep+0x118
nanosleep() at nanosleep+0x6e
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x8010ce92c, rsp =  
0x7fffffffaac8, rbp = 0 ---

Tracing command csh pid 1252 tid 100158 td 0xffffff001284c000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp =  
0x7fffffffe638, rbp = 0x800c59400 ---

Tracing command su pid 1248 tid 100156 td 0xffffff001284cab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3f7
wait4() at wait4+0x35
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x8009e8edc, rsp =  
0x7fffffffe598, rbp = 0x4e4 ---

Tracing command csh pid 1243 tid 100137 td 0xffffff0002917720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp =  
0x7fffffffe558, rbp = 0x800c1f700 ---

Tracing command sshd pid 1242 tid 100154 td 0xffffff001284d390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
seltdwait() at seltdwait+0xac
kern_select() at kern_select+0x620
select() at select+0x5d
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x8013d772c, rsp =  
0x7fffffffdcb8, rbp = 0x7fffffffdd40 ---

Tracing command sshd pid 1234 tid 100060 td 0xffffff000264fab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
soreceive_generic() at soreceive_generic+0xf99
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x8013d77ac, rsp =  
0x7fffffffdcd8, rbp = 0 ---

Tracing command bsdtar pid 1203 tid 100153 td 0xffffff000270aab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x800e447ac, rsp =  
0x7fffffffe7a8, rbp = 0x80101a240 ---

Tracing command bsdtar pid 1202 tid 100057 td 0xffffff0002650390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x79b
vop_stdlock() at vop_stdlock+0x39
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
vacl_get_acl() at vacl_get_acl+0x55
__acl_get_file() at __acl_get_file+0xcf
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (347, FreeBSD ELF64, __acl_get_file), rip = 0x800dbde0c,  
rsp = 0x7fffffffe1d8, rbp = 0x2 ---

Tracing command csh pid 1200 tid 100152 td 0xffffff000270a720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp =  
0x7fffffffe3a8, rbp = 0x800c43200 ---

Tracing command bsdtar pid 1199 tid 100151 td 0xffffff0002af0ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x800e447ac, rsp =  
0x7fffffffe7a8, rbp = 0x80101a240 ---

Tracing command bsdtar pid 1198 tid 100105 td 0xffffff000264c000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
_cv_wait() at _cv_wait+0x17a
rrw_enter() at rrw_enter+0xb1
zfs_freebsd_read() at zfs_freebsd_read+0x55
VOP_READ_APV() at VOP_READ_APV+0xaf
vn_read() at vn_read+0x254
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x800e447ac, rsp =  
0x7fffffffe898, rbp = 0x1a87 ---

Tracing command csh pid 1196 tid 100056 td 0xffffff0002650720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp =  
0x7fffffffe3a8, rbp = 0x800c43200 ---

Tracing command bsdtar pid 1195 tid 100067 td 0xffffff00026d6720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
_sx_slock_hard() at _sx_slock_hard+0x1b7
_sx_slock() at _sx_slock+0xc1
zfs_freebsd_reclaim() at zfs_freebsd_reclaim+0x4a
VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0xb5
vgonel() at vgonel+0x119
vnlru_free() at vnlru_free+0x345
getnewvnode() at getnewvnode+0x24f
zfs_znode_cache_constructor() at zfs_znode_cache_constructor+0x2c
zfs_znode_alloc() at zfs_znode_alloc+0x39
zfs_mknode() at zfs_mknode+0x205
zfs_freebsd_create() at zfs_freebsd_create+0x583
VOP_CREATE_APV() at VOP_CREATE_APV+0xb3
vn_open_cred() at vn_open_cred+0x473
kern_openat() at kern_openat+0x179
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (5, FreeBSD ELF64, open), rip = 0x800e3557c, rsp =  
0x7fffffffe758, rbp = 0x1a4 ---

Tracing command bsdtar pid 1194 tid 100150 td 0xffffff0002cb8390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x79b
vop_stdlock() at vop_stdlock+0x39
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
vacl_get_acl() at vacl_get_acl+0x55
__acl_get_file() at __acl_get_file+0xcf
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (347, FreeBSD ELF64, __acl_get_file), rip = 0x800dbde0c,  
rsp = 0x7fffffffe1d8, rbp = 0x2 ---

Tracing command csh pid 1192 tid 100138 td 0xffffff0002cbc390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp =  
0x7fffffffe3a8, rbp = 0x800c43200 ---

Tracing command bsdtar pid 1191 tid 100113 td 0xffffff0002915720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x800e447ac, rsp =  
0x7fffffffe7a8, rbp = 0x80101a240 ---

Tracing command bsdtar pid 1190 tid 100118 td 0xffffff0002914390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x79b
vop_stdlock() at vop_stdlock+0x39
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
vacl_get_acl() at vacl_get_acl+0x55
__acl_get_file() at __acl_get_file+0xcf
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (347, FreeBSD ELF64, __acl_get_file), rip = 0x800dbde0c,  
rsp = 0x7fffffffe1d8, rbp = 0x2 ---

Tracing command csh pid 1188 tid 100144 td 0xffffff0002cb9ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp =  
0x7fffffffe3a8, rbp = 0x800c43200 ---

Tracing command bsdtar pid 1187 tid 100139 td 0xffffff0002cbc000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x800e447ac, rsp =  
0x7fffffffe7a8, rbp = 0x80101a240 ---

Tracing command bsdtar pid 1186 tid 100123 td 0xffffff0002af0000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x79b
vop_stdlock() at vop_stdlock+0x39
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
vacl_get_acl() at vacl_get_acl+0x55
__acl_get_file() at __acl_get_file+0xcf
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (347, FreeBSD ELF64, __acl_get_file), rip = 0x800dbde0c,  
rsp = 0x7fffffffe1d8, rbp = 0x2 ---

Tracing command csh pid 1184 tid 100146 td 0xffffff0002cb9390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp =  
0x7fffffffe3a8, rbp = 0x800c43200 ---

Tracing command bsdtar pid 1183 tid 100126 td 0xffffff0002aea390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x7b2
vop_stdlock() at vop_stdlock+0x39
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
vget() at vget+0x7b
cache_lookup() at cache_lookup+0x4e0
vfs_cache_lookup() at vfs_cache_lookup+0xc0
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7
lookup() at lookup+0x2eb
namei() at namei+0x4a9
kern_statat_vnhook() at kern_statat_vnhook+0x8f
kern_statat() at kern_statat+0x15
lstat() at lstat+0x2a
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (190, FreeBSD ELF64, lstat), rip = 0x800e354fc, rsp =  
0x7fffffffe778, rbp = 0x801023380 ---

Tracing command bsdtar pid 1182 tid 100066 td 0xffffff000264f000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x79b
vop_stdlock() at vop_stdlock+0x39
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
vacl_get_acl() at vacl_get_acl+0x55
__acl_get_file() at __acl_get_file+0xcf
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (347, FreeBSD ELF64, __acl_get_file), rip = 0x800dbde0c,  
rsp = 0x7fffffffe1d8, rbp = 0x2 ---

Tracing command csh pid 1180 tid 100120 td 0xffffff0002714ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp =  
0x7fffffffe3a8, rbp = 0x800c43200 ---

Tracing command csh pid 1147 tid 100142 td 0xffffff0002cbb390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
tty_wait() at tty_wait+0x48
ttydisc_read() at ttydisc_read+0x2f1
ttydev_read() at ttydev_read+0xab
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x8009ec7ac, rsp =  
0x7fffffffe658, rbp = 0x1 ---

Tracing command sshd pid 1146 tid 100111 td 0xffffff0002916000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
seltdwait() at seltdwait+0xac
kern_select() at kern_select+0x620
select() at select+0x5d
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x8013d772c, rsp =  
0x7fffffffdcb8, rbp = 0x7fffffffdd40 ---

Tracing command sshd pid 1133 tid 100116 td 0xffffff0002914ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
soreceive_generic() at soreceive_generic+0xf99
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x8013d77ac, rsp =  
0x7fffffffdcd8, rbp = 0 ---

Tracing command csh pid 959 tid 100149 td 0xffffff0002cb8720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
tty_wait() at tty_wait+0x48
ttydisc_read() at ttydisc_read+0x2f1
ttydev_read() at ttydev_read+0xab
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x8009ec7ac, rsp =  
0x7fffffffe738, rbp = 0x1 ---

Tracing command su pid 958 tid 100148 td 0xffffff0002cb8ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3f7
wait4() at wait4+0x35
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x8009e8edc, rsp =  
0x7fffffffe598, rbp = 0x3bf ---

Tracing command csh pid 952 tid 100125 td 0xffffff0002aea720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp =  
0x7fffffffe558, rbp = 0x800c1f500 ---

Tracing command sshd pid 951 tid 100130 td 0xffffff0002ae8390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
seltdwait() at seltdwait+0xac
kern_select() at kern_select+0x620
select() at select+0x5d
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x8013d772c, rsp =  
0x7fffffffdcb8, rbp = 0x7fffffffdd40 ---

Tracing command csh pid 946 tid 100141 td 0xffffff0002cbb720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp =  
0x7fffffffe558, rbp = 0x800c1f500 ---

Tracing command sshd pid 945 tid 100140 td 0xffffff0002cbbab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
seltdwait() at seltdwait+0xac
kern_select() at kern_select+0x620
select() at select+0x5d
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x8013d772c, rsp =  
0x7fffffffdcb8, rbp = 0x7fffffffdd40 ---

Tracing command sshd pid 942 tid 100128 td 0xffffff0002ae8ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
soreceive_generic() at soreceive_generic+0xf99
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x8013d77ac, rsp =  
0x7fffffffdcd8, rbp = 0 ---

Tracing command sshd pid 939 tid 100132 td 0xffffff0002ae7ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
soreceive_generic() at soreceive_generic+0xf99
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x8013d77ac, rsp =  
0x7fffffffdcd8, rbp = 0 ---

Tracing command getty pid 908 tid 100117 td 0xffffff0002914720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
tty_wait() at tty_wait+0x48
ttydisc_read() at ttydisc_read+0x2f1
ttydev_read() at ttydev_read+0xab
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80084f7ac, rsp =  
0x7fffffffecc8, rbp = 0 ---

Tracing command getty pid 907 tid 100129 td 0xffffff0002ae8720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
tty_wait() at tty_wait+0x48
ttydisc_read() at ttydisc_read+0x2f1
ttydev_read() at ttydev_read+0xab
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80084f7ac, rsp =  
0x7fffffffecc8, rbp = 0 ---

Tracing command getty pid 906 tid 100106 td 0xffffff0002917390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
tty_wait() at tty_wait+0x48
ttydisc_read() at ttydisc_read+0x2f1
ttydev_read() at ttydev_read+0xab
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80084f7ac, rsp =  
0x7fffffffecc8, rbp = 0 ---

Tracing command getty pid 905 tid 100121 td 0xffffff0002714720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
tty_wait() at tty_wait+0x48
ttydisc_read() at ttydisc_read+0x2f1
ttydev_read() at ttydev_read+0xab
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80084f7ac, rsp =  
0x7fffffffecc8, rbp = 0 ---

Tracing command getty pid 904 tid 100136 td 0xffffff0002917ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
tty_wait() at tty_wait+0x48
ttydisc_read() at ttydisc_read+0x2f1
ttydev_read() at ttydev_read+0xab
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80084f7ac, rsp =  
0x7fffffffecc8, rbp = 0 ---

Tracing command getty pid 903 tid 100135 td 0xffffff0002ae7000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
tty_wait() at tty_wait+0x48
ttydisc_read() at ttydisc_read+0x2f1
ttydev_read() at ttydev_read+0xab
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80084f7ac, rsp =  
0x7fffffffecc8, rbp = 0 ---

Tracing command getty pid 902 tid 100134 td 0xffffff0002ae7390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
tty_wait() at tty_wait+0x48
ttydisc_read() at ttydisc_read+0x2f1
ttydev_read() at ttydev_read+0xab
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80084f7ac, rsp =  
0x7fffffffecc8, rbp = 0 ---

Tracing command getty pid 901 tid 100133 td 0xffffff0002ae7720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
tty_wait() at tty_wait+0x48
ttydisc_read() at ttydisc_read+0x2f1
ttydev_read() at ttydev_read+0xab
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80084f7ac, rsp =  
0x7fffffffecc8, rbp = 0 ---

Tracing command cron pid 853 tid 100119 td 0xffffff0002914000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_sleep() at _sleep+0x23f
kern_nanosleep() at kern_nanosleep+0x118
nanosleep() at nanosleep+0x6e
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80093d92c, rsp =  
0x7fffffffeb28, rbp = 0x3c ---

Tracing command sendmail pid 847 tid 100061 td 0xffffff000256a390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x800d3975c, rsp =  
0x7fffffffcd68, rbp = 0x1 ---

Tracing command sendmail pid 841 tid 100107 td 0xffffff0002917000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_cv_timedwait_sig() at _cv_timedwait_sig+0x18c
seltdwait() at seltdwait+0x56
kern_select() at kern_select+0x620
select() at select+0x5d
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x800dd772c, rsp =  
0x7fffffffc1e8, rbp = 0x7fffffffc280 ---

Tracing command sshd pid 834 tid 100110 td 0xffffff0002916390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
seltdwait() at seltdwait+0xac
kern_select() at kern_select+0x620
select() at select+0x5d
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x8013d772c, rsp =  
0x7fffffffddd8, rbp = 0x2 ---

Tracing command syslogd pid 610 tid 100064 td 0xffffff000264f390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
seltdwait() at seltdwait+0xac
kern_select() at kern_select+0x620
select() at select+0x5d
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x80085272c, rsp =  
0x7fffffffdcb8, rbp = 0x1 ---

Tracing command devd pid 503 tid 100114 td 0xffffff0002915390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
seltdwait() at seltdwait+0xac
kern_select() at kern_select+0x620
select() at select+0x5d
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x440cfc, rsp =  
0x7fffffffe898, rbp = 0x7fffffffe8b0 ---

Tracing command flowcleaner pid 22 tid 100054 td 0xffffff000256aab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_timedwait() at sleepq_timedwait+0x4d
_cv_timedwait() at _cv_timedwait+0x18c
flowtable_
debug.ddb.capture.bufsize: 49152
debug.ddb.capture.inprogress: 0
debug.ddb.capture.maxbufsize: 5242880
debug.ddb.capture.bufoff: 49152
debug.ddb.scripting.unscript:
debug.ddb.scripting.scripts:
debug.ddb.textdump.do_version: 1
debug.ddb.textdump.do_panic: 1
debug.ddb.textdump.do_msgbuf: 1
debug.ddb.textdump.do_ddb: 1
debug.ddb.textdump.pending: 0

_______________________________________________
freebsd-fs@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-unsubscribe@..."

[Stef Walter]

Borja Marcos wrote:
> I've been sending some alltraces to pjd about this easy to reproduce
> problem. I am using zfs send/zfs receive to replicate a dataset from one
> server to another. At 1 minute intervals, an incremental snapshot is
> sent to update the dataset copy. If there is reading activity on the
> dataset copy, a deadlock can happen rendering ZFS and all the FS
> subsystem unusable. I've tried with 8.0RC2 and it still happens.

FWIW, another (or the same) zfs recv deadlock I've been trying to get to
the bottom of:

http://lists.freebsd.org/pipermail/freebsd-fs/2009-October/006999.html

Cheers,

Stef

_______________________________________________
freebsd-fs@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-unsubscribe@..."

[pjd-2]

On Thu, Oct 29, 2009 at 03:52:45PM -0600, Stef Walter wrote:
> Borja Marcos wrote:
> > I've been sending some alltraces to pjd about this easy to reproduce
> > problem. I am using zfs send/zfs receive to replicate a dataset from one
> > server to another. At 1 minute intervals, an incremental snapshot is
> > sent to update the dataset copy. If there is reading activity on the
> > dataset copy, a deadlock can happen rendering ZFS and all the FS
> > subsystem unusable. I've tried with 8.0RC2 and it still happens.

I was able to reproduce it, but I don't have fix yet.

> FWIW, another (or the same) zfs recv deadlock I've been trying to get to
> the bottom of:
>
> http://lists.freebsd.org/pipermail/freebsd-fs/2009-October/006999.html

Could you guys recompile your kernel after uncommenting line:

#CFLAGS+=-DDEBUG=1

in sys/modules/zfs/Makefile?

You should see panic on assertion instead of deadlock, I believe.

--
Pawel Jakub Dawidek                       http://www.wheel.pl
pjd@...                           http://www.FreeBSD.org
FreeBSD committer                         Am I Evil? Yes, I Am!

[Stef Walter]

Pawel Jakub Dawidek wrote:

> On Thu, Oct 29, 2009 at 03:52:45PM -0600, Stef Walter wrote:
>> Borja Marcos wrote:
>>> I've been sending some alltraces to pjd about this easy to reproduce
>>> problem. I am using zfs send/zfs receive to replicate a dataset from one
>>> server to another. At 1 minute intervals, an incremental snapshot is
>>> sent to update the dataset copy. If there is reading activity on the
>>> dataset copy, a deadlock can happen rendering ZFS and all the FS
>>> subsystem unusable. I've tried with 8.0RC2 and it still happens.
>
> I was able to reproduce it, but I don't have fix yet.

Thanks for taking your time to look at it. Super appreciated.

>> FWIW, another (or the same) zfs recv deadlock I've been trying to get to
>> the bottom of:
>>
>> http://lists.freebsd.org/pipermail/freebsd-fs/2009-October/006999.html
>
> Could you guys recompile your kernel after uncommenting line:
>
> #CFLAGS+=-DDEBUG=1
>
> in sys/modules/zfs/Makefile?
>
> You should see panic on assertion instead of deadlock, I believe.

Rebuilt with that change... You're right, now I get this assertion:

login: panic: solaris assert: list_head(&osi->os_dnodes) ==
osi->os_meta_dnode (0xc9060488 == 0xca922000), file:
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c,
line: 459

Full details and backtraces attached.

Thanks again,

Stef

http://lists.freebsd.org/pipermail/freebsd-fs/2009-October/006999.html

----------------------------------------------------------------------
 The Assertion

login: panic: solaris assert: list_head(&osi->os_dnodes) == osi->os_meta_dnode (0xc9060488 == 0xca922000), file: /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c, line: 459
cpuid = 0
KDB: enter: panic
[thread pid 1552 tid 100121 ]
Stopped at      kdb_enter+0x3a: movl    $0,kdb_why


db> bt
Tracing pid 1552 tid 100121 td 0xc658f480
kdb_enter(c0c79511,c0c79511,c629f3cf,e81794d8,0,...) at kdb_enter+0x3a
panic(c629f3cf,c62a3486,c9060488,0,c629f660,...) at panic+0x136
dmu_objset_evict(c7346200,c72b0200,1cb,0,c629cc75,...) at dmu_objset_evict+0x299
dsl_dataset_destroy(c7346200,c62a2ebd,16,e8179984,e8179944,...) at dsl_dataset_destroy+0x24c
dmu_recv_abort_cleanup(e8179984,c62ad24a,1cb,c629c778,e817992c,...) at dmu_recv_abort_cleanup+0x2f
dmu_recv_stream(e8179984,c5ed4690,e817999c,0,0,...) at dmu_recv_stream+0xa0e
zfs_ioc_recv(caab8000,0,0,0,0,...) at zfs_ioc_recv+0x2d4
zfsdev_ioctl(c5f67300,cc285a1c,caab8000,3,c658f480,...) at zfsdev_ioctl+0x7d
devfs_ioctl_f(c5ed4dc8,cc285a1c,caab8000,c981c280,c658f480,...) at devfs_ioctl_f+0xf8
kern_ioctl(c658f480,3,cc285a1c,caab8000,1000000,...) at kern_ioctl+0x1fd
ioctl(c658f480,e8179cf8,c,c0c916cc,c0d5ff48,...) at ioctl+0x134
syscall(e8179d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (54, FreeBSD ELF32, ioctl), eip = 0x282211f3, esp = 0xbfbfbcfc, ebp = 0xbfbfbd28 ---


db> show pcpu
cpuid        = 0
dynamic pcpu    = 0x68ac00
curthread    = 0xc658f480: pid 1552 "zfs"
curpcb       = 0xe8179d90
fpcurthread  = none
idlethread   = 0xc596fb40: pid 11 "idle: cpu0"
APIC ID      = 0
currentldt   = 0x50
spin locks held:


db> show allpcpu
Current CPU: 0

cpuid        = 0
dynamic pcpu    = 0x68ac00
curthread    = 0xc658f480: pid 1552 "zfs"
curpcb       = 0xe8179d90
fpcurthread  = none
idlethread   = 0xc596fb40: pid 11 "idle: cpu0"
APIC ID      = 0
currentldt   = 0x50
spin locks held:


db> show locks
exclusive sx ds->ds_rwlock (ds->ds_rwlock) r = 0 (0xc7346294) locked @ /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c:745
shared sx ds->ds_rwlock (ds->ds_rwlock) r = 0 (0xc7a29494) locked @ /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c:519
exclusive sx zfsvfs->z_online_recv_lock (zfsvfs->z_online_recv_lock) r = 0 (0xd2940198) locked @ /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:2484


db> show alllocks
Process 1552 (zfs) thread 0xc658f480 (100121)
exclusive sx ds->ds_rwlock (ds->ds_rwlock) r = 0 (0xc7346294) locked @ /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c:745
shared sx ds->ds_rwlock (ds->ds_rwlock) r = 0 (0xc7a29494) locked @ /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c:519
exclusive sx zfsvfs->z_online_recv_lock (zfsvfs->z_online_recv_lock) r = 0 (0xd2940198) locked @ /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:2484


db> show lockedvnods
Locked vnodes


db> alltrace

Tracing command zfs pid 1552 tid 100121 td 0xc658f480
kdb_enter(c0c79511,c0c79511,c629f3cf,e81794d8,0,...) at kdb_enter+0x3a
panic(c629f3cf,c62a3486,c9060488,0,c629f660,...) at panic+0x136
dmu_objset_evict(c7346200,c72b0200,1cb,0,c629cc75,...) at dmu_objset_evict+0x299
dsl_dataset_destroy(c7346200,c62a2ebd,16,e8179984,e8179944,...) at dsl_dataset_destroy+0x24c
dmu_recv_abort_cleanup(e8179984,c62ad24a,1cb,c629c778,e817992c,...) at dmu_recv_abort_cleanup+0x2f
dmu_recv_stream(e8179984,c5ed4690,e817999c,0,0,...) at dmu_recv_stream+0xa0e
zfs_ioc_recv(caab8000,0,0,0,0,...) at zfs_ioc_recv+0x2d4
zfsdev_ioctl(c5f67300,cc285a1c,caab8000,3,c658f480,...) at zfsdev_ioctl+0x7d
devfs_ioctl_f(c5ed4dc8,cc285a1c,caab8000,c981c280,c658f480,...) at devfs_ioctl_f+0xf8
kern_ioctl(c658f480,3,cc285a1c,caab8000,1000000,...) at kern_ioctl+0x1fd
ioctl(c658f480,e8179cf8,c,c0c916cc,c0d5ff48,...) at ioctl+0x134
syscall(e8179d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (54, FreeBSD ELF32, ioctl), eip = 0x282211f3, esp = 0xbfbfbcfc, ebp = 0xbfbfbd28 ---

Tracing command zfskern pid 1527 tid 100117 td 0xc61226c0
sched_switch(c61226c0,0,104,191,b9d041d4,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c61226c0,0,c0c7d8d5,26e,c61226c0,...) at sleepq_switch+0x15f
sleepq_timedwait(c669addc,0,e8166c08,1,0,...) at sleepq_timedwait+0x6b
_cv_timedwait(c669addc,c669ad84,7530,1,c669addc,...) at _cv_timedwait+0x250
txg_thread_wait(7530,0,3b9aca00,0,202c7066,...) at txg_thread_wait+0xab
txg_sync_thread(c669ac00,e8166d38,c0c749df,343,c611e000,...) at txg_sync_thread+0x480
fork_exit(c61fd6c0,c669ac00,e8166d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8166d70, ebp = 0 ---

Tracing command zfskern pid 1527 tid 100116 td 0xc6122900
sched_switch(c6122900,0,104,191,b9d06478,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6122900,0,c0c7d8d5,24b,c6122900,...) at sleepq_switch+0x15f
sleepq_wait(c669adec,0,e8162c38,1,0,...) at sleepq_wait+0x63
_cv_wait(c669adec,c669ad84,c669adf4,1,c669adec,...) at _cv_wait+0x240
txg_thread_wait(0,0,c62ad24a,18d,c08c441b,...) at txg_thread_wait+0x4f
txg_quiesce_thread(c669ac00,e8162d38,c0c749df,343,c611e000,...) at txg_quiesce_thread+0xa0
fork_exit(c61fce20,c669ac00,e8162d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8162d70, ebp = 0 ---
       
Tracing command zfskern pid 1527 tid 100114 td 0xc6122d80
sched_switch(c6122d80,0,104,191,b9c5a414,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,4c,...) at mi_switch+0x200
sleepq_switch(c6122d80,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c65882c8,4c,c62b6b3a,0,0,...) at sleepq_wait+0x63
_sleep(c65882c8,c65882dc,24c,c62b6b3a,0,...) at _sleep+0x36b
vdev_geom_worker(c65882c0,e8156d38,c0c749df,343,c611e000,...) at vdev_geom_worker+0x124
fork_exit(c6251040,c65882c0,e8156d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8156d70, ebp = 0 ---

Tracing command zfskern pid 1527 tid 100087 td 0xc6120b40
sched_switch(c6120b40,0,104,191,7710eb92,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6120b40,0,c0c7d8d5,26e,c6120b40,...) at sleepq_switch+0x15f
sleepq_timedwait(c62c2ff4,0,e810bc24,1,0,...) at sleepq_timedwait+0x6b
_cv_timedwait(c62c2ff4,c62c2fe0,3e8,1131,0,...) at _cv_timedwait+0x250
l2arc_feed_thread(0,e810bd38,c0c749df,343,c611e000,...) at l2arc_feed_thread+0x149
fork_exit(c61a4fa0,0,e810bd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe810bd70, ebp = 0 ---

Tracing command zfskern pid 1527 tid 100078 td 0xc6122000
sched_switch(c6122000,0,104,191,7711309a,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6122000,0,c0c7d8d5,26e,c6122000,...) at sleepq_switch+0x15f
sleepq_timedwait(c62babcc,0,c57eac80,1,0,...) at sleepq_timedwait+0x6b
_cv_timedwait(c62babcc,c62babb8,3e8,0,3e8,...) at _cv_timedwait+0x250
arc_reclaim_thread(0,c57ead38,c0c749df,343,c611e000,...) at arc_reclaim_thread+0x3b7
fork_exit(c61a8ac0,0,c57ead38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc57ead70, ebp = 0 ---

Tracing command bash pid 1525 tid 100076 td 0xc5ede240
sched_switch(c5ede240,0,104,191,788fbd72,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ede240,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c08738da,c5f2ae04,0,c0c77c84,c5ede240,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c5f2ae70,0,c57e4b0c,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c5f2ae70,c5f2ae04,c0c81880,4d5,0,...) at _cv_wait_sig+0x240
tty_wait(c5f2ae00,c5f2ae70,c57e4c58,1,0,...) at tty_wait+0x71
ttydisc_read(c5f2ae00,c57e4c58,0,9e,0,...) at ttydisc_read+0x22c
ttydev_read(c6137000,c57e4c58,0,0,1,...) at ttydev_read+0xaa
devfs_read_f(c5ed4738,c57e4c58,c5f40300,0,c5ede240,...) at devfs_read_f+0x7e
dofileread(c57e4c58,ffffffff,ffffffff,0,c5ed4738,...) at dofileread+0x96
kern_readv(c5ede240,0,c57e4c58,c57e4c78,1,...) at kern_readv+0x58
read(c5ede240,c57e4cf8,c,c5ede240,c0d5f9b4,...) at read+0x4f
syscall(c57e4d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x8127b93, esp = 0xbfbfe03c, ebp = 0xbfbfe068 ---

Tracing command sshd pid 1521 tid 100079 td 0xc6121d80
sched_switch(c6121d80,0,104,191,788df0ca,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6121d80,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c08738da,c5e58710,0,c0c77c84,c6121d80,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c5e58724,0,c57eda7c,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c5e58724,c5e58710,c0c7fb95,5d1,c5ed4770,...) at _cv_wait_sig+0x240
seltdwait(c5ed4770,58,c60ff600,c6121d80,200246,...) at seltdwait+0xa2
kern_select(c6121d80,9,286030c8,286030cc,0,0,20,65,280fdd50) at kern_select+0x4f4
select(c6121d80,c57edcf8,14,c0c61635,c0d6038c,...) at select+0x66
syscall(c57edd38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x283cc1d3, esp = 0xbfbfde5c, ebp = 0xbfbfdea8 ---

Tracing command getty pid 1518 tid 100071 td 0xc5eded80
sched_switch(c5eded80,0,104,191,d92742d4,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5eded80,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c08738da,c5c7f404,0,c0c77c84,c5eded80,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c5c7f470,0,c57c0b0c,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c5c7f470,c5c7f404,c0c81880,4d5,0,...) at _cv_wait_sig+0x240
tty_wait(c5c7f400,c5c7f470,c57c0c58,1,0,...) at tty_wait+0x71
ttydisc_read(c5c7f400,c57c0c58,0,9e,0,...) at ttydisc_read+0x22c
ttydev_read(c5b00a00,c57c0c58,0,0,1,...) at ttydev_read+0xaa
devfs_read_f(c5ed3850,c57c0c58,c596b100,0,c5eded80,...) at devfs_read_f+0x7e
dofileread(c57c0c58,ffffffff,ffffffff,0,c5ed3850,...) at dofileread+0x96
kern_readv(c5eded80,0,c57c0c58,c57c0c78,1,...) at kern_readv+0x58
read(c5eded80,c57c0cf8,c,c0c916cc,c0d5f9b4,...) at read+0x4f
syscall(c57c0d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp = 0xbfbfedbc, ebp = 0xbfbfede8 ---

Tracing command getty pid 1517 tid 100083 td 0xc6121480
sched_switch(c6121480,0,104,191,914789b2,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6121480,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0c7d8d5,15b,0,c6121480,c6121480,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c59eca70,0,c0c81ed5,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c59eca70,c0dce270,c0c81880,4d5,0,...) at _cv_wait_sig+0x240
tty_wait(c59eca00,c59eca70,c57f9c58,1,0,...) at tty_wait+0x71
ttydisc_read(c59eca00,c57f9c58,0,9e,0,...) at ttydisc_read+0x22c
ttydev_read(c5c85a00,c57f9c58,0,0,1,...) at ttydev_read+0xaa
devfs_read_f(c5ed3498,c57f9c58,c596b100,0,c6121480,...) at devfs_read_f+0x7e
dofileread(c57f9c58,ffffffff,ffffffff,0,c5ed3498,...) at dofileread+0x96
kern_readv(c6121480,0,c57f9c58,c57f9c78,1,...) at kern_readv+0x58
read(c6121480,c57f9cf8,c,c0c916cc,c0d5f9b4,...) at read+0x4f
syscall(c57f9d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp = 0xbfbfedcc, ebp = 0xbfbfedf8 ---

Tracing command getty pid 1516 tid 100077 td 0xc5ede000
sched_switch(c5ede000,0,104,191,9126c18e,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ede000,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0c7d8d5,15b,0,c5ede000,c5ede000,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c59ed270,0,c0c81ed5,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c59ed270,c0dce270,c0c81880,4d5,0,...) at _cv_wait_sig+0x240
tty_wait(c59ed200,c59ed270,c57e7c58,1,0,...) at tty_wait+0x71
ttydisc_read(c59ed200,c57e7c58,0,9e,0,...) at ttydisc_read+0x22c
ttydev_read(c5b00100,c57e7c58,0,0,1,...) at ttydev_read+0xaa
devfs_read_f(c5ed3af0,c57e7c58,c596b100,0,c5ede000,...) at devfs_read_f+0x7e
dofileread(c57e7c58,ffffffff,ffffffff,0,c5ed3af0,...) at dofileread+0x96
kern_readv(c5ede000,0,c57e7c58,c57e7c78,1,...) at kern_readv+0x58
read(c5ede000,c57e7cf8,c,c0c916cc,c0d5f9b4,...) at read+0x4f
syscall(c57e7d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp = 0xbfbfedcc, ebp = 0xbfbfedf8 ---

Tracing command getty pid 1515 tid 100074 td 0xc5ede6c0
sched_switch(c5ede6c0,0,104,191,9105c10a,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ede6c0,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0c7d8d5,15b,0,c5ede6c0,c5ede6c0,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c59edc70,0,c0c81ed5,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c59edc70,c0dce270,c0c81880,4d5,0,...) at _cv_wait_sig+0x240
tty_wait(c59edc00,c59edc70,c57cac58,1,0,...) at tty_wait+0x71
ttydisc_read(c59edc00,c57cac58,0,9e,0,...) at ttydisc_read+0x22c
ttydev_read(c5c85800,c57cac58,0,0,1,...) at ttydev_read+0xaa
devfs_read_f(c5ed3930,c57cac58,c596b100,0,c5ede6c0,...) at devfs_read_f+0x7e
dofileread(c57cac58,ffffffff,ffffffff,0,c5ed3930,...) at dofileread+0x96
kern_readv(c5ede6c0,0,c57cac58,c57cac78,1,...) at kern_readv+0x58
read(c5ede6c0,c57cacf8,c,c0c916cc,c0d5f9b4,...) at read+0x4f
syscall(c57cad38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp = 0xbfbfedcc, ebp = 0xbfbfedf8 ---

Tracing command getty pid 1514 tid 100081 td 0xc6121900
sched_switch(c6121900,0,104,191,90e4c0de,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6121900,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0c7d8d5,15b,0,c6121900,c6121900,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c59ed670,0,c0c81ed5,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c59ed670,c0dce270,c0c81880,4d5,0,...) at _cv_wait_sig+0x240
tty_wait(c59ed600,c59ed670,c57f3c58,1,0,...) at tty_wait+0x71
ttydisc_read(c59ed600,c57f3c58,0,9e,0,...) at ttydisc_read+0x22c
ttydev_read(c5c85700,c57f3c58,0,0,1,...) at ttydev_read+0xaa
devfs_read_f(c5ed3ce8,c57f3c58,c596b100,0,c6121900,...) at devfs_read_f+0x7e
dofileread(c57f3c58,ffffffff,ffffffff,0,c5ed3ce8,...) at dofileread+0x96
kern_readv(c6121900,0,c57f3c58,c57f3c78,1,...) at kern_readv+0x58
read(c6121900,c57f3cf8,c,c0c916cc,c0d5f9b4,...) at read+0x4f
syscall(c57f3d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp = 0xbfbfedcc, ebp = 0xbfbfedf8 ---

Tracing command getty pid 1513 tid 100063 td 0xc5b06240
sched_switch(c5b06240,0,104,191,90c3f93e,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5b06240,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0c7d8d5,15b,0,c5b06240,c5b06240,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c59ede70,0,c0c81ed5,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c59ede70,c0dce270,c0c81880,4d5,0,...) at _cv_wait_sig+0x240
tty_wait(c59ede00,c59ede70,c5799c58,1,0,...) at tty_wait+0x71
ttydisc_read(c59ede00,c5799c58,0,9e,0,...) at ttydisc_read+0x22c
ttydev_read(c5c85300,c5799c58,0,0,1,...) at ttydev_read+0xaa
devfs_read_f(c5ed3620,c5799c58,c596b100,0,c5b06240,...) at devfs_read_f+0x7e
dofileread(c5799c58,ffffffff,ffffffff,0,c5ed3620,...) at dofileread+0x96
kern_readv(c5b06240,0,c5799c58,c5799c78,1,...) at kern_readv+0x58
read(c5b06240,c5799cf8,c,c0c916cc,c0d5f9b4,...) at read+0x4f
syscall(c5799d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp = 0xbfbfedcc, ebp = 0xbfbfedf8 ---

Tracing command getty pid 1512 tid 100086 td 0xc6120d80
sched_switch(c6120d80,0,104,191,90a2bcba,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6120d80,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0c7d8d5,15b,0,c6120d80,c6120d80,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c59ed470,0,c0c81ed5,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c59ed470,c0dce270,c0c81880,4d5,0,...) at _cv_wait_sig+0x240
tty_wait(c59ed400,c59ed470,e80fcc58,1,0,...) at tty_wait+0x71
ttydisc_read(c59ed400,e80fcc58,0,9e,0,...) at ttydisc_read+0x22c
ttydev_read(c5c85e00,e80fcc58,0,0,1,...) at ttydev_read+0xaa
devfs_read_f(c5ed3bd0,e80fcc58,c596b100,0,c6120d80,...) at devfs_read_f+0x7e
dofileread(e80fcc58,ffffffff,ffffffff,0,c5ed3bd0,...) at dofileread+0x96
kern_readv(c6120d80,0,e80fcc58,e80fcc78,1,...) at kern_readv+0x58
read(c6120d80,e80fccf8,c,c0c916cc,c0d5f9b4,...) at read+0x4f
syscall(e80fcd38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp = 0xbfbfedcc, ebp = 0xbfbfedf8 ---

Tracing command getty pid 1511 tid 100085 td 0xc6121000
sched_switch(c6121000,0,104,191,908225fa,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6121000,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0c7d8d5,15b,0,c6121000,c6121000,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c59ed870,0,c0c81ed5,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c59ed870,c0dce270,c0c81880,4d5,0,...) at _cv_wait_sig+0x240
tty_wait(c59ed800,c59ed870,c57ffc58,1,0,...) at tty_wait+0x71
ttydisc_read(c59ed800,c57ffc58,0,9e,0,...) at ttydisc_read+0x22c
ttydev_read(c5c85900,c57ffc58,0,0,1,...) at ttydev_read+0xaa
devfs_read_f(c5ed3508,c57ffc58,c596b100,0,c6121000,...) at devfs_read_f+0x7e
dofileread(c57ffc58,ffffffff,ffffffff,0,c5ed3508,...) at dofileread+0x96
kern_readv(c6121000,0,c57ffc58,c57ffc78,1,...) at kern_readv+0x58
read(c6121000,c57ffcf8,c,c0c916cc,c0d5f9b4,...) at read+0x4f
syscall(c57ffd38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp = 0xbfbfedcc, ebp = 0xbfbfedf8 ---

Tracing command getty pid 1510 tid 100084 td 0xc6121240
sched_switch(c6121240,0,104,191,918e02aa,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6121240,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0c7d8d5,15b,0,c6121240,c6121240,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c5c7f070,0,c0c81ed5,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c5c7f070,c0dce270,c0c81880,4d5,0,...) at _cv_wait_sig+0x240
tty_wait(c5c7f000,c5c7f070,c57fcc58,1,0,...) at tty_wait+0x71
ttydisc_read(c5c7f000,c57fcc58,0,9e,0,...) at ttydisc_read+0x22c
ttydev_read(c5c86000,c57fcc58,0,0,1,...) at ttydev_read+0xaa
devfs_read_f(c5ed3700,c57fcc58,c596b100,0,c6121240,...) at devfs_read_f+0x7e
dofileread(c57fcc58,ffffffff,ffffffff,0,c5ed3700,...) at dofileread+0x96
kern_readv(c6121240,0,c57fcc58,c57fcc78,1,...) at kern_readv+0x58
read(c6121240,c57fccf8,c,c0c916cc,c0d5f9b4,...) at read+0x4f
syscall(c57fcd38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp = 0xbfbfedcc, ebp = 0xbfbfedf8 ---

Tracing command cron pid 1459 tid 100067 td 0xc5ee06c0
sched_switch(c5ee06c0,0,104,191,7c0f71e0,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,5c,...) at mi_switch+0x200
sleepq_switch(c5ee06c0,0,c0c7d8d5,18b,5c,...) at sleepq_switch+0x15f
sleepq_catch_signals(c351,c08bad00,c5ee06c0,0,100,...) at sleepq_catch_signals+0xb7
sleepq_timedwait_sig(c0dceac4,5c,c0c7a7d5,100,0,...) at sleepq_timedwait_sig+0x1a
_sleep(c0dceac4,0,15c,c0c7a7d5,c351,...) at _sleep+0x31e
kern_nanosleep(c5ee06c0,c57b0c64,c57b0c6c,32,0,...) at kern_nanosleep+0xc1
nanosleep(c5ee06c0,c57b0cf8,8,c0c80283,c0d613a0,...) at nanosleep+0x6f
syscall(c57b0d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x28179c2f, esp = 0xbfbfec8c, ebp = 0xbfbfecb8 ---
       
Tracing command sendmail pid 1453 tid 100069 td 0xc5ee0240
sched_switch(c5ee0240,0,104,191,61ca0cea,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,68,...) at mi_switch+0x200
sleepq_switch(c5ee0240,0,c0c7d8d5,18b,68,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0c7d8d5,15b,0,100,100,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c5f1a058,68,c0c2d13b,100,0,...) at sleepq_wait_sig+0x17
_sleep(c5f1a058,c5f1a088,168,c0c2d13b,0,...) at _sleep+0x354
kern_sigsuspend(c5ee0240,0,0,0,0,...) at kern_sigsuspend+0xe4
sigsuspend(c5ee0240,c57b8cf8,4,c0c80035,c0d61eac,...) at sigsuspend+0x4d
syscall(c57b8d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x28332a3b, esp = 0xbfbfcf9c, ebp = 0xbfbfcfc8 ---

Tracing command sendmail pid 1447 tid 100066 td 0xc5ee0900
sched_switch(c5ee0900,0,104,191,54c1b16c,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ee0900,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c57aca4c,c08738da,c5f2d650,0,c5ee0900,...) at sleepq_catch_signals+0xb7
sleepq_timedwait_sig(c5f2d664,0,c57aca7c,101,0,...) at sleepq_timedwait_sig+0x1a
_cv_timedwait_sig(c5f2d664,c5f2d650,1389,5d1,c5ed32a0,...) at _cv_timedwait_sig+0x250
seltdwait(c57acc28,c57acc30,c6100600,c5ee0900,c188b014,...) at seltdwait+0x8a
kern_select(c5ee0900,5,bfbfc520,0,0,c57acc70,20,5,0) at kern_select+0x4f4
select(c5ee0900,c57accf8,14,c0c80559,c0d6038c,...) at select+0x66
syscall(c57acd38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x283d81d3, esp = 0xbfbfc48c, ebp = 0xbfbfcfb8 ---

Tracing command sshd pid 1440 tid 100068 td 0xc5ee0480
sched_switch(c5ee0480,0,104,191,7cd515d2,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ee0480,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c08738da,c5f2d010,0,c0c77c84,c5ee0480,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c5f2d024,0,c57b4a7c,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c5f2d024,c5f2d010,c0c7fb95,5d1,c5ed34d0,...) at _cv_wait_sig+0x240
seltdwait(c5ed34d0,58,c6100000,c5ee0480,0,...) at seltdwait+0xa2
kern_select(c5ee0480,5,286090ac,0,0,0,20,65,a) at kern_select+0x4f4
select(c5ee0480,c57b4cf8,14,c0c809ab,c0d6038c,...) at select+0x66
syscall(c57b4d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x283cc1d3, esp = 0xbfbfdf1c, ebp = 0xbfbfee38 ---

Tracing command syslogd pid 1210 tid 100072 td 0xc5edeb40
sched_switch(c5edeb40,0,104,191,b0d575ee,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5edeb40,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c08738da,c5c4bd50,0,c0c77c84,c5edeb40,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c5c4bd64,0,c57c4a7c,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c5c4bd64,c5c4bd50,c0c7fb95,5d1,c5ed38c0,...) at _cv_wait_sig+0x240
seltdwait(c5ed38c0,58,c596b100,c5edeb40,0,...) at seltdwait+0xa2
kern_select(c5edeb40,9,282290a8,0,0,0,20,65,281a4ad8) at kern_select+0x4f4
select(c5edeb40,c57c4cf8,14,c0c80135,c0d6038c,...) at select+0x66
syscall(c57c4d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x281921d3, esp = 0xbfbfde7c, ebp = 0xbfbfee28 ---

Tracing command dhclient pid 1161 tid 100075 td 0xc5ede480
sched_switch(c5ede480,0,104,191,5075fdca,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ede480,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c57cdaa8,c08738da,c5f2d4d0,0,c5ede480,...) at sleepq_catch_signals+0xb7
sleepq_timedwait_sig(c5f2d4e4,0,c57cdad8,101,0,...) at sleepq_timedwait_sig+0x1a
_cv_timedwait_sig(c5f2d4e4,c5f2d4d0,5265819,5d1,c57cdb64,...) at _cv_timedwait_sig+0x250
seltdwait(c57cdc5c,c57cdc64,4df,c5ede480,c57cdb5c,...) at seltdwait+0x8a
poll(c5ede480,c57cdcf8,c,c0c18927,c0d6103c,...) at poll+0x300
syscall(c57cdd38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (209, FreeBSD ELF32, poll), eip = 0x2813122f, esp = 0xbfbfedcc, ebp = 0xbfbfedf8 ---

Tracing command dhclient pid 1120 tid 100064 td 0xc5ee0d80
sched_switch(c5ee0d80,0,104,191,c880666e,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ee0d80,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c08738da,c5f2e950,0,c0c77c84,c5ee0d80,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c5f2e964,0,c579dad8,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c5f2e964,c5f2e950,c0c7fb95,5d1,c579db5c,...) at _cv_wait_sig+0x240
seltdwait(c5f6692c,c0c7fb95,4df,c5ee0d80,c579db5c,...) at seltdwait+0xa2
poll(c5ee0d80,c579dcf8,c,c579dcb8,c0d6103c,...) at poll+0x300
syscall(c579dd38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (209, FreeBSD ELF32, poll), eip = 0x2813122f, esp = 0xbfbfedcc, ebp = 0xbfbfedf8 ---

Tracing command devd pid 971 tid 100065 td 0xc5ee0b40
sched_switch(c5ee0b40,0,104,191,1d25f46e,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ee0b40,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c08738da,c5f2de10,0,c0c77c84,c5ee0b40,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c5f2de24,0,c57a1a7c,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c5f2de24,c5f2de10,c0c7fb95,5d1,c5ed3380,...) at _cv_wait_sig+0x240
seltdwait(c5ed3380,58,c596b100,c5ee0b40,c0c717ff,...) at seltdwait+0xa2
kern_select(c5ee0b40,6,bfbfe9b0,0,0,0,20,c0dd2800,b) at kern_select+0x4f4
select(c5ee0b40,c57a1cf8,14,c57a1ca0,c0d6038c,...) at select+0x66
syscall(c57a1d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x80880e3, esp = 0xbfbfe97c, ebp = 0xbfbfee58 ---

Tracing command moused pid 954 tid 100073 td 0xc5ede900
sched_switch(c5ede900,0,104,191,b66153a,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ede900,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c08738da,c5f2d810,0,c0c77c84,c5ede900,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c5f2d824,0,c57c7a7c,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c5f2d824,c5f2d810,c0c7fb95,5d1,c5ed3b28,...) at _cv_wait_sig+0x240
seltdwait(c5ed3b28,58,c596b100,c5ede900,0,...) at seltdwait+0xa2
kern_select(c5ede900,7,bfbfea14,0,0,0,20,65,0) at kern_select+0x4f4
select(c5ede900,c57c7cf8,14,c0c916cc,c0d6038c,...) at select+0x66
syscall(c57c7d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x281ac1d3, esp = 0xbfbfe99c, ebp = 0xbfbfeb38 ---

Tracing command flowcleaner pid 21 tid 100062 td 0xc5b06480
sched_switch(c5b06480,0,104,191,8776d332,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5b06480,0,c0c7d8d5,26e,c5b06480,...) at sleepq_switch+0x15f
sleepq_timedwait(c0f3b548,0,c5791cc4,1,0,...) at sleepq_timedwait+0x6b
_cv_timedwait(c0f3b548,c0f3b550,2710,3f0,0,...) at _cv_timedwait+0x250
flowtable_cleaner(0,c5791d38,c0c749df,343,c5e9b000,...) at flowtable_cleaner+0x1bf
fork_exit(c092a660,0,c5791d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5791d70, ebp = 0 ---
       
Tracing command softdepflush pid 20 tid 100061 td 0xc5b066c0
sched_switch(c5b066c0,0,104,191,9a91c59a,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,44,...) at mi_switch+0x200
sleepq_switch(c5b066c0,0,c0c7d8d5,26e,0,...) at sleepq_switch+0x15f
sleepq_timedwait(c0f46c40,44,c0ca021c,0,0,...) at sleepq_timedwait+0x6b
_sleep(c0f46c40,c0f46be4,44,c0ca021c,3e8,...) at _sleep+0x339
softdep_flush(0,c578ed38,c0c749df,343,c5e9b2a8,...) at softdep_flush+0x244
fork_exit(c0ab3a60,0,c578ed38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc578ed70, ebp = 0 ---

Tracing command vnlru pid 19 tid 100060 td 0xc5b06900
sched_switch(c5b06900,0,104,191,9a9157a6,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,50,...) at mi_switch+0x200
sleepq_switch(c5b06900,0,c0c7d8d5,26e,0,...) at sleepq_switch+0x15f
sleepq_timedwait(c5e9b550,50,c0c879e7,0,0,...) at sleepq_timedwait+0x6b
_sleep(c5e9b550,c0f3b314,250,c0c879e7,3e8,...) at _sleep+0x339
vnlru_proc(0,c578bd38,c0c749df,343,c5e9b550,...) at vnlru_proc+0xe7
fork_exit(c0916060,0,c578bd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc578bd70, ebp = 0 ---

Tracing command syncer pid 18 tid 100059 td 0xc5b06b40
sched_switch(c5b06b40,0,104,191,77355a82,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5b06b40,0,c0c7d8d5,26e,c5b06b40,...) at sleepq_switch+0x15f
sleepq_timedwait(c0f3b354,0,c5788c88,1,0,...) at sleepq_timedwait+0x6b
_cv_timedwait(c0f3b354,c0f3b340,3e8,6cc,4e20,...) at _cv_timedwait+0x250
sched_sync(0,c5788d38,c0c749df,343,c5e9b7f8,...) at sched_sync+0x502
fork_exit(c0915490,0,c5788d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5788d70, ebp = 0 ---

Tracing command bufdaemon pid 17 tid 100058 td 0xc5b06d80
sched_switch(c5b06d80,0,104,191,9a911762,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,44,...) at mi_switch+0x200
sleepq_switch(c5b06d80,0,c0c7d8d5,26e,0,...) at sleepq_switch+0x15f
sleepq_timedwait(c0f3b088,44,c0c84f5d,0,0,...) at sleepq_timedwait+0x6b
_sleep(c0f3b088,c0f3b08c,44,c0c84f5d,3e8,...) at _sleep+0x339
buf_daemon(0,c5785d38,c0c749df,343,c5e9baa0,...) at buf_daemon+0x138
fork_exit(c08fd800,0,c5785d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5785d70, ebp = 0 ---

Tracing command pagezero pid 16 tid 100057 td 0xc5c82000
sched_switch(c5c82000,0,104,191,e6d3fcae,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5c82000,0,c0c7d8d5,26e,0,...) at sleepq_switch+0x15f
sleepq_timedwait(c0f47914,0,c0ca5b25,0,0,...) at sleepq_timedwait+0x6b
_sleep(c0f47914,c0f474d0,0,c0ca5b25,493e0,...) at _sleep+0x339
vm_pagezero(0,c5782d38,c0c749df,343,c5e9bd48,...) at vm_pagezero+0xdc
fork_exit(c0af0df0,0,c5782d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5782d70, ebp = 0 ---

Tracing command vmdaemon pid 15 tid 100056 td 0xc5c82240
sched_switch(c5c82240,0,104,191,794357b6,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,68,...) at mi_switch+0x200
sleepq_switch(c5c82240,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c0f4753c,68,c0c84f5d,0,0,...) at sleepq_wait+0x63
_sleep(c0f4753c,c0f47540,68,c0c84f5d,0,...) at _sleep+0x36b
vm_daemon(0,c577fd38,c0c749df,343,c596e2a8,...) at vm_daemon+0x59
fork_exit(c0aeb240,0,c577fd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc577fd70, ebp = 0 ---

Tracing command pagedaemon pid 9 tid 100055 td 0xc5c82480
sched_switch(c5c82480,0,104,191,41e649ea,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,44,...) at mi_switch+0x200
sleepq_switch(c5c82480,0,c0c7d8d5,26e,0,...) at sleepq_switch+0x15f
sleepq_timedwait(c0f47504,44,c0c84f5d,0,0,...) at sleepq_timedwait+0x6b
_sleep(c0f47504,c0f474d0,44,c0c84f5d,1388,...) at _sleep+0x339
vm_pageout(0,c577cd38,c0c749df,343,c596e550,...) at vm_pageout+0x2bb
fork_exit(c0aec0e0,0,c577cd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc577cd70, ebp = 0 ---

Tracing command sctp_iterator pid 8 tid 100054 td 0xc5c826c0
sched_switch(c5c826c0,0,104,191,be4bf2fa,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5c826c0,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c0f3ce3c,0,c0c92e35,0,0,...) at sleepq_wait+0x63
_sleep(c0f3ce3c,c0f3cd50,0,c0c92e35,0,...) at _sleep+0x36b
sctp_iterator_thread(0,c5779d38,c0c749df,343,c596e7f8,...) at sctp_iterator_thread+0x60
fork_exit(c099ed00,0,c5779d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5779d70, ebp = 0 ---

Tracing command fdc0 pid 7 tid 100050 td 0xc5ad66c0
sched_switch(c5ad66c0,0,104,191,76ec9a8a,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,4c,...) at mi_switch+0x200
sleepq_switch(c5ad66c0,0,c0c7d8d5,26e,0,...) at sleepq_switch+0x15f
sleepq_timedwait(c59ec23c,4c,c0c6f250,0,0,...) at sleepq_timedwait+0x6b
_sleep(c59ec23c,c59ec2f0,4c,c0c6f250,3e8,...) at _sleep+0x339
fdc_thread(c59ec200,c576cd38,c0c749df,343,c596eaa0,...) at fdc_thread+0x2be
fork_exit(c0b6e360,c59ec200,c576cd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc576cd70, ebp = 0 ---

Tracing command fw0_probe pid 6 tid 100047 td 0xc5ad6d80
sched_switch(c5ad6d80,0,104,191,397c48de,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,5c,...) at mi_switch+0x200
sleepq_switch(c5ad6d80,0,c0c7d8d5,18b,5c,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0c7d8d5,15b,0,100,100,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c5b6e000,5c,c0c6f250,100,0,...) at sleepq_wait_sig+0x17
_sleep(c5b6e000,c5b72488,15c,c0c6f250,0,...) at _sleep+0x354
fw_bus_probe_thread(c5b6e000,c5588d38,c0c749df,343,c596ed48,...) at fw_bus_probe_thread+0xa08
fork_exit(c0654740,c5b6e000,c5588d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5588d70, ebp = 0 ---

Tracing command usb pid 14 tid 100042 td 0xc5b05900
sched_switch(c5b05900,0,104,191,ab8bccb2,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5b05900,0,c0c7d8d5,24b,c5b05900,...) at sleepq_switch+0x15f
sleepq_wait(c5aedd0c,0,c555ecb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5aedd0c,c5aeddac,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240
usb_process(c5aedd04,c555ed38,c0c749df,343,c59d9000,...) at usb_process+0x193
fork_exit(c07b7c80,c5aedd04,c555ed38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc555ed70, ebp = 0 ---

Tracing command usb pid 14 tid 100041 td 0xc5b05b40
sched_switch(c5b05b40,0,104,191,1e3c062e,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5b05b40,0,c0c7d8d5,24b,c5b05b40,...) at sleepq_switch+0x15f
sleepq_wait(c5aedcdc,0,c555bcb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5aedcdc,c5aeddac,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240
usb_process(c5aedcd4,c555bd38,c0c749df,343,c59d9000,...) at usb_process+0x193
fork_exit(c07b7c80,c5aedcd4,c555bd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc555bd70, ebp = 0 ---

Tracing command usb pid 14 tid 100040 td 0xc5b05d80
sched_switch(c5b05d80,0,104,191,b9c1c834,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5b05d80,0,c0c7d8d5,24b,c5b05d80,...) at sleepq_switch+0x15f
sleepq_wait(c5aedcac,0,c5558cb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5aedcac,c5aeddac,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240
usb_process(c5aedca4,c5558d38,c0c749df,343,c59d9000,...) at usb_process+0x193
fork_exit(c07b7c80,c5aedca4,c5558d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5558d70, ebp = 0 ---

Tracing command usb pid 14 tid 100039 td 0xc5b06000
sched_switch(c5b06000,0,104,191,8b9b48a0,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5b06000,0,c0c7d8d5,24b,c5b06000,...) at sleepq_switch+0x15f
sleepq_wait(c5aedc7c,0,c5555cb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5aedc7c,c5aeddac,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240
usb_process(c5aedc74,c5555d38,c0c749df,343,c59d9000,...) at usb_process+0x193
fork_exit(c07b7c80,c5aedc74,c5555d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5555d70, ebp = 0 ---

Tracing command usb pid 14 tid 100037 td 0xc5ad4000
sched_switch(c5ad4000,0,104,191,8b9b1c18,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ad4000,0,c0c7d8d5,24b,c5ad4000,...) at sleepq_switch+0x15f
sleepq_wait(c5adadac,0,c554dcb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5adadac,c5adae4c,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240
usb_process(c5adada4,c554dd38,c0c749df,343,c59d9000,...) at usb_process+0x193
fork_exit(c07b7c80,c5adada4,c554dd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc554dd70, ebp = 0 ---

Tracing command usb pid 14 tid 100036 td 0xc5ad4240
sched_switch(c5ad4240,0,104,191,ddca875a,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ad4240,0,c0c7d8d5,24b,c5ad4240,...) at sleepq_switch+0x15f
sleepq_wait(c5adad7c,0,c554acb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5adad7c,c5adae4c,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240
usb_process(c5adad74,c554ad38,c0c749df,343,c59d9000,...) at usb_process+0x193
fork_exit(c07b7c80,c5adad74,c554ad38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc554ad70, ebp = 0 ---

Tracing command usb pid 14 tid 100035 td 0xc5ad4480
sched_switch(c5ad4480,0,104,191,8589b4b4,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ad4480,0,c0c7d8d5,24b,c5ad4480,...) at sleepq_switch+0x15f
sleepq_wait(c5adad4c,0,c5547cb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5adad4c,c5adae4c,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240
usb_process(c5adad44,c5547d38,c0c749df,343,c59d9000,...) at usb_process+0x193
fork_exit(c07b7c80,c5adad44,c5547d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5547d70, ebp = 0 ---

Tracing command usb pid 14 tid 100034 td 0xc5ad46c0
sched_switch(c5ad46c0,0,104,191,85898ec8,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ad46c0,0,c0c7d8d5,24b,c5ad46c0,...) at sleepq_switch+0x15f
sleepq_wait(c5adad1c,0,c5544cb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5adad1c,c5adae4c,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240
usb_process(c5adad14,c5544d38,c0c749df,343,c59d9000,...) at usb_process+0x193
fork_exit(c07b7c80,c5adad14,c5544d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5544d70, ebp = 0 ---

Tracing command usb pid 14 tid 100032 td 0xc5ad4b40
sched_switch(c5ad4b40,0,104,191,8589586c,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ad4b40,0,c0c7d8d5,24b,c5ad4b40,...) at sleepq_switch+0x15f
sleepq_wait(c5ac5dac,0,c553dcb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5ac5dac,c5ac5e4c,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240
usb_process(c5ac5da4,c553dd38,c0c749df,343,c59d9000,...) at usb_process+0x193
fork_exit(c07b7c80,c5ac5da4,c553dd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc553dd70, ebp = 0 ---

Tracing command usb pid 14 tid 100031 td 0xc5ad4d80
sched_switch(c5ad4d80,0,104,191,dd5c7726,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ad4d80,0,c0c7d8d5,24b,c5ad4d80,...) at sleepq_switch+0x15f
sleepq_wait(c5ac5d7c,0,c553acb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5ac5d7c,c5ac5e4c,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240
usb_process(c5ac5d74,c553ad38,c0c749df,343,c59d9000,...) at usb_process+0x193
fork_exit(c07b7c80,c5ac5d74,c553ad38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc553ad70, ebp = 0 ---

Tracing command usb pid 14 tid 100030 td 0xc5ad6000
sched_switch(c5ad6000,0,104,191,7f759cb0,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ad6000,0,c0c7d8d5,24b,c5ad6000,...) at sleepq_switch+0x15f
sleepq_wait(c5ac5d4c,0,c5537cb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5ac5d4c,c5ac5e4c,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240
usb_process(c5ac5d44,c5537d38,c0c749df,343,c59d9000,...) at usb_process+0x193
fork_exit(c07b7c80,c5ac5d44,c5537d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5537d70, ebp = 0 ---

Tracing command usb pid 14 tid 100029 td 0xc5ad6240
sched_switch(c5ad6240,0,104,191,7f757954,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ad6240,0,c0c7d8d5,24b,c5ad6240,...) at sleepq_switch+0x15f
sleepq_wait(c5ac5d1c,0,c5534cb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5ac5d1c,c5ac5e4c,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240
usb_process(c5ac5d14,c5534d38,c0c749df,343,c59d9000,...) at usb_process+0x193
fork_exit(c07b7c80,c5ac5d14,c5534d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5534d70, ebp = 0 ---

Tracing command usb pid 14 tid 100027 td 0xc59b56c0
sched_switch(c59b56c0,0,104,191,de33bc1e,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c59b56c0,0,c0c7d8d5,24b,c59b56c0,...) at sleepq_switch+0x15f
sleepq_wait(c5aaedac,0,c552dcb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5aaedac,c5aaee4c,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240
usb_process(c5aaeda4,c552dd38,c0c749df,343,c59d9000,...) at usb_process+0x193
fork_exit(c07b7c80,c5aaeda4,c552dd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc552dd70, ebp = 0 ---

Tracing command usb pid 14 tid 100026 td 0xc59b5900
sched_switch(c59b5900,0,104,191,de342e32,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c59b5900,0,c0c7d8d5,24b,c59b5900,...) at sleepq_switch+0x15f
sleepq_wait(c5aaed7c,0,c552acb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5aaed7c,c5aaee4c,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240
usb_process(c5aaed74,c552ad38,c0c749df,343,c59d9000,...) at usb_process+0x193
fork_exit(c07b7c80,c5aaed74,c552ad38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc552ad70, ebp = 0 ---

Tracing command usb pid 14 tid 100025 td 0xc59b5b40
sched_switch(c59b5b40,0,104,191,72ec5146,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c59b5b40,0,c0c7d8d5,24b,c59b5b40,...) at sleepq_switch+0x15f
sleepq_wait(c5aaed4c,0,c5527cb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5aaed4c,c5aaee4c,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240
usb_process(c5aaed44,c5527d38,c0c749df,343,c59d9000,...) at usb_process+0x193
fork_exit(c07b7c80,c5aaed44,c5527d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5527d70, ebp = 0 ---

Tracing command usb pid 14 tid 100024 td 0xc59b5d80
sched_switch(c59b5d80,0,104,191,72edde2a,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c59b5d80,0,c0c7d8d5,24b,c59b5d80,...) at sleepq_switch+0x15f
sleepq_wait(c5aaed1c,0,c5524cb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5aaed1c,c5aaee4c,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240
usb_process(c5aaed14,c5524d38,c0c749df,343,c59d9000,...) at usb_process+0x193
fork_exit(c07b7c80,c5aaed14,c5524d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5524d70, ebp = 0 ---

Tracing command xpt_thrd pid 5 tid 100018 td 0xc5a85b40
sched_switch(c5a85b40,0,104,191,795df7fc,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,4c,...) at mi_switch+0x200
sleepq_switch(c5a85b40,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c0d9b0d4,4c,c0c14384,0,0,...) at sleepq_wait+0x63
_sleep(c0d9b0d4,c0d9b0ec,4c,c0c14384,0,...) at _sleep+0x36b
xpt_scanner_thread(0,c54fbd38,c0c749df,343,c59d92a8,...) at xpt_scanner_thread+0x4a
fork_exit(c0483010,0,c54fbd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54fbd70, ebp = 0 ---

Tracing command yarrow pid 13 tid 100011 td 0xc59b5000
sched_switch(c59b5000,0,104,191,d456c068,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c59b5000,0,c0c7d8d5,26e,2,...) at sleepq_switch+0x15f
sleepq_timedwait(c0dce924,0,c0c6f250,2,0,...) at sleepq_timedwait+0x6b
_sleep(c0dce924,0,0,c0c6f250,64,...) at _sleep+0x339
pause(c0c6f250,64,c0c5bcd5,111,0,...) at pause+0x47
random_kthread(0,c54e6d38,c0c749df,343,c59d9550,...) at random_kthread+0x1ef
fork_exit(c07308f0,0,c54e6d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54e6d70, ebp = 0 ---

Tracing command g_down pid 4 tid 100009 td 0xc59b5480
sched_switch(c59b5480,0,104,191,d6a10fd4,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,4c,...) at mi_switch+0x200
sleepq_switch(c59b5480,0,c0c7d8d5,26e,0,...) at sleepq_switch+0x15f
sleepq_timedwait(c0dcc6e4,4c,c0c6f250,0,0,...) at sleepq_timedwait+0x6b
_sleep(c0dcc6e4,c0dcc648,24c,c0c6f250,64,...) at _sleep+0x339
g_io_schedule_down(c59b5480,0,c0c70886,74,0,...) at g_io_schedule_down+0x6b
g_down_procbody(0,c54e0d38,c0c749df,343,c596d000,...) at g_down_procbody+0x8d
fork_exit(c0820a20,0,c54e0d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54e0d70, ebp = 0 ---

Tracing command g_up pid 3 tid 100008 td 0xc596f000
sched_switch(c596f000,0,104,191,d6a139cc,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,4c,...) at mi_switch+0x200
sleepq_switch(c596f000,0,c0c7d8d5,26e,0,...) at sleepq_switch+0x15f
sleepq_timedwait(c0dcc6e0,4c,c0c6f250,0,0,...) at sleepq_timedwait+0x6b
_sleep(c0dcc6e0,c0dcc668,24c,c0c6f250,64,...) at _sleep+0x339
g_io_schedule_up(c596f000,0,c0c70886,5d,0,...) at g_io_schedule_up+0x133
g_up_procbody(0,c54ddd38,c0c749df,343,c596d2a8,...) at g_up_procbody+0x8d
fork_exit(c0820ab0,0,c54ddd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54ddd70, ebp = 0 ---

Tracing command g_event pid 2 tid 100007 td 0xc596f240
sched_switch(c596f240,0,104,191,d4571590,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,4c,...) at mi_switch+0x200
sleepq_switch(c596f240,0,c0c7d8d5,26e,0,...) at sleepq_switch+0x15f
sleepq_timedwait(c0dcc6d8,4c,c0c6f250,0,0,...) at sleepq_timedwait+0x6b
_sleep(c0dcc6d8,0,4c,c0c6f250,64,...) at _sleep+0x339
g_event_procbody(0,c54dad38,c0c749df,343,c596d550,...) at g_event_procbody+0xcb
fork_exit(c0820b40,0,c54dad38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54dad70, ebp = 0 ---

Tracing command intr pid 12 tid 100053 td 0xc5c82900
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100052 td 0xc5c82b40
sched_switch(c5c82b40,0,109,191,d922a5cc,...) at sched_switch+0x406
mi_switch(109,0,c0c74c66,4f6,c5c79070,...) at mi_switch+0x200
ithread_loop(c5c842e0,c5773d38,c0c749df,343,c596d7f8,...) at ithread_loop+0x1f6
fork_exit(c085c730,c5c842e0,c5773d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5773d70, ebp = 0 ---

Tracing command intr pid 12 tid 100051 td 0xc5c82d80
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100049 td 0xc5ad6900
sched_switch(c5ad6900,0,109,191,8409d254,...) at sched_switch+0x406
mi_switch(109,0,c0c74c66,4f6,c59b42f0,...) at mi_switch+0x200
ithread_loop(c5c6c620,c575fd38,c0c749df,343,c596d7f8,...) at ithread_loop+0x1f6
fork_exit(c085c730,c5c6c620,c575fd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc575fd70, ebp = 0 ---

Tracing command intr pid 12 tid 100048 td 0xc5ad6b40
sched_switch(c5ad6b40,0,109,191,766e7c92,...) at sched_switch+0x406
mi_switch(109,0,c0c74c66,4f6,c59b4370,...) at mi_switch+0x200
ithread_loop(c5c6c490,c56f4d38,c0c749df,343,c596d7f8,...) at ithread_loop+0x1f6
fork_exit(c085c730,c5c6c490,c56f4d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc56f4d70, ebp = 0 ---

Tracing command intr pid 12 tid 100045 td 0xc5b05240
sched_switch(c5b05240,0,109,191,79566bac,...) at sched_switch+0x406
mi_switch(109,0,c0c74c66,4f6,c59b41f0,...) at mi_switch+0x200
ithread_loop(c5b078e0,c557fd38,c0c749df,343,c596d7f8,...) at ithread_loop+0x1f6
fork_exit(c085c730,c5b078e0,c557fd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc557fd70, ebp = 0 ---

Tracing command intr pid 12 tid 100043 td 0xc5b056c0
sched_switch(c5b056c0,0,109,191,78975ef2,...) at sched_switch+0x406
mi_switch(109,0,c0c74c66,4f6,c59b3ef0,...) at mi_switch+0x200
ithread_loop(c5b07980,c5576d38,c0c749df,343,c596d7f8,...) at ithread_loop+0x1f6
fork_exit(c085c730,c5b07980,c5576d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5576d70, ebp = 0 ---

Tracing command intr pid 12 tid 100038 td 0xc5a85d80
sched_switch(c5a85d80,0,109,191,b9c1897c,...) at sched_switch+0x406
mi_switch(109,0,c0c74c66,4f6,c59b3df0,...) at mi_switch+0x200
ithread_loop(c5aaa740,c5552d38,c0c749df,343,c596d7f8,...) at ithread_loop+0x1f6
fork_exit(c085c730,c5aaa740,c5552d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5552d70, ebp = 0 ---

Tracing command intr pid 12 tid 100033 td 0xc5ad4900
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100028 td 0xc5ad6480
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100023 td 0xc5a85000
sched_switch(c5a85000,0,109,191,de33447e,...) at sched_switch+0x406
mi_switch(109,0,c0c74c66,4f6,c59b4270,...) at mi_switch+0x200
ithread_loop(c5aa9820,c5521d38,c0c749df,343,c596d7f8,...) at ithread_loop+0x1f6
fork_exit(c085c730,c5aa9820,c5521d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5521d70, ebp = 0 ---

Tracing command intr pid 12 tid 100022 td 0xc5a85240
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100020 td 0xc5a856c0
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100019 td 0xc5a85900
sched_switch(c5a85900,0,109,191,b9c50404,...) at sched_switch+0x406
mi_switch(109,0,c0c74c66,4f6,c5a838f0,...) at mi_switch+0x200
ithread_loop(c590aa40,c54fed38,c0c749df,343,c596d7f8,...) at ithread_loop+0x1f6
fork_exit(c085c730,c590aa40,c54fed38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54fed70, ebp = 0 ---

Tracing command intr pid 12 tid 100013 td 0xc5971b40
sched_switch(c5971b40,0,109,191,387ac886,...) at sched_switch+0x406
mi_switch(109,0,c0c74c66,4f6,c59b3070,...) at mi_switch+0x200
ithread_loop(c590a9b0,c54ecd38,c0c749df,343,c596d7f8,...) at ithread_loop+0x1f6
fork_exit(c085c730,c590a9b0,c54ecd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54ecd70, ebp = 0 ---

Tracing command intr pid 12 tid 100012 td 0xc5971d80
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100006 td 0xc596f480
sched_switch(c596f480,0,109,191,d7c526fc,...) at sched_switch+0x406
mi_switch(109,0,c0c74c66,4f6,c59b3c70,...) at mi_switch+0x200
ithread_loop(c596c0c0,c54d7d38,c0c749df,343,c596d7f8,...) at ithread_loop+0x1f6
fork_exit(c085c730,c596c0c0,c54d7d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54d7d70, ebp = 0 ---

Tracing command intr pid 12 tid 100005 td 0xc596f6c0
sched_switch(c596f6c0,0,109,191,7b7ba1ea,...) at sched_switch+0x406
mi_switch(109,0,c0c74c66,4f6,c59b3cf0,...) at mi_switch+0x200
ithread_loop(c596c0d0,c54d4d38,c0c749df,343,c596d7f8,...) at ithread_loop+0x1f6
fork_exit(c085c730,c596c0d0,c54d4d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54d4d70, ebp = 0 ---

Tracing command intr pid 12 tid 100004 td 0xc596f900
fork_trampoline() at fork_trampoline

Tracing command idle pid 11 tid 100003 td 0xc596fb40
sched_switch(c596fb40,0,608,18c,b9c14e64,...) at sched_switch+0x406
mi_switch(608,0,c0c79c0b,cf,0,...) at mi_switch+0x200
critical_exit(c5953f7c,c596fb40,c5953f7c,c59b3d80,17,...) at critical_exit+0xa8
intr_event_handle(c59b3d80,c54cdc7c,c54cdc74,2710,c0dd2800,...) at intr_event_handle+0xba
intr_execute_handlers(c5953f7c,c54cdc7c,0,c54cdcf8,c0b98a14,...) at intr_execute_handlers+0x49
lapic_handle_intr(34,c54cdc7c) at lapic_handle_intr+0x4c
Xapic_isr1() at Xapic_isr1+0x34
--- interrupt, eip = 0xc08a594c, esp = 0xc54cdcbc, ebp = 0xc54cdcf8 ---
sched_idletd(0,c54cdd38,c0c749df,343,c596daa0,...) at sched_idletd+0x21c
fork_exit(c08a5730,0,c54cdd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54cdd70, ebp = 0 ---

Tracing command init pid 1 tid 100002 td 0xc596fd80
sched_switch(c596fd80,0,104,191,7ccb36e6,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,5c,...) at mi_switch+0x200
sleepq_switch(c596fd80,0,c0c7d8d5,18b,5c,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0c7d8d5,15b,0,100,100,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c596dd48,5c,c0c80146,100,0,...) at sleepq_wait_sig+0x17
_sleep(c596dd48,c596ddd0,15c,c0c80146,0,...) at _sleep+0x354
kern_wait(c596fd80,ffffffff,c54c9c74,0,0,...) at kern_wait+0xb76
wait4(c596fd80,c54c9cf8,10,c0c7ff76,c0d5fa24,...) at wait4+0x3b
syscall(c54c9d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x8054b8f, esp = 0xbfbfe90c, ebp = 0xbfbfe928 ---

Tracing command audit pid 10 tid 100001 td 0xc5971000
sched_switch(c5971000,0,104,191,7959b734,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5971000,0,c0c7d8d5,24b,c5971000,...) at sleepq_switch+0x15f
sleepq_wait(c0f46560,0,c54c6c9c,1,0,...) at sleepq_wait+0x63
_cv_wait(c0f46560,c0f46544,c0c9d95e,194,c0d67918,...) at _cv_wait+0x240
audit_worker(0,c54c6d38,c0c749df,343,c596e000,...) at audit_worker+0x84
fork_exit(c0a87420,0,c54c6d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54c6d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100118 td 0xc6122240
sched_switch(c6122240,0,104,191,c19cab6c,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6122240,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(d34c0cc0,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(d34c0cc0,d34c0cd8,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c72d5c80,e8170d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c72d5c80,e8170d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8170d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100115 td 0xc6122b40
sched_switch(c6122b40,0,104,191,7673f806,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6122b40,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6587940,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6587940,c6587958,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5e6cc40,e8159d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5e6cc40,e8159d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8159d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100113 td 0xc658c000
sched_switch(c658c000,0,104,191,b9c61070,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658c000,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6588240,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6588240,c6588258,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f390,e8153d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f390,e8153d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8153d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100112 td 0xc658c240
sched_switch(c658c240,0,104,191,6f062614,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658c240,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6588200,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6588200,c6588218,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5b0c390,e8150d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5b0c390,e8150d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8150d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100111 td 0xc658c480
sched_switch(c658c480,0,104,191,6f0602b0,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658c480,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6588180,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6588180,c6588198,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f450,e814dd38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f450,e814dd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe814dd70, ebp = 0 ---

Tracing command kernel pid 0 tid 100110 td 0xc658c6c0
sched_switch(c658c6c0,0,104,191,6f05dd6c,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658c6c0,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6588140,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6588140,c6588158,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f4c0,e814ad38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f4c0,e814ad38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe814ad70, ebp = 0 ---

Tracing command kernel pid 0 tid 100109 td 0xc658d240
sched_switch(c658d240,0,104,191,6f0351a8,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658d240,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6588100,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6588100,c6588118,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f460,e8147d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f460,e8147d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8147d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100108 td 0xc658d480
sched_switch(c658d480,0,104,191,6f032cf4,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658d480,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c65880c0,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c65880c0,c65880d8,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f570,e8144d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f570,e8144d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8144d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100107 td 0xc6120000
sched_switch(c6120000,0,104,191,b97558e6,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6120000,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6588080,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6588080,c6588098,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f510,e8141d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f510,e8141d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8141d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100106 td 0xc6120240
sched_switch(c6120240,0,104,191,b85eff06,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6120240,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6588000,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6588000,c6588018,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f500,e813ed38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f500,e813ed38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe813ed70, ebp = 0 ---

Tracing command kernel pid 0 tid 100105 td 0xc658c900
sched_switch(c658c900,0,104,191,b85e93c2,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658c900,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6588000,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6588000,c6588018,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f500,e813bd38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f500,e813bd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe813bd70, ebp = 0 ---

Tracing command kernel pid 0 tid 100104 td 0xc658cb40
sched_switch(c658cb40,0,104,191,b85f7bde,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658cb40,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6588000,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6588000,c6588018,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f500,e8138d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f500,e8138d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8138d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100103 td 0xc658cd80
sched_switch(c658cd80,0,104,191,b85f588e,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658cd80,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6588000,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6588000,c6588018,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f500,e8135d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f500,e8135d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8135d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100102 td 0xc658d000
sched_switch(c658d000,0,104,191,b85f3b6a,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658d000,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6588000,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6588000,c6588018,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f500,e8132d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f500,e8132d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8132d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100101 td 0xc6120480
sched_switch(c6120480,0,104,191,b85f1fb2,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6120480,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6588000,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6588000,c6588018,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f500,e812fd38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f500,e812fd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe812fd70, ebp = 0 ---
       
Tracing command kernel pid 0 tid 100100 td 0xc61206c0
sched_switch(c61206c0,0,104,191,b85ec786,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c61206c0,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6588000,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6588000,c6588018,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f500,e8120d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f500,e8120d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8120d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100099 td 0xc6120900
sched_switch(c6120900,0,104,191,b85f9ab6,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6120900,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6588000,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6588000,c6588018,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f500,e811dd38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f500,e811dd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe811dd70, ebp = 0 ---

Tracing command kernel pid 0 tid 100098 td 0xc658ed80
sched_switch(c658ed80,0,104,191,89d1724a,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658ed80,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6587e80,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6587e80,c6587e98,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f4f0,e811ad38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f4f0,e811ad38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe811ad70, ebp = 0 ---

Tracing command kernel pid 0 tid 100097 td 0xc658eb40
sched_switch(c658eb40,0,104,191,89a3cffe,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658eb40,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6587e80,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6587e80,c6587e98,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f4f0,e8117d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f4f0,e8117d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8117d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100096 td 0xc658d6c0
sched_switch(c658d6c0,0,104,191,89838012,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658d6c0,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6587e80,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6587e80,c6587e98,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f4f0,e812cd38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f4f0,e812cd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe812cd70, ebp = 0 ---

Tracing command kernel pid 0 tid 100095 td 0xc658d900
sched_switch(c658d900,0,104,191,8af49d86,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658d900,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6587e80,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6587e80,c6587e98,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f4f0,e8129d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f4f0,e8129d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8129d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100094 td 0xc658db40
sched_switch(c658db40,0,104,191,8ab29942,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658db40,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6587e80,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6587e80,c6587e98,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f4f0,e8126d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f4f0,e8126d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8126d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100093 td 0xc658dd80
sched_switch(c658dd80,0,104,191,8a832bb8,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658dd80,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6587e80,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6587e80,c6587e98,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f4f0,e8123d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f4f0,e8123d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8123d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100092 td 0xc658e000
sched_switch(c658e000,0,104,191,8a1a880e,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658e000,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6587e80,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6587e80,c6587e98,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f4f0,e8114d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f4f0,e8114d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8114d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100091 td 0xc658e240
sched_switch(c658e240,0,104,191,89f70072,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658e240,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6587e80,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6587e80,c6587e98,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f4f0,e8111d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f4f0,e8111d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8111d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100090 td 0xc658e480
sched_switch(c658e480,0,104,191,7c37d874,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658e480,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6587e40,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6587e40,c6587e58,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f4d0,e810ed38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f4d0,e810ed38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe810ed70, ebp = 0 ---

Tracing command kernel pid 0 tid 100089 td 0xc658e6c0
sched_switch(c658e6c0,0,104,191,b9c3495c,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658e6c0,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6587e00,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6587e00,c6587e18,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f490,e815cd38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f490,e815cd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe815cd70, ebp = 0 ---

Tracing command kernel pid 0 tid 100088 td 0xc658e900
sched_switch(c658e900,0,104,191,b9838a5a,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658e900,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6587dc0,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6587dc0,c6587dd8,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f2b0,e815fd38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f2b0,e815fd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe815fd70, ebp = 0 ---

Tracing command kernel pid 0 tid 100046 td 0xc5b05000
sched_switch(c5b05000,0,104,191,d30d63e4,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5b05000,0,c0c7d8d5,24b,c5b05000,...) at sleepq_switch+0x15f
sleepq_wait(c5b08540,0,c0c7a109,c0c6f250,0,...) at sleepq_wait+0x63
msleep_spin(c5b08540,c5b08558,c0c6f250,0,c0c77c84,...) at msleep_spin+0x21d
taskqueue_thread_loop(c5b7249c,c5585d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0x94
fork_exit(c08bd6f0,c5b7249c,c5585d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5585d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100044 td 0xc5b05480
sched_switch(c5b05480,0,104,191,6b34f380,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5b05480,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c5abec80,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c5abec80,c5abec98,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5b61074,c557bd38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5b61074,c557bd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc557bd70, ebp = 0 ---

Tracing command kernel pid 0 tid 100021 td 0xc5a85480
sched_switch(c5a85480,0,104,191,aa033aea,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5a85480,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c5a5ee80,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c5a5ee80,c5a5ee98,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c0de1288,c5504d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c0de1288,c5504d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5504d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100017 td 0xc5971240
sched_switch(c5971240,0,104,191,91e1f390,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5971240,0,c0c7d8d5,24b,c5971240,...) at sleepq_switch+0x15f
sleepq_wait(c5a5f380,0,c0c7a109,c0c6f250,0,...) at sleepq_wait+0x63
msleep_spin(c5a5f380,c5a5f398,c0c6f250,0,c0c77c84,...) at msleep_spin+0x21d
taskqueue_thread_loop(c0d9dee0,c54f8d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0x94
fork_exit(c08bd6f0,c0d9dee0,c54f8d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54f8d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100016 td 0xc5971480
sched_switch(c5971480,0,104,191,91e1d060,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5971480,0,c0c7d8d5,24b,c5971480,...) at sleepq_switch+0x15f
sleepq_wait(c5a5f380,0,c0c7a109,c0c6f250,0,...) at sleepq_wait+0x63
msleep_spin(c5a5f380,c5a5f398,c0c6f250,0,c0c77c84,...) at msleep_spin+0x21d
taskqueue_thread_loop(c0d9dee0,c54f5d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0x94
fork_exit(c08bd6f0,c0d9dee0,c54f5d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54f5d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100015 td 0xc59716c0
sched_switch(c59716c0,0,104,191,91e1ab74,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c59716c0,0,c0c7d8d5,24b,c59716c0,...) at sleepq_switch+0x15f
sleepq_wait(c5a5f380,0,c0c7a109,c0c6f250,0,...) at sleepq_wait+0x63
msleep_spin(c5a5f380,c5a5f398,c0c6f250,0,c0c77c84,...) at msleep_spin+0x21d
taskqueue_thread_loop(c0d9dee0,c54f2d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0x94
fork_exit(c08bd6f0,c0d9dee0,c54f2d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54f2d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100014 td 0xc5971900
sched_switch(c5971900,0,104,191,91e061e0,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5971900,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c5a5f3c0,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c5a5f3c0,c5a5f3d8,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c0dcd044,c54efd38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c0dcd044,c54efd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54efd70, ebp = 0 ---
       
Tracing command kernel pid 0 tid 100010 td 0xc59b5240
sched_switch(c59b5240,0,104,191,aa3f995a,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c59b5240,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c5955c40,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c5955c40,c5955c58,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c0ddfd20,c54e3d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c0ddfd20,c54e3d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54e3d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100000 td 0xc0dcca50
sched_switch(c0dcca50,0,104,191,8776839a,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,44,...) at mi_switch+0x200
sleepq_switch(c0dcca50,0,c0c7d8d5,26e,0,...) at sleepq_switch+0x15f
sleepq_timedwait(c0dcc7a0,44,c0c7b93e,0,0,...) at sleepq_timedwait+0x6b
_sleep(c0dcc7a0,0,44,c0c7b93e,2710,...) at _sleep+0x339
scheduler(0,141ec00,141ec00,141e000,1425000,...) at scheduler+0x23e
mi_startup() at mi_startup+0x96
begin() at begin+0x2c


----------------------------------------------------------------------
 Head of dmesg

WARNING: console at 0xc0d5a000 has no name
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2009 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.0-RC2 #4: Sat Oct 31 02:28:58 UTC 2009
    root@...:/usr/obj/usr/src/sys/WITHDDB
WARNING: WITNESS option enabled, expect reduced performance.
WARNING: DIAGNOSTIC option enabled, expect reduced performance.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Pentium(R) 4 CPU 2.40GHz (2400.10-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0xf27  Stepping = 7
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x4400<CNXT-ID,xTPR>
real memory  = 2147483648 (2048 MB)
avail memory = 2086248448 (1989 MB)


----------------------------------------------------------------------
 Kernel Config

include GENERIC

ident   WITHDDB

options KDB
options DDB
options BREAK_TO_DEBUGGER

makeoptions        DEBUG=-g
options     INVARIANTS
options     INVARIANT_SUPPORT
options     WITNESS
options     DEBUG_LOCKS
options     DEBUG_VFS_LOCKS
options     DIAGNOSTIC


_______________________________________________
freebsd-fs@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-unsubscribe@..."

[Borja Marcos-2]

On Oct 31, 2009, at 12:28 AM, Pawel Jakub Dawidek wrote:

> On Thu, Oct 29, 2009 at 03:52:45PM -0600, Stef Walter wrote:
>> Borja Marcos wrote:
>>> I've been sending some alltraces to pjd about this easy to reproduce
>>> problem. I am using zfs send/zfs receive to replicate a dataset  
>>> from one
>>> server to another. At 1 minute intervals, an incremental snapshot is
>>> sent to update the dataset copy. If there is reading activity on the
>>> dataset copy, a deadlock can happen rendering ZFS and all the FS
>>> subsystem unusable. I've tried with 8.0RC2 and it still happens.
>
> I was able to reproduce it, but I don't have fix yet.
>
>> FWIW, another (or the same) zfs recv deadlock I've been trying to  
>> get to
>> the bottom of:
>>
>> http://lists.freebsd.org/pipermail/freebsd-fs/2009-October/ 
>> 006999.html
>
> Could you guys recompile your kernel after uncommenting line:
>
> #CFLAGS+=-DDEBUG=1
>
> in sys/modules/zfs/Makefile?
>
> You should see panic on assertion instead of deadlock, I believe.

Aye aye, Sir!

What do you want me to collect? Just an alltrace? Anything else?

If you want the VMWare images to examine the panic by yourself just  
let me know.





Borja.

_______________________________________________
freebsd-fs@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-unsubscribe@..."

[Borja Marcos-2]

On Oct 31, 2009, at 12:28 AM, Pawel Jakub Dawidek wrote:

> Could you guys recompile your kernel after uncommenting line:
>
> #CFLAGS+=-DDEBUG=1
>
> in sys/modules/zfs/Makefile?
>
> You should see panic on assertion instead of deadlock, I believe.

I've started the test. For now I see a bunch of LOR complaints.

# dmesg
Copyright (c) 1992-2009 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.0-RC2 #0: Mon Oct 26 14:40:09 CET 2009
     root@:/pool/newsrc/obj/pool/newsrc/src/sys/DEBUG
WARNING: WITNESS option enabled, expect reduced performance.
WARNING: DIAGNOSTIC option enabled, expect reduced performance.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Core(TM)2 Duo CPU     T8100  @ 2.10GHz (2094.43-MHz K8-
class CPU)
   Origin = "GenuineIntel"  Id = 0x10676  Stepping = 6
   
Features
=
0xfebfbff
<
FPU
,VME
,DE
,PSE
,TSC
,MSR
,PAE
,MCE
,CX8
,APIC
,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS>
   Features2=0x80082201<SSE3,SSSE3,CX16,SSE4.1,<b31>>
   AMD Features=0x20100800<SYSCALL,NX,LM>
   AMD Features2=0x1<LAHF>
   TSC: P-state invariant
real memory  = 780140544 (744 MB)
avail memory = 730873856 (697 MB)
ACPI APIC Table: <PTLTD   APIC  >
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <INTEL 440BX> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
isab0: <PCI-ISA bridge> at device 7.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX4 UDMA33 controller> port  
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x10c0-0x10cf at device 7.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
pci0: <bridge> at device 7.3 (no driver attached)
pci0: <base peripheral> at device 7.7 (no driver attached)
vgapci0: <VGA-compatible display> port 0x10d0-0x10df mem  
0xd0000000-0xd7ffffff,0xd8000000-0xd87fffff irq 16 at device 15.0 on  
pci0
mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x1400-0x14ff mem  
0xd8820000-0xd883ffff,0xd8800000-0xd881ffff irq 17 at device 16.0 on  
pci0
mpt0: [ITHREAD]
mpt0: MPI Version=1.2.0.0
pcib2: <ACPI PCI-PCI bridge> at device 17.0 on pci0
pci2: <ACPI PCI bus> on pcib2
em0: <Intel(R) PRO/1000 Network Connection 6.9.14> port 0x2000-0x203f  
mem 0xd8940000-0xd895ffff,0xd8900000-0xd890ffff irq 18 at device 0.0  
on pci2
em0: Memory Access and/or Bus Master bits were not set!
em0: [FILTER]
em0: Ethernet address: 00:0c:29:fd:ab:03
em1: <Intel(R) PRO/1000 Network Connection 6.9.14> port 0x2040-0x207f  
mem 0xd8960000-0xd897ffff,0xd8910000-0xd891ffff irq 19 at device 1.0  
on pci2
em1: Memory Access and/or Bus Master bits were not set!
em1: [FILTER]
em1: Ethernet address: 00:0c:29:fd:ab:f9
pci2: <multimedia, audio> at device 2.0 (no driver attached)
em2: <Intel(R) PRO/1000 Network Connection 6.9.14> port 0x20c0-0x20ff  
mem 0xd8980000-0xd899ffff,0xd8920000-0xd892ffff irq 17 at device 3.0  
on pci2
em2: Memory Access and/or Bus Master bits were not set!
em2: [FILTER]
em2: Ethernet address: 00:0c:29:fd:ab:0d
em3: <Intel(R) PRO/1000 Network Connection 6.9.14> port 0x2400-0x243f  
mem 0xd89a0000-0xd89bffff,0xd8930000-0xd893ffff irq 19 at device 5.0  
on pci2
em3: Memory Access and/or Bus Master bits were not set!
em3: [FILTER]
em3: Ethernet address: 00:0c:29:fd:ab:17
pcib3: <ACPI PCI-PCI bridge> at device 21.0 on pci0
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> at device 21.1 on pci0
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 21.2 on pci0
pci5: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> at device 21.3 on pci0
pci6: <ACPI PCI bus> on pcib6
pcib7: <ACPI PCI-PCI bridge> at device 21.4 on pci0
pci7: <ACPI PCI bus> on pcib7
pcib8: <ACPI PCI-PCI bridge> at device 21.5 on pci0
pci8: <ACPI PCI bus> on pcib8
pcib9: <ACPI PCI-PCI bridge> at device 21.6 on pci0
pci9: <ACPI PCI bus> on pcib9
pcib10: <ACPI PCI-PCI bridge> at device 21.7 on pci0
pci10: <ACPI PCI bus> on pcib10
pcib11: <ACPI PCI-PCI bridge> at device 22.0 on pci0
pci11: <ACPI PCI bus> on pcib11
pcib12: <ACPI PCI-PCI bridge> at device 22.1 on pci0
pci12: <ACPI PCI bus> on pcib12
pcib13: <ACPI PCI-PCI bridge> at device 22.2 on pci0
pci13: <ACPI PCI bus> on pcib13
pcib14: <ACPI PCI-PCI bridge> at device 22.3 on pci0
pci14: <ACPI PCI bus> on pcib14
pcib15: <ACPI PCI-PCI bridge> at device 22.4 on pci0
pci15: <ACPI PCI bus> on pcib15
pcib16: <ACPI PCI-PCI bridge> at device 22.5 on pci0
pci16: <ACPI PCI bus> on pcib16
pcib17: <ACPI PCI-PCI bridge> at device 22.6 on pci0
pci17: <ACPI PCI bus> on pcib17
pcib18: <ACPI PCI-PCI bridge> at device 22.7 on pci0
pci18: <ACPI PCI bus> on pcib18
pcib19: <ACPI PCI-PCI bridge> at device 23.0 on pci0
pci19: <ACPI PCI bus> on pcib19
pcib20: <ACPI PCI-PCI bridge> at device 23.1 on pci0
pci20: <ACPI PCI bus> on pcib20
pcib21: <ACPI PCI-PCI bridge> at device 23.2 on pci0
pci21: <ACPI PCI bus> on pcib21
pcib22: <ACPI PCI-PCI bridge> at device 23.3 on pci0
pci22: <ACPI PCI bus> on pcib22
pcib23: <ACPI PCI-PCI bridge> at device 23.4 on pci0
pci23: <ACPI PCI bus> on pcib23
pcib24: <ACPI PCI-PCI bridge> at device 23.5 on pci0
pci24: <ACPI PCI bus> on pcib24
pcib25: <ACPI PCI-PCI bridge> at device 23.6 on pci0
pci25: <ACPI PCI bus> on pcib25
pcib26: <ACPI PCI-PCI bridge> at device 23.7 on pci0
pci26: <ACPI PCI bus> on pcib26
pcib27: <ACPI PCI-PCI bridge> at device 24.0 on pci0
pci27: <ACPI PCI bus> on pcib27
pcib28: <ACPI PCI-PCI bridge> at device 24.1 on pci0
pci28: <ACPI PCI bus> on pcib28
pcib29: <ACPI PCI-PCI bridge> at device 24.2 on pci0
pci29: <ACPI PCI bus> on pcib29
pcib30: <ACPI PCI-PCI bridge> at device 24.3 on pci0
pci30: <ACPI PCI bus> on pcib30
pcib31: <ACPI PCI-PCI bridge> at device 24.4 on pci0
pci31: <ACPI PCI bus> on pcib31
pcib32: <ACPI PCI-PCI bridge> at device 24.5 on pci0
pci32: <ACPI PCI bus> on pcib32
pcib33: <ACPI PCI-PCI bridge> at device 24.6 on pci0
pci33: <ACPI PCI bus> on pcib33
pcib34: <ACPI PCI-PCI bridge> at device 24.7 on pci0
pci34: <ACPI PCI bus> on pcib34
acpi_acad0: <AC Adapter> on acpi0
acpi_button0: <Sleep Button> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: [ITHREAD]
psm0: model IntelliMouse, device ID 3
ppc0: <Parallel port> port 0x378-0x37f irq 7 on acpi0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppc0: [ITHREAD]
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
plip0: [ITHREAD]
lpt0: <Printer> on ppbus0
lpt0: [ITHREAD]
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
uart1: [FILTER]
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on  
acpi0
fdc0: [FILTER]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
cpu0: <ACPI CPU> on acpi0
acpi_throttle0: <ACPI CPU Throttling> on cpu0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xca000-0xcafff,
0xcb000-0xcbfff,0xcc000-0xccfff,0xcd000-0xcdfff,0xdc000-0xdffff,
0xe4000-0xe7fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on  
isa0
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is  
present;
             to enable, add "vfs.zfs.prefetch_disable=0" to /boot/
loader.conf.
ZFS WARNING: Recommended minimum kmem_size is 512MB; expect unstable  
behavior.
              Consider tuning vm.kmem_size and vm.kmem_size_max
              in /boot/loader.conf.
ZFS filesystem version 13
ZFS storage pool version 13
Timecounter "TSC" frequency 2094428941 Hz quality 800
Timecounters tick every 10.000 msec
Waiting 5 seconds for SCSI devices to settle
acd0: CDROM <VMware Virtual IDE CDROM Drive/00000001> at ata1-master  
UDMA33
da0 at mpt0 bus 0 target 0 lun 0
da0: <VMware, VMware Virtual S 1.0> Fixed Direct Access SCSI-2 device
da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
da0: Command Queueing enabled
da0: 8192MB (16777216 512 byte sectors: 255H 63S/T 1044C)
da1 at mpt0 bus 0 target 1 lun 0
da1: <VMware, VMware Virtual S 1.0> Fixed Direct Access SCSI-2 device
da1: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
da1: Command Queueing enabled
da1: 4096MB (8388608 512 byte sectors: 255H 63S/T 522C)
da2 at mpt0 bus 0 target 2 lun 0
da2: <VMware, VMware Virtual S 1.0> Fixed Direct Access SCSI-2 device
da2: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
da2: Command Queueing enabled
da2: 4096MB (8388608 512 byte sectors: 255H 63S/T 522C)
WARNING: WITNESS option enabled, expect reduced performance.
WARNING: DIAGNOSTIC option enabled, expect reduced performance.
Trying to mount root from ufs:/dev/da0s1a
Expensive timeout(9) function: 0xffffffff8046e5b0(0xffffffff80e4f5e0)  
0.002432990 s
Expensive timeout(9) function: 0xffffffff8031ffc0(0xffffff8000277000)  
0.008108572 s
Expensive timeout(9) function: 0xffffffff8031ffc0(0xffffff8000265000)  
0.035305324 s
lock order reversal:
  1st 0xffffff800f0c3cc8 bufwait (bufwait) @ /pool/newsrc/src/sys/kern/
vfs_bio.c:2559
  2nd 0xffffff000c721a00 dirhash (dirhash) @ /pool/newsrc/src/sys/ufs/
ufs/ufs_dirhash.c:285
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_xlock() at _sx_xlock+0x55
ufsdirhash_acquire() at ufsdirhash_acquire+0x44
ufsdirhash_remove() at ufsdirhash_remove+0x16
ufs_dirremove() at ufs_dirremove+0x181
ufs_remove() at ufs_remove+0x92
VOP_REMOVE_APV() at VOP_REMOVE_APV+0xd7
kern_unlinkat() at kern_unlinkat+0x252
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (10, FreeBSD ELF64, unlink), rip = 0x80072be5c, rsp =  
0x7fffffffe8b8, rbp = 0x800902300 ---
Waiting (max 60 seconds) for system process `vnlru' to stop...done
Waiting (max 60 seconds) for system process `bufdaemon' to stop...done
Waiting (max 60 seconds) for system process `syncer' to stop...
Syncing disks, vnodes remaining...2 2 1 0 0 done
All buffers synced.
lock order reversal:
  1st 0xffffff0002558308 ufs (ufs) @ /pool/newsrc/src/sys/kern/
vfs_mount.c:1200
  2nd 0xffffff0002558cc8 devfs (devfs) @ /pool/newsrc/src/sys/kern/
vfs_subr.c:2083
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
__lockmgr_args() at __lockmgr_args+0xd03
vop_stdlock() at vop_stdlock+0x39
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
vget() at vget+0x7b
devfs_allocv() at devfs_allocv+0x100
devfs_root() at devfs_root+0x48
dounmount() at dounmount+0x474
vfs_unmountall() at vfs_unmountall+0x54
boot() at boot+0x7d3
reboot() at reboot+0x68
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (55, FreeBSD ELF64, reboot), rip = 0x40892c, rsp =  
0x7fffffffe738, rbp = 0x4023d0 ---
Uptime: 2h1m35s
Rebooting...
Copyright (c) 1992-2009 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.0-RC2 #2: Mon Nov  2 12:08:37 CET 2009
     root@:/pool/newsrc/obj/pool/newsrc/src/sys/DEBUG
WARNING: WITNESS option enabled, expect reduced performance.
WARNING: DIAGNOSTIC option enabled, expect reduced performance.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Core(TM)2 Duo CPU     T8100  @ 2.10GHz (2094.42-MHz K8-
class CPU)
   Origin = "GenuineIntel"  Id = 0x10676  Stepping = 6
   
Features
=
0xfebfbff
<
FPU
,VME
,DE
,PSE
,TSC
,MSR
,PAE
,MCE
,CX8
,APIC
,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS>
   Features2=0x80082201<SSE3,SSSE3,CX16,SSE4.1,<b31>>
   AMD Features=0x20100800<SYSCALL,NX,LM>
   AMD Features2=0x1<LAHF>
   TSC: P-state invariant
real memory  = 780140544 (744 MB)
avail memory = 730472448 (696 MB)
ACPI APIC Table: <PTLTD   APIC  >
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <INTEL 440BX> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
isab0: <PCI-ISA bridge> at device 7.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX4 UDMA33 controller> port  
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x10c0-0x10cf at device 7.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
pci0: <bridge> at device 7.3 (no driver attached)
pci0: <base peripheral> at device 7.7 (no driver attached)
vgapci0: <VGA-compatible display> port 0x10d0-0x10df mem  
0xd0000000-0xd7ffffff,0xd8000000-0xd87fffff irq 16 at device 15.0 on  
pci0
mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x1400-0x14ff mem  
0xd8820000-0xd883ffff,0xd8800000-0xd881ffff irq 17 at device 16.0 on  
pci0
mpt0: [ITHREAD]
mpt0: MPI Version=1.2.0.0
pcib2: <ACPI PCI-PCI bridge> at device 17.0 on pci0
pci2: <ACPI PCI bus> on pcib2
em0: <Intel(R) PRO/1000 Network Connection 6.9.14> port 0x2000-0x203f  
mem 0xd8940000-0xd895ffff,0xd8900000-0xd890ffff irq 18 at device 0.0  
on pci2
em0: Memory Access and/or Bus Master bits were not set!
em0: [FILTER]
em0: Ethernet address: 00:0c:29:fd:ab:03
em1: <Intel(R) PRO/1000 Network Connection 6.9.14> port 0x2040-0x207f  
mem 0xd8960000-0xd897ffff,0xd8910000-0xd891ffff irq 19 at device 1.0  
on pci2
em1: Memory Access and/or Bus Master bits were not set!
em1: [FILTER]
em1: Ethernet address: 00:0c:29:fd:ab:f9
pci2: <multimedia, audio> at device 2.0 (no driver attached)
em2: <Intel(R) PRO/1000 Network Connection 6.9.14> port 0x20c0-0x20ff  
mem 0xd8980000-0xd899ffff,0xd8920000-0xd892ffff irq 17 at device 3.0  
on pci2
em2: Memory Access and/or Bus Master bits were not set!
em2: [FILTER]
em2: Ethernet address: 00:0c:29:fd:ab:0d
em3: <Intel(R) PRO/1000 Network Connection 6.9.14> port 0x2400-0x243f  
mem 0xd89a0000-0xd89bffff,0xd8930000-0xd893ffff irq 19 at device 5.0  
on pci2
em3: Memory Access and/or Bus Master bits were not set!
em3: [FILTER]
em3: Ethernet address: 00:0c:29:fd:ab:17
pcib3: <ACPI PCI-PCI bridge> at device 21.0 on pci0
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> at device 21.1 on pci0
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 21.2 on pci0
pci5: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> at device 21.3 on pci0
pci6: <ACPI PCI bus> on pcib6
pcib7: <ACPI PCI-PCI bridge> at device 21.4 on pci0
pci7: <ACPI PCI bus> on pcib7
pcib8: <ACPI PCI-PCI bridge> at device 21.5 on pci0
pci8: <ACPI PCI bus> on pcib8
pcib9: <ACPI PCI-PCI bridge> at device 21.6 on pci0
pci9: <ACPI PCI bus> on pcib9
pcib10: <ACPI PCI-PCI bridge> at device 21.7 on pci0
pci10: <ACPI PCI bus> on pcib10
pcib11: <ACPI PCI-PCI bridge> at device 22.0 on pci0
pci11: <ACPI PCI bus> on pcib11
pcib12: <ACPI PCI-PCI bridge> at device 22.1 on pci0
pci12: <ACPI PCI bus> on pcib12
pcib13: <ACPI PCI-PCI bridge> at device 22.2 on pci0
pci13: <ACPI PCI bus> on pcib13
pcib14: <ACPI PCI-PCI bridge> at device 22.3 on pci0
pci14: <ACPI PCI bus> on pcib14
pcib15: <ACPI PCI-PCI bridge> at device 22.4 on pci0
pci15: <ACPI PCI bus> on pcib15
pcib16: <ACPI PCI-PCI bridge> at device 22.5 on pci0
pci16: <ACPI PCI bus> on pcib16
pcib17: <ACPI PCI-PCI bridge> at device 22.6 on pci0
pci17: <ACPI PCI bus> on pcib17
pcib18: <ACPI PCI-PCI bridge> at device 22.7 on pci0
pci18: <ACPI PCI bus> on pcib18
pcib19: <ACPI PCI-PCI bridge> at device 23.0 on pci0
pci19: <ACPI PCI bus> on pcib19
pcib20: <ACPI PCI-PCI bridge> at device 23.1 on pci0
pci20: <ACPI PCI bus> on pcib20
pcib21: <ACPI PCI-PCI bridge> at device 23.2 on pci0
pci21: <ACPI PCI bus> on pcib21
pcib22: <ACPI PCI-PCI bridge> at device 23.3 on pci0
pci22: <ACPI PCI bus> on pcib22
pcib23: <ACPI PCI-PCI bridge> at device 23.4 on pci0
pci23: <ACPI PCI bus> on pcib23
pcib24: <ACPI PCI-PCI bridge> at device 23.5 on pci0
pci24: <ACPI PCI bus> on pcib24
pcib25: <ACPI PCI-PCI bridge> at device 23.6 on pci0
pci25: <ACPI PCI bus> on pcib25
pcib26: <ACPI PCI-PCI bridge> at device 23.7 on pci0
pci26: <ACPI PCI bus> on pcib26
pcib27: <ACPI PCI-PCI bridge> at device 24.0 on pci0
pci27: <ACPI PCI bus> on pcib27
pcib28: <ACPI PCI-PCI bridge> at device 24.1 on pci0
pci28: <ACPI PCI bus> on pcib28
pcib29: <ACPI PCI-PCI bridge> at device 24.2 on pci0
pci29: <ACPI PCI bus> on pcib29
pcib30: <ACPI PCI-PCI bridge> at device 24.3 on pci0
pci30: <ACPI PCI bus> on pcib30
pcib31: <ACPI PCI-PCI bridge> at device 24.4 on pci0
pci31: <ACPI PCI bus> on pcib31
pcib32: <ACPI PCI-PCI bridge> at device 24.5 on pci0
pci32: <ACPI PCI bus> on pcib32
pcib33: <ACPI PCI-PCI bridge> at device 24.6 on pci0
pci33: <ACPI PCI bus> on pcib33
pcib34: <ACPI PCI-PCI bridge> at device 24.7 on pci0
pci34: <ACPI PCI bus> on pcib34
acpi_acad0: <AC Adapter> on acpi0
acpi_button0: <Sleep Button> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: [ITHREAD]
psm0: model IntelliMouse, device ID 3
ppc0: <Parallel port> port 0x378-0x37f irq 7 on acpi0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppc0: [ITHREAD]
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
plip0: [ITHREAD]
lpt0: <Printer> on ppbus0
lpt0: [ITHREAD]
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
uart1: [FILTER]
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on  
acpi0
fdc0: [FILTER]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
cpu0: <ACPI CPU> on acpi0
acpi_throttle0: <ACPI CPU Throttling> on cpu0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xca000-0xcafff,
0xcb000-0xcbfff,0xcc000-0xccfff,0xcd000-0xcdfff,0xdc000-0xdffff,
0xe4000-0xe7fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on  
isa0
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is  
present;
             to enable, add "vfs.zfs.prefetch_disable=0" to /boot/
loader.conf.
ZFS WARNING: Recommended minimum kmem_size is 512MB; expect unstable  
behavior.
              Consider tuning vm.kmem_size and vm.kmem_size_max
              in /boot/loader.conf.
ZFS filesystem version 13
ZFS storage pool version 13
Timecounter "TSC" frequency 2094424190 Hz quality 800
Timecounters tick every 10.000 msec
Waiting 5 seconds for SCSI devices to settle
acd0: CDROM <VMware Virtual IDE CDROM Drive/00000001> at ata1-master  
UDMA33
da0 at mpt0 bus 0 target 0 lun 0
da0: <VMware, VMware Virtual S 1.0> Fixed Direct Access SCSI-2 device
da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
da0: Command Queueing enabled
da0: 8192MB (16777216 512 byte sectors: 255H 63S/T 1044C)
da1 at mpt0 bus 0 target 1 lun 0
da1: <VMware, VMware Virtual S 1.0> Fixed Direct Access SCSI-2 device
da1: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
da1: Command Queueing enabled
da1: 4096MB (8388608 512 byte sectors: 255H 63S/T 522C)
Expensive timeout(9) function: 0xffffffff806fd940(0) 0.002228495 s
da2 at mpt0 bus 0 target 2 lun 0
da2: <VMware, VMware Virtual S 1.0> Fixed Direct Access SCSI-2 device
da2: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
da2: Command Queueing enabled
da2: 4096MB (8388608 512 byte sectors: 255H 63S/T 522C)
WARNING: WITNESS option enabled, expect reduced performance.
WARNING: DIAGNOSTIC option enabled, expect reduced performance.
Trying to mount root from ufs:/dev/da0s1a
lock order reversal:
  1st 0xffffff000284d010 buf->b_lock (buf->b_lock) @ /pool/newsrc/src/
sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:
2506
  2nd 0xffffff0002851058 db->db_mtx (db->db_mtx) @ /pool/newsrc/src/
sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/
dbuf.c:421
  3rd 0xffffff000284cdd8 buf->b_lock (buf->b_lock) @ /pool/newsrc/src/
sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:
3014
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_slock() at _sx_slock+0x55
arc_released() at arc_released+0x34
dbuf_set_data() at dbuf_set_data+0x62
dbuf_read_done() at dbuf_read_done+0x10b
arc_read_done() at arc_read_done+0x1d2
zio_done() at zio_done+0x308
zio_execute() at zio_execute+0xb1
arc_read_nolock() at arc_read_nolock+0x3d0
arc_read() at arc_read+0xaf
dbuf_read() at dbuf_read+0x62b
dbuf_findbp() at dbuf_findbp+0x19b
dbuf_hold_impl() at dbuf_hold_impl+0x164
dbuf_hold() at dbuf_hold+0x1b
dnode_hold_impl() at dnode_hold_impl+0xc5
dmu_buf_hold() at dmu_buf_hold+0x34
zap_lockdir() at zap_lockdir+0x6e
zap_lookup_norm() at zap_lookup_norm+0x45
zap_lookup() at zap_lookup+0x2e
dsl_pool_open() at dsl_pool_open+0xe3
spa_load() at spa_load+0x3a9
spa_open_common() at spa_open_common+0x12d
spa_get_stats() at spa_get_stats+0x42
zfs_ioc_pool_stats() at zfs_ioc_pool_stats+0x2c
zfsdev_ioctl() at zfsdev_ioctl+0x8d
devfs_ioctl_f() at devfs_ioctl_f+0x76
kern_ioctl() at kern_ioctl+0xc5
ioctl() at ioctl+0xfd
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (54, FreeBSD ELF64, ioctl), rip = 0x800fe874c, rsp =  
0x7fffffffd808, rbp = 0x801222140 ---
lock order reversal:
  1st 0xffffff0002850058 db->db_mtx (db->db_mtx) @ /pool/newsrc/src/
sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/
dbuf.c:549
  2nd 0xffffff00028580d8 dn->dn_mtx (dn->dn_mtx) @ /pool/newsrc/src/
sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/
dnode.c:1196
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
Expensive timeout(9) function: 0xffffffff80875e50(0xffffffff80e3ff80)  
0.002947581 s
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_xlock() at _sx_xlock+0x55
dnode_block_freed() at dnode_block_freed+0x8e
dbuf_read() at dbuf_read+0x155
dmu_buf_hold_array_by_dnode() at dmu_buf_hold_array_by_dnode+0x12a
dmu_read() at dmu_read+0x80
load_nvlist() at load_nvlist+0x85
spa_load() at spa_load+0x49a
spa_open_common() at spa_open_common+0x12d
spa_get_stats() at spa_get_stats+0x42
zfs_ioc_pool_stats() at zfs_ioc_pool_stats+0x2c
zfsdev_ioctl() at zfsdev_ioctl+0x8d
devfs_ioctl_f() at devfs_ioctl_f+0x76
kern_ioctl() at kern_ioctl+0xc5
ioctl() at ioctl+0xfd
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (54, FreeBSD ELF64, ioctl), rip = 0x800fe874c, rsp =  
0x7fffffffd808, rbp = 0x801222140 ---
lock order reversal:
  1st 0xffffff0002850e70 db->db_mtx (db->db_mtx) @ /pool/newsrc/src/
sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/
dnode_sync.c:381
  2nd 0xffffff000279e140 osi->os_lock (osi->os_lock) @ /pool/newsrc/
src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/
dnode.c:323
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_xlock() at _sx_xlock+0x55
dnode_destroy() at dnode_destroy+0xa6
dnode_buf_pageout() at dnode_buf_pageout+0xb2
dbuf_evict_user() at dbuf_evict_user+0x55
dbuf_clear() at dbuf_clear+0x5e
dnode_evict_dbufs() at dnode_evict_dbufs+0x98
dmu_objset_evict_dbufs() at dmu_objset_evict_dbufs+0x11c
dmu_objset_evict() at dmu_objset_evict+0xbf
dsl_pool_close() at dsl_pool_close+0x52
spa_unload() at spa_unload+0xb2
spa_load() at spa_load+0x4da
spa_open_common() at spa_open_common+0x12d
spa_get_stats() at spa_get_stats+0x42
zfs_ioc_pool_stats() at zfs_ioc_pool_stats+0x2c
zfsdev_ioctl() at zfsdev_ioctl+0x8d
devfs_ioctl_f() at devfs_ioctl_f+0x76
kern_ioctl() at kern_ioctl+0xc5
ioctl() at ioctl+0xfd
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (54, FreeBSD ELF64, ioctl), rip = 0x800fe874c, rsp =  
0x7fffffffd808, rbp = 0x801222140 ---
lock order reversal:
  1st 0xffffff0002850e70 db->db_mtx (db->db_mtx) @ /pool/newsrc/src/
sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/
dbuf.c:1116
  2nd 0xffffff0002881738 dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) @ /pool/
newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/
fs/zfs/dbuf.c:1120
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_xlock() at _sx_xlock+0x55
dbuf_dirty() at dbuf_dirty+0x892
dnode_setdirty() at dnode_setdirty+0x1a9
dbuf_dirty() at dbuf_dirty+0xa53
bplist_vacate() at bplist_vacate+0x4d
spa_sync() at spa_sync+0x297
txg_sync_thread() at txg_sync_thread+0x2d7
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80001ffd30, rbp = 0 ---
lock order reversal:
  1st 0xffffff00028f4438 dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) @ /pool/
newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/
fs/zfs/dbuf.c:1905
  2nd 0xffffff000255e2f0 spa->spa_sync_bplist.bpl_lock (spa-
 >spa_sync_bplist.bpl_lock) @ /pool/newsrc/src/sys/modules/zfs/../../
cddl/contrib/opensolaris/uts/common/fs/zfs/bplist.c:235
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_xlock() at _sx_xlock+0x55
bplist_enqueue_deferred() at bplist_enqueue_deferred+0x47
zio_free() at zio_free+0x105
arc_free() at arc_free+0x11c
dsl_dataset_block_kill() at dsl_dataset_block_kill+0x483
dbuf_write() at dbuf_write+0x24c
dbuf_sync_list() at dbuf_sync_list+0x3eb
dbuf_sync_list() at dbuf_sync_list+0x17f
dnode_sync() at dnode_sync+0xc12
dmu_objset_sync() at dmu_objset_sync+0x134
dsl_pool_sync() at dsl_pool_sync+0x200
spa_sync() at spa_sync+0x35e
txg_sync_thread() at txg_sync_thread+0x2d7
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80001ffd30, rbp = 0 ---
lock order reversal:
  1st 0xffffff0002935098 zfs (zfs) @ /pool/newsrc/src/sys/modules/
zfs/../../cddl/contrib/opensolaris/uts/common/fs/gfs.c:437
  2nd 0xffffff000266d310 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i])  
@ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/
common/fs/zfs/zfs_znode.c:863
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_xlock() at _sx_xlock+0x55
zfs_zget() at zfs_zget+0x23a
zfs_root() at zfs_root+0x50
zfsctl_create() at zfsctl_create+0x82
zfs_mount() at zfs_mount+0x7ef
vfs_donmount() at vfs_donmount+0xcde
nmount() at nmount+0x63
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (378, FreeBSD ELF64, nmount), rip = 0x800f4a4dc, rsp =  
0x7fffffffced8, rbp = 0x7fffffffcef8 ---
lock order reversal:
  1st 0xffffff000292c078 zp->z_name_lock (zp->z_name_lock) @ /pool/
newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/
fs/zfs/zfs_dir.c:212
  2nd 0xffffff000266d330 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i])  
@ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/
common/fs/zfs/zfs_znode.c:863
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_xlock() at _sx_xlock+0x55
zfs_zget() at zfs_zget+0x23a
zfs_dirent_lock() at zfs_dirent_lock+0x4a0
zfs_dirlook() at zfs_dirlook+0x90
zfs_lookup() at zfs_lookup+0x257
zfs_freebsd_lookup() at zfs_freebsd_lookup+0x8d
VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xaf
vfs_cache_lookup() at vfs_cache_lookup+0xf0
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7
lookup() at lookup+0x2eb
namei() at namei+0x4a9
kern_statat_vnhook() at kern_statat_vnhook+0x8f
kern_statat() at kern_statat+0x15
lstat() at lstat+0x2a
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (190, FreeBSD ELF64, lstat), rip = 0x800fd94fc, rsp =  
0x7fffffffcf38, rbp = 0x7fffffffd3d0 ---
lock order reversal:
  1st 0xffffff000266d210 zfsvfs->z_teardown_inactive_lock (zfsvfs-
 >z_teardown_inactive_lock) @ /pool/newsrc/src/sys/modules/zfs/../../
cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:3724
  2nd 0xffffff000266d330 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i])  
@ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/
common/fs/zfs/zfs_znode.c:1023
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_xlock() at _sx_xlock+0x55
zfs_zinactive() at zfs_zinactive+0x8d
zfs_inactive() at zfs_inactive+0x7e
zfs_freebsd_inactive() at zfs_freebsd_inactive+0x1a
VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0xb5
vinactive() at vinactive+0x90
vput() at vput+0x250
kern_statat_vnhook() at kern_statat_vnhook+0xfa
kern_statat() at kern_statat+0x15
lstat() at lstat+0x2a
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (190, FreeBSD ELF64, lstat), rip = 0x800fd94fc, rsp =  
0x7fffffffcf38, rbp = 0x7fffffffd3d0 ---
lock order reversal:
  1st 0xffffff0002914210 zfsvfs->z_teardown_inactive_lock (zfsvfs-
 >z_teardown_inactive_lock) @ /pool/newsrc/src/sys/modules/zfs/../../
cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c:915
  2nd 0xffffff000278f4f8 ds->ds_rwlock (ds->ds_rwlock) @ /pool/newsrc/
src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/
dsl_dataset.c:2864
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_xlock() at _sx_xlock+0x55
dsl_dataset_clone_swap() at dsl_dataset_clone_swap+0x5a
dmu_recv_end() at dmu_recv_end+0x94
zfs_ioc_recv() at zfs_ioc_recv+0x29d
zfsdev_ioctl() at zfsdev_ioctl+0x8d
devfs_ioctl_f() at devfs_ioctl_f+0x76
kern_ioctl() at kern_ioctl+0xc5
ioctl() at ioctl+0xfd
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (54, FreeBSD ELF64, ioctl), rip = 0x800fe874c, rsp =  
0x7fffffffbbf8, rbp = 0x7fffffffc930 ---
lock order reversal:
  1st 0xffffff000278f438 ds->ds_deadlist.bpl_lock (ds-
 >ds_deadlist.bpl_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/
contrib/opensolaris/uts/common/fs/zfs/bplist.c:331
  2nd 0xffffff0002857b88 dn->dn_struct_rwlock (dn->dn_struct_rwlock)  
@ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/
common/fs/zfs/dnode.c:130
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_slock() at _sx_slock+0x55
dnode_verify() at dnode_verify+0x70
dnode_hold_impl() at dnode_hold_impl+0x73
dmu_bonus_hold() at dmu_bonus_hold+0x31
bplist_hold() at bplist_hold+0x48
bplist_space_birthrange() at bplist_space_birthrange+0xb1
dsl_dataset_clone_swap_sync() at dsl_dataset_clone_swap_sync+0xee
dsl_sync_task_group_sync() at dsl_sync_task_group_sync+0x173
dsl_pool_sync() at dsl_pool_sync+0x122
spa_sync() at spa_sync+0x35e
txg_sync_thread() at txg_sync_thread+0x2d7
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80001ffd30, rbp = 0 ---
lock order reversal:
  1st 0xffffff000278f438 ds->ds_deadlist.bpl_lock (ds-
 >ds_deadlist.bpl_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/
contrib/opensolaris/uts/common/fs/zfs/bplist.c:331
  2nd 0xffffff0002ee9888 dn->dn_mtx (dn->dn_mtx) @ /pool/newsrc/src/
sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/
dnode.c:624
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_xlock() at _sx_xlock+0x55
dnode_hold_impl() at dnode_hold_impl+0x184
dmu_bonus_hold() at dmu_bonus_hold+0x31
bplist_hold() at bplist_hold+0x48
bplist_space_birthrange() at bplist_space_birthrange+0xb1
dsl_dataset_clone_swap_sync() at dsl_dataset_clone_swap_sync+0xee
dsl_sync_task_group_sync() at dsl_sync_task_group_sync+0x173
dsl_pool_sync() at dsl_pool_sync+0x122
spa_sync() at spa_sync+0x35e
txg_sync_thread() at txg_sync_thread+0x2d7
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80001ffd30, rbp = 0 ---
lock order reversal:
  1st 0xffffff000278f438 ds->ds_deadlist.bpl_lock (ds-
 >ds_deadlist.bpl_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/
contrib/opensolaris/uts/common/fs/zfs/bplist.c:331
  2nd 0xffffff0002851d28 db->db_mtx (db->db_mtx) @ /pool/newsrc/src/
sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/
dbuf.c:1724
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_xlock() at _sx_xlock+0x55
dbuf_rele() at dbuf_rele+0x2d
dnode_hold_impl() at dnode_hold_impl+0x20f
dmu_bonus_hold() at dmu_bonus_hold+0x31
bplist_hold() at bplist_hold+0x48
bplist_space_birthrange() at bplist_space_birthrange+0xb1
dsl_dataset_clone_swap_sync() at dsl_dataset_clone_swap_sync+0xee
dsl_sync_task_group_sync() at dsl_sync_task_group_sync+0x173
dsl_pool_sync() at dsl_pool_sync+0x122
spa_sync() at spa_sync+0x35e
txg_sync_thread() at txg_sync_thread+0x2d7
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80001ffd30, rbp = 0 ---
lock order reversal:
  1st 0xffffff0002914250 zfsvfs->z_znodes_lock (zfsvfs->z_znodes_lock)  
@ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/
common/fs/zfs/zfs_vfsops.c:1314
  2nd 0xffffff0002914310 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i])  
@ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/
common/fs/zfs/zfs_znode.c:963
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_xlock() at _sx_xlock+0x55
zfs_rezget() at zfs_rezget+0x4a
zfs_resume_fs() at zfs_resume_fs+0x158
zfs_ioc_recv() at zfs_ioc_recv+0x2b4
zfsdev_ioctl() at zfsdev_ioctl+0x8d
devfs_ioctl_f() at devfs_ioctl_f+0x76
kern_ioctl() at kern_ioctl+0xc5
ioctl() at ioctl+0xfd
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (54, FreeBSD ELF64, ioctl), rip = 0x800fe874c, rsp =  
0x7fffffffbbf8, rbp = 0x7fffffffc930 ---
#

_______________________________________________
freebsd-fs@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-unsubscribe@..."

[Borja Marcos-2]

(Resending to the list, I replied only to Pawel)


On Oct 31, 2009, at 12:28 AM, Pawel Jakub Dawidek wrote:

> On Thu, Oct 29, 2009 at 03:52:45PM -0600, Stef Walter wrote:
>> Borja Marcos wrote:
>>> I've been sending some alltraces to pjd about this easy to reproduce
>>> problem. I am using zfs send/zfs receive to replicate a dataset  
>>> from one
>>> server to another. At 1 minute intervals, an incremental snapshot is
>>> sent to update the dataset copy. If there is reading activity on the
>>> dataset copy, a deadlock can happen rendering ZFS and all the FS
>>> subsystem unusable. I've tried with 8.0RC2 and it still happens.
>
> I was able to reproduce it, but I don't have fix yet.
>
>> FWIW, another (or the same) zfs recv deadlock I've been trying to  
>> get to
>> the bottom of:
>>
>> http://lists.freebsd.org/pipermail/freebsd-fs/2009-October/ 
>> 006999.html
>
> Could you guys recompile your kernel after uncommenting line:
>
> #CFLAGS+=-DDEBUG=1
>
> in sys/modules/zfs/Makefile?
>
> You should see panic on assertion instead of deadlock, I believe.

No panic, it just deadlocked.

Last trace of LORs

Nov  2 13:45:44  kernel: lock order reversal:
Nov  2 13:45:44  kernel: 1st 0xffffff00025d4c38 ds-
 >ds_deadlist.bpl_lock (ds->ds_deadlist.bpl_lock) @ /pool/newsrc/src/
sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/
bplist.c:189
Nov  2 13:45:44  kernel: 2nd 0xffffff000279bd40 osi->os_lock (osi-
 >os_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/
opensolaris/uts/common/fs/zfs/dnode.c:705
Nov  2 13:45:44  kernel: KDB: stack backtrace:
Nov  2 13:45:44  kernel: db_trace_self_wrapper() at  
db_trace_self_wrapper+0x2a
Nov  2 13:45:44  kernel: _witness_debugger() at _witness_debugger+0x2e
Nov  2 13:45:44  kernel: witness_checkorder() at witness_checkorder
+0x81e
Nov  2 13:45:44  kernel: _sx_xlock() at _sx_xlock+0x55
Nov  2 13:45:44  kernel: dnode_setdirty() at dnode_setdirty+0xbc
Nov  2 13:45:44  kernel: dbuf_dirty() at dbuf_dirty+0x516
Nov  2 13:45:44  kernel: bplist_enqueue() at bplist_enqueue+0xbd
Nov  2 13:45:44  kernel: dsl_dataset_block_kill() at  
dsl_dataset_block_kill+0x119
Nov  2 13:45:44  kernel: dmu_objset_sync() at dmu_objset_sync+0x1fe
Nov  2 13:45:44  kernel: dsl_pool_sync() at dsl_pool_sync+0x88
Nov  2 13:45:44  kernel: spa_sync() at spa_sync+0x35e
Nov  2 13:45:44  kernel: txg_sync_thread() at txg_sync_thread+0x2d7
Nov  2 13:45:44  kernel: fork_exit() at fork_exit+0x12a
Nov  2 13:45:44  kernel: fork_trampoline() at fork_trampoline+0xe
Nov  2 13:45:44  kernel: --- trap 0, rip = 0, rsp =  
0xffffff80001ffd30, rbp = 0 ---
Nov  2 13:45:44  kernel: lock order reversal:
Nov  2 13:45:44  kernel: 1st 0xffffff0002eb9d38 dr->dt.di.dr_mtx (dr-
 >dt.di.dr_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/
opensolaris/uts/common/fs/zfs/dbuf.c:1905
Nov  2 13:45:44  kernel: 2nd 0xffffff0002ed3000 dn->dn_struct_rwlock  
(dn->dn_struct_rwlock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/
contrib/opensolaris/uts/common/fs/zfs/dbuf.c:543
Nov  2 13:45:44  kernel: KDB: stack backtrace:
Nov  2 13:45:44  kernel: db_trace_self_wrapper() at  
db_trace_self_wrapper+0x2a
Nov  2 13:45:44  kernel: _witness_debugger() at _witness_debugger+0x2e
Nov  2 13:45:44  kernel: witness_checkorder() at witness_checkorder
+0x81e
Nov  2 13:45:44  kernel: _sx_slock() at _sx_slock+0x55
Nov  2 13:45:44  kernel: dbuf_read() at dbuf_read+0x2ad
Nov  2 13:45:44  kernel: dbuf_will_dirty() at dbuf_will_dirty+0x53
Nov  2 13:45:44  kernel: dsl_dataset_block_kill() at  
dsl_dataset_block_kill+0xe9
Nov  2 13:45:44  kernel: dbuf_write() at dbuf_write+0x24c
Nov  2 13:45:44  kernel: dbuf_sync_list() at dbuf_sync_list+0x159
Nov  2 13:45:44  kernel: dbuf_sync_list() at dbuf_sync_list+0x17f
Nov  2 13:45:44  kernel: dnode_sync() at dnode_sync+0xc12
Nov  2 13:45:44  kernel: dmu_objset_sync() at dmu_objset_sync+0x134
Nov  2 13:45:44  kernel: dsl_pool_sync() at dsl_pool_sync+0x88
Nov  2 13:45:44  kernel: spa_sync() at spa_sync+0x35e
Nov  2 13:45:44  kernel: txg_sync_thread() at txg_sync_thread+0x2d7
Nov  2 13:45:44  kernel: fork_exit() at fork_exit+0x12a
Nov  2 13:45:44  kernel: fork_trampoline() at fork_trampoline+0xe
Nov  2 13:45:44  kernel: --- trap 0, rip = 0, rsp =  
0xffffff80001ffd30, rbp = 0 ---
Nov  2 13:46:03  kernel: lock order reversal:
Nov  2 13:46:03  kernel: 1st 0xffffff00025d4c38 ds-
 >ds_deadlist.bpl_lock (ds->ds_deadlist.bpl_lock) @ /pool/newsrc/src/
sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/
bplist.c:152
Nov  2 13:46:03  kernel: 2nd 0xffffff0002f01ae0 dn->dn_dbufs_mtx (dn-
 >dn_dbufs_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/
opensolaris/uts/common/fs/zfs/dbuf.c:1518
Nov  2 13:46:03  kernel: KDB: stack backtrace:
Nov  2 13:46:03  kernel: db_trace_self_wrapper() at  
db_trace_self_wrapper+0x2a
Nov  2 13:46:03  kernel: _witness_debugger() at _witness_debugger+0x2e
Nov  2 13:46:03  kernel: witness_checkorder() at witness_checkorder
+0x81e
Nov  2 13:46:03  kernel: _sx_xlock() at _sx_xlock+0x55
Nov  2 13:46:03  kernel: dbuf_destroy() at dbuf_destroy+0x58
Nov  2 13:46:03  kernel: bplist_cache() at bplist_cache+0x2e
Nov  2 13:46:03  kernel: bplist_iterate() at bplist_iterate+0xb3
Nov  2 13:46:03  kernel: bplist_space_birthrange() at  
bplist_space_birthrange+0x60
Nov  2 13:46:03  kernel: dsl_dataset_clone_swap_sync() at  
dsl_dataset_clone_swap_sync+0xee
Nov  2 13:46:03  kernel: dsl_sync_task_group_sync() at  
dsl_sync_task_group_sync+0x173
Nov  2 13:46:03  kernel: dsl_pool_sync() at dsl_pool_sync+0x122
Nov  2 13:46:03  kernel: spa_sync() at spa_sync+0x35e
Nov  2 13:46:03  kernel: txg_sync_thread() at txg_sync_thread+0x2d7
Nov  2 13:46:03  kernel: fork_exit() at fork_exit+0x12a
Nov  2 13:46:03  kernel: fork_trampoline() at fork_trampoline+0xe
Nov  2 13:46:03  kernel: --- trap 0, rip = 0, rsp =  
0xffffff80001ffd30, rbp = 0 ---
Nov  2 13:46:03  kernel: lock order reversal:
Nov  2 13:46:03  kernel: 1st 0xffffff00025d4c38 ds-
 >ds_deadlist.bpl_lock (ds->ds_deadlist.bpl_lock) @ /pool/newsrc/src/
sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/
bplist.c:152
Nov  2 13:46:03  kernel: 2nd 0xffffffff81121d50 h->hash_mutexes[i] (h-
 >hash_mutexes[i]) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/
contrib/opensolaris/uts/common/fs/zfs/dbuf.c:191
Nov  2 13:46:03  kernel: KDB: stack backtrace:
Nov  2 13:46:03  kernel: db_trace_self_wrapper() at  
db_trace_self_wrapper+0x2a
Nov  2 13:46:03  kernel: _witness_debugger() at _witness_debugger+0x2e
Nov  2 13:46:03  kernel: witness_checkorder() at witness_checkorder
+0x81e
Nov  2 13:46:03  kernel: _sx_xlock() at _sx_xlock+0x55
Nov  2 13:46:03  kernel: dbuf_destroy() at dbuf_destroy+0x111
Nov  2 13:46:03  kernel: bplist_cache() at bplist_cache+0x2e
Nov  2 13:46:03  kernel: bplist_iterate() at bplist_iterate+0xb3
Nov  2 13:46:03  kernel: bplist_space_birthrange() at  
bplist_space_birthrange+0x60
Nov  2 13:46:03  kernel: dsl_dataset_clone_swap_sync() at  
dsl_dataset_clone_swap_sync+0xee
Nov  2 13:46:03  kernel: dsl_sync_task_group_sync() at  
dsl_sync_task_group_sync+0x173
Nov  2 13:46:03  kernel: dsl_pool_sync() at dsl_pool_sync+0x122
Nov  2 13:46:03  kernel: spa_sync() at spa_sync+0x35e
Nov  2 13:46:03  kernel: txg_sync_thread() at txg_sync_thread+0x2d7
Nov  2 13:46:03  kernel: fork_exit() at fork_exit+0x12a
Nov  2 13:46:03  kernel: fork_trampoline() at fork_trampoline+0xe
Nov  2 13:46:03  kernel: --- trap 0, rip = 0, rsp =  
0xffffff80001ffd30, rbp = 0 ---
Nov  2 13:46:35  kernel: lock order reversal:
Nov  2 13:46:35  kernel: 1st 0xffffff000279c398 dp->dp_config_rwlock  
(dp->dp_config_rwlock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/
contrib/opensolaris/uts/common/fs/zfs/dsl_synctask.c:171
Nov  2 13:46:35  kernel: 2nd 0xffffff0002784938 ds->ds_opening_lock  
(ds->ds_opening_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/
contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c:483
Nov  2 13:46:35  kernel: KDB: stack backtrace:
Nov  2 13:46:35  kernel: db_trace_self_wrapper() at  
db_trace_self_wrapper+0x2a
Nov  2 13:46:35  kernel: _witness_debugger() at _witness_debugger+0x2e
Nov  2 13:46:35  kernel: witness_checkorder() at witness_checkorder
+0x81e
Nov  2 13:46:35  kernel: _sx_xlock() at _sx_xlock+0x55
Nov  2 13:46:35  kernel: dmu_objset_create_impl() at  
dmu_objset_create_impl+0x50
Nov  2 13:46:35  kernel: dmu_objset_create_sync() at  
dmu_objset_create_sync+0xfc
Nov  2 13:46:35  kernel: dsl_sync_task_group_sync() at  
dsl_sync_task_group_sync+0x173
Nov  2 13:46:35  kernel: dsl_pool_sync() at dsl_pool_sync+0x122
Nov  2 13:46:35  kernel: spa_sync() at spa_sync+0x35e
Nov  2 13:46:35  kernel: txg_sync_thread() at txg_sync_thread+0x2d7
Nov  2 13:46:35  kernel: fork_exit() at fork_exit+0x12a
Nov  2 13:46:35  kernel: fork_trampoline() at fork_trampoline+0xe
Nov  2 13:46:35  kernel: --- trap 0, rip = 0, rsp =  
0xffffff80001ffd30, rbp = 0 ---
Nov  2 13:46:35  kernel: lock order reversal:
Nov  2 13:46:35  kernel: 1st 0xffffff000279c398 dp->dp_config_rwlock  
(dp->dp_config_rwlock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/
contrib/opensolaris/uts/common/fs/zfs/dsl_synctask.c:171
Nov  2 13:46:35  kernel: 2nd 0xffffff0002fb7578 zfs (zfs) @ /pool/
newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/
fs/zfs/zfs_znode.c:152
Nov  2 13:46:35  kernel: KDB: stack backtrace:
Nov  2 13:46:35  kernel: db_trace_self_wrapper() at  
db_trace_self_wrapper+0x2a
Nov  2 13:46:35  kernel: _witness_debugger() at _witness_debugger+0x2e
Nov  2 13:46:35  kernel: witness_checkorder() at witness_checkorder
+0x81e
Nov  2 13:46:35  kernel: __lockmgr_args() at __lockmgr_args+0xd03
Nov  2 13:46:35  kernel: vop_stdlock() at vop_stdlock+0x39
Nov  2 13:46:35  kernel: VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
Nov  2 13:46:35  kernel: _vn_lock() at _vn_lock+0x57
Nov  2 13:46:35  kernel: zfs_znode_cache_constructor() at  
zfs_znode_cache_constructor+0x65
Nov  2 13:46:35  kernel: zfs_create_fs() at zfs_create_fs+0x2ab
Nov  2 13:46:35  kernel: dmu_objset_create_sync() at  
dmu_objset_create_sync+0x116
Nov  2 13:46:35  kernel: dsl_sync_task_group_sync() at  
dsl_sync_task_group_sync+0x173
Nov  2 13:46:35  kernel: dsl_pool_sync() at dsl_pool_sync+0x122
Nov  2 13:46:35  kernel: spa_sync() at spa_sync+0x35e
Nov  2 13:46:35  kernel: txg_sync_thread() at txg_sync_thread+0x2d7
Nov  2 13:46:35  kernel: fork_exit() at fork_exit+0x12a
Nov  2 13:46:35  kernel: fork_trampoline() at fork_trampoline+0xe
Nov  2 13:46:35  kernel: --- trap 0, rip = 0, rsp =  
0xffffff80001ffd30, rbp = 0 ---
Nov  2 13:46:36  kernel: lock order reversal:
Nov  2 13:46:36  kernel: 1st 0xffffff001a0a7430 db->db_mtx (db-
 >db_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/
opensolaris/uts/common/fs/zfs/dbuf.c:1724
Nov  2 13:46:36  kernel: 2nd 0xffffff001a4c0708 dn->dn_dbufs_mtx (dn-
 >dn_dbufs_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/
opensolaris/uts/common/fs/zfs/dnode_sync.c:373
Nov  2 13:46:36  kernel: KDB: stack backtrace:
Nov  2 13:46:36  kernel: db_trace_self_wrapper() at  
db_trace_self_wrapper+0x2a
Nov  2 13:46:36  kernel: _witness_debugger() at _witness_debugger+0x2e
Nov  2 13:46:36  kernel: witness_checkorder() at witness_checkorder
+0x81e
Nov  2 13:46:36  kernel: _sx_xlock() at _sx_xlock+0x55
Nov  2 13:46:36  kernel: dnode_evict_dbufs() at dnode_evict_dbufs+0x57
Nov  2 13:46:36  kernel: dmu_objset_evict_dbufs() at  
dmu_objset_evict_dbufs+0xd4
Nov  2 13:46:36  kernel: dmu_objset_evict() at dmu_objset_evict+0xbf
Nov  2 13:46:36  kernel: dsl_dataset_evict() at dsl_dataset_evict+0x54
Nov  2 13:46:36  kernel: dbuf_evict_user() at dbuf_evict_user+0x55
Nov  2 13:46:36  kernel: dbuf_rele() at dbuf_rele+0x173
Nov  2 13:46:36  kernel: dsl_pool_zil_clean() at dsl_pool_zil_clean+0x3c
Nov  2 13:46:36  kernel: spa_sync() at spa_sync+0x618
Nov  2 13:46:36  kernel: txg_sync_thread() at txg_sync_thread+0x2d7
Nov  2 13:46:36  kernel: fork_exit() at fork_exit+0x12a
Nov  2 13:46:36  kernel: fork_trampoline() at fork_trampoline+0xe
Nov  2 13:46:36  kernel: --- trap 0, rip = 0, rsp =  
0xffffff80001ffd30, rbp = 0 ---
Nov  2 13:46:36  kernel: lock order reversal:
Nov  2 13:46:36  kernel: 1st 0xffffff001a0a7430 db->db_mtx (db-
 >db_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/
opensolaris/uts/common/fs/zfs/dbuf.c:1724
Nov  2 13:46:36  kernel: 2nd 0xffffff001a4c03d8 dn->dn_struct_rwlock  
(dn->dn_struct_rwlock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/
contrib/opensolaris/uts/common/fs/zfs/dnode_sync.c:409
Nov  2 13:46:36  kernel: KDB: stack backtrace:
Nov  2 13:46:36  kernel: db_trace_self_wrapper() at  
db_trace_self_wrapper+0x2a
Nov  2 13:46:36  kernel: _witness_debugger() at _witness_debugger+0x2e
Nov  2 13:46:36  kernel: witness_checkorder() at witness_checkorder
+0x81e
Nov  2 13:46:36  kernel: _sx_xlock() at _sx_xlock+0x55
Nov  2 13:46:36  kernel: dnode_evict_dbufs() at dnode_evict_dbufs+0x1b8
Nov  2 13:46:36  kernel: dmu_objset_evict_dbufs() at  
dmu_objset_evict_dbufs+0xd4
Nov  2 13:46:36  kernel: dmu_objset_evict() at dmu_objset_evict+0xbf
Nov  2 13:46:36  kernel: dsl_dataset_evict() at dsl_dataset_evict+0x54
Nov  2 13:46:36  kernel: dbuf_evict_user() at dbuf_evict_user+0x55
Nov  2 13:46:36  kernel: dbuf_rele() at dbuf_rele+0x173
Nov  2 13:46:36  kernel: dsl_pool_zil_clean() at dsl_pool_zil_clean+0x3c
Nov  2 13:46:36  kernel: spa_sync() at spa_sync+0x618
Nov  2 13:46:36  kernel: txg_sync_thread() at txg_sync_thread+0x2d7
Nov  2 13:46:36  kernel: fork_exit() at fork_exit+0x12a
Nov  2 13:46:36  kernel: fork_trampoline() at fork_trampoline+0xe
Nov  2 13:46:36  kernel: --- trap 0, rip = 0, rsp =  
0xffffff80001ffd30, rbp = 0 ---
Nov  2 13:46:57  kernel: lock order reversal:
Nov  2 13:46:57  kernel: 1st 0xffffff001a81ec38 dr->dt.di.dr_mtx (dr-
 >dt.di.dr_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/
opensolaris/uts/common/fs/zfs/dbuf.c:1905
Nov  2 13:46:57  kernel: 2nd 0xffffff0002fdb4b0 dn->dn_mtx (dn-
 >dn_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/
opensolaris/uts/common/fs/zfs/dbuf.c:1066
Nov  2 13:46:57  kernel: KDB: stack backtrace:
Nov  2 13:46:57  kernel: db_trace_self_wrapper() at  
db_trace_self_wrapper+0x2a
Nov  2 13:46:57  kernel: _witness_debugger() at _witness_debugger+0x2e
Nov  2 13:46:57  kernel: witness_checkorder() at witness_checkorder
+0x81e
Nov  2 13:46:57  kernel: _sx_xlock() at _sx_xlock+0x55
Nov  2 13:46:57  kernel: dbuf_dirty() at dbuf_dirty+0xa03
Nov  2 13:46:57  kernel: dsl_dataset_block_kill() at  
dsl_dataset_block_kill+0x38b
Nov  2 13:46:57  kernel: dbuf_write() at dbuf_write+0x24c
Nov  2 13:46:57  kernel: dbuf_sync_list() at dbuf_sync_list+0x3eb
Nov  2 13:46:57  kernel: dbuf_sync_list() at dbuf_sync_list+0x17f
Nov  2 13:46:57  last message repeated 5 times
Nov  2 13:46:57  kernel: dnode_sync() at dnode_sync+0xc12
Nov  2 13:46:57  kernel: dmu_objset_sync() at dmu_objset_sync+0x134
Nov  2 13:46:57  kernel: dsl_pool_sync() at dsl_pool_sync+0x88
Nov  2 13:46:57  kernel: spa_sync() at spa_sync+0x35e
Nov  2 13:46:57  kernel: txg_sync_thread() at txg_sync_thread+0x2d7
Nov  2 13:46:57  kernel: fork_exit() at fork_exit+0x12a
Nov  2 13:46:57  kernel: fork_trampoline() at fork_trampoline+0xe
Nov  2 13:46:57  kernel: --- trap 0, rip = 0, rsp =  
0xffffff80001ffd30, rbp = 0 ---
Nov  2 13:46:57  kernel: lock order reversal:
Nov  2 13:46:57  kernel: 1st 0xffffff001a81ec38 dr->dt.di.dr_mtx (dr-
 >dt.di.dr_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/
opensolaris/uts/common/fs/zfs/dbuf.c:1905
Nov  2 13:46:57  kernel: 2nd 0xffffff000279bd40 osi->os_lock (osi-
 >os_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/
opensolaris/uts/common/fs/zfs/dnode.c:705
Nov  2 13:46:57  kernel: KDB: stack backtrace:
Nov  2 13:46:57  kernel: db_trace_self_wrapper() at  
db_trace_self_wrapper+0x2a
Nov  2 13:46:57  kernel: _witness_debugger() at _witness_debugger+0x2e
Nov  2 13:46:57  kernel: witness_checkorder() at witness_checkorder
+0x81e
Nov  2 13:46:57  kernel: _sx_xlock() at _sx_xlock+0x55
Nov  2 13:46:57  kernel: dnode_setdirty() at dnode_setdirty+0xbc
Nov  2 13:46:57  kernel: dbuf_dirty() at dbuf_dirty+0xa53
Nov  2 13:46:57  kernel: dsl_dataset_block_kill() at  
dsl_dataset_block_kill+0x38b
Nov  2 13:46:57  kernel: dbuf_write() at dbuf_write+0x24c
Nov  2 13:46:57  kernel: dbuf_sync_list() at dbuf_sync_list+0x3eb
Nov  2 13:46:57  kernel: dbuf_sync_list() at dbuf_sync_list+0x17f
Nov  2 13:46:57  last message repeated 5 times
Nov  2 13:46:57  kernel: dnode_sync() at dnode_sync+0xc12
Nov  2 13:46:57  kernel: dmu_objset_sync() at dmu_objset_sync+0x134
Nov  2 13:46:57  kernel: dsl_pool_sync() at dsl_pool_sync+0x88
Nov  2 13:46:57  kernel: spa_sync() at spa_sync+0x35e
Nov  2 13:46:57  kernel: txg_sync_thread() at txg_sync_thread+0x2d7
Nov  2 13:46:57  kernel: fork_exit() at fork_exit+0x12a
Nov  2 13:46:57  kernel: fork_trampoline() at fork_trampoline+0xe
Nov  2 13:46:57  kernel: --- trap 0, rip = 0, rsp =  
0xffffff80001ffd30, rbp = 0 ---
Nov  2 13:47:28  kernel: lock order reversal:
Nov  2 13:47:28  kernel: 1st 0xffffff0002fb6a58 syncer (syncer) @ /
pool/newsrc/src/sys/kern/vfs_subr.c:1693
Nov  2 13:47:28  kernel: 2nd 0xffffff001a868350 zfsvfs->z_hold_mtx[i]  
(zfsvfs->z_hold_mtx[i]) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/
contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:863
Nov  2 13:47:28  kernel: KDB: stack backtrace:
Nov  2 13:47:28  kernel: db_trace_self_wrapper() at  
db_trace_self_wrapper+0x2a
Nov  2 13:47:28  kernel: _witness_debugger() at _witness_debugger+0x2e
Nov  2 13:47:28  kernel: witness_checkorder() at witness_checkorder
+0x81e
Nov  2 13:47:28  kernel: _sx_xlock() at _sx_xlock+0x55
Nov  2 13:47:28  kernel: zfs_zget() at zfs_zget+0x23a
Nov  2 13:47:28  kernel: zfs_get_data() at zfs_get_data+0x5e
Nov  2 13:47:28  kernel: zil_commit() at zil_commit+0x5aa
Nov  2 13:47:28  kernel: zfs_sync() at zfs_sync+0xa6
Nov  2 13:47:28  kernel: sync_fsync() at sync_fsync+0x13a
Nov  2 13:47:28  kernel: VOP_F
Nov  2 13:47:28  kernel: SYNC_APV() at VOP_FSYNC_APV+0xb5
Nov  2 13:47:28  kernel: sync_vnode() at sync_vnode+0x157
Nov  2 13:47:28  kernel: sched_sync() at sched_sync+0x1d2
Nov  2 13:47:28  kernel: fork_exit() at fork_exit+0x12a
Nov  2 13:47:28  kernel: fork_trampoline() at fork_trampoline+0xe
Nov  2 13:47:28  kernel: --- trap 0, rip = 0, rsp =  
0xffffff8017aebd30, rbp = 0 ---
Nov  2 13:47:44  kernel: lock order reversal:
Nov  2 13:47:44  kernel: 1st 0xffffff002325d4c0 zp->z_parent_lock (zp-
 >z_parent_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/
opensolaris/uts/common/fs/zfs/zfs_dir.c:379
Nov  2 13:47:44  kernel: 2nd 0xffffff0002914370 zfsvfs->z_hold_mtx[i]  
(zfsvfs->z_hold_mtx[i]) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/
contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:863
Nov  2 13:47:44  kernel: KDB: stack backtrace:
Nov  2 13:47:44  kernel: db_trace_self_wrapper() at  
db_trace_self_wrapper+0x2a
Nov  2 13:47:44  kernel: _witness_debugger() at _witness_debugger+0x2e
Nov  2 13:47:44  kernel: witness_checkorder() at witness_checkorder
+0x81e
Nov  2 13:47:44  kernel: _sx_xlock() at _sx_xlock+0x55
Nov  2 13:47:44  kernel: zfs_zget() at zfs_zget+0x23a
Nov  2 13:47:44  kernel: zfs_dirlook() at zfs_dirlook+0x1fc
Nov  2 13:47:44  kernel: zfs_lookup() at zfs_lookup+0x257
Nov  2 13:47:44  kernel: zfs_freebsd_lookup() at zfs_freebsd_lookup+0x8d
Nov  2 13:47:44  kernel: VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV
+0xaf
Nov  2 13:47:44  kernel: vfs_cache_lookup() at vfs_cache_lookup+0xf0
Nov  2 13:47:44  kernel: VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7
Nov  2 13:47:44  kernel: lookup() at lookup+0x2eb
Nov  2 13:47:44  kernel: namei() at namei+0x4a9
Nov  2 13:47:44  kernel: kern_chdir() at kern_chdir+0x78
Nov  2 13:47:44  kernel: syscall() at syscall+0x1d0
Nov  2 13:47:44  kernel: Xfast_syscall() at Xfast_syscall+0xe1
Nov  2 13:47:44  kernel: --- syscall (12, FreeBSD ELF64, chdir), rip =  
0x800da8a9c, rsp = 0x7fffffffe8b8, rbp = 0x8010135e0 ---





_______________________________________________
freebsd-fs@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-unsubscribe@..."