Help with ddb and lockdebug.

[Brian Buhrow]

        Hello.  I'm trying to debug akernel module I'm working on which is
having locking issues.  I have a 5.0 kernel with  options LOCKDEBUG turned
on, and I'm able to figure out some of what I'm seeing, but I have a couple
of questions, and I hope someone here can help point me in the right
direction so I can make further progress on getting zaptel working on
NetBSD-5.x.  
I'm seeing output like the following and I have a couple of questions:

Mutex error: lockdebug_barrier: spin lock held

lock address : 0x00000000c0b24700 type     :               spin
initialized  : 0x00000000c043a946
shared holds :                  0 exclusive:                  1
shares wanted:                  0 exclusive:                  0
current cpu  :                  1 last held:                  1
current lwp  : 0x00000000ce0a7520 last held: 0x00000000ce0a7520
last locked  : 0x00000000c043a8a6 unlocked : 0x00000000c04280f3
owner field  : 0x0000000000010700 wait/spin:                0/1

        1.   What are the pointers for the locked and unlocked fields?  Are
they pointers to the struct proc structures for the last processes that
locked and unlocked this lock respectively?  If so, how can I examine those
proc structures in ddb?

2.  How do I read the owner field?  It doesn't look like a uid or a pid,
what is it?

3.  Is there documentation on some of this somewhere?  I've read the ddb
man page, and I'm familiar with the 4.4BSD internals book by Kirk Mcusick.
I've been working with NetBSD for years, but I'm having trouble finding
where this output comes from in the source tree.

        In short, if anyone has pointers on how I can get more complete
information on what I'm looking at in ddb, I would be very interested.

-thanks
-Brian

[Jean-Yves Migeon-3]

Brian Buhrow wrote:
> Hello.  I'm trying to debug akernel module I'm working on which is
> having locking issues.  I have a 5.0 kernel with  options LOCKDEBUG turned
> on, and I'm able to figure out some of what I'm seeing, but I have a couple
> of questions, and I hope someone here can help point me in the right
> direction so I can make further progress on getting zaptel working on
> NetBSD-5.x.  
> I'm seeing output like the following and I have a couple of questions:
>
> Mutex error: lockdebug_barrier: spin lock held

The CPU is probably trying to hold more than one spinlock, which can
result in nasty deadlocks. Hence, the panic.

> lock address : 0x00000000c0b24700 type     :               spin
> initialized  : 0x00000000c043a946
> shared holds :                  0 exclusive:                  1
> shares wanted:                  0 exclusive:                  0
> current cpu  :                  1 last held:                  1
> current lwp  : 0x00000000ce0a7520 last held: 0x00000000ce0a7520
> last locked  : 0x00000000c043a8a6 unlocked : 0x00000000c04280f3
> owner field  : 0x0000000000010700 wait/spin:                0/1
>
> 1.   What are the pointers for the locked and unlocked fields?  Are
> they pointers to the struct proc structures for the last processes that
> locked and unlocked this lock respectively?  If so, how can I examine those
> proc structures in ddb?

(provided without warranty)

- lock address: where the lock (spin mutex) resides in virtual memory
- initialized: where (in code) it was initialized
- last locked: where it was acquired for the last time
- unlocked: similar to 'last locked', this time for unlocking
- current lwp: the pointer to the lwp struct that acquired the mutex
when the lockdebug fired

To obtain information about the lwp in ddb:
$ ps/l # will list the different threads, look for the offending one
$ bt/a <lwp_addr> # for backtrace of the associated lwp

See ddb(4)

> 2.  How do I read the owner field?  It doesn't look like a uid or a pid,
> what is it?

For adaptive mutexes, it stores a pointer to the current owner of the
mutex. For spin mutexes, there is no "ownership" context (spin mutexes
are hold for a very short amount of time, and do not sleep), it is just
used as a bit field.

See kern/kern_mutex.c and mutex(9).

> 3.  Is there documentation on some of this somewhere?  I've read the ddb
> man page, and I'm familiar with the 4.4BSD internals book by Kirk Mcusick.

NetBSD is moving to a fine-grained locking model, which is quite
different from the 4.4BSD one. "Solaris internals" might be closer to
what you want.

> I've been working with NetBSD for years, but I'm having trouble finding
> where this output comes from in the source tree.

See kern/subr_lockdebug.c

> In short, if anyone has pointers on how I can get more complete
> information on what I'm looking at in ddb, I would be very interested.

HTH

PS: anyone is welcomed to make additions or corrections, of course.

--
Jean-Yves Migeon
jeanyves.migeon@...