Just for fun

[Andrew Doran-7]

Hi,

A few years back pk@ locked a few more kernel subsystems and tried running
the kernel unlocked to see what would happen. I repeated the test with a
kernel from the newlock2 branch and to my amazement it completes a kernel
build on a 4 way system.

build.sh isn't so good - once it hits 'make objdir' and there's a lot of
intensive file system activity going it will either panic, or lock up in
biowait -> vnlock, somewhere around 30 to 60 seconds in.

Still, I think it's pretty encouraging that things work as well as they do.
I put dmesg and some lockstat output from a couple of runs at the URL below.
Note that the timings for the sleep locks are wrong.

        http://www.netbsd.org/~ad/biglock/

Cheers,
Andrew

[Perry E. Metzger]

Andrew Doran <ad@...> writes:
> A few years back pk@ locked a few more kernel subsystems and tried running
> the kernel unlocked to see what would happen. I repeated the test with a
> kernel from the newlock2 branch and to my amazement it completes a kernel
> build on a 4 way system.
>
> build.sh isn't so good - once it hits 'make objdir' and there's a lot of
> intensive file system activity going it will either panic, or lock up in
> biowait -> vnlock, somewhere around 30 to 60 seconds in.

That's pretty amazing. Should we attribute this to the low probability
of contention in the real world, or to the quality of the stuff on
newlock2? (I suspect a bit of both, which is, as you say, encouraging.)

> Still, I think it's pretty encouraging that things work as well as they do.
> I put dmesg and some lockstat output from a couple of runs at the URL below.
> Note that the timings for the sleep locks are wrong.
>
> http://www.netbsd.org/~ad/biglock/

Perry

[Andrew Doran-7]

On Mon, Jan 15, 2007 at 11:44:21AM -0500, Perry E. Metzger wrote:

> Andrew Doran <ad@...> writes:
> > A few years back pk@ locked a few more kernel subsystems and tried running
> > the kernel unlocked to see what would happen. I repeated the test with a
> > kernel from the newlock2 branch and to my amazement it completes a kernel
> > build on a 4 way system.
> >
> > build.sh isn't so good - once it hits 'make objdir' and there's a lot of
> > intensive file system activity going it will either panic, or lock up in
> > biowait -> vnlock, somewhere around 30 to 60 seconds in.
>
> That's pretty amazing. Should we attribute this to the low probability
> of contention in the real world,

The kernel build doesn't spend a huge amount of time in the kernel, but it
definitely produces contention. I think the majority of it is likely to
result from handling traps for zero fill and pageins.

> or to the quality of the stuff on newlock2? (I suspect a bit of both,
> which is, as you say, encouraging.)

The branch adds one more piece of the puzzle (locking around process state).
That it works comparatively well is a product of all of the locks that have
been trickling into the system for the last .. well, since 4.4BSD I think.

Andrew

[Perry E. Metzger]

Andrew Doran <ad@...> writes:
> The branch adds one more piece of the puzzle (locking around process state).
> That it works comparatively well is a product of all of the locks that have
> been trickling into the system for the last .. well, since 4.4BSD I think.

BTW, I wonder whether we can modify an x86 emulator into a lock
debugger.

Some years ago I read a paper about a system called "taintbochs" that
modified the Bochs emulator to track the flow of password and other
sensitive data inside a kernel.

What if we did something similar but instead taught the emulator about
locks and traced unlocked touches of global data structures?

Perry

[Pavel Cahyna-4]

On Tue, Jan 16, 2007 at 07:53:51AM -0500, Perry E. Metzger wrote:
>
> Andrew Doran <ad@...> writes:
> > The branch adds one more piece of the puzzle (locking around process state).
> > That it works comparatively well is a product of all of the locks that have
> > been trickling into the system for the last .. well, since 4.4BSD I think.
>
> BTW, I wonder whether we can modify an x86 emulator into a lock
> debugger.

Yes: http://www.valgrind.org/info/tools.html#helgrind

[Perry E. Metzger]

Pavel Cahyna <pavel@...> writes:

> On Tue, Jan 16, 2007 at 07:53:51AM -0500, Perry E. Metzger wrote:
>>
>> Andrew Doran <ad@...> writes:
>> > The branch adds one more piece of the puzzle (locking around
>> > process state).  That it works comparatively well is a product of
>> > all of the locks that have been trickling into the system for the
>> > last .. well, since 4.4BSD I think.
>>
>> BTW, I wonder whether we can modify an x86 emulator into a lock
>> debugger.
>
> Yes: http://www.valgrind.org/info/tools.html#helgrind

Can that be used for running a kernel?

Perry
--
Perry E. Metzger perry@...

[David Maxwell-3]

On Tue, Jan 16, 2007 at 07:53:51AM -0500, Perry E. Metzger wrote:
>
> Andrew Doran <ad@...> writes:
> > The branch adds one more piece of the puzzle (locking around process state).
> > That it works comparatively well is a product of all of the locks that have
> > been trickling into the system for the last .. well, since 4.4BSD I think.
>
> What if we did something similar but instead taught the emulator about
> locks and traced unlocked touches of global data structures?

Coverity Prevent also has a set of concurrency checkers that can
identify issues with locking.

Those checkers haven't been turned on yet on scan.coverity.com, but we
are working towards making the additional results available. I just
wanted to let you know that you'll have additional information to call
on when that happens.

--
David Maxwell
Coverity Open Source Strategist
http://scan.coverity.com/