MP and device drivers

[Andrew Doran-7]

Hi all,

I would like to gate entry to character and block drivers so that we acquire
the kernel_lock if the driver is not marked as MP safe.

Here are my initial thoughts:

- Make callout_init() take a flag indicating MP-safety. We'd take the
  kernel_lock just before final dispatch of the callout.

- Put wrappers around the devsw entry points (e.g. bdev_strategy(dev, bp)).
  In the MP case these would check d_flag and maybe grab the kernel_lock.
  We'd need to assert that no other locks are already held.

- Modify generic soft interrupts and hard interupts not to take the kernel
  lock if the handler is marked as MP safe.

- Take the lock in shutdown hooks and perhaps other minor entry points.
  As others have mentioned these should be proper driver entry points but
  that's a whole other can of worms.

There are some problems with this approach:

- There are multiple places in each driver where we have to specify 'MP
  safe'. Ideally we would stash this information somewhere else, like
  cfattach or the lkm info.

- This doesn't handle network drivers and 'oddball' drivers, but that's
  beyond what I'm trying to do.

Can you see other problems with this approach? Thoughts?

Thanks,
Andrew

[YAMAMOTO Takashi]

> - Put wrappers around the devsw entry points (e.g. bdev_strategy(dev, bp)).
>   In the MP case these would check d_flag and maybe grab the kernel_lock.
>   We'd need to assert that no other locks are already held.

do you have a plan how to make callers mp-safe?

YAMAMOTO Takashi

[Andrew Doran-7]

On Thu, Sep 07, 2006 at 09:51:25PM +0900, YAMAMOTO Takashi wrote:

> > - Put wrappers around the devsw entry points (e.g. bdev_strategy(dev, bp)).
> >   In the MP case these would check d_flag and maybe grab the kernel_lock.
> >   We'd need to assert that no other locks are already held.
>
> do you have a plan how to make callers mp-safe?

That's the difficult part and something that I don't have the knowledge or
time to tackle yet, specifically VFS. My intent with this one change is to
make it easier to work on making the core MP safe.

The approach I started with was to one-by-one lock access to the members of
"struct proc" and "struct lwp", but decided that there was no point
proceeding without newlock. I'll check in what limited changes I have in
that area over the next few days.

If I remember correctly, the FreeBSD guys produced a chart with a list of
things that needed to be done, who was working on it, etc. I think that's a
good idea. I'd be happy to produce a first version of that with some high
level tasks on it, maybe next week. If somebody else wants to go ahead and
do that, or discuss here that would be great too.

I also think it would be a good idea to have a policy that no new code gets
committed to (for example) kern/* unless it's MP safe, or the task becomes
more difficult.

What do you think? Do you have other ideas or suggestions?

Thanks,
Andrew

[YAMAMOTO Takashi]

> On Thu, Sep 07, 2006 at 09:51:25PM +0900, YAMAMOTO Takashi wrote:
>
> > > - Put wrappers around the devsw entry points (e.g. bdev_strategy(dev, bp)).
> > >   In the MP case these would check d_flag and maybe grab the kernel_lock.
> > >   We'd need to assert that no other locks are already held.
> >
> > do you have a plan how to make callers mp-safe?
>
> That's the difficult part and something that I don't have the knowledge or
> time to tackle yet, specifically VFS. My intent with this one change is to
> make it easier to work on making the core MP safe.
>
> The approach I started with was to one-by-one lock access to the members of
> "struct proc" and "struct lwp", but decided that there was no point
> proceeding without newlock. I'll check in what limited changes I have in
> that area over the next few days.

a concern is that, unless you never enter the driver with biglock held
or you make all higher level interrupt handlers mp-safe beforehand,
what the driver can do will be very limited due to the locking order
constraint with biglock.

> I also think it would be a good idea to have a policy that no new code gets
> committed to (for example) kern/* unless it's MP safe, or the task becomes
> more difficult.

i agree that mp-safeness should be considered when writing new code.

YAMAMOTO Takashi

[Gavan Fantom]

YAMAMOTO Takashi wrote:
> a concern is that, unless you never enter the driver with biglock held
> or you make all higher level interrupt handlers mp-safe beforehand,
> what the driver can do will be very limited due to the locking order
> constraint with biglock.

Would it be useful then to treat all drivers as non-mp-safe even if
they're flagged safe, unless all drivers with higher level interrupts
are declared mp-safe?

--
Gillette - the best a man can forget