THREADS Management XUL/FIREFOX

[Takis Psarogiannakopoulos]

Hello,

I am looking to improve the performance of Firefox 2/3, XUL etc
In the mozilla code I see the following:

mozilla/netwerk/dns/src/nsHostResolver.cpp:#define MAX_THREADS 8
mozilla/netwerk/base/src/nsIOThreadPool.cpp:#define MAX_THREADS 4
mozilla/security/nss/cmd/vfyserv/vfyserv.h:#define MAX_THREADS 32
mozilla/security/nss/cmd/strsclnt/strsclnt.c:#define MAX_THREADS 128
mozilla/security/nss/cmd/selfserv/selfserv.c:#define MAX_THREADS 4096


Now 8 threads in a process is already a fair amount of overhead if you
assume they are allowed to run with a descent stack. I would say optimum
4-6 I  its good for most systems. Eg I know from a fact that
over 9 threads on a recent Xlib (R70) and you are bound to run on
syncronization issues one way or another.
I was wondering: is there is a max number of threads that, say, the
firefox process can create? In other words is there is a global limit for
the number of LWP? Obviously that 4096 is ... 32 also way too much.
Moreover the DNS resolver is allowed just by it self to create 8 different
threads?  Isnt that too much? (please note the dns resolver threads
there are lwps that need a a fair amount of stack too for the job are
intented to do!)

Kind Regards,
_______________________________________________
dev-performance mailing list
dev-performance@...
https://lists.mozilla.org/listinfo/dev-performance

[L. David Baron]

On Tuesday 2009-02-17 17:37 -0800, Takis Psarogiannakopoulos wrote:
> mozilla/security/nss/cmd/vfyserv/vfyserv.h:#define MAX_THREADS 32
> mozilla/security/nss/cmd/strsclnt/strsclnt.c:#define MAX_THREADS 128
> mozilla/security/nss/cmd/selfserv/selfserv.c:#define MAX_THREADS 4096

These look like command-line utilities that are part of the NSS
library.  We import the full NSS source tree, but I don't think
Firefox uses that part of it.

-David

--
L. David Baron                                 http://dbaron.org/
Mozilla Corporation                       http://www.mozilla.com/
_______________________________________________
dev-performance mailing list
dev-performance@...
https://lists.mozilla.org/listinfo/dev-performance

[Boris Zbarsky]

Takis Psarogiannakopoulos wrote:
> mozilla/netwerk/dns/src/nsHostResolver.cpp:#define MAX_THREADS 8

These are threads that run only one small function; they spend most of
their time waiting on getaddrinfo() system calls.  They shouldn't have
much of a stack.  If this is being an issue in practice we can limit the
max stack of these threads, but is it actually more expensive to allow
(but not use) a bigger stack?

> mozilla/netwerk/base/src/nsIOThreadPool.cpp:#define MAX_THREADS 4

These are threads that handle network I/O.  They don't ever have much
code running on them, and again then spend most of their time waiting on
the network.

> mozilla/security/nss/cmd/vfyserv/vfyserv.h:#define MAX_THREADS 32

This is a sample SSL server.  It's not used in Firefox code.

> mozilla/security/nss/cmd/strsclnt/strsclnt.c:#define MAX_THREADS 128

This is a test SSL client used to stress-test the SSL implentation.
It's not used in Firefox code.

> mozilla/security/nss/cmd/selfserv/selfserv.c:#define MAX_THREADS 4096

This is an SSL server.  It's not used in Firefox code.

Do keep in mind that NSS is used in server-side SSL products, not just
in Firefox, right?

> Now 8 threads in a process is already a fair amount of overhead if you
> assume they are allowed to run with a descent stack.

If they're allowed, or if they do?

> I would say optimum 4-6 I  its good for most systems.

Is there data to back this up?

> Eg I know from a fact that
> over 9 threads on a recent Xlib (R70) and you are bound to run on
> syncronization issues one way or another.

Even if all the threads are spending most of their time in i/o wait?

> I was wondering: is there is a max number of threads that, say, the
> firefox process can create?

Not that I know of.

> In other words is there is a global limit for the number of LWP?

Not that I know of.

 > 32 also way too much.

That depends on what they're doing, really.  If most of them spend most
of their time sleeping, there is no problem other than the kernel
accounting needed to keep track of them, no?

> Moreover the DNS resolver is allowed just by it self to create 8 different
> threads?

Yes, if someone tries to resolve 8 different addresses at once.  Note
that the threads are reaped after an idle timeout, by the way.

> Isnt that too much?

Not that we've seen so far.  Do you have data showing that it is?

> (please note the dns resolver threads
> there are lwps that need a a fair amount of stack too for the job are
> intented to do!)

They do?  Why?

-Boris
_______________________________________________
dev-performance mailing list
dev-performance@...
https://lists.mozilla.org/listinfo/dev-performance

[Takis Psarogiannakopoulos]

David,
Thats correct still 32, 128 and 4096 as max limit numbers isnt that too
high? I would say 16 max ...
Anyways where I am most interested (thats why I posted) is the dns
resolver. 8 threads with considerably descnt stacks to handle dns
requests? Why not 4? In fact I have build a FI2 binary on my SVR4 with 4
such threads and I can tell is much better. So wanted to know how the
number 8 was choosen. I have also reduced the other number from 4 to 3.

Wouldnt should be some kind of managenment control on how many lwps the
firefox should claim? Instead leaving its modules to go on controlled
really only by their own local limits?

Regards,

On Tue, 17 Feb 2009, L. David Baron wrote:

> On Tuesday 2009-02-17 17:37 -0800, Takis Psarogiannakopoulos wrote:
> > mozilla/security/nss/cmd/vfyserv/vfyserv.h:#define MAX_THREADS 32
> > mozilla/security/nss/cmd/strsclnt/strsclnt.c:#define MAX_THREADS 128
> > mozilla/security/nss/cmd/selfserv/selfserv.c:#define MAX_THREADS 4096
>
> These look like command-line utilities that are part of the NSS
> library.  We import the full NSS source tree, but I don't think
> Firefox uses that part of it.
>
> -David
>
> --
> L. David Baron                                 http://dbaron.org/
> Mozilla Corporation                       http://www.mozilla.com/
> _______________________________________________
> dev-performance mailing list
> dev-performance@...
> https://lists.mozilla.org/listinfo/dev-performance
>

_______________________________________________
dev-performance mailing list
dev-performance@...
https://lists.mozilla.org/listinfo/dev-performance

[Boris Zbarsky]

Takis Psarogiannakopoulos wrote:
> Thats correct still 32, 128 and 4096 as max limit numbers isnt that too
> high? I would say 16 max ...

I'm not sure why those numbers are even being talked about in this
context....

> Anyways where I am most interested (thats why I posted) is the dns
> resolver. 8 threads with considerably descnt stacks to handle dns
> requests? Why not 4? In fact I have build a FI2 binary on my SVR4 with 4
> such threads and I can tell is much better. So wanted to know how the
> number 8 was choosen.

Looking at the CVS blame for that file (which you could have done
yourself, of course), points to that number being picked in
<https://bugzilla.mozilla.org/show_bug.cgi?id=205726>, or rather being
added there.  There's no justification given for the number.  You'd have
to ask Darin for his reasons, assuming he remembers 5+ years thence, but
I suspect it was just a "not too big, not too small" number.

As far as "much better", what exactly was much better?  Would limiting
the stack size of the threads this code creates actually help your
situation?  That could certainly be done (and has been considered; see
https://bugzilla.mozilla.org/show_bug.cgi?id=453403 where Patrick
proposed dropping the max stack size to 256KB; the only reasons that
didn't happen are that there was no obvious reason to do it and that
that sort of change should be done independently).

> I have also reduced the other number from 4 to 3.

That's up to you.  We could even make it a preference, I suppose, if
there is really a need for it to be tunable....

> Wouldnt should be some kind of managenment control on how many lwps the
> firefox should claim? Instead leaving its modules to go on controlled
> really only by their own local limits?

Extensions can spawn threads as needed.  The JS engine can spawn
threads.  Web workers can spawn threads.

If a global limit is implemented, what should happen when someone needs
a thread and the global limit has been hit?

It's really not clear to me what problem needs solving here.  Is the
worry the kernel overhead of multiple threads?  Is the worry lock
contention between the multiple threads?  Is the worry something else?

-Boris
_______________________________________________
dev-performance mailing list
dev-performance@...
https://lists.mozilla.org/listinfo/dev-performance

[Takis Psarogiannakopoulos]

Borris its too late here so I'l address only this:

On Tue, 17 Feb 2009, Boris Zbarsky wrote:

> Takis Psarogiannakopoulos wrote:
> > mozilla/netwerk/dns/src/nsHostResolver.cpp:#define MAX_THREADS 8
>
> These are threads that run only one small function; they spend most of
> their time waiting on getaddrinfo() system calls.  They shouldn't have
> much of a stack.  If this is being an issue in practice we can limit the
> max stack of these threads, but is it actually more expensive to allow
> (but not use) a bigger stack?
>

Yes but the function will normally will go in a shared library lresolv or
similar to address this. In other words will go through nspr
prnetdb.c For some reason the code there adds a considerable amount of
stack. There is a buffer to consider when you call reentrant gethostbyname
etc. My primary question here is: why the number 8? Why eg 4 or 5 which
seems more reasonable. If the threads are spending so little time active
why having 8 laying around?

Btw if you restrict their stack too much you would have a problem. Eg in
my SVR4 if they go out of stack they signal SIGSEGV to the process. This
is the case for those threads.
I would also say that in the design of an OS realistically there is a
consideration of how many lwps are in a process regarding real time
performance.


Regards,





_______________________________________________
dev-performance mailing list
dev-performance@...
https://lists.mozilla.org/listinfo/dev-performance

[Takis Psarogiannakopoulos]

On Tue, 17 Feb 2009, Boris Zbarsky wrote:

> Takis Psarogiannakopoulos wrote:
> > Thats correct still 32, 128 and 4096 as max limit numbers isnt that too
> > high? I would say 16 max ...
>
> I'm not sure why those numbers are even being talked about in this
> context....

I dont know about those. I just observed too much stck on dnsresolver and
noticed that locally on places mozilla sets limits for threads sometimes
really  a bit silly?

>
> Looking at the CVS blame for that file (which you could have done
> yourself, of course), points to that number being picked in
> <https://bugzilla.mozilla.org/show_bug.cgi?id=205726>, or rather being
> added there.  There's no justification given for the number.  You'd have
> to ask Darin for his reasons, assuming he remembers 5+ years thence, but
> I suspect it was just a "not too big, not too small" number.
>
I already answered asking this without having seen this message

> As far as "much better", what exactly was much better?

Not clogging the process with unessecary lwps. You only need to to create
those if you really need them. If you dont they why you created them in
the first place.? Is there is a preformance adavantage?

mobile OS's would care for those extra threads.

> Would limiting
> the stack size of the threads this code creates actually help your
> situation?  That could certainly be done (and has been considered; see
> https://bugzilla.mozilla.org/show_bug.cgi?id=453403 where Patrick
> proposed dropping the max stack size to 256KB; the only reasons that
> didn't happen are that there was no obvious reason to do it and that
> that sort of change should be done independently).
>
> > I have also reduced the other number from 4 to 3.
>

On DNS resolver there is considerable stack cause you have to carry
buffers around.

> If a global limit is implemented, what should happen when someone needs
> a thread and the global limit has been hit?

If what you say its true and we allow to spawn threads arbitrarily then
most possibly we have an issue really on the *real time* perfromance of
the evolving firefox process!

>
> It's really not clear to me what problem needs solving here.  Is the
> worry the kernel overhead of multiple threads?  Is the worry lock
> contention between the multiple threads?  Is the worry something else?
>
primarily kernel overhead of multiple threads.


Sorry didnt see your previous email before

Regards,
_______________________________________________
dev-performance mailing list
dev-performance@...
https://lists.mozilla.org/listinfo/dev-performance

[Boris Zbarsky]

Takis Psarogiannakopoulos wrote:
>> Looking at the CVS blame for that file (which you could have done
>> yourself, of course), points to that number being picked in
>> <https://bugzilla.mozilla.org/show_bug.cgi?id=205726>, or rather being
>> added there.  There's no justification given for the number.  You'd have
>> to ask Darin for his reasons, assuming he remembers 5+ years thence, but
>> I suspect it was just a "not too big, not too small" number.
>>
> I already answered asking this without having seen this message

You said:

   My primary question here is: why the number 8? Why eg 4 or 5 which
   seems more reasonable. If the threads are spending so little time
   active why having 8 laying around?

They don't usually lay around.  If they're idle, we shut them down.  If
they're around that means they're actually busy servicing DNS requests.

> Not clogging the process with unessecary lwps.

Please define "unnecessary"?  And "clogging"?  What's actually better
from a user-observable point of view?

> You only need to to create those if you really need them.
 > If you dont they why you created them in
> the first place.? Is there is a preformance adavantage?

I'm not sure what you're asking here.  There's absolutely a performance
advantage to parallelizing DNS requests if those requests are network-bound.

> On DNS resolver there is considerable stack cause you have to carry
> buffers around.

When you say "considerable" what sort of numbers are we talking about?

>> If a global limit is implemented, what should happen when someone needs
>> a thread and the global limit has been hit?
>
> If what you say its true and we allow to spawn threads arbitrarily then
> most possibly we have an issue really on the *real time* perfromance of
> the evolving firefox process!

Firefox isn't a realtime system.  If you mean user responsiveness, then
extra threads are (so far) a win in terms of taking work that can happen
in the background and doing it there without tying up the event loop.
As long as synchronization between the threads is kept to a minimum,
there is no problem as far as I can see.  What am I missing?

I agree that "too many" threads could cause contention for hardware
resources (the processor in particular), which could adversely affect
user responsiveness.  We're not hitting this, from what I can see, and
numbers of cores are growing faster than our thread usage at the moment.

Oh, and no one is allowed to spawn threads "arbitrarily".  If a module
needs to do work in the background it will typically create a single
thread for this work; there are a very few exceptions (like DNS and
networking) that use small threadpools because they often have very
parallel workloads.

> primarily kernel overhead of multiple threads.

Are we talking about very limited hardware with a really crappy kernel?
  Because I don't seem to recall LWP overhead on Linux, say, being
particularly noticeable (and the thread overhead is even less on other
OSes, last I checked).

-Boris
_______________________________________________
dev-performance mailing list
dev-performance@...
https://lists.mozilla.org/listinfo/dev-performance