SEVERE: All threads (700) are currently busy, waiting. Increase maxThreads (700) or check the servlet status

[Joan Monplet Ortega-2]

Hi,

 

We have Apache Web Server & Tomcat 6.x, both are connected throws
mod_proxy_ajp.

 

We have one instance Tomcat and different appBase (webapps).

 

We have a fatal problem, our Tomcat throws the next exception:

 

SEVERE: All threads (700) are currently busy, waiting. Increase
maxThreads (700) or check the servlet status

 

And we checked all servlets and we are sure that all cursors to database
are closed (statement, resultset, etc...).

 

When we have high traffic to our website the problem occurs.

 

Do you have any idea to resolve the problem?

 

Thanks

 

Joan

 

 

 

 

[awarnier]

Joan Monplet Ortega wrote:
>
> SEVERE: All threads (700) are currently busy, waiting. Increase
> maxThreads (700) or check the servlet status
>
That sounds like a lot of threads.
>  
>
> And we checked all servlets and we are sure that all cursors to database
> are closed (statement, resultset, etc...).
>
Yeah, but it looks like they are busy anyway.  What are they doing ?

>  
>
> When we have high traffic to our website the problem occurs.
>
No kidding.
Is that one Tomcat ? And if it is, how much physical memory is available
on that system ?

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Caldarale, Charles R]

> From: Joan Monplet Ortega [mailto:joan@...]
> Subject: SEVERE: All threads (700) are currently busy, waiting.
> Increase maxThreads (700) or check the servlet status
>
> We have Apache Web Server & Tomcat 6.x, both are connected throws
> mod_proxy_ajp.

Exact Tomcat version, please.  6.x covers a multitude of sins.

> We have one instance Tomcat and different appBase (webapps).

Different appBase (multiple <Host> elements) or a different docBase for each webapp?  (Probably doesn't matter.)

> SEVERE: All threads (700) are currently busy, waiting. Increase
> maxThreads (700) or check the servlet status
>
> When we have high traffic to our website the problem occurs.

And a thread dump shows...??
http://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_my_running_webapp_.3F

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Joan Monplet Ortega-2]

Yeah, It's the one Tomcat. We have 4GB RAM on production Server and We run it with 1GB.

The version Tomcat is "Apache Tomcat/6.0.18".

Also, We're working with Lucene but I'm not sure that the problem is here.

We're looking Server Status from Apache Web Server and We have many keepalive connections and If we check the Tomcat' status (I'm using Lambda Probe), We can see that exists a lot of connection with KeepAlive.

We're using mod_proxy_ajp (Apache 2.2.x) for connecting to Apache Web Server & Tomcat.

When we restart Tomcat Web Server, the number of Threads (Stage KeepAlive) grows, and We don't understand why occur that. It seems that Tomcat no unblock Threads whose stages are unblocked. So, In 2 or 3 hours the current Threads busy increase until the Max threads.


What's the best way connecting Apache & Tomcat? mod_jk or mod_proxy_ajp?

What's the meaning stage keepalive in Tomcat?

How to we find the problematic servlet that does the many connections are blocked? Because in LambdaProbe We can't find the exactly servlet url.

Thanks


-----Mensaje original-----
De: André Warnier [mailto:aw@...]
Enviado el: jueves, 05 de noviembre de 2009 23:52
Para: Tomcat Users List
Asunto: Re: SEVERE: All threads (700) are currently busy, waiting. Increase maxThreads (700) or check the servlet status

Joan Monplet Ortega wrote:
>
> SEVERE: All threads (700) are currently busy, waiting. Increase
> maxThreads (700) or check the servlet status
>
That sounds like a lot of threads.
>  
>
> And we checked all servlets and we are sure that all cursors to database
> are closed (statement, resultset, etc...).
>
Yeah, but it looks like they are busy anyway.  What are they doing ?

>  
>
> When we have high traffic to our website the problem occurs.
>
No kidding.
Is that one Tomcat ? And if it is, how much physical memory is available
on that system ?

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[awarnier]

Joan Monplet Ortega wrote:
> Yeah, It's the one Tomcat. We have 4GB RAM on production Server and We run it with 1GB.
>
> The version Tomcat is "Apache Tomcat/6.0.18".
>
> Also, We're working with Lucene but I'm not sure that the problem is here.
>
> We're looking Server Status from Apache Web Server and We have many keepalive connections
and If we check the Tomcat' status (I'm using Lambda Probe), We can see
that exists a lot of connection with KeepAlive.


How long is the KeepAlive in Apache ?

>
> We're using mod_proxy_ajp (Apache 2.2.x) for connecting to Apache Web Server & Tomcat.
>
...

>
>
> What's the best way connecting Apache & Tomcat? mod_jk or mod_proxy_ajp?
>
It depends.. ;-)
mod_jk has been around a lot longer, and there are probably a lot more
users, and there is probably a lot more experience behind it.
But then, they must have said that also about the steam engine, when the
Otto engine came about..

A word on KeepAlive :
The idea if keep-alive connections originally was :
- setting up new TCP connections and closing them down consumes resources
- WWW pages usually contain several links to stylesheets, images,
javascript etc..
- so when the browser connects to get the main page and the server
answers with the main page, the browser may be able to re-use the same
connection to ask for the images, stylesheets, etc..
This avoids having to set up a new connection for each of these objects,
and thus saves time and resources.
Which in principle is good.

But, during a keepalive connection, usually the server needs to dedicate
one process, or child or thread, to this connection.  So this
child/thread is blocked, and cannot server requests from other browsers.

If the KeepAlive timeout is very long, it means that when the browser
has finished requesting all the objects from the page, the connection
still stays alive for a while, because the server waits to see if there
are no more requests coming on the connection.
Only after this timeout has expired, can the server child/thread decide
that there is nothing anymore coming, close the connection, and be
recycled for other connections from other clients.

If your KeepAlive is set for 30 seconds e.g., then it means that for 30
seconds after the last browser request, the child/thread keeps waiting,
doing nothing, and being unavailable for other requests.
That may be what all your threads are doing (or rather, not doing).

I would think that nowadays, a setting of 3 sec. is largely sufficient
in most cases.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[el al]

As Charles suggested, you need a thread dump. There is too much
complexity in your setup to be able to find your problem without
actually inspecting the machine, but a thread dump can take you right
to the source of the problem in a matter of minutes.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Joan Monplet Ortega-2]

This is our configuration about KeepAlive

KeepAlive On
KeepAliveTimeout 3
MaxKeepAliveRequests 500
KeepAliveTimeout 15



-----Mensaje original-----
De: André Warnier [mailto:aw@...]
Enviado el: viernes, 06 de noviembre de 2009 9:03
Para: Tomcat Users List
Asunto: Re: SEVERE: All threads (700) are currently busy, waiting. Increase maxThreads (700) or check the servlet status

Joan Monplet Ortega wrote:
> Yeah, It's the one Tomcat. We have 4GB RAM on production Server and We run it with 1GB.
>
> The version Tomcat is "Apache Tomcat/6.0.18".
>
> Also, We're working with Lucene but I'm not sure that the problem is here.
>
> We're looking Server Status from Apache Web Server and We have many keepalive connections
and If we check the Tomcat' status (I'm using Lambda Probe), We can see
that exists a lot of connection with KeepAlive.


How long is the KeepAlive in Apache ?

>
> We're using mod_proxy_ajp (Apache 2.2.x) for connecting to Apache Web Server & Tomcat.
>
...

>
>
> What's the best way connecting Apache & Tomcat? mod_jk or mod_proxy_ajp?
>
It depends.. ;-)
mod_jk has been around a lot longer, and there are probably a lot more
users, and there is probably a lot more experience behind it.
But then, they must have said that also about the steam engine, when the
Otto engine came about..

A word on KeepAlive :
The idea if keep-alive connections originally was :
- setting up new TCP connections and closing them down consumes resources
- WWW pages usually contain several links to stylesheets, images,
javascript etc..
- so when the browser connects to get the main page and the server
answers with the main page, the browser may be able to re-use the same
connection to ask for the images, stylesheets, etc..
This avoids having to set up a new connection for each of these objects,
and thus saves time and resources.
Which in principle is good.

But, during a keepalive connection, usually the server needs to dedicate
one process, or child or thread, to this connection.  So this
child/thread is blocked, and cannot server requests from other browsers.

If the KeepAlive timeout is very long, it means that when the browser
has finished requesting all the objects from the page, the connection
still stays alive for a while, because the server waits to see if there
are no more requests coming on the connection.
Only after this timeout has expired, can the server child/thread decide
that there is nothing anymore coming, close the connection, and be
recycled for other connections from other clients.

If your KeepAlive is set for 30 seconds e.g., then it means that for 30
seconds after the last browser request, the child/thread keeps waiting,
doing nothing, and being unavailable for other requests.
That may be what all your threads are doing (or rather, not doing).

I would think that nowadays, a setting of 3 sec. is largely sufficient
in most cases.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[awarnier]

Wait...

Joan Monplet Ortega wrote:
> This is our configuration about KeepAlive
>
> KeepAlive On
> KeepAliveTimeout 3  <======   this
> MaxKeepAliveRequests 500
> KeepAliveTimeout 15 <====== seems contradicted by this
>
Remove the second occurrence and try again,
But as the others say, a thread dump of Tomcat would tell you what these
busy threads are really busy doing.

Also, MaxKeepAliveRequests 500, I think, should be reset to the default
(unlimited).
By setting it to 500, you are forcing Apache
  1) to count the requests on the connection
  2) to force a disconnect after 500 requests, no matter if that is in
the middle of a page.
That sounds counterproductive.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Tobias Crefeld-2]

Am Fri, 6 Nov 2009 08:24:14 +0100
schrieb "Joan Monplet Ortega" <joan@...>:

> Yeah, It's the one Tomcat. We have 4GB RAM on production Server and
> We run it with 1GB.

Not too much RAM... how many clients are using this system
simultaneously? Our smallest machines (< 10 seats) have this amount of
RAM configured.

Just as an idea: Do you run a network monitoring with a smaller testing
period?


> When we restart Tomcat Web Server, the number of Threads (Stage
> KeepAlive) grows, and We don't understand why occur that. It seems
> that Tomcat no unblock Threads whose stages are unblocked. So, In 2
> or 3 hours the current Threads busy increase until the Max threads.

Sounds like a bug of the web application.


> What's the best way connecting Apache & Tomcat? mod_jk or
> mod_proxy_ajp?

mod_jk is more complex to configure but has more options.
There is a bug in mod_proxy_ajp if you have large http-1.0-POSTs but
usually it is a good choice.


> How to we find the problematic servlet that does the many connections
> are blocked? Because in LambdaProbe We can't find the exactly servlet
> url.

In applications/(select app)/sessions you get an overview which
web-application (=context) is using the connections.

In detail (without statistics) Tomcat's catalina.out and especially
Apache2's access_log show you which URIs are initiating / keeping alive
the connections.
Afterwards you can search in Lambda Probe under applications/(select
app)/Servlets which Servlet would handle this URIs. There you will also
find some general statistics how much resources each Servlet comsumes
and how often it has been called.


RU,
 Tobias.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Christopher Schultz-2]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

André,

On 11/6/2009 3:02 AM, André Warnier wrote:
> But, during a keepalive connection, usually the server needs to dedicate
> one process, or child or thread, to this connection.  So this
> child/thread is blocked, and cannot server requests from other browsers.
>
> If the KeepAlive timeout is very long, it means that when the browser
> has finished requesting all the objects from the page, the connection
> still stays alive for a while, because the server waits to see if there
> are no more requests coming on the connection.

Or, the client can close the connection which is a pretty good way to
decide that there are no more requests to be made on that connection.

Note that Tomcat's NIO connector puts worker threads back into the
thread pool between keepalive requests on a single connection, and you
can get better thread usage in that way.

> I would think that nowadays, a setting of 3 sec. is largely sufficient
> in most cases.

+1

If the client can't make another request in a keepalive connection fast
enough, it can always open up a new connection and make a new request.

- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkr5xYQACgkQ9CaO5/Lv0PA4iACfXjrmHkgcEAd47RTcMgmyPChF
/mgAnRHLk2CSnliOOe+ROD4EUFxm8CBU
=GP7F
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Christopher Schultz-2]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Tobias,

On 11/6/2009 8:56 AM, Tobias Crefeld wrote:
> Am Fri, 6 Nov 2009 08:24:14 +0100
> schrieb "Joan Monplet Ortega" <joan@...>:
>
>> Yeah, It's the one Tomcat. We have 4GB RAM on production Server and
>> We run it with 1GB.
>
> Not too much RAM... how many clients are using this system
> simultaneously? Our smallest machines (< 10 seats) have this amount of
> RAM configured.

Wow, you must have pretty heavy requests.

We have hundreds of users on a machine that only recently needed its
heap size increased from the default 64MiB heap to make room for them.
We do tend to keep things light, though.

> In applications/(select app)/sessions you get an overview which
> web-application (=context) is using the connections.

Speaking of sessions, if you have synchronized access to the session or
objects in the session, you may be locking-up threads waiting for those
locks to be released.

- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkr5xjcACgkQ9CaO5/Lv0PDuIQCgpQgT25REehyGCQ5iZuEuzVTz
l5cAoK/nSJ3ON+alUFIiODUirzrkQWeb
=RH+Z
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...