RestS3Service locking up

[bengarrett1971]

I'm using jets3t 0.6.0 in a web application (running in tomcat).

I've created a single RestS3Service instance that is being invoked by multiple threads, and I quite frequently see it locking up to such an extent that all 150 tomcat threads get used up, with no calls to methods like RestS3Service.getObjectDetails() seeming to return at all.

It feels like a Thread deadlock sort of problem, but I can't work out what I'm doing wrong.

Any suggestions would be welcome!

[James Murty-3]

Hi,

There are two common reasons for these kinds of lock-ups:
- You are not closing the data streams for object downloads (this isn't likely to be the problem if you are performing getObjectDetails requests)
- You have a limited number of HTTP connections available, and you are starting more threads that use RestS3Service than there are available connections.

The RestS3Service class uses the HttpClient library's MultiThreadedHttpConnectionManager to manage HTTP connections which are a limited resource on most machines. In the jets3t.properties configuration file, the setting "httpclient.max-connections" controls how many simultaneous connections you can open to S3 at once. If you start more application threads than there are connections, the connection manager may not cope and can end up deadlocked.

There are a couple of potential solutions for this issue:
- Add some thread-management code to your application to prevent it from trying to start more RestS3Service operations than there are HTTP threads available.
- Take advantage of the JetS3t classes that have been written to handle this task for you. There are two classes available that will reliably perform multi-threaded requests to S3: the daunting S3ServiceMulti, and the simplified wrapper for this class S3ServiceSimpleMulti. Even if you need to write your own thread-handling code, I would recommend reviewing the S3ServiceMulti implementation to see how I did it. It may not be pretty, but it works...

To confirm 100% that thread handling is the issue, you can turn on debug logging for the "org.apache.commons.httpclient" class and see if you end up with lots of messages complaining that there are no more HTTP connections available.

Hope this helps,
James

On Wed, Jun 4, 2008 at 9:16 PM, bengarrett1971 <ben.garrett@...> wrote:

I'm using jets3t 0.6.0 in a web application (running in tomcat).

I've created a single RestS3Service instance that is being invoked by
multiple threads, and I quite frequently see it locking up to such an extent
that all 150 tomcat threads get used up, with no calls to methods like
RestS3Service.getObjectDetails() seeming to return at all.

It feels like a Thread deadlock sort of problem, but I can't work out what
I'm doing wrong.

Any suggestions would be welcome!

--
View this message in context: http://www.nabble.com/RestS3Service-locking-up-tp17644333p17644333.html
Sent from the JetS3t Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

--
http://www.jamesmurty.com

[bengarrett1971]

Hi James,

Many thanks for the speedy and helpful reply.

I'm pretty confident that I must be hitting the default "httpclient.max-connections" limit of 10.  Because I'm using jets3t in a web application, with the standard one-Thread-per-request, the single RestS3Service would have been handling multiple concurrent calls when the site is busy, potentially many more than 10.

Unfortunately the S3ServiceMulti and S3ServiceSimpleMulti classes don't do what I'm after - I don't have a batch of S3 requests to do at any one time, but rather many individual ones, as many Threads (created by the app server rather than by my application code) do this:

if (file details were found on S3)
{
    return an S3 URL in the resulting page HTML
}
else
{
    return a local URL in the resulting page HTML
}

I've applied an ugly "brute force" fix, creating a new RestS3Service every time - not nice, but something I can optimise later.

Thanks again,
Ben

James Murty-3 wrote:

Hi,

There are two common reasons for these kinds of lock-ups:
- You are not closing the data streams for object downloads (this isn't
likely to be the problem if you are performing getObjectDetails requests)
- You have a limited number of HTTP connections available, and you are
starting more threads that use RestS3Service than there are available
connections.

The RestS3Service class uses the HttpClient library's
MultiThreadedHttpConnectionManager to manage HTTP connections which are a
limited resource on most machines. In the jets3t.properties configuration
file, the setting "httpclient.max-connections" controls how many
simultaneous connections you can open to S3 at once. If you start more
application threads than there are connections, the connection manager may
not cope and can end up deadlocked.

There are a couple of potential solutions for this issue:
- Add some thread-management code to your application to prevent it from
trying to start more RestS3Service operations than there are HTTP threads
available.
- Take advantage of the JetS3t classes that have been written to handle this
task for you. There are two classes available that will reliably perform
multi-threaded requests to S3: the daunting S3ServiceMulti, and the
simplified wrapper for this class S3ServiceSimpleMulti. Even if you need to
write your own thread-handling code, I would recommend reviewing the
S3ServiceMulti implementation to see how I did it. It may not be pretty, but
it works...

To confirm 100% that thread handling is the issue, you can turn on debug
logging for the "org.apache.commons.httpclient" class and see if you end up
with lots of messages complaining that there are no more HTTP connections
available.

Hope this helps,
James


On Wed, Jun 4, 2008 at 9:16 PM, bengarrett1971 <[e-mail]>
wrote:

>
> I'm using jets3t 0.6.0 in a web application (running in tomcat).
>
> I've created a single RestS3Service instance that is being invoked by
> multiple threads, and I quite frequently see it locking up to such an
> extent
> that all 150 tomcat threads get used up, with no calls to methods like
> RestS3Service.getObjectDetails() seeming to return at all.
>
> It feels like a Thread deadlock sort of problem, but I can't work out what
> I'm doing wrong.
>
> Any suggestions would be welcome!
>
> --
> View this message in context:
> http://www.nabble.com/RestS3Service-locking-up-tp17644333p17644333.html
> Sent from the JetS3t Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [e-mail]
> For additional commands, e-mail: [e-mail]
>
>


--
http://www.jamesmurty.com

[James Murty-2]

Hi Ben,

 

Many thanks for the speedy and helpful reply.

You're welcome
 

I'm pretty confident that I must be hitting the default
"httpclient.max-connections" limit of 10.  Because I'm using jets3t in a web
application, with the standard one-Thread-per-request, the single
RestS3Service would have been handling multiple concurrent calls when the
site is busy, potentially many more than 10.

I see, and the number of simultaneous connections is essentially unbounded because it depends on how many active users you have on the web site.
 

Unfortunately the S3ServiceMulti and S3ServiceSimpleMulti classes don't do
what I'm after - I don't have a batch of S3 requests to do at any one time,
but rather many individual ones, as many Threads (created by the app server
rather than by my application code) do this:

No, the existing multi-threaded classes won't help much in this situation.

I've applied an ugly "brute force" fix, creating a new RestS3Service every
time - not nice, but something I can optimise later.

Hmm, there should be a better way to do this without having to recreate the service instances. Apart from the inefficiency of doing this, you will leave open the possibility that you will one day hit the hard limit on simultaneous HTTP connections imposed by your operating system. Depending on the system you are running on, the maximum number of HTTP connections your OS will allow can be surprisingly low.

In the short term, you may be able to increase the "httpclient.max-connections" limit to match the upper limit of HTTP connections your OS can handle then configure your Tomcat server to serve no more than this number of clients. It's a shame applying a limit to the number of client you can serve, but it's a better option that exceeding your outgoing HTTP connection limit. And this will save you from having to recreate RestS3Service objects. Presumably Tomcat will do something sensible if excess client requests arrive, like queueing them up and replying if/when it can?

Longer term, you may have to centralise the S3 communications to be able to better manage the outgoing connections. If your user base grows and every incoming client request also involves a simultaneous outgoing HTTP request to S3, you may quickly reach your system's network connection limit. This may make your application unable to scale beyond a certain point on a single server.

On the bright side, it's an interesting problem ;)

James

--
http://www.jamesmurty.com

[James Murty-2]

It just occurred to me, another option may be to replace the MultiThreadedHttpConnectionManager in RestS3Client with a standard HttpConnectionManager that does not try to impose a limit on the number of simultaneous connections, see: http://hc.apache.org/httpclient-3.x/apidocs/org/apache/commons/httpclient/HttpConnectionManager.html

The HttpConnectionManager manager will simply block until a connection becomes available, or you can request connections with a pre-set timeout value. This could be the simplest solution for your situation, and if it works it is probably something I could work into the JetS3t library as your use-case seems reasonable.

James

On Wed, Jun 4, 2008 at 11:58 PM, James Murty <jmurty@...> wrote:

Hi Ben,

 

Many thanks for the speedy and helpful reply.

You're welcome
 

I'm pretty confident that I must be hitting the default
"httpclient.max-connections" limit of 10.  Because I'm using jets3t in a web
application, with the standard one-Thread-per-request, the single
RestS3Service would have been handling multiple concurrent calls when the
site is busy, potentially many more than 10.

I see, and the number of simultaneous connections is essentially unbounded because it depends on how many active users you have on the web site.
 

Unfortunately the S3ServiceMulti and S3ServiceSimpleMulti classes don't do
what I'm after - I don't have a batch of S3 requests to do at any one time,
but rather many individual ones, as many Threads (created by the app server
rather than by my application code) do this:

No, the existing multi-threaded classes won't help much in this situation.

I've applied an ugly "brute force" fix, creating a new RestS3Service every
time - not nice, but something I can optimise later.

Hmm, there should be a better way to do this without having to recreate the service instances. Apart from the inefficiency of doing this, you will leave open the possibility that you will one day hit the hard limit on simultaneous HTTP connections imposed by your operating system. Depending on the system you are running on, the maximum number of HTTP connections your OS will allow can be surprisingly low.

In the short term, you may be able to increase the "httpclient.max-connections" limit to match the upper limit of HTTP connections your OS can handle then configure your Tomcat server to serve no more than this number of clients. It's a shame applying a limit to the number of client you can serve, but it's a better option that exceeding your outgoing HTTP connection limit. And this will save you from having to recreate RestS3Service objects. Presumably Tomcat will do something sensible if excess client requests arrive, like queueing them up and replying if/when it can?

Longer term, you may have to centralise the S3 communications to be able to better manage the outgoing connections. If your user base grows and every incoming client request also involves a simultaneous outgoing HTTP request to S3, you may quickly reach your system's network connection limit. This may make your application unable to scale beyond a certain point on a single server.

On the bright side, it's an interesting problem ;)

James

--
http://www.jamesmurty.com

--
http://www.jamesmurty.com

[bengarrett1971]

James,

That HttpConnectionManager idea sounds nice; I'll take a look into it and let you know how I get on.

In the short term, I think you are right about altering "httpclient.max-connections" instead of creating new RestS3Services - it is certainly slowing things down (although I'm about to reduce the load by caching the keys of those files found on S3 so that only one initial call is required for each file).

Ben

James Murty-2 wrote:

It just occurred to me, another option may be to replace the
MultiThreadedHttpConnectionManager in RestS3Client with a standard
HttpConnectionManager that does not try to impose a limit on the number of
simultaneous connections, see:
http://hc.apache.org/httpclient-3.x/apidocs/org/apache/commons/httpclient/HttpConnectionManager.html

The HttpConnectionManager manager will simply block until a connection
becomes available, or you can request connections with a pre-set timeout
value. This could be the simplest solution for your situation, and if it
works it is probably something I could work into the JetS3t library as your
use-case seems reasonable.

James


On Wed, Jun 4, 2008 at 11:58 PM, James Murty <[e-mail]> wrote:

> Hi Ben,
>
>
>> Many thanks for the speedy and helpful reply.
>
>
> You're welcome
>
>
>> I'm pretty confident that I must be hitting the default
>> "httpclient.max-connections" limit of 10.  Because I'm using jets3t in a
>> web
>> application, with the standard one-Thread-per-request, the single
>> RestS3Service would have been handling multiple concurrent calls when the
>> site is busy, potentially many more than 10.
>
>
> I see, and the number of simultaneous connections is essentially unbounded
> because it depends on how many active users you have on the web site.
>
>
>> Unfortunately the S3ServiceMulti and S3ServiceSimpleMulti classes don't do
>> what I'm after - I don't have a batch of S3 requests to do at any one
>> time,
>> but rather many individual ones, as many Threads (created by the app
>> server
>> rather than by my application code) do this:
>
>
> No, the existing multi-threaded classes won't help much in this situation.
>
> I've applied an ugly "brute force" fix, creating a new RestS3Service every
>> time - not nice, but something I can optimise later.
>
>
> Hmm, there should be a better way to do this without having to recreate the
> service instances. Apart from the inefficiency of doing this, you will leave
> open the possibility that you will one day hit the hard limit on
> simultaneous HTTP connections imposed by your operating system. Depending on
> the system you are running on, the maximum number of HTTP connections your
> OS will allow can be surprisingly low.
>
> In the short term, you may be able to increase the
> "httpclient.max-connections" limit to match the upper limit of HTTP
> connections your OS can handle then configure your Tomcat server to serve no
> more than this number of clients. It's a shame applying a limit to the
> number of client you can serve, but it's a better option that exceeding your
> outgoing HTTP connection limit. And this will save you from having to
> recreate RestS3Service objects. Presumably Tomcat will do something sensible
> if excess client requests arrive, like queueing them up and replying if/when
> it can?
>
> Longer term, you may have to centralise the S3 communications to be able to
> better manage the outgoing connections. If your user base grows and every
> incoming client request also involves a simultaneous outgoing HTTP request
> to S3, you may quickly reach your system's network connection limit. This
> may make your application unable to scale beyond a certain point on a single
> server.
>
> On the bright side, it's an interesting problem ;)
>
> James
>
> --
> http://www.jamesmurty.com
>
>


--
http://www.jamesmurty.com