« Return to Thread: RestS3Service locking up

Re: RestS3Service locking up

by bengarrett1971 :: Rate this Message:

Hi James,

Many thanks for the speedy and helpful reply.

I'm pretty confident that I must be hitting the default "httpclient.max-connections" limit of 10. Because I'm using jets3t in a web application, with the standard one-Thread-per-request, the single RestS3Service would have been handling multiple concurrent calls when the site is busy, potentially many more than 10.

Unfortunately the S3ServiceMulti and S3ServiceSimpleMulti classes don't do what I'm after - I don't have a batch of S3 requests to do at any one time, but rather many individual ones, as many Threads (created by the app server rather than by my application code) do this:

if (file details were found on S3)
{
return an S3 URL in the resulting page HTML
}
else
{
return a local URL in the resulting page HTML
}

I've applied an ugly "brute force" fix, creating a new RestS3Service every time - not nice, but something I can optimise later.

Thanks again,
Ben

James Murty-3 wrote:

Hi,

There are two common reasons for these kinds of lock-ups:
- You are not closing the data streams for object downloads (this isn't
likely to be the problem if you are performing getObjectDetails requests)
- You have a limited number of HTTP connections available, and you are
starting more threads that use RestS3Service than there are available
connections.

The RestS3Service class uses the HttpClient library's
MultiThreadedHttpConnectionManager to manage HTTP connections which are a
limited resource on most machines. In the jets3t.properties configuration
file, the setting "httpclient.max-connections" controls how many
simultaneous connections you can open to S3 at once. If you start more
application threads than there are connections, the connection manager may
not cope and can end up deadlocked.

There are a couple of potential solutions for this issue:
- Add some thread-management code to your application to prevent it from
trying to start more RestS3Service operations than there are HTTP threads
available.
- Take advantage of the JetS3t classes that have been written to handle this
task for you. There are two classes available that will reliably perform
multi-threaded requests to S3: the daunting S3ServiceMulti, and the
simplified wrapper for this class S3ServiceSimpleMulti. Even if you need to
write your own thread-handling code, I would recommend reviewing the
S3ServiceMulti implementation to see how I did it. It may not be pretty, but
it works...

To confirm 100% that thread handling is the issue, you can turn on debug
logging for the "org.apache.commons.httpclient" class and see if you end up
with lots of messages complaining that there are no more HTTP connections
available.

Hope this helps,
James

On Wed, Jun 4, 2008 at 9:16 PM, bengarrett1971 <ben.garrett@moviestorm.co.uk>
wrote:

>
> I'm using jets3t 0.6.0 in a web application (running in tomcat).
>
> I've created a single RestS3Service instance that is being invoked by
> multiple threads, and I quite frequently see it locking up to such an
> extent
> that all 150 tomcat threads get used up, with no calls to methods like
> RestS3Service.getObjectDetails() seeming to return at all.
>
> It feels like a Thread deadlock sort of problem, but I can't work out what
> I'm doing wrong.
>
> Any suggestions would be welcome!
>
> --
> View this message in context:
> http://www.nabble.com/RestS3Service-locking-up-tp17644333p17644333.html
> Sent from the JetS3t Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@jets3t.dev.java.net
> For additional commands, e-mail: users-help@jets3t.dev.java.net
>
>

--
http://www.jamesmurty.com

« Return to Thread: RestS3Service locking up