Re: RestS3Service locking up

Hi,

There are two common reasons for these kinds of lock-ups:
- You are not closing the data streams for object downloads (this isn't likely to be the problem if you are performing getObjectDetails requests)
- You have a limited number of HTTP connections available, and you are starting more threads that use RestS3Service than there are available connections.

The RestS3Service class uses the HttpClient library's MultiThreadedHttpConnectionManager to manage HTTP connections which are a limited resource on most machines. In the jets3t.properties configuration file, the setting "httpclient.max-connections" controls how many simultaneous connections you can open to S3 at once. If you start more application threads than there are connections, the connection manager may not cope and can end up deadlocked.

There are a couple of potential solutions for this issue:
- Add some thread-management code to your application to prevent it from trying to start more RestS3Service operations than there are HTTP threads available.
- Take advantage of the JetS3t classes that have been written to handle this task for you. There are two classes available that will reliably perform multi-threaded requests to S3: the daunting S3ServiceMulti, and the simplified wrapper for this class S3ServiceSimpleMulti. Even if you need to write your own thread-handling code, I would recommend reviewing the S3ServiceMulti implementation to see how I did it. It may not be pretty, but it works...

To confirm 100% that thread handling is the issue, you can turn on debug logging for the "org.apache.commons.httpclient" class and see if you end up with lots of messages complaining that there are no more HTTP connections available.

Hope this helps,
James





--
http://www.jamesmurty.com