Question on HTTPClient-675

[D H-9]

Hi,
I know that the developers of this project are very busy and don't mean to
intrude.  I'm tasked with finding why a problem is occurring in an
application of ours and haven't been having very much luck with it.  We are
still using HttpClient 3.0.1 and encounter a problem where after a few days
to a week of making HttpClient connections to an SSL site, the application
container will be working fine and then suddenly the response times will get
longer and longer, then they will go through the roof and the application
will stop responding completely until it's restarted.  From what I
understand, a few other people where I work who have looked at this problem
have used profiling tools to narrow down where the problem is happening to
HttpClient but they aren't sure why it is happening.

I saw there is a bug HTTPClient-675 that was fixed by Roland Weber in
version 3.1.  It talks about thread-starving in the
MultiThreadedHttpConnectionManager and made me wonder if we were seeing
something similar occurring.  I emailed Roland and asked if he might know a
test case in 3.0.1 that would cause this problem to occur.  He replied back
and stated that he is no longer on the project and hasn't been involved for
a year, but that I should email the developer mailing list to see if anyone
might know.

The suspected code isn't doing anything complicated, it just looks like
this:

        HttpClient client = new HttpClient();
        PostMethod post = new PostMethod(inUrl);
        post.setRequestEntity(new StringRequestEntity(inPayload, null,
null));
        int statusCode = client.executeMethod(post);
        log.debug(new StringBuilder("inUrl=").append(inUrl).append(",
inPayload=")
                    .append(inPayload).append(", statusCode=")
                    .append(statusCode).toString());
        String payload = readPayload(post);
        return payload;

The readPayload method just uses post.getResponseBodyAsStream and a
BufferedReader to read the response.
The BufferedReader is then closed in a finally block.

If anyone might know how to reproduce that issue or has any thoughts on what
might be happening I'd appreciate it.  The ultimate goal is to be able to
get it to happen with just the isolated code sample, try an upgraded jar
file and if the problem doesn't happen after the upgrade then we could move
forward with doing an upgrade to the jar in Production to see if this
problem goes away.  I've been using JMeter to simulate the calls that were
being made but haven't had much luck with it and i've been working at this
problem for a few weeks and looking on newsgroups for people who might be
having similar problems without much luck.

Sincerely,
David Hamilton

[Ortwin Glück]

David,

> We are still using HttpClient 3.0.1

Consider upgrading.

> I saw there is a bug HTTPClient-675 that was fixed by Roland Weber in
> version 3.1.  It talks about thread-starving in the
> MultiThreadedHttpConnectionManager and made me wonder if we were seeing
> something similar occurring.  

As you are not using the MTCM, that issue does not affect you.

> I emailed Roland

Generally you do not email indidivual developers. They might not even bother
responding.

>
>         HttpClient client = new HttpClient();
>         PostMethod post = new PostMethod(inUrl);
>         post.setRequestEntity(new StringRequestEntity(inPayload, null,
> null));
>         int statusCode = client.executeMethod(post);
>         log.debug(new StringBuilder("inUrl=").append(inUrl).append(",
> inPayload=")
>                     .append(inPayload).append(", statusCode=")
>                     .append(statusCode).toString());
>         String payload = readPayload(post);
>         return payload;
>
> The readPayload method just uses post.getResponseBodyAsStream and a
> BufferedReader to read the response.
> The BufferedReader is then closed in a finally block.

This code does not call post.releaseConnection() at all! It will leak
connections. It could be that you are running out of file descriptors. The call
should be put into a finally block. See the example code on the website.

> Sincerely,
> David Hamilton

Hope that helps,

Ortwin


--
[web]  http://www.odi.ch/
[blog] http://www.odi.ch/weblog/
[pgp]  key 0x81CF3416
       finger print F2B1 B21F F056 D53E 5D79 A5AF 02BE 70F5 81CF 3416

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[D H-9]

Ortwin,

2009/6/15 Ortwin Glück <odi@...>

> David,
>
> > We are still using HttpClient 3.0.1
>
> Consider upgrading.
>
Upgrading is definitely my goal, but I just started working here 3 weeks ago
and it's a large university with a lot of red tape to go through to get
something upgraded.  They want proof this will fix their problem before
upgrading.


>
> > I saw there is a bug HTTPClient-675 that was fixed by Roland Weber in
> > version 3.1.  It talks about thread-starving in the
> > MultiThreadedHttpConnectionManager and made me wonder if we were seeing
> > something similar occurring.
>
> As you are not using the MTCM, that issue does not affect you.
>
That makes sense, my thought was that maybe the condition that caused the
problem he saw would also cause our problem to happen but it looks like that
was a false hope.

> > I emailed Roland
>
> Generally you do not email indidivual developers. They might not even
> bother
> responding.
>
I was actually surprised he did reply, and was extremely polite in the email
to him, I just hadn't seen anybody else's name on the bugzilla ticket and
wasn't sure if anyone else would know the details.  I will definitely
remember that for the future.

>
> >
> >         HttpClient client = new HttpClient();
> >         PostMethod post = new PostMethod(inUrl);
> >         post.setRequestEntity(new StringRequestEntity(inPayload, null,
> > null));
> >         int statusCode = client.executeMethod(post);
> >         log.debug(new StringBuilder("inUrl=").append(inUrl).append(",
> > inPayload=")
> >                     .append(inPayload).append(", statusCode=")
> >                     .append(statusCode).toString());
> >         String payload = readPayload(post);
> >         return payload;
> >
> > The readPayload method just uses post.getResponseBodyAsStream and a
> > BufferedReader to read the response.
> > The BufferedReader is then closed in a finally block.
>
> This code does not call post.releaseConnection() at all! It will leak
> connections. It could be that you are running out of file descriptors. The
> call
> should be put into a finally block. See the example code on the website.
>

That was one of my recommendations the first day I looked at the code but
they want me to be able to duplicate the problem with my sample code first.
Hopefully your reply will help bolster the case for correcting that
oversight in their code.


>
> > Sincerely,
> > David Hamilton
>
> Hope that helps,
>
> Ortwin
>
Thank you very much for your reply and help, I appreciate it.

Sincerely,
David Hamilton

>
>
> --
> [web]  http://www.odi.ch/
> [blog] http://www.odi.ch/weblog/
> [pgp]  key 0x81CF3416
>       finger print F2B1 B21F F056 D53E 5D79 A5AF 02BE 70F5 81CF 3416
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@...
> For additional commands, e-mail: dev-help@...
>
>

[Ortwin Glück]

D H wrote:

>> This code does not call post.releaseConnection() at all! It will leak
>> connections. It could be that you are running out of file descriptors. The
>> call
>> should be put into a finally block. See the example code on the website.
>>
> That was one of my recommendations the first day I looked at the code but
> they want me to be able to duplicate the problem with my sample code first.
> Hopefully your reply will help bolster the case for correcting that
> oversight in their code.
>

OMG. Failure to call releaseConnection() is a gross mistake and should be fixed
in any case. Obviously the original author of this code has never looked at the
provided examples, or read the API Doc.

Anyway if this is so hard to reproduce and profiling didn't give you any idea
what's broken, why do you suspect something in this particular piece of code is
the cause of your problem?

Ortwin

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[D H-9]

2009/6/15 Ortwin Glück <odi@...>

>
>
> D H wrote:
>
>> This code does not call post.releaseConnection() at all! It will leak
>>> connections. It could be that you are running out of file descriptors.
>>> The
>>> call
>>> should be put into a finally block. See the example code on the website.
>>>
>>>  That was one of my recommendations the first day I looked at the code
>> but
>> they want me to be able to duplicate the problem with my sample code
>> first.
>> Hopefully your reply will help bolster the case for correcting that
>> oversight in their code.
>>
>>
> OMG. Failure to call releaseConnection() is a gross mistake and should be
> fixed in any case. Obviously the original author of this code has never
> looked at the provided examples, or read the API Doc.
>

I agree about this, they wanted proof of a fix before any changes would be
made and my manager is still saying there must be proof before code changes
even now after I showed him your email and the documentation from the site.



>
> Anyway if this is so hard to reproduce and profiling didn't give you any
> idea what's broken, why do you suspect something in this particular piece of
> code is the cause of your problem?


That's a very good question, I was given the source code, no explanation of
their code and told HttpClient was the problem.  It is a rather large
codebase so I'm taking their word for it.  They've apparently worked on it
sporadically for a couple of months to isolate it to HttpClient, and my task
is to prove it is causing this problem with JMeter in a self-contained code
sample.  This problem has only been seen in Production and only after almost
a week of running 24/7 so it's hard to duplicate it easily.  I've sent over
two hundred thousand HttpClient requests without seeing the problem so I'd
rather see this code fix go to Production and test that way personally.


I really appreciate you taking the time to answer my emails, thank you very
much.

Sincerely,
David Hamilton


>
> Ortwin
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@...
> For additional commands, e-mail: dev-help@...
>
>

[olegk]

Ortwin Glück wrote:

> David,
>
>> We are still using HttpClient 3.0.1
>
> Consider upgrading.
>
>> I saw there is a bug HTTPClient-675 that was fixed by Roland Weber in
>> version 3.1.  It talks about thread-starving in the
>> MultiThreadedHttpConnectionManager and made me wonder if we were seeing
>> something similar occurring.  
>
> As you are not using the MTCM, that issue does not affect you.
>
>> I emailed Roland
>
> Generally you do not email indidivual developers. They might not even bother
> responding.
>
>>         HttpClient client = new HttpClient();
>>         PostMethod post = new PostMethod(inUrl);
>>         post.setRequestEntity(new StringRequestEntity(inPayload, null,
>> null));
>>         int statusCode = client.executeMethod(post);
>>         log.debug(new StringBuilder("inUrl=").append(inUrl).append(",
>> inPayload=")
>>                     .append(inPayload).append(", statusCode=")
>>                     .append(statusCode).toString());
>>         String payload = readPayload(post);
>>         return payload;
>>
>> The readPayload method just uses post.getResponseBodyAsStream and a
>> BufferedReader to read the response.
>> The BufferedReader is then closed in a finally block.
>
> This code does not call post.releaseConnection() at all! It will leak
> connections. It could be that you are running out of file descriptors. The call
> should be put into a finally block. See the example code on the website.
>

David,

Another big issue is that a new instance of HttpClient is created for
each new request. This is enormously wasteful, as every time there
likely to be an open connection going out of scope, which needs to be
GC-ed in order to free up system resources.

You should re-use HttpClient instance for multiple request and shut down
its connection manager when it is no longer used.

Besides, upgrading to Httpclient 4.0 may be a big jump, but there is no
excuse for not upgrading to HttpClient 3.1

Oleg


>> Sincerely,
>> David Hamilton
>
> Hope that helps,
>
> Ortwin
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[Ortwin Glück]

D H wrote:
>
> I agree about this, they wanted proof of a fix before any changes would be
> made and my manager is still saying there must be proof before code changes
> even now after I showed him your email and the documentation from the site.

Nice way of thinking -- at a university where time and money are infinite
resources. Wishful thinking anywhere else. Enter The Real World.

>> Anyway if this is so hard to reproduce and profiling didn't give you any
>> idea what's broken, why do you suspect something in this particular piece of
>> code is the cause of your problem?
>
>
> That's a very good question, I was given the source code, no explanation of
> their code and told HttpClient was the problem.  

Why am I not convinced? But it's well possible. My best advice I can give is:
Upgrade for God's sake, and fix obvious mistakes in the use of the API. Use some
monitoring tools like JConsole / JVisualVM, jmap, netstat, top and the like to
see if you have a garbage collection problem or any obvious resource leaks, then
take appropriate action.

 > It is a rather large
> codebase so I'm taking their word for it.  They've apparently worked on it
> sporadically for a couple of months to isolate it to HttpClient, and my task
> is to prove it is causing this problem with JMeter in a self-contained code
> sample.  This problem has only been seen in Production and only after almost
> a week of running 24/7 so it's hard to duplicate it easily.  I've sent over
> two hundred thousand HttpClient requests without seeing the problem so I'd
> rather see this code fix go to Production and test that way personally.

Sounds like a a typical production problem to me: it can take weeks to see it,
there is no way you can trigger it on purpose and maybe it has only ever been
encountered on the production system.

Face it, you will not reproduce it locally in reasonable time. It is maybe
dependent on the workload you are running. Maybe it's even platform specific and
may not trigger on your testing platform. And with platform I don't mean just
the OS. Also the processor type (Single core, Multi core) can make a huge
difference.

What can really help you is to expect the situation in production. And instead
of panicing and quickly restarting, take your time, having the right tools at
hand to find out what's going on in this moment. Maybe even a "post mortem" job
that gathers useful information in case this happens during everyone is
sleeping. Is it swapping? Has the VM run out of memory (stack, heap, perm gen,
code) and is constantly GCing? Has the OS run out of file descriptors? To which
signals does it react? Is it creating threads at a high rate? Are there just too
many runnable threads? Is it busy waiting? Is it looping endlessly? Is it I/O
bound? Is it lock contented or even deadlocked? Is it blocking on I/O or
network? Is it waiting for the DB? What's going on on the DB? What's going on on
the network? Is your log detailes enough to give you the information you need?

If all fails you, you may have to live with it and rather setup monitoring
infrastructure that can reliably detect the situation and restart the process.

>
> I really appreciate you taking the time to answer my emails, thank you very
> much.
>
> Sincerely,
> David Hamilton

Ortwin

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[D H-9]

Oleg and Ortwin,

On Mon, Jun 15, 2009 at 4:07 PM, Oleg Kalnichevski <olegk@...> wrote:

> David,
>
> Another big issue is that a new instance of HttpClient is created for each
> new request. This is enormously wasteful, as every time there likely to be
> an open connection going out of scope, which needs to be GC-ed in order to
> free up system resources.


Thank you for all of the advice, I've been talking with my boss quite a bit
and he wants me to document the best practices for HttpClient so that we can
have the developers fix the problems with their code.  The application is a
website that uses well over a hundred different URLs.  I found a
MultiThreadedExample at
http://svn.apache.org/viewvc/httpcomponents/oac.hc3x/trunk/src/examples/MultiThreadedExample.java?revision=608014but
it looks like you have to declare all of the URLs that it is going to
use?

Could I just declare a static MultiThreadedHttpConnectionManager in a
Servlet, statically set the parameters to initialize it and then use it like
this:

        HttpClient client = new HttpClient(connectionManager);
        PostMethod post = new PostMethod(inUrl);
        try {
            post.setRequestEntity(new StringRequestEntity(inPayload, null,
null));
            int statusCode = client.executeMethod(post);
            payload = SlowUtilities.readPayload(post);
        } finally {
            post.releaseConnection();
        }

Or do you recommend using the SimpleHttpConnectionManager for situations
like ours?


> You should re-use HttpClient instance for multiple request and shut down
> its connection manager when it is no longer used.
>

I'm not sure when the connection manager would be shut down though, this is
a 24/7 application for a medical center and really only goes down when this
slowdown problem occurs that renders the site unusable.  Could I just call
closeIdleConnections() every once in a while to do some cleanup?

>
> Besides, upgrading to Httpclient 4.0 may be a big jump, but there is no
> excuse for not upgrading to HttpClient 3.1
>
I'm working on getting them to agree to upgrade to HttpClient 3.1.  The
application depends on a lot of different libraries so I don't know how
easily they could upgrade to HttpClient 4.0 but I'll look into it.

I greatly appreciate everyone's help,
David Hamilton

[olegk]

D H wrote:

> Oleg and Ortwin,
>
> On Mon, Jun 15, 2009 at 4:07 PM, Oleg Kalnichevski <olegk@...> wrote:
>
>> David,
>>
>> Another big issue is that a new instance of HttpClient is created for each
>> new request. This is enormously wasteful, as every time there likely to be
>> an open connection going out of scope, which needs to be GC-ed in order to
>> free up system resources.
>
>
> Thank you for all of the advice, I've been talking with my boss quite a bit
> and he wants me to document the best practices for HttpClient so that we can
> have the developers fix the problems with their code.  The application is a
> website that uses well over a hundred different URLs.  I found a
> MultiThreadedExample at
> http://svn.apache.org/viewvc/httpcomponents/oac.hc3x/trunk/src/examples/MultiThreadedExample.java?revision=608014but
> it looks like you have to declare all of the URLs that it is going to
> use?
>

No, you certainly do not have to.


> Could I just declare a static MultiThreadedHttpConnectionManager in a
> Servlet, statically set the parameters to initialize it and then use it like
> this:
>

You will be much better off using ServletContextListener to create and
shut down MultiThreadedHttpConnectionManager and ServletContext to store
the reference to that instance.

>         HttpClient client = new HttpClient(connectionManager);
>         PostMethod post = new PostMethod(inUrl);
>         try {
>             post.setRequestEntity(new StringRequestEntity(inPayload, null,
> null));
>             int statusCode = client.executeMethod(post);
>             payload = SlowUtilities.readPayload(post);
>         } finally {
>             post.releaseConnection();
>         }
>
> Or do you recommend using the SimpleHttpConnectionManager for situations
> like ours?
>

One should definitely use MultiThreadedHttpConnectionManager in a
servlet container.


>
>> You should re-use HttpClient instance for multiple request and shut down
>> its connection manager when it is no longer used.
>>
>
> I'm not sure when the connection manager would be shut down though, this is
> a 24/7 application for a medical center and really only goes down when this
> slowdown problem occurs that renders the site unusable.

ServletContextListener is your friend.


  Could I just call
> closeIdleConnections() every once in a while to do some cleanup?

Yes, you should.

>
>> Besides, upgrading to Httpclient 4.0 may be a big jump, but there is no
>> excuse for not upgrading to HttpClient 3.1
>>
> I'm working on getting them to agree to upgrade to HttpClient 3.1.  The
> application depends on a lot of different libraries so I don't know how
> easily they could upgrade to HttpClient 4.0 but I'll look into it.
>

HttpClient 4.0 is _massively_ better than 3.x version and is better
documented. You should tell your boss to seriously consider upgrading.
Besides, 3.x code line is effectively at the end of life.

It is just a matter of time I _personally_ will stop answering any
questions about HttpClient 3.x

Cheers

Evil Comrade Oleg

> I greatly appreciate everyone's help,
> David Hamilton
>


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[D H-9]

Oleg,

It has taken some time, but the developers are going to make the recommended
changes that you and Ortwin suggested.

On Wed, Jun 17, 2009 at 2:25 PM, Oleg Kalnichevski <olegk@...> wrote:

> D H wrote:
>
> You will be much better off using ServletContextListener to create and shut
> down MultiThreadedHttpConnectionManager and ServletContext to store the
> reference to that instance.
>
>> One should definitely use MultiThreadedHttpConnectionManager in a servlet
> container.
>

One question that one of our developers had was where to put the single
instance of MultiThreadedHttpConnectionManager to be able to use it across
threads from an EJB?
He will use the ServletContext and ServletContextListener for the
applications but isn't sure for EJBs.


> HttpClient 4.0 is _massively_ better than 3.x version and is better
> documented. You should tell your boss to seriously consider upgrading.
> Besides, 3.x code line is effectively at the end of life.
>
> It is just a matter of time I _personally_ will stop answering any
> questions about HttpClient 3.x


Believe it or not, once the developers saw that they had to upgrade, they
decided to go to the latest version themselves and only have to change the
code once so they are working to convert it now to 4.0.1.  I found the other
application having problems was running on the 2.0.2 version of HttpClient
but that will be fixed very soon.


> Cheers
> Evil Comrade Oleg


Thanks again for your help,
David Hamilton

[Ortwin Glück]

David,

D H wrote:
> One question that one of our developers had was where to put the single
> instance of MultiThreadedHttpConnectionManager to be able to use it across
> threads from an EJB?
> He will use the ServletContext and ServletContextListener for the
> applications but isn't sure for EJBs.

It's a bit of a matter of taste. I think it makes a good candidate for an MBean.
If JBoss is your appserver, you can easily create a ServiceMBean for this
purpose and set it up in the jboss-app.xml of your EAR or any other jboss.xml
(of an EJB-JAR or SAR) of your choice. This way you could also easily add some
MBean operations & properties for adminitrative purposes like flushing the
cache, or returning statistics.

Odi

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[olegk]

On Mon, Jul 20, 2009 at 09:52:42PM +0200, Ortwin Gl?ck wrote:

> David,
>
> D H wrote:
> > One question that one of our developers had was where to put the single
> > instance of MultiThreadedHttpConnectionManager to be able to use it across
> > threads from an EJB?
> > He will use the ServletContext and ServletContextListener for the
> > applications but isn't sure for EJBs.
>
> It's a bit of a matter of taste. I think it makes a good candidate for an MBean.
> If JBoss is your appserver, you can easily create a ServiceMBean for this
> purpose and set it up in the jboss-app.xml of your EAR or any other jboss.xml
> (of an EJB-JAR or SAR) of your choice. This way you could also easily add some
> MBean operations & properties for adminitrative purposes like flushing the
> cache, or returning statistics.
>
> Odi
>

I agree with Odi it all boils down to a matter of personal taste. There are a
few more options to consider

(1) JCA (JEE Connector Architecture) resource adaptor. Portable across
different JEE containers but horribly difficult to develop


(2) If your applications needs to access resource from only one HTTP server (or
very few), my personal recommendation would be to wrap HttpClient plus a
_single_ connection connection manager into a stateless session bean and let
the JEE container take care of instance pooling and thread safety.

Hope this helps

Oleg


> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@...
> For additional commands, e-mail: dev-help@...
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...