HttpCore NIO and recycling connections

[Peter Soles-2]

I'm using HttpCore NIO as part of a web app that issues long polls to a remote server on behalf of clients - basically it is a proxy. I previously used NIO as part of a simulator I was building over the summer and had good success with it!  Now I am trying to use an enhanced version of the code as part of my web app since it is possible that my webapp may have to open many long polls to the remote server.  One of the enhancements is I added to my code is connection recycling.

My webapp operates as expected in my development environment under Windows and I am able to make long polls using recycled connections without errors.  However, when I deploy it to a server running on Linux I get frequent fatalIOExceptions raised by the EventListener.  I traced this to my connection recycler object that attempts to re-use a connection.  It appears that the code sometimes hangs when I issue NHttpConnection.requestOutput() on a recycled connection, and then the code throws the IOException: Connection reset by peer.

I use tcpdump to monitor the outgoing messages from my webapp to the remote server and I observe that the message I am trying to send when I call requestOutput() is never sent over the wire so I suspect that the "connection reset by peer" is not really what is happening.  I actually suspect/hope that I am doing something obviously wrong in my recycle connection code that is causing this to happen.

I've attached my code that implements my NIO communication stage for my web app.  To use, you create a new HttpClientNIO object and then you can send a non-blocking request using HttpClientNIO.sendRequest(request, callback).  When the request completes you will receive notification through the callback.

My connection logic is this:

1. When request arrives, check for available connection on queue. 

2. If there is not an available connection:
    a) call DefaultConnectionIOReactor.connect() start creation of new connection
    b) use connection as described in documentation
    c) when HttpRequestExecutionHandler.handleResponse() has completed it's processing, put connection on queue for re-use.

3. If there is an available connection
    a) make sure connection is active
    b) get the HttpContext from the connection NHttpConnection.getContext()
    c) reset values as needed
    d) call NHttpConnection.requestOutput

Am I missing something?  I should also say that I generally instantiate an SSLClientIOEventDispatch since the remote server uses SSL but I also get fatalIOException when I am not using SSL.

Peter


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@...
For additional commands, e-mail: httpclient-users-help@...

[olegk]

Peter Soles wrote:

> I'm using HttpCore NIO as part of a web app that issues long polls to a
> remote server on behalf of clients - basically it is a proxy. I
> previously used NIO as part of a simulator I was building over the
> summer and had good success with it!  Now I am trying to use an enhanced
> version of the code as part of my web app since it is possible that my
> webapp may have to open many long polls to the remote server.  One of
> the enhancements is I added to my code is connection recycling.
>
> My webapp operates as expected in my development environment under
> Windows and I am able to make long polls using recycled connections
> without errors.  However, when I deploy it to a server running on Linux
> I get frequent fatalIOExceptions raised by the EventListener.  I traced
> this to my connection recycler object that attempts to re-use a
> connection.  It appears that the code sometimes hangs when I issue
> NHttpConnection.requestOutput() on a recycled connection, and then the
> code throws the IOException: Connection reset by peer.
>
> I use tcpdump to monitor the outgoing messages from my webapp to the
> remote server and I observe that the message I am trying to send when I
> call requestOutput() is never sent over the wire so I suspect that the
> "connection reset by peer" is not really what is happening.  I actually
> suspect/hope that I am doing something obviously wrong in my recycle
> connection code that is causing this to happen.
>
> I've attached my code that implements my NIO communication stage for my
> web app.  To use, you create a new HttpClientNIO object and then you can
> send a non-blocking request using HttpClientNIO.sendRequest(request,
> callback).  When the request completes you will receive notification
> through the callback.
>
> My connection logic is this:
>
> 1. When request arrives, check for available connection on queue.
>
> 2. If there is not an available connection:
>     a) call DefaultConnectionIOReactor.connect() start creation of new
> connection
>     b) use connection as described in documentation
>     c) when HttpRequestExecutionHandler.handleResponse() has completed
> it's processing, put connection on queue for re-use.
>
> 3. If there is an available connection
>     a) make sure connection is active
>     b) get the HttpContext from the connection NHttpConnection.getContext()
>     c) reset values as needed
>     d) call NHttpConnection.requestOutput
>
> Am I missing something?  I should also say that I generally instantiate
> an SSLClientIOEventDispatch since the remote server uses SSL but I also
> get fatalIOException when I am not using SSL.
>
> Peter
>

Peter,

It all sounds very reasonable. I have a question, though. When you put
persistent connections on the re-use queue, do you clear interest in
read event notifications? This might explain why the I/O event dispatch
does not get called when a connection is closed by the peer on the
opposite end.

Anyways, for what it is worth to you, I have also put together a very
experimental and very buggy prototype of an async HTTP agent based on
HttpCore NIO and HttpClient 4.0. Feel free to take a look at my
implementation of the non-blocking connection manager and pick some
ideas or bits of code:

http://svn.apache.org/repos/asf/httpcomponents/asynchttpclient/trunk/

This is how this API looks in action:

http://svn.apache.org/repos/asf/httpcomponents/asynchttpclient/trunk/src/examples/org/apache/http/examples/nio/client/AsyncClientRequest.java

You may take a special look at the code that adds logging capability to
some of the HttpCore NIO classes.

http://svn.apache.org/repos/asf/httpcomponents/asynchttpclient/trunk/src/main/java/org/apache/http/impl/nio/conn/LoggingNHttpClientConnection.java
http://svn.apache.org/repos/asf/httpcomponents/asynchttpclient/trunk/src/main/java/org/apache/http/impl/nio/conn/LoggingIOSession.java

This really helps with debugging.

Hope this helps somewhat.

Oleg

>
> ------------------------------------------------------------------------
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@...
> For additional commands, e-mail: httpclient-users-help@...


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@...
For additional commands, e-mail: httpclient-users-help@...

[Peter Soles-2]

Thanks for the response and the pointers to your code.  I'll definitely take
a look!

I'm not clearing the read event notifications in the persistent connection.
In my code, the connection is "born" when I get it from I
NHttpClientHandler.connected() and I just place it on my re-use queue at the
end of HttpRequestExecutionHandler.handleResponse(). When I need to re-use,
I just check NHttpClientConnection.getStatus == ACTIVE and
NHttpClientConnection.isOpen() == true.

Should I be calling NHttpClientConnection.resetOutput()/resetInput() before
I try and re-use the connection?

thanks again!

Peter


On Tue, Sep 22, 2009 at 3:29 PM, Oleg Kalnichevski <olegk@...> wrote:

> Peter Soles wrote:
>
>> I'm using HttpCore NIO as part of a web app that issues long polls to a
>> remote server on behalf of clients - basically it is a proxy. I previously
>> used NIO as part of a simulator I was building over the summer and had good
>> success with it!  Now I am trying to use an enhanced version of the code as
>> part of my web app since it is possible that my webapp may have to open many
>> long polls to the remote server.  One of the enhancements is I added to my
>> code is connection recycling.
>>
>> My webapp operates as expected in my development environment under Windows
>> and I am able to make long polls using recycled connections without errors.
>>  However, when I deploy it to a server running on Linux I get frequent
>> fatalIOExceptions raised by the EventListener.  I traced this to my
>> connection recycler object that attempts to re-use a connection.  It appears
>> that the code sometimes hangs when I issue NHttpConnection.requestOutput()
>> on a recycled connection, and then the code throws the IOException:
>> Connection reset by peer.
>>
>> I use tcpdump to monitor the outgoing messages from my webapp to the
>> remote server and I observe that the message I am trying to send when I call
>> requestOutput() is never sent over the wire so I suspect that the
>> "connection reset by peer" is not really what is happening.  I actually
>> suspect/hope that I am doing something obviously wrong in my recycle
>> connection code that is causing this to happen.
>>
>> I've attached my code that implements my NIO communication stage for my
>> web app.  To use, you create a new HttpClientNIO object and then you can
>> send a non-blocking request using HttpClientNIO.sendRequest(request,
>> callback).  When the request completes you will receive notification through
>> the callback.
>>
>> My connection logic is this:
>>
>> 1. When request arrives, check for available connection on queue.
>> 2. If there is not an available connection:
>>    a) call DefaultConnectionIOReactor.connect() start creation of new
>> connection
>>    b) use connection as described in documentation
>>    c) when HttpRequestExecutionHandler.handleResponse() has completed it's
>> processing, put connection on queue for re-use.
>>
>> 3. If there is an available connection
>>    a) make sure connection is active
>>    b) get the HttpContext from the connection NHttpConnection.getContext()
>>    c) reset values as needed
>>    d) call NHttpConnection.requestOutput
>>
>> Am I missing something?  I should also say that I generally instantiate an
>> SSLClientIOEventDispatch since the remote server uses SSL but I also get
>> fatalIOException when I am not using SSL.
>>
>> Peter
>>
>>
> Peter,
>
> It all sounds very reasonable. I have a question, though. When you put
> persistent connections on the re-use queue, do you clear interest in read
> event notifications? This might explain why the I/O event dispatch does not
> get called when a connection is closed by the peer on the opposite end.
>
> Anyways, for what it is worth to you, I have also put together a very
> experimental and very buggy prototype of an async HTTP agent based on
> HttpCore NIO and HttpClient 4.0. Feel free to take a look at my
> implementation of the non-blocking connection manager and pick some ideas or
> bits of code:
>
> http://svn.apache.org/repos/asf/httpcomponents/asynchttpclient/trunk/
>
> This is how this API looks in action:
>
>
> http://svn.apache.org/repos/asf/httpcomponents/asynchttpclient/trunk/src/examples/org/apache/http/examples/nio/client/AsyncClientRequest.java
>
> You may take a special look at the code that adds logging capability to
> some of the HttpCore NIO classes.
>
>
> http://svn.apache.org/repos/asf/httpcomponents/asynchttpclient/trunk/src/main/java/org/apache/http/impl/nio/conn/LoggingNHttpClientConnection.java
>
> http://svn.apache.org/repos/asf/httpcomponents/asynchttpclient/trunk/src/main/java/org/apache/http/impl/nio/conn/LoggingIOSession.java
>
> This really helps with debugging.
>
> Hope this helps somewhat.
>
> Oleg
>
>
>> ------------------------------------------------------------------------
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: httpclient-users-unsubscribe@...
>> For additional commands, e-mail: httpclient-users-help@...
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@...
> For additional commands, e-mail: httpclient-users-help@...
>
>

[olegk]

Peter Soles wrote:

> Thanks for the response and the pointers to your code.  I'll definitely take
> a look!
>
> I'm not clearing the read event notifications in the persistent connection.
> In my code, the connection is "born" when I get it from I
> NHttpClientHandler.connected() and I just place it on my re-use queue at the
> end of HttpRequestExecutionHandler.handleResponse(). When I need to re-use,
> I just check NHttpClientConnection.getStatus == ACTIVE and
> NHttpClientConnection.isOpen() == true.
>
> Should I be calling NHttpClientConnection.resetOutput()/resetInput() before
> I try and re-use the connection?
>
> thanks again!
>
> Peter
>

Sorry for not being specific enough. Do you call
NHttpClientConnection#suspendInput() anywhere in your code prior to
putting the connection on the re-use queue?

Oleg

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@...
For additional commands, e-mail: httpclient-users-help@...

[Peter Soles-2]

No need to apologize, I certainly appreciate your responses!  To answer your
question, no, I don't call NHttpClientConnection#suspendInput in my code
prior to putting the connection on my re-use queue.

As a possibly irrelevant point, my code functions on my Linux server if I
circumvent my connection recycler but of course I'd rather not do that :-).
Under Windows, the connection recycling seems to work just find and I get no
errors.

Does the fact that tcpdump doesn't report any outgoing message for the
request that fails give me any clues as to what is happening?  I'll have to
take a look at the logging pointers you sent me.

Peter

On Tue, Sep 22, 2009 at 5:06 PM, Oleg Kalnichevski <olegk@...> wrote:

> Peter Soles wrote:
>
>> Thanks for the response and the pointers to your code.  I'll definitely
>> take
>> a look!
>>
>> I'm not clearing the read event notifications in the persistent
>> connection.
>> In my code, the connection is "born" when I get it from I
>> NHttpClientHandler.connected() and I just place it on my re-use queue at
>> the
>> end of HttpRequestExecutionHandler.handleResponse(). When I need to
>> re-use,
>> I just check NHttpClientConnection.getStatus == ACTIVE and
>> NHttpClientConnection.isOpen() == true.
>>
>> Should I be calling NHttpClientConnection.resetOutput()/resetInput()
>> before
>> I try and re-use the connection?
>>
>> thanks again!
>>
>> Peter
>>
>>
> Sorry for not being specific enough. Do you call
> NHttpClientConnection#suspendInput() anywhere in your code prior to putting
> the connection on the re-use queue?
>
>
> Oleg
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@...
> For additional commands, e-mail: httpclient-users-help@...
>
>

[olegk]

Peter Soles wrote:

> No need to apologize, I certainly appreciate your responses!  To answer your
> question, no, I don't call NHttpClientConnection#suspendInput in my code
> prior to putting the connection on my re-use queue.
>
> As a possibly irrelevant point, my code functions on my Linux server if I
> circumvent my connection recycler but of course I'd rather not do that :-).
> Under Windows, the connection recycling seems to work just find and I get no
> errors.
>
> Does the fact that tcpdump doesn't report any outgoing message for the
> request that fails give me any clues as to what is happening?  I'll have to
> take a look at the logging pointers you sent me.
>
> Peter
>

Peter,

To me it is perfectly clear what is happening. While a persistent
connection was sitting idle in the re-use queue it was closed by the
peer on the opposite end and had become half-closed. That explains why
there is no outgoing message recorded by tcpdump. (The message was
submitted to the connection but nothing got sent across the wire as the
socket was already closed on the opposite end.) However, I always
assumed the I/O selector would get immediately notified of such an event
and the protocol handler would be able to react intelligently to it, for
instance, by removing the connection from the queue. The question is why
it does not happen, and since it appears to be platform specific in your
case I tend to suspect a JRE bug / an OS issue.

A debug log would probably be quite helpful.

Oleg

> On Tue, Sep 22, 2009 at 5:06 PM, Oleg Kalnichevski <olegk@...> wrote:
>
>> Peter Soles wrote:
>>
>>> Thanks for the response and the pointers to your code.  I'll definitely
>>> take
>>> a look!
>>>
>>> I'm not clearing the read event notifications in the persistent
>>> connection.
>>> In my code, the connection is "born" when I get it from I
>>> NHttpClientHandler.connected() and I just place it on my re-use queue at
>>> the
>>> end of HttpRequestExecutionHandler.handleResponse(). When I need to
>>> re-use,
>>> I just check NHttpClientConnection.getStatus == ACTIVE and
>>> NHttpClientConnection.isOpen() == true.
>>>
>>> Should I be calling NHttpClientConnection.resetOutput()/resetInput()
>>> before
>>> I try and re-use the connection?
>>>
>>> thanks again!
>>>
>>> Peter
>>>
>>>
>> Sorry for not being specific enough. Do you call
>> NHttpClientConnection#suspendInput() anywhere in your code prior to putting
>> the connection on the re-use queue?
>>
>>
>> Oleg
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: httpclient-users-unsubscribe@...
>> For additional commands, e-mail: httpclient-users-help@...
>>
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@...
For additional commands, e-mail: httpclient-users-help@...