Jetty ClosedChannelException appears to cause AMQ Timeout in ServiceMix

[Tom Purcell]

Hello

Forgive me for cross posting with ServiceMix forum but we're having a problem that may be related to ether JETTY-747 or JETTY-661 (or both). Its happening both on my development machine and our test/uat machines

Development:
ServiceMix 3.3
openSuse 11.1
java version "1.6.0_11"
Java(TM) SE Runtime Environment (build 1.6.0_11-b03)
Java HotSpot(TM) 64-Bit Server VM (build 11.0-b16, mixed mode)

Test/uat:
ServiceMix 3.3
Red Hat Enterprise Linux AS release 4 (Nahant Update 6)
java version "1.6.0_07"
Java(TM) SE Runtime Environment (build 1.6.0_07-b06)
Java HotSpot(TM) 64-Bit Server VM (build 10.0-b23, mixed mode)

I'm using the servicemix-http with a marshaller that extends DefaultHttpProviderMarshaler (RestProviderMarshaler). My process flow is:
External Webapp> JmsQueue > SmxJms > SmxPipeline > SmxEIP:RoutingSlip > SmxBean:MyService > SmxHttp:RestProviderMarshaler > HTTPS REST Response > External Webapp

My process runs fine and completes normally. Then, about a minute later, I get the following stack trace:

15:58:28,126 - DEBUG - jetty                          - EXCEPTION
java.nio.channels.ClosedChannelException
        at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
        at org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:169)
        at org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
        at org.mortbay.jetty.security.SslHttpChannelEndPoint.flush(SslHttpChannelEndPoint.java:480)
        at org.mortbay.jetty.security.SslHttpChannelEndPoint.fill(SslHttpChannelEndPoint.java:326)
        at org.mortbay.jetty.HttpParser.fill(HttpParser.java:785)
        at org.mortbay.jetty.client.HttpConnection.handle(HttpConnection.java:193)
        at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:497)
15:58:28,128 - DEBUG - jetty                          - EOF
 
This stack trace will repeat once or twice if I run the process once. If I run under load (via JMeter) the stack trace will repeat continuously and SMX will essentially lock up until I stop the server. If I set the logging level on Jetty to INFO the SMX server will start putting out the following message over and over:

11:05:53,069 | WARN  | NetworkBridge: java.util.concurrent.ThreadPoolExecutor$Worker@31a83224 | DemandForwardingBridge   | .DemandForwardingBridgeSupport  378 | Network connection between vm://localhost#10 and
 tcp://localhost/127.0.0.1:61919 shutdown due to a remote error: java.io.IOException: Wire format negotiation timeout: peer did not send his wire format.
11:18:33,219 | ERROR | ActiveMQ Transport Initiator: /127.0.0.1:59875 | TransportConnector       | mq.broker.TransportConnector$1  234 | Could not accept connection : Wire format negotiation timeout: peer did
not send his wire format.
11:22:07,958 | WARN  | NetworkBridge: java.util.concurrent.ThreadPoolExecutor$Worker@4a90431e | DemandForwardingBridge   | .DemandForwardingBridgeSupport  378 | Network connection between vm://localhost#36 and
 tcp://localhost/127.0.0.1:61919 shutdown due to a remote error: java.io.IOException: Wire format negotiation timeout: peer did not send his wire format.

Again SMX will be locked but eventually will come back and start to process messages again. I'm guessing that resources in SMX get tied up till Jetty is finished trashing. I'm not sure what to do. Any input would be appreciated.

Thanks
Tom