misbehaviour of "com.swiftmq.jms.ConnectionLostException: Connection closed"

[Leos Bitto]

Hello,

I use SwiftMQ 7.5.1 and my application uses just one JMS connection, but with many sessions (each thread of my application uses one session). When I shutdown the SwiftMQ router while my application runs, after some time (any hint how this time can be configured?) one of my threads gets this JMSException when trying to commit its JMS session:

com.swiftmq.jms.ConnectionLostException: Connection closed
        at com.swiftmq.jms.ExceptionConverter.convert(Unknown Source)
        at com.swiftmq.jms.v750.SessionImpl.commit(Unknown Source)
        at my.class.run(MyClass.java:498)
        at java.lang.Thread.run(Thread.java:619)

This thread has the possibility to react on this JMSException by periodically trying to create a new JMS connection until it succeeds (after I start the SwiftMQ router again). However, the other threads seem to get stuck with this thread dump:

"Handler2_2" prio=6 tid=0x2ded7800 nid=0xa18 in Object.wait() [0x310ff000..0x310ffa14]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x04730a40> (a com.swiftmq.tools.concurrent.Semaphore)
        at java.lang.Object.wait(Object.java:485)
        at com.swiftmq.tools.util.UninterruptableWaiter.doWait(Unknown Source)
        at com.swiftmq.tools.concurrent.Semaphore.waitHere(Unknown Source)
        - locked <0x04730a40> (a com.swiftmq.tools.concurrent.Semaphore)
        at com.swiftmq.tools.requestreply.RequestRegistry.request(Unknown Source)
        at com.swiftmq.jms.v750.SessionImpl.requestTransaction(Unknown Source)
        at com.swiftmq.jms.v750.SessionImpl.commit(Unknown Source)
        at my.class.run(MyClass.java:498)
        at java.lang.Thread.run(Thread.java:619)

"Handler1_1" prio=6 tid=0x2c6b6800 nid=0x1308 in Object.wait() [0x3105f000..0x3105fb14]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x04730a88> (a com.swiftmq.tools.concurrent.Semaphore)
        at java.lang.Object.wait(Object.java:485)
        at com.swiftmq.tools.util.UninterruptableWaiter.doWait(Unknown Source)
        at com.swiftmq.tools.concurrent.Semaphore.waitHere(Unknown Source)
        - locked <0x04730a88> (a com.swiftmq.tools.concurrent.Semaphore)
        at com.swiftmq.tools.requestreply.RequestRegistry.request(Unknown Source)
        at com.swiftmq.jms.v750.SessionImpl.requestTransaction(Unknown Source)
        at com.swiftmq.jms.v750.SessionImpl.commit(Unknown Source)
        at my.class.run(MyClass.java:498)
        at java.lang.Thread.run(Thread.java:619)

Both these threads call session.commit(), one has one MessageProducer, the other has one MessageConsumer in their JMS session. Like the name UninterruptableWaiter indicates, these threads cannot be interrupted - they even prevent me from shutting down my whole application, so I have to use kill -9 which is not nice. I would expect that the call to session.commit() should result in JMSException in this case for all my threads, not just one. Why does this not happen, is that a bug in the SwiftMQ code?


Leos Bitto

[IIT Software]

Transparent Reconnect is enabled by default so the client tries to reconnect until it reaches max-retries. That's apparently what you see. If you want to get informed about connection lost, register a javax.jms.ExceptionListener.

All pending requests wait until the reconnect succeeds or max-retries have been reached. In the latter case the requests are cancelled and the commit returns a JMSException.

You can see the reconnect output if you set

-Dswiftmq.reconnect.debug=true

at your client.

[Leos Bitto]

IIT Software wrote:

Transparent Reconnect is enabled by default so the client tries to reconnect until it reaches max-retries. That's apparently what you see.

That makes sense, thank you for explanation.

IIT Software wrote:

If you want to get informed about connection lost, register a javax.jms.ExceptionListener.

I do this, the method onException gets called, but that is irrelevant to my problem.

IIT Software wrote:

All pending requests wait until the reconnect succeeds or max-retries have been reached. In the latter case the requests are cancelled and the commit returns a JMSException.

My problem is that the commit of my session with MessageConsumer hangs - it does not return JMSException! I am attaching a simple program P2PTest.java which proves this misbehaviour which I consider to be a bug in the SwiftMQ code. How to use it: install SwiftMQ 7.5.1, start smqr1, start "java -cp .;jars/swiftmq.jar;jars/jms.jar P2PTest", stop smqr1 and see that my program never finishes because the thread Consumer is blocked infinitely when calling session.commit() which is in a contradiction with your declaration that JMSException would be thrown in this case.P2PTest.java

[IIT Software]

Just tested it and you're right. But it concerns "only" the case if you stop the router and don't restart it. When the client reaches max-retries (50), it throws a JMSException on producer.commit but not on the consumer.commit. We will have to check this.

However, transparent reconnect works. If you stop the router and restart it before your client reaches max-retries, everything continues as expected.

[Leos Bitto]

IIT Software wrote:

Just tested it and you're right. But it concerns "only" the case if you stop the router and don't restart it.

Not only this case. It also concerns the case when I do start the router, but too late (after max-retries are reached).

IIT Software wrote:

When the client reaches max-retries (50), it throws a JMSException on producer.commit but not on the consumer.commit. We will have to check this.

Exactly. In this case the consumer thread gets blocked and cannot be interrupted, which is a fatal problem.

IIT Software wrote:

However, transparent reconnect works. If you stop the router and restart it before your client reaches max-retries, everything continues as expected.

Sure, but the situation when the router does not become available until max-retries are reached must be handled properly, too.

[Leos Bitto]

I confirm that this bug has been fixed in the SwiftMQ release 7.5.3 - now all producers and consumers get com.swiftmq.jms.ConnectionLostException, as expected.