com.swiftmq.jms.v750.MessageProducerImpl.send() hangs

[Eric Yung]

Hi,

I have a JMS client connecting to SwiftMQ cluster 7.5.0. The client runs for more than 10 days normally. And it hangs suddenly during the method call to com.swiftmq.jms.v750.MessageProducerImpl.send(Message). That call never returns to my client code.

Here are the thread dump taken one at 17:08 and another at 23:06.

INFO   | jvm 1    | 2009/08/11 17:08:25 | "Timer-1" daemon prio=10 tid=0x00002aafbc48cc00 nid=0x66b9 in Object.wait() [0x0000000047452000..0x0000000047452a10]
INFO   | jvm 1    | 2009/08/11 17:08:25 |    java.lang.Thread.State: WAITING (on object monitor)
INFO   | jvm 1    | 2009/08/11 17:08:25 |       at java.lang.Object.wait(Native Method)
INFO   | jvm 1    | 2009/08/11 17:08:25 |       at java.lang.Object.wait(Object.java:485)
INFO   | jvm 1    | 2009/08/11 17:08:25 |       at com.swiftmq.tools.util.UninterruptableWaiter.doWait(Unknown Source)
INFO   | jvm 1    | 2009/08/11 17:08:25 |       at com.swiftmq.tools.concurrent.Semaphore.waitHere(Unknown Source)
INFO   | jvm 1    | 2009/08/11 17:08:25 |       - locked <0x00002aab18d45550> (a com.swiftmq.tools.concurrent.Semaphore)
INFO   | jvm 1    | 2009/08/11 17:08:25 |       at com.swiftmq.tools.requestreply.RequestRegistry.request(Unknown Source)
INFO   | jvm 1    | 2009/08/11 17:08:25 |       at com.swiftmq.jms.v750.MessageProducerImpl.processSend(Unknown Source)
INFO   | jvm 1    | 2009/08/11 17:08:25 |       at com.swiftmq.jms.v750.MessageProducerImpl.send(Unknown Source)


INFO   | jvm 1    | 2009/08/11 23:06:10 | "Timer-1" daemon prio=10 tid=0x00002aafbc48cc00 nid=0x66b9 in Object.wait() [0x0000000047452000..0x0000000047452a10]
INFO   | jvm 1    | 2009/08/11 23:06:10 |    java.lang.Thread.State: WAITING (on object monitor)
INFO   | jvm 1    | 2009/08/11 23:06:10 |       at java.lang.Object.wait(Native Method)
INFO   | jvm 1    | 2009/08/11 23:06:10 |       at java.lang.Object.wait(Object.java:485)
INFO   | jvm 1    | 2009/08/11 23:06:10 |       at com.swiftmq.tools.util.UninterruptableWaiter.doWait(Unknown Source)
INFO   | jvm 1    | 2009/08/11 23:06:10 |       at com.swiftmq.tools.concurrent.Semaphore.waitHere(Unknown Source)
INFO   | jvm 1    | 2009/08/11 23:06:10 |       - locked <0x00002aab1865e120> (a com.swiftmq.tools.concurrent.Semaphore)
INFO   | jvm 1    | 2009/08/11 23:06:10 |       at com.swiftmq.tools.requestreply.RequestRegistry.request(Unknown Source)
INFO   | jvm 1    | 2009/08/11 23:06:10 |       at com.swiftmq.jms.v750.MessageProducerImpl.processSend(Unknown Source)
INFO   | jvm 1    | 2009/08/11 23:06:10 |       at com.swiftmq.jms.v750.MessageProducerImpl.send(Unknown Source)

I have restarted the client and it resumes normal. Do you know what happen?

If this wait can happen anytime, could I use Thread.interrupt() to exit the wait as the code shows the usage of UninterruptableWaiter.doWait()?


Regards,
Eric

[IIT Software]

"Timer-1" means you send your messages driven by a timer? Are you using the sender session from a single thread only? Check the whole thread dump whether all SwiftMQ session threads are blocked. Sometimes the client just runs out of threads if things are done out of message listeners. Look here (Client-Side Threadpools)

You can't interrupt the thread. That's why we use this Uninterruptable. ;)

[IIT Software]

I sounds a bit like this problem (fixed in 7.5.3, coming soon). Can you check the memory consumption of your STANDBY and whether you got an OOM there?

[Eric Yung]

There is NO OOM exception seen in both active and standby SwiftMQ.

But the server got the following exceptions.

+++ High Availability State: ACTIVE/ACTIVE
Got Exception:
    ThreadGroup: jms.ivm.client.session
    ActiveTask : INTRAVM-1/Session/SessionTask
Stack Trace:
java.lang.InternalError: Can't connect to X11 window server using 'localhost:10.0' as the value of the DISPLAY variable.
        at sun.awt.X11GraphicsEnvironment.initDisplay(Native Method)
        at sun.awt.X11GraphicsEnvironment.access$100(X11GraphicsEnvironment.java:52)
        at sun.awt.X11GraphicsEnvironment$1.run(X11GraphicsEnvironment.java:155)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.awt.X11GraphicsEnvironment.<clinit>(X11GraphicsEnvironment.java:131)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:169)
        at java.awt.GraphicsEnvironment.getLocalGraphicsEnvironment(GraphicsEnvironment.java:68)
        at sun.awt.X11.XToolkit.<clinit>(XToolkit.java:89)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:169)
        at java.awt.Toolkit$2.run(Toolkit.java:834)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.awt.Toolkit.getDefaultToolkit(Toolkit.java:826)
        at java.awt.Toolkit.getEventQueue(Toolkit.java:1676)
        at java.awt.EventQueue.invokeLater(EventQueue.java:954)
        at javax.swing.SwingUtilities.invokeLater(SwingUtilities.java:1264)
        at com.swiftmq.admin.cli.v750.RequestProcessor.serviceRequest(Unknown Source)
        at com.swiftmq.tools.requestreply.RequestServiceRegistry.dispatch(Unknown Source)
        at com.swiftmq.admin.mgmt.v750.EndpointImpl.onMessage(Unknown Source)
        at com.swiftmq.jms.v750.MessageConsumerImpl.invokeMessageListener(Unknown Source)
        at com.swiftmq.jms.v750.MessageConsumerImpl.invokeConsumer(Unknown Source)
        at com.swiftmq.jms.v750.SessionImpl$SessionDeliveryQueue.process(Unknown Source)
        at com.swiftmq.tools.queue.SingleProcessorQueue.dequeue(Unknown Source)
        at com.swiftmq.jms.v750.SessionImpl$SessionTask.run(Unknown Source)
        at com.swiftmq.impl.threadpool.standard.PoolThread.run(Unknown Source)

The following exception happens after I install the SNMP swiftlet.

javax.management.InstanceAlreadyExistsException: com.swiftmq.router1:type=sys$net, name1=usage, name2=10.252.80.4_46242
        at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:453)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.internal_addObject(DefaultMBeanServerInterceptor.java:1484)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:963)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:482)
        at com.swiftmq.impl.mgmt.standard.jmx.JMXUtil.registerMBean(Unknown Source)
        at com.swiftmq.impl.mgmt.standard.jmx.EntityMBean.<init>(Unknown Source)
        at com.swiftmq.impl.mgmt.standard.jmx.EntityMBean.entityAdded(Unknown Source)
        at com.swiftmq.mgmt.Entity.notifyEntityWatchListeners(Unknown Source)
        at com.swiftmq.mgmt.Entity.addEntity(Unknown Source)
        at com.swiftmq.impl.net.standard.ConnectionManagerImpl.addConnection(Unknown Source)
        at com.swiftmq.impl.net.standard.scheduler.b.run(Unknown Source)
        at com.swiftmq.impl.threadpool.standard.PoolThread.run(Unknown Source)

[IIT Software]

Eric Yung wrote:

java.lang.InternalError: Can't connect to X11 window server using 'localhost:10.0' as the value of the DISPLAY variable.

You are using CLI from a JAC application (deployed inside the router) on a host which doesn't have X11 installed.

CLI shares a common code base with SwiftMQ Explorer and uses the Swing event queue internally. But you can run it "headless" (without X11) by setting this system property:

-Djava.awt.headless=true

I don't know whether a normal Java app can set this property. In that case you can just set it in the deployment descriptor of your JAC app. If not, you must set it on the command line of both HA instances.

The SNMP error seems to be a follow-up. You have both JMX and SNMP enabled. Is that what you want? If not, disable JMX and use SNMP only.