Memory leak when using CLI admin API

[Karl Cassaigne]

We currently have an application in production running into Tomcat 5.5 and using a SwiftMQ HA 7.20 cluster on a pair of Linux servers with JDK 1.5.

This exact same application has been moved on a pair of new Linux server under Tomcat 6.0, JDK 1.6 and SwiftMQ HA 7.5.2.

The application works fine except we observe a memory leak that consumes 1Go in 24h... We obtained a memory dump on OutOfMemory and analyzed it with Eclipse Memory Analyzer and the result is that 98% of the heap is fullfilled with java.awt.EventQueueItem objects (see memory_leak1.jpg and memory_leak2.jpg) ; it seems that these objects are linked with Router availability notifications.

The fact is that the application is running a thread wich opens a CLI, send some commands, then close the CLI ; it does it every 30 seconds. No RouterListener is added to the CLI.

As soon as we stop the thread, the memory leak stops; Find below some traces we obtained we stopped the thread :

com.swiftmq.tools.requestreply.TimeoutException: Request timeout occured (60000) ms
        at com.swiftmq.admin.mgmt.EndpointFactory$EndpointCreator.request(Unknown Source)
        at com.swiftmq.admin.mgmt.EndpointFactory$EndpointCreator.create(Unknown Source)
        at com.swiftmq.admin.mgmt.EndpointFactory.createEndpoint(Unknown Source)
        at com.swiftmq.admin.cli.CLI.createEndpoint(Unknown Source)
        at com.swiftmq.admin.cli.v750.RequestProcessor$Visitor.visit(Unknown Source)
        at com.swiftmq.mgmt.protocol.v750.RouterAvailableRequest.accept(Unknown Source)
        at com.swiftmq.admin.cli.v750.RequestProcessor$Visitor.run(Unknown Source)
        at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:209)
        at java.awt.EventQueue.dispatchEvent(EventQueue.java:597)
        at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:269)
        at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:184)
        at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:174)
        at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:169)
        at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:161)
        at java.awt.EventDispatchThread.run(EventDispatchThread.java:122)
javax.jms.IllegalStateException: Session is closed
        at com.swiftmq.jms.v750.SessionImpl.verifyState(Unknown Source)
        at com.swiftmq.jms.v750.MessageProducerImpl.verifyState(Unknown Source)
        at com.swiftmq.jms.v750.MessageProducerImpl.send(Unknown Source)
        at com.swiftmq.admin.mgmt.EndpointFactory$EndpointCreator.request(Unknown Source)
        at com.swiftmq.admin.mgmt.EndpointFactory$EndpointCreator.create(Unknown Source)
        at com.swiftmq.admin.mgmt.EndpointFactory.createEndpoint(Unknown Source)
        at com.swiftmq.admin.cli.CLI.createEndpoint(Unknown Source)
        at com.swiftmq.admin.cli.v750.RequestProcessor$Visitor.visit(Unknown Source)
        at com.swiftmq.mgmt.protocol.v750.RouterAvailableRequest.accept(Unknown Source)
        at com.swiftmq.admin.cli.v750.RequestProcessor$Visitor.run(Unknown Source)
        at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:209)
        at java.awt.EventQueue.dispatchEvent(EventQueue.java:597)
        at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:269)
        at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:184)
        at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:174)
        at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:169)
        at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:161)
        at java.awt.EventDispatchThread.run(EventDispatchThread.java:122)
com.swiftmq.tools.requestreply.TimeoutException: Request timeout occured (60000) ms
        at com.swiftmq.admin.mgmt.EndpointFactory$EndpointCreator.request(Unknown Source)
        at com.swiftmq.admin.mgmt.EndpointFactory$EndpointCreator.create(Unknown Source)
        at com.swiftmq.admin.mgmt.EndpointFactory.createEndpoint(Unknown Source)
        at com.swiftmq.admin.cli.CLI.createEndpoint(Unknown Source)
        at com.swiftmq.admin.cli.v750.RequestProcessor$Visitor.visit(Unknown Source)
        at com.swiftmq.mgmt.protocol.v750.RouterAvailableRequest.accept(Unknown Source)
        at com.swiftmq.admin.cli.v750.RequestProcessor$Visitor.run(Unknown Source)
        at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:209)
        at java.awt.EventQueue.dispatchEvent(EventQueue.java:597)
        at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:269)
        at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:184)
        at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:174)
        at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:169)
        at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:161)
        at java.awt.EventDispatchThread.run(EventDispatchThread.java:122)

Thanks for your help.
Best regards,
Karl

[Karl Cassaigne]

I don't know it it's related but we also get some errors we've never seen before (still on the part of the code that uses the CLI) :

java.util.ConcurrentModificationException
        at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100)
        at java.util.TreeMap$EntryIterator.next(TreeMap.java:1136)
        at java.util.TreeMap$EntryIterator.next(TreeMap.java:1131)
        at com.swiftmq.admin.cli.CLI.close(Unknown Source)
        at fr.amadeuspro.jms.client.AdminJmsClientImpl.unconnect(AdminJmsClientImpl.java:121)
        at fr.amadeuspro.mctorouter.module.Supervisor.run(Supervisor.java:379)
        at java.lang.Thread.run(Thread.java:619)

[IIT Software]

The exception above is a synchronization error which might cause the leak.

We had a change in management protocol in 7.5.0. This also effects CLI. In particular it will create management endpoints to all available routers upon availabilty. Before it was during the first "sr" command.

We need to verify it and will provide a fix, if necessary.

[Karl Cassaigne]

For your information, due to the couple of issues we faced with version 7.5.2, we falled back to version 7.2.0. Note that we didn't change anything except the SwiftMQ HA router.

All the observed issues have gone away among wich the most important are :
- scheduled tasks execution starts at the wrong time
- memory leak when using the CLI API
- unexpected ConcurrentModificationException when using the CLI API

We will look forward for a fix in the next release of SwiftMQ since I have no doubt that these problems are very likely to affect some other customers.

Best regards

[IIT Software]

The CLI issues have been fixed already but we are not able to reproduce that jobs are executed at a wrong time. We did not change anything in this area between 7.2.0 and current release. So you might check if that problem is located on your side. Alternatively you may submit an support incident and we will verify your traces.

[Karl Cassaigne]

Thanks for your feedback.

Do you already have an expected date for the fixed version release ? We've some applications that are still in version 7.5.2 and I would prefer not wasting time downgrading them in 7.2.0 if the fix is coming out tomorrow ;-)

I'll do some more testing on the schedule task problem... seems really weird.

[IIT Software]

7.5.4 should be released next week. To get over the CLI problems you can use the 7.2.0 swiftmq.jar at your clients against a 7.5.2 router.

[IIT Software]

This has been fixed in 7.5.4, released today.