SwiftMQ 7.4.1 + spring 2.5 + transactions + ReplyTo == trouble

[Bryan Talbot]

I seem to have run into a few bugs while getting a spring 2.5.x application working with SwiftMQ.  The portion I've had a lot of troubles with listens to a queue and responds to the received message's ReplyTo destination.

I think there are three issues I've run into.

1.  An object leak in SwiftMQ QueueReceiver objects when transactions are used and consumers (and sessions and connections) are not cached.
2.  A bug of springsupport's SingleSharedConnectionFactory that causes all producers to be cached -- even if they're only used once to reply on a temporary queue named in a ReplyTo header.  Since these producers never actually get closed, the max-producers limit for the connection is quickly reached.
3.  A bug causing "transaction closed" exceptions after sending a response only when run on OSX.  This occurs with both Java 1.5 and 1.6.  This bug doesn't seem to happen on Linux.


I don't believe that these are spring bugs in general because these problems don't come up when using the same application on Active MQ -- not that Active MQ behavior is correct -- I've found too many bugs in that to make me want to use it for anything other than testing.

I've attached jar files containing source and binaries for a simple spring app to reproduce these issues.



To summarize, the behaviors I'm seeing are ...

No caching:
Leak QueueReceiver objects and eventually OOM

Cache connection only:
Leak QueueReceiver objects and eventually OOM

Cache connection and session only:
Leak QueueReceiver objects and eventually OOM

Cache connection, session and consumer:
working on linux, fails with transaction closed exception on osx

Use springsupport's SingleSharedConnectionFactory:
no way to disable producer caching which causes 1-time-use producers on temp ReplyTo queues to exceed connection limit.



The number of QueueReceiver objects continues to grow until OOM.  This occurs with just one application connected to SwiftMQ and doesn't require the application to actually be receiving any messages -- it just has to poll for messages.

The leak can be easily detected using Java 1.6 using jmap -histo:live ...

agewi-app-qu02:router$ /usr/java/default/bin/jmap -histo:live <pid> | grep QueueReceiver
 197:             8            640  com.swiftmq.swiftlet.queue.QueueReceiver
 226:             8            384  com.swiftmq.swiftlet.queue.QueueReceiver$PullTransactionRecycler
agewi-app-qu02:router$ /usr/java/default/bin/jmap -histo:live <pid> | grep QueueReceiver
  46:           173          13840  com.swiftmq.swiftlet.queue.QueueReceiver
  65:           173           8304  com.swiftmq.swiftlet.queue.QueueReceiver$PullTransactionRecycler
agewi-app-qu02:router$ /usr/java/default/bin/jmap -histo:live <pid> | grep QueueReceiver
  40:           251          20080  com.swiftmq.swiftlet.queue.QueueReceiver
  52:           251          12048  com.swiftmq.swiftlet.queue.QueueReceiver$PullTransactionRecycler
agewi-app-qu02:router$ /usr/java/default/bin/jmap -histo:live <pid> | grep QueueReceiver
  33:           359          28720  com.swiftmq.swiftlet.queue.QueueReceiver
  45:           359          17232  com.swiftmq.swiftlet.queue.QueueReceiver$PullTransactionRecycler
agewi-app-qu02:router$ /usr/java/default/bin/jmap -histo:live <pid> | grep QueueReceiver
  33:           398          31840  com.swiftmq.swiftlet.queue.QueueReceiver
  43:           398          19104  com.swiftmq.swiftlet.queue.QueueReceiver$PullTransactionRecycler



And finally, here's the exception that occurs when consumer caching is enabled with transactions on OSX ...


2009-04-13 11:41:34,166 [dummyInquiryJmsContainer-1] INFO  org.springframework.jms.listener.DefaultMessageListenerContainer - Setup of JMS message listener invoker failed for destination 'game_item_insertable_iweb' - trying to recover.
 Cause: Could not commit JMS transaction; nested exception is javax.jms.JMSException: com.swiftmq.swiftlet.queue.QueueTransactionClosedException: transaction is closed!
org.springframework.transaction.TransactionSystemException: Could not commit JMS transaction; nested exception is javax.jms.JMSException: com.swiftmq.swiftlet.queue.QueueTransactionClosedException: transaction is closed!
        at org.springframework.jms.connection.JmsTransactionManager.doCommit(JmsTransactionManager.java:242)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:732)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:701)
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:255)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:982)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:974)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:876)
        at java.lang.Thread.run(Thread.java:613)
Caused by: javax.jms.JMSException: com.swiftmq.swiftlet.queue.QueueTransactionClosedException: transaction is closed!
        at com.swiftmq.jms.ExceptionConverter.convert(Unknown Source)
        at com.swiftmq.jms.v630.SessionImpl.commit(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.springframework.jms.connection.CachingConnectionFactory$CachedSessionInvocationHandler.invoke(CachingConnectionFactory.java:332)
        at $Proxy1.commit(Unknown Source)
        at org.springframework.jms.connection.JmsTransactionManager.doCommit(JmsTransactionManager.java:236)
        ... 7 more




Thanks for your help,

Bryan

[Bryan Talbot]

Well, looks like there's no way to attach files for the test case.  So I'll copy and paste the code into a message instead ...

$> cat src/main/java/bat/Main.java
package bat;
import org.springframework.context.support.ClassPathXmlApplicationContext;
public class Main
{
    public static void main( String [] args )
    {
        final ClassPathXmlApplicationContext context = new ClassPathXmlApplicationContext();
        context.setAllowBeanDefinitionOverriding( false );
        context.setConfigLocations( args );
        context.refresh();
        context.registerShutdownHook();        
    }
}


$> cat src/main/java/bat/Responder.java
package bat;
import org.apache.log4j.Logger;
public class Responder
{
    final private Logger logger = Logger.getLogger( Responder.class );
    public String process( String msg )
    {
        final String ret = msg.toUpperCase();
        logger.info( "Processed message and sending response: " + ret );
        return ret;
    }
}


$> cat  src/main/resources/context-swiftmq.xml
<?xml version="1.0" encoding="UTF-8"?>
<beans
    xmlns="http://www.springframework.org/schema/beans"
    xmlns:p="http://www.springframework.org/schema/p"
    xmlns:jms="http://www.springframework.org/schema/jms"
    xmlns:jee="http://www.springframework.org/schema/jee"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd
                        http://www.springframework.org/schema/jms http://www.springframework.org/schema/jms/spring-jms-2.5.xsd
                        http://www.springframework.org/schema/jee http://www.springframework.org/schema/jee/spring-jee-2.5.xsd"
>

    <!--  jndi access for the swiftmq server -->
    <bean id="destinationResolver" class="org.springframework.jms.support.destination.JndiDestinationResolver">
      <property name="jndiEnvironment">
        <props>
          <prop key="java.naming.factory.initial">com.swiftmq.jndi.InitialContextFactoryImpl</prop>
          <prop key="java.naming.provider.url">smqp://localhost:4001/timeout=1000</prop>
        </props>
      </property>
    </bean>

    <jee:jndi-lookup id="jndiConnectionFactory" jndi-name="ConnectionFactory">
      <jee:environment>
        java.naming.factory.initial=com.swiftmq.jndi.InitialContextFactoryImpl
        java.naming.provider.url=smqp://localhost:4001/timeout=1000
      </jee:environment>
    </jee:jndi-lookup>
   
    <bean id="authConnectionFactory" class="org.springframework.jms.connection.UserCredentialsConnectionFactoryAdapter"
          p:targetConnectionFactory-ref="jndiConnectionFactory"
          p:username="admin"
          p:password="secret" />
   
<!--     <bean id="connectionFactory" -->
<!--           class="com.swiftmq.jms.springsupport.SingleSharedConnectionFactory" -->
<!--           p:targetConnectionFactory-ref="authConnectionFactory" /> -->
   
    <bean id="connectionFactory" class="org.springframework.jms.connection.CachingConnectionFactory"
          p:targetConnectionFactory-ref="authConnectionFactory"
          p:cacheProducers="false"
          />

<!--     <bean id="connectionFactory" class="org.springframework.jms.connection.SingleConnectionFactory" -->
<!--           p:targetConnectionFactory-ref="authConnectionFactory" -->
<!--           /> -->

    <bean id="transactionManager" class="org.springframework.jms.connection.JmsTransactionManager"
          p:connectionFactory-ref="connectionFactory" />
   
    <bean id="responder" class="bat.Responder"/>
   

    <bean id="messageListener" class="org.springframework.jms.listener.adapter.MessageListenerAdapter"
          p:defaultListenerMethod="process">
      <constructor-arg ref="responder"/>
    </bean>

    <bean id="jmsContainer" class="org.springframework.jms.listener.DefaultMessageListenerContainer"
          p:transactionManager-ref="transactionManager"
          p:connectionFactory-ref="connectionFactory"
          p:destinationName="testqueue"
          p:destinationResolver-ref="destinationResolver"
          p:messageListener-ref="messageListener"
          />
   
</beans>



$> cat  src/main/resources/context-activemq.xml
<?xml version="1.0" encoding="UTF-8"?>
<beans
    xmlns="http://www.springframework.org/schema/beans"
    xmlns:p="http://www.springframework.org/schema/p"
    xmlns:jms="http://www.springframework.org/schema/jms"
    xmlns:jee="http://www.springframework.org/schema/jee"
    xmlns:amq="http://activemq.apache.org/schema/core"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd
                        http://www.springframework.org/schema/jms http://www.springframework.org/schema/jms/spring-jms-2.5.xsd
                        http://www.springframework.org/schema/jee http://www.springframework.org/schema/jee/spring-jee-2.5.xsd
                        http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core-5.2.0.xsd"
>

  <amq:broker id="amq-broker" useJmx="false" persistent="false">
    <amq:transportConnectors>
      <amq:transportConnector name="stomp" uri="stomp://localhost:61613"/>
      <amq:transportConnector name="tcp" uri="tcp://localhost:4001"/>
    </amq:transportConnectors>
  </amq:broker>
       
  <!--  jndi access for the destinations -->
  <bean id="destinationResolver" class="org.springframework.jms.support.destination.JndiDestinationResolver" depends-on="amq-broker">
    <property name="jndiEnvironment">
      <props>
        <prop key="java.naming.factory.initial">org.apache.activemq.jndi.ActiveMQInitialContextFactory</prop>
        <prop key="java.naming.provider.url">vm://localhost</prop>
        <prop key="queue.testqueue">testqueue</prop>
      </props>
    </property>
  </bean>

  <!-- JMS ConnectionFactory to use, configuring the embedded broker using XML -->
  <amq:connectionFactory id="connectionFactory" brokerURL="vm://localhost"/>
       
  <!-- to enable (connection / session / producer / consumer) caching, change the id of the bean
       above to "amqConnectionFactory" and uncomment the bean below -->

<!--   <bean id="connectionFactory" class="org.springframework.jms.connection.CachingConnectionFactory" -->
<!--         p:targetConnectionFactory-ref="amqConnectionFactory" -->
<!--         p:cacheProducers="false" -->
<!--         /> -->
 
  <bean id="transactionManager" class="org.springframework.jms.connection.JmsTransactionManager"
        p:connectionFactory-ref="connectionFactory" />
 
  <bean id="responder" class="bat.Responder"/>
 
 
  <bean id="messageListener" class="org.springframework.jms.listener.adapter.MessageListenerAdapter"
        p:defaultListenerMethod="process">
    <constructor-arg ref="responder"/>
  </bean>
 
  <bean id="jmsContainer" class="org.springframework.jms.listener.DefaultMessageListenerContainer"
        p:transactionManager-ref="transactionManager"
        p:connectionFactory-ref="connectionFactory"
        p:destinationName="testqueue"
        p:destinationResolver-ref="destinationResolver"
        p:messageListener-ref="messageListener"
        />
 
</beans>


$> cat pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>bat</groupId>
  <artifactId>spring-replyto</artifactId>
  <packaging>jar</packaging>
  <version>0.0.1-SNAPSHOT</version>
  <name>spring-replyto</name>
  <dependencies>
    <dependency>
    <groupId>javax.jms</groupId>
    <artifactId>jms</artifactId>
    <version>1.1</version>
    </dependency>
    <dependency>
    <groupId>org.springframework</groupId>
    <artifactId>spring</artifactId>
    <version>2.5.6</version>
    </dependency>
    <dependency>
    <groupId>org.apache.activemq</groupId>
    <artifactId>activemq-core</artifactId>
    <version>5.2.0</version>
    </dependency>
    <dependency>
    <groupId>org.apache.xbean</groupId>
    <artifactId>xbean-spring</artifactId>
    <version>2.8</version>
    </dependency>
    <dependency>
    <groupId>com.swiftmq</groupId>
    <artifactId>swiftmq</artifactId>
    <version>7.4.1</version>
    </dependency>
    <dependency>
    <groupId>com.swiftmq</groupId>
    <artifactId>springsupport</artifactId>
    <version>7.4.1</version>
    </dependency>
    <dependency>
      <groupId>log4j</groupId>
      <artifactId>log4j</artifactId>
      <version>1.2.15</version>
    </dependency>
  </dependencies>
  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <configuration>
          <source>1.5</source>
          <target>1.5</target>
        </configuration>
      </plugin>
      <plugin>
      <artifactId>maven-assembly-plugin</artifactId>
      <version>2.2-beta-3</version>
      <configuration>
          <tarLongFileMode>gnu</tarLongFileMode>
          <descriptorRefs>
            <descriptor>src</descriptor>
          </descriptorRefs>
      </configuration>
        <executions>
          <execution>
            <id>make-assembly</id>
            <phase>package</phase>
            <goals>
              <goal>single</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-dependency-plugin</artifactId>
        <version>2.1</version>
      </plugin>
    </plugins>
  </build>

  <repositories>
     <!-- redacted -->
  </repositories>
 
</project>

[IIT Software]

You should always use our springsupport.jar. There are issues with Spring, especially with a TransactionManager. For example, if you use the default values on the DefaultMessageListenerContainer, it will use receive(1000) and runs a commit every second. If you don't use pooling, it will create/destroy sessions etc on every go.

I got a strange exception from DefaultMessageListenerContainer stating that a message was received while the listener was stopped. Don't know what that means but I was able to get over it by adding a property. See attached Spring context. swiftmq-context.xml

The problem with many producers for temp queues can be solved a) by using an unidentified producer or b) by reducing the poolExpiration. Both are supported by our springsupport. The usual way is a) in request/reply. I don't know how you can use an unidentified producer in Spring's request/reply.

[Bryan Talbot]

IIT Software wrote:

You should always use our springsupport.jar. There are issues with Spring, especially with a TransactionManager. For example, if you use the default values on the DefaultMessageListenerContainer, it will use receive(1000) and runs a commit every second. If you don't use pooling, it will create/destroy sessions etc on every go.

Yes, this is what seems to cause the object leak in the SwiftMQ router itself.  Note that these leaked objects do not go away when the client disconnects or quits -- the router must be restarted.  This seems like a pretty serious leak that any JMS client might cause.

The connection and session creation issues can also be avoided by using springs SingleConnectionFactory or CachingConnectionFactory which is what I'm using in the sample app above.  The CachingConnectionFactory is configured in the sample to not cache any producers but will still cache connections, sessions, and consumers.

IIT Software wrote:

I got a strange exception from DefaultMessageListenerContainer stating that a message was received while the listener was stopped. Don't know what that means but I was able to get over it by adding a property. See attached Spring context. swiftmq-context.xml

I've not seen that happen before.  Did that occur during startup or shutdown or while the listener was expected to be running?  I've not tried the properties yet, but it doesn't look like those settings are meant to be used with a transaction manager.  From the API docs for setReceiveTimeout:

Set the timeout to use for receive calls, in milliseconds. The default is 1000 ms, that is, 1 second.
NOTE: This value needs to be smaller than the transaction timeout used by the transaction manager (in the appropriate unit, of course). -1 indicates no timeout at all; however, this is only feasible if not running within a transaction manager.  

Since I'm using a JmsTransactionManager, that seems like asking for trouble.

IIT Software wrote:

The problem with many producers for temp queues can be solved a) by using an unidentified producer or b) by reducing the poolExpiration. Both are supported by our springsupport. The usual way is a) in request/reply. I don't know how you can use an unidentified producer in Spring's request/reply.

AFAIK, spring is handling the reply like most JMS applications would.  It calls Session.createQueue() using the name from the ReplyTo header.  A producer is then created from that Destination, then a Message which is finally sent.

It seems that for the caching of connection related objects to be effective at all, the object lifetime must be at least a few seconds or connections or sessions won't be reused.  Is that correct?  If it is, then a large number of messages can be processed during that time with each of them producing a response.  I'd have to make the max-temp-producers-per-connection value very large to accommodate this.

Does the springsupport SingleSharedConnectionFactory perform any other magic besides caching the connection related objects that the CachingConnectionFactory does not?

[IIT Software]

Bryan Talbot wrote:

Yes, this is what seems to cause the object leak in the SwiftMQ router itself.  Note that these leaked objects do not go away when the client disconnects or quits -- the router must be restarted.  This seems like a pretty serious leak that any JMS client might cause.

It only happens with Spring in this scenario but we were not able to figure our why. I'm not sure if this is a bug in SwiftMQ. But anyway, we will dive into it.

Bryan Talbot wrote:

The connection and session creation issues can also be avoided by using springs SingleConnectionFactory or CachingConnectionFactory which is what I'm using in the sample app above.  The CachingConnectionFactory is configured in the sample to not cache any producers but will still cache connections, sessions, and consumers.

Might be but that's simply not tested from us. We have tested our springsupport but we are not able to test any new Spring features etc.

Bryan Talbot wrote:

I've not seen that happen before.  Did that occur during startup or shutdown or while the listener was expected to be running?  I've not tried the properties yet, but it doesn't look like those settings are meant to be used with a transaction manager.  From the API docs for setReceiveTimeout:

It happend if I use samples/P2PRequestor with 2 messages. First one wents through, the second hangs because of this reason.

Bryan Talbot wrote:

 From the API docs for setReceiveTimeout:

Set the timeout to use for receive calls, in milliseconds. The default is 1000 ms, that is, 1 second.
NOTE: This value needs to be smaller than the transaction timeout used by the transaction manager (in the appropriate unit, of course). -1 indicates no timeout at all; however, this is only feasible if not running within a transaction manager.  

Since I'm using a JmsTransactionManager, that seems like asking for trouble.

The question is why there is a transaction timeout at all. JMS local transactions never timeout and it is not spec compliant to call rollback/commit from another thread to terminate a receive. You can only call session.close from another thread. However, setting receiveTimeout=-1 will avoid this strange behavior to commit empty transactions every second. So you should use -1.

Bryan Talbot wrote:

AFAIK, spring is handling the reply like most JMS applications would.  It calls Session.createQueue() using the name from the ReplyTo header.  A producer is then created from that Destination, then a Message which is finally sent.

That is not how request/reply is handled from a JMS application. Message.getJMSReplyTo() returns already a javax.jms.Destination object. In your case it's a javax.jms.TemporaryQueue, not a javax.jms.Queue. If Spring really acts like you describe above, this could be one cause for the problems.

Bryan Talbot wrote:

It seems that for the caching of connection related objects to be effective at all, the object lifetime must be at least a few seconds or connections or sessions won't be reused.  Is that correct?  If it is, then a large number of messages can be processed during that time with each of them producing a response.  I'd have to make the max-temp-producers-per-connection value very large to accommodate this.

Why don't you just use an unidentified producer? You can create it with session.createProducer(null) and pass the destination during send. That's how request/reply should be done. You can create it from our springsupport like any other producer and it will be cached (checkout on create, checkin on close).

[IIT Software]

Bryan Talbot wrote:

Does the springsupport SingleSharedConnectionFactory perform any other magic besides caching the connection related objects that the CachingConnectionFactory does not?

I did not compare them but if you use SwiftMQ, I recommend to use our lib because we tested it and if there is a bug, it will be fixed ASAP.

[IIT Software]

We've found a bug - sorry! It comes up if you reply to temp queues by using pooled identified producers for these temp queues (via Spring's or SwiftMQ's caching, no matter) without really closing them (because the pool converts the close into a pool checkin) & if you use transacted sessions.

It only occurs if you use identified producers for the replies, that is, you create a producer per temp. queue. It does not occur if you use an unidentified producer (session.createProducer(null)). So I recommend to use an unidentified producer which is the recommended way for request/reply anyway (saves resources).

[Bryan Talbot]

As you can guess by the topic and sample application, my application is a Spring pojo-jms application.  I believe that to use unidentified producers, a major portion of that framework would need to be abandoned and replaced with new code.  That's not something I'd look forward to doing.

Can you make that recommendation to the Spring devs to help spring be more friendly with SwiftMQ?  I'd think they'd welcome a patch or the suggestion at least.

This bug is the cause of bug #2 in my original post.  The most serious issue to me is #1.  Are you able to reproduce that object leak?

[IIT Software]

The leak is related (internally caused by the TransactionClosedException).

You should also be able to work around this problem by temporarily switching over to nontransacted sessions for your request/reply until this has been fixed (soon). Patches are provided to tech support customers only, not via this free forum.

[Bryan Talbot]

IIT Software wrote:

The question is why there is a transaction timeout at all. JMS local transactions never timeout and it is not spec compliant to call rollback/commit from another thread to terminate a receive. You can only call session.close from another thread. However, setting receiveTimeout=-1 will avoid this strange behavior to commit empty transactions every second. So you should use -1.

I think the reason the timeout exists is to allow the VM to exit.  When I set the timeout to -1 with the spring app provided, the process hangs on exit.  A thread dump reveals that this thread is still waiting on a receive() call with no timeout.  I have to use "kill -9" to terminate the process.


"jmsContainer-1" prio=5 tid=0x01018b80 nid=0x82d800 in Object.wait() [0xb0f90000..0xb0f90d90]
        at java.lang.Object.wait(Native Method)
        - waiting on <0x255c51a8> (a com.swiftmq.jms.v630.QueueReceiverImpl)
        at java.lang.Object.wait(Object.java:474)
        at com.swiftmq.tools.util.UninterruptableWaiter.doWait(Unknown Source)
        at com.swiftmq.jms.v630.MessageConsumerImpl.receiveMessage(Unknown Source)
        - locked <0x255c51a8> (a com.swiftmq.jms.v630.QueueReceiverImpl)
        at com.swiftmq.jms.v630.MessageConsumerImpl.receive(Unknown Source)
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveMessage(AbstractPollingMessageListenerContainer.java:405)
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:308)
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:241)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:982)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:974)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:876)
        at java.lang.Thread.run(Thread.java:613)

[Bryan Talbot]

I'm a paid up customer.  How should I request the patch when it's available?

[IIT Software]

Yes, I experienced it too. Reason is that you've set a shutdown hook from Spring which leads to a deadlock because, due to the close interception of the pooling, the JMS connection isn't closed actually (which will release the receive). If you use our springsupport, define the destroy method which closea the connection.

[IIT Software]

If you have valid maintenance, you can download new releases free of charge once available..

[IIT Software]

This has also been fixed in 7.5.0, released today.

[batalbot]

There were three issues reported in the first post.

Bryan Talbot wrote:

1.  An object leak in SwiftMQ QueueReceiver objects when transactions are used and consumers (and sessions and connections) are not cached.
2.  A bug of springsupport's SingleSharedConnectionFactory that causes all producers to be cached -- even if they're only used once to reply on a temporary queue named in a ReplyTo header.  Since these producers never actually get closed, the max-producers limit for the connection is quickly reached.
3.  A bug causing "transaction closed" exceptions after sending a response only when run on OSX.  This occurs with both Java 1.5 and 1.6.  This bug doesn't seem to happen on Linux.

I've tested these with 7.5.0 with the following results.

#1 seems to be resolved when consuming from a physical queue but the issue still exists when consuming from a clustered queue (which is my use case unfortunately).  This eventually leads to crashing the swiftmq router (due to OOM) and cannot be cleared by restarting the client.

#2 seems to be unchanged.  SingleSharedConnectionFactory will still fail when used in request-response after responding on 50 (the default value) different temp queues.  This is with using spring's JMS pojo classes so using anonymous producers isnt' an option without discarding the use of those spring features.

I haven't tested #3 much yet but it's the lowest priority issue for me.

[IIT Software]

We will test #1 with clustered queues. Do you use the clustered queue name for your consumer (so the clustered queue will chose a phys. queue for you) or the name of a phys. queue binding?

Concerning #2: You can solve it by specifying a low pool expiration interval, e.g. 1000 (1 sec). The max-producers value of the resource limit group is only a value after which a resource limit exception is thrown. That's actually a tool for an admin to detect clients with resource leaks. Set it to a value you will not exceed within the pool expiration interval of your client (e.g. replies per second).

#3 should work. I've tested it under OSX and Linux.

[batalbot]

IIT Software wrote:

We will test #1 with clustered queues. Do you use the clustered queue name for your consumer (so the clustered queue will chose a phys. queue for you) or the name of a phys. queue binding?

Yes, the consumers connect using the clustered queue name and not the physical queue name.

[IIT Software]

You're right. There is a leak with consumers on a clustered queue if it is used like your case #1. We will fix it soon. Can you live with it meanwhile? The leak doesn't appear if you use our springsupport.

[batalbot]

Yes, we can live with the leak for a bit.

[batalbot]

IIT Software wrote:

Concerning #2: You can solve it by specifying a low pool expiration interval, e.g. 1000 (1 sec). The max-producers value of the resource limit group is only a value after which a resource limit exception is thrown.

I have a question about the pool expiration interval.  Are objects invalidated after this many milliseconds of non-use or after they are this many milliseconds old?

It looks like the expiration interval is a global setting and not per-object-type, so how effective is the caching of sessions and connections if the expiration time is set very small?