Problem shutting down cleanly, with jencks, tomcat....

Hi,

I am having trouble getting a clean shutdown.  I am using tomcat, with Spring, activeMQ, and Jencks....

I am using the Spring's DefaultMessageListenerContainer.   I have included the exception below.  It's spurious, and sometimes the threads just seem to hang (also included a hung thread stack trace below).  Essentially, I think the connections are closing prior to the listener, and so the listener is trying to repair a broken connection....

What I need, I suppose, is to have the listener container quit prior to the activeMQ connections shutdown.  I'm not sure how to ensure this sequence, with spring, et al....

Any ideas?  I wonder if it has to do with the redelivery policies I've set for the connection...

I'm using jencks 2.0, ActiveMQ 4.1.1, Spring 2.0.7, tomcat 6.0.14, Java 6....

Here's the spring definitions (assume reasonable values for the ${} placeholders there):

  <bean id="jmsTransactionManager" class="org.jencks.factory.TransactionManagerFactoryBean" />

  <bean id="connectionManager" class="org.jencks.factory.ConnectionManagerFactoryBean">
    <property name="transactionManager" ref="jmsTransactionManager" />
  </bean>

  <!--  Connection Factory with Limited Redelivery Policy -->
  <bean id="jmsManagedConnectionFactoryLimited" class="org.apache.activemq.ra.ActiveMQManagedConnectionFactory">
    <property name="resourceAdapter" ref="activeMQResourceAdapter" />
    <property name="queuePrefetch" value="1" />
    <property name="initialRedeliveryDelay" value="${env.jms.initialRedeliveryDelay.limited}" />
    <property name="maximumRedeliveries" value="${env.jms.maximumRedeliveries.limited}" />
    <property name="redeliveryBackOffMultiplier" value="${env.jms.redeliveryBackOffMultiplier.limited}" />
    <property name="redeliveryUseExponentialBackOff" value="${env.jms.redeliveryUseExponentialBackOff.limited}" />
  </bean>

  <bean id="jmsConnectionFactoryLimited" class="org.jencks.factory.ConnectionFactoryFactoryBean">
    <property name="managedConnectionFactory" ref="jmsManagedConnectionFactoryLimited" />
    <property name="connectionManager" ref="connectionManager" />
  </bean>

  <bean id="redirectListener"
    class="org.springframework.jms.listener.DefaultMessageListenerContainer">
    <property name="concurrentConsumers" value="150" />
    <property name="maxConcurrentConsumers" value="150" />
    <property name="sessionTransacted" value="true" />
    <property name="connectionFactory" ref="jmsConnectionFactoryLimited" />
    <property name="destinationName" value="my.dest" />
    <property name="messageListener" ref="myListenerMdp" />
  </bean>


Here's a hung thread, which commonly occurs during tomcat shutdown:

"redirectListener-1" prio=5 tid=0x02a11000 nid=0x192c400 in Object.wait() [0xb0f0e000..0xb0f0ecd0]
   java.lang.Thread.State: WAITING (on object monitor)
   JavaThread state: _thread_blocked
Thread: 0x02a11000  [0x192c400] State: _at_safepoint _has_called_back 0 _at_poll_safepoint 0
   JavaThread state: _thread_blocked
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:485)
        at edu.emory.mathcs.backport.java.util.concurrent.locks.CondVar.await(CondVar.java:75)
        - locked <0x25b3bfe0> (a edu.emory.mathcs.backport.java.util.concurrent.locks.CondVar)
        at edu.emory.mathcs.backport.java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:318)
        at org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:42)
        at org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:75)
        at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1175)
        at org.apache.activemq.ActiveMQSession.syncSendPacket(ActiveMQSession.java:1663)
        at org.apache.activemq.ActiveMQMessageConsumer.close(ActiveMQMessageConsumer.java:542)
        at org.springframework.jms.support.JmsUtils.closeMessageConsumer(JmsUtils.java:144)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.clearResources(DefaultMessageListenerContainer.java:898)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:864)
        at java.lang.Thread.run(Thread.java:637)



Here's an exception stack trace, which also commonly occurs:

WARN  [2007-11-06 10:07:38,703] thread:AcitveMQ Connection Worker: tcp://localhost/127.0.0.1:61616      ActiveMQManagedConnection -- Connection failed: javax.jms.JMSException: Cannot lookup a connection that had not been registered: ID: jrosenberg.local-22322-2342323342-1:1
WARN  [2007-11-06 10:07:38,709] thread:AcitveMQ Connection Worker: tcp://localhost/127.0.0.1:61616 GeronimoConnectionEventListener -- connectionErrorOccurred called with null
javax.jms.JMSException: Cannot lookup a connection that had not been registered: ID:szjrosenberg-mbp17.local-60069-1194343647880-1:1
        at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:46)
        at org.apache.activemq.ActiveMQConnection.onAsyncException(ActiveMQConnection.java:1519)
        at org.apache.activemq.ActiveMQConnection$2.run(ActiveMQConnection.java:1492)
        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
        at java.lang.Thread.run(Thread.java:637)
Caused by: java.lang.IllegalStateException: Cannot lookup a connection that had not been registered: ID:szjrosenberg-mbp17.local-60069-1194343647880-1:1
        at org.apache.activemq.broker.TransportConnection.lookupConnectionState(TransportConnection.java:343)
        at org.apache.activemq.broker.TransportConnection.processRemoveSession(TransportConnection.java:634)
        at org.apache.activemq.command.RemoveInfo.visit(RemoveInfo.java:62)
        at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:294)
        at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:185)
        at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:65)
        at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:133)
        at org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:122)
        at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
        at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:137)
        ... 1 more