JMS + SimpleRetryConnectionStrategy +JNDI +OracleJMS ... not working

[kennywest]

Guys,

I am using Mule to connect to an oracle JMS provider. Below is the configuration for the connector:
        <connector name="oracleJmsCustomerConnector" className="org.mule.providers.jms.JmsConnector">
                <properties>
                        <property name="jndiProviderUrl" value="opmn:ormi://host:OC4J_SOJMS"/>
                        <map name="jndiProviderProperties">
                                <property name="java.naming.factory.initial" value="com.evermind.server.rmi.RMIInitialContextFactory"/>
                                <property name="java.naming.security.principal" value="admin"/>
                                <property name="java.naming.security.credentials" value="welcome"/>
                        </map>
                        <property name="connectionFactoryJndiName" value="java:comp/resource/SOJMSDS/QueueConnectionFactories/customers_qt"/>
                        <property name="jndiDestinations" value="true"/>
                        <property name="forceJndiDestinations" value="true"/>
                </properties>
                <connection-strategy className="org.mule.providers.SimpleRetryConnectionStrategy">
                        <properties>
                                <property name="retryCount" value="5"/>
                                <property name="frequency" value="1800000"/>
                                <!-- see http://jira.symphonysoft.com/browse/MULE-708 -->
                                <property name="doThreading" value="true"/>
                        </properties>
                </connection-strategy>
        </connector>

This works like a charm. Somewhere during the night, Mule loses its connection. Not sure what happens on the oracle side, so don't ask ;)

The SimpleRetryConnectionStrategy is configured to try a few times to establish the connection again. This fails, even when oracle is back.

Due to historical reasons, I'm still using 1.3rc4, which was the last release candidate before 1.3 was released. Below is a stacktrace:
ERROR 2007-02-17 01:52:23,643 [UMOManager.4] org.mule.providers.SimpleRetryConnectionStrategy: Failed to connect/reconnect on endpoint: jms:///?address=java:comp/resource/SOJMSDS/Queues/customers_q. Root Exception was: Failed to look up destination: Disconnected: Connection reset(JMS Code: null). Type: class javax.jms.JMSException
org.mule.providers.ConnectException: Initialisation Failure: Failed to look up destination: Disconnected: Connection reset
        at org.mule.providers.jms.SingleJmsMessageReceiver.createConsumer(SingleJmsMessageReceiver.java:197)
        at org.mule.providers.jms.SingleJmsMessageReceiver.doConnect(SingleJmsMessageReceiver.java:69)
        at org.mule.providers.AbstractMessageReceiver.connect(AbstractMessageReceiver.java:344)
        at org.mule.providers.SimpleRetryConnectionStrategy.doConnect(SimpleRetryConnectionStrategy.java:51)
        at org.mule.providers.AbstractConnectionStrategy$1.run(AbstractConnectionStrategy.java:53)
        at org.mule.impl.work.WorkerContext.run(WorkerContext.java:290)
        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1486)
        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:391)
        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:865)
        at org.mule.impl.work.WorkExecutorPoolImpl.execute(WorkExecutorPoolImpl.java:88)
        at org.mule.impl.work.ScheduleWorkExecutor.doExecute(ScheduleWorkExecutor.java:35)
        at org.mule.impl.work.MuleWorkManager.executeWork(MuleWorkManager.java:272)
        at org.mule.impl.work.MuleWorkManager.scheduleWork(MuleWorkManager.java:241)
        at org.mule.providers.AbstractConnectionStrategy.connect(AbstractConnectionStrategy.java:44)
        at org.mule.providers.AbstractMessageReceiver.connect(AbstractMessageReceiver.java:338)
        at org.mule.providers.SimpleRetryConnectionStrategy.doConnect(SimpleRetryConnectionStrategy.java:51)
        at org.mule.providers.AbstractConnectionStrategy$1.run(AbstractConnectionStrategy.java:53)
        at org.mule.impl.work.WorkerContext.run(WorkerContext.java:290)
        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1486)
        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:391)
        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:865)
        at org.mule.impl.work.WorkExecutorPoolImpl.execute(WorkExecutorPoolImpl.java:88)
        at org.mule.impl.work.ScheduleWorkExecutor.doExecute(ScheduleWorkExecutor.java:35)
        at org.mule.impl.work.MuleWorkManager.executeWork(MuleWorkManager.java:272)
        at org.mule.impl.work.MuleWorkManager.scheduleWork(MuleWorkManager.java:241)
        at org.mule.providers.AbstractConnectionStrategy.connect(AbstractConnectionStrategy.java:44)
        at org.mule.providers.AbstractMessageReceiver.start(AbstractMessageReceiver.java:375)
        at org.mule.providers.AbstractConnector.startConnector(AbstractConnector.java:334)
        at org.mule.providers.AbstractConnector.connect(AbstractConnector.java:924)
        at org.mule.providers.SimpleRetryConnectionStrategy.doConnect(SimpleRetryConnectionStrategy.java:51)
        at org.mule.providers.AbstractConnectionStrategy$1.run(AbstractConnectionStrategy.java:53)
        at org.mule.impl.work.WorkerContext.run(WorkerContext.java:290)
        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1486)
        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:391)
        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:865)
        at org.mule.impl.work.WorkExecutorPoolImpl.execute(WorkExecutorPoolImpl.java:88)
        at org.mule.impl.work.ScheduleWorkExecutor.doExecute(ScheduleWorkExecutor.java:35)
        at org.mule.impl.work.MuleWorkManager.executeWork(MuleWorkManager.java:272)
        at org.mule.impl.work.MuleWorkManager.scheduleWork(MuleWorkManager.java:241)
        at org.mule.providers.AbstractConnectionStrategy.connect(AbstractConnectionStrategy.java:44)
        at org.mule.providers.AbstractConnector.connect(AbstractConnector.java:899)
        at org.mule.providers.SimpleRetryConnectionStrategy.doConnect(SimpleRetryConnectionStrategy.java:51)
        at org.mule.providers.AbstractConnectionStrategy$1.run(AbstractConnectionStrategy.java:53)
        at org.mule.impl.work.WorkerContext.run(WorkerContext.java:290)
        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
        at java.lang.Thread.run(Thread.java:595)
Caused by: javax.jms.JMSException: Failed to look up destination: Disconnected: Connection reset
        at org.mule.providers.jms.Jms11Support.getJndiDestination(Jms11Support.java:139)
        at org.mule.providers.jms.Jms11Support.createDestination(Jms11Support.java:118)
        at org.mule.providers.jms.SingleJmsMessageReceiver.createConsumer(SingleJmsMessageReceiver.java:169)
        ... 46 more

Correct me if I'm wrong, but upgrading to 1.3.3 will not fix my problem, I think. I have been looking through the code and I think the root cause of the problem is the fact that I'm not only loosing the connection with my JMSProvider, but also with the JNDI store (hosted on the same machine).
Could it be that the retry strategy is not refreshing its connection with the JNDI store? If so, how can I make sure that the retry strategy refreshes its connection with the JNDI store and _then_ tries to connect with the JMS queues again.
Thanks in advance.

P.S.: I've been searching this list about retry strategies and JMS, but haven't found a similar problem yet.

[kennywest]

I've done some further digging. The JNDI context is created near initJndiContext() in JMSMessageConnector. This is only called from the doConnect(), if connectionFactory == null method.
If somewhere the connection with the JNDI store is lost, the jndiContext member will fail when doing getJndiDestination() in Jms11Support.
Therefore the reconnect strategy does not cover loss of connections with JNDI stores. Am I correct?

Pleas note, I am not criticizing anything, just looking for the right solution ;)

[Andrew Perepelytsya]

Ken,

I think you're right, the JNDI connection is not recycled. Note however, this is not a full explanation. Most probably it's very implementation-specific, e.g. I had similar symptoms (unable to recover) with JBoss when running in HA/clustered mode with HA-JNDI, but it worked flawlessly with normal JNDI.

I've filed http://mule.mulesource.org/jira/browse/MULE-1416 for this.

Andrew

On 2/21/07, kennywest <kenneth.westelinck@...> wrote:

I've done some further digging. The JNDI context is created near
initJndiContext() in JMSMessageConnector. This is only called from the
doConnect(), if connectionFactory == null method.
If somewhere the connection with the JNDI store is lost, the jndiContext
member will fail when doing getJndiDestination() in Jms11Support.
Therefore the reconnect strategy does not cover loss of connections with
JNDI stores. Am I correct?

Pleas note, I am not criticizing anything, just looking for the right
solution ;)

--
View this message in context: http://www.nabble.com/JMS-%2B-SimpleRetryConnectionStrategy-%2BJNDI-%2BOracleJMS-...-not-working-tf3265566.html#a9082171
Sent from the Mule - User mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[kennywest]

Andrew,

Thanks for your reply. I have wrapped the RMIInitialContextFactory and RMIContext objects from Oracle in a set of custom pojos. If a lookup fails, a new RMIContext is created. Not sure if it will work, but worth the try, right ;)
Anyway, I'll keep you updated.


regards,

Kenneth

[Andrew Perepelytsya]

Kenneth,

Can you check if it can be implemented with a standard JDK proxy? That would provide a consistent approach with the invocation handler.

Andrew

On 2/22/07, kennywest <kenneth.westelinck@...> wrote:

Andrew,

Thanks for your reply. I have wrapped the RMIInitialContextFactory and
RMIContext objects from Oracle in a set of custom pojos. If a lookup fails,
a new RMIContext is created. Not sure if it will work, but worth the try,
right ;)
Anyway, I'll keep you updated.


regards,

Kenneth
--
View this message in context: http://www.nabble.com/JMS-%2B-SimpleRetryConnectionStrategy-%2BJNDI-%2BOracleJMS-...-not-working-tf3265566.html#a9103807
Sent from the Mule - User mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[kennywest]

Andrew Perepelytsya wrote:

Kenneth,

Can you check if it can be implemented with a standard JDK proxy? That would
provide a consistent approach with the invocation handler.

Andrew

Ehm ... sure. Andrew, can you elaborate some more on this, I'm not completely sure what you're meaning / trying to accomplish.

[Andrew Perepelytsya]

The java.lang.reflect. Proxy can decorate the context factory and context classes (they both have interfaces) and delegate to the original class. This is effectively an around advice.

The way clients see it, it's still original class.

When such proxy detects a failure, it can 'reinitialise' behind the scene. See the org.mule.providers.jms.xa.ConnectionFactoryWrapper for a good usage sample.

Andrew

On 2/23/07, kennywest <kenneth.westelinck@...> wrote:

Andrew Perepelytsya wrote:
>
> Kenneth,
>
> Can you check if it can be implemented with a standard JDK proxy? That
> would
> provide a consistent approach with the invocation handler.
>
> Andrew
>

Ehm ... sure. Andrew, can you elaborate some more on this, I'm not
completely sure what you're meaning / trying to accomplish.
--
View this message in context: http://www.nabble.com/JMS-%2B-SimpleRetryConnectionStrategy-%2BJNDI-%2BOracleJMS-...-not-working-tf3265566.html#a9115157
Sent from the Mule - User mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[kennywest]

I am still experiencing major problems with 1.3rc4. At startup, I can see the following logged:

ERROR 2007-03-08 08:39:43,839 [main] org.mule.providers.SimpleRetryConnectionStrategy: Failed to connect/reconnect on endpoint: jms:///?address=java:comp/resource/SOJMSDS/Queues/agencies_q. Root Exception was: Not connected(JMS Code: null). Type: class javax.jms.JMSException
org.mule.providers.ConnectException: Initialisation Failure: Not connected
        at org.mule.providers.jms.SingleJmsMessageReceiver.createConsumer(SingleJmsMessageReceiver.java:197)
        at org.mule.providers.jms.SingleJmsMessageReceiver.doConnect(SingleJmsMessageReceiver.java:69)
        at org.mule.providers.AbstractMessageReceiver.connect(AbstractMessageReceiver.java:344)
        at org.mule.providers.SimpleRetryConnectionStrategy.doConnect(SimpleRetryConnectionStrategy.java:51)
        at org.mule.providers.AbstractConnectionStrategy$1.run(AbstractConnectionStrategy.java:53)
        at org.mule.impl.work.WorkerContext.run(WorkerContext.java:290)
        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1486)
        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:391)
        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:865)
        at org.mule.impl.work.WorkExecutorPoolImpl.execute(WorkExecutorPoolImpl.java:88)
        at org.mule.impl.work.ScheduleWorkExecutor.doExecute(ScheduleWorkExecutor.java:35)
        at org.mule.impl.work.MuleWorkManager.executeWork(MuleWorkManager.java:272)
        at org.mule.impl.work.MuleWorkManager.scheduleWork(MuleWorkManager.java:241)
        at org.mule.providers.AbstractConnectionStrategy.connect(AbstractConnectionStrategy.java:44)
        at org.mule.providers.AbstractMessageReceiver.connect(AbstractMessageReceiver.java:338)
        at org.mule.impl.model.ComponentUtil.connectListeners(ComponentUtil.java:179)
        at org.mule.impl.model.ComponentUtil.start(ComponentUtil.java:51)
        at org.mule.impl.model.ComponentUtil.start(ComponentUtil.java:35)
        at org.mule.impl.model.AbstractModel.start(AbstractModel.java:305)
        at org.mule.MuleManager.start(MuleManager.java:755)
        at org.mule.config.builders.MuleXmlConfigurationBuilder.configure(MuleXmlConfigurationBuilder.java:205)
        at org.mule.config.builders.MuleXmlConfigurationBuilder.configure(MuleXmlConfigurationBuilder.java:191)
        at org.mule.MuleServer.initialize(MuleServer.java:245)
        at org.mule.MuleServer.run(MuleServer.java:159)
        at org.mule.MuleServer.start(MuleServer.java:148)
        at org.mule.MuleServer.main(MuleServer.java:121)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.mule.tools.launcher.Launcher.run(Launcher.java:365)
        at org.mule.tools.launcher.Launcher.main(Launcher.java:140)
Caused by: javax.jms.JMSException: Not connected
        at org.mule.providers.jms.JmsConnector.getSession(JmsConnector.java:342)
        at org.mule.providers.jms.JmsConnector.getSession(JmsConnector.java:556)
        at org.mule.providers.jms.SingleJmsMessageReceiver.createConsumer(SingleJmsMessageReceiver.java:156)
        ... 31 more
INFO  2007-03-08 08:39:43,839 [main] org.mule.providers.SimpleRetryConnectionStrategy: Waiting for 1800000ms before reconnecting. Failed attempt 3 of 5

I think it finally manages to connect, but for some reason the component receiving this messages is stopped:
ERROR 2007-03-08 08:39:45,636 [oracleJmsTravelAgentConnector.ReadBETravelAgent.In.receiver.1] org.mule.impl.DefaultExceptionStrategy:
********************************************************************************
Message               : Cannot route event as component "ReadBETravelAgent" is stopped. Component that caused exception is: ReadBETravelAgent. Message payload is of type: oracle.jms.AQjmsTextMessage
Type                  : org.mule.umo.ComponentException
Code                  : 79999
JavaDoc               : http://mule.codehaus.org/docs/apidocs/org/mule/umo/ComponentException.html
Payload               : oracle.jms.AQjmsTextMessage@147bc27
********************************************************************************
Exception stack is:
1. Cannot route event as component "ReadBETravelAgent" is stopped. Component that caused exception is: ReadBETravelAgent. Message payload is of type: oracle.jms.AQjmsTextMessage (org.mule.umo.ComponentException)
  org.mule.impl.model.AbstractComponent:209 (http://mule.codehaus.org/docs/apidocs/org/mule/umo/ComponentException.html)
********************************************************************************
Root Exception stack trace:
org.mule.umo.ComponentException: Cannot route event as component "ReadBETravelAgent" is stopped. Component that caused exception is: ReadBETravelAgent. Message payload is of type: oracle.jms.AQjmsTextMessage
        at org.mule.impl.model.AbstractComponent.dispatchEvent(AbstractComponent.java:209)
        at org.mule.impl.MuleSession.dispatchEvent(MuleSession.java:231)
        at org.mule.routing.inbound.InboundMessageRouter.dispatch(InboundMessageRouter.java:146)
        at org.mule.routing.inbound.InboundMessageRouter.route(InboundMessageRouter.java:125)
        at org.mule.providers.AbstractMessageReceiver$DefaultInternalMessageListener.onMessage(AbstractMessageReceiver.java:457)
        at org.mule.providers.AbstractMessageReceiver.routeMessage(AbstractMessageReceiver.java:263)
        at org.mule.providers.AbstractMessageReceiver.routeMessage(AbstractMessageReceiver.java:217)
        at org.mule.providers.AbstractMessageReceiver.routeMessage(AbstractMessageReceiver.java:211)
        at org.mule.providers.jms.JmsMessageReceiver$Worker.run(JmsMessageReceiver.java:81)
        at org.mule.impl.work.WorkerContext.run(WorkerContext.java:290)
        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
        at java.lang.Thread.run(Thread.java:595)

********************************************************************************

The connector using the following configuration:
        <connector name="oracleJmsTravelAgentConnector" className="org.mule.providers.jms.JmsConnector">
                <properties>
                        <property name="jndiProviderUrl" value="opmn:ormi://host:OC4J_SOJMS"/>
                        <map name="jndiProviderProperties">
                                <property name="java.naming.factory.initial" value="com.evermind.server.rmi.RMIInitialContextFactory"/>
                                <!--property name="java.naming.factory.initial" value="com.thomascook.naming.oracle.RMIInitialContextFactory"/-->
                                <property name="java.naming.security.principal" value="admin"/>
                                <property name="java.naming.security.credentials" value="welcome"/>
                        </map>
                        <property name="connectionFactoryJndiName" value="java:comp/resource/SOJMSDS/QueueConnectionFactories/agencies_qt"/>
                        <property name="jndiDestinations" value="true"/>
                        <property name="forceJndiDestinations" value="true"/>
                </properties>
                <connection-strategy className="org.mule.providers.SimpleRetryConnectionStrategy">
                        <properties>
                                <property name="retryCount" value="5"/>
                                <property name="frequency" value="1800000"/>
                                <!-- see http://jira.symphonysoft.com/browse/MULE-708 -->
                                <property name="doThreading" value="true"/>
                        </properties>
                </connection-strategy>
        </connector>

I am not quite sure what is happening here. Is there anyone who can shed some light on this?