Upgrade ActiveMQ in Geronimo 2.1.4 to improve reliability

[Fredrik Jonson-3]

Hi all,

I'm experiencing some reliability issues with ActiveMQ in G 2.1.4 in
combination with the application I'm working on now. Basically I see the same
three issues described in GERONIMO-4257[0].  Unfortunately I can't reproduce
the issues reliably, but still I'm not comfortable to put G into production at
its current state.

[0] https://issues.apache.org/jira/browse/GERONIMO-4257

As a stop gap solution before G 2.2 is stable I'm considering upgrading
ActiveMQ in G 2.1.4. Running with a stand alone broker using ActiveMQ 5.2 seems
to better, but I'm not entirely confident that we won't run into any unintended
classpath issues while still having ActiveMQ 4.1.2-G20090207 deployed in the
application server. Further It'd be convenient to have the 5.2 broker deployed
in the application server.

What I hope to elicit from you on this list is some feedback, is it reasonable,
etc...

So, here's my plan:

undeploy org.apache.geronimo.plugins/activemq-console-jetty/2.1.4/car
undeploy org.apache.geronimo.configs/activemq-ra/2.1.4/car
undeploy org.apache.geronimo.configs/activemq-broker/2.1.4/car

install-library org.apache.activemq/activemq-core/5.2.0/jar
install-library org.apache.activemq/activeio-core/3.1.0/jar
install-library org.apache.activemq/activemq-rar/5.2.0/jar

# Seems like openejb depends on activemq-broker so i now need to
# create a plugin that artifact-aliases ...configs/activemq-broker/2.1.4/car
# and starts a 5.2.0 broker.
deploy said module

#Is there a uninstall-library command, or is "rm -rf whatever" fine?
rm -rf $G/repository/org/apache/activemq/activemq-core/4.1.2-G20090207
rm -rf $G/repository/org/apache/activemq/activeio-core/3.0.1

That's pretty much it. Are there any glaring omissions in this plan? Anyone
else done anything similar before? Is it reasonable? Any feedback much
appreciated. Thanks!

--
Fredrik Jonson

[Fredrik Jonson-3]

Hi again,

I haven't given up on my little project to replace the old and not so solid
ActiveMQ 4.1.2 in geronimo 2.1.4 with ActiveMQ 5.2.0. Not yet anyway.

Right now I've replaced activem-broker and geronimo-activemq-ra with two custom
packages. And with these packages installed I can start geronimo just fine. I'm
using a modified Geronimo 2.1.4 Jetty 6 JavaEE.

Unfortunately when I try to deploy an ear that contains MDB:s I get an odd
class cast exception:

2009-07-16 13:46:56,854 ERROR [GBeanInstanceState] Error while starting;
  GBean is now in the FAILED state: abstractName="com.example/custom-server-ear/
        1.0.1-SNAPSHOT/ear?J2EEApplication=com.example/custom-server-ear/
        1.0.1-SNAPSHOT/ear,JCAResource=jms-resources,ResourceAdapter=
        jms-resources,ResourceAdapterModule=jms-resources,j2eeType=
        JCAResourceAdapter,name=jms-resources"
java.lang.ClassCastException: org.apache.activemq.ra.ActiveMQResourceAdapter cannot be cast to javax.resource.spi.ResourceAdapter
        at org.apache.geronimo.connector.ResourceAdapterWrapper.<init>(ResourceAdapterWrapper.java:79)
        at org.apache.geronimo.connector.ResourceAdapterWrapperGBean.<init>(ResourceAdapterWrapperGBean.java:51)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at org.apache.geronimo.gbean.runtime.GBeanInstance.createInstance(GBeanInstance.java:948)
        at org.apache.geronimo.gbean.runtime.GBeanInstanceState.attemptFullStart(GBeanInstanceState.java:268)
        at org.apache.geronimo.gbean.runtime.GBeanInstanceState.start(GBeanInstanceState.java:102)
        at org.apache.geronimo.gbean.runtime.GBeanInstanceState.startRecursive(GBeanInstanceState.java:124)
        at org.apache.geronimo.gbean.runtime.GBeanInstance.startRecursive(GBeanInstance.java:555)
        at org.apache.geronimo.kernel.basic.BasicKernel.startRecursiveGBean(BasicKernel.java:379)
        at org.apache.geronimo.kernel.config.ConfigurationUtil.startConfigurationGBeans(ConfigurationUtil.java:456)
        at org.apache.geronimo.kernel.config.KernelConfigurationManager.start(KernelConfigurationManager.java:188)
        at org.apache.geronimo.kernel.config.SimpleConfigurationManager.startConfiguration(SimpleConfigurationManager.java:563)
        at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.geronimo.gbean.runtime.ReflectionMethodInvoker.invoke(ReflectionMethodInvoker.java:34)
        at org.apache.geronimo.gbean.runtime.GBeanOperation.invoke(GBeanOperation.java:124)
        at org.apache.geronimo.gbean.runtime.GBeanInstance.invoke(GBeanInstance.java:832)
        at org.apache.geronimo.gbean.runtime.RawInvoker.invoke(RawInvoker.java:57)
        at org.apache.geronimo.kernel.basic.RawOperationInvoker.invoke(RawOperationInvoker.java:35)
        at org.apache.geronimo.kernel.basic.ProxyMethodInterceptor.intercept(ProxyMethodInterceptor.java:96)
        at org.apache.geronimo.gbean.GBeanLifecycle$$EnhancerByCGLIB$$8e753450.startConfiguration(<generated>)
        at org.apache.geronimo.system.main.EmbeddedDaemon.doStartup(EmbeddedDaemon.java:162)
        at org.apache.geronimo.system.main.EmbeddedDaemon.execute(EmbeddedDaemon.java:79)
        at org.apache.geronimo.kernel.util.MainConfigurationBootstrapper.main(MainConfigurationBootstrapper.java:45)
        at org.apache.geronimo.cli.AbstractCLI.executeMain(AbstractCLI.java:67)
        at org.apache.geronimo.cli.daemon.DaemonCLI.main(DaemonCLI.java:30)

Does anyone have a hunch what might be happening here?

--
Fredrik Jonson

[djencks]

On Jul 16, 2009, at 5:03 AM, Fredrik Jonson wrote:

> Hi again,
>
> I haven't given up on my little project to replace the old and not  
> so solid
> ActiveMQ 4.1.2 in geronimo 2.1.4 with ActiveMQ 5.2.0. Not yet anyway.
>
> Right now I've replaced activem-broker and geronimo-activemq-ra with  
> two custom
> packages. And with these packages installed I can start geronimo  
> just fine. I'm
> using a modified Geronimo 2.1.4 Jetty 6 JavaEE.
>
> Unfortunately when I try to deploy an ear that contains MDB:s I get  
> an odd
> class cast exception:
>
> 2009-07-16 13:46:56,854 ERROR [GBeanInstanceState] Error while  
> starting;
>  GBean is now in the FAILED state: abstractName="com.example/custom-
> server-ear/
>        1.0.1-SNAPSHOT/ear?J2EEApplication=com.example/custom-server-
> ear/
>        1.0.1-SNAPSHOT/ear,JCAResource=jms-resources,ResourceAdapter=
>        jms-resources,ResourceAdapterModule=jms-resources,j2eeType=
>        JCAResourceAdapter,name=jms-resources"
> java.lang.ClassCastException:  
> org.apache.activemq.ra.ActiveMQResourceAdapter cannot be cast to  
> javax.resource.spi.ResourceAdapter
>        at  
> org
> .apache
> .geronimo
> .connector.ResourceAdapterWrapper.<init>(ResourceAdapterWrapper.java:
> 79)
>        at  
> org
> .apache
> .geronimo
> .connector
> .ResourceAdapterWrapperGBean.<init>(ResourceAdapterWrapperGBean.java:
> 51)
>        at  
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>        at  
> sun
> .reflect
> .NativeConstructorAccessorImpl
> .newInstance(NativeConstructorAccessorImpl.java:39)
>        at  
> sun
> .reflect
> .DelegatingConstructorAccessorImpl
> .newInstance(DelegatingConstructorAccessorImpl.java:27)
>        at java.lang.reflect.Constructor.newInstance(Constructor.java:
> 513)
>        at  
> org
> .apache
> .geronimo
> .gbean.runtime.GBeanInstance.createInstance(GBeanInstance.java:948)
>        at  
> org
> .apache
> .geronimo
> .gbean
> .runtime.GBeanInstanceState.attemptFullStart(GBeanInstanceState.java:
> 268)
>        at  
> org
> .apache
> .geronimo
> .gbean.runtime.GBeanInstanceState.start(GBeanInstanceState.java:102)
>        at  
> org
> .apache
> .geronimo
> .gbean
> .runtime.GBeanInstanceState.startRecursive(GBeanInstanceState.java:
> 124)
>        at  
> org
> .apache
> .geronimo
> .gbean.runtime.GBeanInstance.startRecursive(GBeanInstance.java:555)
>        at  
> org
> .apache
> .geronimo
> .kernel.basic.BasicKernel.startRecursiveGBean(BasicKernel.java:379)
>        at  
> org
> .apache
> .geronimo
> .kernel
> .config
> .ConfigurationUtil.startConfigurationGBeans(ConfigurationUtil.java:
> 456)
>        at  
> org
> .apache
> .geronimo
> .kernel
> .config
> .KernelConfigurationManager.start(KernelConfigurationManager.java:188)
>        at  
> org
> .apache
> .geronimo
> .kernel
> .config
> .SimpleConfigurationManager
> .startConfiguration(SimpleConfigurationManager.java:563)
>        at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
>        at  
> sun
> .reflect
> .DelegatingMethodAccessorImpl
> .invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at  
> org
> .apache
> .geronimo
> .gbean
> .runtime.ReflectionMethodInvoker.invoke(ReflectionMethodInvoker.java:
> 34)
>        at  
> org
> .apache
> .geronimo.gbean.runtime.GBeanOperation.invoke(GBeanOperation.java:124)
>        at  
> org
> .apache
> .geronimo.gbean.runtime.GBeanInstance.invoke(GBeanInstance.java:832)
>        at  
> org.apache.geronimo.gbean.runtime.RawInvoker.invoke(RawInvoker.java:
> 57)
>        at  
> org
> .apache
> .geronimo
> .kernel.basic.RawOperationInvoker.invoke(RawOperationInvoker.java:35)
>        at  
> org
> .apache
> .geronimo
> .kernel
> .basic.ProxyMethodInterceptor.intercept(ProxyMethodInterceptor.java:
> 96)
>        at org.apache.geronimo.gbean.GBeanLifecycle$$EnhancerByCGLIB$
> $8e753450.startConfiguration(<generated>)
>        at  
> org
> .apache
> .geronimo.system.main.EmbeddedDaemon.doStartup(EmbeddedDaemon.java:
> 162)
>        at  
> org
> .apache
> .geronimo.system.main.EmbeddedDaemon.execute(EmbeddedDaemon.java:79)
>        at  
> org
> .apache
> .geronimo
> .kernel
> .util
> .MainConfigurationBootstrapper
> .main(MainConfigurationBootstrapper.java:45)
>        at  
> org.apache.geronimo.cli.AbstractCLI.executeMain(AbstractCLI.java:67)
>        at  
> org.apache.geronimo.cli.daemon.DaemonCLI.main(DaemonCLI.java:30)
>
> Does anyone have a hunch what might be happening here?

My guess would be that
- you are using the amq 5.2 rar as supplied by amq, which contains  
waaaay too much stuff including the j2ca spec jar
- the classloader for the rar does not have the jee-specs plugin as a  
parent (or, rather, ancestor)

So the rar is loading the javax.resource classes in its own  
classloader and openejb is trying to load them in the jee-specs  
classloader.

Can I ask why you aren't using 2.2-SNAPSHOT and/or porting our amq 5.x  
integration back to 2.1.x?  AMQ 5.2 did have some tck issues in  
geronimo which is why we are using 5.3-SNAPSHOT and hoping for a  
imminent 5.3 release.

thanks
david jencks

>
> --
> Fredrik Jonson
>

[Fredrik Jonson-3]

David Jencks wrote:
 
>  My guess would be that
>  - you are using the amq 5.2 rar as supplied by amq, which contains
>  waaaay too much stuff including the j2ca spec jar.

No, not anymore. I ran into some issues around that in my first attempt.
Instead I package and depend on a custom and adopted version of
geronimo-activemq-ra.

>  - the classloader for the rar does not have the jee-specs plugin as a
>  parent (or, rather, ancestor).
>
>  So the rar is loading the javax.resource classes in its own  classloader
>  and openejb is trying to load them in the jee-specs  classloader.

Yes, that sounds more likely. Neither my custom version of the activemq-broker
or the geronimo-activemq-ra plugin project pom have any geronimo module as
parent. Is that the correct way to share ancestor? Or perhaps it doesn't have
anything to do with dependencies in the pom, but rather is a matter of how
the plan.xml is configured?

What configuration for the car-maven-plugin decides which classloader a plugin
will use? Does it have anything with the category tag?

>  Can I ask why you aren't using 2.2-SNAPSHOT and/or porting our amq 5.x
>  integration back to 2.1.x? AMQ 5.2 did have some tck issues in
>  geronimo which is why we are using 5.3-SNAPSHOT and hoping for a
>  imminent 5.3 release.

Yes, I'm also hoping that both the AMQ 5.3 and G 2.2 release are imminent.
The gotcha is that our product release is even more imminent.

I considered backporting the 2.2 plugins but was concerned the project would
depend to much on 2.2-specific functionality in dependencies. Also it seemed
less invasive to just cut out those few modules that depended on ActiveMQ,
and replace just the broker and as little as possible otherwise. Hindsight
maybe that wasn't the best assumption. ;)

Another aspect is that we've had good experience with the stability of 5.2
used as a standalone broker for our use cases. I don't know to what extent
5.3 has been tested, so that would seem like a bit of a uncertain choice.

--
Fredrik Jonson

[djencks]

On Jul 17, 2009, at 4:35 AM, Fredrik Jonson wrote:

> David Jencks wrote:
>
>> My guess would be that
>> - you are using the amq 5.2 rar as supplied by amq, which contains
>> waaaay too much stuff including the j2ca spec jar.
>
> No, not anymore. I ran into some issues around that in my first  
> attempt.
> Instead I package and depend on a custom and adopted version of
> geronimo-activemq-ra.
>
>> - the classloader for the rar does not have the jee-specs plugin as a
>> parent (or, rather, ancestor).
>>
>> So the rar is loading the javax.resource classes in its own  
>> classloader
>> and openejb is trying to load them in the jee-specs  classloader.
>
> Yes, that sounds more likely. Neither my custom version of the  
> activemq-broker
> or the geronimo-activemq-ra plugin project pom have any geronimo  
> module as
> parent. Is that the correct way to share ancestor? Or perhaps it  
> doesn't have
> anything to do with dependencies in the pom, but rather is a matter  
> of how
> the plan.xml is configured?
>
> What configuration for the car-maven-plugin decides which  
> classloader a plugin
> will use? Does it have anything with the category tag?

so, basically plugin == classloader + services.  The classloader is  
described by the environment element in the geronimo plan.  If you are  
using the car-maven-plugin to package the plugin, it will construct  
the environment element from your maven dependencies.  So if you have  
a maven dependency on e.g. jee-specs, that will turn into a parent  
classloader of your plugin.  You should do this :-)

>
>> Can I ask why you aren't using 2.2-SNAPSHOT and/or porting our amq  
>> 5.x
>> integration back to 2.1.x? AMQ 5.2 did have some tck issues in
>> geronimo which is why we are using 5.3-SNAPSHOT and hoping for a
>> imminent 5.3 release.
>
> Yes, I'm also hoping that both the AMQ 5.3 and G 2.2 release are  
> imminent.
> The gotcha is that our product release is even more imminent.
>
> I considered backporting the 2.2 plugins but was concerned the  
> project would
> depend to much on 2.2-specific functionality in dependencies. Also  
> it seemed
> less invasive to just cut out those few modules that depended on  
> ActiveMQ,
> and replace just the broker and as little as possible otherwise.  
> Hindsight
> maybe that wasn't the best assumption. ;)
>

What I would try is:

-- copy the activemq-broker and rar modules from 2.2 to your project,  
changing the groupId
-- look at the g. 2.2 processed plans and copy all the dependencies  
into the pom (in 2.2 geronimo follows maven transitive dependencies,  
in 2.1 geronimo does not, so you need more maven dependencies listed  
in 2.1)
-- change the amq version to 5.2 if you want

In production you probably don't need any of the other bits from the  
activmq integration like the admin console stuff.

> Another aspect is that we've had good experience with the stability  
> of 5.2
> used as a standalone broker for our use cases. I don't know to what  
> extent
> 5.3 has been tested, so that would seem like a bit of a uncertain  
> choice.

On the other hand 5.3 definitely has some bug fixes over 5.2.  But if  
5.2 works for you.... use it.

thanks
david jencks
>
> --
> Fredrik Jonson
>