JMS Queue Browsing Receiver

[0x6e6562]

I'm trying to create a JMS receiver that browses a queue in a controlling thread using the QueueBrowser interface and then when it finds something, for each message, it calls a worker thread to consume _a_ message with a non blocking call.

The idea is that the browsing thread can peek at the queue in a non-transactional session and if there is anything to consume, this can be farmed out to separate threads, thereby minimising the time any worker thread would idle when there is potentially work arrived in the queue but not yet farmed out.

I've included the source of the receiver and my stripped down mule-config.

BrowsingMessageReceiver.java
test-mule-config.xml

My problem is that the first enqueued message gets consumed correctly, but the second enqueued message cannot get consumed.

When dequeueing the second message, the call to this consumer's receiveNoWait() method returns null, although there _is_ a message in the queue.

This fact is backed up by the JMX queue browser and the fact that you can create a browser in the same session which allows you to peek at the message.

The logic then sets the rollBackOnly flag and returns.

This then just cycles, because the controlling thread can still see the same message on the queue and hence dispatches another worker thread.

I've tried this with the JMS and the XA transaction managers, to no avail.

The JMS provider is activemq 4.0.1, I haven't tried another provider yet.

I'm not too confident that my transaction handling is correct, as I don't feel that I quite mastered the mule transaction management yet.

What I find strange is the fact that it works for the first message, but then for no subsequent message.

Does anybody have ideas or things that I can try?

[Andrew Perepelytsya]

I know there were issues with queue browsers in ActiveMQ before. Not that it wasn't fixed by v4 (I just don't know now). Before that it resulted in messages getting reordered randomly.

The problem will need some investigation. Could you also list the activemq config file. Generally, the easier you make to reproduce the scenario (test client, etc.) the higher the chances are you'll get feedback.

Andrew

On 8/25/06, 0x6e6562 <ben.hood@...> wrote:

I'm trying to create a JMS receiver that browses a queue in a controlling
thread using the QueueBrowser interface and then when it finds something,
for each message, it calls a worker thread to consume _a_ message with a non
blocking call.

The idea is that the browsing thread can peek at the queue in a
non-transactional session and if there is anything to consume, this can be
farmed out to separate threads, thereby minimising the time any worker
thread would idle when there is potentially work arrived in the queue but
not yet farmed out.

I've included the source of the receiver and my stripped down mule-config.

http://www.nabble.com/user-files/235766/BrowsingMessageReceiver.java
BrowsingMessageReceiver.java
http://www.nabble.com/user-files/235767/test-mule-config.xml
test-mule-config.xml

My problem is that the first enqueued message gets consumed correctly, but
the second enqueued message cannot get consumed.

When dequeueing the second message, the call to this consumer's
receiveNoWait() method returns null, although there _is_ a message in the
queue.

This fact is backed up by the JMX queue browser and the fact that you can
create a browser in the same session which allows you to peek at the
message.

The logic then sets the rollBackOnly flag and returns.

This then just cycles, because the controlling thread can still see the same
message on the queue and hence dispatches another worker thread.

I've tried this with the JMS and the XA transaction managers, to no avail.

The JMS provider is activemq 4.0.1, I haven't tried another provider yet.

I'm not too confident that my transaction handling is correct, as I don't
feel that I quite mastered the mule transaction management yet.

What I find strange is the fact that it works for the first message, but
then for no subsequent message.

Does anybody have ideas or things that I can try?
--
View this message in context: http://www.nabble.com/JMS-Queue-Browsing-Receiver-tf2166297.html#a5988823
Sent from the Mule - User forum at Nabble.com.


---------------------------------------------------------------------
To unsubscribe from this list please visit:

     http://xircles.codehaus.org/manage_email

[0x6e6562]

Andrew,

That's a very good point.

Here's a tiny m2 based project to reproduce the error:

my-app.zip

The main class can be run from eclipse, its called EnqueueingClient.

This starts up activemq and mule.

The eclipse project can be generated by the m2 eclipse plugin.

It initially sends one message to the queue.

You can also send messages via the JMX console for ease of use (its bound to the namespace foo).

Thanks for your help,

Ben

Andrew Perepelytsya wrote:

I know there were issues with queue browsers in ActiveMQ before. Not that it
wasn't fixed by v4 (I just don't know now). Before that it resulted in
messages getting reordered randomly.

The problem will need some investigation. Could you also list the activemq
config file. Generally, the easier you make to reproduce the scenario (test
client, etc.) the higher the chances are you'll get feedback.

[0x6e6562]

Andrew,

I'll have a look at the latest version of activemq and maybe also a different provider.

Sorry it took so long for me to reply, I had to knock up a standalone project to upload.

Thx,

Ben

Andrew Perepelytsya wrote:

I know there were issues with queue browsers in ActiveMQ before. Not that it
wasn't fixed by v4 (I just don't know now). Before that it resulted in
messages getting reordered randomly.

[0x6e6562]

What I forgot to mention is that I also wanted to browse the queue in a non-transactional session, thereby avoiding the blocking receive done currently by the TransactedPollingJmsReceiver. This tries to consume from the queue in a transaction and then rolls back when the transaction times out, which is what I wanted to avoid.

0x6e6562 wrote:

I'm trying to create a JMS receiver that browses a queue in a controlling thread using the QueueBrowser interface and then when it finds something, for each message, it calls a worker thread to consume _a_ message with a non blocking call.

The idea is that the browsing thread can peek at the queue in a non-transactional session and if there is anything to consume, this can be farmed out to separate threads, thereby minimising the time any worker thread would idle when there is potentially work arrived in the queue but not yet farmed out.

[Meeraj]

Ben,

I have tested your BrowsingMessageReceiver with Weblogic JMS provider and seems to be working as expected (dequeueing and processing all the messages instead of stalling after the first). However, with ActiveMQ it appears to be not returning any messages after the first dequeue. I have had a quick look through your receiver. Few minor comments,

1. I don't think you need to bind the browser to the thread local as it is accessed by only the monitor thread.
2. Also, you may not need to start a transaction for browsing.
3. Since you are creating a session and consumer for every dequeue, you may need to make sure they are closed after the transaction commit. You could do this by having them as instance variables in the Work interface implementation and closing them in the release method.

HTH
Meeraj

Andrew,

That's a very good point.

Here's a tiny m2 based project to reproduce the error:

my-app.zip

The main class can be run from eclipse, its called EnqueueingClient.

This starts up activemq and mule.

The eclipse project can be generated by the m2 eclipse plugin.

It initially sends one message to the queue.

You can also send messages via the JMX console for ease of use (its bound to the namespace foo).

Thanks for your help,

Ben

Andrew Perepelytsya wrote:

I know there were issues with queue browsers in ActiveMQ before. Not that it
wasn't fixed by v4 (I just don't know now). Before that it resulted in
messages getting reordered randomly.

The problem will need some investigation. Could you also list the activemq
config file. Generally, the easier you make to reproduce the scenario (test
client, etc.) the higher the chances are you'll get feedback.

[0x6e6562]

Meeraj,

Thanks for trying that one out.

You were right about using a different provider, I tested the BrowsingMessageReceiver with JORAM and it is working fine. So I'll have to rethink whether to use ActiveMQ or not, but it would be nice to know why it's not working with ActiveMQ. I suppose you would also have to evaluate whether ActiveMQ works with XA transactions as well.

I take your point about the thread local stuff and disposing of the objects that are being created. I'll fix this up, get some tests done and then submit the whole thing to JIRA.

I've had another thought about the sleep/poll routine. Essentially this is what the transport has to do when listening on whatever interface it has. So instead of polling (browsing) the message queue for readiness selection, you might be able to exploit a proprietry messaging provider's API for a non-transactional, non-destructive event notification that could spawn off a worker to consume a message in transaction, in a non-blocking fashion. This would replace the poll/sleep routine with a wait/notify mechanism. I guess this would mean developing a separate transport for this. But isn't that what Mule is all about?

What are your thoughts?

Ben


Ben,

I have tested your BrowsingMessageReceiver with Weblogic JMS provider and seems to be working as expected (dequeueing and processing all the messages instead of stalling after the first). However, with ActiveMQ it appears to be not returning any messages after the first dequeue. I have had a quick look through your receiver. Few minor comments,

1. I don't think you need to bind the browser to the thread local as it is accessed by only the monitor thread.
2. Also, you may not need to start a transaction for browsing.
3. Since you are creating a session and consumer for every dequeue, you may need to make sure they are closed after the transaction commit. You could do this by having them as instance variables in the Work interface implementation and closing them in the release method.

HTH
Meeraj

0x6e6562 wrote:

Andrew,

That's a very good point.

Here's a tiny m2 based project to reproduce the error:

my-app.zip

The main class can be run from eclipse, its called EnqueueingClient.

This starts up activemq and mule.

The eclipse project can be generated by the m2 eclipse plugin.

It initially sends one message to the queue.

You can also send messages via the JMX console for ease of use (its bound to the namespace foo).

Thanks for your help,

Ben

Andrew Perepelytsya wrote:

I know there were issues with queue browsers in ActiveMQ before. Not that it
wasn't fixed by v4 (I just don't know now). Before that it resulted in
messages getting reordered randomly.

The problem will need some investigation. Could you also list the activemq
config file. Generally, the easier you make to reproduce the scenario (test
client, etc.) the higher the chances are you'll get feedback.

[Andrew Perepelytsya]

Ben,

This idea is fine and in fact already exploited in Mule. There is an Oracle Advanced Queueing Provider (OAQ), so having one more with a proprietary settings is not a big deal. Granted, however, that binary dependencies' licensing issues are resolved for that provider.

Andrew

might be able to exploit a proprietry messaging provider's API for a
non-transactional, non-destructive event notification that could spawn off a
worker to consume a message in transaction, in a non-blocking fashion. This
would replace the poll/sleep routine with a wait/notify mechanism. I guess
this would mean developing a separate transport for this. But isn't that
what Mule is all about?

What are your thoughts?

Ben


Meeraj wrote:

>
> Ben,
>
> I have tested your BrowsingMessageReceiver with Weblogic JMS provider and
> seems to be working as expected (dequeueing and processing all the
> messages instead of stalling after the first). However, with ActiveMQ it
> appears to be not returning any messages after the first dequeue. I have
> had a quick look through your receiver. Few minor comments,
>
> 1. I don't think you need to bind the browser to the thread local as it is
> accessed by only the monitor thread.
> 2. Also, you may not need to start a transaction for browsing.
> 3. Since you are creating a session and consumer for every dequeue, you
> may need to make sure they are closed after the transaction commit. You
> could do this by having them as instance variables in the Work interface
> implementation and closing them in the release method.
>
> HTH
> Meeraj
>
> 0x6e6562 wrote:
>>
>> Andrew,
>>
>> That's a very good point.
>>
>> Here's a tiny m2 based project to reproduce the error:
>>
>>  http://www.nabble.com/user-files/235770/my-app.zip my-app.zip

>>

>> The main class can be run from eclipse, its called EnqueueingClient.
>>
>> This starts up activemq and mule.
>>
>> The eclipse project can be generated by the m2 eclipse plugin.
>>
>> It initially sends one message to the queue.
>>
>> You can also send messages via the JMX console for ease of use (its bound
>> to the namespace foo).
>>
>> Thanks for your help,
>>
>> Ben
>>
>>
>> Andrew Perepelytsya wrote:
>>>
>>> I know there were issues with queue browsers in ActiveMQ before. Not
>>> that it
>>> wasn't fixed by v4 (I just don't know now). Before that it resulted in
>>> messages getting reordered randomly.
>>>
>>> The problem will need some investigation. Could you also list the
>>> activemq
>>> config file. Generally, the easier you make to reproduce the scenario
>>> (test
>>> client, etc.) the higher the chances are you'll get feedback.
>>>
>>
>>
>
>

--
View this message in context: http://www.nabble.com/JMS-Queue-Browsing-Receiver-tf2166297.html#a6024291
Sent from the Mule - User forum at Nabble.com.


---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[0x6e6562]

I have tried to isolate ActiveMQ's behaviour with a pure JMS client but following the same threading strategy, to no avail, so I have created a test case within a m2 project and have posted it to ActiveMQ's JIRA:

https://issues.apache.org/activemq/browse/AMQ-900

Andrew Perepelytsya wrote:

I know there were issues with queue browsers in ActiveMQ before. Not that it
wasn't fixed by v4 (I just don't know now). Before that it resulted in
messages getting reordered randomly.

The problem will need some investigation. Could you also list the activemq
config file. Generally, the easier you make to reproduce the scenario (test
client, etc.) the higher the chances are you'll get feedback.

Andrew

[Andrew Perepelytsya]

Thanks for picking up the ball, Ben.

Could you please keep the list posted with any subsequent updates from the ActiveMQ side?

Andrew

On 8/29/06, 0x6e6562 <ben.hood@...> wrote:

I have tried to isolate ActiveMQ's behaviour with a pure JMS client but
following the same threading strategy, to no avail, so I have created a test
case within a m2 project and have posted it to ActiveMQ's JIRA:

https://issues.apache.org/activemq/browse/AMQ-900


Andrew Perepelytsya wrote:

>
> I know there were issues with queue browsers in ActiveMQ before. Not that
> it
> wasn't fixed by v4 (I just don't know now). Before that it resulted in
> messages getting reordered randomly.
>
> The problem will need some investigation. Could you also list the activemq
> config file. Generally, the easier you make to reproduce the scenario
> (test
> client, etc.) the higher the chances are you'll get feedback.
>
> Andrew
>

--
View this message in context: http://www.nabble.com/JMS-Queue-Browsing-Receiver-tf2166297.html#a6041786
Sent from the Mule - User forum at Nabble.com.


---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[0x6e6562]

Sure.

In the meantime I'm putting togther a test case for the queue browser using Joram, but I've got to figure out how I'm going to embed it first (rather than have to run it in a separate process).

Ben

Andrew Perepelytsya wrote:

Thanks for picking up the ball, Ben.

Could you please keep the list posted with any subsequent updates from the
ActiveMQ side?

Andrew

On 8/29/06, 0x6e6562 <[e-mail]> wrote:
>
>
> I have tried to isolate ActiveMQ's behaviour with a pure JMS client but
> following the same threading strategy, to no avail, so I have created a
> test
> case within a m2 project and have posted it to ActiveMQ's JIRA:
>
> https://issues.apache.org/activemq/browse/AMQ-900
>
>
> Andrew Perepelytsya wrote:
> >
> > I know there were issues with queue browsers in ActiveMQ before. Not
> that
> > it
> > wasn't fixed by v4 (I just don't know now). Before that it resulted in
> > messages getting reordered randomly.
> >
> > The problem will need some investigation. Could you also list the
> activemq
> > config file. Generally, the easier you make to reproduce the scenario
> > (test
> > client, etc.) the higher the chances are you'll get feedback.
> >
> > Andrew
> >
>
> --
> View this message in context:
> http://www.nabble.com/JMS-Queue-Browsing-Receiver-tf2166297.html#a6041786
> Sent from the Mule - User forum at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list please visit:
>
>     http://xircles.codehaus.org/manage_email
>
>

[0x6e6562]

I've submitted a version with test cases as a patch:

http://jira.symphonysoft.com/browse/MULE-1004

This uses Joram as an embedded JMS provider.

I still haven't heard back from ActiveMQ guys yet whether they know anything about the previously encountered issue.

Ben


Ben,

I have tested your BrowsingMessageReceiver with Weblogic JMS provider and seems to be working as expected (dequeueing and processing all the messages instead of stalling after the first). However, with ActiveMQ it appears to be not returning any messages after the first dequeue. I have had a quick look through your receiver. Few minor comments,

1. I don't think you need to bind the browser to the thread local as it is accessed by only the monitor thread.
2. Also, you may not need to start a transaction for browsing.
3. Since you are creating a session and consumer for every dequeue, you may need to make sure they are closed after the transaction commit. You could do this by having them as instance variables in the Work interface implementation and closing them in the release method.

HTH
Meeraj

0x6e6562 wrote:

Andrew,

That's a very good point.

Here's a tiny m2 based project to reproduce the error:

my-app.zip

The main class can be run from eclipse, its called EnqueueingClient.

This starts up activemq and mule.

The eclipse project can be generated by the m2 eclipse plugin.

It initially sends one message to the queue.

You can also send messages via the JMX console for ease of use (its bound to the namespace foo).

Thanks for your help,

Ben

Andrew Perepelytsya wrote:

I know there were issues with queue browsers in ActiveMQ before. Not that it
wasn't fixed by v4 (I just don't know now). Before that it resulted in
messages getting reordered randomly.

The problem will need some investigation. Could you also list the activemq
config file. Generally, the easier you make to reproduce the scenario (test
client, etc.) the higher the chances are you'll get feedback.

[0x6e6562]

I've looked into a wait/notify mechanism for Joram and though it would be possible I don't think it would be practical because:

1) There is currently no exposed interface for this, so you would have to extend for example the QueueMBean in order to register as a waiter. Then you would have to change the implmentation to do the notification. Although this is actually quite simple (I tried it), it means an API change in Joram.

2) I don't think this would cluster (unless you had a clusterable VM like Terracotta), because the thread notification could only occur in one VM.

3) Because of this added complexity and not being able to quantify the performance gain over a sleep/poll mechanism, I've decided not to pursue this approach.

Andrew Perepelytsya wrote:

Ben,

This idea is fine and in fact already exploited in Mule. There is an Oracle
Advanced Queueing Provider (OAQ), so having one more with a proprietary
settings is not a big deal. Granted, however, that binary dependencies'
licensing issues are resolved for that provider.

Andrew

>
> might be able to exploit a proprietry messaging provider's API for a
> non-transactional, non-destructive event notification that could spawn off
> a
> worker to consume a message in transaction, in a non-blocking fashion.
> This
> would replace the poll/sleep routine with a wait/notify mechanism. I guess
> this would mean developing a separate transport for this. But isn't that
> what Mule is all about?
>
> What are your thoughts?
>
> Ben
>
>
> Meeraj wrote:
> >
> > Ben,
> >
> > I have tested your BrowsingMessageReceiver with Weblogic JMS provider
> and
> > seems to be working as expected (dequeueing and processing all the
> > messages instead of stalling after the first). However, with ActiveMQ it
> > appears to be not returning any messages after the first dequeue. I have
> > had a quick look through your receiver. Few minor comments,
> >
> > 1. I don't think you need to bind the browser to the thread local as it
> is
> > accessed by only the monitor thread.
> > 2. Also, you may not need to start a transaction for browsing.
> > 3. Since you are creating a session and consumer for every dequeue, you
> > may need to make sure they are closed after the transaction commit. You
> > could do this by having them as instance variables in the Work interface
> > implementation and closing them in the release method.
> >
> > HTH
> > Meeraj
> >
> > 0x6e6562 wrote:
> >>
> >> Andrew,
> >>
> >> That's a very good point.
> >>
> >> Here's a tiny m2 based project to reproduce the error:
> >>
> >>  http://www.nabble.com/user-files/235770/my-app.zip my-app.zip
> >>
> >> The main class can be run from eclipse, its called EnqueueingClient.
> >>
> >> This starts up activemq and mule.
> >>
> >> The eclipse project can be generated by the m2 eclipse plugin.
> >>
> >> It initially sends one message to the queue.
> >>
> >> You can also send messages via the JMX console for ease of use (its
> bound
> >> to the namespace foo).
> >>
> >> Thanks for your help,
> >>
> >> Ben
> >>
> >>
> >> Andrew Perepelytsya wrote:
> >>>
> >>> I know there were issues with queue browsers in ActiveMQ before. Not
> >>> that it
> >>> wasn't fixed by v4 (I just don't know now). Before that it resulted in
> >>> messages getting reordered randomly.
> >>>
> >>> The problem will need some investigation. Could you also list the
> >>> activemq
> >>> config file. Generally, the easier you make to reproduce the scenario
> >>> (test
> >>> client, etc.) the higher the chances are you'll get feedback.
> >>>
> >>
> >>
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/JMS-Queue-Browsing-Receiver-tf2166297.html#a6024291
> Sent from the Mule - User forum at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list please visit:
>
>     http://xircles.codehaus.org/manage_email
>
>

[Saravanan0]

Hi Ben,

     I am new in mule.i would like to know how to use jms provider instead of using vm provider.i tried to send a text message from mule. but the same message should be received in mule . how to do that. Can you help me please.if you send simple sample code ,it will be more helpful for me.

Thanks in advance.
Please reply at [e-mail].



Andrew,

That's a very good point.

Here's a tiny m2 based project to reproduce the error:

my-app.zip

The main class can be run from eclipse, its called EnqueueingClient.

This starts up activemq and mule.

The eclipse project can be generated by the m2 eclipse plugin.

It initially sends one message to the queue.

You can also send messages via the JMX console for ease of use (its bound to the namespace foo).

Thanks for your help,

Ben

Andrew Perepelytsya wrote:

I know there were issues with queue browsers in ActiveMQ before. Not that it
wasn't fixed by v4 (I just don't know now). Before that it resulted in
messages getting reordered randomly.

The problem will need some investigation. Could you also list the activemq
config file. Generally, the easier you make to reproduce the scenario (test
client, etc.) the higher the chances are you'll get feedback.