Stuck messages - Dispatch issues

[chrajanirao]

We are seeing issues with ActiveMQ 5.1 and 5.2 RC2 with message dispatching from queues. It is easily reproducible even using the out of the box activemq configuration.

Send messages to the a queue (QueueA) using multiple threads (10 or 20 thread in a loop of 10) using JMeter or your custom code. Have the consumer setup using spring DMLC (DefaultMessageListenerContainer) or with regular JMS API sync or async consumption with multiple consumers. Consumer part can be configured using Camel to consumer from QueueA and put the messages in QueueB within ActiveMQ configuration as well.

After receiving some messages (the number is different each time), the consumers stop receiving any messages even though there are some left on the queue. Basically, the broker don't dispatch the messages  and they are stuck until restart of the broker. Any new messages sent to the queue after this are sometimes dispatched and other times they are stuck too.

This certainly seems like a major bug in the dispatch mechanism. I have found below posts that state the exact problem.

http://www.nabble.com/Pending-Messages-are-shown-in-ActiveMQ-td20241332.html

http://www.nabble.com/Consumer-Listener-stop-receving-message-until-ActiveMQ-restart-td20355247.html

This is a very basic use case and I wonder how the version 5.1 is currently used in production, if anyone is using at all.

We tried with prefetch limit as 1, asyncDispatch as true and false, session transacted as true and false. In all cases, the dispatch problem still exists starting after 100 messages until 1000 messages.

I hope any of the active commiters looks into this issue seriously. Would really appreciate the help.

Thanks,
Rajani.

[Joe Fernandez]

What are the queue's "DequeueCount" and "InFlightCount" attributes registering on the jconsole?

Joe


Get a free ActiveMQ user guide @ http://www.ttmsolutions.com

chrajanirao wrote:

We are seeing issues with ActiveMQ 5.1 and 5.2 RC2 with message dispatching from queues. It is easily reproducible even using the out of the box activemq configuration.

Send messages to the a queue (QueueA) using multiple threads (10 or 20 thread in a loop of 10) using JMeter or your custom code. Have the consumer setup using spring DMLC (DefaultMessageListenerContainer) or with regular JMS API sync or async consumption with multiple consumers. Consumer part can be configured using Camel to consumer from QueueA and put the messages in QueueB within ActiveMQ configuration as well.

After receiving some messages (the number is different each time), the consumers stop receiving any messages even though there are some left on the queue. Basically, the broker don't dispatch the messages  and they are stuck until restart of the broker. Any new messages sent to the queue after this are sometimes dispatched and other times they are stuck too.

This certainly seems like a major bug in the dispatch mechanism. I have found below posts that state the exact problem.

http://www.nabble.com/Pending-Messages-are-shown-in-ActiveMQ-td20241332.html

http://www.nabble.com/Consumer-Listener-stop-receving-message-until-ActiveMQ-restart-td20355247.html

This is a very basic use case and I wonder how the version 5.1 is currently used in production, if anyone is using at all.

We tried with prefetch limit as 1, asyncDispatch as true and false, session transacted as true and false. In all cases, the dispatch problem still exists starting after 100 messages until 1000 messages.

I hope any of the active commiters looks into this issue seriously. Would really appreciate the help.

Thanks,
Rajani.

[Gary Tully]

Once such problem[1] was resolved for 5.2.0 RC3, would it be possible
to validate RC3[2]?

What would really help here is a JUnit test case that demonstrates the problem.

[1] https://issues.apache.org/activemq/browse/AMQ-1984
[2] http://people.apache.org/~gtully/staging-repos/activemq-5.2.0/org/apache/activemq/apache-activemq/5.2.0

2008/11/12 chrajanirao <rajanicm@...>:

>
> We are seeing issues with ActiveMQ 5.1 and 5.2 RC2 with message dispatching
> from queues. It is easily reproducible even using the out of the box
> activemq configuration.
>
> Send messages to the a queue (QueueA) using multiple threads (10 or 20
> thread in a loop of 10) using JMeter or your custom code. Have the consumer
> setup using spring DMLC (DefaultMessageListenerContainer) or with regular
> JMS API sync or async consumption with multiple consumers. Consumer part can
> be configured using Camel to consumer from QueueA and put the messages in
> QueueB within ActiveMQ configuration as well.
>
> After receiving some messages (the number is different each time), the
> consumers stop receiving any messages even though there are some left on the
> queue. Basically, the broker don't dispatch the messages  and they are stuck
> until restart of the broker. Any new messages sent to the queue after this
> are sometimes dispatched and other times they are stuck too.
>
> This certainly seems like a major bug in the dispatch mechanism. I have
> found below posts that state the exact problem.
>
> http://www.nabble.com/Pending-Messages-are-shown-in-ActiveMQ-td20241332.html
>
> http://www.nabble.com/Consumer-Listener-stop-receving-message-until-ActiveMQ-restart-td20355247.html
>
> This is a very basic use case and I wonder how the version 5.1 is currently
> used in production, if anyone is using at all.
>
> We tried with prefetch limit as 1, asyncDispatch as true and false, session
> transacted as true and false. In all cases, the dispatch problem still
> exists starting after 100 messages until 1000 messages.
>
> I hope any of the active commiters looks into this issue seriously. Would
> really appreciate the help.
>
> Thanks,
> Rajani.
> --
> View this message in context: http://www.nabble.com/Stuck-messages---Dispatch-issues-tp20467949p20467949.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>

[chrajanirao]

I tried 5.2 RC3 and it seem to have new bugs. It dispatching duplicates. One of my test run resulted below:

Messages sent: 1000 (using 50 threads 20 times)
Consumers: 2 trasacted

Queue Attributes in JConsole:
DequeCount: 1000
DipatchCount: 4849
EnqueueCount:1000
InFlightCount: 3849
QueueSize: 49

Consumer end:
Received: 2025

Its acting weird. when I tried with Non-transacted+Auto Ack, it sent duplicates and also the broker was still holding on to all of 1000 messages.

I will try to come up with a JUnit test case. But this is faily staright forward case when messages are sent faster than they can be consumed!!

Thanks.

[Gary Tully]

A test case would be great because it would be good to get this issue
fully understood. This is a potential blocker for the 5.2.0 release
IMHO. Can you raise a jira issue to track this?
thanks.

2008/11/14 chrajanirao <rajanicm@...>:

>
> I tried 5.2 RC3 and it seem to have new bugs. It dispatching duplicates. One
> of my test run resulted below:
>
> Messages sent: 1000 (using 50 threads 20 times)
> Consumers: 2 trasacted
>
> Queue Attributes in JConsole:
> DequeCount: 1000
> DipatchCount: 4849
> EnqueueCount:1000
> InFlightCount: 3849
> QueueSize: 49
>
> Consumer end:
> Received: 2025
>
> Its acting weird. when I tried with Non-transacted+Auto Ack, it sent
> duplicates and also the broker was still holding on to all of 1000 messages.
>
> I will try to come up with a JUnit test case. But this is faily staright
> forward case when messages are sent faster than they can be consumed!!
>
> Thanks.
>
> --
> View this message in context: http://www.nabble.com/Stuck-messages---Dispatch-issues-tp20467949p20492251.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>

[chrajanirao]

Here is the issue that I created. Although the test case may not be of much help as I could not get it to fail with embedded broker, hope you would look into the issue.

https://issues.apache.org/activemq/browse/AMQ-2009

Thanks,
Rajani.

Gary Tully wrote:

A test case would be great because it would be good to get this issue
fully understood. This is a potential blocker for the 5.2.0 release
IMHO. Can you raise a jira issue to track this?
thanks.

2008/11/14 chrajanirao <[e-mail]>:
>
> I tried 5.2 RC3 and it seem to have new bugs. It dispatching duplicates. One
> of my test run resulted below:
>
> Messages sent: 1000 (using 50 threads 20 times)
> Consumers: 2 trasacted
>
> Queue Attributes in JConsole:
> DequeCount: 1000
> DipatchCount: 4849
> EnqueueCount:1000
> InFlightCount: 3849
> QueueSize: 49
>
> Consumer end:
> Received: 2025
>
> Its acting weird. when I tried with Non-transacted+Auto Ack, it sent
> duplicates and also the broker was still holding on to all of 1000 messages.
>
> I will try to come up with a JUnit test case. But this is faily staright
> forward case when messages are sent faster than they can be consumed!!
>
> Thanks.
>
> --
> View this message in context: http://www.nabble.com/Stuck-messages---Dispatch-issues-tp20467949p20492251.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>

[couzteau]

Has this been solved yet? I'm seeing it with 5.2.0.

I'm seeing an issue that occurs on some machines (where machines are identical regarding OS, Java version and hardware).

Consumers that are on the same machine as the producer usually work fine.

It's a major blocker for us - any comments highly appreciated.


TIA  Jacques

chrajanirao wrote:

We are seeing issues with ActiveMQ 5.1 and 5.2 RC2 with message dispatching from queues. It is easily reproducible even using the out of the box activemq configuration.

Send messages to the a queue (QueueA) using multiple threads (10 or 20 thread in a loop of 10) using JMeter or your custom code. Have the consumer setup using spring DMLC (DefaultMessageListenerContainer) or with regular JMS API sync or async consumption with multiple consumers. Consumer part can be configured using Camel to consumer from QueueA and put the messages in QueueB within ActiveMQ configuration as well.

After receiving some messages (the number is different each time), the consumers stop receiving any messages even though there are some left on the queue. Basically, the broker don't dispatch the messages  and they are stuck until restart of the broker. Any new messages sent to the queue after this are sometimes dispatched and other times they are stuck too.

This certainly seems like a major bug in the dispatch mechanism. I have found below posts that state the exact problem.

http://www.nabble.com/Pending-Messages-are-shown-in-ActiveMQ-td20241332.html

http://www.nabble.com/Consumer-Listener-stop-receving-message-until-ActiveMQ-restart-td20355247.html

This is a very basic use case and I wonder how the version 5.1 is currently used in production, if anyone is using at all.

We tried with prefetch limit as 1, asyncDispatch as true and false, session transacted as true and false. In all cases, the dispatch problem still exists starting after 100 messages until 1000 messages.

I hope any of the active commiters looks into this issue seriously. Would really appreciate the help.

Thanks,
Rajani.

[Gary Tully]

in the absence of a test case, the best approach is to try out a
5.3-SNAPSHOT

2009/7/30 couzteau <couzteau@...>

>
> Has this been solved yet? I'm seeing it with 5.2.0.
>
> I'm seeing an issue that occurs on some machines (where machines are
> identical regarding OS, Java version and hardware).
>
> Consumers that are on the same machine as the producer usually work fine.
>
> It's a major blocker for us - any comments highly appreciated.
>
>
> TIA  Jacques
>
>
>
>
> chrajanirao wrote:
> >
> > We are seeing issues with ActiveMQ 5.1 and 5.2 RC2 with message
> > dispatching from queues. It is easily reproducible even using the out of
> > the box activemq configuration.
> >
> > Send messages to the a queue (QueueA) using multiple threads (10 or 20
> > thread in a loop of 10) using JMeter or your custom code. Have the
> > consumer setup using spring DMLC (DefaultMessageListenerContainer) or
> with
> > regular JMS API sync or async consumption with multiple consumers.
> > Consumer part can be configured using Camel to consumer from QueueA and
> > put the messages in QueueB within ActiveMQ configuration as well.
> >
> > After receiving some messages (the number is different each time), the
> > consumers stop receiving any messages even though there are some left on
> > the queue. Basically, the broker don't dispatch the messages  and they
> are
> > stuck until restart of the broker. Any new messages sent to the queue
> > after this are sometimes dispatched and other times they are stuck too.
> >
> > This certainly seems like a major bug in the dispatch mechanism. I have
> > found below posts that state the exact problem.
> >
> >
> http://www.nabble.com/Pending-Messages-are-shown-in-ActiveMQ-td20241332.html
> >
> >
> http://www.nabble.com/Consumer-Listener-stop-receving-message-until-ActiveMQ-restart-td20355247.html
> >
> > This is a very basic use case and I wonder how the version 5.1 is
> > currently used in production, if anyone is using at all.
> >
> > We tried with prefetch limit as 1, asyncDispatch as true and false,
> > session transacted as true and false. In all cases, the dispatch problem
> > still exists starting after 100 messages until 1000 messages.
> >
> > I hope any of the active commiters looks into this issue seriously. Would
> > really appreciate the help.
> >
> > Thanks,
> > Rajani.
> >
>
> --
> View this message in context:
> http://www.nabble.com/Stuck-messages---Dispatch-issues-tp20467949p24733104.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>

--
http://blog.garytully.com

Open Source Integration
http://fusesource.com

[MatsH]

Hi Jacques,

I've just recently logged the following issue:

https://issues.apache.org/activemq/browse/AMQ-2332

It sounds exactly the same as the original email in this thread, but it
is however different from the other threads the original poster linked
to.

Basically it's possible to deadlock queues, and they will then not
deliver or accept any more messages until the broker is restarted.

Regards,

Mats

> -----Original Message-----
> From: couzteau [mailto:couzteau@...]
> Sent: Thursday, July 30, 2009 2:57 AM
> To: users@...
> Subject: Re: Stuck messages - Dispatch issues
>
>
> Has this been solved yet? I'm seeing it with 5.2.0.
>
> I'm seeing an issue that occurs on some machines (where machines are
> identical regarding OS, Java version and hardware).
>
> Consumers that are on the same machine as the producer usually work

fine.

>
> It's a major blocker for us - any comments highly appreciated.
>
>
> TIA  Jacques
>
>
>
>
> chrajanirao wrote:
> >
> > We are seeing issues with ActiveMQ 5.1 and 5.2 RC2 with message
> > dispatching from queues. It is easily reproducible even using the

out of
> > the box activemq configuration.
> >
> > Send messages to the a queue (QueueA) using multiple threads (10 or
20
> > thread in a loop of 10) using JMeter or your custom code. Have the
> > consumer setup using spring DMLC (DefaultMessageListenerContainer)
or
> with
> > regular JMS API sync or async consumption with multiple consumers.
> > Consumer part can be configured using Camel to consumer from QueueA
and
> > put the messages in QueueB within ActiveMQ configuration as well.
> >
> > After receiving some messages (the number is different each time),
the
> > consumers stop receiving any messages even though there are some
left on
> > the queue. Basically, the broker don't dispatch the messages  and
they
> are
> > stuck until restart of the broker. Any new messages sent to the
queue
> > after this are sometimes dispatched and other times they are stuck
too.
> >
> > This certainly seems like a major bug in the dispatch mechanism. I
have

> > found below posts that state the exact problem.
> >
> > http://www.nabble.com/Pending-Messages-are-shown-in-ActiveMQ-
> td20241332.html
> >
> > http://www.nabble.com/Consumer-Listener-stop-receving-message-until-
> ActiveMQ-restart-td20355247.html
> >
> > This is a very basic use case and I wonder how the version 5.1 is
> > currently used in production, if anyone is using at all.
> >
> > We tried with prefetch limit as 1, asyncDispatch as true and false,
> > session transacted as true and false. In all cases, the dispatch

problem

> > still exists starting after 100 messages until 1000 messages.
> >
> > I hope any of the active commiters looks into this issue seriously.
> Would
> > really appreciate the help.
> >
> > Thanks,
> > Rajani.
> >
>
> --
> View this message in context: http://www.nabble.com/Stuck-messages---
> Dispatch-issues-tp20467949p24733104.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

[couzteau]

Thanks for the info!

I integrated 5.3-snapshot but ran into compile issues with the latest jar - So I abandoned that.

Gary Tully wrote:

in the absence of a test case, the best approach is to try out a
5.3-SNAPSHOT

2009/7/30 couzteau <[e-mail]>

>
> Has this been solved yet? I'm seeing it with 5.2.0.
>
> I'm seeing an issue that occurs on some machines (where machines are
> identical regarding OS, Java version and hardware).
>
> Consumers that are on the same machine as the producer usually work fine.
>
> It's a major blocker for us - any comments highly appreciated.
>
>
> TIA  Jacques
>
>
>
>
> chrajanirao wrote:
> >
> > We are seeing issues with ActiveMQ 5.1 and 5.2 RC2 with message
> > dispatching from queues. It is easily reproducible even using the out of
> > the box activemq configuration.
> >
> > Send messages to the a queue (QueueA) using multiple threads (10 or 20
> > thread in a loop of 10) using JMeter or your custom code. Have the
> > consumer setup using spring DMLC (DefaultMessageListenerContainer) or
> with
> > regular JMS API sync or async consumption with multiple consumers.
> > Consumer part can be configured using Camel to consumer from QueueA and
> > put the messages in QueueB within ActiveMQ configuration as well.
> >
> > After receiving some messages (the number is different each time), the
> > consumers stop receiving any messages even though there are some left on
> > the queue. Basically, the broker don't dispatch the messages  and they
> are
> > stuck until restart of the broker. Any new messages sent to the queue
> > after this are sometimes dispatched and other times they are stuck too.
> >
> > This certainly seems like a major bug in the dispatch mechanism. I have
> > found below posts that state the exact problem.
> >
> >
> http://www.nabble.com/Pending-Messages-are-shown-in-ActiveMQ-td20241332.html
> >
> >
> http://www.nabble.com/Consumer-Listener-stop-receving-message-until-ActiveMQ-restart-td20355247.html
> >
> > This is a very basic use case and I wonder how the version 5.1 is
> > currently used in production, if anyone is using at all.
> >
> > We tried with prefetch limit as 1, asyncDispatch as true and false,
> > session transacted as true and false. In all cases, the dispatch problem
> > still exists starting after 100 messages until 1000 messages.
> >
> > I hope any of the active commiters looks into this issue seriously. Would
> > really appreciate the help.
> >
> > Thanks,
> > Rajani.
> >
>
> --
> View this message in context:
> http://www.nabble.com/Stuck-messages---Dispatch-issues-tp20467949p24733104.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>


--
http://blog.garytully.com

Open Source Integration
http://fusesource.com

[Gary Tully]

can you share the details of the compilation issues so we can help diagnose?

2009/7/30 couzteau <couzteau@...>

>
> Thanks for the info!
>
> I integrated 5.3-snapshot but ran into compile issues with the latest jar -
> So I abandoned that.
>
>
>
> Gary Tully wrote:
> >
> > in the absence of a test case, the best approach is to try out a
> > 5.3-SNAPSHOT
> >
> > 2009/7/30 couzteau <couzteau@...>
> >
> >>
> >> Has this been solved yet? I'm seeing it with 5.2.0.
> >>
> >> I'm seeing an issue that occurs on some machines (where machines are
> >> identical regarding OS, Java version and hardware).
> >>
> >> Consumers that are on the same machine as the producer usually work
> fine.
> >>
> >> It's a major blocker for us - any comments highly appreciated.
> >>
> >>
> >> TIA  Jacques
> >>
> >>
> >>
> >>
> >> chrajanirao wrote:
> >> >
> >> > We are seeing issues with ActiveMQ 5.1 and 5.2 RC2 with message
> >> > dispatching from queues. It is easily reproducible even using the out
> >> of
> >> > the box activemq configuration.
> >> >
> >> > Send messages to the a queue (QueueA) using multiple threads (10 or 20
> >> > thread in a loop of 10) using JMeter or your custom code. Have the
> >> > consumer setup using spring DMLC (DefaultMessageListenerContainer) or
> >> with
> >> > regular JMS API sync or async consumption with multiple consumers.
> >> > Consumer part can be configured using Camel to consumer from QueueA
> and
> >> > put the messages in QueueB within ActiveMQ configuration as well.
> >> >
> >> > After receiving some messages (the number is different each time), the
> >> > consumers stop receiving any messages even though there are some left
> >> on
> >> > the queue. Basically, the broker don't dispatch the messages  and they
> >> are
> >> > stuck until restart of the broker. Any new messages sent to the queue
> >> > after this are sometimes dispatched and other times they are stuck
> too.
> >> >
> >> > This certainly seems like a major bug in the dispatch mechanism. I
> have
> >> > found below posts that state the exact problem.
> >> >
> >> >
> >>
> http://www.nabble.com/Pending-Messages-are-shown-in-ActiveMQ-td20241332.html
> >> >
> >> >
> >>
> http://www.nabble.com/Consumer-Listener-stop-receving-message-until-ActiveMQ-restart-td20355247.html
> >> >
> >> > This is a very basic use case and I wonder how the version 5.1 is
> >> > currently used in production, if anyone is using at all.
> >> >
> >> > We tried with prefetch limit as 1, asyncDispatch as true and false,
> >> > session transacted as true and false. In all cases, the dispatch
> >> problem
> >> > still exists starting after 100 messages until 1000 messages.
> >> >
> >> > I hope any of the active commiters looks into this issue seriously.
> >> Would
> >> > really appreciate the help.
> >> >
> >> > Thanks,
> >> > Rajani.
> >> >
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/Stuck-messages---Dispatch-issues-tp20467949p24733104.html
> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
> > --
> > http://blog.garytully.com
> >
> > Open Source Integration
> > http://fusesource.com
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Stuck-messages---Dispatch-issues-tp20467949p24744031.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>

--
http://blog.garytully.com

Open Source Integration
http://fusesource.com