How to maintain an historical view of queue contents?

[stirlingc]

Hello,

I have an architecture in which one or more message producers place messages in a shared queue and a consumer pool processes the messages.   While I can use JMX to monitor the number of messages produced and consumed, the messages are consumed so fast that I often can't browse them with JMX.

To monitor the system and debug issues, I would like to keep a JMX-viewable history of the most recently produced messages.  My thoughts were to use a composite destination so that any message placed in the shared queue are replicated to another queue without any consumers.

The problem is that I would like to cap the size of this historical queue, preferably by count (e.g., only store the last 200 messages that were sent).  Alternatively, I could use message time-to-live (TTL) to age the messages within the queue.  The problem with TTL is that it must be set by the message producer and would therefore impact both the shared and historical queues.  Ideally, I'd like to configure the composite destination to set the TTL only on the messages replicated to the historical queue.  Unfortunately, it seems like the only way to do this is to create a consumer for the "primary" historical queue which simply sets the TTL and forwards the messages to a "secondary" historical queue.

So here are my questions:

1) Is it possible bound the queue size so that if a new message and the queue size is exceeded, the oldest message in the queue is removed/expired?  E.g., make a rollover queue that shows just the last 200 messages sent.  I imagine this can be done with application code and a queue browser, but I'd prefer to achieve it with configuration.

2) If a fixed size is not possible, can the time-to-live on a message be set as a property of the queue's configuration rather than the message, or as a property applied during resolution of a composite destination?

3) Is there a better way to achieve my goal of creating a JMX-viewable bounded history of a queue's contents?

Thanks for taking the time to read this,
Stirling

[Joe Fernandez]

Sounds like you're wanting to implement a message-level audit trail? With Camel, you could very quickly  implement a wiretap message pattern (http://camel.apache.org/wire-tap.html) and have the wiretap route the tapped messages to a log4j appender. The appender's logfile could rollover based on size and/or time. You could then write a simple MBean that is loaded via a plugin and used to query the appender's log file. With this approach, you'd have much more control.

You may also want to check out this Camel-based monitoring tool for ActiveMQ.

http://www.ttmsolutions.com/Transactional_Software_Solutions/Active_Monitor_AMon.php

Joe
http://www.ttmsolutions.com

stirlingc wrote:

Hello,

I have an architecture in which one or more message producers place messages in a shared queue and a consumer pool processes the messages.   While I can use JMX to monitor the number of messages produced and consumed, the messages are consumed so fast that I often can't browse them with JMX.

To monitor the system and debug issues, I would like to keep a JMX-viewable history of the most recently produced messages.  My thoughts were to use a composite destination so that any message placed in the shared queue are replicated to another queue without any consumers.

The problem is that I would like to cap the size of this historical queue, preferably by count (e.g., only store the last 200 messages that were sent).  Alternatively, I could use message time-to-live (TTL) to age the messages within the queue.  The problem with TTL is that it must be set by the message producer and would therefore impact both the shared and historical queues.  Ideally, I'd like to configure the composite destination to set the TTL only on the messages replicated to the historical queue.  Unfortunately, it seems like the only way to do this is to create a consumer for the "primary" historical queue which simply sets the TTL and forwards the messages to a "secondary" historical queue.

So here are my questions:

1) Is it possible bound the queue size so that if a new message and the queue size is exceeded, the oldest message in the queue is removed/expired?  E.g., make a rollover queue that shows just the last 200 messages sent.  I imagine this can be done with application code and a queue browser, but I'd prefer to achieve it with configuration.

2) If a fixed size is not possible, can the time-to-live on a message be set as a property of the queue's configuration rather than the message, or as a property applied during resolution of a composite destination?

3) Is there a better way to achieve my goal of creating a JMX-viewable bounded history of a queue's contents?

Thanks for taking the time to read this,
Stirling

[stirlingc]

Joe Fernandez wrote:

Sounds like you're wanting to implement a message-level audit trail? With Camel, you could very quickly  implement a wiretap message pattern (http://camel.apache.org/wire-tap.html) and have the wiretap route the tapped messages to a log4j appender. The appender's logfile could rollover based on size and/or time. You could then write a simple MBean that is loaded via a plugin and used to query the appender's log file. With this approach, you'd have much more control.

Thanks for the idea, that sounds like a possible solution.  I was hoping, however, to do this with the stock AMQ MBeans and queue configuration rather than introducing new code into our system, however, it looks like I might have to do this.  The existing MBean views are sufficient for our purposes, but the problem is that the messages are so short-lived in the queue that they are rarely browseable.

I found the following open ticket for implemented fixed-size rollover queues, so this answers my first question:

http://issues.apache.org/activemq/browse/AMQ-1648

[Joe Fernandez]

Re AMQ-1648; it would be ideal to avoid as much as possible having to evict/lose messages, because a queue or some other system resource has reached its max capacity. Instead, you'd like to proactively monitor system resources and invoke or trigger some action (e.g., whenever a resource reaches a predefined threshold) that can alleviate the situation and thus avert losing messages. Check out AMon. It gives you that facility to  do just that.

Joe
http://www.ttmsolutions.com

stirlingc wrote:

Joe Fernandez wrote:

Sounds like you're wanting to implement a message-level audit trail? With Camel, you could very quickly  implement a wiretap message pattern (http://camel.apache.org/wire-tap.html) and have the wiretap route the tapped messages to a log4j appender. The appender's logfile could rollover based on size and/or time. You could then write a simple MBean that is loaded via a plugin and used to query the appender's log file. With this approach, you'd have much more control.

Thanks for the idea, that sounds like a possible solution.  I was hoping, however, to do this with the stock AMQ MBeans and queue configuration rather than introducing new code into our system, however, it looks like I might have to do this.  The existing MBean views are sufficient for our purposes, but the problem is that the messages are so short-lived in the queue that they are rarely browseable.

I found the following open ticket for implemented fixed-size rollover queues, so this answers my first question:

http://issues.apache.org/activemq/browse/AMQ-1648