guaranteed message processing question...

[boday ()]

I'm using Camel2 in SMX 3.3.1.  I have a simple Camel route as follows...

from("activemq:inboundMessages?maxConcurrentConsumers=50")
.process(new Processor1())
.process(new Processor2())
.to("activemq:finishedQueue");

How do I guarantee that a message will not be removed from the "inboundMessages" queue until it reaches the "finishedQueue"?  I need to make sure that messages are not lost in the event of a server crash/restart, etc.  Currently, this process is losing messages if I restart the server midway through a load test.

Do I have to use a transactional client approach (if so, how exactly)?  I tried this briefly but haven't been able to get it to work.  Is there another option like using the JMS acknowledgement mode or something?

thanks

[Christian Schneider]

As far as I know the transactional client aproach is the only good way
to guarantee that no message is lost.
You can take a look at these pages:
http://camel.apache.org/jms.html   section: Enabling Transacted Consumption
http://camel.apache.org/transactional-client.html section: Camel 2.0 -
JMS Sample

This thread could also help:
http://old.nabble.com/Problem-with-SOAP-JMS-and-transactions-to26162087.html

As soon as you start using transactions you also will have to care about
rollbacks and redeliveries like described in the thread.

Greetings

Christian


boday schrieb:

> I'm using Camel2 in SMX 3.3.1.  I have a simple Camel route as follows...
>
> from("activemq:inboundMessages?maxConcurrentConsumers=50")
> .process(new Processor1())
> .process(new Processor2())
> .to("activemq:finishedQueue");
>
> How do I guarentee that a message will not be removed from the
> "inboundMessages" queue until it reaches the "finishedQueue"?  I need to
> make sure that messages are not lost in the event of a server crash/restart,
> etc.  Currently, this process is loosing messages if I restart the server
> midway through a load test.
>
> Do I have to use a transactional client approach (if so, how exactly)?  I
> tried this briefly but haven't been able to get it to work.  Is there
> another option like using the JMS acknowledgement mode or something?
>
> thanks
>
> -----
> Ben - Senior Consultant
>
>  

[Claus Ibsen-2]

Yeah as Christian says you need to use transactions for that.


On Wed, Nov 4, 2009 at 8:23 PM, Christian Schneider
<chris@...> wrote:

> As far as I know the transactional client aproach is the only good way to
> guarantee that no message is lost.
> You can take a look at these pages:
> http://camel.apache.org/jms.html   section: Enabling Transacted Consumption
> http://camel.apache.org/transactional-client.html section: Camel 2.0 - JMS
> Sample
>
> This thread could also help:
> http://old.nabble.com/Problem-with-SOAP-JMS-and-transactions-to26162087.html
>
> As soon as you start using transactions you also will have to care about
> rollbacks and redeliveries like described in the thread.
>
> Greetings
>
> Christian
>
>
> boday schrieb:
>>
>> I'm using Camel2 in SMX 3.3.1.  I have a simple Camel route as follows...
>>
>> from("activemq:inboundMessages?maxConcurrentConsumers=50")
>> .process(new Processor1())
>> .process(new Processor2())
>> .to("activemq:finishedQueue");
>>
>> How do I guarentee that a message will not be removed from the
>> "inboundMessages" queue until it reaches the "finishedQueue"?  I need to
>> make sure that messages are not lost in the event of a server
>> crash/restart,
>> etc.  Currently, this process is loosing messages if I restart the server
>> midway through a load test.
>>
>> Do I have to use a transactional client approach (if so, how exactly)?  I
>> tried this briefly but haven't been able to get it to work.  Is there
>> another option like using the JMS acknowledgement mode or something?
>>
>> thanks
>>
>> -----
>> Ben - Senior Consultant
>>
>>
>
>

--
Claus Ibsen
Apache Camel Committer

Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus

[boday]

thanks guys.  I added some parameters to the queue to make the route "transactional"...

from("activemq:inboundMessages?maxConcurrentConsumers=50&lazyCreateTransactionManager=true&transacted=true")
.process(new Processor1())
.process(new Processor2())
.to("activemq:finishedQueue");

Now I'm getting duplicates delivered.  My test pumped 10k messages into the queue via an external client.  During the test, I restarted SMX 2 times.  After the test concluded, I had 10,003 messages in the finishedQueue.  This is better than losing messages, but still not the desired behavior...any ideas?

One more question, what happens during exception handling?  Suppose I have the following policy to retry 3 times, then go to an error queue...

onException(Exception.class).handled(true).maximumRedeliveries(3).to("activemq:errorQueue");

I assume the message is still in the inbound queue until either delivered to the finished queue or error queue, correct?  So at any given time the sum of the inbound, finished and error queues should equal to the number of messages sent, correct?

I'm working on a unit test for this now, but wanted to make sure I understood the intended behavior first...

thanks in advance

[Claus Ibsen-2]

Hi

Do you use persistent queues? Then make sure the queues are empty when
testing. Maybe deleting the AMQ data folder before testing.


On Thu, Nov 5, 2009 at 12:23 AM, boday <boday@...> wrote:

>
> thanks guys.  I added some parameters to the queue to make the route
> "transactional"...
>
> from("activemq:inboundMessages?maxConcurrentConsumers=50&lazyCreateTransactionManager=true&transacted=true")
> .process(new Processor1())
> .process(new Processor2())
> .to("activemq:finishedQueue");
>
> Now I'm getting duplicates delivered.  My test pumped 10k messages into the
> queue via an external client.  During the test, I restarted SMX 2 times.
> After the test concluded, I had 10,003 messages in the finishedQueue.  This
> is better than losing messages, but still not the desired behavior...any
> ideas?
>
> One more question, what happens during exception handling?  Suppose I have
> the following policy to retry 3 times, then go to an error queue...
>
> onException(Exception.class).handled(true).maximumRedeliveries(3).to("activemq:errorQueue");
>
> I assume the message is still in the inbound queue until either delivered to
> the finished queue or error queue, correct?  So at any given time the sum of
> the inbound, finished and error queues should equal to the number of
> messages sent, correct?
>
> I'm working on a unit test for this now, but wanted to make sure I
> understood the intended behavior first...
>
> thanks in advance
>
>
> -----
> Ben - Senior Consultant
>
> --
> View this message in context: http://old.nabble.com/guaranteed-message-processing-question...-tp26202573p26206508.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>

--
Claus Ibsen
Apache Camel Committer

Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus

[boday ()]

Claus, I am using persistent queues, but I purge them prior to each test.

I've rerun this test multiple times and it does deliver the correct number if no exceptions are thrown.  Its only when the exception handling kicks in that I end up with duplicates being delivered.

Any other thoughts on this?  What happens exactly when an exception is thrown in a transacted route?

Can someone verify if this scenario is correct?

flow w/o exception...

-message sent to InboundQ (external process)
-message received by Processor1 and marked as sent/waiting for ACK in InboundQ
-message received by Processor2
-message sent to finishedQueue
-ACK sent back to InboundMessage queue
-message is removed from InboundQ

flow w/exception...

-message sent to InboundQ (external process)
-message received by Processor1 and marked as sent/waiting for ACK in InboundQ
-message received by Processor2, then throws an Exception
-onException clause kicks in and retries Processor2 (up to 3 times)
   -if success, message sent to finishedQueue, ACK sent back to InboundQ
   -if 3 failures reached, message sent to errorQueue, ACK send back to InboundQ
-message is removed from InboundQ

Also, I've read that this can also be accomplished using CLIENT_ACKNOWLEDGE mode instead of transactions.  I'm yet to find a Camel routing example of doing this.  Can someone point me to a unit test that does this?  How do I explicitly ACK back at the end of my route to control this??

thanks in advance...

Claus Ibsen-2 wrote:

Hi

Do you use persistent queues? Then make sure the queues are empty when
testing. Maybe deleting the AMQ data folder before testing.

[Claus Ibsen-2]

Hi

Could you ask / search at the AMQ forum. The questions is now more
related to messaging in general how to do reliable bridges.
I remember there is also a QoS term (Quality of Service) which specified such as
- duplicates allowed
- at most one
- etc.

Try to see if that can help, looks as if duplicates allowed is the
kind of mode you run with.


On Fri, Nov 6, 2009 at 8:27 PM, boday <boday@...> wrote:

>
> Claus, I am using persistent queues, but I purge them prior to each test.
>
> I've rerun this test multiple times and it does deliver the correct number
> if no exceptions are thrown.  Its only when the exception handling kicks in
> that I end up with duplicates being delivered.
>
> Any other thoughts on this?  What happens exactly when an exception is
> thrown in a transacted route?
>
> Can someone verify if this scenario is correct?
>
> flow w/o exception...
>
> -message sent by InboundQ (external process)
> -message received by Processor1 and marked as sent/waiting for ACK in
> InboundQ
> -message received by Processor2
> -message sent to finishedQueue
> -ACK sent back to InboundMessage queue
> -message is removed from InboundQ
>
> w/exception
>
> -message sent by InboundQ (external process)
> -message received by Processor1 and marked as sent/waiting for ACK in
> InboundQ
> -message received by Processor2 (throws Exception)
> -onException clause kicks in and retries Processor2 3 times
>   -if success, message sent to finishedQueue, ACK sent back to InboundQ
>   -if 3 failures reached, message sent to errorQueue, ACK send back to
> InboundQ
> -message is removed from InboundQ
>
> thanks in advance...
>
>
>
>
>
> Claus Ibsen-2 wrote:
>>
>> Hi
>>
>> Do you use persistent queues? Then make sure the queues are empty when
>> testing. Maybe deleting the AMQ data folder before testing.
>>
>
>
> -----
> Ben - Senior Consultant
>
> --
> View this message in context: http://old.nabble.com/guaranteed-message-processing-question...-tp26202573p26230870.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>

--
Claus Ibsen
Apache Camel Committer

Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus

[boday]

Claus, my questions are really specific to how Camel wraps the underlying AMQ technologies.  In particular, how Camel handles transactions and client acknowledgements when exceptions are raised and redelivery policies applied (using Camel's onException clause, etc).

Also, I'm not running in DUPS_OK mode explicitly (using AUTO_ACK mode).  But this seems to be the behavior that I'm seeing...

On another note, I did purchase the "Camel In Action" pre-release (which I see you are an author of) but haven't found much on this specific topic yet.  Will subsequent chapters will cover this?

thanks again...

Claus Ibsen-2 wrote:

Hi

Could you ask / search at the AMQ forum. The questions is now more
related to messaging in general how to do reliable bridges.
I remember there is also a QoS term (Quality of Service) which specified such as
- duplicates allowed
- at most one
- etc.

Try to see if that can help, looks as if duplicates allowed is the
kind of mode you run with.

[Claus Ibsen-2]

On Mon, Nov 9, 2009 at 6:14 PM, boday <boday@...> wrote:
>
> Claus, my questions are really specific to how Camel wraps the underlying AMQ
> technologies.  In particular, how Camel handles transactions and client
> acknowledgements when exceptions are raised and redelivery policies applied
> (using Camel's onException clause, etc).
>

camel-jms uses Spring DMLC as the message listener.
And the spring transaction manager for TX.



> Also, I'm not running in DUPS_OK mode explicitly (using AUTO_ACK mode).  But
> this seems to be the behavior that I'm seeing...
>
> On another note, I did purchase the "Camel In Action" pre-release (which I
> see you are an author of) but haven't found much on this specific topic yet.
> Will subsequent chapters will cover this?
>

Yeah chapter 9 is about transactions.

> thanks again...
>
>
> Claus Ibsen-2 wrote:
>>
>> Hi
>>
>> Could you ask / search at the AMQ forum. The questions is now more
>> related to messaging in general how to do reliable bridges.
>> I remember there is also a QoS term (Quality of Service) which specified
>> such as
>> - duplicates allowed
>> - at most one
>> - etc.
>>
>> Try to see if that can help, looks as if duplicates allowed is the
>> kind of mode you run with.
>>
>>
>
>
> -----
> Ben - Senior Consultant
>
> --
> View this message in context: http://old.nabble.com/guaranteed-message-processing-question...-tp26202573p26270070.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>

--
Claus Ibsen
Apache Camel Committer

Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus

[boday]

I've been performance testing my newly transacted route, but have seen a significant performance drop.  

I assumed there would be a bit of overhead with using transactions, but my route goes from ~250 message/second to ~50 messages/second when I add the "?transacted=true" to it (no other changes).

Has anyone else seen this type of drop in throughput?  I'm hoping it has something to do with the Tx manager configuration or something.  I'm using the default tx.xml file that shipped with Servicemix 3.3.1...

thanks...