JMS: Howto reconnect on connection lost

[dialsc]

hello all,

i've been playing around with retryPolicies in order to get reconnected to the jms once the queue has crashed and restarted. up to now i was able to write a custom etryPolicy which does the job but i'm very unhappy with this sollution.

if i just implement the RetryPolicyTemplate as found in the documentation, each call to a service that puts a message to the queue blocks untill the queue gets available. that's very bad. i was also playing around with asynchronous retryPolicies but found out that every request to the service starts a new retryPolicy job. the poblem here is, beside it's very bad to have multiple retries running, that only the first one stops once the connection has been reestablished because afterwards an exception is thrown telling that there is allready a client connected to the queue using the specific clientId. for sure this is true, isn't it.

so here i am and i've got no idea how to solve my problem. i just want to be able to tell the jms-connector(s) to reconnect to the queue once it is up again. am i missing something? why doesn't mule can handle this? i'm sure this is a big problem for every one creating a mule service based on a messaging infrustructure, isn't it?

has anyone an idea how i could solve this problem?

greez,

dialsc

[David Dossot-3]

Have you checked: http://www.mulesoft.org/display/COMMONRETRYPOLICIES ?

D.

On Wed, Nov 4, 2009 at 3:24 PM, dialsc <dirk.a.schaefer@...> wrote:

hello all,

i've been playing around with retryPolicies in order to get reconnected to
the jms once the queue has crashed and restarted. up to now i was able to
write a custom etryPolicy which does the job but i'm very unhappy with this
sollution.

if i just implement the RetryPolicyTemplate as found in the documentation,
each call to a service that puts a message to the queue blocks untill the
queue gets available. that's very bad. i was also playing around with
asynchronous retryPolicies but found out that every request to the service
starts a new retryPolicy job. the poblem here is, beside it's very bad to
have multiple retries running, that only the first one stops once the
connection has been reestablished because afterwards an exception is thrown
telling that there is allready a client connected to the queue using the
specific clientId. for sure this is true, isn't it.

so here i am and i've got no idea how to solve my problem. i just want to be
able to tell the jms-connector(s) to reconnect to the queue once it is up
again. am i missing something? why doesn't mule can handle this? i'm sure
this is a big problem for every one creating a mule service based on a
messaging infrustructure, isn't it?

has anyone an idea how i could solve this problem?

greez,

dialsc
--
View this message in context: http://old.nabble.com/JMS%3A-Howto-reconnect-on-connection-lost-tp26206517p26206517.html
Sent from the Mule - User mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

   http://xircles.codehaus.org/manage_email

[dialsc]

hi, no i haven't checked this side but its only describing retryPolicies and as i wrote, this is not what i need. therefore this doesn't help me. i need to get the mule jms-connector to reconnet to the queue when the connection has been lost. i do not need to retry to put messages into the queue.

greez,

dialsc

David Dossot-3 wrote:

Have you checked: http://www.mulesoft.org/display/COMMONRETRYPOLICIES ?

D.

On Wed, Nov 4, 2009 at 3:24 PM, dialsc <[e-mail]> wrote:

>
> hello all,
>
> i've been playing around with retryPolicies in order to get reconnected to
> the jms once the queue has crashed and restarted. up to now i was able to
> write a custom etryPolicy which does the job but i'm very unhappy with this
> sollution.
>
> if i just implement the RetryPolicyTemplate as found in the documentation,
> each call to a service that puts a message to the queue blocks untill the
> queue gets available. that's very bad. i was also playing around with
> asynchronous retryPolicies but found out that every request to the service
> starts a new retryPolicy job. the poblem here is, beside it's very bad to
> have multiple retries running, that only the first one stops once the
> connection has been reestablished because afterwards an exception is thrown
> telling that there is allready a client connected to the queue using the
> specific clientId. for sure this is true, isn't it.
>
> so here i am and i've got no idea how to solve my problem. i just want to
> be
> able to tell the jms-connector(s) to reconnect to the queue once it is up
> again. am i missing something? why doesn't mule can handle this? i'm sure
> this is a big problem for every one creating a mule service based on a
> messaging infrustructure, isn't it?
>
> has anyone an idea how i could solve this problem?
>
> greez,
>
> dialsc
> --
> View this message in context:
> http://old.nabble.com/JMS%3A-Howto-reconnect-on-connection-lost-tp26206517p26206517.html
> Sent from the Mule - User mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>
>

[Andrew Perepelytsya]

I think you misunderstood the concept. JMS and Mule are message-based systems, thus it's very different from e.g. a client-server tcp connection when you want to maintain this connection open all the time. From the client perspective, you don't really care if connection is alive at any point, as long as you can *connect and send/receive* a message when you need it. Thus, I recommend you check out the link David posted above.

HTH,
Andrew

[dialsc]

hi,

first of all i have to say that i do not claim the following to be true but this is how i understand the whole thing so please correct me if i'm wrong.

no i didn't missunderstand it. i know that it's message based. nevertheless the problem is still the same. two things about it.

1. if the queue turns off, mule doesn't reconnect to it at all until i start using a retryPolicy.
2. if the connection to the queue has been lost, messages from the queue to the mule instance can and will never be received by mule once the queue is up again.

in generell i've got the following possibilities with retryPolicies:

1. use one that tries to deliver a message for a configured amount of times.
2. use one that tries to deliver a message endless.

both of them do block any request (thread) until the queue gets available. but i do not want to have all request blocked until then. i need to have an exception thrown in such cases. if each request is being blocked, how log will it take until mule runs out of threads? furthermore i cannot accept to have clients waiting for responses until any timeout for nothing. i want them to receive an exception. i'm trying to integrate mule in an environment with heavy load and very high demands for reliability and performance. i cannot not go any further with such a behaviour or i do not see the way, at least.

for sure there is the possibility to use asynchronous retryPolicies but these bring another problem into the game. first of all its only available in the enterprise edition which is - as you certainly know - incredible expensive. second and here comes the real problem. for each request a new retryPolicy "worker" (thread) is being started. once the queue is available, the first one "wins" and the connector gets reconnected. but all the other retryPolicyWorks are still there and still try to do their job. but once the connection has been reestablished, there will always be an exception thrown that there is already a client using cientId XY connected to the queue and therefore all these retryPolicyWorkers keep working for ever as they keep receiving exceptions (JMSException).

what i need, and i'm absolutly sure lot of other people too, is a simple behaviour of the jmsConnector. throw an exception when the queue is not available and if it is not available, try to reconnect to it until its up again. this is such a simple and expected behaviour i cannot believe that it is not implemented this way. in my opinion how mule handles this is conceptually wrong. actually mule's retryPolicy concept for sure is a good thing if you whant exact such things but blocking requests and|or starting multiple threads where only the first one gets "stopped" after a reconnect is just wrong.

from my point of view we need to get the possibility to get it working the way i described it above. of course it could be integrated into mule thus this behaviour would be configurable but being able to tell mule - the jmsConnector - to work this way is something we realy need.

i think i will open a bug report at mule's jira.

greez,

dialsc

Andrew Perepelytsya wrote:

I think you misunderstood the concept. JMS and Mule are message-based
systems, thus it's very different from e.g. a client-server tcp connection
when you want to maintain this connection open all the time. From the client
perspective, you don't really care if connection is alive at any point, as
long as you can *connect and send/receive* a message when you need it. Thus,
I recommend you check out the link David posted above.

HTH,
Andrew

[Andrew Perepelytsya]

On Fri, Nov 6, 2009 at 6:20 AM, dialsc <dirk.a.schaefer@...> wrote:

what i need, and i'm absolutly sure lot of other people too, is a simple
behaviour of the jmsConnector. throw an exception when the queue is not
available and if it is not available, try to reconnect to it until its up
again. this is such a simple and expected behaviour i cannot believe that it
is not implemented this way. in my opinion how mule handles this is
conceptually wrong. actually mule's retryPolicy concept for sure is a good
thing if you whant exact such things but blocking requests and|or starting
multiple threads where only the first one gets "stopped" after a reconnect
is just wrong.

I'm sure it all looks 'simple' and we're 'so dumb not to see it', but believe me, there's a reason why things are the way they are. First, when you mentioned you wrote a custom exception strategy, were you sure it was written correctly, including concurrent access? Have you worked with common retry policies or tried Mule EE ones?

Now, designing reliable systems is a serious science, and sorry, but 'Mule simply taking the message and retrying to connect to queue' simply doesn't cut it. There are at least 2 points of failure in this 'simple' solution. Add transactions to the mix, and things get another dimension totally.

Well, my point is not to rant, but just communicating the message - things are much much more complex than they are on the surface. Mule's JMS transport is probably the most actively maintained and enhanced one of all, and it's been over 4 years like that (it was available before that of course, I'm only talking of my time). If there were an option to do simple things like that and still not breaking everything else, it would've been done.

Here's some food for thought - ask people on the httpclient list about the default retry, and see what kind of problems it causes and solves. And this is just http, whereas jms has much higher expectation as a major building block of reliable fault-tolerant systems.

So, let's be cooperative and break down posts into individual, reproducible problems to discuss/fix. Accompanied by a good measure of supporting information. Starting with a Mule version in use.

Andrew

[Gerwin Postma]

We are having the same problem with Tibco JMS server restarts. I agree with dialsc that the way Mule is handling restarts of JMS server is inadequate. At least the option has to be provided, where Mule just reconnects when the server comes back up and throws exceptions till then. Blocking threads is not even an option. I find it hard to believe that an ESB that is supposed to integrate multiple systems handles system restarts so awkwardly.

Mule Version 2.2.1 running on Websphere 6.1

Regards,
Sunil

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[rotten]

Seconded.   Catching broken endpoints and then having to restart our ESB is probably our main task/issue supporting this tool in production.

We've had the problem with Exchange server restarts (the mail transport blows up).

And LDAP restarts -- although we are testing the new ldap transport point release which does seem to be more robust.

For JMS, we followed the advice from this thread:
http://www.nabble.com/Jms-Transport-Not-able-to-send---receive-messages-td19602535.html

That works pretty well for auto-retrying.  It doesn't help with configuring alternate processing (you might be able to use an exception router), nor does it help if the other end is down when you try to start mule up.  (It won't start if you can't make the connections immediately.)

Sunil Kukde wrote:

We are having the same problem with Tibco JMS server restarts. I agree with dialsc that the way Mule is handling restarts of JMS server is inadequate. At least the option has to be provided, where Mule just reconnects when the server comes back up and throws exceptions till then. Blocking threads is not even an option. I find it hard to believe that an ESB that is supposed to integrate multiple systems handles system restarts so awkwardly.

Mule Version 2.2.1 running on Websphere 6.1

Regards,
Sunil

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Andrew Perepelytsya]

does it help if the other end is down when you try to start mule up.  (It
won't start if you can't make the connections immediately.)

Sounds like you never tried async retry strategies in Mule EE.

Andrew

[rotten]

If the other end is down, and you have an inbound or outbound JMS connector, Mule will not start.

If Mule is already running and the JMS connection goes down, the retry strategy will keep Mule alive and back-up the messages, for as long as you have it configured and the capacity to do so.

Andrew Perepelytsya wrote:

> does it help if the other end is down when you try to start mule up.  (It
> won't start if you can't make the connections immediately.)
>
>
Sounds like you never tried async retry strategies in Mule EE.

Andrew

[David Dossot-3]

The AdaptiveRetryPolicyTemplateWrapper from Common Retry Policies lets Mule start even if the JMS provider is down. After failing once synchronously on start-up, it schedules a threaded connection retry for as many attempts you want (including for ever).

D.

On Thu, Nov 12, 2009 at 10:28 AM, rotten <rickotten@...> wrote:

If the other end is down, and you have an inbound or outbound JMS connector,
Mule will not start.

If Mule is already running and the JMS connection goes down, the retry
strategy will keep Mule alive and back-up the messages, for as long as you
have it configured and the capacity to do so.

Andrew Perepelytsya wrote:
>
>> does it help if the other end is down when you try to start mule up.  (It
>> won't start if you can't make the connections immediately.)
>>
>>
> Sounds like you never tried async retry strategies in Mule EE.
>
> Andrew
>
>

--
View this message in context: http://old.nabble.com/JMS%3A-Howto-reconnect-on-connection-lost-tp26206517p26323847.html

Sent from the Mule - User mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

   http://xircles.codehaus.org/manage_email

[dialsc]

hi,

i've tried all the possibilities offered by mule so far. i've tried the community implementation of the retryPolicies. i've also tried my own implementation and last but not least i've also tried the ee async retryPolicy.

here are my conclusions.

1. non async - also known as sync ;) - retryPolicies block every request that comes in as long as the policy is not giving up because e.g. maxRetries has been reached. this is absolutly inacceptable. how long do you think will it take until mule runs out of threads in an busy environment? i will not discuss the fact that clients then need to wait until a timout occurs. that is simply a way we cannot accept in an enterprise environment.

2. the async (ee) retryPolicy starts a retryPolicy-thread for each request. once the queue is available again, only the first thread operating then will "win" and the connection is reestablished. ok, but what's about all the other threads? they keep running because each of them continues to receive an exception which is thrown by the jmsImplementation as the queue it self returns an exception upon each connection attempt because there is allready a client connected using the specific clientId. remember, that was the first thread connecting.

both ways are unusable in an enterprise environment!

so what can we do otherwise? we can go on not using retryPolicies at all. in this case we receive an exception because the queue has been lost. but what is then? then never again a reconnect will be tried. that's unusable, too.

for sure the idea of retryPolicies is a good thing. but the way they work is bad. blocking requests or starting multiple threads of which only the first one gets stopped and all the others continue producing errors until mule gets stopped is - at least for me - unusable.

last but not least here the very worst thing about it. if we do not use retryPolicies, we will never get reconnected to the queue. what about inbound endpoints in this case? we will never receive any message again until we restart mule!

as Andrew Perepelytsya mentioned, the jmsConnector is one of the most complicated connectors and they spend a lot of work in it and it's concepts. please understand that i do not want to blame you for this work, realy not. but the way it is is not enterprise ready, realy not!

here is my suggesstion on how we could improve the behaviour of the connector.

1. spend the connector a config flag which defines if it throws an exception while processing messages when the queue is offline or not.
2. do not start more than one retryPolicyThreads in the async way
3. here is my personal favourit. if the connection to the queue gets lost, start a single thread that tries to reconnect to the queue independed of whether we are using retryPolicies or not. we realy need the connector to reconnect to the queue by its own in any case.

this way we could keep the concept but we could improve it very much.

to point it out one more time. in an enterprise environment it is inacceptable to block all client request as long as the connection is lost. it is also inaccaptable to start multiple threads were only one gets stopped and last but not least it is inacceptable that the connector does not try to reconnect especially if we have an inbound endpoint listening for messages from the queue.

please understand that i do not want to whinge around here. i need to get this problem solved and i'm trying to make reasonable suggesstions on how we could do it base on my investigations and understandings of the problem. if my investigation results are wrong, please correct me. but i've undestood the messaging concept as well as i know what i'm talking about regarding the enterprise environment and these are my results so far.

greez,

dialsc

David Dossot-3 wrote:

The AdaptiveRetryPolicyTemplateWrapper from Common Retry Policies lets Mule
start even if the JMS provider is down. After failing once synchronously on
start-up, it schedules a threaded connection retry for as many attempts you
want (including for ever).

D.


On Thu, Nov 12, 2009 at 10:28 AM, rotten <[e-mail]> wrote:

>
> If the other end is down, and you have an inbound or outbound JMS
> connector,
> Mule will not start.
>
> If Mule is already running and the JMS connection goes down, the retry
> strategy will keep Mule alive and back-up the messages, for as long as you
> have it configured and the capacity to do so.
>
>
>
> Andrew Perepelytsya wrote:
> >
> >> does it help if the other end is down when you try to start mule up.
>  (It
> >> won't start if you can't make the connections immediately.)
> >>
> >>
> > Sounds like you never tried async retry strategies in Mule EE.
> >
> > Andrew
> >
> >
>
> --
> View this message in context:
> http://old.nabble.com/JMS%3A-Howto-reconnect-on-connection-lost-tp26206517p26323847.html
> Sent from the Mule - User mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>
>

[David Dossot-3]

I am truly puzzled. Like many others, I have successfully used Mule with JMS providers in enterprise environments. These JMS providers would go up and down but usually a combination of retry policies (when they were introduced), connector and service error handlers, error based conditional routing and ad-hoc retry strategies always did me well both for inbound and outbound scenarios.

My impression is that you focus a lot on the JMS transport while not enough considering all the other stuff you get from Mule around it.

Still my 2 cents.
D.

On Fri, Nov 13, 2009 at 3:22 AM, dialsc <dirk.a.schaefer@...> wrote:

hi,

i've tried all the possibilities offered by mule so far. i've tried the
community implementation of the retryPolicies. i've also tried my own
implementation and last but not least i've also tried the ee async
retryPolicy.

here are my conclusions.

1. non async - also known as sync ;) - retryPolicies block every request
that comes in as long as the policy is not giving up because e.g. maxRetries
has been reached. this is absolutly inacceptable. how long do you think will
it take until mule runs out of threads in an busy environment? i will not
discuss the fact that clients then need to wait until a timout occurs. that
is simply a way we cannot accept in an enterprise environment.

2. the async (ee) retryPolicy starts a retryPolicy-thread for each request.
once the queue is available again, only the first thread operating then will
"win" and the connection is reestablished. ok, but what's about all the
other threads? they keep running because each of them continues to receive
an exception which is thrown by the jmsImplementation as the queue it self
returns an exception upon each connection attempt because there is allready
a client connected using the specific clientId. remember, that was the first
thread connecting.

both ways are unusable in an enterprise environment!

so what can we do otherwise? we can go on not using retryPolicies at all. in
this case we receive an exception because the queue has been lost. but what
is then? then never again a reconnect will be tried. that's unusable, too.

for sure the idea of retryPolicies is a good thing. but the way they work is
bad. blocking requests or starting multiple threads of which only the first
one gets stopped and all the others continue producing errors until mule
gets stopped is - at least for me - unusable.

last but not least here the very worst thing about it. if we do not use
retryPolicies, we will never get reconnected to the queue. what about
inbound endpoints in this case? we will never receive any message again
until we restart mule!

as Andrew Perepelytsya mentioned, the jmsConnector is one of the most
complicated connectors and they spend a lot of work in it and it's concepts.
please understand that i do not want to blame you for this work, realy not.
but the way it is is not enterprise ready, realy not!

here is my suggesstion on how we could improve the behaviour of the
connector.

1. spend the connector a config flag which defines if it throws an exception
while processing messages when the queue is offline or not.
2. do not start more than one retryPolicyThreads in the async way
3. here is my personal favourit. if the connection to the queue gets lost,
start a single thread that tries to reconnect to the queue independed of
whether we are using retryPolicies or not. we realy need the connector to
reconnect to the queue by its own in any case.

this way we could keep the concept but we could improve it very much.

to point it out one more time. in an enterprise environment it is
inacceptable to block all client request as long as the connection is lost.
it is also inaccaptable to start multiple threads were only one gets stopped
and last but not least it is inacceptable that the connector does not try to
reconnect especially if we have an inbound endpoint listening for messages
from the queue.

please understand that i do not want to whinge around here. i need to get
this problem solved and i'm trying to make reasonable suggesstions on how we
could do it base on my investigations and understandings of the problem. if
my investigation results are wrong, please correct me. but i've undestood
the messaging concept as well as i know what i'm talking about regarding the
enterprise environment and these are my results so far.

greez,

dialsc



David Dossot-3 wrote:
>

> The AdaptiveRetryPolicyTemplateWrapper from Common Retry Policies lets
> Mule
> start even if the JMS provider is down. After failing once synchronously
> on
> start-up, it schedules a threaded connection retry for as many attempts
> you
> want (including for ever).
>
> D.
>
>
> On Thu, Nov 12, 2009 at 10:28 AM, rotten <rickotten@...> wrote:
>
>>
>> If the other end is down, and you have an inbound or outbound JMS
>> connector,
>> Mule will not start.
>>
>> If Mule is already running and the JMS connection goes down, the retry
>> strategy will keep Mule alive and back-up the messages, for as long as
>> you
>> have it configured and the capacity to do so.
>>
>>
>>
>> Andrew Perepelytsya wrote:
>> >
>> >> does it help if the other end is down when you try to start mule up.
>>  (It
>> >> won't start if you can't make the connections immediately.)
>> >>
>> >>
>> > Sounds like you never tried async retry strategies in Mule EE.
>> >
>> > Andrew
>> >
>> >
>>
>> --
>> View this message in context:
>> http://old.nabble.com/JMS%3A-Howto-reconnect-on-connection-lost-tp26206517p26323847.html
>> Sent from the Mule - User mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe from this list, please visit:
>>
>>    http://xircles.codehaus.org/manage_email
>>
>>
>>
>
>

--
View this message in context: http://old.nabble.com/JMS%3A-Howto-reconnect-on-connection-lost-tp26206517p26335106.html

Sent from the Mule - User mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

   http://xircles.codehaus.org/manage_email

[dialsc]

hi,

as i pointed out, i'm only telling the findings i've got so far. for sure it's possible that i'm missing something that's already there in order to get my problem solved but apparently i'm unable to figure out what and how to use it.

please accept my appologies but it doesn't help me just to say "we are able to do it".

let me say one more time that i do not want to rail against mule's implementation. i need to solve this problem and if i can with any of mule's methods|constructs that's fine because then i do not need to wait until any further release. but how can i solve it, that's the information i need.

can anybody please give me a hint on how to get the jmsConnector reconnected to a missing queue without the retryPolicy construct. that construct does not work for me as i cannot accept requests (message flows!) getting blocked and the problem with the async ones i already explained.

greez,

dialsc

David Dossot-3 wrote:

I am truly puzzled. Like many others, I have successfully used Mule with JMS
providers in enterprise environments. These JMS providers would go up and
down but usually a combination of retry policies (when they were
introduced), connector and service error handlers, error based conditional
routing and ad-hoc retry strategies always did me well both for inbound and
outbound scenarios.

My impression is that you focus a lot on the JMS transport while not enough
considering all the other stuff you get from Mule around it.

Still my 2 cents.
D.


On Fri, Nov 13, 2009 at 3:22 AM, dialsc <[e-mail]> wrote:

>
> hi,
>
> i've tried all the possibilities offered by mule so far. i've tried the
> community implementation of the retryPolicies. i've also tried my own
> implementation and last but not least i've also tried the ee async
> retryPolicy.
>
> here are my conclusions.
>
> 1. non async - also known as sync ;) - retryPolicies block every request
> that comes in as long as the policy is not giving up because e.g.
> maxRetries
> has been reached. this is absolutly inacceptable. how long do you think
> will
> it take until mule runs out of threads in an busy environment? i will not
> discuss the fact that clients then need to wait until a timout occurs. that
> is simply a way we cannot accept in an enterprise environment.
>
> 2. the async (ee) retryPolicy starts a retryPolicy-thread for each request.
> once the queue is available again, only the first thread operating then
> will
> "win" and the connection is reestablished. ok, but what's about all the
> other threads? they keep running because each of them continues to receive
> an exception which is thrown by the jmsImplementation as the queue it self
> returns an exception upon each connection attempt because there is allready
> a client connected using the specific clientId. remember, that was the
> first
> thread connecting.
>
> both ways are unusable in an enterprise environment!
>
> so what can we do otherwise? we can go on not using retryPolicies at all.
> in
> this case we receive an exception because the queue has been lost. but what
> is then? then never again a reconnect will be tried. that's unusable, too.
>
> for sure the idea of retryPolicies is a good thing. but the way they work
> is
> bad. blocking requests or starting multiple threads of which only the first
> one gets stopped and all the others continue producing errors until mule
> gets stopped is - at least for me - unusable.
>
> last but not least here the very worst thing about it. if we do not use
> retryPolicies, we will never get reconnected to the queue. what about
> inbound endpoints in this case? we will never receive any message again
> until we restart mule!
>
> as Andrew Perepelytsya mentioned, the jmsConnector is one of the most
> complicated connectors and they spend a lot of work in it and it's
> concepts.
> please understand that i do not want to blame you for this work, realy not.
> but the way it is is not enterprise ready, realy not!
>
> here is my suggesstion on how we could improve the behaviour of the
> connector.
>
> 1. spend the connector a config flag which defines if it throws an
> exception
> while processing messages when the queue is offline or not.
> 2. do not start more than one retryPolicyThreads in the async way
> 3. here is my personal favourit. if the connection to the queue gets lost,
> start a single thread that tries to reconnect to the queue independed of
> whether we are using retryPolicies or not. we realy need the connector to
> reconnect to the queue by its own in any case.
>
> this way we could keep the concept but we could improve it very much.
>
> to point it out one more time. in an enterprise environment it is
> inacceptable to block all client request as long as the connection is lost.
> it is also inaccaptable to start multiple threads were only one gets
> stopped
> and last but not least it is inacceptable that the connector does not try
> to
> reconnect especially if we have an inbound endpoint listening for messages
> from the queue.
>
> please understand that i do not want to whinge around here. i need to get
> this problem solved and i'm trying to make reasonable suggesstions on how
> we
> could do it base on my investigations and understandings of the problem. if
> my investigation results are wrong, please correct me. but i've undestood
> the messaging concept as well as i know what i'm talking about regarding
> the
> enterprise environment and these are my results so far.
>
> greez,
>
> dialsc
>
>
>
> David Dossot-3 wrote:
> >
> > The AdaptiveRetryPolicyTemplateWrapper from Common Retry Policies lets
> > Mule
> > start even if the JMS provider is down. After failing once synchronously
> > on
> > start-up, it schedules a threaded connection retry for as many attempts
> > you
> > want (including for ever).
> >
> > D.
> >
> >
> > On Thu, Nov 12, 2009 at 10:28 AM, rotten <[e-mail]> wrote:
> >
> >>
> >> If the other end is down, and you have an inbound or outbound JMS
> >> connector,
> >> Mule will not start.
> >>
> >> If Mule is already running and the JMS connection goes down, the retry
> >> strategy will keep Mule alive and back-up the messages, for as long as
> >> you
> >> have it configured and the capacity to do so.
> >>
> >>
> >>
> >> Andrew Perepelytsya wrote:
> >> >
> >> >> does it help if the other end is down when you try to start mule up.
> >>  (It
> >> >> won't start if you can't make the connections immediately.)
> >> >>
> >> >>
> >> > Sounds like you never tried async retry strategies in Mule EE.
> >> >
> >> > Andrew
> >> >
> >> >
> >>
> >> --
> >> View this message in context:
> >>
> http://old.nabble.com/JMS%3A-Howto-reconnect-on-connection-lost-tp26206517p26323847.html
> >> Sent from the Mule - User mailing list archive at Nabble.com.
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe from this list, please visit:
> >>
> >>    http://xircles.codehaus.org/manage_email
> >>
> >>
> >>
> >
> >
>
> --
> View this message in context:
> http://old.nabble.com/JMS%3A-Howto-reconnect-on-connection-lost-tp26206517p26335106.html
> Sent from the Mule - User mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>
>

[David Dossot-3]

To be frank I never had to deal with disappearing JMS destinations, just complete JMS providers that would occasionally go down (which seems less nasty to deal with, from what you are saying). So I realize your problem is different from the ones I've been facing.

I guess you still have a few options open:

• Try to solve the issue by using Mule's building blocks, like a custom transport exception handler or, if it makes sense for your project (which we don't know as you haven't shared any config), build an alternate JMS message fetching service that can be suspended with Quartz, Interceptors, Notification Handlers and the like.
• Try to solve the issue by extending the relevant classes of the JMS transport you think need correction so the "disappearing destination" problem gets fixed. If your fix is generic enough, you may then decide to contribute it back.
• Get consulting help from MuleSoft or Ricston, as it is the kind of particular problem that can benefit from knowledgeable resources contracted to invest time into reproducing and fixing it.
  

I'm tempted to add a fourth option: switch to a more sturdy JMS provider, thought the fact you mentioned you're in an "enterprise environment" probably means there is almost no room for changing such a piece of IT infrastructure.

HTH
D.

On Sat, Nov 14, 2009 at 12:06 AM, dialsc <dirk.a.schaefer@...> wrote:

hi,

as i pointed out, i'm only telling the findings i've got so far. for sure
it's possible that i'm missing something that's already there in order to
get my problem solved but apparently i'm unable to figure out what and how
to use it.

please accept my appologies but it doesn't help me just to say "we are able
to do it".

let me say one more time that i do not want to rail against mule's
implementation. i need to solve this problem and if i can with any of mule's
methods|constructs that's fine because then i do not need to wait until any
further release. but how can i solve it, that's the information i need.

can anybody please give me a hint on how to get the jmsConnector reconnected
to a missing queue without the retryPolicy construct. that construct does
not work for me as i cannot accept requests (message flows!) getting blocked
and the problem with the async ones i already explained.

greez,

dialsc


David Dossot-3 wrote:
>
> I am truly puzzled. Like many others, I have successfully used Mule with
> JMS
> providers in enterprise environments. These JMS providers would go up and
> down but usually a combination of retry policies (when they were
> introduced), connector and service error handlers, error based conditional
> routing and ad-hoc retry strategies always did me well both for inbound
> and
> outbound scenarios.
>
> My impression is that you focus a lot on the JMS transport while not
> enough
> considering all the other stuff you get from Mule around it.
>
> Still my 2 cents.
> D.
>
>
> On Fri, Nov 13, 2009 at 3:22 AM, dialsc <dirk.a.schaefer@...>
> wrote:
>
>>
>> hi,
>>
>> i've tried all the possibilities offered by mule so far. i've tried the
>> community implementation of the retryPolicies. i've also tried my own
>> implementation and last but not least i've also tried the ee async
>> retryPolicy.
>>
>> here are my conclusions.
>>
>> 1. non async - also known as sync ;) - retryPolicies block every request
>> that comes in as long as the policy is not giving up because e.g.
>> maxRetries
>> has been reached. this is absolutly inacceptable. how long do you think
>> will
>> it take until mule runs out of threads in an busy environment? i will not
>> discuss the fact that clients then need to wait until a timout occurs.
>> that
>> is simply a way we cannot accept in an enterprise environment.
>>
>> 2. the async (ee) retryPolicy starts a retryPolicy-thread for each
>> request.
>> once the queue is available again, only the first thread operating then
>> will
>> "win" and the connection is reestablished. ok, but what's about all the
>> other threads? they keep running because each of them continues to
>> receive
>> an exception which is thrown by the jmsImplementation as the queue it
>> self
>> returns an exception upon each connection attempt because there is
>> allready
>> a client connected using the specific clientId. remember, that was the
>> first
>> thread connecting.
>>
>> both ways are unusable in an enterprise environment!
>>
>> so what can we do otherwise? we can go on not using retryPolicies at all.
>> in
>> this case we receive an exception because the queue has been lost. but
>> what
>> is then? then never again a reconnect will be tried. that's unusable,
>> too.
>>
>> for sure the idea of retryPolicies is a good thing. but the way they work
>> is
>> bad. blocking requests or starting multiple threads of which only the
>> first
>> one gets stopped and all the others continue producing errors until mule
>> gets stopped is - at least for me - unusable.
>>
>> last but not least here the very worst thing about it. if we do not use
>> retryPolicies, we will never get reconnected to the queue. what about
>> inbound endpoints in this case? we will never receive any message again
>> until we restart mule!
>>
>> as Andrew Perepelytsya mentioned, the jmsConnector is one of the most
>> complicated connectors and they spend a lot of work in it and it's
>> concepts.
>> please understand that i do not want to blame you for this work, realy
>> not.
>> but the way it is is not enterprise ready, realy not!
>>
>> here is my suggesstion on how we could improve the behaviour of the
>> connector.
>>
>> 1. spend the connector a config flag which defines if it throws an
>> exception
>> while processing messages when the queue is offline or not.
>> 2. do not start more than one retryPolicyThreads in the async way
>> 3. here is my personal favourit. if the connection to the queue gets
>> lost,
>> start a single thread that tries to reconnect to the queue independed of
>> whether we are using retryPolicies or not. we realy need the connector to
>> reconnect to the queue by its own in any case.
>>
>> this way we could keep the concept but we could improve it very much.
>>
>> to point it out one more time. in an enterprise environment it is
>> inacceptable to block all client request as long as the connection is
>> lost.
>> it is also inaccaptable to start multiple threads were only one gets
>> stopped
>> and last but not least it is inacceptable that the connector does not try
>> to
>> reconnect especially if we have an inbound endpoint listening for
>> messages
>> from the queue.
>>
>> please understand that i do not want to whinge around here. i need to get
>> this problem solved and i'm trying to make reasonable suggesstions on how
>> we
>> could do it base on my investigations and understandings of the problem.
>> if
>> my investigation results are wrong, please correct me. but i've undestood
>> the messaging concept as well as i know what i'm talking about regarding
>> the
>> enterprise environment and these are my results so far.
>>
>> greez,
>>
>> dialsc
>>
>>
>>
>> David Dossot-3 wrote:
>> >
>> > The AdaptiveRetryPolicyTemplateWrapper from Common Retry Policies lets
>> > Mule
>> > start even if the JMS provider is down. After failing once
>> synchronously
>> > on
>> > start-up, it schedules a threaded connection retry for as many attempts
>> > you
>> > want (including for ever).
>> >
>> > D.
>> >
>> >
>> > On Thu, Nov 12, 2009 at 10:28 AM, rotten <rickotten@...> wrote:
>> >
>> >>
>> >> If the other end is down, and you have an inbound or outbound JMS
>> >> connector,
>> >> Mule will not start.
>> >>
>> >> If Mule is already running and the JMS connection goes down, the retry
>> >> strategy will keep Mule alive and back-up the messages, for as long as
>> >> you
>> >> have it configured and the capacity to do so.
>> >>
>> >>
>> >>
>> >> Andrew Perepelytsya wrote:
>> >> >
>> >> >> does it help if the other end is down when you try to start mule
>> up.
>> >>  (It
>> >> >> won't start if you can't make the connections immediately.)
>> >> >>
>> >> >>
>> >> > Sounds like you never tried async retry strategies in Mule EE.
>> >> >
>> >> > Andrew
>> >> >
>> >> >
>> >>
>> >> --
>> >> View this message in context:
>> >>
>> http://old.nabble.com/JMS%3A-Howto-reconnect-on-connection-lost-tp26206517p26323847.html
>> >> Sent from the Mule - User mailing list archive at Nabble.com.
>> >>
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe from this list, please visit:
>> >>
>> >>    http://xircles.codehaus.org/manage_email
>> >>
>> >>
>> >>
>> >
>> >
>>
>> --
>> View this message in context:
>> http://old.nabble.com/JMS%3A-Howto-reconnect-on-connection-lost-tp26206517p26335106.html
>> Sent from the Mule - User mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe from this list, please visit:
>>
>>    http://xircles.codehaus.org/manage_email
>>
>>
>>
>
>

--
View this message in context: http://old.nabble.com/JMS%3A-Howto-reconnect-on-connection-lost-tp26206517p26348022.html

Sent from the Mule - User mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

   http://xircles.codehaus.org/manage_email