Some (not all) one-way messages locked and never consumed

[jchang]

I have one-way messages that never get consumed.  In the explorer, I see a lock next to them and they never drop off.  I have sent them with a quality of service time-to-live of 10 seconds, but they never disappear.  However, it is just some such messages that fail to be consumed.  I can have one fail, a minute later the same kind of message on the same queue get through fine, and the next minute one gets locked and never consumed.  I see other similar threads, but the discussions involve checking the reply queue, and I don't think that applies to me.  Thanks in advance.

[IIT Software]

A message is locked when

a) it has been delivered into a client-side consumer cache but has not been consumed yet (not delivered by receive() or onMessage);

b) it is part of an XA transaction which has been started but is not finished yet.

Mostly it's a). That is you have another consumer on that queue which has this message in its cache. Do you use Spring?

[jchang]

Thanks for the response.  I do see from other threads what you write below, that the message is locked when it is in the consumer's cache but is not yet consumed.  However, I don't know where to go with this information.  I'm afraid I don't see how this point answers the question of why it is never consumed.  Can you help with that?

Yes, I am using Spring.  Thanks.

[jchang]

Oh, and I should add in case it is useful, there is one consumer only on the queue.  Thanks.

[IIT Software]

jchang wrote:

Oh, and I should add in case it is useful, there is one consumer only on the queue.  Thanks.

Good to know.

Because you use Spring, I assume it's related to that. Spring exposes some weird behavior if you use DefaultMessageListenerContainer with the default values in conjunction with a transaction manager. I recommend to use our springsupport library to get over that.

Please let me know if that works.

[jchang]

We are not using a transaction manager for this.

[IIT Software]

Then post your spring config, please...

[jchang]

Here is the top-level file.  At the bottom, you see it imports another file (ApplicationImplementation.spring.xml).  I'll post that one in a few minutes, too.ApplicationDeployment.spring.xml

[jchang]

And here is the ApplicationImplementation.spring.xml that is imported at the bottom of the last file I sent.  ApplicationImplementation.spring.xml

[jchang]

Note in the first file I posted (ApplicationDeployment.spring.xml) we are using com.swiftmq.jms.springsupport.SingleSharedConnectionFactory.  Thanks.

[IIT Software]

As far as I see you have concurrent consumers:

<property name="concurrentConsumers" value="30"/>
<property name="maxConcurrentConsumers" value="50"/>

Try to set this property to "-1":

<property name="receiveTimeout" value="30000"/>

[jchang]

Woops.  Please use this version of ApplicationImlementation.spring.xml, not the first I posted.  It is almost identical.  ApplicationImplementation.spring.xml

[jchang]

You mean set the receiveTimeout to -1, not the concurrentConsumers and maxConcurrentConsumers?  Or those, too?

[jchang]

Actually, I see in the service that is giving us trouble, this value (receiveTimeout) is set to 300000.  At any rate, you still want me to set it to -1?  And what about the concurrentConsumers and maxConcurrentConsumers?

[jchang]

Oh, and in case it was not clear, this is config from the message consumer side, not the message producer side.  And please do clarify specifically which params to set to -1; I was not clear on that from your instructions above.  Thanks.

[IIT Software]

I would change the receiveTimeout to -1. Now it's set to 30 secs. If that does not work, set the concurrent consumer to 1 to ensure that you have a single consumer.

So again.

First try:

<property name="concurrentConsumers" value="30"/>
<property name="maxConcurrentConsumers" value="50"/>

<property name="receiveTimeout" value="-1"/>

Second try:

<property name="concurrentConsumers" value="1"/>
<property name="maxConcurrentConsumers" value="1"/>

<property name="receiveTimeout" value="-1"/>

[jchang]

OK, thanks.  I'll give that a try.  BTW, the messages that are getting hung up are being sent the endpoint configured as '#ApplicationImplementation' in the ApplicationDeployment.spring.xml which I posted.  So, I can see how changing config params on that as you advised (concurrentConsumer and maxConcurrentConsumer) might make a difference.  

Again, the messges that never get consumed are one-way messages, so they just get dumped on the queue ApplicationImplementation and consumed; there is no call back of any sort.  As such, I can't see how the other change you advised (receiveTimeout on the agentCallbackServiceFactory bean) would make any difference.  This is a call back bean that sends messages out onto the ApplicationReply queue.  It is related to different methods.  This reply queue is for calls back that have nothing to do with the messages giving me trouble.  As my problem is with one-way messages, there is no reply queue involved.  Why would changing timeouts here make a difference?  (That said, I'll give it a try...was just hoping to understand things better.)  

Thanks.

[IIT Software]

The receiveTimeout and concurrentConsumer stuff can lead to these locked messages with Spring. That's why I recommended it. But you are using Apache CXF on top so I don't know how this effects behavior. But if that doesn't work you should ask someone from the Apache CXF side.

[jchang]

Before changing these params, which necessitate a restart (so far as I know how to do), I decided to see what impact simply restarting the message consumer would have.  That way, if the problem went away (if only temporarily), I would know if the solution (if only temporary) was due to the restart or the config change.  The restart w/o config change did, at least temporarily, "solve" the problem.  The backed-up locked messages were consumed right away and nothing new has blocked up.  So, I'll watch it to see if it degrades again.  

My question at this point: given what you suspected to be the problem, is the above behavior expected?

[IIT Software]

It indicates that this client itself did lock the messages, hence it seems to be a concurrency issue with CXF & Spring. Proceed as recommended and let me know if that solves the problem.