Usage Manager for durable offline subscriber

[Nicky Sandhu]

On revision 581885 I turned on debug messages on Usage.java and the commented lines of System.out on Message.java and I see the memory usage increase beyond 100% to something like 16500%. This causes a problem for the next thing that calls Usage.waitForSpace(timeout) and causes an indefinite hang

The Usage.increaseUsage(value) does not check to see if it's full and so can keep allocating even beyond 100%. This seems to be happening a lot from Message.incrementReferenceCount() with no corresponding Message.decrementReferenceCount() happening.

In my case it happens when a publishing to a topic (~ 1000 msgs) with a offline durable subscribe with multiple hard disconnects from the subscriber.

[rajdavies]

Hi Nicky,

which version is that?


cheers,

Rob

http://rajdavies.blogspot.com/



On Oct 5, 2007, at 1:17 AM, Nicky Sandhu wrote:

>
> On revision 581885 I turned on debug messages on Usage.java and the  
> commented
> lines of System.out on Message.java and I see the memory usage  
> increase
> beyond 100% to something like 16500%. This causes a problem for the  
> next
> thing that calls Usage.waitForSpace(timeout) and causes an  
> indefinite hang
>
> The Usage.increaseUsage(value) does not check to see if it's full  
> and so can
> keep allocating even beyond 100%. This seems to be happening a lot  
> from
> Message.incrementReferenceCount() with no corresponding
> Message.decrementReferenceCount() happening.
>
> In my case it happens when a publishing to a topic (~ 1000 msgs)  
> with a
> offline durable subscribe with multiple hard disconnects from the
> subscriber.
> --
> View this message in context: http://www.nabble.com/Usage-Manager- 
> for-durable-offline-subscriber-tf4572289s2354.html#a13051280
> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>

[rajdavies]

oops - that's the latest- must drink coffee before reading email ;)
Any chance of a test case ?

cheers,

Rob

rajdavies wrote:

Hi Nicky,

which version is that?


cheers,

Rob

http://rajdavies.blogspot.com/



On Oct 5, 2007, at 1:17 AM, Nicky Sandhu wrote:

>
> On revision 581885 I turned on debug messages on Usage.java and the  
> commented
> lines of System.out on Message.java and I see the memory usage  
> increase
> beyond 100% to something like 16500%. This causes a problem for the  
> next
> thing that calls Usage.waitForSpace(timeout) and causes an  
> indefinite hang
>
> The Usage.increaseUsage(value) does not check to see if it's full  
> and so can
> keep allocating even beyond 100%. This seems to be happening a lot  
> from
> Message.incrementReferenceCount() with no corresponding
> Message.decrementReferenceCount() happening.
>
> In my case it happens when a publishing to a topic (~ 1000 msgs)  
> with a
> offline durable subscribe with multiple hard disconnects from the
> subscriber.
> --
> View this message in context: http://www.nabble.com/Usage-Manager- 
> for-durable-offline-subscriber-tf4572289s2354.html#a13051280
> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>

[Nicky Sandhu]

I have been trying to reproduce it in a unit test. No luck so far :( I have hunch its the client disconnections and reconnections without a proper session.close

It would be great to have Usage exposed via MBeans. Any plans to do that?

IMHO, reference counting is extremely difficult to implement especially when increment and decrements are async. Any other strategies under consideration ?

oops - that's the latest- must drink coffee before reading email ;)
Any chance of a test case ?

cheers,

Rob

rajdavies wrote:

Hi Nicky,

which version is that?


cheers,

Rob

http://rajdavies.blogspot.com/



On Oct 5, 2007, at 1:17 AM, Nicky Sandhu wrote:

>
> On revision 581885 I turned on debug messages on Usage.java and the  
> commented
> lines of System.out on Message.java and I see the memory usage  
> increase
> beyond 100% to something like 16500%. This causes a problem for the  
> next
> thing that calls Usage.waitForSpace(timeout) and causes an  
> indefinite hang
>
> The Usage.increaseUsage(value) does not check to see if it's full  
> and so can
> keep allocating even beyond 100%. This seems to be happening a lot  
> from
> Message.incrementReferenceCount() with no corresponding
> Message.decrementReferenceCount() happening.
>
> In my case it happens when a publishing to a topic (~ 1000 msgs)  
> with a
> offline durable subscribe with multiple hard disconnects from the
> subscriber.
> --
> View this message in context: http://www.nabble.com/Usage-Manager- 
> for-durable-offline-subscriber-tf4572289s2354.html#a13051280
> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>

[Nicky Sandhu]

Still no luck with a unit test but here's some info from a debugging session
I am using a Journaled persistence adapter with Derby backend. I have registered a durable subscriber that  is then disconnected. A producer produces ~ 1000 msgs to the topic. The durable subscriber comes on line (after a broker restart sometimes) and I put a conditional breakpoint (newPercentUsage > 100) on the running remote broker and here's the stack trace where it is about to exceed the limit



Hope this helps...

Nicky Sandhu wrote:

I have been trying to reproduce it in a unit test. No luck so far :( I have hunch its the client disconnections and reconnections without a proper session.close

On Oct 5, 2007, at 1:17 AM, Nicky Sandhu wrote:

>
> On revision 581885 I turned on debug messages on Usage.java and the  
> commented
> lines of System.out on Message.java and I see the memory usage  
> increase
> beyond 100% to something like 16500%. This causes a problem for the  
> next
> thing that calls Usage.waitForSpace(timeout) and causes an  
> indefinite hang
>
> The Usage.increaseUsage(value) does not check to see if it's full  
> and so can
> keep allocating even beyond 100%. This seems to be happening a lot  
> from
> Message.incrementReferenceCount() with no corresponding
> Message.decrementReferenceCount() happening.
>
> In my case it happens when a publishing to a topic (~ 1000 msgs)  
> with a
> offline durable subscribe with multiple hard disconnects from the
> subscriber.
> --
> View this message in context: http://www.nabble.com/Usage-Manager- 
> for-durable-offline-subscriber-tf4572289s2354.html#a13051280
> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>

[Nicky Sandhu]

I have a test case that reproduces the problem. WARNING: This test case is not a unit test even though it uses the junit framework.

Please have a look at https://issues.apache.org/activemq/browse/AMQ-1449.



oops - that's the latest- must drink coffee before reading email ;)
Any chance of a test case ?

cheers,

Rob

rajdavies wrote:

Hi Nicky,

which version is that?


cheers,

Rob

http://rajdavies.blogspot.com/



On Oct 5, 2007, at 1:17 AM, Nicky Sandhu wrote:

>
> On revision 581885 I turned on debug messages on Usage.java and the  
> commented
> lines of System.out on Message.java and I see the memory usage  
> increase
> beyond 100% to something like 16500%. This causes a problem for the  
> next
> thing that calls Usage.waitForSpace(timeout) and causes an  
> indefinite hang
>
> The Usage.increaseUsage(value) does not check to see if it's full  
> and so can
> keep allocating even beyond 100%. This seems to be happening a lot  
> from
> Message.incrementReferenceCount() with no corresponding
> Message.decrementReferenceCount() happening.
>
> In my case it happens when a publishing to a topic (~ 1000 msgs)  
> with a
> offline durable subscribe with multiple hard disconnects from the
> subscriber.
> --
> View this message in context: http://www.nabble.com/Usage-Manager- 
> for-durable-offline-subscriber-tf4572289s2354.html#a13051280
> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>