4.0 Consumer OutOfMemoryError bug?

[skarthik]

Hi,

I am using the latest activemq-4.0 (downloaded today 5/30/2006) The following problem exists in activemq-4.0 downloaded on 5/8/2006 as well, but NOT in RC2.

The setup includes one activemq server, one simple producer process and one simple consumer process. After the consumer receives around 64MB of messages (6200 10KB messages or 998 64KB messages or 634 100KB messages), it throws the following error on its console

Exception in thread "ActiveMQ Transport: tcp:///143.102.32.173:8000" java.lang.OutOfMemoryError: Java heap space

The producer publishes the messages at a rate of 10 messages per second on a Topic using DeliveryMode.PERSISTENT. The test programs are very simple and only just log the messages sent and recieved. So they should not have any memory leaks. The JMS server uses default configuration (persistent broker)

Is there any configuration that needs to be specified to eliminate this problem? Like I mentioned at the start, the exact same test code works fine with RC2 version (which was verified/re-tested today).

thanks
karthik
 

[James.Strachan]

On 5/30/06, skarthik <karthik.sethuraman@...> wrote:

>
> Hi,
>
> I am using the latest activemq-4.0 (downloaded today 5/30/2006) The
> following problem exists in activemq-4.0 downloaded on 5/8/2006 as well, but
> NOT in RC2.
>
> The setup includes one activemq server, one simple producer process and one
> simple consumer process. After the consumer receives around 64MB of messages
> (6200 10KB messages or 998 64KB messages or 634 100KB messages), it throws
> the following error on its console
>
> Exception in thread "ActiveMQ Transport: tcp:///143.102.32.173:8000"
> java.lang.OutOfMemoryError: Java heap space
>
> The producer publishes the messages at a rate of 10 messages per second on a
> Topic using DeliveryMode.PERSISTENT. The test programs are very simple and
> only just log the messages sent and recieved. So they should not have any
> memory leaks. The JMS server uses default configuration (persistent broker)
>
> Is there any configuration that needs to be specified to eliminate this
> problem? Like I mentioned at the start, the exact same test code works fine
> with RC2 version (which was verified/re-tested today).

So the JVM that runs out of memory - is that just the consumer, or is
it the consumer, the broker and producer? Whats the heap size of that
JVM? What acknowldgement mode are you using?

--

James
-------
http://radio.weblogs.com/0112098/

[skarthik]

Only the consumer JVM (JDK 1.5) runs out of memory. I am using the default JVM heap sizes which I think is 64MB.

 But I put some logging statements in the consumer code to display memory statistics after receiving every message (using Runtime.maxMemory(), Runtime.totalMemory() and Runtime.freeMemory()). I see that max memory is 64MB and the total memory used slowly creeps up to that max mark and then encounters the OutOfMemoryError.

 Also I am using a non-durable subscriber, and am using subscriber.receive () with timeout to pull the messages from the Topic (NOT using MessageListener interface). Also the TopicSessions used are non-transacted with Session.AUTO_ACKNOWLEDGE mode.

Hope this helps.
karthik

[James.Strachan]

You could try lowering the prefetch size to reduce the amount of
Message objects which are loaded into the consumer....

http://incubator.apache.org/activemq/what-is-the-prefetch-limit-for.html

Try set it to a low value like 100 and see if that fixes it.

On 5/31/06, skarthik <karthik.sethuraman@...> wrote:

>
> Only the consumer JVM (JDK 1.5) runs out of memory. I am using the default
> JVM heap sizes which I think is 64MB.
>
>  But I put some logging statements in the consumer code to display memory
> statistics after receiving every message (using Runtime.maxMemory(),
> Runtime.totalMemory() and Runtime.freeMemory()). I see that max memory is
> 64MB and the total memory used slowly creeps up to that max mark and then
> encounters the OutOfMemoryError.
>
>  Also I am using a non-durable subscriber, and am using subscriber.receive
> () with timeout to pull the messages from the Topic (NOT using
> MessageListener interface). Also the TopicSessions used are non-transacted
> with Session.AUTO_ACKNOWLEDGE mode.
>
> Hope this helps.
> karthik
> --
> View this message in context: http://www.nabble.com/4.0+Consumer+OutOfMemoryError+bug--t1707655.html#a4645070
> Sent from the ActiveMQ - User forum at Nabble.com.
>
>

--

James
-------
http://radio.weblogs.com/0112098/

[skarthik]

Hi James,

Well I read the information on prefetch limit (the link you specified) and iIMHO I don't believe that is the issue.

First, I lowered the rate of messages published to 1 per second and I still see the heap size growing at the same rate (per KB of messages received) and it eventually hits the OutOfMemoryError. There is only one publisher and one consumer connected the JMS server.

Second my consumer is a test program and is very fast. All it does is display the messages received. Also the log time of the published messages (logged by producer before publish) and received messages (logged by consumer after received) are the same (down to the last millisecond).

I do feel there maybe a critical memory leak somewhere. Also as I mentioned in my previous post, the RC2 build does not have this problem.

thanks
karthik

[amerigo5]

Can you post your code and activemq.conf?

Also, make sure that your client code is using the correct version of jar files.

Thanks.

[skarthik]

HI,

I would like send in the test code files as zip file (along with some script files to compile and run it). Is there anyway I can attach it or any email address to send it to?

Thanks
 karthik

[James.Strachan]

On 5/31/06, skarthik <karthik.sethuraman@...> wrote:
>
> HI,
>
> I would like send in the test code files as zip file (along with some script
> files to compile and run it). Is there anyway I can attach it or any email
> address to send it to?

The easiest way is to raise a JIRA issue in our issue tracker then
attach whatever source files & config files to that then we can try
replicate your issue

http://incubator.apache.org/activemq/support.html
--

James
-------
http://radio.weblogs.com/0112098/

[skarthik]

I have raised the follwing issue in JIRA, which has the test code attached.

http://issues.apache.org/activemq/browse/AMQ-735

karthik

[skarthik]

Hi,

Just following up. Has anybody from the development team had a chance to look at this problem and the test code I posted? Was wondering if this is a geniune problem or is it something I am doing wrong?

thanks,
karthik

[Hiram Chirino]

Sorry.. not yet. I've been swamped..  I'll try hooking up a memory
profiler to your test case perhaps early next week.

On 6/6/06, skarthik <karthik.sethuraman@...> wrote:

>
> Hi,
>
> Just following up. Has anybody from the development team had a chance to
> look at this problem and the test code I posted? Was wondering if this is a
> geniune problem or is it something I am doing wrong?
>
> thanks,
> karthik
> --
> View this message in context: http://www.nabble.com/4.0-Consumer-OutOfMemoryError-bug--t1707655.html#a4733986
> Sent from the ActiveMQ - User forum at Nabble.com.
>
>

--
Regards,
Hiram

[James.Strachan]

BTW in any kind of memory leak type situation, it can help us to help
you quicker if you run a profiler on the test case which will then
report the types of objects which are leaking. e.g. if you run
JProfiler or YourKit on the test case and attach the memory output
file to the JIRA (probably zipped up if its huge) then we can
typically diagnose the issue quicker.

On 6/6/06, Hiram Chirino <hiram@...> wrote:

> Sorry.. not yet. I've been swamped..  I'll try hooking up a memory
> profiler to your test case perhaps early next week.
>
> On 6/6/06, skarthik <karthik.sethuraman@...> wrote:
> >
> > Hi,
> >
> > Just following up. Has anybody from the development team had a chance to
> > look at this problem and the test code I posted? Was wondering if this is a
> > geniune problem or is it something I am doing wrong?
> >
> > thanks,
> > karthik
> > --
> > View this message in context: http://www.nabble.com/4.0-Consumer-OutOfMemoryError-bug--t1707655.html#a4733986
> > Sent from the ActiveMQ - User forum at Nabble.com.
> >
> >
>
>
> --
> Regards,
> Hiram
>

--

James
-------
http://radio.weblogs.com/0112098/

[FWang]

I ran JProbe on karthik's code and I suspect that MessageDispath has the memory leak. Please see the attached screen shot from "Memory Leak Detector" in JProbe.

I have put a copy of zip file (activemq_735.zip) on Jira as well. (http://issues.apache.org/activemq/browse/AMQ-735)  

thanks,

[James.Strachan]

On 6/7/06, FWang <feng.wang@...> wrote:
>
>
> I ran JProbe on karthik's code and I suspect that MessageDispath has the
> memory leak. Please see the attached screen shot from "Memory Leak Detector"
> in JProbe.

Thanks for doing that!

Though I don't really understand the pictures; if my reading of them
is correct the test program has created 124596 JMS Session objects;
I'm not surprised that ran out of RAM - thats a massive number of
sessions; you generally only need at most one per thread.

JMS Sessions are created by JMS client code - they are not controlled
by the JMS provider - so if too many of those are created I suspect
its your application code thats at fault.

Also imagine when you start consuming topics on that many sessions;
each consumer on the session will receive its own copy of the message
- then we prefech a few thousand messages by default per consumer - so
running out of RAM with that configuration doesnt' seem like a memory
leak - it seems like normal behaviour.

http://incubator.apache.org/activemq/what-is-the-prefetch-limit-for.html

(Unless the number next to the objects in the pictures is not the
number of instances created in which case please disregard this entire
email :)

--

James
-------
http://radio.weblogs.com/0112098/

[FWang]

I have post more info on http://issues.apache.org/activemq/browse/AMQ-735. Has anyone investigate the issue?

thanks,

[James.Strachan]

On 6/9/06, FWang <feng.wang@...> wrote:
>
>
> I have post more info on http://issues.apache.org/activemq/browse/AMQ-735.
> Has anyone investigate the issue?

I added a comment to the issue. I'm not yet convinced you are
experiencing any memory leak; its just the effect of the pre-fetch. If
you want to use a small heap size with messages of around 10K then
you'll probably need to reduce the pre-fetch values for topics to
something like 10

http://incubator.apache.org/activemq/what-is-the-prefetch-limit-for.html

--

James
-------
http://radio.weblogs.com/0112098/

[skarthik]

James,

Thanks for your valuable time on following up on this issue.

Well, I tried out some more test scenarios over the weekend and now DON'T think it is a memory leak problem, but definitely some kind of cleanup problem. Also I would like to reiterate that these issuses did not exist in RC2 build.

I ran the test programs submitted in JIRA, with following parameters.

Producer publishes messages of 10K size.

The consumer heap was increased to 220 MB, to allow the consumer to store more than 20,000 messages (if required). Used the default topic prefetch values. (i think you mentioned it was 4000).
With this heap size, the consumer runs out of memory after consuming around 19,000 messages.

Now increased the consumer heap size to 225MB. Then a peculiar thing happens. After using up almost all of the heap memory (225 MB, and consuming about 21500-odd messages), the whole heap (about 210MB) is freed up in one shot. And this cycle (of using upto 225 MB and then freeing up all of it) continues after consuming every 21500-odd messages and the consumer never runs out of memory.

So the question is why does it wait for upto 225 MB of memory to be used up before some kind of cleanup kicks in.

 BTW, since the memory profile test case reported by Feng (my co-worker) in the JIRA issue-tracker, involved messages of 64K and a heap size of only 64MB, I can see why it might be difficult to convince that a problem exists. But I don't think the prefetch size is an issue, since the consumer consumes the messages at the same rate as the producer, unless the messages are kept around even after the consumer has processed it (my understanding of activemq internals/behavior is not deep). Also for our application (not the test code), we would like to keep the default prefetch size (around 4000-5000).

Anyways, hope you guys get a chance to look at this problem as time is running out for us (we have a deadline to deilver by this month-end). Also would like to mention that the reason we are stressing on this issue is, that using activemq 4.0, our actual application (not the test code submitted) does not pass through our QA department. RC2 build does, but it has some known issues.

Thanks,
Karthik

[James.Strachan]

For high performance we have an optimised acknowledgement mode which
is an attempt at acknowledging in batches rather than on every single
message. This is on by default in 4.0 but we've since disabled it as
some folks have experienced problems in some scenarios. Batching
acknowledgements causes the client to keep consumed messages around
until they are acknowledged; which further increases the RAM usage.
Given you are consumed over heap size you might want to disable this
feature...

http://incubator.apache.org/activemq/connection-configuration-uri.html

so connect via http://localhost:61616?jms.optimizeAcknowledge=false

(in 4.0.1 this will be the default value).

Other than that as I said, if you don't want a heap of such a large
size then just reduce down the prefetech; things won't run quite as
fast but if you've such little RAM on the boxes you are running
ActiveMQ then you have to trade off RAM with performance.



On 6/12/06, skarthik <karthik.sethuraman@...> wrote:

>
> James,
>
> Thanks for your valuable time on following up on this issue.
>
> Well, I tried out some more test scenarios over the weekend and now DON'T
> think it is a memory leak problem, but definitely some kind of cleanup
> problem. Also I would like to reiterate that these issuses did not exist in
> RC2 build.
>
> I ran the test programs submitted in JIRA, with following parameters.
>
> Producer publishes messages of 10K size.
>
> The consumer heap was increased to 220 MB, to allow the consumer to store
> more than 20,000 messages (if required). Used the default topic prefetch
> values. (i think you mentioned it was 4000).
> With this heap size, the consumer runs out of memory after consuming around
> 19,000 messages.
>
> Now increased the consumer heap size to 225MB. Then a peculiar thing
> happens. After using up almost all of the heap memory (225 MB, and consuming
> about 21500-odd messages), the whole heap (about 210MB) is freed up in one
> shot. And this cycle (of using upto 225 MB and then freeing up all of it)
> continues after consuming every 21500-odd messages and the consumer never
> runs out of memory.
>
> So the question is why does it wait for upto 225 MB of memory to be used up
> before some kind of cleanup kicks in.
>
>  BTW, since the memory profile test case reported by Feng (my co-worker) in
> the JIRA issue-tracker, involved messages of 64K and a heap size of only
> 64MB, I can see why it might be difficult to convince that a problem exists.
> But I don't think the prefetch size is an issue, since the consumer consumes
> the messages at the same rate as the producer, unless the messages are kept
> around even after the consumer has processed it (my understanding of
> activemq internals/behavior is not deep). Also for our application (not the
> test code), we would like to keep the default prefetch size (around
> 4000-5000).
>
> Anyways, hope you guys get a chance to look at this problem as time is
> running out for us (we have a deadline to deilver by this month-end). Also
> would like to mention that the reason we are stressing on this issue is,
> that using activemq 4.0, our actual application (not the test code
> submitted) does not pass through our QA department. RC2 build does, but it
> has some known issues.
>
> Thanks,
> Karthik
>
> --
> View this message in context: http://www.nabble.com/4.0-Consumer-OutOfMemoryError-bug--t1707655.html#a4822824
> Sent from the ActiveMQ - User forum at Nabble.com.
>
>

--

James
-------
http://radio.weblogs.com/0112098/

[skarthik]

Thanks James,

It looks like batch acknowledgement was the problem. Once omtimizeAcknowledgeis turned OFF, the consumer memory never grows beyond 1-2 MB which is what were expecting. We did NOT have to tune the prefetch size (which BTW we found out is 32000 by default).

 I don't know what is the algorithm used for determining when the grouped messages need to be acknowledged, but if in case you guys are going to revisit it, maybe it should take available heap size into consideration (just a suggestion).

Thanks again for your time.

karthik

[jevans12]

Are prefetch policy and optimized ack's Autonomous? As I see a consumer will ask for 65% of the prefetch policy messages. Does the batching of ack's follow this? as I receive the 65th (of 100) message a batch ack will be sent for those 65?