Another Heap-Space question

[tabshift]

We produce a lot of messages that might be consumed far later than they were produced.
Even when we run the broker with a heap space of 1024 Mb we run into an out-of-heapspace-error.
What can we do to prevent this?

The msgs are not too big, but 500.000 msgs of 2Kb are also a large amount of data.

[IIT Software]

Each queue has a cache which is 500 messages by default. Only those messages are held in memory. The others are stored in persistent store or in swap file (non-persistent messages). There is only a few bytes messages index held in memory for each message.

So let's assume you have 30 queues, 500 msgs/cache per queue, messages not larger than 2K =

30 x 500 x 2 = 30000K = ~30 MB.

Plus the message index of, say, 100 bytes per message =

100 x 500000 = 50000000 bytes = 49000K = ~47 MB

The persistent store has a cache as well, per default 2048 pages á 2K =

4096K = 4 MB

There are also 256 KB network buffer per connection. Say, you have 100 connections:

100 x 256 = 25600K = ~25MB

Makes 30 + 47 + 4 + 25 = 106 MB heap space in that configuration.

IMO, there must be another problem. Are you sure that you don't use a producer concurrently? 7.0.1 fixes that at the client side so you must use 7.0.1 swiftmq.jar at the client.

[IIT Software]

I did a test with 510000 messages (30 queues, each 17000, message size 2K). Here is a screenshot from SwiftMQ Explorer, showing the memory consumption and the queue sizes.

 

[tabshift]

Okay, thanks a lot.
We will try that!

[tabshift]

That's interesting!
I started the router with 1024 MB of memory.
The process-table tells me:

java -server -Xmx1024M -cp ../../jars/swiftmq.jar:../../jars/jndi.jar:../../jars/jms.jar:../../jars/jsse.jar:../../jars/jnet.jar:../../jars/jcert.jar:../../jars/dom4j-full.jar:../../jars/jta-spec1_0_1.jar com.swiftmq.HARouter ../../config/replicated/instance1/routerconfig.xml

My explorer reports a Total Memory of 204508 KB ?!
Main memory of the server machine is 2 Gigs!

What's wrong?

TIA
Thomas

[IIT Software]

tabshift wrote:

I started the router with 1024 MB of memory.

That is the max heap size (-Xmx). There is another option for the min heap size (-Xmn). If you set both to the same value then your process will have this size all the time.

tabshift wrote:

My explorer reports a Total Memory of 204508 KB ?!

That's 200 MB, nothing special actually. Did you get that right after startup without messages in your queues? It's important what you see as free memory. My screenshot was from an ACTIVE HA instance running on my XP box (win32). If I run the same on a Linux x64 system (64 bit), the memory consumption goes up because of the 64 bit addressing.

[tabshift]

So now i made an upgrade to SwiftMQ 7.0.1.
I started the broker with
java -server -Xmx768M -Xmn768M -cp ../../jars/swiftmq.jar:../../jars/jndi.jar:../../jars/jms.jar:../../jars/jsse.jar:../../jars/jnet.jar:../../jars/jcert.jar:../../jars/dom4j-full.jar:../../jars/jta-spec1_0_1.jar com.swiftmq.HARouter ../../config/replicated/instance1/routerconfig.xml

(master and slave)

Our developers wrote a test-tool that behaves just like our main application concerning JMS.
It works whithin the specifications and I produced 1500 Kb messages in 25 threads a hundred times.

The active Instance (master) reported
+++ High Availability State: ACTIVE/ACTIVE
Got OutOfMemoryError:
    ThreadGroup: store.log
    ActiveTask : LogProcessor
Stack Trace:
java.lang.OutOfMemoryError: Java heap space
heartbeat counter reaches 0, closing replication connections
+++ High Availability State: STANDALONE/STANDALONE

the slave reported

Got Exception:
    ThreadGroup: hacontroller.stagecontroller
    ActiveTask : PipelineQueue, dispatchToken=sys$hacontroller.stagecontroller
Stack Trace:
java.lang.NullPointerException
        at com.swiftmq.tools.util.DataByteArrayInputStream.setBuffer(Unknown Source)
        at com.swiftmq.impl.store.standard_ha.v600.SinkProxyImpl.newReplicationItem(Unknown Source)
        at com.swiftmq.impl.hacontroller.standard.v600.stage.StandbyStage.visit(Unknown Source)
        at com.swiftmq.impl.hacontroller.standard.v600.smqpha.UpdateDeliveryRequest.accept(Unknown Source)
        at com.swiftmq.impl.hacontroller.standard.v600.stage.StandbyStage.process(Unknown Source)
        at com.swiftmq.impl.hacontroller.standard.stage.StageController.visit(Unknown Source)
        at com.swiftmq.impl.hacontroller.standard.stage.po.PORequestReceived.accept(Unknown Source)
        at com.swiftmq.tools.pipeline.PipelineQueue.process(Unknown Source)
        at com.swiftmq.tools.queue.SingleProcessorQueue.dequeue(Unknown Source)
        at com.swiftmq.tools.pipeline.PipelineQueue$QueueProcessor.run(Unknown Source)
        at com.swiftmq.impl.threadpool.standard.PoolThread.run(Unknown Source)
heartbeat counter reaches 0, closing replication connections

and the memory-monitor reported



I can send the test-application (also the source) and our configuration, whatever you need, by mail.
You know my mail-adress from our last official incident.

Maybe it is a problem that just affects Linux?
What can I do?

TIA
Thomas