Re: Performance issues with 7.4.0 Cluster

Today we had the issue again:

- Standby HA Node shows 1.3 GB in old gen and we got an OOM:

INFO   | jvm 1    | 2009/07/24 13:07:17 | java.lang.OutOfMemoryError: Java heap space
INFO   | jvm 1    | 2009/07/24 13:07:17 | Got OutOfMemoryError:
INFO   | jvm 1    | 2009/07/24 13:07:17 |     ThreadGroup: hacontroller.stagecontroller
INFO   | jvm 1    | 2009/07/24 13:07:17 |     ActiveTask : PipelineQueue, dispatchToken=sys$hacontroller.stagecontroller
INFO   | jvm 1    | 2009/07/24 13:07:17 | Stack Trace:
INFO   | jvm 1    | 2009/07/24 13:07:17 | java.lang.OutOfMemoryError: Java heap space
INFO   | jvm 1    | 2009/07/24 13:07:17 |       at com.swiftmq.tools.collection.IntRingBuffer.add(Unknown Source)
INFO   | jvm 1    | 2009/07/24 13:07:17 |       at com.swiftmq.impl.store.standard.cache.StableStore.a(Unknown Source)
INFO   | jvm 1    | 2009/07/24 13:07:17 |       at com.swiftmq.impl.store.standard.cache.StableStore.free(Unknown Source)
INFO   | jvm 1    | 2009/07/24 13:07:17 |       at com.swiftmq.impl.store.standard_ha.v600.StandbyVisitor.visit(Unknown Source)
INFO   | jvm 1    | 2009/07/24 13:07:17 |       at com.swiftmq.impl.store.standard_ha.v600.protocol.PageDBFreeRequest.accept(Unknown Source)
INFO   | jvm 1    | 2009/07/24 13:07:17 |       at com.swiftmq.impl.store.standard_ha.v600.StandbyVisitor.visit(Unknown Source)
INFO   | jvm 1    | 2009/07/24 13:07:17 |       at com.swiftmq.impl.store.standard_ha.v600.protocol.TransactionEndRequest.accept(Unknown Source)
INFO   | jvm 1    | 2009/07/24 13:07:17 |       at com.swiftmq.impl.store.standard_ha.v600.SinkProxyImpl.newReplicationItem(Unknown Source)
INFO   | jvm 1    | 2009/07/24 13:07:17 |       at com.swiftmq.impl.hacontroller.standard.v600.stage.StandbyStage.visit(Unknown Source)
INFO   | jvm 1    | 2009/07/24 13:07:17 |       at com.swiftmq.impl.hacontroller.standard.v600.smqpha.UpdateDeliveryRequest.accept(Unknown Source)
INFO   | jvm 1    | 2009/07/24 13:07:17 |       at com.swiftmq.impl.hacontroller.standard.v600.stage.StandbyStage.process(Unknown Source)
INFO   | jvm 1    | 2009/07/24 13:07:17 |       at com.swiftmq.impl.hacontroller.standard.stage.StageController.visit(Unknown Source)
INFO   | jvm 1    | 2009/07/24 13:07:17 |       at com.swiftmq.impl.hacontroller.standard.stage.po.PORequestReceived.accept(Unknown Source)
INFO   | jvm 1    | 2009/07/24 13:07:17 |       at com.swiftmq.tools.pipeline.PipelineQueue.process(Unknown Source)
INFO   | jvm 1    | 2009/07/24 13:07:17 |       at com.swiftmq.tools.queue.SingleProcessorQueue.dequeue(Unknown Source)
INFO   | jvm 1    | 2009/07/24 13:07:17 |       at com.swiftmq.tools.pipeline.PipelineQueue$QueueProcessor.run(Unknown Source)
INFO   | jvm 1    | 2009/07/24 13:07:17 |       at com.swiftmq.impl.threadpool.standard.PoolThread.run(Unknown Source)


At same time producers can't longer deliver any Msgs to active node also, which is a quite unexpected behavior for a cluster/HA solution at all.

Restarting Standby having the OOM solves issue on Active node also.

Today I analysed an older HeapDump which was generated on same host/cluster with same issue and seems there is a bug, please see attachment. SmwiftMQ_74_HA_HeapDump_Standby.pdf

If needed / helpful I can send (138 MB) by ftp the compressed HeapDump.

One more srceenshot from different HeapAnalyser:



Thanks,
Michael