|
java.lang.OutOfMemoryError: allocLargeObjectOrArray
SwiftMQ (still 4.5.1.) was running very stable since years, on current server (Intel Quad Core / 4 GB RAM) since about 10 month without issues. JVM has 2048 MB Memory, we run jrockit-R27.4.0-jdk1.5.0_12, OS/JVM/Wrapper is 64Bit.
Since some weeks we get an exception and JVM is crashing (no configuration changes had been done):
INFO | jvm 1 | 2009/01/09 10:49:38 | Exception in thread "PoolThread" java.lang.OutOfMemoryError: allocLargeObjectOrArray - Object size: 131088, Num elements: 131072
INFO | jvm 1 | 2009/01/09 10:49:58 | Exception in thread "PoolThread" java.lang.OutOfMemoryError: allocLargeObjectOrArray - Object size: 2064, Num elements: 2048
INFO | jvm 1 | 2009/01/12 09:50:44 | Exception in thread "PoolThread" java.lang.OutOfMemoryError: allocLargeObjectOrArray - Object size: 3984904, Num elements: 996221
INFO | jvm 1 | 2009/01/12 09:51:26 | Exception in thread "PoolThread" java.lang.OutOfMemoryError: allocLargeObjectOrArray - Object size: 131088, Num elements: 131072
INFO | jvm 1 | 2009/01/13 16:13:14 | Exception in thread "PoolThread" java.lang.OutOfMemoryError: allocLargeObjectOrArray - Object size: 131088, Num elements: 131072
INFO | jvm 1 | 2009/01/13 16:13:20 | Exception in thread "PoolThread" java.lang.OutOfMemoryError: allocLargeObjectOrArray - Object size: 131088, Num elements: 131072
INFO | jvm 1 | 2009/01/13 16:13:14 | Exception in thread "PoolThread" java.lang.OutOfMemoryError: allocLargeObjectOrArray - Object size: 131088, Num elements: 131072
INFO | jvm 1 | 2009/01/13 16:13:14 | at com.swiftmq.net.protocol.smqp.SMQPOutputHandler.ensureInput(Unknown Source)
INFO | jvm 1 | 2009/01/13 16:13:14 | at com.swiftmq.net.protocol.smqp.SMQPOutputHandler.addByte(Unknown Source)
INFO | jvm 1 | 2009/01/13 16:13:14 | at com.swiftmq.net.protocol.ProtocolOutputHandler.write(Unknown Source)
INFO | jvm 1 | 2009/01/13 16:13:14 | at com.swiftmq.impl.net.nio.CountableNioOutputStream.write(Unknown Source)
INFO | jvm 1 | 2009/01/13 16:13:14 | at com.swiftmq.tools.util.DataStreamOutputStream.writeInt(Unknown Source)
INFO | jvm 1 | 2009/01/13 16:13:14 | at com.swiftmq.tools.dump.Dumpalizer.dump(Unknown Source)
INFO | jvm 1 | 2009/01/13 16:13:14 | at com.swiftmq.impl.jms.standard.VersionSelector.a(VersionSelector.java:4)
INFO | jvm 1 | 2009/01/13 16:13:14 | at com.swiftmq.impl.jms.standard.VersionSelector.dataAvailable(VersionSelector.java:33)
INFO | jvm 1 | 2009/01/13 16:13:15 | at com.swiftmq.impl.net.nio.scheduler.NioHandler.chunkCompleted(Unknown Source)
INFO | jvm 1 | 2009/01/13 16:13:15 | at com.swiftmq.net.protocol.smqp.SMQPInputHandler.setBytesWritten(Unknown Source)
INFO | jvm 1 | 2009/01/13 16:13:15 | at com.swiftmq.net.protocol.smqp.SMQPInputHandler.setBytesWritten(Unknown Source)
INFO | jvm 1 | 2009/01/13 16:13:15 | at com.swiftmq.impl.net.nio.scheduler.NioHandler.onEvent(Unknown Source)
INFO | jvm 1 | 2009/01/13 16:13:15 | at com.swiftmq.impl.net.nio.scheduler.SelectTask.run(Unknown Source)
INFO | jvm 1 | 2009/01/13 16:13:15 | at com.swiftmq.impl.threadpool.standard.PoolThread.run(Unknown Source)
INFO | jvm 1 | 2009/01/13 16:13:20 | Exception in thread "PoolThread" java.lang.OutOfMemoryError: allocLargeObjectOrArray - Object size: 131088, Num elements: 131072
INFO | jvm 1 | 2009/01/13 16:13:20 | at com.swiftmq.net.protocol.smqp.SMQPInputHandler.createInputBuffer(Unknown Source)
INFO | jvm 1 | 2009/01/13 16:13:20 | at com.swiftmq.impl.net.nio.scheduler.NioHandler.<init>(Unknown Source)
INFO | jvm 1 | 2009/01/13 16:13:20 | at com.swiftmq.impl.net.nio.scheduler.NioTCPListener.onEvent(Unknown Source)
INFO | jvm 1 | 2009/01/13 16:13:20 | at com.swiftmq.impl.net.nio.scheduler.SelectTask.run(Unknown Source)
INFO | jvm 1 | 2009/01/13 16:13:20 | at com.swiftmq.impl.threadpool.standard.PoolThread.run(Unknown Source)
STATUS | wrapper | 2009/01/13 16:21:20 | TERM trapped. Shutting down.
Any help is welcome, in case further informations are needed please let me know.
Thanks,
Michael
|
|
Re: java.lang.OutOfMemoryError: allocLargeObjectOrArray
You either have too many connections or you send large messages or both. Try to reduce the smqp-consumer-cache-size of your connection factory. Default is 500 messages. Set it to 50 may be.
Some background infos are here.
|
|
Re: java.lang.OutOfMemoryError: allocLargeObjectOrArray
I have replaced JVM from jrockit-R27.4.0-jdk1.5.0_12/64Bit with SUN JVM version 1.5.0_16-b02 64Bit.
Since this change of the JVM so far (1,5 days) the issues has not occured again. But this means not too much, may be after 2 weeks without issues I can say if this was the fix and this JRockit-version the reason.
I have two more questions:
1) Max threads:
I have seen that default in 4.5.1 is:
ThreadPool Swiftlet -> Pools -> jms.connection ->> max threads 1
ThreadPool Swiftlet -> Pools -> jms.session ->> max threads 1
In version 7.x. this has changed per default from min/max to 5 for both.
May it make sense also to increase this in 4.5.1 version or is there a reason to do it not?
2) Recently released 7.4.0 version
Are there already any expierences or known issues with this version?
We work on upgrade 4.5.1 to 7.x and I would like to know if we should go for 7.3.2. or 7.4.
Thanks,
Michael
|
|
Re: java.lang.OutOfMemoryError: allocLargeObjectOrArray
m_oehme wrote:
1) Max threads:
I have seen that default in 4.5.1 is:
ThreadPool Swiftlet -> Pools -> jms.connection ->> max threads 1
ThreadPool Swiftlet -> Pools -> jms.session ->> max threads 1
In version 7.x. this has changed per default from min/max to 5 for both.
May it make sense also to increase this in 4.5.1 version or is there a reason to do it not?
We had this set to 1 each in earlier releases due to performance reasons (to avoid thread context switches). Backdraw was that this single thread per pool may block (i.e. on slow connections etc) so we use 5 now as the default. You can change it in 4.5.1 too. No problem.
m_oehme wrote:
2) Recently released 7.4.0 version
Are there already any expierences or known issues with this version?
We work on upgrade 4.5.1 to 7.x and I would like to know if we should go for 7.3.2. or 7.4.
You should always upgrade to the latest release, thus 7.4.0. See the release notes for compatibility and upgrade instructions.
|
|
Re: java.lang.OutOfMemoryError: allocLargeObjectOrArray
Today JVM (this time SUN JVM) chrashed again with a bit different Exception:
INFO | jvm 1 | 2009/01/19 07:47:52 | Exception in thread "PoolThread" java.lang.OutOfMemoryError: Java heap space
INFO | jvm 1 | 2009/01/19 07:47:52 | at com.swiftmq.net.protocol.smqp.SMQPOutputHandler.ensureInput(Unknown Source)
INFO | jvm 1 | 2009/01/19 07:47:52 | at com.swiftmq.net.protocol.smqp.SMQPOutputHandler.addByte(Unknown Source)
INFO | jvm 1 | 2009/01/19 07:47:52 | at com.swiftmq.net.protocol.ProtocolOutputHandler.write(Unknown Source)
INFO | jvm 1 | 2009/01/19 07:47:52 | at com.swiftmq.impl.net.nio.CountableNioOutputStream.write(Unknown Source)
INFO | jvm 1 | 2009/01/19 07:47:52 | at com.swiftmq.tools.util.DataStreamOutputStream.writeInt(Unknown Source)
INFO | jvm 1 | 2009/01/19 07:47:52 | at com.swiftmq.tools.dump.Dumpalizer.dump(Unknown Source)
INFO | jvm 1 | 2009/01/19 07:47:52 | at com.swiftmq.impl.jms.standard.VersionSelector.a(VersionSelector.java:4)
INFO | jvm 1 | 2009/01/19 07:47:58 | at com.swiftmq.impl.jms.standard.VersionSelector.dataAvailable(VersionSelector.java:33)
INFO | jvm 1 | 2009/01/19 07:47:58 | at com.swiftmq.impl.net.nio.scheduler.NioHandler.chunkCompleted(Unknown Source)
INFO | jvm 1 | 2009/01/19 07:47:58 | at com.swiftmq.net.protocol.smqp.SMQPInputHandler.setBytesWritten(Unknown Source)
INFO | jvm 1 | 2009/01/19 07:47:58 | at com.swiftmq.net.protocol.smqp.SMQPInputHandler.setBytesWritten(Unknown Source)
INFO | jvm 1 | 2009/01/19 07:47:58 | at com.swiftmq.impl.net.nio.scheduler.NioHandler.onEvent(Unknown Source)
INFO | jvm 1 | 2009/01/19 07:47:58 | at com.swiftmq.impl.net.nio.scheduler.SelectTask.run(Unknown Source)
INFO | jvm 1 | 2009/01/19 07:47:58 | at com.swiftmq.impl.threadpool.standard.PoolThread.run(Unknown Source)
INFO | jvm 1 | 2009/01/19 07:48:02 | Exception in thread "PoolThread" java.lang.OutOfMemoryError: Java heap space
INFO | jvm 1 | 2009/01/19 07:48:02 | at com.swiftmq.impl.store.standard.cache.CacheManager.a(Unknown Source)
INFO | jvm 1 | 2009/01/19 07:48:02 | at com.swiftmq.impl.store.standard.cache.CacheManager.b(Unknown Source)
INFO | jvm 1 | 2009/01/19 07:48:02 | at com.swiftmq.impl.store.standard.cache.CacheManager.createAndPin(Unknown Source)
INFO | jvm 1 | 2009/01/19 07:48:02 | at com.swiftmq.impl.store.standard.index.PageOutputStream.c(Unknown Source)
INFO | jvm 1 | 2009/01/19 07:48:02 | at com.swiftmq.impl.store.standard.index.PageOutputStream.write(Unknown Source)
INFO | jvm 1 | 2009/01/19 07:48:02 | at com.swiftmq.impl.store.standard.index.PageOutputStream.writeInt(Unknown Source)
INFO | jvm 1 | 2009/01/19 07:48:02 | at com.swiftmq.jms.MessageImpl.writeContent(Unknown Source)
INFO | jvm 1 | 2009/01/19 07:48:02 | at com.swiftmq.impl.store.standard.index.QueueIndex.add(Unknown Source)
INFO | jvm 1 | 2009/01/19 07:48:02 | at com.swiftmq.impl.store.standard.StoreWriteTransactionImpl.insert(Unknown Source)
INFO | jvm 1 | 2009/01/19 07:48:02 | at com.swiftmq.impl.queue.standard.MessageQueue.a(Unknown Source)
INFO | jvm 1 | 2009/01/19 07:48:02 | at com.swiftmq.impl.queue.standard.MessageQueue.prepare(Unknown Source)
INFO | jvm 1 | 2009/01/19 07:48:02 | at com.swiftmq.swiftlet.queue.QueueTransaction.prepare(Unknown Source)
INFO | jvm 1 | 2009/01/19 07:48:02 | at com.swiftmq.impl.routing.single.connection.v400.s.a(Unknown Source)
INFO | jvm 1 | 2009/01/19 07:48:02 | at com.swiftmq.impl.routing.single.connection.v400.DeliveryStage.a(Unknown Source)
INFO | jvm 1 | 2009/01/19 07:48:02 | at com.swiftmq.impl.routing.single.connection.v400.DeliveryStage.a(Unknown Source)
INFO | jvm 1 | 2009/01/19 07:48:02 | at com.swiftmq.impl.routing.single.connection.v400.q.visited(Unknown Source)
INFO | jvm 1 | 2009/01/19 07:48:02 | at com.swiftmq.impl.routing.single.smqpr.SMQRVisitor.a(Unknown Source)
INFO | jvm 1 | 2009/01/19 07:48:02 | at com.swiftmq.impl.routing.single.smqpr.SMQRVisitor.visit(Unknown Source)
INFO | jvm 1 | 2009/01/19 07:48:02 | at com.swiftmq.impl.routing.single.smqpr.v400.TransactionRequest.accept(Unknown Source)
INFO | jvm 1 | 2009/01/19 07:48:02 | at com.swiftmq.impl.routing.single.connection.v400.DeliveryStage.process(Unknown Source)
INFO | jvm 1 | 2009/01/19 07:48:02 | at com.swiftmq.impl.routing.single.connection.stage.StageQueue.process(Unknown Source)
INFO | jvm 1 | 2009/01/19 07:48:02 | at com.swiftmq.tools.queue.SingleProcessorQueue.dequeue(Unknown Source)
INFO | jvm 1 | 2009/01/19 07:48:02 | at com.swiftmq.impl.routing.single.connection.stage.g.run(Unknown Source)
INFO | jvm 1 | 2009/01/19 07:48:02 | at com.swiftmq.impl.threadpool.standard.PoolThread.run(Unknown Source)
Any further advices to solve this?
As mentioned the JVM was replaced and the number of threads increased.
Thanks,
Michael
|
|
Re: java.lang.OutOfMemoryError: allocLargeObjectOrArray
Neither the JVM nor the number of threads is the problem. You probably send large messages or have too many connections or both. What helps is to decrease the smqp-consumer-cache-size of your connection factory.
For example:
<connection-factories>
<connection-factory name="plainsocket@router1" smqp-consumer-cache-size="10"/>
</connection-factories>
|
|
Re: java.lang.OutOfMemoryError: allocLargeObjectOrArray
The Messages are quite small (we send binary messages of about 150 Bytes via JMS/SwiftMQ).
We have a forwarding from one Router to another, second one is crashing, first one processing same messages does not crash (this one have 3 producers and one consumer).
We have about 20 connections on the router having the issue (using NIO), one producer (one connection) and one consumer. Therefore I do not think issue is related to consumer cache size (might be wrong ...), also there haven't been any changes (configuration/load/etc), also number of messages hasen't really changed (we talk about 350 transactional XA Msg/Second).
What is quite interessting is the usage of Survivor space, please see screenshot.
|
|
Re: java.lang.OutOfMemoryError: allocLargeObjectOrArray
Well, if there were no change at all, the behavior won't change at all. Since you did not change the SwiftMQ Router configuration etc, it is very likely the memory consumption is caused by your JMS clients. To get an OOM it is enough if the is a single huge message sent into the system. I therefore recommend to reduce the consumer cache size. Keep in mind this effects throughput so you shouldn't set it to one but to 50 and if that doesn't work to 10.
|
|
Re: java.lang.OutOfMemoryError: allocLargeObjectOrArray
If we talk about the JMS Swiftlet (XA/ASF), this cache is (and was already before the issues happens) still at "10" (not changed):
<listeners>
<listener name="plainsocket" bindaddress="192.168.95.126" port="4001">
<connection-factories>
<connection-factory name="QueueConnectionFactory" smqp-consumer-cache-size="10"/>
<connection-factory name="TopicConnectionFactory"/>
<connection-factory name="plainsocket@msgrouter10" smqp-consumer-cache-size="10"/>
<connection-factory name="plainsocket_appserver@msgrouter10" thread-context-classloader-for-getobject="true"/>
<connection-factory name="plainsocket_recover@msgrouter10" smqp-consumer-cache-size="10"/>
</connection-factories>
<host-access-list/>
</listener>
Are there any further recommendations? If it is helpful, I can send you (by Mail) routerconfig.xml
Thanks a lot,
Michael
|
|
Re: java.lang.OutOfMemoryError: allocLargeObjectOrArray
Looks fine, except you use TopicConnectionFactory.
You can send the routerconfig.xml by using "Reply to author". I'll take a look.
|
|
Re: java.lang.OutOfMemoryError: allocLargeObjectOrArray
Good morning,
we spend late yesterday and this morning again a lot of time on this issue.
It seems that we have a dependency between memory usage and tmp$-Queues.
It seems that the tmp-Queues created are allocating memory which could not be Garbage collected, and so JVM run's to an OOM. After giving JVM 512 MB more Heap it was just a question of time until the heap was completely used again. We see no possible GC's on old Gen and that Survivor was empty, seems no space avaliable. This seems to be the same behavior allways when OOM happens.
The tmp-Queues and the allocated memory are not released after restart of router, only when we move db/transactions and start "empty".
In which cases a tmp-Queue is created and not released (in 4.5.1.)? Please see also screenshot for Queue config. Would it make sense and help to change to persitent?
Thanks,
Michael
|
|
Re: java.lang.OutOfMemoryError: allocLargeObjectOrArray
|
|
Re: java.lang.OutOfMemoryError: allocLargeObjectOrArray
Your configration is ok, except the trace settings. You actually cannot create your own trace spaces because all kernel Swiftlets do their trace to the kernel space.
But that's a minor issue.
|
|
Re: java.lang.OutOfMemoryError: allocLargeObjectOrArray
Update (Issue was with 4.7.0):
Reason was finally bug as described in Release Notes of version 5.1.0:
"Store Swiftlet: Unpin of pages doesn't work sometimes which leads to memory leak and increase of page.db"
Upgrade to 7.4.0 was quite smooth, but swiftmq.jar on producers/consumers was also required to upgrade due to change in challange/response (introduced with 5.0.0: Authentication, ChallengeResponse: Bug in response generator fixed).
7.4.0 is now running since about 10 weeks without issues :-)
|
