« Return to Thread: Stresstest stops with QueueException: too many open files

Stresstest stops with QueueException: too many open files

by Dirk Grosskopf :: Rate this Message:

Reply to Author | View in Thread

We are currently tuning our production environment by changing various SwiftMQ settings. In order to simulate our production environment, we coded a simple message producer, that constantly creates text messages with variable message size ( 300 to 1MB text body size). The router platform is RHEL 5.2 with Sun Java JDK 1.6.0_11 64 Bit. The initial router configuration was simple out of the box, except we increased the maximum Java heap size to 2GB.

We are now facing the following problem:
When sending non persistent messages, the producer process receives the following exception after 82,000 messages on the queue:


com.swiftmq.swiftlet.queue.QueueException: com.swiftmq.swiftlet.store.StoreException:
/opt/swiftmq_7_4_0/scripts/unix/../../store/router1/swap/TESTQUEUE-955.swap (Too many open files)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.swiftmq.tools.requestreply.ReplyNE.readContent(Unknown Source)
at com.swiftmq.jms.smqp.v510.ProduceMessageReply.readContent(Unknown Source)
at com.swiftmq.tools.dump.Dumpalizer.construct(Unknown Source)
at com.swiftmq.jms.v510.ConnectionImpl.dataAvailable(Unknown Source)
at com.swiftmq.net.client.BlockingConnection.chunkCompleted(Unknown Source)
at com.swiftmq.net.protocol.smqp.SMQPInputHandler.setBytesWritten(Unknown Source)
at com.swiftmq.net.protocol.smqp.SMQPInputHandler.setBytesWritten(Unknown Source)
at com.swiftmq.net.client.BlockingConnection.run(Unknown Source)


After that, no new connection can be established to the router. The router must be shut down, but then all messages are lost. We can reproduce this behaviour.
Some notes:

- There is no problem with persistent messages
- We tried version 5.2.5 and 7.4.0 swiftmq.jar for the producer. No difference
- The producer opens one session (auto acknowledge) and constantly puts new message instances
- We know, SwiftMQ is not a database and 100,000 messages in a queue is not an ideal situation, but we have a high traffic environment and 100,000 messages in a single queue happens sometimes, when back end systems are down.
- We tried to use larger client input buffer sizes, but nothing changed
- The disk is not full


Any hints?

Regards DG

 « Return to Thread: Stresstest stops with QueueException: too many open files