Stresstest stops with QueueException: too many open files

[Dirk Grosskopf]

We are currently tuning our production environment by changing various SwiftMQ settings. In order to simulate our production environment, we coded a simple message producer, that constantly creates text messages with variable message size ( 300 to 1MB text body size). The router platform is RHEL 5.2 with Sun Java JDK 1.6.0_11 64 Bit. The initial router configuration was simple out of the box, except we increased the maximum Java heap size to 2GB.

We are now facing the following problem:
When sending non persistent messages, the producer process receives the following exception after 82,000 messages on the queue:

---

com.swiftmq.swiftlet.queue.QueueException: com.swiftmq.swiftlet.store.StoreException:
/opt/swiftmq_7_4_0/scripts/unix/../../store/router1/swap/TESTQUEUE-955.swap (Too many open files)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.swiftmq.tools.requestreply.ReplyNE.readContent(Unknown Source)
at com.swiftmq.jms.smqp.v510.ProduceMessageReply.readContent(Unknown Source)
at com.swiftmq.tools.dump.Dumpalizer.construct(Unknown Source)
at com.swiftmq.jms.v510.ConnectionImpl.dataAvailable(Unknown Source)
at com.swiftmq.net.client.BlockingConnection.chunkCompleted(Unknown Source)
at com.swiftmq.net.protocol.smqp.SMQPInputHandler.setBytesWritten(Unknown Source)
at com.swiftmq.net.protocol.smqp.SMQPInputHandler.setBytesWritten(Unknown Source)
at com.swiftmq.net.client.BlockingConnection.run(Unknown Source)


---

After that, no new connection can be established to the router. The router must be shut down, but then all messages are lost. We can reproduce this behaviour.
Some notes:

- There is no problem with persistent messages
- We tried version 5.2.5 and 7.4.0 swiftmq.jar for the producer. No difference
- The producer opens one session (auto acknowledge) and constantly puts new message instances
- We know, SwiftMQ is not a database and 100,000 messages in a queue is not an ideal situation, but we have a high traffic environment and 100,000 messages in a single queue happens sometimes, when back end systems are down.
- We tried to use larger client input buffer sizes, but nothing changed
- The disk is not full


Any hints?

Regards DG

[IIT Software]

Nonpersistent messages are swapped to disk. Check Store Swiftlet/Swap and then "rollover-size". This is the file size at which a current .swap file is closed and a new one is created. The old .swap files are deleted when all messages of that particular file are consumed. Default for rollover-size is 10 MB.

What happens on your side is that 900+ swap files are created but never deleted (because you don't consume) and that you've reached your ulimit which seems to be 1024. Check ulimit -n.

I recommed to increase ulimit or to increase rollover-size, e.g. 50 MB.

[IIT Software]

Dirk Grosskopf wrote:

<code>
com.swiftmq.swiftlet.queue.QueueException: com.swiftmq.swiftlet.store.StoreException:<br /> /opt/swiftmq_7_4_0/scripts/unix/../../store/router1/swap/TESTQUEUE-955.swap (Too many open files)<br />
        at com.swiftmq.jms.smqp.v510.ProduceMessageReply.readContent(Unknown Source)<br />
</code>

Btw, you are using a 7.4.0 router but your client is still at 5.x.

[Dirk Grosskopf]

Thanks

It works now.

We are using the 5.2.5 client for legacy reasons, but plan to migrate the client to 7.4.0 as soon as possible.

Regards Dirk