[jira] Created: (AMQ-1873) Stomp connections doesn't seem to be released cleanly (Too many open files error)

View: New views

2 Messages — Rating Filter: Alert me

[jira] Created: (AMQ-1873) Stomp connections doesn't seem to be released cleanly (Too many open files error)

by JIRA jira@apache.org :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Stomp connections doesn't seem to be released cleanly (Too many open files error)
---------------------------------------------------------------------------------

Key: AMQ-1873
URL: https://issues.apache.org/activemq/browse/AMQ-1873
Project: ActiveMQ
Issue Type: Bug
Affects Versions: 5.1.0
Environment: Linux RHEL 4
java version "1.5.0_15"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_15-b04)
Java HotSpot(TM) Server VM (build 1.5.0_15-b04, mixed mode)

Reporter: Julien Devemy
Attachments: activemq.xml

I'm actually testing ActiveMQ with a python stomp Client (stomp.py http://www.briggs.net.nz/log/projects/stomppy/).

I'm trying to load the broker with ~200 producers (on ~200 hosts) feeding one TOPIC for only one consumer.

Everything is working almost fine until the client connection fails.

The ActiveMQ log displays :
...
2008-07-30 09:28:43,665 [localhost:61613] ERROR TransportConnector - Could not accept connection : Too many open files
2008-07-30 09:28:43,665 [localhost:61613] DEBUG TransportConnector - Reason: Too many open files
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
at java.net.ServerSocket.implAccept(ServerSocket.java:450)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at org.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportServer.java:221)
at java.lang.Thread.run(Thread.java:595)

When I look at the open files (or connections) (with lsof) I see 65446 lines like :

java 26765 root *066u sock 0,4 7534034 can't identify protocol

This seems to indicate that ActiveMQ doesn't totally released the UNIX socket

I'm attaching the activemq.xml conf file.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (AMQ-1873) Stomp connections doesn't seem to be released cleanly (Too many open files error)

by JIRA jira@apache.org :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

[ https://issues.apache.org/activemq/browse/AMQ-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=55096#action_55096 ]

Alex Hollerith commented on AMQ-1873:
-------------------------------------

config:
<policyEntry queue=">" producerFlowControl="true" memoryLimit="5mb">

setup:
1 perl stomp producer producing into a queue,
connecting and disconnecting on every post,
rather low frequency of posts (0.5/minute)
0 consumers

behaviour:
works OK until around 68 messages are in the queue (surely depends on the size of the messages)

after that you get this in the log:
2009-10-29 20:32:05,189 | INFO | Usage Manager memory limit reached on queue://test.soccerfeeds.queue. Producers will be throttled to the rate at which messages are removed <...>

And while the activemq service is in that "throttling producers" state you will see CLOSE_WAIT sockets building up:
tcp 0 0 ::ffff:10.60.1.51:61613 ::ffff:10.60.1.206:41519 CLOSE_WAIT
tcp 0 0 ::ffff:10.60.1.51:61613 ::ffff:10.60.1.206:36141 CLOSE_WAIT
tcp 0 0 ::ffff:10.60.1.51:61613 ::ffff:10.60.1.206:45840 CLOSE_WAIT
tcp 0 0 ::ffff:10.60.1.51:61613 ::ffff:10.60.1.206:43793 CLOSE_WAIT
tcp 0 0 ::ffff:10.60.1.51:61613 ::ffff:10.60.1.206:40212 CLOSE_WAIT
tcp 0 0 ::ffff:10.60.1.51:61613 ::ffff:10.60.1.206:44060 CLOSE_WAIT
tcp 0 0 ::ffff:10.60.1.51:61613 ::ffff:10.60.1.206:43776 CLOSE_WAIT
tcp 0 0 ::ffff:10.60.1.51:61613 ::ffff:10.60.1.206:44032 CLOSE_WAIT
tcp 0 0 ::ffff:10.60.1.51:61613 ::ffff:10.60.1.206:43781 CLOSE_WAIT
tcp 0 0 ::ffff:10.60.1.51:61613 ::ffff:10.60.1.206:40200 CLOSE_WAIT
tcp 0 0 ::ffff:10.60.1.51:61613 ::ffff:10.60.1.206:44045 CLOSE_WAIT

You can watch the numbers grow with:
watch --interval=5 'netstat -an |grep tcp |grep 61613 | grep CLOSE_WAIT |wc -l'

Every post increases the number of CLOSE_WAIT sockets by 1. And the sockets will not go away, the number seems to be steadily growing.

Just consume one single message (we did it via the admin webinterface) and the number of sockets in CLOSE_WAIT drops to 0 instantly.

[root@bladedist01 activemq]# netstat -an |grep tcp |grep 61613
tcp 0 0 :::61613 :::* LISTEN

Our theory is that activemq does somehow manage to build up sockets in CLOSE_WAIT state while it is "throttling producers" mode on a given queue until, eventually, the system runs out of resources (file descriptors in this case).

> Stomp connections doesn't seem to be released cleanly (Too many open files error)
> ---------------------------------------------------------------------------------
>
> Key: AMQ-1873
> URL: https://issues.apache.org/activemq/browse/AMQ-1873
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.1.0
> Environment: Linux RHEL 4
> java version "1.5.0_15"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_15-b04)
> Java HotSpot(TM) Server VM (build 1.5.0_15-b04, mixed mode)
> Reporter: Julien Devemy
> Fix For: 5.4.0
>
> Attachments: activemq.xml
>
>
> I'm actually testing ActiveMQ with a python stomp Client (stomp.py http://www.briggs.net.nz/log/projects/stomppy/).
> I'm trying to load the broker with ~200 producers (on ~200 hosts) feeding one TOPIC for only one consumer.
> Everything is working almost fine until the client connection fails.
> The ActiveMQ log displays :
> ...
> 2008-07-30 09:28:43,665 [localhost:61613] ERROR TransportConnector - Could not accept connection : Too many open files
> 2008-07-30 09:28:43,665 [localhost:61613] DEBUG TransportConnector - Reason: Too many open files
> java.net.SocketException: Too many open files
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
> at java.net.ServerSocket.implAccept(ServerSocket.java:450)
> at java.net.ServerSocket.accept(ServerSocket.java:421)
> at org.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportServer.java:221)
> at java.lang.Thread.run(Thread.java:595)
> When I look at the open files (or connections) (with lsof) I see 65446 lines like :
> java 26765 root *066u sock 0,4 7534034 can't identify protocol
> This seems to indicate that ActiveMQ doesn't totally released the UNIX socket
> I'm attaching the activemq.xml conf file.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.