|

»

»

[jira] Created: (AMQ-1375) ActiveMQ hangs, cannot send message/create temp queue

View: New views

11 Messages — Rating Filter: Alert me

	[jira] Created: (AMQ-1375) ActiveMQ hangs, cannot send message/create temp queue by JIRA jira@apache.org :: Rate this Message: Reply to Author \| View Threaded \| Show Only this Message ActiveMQ hangs, cannot send message/create temp queue ----------------------------------------------------- Key: AMQ-1375 URL: https://issues.apache.org/activemq/browse/AMQ-1375 Project: ActiveMQ Issue Type: Bug Components: Broker Affects Versions: 4.1.1 Environment: Running an embedded broker in standalone java application based on Spring framework on Solaris 10 with JVM version 1.5.0_09. Using non persisten messaging with 512MB limit in memory manager of broker. Reporter: Trevor Higgins Attachments: jstack_basic.txt We are experiencing severe issues with AMQ blocking sending. We have a basic setup with one standalone server process and many clients. Each client has its own temp queue for P2P request/response type communication. There is also a number of topics broadcast to all clients. Our clients receive messages in an MDP fashion with each message being processed on a single threaded executor by an anonymous Runnable as follows: {code} public void onMessage(final Message message) { threadRunner.submit(new Runnable() { public void run() { try { for(final IReceiverListener listener : listeners) { listener.onReceive(message); } } catch (Throwable e) { e.printStackTrace(); } } }); } {code} Every so often, the entire AMQ blocks and we cannot send anything anymore. We also cannot create any new temp queues, which means no new client can start up (each creates its own temp queue during startup). Connections are be created and from that a session, but the session.createTemporaryQueue() call alwaysy hangs. This is similar to AMQ-1278, but as we are cannot use 5.0 until stable release....... Here is the stack of the client trying to create a temp queue at Object.wait(long, int) line: not available [native method] at CondVar(Object).wait() line: 199 at CondVar.await() line: 75 at ArrayBlockingQueue.take() line: 318 at FutureResponse.getResult() line: 42 at ResponseCorrelator.request(Object) line: 75 at ActiveMQConnection.syncSendPacket(Command) line: 1175 at ActiveMQConnection.createTempDestination(boolean) line: 1594 at ActiveMQSession.createTemporaryQueue() line: 1134 at TempDestinationResolver.resolveDestinationName(Session, String, boolean) line: 16 at SimpleMessageListenerContainer(JmsDestinationAccessor).resolveDestinationName(Session, String) line: 100 at SimpleMessageListenerContainer.createListenerConsumer(Session) line: 188 at SimpleMessageListenerContainer.doInitialize() line: 171 at SimpleMessageListenerContainer(AbstractJmsListeningContainer).initialize() line: 160 at SimpleMessageListenerContainer(AbstractJmsListeningContainer).afterPropertiesSet() line: 133 at DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).invokeInitMethods(String, Object, RootBeanDefinition) line: 1202 at DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).initializeBean(String, Object, RootBeanDefinition) line: 1172 at DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).createBean(String, RootBeanDefinition, Object[]) line: 428 at AbstractBeanFactory$1.getObject() line: 251 at DefaultListableBeanFactory(DefaultSingletonBeanRegistry).getSingleton(String, ObjectFactory) line: 156 at DefaultListableBeanFactory(AbstractBeanFactory).getBean(String, Class, Object[]) line: 248 at DefaultListableBeanFactory(AbstractBeanFactory).getBean(String) line: 160 at DefaultListableBeanFactory.preInstantiateSingletons() line: 287 at FileSystemXmlApplicationContext(AbstractApplicationContext).refresh() line: 352 at FileSystemXmlApplicationContext.<init>(String[], ApplicationContext) line: 93 at FileSystemXmlApplicationContext.<init>(String[]) line: 77 at FileSystemXmlApplicationContext.<init>(String) line: 67 at BindingService.<init>() line: 65 at BindingService.getInstance() line: 43 at TopologyServiceFactory.initialize() line: 123 at TopologyManager.initialize() line: 31 at TopologyView$3.run(IProgressMonitor) line: 159 at Worker.run() line: 55 I have attached a JStack of the server process with embedded broker also. From our client side the only thing I am dubious about is the retention of the reference to the message until the anonymous Runnable is executed. Could this cause issues is for some reason the execution blocked, will the messages build up in AMQ? From looking at JConsole for the broker, the memoryPercentageUsed is zero so it doesnt appear to be a RAM issue. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
	[jira] Updated: (AMQ-1375) ActiveMQ hangs, cannot send message/create temp queue by JIRA jira@apache.org :: Rate this Message: Reply to Author \| View Threaded \| Show Only this Message [ https://issues.apache.org/activemq/browse/AMQ-1375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Trevor Higgins updated AMQ-1375: -------------------------------- Attachment: jstack_basic.txt > ActiveMQ hangs, cannot send message/create temp queue > ----------------------------------------------------- > > Key: AMQ-1375 > URL: https://issues.apache.org/activemq/browse/AMQ-1375 > Project: ActiveMQ > Issue Type: Bug > Components: Broker > Affects Versions: 4.1.1 > Environment: Running an embedded broker in standalone java application based on Spring framework on Solaris 10 with JVM version 1.5.0_09. Using non persisten messaging with 512MB limit in memory manager of broker. > Reporter: Trevor Higgins > Attachments: jstack_basic.txt > > > We are experiencing severe issues with AMQ blocking sending. We have a basic setup with one standalone server process and many clients. Each client has its own temp queue for P2P request/response type communication. There is also a number of topics broadcast to all clients. Our clients receive messages in an MDP fashion with each message being processed on a single threaded executor by an anonymous Runnable as follows: > {code} > public void onMessage(final Message message) { > threadRunner.submit(new Runnable() { > public void run() { > try { > for(final IReceiverListener listener : listeners) { > listener.onReceive(message); > } > } catch (Throwable e) { > e.printStackTrace(); > } > } > }); > } > {code} > Every so often, the entire AMQ blocks and we cannot send anything anymore. We also cannot create any new temp queues, which means no new client can start up (each creates its own temp queue during startup). Connections are be created and from that a session, but the session.createTemporaryQueue() call alwaysy hangs. This is similar to AMQ-1278, but as we are cannot use 5.0 until stable release....... > Here is the stack of the client trying to create a temp queue > at Object.wait(long, int) line: not available [native method] > at CondVar(Object).wait() line: 199 > at CondVar.await() line: 75 > at ArrayBlockingQueue.take() line: 318 > at FutureResponse.getResult() line: 42 > at ResponseCorrelator.request(Object) line: 75 > at ActiveMQConnection.syncSendPacket(Command) line: 1175 > at ActiveMQConnection.createTempDestination(boolean) line: 1594 > at ActiveMQSession.createTemporaryQueue() line: 1134 > at TempDestinationResolver.resolveDestinationName(Session, String, boolean) line: 16 > at SimpleMessageListenerContainer(JmsDestinationAccessor).resolveDestinationName(Session, String) line: 100 > at SimpleMessageListenerContainer.createListenerConsumer(Session) line: 188 > at SimpleMessageListenerContainer.doInitialize() line: 171 > at SimpleMessageListenerContainer(AbstractJmsListeningContainer).initialize() line: 160 > at SimpleMessageListenerContainer(AbstractJmsListeningContainer).afterPropertiesSet() line: 133 > at DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).invokeInitMethods(String, Object, RootBeanDefinition) line: 1202 > at DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).initializeBean(String, Object, RootBeanDefinition) line: 1172 > at DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).createBean(String, RootBeanDefinition, Object[]) line: 428 > at AbstractBeanFactory$1.getObject() line: 251 > at DefaultListableBeanFactory(DefaultSingletonBeanRegistry).getSingleton(String, ObjectFactory) line: 156 > at DefaultListableBeanFactory(AbstractBeanFactory).getBean(String, Class, Object[]) line: 248 > at DefaultListableBeanFactory(AbstractBeanFactory).getBean(String) line: 160 > at DefaultListableBeanFactory.preInstantiateSingletons() line: 287 > at FileSystemXmlApplicationContext(AbstractApplicationContext).refresh() line: 352 > at FileSystemXmlApplicationContext.<init>(String[], ApplicationContext) line: 93 > at FileSystemXmlApplicationContext.<init>(String[]) line: 77 > at FileSystemXmlApplicationContext.<init>(String) line: 67 > at BindingService.<init>() line: 65 > at BindingService.getInstance() line: 43 > at TopologyServiceFactory.initialize() line: 123 > at TopologyManager.initialize() line: 31 > at TopologyView$3.run(IProgressMonitor) line: 159 > at Worker.run() line: 55 > I have attached a JStack of the server process with embedded broker also. > From our client side the only thing I am dubious about is the retention of the reference to the message until the anonymous Runnable is executed. Could this cause issues is for some reason the execution blocked, will the messages build up in AMQ? > From looking at JConsole for the broker, the memoryPercentageUsed is zero so it doesnt appear to be a RAM issue. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
	[jira] Updated: (AMQ-1375) ActiveMQ hangs, cannot send message/create temp queue by JIRA jira@apache.org :: Rate this Message: Reply to Author \| View Threaded \| Show Only this Message [ https://issues.apache.org/activemq/browse/AMQ-1375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Trevor Higgins updated AMQ-1375: -------------------------------- Attachment: (was: jstack_basic.txt) > ActiveMQ hangs, cannot send message/create temp queue > ----------------------------------------------------- > > Key: AMQ-1375 > URL: https://issues.apache.org/activemq/browse/AMQ-1375 > Project: ActiveMQ > Issue Type: Bug > Components: Broker > Affects Versions: 4.1.1 > Environment: Running an embedded broker in standalone java application based on Spring framework on Solaris 10 with JVM version 1.5.0_09. Using non persisten messaging with 512MB limit in memory manager of broker. > Reporter: Trevor Higgins > Attachments: jstack_basic.txt > > > We are experiencing severe issues with AMQ blocking sending. We have a basic setup with one standalone server process and many clients. Each client has its own temp queue for P2P request/response type communication. There is also a number of topics broadcast to all clients. Our clients receive messages in an MDP fashion with each message being processed on a single threaded executor by an anonymous Runnable as follows: > {code} > public void onMessage(final Message message) { > threadRunner.submit(new Runnable() { > public void run() { > try { > for(final IReceiverListener listener : listeners) { > listener.onReceive(message); > } > } catch (Throwable e) { > e.printStackTrace(); > } > } > }); > } > {code} > Every so often, the entire AMQ blocks and we cannot send anything anymore. We also cannot create any new temp queues, which means no new client can start up (each creates its own temp queue during startup). Connections are be created and from that a session, but the session.createTemporaryQueue() call alwaysy hangs. This is similar to AMQ-1278, but as we are cannot use 5.0 until stable release....... > Here is the stack of the client trying to create a temp queue > at Object.wait(long, int) line: not available [native method] > at CondVar(Object).wait() line: 199 > at CondVar.await() line: 75 > at ArrayBlockingQueue.take() line: 318 > at FutureResponse.getResult() line: 42 > at ResponseCorrelator.request(Object) line: 75 > at ActiveMQConnection.syncSendPacket(Command) line: 1175 > at ActiveMQConnection.createTempDestination(boolean) line: 1594 > at ActiveMQSession.createTemporaryQueue() line: 1134 > at TempDestinationResolver.resolveDestinationName(Session, String, boolean) line: 16 > at SimpleMessageListenerContainer(JmsDestinationAccessor).resolveDestinationName(Session, String) line: 100 > at SimpleMessageListenerContainer.createListenerConsumer(Session) line: 188 > at SimpleMessageListenerContainer.doInitialize() line: 171 > at SimpleMessageListenerContainer(AbstractJmsListeningContainer).initialize() line: 160 > at SimpleMessageListenerContainer(AbstractJmsListeningContainer).afterPropertiesSet() line: 133 > at DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).invokeInitMethods(String, Object, RootBeanDefinition) line: 1202 > at DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).initializeBean(String, Object, RootBeanDefinition) line: 1172 > at DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).createBean(String, RootBeanDefinition, Object[]) line: 428 > at AbstractBeanFactory$1.getObject() line: 251 > at DefaultListableBeanFactory(DefaultSingletonBeanRegistry).getSingleton(String, ObjectFactory) line: 156 > at DefaultListableBeanFactory(AbstractBeanFactory).getBean(String, Class, Object[]) line: 248 > at DefaultListableBeanFactory(AbstractBeanFactory).getBean(String) line: 160 > at DefaultListableBeanFactory.preInstantiateSingletons() line: 287 > at FileSystemXmlApplicationContext(AbstractApplicationContext).refresh() line: 352 > at FileSystemXmlApplicationContext.<init>(String[], ApplicationContext) line: 93 > at FileSystemXmlApplicationContext.<init>(String[]) line: 77 > at FileSystemXmlApplicationContext.<init>(String) line: 67 > at BindingService.<init>() line: 65 > at BindingService.getInstance() line: 43 > at TopologyServiceFactory.initialize() line: 123 > at TopologyManager.initialize() line: 31 > at TopologyView$3.run(IProgressMonitor) line: 159 > at Worker.run() line: 55 > I have attached a JStack of the server process with embedded broker also. > From our client side the only thing I am dubious about is the retention of the reference to the message until the anonymous Runnable is executed. Could this cause issues is for some reason the execution blocked, will the messages build up in AMQ? > From looking at JConsole for the broker, the memoryPercentageUsed is zero so it doesnt appear to be a RAM issue. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
	[jira] Updated: (AMQ-1375) ActiveMQ hangs, cannot send message/create temp queue by JIRA jira@apache.org :: Rate this Message: Reply to Author \| View Threaded \| Show Only this Message [ https://issues.apache.org/activemq/browse/AMQ-1375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Trevor Higgins updated AMQ-1375: -------------------------------- Attachment: (was: jstack_basic.txt) > ActiveMQ hangs, cannot send message/create temp queue > ----------------------------------------------------- > > Key: AMQ-1375 > URL: https://issues.apache.org/activemq/browse/AMQ-1375 > Project: ActiveMQ > Issue Type: Bug > Components: Broker > Affects Versions: 4.1.1 > Environment: Running an embedded broker in standalone java application based on Spring framework on Solaris 10 with JVM version 1.5.0_09. Using non persisten messaging with 512MB limit in memory manager of broker. > Reporter: Trevor Higgins > Attachments: jstack_basic.txt > > > We are experiencing severe issues with AMQ blocking sending. We have a basic setup with one standalone server process and many clients. Each client has its own temp queue for P2P request/response type communication. There is also a number of topics broadcast to all clients. Our clients receive messages in an MDP fashion with each message being processed on a single threaded executor by an anonymous Runnable as follows: > {code} > public void onMessage(final Message message) { > threadRunner.submit(new Runnable() { > public void run() { > try { > for(final IReceiverListener listener : listeners) { > listener.onReceive(message); > } > } catch (Throwable e) { > e.printStackTrace(); > } > } > }); > } > {code} > Every so often, the entire AMQ blocks and we cannot send anything anymore. We also cannot create any new temp queues, which means no new client can start up (each creates its own temp queue during startup). Connections are be created and from that a session, but the session.createTemporaryQueue() call alwaysy hangs. This is similar to AMQ-1278, but as we are cannot use 5.0 until stable release....... > Here is the stack of the client trying to create a temp queue > at Object.wait(long, int) line: not available [native method] > at CondVar(Object).wait() line: 199 > at CondVar.await() line: 75 > at ArrayBlockingQueue.take() line: 318 > at FutureResponse.getResult() line: 42 > at ResponseCorrelator.request(Object) line: 75 > at ActiveMQConnection.syncSendPacket(Command) line: 1175 > at ActiveMQConnection.createTempDestination(boolean) line: 1594 > at ActiveMQSession.createTemporaryQueue() line: 1134 > at TempDestinationResolver.resolveDestinationName(Session, String, boolean) line: 16 > at SimpleMessageListenerContainer(JmsDestinationAccessor).resolveDestinationName(Session, String) line: 100 > at SimpleMessageListenerContainer.createListenerConsumer(Session) line: 188 > at SimpleMessageListenerContainer.doInitialize() line: 171 > at SimpleMessageListenerContainer(AbstractJmsListeningContainer).initialize() line: 160 > at SimpleMessageListenerContainer(AbstractJmsListeningContainer).afterPropertiesSet() line: 133 > at DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).invokeInitMethods(String, Object, RootBeanDefinition) line: 1202 > at DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).initializeBean(String, Object, RootBeanDefinition) line: 1172 > at DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).createBean(String, RootBeanDefinition, Object[]) line: 428 > at AbstractBeanFactory$1.getObject() line: 251 > at DefaultListableBeanFactory(DefaultSingletonBeanRegistry).getSingleton(String, ObjectFactory) line: 156 > at DefaultListableBeanFactory(AbstractBeanFactory).getBean(String, Class, Object[]) line: 248 > at DefaultListableBeanFactory(AbstractBeanFactory).getBean(String) line: 160 > at DefaultListableBeanFactory.preInstantiateSingletons() line: 287 > at FileSystemXmlApplicationContext(AbstractApplicationContext).refresh() line: 352 > at FileSystemXmlApplicationContext.<init>(String[], ApplicationContext) line: 93 > at FileSystemXmlApplicationContext.<init>(String[]) line: 77 > at FileSystemXmlApplicationContext.<init>(String) line: 67 > at BindingService.<init>() line: 65 > at BindingService.getInstance() line: 43 > at TopologyServiceFactory.initialize() line: 123 > at TopologyManager.initialize() line: 31 > at TopologyView$3.run(IProgressMonitor) line: 159 > at Worker.run() line: 55 > I have attached a JStack of the server process with embedded broker also. > From our client side the only thing I am dubious about is the retention of the reference to the message until the anonymous Runnable is executed. Could this cause issues is for some reason the execution blocked, will the messages build up in AMQ? > From looking at JConsole for the broker, the memoryPercentageUsed is zero so it doesnt appear to be a RAM issue. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
	[jira] Updated: (AMQ-1375) ActiveMQ hangs, cannot send message/create temp queue by JIRA jira@apache.org :: Rate this Message: Reply to Author \| View Threaded \| Show Only this Message [ https://issues.apache.org/activemq/browse/AMQ-1375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Trevor Higgins updated AMQ-1375: -------------------------------- Attachment: jstack_basic.txt > ActiveMQ hangs, cannot send message/create temp queue > ----------------------------------------------------- > > Key: AMQ-1375 > URL: https://issues.apache.org/activemq/browse/AMQ-1375 > Project: ActiveMQ > Issue Type: Bug > Components: Broker > Affects Versions: 4.1.1 > Environment: Running an embedded broker in standalone java application based on Spring framework on Solaris 10 with JVM version 1.5.0_09. Using non persisten messaging with 512MB limit in memory manager of broker. > Reporter: Trevor Higgins > Attachments: jstack_basic.txt > > > We are experiencing severe issues with AMQ blocking sending. We have a basic setup with one standalone server process and many clients. Each client has its own temp queue for P2P request/response type communication. There is also a number of topics broadcast to all clients. Our clients receive messages in an MDP fashion with each message being processed on a single threaded executor by an anonymous Runnable as follows: > {code} > public void onMessage(final Message message) { > threadRunner.submit(new Runnable() { > public void run() { > try { > for(final IReceiverListener listener : listeners) { > listener.onReceive(message); > } > } catch (Throwable e) { > e.printStackTrace(); > } > } > }); > } > {code} > Every so often, the entire AMQ blocks and we cannot send anything anymore. We also cannot create any new temp queues, which means no new client can start up (each creates its own temp queue during startup). Connections are be created and from that a session, but the session.createTemporaryQueue() call alwaysy hangs. This is similar to AMQ-1278, but as we are cannot use 5.0 until stable release....... > Here is the stack of the client trying to create a temp queue > at Object.wait(long, int) line: not available [native method] > at CondVar(Object).wait() line: 199 > at CondVar.await() line: 75 > at ArrayBlockingQueue.take() line: 318 > at FutureResponse.getResult() line: 42 > at ResponseCorrelator.request(Object) line: 75 > at ActiveMQConnection.syncSendPacket(Command) line: 1175 > at ActiveMQConnection.createTempDestination(boolean) line: 1594 > at ActiveMQSession.createTemporaryQueue() line: 1134 > at TempDestinationResolver.resolveDestinationName(Session, String, boolean) line: 16 > at SimpleMessageListenerContainer(JmsDestinationAccessor).resolveDestinationName(Session, String) line: 100 > at SimpleMessageListenerContainer.createListenerConsumer(Session) line: 188 > at SimpleMessageListenerContainer.doInitialize() line: 171 > at SimpleMessageListenerContainer(AbstractJmsListeningContainer).initialize() line: 160 > at SimpleMessageListenerContainer(AbstractJmsListeningContainer).afterPropertiesSet() line: 133 > at DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).invokeInitMethods(String, Object, RootBeanDefinition) line: 1202 > at DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).initializeBean(String, Object, RootBeanDefinition) line: 1172 > at DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).createBean(String, RootBeanDefinition, Object[]) line: 428 > at AbstractBeanFactory$1.getObject() line: 251 > at DefaultListableBeanFactory(DefaultSingletonBeanRegistry).getSingleton(String, ObjectFactory) line: 156 > at DefaultListableBeanFactory(AbstractBeanFactory).getBean(String, Class, Object[]) line: 248 > at DefaultListableBeanFactory(AbstractBeanFactory).getBean(String) line: 160 > at DefaultListableBeanFactory.preInstantiateSingletons() line: 287 > at FileSystemXmlApplicationContext(AbstractApplicationContext).refresh() line: 352 > at FileSystemXmlApplicationContext.<init>(String[], ApplicationContext) line: 93 > at FileSystemXmlApplicationContext.<init>(String[]) line: 77 > at FileSystemXmlApplicationContext.<init>(String) line: 67 > at BindingService.<init>() line: 65 > at BindingService.getInstance() line: 43 > at TopologyServiceFactory.initialize() line: 123 > at TopologyManager.initialize() line: 31 > at TopologyView$3.run(IProgressMonitor) line: 159 > at Worker.run() line: 55 > I have attached a JStack of the server process with embedded broker also. > From our client side the only thing I am dubious about is the retention of the reference to the message until the anonymous Runnable is executed. Could this cause issues is for some reason the execution blocked, will the messages build up in AMQ? > From looking at JConsole for the broker, the memoryPercentageUsed is zero so it doesnt appear to be a RAM issue. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
	[jira] Commented: (AMQ-1375) ActiveMQ hangs, cannot send message/create temp queue by JIRA jira@apache.org :: Rate this Message: Reply to Author \| View Threaded \| Show Only this Message [ https://issues.apache.org/activemq/browse/AMQ-1375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_40061 ] James Strachan commented on AMQ-1375: ------------------------------------- The stack trace just implies that the current JMS connection is blocked; I suspect its due to a producer being blocked? Are you sharing the connection in spring's listener container stuff? http://activemq.apache.org/my-producer-blocks.html I wonder - do you have any way of reproducing this? e.g. a JUnit test case? Also can you reproduce this on 5.0; which has much more flexible flow control which avoids blocking the entire connection if a single producer is blocked? > ActiveMQ hangs, cannot send message/create temp queue > ----------------------------------------------------- > > Key: AMQ-1375 > URL: https://issues.apache.org/activemq/browse/AMQ-1375 > Project: ActiveMQ > Issue Type: Bug > Components: Broker > Affects Versions: 4.1.1 > Environment: Running an embedded broker in standalone java application based on Spring framework on Solaris 10 with JVM version 1.5.0_09. Using non persisten messaging with 512MB limit in memory manager of broker. > Reporter: Trevor Higgins > Attachments: jstack_basic.txt > > > We are experiencing severe issues with AMQ blocking sending. We have a basic setup with one standalone server process and many clients. Each client has its own temp queue for P2P request/response type communication. There is also a number of topics broadcast to all clients. Our clients receive messages in an MDP fashion with each message being processed on a single threaded executor by an anonymous Runnable as follows: > {code} > public void onMessage(final Message message) { > threadRunner.submit(new Runnable() { > public void run() { > try { > for(final IReceiverListener listener : listeners) { > listener.onReceive(message); > } > } catch (Throwable e) { > e.printStackTrace(); > } > } > }); > } > {code} > Every so often, the entire AMQ blocks and we cannot send anything anymore. We also cannot create any new temp queues, which means no new client can start up (each creates its own temp queue during startup). Connections are be created and from that a session, but the session.createTemporaryQueue() call alwaysy hangs. This is similar to AMQ-1278, but as we are cannot use 5.0 until stable release....... > Here is the stack of the client trying to create a temp queue > at Object.wait(long, int) line: not available [native method] > at CondVar(Object).wait() line: 199 > at CondVar.await() line: 75 > at ArrayBlockingQueue.take() line: 318 > at FutureResponse.getResult() line: 42 > at ResponseCorrelator.request(Object) line: 75 > at ActiveMQConnection.syncSendPacket(Command) line: 1175 > at ActiveMQConnection.createTempDestination(boolean) line: 1594 > at ActiveMQSession.createTemporaryQueue() line: 1134 > at TempDestinationResolver.resolveDestinationName(Session, String, boolean) line: 16 > at SimpleMessageListenerContainer(JmsDestinationAccessor).resolveDestinationName(Session, String) line: 100 > at SimpleMessageListenerContainer.createListenerConsumer(Session) line: 188 > at SimpleMessageListenerContainer.doInitialize() line: 171 > at SimpleMessageListenerContainer(AbstractJmsListeningContainer).initialize() line: 160 > at SimpleMessageListenerContainer(AbstractJmsListeningContainer).afterPropertiesSet() line: 133 > at DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).invokeInitMethods(String, Object, RootBeanDefinition) line: 1202 > at DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).initializeBean(String, Object, RootBeanDefinition) line: 1172 > at DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).createBean(String, RootBeanDefinition, Object[]) line: 428 > at AbstractBeanFactory$1.getObject() line: 251 > at DefaultListableBeanFactory(DefaultSingletonBeanRegistry).getSingleton(String, ObjectFactory) line: 156 > at DefaultListableBeanFactory(AbstractBeanFactory).getBean(String, Class, Object[]) line: 248 > at DefaultListableBeanFactory(AbstractBeanFactory).getBean(String) line: 160 > at DefaultListableBeanFactory.preInstantiateSingletons() line: 287 > at FileSystemXmlApplicationContext(AbstractApplicationContext).refresh() line: 352 > at FileSystemXmlApplicationContext.<init>(String[], ApplicationContext) line: 93 > at FileSystemXmlApplicationContext.<init>(String[]) line: 77 > at FileSystemXmlApplicationContext.<init>(String) line: 67 > at BindingService.<init>() line: 65 > at BindingService.getInstance() line: 43 > at TopologyServiceFactory.initialize() line: 123 > at TopologyManager.initialize() line: 31 > at TopologyView$3.run(IProgressMonitor) line: 159 > at Worker.run() line: 55 > I have attached a JStack of the server process with embedded broker also. > From our client side the only thing I am dubious about is the retention of the reference to the message until the anonymous Runnable is executed. Could this cause issues is for some reason the execution blocked, will the messages build up in AMQ? > From looking at JConsole for the broker, the memoryPercentageUsed is zero so it doesnt appear to be a RAM issue. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
	[jira] Commented: (AMQ-1375) ActiveMQ hangs, cannot send message/create temp queue by JIRA jira@apache.org :: Rate this Message: Reply to Author \| View Threaded \| Show Only this Message [ https://issues.apache.org/activemq/browse/AMQ-1375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_40075 ] Trevor Higgins commented on AMQ-1375: ------------------------------------- Can you explain why a producer would be blocked if all the client does when handling the message is add it to the executor in a Runnable? The reason for doing this was specifically to avoid taking time in the handling the message to prevent blocking the producer.... There is a session & producer created for each client request in a thread local fashion, but only one connection is created on server side. We dont share connection between client and server. Server has one connection only, created using connectionFactory.createConnection(); where connectionFactory is defined as follows in a server side Spring config file: <bean id="connectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory" depends-on="broker"> <property name="brokerURL" value="tcp://localhost:50057"/> </bean> Each client listener is defined as follows in a separate client XML file: <bean id="destinationResolver" class="com.xxx.yyy.platform.core.transport.activemq.TempDestinationResolver"/> <bean id="connectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory"> <property name="brokerURL" value="tcp://${jms.server}:${jms.port}"/> </bean> <bean id="listenerContainer" class="org.springframework.jms.listener.SimpleMessageListenerContainer" destroy-method="destroy"> <property name="concurrentConsumers" value="1"/> <property name="connectionFactory" ref="connectionFactory" /> <property name="messageListener" ref="messageListener" /> <property name="destinationName" value="destination"/> <property name="destinationResolver" ref="destinationResolver"/> </bean> <bean id="messageListener" class="com.xxx.yyy.platform.core.transport.activemq.MessageDispatcher" destroy-method="dispose"> <property name="destinationFactory" ref="destinationResolver"/> <property name="messageConverter"> <bean class="org.springframework.jms.support.converter.SimpleMessageConverter"/> </property> </bean> Hence the connections are separate entities created by the ActiveMQConnectionFactory. Im am trying to reproduce this without our GUI/server stuff, just a simple AMQ Junit............ > ActiveMQ hangs, cannot send message/create temp queue > ----------------------------------------------------- > > Key: AMQ-1375 > URL: https://issues.apache.org/activemq/browse/AMQ-1375 > Project: ActiveMQ > Issue Type: Bug > Components: Broker > Affects Versions: 4.1.1 > Environment: Running an embedded broker in standalone java application based on Spring framework on Solaris 10 with JVM version 1.5.0_09. Using non persisten messaging with 512MB limit in memory manager of broker. > Reporter: Trevor Higgins > Attachments: jstack_basic.txt > > > We are experiencing severe issues with AMQ blocking sending. We have a basic setup with one standalone server process and many clients. Each client has its own temp queue for P2P request/response type communication. There is also a number of topics broadcast to all clients. Our clients receive messages in an MDP fashion with each message being processed on a single threaded executor by an anonymous Runnable as follows: > {code} > public void onMessage(final Message message) { > threadRunner.submit(new Runnable() { > public void run() { > try { > for(final IReceiverListener listener : listeners) { > listener.onReceive(message); > } > } catch (Throwable e) { > e.printStackTrace(); > } > } > }); > } > {code} > Every so often, the entire AMQ blocks and we cannot send anything anymore. We also cannot create any new temp queues, which means no new client can start up (each creates its own temp queue during startup). Connections are be created and from that a session, but the session.createTemporaryQueue() call alwaysy hangs. This is similar to AMQ-1278, but as we are cannot use 5.0 until stable release....... > Here is the stack of the client trying to create a temp queue > at Object.wait(long, int) line: not available [native method] > at CondVar(Object).wait() line: 199 > at CondVar.await() line: 75 > at ArrayBlockingQueue.take() line: 318 > at FutureResponse.getResult() line: 42 > at ResponseCorrelator.request(Object) line: 75 > at ActiveMQConnection.syncSendPacket(Command) line: 1175 > at ActiveMQConnection.createTempDestination(boolean) line: 1594 > at ActiveMQSession.createTemporaryQueue() line: 1134 > at TempDestinationResolver.resolveDestinationName(Session, String, boolean) line: 16 > at SimpleMessageListenerContainer(JmsDestinationAccessor).resolveDestinationName(Session, String) line: 100 > at SimpleMessageListenerContainer.createListenerConsumer(Session) line: 188 > at SimpleMessageListenerContainer.doInitialize() line: 171 > at SimpleMessageListenerContainer(AbstractJmsListeningContainer).initialize() line: 160 > at SimpleMessageListenerContainer(AbstractJmsListeningContainer).afterPropertiesSet() line: 133 > at DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).invokeInitMethods(String, Object, RootBeanDefinition) line: 1202 > at DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).initializeBean(String, Object, RootBeanDefinition) line: 1172 > at DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).createBean(String, RootBeanDefinition, Object[]) line: 428 > at AbstractBeanFactory$1.getObject() line: 251 > at DefaultListableBeanFactory(DefaultSingletonBeanRegistry).getSingleton(String, ObjectFactory) line: 156 > at DefaultListableBeanFactory(AbstractBeanFactory).getBean(String, Class, Object[]) line: 248 > at DefaultListableBeanFactory(AbstractBeanFactory).getBean(String) line: 160 > at DefaultListableBeanFactory.preInstantiateSingletons() line: 287 > at FileSystemXmlApplicationContext(AbstractApplicationContext).refresh() line: 352 > at FileSystemXmlApplicationContext.<init>(String[], ApplicationContext) line: 93 > at FileSystemXmlApplicationContext.<init>(String[]) line: 77 > at FileSystemXmlApplicationContext.<init>(String) line: 67 > at BindingService.<init>() line: 65 > at BindingService.getInstance() line: 43 > at TopologyServiceFactory.initialize() line: 123 > at TopologyManager.initialize() line: 31 > at TopologyView$3.run(IProgressMonitor) line: 159 > at Worker.run() line: 55 > I have attached a JStack of the server process with embedded broker also. > From our client side the only thing I am dubious about is the retention of the reference to the message until the anonymous Runnable is executed. Could this cause issues is for some reason the execution blocked, will the messages build up in AMQ? > From looking at JConsole for the broker, the memoryPercentageUsed is zero so it doesnt appear to be a RAM issue. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
	[jira] Commented: (AMQ-1375) ActiveMQ hangs, cannot send message/create temp queue by JIRA jira@apache.org :: Rate this Message: Reply to Author \| View Threaded \| Show Only this Message [ https://issues.apache.org/activemq/browse/AMQ-1375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=51809#action_51809 ] Vadim Pesochinskiy commented on AMQ-1375: ----------------------------------------- I found similar stack traces in following issues like AMQ-768 and AMQ-2147. I have the same issue in production now with syncSendPacket(Command) blocking forever. I am thinking that you would have a problem anywhere this method is called, because result may never come back. In AMQ-768 it does not come back because of connection reset or a server side error, in AMQ-2147 it does not come back because of memory issues on the client side. Here is my stack trace (amq v5.1.0), which is identical to the one listed above, but is parseable by IDEA. java.lang.Thread.State: WAITING on edu.emory.mathcs.backport.java.util.concurrent.locks.CondVar@15ee98b5 at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at edu.emory.mathcs.backport.java.util.concurrent.locks.CondVar.await(CondVar.java:75) at edu.emory.mathcs.backport.java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:318) at org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:42) at org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:75) at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1175) at org.apache.activemq.ActiveMQConnection.createTempDestination(ActiveMQConnection.java:1600) at org.apache.activemq.ActiveMQSession.createTemporaryQueue(ActiveMQSession.java:1134) at org.logicblaze.lingo.jms.impl.SingleThreadedRequestor.createTemporaryDestination(SingleThreadedRequestor.java:109) at org.logicblaze.lingo.jms.impl.SingleThreadedRequestor.<init>(SingleThreadedRequestor.java:56) at org.logicblaze.lingo.jms.impl.MultiplexingRequestor.<init>(MultiplexingRequestor.java:64) at org.logicblaze.lingo.jms.impl.MultiplexingRequestor.<init>(MultiplexingRequestor.java:69) at org.logicblaze.lingo.jms.impl.MultiplexingRequestor.newInstance(MultiplexingRequestor.java:55) > ActiveMQ hangs, cannot send message/create temp queue > ----------------------------------------------------- > > Key: AMQ-1375 > URL: https://issues.apache.org/activemq/browse/AMQ-1375 > Project: ActiveMQ > Issue Type: Bug > Components: Broker > Affects Versions: 4.1.1 > Environment: Running an embedded broker in standalone java application based on Spring framework on Solaris 10 with JVM version 1.5.0_09. Using non persisten messaging with 512MB limit in memory manager of broker. > Reporter: Trevor Higgins > Fix For: 5.4.0 > > Attachments: jstack_basic.txt > > > We are experiencing severe issues with AMQ blocking sending. We have a basic setup with one standalone server process and many clients. Each client has its own temp queue for P2P request/response type communication. There is also a number of topics broadcast to all clients. Our clients receive messages in an MDP fashion with each message being processed on a single threaded executor by an anonymous Runnable as follows: > {code} > public void onMessage(final Message message) { > threadRunner.submit(new Runnable() { > public void run() { > try { > for(final IReceiverListener listener : listeners) { > listener.onReceive(message); > } > } catch (Throwable e) { > e.printStackTrace(); > } > } > }); > } > {code} > Every so often, the entire AMQ blocks and we cannot send anything anymore. We also cannot create any new temp queues, which means no new client can start up (each creates its own temp queue during startup). Connections are be created and from that a session, but the session.createTemporaryQueue() call alwaysy hangs. This is similar to AMQ-1278, but as we are cannot use 5.0 until stable release....... > Here is the stack of the client trying to create a temp queue > at Object.wait(long, int) line: not available [native method] > at CondVar(Object).wait() line: 199 > at CondVar.await() line: 75 > at ArrayBlockingQueue.take() line: 318 > at FutureResponse.getResult() line: 42 > at ResponseCorrelator.request(Object) line: 75 > at ActiveMQConnection.syncSendPacket(Command) line: 1175 > at ActiveMQConnection.createTempDestination(boolean) line: 1594 > at ActiveMQSession.createTemporaryQueue() line: 1134 > at TempDestinationResolver.resolveDestinationName(Session, String, boolean) line: 16 > at SimpleMessageListenerContainer(JmsDestinationAccessor).resolveDestinationName(Session, String) line: 100 > at SimpleMessageListenerContainer.createListenerConsumer(Session) line: 188 > at SimpleMessageListenerContainer.doInitialize() line: 171 > at SimpleMessageListenerContainer(AbstractJmsListeningContainer).initialize() line: 160 > at SimpleMessageListenerContainer(AbstractJmsListeningContainer).afterPropertiesSet() line: 133 > at DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).invokeInitMethods(String, Object, RootBeanDefinition) line: 1202 > at DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).initializeBean(String, Object, RootBeanDefinition) line: 1172 > at DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).createBean(String, RootBeanDefinition, Object[]) line: 428 > at AbstractBeanFactory$1.getObject() line: 251 > at DefaultListableBeanFactory(DefaultSingletonBeanRegistry).getSingleton(String, ObjectFactory) line: 156 > at DefaultListableBeanFactory(AbstractBeanFactory).getBean(String, Class, Object[]) line: 248 > at DefaultListableBeanFactory(AbstractBeanFactory).getBean(String) line: 160 > at DefaultListableBeanFactory.preInstantiateSingletons() line: 287 > at FileSystemXmlApplicationContext(AbstractApplicationContext).refresh() line: 352 > at FileSystemXmlApplicationContext.<init>(String[], ApplicationContext) line: 93 > at FileSystemXmlApplicationContext.<init>(String[]) line: 77 > at FileSystemXmlApplicationContext.<init>(String) line: 67 > at BindingService.<init>() line: 65 > at BindingService.getInstance() line: 43 > at TopologyServiceFactory.initialize() line: 123 > at TopologyManager.initialize() line: 31 > at TopologyView$3.run(IProgressMonitor) line: 159 > at Worker.run() line: 55 > I have attached a JStack of the server process with embedded broker also. > From our client side the only thing I am dubious about is the retention of the reference to the message until the anonymous Runnable is executed. Could this cause issues is for some reason the execution blocked, will the messages build up in AMQ? > From looking at JConsole for the broker, the memoryPercentageUsed is zero so it doesnt appear to be a RAM issue. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
	[jira] Commented: (AMQ-1375) ActiveMQ hangs, cannot send message/create temp queue by JIRA jira@apache.org :: Rate this Message: Reply to Author \| View Threaded \| Show Only this Message [ https://issues.apache.org/activemq/browse/AMQ-1375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=52625#action_52625 ] Dan Tran commented on AMQ-1375: ------------------------------- I have a similiar case with embedded AMQ 5.2 + tomcat 6.x + java 6, where my embedded jms client ( via vm:// ...) send operation hangs when the broker is heavily processing messages from an another external jms client ( via tcp:// ) . We are using non persistent messages. Any advice to work around this issue is greatly appreciated. here is the stack trace Daemon Thread [Thread-14] (Suspended) Unsafe.park(boolean, long) line: not available [native method] LockSupport.park(Object) line: 158 AbstractQueuedSynchronizer$ConditionObject.await() line: 1925 ArrayBlockingQueue<E>.take() line: 317 FutureResponse.getResult() line: 40 ResponseCorrelator.request(Object) line: 80 ActiveMQConnection.syncSendPacket(Command) line: 1233 ActiveMQSession.send(ActiveMQMessageProducer, ActiveMQDestination, Message, int, int, long, MemoryUsage, int) line: 1659 ActiveMQMessageProducer.send(Destination, Message, int, int, long) line: 227 ActiveMQMessageProducer(ActiveMQMessageProducerSupport).send(Message) line: 241 JmsTemplate.doSend(MessageProducer, Message) line: 597 JmsTemplate.doSend(Session, Destination, MessageCreator) line: 574 JmsTemplate$4.doInJms(Session) line: 551 JmsTemplate.execute(SessionCallback, boolean) line: 471 JmsTemplate.send(String, MessageCreator) line: 548 OracleSGAAgentProxy(AbstractAsyncAgentProxy).sendMessageToAgent(String) line: 92 OracleSGAAgentProxy(AbstractAsyncAgentProxy).pingAgent() line: 147 OracleSGAAgentProxy(AbstractAsyncAgentProxy).run() line: 173 Thread.run() line: 619 > ActiveMQ hangs, cannot send message/create temp queue > ----------------------------------------------------- > > Key: AMQ-1375 > URL: https://issues.apache.org/activemq/browse/AMQ-1375 > Project: ActiveMQ > Issue Type: Bug > Components: Broker > Affects Versions: 4.1.1 > Environment: Running an embedded broker in standalone java application based on Spring framework on Solaris 10 with JVM version 1.5.0_09. Using non persisten messaging with 512MB limit in memory manager of broker. > Reporter: Trevor Higgins > Fix For: 5.4.0 > > Attachments: jstack_basic.txt > > > We are experiencing severe issues with AMQ blocking sending. We have a basic setup with one standalone server process and many clients. Each client has its own temp queue for P2P request/response type communication. There is also a number of topics broadcast to all clients. Our clients receive messages in an MDP fashion with each message being processed on a single threaded executor by an anonymous Runnable as follows: > {code} > public void onMessage(final Message message) { > threadRunner.submit(new Runnable() { > public void run() { > try { > for(final IReceiverListener listener : listeners) { > listener.onReceive(message); > } > } catch (Throwable e) { > e.printStackTrace(); > } > } > }); > } > {code} > Every so often, the entire AMQ blocks and we cannot send anything anymore. We also cannot create any new temp queues, which means no new client can start up (each creates its own temp queue during startup). Connections are be created and from that a session, but the session.createTemporaryQueue() call alwaysy hangs. This is similar to AMQ-1278, but as we are cannot use 5.0 until stable release....... > Here is the stack of the client trying to create a temp queue > at Object.wait(long, int) line: not available [native method] > at CondVar(Object).wait() line: 199 > at CondVar.await() line: 75 > at ArrayBlockingQueue.take() line: 318 > at FutureResponse.getResult() line: 42 > at ResponseCorrelator.request(Object) line: 75 > at ActiveMQConnection.syncSendPacket(Command) line: 1175 > at ActiveMQConnection.createTempDestination(boolean) line: 1594 > at ActiveMQSession.createTemporaryQueue() line: 1134 > at TempDestinationResolver.resolveDestinationName(Session, String, boolean) line: 16 > at SimpleMessageListenerContainer(JmsDestinationAccessor).resolveDestinationName(Session, String) line: 100 > at SimpleMessageListenerContainer.createListenerConsumer(Session) line: 188 > at SimpleMessageListenerContainer.doInitialize() line: 171 > at SimpleMessageListenerContainer(AbstractJmsListeningContainer).initialize() line: 160 > at SimpleMessageListenerContainer(AbstractJmsListeningContainer).afterPropertiesSet() line: 133 > at DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).invokeInitMethods(String, Object, RootBeanDefinition) line: 1202 > at DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).initializeBean(String, Object, RootBeanDefinition) line: 1172 > at DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).createBean(String, RootBeanDefinition, Object[]) line: 428 > at AbstractBeanFactory$1.getObject() line: 251 > at DefaultListableBeanFactory(DefaultSingletonBeanRegistry).getSingleton(String, ObjectFactory) line: 156 > at DefaultListableBeanFactory(AbstractBeanFactory).getBean(String, Class, Object[]) line: 248 > at DefaultListableBeanFactory(AbstractBeanFactory).getBean(String) line: 160 > at DefaultListableBeanFactory.preInstantiateSingletons() line: 287 > at FileSystemXmlApplicationContext(AbstractApplicationContext).refresh() line: 352 > at FileSystemXmlApplicationContext.<init>(String[], ApplicationContext) line: 93 > at FileSystemXmlApplicationContext.<init>(String[]) line: 77 > at FileSystemXmlApplicationContext.<init>(String) line: 67 > at BindingService.<init>() line: 65 > at BindingService.getInstance() line: 43 > at TopologyServiceFactory.initialize() line: 123 > at TopologyManager.initialize() line: 31 > at TopologyView$3.run(IProgressMonitor) line: 159 > at Worker.run() line: 55 > I have attached a JStack of the server process with embedded broker also. > From our client side the only thing I am dubious about is the retention of the reference to the message until the anonymous Runnable is executed. Could this cause issues is for some reason the execution blocked, will the messages build up in AMQ? > From looking at JConsole for the broker, the memoryPercentageUsed is zero so it doesnt appear to be a RAM issue. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
	[jira] Commented: (AMQ-1375) ActiveMQ hangs, cannot send message/create temp queue by JIRA jira@apache.org :: Rate this Message: Reply to Author \| View Threaded \| Show Only this Message [ https://issues.apache.org/activemq/browse/AMQ-1375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=52626#action_52626 ] Dan Tran commented on AMQ-1375: ------------------------------- and it hangs after i stop the external client and drain all of its messages > ActiveMQ hangs, cannot send message/create temp queue > ----------------------------------------------------- > > Key: AMQ-1375 > URL: https://issues.apache.org/activemq/browse/AMQ-1375 > Project: ActiveMQ > Issue Type: Bug > Components: Broker > Affects Versions: 4.1.1 > Environment: Running an embedded broker in standalone java application based on Spring framework on Solaris 10 with JVM version 1.5.0_09. Using non persisten messaging with 512MB limit in memory manager of broker. > Reporter: Trevor Higgins > Fix For: 5.4.0 > > Attachments: jstack_basic.txt > > > We are experiencing severe issues with AMQ blocking sending. We have a basic setup with one standalone server process and many clients. Each client has its own temp queue for P2P request/response type communication. There is also a number of topics broadcast to all clients. Our clients receive messages in an MDP fashion with each message being processed on a single threaded executor by an anonymous Runnable as follows: > {code} > public void onMessage(final Message message) { > threadRunner.submit(new Runnable() { > public void run() { > try { > for(final IReceiverListener listener : listeners) { > listener.onReceive(message); > } > } catch (Throwable e) { > e.printStackTrace(); > } > } > }); > } > {code} > Every so often, the entire AMQ blocks and we cannot send anything anymore. We also cannot create any new temp queues, which means no new client can start up (each creates its own temp queue during startup). Connections are be created and from that a session, but the session.createTemporaryQueue() call alwaysy hangs. This is similar to AMQ-1278, but as we are cannot use 5.0 until stable release....... > Here is the stack of the client trying to create a temp queue > at Object.wait(long, int) line: not available [native method] > at CondVar(Object).wait() line: 199 > at CondVar.await() line: 75 > at ArrayBlockingQueue.take() line: 318 > at FutureResponse.getResult() line: 42 > at ResponseCorrelator.request(Object) line: 75 > at ActiveMQConnection.syncSendPacket(Command) line: 1175 > at ActiveMQConnection.createTempDestination(boolean) line: 1594 > at ActiveMQSession.createTemporaryQueue() line: 1134 > at TempDestinationResolver.resolveDestinationName(Session, String, boolean) line: 16 > at SimpleMessageListenerContainer(JmsDestinationAccessor).resolveDestinationName(Session, String) line: 100 > at SimpleMessageListenerContainer.createListenerConsumer(Session) line: 188 > at SimpleMessageListenerContainer.doInitialize() line: 171 > at SimpleMessageListenerContainer(AbstractJmsListeningContainer).initialize() line: 160 > at SimpleMessageListenerContainer(AbstractJmsListeningContainer).afterPropertiesSet() line: 133 > at DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).invokeInitMethods(String, Object, RootBeanDefinition) line: 1202 > at DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).initializeBean(String, Object, RootBeanDefinition) line: 1172 > at DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).createBean(String, RootBeanDefinition, Object[]) line: 428 > at AbstractBeanFactory$1.getObject() line: 251 > at DefaultListableBeanFactory(DefaultSingletonBeanRegistry).getSingleton(String, ObjectFactory) line: 156 > at DefaultListableBeanFactory(AbstractBeanFactory).getBean(String, Class, Object[]) line: 248 > at DefaultListableBeanFactory(AbstractBeanFactory).getBean(String) line: 160 > at DefaultListableBeanFactory.preInstantiateSingletons() line: 287 > at FileSystemXmlApplicationContext(AbstractApplicationContext).refresh() line: 352 > at FileSystemXmlApplicationContext.<init>(String[], ApplicationContext) line: 93 > at FileSystemXmlApplicationContext.<init>(String[]) line: 77 > at FileSystemXmlApplicationContext.<init>(String) line: 67 > at BindingService.<init>() line: 65 > at BindingService.getInstance() line: 43 > at TopologyServiceFactory.initialize() line: 123 > at TopologyManager.initialize() line: 31 > at TopologyView$3.run(IProgressMonitor) line: 159 > at Worker.run() line: 55 > I have attached a JStack of the server process with embedded broker also. > From our client side the only thing I am dubious about is the retention of the reference to the message until the anonymous Runnable is executed. Could this cause issues is for some reason the execution blocked, will the messages build up in AMQ? > From looking at JConsole for the broker, the memoryPercentageUsed is zero so it doesnt appear to be a RAM issue. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
	[jira] Updated: (AMQ-1375) ActiveMQ hangs, cannot send message/create temp queue by JIRA jira@apache.org :: Rate this Message: Reply to Author \| View Threaded \| Show Only this Message [ https://issues.apache.org/activemq/browse/AMQ-1375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Snyder updated AMQ-1375: ------------------------------ Fix Version/s: (was: AGING_TO_DIE) 5.4.0 > ActiveMQ hangs, cannot send message/create temp queue > ----------------------------------------------------- > > Key: AMQ-1375 > URL: https://issues.apache.org/activemq/browse/AMQ-1375 > Project: ActiveMQ > Issue Type: Bug > Components: Broker > Affects Versions: 4.1.1 > Environment: Running an embedded broker in standalone java application based on Spring framework on Solaris 10 with JVM version 1.5.0_09. Using non persisten messaging with 512MB limit in memory manager of broker. > Reporter: Trevor Higgins > Fix For: 5.4.0 > > Attachments: jstack_basic.txt > > > We are experiencing severe issues with AMQ blocking sending. We have a basic setup with one standalone server process and many clients. Each client has its own temp queue for P2P request/response type communication. There is also a number of topics broadcast to all clients. Our clients receive messages in an MDP fashion with each message being processed on a single threaded executor by an anonymous Runnable as follows: > {code} > public void onMessage(final Message message) { > threadRunner.submit(new Runnable() { > public void run() { > try { > for(final IReceiverListener listener : listeners) { > listener.onReceive(message); > } > } catch (Throwable e) { > e.printStackTrace(); > } > } > }); > } > {code} > Every so often, the entire AMQ blocks and we cannot send anything anymore. We also cannot create any new temp queues, which means no new client can start up (each creates its own temp queue during startup). Connections are be created and from that a session, but the session.createTemporaryQueue() call alwaysy hangs. This is similar to AMQ-1278, but as we are cannot use 5.0 until stable release....... > Here is the stack of the client trying to create a temp queue > at Object.wait(long, int) line: not available [native method] > at CondVar(Object).wait() line: 199 > at CondVar.await() line: 75 > at ArrayBlockingQueue.take() line: 318 > at FutureResponse.getResult() line: 42 > at ResponseCorrelator.request(Object) line: 75 > at ActiveMQConnection.syncSendPacket(Command) line: 1175 > at ActiveMQConnection.createTempDestination(boolean) line: 1594 > at ActiveMQSession.createTemporaryQueue() line: 1134 > at TempDestinationResolver.resolveDestinationName(Session, String, boolean) line: 16 > at SimpleMessageListenerContainer(JmsDestinationAccessor).resolveDestinationName(Session, String) line: 100 > at SimpleMessageListenerContainer.createListenerConsumer(Session) line: 188 > at SimpleMessageListenerContainer.doInitialize() line: 171 > at SimpleMessageListenerContainer(AbstractJmsListeningContainer).initialize() line: 160 > at SimpleMessageListenerContainer(AbstractJmsListeningContainer).afterPropertiesSet() line: 133 > at DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).invokeInitMethods(String, Object, RootBeanDefinition) line: 1202 > at DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).initializeBean(String, Object, RootBeanDefinition) line: 1172 > at DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).createBean(String, RootBeanDefinition, Object[]) line: 428 > at AbstractBeanFactory$1.getObject() line: 251 > at DefaultListableBeanFactory(DefaultSingletonBeanRegistry).getSingleton(String, ObjectFactory) line: 156 > at DefaultListableBeanFactory(AbstractBeanFactory).getBean(String, Class, Object[]) line: 248 > at DefaultListableBeanFactory(AbstractBeanFactory).getBean(String) line: 160 > at DefaultListableBeanFactory.preInstantiateSingletons() line: 287 > at FileSystemXmlApplicationContext(AbstractApplicationContext).refresh() line: 352 > at FileSystemXmlApplicationContext.<init>(String[], ApplicationContext) line: 93 > at FileSystemXmlApplicationContext.<init>(String[]) line: 77 > at FileSystemXmlApplicationContext.<init>(String) line: 67 > at BindingService.<init>() line: 65 > at BindingService.getInstance() line: 43 > at TopologyServiceFactory.initialize() line: 123 > at TopologyManager.initialize() line: 31 > at TopologyView$3.run(IProgressMonitor) line: 159 > at Worker.run() line: 55 > I have attached a JStack of the server process with embedded broker also. > From our client side the only thing I am dubious about is the retention of the reference to the message until the anonymous Runnable is executed. Could this cause issues is for some reason the execution blocked, will the messages build up in AMQ? > From looking at JConsole for the broker, the memoryPercentageUsed is zero so it doesnt appear to be a RAM issue. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.