Message Head of line blocking

[David Erickson]

Hi All,
I am using AMQ inside a Spring IoC container.  It is a very simple setup with the following config:

  <!--  ActiveMQ Broker -->
  <amq:broker useJmx="false" persistent="false">
    <amq:transportConnectors>
      <amq:transportConnector uri="tcp://localhost:0" />
    </amq:transportConnectors>
   
  </amq:broker>

   <!--  ActiveMQ destinations to use  -->
  <amq:queue id="mailDestination"  physicalName="mailsender"/>

  <!-- JMS ConnectionFactory to use, configuring the embedded broker using XML -->
  <amq:connectionFactory id="jmsFactory" brokerURL="vm://localhost">
    <amq:redeliveryPolicy>
      <amq:redeliveryPolicy backOffMultiplier="2" useExponentialBackOff="true" initialRedeliveryDelay="1000" maximumRedeliveries="-1"/>
    </amq:redeliveryPolicy>
  </amq:connectionFactory>
 
  <bean id="myJmsTemplate" class="org.springframework.jms.core.JmsTemplate">
    <property name="connectionFactory">
      <!-- lets wrap in a pool to avoid creating a connection per send -->
      <bean class="org.apache.activemq.pool.PooledConnectionFactory">
        <property name="connectionFactory">
          <ref local="jmsFactory"/>
        </property>
      </bean>
    </property>
    <property name="pubSubDomain">
      <value>false</value>
    </property>
  </bean>

  <bean id="mailContainer" class="org.springframework.jms.listener.DefaultMessageListenerContainer">
      <property name="connectionFactory" ref="jmsFactory"/>
      <property name="destination" ref="mailDestination"/>
      <property name="messageListener" ref="mailConsumer"/>
      <property name="sessionTransacted" value="true"/>
  </bean>

Along with one corresponding producer and consumer.

My question is this, I noticed that my consumer was having problems sending one of the emails in the queue, it was throwing an exception, being placed back in the queue, then the entire queue was waiting the backoff time for this message to attempt to be delivered again, over and over again, producing a head of line blocking problem (no messages can be delivered until the one with the error is delivered).

Is there a solution for this? Can the message that failed be pushed to a re-delivery queue, or a troubled queue rather than the main queue? Or can the consumer walk through the queue looking for a message that is ready to be delivered rather than blocking on the front message that isn't set to be delivered for some long amount of time because of a previous failure and backoff time?

Thanks!

[Dejan Bosanac]

Hi David,

it's all configured in your "redeliveryPolicy", where you instructed the
broker to try resending the message indefinitely.

See http://activemq.apache.org/message-redelivery-and-dlq-handling.html for
more information how redelivery works and how you can use "Dead letter
queues"

Cheers
--
Dejan Bosanac - http://twitter.com/dejanb

Open Source Integration - http://fusesource.com/
ActiveMQ in Action - http://www.manning.com/snyder/
Blog - http://www.nighttale.net


On Sat, Nov 7, 2009 at 7:35 PM, David Erickson <derickso@...>wrote:

>
> Hi All,
> I am using AMQ inside a Spring IoC container.  It is a very simple setup
> with the following config:
>
>  <!--  ActiveMQ Broker -->
>  <amq:broker useJmx="false" persistent="false">
>    <amq:transportConnectors>
>      <amq:transportConnector uri="tcp://localhost:0" />
>    </amq:transportConnectors>
>
>  </amq:broker>
>
>   <!--  ActiveMQ destinations to use  -->
>  <amq:queue id="mailDestination"  physicalName="mailsender"/>
>
>  <!-- JMS ConnectionFactory to use, configuring the embedded broker using
> XML -->
>  <amq:connectionFactory id="jmsFactory" brokerURL="vm://localhost">
>    <amq:redeliveryPolicy>
>      <amq:redeliveryPolicy backOffMultiplier="2"
> useExponentialBackOff="true" initialRedeliveryDelay="1000"
> maximumRedeliveries="-1"/>
>    </amq:redeliveryPolicy>
>  </amq:connectionFactory>
>
>  <bean id="myJmsTemplate" class="org.springframework.jms.core.JmsTemplate">
>    <property name="connectionFactory">
>      <!-- lets wrap in a pool to avoid creating a connection per send -->
>      <bean class="org.apache.activemq.pool.PooledConnectionFactory">
>        <property name="connectionFactory">
>          <ref local="jmsFactory"/>
>        </property>
>      </bean>
>    </property>
>    <property name="pubSubDomain">
>      <value>false</value>
>    </property>
>  </bean>
>
>  <bean id="mailContainer"
> class="org.springframework.jms.listener.DefaultMessageListenerContainer">
>      <property name="connectionFactory" ref="jmsFactory"/>
>      <property name="destination" ref="mailDestination"/>
>      <property name="messageListener" ref="mailConsumer"/>
>      <property name="sessionTransacted" value="true"/>
>  </bean>
>
> Along with one corresponding producer and consumer.
>
> My question is this, I noticed that my consumer was having problems sending
> one of the emails in the queue, it was throwing an exception, being placed
> back in the queue, then the entire queue was waiting the backoff time for
> this message to attempt to be delivered again, over and over again,
> producing a head of line blocking problem (no messages can be delivered
> until the one with the error is delivered).
>
> Is there a solution for this? Can the message that failed be pushed to a
> re-delivery queue, or a troubled queue rather than the main queue? Or can
> the consumer walk through the queue looking for a message that is ready to
> be delivered rather than blocking on the front message that isn't set to be
> delivered for some long amount of time because of a previous failure and
> backoff time?
>
> Thanks!
> --
> View this message in context:
> http://old.nabble.com/Message-Head-of-line-blocking-tp26247283p26247283.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>

[David Erickson]

Hi Dejan,
Thanks for the reply.  Do I need to explicitly declare the DLQ from within my Spring xml or is it created automatically?  And I imagine I will need to put a consumer on the queue to do anything with it?

Thanks,
David

Dejan Bosanac wrote:

Hi David,

it's all configured in your "redeliveryPolicy", where you instructed the
broker to try resending the message indefinitely.

See http://activemq.apache.org/message-redelivery-and-dlq-handling.html for
more information how redelivery works and how you can use "Dead letter
queues"

Cheers
--
Dejan Bosanac - http://twitter.com/dejanb

Open Source Integration - http://fusesource.com/
ActiveMQ in Action - http://www.manning.com/snyder/
Blog - http://www.nighttale.net


On Sat, Nov 7, 2009 at 7:35 PM, David Erickson <[e-mail]>wrote:

>
> Hi All,
> I am using AMQ inside a Spring IoC container.  It is a very simple setup
> with the following config:
>
>  <!--  ActiveMQ Broker -->
>  <amq:broker useJmx="false" persistent="false">
>    <amq:transportConnectors>
>      <amq:transportConnector uri="tcp://localhost:0" />
>    </amq:transportConnectors>
>
>  </amq:broker>
>
>   <!--  ActiveMQ destinations to use  -->
>  <amq:queue id="mailDestination"  physicalName="mailsender"/>
>
>  <!-- JMS ConnectionFactory to use, configuring the embedded broker using
> XML -->
>  <amq:connectionFactory id="jmsFactory" brokerURL="vm://localhost">
>    <amq:redeliveryPolicy>
>      <amq:redeliveryPolicy backOffMultiplier="2"
> useExponentialBackOff="true" initialRedeliveryDelay="1000"
> maximumRedeliveries="-1"/>
>    </amq:redeliveryPolicy>
>  </amq:connectionFactory>
>
>  <bean id="myJmsTemplate" class="org.springframework.jms.core.JmsTemplate">
>    <property name="connectionFactory">
>      <!-- lets wrap in a pool to avoid creating a connection per send -->
>      <bean class="org.apache.activemq.pool.PooledConnectionFactory">
>        <property name="connectionFactory">
>          <ref local="jmsFactory"/>
>        </property>
>      </bean>
>    </property>
>    <property name="pubSubDomain">
>      <value>false</value>
>    </property>
>  </bean>
>
>  <bean id="mailContainer"
> class="org.springframework.jms.listener.DefaultMessageListenerContainer">
>      <property name="connectionFactory" ref="jmsFactory"/>
>      <property name="destination" ref="mailDestination"/>
>      <property name="messageListener" ref="mailConsumer"/>
>      <property name="sessionTransacted" value="true"/>
>  </bean>
>
> Along with one corresponding producer and consumer.
>
> My question is this, I noticed that my consumer was having problems sending
> one of the emails in the queue, it was throwing an exception, being placed
> back in the queue, then the entire queue was waiting the backoff time for
> this message to attempt to be delivered again, over and over again,
> producing a head of line blocking problem (no messages can be delivered
> until the one with the error is delivered).
>
> Is there a solution for this? Can the message that failed be pushed to a
> re-delivery queue, or a troubled queue rather than the main queue? Or can
> the consumer walk through the queue looking for a message that is ready to
> be delivered rather than blocking on the front message that isn't set to be
> delivered for some long amount of time because of a previous failure and
> backoff time?
>
> Thanks!
> --
> View this message in context:
> http://old.nabble.com/Message-Head-of-line-blocking-tp26247283p26247283.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>


-----
Dejan Bosanac

Open Source Integration - http://fusesource.com/
ActiveMQ in Action - http://www.manning.com/snyder/
Blog - http://www.nighttale.net

[Dejan Bosanac]

Hi,

there's a default DLQ queue, but you can customize this if you need. You can
put a consumer there, or use web console to browse failed messages and do
appropriate actions manually.

Cheers
--
Dejan Bosanac - http://twitter.com/dejanb

Open Source Integration - http://fusesource.com/
ActiveMQ in Action - http://www.manning.com/snyder/
Blog - http://www.nighttale.net


On Mon, Nov 9, 2009 at 6:03 PM, David Erickson <derickso@...>wrote:

>
> Hi Dejan,
> Thanks for the reply.  Do I need to explicitly declare the DLQ from within
> my Spring xml or is it created automatically?  And I imagine I will need to
> put a consumer on the queue to do anything with it?
>
> Thanks,
> David
>
>
>
> Dejan Bosanac wrote:
> >
> > Hi David,
> >
> > it's all configured in your "redeliveryPolicy", where you instructed the
> > broker to try resending the message indefinitely.
> >
> > See http://activemq.apache.org/message-redelivery-and-dlq-handling.html
> > for
> > more information how redelivery works and how you can use "Dead letter
> > queues"
> >
> > Cheers
> > --
> > Dejan Bosanac - http://twitter.com/dejanb
> >
> > Open Source Integration - http://fusesource.com/
> > ActiveMQ in Action - http://www.manning.com/snyder/
> > Blog - http://www.nighttale.net
> >
> >
> > On Sat, Nov 7, 2009 at 7:35 PM, David Erickson
> > <derickso@...>wrote:
> >
> >>
> >> Hi All,
> >> I am using AMQ inside a Spring IoC container.  It is a very simple setup
> >> with the following config:
> >>
> >>  <!--  ActiveMQ Broker -->
> >>  <amq:broker useJmx="false" persistent="false">
> >>    <amq:transportConnectors>
> >>      <amq:transportConnector uri="tcp://localhost:0" />
> >>    </amq:transportConnectors>
> >>
> >>  </amq:broker>
> >>
> >>   <!--  ActiveMQ destinations to use  -->
> >>  <amq:queue id="mailDestination"  physicalName="mailsender"/>
> >>
> >>  <!-- JMS ConnectionFactory to use, configuring the embedded broker
> using
> >> XML -->
> >>  <amq:connectionFactory id="jmsFactory" brokerURL="vm://localhost">
> >>    <amq:redeliveryPolicy>
> >>      <amq:redeliveryPolicy backOffMultiplier="2"
> >> useExponentialBackOff="true" initialRedeliveryDelay="1000"
> >> maximumRedeliveries="-1"/>
> >>    </amq:redeliveryPolicy>
> >>  </amq:connectionFactory>
> >>
> >>  <bean id="myJmsTemplate"
> >> class="org.springframework.jms.core.JmsTemplate">
> >>    <property name="connectionFactory">
> >>      <!-- lets wrap in a pool to avoid creating a connection per send
> -->
> >>      <bean class="org.apache.activemq.pool.PooledConnectionFactory">
> >>        <property name="connectionFactory">
> >>          <ref local="jmsFactory"/>
> >>        </property>
> >>      </bean>
> >>    </property>
> >>    <property name="pubSubDomain">
> >>      <value>false</value>
> >>    </property>
> >>  </bean>
> >>
> >>  <bean id="mailContainer"
> >>
> class="org.springframework.jms.listener.DefaultMessageListenerContainer">
> >>      <property name="connectionFactory" ref="jmsFactory"/>
> >>      <property name="destination" ref="mailDestination"/>
> >>      <property name="messageListener" ref="mailConsumer"/>
> >>      <property name="sessionTransacted" value="true"/>
> >>  </bean>
> >>
> >> Along with one corresponding producer and consumer.
> >>
> >> My question is this, I noticed that my consumer was having problems
> >> sending
> >> one of the emails in the queue, it was throwing an exception, being
> >> placed
> >> back in the queue, then the entire queue was waiting the backoff time
> for
> >> this message to attempt to be delivered again, over and over again,
> >> producing a head of line blocking problem (no messages can be delivered
> >> until the one with the error is delivered).
> >>
> >> Is there a solution for this? Can the message that failed be pushed to a
> >> re-delivery queue, or a troubled queue rather than the main queue? Or
> can
> >> the consumer walk through the queue looking for a message that is ready
> >> to
> >> be delivered rather than blocking on the front message that isn't set to
> >> be
> >> delivered for some long amount of time because of a previous failure and
> >> backoff time?
> >>
> >> Thanks!
> >> --
> >> View this message in context:
> >>
> http://old.nabble.com/Message-Head-of-line-blocking-tp26247283p26247283.html
> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
> > -----
> > Dejan Bosanac
> >
> > Open Source Integration - http://fusesource.com/
> > ActiveMQ in Action - http://www.manning.com/snyder/
> > Blog - http://www.nighttale.net
> >
>
> --
> View this message in context:
> http://old.nabble.com/Message-Head-of-line-blocking-tp26247283p26269877.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>