[jira] Created: (AMQ-2184) Messages keep hanging in JDBCStore without delivery to client

Messages keep hanging in JDBCStore without delivery to client
-------------------------------------------------------------

                 Key: AMQ-2184
                 URL: https://issues.apache.org/activemq/browse/AMQ-2184
             Project: ActiveMQ
          Issue Type: Bug
          Components: Broker
    Affects Versions: 5.2.0, 5.1.0
         Environment: OS: Linux version 2.6.26-amd64-85 (root@etch64) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21))
ActiveMQ: 5.1.0
1 Broker, 12 JBoss's  with round about 20 consumers and  20 Producers on 7 JDBC persistant queues
Clients use transacted consumers.
            Reporter: Norbert Pfistner
            Priority: Blocker
             Fix For: 5.3.0


We currently suffer from hanging messages in the JDBC persistant storage in our production environment.

Occasionally (once in a day or week) a few Messages (most time 1 or 2, sometimes more) seems to stuck in the DB Table activemq_msg without being delivered even once to a consumer. They are never deliverd to a consumer until the broker is rebooted. Strange enough new messages produced into the same queue with those stuck messages are delivered to the consumers as normal. It simply looks like the borker is blind about the stuck messages.

Our system depends on processing every message, so missing even one of them is realy a big problem in production.

Everything seems to be fine: No warnings, errors or even debug logging gives any hint about a problem.
Unfortunately we can't debug the broker 'cuse it's our production environment. Our Testing envirenment is not capable to reproduce the problem (actually it's only one Machine with a Broker and a signle jaboss running a small number of producers/clients).

Our messaging characteristics show  moderate producers and slow consumers. The count of simultanous messages is moderate (about 300-1000). Actually the producers have hot phases generating bulks of Messages followed by phases of long inactivity. The consumers are slow related to the producers but always keep within an hour or two. We do not experience any memory problems.



Because this bug may be related to https://issues.apache.org/activemq/browse/AMQ-1918, I had a look at the code around the StoreCursors and stumbled apon a strange code fragment called when removing entries.

The suspicous code is found in org.apache.activemq.broker.region.cursors.AbstractStoreCursor.remove() in 5.1.0, 5.2 .0 and 5.3.0:

    public final synchronized void remove() {
        size--;
        if (size==0 && isStarted() && cacheEnabled) {
            cacheEnabled=true;
        }
    }

As far as i can see, the if statement ist actually a NOP, 'cause it will not modify cacheEnabled in any case.

Maybe this causes some strange effects when using the StoreCursor's derived from this class..





--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/activemq/browse/AMQ-2184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Norbert Pfistner updated AMQ-2184:
----------------------------------

    Attachment: NonBrowsableQueueEntries.txt
                NonBrowsableQueueEntryList.jpg
                NonBrowsableQueueEntries.jpg


Our testing system now also shows this strange behaviour.
The Webmonitor lists correctly the 25 pending messages.
Browsing the lists shows them to be empty -- which is obviously not true,
as revealed by a direct select to the database storage in table 'activemq_msgs'.

I attached the screenshots from the web monitor and the db select.

The broker simply forgets some messages.
The only workaround seems to constantly reboot the broker, which is really annoying
and not the kind of thing we'd like to do in a productive environment.

This issue is really is realy urgent!

> Messages keep hanging in JDBCStore without delivery to client
> -------------------------------------------------------------
>
>                 Key: AMQ-2184
>                 URL: https://issues.apache.org/activemq/browse/AMQ-2184
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.1.0, 5.2.0
>         Environment: OS: Linux version 2.6.26-amd64-85 (root@etch64) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21))
> ActiveMQ: 5.1.0
> 1 Broker, 12 JBoss's  with round about 20 consumers and  20 Producers on 7 JDBC persistant queues
> Clients use transacted consumers.
>            Reporter: Norbert Pfistner
>            Priority: Blocker
>             Fix For: 5.3.0
>
>         Attachments: NonBrowsableQueueEntries.jpg, NonBrowsableQueueEntries.txt, NonBrowsableQueueEntryList.jpg
>
>
> We currently suffer from hanging messages in the JDBC persistant storage in our production environment.
> Occasionally (once in a day or week) a few Messages (most time 1 or 2, sometimes more) seems to stuck in the DB Table activemq_msg without being delivered even once to a consumer. They are never deliverd to a consumer until the broker is rebooted. Strange enough new messages produced into the same queue with those stuck messages are delivered to the consumers as normal. It simply looks like the borker is blind about the stuck messages.
> Our system depends on processing every message, so missing even one of them is realy a big problem in production.
> Everything seems to be fine: No warnings, errors or even debug logging gives any hint about a problem.
> Unfortunately we can't debug the broker 'cuse it's our production environment. Our Testing envirenment is not capable to reproduce the problem (actually it's only one Machine with a Broker and a signle jaboss running a small number of producers/clients).
> Our messaging characteristics show  moderate producers and slow consumers. The count of simultanous messages is moderate (about 300-1000). Actually the producers have hot phases generating bulks of Messages followed by phases of long inactivity. The consumers are slow related to the producers but always keep within an hour or two. We do not experience any memory problems.
> Because this bug may be related to https://issues.apache.org/activemq/browse/AMQ-1918, I had a look at the code around the StoreCursors and stumbled apon a strange code fragment called when removing entries.
> The suspicous code is found in org.apache.activemq.broker.region.cursors.AbstractStoreCursor.remove() in 5.1.0, 5.2 .0 and 5.3.0:
>     public final synchronized void remove() {
>         size--;
>         if (size==0 && isStarted() && cacheEnabled) {
>             cacheEnabled=true;
>         }
>     }
> As far as i can see, the if statement ist actually a NOP, 'cause it will not modify cacheEnabled in any case.
> Maybe this causes some strange effects when using the StoreCursor's derived from this class..

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/activemq/browse/AMQ-2184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Norbert Pfistner updated AMQ-2184:
----------------------------------

    Attachment: NonBrowsableMessaagesJConsole.jpg

I also added the JConsole for the IMPORT_READY queue.

It shows the correct queue size of 1, but the message ist still stuck in the queue.

> Messages keep hanging in JDBCStore without delivery to client
> -------------------------------------------------------------
>
>                 Key: AMQ-2184
>                 URL: https://issues.apache.org/activemq/browse/AMQ-2184
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.1.0, 5.2.0
>         Environment: OS: Linux version 2.6.26-amd64-85 (root@etch64) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21))
> ActiveMQ: 5.1.0
> 1 Broker, 12 JBoss's  with round about 20 consumers and  20 Producers on 7 JDBC persistant queues
> Clients use transacted consumers.
>            Reporter: Norbert Pfistner
>            Priority: Blocker
>             Fix For: 5.3.0
>
>         Attachments: NonBrowsableMessaagesJConsole.jpg, NonBrowsableQueueEntries.jpg, NonBrowsableQueueEntries.txt, NonBrowsableQueueEntryList.jpg
>
>
> We currently suffer from hanging messages in the JDBC persistant storage in our production environment.
> Occasionally (once in a day or week) a few Messages (most time 1 or 2, sometimes more) seems to stuck in the DB Table activemq_msg without being delivered even once to a consumer. They are never deliverd to a consumer until the broker is rebooted. Strange enough new messages produced into the same queue with those stuck messages are delivered to the consumers as normal. It simply looks like the borker is blind about the stuck messages.
> Our system depends on processing every message, so missing even one of them is realy a big problem in production.
> Everything seems to be fine: No warnings, errors or even debug logging gives any hint about a problem.
> Unfortunately we can't debug the broker 'cuse it's our production environment. Our Testing envirenment is not capable to reproduce the problem (actually it's only one Machine with a Broker and a signle jaboss running a small number of producers/clients).
> Our messaging characteristics show  moderate producers and slow consumers. The count of simultanous messages is moderate (about 300-1000). Actually the producers have hot phases generating bulks of Messages followed by phases of long inactivity. The consumers are slow related to the producers but always keep within an hour or two. We do not experience any memory problems.
> Because this bug may be related to https://issues.apache.org/activemq/browse/AMQ-1918, I had a look at the code around the StoreCursors and stumbled apon a strange code fragment called when removing entries.
> The suspicous code is found in org.apache.activemq.broker.region.cursors.AbstractStoreCursor.remove() in 5.1.0, 5.2 .0 and 5.3.0:
>     public final synchronized void remove() {
>         size--;
>         if (size==0 && isStarted() && cacheEnabled) {
>             cacheEnabled=true;
>         }
>     }
> As far as i can see, the if statement ist actually a NOP, 'cause it will not modify cacheEnabled in any case.
> Maybe this causes some strange effects when using the StoreCursor's derived from this class..

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/activemq/browse/AMQ-2184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Norbert Pfistner updated AMQ-2184:
----------------------------------

    Priority: Trivial  (was: Blocker)

Problem solved:

Eventually we started not only one but actually four brokers.
All of them had the same datasource in the DB.
The main mistake was that we had the following setting which prevented the exclusive DB Table lock for a valid master/slave configuration:

    <bean id="persistenceAdapter" class="org.apache.activemq.store.jdbc.JDBCPersistenceAdapter">
        <property name="dataSource" ref="brokerDataSource"/>
        <property name="useDatabaseLock" value="false"/>
    </bean>

After shutting down the additional brokers we do not experience hanging massages any more.

> Messages keep hanging in JDBCStore without delivery to client
> -------------------------------------------------------------
>
>                 Key: AMQ-2184
>                 URL: https://issues.apache.org/activemq/browse/AMQ-2184
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.1.0, 5.2.0
>         Environment: OS: Linux version 2.6.26-amd64-85 (root@etch64) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21))
> ActiveMQ: 5.1.0
> 1 Broker, 12 JBoss's  with round about 20 consumers and  20 Producers on 7 JDBC persistant queues
> Clients use transacted consumers.
>            Reporter: Norbert Pfistner
>            Priority: Trivial
>             Fix For: 5.3.0
>
>         Attachments: NonBrowsableMessaagesJConsole.jpg, NonBrowsableQueueEntries.jpg, NonBrowsableQueueEntries.txt, NonBrowsableQueueEntryList.jpg
>
>
> We currently suffer from hanging messages in the JDBC persistant storage in our production environment.
> Occasionally (once in a day or week) a few Messages (most time 1 or 2, sometimes more) seems to stuck in the DB Table activemq_msg without being delivered even once to a consumer. They are never deliverd to a consumer until the broker is rebooted. Strange enough new messages produced into the same queue with those stuck messages are delivered to the consumers as normal. It simply looks like the borker is blind about the stuck messages.
> Our system depends on processing every message, so missing even one of them is realy a big problem in production.
> Everything seems to be fine: No warnings, errors or even debug logging gives any hint about a problem.
> Unfortunately we can't debug the broker 'cuse it's our production environment. Our Testing envirenment is not capable to reproduce the problem (actually it's only one Machine with a Broker and a signle jaboss running a small number of producers/clients).
> Our messaging characteristics show  moderate producers and slow consumers. The count of simultanous messages is moderate (about 300-1000). Actually the producers have hot phases generating bulks of Messages followed by phases of long inactivity. The consumers are slow related to the producers but always keep within an hour or two. We do not experience any memory problems.
> Because this bug may be related to https://issues.apache.org/activemq/browse/AMQ-1918, I had a look at the code around the StoreCursors and stumbled apon a strange code fragment called when removing entries.
> The suspicous code is found in org.apache.activemq.broker.region.cursors.AbstractStoreCursor.remove() in 5.1.0, 5.2 .0 and 5.3.0:
>     public final synchronized void remove() {
>         size--;
>         if (size==0 && isStarted() && cacheEnabled) {
>             cacheEnabled=true;
>         }
>     }
> As far as i can see, the if statement ist actually a NOP, 'cause it will not modify cacheEnabled in any case.
> Maybe this causes some strange effects when using the StoreCursor's derived from this class..

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/activemq/browse/AMQ-2184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rob Davies reassigned AMQ-2184:
-------------------------------

    Assignee: Rob Davies

> Messages keep hanging in JDBCStore without delivery to client
> -------------------------------------------------------------
>
>                 Key: AMQ-2184
>                 URL: https://issues.apache.org/activemq/browse/AMQ-2184
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.1.0, 5.2.0
>         Environment: OS: Linux version 2.6.26-amd64-85 (root@etch64) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21))
> ActiveMQ: 5.1.0
> 1 Broker, 12 JBoss's  with round about 20 consumers and  20 Producers on 7 JDBC persistant queues
> Clients use transacted consumers.
>            Reporter: Norbert Pfistner
>            Assignee: Rob Davies
>            Priority: Trivial
>             Fix For: 5.4.0
>
>         Attachments: NonBrowsableMessaagesJConsole.jpg, NonBrowsableQueueEntries.jpg, NonBrowsableQueueEntries.txt, NonBrowsableQueueEntryList.jpg
>
>
> We currently suffer from hanging messages in the JDBC persistant storage in our production environment.
> Occasionally (once in a day or week) a few Messages (most time 1 or 2, sometimes more) seems to stuck in the DB Table activemq_msg without being delivered even once to a consumer. They are never deliverd to a consumer until the broker is rebooted. Strange enough new messages produced into the same queue with those stuck messages are delivered to the consumers as normal. It simply looks like the borker is blind about the stuck messages.
> Our system depends on processing every message, so missing even one of them is realy a big problem in production.
> Everything seems to be fine: No warnings, errors or even debug logging gives any hint about a problem.
> Unfortunately we can't debug the broker 'cuse it's our production environment. Our Testing envirenment is not capable to reproduce the problem (actually it's only one Machine with a Broker and a signle jaboss running a small number of producers/clients).
> Our messaging characteristics show  moderate producers and slow consumers. The count of simultanous messages is moderate (about 300-1000). Actually the producers have hot phases generating bulks of Messages followed by phases of long inactivity. The consumers are slow related to the producers but always keep within an hour or two. We do not experience any memory problems.
> Because this bug may be related to https://issues.apache.org/activemq/browse/AMQ-1918, I had a look at the code around the StoreCursors and stumbled apon a strange code fragment called when removing entries.
> The suspicous code is found in org.apache.activemq.broker.region.cursors.AbstractStoreCursor.remove() in 5.1.0, 5.2 .0 and 5.3.0:
>     public final synchronized void remove() {
>         size--;
>         if (size==0 && isStarted() && cacheEnabled) {
>             cacheEnabled=true;
>         }
>     }
> As far as i can see, the if statement ist actually a NOP, 'cause it will not modify cacheEnabled in any case.
> Maybe this causes some strange effects when using the StoreCursor's derived from this class..

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/activemq/browse/AMQ-2184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rob Davies resolved AMQ-2184.
-----------------------------

       Resolution: Fixed
    Fix Version/s:     (was: 5.4.0)
                   5.3.0

This looks fixed now - thanks Norbert

> Messages keep hanging in JDBCStore without delivery to client
> -------------------------------------------------------------
>
>                 Key: AMQ-2184
>                 URL: https://issues.apache.org/activemq/browse/AMQ-2184
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.1.0, 5.2.0
>         Environment: OS: Linux version 2.6.26-amd64-85 (root@etch64) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21))
> ActiveMQ: 5.1.0
> 1 Broker, 12 JBoss's  with round about 20 consumers and  20 Producers on 7 JDBC persistant queues
> Clients use transacted consumers.
>            Reporter: Norbert Pfistner
>            Assignee: Rob Davies
>            Priority: Trivial
>             Fix For: 5.3.0
>
>         Attachments: NonBrowsableMessaagesJConsole.jpg, NonBrowsableQueueEntries.jpg, NonBrowsableQueueEntries.txt, NonBrowsableQueueEntryList.jpg
>
>
> We currently suffer from hanging messages in the JDBC persistant storage in our production environment.
> Occasionally (once in a day or week) a few Messages (most time 1 or 2, sometimes more) seems to stuck in the DB Table activemq_msg without being delivered even once to a consumer. They are never deliverd to a consumer until the broker is rebooted. Strange enough new messages produced into the same queue with those stuck messages are delivered to the consumers as normal. It simply looks like the borker is blind about the stuck messages.
> Our system depends on processing every message, so missing even one of them is realy a big problem in production.
> Everything seems to be fine: No warnings, errors or even debug logging gives any hint about a problem.
> Unfortunately we can't debug the broker 'cuse it's our production environment. Our Testing envirenment is not capable to reproduce the problem (actually it's only one Machine with a Broker and a signle jaboss running a small number of producers/clients).
> Our messaging characteristics show  moderate producers and slow consumers. The count of simultanous messages is moderate (about 300-1000). Actually the producers have hot phases generating bulks of Messages followed by phases of long inactivity. The consumers are slow related to the producers but always keep within an hour or two. We do not experience any memory problems.
> Because this bug may be related to https://issues.apache.org/activemq/browse/AMQ-1918, I had a look at the code around the StoreCursors and stumbled apon a strange code fragment called when removing entries.
> The suspicous code is found in org.apache.activemq.broker.region.cursors.AbstractStoreCursor.remove() in 5.1.0, 5.2 .0 and 5.3.0:
>     public final synchronized void remove() {
>         size--;
>         if (size==0 && isStarted() && cacheEnabled) {
>             cacheEnabled=true;
>         }
>     }
> As far as i can see, the if statement ist actually a NOP, 'cause it will not modify cacheEnabled in any case.
> Maybe this causes some strange effects when using the StoreCursor's derived from this class..

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/activemq/browse/AMQ-2184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Norbert Pfistner reopened AMQ-2184:
-----------------------------------


Unfortunately we are experiencing the prblem again -- now with a single standalone broker.

The hanging mesages are listed in the database but not in the visible in the webconsole (browse).

Restarting the connector (using JMX) seems to to be a hotfix (but obvoiously not a solution to the problem).

The listeners wait receiving (as stated by a thread dump), so it seems not to be a client code problem.

BTW: Still using activemq 5.1

http://issues.apache.org/activemq/browse/AMQ-1918 may be related to this problem.

> Messages keep hanging in JDBCStore without delivery to client
> -------------------------------------------------------------
>
>                 Key: AMQ-2184
>                 URL: https://issues.apache.org/activemq/browse/AMQ-2184
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.1.0, 5.2.0
>         Environment: OS: Linux version 2.6.26-amd64-85 (root@etch64) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21))
> ActiveMQ: 5.1.0
> 1 Broker, 12 JBoss's  with round about 20 consumers and  20 Producers on 7 JDBC persistant queues
> Clients use transacted consumers.
>            Reporter: Norbert Pfistner
>            Assignee: Rob Davies
>            Priority: Trivial
>             Fix For: 5.3.0
>
>         Attachments: NonBrowsableMessaagesJConsole.jpg, NonBrowsableQueueEntries.jpg, NonBrowsableQueueEntries.txt, NonBrowsableQueueEntryList.jpg
>
>
> We currently suffer from hanging messages in the JDBC persistant storage in our production environment.
> Occasionally (once in a day or week) a few Messages (most time 1 or 2, sometimes more) seems to stuck in the DB Table activemq_msg without being delivered even once to a consumer. They are never deliverd to a consumer until the broker is rebooted. Strange enough new messages produced into the same queue with those stuck messages are delivered to the consumers as normal. It simply looks like the borker is blind about the stuck messages.
> Our system depends on processing every message, so missing even one of them is realy a big problem in production.
> Everything seems to be fine: No warnings, errors or even debug logging gives any hint about a problem.
> Unfortunately we can't debug the broker 'cuse it's our production environment. Our Testing envirenment is not capable to reproduce the problem (actually it's only one Machine with a Broker and a signle jaboss running a small number of producers/clients).
> Our messaging characteristics show  moderate producers and slow consumers. The count of simultanous messages is moderate (about 300-1000). Actually the producers have hot phases generating bulks of Messages followed by phases of long inactivity. The consumers are slow related to the producers but always keep within an hour or two. We do not experience any memory problems.
> Because this bug may be related to https://issues.apache.org/activemq/browse/AMQ-1918, I had a look at the code around the StoreCursors and stumbled apon a strange code fragment called when removing entries.
> The suspicous code is found in org.apache.activemq.broker.region.cursors.AbstractStoreCursor.remove() in 5.1.0, 5.2 .0 and 5.3.0:
>     public final synchronized void remove() {
>         size--;
>         if (size==0 && isStarted() && cacheEnabled) {
>             cacheEnabled=true;
>         }
>     }
> As far as i can see, the if statement ist actually a NOP, 'cause it will not modify cacheEnabled in any case.
> Maybe this causes some strange effects when using the StoreCursor's derived from this class..

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/activemq/browse/AMQ-2184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=52698#action_52698 ]

Rob Davies commented on AMQ-2184:
---------------------------------

Hi Norbert - we don't do patches to the 5.1 branch - is this an issue with 5.3 (trunk currently) ? If not can you close it ?

> Messages keep hanging in JDBCStore without delivery to client
> -------------------------------------------------------------
>
>                 Key: AMQ-2184
>                 URL: https://issues.apache.org/activemq/browse/AMQ-2184
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.1.0, 5.2.0
>         Environment: OS: Linux version 2.6.26-amd64-85 (root@etch64) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21))
> ActiveMQ: 5.1.0
> 1 Broker, 12 JBoss's  with round about 20 consumers and  20 Producers on 7 JDBC persistant queues
> Clients use transacted consumers.
>            Reporter: Norbert Pfistner
>            Assignee: Rob Davies
>            Priority: Trivial
>             Fix For: 5.3.0
>
>         Attachments: NonBrowsableMessaagesJConsole.jpg, NonBrowsableQueueEntries.jpg, NonBrowsableQueueEntries.txt, NonBrowsableQueueEntryList.jpg
>
>
> We currently suffer from hanging messages in the JDBC persistant storage in our production environment.
> Occasionally (once in a day or week) a few Messages (most time 1 or 2, sometimes more) seems to stuck in the DB Table activemq_msg without being delivered even once to a consumer. They are never deliverd to a consumer until the broker is rebooted. Strange enough new messages produced into the same queue with those stuck messages are delivered to the consumers as normal. It simply looks like the borker is blind about the stuck messages.
> Our system depends on processing every message, so missing even one of them is realy a big problem in production.
> Everything seems to be fine: No warnings, errors or even debug logging gives any hint about a problem.
> Unfortunately we can't debug the broker 'cuse it's our production environment. Our Testing envirenment is not capable to reproduce the problem (actually it's only one Machine with a Broker and a signle jaboss running a small number of producers/clients).
> Our messaging characteristics show  moderate producers and slow consumers. The count of simultanous messages is moderate (about 300-1000). Actually the producers have hot phases generating bulks of Messages followed by phases of long inactivity. The consumers are slow related to the producers but always keep within an hour or two. We do not experience any memory problems.
> Because this bug may be related to https://issues.apache.org/activemq/browse/AMQ-1918, I had a look at the code around the StoreCursors and stumbled apon a strange code fragment called when removing entries.
> The suspicous code is found in org.apache.activemq.broker.region.cursors.AbstractStoreCursor.remove() in 5.1.0, 5.2 .0 and 5.3.0:
>     public final synchronized void remove() {
>         size--;
>         if (size==0 && isStarted() && cacheEnabled) {
>             cacheEnabled=true;
>         }
>     }
> As far as i can see, the if statement ist actually a NOP, 'cause it will not modify cacheEnabled in any case.
> Maybe this causes some strange effects when using the StoreCursor's derived from this class..

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/activemq/browse/AMQ-2184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=52716#action_52716 ]

Norbert Pfistner commented on AMQ-2184:
---------------------------------------

As long as we can see it's an issue somehow related to the database persistance. We still did not get any pin on it.
Currently we use 5.1, so as Dave mentioned in the comment above I'll close it.

> Messages keep hanging in JDBCStore without delivery to client
> -------------------------------------------------------------
>
>                 Key: AMQ-2184
>                 URL: https://issues.apache.org/activemq/browse/AMQ-2184
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.1.0, 5.2.0
>         Environment: OS: Linux version 2.6.26-amd64-85 (root@etch64) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21))
> ActiveMQ: 5.1.0
> 1 Broker, 12 JBoss's  with round about 20 consumers and  20 Producers on 7 JDBC persistant queues
> Clients use transacted consumers.
>            Reporter: Norbert Pfistner
>            Assignee: Rob Davies
>            Priority: Trivial
>             Fix For: 5.3.0
>
>         Attachments: NonBrowsableMessaagesJConsole.jpg, NonBrowsableQueueEntries.jpg, NonBrowsableQueueEntries.txt, NonBrowsableQueueEntryList.jpg
>
>
> We currently suffer from hanging messages in the JDBC persistant storage in our production environment.
> Occasionally (once in a day or week) a few Messages (most time 1 or 2, sometimes more) seems to stuck in the DB Table activemq_msg without being delivered even once to a consumer. They are never deliverd to a consumer until the broker is rebooted. Strange enough new messages produced into the same queue with those stuck messages are delivered to the consumers as normal. It simply looks like the borker is blind about the stuck messages.
> Our system depends on processing every message, so missing even one of them is realy a big problem in production.
> Everything seems to be fine: No warnings, errors or even debug logging gives any hint about a problem.
> Unfortunately we can't debug the broker 'cuse it's our production environment. Our Testing envirenment is not capable to reproduce the problem (actually it's only one Machine with a Broker and a signle jaboss running a small number of producers/clients).
> Our messaging characteristics show  moderate producers and slow consumers. The count of simultanous messages is moderate (about 300-1000). Actually the producers have hot phases generating bulks of Messages followed by phases of long inactivity. The consumers are slow related to the producers but always keep within an hour or two. We do not experience any memory problems.
> Because this bug may be related to https://issues.apache.org/activemq/browse/AMQ-1918, I had a look at the code around the StoreCursors and stumbled apon a strange code fragment called when removing entries.
> The suspicous code is found in org.apache.activemq.broker.region.cursors.AbstractStoreCursor.remove() in 5.1.0, 5.2 .0 and 5.3.0:
>     public final synchronized void remove() {
>         size--;
>         if (size==0 && isStarted() && cacheEnabled) {
>             cacheEnabled=true;
>         }
>     }
> As far as i can see, the if statement ist actually a NOP, 'cause it will not modify cacheEnabled in any case.
> Maybe this causes some strange effects when using the StoreCursor's derived from this class..

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/activemq/browse/AMQ-2184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Norbert Pfistner closed AMQ-2184.
---------------------------------

       Resolution: Won't Fix
    Fix Version/s:     (was: 5.3.0)
                   5.1.0

It seems that we do not get any clear solution for this bug for version 5.1.

> Messages keep hanging in JDBCStore without delivery to client
> -------------------------------------------------------------
>
>                 Key: AMQ-2184
>                 URL: https://issues.apache.org/activemq/browse/AMQ-2184
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.1.0, 5.2.0
>         Environment: OS: Linux version 2.6.26-amd64-85 (root@etch64) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21))
> ActiveMQ: 5.1.0
> 1 Broker, 12 JBoss's  with round about 20 consumers and  20 Producers on 7 JDBC persistant queues
> Clients use transacted consumers.
>            Reporter: Norbert Pfistner
>            Assignee: Rob Davies
>            Priority: Trivial
>             Fix For: 5.1.0
>
>         Attachments: NonBrowsableMessaagesJConsole.jpg, NonBrowsableQueueEntries.jpg, NonBrowsableQueueEntries.txt, NonBrowsableQueueEntryList.jpg
>
>
> We currently suffer from hanging messages in the JDBC persistant storage in our production environment.
> Occasionally (once in a day or week) a few Messages (most time 1 or 2, sometimes more) seems to stuck in the DB Table activemq_msg without being delivered even once to a consumer. They are never deliverd to a consumer until the broker is rebooted. Strange enough new messages produced into the same queue with those stuck messages are delivered to the consumers as normal. It simply looks like the borker is blind about the stuck messages.
> Our system depends on processing every message, so missing even one of them is realy a big problem in production.
> Everything seems to be fine: No warnings, errors or even debug logging gives any hint about a problem.
> Unfortunately we can't debug the broker 'cuse it's our production environment. Our Testing envirenment is not capable to reproduce the problem (actually it's only one Machine with a Broker and a signle jaboss running a small number of producers/clients).
> Our messaging characteristics show  moderate producers and slow consumers. The count of simultanous messages is moderate (about 300-1000). Actually the producers have hot phases generating bulks of Messages followed by phases of long inactivity. The consumers are slow related to the producers but always keep within an hour or two. We do not experience any memory problems.
> Because this bug may be related to https://issues.apache.org/activemq/browse/AMQ-1918, I had a look at the code around the StoreCursors and stumbled apon a strange code fragment called when removing entries.
> The suspicous code is found in org.apache.activemq.broker.region.cursors.AbstractStoreCursor.remove() in 5.1.0, 5.2 .0 and 5.3.0:
>     public final synchronized void remove() {
>         size--;
>         if (size==0 && isStarted() && cacheEnabled) {
>             cacheEnabled=true;
>         }
>     }
> As far as i can see, the if statement ist actually a NOP, 'cause it will not modify cacheEnabled in any case.
> Maybe this causes some strange effects when using the StoreCursor's derived from this class..

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.