VM memory leak or expected functionality

[Daniel Feist]

Hi,

Just a quick heads up on an issue I had a quick look into this morning
and to see if anyone has any good ideas ...

http://mule.mulesource.org/jira/browse/MULE-2158
(The issue is reported against 2.x but it's likely it also affects 1.4.x.)

This issue was initially reported as a memory leak which appeared when
performing stress testing as part of MulticastRouterMule2136TestCase
test case.  After looking into this memory leak, I found that there is
no bug and this seems to be expected behaviour.

The reason for this is that there are outbound vm endpoints with event
queuing enabled that have no vm receiver configured with the same uri,
and neither does any client code pull events from the queue via the
dispatcher (using recieve()/request()).  The result of course is that
these message just pile up in the MessagePersistenceStrategy's hashmap
until we get a OutOfMemoryException.

(MemoryPersistenceStrategy is used by default at least in 2.x)

The question is what should be done about this:

- nothing. it's expected
- nothing. it's expected, but document somewhere.
- it's expected, but we need better error/warn messages so user
doesn't get a OME or if they do then know why they are getting (how
would we do this?).
- set up a dummy receiver if none configured? ( not sure this would
work though because of the receive() usage)

Dan

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Andrew Perepelytsya]

Doesn't file persistent strategy help there?

On Dec 10, 2007 1:19 PM, Daniel Feist <dfeist@...> wrote:

Hi,

Just a quick heads up on an issue I had a quick look into this morning
and to see if anyone has any good ideas ...

http://mule.mulesource.org/jira/browse/MULE-2158
(The issue is reported against 2.x but it's likely it also affects 1.4.x.)

This issue was initially reported as a memory leak which appeared when
performing stress testing as part of MulticastRouterMule2136TestCase
test case.  After looking into this memory leak, I found that there is
no bug and this seems to be expected behaviour.

The reason for this is that there are outbound vm endpoints with event
queuing enabled that have no vm receiver configured with the same uri,
and neither does any client code pull events from the queue via the
dispatcher (using recieve()/request()).  The result of course is that
these message just pile up in the MessagePersistenceStrategy's hashmap
until we get a OutOfMemoryException.

(MemoryPersistenceStrategy is used by default at least in 2.x)

The question is what should be done about this:

- nothing. it's expected
- nothing. it's expected, but document somewhere.
- it's expected, but we need better error/warn messages so user
doesn't get a OME or if they do then know why they are getting (how
would we do this?).
- set up a dummy receiver if none configured? ( not sure this would
work though because of the receive() usage)

Dan

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Daniel Feist]

It would avoid the OutOfMemoryException if it were used yes but eventually you'll run out of disk space if you have an outbound vm endpoint with queuing and no consumer.

So the root issue/question is still exists.. what to do if a user has such a queued endpoint with no consumer in their configuration, nothing? nothing but warn them somehow? or do something else?

Dan

On 10 Dec 2007, at 15:23, Andrew Perepelytsya wrote:

Doesn't file persistent strategy help there?

On Dec 10, 2007 1:19 PM, Daniel Feist <dfeist@...> wrote:

Hi,

Just a quick heads up on an issue I had a quick look into this morning
and to see if anyone has any good ideas ...

http://mule.mulesource.org/jira/browse/MULE-2158
(The issue is reported against 2.x but it's likely it also affects 1.4.x.)

This issue was initially reported as a memory leak which appeared when
performing stress testing as part of MulticastRouterMule2136TestCase
test case.  After looking into this memory leak, I found that there is
no bug and this seems to be expected behaviour.

The reason for this is that there are outbound vm endpoints with event
queuing enabled that have no vm receiver configured with the same uri,
and neither does any client code pull events from the queue via the
dispatcher (using recieve()/request()).  The result of course is that
these message just pile up in the MessagePersistenceStrategy's hashmap
until we get a OutOfMemoryException.

(MemoryPersistenceStrategy is used by default at least in 2.x)

The question is what should be done about this:

- nothing. it's expected
- nothing. it's expected, but document somewhere.
- it's expected, but we need better error/warn messages so user
doesn't get a OME or if they do then know why they are getting (how
would we do this?).
- set up a dummy receiver if none configured? ( not sure this would
work though because of the receive() usage)

Dan

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Andrew Perepelytsya]

There's nothing for us to do, it's a misconfig in this case. I thought there was a testcase issue currently.

On Dec 10, 2007 1:45 PM, Daniel Feist <dfeist@...> wrote:

It would avoid the OutOfMemoryException if it were used yes but eventually you'll run out of disk space if you have an outbound vm endpoint with queuing and no consumer.

So the root issue/question is still exists.. what to do if a user has such a queued endpoint with no consumer in their configuration, nothing? nothing but warn them somehow? or do something else?

Dan

On 10 Dec 2007, at 15:23, Andrew Perepelytsya wrote:

Doesn't file persistent strategy help there?

On Dec 10, 2007 1:19 PM, Daniel Feist <dfeist@...> wrote:

Hi,

Just a quick heads up on an issue I had a quick look into this morning
and to see if anyone has any good ideas ...

http://mule.mulesource.org/jira/browse/MULE-2158
(The issue is reported against 2.x but it's likely it also affects 1.4.x.)

This issue was initially reported as a memory leak which appeared when
performing stress testing as part of MulticastRouterMule2136TestCase
test case.  After looking into this memory leak, I found that there is
no bug and this seems to be expected behaviour.

The reason for this is that there are outbound vm endpoints with event
queuing enabled that have no vm receiver configured with the same uri,
and neither does any client code pull events from the queue via the
dispatcher (using recieve()/request()).  The result of course is that
these message just pile up in the MessagePersistenceStrategy's hashmap
until we get a OutOfMemoryException.

(MemoryPersistenceStrategy is used by default at least in 2.x)

The question is what should be done about this:

- nothing. it's expected
- nothing. it's expected, but document somewhere.
- it's expected, but we need better error/warn messages so user
doesn't get a OME or if they do then know why they are getting (how
would we do this?).
- set up a dummy receiver if none configured? ( not sure this would
work though because of the receive() usage)

Dan

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Daniel Feist]

Yes, the testcase is failing with OutOfMemoryException because it has an outbound endpoint with no consumer.

I just wanted to throw out the more general question of what to do in these situations, if anything...

Dan

On 10 Dec 2007, at 15:46, Andrew Perepelytsya wrote:

There's nothing for us to do, it's a misconfig in this case. I thought there was a testcase issue currently.

On Dec 10, 2007 1:45 PM, Daniel Feist <dfeist@...> wrote:

It would avoid the OutOfMemoryException if it were used yes but eventually you'll run out of disk space if you have an outbound vm endpoint with queuing and no consumer.

So the root issue/question is still exists.. what to do if a user has such a queued endpoint with no consumer in their configuration, nothing? nothing but warn them somehow? or do something else?

Dan

On 10 Dec 2007, at 15:23, Andrew Perepelytsya wrote:

Doesn't file persistent strategy help there?

On Dec 10, 2007 1:19 PM, Daniel Feist <dfeist@...> wrote:

Hi,

Just a quick heads up on an issue I had a quick look into this morning
and to see if anyone has any good ideas ...

http://mule.mulesource.org/jira/browse/MULE-2158
(The issue is reported against 2.x but it's likely it also affects 1.4.x.)

This issue was initially reported as a memory leak which appeared when
performing stress testing as part of MulticastRouterMule2136TestCase
test case.  After looking into this memory leak, I found that there is
no bug and this seems to be expected behaviour.

The reason for this is that there are outbound vm endpoints with event
queuing enabled that have no vm receiver configured with the same uri,
and neither does any client code pull events from the queue via the
dispatcher (using recieve()/request()).  The result of course is that
these message just pile up in the MessagePersistenceStrategy's hashmap
until we get a OutOfMemoryException.

(MemoryPersistenceStrategy is used by default at least in 2.x)

The question is what should be done about this:

- nothing. it's expected
- nothing. it's expected, but document somewhere.
- it's expected, but we need better error/warn messages so user
doesn't get a OME or if they do then know why they are getting (how
would we do this?).
- set up a dummy receiver if none configured? ( not sure this would
work though because of the receive() usage)

Dan

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Holger Hoffstätte-2]

Daniel Feist wrote:
> Yes, the testcase is failing with OutOfMemoryException because it has an
> outbound endpoint with no consumer.
>
> I just wanted to throw out the more general question of what to do in
> these situations, if anything...

IMHO just eat the event.

-h

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Daniel Feist]

To "just eat the event":

- You would need to be able to detect (at startup? in runtime?) that  
this will happen and then put something in place to eat it.

- You would not be taking into account the receiver()/request()  
usage.  Because even though there is no consumer/receiver configured a  
receive()/request() could be performed at anytime programatically via  
MuleClient or other, and if you have already eaten the event, the  
client code isn't going to get anything.

Dan


On 10 Dec 2007, at 16:22, Holger Hoffstätte wrote:

> Daniel Feist wrote:
>> Yes, the testcase is failing with OutOfMemoryException because it  
>> has an
>> outbound endpoint with no consumer.
>>
>> I just wanted to throw out the more general question of what to do in
>> these situations, if anything...
>
> IMHO just eat the event.
>
> -h
>
> ---------------------------------------------------------------------
> To unsubscribe from this list please visit:
>
>    http://xircles.codehaus.org/manage_email
>

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Holger Hoffstätte-2]

Daniel Feist wrote:
> To "just eat the event":
>
> - You would need to be able to detect (at startup? in runtime?) that
> this will happen and then put something in place to eat it.

InMemoryCookieMonsterReceiver? :)

> - You would not be taking into account the receiver()/request() usage.
> Because even though there is no consumer/receiver configured a
> receive()/request() could be performed at anytime programatically via
> MuleClient or other, and if you have already eaten the event, the client
> code isn't going to get anything.

That is actually a good point..you can look at the
IdempotentInMemoryMessageIdStore for how to write a concurrent expiring
store that has fixed upper bounds.
Of course the other, easier alternative would be to just use a JavaSpace
and get over it. :-D

/me hums a happy tune

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Ross Mason-3]

I think it would be consistent to apply a property similar to the ThreadingProfile.whenPoolExhausted.,  on the QueuingProfile we could have a similar setting that allows for - wait (with time out), Error and eat, Eat oldest, etc. 

We should probably start firing ControlNotifications when queues and pools get exhausted and also allow users to set high water marks so that they get a warning control message when the water mark is reached.

On 10 Dec 2007, at 19:50, Daniel Feist wrote:

Yes, the testcase is failing with OutOfMemoryException because it has an outbound endpoint with no consumer.

I just wanted to throw out the more general question of what to do in these situations, if anything...

Dan

On 10 Dec 2007, at 15:46, Andrew Perepelytsya wrote:

There's nothing for us to do, it's a misconfig in this case. I thought there was a testcase issue currently.

On Dec 10, 2007 1:45 PM, Daniel Feist <dfeist@...> wrote:

It would avoid the OutOfMemoryException if it were used yes but eventually you'll run out of disk space if you have an outbound vm endpoint with queuing and no consumer.

So the root issue/question is still exists.. what to do if a user has such a queued endpoint with no consumer in their configuration, nothing? nothing but warn them somehow? or do something else?

Dan

On 10 Dec 2007, at 15:23, Andrew Perepelytsya wrote:

Doesn't file persistent strategy help there?

On Dec 10, 2007 1:19 PM, Daniel Feist <dfeist@...> wrote:

Hi,

Just a quick heads up on an issue I had a quick look into this morning
and to see if anyone has any good ideas ...

http://mule.mulesource.org/jira/browse/MULE-2158
(The issue is reported against 2.x but it's likely it also affects 1.4.x.)

This issue was initially reported as a memory leak which appeared when
performing stress testing as part of MulticastRouterMule2136TestCase
test case.  After looking into this memory leak, I found that there is
no bug and this seems to be expected behaviour.

The reason for this is that there are outbound vm endpoints with event
queuing enabled that have no vm receiver configured with the same uri,
and neither does any client code pull events from the queue via the
dispatcher (using recieve()/request()).  The result of course is that
these message just pile up in the MessagePersistenceStrategy's hashmap
until we get a OutOfMemoryException.

(MemoryPersistenceStrategy is used by default at least in 2.x)

The question is what should be done about this:

- nothing. it's expected
- nothing. it's expected, but document somewhere.
- it's expected, but we need better error/warn messages so user
doesn't get a OME or if they do then know why they are getting (how
would we do this?).
- set up a dummy receiver if none configured? ( not sure this would
work though because of the receive() usage)

Dan

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Rahul S]

We are also facing memory issues while using the MulticastingRouter.
The configuration is as follows

<model name="customer-information">
        <mule-descriptor name="CustomerInfo" implementation="components.processor.CPProcessor" >
            <inbound-router>
                <endpoint  address="cpService" transformers="ByteArrayToObject" />
                     </inbound-router>
        <outbound-router>
                <router className="org.mule.routing.outbound.MulticastingRouter">
                    <endpoint   address="eSourceUpdateProfileService"  connector="jmsConnector" synchronous="false" />
                    <endpoint   address="bkUpdateProfileService"  connector="jmsConnector" synchronous="false" />
                 </router>
             </outbound-router>
   </mule-descriptor>
     </model>    
     <model name="esource-information">
        <mule-descriptor name="eSrcInfo" implementation="components.processor.ESourceProcessor" >

            <inbound-router>
                <endpoint   address="eSourceUpdateProfileService"  />
                     </inbound-router>
                               
        </mule-descriptor>
       </model>
     
     <model name="bk-information">
        <mule-descriptor name="BKInfo" implementation="components.processor.BKProcessor" >

            <inbound-router>
                <endpoint    address="bkUpdateProfileService"  />
                     </inbound-router>
                             
        </mule-descriptor>
     </model>    


Is this the root cause of the memory issue?
Any suggestions on how this can be modified so that the problem can be avoided.

Any help in this regard is highly appreciated

Andrew Perepelytsya wrote:

There's nothing for us to do, it's a misconfig in this case. I thought there
was a testcase issue currently.

On Dec 10, 2007 1:45 PM, Daniel Feist <[e-mail]> wrote:

> It would avoid the OutOfMemoryException if it were used
> yes but eventually you'll run out of disk space if you have an outbound vm
> endpoint with queuing and no consumer.
> So the root issue/question is still exists.. what to do if a user has such
> a queued endpoint with no consumer in their configuration, nothing? nothing
> but warn them somehow? or do something else?
>
> Dan
>
>
> On 10 Dec 2007, at 15:23, Andrew Perepelytsya wrote:
>
> Doesn't file persistent strategy help there?
>
> On Dec 10, 2007 1:19 PM, Daniel Feist <[e-mail]> wrote:
>
> > Hi,
> >
> > Just a quick heads up on an issue I had a quick look into this morning
> > and to see if anyone has any good ideas ...
> >
> > http://mule.mulesource.org/jira/browse/MULE-2158
> > (The issue is reported against 2.x but it's likely it also affects 1.4.x
> > .)
> >
> > This issue was initially reported as a memory leak which appeared when
> > performing stress testing as part of MulticastRouterMule2136TestCase
> > test case.  After looking into this memory leak, I found that there is
> > no bug and this seems to be expected behaviour.
> >
> > The reason for this is that there are outbound vm endpoints with event
> > queuing enabled that have no vm receiver configured with the same uri,
> > and neither does any client code pull events from the queue via the
> > dispatcher (using recieve()/request()).  The result of course is that
> > these message just pile up in the MessagePersistenceStrategy's hashmap
> > until we get a OutOfMemoryException.
> >
> > (MemoryPersistenceStrategy is used by default at least in 2.x)
> >
> > The question is what should be done about this:
> >
> > - nothing. it's expected
> > - nothing. it's expected, but document somewhere.
> > - it's expected, but we need better error/warn messages so user
> > doesn't get a OME or if they do then know why they are getting (how
> > would we do this?).
> > - set up a dummy receiver if none configured? ( not sure this would
> > work though because of the receive() usage)
> >
> > Dan
> >
> > ---------------------------------------------------------------------
> > To unsubscribe from this list please visit:
> >
> >     http://xircles.codehaus.org/manage_email
> >
> >
>
>