Event Correlator and Processed Groups

Hi all,

I was looking through the EventCorrelator code and noticed the following method:

    protected final BoundedFifoBuffer processedGroups = new BoundedFifoBuffer(MAX_PROCESSED_GROUPS);


    protected boolean isGroupAlreadyProcessed(Object id)
    {
        synchronized (groupsLock)
        {
            return processedGroups.contains(id);
        }
    }


It appears that everytime a new event group is received on the aggregator, it will traverse the entire queue of already processed groups to see if it has been processed yet.  Since this queue is currently sitting at 50000 groups, this seems to be very inefficient.  Wouldn't it make sense to also keep the processed groups in a collection object that offers a faster search mechanism?  I understand we still need the queue, but the memory impact should be minimal as these are very small objects.  Just looking for opinions and suggestions.

Thanks,
Rob

[Andrew Perepelytsya]

Do you have any numbers that it's really inefficient or is it just a theoretical assumption? I mean - 50000 objects is nothing to modern JVMs and hardware.

Andrew

On Thu, Feb 26, 2009 at 11:19 AM, rstacy@... <rstacy@...> wrote:

Hi all,

I was looking through the EventCorrelator code and noticed the following
method:

   protected final BoundedFifoBuffer processedGroups = new
BoundedFifoBuffer(MAX_PROCESSED_GROUPS);


   protected boolean isGroupAlreadyProcessed(Object id)
   {
       synchronized (groupsLock)
       {
           return processedGroups.contains(id);
       }
   }


It appears that everytime a new event group is received on the aggregator,
it will traverse the entire queue of already processed groups to see if it
has been processed yet.  Since this queue is currently sitting at 50000
groups, this seems to be very inefficient.  Wouldn't it make sense to also
keep the processed groups in a collection object that offers a faster search
mechanism?  I understand we still need the queue, but the memory impact
should be minimal as these are very small objects.  Just looking for
opinions and suggestions.

Thanks,
Rob

--
View this message in context: http://www.nabble.com/Event-Correlator-and-Processed-Groups-tp22227630p22227630.html
Sent from the Mule - Dev mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

   http://xircles.codehaus.org/manage_email

I have no numbers at this point.  I will try and get some.  It is just an assumption at this point.  It definitely seems that there are more efficient ways of doing it.
 

Andrew Perepelytsya wrote:

Do you have any numbers that it's really inefficient or is it just a
theoretical assumption? I mean - 50000 objects is nothing to modern JVMs and
hardware.

Andrew

On Thu, Feb 26, 2009 at 11:19 AM, [e-mail] <[e-mail]>wrote:

>
> Hi all,
>
> I was looking through the EventCorrelator code and noticed the following
> method:
>
>    protected final BoundedFifoBuffer processedGroups = new
> BoundedFifoBuffer(MAX_PROCESSED_GROUPS);
>
>
>    protected boolean isGroupAlreadyProcessed(Object id)
>    {
>        synchronized (groupsLock)
>        {
>            return processedGroups.contains(id);
>        }
>    }
>
>
> It appears that everytime a new event group is received on the aggregator,
> it will traverse the entire queue of already processed groups to see if it
> has been processed yet.  Since this queue is currently sitting at 50000
> groups, this seems to be very inefficient.  Wouldn't it make sense to also
> keep the processed groups in a collection object that offers a faster
> search
> mechanism?  I understand we still need the queue, but the memory impact
> should be minimal as these are very small objects.  Just looking for
> opinions and suggestions.
>
> Thanks,
> Rob
>
> --
> View this message in context:
> http://www.nabble.com/Event-Correlator-and-Processed-Groups-tp22227630p22227630.html
> Sent from the Mule - Dev mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>
>

[Richard Clayton]

We just ran into this issue as well. Here's some timing numbers we got on the isGroupAlreadyProcessed method:

First 10K calls: 40 seconds
Second 10K calls: 120 seconds

So this absolutely affects performance. As Rob pointed out, this is easily corrected. We added a HashSet that tracks the group ids in the BoundedFifoBuffer, let it handle the contains calls, and the total time for the isGroupAlreadyProcessed method calls drops to a fraction of a second.

Our Collection Aggregator router now rocks even on high volumes.



I have no numbers at this point.  I will try and get some.  It is just an assumption at this point.  It definitely seems that there are more efficient ways of doing it.
 

Andrew Perepelytsya wrote:

Do you have any numbers that it's really inefficient or is it just a
theoretical assumption? I mean - 50000 objects is nothing to modern JVMs and
hardware.

Andrew

On Thu, Feb 26, 2009 at 11:19 AM, [e-mail] <[e-mail]>wrote:

>
> Hi all,
>
> I was looking through the EventCorrelator code and noticed the following
> method:
>
>    protected final BoundedFifoBuffer processedGroups = new
> BoundedFifoBuffer(MAX_PROCESSED_GROUPS);
>
>
>    protected boolean isGroupAlreadyProcessed(Object id)
>    {
>        synchronized (groupsLock)
>        {
>            return processedGroups.contains(id);
>        }
>    }
>
>
> It appears that everytime a new event group is received on the aggregator,
> it will traverse the entire queue of already processed groups to see if it
> has been processed yet.  Since this queue is currently sitting at 50000
> groups, this seems to be very inefficient.  Wouldn't it make sense to also
> keep the processed groups in a collection object that offers a faster
> search
> mechanism?  I understand we still need the queue, but the memory impact
> should be minimal as these are very small objects.  Just looking for
> opinions and suggestions.
>
> Thanks,
> Rob
>
> --
> View this message in context:
> http://www.nabble.com/Event-Correlator-and-Processed-Groups-tp22227630p22227630.html
> Sent from the Mule - Dev mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>
>

[Andrew Perepelytsya]

Mind submitting a patch?

Andrew

On Apr 30, 2009 6:03 PM, "Richard Clayton" <richard@...> wrote:


We just ran into this issue as well. Here's some timing numbers we got on the
isGroupAlreadyProcessed method:

First 10K calls: 40 seconds
Second 10K calls: 120 seconds

So this absolutely affects performance. As Rob pointed out, this is easily
corrected. We added a HashSet that tracks the group ids in the
BoundedFifoBuffer, let it handle the contains calls, and the total time for
the isGroupAlreadyProcessed method calls drops to a fraction of a second.

Our Collection Aggregator router now rocks even on high volumes.

rstacy@... wrote: > > I have no numbers at this point. I will try and get some. It is j...

View this message in context: http://www.nabble.com/Event-Correlator-and-Processed-Groups-tp22227630p23325142.html

Sent from the Mule - Dev mailing list archive at Nabble.com. -------------------------------------...

[Richard Clayton]

Will do. Do I submit a Jira issue or do you?

Richard

Andrew Perepelytsya wrote:

Mind submitting a patch?

Andrew

On Apr 30, 2009 6:03 PM, "Richard Clayton" <[e-mail]> wrote:


We just ran into this issue as well. Here's some timing numbers we got on
the
isGroupAlreadyProcessed method:

First 10K calls: 40 seconds
Second 10K calls: 120 seconds

So this absolutely affects performance. As Rob pointed out, this is easily
corrected. We added a HashSet that tracks the group ids in the
BoundedFifoBuffer, let it handle the contains calls, and the total time for
the isGroupAlreadyProcessed method calls drops to a fraction of a second.

Our Collection Aggregator router now rocks even on high volumes.

[e-mail] wrote: > > I have no numbers at this point. I will try and
get some. It is j...
View this message in context:
http://www.nabble.com/Event-Correlator-and-Processed-Groups-tp22227630p23325142.html

Sent from the Mule - Dev mailing list archive at Nabble.com.
-------------------------------------...

[Andrew Perepelytsya]

Richard,

You create a new issue at http://www.mulesource.org/jira/browse/MULE (free account registration if needed). Choose a Patch Submission issue type, I'll handle it from there. You can find more details here: http://www.mulesource.org/display/MULE/Getting+Involved

HTH,
Andrew

On Thu, Apr 30, 2009 at 6:37 PM, Richard Clayton <richard@...> wrote:

Will do. Do I submit a Jira issue or do you?

Richard

Andrew Perepelytsya wrote:
>
> Mind submitting a patch?
>
> Andrew
>
> On Apr 30, 2009 6:03 PM, "Richard Clayton" <richard@...> wrote:
>
>
> We just ran into this issue as well. Here's some timing numbers we got on
> the
> isGroupAlreadyProcessed method:
>
> First 10K calls: 40 seconds
> Second 10K calls: 120 seconds
>
> So this absolutely affects performance. As Rob pointed out, this is easily
> corrected. We added a HashSet that tracks the group ids in the
> BoundedFifoBuffer, let it handle the contains calls, and the total time
> for
> the isGroupAlreadyProcessed method calls drops to a fraction of a second.
>
> Our Collection Aggregator router now rocks even on high volumes.
>
> rstacy@... wrote: > > I have no numbers at this point. I will try
> and
> get some. It is j...
> View this message in context:
> http://www.nabble.com/Event-Correlator-and-Processed-Groups-tp22227630p23325142.html
>
> Sent from the Mule - Dev mailing list archive at Nabble.com.
> -------------------------------------...
>
>

--
View this message in context: http://www.nabble.com/Event-Correlator-and-Processed-Groups-tp22227630p23325578.html

Sent from the Mule - Dev mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

   http://xircles.codehaus.org/manage_email

[LucDP]

Hi,

Can someone post the JIRA details for this patch ? A search with JIRA did not return the hoped result.

Thanks!

Luc

Andrew Perepelytsya wrote:

Richard,

You create a new issue at http://www.mulesource.org/jira/browse/MULE (free
account registration if needed). Choose a Patch Submission issue type, I'll
handle it from there. You can find more details here:
http://www.mulesource.org/display/MULE/Getting+Involved

HTH,
Andrew

On Thu, Apr 30, 2009 at 6:37 PM, Richard Clayton <[e-mail]>wrote:

>
> Will do. Do I submit a Jira issue or do you?
>
> Richard
>
>
> Andrew Perepelytsya wrote:
> >
> > Mind submitting a patch?
> >
> > Andrew
> >
> > On Apr 30, 2009 6:03 PM, "Richard Clayton" <[e-mail]> wrote:
> >
> >
> > We just ran into this issue as well. Here's some timing numbers we got on
> > the
> > isGroupAlreadyProcessed method:
> >
> > First 10K calls: 40 seconds
> > Second 10K calls: 120 seconds
> >
> > So this absolutely affects performance. As Rob pointed out, this is
> easily
> > corrected. We added a HashSet that tracks the group ids in the
> > BoundedFifoBuffer, let it handle the contains calls, and the total time
> > for
> > the isGroupAlreadyProcessed method calls drops to a fraction of a second.
> >
> > Our Collection Aggregator router now rocks even on high volumes.
> >
> > [e-mail] wrote: > > I have no numbers at this point. I will try
> > and
> > get some. It is j...
> > View this message in context:
> >
> http://www.nabble.com/Event-Correlator-and-Processed-Groups-tp22227630p23325142.html
> >
> > Sent from the Mule - Dev mailing list archive at Nabble.com.
> > -------------------------------------...
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Event-Correlator-and-Processed-Groups-tp22227630p23325578.html
> Sent from the Mule - Dev mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>
>

[Andrew Perepelytsya]

Hi Luc,

What exactly were you trying to locate? The patch is attached to jira posted above, here's the link once again: http://www.mulesource.org/jira/browse/MULE-4337

HTH,
Andrew

On Fri, Nov 6, 2009 at 9:59 AM, LucDP <luc@...> wrote:

Hi,

Can someone post the JIRA details for this patch ? A search with JIRA did
not return the hoped result.

Thanks!

Luc

Andrew Perepelytsya wrote:
>
> Richard,
>
> You create a new issue at http://www.mulesource.org/jira/browse/MULE (free
> account registration if needed). Choose a Patch Submission issue type,
> I'll
> handle it from there. You can find more details here:
> http://www.mulesource.org/display/MULE/Getting+Involved
>
> HTH,
> Andrew
>
> On Thu, Apr 30, 2009 at 6:37 PM, Richard Clayton
> <richard@...>wrote:
>
>>
>> Will do. Do I submit a Jira issue or do you?
>>
>> Richard
>>
>>
>> Andrew Perepelytsya wrote:
>> >
>> > Mind submitting a patch?
>> >
>> > Andrew
>> >
>> > On Apr 30, 2009 6:03 PM, "Richard Clayton" <richard@...> wrote:
>> >
>> >
>> > We just ran into this issue as well. Here's some timing numbers we got
>> on
>> > the
>> > isGroupAlreadyProcessed method:
>> >
>> > First 10K calls: 40 seconds
>> > Second 10K calls: 120 seconds
>> >
>> > So this absolutely affects performance. As Rob pointed out, this is
>> easily
>> > corrected. We added a HashSet that tracks the group ids in the
>> > BoundedFifoBuffer, let it handle the contains calls, and the total time
>> > for
>> > the isGroupAlreadyProcessed method calls drops to a fraction of a
>> second.
>> >
>> > Our Collection Aggregator router now rocks even on high volumes.
>> >
>> > rstacy@... wrote: > > I have no numbers at this point. I will
>> try
>> > and
>> > get some. It is j...
>> > View this message in context:
>> >
>> http://www.nabble.com/Event-Correlator-and-Processed-Groups-tp22227630p23325142.html
>> >
>> > Sent from the Mule - Dev mailing list archive at Nabble.com.
>> > -------------------------------------...
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Event-Correlator-and-Processed-Groups-tp22227630p23325578.html
>> Sent from the Mule - Dev mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe from this list, please visit:
>>
>>    http://xircles.codehaus.org/manage_email
>>
>>
>>
>
>

--
View this message in context: http://old.nabble.com/Event-Correlator-and-Processed-Groups-tp22227630p26230787.html

Sent from the Mule - Dev mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

   http://xircles.codehaus.org/manage_email