Re: Performance problems

[Gerben Vos]

Jim and Artem, thanks for your suggestions.

The tables are read/write. Although the action we're having problems with is a search and we only read the tables, we also need to do write an audit entry for each found item to a different table.

The string GUIDs are necessary because Oracle can't support binary GUIDs. I don't know why we haven't chosen some kind of auto-increment integer originally; probably also because different databases handle that differently.

We did some profiling. Most time is spent in the function ObjectContext.ImportRowsFromTable(). We didn't really notice any events, but we'll investigate that further.

Attached is a (compressed and somewhat sanitized) log4net log file for a search query which returned 2419 "Record" objects. The most notable gap is between 17:24:51,597 and  17:26:05,238, we suspect that's spent in ImportRowsFromTable(). This function calls HitSpans(), which seems to take most of its time; nevertheless, without spans, the whole operation took even longer.

                                        Gerben Vos.


---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Jim Arnold]

Hi Gerben,

The reason I asked if the table was read-only is that, if you are searching and reporting on lots of records that won't be manipulated in your application, then maybe Neo isn't the best choice!  That's probably not the answer you wanted, but I would be inclined to use something more lightweight than DataTables (which Neo is based on) for this scenario.  That's not to say that Neo isn't useful for the rest of your application, but you might want to use a custom query and object model here if you can't get the performance you need.

(Re: Oracle supporting GUIDs.  It may not have a GUID type, but if you are creating your own IDs you could use the System.Guid type and store them as RAW(16) in the database.  Might not help you in this case though).

Anyway, I will try to have a look at the spans code and see if I can find anything obviously slow.

Jim

"Gerben Vos" <Gerben@...>

19/06/2007 17:20

Please respond to
user@...

To	<user@...>
cc	"Siebe Bloembergen" <Siebe@...>, "Lieuwe Visser" <Lieuwe@...>
Subject	Re: [neo-user] Performance problems

Jim and Artem, thanks for your suggestions.

The tables are read/write. Although the action we're having problems with is a search and we only read the tables, we also need to do write an audit entry for each found item to a different table.

The string GUIDs are necessary because Oracle can't support binary GUIDs. I don't know why we haven't chosen some kind of auto-increment integer originally; probably also because different databases handle that differently.

We did some profiling. Most time is spent in the function ObjectContext.ImportRowsFromTable(). We didn't really notice any events, but we'll investigate that further.

Attached is a (compressed and somewhat sanitized) log4net log file for a search query which returned 2419 "Record" objects. The most notable gap is between 17:24:51,597 and  17:26:05,238, we suspect that's spent in ImportRowsFromTable(). This function calls HitSpans(), which seems to take most of its time; nevertheless, without spans, the whole operation took even longer.

                                                                                    Gerben Vos.
[attachment "zylog3.gz" deleted by Jim Arnold/UK/ThoughtWorks] ---------------------------------------------------------------------
To unsubscribe from this list please visit:

   http://xircles.codehaus.org/manage_email

[Artem-5]

Hi Jim,

Tuesday, June 19, 2007, 8:48:06 PM, you wrote:

>
> Anyway, I will try to have a look at the spans code and see if I
> can find anything obviously slow.
>  
> Jim

Suppose you have 1000 Order objects and 10 000 OrderDetail objects.
Suppose you fetch all Order objects with the "Details" span. Now you
have all your objects in memory, but you still have to populate the
Details relationship for each Order. This is what HitSpans does.

Basically, you save yourself 1000 database queries, but you still do
1000 in-memory queries. So, for each Order, you check all the
in-memory OrderDetail objects and attach whatever fits to the Details.
That makes 1000 x 10 000 operations! The problem is much worse with path
spans, like, you have Customers and "Orders.Details" span.

So, my suggestion is simple. Instead of checking each OrderDetail
against each Order, why not just take an OrderDetail, find the
corresponding Order, and add it to the relationship? Finding the
Details for an Order is slow, since we have to loop through the entire
collection and evaluate the qualifier. Finding the Order for a Detail
is fast, since we find it by its primary key.

So, we have now, sort of:
foreach Order order
        foreach OrderDetail detail
                if (detail.Order == order)  AddToRelationship(order.OrderDetails)

And I suggest:
foreach OrderDetail  detail
        AddToRelationship(detail.Order.OrderDetails)

Clearly, this is only 10 000 instead of 1000 x 10 000 operations.

Artem


---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Jim Arnold]

Artem,

I haven't thought much about this yet but it sounds reasonable.  I'm seeing Erik tomorrow so I'll talk to him about it, and hopefully I have some time next week to work on spans.

Jim

Artem <gamma@...>

20/06/2007 15:00

Please respond to
user@...

To	Jim Arnold <user@...>
cc
Subject	Re[2]: [neo-user] Performance problems

Hi Jim,

Tuesday, June 19, 2007, 8:48:06 PM, you wrote:

>
> Anyway, I will try to have a look at the spans code and see if I
> can find anything obviously slow.
>  
> Jim

Suppose you have 1000 Order objects and 10 000 OrderDetail objects.
Suppose you fetch all Order objects with the "Details" span. Now you
have all your objects in memory, but you still have to populate the
Details relationship for each Order. This is what HitSpans does.

Basically, you save yourself 1000 database queries, but you still do
1000 in-memory queries. So, for each Order, you check all the
in-memory OrderDetail objects and attach whatever fits to the Details.
That makes 1000 x 10 000 operations! The problem is much worse with path
spans, like, you have Customers and "Orders.Details" span.

So, my suggestion is simple. Instead of checking each OrderDetail
against each Order, why not just take an OrderDetail, find the
corresponding Order, and add it to the relationship? Finding the
Details for an Order is slow, since we have to loop through the entire
collection and evaluate the qualifier. Finding the Order for a Detail
is fast, since we find it by its primary key.

So, we have now, sort of:
foreach Order order
       foreach OrderDetail detail
               if (detail.Order == order)  AddToRelationship(order.OrderDetails)

And I suggest:
foreach OrderDetail  detail
       AddToRelationship(detail.Order.OrderDetails)

Clearly, this is only 10 000 instead of 1000 x 10 000 operations.

Artem


---------------------------------------------------------------------
To unsubscribe from this list please visit:

   http://xircles.codehaus.org/manage_email

[Artem-5]

Jim,

Wednesday, June 20, 2007, 6:10:00 PM, you wrote:

>  
> Artem,
>  
> I haven't thought much about this yet but it sounds reasonable.
>  I'm seeing Erik tomorrow so I'll talk to him about it, and
> hopefully I have some time next week to work on spans.
>  
> Jim

I'm so happy this project is not abandoned! Any chance to see it
developed further?

In fact, I use it a lot, and I've got a couple of suggestions as well
as bugs, and I'll be happy to contribute a bit.

Artem


---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[PC]

I have been using Neo on several projects for almost 2 years now, including a web framework. It would be great if Erik would resurrect the project. If not, perhaps he would give his blessing to a branch?

 

PCBender

 

On 6/20/07, Artem <gamma@...> wrote:

Jim,

Wednesday, June 20, 2007, 6:10:00 PM, you wrote:

>
> Artem,
>
> I haven't thought much about this yet but it sounds reasonable.
> I'm seeing Erik tomorrow so I'll talk to him about it, and
> hopefully I have some time next week to work on spans.
>
> Jim

I'm so happy this project is not abandoned! Any chance to see it
developed further?

In fact, I use it a lot, and I've got a couple of suggestions as well
as bugs, and I'll be happy to contribute a bit.

Artem


---------------------------------------------------------------------
To unsubscribe from this list please visit:

   http://xircles.codehaus.org/manage_email

[Chris Tinsley]

I have been using NEO in a high volume scenario for 3 years, and I would love to see some movement on it as well.  I'm willing to invest some time in contributing, but we do need some direction.  

I liked the work that was done to include generics in the code, but it never made its way to source control.

Chris Tinsley


________________________________________
From: PC Bender [mailto:pcbender@...]
Sent: Wednesday, June 20, 2007 4:07 PM
To: user@...
Subject: Re: Re[4]: [neo-user] Performance problems

I have been using Neo on several projects for almost 2 years now, including a web framework. It would be great if Erik would resurrect the project. If not, perhaps he would give his blessing to a branch?
 
PCBender

 
On 6/20/07, Artem <gamma@...> wrote:
Jim,

Wednesday, June 20, 2007, 6:10:00 PM, you wrote:

>
> Artem,
>
> I haven't thought much about this yet but it sounds reasonable.
> I'm seeing Erik tomorrow so I'll talk to him about it, and
> hopefully I have some time next week to work on spans.
>
> Jim

I'm so happy this project is not abandoned! Any chance to see it
developed further?

In fact, I use it a lot, and I've got a couple of suggestions as well
as bugs, and I'll be happy to contribute a bit.

Artem


---------------------------------------------------------------------
To unsubscribe from this list please visit:

   http://xircles.codehaus.org/manage_email


---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email