Performance problems: some more ideas

View: New views

2 Messages — Rating Filter: Alert me

Performance problems: some more ideas

by Gerben Vos :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Dear list,

continuing on our earlier performance problems which we told you
about a month or two ago, we have implemented the "loop inversion"
for the spans that Artem proposed. The code is still a bit a proof
of concept and not up to good coding standards yet, but I've appended
the patches so you can have a look. Unfortunately, it shaved only a
few percent off the time spent on our queries with spans.

I'm now wondering why ObjectContext.GetObjects() first
loads the SQL query results into a DataSet, and then calls
FetchObjectsFromObjectTable() to again search within that
DataSet using a loop and EvaluateWithObject() -- something the
database server basically has already done for us. Our code spends
extraordinary amounts of time inside FetchObjectsFromObjectTable().

I think it would make more sense to construct the result list
inside FetchObjectsFromStore() itself. It would need a little
extra administration, but I feel that that would easily outweigh
the time which is IMHO now wasted.

Is there any good reason why this is not done already? Maybe the call
to DbDataAdapter.Fill() inside DbDataStore.FillTable() somehow adds
unnecessary cruft into the DataTable that needs to be filtered out?
Or am I missing something else?

Any thoughts on this would be appreciated.

Gerben Vos,
ZyLAB Technologies.

--- C:/Documents and Settings/Gerben/Local Settings/Temp/TCVb4fc.tmp/ObjectRelationBase.1.1.cs Tue Jun 12 16:21:46 2007
+++ C:/Documents and Settings/Gerben/Local Settings/Temp/TCVb4fc.tmp/ObjectRelationBase.1.2.cs Tue Aug 21 11:59:06 2007
@@ -240,6 +240,53 @@
Load();
}

+ public virtual IList HitSpansHelper(IList objects, string pathElement) {
+ if (innerList == null) {
+
+ IList childObjects = Owner.Context.ObjectTable.GetObjects(foreignTableName);
+
+ if(childObjects == null) {
+ return new ArrayList();
+ }
+
+ IList addedChildObjects = new ArrayList();
+ foreach (IEntityObject childObject in childObjects) {
+ IEntityObject owner = Owner.Context.GetObjectFromTable(
+ Relation.ParentEntity.TableName,
+ new object[] { childObject.Row[foreignColumnName] }
+ );
+
+ if (owner == null) {
+ continue;
+ }
+
+ addedChildObjects.Add(childObject);
+ ObjectRelationBase relation = (ObjectRelationBase)ObjectHelper.GetPropertyOrField(owner, pathElement);
+ if (relation.innerList == null) {
+ relation.innerList = new ArrayList();
+ }
+
+ relation.innerList.Add(childObject);
+ }
+
+ foreach (IEntityObject eo2 in objects) {
+ object obj = ObjectHelper.GetPropertyOrField(eo2, pathElement);
+ ObjectRelationBase rel = obj as ObjectRelationBase;
+
+ rel.OnListChanged(new ListChangedEventArgs(ListChangedType.Reset, -1));
+ eo2.Context.RegisterForColumnChanges(new ColumnChangeHandler(OnColumnChanging), rel.foreignTableName, rel.foreignColumnName);
+ eo2.Context.RegisterForRowChanges(new RowChangeHandler(OnRowDeleting), rel.foreignTableName);
+
+ if (rel.onListChanged != null) {
+ rel.Owner.Context.EntityObjectChanged += new EntityObjectChangedHandler(rel.OnEntityObjectChanged);
+ }
+ }
+
+ return addedChildObjects;
+ }
+ return new ArrayList();
+ }
+
public virtual void InvalidateCache()
{
if(innerList == null)
--- C:/Documents and Settings/Gerben/Local Settings/Temp/TCV08dc.tmp/ObjectContext.1.1.cs Thu May 31 17:11:06 2007
+++ C:/Documents and Settings/Gerben/Local Settings/Temp/TCV08dc.tmp/ObjectContext.1.2.cs Tue Aug 21 12:56:56 2007
@@ -1369,31 +1369,40 @@
HitSpans(mainObjects, fetchSpec.Spans);
}

- private void HitSpans(IList objects, params string[] spans)
- {
+
+ private void HitSpans(IList objects, params string[] spans) {
bool didIgnoreStore = ignoresDataStore;
ignoresDataStore = true;
- foreach(IEntityObject eo in objects)
- {
- foreach(string span in spans)
- {
+
+ ArrayList nonRelationSpans = new ArrayList();
+ foreach(string span in spans) {
+ if (objects.Count > 0) {
+ IEntityObject eo = (IEntityObject)objects[0];
+
string[] pathElements = span.Split(new char[]{'.'}, 2);
- object obj = ObjectHelper.GetPropertyOrField(eo, pathElements[0]);
- IList list = null;
- ObjectRelationBase rel = obj as ObjectRelationBase;
- if(rel != null)
- {
- rel.Touch();
- list = rel;
- }
- else
- {
- list = new object[] { obj };
- }
- if(pathElements.Length > 1)
+ object obj0 = ObjectHelper.GetPropertyOrField(eo, pathElements[0]);
+ ObjectRelationBase rel0 = obj0 as ObjectRelationBase;
+ if(rel0 != null) {
+ IList list = rel0.HitSpansHelper(objects, pathElements[0]);
+ if (pathElements.Length > 1) {
HitSpans(list, pathElements[1]);
}
+ } else {
+ nonRelationSpans.Add(span);
+ }
+ }
}
+
+ foreach (string span in nonRelationSpans) {
+ foreach(IEntityObject eo in objects) {
+ string[] pathElements = span.Split(new char[]{'.'}, 2);
+ object obj0 = ObjectHelper.GetPropertyOrField(eo, pathElements[0]);
+ if(pathElements.Length > 1) {
+ HitSpans(new object[] { obj0 }, pathElements[1]);
+ }
+ }
+ }
+
ignoresDataStore = didIgnoreStore;
}

--- C:/Documents and Settings/Gerben/Local Settings/Temp/TCV08dc.tmp/ObjectContext.1.1.cs Thu May 31 17:11:06 2007
+++ C:/Documents and Settings/Gerben/Local Settings/Temp/TCV08dc.tmp/ObjectContext.1.2.cs Tue Aug 21 12:56:56 2007
@@ -1369,34 +1369,43 @@
HitSpans(mainObjects, fetchSpec.Spans);
}

- private void HitSpans(IList objects, params string[] spans)
- {
- bool didIgnoreStore = ignoresDataStore;
- ignoresDataStore = true;
- foreach(IEntityObject eo in objects)
- {
- foreach(string span in spans)
- {
- string[] pathElements = span.Split(new char[]{'.'}, 2);
- object obj = ObjectHelper.GetPropertyOrField(eo, pathElements[0]);
- IList list = null;
- ObjectRelationBase rel = obj as ObjectRelationBase;
- if(rel != null)
- {
- rel.Touch();
- list = rel;
- }
- else
- {
- list = new object[] { obj };
- }
- if(pathElements.Length > 1)
- HitSpans(list, pathElements[1]);
- }
- }
- ignoresDataStore = didIgnoreStore;
- }
-
+
+ private void HitSpans(IList objects, params string[] spans) {
+ bool didIgnoreStore = ignoresDataStore;
+ ignoresDataStore = true;
+
+ ArrayList nonRelationSpans = new ArrayList();
+ foreach(string span in spans) {
+ if (objects.Count > 0) {
+ IEntityObject eo = (IEntityObject)objects[0];
+
+ string[] pathElements = span.Split(new char[]{'.'}, 2);
+ object obj0 = ObjectHelper.GetPropertyOrField(eo, pathElements[0]);
+ ObjectRelationBase rel0 = obj0 as ObjectRelationBase;
+ if(rel0 != null) {
+ IList list = rel0.HitSpansHelper(objects, pathElements[0]);
+ if (pathElements.Length > 1) {
+ HitSpans(list, pathElements[1]);
+ }
+ } else {
+ nonRelationSpans.Add(span);
+ }
+ }
+ }
+
+ foreach (string span in nonRelationSpans) {
+ foreach(IEntityObject eo in objects) {
+ string[] pathElements = span.Split(new char[]{'.'}, 2);
+ object obj0 = ObjectHelper.GetPropertyOrField(eo, pathElements[0]);
+ if(pathElements.Length > 1) {
+ HitSpans(new object[] { obj0 }, pathElements[1]);
+ }
+ }
+ }
+
+ ignoresDataStore = didIgnoreStore;
+ }
+

//--------------------------------------------------------------------------------------
// IDataStore Impl

---------------------------------------------------------------------
To unsubscribe from this list please visit:

http://xircles.codehaus.org/manage_email

Re: Performance problems: some more ideas

by Artem-5 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Dear list,

The main reason why we do a double work (once while fetching the rows
and second evaluating the in-memory objects) is that we want to have
an in-memory database. There are numerous advantages: we can cache
results, we can use a disconnected scenario, we can create new objects
or alter existing ones and without saving they are going to
participate in the second stage of the query execution.

The main problem with in-memory evaluation is that it doesn't
implement indexing like the ordinary database. More precisely, it has
only indexing for primary keys (which is buggy, btw). So, the spans
problem could be solved if we implemented at least indexing for
foreign keys.

So, perhaps we could get involved with the project, or at least create
a branch?

Artem

Tuesday, August 21, 2007, 6:28:23 PM, you wrote:

> Dear list,

> continuing on our earlier performance problems which we told you
> about a month or two ago, we have implemented the "loop inversion"
> for the spans that Artem proposed. The code is still a bit a proof
> of concept and not up to good coding standards yet, but I've appended
> the patches so you can have a look. Unfortunately, it shaved only a
> few percent off the time spent on our queries with spans.

> I'm now wondering why ObjectContext.GetObjects() first
> loads the SQL query results into a DataSet, and then calls
> FetchObjectsFromObjectTable() to again search within that
> DataSet using a loop and EvaluateWithObject() -- something the
> database server basically has already done for us. Our code spends
> extraordinary amounts of time inside FetchObjectsFromObjectTable().

> I think it would make more sense to construct the result list
> inside FetchObjectsFromStore() itself. It would need a little
> extra administration, but I feel that that would easily outweigh
> the time which is IMHO now wasted.

> Is there any good reason why this is not done already? Maybe the call
> to DbDataAdapter.Fill() inside DbDataStore.FillTable() somehow adds
> unnecessary cruft into the DataTable that needs to be filtered out?
> Or am I missing something else?

> Any thoughts on this would be appreciated.

> Gerben Vos,
> ZyLAB Technologies.

---------------------------------------------------------------------
To unsubscribe from this list please visit:

http://xircles.codehaus.org/manage_email