personview changes

[Benny Malengier]

Nick,

I have been working on personview today,
http://www.gramps-project.org/bugs/view.php?id=3275
It looks nice, but I see a problem: on_iter_next is too slow.

I see two possible solutions:

1/do like in flatbasemodel, and add an index like the handle2index
map. This could be added as data in self.tree, as the third entry.
Then on_iter_next(iter) would do: index = tree[iter][2] +1, and see if
that exists in children to return it.
Problem is the same as in flatbasemodel: on insert and delete: update
the affected indexes. Not too bad, it is done in memory (see how
adding/deleting is quick is eg eventview), but still annoying.

2/go for a linked list approach. The linked list would be done with a
class Node, as in
http://stackoverflow.com/questions/280243/python-linked-list
In this setup, self.tree[iter] = (level, Node), where Node.key = iter,
and Node.next links to the next Node and
self.children[iter] = a list of Node which also form a correct linked list.
In this setup on_iter_next(iter) would do: retrun self.tree[iter][1].next()
The advantage over 1/ is that on insert and delete, no loops to update
indexes must be performed, but the code for insert/delete has to be
done once correctly :-)

Any preferences or other ideas?
To see that above is a problem in the present implementation, some timings:

GRAMPS 3.1 on 30000+ Persons:
8594: DEBUG: _PeopleModel.py: line 316: PeopleModel rebuild_data 4.24 sec
8595: DEBUG: _PeopleModel.py: line 296: PeopleModel __init__ 4.24 sec
8688: DEBUG: PersonView.py: line 555: PersonView build_tree 4.32 sec
8688: DEBUG: PersonView.py: line 556:  setting model 0.07 sec
39118: DEBUG: PersonView.py: line 773: PersonView person_removed 4.65 sec
56205: DEBUG: PersonView.py: line 746: PersonView person_added 4.75 sec

GRAMSP trunk with my patchset:
8281: DEBUG: treebasemodel.py: line 285: PeopleModel rebuild_data 4.72 sec
8281: DEBUG: treebasemodel.py: line 169: PeopleModel __init__ 4.72 sec
11897: DEBUG: listview.py: line 256: PersonView build_tree 8.29 sec
11897: DEBUG: listview.py: line 261: parts 4.72 , 0.01 , 3.56 , 0.0 , 0.0
60672: DEBUG: listview.py: line 624:    PersonView row_add 0.0 sec
72629: DEBUG: treebasemodel.py: line 465: PeopleModel add_row_by_handle 0.01 sec
179444: DEBUG: treebasemodel.py: line 485: PeopleModel
delete_row_by_handle 0.11 sec
179444: DEBUG: listview.py: line 655:    PersonView row_delete 0.11 sec

So the row add and row delete are now instantanious (5 sec to 0.1sec)
which is what we want to achieve, building the data is a bit more
expensive (4.24 to 4.72) which is completely acceptable, but showing
the treeview itself (which happens on attach of the model) goes from
0.07 sec to 3.56.
In other words, we need something as good as the old NodeTreeMap with
it's path2iter and iter2path to make the iter operations faster.
Sorting on the columns also suffers from this.

Some other attention points:
1/ remove_node should use del, not a temporary list. Should also be
recusive as deleting a leaf can mean removing all upper groups.
2/ I'd like to keep in memory the entire map so as not to hit the
database on search and filter, just like in the flatbasemodel
3/ not sure how reverse works with this rows_reordered call. This is
not done in flatbasemodel and seems to work fine. Here setting up the
sequence for call to rows_reodered takes quite some work.

My preference goes at the moment towards committing this patch, and
then doing further changes. The patchset just becomes too large to
handle otherwise.

Benny

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Gramps-devel mailing list
Gramps-devel@...
https://lists.sourceforge.net/lists/listinfo/gramps-devel

[Gerald Britton-2]

Just thinking out loud here.  What about using dictionaries instead of
lists?  You'd have to use some sort of compound key for uniqueness and
ordering, I suppose, and it would use more memory.  All operations are
amortized O(1) though.

On Fri, Nov 6, 2009 at 4:04 PM, Benny Malengier
<benny.malengier@...> wrote:

> Nick,
>
> I have been working on personview today,
> http://www.gramps-project.org/bugs/view.php?id=3275
> It looks nice, but I see a problem: on_iter_next is too slow.
>
> I see two possible solutions:
>
> 1/do like in flatbasemodel, and add an index like the handle2index
> map. This could be added as data in self.tree, as the third entry.
> Then on_iter_next(iter) would do: index = tree[iter][2] +1, and see if
> that exists in children to return it.
> Problem is the same as in flatbasemodel: on insert and delete: update
> the affected indexes. Not too bad, it is done in memory (see how
> adding/deleting is quick is eg eventview), but still annoying.
>
> 2/go for a linked list approach. The linked list would be done with a
> class Node, as in
> http://stackoverflow.com/questions/280243/python-linked-list
> In this setup, self.tree[iter] = (level, Node), where Node.key = iter,
> and Node.next links to the next Node and
> self.children[iter] = a list of Node which also form a correct linked list.
> In this setup on_iter_next(iter) would do: retrun self.tree[iter][1].next()
> The advantage over 1/ is that on insert and delete, no loops to update
> indexes must be performed, but the code for insert/delete has to be
> done once correctly :-)
>
> Any preferences or other ideas?
> To see that above is a problem in the present implementation, some timings:
>
> GRAMPS 3.1 on 30000+ Persons:
> 8594: DEBUG: _PeopleModel.py: line 316: PeopleModel rebuild_data 4.24 sec
> 8595: DEBUG: _PeopleModel.py: line 296: PeopleModel __init__ 4.24 sec
> 8688: DEBUG: PersonView.py: line 555: PersonView build_tree 4.32 sec
> 8688: DEBUG: PersonView.py: line 556:  setting model 0.07 sec
> 39118: DEBUG: PersonView.py: line 773: PersonView person_removed 4.65 sec
> 56205: DEBUG: PersonView.py: line 746: PersonView person_added 4.75 sec
>
> GRAMSP trunk with my patchset:
> 8281: DEBUG: treebasemodel.py: line 285: PeopleModel rebuild_data 4.72 sec
> 8281: DEBUG: treebasemodel.py: line 169: PeopleModel __init__ 4.72 sec
> 11897: DEBUG: listview.py: line 256: PersonView build_tree 8.29 sec
> 11897: DEBUG: listview.py: line 261: parts 4.72 , 0.01 , 3.56 , 0.0 , 0.0
> 60672: DEBUG: listview.py: line 624:    PersonView row_add 0.0 sec
> 72629: DEBUG: treebasemodel.py: line 465: PeopleModel add_row_by_handle 0.01 sec
> 179444: DEBUG: treebasemodel.py: line 485: PeopleModel
> delete_row_by_handle 0.11 sec
> 179444: DEBUG: listview.py: line 655:    PersonView row_delete 0.11 sec
>
> So the row add and row delete are now instantanious (5 sec to 0.1sec)
> which is what we want to achieve, building the data is a bit more
> expensive (4.24 to 4.72) which is completely acceptable, but showing
> the treeview itself (which happens on attach of the model) goes from
> 0.07 sec to 3.56.
> In other words, we need something as good as the old NodeTreeMap with
> it's path2iter and iter2path to make the iter operations faster.
> Sorting on the columns also suffers from this.
>
> Some other attention points:
> 1/ remove_node should use del, not a temporary list. Should also be
> recusive as deleting a leaf can mean removing all upper groups.
> 2/ I'd like to keep in memory the entire map so as not to hit the
> database on search and filter, just like in the flatbasemodel
> 3/ not sure how reverse works with this rows_reordered call. This is
> not done in flatbasemodel and seems to work fine. Here setting up the
> sequence for call to rows_reodered takes quite some work.
>
> My preference goes at the moment towards committing this patch, and
> then doing further changes. The patchset just becomes too large to
> handle otherwise.
>
> Benny
>
> ------------------------------------------------------------------------------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
> trial. Simplify your report design, integration and deployment - and focus on
> what you do best, core application coding. Discover what's new with
> Crystal Reports now.  http://p.sf.net/sfu/bobj-july
> _______________________________________________
> Gramps-devel mailing list
> Gramps-devel@...
> https://lists.sourceforge.net/lists/listinfo/gramps-devel
>

--
Gerald Britton

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Gramps-devel mailing list
Gramps-devel@...
https://lists.sourceforge.net/lists/listinfo/gramps-devel

[Benny Malengier]

2009/11/6 Gerald Britton <gerald.britton@...>:
> Just thinking out loud here.  What about using dictionaries instead of
> lists?  You'd have to use some sort of compound key for uniqueness and
> ordering, I suppose, and it would use more memory.  All operations are
> amortized O(1) though.

The key retrieval things are already dictionaries. The parts that need
an ordering are in lists. You need to store the order as you need to
know what is the item on path [400, 124], so group node 400, subnode
124.

Benny

>
> On Fri, Nov 6, 2009 at 4:04 PM, Benny Malengier
> <benny.malengier@...> wrote:
>> Nick,
>>
>> I have been working on personview today,
>> http://www.gramps-project.org/bugs/view.php?id=3275
>> It looks nice, but I see a problem: on_iter_next is too slow.
>>
>> I see two possible solutions:
>>
>> 1/do like in flatbasemodel, and add an index like the handle2index
>> map. This could be added as data in self.tree, as the third entry.
>> Then on_iter_next(iter) would do: index = tree[iter][2] +1, and see if
>> that exists in children to return it.
>> Problem is the same as in flatbasemodel: on insert and delete: update
>> the affected indexes. Not too bad, it is done in memory (see how
>> adding/deleting is quick is eg eventview), but still annoying.
>>
>> 2/go for a linked list approach. The linked list would be done with a
>> class Node, as in
>> http://stackoverflow.com/questions/280243/python-linked-list
>> In this setup, self.tree[iter] = (level, Node), where Node.key = iter,
>> and Node.next links to the next Node and
>> self.children[iter] = a list of Node which also form a correct linked list.
>> In this setup on_iter_next(iter) would do: retrun self.tree[iter][1].next()
>> The advantage over 1/ is that on insert and delete, no loops to update
>> indexes must be performed, but the code for insert/delete has to be
>> done once correctly :-)
>>
>> Any preferences or other ideas?
>> To see that above is a problem in the present implementation, some timings:
>>
>> GRAMPS 3.1 on 30000+ Persons:
>> 8594: DEBUG: _PeopleModel.py: line 316: PeopleModel rebuild_data 4.24 sec
>> 8595: DEBUG: _PeopleModel.py: line 296: PeopleModel __init__ 4.24 sec
>> 8688: DEBUG: PersonView.py: line 555: PersonView build_tree 4.32 sec
>> 8688: DEBUG: PersonView.py: line 556:  setting model 0.07 sec
>> 39118: DEBUG: PersonView.py: line 773: PersonView person_removed 4.65 sec
>> 56205: DEBUG: PersonView.py: line 746: PersonView person_added 4.75 sec
>>
>> GRAMSP trunk with my patchset:
>> 8281: DEBUG: treebasemodel.py: line 285: PeopleModel rebuild_data 4.72 sec
>> 8281: DEBUG: treebasemodel.py: line 169: PeopleModel __init__ 4.72 sec
>> 11897: DEBUG: listview.py: line 256: PersonView build_tree 8.29 sec
>> 11897: DEBUG: listview.py: line 261: parts 4.72 , 0.01 , 3.56 , 0.0 , 0.0
>> 60672: DEBUG: listview.py: line 624:    PersonView row_add 0.0 sec
>> 72629: DEBUG: treebasemodel.py: line 465: PeopleModel add_row_by_handle 0.01 sec
>> 179444: DEBUG: treebasemodel.py: line 485: PeopleModel
>> delete_row_by_handle 0.11 sec
>> 179444: DEBUG: listview.py: line 655:    PersonView row_delete 0.11 sec
>>
>> So the row add and row delete are now instantanious (5 sec to 0.1sec)
>> which is what we want to achieve, building the data is a bit more
>> expensive (4.24 to 4.72) which is completely acceptable, but showing
>> the treeview itself (which happens on attach of the model) goes from
>> 0.07 sec to 3.56.
>> In other words, we need something as good as the old NodeTreeMap with
>> it's path2iter and iter2path to make the iter operations faster.
>> Sorting on the columns also suffers from this.
>>
>> Some other attention points:
>> 1/ remove_node should use del, not a temporary list. Should also be
>> recusive as deleting a leaf can mean removing all upper groups.
>> 2/ I'd like to keep in memory the entire map so as not to hit the
>> database on search and filter, just like in the flatbasemodel
>> 3/ not sure how reverse works with this rows_reordered call. This is
>> not done in flatbasemodel and seems to work fine. Here setting up the
>> sequence for call to rows_reodered takes quite some work.
>>
>> My preference goes at the moment towards committing this patch, and
>> then doing further changes. The patchset just becomes too large to
>> handle otherwise.
>>
>> Benny
>>
>> ------------------------------------------------------------------------------
>> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
>> trial. Simplify your report design, integration and deployment - and focus on
>> what you do best, core application coding. Discover what's new with
>> Crystal Reports now.  http://p.sf.net/sfu/bobj-july
>> _______________________________________________
>> Gramps-devel mailing list
>> Gramps-devel@...
>> https://lists.sourceforge.net/lists/listinfo/gramps-devel
>>
>
>
>
> --
> Gerald Britton
>

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Gramps-devel mailing list
Gramps-devel@...
https://lists.sourceforge.net/lists/listinfo/gramps-devel

[Benny Malengier]

These changes are now in trunk. This is however not the final version.
Commit 13515 does the following:

Personview is now derives of the same classes as the other views, so a
very nice code cleanup
Personview now uses a generic treebasemodel, that in the future will
be reused for other views.
Personview now allows column sorting.

There are some dramatic slowdowns as to the old code, as well as speed
ups. What is slower:
* showing person view is two times slower
* selecting a person is very slow on large family trees
Dramatically faster is however insert and delete which no longer
depend on the size of the tree, so common actions that happen a lot
during workflow.

The following work will now be done:

* fix column order dialog: group columns may not be reordered or
removed, so the column dialog must be aware of this
* increase speed, I intend to try linked lists instead of the index to
path and path to index maps we used to have. It should be cleaner and
a lot faster. If this works out, also the flatbasemodel will use it,
so as to have one way of coding.
* increase speed of filter/search on a build tree by keeping the full
tree in memory.
* make views plugins, so we can have both tree personview and flat
personview. My idea now is to have one person selection in the
sidebar, and then on the toolbar reserve a section to switch views of
the same category.

Feel free to jump in.

Benny

2009/11/6 Benny Malengier <benny.malengier@...>:

> Nick,
>
> I have been working on personview today,
> http://www.gramps-project.org/bugs/view.php?id=3275
> It looks nice, but I see a problem: on_iter_next is too slow.
>
> I see two possible solutions:
>
> 1/do like in flatbasemodel, and add an index like the handle2index
> map. This could be added as data in self.tree, as the third entry.
> Then on_iter_next(iter) would do: index = tree[iter][2] +1, and see if
> that exists in children to return it.
> Problem is the same as in flatbasemodel: on insert and delete: update
> the affected indexes. Not too bad, it is done in memory (see how
> adding/deleting is quick is eg eventview), but still annoying.
>
> 2/go for a linked list approach. The linked list would be done with a
> class Node, as in
> http://stackoverflow.com/questions/280243/python-linked-list
> In this setup, self.tree[iter] = (level, Node), where Node.key = iter,
> and Node.next links to the next Node and
> self.children[iter] = a list of Node which also form a correct linked list.
> In this setup on_iter_next(iter) would do: retrun self.tree[iter][1].next()
> The advantage over 1/ is that on insert and delete, no loops to update
> indexes must be performed, but the code for insert/delete has to be
> done once correctly :-)
>
> Any preferences or other ideas?
> To see that above is a problem in the present implementation, some timings:
>
> GRAMPS 3.1 on 30000+ Persons:
> 8594: DEBUG: _PeopleModel.py: line 316: PeopleModel rebuild_data 4.24 sec
> 8595: DEBUG: _PeopleModel.py: line 296: PeopleModel __init__ 4.24 sec
> 8688: DEBUG: PersonView.py: line 555: PersonView build_tree 4.32 sec
> 8688: DEBUG: PersonView.py: line 556:  setting model 0.07 sec
> 39118: DEBUG: PersonView.py: line 773: PersonView person_removed 4.65 sec
> 56205: DEBUG: PersonView.py: line 746: PersonView person_added 4.75 sec
>
> GRAMSP trunk with my patchset:
> 8281: DEBUG: treebasemodel.py: line 285: PeopleModel rebuild_data 4.72 sec
> 8281: DEBUG: treebasemodel.py: line 169: PeopleModel __init__ 4.72 sec
> 11897: DEBUG: listview.py: line 256: PersonView build_tree 8.29 sec
> 11897: DEBUG: listview.py: line 261: parts 4.72 , 0.01 , 3.56 , 0.0 , 0.0
> 60672: DEBUG: listview.py: line 624:    PersonView row_add 0.0 sec
> 72629: DEBUG: treebasemodel.py: line 465: PeopleModel add_row_by_handle 0.01 sec
> 179444: DEBUG: treebasemodel.py: line 485: PeopleModel
> delete_row_by_handle 0.11 sec
> 179444: DEBUG: listview.py: line 655:    PersonView row_delete 0.11 sec
>
> So the row add and row delete are now instantanious (5 sec to 0.1sec)
> which is what we want to achieve, building the data is a bit more
> expensive (4.24 to 4.72) which is completely acceptable, but showing
> the treeview itself (which happens on attach of the model) goes from
> 0.07 sec to 3.56.
> In other words, we need something as good as the old NodeTreeMap with
> it's path2iter and iter2path to make the iter operations faster.
> Sorting on the columns also suffers from this.
>
> Some other attention points:
> 1/ remove_node should use del, not a temporary list. Should also be
> recusive as deleting a leaf can mean removing all upper groups.
> 2/ I'd like to keep in memory the entire map so as not to hit the
> database on search and filter, just like in the flatbasemodel
> 3/ not sure how reverse works with this rows_reordered call. This is
> not done in flatbasemodel and seems to work fine. Here setting up the
> sequence for call to rows_reodered takes quite some work.
>
> My preference goes at the moment towards committing this patch, and
> then doing further changes. The patchset just becomes too large to
> handle otherwise.
>
> Benny
>

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Gramps-devel mailing list
Gramps-devel@...
https://lists.sourceforge.net/lists/listinfo/gramps-devel

[Benny Malengier]

For those people who want to see timings themselves, starting in trunk
and branch31 gramps with the command:

python src/gramps.py --debug='.gui.treebasemodel'
--debug='.gui.personview' --debug='.gui.peoplemodel'
--debug='.gui.listview'

prints debug info with timings to the terminal.

Benny


2009/11/7 Benny Malengier <benny.malengier@...>:

> These changes are now in trunk. This is however not the final version.
> Commit 13515 does the following:
>
> Personview is now derives of the same classes as the other views, so a
> very nice code cleanup
> Personview now uses a generic treebasemodel, that in the future will
> be reused for other views.
> Personview now allows column sorting.
>
> There are some dramatic slowdowns as to the old code, as well as speed
> ups. What is slower:
> * showing person view is two times slower
> * selecting a person is very slow on large family trees
> Dramatically faster is however insert and delete which no longer
> depend on the size of the tree, so common actions that happen a lot
> during workflow.
>
> The following work will now be done:
>
> * fix column order dialog: group columns may not be reordered or
> removed, so the column dialog must be aware of this
> * increase speed, I intend to try linked lists instead of the index to
> path and path to index maps we used to have. It should be cleaner and
> a lot faster. If this works out, also the flatbasemodel will use it,
> so as to have one way of coding.
> * increase speed of filter/search on a build tree by keeping the full
> tree in memory.
> * make views plugins, so we can have both tree personview and flat
> personview. My idea now is to have one person selection in the
> sidebar, and then on the toolbar reserve a section to switch views of
> the same category.
>
> Feel free to jump in.
>
> Benny
>
> 2009/11/6 Benny Malengier <benny.malengier@...>:
>> Nick,
>>
>> I have been working on personview today,
>> http://www.gramps-project.org/bugs/view.php?id=3275
>> It looks nice, but I see a problem: on_iter_next is too slow.
>>
>> I see two possible solutions:
>>
>> 1/do like in flatbasemodel, and add an index like the handle2index
>> map. This could be added as data in self.tree, as the third entry.
>> Then on_iter_next(iter) would do: index = tree[iter][2] +1, and see if
>> that exists in children to return it.
>> Problem is the same as in flatbasemodel: on insert and delete: update
>> the affected indexes. Not too bad, it is done in memory (see how
>> adding/deleting is quick is eg eventview), but still annoying.
>>
>> 2/go for a linked list approach. The linked list would be done with a
>> class Node, as in
>> http://stackoverflow.com/questions/280243/python-linked-list
>> In this setup, self.tree[iter] = (level, Node), where Node.key = iter,
>> and Node.next links to the next Node and
>> self.children[iter] = a list of Node which also form a correct linked list.
>> In this setup on_iter_next(iter) would do: retrun self.tree[iter][1].next()
>> The advantage over 1/ is that on insert and delete, no loops to update
>> indexes must be performed, but the code for insert/delete has to be
>> done once correctly :-)
>>
>> Any preferences or other ideas?
>> To see that above is a problem in the present implementation, some timings:
>>
>> GRAMPS 3.1 on 30000+ Persons:
>> 8594: DEBUG: _PeopleModel.py: line 316: PeopleModel rebuild_data 4.24 sec
>> 8595: DEBUG: _PeopleModel.py: line 296: PeopleModel __init__ 4.24 sec
>> 8688: DEBUG: PersonView.py: line 555: PersonView build_tree 4.32 sec
>> 8688: DEBUG: PersonView.py: line 556:  setting model 0.07 sec
>> 39118: DEBUG: PersonView.py: line 773: PersonView person_removed 4.65 sec
>> 56205: DEBUG: PersonView.py: line 746: PersonView person_added 4.75 sec
>>
>> GRAMSP trunk with my patchset:
>> 8281: DEBUG: treebasemodel.py: line 285: PeopleModel rebuild_data 4.72 sec
>> 8281: DEBUG: treebasemodel.py: line 169: PeopleModel __init__ 4.72 sec
>> 11897: DEBUG: listview.py: line 256: PersonView build_tree 8.29 sec
>> 11897: DEBUG: listview.py: line 261: parts 4.72 , 0.01 , 3.56 , 0.0 , 0.0
>> 60672: DEBUG: listview.py: line 624:    PersonView row_add 0.0 sec
>> 72629: DEBUG: treebasemodel.py: line 465: PeopleModel add_row_by_handle 0.01 sec
>> 179444: DEBUG: treebasemodel.py: line 485: PeopleModel
>> delete_row_by_handle 0.11 sec
>> 179444: DEBUG: listview.py: line 655:    PersonView row_delete 0.11 sec
>>
>> So the row add and row delete are now instantanious (5 sec to 0.1sec)
>> which is what we want to achieve, building the data is a bit more
>> expensive (4.24 to 4.72) which is completely acceptable, but showing
>> the treeview itself (which happens on attach of the model) goes from
>> 0.07 sec to 3.56.
>> In other words, we need something as good as the old NodeTreeMap with
>> it's path2iter and iter2path to make the iter operations faster.
>> Sorting on the columns also suffers from this.
>>
>> Some other attention points:
>> 1/ remove_node should use del, not a temporary list. Should also be
>> recusive as deleting a leaf can mean removing all upper groups.
>> 2/ I'd like to keep in memory the entire map so as not to hit the
>> database on search and filter, just like in the flatbasemodel
>> 3/ not sure how reverse works with this rows_reordered call. This is
>> not done in flatbasemodel and seems to work fine. Here setting up the
>> sequence for call to rows_reodered takes quite some work.
>>
>> My preference goes at the moment towards committing this patch, and
>> then doing further changes. The patchset just becomes too large to
>> handle otherwise.
>>
>> Benny
>>
>

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Gramps-devel mailing list
Gramps-devel@...
https://lists.sourceforge.net/lists/listinfo/gramps-devel

[Nick Hall-6]

Benny Malengier wrote:
> Nick,
>
> I have been working on personview today,
> http://www.gramps-project.org/bugs/view.php?id=3275
> It looks nice, but I see a problem: on_iter_next is too slow.
>
>  
The 2 functions called by the treeview the most are on_iter_next and
on_get_value.  I don't  understand why on_iter_next is called so much,
the treeview seems to call it far more times than is needed for the
displayed part of the tree.  Having said that, there is nothing we can
do about it so we need to speed up on_iter_next.

> I see two possible solutions:
>
> 1/do like in flatbasemodel, and add an index like the handle2index
> map. This could be added as data in self.tree, as the third entry.
> Then on_iter_next(iter) would do: index = tree[iter][2] +1, and see if
> that exists in children to return it.
> Problem is the same as in flatbasemodel: on insert and delete: update
> the affected indexes. Not too bad, it is done in memory (see how
> adding/deleting is quick is eg eventview), but still annoying.
>
>  

Yes, this is like having a flatbasemodel for every parent node in the
tree.  It should have similar performance implications to the flat model.

> 2/go for a linked list approach. The linked list would be done with a
> class Node, as in
> http://stackoverflow.com/questions/280243/python-linked-list
> In this setup, self.tree[iter] = (level, Node), where Node.key = iter,
> and Node.next links to the next Node and
> self.children[iter] = a list of Node which also form a correct linked list.
> In this setup on_iter_next(iter) would do: retrun self.tree[iter][1].next()
> The advantage over 1/ is that on insert and delete, no loops to update
> indexes must be performed, but the code for insert/delete has to be
> done once correctly :-)
>
>  

This would involve more work but I think it is worth a try.  The gtk
ListStore is implemented using linked lists.  I almost started coding a
linked list implementation but then thought it was better to wait for
your comments first.

Do you want me to code this or do you want to do it yourself?

> Any preferences or other ideas?
> To see that above is a problem in the present implementation, some timings:
>
> GRAMPS 3.1 on 30000+ Persons:
> 8594: DEBUG: _PeopleModel.py: line 316: PeopleModel rebuild_data 4.24 sec
> 8595: DEBUG: _PeopleModel.py: line 296: PeopleModel __init__ 4.24 sec
> 8688: DEBUG: PersonView.py: line 555: PersonView build_tree 4.32 sec
> 8688: DEBUG: PersonView.py: line 556:  setting model 0.07 sec
> 39118: DEBUG: PersonView.py: line 773: PersonView person_removed 4.65 sec
> 56205: DEBUG: PersonView.py: line 746: PersonView person_added 4.75 sec
>
> GRAMSP trunk with my patchset:
> 8281: DEBUG: treebasemodel.py: line 285: PeopleModel rebuild_data 4.72 sec
> 8281: DEBUG: treebasemodel.py: line 169: PeopleModel __init__ 4.72 sec
> 11897: DEBUG: listview.py: line 256: PersonView build_tree 8.29 sec
> 11897: DEBUG: listview.py: line 261: parts 4.72 , 0.01 , 3.56 , 0.0 , 0.0
> 60672: DEBUG: listview.py: line 624:    PersonView row_add 0.0 sec
> 72629: DEBUG: treebasemodel.py: line 465: PeopleModel add_row_by_handle 0.01 sec
> 179444: DEBUG: treebasemodel.py: line 485: PeopleModel
> delete_row_by_handle 0.11 sec
> 179444: DEBUG: listview.py: line 655:    PersonView row_delete 0.11 sec
>
> So the row add and row delete are now instantanious (5 sec to 0.1sec)
> which is what we want to achieve, building the data is a bit more
> expensive (4.24 to 4.72) which is completely acceptable, but showing
> the treeview itself (which happens on attach of the model) goes from
> 0.07 sec to 3.56.
> In other words, we need something as good as the old NodeTreeMap with
> it's path2iter and iter2path to make the iter operations faster.
> Sorting on the columns also suffers from this.
>
>  

Your timings show similar results to mine.  I am using smaller data sets
though.

> Some other attention points:
> 1/ remove_node should use del, not a temporary list. Should also be
> recusive as deleting a leaf can mean removing all upper groups.
>  
Both adding and deleting will involve more than one node, in general.

> 2/ I'd like to keep in memory the entire map so as not to hit the
> database on search and filter, just like in the flatbasemodel
>  
I was always intending to do this but I wanted to get the basic design
right first before optimizing.  Your optimization for personview was
necessary though in order for us to be able to do proper performance
comparisons between versions.

> 3/ not sure how reverse works with this rows_reordered call. This is
> not done in flatbasemodel and seems to work fine. Here setting up the
> sequence for call to rows_reodered takes quite some work.
>
>  
When the order of rows in the model changes it needs to tell the
treeview so that it can update the display.  I think that rows_reordered
is the way it should be done.  It doesn't seem to work without it - this
is why I was removing and re-adding the model in one version of my old
code.  I don't know how we get away with it in the flat model.

> My preference goes at the moment towards committing this patch, and
> then doing further changes. The patchset just becomes too large to
> handle otherwise.
>
>  
I see you have done this now.  Good idea.


> Benny
>
>
>  
Regards,

Nick.


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Gramps-devel mailing list
Gramps-devel@...
https://lists.sourceforge.net/lists/listinfo/gramps-devel

[Doug Blank-2]

On Sat, Nov 7, 2009 at 8:11 AM, Benny Malengier
<benny.malengier@...> wrote:

> These changes are now in trunk. This is however not the final version.
> Commit 13515 does the following:
>
> Personview is now derives of the same classes as the other views, so a
> very nice code cleanup
> Personview now uses a generic treebasemodel, that in the future will
> be reused for other views.
> Personview now allows column sorting.
>
> There are some dramatic slowdowns as to the old code, as well as speed
> ups. What is slower:
> * showing person view is two times slower
> * selecting a person is very slow on large family trees
> Dramatically faster is however insert and delete which no longer
> depend on the size of the tree, so common actions that happen a lot
> during workflow.
>
> The following work will now be done:
>
> * fix column order dialog: group columns may not be reordered or
> removed, so the column dialog must be aware of this

How does one use an old Family Tree with changes to the personview
columns? The new code seems to work on newly created trees, but shows
the wrong columns with old data. How can I reset my edited personview
columns? Where is that info stored? I tried ./autogen.sh and deleting
.gramps/*.ini. Or is there something we can do to the old data before
apply the patch that will still make those trees work with the new
view code?

-Doug

> * increase speed, I intend to try linked lists instead of the index to
> path and path to index maps we used to have. It should be cleaner and
> a lot faster. If this works out, also the flatbasemodel will use it,
> so as to have one way of coding.
> * increase speed of filter/search on a build tree by keeping the full
> tree in memory.
> * make views plugins, so we can have both tree personview and flat
> personview. My idea now is to have one person selection in the
> sidebar, and then on the toolbar reserve a section to switch views of
> the same category.
>
> Feel free to jump in.
>
> Benny
>
> 2009/11/6 Benny Malengier <benny.malengier@...>:
>> Nick,
>>
>> I have been working on personview today,
>> http://www.gramps-project.org/bugs/view.php?id=3275
>> It looks nice, but I see a problem: on_iter_next is too slow.
>>
>> I see two possible solutions:
>>
>> 1/do like in flatbasemodel, and add an index like the handle2index
>> map. This could be added as data in self.tree, as the third entry.
>> Then on_iter_next(iter) would do: index = tree[iter][2] +1, and see if
>> that exists in children to return it.
>> Problem is the same as in flatbasemodel: on insert and delete: update
>> the affected indexes. Not too bad, it is done in memory (see how
>> adding/deleting is quick is eg eventview), but still annoying.
>>
>> 2/go for a linked list approach. The linked list would be done with a
>> class Node, as in
>> http://stackoverflow.com/questions/280243/python-linked-list
>> In this setup, self.tree[iter] = (level, Node), where Node.key = iter,
>> and Node.next links to the next Node and
>> self.children[iter] = a list of Node which also form a correct linked list.
>> In this setup on_iter_next(iter) would do: retrun self.tree[iter][1].next()
>> The advantage over 1/ is that on insert and delete, no loops to update
>> indexes must be performed, but the code for insert/delete has to be
>> done once correctly :-)
>>
>> Any preferences or other ideas?
>> To see that above is a problem in the present implementation, some timings:
>>
>> GRAMPS 3.1 on 30000+ Persons:
>> 8594: DEBUG: _PeopleModel.py: line 316: PeopleModel rebuild_data 4.24 sec
>> 8595: DEBUG: _PeopleModel.py: line 296: PeopleModel __init__ 4.24 sec
>> 8688: DEBUG: PersonView.py: line 555: PersonView build_tree 4.32 sec
>> 8688: DEBUG: PersonView.py: line 556:  setting model 0.07 sec
>> 39118: DEBUG: PersonView.py: line 773: PersonView person_removed 4.65 sec
>> 56205: DEBUG: PersonView.py: line 746: PersonView person_added 4.75 sec
>>
>> GRAMSP trunk with my patchset:
>> 8281: DEBUG: treebasemodel.py: line 285: PeopleModel rebuild_data 4.72 sec
>> 8281: DEBUG: treebasemodel.py: line 169: PeopleModel __init__ 4.72 sec
>> 11897: DEBUG: listview.py: line 256: PersonView build_tree 8.29 sec
>> 11897: DEBUG: listview.py: line 261: parts 4.72 , 0.01 , 3.56 , 0.0 , 0.0
>> 60672: DEBUG: listview.py: line 624:    PersonView row_add 0.0 sec
>> 72629: DEBUG: treebasemodel.py: line 465: PeopleModel add_row_by_handle 0.01 sec
>> 179444: DEBUG: treebasemodel.py: line 485: PeopleModel
>> delete_row_by_handle 0.11 sec
>> 179444: DEBUG: listview.py: line 655:    PersonView row_delete 0.11 sec
>>
>> So the row add and row delete are now instantanious (5 sec to 0.1sec)
>> which is what we want to achieve, building the data is a bit more
>> expensive (4.24 to 4.72) which is completely acceptable, but showing
>> the treeview itself (which happens on attach of the model) goes from
>> 0.07 sec to 3.56.
>> In other words, we need something as good as the old NodeTreeMap with
>> it's path2iter and iter2path to make the iter operations faster.
>> Sorting on the columns also suffers from this.
>>
>> Some other attention points:
>> 1/ remove_node should use del, not a temporary list. Should also be
>> recusive as deleting a leaf can mean removing all upper groups.
>> 2/ I'd like to keep in memory the entire map so as not to hit the
>> database on search and filter, just like in the flatbasemodel
>> 3/ not sure how reverse works with this rows_reordered call. This is
>> not done in flatbasemodel and seems to work fine. Here setting up the
>> sequence for call to rows_reodered takes quite some work.
>>
>> My preference goes at the moment towards committing this patch, and
>> then doing further changes. The patchset just becomes too large to
>> handle otherwise.
>>
>> Benny
>>
>
> ------------------------------------------------------------------------------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
> trial. Simplify your report design, integration and deployment - and focus on
> what you do best, core application coding. Discover what's new with
> Crystal Reports now.  http://p.sf.net/sfu/bobj-july
> _______________________________________________
> Gramps-devel mailing list
> Gramps-devel@...
> https://lists.sourceforge.net/lists/listinfo/gramps-devel
>

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Gramps-devel mailing list
Gramps-devel@...
https://lists.sourceforge.net/lists/listinfo/gramps-devel

[Benny Malengier]

Ok,

this is a serious issue. It will mean a database upgrade I think, as
this is stored in gen/read.py
Just edit  get_person_column_order once to return the default order

Benny

2009/11/7 Doug Blank <doug.blank@...>:

> On Sat, Nov 7, 2009 at 8:11 AM, Benny Malengier
> <benny.malengier@...> wrote:
>> These changes are now in trunk. This is however not the final version.
>> Commit 13515 does the following:
>>
>> Personview is now derives of the same classes as the other views, so a
>> very nice code cleanup
>> Personview now uses a generic treebasemodel, that in the future will
>> be reused for other views.
>> Personview now allows column sorting.
>>
>> There are some dramatic slowdowns as to the old code, as well as speed
>> ups. What is slower:
>> * showing person view is two times slower
>> * selecting a person is very slow on large family trees
>> Dramatically faster is however insert and delete which no longer
>> depend on the size of the tree, so common actions that happen a lot
>> during workflow.
>>
>> The following work will now be done:
>>
>> * fix column order dialog: group columns may not be reordered or
>> removed, so the column dialog must be aware of this
>
> How does one use an old Family Tree with changes to the personview
> columns? The new code seems to work on newly created trees, but shows
> the wrong columns with old data. How can I reset my edited personview
> columns? Where is that info stored? I tried ./autogen.sh and deleting
> .gramps/*.ini. Or is there something we can do to the old data before
> apply the patch that will still make those trees work with the new
> view code?
>

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Gramps-devel mailing list
Gramps-devel@...
https://lists.sourceforge.net/lists/listinfo/gramps-devel

[Nick Hall-6]

Benny Malengier wrote:

> These changes are now in trunk. This is however not the final version.
> Commit 13515 does the following:
>
> Personview is now derives of the same classes as the other views, so a
> very nice code cleanup
> Personview now uses a generic treebasemodel, that in the future will
> be reused for other views.
> Personview now allows column sorting.
>
> There are some dramatic slowdowns as to the old code, as well as speed
> ups. What is slower:
> * showing person view is two times slower
> * selecting a person is very slow on large family trees
> Dramatically faster is however insert and delete which no longer
> depend on the size of the tree, so common actions that happen a lot
> during workflow.
>
> The following work will now be done:
>
> * fix column order dialog: group columns may not be reordered or
> removed, so the column dialog must be aware of this
>  

This will be fairly easy to do but we need to think about how it will
work if a view has more than one display mode.

Does the view share the column order across all display modes or does
each display (model) have its own order?

For flat models we display all columns as we do now.

There are 2 options for tree models:

1. Display all the columns as we do for the flat model.  The view
ignores the hierarchical column and places it as the first column even
if the user has disabled it.

2. Remove the hierarchical column from the column selector.  The problem
here is that if we share the column order with the flat view where do we
put the hierarchical column back in for the flat view?

For example, the flat view makes the hierarchical column the third
column.  Then the user re-orders the columns for the hierarchical
display.  Then the user goes back to the flat view.  Does this cause a
problem?  Do we need to store a column order for each display rather
than each view.

> * increase speed, I intend to try linked lists instead of the index to
> path and path to index maps we used to have. It should be cleaner and
> a lot faster. If this works out, also the flatbasemodel will use it,
> so as to have one way of coding.
>  
Let me know if you need any help.

> * increase speed of filter/search on a build tree by keeping the full
> tree in memory.
>  
This needs to be done after we decide which approach is best.

> * make views plugins, so we can have both tree personview and flat
> personview.
I assume that you are working on this yourself.

> My idea now is to have one person selection in the
> sidebar, and then on the toolbar reserve a section to switch views of
> the same category.
>  
As you know I have started to do some work on the design of a new
sidebar in the bug tracker - 0001644: need to brainstorm alternatives to
the current sidebar/navbar

http://www.gramps-project.org/bugs/view.php?id=1644

I had another idea to put a small button bar in the "views" navigation
mode of the navigation bar.  Have a look at the "File Browser" sidebar
of gedit to see what I mean.  I think that this as closer to your
original intentions but will still be flexible for the future.

I'll post a summary of my thoughts so far in the original thread.


Regards,


Nick.

> Feel free to jump in.
>
> Benny
>
> 2009/11/6 Benny Malengier <benny.malengier@...>:
>  
>> Nick,
>>
>> I have been working on personview today,
>> http://www.gramps-project.org/bugs/view.php?id=3275
>> It looks nice, but I see a problem: on_iter_next is too slow.
>>
>> I see two possible solutions:
>>
>> 1/do like in flatbasemodel, and add an index like the handle2index
>> map. This could be added as data in self.tree, as the third entry.
>> Then on_iter_next(iter) would do: index = tree[iter][2] +1, and see if
>> that exists in children to return it.
>> Problem is the same as in flatbasemodel: on insert and delete: update
>> the affected indexes. Not too bad, it is done in memory (see how
>> adding/deleting is quick is eg eventview), but still annoying.
>>
>> 2/go for a linked list approach. The linked list would be done with a
>> class Node, as in
>> http://stackoverflow.com/questions/280243/python-linked-list
>> In this setup, self.tree[iter] = (level, Node), where Node.key = iter,
>> and Node.next links to the next Node and
>> self.children[iter] = a list of Node which also form a correct linked list.
>> In this setup on_iter_next(iter) would do: retrun self.tree[iter][1].next()
>> The advantage over 1/ is that on insert and delete, no loops to update
>> indexes must be performed, but the code for insert/delete has to be
>> done once correctly :-)
>>
>> Any preferences or other ideas?
>> To see that above is a problem in the present implementation, some timings:
>>
>> GRAMPS 3.1 on 30000+ Persons:
>> 8594: DEBUG: _PeopleModel.py: line 316: PeopleModel rebuild_data 4.24 sec
>> 8595: DEBUG: _PeopleModel.py: line 296: PeopleModel __init__ 4.24 sec
>> 8688: DEBUG: PersonView.py: line 555: PersonView build_tree 4.32 sec
>> 8688: DEBUG: PersonView.py: line 556:  setting model 0.07 sec
>> 39118: DEBUG: PersonView.py: line 773: PersonView person_removed 4.65 sec
>> 56205: DEBUG: PersonView.py: line 746: PersonView person_added 4.75 sec
>>
>> GRAMSP trunk with my patchset:
>> 8281: DEBUG: treebasemodel.py: line 285: PeopleModel rebuild_data 4.72 sec
>> 8281: DEBUG: treebasemodel.py: line 169: PeopleModel __init__ 4.72 sec
>> 11897: DEBUG: listview.py: line 256: PersonView build_tree 8.29 sec
>> 11897: DEBUG: listview.py: line 261: parts 4.72 , 0.01 , 3.56 , 0.0 , 0.0
>> 60672: DEBUG: listview.py: line 624:    PersonView row_add 0.0 sec
>> 72629: DEBUG: treebasemodel.py: line 465: PeopleModel add_row_by_handle 0.01 sec
>> 179444: DEBUG: treebasemodel.py: line 485: PeopleModel
>> delete_row_by_handle 0.11 sec
>> 179444: DEBUG: listview.py: line 655:    PersonView row_delete 0.11 sec
>>
>> So the row add and row delete are now instantanious (5 sec to 0.1sec)
>> which is what we want to achieve, building the data is a bit more
>> expensive (4.24 to 4.72) which is completely acceptable, but showing
>> the treeview itself (which happens on attach of the model) goes from
>> 0.07 sec to 3.56.
>> In other words, we need something as good as the old NodeTreeMap with
>> it's path2iter and iter2path to make the iter operations faster.
>> Sorting on the columns also suffers from this.
>>
>> Some other attention points:
>> 1/ remove_node should use del, not a temporary list. Should also be
>> recusive as deleting a leaf can mean removing all upper groups.
>> 2/ I'd like to keep in memory the entire map so as not to hit the
>> database on search and filter, just like in the flatbasemodel
>> 3/ not sure how reverse works with this rows_reordered call. This is
>> not done in flatbasemodel and seems to work fine. Here setting up the
>> sequence for call to rows_reodered takes quite some work.
>>
>> My preference goes at the moment towards committing this patch, and
>> then doing further changes. The patchset just becomes too large to
>> handle otherwise.
>>
>> Benny
>>
>>    
>
> ------------------------------------------------------------------------------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
> trial. Simplify your report design, integration and deployment - and focus on
> what you do best, core application coding. Discover what's new with
> Crystal Reports now.  http://p.sf.net/sfu/bobj-july
> _______________________________________________
> Gramps-devel mailing list
> Gramps-devel@...
> https://lists.sourceforge.net/lists/listinfo/gramps-devel
>  

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Gramps-devel mailing list
Gramps-devel@...
https://lists.sourceforge.net/lists/listinfo/gramps-devel

[Benny Malengier]

2009/11/7 Nick Hall <nick__hall@...>:

>
>
> Benny Malengier wrote:
>>
>> Nick,
>>
>> I have been working on personview today,
>> http://www.gramps-project.org/bugs/view.php?id=3275
>> It looks nice, but I see a problem: on_iter_next is too slow.
>>
>>
>
> The 2 functions called by the treeview the most are on_iter_next and
> on_get_value.  I don't  understand why on_iter_next is called so much, the
> treeview seems to call it far more times than is needed for the displayed
> part of the tree.  Having said that, there is nothing we can do about it so
> we need to speed up on_iter_next.
>
>> I see two possible solutions:
>>
>> 1/do like in flatbasemodel, and add an index like the handle2index
>> map. This could be added as data in self.tree, as the third entry.
>> Then on_iter_next(iter) would do: index = tree[iter][2] +1, and see if
>> that exists in children to return it.
>> Problem is the same as in flatbasemodel: on insert and delete: update
>> the affected indexes. Not too bad, it is done in memory (see how
>> adding/deleting is quick is eg eventview), but still annoying.
>>
>>
>
> Yes, this is like having a flatbasemodel for every parent node in the tree.
>  It should have similar performance implications to the flat model.
>
>> 2/go for a linked list approach. The linked list would be done with a
>> class Node, as in
>> http://stackoverflow.com/questions/280243/python-linked-list
>> In this setup, self.tree[iter] = (level, Node), where Node.key = iter,
>> and Node.next links to the next Node and
>> self.children[iter] = a list of Node which also form a correct linked
>> list.
>> In this setup on_iter_next(iter) would do: retrun
>> self.tree[iter][1].next()
>> The advantage over 1/ is that on insert and delete, no loops to update
>> indexes must be performed, but the code for insert/delete has to be
>> done once correctly :-)
>>
>>
>
> This would involve more work but I think it is worth a try.  The gtk
> ListStore is implemented using linked lists.  I almost started coding a
> linked list implementation but then thought it was better to wait for your
> comments first.
>
> Do you want me to code this or do you want to do it yourself?

It looks like a very fun thing to implement. I will however start with
making views plugins before tackling this, so if you have time, go
ahead and start it. We need the plugin views for some larger patches
in the tracker.

I would not deviate too much from the present setup. So what would be needed:
1/a class LLNode for linked list nodes. That would be like in
http://stackoverflow.com/questions/280243/python-linked-list which
actually uses the names LISP
(http://en.wikipedia.org/wiki/Lisp_programming_language) gives to
these. So car and cdr (http://en.wikipedia.org/wiki/CAR_and_CDR ), I
would go with that too. And add to car the things we want to store

2/instead of self.tree mapping key_level-1 to (key_level,
handle_level-1),  we would have self.tree mapping key_level-1 to
(key_level, LLNode_level-1).
And self.children would then map key_level to a list of LLNode on
level-1. The list should correspond with the list as constructucted by
transversing the linkedlist itself, but we would want a real python
list too, so as to be able to quickly map a path to the correct iter.

With this construct all iter methods should be fast.
A delete of an iter would mean obtaining parent key, and finding it in
self.children (bisect_left is again fastest), do a LLList delete.
Adding a node would mean a bisect_search to know where to insert, then
adding the Node in the linked list.

Yes, looks like fun to do. I'm sure my idea as given above still has
some holes :-)
Go ahead and try it.

<snip>

> Your timings show similar results to mine.  I am using smaller data sets
> though.
>
>> Some other attention points:
>> 1/ remove_node should use del, not a temporary list. Should also be
>> recusive as deleting a leaf can mean removing all upper groups.
>>
>
> Both adding and deleting will involve more than one node, in general.

Well, for personview it is still easy, a real test for the code will
be the tree placeview. That will allow to remove any bugs not visible
with the personview

>> 2/ I'd like to keep in memory the entire map so as not to hit the
>> database on search and filter, just like in the flatbasemodel
>>
>
> I was always intending to do this but I wanted to get the basic design right
> first before optimizing.  Your optimization for personview was necessary
> though in order for us to be able to do proper performance comparisons
> between versions.
>
>> 3/ not sure how reverse works with this rows_reordered call. This is
>> not done in flatbasemodel and seems to work fine. Here setting up the
>> sequence for call to rows_reodered takes quite some work.
>
> When the order of rows in the model changes it needs to tell the treeview so
> that it can update the display.  I think that rows_reordered is the way it
> should be done.  It doesn't seem to work without it - this is why I was
> removing and re-adding the model in one version of my old code.  I don't
> know how we get away with it in the flat model.

Once adding the model to the view is fast, we can just remove the
model, set reverse, and reattach, without the need to signal row
changed on every single row (as the entire model changed as far as the
treeview is concerned). Probably flatbasemodel has something like that
happening behind the scenes

Benny

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Gramps-devel mailing list
Gramps-devel@...
https://lists.sourceforge.net/lists/listinfo/gramps-devel

[Benny Malengier]

2009/11/7 Nick Hall <nick__hall@...>:

>
>
> Benny Malengier wrote:
>>
>> These changes are now in trunk. This is however not the final version.
>> Commit 13515 does the following:
>>
>> Personview is now derives of the same classes as the other views, so a
>> very nice code cleanup
>> Personview now uses a generic treebasemodel, that in the future will
>> be reused for other views.
>> Personview now allows column sorting.
>>
>> There are some dramatic slowdowns as to the old code, as well as speed
>> ups. What is slower:
>> * showing person view is two times slower
>> * selecting a person is very slow on large family trees
>> Dramatically faster is however insert and delete which no longer
>> depend on the size of the tree, so common actions that happen a lot
>> during workflow.
>>
>> The following work will now be done:
>>
>> * fix column order dialog: group columns may not be reordered or
>> removed, so the column dialog must be aware of this
>>
>
> This will be fairly easy to do but we need to think about how it will work
> if a view has more than one display mode.
>
> Does the view share the column order across all display modes or does each
> display (model) have its own order?
>
> For flat models we display all columns as we do now.
>
> There are 2 options for tree models:
>
> 1. Display all the columns as we do for the flat model.  The view ignores
> the hierarchical column and places it as the first column even if the user
> has disabled it.
>
> 2. Remove the hierarchical column from the column selector.  The problem
> here is that if we share the column order with the flat view where do we put
> the hierarchical column back in for the flat view?
>
> For example, the flat view makes the hierarchical column the third column.
>  Then the user re-orders the columns for the hierarchical display.  Then the
> user goes back to the flat view.  Does this cause a problem?  Do we need to
> store a column order for each display rather than each view.

I would remove it from database and have columns in the ini file
connected to a specific view.  Let's see the others agree.

>> * increase speed, I intend to try linked lists instead of the index to
>> path and path to index maps we used to have. It should be cleaner and
>> a lot faster. If this works out, also the flatbasemodel will use it,
>> so as to have one way of coding.
>>
>
> Let me know if you need any help.

yes

>> * increase speed of filter/search on a build tree by keeping the full
>> tree in memory.
>>
>
> This needs to be done after we decide which approach is best.
>
ok

>> * make views plugins, so we can have both tree personview and flat
>> personview.
>
> I assume that you are working on this yourself.
>

I will do this first

>> My idea now is to have one person selection in the
>> sidebar, and then on the toolbar reserve a section to switch views of
>> the same category.
>>
>
> As you know I have started to do some work on the design of a new sidebar in
> the bug tracker - 0001644: need to brainstorm alternatives to the current
> sidebar/navbar
>
> http://www.gramps-project.org/bugs/view.php?id=1644
>
> I had another idea to put a small button bar in the "views" navigation mode
> of the navigation bar.  Have a look at the "File Browser" sidebar of gedit
> to see what I mean.  I think that this as closer to your original intentions
> but will still be flexible for the future.
>
> I'll post a summary of my thoughts so far in the original thread.

Ok, as it is november, I think there is too little time to now change
to dramatically for 3.2, so let's push a real rewrite to 3.3
In the meantime, I would go for a minimal change:
1/ the sidebar shows predefined categories
2/different views for the same category show as buttons in the
toolbar, clicking it removes the previous view and shows the new one.

I think above is least intrusive and allows us to have all fully
tested before 3.2. Should you have really a lot of time, you can
already design more extensive changes, but I'm afraid actually doing
it would not fit in the timeframe of a 3.2 release.

Benny

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Gramps-devel mailing list
Gramps-devel@...
https://lists.sourceforge.net/lists/listinfo/gramps-devel

[Doug Blank-2]

On Sat, Nov 7, 2009 at 11:11 AM, Benny Malengier
<benny.malengier@...> wrote:

> 2009/11/7 Nick Hall <nick__hall@...>:
>>
>>
>> Benny Malengier wrote:
>>>
>>> These changes are now in trunk. This is however not the final version.
>>> Commit 13515 does the following:
>>>
>>> Personview is now derives of the same classes as the other views, so a
>>> very nice code cleanup
>>> Personview now uses a generic treebasemodel, that in the future will
>>> be reused for other views.
>>> Personview now allows column sorting.
>>>
>>> There are some dramatic slowdowns as to the old code, as well as speed
>>> ups. What is slower:
>>> * showing person view is two times slower
>>> * selecting a person is very slow on large family trees
>>> Dramatically faster is however insert and delete which no longer
>>> depend on the size of the tree, so common actions that happen a lot
>>> during workflow.
>>>
>>> The following work will now be done:
>>>
>>> * fix column order dialog: group columns may not be reordered or
>>> removed, so the column dialog must be aware of this
>>>
>>
>> This will be fairly easy to do but we need to think about how it will work
>> if a view has more than one display mode.
>>
>> Does the view share the column order across all display modes or does each
>> display (model) have its own order?
>>
>> For flat models we display all columns as we do now.
>>
>> There are 2 options for tree models:
>>
>> 1. Display all the columns as we do for the flat model.  The view ignores
>> the hierarchical column and places it as the first column even if the user
>> has disabled it.
>>
>> 2. Remove the hierarchical column from the column selector.  The problem
>> here is that if we share the column order with the flat view where do we put
>> the hierarchical column back in for the flat view?
>>
>> For example, the flat view makes the hierarchical column the third column.
>>  Then the user re-orders the columns for the hierarchical display.  Then the
>> user goes back to the flat view.  Does this cause a problem?  Do we need to
>> store a column order for each display rather than each view.
>

Go gramps team!

> I would remove it from database and have columns in the ini file
> connected to a specific view.  Let's see the others agree.

+1. We should separate display data from the genealogical data. It
should make the genealogy data more safe, and not have to be upgraded
if we separate. There is an additional question about whether it
should be stored in *a* database (separate from the core data) or just
in the ini file. The column ordering doesn't change much, and there
won't be too many, so I think the ini file is fine.

I hope that we can address the views reorg sooner than later. There
are some parts of gramplets/tools that I think can be fixed with this.
Maybe we can have a 3.3 plan in place to come out 6 months after 3.2?

-Doug

>>> * increase speed, I intend to try linked lists instead of the index to
>>> path and path to index maps we used to have. It should be cleaner and
>>> a lot faster. If this works out, also the flatbasemodel will use it,
>>> so as to have one way of coding.
>>>
>>
>> Let me know if you need any help.
>
> yes
>
>>> * increase speed of filter/search on a build tree by keeping the full
>>> tree in memory.
>>>
>>
>> This needs to be done after we decide which approach is best.
>>
> ok
>
>>> * make views plugins, so we can have both tree personview and flat
>>> personview.
>>
>> I assume that you are working on this yourself.
>>
>
> I will do this first
>
>>> My idea now is to have one person selection in the
>>> sidebar, and then on the toolbar reserve a section to switch views of
>>> the same category.
>>>
>>
>> As you know I have started to do some work on the design of a new sidebar in
>> the bug tracker - 0001644: need to brainstorm alternatives to the current
>> sidebar/navbar
>>
>> http://www.gramps-project.org/bugs/view.php?id=1644
>>
>> I had another idea to put a small button bar in the "views" navigation mode
>> of the navigation bar.  Have a look at the "File Browser" sidebar of gedit
>> to see what I mean.  I think that this as closer to your original intentions
>> but will still be flexible for the future.
>>
>> I'll post a summary of my thoughts so far in the original thread.
>
> Ok, as it is november, I think there is too little time to now change
> to dramatically for 3.2, so let's push a real rewrite to 3.3
> In the meantime, I would go for a minimal change:
> 1/ the sidebar shows predefined categories
> 2/different views for the same category show as buttons in the
> toolbar, clicking it removes the previous view and shows the new one.
>
> I think above is least intrusive and allows us to have all fully
> tested before 3.2. Should you have really a lot of time, you can
> already design more extensive changes, but I'm afraid actually doing
> it would not fit in the timeframe of a 3.2 release.

> Benny
>
> ------------------------------------------------------------------------------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
> trial. Simplify your report design, integration and deployment - and focus on
> what you do best, core application coding. Discover what's new with
> Crystal Reports now.  http://p.sf.net/sfu/bobj-july
> _______________________________________________
> Gramps-devel mailing list
> Gramps-devel@...
> https://lists.sourceforge.net/lists/listinfo/gramps-devel
>

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Gramps-devel mailing list
Gramps-devel@...
https://lists.sourceforge.net/lists/listinfo/gramps-devel

[Benny Malengier]

2009/11/7 Doug Blank <doug.blank@...>:

>>
>
> Go gramps team!
>
>> I would remove it from database and have columns in the ini file
>> connected to a specific view.  Let's see the others agree.
>
> +1. We should separate display data from the genealogical data. It
> should make the genealogy data more safe, and not have to be upgraded
> if we separate. There is an additional question about whether it
> should be stored in *a* database (separate from the core data) or just
> in the ini file. The column ordering doesn't change much, and there
> won't be too many, so I think the ini file is fine.

But we could store the width of the columns after the person changes
that. I personally find it annoying that is not stored....

B.
>
> I hope that we can address the views reorg sooner than later. There
> are some parts of gramplets/tools that I think can be fixed with this.
> Maybe we can have a 3.3 plan in place to come out 6 months after 3.2?
>
> -Doug
>

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Gramps-devel mailing list
Gramps-devel@...
https://lists.sourceforge.net/lists/listinfo/gramps-devel

[Nick Hall-6]

Benny Malengier wrote:

> 2009/11/7 Nick Hall <nick__hall@...>:
>  
>> Benny Malengier wrote:
>>    
>>> These changes are now in trunk. This is however not the final version.
>>> Commit 13515 does the following:
>>>
>>> Personview is now derives of the same classes as the other views, so a
>>> very nice code cleanup
>>> Personview now uses a generic treebasemodel, that in the future will
>>> be reused for other views.
>>> Personview now allows column sorting.
>>>
>>> There are some dramatic slowdowns as to the old code, as well as speed
>>> ups. What is slower:
>>> * showing person view is two times slower
>>> * selecting a person is very slow on large family trees
>>> Dramatically faster is however insert and delete which no longer
>>> depend on the size of the tree, so common actions that happen a lot
>>> during workflow.
>>>
>>> The following work will now be done:
>>>
>>> * fix column order dialog: group columns may not be reordered or
>>> removed, so the column dialog must be aware of this
>>>
>>>      
>> This will be fairly easy to do but we need to think about how it will work
>> if a view has more than one display mode.
>>
>> Does the view share the column order across all display modes or does each
>> display (model) have its own order?
>>
>> For flat models we display all columns as we do now.
>>
>> There are 2 options for tree models:
>>
>> 1. Display all the columns as we do for the flat model.  The view ignores
>> the hierarchical column and places it as the first column even if the user
>> has disabled it.
>>
>> 2. Remove the hierarchical column from the column selector.  The problem
>> here is that if we share the column order with the flat view where do we put
>> the hierarchical column back in for the flat view?
>>
>> For example, the flat view makes the hierarchical column the third column.
>>  Then the user re-orders the columns for the hierarchical display.  Then the
>> user goes back to the flat view.  Does this cause a problem?  Do we need to
>> store a column order for each display rather than each view.
>>    
>
> I would remove it from database and have columns in the ini file
> connected to a specific view.  Let's see the others agree.
>
>  
>>> * increase speed, I intend to try linked lists instead of the index to
>>> path and path to index maps we used to have. It should be cleaner and
>>> a lot faster. If this works out, also the flatbasemodel will use it,
>>> so as to have one way of coding.
>>>
>>>      
>> Let me know if you need any help.
>>    
>
> yes
>
>  
>>> * increase speed of filter/search on a build tree by keeping the full
>>> tree in memory.
>>>
>>>      
>> This needs to be done after we decide which approach is best.
>>
>>    
> ok
>
>  
>>> * make views plugins, so we can have both tree personview and flat
>>> personview.
>>>      
>> I assume that you are working on this yourself.
>>
>>    
>
> I will do this first
>
>  
>>> My idea now is to have one person selection in the
>>> sidebar, and then on the toolbar reserve a section to switch views of
>>> the same category.
>>>
>>>      
>> As you know I have started to do some work on the design of a new sidebar in
>> the bug tracker - 0001644: need to brainstorm alternatives to the current
>> sidebar/navbar
>>
>> http://www.gramps-project.org/bugs/view.php?id=1644
>>
>> I had another idea to put a small button bar in the "views" navigation mode
>> of the navigation bar.  Have a look at the "File Browser" sidebar of gedit
>> to see what I mean.  I think that this as closer to your original intentions
>> but will still be flexible for the future.
>>
>> I'll post a summary of my thoughts so far in the original thread.
>>    
>
>  

I have posted a summary of the ideas for the new interface in the other
thread.


> Ok, as it is november, I think there is too little time to now change
> to dramatically for 3.2, so let's push a real rewrite to 3.3
>  
I was only proposing implementing the navigation bar in this release.  
What I was suggesting though was to get some idea of a longer term
design first.  In this way way could implement a navigation bar that was
expandable and flexible for the future. (hence the pluggable
multi-function idea.)

> In the meantime, I would go for a minimal change:
> 1/ the sidebar shows predefined categories
> 2/different views for the same category show as buttons in the
> toolbar, clicking it removes the previous view and shows the new one.
>
> I think above is least intrusive and allows us to have all fully
> tested before 3.2. Should you have really a lot of time, you can
> already design more extensive changes, but I'm afraid actually doing
> it would not fit in the timeframe of a 3.2 release.
>  

I am particularly interested in getting involved in some front-end
development so I'll look forward to 3.3 to contribute some more to this.

Nick.
> Benny
>
>
>  

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Gramps-devel mailing list
Gramps-devel@...
https://lists.sourceforge.net/lists/listinfo/gramps-devel