Custom Filtering Large Dataset

[adamarmistead]

Hello,

I need to create a tree component.  I doesn't matter to me if it is an actual tree or a treetable.  What I need is to display a list of clients, their names would be the top level nodes, and the child nodes under the client would display a few details about the client.  Say SSN, birth date, a customer id number, other names for the client (aliases), etc.  I need to filter the top level nodes of the tree (client names) only, not the child nodes.  I want to show the top level node if it or any of its children match the TextMatcherEditor filter values. Otherwise I want to hide the top level node.  I don't want the child nodes to be filtered at all, so either the top level node is hidden with no access to its children, or its not and all of its child nodes are shown.

Currently I am creating the tree structure by adding a Client object to a thread safe BasicEventList then adding a bunch of FakeClient objects that extends Client but only hold a single detail value to the list to get my tree structure.  Not sure this is the way to do this, it seems inefficient as i now have like 5n objects in my list instead of just n objects.  I then create my FilterList with a ThreadedMatcherEditor, a SwingProxyList, a TreeList, an EventTableModel from which i create my treetable.  I also need to work with a client list of like 50,000+ clients with each having like 5 to 10 details to display, so I would like something in the end that is fast enough to reasonably filter up to like a million nodes fairly quickly to allow for future client growth.

So, is it possible to filter a list in the manner i have described? and does it sound like I'm doing this reasonably correctly?  Any help, tips or criticisms would be greatly appreciated.  I can post code if that would help.

[Kevin Day-7]

Some parts of this message have been removed. Learn more about Nabble's security policy.

Does each detail hold a reference to the Client (or is there an efficient way of obtaining the Client given a reference)?  If so, then you can have the list hold only details.  Then when you build your tree path, you put the client first, then the detail itself.

 

TreeList will create a row to hold the Client object

 

With this approach in mind, the filter gets applied to the list of details.  Then the filtered list gets passed into TreeList (or the event proxy list if you want to move to the dispatch thread at that stage).

 

 

If the items don't contain a reference to their Client, you can use a CompositeList and a FunctionList to create an intermediate object that does hold the reference back to Client.

 

 

When I first started using TreeList, I had a hard time inverting my thinking.  I was so used to having top down data structures that GL's bottom-up approach took a bit of getting used to...

 

- K

 

----------------------- Original Message -----------------------

  

From: adamarmistead adamarmistead@...

To: users@...

Cc: 

Date: Wed, 12 Aug 2009 08:46:58 -0700 (PDT)

Subject: Custom Filtering Large Dataset

  

Hello,

I need to create a tree component.  I doesn't matter to me if it is an
actual tree or a treetable.  What I need is to display a list of clients,
their names would be the top level nodes, and the child nodes under the
client would display a few details about the client.  Say SSN, birth date, a
customer id number, other names for the client (aliases), etc.  I need to
filter the top level nodes of the tree (client names) only, not the child
nodes.  I want to show the top level node if it or any of its children match
the TextMatcherEditor filter values. Otherwise I want to hide the top level
node.  I don't want the child nodes to be filtered at all, so either the top
level node is hidden with no access to its children, or its not and all of
its child nodes are shown.

Currently I am creating the tree structure by adding a Client object to a
thread safe BasicEventList then adding a bunch of FakeClient objects that
extends Client but only hold a single detail value to the list to get my
tree structure.  Not sure this is the way to do this, it seems inefficient
as i now have like 5n objects in my list instead of just n objects.  I then
create my FilterList with a ThreadedMatcherEditor, a SwingProxyList, a
TreeList, an EventTableModel from which i create my treetable.  I also need
to work with a client list of like 50,000+ clients with each having like 5
to 10 details to display, so I would like something in the end that is fast
enough to reasonably filter up to like a million nodes fairly quickly to
allow for future client growth.

So, is it possible to filter a list in the manner i have described? and does
it sound like I'm doing this reasonably correctly?  Any help, tips or
criticisms would be greatly appreciated.  I can post code if that would
help.
--
View this message in context: http://www.nabble.com/Custom-Filtering-Large-Dataset-tp24939403p24939403.html
Sent from the GlazedLists - User mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

--------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@... For additional commands, e-mail: users-help@...

[adamarmistead]

I see what you are saying, i think.  It sounds backwards from what I need to do though I think.  I need to filter Clients only, not the details but i need to filter the clients based on the details.

If I have:

-Smith, Bob D.
--Smith, Robert
--Alias 2
--Alias 3
--123456
--123-45-6789
--08/14/1957
-Doe, Jane J.
--Hubbard, Old Mother
--Alias 2
--987654
--08/12/1957

and the user types '9' in the filter text field.  I want to hide Bob Smith and all his child nodes but still display all Jane Doe and ALL her child nodes.  So I only want to hide first level nodes, not their children.
I would like the clients sorted by names; last, first, middle initial.  Don't think it matters if child nodes are sorted or not.  So far this is still stumping me.

Kevin Day wrote:

Does each detail hold a reference to the Client (or is there an efficient way of obtaining the Client given a reference)?  If so, then you can have the list hold only details.  Then when you build your tree path, you put the client first, then the detail itself.
 
TreeList will create a row to hold the Client object
 
With this approach in mind, the filter gets applied to the list of details.  Then the filtered list gets passed into TreeList (or the event proxy list if you want to move to the dispatch thread at that stage).
 
 
If the items don't contain a reference to their Client, you can use a CompositeList and a FunctionList to create an intermediate object that does hold the reference back to Client.
 
 
When I first started using TreeList, I had a hard time inverting my thinking.  I was so used to having top down data structures that GL's bottom-up approach took a bit of getting used to...
 
- K
 

[adamarmistead ()]

Ok, I have managed to get the filtering working the way I wanted by ditching generics half way through my tree creation procedure. I created an EventList<Client> to hold my client list, then a FilteredList<Client> to filter clients, then a CollectionList<Client, ClientDetail> to generate the subnodes for my clients (ClientDetail no longer subclasses Client).  Then I dropped generics and created a SwingThreadProxyList to pass to my TreeList to create my EventTableModel from.  Now I can show client details as child nodes and filter just the parent nodes by telling my TextFilterator to consider client names and all the properties of the Client that are displayed as child nodes.

Here is my current stumbling block;  GlazedLists are really cool, they allow me to do a lot of really powerful things very easily, and they seem to be very efficient...until I touch a TreeList.  TreeList seems to work, but it is wicked slow.

Anyone have any ideas on how to display things in a tree format that is not SUPER slow to filter?  The only thing I have found that is even reasonably fast is in a post from JavaRanch here where Ed Kawas posted a link to his filter tree implementation which can be found here.

hopefully this helps someone.  If anyone knows of another way to quickly filter trees with large datasets I would be glad to hear it.