Using Lucene with large sets of documents in a collection

> My team is fairly new to eXist, and we’re in the process of porting an
> application from MarkLogic Server to eXist.   We found some
> interesting experiencing that might be useful to others trying to make
> fragmenting decisions.   I’d also like to know if this is typical or
> expected behavior.
>
>  
>
> NOTE: We are running eXist in tomcat on a Windows Server with the
> db-connection cacheSize set to 1536Mb and the collection cache set to
> 48Mb.
>
>  
>
> The app takes many types of documents, but the largest are very large
> (up to 125Mb) .  For various reasons our app chunks the documents into
> about 10,000 “flattened” document  chunks using a Java app server
> before being stored in the database in a single collection.  In ML
> server this process took about 10 minutes to chunk, store, and
> index.   The following is our account of doing the same thing in
> eXist.
>
>  
>
> First cut in eXist….
>
> We were loading and indexing the “chunks” using lucene into a single
> collection one at a time.  It took over 5 hours before the memory was
> consumed and the database crashed after saving around 2000 fragments.
> Interesting note is that the memory consumption got higher and higher
> as the process continued. We were very discouraged and thought that
> perhaps eXist wasn’t the right choice for our app.  
>
>  
>
> Second cut…  
>
> After reading some posts from people doing similar things, we decided
> to first turn off any full-text indexing for the collection, then load
> the 10,000 fragments into about 30 sub-collections of 300-400 docs
> each, then add the lucene definitions and do the full-text indexing on
> the whole collection.  This took <10 minutes to load the data into
> eXist, then it took about 30 minutes to do the indexing.  Although
> this was much better, the database still consumed dangerously high
> amounts of memory, and any concurrent use could take it down.  Memory
> stayed high after the process completed, and sometimes subsequent
> queries could take down the db.  This was not acceptable to even
> consider for a production app.
>
>  
>
> Third cut…
>
> We decided to switch from chunking to flattening (to remove hierarchy
> issues).  Instead of chunking into separate docs, we now recombine the
> flattened chunks into a single XML document that is about 140Mb in
> size.  This was the magic bullet for eXist/Lucene.  The big flattened
> doc loads the XML into eXist and builds the lucene indexes in 3-4
> minutes total.  Queries are very fast, and the memory never goes above
> about 1/3 of availability during the load process.  We are once again
> encouraged.
>
>  
>
> I want to re-iterate that this only became an issue when we added the
> Lucene indexing on large numbers of documents.  This approach for us
> requires a fair amount of re-architecting of our application to change
> what we had looking for full documents to instead look for nodes in
> the flattened XML document.   However, we’re very optimistic again
> that this will provide all the full-text and XML aware searching that
> we were using in ML Server.  And if we ever need to also support ML
> this approach should work with that platform as well.
>
>  
>
>
> ------------------------------------------------------------------------------
> Come build with us! The BlackBerry(R) Developer Conference in SF, CA
> is the only developer event you need to attend this year. Jumpstart your
> developing skills, take BlackBerry mobile applications to market and stay
> ahead of the curve. Join us from November 9 - 12, 2009. Register now!
> http://p.sf.net/sfu/devconference
> _______________________________________________ Exist-open mailing list Exist-open@... https://lists.sourceforge.net/lists/listinfo/exist-open

> My team is fairly new to eXist, and we’re in the process of porting an
> application from MarkLogic Server to eXist.   We found some interesting
> experiencing that might be useful to others trying to make fragmenting
> decisions.   I’d also like to know if this is typical or expected behavior.
>
>
>
> NOTE: We are running eXist in tomcat on a Windows Server with the
> db-connection cacheSize set to 1536Mb and the collection cache set to 48Mb.
>
>
>
> The app takes many types of documents, but the largest are very large (up to
> 125Mb) .  For various reasons our app chunks the documents into about 10,000
> “flattened” document  chunks using a Java app server before being stored in
> the database in a single collection.  In ML server this process took about
> 10 minutes to chunk, store, and index.   The following is our account of
> doing the same thing in eXist.
>
>
>
> First cut in eXist….
>
> We were loading and indexing the “chunks” using lucene into a single
> collection one at a time.  It took over 5 hours before the memory was
> consumed and the database crashed after saving around 2000 fragments.
> Interesting note is that the memory consumption got higher and higher as the
> process continued. We were very discouraged and thought that perhaps eXist
> wasn’t the right choice for our app.
>
>
>
> Second cut…
>
> After reading some posts from people doing similar things, we decided to
> first turn off any full-text indexing for the collection, then load the
> 10,000 fragments into about 30 sub-collections of 300-400 docs each, then
> add the lucene definitions and do the full-text indexing on the whole
> collection.  This took <10 minutes to load the data into eXist, then it took
> about 30 minutes to do the indexing.  Although this was much better, the
> database still consumed dangerously high amounts of memory, and any
> concurrent use could take it down.  Memory stayed high after the process
> completed, and sometimes subsequent queries could take down the db.  This
> was not acceptable to even consider for a production app.
>
>
>
> Third cut…
>
> We decided to switch from chunking to flattening (to remove hierarchy
> issues).  Instead of chunking into separate docs, we now recombine the
> flattened chunks into a single XML document that is about 140Mb in size.
> This was the magic bullet for eXist/Lucene.  The big flattened doc loads the
> XML into eXist and builds the lucene indexes in 3-4 minutes total.  Queries
> are very fast, and the memory never goes above about 1/3 of availability
> during the load process.  We are once again encouraged.
>
>
>
> I want to re-iterate that this only became an issue when we added the Lucene
> indexing on large numbers of documents.  This approach for us requires a
> fair amount of re-architecting of our application to change what we had
> looking for full documents to instead look for nodes in the flattened XML
> document.   However, we’re very optimistic again that this will provide all
> the full-text and XML aware searching that we were using in ML Server.  And
> if we ever need to also support ML this approach should work with that
> platform as well.

>>
>>
>> Third cut…
>>
>> We decided to switch from chunking to flattening (to remove hierarchy
>> issues).  Instead of chunking into separate docs, we now recombine  
>> the
>> flattened chunks into a single XML document that is about 140Mb in  
>> size.
>> This was the magic bullet for eXist/Lucene.  The big flattened doc  
>> loads the
>> XML into eXist and builds the lucene indexes in 3-4 minutes total.  
>> Queries
>> are very fast, and the memory never goes above about 1/3 of  
>> availability
>> during the load process.  We are once again encouraged.
>>
>>
>>
>> I want to re-iterate that this only became an issue when we added  
>> the Lucene
>> indexing on large numbers of documents.  This approach for us  
>> requires a
>> fair amount of re-architecting of our application to change what we  
>> had
>> looking for full documents to instead look for nodes in the  
>> flattened XML
>> document.   However, we’re very optimistic again that this will  
>> provide all
>> the full-text and XML aware searching that we were using in ML  
>> Server.  And
>> if we ever need to also support ML this approach should work with  
>> that
>> platform as well.

>>
>>
>> Third cut…
>>
>> We decided to switch from chunking to flattening (to remove hierarchy
>> issues).  Instead of chunking into separate docs, we now recombine  
>> the
>> flattened chunks into a single XML document that is about 140Mb in  
>> size.
>> This was the magic bullet for eXist/Lucene.  The big flattened doc  
>> loads the
>> XML into eXist and builds the lucene indexes in 3-4 minutes total.  
>> Queries
>> are very fast, and the memory never goes above about 1/3 of  
>> availability
>> during the load process.  We are once again encouraged.
>>
>>
>>
>> I want to re-iterate that this only became an issue when we added  
>> the Lucene
>> indexing on large numbers of documents.  This approach for us  
>> requires a
>> fair amount of re-architecting of our application to change what we  
>> had
>> looking for full documents to instead look for nodes in the  
>> flattened XML
>> document.   However, we’re very optimistic again that this will  
>> provide all
>> the full-text and XML aware searching that we were using in ML  
>> Server.  And
>> if we ever need to also support ML this approach should work with  
>> that
>> platform as well.

>>
>>
>> Third cut.
>>
>> We decided to switch from chunking to flattening (to remove hierarchy
>> issues).  Instead of chunking into separate docs, we now recombine  
>> the
>> flattened chunks into a single XML document that is about 140Mb in  
>> size.
>> This was the magic bullet for eXist/Lucene.  The big flattened doc  
>> loads the
>> XML into eXist and builds the lucene indexes in 3-4 minutes total.  
>> Queries
>> are very fast, and the memory never goes above about 1/3 of  
>> availability
>> during the load process.  We are once again encouraged.
>>
>>
>>
>> I want to re-iterate that this only became an issue when we added  
>> the Lucene
>> indexing on large numbers of documents.  This approach for us  
>> requires a
>> fair amount of re-architecting of our application to change what we  
>> had
>> looking for full documents to instead look for nodes in the  
>> flattened XML
>> document.   However, we're very optimistic again that this will  
>> provide all
>> the full-text and XML aware searching that we were using in ML  
>> Server.  And
>> if we ever need to also support ML this approach should work with  
>> that
>> platform as well.