|
»
»
« Return to Thread: Slow response
Re: Slow response
by Mike Klaas
::
Rate this Message:
On 6-Sep-07, at 3:16 PM, Aaron Hammond wrote:
> Thank-you for your response, this does shed some light on the subject.
> Our basic question was why were we seeing slower responses the smaller
> our result set got.
>
> Currently we are searching about 1.2 million documents with the source
> document about 2KB, but we do duplicate some of the data. I bumped
> up my
> filterCache to 5 million and the 2nd search I did for an non-indexed
> term came back in 2.1 seconds so that is much improved. I am a little
> concerned about having this value so high but this is our problem
> and we
> will play with it.
>
> I do have a few follow-up questions. First, in regards to the
> filterCache once a single search has been done and facets
> requested, as
> long as new facets aren't requested and the size is large enough then
> the filters will remain in the cache, correct?
>
> Also, you mention that faceting is more a "function of the number
> of the
> number of terms in the field". The 2 fields causing our problems are
> Authors and Subjects. If we divided up the data that made these facets
> into more specific fields (Primary author, secondary author, etc.)
> would
> this perform better? So the number of facet fields would increase but
> the unique terms for a given facet should be less.
There are essentially two facet computation strategies:
1. cached bitsets: a bitset for each term is generated and
intersected with the query restul bitset. This is more general and
performs well up to a few thousand terms.
2. field enumeration: cache the field contents, and generate counts
using this data. Relatively independent of #unique terms, but
requires at most a single facet value per field per document.
So, if you factor author into Primary author/Secondary author, where
each is guaranteed to only have one value per doc, this could greatly
accelerate your faceting. There are probably fewer unique subjects,
so strategy 1 is likely fine.
To use strategy 2, just make sure that multivalued="false" is set for
those fields in schema.xml
-Mike
> Thank-you for your response, this does shed some light on the subject.
> Our basic question was why were we seeing slower responses the smaller
> our result set got.
>
> Currently we are searching about 1.2 million documents with the source
> document about 2KB, but we do duplicate some of the data. I bumped
> up my
> filterCache to 5 million and the 2nd search I did for an non-indexed
> term came back in 2.1 seconds so that is much improved. I am a little
> concerned about having this value so high but this is our problem
> and we
> will play with it.
>
> I do have a few follow-up questions. First, in regards to the
> filterCache once a single search has been done and facets
> requested, as
> long as new facets aren't requested and the size is large enough then
> the filters will remain in the cache, correct?
>
> Also, you mention that faceting is more a "function of the number
> of the
> number of terms in the field". The 2 fields causing our problems are
> Authors and Subjects. If we divided up the data that made these facets
> into more specific fields (Primary author, secondary author, etc.)
> would
> this perform better? So the number of facet fields would increase but
> the unique terms for a given facet should be less.
There are essentially two facet computation strategies:
1. cached bitsets: a bitset for each term is generated and
intersected with the query restul bitset. This is more general and
performs well up to a few thousand terms.
2. field enumeration: cache the field contents, and generate counts
using this data. Relatively independent of #unique terms, but
requires at most a single facet value per field per document.
So, if you factor author into Primary author/Secondary author, where
each is guaranteed to only have one value per doc, this could greatly
accelerate your faceting. There are probably fewer unique subjects,
so strategy 1 is likely fine.
To use strategy 2, just make sure that multivalued="false" is set for
those fields in schema.xml
-Mike
« Return to Thread: Slow response
| Free embeddable forum powered by Nabble | Forum Help |
