Optimizing unordered queries

[Nigel-13]

I recently posted some questions about performance problems with large
indexes.  One key thing about our situation is that we don't need sorted
results (either by relevance or any other key).  I've been looking into our
memory usage and tracing through some code, which in combination with the
recent posts about setTermInfosIndexDivisor got me thinking about the best
way to do a query where the order of results doesn't matter.

Currently we're (perhaps naively) doing the equivalent of
query.weight(searcher).scorer(reader).score(collector).  Obviously there's a
certain amount of unnecessary calculation that results from this if you
don't care about sorting.  Are there any general recommendations for
unordered searching?  (We already omit norms.)

(More details: Of particular interest are things that access the TermInfos,
since that's the major source of RAM usage: if a smaller number of TermInfos
were needed then we could perhaps use an aggressive index divisor setting to
save RAM without a performance penalty.  For example, I was thinking about a
custom Similarity implementation that skipped the idf calculations, since
those have to hit the TermInfos.)

Thanks,
Chris

[eks dev]

You omitNorms(), did you also omitTf()?

when something like https://issues.apache.org/jira/browse/LUCENE-1345 gets commited, you will have a posibility to see some benefits (e.g. by packing single postings lists as Filters). The code there optimises exactly that case as filters contain no Scoring information ...


try:




----- Original Message ----

> From: Nigel <nigelspleen@...>
> To: java-user@...
> Sent: Friday, 26 June, 2009 4:11:53
> Subject: Optimizing unordered queries
>
> I recently posted some questions about performance problems with large
> indexes.  One key thing about our situation is that we don't need sorted
> results (either by relevance or any other key).  I've been looking into our
> memory usage and tracing through some code, which in combination with the
> recent posts about setTermInfosIndexDivisor got me thinking about the best
> way to do a query where the order of results doesn't matter.
>
> Currently we're (perhaps naively) doing the equivalent of
> query.weight(searcher).scorer(reader).score(collector).  Obviously there's a
> certain amount of unnecessary calculation that results from this if you
> don't care about sorting.  Are there any general recommendations for
> unordered searching?  (We already omit norms.)
>
> (More details: Of particular interest are things that access the TermInfos,
> since that's the major source of RAM usage: if a smaller number of TermInfos
> were needed then we could perhaps use an aggressive index divisor setting to
> save RAM without a performance penalty.  For example, I was thinking about a
> custom Similarity implementation that skipped the idf calculations, since
> those have to hit the TermInfos.)
>
> Thanks,
> Chris

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@...
For additional commands, e-mail: java-user-help@...

[eks dev]

also see, http://lucene.apache.org/java/2_2_0/api/org/apache/lucene/search/BooleanQuery.html#getAllowDocsOutOfOrder()





----- Original Message ----

> From: Nigel <nigelspleen@...>
> To: java-user@...
> Sent: Friday, 26 June, 2009 4:11:53
> Subject: Optimizing unordered queries
>
> I recently posted some questions about performance problems with large
> indexes.  One key thing about our situation is that we don't need sorted
> results (either by relevance or any other key).  I've been looking into our
> memory usage and tracing through some code, which in combination with the
> recent posts about setTermInfosIndexDivisor got me thinking about the best
> way to do a query where the order of results doesn't matter.
>
> Currently we're (perhaps naively) doing the equivalent of
> query.weight(searcher).scorer(reader).score(collector).  Obviously there's a
> certain amount of unnecessary calculation that results from this if you
> don't care about sorting.  Are there any general recommendations for
> unordered searching?  (We already omit norms.)
>
> (More details: Of particular interest are things that access the TermInfos,
> since that's the major source of RAM usage: if a smaller number of TermInfos
> were needed then we could perhaps use an aggressive index divisor setting to
> save RAM without a performance penalty.  For example, I was thinking about a
> custom Similarity implementation that skipped the idf calculations, since
> those have to hit the TermInfos.)
>
> Thanks,
> Chris

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@...
For additional commands, e-mail: java-user-help@...

[Michael McCandless-2]

On Thu, Jun 25, 2009 at 10:11 PM, Nigel<nigelspleen@...> wrote:

> Currently we're (perhaps naively) doing the equivalent of
> query.weight(searcher).scorer(reader).score(collector).  Obviously there's a
> certain amount of unnecessary calculation that results from this if you
> don't care about sorting.  Are there any general recommendations for
> unordered searching?  (We already omit norms.)

As of 2.9, scoring is optional with the new Collector.  Ie, if your
Collector doesn't call scorer.score() then the score won't be
computed.

Also, 2.9 will enable out-of-docID-order scoring if your
Collector.acceptsDocsOutOfOrder returns true, but currently it's
disjunction only (plus up to 32 prohibited terms) that will see a
speedup from this.

What kind of queries are you running?

I assume your Collector simply aggregates the list of docIDs hit?  Ie
no sorting, priority queue, etc., in it.

> (More details: Of particular interest are things that access the TermInfos,
> since that's the major source of RAM usage: if a smaller number of TermInfos
> were needed then we could perhaps use an aggressive index divisor setting to
> save RAM without a performance penalty.  For example, I was thinking about a
> custom Similarity implementation that skipped the idf calculations, since
> those have to hit the TermInfos.)

Unfortunately the TermInfos must still be hit to look up the
freq/proxOffset in the postings files.

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@...
For additional commands, e-mail: java-user-help@...

[Nigel-13]

On Fri, Jun 26, 2009 at 10:51 AM, eks dev <eksdev@...> wrote:

>
> You omitNorms(), did you also omitTf()?


We did, but had to include TF after all since omitting it also dropped
position information, which we needed for phrase queries.  I didn't think it
was possible to remove just the TFs without the positions, but please let me
know if that's incorrect.

Thanks,
Chris

[Nigel-13]

On Fri, Jun 26, 2009 at 10:52 AM, eks dev <eksdev@...> wrote:

>
> also see,
> http://lucene.apache.org/java/2_2_0/api/org/apache/lucene/search/BooleanQuery.html#getAllowDocsOutOfOrder()<http://lucene.apache.org/java/2_2_0/api/org/apache/lucene/search/BooleanQuery.html#getAllowDocsOutOfOrder%28%29>


Interesting, thank you.  I'm not sure how many of our queries will fall into
the category of things that will be improved by this setting, but every
little bit helps.

Chris

[Nigel-13]

On Fri, Jun 26, 2009 at 11:06 AM, Michael McCandless <
lucene@...> wrote:

> On Thu, Jun 25, 2009 at 10:11 PM, Nigel<nigelspleen@...> wrote:
>
> > Currently we're (perhaps naively) doing the equivalent of
> > query.weight(searcher).scorer(reader).score(collector).  Obviously
> there's a
> > certain amount of unnecessary calculation that results from this if you
> > don't care about sorting.  Are there any general recommendations for
> > unordered searching?  (We already omit norms.)
>
> As of 2.9, scoring is optional with the new Collector.  Ie, if your
> Collector doesn't call scorer.score() then the score won't be
> computed.
>
> Also, 2.9 will enable out-of-docID-order scoring if your
> Collector.acceptsDocsOutOfOrder returns true, but currently it's
> disjunction only (plus up to 32 prohibited terms) that will see a
> speedup from this.
>
> What kind of queries are you running?

There's quite a mix.  At the simplest they're a conjunction of 3-5 terms.
At the most complex there are sometimes a hundred or more terms in various
nested boolean combinations.  So it looks like 2.9 is where we'll see some
benefits.

I assume your Collector simply aggregates the list of docIDs hit?  Ie
> no sorting, priority queue, etc., in it.
>

Exactly: it just collects all the ids and later we either load all hits or a
random subsample.

> > (More details: Of particular interest are things that access the
> TermInfos,
> > since that's the major source of RAM usage: if a smaller number of
> TermInfos
> > were needed then we could perhaps use an aggressive index divisor setting
> to
> > save RAM without a performance penalty.  For example, I was thinking
> about a
> > custom Similarity implementation that skipped the idf calculations, since
> > those have to hit the TermInfos.)
>
> Unfortunately the TermInfos must still be hit to look up the
> freq/proxOffset in the postings files.

But for that data you only have to hit the TermInfos for the terms you're
searching, correct?  So, assuming that there are vastly more terms in the
index than you ever actually use in a query, we could rely on the LRU cache
to keep the queried TermInfos around, rather than loading all of them
up-front.  This was a hypothesis based on some tracing through the code but
not a lot of knowledge of Lucene internals, so please steer me back to
reality if necessary.  (-:

Thanks,
Chris

[Michael McCandless-2]

On Sun, Jun 28, 2009 at 9:08 PM, Nigel<nigelspleen@...> wrote:

>> Unfortunately the TermInfos must still be hit to look up the
>> freq/proxOffset in the postings files.
>
> But for that data you only have to hit the TermInfos for the terms you're
> searching, correct?  So, assuming that there are vastly more terms in the
> index than you ever actually use in a query, we could rely on the LRU cache
> to keep the queried TermInfos around, rather than loading all of them
> up-front.  This was a hypothesis based on some tracing through the code but
> not a lot of knowledge of Lucene internals, so please steer me back to
> reality if necessary.  (-:

Right, it's only the terms in your query that need to be looked up.

There's already an LRU cache for the Term -> TermInfos lookup, in
TermInfosReader (hard wired to size 1024).  It was created as a "short
term" cache so that queries that look up the same term twice (eg once
for idf and once to get freq/prox postings position), would result in
only one real lookup.  You could try privately experimenting w/ that
value to see if you can get a reasonable it rate?

Lucene only loads the "indexed" terms (= every 128th) into RAM, by default.

Mike

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@...
For additional commands, e-mail: java-user-help@...

[Nigel-13]

On Mon, Jun 29, 2009 at 6:28 AM, Michael McCandless <
lucene@...> wrote:

> On Sun, Jun 28, 2009 at 9:08 PM, Nigel<nigelspleen@...> wrote:
> >> Unfortunately the TermInfos must still be hit to look up the
> >> freq/proxOffset in the postings files.
> >
> > But for that data you only have to hit the TermInfos for the terms you're
> > searching, correct?  So, assuming that there are vastly more terms in the
> > index than you ever actually use in a query, we could rely on the LRU
> cache
> > to keep the queried TermInfos around, rather than loading all of them
> > up-front.  This was a hypothesis based on some tracing through the code
> but
> > not a lot of knowledge of Lucene internals, so please steer me back to
> > reality if necessary.  (-:
>
> Right, it's only the terms in your query that need to be looked up.
>
> There's already an LRU cache for the Term -> TermInfos lookup, in
> TermInfosReader (hard wired to size 1024).  It was created as a "short
> term" cache so that queries that look up the same term twice (eg once
> for idf and once to get freq/prox postings position), would result in
> only one real lookup.  You could try privately experimenting w/ that
> value to see if you can get a reasonable it rate?

Exactly, the TermInfosReader cache is what I was thinking of.  Thanks for
the confirmation; I'll give that a try.

Lucene only loads the "indexed" terms (= every 128th) into RAM, by default.


Ah, I was confused by the index divisor being 1 by default: I thought it
meant that all terms were being loaded.  I see now in SegmentTermEnum that
the every-128th behavior is implemented at a lower level.

But I'm even more confused about why we have so many terms in memory.  A
heap dump shows over 270 million TermInfos, so if that's only 128th of the
total then we REALLY have a lot of terms.  (-:  We do have a lot of docs
(about 250 million), and we do have a couple unique per-document values, but
even so I can't see how we could get to 270 million x 128 terms.  (The heap
dump numbers are stable across the index close-and-reopen cycle, so I don't
think we're leaking.)

Thanks,
Chris

[Michael McCandless-2]

On Mon, Jun 29, 2009 at 9:33 AM, Nigel<nigelspleen@...> wrote:

> Ah, I was confused by the index divisor being 1 by default: I thought it
> meant that all terms were being loaded.  I see now in SegmentTermEnum that
> the every-128th behavior is implemented at a lower level.
>
> But I'm even more confused about why we have so many terms in memory.  A
> heap dump shows over 270 million TermInfos, so if that's only 128th of the
> total then we REALLY have a lot of terms.  (-:  We do have a lot of docs
> (about 250 million), and we do have a couple unique per-document values, but
> even so I can't see how we could get to 270 million x 128 terms.  (The heap
> dump numbers are stable across the index close-and-reopen cycle, so I don't
> think we're leaking.)

You could use CheckIndex to see how many terms are in your index.

If you do the heap dump after opening a fresh reader and not running
any searches yet, you see 270 million TermInfos?

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@...
For additional commands, e-mail: java-user-help@...

[Nigel-13]

On Mon, Jul 6, 2009 at 12:37 PM, Michael McCandless <
lucene@...> wrote:

> On Mon, Jun 29, 2009 at 9:33 AM, Nigel<nigelspleen@...> wrote:
>
> > Ah, I was confused by the index divisor being 1 by default: I thought it
> > meant that all terms were being loaded.  I see now in SegmentTermEnum
> that
> > the every-128th behavior is implemented at a lower level.
> >
> > But I'm even more confused about why we have so many terms in memory.  A
> > heap dump shows over 270 million TermInfos, so if that's only 128th of
> the
> > total then we REALLY have a lot of terms.  (-:  We do have a lot of docs
> > (about 250 million), and we do have a couple unique per-document values,
> but
> > even so I can't see how we could get to 270 million x 128 terms.  (The
> heap
> > dump numbers are stable across the index close-and-reopen cycle, so I
> don't
> > think we're leaking.)
>
> You could use CheckIndex to see how many terms are in your index.
>
> If you do the heap dump after opening a fresh reader and not running
> any searches yet, you see 270 million TermInfos?

Thanks, Mike.  I'm just coming back to this after taking some time to
educate myself better on Lucene internals, mostly by reading and tracing
through code.

I think now that the 270 million TermInfo number must have been user error
on my part, as I can't reproduce those values.  What I do see is about 8
million loaded TermInfos.  That matches what I expect by examining indexes
with CheckIndex: there are about 250 million terms per index, and we have 4
indexes loaded, so 1 billion terms / 128 = 8 million cached.  So, that's
still a big number (about 2gb including the associated Strings and arrays),
but at least it makes sense now.

My next thought, which I'll try as soon as I can set up some reproducible
benchmarks, is using a larger index divisor, perhaps combined with a larger
LRU TermInfo cache.  But this seems like such an easy win that I wonder why
it isn't mentioned more often (at least, I haven't seen much discussion of
it in the java-user archives).  For example, if I simply increase the index
divisor from 1 to 4, I can cut my Lucene usage from 2gb to 500mb (meaning
less GC and more OS cache).  That means much more seeking to find non-cached
terms, but increasing the LRU cache to 100,000 (for example) would allow all
(I think) of our searched terms to be cached, at a fraction of the RAM cost
of the 8 million terms cached now.  (The first-time use of any term would of
course be slower, but most search terms are used repeatedly, and it seems
like a small price to pay for such a RAM win.)  Anyway, I'm curious if there
are any obvious flaws in this plan.

Thanks,
Chris

[Michael McCandless-2]

OK good to hear you have a sane number of TermInfos now...

I think many apps don't have nearly as many unique terms as you do;
your approach (increase index divisor & LRU cache) sounds reasonable.
It'll make warming more important.  Please report back how it goes!

Lucene is unfortunately rather wasteful in how it loads the terms
index in RAM; there is a good improvement I've been wanting to
implement but haven't gotten to yet... the details are described here:

  http://mail-archives.apache.org/mod_mbox/lucene-java-user/200906.mbox/%3C85d3c3b60906101313t77d8b16atc4a2644ecd158e9@...%3E

If anyone has the "itch" this'd make a nice self-contained project and
solid improvement to Lucene...

Mike

On Mon, Jul 6, 2009 at 10:31 PM, Nigel<nigelspleen@...> wrote:

> On Mon, Jul 6, 2009 at 12:37 PM, Michael McCandless <
> lucene@...> wrote:
>
>> On Mon, Jun 29, 2009 at 9:33 AM, Nigel<nigelspleen@...> wrote:
>>
>> > Ah, I was confused by the index divisor being 1 by default: I thought it
>> > meant that all terms were being loaded.  I see now in SegmentTermEnum
>> that
>> > the every-128th behavior is implemented at a lower level.
>> >
>> > But I'm even more confused about why we have so many terms in memory.  A
>> > heap dump shows over 270 million TermInfos, so if that's only 128th of
>> the
>> > total then we REALLY have a lot of terms.  (-:  We do have a lot of docs
>> > (about 250 million), and we do have a couple unique per-document values,
>> but
>> > even so I can't see how we could get to 270 million x 128 terms.  (The
>> heap
>> > dump numbers are stable across the index close-and-reopen cycle, so I
>> don't
>> > think we're leaking.)
>>
>> You could use CheckIndex to see how many terms are in your index.
>>
>> If you do the heap dump after opening a fresh reader and not running
>> any searches yet, you see 270 million TermInfos?
>
>
> Thanks, Mike.  I'm just coming back to this after taking some time to
> educate myself better on Lucene internals, mostly by reading and tracing
> through code.
>
> I think now that the 270 million TermInfo number must have been user error
> on my part, as I can't reproduce those values.  What I do see is about 8
> million loaded TermInfos.  That matches what I expect by examining indexes
> with CheckIndex: there are about 250 million terms per index, and we have 4
> indexes loaded, so 1 billion terms / 128 = 8 million cached.  So, that's
> still a big number (about 2gb including the associated Strings and arrays),
> but at least it makes sense now.
>
> My next thought, which I'll try as soon as I can set up some reproducible
> benchmarks, is using a larger index divisor, perhaps combined with a larger
> LRU TermInfo cache.  But this seems like such an easy win that I wonder why
> it isn't mentioned more often (at least, I haven't seen much discussion of
> it in the java-user archives).  For example, if I simply increase the index
> divisor from 1 to 4, I can cut my Lucene usage from 2gb to 500mb (meaning
> less GC and more OS cache).  That means much more seeking to find non-cached
> terms, but increasing the LRU cache to 100,000 (for example) would allow all
> (I think) of our searched terms to be cached, at a fraction of the RAM cost
> of the 8 million terms cached now.  (The first-time use of any term would of
> course be slower, but most search terms are used repeatedly, and it seems
> like a small price to pay for such a RAM win.)  Anyway, I'm curious if there
> are any obvious flaws in this plan.
>
> Thanks,
> Chris
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@...
For additional commands, e-mail: java-user-help@...

[Jason Rutherglen-2]

Ah ok, I was thinking we'd wait for the new flex indexing patch.
I had started working along these lines before and will take it
on as a project (which is I believe reducing the memory
consumption of the term dictionary).

I plan to segue it into the tag index at some point.

On Tue, Jul 7, 2009 at 2:43 AM, Michael McCandless <
lucene@...> wrote:

> OK good to hear you have a sane number of TermInfos now...
>
> I think many apps don't have nearly as many unique terms as you do;
> your approach (increase index divisor & LRU cache) sounds reasonable.
> It'll make warming more important.  Please report back how it goes!
>
> Lucene is unfortunately rather wasteful in how it loads the terms
> index in RAM; there is a good improvement I've been wanting to
> implement but haven't gotten to yet... the details are described here:
>
>
> http://mail-archives.apache.org/mod_mbox/lucene-java-user/200906.mbox/%3C85d3c3b60906101313t77d8b16atc4a2644ecd158e9@...%3E
>
> If anyone has the "itch" this'd make a nice self-contained project and
> solid improvement to Lucene...
>
> Mike
>
> On Mon, Jul 6, 2009 at 10:31 PM, Nigel<nigelspleen@...> wrote:
> > On Mon, Jul 6, 2009 at 12:37 PM, Michael McCandless <
> > lucene@...> wrote:
> >
> >> On Mon, Jun 29, 2009 at 9:33 AM, Nigel<nigelspleen@...> wrote:
> >>
> >> > Ah, I was confused by the index divisor being 1 by default: I thought
> it
> >> > meant that all terms were being loaded.  I see now in SegmentTermEnum
> >> that
> >> > the every-128th behavior is implemented at a lower level.
> >> >
> >> > But I'm even more confused about why we have so many terms in memory.
>  A
> >> > heap dump shows over 270 million TermInfos, so if that's only 128th of
> >> the
> >> > total then we REALLY have a lot of terms.  (-:  We do have a lot of
> docs
> >> > (about 250 million), and we do have a couple unique per-document
> values,
> >> but
> >> > even so I can't see how we could get to 270 million x 128 terms.  (The
> >> heap
> >> > dump numbers are stable across the index close-and-reopen cycle, so I
> >> don't
> >> > think we're leaking.)
> >>
> >> You could use CheckIndex to see how many terms are in your index.
> >>
> >> If you do the heap dump after opening a fresh reader and not running
> >> any searches yet, you see 270 million TermInfos?
> >
> >
> > Thanks, Mike.  I'm just coming back to this after taking some time to
> > educate myself better on Lucene internals, mostly by reading and tracing
> > through code.
> >
> > I think now that the 270 million TermInfo number must have been user
> error
> > on my part, as I can't reproduce those values.  What I do see is about 8
> > million loaded TermInfos.  That matches what I expect by examining
> indexes
> > with CheckIndex: there are about 250 million terms per index, and we have
> 4
> > indexes loaded, so 1 billion terms / 128 = 8 million cached.  So, that's
> > still a big number (about 2gb including the associated Strings and
> arrays),
> > but at least it makes sense now.
> >
> > My next thought, which I'll try as soon as I can set up some reproducible
> > benchmarks, is using a larger index divisor, perhaps combined with a
> larger
> > LRU TermInfo cache.  But this seems like such an easy win that I wonder
> why
> > it isn't mentioned more often (at least, I haven't seen much discussion
> of
> > it in the java-user archives).  For example, if I simply increase the
> index
> > divisor from 1 to 4, I can cut my Lucene usage from 2gb to 500mb (meaning
> > less GC and more OS cache).  That means much more seeking to find
> non-cached
> > terms, but increasing the LRU cache to 100,000 (for example) would allow
> all
> > (I think) of our searched terms to be cached, at a fraction of the RAM
> cost
> > of the 8 million terms cached now.  (The first-time use of any term would
> of
> > course be slower, but most search terms are used repeatedly, and it seems
> > like a small price to pay for such a RAM win.)  Anyway, I'm curious if
> there
> > are any obvious flaws in this plan.
> >
> > Thanks,
> > Chris
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@...
> For additional commands, e-mail: java-user-help@...
>
>

[Nigel-13]

I created a benchmark test using real queries from our logs.  I kept the LRU
cache the same for now and varied the index divisor:

index divisor = 1: 768 sec.
index divisor = 4: 788 sec. (+ 3%)
index divisor = 8: 855 sec. (+ 11%)
index divisor = 16: 997 sec. (+ 30%)

This is exciting news for me, as it means we can cut our memory usage to
about 1/4th of what it is now, with negligible performance penalty.  And I'm
hoping that in real use the performance actually improves, as more RAM is
available for OS caching, or if I can reclaim some of the saved RAM for a
larger LRU cache, filter caching, etc.

I'll report further results when I get them.

Thanks,
Chris

On Tue, Jul 7, 2009 at 5:43 AM, Michael McCandless <
lucene@...> wrote:

> OK good to hear you have a sane number of TermInfos now...
>
> I think many apps don't have nearly as many unique terms as you do;
> your approach (increase index divisor & LRU cache) sounds reasonable.
> It'll make warming more important.  Please report back how it goes!
>
> > My next thought, which I'll try as soon as I can set up some reproducible
> > benchmarks, is using a larger index divisor, perhaps combined with a
> larger
> > LRU TermInfo cache.  But this seems like such an easy win that I wonder
> why
> > it isn't mentioned more often (at least, I haven't seen much discussion
> of
> > it in the java-user archives).  For example, if I simply increase the
> index
> > divisor from 1 to 4, I can cut my Lucene usage from 2gb to 500mb (meaning
> > less GC and more OS cache).  That means much more seeking to find
> non-cached
> > terms, but increasing the LRU cache to 100,000 (for example) would allow
> all
> > (I think) of our searched terms to be cached, at a fraction of the RAM
> cost
> > of the 8 million terms cached now.  (The first-time use of any term would
> of
> > course be slower, but most search terms are used repeatedly, and it seems
> > like a small price to pay for such a RAM win.)  Anyway, I'm curious if
> there
> > are any obvious flaws in this plan.
>

[eks dev]

Hi,
We just upgraded to 2.9 and noticed some (to me) not expected OOM.

We use MMapDirectory and after upgrade, on exactly the same Index/machine/jvm/params/setup... we cannot start index as mapping screams "No memory"

any explanation why this could be the case?





---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@...
For additional commands, e-mail: java-user-help@...

[Michael McCandless-2]

I would not expect a 2.9 IndexReader to consume more RAM.  Was this
definitely the case?  (It wasn't just a matter of other processes
taking up RAM).  If so, we should drill in to understand the root
cause / regression.

One thing you can do in 2.9 is IndexReader.setDisableFakeNorms(true),
to prevent allocation of the "fake" byte[maxDoc()] norms array, to
save RAM.

Mike

On Sun, Jul 12, 2009 at 7:22 AM, eks dev<eksdev@...> wrote:

>
> Hi,
> We just upgraded to 2.9 and noticed some (to me) not expected OOM.
>
> We use MMapDirectory and after upgrade, on exactly the same Index/machine/jvm/params/setup... we cannot start index as mapping screams "No memory"
>
> any explanation why this could be the case?
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@...
> For additional commands, e-mail: java-user-help@...
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@...
For additional commands, e-mail: java-user-help@...