Nabble - Lucene - Java Users

RE: [VOTE] Push fast-vector-highlighter mvn artifacts for 3.0.0 and 2.9.1

2009-12-09T01:28:10Z

Hi all,

The missing maven artifacts for the fast-vector-highlighter contrib of
Lucene Java in version 2.9.1 and 3.0.0 are now available at:

http://repo1.maven.org/maven2/org/apache/lucene/
http://repo2.maven.org/maven2/org/apache/lucene/

Uwe

-----
Uwe Schindler
uschindler@...
Apache Lucene Java Committer
Bremen, Germany
http://lucene.apache.org/java/docs/

> From: Uwe Schindler [mailto:uwe@...]
> Sent: Tuesday, December 08, 2009 10:41 PM
> To: java-dev@...; general@...
> Subject: RE: [VOTE] Push fast-vector-highlighter mvn artifacts for 3.0.0
> and 2.9.1
>
> I got 3 binding votes from Grant, Mike, and Ted (and one from Simon, who
> was
> a big help on Sunday evening when I created the artifacts), so I push the
> maven artifacts onto the rsync repo in few minutes.
>
> Thanks!
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@...
>
> > -----Original Message-----
> > From: Uwe Schindler [mailto:uwe@...]
> > Sent: Tuesday, December 08, 2009 7:03 PM
> > To: java-dev@...
> > Subject: [VOTE] Push fast-vector-highlighter mvn artifacts for 3.0.0 and
> > 2.9.1
> >
> > Sorry,
> >
> > I initially didn't want to start a vote, as Grant only proposed to
> "maybe
> > start one". But nobody responded (esp. to the questions in this mail) I
> > ask
> > again, an I will start the vote for now.
> >
> >
> ==========================================================================
> > ==
> > Please vote, that the missing artifacts for of fast-verctor-highlighter
> of
> > Lucene Java 2.9.1 and 3.0.0 should be pushed to repoX.maven.org.
> >
> > You can find the artifacts here:
> > http://people.apache.org/~uschindler/staging-area/
> >
> > This dir contains only the maven folder to be copied to maven-rsync dir
> on
> > p.a.o. The top-level version in the maven metadata is 3.0.0, which
> > conforms
> > to the current state on maven (so during merging both folders during
> > build,
> > I set preference to metadata.xml of 3.0.0).
> >
> > All files are signed by my PGP key (even the 2.9.1 ones; that release
> was
> > originally built by Mike McCandless).
> >
> ==========================================================================
> > ==
> >
> > What I additionally found out until now (because Simon nagged me):
> >
> > If you compare the JAR files inside the binary ZIP file from the apache
> > archive and the JAR files directly published on maven (for the other
> > contribs), the MD5s/SHA1s are different even as they are created from
> the
> > same source code (because the timestamps inside the JAR are different,
> for
> > 2.9.1 another JDK compiler/platform was used). This interestingly does
> not
> > apply to lucene-core.jar in 3.0. Because of that I see no problem with
> > this
> > maven release, even that they are not the orginal JAR files from the
> > binary
> > distrib.
> >
> > What is not nice, is that the svn revision number in the manifest is
> > different, but else is exactly the same, see my comments below in
> earlier
> > mails about changing the ant script for showing the SVN rev of the last
> > changed file.
> >
> > So if nobody objects to release these rebuild jar files, all signed by
> my
> > key, I would like to simply put them on the maven-rsync folder.
> >
> > Uwe
> >
> > -----
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: uwe@...
> >
> >
> > > -----Original Message-----
> > > From: Chris Hostetter [mailto:hossman_lucene@...]
> > > Sent: Tuesday, December 08, 2009 6:48 PM
> > > To: java-dev@...
> > > Subject: Re: (NAG) Push fast-vector-highlighter mvn artifacts for 3.0
> > and
> > > 2.9
> > >
> > >
> > > : What to do now, any votes on adding the missing maven artifacts for
> > > : fast-vector-highlighter to 2.9.1 and 3.0.0 on the apache maven
> > > reposititory?
> > >
> > > It's not even clear to me that anything special needs to be done
> before
> > > publishing those jars to maven. 2.9.1 and 3.0.0 were already voted on
> > and
> > > released -- including all of the source code in them.
> > >
> > > The safest bet least likely to anger the process gods is just to call
> a
> > > vote (new thread with VOTE in the subject) and cast a vote ...
> > considering
> > > the sources has already been reviewed it should go pretty quick.
> > >
> > > :
> > > : > I rebuilt the maven-dir for 2.9.1 and 3.0.0, merged them (3.0.0 is
> > > top-
> > > : > level
> > > : > version) and extracted only fast-vector-highlighter:
> > > : >
> > > : > http://people.apache.org/~uschindler/staging-area/
> > > : >
> > > : > I will copy this dir to the maven folder on people.a.o, when I got
> > > votes
> > > : > (how many)? At least someone should check the signatures.
> > > : >
> > > : > By the way, we have a small error in our ant build.xml that
> inserts
> > > : > svnversion into the manifest file. This version is not the version
> > of
> > > the
> > > : > last changed item (would be svnversion -c) but the current svn
> > > version,
> > > : > even
> > > : > that I checked out the corresponding tags. It's no problem at all,
> > but
> > > not
> > > : > very nice.
> > > : >
> > > : > Maybe we should change build.xml to call "svnversion -c" in
> future,
> > to
> > > get
> > > : > the real number.
> > > : >
> > > : > Uwe
> > > : >
> > > : > -----
> > > : > Uwe Schindler
> > > : > H.-H.-Meier-Allee 63, D-28213 Bremen
> > > : > http://www.thetaphi.de
> > > : > eMail: uwe@...
> > > : >
> > > : >
> > > : > > -----Original Message-----
> > > : > > From: Grant Ingersoll [mailto:gsingers@...]
> > > : > > Sent: Saturday, December 05, 2009 10:26 PM
> > > : > > To: java-dev@...
> > > : > > Subject: Re: Push fast-vector-highlighter mvn artifacts for 3.0
> > and
> > > 2.9
> > > : > >
> > > : > > I suppose we could put up the artifacts on a dev site and then
> we
> > > could
> > > : > > vote to release both of them pretty quickly. I think that
> should
> > be
> > > : > easy
> > > : > > to do, since it pretty much only involves verifying the jar and
> > the
> > > : > > signatures.
> > > : > >
> > > : > > On Dec 5, 2009, at 1:03 PM, Simon Willnauer wrote:
> > > : > >
> > > : > > > hi folks,
> > > : > > > The maven artifacts for fast-vector-highlighter have never
> been
> > > pushed
> > > : > > > since it was released because there were no pom.xml.template
> > > inside
> > > : > > > the module. I added a pom file a day ago in the context of
> > > : > > > LUCENE-2107. I already talked to uwe and grant how to deal
> with
> > > this
> > > : > > > issues and if we should push the artifact for Lucene 2.9 /
> 3.0.
> > > Since
> > > : > > > this is only a metadata file we could consider rebuilding the
> > > : > > > artefacts and publish them for those releases. I can not
> > remember
> > > that
> > > : > > > anything like that happened before, so we should discuss how
> to
> > > deal
> > > : > > > with this situation and if we should wait until 3.1.
> > > : > > >
> > > : > > > simon
> > > : > > >
> > > : > > > --------------------------------------------------------------
> --
> > --
> > > ---
> > > : > > > To unsubscribe, e-mail: java-dev-unsubscribe@...
> > > : > > > For additional commands, e-mail: java-dev-
> help@...
> > > : > > >
> > > : > >
> > > : > >
> > > : > >
> > > : > > ----------------------------------------------------------------
> --
> > --
> > > -
> > > : > > To unsubscribe, e-mail: java-dev-unsubscribe@...
> > > : > > For additional commands, e-mail: java-dev-help@...
> > > : >
> > > : >
> > > : >
> > > : > ------------------------------------------------------------------
> --
> > -
> > > : > To unsubscribe, e-mail: java-dev-unsubscribe@...
> > > : > For additional commands, e-mail: java-dev-help@...
> > > :
> > > :
> > > :
> > > : --------------------------------------------------------------------
> -
> > > : To unsubscribe, e-mail: java-dev-unsubscribe@...
> > > : For additional commands, e-mail: java-dev-help@...
> > > :
> > >
> > >
> > >
> > > -Hoss
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-dev-unsubscribe@...
> > > For additional commands, e-mail: java-dev-help@...
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-dev-unsubscribe@...
> > For additional commands, e-mail: java-dev-help@...
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@...
For additional commands, e-mail: java-user-help@...

Re: HOW to do date range searchi in 3.0

2009-12-08T23:12:21Z

Thanks, Uwe. I've found the problem. the updateTime field is lost when i
converted my index from an older version.

Another question, is there any detailed tutorial about Lucene 3.0.0?

2009/12/9 Uwe Schindler <uwe@...>

> How did you index your date?
>
> I would suggest to reindex the date using NumericField! And then query
> using
> NumericRangeQuery. If reindexing is not possible the Query like you have
> done, should work. Please give us examples of how you indexed and how you
> query.
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@...
>
> > -----Original Message-----
> > From: Weiwei Wang [mailto:ww.wang.cs@...]
> > Sent: Wednesday, December 09, 2009 4:23 AM
> > To: java-user@...
> > Subject: HOW to do date range searchi in 3.0
> >
> > Hi, all
> > I need to do a date range search like date:[a previous time to null]
> > I used a filter to do this job, the code is shown below:
> > Calendar c = Calendar.getInstance();
> > c.setTimeInMillis(c.getTimeInMillis() -
> > parameter.getRecentUpdateConstraint()
> > * RosaCrawlerConstants.ONE_DAY_IN_MILLISECOND);
> > String fromTime = DateTools.dateToString(c.getTime(),
> > DateTools.Resolution.DAY);
> > Query updateTimeRange = new
> > TermRangeQuery("updateTime",fromTime,null,true,false);
> > query.add(updateTimeRange, BooleanClause.Occur.MUST);
> >
> > However, it doesn't work as before in version 2.4.1(I'm updating my
> > project
> > from version 2.4.1 to lucenen 3.0.0)
> >
> > Could anybody here offer me a solution?
> > --
> > Weiwei Wang
> > Alex Wang
> > 王巍巍
> > Room 403, Mengmin Wei Building
> > Computer Science Department
> > Gulou Campus of Nanjing University
> > Nanjing, P.R.China, 210093
> >
> > Homepage: http://cs.nju.edu.cn/rl/weiweiwang
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@...
> For additional commands, e-mail: java-user-help@...
>
>

--
Weiwei Wang
Alex Wang
王巍巍
Room 403, Mengmin Wei Building
Computer Science Department
Gulou Campus of Nanjing University
Nanjing, P.R.China, 210093

Homepage: http://cs.nju.edu.cn/rl/weiweiwang

RE: HOW to do date range searchi in 3.0

2009-12-08T23:09:24Z

How did you index your date?

I would suggest to reindex the date using NumericField! And then query using
NumericRangeQuery. If reindexing is not possible the Query like you have
done, should work. Please give us examples of how you indexed and how you
query.

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@...

> -----Original Message-----
> From: Weiwei Wang [mailto:ww.wang.cs@...]
> Sent: Wednesday, December 09, 2009 4:23 AM
> To: java-user@...
> Subject: HOW to do date range searchi in 3.0
>
> Hi, all
> I need to do a date range search like date:[a previous time to null]
> I used a filter to do this job, the code is shown below:
> Calendar c = Calendar.getInstance();
> c.setTimeInMillis(c.getTimeInMillis() -
> parameter.getRecentUpdateConstraint()
> * RosaCrawlerConstants.ONE_DAY_IN_MILLISECOND);
> String fromTime = DateTools.dateToString(c.getTime(),
> DateTools.Resolution.DAY);
> Query updateTimeRange = new
> TermRangeQuery("updateTime",fromTime,null,true,false);
> query.add(updateTimeRange, BooleanClause.Occur.MUST);
>
> However, it doesn't work as before in version 2.4.1(I'm updating my
> project
> from version 2.4.1 to lucenen 3.0.0)
>
> Could anybody here offer me a solution?
> --
> Weiwei Wang
> Alex Wang
> 王巍巍
> Room 403, Mengmin Wei Building
> Computer Science Department
> Gulou Campus of Nanjing University
> Nanjing, P.R.China, 210093
>
> Homepage: http://cs.nju.edu.cn/rl/weiweiwang

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@...
For additional commands, e-mail: java-user-help@...

Re: FileNotFoundException on index

2009-12-08T22:56:05Z

Hi Mike,

Missed your response on this,
What I was doing was physically removing index/write.lock if older than 8
hours, allowing another process of my indexer to run. I realize in
hindsight that there is no reason why I should be doing this and it was
really stupid. I think I was under the impression one of my pylucene
processes was hanging.

On Fri, Oct 9, 2009 at 3:44 AM, Michael McCandless <
lucene@...> wrote:

> You can use o.a.l.index.CheckIndex to fix the index. It will remove
> references to any segments that are missing or have problems during
> testing. First run it without -fix to see what problems there are.
> Then take a backup of the index. Then run it with -fix. The index
> will lose all docs in those segments that it removes.
>
> Can you describe what led up to this? Is it repeatable?
>
> Mike
>
> On Fri, Oct 9, 2009 at 12:37 AM, Max Lynch <ihasmax@...> wrote:
> > Missed your response, thanks Bernd.
> >
> > I don't think that's it, since I haven't been executing any commands like
> > that. The only thing I could think of is corruption. I've got the index
> > backed up in case there is a way to fix it (it won't matter in a week or
> so
> > since I cull any documents older than 25 days).
> >
> > Is there a way to fix this?
> >
> > Thanks.
> >
> > On Thu, Oct 8, 2009 at 3:01 AM, Bernd Fondermann <
> > bernd.fondermann@...> wrote:
> >
> >> Hi Max
> >>
> >> just a guess: maybe you deleted all *.c source files in that area and
> >> unintentionally deleted this index file, too.
> >>
> >> Bernd
> >>
> >> On Fri, Oct 2, 2009 at 17:10, Max Lynch <ihasmax@...> wrote:
> >> > I'm getting this error when I try to run my searcher and my indexer:
> >> >
> >> > Traceback (most recent call last):
> >> > self.searcher = lucene.IndexSearcher(self.directory)
> >> > JavaError: java.io.FileNotFoundException:
> >> /home/spider/misc/index/_275c.cfs
> >> > (No such file or directory)
> >> >
> >> > I don't know anything about the format of the Lucene index, but I
> notice
> >> I
> >> > have several _275* files from b to j but no c.
> >> >
> >> > Any ideas?
> >> >
> >> > Thanks.
> >> >
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@...
> >> For additional commands, e-mail: java-user-help@...
> >>
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@...
> For additional commands, e-mail: java-user-help@...
>
>

HOW to do date range searchi in 3.0

2009-12-08T19:23:02Z

Hi, all
I need to do a date range search like date:[a previous time to null]
I used a filter to do this job, the code is shown below:
Calendar c = Calendar.getInstance();
c.setTimeInMillis(c.getTimeInMillis() -
parameter.getRecentUpdateConstraint()
* RosaCrawlerConstants.ONE_DAY_IN_MILLISECOND);
String fromTime = DateTools.dateToString(c.getTime(),
DateTools.Resolution.DAY);
Query updateTimeRange = new
TermRangeQuery("updateTime",fromTime,null,true,false);
query.add(updateTimeRange, BooleanClause.Occur.MUST);

However, it doesn't work as before in version 2.4.1(I'm updating my project
from version 2.4.1 to lucenen 3.0.0)

Could anybody here offer me a solution?
--
Weiwei Wang
Alex Wang
王巍巍
Room 403, Mengmin Wei Building
Computer Science Department
Gulou Campus of Nanjing University
Nanjing, P.R.China, 210093

Homepage: http://cs.nju.edu.cn/rl/weiweiwang

NearSpansUnordered payloads not returning all the time

2009-12-08T16:34:07Z

Howdy,

I am wondering if anyone has seen
NearSpansUnordered.getPayload() not return payloads that are
verifiably accessible via IR.termPositions? It's a bit confusing
because most of the time they're returned properly.

I suspect the payload logic gets tripped up in
NearSpansUnordered. I'll put together a test case, however the
difficulty is that we're only seeing the issue with largish 800
MB indexes, which could make the test case a little crazy.

Jason

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@...
For additional commands, e-mail: java-user-help@...

RE: TopFieldDocCollector and v3.0.0

2009-12-08T13:43:53Z

Sorry wrong word, Germans often have the problem with English "must". It has
to be to be "but you must not".

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@...

> -----Original Message-----
> From: Steven A Rowe [mailto:sarowe@...]
> Sent: Tuesday, December 08, 2009 8:42 PM
> To: java-user@...
> Subject: RE: TopFieldDocCollector and v3.0.0
>
> Hi Uwe,
>
> On 12/08/2009 at 9:40 AM, Uwe Schindler wrote:
> > After the move to 3.0, you can (but you must not) further update
> > your code to use generics, which is not really needed but will
> > remove all compiler warnings.
>
> This sounds like you're telling people that although they are able to
> update their code to use generics, it is forbidden.
>
> I'm sure, though, that you mean that they are not required to do so:
> something like "but you need not" rather than "but you must not".
>
> Steve
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@...
> For additional commands, e-mail: java-user-help@...

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@...
For additional commands, e-mail: java-user-help@...

Re: question related to Indexing

2009-12-08T12:41:48Z

Hello Tom and Erick,
I am really sorry for posting such a dull
question. Meanwhile I have explored a few other parts of API,
fortunately I have found a place which could exaclty fit for my case.
Thanks for patiently trying to understand my question.. and warning
me.
Bye.

On Tue, Dec 8, 2009 at 9:21 PM, Tom Hill <solr-list@...> wrote:

> If you tell us WHY you want to do this, rather than HOW you want to do it,
> the chances are much better that someone can help.
>
> What's the business motivation here? What does the end user want to
> achieve?
>
> Tom
>
> On Tue, Dec 8, 2009 at 8:16 AM, Phanindra Reva <reva.phanindra@...>wrote:
>
>> Hello,
>> Thanks for the reply. *strange* was expected. I am trying to
>> store field names as payloads, so I need unedited field names during
>> analysis part. And later my plan is to replace all the field names
>> with a default value and then store the document in the index. So, If
>> its possible to get the reference of the Document after the analysis,
>> I could modify all the field names. Even a way of modifying ( of
>> course , it should be after analysis and before adding to the index )
>> the field-name values that are going to be added to the index will
>> suffice.
>> I guess.. this time you feel its much more strange, but that's my
>> task for which above mentioned is one way to accomplish.
>> Thanks.
>>
>> On Tue, Dec 8, 2009 at 4:55 PM, Erick Erickson <erickerickson@...>
>> wrote:
>> > You're right, it *does* seem strange <G>....
>> >
>> > I'm having a really hard time imagining a use-case
>> > for this capability, so it's hard to suggest
>> > an approach. Perhaps you could supply
>> > an outline of your use-case? This may be
>> > an XY problem.
>> >
>> > Best
>> > Erick
>> >
>> > On Tue, Dec 8, 2009 at 10:12 AM, Phanindra Reva <
>> reva.phanindra@...>wrote:
>> >
>> >> Hello All,
>> >> I am a newbie using Lucene. To be brief, I am just
>> >> wondering whether is there a point where we get the access to the
>> >> org.apache.lucene.document.Document (which is being indexed at the
>> >> moment) after the analysing part is completed but exactly before it
>> >> is added to the index. My whole aim is to modify all the field names
>> >> present in the document before its being added to the index, but I
>> >> need those field names un-edited during the analysis part.
>> >> If it seems strange.. please don't mind.
>> >> Thanks.
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: java-user-unsubscribe@...
>> >> For additional commands, e-mail: java-user-help@...
>> >>
>> >>
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@...
>> For additional commands, e-mail: java-user-help@...
>>
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@...
For additional commands, e-mail: java-user-help@...

Re: question related to Indexing

2009-12-08T12:21:36Z

If you tell us WHY you want to do this, rather than HOW you want to do it,
the chances are much better that someone can help.

What's the business motivation here? What does the end user want to
achieve?

Tom

On Tue, Dec 8, 2009 at 8:16 AM, Phanindra Reva <reva.phanindra@...>wrote:

> Hello,
> Thanks for the reply. *strange* was expected. I am trying to
> store field names as payloads, so I need unedited field names during
> analysis part. And later my plan is to replace all the field names
> with a default value and then store the document in the index. So, If
> its possible to get the reference of the Document after the analysis,
> I could modify all the field names. Even a way of modifying ( of
> course , it should be after analysis and before adding to the index )
> the field-name values that are going to be added to the index will
> suffice.
> I guess.. this time you feel its much more strange, but that's my
> task for which above mentioned is one way to accomplish.
> Thanks.
>
> On Tue, Dec 8, 2009 at 4:55 PM, Erick Erickson <erickerickson@...>
> wrote:
> > You're right, it *does* seem strange <G>....
> >
> > I'm having a really hard time imagining a use-case
> > for this capability, so it's hard to suggest
> > an approach. Perhaps you could supply
> > an outline of your use-case? This may be
> > an XY problem.
> >
> > Best
> > Erick
> >
> > On Tue, Dec 8, 2009 at 10:12 AM, Phanindra Reva <
> reva.phanindra@...>wrote:
> >
> >> Hello All,
> >> I am a newbie using Lucene. To be brief, I am just
> >> wondering whether is there a point where we get the access to the
> >> org.apache.lucene.document.Document (which is being indexed at the
> >> moment) after the analysing part is completed but exactly before it
> >> is added to the index. My whole aim is to modify all the field names
> >> present in the document before its being added to the index, but I
> >> need those field names un-edited during the analysis part.
> >> If it seems strange.. please don't mind.
> >> Thanks.
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@...
> >> For additional commands, e-mail: java-user-help@...
> >>
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@...
> For additional commands, e-mail: java-user-help@...
>
>

RE: TopFieldDocCollector and v3.0.0

2009-12-08T11:41:34Z

Hi Uwe,

On 12/08/2009 at 9:40 AM, Uwe Schindler wrote:
> After the move to 3.0, you can (but you must not) further update
> your code to use generics, which is not really needed but will
> remove all compiler warnings.

This sounds like you're telling people that although they are able to update their code to use generics, it is forbidden.

I'm sure, though, that you mean that they are not required to do so: something like "but you need not" rather than "but you must not".

Steve

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@...
For additional commands, e-mail: java-user-help@...

Re: Problem searching field with % as value

2009-12-08T09:18:23Z

If you store the field unanalyzed it will be indexed as is. You can
then search for it via a TermQuery, or use QueryParser with
PerFieldAnalyzerWrapper specifying KeywordAnalyzer for the field
containing this character.

Another approach is to replace the % with something easier to work
with. You could do this yourself or, I think, with MappingCharFilter.

Personally I'd probably replace "%" with "percent" somewhere in my
code with a simple string replacement.

--
Ian.

On Tue, Dec 8, 2009 at 5:02 PM, kanayo <richies4all@...> wrote:

>
> Thanks for your reply Erick.
>
> In Luke, its also not working. I tried to retrieving values from the field
> by specifying the field as the search field and then specify % as the search
> parameter which using StandardAnalyzer but nothing is displayed. Also while
> Luke shows the query details for other search values, it dosent show query
> details for search value of %.
>
> I think it is not Tokenized in the index. Is there anything else i can do to
> be able to retrieve values from fields comprising of just %?
>
> Thanks for your assistance.
>
>
> Erick Erickson wrote:
>>
>> Try printing out query.toString() to see what's actually being
>> sent to the searcher.
>>
>> You can try the same thing in Luke, specifying StandardAnalyzer
>> to parse queries.
>>
>> Are you sure you're specifying the fields in the query and not just the
>> '%'? That would go against your default field.
>>
>> When you say that you can see the fields in luke, are you storing the
>> field?
>> Because what you may be seeing is the *stored* value rather than the
>> *tokens*.
>> Make sure you're looking at the tokens in Luke..
>>
>> If none of that helps, could you post a code snippet or two (index and
>> query)?
>>
>> Best
>> Erick
>>
>> On Tue, Dec 8, 2009 at 11:04 AM, kanayo <richies4all@...> wrote:
>>
>>>
>>> Hi,
>>>
>>> I am a newbie to lucene. I am using Standard Analyzer in my lucene
>>> project.
>>> I am indexing some fields which may contain only "%" as a field value and
>>> it
>>> indexes fine and i can view the value against the field in the index
>>> using
>>> Luke.
>>>
>>> However when i try to retrieve the same field using indexsearcher and
>>> passing "%" as a query parameter nothing is retrieved. It is simply being
>>> ignored. I have also tried to escape the "%" while searching but still no
>>> results.
>>>
>>> Is there anything am not doing right?
>>>
>>> Thanks in advance for your assistance.
>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/Problem-searching-field-with---as-value-tp26696184p26696184.html
>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@...
>>> For additional commands, e-mail: java-user-help@...
>>>
>>>
>>
>>
>
> --
> View this message in context: http://old.nabble.com/Problem-searching-field-with---as-value-tp26696184p26696993.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@...
> For additional commands, e-mail: java-user-help@...
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@...
For additional commands, e-mail: java-user-help@...

Re: Problem searching field with % as value

2009-12-08T09:02:21Z

Thanks for your reply Erick.

In Luke, its also not working. I tried to retrieving values from the field by specifying the field as the search field and then specify % as the search parameter which using StandardAnalyzer but nothing is displayed. Also while Luke shows the query details for other search values, it dosent show query details for search value of %.

I think it is not Tokenized in the index. Is there anything else i can do to be able to retrieve values from fields comprising of just %?

Thanks for your assistance.

Erick Erickson wrote:

Try printing out query.toString() to see what's actually being
sent to the searcher.

You can try the same thing in Luke, specifying StandardAnalyzer
to parse queries.

Are you sure you're specifying the fields in the query and not just the
'%'? That would go against your default field.

When you say that you can see the fields in luke, are you storing the field?
Because what you may be seeing is the *stored* value rather than the
*tokens*.
Make sure you're looking at the tokens in Luke..

If none of that helps, could you post a code snippet or two (index and
query)?

Best
Erick

On Tue, Dec 8, 2009 at 11:04 AM, kanayo <richies4all@gmail.com> wrote:

>
> Hi,
>
> I am a newbie to lucene. I am using Standard Analyzer in my lucene project.
> I am indexing some fields which may contain only "%" as a field value and
> it
> indexes fine and i can view the value against the field in the index using
> Luke.
>
> However when i try to retrieve the same field using indexsearcher and
> passing "%" as a query parameter nothing is retrieved. It is simply being
> ignored. I have also tried to escape the "%" while searching but still no
> results.
>
> Is there anything am not doing right?
>
> Thanks in advance for your assistance.
>
>
> --
> View this message in context:
> http://old.nabble.com/Problem-searching-field-with---as-value-tp26696184p26696184.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Re: Problem searching field with % as value

2009-12-08T08:20:01Z

Try printing out query.toString() to see what's actually being
sent to the searcher.

You can try the same thing in Luke, specifying StandardAnalyzer
to parse queries.

Are you sure you're specifying the fields in the query and not just the
'%'? That would go against your default field.

When you say that you can see the fields in luke, are you storing the field?
Because what you may be seeing is the *stored* value rather than the
*tokens*.
Make sure you're looking at the tokens in Luke..

If none of that helps, could you post a code snippet or two (index and
query)?

Best
Erick

On Tue, Dec 8, 2009 at 11:04 AM, kanayo <richies4all@...> wrote:

>
> Hi,
>
> I am a newbie to lucene. I am using Standard Analyzer in my lucene project.
> I am indexing some fields which may contain only "%" as a field value and
> it
> indexes fine and i can view the value against the field in the index using
> Luke.
>
> However when i try to retrieve the same field using indexsearcher and
> passing "%" as a query parameter nothing is retrieved. It is simply being
> ignored. I have also tried to escape the "%" while searching but still no
> results.
>
> Is there anything am not doing right?
>
> Thanks in advance for your assistance.
>
>
> --
> View this message in context:
> http://old.nabble.com/Problem-searching-field-with---as-value-tp26696184p26696184.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@...
> For additional commands, e-mail: java-user-help@...
>
>

Re: question related to Indexing

2009-12-08T08:16:25Z

Hello,
Thanks for the reply. *strange* was expected. I am trying to
store field names as payloads, so I need unedited field names during
analysis part. And later my plan is to replace all the field names
with a default value and then store the document in the index. So, If
its possible to get the reference of the Document after the analysis,
I could modify all the field names. Even a way of modifying ( of
course , it should be after analysis and before adding to the index )
the field-name values that are going to be added to the index will
suffice.
I guess.. this time you feel its much more strange, but that's my
task for which above mentioned is one way to accomplish.
Thanks.

On Tue, Dec 8, 2009 at 4:55 PM, Erick Erickson <erickerickson@...> wrote:

> You're right, it *does* seem strange <G>....
>
> I'm having a really hard time imagining a use-case
> for this capability, so it's hard to suggest
> an approach. Perhaps you could supply
> an outline of your use-case? This may be
> an XY problem.
>
> Best
> Erick
>
> On Tue, Dec 8, 2009 at 10:12 AM, Phanindra Reva <reva.phanindra@...>wrote:
>
>> Hello All,
>> I am a newbie using Lucene. To be brief, I am just
>> wondering whether is there a point where we get the access to the
>> org.apache.lucene.document.Document (which is being indexed at the
>> moment) after the analysing part is completed but exactly before it
>> is added to the index. My whole aim is to modify all the field names
>> present in the document before its being added to the index, but I
>> need those field names un-edited during the analysis part.
>> If it seems strange.. please don't mind.
>> Thanks.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@...
>> For additional commands, e-mail: java-user-help@...
>>
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@...
For additional commands, e-mail: java-user-help@...

Problem searching field with % as value

2009-12-08T08:04:42Z

Hi,

I am a newbie to lucene. I am using Standard Analyzer in my lucene project.
I am indexing some fields which may contain only "%" as a field value and it indexes fine and i can view the value against the field in the index using Luke.

However when i try to retrieve the same field using indexsearcher and passing "%" as a query parameter nothing is retrieved. It is simply being ignored. I have also tried to escape the "%" while searching but still no results.

Is there anything am not doing right?

Thanks in advance for your assistance.

Re: question related to Indexing

2009-12-08T07:55:52Z

You're right, it *does* seem strange <G>....

I'm having a really hard time imagining a use-case
for this capability, so it's hard to suggest
an approach. Perhaps you could supply
an outline of your use-case? This may be
an XY problem.

Best
Erick

On Tue, Dec 8, 2009 at 10:12 AM, Phanindra Reva <reva.phanindra@...>wrote:

> Hello All,
> I am a newbie using Lucene. To be brief, I am just
> wondering whether is there a point where we get the access to the
> org.apache.lucene.document.Document (which is being indexed at the
> moment) after the analysing part is completed but exactly before it
> is added to the index. My whole aim is to modify all the field names
> present in the document before its being added to the index, but I
> need those field names un-edited during the analysis part.
> If it seems strange.. please don't mind.
> Thanks.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@...
> For additional commands, e-mail: java-user-help@...
>
>

question related to Indexing

2009-12-08T07:12:46Z

Hello All,
I am a newbie using Lucene. To be brief, I am just
wondering whether is there a point where we get the access to the
org.apache.lucene.document.Document (which is being indexed at the
moment) after the analysing part is completed but exactly before it
is added to the index. My whole aim is to modify all the field names
present in the document before its being added to the index, but I
need those field names un-edited during the analysis part.
If it seems strange.. please don't mind.
Thanks.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@...
For additional commands, e-mail: java-user-help@...

RE: TopFieldDocCollector and v3.0.0

2009-12-08T06:40:24Z

The only difference to 3.0 is, that after moving to 3.0, you can remove lots
of unsafe casts and use generics (which does not work in 2.9, as it is Java
1.4 only). So this is the good thing when directly moving to 3.0.

But as the release notes for 3.0 denote, for new users that want to start
new projects, 3.0 is the best start because they get a clean generified and
type safe API (whichout old crap). For users porting from older versions,
the move through 2.9.1 is recommended. After the move to 3.0, you can (but
you must not) further update your code to use generics, which is not really
needed but will remove all compiler warnings.

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@...

> -----Original Message-----
> From: Erick Erickson [mailto:erickerickson@...]
> Sent: Tuesday, December 08, 2009 2:24 PM
> To: java-user@...
> Subject: Re: TopFieldDocCollector and v3.0.0
>
> You might want to move to 2.9.1 first, find and fix all the deprecations
> and *then* move to 3.x.
>
> It seems like more work, but it's actually not. Especially if you have
> reasonable
> unit tests. Since lots of effort has been put into maintaining backwards
> compatibility in the 2.X versions, 2.9.1 should "just work" for you. Then
> you can remove the deprecated methods a little bit at a time rather than
> all at once. It's always tempting to do things in "one big edit", but then
> you
> never know which one of the many changes actually broke something. Unless
> you can code perfectly the first time, in which case I'd recommend we hire
> you and pay you a gazillion dollars <G>....
>
> Once all the deprecations are removed from 2.9.1, you shouldn't have to do
> anything to move to 3.x. So you get the best of both worlds....
>
> Best
> Erick
>
>
> On Tue, Dec 8, 2009 at 1:44 AM, Antony Bowesman <adb@...> wrote:
>
> > I'm on 2.3.2 and looking to move to 2.9.1 or 3.0.0
> >
> > In 2.9.1 TopFieldDocCollector is
> >
> > "Deprecated. Please use TopFieldCollector instead."
> >
> > in 3.0.0 TopFieldCollector says
> >
> > NOTE: This API is experimental and might change in incompatible ways in
> the
> > next release
> >
> > What is the suggested path for migrating TopFieldDocCollector usage?
> >
> > Antony
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@...
> > For additional commands, e-mail: java-user-help@...
> >
> >

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@...
For additional commands, e-mail: java-user-help@...

Re: IndexDivisor

2009-12-08T05:46:28Z

Visiting all the index terms, which must be done for any divisor !=
-1, generates a good amount of garbage. So if you're including that
garbage in your memory measurement, that would explain what you're
seeing, and switching to a memory profiler should show the true RAM
usage.

Mike

On Tue, Dec 8, 2009 at 6:29 AM, Ganesh <emailgane@...> wrote:

> Thanks Mike...
>
> As i explained before, I created a small app, which loads all the db, does term search (using term query) and calculates the memory consumption. I tried this with divisor value but after 100 there seems to be no difference.
>
> Just load the database with different divisor value and then perform warming using TermQuery. I feel there should be some difference when opening with different value. But it didn't. I will try to perform the same using profiler.
>
> Regards
> Ganesh
>
>
> ----- Original Message -----
> From: "Michael McCandless" <lucene@...>
> To: <java-user@...>
> Sent: Sunday, December 06, 2009 3:14 PM
> Subject: Re: IndexDivisor
>
>
> You really have to use a profiler, to trust the RAM measurements.
>
> Certain queries, eg function queries, can load the field cache. What
> kind of queries are you running?
>
> Mike
>
> On Fri, Dec 4, 2009 at 9:11 AM, Ganesh <emailgane@...> wrote:
>> I didn't run with profiler. I created a test app and run that.. I am opening multiple database.
>>
>> IndexReader opened with IndexDivisor: 100 //Open the reader with the divisor value
>> TermCount: 7046764 //Available unique terms in the db
>> Warmup done: //warming up
>> Memory: UsedMemory: 4 :FreeMemory:788 //Memory consumed after GC.
>>
>> I couldn't see the memory difference for the value btw 100 to 10000. I will try to run with profiler and get back to you.
>>
>> Other than terms what all the other things which consumes memory. There is no deletions, no norms and i am not doing sorting
>>
>> Regards
>> Ganesh
>>
>>
>> ----- Original Message -----
>> From: "Michael McCandless" <lucene@...>
>> To: <java-user@...>
>> Sent: Friday, December 04, 2009 7:01 PM
>> Subject: Re: IndexDivisor
>>
>>
>> I'm confused -- what are these attachments? Output from a memory profiler?
>>
>> Can you post the app you created?
>>
>> Mike
>>
>> On Fri, Dec 4, 2009 at 12:24 AM, Ganesh <emailgane@...> wrote:
>>> Thanks mike..
>>>
>>> Please find the attached file. I ran the testing for 1,100,1000,10000 divisor value. There is difference from 1 to 100 but there is no difference between 100 to 10000.
>>>
>>> I created a new application, in which i opened all reader and searcher and warmed up. Sleep for a minute and close all the searcher and reader.
>>>
>>> There is no deletions, no norms and i am not doing sorting.
>>>
>>> Regards
>>> Ganesh
>>>
>>>
>>>
>>> ----- Original Message -----
>>> From: "Michael McCandless" <lucene@...>
>>> To: <java-user@...>
>>> Sent: Thursday, December 03, 2009 9:35 PM
>>> Subject: Re: IndexDivisor
>>>
>>>
>>> Can you run w/ a memory profiler? I don't trust that gc is truly running.
>>>
>>> Mike
>>>
>>> On Thu, Dec 3, 2009 at 10:47 AM, Ganesh <emailgane@...> wrote:
>>>> I am doing GC before calculating the memory. Even i set my indexdivisor to 10000 but there seems to be no change.
>>>>
>>>> Below are my stats
>>>> IndexDivisor Memory
>>>> -1 7 MB
>>>> 1 486 MB
>>>> 100 180 MB
>>>> 1000 176 MB.
>>>> 10000 176MB
>>>>
>>>> Regards
>>>> Ganesh
>>>>
>>>> ----- Original Message -----
>>>> From: "Danil ŢORIN" <torindan@...>
>>>> To: <java-user@...>
>>>> Sent: Thursday, December 03, 2009 8:35 PM
>>>> Subject: Re: IndexDivisor
>>>>
>>>>
>>>> Run System.gc() exactly before measuring memory usage.
>>>>
>>>> On sun jvm it will FORCE gc (unless DisableExplicitGC is used).
>>>>
>>>>
>>>> On Thu, Dec 3, 2009 at 16:30, Ganesh <emailgane@...> wrote:
>>>>> Thanks mike.
>>>>>
>>>>> I am opening the reader and warming it up and then calculating the memory consumed.
>>>>> long usedMemory = runtime.totalMemory() - runtime.freeMemory();
>>>>>
>>>>> Regards
>>>>> Ganesh
>>>>>
>>>>> ----- Original Message -----
>>>>> From: "Michael McCandless" <lucene@...>
>>>>> To: <java-user@...>
>>>>> Sent: Thursday, December 03, 2009 6:22 PM
>>>>> Subject: Re: IndexDivisor
>>>>>
>>>>>
>>>>> On Thu, Dec 3, 2009 at 7:15 AM, Ganesh <emailgane@...> wrote:
>>>>>
>>>>>> Below are my stats
>>>>>> IndexDivisor Memory
>>>>>> -1 7 MB
>>>>>> 1 486 MB
>>>>>> 100 180 MB
>>>>>> 1000 176 MB.
>>>>>
>>>>> Do you simply create the IndexWriter & IndexReader, but do no
>>>>> searching/indexing?
>>>>>
>>>>> How are you measuring RAM? You should use a RAM profiler (eg YourKit)
>>>>> to get the "real" usage, not counting garbage that's yet to be
>>>>> collected.
>>>>>
>>>>> My guess is the shear process of Lucene scanning your terms index
>>>>> creates lots of garbage and you're measuring the RAM consumed by that
>>>>> garbage. The garbage should be harmless (it'll eventually get GCd &
>>>>> reused).
>>>>>
>>>>>> 1. Whether term vector will consume RAM. I think it should only consume disk space.
>>>>>
>>>>> No RAM is consumed.
>>>>>
>>>>>> 2. By setting RAM buffer for IW, Does it will allocate memory Or based on the usage the memory will be allocated / increased.
>>>>>
>>>>> Only a little RAM is allocated up front, I think. It's only as you
>>>>> start indexing docs that the big RAM is really allocated. After a
>>>>> flush the RAM is then reused (ie, not freed) for subsequent segments.
>>>>>
>>>>> Mike
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@...
>>>>> For additional commands, e-mail: java-user-help@...
>>>>>
>>>>> Send instant messages to your online friends http://in.messenger.yahoo.com
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@...
>>>>> For additional commands, e-mail: java-user-help@...
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@...
>>>> For additional commands, e-mail: java-user-help@...
>>>>
>>>> Send instant messages to your online friends http://in.messenger.yahoo.com
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@...
>>>> For additional commands, e-mail: java-user-help@...
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@...
>>> For additional commands, e-mail: java-user-help@...
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@...
>>> For additional commands, e-mail: java-user-help@...
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@...
>> For additional commands, e-mail: java-user-help@...
>>
>> Send instant messages to your online friends http://in.messenger.yahoo.com
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@...
>> For additional commands, e-mail: java-user-help@...
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@...
> For additional commands, e-mail: java-user-help@...
>
> Send instant messages to your online friends http://in.messenger.yahoo.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@...
> For additional commands, e-mail: java-user-help@...
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@...
For additional commands, e-mail: java-user-help@...

Re: IndexWriter creates multiple .cfs files

2009-12-08T05:27:02Z

If you're using reopen, be sure to close the old reader
if the new one isn't identical, something like:

IndexReader new = r.reopen();
if (new != reader) {
... // reader was reopened
reader.close();
}
reader = new;

Erick

On Tue, Dec 8, 2009 at 6:13 AM, Cool The Breezer
<techcool.kumar@...>wrote:

> Thanks Mike for your timely suggestion. Somehow readers are not reopened
> properly.
>
>
>
>
>
> ----- Original Message ----
> From: Michael McCandless <lucene@...>
> To: java-user@...
> Sent: Tue, December 8, 2009 3:31:22 PM
> Subject: Re: IndexWriter creates multiple .cfs files
>
> IndexWriter takes care of merging the CFSs down, over time. Have you
> changed your mergeFactor? It's odd to see 100s of CFSs.
>
> Or maybe you're not closing the old reader on reopening a new one?
> That would prevent deletion of the files.
>
> Mike
>
> On Tue, Dec 8, 2009 at 1:43 AM, Cool The Breezer
> <techcool.kumar@...> wrote:
> > Thanks Jason for quick reply and suggestion. I am definitely going to
> change directory implementation policy to have one .cfs file instead of
> hundreds in one directory.
> >
> > regards,
> > Ranjan
> >
> >
> > ----- Original Message ----
> > From: Jason Rutherglen <jason.rutherglen@...>
> > To: java-user@...
> > Sent: Tue, December 8, 2009 12:00:59 PM
> > Subject: Re: IndexWriter creates multiple .cfs files
> >
> > RB,
> >
> > That's expected behavior, each .cfs corresponds to all of a
> > segment's files. You could write your own directory
> > implementation that underneath writes to a single file. It's
> > usually good to present what you're trying to accomplish (i.e.
> > the why).
> >
> > Jason
> >
> > On Mon, Dec 7, 2009 at 10:25 PM, Cool The Breezer
> > <techcool.kumar@...> wrote:
> >> Hello Group,
> >> I am continuously updating an index and at the same
> time searcher also searches the index, which resulted in multiple .cfs files
> for each commit by IndexWriter. I am not sure whether this is an expected
> behavior or I need to merge each time after IndexWriter commits. In between
> the writer's commit, I reopen the IndexReader but still .cfs files keep on
> increasing. I would like to have one .cfs file which can be reopened
> frequently by reader and at the same time progressive commit by writer. Is
> it possible?
> >>
> >> - RB
> >>
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@...
> >> For additional commands, e-mail: java-user-help@...
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@...
> > For additional commands, e-mail: java-user-help@...
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@...
> > For additional commands, e-mail: java-user-help@...
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@...
> For additional commands, e-mail: java-user-help@...
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@...
> For additional commands, e-mail: java-user-help@...
>
>

Re: TopFieldDocCollector and v3.0.0

2009-12-08T05:24:21Z

You might want to move to 2.9.1 first, find and fix all the deprecations
and *then* move to 3.x.

It seems like more work, but it's actually not. Especially if you have
reasonable
unit tests. Since lots of effort has been put into maintaining backwards
compatibility in the 2.X versions, 2.9.1 should "just work" for you. Then
you can remove the deprecated methods a little bit at a time rather than
all at once. It's always tempting to do things in "one big edit", but then
you
never know which one of the many changes actually broke something. Unless
you can code perfectly the first time, in which case I'd recommend we hire
you and pay you a gazillion dollars <G>....

Once all the deprecations are removed from 2.9.1, you shouldn't have to do
anything to move to 3.x. So you get the best of both worlds....

Best
Erick

On Tue, Dec 8, 2009 at 1:44 AM, Antony Bowesman <adb@...> wrote:

> I'm on 2.3.2 and looking to move to 2.9.1 or 3.0.0
>
> In 2.9.1 TopFieldDocCollector is
>
> "Deprecated. Please use TopFieldCollector instead."
>
> in 3.0.0 TopFieldCollector says
>
> NOTE: This API is experimental and might change in incompatible ways in the
> next release
>
> What is the suggested path for migrating TopFieldDocCollector usage?
>
> Antony
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@...
> For additional commands, e-mail: java-user-help@...
>
>

Re: org.apache.lucene.search.RemoteSearchable missing

2009-12-08T04:39:03Z

Thanks, so many changes in 3.0.0

On Tue, Dec 8, 2009 at 8:32 PM, Mark Miller <markrmiller@...> wrote:

> Weiwei Wang wrote:
> > Hi,all,
> > I can't not find this class in the downloaded jar and I can't figure
> out
> > what's wrong.
> > Does anybody here know how to fix it?
> >
> >
> Its now in the remote Contrib.
>
> --
> - Mark
>
> http://www.lucidimagination.com
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@...
> For additional commands, e-mail: java-user-help@...
>
>

Re: org.apache.lucene.search.RemoteSearchable missing

2009-12-08T04:32:34Z

Weiwei Wang wrote:
> Hi,all,
> I can't not find this class in the downloaded jar and I can't figure out
> what's wrong.
> Does anybody here know how to fix it?
>
>
Its now in the remote Contrib.

--
- Mark

http://www.lucidimagination.com

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@...
For additional commands, e-mail: java-user-help@...

org.apache.lucene.search.RemoteSearchable missing

2009-12-08T04:31:38Z

Hi,all,
I can't not find this class in the downloaded jar and I can't figure out
what's wrong.
Does anybody here know how to fix it?

--
Weiwei Wang
Alex Wang
王巍巍
Room 403, Mengmin Wei Building
Computer Science Department
Gulou Campus of Nanjing University
Nanjing, P.R.China, 210093

Homepage: http://cs.nju.edu.cn/rl/weiweiwang

Re: IndexDivisor

2009-12-08T03:29:40Z

Thanks Mike...

As i explained before, I created a small app, which loads all the db, does term search (using term query) and calculates the memory consumption. I tried this with divisor value but after 100 there seems to be no difference.

Just load the database with different divisor value and then perform warming using TermQuery. I feel there should be some difference when opening with different value. But it didn't. I will try to perform the same using profiler.

Regards
Ganesh

----- Original Message -----
From: "Michael McCandless" <lucene@...>
To: <java-user@...>
Sent: Sunday, December 06, 2009 3:14 PM
Subject: Re: IndexDivisor

You really have to use a profiler, to trust the RAM measurements.

Certain queries, eg function queries, can load the field cache. What
kind of queries are you running?

Mike

On Fri, Dec 4, 2009 at 9:11 AM, Ganesh <emailgane@...> wrote:

> I didn't run with profiler. I created a test app and run that.. I am opening multiple database.
>
> IndexReader opened with IndexDivisor: 100 //Open the reader with the divisor value
> TermCount: 7046764 //Available unique terms in the db
> Warmup done: //warming up
> Memory: UsedMemory: 4 :FreeMemory:788 //Memory consumed after GC.
>
> I couldn't see the memory difference for the value btw 100 to 10000. I will try to run with profiler and get back to you.
>
> Other than terms what all the other things which consumes memory. There is no deletions, no norms and i am not doing sorting
>
> Regards
> Ganesh
>
>
> ----- Original Message -----
> From: "Michael McCandless" <lucene@...>
> To: <java-user@...>
> Sent: Friday, December 04, 2009 7:01 PM
> Subject: Re: IndexDivisor
>
>
> I'm confused -- what are these attachments? Output from a memory profiler?
>
> Can you post the app you created?
>
> Mike
>
> On Fri, Dec 4, 2009 at 12:24 AM, Ganesh <emailgane@...> wrote:
>> Thanks mike..
>>
>> Please find the attached file. I ran the testing for 1,100,1000,10000 divisor value. There is difference from 1 to 100 but there is no difference between 100 to 10000.
>>
>> I created a new application, in which i opened all reader and searcher and warmed up. Sleep for a minute and close all the searcher and reader.
>>
>> There is no deletions, no norms and i am not doing sorting.
>>
>> Regards
>> Ganesh
>>
>>
>>
>> ----- Original Message -----
>> From: "Michael McCandless" <lucene@...>
>> To: <java-user@...>
>> Sent: Thursday, December 03, 2009 9:35 PM
>> Subject: Re: IndexDivisor
>>
>>
>> Can you run w/ a memory profiler? I don't trust that gc is truly running.
>>
>> Mike
>>
>> On Thu, Dec 3, 2009 at 10:47 AM, Ganesh <emailgane@...> wrote:
>>> I am doing GC before calculating the memory. Even i set my indexdivisor to 10000 but there seems to be no change.
>>>
>>> Below are my stats
>>> IndexDivisor Memory
>>> -1 7 MB
>>> 1 486 MB
>>> 100 180 MB
>>> 1000 176 MB.
>>> 10000 176MB
>>>
>>> Regards
>>> Ganesh
>>>
>>> ----- Original Message -----
>>> From: "Danil ŢORIN" <torindan@...>
>>> To: <java-user@...>
>>> Sent: Thursday, December 03, 2009 8:35 PM
>>> Subject: Re: IndexDivisor
>>>
>>>
>>> Run System.gc() exactly before measuring memory usage.
>>>
>>> On sun jvm it will FORCE gc (unless DisableExplicitGC is used).
>>>
>>>
>>> On Thu, Dec 3, 2009 at 16:30, Ganesh <emailgane@...> wrote:
>>>> Thanks mike.
>>>>
>>>> I am opening the reader and warming it up and then calculating the memory consumed.
>>>> long usedMemory = runtime.totalMemory() - runtime.freeMemory();
>>>>
>>>> Regards
>>>> Ganesh
>>>>
>>>> ----- Original Message -----
>>>> From: "Michael McCandless" <lucene@...>
>>>> To: <java-user@...>
>>>> Sent: Thursday, December 03, 2009 6:22 PM
>>>> Subject: Re: IndexDivisor
>>>>
>>>>
>>>> On Thu, Dec 3, 2009 at 7:15 AM, Ganesh <emailgane@...> wrote:
>>>>
>>>>> Below are my stats
>>>>> IndexDivisor Memory
>>>>> -1 7 MB
>>>>> 1 486 MB
>>>>> 100 180 MB
>>>>> 1000 176 MB.
>>>>
>>>> Do you simply create the IndexWriter & IndexReader, but do no
>>>> searching/indexing?
>>>>
>>>> How are you measuring RAM? You should use a RAM profiler (eg YourKit)
>>>> to get the "real" usage, not counting garbage that's yet to be
>>>> collected.
>>>>
>>>> My guess is the shear process of Lucene scanning your terms index
>>>> creates lots of garbage and you're measuring the RAM consumed by that
>>>> garbage. The garbage should be harmless (it'll eventually get GCd &
>>>> reused).
>>>>
>>>>> 1. Whether term vector will consume RAM. I think it should only consume disk space.
>>>>
>>>> No RAM is consumed.
>>>>
>>>>> 2. By setting RAM buffer for IW, Does it will allocate memory Or based on the usage the memory will be allocated / increased.
>>>>
>>>> Only a little RAM is allocated up front, I think. It's only as you
>>>> start indexing docs that the big RAM is really allocated. After a
>>>> flush the RAM is then reused (ie, not freed) for subsequent segments.
>>>>
>>>> Mike
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@...
>>>> For additional commands, e-mail: java-user-help@...
>>>>
>>>> Send instant messages to your online friends http://in.messenger.yahoo.com
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@...
>>>> For additional commands, e-mail: java-user-help@...
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@...
>>> For additional commands, e-mail: java-user-help@...
>>>
>>> Send instant messages to your online friends http://in.messenger.yahoo.com
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@...
>>> For additional commands, e-mail: java-user-help@...
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@...
>> For additional commands, e-mail: java-user-help@...
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@...
>> For additional commands, e-mail: java-user-help@...
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@...
> For additional commands, e-mail: java-user-help@...
>
> Send instant messages to your online friends http://in.messenger.yahoo.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@...
> For additional commands, e-mail: java-user-help@...
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@...
For additional commands, e-mail: java-user-help@...

Send instant messages to your online friends http://in.messenger.yahoo.com

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@...
For additional commands, e-mail: java-user-help@...

Re: IndexWriter creates multiple .cfs files

2009-12-08T03:13:48Z

Thanks Mike for your timely suggestion. Somehow readers are not reopened properly.

----- Original Message ----
From: Michael McCandless <lucene@...>
To: java-user@...
Sent: Tue, December 8, 2009 3:31:22 PM
Subject: Re: IndexWriter creates multiple .cfs files

IndexWriter takes care of merging the CFSs down, over time. Have you
changed your mergeFactor? It's odd to see 100s of CFSs.

Or maybe you're not closing the old reader on reopening a new one?
That would prevent deletion of the files.

Mike

On Tue, Dec 8, 2009 at 1:43 AM, Cool The Breezer
<techcool.kumar@...> wrote:

> Thanks Jason for quick reply and suggestion. I am definitely going to change directory implementation policy to have one .cfs file instead of hundreds in one directory.
>
> regards,
> Ranjan
>
>
> ----- Original Message ----
> From: Jason Rutherglen <jason.rutherglen@...>
> To: java-user@...
> Sent: Tue, December 8, 2009 12:00:59 PM
> Subject: Re: IndexWriter creates multiple .cfs files
>
> RB,
>
> That's expected behavior, each .cfs corresponds to all of a
> segment's files. You could write your own directory
> implementation that underneath writes to a single file. It's
> usually good to present what you're trying to accomplish (i.e.
> the why).
>
> Jason
>
> On Mon, Dec 7, 2009 at 10:25 PM, Cool The Breezer
> <techcool.kumar@...> wrote:
>> Hello Group,
>> I am continuously updating an index and at the same time searcher also searches the index, which resulted in multiple .cfs files for each commit by IndexWriter. I am not sure whether this is an expected behavior or I need to merge each time after IndexWriter commits. In between the writer's commit, I reopen the IndexReader but still .cfs files keep on increasing. I would like to have one .cfs file which can be reopened frequently by reader and at the same time progressive commit by writer. Is it possible?
>>
>> - RB
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@...
>> For additional commands, e-mail: java-user-help@...
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@...
> For additional commands, e-mail: java-user-help@...
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@...
> For additional commands, e-mail: java-user-help@...
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@...
For additional commands, e-mail: java-user-help@...

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@...
For additional commands, e-mail: java-user-help@...

Re: heap memory issues when sorting by a string field

2009-12-08T02:43:26Z

I've opened LUCENE-2135.

Mike

On Tue, Dec 8, 2009 at 5:36 AM, Michael McCandless
<lucene@...> wrote:

> This is a rather disturbing implementation detail of WeakHashMap, that
> it needs the one extra step (invoking one of its methods) for its weak
> keys to be reclaimable.
>
> Maybe on IndexReader.close(), Lucene should go and evict all entries
> in the FieldCache associated with that reader. Ie, step through the
> sub-readers, and if they are truly closed as well (not shared w/ other
> readers), evict. I'll open an issue.
>
> Even in TCK's code fragment, it's not until the final line is done
> executing, that the cache key even loses all hard references, because
> it's that line that assigns to sSearcher, replacing the strong
> reference to the old searcher. Inserting sSearcher = null prior to
> that would drop the hard reference sooner, but because of this impl
> detail of WeakHashMap, something would still have to touch it (eg, a
> warmup query that hits the field cache) before it's reclaimable.
>
> Mike
>
> On Mon, Dec 7, 2009 at 7:38 PM, Tom Hill <solr-list@...> wrote:
>> Hi -
>>
>> If I understand correctly, WeakHashMap does not free the memory for the
>> value (cached data) when the key is nulled, or even when the key is garbage
>> collected.
>>
>> It requires one more step: a method on WeakHashMap must be called to allow
>> it to release its hard reference to the cached data. It appears that most
>> methods in WeakHashMap end up calling expungeStaleEntries, which will clear
>> the hard reference. But you have to call some method on the map, before the
>> memory is eligible for garbage collection.
>>
>> So it requires four stages to free the cached data. Null the key; A GC to
>> release the weak reference to the key; A call to some method on the map;
>> Then the next GC cycle should free the value.
>>
>> So it seems possible that you could end up with double memory usage for a
>> time. If you don't have a GC between the time that you close the old reader,
>> and you start to load the field cache entry for the next reader, then the
>> key may still be hanging around uncollected.
>>
>> At that point, it may run a GC when you allocate the new cache, but that's
>> only the first GC. It can't free the cached data until after the next call
>> to expungeStaleEntries, so for a while you have both caches around.
>>
>> This extra usage could cause things to move into tenured space. Could this
>> be causing your problem?
>>
>> Workaround would be to cause some method to be called on the WeakHashMap.
>> You don't want to call get(), since that will try to populate the cache.
>> Maybe if you tried putting a small value to the cache, and doing a GC, and
>> see if your memory drops then.
>>
>>
>> Tom
>>
>>
>>
>> On Mon, Dec 7, 2009 at 1:48 PM, TCK <moonwatcher32329@...> wrote:
>>
>>> Thanks for the response. But I'm definitely calling close() on the old
>>> reader and opening a new one (not using reopen). Also, to simplify the
>>> analysis, I did my test with a single-threaded requester to eliminate any
>>> concurrency issues.
>>>
>>> I'm doing:
>>> sSearcher.getIndexReader().close();
>>> sSearcher.close(); // this actually seems to be a no-op
>>> IndexReader newIndexReader = IndexReader.open(newDirectory);
>>> sSearcher = new IndexSearcher(newIndexReader);
>>>
>>> Btw, isn't it bad practice anyway to have an unbounded cache? Are there any
>>> plans to replace the HashMaps used for the innerCaches with an actual
>>> size-bounded cache with some eviction policy (perhaps EhCache or something)
>>> ?
>>>
>>> Thanks again,
>>> TCK
>>>
>>>
>>>
>>>
>>> On Mon, Dec 7, 2009 at 4:37 PM, Erick Erickson <erickerickson@...
>>> >wrote:
>>>
>>> > What this sounds like is that you're not really closing your
>>> > readers even though you think you are. Sorting indeed uses up
>>> > significant memory when it populates internal caches and keeps
>>> > it around for later use (which is one of the reasons that warming
>>> > queries matter). But if you really do close the reader, I'm pretty
>>> > sure the memory should be GC-able.
>>> >
>>> > One thing that trips people up is IndexReader.reopen(). If it
>>> > returns a reader different than the original, you *must* close the
>>> > old one. If you don't, the old reader is still hanging around and
>>> > memory won't be returne.... An example from the Javadocs...
>>> >
>>> > IndexReader reader = ...
>>> > ...
>>> > IndexReader new = r.reopen();
>>> > if (new != reader) {
>>> > ... // reader was reopened
>>> > reader.close();
>>> > }
>>> > reader = new;
>>> > ...
>>> >
>>> >
>>> > If this is irrelevant, could you post your close/open
>>> >
>>> > code?
>>> >
>>> > HTH
>>> >
>>> > Erick
>>> >
>>> >
>>> > On Mon, Dec 7, 2009 at 4:27 PM, TCK <moonwatcher32329@...> wrote:
>>> >
>>> > > Hi,
>>> > > I'm having heap memory issues when I do lucene queries involving
>>> sorting
>>> > by
>>> > > a string field. Such queries seem to load a lot of data in to the heap.
>>> > > Moreover lucene seems to hold on to references to this data even after
>>> > the
>>> > > index reader has been closed and a full GC has been run. Some of the
>>> > > consequences of this are that in my generational heap configuration a
>>> lot
>>> > > of
>>> > > memory gets promoted to tenured space each time I close the old index
>>> > > reader
>>> > > and after opening and querying using a new one, and the tenured space
>>> > > eventually gets fragmented causing a lot of promotion failures
>>> resulting
>>> > in
>>> > > jvm hangs while the jvm does stop-the-world GCs.
>>> > >
>>> > > Does anyone know any workarounds to avoid these memory issues when
>>> doing
>>> > > such lucene queries?
>>> > >
>>> > > My profiling showed that even after a full GC lucene is holding on to a
>>> > lot
>>> > > of references to field value data notably via the
>>> > > FieldCacheImpl/ExtendedFieldCacheImpl. I noticed that the WeakHashMap
>>> > > readerCaches are using unbounded HashMaps as the innerCaches and I used
>>> > > reflection to replace these innerCaches with dummy empty HashMaps, but
>>> > > still
>>> > > I'm seeing the same behavior. I wondered if anyone has gone through
>>> these
>>> > > same issues before and would offer any advice.
>>> > >
>>> > > Thanks a lot,
>>> > > TCK
>>> > >
>>> >
>>>
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@...
For additional commands, e-mail: java-user-help@...

Re: heap memory issues when sorting by a string field

2009-12-08T02:36:24Z

This is a rather disturbing implementation detail of WeakHashMap, that
it needs the one extra step (invoking one of its methods) for its weak
keys to be reclaimable.

Maybe on IndexReader.close(), Lucene should go and evict all entries
in the FieldCache associated with that reader. Ie, step through the
sub-readers, and if they are truly closed as well (not shared w/ other
readers), evict. I'll open an issue.

Even in TCK's code fragment, it's not until the final line is done
executing, that the cache key even loses all hard references, because
it's that line that assigns to sSearcher, replacing the strong
reference to the old searcher. Inserting sSearcher = null prior to
that would drop the hard reference sooner, but because of this impl
detail of WeakHashMap, something would still have to touch it (eg, a
warmup query that hits the field cache) before it's reclaimable.

Mike

On Mon, Dec 7, 2009 at 7:38 PM, Tom Hill <solr-list@...> wrote:

> Hi -
>
> If I understand correctly, WeakHashMap does not free the memory for the
> value (cached data) when the key is nulled, or even when the key is garbage
> collected.
>
> It requires one more step: a method on WeakHashMap must be called to allow
> it to release its hard reference to the cached data. It appears that most
> methods in WeakHashMap end up calling expungeStaleEntries, which will clear
> the hard reference. But you have to call some method on the map, before the
> memory is eligible for garbage collection.
>
> So it requires four stages to free the cached data. Null the key; A GC to
> release the weak reference to the key; A call to some method on the map;
> Then the next GC cycle should free the value.
>
> So it seems possible that you could end up with double memory usage for a
> time. If you don't have a GC between the time that you close the old reader,
> and you start to load the field cache entry for the next reader, then the
> key may still be hanging around uncollected.
>
> At that point, it may run a GC when you allocate the new cache, but that's
> only the first GC. It can't free the cached data until after the next call
> to expungeStaleEntries, so for a while you have both caches around.
>
> This extra usage could cause things to move into tenured space. Could this
> be causing your problem?
>
> Workaround would be to cause some method to be called on the WeakHashMap.
> You don't want to call get(), since that will try to populate the cache.
> Maybe if you tried putting a small value to the cache, and doing a GC, and
> see if your memory drops then.
>
>
> Tom
>
>
>
> On Mon, Dec 7, 2009 at 1:48 PM, TCK <moonwatcher32329@...> wrote:
>
>> Thanks for the response. But I'm definitely calling close() on the old
>> reader and opening a new one (not using reopen). Also, to simplify the
>> analysis, I did my test with a single-threaded requester to eliminate any
>> concurrency issues.
>>
>> I'm doing:
>> sSearcher.getIndexReader().close();
>> sSearcher.close(); // this actually seems to be a no-op
>> IndexReader newIndexReader = IndexReader.open(newDirectory);
>> sSearcher = new IndexSearcher(newIndexReader);
>>
>> Btw, isn't it bad practice anyway to have an unbounded cache? Are there any
>> plans to replace the HashMaps used for the innerCaches with an actual
>> size-bounded cache with some eviction policy (perhaps EhCache or something)
>> ?
>>
>> Thanks again,
>> TCK
>>
>>
>>
>>
>> On Mon, Dec 7, 2009 at 4:37 PM, Erick Erickson <erickerickson@...
>> >wrote:
>>
>> > What this sounds like is that you're not really closing your
>> > readers even though you think you are. Sorting indeed uses up
>> > significant memory when it populates internal caches and keeps
>> > it around for later use (which is one of the reasons that warming
>> > queries matter). But if you really do close the reader, I'm pretty
>> > sure the memory should be GC-able.
>> >
>> > One thing that trips people up is IndexReader.reopen(). If it
>> > returns a reader different than the original, you *must* close the
>> > old one. If you don't, the old reader is still hanging around and
>> > memory won't be returne.... An example from the Javadocs...
>> >
>> > IndexReader reader = ...
>> > ...
>> > IndexReader new = r.reopen();
>> > if (new != reader) {
>> > ... // reader was reopened
>> > reader.close();
>> > }
>> > reader = new;
>> > ...
>> >
>> >
>> > If this is irrelevant, could you post your close/open
>> >
>> > code?
>> >
>> > HTH
>> >
>> > Erick
>> >
>> >
>> > On Mon, Dec 7, 2009 at 4:27 PM, TCK <moonwatcher32329@...> wrote:
>> >
>> > > Hi,
>> > > I'm having heap memory issues when I do lucene queries involving
>> sorting
>> > by
>> > > a string field. Such queries seem to load a lot of data in to the heap.
>> > > Moreover lucene seems to hold on to references to this data even after
>> > the
>> > > index reader has been closed and a full GC has been run. Some of the
>> > > consequences of this are that in my generational heap configuration a
>> lot
>> > > of
>> > > memory gets promoted to tenured space each time I close the old index
>> > > reader
>> > > and after opening and querying using a new one, and the tenured space
>> > > eventually gets fragmented causing a lot of promotion failures
>> resulting
>> > in
>> > > jvm hangs while the jvm does stop-the-world GCs.
>> > >
>> > > Does anyone know any workarounds to avoid these memory issues when
>> doing
>> > > such lucene queries?
>> > >
>> > > My profiling showed that even after a full GC lucene is holding on to a
>> > lot
>> > > of references to field value data notably via the
>> > > FieldCacheImpl/ExtendedFieldCacheImpl. I noticed that the WeakHashMap
>> > > readerCaches are using unbounded HashMaps as the innerCaches and I used
>> > > reflection to replace these innerCaches with dummy empty HashMaps, but
>> > > still
>> > > I'm seeing the same behavior. I wondered if anyone has gone through
>> these
>> > > same issues before and would offer any advice.
>> > >
>> > > Thanks a lot,
>> > > TCK
>> > >
>> >
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@...
For additional commands, e-mail: java-user-help@...

Re: IndexWriter creates multiple .cfs files

2009-12-08T02:01:22Z

IndexWriter takes care of merging the CFSs down, over time. Have you
changed your mergeFactor? It's odd to see 100s of CFSs.

Or maybe you're not closing the old reader on reopening a new one?
That would prevent deletion of the files.

Mike

On Tue, Dec 8, 2009 at 1:43 AM, Cool The Breezer
<techcool.kumar@...> wrote:

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@...
For additional commands, e-mail: java-user-help@...

RE: TopFieldDocCollector and v3.0.0

2009-12-07T23:39:56Z

That's a java docs issue and already fixed for the 3.0 branch. You have to
use the no longer experimental API.

Any by the way: Experimental means not "bad" or "functionally unstable", it
just means that the API might change on later releases.

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@...

> -----Original Message-----
> From: Antony Bowesman [mailto:adb@...]
> Sent: Tuesday, December 08, 2009 7:44 AM
> To: Lucene User
> Subject: TopFieldDocCollector and v3.0.0
>
> I'm on 2.3.2 and looking to move to 2.9.1 or 3.0.0
>
> In 2.9.1 TopFieldDocCollector is
>
> "Deprecated. Please use TopFieldCollector instead."
>
> in 3.0.0 TopFieldCollector says
>
> NOTE: This API is experimental and might change in incompatible ways in
> the next
> release
>
> What is the suggested path for migrating TopFieldDocCollector usage?
>
> Antony
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@...
> For additional commands, e-mail: java-user-help@...

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@...
For additional commands, e-mail: java-user-help@...

Re: heap memory issues when sorting by a string field

2009-12-07T22:51:46Z

TCK,

CSIndexInput is returned by SegmentReader.getFieldCacheKey()

If you think it's an issue, then it'd be good to open an issue and submit
some code as a patch, maybe a test case showing the WHM isn't
removing values like it's supposed to.

Jason

On Mon, Dec 7, 2009 at 10:45 PM, TCK <moonwatcher32329@...> wrote:

> Thanks for the feedback guys. The evidence I have collected does point to an
> issue either in the java WeakHashMap implementation or in Lucene's use of
> it. In particular, I used reflection to replace the WeakHashMap instances
> with my own dummy Map that does a no-op for the put operation, and although
> this caused tons more garbage to be created for each search query the CMS
> executions always collected that garbage and brought down the tenured space
> usage to a small value.
>
> Before that, I tried calling clear() on the WeakHashMap instances and then
> calling size() which calls expungeStaleEntries() but that didn't help. There
> may be indeed be a case of double memory usage for a while (until enough CMS
> executions have run) as Tom pointed out, but I don't fully understand what's
> happenning yet.
>
> Also, bizzarely during my introspection of the readerCache WeakHashMap
> instances I found an instance of
> org.apache.lucene.index.CompoundFileReader$CSIndexInput as a key. Looking at
> the code I don't see how anything other than an IndexReader could possibly
> get there, and this class certainly doesn't extend IndexReader. Any ideas?
> :-) Maybe I need to get more sleep.
>
> Btw, is searching with a sort by a text field not a common use-case of
> lucene? I've been testing with only 1Gb indexes and I'm pretty sure there
> are much larger indexes out there.
>
> Cheers,
> TCK
>
>
>
>
>
> On Mon, Dec 7, 2009 at 7:57 PM, Tom Hill <solr-list@...> wrote:
>
>> Hey, that's a nice little Class! I hadn't see it before. But it sounds like
>> the asynchronous cleanup might deal with the problem I mentioned above (but
>> I haven't looked at the code yet).
>>
>> It's an apache license - but you mentioned something about no third party
>> libraries. Is that a policy for Lucene?
>>
>> Thanks,
>>
>> Tom
>>
>>
>> On Mon, Dec 7, 2009 at 4:44 PM, Jason Rutherglen <
>> jason.rutherglen@...
>> > wrote:
>>
>> > I wonder if Google Collections (even though we don't use third party
>> > libraries) concurrent map, which supports weak keys, handles the
>> > removal of weakly referenced keys in a more elegant way than Java's
>> > WeakHashMap?
>> >
>> > On Mon, Dec 7, 2009 at 4:38 PM, Tom Hill <solr-list@...>
>> wrote:
>> > > Hi -
>> > >
>> > > If I understand correctly, WeakHashMap does not free the memory for the
>> > > value (cached data) when the key is nulled, or even when the key is
>> > garbage
>> > > collected.
>> > >
>> > > It requires one more step: a method on WeakHashMap must be called to
>> > allow
>> > > it to release its hard reference to the cached data. It appears that
>> most
>> > > methods in WeakHashMap end up calling expungeStaleEntries, which will
>> > clear
>> > > the hard reference. But you have to call some method on the map, before
>> > the
>> > > memory is eligible for garbage collection.
>> > >
>> > > So it requires four stages to free the cached data. Null the key; A GC
>> to
>> > > release the weak reference to the key; A call to some method on the
>> map;
>> > > Then the next GC cycle should free the value.
>> > >
>> > > So it seems possible that you could end up with double memory usage for
>> a
>> > > time. If you don't have a GC between the time that you close the old
>> > reader,
>> > > and you start to load the field cache entry for the next reader, then
>> the
>> > > key may still be hanging around uncollected.
>> > >
>> > > At that point, it may run a GC when you allocate the new cache, but
>> > that's
>> > > only the first GC. It can't free the cached data until after the next
>> > call
>> > > to expungeStaleEntries, so for a while you have both caches around.
>> > >
>> > > This extra usage could cause things to move into tenured space. Could
>> > this
>> > > be causing your problem?
>> > >
>> > > Workaround would be to cause some method to be called on the
>> WeakHashMap.
>> > > You don't want to call get(), since that will try to populate the
>> cache.
>> > > Maybe if you tried putting a small value to the cache, and doing a GC,
>> > and
>> > > see if your memory drops then.
>> > >
>> > >
>> > > Tom
>> > >
>> > >
>> > >
>> > > On Mon, Dec 7, 2009 at 1:48 PM, TCK <moonwatcher32329@...>
>> wrote:
>> > >
>> > >> Thanks for the response. But I'm definitely calling close() on the old
>> > >> reader and opening a new one (not using reopen). Also, to simplify the
>> > >> analysis, I did my test with a single-threaded requester to eliminate
>> > any
>> > >> concurrency issues.
>> > >>
>> > >> I'm doing:
>> > >> sSearcher.getIndexReader().close();
>> > >> sSearcher.close(); // this actually seems to be a no-op
>> > >> IndexReader newIndexReader = IndexReader.open(newDirectory);
>> > >> sSearcher = new IndexSearcher(newIndexReader);
>> > >>
>> > >> Btw, isn't it bad practice anyway to have an unbounded cache? Are
>> there
>> > any
>> > >> plans to replace the HashMaps used for the innerCaches with an actual
>> > >> size-bounded cache with some eviction policy (perhaps EhCache or
>> > something)
>> > >> ?
>> > >>
>> > >> Thanks again,
>> > >> TCK
>> > >>
>> > >>
>> > >>
>> > >>
>> > >> On Mon, Dec 7, 2009 at 4:37 PM, Erick Erickson <
>> erickerickson@...
>> > >> >wrote:
>> > >>
>> > >> > What this sounds like is that you're not really closing your
>> > >> > readers even though you think you are. Sorting indeed uses up
>> > >> > significant memory when it populates internal caches and keeps
>> > >> > it around for later use (which is one of the reasons that warming
>> > >> > queries matter). But if you really do close the reader, I'm pretty
>> > >> > sure the memory should be GC-able.
>> > >> >
>> > >> > One thing that trips people up is IndexReader.reopen(). If it
>> > >> > returns a reader different than the original, you *must* close the
>> > >> > old one. If you don't, the old reader is still hanging around and
>> > >> > memory won't be returne.... An example from the Javadocs...
>> > >> >
>> > >> > IndexReader reader = ...
>> > >> > ...
>> > >> > IndexReader new = r.reopen();
>> > >> > if (new != reader) {
>> > >> > ... // reader was reopened
>> > >> > reader.close();
>> > >> > }
>> > >> > reader = new;
>> > >> > ...
>> > >> >
>> > >> >
>> > >> > If this is irrelevant, could you post your close/open
>> > >> >
>> > >> > code?
>> > >> >
>> > >> > HTH
>> > >> >
>> > >> > Erick
>> > >> >
>> > >> >
>> > >> > On Mon, Dec 7, 2009 at 4:27 PM, TCK <moonwatcher32329@...>
>> > wrote:
>> > >> >
>> > >> > > Hi,
>> > >> > > I'm having heap memory issues when I do lucene queries involving
>> > >> sorting
>> > >> > by
>> > >> > > a string field. Such queries seem to load a lot of data in to the
>> > heap.
>> > >> > > Moreover lucene seems to hold on to references to this data even
>> > after
>> > >> > the
>> > >> > > index reader has been closed and a full GC has been run. Some of
>> the
>> > >> > > consequences of this are that in my generational heap
>> configuration
>> > a
>> > >> lot
>> > >> > > of
>> > >> > > memory gets promoted to tenured space each time I close the old
>> > index
>> > >> > > reader
>> > >> > > and after opening and querying using a new one, and the tenured
>> > space
>> > >> > > eventually gets fragmented causing a lot of promotion failures
>> > >> resulting
>> > >> > in
>> > >> > > jvm hangs while the jvm does stop-the-world GCs.
>> > >> > >
>> > >> > > Does anyone know any workarounds to avoid these memory issues when
>> > >> doing
>> > >> > > such lucene queries?
>> > >> > >
>> > >> > > My profiling showed that even after a full GC lucene is holding on
>> > to a
>> > >> > lot
>> > >> > > of references to field value data notably via the
>> > >> > > FieldCacheImpl/ExtendedFieldCacheImpl. I noticed that the
>> > WeakHashMap
>> > >> > > readerCaches are using unbounded HashMaps as the innerCaches and I
>> > used
>> > >> > > reflection to replace these innerCaches with dummy empty HashMaps,
>> > but
>> > >> > > still
>> > >> > > I'm seeing the same behavior. I wondered if anyone has gone
>> through
>> > >> these
>> > >> > > same issues before and would offer any advice.
>> > >> > >
>> > >> > > Thanks a lot,
>> > >> > > TCK
>> > >> > >
>> > >> >
>> > >>
>> > >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: java-user-unsubscribe@...
>> > For additional commands, e-mail: java-user-help@...
>> >
>> >
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@...
For additional commands, e-mail: java-user-help@...

Re: heap memory issues when sorting by a string field

2009-12-07T22:48:25Z

> It's an apache license - but you mentioned something about no third party
> libraries. Is that a policy for Lucene?

Pretty much... Though one can always submit a patch anyways.

On Mon, Dec 7, 2009 at 4:57 PM, Tom Hill <solr-list@...> wrote:

> Hey, that's a nice little Class! I hadn't see it before. But it sounds like
> the asynchronous cleanup might deal with the problem I mentioned above (but
> I haven't looked at the code yet).
>
> It's an apache license - but you mentioned something about no third party
> libraries. Is that a policy for Lucene?
>
> Thanks,
>
> Tom
>
>
> On Mon, Dec 7, 2009 at 4:44 PM, Jason Rutherglen <jason.rutherglen@...
>> wrote:
>
>> I wonder if Google Collections (even though we don't use third party
>> libraries) concurrent map, which supports weak keys, handles the
>> removal of weakly referenced keys in a more elegant way than Java's
>> WeakHashMap?
>>
>> On Mon, Dec 7, 2009 at 4:38 PM, Tom Hill <solr-list@...> wrote:
>> > Hi -
>> >
>> > If I understand correctly, WeakHashMap does not free the memory for the
>> > value (cached data) when the key is nulled, or even when the key is
>> garbage
>> > collected.
>> >
>> > It requires one more step: a method on WeakHashMap must be called to
>> allow
>> > it to release its hard reference to the cached data. It appears that most
>> > methods in WeakHashMap end up calling expungeStaleEntries, which will
>> clear
>> > the hard reference. But you have to call some method on the map, before
>> the
>> > memory is eligible for garbage collection.
>> >
>> > So it requires four stages to free the cached data. Null the key; A GC to
>> > release the weak reference to the key; A call to some method on the map;
>> > Then the next GC cycle should free the value.
>> >
>> > So it seems possible that you could end up with double memory usage for a
>> > time. If you don't have a GC between the time that you close the old
>> reader,
>> > and you start to load the field cache entry for the next reader, then the
>> > key may still be hanging around uncollected.
>> >
>> > At that point, it may run a GC when you allocate the new cache, but
>> that's
>> > only the first GC. It can't free the cached data until after the next
>> call
>> > to expungeStaleEntries, so for a while you have both caches around.
>> >
>> > This extra usage could cause things to move into tenured space. Could
>> this
>> > be causing your problem?
>> >
>> > Workaround would be to cause some method to be called on the WeakHashMap.
>> > You don't want to call get(), since that will try to populate the cache.
>> > Maybe if you tried putting a small value to the cache, and doing a GC,
>> and
>> > see if your memory drops then.
>> >
>> >
>> > Tom
>> >
>> >
>> >
>> > On Mon, Dec 7, 2009 at 1:48 PM, TCK <moonwatcher32329@...> wrote:
>> >
>> >> Thanks for the response. But I'm definitely calling close() on the old
>> >> reader and opening a new one (not using reopen). Also, to simplify the
>> >> analysis, I did my test with a single-threaded requester to eliminate
>> any
>> >> concurrency issues.
>> >>
>> >> I'm doing:
>> >> sSearcher.getIndexReader().close();
>> >> sSearcher.close(); // this actually seems to be a no-op
>> >> IndexReader newIndexReader = IndexReader.open(newDirectory);
>> >> sSearcher = new IndexSearcher(newIndexReader);
>> >>
>> >> Btw, isn't it bad practice anyway to have an unbounded cache? Are there
>> any
>> >> plans to replace the HashMaps used for the innerCaches with an actual
>> >> size-bounded cache with some eviction policy (perhaps EhCache or
>> something)
>> >> ?
>> >>
>> >> Thanks again,
>> >> TCK
>> >>
>> >>
>> >>
>> >>
>> >> On Mon, Dec 7, 2009 at 4:37 PM, Erick Erickson <erickerickson@...
>> >> >wrote:
>> >>
>> >> > What this sounds like is that you're not really closing your
>> >> > readers even though you think you are. Sorting indeed uses up
>> >> > significant memory when it populates internal caches and keeps
>> >> > it around for later use (which is one of the reasons that warming
>> >> > queries matter). But if you really do close the reader, I'm pretty
>> >> > sure the memory should be GC-able.
>> >> >
>> >> > One thing that trips people up is IndexReader.reopen(). If it
>> >> > returns a reader different than the original, you *must* close the
>> >> > old one. If you don't, the old reader is still hanging around and
>> >> > memory won't be returne.... An example from the Javadocs...
>> >> >
>> >> > IndexReader reader = ...
>> >> > ...
>> >> > IndexReader new = r.reopen();
>> >> > if (new != reader) {
>> >> > ... // reader was reopened
>> >> > reader.close();
>> >> > }
>> >> > reader = new;
>> >> > ...
>> >> >
>> >> >
>> >> > If this is irrelevant, could you post your close/open
>> >> >
>> >> > code?
>> >> >
>> >> > HTH
>> >> >
>> >> > Erick
>> >> >
>> >> >
>> >> > On Mon, Dec 7, 2009 at 4:27 PM, TCK <moonwatcher32329@...>
>> wrote:
>> >> >
>> >> > > Hi,
>> >> > > I'm having heap memory issues when I do lucene queries involving
>> >> sorting
>> >> > by
>> >> > > a string field. Such queries seem to load a lot of data in to the
>> heap.
>> >> > > Moreover lucene seems to hold on to references to this data even
>> after
>> >> > the
>> >> > > index reader has been closed and a full GC has been run. Some of the
>> >> > > consequences of this are that in my generational heap configuration
>> a
>> >> lot
>> >> > > of
>> >> > > memory gets promoted to tenured space each time I close the old
>> index
>> >> > > reader
>> >> > > and after opening and querying using a new one, and the tenured
>> space
>> >> > > eventually gets fragmented causing a lot of promotion failures
>> >> resulting
>> >> > in
>> >> > > jvm hangs while the jvm does stop-the-world GCs.
>> >> > >
>> >> > > Does anyone know any workarounds to avoid these memory issues when
>> >> doing
>> >> > > such lucene queries?
>> >> > >
>> >> > > My profiling showed that even after a full GC lucene is holding on
>> to a
>> >> > lot
>> >> > > of references to field value data notably via the
>> >> > > FieldCacheImpl/ExtendedFieldCacheImpl. I noticed that the
>> WeakHashMap
>> >> > > readerCaches are using unbounded HashMaps as the innerCaches and I
>> used
>> >> > > reflection to replace these innerCaches with dummy empty HashMaps,
>> but
>> >> > > still
>> >> > > I'm seeing the same behavior. I wondered if anyone has gone through
>> >> these
>> >> > > same issues before and would offer any advice.
>> >> > >
>> >> > > Thanks a lot,
>> >> > > TCK
>> >> > >
>> >> >
>> >>
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@...
>> For additional commands, e-mail: java-user-help@...
>>
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@...
For additional commands, e-mail: java-user-help@...

Re: heap memory issues when sorting by a string field

2009-12-07T22:45:34Z

Thanks for the feedback guys. The evidence I have collected does point to an
issue either in the java WeakHashMap implementation or in Lucene's use of
it. In particular, I used reflection to replace the WeakHashMap instances
with my own dummy Map that does a no-op for the put operation, and although
this caused tons more garbage to be created for each search query the CMS
executions always collected that garbage and brought down the tenured space
usage to a small value.

Before that, I tried calling clear() on the WeakHashMap instances and then
calling size() which calls expungeStaleEntries() but that didn't help. There
may be indeed be a case of double memory usage for a while (until enough CMS
executions have run) as Tom pointed out, but I don't fully understand what's
happenning yet.

Also, bizzarely during my introspection of the readerCache WeakHashMap
instances I found an instance of
org.apache.lucene.index.CompoundFileReader$CSIndexInput as a key. Looking at
the code I don't see how anything other than an IndexReader could possibly
get there, and this class certainly doesn't extend IndexReader. Any ideas?
:-) Maybe I need to get more sleep.

Btw, is searching with a sort by a text field not a common use-case of
lucene? I've been testing with only 1Gb indexes and I'm pretty sure there
are much larger indexes out there.

Cheers,
TCK

On Mon, Dec 7, 2009 at 7:57 PM, Tom Hill <solr-list@...> wrote:

> Hey, that's a nice little Class! I hadn't see it before. But it sounds like
> the asynchronous cleanup might deal with the problem I mentioned above (but
> I haven't looked at the code yet).
>
> It's an apache license - but you mentioned something about no third party
> libraries. Is that a policy for Lucene?
>
> Thanks,
>
> Tom
>
>
> On Mon, Dec 7, 2009 at 4:44 PM, Jason Rutherglen <
> jason.rutherglen@...
> > wrote:
>
> > I wonder if Google Collections (even though we don't use third party
> > libraries) concurrent map, which supports weak keys, handles the
> > removal of weakly referenced keys in a more elegant way than Java's
> > WeakHashMap?
> >
> > On Mon, Dec 7, 2009 at 4:38 PM, Tom Hill <solr-list@...>
> wrote:
> > > Hi -
> > >
> > > If I understand correctly, WeakHashMap does not free the memory for the
> > > value (cached data) when the key is nulled, or even when the key is
> > garbage
> > > collected.
> > >
> > > It requires one more step: a method on WeakHashMap must be called to
> > allow
> > > it to release its hard reference to the cached data. It appears that
> most
> > > methods in WeakHashMap end up calling expungeStaleEntries, which will
> > clear
> > > the hard reference. But you have to call some method on the map, before
> > the
> > > memory is eligible for garbage collection.
> > >
> > > So it requires four stages to free the cached data. Null the key; A GC
> to
> > > release the weak reference to the key; A call to some method on the
> map;
> > > Then the next GC cycle should free the value.
> > >
> > > So it seems possible that you could end up with double memory usage for
> a
> > > time. If you don't have a GC between the time that you close the old
> > reader,
> > > and you start to load the field cache entry for the next reader, then
> the
> > > key may still be hanging around uncollected.
> > >
> > > At that point, it may run a GC when you allocate the new cache, but
> > that's
> > > only the first GC. It can't free the cached data until after the next
> > call
> > > to expungeStaleEntries, so for a while you have both caches around.
> > >
> > > This extra usage could cause things to move into tenured space. Could
> > this
> > > be causing your problem?
> > >
> > > Workaround would be to cause some method to be called on the
> WeakHashMap.
> > > You don't want to call get(), since that will try to populate the
> cache.
> > > Maybe if you tried putting a small value to the cache, and doing a GC,
> > and
> > > see if your memory drops then.
> > >
> > >
> > > Tom
> > >
> > >
> > >
> > > On Mon, Dec 7, 2009 at 1:48 PM, TCK <moonwatcher32329@...>
> wrote:
> > >
> > >> Thanks for the response. But I'm definitely calling close() on the old
> > >> reader and opening a new one (not using reopen). Also, to simplify the
> > >> analysis, I did my test with a single-threaded requester to eliminate
> > any
> > >> concurrency issues.
> > >>
> > >> I'm doing:
> > >> sSearcher.getIndexReader().close();
> > >> sSearcher.close(); // this actually seems to be a no-op
> > >> IndexReader newIndexReader = IndexReader.open(newDirectory);
> > >> sSearcher = new IndexSearcher(newIndexReader);
> > >>
> > >> Btw, isn't it bad practice anyway to have an unbounded cache? Are
> there
> > any
> > >> plans to replace the HashMaps used for the innerCaches with an actual
> > >> size-bounded cache with some eviction policy (perhaps EhCache or
> > something)
> > >> ?
> > >>
> > >> Thanks again,
> > >> TCK
> > >>
> > >>
> > >>
> > >>
> > >> On Mon, Dec 7, 2009 at 4:37 PM, Erick Erickson <
> erickerickson@...
> > >> >wrote:
> > >>
> > >> > What this sounds like is that you're not really closing your
> > >> > readers even though you think you are. Sorting indeed uses up
> > >> > significant memory when it populates internal caches and keeps
> > >> > it around for later use (which is one of the reasons that warming
> > >> > queries matter). But if you really do close the reader, I'm pretty
> > >> > sure the memory should be GC-able.
> > >> >
> > >> > One thing that trips people up is IndexReader.reopen(). If it
> > >> > returns a reader different than the original, you *must* close the
> > >> > old one. If you don't, the old reader is still hanging around and
> > >> > memory won't be returne.... An example from the Javadocs...
> > >> >
> > >> > IndexReader reader = ...
> > >> > ...
> > >> > IndexReader new = r.reopen();
> > >> > if (new != reader) {
> > >> > ... // reader was reopened
> > >> > reader.close();
> > >> > }
> > >> > reader = new;
> > >> > ...
> > >> >
> > >> >
> > >> > If this is irrelevant, could you post your close/open
> > >> >
> > >> > code?
> > >> >
> > >> > HTH
> > >> >
> > >> > Erick
> > >> >
> > >> >
> > >> > On Mon, Dec 7, 2009 at 4:27 PM, TCK <moonwatcher32329@...>
> > wrote:
> > >> >
> > >> > > Hi,
> > >> > > I'm having heap memory issues when I do lucene queries involving
> > >> sorting
> > >> > by
> > >> > > a string field. Such queries seem to load a lot of data in to the
> > heap.
> > >> > > Moreover lucene seems to hold on to references to this data even
> > after
> > >> > the
> > >> > > index reader has been closed and a full GC has been run. Some of
> the
> > >> > > consequences of this are that in my generational heap
> configuration
> > a
> > >> lot
> > >> > > of
> > >> > > memory gets promoted to tenured space each time I close the old
> > index
> > >> > > reader
> > >> > > and after opening and querying using a new one, and the tenured
> > space
> > >> > > eventually gets fragmented causing a lot of promotion failures
> > >> resulting
> > >> > in
> > >> > > jvm hangs while the jvm does stop-the-world GCs.
> > >> > >
> > >> > > Does anyone know any workarounds to avoid these memory issues when
> > >> doing
> > >> > > such lucene queries?
> > >> > >
> > >> > > My profiling showed that even after a full GC lucene is holding on
> > to a
> > >> > lot
> > >> > > of references to field value data notably via the
> > >> > > FieldCacheImpl/ExtendedFieldCacheImpl. I noticed that the
> > WeakHashMap
> > >> > > readerCaches are using unbounded HashMaps as the innerCaches and I
> > used
> > >> > > reflection to replace these innerCaches with dummy empty HashMaps,
> > but
> > >> > > still
> > >> > > I'm seeing the same behavior. I wondered if anyone has gone
> through
> > >> these
> > >> > > same issues before and would offer any advice.
> > >> > >
> > >> > > Thanks a lot,
> > >> > > TCK
> > >> > >
> > >> >
> > >>
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@...
> > For additional commands, e-mail: java-user-help@...
> >
> >
>

TopFieldDocCollector and v3.0.0

2009-12-07T22:44:25Z

I'm on 2.3.2 and looking to move to 2.9.1 or 3.0.0

In 2.9.1 TopFieldDocCollector is

"Deprecated. Please use TopFieldCollector instead."

in 3.0.0 TopFieldCollector says

NOTE: This API is experimental and might change in incompatible ways in the next
release

What is the suggested path for migrating TopFieldDocCollector usage?

Antony

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@...
For additional commands, e-mail: java-user-help@...

Re: IndexWriter creates multiple .cfs files

2009-12-07T22:43:06Z

Thanks Jason for quick reply and suggestion. I am definitely going to change directory implementation policy to have one .cfs file instead of hundreds in one directory.

regards,
Ranjan

----- Original Message ----
From: Jason Rutherglen <jason.rutherglen@...>
To: java-user@...
Sent: Tue, December 8, 2009 12:00:59 PM
Subject: Re: IndexWriter creates multiple .cfs files

RB,

That's expected behavior, each .cfs corresponds to all of a
segment's files. You could write your own directory
implementation that underneath writes to a single file. It's
usually good to present what you're trying to accomplish (i.e.
the why).

Jason

On Mon, Dec 7, 2009 at 10:25 PM, Cool The Breezer
<techcool.kumar@...> wrote:

> Hello Group,
> I am continuously updating an index and at the same time searcher also searches the index, which resulted in multiple .cfs files for each commit by IndexWriter. I am not sure whether this is an expected behavior or I need to merge each time after IndexWriter commits. In between the writer's commit, I reopen the IndexReader but still .cfs files keep on increasing. I would like to have one .cfs file which can be reopened frequently by reader and at the same time progressive commit by writer. Is it possible?
>
> - RB
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@...
> For additional commands, e-mail: java-user-help@...
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@...
For additional commands, e-mail: java-user-help@...

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@...
For additional commands, e-mail: java-user-help@...