Nabble - Lucene - Java Developer

[jira] Commented: (LUCENE-2034) Massive Code Duplication in Contrib Analyzers - unifly the analyzer ctors

2009-12-24T04:00:30Z

[ https://issues.apache.org/jira/browse/LUCENE-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794396#action_12794396 ]

Simon Willnauer commented on LUCENE-2034:
-----------------------------------------

With LUCENE-2169 this patch becomes even more valuable. The creation of Analyzers extending StopwordAnalyzerBase will be way faster than in previous versions, with this in mind we should add a pointer to StopwordAnalyzerBase that recommends using charArraySet for stopwords and in the next step we should get WordlistLoader refactored and deprecate all public HashSet * methods in favour of Set<?> with CharArraySet as an internal implementation. Unifiying the way to create / load stopwords outside of a StopwordAnalyzerBase subclass goes the same way I would guess. We need one way to do it though. I will create correspondent issues within the next days.

> Massive Code Duplication in Contrib Analyzers - unifly the analyzer ctors
> -------------------------------------------------------------------------
>
> Key: LUCENE-2034
> URL: https://issues.apache.org/jira/browse/LUCENE-2034
> Project: Lucene - Java
> Issue Type: Improvement
> Components: contrib/analyzers
> Affects Versions: 2.9
> Reporter: Simon Willnauer
> Assignee: Robert Muir
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2034,patch, LUCENE-2034,patch, LUCENE-2034.patch, LUCENE-2034.patch, LUCENE-2034.patch, LUCENE-2034.patch, LUCENE-2034.patch, LUCENE-2034.patch, LUCENE-2034.patch, LUCENE-2034.patch, LUCENE-2034.patch, LUCENE-2034.txt
>
>
> Due to the variouse tokenStream APIs we had in lucene analyzer subclasses need to implement at least one of the methodes returning a tokenStream. When you look at the code it appears to be almost identical if both are implemented in the same analyzer. Each analyzer defnes the same inner class (SavedStreams) which is unnecessary.
> In contrib almost every analyzer uses stopwords and each of them creates his own way of loading them or defines a large number of ctors to load stopwords from a file, set, arrays etc.. those ctors should be removed / deprecated and eventually removed.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

[jira] Commented: (LUCENE-2162) ExtendableQueryParser should allow extensions to access the toplevel parser settings/ properties

2009-12-24T03:54:29Z

[ https://issues.apache.org/jira/browse/LUCENE-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794395#action_12794395 ]

Simon Willnauer commented on LUCENE-2162:
-----------------------------------------

I plan to commit this until Dec. 28. If nobody objects.

> ExtendableQueryParser should allow extensions to access the toplevel parser settings/ properties
> ------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2162
> URL: https://issues.apache.org/jira/browse/LUCENE-2162
> Project: Lucene - Java
> Issue Type: Improvement
> Components: contrib/*
> Reporter: Simon Willnauer
> Assignee: Simon Willnauer
> Priority: Trivial
> Fix For: 3.1
>
> Attachments: LUCENE-2162.patch
>
>
> Based on the latest discussions in LUCENE-2039 this issue will expose the toplevel parser via the ExtensionQuery so that ExtensionParsers can access properties like getAllowLeadingWildcards() from the top level parser.

[jira] Resolved: (LUCENE-2179) CharArraySet.clear()

2009-12-23T15:27:29Z

[ https://issues.apache.org/jira/browse/LUCENE-2179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uwe Schindler resolved LUCENE-2179.
-----------------------------------

Resolution: Fixed
Lucene Fields: [New, Patch Available] (was: [New])

Committed revision: 893655

> CharArraySet.clear()
> --------------------
>
> Key: LUCENE-2179
> URL: https://issues.apache.org/jira/browse/LUCENE-2179
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Analysis
> Reporter: Robert Muir
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2179.patch
>
>
> I needed CharArraySet.clear() for something I was working on in Solr in a tokenstream.
> instead I ended up using CharArrayMap<Boolean> because it supported .clear()
> it would be better to use a set though, currently it will throw UOE for .clear() because AbstractSet will call iterator.remove() which throws UOE.
> In Solr, the very similar CharArrayMap.clear() looks like this:
> {code}
> @Override
> public void clear() {
> count = 0;
> Arrays.fill(keys,null);
> Arrays.fill(values,null);
> }
> {code}
> I think we can do a similar thing as long as we throw UOE for the UnmodifiableCharArraySet
> will submit a patch later tonight (unless someone is bored and has nothing better to do)

[jira] Resolved: (LUCENE-2169) Speedup of CharArraySet#copy if a CharArraySet instance is passed to copy.

2009-12-23T15:27:29Z

[ https://issues.apache.org/jira/browse/LUCENE-2169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uwe Schindler resolved LUCENE-2169.
-----------------------------------

Resolution: Fixed
Lucene Fields: [New, Patch Available] (was: [New])

Committed revision: 893655

> Speedup of CharArraySet#copy if a CharArraySet instance is passed to copy.
> --------------------------------------------------------------------------
>
> Key: LUCENE-2169
> URL: https://issues.apache.org/jira/browse/LUCENE-2169
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Analysis
> Affects Versions: 3.1
> Reporter: Simon Willnauer
> Assignee: Uwe Schindler
> Fix For: 3.1
>
> Attachments: LUCENE-2169.patch, LUCENE-2169.patch, LUCENE-2169.patch, LUCENE-2169.patch
>
>
> the copy method should use the entries array itself to copy the set internally instead of iterating over all values. this would speedup copying even small set

[jira] Commented: (LUCENE-2179) CharArraySet.clear()

2009-12-23T12:48:29Z

[ https://issues.apache.org/jira/browse/LUCENE-2179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794218#action_12794218 ]

Uwe Schindler commented on LUCENE-2179:
---------------------------------------

For this to work, in BW branch the clear() test will be disabled.

[jira] Commented: (LUCENE-2179) CharArraySet.clear()

2009-12-23T12:26:29Z

[ https://issues.apache.org/jira/browse/LUCENE-2179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794208#action_12794208 ]

Uwe Schindler commented on LUCENE-2179:
---------------------------------------

I will commit this together with LUCENE-2169.

[jira] Commented: (LUCENE-2169) Speedup of CharArraySet#copy if a CharArraySet instance is passed to copy.

2009-12-23T12:24:29Z

[ https://issues.apache.org/jira/browse/LUCENE-2169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794207#action_12794207 ]

Uwe Schindler commented on LUCENE-2169:
---------------------------------------

OK.

I will commit this together with LUCENE-2179.

[jira] Commented: (LUCENE-2179) CharArraySet.clear()

2009-12-23T11:12:29Z

[ https://issues.apache.org/jira/browse/LUCENE-2179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794162#action_12794162 ]

Simon Willnauer commented on LUCENE-2179:
-----------------------------------------

patch looks good uwe! +1 from my side

[jira] Commented: (LUCENE-2169) Speedup of CharArraySet#copy if a CharArraySet instance is passed to copy.

2009-12-23T11:08:29Z

[ https://issues.apache.org/jira/browse/LUCENE-2169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794160#action_12794160 ]

Simon Willnauer commented on LUCENE-2169:
-----------------------------------------

bq. But you now changed the behaviour of copy(). Before the patch it changed the CharArraySets matchVersion... That was what my patch was doing to preserve this behaviour.

the CharArraySet did not have a Version before 3.1 so this code has never been released. Changing this behavior is fine and will not break anything though.

simon

[jira] Commented: (LUCENE-2026) Refactoring of IndexWriter

2009-12-23T11:08:29Z

[ https://issues.apache.org/jira/browse/LUCENE-2026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794161#action_12794161 ]

Michael McCandless commented on LUCENE-2026:
--------------------------------------------

{quote}
For what it's worth, we haven't really solved that problem in Lucy either.
The sliding window abstraction we wrapped around mmap/MapViewOfFile largely
solved the problem of running out of address space on 32-bit operating
systems. However, there's currently no way to invoke madvise through Lucy's
IO abstraction layer - it's a little tricky with compound files.

Linux, at least, requires that the buffer supplied to madvise be page-aligned.
So, say we're starting off on a posting list, and we want to communicate to
the OS that it should treat the region we're about to read as MADV_SEQUENTIAL.
If the start of the postings file is in the middle of a 4k page and the file
right before it is a term dictionary, we don't want to indicate that that
region should be treated as sequential.

I'm not sure how to solve that problem without violating the encapsulation of
the compound file model. Hmm, maybe we could store metadata about the virtual
files indicating usage patterns (sequential, random, etc.)? Since files are
generally part of dedicated data structures whose usage patterns are known at
index time.

Or maybe we just punt on that use case and worry only about segment merging.
{quote}

Storing metadata seems OK. It'd be optional for codecs to declare that...

{quote}
Hmm, wouldn't the act of deleting a file (and releasing all file descriptors) tell
the OS that it's free to recycle any memory pages associated with it?
{quote}

It better!

{quote}
bq. Actually why can't ord & offset be one, for the string sort cache? Ie, if you write your string data in sort order, then the offsets are also in sort order? (I think we may have discussed this already?)

Right, we discussed this on lucy-dev last spring:

http://markmail.org/message/epc56okapbgit5lw
{quote}

OK I'll go try to catch up... but I'm about to drop [sort of]
offline for a week and a half! There's alot of reading there! Should
be a prereq that we first go back and re-read what we said "the last
time"... ;)

{quote}
Incidentally, some of this thread replays our exchange at the top of
LUCENE-1458 from a year ago. It was fun to go back and reread that: in the
interrim, we've implemented segment-centric search and memory mapped field
caches and term dictionaries, both of which were first discussed back then.
{quote}

Nice!

{quote}
Ords are great for low cardinality fields of all kinds, but become less
efficient for high cardinality primitive numeric fields. For simplicity's
sake, the prototype implementation of mmap'd field caches in KS always uses
ords.
{quote}

Right...

{quote}
bq. You don't want to have to create Lucy's equivalent of the JMM...

The more I think about making Lucy classes thread safe, the harder it seems.
I'd like to make it possible to share a Schema across threads, for
instance, but that means all its Analyzers, etc have to be thread-safe as
well, which isn't practical when you start getting into contributed
subclasses.

Even if we succeed in getting Folders and FileHandles thread safe, it will be
hard for the user to keep track of what they can and can't do across threads.
"Don't share anything" is a lot easier to understand.

We reap a big benefit by making Lucy's metaclass infrastructure thread-safe.
Beyond that, seems like there's a lot of pain for little gain.
{quote}

Yeah. Threads are not easy :(

> Refactoring of IndexWriter
> --------------------------
>
> Key: LUCENE-2026
> URL: https://issues.apache.org/jira/browse/LUCENE-2026
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
>
> I've been thinking for a while about refactoring the IndexWriter into
> two main components.
> One could be called a SegmentWriter and as the
> name says its job would be to write one particular index segment. The
> default one just as today will provide methods to add documents and
> flushes when its buffer is full.
> Other SegmentWriter implementations would do things like e.g. appending or
> copying external segments [what addIndexes*() currently does].
> The second component's job would it be to manage writing the segments
> file and merging/deleting segments. It would know about
> DeletionPolicy, MergePolicy and MergeScheduler. Ideally it would
> provide hooks that allow users to manage external data structures and
> keep them in sync with Lucene's data during segment merges.
> API wise there are things we have to figure out, such as where the
> updateDocument() method would fit in, because its deletion part
> affects all segments, whereas the new document is only being added to
> the new segment.
> Of course these should be lower level APIs for things like parallel
> indexing and related use cases. That's why we should still provide
> easy to use APIs like today for people who don't need to care about
> per-segment ops during indexing. So the current IndexWriter could
> probably keeps most of its APIs and delegate to the new classes.

[jira] Commented: (LUCENE-2169) Speedup of CharArraySet#copy if a CharArraySet instance is passed to copy.

2009-12-23T11:04:29Z

[ https://issues.apache.org/jira/browse/LUCENE-2169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794157#action_12794157 ]

Uwe Schindler commented on LUCENE-2169:
---------------------------------------

But you now changed the behaviour of copy(). Before the patch it changed the CharArraySets matchVersion... That was what my patch was doing to preserve this behaviour.

[jira] Updated: (LUCENE-2169) Speedup of CharArraySet#copy if a CharArraySet instance is passed to copy.

2009-12-23T10:52:30Z

[ https://issues.apache.org/jira/browse/LUCENE-2169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Simon Willnauer updated LUCENE-2169:
------------------------------------

Attachment: LUCENE-2169.patch

patch that incorporates uwes additions to the testcases and enforces the Version of the source set in the copy if it is an instance of CharArraySet

[jira] Commented: (LUCENE-2169) Speedup of CharArraySet#copy if a CharArraySet instance is passed to copy.

2009-12-23T10:52:30Z

[ https://issues.apache.org/jira/browse/LUCENE-2169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794147#action_12794147 ]

Simon Willnauer commented on LUCENE-2169:
-----------------------------------------

Uwe, when I first looked at your patch I thought that is a good idea but once I looked at the usecases of CharArraySet differentiating between matchVersion if the given set is an instance of CharArraySet is not idea IMO. Imagine you create an analyzer with CharArraySet the analyzer will use its own given version together with the copy method internally if the analyzer is created with a different version than the provided stopset (which is already a CharArraySet) copy could change the behavior due to the given version with is actually the matchVersion for the analyzer not for the set. I would leave the decision to the user if a copy with a different version is what the user wants. If the version should not be preserved and the set to copy is a charArraySet users should use the constructor directly. I will attach a patch shortly

[jira] Updated: (LUCENE-2179) CharArraySet.clear()

2009-12-23T10:48:30Z

[ https://issues.apache.org/jira/browse/LUCENE-2179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uwe Schindler updated LUCENE-2179:
----------------------------------

Attachment: LUCENE-2179.patch

Patch for clear() and modified test. The test already checks, that the unmodifiable set cannot be cleared.

> CharArraySet.clear()
> --------------------
>
> Key: LUCENE-2179
> URL: https://issues.apache.org/jira/browse/LUCENE-2179
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Analysis
> Reporter: Robert Muir
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2179.patch
>
>
> I needed CharArraySet.clear() for something I was working on in Solr in a tokenstream.
> instead I ended up using CharArrayMap<Boolean> because it supported .clear()
> it would be better to use a set though, currently it will throw UOE for .clear() because AbstractSet will call iterator.remove() which throws UOE.
> In Solr, the very similar CharArrayMap.clear() looks like this:
> {code}
> @Override
> public void clear() {
> count = 0;
> Arrays.fill(keys,null);
> Arrays.fill(values,null);
> }
> {code}
> I think we can do a similar thing as long as we throw UOE for the UnmodifiableCharArraySet
> will submit a patch later tonight (unless someone is bored and has nothing better to do)

[jira] Assigned: (LUCENE-2179) CharArraySet.clear()

2009-12-23T10:48:30Z

[ https://issues.apache.org/jira/browse/LUCENE-2179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uwe Schindler reassigned LUCENE-2179:
-------------------------------------

Assignee: Uwe Schindler

[jira] Commented: (LUCENE-2026) Refactoring of IndexWriter

2009-12-23T10:26:29Z

[ https://issues.apache.org/jira/browse/LUCENE-2026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794137#action_12794137 ]

Marvin Humphrey commented on LUCENE-2026:
-----------------------------------------

> we can't give hints to the OS to tell it not to cache certain reads/writes
> (ie segment merging),

For what it's worth, we haven't really solved that problem in Lucy either.
The sliding window abstraction we wrapped around mmap/MapViewOfFile largely
solved the problem of running out of address space on 32-bit operating
systems. However, there's currently no way to invoke madvise through Lucy's
IO abstraction layer -- it's a little tricky with compound files.

Linux, at least, requires that the buffer supplied to madvise be page-aligned.
So, say we're starting off on a posting list, and we want to communicate to
the OS that it should treat the region we're about to read as MADV_SEQUENTIAL.
If the start of the postings file is in the middle of a 4k page and the file
right before it is a term dictionary, we don't want to indicate that that
region should be treated as sequential.

I'm not sure how to solve that problem without violating the encapsulation of
the compound file model. Hmm, maybe we could store metadata about the virtual
files indicating usage patterns (sequential, random, etc.)? Since files are
generally part of dedicated data structures whose usage patterns are known at
index time.

Or maybe we just punt on that use case and worry only about segment merging.
Hmm, wouldn't the act of deleting a file (and releasing all file descriptors) tell
the OS that it's free to recycle any memory pages associated with it?

> Actually why can't ord & offset be one, for the string sort cache?
> Ie, if you write your string data in sort order, then the offsets are
> also in sort order? (I think we may have discussed this already?)

Right, we discussed this on lucy-dev last spring:

http://markmail.org/message/epc56okapbgit5lw

Incidentally, some of this thread replays our exchange at the top of
LUCENE-1458 from a year ago. It was fun to go back and reread that: in the
interrim, we've implemented segment-centric search and memory mapped field
caches and term dictionaries, both of which were first discussed back then.
:)

Ords are great for low cardinality fields of all kinds, but become less
efficient for high cardinality primitive numeric fields. For simplicity's
sake, the prototype implementation of mmap'd field caches in KS always uses
ords.

> You don't want to have to create Lucy's equivalent of the JMM...

The more I think about making Lucy classes thread safe, the harder it seems.
:( I'd like to make it possible to share a Schema across threads, for
instance, but that means all its Analyzers, etc have to be thread-safe as
well, which isn't practical when you start getting into contributed
subclasses.

Even if we succeed in getting Folders and FileHandles thread safe, it will be
hard for the user to keep track of what they can and can't do across threads.
"Don't share anything" is a lot easier to understand.

We reap a big benefit by making Lucy's metaclass infrastructure thread-safe.
Beyond that, seems like there's a lot of pain for little gain.

[jira] Created: (LUCENE-2179) CharArraySet.clear()

2009-12-23T10:10:29Z

CharArraySet.clear()
--------------------

Key: LUCENE-2179
URL: https://issues.apache.org/jira/browse/LUCENE-2179
Project: Lucene - Java
Issue Type: Improvement
Components: Analysis
Reporter: Robert Muir
Priority: Minor
Fix For: 3.1

I needed CharArraySet.clear() for something I was working on in Solr in a tokenstream.

instead I ended up using CharArrayMap<Boolean> because it supported .clear()

it would be better to use a set though, currently it will throw UOE for .clear() because AbstractSet will call iterator.remove() which throws UOE.

In Solr, the very similar CharArrayMap.clear() looks like this:
{code}
@Override
public void clear() {
count = 0;
Arrays.fill(keys,null);
Arrays.fill(values,null);
}
{code}

I think we can do a similar thing as long as we throw UOE for the UnmodifiableCharArraySet

will submit a patch later tonight (unless someone is bored and has nothing better to do)

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

[jira] Commented: (LUCENE-2026) Refactoring of IndexWriter

2009-12-23T08:28:30Z

[ https://issues.apache.org/jira/browse/LUCENE-2026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794095#action_12794095 ]

Michael McCandless commented on LUCENE-2026:
--------------------------------------------

{quote}
bq. Very interesting - thanks. So it also factors in how much the page was used in the past, not just how long it's been since the page was last used.

In theory, I think that means the term dictionary will tend to be
favored over the posting lists. In practice... hard to say, it would
be difficult to test.
{quote}

Right... though, I think the top "trunks" frequently used by the
binary search, will stay hot. But as you get deeper into the terms
index, it's not as clear.

{quote}
bq. And of course Java pretty much forces threads-as-concurrency (JVM startup time, hotspot compilation, are costly).

Yes. Java does a lot of stuff that most operating systems can also do, but of
course provides a coherent platform-independent interface. In Lucy we're
going to try to go back to the OS for some of the stuff that Java likes to
take over - provided that we can develop a sane genericized interface using
configuration probing and #ifdefs.

It's nice that as long as the box is up our OS-as-JVM is always running, so we
don't have to worry about its (quite lengthy) startup time.
{quote}

OS as JVM is a nice analogy. Java of course gets in the way, too,
like we cannot properly set IO priorities, we can't give hints to the
OS to tell it not to cache certain reads/writes (ie segment merging),
can't pin pages ;), etc.

{quote}
bq. Right, this is how Lucy would force warming.

I think slurp-instead-of-mmap is orthogonal to warming, because we can warm
file-backed RAM structures by forcing them into the IO cache, using either the
cat-to-dev-null trick or something more sophisticated. The
slurp-instead-of-mmap setting would cause warming as a side effect, but the
main point would be to attempt to persuade the virtual memory system that
certain data structures should have a higher status and not be paged out as
quickly.
{quote}

Woops, sorry, I misread -- now I understand. You can easily make
certain files ram resident, and then be like Lucene (except the data
structures are more compact). Nice.

{quote}
bq. But, even within that CFS file, these three sub-files will not be local? Ie you'll still have to hit three pages per "lookup" right?

They'll be next to each other in the compound file because CompoundFileWriter
orders them alphabetically. For big segments, though, you're right that they
won't be right next to each other, and you could possibly incur as many as
three page faults when retrieving a sort cache value.

But what are the alternatives for variable width data like strings? You need
the ords array anyway for efficient comparisons, so what's left are the
offsets array and the character data.

An array of String objects isn't going to have better locality than one solid
block of memory dedicated to offsets and another solid block of memory
dedicated to file data, and it's no fewer derefs even if the string object
stores its character data inline - more if it points to a separate allocation
(like Lucy's CharBuf does, since it's mutable).

For each sort cache value lookup, you're going to need to access two blocks of
memory.

With the array of String objects, the first is the memory block dedicated
to the array, and the second is the memory block dedicated to the String
object itself, which contains the character data.
With the file-backed block sort cache, the first memory block is the
offsets array, and the second is the character data array.
I think the locality costs should be approximately the same... have I missed
anything?
{quote}

You're right, Lucene risks 3 (ord array, String array, String object)
page faults on each lookup as well.

Actually why can't ord & offset be one, for the string sort cache?
Ie, if you write your string data in sort order, then the offsets are
also in sort order? (I think we may have discussed this already?)

{quote}
bq. And it seems like Lucy would not need anything crazy-os-specific wrt threads?

It depends on how many classes we want to make thread-safe, and it's not just
the OS, it's the host.

The bare minimum is simply to make Lucy thread-safe as a library. That's
pretty close, because Lucy studiously avoided global variables whenever
possible. The only problems that have to be addressed are the VTable_registry
Hash, race conditions when creating new subclasses via dynamic VTable
singletons, and refcounts on the VTable objects themselves.

Once those issues are taken care of, you'll be able to use Lucy objects in
separate threads with no problem, e.g. one Searcher per thread.

However, if you want to share Lucy objects (other than VTables) across
threads, all of a sudden we have to start thinking about "synchronized",
"volatile", etc. Such constructs may not be efficient or even possible under
some threading models.
{quote}

OK it is indeed hairy. You don't want to have to create Lucy's
equivalent of the JMM...

{quote}
bq. Hmm I'd guess that field cache is slowish; deleted docs & norms are very fast; terms index is somewhere in between.

That jibes with my own experience. So maybe consider file-backed sort caches
in Lucene, while keeping the status quo for everything else?
{quote}

Perhaps, but it'd still make me nervous ;) When we get
CSF (LUCENE-1231) online we should make it
pluggable enough so that one could create an mmap impl.

{quote}
bq. You're right, you'd get two readers for seg_12 in that case. By "pool" I meant you're tapping into all the sub-readers that the existing reader have opened - the reader is your pool of sub-readers.

Each unique SegReader will also have dedicated "sub-reader" objects: two
"seg_12" SegReaders means two "seg_12" DocReaders, two "seg_12"
PostingsReaders, etc. However, all those sub-readers will share the same
file-backed RAM data, so in that sense they're pooled.
{quote}

OK

[jira] Commented: (LUCENE-2026) Refactoring of IndexWriter

2009-12-22T19:59:29Z

[ https://issues.apache.org/jira/browse/LUCENE-2026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793918#action_12793918 ]

Marvin Humphrey commented on LUCENE-2026:
-----------------------------------------

> Very interesting - thanks. So it also factors in how much the page
> was used in the past, not just how long it's been since the page was
> last used.

In theory, I think that means the term dictionary will tend to be favored over
the posting lists. In practice... hard to say, it would be difficult to test.
:(

> Even smallish indexes can see the pages swapped out?

Yes, you're right -- the wait time to get at a small term dictionary isn't
necessarily small. I've amended my previous post, thanks.

> And of course Java pretty much forces threads-as-concurrency (JVM
> startup time, hotspot compilation, are costly).

Yes. Java does a lot of stuff that most operating systems can also do, but of
course provides a coherent platform-independent interface. In Lucy we're
going to try to go back to the OS for some of the stuff that Java likes to
take over -- provided that we can develop a sane genericized interface using
configuration probing and #ifdefs.

It's nice that as long as the box is up our OS-as-JVM is always running, so we
don't have to worry about its (quite lengthy) startup time.

> Right, this is how Lucy would force warming.

I think slurp-instead-of-mmap is orthogonal to warming, because we can warm
file-backed RAM structures by forcing them into the IO cache, using either the
cat-to-dev-null trick or something more sophisticated. The
slurp-instead-of-mmap setting would cause warming as a side effect, but the
main point would be to attempt to persuade the virtual memory system that
certain data structures should have a higher status and not be paged out as
quickly.

> But, even within that CFS file, these three sub-files will not be
> local? Ie you'll still have to hit three pages per "lookup" right?

They'll be next to each other in the compound file because CompoundFileWriter
orders them alphabetically. For big segments, though, you're right that they
won't be right next to each other, and you could possibly incur as many as
three page faults when retrieving a sort cache value.

But what are the alternatives for variable width data like strings? You need
the ords array anyway for efficient comparisons, so what's left are the
offsets array and the character data.

An array of String objects isn't going to have better locality than one solid
block of memory dedicated to offsets and another solid block of memory
dedicated to file data, and it's no fewer derefs even if the string object
stores its character data inline -- more if it points to a separate allocation
(like Lucy's CharBuf does, since it's mutable).

For each sort cache value lookup, you're going to need to access two blocks of
memory.

* With the array of String objects, the first is the memory block dedicated
to the array, and the second is the memory block dedicated to the String
object itself, which contains the character data.
* With the file-backed block sort cache, the first memory block is the
offsets array, and the second is the character data array.

I think the locality costs should be approximately the same... have I missed
anything?

> Write-once is good for Lucene too.

Hellyeah.

> And it seems like Lucy would not need anything crazy-os-specific wrt
> threads?

It depends on how many classes we want to make thread-safe, and it's not just
the OS, it's the host.

The bare minimum is simply to make Lucy thread-safe as a library. That's
pretty close, because Lucy studiously avoided global variables whenever
possible. The only problems that have to be addressed are the VTable_registry
Hash, race conditions when creating new subclasses via dynamic VTable
singletons, and refcounts on the VTable objects themselves.

Once those issues are taken care of, you'll be able to use Lucy objects in
separate threads with no problem, e.g. one Searcher per thread.

However, if you want to *share* Lucy objects (other than VTables) across
threads, all of a sudden we have to start thinking about "synchronized",
"volatile", etc. Such constructs may not be efficient or even possible under
some threading models.

> Hmm I'd guess that field cache is slowish; deleted docs & norms are
> very fast; terms index is somewhere in between.

That jibes with my own experience. So maybe consider file-backed sort caches
in Lucene, while keeping the status quo for everything else?

> You're right, you'd get two readers for seg_12 in that case. By
> "pool" I meant you're tapping into all the sub-readers that the
> existing reader have opened - the reader is your pool of sub-readers.

Each unique SegReader will also have dedicated "sub-reader" objects: two
"seg_12" SegReaders means two "seg_12" DocReaders, two "seg_12"
PostingsReaders, etc. However, all those sub-readers will share the same
file-backed RAM data, so in that sense they're pooled.

[jira] Issue Comment Edited: (LUCENE-2026) Refactoring of IndexWriter

2009-12-22T19:55:29Z

[ https://issues.apache.org/jira/browse/LUCENE-2026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793431#action_12793431 ]

Marvin Humphrey edited comment on LUCENE-2026 at 12/23/09 3:54 AM:
-------------------------------------------------------------------

> I guess my confusion is what are all the other benefits of using
> file-backed RAM? You can efficiently use process only concurrency
> (though shared memory is technically an option for this too), and you
> have wicked fast open times (but, you still must warm, just like
> Lucene).

Processes are Lucy's primary concurrency model. ("The OS is our JVM.")
Making process-only concurrency efficient isn't optional -- it's a *core*
*concern*.

> What else? Oh maybe the ability to inform OS not to cache
> eg the reads done when merging segments. That's one I sure wish
> Lucene could use...

Lightweight searchers mean architectural freedom.

Create 2, 10, 100, 1000 Searchers without a second thought -- as many as you
need for whatever app architecture you just dreamed up -- then destroy them
just as effortlessly. Add another worker thread to your search server without
having to consider the RAM requirements of a heavy searcher object. Create a
command-line app to search a documentation index without worrying about
daemonizing it. Etc.

If your normal development pattern is a single monolithic Java process, then
that freedom might not mean much to you. But with their low per-object RAM
requirements and fast opens, lightweight searchers are easy to use within a
lot of other development patterns. For example: lightweight searchers work
well for maxing out multiple CPU cores under process-only concurrency.

> In exchange you risk the OS making poor choices about what gets
> swapped out (LRU policy is too simplistic... not all pages are created
> equal),

The Linux virtual memory system, at least, is not a pure LRU. It utilizes a
page aging algo which prioritizes pages that have historically been accessed
frequently even when they have not been accessed recently:

{panel}
http://sunsite.nus.edu.sg/LDP/LDP/tlk/node40.html

The default action when a page is first allocated, is to give it an
initial age of 3. Each time it is touched (by the memory management
subsystem) it's age is increased by 3 to a maximum of 20. Each time the
Kernel swap daemon runs it ages pages, decrementing their age by 1.
{panel}

And while that system may not be ideal from our standpoint, it's still pretty
good. In general, the operating system's virtual memory scheme is going to
work fine as designed, for us and everyone else, and minimize memory
availability wait times.

When will swapping out the term dictionary be a problem?

* For indexes where queries are made frequently, no problem.
* Foir systems with plenty of RAM, no problem.
* For systems that aren't very busy, no problem.
* -For small indexes, no problem.-

The only situation we're talking about is infrequent queries against -large-
indexes on busy boxes where RAM isn't abundant. Under those circumstances, it
*might* be noticable that Lucy's term dictionary gets paged out somewhat
sooner than Lucene's.

But in general, if the term dictionary gets paged out, so what? Nobody was
using it. Maybe nobody will make another query against that index until next
week. Maybe the OS made the right decision.

OK, so there's a vulnerable bubble where the the query rate against
-a large index- an index is neither too fast nor too slow, on busy machines
where RAM isn't abundant. I don't think that bubble ought to drive major
architectural decisions.

Let me turn your question on its head. What does Lucene gain in return for
the slow index opens and large process memory footprint of its heavy
searchers?

> I do love how pure the file-backed RAM approach is, but I worry that
> down the road it'll result in erratic search performance in certain
> app profiles.

If necessary, there's a straightforward remedy: slurp the relevant files into
RAM at object construction rather than mmap them. The rest of the code won't
know the difference between malloc'd RAM and mmap'd RAM. The slurped files
won't take up any more space than the analogous Lucene data structures; more
likely, they'll take up less.

That's the kind of setting we'd hide away in the IndexManager class rather
than expose as prominent API, and it would be a hint to index components
rather than an edict.

> Yeah, that you need 3 files for the string sort cache is a little
> spooky... that's 3X the chance of a page fault.

Not when using the compound format.

> But the CFS construction must also go through the filesystem (like
> Lucene) right? So you still incur IO load of creating the small
> files, then 2nd pass to consolidate.

Yes.

> I think we may need to largely take "time" out of our programming
> languages, eg switch to much more declarative code, or
> something... wanna port Lucy to Erlang?
>
> But I'm not sure process only concurrency, sharing only via
> file-backed memory, is the answer either

I think relying heavily on file-backed memory is particularly appropriate for
Lucy because the write-once file format works well with MAP_SHARED memory
segments. If files were being modified and had to be protected with
semaphores, it wouldn't be as sweet a match.

Focusing on process-only concurrency also works well for Lucy because host
threading models differ substantially and so will only be accessible via a
generalized interface from the Lucy C core. It will be difficult to tune
threading performance through that layer of indirection -- I'm guessing beyond
the ability of most developers since few will be experts in multiple host
threading models. In contrast, expertise in process level concurrency will be
easier to come by and to nourish.

> Using Zoie you can make reopen time insanely fast (much faster than I
> think necessary for most apps), but at the expense of some expected
> hit to searching/indexing throughput. I don't think that's the right
> tradeoff for Lucene.

But as Jake pointed out early in the thread, Zoie achieves those insanely fast
reopens without tight coupling to IndexWriter and its components. The
auxiliary RAM index approach is well proven.

> Do you have any hard numbers on how much time it takes Lucene to load
> from a hot IO cache, populating its RAM resident data structures?

Hmm, I don't spend a lot of time working with Lucene directly, so I might not
be the person most likely to have data like that at my fingertips. Maybe that
McCandless dude can help you out, he runs a lot of benchmarks. ;)

Or maybe ask the Solr folks? I see them on solr-user all the time talking
about "MaxWarmingSearchers". ;)

> OK. Then, you are basically pooling your readers Ie, you do allow
> in-process sharing, but only among readers.

Not sure about that. Lucy's IndexReader.reopen() would open new SegReaders for
each new segment, but they would be private to each parent PolyReader. So if
you reopened two IndexReaders at the same time after e.g. segment "seg_12"
had been added, each would create a new, private SegReader for "seg_12".

*Edit*: updated to correct assertions about virtual memory performance with
small indexes.

was (Author: creamyg):
> I guess my confusion is what are all the other benefits of using
> file-backed RAM? You can efficiently use process only concurrency
> (though shared memory is technically an option for this too), and you
> have wicked fast open times (but, you still must warm, just like
> Lucene).

Processes are Lucy's primary concurrency model. ("The OS is our JVM.")
Making process-only concurrency efficient isn't optional -- it's a *core*
*concern*.

> What else? Oh maybe the ability to inform OS not to cache
> eg the reads done when merging segments. That's one I sure wish
> Lucene could use...

Lightweight searchers mean architectural freedom.

Create 2, 10, 100, 1000 Searchers without a second thought -- as many as you
need for whatever app architecture you just dreamed up -- then destroy them
just as effortlessly. Add another worker thread to your search server without
having to consider the RAM requirements of a heavy searcher object. Create a
command-line app to search a documentation index without worrying about
daemonizing it. Etc.

If your normal development pattern is a single monolithic Java process, then
that freedom might not mean much to you. But with their low per-object RAM
requirements and fast opens, lightweight searchers are easy to use within a
lot of other development patterns. For example: lightweight searchers work
well for maxing out multiple CPU cores under process-only concurrency.

> In exchange you risk the OS making poor choices about what gets
> swapped out (LRU policy is too simplistic... not all pages are created
> equal),

The Linux virtual memory system, at least, is not a pure LRU. It utilizes a
page aging algo which prioritizes pages that have historically been accessed
frequently even when they have not been accessed recently:

{panel}
http://sunsite.nus.edu.sg/LDP/LDP/tlk/node40.html

The default action when a page is first allocated, is to give it an
initial age of 3. Each time it is touched (by the memory management
subsystem) it's age is increased by 3 to a maximum of 20. Each time the
Kernel swap daemon runs it ages pages, decrementing their age by 1.
{panel}

And while that system may not be ideal from our standpoint, it's still pretty
good. In general, the operating system's virtual memory scheme is going to
work fine as designed, for us and everyone else, and minimize memory
availability wait times.

When will swapping out the term dictionary be a problem?

* For indexes where queries are made frequently, no problem.
* Foir systems with plenty of RAM, no problem.
* For systems that aren't very busy, no problem.
* For small indexes, no problem.

The only situation we're talking about is infrequent queries against large
indexes on busy boxes where RAM isn't abundant. Under those circumstances, it
*might* be noticable that Lucy's term dictionary gets paged out somewhat
sooner than Lucene's.

But in general, if the term dictionary gets paged out, so what? Nobody was
using it. Maybe nobody will make another query against that index until next
week. Maybe the OS made the right decision.

OK, so there's a vulnerable bubble where the the query rate against a large
index is neither too fast nor too slow, on busy machines where RAM isn't
abundant. I don't think that bubble ought to drive major architectural
decisions.

Let me turn your question on its head. What does Lucene gain in return for
the slow index opens and large process memory footprint of its heavy
searchers?

> I do love how pure the file-backed RAM approach is, but I worry that
> down the road it'll result in erratic search performance in certain
> app profiles.

If necessary, there's a straightforward remedy: slurp the relevant files into
RAM at object construction rather than mmap them. The rest of the code won't
know the difference between malloc'd RAM and mmap'd RAM. The slurped files
won't take up any more space than the analogous Lucene data structures; more
likely, they'll take up less.

That's the kind of setting we'd hide away in the IndexManager class rather
than expose as prominent API, and it would be a hint to index components
rather than an edict.

> Yeah, that you need 3 files for the string sort cache is a little
> spooky... that's 3X the chance of a page fault.

Not when using the compound format.

> But the CFS construction must also go through the filesystem (like
> Lucene) right? So you still incur IO load of creating the small
> files, then 2nd pass to consolidate.

Yes.

> I think we may need to largely take "time" out of our programming
> languages, eg switch to much more declarative code, or
> something... wanna port Lucy to Erlang?
>
> But I'm not sure process only concurrency, sharing only via
> file-backed memory, is the answer either

I think relying heavily on file-backed memory is particularly appropriate for
Lucy because the write-once file format works well with MAP_SHARED memory
segments. If files were being modified and had to be protected with
semaphores, it wouldn't be as sweet a match.

Focusing on process-only concurrency also works well for Lucy because host
threading models differ substantially and so will only be accessible via a
generalized interface from the Lucy C core. It will be difficult to tune
threading performance through that layer of indirection -- I'm guessing beyond
the ability of most developers since few will be experts in multiple host
threading models. In contrast, expertise in process level concurrency will be
easier to come by and to nourish.

> Using Zoie you can make reopen time insanely fast (much faster than I
> think necessary for most apps), but at the expense of some expected
> hit to searching/indexing throughput. I don't think that's the right
> tradeoff for Lucene.

But as Jake pointed out early in the thread, Zoie achieves those insanely fast
reopens without tight coupling to IndexWriter and its components. The
auxiliary RAM index approach is well proven.

> Do you have any hard numbers on how much time it takes Lucene to load
> from a hot IO cache, populating its RAM resident data structures?

Hmm, I don't spend a lot of time working with Lucene directly, so I might not
be the person most likely to have data like that at my fingertips. Maybe that
McCandless dude can help you out, he runs a lot of benchmarks. ;)

Or maybe ask the Solr folks? I see them on solr-user all the time talking
about "MaxWarmingSearchers". ;)

> OK. Then, you are basically pooling your readers Ie, you do allow
> in-process sharing, but only among readers.

Not sure about that. Lucy's IndexReader.reopen() would open new SegReaders for
each new segment, but they would be private to each parent PolyReader. So if
you reopened two IndexReaders at the same time after e.g. segment "seg_12"
had been added, each would create a new, private SegReader for "seg_12".

[jira] Commented: (LUCENE-2120) Possible file handle leak in near real-time reader

2009-12-22T13:50:29Z

[ https://issues.apache.org/jira/browse/LUCENE-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793803#action_12793803 ]

John Wang commented on LUCENE-2120:
-----------------------------------

Yes we have done perf tests.
We see no indexing throughput improvement, query throughput improved by 40%.

> Possible file handle leak in near real-time reader
> --------------------------------------------------
>
> Key: LUCENE-2120
> URL: https://issues.apache.org/jira/browse/LUCENE-2120
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 3.1
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 3.1
>
>
> Spinoff of LUCENE-1526: Jake/John hit file descriptor exhaustion when testing NRT.
> I've tried to repro this, stress testing NRT, saturating reopens, indexing, searching, but haven't found any issue.
> Let's try to get to the bottom of it, here...

[jira] Commented: (LUCENE-2120) Possible file handle leak in near real-time reader

2009-12-22T12:36:29Z

[ https://issues.apache.org/jira/browse/LUCENE-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793775#action_12793775 ]

Michael McCandless commented on LUCENE-2120:
--------------------------------------------

I briefly diff'd the BR_DELETE_OPT branch in Zoie -- it looks like you switched (via custom TermDocs/Positions) to enforcing deleted docs using an iterator instead of double-check random-access, right? I had tried the same thing with Lucene (a while back now), under LUCENE-1476/LUCENE-1536, and found that the simple BitVector gave much better performance than an iterator (which then led to applying filters random-access as well). Have you tested performance of this switch? Maybe it works out to be faster, net/net, than doing the double-deletions check?

[jira] Commented: (LUCENE-2026) Refactoring of IndexWriter

2009-12-22T11:10:29Z

[ https://issues.apache.org/jira/browse/LUCENE-2026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793737#action_12793737 ]

Michael McCandless commented on LUCENE-2026:
--------------------------------------------

{quote}
Processes are Lucy's primary concurrency model. ("The OS is our JVM.")
Making process-only concurrency efficient isn't optional - it's a core
concern.
{quote}

OK

{quote}
Lightweight searchers mean architectural freedom.

Create 2, 10, 100, 1000 Searchers without a second thought - as many as you
need for whatever app architecture you just dreamed up - then destroy them
just as effortlessly. Add another worker thread to your search server without
having to consider the RAM requirements of a heavy searcher object. Create a
command-line app to search a documentation index without worrying about
daemonizing it. Etc.
{quote}

This is definitely neat.

{quote}
The Linux virtual memory system, at least, is not a pure LRU. It utilizes a
page aging algo which prioritizes pages that have historically been accessed
frequently even when they have not been accessed recently:

http://sunsite.nus.edu.sg/LDP/LDP/tlk/node40.html
{quote}

Very interesting -- thanks. So it also factors in how much the page
was used in the past, not just how long it's been since the page was
last used.

{quote}
When will swapping out the term dictionary be a problem?

For indexes where queries are made frequently, no problem.
Foir systems with plenty of RAM, no problem.
For systems that aren't very busy, no problem.
For small indexes, no problem.
The only situation we're talking about is infrequent queries against large
indexes on busy boxes where RAM isn't abundant. Under those circumstances, it
might be noticable that Lucy's term dictionary gets paged out somewhat
sooner than Lucene's.
{quote}

Even smallish indexes can see the pages swapped out? I'd think at
low-to-moderate search traffic, any index could be at risk, depdending
on whether other stuff in the machine wanting RAM or IO cache is
running.

{quote}
But in general, if the term dictionary gets paged out, so what? Nobody was
using it. Maybe nobody will make another query against that index until next
week. Maybe the OS made the right decision.
{quote}

You can't afford many page faults until the latency becomes very
apparent (until we're all on SSDs... at which point this may all be
moot).

Right -- the metric that the swapper optimizes is overall efficient
use of the machine's resources.

But I think that's often a poor metric for search apps... I think
consistency on the search latency is more important, though I agree it
depends very much on the app.

I don't like the same behavior in my desktop -- when I switch to my
mail client, I don't want to wait 10 seconds for it to swap the pages
back in.

{quote}
Let me turn your question on its head. What does Lucene gain in return for
the slow index opens and large process memory footprint of its heavy
searchers?
{quote}

Consistency in the search time. Assuming the OS doesn't swap our
pages out...

And of course Java pretty much forces threads-as-concurrency (JVM
startup time, hotspot compilation, are costly).

{quote}
If necessary, there's a straightforward remedy: slurp the relevant files into
RAM at object construction rather than mmap them. The rest of the code won't
know the difference between malloc'd RAM and mmap'd RAM. The slurped files
won't take up any more space than the analogous Lucene data structures; more
likely, they'll take up less.

That's the kind of setting we'd hide away in the IndexManager class rather
than expose as prominent API, and it would be a hint to index components
rather than an edict.
{quote}

Right, this is how Lucy would force warming.

{quote}
bq. Yeah, that you need 3 files for the string sort cache is a little spooky... that's 3X the chance of a page fault.

Not when using the compound format.
{quote}

But, even within that CFS file, these three sub-files will not be
local? Ie you'll still have to hit three pages per "lookup" right?

{quote}
I think relying heavily on file-backed memory is particularly appropriate for
Lucy because the write-once file format works well with MAP_SHARED memory
segments. If files were being modified and had to be protected with
semaphores, it wouldn't be as sweet a match.
{quote}

Write-once is good for Lucene too.

{quote}
Focusing on process-only concurrency also works well for Lucy because host
threading models differ substantially and so will only be accessible via a
generalized interface from the Lucy C core. It will be difficult to tune
threading performance through that layer of indirection - I'm guessing beyond
the ability of most developers since few will be experts in multiple host
threading models. In contrast, expertise in process level concurrency will be
easier to come by and to nourish.
{quote}

I'm confused by this -- eg Python does a great job presenting a simple
threads interface and implementing it on major OSs. And it seems like
Lucy would not need anything crazy-os-specific wrt threads?

{quote}
bq. Do you have any hard numbers on how much time it takes Lucene to load from a hot IO cache, populating its RAM resident data structures?

Hmm, I don't spend a lot of time working with Lucene directly, so I might not
be the person most likely to have data like that at my fingertips. Maybe that
McCandless dude can help you out, he runs a lot of benchmarks.
{quote}

Hmm ;) I'd guess that field cache is slowish; deleted docs & norms are
very fast; terms index is somewhere in between.

bq. Or maybe ask the Solr folks? I see them on solr-user all the time talking about "MaxWarmingSearchers".

Hmm -- not sure what's up with that. Looks like maybe it's the
auto-warming that might happen after a commit.

{quote}
bq. OK. Then, you are basically pooling your readers Ie, you do allow in-process sharing, but only among readers.

Not sure about that. Lucy's IndexReader.reopen() would open new SegReaders for
each new segment, but they would be private to each parent PolyReader. So if
you reopened two IndexReaders at the same time after e.g. segment "seg_12"
had been added, each would create a new, private SegReader for "seg_12".
{quote}

You're right, you'd get two readers for seg_12 in that case. By
"pool" I meant you're tapping into all the sub-readers that the
existing reader have opened -- the reader is your pool of sub-readers.

Re: 3.0 api change

2009-12-22T10:09:53Z

Thanks Uwe for clearing this up!

-John

On Mon, Dec 21, 2009 at 11:22 PM, Uwe Schindler <uwe@...> wrote:

This method was accidentially removed because it had no javadocs and was “somehow invisible”. This is the second one found, that was accidentially removed (the other was IR.getTermIndexDivisor).

As JIRA is not working a the moment, I will add a comment to the Sort deprecations removal issue (LUCENE-1972) later, but it is fixed now in 3.0 branch revision 893095 (and also trunk)

Thanks!

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@...

From: John Wang [mailto:john.wang@...]
Sent: Tuesday, December 22, 2009 3:16 AM
To: java-user@...; java-dev@...
Subject: Fwd: 3.0 api change

Any comments?

Did we just unintentionally remove getFieldComparatorSource in 3.0.0?

-John

---------- Forwarded message ----------
From: John Wang <john.wang@...>
Date: Mon, Dec 21, 2009 at 11:21 AM
Subject: 3.0 api change
To: Lucene Users List <lucene-user@...>, lucene-dev@...

Hi guys:

I noticed SortField.getComparatorSource was removed (replaced by getComparator) and it is not documented in CHANGES.TXT. This api was introduced in 2.9.0.

Is this intentional?

Thanks

-John

[jira] Resolved: (LUCENE-2178) Benchmark contrib should allow multiple locations in ext.classpath

2009-12-22T09:08:30Z

[ https://issues.apache.org/jira/browse/LUCENE-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael McCandless resolved LUCENE-2178.
----------------------------------------

Resolution: Fixed
Fix Version/s: 3.1

> Benchmark contrib should allow multiple locations in ext.classpath
> ------------------------------------------------------------------
>
> Key: LUCENE-2178
> URL: https://issues.apache.org/jira/browse/LUCENE-2178
> Project: Lucene - Java
> Issue Type: Improvement
> Components: contrib/benchmark
> Affects Versions: 3.0
> Reporter: Steven Rowe
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 3.1
>
>
> When {{ant run-task}} is invoked with the {{-Dbenchmark.ext.classpath=...}} option, only a single location may be specified. If a classpath with more than one location is specified, none of the locations is put on the classpath for the invoked JVM.

[jira] Assigned: (LUCENE-2178) Benchmark contrib should allow multiple locations in ext.classpath

2009-12-22T09:04:29Z

[ https://issues.apache.org/jira/browse/LUCENE-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael McCandless reassigned LUCENE-2178:
------------------------------------------

Assignee: Michael McCandless

> Benchmark contrib should allow multiple locations in ext.classpath
> ------------------------------------------------------------------
>
> Key: LUCENE-2178
> URL: https://issues.apache.org/jira/browse/LUCENE-2178
> Project: Lucene - Java
> Issue Type: Improvement
> Components: contrib/benchmark
> Affects Versions: 3.0
> Reporter: Steven Rowe
> Assignee: Michael McCandless
> Priority: Minor
>
> When {{ant run-task}} is invoked with the {{-Dbenchmark.ext.classpath=...}} option, only a single location may be specified. If a classpath with more than one location is specified, none of the locations is put on the classpath for the invoked JVM.

[jira] Commented: (LUCENE-2178) Benchmark contrib should allow multiple locations in ext.classpath

2009-12-22T09:04:29Z

[ https://issues.apache.org/jira/browse/LUCENE-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793673#action_12793673 ]

Michael McCandless commented on LUCENE-2178:
--------------------------------------------

Looks good -- I'll commit. Thanks Steven!

> Benchmark contrib should allow multiple locations in ext.classpath
> ------------------------------------------------------------------
>
> Key: LUCENE-2178
> URL: https://issues.apache.org/jira/browse/LUCENE-2178
> Project: Lucene - Java
> Issue Type: Improvement
> Components: contrib/benchmark
> Affects Versions: 3.0
> Reporter: Steven Rowe
> Assignee: Michael McCandless
> Priority: Minor
>
> When {{ant run-task}} is invoked with the {{-Dbenchmark.ext.classpath=...}} option, only a single location may be specified. If a classpath with more than one location is specified, none of the locations is put on the classpath for the invoked JVM.

[jira] Commented: (LUCENE-2178) Benchmark contrib should allow multiple locations in ext.classpath

2009-12-22T06:30:29Z

[ https://issues.apache.org/jira/browse/LUCENE-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793627#action_12793627 ]

Steven Rowe commented on LUCENE-2178:
-------------------------------------

Trivial patch to fix (works with single or multiple locations):

{code}
Index: contrib/benchmark/build.xml
===================================================================
--- contrib/benchmark/build.xml (revision 892657)
+++ contrib/benchmark/build.xml (working copy)
@@ -114,7 +114,7 @@
<path id="run.classpath">
<path refid="classpath"/>
<pathelement location="${build.dir}/classes/java"/>
- <pathelement location="${benchmark.ext.classpath}"/>
+ <pathelement path="${benchmark.ext.classpath}"/>
</path>

<property name="task.alg" location="conf/micro-standard.alg"/>
{code}

> Benchmark contrib should allow multiple locations in ext.classpath
> ------------------------------------------------------------------
>
> Key: LUCENE-2178
> URL: https://issues.apache.org/jira/browse/LUCENE-2178
> Project: Lucene - Java
> Issue Type: Improvement
> Components: contrib/benchmark
> Affects Versions: 3.0
> Reporter: Steven Rowe
> Priority: Minor
>
> When {{ant run-task}} is invoked with the {{-Dbenchmark.ext.classpath=...}} option, only a single location may be specified. If a classpath with more than one location is specified, none of the locations is put on the classpath for the invoked JVM.

[jira] Updated: (LUCENE-2178) Benchmark contrib should allow multiple locations in ext.classpath

2009-12-22T06:28:29Z

[ https://issues.apache.org/jira/browse/LUCENE-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Steven Rowe updated LUCENE-2178:
--------------------------------

Description: When {{ant run-task}} is invoked with the {{-Dbenchmark.ext.classpath=...}} option, only a single location may be specified. If a classpath with more than one location is specified, none of the locations is put on the classpath for the invoked JVM. (was: When {{ant run-task}} is invoked with the {{-Dbenchmark.ext.classpath=...} option, only a single location may be specified. If a classpath with more than one location is specified, none of the locations is put on the classpath for the invoked JVM.)

> Benchmark contrib should allow multiple locations in ext.classpath
> ------------------------------------------------------------------
>
> Key: LUCENE-2178
> URL: https://issues.apache.org/jira/browse/LUCENE-2178
> Project: Lucene - Java
> Issue Type: Improvement
> Components: contrib/benchmark
> Affects Versions: 3.0
> Reporter: Steven Rowe
> Priority: Minor
>
> When {{ant run-task}} is invoked with the {{-Dbenchmark.ext.classpath=...}} option, only a single location may be specified. If a classpath with more than one location is specified, none of the locations is put on the classpath for the invoked JVM.

[jira] Resolved: (LUCENE-2177) The Field ctors that take byte[] shouldn't take Store, since it must be YES

2009-12-22T06:26:29Z

[ https://issues.apache.org/jira/browse/LUCENE-2177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael McCandless resolved LUCENE-2177.
----------------------------------------

Resolution: Fixed

> The Field ctors that take byte[] shouldn't take Store, since it must be YES
> ---------------------------------------------------------------------------
>
> Key: LUCENE-2177
> URL: https://issues.apache.org/jira/browse/LUCENE-2177
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Other
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Trivial
> Fix For: 3.1
>
> Attachments: LUCENE-2177.patch
>
>
> API silliness. Makes you think you can set Store.NO for binary fields. This used to be meaningful when we also accepted COMPRESS, but now it's an orphan.

[jira] Created: (LUCENE-2178) Benchmark contrib should allow multiple locations in ext.classpath

2009-12-22T06:26:29Z

Benchmark contrib should allow multiple locations in ext.classpath
------------------------------------------------------------------

Key: LUCENE-2178
URL: https://issues.apache.org/jira/browse/LUCENE-2178
Project: Lucene - Java
Issue Type: Improvement
Components: contrib/benchmark
Affects Versions: 3.0
Reporter: Steven Rowe
Priority: Minor

When {{ant run-task}} is invoked with the {{-Dbenchmark.ext.classpath=...} option, only a single location may be specified. If a classpath with more than one location is specified, none of the locations is put on the classpath for the invoked JVM.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

[jira] Updated: (LUCENE-1972) Remove (deprecated) ExtendedFieldCache and Auto/Custom caches and lot's of deprecated sort logic

2009-12-22T00:24:29Z

[ https://issues.apache.org/jira/browse/LUCENE-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uwe Schindler updated LUCENE-1972:
----------------------------------

Attachment: LUCENE-1972-fix.patch

Attached is the patch, committed in 3.0 branch and trunk (rev 893104) that fixes the accidental remove of SortField.getComparatorSource().

Thanks John Wang!

> Remove (deprecated) ExtendedFieldCache and Auto/Custom caches and lot's of deprecated sort logic
> ------------------------------------------------------------------------------------------------
>
> Key: LUCENE-1972
> URL: https://issues.apache.org/jira/browse/LUCENE-1972
> Project: Lucene - Java
> Issue Type: Task
> Components: Search
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 3.0
>
> Attachments: LUCENE-1972-2.patch, LUCENE-1972-bw.patch, LUCENE-1972-fix.patch, LUCENE-1972.patch
>
>
> Remove (deprecated) ExtendedFieldCache and Auto/Custom caches and sort

RE: 3.0 api change

2009-12-21T23:22:43Z

This method was accidentially removed because it had no javadocs and was “somehow invisible”. This is the second one found, that was accidentially removed (the other was IR.getTermIndexDivisor).

As JIRA is not working a the moment, I will add a comment to the Sort deprecations removal issue (LUCENE-1972) later, but it is fixed now in 3.0 branch revision 893095 (and also trunk)

Thanks!

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@...

From: John Wang [mailto:john.wang@...]
Sent: Tuesday, December 22, 2009 3:16 AM
To: java-user@...; java-dev@...
Subject: Fwd: 3.0 api change

Any comments?

Did we just unintentionally remove getFieldComparatorSource in 3.0.0?

-John

---------- Forwarded message ----------
From: John Wang <john.wang@...>
Date: Mon, Dec 21, 2009 at 11:21 AM
Subject: 3.0 api change
To: Lucene Users List <lucene-user@...>, lucene-dev@...

Hi guys:

I noticed SortField.getComparatorSource was removed (replaced by getComparator) and it is not documented in CHANGES.TXT. This api was introduced in 2.9.0.

Is this intentional?

Thanks

-John

Fwd: 3.0 api change

2009-12-21T18:15:54Z

Any comments?

Did we just unintentionally remove getFieldComparatorSource in 3.0.0?

-John

I noticed SortField.getComparatorSource was removed (replaced by getComparator) and it is not documented in CHANGES.TXT. This api was introduced in 2.9.0.

Is this intentional?

Thanks

-John

[jira] Commented: (LUCENE-2026) Refactoring of IndexWriter

2009-12-21T16:09:18Z

[ https://issues.apache.org/jira/browse/LUCENE-2026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793431#action_12793431 ]

Marvin Humphrey commented on LUCENE-2026:
-----------------------------------------

> I guess my confusion is what are all the other benefits of using
> file-backed RAM? You can efficiently use process only concurrency
> (though shared memory is technically an option for this too), and you
> have wicked fast open times (but, you still must warm, just like
> Lucene).

Processes are Lucy's primary concurrency model. ("The OS is our JVM.")
Making process-only concurrency efficient isn't optional -- it's a *core*
*concern*.

> What else? Oh maybe the ability to inform OS not to cache
> eg the reads done when merging segments. That's one I sure wish
> Lucene could use...

Lightweight searchers mean architectural freedom.

Create 2, 10, 100, 1000 Searchers without a second thought -- as many as you
need for whatever app architecture you just dreamed up -- then destroy them
just as effortlessly. Add another worker thread to your search server without
having to consider the RAM requirements of a heavy searcher object. Create a
command-line app to search a documentation index without worrying about
daemonizing it. Etc.

If your normal development pattern is a single monolithic Java process, then
that freedom might not mean much to you. But with their low per-object RAM
requirements and fast opens, lightweight searchers are easy to use within a
lot of other development patterns. For example: lightweight searchers work
well for maxing out multiple CPU cores under process-only concurrency.

> In exchange you risk the OS making poor choices about what gets
> swapped out (LRU policy is too simplistic... not all pages are created
> equal),

The Linux virtual memory system, at least, is not a pure LRU. It utilizes a
page aging algo which prioritizes pages that have historically been accessed
frequently even when they have not been accessed recently:

{panel}
http://sunsite.nus.edu.sg/LDP/LDP/tlk/node40.html

The default action when a page is first allocated, is to give it an
initial age of 3. Each time it is touched (by the memory management
subsystem) it's age is increased by 3 to a maximum of 20. Each time the
Kernel swap daemon runs it ages pages, decrementing their age by 1.
{panel}

And while that system may not be ideal from our standpoint, it's still pretty
good. In general, the operating system's virtual memory scheme is going to
work fine as designed, for us and everyone else, and minimize memory
availability wait times.

When will swapping out the term dictionary be a problem?

* For indexes where queries are made frequently, no problem.
* Foir systems with plenty of RAM, no problem.
* For systems that aren't very busy, no problem.
* For small indexes, no problem.

The only situation we're talking about is infrequent queries against large
indexes on busy boxes where RAM isn't abundant. Under those circumstances, it
*might* be noticable that Lucy's term dictionary gets paged out somewhat
sooner than Lucene's.

But in general, if the term dictionary gets paged out, so what? Nobody was
using it. Maybe nobody will make another query against that index until next
week. Maybe the OS made the right decision.

OK, so there's a vulnerable bubble where the the query rate against a large
index is neither too fast nor too slow, on busy machines where RAM isn't
abundant. I don't think that bubble ought to drive major architectural
decisions.

Let me turn your question on its head. What does Lucene gain in return for
the slow index opens and large process memory footprint of its heavy
searchers?

> I do love how pure the file-backed RAM approach is, but I worry that
> down the road it'll result in erratic search performance in certain
> app profiles.

If necessary, there's a straightforward remedy: slurp the relevant files into
RAM at object construction rather than mmap them. The rest of the code won't
know the difference between malloc'd RAM and mmap'd RAM. The slurped files
won't take up any more space than the analogous Lucene data structures; more
likely, they'll take up less.

That's the kind of setting we'd hide away in the IndexManager class rather
than expose as prominent API, and it would be a hint to index components
rather than an edict.

> Yeah, that you need 3 files for the string sort cache is a little
> spooky... that's 3X the chance of a page fault.

Not when using the compound format.

> But the CFS construction must also go through the filesystem (like
> Lucene) right? So you still incur IO load of creating the small
> files, then 2nd pass to consolidate.

Yes.

> I think we may need to largely take "time" out of our programming
> languages, eg switch to much more declarative code, or
> something... wanna port Lucy to Erlang?
>
> But I'm not sure process only concurrency, sharing only via
> file-backed memory, is the answer either

I think relying heavily on file-backed memory is particularly appropriate for
Lucy because the write-once file format works well with MAP_SHARED memory
segments. If files were being modified and had to be protected with
semaphores, it wouldn't be as sweet a match.

Focusing on process-only concurrency also works well for Lucy because host
threading models differ substantially and so will only be accessible via a
generalized interface from the Lucy C core. It will be difficult to tune
threading performance through that layer of indirection -- I'm guessing beyond
the ability of most developers since few will be experts in multiple host
threading models. In contrast, expertise in process level concurrency will be
easier to come by and to nourish.

> Using Zoie you can make reopen time insanely fast (much faster than I
> think necessary for most apps), but at the expense of some expected
> hit to searching/indexing throughput. I don't think that's the right
> tradeoff for Lucene.

But as Jake pointed out early in the thread, Zoie achieves those insanely fast
reopens without tight coupling to IndexWriter and its components. The
auxiliary RAM index approach is well proven.

> Do you have any hard numbers on how much time it takes Lucene to load
> from a hot IO cache, populating its RAM resident data structures?

Hmm, I don't spend a lot of time working with Lucene directly, so I might not
be the person most likely to have data like that at my fingertips. Maybe that
McCandless dude can help you out, he runs a lot of benchmarks. ;)

Or maybe ask the Solr folks? I see them on solr-user all the time talking
about "MaxWarmingSearchers". ;)

> OK. Then, you are basically pooling your readers Ie, you do allow
> in-process sharing, but only among readers.

Not sure about that. Lucy's IndexReader.reopen() would open new SegReaders for
each new segment, but they would be private to each parent PolyReader. So if
you reopened two IndexReaders at the same time after e.g. segment "seg_12"
had been added, each would create a new, private SegReader for "seg_12".