Nabble - Solr

Re: Solr CPU usage

2009-11-27T09:09:49Z

Yonik,
Am running both my server and client on ubuntu machines. The client is on a
different box. The server CPU and RAM are both well below 50%.

Girish Redekar
http://girishredekar.net

On Fri, Nov 27, 2009 at 10:07 PM, Yonik Seeley
<yonik@...>wrote:

> On Fri, Nov 27, 2009 at 9:30 AM, Girish Redekar
> <girish.redekar@...> wrote:
> > Hi
> >
> > I'm testing my Solr instance with multiple simultaneous requests. Here's
> my
> > test.
> >
> > For an index of ~200K docs, I query Solr with 10 simultaneous threads.
> Can
> > someone help me explain/improve the following observations:
> >
> > 1) Solr doesn't seem to use all the available CPU to improve response
> times
> > (query times are good, but the time required to return documents aren't
> so
> > good). My CPU seems to be running at ~30%
> > 2) As expected, time for response increases as the num of requested
> results
> > increase. What's surprising (and perplexing) is that Solr seems to use
> > *more* of the CPU when I'm requesting *fewer* docs. Consequently, its
> > performance in returning a larger result set is very bad
>
> This may point to another bottleneck - if the OS is low on free RAM,
> it could be disk IO.
> If this is on Windows, you could have contention reading the index files.
> Otherwise, you may have a bottleneck in network IO. Is the client you
> are testing with on the same box?
> Is this Solr 1.4?
>
> -Yonik
> http://www.lucidimagination.com
>

From forum: Solr - User

Re: Links to in-dev features on wiki?

2009-11-27T09:06:18Z

: Pages such as ZooKeeperIntegration and KattaIntegration used to be linked
: off the Wiki front page. Now I can't find them any more. Where have those
: links been moved to?

They're both linked to from "IntegratingSolr" (and "DistributedSearch")

FYI: for any page, you can get a list of all the in-bound wiki links by
clicking on the page title (above the navigation) which takes you to links
like these...

http://wiki.apache.org/solr/KattaIntegration?action=fullsearch&context=180&value=linkto%3A%22KattaIntegration%22
http://wiki.apache.org/solr/ZooKeeperIntegration?action=fullsearch&context=180&value=linkto%3A%22ZooKeeperIntegration%22

-Hoss

From forum: Solr - Dev

Re: Solr CPU usage

2009-11-27T08:37:33Z

On Fri, Nov 27, 2009 at 9:30 AM, Girish Redekar
<girish.redekar@...> wrote:

> Hi
>
> I'm testing my Solr instance with multiple simultaneous requests. Here's my
> test.
>
> For an index of ~200K docs, I query Solr with 10 simultaneous threads. Can
> someone help me explain/improve the following observations:
>
> 1) Solr doesn't seem to use all the available CPU to improve response times
> (query times are good, but the time required to return documents aren't so
> good). My CPU seems to be running at ~30%
> 2) As expected, time for response increases as the num of requested results
> increase. What's surprising (and perplexing) is that Solr seems to use
> *more* of the CPU when I'm requesting *fewer* docs. Consequently, its
> performance in returning a larger result set is very bad

This may point to another bottleneck - if the OS is low on free RAM,
it could be disk IO.
If this is on Windows, you could have contention reading the index files.
Otherwise, you may have a bottleneck in network IO. Is the client you
are testing with on the same box?
Is this Solr 1.4?

-Yonik
http://www.lucidimagination.com

From forum: Solr - User

Solr CPU usage

2009-11-27T06:30:39Z

Hi

I'm testing my Solr instance with multiple simultaneous requests. Here's my
test.

For an index of ~200K docs, I query Solr with 10 simultaneous threads. Can
someone help me explain/improve the following observations:

1) Solr doesn't seem to use all the available CPU to improve response times
(query times are good, but the time required to return documents aren't so
good). My CPU seems to be running at ~30%
2) As expected, time for response increases as the num of requested results
increase. What's surprising (and perplexing) is that Solr seems to use
*more* of the CPU when I'm requesting *fewer* docs. Consequently, its
performance in returning a larger result set is very bad
3) To counter 1, is there a way to make two Solr instances search on the
same index (so that concurrent requests are served faster)

Any help in this regard would be very useful.

Thanks !
Girish Redekar
http://girishredekar.net

From forum: Solr - User

What does this error mean?

2009-11-27T05:46:53Z

NFO: start commit(optimize=false,waitFlush=true,waitSearcher=true,expungeDeletes=false)
Nov 27, 2009 3:45:35 AM
org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: {} 0 634
Nov 27, 2009 3:45:35 AM org.apache.solr.core.SolrCore getSearcher
WARNING: [nutch] Error opening new searcher. exceeded limit of
maxWarmingSearchers=2, try again later.
Nov 27, 2009 3:45:35 AM
org.apache.solr.update.processor.LogUpdateProcessor finishINFO: {} 0
635
Nov 27, 2009 3:45:35 AM org.apache.solr.common.SolrException log
SEVERE: org.apache.solr.common.SolrException: Error opening new
searcher. exceeded limit of maxWarmingSear
chers=2, try again later.
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1029)
at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:418)
at org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.jav
a:85)
at org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:107)
at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:48)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454)
at java.lang.Thread.run(Thread.java:619)

Nov 27, 2009 3:45:35 AM org.apache.solr.core.SolrCore execute
INFO: [nutch] webapp=/solrChunk path=/update
params={waitSearcher=true&commit=true&wt=javabin&waitFlush=true&version=1}
status=503 QTime=634
Nov 27, 2009 3:45:35 AM org.apache.solr.common.SolrException log
SEVERE: org.apache.solr.common.SolrException: Error opening new
searcher. exceeded limit of maxWarmingSearchers=2, try again later.
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1029)
at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:418)
at org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:85)
at org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:107)
at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:48)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454)
at java.lang.Thread.run(Thread.java:619)

Nov 27, 2009 3:45:35 AM org.apache.solr.core.SolrCore execute
INFO: [nutch] webapp=/solrChunk path=/update
params={waitSearcher=true&commit=true&wt=javabin&waitFlush=true&version=1}
status=503 QTime=635

--
http://www.linkedin.com/in/paultomblin
http://careers.stackoverflow.com/ptomblin

From forum: Solr - User

restore space between words by spell checker

2009-11-27T03:09:08Z

Hi

If a user issued a misspelled query, forgetting to place space between
words, is it possible to fix it with a spell checker or by some other
mechanism?

For example, if we get query "tommyhitfiger" and have terms "tommy" and
"hitfiger" in the index, how to fix the query?

--
Andrew Klochkov
Senior Software Engineer,
Grid Dynamics

From forum: Solr - User

Re: Retrieving large num of docs

2009-11-27T01:46:38Z

> Hi Andrew,
> We are running solr using its http interface from python.
> From the resources
> I could find, EmbeddedSolrServer is possible only if I am
> using solr from a
> java program. It will be useful to understand if a
> significant part of the
> performance increase is due to bypassing HTTP before going
> down this path.
>
> In the mean time I am trying my luck with the other
> suggestions. Can you
> share the patch that helps cache solr documents instead of
> lucene documents?

May be these links can help
http://wiki.apache.org/lucene-java/ImproveSearchingSpeed
http://wiki.apache.org/lucene-java/ImproveIndexingSpeed
http://www.lucidimagination.com/Downloads/LucidGaze-for-Solr

how often do you update your index?
is your index optimized?
configuring caching can also help:

http://wiki.apache.org/solr/SolrCaching
http://wiki.apache.org/solr/SolrPerformanceFactors

From forum: Solr - User

Re: Retrieving large num of docs

2009-11-27T01:02:09Z

Hi Andrew,
We are running solr using its http interface from python. From the resources
I could find, EmbeddedSolrServer is possible only if I am using solr from a
java program. It will be useful to understand if a significant part of the
performance increase is due to bypassing HTTP before going down this path.

In the mean time I am trying my luck with the other suggestions. Can you
share the patch that helps cache solr documents instead of lucene documents?

On a different note, I am wondering why does it take 4 - 5 seconds for Solr
to return the ID's of ranked documents when it can rank the results in about
20 milli seconds? Am I missing something here?

Thanks,
Raghu

On Fri, Nov 27, 2009 at 2:15 AM, Andrey Klochkov <aklochkov@...
> wrote:

> Hi
>
> We obtain ALL documents for every query, the index size is about 50k. We
> use
> number of stored fields. Often the result set size is several thousands of
> docs.
>
> We performed the following things to make it faster:
>
> 1. Use EmbeddedSolrServer
> 2. Patch Solr to avoid unnecessary marshalling while using
> EmbeddedSolrServer (there's an issue in Solr JIRA)
> 3. Patch Solr to cache SolrDocument instances instead of Lucene's Document
> instances. I was going to share this patch, but then decided that our usage
> of Solr is not common and this functionality is useless in most cases
> 4. We have all documents in cache
> 5. In fact our index is stored in a data grid, not a file system. But as
> tests showed this is not important because standard FSDirectory is faster
> if
> you have enough of RAM free for OS caches.
>
> These changes improved the performance very much, so in the end we have
> performance comparable (about 3-5 times slower) to the "proper" Solr usage
> (obtaining first 20 documents).
>
> To get more details on how different Solr components perform we injected
> perf4j statements into key points in the code. And a profiler was helpful
> too.
>
> Hope it helps somehow.
>
> On Thu, Nov 26, 2009 at 8:48 PM, Raghuveer Kancherla <
> raghuveer.kancherla@...> wrote:
>
> > Hi,
> > I am using Solr1.4 for searching through half a million documents. The
> > problem is, I want to retrieve nearly 200 documents for each search
> query.
> > The query time in Solr logs is showing 0.02 seconds and I am fairly happy
> > with that. However Solr is taking a long time (4 to 5 secs) to return the
> > results (I think it is because of the number of docs I am requesting). I
> > tried returning only the id's (unique key) without any other stored
> fields,
> > but it is not helping me improve the response times (time to return the
> > id's
> > of matching documents).
> > I understand that retrieving 200 documents for each search term is
> > impractical in most scenarios but I dont have any other option. Any
> > pointers
> > on how to improve the response times will be a great help.
> >
> > Thanks,
> > Raghu
> >
>
>
>
> --
> Andrew Klochkov
> Senior Software Engineer,
> Grid Dynamics
>

From forum: Solr - User

[jira] Updated: (SOLR-1537) Dedupe Sharded Search Results by Shard Order or Score

2009-11-26T20:37:40Z

[ https://issues.apache.org/jira/browse/SOLR-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dennis Kubes updated SOLR-1537:
-------------------------------

Attachment: SOLR-1537-20091126-4.patch

Final patch. This incorporates an updated version of SOLR-1143, allowing the return of partial search results. This patch fixes bugs in the number of results returned, sorting order, errors on edge conditions, among others. This patch also supercedes SOLR-1143 bringing all unit tests up to date and adding enhanced functionality to allow returning partial results when servers names are mispelled or there are other errors besides simple connection errors. Headers have been added to show the number of shards failing and the names of those shards. Unit test have been added to demonstrate dedup of search results by shard order. This patch passes all current unit tests.

> Dedupe Sharded Search Results by Shard Order or Score
> -----------------------------------------------------
>
> Key: SOLR-1537
> URL: https://issues.apache.org/jira/browse/SOLR-1537
> Project: Solr
> Issue Type: Improvement
> Components: search
> Affects Versions: 1.4, 1.5
> Environment: All
> Reporter: Dennis Kubes
> Fix For: 1.5
>
> Attachments: SOLR-1537-20091126-4.patch, solr-dedupe-20091031-2.patch, solr-dedupe-20091031.patch, solr-dedupe-20091106-3.patch
>
>
> Allows sharded search results to dedupe results by ID based on either the order of the shards in the shards param or by score. Allows the result returned to be deterministic. If by shards then shards that appear first in the shards param have a higher precedence than shards that appear later. If by score then higher scores beat out lower scores. This doesn't allow multiple duplicates because currently SOLR only permits a single result by ID to be returned.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

From forum: Solr - Dev

Re: WordDelimiterFilter and acronyms normalization

2009-11-26T12:50:49Z

> Is there any ready-for-use filter which performs acronyms
> normalization such
> as "I.N.C."->"INC"?
>
> I see that Lucene's StandardFilter can do this but we can't
> use it as we're
> using WhitespaceTokenizer instead of StandardTokenizer.
>

I am bad at regular expressions but if you can write a regex for that replacement solr.PatternReplaceFilterFactory can do that.

<filter class="solr.PatternReplaceFilterFactory" pattern="([^a-z])" replacement="" replace="all" />

From forum: Solr - User

Re: Re: Sending Tika parse result to Solr

2009-11-26T12:49:37Z

Hello,

i want to send the Tika parse results of my data to my Solr-Server.

My File-Server is not my Solr-Server, so Solr Cell is no option for me.

In Lucene i can pass my Reader Object (as an result of the parsing) to a Lucene Document for indexing.

Is this also possible with Solr? Or is there an other or better way to do this?

I'm using SolrJ for the connection.

You can't pass your reader object, but I have opened https://issues.apache.org/jira/browse/SOLR-1526 to provide a SolrJ client side equivalent of Solr Cell. If you'd like to contribute a patch that would be great. Basically, you just need to have your Handler override create a SolrInputDocument (batches, that is) and then send them to Solr.

Is there any documentation how to do that? I'm new to Solr and don't exactly understand what you mean with that. Some detailed informations would be great.

Thank you in anticipation!

Using the Streaming server may also fit well with this model.

--------------------------

Grant Ingersoll

http://www.lucidimagination.com/

Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids) using Solr/Lucene:

http://www.lucidimagination.com/search

smime.p7s (6K) Download Attachment

From forum: Solr - User

Re: Retrieving large num of docs

2009-11-26T12:45:05Z

Hi

We obtain ALL documents for every query, the index size is about 50k. We use
number of stored fields. Often the result set size is several thousands of
docs.

We performed the following things to make it faster:

1. Use EmbeddedSolrServer
2. Patch Solr to avoid unnecessary marshalling while using
EmbeddedSolrServer (there's an issue in Solr JIRA)
3. Patch Solr to cache SolrDocument instances instead of Lucene's Document
instances. I was going to share this patch, but then decided that our usage
of Solr is not common and this functionality is useless in most cases
4. We have all documents in cache
5. In fact our index is stored in a data grid, not a file system. But as
tests showed this is not important because standard FSDirectory is faster if
you have enough of RAM free for OS caches.

These changes improved the performance very much, so in the end we have
performance comparable (about 3-5 times slower) to the "proper" Solr usage
(obtaining first 20 documents).

To get more details on how different Solr components perform we injected
perf4j statements into key points in the code. And a profiler was helpful
too.

Hope it helps somehow.

On Thu, Nov 26, 2009 at 8:48 PM, Raghuveer Kancherla <
raghuveer.kancherla@...> wrote:

> Hi,
> I am using Solr1.4 for searching through half a million documents. The
> problem is, I want to retrieve nearly 200 documents for each search query.
> The query time in Solr logs is showing 0.02 seconds and I am fairly happy
> with that. However Solr is taking a long time (4 to 5 secs) to return the
> results (I think it is because of the number of docs I am requesting). I
> tried returning only the id's (unique key) without any other stored fields,
> but it is not helping me improve the response times (time to return the
> id's
> of matching documents).
> I understand that retrieving 200 documents for each search term is
> impractical in most scenarios but I dont have any other option. Any
> pointers
> on how to improve the response times will be a great help.
>
> Thanks,
> Raghu
>

--
Andrew Klochkov
Senior Software Engineer,
Grid Dynamics

From forum: Solr - User

Re: Wildcard searches within phrases to use proximity

2009-11-26T11:03:32Z

> That'd be great. Please open an issue in Jira and attach a
> patch. See
> http://wiki.apache.org/solr/HowToContribute
>

Hi Shalin,
I opened an issue (SOLR-1604) and attached a patch as well as a maven project to enable this feature without applying the patch. I couldn't consume ComplexPhraseQueryParser from lucene-misc-2.9.1.jar. Because there is a fixed bug but it is not included in lucene release. LUCENE-1486 says guidance needed from the Solr team about preferred course of action.

I will add more test cases to the patch in the future.

Thanks.

From forum: Solr - User

[jira] Updated: (SOLR-1604) Wildcards, ORs etc inside Phrase Queries

2009-11-26T10:40:39Z

[ https://issues.apache.org/jira/browse/SOLR-1604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ahmet Arslan updated SOLR-1604:
-------------------------------

Attachment: ComplexPhraseQueryParser.java
SOLR-1604.patch

There is a fixed but forgotten bug in ComplexPhraseQuery at lucene-misc-2.9.1.jar.
This patch includes addition of a jar to trunk/lib directory.
This jar has the corrected version of ComplexPhraseQueryParser.
I couldn't include the corrected version of ComplexPhraseQueryParser in patch because it accesses a package-private field in Lucene's QueryParser.

> Wildcards, ORs etc inside Phrase Queries
> ----------------------------------------
>
> Key: SOLR-1604
> URL: https://issues.apache.org/jira/browse/SOLR-1604
> Project: Solr
> Issue Type: Improvement
> Components: search
> Affects Versions: 1.4
> Reporter: Ahmet Arslan
> Priority: Minor
> Attachments: ComplexPhrase.zip, ComplexPhraseQueryParser.java, SOLR-1604.patch
>
>
> Solr Plugin for ComplexPhraseQueryParser (LUCENE-1486) which supports wildcards, ORs, ranges, fuzzies inside phrase queries.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

From forum: Solr - Dev

[jira] Updated: (SOLR-1604) Wildcards, ORs etc inside Phrase Queries

2009-11-26T10:18:39Z

[ https://issues.apache.org/jira/browse/SOLR-1604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ahmet Arslan updated SOLR-1604:
-------------------------------

Attachment: ComplexPhrase.zip

To enable ComplexPhraseQueryParser in Solr 1.4.0

1-) extract ComplexPhrase.zip and run 'mvn package'
2-) copy the ComplexPhrase/target/ComplexPhrase-1.0.jar to solrhome/lib directory
3-) register queryparser to solrhome/conf/solrconfig.xml by adding
<queryParser name="complexphrase" class="org.apache.solr.search.ComplexPhraseQParserPlugin" />
4-) enable it by appending &defType=complexphrase to search url.
5-) More permanent usage can be configured in solrconfig.xml
<requestHandler name="standard" class="solr.StandardRequestHandler" default="true">
<lst name="defaults">
<str name="defType">complexphrase</str>
</lst>
</requestHandler>

> Wildcards, ORs etc inside Phrase Queries
> ----------------------------------------
>
> Key: SOLR-1604
> URL: https://issues.apache.org/jira/browse/SOLR-1604
> Project: Solr
> Issue Type: Improvement
> Components: search
> Affects Versions: 1.4
> Reporter: Ahmet Arslan
> Priority: Minor
> Attachments: ComplexPhrase.zip
>
>
> Solr Plugin for ComplexPhraseQueryParser (LUCENE-1486) which supports wildcards, ORs, ranges, fuzzies inside phrase queries.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

From forum: Solr - Dev

Re: Deduplication in 1.4

2009-11-26T10:06:44Z

Two sites that use field-collapsing:
1) www.ilocal.nl
2) www.welke.nl
I'm not sure what you mean with double-tripping? The sites mentioned
do not have performance problems that are caused by field collapsing.

Field-collapsing currently only supports quasi distributed
field-collapsing (as I have described on the Solr wiki). Currently I
don't know a distributed field-collapsing algorithm that works
properly and does not influence the search time in such a way that the
search becomes slow.

Martijn

2009/11/26 Otis Gospodnetic <otis_gospodnetic@...>:

> Hi Martijn,
>
>
> ----- Original Message ----
>
>> From: Martijn v Groningen <martijn.is.hier@...>
>> To: solr-user@...
>> Sent: Thu, November 26, 2009 3:19:40 AM
>> Subject: Re: Deduplication in 1.4
>>
>> Field collapsing has been used by many in their production
>> environment.
>
> Got any pointers to public sites you know use it? I know of a high traffic site that used an early version, and it caused performance problems. Is double-tripping still required?
>
>> The last few months the stability of the patch grew as
>> quiet some bugs were fixed. The only big feature missing currently is
>> caching of the collapsing algorithm. I'm currently working on that and
>
> Is it also full distributed-search-ready?
>
>> I will put it in a new patch in the coming next days. So yes the
>> patch is very near being production ready.
>
> Thanks,
> Otis
>
>> Martijn
>>
>> 2009/11/26 KaktuChakarabati :
>> >
>> > Hey Otis,
>> > Yep, I realized this myself after playing some with the dedupe feature
>> > yesterday.
>> > So it does look like Field collapsing is what I need pretty much.
>> > Any idea on how close it is to being production-ready?
>> >
>> > Thanks,
>> > -Chak
>> >
>> > Otis Gospodnetic wrote:
>> >>
>> >> Hi,
>> >>
>> >> As far as I know, the point of deduplication in Solr (
>> >> http://wiki.apache.org/solr/Deduplication ) is to detect a duplicate
>> >> document before indexing it in order to avoid duplicates in the index in
>> >> the first place.
>> >>
>> >> What you are describing is closer to field collapsing patch in SOLR-236.
>> >>
>> >> Otis
>> >> --
>> >> Sematext is hiring -- http://sematext.com/about/jobs.html?mls
>> >> Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
>> >>
>> >>
>> >>
>> >> ----- Original Message ----
>> >>> From: KaktuChakarabati
>> >>> To: solr-user@...
>> >>> Sent: Tue, November 24, 2009 5:29:00 PM
>> >>> Subject: Deduplication in 1.4
>> >>>
>> >>>
>> >>> Hey,
>> >>> I've been trying to find some documentation on using this feature in 1.4
>> >>> but
>> >>> Wiki page is alittle sparse..
>> >>> In specific, here's what i'm trying to do:
>> >>>
>> >>> I have a field, say 'duplicate_group_id' that i'll populate based on some
>> >>> offline documents deduplication process I have.
>> >>>
>> >>> All I want is for solr to compute a 'duplicate_signature' field based on
>> >>> this one at update time, so that when i search for documents later, all
>> >>> documents with same original 'duplicate_group_id' value will be rolled up
>> >>> (e.g i'll just get the first one that came back according to relevancy).
>> >>>
>> >>> I enabled the deduplication processor and put it into updater, but i'm
>> >>> not
>> >>> seeing any difference in returned results (i.e results with same
>> >>> duplicate_id are returned separately..)
>> >>>
>> >>> is there anything i need to supply in query-time for this to take effect?
>> >>> what should be the behaviour? is there any working example of this?
>> >>>
>> >>> Anything will be helpful..
>> >>>
>> >>> Thanks,
>> >>> Chak
>> >>> --
>> >>> View this message in context:
>> >>> http://old.nabble.com/Deduplication-in-1.4-tp26504403p26504403.html
>> >>> Sent from the Solr - User mailing list archive at Nabble.com.
>> >>
>> >>
>> >>
>> >
>> > --
>> > View this message in context:
>> http://old.nabble.com/Deduplication-in-1.4-tp26504403p26522386.html
>> > Sent from the Solr - User mailing list archive at Nabble.com.
>> >
>> >
>
>

From forum: Solr - User

Retrieving large num of docs

2009-11-26T09:48:11Z

Hi,
I am using Solr1.4 for searching through half a million documents. The
problem is, I want to retrieve nearly 200 documents for each search query.
The query time in Solr logs is showing 0.02 seconds and I am fairly happy
with that. However Solr is taking a long time (4 to 5 secs) to return the
results (I think it is because of the number of docs I am requesting). I
tried returning only the id's (unique key) without any other stored fields,
but it is not helping me improve the response times (time to return the id's
of matching documents).
I understand that retrieving 200 documents for each search term is
impractical in most scenarios but I dont have any other option. Any pointers
on how to improve the response times will be a great help.

Thanks,
Raghu

From forum: Solr - User

Maximum number of fields allowed in a Solr document

2009-11-26T09:47:36Z

Hi,

We are in the process of designing a Solr app where we might have
millions of documents and within each of the document, we might have
thousands of dynamic fields. These fields are small and only contain
an integer, which needs to be retrievable and sortable.

My questions is:

1. Is there a limit on the number of fields allowed per document?
2. What is the performance impact for such design?
3. Has anyone done this before and is it a wise thing to do?

Thanks,

Alex

From forum: Solr - User

Re: Multi-Term Synonyms

2009-11-26T09:08:30Z

Hi Brad,

I was trying this, too, and there is a possibility how to get multi-term
synonyms to work properly. I wrote my solution already on this list.

My solution was as follows:

[cite]
after your hints that had partially confirmed my considerations, I had
made some tests with the FieldQParser. At the beginning, I had have some
problems, but finally, I was able to solve the problem of multi-word
synonyms at query time in a way that is suitable for us - and possibly
for others, too.

At my solution, I re-used the FieldQParserPlugin. At first, I ported it
to the new API (incrementToken instead of next, etc.) and then I
modified the code so, that no PhraseQueries will be created but only
BooleanQueries.

Now with my new QParserPlugin that based on the FieldQParserPlugin, it's
possible to search for things like "foo bar baz", where "foo bar" has to
be changed to "foo_bar" and where at the end the tokens "foo_bar" und
"baz" will be created, so that both could match independently.
[/cite]

Our current version is re-worked again, so that also multi-field queries
are possible.

If you want to use such a solution, you have probably to go without
complex query parsing et cetera. I also have to write your own modified
QParser, that fit your special needs. Also some higher features, like
they are offered by other QParsers could be integrated. It's all up to
you and your needs.

Patrick

brad anderson schrieb:

> Thanks for the help. Can't believe I missed that part in the wiki.
>
> 2009/11/24 Tom Hill <solr-list@...>
>
>> Hi Brad,
>>
>>
>> I suspect that this section from the wiki for SynonymFilterFactory might be
>> relevant:
>>
>>
>> http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.SynonymFilterFactory
>>
>> *"Keep in mind that while the SynonymFilter will happily work with synonyms
>> containing multiple words (ie: "**sea biscuit, sea biscit, seabiscuit**")
>> The recommended approach for dealing with synonyms like this, is to expand
>> the synonym when indexing. This is because there are two potential issues
>> that can arrise at query time:*
>>
>> 1.
>>
>> *The Lucene QueryParser tokenizes on white space before giving any text
>> to the Analyzer, so if a person searches for the words **sea biscit** the
>> analyzer will be given the words "sea" and "biscit" seperately, and will
>> not
>> know that they match a synonym."*
>>
>> ...
>>
>> Tom
>>
>> On Tue, Nov 24, 2009 at 10:47 AM, brad anderson <solrinterest@...
>>> wrote:
>>> Hi Folks,
>>>
>>> I was trying to get multi term synonyms to work. I'm experiencing some
>>> strange behavior and would like some feedback.
>>>
>>> In the synonyms file I have the line:
>>>
>>> thomas, boll holly, thomas a, john q => tom
>>>
>>> And I have a document with the text field as;
>>>
>>> tom
>>>
>>> However, when I do a search on boll holly, it does not return the
>> document
>>> with tom. The same thing happens if I do a query on john q. But if I do a
>>> query on thomas, it gives me the document. Also, if I quote "boll holly"
>> or
>>> "john q" it gives back the document.
>>>
>>> When I look at the analyzer page on the solr admin page, it is
>> transforming
>>> "boll holly" to "tom" when it isn't quoted. Why is it that it is not
>>> returning the document? Is there some configuration I can make so it does
>>> return the document if I do an unquoted search on "boll holly"?
>>>
>>> My synonym filter is defined as follows, and is only defined on the query
>>> side:
>>>
>>> <filter class="solr.SynonymFilterFactory"
>>> synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
>>>
>>>
>>> I've also tried changing the synonym file to be
>>>
>>> tom, thomas, boll holly, thomas a, john q
>>>
>>> This produces the same results.
>>>
>>> Thanks,
>>> Brad
>>>
>

From forum: Solr - User

Re: configure solr

2009-11-26T09:01:10Z

Hi

1. Issue with jetty:
When you start the jetty server by running start.jar, just look at the logs
to verify whether jetty has started successfully or not. At times, the port
you are using to start jetty(in your case 8983) could be used by some other
apps, which can cause issues in start up.

2. Details are provided step by step in the wiki as mentioned by eric:
http://wiki.apache.org/solr/SolrTomcat

But I hope following points can be of some help in debugging issues with
Tomcat installation:

:: Configure Tomcat to recognise the solr home directory you created, by
adding the Java Option -Dsolr.solr.home=c:\web\solr
:: Also, you can easily debug the root cause of problem by looking into
catalina.out file under logs folder of your tomcat installation.

Thanks
Dipti

On Wed, Nov 25, 2009 at 3:25 AM, Jill Han <jill.han@...> wrote:

> Hi,
>
> I just downloaded solr -1.4.0 to my computer, C:\apache-solr-1.4.0.
>
> 1. I followed the instruction to run the sample, java -jar
> start.jar at C:\apache-solr-1.4.0\example
>
> And then go to http://localhost:8983/solr/admin, however, I got
>
>
> HTTP ERROR: 404
>
> NOT_FOUND
>
> RequestURI=/solr/admin
>
> Powered by jetty:// <http://jetty.mortbay.org>
>
> Did I miss something?
>
> 2. Since I can't get sample run, I tried to run it on tomcat
> server(5.5) directly as
>
> a. Copy/paste apache-solr-1.4.0.war to C:\Tomcat 5.5\webapps,
>
> b. Go to http://localhost:8080/apache-solr-1.4.0/
>
> The error message is" HTTP Status 500 - Severe errors in solr
> configuration.."
>
> 3. How to configure it on tomcat server?
>
> Your help is appreciated very much as always,
>
> Jill
>
>
>
>
>
>

From forum: Solr - User

Re: Creating Facets

2009-11-26T08:40:31Z

Examples can be found out at:
http://www.lucidimagination.com/Community/Hear-from-the-Experts/Articles/Faceted-Search-Solr

Simple configuration works with setting facet=true&facet.field=xyz

Thanks
Dipti

On Wed, Nov 25, 2009 at 3:29 AM, Lance Norskog <goksron@...> wrote:

> There is nothing special to configure. All facet processing happens
> during processing the query.
>
> On Tue, Nov 24, 2009 at 9:56 AM, Tommy Molto <tommymolto@...> wrote:
> > People,
> >
> > I look in the solr wiki and only found about the use of the fecets, not
> how
> > to configure it in the schema or solrconfig. Any tip how to do it?
> >
> > Att,
> >
>
>
>
> --
> Lance Norskog
> goksron@...
>

From forum: Solr - User

[jira] Updated: (SOLR-1357) SolrInputDocument cannot process dynamic fields

2009-11-26T08:18:39Z

[ https://issues.apache.org/jira/browse/SOLR-1357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Lars Grote updated SOLR-1357:
-----------------------------

Attachment: SOLR-1357.patch

provides the behavior like shown in my post on November the 26. 2009

> SolrInputDocument cannot process dynamic fields
> -----------------------------------------------
>
> Key: SOLR-1357
> URL: https://issues.apache.org/jira/browse/SOLR-1357
> Project: Solr
> Issue Type: Improvement
> Components: clients - java
> Reporter: Avlesh Singh
> Attachments: SOLR-1357.patch
>
>
> Adding data via {{SolrInputDocument}} is normally done by calling the {{addField}} method with a field name, field value and an optional boost. In case of dynamic fields, if field names are known upfront, then caller of this method just passes in the right name and it automatically works.
> This does not go well with users who use {{@interface Field}} annotations for automatic binding.
> As of SOLR-1129, users can annotate {{Map<String, String> propertyName}} with {{@Field ("field_*")}} kind of annotations to bind dynamic field data to. {{SolrInputDocument}} should exhibit the same behavior. The field {{value}} currently supported are - primitive, array, collection or an instance of Iterable. It can also take {{Map}} as values. If the field, for which {{addField}} method is called, is of dynamicField type (which can be derived from the field name), then the keys of the {{Map}}, passed as value, should be used to "compose" the correct field name.
> This should be supported
> {code:java}
> //This code sample should populate the dynamic fields "brands_Nokia" and "brands_Samsung"
> public class MyBean{
> @Field("brands_*)
> Map<String, Integer> brands;
>
> ...
> }
> Map<String, String> brands= new HashMap<String, String>();
> brands.put("Nokia", 1000);
> brands.put("Samsung", 100);
> MyBean myBean = new MyBean();
> myBean.setBrands(brands);
> solrServer.addBean(myBean);
> {code}
> We can think of supporting this too ...
> {code:java}
> //This code sample should populate the dynamic fields "brands_Nokia" and "brands_Samsung"
> Map<String, String> brands= new HashMap<String, String>();
> brands.put("Nokia", 1000);
> brands.put("Samsung", 100);
> SolrInputDocument doc = new SolrInputDocument();
> doc.addField("brands_*", brands);
> {code}

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

From forum: Solr - Dev

[jira] Updated: (SOLR-1357) SolrInputDocument cannot process dynamic fields

2009-11-26T08:18:39Z

[ https://issues.apache.org/jira/browse/SOLR-1357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Lars Grote updated SOLR-1357:
-----------------------------

Comment: was deleted

(was: provides the behavior like shown in my post on November the 26. 2009)

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

From forum: Solr - Dev

[jira] Commented: (SOLR-1357) SolrInputDocument cannot process dynamic fields

2009-11-26T08:14:39Z

[ https://issues.apache.org/jira/browse/SOLR-1357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782925#action_12782925 ]

Lars Grote commented on SOLR-1357:
----------------------------------

Hello,
I think the way Avlesh suggested is more intuitive. At least I tried exactly that and wondered why it did not work.

This is what I think would be the best:

{code:java}
// a Field like this:
@Field("*_brands")
Map<String,Integer> brands;
{code}

you should add data like this:
{code:java}
brands.put("Nokia", 1000);
brands.put("Samsung", 100);
{code}

and get the data out like this:
{code:java}
bean.getBrands().get("Nokia");
bean.getBrands().get("Samsung");
{code}

I used the idear of Jean and enhanced it to work like shown above.
I attached a patch for this, maybe someone can have a look. It is not well tested.

I have never upload a Patch or something before. so if i'm doing something wrong, or do not follow the rules or anything.
Please tell me.

Regards, Lars

> SolrInputDocument cannot process dynamic fields
> -----------------------------------------------
>
> Key: SOLR-1357
> URL: https://issues.apache.org/jira/browse/SOLR-1357
> Project: Solr
> Issue Type: Improvement
> Components: clients - java
> Reporter: Avlesh Singh
>
> Adding data via {{SolrInputDocument}} is normally done by calling the {{addField}} method with a field name, field value and an optional boost. In case of dynamic fields, if field names are known upfront, then caller of this method just passes in the right name and it automatically works.
> This does not go well with users who use {{@interface Field}} annotations for automatic binding.
> As of SOLR-1129, users can annotate {{Map<String, String> propertyName}} with {{@Field ("field_*")}} kind of annotations to bind dynamic field data to. {{SolrInputDocument}} should exhibit the same behavior. The field {{value}} currently supported are - primitive, array, collection or an instance of Iterable. It can also take {{Map}} as values. If the field, for which {{addField}} method is called, is of dynamicField type (which can be derived from the field name), then the keys of the {{Map}}, passed as value, should be used to "compose" the correct field name.
> This should be supported
> {code:java}
> //This code sample should populate the dynamic fields "brands_Nokia" and "brands_Samsung"
> public class MyBean{
> @Field("brands_*)
> Map<String, Integer> brands;
>
> ...
> }
> Map<String, String> brands= new HashMap<String, String>();
> brands.put("Nokia", 1000);
> brands.put("Samsung", 100);
> MyBean myBean = new MyBean();
> myBean.setBrands(brands);
> solrServer.addBean(myBean);
> {code}
> We can think of supporting this too ...
> {code:java}
> //This code sample should populate the dynamic fields "brands_Nokia" and "brands_Samsung"
> Map<String, String> brands= new HashMap<String, String>();
> brands.put("Nokia", 1000);
> brands.put("Samsung", 100);
> SolrInputDocument doc = new SolrInputDocument();
> doc.addField("brands_*", brands);
> {code}

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

From forum: Solr - Dev

Re: solr+jetty logging to syslog?

2009-11-26T08:04:58Z

With 1.4

-Add log4j jars to Solr

-Configure de SyslogAppender with something like:
log4j.appender.solrLog=org.apache.log4j.net.SyslogAppender
log4j.appender.solrLog.Facility=LOCAL0
log4j.appender.solrLog.SyslogHost=127.0.0.1
log4j.appender.solrLog.layout=org.apache.log4j.PatternLayout
log4j.appender.solrLog.layout.ConversionPattern=solr: %-4r [%t] %-5p %c - %m%n

-Install syslog-ng and let syslog accept udp packets. To do that uncomment in syslog-ng.conf the line
udp();
in
# all known message sources
source s_all {

Otis Gospodnetic wrote:

Not many people do that, judging from http://www.google.com/search?&q=+solr%20+syslogd .

But I think this is really not a Solr-specific question. Isn't the question really "how do I configure log4j to log to syslogd?". Oh, and then "how do I configure slf4j to use log4j?"

The answer to the first one is "by using SyslogAppender" (google says so)
The answer to the second one might be on http://fernandoribeiro.eti.br/2006/05/24/how-to-use-slf4j-with-log4j/

Otis
--
Sematext is hiring -- http://sematext.com/about/jobs.html?mls
Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR

----- Original Message ----
> From: Steve Conover <sconover@gmail.com>
> To: solr-user@lucene.apache.org
> Sent: Sat, November 21, 2009 4:09:57 PM
> Subject: Re: solr+jetty logging to syslog?
>
> Does no one send solr logging to syslog?
>
> On Thu, Nov 19, 2009 at 5:54 PM, Steve Conover wrote:
> > The solution involves slf4j to log4j to syslog (at least, for solr),
> > but I'm having some trouble stringing all the parts together. If
> > anyone is doing this, would you mind posting how you use slf4j-log4j
> > jar, what your log4j.properties looks like, what your java system
> > properties settings are, and anything else you think is relevant?
> >
> > Much appreciated
> >
> > -Steve
> >

From forum: Solr - User

Intensive querying give odd results in search

2009-11-26T07:44:55Z

Hi, All.

I have a problem regarding intensive query requesting.
I'm using SolrJ client through http in the client side and Solr 1.4 and tomcat 6.0.20 on the server side.

My purpose is to execute 3 different queries for each word in a list of words and get the number of results.

On the client
**********
String queryText = "titexpl:NASA";

SolrServer server = ServerConSearch.getInstance().getServer();
SolrQuery query = new SolrQuery();
query.setQuery(queryText);
query.addField("*");
query.addField("score");
query.setRows(0);
QueryResponse rsp = server.query(query);
final long resultsNum = rsp.getResults().getNumFound();

My problem is that I get different results for the same query. For example, if I search "titexpl:NASA" I get 10 results for the first time but after some tries I get 0 results. After more tries, I get 10 results again an so on.

I tried with Tomcat running on Windows Vista and Ubuntu (in VMWare) but always got the same problem. Also used the default Jetty that cames with Solr, but no luck.

Any thoughts?

Ze Marques

PS: No index update is being done during the queries.
PS 2: Adding a sleep of 50ms for each query request, seems to solve the problem.

From forum: Solr - User

Re: SOLR-1131 - Multiple Fields per Field Type

2009-11-26T07:04:52Z

On Nov 25, 2009, at 8:24 PM, Chris Hostetter wrote:

>
> I'm having a hard time wrapping my head arround this entire concept ... i
> know part of my problem is that your example use case seems somewhat
> nonsensical...
>
> : As a simple proof of concept, imagine that I define a new FieldType
> : called PlusMinusIntFieldType that extends IntField. This FieldType
> : takes in an int value and outputs two Fields: one with the original
> : value and one with the negative of the value.
> ...
>
> : OK, on the search side is where it gets tricky. The whole point of this
> : exercise is that the details are hidden from the user in the generic
> : case. Thus, a query of plusMinus:5 should automatically expand to
> : (plusMinus__0:5 OR plusMinus__1:-5). Of course, an expert user should
>
> ...nothing could match plusMinus__0:5 that didn't also match
> plusMinus__1:-5, so i don't really understand what the point of using the
> field expansion for a use case like this would be ... and that's making it
> hard for me to try and understand how this sort of system
> could/should/would be used at query time.

Kind of, if a user just inputs plusMinus:5, then sure, but they may also want to just search the negative portion. More importantly, though, they may have a QParser or some other component that can appropriately select one of the fields w/o the user knowing.

>
> perhaps a more realistic example would be helpful?
>
> ...or even some differnet simple and contrived examples that demonstrate
> how this could be usefull in a way that isn't possible with a single
> field.

OK, a more concrete example is spatial. A user will want to index a point as a lat lon. So, they index: <field name="latLon">49, -79</field>.

The implementation of how this gets indexed can be done in several ways. For starters, it can be represented as a single field using Geohash or even just as a string (even if that isn't useful for much). We don't need S-1131 for that at all. Next, they may just want to represent it as a two fields: one for the lat and one for the long. Again, not super hard to do now, but it requires the user to set it up, whereas with a LatLonFieldType, this would be hidden from them. Finally, consider the cartesian tier case. In this case, a single lat lon point could be mapped to a whole slew of tiers, where each tier is like a zoom level on a map application (like Google Maps). Here, we could have a CartesianTierFieldType, that takes in the lower and upper bounds of the tiers to represent, i.e. tier 4 through 17, and this would output 13 different fields. Local Solr currently handles this through dynamic fields and user level knowledge of the magic fields used.

For this case, there are several different search patterns:
1. The user may know the tier they want to search at and thus input tier and a zoom level.
2. User invokes a QParser to build a bounding box (see https://issues.apache.org/jira/browse/SOLR-1568) and the Parser is responsible for creating a filter that chooses the most appropriate tier to search against. So, the user might just say: {!tier lat=X lon=Y dist=10} and it will pick the most appropriate tier, whereas putting in dist=50 would likely pick a different tier.

Does that help?

BTW, all of this is tracked via SOLR-773.

From forum: Solr - Dev

WordDelimiterFilter and acronyms normalization

2009-11-26T06:44:35Z

Hi all!

Is there any ready-for-use filter which performs acronyms normalization such
as "I.N.C."->"INC"?

I see that Lucene's StandardFilter can do this but we can't use it as we're
using WhitespaceTokenizer instead of StandardTokenizer.

--
Andrew Klochkov
Senior Software Engineer,
Grid Dynamics

From forum: Solr - User

RE: schema-based Index-time field boosting

2009-11-26T06:29:38Z

Hi Chris, thanks for replying!

OK, now I'm going to take the bait ;)

I am talking about field boosting rather than document boosting, ie. I
would like some fields (say eg. title) to be "louder" than others,
across ALL documents. I believe you are at least partially talking
about document boosting, which clearly applies on a per-document basis.

If it helps, consider a schema version of the following, from
org.apache.solr.common.SolrInputDocument:

/**
* Adds a field with the given name, value and boost. If a field with
the name already exists, then it is updated to
* the new value and boost.
*
* @param name Name of the field to add
* @param value Value of the field
* @param boost Boost value for the field
*/
public void addField(String name, Object value, float boost )
{
SolrInputField field = _fields.get( name );
if( field == null || field.value == null ) {
setField(name, value, boost);
}
else {
field.addValue( value, boost );
}
}

Where a constant boost value is applied consistently to a given field.
That is what I was mistakenly hoping to achieve in the schema. I still
think it would be a good idea BTW. Regards,

Ian.

-----Original Message-----
From: Chris Hostetter [mailto:hossman_lucene@...]
Sent: 23 November 2009 18:34
To: solr-user@...
Subject: RE: schema-based Index-time field boosting

: Yeah, like I said, I was mistaken about setting field boost in
: schema.xml - doesn't mean it's a bad idea though. At any rate, from
: your penultimate sentence I reckon at least one of us is still
confused
: about field boosting, feel free to reply if you think it's me ;)

Yeah ... i think it's you. like i said...

: field boosting only makes sense if it's only applied to some of the
: documents in the index, if every document has an index time boost on
: fieldX, then that boost is meaningless.

...if there was a way to oost fields at index time that was configured
in the schema.xml, then every doc would get that boost on it's instances
of those fields but the only purpose of index time boosting is to
indicate that one document is more significant then another doc -- if
every doc gets the same boost, it becomes a No-OP.

(think about the math -- field boosts become multipliers in the
fieldNorm
-- if every doc gets the same multiplier, then there is no net effect)

-Hoss

Web design and intelligent Content Management. www.twitter.com/gossinteractive

Registered Office: c/o Bishop Fleming, Cobourg House, Mayflower Street, Plymouth, PL1 1LG. Company Registration No: 3553908

This email contains proprietary information, some or all of which may be legally privileged. It is for the intended recipient only. If an addressing or transmission error has misdirected this email, please notify the author by replying to this email. If you are not the intended recipient you may not use, disclose, distribute, copy, print or rely on this email.

Email transmission cannot be guaranteed to be secure or error free, as information may be intercepted, corrupted, lost, destroyed, arrive late or incomplete or contain viruses. This email and any files attached to it have been checked with virus detection software before transmission. You should nonetheless carry out your own virus check before opening any attachment. GOSS Interactive Ltd accepts no liability for any loss or damage that may be caused by software viruses.

From forum: Solr - User

SolrException caused by illegal character

2009-11-26T05:54:20Z

Hi,
I upgradeed to Solr 1.4 and tried to reindex the data. After few
thousand of reindexed documents an exception is thrown, I did not meet
this using 1.3 before. Do you have any idea what caused the problem?
Thanks.

SEVERE: org.apache.solr.common.SolrException: Illegal character
((CTRL-CHAR, code 3))
at [row,col {unknown-source}]: [6495,39]
at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:72)
at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:54)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
at org.mortbay.jetty.Server.handle(Server.java:285)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:502)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:835)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:641)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:208)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226)
at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
Caused by: com.ctc.wstx.exc.WstxUnexpectedCharException: Illegal
character ((CTRL-CHAR, code 3))
at [row,col {unknown-source}]: [6495,39]
at com.ctc.wstx.sr.StreamScanner.throwInvalidSpace(StreamScanner.java:675)
at com.ctc.wstx.sr.BasicStreamReader.readTextPrimary(BasicStreamReader.java:4556)
at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2888)
at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1019)
at org.apache.solr.handler.XMLLoader.readDoc(XMLLoader.java:273)
at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:138)
at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69)
... 22 more

From forum: Solr - User

Re: param "version" and diferences in /admin/ping response

2009-11-26T05:16:09Z

Tank you Chris... I didn't see it. I was looking for something related
with the PingRequestHandler.
Regards.
Nestor Oviedo

On Wed, Nov 25, 2009 at 7:09 PM, Chris Hostetter
<hossman_lucene@...> wrote:
> : Hi everyone!
> : Can anyone tell me what's the meaning of the param "version" ?? There
> : isn't anything about it in the Solr documentation.
>
> http://wiki.apache.org/solr/XMLResponseFormat#A.27version.27
>
> -Hoss
>
>

From forum: Solr - User

Re: [jira] Created: (SOLR-1592) Refactor XMLWriter startTag to allow arbitrary attributes to be written

2009-11-26T04:29:33Z

Chris Hostetter wrote:
> Perhaps instead of making API additions/generalizations to XMLWriter, we
> should approach the problem differnelty and think about what a good
> reusable general pupose XML generation toolkit that could be used by
> custom response writers should look like, and then implement that --
> refactoring the existing XMLWriter class to use/extend this new class
> where possible?

I agree completely. Looking at solr/lucene from an XML/XSLT/X... background my first impression
is that the XML/XSLT features are just what is absolutely necessary to get it running. As we are
going to use solr in an XML based environment, we'll be happy to exchange ideas about "full" (or
at least "more" xml support.

--
Age Jan Kuperus
FB ICT / Wageningen UR Library

From forum: Solr - Dev

Re: Deduplication in 1.4

2009-11-26T03:09:45Z

Hi Martijn,

----- Original Message ----

> From: Martijn v Groningen <martijn.is.hier@...>
> To: solr-user@...
> Sent: Thu, November 26, 2009 3:19:40 AM
> Subject: Re: Deduplication in 1.4
>
> Field collapsing has been used by many in their production
> environment.

Got any pointers to public sites you know use it? I know of a high traffic site that used an early version, and it caused performance problems. Is double-tripping still required?

> The last few months the stability of the patch grew as
> quiet some bugs were fixed. The only big feature missing currently is
> caching of the collapsing algorithm. I'm currently working on that and

Is it also full distributed-search-ready?

> I will put it in a new patch in the coming next days. So yes the
> patch is very near being production ready.

Thanks,
Otis

> Martijn
>
> 2009/11/26 KaktuChakarabati :
> >
> > Hey Otis,
> > Yep, I realized this myself after playing some with the dedupe feature
> > yesterday.
> > So it does look like Field collapsing is what I need pretty much.
> > Any idea on how close it is to being production-ready?
> >
> > Thanks,
> > -Chak
> >
> > Otis Gospodnetic wrote:
> >>
> >> Hi,
> >>
> >> As far as I know, the point of deduplication in Solr (
> >> http://wiki.apache.org/solr/Deduplication ) is to detect a duplicate
> >> document before indexing it in order to avoid duplicates in the index in
> >> the first place.
> >>
> >> What you are describing is closer to field collapsing patch in SOLR-236.
> >>
> >> Otis
> >> --
> >> Sematext is hiring -- http://sematext.com/about/jobs.html?mls
> >> Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
> >>
> >>
> >>
> >> ----- Original Message ----
> >>> From: KaktuChakarabati
> >>> To: solr-user@...
> >>> Sent: Tue, November 24, 2009 5:29:00 PM
> >>> Subject: Deduplication in 1.4
> >>>
> >>>
> >>> Hey,
> >>> I've been trying to find some documentation on using this feature in 1.4
> >>> but
> >>> Wiki page is alittle sparse..
> >>> In specific, here's what i'm trying to do:
> >>>
> >>> I have a field, say 'duplicate_group_id' that i'll populate based on some
> >>> offline documents deduplication process I have.
> >>>
> >>> All I want is for solr to compute a 'duplicate_signature' field based on
> >>> this one at update time, so that when i search for documents later, all
> >>> documents with same original 'duplicate_group_id' value will be rolled up
> >>> (e.g i'll just get the first one that came back according to relevancy).
> >>>
> >>> I enabled the deduplication processor and put it into updater, but i'm
> >>> not
> >>> seeing any difference in returned results (i.e results with same
> >>> duplicate_id are returned separately..)
> >>>
> >>> is there anything i need to supply in query-time for this to take effect?
> >>> what should be the behaviour? is there any working example of this?
> >>>
> >>> Anything will be helpful..
> >>>
> >>> Thanks,
> >>> Chak
> >>> --
> >>> View this message in context:
> >>> http://old.nabble.com/Deduplication-in-1.4-tp26504403p26504403.html
> >>> Sent from the Solr - User mailing list archive at Nabble.com.
> >>
> >>
> >>
> >
> > --
> > View this message in context:
> http://old.nabble.com/Deduplication-in-1.4-tp26504403p26522386.html
> > Sent from the Solr - User mailing list archive at Nabble.com.
> >
> >

From forum: Solr - User

Links to in-dev features on wiki?

2009-11-26T02:17:35Z

Pages such as ZooKeeperIntegration and KattaIntegration used to be linked
off the Wiki front page. Now I can't find them any more. Where have those
links been moved to?

--
Regards,
Shalin Shekhar Mangar.

From forum: Solr - Dev

Re: Looking for Best Practices: Analyzers vs. UpdateRequestProcessors?

2009-11-26T02:07:15Z

On Wed, Nov 25, 2009 at 9:52 PM, Andreas Kahl <Andreas_Kahl@...> wrote:

> Hello,
>
> are there any general criteria when to use Analyzers to implement an
> indexing function and when it is better to use UpdateRequestProcessors?
>
> The main difference I found in the documentation was that
> UpdateRequestProcessors are able to manipulate several fields at once
> (create, read, update, delete), while Analyzers operate on the contents of a
> single field at once.
>
>

Analyzers can only change indexed content. If a field is marked as "stored",
then it is stored and retrieved un-modified. If you want to modify the
"stored" part as well, then only an UpdateRequestProcessor can do that. In
other words, the field's value after applying UpdateRequestProcessors is fed
into analyzers (for indexed field) and stored verbatim (for stored fields).

--
Regards,
Shalin Shekhar Mangar.

From forum: Solr - User