Nabble - Solr - User

Re: Response writer configs

2009-12-01T20:08:52Z

On Dec 1, 2009, at 9:04 PM, Ross wrote:
> I'm starting to play with Solr. This might be a silly question and not
> particularly important but I'm curious.
>
> I setup the example site using the tutorial. It works very well. I was
> looking around the config files and notice that in my solrconfig.xml
> that the queryResponseWriter area is commented out but they all still
> work. wt=php etc returns the php format. How is it working if they're
> not defined? Are they defined elsewhere?

Good question. Solr defines (in SolrCore) a default set of response
writers.

static{
HashMap<String, QueryResponseWriter> m= new HashMap<String,
QueryResponseWriter>();
m.put("xml", new XMLResponseWriter());
m.put("standard", m.get("xml"));
m.put("json", new JSONResponseWriter());
m.put("python", new PythonResponseWriter());
m.put("php", new PHPResponseWriter());
m.put("phps", new PHPSerializedResponseWriter());
m.put("ruby", new RubyResponseWriter());
m.put("raw", new RawResponseWriter());
m.put("javabin", new BinaryResponseWriter());
DEFAULT_RESPONSE_WRITERS = Collections.unmodifiableMap(m);
}

Note that these built in ones can be overridden, but not undefined, by
registering a response writer with the same name as a built in one.

Erik

Response writer configs

2009-12-01T18:04:49Z

Hi all

I'm starting to play with Solr. This might be a silly question and not
particularly important but I'm curious.

I setup the example site using the tutorial. It works very well. I was
looking around the config files and notice that in my solrconfig.xml
that the queryResponseWriter area is commented out but they all still
work. wt=php etc returns the php format. How is it working if they're
not defined? Are they defined elsewhere?

Thanks
Ross

question about schemas

2009-12-01T15:27:10Z

I just started using Solr, and I am trying to figure out how to setup my schema. I know that Solr doesn’t have JOINs, and so I am having some difficulty figuring out how would I setup a schema for the following fictional situation. For example, let us say that :

- I have a 10000+ customers, each having some specific info (StoreId , Name, Phone, Address, City, State, Zip, etc)
- Each customer has a subset of the 100+ products I am looking to track, each product having some specific info (ProductId, Name, Width, Height, Depth, Weight, Density, etc)
- I want to be able to search by the product info but have facets return the number of customers, rather than the number of products, that meet my criteria
- I want to display (and sort) customers based on my product search

In relational databases, I would simply create two tables (customer and product) and JOIN them. I could then craft a sql query to count the number of distinct StoreId values in the result (something like facets).

In Solr, however, there are no joins. As far as I can tell, my options are to:

- create two Solr instances, one with customer info and one with product info; I would search the product Solr instance and identify the StoreId values return, and then use that info to search the customer Solr instance to get the customer info. The problem with this is the second query could have ten thousand ANDs (one for each StoreId returned by the first query)
- create a single Solr instance that contains a denormalized version of the data where each doc would contain both the customer info and the product info for a given product. The problem with this is that my facets would return the number of products, not the number of customers
- create a single Solr instance that contains a denormalized version of the data where each doc contains the customer info and info for ALL products that the customer might have (likely done via dynamicfields). The problem with this is that my schema would be a bit messy and that my queries could have hundreds of ANDs and Ors (one AND for each product field, and one OR for each product); for example, q=((Width1:50 AND Density1:7) OR (Width2:50 AND Density2:7) OR …)

Does anyone have any advice on this? Are there other schemas that might work? Hopefully the example makes sense.

Re: Maximum number of fields allowed in a Solr document

2009-12-01T13:47:45Z

Lucene creates an array of one item per document for every field you
sort on. If you sort on a thousand fields, Lucene will create 1000
different arrays of 500K ints. I assume there is some sort of cache of
these arrays. In Solr, it is also possible to sort using a function as
the relevance value. This is rather slow, and caches no data between
queries.

You may want to do sorting in your front-end applications, or get
database ids from Solr and do sorting in the database query.

On Mon, Nov 30, 2009 at 7:14 AM, Alex Wang <awang@...> wrote:

> Thanks Otis for the reply. Yes this will be pretty memory intensive.
> The size of the index is 5 cores with a maximum of 500K documents each
> core. I did search the archives before but did not find any definite
> answer. Thanks again!
>
> Alex
>
>
>
> On Nov 27, 2009, at 11:09 PM, Otis Gospodnetic wrote:
>
>> Hi Alex,
>>
>> There is no build-in limit. The limit is going to be dictated by
>> your hardware resources. In particular, this sounds like a memory
>> intensive app because of sorting on lots of different fields. You
>> didn't mention the size of your index, but that's a factor, too.
>> Once in a while people on the list mention cases with lots and lots
>> of fields, so I'd check ML archives.
>>
>> Otis
>> --
>> Sematext is hiring -- http://sematext.com/about/jobs.html?mls
>> Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
>>
>>
>>
>> ----- Original Message ----
>>> From: Alex Wang <awang@...>
>>> To: "solr-user@..." <solr-user@...>
>>> Sent: Thu, November 26, 2009 12:47:36 PM
>>> Subject: Maximum number of fields allowed in a Solr document
>>>
>>> Hi,
>>>
>>> We are in the process of designing a Solr app where we might have
>>> millions of documents and within each of the document, we might have
>>> thousands of dynamic fields. These fields are small and only contain
>>> an integer, which needs to be retrievable and sortable.
>>>
>>> My questions is:
>>>
>>> 1. Is there a limit on the number of fields allowed per document?
>>> 2. What is the performance impact for such design?
>>> 3. Has anyone done this before and is it a wise thing to do?
>>>
>>> Thanks,
>>>
>>> Alex
>>
>
>

--
Lance Norskog
goksron@...

Re: Creating Facets

2009-12-01T13:21:01Z

Also, facet values are pulled from the indexed terms rather than the
stored source fields. This means that if you want to facet on a field,
that field has to be declared 'indexed="true"' in schema.xml.

On Thu, Nov 26, 2009 at 8:40 AM, dipti khullar <dipti.khullar@...> wrote:

> Examples can be found out at:
> http://www.lucidimagination.com/Community/Hear-from-the-Experts/Articles/Faceted-Search-Solr
>
> Simple configuration works with setting facet=true&facet.field=xyz
>
> Thanks
> Dipti
>
> On Wed, Nov 25, 2009 at 3:29 AM, Lance Norskog <goksron@...> wrote:
>
>> There is nothing special to configure. All facet processing happens
>> during processing the query.
>>
>> On Tue, Nov 24, 2009 at 9:56 AM, Tommy Molto <tommymolto@...> wrote:
>> > People,
>> >
>> > I look in the solr wiki and only found about the use of the fecets, not
>> how
>> > to configure it in the schema or solrconfig. Any tip how to do it?
>> >
>> > Att,
>> >
>>
>>
>>
>> --
>> Lance Norskog
>> goksron@...
>>
>

--
Lance Norskog
goksron@...

NYC Search & Discovery Meetup

2009-12-01T12:39:08Z

Hello,

For those living in or near NYC, you may be interested in joining (and/or presenting?) at the NYC Search & Discovery Meetup.
Topics are: search, machine learning, data mining, NLP, information gathering, information extraction, etc.

http://www.meetup.com/NYC-Search-and-Discovery/

Our previous/first meetup was about solr-python and parse.ly (a service that makes use of Solr and solr-python).

Tomorrow (December 2 2009) we have:

Incorporating Probabilistic Retrieval Knowledge into TFIDF-based Search Engine

You can RSVP at:
http://www.meetup.com/NYC-Search-and-Discovery/calendar/11745435/

Otis
--
Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch

Re: Where to put ExternalRequestHandler and Tika jars

2009-12-01T11:59:05Z

Now it is working.. It was a libraries problem as you said...

thanks!

On Mon, Nov 30, 2009 at 12:25 PM, javaxmlsoapdev <vikasdp@...> wrote:

>
> Yes. code I posted in first thread does work. And I am able to retrieve
> data
> from the document index. did you include all required jars in deployed solr
> application's lib folder? what errors are you seeing?
>
> Juan Pedro Danculovic wrote:
> >
> > HI! does your example finally works? I index the data with solrj and I
> > have
> > the same problem and could not retrieve file data.
> >
> >
> > On Wed, Nov 25, 2009 at 3:41 PM, javaxmlsoapdev <vikasdp@...>
> wrote:
> >
> >>
> >> grrrrrrrr. I had to include tika and related parsing jars into
> >> tomcat/webapps/solr/WEB-INF/lib.. this was an embarrassing mistake.
> >> apologies for all the noise.
> >>
> >> Thanks,
> >> --
> >> View this message in context:
> >>
> http://old.nabble.com/Where-to-put-ExternalRequestHandler-and-Tika-jars-tp26515579p26518100.html
> >> Sent from the Solr - User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
>
> --
> View this message in context:
> http://old.nabble.com/Where-to-put-ExternalRequestHandler-and-Tika-jars-tp26515579p26576242.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>

Re: latency in solr response is observed after index is updated

2009-12-01T10:42:11Z

What would be the average doc size. What is the autoCommit frequency
set in solrconfig.xml .

Another place to look at is the field cache size and the nature of
warmup queries run after a new searcher is created ( happens due to a
commit ).

Bharath Venkatesh wrote:

> Hi Kalidoss,
>
> I am not aware of using solr-config for committing the document .
> but I have mentioned below how we update and commit documents:
>
> curl http://solr_url/update --data-binary @feeds.xml -H
> 'Content-type:text/xml; charset=utf-8'
> curl http://solr_url/update --data-binary '<commit/>' -H
> 'Content-type:text/xml; charset=utf-8'
>
> where feeds.xml contains the document in xml format
>
> we have master and slave replication for solr server.
>
> updates happens in master , snappuller and snapinstaller is run on
> slaves periodically
> queries don't happen at master , only happens at slaves
>
> is there any thing which can be said from above information ?
>
> Thanks,
> Bharath
>
>
>
> -----Original Message-----
> From: kalidoss [mailto:kalidoss.muthuramalingam@...]
> Sent: Tue 12/1/2009 2:38 PM
> To: solr-user@...
> Subject: Re: latency in solr response is observed after index is updated
>
> r u using solr-config for committing the document?
>
> bharath venkatesh wrote:
>
>> Hi,
>>
>> We are observing latency (some times huge latency upto 10-20 secs)
>> in solr response after index is updated . whats the reason of this
>> latency and how can it be minimized ?
>> Note: our index size is pretty large.
>>
>> any help would be appreciated as we largely affected by it
>>
>> Thanks in advance.
>> Bharath
>>
>
>
>
>
>
>
> This message is intended only for the use of the addressee and may contain information that is privileged, confidential
> and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the
> employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any
> dissemination, distribution or copying of this communication is strictly prohibited. If you have received this e-mail
> in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from your system.
>

Re: getting total index size & last update date/time from query

2009-12-01T07:49:48Z

The drawback to stats.jsp is that the XML it just that. It isn't in
standard response writer format, so you can only get XML, not JSON,
etc like you can with /admin/luke.

And as mentioned in my previous post, use numTerms=0 for /admin/luke
to speed it up.

Erik

On Nov 21, 2009, at 12:00 AM, Lance Norskog wrote:

> solr/admin/stats.jsp gives a much larger XML dump and also includes
> these two data items.
>
> Note that Luke can walk the entire index data structures, so if you
> have a large index it's like playing with fire.
>
> On Thu, Nov 19, 2009 at 8:54 AM, Binkley, Peter
> <Peter.Binkley@...> wrote:
>> The Luke request handler (normally available at <solr>/admin/luke)
>> will
>> give you the document count (not size on the disk, though, if that's
>> what you want) and last update and other info:
>>
>> <lst name="index">
>> <int name="numDocs">14591</int>
>> <int name="maxDoc">14598</int>
>> <int name="numTerms">128730</int>
>> <long name="version">1196962176380</long>
>> <bool name="optimized">false</bool>
>> <bool name="current">true</bool>
>> <bool name="hasDeletions">true</bool>
>> <str name="directory">.... s/solr/data/index</str>
>> <date name="lastModified">2009-11-19T16:44:45Z</date>
>> </lst>
>>
>> See http://wiki.apache.org/solr/LukeRequestHandler
>>
>> Peter
>>
>>
>>
>>> -----Original Message-----
>>> From: Joel Nylund [mailto:jnylund@...]
>>> Sent: Thursday, November 19, 2009 8:31 AM
>>> To: solr-user@...
>>> Subject: getting total index size & last update date/time from query
>>>
>>> Hi,
>>>
>>> Looking for total number of documents in my index and the
>>> last updated date/time of the index.
>>>
>>> Is there a way to get this through the standard query q=?
>>>
>>> if not, what is the best way to get this info from solr.
>>>
>>> thanks
>>> Joel
>>>
>>>
>>
>
>
>
> --
> Lance Norskog
> goksron@...

Re: getting total index size & last update date/time from query

2009-12-01T07:48:30Z

Use numTerms=0 - /admin/luke?numTerms=0 that'll speed it up tons.

Erik

On Dec 1, 2009, at 10:46 AM, Joel Nylund wrote:

> Hi, Luke worked, but we are finding it really slow in our
> environment (8-10 seconds). Is there a way to just get document
> count & last index time with a faster call, possibly passing
> something to luke?
>
> thanks
> Joel
>
> On Nov 19, 2009, at 11:54 AM, Binkley, Peter wrote:
>
>> The Luke request handler (normally available at <solr>/admin/luke)
>> will
>> give you the document count (not size on the disk, though, if that's
>> what you want) and last update and other info:
>>
>> <lst name="index">
>> <int name="numDocs">14591</int>
>> <int name="maxDoc">14598</int>
>> <int name="numTerms">128730</int>
>> <long name="version">1196962176380</long>
>> <bool name="optimized">false</bool>
>> <bool name="current">true</bool>
>> <bool name="hasDeletions">true</bool>
>> <str name="directory">.... s/solr/data/index</str>
>> <date name="lastModified">2009-11-19T16:44:45Z</date>
>> </lst>
>>
>> See http://wiki.apache.org/solr/LukeRequestHandler
>>
>> Peter
>>
>>
>>
>>> -----Original Message-----
>>> From: Joel Nylund [mailto:jnylund@...]
>>> Sent: Thursday, November 19, 2009 8:31 AM
>>> To: solr-user@...
>>> Subject: getting total index size & last update date/time from query
>>>
>>> Hi,
>>>
>>> Looking for total number of documents in my index and the
>>> last updated date/time of the index.
>>>
>>> Is there a way to get this through the standard query q=?
>>>
>>> if not, what is the best way to get this info from solr.
>>>
>>> thanks
>>> Joel
>>>
>>>
>

Re: getting total index size & last update date/time from query

2009-12-01T07:46:01Z

Hi, Luke worked, but we are finding it really slow in our environment
(8-10 seconds). Is there a way to just get document count & last index
time with a faster call, possibly passing something to luke?

thanks
Joel

On Nov 19, 2009, at 11:54 AM, Binkley, Peter wrote:

> The Luke request handler (normally available at <solr>/admin/luke)
> will
> give you the document count (not size on the disk, though, if that's
> what you want) and last update and other info:
>
> <lst name="index">
> <int name="numDocs">14591</int>
> <int name="maxDoc">14598</int>
> <int name="numTerms">128730</int>
> <long name="version">1196962176380</long>
> <bool name="optimized">false</bool>
> <bool name="current">true</bool>
> <bool name="hasDeletions">true</bool>
> <str name="directory">.... s/solr/data/index</str>
> <date name="lastModified">2009-11-19T16:44:45Z</date>
> </lst>
>
> See http://wiki.apache.org/solr/LukeRequestHandler
>
> Peter
>
>
>
>> -----Original Message-----
>> From: Joel Nylund [mailto:jnylund@...]
>> Sent: Thursday, November 19, 2009 8:31 AM
>> To: solr-user@...
>> Subject: getting total index size & last update date/time from query
>>
>> Hi,
>>
>> Looking for total number of documents in my index and the
>> last updated date/time of the index.
>>
>> Is there a way to get this through the standard query q=?
>>
>> if not, what is the best way to get this info from solr.
>>
>> thanks
>> Joel
>>
>>

Re: How to avoid hardcoding masterUrl in slave solrconfig.xml?

2009-12-01T06:11:00Z

Thanks, Noble....That did the trick!

- Bill

--------------------------------------------------
From: "Noble Paul നോബിള്‍ नोब्ळ्" <noble.paul@...>
Sent: Monday, November 30, 2009 10:20 PM
To: <solr-user@...>
Subject: Re: How to avoid hardcoding masterUrl in slave solrconfig.xml?

> remove the <lst name="slave"> section from your solrconfig. It should be
> fine
>
> On Tue, Dec 1, 2009 at 6:59 AM, William Pierce <evalsinca@...>
> wrote:
>> Hi, Joe:
>>
>> I tried with the "fetchIndex" all lower-cased, and still the same result.
>> What do you specify for masterUrl in the solrconfig.xml on the slave?
>> it
>> seems to me that if I remove the element, I get the exception I wrote
>> about. If I set it to some dummy url, then I get an invalid url
>> message
>> when I run the command=details on the slave replication handler.
>>
>> What I am doing does not look out of the ordinary. I want to control
>> the
>> masterurl and the time of replication by myself. As such I want neither
>> the
>> masterUrl nor the polling interval in the config file. Can you share
>> relevant snippets of your config file and the exact url your code is
>> generating?
>>
>> Thanks,
>>
>> - Bill
>>
>> --------------------------------------------------
>> From: "Joe Kessel" <isjustjoe@...>
>> Sent: Monday, November 30, 2009 3:45 PM
>> To: <solr-user@...>
>> Subject: RE: How to avoid hardcoding masterUrl in slave solrconfig.xml?
>>
>>>
>>> I do something very similar and it works for me. I noticed on your URL
>>> that you have a mixed case fetchIndex, which the request handler is
>>> checking
>>> for fetchindex, all lowercase. If it is not that simple I can try to
>>> see
>>> the exact url my code is generating.
>>>
>>>
>>>
>>> Hope it helps,
>>>
>>> Joe
>>>
>>>> From: evalsinca@...
>>>> To: solr-user@...
>>>> Subject: Re: How to avoid hardcoding masterUrl in slave solrconfig.xml?
>>>> Date: Mon, 30 Nov 2009 13:48:38 -0800
>>>>
>>>> Folks:
>>>>
>>>> Sorry for this repost! It looks like this email went out twice....
>>>>
>>>> Thanks,
>>>>
>>>> - Bill
>>>>
>>>> --------------------------------------------------
>>>> From: "William Pierce" <evalsinca@...>
>>>> Sent: Monday, November 30, 2009 1:47 PM
>>>> To: <solr-user@...>
>>>> Subject: How to avoid hardcoding masterUrl in slave solrconfig.xml?
>>>>
>>>> > Folks:
>>>> >
>>>> > I do not want to hardcode the masterUrl in the solrconfig.xml of my >
>>>> > slave.
>>>> > If the masterUrl tag is missing from the config file, I am getting an
>>>> > exception in solr saying that the masterUrl is required. So I set it
>>>> > to
>>>> > some dummy value, comment out the poll interval element, and issue a
>>>> > replication command manually like so:
>>>> >
>>>> >
>>>> > http://localhost:port/postings/replication?command=fetchIndex&masterUrl=http://localhost:port/postingsmaster/replication
>>>> >
>>>> > Now no internal exception, solr responds with a status "OK" for the >
>>>> > above
>>>> > request, the tomcat logs show no error but the index is not
>>>> > replicated.
>>>> > When I issue the details command to the slave, I see that it ignored
>>>> > >
>>>> > the
>>>> > masterUrl on the command line but instead complains that the master
>>>> > url
>>>> > > in
>>>> > the config file (which I had set to a dummy value) is not correct.
>>>> >
>>>> > (Just fyi, I have tried sending in the masterUrl to the above command
>>>> > >
>>>> > with
>>>> > url encoding and also without. in both cases, I got the same result.)
>>>> >
>>>> > Soooo....how exactly do I avoid hardcoding the masterUrl in the
>>>> > config
>>>> > file? Any pointers/help will be greatly appreciated!
>>>> >
>>>> > - Bill
>>>>
>>>
>>> _________________________________________________________________
>>> Bing brings you maps, menus, and reviews organized in one place.
>>>
>>> http://www.bing.com/search?q=restaurants&form=MFESRP&publ=WLHMTAG&crea=TEXT_MFESRP_Local_MapsMenu_Resturants_1x1
>>
>>
>
>
>
> --
> -----------------------------------------------------
> Noble Paul | Principal Engineer| AOL | http://aol.com
>

Re: Phrase Query Issue

2009-12-01T05:20:58Z

Well, I don't think this is done normally. I'm assuming here that
you've simplified the example and your *real* requirements aren't
quite what you're showing, because what you've described seems
really confusing to a user.

That said, I don't think SOLR has a way to handle this out-of-the-box.
What you're asking for is for the query to search different fields based
on the *results* of the parsing. You have to deal with someone searching
for "tests" OR "run", which presumably should return anything
with "test", "tests", "testing", "run". But should NOT return a doc
with only "running" in the title. But to the parser, this starts out as
title:(tests run)....

So, you really need to see the results of feeding any term through
the stemmer. If it comes back the same as what you fed in, go
against one field. If it comes back stemmed, go against a
different field.

You'll probably have to create a custom parser that pre-processes
each term and determines which field should be searched, search
the SOLR list as creating custom parsers has been discussed
before by people more knowledgeable than me...

If my assumption that there are parts of this problem that you've
simplified is NOT correct, perhaps you could explain the use-case
that leads to this requirement....

But if anyone has a better idea, now's the time to speak up <G>.

Best
Erick

On Tue, Dec 1, 2009 at 12:13 AM, ravicv <ravichandrab4u@...> wrote:

>
> Thanks Erick But their any way to get phrase search and normal search
> together.
>
> for Example if I search for "test" it should only give test result but not
> tests..
> And if we search for tests it should give all results matching with
> tests(this should include stemmer results also)
>
> Please clarify me how normally this scenario is implemented?
>
> I am using field title with text type which contains
> SnowballPorterFilterFactory filter.
>
> And another field title_exact with out SnowballPorterFilterFactory filter.
>
> Both are copied using copy field.
>
> <copyField source="title" dest="text"/>
> <copyField source="title_exact" dest="text_exact"/>
> But as text type is having a SnowballPorterFilterFactory filter it stores
> tests as test.So always if I search for "test" tests also will come in my
> search result.
> Please tell me a way to avoid it.
>
> Thanks
> Ravichandra
> --
> View this message in context:
> http://old.nabble.com/Phrase-Query-Issue-tp22863529p26586788.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>

RE: latency in solr response is observed after index is updated

2009-12-01T02:13:30Z

Hi Kalidoss,

I am not aware of using solr-config for committing the document .
but I have mentioned below how we update and commit documents:

curl http://solr_url/update --data-binary @feeds.xml -H
'Content-type:text/xml; charset=utf-8'
curl http://solr_url/update --data-binary '<commit/>' -H
'Content-type:text/xml; charset=utf-8'

where feeds.xml contains the document in xml format

we have master and slave replication for solr server.

updates happens in master , snappuller and snapinstaller is run on
slaves periodically
queries don't happen at master , only happens at slaves

is there any thing which can be said from above information ?

Thanks,
Bharath

-----Original Message-----
From: kalidoss [mailto:kalidoss.muthuramalingam@...]
Sent: Tue 12/1/2009 2:38 PM
To: solr-user@...
Subject: Re: latency in solr response is observed after index is updated

r u using solr-config for committing the document?

bharath venkatesh wrote:

This message is intended only for the use of the addressee and may contain information that is privileged, confidential
and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the
employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any
dissemination, distribution or copying of this communication is strictly prohibited. If you have received this e-mail
in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from your system.

Re: Retrieving large num of docs

2009-12-01T01:47:56Z

Hi Hoss/Andrew,
I think I solved the problem of retrieving 300 docs per request for now. The
problem was that I was storing 2 moderately large multivalued text fields
though I was not retrieving them during search time. I reindexed all my
data without storing these fields. Now the response time (time for Solr to
return the http response) is very close to the QTime Solr is showing in the
logs.

Thanks for all the help,
Raghu

On Mon, Nov 30, 2009 at 11:37 AM, Raghuveer Kancherla <
raghuveer.kancherla@...> wrote:

> Thanks Hoss,
> In my previous mail, I was measuring the system time difference between
> sending a (http) request and receiving a response. This was being run on a
> (different) client machine
>
> Like you suggested, I tried to time the response on the server itself as
> follows:
>
> $ /usr/bin/time -p curl -sS -o solr.out "
> http://localhost:1212/solr/select/?rows=300&q=%28ResumeAllText%3A%28%28%28%22java+j2ee%22+%28java+j2ee%29%29%29%5E4%29%5E1.0%29&start=0&wt=python
> "
> real 3.49
>
> user 0.00
> sys 0.00
>
> The query time in solr log shows me Qtime=600
> size of solr.out is 843 kB.
>
> As you've mentioned, Solr shouldn't give these kind of numbers for 300
> docs, and we're quite perplexed as to whats going on.
>
> Thanks,
> Raghu
>
>
>
>
> On Mon, Nov 30, 2009 at 6:00 AM, Chris Hostetter <hossman_lucene@...
> > wrote:
>
>>
>> : I am using Solr1.4 for searching through half a million documents. The
>> : problem is, I want to retrieve nearly 200 documents for each search
>> query.
>> : The query time in Solr logs is showing 0.02 seconds and I am fairly
>> happy
>> : with that. However Solr is taking a long time (4 to 5 secs) to return
>> the
>> : results (I think it is because of the number of docs I am requesting). I
>> : tried returning only the id's (unique key) without any other stored
>> fields,
>> : but it is not helping me improve the response times (time to return the
>> id's
>> : of matching documents).
>>
>> What exactly does your request URL look like, and how exactly are you
>> timing the total response time?
>>
>> 200 isn't a very big number for the rows param -- people who want to get
>> 100K documents back in their response at a time may have problems, but 200
>> is not that big.
>>
>> so like i said: how exactly are you timing things?
>>
>> My guess: it's more likely that network overhead or the performance of
>> your client code (reading the data off the wire) is causing your timing
>> code to seem slow, then it is that Solr is taking 5 seconds to write out
>> those document IDs.
>>
>> I suspect if you try hitting the same exact URL using curl via localhost,
>> you'll see the total response time be a lot less then 5 seconds.
>>
>> Here's an example of a query that asks solr to return *every* field from
>> 500 documents, in the XML format. And these are not small documents...
>>
>> $ /usr/bin/time -p curl -sS -o /tmp/solr.out "
>> http://localhost:5051/solr/select/?q=doctype:product&version=2.2&start=0&rows=500&indent=on
>> "
>> real 0.07
>> user 0.00
>> sys 0.00
>> [chrish@c18-ssa-so-dfll-qry1 ~]$ du -sh /tmp/solr.out
>> 1.6M /tmp/solr.out
>>
>> ...that's 1.6 MB of 500 Solr documents with all of their fields in
>> verbose XML format (including indenting) fetched in 70ms.
>>
>> If it's taking 5 seconds for you to get just the ids of 200 docs, you've
>> got a problem somewhere and i'm 99% certain it's not in Solr.
>>
>> what does a similar "time curl" command for your URL look like when you
>> run it on your solr server?
>>
>>
>> -Hoss
>>
>>
>

Re: latency in solr response is observed after index is updated

2009-12-01T01:08:29Z

r u using solr-config for committing the document?

bharath venkatesh wrote:

> Hi,
>
> We are observing latency (some times huge latency upto 10-20 secs)
> in solr response after index is updated . whats the reason of this
> latency and how can it be minimized ?
> Note: our index size is pretty large.
>
> any help would be appreciated as we largely affected by it
>
> Thanks in advance.
> Bharath
> This message is intended only for the use of the addressee and may
> contain information that is privileged, confidential and exempt from
> disclosure under applicable law. If the reader of this message is not
> the intended recipient, or the employee or agent responsible for
> delivering the message to the intended recipient, you are hereby
> notified that any dissemination, distribution or copying of this
> communication is strictly prohibited. If you have received this e-mail
> in error, please notify us immediately by return e-mail and delete
> this e-mail and all attachments from your system.
>
>

Get your world in your inbox!

Mail, widgets, documents, spreadsheets, organizer and much more with your Sifymail WIYI id!
Log on to http://www.sify.com

********** DISCLAIMER **********
Information contained and transmitted by this E-MAIL is proprietary to
Sify Limited and is intended for use only by the individual or entity to
which it is addressed, and may contain information that is privileged,
confidential or exempt from disclosure under applicable law. If this is a
forwarded message, the content of this E-MAIL may not have been sent with
the authority of the Company. If you are not the intended recipient, an
agent of the intended recipient or a person responsible for delivering the
information to the named recipient, you are notified that any use,
distribution, transmission, printing, copying or dissemination of this
information in any way or in any manner is strictly prohibited. If you have
received this communication in error, please delete this mail & notify us
immediately at admin@...

latency in solr response is observed after index is updated

2009-12-01T01:04:17Z

Hi,

We are observing latency (some times huge latency upto 10-20 secs)
in solr response after index is updated . whats the reason of this
latency and how can it be minimized ?

Note: our index size is pretty large.

any help would be appreciated as we largely affected by it

Thanks in advance.
Bharath
This message is intended only for the use of the addressee and may contain information that is privileged, confidential
and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the
employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any
dissemination, distribution or copying of this communication is strictly prohibited. If you have received this e-mail
in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from your system.

Re: 'Connection reset' in DataImportHandler Development Console

2009-12-01T00:41:19Z

The debug tool for DIH dires queries in sync mode. it waits fro the
import to complete for the page to show up. If the process takes long
you r likely to see the connection reset message.

For get about debug. what exactly do you want to do?

2009/8/17 Noble Paul നോബിള്‍ नोब्ळ् <noble.paul@...>:

> apparently I do not see any command full-import, delta-import being
> fired. Is that true?
>
> On Mon, Aug 17, 2009 at 5:55 PM, Andrew Clegg<andrew.clegg@...> wrote:
>>
>> Hi folks,
>>
>> I'm trying to use the Debug Now button in the development console to test
>> the effects of some changes in my data import config (see attached).
>>
>> However, each time I click it, the right-hand frame fails to load -- it just
>> gets replaced with the standard 'connection reset' message from Firefox, as
>> if the server's dropped the HTTP connection.
>>
>> Everything else seems okay -- I can run queries in Solr Admin without any
>> problems, and all the other buttons in the dev console work -- status,
>> document count, reload config etc.
>>
>> There's nothing suspicious in Tomcat's catalina.out either. If I hit Reload
>> Config, then Status, then Debug Now, I get this:
>>
>>
>> 17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImportHandler
>> processConfiguration
>> INFO: Processing configuration from solrconfig.xml: {config=dataconfig.xml}
>> 17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImporter
>> loadDataConfig
>> INFO: Data Configuration loaded successfully
>> 17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImporter
>> verifyWithSchema
>> INFO: id is a required field in SolrSchema . But not found in DataConfig
>> 17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImporter
>> verifyWithSchema
>> INFO: title is a required field in SolrSchema . But not found in DataConfig
>> 17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImporter
>> verifyWithSchema
>> INFO: doc_type is a required field in SolrSchema . But not found in
>> DataConfig
>> 17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImporter
>> verifyWithSchema
>> INFO: id is a required field in SolrSchema . But not found in DataConfig
>> 17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImporter
>> verifyWithSchema
>> INFO: title is a required field in SolrSchema . But not found in DataConfig
>> 17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImporter
>> verifyWithSchema
>> INFO: doc_type is a required field in SolrSchema . But not found in
>> DataConfig
>> 17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImporter
>> verifyWithSchema
>> INFO: id is a required field in SolrSchema . But not found in DataConfig
>> 17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImporter
>> verifyWithSchema
>> INFO: title is a required field in SolrSchema . But not found in DataConfig
>> 17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImporter
>> verifyWithSchema
>> INFO: doc_type is a required field in SolrSchema . But not found in
>> DataConfig
>> 17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImporter
>> verifyWithSchema
>> INFO: id is a required field in SolrSchema . But not found in DataConfig
>> 17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImporter
>> verifyWithSchema
>> INFO: title is a required field in SolrSchema . But not found in DataConfig
>> 17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImporter
>> verifyWithSchema
>> INFO: doc_type is a required field in SolrSchema . But not found in
>> DataConfig
>> 17-Aug-2009 13:12:12 org.apache.solr.core.SolrCore execute
>> INFO: [] webapp=/solr path=/select
>> params={clean=false&command=reload-config&commit=true&qt=/dataimport}
>> status=0 QTime=5
>> 17-Aug-2009 13:12:21 org.apache.solr.core.SolrCore execute
>> INFO: [] webapp=/solr path=/select
>> params={clean=false&command=status&commit=true&qt=/dataimport} status=0
>> QTime=0
>>
>>
>> (The warnings are because the doc_type field comes out of the JDBC result
>> set automatically by column name -- this isn't a problem.)
>>
>> Also, there's no entry in the Tomcat access log for the debug request
>> either, just the first two:
>>
>>
>> [17/Aug/2009:13:12:12 +0100] HTTP/1.1 cookie:- request:- GET /solr/select
>> 200 ?clean=false&commit=true&qt=%2Fdataimport&command=reload-config GET
>> /solr/select?clean=false&commit=t
>> rue&qt=%2Fdataimport&command=reload-config HTTP/1.1
>> [17/Aug/2009:13:12:21 +0100] HTTP/1.1 cookie:- request:- GET /solr/select
>> 200 ?clean=false&commit=true&qt=%2Fdataimport&command=status GET
>> /solr/select?clean=false&commit=true&qt=
>> %2Fdataimport&command=status HTTP/1.1
>>
>>
>> PS... Nightly build, 30th of July.
>>
>> Thanks,
>>
>> Andrew.
>>
>> http://www.nabble.com/file/p25005850/dataconfig.xml dataconfig.xml
>> --
>> View this message in context: http://www.nabble.com/%27Connection-reset%27-in-DataImportHandler-Development-Console-tp25005850p25005850.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> -----------------------------------------------------
> Noble Paul | Principal Engineer| AOL | http://aol.com
>

--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com

Re: Thought that masterUrl in slave solrconfig.xml is optional...

2009-12-01T00:35:26Z

remove the slave section completely and startup will go thru fine

On Tue, Dec 1, 2009 at 2:47 AM, William Pierce <evalsinca@...> wrote:

> Folks:
>
> Reading the wiki, I saw the following statement:
> "Force a fetchindex on slave from master command : http://slave_host:port/solr/replication?command=fetchindex
>
> It is possible to pass on extra attribute 'masterUrl' or other attributes like 'compression' (or any other parameter which is specified in the <lst name="slave"> tag) to do a one time replication from a master. This obviates the need for hardcoding the master in the slave. "
> In my case, I cannot hardcode the masterurl in the config file. I want a cron job to issue the replication commands for each of the slaves.
>
> So I issued the following command:
>
> http://localhost/postings/replication?command=fetchIndex&masterUrl=http%3a%2f%2flocalhost%2fpostingsmaster
>
> I got the following exception:
>
> HTTP Status 500 - Severe errors in solr configuration. Check your log files for more detailed information on what may be wrong. If you want solr to continue after configuration errors, change: <abortOnConfigurationError>false</abortOnConfigurationError> in null -------------------------------------------------------------
>
> org.apache.solr.common.SolrException: 'masterUrl' is required for a slave at org.apache.solr.handler.SnapPuller.<init>(SnapPuller.java:126) at
>
> ....other lines removed....
>
> Why is error message asking me to specify the masterUrl in the config file when the wiki states that this is optional? Or, am I understanding this incorrectly?
>
> Thanks,
>
> - Bill
>
>
>

--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com

Re: How to avoid hardcoding masterUrl in slave solrconfig.xml?

2009-11-30T22:20:20Z

remove the <lst name="slave"> section from your solrconfig. It should be fine

On Tue, Dec 1, 2009 at 6:59 AM, William Pierce <evalsinca@...> wrote:

> Hi, Joe:
>
> I tried with the "fetchIndex" all lower-cased, and still the same result.
> What do you specify for masterUrl in the solrconfig.xml on the slave? it
> seems to me that if I remove the element, I get the exception I wrote
> about. If I set it to some dummy url, then I get an invalid url message
> when I run the command=details on the slave replication handler.
>
> What I am doing does not look out of the ordinary. I want to control the
> masterurl and the time of replication by myself. As such I want neither the
> masterUrl nor the polling interval in the config file. Can you share
> relevant snippets of your config file and the exact url your code is
> generating?
>
> Thanks,
>
> - Bill
>
> --------------------------------------------------
> From: "Joe Kessel" <isjustjoe@...>
> Sent: Monday, November 30, 2009 3:45 PM
> To: <solr-user@...>
> Subject: RE: How to avoid hardcoding masterUrl in slave solrconfig.xml?
>
>>
>> I do something very similar and it works for me. I noticed on your URL
>> that you have a mixed case fetchIndex, which the request handler is checking
>> for fetchindex, all lowercase. If it is not that simple I can try to see
>> the exact url my code is generating.
>>
>>
>>
>> Hope it helps,
>>
>> Joe
>>
>>> From: evalsinca@...
>>> To: solr-user@...
>>> Subject: Re: How to avoid hardcoding masterUrl in slave solrconfig.xml?
>>> Date: Mon, 30 Nov 2009 13:48:38 -0800
>>>
>>> Folks:
>>>
>>> Sorry for this repost! It looks like this email went out twice....
>>>
>>> Thanks,
>>>
>>> - Bill
>>>
>>> --------------------------------------------------
>>> From: "William Pierce" <evalsinca@...>
>>> Sent: Monday, November 30, 2009 1:47 PM
>>> To: <solr-user@...>
>>> Subject: How to avoid hardcoding masterUrl in slave solrconfig.xml?
>>>
>>> > Folks:
>>> >
>>> > I do not want to hardcode the masterUrl in the solrconfig.xml of my >
>>> > slave.
>>> > If the masterUrl tag is missing from the config file, I am getting an
>>> > exception in solr saying that the masterUrl is required. So I set it to
>>> > some dummy value, comment out the poll interval element, and issue a
>>> > replication command manually like so:
>>> >
>>> >
>>> > http://localhost:port/postings/replication?command=fetchIndex&masterUrl=http://localhost:port/postingsmaster/replication
>>> >
>>> > Now no internal exception, solr responds with a status "OK" for the >
>>> > above
>>> > request, the tomcat logs show no error but the index is not replicated.
>>> > When I issue the details command to the slave, I see that it ignored >
>>> > the
>>> > masterUrl on the command line but instead complains that the master url
>>> > > in
>>> > the config file (which I had set to a dummy value) is not correct.
>>> >
>>> > (Just fyi, I have tried sending in the masterUrl to the above command >
>>> > with
>>> > url encoding and also without. in both cases, I got the same result.)
>>> >
>>> > Soooo....how exactly do I avoid hardcoding the masterUrl in the config
>>> > file? Any pointers/help will be greatly appreciated!
>>> >
>>> > - Bill
>>>
>>
>> _________________________________________________________________
>> Bing brings you maps, menus, and reviews organized in one place.
>>
>> http://www.bing.com/search?q=restaurants&form=MFESRP&publ=WLHMTAG&crea=TEXT_MFESRP_Local_MapsMenu_Resturants_1x1
>
>

--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com

Re: HTML Stripping slower in Solr 1.4?

2009-11-30T22:17:46Z

Robin,

Thank you for reporting this. Performance degradation of HTML Stripper
could be in 1.4. I opened a ticket in Lucene:

https://issues.apache.org/jira/browse/LUCENE-2098

Koji

--
http://www.rondhuit.com/en/

Re: Phrase Query Issue

2009-11-30T21:13:44Z

Thanks Erick But their any way to get phrase search and normal search together.

for Example if I search for "test" it should only give test result but not tests..
And if we search for tests it should give all results matching with tests(this should include stemmer results also)

Please clarify me how normally this scenario is implemented?

I am using field title with text type which contains SnowballPorterFilterFactory filter.

And another field title_exact with out SnowballPorterFilterFactory filter.

Both are copied using copy field.

<copyField source="title" dest="text"/>
<copyField source="title_exact" dest="text_exact"/>
But as text type is having a SnowballPorterFilterFactory filter it stores tests as test.So always if I search for "test" tests also will come in my search result.
Please tell me a way to avoid it.

Thanks
Ravichandra

HTML Stripping slower in Solr 1.4?

2009-11-30T20:18:39Z

Hello,

Our schema in Sol 1.3 looked like:

<tokenizer class="solr.HTMLStripStandardTokenizerFactory"/>
<filter class="solr.StandardFilterFactory"/>

It takes 30s to index 1500 docs. When we run the same in Sol 1.4 it take 70s.

I noticed that HTMLStripStandardTokenizerFactory was deprecated. So
changed the schema to:
<charFilter class="solr.HTMLStripCharFilterFactory"/>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StandardFilterFactory"/>

It still takes 70s.

Instead, if I use the schema:
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StandardFilterFactory"/>

It takes 30s in both 1.3 and 1.4.

I am not sure if HTMLStrip has become slower in 1.4 or HTML stripping
impacts perf down stream in 1.4. Before I started writing a unit test
with a TokenizerChain, I wanted to check if I am doing something
fundamentally wrong.

Robin

Re: : Returning no results

2009-11-30T20:15:29Z

Are you sure you're hitting the same core? Did a commit? Are you
possibly using the dismax query parser (where *:* is fairly
meaningless)?

Erik

On Nov 30, 2009, at 3:54 PM, Giovanni Fernandez-Kincade wrote:

> Hi,
> I created a brand new core (on Solr 1.4), added a few documents and
> then searched for *:*, but got no results. Strangely enough, if I
> search for a specific document I know is in the index, like say
> "versionId:3", I get the expected result.
>
> Any ideas on why that might be?
>
> Thank,
> Gio.

Re: nested solr queries

2009-11-30T17:49:03Z

: thanks for your help so do you think I should execute solr queries twice ?
: or is there any other workarounds

http://people.apache.org/~hossman/#xyproblem
XY Problem

Your question appears to be an "XY Problem" ... that is: you are dealing
with "X", you are assuming "Y" will help you, and you are asking about "Y"
without giving more details about the "X" so that we can understand the
full issue. Perhaps the best solution doesn't involve "Y" at all?
See Also: http://www.perlmonks.org/index.pl?node_id=542341

-Hoss

Re: Sorting Facets by First Occurrence

2009-11-30T17:33:16Z

On Nov 30, 2009, at 5:15 PM, Chris Hostetter wrote:

> All of Solr's existing faceting code is based on the DocSet which is an
> unordered set of all matching documents -- i suspect your existing
> application only reordered the facets based on their appearance in the
> first N docs (possibly just the first page, but maybe more) so doing
> something like that using the DocList would certainly be feasible. if
> your number of facet constraints is low enough that they are all returned
> everytime then doing it in the client is probably the easiest -- but if
> you have to worry about facet.limit preventing something from being
> returned that might otherwise bubble to the top of your list when you
> reorder it then you'll need to customise the FacetComponent.

You are right, I left out a few important bits there. Tried to be brief and succeeding in being vague? :)

Effectively I was ordering the facet based on the N documents in the current "page". My thought that his was a good feature for a facet now seems incorrect, as my needs are limited to the current page, not the whole set of results.

I'll probably elect to fetching data from the facets based on the page of documents I'm showing. Thanks for the discussion, it helped! :)

Cory G Watson
http://www.onemogin.com

Re: How to avoid hardcoding masterUrl in slave solrconfig.xml?

2009-11-30T17:29:33Z

Hi, Joe:

I tried with the "fetchIndex" all lower-cased, and still the same result.
What do you specify for masterUrl in the solrconfig.xml on the slave? it
seems to me that if I remove the element, I get the exception I wrote
about. If I set it to some dummy url, then I get an invalid url message
when I run the command=details on the slave replication handler.

What I am doing does not look out of the ordinary. I want to control the
masterurl and the time of replication by myself. As such I want neither the
masterUrl nor the polling interval in the config file. Can you share
relevant snippets of your config file and the exact url your code is
generating?

Thanks,

- Bill

--------------------------------------------------
From: "Joe Kessel" <isjustjoe@...>
Sent: Monday, November 30, 2009 3:45 PM
To: <solr-user@...>
Subject: RE: How to avoid hardcoding masterUrl in slave solrconfig.xml?

>
> I do something very similar and it works for me. I noticed on your URL
> that you have a mixed case fetchIndex, which the request handler is
> checking for fetchindex, all lowercase. If it is not that simple I can
> try to see the exact url my code is generating.
>
>
>
> Hope it helps,
>
> Joe
>
>> From: evalsinca@...
>> To: solr-user@...
>> Subject: Re: How to avoid hardcoding masterUrl in slave solrconfig.xml?
>> Date: Mon, 30 Nov 2009 13:48:38 -0800
>>
>> Folks:
>>
>> Sorry for this repost! It looks like this email went out twice....
>>
>> Thanks,
>>
>> - Bill
>>
>> --------------------------------------------------
>> From: "William Pierce" <evalsinca@...>
>> Sent: Monday, November 30, 2009 1:47 PM
>> To: <solr-user@...>
>> Subject: How to avoid hardcoding masterUrl in slave solrconfig.xml?
>>
>> > Folks:
>> >
>> > I do not want to hardcode the masterUrl in the solrconfig.xml of my
>> > slave.
>> > If the masterUrl tag is missing from the config file, I am getting an
>> > exception in solr saying that the masterUrl is required. So I set it to
>> > some dummy value, comment out the poll interval element, and issue a
>> > replication command manually like so:
>> >
>> > http://localhost:port/postings/replication?command=fetchIndex&masterUrl=http://localhost:port/postingsmaster/replication
>> >
>> > Now no internal exception, solr responds with a status "OK" for the
>> > above
>> > request, the tomcat logs show no error but the index is not replicated.
>> > When I issue the details command to the slave, I see that it ignored
>> > the
>> > masterUrl on the command line but instead complains that the master url
>> > in
>> > the config file (which I had set to a dummy value) is not correct.
>> >
>> > (Just fyi, I have tried sending in the masterUrl to the above command
>> > with
>> > url encoding and also without. in both cases, I got the same result.)
>> >
>> > Soooo....how exactly do I avoid hardcoding the masterUrl in the config
>> > file? Any pointers/help will be greatly appreciated!
>> >
>> > - Bill
>>
>
> _________________________________________________________________
> Bing brings you maps, menus, and reviews organized in one place.
> http://www.bing.com/search?q=restaurants&form=MFESRP&publ=WLHMTAG&crea=TEXT_MFESRP_Local_MapsMenu_Resturants_1x1

RE: How to avoid hardcoding masterUrl in slave solrconfig.xml?

2009-11-30T15:45:59Z

I do something very similar and it works for me. I noticed on your URL that you have a mixed case fetchIndex, which the request handler is checking for fetchindex, all lowercase. If it is not that simple I can try to see the exact url my code is generating.

Hope it helps,

Joe

> From: evalsinca@...
> To: solr-user@...
> Subject: Re: How to avoid hardcoding masterUrl in slave solrconfig.xml?
> Date: Mon, 30 Nov 2009 13:48:38 -0800
>
> Folks:
>
> Sorry for this repost! It looks like this email went out twice....
>
> Thanks,
>
> - Bill
>
> --------------------------------------------------
> From: "William Pierce" <evalsinca@...>
> Sent: Monday, November 30, 2009 1:47 PM
> To: <solr-user@...>
> Subject: How to avoid hardcoding masterUrl in slave solrconfig.xml?
>
> > Folks:
> >
> > I do not want to hardcode the masterUrl in the solrconfig.xml of my slave.
> > If the masterUrl tag is missing from the config file, I am getting an
> > exception in solr saying that the masterUrl is required. So I set it to
> > some dummy value, comment out the poll interval element, and issue a
> > replication command manually like so:
> >
> > http://localhost:port/postings/replication?command=fetchIndex&masterUrl=http://localhost:port/postingsmaster/replication
> >
> > Now no internal exception, solr responds with a status "OK" for the above
> > request, the tomcat logs show no error but the index is not replicated.
> > When I issue the details command to the slave, I see that it ignored the
> > masterUrl on the command line but instead complains that the master url in
> > the config file (which I had set to a dummy value) is not correct.
> >
> > (Just fyi, I have tried sending in the masterUrl to the above command with
> > url encoding and also without. in both cases, I got the same result.)
> >
> > Soooo....how exactly do I avoid hardcoding the masterUrl in the config
> > file? Any pointers/help will be greatly appreciated!
> >
> > - Bill
>

_________________________________________________________________
Bing brings you maps, menus, and reviews organized in one place.
http://www.bing.com/search?q=restaurants&form=MFESRP&publ=WLHMTAG&crea=TEXT_MFESRP_Local_MapsMenu_Resturants_1x1

Re: Sorting Facets by First Occurrence

2009-11-30T15:15:42Z

: I'm working on replacing a custom, internal search implementation with
: Solr. I'm having great success, with one small exception.
...
: For example, if a search yielded 10 results, 1 - 10, and hit 1 is in
: category 'Toys', hit 2 through 9 are in 'Sports' and the last is in
: 'Household' then the facet would look like:

...and what if you had 100,000 results? did it really look at every doc
that matched a query to decide the facet ordering, or did it stop at 10?

: So, the question I _really_ have is: how can I implement this feature?

It depends ... how was the previous version implemented?

(you mentioned it was a custom internal solution, so i assume you have
access to the code and can explain it to us in psuedo code ... that would
give people the best insight into what exactly it was doing to make a
Solr based comparison)

: I could examine the results i'm returned and create my own facet
: order from it, but I thought this might be useful for others. I don't
: know my way around Solr's source, so I though dropping a note to the
: list would be faster than code spelunking with no light.

All of Solr's existing faceting code is based on the DocSet which is an
unordered set of all matching documents -- i suspect your existing
application only reordered the facets based on their appearance in the
first N docs (possibly just the first page, but maybe more) so doing
something like that using the DocList would certainly be feasible. if
your number of facet constraints is low enough that they are all returned
everytime then doing it in the client is probably the easiest -- but if
you have to worry about facet.limit preventing something from being
returned that might otherwise bubble to the top of your list when you
reorder it then you'll need to customise the FacetComponent.

-Hoss

RE: schema-based Index-time field boosting

2009-11-30T15:04:54Z

: I am talking about field boosting rather than document boosting, ie. I
: would like some fields (say eg. title) to be "louder" than others,
: across ALL documents. I believe you are at least partially talking
: about document boosting, which clearly applies on a per-document basis.

index time boosts are all the same -- it doesn't matter if htey are field
boosts or document boosts -- a document boost is just a field boost for
every field in the document.

: If it helps, consider a schema version of the following, from
: org.apache.solr.common.SolrInputDocument:
:
: /**
: * Adds a field with the given name, value and boost. If a field with
: the name already exists, then it is updated to
: * the new value and boost.
: *
: * @param name Name of the field to add
: * @param value Value of the field
: * @param boost Boost value for the field
: */
: public void addField(String name, Object value, float boost )

...

: Where a constant boost value is applied consistently to a given field.
: That is what I was mistakenly hoping to achieve in the schema. I still
: think it would be a good idea BTW. Regards,

But now we're right back to what i was trying to explain before: index
time boost values like these are only used as a multiplier in the
fieldNorm. when included as part of the document data, you can specify a
fieldBoost for fieldX of docA that's greater then the boost for fieldX
of docB and that will make docA score higher then docB when fieldX
contains the same number of matches and is hte same length -- but if you
apply a constant boost of B to fieldX for every doc (which is what a
feature to hardcode boosts in schema.xml might give you) then the net
effect would be zero when scoring docA and docB, because the fieldNorm's
for fieldX in both docs would include the exact same multiplier.

-Hoss

Re: schema-based Index-time field boosting

2009-11-30T14:58:41Z

: Coming in a bit late but I would like a variant that is not a No-OP.
: Think of something like title:searchstring^10 OR catch_all:searchstring
: Of course I can always add the boosting at query time but it would make
: life easier if I could define a default boost in the schema so that my
: query could just be title:searchstring OR catch_all:searchstring
: but still get the boost for the title field.

That would be a query time boost -- not an index time boost. An index
time boost is a very specific concept that refers to increaseing the
fieldNorm of a field in a specific document.

What you are describing would be a query time boost, in which solr knows
that certain query clauses should be worth more. You can
already do esentailly this exact same thing using the dismax parser, but i
suppose it would also be possible to modify the LuceneQParser so that it
could take in configuration that would tell it to apply different default
query boosts to different fields.

-Hoss

Re: $DeleteDocbyQuery in solr 1.4 is not working

2009-11-30T14:34:14Z

Ok, I think I figured out what might be happening. It appears that the DataImporter issues the commit command without the expungeDeletes option set to true (default in 1.4 for a commit command is for expungeDeletes to be set to false). You can get around this by issuing the commit command manually:

curl "http://localhost:8983/solr/dataimport?command=delta-import&commit=false&optimize=false"
curl "http://localhost:8983/solr/update" --data-binary '<commit expungeDeletes="true" />' -H "Content-type:text/xml; charset=utf-8"

That fixed the issue for me, although ideally the DataImportHandler should run with the expungeDeletes option set to true, so that this could all happen in one command without having to wait for the DIH to finish before issuing the commit command.

Is there a way to set the expungeDeletes option for the commit command of the DataImportHandler, so that delta imports with deletes can happen automatically?

Mark.El wrote:

Thanks I will look into it!

Re: How to avoid hardcoding masterUrl in slave solrconfig.xml?

2009-11-30T13:48:38Z

Folks:

Sorry for this repost! It looks like this email went out twice....

Thanks,

- Bill

--------------------------------------------------
From: "William Pierce" <evalsinca@...>
Sent: Monday, November 30, 2009 1:47 PM
To: <solr-user@...>
Subject: How to avoid hardcoding masterUrl in slave solrconfig.xml?

> Folks:
>
> I do not want to hardcode the masterUrl in the solrconfig.xml of my slave.
> If the masterUrl tag is missing from the config file, I am getting an
> exception in solr saying that the masterUrl is required. So I set it to
> some dummy value, comment out the poll interval element, and issue a
> replication command manually like so:
>
> http://localhost:port/postings/replication?command=fetchIndex&masterUrl=http://localhost:port/postingsmaster/replication
>
> Now no internal exception, solr responds with a status "OK" for the above
> request, the tomcat logs show no error but the index is not replicated.
> When I issue the details command to the slave, I see that it ignored the
> masterUrl on the command line but instead complains that the master url in
> the config file (which I had set to a dummy value) is not correct.
>
> (Just fyi, I have tried sending in the masterUrl to the above command with
> url encoding and also without. in both cases, I got the same result.)
>
> Soooo....how exactly do I avoid hardcoding the masterUrl in the config
> file? Any pointers/help will be greatly appreciated!
>
> - Bill

How to avoid hardcoding masterUrl in slave solrconfig.xml?

2009-11-30T13:47:10Z

Folks:

I do not want to hardcode the masterUrl in the solrconfig.xml of my slave. If the masterUrl tag is missing from the config file, I am getting an exception in solr saying that the masterUrl is required. So I set it to some dummy value, comment out the poll interval element, and issue a replication command manually like so:

http://localhost:port/postings/replication?command=fetchIndex&masterUrl=http://localhost:port/postingsmaster/replication

Now no internal exception, solr responds with a status "OK" for the above request, the tomcat logs show no error but the index is not replicated. When I issue the details command to the slave, I see that it ignored the masterUrl on the command line but instead complains that the master url in the config file (which I had set to a dummy value) is not correct.

(Just fyi, I have tried sending in the masterUrl to the above command with url encoding and also without. in both cases, I got the same result.)

Soooo....how exactly do I avoid hardcoding the masterUrl in the config file? Any pointers/help will be greatly appreciated!

- Bill

Sorting Facets by First Occurrence

2009-11-30T13:37:54Z

I'm working on replacing a custom, internal search implementation with
Solr. I'm having great success, with one small exception.

When implementing our version of faceting, one of our facets had a
peculiar sort order. It was dictated by the order in which the field
occurred in the results. The first time a value occurred it was added
to the list and regardless of the number of times it occurred, it
always stayed at the top.

For example, if a search yielded 10 results, 1 - 10, and hit 1 is in
category 'Toys', hit 2 through 9 are in 'Sports' and the last is in
'Household' then the facet would look like:

facet.fields -> category -> [ Toys: 1, Sports: 8 Household: 1 ]

The facet.sort only gives me the option to sort highest count first or
alphabetically.

So, the question I _really_ have is: how can I implement this feature?
I could examine the results i'm returned and create my own facet
order from it, but I thought this might be useful for others. I don't
know my way around Solr's source, so I though dropping a note to the
list would be faster than code spelunking with no light.

--
Cory 'G' Watson
http://www.onemogin.com

Nabble - Solr - User

Re: Response writer configs

Response writer configs

question about schemas

Re: Maximum number of fields allowed in a Solr document

Re: Creating Facets

NYC Search & Discovery Meetup

Re: Where to put ExternalRequestHandler and Tika jars

Re: latency in solr response is observed after index is updated

Re: getting total index size & last update date/time from query

Re: getting total index size & last update date/time from query

Re: getting total index size & last update date/time from query

Re: How to avoid hardcoding masterUrl in slave solrconfig.xml?

Re: Phrase Query Issue

RE: latency in solr response is observed after index is updated

Re: Retrieving large num of docs

Re: latency in solr response is observed after index is updated

latency in solr response is observed after index is updated

Re: 'Connection reset' in DataImportHandler Development Console

Re: Thought that masterUrl in slave solrconfig.xml is optional...

Re: How to avoid hardcoding masterUrl in slave solrconfig.xml?

Re: HTML Stripping slower in Solr 1.4?

Re: Phrase Query Issue

HTML Stripping slower in Solr 1.4?

Re: *:* Returning no results

Re: nested solr queries

Re: Sorting Facets by First Occurrence

Re: How to avoid hardcoding masterUrl in slave solrconfig.xml?

RE: How to avoid hardcoding masterUrl in slave solrconfig.xml?

Re: Sorting Facets by First Occurrence

RE: schema-based Index-time field boosting

Re: schema-based Index-time field boosting

Re: $DeleteDocbyQuery in solr 1.4 is not working

Re: How to avoid hardcoding masterUrl in slave solrconfig.xml?

How to avoid hardcoding masterUrl in slave solrconfig.xml?

Sorting Facets by First Occurrence

Re: : Returning no results