Nabble - Solr

Re: Question about the message "Indexing failed. Rolled back all changes."

2009-11-10T00:02:41Z

No. I did not check the logs.

But even after I successfully index data using
http://host:port/solr-example/dataimport?command=full-import&commit=true&clean=true,
do solr search which returns meaningful results, and then visit
http://host:port/solr-example/dataimport?command=status, I can see the
following result

<response>
-
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">1</int>
</lst>
-
<lst name="initArgs">
-
<lst name="defaults">
<str name="config">data-config.xml</str>
</lst>
</lst>
<str name="command">status</str>
<str name="status">idle</str>
<str name="importResponse"/>
-
<lst name="statusMessages">
<str name="Time Elapsed">0:2:11.426</str>
<str name="Total Requests made to DataSource">584</str>
<str name="Total Rows Fetched">1538</str>
<str name="Total Documents Skipped">0</str>
<str name="Full Dump Started">2009-11-09 23:54:41</str>
*<str name="">Indexing failed. Rolled back all changes.</str>*
<str name="Committed">2009-11-09 23:54:42</str>
<str name="Optimized">2009-11-09 23:54:42</str>
<str name="Rolledback">2009-11-09 23:54:42</str>
</lst>
-
<str name="WARNING">
This response format is experimental. It is likely to change in the future.
</str>
</response>

On Mon, Nov 9, 2009 at 7:39 AM, Shalin Shekhar Mangar <
shalinmangar@...> wrote:

> On Sat, Nov 7, 2009 at 1:10 PM, Bertie Shen <bertie.shen@...> wrote:
>
> >
> > When I use
> > http://localhost:8180/solr/admin/dataimport.jsp?handler=/dataimport to
> > debug
> > the indexing config file, I always see the status message on the right
> part
> > <str name="">Indexing failed. Rolled back all changes.</str>, even the
> > indexing process looks to be successful. I am not sure whether you guys
> > have
> > seen the same phenomenon or not. BTW, I usually check the checkbox Clean
> > and sometimes check Commit box, and then click Debug Now button.
> >
> >
> Do you see any exceptions in the logs?
>
> --
> Regards,
> Shalin Shekhar Mangar.
>

From forum: Solr - User

Re: How TEXT field make sortable?

2009-11-09T23:36:31Z

>
> Can some one help me how we can sort the text field.
>
You CANNOT sort on a "text" field. Sorting can only be done on an
untokenized field (e.g string, sint, sfloat etc fields)

Cheers
Avlesh

On Tue, Nov 10, 2009 at 11:44 AM, deepak agrawal <dk.agwl@...> wrote:

> Can some one help me how we can sort the text field.
>
> <field name="TITLE" type="text" indexed="true" stored="true"/>
>
> --
> DEEPAK AGRAWAL
> +91-9379433455
> GOOD LUCK.....
>

From forum: Solr - User

Re: A question about how to make schema.xml change take effect

2009-11-09T23:15:13Z

Oh. Sorry, take back what I said. Most of my config change is at
data-config.xml, not schema.xml.

I just made a change for field data type in schema.xml and noticed that I
have to restart tomcat.

On Mon, Nov 9, 2009 at 10:37 PM, Ritesh Gurung <ritesh@...> wrote:

> Well everytime you make change in schema.xml file you need restart the
> tomcat server.
>
> On Tue, Nov 10, 2009 at 11:59 AM, Bertie Shen <bertie.shen@...>
> wrote:
> > Hey folks,
> >
> > When I update schema.xml, I found most of time I do not need to restart
> > tomcat in order to make change take effect. But sometimes, I have to
> restart
> > tomcat server to make change take effect.
> >
> > For example, when I change a field data type from sint to tlong, I
> called
> > http://host:port
> /solr/dataimport?command=full-import&commit=true&clean=true.
> > I clicked [Schema] link from admin page and found data type is tlong; but
> > click [Schema Browser] and that field link, I found the data type is
> still
> > sint. When I make a search, the result also shows the field is still
> sint.
> > The only way to make the change effective I found is to restart tomcat.
> >
> > I want to confirm whether it is intended or it is a bug.
> >
> > Thanks.
> >
>

From forum: Solr - User

Re: A question about how to make schema.xml change take effect

2009-11-09T23:14:32Z

if your are using a multicore instance you may just reload the core

On Tue, Nov 10, 2009 at 12:07 PM, Ritesh Gurung <ritesh@...> wrote:

> Well everytime you make change in schema.xml file you need restart the
> tomcat server.
>
> On Tue, Nov 10, 2009 at 11:59 AM, Bertie Shen <bertie.shen@...> wrote:
>> Hey folks,
>>
>> When I update schema.xml, I found most of time I do not need to restart
>> tomcat in order to make change take effect. But sometimes, I have to restart
>> tomcat server to make change take effect.
>>
>> For example, when I change a field data type from sint to tlong, I called
>> http://host:port/solr/dataimport?command=full-import&commit=true&clean=true.
>> I clicked [Schema] link from admin page and found data type is tlong; but
>> click [Schema Browser] and that field link, I found the data type is still
>> sint. When I make a search, the result also shows the field is still sint.
>> The only way to make the change effective I found is to restart tomcat.
>>
>> I want to confirm whether it is intended or it is a bug.
>>
>> Thanks.
>>
>

--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com

From forum: Solr - User

Re: A question about how to make schema.xml change take effect

2009-11-09T22:37:37Z

Well everytime you make change in schema.xml file you need restart the
tomcat server.

On Tue, Nov 10, 2009 at 11:59 AM, Bertie Shen <bertie.shen@...> wrote:

> Hey folks,
>
> When I update schema.xml, I found most of time I do not need to restart
> tomcat in order to make change take effect. But sometimes, I have to restart
> tomcat server to make change take effect.
>
> For example, when I change a field data type from sint to tlong, I called
> http://host:port/solr/dataimport?command=full-import&commit=true&clean=true.
> I clicked [Schema] link from admin page and found data type is tlong; but
> click [Schema Browser] and that field link, I found the data type is still
> sint. When I make a search, the result also shows the field is still sint.
> The only way to make the change effective I found is to restart tomcat.
>
> I want to confirm whether it is intended or it is a bug.
>
> Thanks.
>

From forum: Solr - User

A question about how to make schema.xml change take effect

2009-11-09T22:29:28Z

Hey folks,

When I update schema.xml, I found most of time I do not need to restart
tomcat in order to make change take effect. But sometimes, I have to restart
tomcat server to make change take effect.

For example, when I change a field data type from sint to tlong, I called
http://host:port/solr/dataimport?command=full-import&commit=true&clean=true.
I clicked [Schema] link from admin page and found data type is tlong; but
click [Schema Browser] and that field link, I found the data type is still
sint. When I make a search, the result also shows the field is still sint.
The only way to make the change effective I found is to restart tomcat.

I want to confirm whether it is intended or it is a bug.

Thanks.

From forum: Solr - User

Re: Are subqueries possible in Solr? If so, are they performant?

2009-11-09T22:23:49Z

Thanks Otis for your response.

Is it possible to get result of one solr query feed into another Solr Query?

Issue which I am facing right now is::
I am getting results from one query and I just need 2 index attribute values . These index attribute values are used for form new Query to Solr.

Since Solr gives result only for GET request, hence there is restriction on : forming query with all values.

Please do send your views on above problem

Thanks
~Vikrant

Otis Gospodnetic wrote:

You can mimic them by combining 2 clauses with an AND.
e.g.
cookies
vs.
cookies AND vanilla

Otis
--
Sematext is hiring -- http://sematext.com/about/jobs.html?mls
Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR

----- Original Message ----
> From: Vicky_Dev <vikrantv_shirbhate@yahoo.co.in>
> To: solr-user@lucene.apache.org
> Sent: Mon, November 9, 2009 1:48:03 PM
> Subject: Re: Are subqueries possible in Solr? If so, are they performant?
>
>
>
> Hi Team,
> Is it possible to write subqueries in dismaxrequest handler?
>
> ~Vikrant
>
>
> Edoardo Marcora wrote:
> >
> > Does Solr have the ability to do subqueries, like this one (in SQL):
> >
> > SELECT id, first_name
> > FROM student_details
> > WHERE first_name IN (SELECT first_name
> > FROM student_details
> > WHERE subject= 'Science');
> >
> > If so, how performant is this kind of queries?
> >
>
> --
> View this message in context:
> http://old.nabble.com/Are-subqueries-possible-in-Solr--If-so%2C-are-they-performant--tp24467023p26271600.html
> Sent from the Solr - User mailing list archive at Nabble.com.

From forum: Solr - User

Re: Are subqueries possible in Solr? If so, are they performant?

2009-11-09T22:23:05Z

Thanks Otis for your response. Is it possible to get result of one solr query feed into another Solr Query? Issue which I am facing right now is:: I am getting results from one query and I just need 2 index attribute values . These index attribute values are used for form new Query to Solr. Since Solr gives result only for GET request, hence there is restriction on : forming query with all values. Please do send your views on above problem Thanks ~Vikrant

Otis Gospodnetic wrote:

You can mimic them by combining 2 clauses with an AND. e.g. cookies vs. cookies AND vanilla Otis -- Sematext is hiring -- http://sematext.com/about/jobs.html?mls Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR ----- Original Message ---- > From: Vicky_Dev > To: solr-user@lucene.apache.org > Sent: Mon, November 9, 2009 1:48:03 PM > Subject: Re: Are subqueries possible in Solr? If so, are they performant? > > > > Hi Team, > Is it possible to write subqueries in dismaxrequest handler? > > ~Vikrant > > > Edoardo Marcora wrote: > > > > Does Solr have the ability to do subqueries, like this one (in SQL): > > > > SELECT id, first_name > > FROM student_details > > WHERE first_name IN (SELECT first_name > > FROM student_details > > WHERE subject= 'Science'); > > > > If so, how performant is this kind of queries? > > > > -- > View this message in context: > http://old.nabble.com/Are-subqueries-possible-in-Solr--If-so%2C-are-they-performant--tp24467023p26271600.html > Sent from the Solr - User mailing list archive at Nabble.com.

From forum: Solr - User

How TEXT field make sortable?

2009-11-09T22:14:50Z

Can some one help me how we can sort the text field.

<field name="TITLE" type="text" indexed="true" stored="true"/>

--
DEEPAK AGRAWAL
+91-9379433455
GOOD LUCK.....

From forum: Solr - User

Re: Solr Internal exception on startup...

2009-11-09T20:11:13Z

This looks like a file system access control configuration problem,
not a Solr, Lucene, or even Java problem.

You could disable the various security things like SELinux and test it again.

On Mon, Nov 9, 2009 at 1:14 PM, William Pierce <evalsinca@...> wrote:

> All,
>
> I realized that the stack trace I had sent in my previous email was
> truncated to not include the solr portions.....here is the fuller stack
> trace:
>
> SEVERE: Exception starting filter SolrRequestFilter
> org.apache.solr.common.SolrException: java.security.AccessControlException:
> access denied (java.io.FilePermission
> /home/ubuntu/apps/solr/tomcatweb/resumes/lib read)
> at
> org.apache.solr.servlet.SolrDispatchFilter.<init>(SolrDispatchFilter.java:68)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at java.lang.Class.newInstance0(Class.java:355)
> at java.lang.Class.newInstance(Class.java:308)
> at
> org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:255)
> at
> org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:397)
> at
> org.apache.catalina.core.ApplicationFilterConfig.<init>(ApplicationFilterConfig.java:108)
> at
> org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:3800)
> at
> org.apache.catalina.core.StandardContext.start(StandardContext.java:4450)
> at
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791)
> at
> org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:123)
> at
> org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:769)
> at
> org.apache.catalina.core.StandardHost.addChild(StandardHost.java:526)
> at
> org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:630)
> at
> org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:556)
> at
> org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:491)
> at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1206)
> at
> org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:314)
> at
> org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119)
> Nov 9, 2009 9:08:57 PM org.apache.catalina.core.StandardContext filterStart
> SEVERE: Exception starting filter SolrRequestFilter
> org.apache.solr.common.SolrException: java.security.AccessControlException:
> access denied (java.io.FilePermission
> /home/ubuntu/apps/solr/tomcatweb/resumes/lib read)
> at
> org.apache.solr.servlet.SolrDispatchFilter.<init>(SolrDispatchFilter.java:68)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at java.lang.Class.newInstance0(Class.java:355)
> at java.lang.Class.newInstance(Class.java:308)
> at
> org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:255)
> at
> org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:397)
> at
> org.apache.catalina.core.ApplicationFilterConfig.<init>(ApplicationFilterConfig.java:108)
> at
> org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:3800)
>
> Cheers,
>
> - Bill
>
> --------------------------------------------------
> From: "William Pierce" <evalsinca@...>
> Sent: Monday, November 09, 2009 12:49 PM
> To: <solr-user@...>
> Subject: Solr Internal exception on startup...
>
>> Folks:
>>
>> I am encountering an internal exception running solr on an Ubuntu 9.04
>> box, running tomcat 6. I have deposited the solr nightly bits (as of
>> October 7) into the folder: /usr/share/tomcat6/lib
>>
>> The exception from the log says:
>>
>> Nov 9, 2009 8:26:13 PM org.apache.catalina.core.StandardContext
>> filterStart
>> SEVERE: Exception starting filter SolrRequestFilter
>> org.apache.solr.common.SolrException:
>> java.security.AccessControlException: access denied (java.io.FilePermission
>> /home/ubuntu/apps/solr/tomcatweb/prod/lib read)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.<init>(SolrDispatchFilter.java:68)
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> Method)
>> at
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>> at java.lang.Class.newInstance0(Class.java:355)
>> at java.lang.Class.newInstance(Class.java:308)
>> at
>> org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:255)
>> at
>> org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:397)
>> at
>> org.apache.catalina.core.ApplicationFilterConfig.<init>(ApplicationFilterConfig.java:108)
>> at
>> org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:3800)
>>
>>
>> This is strange because the documentation says that the "lib" folder is
>> optional. (As a point of reference, I don't have a lib folder for my
>> windows installation). In any event, I created an empty "lib' folder and I
>> am still getting this same exception. (I gave the lib folder 777
>> permission.)
>>
>> <Context docBase="/usr/share/tomcat6/lib/solr-nightly-10-07-2009.war"
>> debug="0" crossContext="true" >
>> <Environment name="solr/home" type="java.lang.String"
>> value="/home/ubuntu/apps/solr/tomcatweb/prod" override="true" />
>> </Context>
>>
>> Under the folder /home/ubuntu/apps/solr/tomcatweb/prod are all solr
>> folders (conf, data).
>>
>> Can anybody help me here with what looks like a basic configuration error
>> on my part?
>>
>> Thanks,
>>
>> - Bill
>>
>>
>

--
Lance Norskog
goksron@...

From forum: Solr - User

Re: DocumentObjectBinder.getBean in solrj

2009-11-09T20:09:13Z

That's great!

Please make a new issue in JIRA asking for this feature, and attach
your patch. Be sure to include unit tests.

On Mon, Nov 9, 2009 at 12:48 PM, Christian López Espínola
<penyaskito@...> wrote:

> On Mon, Nov 9, 2009 at 8:26 PM, Christian López Espínola
> <penyaskito@...> wrote:
>> Hi,
>>
>> I need to create different beans from a search result, according to a
>> stored field with the classname.
>> What I'd find useful would be a getBean method in
>> DocumentObjectBinder, instead of the getBeans method (my list would
>> have different class instances).
>>
>> Is there any workaround for this? If not, would anyone be interested on this?
>> I can contribute a patch for this.
>>
>> TIA.
>
> Attached is the patch I would like to have in Solrj.
> If there is any problem with it please let me know. I followed the
> HowToContribute wiki page and I hope that I didn't miss any steps.
>
>>
>> --
>> Cheers,
>>
>> Christian López Espínola <penyaskito>
>>
>
>
>
> --
> Cheers,
>
> Christian López Espínola <penyaskito>
>

--
Lance Norskog
goksron@...

From forum: Solr - Dev

Re: [DIH] blocking import operation

2009-11-09T19:56:08Z

DIH imports are really long running. There is a good chance that the
connection times out or breaks in between.

how about a callback?

On Tue, Nov 10, 2009 at 12:12 AM, Sascha Szott <szott@...> wrote:

> Hi all,
>
> currently, DIH's import operation(s) only works asynchronously. Therefore,
> after submitting an import request, DIH returns immediately, while the
> import process (in case a large amount of data needs to be indexed)
> continues asynchronously behind the scenes.
>
> So, what is the recommended way to check if the import process has already
> finished? Or still better, is there any method / workaround that will block
> the import operation's caller until the operation has finished?
>
> In my application, the DIH receives some URL parameters which are used for
> determining the database name that is used within data-config.xml, e.g.
>
> http://localhost:8983/solr/dataimport?command=full-import&dbname=foo
>
> Since only one DIH, /dataimport, is defined, but several database needs to
> be indexed, it is required to issue this command several times, e.g.
>
> http://localhost:8983/solr/dataimport?command=full-import&dbname=foo
>
> ... wait until /dataimport?command=status says "Indexing completed" (but
> without using a loop that checks it again and again) ...
>
> http://localhost:8983/solr/dataimport?command=full-import&dbname=bar&clean=false
>
>
> A suitable solution, at least IMHO, would be to have an additional DIH
> parameter which determines whether the import call is blocking on
> non-blocking, the default. As far as I see, this could be accomplished since
> Solr can execute more than one import operation at a time (it starts a new
> thread for each). Perhaps, my question is somehow related to the discussion
> [1] on ParallelDataImportHandler.
>
> Best,
> Sascha
>
> [1] http://www.lucidimagination.com/search/document/a9b26ade46466ee
>
>

--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com

From forum: Solr - User

[jira] Assigned: (SOLR-1551) Provide DocumentObjectBinder.getBean

2009-11-09T19:53:32Z

[ https://issues.apache.org/jira/browse/SOLR-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Noble Paul reassigned SOLR-1551:
--------------------------------

Assignee: Noble Paul

> Provide DocumentObjectBinder.getBean
> ------------------------------------
>
> Key: SOLR-1551
> URL: https://issues.apache.org/jira/browse/SOLR-1551
> Project: Solr
> Issue Type: Improvement
> Components: clients - java
> Affects Versions: 1.1.0, 1.2, 1.3
> Reporter: Christian Lopez Espinola
> Assignee: Noble Paul
> Fix For: 1.5
>
> Attachments: getBeanPatch.patch
>
>
> It would be useful being able of creating the beans individually, for people who have different entities indexed with a unique schema.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

From forum: Solr - Dev

[jira] Updated: (SOLR-1551) Provide DocumentObjectBinder.getBean

2009-11-09T19:53:32Z

[ https://issues.apache.org/jira/browse/SOLR-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Noble Paul updated SOLR-1551:
-----------------------------

Fix Version/s: 1.5

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

From forum: Solr - Dev

[jira] Updated: (SOLR-1549) SqlEntityProcessor does not recognize onError attribute

2009-11-09T19:51:32Z

[ https://issues.apache.org/jira/browse/SOLR-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Noble Paul updated SOLR-1549:
-----------------------------

Fix Version/s: 1.5

> SqlEntityProcessor does not recognize onError attribute
> -------------------------------------------------------
>
> Key: SOLR-1549
> URL: https://issues.apache.org/jira/browse/SOLR-1549
> Project: Solr
> Issue Type: Improvement
> Components: contrib - DataImportHandler
> Affects Versions: 1.4, 1.5
> Reporter: Sascha Szott
> Assignee: Noble Paul
> Priority: Minor
> Fix For: 1.5
>
>
> Unfortunately, the SqlEntityProcessor does not recognize the value of an entity's onError attribute in DIH's data config file. Therefore, in cases where SQL exceptions are thrown somewhere inside the constructor of ResultSetIterators (which is an inner class of JdbcDataSource), Solr's import exits immediately, even though onError is set to continue or skip.
> In my opinion, use cases exist that will profit from database related exception handling inside of Solr (e.g., in cases where the existence of certain database tables or views is not predictable).

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

From forum: Solr - Dev

[jira] Assigned: (SOLR-1549) SqlEntityProcessor does not recognize onError attribute

2009-11-09T19:51:32Z

[ https://issues.apache.org/jira/browse/SOLR-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Noble Paul reassigned SOLR-1549:
--------------------------------

Assignee: Noble Paul

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

From forum: Solr - Dev

Re: DataImportHandler Error (SEVERE: Ignoring Error when closing connection)

2009-11-09T19:50:08Z

this should not cause any harm. it is just closing the connection at
the end of the process. But the stacktrace could be avoided

On Tue, Nov 10, 2009 at 2:39 AM, Pradeep Pujari <Pradeepp@...> wrote:

> I am getting the following error in solr/trunk code while do a full-dataimport from db2 database.
>
> INFO: Read dataimport.properties
> Nov 9, 2009 1:02:55 PM org.apache.solr.handler.dataimport.SolrWriter persist
> INFO: Wrote last indexed time to dataimport.properties
> Nov 9, 2009 1:02:55 PM org.apache.solr.handler.dataimport.DocBuilder execute
> INFO: Time taken = 0:0:2.704
> Nov 9, 2009 1:02:55 PM org.apache.solr.handler.dataimport.JdbcDataSource closeConnection
> SEVERE: Ignoring Error when closing connection
> com.ibm.db2.jcc.a.SqlException: java.sql.Connection.close() requested while a transaction is in progress on the connection.The transaction remains active, and the connection cannot be closed.
> at com.ibm.db2.jcc.a.p.r(p.java:1385)
> at com.ibm.db2.jcc.a.p.u(p.java:1419)
> at com.ibm.db2.jcc.a.p.s(p.java:1395)
> at com.ibm.db2.jcc.a.p.close(p.java:1378)
> at org.apache.solr.handler.dataimport.JdbcDataSource.closeConnection(JdbcDataSource.java:399)
> at org.apache.solr.handler.dataimport.JdbcDataSource.close(JdbcDataSource.java:390)
> at org.apache.solr.handler.dataimport.DataConfig$Entity.clearCache(DataConfig.java:173)
> at org.apache.solr.handler.dataimport.DataConfig.clearCaches(DataConfig.java:331)
> at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:339)
> at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:389)
> at org.apache.solr.handler.dataimport.DataImporter$3.run(DataImporter.java:370)
>
>
>

--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com

From forum: Solr - Dev

Re: Specifying multiple documents in DataImportHandler dataConfig

2009-11-09T19:15:24Z

HI Lance,

I think you are discussing a different issue here. We are talking about
each row from each table represents a document in index. You look to discuss
about some documents may have multi-value fields which are stored in a
separate table in RDBMS because of normalization.

On Mon, Nov 9, 2009 at 6:01 PM, Lance Norskog <goksron@...> wrote:

> There is a more fundamental problem here: Solr/Lucene index only
> implements one table. If you have data from multiple tables in a
> normalized index, you have denormalize the multi-table DB schema to
> make a single-table Solr/Lucene index.
>
> Your indexing will probably be faster if you a join in SQL to supply
> your entire set of fields per database request.
>
> 2009/11/7 Noble Paul നോബിള്‍ नोब्ळ् <noble.paul@...>:
> > On Sun, Nov 8, 2009 at 8:25 AM, Bertie Shen <bertie.shen@...>
> wrote:
> >> I have figured out a way to solve this problem: just specify a
> >> single <document> blah blah blah </document>. Under <document>, specify
> >> multiple top level entity entries, each of which corresponds to one
> table
> >> data.
> >>
> >> So each top level entry will map one row in it to a document in Lucene
> >> index. <document> in DIH is *NOT* mapped to a document in Lucene index
> while
> >> top-level entity is. I feel <document> tag is redundant and misleading
> in
> >> data config and thus should be removed.
> >
> > There are some common attributes specified at the <document> level .
> > It still acts as a container tag .
> >>
> >> Cheers.
> >>
> >> On Sat, Nov 7, 2009 at 9:43 AM, Bertie Shen <bertie.shen@...>
> wrote:
> >>
> >>> I have the same problem. I had thought we could specify multiple
> <document>
> >>> blah blah blah</document>s, each of which is mapping one table in the
> RDBMS.
> >>> But I found it was not the case. It only picks the first <document>blah
> blah
> >>> blah</document> to do indexing.
> >>>
> >>> I think Rupert's and my request are pretty common. Basically there are
> >>> multiple tables in RDBMS, and we want each row in each table become a
> >>> document in Lucene index. How can we write one data config.xml file to
> let
> >>> DataImportHandler import multiple tables at the same time?
> >>>
> >>> Rupert, have you figured out a way to do it?
> >>>
> >>> Thanks.
> >>>
> >>>
> >>>
> >>> On Tue, Sep 8, 2009 at 3:42 PM, Rupert Fiasco <rufiasco@...>
> wrote:
> >>>
> >>>> Maybe I should be more clear: I have multiple tables in my DB that I
> >>>> need to save to my Solr index. In my app code I have logic to persist
> >>>> each table, which maps to an application model to Solr. This is fine.
> >>>> I am just trying to speed up indexing time by using DIH instead of
> >>>> going through my application. From what I understand of DIH I can
> >>>> specify one dataSource element and then a series of document/entity
> >>>> sets, for each of my models. But like I said before, DIH only appears
> >>>> to want to index the first document declared under the dataSource tag.
> >>>>
> >>>> -Rupert
> >>>>
> >>>> On Tue, Sep 8, 2009 at 4:05 PM, Rupert Fiasco<rufiasco@...>
> wrote:
> >>>> > I am using the DataImportHandler with a JDBC datasource. From my
> >>>> > understanding of DIH, for each of my "content types" e.g. Blog
> posts,
> >>>> > Mesh Categories, etc I would construct a series of document/entity
> >>>> > sets, like
> >>>> >
> >>>> > <dataConfig>
> >>>> > <dataSource driver="com.mysql.jdbc.Driver" url="jdbc:mysql://...."
> />
> >>>> >
> >>>> > 
> >>>> > <document name="blog_entries">
> >>>> > <entity name="blog_entries" query="select
> >>>> > id,title,keywords,summary,data,title as name_fc,'BlogEntry' as type
> >>>> > from blog_entries">
> >>>> > <field column="id" name="pk_i" />
> >>>> > <field column="id" name="id" />
> >>>> > <field column="title" name="text_t" />
> >>>> > <field column="data" name="text_t" />
> >>>> > </entity>
> >>>> > </document>
> >>>> >
> >>>> > 
> >>>> > <document name="mesh_category">
> >>>> > <entity name="mesh_categories" query="select
> >>>> > id,name,node_key,name as name_fc,'MeshCategory' as type from
> >>>> > mesh_categories">
> >>>> > <field column="id" name="pk_i" />
> >>>> > <field column="id" name="id" />
> >>>> > <field column="name" name="text_t" />
> >>>> > <field column="node_key" name="string" />
> >>>> > <field column="name_fc" name="facet_value" />
> >>>> > <field column="type" name="type_t" />
> >>>> > </entity>
> >>>> > </document>
> >>>> > </datasource>
> >>>> > </dataConfig>
> >>>> >
> >>>> >
> >>>> > Solr parses this just fine and allows me to issue a
> >>>> > /dataimport?command=full-import and it runs, but it only runs
> against
> >>>> > the "first" document (blog_entries). It doesnt run against the 2nd
> >>>> > document (mesh_categories).
> >>>> >
> >>>> > If I remove the 2 document elements and wrap both entity sets in
> just
> >>>> > one document tag, then both sets get indexed, which seemingly
> achieves
> >>>> > my goal. This just doesnt make sense from my understanding of how
> DIH
> >>>> > works. My 2 content types are indeed separate so they logically
> >>>> > represent two document types, not one.
> >>>> >
> >>>> > Is this correct? What am I missing here?
> >>>> >
> >>>> > Thanks
> >>>> > -Rupert
> >>>> >
> >>>>
> >>>
> >>>
> >>
> >
> >
> >
> > --
> > -----------------------------------------------------
> > Noble Paul | Principal Engineer| AOL | http://aol.com
> >
>
>
>
> --
> Lance Norskog
> goksron@...
>

From forum: Solr - User

[jira] Updated: (SOLR-1506) Search multiple cores using MultiReader

2009-11-09T18:25:32Z

[ https://issues.apache.org/jira/browse/SOLR-1506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jason Rutherglen updated SOLR-1506:
-----------------------------------

Attachment: SOLR-1506.patch

MultiReader doesn't support reopen with the readOnly parameter. This patch adds a test case for commit on the proxy, and a workaround (if unsupported is caught, then regular reopen is called).

> Search multiple cores using MultiReader
> ---------------------------------------
>
> Key: SOLR-1506
> URL: https://issues.apache.org/jira/browse/SOLR-1506
> Project: Solr
> Issue Type: Improvement
> Components: search
> Affects Versions: 1.4
> Reporter: Jason Rutherglen
> Priority: Trivial
> Fix For: 1.5
>
> Attachments: SOLR-1506.patch, SOLR-1506.patch, SOLR-1506.patch
>
>
> I need to search over multiple cores, and SOLR-1477 is more
> complicated than expected, so here we'll create a MultiReader
> over the cores to allow searching on them.
> Maybe in the future we can add parallel searching however
> SOLR-1477, if it gets completed, provides that out of the box.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

From forum: Solr - Dev

Re: de-boosting certain facets during search

2009-11-09T18:14:49Z

Paul,

Inline below...

On Nov 9, 2009, at 6:28 PM, Paul Rosen wrote:
> If I could just create the desired URL, I can probably work
> backwards and construct the correct ruby call.

Right, this list will always serve you best if you take the Ruby out
of the equation. solr-ruby, while cool and all, isn't very well known
by many, but Solr URLs are universal lingo here.

> http://localhost:8983/solr/resources/select?hl.fragsize=600
> &hl=true
> &facet.field=genre
> &facet.field=archive
> &facet.limit=-1
> &qt=standard
> &start=0
> &fq=archive%3A%22blake%22
> &hl.fl=text
> &fl=uri%2Carchive%2Cdate_label%2Cgenre
> &facet=true
> &q=%28history%29
> &rows=60
> &facet.missing=true
> &facet.mincount=1
>
> What this search returns from my index is 53 hits. The first 43
> contain the genre field value "Citation" and the last 10 do not
> (they contain other values in that field.)
>
> Note: the genre field is multivalued, if that matters.

It matters if you want to sort by genre. It doesn't make sense to
sort by a multivalued field though.

> I'd like the search to put all of the objects that contain genre
> "Citation" below the 10 objects that do not contain that genre.

Are you dogmatic about them _all_ appearing below? Or might it be ok
if a Citation that has substantially better term matching than another
type of object appear ahead in the results?

> I've read the various pages on boosting, but since I'm not actively
> searching on the field that I want to put a boost value on, I'm not
> sure how to go about this.

How this is done is dependent on the query parser. You're using the
Lucene query parser. Something like this might work for you:

http://localhost:8983/solr/select?q=ipod%20%20OR%20%28ipod%20-manu:Belkin%29
^5&debugQuery=true

unurlencoded, that is q=ipod OR (ipod -manu:Belkin)^5, where the
users query is repeated in a second clause that boosts up all
documents that are not of a particular manufacturer using the example
docs that Solr ships with.

Be sure to use debugQuery=true to look at the score explanations (try
looking at the output in the wt=ruby&indent=on format for best
readability).

Additionally...

>
> Thanks for any hints.
>
> Paul Rosen wrote:
>> Hi,
>> I'm using solr-ruby-0.0.8 and solr 1.4.
>> My data contains a faceted field called "genre". We would like one
>> particular genre, (the one named "Citation") to show up last in the
>> results.
>> I'm having trouble figuring out how to add the boost parameter to
>> the solr-ruby call. Here is my code:
>> req = Solr::Request::Standard.new(:start => start,
>> :rows => max,
>> :sort => sort_param,
>> :query => query,
>> :filter_queries => filter_queries,
>> :field_list => @field_list,
>> :facets => {:fields => @facet_fields,
>> :mincount => 1,
>> :missing => true,
>> :limit => -1},
>> :highlighting => {:field_list => ['text'],
>> :fragment_size => 600},
>> :shards => @cores)
>> response = @solr.send(req)
>> Do I just format it inside my query, like this:
>> query = query + "AND genre:Citation^.01"
>> or in filter_query, like this:
>> filter_queries.push("genre:Citation^.01")
>> or is there a hash parameter that I set?

filter queries (fq) do not contribute to the score, so boosting them
makes no score difference at all.

>> (Note that the user can select Citation explicitly. I'll probably
>> special case that.)
>> I've tried variations of the above, but I've had no luck so far.
>> Thanks,
>> Paul
>

Erik

From forum: Solr - User

Re: Segment file not found error - after replicating

2009-11-09T18:13:57Z

Madu,

So are you saying that all slaves have the exact same index, and that index is exactly the same as the one on the master, yet only some of those slaves exhibit this error, while others do not? Mind listing index directories of 1) master 2) slave without errors, 3) slave with errors and doing:
du -s /path/to/index/on/master
du -s /path/to/index/on/slave/without/errors
du -s /path/to/index/on/slave/with/errors

Otis
--
Sematext is hiring -- http://sematext.com/about/jobs.html?mls
Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR

----- Original Message ----

> From: Maduranga Kannangara <mkannangara@...>
> To: "solr-user@..." <solr-user@...>
> Sent: Mon, November 9, 2009 7:47:04 PM
> Subject: RE: Segment file not found error - after replicating
>
> Thanks Otis!
>
> Yes, I checked the index directories and they are 100% same, both timestamp and
> size wise.
>
> Not all the slaves face this issue. I would say roughly 50% has this trouble.
>
> Logs do not have any errors too :-(
>
> Any other things I should do/look at?
>
> Cheers
> Madu
>
>
> -----Original Message-----
> From: Otis Gospodnetic [mailto:otis_gospodnetic@...]
> Sent: Tuesday, 10 November 2009 9:26 AM
> To: solr-user@...
> Subject: Re: Segment file not found error - after replicating
>
> It's hard to troubleshoot blindly like this, but have you tried manually
> comparing the contents of the index dir on the master and on the slave(s)?
> If they are out of sync, have you tried forcing of replication to see if one of
> the subsequent replication attempts gets the dirs in sync?
> Do you have more than 1 slave and do they all start having this problem at the
> same time?
> Any errors in the logs for any of the scripts involved in replication in 1.3?
>
> Otis
> --
> Sematext is hiring -- http://sematext.com/about/jobs.html?mls
> Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
>
>
>
> ----- Original Message ----
> > From: Maduranga Kannangara
> > To: "solr-user@..."
> > Sent: Sun, November 8, 2009 10:30:44 PM
> > Subject: Segment file not found error - after replicating
> >
> > Hi guys,
> >
> > We use Solr 1.3 for indexing large amounts of data (50G avg) on Linux
> > environment and use the replication scripts to make replicas those live in
> load
> > balancing slaves.
> >
> > The issue we face quite often (only in Linux servers) is that they tend to not
>
> > been able to find the segment file (segment_x etc) after the replicating
> > completed. As this has become quite common, we started hitting a serious
> issue.
> >
> > Below is a stack trace, if that helps and any help on this matter is greatly
> > appreciated.
> >
> > --------------------------------
> >
> > Nov 5, 2009 11:34:46 PM org.apache.solr.util.plugin.AbstractPluginLoader load
> > INFO: created /admin/: org.apache.solr.handler.admin.AdminHandlers
> > Nov 5, 2009 11:34:46 PM org.apache.solr.util.plugin.AbstractPluginLoader load
> > INFO: created /admin/ping: org.apache.solr.handler.PingRequestHandler
> > Nov 5, 2009 11:34:46 PM org.apache.solr.util.plugin.AbstractPluginLoader load
> > INFO: created /debug/dump: org.apache.solr.handler.DumpRequestHandler
> > Nov 5, 2009 11:34:46 PM org.apache.solr.util.plugin.AbstractPluginLoader load
> > INFO: created gap: org.apache.solr.highlight.GapFragmenter
> > Nov 5, 2009 11:34:46 PM org.apache.solr.util.plugin.AbstractPluginLoader load
> > INFO: created regex: org.apache.solr.highlight.RegexFragmenter
> > Nov 5, 2009 11:34:46 PM org.apache.solr.util.plugin.AbstractPluginLoader load
> > INFO: created html: org.apache.solr.highlight.HtmlFormatter
> > Nov 5, 2009 11:34:46 PM org.apache.solr.servlet.SolrDispatchFilter init
> > SEVERE: Could not start SOLR. Check solr/home property
> > java.lang.RuntimeException: java.io.FileNotFoundException:
> > /solrinstances/solrhome01/data/index/segments_v (No such file or directory)
> > at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:960)
> > at org.apache.solr.core.SolrCore.(SolrCore.java:470)
> > at
> >
> org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:119)
> > at
> > org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:69)
> > at
> >
> org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:275)
> > at
> >
> org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:397)
> > at
> >
> org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:108)
> > at
> >
> org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:3709)
> > at
> > org.apache.catalina.core.StandardContext.start(StandardContext.java:4363)
> > at
> > org.apache.catalina.core.StandardContext.reload(StandardContext.java:3099)
> > at
> > org.apache.catalina.manager.ManagerServlet.reload(ManagerServlet.java:916)
> > at
> >
> org.apache.catalina.manager.HTMLManagerServlet.reload(HTMLManagerServlet.java:536)
> > at
> >
> org.apache.catalina.manager.HTMLManagerServlet.doGet(HTMLManagerServlet.java:114)
> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
> > at
> >
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> > at
> >
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> > at com.jamonapi.JAMonFilter.doFilter(JAMonFilter.java:57)
> > at
> >
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> > at
> >
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> > at
> >
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
> > at
> >
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
> > at
> >
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525)
> > at
> > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
> > at
> > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> > at
> >
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> > at
> > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
> > at
> > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:845)
> > at
> >
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
> > at
> > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
> > at java.lang.Thread.run(Thread.java:619)
> > Caused by: java.io.FileNotFoundException:
> > /solrinstances/solrhome01/data/index/segments_v (No such file or directory)
> > at java.io.RandomAccessFile.open(Native Method)
> > at java.io.RandomAccessFile.(RandomAccessFile.java:212)
> > at
> >
> org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.(FSDirectory.java:552)
> > at
> > org.apache.lucene.store.FSDirectory$FSIndexInput.(FSDirectory.java:582)
> > at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
> > at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:482)
> > at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:214)
> > at
> >
> org.apache.lucene.index.DirectoryIndexReader$1.doBody(DirectoryIndexReader.java:94)
> > at
> >
> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:653)
> > at
> >
> org.apache.lucene.index.DirectoryIndexReader.open(DirectoryIndexReader.java:111)
> > at org.apache.lucene.index.IndexReader.open(IndexReader.java:316)
> > at org.apache.lucene.index.IndexReader.open(IndexReader.java:237)
> > at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:951)
> > ... 30 more
> > Nov 5, 2009 11:34:46 PM org.apache.solr.common.SolrException log
> > SEVERE: java.lang.RuntimeException: java.io.FileNotFoundException:
> > /solrinstances/solrhome01/data/index/segments_v (No such file or directory)
> > at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:960)
> > at org.apache.solr.core.SolrCore.(SolrCore.java:470)
> > at
> >
> org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:119)
> > at
> > org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:69)
> > at
> >
> org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:275)
> > at
> >
> org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:397)
> > at
> >
> org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:108)
> > at
> >
> org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:3709)
> > at
> > org.apache.catalina.core.StandardContext.start(StandardContext.java:4363)
> > at
> > org.apache.catalina.core.StandardContext.reload(StandardContext.java:3099)
> > at
> > org.apache.catalina.manager.ManagerServlet.reload(ManagerServlet.java:916)
> > at
> >
> org.apache.catalina.manager.HTMLManagerServlet.reload(HTMLManagerServlet.java:536)
> > at
> >
> org.apache.catalina.manager.HTMLManagerServlet.doGet(HTMLManagerServlet.java:114)
> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
> > at
> >
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> > at
> >
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> > at com.jamonapi.JAMonFilter.doFilter(JAMonFilter.java:57)
> > at
> >
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> > at
> >
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> > at
> >
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
> > at
> >
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
> > at
> >
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525)
> > at
> > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
> > at
> > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> > at
> >
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> > at
> > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
> > at
> > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:845)
> > at
> >
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
> > at
> > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
> > at java.lang.Thread.run(Thread.java:619)
> > Caused by: java.io.FileNotFoundException:
> > /solrinstances/solrhome01/data/index/segments_v (No such file or directory)
> > at java.io.RandomAccessFile.open(Native Method)
> > at java.io.RandomAccessFile.(RandomAccessFile.java:212)
> > at
> >
> org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.(FSDirectory.java:552)
> > at
> > org.apache.lucene.store.FSDirectory$FSIndexInput.(FSDirectory.java:582)
> > at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
> > at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:482)
> > at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:214)
> > at
> >
> org.apache.lucene.index.DirectoryIndexReader$1.doBody(DirectoryIndexReader.java:94)
> > at
> >
> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:653)
> > at
> >
> org.apache.lucene.index.DirectoryIndexReader.open(DirectoryIndexReader.java:111)
> > at org.apache.lucene.index.IndexReader.open(IndexReader.java:316)
> > at org.apache.lucene.index.IndexReader.open(IndexReader.java:237)
> > at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:951)
> > ... 30 more
> >
> > Nov 5, 2009 11:34:46 PM org.apache.solr.servlet.SolrDispatchFilter init
> > INFO: SolrDispatchFilter.init() done
> > Nov 5, 2009 11:34:46 PM org.apache.solr.servlet.SolrServlet init
> > INFO: SolrServlet.init()
> >
> > --------------------------------
> >
> > Steps to re-produce the error (However, for me did not work in my local box.
> > Also remote server is too far away to remote-debug!).
> >
> > - Post some new data to the master server (Usually about 1Gb worth text
> files)
> > - Run the replicate script in slave Solr instance
> > - Try to login to admin in slave Solr instance
> >
> > And you should see above stack trace even in the Tomcat output.
> >
> >
> > Thanks in advance.
> > Madu

From forum: Solr - User

Re: synonym payload boosting

2009-11-09T18:11:49Z

David, when you get this working would you consider writing a case
study on the wiki? Nothing complex, just something that describes how
you did several customizations to create a new feature.

On Mon, Nov 9, 2009 at 4:10 AM, Grant Ingersoll <gsingers@...> wrote:

>
> On Nov 9, 2009, at 4:41 AM, David Ginzburg wrote:
>
>> I have found this
>>
>> https://issues.apache.org/jira/browse/SOLR-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
>> patch
>> But i don't want to use any function, just the normal scoring and the
>> similarity class I have written.
>> Can you point me to modifications I need (if any) ?
>>
>>
>
> Amhet's point is that you need some query that will actually invoke the
> payload in scoring. PayloadTermQuery and PayloadNearQuery are the two that
> do this in Lucene. You can certainly write your own, as well.
>
> -Grant
>
>>
>> On Sun, Nov 8, 2009 at 16:33, AHMET ARSLAN <iorixxx@...> wrote:
>>
>>> Additionaly you need to modify your queryparser to return
>>> BoostingTermQuery, PayloadTermQuery, PayloadNearQuery etc.
>>>
>>> With these types of Queries scorePayload method invoked.
>>>
>>> Hope this helps.
>>>
>>> --- On Sun, 11/8/09, David Ginzburg <david@...> wrote:
>>>
>>>> From: David Ginzburg <david@...>
>>>> Subject: synonym payload boosting
>>>> To: solr-user@...
>>>> Date: Sunday, November 8, 2009, 4:06 PM
>>>> Hi,
>>>> I have a field and a wighted synonym map.
>>>> I have indexed the synonyms with the weight as payload.
>>>> my code snippet from my filter
>>>>
>>>> *public Token next(final Token reusableToken) throws
>>>> IOException *
>>>> * . *
>>>> * . *
>>>> * .*
>>>> * Payload boostPayload;*
>>>> *
>>>> *
>>>> * for (Synonym synonym : syns)
>>>> {*
>>>> * *
>>>> * Token newTok =
>>>> new Token(nToken.startOffset(),
>>>> nToken.endOffset(), "SYNONYM");*
>>>> *
>>>> newTok.setTermBuffer(synonym.getToken().toCharArray(), 0,
>>>> synonym.getToken().length());*
>>>> * // set the
>>>> position increment to zero*
>>>> * // this tells
>>>> lucene the synonym is*
>>>> * // in the exact
>>>> same location as the originating word*
>>>> *
>>>> newTok.setPositionIncrement(0);*
>>>> * boostPayload =
>>>> new
>>>> Payload(PayloadHelper.encodeFloat(synonym.getWieght()));*
>>>> *
>>>> newTok.setPayload(boostPayload);*
>>>> *
>>>> *
>>>> I have put it in the index time analyzer : this is my field
>>>> definition:
>>>>
>>>> *
>>>> <fieldType name="PersonName" class="solr.TextField"
>>>> positionIncrementGap="100" >
>>>> <analyzer type="index">
>>>> <tokenizer
>>>> class="solr.WhitespaceTokenizerFactory"/>
>>>> <filter
>>>> class="solr.StopFilterFactory" ignoreCase="true"
>>>> words="stopwords.txt"/>
>>>> <filter
>>>> class="solr.LowerCaseFilterFactory"/>
>>>> <filter
>>>> class="com.digitaltrowel.solr.DTSynonymFactory"
>>>> FreskoFunction="names_with_scoresPipe23Columns.txt"
>>>> ignoreCase="true"
>>>> expand="false"/>
>>>>
>>>> 
>>>> 
>>>> </analyzer>
>>>> <analyzer type="query">
>>>> <tokenizer
>>>> class="solr.WhitespaceTokenizerFactory"/>
>>>> <filter
>>>> class="solr.LowerCaseFilterFactory"/>
>>>> 
>>>> <filter
>>>> class="solr.StopFilterFactory" ignoreCase="true"
>>>> words="stopwords.txt"/>
>>>> 
>>>> 
>>>>
>>>> </analyzer>
>>>> </fieldType>
>>>>
>>>>
>>>> my similarity class is
>>>> public class BoostingSymilarity extends DefaultSimilarity
>>>> {
>>>>
>>>>
>>>> public BoostingSymilarity(){
>>>> super();
>>>>
>>>> }
>>>> @Override
>>>> public float scorePayload(String field,
>>>> byte [] payload, int offset,
>>>> int length)
>>>> {
>>>> double weight = PayloadHelper.decodeFloat(payload, 0);
>>>> return (float)weight;
>>>> }
>>>>
>>>> @Override public float coord(int overlap, int maxoverlap)
>>>> {
>>>> return 1.0f;
>>>> }
>>>>
>>>> @Override public float idf(int docFreq, int numDocs)
>>>> {
>>>> return 1.0f;
>>>> }
>>>>
>>>> @Override public float lengthNorm(String fieldName, int
>>>> numTerms)
>>>> {
>>>> return 1.0f;
>>>> }
>>>>
>>>> @Override public float tf(float freq)
>>>> {
>>>> return 1.0f;
>>>> }
>>>> }
>>>>
>>>> My problem is that scorePayload method does not get called
>>>> at search time
>>>> like the other methods in my similarity class.
>>>> I tested and verified it with break points.
>>>> What am I doing wrong?
>>>> I used solr 1.3 and thinking of the payload boos support in
>>>> solr 1.4.
>>>>
>>>>
>>>> *
>>>>
>>>
>>> __________________________________________________
>>> Do You Yahoo!?
>>> Tired of spam? Yahoo! Mail has the best spam protection around
>>> http://mail.yahoo.com
>>>
>>
>>
>>
>> --
>> Regards
>>
>> _____________________
>> David Ginzburg
>> Developer, Digital Trowel
>> 1 Hayarden St., Airport City
>> [POB 169, NATBAG]
>> Lod, 70151, Israel
>> http://www.digitaltrowel.com/
>> Office: +972 73 240 522
>> Mobile: +972 50 496 0595
>>
>> CHECK OUT OUR NEW TEXT MINING BLOG:
>> http://mineyourbusiness.wordpress.com/
>
> --------------------------
> Grant Ingersoll
> http://www.lucidimagination.com/
>
> Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids) using
> Solr/Lucene:
> http://www.lucidimagination.com/search
>
>

--
Lance Norskog
goksron@...

From forum: Solr - User

Re: Specifying multiple documents in DataImportHandler dataConfig

2009-11-09T18:01:38Z

There is a more fundamental problem here: Solr/Lucene index only
implements one table. If you have data from multiple tables in a
normalized index, you have denormalize the multi-table DB schema to
make a single-table Solr/Lucene index.

Your indexing will probably be faster if you a join in SQL to supply
your entire set of fields per database request.

2009/11/7 Noble Paul നോബിള്‍ नोब्ळ् <noble.paul@...>:

> On Sun, Nov 8, 2009 at 8:25 AM, Bertie Shen <bertie.shen@...> wrote:
>> I have figured out a way to solve this problem: just specify a
>> single <document> blah blah blah </document>. Under <document>, specify
>> multiple top level entity entries, each of which corresponds to one table
>> data.
>>
>> So each top level entry will map one row in it to a document in Lucene
>> index. <document> in DIH is *NOT* mapped to a document in Lucene index while
>> top-level entity is. I feel <document> tag is redundant and misleading in
>> data config and thus should be removed.
>
> There are some common attributes specified at the <document> level .
> It still acts as a container tag .
>>
>> Cheers.
>>
>> On Sat, Nov 7, 2009 at 9:43 AM, Bertie Shen <bertie.shen@...> wrote:
>>
>>> I have the same problem. I had thought we could specify multiple <document>
>>> blah blah blah</document>s, each of which is mapping one table in the RDBMS.
>>> But I found it was not the case. It only picks the first <document>blah blah
>>> blah</document> to do indexing.
>>>
>>> I think Rupert's and my request are pretty common. Basically there are
>>> multiple tables in RDBMS, and we want each row in each table become a
>>> document in Lucene index. How can we write one data config.xml file to let
>>> DataImportHandler import multiple tables at the same time?
>>>
>>> Rupert, have you figured out a way to do it?
>>>
>>> Thanks.
>>>
>>>
>>>
>>> On Tue, Sep 8, 2009 at 3:42 PM, Rupert Fiasco <rufiasco@...> wrote:
>>>
>>>> Maybe I should be more clear: I have multiple tables in my DB that I
>>>> need to save to my Solr index. In my app code I have logic to persist
>>>> each table, which maps to an application model to Solr. This is fine.
>>>> I am just trying to speed up indexing time by using DIH instead of
>>>> going through my application. From what I understand of DIH I can
>>>> specify one dataSource element and then a series of document/entity
>>>> sets, for each of my models. But like I said before, DIH only appears
>>>> to want to index the first document declared under the dataSource tag.
>>>>
>>>> -Rupert
>>>>
>>>> On Tue, Sep 8, 2009 at 4:05 PM, Rupert Fiasco<rufiasco@...> wrote:
>>>> > I am using the DataImportHandler with a JDBC datasource. From my
>>>> > understanding of DIH, for each of my "content types" e.g. Blog posts,
>>>> > Mesh Categories, etc I would construct a series of document/entity
>>>> > sets, like
>>>> >
>>>> > <dataConfig>
>>>> > <dataSource driver="com.mysql.jdbc.Driver" url="jdbc:mysql://...." />
>>>> >
>>>> > 
>>>> > <document name="blog_entries">
>>>> > <entity name="blog_entries" query="select
>>>> > id,title,keywords,summary,data,title as name_fc,'BlogEntry' as type
>>>> > from blog_entries">
>>>> > <field column="id" name="pk_i" />
>>>> > <field column="id" name="id" />
>>>> > <field column="title" name="text_t" />
>>>> > <field column="data" name="text_t" />
>>>> > </entity>
>>>> > </document>
>>>> >
>>>> > 
>>>> > <document name="mesh_category">
>>>> > <entity name="mesh_categories" query="select
>>>> > id,name,node_key,name as name_fc,'MeshCategory' as type from
>>>> > mesh_categories">
>>>> > <field column="id" name="pk_i" />
>>>> > <field column="id" name="id" />
>>>> > <field column="name" name="text_t" />
>>>> > <field column="node_key" name="string" />
>>>> > <field column="name_fc" name="facet_value" />
>>>> > <field column="type" name="type_t" />
>>>> > </entity>
>>>> > </document>
>>>> > </datasource>
>>>> > </dataConfig>
>>>> >
>>>> >
>>>> > Solr parses this just fine and allows me to issue a
>>>> > /dataimport?command=full-import and it runs, but it only runs against
>>>> > the "first" document (blog_entries). It doesnt run against the 2nd
>>>> > document (mesh_categories).
>>>> >
>>>> > If I remove the 2 document elements and wrap both entity sets in just
>>>> > one document tag, then both sets get indexed, which seemingly achieves
>>>> > my goal. This just doesnt make sense from my understanding of how DIH
>>>> > works. My 2 content types are indeed separate so they logically
>>>> > represent two document types, not one.
>>>> >
>>>> > Is this correct? What am I missing here?
>>>> >
>>>> > Thanks
>>>> > -Rupert
>>>> >
>>>>
>>>
>>>
>>
>
>
>
> --
> -----------------------------------------------------
> Noble Paul | Principal Engineer| AOL | http://aol.com
>

--
Lance Norskog
goksron@...

From forum: Solr - User

Re: keep index in production and snapshots in separate phisical disks

2009-11-09T17:57:20Z

: Is there any way to make snapinstaller install the index in
: spanpshot20091023124543 (for example) from another disk? I am asking this

you're talking about the old script based replication correct? ... i don't
think that's possible since it relied on hardlinks and atomic move
operations. i don't think those work across physical disks.

: because I would like not to optimize the index in the master (if I do that
: it takes a long time to send it via rsync if it is so big). This way I would
: just have to send the new segments.

in Solr 1.4 the disadvantages of having a non-optimized index are fading
away ... before you worry too much about this you might want to run some
tests and verify that you really need to optimize to meet your performance
targets.

: In the slave I would have 2 phisical disks. Snappuller would send the
: snapshot to a disk (here the index would not be optimized). Snapinstaller
: would install the snapshot in the other disk, optimize it and open the
: newIndexReader. The optimization should be done in the disk wich contains
: the "not in production index" to not affect the search request speed.
: Any idea what should I hack to reach this goal in case it is possible?

I'm not really familiar with the new java based replication code, but i
suspect this could be setup fairly easily by running two solr instances on
your slave boxes ... one serving as a repeater (ie: both a slave and a
master) to the other. on the repeater port you would run the optimize,
and then the leaf level slave (serving queries to end users) would
replicate that optimized index over the loopback address (no network
overhead, should be ~fast as a file copy from a different disk)

...this is all just theory mind you, i've never tried this.

-Hoss

From forum: Solr - User

Re: Lowering ranking of certain documents while search in Solr

2009-11-09T17:38:09Z

Some hard-won knowledge: don't be afraid to re-index. Changing the
schema without re-indexing can lead to mountains of trouble.

On Mon, Nov 9, 2009 at 5:36 PM, Lance Norskog <goksron@...> wrote:

> The ExternalFileField is the tool for this.
>
> It is a field type which reads an array of floating point values from
> a file, one per document. You can boost on it. (I don't know if sort
> works.) There is no direct documentation that I found. Here are some
> hints.
>
> http://www.lucidimagination.com/search/?q=ExternalFileField
>
> http://lucene.apache.org/solr/api/org/apache/solr/schema/ExternalFileField.html
>
>
> On Fri, Nov 6, 2009 at 9:44 PM, Bhavnik Gajjar
> <bhavnik.gajjar@...> wrote:
>> Thanks again,
>>
>> This solution requires me to re-index 2 millions documents again in Solr!!!
>>
>> I would prefer something which can be done in ad-hoc way on my existing working/live application.
>>
>> Also, regarding the solution [q=hello +(source:source1^100 source:source2^0.1 ...)], I think if sorting is being done on some other field (rather, score desc) then it will messup documents!
>>
>> Is there a way to use function query or so then?
>>
>> Regards,
>> Bhavnik Gajjar
>>
>> ----- Original Message -----
>> From: "Avlesh Singh" <avlesh@...>
>> To: <solr-user@...>
>> Sent: Friday, November 06, 2009 10:25 PM
>> Subject: Re: Lowering ranking of certain documents while search in Solr
>>
>>
>>> How about adding an extra sint field, say "source_boost". For your 199
>>> sources the indexed value can be 0. For the "underprivileged" source it can
>>> be -1. Adding "sort=source_boost desc,score desc" would do the needful.
>>>
>>> Works?
>>>
>>> Cheers
>>> Avlesh
>>>
>>> On Fri, Nov 6, 2009 at 4:55 PM, Bhavnik Gajjar <
>>> bhavnik.gajjar@...> wrote:
>>>
>>>> Thanks for your inputs,
>>>>
>>>> This solution works in general!!
>>>>
>>>> In my system, there are around 200 different sources and out of that only
>>>> one source is needed with lower rank. Now, if I go with the solution you
>>>> provided then I need 199 sources with individual boost and then one source
>>>> with lower boost value. Now, I'm afraid that, this would cause performance
>>>> issue in Solr. Is there any other solution to work with this then?
>>>>
>>>> Note: our massive application is already built with standard request
>>>> handler (I can't use dismax at this stage!!)
>>>>
>>>> Seeking for positive reply!
>>>>
>>>> Thanks again,
>>>> Bhavnik Gajjar
>>>>
>>>>
>>>> ----- Original Message -----
>>>> From: "Avlesh Singh" <avlesh@...>
>>>> To: <solr-user@...>
>>>> Sent: Thursday, November 05, 2009 12:00 PM
>>>> Subject: Re: Lowering ranking of certain documents while search in Solr
>>>>
>>>>
>>>> > The query "q=hello&start=0&rows=20&fl=Source&bq=(*:*
>>>> -Source:Source2^10000)"
>>>> > should work as expected. There can be two reasons for it not to work -
>>>> >
>>>> > 1. You might be using a StandardRequestHandler. Try Dismax.
>>>> > 2. Your field "Source" might not be indexed.
>>>> >
>>>> > With the StandardRequestHandler, if your "sources" are known (and
>>>> limited),
>>>> > you can try this too -
>>>> > q=hello +(source:source1^100 source:source2^0.1 ...)
>>>> >
>>>> > Cheers
>>>> > Avlesh
>>>> >
>>>> > On Thu, Nov 5, 2009 at 11:42 AM, Bhavnik Gajjar <
>>>> > bhavnik.gajjar@...> wrote:
>>>> >
>>>> >> Hi,
>>>> >>
>>>> >> I have tried with
>>>> >> q=hello&start=0&rows=20&fl=Source&bq=(*:* -Source:Source2^10000)
>>>> >>
>>>> >> But it doesn't change any ordering of results
>>>> >>
>>>> >> Am I doing wrong? Please advise.
>>>> >>
>>>> >> Note: We are also using _val_ hook in Solr query to get recent documents
>>>> on
>>>> >> top of results. (_val_:"recip(rord(DateAdded),1,1000,1000)")
>>>> >>
>>>> >> It would be really good if you can give me an exact example of Solr
>>>> query
>>>> >> that meets my need
>>>> >>
>>>> >> Thanks,
>>>> >> Bhavnik
>>>> >>
>>>> >> ----- Original Message -----
>>>> >> From: "Avlesh Singh" <avlesh@...>
>>>> >> To: <solr-user@...>
>>>> >> Sent: Thursday, November 05, 2009 11:19 AM
>>>> >> Subject: Re: Lowering ranking of certain documents while search in Solr
>>>> >>
>>>> >>
>>>> >> > This is what I meant -
>>>> >> >
>>>> >>
>>>> http://wiki.apache.org/solr/SolrRelevancyFAQ#How_do_I_give_a_very_low_boost_to_documents_that_match_my_query
>>>> >> >
>>>> >> > Cheers
>>>> >> > Avlesh
>>>> >> >
>>>> >> > On Thu, Nov 5, 2009 at 11:13 AM, Bhavnik Gajjar <
>>>> >> > bhavnik.gajjar@...> wrote:
>>>> >> >
>>>> >> >> Hi Avlesh,
>>>> >> >>
>>>> >> >> Thanks for your inputs.
>>>> >> >>
>>>> >> >> What exactly I mean is.
>>>> >> >>
>>>> >> >> Solr has two fields named, Text and Source. Search is performed in
>>>> >> [Text]
>>>> >> >> field. Now suppose 1000 results come for a search. By default, this
>>>> >> search
>>>> >> >> returns records according to boost factor specified in Solr query
>>>> >> already.
>>>> >> >> Result would look somewhat like..
>>>> >> >> Documents with value in [Source] Solr field
>>>> >> >> Source1
>>>> >> >> Source2
>>>> >> >> Source3
>>>> >> >> Source1
>>>> >> >> Source3
>>>> >> >> Source2
>>>> >> >> etc..
>>>> >> >>
>>>> >> >> But, what we want to achieve is.. this search should return documents
>>>> >> which
>>>> >> >> has value [Source2] in [Source] field should come with the lowest
>>>> >> ranking
>>>> >> >> like,
>>>> >> >> Source1
>>>> >> >> Source3
>>>> >> >> Source1
>>>> >> >> Source3
>>>> >> >> Source2
>>>> >> >> Source2
>>>> >> >>
>>>> >> >> The solution you gave actually affects boosting of [Text] solr field
>>>> not
>>>> >> >> [Source] field.
>>>> >> >>
>>>> >> >> Thanks,
>>>> >> >> Bhavnik Gajjar
>>>> >> >>
>>>> >> >>
>>>> >> >> ----- Original Message -----
>>>> >> >> From: "Avlesh Singh" <avlesh@...>
>>>> >> >> To: <solr-user@...>
>>>> >> >> Sent: Thursday, November 05, 2009 10:19 AM
>>>> >> >> Subject: Re: Lowering ranking of certain documents while search in
>>>> Solr
>>>> >> >>
>>>> >> >>
>>>> >> >> > Query terms can be "boosted" to alter rank of documents. Read more
>>>> >> here -
>>>> >> >> >
>>>> >>
>>>> http://wiki.apache.org/solr/SolrRelevancyCookbook#Boosting_Ranking_Terms
>>>> >> >> >
>>>> >> >> > Cheers
>>>> >> >> > Avlesh
>>>> >> >> >
>>>> >> >> > On Thu, Nov 5, 2009 at 10:14 AM, Bhavnik Gajjar <
>>>> >> >> > bhavnik.gajjar@...> wrote:
>>>> >> >> >
>>>> >> >> >> Hi,
>>>> >> >> >>
>>>> >> >> >> Suppose, there are some documents already stored in Solr. These
>>>> >> >> >> documents/records belong to various sources like, source1, source2
>>>> >> etc
>>>> >> >> >> (stored in 'Source' Solr field). Now, when user searches for
>>>> >> documents
>>>> >> >> >> (simple text search) then, is there any possibilities in Solr so
>>>> that
>>>> >> >> >> results of certain sources (eg, Source2) always come with lower
>>>> rank?
>>>> >> >> (ie,
>>>> >> >> >> such sources always come in trailing pages).
>>>> >> >> >>
>>>> >> >> >> I believe, there should be some way for this in functional query;
>>>> but
>>>> >> >> not
>>>> >> >> >> sure!
>>>> >> >> >>
>>>> >> >> >> Any help on this is greately appreciated.
>>>> >> >> >>
>>>> >> >> >> Thanks in advance,
>>>> >> >> >> Bhavnik Gajjar
>>>> >> >> >>
>>>> >> >> >>
>>>> >> >> >> The contents of this eMail including the contents of attachment(s)
>>>> >> are
>>>> >> >> >> privileged and confidential material of Gateway NINtec Pvt. Ltd.
>>>> >> (GNPL)
>>>> >> >> and
>>>> >> >> >> should not be disclosed to, used by or copied in any manner by
>>>> anyone
>>>> >> >> other
>>>> >> >> >> than the intended addressee(s). If this eMail has been received by
>>>> >> >> error,
>>>> >> >> >> please advise the sender immediately and delete it from your
>>>> system.
>>>> >> The
>>>> >> >> >> views expressed in this eMail message are those of the individual
>>>> >> >> sender,
>>>> >> >> >> except where the sender expressly, and with authority, states them
>>>> to
>>>> >> be
>>>> >> >> the
>>>> >> >> >> views of GNPL. Any unauthorized review, use, disclosure,
>>>> >> dissemination,
>>>> >> >> >> forwarding, printing or copying of this eMail or any action taken
>>>> in
>>>> >> >> >> reliance on this eMail is strictly prohibited and may be unlawful.
>>>> >> This
>>>> >> >> >> eMail may contain viruses. GNPL has taken every reasonable
>>>> precaution
>>>> >> to
>>>> >> >> >> minimize this risk, but is not liable for any damage you may
>>>> sustain
>>>> >> as
>>>> >> >> a
>>>> >> >> >> result of any virus in this eMail. You should carry out your own
>>>> >> virus
>>>> >> >> >> checks before opening the eMail or attachment(s). GNPL is neither
>>>> >> liable
>>>> >> >> for
>>>> >> >> >> the proper and complete transmission of the information contained
>>>> in
>>>> >> >> this
>>>> >> >> >> communication nor for any delay in its receipt. GNPL reserves the
>>>> >> right
>>>> >> >> to
>>>> >> >> >> monitor and review the content of all messages sent to or from
>>>> this
>>>> >> >> eMail
>>>> >> >> >> address and may be stored on the GNPL eMail system. In case this
>>>> >> eMail
>>>> >> >> has
>>>> >> >> >> reached you in error, and you would no longer like to receive
>>>> eMails
>>>> >> >> from
>>>> >> >> >> us, then please send an eMail to dnd@...
>>>> >> >> >>
>>>> >> >> >
>>>> >> >>
>>>> >> >>
>>>> >> >> The contents of this eMail including the contents of attachment(s)
>>>> are
>>>> >> >> privileged and confidential material of Gateway NINtec Pvt. Ltd.
>>>> (GNPL)
>>>> >> and
>>>> >> >> should not be disclosed to, used by or copied in any manner by anyone
>>>> >> other
>>>> >> >> than the intended addressee(s). If this eMail has been received by
>>>> >> error,
>>>> >> >> please advise the sender immediately and delete it from your system.
>>>> The
>>>> >> >> views expressed in this eMail message are those of the individual
>>>> >> sender,
>>>> >> >> except where the sender expressly, and with authority, states them to
>>>> be
>>>> >> the
>>>> >> >> views of GNPL. Any unauthorized review, use, disclosure,
>>>> dissemination,
>>>> >> >> forwarding, printing or copying of this eMail or any action taken in
>>>> >> >> reliance on this eMail is strictly prohibited and may be unlawful.
>>>> This
>>>> >> >> eMail may contain viruses. GNPL has taken every reasonable precaution
>>>> to
>>>> >> >> minimize this risk, but is not liable for any damage you may sustain
>>>> as
>>>> >> a
>>>> >> >> result of any virus in this eMail. You should carry out your own
>>>> virus
>>>> >> >> checks before opening the eMail or attachment(s). GNPL is neither
>>>> liable
>>>> >> for
>>>> >> >> the proper and complete transmission of the information contained in
>>>> >> this
>>>> >> >> communication nor for any delay in its receipt. GNPL reserves the
>>>> right
>>>> >> to
>>>> >> >> monitor and review the content of all messages sent to or from this
>>>> >> eMail
>>>> >> >> address and may be stored on the GNPL eMail system. In case this
>>>> eMail
>>>> >> has
>>>> >> >> reached you in error, and you would no longer like to receive eMails
>>>> >> from
>>>> >> >> us, then please send an eMail to dnd@...
>>>> >> >>
>>>> >> >
>>>> >>
>>>> >>
>>>> >> The contents of this eMail including the contents of attachment(s) are
>>>> >> privileged and confidential material of Gateway NINtec Pvt. Ltd. (GNPL)
>>>> and
>>>> >> should not be disclosed to, used by or copied in any manner by anyone
>>>> other
>>>> >> than the intended addressee(s). If this eMail has been received by
>>>> error,
>>>> >> please advise the sender immediately and delete it from your system. The
>>>> >> views expressed in this eMail message are those of the individual
>>>> sender,
>>>> >> except where the sender expressly, and with authority, states them to be
>>>> the
>>>> >> views of GNPL. Any unauthorized review, use, disclosure, dissemination,
>>>> >> forwarding, printing or copying of this eMail or any action taken in
>>>> >> reliance on this eMail is strictly prohibited and may be unlawful. This
>>>> >> eMail may contain viruses. GNPL has taken every reasonable precaution to
>>>> >> minimize this risk, but is not liable for any damage you may sustain as
>>>> a
>>>> >> result of any virus in this eMail. You should carry out your own virus
>>>> >> checks before opening the eMail or attachment(s). GNPL is neither liable
>>>> for
>>>> >> the proper and complete transmission of the information contained in
>>>> this
>>>> >> communication nor for any delay in its receipt. GNPL reserves the right
>>>> to
>>>> >> monitor and review the content of all messages sent to or from this
>>>> eMail
>>>> >> address and may be stored on the GNPL eMail system. In case this eMail
>>>> has
>>>> >> reached you in error, and you would no longer like to receive eMails
>>>> from
>>>> >> us, then please send an eMail to dnd@...
>>>> >>
>>>> >
>>>>
>>>>
>>>> The contents of this eMail including the contents of attachment(s) are
>>>> privileged and confidential material of Gateway NINtec Pvt. Ltd. (GNPL) and
>>>> should not be disclosed to, used by or copied in any manner by anyone other
>>>> than the intended addressee(s). If this eMail has been received by error,
>>>> please advise the sender immediately and delete it from your system. The
>>>> views expressed in this eMail message are those of the individual sender,
>>>> except where the sender expressly, and with authority, states them to be the
>>>> views of GNPL. Any unauthorized review, use, disclosure, dissemination,
>>>> forwarding, printing or copying of this eMail or any action taken in
>>>> reliance on this eMail is strictly prohibited and may be unlawful. This
>>>> eMail may contain viruses. GNPL has taken every reasonable precaution to
>>>> minimize this risk, but is not liable for any damage you may sustain as a
>>>> result of any virus in this eMail. You should carry out your own virus
>>>> checks before opening the eMail or attachment(s). GNPL is neither liable for
>>>> the proper and complete transmission of the information contained in this
>>>> communication nor for any delay in its receipt. GNPL reserves the right to
>>>> monitor and review the content of all messages sent to or from this eMail
>>>> address and may be stored on the GNPL eMail system. In case this eMail has
>>>> reached you in error, and you would no longer like to receive eMails from
>>>> us, then please send an eMail to dnd@...
>>>>
>>>
>>
>>
>> The contents of this eMail including the contents of attachment(s) are privileged and confidential material of Gateway NINtec Pvt. Ltd. (GNPL) and should not be disclosed to, used by or copied in any manner by anyone other than the intended addressee(s). If this eMail has been received by error, please advise the sender immediately and delete it from your system. The views expressed in this eMail message are those of the individual sender, except where the sender expressly, and with authority, states them to be the views of GNPL. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this eMail or any action taken in reliance on this eMail is strictly prohibited and may be unlawful. This eMail may contain viruses. GNPL has taken every reasonable precaution to minimize this risk, but is not liable for any damage you may sustain as a result of any virus in this eMail. You should carry out your own virus checks before opening the eMail or attachment(s). GNPL is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt. GNPL reserves the right to monitor and review the content of all messages sent to or from this eMail address and may be stored on the GNPL eMail system. In case this eMail has reached you in error, and you would no longer like to receive eMails from us, then please send an eMail to dnd@...
>>
>
>
>
> --
> Lance Norskog
> goksron@...
>

--
Lance Norskog
goksron@...

From forum: Solr - User

Re: Lowering ranking of certain documents while search in Solr

2009-11-09T17:36:52Z

The ExternalFileField is the tool for this.

It is a field type which reads an array of floating point values from
a file, one per document. You can boost on it. (I don't know if sort
works.) There is no direct documentation that I found. Here are some
hints.

http://www.lucidimagination.com/search/?q=ExternalFileField

http://lucene.apache.org/solr/api/org/apache/solr/schema/ExternalFileField.html

On Fri, Nov 6, 2009 at 9:44 PM, Bhavnik Gajjar
<bhavnik.gajjar@...> wrote:

> Thanks again,
>
> This solution requires me to re-index 2 millions documents again in Solr!!!
>
> I would prefer something which can be done in ad-hoc way on my existing working/live application.
>
> Also, regarding the solution [q=hello +(source:source1^100 source:source2^0.1 ...)], I think if sorting is being done on some other field (rather, score desc) then it will messup documents!
>
> Is there a way to use function query or so then?
>
> Regards,
> Bhavnik Gajjar
>
> ----- Original Message -----
> From: "Avlesh Singh" <avlesh@...>
> To: <solr-user@...>
> Sent: Friday, November 06, 2009 10:25 PM
> Subject: Re: Lowering ranking of certain documents while search in Solr
>
>
>> How about adding an extra sint field, say "source_boost". For your 199
>> sources the indexed value can be 0. For the "underprivileged" source it can
>> be -1. Adding "sort=source_boost desc,score desc" would do the needful.
>>
>> Works?
>>
>> Cheers
>> Avlesh
>>
>> On Fri, Nov 6, 2009 at 4:55 PM, Bhavnik Gajjar <
>> bhavnik.gajjar@...> wrote:
>>
>>> Thanks for your inputs,
>>>
>>> This solution works in general!!
>>>
>>> In my system, there are around 200 different sources and out of that only
>>> one source is needed with lower rank. Now, if I go with the solution you
>>> provided then I need 199 sources with individual boost and then one source
>>> with lower boost value. Now, I'm afraid that, this would cause performance
>>> issue in Solr. Is there any other solution to work with this then?
>>>
>>> Note: our massive application is already built with standard request
>>> handler (I can't use dismax at this stage!!)
>>>
>>> Seeking for positive reply!
>>>
>>> Thanks again,
>>> Bhavnik Gajjar
>>>
>>>
>>> ----- Original Message -----
>>> From: "Avlesh Singh" <avlesh@...>
>>> To: <solr-user@...>
>>> Sent: Thursday, November 05, 2009 12:00 PM
>>> Subject: Re: Lowering ranking of certain documents while search in Solr
>>>
>>>
>>> > The query "q=hello&start=0&rows=20&fl=Source&bq=(*:*
>>> -Source:Source2^10000)"
>>> > should work as expected. There can be two reasons for it not to work -
>>> >
>>> > 1. You might be using a StandardRequestHandler. Try Dismax.
>>> > 2. Your field "Source" might not be indexed.
>>> >
>>> > With the StandardRequestHandler, if your "sources" are known (and
>>> limited),
>>> > you can try this too -
>>> > q=hello +(source:source1^100 source:source2^0.1 ...)
>>> >
>>> > Cheers
>>> > Avlesh
>>> >
>>> > On Thu, Nov 5, 2009 at 11:42 AM, Bhavnik Gajjar <
>>> > bhavnik.gajjar@...> wrote:
>>> >
>>> >> Hi,
>>> >>
>>> >> I have tried with
>>> >> q=hello&start=0&rows=20&fl=Source&bq=(*:* -Source:Source2^10000)
>>> >>
>>> >> But it doesn't change any ordering of results
>>> >>
>>> >> Am I doing wrong? Please advise.
>>> >>
>>> >> Note: We are also using _val_ hook in Solr query to get recent documents
>>> on
>>> >> top of results. (_val_:"recip(rord(DateAdded),1,1000,1000)")
>>> >>
>>> >> It would be really good if you can give me an exact example of Solr
>>> query
>>> >> that meets my need
>>> >>
>>> >> Thanks,
>>> >> Bhavnik
>>> >>
>>> >> ----- Original Message -----
>>> >> From: "Avlesh Singh" <avlesh@...>
>>> >> To: <solr-user@...>
>>> >> Sent: Thursday, November 05, 2009 11:19 AM
>>> >> Subject: Re: Lowering ranking of certain documents while search in Solr
>>> >>
>>> >>
>>> >> > This is what I meant -
>>> >> >
>>> >>
>>> http://wiki.apache.org/solr/SolrRelevancyFAQ#How_do_I_give_a_very_low_boost_to_documents_that_match_my_query
>>> >> >
>>> >> > Cheers
>>> >> > Avlesh
>>> >> >
>>> >> > On Thu, Nov 5, 2009 at 11:13 AM, Bhavnik Gajjar <
>>> >> > bhavnik.gajjar@...> wrote:
>>> >> >
>>> >> >> Hi Avlesh,
>>> >> >>
>>> >> >> Thanks for your inputs.
>>> >> >>
>>> >> >> What exactly I mean is.
>>> >> >>
>>> >> >> Solr has two fields named, Text and Source. Search is performed in
>>> >> [Text]
>>> >> >> field. Now suppose 1000 results come for a search. By default, this
>>> >> search
>>> >> >> returns records according to boost factor specified in Solr query
>>> >> already.
>>> >> >> Result would look somewhat like..
>>> >> >> Documents with value in [Source] Solr field
>>> >> >> Source1
>>> >> >> Source2
>>> >> >> Source3
>>> >> >> Source1
>>> >> >> Source3
>>> >> >> Source2
>>> >> >> etc..
>>> >> >>
>>> >> >> But, what we want to achieve is.. this search should return documents
>>> >> which
>>> >> >> has value [Source2] in [Source] field should come with the lowest
>>> >> ranking
>>> >> >> like,
>>> >> >> Source1
>>> >> >> Source3
>>> >> >> Source1
>>> >> >> Source3
>>> >> >> Source2
>>> >> >> Source2
>>> >> >>
>>> >> >> The solution you gave actually affects boosting of [Text] solr field
>>> not
>>> >> >> [Source] field.
>>> >> >>
>>> >> >> Thanks,
>>> >> >> Bhavnik Gajjar
>>> >> >>
>>> >> >>
>>> >> >> ----- Original Message -----
>>> >> >> From: "Avlesh Singh" <avlesh@...>
>>> >> >> To: <solr-user@...>
>>> >> >> Sent: Thursday, November 05, 2009 10:19 AM
>>> >> >> Subject: Re: Lowering ranking of certain documents while search in
>>> Solr
>>> >> >>
>>> >> >>
>>> >> >> > Query terms can be "boosted" to alter rank of documents. Read more
>>> >> here -
>>> >> >> >
>>> >>
>>> http://wiki.apache.org/solr/SolrRelevancyCookbook#Boosting_Ranking_Terms
>>> >> >> >
>>> >> >> > Cheers
>>> >> >> > Avlesh
>>> >> >> >
>>> >> >> > On Thu, Nov 5, 2009 at 10:14 AM, Bhavnik Gajjar <
>>> >> >> > bhavnik.gajjar@...> wrote:
>>> >> >> >
>>> >> >> >> Hi,
>>> >> >> >>
>>> >> >> >> Suppose, there are some documents already stored in Solr. These
>>> >> >> >> documents/records belong to various sources like, source1, source2
>>> >> etc
>>> >> >> >> (stored in 'Source' Solr field). Now, when user searches for
>>> >> documents
>>> >> >> >> (simple text search) then, is there any possibilities in Solr so
>>> that
>>> >> >> >> results of certain sources (eg, Source2) always come with lower
>>> rank?
>>> >> >> (ie,
>>> >> >> >> such sources always come in trailing pages).
>>> >> >> >>
>>> >> >> >> I believe, there should be some way for this in functional query;
>>> but
>>> >> >> not
>>> >> >> >> sure!
>>> >> >> >>
>>> >> >> >> Any help on this is greately appreciated.
>>> >> >> >>
>>> >> >> >> Thanks in advance,
>>> >> >> >> Bhavnik Gajjar
>>> >> >> >>
>>> >> >> >>
>>> >> >> >> The contents of this eMail including the contents of attachment(s)
>>> >> are
>>> >> >> >> privileged and confidential material of Gateway NINtec Pvt. Ltd.
>>> >> (GNPL)
>>> >> >> and
>>> >> >> >> should not be disclosed to, used by or copied in any manner by
>>> anyone
>>> >> >> other
>>> >> >> >> than the intended addressee(s). If this eMail has been received by
>>> >> >> error,
>>> >> >> >> please advise the sender immediately and delete it from your
>>> system.
>>> >> The
>>> >> >> >> views expressed in this eMail message are those of the individual
>>> >> >> sender,
>>> >> >> >> except where the sender expressly, and with authority, states them
>>> to
>>> >> be
>>> >> >> the
>>> >> >> >> views of GNPL. Any unauthorized review, use, disclosure,
>>> >> dissemination,
>>> >> >> >> forwarding, printing or copying of this eMail or any action taken
>>> in
>>> >> >> >> reliance on this eMail is strictly prohibited and may be unlawful.
>>> >> This
>>> >> >> >> eMail may contain viruses. GNPL has taken every reasonable
>>> precaution
>>> >> to
>>> >> >> >> minimize this risk, but is not liable for any damage you may
>>> sustain
>>> >> as
>>> >> >> a
>>> >> >> >> result of any virus in this eMail. You should carry out your own
>>> >> virus
>>> >> >> >> checks before opening the eMail or attachment(s). GNPL is neither
>>> >> liable
>>> >> >> for
>>> >> >> >> the proper and complete transmission of the information contained
>>> in
>>> >> >> this
>>> >> >> >> communication nor for any delay in its receipt. GNPL reserves the
>>> >> right
>>> >> >> to
>>> >> >> >> monitor and review the content of all messages sent to or from
>>> this
>>> >> >> eMail
>>> >> >> >> address and may be stored on the GNPL eMail system. In case this
>>> >> eMail
>>> >> >> has
>>> >> >> >> reached you in error, and you would no longer like to receive
>>> eMails
>>> >> >> from
>>> >> >> >> us, then please send an eMail to dnd@...
>>> >> >> >>
>>> >> >> >
>>> >> >>
>>> >> >>
>>> >> >> The contents of this eMail including the contents of attachment(s)
>>> are
>>> >> >> privileged and confidential material of Gateway NINtec Pvt. Ltd.
>>> (GNPL)
>>> >> and
>>> >> >> should not be disclosed to, used by or copied in any manner by anyone
>>> >> other
>>> >> >> than the intended addressee(s). If this eMail has been received by
>>> >> error,
>>> >> >> please advise the sender immediately and delete it from your system.
>>> The
>>> >> >> views expressed in this eMail message are those of the individual
>>> >> sender,
>>> >> >> except where the sender expressly, and with authority, states them to
>>> be
>>> >> the
>>> >> >> views of GNPL. Any unauthorized review, use, disclosure,
>>> dissemination,
>>> >> >> forwarding, printing or copying of this eMail or any action taken in
>>> >> >> reliance on this eMail is strictly prohibited and may be unlawful.
>>> This
>>> >> >> eMail may contain viruses. GNPL has taken every reasonable precaution
>>> to
>>> >> >> minimize this risk, but is not liable for any damage you may sustain
>>> as
>>> >> a
>>> >> >> result of any virus in this eMail. You should carry out your own
>>> virus
>>> >> >> checks before opening the eMail or attachment(s). GNPL is neither
>>> liable
>>> >> for
>>> >> >> the proper and complete transmission of the information contained in
>>> >> this
>>> >> >> communication nor for any delay in its receipt. GNPL reserves the
>>> right
>>> >> to
>>> >> >> monitor and review the content of all messages sent to or from this
>>> >> eMail
>>> >> >> address and may be stored on the GNPL eMail system. In case this
>>> eMail
>>> >> has
>>> >> >> reached you in error, and you would no longer like to receive eMails
>>> >> from
>>> >> >> us, then please send an eMail to dnd@...
>>> >> >>
>>> >> >
>>> >>
>>> >>
>>> >> The contents of this eMail including the contents of attachment(s) are
>>> >> privileged and confidential material of Gateway NINtec Pvt. Ltd. (GNPL)
>>> and
>>> >> should not be disclosed to, used by or copied in any manner by anyone
>>> other
>>> >> than the intended addressee(s). If this eMail has been received by
>>> error,
>>> >> please advise the sender immediately and delete it from your system. The
>>> >> views expressed in this eMail message are those of the individual
>>> sender,
>>> >> except where the sender expressly, and with authority, states them to be
>>> the
>>> >> views of GNPL. Any unauthorized review, use, disclosure, dissemination,
>>> >> forwarding, printing or copying of this eMail or any action taken in
>>> >> reliance on this eMail is strictly prohibited and may be unlawful. This
>>> >> eMail may contain viruses. GNPL has taken every reasonable precaution to
>>> >> minimize this risk, but is not liable for any damage you may sustain as
>>> a
>>> >> result of any virus in this eMail. You should carry out your own virus
>>> >> checks before opening the eMail or attachment(s). GNPL is neither liable
>>> for
>>> >> the proper and complete transmission of the information contained in
>>> this
>>> >> communication nor for any delay in its receipt. GNPL reserves the right
>>> to
>>> >> monitor and review the content of all messages sent to or from this
>>> eMail
>>> >> address and may be stored on the GNPL eMail system. In case this eMail
>>> has
>>> >> reached you in error, and you would no longer like to receive eMails
>>> from
>>> >> us, then please send an eMail to dnd@...
>>> >>
>>> >
>>>
>>>
>>> The contents of this eMail including the contents of attachment(s) are
>>> privileged and confidential material of Gateway NINtec Pvt. Ltd. (GNPL) and
>>> should not be disclosed to, used by or copied in any manner by anyone other
>>> than the intended addressee(s). If this eMail has been received by error,
>>> please advise the sender immediately and delete it from your system. The
>>> views expressed in this eMail message are those of the individual sender,
>>> except where the sender expressly, and with authority, states them to be the
>>> views of GNPL. Any unauthorized review, use, disclosure, dissemination,
>>> forwarding, printing or copying of this eMail or any action taken in
>>> reliance on this eMail is strictly prohibited and may be unlawful. This
>>> eMail may contain viruses. GNPL has taken every reasonable precaution to
>>> minimize this risk, but is not liable for any damage you may sustain as a
>>> result of any virus in this eMail. You should carry out your own virus
>>> checks before opening the eMail or attachment(s). GNPL is neither liable for
>>> the proper and complete transmission of the information contained in this
>>> communication nor for any delay in its receipt. GNPL reserves the right to
>>> monitor and review the content of all messages sent to or from this eMail
>>> address and may be stored on the GNPL eMail system. In case this eMail has
>>> reached you in error, and you would no longer like to receive eMails from
>>> us, then please send an eMail to dnd@...
>>>
>>
>
>
> The contents of this eMail including the contents of attachment(s) are privileged and confidential material of Gateway NINtec Pvt. Ltd. (GNPL) and should not be disclosed to, used by or copied in any manner by anyone other than the intended addressee(s). If this eMail has been received by error, please advise the sender immediately and delete it from your system. The views expressed in this eMail message are those of the individual sender, except where the sender expressly, and with authority, states them to be the views of GNPL. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this eMail or any action taken in reliance on this eMail is strictly prohibited and may be unlawful. This eMail may contain viruses. GNPL has taken every reasonable precaution to minimize this risk, but is not liable for any damage you may sustain as a result of any virus in this eMail. You should carry out your own virus checks before opening the eMail or attachment(s). GNPL is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt. GNPL reserves the right to monitor and review the content of all messages sent to or from this eMail address and may be stored on the GNPL eMail system. In case this eMail has reached you in error, and you would no longer like to receive eMails from us, then please send an eMail to dnd@...
>

--
Lance Norskog
goksron@...

From forum: Solr - User

RE: Solr on OOM

2009-11-09T17:36:13Z

Ok, I'll see if I can get their heap dump. Thanks.

-----Original Message-----
From:
solr-user-return-29005-Laurent.Vauthrin=disney.com@...
[mailto:solr-user-return-29005-Laurent.Vauthrin=disney.com@...
.org] On Behalf Of Otis Gospodnetic
Sent: Monday, November 09, 2009 2:17 PM
To: solr-user@...
Subject: Re: Solr on OOM

Laurent,

The autowarmCounts look biggest, but they are probably not causing OOMs.
Maybe you can see how big the caches are right before you OOM.
Or you can also start the JVM with -XX:+HeapDumpOnOutOfMemoryError and
even specify the file where the heap should be dumped. You can then
analyze it and see what's eating the memory.

Otis
--
Sematext is hiring -- http://sematext.com/about/jobs.html?mls
Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR

----- Original Message ----

> From: "Vauthrin, Laurent" <Laurent.Vauthrin@...>
> To: solr-user@...
> Sent: Mon, November 9, 2009 2:22:26 PM
> Subject: Solr on OOM
>
> Hello,
>
>
>
> One of our deployed Solr (1.3) setup is having out of memory issues

and
> I'm not sure how to troubleshoot it. I've read a few posts (including
>
http://old.nabble.com/Debugging-Solr-memory-usage-heap-problems-ts883279

> 4.html#a8832794) but I think this situation is slightly different.
>
>
>
> Here's the setup:
>
> 1 master and 1 slave are located on a the same VM (using a 64-bit JVM)
>
> 1 slave running on its own VM (using a 64-bit JVM)
>
> From what I've been told, nothing else is running on those VMs.
>
> Index size is about 100-200 MB.
>
>
>
> Solrconfig.xml cache settings:
>
>
>
>
> autowarmCount="5000"/>
>
>
> initialSize="5000" autowarmCount="5000"/>
>
>
> initialSize="5000"/>
>
>
>
> Both slaves at some point have gone out of memory (though not both at
> the same time) when receiving a moderate load of queries (a few

queries
> per second - don't have an exact stat here). We started with a heap
> size of 1GB and ended up having to bump it up to 3.5GB. It seems
really
> odd that we'd have to have a heap size that large when the index
itself
> is not really big. Any thoughts on what could be really off here?
Is
> there a way to determine the cache sizes in bytes? I noticed that
there
> was a thread about other issues running Solr on VMs, has anyone else
had
> problems using VMWare? From what I'm told, it seems like moving to
> physical servers won't be a fast/easy change so I'm looking for any
help
> I can get for this configuration.
>
>
>
> Thanks,
> Laurent Vauthrin

From forum: Solr - User

RE: Segment file not found error - after replicating

2009-11-09T16:47:04Z

Thanks Otis!

Yes, I checked the index directories and they are 100% same, both timestamp and size wise.

Not all the slaves face this issue. I would say roughly 50% has this trouble.

Logs do not have any errors too :-(

Any other things I should do/look at?

Cheers
Madu

-----Original Message-----
From: Otis Gospodnetic [mailto:otis_gospodnetic@...]
Sent: Tuesday, 10 November 2009 9:26 AM
To: solr-user@...
Subject: Re: Segment file not found error - after replicating

It's hard to troubleshoot blindly like this, but have you tried manually comparing the contents of the index dir on the master and on the slave(s)?
If they are out of sync, have you tried forcing of replication to see if one of the subsequent replication attempts gets the dirs in sync?
Do you have more than 1 slave and do they all start having this problem at the same time?
Any errors in the logs for any of the scripts involved in replication in 1.3?

Otis
--
Sematext is hiring -- http://sematext.com/about/jobs.html?mls
Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR

----- Original Message ----

> From: Maduranga Kannangara <mkannangara@...>
> To: "solr-user@..." <solr-user@...>
> Sent: Sun, November 8, 2009 10:30:44 PM
> Subject: Segment file not found error - after replicating
>
> Hi guys,
>
> We use Solr 1.3 for indexing large amounts of data (50G avg) on Linux
> environment and use the replication scripts to make replicas those live in load
> balancing slaves.
>
> The issue we face quite often (only in Linux servers) is that they tend to not
> been able to find the segment file (segment_x etc) after the replicating
> completed. As this has become quite common, we started hitting a serious issue.
>
> Below is a stack trace, if that helps and any help on this matter is greatly
> appreciated.
>
> --------------------------------
>
> Nov 5, 2009 11:34:46 PM org.apache.solr.util.plugin.AbstractPluginLoader load
> INFO: created /admin/: org.apache.solr.handler.admin.AdminHandlers
> Nov 5, 2009 11:34:46 PM org.apache.solr.util.plugin.AbstractPluginLoader load
> INFO: created /admin/ping: org.apache.solr.handler.PingRequestHandler
> Nov 5, 2009 11:34:46 PM org.apache.solr.util.plugin.AbstractPluginLoader load
> INFO: created /debug/dump: org.apache.solr.handler.DumpRequestHandler
> Nov 5, 2009 11:34:46 PM org.apache.solr.util.plugin.AbstractPluginLoader load
> INFO: created gap: org.apache.solr.highlight.GapFragmenter
> Nov 5, 2009 11:34:46 PM org.apache.solr.util.plugin.AbstractPluginLoader load
> INFO: created regex: org.apache.solr.highlight.RegexFragmenter
> Nov 5, 2009 11:34:46 PM org.apache.solr.util.plugin.AbstractPluginLoader load
> INFO: created html: org.apache.solr.highlight.HtmlFormatter
> Nov 5, 2009 11:34:46 PM org.apache.solr.servlet.SolrDispatchFilter init
> SEVERE: Could not start SOLR. Check solr/home property
> java.lang.RuntimeException: java.io.FileNotFoundException:
> /solrinstances/solrhome01/data/index/segments_v (No such file or directory)
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:960)
> at org.apache.solr.core.SolrCore.(SolrCore.java:470)
> at
> org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:119)
> at
> org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:69)
> at
> org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:275)
> at
> org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:397)
> at
> org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:108)
> at
> org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:3709)
> at
> org.apache.catalina.core.StandardContext.start(StandardContext.java:4363)
> at
> org.apache.catalina.core.StandardContext.reload(StandardContext.java:3099)
> at
> org.apache.catalina.manager.ManagerServlet.reload(ManagerServlet.java:916)
> at
> org.apache.catalina.manager.HTMLManagerServlet.reload(HTMLManagerServlet.java:536)
> at
> org.apache.catalina.manager.HTMLManagerServlet.doGet(HTMLManagerServlet.java:114)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at com.jamonapi.JAMonFilter.doFilter(JAMonFilter.java:57)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
> at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
> at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:845)
> at
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
> at
> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.FileNotFoundException:
> /solrinstances/solrhome01/data/index/segments_v (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(RandomAccessFile.java:212)
> at
> org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.(FSDirectory.java:552)
> at
> org.apache.lucene.store.FSDirectory$FSIndexInput.(FSDirectory.java:582)
> at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
> at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:482)
> at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:214)
> at
> org.apache.lucene.index.DirectoryIndexReader$1.doBody(DirectoryIndexReader.java:94)
> at
> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:653)
> at
> org.apache.lucene.index.DirectoryIndexReader.open(DirectoryIndexReader.java:111)
> at org.apache.lucene.index.IndexReader.open(IndexReader.java:316)
> at org.apache.lucene.index.IndexReader.open(IndexReader.java:237)
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:951)
> ... 30 more
> Nov 5, 2009 11:34:46 PM org.apache.solr.common.SolrException log
> SEVERE: java.lang.RuntimeException: java.io.FileNotFoundException:
> /solrinstances/solrhome01/data/index/segments_v (No such file or directory)
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:960)
> at org.apache.solr.core.SolrCore.(SolrCore.java:470)
> at
> org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:119)
> at
> org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:69)
> at
> org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:275)
> at
> org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:397)
> at
> org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:108)
> at
> org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:3709)
> at
> org.apache.catalina.core.StandardContext.start(StandardContext.java:4363)
> at
> org.apache.catalina.core.StandardContext.reload(StandardContext.java:3099)
> at
> org.apache.catalina.manager.ManagerServlet.reload(ManagerServlet.java:916)
> at
> org.apache.catalina.manager.HTMLManagerServlet.reload(HTMLManagerServlet.java:536)
> at
> org.apache.catalina.manager.HTMLManagerServlet.doGet(HTMLManagerServlet.java:114)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at com.jamonapi.JAMonFilter.doFilter(JAMonFilter.java:57)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
> at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
> at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:845)
> at
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
> at
> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.FileNotFoundException:
> /solrinstances/solrhome01/data/index/segments_v (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(RandomAccessFile.java:212)
> at
> org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.(FSDirectory.java:552)
> at
> org.apache.lucene.store.FSDirectory$FSIndexInput.(FSDirectory.java:582)
> at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
> at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:482)
> at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:214)
> at
> org.apache.lucene.index.DirectoryIndexReader$1.doBody(DirectoryIndexReader.java:94)
> at
> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:653)
> at
> org.apache.lucene.index.DirectoryIndexReader.open(DirectoryIndexReader.java:111)
> at org.apache.lucene.index.IndexReader.open(IndexReader.java:316)
> at org.apache.lucene.index.IndexReader.open(IndexReader.java:237)
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:951)
> ... 30 more
>
> Nov 5, 2009 11:34:46 PM org.apache.solr.servlet.SolrDispatchFilter init
> INFO: SolrDispatchFilter.init() done
> Nov 5, 2009 11:34:46 PM org.apache.solr.servlet.SolrServlet init
> INFO: SolrServlet.init()
>
> --------------------------------
>
> Steps to re-produce the error (However, for me did not work in my local box.
> Also remote server is too far away to remote-debug!).
>
> - Post some new data to the master server (Usually about 1Gb worth text files)
> - Run the replicate script in slave Solr instance
> - Try to login to admin in slave Solr instance
>
> And you should see above stack trace even in the Tomcat output.
>
>
> Thanks in advance.
> Madu

From forum: Solr - User

Solr 1.4 - Pre release post (portuguese)

2009-11-09T16:43:29Z

Hello all,

I've blogged about the Solr 1.4 and some of its features in my blog.
It's written in Brazilian Portuguese, hope you all can enjoy (thanks google
translate).

The link is: http://lucastex.com.br/2009/11/09/solr-1-4-mais-que-pronto/
<http://lucastex.com.br/2009/11/09/solr-1-4-mais-que-pronto/>

[]s,

Lucas Frare Teixeira .·.
- lucastex@...
- lucastex.com.br
- blog.lucastex.com
- twitter.com/lucastex

From forum: Solr - User

Oddness with Phrase Query

2009-11-09T16:06:06Z

I have a document with the title "Here, there be dragons" and a body.

When I search for

q = Here, there be dragons
qf = title^2.0 body^0.8
qt = dismax

Which is parsed as

+DisjunctionMaxQuery((content:"here dragon"^0.8 | title:"here
dragon"^2.0)~0.01) ()

I get the document as the first hit which is what I'd suspect.

However, if change the query to

q = "Here, there be dragons"

(with quotes)

which is parsed as

+DisjunctionMaxQuery((content:"here dragon"^0.8 | title:"here
dragon"^2.0)~0.01) ()

then I don't get the document at all. Which is not what I'd suspect.

I've tried modifying the phrase slop but still don't get any results
back.

Am I doing something wrong - do I have to have an untokenized copy of
fields lying around?

Thanks,

Simon

From forum: Solr - User

[jira] Updated: (SOLR-1551) Provide DocumentObjectBinder.getBean

2009-11-09T15:47:33Z

[ https://issues.apache.org/jira/browse/SOLR-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Christian Lopez Espinola updated SOLR-1551:
-------------------------------------------

Attachment: getBeanPatch.patch

Patch that provides the DocumentObjectBinder.getBean and unit test.

> Provide DocumentObjectBinder.getBean
> ------------------------------------
>
> Key: SOLR-1551
> URL: https://issues.apache.org/jira/browse/SOLR-1551
> Project: Solr
> Issue Type: Improvement
> Components: clients - java
> Affects Versions: 1.1.0, 1.2, 1.3
> Reporter: Christian Lopez Espinola
> Attachments: getBeanPatch.patch
>
>
> It would be useful being able of creating the beans individually, for people who have different entities indexed with a unique schema.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

From forum: Solr - Dev

[jira] Issue Comment Edited: (SOLR-236) Field collapsing

2009-11-09T15:47:32Z

[ https://issues.apache.org/jira/browse/SOLR-236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775192#action_12775192 ]

Michael Gundlach edited comment on SOLR-236 at 11/9/09 11:45 PM:
-----------------------------------------------------------------

I've found an NPE that occurs when performing quasi-distributed field collapsing.

My company only has one use case for field collapsing: collapsing on email address. Our index is spread across multiple cores. We found that if we shard by email address, so that all documents with a given email address are guaranteed to appear on the same core, then we can do distributed field collapsing.

We add &collapse.field=email and &shards=core1,core2,... to a regular query. Each core collapses on email and sends the results back to the requestor. Since no emails appear on more than one core, we've accomplished distributed search. We do lose the <collapse_count> section, but that's not needed for our purpose -- we just need an accurate total document count, and to have no more than one document for a given email address in the results.

Unfortunately, this throws an NPE when searching on a tokenized field. Searching string fields is fine. I don't understand exactly why the NPE appears, but I did bandaid over it by checking explicitly for nulls at the appropriate line in the code. No more NPE.

There's a downside, which is that if we attempt to collapse on a field other than email -- one which has documents appearing in multiple cores -- the results are buggy: the first search returns few documents, and the number of documents actually displayed don't always match the "numFound" value. Then upon refresh we get what we think is the correct numFound, and the correct list of documents. This doesn't bother me too much, as you're guaranteed to get incorrect answers from the collapse code anyway when collapsing on a field that you didn't use as your key for sharding.

In the spirit of Yonik's law of patches, I have made two imperfect patches attempting to contribute the fix, or at least point out the error:

1. I pulled trunk, applied the latest SOLR-236 patch, made my 2 line change, and created a patch file. The resultant patch file looks very different from the latest SOLR-236 patchfile, so I assume I did something wrong.

2. I pulled trunk, made my 2 line change, and created another patch file. This file is tiny but of course is missing all of the field collapsing changes.

Would you like me to post either of these patchfiles to this issue? Or is it sufficient to just tell you that the NPE occured in QueryComponent.java on line 556? ("rb._responseDocs.set(sdoc.positionInResponse, doc);" where sdoc was null.) Perhaps my use case is extraordinary enough that you're happy leaving the NPE in place and telling other users to not do what I'm doing?

Thanks!
Michael

was (Author: gundlach):
I've found an NPE that occurs when performing quasi-distributed field collapsing.

My company only has one use case for field collapsing: collapsing on email address. Our index is spread across multiple cores. We found that if we shard by email address, so that a given all documents with a given email address are guaranteed to appear on the same core, then we can do distributed field collapsing.

We add &collapse.field=email and &shards=core1,core2,... to a regular query. Each core collapses on email and sends the results back to the requestor. Since no emails appear on more than one core, we've accomplished distributed search. We do lose the <collapse_count> section, but that's not needed for our purpose -- we just need an accurate total document count, and to have no more than one document for a given email address in the results.

Unfortunately, this throws an NPE when searching on a tokenized field. Searching string fields is fine. I don't understand exactly why the NPE appears, but I did bandaid over it by checking explicitly for nulls at the appropriate line in the code. No more NPE.

There's a downside, which is that if we attempt to collapse on a field other than email -- one which has documents appearing in multiple cores -- the results are buggy: the first search returns few documents, and the number of documents actually displayed don't always match the "numFound" value. Then upon refresh we get what we think is the correct numFound, and the correct list of documents. This doesn't bother me too much, as you're guaranteed to get incorrect answers from the collapse code anyway when collapsing on a field that you didn't use as your key for sharding.

In the spirit of Yonik's law of patches, I have made two imperfect patches attempting to contribute the fix, or at least point out the error:

1. I pulled trunk, applied the latest SOLR-236 patch, made my 2 line change, and created a patch file. The resultant patch file looks very different from the latest SOLR-236 patchfile, so I assume I did something wrong.

2. I pulled trunk, made my 2 line change, and created another patch file. This file is tiny but of course is missing all of the field collapsing changes.

Would you like me to post either of these patchfiles to this issue? Or is it sufficient to just tell you that the NPE occured in QueryComponent.java on line 556? ("rb._responseDocs.set(sdoc.positionInResponse, doc);" where sdoc was null.) Perhaps my use case is extraordinary enough that you're happy leaving the NPE in place and telling other users to not do what I'm doing?

Thanks!
Michael

> Field collapsing
> ----------------
>
> Key: SOLR-236
> URL: https://issues.apache.org/jira/browse/SOLR-236
> Project: Solr
> Issue Type: New Feature
> Components: search
> Affects Versions: 1.3
> Reporter: Emmanuel Keller
> Fix For: 1.5
>
> Attachments: collapsing-patch-to-1.3.0-dieter.patch, collapsing-patch-to-1.3.0-ivan.patch, collapsing-patch-to-1.3.0-ivan_2.patch, collapsing-patch-to-1.3.0-ivan_3.patch, field-collapse-3.patch, field-collapse-4-with-solrj.patch, field-collapse-5.patch, field-collapse-5.patch, field-collapse-5.patch, field-collapse-5.patch, field-collapse-5.patch, field-collapse-5.patch, field-collapse-5.patch, field-collapse-5.patch, field-collapse-5.patch, field-collapse-solr-236-2.patch, field-collapse-solr-236.patch, field-collapsing-extended-592129.patch, field_collapsing_1.1.0.patch, field_collapsing_1.3.patch, field_collapsing_dsteigerwald.diff, field_collapsing_dsteigerwald.diff, field_collapsing_dsteigerwald.diff, SOLR-236-FieldCollapsing.patch, SOLR-236-FieldCollapsing.patch, SOLR-236-FieldCollapsing.patch, solr-236.patch, SOLR-236_collapsing.patch, SOLR-236_collapsing.patch
>
>
> This patch include a new feature called "Field collapsing".
> "Used in order to collapse a group of results with similar value for a given field to a single entry in the result set. Site collapsing is a special case of this, where all results for a given web site is collapsed into one or two entries in the result set, typically with an associated "more documents from this site" link. See also Duplicate detection."
> http://www.fastsearch.com/glossary.aspx?m=48&amid=299
> The implementation add 3 new query parameters (SolrParams):
> "collapse.field" to choose the field used to group results
> "collapse.type" normal (default value) or adjacent
> "collapse.max" to select how many continuous results are allowed before collapsing
> TODO (in progress):
> - More documentation (on source code)
> - Test cases
> Two patches:
> - "field_collapsing.patch" for current development version
> - "field_collapsing_1.1.0.patch" for Solr-1.1.0
> P.S.: Feedback and misspelling correction are welcome ;-)

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

From forum: Solr - Dev

[jira] Commented: (SOLR-236) Field collapsing

2009-11-09T15:45:32Z

[ https://issues.apache.org/jira/browse/SOLR-236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775192#action_12775192 ]

Michael Gundlach commented on SOLR-236:
---------------------------------------

I've found an NPE that occurs when performing quasi-distributed field collapsing.

My company only has one use case for field collapsing: collapsing on email address. Our index is spread across multiple cores. We found that if we shard by email address, so that a given all documents with a given email address are guaranteed to appear on the same core, then we can do distributed field collapsing.

We add &collapse.field=email and &shards=core1,core2,... to a regular query. Each core collapses on email and sends the results back to the requestor. Since no emails appear on more than one core, we've accomplished distributed search. We do lose the <collapse_count> section, but that's not needed for our purpose -- we just need an accurate total document count, and to have no more than one document for a given email address in the results.

Unfortunately, this throws an NPE when searching on a tokenized field. Searching string fields is fine. I don't understand exactly why the NPE appears, but I did bandaid over it by checking explicitly for nulls at the appropriate line in the code. No more NPE.

There's a downside, which is that if we attempt to collapse on a field other than email -- one which has documents appearing in multiple cores -- the results are buggy: the first search returns few documents, and the number of documents actually displayed don't always match the "numFound" value. Then upon refresh we get what we think is the correct numFound, and the correct list of documents. This doesn't bother me too much, as you're guaranteed to get incorrect answers from the collapse code anyway when collapsing on a field that you didn't use as your key for sharding.

In the spirit of Yonik's law of patches, I have made two imperfect patches attempting to contribute the fix, or at least point out the error:

1. I pulled trunk, applied the latest SOLR-236 patch, made my 2 line change, and created a patch file. The resultant patch file looks very different from the latest SOLR-236 patchfile, so I assume I did something wrong.

2. I pulled trunk, made my 2 line change, and created another patch file. This file is tiny but of course is missing all of the field collapsing changes.

Would you like me to post either of these patchfiles to this issue? Or is it sufficient to just tell you that the NPE occured in QueryComponent.java on line 556? ("rb._responseDocs.set(sdoc.positionInResponse, doc);" where sdoc was null.) Perhaps my use case is extraordinary enough that you're happy leaving the NPE in place and telling other users to not do what I'm doing?

Thanks!
Michael

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

From forum: Solr - Dev

[jira] Created: (SOLR-1551) Provide DocumentObjectBinder.getBean

2009-11-09T15:45:32Z

Provide DocumentObjectBinder.getBean
------------------------------------

Key: SOLR-1551
URL: https://issues.apache.org/jira/browse/SOLR-1551
Project: Solr
Issue Type: Improvement
Components: clients - java
Affects Versions: 1.3, 1.2, 1.1.0
Reporter: Christian Lopez Espinola

It would be useful being able of creating the beans individually, for people who have different entities indexed with a unique schema.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

From forum: Solr - Dev

Re: de-boosting certain facets during search

2009-11-09T15:28:01Z

I'm still going around in a circle on this. I'm not sure why it's not
sinking in...

If I could just create the desired URL, I can probably work backwards
and construct the correct ruby call.

Here is the URL that I'm currently creating (I've added newlines here
for readability):

http://localhost:8983/solr/resources/select?hl.fragsize=600
&hl=true
&facet.field=genre
&facet.field=archive
&facet.limit=-1
&qt=standard
&start=0
&fq=archive%3A%22blake%22
&hl.fl=text
&fl=uri%2Carchive%2Cdate_label%2Cgenre
&facet=true
&q=%28history%29
&rows=60
&facet.missing=true
&facet.mincount=1

What this search returns from my index is 53 hits. The first 43 contain
the genre field value "Citation" and the last 10 do not (they contain
other values in that field.)

Note: the genre field is multivalued, if that matters.

I'd like the search to put all of the objects that contain genre
"Citation" below the 10 objects that do not contain that genre.

I've read the various pages on boosting, but since I'm not actively
searching on the field that I want to put a boost value on, I'm not sure
how to go about this.

Thanks for any hints.

Paul Rosen wrote:

> Hi,
>
> I'm using solr-ruby-0.0.8 and solr 1.4.
>
> My data contains a faceted field called "genre". We would like one
> particular genre, (the one named "Citation") to show up last in the
> results.
>
> I'm having trouble figuring out how to add the boost parameter to the
> solr-ruby call. Here is my code:
>
> req = Solr::Request::Standard.new(:start => start,
> :rows => max,
> :sort => sort_param,
> :query => query,
> :filter_queries => filter_queries,
> :field_list => @field_list,
> :facets => {:fields => @facet_fields,
> :mincount => 1,
> :missing => true,
> :limit => -1},
> :highlighting => {:field_list => ['text'],
> :fragment_size => 600},
> :shards => @cores)
>
> response = @solr.send(req)
>
> Do I just format it inside my query, like this:
>
> query = query + "AND genre:Citation^.01"
>
> or in filter_query, like this:
>
> filter_queries.push("genre:Citation^.01")
>
> or is there a hash parameter that I set?
>
> (Note that the user can select Citation explicitly. I'll probably
> special case that.)
>
> I've tried variations of the above, but I've had no luck so far.
>
> Thanks,
> Paul

From forum: Solr - User