Nabble - Lucene - Ruby Developer

Re: reset solr index

2009-05-02T18:40:44Z

On May 1, 2009, at 11:32 PM, Geetha wrote:
> I've installed solr-ruby and integrated the same also.
>
> I need to clear the solr index.
>
> I need a way to clear the db and also need a function (through solr
> ruby) that will allow us to clear everything.

How are you integrating the DB and Solr? acts_as_solr?

Clearing the Solr index from solr-ruby is as easy as just sending a
delete_by_query of *:* and doing a commit. Details were replied to on
the solr-user list the other day:

<http://www.lucidimagination.com/search/document/ff32b0089f2bf142/how_to_reset_the_index_in_solr
>

Erik

reset solr index

2009-05-01T20:32:05Z

I have deployed rails application and using search engine is solr.

I've installed solr-ruby and integrated the same also.

I need to clear the solr index.

I need a way to clear the db and also need a function (through solr
ruby) that will allow us to clear everything.

--

Best Regards,
**
*Geetha S *| System and Software Engineer
email: geetha@... <mailto:geetha@...>

*
*

*
*Visit us at **Internet World, UK** (**28th-30th Apr 2009**)*
*Click here for FREE TICKETS:**
*http://www.angleritech.com/company/latest-technology-news-events.html
*

*ANGLER Technologies India - Your Offshore Development Partner -- An ISO
9001 Company*

Contact us for your high quality Software Outsourcing
<http://www.angleritech.com/offshore/outsourced_product_development.html>,
E-Business Products <http://www.angleritech.com/ebusiness/index.html>
and Design Solutions <http://www.angleritech.com/design/index.html>

/* */

web : www.angleritech.com <http://www.angleritech.com/>
tel : +91 422 2312707, 2313938
fax : +91 422 2313936
address :* *1144 Trichy Road, Coimbatore, 641045, India

offices <http://www.angleritech.com/contact/index.html>_ _ :
India | USA | UK | Canada | Europe | UAE | South Africa | Singapore |
Hong Kong

* *

*Disclaimer: *The information in the email, files and communication are
strictly confidential and meant for the intended recipients. It may
contain proprietary information. If you are not an intended recipient;
any form of disclosure, copyright, distribution and any other means of
use of information is unauthorised and subject to legal implications. We
do not accept any liability for the transmission of incomplete, delayed
communication and recipients must check this email and any attachments
for the presence of viruses before downloading them.

Re: solr-ruby - let's get a 1.0!

2009-04-09T11:30:17Z

So others know... RSolr had some of the object oriented document code
underneath the covers. But just a few days ago, Mat generously created a
patch to expose the option of adding document objects instead of hashes
only. Thanks Mat! Have you had a chance to check out the latest version?

I think that RSolr now looks a lot like what we talked about here:
http://markmail.org/message/ujesla5hdgch7xrd

I hope that people will continue to give feedback. Any other ruby+solr users
out there?

Matt

On Mon, Apr 6, 2009 at 6:52 PM, outoftime <mat@...> wrote:

>
> Hi all - just wanted to give a big +1 to Matt on this one. I've been
> working
> for a while on a higher-level Solr library in Ruby called Sunspot that
> allows easy indexing/searching of Ruby objects (using a prepackaged
> schema.xml, etc), abstracting all Solr/Lucene syntax away from the API
> consumer.
>
> The current (incomplete) version is built on solr-ruby, but a while ago I
> came across the LocalSolr plugin and thought it would be lovely to include
> LocalSolr support in Sunspot. I began looking for an alternative to
> solr-ruby that would allow me to cleanly use parameters and query types
> that
> aren't part of Solr core, and ran across RSolr. I think passing params
> through transparently is absolutely the right approach here - thus
> developers of higher-level APIs, or application developers who want to
> interact directly with Solr at a low level, are free to add integration
> with
> whatever Solr extensions they might need, without any explicit support from
> the Solr adapter layer.
>
> Having the option of using curb with no extra effort is also a huge bonus.
>
> One thing I'd like to see in RSolr/solr-ruby 1.0 is an object-oriented
> approach to building documents - in particular, it would be nice to have
> fine-grained control over field boost and that sort of thing, and having an
> explicit Document object would be an intuitive way to do that. Here's a
> simple example from an email discussion (that's been on hold for a few
> weeks
> - sorry Matt!) that Matt and I have been having:
>
> doc = RSolr::Document.new
> doc.id = 12
> doc.add_field('title_s', 'My Title', :boost => 2.0)
> solr.add(doc)
>
> It would make sense to me to have this be RSolr/solr-ruby's internal
> representation of documents, with appropriate conversion from the simpler
> hash format when that is sufficient for API users' expression of documents.
> If time allows, I'm hoping to contribute a patch to RSolr to this effect in
> the near future, but of course it would be interesting to hear everyone's
> thoughts.
>
> If anyone's curious, Sunspot is at http://github.com/outoftime/sunspot -
> comments, criticism, etc. more than welcome.
>
> Thanks!
> Mat
>
>
> goodieboy wrote:
> >
> > Solr-Ruby folks,
> >
> > A while ago, I started some discussions about updating solr-ruby. Some of
> > the things I brought up were related to the way that solr-ruby handles
> > param
> > mapping and response/request handling. Other folks have expressed the
> need
> > for jRuby support.
> >
> > In an effort to get some development happening on this front, Erik
> Hatcher
> > created a refectoring branch within the solr-ruby repo. I ended up doing
> > my
> > own development in a github repository because I had all kinds of nutty
> > ideas, and didn't want to bombard Erik with patches.
> >
> > I now have what I'm considering to be a very solid code base, complete
> > with
> > jRuby support. This library does no param mapping, and no
> request/response
> > class mapping; what you put in is what Solr gets, and what Solr returns
> is
> > what you get. In addition, it features a http adapter layer, so you can
> > switch between different http client implementations. The built in
> > adapters
> > are Net::HTTP and Curb. Etc..
> >
> > I'm asking the solr-ruby community (YOU!) to please have a look at this
> > library and offer your thoughts/opinions on it's capabilities and design.
> > Do
> > you seen this as a possible refectored base for solr-ruby? Is there
> > anything
> > there that you think could benefit solr-ruby? What are your thoughts on
> > getting solr-ruby to a 1.0 release?
> >
> > Here's the library I've been working on:
> > http://github.com/mwmitchell/rsolr/tree/master
> >
> > Thanks!
> >
> > Matt
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/solr-ruby---let%27s-get-a-1.0%21-tp22501321p22919097.html
> Sent from the Lucene - Ruby Developer mailing list archive at Nabble.com.
>
>

Re: solr-ruby - let's get a 1.0!

2009-04-06T15:52:36Z

Hi all - just wanted to give a big +1 to Matt on this one. I've been working for a while on a higher-level Solr library in Ruby called Sunspot that allows easy indexing/searching of Ruby objects (using a prepackaged schema.xml, etc), abstracting all Solr/Lucene syntax away from the API consumer.

The current (incomplete) version is built on solr-ruby, but a while ago I came across the LocalSolr plugin and thought it would be lovely to include LocalSolr support in Sunspot. I began looking for an alternative to solr-ruby that would allow me to cleanly use parameters and query types that aren't part of Solr core, and ran across RSolr. I think passing params through transparently is absolutely the right approach here - thus developers of higher-level APIs, or application developers who want to interact directly with Solr at a low level, are free to add integration with whatever Solr extensions they might need, without any explicit support from the Solr adapter layer.

Having the option of using curb with no extra effort is also a huge bonus.

One thing I'd like to see in RSolr/solr-ruby 1.0 is an object-oriented approach to building documents - in particular, it would be nice to have fine-grained control over field boost and that sort of thing, and having an explicit Document object would be an intuitive way to do that. Here's a simple example from an email discussion (that's been on hold for a few weeks - sorry Matt!) that Matt and I have been having:

doc = RSolr::Document.new
doc.id = 12
doc.add_field('title_s', 'My Title', :boost => 2.0)
solr.add(doc)

It would make sense to me to have this be RSolr/solr-ruby's internal representation of documents, with appropriate conversion from the simpler hash format when that is sufficient for API users' expression of documents. If time allows, I'm hoping to contribute a patch to RSolr to this effect in the near future, but of course it would be interesting to hear everyone's thoughts.

If anyone's curious, Sunspot is at http://github.com/outoftime/sunspot - comments, criticism, etc. more than welcome.

Thanks!
Mat

goodieboy wrote:

Solr-Ruby folks,

A while ago, I started some discussions about updating solr-ruby. Some of
the things I brought up were related to the way that solr-ruby handles param
mapping and response/request handling. Other folks have expressed the need
for jRuby support.

In an effort to get some development happening on this front, Erik Hatcher
created a refectoring branch within the solr-ruby repo. I ended up doing my
own development in a github repository because I had all kinds of nutty
ideas, and didn't want to bombard Erik with patches.

I now have what I'm considering to be a very solid code base, complete with
jRuby support. This library does no param mapping, and no request/response
class mapping; what you put in is what Solr gets, and what Solr returns is
what you get. In addition, it features a http adapter layer, so you can
switch between different http client implementations. The built in adapters
are Net::HTTP and Curb. Etc..

I'm asking the solr-ruby community (YOU!) to please have a look at this
library and offer your thoughts/opinions on it's capabilities and design. Do
you seen this as a possible refectored base for solr-ruby? Is there anything
there that you think could benefit solr-ruby? What are your thoughts on
getting solr-ruby to a 1.0 release?

Here's the library I've been working on:
http://github.com/mwmitchell/rsolr/tree/master

Thanks!

Matt

solr-ruby - let's get a 1.0!

2009-03-13T10:24:05Z

Solr-Ruby folks,

A while ago, I started some discussions about updating solr-ruby. Some of
the things I brought up were related to the way that solr-ruby handles param
mapping and response/request handling. Other folks have expressed the need
for jRuby support.

In an effort to get some development happening on this front, Erik Hatcher
created a refectoring branch within the solr-ruby repo. I ended up doing my
own development in a github repository because I had all kinds of nutty
ideas, and didn't want to bombard Erik with patches.

I now have what I'm considering to be a very solid code base, complete with
jRuby support. This library does no param mapping, and no request/response
class mapping; what you put in is what Solr gets, and what Solr returns is
what you get. In addition, it features a http adapter layer, so you can
switch between different http client implementations. The built in adapters
are Net::HTTP and Curb. Etc..

I'm asking the solr-ruby community (YOU!) to please have a look at this
library and offer your thoughts/opinions on it's capabilities and design. Do
you seen this as a possible refectored base for solr-ruby? Is there anything
there that you think could benefit solr-ruby? What are your thoughts on
getting solr-ruby to a 1.0 release?

Here's the library I've been working on:
http://github.com/mwmitchell/rsolr/tree/master

Thanks!

Matt

Re: sorl-ruby facet.method

2009-03-05T13:33:56Z

I agree that remapping params is not that much fun. I certainly vote for
just passing them through and it will be easier to keep up with the latest
as well.

I created:

https://issues.apache.org/jira/browse/SOLR-1047

Let me know if there is something else I can do to help.

On Thu, Mar 5, 2009 at 3:19 PM, Erik Hatcher <erik@...>wrote:

> First, note we have a ruby-dev@... list which focuses
> primarily on the solr-ruby library, flare, and other Ruby specific things.
> But this forum is as good as any, though I'm CC'ing ruby-dev too.
>
> On Mar 5, 2009, at 12:59 PM, Ian Connor wrote:
>
>> Is there a way to specify the facet.method using solr-ruby. I tried to add
>> it like this:
>>
>> hash["facet.method"] = @params[:facets][:method] if
>> @params[:facets][:method]
>>
>
> That's a reasonable addition, however we're about to do a refactoring of
> solr-ruby to bring in the great contributions Matt Mitchell has been doing
> with his RSolr project. We're going to strip away all the parameter
> mangling/mapping and just simply pass through parameters to Solr (and leave
> clever mapping of things like :query -> &q= to folks that want to add that
> construct to their own applications).
>
> to line 78 of standard.rb and it works when you add it to the facets Hash.
>> However, if there is another place that I could set that would be great.
>>
>
> One option is to provide your own request/response classes, subclassing
> Solr::Request(and Response)::Standard if you want to just hack this for now.
>
> I am also happy to submit a patch on a ticket if that works.
>>
>
> The above being said, I'd gladly commit your patch right away. Submit it
> via JIRA and consider it done. We'll do one final release (0.9?) of the
> current solr-ruby library before we gut it and simplify it (and do our best
> to provide backwards compatibility to the 0.x versions) for a 1.0 version.
>
> Erik
>
>

--
Regards,

Ian Connor
1 Leighton St #723
Cambridge, MA 02141
Call Center Phone: +1 (714) 239 3875 (24 hrs)
Fax: +1(770) 818 5697
Skype: ian.connor

Re: sorl-ruby facet.method

2009-03-05T12:19:08Z

First, note we have a ruby-dev@... list which focuses
primarily on the solr-ruby library, flare, and other Ruby specific
things. But this forum is as good as any, though I'm CC'ing ruby-dev
too.

On Mar 5, 2009, at 12:59 PM, Ian Connor wrote:
> Is there a way to specify the facet.method using solr-ruby. I tried
> to add
> it like this:
>
> hash["facet.method"] = @params[:facets][:method] if
> @params[:facets][:method]

That's a reasonable addition, however we're about to do a refactoring
of solr-ruby to bring in the great contributions Matt Mitchell has
been doing with his RSolr project. We're going to strip away all the
parameter mangling/mapping and just simply pass through parameters to
Solr (and leave clever mapping of things like :query -> &q= to folks
that want to add that construct to their own applications).

> to line 78 of standard.rb and it works when you add it to the facets
> Hash.
> However, if there is another place that I could set that would be
> great.

One option is to provide your own request/response classes,
subclassing Solr::Request(and Response)::Standard if you want to just
hack this for now.

> I am also happy to submit a patch on a ticket if that works.

The above being said, I'd gladly commit your patch right away. Submit
it via JIRA and consider it done. We'll do one final release (0.9?)
of the current solr-ruby library before we gut it and simplify it (and
do our best to provide backwards compatibility to the 0.x versions)
for a 1.0 version.

Erik

Re: [acts_as_solr] Re: acts_as_solr returns all db rows when using model name as query

2009-01-17T10:58:43Z

Cross posting a little...

On Jan 17, 2009, at 7:24 AM, Matthew Rudy Jacobs wrote:
> Nice Erik.
>
> http://wiki.apache.org/solr/CommonQueryParameters#fq
>
> I need to get to understand solr as a service, rather than just
> understanding ActsAsSolr.
> Feels like most of our queries are done wrong, as we don't use this
> "filter query" param.

Yeah, for sure. acts_as_solr really could shine with more pluggable
param handling/mapping to Solr. There's a lot of Solr trickery that
keeps on getting added. Faceting, for example, can really do some
wacky cool stuff with the local params:
<http://wiki.apache.org/solr/SimpleFacetParameters#head-f277d409b221b407d9c5430f552bf40ee6185c4c
>

It'd be sweet to have some Rubyish way to map those things. Matt,
CC'd, has some really great RSolr stuff rolling on github. I've been
so swamped to give the attention to these things, sadly.

Regarding consolidating the codebase: I'd really like to see folks
maintaining the solr-ruby library, pulling in RSolr and getting rid of
the ugly stuff, make the good stuff even better, and keep a nice test
suite. Not being a relational database kinda guy, I've not really
given aas much attention, but it deserves it. I invite the aas, solr-
ruby, and RSolr communities (and Flare too, if there is anyone using
it besides me in demos ;) to consider proposing and submitting code to
the /clients/ruby/solr-ruby (or renamed to rsolr?) Subversion repo.
It's not as easy a process as just propping up a github area, but it
does offer support from the broader Solr community and allows this
Ruby library to just simply ship with each version of Solr as a built-
in gem, plugin, what-have-you that is 100% compatible, etc.
Interested parties in aiming towards this, cull together on ruby-
dev@lucene and get us a codebase/patches to bringing in AAS, and any
other best-of Solr/Ruby goodness.

Erik

Re: solr ruby code update

2008-12-29T11:22:53Z

There have been lots of updates on this. More tests and features! And yes,
:auto_commit is gone :)

github url:

http://github.com/mwmitchell/solr/tree/master

Matt

Re: solr ruby code update

2008-12-17T07:29:02Z

On Dec 17, 2008, at 8:39 AM, Matt Mitchell wrote:
> The result object now has methods that WillPaginate can hook into,
> namely:
> current_page, total_pages, next_page and previous_page. With those
> methods,
> the will_paginate view helper can create the pagination view widget.

Gotcha, so WillPaginate makes it nicer to generate paging controls,
but isn't itself cursoring through Solr. it provides parameters that
are used elsewhere in another action. Got it (I think).

I was thinking WillPaginate was maybe doing something like
Solr::Importer::SolrSource - but I was confused.

> <%= will_paginate(@docs) %>

Nah, I think @response is better now that I think about it. You'll
want the current start/rows back from Solr in the response. @docs
won't give you that, right?

Erik

Re: solr ruby code update

2008-12-17T05:39:48Z

Erik,

I'm glad you like it so far, definitely some things to work out still but
I'm feeling pretty happy with it.

The pagination is real, not just a huge doc set using array cursors. The
library is converting :page and :per_page to "rows" and "start. You use it
by first querying, using the :page and :per_page params like:

result = solr.query(:page=>1, :per_page=>10, :q=>'*:*')
# or "search"
result = solr.search('ipod', :page=>1, :per_page=>10)

The result object now has methods that WillPaginate can hook into, namely:
current_page, total_pages, next_page and previous_page. With those methods,
the will_paginate view helper can create the pagination view widget. Of
course, the :page param would probably be dynamic; in Rails/Merb --
params[:page].

But you bring up a good point about where the pagination should be
"happening". Maybe instead of paginating the top-level "result", it'd be
more intuitive to do it on just a document set like:

result = solr.query(:page=>1, :per_page=>10, :q=>'*:*')
@docs = result.documents

<%= will_paginate(@docs) %>

What do you think?

Thanks for having a look and please do play with it when you get a chance!

Matt

On Wed, Dec 17, 2008 at 12:12 AM, Erik Hatcher
<erik@...>wrote:

> tell us more about the willpaginate stuff. your example:
>
> <%= will_paginate(@response) %>
>
> does this paginate a live cursor through solr by requesting start/row
> windows? or is it paginating through the docs returned to the client?
> shouldn't you paginate an @query or an @request than the response?
>
> Erik
>
>
> On Dec 16, 2008, at 11:52 PM, Matt Mitchell wrote:
>
> I've been re-writing/factoring the solr ruby gem -- and the source code
>> can
>> be found here:
>>
>> http://github.com/mwmitchell/solr/tree/master
>>
>> The current version is 0.5.1 and can be installed by running:
>>
>> gem sources -a http://gems.github.com
>> sudo gem install mwmitchell-solr
>>
>> More docs coming, but for the nitty gritty, have a look at some of the
>> tests
>> if needed.
>>
>> If you use solr-ruby at all, please have a look. I would love to hear from
>> some of the ruby folks out there!
>>
>> Cheers,
>>
>> Matt
>>
>
>

Re: solr ruby code update

2008-12-16T21:25:39Z

one concept i think we can remove from next gen solr-ruby's....
autocommit. the solr side can handle that configuration, and it's
often not really even a good idea to commit until your job is done, as
we now have rollback features and solr really should be able to handle
lots of uncommitted docs.

thoughts? maybe i'm wrong and having a client api sending commits
periodically in a big batch is warranted.

Erik

On Dec 16, 2008, at 11:52 PM, Matt Mitchell wrote:

> I've been re-writing/factoring the solr ruby gem -- and the source
> code can
> be found here:
>
> http://github.com/mwmitchell/solr/tree/master
>
> The current version is 0.5.1 and can be installed by running:
>
> gem sources -a http://gems.github.com
> sudo gem install mwmitchell-solr
>
> More docs coming, but for the nitty gritty, have a look at some of
> the tests
> if needed.
>
> If you use solr-ruby at all, please have a look. I would love to
> hear from
> some of the ruby folks out there!
>
> Cheers,
>
> Matt

Re: solr ruby code update

2008-12-16T21:12:08Z

tell us more about the willpaginate stuff. your example:

<%= will_paginate(@response) %>

does this paginate a live cursor through solr by requesting start/row
windows? or is it paginating through the docs returned to the
client? shouldn't you paginate an @query or an @request than the
response?

Erik

On Dec 16, 2008, at 11:52 PM, Matt Mitchell wrote:

Re: solr ruby code update

2008-12-16T21:09:35Z

wow, at first glance this looks spectacular, Matt! one of these
dreamy days i'll carve out time to really play... maybe sooner if you
keep dazzling us like this.

On Dec 16, 2008, at 11:52 PM, Matt Mitchell wrote:

solr ruby code update

2008-12-16T20:52:59Z

I've been re-writing/factoring the solr ruby gem -- and the source code can
be found here:

http://github.com/mwmitchell/solr/tree/master

The current version is 0.5.1 and can be installed by running:

gem sources -a http://gems.github.com
sudo gem install mwmitchell-solr

More docs coming, but for the nitty gritty, have a look at some of the tests
if needed.

If you use solr-ruby at all, please have a look. I would love to hear from
some of the ruby folks out there!

Cheers,

Matt

Re: quick jruby + solr benchmarks

2008-12-03T08:41:52Z

The other night I spent a few hours messing with EventMachine, Curb
(libcurl ruby lib) and RFuzz. EventMachine's HTTP2 is just missing
some of the POST features I need, and I didn't want to figure out how
to build what I needed from EventMachine's low-level features. RFuzz
works, but then would crap out completely or go from well under a
second to 20+ seconds to complete a request. I suspect it's not
designed for the large POSTs I need. Curb (which is used with "require
'curl'"--why do some gem authors not name the gem and the library the
same dang thing???) works great. It's not any faster than net/http,
but judging from those tests, I should be saving a lot of CPU.

Jamie

On Dec 3, 2008, at 10:05 AM, Matt Mitchell wrote:

> Thanks Jamie. That's kind of shocking actually. What client library
> do you
> use?
>
> On Sun, Nov 30, 2008 at 1:38 PM, Jamie Orchard-Hays <jamie@...
> >wrote:
>
>> Here's something to note when using net/http in Ruby (which open-uri
>> wraps). Even though it's about as fast as other options, it uses a
>> huge cpu
>> load when compared to others (on ruby 1.8.6):
>>
>> http://apocryph.org/
>> more_indepth_analysis_ruby_http_client_performance
>>
>>
>>
>> On Nov 26, 2008, at 12:06 PM, Matt Mitchell wrote:
>>
>> Interesting. My main goal was to get a feel for how jruby and the
>>> direct/embedded stuff compared to mri ruby and straight up http. But
>>> obviously, the data and these tests are not realistic at all.
>>> Thanks for
>>> your feedback guys.
>>>
>>> Matt
>>>
>>> On Wed, Nov 26, 2008 at 10:34 AM, Erik Hatcher
>>> <erik@...>wrote:
>>>
>>> I just had a brief conversation with Yonik on this to get his way
>>> more
>>>> expert opinion, and it really boils down to this in this particular
>>>> test...
>>>> the query itself is incredibly fast (1 millisecond or less QTime
>>>> Solr
>>>> reports) since there are no documents. So what these differences
>>>> are
>>>> showing is merely the difference between HTTP and a method call -
>>>> with
>>>> nothing else (of note) going on.
>>>>
>>>> In a realer world scenario, the HTTP overhead makes less
>>>> difference as
>>>> the
>>>> work being done in the query/faceting overshadows the communication
>>>> overhead.
>>>>
>>>> There's lies, damned lies, and benchmarks :)
>>>>
>>>> Erik
>>>>
>>>>
>>>>
>>>> On Nov 26, 2008, at 9:54 AM, Matt Mitchell wrote:
>>>>
>>>> Yeah I overlooked all of that. Thanks Erik. So could a better
>>>> query test
>>>>
>>>>> be
>>>>> an incremental one based on id like:
>>>>>
>>>>> 100.times do |id|
>>>>> q = "id:#{id}"
>>>>> # query request here...
>>>>> end
>>>>>
>>>>> ?
>>>>>
>>>>> Would you happen to know why the solr home and data dir never
>>>>> really
>>>>> change?
>>>>> Anytime I use commons http or embedded, a "solr" directory is
>>>>> created in
>>>>> the
>>>>> same directory as my script. Even though I'm setting the home
>>>>> and data
>>>>> dir
>>>>> in my code?
>>>>>
>>>>> Matt
>>>>>
>>>>> On Wed, Nov 26, 2008 at 3:28 AM, Erik Hatcher <
>>>>> erik@...
>>>>>
>>>>>> wrote:
>>>>>>
>>>>>
>>>>> just a couple of quick code comments...
>>>>>
>>>>>>
>>>>>> On Nov 25, 2008, at 6:04 PM, Matt Mitchell wrote:
>>>>>>
>>>>>> # EmbeddedSolrServer
>>>>>>
>>>>>>> def embedded(solr_home)
>>>>>>> @embedded ||= (
>>>>>>> import org.apache.solr.client.solrj.embedded.EmbeddedSolrServer
>>>>>>> import org.apache.solr.core.CoreContainer
>>>>>>> import org.apache.solr.core.CoreDescriptor
>>>>>>> import org.apache.solr.client.solrj.SolrQuery
>>>>>>> core_name = 'main-core'
>>>>>>> container = CoreContainer.new
>>>>>>> descriptor = CoreDescriptor.new(container, core_name, solr_home)
>>>>>>> core = container.create(descriptor)
>>>>>>>
>>>>>>>
>>>>>>> You'll want to close that core, otherwise the JVM doesn't
>>>>>>> exit. I
>>>>>> changed
>>>>>> this to:
>>>>>>
>>>>>> @core = ....
>>>>>>
>>>>>> container.register(core_name, core, false)
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> and used @core there.
>>>>>>
>>>>>> query = {'qt' => 'standard', 'q'=>'ipod', 'facet.field' => 'cat'}
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Note that faceting is not enabled unless there is also a
>>>>>>> &facet=on
>>>>>>
>>>>>> params = hash_to_params(query)
>>>>>>
>>>>>>
>>>>>>> max = 1000
>>>>>>>
>>>>>>> Benchmark.bm do |x|
>>>>>>> x.report 'http commons' do
>>>>>>> max.times do
>>>>>>> http_commons.query(params)
>>>>>>> end
>>>>>>> end
>>>>>>> x.report 'embedded' do
>>>>>>> max.times do
>>>>>>> embedded(solr_home).query(params)
>>>>>>> end
>>>>>>> end
>>>>>>> end
>>>>>>>
>>>>>>>
>>>>>>> And I added an:
>>>>>>
>>>>>> @core.close
>>>>>>
>>>>>> at the end.
>>>>>>
>>>>>> Erik
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>
>>

Re: quick jruby + solr benchmarks

2008-12-03T07:05:18Z

Thanks Jamie. That's kind of shocking actually. What client library do you
use?

On Sun, Nov 30, 2008 at 1:38 PM, Jamie Orchard-Hays <jamie@...>wrote:

> Here's something to note when using net/http in Ruby (which open-uri
> wraps). Even though it's about as fast as other options, it uses a huge cpu
> load when compared to others (on ruby 1.8.6):
>
> http://apocryph.org/more_indepth_analysis_ruby_http_client_performance
>
>
>
> On Nov 26, 2008, at 12:06 PM, Matt Mitchell wrote:
>
> Interesting. My main goal was to get a feel for how jruby and the
>> direct/embedded stuff compared to mri ruby and straight up http. But
>> obviously, the data and these tests are not realistic at all. Thanks for
>> your feedback guys.
>>
>> Matt
>>
>> On Wed, Nov 26, 2008 at 10:34 AM, Erik Hatcher
>> <erik@...>wrote:
>>
>> I just had a brief conversation with Yonik on this to get his way more
>>> expert opinion, and it really boils down to this in this particular
>>> test...
>>> the query itself is incredibly fast (1 millisecond or less QTime Solr
>>> reports) since there are no documents. So what these differences are
>>> showing is merely the difference between HTTP and a method call - with
>>> nothing else (of note) going on.
>>>
>>> In a realer world scenario, the HTTP overhead makes less difference as
>>> the
>>> work being done in the query/faceting overshadows the communication
>>> overhead.
>>>
>>> There's lies, damned lies, and benchmarks :)
>>>
>>> Erik
>>>
>>>
>>>
>>> On Nov 26, 2008, at 9:54 AM, Matt Mitchell wrote:
>>>
>>> Yeah I overlooked all of that. Thanks Erik. So could a better query test
>>>
>>>> be
>>>> an incremental one based on id like:
>>>>
>>>> 100.times do |id|
>>>> q = "id:#{id}"
>>>> # query request here...
>>>> end
>>>>
>>>> ?
>>>>
>>>> Would you happen to know why the solr home and data dir never really
>>>> change?
>>>> Anytime I use commons http or embedded, a "solr" directory is created in
>>>> the
>>>> same directory as my script. Even though I'm setting the home and data
>>>> dir
>>>> in my code?
>>>>
>>>> Matt
>>>>
>>>> On Wed, Nov 26, 2008 at 3:28 AM, Erik Hatcher <
>>>> erik@...
>>>>
>>>>> wrote:
>>>>>
>>>>
>>>> just a couple of quick code comments...
>>>>
>>>>>
>>>>> On Nov 25, 2008, at 6:04 PM, Matt Mitchell wrote:
>>>>>
>>>>> # EmbeddedSolrServer
>>>>>
>>>>>> def embedded(solr_home)
>>>>>> @embedded ||= (
>>>>>> import org.apache.solr.client.solrj.embedded.EmbeddedSolrServer
>>>>>> import org.apache.solr.core.CoreContainer
>>>>>> import org.apache.solr.core.CoreDescriptor
>>>>>> import org.apache.solr.client.solrj.SolrQuery
>>>>>> core_name = 'main-core'
>>>>>> container = CoreContainer.new
>>>>>> descriptor = CoreDescriptor.new(container, core_name, solr_home)
>>>>>> core = container.create(descriptor)
>>>>>>
>>>>>>
>>>>>> You'll want to close that core, otherwise the JVM doesn't exit. I
>>>>> changed
>>>>> this to:
>>>>>
>>>>> @core = ....
>>>>>
>>>>> container.register(core_name, core, false)
>>>>>
>>>>>
>>>>>>
>>>>>> and used @core there.
>>>>>
>>>>> query = {'qt' => 'standard', 'q'=>'ipod', 'facet.field' => 'cat'}
>>>>>
>>>>>
>>>>>>
>>>>>> Note that faceting is not enabled unless there is also a &facet=on
>>>>>
>>>>> params = hash_to_params(query)
>>>>>
>>>>>
>>>>>> max = 1000
>>>>>>
>>>>>> Benchmark.bm do |x|
>>>>>> x.report 'http commons' do
>>>>>> max.times do
>>>>>> http_commons.query(params)
>>>>>> end
>>>>>> end
>>>>>> x.report 'embedded' do
>>>>>> max.times do
>>>>>> embedded(solr_home).query(params)
>>>>>> end
>>>>>> end
>>>>>> end
>>>>>>
>>>>>>
>>>>>> And I added an:
>>>>>
>>>>> @core.close
>>>>>
>>>>> at the end.
>>>>>
>>>>> Erik
>>>>>
>>>>>
>>>>>
>>>>>
>>>
>

How to add a Icon using RubyQt

2008-12-01T20:53:07Z

Hi to all,

I am developing an application using RubyQt. In that i need to keep image icons. I want to know how to do, can any one help me regarding this. I am waiting for your replys.

Regards,

Narasimha Raju. Naidu

Re: quick jruby + solr benchmarks

2008-11-30T10:38:14Z

Here's something to note when using net/http in Ruby (which open-uri
wraps). Even though it's about as fast as other options, it uses a
huge cpu load when compared to others (on ruby 1.8.6):

http://apocryph.org/more_indepth_analysis_ruby_http_client_performance

On Nov 26, 2008, at 12:06 PM, Matt Mitchell wrote:

> Interesting. My main goal was to get a feel for how jruby and the
> direct/embedded stuff compared to mri ruby and straight up http. But
> obviously, the data and these tests are not realistic at all. Thanks
> for
> your feedback guys.
>
> Matt
>
> On Wed, Nov 26, 2008 at 10:34 AM, Erik Hatcher
> <erik@...>wrote:
>
>> I just had a brief conversation with Yonik on this to get his way
>> more
>> expert opinion, and it really boils down to this in this particular
>> test...
>> the query itself is incredibly fast (1 millisecond or less QTime Solr
>> reports) since there are no documents. So what these differences are
>> showing is merely the difference between HTTP and a method call -
>> with
>> nothing else (of note) going on.
>>
>> In a realer world scenario, the HTTP overhead makes less difference
>> as the
>> work being done in the query/faceting overshadows the communication
>> overhead.
>>
>> There's lies, damned lies, and benchmarks :)
>>
>> Erik
>>
>>
>>
>> On Nov 26, 2008, at 9:54 AM, Matt Mitchell wrote:
>>
>> Yeah I overlooked all of that. Thanks Erik. So could a better query
>> test
>>> be
>>> an incremental one based on id like:
>>>
>>> 100.times do |id|
>>> q = "id:#{id}"
>>> # query request here...
>>> end
>>>
>>> ?
>>>
>>> Would you happen to know why the solr home and data dir never really
>>> change?
>>> Anytime I use commons http or embedded, a "solr" directory is
>>> created in
>>> the
>>> same directory as my script. Even though I'm setting the home and
>>> data dir
>>> in my code?
>>>
>>> Matt
>>>
>>> On Wed, Nov 26, 2008 at 3:28 AM, Erik Hatcher <erik@...
>>>> wrote:
>>>
>>> just a couple of quick code comments...
>>>>
>>>> On Nov 25, 2008, at 6:04 PM, Matt Mitchell wrote:
>>>>
>>>> # EmbeddedSolrServer
>>>>> def embedded(solr_home)
>>>>> @embedded ||= (
>>>>> import org.apache.solr.client.solrj.embedded.EmbeddedSolrServer
>>>>> import org.apache.solr.core.CoreContainer
>>>>> import org.apache.solr.core.CoreDescriptor
>>>>> import org.apache.solr.client.solrj.SolrQuery
>>>>> core_name = 'main-core'
>>>>> container = CoreContainer.new
>>>>> descriptor = CoreDescriptor.new(container, core_name, solr_home)
>>>>> core = container.create(descriptor)
>>>>>
>>>>>
>>>> You'll want to close that core, otherwise the JVM doesn't exit. I
>>>> changed
>>>> this to:
>>>>
>>>> @core = ....
>>>>
>>>> container.register(core_name, core, false)
>>>>
>>>>>
>>>>>
>>>> and used @core there.
>>>>
>>>> query = {'qt' => 'standard', 'q'=>'ipod', 'facet.field' => 'cat'}
>>>>
>>>>>
>>>>>
>>>> Note that faceting is not enabled unless there is also a &facet=on
>>>>
>>>> params = hash_to_params(query)
>>>>
>>>>>
>>>>> max = 1000
>>>>>
>>>>> Benchmark.bm do |x|
>>>>> x.report 'http commons' do
>>>>> max.times do
>>>>> http_commons.query(params)
>>>>> end
>>>>> end
>>>>> x.report 'embedded' do
>>>>> max.times do
>>>>> embedded(solr_home).query(params)
>>>>> end
>>>>> end
>>>>> end
>>>>>
>>>>>
>>>> And I added an:
>>>>
>>>> @core.close
>>>>
>>>> at the end.
>>>>
>>>> Erik
>>>>
>>>>
>>>>
>>

Re: quick jruby + solr benchmarks

2008-11-26T09:06:57Z

Interesting. My main goal was to get a feel for how jruby and the
direct/embedded stuff compared to mri ruby and straight up http. But
obviously, the data and these tests are not realistic at all. Thanks for
your feedback guys.

Matt

On Wed, Nov 26, 2008 at 10:34 AM, Erik Hatcher
<erik@...>wrote:

> I just had a brief conversation with Yonik on this to get his way more
> expert opinion, and it really boils down to this in this particular test...
> the query itself is incredibly fast (1 millisecond or less QTime Solr
> reports) since there are no documents. So what these differences are
> showing is merely the difference between HTTP and a method call - with
> nothing else (of note) going on.
>
> In a realer world scenario, the HTTP overhead makes less difference as the
> work being done in the query/faceting overshadows the communication
> overhead.
>
> There's lies, damned lies, and benchmarks :)
>
> Erik
>
>
>
> On Nov 26, 2008, at 9:54 AM, Matt Mitchell wrote:
>
> Yeah I overlooked all of that. Thanks Erik. So could a better query test
>> be
>> an incremental one based on id like:
>>
>> 100.times do |id|
>> q = "id:#{id}"
>> # query request here...
>> end
>>
>> ?
>>
>> Would you happen to know why the solr home and data dir never really
>> change?
>> Anytime I use commons http or embedded, a "solr" directory is created in
>> the
>> same directory as my script. Even though I'm setting the home and data dir
>> in my code?
>>
>> Matt
>>
>> On Wed, Nov 26, 2008 at 3:28 AM, Erik Hatcher <erik@...
>> >wrote:
>>
>> just a couple of quick code comments...
>>>
>>> On Nov 25, 2008, at 6:04 PM, Matt Mitchell wrote:
>>>
>>> # EmbeddedSolrServer
>>>> def embedded(solr_home)
>>>> @embedded ||= (
>>>> import org.apache.solr.client.solrj.embedded.EmbeddedSolrServer
>>>> import org.apache.solr.core.CoreContainer
>>>> import org.apache.solr.core.CoreDescriptor
>>>> import org.apache.solr.client.solrj.SolrQuery
>>>> core_name = 'main-core'
>>>> container = CoreContainer.new
>>>> descriptor = CoreDescriptor.new(container, core_name, solr_home)
>>>> core = container.create(descriptor)
>>>>
>>>>
>>> You'll want to close that core, otherwise the JVM doesn't exit. I
>>> changed
>>> this to:
>>>
>>> @core = ....
>>>
>>> container.register(core_name, core, false)
>>>
>>>>
>>>>
>>> and used @core there.
>>>
>>> query = {'qt' => 'standard', 'q'=>'ipod', 'facet.field' => 'cat'}
>>>
>>>>
>>>>
>>> Note that faceting is not enabled unless there is also a &facet=on
>>>
>>> params = hash_to_params(query)
>>>
>>>>
>>>> max = 1000
>>>>
>>>> Benchmark.bm do |x|
>>>> x.report 'http commons' do
>>>> max.times do
>>>> http_commons.query(params)
>>>> end
>>>> end
>>>> x.report 'embedded' do
>>>> max.times do
>>>> embedded(solr_home).query(params)
>>>> end
>>>> end
>>>> end
>>>>
>>>>
>>> And I added an:
>>>
>>> @core.close
>>>
>>> at the end.
>>>
>>> Erik
>>>
>>>
>>>
>

Re: quick jruby + solr benchmarks

2008-11-26T07:34:44Z

I just had a brief conversation with Yonik on this to get his way more
expert opinion, and it really boils down to this in this particular
test... the query itself is incredibly fast (1 millisecond or less
QTime Solr reports) since there are no documents. So what these
differences are showing is merely the difference between HTTP and a
method call - with nothing else (of note) going on.

In a realer world scenario, the HTTP overhead makes less difference as
the work being done in the query/faceting overshadows the
communication overhead.

There's lies, damned lies, and benchmarks :)

Erik

On Nov 26, 2008, at 9:54 AM, Matt Mitchell wrote:

> Yeah I overlooked all of that. Thanks Erik. So could a better query
> test be
> an incremental one based on id like:
>
> 100.times do |id|
> q = "id:#{id}"
> # query request here...
> end
>
> ?
>
> Would you happen to know why the solr home and data dir never really
> change?
> Anytime I use commons http or embedded, a "solr" directory is
> created in the
> same directory as my script. Even though I'm setting the home and
> data dir
> in my code?
>
> Matt
>
> On Wed, Nov 26, 2008 at 3:28 AM, Erik Hatcher <erik@...
> >wrote:
>
>> just a couple of quick code comments...
>>
>> On Nov 25, 2008, at 6:04 PM, Matt Mitchell wrote:
>>
>>> # EmbeddedSolrServer
>>> def embedded(solr_home)
>>> @embedded ||= (
>>> import org.apache.solr.client.solrj.embedded.EmbeddedSolrServer
>>> import org.apache.solr.core.CoreContainer
>>> import org.apache.solr.core.CoreDescriptor
>>> import org.apache.solr.client.solrj.SolrQuery
>>> core_name = 'main-core'
>>> container = CoreContainer.new
>>> descriptor = CoreDescriptor.new(container, core_name, solr_home)
>>> core = container.create(descriptor)
>>>
>>
>> You'll want to close that core, otherwise the JVM doesn't exit. I
>> changed
>> this to:
>>
>> @core = ....
>>
>> container.register(core_name, core, false)
>>>
>>
>> and used @core there.
>>
>> query = {'qt' => 'standard', 'q'=>'ipod', 'facet.field' => 'cat'}
>>>
>>
>> Note that faceting is not enabled unless there is also a &facet=on
>>
>> params = hash_to_params(query)
>>>
>>> max = 1000
>>>
>>> Benchmark.bm do |x|
>>> x.report 'http commons' do
>>> max.times do
>>> http_commons.query(params)
>>> end
>>> end
>>> x.report 'embedded' do
>>> max.times do
>>> embedded(solr_home).query(params)
>>> end
>>> end
>>> end
>>>
>>
>> And I added an:
>>
>> @core.close
>>
>> at the end.
>>
>> Erik
>>
>>

Re: quick jruby + solr benchmarks

2008-11-26T07:10:15Z

On Nov 26, 2008, at 9:54 AM, Matt Mitchell wrote:

> Yeah I overlooked all of that. Thanks Erik. So could a better query
> test be
> an incremental one based on id like:
>
> 100.times do |id|
> q = "id:#{id}"
> # query request here...
> end
>
> ?

Testing is an art form. Depends on what you are testing. Issuing
entirely unique queries is not very real-world either, but at least it
will cause the bypassing of query and HTTP caching shortcuts.

Many organizations mine their query logs to get a set of
representative queries to test with, for example.

I think your point is proven - EmbeddedSolrServer itself is faster
than CommonsHttpSolrServer. But would you deploy that way? Is your
front-end going to be merged with Solr itself? That may or may not be
very viable, depending on the resources the front-end and Solr needs
and how much system resources you have. What about doing load
balancing? You're then stuck with load balancing your front-end in
tandem with Solr itself.

Again, it all boils down to what you're after with the benchmarks.
And I'm not a benchmarking performance savvy person myself, so I'm not
sure where to take it from here. It's an interesting test, for sure,
and I'd like to have it reviewed by others that really know their
stuff in this realm and with Solr itself that can elaborate on why
there is such a huge difference in speed. Is it just HTTP and
serialize/unserialize overhead? (I tend to doubt that, but don't know)

> Would you happen to know why the solr home and data dir never really
> change?
> Anytime I use commons http or embedded, a "solr" directory is
> created in the
> same directory as my script. Even though I'm setting the home and
> data dir
> in my code?

I don't know at the moment, I'd have to dig deeper.

Erik

Re: quick jruby + solr benchmarks

2008-11-26T06:54:22Z

Yeah I overlooked all of that. Thanks Erik. So could a better query test be
an incremental one based on id like:

100.times do |id|
q = "id:#{id}"
# query request here...
end

?

Would you happen to know why the solr home and data dir never really change?
Anytime I use commons http or embedded, a "solr" directory is created in the
same directory as my script. Even though I'm setting the home and data dir
in my code?

Matt

On Wed, Nov 26, 2008 at 3:28 AM, Erik Hatcher <erik@...>wrote:

> just a couple of quick code comments...
>
> On Nov 25, 2008, at 6:04 PM, Matt Mitchell wrote:
>
>> # EmbeddedSolrServer
>> def embedded(solr_home)
>> @embedded ||= (
>> import org.apache.solr.client.solrj.embedded.EmbeddedSolrServer
>> import org.apache.solr.core.CoreContainer
>> import org.apache.solr.core.CoreDescriptor
>> import org.apache.solr.client.solrj.SolrQuery
>> core_name = 'main-core'
>> container = CoreContainer.new
>> descriptor = CoreDescriptor.new(container, core_name, solr_home)
>> core = container.create(descriptor)
>>
>
> You'll want to close that core, otherwise the JVM doesn't exit. I changed
> this to:
>
> @core = ....
>
> container.register(core_name, core, false)
>>
>
> and used @core there.
>
> query = {'qt' => 'standard', 'q'=>'ipod', 'facet.field' => 'cat'}
>>
>
> Note that faceting is not enabled unless there is also a &facet=on
>
> params = hash_to_params(query)
>>
>> max = 1000
>>
>> Benchmark.bm do |x|
>> x.report 'http commons' do
>> max.times do
>> http_commons.query(params)
>> end
>> end
>> x.report 'embedded' do
>> max.times do
>> embedded(solr_home).query(params)
>> end
>> end
>> end
>>
>
> And I added an:
>
> @core.close
>
> at the end.
>
> Erik
>
>

Re: quick jruby + solr benchmarks

2008-11-26T00:28:33Z

just a couple of quick code comments...

On Nov 25, 2008, at 6:04 PM, Matt Mitchell wrote:

> # EmbeddedSolrServer
> def embedded(solr_home)
> @embedded ||= (
> import org.apache.solr.client.solrj.embedded.EmbeddedSolrServer
> import org.apache.solr.core.CoreContainer
> import org.apache.solr.core.CoreDescriptor
> import org.apache.solr.client.solrj.SolrQuery
> core_name = 'main-core'
> container = CoreContainer.new
> descriptor = CoreDescriptor.new(container, core_name, solr_home)
> core = container.create(descriptor)

You'll want to close that core, otherwise the JVM doesn't exit. I
changed this to:

@core = ....

> container.register(core_name, core, false)

and used @core there.

> query = {'qt' => 'standard', 'q'=>'ipod', 'facet.field' => 'cat'}

Note that faceting is not enabled unless there is also a &facet=on

> params = hash_to_params(query)
>
> max = 1000
>
> Benchmark.bm do |x|
> x.report 'http commons' do
> max.times do
> http_commons.query(params)
> end
> end
> x.report 'embedded' do
> max.times do
> embedded(solr_home).query(params)
> end
> end
> end

And I added an:

@core.close

at the end.

Erik

Re: quick jruby + solr benchmarks

2008-11-25T23:56:37Z

On Nov 25, 2008, at 7:13 PM, Jamie Orchard-Hays wrote:
> So about 2x? Not bad. I wonder what running httperf against a simple
> app would show.

Keep in mind these points:

* Solr's query cache. Repeating a query 1000 times is really only
executing the query one time and pulling the document set the rest of
the time, except...

* Solr supports HTTP cache headers. Thus a "smart" HTTP client
that is HTTP cache savvy will get 304's for 999 of those queries
without Solr doing anything but checking the HTTP request headers and
the current state of the index. Note that Matt's benchmark code is
not HTTP cache savvy at the moment (not a flaw per se, just worth
noting).

Erik

Re: quick jruby + solr benchmarks

2008-11-25T21:10:54Z

Yeah that type of benchmark would probably be a lot more useful. I'll see if
I can get something like that going. I've never really done benchmarking
before. Any general tips?

matt

On Tue, Nov 25, 2008 at 7:13 PM, Jamie Orchard-Hays <jamie@...>wrote:

> So about 2x? Not bad. I wonder what running httperf against a simple app
> would show.
>
>
>
> On Nov 25, 2008, at 6:04 PM, Matt Mitchell wrote:
>
> I'm starting to experiment with benchmarks/jruby + solr and just wanted to
>> get this out there -- getting ready for a week vacation :)
>>
>> In my solr-ruby 'refactoring' progress, I'm finding some interesting
>> results
>> and will try to post in the next few weeks.
>>
>> This is jruby 1.1.4 and solr 1.3 (empty index) -- using the standard Ruby
>> "Benchmark" library.
>>
>> The script:
>>
>> #
>>
>> require 'java'
>> require 'benchmark'
>>
>> solr_dist_root = File.expand_path(File.join(File.dirname(__FILE__), '..',
>> 'apache-solr-1.3.0'))
>> solr_home = File.join(solr_dist_root, 'example', 'solr')
>>
>> def require_jars(dir)
>> jar_pattern = File.join(dir,"**", "*.jar")
>> jar_files = Dir.glob(jar_pattern)
>> jar_files.each {|jar_file| require jar_file}
>> end
>>
>> def hash_to_params(hash_params)
>> import org.apache.solr.common.params.ModifiableSolrParams
>> query = ModifiableSolrParams.new
>> query.instance_eval do
>> alias _add add
>> def add(field, values)
>> _add(field.to_s, (values.is_a?(Array) ? values :
>> [values]).to_java(:string))
>> end
>> end
>> hash_params.each_pair do |k,v|
>> query.add k, v
>> end
>> query
>> end
>>
>> require_jars(File.join(solr_dist_root, "lib"))
>> require_jars(File.join(solr_dist_root, "dist"))
>>
>> # HttpCommons
>> def http_commons
>> @http_commons ||= (
>> import org.apache.solr.client.solrj.impl.CommonsHttpSolrServer
>> import org.apache.solr.common.params.MapSolrParams
>> solr = CommonsHttpSolrServer.new("http://localhost:8983/solr")
>> )
>> end
>>
>> # EmbeddedSolrServer
>> def embedded(solr_home)
>> @embedded ||= (
>> import org.apache.solr.client.solrj.embedded.EmbeddedSolrServer
>> import org.apache.solr.core.CoreContainer
>> import org.apache.solr.core.CoreDescriptor
>> import org.apache.solr.client.solrj.SolrQuery
>> core_name = 'main-core'
>> container = CoreContainer.new
>> descriptor = CoreDescriptor.new(container, core_name, solr_home)
>> core = container.create(descriptor)
>> container.register(core_name, core, false)
>> solr = EmbeddedSolrServer.new(container, core_name)
>> )
>> end
>>
>> query = {'qt' => 'standard', 'q'=>'ipod', 'facet.field' => 'cat'}
>> params = hash_to_params(query)
>>
>> max = 1000
>>
>> Benchmark.bm do |x|
>> x.report 'http commons' do
>> max.times do
>> http_commons.query(params)
>> end
>> end
>> x.report 'embedded' do
>> max.times do
>> embedded(solr_home).query(params)
>> end
>> end
>> end
>>
>> # THE RESULTS
>>
>> # http commons
>> # 4.634000 0.000000 4.634000 ( 4.633849)
>> # 4.454000 0.000000 4.454000 ( 4.453764)
>> # 3.908000 0.000000 3.908000 ( 3.907367)
>>
>> # embedded
>> # 2.152000 0.000000 2.152000 ( 2.152226)
>> # 2.191000 0.000000 2.191000 ( 2.191359)
>> # 2.083000 0.000000 2.083000 ( 2.082696)
>>
>
>

Re: quick jruby + solr benchmarks

2008-11-25T21:08:13Z

Looks like jruby + DirectSolrConnection are on top. I'll try to get some
update queries next.

1,000 iterations VS 10,000 iterations
Added Ruby MRI 1.8.6, using open-uri / http
Added jruby using open-uri / http

"Benchmark'" standard lib
solr 1.3
empty index
query = ipod

# jruby + CommonsHttpSolrServer
# user system total real
# 1000 iterations
# 4.335000 0.000000 4.335000 ( 4.334744)
# 4.335000 0.000000 4.335000 ( 4.334730)
# 10000 iterations
# 32.355000 0.000000 32.355000 ( 32.354999)
# 32.303000 0.000000 32.303000 ( 32.302859)
# 32.323000 0.000000 32.323000 ( 32.323368)

# jruby + EmbeddedSolrServer
# user system total real
# 1000 iterations
# 2.268000 0.000000 2.268000 ( 2.267976)
# 2.357000 0.000000 2.357000 ( 2.356588)
# 10000 iterations
# 10.650000 0.000000 10.650000 ( 10.649839)
# 8.099000 0.000000 8.099000 ( 8.099088)
# 8.119000 0.000000 8.119000 ( 8.118807)

# jruby + DirectSolrConnection
# user system total real
# 1000 iterations
# 1.593000 0.000000 1.593000 ( 1.592349)
# 1.595000 0.000000 1.595000 ( 1.594842)
# 10000 iterations
# 10.708000 0.000000 10.708000 ( 10.707790)
# 6.952000 0.000000 6.952000 ( 6.951736)
# 7.939000 0.000000 7.939000 ( 7.939191)

# ruby mri + http / open-uri
# user system total real
# 1000 iterations
# 0.760000 0.310000 1.070000 ( 1.607703)
# 0.730000 0.300000 1.030000 ( 1.619739)
# 0.760000 0.330000 1.090000 ( 1.907517)
# 0.740000 0.300000 1.040000 ( 1.543832)
# 10000 iterations
# 7.300000 2.970000 10.270000 ( 15.452759)
# 7.290000 2.960000 10.250000 ( 15.585011)
# 7.330000 2.980000 10.310000 ( 15.781377)

# jruby + http / open-uri
# user system total real
# 10000 iterations
# 27.583000 0.000000 27.583000 ( 27.582765)
# 25.620000 0.000000 25.620000 ( 25.620403)
# 25.474000 0.000000 25.474000 ( 25.473653)

Re: quick jruby + solr benchmarks

2008-11-25T16:13:26Z

So about 2x? Not bad. I wonder what running httperf against a simple
app would show.

On Nov 25, 2008, at 6:04 PM, Matt Mitchell wrote:

> I'm starting to experiment with benchmarks/jruby + solr and just
> wanted to
> get this out there -- getting ready for a week vacation :)
>
> In my solr-ruby 'refactoring' progress, I'm finding some interesting
> results
> and will try to post in the next few weeks.
>
> This is jruby 1.1.4 and solr 1.3 (empty index) -- using the standard
> Ruby
> "Benchmark" library.
>
> The script:
>
> #
>
> require 'java'
> require 'benchmark'
>
> solr_dist_root = File.expand_path(File.join(File.dirname(__FILE__),
> '..',
> 'apache-solr-1.3.0'))
> solr_home = File.join(solr_dist_root, 'example', 'solr')
>
> def require_jars(dir)
> jar_pattern = File.join(dir,"**", "*.jar")
> jar_files = Dir.glob(jar_pattern)
> jar_files.each {|jar_file| require jar_file}
> end
>
> def hash_to_params(hash_params)
> import org.apache.solr.common.params.ModifiableSolrParams
> query = ModifiableSolrParams.new
> query.instance_eval do
> alias _add add
> def add(field, values)
> _add(field.to_s, (values.is_a?(Array) ? values :
> [values]).to_java(:string))
> end
> end
> hash_params.each_pair do |k,v|
> query.add k, v
> end
> query
> end
>
> require_jars(File.join(solr_dist_root, "lib"))
> require_jars(File.join(solr_dist_root, "dist"))
>
> # HttpCommons
> def http_commons
> @http_commons ||= (
> import org.apache.solr.client.solrj.impl.CommonsHttpSolrServer
> import org.apache.solr.common.params.MapSolrParams
> solr = CommonsHttpSolrServer.new("http://localhost:8983/solr")
> )
> end
>
> # EmbeddedSolrServer
> def embedded(solr_home)
> @embedded ||= (
> import org.apache.solr.client.solrj.embedded.EmbeddedSolrServer
> import org.apache.solr.core.CoreContainer
> import org.apache.solr.core.CoreDescriptor
> import org.apache.solr.client.solrj.SolrQuery
> core_name = 'main-core'
> container = CoreContainer.new
> descriptor = CoreDescriptor.new(container, core_name, solr_home)
> core = container.create(descriptor)
> container.register(core_name, core, false)
> solr = EmbeddedSolrServer.new(container, core_name)
> )
> end
>
> query = {'qt' => 'standard', 'q'=>'ipod', 'facet.field' => 'cat'}
> params = hash_to_params(query)
>
> max = 1000
>
> Benchmark.bm do |x|
> x.report 'http commons' do
> max.times do
> http_commons.query(params)
> end
> end
> x.report 'embedded' do
> max.times do
> embedded(solr_home).query(params)
> end
> end
> end
>
> # THE RESULTS
>
> # http commons
> # 4.634000 0.000000 4.634000 ( 4.633849)
> # 4.454000 0.000000 4.454000 ( 4.453764)
> # 3.908000 0.000000 3.908000 ( 3.907367)
>
> # embedded
> # 2.152000 0.000000 2.152000 ( 2.152226)
> # 2.191000 0.000000 2.191000 ( 2.191359)
> # 2.083000 0.000000 2.083000 ( 2.082696)

quick jruby + solr benchmarks

2008-11-25T15:04:38Z

I'm starting to experiment with benchmarks/jruby + solr and just wanted to
get this out there -- getting ready for a week vacation :)

In my solr-ruby 'refactoring' progress, I'm finding some interesting results
and will try to post in the next few weeks.

This is jruby 1.1.4 and solr 1.3 (empty index) -- using the standard Ruby
"Benchmark" library.

The script:

#

require 'java'
require 'benchmark'

solr_dist_root = File.expand_path(File.join(File.dirname(__FILE__), '..',
'apache-solr-1.3.0'))
solr_home = File.join(solr_dist_root, 'example', 'solr')

def require_jars(dir)
jar_pattern = File.join(dir,"**", "*.jar")
jar_files = Dir.glob(jar_pattern)
jar_files.each {|jar_file| require jar_file}
end

def hash_to_params(hash_params)
import org.apache.solr.common.params.ModifiableSolrParams
query = ModifiableSolrParams.new
query.instance_eval do
alias _add add
def add(field, values)
_add(field.to_s, (values.is_a?(Array) ? values :
[values]).to_java(:string))
end
end
hash_params.each_pair do |k,v|
query.add k, v
end
query
end

require_jars(File.join(solr_dist_root, "lib"))
require_jars(File.join(solr_dist_root, "dist"))

# HttpCommons
def http_commons
@http_commons ||= (
import org.apache.solr.client.solrj.impl.CommonsHttpSolrServer
import org.apache.solr.common.params.MapSolrParams
solr = CommonsHttpSolrServer.new("http://localhost:8983/solr")
)
end

# EmbeddedSolrServer
def embedded(solr_home)
@embedded ||= (
import org.apache.solr.client.solrj.embedded.EmbeddedSolrServer
import org.apache.solr.core.CoreContainer
import org.apache.solr.core.CoreDescriptor
import org.apache.solr.client.solrj.SolrQuery
core_name = 'main-core'
container = CoreContainer.new
descriptor = CoreDescriptor.new(container, core_name, solr_home)
core = container.create(descriptor)
container.register(core_name, core, false)
solr = EmbeddedSolrServer.new(container, core_name)
)
end

query = {'qt' => 'standard', 'q'=>'ipod', 'facet.field' => 'cat'}
params = hash_to_params(query)

max = 1000

Benchmark.bm do |x|
x.report 'http commons' do
max.times do
http_commons.query(params)
end
end
x.report 'embedded' do
max.times do
embedded(solr_home).query(params)
end
end
end

# THE RESULTS

# http commons
# 4.634000 0.000000 4.634000 ( 4.633849)
# 4.454000 0.000000 4.454000 ( 4.453764)
# 3.908000 0.000000 3.908000 ( 3.907367)

# embedded
# 2.152000 0.000000 2.152000 ( 2.152226)
# 2.191000 0.000000 2.191000 ( 2.191359)
# 2.083000 0.000000 2.083000 ( 2.082696)

ruby/json response for spellcheck component

2008-10-14T10:36:57Z

Hi,

I'm working on a patch to support the new SpellCheckComponent in solr-ruby.
I see that the response in some cases has duplicate keys. Here is an
example:

'suggestions'=>[

'buk',{'numFound'=>5,'startOffset'=>0,'endOffset'=>3,'origFreq'=>0,'suggestion'=>{'frequency'=>27,'word'=>'bul'},'suggestion'=>{'frequency'=>5,'word'=>'bus'},'suggestion'=>{'frequency'=>3,'word'=>'bok'},'suggestion'=>{'frequency'=>3,'word'=>'buy'},'suggestion'=>{'frequency'=>2,'word'=>'bulk'}},

'bot',{'numFound'=>5,'startOffset'=>4,'endOffset'=>7,'origFreq'=>1,'suggestion'=>{'frequency'=>57,'word'=>'box'},'suggestion'=>{'frequency'=>20,'word'=>'tot'},'suggestion'=>{'frequency'=>20,'word'=>'both'},'suggestion'=>{'frequency'=>18,'word'=>'boy'},'suggestion'=>{'frequency'=>7,'word'=>'mot'}},
'correctlySpelled',false,
'collation','bul box'
]

When this is eval'd in Ruby, only the last instance of a duplicate key
(suggestion) makes it (maybe I'm completely missing something though) - Was
this the desired format and if so, anyone have any ideas on how to handle
this sort of thing? I'd vote for changing 'suggestion'=>{} to
'suggestions'=>[{},{}] etc..

Also, a possible gotcha for this format is... what if one of your terms was
'suggestion' or 'collation'? The spellcheck response fields are mixed right
in with the terms. What about having a separate key called 'terms':

'suggestions'=>[
'collation', 'bul box',
'correctlySpelled',false,
'terms',[
'buk', {'numFound'=>5, 'suggestions'=>[{},{}]},
'bot', {'numFound'=>5m 'suggestions'=>[{},{}]}
]
]

etc..

Matt

DirectSolrConnection / JRuby

2008-10-08T18:08:12Z

DirectSolrConnection does work out pretty nicely:

require 'java'
require 'erb'

solr_dist_root = "/Users/erik/apache-solr-1.3.0"
solr_home = "/Users/erik/apache-solr-1.3.0/example/solr"

def require_jars(dir)
jar_pattern = File.join(dir,"**", "*.jar")
jar_files = Dir.glob(jar_pattern)

jar_files.each {|jar_file| require jar_file}
end

require_jars(File.join(solr_dist_root, "lib"))
require_jars(File.join(solr_dist_root, "dist"))

import org.apache.solr.client.solrj.impl.CommonsHttpSolrServer
import org.apache.solr.client.solrj.embedded.EmbeddedSolrServer
import org.apache.solr.core.CoreContainer
import org.apache.solr.core.CoreDescriptor
import org.apache.solr.common.params.ModifiableSolrParams
import org.apache.solr.common.params.MapSolrParams
import org.apache.solr.servlet.DirectSolrConnection

container = CoreContainer.new
descriptor = CoreDescriptor.new(container, "core1", solr_home)
core = container.create(descriptor)
container.register("core1", core, false)

class Request
def initialize(params)
@params = params
end

def handler
'select'
end

def content_type
'application/x-www-form-urlencoded; charset=utf-8'
end

def to_s
http_params = []
@params.each do |key,value|
if value.respond_to? :each
value.each { |v| http_params <<
"#{key}=#{ERB::Util::url_encode(v)}" unless v.nil?}
else
http_params << "#{key}=#{ERB::Util::url_encode(value)}"
unless value.nil?
end
end

http_params.join("&")
end
end

solr = DirectSolrConnection.new(solr_home,File.join(solr_home,"solr",
"data"),nil)
request = Request.new(:qt => 'standard', :q => '*:*', 'facet.field' =>
['cat'], :wt=>'ruby')
response = solr.request("/#{request.handler}?#{request.to_s}",nil)
puts response
solr.close
core.close

The response is the raw String response, suitable for eval'ing if
wt=ruby.

So, for embedded SolrJRuby work, this will be the most transparent way
to go I think.

The question I have is whether we should concern ourselves with
SolrJ's CommonsHttpSolrServer? I suspect if you're in a JRuby
environment that SolrJ's API will be faster performance-wise than
eval'ing Ruby responses. Testing warranted for sure.

Erik

Fwd: svn commit: r702993 - /lucene/solr/trunk/src/webapp/src/org/apache/solr/servlet/DirectSolrConnection.java

2008-10-08T14:37:51Z

Ryan's commit reminded me of DirectSolrConnection - this looks like an
even easier thing to plug into solr-(j)ruby than EmbeddedSolrServer.
I'll take a stab at that soon. We still should support SolrServer and
what it offers, but DirectSolrConnection will make things more easily
transparent at first.

Erik

Begin forwarded message:

> From: ryan@...
> Date: October 8, 2008 5:23:21 PM EDT
> To: solr-commits@...
> Subject: svn commit: r702993 - /lucene/solr/trunk/src/webapp/src/org/
> apache/solr/servlet/DirectSolrConnection.java
> Reply-To: solr-dev@...
>
> Author: ryan
> Date: Wed Oct 8 14:23:21 2008
> New Revision: 702993
>
> URL: http://svn.apache.org/viewvc?rev=702993&view=rev
> Log:
> SOLR-721 -- DirectSolrConnection now uses CoreContainers
>
> Modified:
> lucene/solr/trunk/src/webapp/src/org/apache/solr/servlet/
> DirectSolrConnection.java
>
> Modified: lucene/solr/trunk/src/webapp/src/org/apache/solr/servlet/
> DirectSolrConnection.java
> URL: http://svn.apache.org/viewvc/lucene/solr/trunk/src/webapp/src/org/apache/solr/servlet/DirectSolrConnection.java?rev=702993&r1=702992&r2=702993&view=diff
> =
> =
> =
> =
> =
> =
> =
> =
> ======================================================================
> --- lucene/solr/trunk/src/webapp/src/org/apache/solr/servlet/
> DirectSolrConnection.java (original)
> +++ lucene/solr/trunk/src/webapp/src/org/apache/solr/servlet/
> DirectSolrConnection.java Wed Oct 8 14:23:21 2008
> @@ -29,8 +29,11 @@
> import org.apache.solr.common.params.SolrParams;
> import org.apache.solr.common.util.ContentStream;
> import org.apache.solr.common.util.ContentStreamBase;
> +import org.apache.solr.core.CoreContainer;
> +import org.apache.solr.core.CoreDescriptor;
> import org.apache.solr.core.SolrConfig;
> import org.apache.solr.core.SolrCore;
> +import org.apache.solr.core.SolrResourceLoader;
> import org.apache.solr.request.QueryResponseWriter;
> import org.apache.solr.request.SolrQueryRequest;
> import org.apache.solr.request.SolrQueryResponse;
> @@ -101,16 +104,19 @@
> }
> }
>
> - // Initialize SolrConfig
> - SolrConfig config = null;
> + if( instanceDir == null ) {
> + instanceDir = SolrResourceLoader.locateInstanceDir();
> + }
> +
> + // Initialize
> try {
> - config = new SolrConfig(instanceDir,
> SolrConfig.DEFAULT_CONF_FILE, null);
> - instanceDir = config.getResourceLoader().getInstanceDir();
> -
> - // If the Data directory is specified, initialize SolrCore
> directly
> - IndexSchema schema = new IndexSchema(config, instanceDir+"/
> conf/schema.xml", null);
> - core = new SolrCore( null, dataDir, config, schema, null );
> - parser = new SolrRequestParsers( config );
> + CoreContainer cores = new CoreContainer(new
> SolrResourceLoader(instanceDir));
> + SolrConfig solrConfig = new SolrConfig(instanceDir,
> SolrConfig.DEFAULT_CONF_FILE, null);
> + CoreDescriptor dcore = new CoreDescriptor(cores, "",
> solrConfig.getResourceLoader().getInstanceDir());
> + IndexSchema indexSchema = new IndexSchema(solrConfig,
> instanceDir+"/conf/schema.xml", null);
> + core = new SolrCore( null, dataDir, solrConfig, indexSchema,
> dcore);
> + cores.register("", core, false);
> + parser = new SolrRequestParsers( solrConfig );
> }
> catch (Exception ee) {
> throw new RuntimeException(ee);
>

Re: jruby and solrj

2008-10-06T20:40:02Z

I've created a simple bare minimum server.rb that - similar to SolrJ's
SolrServer abstraction in (currently mostly empty) methods.

And in test/ I've checked in a JRuby example showing the difference
between going through a pure Ruby API and the SolrJ API. You can run
it this way from the root of the solr-ruby-refactoring branch:

jruby -Ilib test/temp.rb

I think the goal is to achieve something like this being equivalent
for both MRI and SolrJ-Ruby, regardless of the solr object type:

response = eval(solr.request(:qt => 'standard', :q => '*:*',
'facet.field' => ['cat'], :wt=>'ruby'))
puts response['response']['numFound']

Not quite sure where the eval belongs, so I left it out of
Solr::Server for now. We want to be able to support raw responses
through solr-ruby also, for use cases like letting Solr's JSON
response pass through a Ruby application server (or perhaps even to
use JSON within solr-ruby, if it parses faster).

Shouldn't be too hard to get there from here, eh Matt?

Erik

Re: jruby and solrj

2008-10-06T13:30:56Z

On Oct 6, 2008, at 4:19 PM, Matt Mitchell wrote:
> Awesome, thanks Erik! I really like the idea of bypassing SolrQuery,
> and
> just using a straight up Hash. It'll be even easier to keep the API
> consistent between jruby and standard ruby now.

Here's a cute trick:

query = {'qt' => 'standard', 'q'=>'*:*', 'facet.field' => 'cat'}
response = solr.query(MapSolrParams.new(query))

But that doesn't work, as-is, with an array for the facet.field
parameter, nor does it work for symbols, so this doesn't work for two
reasons:

solr.query(MapSolrParams.new(
:qt => 'standard', :q=>'*:*', 'facet.field' => ['cat']
)

Getting warmer! :)

Erik

Re: jruby and solrj

2008-10-06T13:19:10Z

Awesome, thanks Erik! I really like the idea of bypassing SolrQuery, and
just using a straight up Hash. It'll be even easier to keep the API
consistent between jruby and standard ruby now.

On Mon, Oct 6, 2008 at 3:59 PM, Erik Hatcher <erik@...>wrote:

> Unfortunately, to call varargs from JRuby (so I just learned from irc's
> #jruby), you have to do this:
>
> query.add_facet_field(['cat'].to_java :string)
>
> anyway, for our solr-jruby purposes, i think we can stick to using the
> ModifiableSolrParams API instead of SolrQuery, but we still get varargs
> issues.
>
> thankfully Ruby solves this relatively nicely...
>
> solr = EmbeddedSolrServer.new(container, "core1")
> query = ModifiableSolrParams.new
> def query.abcd(key, values)
> add(key,(values.is_a?(Array) ? values : [values]).to_java(:string))
> end
> query.abcd('q','*:*')
> query.abcd('facet.field',['cat'])
> response = solr.query(query)
>
> We'll just need to sprinkle a little Ruby fairy dust on some of this clunky
> stuff. It's not all that bad. But the varargs one is strange to me, but
> I'm sure there is some deep technical reason why that support isn't
> intuitive.
>
> Erik
>
>
>
> On Oct 6, 2008, at 1:36 AM, Ryan McKinley wrote:
>
> just guessing, but try passing that as a string[] rather then just a
>> string
>>
>> The java function takes ( string ... v ) -- in java that means you can
>> pass either a string or an array, or a variable length list of strings. I'm
>> guessing that does something funny with jruby
>>
>> ryan
>>
>>
>> On Oct 5, 2008, at 10:54 PM, Matt Mitchell wrote:
>>
>> I've been messing around with jRuby and SolrJ with nice results. But
>>> can't
>>> seem to get the SolrQuery.addFacetField method to work. I get this error:
>>>
>>> "for method addFacetField expected [[Ljava.lang.String;]; got:
>>> [java.lang.String]; error: argument type mismatch (TypeError)"
>>>
>>> I'm using it like:
>>>
>>> query.add_facet_field 'cat'
>>>
>>> and also tried:
>>>
>>> query.addFacetField 'cat'
>>>
>>> Any ideas as to what I'm doing wrong?
>>>
>>> Matt
>>>
>>> p.s. The other methods I've tried work great: set_query, set_query_type,
>>> set_facet, set_facet_limit, set_facet_min_count, set_include_score,
>>> add_sort_field.
>>>
>>
>