[jira] Created: (SOLR-1395) Integrate Katta

Integrate Katta
---------------

                 Key: SOLR-1395
                 URL: https://issues.apache.org/jira/browse/SOLR-1395
             Project: Solr
          Issue Type: New Feature
    Affects Versions: 1.4
            Reporter: Jason Rutherglen
            Priority: Minor
             Fix For: 1.5


We'll integrate Katta into Solr so that:

* Distributed search uses Hadoop RPC

* Shard/SolrCore distribution and management

* Zookeeper based failover

* Indexes may be built using Hadoop

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/SOLR-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749389#action_12749389 ]

Noble Paul commented on SOLR-1395:
----------------------------------

Why should this be a Solr issue? What is missing in Solr which prevents you from integrating katta into Solr as a some kind of plugin?

> Integrate Katta
> ---------------
>
>                 Key: SOLR-1395
>                 URL: https://issues.apache.org/jira/browse/SOLR-1395
>             Project: Solr
>          Issue Type: New Feature
>    Affects Versions: 1.4
>            Reporter: Jason Rutherglen
>            Priority: Minor
>             Fix For: 1.5
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> We'll integrate Katta into Solr so that:
> * Distributed search uses Hadoop RPC
> * Shard/SolrCore distribution and management
> * Zookeeper based failover
> * Indexes may be built using Hadoop

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/SOLR-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jason Rutherglen updated SOLR-1395:
-----------------------------------

    Attachment: KATTA-SOLR.patch
                SOLR-1395.patch

This is our first cut at integrating Katta with Solr. The
KattaClientTest test case shows a Katta cluster being created
locally, a couple of cores/shards being placed into the cluster,
then a query being executed that returns the correct number of
results. It takes about 30s - 1.5min to run (hopefully that can
be reduced?).

Today Solr shards map to Solr servers. Here we map shards to
cores, where there can be multiple shards per server or in Katta
parlance a node. We assume the shards exist in Hadoop HDFS.
Katta copies the shards to a local Solr server to make them
searchable (and incrementally updateable).

h3. Instructions for Installation

* Download Katta trunk "svn co
https://katta.svn.sourceforge.net/svnroot/katta/trunk
kattatrunk". Download the KATTA-SOLR.patch to kattatrunk. run "patch
-p 0 -i KATTA-SOLR.patch", "ant -jar", "ant jar-test".

* Download a Solr trunk "svn co
http://svn.apache.org/repos/asf/lucene/solr/trunk solrtrunk".
Copy from kattatrunk: lib/log4j-1.2.13.jar
lib/zookeeper-3.1.1.jar lib/hadoop-core-0.19.0.jar
build/katta-core-0.6-dev.jar build/test-katta-core-0.6-dev.jar
to solrtrunk/lib

* Download SOLR-1395.patch to solrtrunk. Run "patch -p 0 -i
SOLR-1395.patch".

* Run a test while in solrtrunk "ant test-core
-Dtestcase=KattaClientTest"

h3. General Notes

* SearchHandler's HttpCommComponent has been abstracted out.
There's a CommComponent interface, AbstractCommComponent
implements the generic multithreading ShardRequest ->
ShardResponse logic. EmbeddedSearchHandler executes requests on
a set of local cores. HttpCommComponent implements requests over
HTTP. KattaCommComponent distributes requests using Katta's
Hadoop RPC mechanism.

* The patch enables all of Solr's distributed request types. All
current distributed requests should work as is with no
modifications.

* Shards/Solr cores may be managed dynamically and remotely
administered from a centralized location (whereas today Solr
typically requires SSHing and manually editing files etc)

* Solr Katta has built in failover, this is tested in
KattaClientFailoverTest

* When a shard is deployed to a Solr server, the schema and
solrconfig are deployed with it. This begs the question of how
updates to the solrconfig and schema are deployed. Redeploying
solrconfig is fairly simple, whereas a schema change implies
recreating the entire shard.

* Maybe there's an easy way to interface with Hadoop index
creation (i.e. as easy as Solr's HTTP based update command)

The patch was created by Jason Venner and Jason Rutherglen

> Integrate Katta
> ---------------
>
>                 Key: SOLR-1395
>                 URL: https://issues.apache.org/jira/browse/SOLR-1395
>             Project: Solr
>          Issue Type: New Feature
>    Affects Versions: 1.4
>            Reporter: Jason Rutherglen
>            Priority: Minor
>             Fix For: 1.5
>
>         Attachments: KATTA-SOLR.patch, SOLR-1395.patch
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> We'll integrate Katta into Solr so that:
> * Distributed search uses Hadoop RPC
> * Shard/SolrCore distribution and management
> * Zookeeper based failover
> * Indexes may be built using Hadoop

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/SOLR-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12751980#action_12751980 ]

Stefan Groschupf commented on SOLR-1395:
----------------------------------------

Jason please note that the latest katta code is actually in sourceforges git repo not in svn.

> Integrate Katta
> ---------------
>
>                 Key: SOLR-1395
>                 URL: https://issues.apache.org/jira/browse/SOLR-1395
>             Project: Solr
>          Issue Type: New Feature
>    Affects Versions: 1.4
>            Reporter: Jason Rutherglen
>            Priority: Minor
>             Fix For: 1.5
>
>         Attachments: KATTA-SOLR.patch, SOLR-1395.patch
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> We'll integrate Katta into Solr so that:
> * Distributed search uses Hadoop RPC
> * Shard/SolrCore distribution and management
> * Zookeeper based failover
> * Indexes may be built using Hadoop

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/SOLR-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12754241#action_12754241 ]

Jason Rutherglen commented on SOLR-1395:
----------------------------------------

I added a wiki page at: http://wiki.apache.org/solr/KattaIntegration

> Integrate Katta
> ---------------
>
>                 Key: SOLR-1395
>                 URL: https://issues.apache.org/jira/browse/SOLR-1395
>             Project: Solr
>          Issue Type: New Feature
>    Affects Versions: 1.4
>            Reporter: Jason Rutherglen
>            Priority: Minor
>             Fix For: 1.5
>
>         Attachments: KATTA-SOLR.patch, SOLR-1395.patch
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> We'll integrate Katta into Solr so that:
> * Distributed search uses Hadoop RPC
> * Shard/SolrCore distribution and management
> * Zookeeper based failover
> * Indexes may be built using Hadoop

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/SOLR-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jason Rutherglen updated SOLR-1395:
-----------------------------------

    Attachment: hadoop-core-0.19.0.jar
                log4j-1.2.13.jar
                zookeeper-3.2.1.jar

These are the external libraries necessary to run the test

> Integrate Katta
> ---------------
>
>                 Key: SOLR-1395
>                 URL: https://issues.apache.org/jira/browse/SOLR-1395
>             Project: Solr
>          Issue Type: New Feature
>    Affects Versions: 1.4
>            Reporter: Jason Rutherglen
>            Priority: Minor
>             Fix For: 1.5
>
>         Attachments: hadoop-core-0.19.0.jar, KATTA-SOLR.patch, log4j-1.2.13.jar, SOLR-1395.patch, zookeeper-3.2.1.jar
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> We'll integrate Katta into Solr so that:
> * Distributed search uses Hadoop RPC
> * Shard/SolrCore distribution and management
> * Zookeeper based failover
> * Indexes may be built using Hadoop

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/SOLR-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12754662#action_12754662 ]

jv ning commented on SOLR-1395:
-------------------------------

Jason and I have a couple of small changes that make this simpler to use, and a first faq entry.
If you get a NullPointerException in mergeId's a likely cause is a schema missmatch on the unique id field between an index served by a shard, and the top level solr instance performing the search.

> Integrate Katta
> ---------------
>
>                 Key: SOLR-1395
>                 URL: https://issues.apache.org/jira/browse/SOLR-1395
>             Project: Solr
>          Issue Type: New Feature
>    Affects Versions: 1.4
>            Reporter: Jason Rutherglen
>            Priority: Minor
>             Fix For: 1.5
>
>         Attachments: hadoop-core-0.19.0.jar, KATTA-SOLR.patch, log4j-1.2.13.jar, SOLR-1395.patch, zookeeper-3.2.1.jar
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> We'll integrate Katta into Solr so that:
> * Distributed search uses Hadoop RPC
> * Shard/SolrCore distribution and management
> * Zookeeper based failover
> * Indexes may be built using Hadoop

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/SOLR-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12754700#action_12754700 ]

Noble Paul commented on SOLR-1395:
----------------------------------

Jason , why don't you separate issue for the CommComponent.It is useful for Solr even w/o Katta

> Integrate Katta
> ---------------
>
>                 Key: SOLR-1395
>                 URL: https://issues.apache.org/jira/browse/SOLR-1395
>             Project: Solr
>          Issue Type: New Feature
>    Affects Versions: 1.4
>            Reporter: Jason Rutherglen
>            Priority: Minor
>             Fix For: 1.5
>
>         Attachments: hadoop-core-0.19.0.jar, KATTA-SOLR.patch, log4j-1.2.13.jar, SOLR-1395.patch, zookeeper-3.2.1.jar
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> We'll integrate Katta into Solr so that:
> * Distributed search uses Hadoop RPC
> * Shard/SolrCore distribution and management
> * Zookeeper based failover
> * Indexes may be built using Hadoop

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/SOLR-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755228#action_12755228 ]

Jason Rutherglen commented on SOLR-1395:
----------------------------------------

Noble, great idea!  I opened an issue at SOLR-1431.

> Integrate Katta
> ---------------
>
>                 Key: SOLR-1395
>                 URL: https://issues.apache.org/jira/browse/SOLR-1395
>             Project: Solr
>          Issue Type: New Feature
>    Affects Versions: 1.4
>            Reporter: Jason Rutherglen
>            Priority: Minor
>             Fix For: 1.5
>
>         Attachments: hadoop-core-0.19.0.jar, KATTA-SOLR.patch, log4j-1.2.13.jar, SOLR-1395.patch, zookeeper-3.2.1.jar
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> We'll integrate Katta into Solr so that:
> * Distributed search uses Hadoop RPC
> * Shard/SolrCore distribution and management
> * Zookeeper based failover
> * Indexes may be built using Hadoop

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/SOLR-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jason Rutherglen updated SOLR-1395:
-----------------------------------

    Attachment: SOLR-1395.patch

New patch updated to Katta's latest from Git.  It's slimmed down a bit, removing the various extraneous config files etc.

> Integrate Katta
> ---------------
>
>                 Key: SOLR-1395
>                 URL: https://issues.apache.org/jira/browse/SOLR-1395
>             Project: Solr
>          Issue Type: New Feature
>    Affects Versions: 1.4
>            Reporter: Jason Rutherglen
>            Priority: Minor
>             Fix For: 1.5
>
>         Attachments: hadoop-core-0.19.0.jar, KATTA-SOLR.patch, log4j-1.2.13.jar, SOLR-1395.patch, SOLR-1395.patch, zookeeper-3.2.1.jar
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> We'll integrate Katta into Solr so that:
> * Distributed search uses Hadoop RPC
> * Shard/SolrCore distribution and management
> * Zookeeper based failover
> * Indexes may be built using Hadoop

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/SOLR-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jason Rutherglen updated SOLR-1395:
-----------------------------------

    Attachment: zkclient-0.1-dev.jar
                katta-core-0.6-dev.jar
                test-katta-core-0.6-dev.jar

Copy these libraries into lib/ before executing the test.  The Katta jars are somewhat custom.  I'll post a patch there shortly.

> Integrate Katta
> ---------------
>
>                 Key: SOLR-1395
>                 URL: https://issues.apache.org/jira/browse/SOLR-1395
>             Project: Solr
>          Issue Type: New Feature
>    Affects Versions: 1.4
>            Reporter: Jason Rutherglen
>            Priority: Minor
>             Fix For: 1.5
>
>         Attachments: hadoop-core-0.19.0.jar, katta-core-0.6-dev.jar, KATTA-SOLR.patch, log4j-1.2.13.jar, SOLR-1395.patch, SOLR-1395.patch, test-katta-core-0.6-dev.jar, zkclient-0.1-dev.jar, zookeeper-3.2.1.jar
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> We'll integrate Katta into Solr so that:
> * Distributed search uses Hadoop RPC
> * Shard/SolrCore distribution and management
> * Zookeeper based failover
> * Indexes may be built using Hadoop

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/SOLR-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jason Rutherglen updated SOLR-1395:
-----------------------------------

    Attachment:     (was: KATTA-SOLR.patch)

> Integrate Katta
> ---------------
>
>                 Key: SOLR-1395
>                 URL: https://issues.apache.org/jira/browse/SOLR-1395
>             Project: Solr
>          Issue Type: New Feature
>    Affects Versions: 1.4
>            Reporter: Jason Rutherglen
>            Priority: Minor
>             Fix For: 1.5
>
>         Attachments: hadoop-core-0.19.0.jar, katta-core-0.6-dev.jar, log4j-1.2.13.jar, SOLR-1395.patch, SOLR-1395.patch, test-katta-core-0.6-dev.jar, zkclient-0.1-dev.jar, zookeeper-3.2.1.jar
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> We'll integrate Katta into Solr so that:
> * Distributed search uses Hadoop RPC
> * Shard/SolrCore distribution and management
> * Zookeeper based failover
> * Indexes may be built using Hadoop

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/SOLR-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jason Rutherglen updated SOLR-1395:
-----------------------------------

    Attachment: SOLR-1395.patch

Updated the KattaRequest class to properly serialize the SolrParams.

> Integrate Katta
> ---------------
>
>                 Key: SOLR-1395
>                 URL: https://issues.apache.org/jira/browse/SOLR-1395
>             Project: Solr
>          Issue Type: New Feature
>    Affects Versions: 1.4
>            Reporter: Jason Rutherglen
>            Priority: Minor
>             Fix For: 1.5
>
>         Attachments: hadoop-core-0.19.0.jar, katta-core-0.6-dev.jar, log4j-1.2.13.jar, SOLR-1395.patch, SOLR-1395.patch, SOLR-1395.patch, test-katta-core-0.6-dev.jar, zkclient-0.1-dev.jar, zookeeper-3.2.1.jar
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> We'll integrate Katta into Solr so that:
> * Distributed search uses Hadoop RPC
> * Shard/SolrCore distribution and management
> * Zookeeper based failover
> * Indexes may be built using Hadoop

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/SOLR-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

jv ning updated SOLR-1395:
--------------------------

    Attachment: solr-1395-1431.patch

the file /tmp/solr-1395-1431.patch is a combined patch of 1431 and 1395.
A small api change in the query parser required a small code change.

> Integrate Katta
> ---------------
>
>                 Key: SOLR-1395
>                 URL: https://issues.apache.org/jira/browse/SOLR-1395
>             Project: Solr
>          Issue Type: New Feature
>    Affects Versions: 1.4
>            Reporter: Jason Rutherglen
>            Priority: Minor
>             Fix For: 1.5
>
>         Attachments: hadoop-core-0.19.0.jar, katta-core-0.6-dev.jar, log4j-1.2.13.jar, solr-1395-1431.patch, SOLR-1395.patch, SOLR-1395.patch, SOLR-1395.patch, test-katta-core-0.6-dev.jar, zkclient-0.1-dev.jar, zookeeper-3.2.1.jar
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> We'll integrate Katta into Solr so that:
> * Distributed search uses Hadoop RPC
> * Shard/SolrCore distribution and management
> * Zookeeper based failover
> * Indexes may be built using Hadoop

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/SOLR-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12760635#action_12760635 ]

jv ning edited comment on SOLR-1395 at 9/29/09 9:07 AM:
--------------------------------------------------------

the file /tmp/solr-1395-1431.patch is a combined patch of 1431 and 1395.
A small api change in the query string parser required a small code change.

      was (Author: jv_ning):
    the file /tmp/solr-1395-1431.patch is a combined patch of 1431 and 1395.
A small api change in the query parser required a small code change.
 

> Integrate Katta
> ---------------
>
>                 Key: SOLR-1395
>                 URL: https://issues.apache.org/jira/browse/SOLR-1395
>             Project: Solr
>          Issue Type: New Feature
>    Affects Versions: 1.4
>            Reporter: Jason Rutherglen
>            Priority: Minor
>             Fix For: 1.5
>
>         Attachments: hadoop-core-0.19.0.jar, katta-core-0.6-dev.jar, log4j-1.2.13.jar, solr-1395-1431.patch, SOLR-1395.patch, SOLR-1395.patch, SOLR-1395.patch, test-katta-core-0.6-dev.jar, zkclient-0.1-dev.jar, zookeeper-3.2.1.jar
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> We'll integrate Katta into Solr so that:
> * Distributed search uses Hadoop RPC
> * Shard/SolrCore distribution and management
> * Zookeeper based failover
> * Indexes may be built using Hadoop

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/SOLR-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12760635#action_12760635 ]

jv ning edited comment on SOLR-1395 at 9/29/09 9:08 AM:
--------------------------------------------------------

the file /tmp/solr-1395-1431.patch is a combined patch of 1431 and 1395.
A small api change in the query string creator required a small code change.
ClientUtils.toQueryString, now prefixes the returned query string with a '?' character

      was (Author: jv_ning):
    the file /tmp/solr-1395-1431.patch is a combined patch of 1431 and 1395.
A small api change in the query string parser required a small code change.
 

> Integrate Katta
> ---------------
>
>                 Key: SOLR-1395
>                 URL: https://issues.apache.org/jira/browse/SOLR-1395
>             Project: Solr
>          Issue Type: New Feature
>    Affects Versions: 1.4
>            Reporter: Jason Rutherglen
>            Priority: Minor
>             Fix For: 1.5
>
>         Attachments: hadoop-core-0.19.0.jar, katta-core-0.6-dev.jar, log4j-1.2.13.jar, solr-1395-1431.patch, SOLR-1395.patch, SOLR-1395.patch, SOLR-1395.patch, test-katta-core-0.6-dev.jar, zkclient-0.1-dev.jar, zookeeper-3.2.1.jar
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> We'll integrate Katta into Solr so that:
> * Distributed search uses Hadoop RPC
> * Shard/SolrCore distribution and management
> * Zookeeper based failover
> * Indexes may be built using Hadoop

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/SOLR-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jason Venner (at ning) updated SOLR-1395:
-----------------------------------------

    Attachment: katta.zk.properties
                katta.node.properties
                solr-1395-1431-3.patch

/tmp/solr-1395-1431-3.patch contains an additional unit test for the query string serialization code, and two additional classes that allow for deployment to katta.

WIth this jar, a katta client node may be started via
katta-daemon.sh start katta\ startNode org.apache.solr.katta.DeployableSolrKattaServer
The system properties that control the node startup are

solr.server.name - the property to look for the server name, default proxy
solr.home - the property to look for the server root, default solrHome
solr.config.file - the property to look for the server config file name, default solr.xml

These will be used to find a solr configuration to run the embedded server which will search the deployed shards.

Index shards may be deployed via the standard katta mechanism of katta addIndex index-name shared-path-to-index
I use the zip files produced by SOLR-1301 and deploy from hdfs.

For searching, create a solr configuration with a handler:

  <requestHandler name="standard" class="solr.KattaRequestHandler" default="true">
    <!-- default values for query parameters -->
     <lst name="defaults">
       <str name="echoParams">explicit</str>
       <!--
       <int name="rows">10</int>
       <str name="fl">*</str>
       <str name="version">2.1</str>
        -->
           <str name="shards">*</str>
     </lst>
  </requestHandler>

This will search all deployed shards, replace the shards parameter with an explicit shard list if you only wish to query an explicit subset with this query handler.

The solr instance for search will need the zookeeper information.

conf/katta.node.properties
conf/katta.zk.properties, replace the zookeeper nodes with your clusters nodes

I tend to run java -d64 -Xmx2g -Dkatta.request.timeout=100000 start.jar
for my testing work as my cluster is on the far side of a couple of firewals

> Integrate Katta
> ---------------
>
>                 Key: SOLR-1395
>                 URL: https://issues.apache.org/jira/browse/SOLR-1395
>             Project: Solr
>          Issue Type: New Feature
>    Affects Versions: 1.4
>            Reporter: Jason Rutherglen
>            Priority: Minor
>             Fix For: 1.5
>
>         Attachments: hadoop-core-0.19.0.jar, katta-core-0.6-dev.jar, katta.node.properties, katta.zk.properties, log4j-1.2.13.jar, solr-1395-1431-3.patch, solr-1395-1431.patch, SOLR-1395.patch, SOLR-1395.patch, SOLR-1395.patch, test-katta-core-0.6-dev.jar, zkclient-0.1-dev.jar, zookeeper-3.2.1.jar
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> We'll integrate Katta into Solr so that:
> * Distributed search uses Hadoop RPC
> * Shard/SolrCore distribution and management
> * Zookeeper based failover
> * Indexes may be built using Hadoop

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/SOLR-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12760726#action_12760726 ]

Jason Venner (at ning) edited comment on SOLR-1395 at 9/29/09 3:09 PM:
-----------------------------------------------------------------------

/tmp/solr-1395-1431-3.patch contains an additional unit test for the query string serialization code, and two additional classes that allow for deployment to katta.

WIth this jar, a katta client node may be started via
katta-daemon.sh start katta\ startNode org.apache.solr.katta.DeployableSolrKattaServer
The system properties that control the node startup are

solr.server.name - the property to look for the server name, default proxy
solr.home - the property to look for the server root, default solrHome
solr.config.file - the property to look for the server config file name, default solr.xml

These will be used to find a solr configuration to run the embedded server which will search the deployed shards.

Index shards may be deployed via the standard katta mechanism of katta addIndex index-name shared-path-to-index
I use the zip files produced by SOLR-1301 and deploy from hdfs.

For searching, create a solr configuration with a handler:

  <requestHandler name="standard" class="solr.KattaRequestHandler" default="true">
    <!-- default values for query parameters -->
     <lst name="defaults">
       <str name="echoParams">explicit</str>
       <!--
       <int name="rows">10</int>
       <str name="fl">*</str>
       <str name="version">2.1</str>
        -->
           <str name="shards">*</str>
     </lst>
  </requestHandler>

This will search all deployed shards, replace the shards parameter with an explicit shard list if you only wish to query an explicit subset with this query handler.

The solr instance for search will need the zookeeper information.

conf/katta.node.properties
conf/katta.zk.properties, replace the zookeeper nodes with your clusters nodes

I tend to run java -d64 -Xmx2g -Dkatta.request.timeout=100000 start.jar
for my testing work as my cluster is on the far side of a couple of firewallss

I also have to store my katta.zk.properties file in the start.jar for some reason



      was (Author: jv_ning):
    /tmp/solr-1395-1431-3.patch contains an additional unit test for the query string serialization code, and two additional classes that allow for deployment to katta.

WIth this jar, a katta client node may be started via
katta-daemon.sh start katta\ startNode org.apache.solr.katta.DeployableSolrKattaServer
The system properties that control the node startup are

solr.server.name - the property to look for the server name, default proxy
solr.home - the property to look for the server root, default solrHome
solr.config.file - the property to look for the server config file name, default solr.xml

These will be used to find a solr configuration to run the embedded server which will search the deployed shards.

Index shards may be deployed via the standard katta mechanism of katta addIndex index-name shared-path-to-index
I use the zip files produced by SOLR-1301 and deploy from hdfs.

For searching, create a solr configuration with a handler:

  <requestHandler name="standard" class="solr.KattaRequestHandler" default="true">
    <!-- default values for query parameters -->
     <lst name="defaults">
       <str name="echoParams">explicit</str>
       <!--
       <int name="rows">10</int>
       <str name="fl">*</str>
       <str name="version">2.1</str>
        -->
           <str name="shards">*</str>
     </lst>
  </requestHandler>

This will search all deployed shards, replace the shards parameter with an explicit shard list if you only wish to query an explicit subset with this query handler.

The solr instance for search will need the zookeeper information.

conf/katta.node.properties
conf/katta.zk.properties, replace the zookeeper nodes with your clusters nodes

I tend to run java -d64 -Xmx2g -Dkatta.request.timeout=100000 start.jar
for my testing work as my cluster is on the far side of a couple of firewals


 

> Integrate Katta
> ---------------
>
>                 Key: SOLR-1395
>                 URL: https://issues.apache.org/jira/browse/SOLR-1395
>             Project: Solr
>          Issue Type: New Feature
>    Affects Versions: 1.4
>            Reporter: Jason Rutherglen
>            Priority: Minor
>             Fix For: 1.5
>
>         Attachments: hadoop-core-0.19.0.jar, katta-core-0.6-dev.jar, katta.node.properties, katta.zk.properties, log4j-1.2.13.jar, solr-1395-1431-3.patch, solr-1395-1431.patch, SOLR-1395.patch, SOLR-1395.patch, SOLR-1395.patch, test-katta-core-0.6-dev.jar, zkclient-0.1-dev.jar, zookeeper-3.2.1.jar
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> We'll integrate Katta into Solr so that:
> * Distributed search uses Hadoop RPC
> * Shard/SolrCore distribution and management
> * Zookeeper based failover
> * Indexes may be built using Hadoop

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/SOLR-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12760955#action_12760955 ]

Jason Rutherglen commented on SOLR-1395:
----------------------------------------

Jason, Can you upload a SOLR-1395 only patch?  That will help in seeing the SOLR-1395 specific changes.

I think the next step is to remove the dependency on separate property files, as I find these hard to manage (they are too numerous).

> Integrate Katta
> ---------------
>
>                 Key: SOLR-1395
>                 URL: https://issues.apache.org/jira/browse/SOLR-1395
>             Project: Solr
>          Issue Type: New Feature
>    Affects Versions: 1.4
>            Reporter: Jason Rutherglen
>            Priority: Minor
>             Fix For: 1.5
>
>         Attachments: hadoop-core-0.19.0.jar, katta-core-0.6-dev.jar, katta.node.properties, katta.zk.properties, log4j-1.2.13.jar, solr-1395-1431-3.patch, solr-1395-1431.patch, SOLR-1395.patch, SOLR-1395.patch, SOLR-1395.patch, test-katta-core-0.6-dev.jar, zkclient-0.1-dev.jar, zookeeper-3.2.1.jar
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> We'll integrate Katta into Solr so that:
> * Distributed search uses Hadoop RPC
> * Shard/SolrCore distribution and management
> * Zookeeper based failover
> * Indexes may be built using Hadoop

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/SOLR-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12760968#action_12760968 ]

Jason Venner (at ning) commented on SOLR-1395:
----------------------------------------------

I was unable to separate them cleanly, so no.

> Integrate Katta
> ---------------
>
>                 Key: SOLR-1395
>                 URL: https://issues.apache.org/jira/browse/SOLR-1395
>             Project: Solr
>          Issue Type: New Feature
>    Affects Versions: 1.4
>            Reporter: Jason Rutherglen
>            Priority: Minor
>             Fix For: 1.5
>
>         Attachments: hadoop-core-0.19.0.jar, katta-core-0.6-dev.jar, katta.node.properties, katta.zk.properties, log4j-1.2.13.jar, solr-1395-1431-3.patch, solr-1395-1431.patch, SOLR-1395.patch, SOLR-1395.patch, SOLR-1395.patch, test-katta-core-0.6-dev.jar, zkclient-0.1-dev.jar, zookeeper-3.2.1.jar
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> We'll integrate Katta into Solr so that:
> * Distributed search uses Hadoop RPC
> * Shard/SolrCore distribution and management
> * Zookeeper based failover
> * Indexes may be built using Hadoop

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.