Opaque replication failures

[Michael Ford]

Hi,

I have a multicore Solr 1.4 setup.  core_master is a 3.7G master for
replication, and core_slave is a 500 byte slave pointing to the
master.  I'm using the example replication configuration from
solrconfig.xml, with ${enable.master} and ${enable.slave} properties
so that the master and slave can use the same solrconfig.xml.

When I attempt to replicate (every 60 seconds or by pressing the
button on the slave replication admin page), it doesn't work.
Unfortunately, neither the admin page nor the REST API "details"
command show anything useful, and the logs show no errors.

How can I get insight into what is causing the failure?  I assume it's
some configuration problem but don't know where to start.

Thanks in advance for any help!  Config files are below.
Michael



Here is my solr.xml:

<?xml version='1.0' encoding='UTF-8'?><solr sharedLib="lib" persistent="true">
<cores adminPath="/admin/cores" shareSchema="true">
  <core name="core_master" instanceDir="." dataDir="/home/search/solr/data/5">
    <property name="enable.master" value="true" />
  </core>
  <core name="core_slave" instanceDir="." dataDir="/home/search/solr/data/1">
    <property name="enable.slave" value="true" />
  </core>
</cores>
</solr>

And here's the relevant chunk of my solrconfig.xml:

<requestHandler name="/replication" class="solr.ReplicationHandler" >
    <lst name="master">
        <str name="enable">${enable.master:false}</str>
        <str name="replicateAfter">commit</str>
    </lst>
    <lst name="slave">
        <str name="enable">${enable.slave:false}</str>
        <str name="masterUrl">http://localhost:31000/solr/core_master/replication</str>
        <str name="pollInterval">00:00:60</str>
     </lst>
</requestHandler>

Here's what the "details" command on the slave has to say -- nothing
explanatory that I can see.  Is the "isReplicating=false" worrying?

<lst name="details">

  <str name="indexSize">589 bytes</str>
  <str name="indexPath">/home/search/solr/data/1/index</str>
  <arr name="commits"/>
  <str name="isMaster">false</str>
  <str name="isSlave">true</str>
  <long name="indexVersion">1254772638413</long>
  <long name="generation">2</long>

  <lst name="slave">
    <lst name="masterDetails">
      <str name="indexSize">3.75 GB</str>
      <str name="indexPath">/home/search/solr/data/5/index</str>
      <arr name="commits"/>
      <str name="isMaster">true</str>
      <str name="isSlave">false</str>
      <long name="indexVersion">1254772639291</long>
      <long name="generation">156</long>
    </lst>
    <str name="masterUrl">http://localhost:31000/solr/core_master/replication</str>
    <str name="pollInterval">00:00:60</str>
    <str name="indexReplicatedAt">Wed Oct 14 14:25:22 EDT 2009</str>

    <arr name="indexReplicatedAtList">
      <str>Wed Oct 14 14:25:22 EDT 2009</str>
      <str>Wed Oct 14 14:25:22 EDT 2009</str>
      <str>Wed Oct 14 14:25:21 EDT 2009</str>
      <str>Wed Oct 14 14:24:27 EDT 2009</str>
      (etc)
    </arr>
    <arr name="replicationFailedAtList">
      <str>Wed Oct 14 14:25:22 EDT 2009</str>
      <str>Wed Oct 14 14:25:22 EDT 2009</str>
      <str>Wed Oct 14 14:25:21 EDT 2009</str>
      <str>Wed Oct 14 14:24:27 EDT 2009</str>
      (etc)
    </arr>

    <str name="timesIndexReplicated">1481</str>
    <str name="lastCycleBytesDownloaded">0</str>
    <str name="timesFailed">1481</str>
    <str name="replicationFailedAt">Wed Oct 14 14:25:22 EDT 2009</str>
    <str name="previousCycleTimeInSeconds">0</str>
    <str name="isReplicating">false</str>
  </lst>

</lst>

[Michael Ford]

I just tried setting up replication between two cores with the Nov 2
nightly, and got the same result as below: replication reports as
failed but doesn't tell me why.

Is replication not allowed from a master core to a slave core within
the same Solr instance?  Or is there a way for me to find out if there
is something wrong with my index (which otherwise appears OK)?

Thanks,
Michael

On Wed, Oct 14, 2009 at 1:33 PM, Michael <solrcoder@...> wrote:

> Hi,
>
> I have a multicore Solr 1.4 setup.  core_master is a 3.7G master for
> replication, and core_slave is a 500 byte slave pointing to the
> master.  I'm using the example replication configuration from
> solrconfig.xml, with ${enable.master} and ${enable.slave} properties
> so that the master and slave can use the same solrconfig.xml.
>
> When I attempt to replicate (every 60 seconds or by pressing the
> button on the slave replication admin page), it doesn't work.
> Unfortunately, neither the admin page nor the REST API "details"
> command show anything useful, and the logs show no errors.
>
> How can I get insight into what is causing the failure?  I assume it's
> some configuration problem but don't know where to start.
>
> Thanks in advance for any help!  Config files are below.
> Michael
>
>
>
> Here is my solr.xml:
>
> <?xml version='1.0' encoding='UTF-8'?><solr sharedLib="lib" persistent="true">
> <cores adminPath="/admin/cores" shareSchema="true">
>  <core name="core_master" instanceDir="." dataDir="/home/search/solr/data/5">
>    <property name="enable.master" value="true" />
>  </core>
>  <core name="core_slave" instanceDir="." dataDir="/home/search/solr/data/1">
>    <property name="enable.slave" value="true" />
>  </core>
> </cores>
> </solr>
>
> And here's the relevant chunk of my solrconfig.xml:
>
> <requestHandler name="/replication" class="solr.ReplicationHandler" >
>    <lst name="master">
>        <str name="enable">${enable.master:false}</str>
>        <str name="replicateAfter">commit</str>
>    </lst>
>    <lst name="slave">
>        <str name="enable">${enable.slave:false}</str>
>        <str name="masterUrl">http://localhost:31000/solr/core_master/replication</str>
>        <str name="pollInterval">00:00:60</str>
>     </lst>
> </requestHandler>
>
> Here's what the "details" command on the slave has to say -- nothing
> explanatory that I can see.  Is the "isReplicating=false" worrying?
>
> <lst name="details">
>
>  <str name="indexSize">589 bytes</str>
>  <str name="indexPath">/home/search/solr/data/1/index</str>
>  <arr name="commits"/>
>  <str name="isMaster">false</str>
>  <str name="isSlave">true</str>
>  <long name="indexVersion">1254772638413</long>
>  <long name="generation">2</long>
>
>  <lst name="slave">
>    <lst name="masterDetails">
>      <str name="indexSize">3.75 GB</str>
>      <str name="indexPath">/home/search/solr/data/5/index</str>
>      <arr name="commits"/>
>      <str name="isMaster">true</str>
>      <str name="isSlave">false</str>
>      <long name="indexVersion">1254772639291</long>
>      <long name="generation">156</long>
>    </lst>
>    <str name="masterUrl">http://localhost:31000/solr/core_master/replication</str>
>    <str name="pollInterval">00:00:60</str>
>    <str name="indexReplicatedAt">Wed Oct 14 14:25:22 EDT 2009</str>
>
>    <arr name="indexReplicatedAtList">
>      <str>Wed Oct 14 14:25:22 EDT 2009</str>
>      <str>Wed Oct 14 14:25:22 EDT 2009</str>
>      <str>Wed Oct 14 14:25:21 EDT 2009</str>
>      <str>Wed Oct 14 14:24:27 EDT 2009</str>
>      (etc)
>    </arr>
>    <arr name="replicationFailedAtList">
>      <str>Wed Oct 14 14:25:22 EDT 2009</str>
>      <str>Wed Oct 14 14:25:22 EDT 2009</str>
>      <str>Wed Oct 14 14:25:21 EDT 2009</str>
>      <str>Wed Oct 14 14:24:27 EDT 2009</str>
>      (etc)
>    </arr>
>
>    <str name="timesIndexReplicated">1481</str>
>    <str name="lastCycleBytesDownloaded">0</str>
>    <str name="timesFailed">1481</str>
>    <str name="replicationFailedAt">Wed Oct 14 14:25:22 EDT 2009</str>
>    <str name="previousCycleTimeInSeconds">0</str>
>    <str name="isReplicating">false</str>
>  </lst>
>
> </lst>
>

[Michael Ford]

OK, I've solved this.  For posterity:

The master doesn't make anything available for replication unless you
set replicateAfter="startup", or unless you set
replicateAfter="commit" and then both add a document and execute a
commit.  If you don't do one of those, even manually clicking
"Replicate Now" on the slave will show failures without explaining
why.

With replicateAfter="startup" and "commit" I was able to get a slave
core in the same Solr instance to replicate upon startup and upon
add-doc-and-commit.

Michael

On Tue, Nov 3, 2009 at 11:53 AM, Michael <solrcoder@...> wrote:

> I just tried setting up replication between two cores with the Nov 2
> nightly, and got the same result as below: replication reports as
> failed but doesn't tell me why.
>
> Is replication not allowed from a master core to a slave core within
> the same Solr instance?  Or is there a way for me to find out if there
> is something wrong with my index (which otherwise appears OK)?
>
> Thanks,
> Michael
>
> On Wed, Oct 14, 2009 at 1:33 PM, Michael <solrcoder@...> wrote:
>> Hi,
>>
>> I have a multicore Solr 1.4 setup.  core_master is a 3.7G master for
>> replication, and core_slave is a 500 byte slave pointing to the
>> master.  I'm using the example replication configuration from
>> solrconfig.xml, with ${enable.master} and ${enable.slave} properties
>> so that the master and slave can use the same solrconfig.xml.
>>
>> When I attempt to replicate (every 60 seconds or by pressing the
>> button on the slave replication admin page), it doesn't work.
>> Unfortunately, neither the admin page nor the REST API "details"
>> command show anything useful, and the logs show no errors.
>>
>> How can I get insight into what is causing the failure?  I assume it's
>> some configuration problem but don't know where to start.
>>
>> Thanks in advance for any help!  Config files are below.
>> Michael
>>
>>
>>
>> Here is my solr.xml:
>>
>> <?xml version='1.0' encoding='UTF-8'?><solr sharedLib="lib" persistent="true">
>> <cores adminPath="/admin/cores" shareSchema="true">
>>  <core name="core_master" instanceDir="." dataDir="/home/search/solr/data/5">
>>    <property name="enable.master" value="true" />
>>  </core>
>>  <core name="core_slave" instanceDir="." dataDir="/home/search/solr/data/1">
>>    <property name="enable.slave" value="true" />
>>  </core>
>> </cores>
>> </solr>
>>
>> And here's the relevant chunk of my solrconfig.xml:
>>
>> <requestHandler name="/replication" class="solr.ReplicationHandler" >
>>    <lst name="master">
>>        <str name="enable">${enable.master:false}</str>
>>        <str name="replicateAfter">commit</str>
>>    </lst>
>>    <lst name="slave">
>>        <str name="enable">${enable.slave:false}</str>
>>        <str name="masterUrl">http://localhost:31000/solr/core_master/replication</str>
>>        <str name="pollInterval">00:00:60</str>
>>     </lst>
>> </requestHandler>
>>
>> Here's what the "details" command on the slave has to say -- nothing
>> explanatory that I can see.  Is the "isReplicating=false" worrying?
>>
>> <lst name="details">
>>
>>  <str name="indexSize">589 bytes</str>
>>  <str name="indexPath">/home/search/solr/data/1/index</str>
>>  <arr name="commits"/>
>>  <str name="isMaster">false</str>
>>  <str name="isSlave">true</str>
>>  <long name="indexVersion">1254772638413</long>
>>  <long name="generation">2</long>
>>
>>  <lst name="slave">
>>    <lst name="masterDetails">
>>      <str name="indexSize">3.75 GB</str>
>>      <str name="indexPath">/home/search/solr/data/5/index</str>
>>      <arr name="commits"/>
>>      <str name="isMaster">true</str>
>>      <str name="isSlave">false</str>
>>      <long name="indexVersion">1254772639291</long>
>>      <long name="generation">156</long>
>>    </lst>
>>    <str name="masterUrl">http://localhost:31000/solr/core_master/replication</str>
>>    <str name="pollInterval">00:00:60</str>
>>    <str name="indexReplicatedAt">Wed Oct 14 14:25:22 EDT 2009</str>
>>
>>    <arr name="indexReplicatedAtList">
>>      <str>Wed Oct 14 14:25:22 EDT 2009</str>
>>      <str>Wed Oct 14 14:25:22 EDT 2009</str>
>>      <str>Wed Oct 14 14:25:21 EDT 2009</str>
>>      <str>Wed Oct 14 14:24:27 EDT 2009</str>
>>      (etc)
>>    </arr>
>>    <arr name="replicationFailedAtList">
>>      <str>Wed Oct 14 14:25:22 EDT 2009</str>
>>      <str>Wed Oct 14 14:25:22 EDT 2009</str>
>>      <str>Wed Oct 14 14:25:21 EDT 2009</str>
>>      <str>Wed Oct 14 14:24:27 EDT 2009</str>
>>      (etc)
>>    </arr>
>>
>>    <str name="timesIndexReplicated">1481</str>
>>    <str name="lastCycleBytesDownloaded">0</str>
>>    <str name="timesFailed">1481</str>
>>    <str name="replicationFailedAt">Wed Oct 14 14:25:22 EDT 2009</str>
>>    <str name="previousCycleTimeInSeconds">0</str>
>>    <str name="isReplicating">false</str>
>>  </lst>
>>
>> </lst>
>>
>