[jira] Created: (SOLR-1143) Return partial results when a connection to a shard is refused

Return partial results when a connection to a shard is refused
--------------------------------------------------------------

                 Key: SOLR-1143
                 URL: https://issues.apache.org/jira/browse/SOLR-1143
             Project: Solr
          Issue Type: Improvement
          Components: search
            Reporter: Nicolas Dessaigne


If any shard is down in a distributed search, a ConnectException it thrown.

Here's a little patch that change this behaviour: if we can't connect to a shard (ConnectException), we get partial results from the active shards. As for TimeOut parameter (https://issues.apache.org/jira/browse/SOLR-502), we set the parameter "partialResults" at true.

This patch also adresses a problem expressed in the mailing list about a year ago (http://www.nabble.com/partialResults,-distributed-search---SOLR-502-td19002610.html)

We have a use case that needs this behaviour and we would like to know your thougths about such a behaviour? Should it be the default behaviour for distributed search?


--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/SOLR-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicolas Dessaigne updated SOLR-1143:
------------------------------------

    Attachment: SOLR-1143.patch

> Return partial results when a connection to a shard is refused
> --------------------------------------------------------------
>
>                 Key: SOLR-1143
>                 URL: https://issues.apache.org/jira/browse/SOLR-1143
>             Project: Solr
>          Issue Type: Improvement
>          Components: search
>            Reporter: Nicolas Dessaigne
>         Attachments: SOLR-1143.patch
>
>
> If any shard is down in a distributed search, a ConnectException it thrown.
> Here's a little patch that change this behaviour: if we can't connect to a shard (ConnectException), we get partial results from the active shards. As for TimeOut parameter (https://issues.apache.org/jira/browse/SOLR-502), we set the parameter "partialResults" at true.
> This patch also adresses a problem expressed in the mailing list about a year ago (http://www.nabble.com/partialResults,-distributed-search---SOLR-502-td19002610.html)
> We have a use case that needs this behaviour and we would like to know your thougths about such a behaviour? Should it be the default behaviour for distributed search?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/SOLR-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yonik Seeley updated SOLR-1143:
-------------------------------

    Fix Version/s: 1.4

Seems like this is something we should consider for 1.4

> Return partial results when a connection to a shard is refused
> --------------------------------------------------------------
>
>                 Key: SOLR-1143
>                 URL: https://issues.apache.org/jira/browse/SOLR-1143
>             Project: Solr
>          Issue Type: Improvement
>          Components: search
>            Reporter: Nicolas Dessaigne
>             Fix For: 1.4
>
>         Attachments: SOLR-1143.patch
>
>
> If any shard is down in a distributed search, a ConnectException it thrown.
> Here's a little patch that change this behaviour: if we can't connect to a shard (ConnectException), we get partial results from the active shards. As for TimeOut parameter (https://issues.apache.org/jira/browse/SOLR-502), we set the parameter "partialResults" at true.
> This patch also adresses a problem expressed in the mailing list about a year ago (http://www.nabble.com/partialResults,-distributed-search---SOLR-502-td19002610.html)
> We have a use case that needs this behaviour and we would like to know your thougths about such a behaviour? Should it be the default behaviour for distributed search?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/SOLR-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12738678#action_12738678 ]

Grant Ingersoll commented on SOLR-1143:
---------------------------------------

This needs tests.  

> Return partial results when a connection to a shard is refused
> --------------------------------------------------------------
>
>                 Key: SOLR-1143
>                 URL: https://issues.apache.org/jira/browse/SOLR-1143
>             Project: Solr
>          Issue Type: Improvement
>          Components: search
>            Reporter: Nicolas Dessaigne
>             Fix For: 1.4
>
>         Attachments: SOLR-1143.patch
>
>
> If any shard is down in a distributed search, a ConnectException it thrown.
> Here's a little patch that change this behaviour: if we can't connect to a shard (ConnectException), we get partial results from the active shards. As for TimeOut parameter (https://issues.apache.org/jira/browse/SOLR-502), we set the parameter "partialResults" at true.
> This patch also adresses a problem expressed in the mailing list about a year ago (http://www.nabble.com/partialResults,-distributed-search---SOLR-502-td19002610.html)
> We have a use case that needs this behaviour and we would like to know your thougths about such a behaviour? Should it be the default behaviour for distributed search?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/SOLR-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743478#action_12743478 ]

Artem Russakovskii commented on SOLR-1143:
------------------------------------------

+1 for importance of this feature. If I have 10 shards, I should be able to handle 1 of them going down without returning 0 results to the user.

> Return partial results when a connection to a shard is refused
> --------------------------------------------------------------
>
>                 Key: SOLR-1143
>                 URL: https://issues.apache.org/jira/browse/SOLR-1143
>             Project: Solr
>          Issue Type: Improvement
>          Components: search
>            Reporter: Nicolas Dessaigne
>             Fix For: 1.4
>
>         Attachments: SOLR-1143.patch
>
>
> If any shard is down in a distributed search, a ConnectException it thrown.
> Here's a little patch that change this behaviour: if we can't connect to a shard (ConnectException), we get partial results from the active shards. As for TimeOut parameter (https://issues.apache.org/jira/browse/SOLR-502), we set the parameter "partialResults" at true.
> This patch also adresses a problem expressed in the mailing list about a year ago (http://www.nabble.com/partialResults,-distributed-search---SOLR-502-td19002610.html)
> We have a use case that needs this behaviour and we would like to know your thougths about such a behaviour? Should it be the default behaviour for distributed search?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/SOLR-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743481#action_12743481 ]

Jason Rutherglen commented on SOLR-1143:
----------------------------------------

What happens today when a query times out?  

> Return partial results when a connection to a shard is refused
> --------------------------------------------------------------
>
>                 Key: SOLR-1143
>                 URL: https://issues.apache.org/jira/browse/SOLR-1143
>             Project: Solr
>          Issue Type: Improvement
>          Components: search
>            Reporter: Nicolas Dessaigne
>             Fix For: 1.4
>
>         Attachments: SOLR-1143.patch
>
>
> If any shard is down in a distributed search, a ConnectException it thrown.
> Here's a little patch that change this behaviour: if we can't connect to a shard (ConnectException), we get partial results from the active shards. As for TimeOut parameter (https://issues.apache.org/jira/browse/SOLR-502), we set the parameter "partialResults" at true.
> This patch also adresses a problem expressed in the mailing list about a year ago (http://www.nabble.com/partialResults,-distributed-search---SOLR-502-td19002610.html)
> We have a use case that needs this behaviour and we would like to know your thougths about such a behaviour? Should it be the default behaviour for distributed search?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/SOLR-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743775#action_12743775 ]

Lance Norskog commented on SOLR-1143:
-------------------------------------

If the search subsystem has a problem, the ops team wants to know about it and fix it. This just hides problems.

An example: 2 servers with the same shard are behind a load balancer. One server fails. The load balancer notices this and directs all traffic to the other server.

This is a production network which serves and outside API, where everything is supposed to work >from the viewpoint of the outside API<. When the load balancer gets a failure it usually returns an error on that one request, then marks the server down. So that one search request eventually returns with a "temporary error" condition.

These search requests come from an app server which serves the API. The app server then has the option of retrying one or two times, or returning "service not happy" to the outside calling app.

When I have a problem in my system, I want to find it and fix it.  Ignoring shard errors is ok as an option, and should be there. But, please do not make it the default.  Hiding failures should never be the default.

> Return partial results when a connection to a shard is refused
> --------------------------------------------------------------
>
>                 Key: SOLR-1143
>                 URL: https://issues.apache.org/jira/browse/SOLR-1143
>             Project: Solr
>          Issue Type: Improvement
>          Components: search
>            Reporter: Nicolas Dessaigne
>             Fix For: 1.4
>
>         Attachments: SOLR-1143.patch
>
>
> If any shard is down in a distributed search, a ConnectException it thrown.
> Here's a little patch that change this behaviour: if we can't connect to a shard (ConnectException), we get partial results from the active shards. As for TimeOut parameter (https://issues.apache.org/jira/browse/SOLR-502), we set the parameter "partialResults" at true.
> This patch also adresses a problem expressed in the mailing list about a year ago (http://www.nabble.com/partialResults,-distributed-search---SOLR-502-td19002610.html)
> We have a use case that needs this behaviour and we would like to know your thougths about such a behaviour? Should it be the default behaviour for distributed search?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/SOLR-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Martijn van Groningen updated SOLR-1143:
----------------------------------------

    Attachment: SOLR-1143-2.patch

I have added a test in _TestDistributedSearch_ class. This test sets up a cluster of shards and then kills one shard and then it expects that the search request as a whole to continue. The _TestDistributedSearch_ class in general tests distributed search by having a non distributed instance and a cluster of shards both have the same documents. All results from the cluster are compared with results from the non distributed instance. Some things in the test I added like facets and maxScore could not be tested because one shard in the cluster is down (so part of the corpus is missing). Only the documents that are returned from the shards are compared against the documents in the non distributed instance.

I have also included the option to disable / enable partial results as Lance described. I agree with Lance that ignoring a  shard failure should *not* be enabled by default, if you do not know about this feature then finding the cause of the actual problem might be difficult.

In this patch you return a partial result when a shard has failed by setting _partialResults_ to _true_ in the request or if you want it to for all requests your can add _<bool name="return-partial-results">true</bool>_  to your search handler in your solrconfig.xml. If both are not specified, partial results are disabled. Currently the _partialResults_ parameter overrides the _return-partial-results_ property in the search handler.

> Return partial results when a connection to a shard is refused
> --------------------------------------------------------------
>
>                 Key: SOLR-1143
>                 URL: https://issues.apache.org/jira/browse/SOLR-1143
>             Project: Solr
>          Issue Type: Improvement
>          Components: search
>            Reporter: Nicolas Dessaigne
>             Fix For: 1.4
>
>         Attachments: SOLR-1143-2.patch, SOLR-1143.patch
>
>
> If any shard is down in a distributed search, a ConnectException it thrown.
> Here's a little patch that change this behaviour: if we can't connect to a shard (ConnectException), we get partial results from the active shards. As for TimeOut parameter (https://issues.apache.org/jira/browse/SOLR-502), we set the parameter "partialResults" at true.
> This patch also adresses a problem expressed in the mailing list about a year ago (http://www.nabble.com/partialResults,-distributed-search---SOLR-502-td19002610.html)
> We have a use case that needs this behaviour and we would like to know your thougths about such a behaviour? Should it be the default behaviour for distributed search?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/SOLR-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744327#action_12744327 ]

Artem Russakovskii commented on SOLR-1143:
------------------------------------------

Any idea when this will be approved for pushing into trunk?

> Return partial results when a connection to a shard is refused
> --------------------------------------------------------------
>
>                 Key: SOLR-1143
>                 URL: https://issues.apache.org/jira/browse/SOLR-1143
>             Project: Solr
>          Issue Type: Improvement
>          Components: search
>            Reporter: Nicolas Dessaigne
>             Fix For: 1.4
>
>         Attachments: SOLR-1143-2.patch, SOLR-1143.patch
>
>
> If any shard is down in a distributed search, a ConnectException it thrown.
> Here's a little patch that change this behaviour: if we can't connect to a shard (ConnectException), we get partial results from the active shards. As for TimeOut parameter (https://issues.apache.org/jira/browse/SOLR-502), we set the parameter "partialResults" at true.
> This patch also adresses a problem expressed in the mailing list about a year ago (http://www.nabble.com/partialResults,-distributed-search---SOLR-502-td19002610.html)
> We have a use case that needs this behaviour and we would like to know your thougths about such a behaviour? Should it be the default behaviour for distributed search?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/SOLR-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744518#action_12744518 ]

Grant Ingersoll commented on SOLR-1143:
---------------------------------------

I'm not sure about the need for the return-partial-results static boolean.  This could just be handled through the RequestHandler defaults, right?

> Return partial results when a connection to a shard is refused
> --------------------------------------------------------------
>
>                 Key: SOLR-1143
>                 URL: https://issues.apache.org/jira/browse/SOLR-1143
>             Project: Solr
>          Issue Type: Improvement
>          Components: search
>            Reporter: Nicolas Dessaigne
>             Fix For: 1.4
>
>         Attachments: SOLR-1143-2.patch, SOLR-1143.patch
>
>
> If any shard is down in a distributed search, a ConnectException it thrown.
> Here's a little patch that change this behaviour: if we can't connect to a shard (ConnectException), we get partial results from the active shards. As for TimeOut parameter (https://issues.apache.org/jira/browse/SOLR-502), we set the parameter "partialResults" at true.
> This patch also adresses a problem expressed in the mailing list about a year ago (http://www.nabble.com/partialResults,-distributed-search---SOLR-502-td19002610.html)
> We have a use case that needs this behaviour and we would like to know your thougths about such a behaviour? Should it be the default behaviour for distributed search?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/SOLR-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744518#action_12744518 ]

Grant Ingersoll edited comment on SOLR-1143 at 8/18/09 7:19 AM:
----------------------------------------------------------------

{quote}
In this patch you return a partial result when a shard has failed by setting partialResults to true in the request or if you want it to for all requests your can add <bool name="return-partial-results">true</bool> to your search handler in your solrconfig.xml. If both are not specified, partial results are disabled. Currently the partialResults parameter overrides the return-partial-results property in the search handler.
{quote}

I'm not sure about the need for the return-partial-results static boolean.  This could just be handled through the RequestHandler defaults, right?

      was (Author: gsingers):
    I'm not sure about the need for the return-partial-results static boolean.  This could just be handled through the RequestHandler defaults, right?
 

> Return partial results when a connection to a shard is refused
> --------------------------------------------------------------
>
>                 Key: SOLR-1143
>                 URL: https://issues.apache.org/jira/browse/SOLR-1143
>             Project: Solr
>          Issue Type: Improvement
>          Components: search
>            Reporter: Nicolas Dessaigne
>             Fix For: 1.4
>
>         Attachments: SOLR-1143-2.patch, SOLR-1143.patch
>
>
> If any shard is down in a distributed search, a ConnectException it thrown.
> Here's a little patch that change this behaviour: if we can't connect to a shard (ConnectException), we get partial results from the active shards. As for TimeOut parameter (https://issues.apache.org/jira/browse/SOLR-502), we set the parameter "partialResults" at true.
> This patch also adresses a problem expressed in the mailing list about a year ago (http://www.nabble.com/partialResults,-distributed-search---SOLR-502-td19002610.html)
> We have a use case that needs this behaviour and we would like to know your thougths about such a behaviour? Should it be the default behaviour for distributed search?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/SOLR-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Martijn van Groningen updated SOLR-1143:
----------------------------------------

    Attachment: SOLR-1143-3.patch

You are right Grant. I guess I forgot about the request handler defaults when I was creating this patch...
I have removed the possibility to configure partial results via the _return-partial-results_ property in the latest patch. The request handler defaults can perfectly configure partial results to be enabled by default.

> Return partial results when a connection to a shard is refused
> --------------------------------------------------------------
>
>                 Key: SOLR-1143
>                 URL: https://issues.apache.org/jira/browse/SOLR-1143
>             Project: Solr
>          Issue Type: Improvement
>          Components: search
>            Reporter: Nicolas Dessaigne
>             Fix For: 1.4
>
>         Attachments: SOLR-1143-2.patch, SOLR-1143-3.patch, SOLR-1143.patch
>
>
> If any shard is down in a distributed search, a ConnectException it thrown.
> Here's a little patch that change this behaviour: if we can't connect to a shard (ConnectException), we get partial results from the active shards. As for TimeOut parameter (https://issues.apache.org/jira/browse/SOLR-502), we set the parameter "partialResults" at true.
> This patch also adresses a problem expressed in the mailing list about a year ago (http://www.nabble.com/partialResults,-distributed-search---SOLR-502-td19002610.html)
> We have a use case that needs this behaviour and we would like to know your thougths about such a behaviour? Should it be the default behaviour for distributed search?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/SOLR-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747462#action_12747462 ]

Grant Ingersoll commented on SOLR-1143:
---------------------------------------

Small, FYI on patch submission:  No need to name them XXXX-1, XXXX-2, etc.  JIRA will actually version them automatically and gray out all but the most current one.  Doing so makes it easier to see what is the current patch w/o reading every one.

> Return partial results when a connection to a shard is refused
> --------------------------------------------------------------
>
>                 Key: SOLR-1143
>                 URL: https://issues.apache.org/jira/browse/SOLR-1143
>             Project: Solr
>          Issue Type: Improvement
>          Components: search
>            Reporter: Nicolas Dessaigne
>             Fix For: 1.4
>
>         Attachments: SOLR-1143-2.patch, SOLR-1143-3.patch, SOLR-1143.patch
>
>
> If any shard is down in a distributed search, a ConnectException it thrown.
> Here's a little patch that change this behaviour: if we can't connect to a shard (ConnectException), we get partial results from the active shards. As for TimeOut parameter (https://issues.apache.org/jira/browse/SOLR-502), we set the parameter "partialResults" at true.
> This patch also adresses a problem expressed in the mailing list about a year ago (http://www.nabble.com/partialResults,-distributed-search---SOLR-502-td19002610.html)
> We have a use case that needs this behaviour and we would like to know your thougths about such a behaviour? Should it be the default behaviour for distributed search?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/SOLR-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Grant Ingersoll reassigned SOLR-1143:
-------------------------------------

    Assignee: Grant Ingersoll

> Return partial results when a connection to a shard is refused
> --------------------------------------------------------------
>
>                 Key: SOLR-1143
>                 URL: https://issues.apache.org/jira/browse/SOLR-1143
>             Project: Solr
>          Issue Type: Improvement
>          Components: search
>            Reporter: Nicolas Dessaigne
>            Assignee: Grant Ingersoll
>             Fix For: 1.4
>
>         Attachments: SOLR-1143-2.patch, SOLR-1143-3.patch, SOLR-1143.patch
>
>
> If any shard is down in a distributed search, a ConnectException it thrown.
> Here's a little patch that change this behaviour: if we can't connect to a shard (ConnectException), we get partial results from the active shards. As for TimeOut parameter (https://issues.apache.org/jira/browse/SOLR-502), we set the parameter "partialResults" at true.
> This patch also adresses a problem expressed in the mailing list about a year ago (http://www.nabble.com/partialResults,-distributed-search---SOLR-502-td19002610.html)
> We have a use case that needs this behaviour and we would like to know your thougths about such a behaviour? Should it be the default behaviour for distributed search?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/SOLR-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747567#action_12747567 ]

Martijn van Groningen commented on SOLR-1143:
---------------------------------------------

Hi Grant, thanks for mentioning that, I did not realized that.

I have two other ideas about returning a partial result that might be usable in this patch:
1) Currently when a partial result is returned the response does not tell you which shard has failed, it only tells you that it is a partial result. Wouldn't it be handy to include hostnames or ip addresses in the response of the shards that had a connection timeout?

2) A partial result is only returned when when a connection exception occurs, is it practical to return a partial result when another type of exception occurs? Let say one shard has a corrupted index and therefore while searching only that shard throws an exception, I can imagine that in such situation it is also useful to return a partial result instead of only returning an error for the complete search. a

> Return partial results when a connection to a shard is refused
> --------------------------------------------------------------
>
>                 Key: SOLR-1143
>                 URL: https://issues.apache.org/jira/browse/SOLR-1143
>             Project: Solr
>          Issue Type: Improvement
>          Components: search
>            Reporter: Nicolas Dessaigne
>            Assignee: Grant Ingersoll
>             Fix For: 1.4
>
>         Attachments: SOLR-1143-2.patch, SOLR-1143-3.patch, SOLR-1143.patch
>
>
> If any shard is down in a distributed search, a ConnectException it thrown.
> Here's a little patch that change this behaviour: if we can't connect to a shard (ConnectException), we get partial results from the active shards. As for TimeOut parameter (https://issues.apache.org/jira/browse/SOLR-502), we set the parameter "partialResults" at true.
> This patch also adresses a problem expressed in the mailing list about a year ago (http://www.nabble.com/partialResults,-distributed-search---SOLR-502-td19002610.html)
> We have a use case that needs this behaviour and we would like to know your thougths about such a behaviour? Should it be the default behaviour for distributed search?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/SOLR-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747567#action_12747567 ]

Martijn van Groningen edited comment on SOLR-1143 at 8/25/09 11:55 AM:
-----------------------------------------------------------------------

Hi Grant, thanks for mentioning that, I did not realized that.

I have two other ideas about returning a partial result that might be usable in this patch:
1) Currently when a partial result is returned the response does not tell you which shard has failed, it only tells you that it is a partial result. Wouldn't it be handy to include hostnames or ip addresses in the response of the shards that had a connection timeout?

2) A partial result is only returned when when a connection exception occurs, is it practical to return a partial result when another type of exception occurs? Let say one shard has a corrupted index and therefore while searching only that shard throws an exception, I can imagine that in such situation it is also useful to return a partial result instead of only returning an error for the complete search.

      was (Author: martijn):
    Hi Grant, thanks for mentioning that, I did not realized that.

I have two other ideas about returning a partial result that might be usable in this patch:
1) Currently when a partial result is returned the response does not tell you which shard has failed, it only tells you that it is a partial result. Wouldn't it be handy to include hostnames or ip addresses in the response of the shards that had a connection timeout?

2) A partial result is only returned when when a connection exception occurs, is it practical to return a partial result when another type of exception occurs? Let say one shard has a corrupted index and therefore while searching only that shard throws an exception, I can imagine that in such situation it is also useful to return a partial result instead of only returning an error for the complete search. a
 

> Return partial results when a connection to a shard is refused
> --------------------------------------------------------------
>
>                 Key: SOLR-1143
>                 URL: https://issues.apache.org/jira/browse/SOLR-1143
>             Project: Solr
>          Issue Type: Improvement
>          Components: search
>            Reporter: Nicolas Dessaigne
>            Assignee: Grant Ingersoll
>             Fix For: 1.4
>
>         Attachments: SOLR-1143-2.patch, SOLR-1143-3.patch, SOLR-1143.patch
>
>
> If any shard is down in a distributed search, a ConnectException it thrown.
> Here's a little patch that change this behaviour: if we can't connect to a shard (ConnectException), we get partial results from the active shards. As for TimeOut parameter (https://issues.apache.org/jira/browse/SOLR-502), we set the parameter "partialResults" at true.
> This patch also adresses a problem expressed in the mailing list about a year ago (http://www.nabble.com/partialResults,-distributed-search---SOLR-502-td19002610.html)
> We have a use case that needs this behaviour and we would like to know your thougths about such a behaviour? Should it be the default behaviour for distributed search?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/SOLR-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747619#action_12747619 ]

Grant Ingersoll commented on SOLR-1143:
---------------------------------------

I'm not sure I'm following the changes in SearchHandler.

AIUI, before, we check for ShardResponses via comm.takeCompletedOrError() and then process the error and check for an exception.  If there is an exception, we throw it, essentially.

In the new code, it is replaced by just take() which returns the response, null or an exception.  We then iterate over the whole set of responses every time we enter the while (rb.outgoing...) loop.

However, why wouldn't you just keep the existing takeCompletedOrError, check to see if that shard is an error and handle it.  At the end of the loop it should be easy to determine if the number of requests sent equals the number received and then add the partial results indicator, and, potentially, indicate which shards failed.

What am I not understanding?  Basically, I don't get the need for:
{code}
for (ShardResponse shardRsp : srsp.getShardRequest().responses)
...
{code}

> Return partial results when a connection to a shard is refused
> --------------------------------------------------------------
>
>                 Key: SOLR-1143
>                 URL: https://issues.apache.org/jira/browse/SOLR-1143
>             Project: Solr
>          Issue Type: Improvement
>          Components: search
>            Reporter: Nicolas Dessaigne
>            Assignee: Grant Ingersoll
>             Fix For: 1.4
>
>         Attachments: SOLR-1143-2.patch, SOLR-1143-3.patch, SOLR-1143.patch
>
>
> If any shard is down in a distributed search, a ConnectException it thrown.
> Here's a little patch that change this behaviour: if we can't connect to a shard (ConnectException), we get partial results from the active shards. As for TimeOut parameter (https://issues.apache.org/jira/browse/SOLR-502), we set the parameter "partialResults" at true.
> This patch also adresses a problem expressed in the mailing list about a year ago (http://www.nabble.com/partialResults,-distributed-search---SOLR-502-td19002610.html)
> We have a use case that needs this behaviour and we would like to know your thougths about such a behaviour? Should it be the default behaviour for distributed search?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/SOLR-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747641#action_12747641 ]

Martijn van Groningen commented on SOLR-1143:
---------------------------------------------

Actually your approach makes more sense, because that is more efficient. But the takeCompletedOrError() method may then not directly return when a shard failure occurs, because then you might lose the response from the other shards. I initially tried to change take() to takeCompletedOrError(), but then I noticed this problem.

> Return partial results when a connection to a shard is refused
> --------------------------------------------------------------
>
>                 Key: SOLR-1143
>                 URL: https://issues.apache.org/jira/browse/SOLR-1143
>             Project: Solr
>          Issue Type: Improvement
>          Components: search
>            Reporter: Nicolas Dessaigne
>            Assignee: Grant Ingersoll
>             Fix For: 1.4
>
>         Attachments: SOLR-1143-2.patch, SOLR-1143-3.patch, SOLR-1143.patch
>
>
> If any shard is down in a distributed search, a ConnectException it thrown.
> Here's a little patch that change this behaviour: if we can't connect to a shard (ConnectException), we get partial results from the active shards. As for TimeOut parameter (https://issues.apache.org/jira/browse/SOLR-502), we set the parameter "partialResults" at true.
> This patch also adresses a problem expressed in the mailing list about a year ago (http://www.nabble.com/partialResults,-distributed-search---SOLR-502-td19002610.html)
> We have a use case that needs this behaviour and we would like to know your thougths about such a behaviour? Should it be the default behaviour for distributed search?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/SOLR-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747933#action_12747933 ]

Grant Ingersoll commented on SOLR-1143:
---------------------------------------

I don't follow.  The only difference between the two methods is that takeOrError returns immediately if there was an error and doesn't put it in the response list, which is what you are checking for in your loop anyway.  From what I can tell the while loop isn't going to break until all pending are accounted for, either by error or by valid results.  I don't see how it is beneficial to examine every shard response every time and I don't see why that would prevent you from losing responses as it is independent of the request sent.

> Return partial results when a connection to a shard is refused
> --------------------------------------------------------------
>
>                 Key: SOLR-1143
>                 URL: https://issues.apache.org/jira/browse/SOLR-1143
>             Project: Solr
>          Issue Type: Improvement
>          Components: search
>            Reporter: Nicolas Dessaigne
>            Assignee: Grant Ingersoll
>             Fix For: 1.4
>
>         Attachments: SOLR-1143-2.patch, SOLR-1143-3.patch, SOLR-1143.patch
>
>
> If any shard is down in a distributed search, a ConnectException it thrown.
> Here's a little patch that change this behaviour: if we can't connect to a shard (ConnectException), we get partial results from the active shards. As for TimeOut parameter (https://issues.apache.org/jira/browse/SOLR-502), we set the parameter "partialResults" at true.
> This patch also adresses a problem expressed in the mailing list about a year ago (http://www.nabble.com/partialResults,-distributed-search---SOLR-502-td19002610.html)
> We have a use case that needs this behaviour and we would like to know your thougths about such a behaviour? Should it be the default behaviour for distributed search?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/SOLR-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748128#action_12748128 ]

Martijn van Groningen commented on SOLR-1143:
---------------------------------------------

Sorry for my confusing comment. I meant to say takeOrError() does return immediately when an exception occurs. To avoid more confusion I will sketch a situation from what I currently understand from the code to show that takeOrError() should not be used when returning partial result.

For each stage a number of requests may be send to the shards and a number of responses may be returned from the shards for further processing.
Lets say we have three shards and we send a shard request in a certain stage to all three shards. If the first response contains an error the current behaviour is to return the response immediately, without adding the two other responses (that did return without an error). Because of this the so called partial result might contain less data or even nothing.  Therefore I think take() should be used there. I think takeOrError() is only suitable when not using partial result.

{code:java}
ShardResponse takeCompletedOrError() {
    while (pending.size() > 0) {
      try {
        Future<ShardResponse> future = completionService.take();
        pending.remove(future);
        ShardResponse rsp = future.get();
        if (rsp.getException() != null) return rsp; // now we return and if there are more pending results, we lose them
        ...............
        rsp.getShardRequest().responses.add(rsp);
        if (rsp.getShardRequest().responses.size() == rsp.getShardRequest().actualShards.length) {
          return rsp;
        }
      } catch (InterruptedException e) {
      ......
    }
    return null;
  }
{code}

Again this what I understand from the code. What do you think about this?

I also did some more thinking about how to improve shard failures. Currently if a shard fails in a early stage of the distributed search we keep sending requests to the shard, although we noticed in a previous stage that it was not responding. You think that it is a good idea to mark a shard as failed, so that it will not use the shard that is marked as failed for the current running search?

> Return partial results when a connection to a shard is refused
> --------------------------------------------------------------
>
>                 Key: SOLR-1143
>                 URL: https://issues.apache.org/jira/browse/SOLR-1143
>             Project: Solr
>          Issue Type: Improvement
>          Components: search
>            Reporter: Nicolas Dessaigne
>            Assignee: Grant Ingersoll
>             Fix For: 1.4
>
>         Attachments: SOLR-1143-2.patch, SOLR-1143-3.patch, SOLR-1143.patch
>
>
> If any shard is down in a distributed search, a ConnectException it thrown.
> Here's a little patch that change this behaviour: if we can't connect to a shard (ConnectException), we get partial results from the active shards. As for TimeOut parameter (https://issues.apache.org/jira/browse/SOLR-502), we set the parameter "partialResults" at true.
> This patch also adresses a problem expressed in the mailing list about a year ago (http://www.nabble.com/partialResults,-distributed-search---SOLR-502-td19002610.html)
> We have a use case that needs this behaviour and we would like to know your thougths about such a behaviour? Should it be the default behaviour for distributed search?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.