[jira] Created: (SOLR-1543) StreamingUpdateSolrServer threading bug

StreamingUpdateSolrServer threading bug
---------------------------------------

                 Key: SOLR-1543
                 URL: https://issues.apache.org/jira/browse/SOLR-1543
             Project: Solr
          Issue Type: Bug
    Affects Versions: 1.4
            Reporter: Yonik Seeley


StreamingUpdateSolrServer - indexing process stops in a couple of hours
http://search.lucidimagination.com/search/document/352d3c8cd3bbc8cb

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/SOLR-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774141#action_12774141 ]

Yonik Seeley commented on SOLR-1543:
------------------------------------

I don't totally understand this code (and it doesn't have enough comments as to how some of the threading/locking works), but I do notice that blockUntilFinished doesn't unlock the lock in a finally block.

> StreamingUpdateSolrServer threading bug
> ---------------------------------------
>
>                 Key: SOLR-1543
>                 URL: https://issues.apache.org/jira/browse/SOLR-1543
>             Project: Solr
>          Issue Type: Bug
>    Affects Versions: 1.4
>            Reporter: Yonik Seeley
>
> StreamingUpdateSolrServer - indexing process stops in a couple of hours
> http://search.lucidimagination.com/search/document/352d3c8cd3bbc8cb

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/SOLR-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yonik Seeley updated SOLR-1543:
-------------------------------

    Attachment: SOLR-1543.patch

This patch most likely does not fix the problem (I don't even know how to reproduce the problem at this point).  But it does:
 - unlock the lock in a finally block
 - fix thread visibility of the lock by making it volatile
 - change the lock in each runner to be a countdown latch since the only purpose appears to be to wait for the runner to stop.  This fixes a race condition where previously the lock/unlock pair in blockUntilFinished() could possibly execute *before* the runner itself locks the lock.

There are other race conditions and thread safety problems I see, but I'm not sure if any are serious or not.

> StreamingUpdateSolrServer threading bug
> ---------------------------------------
>
>                 Key: SOLR-1543
>                 URL: https://issues.apache.org/jira/browse/SOLR-1543
>             Project: Solr
>          Issue Type: Bug
>    Affects Versions: 1.4
>            Reporter: Yonik Seeley
>         Attachments: SOLR-1543.patch
>
>
> StreamingUpdateSolrServer - indexing process stops in a couple of hours
> http://search.lucidimagination.com/search/document/352d3c8cd3bbc8cb

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/SOLR-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774172#action_12774172 ]

Yonik Seeley commented on SOLR-1543:
------------------------------------

OK, I'm able to reproduce the problem with the test code in the referenced thread... looking into why.

> StreamingUpdateSolrServer threading bug
> ---------------------------------------
>
>                 Key: SOLR-1543
>                 URL: https://issues.apache.org/jira/browse/SOLR-1543
>             Project: Solr
>          Issue Type: Bug
>    Affects Versions: 1.4
>            Reporter: Yonik Seeley
>         Attachments: SOLR-1543.patch
>
>
> StreamingUpdateSolrServer - indexing process stops in a couple of hours
> http://search.lucidimagination.com/search/document/352d3c8cd3bbc8cb

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/SOLR-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774179#action_12774179 ]

Yonik Seeley commented on SOLR-1543:
------------------------------------

Found it.

All of the threads check "lock" and block on it.
When the lock unlocks, one thread locks it again, gets through, but never unlocks it because the lock instance var as been nulled.  The solution is to copy the lock var into a local variable and use that for all lock operations.

> StreamingUpdateSolrServer threading bug
> ---------------------------------------
>
>                 Key: SOLR-1543
>                 URL: https://issues.apache.org/jira/browse/SOLR-1543
>             Project: Solr
>          Issue Type: Bug
>    Affects Versions: 1.4
>            Reporter: Yonik Seeley
>         Attachments: SOLR-1543.patch
>
>
> StreamingUpdateSolrServer - indexing process stops in a couple of hours
> http://search.lucidimagination.com/search/document/352d3c8cd3bbc8cb

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/SOLR-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yonik Seeley updated SOLR-1543:
-------------------------------

    Attachment: SOLR-1543.patch

OK, here's the patch, with the lock converted to a latch so all blocked threads can be released at once.

> StreamingUpdateSolrServer threading bug
> ---------------------------------------
>
>                 Key: SOLR-1543
>                 URL: https://issues.apache.org/jira/browse/SOLR-1543
>             Project: Solr
>          Issue Type: Bug
>    Affects Versions: 1.4
>            Reporter: Yonik Seeley
>         Attachments: SOLR-1543.patch, SOLR-1543.patch
>
>
> StreamingUpdateSolrServer - indexing process stops in a couple of hours
> http://search.lucidimagination.com/search/document/352d3c8cd3bbc8cb

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/SOLR-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yonik Seeley resolved SOLR-1543.
--------------------------------

       Resolution: Fixed
    Fix Version/s: 1.4

committed on trunk and 1.4 branch.

> StreamingUpdateSolrServer threading bug
> ---------------------------------------
>
>                 Key: SOLR-1543
>                 URL: https://issues.apache.org/jira/browse/SOLR-1543
>             Project: Solr
>          Issue Type: Bug
>    Affects Versions: 1.4
>            Reporter: Yonik Seeley
>             Fix For: 1.4
>
>         Attachments: SOLR-1543.patch, SOLR-1543.patch
>
>
> StreamingUpdateSolrServer - indexing process stops in a couple of hours
> http://search.lucidimagination.com/search/document/352d3c8cd3bbc8cb

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.