[jira] Created: (HADOOP-2649) The ReplicationMonitor sleep period should be configurable

The ReplicationMonitor sleep period should be configurable
----------------------------------------------------------

                 Key: HADOOP-2649
                 URL: https://issues.apache.org/jira/browse/HADOOP-2649
             Project: Hadoop
          Issue Type: Bug
          Components: dfs
            Reporter: dhruba borthakur
             Fix For: 0.16.0


The HDFS Namenode computes replication work for datanodes once every 3 seconds. This should be a configurable  value. On large clusters, there could be many many blocks in the neededReplication queue and computing replication work for datanodes once every 3 seconds might consume lots of CPU on namenode.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur updated HADOOP-2649:
-------------------------------------

    Attachment: configurableReplicationPeriod.patch

Make the Replication Monitor periodicity configurable. The default value of 3 seconds remains the same as before.

> The ReplicationMonitor sleep period should be configurable
> ----------------------------------------------------------
>
>                 Key: HADOOP-2649
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2649
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: dhruba borthakur
>             Fix For: 0.16.0
>
>         Attachments: configurableReplicationPeriod.patch
>
>
> The HDFS Namenode computes replication work for datanodes once every 3 seconds. This should be a configurable  value. On large clusters, there could be many many blocks in the neededReplication queue and computing replication work for datanodes once every 3 seconds might consume lots of CPU on namenode.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur reassigned HADOOP-2649:
----------------------------------------

    Assignee: dhruba borthakur

> The ReplicationMonitor sleep period should be configurable
> ----------------------------------------------------------
>
>                 Key: HADOOP-2649
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2649
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>             Fix For: 0.16.0
>
>         Attachments: configurableReplicationPeriod.patch
>
>
> The HDFS Namenode computes replication work for datanodes once every 3 seconds. This should be a configurable  value. On large clusters, there could be many many blocks in the neededReplication queue and computing replication work for datanodes once every 3 seconds might consume lots of CPU on namenode.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12560569#action_12560569 ]

Hairong Kuang commented on HADOOP-2649:
---------------------------------------

+1 The patch looks good to me.

It would be nicer if the replication monitor can adjust its frequency dynamically. It slows itself down when the namenode becomes busy and replicates more aggressively when the namenode becomes idle.

> The ReplicationMonitor sleep period should be configurable
> ----------------------------------------------------------
>
>                 Key: HADOOP-2649
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2649
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>             Fix For: 0.16.0
>
>         Attachments: configurableReplicationPeriod.patch
>
>
> The HDFS Namenode computes replication work for datanodes once every 3 seconds. This should be a configurable  value. On large clusters, there could be many many blocks in the neededReplication queue and computing replication work for datanodes once every 3 seconds might consume lots of CPU on namenode.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur updated HADOOP-2649:
-------------------------------------

    Status: Patch Available  (was: Open)

> The ReplicationMonitor sleep period should be configurable
> ----------------------------------------------------------
>
>                 Key: HADOOP-2649
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2649
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>             Fix For: 0.16.0
>
>         Attachments: configurableReplicationPeriod.patch
>
>
> The HDFS Namenode computes replication work for datanodes once every 3 seconds. This should be a configurable  value. On large clusters, there could be many many blocks in the neededReplication queue and computing replication work for datanodes once every 3 seconds might consume lots of CPU on namenode.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12560570#action_12560570 ]

dhruba borthakur commented on HADOOP-2649:
------------------------------------------

Hi hairong, thanks for ur review comments. I plan to make this period dymanic (as per your suggestion) in the jira that is linked to this issue HADOOP-2606.

> The ReplicationMonitor sleep period should be configurable
> ----------------------------------------------------------
>
>                 Key: HADOOP-2649
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2649
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>             Fix For: 0.16.0
>
>         Attachments: configurableReplicationPeriod.patch
>
>
> The HDFS Namenode computes replication work for datanodes once every 3 seconds. This should be a configurable  value. On large clusters, there could be many many blocks in the neededReplication queue and computing replication work for datanodes once every 3 seconds might consume lots of CPU on namenode.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12560757#action_12560757 ]

Hadoop QA commented on HADOOP-2649:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12373461/configurableReplicationPeriod.patch
against trunk revision r613446.

    @author +1.  The patch does not contain any @author tags.

    javadoc +1.  The javadoc tool did not generate any warning messages.

    javac +1.  The applied patch does not generate any new compiler warnings.

    findbugs +1.  The patch does not introduce any new Findbugs warnings.

    core tests -1.  The patch failed core unit tests.

    contrib tests +1.  The patch passed contrib unit tests.

Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1657/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1657/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1657/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1657/console

This message is automatically generated.

> The ReplicationMonitor sleep period should be configurable
> ----------------------------------------------------------
>
>                 Key: HADOOP-2649
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2649
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>             Fix For: 0.16.0
>
>         Attachments: configurableReplicationPeriod.patch
>
>
> The HDFS Namenode computes replication work for datanodes once every 3 seconds. This should be a configurable  value. On large clusters, there could be many many blocks in the neededReplication queue and computing replication work for datanodes once every 3 seconds might consume lots of CPU on namenode.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur updated HADOOP-2649:
-------------------------------------

    Status: Open  (was: Patch Available)

> The ReplicationMonitor sleep period should be configurable
> ----------------------------------------------------------
>
>                 Key: HADOOP-2649
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2649
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>             Fix For: 0.16.0
>
>         Attachments: configurableReplicationPeriod.patch
>
>
> The HDFS Namenode computes replication work for datanodes once every 3 seconds. This should be a configurable  value. On large clusters, there could be many many blocks in the neededReplication queue and computing replication work for datanodes once every 3 seconds might consume lots of CPU on namenode.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur updated HADOOP-2649:
-------------------------------------

    Attachment: configurableReplicationPeriod.patch

Merged patch with latest trunk. Also, fixed the problem with Test Decommission failing because it was not progressing quickly.

> The ReplicationMonitor sleep period should be configurable
> ----------------------------------------------------------
>
>                 Key: HADOOP-2649
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2649
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>             Fix For: 0.16.0
>
>         Attachments: configurableReplicationPeriod.patch, configurableReplicationPeriod.patch
>
>
> The HDFS Namenode computes replication work for datanodes once every 3 seconds. This should be a configurable  value. On large clusters, there could be many many blocks in the neededReplication queue and computing replication work for datanodes once every 3 seconds might consume lots of CPU on namenode.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur updated HADOOP-2649:
-------------------------------------

    Status: Patch Available  (was: Open)

> The ReplicationMonitor sleep period should be configurable
> ----------------------------------------------------------
>
>                 Key: HADOOP-2649
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2649
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>             Fix For: 0.16.0
>
>         Attachments: configurableReplicationPeriod.patch, configurableReplicationPeriod.patch
>
>
> The HDFS Namenode computes replication work for datanodes once every 3 seconds. This should be a configurable  value. On large clusters, there could be many many blocks in the neededReplication queue and computing replication work for datanodes once every 3 seconds might consume lots of CPU on namenode.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Chansler updated HADOOP-2649:
------------------------------------

    Priority: Blocker  (was: Major)

Promoted for consideration in 16.

> The ReplicationMonitor sleep period should be configurable
> ----------------------------------------------------------
>
>                 Key: HADOOP-2649
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2649
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: configurableReplicationPeriod.patch, configurableReplicationPeriod.patch
>
>
> The HDFS Namenode computes replication work for datanodes once every 3 seconds. This should be a configurable  value. On large clusters, there could be many many blocks in the neededReplication queue and computing replication work for datanodes once every 3 seconds might consume lots of CPU on namenode.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12561429#action_12561429 ]

Hadoop QA commented on HADOOP-2649:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12373724/configurableReplicationPeriod.patch
against trunk revision r614192.

    @author +1.  The patch does not contain any @author tags.

    javadoc +1.  The javadoc tool did not generate any warning messages.

    javac +1.  The applied patch does not generate any new compiler warnings.

    findbugs +1.  The patch does not introduce any new Findbugs warnings.

    core tests +1.  The patch passed core unit tests.

    contrib tests +1.  The patch passed contrib unit tests.

Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1675/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1675/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1675/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1675/console

This message is automatically generated.

> The ReplicationMonitor sleep period should be configurable
> ----------------------------------------------------------
>
>                 Key: HADOOP-2649
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2649
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: configurableReplicationPeriod.patch, configurableReplicationPeriod.patch
>
>
> The HDFS Namenode computes replication work for datanodes once every 3 seconds. This should be a configurable  value. On large clusters, there could be many many blocks in the neededReplication queue and computing replication work for datanodes once every 3 seconds might consume lots of CPU on namenode.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur updated HADOOP-2649:
-------------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

I just committed this.

> The ReplicationMonitor sleep period should be configurable
> ----------------------------------------------------------
>
>                 Key: HADOOP-2649
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2649
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: configurableReplicationPeriod.patch, configurableReplicationPeriod.patch
>
>
> The HDFS Namenode computes replication work for datanodes once every 3 seconds. This should be a configurable  value. On large clusters, there could be many many blocks in the neededReplication queue and computing replication work for datanodes once every 3 seconds might consume lots of CPU on namenode.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12561709#action_12561709 ]

Hudson commented on HADOOP-2649:
--------------------------------

Integrated in Hadoop-Nightly #374 (See [http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/374/])

> The ReplicationMonitor sleep period should be configurable
> ----------------------------------------------------------
>
>                 Key: HADOOP-2649
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2649
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: configurableReplicationPeriod.patch, configurableReplicationPeriod.patch
>
>
> The HDFS Namenode computes replication work for datanodes once every 3 seconds. This should be a configurable  value. On large clusters, there could be many many blocks in the neededReplication queue and computing replication work for datanodes once every 3 seconds might consume lots of CPU on namenode.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.