[jira] Created: (HADOOP-1188) processIOError() should update fstime file

processIOError() should update fstime file
------------------------------------------

                 Key: HADOOP-1188
                 URL: https://issues.apache.org/jira/browse/HADOOP-1188
             Project: Hadoop
          Issue Type: Bug
          Components: dfs
    Affects Versions: 0.12.2, 0.11.2, 0.10.1
            Reporter: Konstantin Shvachko
             Fix For: 0.13.0


Name-node can have multiple directories to store the name space image and edits.
During startup the name-node selects the latest image and reads it in memory.
fstime stores the time of the latest checkpoint.
If one of the directories becomes inaccessible during normal operation the name-node
excludes it from the list, and never writes anything into it until restarted.
Now if the the name-node restarts after excluding one of the directories, and if that
excluded directory contained the latest image, then all modifications of the name space
recorded in edits files in other directories starting from the moment the directory was
excluded will be lost.
We should update fstime in all remaining good directories after the exclusion.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12486055 ]

dhruba borthakur commented on HADOOP-1188:
------------------------------------------

+1. Sounds great.

> processIOError() should update fstime file
> ------------------------------------------
>
>                 Key: HADOOP-1188
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1188
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.10.1, 0.11.2, 0.12.2
>            Reporter: Konstantin Shvachko
>             Fix For: 0.13.0
>
>
> Name-node can have multiple directories to store the name space image and edits.
> During startup the name-node selects the latest image and reads it in memory.
> fstime stores the time of the latest checkpoint.
> If one of the directories becomes inaccessible during normal operation the name-node
> excludes it from the list, and never writes anything into it until restarted.
> Now if the the name-node restarts after excluding one of the directories, and if that
> excluded directory contained the latest image, then all modifications of the name space
> recorded in edits files in other directories starting from the moment the directory was
> excluded will be lost.
> We should update fstime in all remaining good directories after the exclusion.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur updated HADOOP-1188:
-------------------------------------

    Fix Version/s: 0.15.0

> processIOError() should update fstime file
> ------------------------------------------
>
>                 Key: HADOOP-1188
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1188
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.10.1, 0.11.2, 0.12.2
>            Reporter: Konstantin Shvachko
>             Fix For: 0.15.0
>
>
> Name-node can have multiple directories to store the name space image and edits.
> During startup the name-node selects the latest image and reads it in memory.
> fstime stores the time of the latest checkpoint.
> If one of the directories becomes inaccessible during normal operation the name-node
> excludes it from the list, and never writes anything into it until restarted.
> Now if the the name-node restarts after excluding one of the directories, and if that
> excluded directory contained the latest image, then all modifications of the name space
> recorded in edits files in other directories starting from the moment the directory was
> excluded will be lost.
> We should update fstime in all remaining good directories after the exclusion.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553528 ]

dhruba borthakur commented on HADOOP-1188:
------------------------------------------

It is possible that updating the fstime on the good directories could fail too. So, a namenode restart should pick all those directories that have the latest (and same) fstime and fail to restart if the sizes of the edit files in those directories are not exactly the same.

> processIOError() should update fstime file
> ------------------------------------------
>
>                 Key: HADOOP-1188
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1188
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.10.1, 0.11.2, 0.12.2
>            Reporter: Konstantin Shvachko
>
> Name-node can have multiple directories to store the name space image and edits.
> During startup the name-node selects the latest image and reads it in memory.
> fstime stores the time of the latest checkpoint.
> If one of the directories becomes inaccessible during normal operation the name-node
> excludes it from the list, and never writes anything into it until restarted.
> Now if the the name-node restarts after excluding one of the directories, and if that
> excluded directory contained the latest image, then all modifications of the name space
> recorded in edits files in other directories starting from the moment the directory was
> excluded will be lost.
> We should update fstime in all remaining good directories after the exclusion.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Chansler updated HADOOP-1188:
------------------------------------

    Priority: Blocker  (was: Major)

Promoted for consideration in 16.

> processIOError() should update fstime file
> ------------------------------------------
>
>                 Key: HADOOP-1188
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1188
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.10.1, 0.11.2, 0.12.2
>            Reporter: Konstantin Shvachko
>            Priority: Blocker
>
> Name-node can have multiple directories to store the name space image and edits.
> During startup the name-node selects the latest image and reads it in memory.
> fstime stores the time of the latest checkpoint.
> If one of the directories becomes inaccessible during normal operation the name-node
> excludes it from the list, and never writes anything into it until restarted.
> Now if the the name-node restarts after excluding one of the directories, and if that
> excluded directory contained the latest image, then all modifications of the name space
> recorded in edits files in other directories starting from the moment the directory was
> excluded will be lost.
> We should update fstime in all remaining good directories after the exclusion.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur updated HADOOP-1188:
-------------------------------------

    Fix Version/s: 0.16.0

> processIOError() should update fstime file
> ------------------------------------------
>
>                 Key: HADOOP-1188
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1188
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.10.1, 0.11.2, 0.12.2
>            Reporter: Konstantin Shvachko
>            Priority: Blocker
>             Fix For: 0.16.0
>
>
> Name-node can have multiple directories to store the name space image and edits.
> During startup the name-node selects the latest image and reads it in memory.
> fstime stores the time of the latest checkpoint.
> If one of the directories becomes inaccessible during normal operation the name-node
> excludes it from the list, and never writes anything into it until restarted.
> Now if the the name-node restarts after excluding one of the directories, and if that
> excluded directory contained the latest image, then all modifications of the name space
> recorded in edits files in other directories starting from the moment the directory was
> excluded will be lost.
> We should update fstime in all remaining good directories after the exclusion.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur reassigned HADOOP-1188:
----------------------------------------

    Assignee: Konstantin Shvachko

> processIOError() should update fstime file
> ------------------------------------------
>
>                 Key: HADOOP-1188
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1188
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.10.1, 0.11.2, 0.12.2
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>            Priority: Blocker
>             Fix For: 0.16.0
>
>
> Name-node can have multiple directories to store the name space image and edits.
> During startup the name-node selects the latest image and reads it in memory.
> fstime stores the time of the latest checkpoint.
> If one of the directories becomes inaccessible during normal operation the name-node
> excludes it from the list, and never writes anything into it until restarted.
> Now if the the name-node restarts after excluding one of the directories, and if that
> excluded directory contained the latest image, then all modifications of the name space
> recorded in edits files in other directories starting from the moment the directory was
> excluded will be lost.
> We should update fstime in all remaining good directories after the exclusion.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nigel Daley updated HADOOP-1188:
--------------------------------

    Fix Version/s:     (was: 0.16.0)
                   0.16.1

> processIOError() should update fstime file
> ------------------------------------------
>
>                 Key: HADOOP-1188
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1188
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.10.1, 0.11.2, 0.12.2
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>            Priority: Blocker
>             Fix For: 0.16.1
>
>
> Name-node can have multiple directories to store the name space image and edits.
> During startup the name-node selects the latest image and reads it in memory.
> fstime stores the time of the latest checkpoint.
> If one of the directories becomes inaccessible during normal operation the name-node
> excludes it from the list, and never writes anything into it until restarted.
> Now if the the name-node restarts after excluding one of the directories, and if that
> excluded directory contained the latest image, then all modifications of the name space
> recorded in edits files in other directories starting from the moment the directory was
> excluded will be lost.
> We should update fstime in all remaining good directories after the exclusion.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.