[jira] Created: (HADOOP-2652) Fix permission issues for HftpFileSystem

Fix permission issues for HftpFileSystem
----------------------------------------

                 Key: HADOOP-2652
                 URL: https://issues.apache.org/jira/browse/HADOOP-2652
             Project: Hadoop
          Issue Type: Bug
          Components: dfs
            Reporter: Tsz Wo (Nicholas), SZE
            Assignee: Tsz Wo (Nicholas), SZE


Similar to HADOOP-2614, HftpFileSystem requires UserGroupInformation to access NameNode.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-2652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-2652:
-------------------------------------------

    Attachment: 2652_20080117.patch

> Fix permission issues for HftpFileSystem
> ----------------------------------------
>
>                 Key: HADOOP-2652
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2652
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: 2652_20080117.patch
>
>
> Similar to HADOOP-2614, HftpFileSystem requires UserGroupInformation to access NameNode.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-2652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-2652:
-------------------------------------------

    Attachment: 2652_20080117b.patch

> Fix permission issues for HftpFileSystem
> ----------------------------------------
>
>                 Key: HADOOP-2652
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2652
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: 2652_20080117.patch, 2652_20080117b.patch
>
>
> Similar to HADOOP-2614, HftpFileSystem requires UserGroupInformation to access NameNode.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-2652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12560566#action_12560566 ]

Hairong Kuang commented on HADOOP-2652:
---------------------------------------

The patch looks good to me. It would be nicer if the replication monitor can adjust its frequency dynamically. It slows itself down when the namenode becomes busy and replicates more aggressively when the namenode becomes idle.

> Fix permission issues for HftpFileSystem
> ----------------------------------------
>
>                 Key: HADOOP-2652
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2652
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: 2652_20080117.patch, 2652_20080117b.patch
>
>
> Similar to HADOOP-2614, HftpFileSystem requires UserGroupInformation to access NameNode.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-2652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-2652:
----------------------------------

    Comment: was deleted

> Fix permission issues for HftpFileSystem
> ----------------------------------------
>
>                 Key: HADOOP-2652
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2652
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: 2652_20080117.patch, 2652_20080117b.patch
>
>
> Similar to HADOOP-2614, HftpFileSystem requires UserGroupInformation to access NameNode.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-2652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-2652:
-------------------------------------------

    Priority: Blocker  (was: Major)

> Fix permission issues for HftpFileSystem
> ----------------------------------------
>
>                 Key: HADOOP-2652
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2652
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>            Priority: Blocker
>         Attachments: 2652_20080117.patch, 2652_20080117b.patch
>
>
> Similar to HADOOP-2614, HftpFileSystem requires UserGroupInformation to access NameNode.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-2652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12561445#action_12561445 ]

Hairong Kuang commented on HADOOP-2652:
---------------------------------------

1. HftpFileSystem.initialize should throw a LoginException when login failed. Otherwise, A NullPointerException might be thrown when a URL is constructed.
2. ListPathsServlet and FileDataServlet talks to namenode through RPC not by direct method call. Shall we concern about the performance?
3. ListPathsServlet.writeInfo should also write a path's permission-related info back to the client.
4, For secondary namenode communication, I think it is better to check the user in getFileServer and getInfoServer. This means that these two methods can not be static.
5. Can we also enforce user checking for NamenodeProtocol?
6. The patch passes the user info to StreamFile servlets but the user info is not used. Since currently dfs does not check user authorization for client to datanode communications, I feel it is better to do it later on.

> Fix permission issues for HftpFileSystem
> ----------------------------------------
>
>                 Key: HADOOP-2652
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2652
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>            Priority: Blocker
>         Attachments: 2652_20080117.patch, 2652_20080117b.patch
>
>
> Similar to HADOOP-2614, HftpFileSystem requires UserGroupInformation to access NameNode.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-2652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12561511#action_12561511 ]

Tsz Wo (Nicholas), SZE commented on HADOOP-2652:
------------------------------------------------

Regarding to Hairong's comment:

1. Sure, I will make it throwing an IOException warpping a LoginException.  Then, we don't have to change the FileSystem API.

2. Comparing to file transfer time, RPC overhead is negligible.  So it is fine to do RPC calls.

Right now, we need RPC calls for passing ugi to name node.  Once we have changed the way for storing ugi (by thread local variable or by Subject) in name node late on, we could get rid of RPC.

3. They were already done by some previous patches.

4 & 5. Actually, I should not change the secondary namenode in this issue since it is not related.  I will create another issue for that and NamenodeProtocol.

6. StreamFile servlet first stores ugi to conf and then use it to create a DFSClient.  So the ugi is used indeed.

> Fix permission issues for HftpFileSystem
> ----------------------------------------
>
>                 Key: HADOOP-2652
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2652
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>            Priority: Blocker
>         Attachments: 2652_20080117.patch, 2652_20080117b.patch
>
>
> Similar to HADOOP-2614, HftpFileSystem requires UserGroupInformation to access NameNode.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-2652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nigel Daley updated HADOOP-2652:
--------------------------------

    Fix Version/s: 0.16.0

> Fix permission issues for HftpFileSystem
> ----------------------------------------
>
>                 Key: HADOOP-2652
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2652
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: 2652_20080117.patch, 2652_20080117b.patch
>
>
> Similar to HADOOP-2614, HftpFileSystem requires UserGroupInformation to access NameNode.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-2652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-2652:
-------------------------------------------

    Attachment: 2652_20080122.patch

> Fix permission issues for HftpFileSystem
> ----------------------------------------
>
>                 Key: HADOOP-2652
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2652
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: 2652_20080117.patch, 2652_20080117b.patch, 2652_20080122.patch
>
>
> Similar to HADOOP-2614, HftpFileSystem requires UserGroupInformation to access NameNode.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-2652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12561771#action_12561771 ]

Hairong Kuang commented on HADOOP-2652:
---------------------------------------

+1 The patch looks good.

> Fix permission issues for HftpFileSystem
> ----------------------------------------
>
>                 Key: HADOOP-2652
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2652
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: 2652_20080117.patch, 2652_20080117b.patch, 2652_20080122.patch
>
>
> Similar to HADOOP-2614, HftpFileSystem requires UserGroupInformation to access NameNode.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-2652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-2652:
-------------------------------------------

    Status: Patch Available  (was: Open)

> Fix permission issues for HftpFileSystem
> ----------------------------------------
>
>                 Key: HADOOP-2652
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2652
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: 2652_20080117.patch, 2652_20080117b.patch, 2652_20080122.patch
>
>
> Similar to HADOOP-2614, HftpFileSystem requires UserGroupInformation to access NameNode.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.