[jira] Created: (HADOOP-2626) RawLocalFileStatus is badly handling URIs

RawLocalFileStatus is badly handling URIs
-----------------------------------------

                 Key: HADOOP-2626
                 URL: https://issues.apache.org/jira/browse/HADOOP-2626
             Project: Hadoop
          Issue Type: Bug
          Components: fs
    Affects Versions: 0.15.2
            Reporter: Frédéric Bertin


as a result, files with special characters (that get encoded when translated to URIs) are badly handled using a local filesystem.

{{new Path(f.toURI().toString()))}} should be replaced by {{new Path(f.toURI().getPath()))}}

IMHO, each call to {{toURI().toString()}} should be considered suspicious. There's another one in the class CopyFiles at line 641.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-2626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Friol updated HADOOP-2626:
---------------------------------

    Attachment: patch-Hadoop-2626.diff

Here is a patch.

> RawLocalFileStatus is badly handling URIs
> -----------------------------------------
>
>                 Key: HADOOP-2626
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2626
>             Project: Hadoop
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.15.2
>            Reporter: Frédéric Bertin
>         Attachments: patch-Hadoop-2626.diff
>
>
> as a result, files with special characters (that get encoded when translated to URIs) are badly handled using a local filesystem.
> {{new Path(f.toURI().toString()))}} should be replaced by {{new Path(f.toURI().getPath()))}}
> IMHO, each call to {{toURI().toString()}} should be considered suspicious. There's another one in the class CopyFiles at line 641.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-2626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Friol updated HADOOP-2626:
---------------------------------

    Status: Patch Available  (was: Open)

> RawLocalFileStatus is badly handling URIs
> -----------------------------------------
>
>                 Key: HADOOP-2626
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2626
>             Project: Hadoop
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.15.2
>            Reporter: Frédéric Bertin
>         Attachments: patch-Hadoop-2626.diff
>
>
> as a result, files with special characters (that get encoded when translated to URIs) are badly handled using a local filesystem.
> {{new Path(f.toURI().toString()))}} should be replaced by {{new Path(f.toURI().getPath()))}}
> IMHO, each call to {{toURI().toString()}} should be considered suspicious. There's another one in the class CopyFiles at line 641.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-2626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12559731#action_12559731 ]

Hadoop QA commented on HADOOP-2626:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12373296/patch-Hadoop-2626.diff
against trunk revision r612614.

    @author +1.  The patch does not contain any @author tags.

    patch -1.  The patch command could not apply the patch.

Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1617/console

This message is automatically generated.

> RawLocalFileStatus is badly handling URIs
> -----------------------------------------
>
>                 Key: HADOOP-2626
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2626
>             Project: Hadoop
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.15.2
>            Reporter: Frédéric Bertin
>         Attachments: patch-Hadoop-2626.diff
>
>
> as a result, files with special characters (that get encoded when translated to URIs) are badly handled using a local filesystem.
> {{new Path(f.toURI().toString()))}} should be replaced by {{new Path(f.toURI().getPath()))}}
> IMHO, each call to {{toURI().toString()}} should be considered suspicious. There's another one in the class CopyFiles at line 641.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-2626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-2626:
---------------------------------

    Status: Open  (was: Patch Available)

This patch puts an unqualified path in the returned FileStatus.  That's not strictly a bug, but we've found that it's always safest to return fully-qualified paths whenever we can.

To convert the java.io.File to a Path, we might use new Path(file.getPath()).makeQualified(fs).  Perhaps this should be added as a fileToPath method, since there's already a path2File() method.

> RawLocalFileStatus is badly handling URIs
> -----------------------------------------
>
>                 Key: HADOOP-2626
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2626
>             Project: Hadoop
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.15.2
>            Reporter: Frédéric Bertin
>         Attachments: patch-Hadoop-2626.diff
>
>
> as a result, files with special characters (that get encoded when translated to URIs) are badly handled using a local filesystem.
> {{new Path(f.toURI().toString()))}} should be replaced by {{new Path(f.toURI().getPath()))}}
> IMHO, each call to {{toURI().toString()}} should be considered suspicious. There's another one in the class CopyFiles at line 641.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-2626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Friol updated HADOOP-2626:
---------------------------------

    Attachment:     (was: patch-Hadoop-2626.diff)

> RawLocalFileStatus is badly handling URIs
> -----------------------------------------
>
>                 Key: HADOOP-2626
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2626
>             Project: Hadoop
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.15.2
>            Reporter: Frédéric Bertin
>         Attachments: HADOOP-2626.patch
>
>
> as a result, files with special characters (that get encoded when translated to URIs) are badly handled using a local filesystem.
> {{new Path(f.toURI().toString()))}} should be replaced by {{new Path(f.toURI().getPath()))}}
> IMHO, each call to {{toURI().toString()}} should be considered suspicious. There's another one in the class CopyFiles at line 641.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-2626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Friol updated HADOOP-2626:
---------------------------------

    Attachment: HADOOP-2626.patch

> RawLocalFileStatus is badly handling URIs
> -----------------------------------------
>
>                 Key: HADOOP-2626
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2626
>             Project: Hadoop
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.15.2
>            Reporter: Frédéric Bertin
>         Attachments: HADOOP-2626.patch
>
>
> as a result, files with special characters (that get encoded when translated to URIs) are badly handled using a local filesystem.
> {{new Path(f.toURI().toString()))}} should be replaced by {{new Path(f.toURI().getPath()))}}
> IMHO, each call to {{toURI().toString()}} should be considered suspicious. There's another one in the class CopyFiles at line 641.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-2626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Friol updated HADOOP-2626:
---------------------------------

    Status: Patch Available  (was: Open)

What about this patch then ?

> RawLocalFileStatus is badly handling URIs
> -----------------------------------------
>
>                 Key: HADOOP-2626
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2626
>             Project: Hadoop
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.15.2
>            Reporter: Frédéric Bertin
>         Attachments: HADOOP-2626.patch
>
>
> as a result, files with special characters (that get encoded when translated to URIs) are badly handled using a local filesystem.
> {{new Path(f.toURI().toString()))}} should be replaced by {{new Path(f.toURI().getPath()))}}
> IMHO, each call to {{toURI().toString()}} should be considered suspicious. There's another one in the class CopyFiles at line 641.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-2626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12559886#action_12559886 ]

Edward Yoon commented on HADOOP-2626:
-------------------------------------

If local file included a '%' character in a file name, we can't copy to dfs becuase RawLocalFileSystem.getPath() returns urlencoded '%25'.
IMO, I think it is a bug.

> RawLocalFileStatus is badly handling URIs
> -----------------------------------------
>
>                 Key: HADOOP-2626
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2626
>             Project: Hadoop
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.15.2
>            Reporter: Frédéric Bertin
>         Attachments: HADOOP-2626.patch
>
>
> as a result, files with special characters (that get encoded when translated to URIs) are badly handled using a local filesystem.
> {{new Path(f.toURI().toString()))}} should be replaced by {{new Path(f.toURI().getPath()))}}
> IMHO, each call to {{toURI().toString()}} should be considered suspicious. There's another one in the class CopyFiles at line 641.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-2626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12559886#action_12559886 ]

udanax edited comment on HADOOP-2626 at 1/17/08 2:15 AM:
--------------------------------------------------------------

If local file included a '%' character in a file name, we can't copy to dfs becuase FileStatus.getPath() returns urlencoded '%25'.
IMO, I think it is a bug.

      was (Author: udanax):
    If local file included a '%' character in a file name, we can't copy to dfs becuase RawLocalFileSystem.getPath() returns urlencoded '%25'.
IMO, I think it is a bug.
 

> RawLocalFileStatus is badly handling URIs
> -----------------------------------------
>
>                 Key: HADOOP-2626
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2626
>             Project: Hadoop
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.15.2
>            Reporter: Frédéric Bertin
>         Attachments: HADOOP-2626.patch
>
>
> as a result, files with special characters (that get encoded when translated to URIs) are badly handled using a local filesystem.
> {{new Path(f.toURI().toString()))}} should be replaced by {{new Path(f.toURI().getPath()))}}
> IMHO, each call to {{toURI().toString()}} should be considered suspicious. There's another one in the class CopyFiles at line 641.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-2626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12559889#action_12559889 ]

Edward Yoon commented on HADOOP-2626:
-------------------------------------

For example :

FileStatus fileStatus = new RawLocalFileStatus(new File("udanax/15.7%.dat"), getDefaultBlockSize());
LOG.info(fileStatus.getPath()) // It will log the 'file:/root/workspace/hadoop/udanax/15.7%25.dat' string.

> RawLocalFileStatus is badly handling URIs
> -----------------------------------------
>
>                 Key: HADOOP-2626
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2626
>             Project: Hadoop
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.15.2
>            Reporter: Frédéric Bertin
>         Attachments: HADOOP-2626.patch
>
>
> as a result, files with special characters (that get encoded when translated to URIs) are badly handled using a local filesystem.
> {{new Path(f.toURI().toString()))}} should be replaced by {{new Path(f.toURI().getPath()))}}
> IMHO, each call to {{toURI().toString()}} should be considered suspicious. There's another one in the class CopyFiles at line 641.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-2626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12559966#action_12559966 ]

Nigel Daley commented on HADOOP-2626:
-------------------------------------

Please include a test case.

> RawLocalFileStatus is badly handling URIs
> -----------------------------------------
>
>                 Key: HADOOP-2626
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2626
>             Project: Hadoop
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.15.2
>            Reporter: Frédéric Bertin
>         Attachments: HADOOP-2626.patch
>
>
> as a result, files with special characters (that get encoded when translated to URIs) are badly handled using a local filesystem.
> {{new Path(f.toURI().toString()))}} should be replaced by {{new Path(f.toURI().getPath()))}}
> IMHO, each call to {{toURI().toString()}} should be considered suspicious. There's another one in the class CopyFiles at line 641.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-2626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12560043#action_12560043 ]

Doug Cutting commented on HADOOP-2626:
--------------------------------------

> What about this patch then ?

That looks better to me, in that the returned Path is now fully qualified.  Does it handle escapes any better than before?  If not, 'new Path(file.toUri().getPath()).makeQualified(fs)' may do better.

As Nigel indicates, some test cases would be very useful.

> RawLocalFileStatus is badly handling URIs
> -----------------------------------------
>
>                 Key: HADOOP-2626
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2626
>             Project: Hadoop
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.15.2
>            Reporter: Frédéric Bertin
>         Attachments: HADOOP-2626.patch
>
>
> as a result, files with special characters (that get encoded when translated to URIs) are badly handled using a local filesystem.
> {{new Path(f.toURI().toString()))}} should be replaced by {{new Path(f.toURI().getPath()))}}
> IMHO, each call to {{toURI().toString()}} should be considered suspicious. There's another one in the class CopyFiles at line 641.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-2626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12560193#action_12560193 ]

Hadoop QA commented on HADOOP-2626:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12373373/HADOOP-2626.patch
against trunk revision r613022.

    @author +1.  The patch does not contain any @author tags.

    javadoc +1.  The javadoc tool did not generate any warning messages.

    javac +1.  The applied patch does not generate any new compiler warnings.

    findbugs +1.  The patch does not introduce any new Findbugs warnings.

    core tests +1.  The patch passed core unit tests.

    contrib tests -1.  The patch failed contrib unit tests.

Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1631/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1631/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1631/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1631/console

This message is automatically generated.

> RawLocalFileStatus is badly handling URIs
> -----------------------------------------
>
>                 Key: HADOOP-2626
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2626
>             Project: Hadoop
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.15.2
>            Reporter: Frédéric Bertin
>         Attachments: HADOOP-2626.patch
>
>
> as a result, files with special characters (that get encoded when translated to URIs) are badly handled using a local filesystem.
> {{new Path(f.toURI().toString()))}} should be replaced by {{new Path(f.toURI().getPath()))}}
> IMHO, each call to {{toURI().toString()}} should be considered suspicious. There's another one in the class CopyFiles at line 641.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-2626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12560263#action_12560263 ]

Thomas Friol commented on HADOOP-2626:
--------------------------------------

Well, it has at least fix our problem. So it seems to be ok.
Will try add unit tests soon but if someone has time to do it now, please, do not hesitate. :-)

BTW, the patch is failing in hudson because of a failing unit test which seems not to be related to this modification.

> RawLocalFileStatus is badly handling URIs
> -----------------------------------------
>
>                 Key: HADOOP-2626
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2626
>             Project: Hadoop
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.15.2
>            Reporter: Frédéric Bertin
>         Attachments: HADOOP-2626.patch
>
>
> as a result, files with special characters (that get encoded when translated to URIs) are badly handled using a local filesystem.
> {{new Path(f.toURI().toString()))}} should be replaced by {{new Path(f.toURI().getPath()))}}
> IMHO, each call to {{toURI().toString()}} should be considered suspicious. There's another one in the class CopyFiles at line 641.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-2626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-2626:
---------------------------------

    Fix Version/s: 0.16.0
         Assignee: Doug Cutting

> RawLocalFileStatus is badly handling URIs
> -----------------------------------------
>
>                 Key: HADOOP-2626
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2626
>             Project: Hadoop
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.15.2
>            Reporter: Frédéric Bertin
>            Assignee: Doug Cutting
>             Fix For: 0.16.0
>
>         Attachments: HADOOP-2626.patch
>
>
> as a result, files with special characters (that get encoded when translated to URIs) are badly handled using a local filesystem.
> {{new Path(f.toURI().toString()))}} should be replaced by {{new Path(f.toURI().getPath()))}}
> IMHO, each call to {{toURI().toString()}} should be considered suspicious. There's another one in the class CopyFiles at line 641.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.