[jira] Created: (HADOOP-2404) HADOOP-2185 breaks compatibility with hadoop-0.15.0

HADOOP-2185 breaks compatibility with hadoop-0.15.0
---------------------------------------------------

                 Key: HADOOP-2404
                 URL: https://issues.apache.org/jira/browse/HADOOP-2404
             Project: Hadoop
          Issue Type: Bug
          Components: conf
    Affects Versions: 0.16.0
            Reporter: Arun C Murthy
            Assignee: Konstantin Shvachko
            Priority: Blocker
             Fix For: 0.16.0


HADOOP-2185 removed the following configuration parameters:

{noformat}
dfs.secondary.info.port
dfs.datanode.port
dfs.info.port
mapred.job.tracker.info.port
tasktracker.http.port
{noformat}

and changed the following configuration parameters:
{noformat}
dfs.secondary.info.bindAddress
dfs.datanode.bindAddress
dfs.info.bindAddress
mapred.job.tracker.info.bindAddress
mapred.task.tracker.report.bindAddress
tasktracker.http.bindAddress
{noformat}

without a backward-compatibility story.

Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12550445 ]

Hemanth Yamijala commented on HADOOP-2404:
------------------------------------------

One of the applications that failed was Hadoop-On-Demand, which is being used internally at Yahoo. HOD brings up the DFS and MapReduce web UI on dynamically allocated cluster nodes using the older configuration items. With Hadoop trunk, these fail, and the web UI is brought up on the default nodes. HOD thus incorrectly reports the ports where the info servers are brought up.

> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
>                 Key: HADOOP-2404
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2404
>             Project: Hadoop
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.16.0
>            Reporter: Arun C Murthy
>            Assignee: Konstantin Shvachko
>            Priority: Blocker
>             Fix For: 0.16.0
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12550711 ]

Arun C Murthy commented on HADOOP-2404:
---------------------------------------

Konstantin, could you also please fix any documentation, if needed?

It would be great to atleast clarify the behaviour wrt port-rolling etc. (maybe cluster_setup.html)? Thanks!

> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
>                 Key: HADOOP-2404
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2404
>             Project: Hadoop
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.16.0
>            Reporter: Arun C Murthy
>            Assignee: Konstantin Shvachko
>            Priority: Blocker
>             Fix For: 0.16.0
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12550717 ]

Konstantin Shvachko commented on HADOOP-2404:
---------------------------------------------

Arun, There are two +1s from you on HADOOP-2185.
It also clarifies about the port rolling and etc.

> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
>                 Key: HADOOP-2404
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2404
>             Project: Hadoop
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.16.0
>            Reporter: Arun C Murthy
>            Assignee: Konstantin Shvachko
>            Priority: Blocker
>             Fix For: 0.16.0
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12550831 ]

Konstantin Shvachko commented on HADOOP-2404:
---------------------------------------------

HADOOP-2185 introduced two types of changes:
# Change in semantics: no port rolling for any ports.
# Configuration api changes: 5 properties have been removed and 5 renamed.

Here is the compatibility plan.
# Semantical changes will not be backward compatible.
# Old configuration variables will be recognized in 0.16 as described below.

If a new name of a configuration parameter is present, then old names corresponding
to this parameter will be ignored.

If old names of a configuration parameter are present but not the new ones
the old parameters will be converted to the new according to the conversion
table provided in HADOOP-2185.

The conversion will be done right after loading all Configuration resources
(see Configuration.loadResources()) and Hadoop will further work with the
converted configuration.

I was assured this will work fine for HOD and Pig.
But if your application uses tricks like:
create config; call new JobTracker(config); then get the job-tracker info port
by calling config.get("mapred.job.tracker.info.port")
introduced by HADOOP-1085 this will not work. You will need to change your application
to use config.get("mapred.job.tracker.http.bindAddress") and then extract the port.

> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
>                 Key: HADOOP-2404
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2404
>             Project: Hadoop
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.16.0
>            Reporter: Arun C Murthy
>            Assignee: Konstantin Shvachko
>            Priority: Blocker
>             Fix For: 0.16.0
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Shvachko updated HADOOP-2404:
----------------------------------------

    Attachment: ConfigurationConverter.patch

This patch performs configuration conversion as discussed.
I tested it with new, old, and mixed configurations. It works for me.
Could anybody please verify it on HOD.

> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
>                 Key: HADOOP-2404
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2404
>             Project: Hadoop
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.16.0
>            Reporter: Arun C Murthy
>            Assignee: Konstantin Shvachko
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12550869 ]

dhruba borthakur commented on HADOOP-2404:
------------------------------------------

I am unable to get it to working with this patch, I see this exception on all datanodes logs:

2007-12-12 07:15:14,063 ERROR org.apache.hadoop.dfs.DataNode: java.net.BindException: Address already in use
        at java.net.PlainSocketImpl.socketBind(Native Method)
        at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:359)
        at java.net.ServerSocket.bind(ServerSocket.java:319)
        at java.net.ServerSocket.<init>(ServerSocket.java:185)
        at org.apache.hadoop.dfs.DataNode.startDataNode(DataNode.java:278)
        at org.apache.hadoop.dfs.DataNode.<init>(DataNode.java:204)
        at org.apache.hadoop.dfs.DataNode.makeInstance(DataNode.java:2204)
        at org.apache.hadoop.dfs.DataNode.run(DataNode.java:2148)
        at org.apache.hadoop.dfs.DataNode.createDataNode(DataNode.java:2169)
        at org.apache.hadoop.dfs.DataNode.main(DataNode.java:2360)

No message about "deprecated configuration parameter" appear in the datanode logs.

> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
>                 Key: HADOOP-2404
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2404
>             Project: Hadoop
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.16.0
>            Reporter: Arun C Murthy
>            Assignee: Konstantin Shvachko
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12551080 ]

Doug Cutting commented on HADOOP-2404:
--------------------------------------

-1 We should not fix this in Configuration.java, but rather in the classes that use the configuration.  So, if you wish to do this via Configuration fixup, add it as a method in HDFS, and call that method in HDFS where it constructs its configurations.

> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
>                 Key: HADOOP-2404
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2404
>             Project: Hadoop
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.16.0
>            Reporter: Arun C Murthy
>            Assignee: Konstantin Shvachko
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12551099 ]

Konstantin Shvachko commented on HADOOP-2404:
---------------------------------------------

Dhruba, this means that there is something else running on the same port. Most likely another data-node on the default port.
As I mentioned before this patch does not reverse port rolling behavior, just provides support for the deprecated config parameters.
And "deprecated configuration parameter" message does not appear because your configuration does not contain old names.
The application you use should explicitly specify port 0 if the actual port # is meant to be arbitrary.

Doug, I agree this would be a better solution. But this will be a much bigger change now and then when we remove
deprecation. There are 5 servers involved. So at least 5 places should be patched if we go your way.
The problem is only with HOD for now as I understand. HOD does not use Configuration class, but rather
generates xml files by directly writing into them. So there is no way they can call any methods at all.

> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
>                 Key: HADOOP-2404
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2404
>             Project: Hadoop
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.16.0
>            Reporter: Arun C Murthy
>            Assignee: Konstantin Shvachko
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12551105 ]

Doug Cutting commented on HADOOP-2404:
--------------------------------------

> There are 5 servers involved.

Then that's five one-line changes.  I'd rather see those than establish the precedent of making application-specific changes in Configuration.java.  Each server can call the same method.  When that method is removed in the next release the compiler will identify the five calls to it that must be removed.

Better yet, you could use static methods to access these parameters.  So instead of directly reading dfs.http.bindAddress, add a package-private static method getHttpBindAddress(Configuration) that HDFS calls instead of directly reading the Configuration property in six different places.  That method can implement back-compatibility.

> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
>                 Key: HADOOP-2404
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2404
>             Project: Hadoop
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.16.0
>            Reporter: Arun C Murthy
>            Assignee: Konstantin Shvachko
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur updated HADOOP-2404:
-------------------------------------

    Status: Patch Available  (was: Open)

> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
>                 Key: HADOOP-2404
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2404
>             Project: Hadoop
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.16.0
>            Reporter: Arun C Murthy
>            Assignee: Konstantin Shvachko
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: asyncRPC-9.patch, ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur updated HADOOP-2404:
-------------------------------------

    Attachment: asyncRPC-9.patch

I inserted a LOG message if asyncWrite generated an exception.

> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
>                 Key: HADOOP-2404
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2404
>             Project: Hadoop
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.16.0
>            Reporter: Arun C Murthy
>            Assignee: Konstantin Shvachko
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: asyncRPC-9.patch, ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur updated HADOOP-2404:
-------------------------------------

    Status: Open  (was: Patch Available)

Wrong patch, Oops, sorry!

> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
>                 Key: HADOOP-2404
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2404
>             Project: Hadoop
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.16.0
>            Reporter: Arun C Murthy
>            Assignee: Konstantin Shvachko
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: asyncRPC-9.patch, ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur updated HADOOP-2404:
-------------------------------------

    Attachment:     (was: asyncRPC-9.patch)

> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
>                 Key: HADOOP-2404
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2404
>             Project: Hadoop
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.16.0
>            Reporter: Arun C Murthy
>            Assignee: Konstantin Shvachko
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Shvachko updated HADOOP-2404:
----------------------------------------

    Comment: was deleted

> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
>                 Key: HADOOP-2404
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2404
>             Project: Hadoop
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.16.0
>            Reporter: Arun C Murthy
>            Assignee: Konstantin Shvachko
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Shvachko updated HADOOP-2404:
----------------------------------------

    Comment: was deleted

> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
>                 Key: HADOOP-2404
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2404
>             Project: Hadoop
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.16.0
>            Reporter: Arun C Murthy
>            Assignee: Konstantin Shvachko
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12551673 ]

Konstantin Shvachko commented on HADOOP-2404:
---------------------------------------------

May be should just declare that this change is incompatible and not provide any conversion of the changed parameters at all.
We cannot provide true compatibility. Whoever relied on the default port usage expecting (consciously or unconsciously) them
to be rolled if they have are busy, would have to explicitly specify 0-ports. This means the configuration should be changed,
so users might change the names as well.

> instead of directly reading "dfs.http.bindAddress", add a package-private static method getHttpBindAddress(Configuration)
+1  I really hate that we keep using raw names rather than providing getters/setters/
This should be done for all configuration parameters imo, and probably belongs to HADOOP-2385.

> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
>                 Key: HADOOP-2404
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2404
>             Project: Hadoop
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.16.0
>            Reporter: Arun C Murthy
>            Assignee: Konstantin Shvachko
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12551673 ]

shv edited comment on HADOOP-2404 at 12/13/07 5:51 PM:
-----------------------------------------------------------------------

May be we should just declare that this change is incompatible and not provide any conversion of the changed parameters at all.
We cannot provide true compatibility. Whoever relied on the default port usage expecting (consciously or unconsciously) them
to be rolled if they are busy, would have to explicitly specify 0-ports. This means the configuration should be changed,
so users might change the names as well.

> instead of directly reading "dfs.http.bindAddress", add a package-private static method getHttpBindAddress(Configuration)
+1  I really hate that we keep using raw names rather than providing getters/setters.
This should be done for all configuration parameters imo, and probably belongs to HADOOP-2385.

      was (Author: shv):
    May be should just declare that this change is incompatible and not provide any conversion of the changed parameters at all.
We cannot provide true compatibility. Whoever relied on the default port usage expecting (consciously or unconsciously) them
to be rolled if they have are busy, would have to explicitly specify 0-ports. This means the configuration should be changed,
so users might change the names as well.

> instead of directly reading "dfs.http.bindAddress", add a package-private static method getHttpBindAddress(Configuration)
+1  I really hate that we keep using raw names rather than providing getters/setters/
This should be done for all configuration parameters imo, and probably belongs to HADOOP-2385.
 

> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
>                 Key: HADOOP-2404
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2404
>             Project: Hadoop
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.16.0
>            Reporter: Arun C Murthy
>            Assignee: Konstantin Shvachko
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hemanth Yamijala updated HADOOP-2404:
-------------------------------------


bq. May be we should just declare that this change is incompatible and not provide any conversion of the changed parameters at all. We cannot provide true compatibility. Whoever relied on the default port usage expecting (consciously or unconsciously) them to be rolled if they are busy, would have to explicitly specify 0-ports. This means the configuration should be changed, so users might change the names as well.

I feel this is a fair argument. We are OK to make this change in our application.

> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
>                 Key: HADOOP-2404
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2404
>             Project: Hadoop
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.16.0
>            Reporter: Arun C Murthy
>            Assignee: Konstantin Shvachko
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12552832 ]

Arun C Murthy commented on HADOOP-2404:
---------------------------------------

{quote}
May be we should just declare that this change is incompatible and not provide any conversion of the changed parameters at all.
We cannot provide true compatibility. Whoever relied on the default port usage expecting (consciously or unconsciously) them
to be rolled if they are busy, would have to explicitly specify 0-ports. This means the configuration should be changed,
so users might change the names as well.
{quote}

I'm *ok* with some pieces changing semantics, but I worry about current configs breaking - to be honest most people don't care as much about the port-rolling semantics as they do about having their current hadoop-site.xml work as-is. Given that I believe it's important to maintain compatibility... however I'm willing to listen.

> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
>                 Key: HADOOP-2404
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2404
>             Project: Hadoop
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.16.0
>            Reporter: Arun C Murthy
>            Assignee: Konstantin Shvachko
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.