[jira] Created: (HADOOP-1985) Abstract node to switch mapping into a topology service class used by namenode and jobtracker

Abstract node to switch mapping into a topology service class used by namenode and jobtracker
---------------------------------------------------------------------------------------------

                 Key: HADOOP-1985
                 URL: https://issues.apache.org/jira/browse/HADOOP-1985
             Project: Hadoop
          Issue Type: New Feature
            Reporter: eric baldeschwieler


In order to implement switch locality in MapReduce, we need to have switch location in both the namenode and job tracker.  Currently the namenode asks the data nodes for this info and they run a local script to answer this question.  In our environment and others that I know of there is no reason to push this to each node.  It is easier to maintain a centralized script that maps node DNS names to switch strings.

I propose that we build a new class that caches known DNS name to switch mappings and invokes a loadable class or a configurable system call to resolve unknown DNS to switch mappings.  We can then add this to the namenode to support the current block to switch mapping needs and simplify the data nodes.  We can also add this same callout to the job tracker and then implement rack locality logic there without needing to chane the filesystem API or the split planning API.

Not only is this the least intrusive path to building racklocal MR I can ID, it is also future compatible to future infrastructures that may derive topology on the fly, etc, etc...

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12533350 ]

Runping Qi commented on HADOOP-1985:
------------------------------------


yes, DNS name (hostname) to switch id mapping should be managed just like the hostname to IP mapping.
The info should be available to the DFS  namenode, datanodes and applications in the same way.
Job tracker uses this info for task assignment. In general, DFS client should also use this info to decide where to fetch a needed block.

> Abstract node to switch mapping into a topology service class used by namenode and jobtracker
> ---------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-1985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1985
>             Project: Hadoop
>          Issue Type: New Feature
>            Reporter: eric baldeschwieler
>
> In order to implement switch locality in MapReduce, we need to have switch location in both the namenode and job tracker.  Currently the namenode asks the data nodes for this info and they run a local script to answer this question.  In our environment and others that I know of there is no reason to push this to each node.  It is easier to maintain a centralized script that maps node DNS names to switch strings.
> I propose that we build a new class that caches known DNS name to switch mappings and invokes a loadable class or a configurable system call to resolve unknown DNS to switch mappings.  We can then add this to the namenode to support the current block to switch mapping needs and simplify the data nodes.  We can also add this same callout to the job tracker and then implement rack locality logic there without needing to chane the filesystem API or the split planning API.
> Not only is this the least intrusive path to building racklocal MR I can ID, it is also future compatible to future infrastructures that may derive topology on the fly, etc, etc...

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das reassigned HADOOP-1985:
-----------------------------------

    Assignee: Devaraj Das

> Abstract node to switch mapping into a topology service class used by namenode and jobtracker
> ---------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-1985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1985
>             Project: Hadoop
>          Issue Type: New Feature
>            Reporter: eric baldeschwieler
>            Assignee: Devaraj Das
>
> In order to implement switch locality in MapReduce, we need to have switch location in both the namenode and job tracker.  Currently the namenode asks the data nodes for this info and they run a local script to answer this question.  In our environment and others that I know of there is no reason to push this to each node.  It is easier to maintain a centralized script that maps node DNS names to switch strings.
> I propose that we build a new class that caches known DNS name to switch mappings and invokes a loadable class or a configurable system call to resolve unknown DNS to switch mappings.  We can then add this to the namenode to support the current block to switch mapping needs and simplify the data nodes.  We can also add this same callout to the job tracker and then implement rack locality logic there without needing to chane the filesystem API or the split planning API.
> Not only is this the least intrusive path to building racklocal MR I can ID, it is also future compatible to future infrastructures that may derive topology on the fly, etc, etc...

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12538717 ]

Devaraj Das commented on HADOOP-1985:
-------------------------------------

Some thoughts -
1) Make the DNS->Switch mapping an interface class.
    1.1) interface DNStoSwitchMap {
               public String resolve(String dnsname);
            }
    1.2) The switch string format is the same as it exists today (documented in https://issues.apache.org/jira/secure/attachment/12345251/Rack_aware_HDFS_proposal.pdf). That will make things work in the non-typical setup with 3+ levels of nodes.
    1.3) The default implementation of the interface, packaged with hadoop, could simply look up a table of dns->switch mapping created statically.

2) The DataNode, today, takes the location as an argument. This is not needed anymore, and hence the associated code would go away.
3) The DataNode sends the location information as part of the registration. The NetworkTopology is derived at the NameNode. Using the interface mentioned in (1), the NameNode can create the topology all by itself.

4) The JobTracker creates the NetworkTopology for the TaskTrackers exactly how the NameNode does it.
5) The JobTracker assigns tasks first on node-locality basis, then on rack-locality basis.

In our environment,  "distance-basis" (o.a.h.n.NetworkTopology.getDistance), isn't that much relevant. But we might make the framework ready for it as well (assuming it is not too much work).

Does the above make sense?

> Abstract node to switch mapping into a topology service class used by namenode and jobtracker
> ---------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-1985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1985
>             Project: Hadoop
>          Issue Type: New Feature
>            Reporter: eric baldeschwieler
>            Assignee: Devaraj Das
>
> In order to implement switch locality in MapReduce, we need to have switch location in both the namenode and job tracker.  Currently the namenode asks the data nodes for this info and they run a local script to answer this question.  In our environment and others that I know of there is no reason to push this to each node.  It is easier to maintain a centralized script that maps node DNS names to switch strings.
> I propose that we build a new class that caches known DNS name to switch mappings and invokes a loadable class or a configurable system call to resolve unknown DNS to switch mappings.  We can then add this to the namenode to support the current block to switch mapping needs and simplify the data nodes.  We can also add this same callout to the job tracker and then implement rack locality logic there without needing to chane the filesystem API or the split planning API.
> Not only is this the least intrusive path to building racklocal MR I can ID, it is also future compatible to future infrastructures that may derive topology on the fly, etc, etc...

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12538717 ]

devaraj edited comment on HADOOP-1985 at 10/30/07 3:43 AM:
---------------------------------------------------------------

Some thoughts -
1) Make the DNS->Switch mapping an interface class.
    1.1) interface DNStoSwitchMap {
               public String resolve(String dnsname);
            }
    1.2) The switch string format is the same as it exists today (documented in https://issues.apache.org/jira/secure/attachment/12345251/Rack_aware_HDFS_proposal.pdf). That will make things work in the non-typical setup with 3+ levels of nodes.
    1.3) The default implementation of the interface, packaged with hadoop, could simply look up a table of dns->switch mapping created statically.

2) The DataNode, today, takes the location as an argument. This is not needed anymore, and hence the associated code would go away.
3) The DataNode sends the location information as part of the registration. The NetworkTopology is derived at the NameNode. Using the interface mentioned in (1), the NameNode can create the topology all by itself.

4) The JobTracker creates the NetworkTopology for the TaskTrackers exactly how the NameNode does it.
5) The JobTracker assigns tasks first on node-locality basis, then on rack-locality basis.

In our environment,  task placement based on "distance" (o.a.h.n.NetworkTopology.getDistance), isn't that much relevant since we only have flat racks of machines. But we might make the framework ready for it as well (assuming it is not too much work).

Does the above make sense?

      was (Author: devaraj):
    Some thoughts -
1) Make the DNS->Switch mapping an interface class.
    1.1) interface DNStoSwitchMap {
               public String resolve(String dnsname);
            }
    1.2) The switch string format is the same as it exists today (documented in https://issues.apache.org/jira/secure/attachment/12345251/Rack_aware_HDFS_proposal.pdf). That will make things work in the non-typical setup with 3+ levels of nodes.
    1.3) The default implementation of the interface, packaged with hadoop, could simply look up a table of dns->switch mapping created statically.

2) The DataNode, today, takes the location as an argument. This is not needed anymore, and hence the associated code would go away.
3) The DataNode sends the location information as part of the registration. The NetworkTopology is derived at the NameNode. Using the interface mentioned in (1), the NameNode can create the topology all by itself.

4) The JobTracker creates the NetworkTopology for the TaskTrackers exactly how the NameNode does it.
5) The JobTracker assigns tasks first on node-locality basis, then on rack-locality basis.

In our environment,  "distance-basis" (o.a.h.n.NetworkTopology.getDistance), isn't that much relevant. But we might make the framework ready for it as well (assuming it is not too much work).

Does the above make sense?
 

> Abstract node to switch mapping into a topology service class used by namenode and jobtracker
> ---------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-1985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1985
>             Project: Hadoop
>          Issue Type: New Feature
>            Reporter: eric baldeschwieler
>            Assignee: Devaraj Das
>
> In order to implement switch locality in MapReduce, we need to have switch location in both the namenode and job tracker.  Currently the namenode asks the data nodes for this info and they run a local script to answer this question.  In our environment and others that I know of there is no reason to push this to each node.  It is easier to maintain a centralized script that maps node DNS names to switch strings.
> I propose that we build a new class that caches known DNS name to switch mappings and invokes a loadable class or a configurable system call to resolve unknown DNS to switch mappings.  We can then add this to the namenode to support the current block to switch mapping needs and simplify the data nodes.  We can also add this same callout to the job tracker and then implement rack locality logic there without needing to chane the filesystem API or the split planning API.
> Not only is this the least intrusive path to building racklocal MR I can ID, it is also future compatible to future infrastructures that may derive topology on the fly, etc, etc...

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12538857 ]

eric baldeschwieler commented on HADOOP-1985:
---------------------------------------------

works for me

> Abstract node to switch mapping into a topology service class used by namenode and jobtracker
> ---------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-1985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1985
>             Project: Hadoop
>          Issue Type: New Feature
>            Reporter: eric baldeschwieler
>            Assignee: Devaraj Das
>
> In order to implement switch locality in MapReduce, we need to have switch location in both the namenode and job tracker.  Currently the namenode asks the data nodes for this info and they run a local script to answer this question.  In our environment and others that I know of there is no reason to push this to each node.  It is easier to maintain a centralized script that maps node DNS names to switch strings.
> I propose that we build a new class that caches known DNS name to switch mappings and invokes a loadable class or a configurable system call to resolve unknown DNS to switch mappings.  We can then add this to the namenode to support the current block to switch mapping needs and simplify the data nodes.  We can also add this same callout to the job tracker and then implement rack locality logic there without needing to chane the filesystem API or the split planning API.
> Not only is this the least intrusive path to building racklocal MR I can ID, it is also future compatible to future infrastructures that may derive topology on the fly, etc, etc...

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12538872 ]

Allen Wittenauer commented on HADOOP-1985:
------------------------------------------

Just a few notes.

From an ops perspective, it is important that this mapping be highly pluggable in an easy way.  The ability to have hadoop call some sort of executable (not necessarily a script) means we can do fancy things with /etc/netmasks or LDAP lookups or ... .  Ideally, every sort of mapping would have a callout rather than having one big one. KISS is important here.  [Remember, most admins--myself included--are not hardcore Java people. ]

FWIW, most implementations of autofs include similar functionality called executable maps where the key is passed to an exec and the exec returns the location of the mount.  So the practice has at least a little bit of traction.  [In fact, auto.net aka /net on Linux uses this method.]

Additionally,I think moving this functionality to be done on the namenode makes this significantly easier to manage as a grid scales up.  There is also the issue of should the namenode 'trust' the datanode to report the proper location.  I understand that the datanode and namenode have to trust each other at some point during node bringup, but I think it makes a lot of sense to let the namenode be in charge of data locality.

Hopefuly this was helpful.

> Abstract node to switch mapping into a topology service class used by namenode and jobtracker
> ---------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-1985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1985
>             Project: Hadoop
>          Issue Type: New Feature
>            Reporter: eric baldeschwieler
>            Assignee: Devaraj Das
>
> In order to implement switch locality in MapReduce, we need to have switch location in both the namenode and job tracker.  Currently the namenode asks the data nodes for this info and they run a local script to answer this question.  In our environment and others that I know of there is no reason to push this to each node.  It is easier to maintain a centralized script that maps node DNS names to switch strings.
> I propose that we build a new class that caches known DNS name to switch mappings and invokes a loadable class or a configurable system call to resolve unknown DNS to switch mappings.  We can then add this to the namenode to support the current block to switch mapping needs and simplify the data nodes.  We can also add this same callout to the job tracker and then implement rack locality logic there without needing to chane the filesystem API or the split planning API.
> Not only is this the least intrusive path to building racklocal MR I can ID, it is also future compatible to future infrastructures that may derive topology on the fly, etc, etc...

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12539020 ]

eric baldeschwieler commented on HADOOP-1985:
---------------------------------------------

I agree that an exec as a simple to config option should be required.

> Abstract node to switch mapping into a topology service class used by namenode and jobtracker
> ---------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-1985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1985
>             Project: Hadoop
>          Issue Type: New Feature
>            Reporter: eric baldeschwieler
>            Assignee: Devaraj Das
>
> In order to implement switch locality in MapReduce, we need to have switch location in both the namenode and job tracker.  Currently the namenode asks the data nodes for this info and they run a local script to answer this question.  In our environment and others that I know of there is no reason to push this to each node.  It is easier to maintain a centralized script that maps node DNS names to switch strings.
> I propose that we build a new class that caches known DNS name to switch mappings and invokes a loadable class or a configurable system call to resolve unknown DNS to switch mappings.  We can then add this to the namenode to support the current block to switch mapping needs and simplify the data nodes.  We can also add this same callout to the job tracker and then implement rack locality logic there without needing to chane the filesystem API or the split planning API.
> Not only is this the least intrusive path to building racklocal MR I can ID, it is also future compatible to future infrastructures that may derive topology on the fly, etc, etc...

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12540233 ]

Hairong Kuang commented on HADOOP-1985:
---------------------------------------

Currently in dfs a datanode can get its network location either from the command line or by running a pluggable script at the startup time. The property is defined in the default configuration file as below.

<property>
  <name>dfs.network.script</name>
  <value></value>
  <description>
        Specifies a script name that print the network location path
        of the current machine.
  </description>
</property>

> Abstract node to switch mapping into a topology service class used by namenode and jobtracker
> ---------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-1985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1985
>             Project: Hadoop
>          Issue Type: New Feature
>            Reporter: eric baldeschwieler
>            Assignee: Devaraj Das
>
> In order to implement switch locality in MapReduce, we need to have switch location in both the namenode and job tracker.  Currently the namenode asks the data nodes for this info and they run a local script to answer this question.  In our environment and others that I know of there is no reason to push this to each node.  It is easier to maintain a centralized script that maps node DNS names to switch strings.
> I propose that we build a new class that caches known DNS name to switch mappings and invokes a loadable class or a configurable system call to resolve unknown DNS to switch mappings.  We can then add this to the namenode to support the current block to switch mapping needs and simplify the data nodes.  We can also add this same callout to the job tracker and then implement rack locality logic there without needing to chane the filesystem API or the split planning API.
> Not only is this the least intrusive path to building racklocal MR I can ID, it is also future compatible to future infrastructures that may derive topology on the fly, etc, etc...

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-1985:
--------------------------------

    Attachment: 1985.new.patch

Attached is one version of the patch. Hasn't been tested on large clusters yet. Here are the main changes:
1) DFS part updated to use the newly defined DNSToSwitchMapping interface.
  1.1) The datanode doesn't send the switch info as part of registration, rather, the namenode gets that info through the
  DNSToSwitchMapping.resolve
2) The default implementation of the DNSToSwitchMapping assumes a script based resolution (ScriptBasedMapping). If the script is defined, then DEFAULT_RACK is assumed.
3) The JobTracker maintains the network topology and updates it (if required) whenever a tasktracker sends a heartbeat.
4) The JobInProgress maintains a compile-time-configurable number of task caches. For e.g., the first level cache is the map of leaf level Nodes to TIPs, the second level is the map of the rack level nodes to TIPs in that rack, the third level is for the level above rack and so on.. The default number of caches here is hardcoded to 2.
5) At runtime, the findNewTask would use these caches to assign a task to a checked-in tasktracker.

Patch up for review.

> Abstract node to switch mapping into a topology service class used by namenode and jobtracker
> ---------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-1985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1985
>             Project: Hadoop
>          Issue Type: New Feature
>            Reporter: eric baldeschwieler
>            Assignee: Devaraj Das
>         Attachments: 1985.new.patch
>
>
> In order to implement switch locality in MapReduce, we need to have switch location in both the namenode and job tracker.  Currently the namenode asks the data nodes for this info and they run a local script to answer this question.  In our environment and others that I know of there is no reason to push this to each node.  It is easier to maintain a centralized script that maps node DNS names to switch strings.
> I propose that we build a new class that caches known DNS name to switch mappings and invokes a loadable class or a configurable system call to resolve unknown DNS to switch mappings.  We can then add this to the namenode to support the current block to switch mapping needs and simplify the data nodes.  We can also add this same callout to the job tracker and then implement rack locality logic there without needing to chane the filesystem API or the split planning API.
> Not only is this the least intrusive path to building racklocal MR I can ID, it is also future compatible to future infrastructures that may derive topology on the fly, etc, etc...

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-1985:
--------------------------------

    Attachment: 1985.v1.patch

Attached is a tested patch. The patch has better documentation too. One of the important changes in the patch to do with testcases is the way it handles multiple datanodes in the same machine. Since the namenode should be able to distinguish between them in terms of the dnsToRackId mapping, I added a configuration option called "slave.host.name" that would take effect only when the framework is run under junit. Ditto applies to the jobtracker/tasktrackers. Also all communications to these dummy hostnames are redirected to "localhost" (indirectly via NetUtils.createSocketAddress).

Patch is up for review.

> Abstract node to switch mapping into a topology service class used by namenode and jobtracker
> ---------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-1985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1985
>             Project: Hadoop
>          Issue Type: New Feature
>            Reporter: eric baldeschwieler
>            Assignee: Devaraj Das
>         Attachments: 1985.new.patch, 1985.v1.patch
>
>
> In order to implement switch locality in MapReduce, we need to have switch location in both the namenode and job tracker.  Currently the namenode asks the data nodes for this info and they run a local script to answer this question.  In our environment and others that I know of there is no reason to push this to each node.  It is easier to maintain a centralized script that maps node DNS names to switch strings.
> I propose that we build a new class that caches known DNS name to switch mappings and invokes a loadable class or a configurable system call to resolve unknown DNS to switch mappings.  We can then add this to the namenode to support the current block to switch mapping needs and simplify the data nodes.  We can also add this same callout to the job tracker and then implement rack locality logic there without needing to chane the filesystem API or the split planning API.
> Not only is this the least intrusive path to building racklocal MR I can ID, it is also future compatible to future infrastructures that may derive topology on the fly, etc, etc...

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-1985:
--------------------------------

    Fix Version/s: 0.16.0

> Abstract node to switch mapping into a topology service class used by namenode and jobtracker
> ---------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-1985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1985
>             Project: Hadoop
>          Issue Type: New Feature
>            Reporter: eric baldeschwieler
>            Assignee: Devaraj Das
>             Fix For: 0.16.0
>
>         Attachments: 1985.new.patch, 1985.v1.patch
>
>
> In order to implement switch locality in MapReduce, we need to have switch location in both the namenode and job tracker.  Currently the namenode asks the data nodes for this info and they run a local script to answer this question.  In our environment and others that I know of there is no reason to push this to each node.  It is easier to maintain a centralized script that maps node DNS names to switch strings.
> I propose that we build a new class that caches known DNS name to switch mappings and invokes a loadable class or a configurable system call to resolve unknown DNS to switch mappings.  We can then add this to the namenode to support the current block to switch mapping needs and simplify the data nodes.  We can also add this same callout to the job tracker and then implement rack locality logic there without needing to chane the filesystem API or the split planning API.
> Not only is this the least intrusive path to building racklocal MR I can ID, it is also future compatible to future infrastructures that may derive topology on the fly, etc, etc...

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-1985:
--------------------------------

    Status: Patch Available  (was: Open)

Passing it through hudson.

> Abstract node to switch mapping into a topology service class used by namenode and jobtracker
> ---------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-1985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1985
>             Project: Hadoop
>          Issue Type: New Feature
>            Reporter: eric baldeschwieler
>            Assignee: Devaraj Das
>             Fix For: 0.16.0
>
>         Attachments: 1985.new.patch, 1985.v1.patch
>
>
> In order to implement switch locality in MapReduce, we need to have switch location in both the namenode and job tracker.  Currently the namenode asks the data nodes for this info and they run a local script to answer this question.  In our environment and others that I know of there is no reason to push this to each node.  It is easier to maintain a centralized script that maps node DNS names to switch strings.
> I propose that we build a new class that caches known DNS name to switch mappings and invokes a loadable class or a configurable system call to resolve unknown DNS to switch mappings.  We can then add this to the namenode to support the current block to switch mapping needs and simplify the data nodes.  We can also add this same callout to the job tracker and then implement rack locality logic there without needing to chane the filesystem API or the split planning API.
> Not only is this the least intrusive path to building racklocal MR I can ID, it is also future compatible to future infrastructures that may derive topology on the fly, etc, etc...

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12551804 ]

Hadoop QA commented on HADOOP-1985:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12371666/1985.v1.patch
against trunk revision r604058.

    @author +1.  The patch does not contain any @author tags.

    javadoc +1.  The javadoc tool did not generate any warning messages.

    javac +1.  The applied patch does not generate any new compiler warnings.

    findbugs -1.  The patch appears to introduce 2 new Findbugs warnings.

    core tests +1.  The patch passed core unit tests.

    contrib tests -1.  The patch failed contrib unit tests.

Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1344/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1344/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1344/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1344/console

This message is automatically generated.

> Abstract node to switch mapping into a topology service class used by namenode and jobtracker
> ---------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-1985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1985
>             Project: Hadoop
>          Issue Type: New Feature
>            Reporter: eric baldeschwieler
>            Assignee: Devaraj Das
>             Fix For: 0.16.0
>
>         Attachments: 1985.new.patch, 1985.v1.patch
>
>
> In order to implement switch locality in MapReduce, we need to have switch location in both the namenode and job tracker.  Currently the namenode asks the data nodes for this info and they run a local script to answer this question.  In our environment and others that I know of there is no reason to push this to each node.  It is easier to maintain a centralized script that maps node DNS names to switch strings.
> I propose that we build a new class that caches known DNS name to switch mappings and invokes a loadable class or a configurable system call to resolve unknown DNS to switch mappings.  We can then add this to the namenode to support the current block to switch mapping needs and simplify the data nodes.  We can also add this same callout to the job tracker and then implement rack locality logic there without needing to chane the filesystem API or the split planning API.
> Not only is this the least intrusive path to building racklocal MR I can ID, it is also future compatible to future infrastructures that may derive topology on the fly, etc, etc...

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-1985:
--------------------------------

    Attachment: 1985.v2.patch

Fixed the findbugs issues.

> Abstract node to switch mapping into a topology service class used by namenode and jobtracker
> ---------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-1985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1985
>             Project: Hadoop
>          Issue Type: New Feature
>            Reporter: eric baldeschwieler
>            Assignee: Devaraj Das
>             Fix For: 0.16.0
>
>         Attachments: 1985.new.patch, 1985.v1.patch, 1985.v2.patch
>
>
> In order to implement switch locality in MapReduce, we need to have switch location in both the namenode and job tracker.  Currently the namenode asks the data nodes for this info and they run a local script to answer this question.  In our environment and others that I know of there is no reason to push this to each node.  It is easier to maintain a centralized script that maps node DNS names to switch strings.
> I propose that we build a new class that caches known DNS name to switch mappings and invokes a loadable class or a configurable system call to resolve unknown DNS to switch mappings.  We can then add this to the namenode to support the current block to switch mapping needs and simplify the data nodes.  We can also add this same callout to the job tracker and then implement rack locality logic there without needing to chane the filesystem API or the split planning API.
> Not only is this the least intrusive path to building racklocal MR I can ID, it is also future compatible to future infrastructures that may derive topology on the fly, etc, etc...

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-1985:
--------------------------------

    Status: Open  (was: Patch Available)

> Abstract node to switch mapping into a topology service class used by namenode and jobtracker
> ---------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-1985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1985
>             Project: Hadoop
>          Issue Type: New Feature
>            Reporter: eric baldeschwieler
>            Assignee: Devaraj Das
>             Fix For: 0.16.0
>
>         Attachments: 1985.new.patch, 1985.v1.patch, 1985.v2.patch
>
>
> In order to implement switch locality in MapReduce, we need to have switch location in both the namenode and job tracker.  Currently the namenode asks the data nodes for this info and they run a local script to answer this question.  In our environment and others that I know of there is no reason to push this to each node.  It is easier to maintain a centralized script that maps node DNS names to switch strings.
> I propose that we build a new class that caches known DNS name to switch mappings and invokes a loadable class or a configurable system call to resolve unknown DNS to switch mappings.  We can then add this to the namenode to support the current block to switch mapping needs and simplify the data nodes.  We can also add this same callout to the job tracker and then implement rack locality logic there without needing to chane the filesystem API or the split planning API.
> Not only is this the least intrusive path to building racklocal MR I can ID, it is also future compatible to future infrastructures that may derive topology on the fly, etc, etc...

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-1985:
--------------------------------

    Status: Patch Available  (was: Open)

Rerunning through hudson.

> Abstract node to switch mapping into a topology service class used by namenode and jobtracker
> ---------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-1985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1985
>             Project: Hadoop
>          Issue Type: New Feature
>            Reporter: eric baldeschwieler
>            Assignee: Devaraj Das
>             Fix For: 0.16.0
>
>         Attachments: 1985.new.patch, 1985.v1.patch, 1985.v2.patch
>
>
> In order to implement switch locality in MapReduce, we need to have switch location in both the namenode and job tracker.  Currently the namenode asks the data nodes for this info and they run a local script to answer this question.  In our environment and others that I know of there is no reason to push this to each node.  It is easier to maintain a centralized script that maps node DNS names to switch strings.
> I propose that we build a new class that caches known DNS name to switch mappings and invokes a loadable class or a configurable system call to resolve unknown DNS to switch mappings.  We can then add this to the namenode to support the current block to switch mapping needs and simplify the data nodes.  We can also add this same callout to the job tracker and then implement rack locality logic there without needing to chane the filesystem API or the split planning API.
> Not only is this the least intrusive path to building racklocal MR I can ID, it is also future compatible to future infrastructures that may derive topology on the fly, etc, etc...

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12552127 ]

Hadoop QA commented on HADOOP-1985:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12371728/1985.v2.patch
against trunk revision r604451.

    @author +1.  The patch does not contain any @author tags.

    javadoc +1.  The javadoc tool did not generate any warning messages.

    javac +1.  The applied patch does not generate any new compiler warnings.

    findbugs +1.  The patch does not introduce any new Findbugs warnings.

    core tests +1.  The patch passed core unit tests.

    contrib tests +1.  The patch passed contrib unit tests.

Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1356/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1356/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1356/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1356/console

This message is automatically generated.

> Abstract node to switch mapping into a topology service class used by namenode and jobtracker
> ---------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-1985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1985
>             Project: Hadoop
>          Issue Type: New Feature
>            Reporter: eric baldeschwieler
>            Assignee: Devaraj Das
>             Fix For: 0.16.0
>
>         Attachments: 1985.new.patch, 1985.v1.patch, 1985.v2.patch
>
>
> In order to implement switch locality in MapReduce, we need to have switch location in both the namenode and job tracker.  Currently the namenode asks the data nodes for this info and they run a local script to answer this question.  In our environment and others that I know of there is no reason to push this to each node.  It is easier to maintain a centralized script that maps node DNS names to switch strings.
> I propose that we build a new class that caches known DNS name to switch mappings and invokes a loadable class or a configurable system call to resolve unknown DNS to switch mappings.  We can then add this to the namenode to support the current block to switch mapping needs and simplify the data nodes.  We can also add this same callout to the job tracker and then implement rack locality logic there without needing to chane the filesystem API or the split planning API.
> Not only is this the least intrusive path to building racklocal MR I can ID, it is also future compatible to future infrastructures that may derive topology on the fly, etc, etc...

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-1985:
--------------------------------

    Attachment: 1985.v3.patch

Fixed an issue in the Scan benchmark.

> Abstract node to switch mapping into a topology service class used by namenode and jobtracker
> ---------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-1985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1985
>             Project: Hadoop
>          Issue Type: New Feature
>            Reporter: eric baldeschwieler
>            Assignee: Devaraj Das
>             Fix For: 0.16.0
>
>         Attachments: 1985.new.patch, 1985.v1.patch, 1985.v2.patch, 1985.v3.patch
>
>
> In order to implement switch locality in MapReduce, we need to have switch location in both the namenode and job tracker.  Currently the namenode asks the data nodes for this info and they run a local script to answer this question.  In our environment and others that I know of there is no reason to push this to each node.  It is easier to maintain a centralized script that maps node DNS names to switch strings.
> I propose that we build a new class that caches known DNS name to switch mappings and invokes a loadable class or a configurable system call to resolve unknown DNS to switch mappings.  We can then add this to the namenode to support the current block to switch mapping needs and simplify the data nodes.  We can also add this same callout to the job tracker and then implement rack locality logic there without needing to chane the filesystem API or the split planning API.
> Not only is this the least intrusive path to building racklocal MR I can ID, it is also future compatible to future infrastructures that may derive topology on the fly, etc, etc...

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12554686 ]

Devaraj Das commented on HADOOP-1985:
-------------------------------------

I ran the Scan benchmark attached in the patch (the benchmark just scans inputs; no sort/shuffle/reduce).

The input data was generated on a cluster of  ~300 machines. Randomwriter with the following config was run - 40 maps per host, each map configured to generate 1G, dfs blk size 256 MB. The input data set was thus around 11.6 TB.

Another cluster of ~900 nodes, with its dfs pointing to the earlier 300 node cluster, was used to run the Scan benchmark. The number of maps was equal to the number of dfs blocks in the input.

The two clusters had common racks but no common nodes. With the rack aware patch, the scan program took 25 minutes (with 90% rack-local tasks), and without the patch, the scan took around 35 minutes, ~30% improvement.

> Abstract node to switch mapping into a topology service class used by namenode and jobtracker
> ---------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-1985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1985
>             Project: Hadoop
>          Issue Type: New Feature
>            Reporter: eric baldeschwieler
>            Assignee: Devaraj Das
>             Fix For: 0.16.0
>
>         Attachments: 1985.new.patch, 1985.v1.patch, 1985.v2.patch, 1985.v3.patch
>
>
> In order to implement switch locality in MapReduce, we need to have switch location in both the namenode and job tracker.  Currently the namenode asks the data nodes for this info and they run a local script to answer this question.  In our environment and others that I know of there is no reason to push this to each node.  It is easier to maintain a centralized script that maps node DNS names to switch strings.
> I propose that we build a new class that caches known DNS name to switch mappings and invokes a loadable class or a configurable system call to resolve unknown DNS to switch mappings.  We can then add this to the namenode to support the current block to switch mapping needs and simplify the data nodes.  We can also add this same callout to the job tracker and then implement rack locality logic there without needing to chane the filesystem API or the split planning API.
> Not only is this the least intrusive path to building racklocal MR I can ID, it is also future compatible to future infrastructures that may derive topology on the fly, etc, etc...

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.