[HOD] "getServiceAddr addr mapred(hdfs): not found" errors

[Jik-Soo Kim]

Dear whom it may concern,

Hi, I am a novice user of Hadoop and Hadoop On Demand (HOD) system.
Currently I am using Hadoop version of 0.20.1 (I also tested with 0.18.3
version) and 4 nodes
including a head and three compute nodes. I already installed torque
resource manager
and tested it on top of my small cluster.

When I statically configured Hadoop cluster, HDFS and MapReduce could be run
without
any major problems. However, when I tried to set-up a HOD cluster, I ran
into a problem where
HDFS and MapReduce services could not be found (or started). Here is a
portion of the log-file
of Ringmaster node.

****************************************************************************************************************************************
[2009-10-12 17:55:17,534] DEBUG/10 ringMaster:567 - Getting service ID.
[2009-10-12 17:55:17,534] DEBUG/10 ringMaster:571 - Got service ID:
84.master
[2009-10-12 17:55:17,536] DEBUG/10 ringMaster:589 - Service registry @
http://master:59753
[2009-10-12 17:55:17,538] DEBUG/10 ringMaster:712 - Returning Hadoop
directory as: /home/hadoop/hadoop-0.18.3
[2009-10-12 17:55:17,540] DEBUG/10 util:215 - Executing command
/home/hadoop/hadoop-0.18.3/bin/hadoop version to find hadoop version
[2009-10-12 17:55:19,817] DEBUG/10 util:224 - Version from hadoop command:
Hadoop 0.18.3
[2009-10-12 17:55:19,820] DEBUG/10 ringMaster:117 - Using max-connect value
30
[2009-10-12 17:55:19,822] INFO/20 ringMaster:61 - Twisted interface not
found. Using hodXMLRPCServer.
[2009-10-12 17:55:19,830] DEBUG/10 ringMaster:73 - Ringmaster RPC Server at
50804
[2009-10-12 17:55:19,831] DEBUG/10 ringMaster:651 - Download not set.
[2009-10-12 17:55:19,832] DEBUG/10 ringMaster:654 - hadoop 84.master
slave3.local ringmaster hod
[2009-10-12 17:55:19,844] DEBUG/10 ringMaster:666 - Registered with serivce
registry: http://master:59753.
[2009-10-12 17:55:19,846] DEBUG/10 ringMaster:712 - Returning Hadoop
directory as: /home/hadoop/hadoop-0.18.3
[2009-10-12 17:55:19,846] DEBUG/10 ringMaster:686 -
hadoopdir=/home/hadoop/hadoop-0.18.3, java-home=/usr/local/jdk1.6.0_16
[2009-10-12 17:55:19,849] DEBUG/10 util:215 - Executing command
/home/hadoop/hadoop-0.18.3/bin/hadoop version to find hadoop version
[2009-10-12 17:55:20,437] DEBUG/10 ringMaster:479 - getServiceAddr name:
hdfs
[2009-10-12 17:55:20,440] DEBUG/10 ringMaster:487 - getServiceAddr service:
<hodlib.GridServices.hdfs.Hdfs instance at 0xe0ab878>
[2009-10-12 17:55:20,442] DEBUG/10 ringMaster:504 - getServiceAddr addr
hdfs: not found
[2009-10-12 17:55:21,429] DEBUG/10 ringMaster:479 - getServiceAddr name:
hdfs
[2009-10-12 17:55:21,430] DEBUG/10 ringMaster:487 - getServiceAddr service:
<hodlib.GridServices.hdfs.Hdfs instance at 0xe0ab878>
[2009-10-12 17:55:21,430] DEBUG/10 ringMaster:504 - getServiceAddr addr
hdfs: not found
[2009-10-12 17:55:22,157] DEBUG/10 util:224 - Version from hadoop command:
Hadoop 0.18.3
[2009-10-12 17:55:22,158] DEBUG/10 ringMaster:693 - starting jt monitor
[2009-10-12 17:55:22,160] DEBUG/10 ringMaster:909 - Entered start method.
[2009-10-12 17:55:22,161] DEBUG/10 ringMaster:920 -
/home/hadoop/hadoop-0.18.3/contrib/hod/bin/hodring
--hodring.tarball-retry-initial-time 1.0 --hodring.cmd-retry-initial-time
2.0 --hodring.cmd-retry-interval 2.0 --hodring.service-id 84.master
--hodring.temp-dir /tmp/hod --hodring.http-port-range 8000-9000
--hodring.userid hadoop --hodring.java-home /usr/local/jdk1.6.0_16
--hodring.svcrgy-addr master:59753 --hodring.tarball-retry-interval 3.0
--hodring.log-dir /home/hadoop/hadoop-0.18.3/contrib/hod/logs
--hodring.mapred-system-dir-root /mapredsystem --hodring.xrs-port-range
32768-65536 --hodring.debug 4 --hodring.ringmaster-xrs-addr slave3:50804
--hodring.register
[2009-10-12 17:55:22,163] DEBUG/10 torque:147 - pbsdsh command:
/usr/local/bin/pbsdsh /home/hadoop/hadoop-0.18.3/contrib/hod/bin/hodring
--hodring.tarball-retry-initial-time 1.0 --hodring.cmd-retry-initial-time
2.0 --hodring.cmd-retry-interval 2.0 --hodring.service-id 84.master
--hodring.temp-dir /tmp/hod --hodring.http-port-range 8000-9000
--hodring.userid hadoop --hodring.java-home /usr/local/jdk1.6.0_16
--hodring.svcrgy-addr master:59753 --hodring.tarball-retry-interval 3.0
--hodring.log-dir /home/hadoop/hadoop-0.18.3/contrib/hod/logs
--hodring.mapred-system-dir-root /mapredsystem --hodring.xrs-port-range
32768-65536 --hodring.debug 4 --hodring.ringmaster-xrs-addr slave3:50804
--hodring.register
[2009-10-12 17:55:22,164] DEBUG/10 ringMaster:479 - getServiceAddr name:
mapred
[2009-10-12 17:55:22,164] DEBUG/10 ringMaster:487 - getServiceAddr service:
<hodlib.GridServices.mapred.MapReduce instance at 0xe0ab908>
[2009-10-12 17:55:22,166] DEBUG/10 ringMaster:925 - Returned from
runWorkers.
[2009-10-12 17:55:22,167] DEBUG/10 ringMaster:504 - getServiceAddr addr
mapred: not found
[2009-10-12 17:55:22,439] DEBUG/10 ringMaster:479 - getServiceAddr name:
hdfs
...........
****************************************************************************************************************************************

And the following is the "hodrc" configuration file.


****************************************************************************************************************************************
[hod]
stream                          = True
java-home                       = /usr/local/jdk1.6.0_16/
cluster                         = hadoop-cluster
cluster-factor                  = 1.8
xrs-port-range                  = 32768-65536
debug                           = 3
allocate-wait-time              = 3600
temp-dir                        = /tmp/hod
[ringmaster]
register                        = True
stream                          = False
temp-dir                        = /tmp/hod
http-port-range                 = 8000-9000
work-dirs                       = /tmp/hod/1,/tmp/hod/2
xrs-port-range                  = 32768-65536
debug                           = 4
[hodring]
stream                          = False
temp-dir                        = /tmp/hod
register                        = True
java-home                       = /usr/local/jdk1.6.0_16/
http-port-range                 = 8000-9000
xrs-port-range                  = 32768-65536
debug                           = 3
[resource_manager]
queue                           = hadoopQ
batch-home                      = /usr/local/
id                              = torque
#env-vars                       = HOD_PYTHON_HOME=/usr/local/bin/python
[gridservice-mapred]
external                        = False
pkgs                            = /usr/local/hadoop-0.20.1/
tracker_port                    = 8030
info_port                       = 50080
[gridservice-hdfs]
external                        = False
pkgs                            = /usr/local/hadoop-0.20.1/
fs_port                         = 8020
info_port                       = 50070

****************************************************************************************************************************************

It looks like ringMaster cannot locate both of hdfs and mapred service
addresses. If you have any idea of this, please
let me know and any help will be greatly appreciated.

Best regards,

Jik-Soo Kim