tag:old.nabble.com,2006:forum-30592Nabble - Hadoop Core2009-12-15T03:36:10Ztag:old.nabble.com,2006:post-26793409File _partition.lst does not exist.2009-12-15T03:36:10Z2009-12-15T03:36:10ZafarsekHi,
<br>I'm using the InputSampler.RandomSampler to perform a partition sampling. It should create a file called _partition.lst that should be use later on by the partitionner class.
<br><br>For some reason it doesn't work and I get a
<br>java.io.FileNotFoundException: File _partition.lst does not exist.
<br>Below the code: it consists of a mapper only job, taking as input a file in a SequenceFileInputFormat that was generated by a previous job.
<br><br>Thanks a lot in advance for any insights.
<br><br>public class WordCountSorted {
<br><br> public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, IntWritable, Text> {
<br> // private final static IntWritable one = new IntWritable(1);
<br> private Text word = new Text();
<br><br> public void map(LongWritable key, Text value, OutputCollector<IntWritable, Text> output, Reporter reporter) throws IOException {
<br> String line = value.toString();
<br> String[] tokens = line.split("\t");
<br> int nbOccurences = Integer.parseInt(tokens[1]);
<br> word.set(tokens[0]);
<br> output.collect(new IntWritable(nbOccurences),word );
<br> }
<br> }
<br><br> public static void main(String[] args) throws Exception {
<br> JobConf conf = new JobConf(WordCountSorted.class);
<br> conf.setJobName("wordcountsorted");
<br><br> FileInputFormat.setInputPaths(conf, new Path(args[0]));
<br> FileOutputFormat.setOutputPath(conf, new Path(args[1]));
<br><br>
<br> conf.setInputFormat(SequenceFileInputFormat.class);
<br>
<br> conf.setOutputKeyClass(IntWritable.class);
<br> conf.setOutputValueClass(Text.class);
<br><br> conf.setMapperClass(Map.class);
<br> conf.setReducerClass(IdentityReducer.class);
<br><br><br> conf.setNumReduceTasks(2);
<br><br><br><br><br> InputSampler.Sampler<IntWritable, Text> sampler =
<br> new InputSampler.RandomSampler<IntWritable, Text>(0.1, 100, 10);
<br> InputSampler.writePartitionFile(conf, sampler);
<br><br> conf.setPartitionerClass(TotalOrderPartitioner.class);
<br><br> JobClient.runJob(conf);
<br> }
<br>}<p>From forum: <a href="http://old.nabble.com/Hadoop-core-user-f30590.html" embed="fixTarget[30590]" target="_top" >Hadoop core-user</a></p>tag:old.nabble.com,2006:post-267744610.18.3-14.cloudera.CH0_3 RHEL5: hadoop-site.xml properties not being read when formatting the Hadoop namenode2009-12-14T00:38:43Z2009-12-14T00:38:43ZDavid StemmerI'm trying to set up a basic Hadoop single-node cluster on a Red Hat Enterprise Linux 5 system. The Hadoop version is 0.18.3-14.cloudera.CH0_3. Unfortunately I am having problems getting Hadoop to read the configuration properties from hadoop-site.xml. Specifically, when I try to run the following command:
<br><br>${HADOOP_HOME}/bin/hadoop namenode -format
<br><br>...it seems that Hadoop is not using the storage directory that I have configured for HDFS. I get the following output:
<br><br>---------------------------------------------------------------------
<br>---------------------------------------------------------------------
<br><br> 09/12/14 02:24:14 INFO dfs.NameNode: STARTUP_MSG:
<br> /************************************************************
<br> STARTUP_MSG: Starting NameNode
<br> STARTUP_MSG: host = #####/###.##.#.##
<br> STARTUP_MSG: args = [-format]
<br> STARTUP_MSG: version = 0.18.3-14.cloudera.CH0_3
<br> STARTUP_MSG: build = -r HEAD; compiled by 'root' on Mon Jul 6 15:02:31 EDT 2009
<br> ************************************************************/
<br> Re-format filesystem in /tmp/hadoop-myusername/dfs/name ? (Y or N) Y
<br> 09/12/14 02:24:17 INFO fs.FSNamesystem: fsOwner=myusername,ugrad
<br> 09/12/14 02:24:17 INFO fs.FSNamesystem: supergroup=supergroup
<br> 09/12/14 02:24:17 INFO fs.FSNamesystem: isPermissionEnabled=true
<br> 09/12/14 02:24:17 INFO dfs.Storage: Image file of size 82 saved in 0 seconds.
<br> 09/12/14 02:24:17 INFO dfs.Storage: Storage directory /tmp/hadoop-myusername/dfs/name has been successfully formatted.
<br> 09/12/14 02:24:17 INFO dfs.NameNode: SHUTDOWN_MSG:
<br> /************************************************************
<br> SHUTDOWN_MSG: Shutting down NameNode at #####/###.##.#.##
<br> ************************************************************/
<br><br>---------------------------------------------------------------------
<br><br>I do not want it to format /tmp/hadoop-myusername/dfs/name as the storage directory. Here is what is in my hadoop-site.xml file. Notice that hadoop.tmp.dir is set /home/u/fall06/myusername/hadoop_tmp:
<br><br>---------------------------------------------------------------------
<br>---------------------------------------------------------------------
<br><br> <?xml version="1.0"?>
<br> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<br><br> <!-- Put site-specific property overrides in this file. -->
<br><br> <configuration>
<br><br> <property>
<br> <name>hadoop.tmp.dir</name>
<br> <value>/home/u/fall06/myusername/hadoop_tmp</value>
<br> </property>
<br>
<br> <property>
<br> <name>fs.default.name</name>
<br> <value>hdfs://localhost:50031</value>
<br> </property>
<br><br> <property>
<br> <name>mapred.job.tracker</name>
<br> <value>localhost:50032</value>
<br> </property>
<br><br> <property>
<br> <name>dfs.replication</name>
<br> <value>1</value>
<br> </property>
<br> </configuration>
<br><br>---------------------------------------------------------------------
<br><br><br>Here are the relevent lines of my .bash_profile where I set some of the Hadoop environment variables:
<br><br>---------------------------------------------------------------------
<br>---------------------------------------------------------------------
<br><br> export HADOOP_HOME=/home/u/fall06/myusername/Desktop/hadoop_project/hadoop
<br> export HADOOP_CONF_DIR=/home/u/fall06/myusername/Desktop/hadoop_project/hadoop/conf
<br><br>---------------------------------------------------------------------
<br><br>A few more details on the system setup:
<br><br>- I did not install Hadoop on the system myself; it was installed by a system administrator.
<br>- Because the default Hadoop directory is read/execute only, I copied the Hadoop directory to a folder that I own and ran in from there. My HADOOP_HOME enviromnent variable is set to this duplicate directory.
<br>- hadoop_site.xml is in my ${HADOOP_HOME}/conf directory.
<br>- I did try to echo what ${HADOOP_CONF_DIR} was being set to while '{HADOOP_HOME}/bin/hadoop namenode -format' was running. The result was this:
<br><br>---------------------------------------------------------------------
<br>---------------------------------------------------------------------
<br> /home/u/fall06/myusername/Desktop/hadoop_project/conf
<br>---------------------------------------------------------------------
<br><br>That's what I would expect, so I'm stumped. Does anyone have any ideas what I might be doing wrong, or know of more tests I can run to figure out the problem? Any and all advice is appreciated.<p>From forum: <a href="http://old.nabble.com/Hadoop-core-user-f30590.html" embed="fixTarget[30590]" target="_top" >Hadoop core-user</a></p>tag:old.nabble.com,2006:post-26694569Re: multiple file input2009-12-08T06:18:38Z2009-12-08T06:18:38Zlaser08150815<blockquote class="quote light-black dark-border-color"><div class="quote light-border-color">
<div class="quote-author" style="font-weight: bold;">pmg wrote:</div>
<div class="quote-message shrinkable-quote">I am evaluating hadoop for a problem that do a Cartesian product of input from one file of 600K (File A) with another set of file set (FileB1, FileB2, FileB3) with 2 millions line in total.
<br><br>Each line from FileA gets compared with every line from FileB1, FileB2 etc. etc. FileB1, FileB2 etc. are in a different input directory
<br><br>So....
<br><br>Two input directories
<br><br>1. input1 directory with a single file of 600K records - FileA
<br>2. input2 directory segmented into different files with 2Million records - FileB1, FileB2 etc.
<br><br>How can I have a map that reads a line from a FileA in directory input1 and compares the line with each line from input2?
<br><br>What is the best way forward? I have seen plenty of examples that maps each record from single input file and reduces into an output forward.
<br><br>thanks
</div>
</div></blockquote>
<br>I had a similar problem and solved it by writing a custom InputFormat (see attachment). You should improve the methods ACrossBInputSplit.getLength , ACrossBRecordReader.getPos and ACrossBRecordReader.getProgress.<p>From forum: <a href="http://old.nabble.com/Hadoop-core-user-f30590.html" embed="fixTarget[30590]" target="_top" >Hadoop core-user</a></p>tag:old.nabble.com,2006:post-26683812Hadoop / Data Migration Specialist Needed2009-12-07T12:43:04Z2009-12-07T12:43:04ZalevinHello. My name is Alex Levin and I am the COO of Brilig (www.brilig.com) a startup in New York focused on the online advertising space. We are looking to hire a Hadoop / Data Migration Specialist to play a crucial role in converting new client's data onto Brilig's service platforms. We are looking for a talented and energetic full-time freelance programmer to work both remotely and onsite at our Manhattan location. The Specialist will work with our clients' technical teams to determine the optimal formats and requirements to create files for subsequent import into a Brilig distributed file system during the implementation process. This process involves extracting, scrubbing, combining, transforming, validating and importing large data tables into final data sets suitable for loading into Brilig's defined databases. The Specialist will be responsible for creating/editing the database structure and writing of all import scripts and programs. The ability to work on multiple projects simultaneously while meeting tight deadlines is critical. This position is initially contract but may eventually lead to a full time position in our fun and exciting startup. Must be able to travel to client meetings and work independently.
<br><br>Responsibilities:
<br>- Subject Matter Expert on software tools used in the entire data migration process from extraction to validation and load
<br>- Design, develop and execute quality data movement processes that are consistent, repeatable and scalable
<br><br>- Streamline testing, audit and validation processes through data scrubbing routines and presentation of audit reports prior to load
<br>- Roll out newly developed processes via documentation and training
<br>- Maintain and manage a template library of executed solutions to leverage against future opportunities
<br>- Identify, clarify, and resolve issues and risks, escalating them as needed
<br>- Build and nourish strong business relationships with external clients
<br><br>Please include:
<br><br>Salary Requirements
<br>Availability
<br><br>Experience:
<br><br>At least 2 years experience with Htable, HBase and Hadoop
<br>At least 3-5 years experience in the development of java applications
<br>Use of XML and other protocols for data exchange between systems
<br>SQL database design and implementation
<br>Experience with Eclipse, Maven, and SVN a plus
<br>Excellent communication skills with both technical and non-technical colleagues
<br>Upper management and client facing skills
<br>Interest in keeping up with technology advances
<br><br>PLEASE NOTE:
<br>US citizens and Green Card Holders and those authorized to work in the US only. We are unable to sponsor or transfer H-1B candidates.
<br><br>Contact:
<br>Alex Levin, COO
<br>Brilig
<br>alevin@brilig.com<p>From forum: <a href="http://old.nabble.com/Hadoop-core-user-f30590.html" embed="fixTarget[30590]" target="_top" >Hadoop core-user</a></p>tag:old.nabble.com,2006:post-26681351writing files to HDFS (from c++/pipes)2009-12-07T10:05:54Z2009-12-07T10:05:54ZhorsonHi,
<br>i want to write a file to hdfs, using hadoop pipes. can anyone tell me how to do that?
<br>Im using an external library that writes its output to disk, so probably i have to read that output and write it to the distributed filesystem?
<br>I found only FSDataOutputStream, a java class.
<br>Can anyone help?
<br>Moreover, can anyone tell me where i can find goot documentation about hadoop pipes? Nearly everything i find is java specific or general information about MapReduce. I looked at the hadoop pipes source and it looked very restricted, can i do everything in hadoop pipes that possible in java?
<br><br>Thank for your help
<br>horson<p>From forum: <a href="http://old.nabble.com/Hadoop-core-user-f30590.html" embed="fixTarget[30590]" target="_top" >Hadoop core-user</a></p>tag:old.nabble.com,2006:post-26635758Start Hadoop env using JAVA or HADOOP APIs (InProcess)2009-12-03T20:25:47Z2009-12-03T20:25:47ZsamuellawrenceHai,
<br><br>I have to start the HADOOP environment using java code (inprocess). I would like to use the APIs to start it.
<br><br>Could anyone please give me snippet or a link.
<br><br>Thanks in Advance.<p>From forum: <a href="http://old.nabble.com/Hadoop-core-user-f30590.html" embed="fixTarget[30590]" target="_top" >Hadoop core-user</a></p>tag:old.nabble.com,2006:post-26605201Re: About Hadoop pseudo distribution2009-12-02T00:30:40Z2009-12-02T00:30:40ZDoss_IPHHi
<br>Hadoop pseudo distribution runs five java processors which are given below
<br> 1. namenode,
<br> 2. secondarynamenode
<br> 3. datanode
<br> 4 jobtracker
<br> 5. tasktracker
<br><br>As you know namenode, secondarynamenode, datanode processors are for HDFS and jobtracker ,tastktracker are for MR (Map Reduce).
<br><br><blockquote class="quote light-black dark-border-color"><div class="quote light-border-color">
<div class="quote-author" style="font-weight: bold;">kvorion wrote:</div>
<div class="quote-message shrinkable-quote">Hi All,
<br><br>I have been trying to set up a hadoop cluster on a number of machines, a few of which are multicore machines. I have been wondering whether the hadoop pseudo distribution is something that can help me take advantage of the multiple cores on my machines. All the tutorials say that the pseudo distribution mode lets you start each daemon in a separate java process. I have the following configuration settings for hadoop-site.xml:
<br><br><property>
<br> <name>fs.default.name</name>
<br> <value>hdfs://athena:9000</value>
<br></property>
<br><br><property>
<br> <name>mapred.job.tracker</name>
<br> <value>athena:9001</value>
<br></property>
<br><br><property>
<br> <name>dfs.replication</name>
<br> <value>2</value>
<br></property>
<br><br>I am not sure if this is really running in the pseudo-distribution mode. Are there any indicators or outputs that confirm what mode you are running in?
<br></div>
</div></blockquote>
<p>From forum: <a href="http://old.nabble.com/Hadoop-core-user-f30590.html" embed="fixTarget[30590]" target="_top" >Hadoop core-user</a></p>tag:old.nabble.com,2006:post-26588090Help! FSNamesystem initialization failed.2009-12-01T00:51:25Z2009-12-01T00:51:25ZSergio PuchiniHi, suddenly I’ve got problem with starting Namenode:
<br><br>[hadoop@hadoop1 hadoop]$ bin/hadoop namenode
<br>09/12/01 11:26:51 INFO namenode.NameNode: STARTUP_MSG:
<br>/************************************************************
<br>STARTUP_MSG: Starting NameNode
<br>STARTUP_MSG: host = hadoop1.bcr.ru/81.68.243.18
<br>STARTUP_MSG: args = []
<br>STARTUP_MSG: version = 0.19.2-dev
<br>STARTUP_MSG: build = <a href="http://svn.apache.org/repos/asf/hadoop/core/branches/bran" target="_top" rel="nofollow">http://svn.apache.org/repos/asf/hadoop/core/branches/bran</a><br>ch-0.19 -r 755955; compiled by 'maksim07' on Sun Mar 22 14:29:37 MSK 2009
<br>************************************************************/
<br>09/12/01 11:26:51 INFO metrics.RpcMetrics: Initializing RPC Metrics with hostName=NameNode, port=9000
<br>09/12/01 11:26:51 INFO namenode.NameNode: Namenode up at: hadoop1.bcr.ru/81.68.243.18:9000
<br>09/12/01 11:26:51 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=NameNode, sessionId=null
<br>09/12/01 11:26:51 INFO metrics.NameNodeMetrics: Initializing NameNodeMeterics using context object:org.apache.hadoop.metrics.spi.NullContext
<br>09/12/01 11:26:52 INFO namenode.FSNamesystem: fsOwner=hadoop,hadoop,dba
<br>09/12/01 11:26:52 INFO namenode.FSNamesystem: supergroup=supergroup
<br>09/12/01 11:26:52 INFO namenode.FSNamesystem: isPermissionEnabled=true
<br>09/12/01 11:26:52 INFO metrics.FSNamesystemMetrics: Initializing FSNamesystemMet
<br>rics using context object:org.apache.hadoop.metrics.spi.NullContext
<br>09/12/01 11:26:52 INFO namenode.FSNamesystem: Registered FSNamesystemStatusMBean
<br>09/12/01 11:26:52 INFO common.Storage: Number of files = 30079
<br>09/12/01 11:26:53 INFO common.Storage: Number of files under construction = 0
<br>09/12/01 11:26:53 INFO common.Storage: Image file of size 4451066 loaded in 1 seconds.
<br>09/12/01 11:27:11 ERROR namenode.FSNamesystem: FSNamesystem initialization failed.
<br>java.io.IOException: Incorrect data format. logVersion is -18 but writables.length is 0.
<br> at org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.java:542)
<br> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:973)
<br> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:793)
<br> at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:352)
<br> at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87)
<br> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:309)
<br> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:288)
<br> at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:163)
<br> at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:208)
<br> at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:194)
<br> at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:859)
<br> at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:868)
<br>09/12/01 11:27:11 INFO ipc.Server: Stopping server on 9000
<br>09/12/01 11:27:11 ERROR namenode.NameNode: java.io.IOException: Incorrect data f
<br>ormat. logVersion is -18 but writables.length is 0.
<br> at org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.java:542)
<br> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:973)
<br> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:793)
<br> at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:352)
<br> at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87)
<br> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:309)
<br> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:288)
<br> at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:163)
<br> at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:208)
<br> at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:194)
<br> at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:859)
<br> at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:868)
<br>09/12/01 11:27:11 INFO namenode.NameNode: SHUTDOWN_MSG:
<br>/************************************************************
<br>SHUTDOWN_MSG: Shutting down NameNode at hadoop1.bcr.ru/81.68.243.18
<br>************************************************************/
<br><br>Any ideas?
<br><br>Thanks!
<br>Sergio Puchini.
<br><br><p>From forum: <a href="http://old.nabble.com/Hadoop-core-user-f30590.html" embed="fixTarget[30590]" target="_top" >Hadoop core-user</a></p>tag:old.nabble.com,2006:post-26574876How to change the MapReduce output coder?2009-11-30T05:41:18Z2009-11-30T05:41:18ZgpcusterI know the default output coder is <b>utf-8</b>, but how can we change it into a other coder? like gbk?
<br><br>thx.<p>From forum: <a href="http://old.nabble.com/Hadoop-core-user-f30590.html" embed="fixTarget[30590]" target="_top" >Hadoop core-user</a></p>tag:old.nabble.com,2006:post-26558264hadoop-0.20.1\src\hdfs\org\apache\hadoop\hdfs\server2009-11-28T16:11:34Z2009-11-28T16:11:34Zdzisaacswhy is this folder called "server"
<br><br>what is this folder for?<p>From forum: <a href="http://old.nabble.com/Hadoop-core-user-f30590.html" embed="fixTarget[30590]" target="_top" >Hadoop core-user</a></p>tag:old.nabble.com,2006:post-26491406Re: Using hadoop for Matrix Multiplication in NFS?2009-11-23T23:17:59Z2009-11-23T23:17:59ZMartin Mituzasii) I once run the mapred program DistCp to copy data from NFS into HDFS. I mount the file system directory to each node. Thus it can be connected.
<br>
<br><blockquote class="quote light-black dark-border-color"><div class="quote light-border-color">
<div class="quote-author" style="font-weight: bold;">Gimick wrote:</div>
<div class="quote-message shrinkable-quote">Hi, I am new to hadoop. I am planning to do matrix multiplication(of order millions) using hadoop.
<br><br>I have a few queries regarding the above.
<br><br>i) Will using hadoop be a fix for this or should I try some other approaches?
<br>ii) I will be using it in NFS. Will using hadoop still be a good option?
<br><br>If I can use hadoop for this problem, could you plz send links to configure hadoop-site.xml file for a nfs system.
<br><br>P.S. I tried a few setup instructions via search, but everything seems to give "Unable to connect to ...." error.
</div>
</div></blockquote>
<p>From forum: <a href="http://old.nabble.com/Hadoop-core-user-f30590.html" embed="fixTarget[30590]" target="_top" >Hadoop core-user</a></p>tag:old.nabble.com,2006:post-26443615PIG bin/labeling relation2009-11-20T10:17:59Z2009-11-20T10:17:59Zdrd_I am using PIG and this is what I am trying to do:
<br><br>1) Sort a relation A into B by a field x. The smallest value of x is first. Just use SORT.
<br><br>2) Label each tuple in B with a number denoting its order in the sorted relation. So the first tuple would be labeled with a 1, the second tuple with a 2, the third with a 3 and so on. Not certain how to do this.
<br><br>3) Derive a relation C where each row is a bag of tuples. The first row contains the first n1 tuples from relation B, the second row contains the tuples from B labeled (n1 + 1) to n2 from, the third row contains the tuples from B labeled (n2 + 1) to n3 and so on to n100. This step is simple (just use filter) once we've labeled each tuple in B with a number.
<br><br>The question: how do I do step 2).
<br><br>thanks<p>From forum: <a href="http://old.nabble.com/Hadoop-core-user-f30590.html" embed="fixTarget[30590]" target="_top" >Hadoop core-user</a></p>tag:old.nabble.com,2006:post-26332382Using hadoop for Matrix Multiplication in NFS?2009-11-12T23:21:50Z2009-11-12T23:21:50ZGimickHi, I am new to hadoop. I am planning to do matrix multiplication(of order millions) using hadoop.
<br><br>I have a few queries regarding the above.
<br><br>i) Will using hadoop be a fix for this or should I try some other approaches?
<br>ii) I will be using it in NFS. Will using hadoop still be a good option?
<br><br>If I can use hadoop for this problem, could you plz send links to configure hadoop-site.xml file for a nfs system.
<br><br>P.S. I tried a few setup instructions via search, but everything seems to give "Unable to connect to ...." error.
<br><p>From forum: <a href="http://old.nabble.com/Hadoop-core-user-f30590.html" embed="fixTarget[30590]" target="_top" >Hadoop core-user</a></p>tag:old.nabble.com,2006:post-26322618Help !! Hadoop installation to One machine has 24 CPU 16 disk (Each one 2 TB)2009-11-12T09:17:01Z2009-11-12T09:17:01ZdgokerHi
<br><br>I installed the hadoop to one server which has following configurations
<br><br> 24 CPU,
<br> 72 GB RAM
<br> 17 Disk (2 TB)
<br><br>
<br>All configuration belongs to Hadoop and Pig are is default settings. ın order to run process efficiently waht should be the following configuration settings. The settings i find on forums usually 4 CPU machines and clustered system.
<br><br><br>What do you suggest me following settings?
<br><br><br>mapred.tasktracker.reduce.tasks.maximum ?
<br>mapred.map.tasks ?
<br>mapred.reduce.tasks ?
<br>dfs.datanode.handler.count ?
<br><br><br><p>From forum: <a href="http://old.nabble.com/Hadoop-core-user-f30590.html" embed="fixTarget[30590]" target="_top" >Hadoop core-user</a></p>tag:old.nabble.com,2006:post-26322437java code - parsing string -outofindex error -File on HDFS2009-11-12T09:05:50Z2009-11-12T09:05:50ZdgokerWhen i try to read a file from HDFS, i want to substring the record by java code (javada line.substring(1,9)) but i am getting outofindex error. When i do the same operation on same file by PIG code there is no problem.
<br><br>I took following error, What shoul i do in Java.
<br><br><br>java.lang.StringIndexOutOfBoundsException: String index out of range: 9
<br> at java.lang.String.substring(String.java:1934)
<br> at WordCount$Map.map(WordCount.java:19)
<br> at WordCount$Map.map(WordCount.java:12)
<br> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
<br> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
<br> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
<br> at org.apache.hadoop.mapred.Child.main(Child.java:170)
<br><p>From forum: <a href="http://old.nabble.com/Hadoop-core-dev-f30593.html" embed="fixTarget[30593]" target="_top" >Hadoop core-dev</a></p>tag:old.nabble.com,2006:post-26322382About Hadoop pseudo distribution2009-11-12T09:02:57Z2009-11-12T09:02:57ZkvorionHi All,
<br><br>I have been trying to set up a hadoop cluster on a number of machines, a few of which are multicore machines. I have been wondering whether the hadoop pseudo distribution is something that can help me take advantage of the multiple cores on my machines. All the tutorials say that the pseudo distribution mode lets you start each daemon in a separate java process. I have the following configuration settings for hadoop-site.xml:
<br><br><property>
<br> <name>fs.default.name</name>
<br> <value>hdfs://athena:9000</value>
<br></property>
<br><br><property>
<br> <name>mapred.job.tracker</name>
<br> <value>athena:9001</value>
<br></property>
<br><br><property>
<br> <name>dfs.replication</name>
<br> <value>2</value>
<br></property>
<br><br>I am not sure if this is really running in the pseudo-distribution mode. Are there any indicators or outputs that confirm what mode you are running in?
<br><br><p>From forum: <a href="http://old.nabble.com/Hadoop-core-user-f30590.html" embed="fixTarget[30590]" target="_top" >Hadoop core-user</a></p>tag:old.nabble.com,2006:post-26305272Re: Could not find any valid local directory for taskTracker2009-11-11T09:33:28Z2009-11-11T09:33:28ZSaju K KHi,
<br><br>Did you get a solution for this problem ,we are facing a similar problem
<br><br>saju
<br><blockquote class="quote light-black dark-border-color"><div class="quote light-border-color">
<div class="quote-author" style="font-weight: bold;">Pallavi Palleti wrote:</div>
<div class="quote-message shrinkable-quote">Hi,
<br> I got below error while running my hadoop task. But, when I tried after few hours, it worked fine.
<br>Can some one please tell me why this error occured?
<br><br>ERROR Below:
<br><br>Error initializing task_200805161358_0158_m_000000_0:
<br>org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for taskTracker/jobcache/job_200805161358_0158/job.xml
<br> at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:313)
<br> at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
<br> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:590)
<br> at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1172)
<br> at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:836)
<br> at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:1208)
<br> at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:1924)
<br><br>08/05/29 23:30:35 WARN mapred.JobClient: Error reading task outputhttp://hadoop31:50060/tasklog?plaintext=true&taskid=task_200805161358_0158_m_000000_0&filter=stdout
<br>08/05/29 23:30:35 WARN mapred.JobClient: Error reading task outputhttp://hadoop31:50060/tasklog?plaintext=true&taskid=task_200805161358_0158_m_000000_0&filter=stderr
<br>08/05/29 23:30:37 INFO mapred.JobClient: map 83% reduce 0%
<br><br><br>Thanks
</div>
</div></blockquote>
<p>From forum: <a href="http://old.nabble.com/Hadoop-core-user-f30590.html" embed="fixTarget[30590]" target="_top" >Hadoop core-user</a></p>tag:old.nabble.com,2006:post-26266407Using jobtracker api for developing UI in hadoop 0.19.12009-11-09T05:28:06Z2009-11-09T05:28:06Zpnd Hi i am new to hadoop been using it for few weeks now, i tried few map/reduce example and could see logs in jobtracker.jsp.
<br>Is there any way to use jobtracker api(<a href="http://hadoop.apache.org/common/docs/r0.19.1/api/org/apache/hadoop/mapred/JobTracker.html" target="_top" rel="nofollow">http://hadoop.apache.org/common/docs/r0.19.1/api/org/apache/hadoop/mapred/JobTracker.html</a>) for tracking the jobs instead using jobtracker jsp because I am trying to build new UI to show the job progress according to my requirements.
<br><br>Thanks in advance.
<br><p>From forum: <a href="http://old.nabble.com/Hadoop-core-user-f30590.html" embed="fixTarget[30590]" target="_top" >Hadoop core-user</a></p>tag:old.nabble.com,2006:post-26142981Re: What if an XML file cross boundary of HDFS chunks?2009-10-31T07:38:57Z2009-10-31T07:38:57ZOliver Fischer-3-----BEGIN PGP SIGNED MESSAGE-----
<br>Hash: SHA1
<br><br>Hello Jeff,
<br><br>does it means, that there is no programmatic possibility to define where
<br>a logical file will be splitted independent of the distribution of it
<br>blocks in the HDFS?
<br><br>Regards
<br><br>Oliver
<br><br>Jeff Zhang schrieb:
<div class='shrinkable-quote'><br>> Hi Steve,
<br>>
<br>> When you want to read xml, you should provide your custom InputFormat which
<br>> extends FileInputFormat.
<br>>
<br>> and override the method isSplitable to not split a file , that means one xml
<br>> file for one mapper.
<br>>
<br>>
<br>> protected boolean isSplitable(FileSystem fs, Path filename) {
<br>> return false;
<br>> }
</div><br><br>- --
<br>Oliver B. Fischer, Schönhauser Allee 64, 10437 Berlin
<br>Tel. +49 30 44793251, Mobil: +49 178 7903538
<br>Mail: <a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26142981&i=0" target="_top" rel="nofollow">o.b.fischer@...</a> Blog: <a href="http://www.swe-blog.net" target="_top" rel="nofollow">http://www.swe-blog.net</a><br>-----BEGIN PGP SIGNATURE-----
<br>Version: GnuPG v1.4.9 (MingW32)
<br>Comment: Using GnuPG with Mozilla - <a href="http://enigmail.mozdev.org/" target="_top" rel="nofollow">http://enigmail.mozdev.org/</a><br><br>iQEcBAEBAgAGBQJK7EwBAAoJELeiwuwqd1DGO/wIAJl8wwf6uNgm/ZwsGh8M1xvz
<br>wSEH9sD2cfjUSV3rmpHndKEfSTEOeHvvaJmJn24K9HhB9w8QyDogAgHawCdBY2TE
<br>K27n4wqSGlbLpQz4XmKUOVtFSooeEPUT58Jn2aMAno+nrWHM7oq9tuCJAAYkBexV
<br>wCrc7eE+o55TlAlx+LDWWS9mJrdTNBYqzoHh0gnWsEGm98CWvzn08tNA/L2moJbQ
<br>HZwnWzfgEBKBwAZUOYLFt2GigIYN3GE0pMp33BgjWi91zPwGSk7Bcq7XhObLK7o/
<br>uYxS+s3BTkLy+R6ngjOW1NLvg6STX37FpFNZowDmPt8Bzd8GxAefnqcxkVcnb90=
<br>=t6vV
<br>-----END PGP SIGNATURE-----
<br><br><p>From forum: <a href="http://old.nabble.com/Hadoop-core-dev-f30593.html" embed="fixTarget[30593]" target="_top" >Hadoop core-dev</a></p>tag:old.nabble.com,2006:post-26123313Re: What if an XML file cross boundary of HDFS chunks?2009-10-29T17:42:37Z2009-10-29T17:42:37ZJeff Zhang-4Hi Steve,
<br><br>When you want to read xml, you should provide your custom InputFormat which
<br>extends FileInputFormat.
<br><br>and override the method isSplitable to not split a file , that means one xml
<br>file for one mapper.
<br><br><br> protected boolean isSplitable(FileSystem fs, Path filename) {
<br> return false;
<br> }
<br><br><br><br>Best Regards,
<br><br>Jeff zhang
<br><br><br><br>On Thu, Oct 29, 2009 at 12:32 PM, Steve Gao <<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26123313&i=0" target="_top" rel="nofollow">steve.gao@...</a>> wrote:
<br><div class='shrinkable-quote'><br>>
<br>> Does anybody have the similar issue? If you store XML files in HDFS, how
<br>> can you make sure a chunk reads by a mapper does not contain partial data of
<br>> an XML segment?
<br>>
<br>> For example:
<br>>
<br>> <title>
<br>> <book>book1</book>
<br>> <author>me</author>
<br>> ..............what if this is the boundary of a chunk?...................
<br>> <year>2009</year>
<br>> <book>book2</book>
<br>>
<br>> <author>me</author>
<br>>
<br>> <year>2009</year>
<br>> <book>book3</book>
<br>>
<br>> <author>me</author>
<br>>
<br>> <year>2009</year>
<br>> <title>
<br>>
<br>>
<br>>
<br>>
<br>>
<br>>
<br>>
<br></div><p>From forum: <a href="http://old.nabble.com/Hadoop-core-dev-f30593.html" embed="fixTarget[30593]" target="_top" >Hadoop core-dev</a></p>tag:old.nabble.com,2006:post-26120236What if an XML file cross boundary of HDFS chunks?2009-10-29T13:32:21Z2009-10-29T13:32:21Zmeili100<br>Does anybody have the similar issue? If you store XML files in HDFS, how can you make sure a chunk reads by a mapper does not contain partial data of an XML segment?
<br><br>For example:
<br><br><title>
<br><book>book1</book>
<br><author>me</author>
<br>..............what if this is the boundary of a chunk?...................
<br><year>2009</year>
<br><book>book2</book>
<br><br><author>me</author>
<br><br><year>2009</year>
<br><book>book3</book>
<br><br><author>me</author>
<br><br><year>2009</year>
<br><title>
<br><br><br><br>
<br><br><br><p>From forum: <a href="http://old.nabble.com/Hadoop-core-dev-f30593.html" embed="fixTarget[30593]" target="_top" >Hadoop core-dev</a></p>tag:old.nabble.com,2006:post-26018722Can I have multiple reducers?2009-10-22T16:17:18Z2009-10-22T16:17:18ZForhadoopHello,
In my application I need to reduce the original reducer output keys further.
I was reading about Chainreducer and Chainmappers but looks like it is for :
one or more mapper -> reducer -> 0 or more mappers
I need something like:
one or more mapper -> reducer -> reducer
Please help me figure out the best way to achieve it. Currently, the only options seems like I write another map reduce application and run it separately after the first map-reduce application. In this second application, the mapper will be dummy and won't do anything. The reducer will further club the first run outputs.
Any other comments such as this is not a good programming practice are welcome, so that I know I am in the wrong direction..<p>From forum: <a href="http://old.nabble.com/Hadoop-core-user-f30590.html" embed="fixTarget[30590]" target="_top" >Hadoop core-user</a></p>tag:old.nabble.com,2006:post-26003660Re: Need Info2009-10-21T20:49:16Z2009-10-21T20:49:16ZDoss_IPH<br>Hi,
<br> you can use this pseudo code for loading data to HDFS.
<br><br>import java.io.File;
<br>import java.net.URI;
<br>import org.apache.hadoop.conf.Configuration;
<br>import org.apache.hadoop.fs.FileStatus;
<br>import org.apache.hadoop.fs.FileSystem;
<br>import org.apache.hadoop.fs.Path;
<br>import org.apache.hadoop.hdfs.DistributedFileSystem;
<br><br>/**
<br> * @author: Arockia Doss S
<br> * @emailto: doss@intellipowerhive.com
<br> * @url: <a href="http://www.intellipowerhive.com,http://www.dossinfotech.com" target="_top" rel="nofollow">http://www.intellipowerhive.com,http://www.dossinfotech.com</a><br> * @comments: You can use and modify this code for your use.
<br> * @About this: This below code works in hadoop-0.19.0 version platform.
<br> * If you want to test this code, you have set the hadoop libraries in your class path.
<br> * You need to give set of parameters before running it (like Hadoop path, Host, Users).
<br> */
<br><br>public class HadoopConfiguration {
<br> //Hadoop Absolute Path
<br> private static final String CLUSTERPATH="/home/hadoop-0.19.0/";
<br> private static final String SITEFILE = "conf/hadoop-site.xml";
<br> private static final String DEFAULTFILE = "conf/hadoop-default.xml";
<br> //Hadoop Name Node Host
<br> private static final String HADOOPHOST = "192.168.1.11";
<br> //Hadoop Root and its users list
<br> private static final String HOSTUSERS = "root,doss";
<br> private static Configuration conf = new Configuration();
<br> private static DistributedFileSystem dfs = new DistributedFileSystem();
<br>
<br> public HadoopConfiguration() throws java.lang.Exception{
<br> Path sitepath = new Path(CLUSTERPATH+SITEFILE);
<br> Path defaultpath = new Path(CLUSTERPATH+DEFAULTFILE);
<br> getConf().set("fs.default.name","hdfs://"+HADOOPHOST+":9000/");
<br> getConf().addResource(sitepath);
<br> getConf().addResource(defaultpath);
<br> getConf().set("hadoop.job.ugi", HOSTUSERS);
<br> dfs.initialize(new URI("hdfs://"+HADOOPHOST+":9000/"), conf);
<br> }
<br>
<br> public static Configuration getConf(){
<br> return conf;
<br> }
<br>
<br> public static void main(String[] args){
<br> try{
<br> HadoopConfiguration h = new HadoopConfiguration();
<br> FileSystem fs = FileSystem.get(h.getConf());
<br>
<br> //Copy sample.xls file to HDFS, The file will be there after copying it.
<br> fs.copyFromLocalFile(new Path("/home/sample.xls"),new Path("/home/xls/"));
<br>
<br> //Move sample.doc file to HDFS, The file will not be there after moving it.
<br> fs.moveFromLocalFile(new Path("/home/sample.doc"),new Path("/home/doc/"));
<br>
<br> //This below code gives to list the files from HDFS
<br> FileStatus[] fileStatus = fs.listStatus(new Path("/home/xls"));
<br> for(int i=0;i<fileStatus.length;i++){
<br> Path path = fileStatus[i].getPath();
<br> }
<br>
<br> }catch(java.lang.Exception e){
<br> System.out.println(e);
<br> }
<br> }
<br>
<br>}
<br><br><br><br><quote author="shwitzu"><br>Thanks for Responding.
<br><br>I read about HDFS and understood how it works and I also installed hadoop in my windows using cygwin and tried a sample driver code and made sure it works.
<br><br>But my concern is, given the problem statement how should I proceed
<br><br>Could you please give me some clue/ pseudo code or a design.
<br><br>Thanks in anticipation.
<br><br><br><blockquote class="quote light-black dark-border-color"><div class="quote light-border-color">
<div class="quote-author" style="font-weight: bold;">Doss_IPH wrote:</div>
<div class="quote-message shrinkable-quote">First and for most, you need to understand about hadoop platform infrastructures.
<br>Currently, I am working in real time application using hadoop. I think that Hadoop will be fit to your requirements.
<br>Hadoop is mainly for three things,
<br>1. Scalability no limit for storage
<br>2. Peta bytes of data processing in distributed parallel mode.
<br>3. Fault tolerance (Automatically Block Replication) recovering data from failure.
<br><br><blockquote class="quote light-black dark-border-color"><div class="quote light-border-color">
<div class="quote-author" style="font-weight: bold;">shwitzu wrote:</div>
<div class="quote-message shrinkable-quote">Hello Sir!
<br><br>I am new to hadoop. I have a project based on webservices. I have my information in 4 databases with different files in each one of them. Say, images in one, video, documents etc. My task is to develop a web service which accepts the keyword from the client and process the request and send back the actual requested file back to the user. Now I have to use Hadoop distributed file system in this project.
<br><br>I have the following questions:
<br><br>1) How should I start with the design?
<br>2) Should I upload all the files and create Map, Reduce and Driver code and once I run my application will it automatically go the file system and get back the results to me?
<br>3) How do i handle the binary data? I want to store binary format data using MTOM in my databse.
<br><br>Please let me know how I should proceed. I dont know much about this hadoop and am I searching for some help. It would be great if you could assist me. Thanks again
</div>
</div></blockquote>
</quote></div>
</div></blockquote>
<p>From forum: <a href="http://old.nabble.com/Hadoop-core-user-f30590.html" embed="fixTarget[30590]" target="_top" >Hadoop core-user</a></p>tag:old.nabble.com,2006:post-25996385Regd. Hadoop Implementation2009-10-21T10:02:55Z2009-10-28T15:51:40ZshwitzuThanks for Responding,
<br><br>I read about HDFS and understood how it works and I also installed hadoop in my windows using cygwin and tried a sample driver code and made sure it works.
<br><br>But my concern is, given the problem statement how should I proceed
<br><br>Could you please give me some clue/ pseudo code or a design.
<br><br>Thanks in anticipation.
<br><br><br><quote author="Doss_IPH"><br>First and for most, you need to understand about hadoop platform infrastructures.
<br>Currently, I am working in real time application using hadoop. I think that Hadoop will be fit to your requirements.
<br>Hadoop is mainly for three things,
<br>1. Scalability no limit for storage
<br>2. Peta bytes of data processing in distributed parallel mode.
<br>3. Fault tolerance (Automatically Block Replication) recovering data from failure.
<br><br><blockquote class="quote light-black dark-border-color"><div class="quote light-border-color">
<div class="quote-author" style="font-weight: bold;">shwitzu wrote:</div>
<div class="quote-message shrinkable-quote">Hello Sir!
<br><br>I am new to hadoop. I have a project based on webservices. I have my information in 4 databases with different files in each one of them. Say, images in one, video, documents etc. My task is to develop a web service which accepts the keyword from the client and process the request and send back the actual requested file back to the user. Now I have to use Hadoop distributed file system in this project.
<br><br>I have the following questions:
<br><br>1) How should I start with the design?
<br>2) Should I upload all the files and create Map, Reduce and Driver code and once I run my application will it automatically go the file system and get back the results to me?
<br>3) How do i handle the binary data? I want to store binary format data using MTOM in my databse.
<br><br>Please let me know how I should proceed. I dont know much about this hadoop and am I searching for some help. It would be great if you could assist me. Thanks again
</div>
</div></blockquote>
</quote><br><p>From forum: <a href="http://old.nabble.com/Hadoop-core-user-f30590.html" embed="fixTarget[30590]" target="_top" >Hadoop core-user</a></p>tag:old.nabble.com,2006:post-25995151Re: Datanode Throwing NoRouteToHostException2009-10-21T08:46:18Z2009-10-21T08:46:18ZlazikidThanks very much for your reply and time. I found it was a firewall issue. After stopping iptables, everything worked fine.
<br><br>Thanks again.
<br><br><quote author="Doss_IPH"><br>Comment all the IPs in /etc/hosts first except localhost or 127.0.0.1 as i have given below. Then, give in conf/hadoop-site.xml
<br>fs.default.name parameter value is hdfs://hdfs://[Master Node IP]:9000/
<br>mapred.job.tracker parameter value is hdfs://[Job Tracker IP]:9001
<br>Make sure you have enabled ssh and sshd running without password.
<br><p>From forum: <a href="http://old.nabble.com/Hadoop-core-user-f30590.html" embed="fixTarget[30590]" target="_top" >Hadoop core-user</a></p>tag:old.nabble.com,2006:post-25986574Re: DataNode is shutting down2009-10-20T22:09:25Z2009-10-20T22:09:25ZDoss_IPHHi!
<br><br>Minimum HDFS replication must be 3.
<br><br><blockquote class="quote light-black dark-border-color"><div class="quote light-border-color">
<div class="quote-author" style="font-weight: bold;">yibo820217 wrote:</div>
<div class="quote-message shrinkable-quote">hi all,there is my problem.
<br>when add a datanode to hadoop,the way is;
<br>1.in namenode add the new datanode to conf/slave
<br><br>2.in new datanode cd $HADOOP_HOME then
<br>$ bin/hadoop-daemon.sh start datanode
<br>$ bin/hadoop-daemon.sh start tasktracker
<br><br>3.in namenode,
<br>$bin/hadoop balancer
<br><br>and the new datanode is added to hadoop,but a old datanode is shutdown
<br>and here is the logs in datanode which is shutdown
<br><br>2009-10-14 13:16:30,604 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 3 blocks got processed in 5 msecs
<br>2009-10-14 13:48:44,395 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeCommand action: DNA_REGISTER
<br>2009-10-14 13:48:47,402 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeCommand action: DNA_REGISTER
<br>2009-10-14 13:48:50,403 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeCommand action: DNA_REGISTER
<br>2009-10-14 13:48:53,407 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeCommand action: DNA_REGISTER
<br>2009-10-14 13:48:56,418 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeCommand action: DNA_REGISTER
<br>2009-10-14 13:48:59,415 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeCommand action: DNA_REGISTER
<br>2009-10-14 13:49:02,420 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: DataNode is shutting down: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.protocol.UnregisteredDatanodeException: Data node 100.207.100.33:50010 is attempting to report storage ID DS-1277539940-100.207.100.33-50010-1255486116525. Node 100.207.100.25:50010 is expected to serve this storage.
<br> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDatanode(FSNamesystem.java:3914)
<br> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.processReport(FSNamesystem.java:2885)
<br> at org.apache.hadoop.hdfs.server.namenode.NameNode.blockReport(NameNode.java:715)
<br> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
<br> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
<br> at java.lang.reflect.Method.invoke(Method.java:597)
<br> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
<br> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
<br> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
<br> at java.security.AccessController.doPrivileged(Native Method)
<br> at javax.security.auth.Subject.doAs(Subject.java:396)
<br> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
<br><br> at org.apache.hadoop.ipc.Client.call(Client.java:739)
<br> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
<br> at $Proxy4.blockReport(Unknown Source)
<br> at org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:756)
<br> at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1186)
<br> at java.lang.Thread.run(Thread.java:619)
<br><br>2009-10-14 13:49:02,527 INFO org.apache.hadoop.ipc.Server: Stopping server on 50020
<br>2009-10-14 13:49:02,528 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 50020: exiting
<br>2009-10-14 13:49:02,529 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
<br>2009-10-14 13:49:02,529 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for threadgroup to exit, active threads is 1
<br>2009-10-14 13:49:02,528 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 50020: exiting
<br>2009-10-14 13:49:02,528 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 50020: exiting
<br>2009-10-14 13:49:02,529 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(10.207.0.33:50010, storageID=DS-1277539940-100.207.100.33-50010-1255486116525, infoPort=50075, ipcPort=50020):DataXceiveServer: java.nio.channels.AsynchronousCloseException
<br> at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
<br> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
<br> at sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
<br> at org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:130)
<br> at java.lang.Thread.run(Thread.java:619)
<br><br>2009-10-14 13:49:02,530 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 50020
<br>2009-10-14 13:49:03,267 INFO org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Exiting DataBlockScanner thread.
<br>2009-10-14 13:49:03,530 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for threadgroup to exit, active threads is 0
<br>2009-10-14 13:49:03,635 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(10.207.0.33:50010, storageID=DS-1277539940-100.207.100.33-50010-1255486116525, infoPort=50075, ipcPort=50020):Finishing DataNode in: FSDataset{dirpath='/data0/hadoop/hadoopfs/data/current'}
<br>2009-10-14 13:49:03,635 INFO org.apache.hadoop.ipc.Server: Stopping server on 50020
<br>2009-10-14 13:49:03,635 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for threadgroup to exit, active threads is 0
<br>2009-10-14 13:49:03,636 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
<br>/************************************************************
<br>SHUTDOWN_MSG: Shutting down DataNode at cent52ip33/100.207.100.33
<br>************************************************************/
<br><br>----hdfs-core.xml----
<br>...
<br> <property>
<br> <name>dfs.replication</name>
<br> <value>1</value>
<br> </property>
<br>...
<br><br>can anybody help me please?
<br><br>Thanks!
<br><br>Darren.
<br><br><br></div>
</div></blockquote>
<p>From forum: <a href="http://old.nabble.com/Hadoop-core-user-f30590.html" embed="fixTarget[30590]" target="_top" >Hadoop core-user</a></p>tag:old.nabble.com,2006:post-25986550Re: Need Info2009-10-20T22:05:18Z2009-10-20T22:05:18ZDoss_IPHFirst and for most, you need to understand about hadoop platform infrastructures.
<br>Currently, I am working in real time application using hadoop. I think that Hadoop will be fit to your requirements.
<br>Hadoop is mainly for three things,
<br>1. Scalability no limit for storage
<br>2. Peta bytes of data processing in distributed parallel mode.
<br>3. Fault tolerance (Automatically Block Replication) recovering data from failure.
<br><br><blockquote class="quote light-black dark-border-color"><div class="quote light-border-color">
<div class="quote-author" style="font-weight: bold;">shwitzu wrote:</div>
<div class="quote-message shrinkable-quote">Hello Sir!
<br><br>I am new to hadoop. I have a project based on webservices. I have my information in 4 databases with different files in each one of them. Say, images in one, video, documents etc. My task is to develop a web service which accepts the keyword from the client and process the request and send back the actual requested file back to the user. Now I have to use Hadoop distributed file system in this project.
<br><br>I have the following questions:
<br><br>1) How should I start with the design?
<br>2) Should I upload all the files and create Map, Reduce and Driver code and once I run my application will it automatically go the file system and get back the results to me?
<br>3) How do i handle the binary data? I want to store binary format data using MTOM in my databse.
<br><br>Please let me know how I should proceed. I dont know much about this hadoop and am I searching for some help. It would be great if you could assist me. Thanks again
</div>
</div></blockquote>
<p>From forum: <a href="http://old.nabble.com/Hadoop-core-user-f30590.html" embed="fixTarget[30590]" target="_top" >Hadoop core-user</a></p>tag:old.nabble.com,2006:post-25986468Re: Datanode Throwing NoRouteToHostException2009-10-20T21:46:49Z2009-10-20T21:46:49ZDoss_IPHComment all the IPs in /etc/hosts first except localhost or 127.0.0.1 as i have given below. Then, give in conf/hadoop-site.xml
<br>fs.default.name parameter value is hdfs://hdfs://[Master Node IP]:9000/
<br>mapred.job.tracker parameter value is hdfs://[Job Tracker IP]:9001
<br>Make sure you have enabled ssh and sshd running without password.
<br><br><blockquote class="quote light-black dark-border-color"><div class="quote light-border-color">
<div class="quote-author" style="font-weight: bold;">lazikid wrote:</div>
<div class="quote-message shrinkable-quote">I need some help with setting up a Hadoop cluster. The datanode on the slave is not coming up throwing java.net.NoRouteToHostException: No route to host. Please see the details below.
<br><br>I have a centos master and a fedora slave. Both have Java 6 and Hadoop 0.20.1. I instaleed hadoop under /opt in both machines.
<br><br>The machines can ping one another using hostname and ip. They can also make password-less ssh connection to one another using both hostname and ip.
<br><br>This is what I have in the /etc/hosts file for the master :
<br><br>************************************************
<br>127.0.0.1 localhost.localdomain localhost
<br>#192.168.1.125 centos
<br>#192.168.1.102 fedora
<br>::1 localhost6.localdomain6 localhost6
<br>************************************************
<br><br>This is what I have in the /etc/hosts for the slave
<br><br>************************************************
<br>127.0.0.1 localhost.localdomain localhost
<br>#192.168.1.104 ubuntu64
<br>#192.168.1.102 fedora
<br>#192.168.1.125 centos
<br>::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
<br>************************************************
<br><br><br><br>These are my configuration files :
<br><br>1. core-site.xml
<br><br><configuration>
<br><property>
<br> <name>fs.default.name</name>
<br> <value>hdfs://centos:54310</value>
<br> <description>determine the host, port, etc. for a filesystem.</description>
<br></property>
<br></configuration>
<br><br>2. mapred-site.xml
<br><br><configuration>
<br><property>
<br> <name>mapred.job.tracker</name>
<br> <value>hdfs://centos:54311</value>
<br> <description>....</description>
<br></property>
<br></configuration>
<br><br>3. masters
<br>
<br> centos
<br><br>4. slaves
<br><br> fedora
<br><br><br>I issued bin/start-dfs.sh from the master(centos) machine and this is what I see :
<br><br>starting namenode, logging to xxxxxxxxxxx
<br>fedora: starting datanode, logging to xxxxxxx
<br>centos: starting secondarynamenode, logging to xxxxxx
<br><br>When I checked the slave(fedora) logs, this what I see :
<br><br>*************************************************************************
<br>STARTUP_MSG: Starting DataNode
<br>STARTUP_MSG: host = fedora/192.168.1.102
<br>STARTUP_MSG: args = []
<br>STARTUP_MSG: version = 0.20.1
<br>STARTUP_MSG: build = <a href="http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.1-rc1" target="_top" rel="nofollow">http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.1-rc1</a> -r 810220; compiled by 'oom' on Tue Sep 1 20:55:56 UTC
<br><br>2009
<br>************************************************************/
<br>2009-10-17 17:05:43,385 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: centos/192.168.1.125:54310. Already tried 0 time(s).
<br>2009-10-17 17:05:44,387 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: centos/192.168.1.125:54310. Already tried 1 time(s).
<br>2009-10-17 17:05:45,389 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: centos/192.168.1.125:54310. Already tried 2 time(s).
<br>2009-10-17 17:05:46,390 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: centos/192.168.1.125:54310. Already tried 3 time(s).
<br>2009-10-17 17:05:47,388 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: centos/192.168.1.125:54310. Already tried 4 time(s).
<br>2009-10-17 17:05:48,390 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: centos/192.168.1.125:54310. Already tried 5 time(s).
<br>2009-10-17 17:05:49,393 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: centos/192.168.1.125:54310. Already tried 6 time(s).
<br>2009-10-17 17:05:50,394 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: centos/192.168.1.125:54310. Already tried 7 time(s).
<br>2009-10-17 17:05:51,395 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: centos/192.168.1.125:54310. Already tried 8 time(s).
<br>2009-10-17 17:05:52,398 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: centos/192.168.1.125:54310. Already tried 9 time(s).
<br>2009-10-17 17:05:52,404 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Call to centos/192.168.1.125:54310 failed on local
<br><br>exception: java.net.NoRouteToHostException: No route to host
<br> at org.apache.hadoop.ipc.Client.wrapException(Client.java:774)
<br> at org.apache.hadoop.ipc.Client.call(Client.java:742)
<br> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
<br> at $Proxy4.getProtocolVersion(Unknown Source)
<br> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
<br> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:346)
<br> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:383)
<br> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:314)
<br> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:291)
<br> at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:269)
<br> at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:216)
<br> at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1283)
<br> at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1238)
<br> at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1246)
<br> at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1368)
<br>Caused by: java.net.NoRouteToHostException: No route to host
<br> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
<br> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
<br> at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
<br> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
<br> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:304)
<br> at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
<br> at org.apache.hadoop.ipc.Client.getConnection(Client.java:859)
<br> at org.apache.hadoop.ipc.Client.call(Client.java:719)
<br> ... 13 more
<br><br>2009-10-17 17:05:52,407 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
<br>/************************************************************
<br>SHUTDOWN_MSG: Shutting down DataNode at fedora/192.168.1.102
<br>************************************************************/
<br><br>*************************************************************************
<br><br><br><br><br><br>The log on the master shows :
<br><br>2009-10-17 17:05:39,352 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
<br>/************************************************************
<br>STARTUP_MSG: Starting NameNode
<br>STARTUP_MSG: host = centos/192.168.1.125
<br>STARTUP_MSG: args = []
<br>STARTUP_MSG: version = 0.20.1
<br>STARTUP_MSG: build = <a href="http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.1-rc1" target="_top" rel="nofollow">http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.1-rc1</a> -r 810220; compiled by 'oom' on Tue Sep 1 20:55:56 UTC
<br><br>2009
<br>************************************************************/
<br>2009-10-17 17:05:39,615 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=NameNode, port=54310
<br>2009-10-17 17:05:39,620 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Namenode up at: centos/192.168.1.125:54310
<br>2009-10-17 17:05:39,622 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=NameNode, sessionId=null
<br>2009-10-17 17:05:39,624 INFO org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics: Initializing NameNodeMeterics using context
<br><br>object:org.apache.hadoop.metrics.spi.NullContext
<br>2009-10-17 17:05:39,779 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=joe,joe
<br>2009-10-17 17:05:39,780 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup=supergroup
<br>2009-10-17 17:05:39,781 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled=true
<br>2009-10-17 17:05:39,877 INFO org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics: Initializing FSNamesystemMetrics using context
<br><br>object:org.apache.hadoop.metrics.spi.NullContext
<br>2009-10-17 17:05:39,883 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered FSNamesystemStatusMBean
<br>2009-10-17 17:05:39,929 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files = 1
<br>2009-10-17 17:05:39,933 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files under construction = 0
<br>2009-10-17 17:05:39,933 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file of size 97 loaded in 0 seconds.
<br>2009-10-17 17:05:39,934 INFO org.apache.hadoop.hdfs.server.common.Storage: Edits file /tmp/hadoop-joe/dfs/name/current/edits of size 4 edits # 0
<br><br>loaded in 0 seconds.
<br>2009-10-17 17:05:39,972 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file of size 97 saved in 0 seconds.
<br>2009-10-17 17:05:40,130 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished loading FSImage in 469 msecs
<br>2009-10-17 17:05:40,131 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Total number of blocks = 0
<br>2009-10-17 17:05:40,131 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of invalid blocks = 0
<br>2009-10-17 17:05:40,131 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of under-replicated blocks = 0
<br>2009-10-17 17:05:40,132 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of over-replicated blocks = 0
<br>2009-10-17 17:05:40,132 INFO org.apache.hadoop.hdfs.StateChange: STATE* Leaving safe mode after 0 secs.
<br>2009-10-17 17:05:40,132 INFO org.apache.hadoop.hdfs.StateChange: STATE* Network topology has 0 racks and 0 datanodes
<br>2009-10-17 17:05:40,132 INFO org.apache.hadoop.hdfs.StateChange: STATE* UnderReplicatedBlocks has 0 blocks
<br>2009-10-17 17:05:40,320 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
<br>2009-10-17 17:05:40,518 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1.
<br><br>Opening the listener on 50070
<br>2009-10-17 17:05:40,520 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50070 webServer.getConnectors()[0].getLocalPort()
<br><br>returned 50070
<br>2009-10-17 17:05:40,520 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50070
<br>2009-10-17 17:05:40,520 INFO org.mortbay.log: jetty-6.1.14
<br>2009-10-17 17:05:46,612 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50070
<br>2009-10-17 17:05:46,612 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at: 0.0.0.0:50070
<br>2009-10-17 17:05:46,613 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
<br>2009-10-17 17:05:46,615 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 54310: starting
<br>2009-10-17 17:05:46,629 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 54310: starting
<br>2009-10-17 17:05:46,634 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 54310: starting
<br>2009-10-17 17:05:46,636 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 54310: starting
<br>2009-10-17 17:05:46,637 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 54310: starting
<br>2009-10-17 17:05:46,641 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 54310: starting
<br>2009-10-17 17:05:46,642 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 54310: starting
<br>2009-10-17 17:05:46,643 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 54310: starting
<br>2009-10-17 17:05:46,643 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 54310: starting
<br>2009-10-17 17:05:46,900 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 54310: starting
<br>2009-10-17 17:05:46,916 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 54310: starting
<br>2009-10-17 17:11:10,967 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 192.168.1.125
<br>2009-10-17 17:11:10,968 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of transactions: 0 Total time for transactions(ms): 0Number of
<br><br>transactions batched in Syncs: 0 Number of syncs: 0 SyncTimes(ms): 0
<br>2009-10-17 17:11:11,518 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll FSImage from 192.168.1.125
<br>2009-10-17 17:11:11,519 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of transactions: 0 Total time for transactions(ms): 0Number of
<br><br>transactions batched in Syncs: 0 Number of syncs: 1 SyncTimes(ms): 2
<br><br><br>
</div>
</div></blockquote>
<p>From forum: <a href="http://old.nabble.com/Hadoop-core-user-f30590.html" embed="fixTarget[30590]" target="_top" >Hadoop core-user</a></p>tag:old.nabble.com,2006:post-25973811Re: editing etc hosts files of a cluster2009-10-20T04:58:33Z2009-10-20T04:58:33ZRamesh.RamasamyThis following script does the work, but testing against /etc/hosts file instead of /home/hadoop/hosts/hosts fails, I'm getting permission denied error..
<br><br>hadoop@master:sed$ cat applyhadoopcluster.sh
<br>#!/bin/bash
<br>for i in `cat clusternodes`
<br>do
<br> # Remove the exiting entiries of Hadoop cluster, if any
<br> ssh hadoop@$i "sed '/#HADOOP_HOST_ENTRIES_STARTING/,/#HADOOP_HOST_ENTRIES_ENDING/d' /home/hadoop/hosts/hosts -i"
<br><br> # Insert the Hadoop cluster entries
<br> ssh hadoop@$i "sed '1i\#HADOOP_HOST_ENTRIES_STARTING' /home/hadoop/hosts/hosts -i"
<br> count=2
<br> while read j
<br> do
<br> #echo $j
<br> ssh -n hadoop@$i "sed '2i$j' /home/hadoop/hosts/hosts -i"
<br> count=`expr $count + 1`
<br> #echo $count
<br> done < clusterhostnames
<br> ssh hadoop@$i "sed ${count}i\#HADOOP_HOST_ENTRIES_ENDING /home/hadoop/hosts/hosts -i"
<br>done
<br><br>hadoop@master:sed$ cat clusterhostnames
<br>172.25.149.37 master
<br>172.25.149.38 slave2
<br>172.25.149.66 slave1
<br><br>hadoop@master:sed$ cat clusternodes
<br>master
<br>slave2
<br>slave1
<br><p>From forum: <a href="http://old.nabble.com/Hadoop-core-user-f30590.html" embed="fixTarget[30590]" target="_top" >Hadoop core-user</a></p>tag:old.nabble.com,2006:post-25958579editing etc hosts files of a cluster2009-10-19T06:40:18Z2009-10-19T06:40:18ZRamesh.RamasamyHi,
<br><br>I have a cluster setup with 3 nodes, and I'm adding hostname details (in /etc/hosts) manually in each node. Seems it is not an effective approach. How this scenario is handled in big clusters?
<br><br>Is there any simple of way to add the hostname details in all the nodes by editing a single entry/file/script?
<br><br>Thanks and Regards,
<br>Ramesh
<br><br><p>From forum: <a href="http://old.nabble.com/Hadoop-core-user-f30590.html" embed="fixTarget[30590]" target="_top" >Hadoop core-user</a></p>tag:old.nabble.com,2006:post-25951789Seattle / NW Hadoop, Lucene, Apache "Cloud Stack" Meetup, Wed Oct 28 6:45pm2009-10-18T17:10:49Z2009-10-18T17:10:49ZBrickMcLargeHugeGreetings,
<br><br>(You're receiving this e-mail because you're on a DL or I think you'd
<br>be interested)
<br><br>It's time for another Hadoop/Lucene/Apache "Cloud" stack meetup! This
<br>month it'll be on Wednesday, the 28th, at 6:45 pm.
<br><br>A *huge* thanks for everyone who showed up last month, and to Facebook
<br>for sending someone awesome to speak about Hive. We learned quite a
<br>bit!
<br><br>For October, we will have someone speaking about Cascading, and how it
<br>helps workflow abstraction with MapReduce. Very useful stuff to know.
<br><br>We've had great attendance in the past few months, let's keep it up!
<br>I'm always amazed by the things I learn from everyone.
<br><br>We're at the University of Washington, Allen Computer Science Center
<br>(not Electrical Engineering)
<br><br>Map: <a href="http://www.washington.edu/home/maps/?CSE" target="_top" rel="nofollow">http://www.washington.edu/home/maps/?CSE</a><br><br>Room: 303 -or- the Entry level. If there are changes, signs will be posted.
<br><br>More Info:
<br><br>The meetup is about 2 hours (and there's usually food): we'll have two
<br>in-depth talks, and then several "lightning talks" of 5 minutes. We'll
<br>then have discussion and 'social time'. Let me know if you're
<br>interested in speaking or attending. We'd like to focus on education,
<br>so feel free to ask questions.
<br><br>Contact: Bradford Stephens, 904-415-3009, <a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=25951789&i=0" target="_top" rel="nofollow">bradfordstephens@...</a>
<br><br>--
<br><a href="http://www.drawntoscaleconsulting.com" target="_top" rel="nofollow">http://www.drawntoscaleconsulting.com</a> - Scalability, Hadoop, HBase,
<br>and Distributed Lucene Consulting
<br><br><a href="http://www.roadtofailure.com" target="_top" rel="nofollow">http://www.roadtofailure.com</a> -- The Fringes of Scalability, Social
<br>Media, and Computer Science
<br><p>From forum: <a href="http://old.nabble.com/Hadoop-core-user-f30590.html" embed="fixTarget[30590]" target="_top" >Hadoop core-user</a></p>tag:old.nabble.com,2006:post-25947843HTTP secure access2009-10-18T08:50:09Z2009-10-18T08:50:09ZBM-13Hi!
<br>Well, I have a kinda simple question, but I can not spot a proper doc
<br>for it: how you, guys, restricting access to the web interfaces? :-)
<br><br>It is somewhere in jetty or there is no feature like this? I am OK
<br>with a simple basic authentication, but I don't really like when
<br>others are staring at logs that are easily accessible through the web
<br>within the same network...
<br><br>--
<br>Kind regards, BM
<br><br>Things, that are stupid at the beginning, rarely ends up wisely.
<br><p>From forum: <a href="http://old.nabble.com/Hadoop-core-user-f30590.html" embed="fixTarget[30590]" target="_top" >Hadoop core-user</a></p>tag:old.nabble.com,2006:post-25946798Datanode Throwing NoRouteToHostException2009-10-18T06:44:24Z2009-10-18T06:44:24ZlazikidI need some help with setting up a Hadoop cluster. The datanode on the slave is not coming up throwing java.net.NoRouteToHostException: No route to host. Please see the details below.
<br><br>I have a centos master and a fedora slave. Both have Java 6 and Hadoop 0.20.1. I instaleed hadoop under /opt in both machines.
<br><br>The machines can ping one another using hostname and ip. They can also make password-less ssh connection to one another using both hostname and ip.
<br><br>This is what I have in the /etc/hosts file for the master :
<br><br>************************************************
<br>127.0.0.1 localhost.localdomain localhost
<br>192.168.1.125 centos
<br>192.168.1.102 fedora
<br>::1 localhost6.localdomain6 localhost6
<br>************************************************
<br><br>This is what I have in the /etc/hosts for the slave
<br><br>************************************************
<br>127.0.0.1 localhost.localdomain localhost
<br>192.168.1.104 ubuntu64
<br>192.168.1.102 fedora
<br>192.168.1.125 centos
<br>::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
<br>************************************************
<br><br><br><br>These are my configuration files :
<br><br>1. core-site.xml
<br><br><configuration>
<br><property>
<br> <name>fs.default.name</name>
<br> <value>hdfs://centos:54310</value>
<br> <description>determine the host, port, etc. for a filesystem.</description>
<br></property>
<br></configuration>
<br><br>2. mapred-site.xml
<br><br><configuration>
<br><property>
<br> <name>mapred.job.tracker</name>
<br> <value>hdfs://centos:54311</value>
<br> <description>....</description>
<br></property>
<br></configuration>
<br><br>3. masters
<br>
<br> centos
<br><br>4. slaves
<br><br> fedora
<br><br><br>I issued bin/start-dfs.sh from the master(centos) machine and this is what I see :
<br><br>starting namenode, logging to xxxxxxxxxxx
<br>fedora: starting datanode, logging to xxxxxxx
<br>centos: starting secondarynamenode, logging to xxxxxx
<br><br>When I checked the slave(fedora) logs, this what I see :
<br><br>*************************************************************************
<br>STARTUP_MSG: Starting DataNode
<br>STARTUP_MSG: host = fedora/192.168.1.102
<br>STARTUP_MSG: args = []
<br>STARTUP_MSG: version = 0.20.1
<br>STARTUP_MSG: build = <a href="http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.1-rc1" target="_top" rel="nofollow">http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.1-rc1</a> -r 810220; compiled by 'oom' on Tue Sep 1 20:55:56 UTC
<br><br>2009
<br>************************************************************/
<br>2009-10-17 17:05:43,385 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: centos/192.168.1.125:54310. Already tried 0 time(s).
<br>2009-10-17 17:05:44,387 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: centos/192.168.1.125:54310. Already tried 1 time(s).
<br>2009-10-17 17:05:45,389 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: centos/192.168.1.125:54310. Already tried 2 time(s).
<br>2009-10-17 17:05:46,390 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: centos/192.168.1.125:54310. Already tried 3 time(s).
<br>2009-10-17 17:05:47,388 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: centos/192.168.1.125:54310. Already tried 4 time(s).
<br>2009-10-17 17:05:48,390 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: centos/192.168.1.125:54310. Already tried 5 time(s).
<br>2009-10-17 17:05:49,393 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: centos/192.168.1.125:54310. Already tried 6 time(s).
<br>2009-10-17 17:05:50,394 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: centos/192.168.1.125:54310. Already tried 7 time(s).
<br>2009-10-17 17:05:51,395 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: centos/192.168.1.125:54310. Already tried 8 time(s).
<br>2009-10-17 17:05:52,398 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: centos/192.168.1.125:54310. Already tried 9 time(s).
<br>2009-10-17 17:05:52,404 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Call to centos/192.168.1.125:54310 failed on local
<br><br>exception: java.net.NoRouteToHostException: No route to host
<br> at org.apache.hadoop.ipc.Client.wrapException(Client.java:774)
<br> at org.apache.hadoop.ipc.Client.call(Client.java:742)
<br> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
<br> at $Proxy4.getProtocolVersion(Unknown Source)
<br> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
<br> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:346)
<br> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:383)
<br> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:314)
<br> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:291)
<br> at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:269)
<br> at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:216)
<br> at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1283)
<br> at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1238)
<br> at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1246)
<br> at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1368)
<br>Caused by: java.net.NoRouteToHostException: No route to host
<br> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
<br> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
<br> at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
<br> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
<br> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:304)
<br> at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
<br> at org.apache.hadoop.ipc.Client.getConnection(Client.java:859)
<br> at org.apache.hadoop.ipc.Client.call(Client.java:719)
<br> ... 13 more
<br><br>2009-10-17 17:05:52,407 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
<br>/************************************************************
<br>SHUTDOWN_MSG: Shutting down DataNode at fedora/192.168.1.102
<br>************************************************************/
<br><br>*************************************************************************
<br><br><br><br><br><br>The log on the master shows :
<br><br>2009-10-17 17:05:39,352 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
<br>/************************************************************
<br>STARTUP_MSG: Starting NameNode
<br>STARTUP_MSG: host = centos/192.168.1.125
<br>STARTUP_MSG: args = []
<br>STARTUP_MSG: version = 0.20.1
<br>STARTUP_MSG: build = <a href="http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.1-rc1" target="_top" rel="nofollow">http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.1-rc1</a> -r 810220; compiled by 'oom' on Tue Sep 1 20:55:56 UTC
<br><br>2009
<br>************************************************************/
<br>2009-10-17 17:05:39,615 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=NameNode, port=54310
<br>2009-10-17 17:05:39,620 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Namenode up at: centos/192.168.1.125:54310
<br>2009-10-17 17:05:39,622 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=NameNode, sessionId=null
<br>2009-10-17 17:05:39,624 INFO org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics: Initializing NameNodeMeterics using context
<br><br>object:org.apache.hadoop.metrics.spi.NullContext
<br>2009-10-17 17:05:39,779 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=joe,joe
<br>2009-10-17 17:05:39,780 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup=supergroup
<br>2009-10-17 17:05:39,781 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled=true
<br>2009-10-17 17:05:39,877 INFO org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics: Initializing FSNamesystemMetrics using context
<br><br>object:org.apache.hadoop.metrics.spi.NullContext
<br>2009-10-17 17:05:39,883 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered FSNamesystemStatusMBean
<br>2009-10-17 17:05:39,929 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files = 1
<br>2009-10-17 17:05:39,933 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files under construction = 0
<br>2009-10-17 17:05:39,933 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file of size 97 loaded in 0 seconds.
<br>2009-10-17 17:05:39,934 INFO org.apache.hadoop.hdfs.server.common.Storage: Edits file /tmp/hadoop-joe/dfs/name/current/edits of size 4 edits # 0
<br><br>loaded in 0 seconds.
<br>2009-10-17 17:05:39,972 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file of size 97 saved in 0 seconds.
<br>2009-10-17 17:05:40,130 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished loading FSImage in 469 msecs
<br>2009-10-17 17:05:40,131 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Total number of blocks = 0
<br>2009-10-17 17:05:40,131 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of invalid blocks = 0
<br>2009-10-17 17:05:40,131 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of under-replicated blocks = 0
<br>2009-10-17 17:05:40,132 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of over-replicated blocks = 0
<br>2009-10-17 17:05:40,132 INFO org.apache.hadoop.hdfs.StateChange: STATE* Leaving safe mode after 0 secs.
<br>2009-10-17 17:05:40,132 INFO org.apache.hadoop.hdfs.StateChange: STATE* Network topology has 0 racks and 0 datanodes
<br>2009-10-17 17:05:40,132 INFO org.apache.hadoop.hdfs.StateChange: STATE* UnderReplicatedBlocks has 0 blocks
<br>2009-10-17 17:05:40,320 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
<br>2009-10-17 17:05:40,518 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1.
<br><br>Opening the listener on 50070
<br>2009-10-17 17:05:40,520 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50070 webServer.getConnectors()[0].getLocalPort()
<br><br>returned 50070
<br>2009-10-17 17:05:40,520 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50070
<br>2009-10-17 17:05:40,520 INFO org.mortbay.log: jetty-6.1.14
<br>2009-10-17 17:05:46,612 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50070
<br>2009-10-17 17:05:46,612 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at: 0.0.0.0:50070
<br>2009-10-17 17:05:46,613 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
<br>2009-10-17 17:05:46,615 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 54310: starting
<br>2009-10-17 17:05:46,629 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 54310: starting
<br>2009-10-17 17:05:46,634 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 54310: starting
<br>2009-10-17 17:05:46,636 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 54310: starting
<br>2009-10-17 17:05:46,637 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 54310: starting
<br>2009-10-17 17:05:46,641 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 54310: starting
<br>2009-10-17 17:05:46,642 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 54310: starting
<br>2009-10-17 17:05:46,643 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 54310: starting
<br>2009-10-17 17:05:46,643 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 54310: starting
<br>2009-10-17 17:05:46,900 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 54310: starting
<br>2009-10-17 17:05:46,916 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 54310: starting
<br>2009-10-17 17:11:10,967 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 192.168.1.125
<br>2009-10-17 17:11:10,968 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of transactions: 0 Total time for transactions(ms): 0Number of
<br><br>transactions batched in Syncs: 0 Number of syncs: 0 SyncTimes(ms): 0
<br>2009-10-17 17:11:11,518 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll FSImage from 192.168.1.125
<br>2009-10-17 17:11:11,519 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of transactions: 0 Total time for transactions(ms): 0Number of
<br><br>transactions batched in Syncs: 0 Number of syncs: 1 SyncTimes(ms): 2
<br><br><br> <p>From forum: <a href="http://old.nabble.com/Hadoop-core-user-f30590.html" embed="fixTarget[30590]" target="_top" >Hadoop core-user</a></p>tag:old.nabble.com,2006:post-25925504Re: fuse:The ratio of reported blocks 0.0000 has not reached the threshold 0.99902009-10-16T06:31:18Z2009-10-16T06:31:18ZBrian BockelmanHey,
<br><br>I thought this sort of basic safemode operation is covered well in the
<br>manual. Do you have any questions or suggestions for the documentation?
<br><br>Brian
<br><br>On Oct 16, 2009, at 3:12 AM, yibo820217 wrote:
<br><div class='shrinkable-quote'><div class='shrinkable-quote'><br>>
<br>>
<br>> It seems lke the normal procedure to restart the hdfs.
<br>>
<br>>
<br>>
<br>>
<br>> yibo820217 wrote:
<br>>>
<br>>> hi,recently,i got some problem.
<br>>> at first,I start the hadoop
<br>>> #bin/start-all.sh
<br>>> then I look the logs
<br>>> #cat hadoop-root-namenode-cent52ip32.log
<br>>>
<br>>> 2009-10-13 22:12:29,733 INFO
<br>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
<br>>>
<br>>> ugi=root,root,bin,daemon,sys,adm,disk,wheel ip=/127.0.0.1
<br>>> cmd=listStatus src=/tmp/hadoop-root/mapred/system
<br>>>
<br>>> dst=null perm=null
<br>>> 2009-10-13 22:12:29,736 INFO org.apache.hadoop.ipc.Server: IPC Server
<br>>> handler 5 on 9000, call delete(/tmp/hadoop-
<br>>>
<br>>> root/mapred/system, true) from 127.0.0.1:37371: error:
<br>>> org.apache.hadoop.hdfs.server.namenode.SafeModeException:
<br>>>
<br>>> Cannot delete /tmp/hadoop-root/mapred/system. Name node is in safe
<br>>> mode.
<br>>> The ratio of reported blocks 0.0000 has not reached the threshold
<br>>> 0.9990.
<br>>> Safe mode will be turned off
<br>>> automatically.
<br>>> org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot
<br>>> delete
<br>>> /tmp/hadoop-root/mapred/system. Name node is
<br>>>
<br>>> in safe mode.
<br>>> The ratio of reported blocks 0.0000 has not reached the threshold
<br>>> 0.9990.
<br>>> Safe mode will be turned off
<br>>>
<br>>> automatically.
<br>>> at
<br>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal
<br>>> (FSNamesystem.java:1696)
<br>>> at
<br>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete
<br>>> (FSNamesystem.java:1676)
<br>>> at
<br>>> org.apache.hadoop.hdfs.server.namenode.NameNode.delete
<br>>> (NameNode.java:517)
<br>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
<br>>> at
<br>>> sun.reflect.NativeMethodAccessorImpl.invoke
<br>>> (NativeMethodAccessorImpl.java:39)
<br>>> at
<br>>> sun.reflect.DelegatingMethodAccessorImpl.invoke
<br>>> (DelegatingMethodAccessorImpl.java:25)
<br>>> at java.lang.reflect.Method.invoke(Method.java:597)
<br>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
<br>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
<br>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
<br>>> at java.security.AccessController.doPrivileged(Native Method)
<br>>> at javax.security.auth.Subject.doAs(Subject.java:396)
<br>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
<br>>>
<br>>> the same message have appear 10 times,the the safemode is off
<br>>> [root@cent52ip32 hadoop]# bin/hadoop dfsadmin -safemode get
<br>>> Safe mode is OFF
<br>>>
<br>>> I don't know the reason,please help me.
<br>>> thanks very much.
<br>>>
<br>>
<br>> --
<br>> View this message in context: <a href="http://www.nabble.com/fuse%3AThe-ratio-of-reported-blocks-0.0000-has-not-reached-the-threshold-0.9990-tp25868687p25921487.html" target="_top">http://www.nabble.com/fuse%3AThe-ratio-of-reported-blocks-0.0000-has-not-reached-the-threshold-0.9990-tp25868687p25921487.html</a><br>> Sent from the Hadoop core-user mailing list archive at Nabble.com.
</div></div><br /> <div class="small"><br/><img src="http://old.nabble.com/images/icon_attachment.gif" > <strong>smime.p7s</strong> (3K) <a href="http://old.nabble.com/attachment/25925504/0/smime.p7s" target="_top">Download Attachment</a></div><p>From forum: <a href="http://old.nabble.com/Hadoop-core-user-f30590.html" embed="fixTarget[30590]" target="_top" >Hadoop core-user</a></p>tag:old.nabble.com,2006:post-25921487Re: fuse:The ratio of reported blocks 0.0000 has not reached the threshold 0.99902009-10-16T01:12:00Z2009-10-16T01:12:00Zyibo820217<br>It seems lke the normal procedure to restart the hdfs.
<br><br><br><br><blockquote class="quote light-black dark-border-color"><div class="quote light-border-color">
<div class="quote-author" style="font-weight: bold;">yibo820217 wrote:</div>
<div class="quote-message shrinkable-quote">hi,recently,i got some problem.
<br>at first,I start the hadoop
<br>#bin/start-all.sh
<br>then I look the logs
<br>#cat hadoop-root-namenode-cent52ip32.log
<br><br>2009-10-13 22:12:29,733 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
<br><br>ugi=root,root,bin,daemon,sys,adm,disk,wheel ip=/127.0.0.1 cmd=listStatus src=/tmp/hadoop-root/mapred/system
<br><br> dst=null perm=null
<br>2009-10-13 22:12:29,736 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 9000, call delete(/tmp/hadoop-
<br><br>root/mapred/system, true) from 127.0.0.1:37371: error: org.apache.hadoop.hdfs.server.namenode.SafeModeException:
<br><br><b><i>Cannot delete /tmp/hadoop-root/mapred/system. Name node is in safe mode.
<br>The ratio of reported blocks 0.0000 has not reached the threshold 0.9990. Safe mode will be turned off </i></b><br>automatically.
<br>org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete /tmp/hadoop-root/mapred/system. Name node is
<br><br>in safe mode.
<br>The ratio of reported blocks 0.0000 has not reached the threshold 0.9990. Safe mode will be turned off
<br><br>automatically.
<br> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1696)
<br> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1676)
<br> at org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:517)
<br> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
<br> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
<br> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
<br> at java.lang.reflect.Method.invoke(Method.java:597)
<br> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
<br> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
<br> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
<br> at java.security.AccessController.doPrivileged(Native Method)
<br> at javax.security.auth.Subject.doAs(Subject.java:396)
<br> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
<br><br>the same message have appear 10 times,the the safemode is off
<br>[root@cent52ip32 hadoop]# bin/hadoop dfsadmin -safemode get
<br>Safe mode is OFF
<br><br>I don't know the reason,please help me.
<br>thanks very much.
</div>
</div></blockquote>
<p>From forum: <a href="http://old.nabble.com/Hadoop-core-user-f30590.html" embed="fixTarget[30590]" target="_top" >Hadoop core-user</a></p>