Nabble - Hadoop lucene-users

hadoop log file for localfile system

2009-11-04T03:46:11Z

Hi i am new to hadoop been using it for few weeks now, i tried few map/reduce example and could see logs in jobtracker.jsp, . we can view status of the job, every task, config, counters, exceptions and even anything that is written to System.out, System.err and syslogs thats really nice but the problem is, This log can be used only for HDFS but not from localfile system.
I have to use local file system , so is there any way to have log for local file system in hadoop.or is there anyway to use same jobtracker.jsb for local file system too. i am really stuck with this problem. it would be really great help if anyone could help me out.

Thanks in advance.

A way to input xml files in mapreduce

2009-11-02T14:27:30Z

I am new to hadoop and still learning most of the details. I am working on an application that will take input from lots of small xml files. Each xml files has some record that I want to parse and input data in a hbase table. How should I go about parsing xml files and input in map functions. Should I have one mapper per xml file or is there another way of doing this? Thanks for your help and time.

Thanks
Vipul

Problem in the way reducer and combiner operate(Merging)

2009-10-20T14:30:47Z

2009-10-19 17:54:16,221 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=SHUFFLE, sessionId=
2009-10-19 17:54:17,632 INFO org.apache.hadoop.mapred.ReduceTask: ShuffleRamManager: MemoryLimit=78643200, MaxSingleShuffleLimit=19660800
2009-10-19 17:54:17,761 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200910191750_0001_r_000000_0 Thread started: Thread for merging on-disk files
2009-10-19 17:54:17,762 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200910191750_0001_r_000000_0 Thread waiting: Thread for merging on-disk files
2009-10-19 17:54:17,763 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200910191750_0001_r_000000_0 Thread started: Thread for merging in memory files
2009-10-19 17:54:17,782 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200910191750_0001_r_000000_0 Need another 2 map output(s) where 0 is already in progress
2009-10-19 17:54:17,793 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200910191750_0001_r_000000_0: Got 0 new map-outputs & number of known map outputs is 0
2009-10-19 17:54:17,793 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200910191750_0001_r_000000_0 Scheduled 0 of 0 known outputs (0 slow hosts and 0 dup hosts)
2009-10-19 17:54:22,816 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200910191750_0001_r_000000_0: Got 2 new map-outputs & number of known map outputs is 2
2009-10-19 17:54:22,818 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200910191750_0001_r_000000_0 Scheduled 1 of 2 known outputs (0 slow hosts and 1 dup hosts)
2009-10-19 17:54:23,041 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling 44 bytes (44 raw bytes) into RAM from attempt_200910191750_0001_m_000000_0
2009-10-19 17:54:23,043 INFO org.apache.hadoop.mapred.ReduceTask: Read 44 bytes from map-output for attempt_200910191750_0001_m_000000_0
2009-10-19 17:54:23,047 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 from attempt_200910191750_0001_m_000000_0 -> (4, 8) from localhost
2009-10-19 17:54:24,808 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200910191750_0001_r_000000_0 Scheduled 1 of 1 known outputs (0 slow hosts and 0 dup hosts)
2009-10-19 17:54:24,824 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling 44 bytes (44 raw bytes) into RAM from attempt_200910191750_0001_m_000001_0
2009-10-19 17:54:24,825 INFO org.apache.hadoop.mapred.ReduceTask: Read 44 bytes from map-output for attempt_200910191750_0001_m_000001_0
2009-10-19 17:54:24,825 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 from attempt_200910191750_0001_m_000001_0 -> (4, 8) from localhost
2009-10-19 17:54:25,800 INFO org.apache.hadoop.mapred.ReduceTask: Closed ram manager
2009-10-19 17:54:25,817 INFO org.apache.hadoop.mapred.ReduceTask: Interleaved on-disk merge complete: 0 files left.
2009-10-19 17:54:25,973 INFO org.apache.hadoop.mapred.ReduceTask: Initiating in-memory merge with 2 segments...
2009-10-19 17:54:25,984 INFO org.apache.hadoop.mapred.Merger: Merging 2 sorted segments
2009-10-19 17:54:25,985 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 88 bytes
2009-10-19 17:54:25,991 FATAL mma.CombA: Running Identity Combiner...
2009-10-19 17:54:25,992 FATAL mma.CombA: Identity Key is1 Combiner processed #: 2
2009-10-19 17:54:25,992 FATAL mma.CombA: Running Identity Combiner...
2009-10-19 17:54:25,992 FATAL mma.CombA: Identity Key is4 Combiner processed #: 1
2009-10-19 17:54:25,993 FATAL mma.CombA: Running Identity Combiner...
2009-10-19 17:54:25,993 FATAL mma.CombA: Identity Key is3 Combiner processed #: 1
2009-10-19 17:54:25,993 FATAL mma.CombA: Running Identity Combiner...
2009-10-19 17:54:25,993 FATAL mma.CombA: Identity Key is4 Combiner processed #: 1
2009-10-19 17:54:25,993 FATAL mma.CombA: Running Identity Combiner...
2009-10-19 17:54:25,993 FATAL mma.CombA: Identity Key is3 Combiner processed #: 1
2009-10-19 17:54:25,996 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200910191750_0001_r_000000_0 Merge of the 2 files in-memory complete. Local file is /var/lib/hadoop/cache/hadoop/mapred/local/taskTracker/jobcache/job_200910191750_0001/attempt_200910191750_0001_r_000000_0/output/map_0.out of size 86
2009-10-19 17:54:25,996 INFO org.apache.hadoop.mapred.ReduceTask: In-memory merge complete: 0 files left.
2009-10-19 17:54:25,998 INFO org.apache.hadoop.mapred.ReduceTask: Initiating final on-disk merge with 1 files
2009-10-19 17:54:25,999 INFO org.apache.hadoop.mapred.Merger: Merging 1 sorted segments
2009-10-19 17:54:26,012 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 86 bytes
2009-10-19 17:54:26,081 FATAL mma.RedA: THE KEY WAS 'PAVG'
2009-10-19 17:54:26,081 FATAL mma.RedA: AVG ENCTRD:0.0
2009-10-19 17:54:26,082 FATAL mma.RedA: AVG ENCTRD:3.0
2009-10-19 17:54:26,083 FATAL mma.RedA: Executed with key:1 Reducer with2.0
2009-10-19 17:54:26,084 FATAL mma.RedA: THE KEY WAS 'MIN'
2009-10-19 17:54:26,084 FATAL mma.RedA: MIN ENCTRD:3.0
2009-10-19 17:54:26,084 FATAL mma.RedA: Executed with key:4 Reducer with1.0
2009-10-19 17:54:26,084 FATAL mma.RedA: THE KEY WAS 'MAX'
2009-10-19 17:54:26,084 FATAL mma.RedA: MAX ENCTRD:3.0
2009-10-19 17:54:26,084 FATAL mma.RedA: Executed with key:3 Reducer with1.0
2009-10-19 17:54:26,084 FATAL mma.RedA: THE KEY WAS 'MIN'
2009-10-19 17:54:26,084 FATAL mma.RedA: MIN ENCTRD:1.0
2009-10-19 17:54:26,084 FATAL mma.RedA: Executed with key:4 Reducer with1.0
2009-10-19 17:54:26,085 FATAL mma.RedA: THE KEY WAS 'MAX'
2009-10-19 17:54:26,086 FATAL mma.RedA: MAX ENCTRD:2.0
2009-10-19 17:54:26,087 FATAL mma.RedA: Executed with key:3 Reducer with1.0
2009-10-19 17:54:26,215 INFO org.apache.hadoop.mapred.TaskRunner: Task 'attempt_200910191750_0001_r_000000_0' done.

The above is the log of my program, which simply does following:
1.Map outputs a kv pair, the key is set to value UNPROCESSED (the key is intwritable)
2.Combiner takes kv pair with value UNPROCESSED and from the list for this key, finds out min,max,avg and outputs
three kv pairs, one for min with key MIN, one for MAX,one for AVG.

3.Reducer should then get the list of values for key MIN, list for MAX, and list for AVG.

Problem:

As seen from log above during merging process, the output isnt sorted as per the key, thus my reducer seemingly gets two lists for MIN and two list for MAX instead of getting 1 key and a list for MAX, and same for min.

I wonder what the problem is, I concur that the problem is in merging step....

Does anyone know whats wrong in above ,from the log?

map function

2009-10-12T09:40:40Z

Can map function be called recursively?

Is anyone interested in working in Spain (Madrid or Barcelona), in Telefónica R&D as a Data Analysis Expert?

2009-10-05T07:09:49Z

Please, check the vacancy:

Telefonica is in the process of converting data into information of our customers using modern data analysis platforms to personalize our services. The successful candidate will:

Technically lead teams of engineers and be responsible for guiding personalization products throughout the execution cycle, focusing specifically on analyzing, designing, implementing and tailoring our solutions to all the markets where Telefonica does business.

Take part in the Telefonica Group Data Analysis strategy through different global initiatives.

Drive the architecture and take decisions on technologies for developing global products implementing Telefonica personalization strategy.

PROFILE:

Requirements:

- Graduate degree on computer science or engineering program (BA or MS degree in computer science, engineering or other technical field preferred).
- Have a wide technical and business knowledge of modern Data Analysis Platforms and personalization Technologies, distributed databases, data modeling, parallel processing, distributed and cloud computing, recommendation systems, inference technologies, etc…
- Full knowledge of technology and business in his area (companies, universities, official organisms, products, other operators…)

Experience:

- Entrepreneurial drive, demonstrated ability to achieve stretch goals in an innovative and fast paced environment.
- 5+ years relevant experience in the Data Analysis & Personalization Technologies area.
- 10+ years in architecting software products.
- Proven track record for product delivery in real world.(At least 5 or more complete product development cycles)
- Able to fit in well within an informal startup-type environment and to provide hands-on development.
- Participation in technical strategic decisions.

Skills:

- Very strong technical skills and deep domain knowledge in the Area.
- Technical leadership and change focused. You will have to become a role model for the Data Analysis & Personalization Technologies Group.
- Excellent technical vision and innovative skills.
- Sound knowledge in software architecture for data analysis and personalization products.
- Strong communication skills that make you able to interact within a global corporation at a Sr Management level.
- That, not easy to find, mix of intelligence, integrity, domain knowledge, verbal agility, and diplomacy which allows you to rapidly earn the trust of technically-astute teams across the company.

Other interesting information:

- Availability to travel.
- English Proficiency level; Spanish conversational level.
- Demonstrated strong performance in prior roles, with increasing levels of responsibility.
- The position will be based in Barcelona or Madrid, Spain.

Send your CV to: e.seleccion1@tid.es

Thanks a lot!

Exception in thread "main" org.apache.hadoop.ipc.RemoteException: java.io.IOExce

2009-09-23T12:29:56Z

I am having an issue starting a job on a hadoop pseudo cluster. I am getting an exception see below. I tried to override the mapred.local.dir with -D parameter and by passing a custom config as well

Here is the custom config declaration:

zeltoa01@hadoop01:~/dev/etl/Hadoop/ybhadoop$ cat etc/hadoop-cluster.xml
<?xml version="1.0"?>
<configuration>
<property>
<name>mapred.local.dir</name>
<value>${hadoop.tmp.dir}/mapred/local</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/hadoop-datastore</value>
<description>A base for other temporary directories.</description>
</property>

zeltoa01@hadoop01:~/dev/etl/Hadoop/ybhadoop$ hadoop jar ybhadoop.jar com.yellow
book.data.hadoop.jobs.ListingsJobStandAlone -Dmapred.local.dir=/home/zeltoa01/m
apred_dir/
09/09/23 15:17:08 INFO mapred.FileInputFormat: Total input paths to process : 1
Exception in thread "main" org.apache.hadoop.ipc.RemoteException: java.io.IOExce
ption: No valid local directories in property: mapred.local.dir
at org.apache.hadoop.conf.Configuration.getLocalPath(Configuration.java:
938)
at org.apache.hadoop.mapred.JobConf.getLocalPath(JobConf.java:279)
at org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:256)

at org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:240)

at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3024)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)

at org.apache.hadoop.ipc.Client.call(Client.java:739)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at org.apache.hadoop.mapred.$Proxy1.submitJob(Unknown Source)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:8
41)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:771)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1290)
at com.yellowbook.data.hadoop.jobs.ListingsJobStandAlone.run(Unknown Sou
rce)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at com.yellowbook.data.hadoop.jobs.ListingsJobStandAlone.main(Unknown So
urce)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:185)

Re: Delete replicated blocks?

2009-08-27T11:00:21Z

I don't know for sure, but running the rebalancer might do this for you.

<
http://hadoop.apache.org/common/docs/r0.20.0/hdfs_user_guide.html#Rebalancer
>

Alex

On Thu, Aug 27, 2009 at 9:18 AM, Michael Thomas <thomas@...>wrote:

> dfs.replication is only used by the client at the time the files are
> written. Changing this setting will not automatically change the
> replication level on existing files. To do that, you need to use the
> hadoop cli:
>
> hadoop fs -setrep -R 1 /
>
> --Mike
>
>
> Vladimir Klimontovich wrote:
> > This will happen automatically.
> > On Aug 27, 2009, at 6:04 PM, Andy Liu wrote:
> >
> >> I'm running a test Hadoop cluster, which had a dfs.replication value
> >> of 3.
> >> I'm now running out of disk space, so I've reduced dfs.replication to
> >> 1 and
> >> restarted my datanodes. Is there a way to free up the over-replicated
> >> blocks, or does this happen automatically at some point?
> >>
> >> Thanks,
> >> Andy
> >
> > ---
> > Vladimir Klimontovich,
> > skype: klimontovich
> > GoogleTalk/Jabber: klimontovich@...
> > Cell phone: +7926 890 2349
> >
>
>

Re: Delete replicated blocks?

2009-08-27T09:18:00Z

dfs.replication is only used by the client at the time the files are
written. Changing this setting will not automatically change the
replication level on existing files. To do that, you need to use the
hadoop cli:

hadoop fs -setrep -R 1 /

--Mike

Vladimir Klimontovich wrote:

> This will happen automatically.
> On Aug 27, 2009, at 6:04 PM, Andy Liu wrote:
>
>> I'm running a test Hadoop cluster, which had a dfs.replication value
>> of 3.
>> I'm now running out of disk space, so I've reduced dfs.replication to
>> 1 and
>> restarted my datanodes. Is there a way to free up the over-replicated
>> blocks, or does this happen automatically at some point?
>>
>> Thanks,
>> Andy
>
> ---
> Vladimir Klimontovich,
> skype: klimontovich
> GoogleTalk/Jabber: klimontovich@...
> Cell phone: +7926 890 2349
>

smime.p7s (5K) Download Attachment

Re: Delete replicated blocks?

2009-08-27T07:05:37Z

This will happen automatically.
On Aug 27, 2009, at 6:04 PM, Andy Liu wrote:

> I'm running a test Hadoop cluster, which had a dfs.replication value
> of 3.
> I'm now running out of disk space, so I've reduced dfs.replication
> to 1 and
> restarted my datanodes. Is there a way to free up the over-replicated
> blocks, or does this happen automatically at some point?
>
> Thanks,
> Andy

---
Vladimir Klimontovich,
skype: klimontovich
GoogleTalk/Jabber: klimontovich@...
Cell phone: +7926 890 2349

Delete replicated blocks?

2009-08-27T07:04:17Z

I'm running a test Hadoop cluster, which had a dfs.replication value of 3.
I'm now running out of disk space, so I've reduced dfs.replication to 1 and
restarted my datanodes. Is there a way to free up the over-replicated
blocks, or does this happen automatically at some point?

Thanks,
Andy

Delete replicated blocks?

2009-08-27T07:00:19Z

I'm running a test Hadoop cluster, which had

Hadoop performance using Mahout

2009-07-23T06:54:15Z

First things first: I want to salute you all and thank you for developing a distributed engine such as Hadoop. It certainly helped me at work. I am now in the process of writing an application for user clustering based on their historical behavior as consumers. For clustering/classification algorithms I resorted to Apache Mahout.

Here's the thing: I generated a pretty small dataset of about ~62MB and set up a small cluster of 5 datanodes and a namenode/jobtracker (runnning on the same machine). Of the datanodes, two of them are four-core processors and the remaining are two-cores (totaling fourteen slaves nodes)... and I tend to think that's more than enough processing power to finish the task in a relatively considerate time, which is exactly what it is not happening. Each MR job is taking about ~3hs to complete, as shown by the jobtracker web UI:

Hadoop job_200907221734_0004
Finished in: 2hrs, 34mins, 3sec

Hadoop job_200907221734_0005
Finished in: 2hrs, 59mins, 34sec

The clustering algorithms runs several iterations of MR phases until it converges, and it takes more than 30hs. in total to complete. For such a small dataset, this is unacceptable and I'm quite sure is has something to do with my cluster configuration and/or how block and its sizes are treated in HDFS. Moreover -and this is quite puzzling to me-, every core on every machine is running at its full capacity almost constantly and it doesn't seem to be any idle time in between tasks. Here are my .xml conf files (just relevant lines):

(mapred-site.xml)
<name>mapred.job.tracker</name>
<value>hdfs://hadoop-jobtracker:54311/</value>
<final>true</final>

<name>mapred.reduce.tasks</name>
<value>98</value> <-- 1.75*14*4 (as suggested by Hadoop's documentation)

<name>mapred.tasktracker.reduce.tasks.maximum</name>
<value>4</value>

<name>mapred.tasktracker.map.tasks.maximum</name>
<value>4</value>

<name>mapred.map.tasks</name>
<value>17</value> <-- With 4MB set as dfs.block.size and having -put the 62MB dataset file with -D dfs.block.size=4194304, there should be ~16 map tasks spawned.

<name>mapred.tasktracker.tasks.maximum</name>
<value>20</value>

(hdfs-site.xml)
<name>dfs.replication</name>
<value>5</value>

<name>dfs.block.size</name>
<value>4194304</value>

(core-site.xml)
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop/hadoop-datastore/hadoop-${user.name}</value>

<name>fs.default.name</name>
<value>hdfs://hadoop-namenode:54310/</value>
<final>true</final>

Of course, hadoop-namenode and hadoop-jobtracker are both defined in /etc/hosts and they both reference the same IP. No firewall is enabled on the network. The doesn't seem to be any errors output on the datanode/jobtracker's logs, either. Is there something I should be taking into account, that I am currently not? What could be the cause of such poor performance? Overhead due to copying small bits of data through the nodes, perhaps? Any pointers would be generously appreciated.

Error in running hadoop examples

2009-07-16T10:04:25Z

Hi

I am relatively new to using hadoop. After installing hadoop on 3 machines i tried running the word count example one one of the machines running as a single node only. However when i try to tun the word count example using the following command on the terminal:

hadoop@user5:~$ /home/hadoop/Desktop/hadoop/bin/hadoop jar /home/hadoop/Desktop/hadoop/hadoop-0.19.1-examples.jar wordcount gutenberg gut-out

where : hadoop is my user account and gutenberg is where the txt files for the word count example are stored and gut-out is where the result is to be stored

it starts the map-reduce however the recduce gets stuck at 0 % even though map reaches 100 % and the output on the console is as follows. I need help. Have been stuck on this problem since 3 days !

09/07/16 12:32:01 INFO mapred.FileInputFormat: Total input paths to process : 3
09/07/16 12:32:44 INFO mapred.JobClient: Running job: job_200907161230_0001
09/07/16 12:32:45 INFO mapred.JobClient: map 0% reduce 0%
09/07/16 12:33:33 INFO mapred.JobClient: map 1% reduce 0%
09/07/16 12:33:37 INFO mapred.JobClient: map 3% reduce 0%
09/07/16 12:33:54 INFO mapred.JobClient: map 5% reduce 0%
09/07/16 12:33:57 INFO mapred.JobClient: map 7% reduce 0%
09/07/16 12:34:07 INFO mapred.JobClient: map 9% reduce 0%
09/07/16 12:34:14 INFO mapred.JobClient: map 11% reduce 0%
09/07/16 12:34:21 INFO mapred.JobClient: map 12% reduce 0%
09/07/16 12:34:29 INFO mapred.JobClient: map 14% reduce 0%
09/07/16 12:34:37 INFO mapred.JobClient: map 16% reduce 0%
09/07/16 12:34:44 INFO mapred.JobClient: map 18% reduce 0%
09/07/16 12:34:51 INFO mapred.JobClient: map 20% reduce 0%
09/07/16 12:34:58 INFO mapred.JobClient: map 22% reduce 0%
09/07/16 12:35:09 INFO mapred.JobClient: map 24% reduce 0%
09/07/16 12:35:41 INFO mapred.JobClient: map 25% reduce 0%
09/07/16 12:36:01 INFO mapred.JobClient: map 27% reduce 0%
09/07/16 12:36:10 INFO mapred.JobClient: map 29% reduce 0%
09/07/16 12:36:34 INFO mapred.JobClient: map 31% reduce 0%
09/07/16 12:36:58 INFO mapred.JobClient: map 33% reduce 0%
09/07/16 12:37:08 INFO mapred.JobClient: map 35% reduce 0%
09/07/16 12:37:15 INFO mapred.JobClient: map 37% reduce 0%
09/07/16 12:37:29 INFO mapred.JobClient: map 38% reduce 0%
09/07/16 12:37:31 INFO mapred.JobClient: map 40% reduce 0%
09/07/16 12:37:47 INFO mapred.JobClient: map 42% reduce 0%
09/07/16 12:37:48 INFO mapred.JobClient: map 44% reduce 0%
09/07/16 12:38:04 INFO mapred.JobClient: map 46% reduce 0%
09/07/16 12:38:06 INFO mapred.JobClient: map 48% reduce 0%
09/07/16 12:38:22 INFO mapred.JobClient: map 49% reduce 0%
09/07/16 12:38:23 INFO mapred.JobClient: map 51% reduce 0%
09/07/16 12:38:39 INFO mapred.JobClient: map 53% reduce 0%
09/07/16 12:38:40 INFO mapred.JobClient: map 55% reduce 0%
09/07/16 12:39:17 INFO mapred.JobClient: map 59% reduce 0%
09/07/16 12:39:37 INFO mapred.JobClient: Task Id : attempt_200907161230_0001_m_000000_0, Status : FAILED
Too many fetch-failures
09/07/16 12:39:37 WARN mapred.JobClient: Error reading task outputConnection refused
09/07/16 12:39:37 WARN mapred.JobClient: Error reading task outputConnection refused
09/07/16 12:39:43 INFO mapred.JobClient: map 61% reduce 0%
09/07/16 12:40:06 INFO mapred.JobClient: map 64% reduce 0%
09/07/16 12:40:25 INFO mapred.JobClient: map 66% reduce 0%
09/07/16 12:40:27 INFO mapred.JobClient: map 68% reduce 0%
09/07/16 12:40:46 INFO mapred.JobClient: map 70% reduce 0%
09/07/16 12:40:48 INFO mapred.JobClient: map 72% reduce 0%
09/07/16 12:41:06 INFO mapred.JobClient: map 74% reduce 0%
09/07/16 12:41:07 INFO mapred.JobClient: map 75% reduce 0%
09/07/16 12:41:27 INFO mapred.JobClient: map 77% reduce 0%
09/07/16 12:41:28 INFO mapred.JobClient: map 79% reduce 0%
09/07/16 12:41:44 INFO mapred.JobClient: map 81% reduce 0%
09/07/16 12:41:47 INFO mapred.JobClient: map 83% reduce 0%
09/07/16 12:42:03 INFO mapred.JobClient: map 85% reduce 0%
09/07/16 12:42:06 INFO mapred.JobClient: map 87% reduce 0%
09/07/16 12:42:42 INFO mapred.JobClient: map 88% reduce 0%
09/07/16 12:42:45 INFO mapred.JobClient: map 90% reduce 0%
09/07/16 12:43:37 INFO mapred.JobClient: map 92% reduce 0%
09/07/16 12:43:40 INFO mapred.JobClient: map 94% reduce 0%
09/07/16 12:44:30 INFO mapred.JobClient: map 96% reduce 0%
09/07/16 12:44:34 INFO mapred.JobClient: map 98% reduce 0%
09/07/16 12:45:21 INFO mapred.JobClient: map 100% reduce 0%
09/07/16 12:46:27 INFO mapred.JobClient: Task Id : attempt_200907161230_0001_m_000001_0, Status : FAILED
Too many fetch-failures
09/07/16 12:46:27 WARN mapred.JobClient: Error reading task outputConnection refused
09/07/16 12:46:27 WARN mapred.JobClient: Error reading task outputConnection refused
09/07/16 12:46:32 INFO mapred.JobClient: map 98% reduce 0%
09/07/16 12:46:34 INFO mapred.JobClient: map 100% reduce 0%
09/07/16 12:52:46 INFO mapred.JobClient: Task Id : attempt_200907161230_0001_m_000002_0, Status : FAILED
Too many fetch-failures
09/07/16 12:52:46 WARN mapred.JobClient: Error reading task outputConnection refused
09/07/16 12:52:46 WARN mapred.JobClient: Error reading task outputConnection refused
09/07/16 12:52:51 INFO mapred.JobClient: map 98% reduce 0%
09/07/16 12:53:01 INFO mapred.JobClient: map 100% reduce 0%
09/07/16 12:59:02 INFO mapred.JobClient: Task Id : attempt_200907161230_0001_m_000003_0, Status : FAILED
Too many fetch-failures
09/07/16 12:59:02 WARN mapred.JobClient: Error reading task outputConnection refused
09/07/16 12:59:02 WARN mapred.JobClient: Error reading task outputConnection refused
09/07/16 12:59:07 INFO mapred.JobClient: map 98% reduce 0%
09/07/16 12:59:15 INFO mapred.JobClient: map 100% reduce 0%

Re: could only be replicated to 0 nodes, instead of 1

2009-07-13T03:25:02Z

The full error message is
09/07/02 16:28:09 WARN hdfs.DFSClient: NotReplicatedYetException sleeping /user/hadoop/count/count/temp1 retries left 1
09/07/02 16:28:12 WARN hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /user/hadoop/count/count/temp1 could only be replicated to 0 nodes, instead of 1
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1280)
at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

at org.apache.hadoop.ipc.Client.call(Client.java:697)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
at $Proxy0.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy0.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2814)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2696)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1996)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)

could only be replicated to 0 nodes, instead of 1

2009-07-13T03:20:37Z

Hi, All

I just start to use Hadoop few days ago. I met the error message
" WARN hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /user/hadoop/count/count/temp1 could only be replicated to 0 nodes, instead of 1"
while trying to copy data files to DFS after Hadoop is started.

I did all the settings according to the "Running_Hadoop_On_Ubuntu_Linux_(Single-Node_Cluster)"'s instruction, and I don't know what's wrong. Besides, during the process, no error message is written to log files.

Also, according to "http://localhost.localdomain:50070/dfshealth.jsp", I have one live namenode. By the broswer, I even can see the first data file is created in DFS, but the size of it is 0.

Things I've tried:
1. Stop hadoop, re-format DFS and start hadoop again.
2. Change "localhost" to "127.0.0.1"

But neigher of them works.

Could anyone help me or give me a hint?

Thanks.

Anthony

Re: SVN gone ?

2009-06-30T14:28:34Z

Yes I seem to be an idiot https.... However the page refer to http

/M

On Tue, Jun 30, 2009 at 11:25 PM, Marcus Herou
<marcus.herou@...>wrote:

> Am I total moron or have the Subversion repo gone fishing ?
>
> I noticed that yesterday when I did a svn up.
>
> I get a 404 on this url: http://svn.apache.org/repos/asf/hadoop/core/
> which is refferred to from:
> http://hadoop.apache.org/core/version_control.html
>
> Cheers
>
> /Marcus
>
> --
> Marcus Herou CTO and co-founder Tailsweep AB
> +46702561312
> marcus.herou@...
> http://www.tailsweep.com/
>
>

--
Marcus Herou CTO and co-founder Tailsweep AB
+46702561312
marcus.herou@...
http://www.tailsweep.com/

SVN gone ?

2009-06-30T14:25:09Z

Am I total moron or have the Subversion repo gone fishing ?

I noticed that yesterday when I did a svn up.

I get a 404 on this url: http://svn.apache.org/repos/asf/hadoop/core/
which is refferred to from:
http://hadoop.apache.org/core/version_control.html

Cheers

/Marcus

--
Marcus Herou CTO and co-founder Tailsweep AB
+46702561312
marcus.herou@...
http://www.tailsweep.com/

Can I post pig questions on this forum?

2009-06-26T16:34:05Z

hadoop lucene integration

2009-06-25T21:53:40Z

hi all

I've work experience with lucene , but am new to hadoop , i created a index by lucene , please any1 tell me how to use hadoop for my lucene index for distributed file system , if possible can any1 send me an example or the link in which i can use it for my index. Please .

BlockAlreadyExistsException

2009-05-15T18:35:33Z

when I used nutch1.0 fetch data to hadoop,and I have 1 mater 10 clusters all with 4G Memory,1T Hard disk of ubuntu system.
my config is
master:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

<property>
<name>dfs.block.size</name>
<value>134217728</value>
<final>true</final>
</property>
<property>
<name>dfs.data.dir</name>
<value>/data/filesystem/data</value>
<final>true</final>
</property>
<property>
<name>dfs.datanode.du.reserved</name>
<value>1073741824</value>
<final>true</final>
</property>
<property>
<name>dfs.datanode.handler.count</name>
<value>3</value>
<final>true</final>
</property>
<property>
<name>dfs.name.dir</name>
<value>/data/filesystem/namenode</value>
<final>true</final>
</property>
<property>
<name>dfs.namenode.handler.count</name>
<value>5</value>
<final>true</final>
</property>
<property>
<name>dfs.permissions</name>
<value>True</value>
<final>true</final>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>fs.checkpoint.dir</name>
<value>/data/filesystem/secondary-nn</value>
<final>true</final>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://ubuntu76:9000</value>
</property>
<property>
<name>fs.trash.interval</name>
<value>1440</value>
<final>true</final>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/tmp/hadoop-${user.name}</value>
<final>true</final>
</property>
<property>
<name>io.file.buffer.size</name>
<value>65536</value>
</property>
<property>
<name>mapred.child.java.opts</name>
<value>-Xmx1945m</value>
</property>
<property>
<name>mapred.child.ulimit</name>
<value>3983360</value>
<final>true</final>
</property>
<property>
<name>mapred.job.tracker</name>
<value>ubuntu76:9001</value>
</property>
<property>
<name>mapred.job.tracker.handler.count</name>
<value>5</value>
<final>true</final>
</property>
<property>
<name>mapred.local.dir</name>
<value>${hadoop.tmp.dir}/mapred/local</value>
<final>true</final>
</property>
<property>
<name>mapred.map.tasks.speculative.execution</name>
<value>true</value>
</property>
<property>
<name>mapred.reduce.parallel.copies</name>
<value>10</value>
</property>
<property>
<name>mapred.reduce.tasks</name>
<value>10</value>
</property>
<property>
<name>mapred.reduce.tasks.speculative.execution</name>
<value>false</value>
</property>
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<value>1</value>
<final>true</final>
</property>
<property>
<name>mapred.tasktracker.reduce.tasks.maximum</name>
<value>1</value>
<final>true</final>
</property>
<property>
<name>tasktracker.http.threads</name>
<value>12</value>
<final>true</final>
</property>
</configuration>

slaver
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

<property>
<name>dfs.block.size</name>
<value>134217728</value>
<final>true</final>
</property>
<property>
<name>dfs.data.dir</name>
<value>/data/filesystem/data</value>
<final>true</final>
</property>
<property>
<name>dfs.datanode.du.reserved</name>
<value>1073741824</value>
<final>true</final>
</property>
<property>
<name>dfs.datanode.handler.count</name>
<value>3</value>
<final>true</final>
</property>
<property>
<name>dfs.name.dir</name>
<value>/data/filesystem/namenode</value>
<final>true</final>
</property>
<property>
<name>dfs.namenode.handler.count</name>
<value>5</value>
<final>true</final>
</property>
<property>
<name>dfs.permissions</name>
<value>True</value>
<final>true</final>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>fs.checkpoint.dir</name>
<value>/data/filesystem/secondary-nn</value>
<final>true</final>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://ubuntu76:9000</value>
</property>
<property>
<name>fs.trash.interval</name>
<value>1440</value>
<final>true</final>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/tmp/hadoop-${user.name}</value>
<final>true</final>
</property>
<property>
<name>io.file.buffer.size</name>
<value>65536</value>
</property>
<property>
<name>mapred.child.java.opts</name>
<value>-Xmx1945m</value>
</property>
<property>
<name>mapred.child.ulimit</name>
<value>3983360</value>
<final>true</final>
</property>
<property>
<name>mapred.job.tracker</name>
<value>ubuntu76:9001</value>
</property>
<property>
<name>mapred.job.tracker.handler.count</name>
<value>5</value>
<final>true</final>
</property>
<property>
<name>mapred.local.dir</name>
<value>/data/filesystem/mapred/local</value>
<final>true</final>
</property>
<property>
<name>mapred.map.tasks.speculative.execution</name>
<value>true</value>
</property>
<property>
<name>mapred.reduce.parallel.copies</name>
<value>10</value>
</property>
<property>
<name>mapred.reduce.tasks</name>
<value>10</value>
</property>
<property>
<name>mapred.reduce.tasks.speculative.execution</name>
<value>false</value>
</property>
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<value>1</value>
<final>true</final>
</property>
<property>
<name>mapred.tasktracker.reduce.tasks.maximum</name>
<value>1</value>
<final>true</final>
</property>
<property>
<name>tasktracker.http.threads</name>
<value>12</value>
<final>true</final>
</property>
</configuration>

but when I excute command "bin/nutch crawl urls -dir crawled -depth 3" in datanode logs file have exception like this:
009-05-15 21:04:10,328 ERROR datanode.DataNode - DatanodeRegistration(113.45.58.77:50010, storageID=DS-1293122987-113.45.58.77-50010-1242435813708, infoPort=50075, ipcPort=50020):DataXceiver
org.apache.hadoop.hdfs.server.datanode.BlockAlreadyExistsException: Block blk_5260319812111246094_1002 is valid, and cannot be written to.
at org.apache.hadoop.hdfs.server.datanode.FSDataset.writeToBlock(FSDataset.java:975)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:97)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:259)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103)
at java.lang.Thread.run(Thread.java:619)
2009-05-15 21:04:39,750 WARN datanode.DataNode - DatanodeRegistration(113.45.58.77:50010, storageID=DS-1293122987-113.45.58.77-50010-1242435813708, infoPort=50075, ipcPort=50020):Failed to transfer blk_9220934476097358434_1017 to 113.45.58.78:50010 got java.net.SocketException: Original Exception : java.io.IOException: Connection reset by peer
at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:418)
at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:519)
at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:199)
at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendChunks(BlockSender.java:313)
at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:400)
at org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:1108)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: Connection reset by peer
... 8 more

Re: hadoop-a small doubt

2009-04-20T22:06:05Z

What is the exact purpose that you want a system not in hadoop cluster to access the namenode or datanode? If it is simply to write data to HDFS from local system and then to copy back data from HDFS to local system simply use hadoop file system's shell commands.

Hope this helps!

deepya wrote:

Hi,
I am SreeDeepya doing MTech in IIIT.I am working on a project named cost effective and scalable storage server.I configured a small hadoop cluster with only two nodes one namenode and one datanode.I am new to hadoop.
I have a small doubt.

Can a system not in the hadoop cluster access the namenode or the datanode????If yes,then can you please tell me the necessary configurations that has to be done.

Thanks in advance.

SreeDeepya

Copying files from HDFS to remote database

2009-04-20T21:40:43Z

Our application is using hadoop to parallelize jobs across ec2 cluster. HDFS is used to store output files. How would you ideally copy output files from HDFS to remote databases?

Thanks,
Parul V. Kudtarkar

Not able to subscribe to pig user/dev mailing list

2009-04-20T06:14:16Z

Hi all,
I am not able to subscribe to pig mailing list (both dev and user). Here is the error message that I am getting when I tried to confirm the subscribtion.

Your message did not reach some or all of the intended recipients.

Subject: pig-dev-sc.1239701669.ohbefaiphgajdgbcjmjg-pallavi.palleti=corp.aol.com@hadoop.apache.org
Sent: 14/04/2009 3:11 PM

The following recipient(s) cannot be reached:

pig-dev-request@hadoop.apache.org on 14/04/2009 3:12 PM
The message could not be delivered because the recipient's mailbox is full.
< omr-m35.mx.aol.com #5.2.2 SMTP; 552 spam score (6.2) exceeded threshold>

I tried couple of times so far but not able to register. Did any one else face same issue?

Thanks
Pallavi

Re: hadoop-a small doubt

2009-04-15T08:56:43Z

Hey ,
You can do that.That system should have same usrname like those of cluster and ofcourse it should be able to ssh name node.Also it should have hadoop and its hadoop-site.xml should be similar .Then u can access namenode,hdfs etc.

if you are willing to see the web interface that can be done easily using any system.

deepya wrote:

Hi,
I am SreeDeepya doing MTech in IIIT.I am working on a project named cost effective and scalable storage server.I configured a small hadoop cluster with only two nodes one namenode and one datanode.I am new to hadoop.
I have a small doubt.

Can a system not in the hadoop cluster access the namenode or the datanode????If yes,then can you please tell me the necessary configurations that has to be done.

Thanks in advance.

SreeDeepya

hadoop-a small doubt

2009-03-28T22:29:24Z

Hi,
I am SreeDeepya doing MTech in IIIT.I am working on a project named cost effective and scalable storage server.I configured a small hadoop cluster with only two nodes one namenode and one datanode.I am new to hadoop.
I have a small doubt.

Can a system not in the hadoop cluster access the namenode or the datanode????If yes,then can you please tell me the necessary configurations that has to be done.

Thanks in advance.

SreeDeepya

streaming inputformat: class not found

2009-03-10T10:30:16Z

Hello,

I'm try to run a mapreduce job on a data file in which the keys and values alternate rows. E.g.

key1
value1
key2
...

I've written my own InputFormat by extending FileInputFormat (the code for this class is below.) The problem is that when I run hadoop streaming with the command

bin/hadoop jar contrib/streaming/hadoop-0.18.3-streaming.jar -mapper mapper.pl -reducer org.apache.hadoop.mapred.lib.IdentityReducer -input test.data -output test-output -file <pathToMapper.pl> -inputformat MyFormatter

I get the error

-inputformat : class not found : MyFormatter
java.lang.RuntimeException: -inputformat : class not found : MyFormatter
at org.apache.hadoop.streaming.StreamJob.fail(StreamJob.java:550)
...

I have tried putting .java, .class, and .jar file of MyFormatter in the job jar using the -file parameter. I have also tried putting them on the hdfs using -copyFromLocal, but I still get the same error. Can anyone give me some hints as to what the problem might be? Also, I tried to hack together my formatter based on the hadoop examples, so does it seems like it should properly process the input files I described above?

Trevis

<imports ommitted>

public final class MyFormatter extends org.apache.hadoop.mapred.FileInputFormat<Text, Text>

{

@Override
public RecordReader<Text, Text> getRecordReader( InputSplit split,
JobConf job, Reporter reporter ) throws IOException
{
return new MyRecordReader( job, (FileSplit) split );
}

static class MyRecordReader implements RecordReader<Text, Text>
{
private LineRecordReader _in = null;
private LongWritable _junk = null;

public FastaRecordReader( JobConf job, FileSplit split ) throws IOException
{
_junk = new LongWritable();

_in = new LineRecordReader( job, split );
}

@Override
public void close() throws IOException
{
_in.close();
}

@Override
public Text createKey()
{
return new Text();
}

@Override
public Text createValue()
{
return new Text();
}

@Override
public long getPos() throws IOException
{
return _in.getPos();
}

@Override
public float getProgress() throws IOException
{
return _in.getProgress();
}

@Override
public boolean next( Text key, Text value ) throws IOException
{
if ( _in.next( _junk, key ) )
{
if ( _in.next( _junk, value ) )
{
return true;
}
}

key.clear();
value.clear();

return false;
}
}
}

Re: bypass ssh to upload files on HDFS

2008-11-28T07:56:59Z

thanks for your reply, can you suggest the file in hadoop which i can make changes to irradicate ssh.

I'm not sure if this is what you meant , but you can start an http service on on of the hdfs machines and use it upload files from the local machine to the HDFS , without using ssh. (remoteServer -> httpService on local , submit to local HDFS)

jas69 wrote:

Hi,

Please suggest me a way out. My interest is to bypass ssh and upload files from local filesystem to HDFS without the use of ssh service.

Regards.

Re: bypass ssh to upload files on HDFS

2008-11-27T10:06:06Z

I'm not sure if this is what you meant , but you can start an http service on on of the hdfs machines and use it upload files from the local machine to the HDFS , without using ssh. (remoteServer -> httpService on local , submit to local HDFS)

jas69 wrote:

Hi,

Please suggest me a way out. My interest is to bypass ssh and upload files from local filesystem to HDFS without the use of ssh service.

Regards.

bypass ssh to upload files on HDFS

2008-11-27T09:36:19Z

Hi,

Please suggest me a way out. My interest is to bypass ssh and upload files from local filesystem to HDFS without the use of ssh service.

Regards.

Block placement in HDFS

2008-11-24T18:59:28Z

Hi everyone,

I was wondering whether it is possible to control the placement of the blocks of a file in HDFS. Is it possible to instruct HDFS about which nodes will hold the block replicas?

Thanks!

Re: "could only be replicated to 0 nodes, instead of 1"

2008-11-13T12:28:32Z

Hi,
If you are getting this in windows environment (2003 64 bit). We have faced the same problem. Now we tried the following steps and it started working.
1)Install cygwin and ssh.
2) Downloaded the stable version Hadoop - hadoop-0.17.2.1.tar.gz as on 13/Nov/2008
3) Untar it via cygwin (tar xvfz hadoop-0.17.2.1.tar.gz). please DONOT use WINZIP to untar.
4) We tried running the sudo distribution example provided in quickstart (http://hadoop.apache.org/core/docs/current/quickstart.html) and it worked.

Thanks
Arul and Limin
eBay Inc.,

jerrro wrote:

I am trying to install/configure hadoop on a cluster with several computers. I followed exactly the instructions in the hadoop website for configuring multiple slaves, and when I run start-all.sh I get no errors - both datanode and tasktracker are reported to be running (doing ps awux | grep hadoop on the slave nodes returns two java processes). Also, the log files are empty - nothing is printed there. Still, when I try to use bin/hadoop dfs -put,
I get the following error:

# bin/hadoop dfs -put w.txt w.txt
put: java.io.IOException: File /user/scohen/w4.txt could only be replicated to 0 nodes, instead of 1

and a file of size 0 is created on the DFS (bin/hadoop dfs -ls shows it).

I couldn't find much information about this error, but I did manage to see somewhere it might mean that there are no datanodes running. But as I said, start-all does not give any errors. Any ideas what could be problem?

Thanks.

Jerr.

Hadoop Streaming - running a jar file

2008-11-11T10:50:35Z

Hi

I have a jar file which takes input from stdin and writes something on stdout. i.e. When I run

java -jar A.jar < input

It prints the required output.

However, when I run it as a mapper in hadoop streaming using the command

$HADOOP_HOME/bin/hadoop jar ....streaming.jar -input .. -output ... -mapper 'java -jar A.jar' -reducer NONE

i get the broken pipe exception.

the error message is

additionalConfSpec_:null
null=@@@userJobConfProps_.get(stream.shipped.hadoopstreaming
packageJobJar: [/mnt/hadoop/HADOOP/hadoop-0.16.3/tmp/dir/hadoop-hadoop/hadoop-unjar45410/] [] /tmp/streamjob45411.jar tmpDir=null
08/11/11 23:20:14 INFO mapred.FileInputFormat: Total input paths to process : 1
08/11/11 23:20:14 INFO streaming.StreamJob: getLocalDirs(): [/mnt/hadoop/HADOOP/hadoop-0.16.3/tmp/mapred]
08/11/11 23:20:14 INFO streaming.StreamJob: Running job: job_200811111724_0014
08/11/11 23:20:14 INFO streaming.StreamJob: To kill this job, run:
08/11/11 23:20:14 INFO streaming.StreamJob: /mnt/hadoop/HADOOP/hadoop-0.16.3/bin/../bin/hadoop job -Dmapred.job.tracker=10.105.41.25:54311 -kill job_200811111724_0014
08/11/11 23:20:15 INFO streaming.StreamJob: Tracking URL: http://sayali:50030/jobdetails.jsp?jobid=job_200811111724_0014
08/11/11 23:20:16 INFO streaming.StreamJob: map 0% reduce 0%
08/11/11 23:21:00 INFO streaming.StreamJob: map 100% reduce 100%
08/11/11 23:21:00 INFO streaming.StreamJob: To kill this job, run:
08/11/11 23:21:00 INFO streaming.StreamJob: /mnt/hadoop/HADOOP/hadoop-0.16.3/bin/../bin/hadoop job -Dmapred.job.tracker=10.105.41.25:54311 -kill job_200811111724_0014
08/11/11 23:21:00 INFO streaming.StreamJob: Tracking URL: http://sayali:50030/jobdetails.jsp?jobid=job_200811111724_0014
08/11/11 23:21:00 ERROR streaming.StreamJob: Job not Successful!
08/11/11 23:21:00 INFO streaming.StreamJob: killJob...
Streaming Job Failed!

Could some one please help me with any ideas or pointers.

regards
Amit

How does an offline Datanode come back up ?

2008-10-27T10:22:46Z

Hi All,

Ive been working michael nolls multi-node cluster setup example (Running_Hadoop_On_Ubuntu_Linux) for hadoop and I have a working setup. I then on my slave machine -- which is currently running a datanode killed the process in an effort to try to simulate some sort of failure on the slave machine datanode. I had assumed that the namenode would have been polling its datanodes and thus attempted to bring up any node that goes down. On looking at my slave machine it seems that the datanode process is still down (I've checked jps).

Obviously im missing something ! Does hadoop look after its datanodes ? Is there a config setting that i may have missed ? Do I need to create some sort of external tool to pool and attempt to bring up nodes that have gone down ?

Thanks
Will

Hadoop with image processing

2008-10-15T07:31:54Z

!!! MEMBERS OF core-user@hadoop.apache.org, DON'T READ THIS
----------------

Hi to all, I started to work on a hadoop-based project.
In our application, there are a huge number of images with a regular pattern, differing in 4 parts/blocks.
System takes an image as input and looks for a similar image, considering if all these 4 parts match.
(System finds all the matches, even after finding one).
Each of these parts are independent, result of each part computed separately, these are
printed on the screen and then an average matching percentage is calculated from these.

(I can write more detailed information if needed)

Could you suggest a structure? or any ideas to have a better result?

Images can be divided into 4 parts, I see that. But folder structure of images are important and
I have no idea with that. Images are kept in DB (can be changed, if folder structure is better)
Is two stage of map-reduce operations better? First, one map-reduce for each image,
then a second map-reduce for every part of one image.
But as far as I know, the slowest computation slows down whole operation.

Thanks in advance..

Searching Lucene Index built using Hadoop

2008-10-06T10:26:43Z

I'm trying to index a large dataset using Hadoop+Lucene. I used the example under hadoop/trunk/src/conrib/index/ for indexing. I'm unable to find a way to search the index that was successfully built.

I tried copying over the index to one machine and merging them using IndexWriter.addIndexesNoOptimize().

I would like hear your input on the best way to index+search large datasets.

Thanks,
Saranath