« Return to Thread: does anyone have idea on how to run multiple sequential jobs with bash script

does anyone have idea on how to run multiple sequential jobs with bash script

by Richard Zhang-2 :: Rate this Message:

Reply to Author | View in Thread

Hello folks:
I am running several hadoop applications on hdfs. To save the efforts in
issuing the set of commands every time, I am trying to use bash script to
run the several applications sequentially. To let the job finishes before it
is proceeding to the next job, I am using wait in the script like below.

sh bin/start-all.sh
wait
echo cluster start
(bin/hadoop jar hadoop-0.17.0-examples.jar randomwriter -D
test.randomwrite.bytes_per_map=107374182 rand)
wait
bin/hadoop jar hadoop-0.17.0-examples.jar randomtextwriter  -D
test.randomtextwrite.total_bytes=107374182 rand-text
bin/stop-all.sh
echo finished hdfs randomwriter experiment


However, it always give the error like below. Does anyone have better idea
on how to run the multiple sequential jobs with bash script?

HadoopScript.sh: line 39: wait: pid 10 is not a child of this shell

org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.mapred.JobTracker$IllegalStateException: Job tracker still
initializing
        at
org.apache.hadoop.mapred.JobTracker.ensureRunning(JobTracker.java:1722)
        at
org.apache.hadoop.mapred.JobTracker.getNewJobId(JobTracker.java:1730)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)

        at org.apache.hadoop.ipc.Client.call(Client.java:557)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
        at $Proxy1.getNewJobId(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
        at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
        at $Proxy1.getNewJobId(Unknown Source)
        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:696)
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:973)
        at
org.apache.hadoop.examples.RandomWriter.run(RandomWriter.java:276)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at
org.apache.hadoop.examples.RandomWriter.main(RandomWriter.java:287)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
        at
org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
        at
org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:53)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
        at org.apache.hadoop.mapred.JobShell.run(JobShell.java:194)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
        at org.apache.hadoop.mapred.JobShell.main(JobShell.java:220)

 « Return to Thread: does anyone have idea on how to run multiple sequential jobs with bash script