Re: Error reporting from map function

by ojh06 :: Rate this Message:

Hi,
Well I've figured some of it out. I needed to initialise the new
JobClient after setting its configuration. So the code looks like:
JobClient aJC = new JobClient();
String jobid = jobConf.get("mapred.job.id");
aJC.setConf(jobConf);
aJC.init();
This works fine in the Map function for killing the job. However, I'm
still getting an exception in the launch function, on the
getMapTasksReports() line, albeit at a deeper level. THe exception I
get is:
org.apache.hadoop.ipc.RemoteException: java.io.IOException:
java.lang.NullPointerException
at java.util.TreeMap.compare(TreeMap.java:1093)
at java.util.TreeMap.getEntry(TreeMap.java:347)
at java.util.TreeMap.get(TreeMap.java:265)
at
org.apache.hadoop.mapred.JobTracker.getMapTaskReports(JobTracker.java:1522)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:341)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:573)

at org.apache.hadoop.ipc.Client.call(Client.java:471)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:165)
at $Proxy1.getMapTaskReports(Unknown Source)
at
org.apache.hadoop.mapred.JobClient.getMapTaskReports(JobClient.java:505)
at imperial.oliverhaggarty.RTAMapRed1a.launch(RTAMapRed1a.java:344)
at imperial.oliverhaggarty.RTAMapRed1a.main(RTAMapRed1a.java:695)
I've called runJob() in a separate thread, as I thought once the job
had finished the set of TaskReports might get deleted. I've also put a
sleep in before I start checking for TaskReports to give time for
everything to initialise. I've copied the code below. Hopefully one of
you Hadoop guru's can point me in the right direction!

Cheers,
Ollie

Code:
Thread t = new Thread() {
public void run() {
System.out.println("In thread");
try {
JobClient.runJob(jobConf);
}
catch(IOException e) {
e.printStackTrace();
}
}
};
t.start();
System.out.println("Afterrunningthread");
try {
Thread.sleep(20000);
}
catch(InterruptedException e) {
e.printStackTrace();
}
JobClient aJC = new JobClient();
String jobid = jobConf.get("mapred.job.id");
aJC.setConf(jobConf);
aJC.init();
TaskReport [] treps = null;
do {
try {
/*GetExcetpion here-->*/ treps = aJC.getMapTaskReports(jobid);
}
catch(Exception e) {
e.printStackTrace();
break;
}
for(TaskReport trep : treps) {
System.out.println(trep.getState());
}
try {
Thread.sleep(1000);
}
catch (InterruptedException e) {
e.printStackTrace();
}
}
while(treps != null && treps.length > 0);

Quoting ojh06@...:

> Hi Michael,
> Thanks for the reply. I've tried to write some code to do this now but
> its not working. I was wondering if there's anything obviously wrong?
> After my runJob() I put (just as a test):
>
> JobClient aJC = new JobClient();
> String jobid = jobConf.get("mapred.job.id");
> aJC.setConf(jobConf); //I've tried with and without this line
> TaskReport [] treps = aJC.getMapTaskReports(jobid);
> for(TaskReport trep : treps) {
> System.out.println(trep.getState());
> }
>
> However, when I run it, I get a NullPointerException on the
> aJC.getMapTasksReports() line. I know its getting the correct jobid.
>
> I've also tried similar code in my Map function for killing the class,
> but I get a NullPointerException when I try and do:
> RunningJob rj = aJC.getJob(jobid);
>
> I'm thinking the new JobClient class needs to be connected to the
> JobTracker in some way? If so, could someone explain how this is done?
> Or am I way off?
>
> Thanks,
> Ollie
>
> Quoting Michael Bieniosek <michael@...>:
>
>>
>> On 8/2/07 5:20 AM, "ojh06@..." <ojh06@...> wrote:
>>
>>> I've found the
>>> getMapTaskReports method in the JobClient class, but can't work out
>>> how to access it other than by creating a new instance of JobClient -
>>> but then that JobClient would be a differnt one to the one that was
>>> running my job, so would access a different set of TaskReports?
>>
>> That doesn't matter -- jobs are bound to jobtrackers, not to jobclients.
>> You can create a new JobClient and access all the jobs that the jobtracker
>> knows about.
>>
>> -Michael
>>
>>

« Return to Thread: Error reporting from map function