MRUnit test case for Driver - hadoop

Question

I have written MRunit with following code:
Configuration conf = new Configuration();
conf.set("fs.defaultFS", "file:///");
conf.set("fs.default.name", "file:///");
conf.set("mapreduce.framework.name", "local");
conf.setInt("mapreduce.task.io.sort.mb", 1);
Path input = new Path("input/ncdc/micro");
Path output = new Path("output");
FileSystem fs = FileSystem.getLocal(conf);
fs.delete(output, true); // delete old output
VisitedItemFlattenDriver driver = new VisitedItemFlattenDriver();
driver.setConf(conf);
int exitCode = driver.run(new String[] {
input.toString(), output.toString(), "false" });
But when I execute the Junit test case from eclipse. I 'm getting exception as below:-
java.lang.NullPointerException
at java.lang.ProcessBuilder.start(ProcessBuilder.java:441)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:404)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:678)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:661)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:639)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:435)
at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:277)
at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:125)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:344)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1286)
at biz.ds.www.preprocess.visiteditem.VisitedItemFlattenDriver.run(VisitedItemFlattenDriver.java:69)
I'm not sure what is causing this error as I just intent to unit test my class:
public class VisitedItemFlattenDriver extends Configured implements Tool {
...}
I deeply appreciate if some one when guide how to resolve the error.

Answer 1

I tried a couple of options to resolve the problem and spend many hours to do so..
Firstly, I searched for an option and found to add winutils.exe, and .dll files to hadoop/bin. I tried the step and also set HADOOP_HOME environment variable.
Somehow above mentioned error got resolved and I was then stuck in to an different error like below:
java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
It was obvious that error is due to some compatibility issues. But then I did some searching and found that it can be resolved if we upgrade JRE from 32 bit to 64 bit.
Earlier I was using JDK 6 32 bit and then I updated it to JDK 6 64 bit. It did not resolved my problem. I also tried to use minidfscluster for MR unit but that to gave same error.
But then I used JDK 7 64 bit for my code and the problem was resolved and it ran successfully.
** Note: I 'm using Hadoop version 2.2.0