How to eliminate Error util.Shell: Failed to locate the winutils binary - hadoop

Question

I am executing a remote job from a windows machine(the client) under eclipse, I clarify that I dont have any hadoop installation on my windows client, and I dont needed, I am executing the hadoop job remotely, and hadoop is installed on a linux machine.
Everything is executed correctly, but I would like to get rid of this ERROR:
14/09/22 11:49:49 ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:355)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:370)
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:363)
at sun.misc.Unsafe.ensureClassInitialized(Native Method)
at sun.reflect.UnsafeFieldAccessorFactory.newFieldAccessor(Unknown Source)
at sun.reflect.ReflectionFactory.newFieldAccessor(Unknown Source)
at java.lang.reflect.Field.acquireFieldAccessor(Unknown Source)
at java.lang.reflect.Field.getFieldAccessor(Unknown Source)
at java.lang.reflect.Field.set(Unknown Source)
at MyFirstJob.main(MyFirstJob.java:45)
Do you know how to make this exception not hapenning ?

Answer 1

Install the winutils.exe, there is no other way of fixing this error.
Here is a little context: Hadoop will write some files locally (e.g. the job configs) before uploading them to the cluster. Thus it will need to set permissions, write some files or create directories.
In case it doesn't find the binary, it will fallback to the Java implementations anyway, so you don't need to worry. However, there is no built-in configuration to turn this message off, so the only way to really fixing it is to recompile your hadoop-common jar without this error (I guess installing winutils isn't that bad compared to it).

Answer 2

Copy org.apache.hadoop.util.Shell.java into your project.
You can comment out the below line,to remove the Error.
throw new IOException("Could not locate executable " + fullExeName + " in the Hadoop binaries.");
Also for Windows check,
Error while running Mapreduce(yarn)from windows eclipse

Answer 3

I saw a suggestion somewhere to just create an empty file with that name, to get rid of the error. I think I tried it once and it worked - feel free to try if it works for you. The file can be created on-the-fly if needed.