Hadoop Remote file creation fails - hadoop

Question

I am trying to create a file in HDFS from outside the cluster (HDFS apis), like this:
Configuration conf = new Configuration();
conf.set("mapred.job.tracker", "192.168.56.101:54310");
conf.set("fs.default.name", "hdfs://192.168.56.101:54311");
FileSystem fs = FileSystem.get(conf);
fs.createNewFile(new Path("/app/hadoop/tmp/data/tools.txt"));
Getting error:
Exception in thread "main" org.apache.hadoop.ipc.RemoteException: java.io.IOException: Unknown protocol to job tracker: org.apache.hadoop.hdfs.protocol.ClientProtocol
at org.apache.hadoop.mapred.JobTracker.getProtocolVersion(JobTracker.java:370)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:622)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1432)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1428)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1426)
at org.apache.hadoop.ipc.Client.call(Client.java:1113)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
at $Proxy1.getProtocolVersion(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
at $Proxy1.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.checkVersion(RPC.java:422)
at org.apache.hadoop.hdfs.DFSClient.createNamenode(DFSClient.java:183)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:281)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:245)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:100)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1446)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1464)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:263)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:124)
at LineCounter.main(LineCounter.java:107)

Answer 1

Namenode exposes two ports one is ipc port(8020 by default) and WebUI(50070 default). For accessing HDFS filesystem from a remote machine no need to set the property mapred.job.tracker only fs.default.name would be sufficient and make sure that you are using the correct hdfs dependant library versions as cluster and namenode is accessible from the remote machine.
Before connecting to remote hdfs find the value of fs.default.name(8020 default) by checking core-site.xml file from anynodes or edgenode in the cluster. You can make sure the ipc port is correct or not by executing the following command in any of the nodes
hadoop fs -ls hdfs://192.168.56.101:54310/
If you are able to access hdfs you can give (above hdsf URI) in conf.set method

Answer 2

Can you please check if Name node and Jobtracker deamons are running and ports you have mentioned are correct.

Answer 3

I use the following and it works for me on a single node cluster on Linux VM:
Configuration conf = new Configuration();
conf.set("fs.default.name", "hdfs://localhost:9000");
I would suggest you try 9000 as the port number, else find out the correct port number that needs to be used from your Hadoop configuration file "site-core.xml". My file is something like this:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>~/hacking/hd-data/tmp</value>
</property>
<property>
<name>fs.checkpoint.dir</name>
<value>~/hacking/hd-data/snn</value>
</property>
</configuration>
Also, as sachinjose pointed out, I would suggest too that you remove the following from your code:
conf.set("mapred.job.tracker", "192.168.56.101:54310");