I started following an online tutorial to configure multi ndoes on my single local VM. here is the hosts on master node:
127.0.0.1 localhost
192.168.96.132 hadoop
192.168.96.135 hadoop1
192.168.96.136 hadoop2
ssh:ALL:allow
sshd:ALL:allow
Here is the command that used to work:hdfs dfs -ls
Now I am seeing error message below:
ls: Call From hadoop/192.168.96.132 to hadoop:9000 failed on connection exception:
java.net.ConnectException: Connection refused;
For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
What is wrong with my configuration? where should I check and correct it?
Thank you very much.
First try to
ping hadoop,
ping hadoop1 and
ping hadoop2.
Ex: ping hadoop
Then just try to connect via ssh
The syntax is
ssh username#hadoop
ssh username#hadoop1
ssh username#hadoop2
Then see the results to find out whether the systems are connecting or not.
Related
I'm trying to set up two types of hadoop clusters: one standalone via SSH localhost and the other in aws ec2.
Both fail for similar issues: a connection refused error.
Here are some pictures of the issues: This is the result of ssh localhost
The next is: the failed run.
This is the relevenat portion of ~/.ssh/config
I can run hadoop, hdfs, yarn, and all the other commands. But, when I actually type this and run it, it fails:
Of note, I'm following this tutorial for the aws ec2 cluster, (this command is almost at the end). https://awstip.com/setting-up-multi-node-apache-hadoop-cluster-on-aws-ec2-from-scratch-2e9caa6881bd
Which is failling on this command: scp hadoop-env.sh core-site.xml hdfs-site.xml mapred-site.xml yarn-site.xml ubuntu#ec2-54-209-221-47.compute-1.amazonaws.com:/home/ubuntu/hadoop/conf
That's not my ec2 link; it's from the example, but that's where it's faiing with the same error as the 2nd and 4th pictures.
I have hadoop installed in pseudo-distributed mode.
When running the command
hadoop fs -ls
I am getting the following error:
ls: Call From kali/127.0.1.1 to localhost:9000 failed on connection exception:
java.net.ConnectException: Connection refused;
For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
Any suggestions?
If you read the link in the error, I see two immediate points that need addressed.
If the error message says the remote service is on "127.0.0.1" or "localhost" that means the configuration file is telling the client that the service is on the local server. If your client is trying to talk to a remote system, then your configuration is broken.
You should treat pseudodistributed mode as a remote system, even if it is only running locally.
For HDFS, you can resolve that by putting your computer hostname (preferably the full FQDN for your domain), as the HDFS address in core-site.xml. For your case, hdfs://kali:9000 should be enough
Check that there isn't an entry for your hostname mapped to 127.0.0.1 or 127.0.1.1 in /etc/hosts (Ubuntu is notorious for this).
I'm not completely sure why it needs removed, but the general answer I can think of is that Hadoop is a distributed system, and as I mentioned, treat the pseudodistributed mode as if it's remote HDFS server. Therefore, no loopback addresses should use your computers hostname
For example, remove the second line of this
127.0.0.1 localhost
127.0.1.1 kali
Or remove the hostname from this
127.0.0.1 localhost kali
Most importantly (emphasis added)
None of these are Hadoop problems, they are hadoop, host, network and firewall configuration issues
I am using VM with Ambari 2.2 and HDP 2.3 and installing services using Ambari user interface. Issue is NameNode not starting and log indicates error saying port is in use 50070. I tried netstat and other tools to find out if anything is running on port 50070, it is not. I also tried changing 50070 to 50071 but error remains the same except it now says port is in use 50071.Below is the error I get in ambari error file:
resource_management.core.exceptions.Fail: Execution of 'ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ; /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf start namenode'' returned 1. starting namenode, logging to /var/log/hadoop/hdfs/hadoop-hdfs-namenode-hdp-m.out
2016-02-07 11:52:47,058 ERROR namenode.NameNode (NameNode.java:main(1712)) - Failed to start namenode.
java.net.BindException: Port in use: hdp-m.samitsolutions.com:50070
When using Ambari, I came across the port is in use 50070 problem. I found it's actually caused by the mismatch of NameNode's host, not port. In sometimes, Ambari will start namenode on HostB and HostC, while your configure are HostA and HostC.
Such a situation could by caused by: Update wrong namenode config when moving namenode
I tried to connect hadoop cluster from my local machine but getting the following given below error
[ashsshar#slc01nwj work]$ hive --config /<hive_conf_file_location>/conf -S -f /<hive_scirpt>
1 2.0.0-cdh4.2.0
FAILED: IllegalArgumentException java.net.UnknownHostException: <cluster_name>
[ashsshar#slc01nwj work]$
I also tried executing the same hive script on hadoop cluster that worked fine.
UnknownHostException means that the hostname resolution is not happening from your local machine. Check if you can ping to your cluster with it's hostname
I am getting below error when I am trying to configure hadoop plugin in eclipse.
Error:call to localhost:54310 failed on connection exception:java.net.connectException:Connection refused:no further informaion
Hadoop version is 1.0.4
I have installed hadoop in Linux and I am running my Eclipse using Windows.
In the hadoop location window, I have tried with host as localhost and linux server.
MR Master: Host: localhost and port 54311
DFS Master: Host: localhost and port 54310
MR Master: Host: <Linux server name> and port 54311
DFS Master: Host: <Linux server name> and port 54310
In my mapred-site.xml I see this entry entry localhost:54311.
ConnectionRefused error is, you are trying to connect a directory which you dont have permission to read/write.
This may be caused by, a directory created another user(e.g. root) and your master machine is trying to read from/write to that directory.
It is more likely that you are trying to read an input from wrong place. Check your input directory if there is no problem with it, check your output directory