I have created a hadoop psuedo-distributed cluster on VirtualBox Unbutu 12.04.
Running jps command shows that DataNode and NameNode processes are up.
I am trying to execute the following DFS command
hadoop fs -put conf input
But the above command or any other DFS command fails with
Bad connection to FS. Command aborted. exception: No FileSystem for scheme: hsdf
Any suggestions to get the above working?
Please check the path of the conf and input file your trying to pass.
Related
Functionally, I was wondering whether there is a difference between starting a hadoop namenode with the command:
$HADOOP_HOME/sbin/hadoop-daemons.sh --config $HADOOP_CONF_DIR --script hdfs start datanode
and:
hdfs datanode
The first command gives me an error stating that the datanode cannot ssh into itself(this is running on a docker container), while the second command seems to run without that issue. The official hadoop documentation for this version, (2.9.1) doesn't mention "hdfs datanode" as a way to start a datanode.
I have installed hadoop 2.7.3 in my ubuntu 16.10. I want to create a multinode cluster and I have done all the steps till formatting the namenode but "hadoop-daemon.sh start namenode command is not working. When I type this command it shows" hadoop-daemon.sh: command not found.
You need to execute following command. sbin/start-dfs.sh or sbin/start-yarn.sh
It is my first time in installing Hadoop on my Linux (Fedora distro) running on VM (using Parallel on my Mac). And I followed every step on this video and including the textual version of it.And then when I run it on localhost (or the equivalent value from hostname) in port 50070, I got the following message.
...can't establish a connection to the server at localhost:50070
When I run the jps by the way command I don't have the datanode and namenode unlike at the end of the textual version tutorial which has the following:
While mine has only the following processes running:
6021 NodeManager
3947 SecondaryNameNode
5788 ResourceManager
8941 Jps
When I run the hadoop namenode command I have some of the following [redacted] error:
Cannot access storage directory /usr/local/hadoop_store/hdfs/namenode
16/10/11 21:52:45 WARN namenode.FSNamesystem: Encountered exception loading fsimage
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /usr/local/hadoop_store/hdfs/namenode is in an inconsistent state: storage directory does not exist or is not accessible.
I tried to access by the way the above mentioned directories and it existed.
Any hint for this newbie? ;-)
You would need to give read and write permission to user with which you are running the services on directory /usr/local/hadoop_store/hdfs/namenode.
Once done, you should run format command using hadoop namenode -format
Then try to start your services.
delete files /app/hadoop/tmp/*
and try again formatting the namenode and then start-dfs.sh & start-yarn.sh
I am setting up for spark with Hadoop 2.3.0 on Mesos 0.21.0. when I try spark on the master, I get these error messages fro stderr of mesos slave:
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1229 12:34:45.923665 8571 fetcher.cpp:76] Fetching URI
'hdfs://10.170.207.41/spark/spark-1.2.0.tar.gz'
I1229 12:34:45.925240 8571 fetcher.cpp:105] Downloading resource from
'hdfs://10.170.207.41/spark/spark-1.2.0.tar.gz' to
'/tmp/mesos/slaves/20141226-161203-701475338-5050-6942-S0/frameworks/20141229-111020-701475338-5050-985-0001/executors/20141226-161203-701475338-5050-6942-S0/runs/8ef30e72-d8cf-4218-8a62-bccdf673b5aa/spark-1.2.0.tar.gz'
E1229 12:34:45.927089 8571 fetcher.cpp:109] HDFS copyToLocal failed:
hadoop fs -copyToLocal 'hdfs://10.170.207.41/spark/spark-1.2.0.tar.gz'
'/tmp/mesos/slaves/20141226-161203-701475338-5050-6942-S0/frameworks/20141229-111020-701475338-5050-985-0001/executors/20141226-161203-701475338-5050-6942-S0/runs/8ef30e72-d8cf-4218-8a62-bccdf673b5aa/spark-1.2.0.tar.gz'
sh: 1: hadoop: not found
Failed to fetch: hdfs://10.170.207.41/spark/spark-1.2.0.tar.gz
Failed to synchronize with slave (it's probably exited)
The interesting thing is that when i switch to the slave node and run the same command
hadoop fs -copyToLocal 'hdfs://10.170.207.41/spark/spark-1.2.0.tar.gz'
'/tmp/mesos/slaves/20141226-161203-701475338-5050-6942-S0/frameworks/20141229-111020-701475338-5050-985-0001/executors/20141226-161203-701475338-5050-6942-S0/runs/8ef30e72-d8cf-4218-8a62-bccdf673b5aa/spark-1.2.0.tar.gz'
, it goes well.
When starting mesos slave, you have to specify the path to your hadoop installation through the following parameter:
--hadoop_home=/path/to/hadoop
Without that it just didn't work for me, even though I had the HADOOP_HOME environment variable set up.
I am very new to hadoop. I am referring "hadoop for dummies" book.
I have setup a vm with following specs
hadoop version 2.0.6-alpha
bigtop
os centos
problem is while running any hdfs file system command I am getting following error
example command : hadoop hdfs dfs -ls
error : Could not find or load main class hdfs
Please advice
Regards,
Try running:
hadoop fs -ls
or
hdfs dfs -ls
what do they return?
fs and dfs are the same commands.
Difference between `hadoop dfs` and `hadoop fs`
Remove either hadoop or hdfs and the command should run.