I'm trying to mount my HDFS using the NFS gateway as it is documented here:
http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HdfsNfsGateway.html
Unfortunately, following the documentation step by step does not work for me (Hadoop 2.7.1 on CentOS 6.6). When executing the mount command I receive the following error message:
[root#server1 ~]# mount -t nfs -o vers=3,proto=tcp,nolock,noacl,sync
server1:/ /hdfsmount/ mount.nfs: mounting server1:/ failed, reason
given by server: No such file or directory
I created the folder hdfsmount so that I can say it definitely exists. My questions are now:
Did anyone faced the same issue as I do?
Do I have to configure the NFS server before I start following the steps in the documentation (e.g. I read about editing /etc/exports).
Any help is highly apreciated!
I found the problem deep in the logs. When executing the command (see below) to start the nfs3 component of HDFS, the executing user needs permissions to delete /tmp/.hdfs-nfs which is configured as nfs.dump.dir in core-site.xml.
If the permissions are not set, you'll receive a log message like:
15/08/12 01:19:56 WARN fs.FileUtil: Failed to delete file or dir
[/tmp/.hdfs-nfs]: it still exists. Exception in thread "main"
java.io.IOException: Cannot remove current dump directory:
/tmp/.hdfs-nfs
Another option is to simply start the nfs component as root.
[root]> /usr/local/hadoop/sbin/hadoop-daemon.sh --script /usr/local/hadoop/bin/hdfs start nfs3
Related
It is my first time in installing Hadoop on my Linux (Fedora distro) running on VM (using Parallel on my Mac). And I followed every step on this video and including the textual version of it.And then when I run it on localhost (or the equivalent value from hostname) in port 50070, I got the following message.
...can't establish a connection to the server at localhost:50070
When I run the jps by the way command I don't have the datanode and namenode unlike at the end of the textual version tutorial which has the following:
While mine has only the following processes running:
6021 NodeManager
3947 SecondaryNameNode
5788 ResourceManager
8941 Jps
When I run the hadoop namenode command I have some of the following [redacted] error:
Cannot access storage directory /usr/local/hadoop_store/hdfs/namenode
16/10/11 21:52:45 WARN namenode.FSNamesystem: Encountered exception loading fsimage
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /usr/local/hadoop_store/hdfs/namenode is in an inconsistent state: storage directory does not exist or is not accessible.
I tried to access by the way the above mentioned directories and it existed.
Any hint for this newbie? ;-)
You would need to give read and write permission to user with which you are running the services on directory /usr/local/hadoop_store/hdfs/namenode.
Once done, you should run format command using hadoop namenode -format
Then try to start your services.
delete files /app/hadoop/tmp/*
and try again formatting the namenode and then start-dfs.sh & start-yarn.sh
I am installing hadoop on my windows PC. I was able to start yarn and dfs. I ran this command hadoop fs -mkdir /in
It displays the following error:
-mkdir: java.net.UnknownHostException: master
I am newbie Please explain me what has to be done ?
The error indicates the host master cannot be resolved to an IP address. You probably need to add a mapping for it in your hosts file so it can be resolved.
I am using centos 7. Downloaded and untarred hadoop 2.4.0 and followed the instruction as per the link Hadoop 2.4.0 setup
Ran the following command.
./hdfs namenode -format
Got this error :
Error: Could not find or load main class org.apache.hadoop.hdfs.server.namenode.NameNode
I see a number of posts with the same error with no accepted answers and I have tried them all without any luck.
This error can occur if the necessary jarfiles are not readable by the user running the "./hdfs" command or are misplaced so that they can't be found by hadoop/libexec/hadoop-config.sh.
Check the permissions on the jarfiles under: hadoop-install/share/hadoop/*:
ls -l share/hadoop/*/*.jar
and if necessary, chmod them as the owner of the respective files to ensure they're readable. Something like chmod 644 should be sufficient to at least check if that fixes the initial problem. For the more permanent fix, you'll likely want to run the hadoop commands as the same user that owns all the files.
I followed the link Setup hadoop 2.4.0
and I was able to get over the error message.
Seems like the documentation on hadoop site is not complete.
I am trying open a file from HDFS throught the webhdfs API. I can create files and upload them, but once I try to open I get this error
{"RemoteException":{"exception":"NullPointerException","javaClassName":"java.lang.NullPointerException","message":null}}
using the following command
curl -i -X GET "http://ec2-xx-xx-xx-xx.eu-west-1.compute.amazonaws.com:50070/webhdfs/v1/tmp/tmp.txt?op=OPEN"
I tried this from multiple machines (from the master node, or remotely) I get the same error. It's running on CHD4.6.
thanks,
apparently this is a bug in CDH running on Ubuntu 12.04, due to /run mounted with noexec
can be resolved as follows:
sudo mount -o remount,exec /run
I am trying to invoke Sqoop through a map reduce program using
Sqoop.runTool(arguments,_conf);
When executing, I receive the following error
Exception in thread "main" java.lang.RuntimeException: Could not create temporary directory: /tmp/sqoop-hdfs/compile/a609226c19d65f561dd7035c00d318f6; check for a directory permissions issue on /tmp.
I have set the permissions on /tmp and it's subdirectories in HDFS to 777
I can invoke the same command fine through command line using sudo -u hdfs sqoop ...
This is Cloudera's hadoop distirbution and I am running the job as hdfs user.
This probably isn't the /tmp directory in HDFS, but rather then /tmp directory on the local file system - whats the permissions on that directory (and would also explain why it works when you 'sudo' the command)
Just clean /tmp/sqoop-hdfs/compile floder it works