I am getting this error in hadoop while trying to create a dir.
2023-02-07 23:43:38,731 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
mkdir: `/BigDataFirstName': Input/output error
please explain me what to do step by step if possible.
i tried some stuff from internet but didnt work.
like
$ hadoop-daemon.sh stop namenode
$ hadoop-daemon.sh stop datanode
and then start it again.
The first line, you can ignore
The second line simply says you're unable to create that directory. You'll need to first format the namenode using hdfs namenode -format, if you haven't, and only then should you start the namenode, then datanode.
If you stop your machine without stopping the namenode cleanly, it may become corrupt and not start correctly, causing other issues when trying to use hdfs commands
Related
I have recently installed hadoop 3.2.0 on Linux and trying to issue commands like hdfs dfs -ls /. But I don't see any output at all, it seems like stuck somewhere.
I also get the warning message:
'Unable to load native-hadoop library for your platform... using builtin-java classes where applicable'
But that is all. Please let me know how to resolve this.
$hadoop fs -ls /
no output, command stuck
I set up and configured a multi-node Hadoop .Will appear when I start
My Ubuntu is 16.04 and Hadoop is 3.0.2
Starting namenodes on [master]
Starting datanodes
localhost: ERROR: Cannot set priority of datanode process 2984
Starting secondary namenodes [master]
master: ERROR: Cannot set priority of secondarynamenode process 3175
2018-07-17 02:19:39,470 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting resourcemanager
Starting nodemanagers
Who can tell me which link is wrong?
I had the same error and fixed it by ensuring that the datanode and namenode locations have the right permissions and are owned by the user starting hadoop daemons.
Check that
The directory path properties in hdfs-site.xml under $HADOOP_CONF_DIR are pointing to valid locations.
dfs.namenode.name.dir
dfs.datanode.data.dir
dfs.namenode.checkpoint.dir
Hadoop user must have write permission for these paths
If the write permission is not present for the mentioned paths, then the processes might not start and the error you see can occur.
I had the same error, and tried the above method, but it doesn't work.
I set XXX_USER in all xxx-env.sh files, and got the same result.
Finally I set HADOOP_SHELL_EXECNAME="root" in ${HADOOP_HOME}/bin/hdfs, and the error disappeared.
The default value of HADOOP_SHELL_EXECNAME is "HDFS".
I had the same error when I renamed my Ubuntu home directory, and had to edit core-site.xml, changing the value of the property hadoop.tmp.dir to the new path.
Just append the word "native" to your HADOOP_OPTS like this:
export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=$HADOOP_HOME/lib/native"
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
I had the same issue, you just need to check hadoop/logs directory and look for a .log file for datanode, type more nameofthefile.log and check for the errors, mine was a problem in to configuration, I fixed it and it worked.
I have installed Hadoop 2.6.0 on my laptop which runs Ubuntu 14.04 lts.
Below is the link I followed for Hadoop installation: https://github.com/ev2900/YouTube_Vedio/blob/master/Install%20Hadoop%202.6%20on%20Ubuntu%2014.04%20LTS/Install%20Hadoop%202.6%20--%20Virtual%20Box%20Ubuntu%2014.04%20LTS.txt
After installation, I ran two commands:
hadoop namenode -format - It works fine
hadoop fs -ls - It is giving the following error
15/11/15 16:15:28 WARN
util.NativeCodeLoader: Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
ls: `.': No such file or directory
help me solve the error.
15/11/15 16:15:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable is a perpetual annoyance and not an error, so don't worry about that.
The ls: '.': No such file or directory error means that you haven't made your home directory yet, so you're trying to ls on a folder that doesn't exist. Do the following (as HDFS root user) to create your home folder. Ensure it has the correct permissions (which I guess depends on what specifically you want to do re: groups etc):
hdfs dfs -mkdir -p /user/'your-username'
I am trying to upgrade HDFS from 1.2.1 to version 2.6. However, whenever I run start-dfs.sh -upgrade command, I get the below error:
hduser#Cluster1-NN:/usr/local/hadoop2/hadoop-2.6.0/etc_bkp/hadoop$ $HADOOP_NEW_HOME/sbin/start-dfs.sh -upgrade
15/05/17 12:45:58 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [nn]
Error: Please specify one of --hosts or --hostnames options and not both.
nn: starting datanode, logging to /var/hadoop/logs/hadoop-hduser-datanode-Cluster1-NN.out
dn1: starting datanode, logging to /var/hadoop/logs/hadoop-hduser-datanode-Cluster1-DN1.out
dn2: starting datanode, logging to /var/hadoop/logs/hadoop-hduser-datanode-Cluster1-DN2.out
Starting secondary namenodes [0.0.0.0]
Error: Please specify one of --hosts or --hostnames options and not both.
Please let me know if any of you experts have come across such error.
I got the same problem on Arch Linux with newly installed Hadoop 2.7.1. I'm not sure whether my case is the same as yours or not, but my experience should help. I just comment out the line HADOOP_SLAVES=/etc/hadoop/slaves in /etc/profile.d/hadoop.sh and re-login. Both accessing HDFS and running streaming jobs work for me.
The cause is that the Arch-specific script /etc/profile.d/hadoop.sh declares the $HADOOP_SLAVES environment variable. And in start-dfs.sh, hadoop-daemons.sh is called with --hostnames arguments. This confuses libexec/hadoop-config.sh.
You may want to type echo $HADOOP_SLAVES as the hadoop user. If there are non-empty outputs, check your .bashrc and/or other shell startup scripts. Hope that helps :)
Maybe it is short of some hadoop library.Can you show the detail information of namenode logs?
I've configured hadoop-2.2.0 on Ubuntu/linux but when I tried to run it via start-dfs.sh and start-yarn it gave me this error:
Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
And when I go to localhost:50070/nn_browsedfscontent.jsp then it gives me the following error:
Can't browse the DFS since there are no live nodes available to redirect to.
So I followed this link to build hadoop from source but the problem still persists. Help needed!
Try hadoop-daemon.sh start namenode and then hadoop-daemon.sh start datanode and check your browser at localhost:50070