I run hadoop-2.0.5-alpha.
When I list hdfs files, I get this Exception:
bin/hadoop fs -ls /
13/07/07 18:47:36 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
ls: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Message missing required fields: callId, status;
My core-site.xml looks like that:
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:8020</value>
</property>
</configuration>
What could be wrong?
If you have multpile versions of hadoop installed on your system, verify your PATH. You may be using the wrong version of hadoop as the client.
I ran into this problem when I had two versions of hadoop installed: hadoop-1.1.2 and hadoop-2.1.0-beta. It turned out that my path was incorrect and I was attempting to run the hadoop command from hadoop-1.1.2 against hadoop 2.1.0-beta.
In addition to your PATH, check the settings of your HADOOP_CONF_DIR or even HADOOP_HOME environment variables to be sure they are pointing to the correct directory for your hadoop 2 installation.
Related
I try to re-read the hosts and exclude files to update the set of Datanodes that are allowed to connect to the Namenode, after I finished the configuration, I execute "hdfs dfsadmin -refreshNodes"，but it does't work.
My hadoop version is 3.1.2 32bit, os is centos 7 64bit.
<property>
<name>dfs.hosts</name>
<value>/opt/install/hadoop-3.1.2/etc/hadoop/dfs.hosts</value>
</property>
after i execute the refresh command, the console display:
2019-07-07 09:31:15,468 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Refresh nodes successful
I have seen a lot of folks getting problem with hadoop installation. I went through all the related stackoverflow questions, but could not fix the problem.
The problem is :
hdfs dfs -ls
16/09/27 09:43:42 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
ls: `.': No such file or directory
I am using ubuntu 16.04 and I downloaded hadoop stable version 2.7.2 from Apache mirror:
http://apache.spinellicreations.com/hadoop/common/
I have installed java and ssh already.
which java
java is /usr/bin/java
which javac
javac is /usr/bin/javac
which ssh
ssh is /usr/bin/ssh
echo $JAVA_HOME
/usr/lib/jvm/java-9-openjdk-amd64
Note:
sudo update-alternatives --config java
There are 2 choices for the alternative java (providing /usr/bin/java).
Selection Path Priority Status
------------------------------------------------------------
* 0 /usr/lib/jvm/java-9-openjdk-amd64/bin/java 1091 auto mode
1 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java 1081 manual mode
2 /usr/lib/jvm/java-9-openjdk-amd64/bin/java 1091 manual mode
Press <enter> to keep the current choice[*], or type selection number:
hadoop environment variables in ~/.bashrc
export JAVA_HOME=/usr/lib/jvm/java-9-openjdk-amd64
export HADOOP_INSTALL=/home/bhishan/hadoop-2.7.2
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib"
export PATH=$PATH:$HADOOP_HOME/bin
Modification of file:
/home/bhishan/hadoop-2.7.2/etc/hadoop/hadoop-env.sh
Added a one line at the end:
export JAVA_HOME=/usr/lib/jvm/java-9-openjdk-amd64
The link to hadoop-env.sh in the pastebin is here:
http://pastebin.com/a3iPjB04
Then I created some empty directories:
/home/bhishan/hadoop-2.7.2/tmp
/home/bhishan/hadoop-2.7.2/etc/hadoop/hadoop_store
/home/bhishan/hadoop-2.7.2/etc/hadoop/hadoop_store/hdfs
/home/bhishan/hadoop-2.7.2etc/hadoop/hadoop_store/hdfs/datanode
/home/bhishan/hadoop-2.7.2/etc/hadoop/hadoop_store/hdfs/namenode
Modifications to the file: /home/bhishan/hadoop-2.7.2/etc/hadoop/hdfs-site.xml
<property>
<name>dfs.replication</name>
<value>1</value>
<description>Default block replication.
The actual number of replications can be specified when the file is created.
The default is used if replication is not specified in create time.
</description>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/home/bhishan/hadoop-2.7.2/etc/hadoop/hadoop_store/hdfs/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/home/bhishan/hadoop-2.7.2/etc/hadoop/hadoop_store/hdfs/datanode</value>
</property>
The link in the pastebin is this:
http://pastebin.com/cha7ZBr8
Modifications to the file: /home/bhishan/hadoop-2.7.2/etc/hadoop/core-site.xml
is following:
hadoop.tmp.dir
/home/bhishan/hadoop-2.7.2/tmp A base
for other temporary directories.
fs.default.name
hdfs://localhost:54310 The name of the
default file system. A URI whose scheme and authority determine the
FileSystem implementation. The uri's scheme determines the config
property (fs.SCHEME.impl) naming the FileSystem implementation
class. The uri's authority is used to determine the host, port,
etc. for a filesystem.
The link to the pastebin for core-site.xml is this:
http://pastebin.com/D184DuGB
The Modification to file are given below: /home/bhishan/hadoop-2.7.2/etc/hadoop/mapred-site.xml
mapred.job.tracker
localhost:54311 The host and port that
the MapReduce job tracker runs at. If "local", then jobs are run
in-process as a single map and reduce task.
The pastebin link is:
http://pastebin.com/nVxs8nMm
when I type hostname in the terminal it says BP
cat /etc/hosts
127.0.0.1 localhost BP
127.0.1.1 localhost
The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
I have also disabled ipv6
cat /etc/sysctl.conf
net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1
net.ipv6.conf.lo.disable_ipv6=1
hadoop descriptions
hadoop version
Hadoop 2.7.2
which hadoop
hadoop is /home/bhishan/hadoop-2.7.2/bin/hadoop
which hdfs
hdfs is /home/bhishan/hadoop-2.7.2/bin/hdfs
Restarting hadoop
cd /home/bhishan/hadoop-2.7.2/sbin
stop-dfs.sh
stop-yarn.sh
cd /home/bhishan/hadoop-2.7.2/tmp && rm -Rf *
hadoop namenode -format
start-dfs.sh
start-yarn.sh
Now the error comes
hdfs dfs -ls
16/09/26 23:53:14 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes
where applicable ls: `.': No such file or directory
checking jps
jps
6688 sun.tools.jps.Jps
3909 SecondaryNameNode
3525 NameNode
4327 NodeManager
4184 ResourceManager
3662 DataNode
checknative
hadoop checknative -a
16/09/27 09:28:18 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Native library checking:
hadoop: false
zlib: false
snappy: false
lz4: false
bzip2: false
openssl: false
16/09/27 09:28:18 INFO util.ExitUtil: Exiting with status 1
Then I installed missing libraries:
a) which hadoop gives Hadoop 2.7.2
b) sudo apt-get install --reinstall zlibc zlib1g zlib1g-dev
From synaptic manager I can see following libraries installed:
zlib1g, zlib1g-dev , zlib1g:i386, zlibc
c) Installed snappy and python-snappy.
d) In Synaptic manager I can see lz4
liblz4-1, liblz4-tool, python-lz4, python3-lz4
e) bzip2 is already installed.
f) openssl is already installed.
All checknative are false and I can not run hdfs dfs -ls
I could not find any errors till now. Any help will be appreciated.
Also, I am trying to run hadoop in Single laptop with four cores. The version is 2.7.2, How is version 3.0, If I have to reinstall the hadoop from Scratch, may be I should go with hadoop3. Suggestions will be welcomed.
Related links:
Result of hdfs dfs -ls command
hdfs dfs ls not working after multiple nodes configured
hadoop fs -ls does not work
Namenode not getting started
No Namenode or Datanode or Secondary NameNode to stop
Hadoop 2.6.1 Warning: WARN util.NativeCodeLoader
Hadoop 2.2.0 Setup (Pseudo-Distributed Mode): ERROR// Warn util.NativeCodeLoader: unable to load native-hadoop library
Command "hadoop fs -ls ." does not work
And, also,
hadoop fs -mkdir failed on connection exception
Hadoop cluster setup - java.net.ConnectException: Connection refused
Hadoop (local and host destination do not match) after installing hive
Help will be truly appreciated!
From this error:
hdfs dfs -ls
16/09/27 09:43:42 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
ls: `.': No such file or directory
Ignore the warning about the native libraries - the command should work fine even with that warning.
When you run hdfs dfs -ls with no path as you have done, it attempts to list the contents of your home directory in HDFS, which is /user/ by default. In this case, I suspect this issue is simply that your user directory does not exist.
Does it work OK if you run:
hadoop fs -ls /
And then do:
hadoop fs -mkdir -p /user/<your_user_name>
hadoop fs -ls
Hi stackoverflow community,
so I've been wanting to install hadoop, but I have come to a problem.
I've looked at other approaches, but I still keep receiving. I am completely new to hadoop, so I don't really know where to go. I am on a macbook pro with El Capitan if relevant. Once I make sbin/start-dfs.sh I receive this:
sbin/start-dfs.sh
16/05/10 11:09:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Incorrect configuration: namenode address dfs.namenode.servicerpc-address or dfs.namenode.rpc-address is not configured.
Starting namenodes on []
Password:
localhost: /usr/local/Cellar/hadoop/2.7.2/libexec/sbin/hadoop-daemon.sh: line 69: [: MacBook: integer expression expected
localhost: starting namenode, logging to /usr/local/Cellar/hadoop/2.7.2/libexec/logs/hadoop-name-namenode-name’s
localhost: Error: Could not find or load main class MacBook
The hadoop-daemon.sh is:
The relevant XMLs are as follow:
core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
If anything is wanted I will freely provide. Thank you for all the help and I truly appreciate it, since I really want to start using Hadoop.
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_65.jdk/Contents/Home
export HADOOP_PREFIX=/usr/local/Cellar/hadoop
Hey so this is an update if anyone is considered: I now get this
WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [myIP#]
New note: I am redoing the process with and refollowing this guide. Whether or not success is mine, I will post my update here :)!
zhongyaonan.com/hadoop-tutorial/…
Looks like your conf directory is not set properly try following steps
export HADOOP_CONF_DIR = $HADOOP_HOME/etc/hadoop
hdfs namenode -format
hdfs getconf -namenodes
./start-dfs.sh
I have started working in hadoop, i am a beginner. I have succuefully install the hadoop-2.6.0 in ubuntu 15.04 64 bit.
The commond like start-all.sh, start-dfs.sh etc are working nicely.
I am facing the problem when i am trying to move the local file system to HDFS.
Like in copyFromLocal command:
hadoop dfs -copyFromLocal ~/Hadoop/test/text2.txt ~/Hadoop/test_hds/input.txt
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
15/06/04 23:18:29 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
copyFromLocal: Call From royaljay-Inspiron-N4010/127.0.1.1 to localhost:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
Same problem in mkdir command:
hadoop dfs -put ~/test/test/test1.txt hd.txt
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
15/06/03 20:49:18 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
put: Cannot create file/user/hduser/hd.txt.COPYING. Name node is in safe mode.
I have found many solutions, but no one is working out.
If anyone have idea about this please tell me.
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
You should not use hadoop dfs, instead use the following command:
hdfs dfs -copyFromLocal ...
Do not use ~, instead mention full path like /home/hadoop/Hadoop/test/text2.txt
Call From royaljay-Inspiron-N4010/127.0.1.1 to localhost:9000 failed
on connection exception: java.net.ConnectException: Connection
refused; For more details see:
http://wiki.apache.org/hadoop/ConnectionRefused
127.0.1.1 will cause loopback problems. Remove the line with 127.0.1.1 from /etc/hosts.
NOTE: For copying files from local filesystem to HDFS, try using -put command instead of -copyFromLocal.
I think I have installed hadoop correctly. If I do jps I can see the namenode and datanode, no problem.
When I type hadoop fs -ls . I get the error:
Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /opt/db/hadoop-2.4.1/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
14/08/08 12:42:44 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
ls: '.': No such file or directory
When I type hadoop dfs -ls . I get the error:
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /opt/db/hadoop-2.4.1/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
14/08/08 12:43:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
ls: '.': No such file or directory
And when I type hadoop hdfs -ls . I get the error:
Error: Could not find or load main class hdfs
This is regardless of whether I put '.' or '/' or whatever directory I'm in.
What does this all mean? How can I get normal, expected output? What am I missing?
Use
hdfs dfs -ls ...
I dont think there is such a thing as hadoop hdfs
use the command as follows
bin/hadoop fs -ls /