I am trying to configure HDFS for spark. Simple running spark-sub
mit with --master spark://IP:7077 --deploy-mode cluster.... ends up with
16/04/08 10:16:55 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Then it end work.
I downloaded and lanuched Hadoop cluster, for testing purposes only one machine. I also set envirtoment variables, althouth I think that I forget about some of them. In the fact I set:
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:/usr/local/hadoop/bin:/usr/local/spark/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/lib/native/
Could you try to help me ?
I followed it: http://www.ccs.neu.edu/home/cbw/spark.html
Related
I have recently installed hadoop 3.2.0 on Linux and trying to issue commands like hdfs dfs -ls /. But I don't see any output at all, it seems like stuck somewhere.
I also get the warning message:
'Unable to load native-hadoop library for your platform... using builtin-java classes where applicable'
But that is all. Please let me know how to resolve this.
$hadoop fs -ls /
no output, command stuck
I am trying to manually start a master node on spark (2.1.0) on Windows 7 but the process hangs before it is setup.
$ bin\spark-class org.apache.spark.deploy.master.Master
17/05/17 14:23:52 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
It gets stuck here indefinitely (more than 10 mins)
My spark installation works fine otherwise, I have used pyspark to write and run scripts locally using pyspark --master local[x]. I am using winutils as this is being ran in standalone mode.
Also I have 2 other machines that I wish to use as workers, these work fine when I run this command on them (setup is near instant) and all environment variables appear to be setup the same on these workers as my (intended) master.
For anyone else coming across this issue I do not know the cause but a fresh download of spark placed in the same location resolved the problem.
I started by following these instructions to install a hbase and configure it to hit my BigTable instance. That all works fine, however next up I wanted to additionally configure this installation so I can run hbase shell from anywhere.
So I added the following to my .zshrc:
export HBASE_HOME=/path/to/my/hbase
export PATH=$HBASE_HOME:...
When I run hbase shell now I get the following:
2017-04-28 09:58:45,069 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
NativeException: java.io.IOException: java.lang.ClassNotFoundException: com.google.cloud.bigtable.hbase1_2.BigtableConnection
initialize at /Users/mmscibor/.hbase/lib/ruby/hbase/hbase.rb:42
(root) at /Users/mmscibor/.hbase/bin/hirb.rb:131
I figured something was up with where it was looking for it's .jars and noticed that the .tar I downloaded had a lib directory so additionally tried:
hbase shell -cp $HBASE_HOME/lib/
But no luck. However, if I navigate to $HBASE_HOME and run hbase shell everything works fine again.
What am I missing here?
You are probably running into the issue described here:
https://github.com/GoogleCloudPlatform/cloud-bigtable-examples/issues/226
You need to set GOOGLE_APPLICATION_CREDENTIALS in your environment, or run gcloud auth application-default login.
I have installed Hadoop 2.6.0 on my laptop which runs Ubuntu 14.04 lts.
Below is the link I followed for Hadoop installation: https://github.com/ev2900/YouTube_Vedio/blob/master/Install%20Hadoop%202.6%20on%20Ubuntu%2014.04%20LTS/Install%20Hadoop%202.6%20--%20Virtual%20Box%20Ubuntu%2014.04%20LTS.txt
After installation, I ran two commands:
hadoop namenode -format - It works fine
hadoop fs -ls - It is giving the following error
15/11/15 16:15:28 WARN
util.NativeCodeLoader: Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
ls: `.': No such file or directory
help me solve the error.
15/11/15 16:15:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable is a perpetual annoyance and not an error, so don't worry about that.
The ls: '.': No such file or directory error means that you haven't made your home directory yet, so you're trying to ls on a folder that doesn't exist. Do the following (as HDFS root user) to create your home folder. Ensure it has the correct permissions (which I guess depends on what specifically you want to do re: groups etc):
hdfs dfs -mkdir -p /user/'your-username'
I've configured hadoop-2.2.0 on Ubuntu/linux but when I tried to run it via start-dfs.sh and start-yarn it gave me this error:
Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
And when I go to localhost:50070/nn_browsedfscontent.jsp then it gives me the following error:
Can't browse the DFS since there are no live nodes available to redirect to.
So I followed this link to build hadoop from source but the problem still persists. Help needed!
Try hadoop-daemon.sh start namenode and then hadoop-daemon.sh start datanode and check your browser at localhost:50070