I am completely new to Hadoop and I am trying to install Hadoop single node cluster on ubuntu but I am unable to figure out the reason I am unable to.I am following the tutorials in the following link "http://codesfusion.blogspot.in/2013/10/setup-hadoop-2x-220-on-ubuntu.html?m=1"
Everything went smoothly but when I give the command "Hadoop version" I get the following error.
"/usr/local/hadoop/bin/hadoop: line 133: /usr/lib/jvm/jdk//bin/java: No such file or directory"
I also opened the same file and searched the entire file but could not find such a line at all .
my .bashrc
export JAVA_HOME=/usr/lib/jvm/jdk/
export HADOOP_INSTALL=/usr/local/hadoop
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL
###end of paste
After that I opened hadoop-env.sh and pasted this ,the java home
export JAVA_HOME=/usr/lib/jvm/jdk/
Later I re-logged in and checked the hadoop version I am getting this error
"/usr/local/hadoop/bin/hadoop: line 133: /usr/lib/jvm/jdk//bin/java: No such file or directory"
I also cross verified that particular file but there is no line as such .Anybody kindly help me with this since I am new to this.
I found the solution.
First remove / from the end /usr/lib/jvm/jdk/ in bot bashrc and hadoop-env.sh
navigate to /usr/lib/jvm/jdk/bin
see if it has java folder or not. If its not there then check if u have made the correct soft link.
You must create a soft link for folder that has java in it so check before this command:
$ cd /usr/lib/jvm
$ ln -s java-7-openjdk-amd64 jdk
in above step as u might have seen in the tutorial change as following
$ cd /usr/lib/jvm
$ ln -s java-7-openjdk-amd64/ jdk
the 7 here is dependent on verion of jdk u have so check that and change accordingly.
I have jdk 6 so i changed for java-6-**
hope it works
This is error due to $JAVA_HOME variable. Change this variable path. you will be free from the error.
go to .bashrc using this command
vim ~/.bashrc
Change the JAVA_HOME variable
export JAVA_HOME=/usr/lib/jvm/jdk
export PATH=PATH:$PATH/bin
if you have jdk 8 then replace jdk with java-8-oracle .
export JAVA_HOME=/usr/lib/jvm/java-8-oracle
export PATH=PATH:$PATH/bin
restart you terminal and check the java command first, then hadoop command.
Related
I tried to install laravel Framework in my mac OSX 10 .And I ended modifying my ~/.bash_profile to add the laravel command. laravel command seems to work normally .But another problem came out . ls ,sudo ...and other shell commands does not work .
-bash: ls: command not found
My bash.profile file contains the two links
export PATH="/Applications/MAMP/bin/php/php5.6/bin"
export PATH="$PATH:$HOME/.composer/vendor/bin"
check in ~/.bash_profile if this PATHs exist
export PATH=/opt/local/bin:/opt/local/sbin:$PATH
your complete line should looks like this
export PATH=/Applications/MAMP/bin/php/php5.6/bin:$HOME/.composer/vendor/bin:/opt/local/bin:/opt/local/sbin:$PATH
reboot needed.
The binaries for the above mentioned utilities for debian distribution are usually in /bin or /usr/bin directory. Of course exporting path won't help if the binaries are not there.
There could be one simple approach to it -
Check for the location of binaries and then export the path in the bash_profile or .bashrc.
Run the bashrc script (. ~/.bashrc) following you may not require a reboot.
Where is the classpath for hadoop set?
When I run the below command it gives me the classpath. Where is the classpath set?
bin/hadoop classpath
I'm using hadoop 2.6.0
Open your bash profile (~/.profile or ~/.bash_profile) for editing and add the following:
export HADOOP_HOME="/usr/local/Cellar/hadoop" then Replace with your own path
export HADOOP_CLASSPATH=$(find $HADOOP_HOME -name '*.jar' | xargs echo | tr ' ' ':')
Save the changes and reload.
source ~/.profile
As said by almas shaikh it's set in hadoop-config.sh, but you could add more jars to it in hadoop-env.sh
Here is a relevant code from hadoop-env.sh which adds additional jars like capacity-scheduler and aws jar's.
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop"}
# Extra Java CLASSPATH elements. Automatically insert capacity-scheduler.
for f in $HADOOP_HOME/contrib/capacity-scheduler/*.jar; do
if [ "$HADOOP_CLASSPATH" ]; then
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$f
else
export HADOOP_CLASSPATH=$f
fi
done
# ... some other lines omitted
# Add Aws jar
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:share/hadoop/tools/lib/*
When you run hadoop command, it sources a file hadoop-config.sh that resides in $HADOOP_HDFS_HOME/libexec which sets your classpath (CLASSPATH) by picking jars residing in various directories viz.
$HADOOP_HDFS_HOME/share/hadoop/mapreduce
$HADOOP_HDFS_HOME/share/hadoop/common
$HADOOP_HDFS_HOME/share/hadoop/hdfs etc.
As per this blog post, it is in an environment variable named HADOOP_CLASSPATH. You can set it as you would any other environment variable, the specifics of which depend on which shell you use. If you use bash, then you can call like export HADOOP_CLASSPATH=/path/to/wherever:/path/to/wherever/else.
I also encountered the problem and have solved it, but my hadoop version is 2.10.1.
I hope it have some help for people who use a newer hadoop version. So far, the following methods should have worked as well in the latest hadoop version 3.3.0.
You just need to edit your .bashrc or .profile, I will give an example of .bashrc.
# edit .bashrc
$ vim ~/.bashrc
Add HADOOP_HOME, PATH of hadoop bin direcotry and HADOOP_CLASSPATH in .bashrc.
# export HADOOP_HOME=${your hadoop install directory}, an example as follows:
export HADOOP_HOME=/usr/local/hadoop-2.10.1
export PATH=${HADOOP_HOME}/bin:${PATH}
export HADOOP_CLASSPATH=`hadoop classpath`
Then,
$ source ~/.bashrc
I have installed hadoop using tar files. i have added $HADOOP_PREFIX=/usr/local/hadoop in the .bashrc file . everything was working fine. Now , i installed Hadoop using Horton's Ambari. I have removed the previous hadoop environment variable $HADOOP_PREFIX from all the system from .bashrc file.
Now when i give the command echo $HADOOP_PREFIX it is still showing the old path /usr/local/hadoop . Is there any way to remove that variable.?
delete the $HADOOP_PREFIX from .bahrc file then run this command
unset HADOOP_PREFIX
I'm trying to use a script in grunt and node to dump a database.
When I run It return me this error:
/bin/sh: mysqldump: command not found
I have already contacted the creator of the script and we have understand that the problem is the configuration of my localhost.
I'm using XAMPP 1.8.2-3 in OSX
How can I use correctly mysqldump
Thanks
Try:
export PATH=$PATH:/Applications/XAMPP/bin
Then run your script.
So you don't have to do this in the future, check your .profile file in your home directory. There should be a line that starts with export PATH, something like (only an example):
export PATH=/opt/local/bin:/opt/local/sbin:$PATH
Just add your XAMPP path to it:
export PATH=/opt/local/bin:/opt/local/sbin:/Applications/XAMPP/bin:$PATH
The different directories are separated by colons only, don't add any whitespace.
I am running Hadoop on local mode on my windows 7 machine (32 Bit).
I've installed HIVE/PIG/Hadoop/Java6 all on the C: drive.
I am using Cygwin version: 2.819.
I've mounted C: on the cygwin.
I am able to run hadoop commands from the cygwin terminal for example : fs -ls etc.
I am also able to start grunt and hive shells.
But the real problem is:
Any command I enter on grunt shell (example: fs -ls or records = LOAD..... ) I do not see any output, it kind of hangs. Similarly with the hive prompt if I give the command as show tables ; I do not see any output just cursor keeps on blinking! Any keyboard inputs and gives NOTHING. System appears to be doing NOTHING.
To me everything looks fine but definitely something is going wrong :-)
I am not sure if I am missing something.
Any help will be highly appreciated. I am attaching my classpath and environment variables from .bashrc file:
export JAVA_HOME=/c/Java/jdk1.6.0_34
export HADOOP_HOME=/c/Hadoop
export PIG_HOME=/c/PIG
export HIVE_HOME=/c/Hive
export HADOOP_BIN=$HADOOP_HOME/bin/hadoop
export PATH=$PATH:/c/Java/jdk1.6.0_34/bin
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HIVE_HOME/bin
export PATH=$PATH:$PIG_HOME/bin
The above links did not help in my case. The following solution - to change the CYGWIN shortcut properties helped.
This seems to be a bug. Other people are reporting the same problem here and here. Someone solved this problem following the tutorial linked here; you can try that and see if it fixes it in your case.