i am learning sqoop from few days and successfully installed and configure with hadoop.
hadoop_usr#sawai-Lenovo-G580:/usr/local/sqoop/bin$ sqoop2-server start
Setting conf dir: /usr/local/sqoop/bin/conf
Sqoop home directory: /usr/local/sqoop
The Sqoop server is already started.
hadoop_usr#sawai-Lenovo-G580:/usr/local/sqoop/bin$ sqoop
sqoop: command not found
sqoop server is already running and when i try to fire sqoop command then i get error message. command not found. sqoop home is already in path
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
export HADOOP_INSTALL=$HADOOP_HOME
export SQOOP_HOME=/usr/local/sqoop
export SQOOP_CONF_DIR=$SQOOP_HOME/conf
export SQOOP_CLASSPATH=$SQOOP_HOME/server/lib
export PATH=$PATH:$SQOOP_HOME/bin:$SQOOP_CONF:$SQOOP_CLASSPATH
$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/local/hadoop/sbin:/usr/local/hadoop/bin:/usr/local/sqoop/bin::/usr/local/sqoop/server/lib
Please help me to resolve this issue.
Thanks in advance.
command not found error in most of the cases happens because of path is not set for same.
Kindly set the paths of sqoop, which you already have done.
export PATH=$PATH:$SQOOP_HOME/bin:$SQOOP_CONF:$SQOOP_CLASSPATH
Compile the file where you have set $PATH or restart your terminal.
put below command in .bashrc file
export SQOOP_HOME=/home/pj/sqoop
export PATH=$PATH:$SQOOP_HOME/bin
and restart .bashrc
source .bashrc
If still issue persist, restart terminal.
Related
I am running windows 10, hadoop 2.8.5 and hive 2.3.5. I installed Hive while following this guide:
https://kontext.tech/docs/DataAndBusinessIntelligence/p/apache-hive-300-installation-on-windows-10-step-by-step-guide
Each command in hive console ends with this exception after a while:
hive> show tables;
FAILED: SemanticException org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
Before anyone points out this is a duplicate question, none of the other solved my problem, namely:
question 1
question 2
and others that I do not have links for at the moment. This is what I did:
run hadoop by using start-all.cmd, this works fine.
in cygwin, I created the metastore schematool -dbType derby -initSchema. Again, this finishes fine.
run hive server: hive --service hiveserver2 start
start hive console: hive
All of these work, but no command in hive console finishes. My .bashrc for cygwin:
export JAVA_HOME="/cygdrive/c/Program Files/Java/jdk1.8.0_212"
export PATH="$PATH:$JAVA_HOME/bin"
export HADOOP_HOME='/cygdrive/c/hadoop-2.8.5'
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin
export HIVE_HOME='/cygdrive/c/hive-2.3.5'
export PATH=$PATH:$HIVE_HOME/bin
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HIVE_HOME/lib/*.jar
I have tried generating schema anew many times, and even setting the path in hive-site.xml in javax.jdo.option.ConnectionURL setting, and starting hive from HIVE_HOME and other directories, to no avail. I have run out of ideas as to what to try next.
I just downloaded hadoop, spark and hive for a MOOC. I am running Ubuntu17-10 via Virtualmachine.
I can run different hadoop commands but when I want to run "spark-shell", I get an error:
bin/spark-shell: line 57: home/dieter/spark-2.1.0-bin-hadoop2.7/bin/spark- submit: No such file or directory
When I run spark-shell inside the bin folder, it says:
spark-shell: command not found
I updated my .bashrc file with the following:
export HADOOP_HOME=home/user/hadoop-2.7.3
PATH=$HADOOP_HOME/bin:$PATH
export SPARK_HOME=home/user/spark-2.1.0-bin-hadoop2.7
PATH=$SPARK_HOME/bin:$PATH
export HIVE_HOME=home/user/apache-hive-2.1.1-bin
PATH=$HIVE_HOME/bin:$PATH
Please help! Thanks in advance
I am trying to run Spark using yarn and I am running into this error:
Exception in thread "main" java.lang.Exception: When running with master 'yarn' either HADOOP_CONF_DIR or YARN_CONF_DIR must be set in the environment.
I am not sure where the "environment" is (what specific file?). I tried using:
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
in the bash_profile, but this doesn't seem to help.
While running spark using Yarn, you need to add following line in to spark-env.sh
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
Note: check $HADOOP_HOME/etc/hadoop is correct one in your environment. And spark-env.sh contains export of HADOOP_HOME as well.
For the Windows environment, open file load-spark-env.cmd in the Spark bin folder and add the following line:
set HADOOP_CONF_DIR=%HADOOP_HOME%\etc\hadoop
just an update to answer by Shubhangi,
cd $SPARK_HOME/bin
sudo nano load-spark-env.sh
add below lines , save and exit
export SPARK_LOCAL_IP="127.0.0.1"
export HADOOP_CONF_DIR="$HADOOP_HOME/etc/hadoop"
export YARN_CONF_DIR="$HADOOP_HOME/etc/hadoop"
I run a import command on sqoop and I face the below issue. Can someone help me with this.
Error: /usr/local/sqoop-1.4.5.bin__hadoop-2.0.4-alpha/bin/../../hadoop-mapreduce does not exist!
Please set $HADOOP_MAPRED_HOME to the root of your Hadoop MapReduce installation.
My bashrc:
export JAVA_HOME=$(/usr/libexec/java_home)
export HADOOP_HOME=/usr/local/Cellar/hadoop/2.6.0/libexec
export HADOOP_YARN_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_CONF_DIR=$HADOOP_HOME
export HADOOP_PID_DIR=$HADOOP_HOME/pids
export HADOOP_LOG_DIR=$HADOOP_HOME/logs
export HADOOP_HOME_WARN_SUPPRESS=true
export HADOOP_PREFIX=$HADOOP_HOME
export PATH=.:$JAVA_HOME/bin:$HADOOP_HOME/bin:/usr/local/sqoop/bin:$PATH
Your Sqoop installation in PATH and error does not match.
export PATH=.:$JAVA_HOME/bin:$HADOOP_HOME/bin:/usr/local/sqoop/bin:$PATH
Error: /usr/local/sqoop-1.4.5.bin__hadoop-2.0.4-alpha/bin/../../hadoop-mapreduce does not exist! Please set $HADOOP_MAPRED_HOME to the root of your Hadoop MapReduce installation.
1. Your PATH says sqoop is located at /usr/local/sqoop but your error points sqoop location to /usr/local/sqoop-1.4.5.... Give the correct location for sqoop and hadoop home.
2. Export HADOOP_MAPRED_HOME in sqoop.sh (found in $SQOOP_HOME/bin). Now execute, sqoop import command.
I install mahout-0.7 and hadoop-1.2.1 on linux (centos).hadoop config as multi_node.
I created a user named hadoop and install mahout and hadoop in path /home/hadoop/opt/
I set MAHOU_HOME and HADOOP_HOME and MAHOUT_LOCAL , .... in .bashrc file in the user's environment hadoop
# .bashrc
# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi
# User specific aliases and functions
export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.71/jre
export HADOOP_HOME=/home/hadoop/opt/hadoop
export PATH=$PATH:$HADOOP_HOME/bin
export HADOOP_CONF_DIR=/opt/hadoop/conf
export MAHOUT_HOME=/home/hadoop/opt/mahout
export MAHOUT_CONF_DIR=$MAHOUT_HOME/conf
export PATH=$PATH:$MAHOUT_HOME/bin
I want to run mahout on hadoop systemfile ,When I run the following command, I get an error
command:
hadoop#master mahout$ bin/mahout fpg -i /home/hadoop/output.dat -o patterns -method mapreduce -k 50 -s 2
error:
MAHOUT_LOCAL is not set; adding HADOOP_CONF_DIR to classpath.
hadoop binary is not in PATH,HADOOP_HOME/bin,HADOOP_PREFIX/bin, running locally
Error occurred during initialization of VM
Could not reserve enough space for object heap
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Please help me. I tried but could not fix the error.
It seems that there are some conflicts in your configurations and usage.
In the fist look you can make sure about these:
To make sure that you've set the Mahout path correctly use this command:
echo $MAHOUT_LOCAL
This should not return an empty string (when you run mahout locally)
Also HADOOP_CONF_DIR should be set to $HADOOP_HOME/conf
Here's a list of popular environment variables for Hadoop:
#HADOOP VARIABLES START
export JAVA_HOME=/path/to/jdk1.8.0/ #your jdk path
export HADOOP_HOME=/usr/local/hadoop #your hadoop path
export HADOOP_INSTALL=/usr/local/hadoop #your hadoop path
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib"
export HADOOP_CLASSPATH=/home/hduser/lib/* #thir party libraries to be loaded with Hadoop
#HADOOP VARIABLES END
Also you get a heap error and you should increase your heap size so JVM will be enable to initialize
Also you may help to solve your error by adding more info about your cluster:
how many machine are you using?
what is the hardware spec of these machines?
what distribution and version of Hadoop?