When running sqoop version, getting this error:
hadoopusr#houssein:~$ sqoop version
Warning: /usr/lib/sqoop/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /usr/lib/sqoop/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /usr/lib/sqoop/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
/usr/local/hadoop/libexec/hadoop-functions.sh: line 2326: HADOOP_ORG.APACHE.SQOOP.SQOOP_USER: bad substitution
/usr/local/hadoop/libexec/hadoop-functions.sh: line 2421: HADOOP_ORG.APACHE.SQOOP.SQOOP_OPTS: bad substitution
2019-02-24 04:43:16,533 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
Sqoop 1.4.7
git commit id 2328971411f57f0cb683dfb79d19d4d19d185dd8
Compiled by maugli on Thu Dec 21 15:59:58 STD 2017`
Any help?
First, change the version of Hadoop to 2.6.x.
The environment warnings say that you need put each dependency in the correspondent directory.
I am assuming the below:
SQOOP_HOME and other params are configured in .profile
the HADOOP_COMMON_HOME & HADOOP_MAPRED_HOME setup are already done in $SQOOP_HOME/conf path.
I too faced a similar issue but it resolved after i added the mysql connectors
I downloaded the below from the link given below, the file mysql-connector-java-8.0.15.tar.gz (you can google and download it from a different link as well)
[Link to download][1]
Steps to configure the mysql connector.
Once the mysql-connector-java gz package is downloaded, Run the below commands to unzip it:
tar -xvf mysql-connector-java-8.0.15.tar.gz
mv mysql-connector-java-8.0.15/mysql-connector-java-8.0.15.jar /$SQOOP_HOME/lib
Finally, to verify the installation, run:
sqoop version
This will output version about sqoop
Hope this helps!!
Related
OS: Windows 10
Sqoop Version: 1.4.7
Hadoop Version: 2.9.2
Hi after installing Sqoop on win 10 following the instructions from
https://medium.com/analytics-vidhya/sqoop-how-to-install-in-5-steps-in-windows-10-ca2f17e11e75
i then try to check the install using the "sqoop version" command, but all i get is following error:
it fails:
C:\Users\iamme>sqoop version
Warning: HBASE_HOME and HBASE_VERSION not set.
Warning: HCAT_HOME not set
Warning: HCATALOG_HOME does not exist HCatalog imports will fail.
Please set HCATALOG_HOME to the root of your HCatalog installation.
Warning: ACCUMULO_HOME not set.
Warning: ZOOKEEPER_HOME not set.
Warning: HBASE_HOME does not exist HBase imports will fail.
Please set HBASE_HOME to the root of your HBase installation.
Warning: ACCUMULO_HOME does not exist Accumulo imports will fail.
Please set ACCUMULO_HOME to the root of your Accumulo installation.
Warning: ZOOKEEPER_HOME does not exist Accumulo imports will fail.
Please set ZOOKEEPER_HOME to the root of your Zookeeper installation.
Error: Could not find or load main class org.apache.sqoop.Sqoop
Caused by: java.lang.ClassNotFoundException: org.apache.sqoop.Sqoop
I checked all Hadoop and Sqoop ENV Variables, and they seem to be fine.
Any clue what might be wrong?
Best Regards,
R.S.
I am trying to install sqoop 1.4.7 in windows 10 on hadoop 3.3.0 ,
on using ./configure-sqoop on GIT bash I get following o/p:
Warning: C:\sqoop_data\sqoop-1.4.7.bin__hadoop-2.6.0/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: C:\sqoop_data\sqoop-1.4.7.bin__hadoop-2.6.0/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: C:\sqoop_data\sqoop-1.4.7.bin__hadoop-2.6.0/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: C:\sqoop_data\sqoop-1.4.7.bin__hadoop-2.6.0/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
on verifying the installation using sqoop.cmd version , I get:
Warning: HBASE_HOME and HBASE_VERSION not set.
Warning: HCAT_HOME not set
Warning: HCATALOG_HOME does not exist HCatalog imports will fail.
Please set HCATALOG_HOME to the root of your HCatalog installation.
Warning: ACCUMULO_HOME not set.
Warning: ZOOKEEPER_HOME not set.
Warning: HBASE_HOME does not exist HBase imports will fail.
Please set HBASE_HOME to the root of your HBase installation.
Warning: ACCUMULO_HOME does not exist Accumulo imports will fail.
Please set ACCUMULO_HOME to the root of your Accumulo installation.
Warning: ZOOKEEPER_HOME does not exist Accumulo imports will fail.
Please set ZOOKEEPER_HOME to the root of your Zookeeper installation.
The system cannot find the path specified.
Please help with a solution to this problem
it depends if you are working on a server-side or client-side.
If you are on the server-side do the following:
Copy the Sqoop artifact to the machine where you want to run Sqoop server. The Sqoop server acts as a Hadoop client, therefore Hadoop libraries (Yarn, Mapreduce, and HDFS jar files) and configuration files (core-site.xml, mapreduce-site.xml, ...) must be available on this node. You do not need to run any Hadoop related services - running the server on a “gateway” node is perfectly fine.
Decompress Sqoop distribution tarball
tar -xvf sqoop-<version>-bin-hadoop<hadoop-version>.tar.gz
Move decompressed content to any location
mv sqoop-<version>-bin-hadoop<hadoop version>.tar.gz /usr/lib/sqoop
Change working directory
cd /usr/lib/sqoop
And sqoopt needs the enviromantal variables to be poiting at Hadoop libraries. So set and export the variables like that:
# Export HADOOP_HOME variable
export HADOOP_HOME=/...
# Or alternatively HADOOP_*_HOME variables
export HADOOP_COMMON_HOME=/...
export HADOOP_HDFS_HOME=/...
export HADOOP_MAPRED_HOME=/...
export HADOOP_YARN_HOME=/...
Sqoop server will need to impersonate users to access HDFS, so edit the core-site.xml file:
<property>
<name>hadoop.proxyuser.sqoop2.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.sqoop2.groups</name>
<value>*</value>
</property>
For client installation
Just copy Sqoop distribution artifact on target machine and unzip it in desired location. You can start client with following command:
sqoop2-shell
for the path variable, All user and administrator facing shell commands are stored in bin/ directory. It’s recommended to add this directory to your $PATH for easier execution, for example:
PATH=$PATH:`pwd`/bin/
This combination works well for me:
Hadoop: 3.3.1
Sqoop: 1.4.7
Connector/J: 5.1.49
Note:
You must replaced "commons-lang3-3.4.jar" by "commons-lang-2.6.jar" in "$SQOOP_HOME/lib" to avoid error "java.lang.ClassNotFoundException: org.apache.commons.lang.StringUtils" when using "Sqoop import".
My VM details:
Cloudera Quickstart VM 5.5.0
VM = VM workstation 12 player
Windows = Windows 10 / 64 bit
Java = Java 1.8
when I run the "sqoop"command , I'm facing the error below:
**Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.**
Can you please help to rectify this error?
In addition to
>>> ACCUMULO_HOME='/var/lib/accumulo'
and
>>> export ACCUMULO_HOME
don't forget to create the directory
>>> sudo mkdir /var/lib/accumulo
Set the variable ACCUMULO_HOME to /var/lib/accumulo and export this variable.
This will resolve this warning.
If you happen to be installing through the Apache Bigtop, you may need to use
$ export ACCUMULO_HOME=/usr/bin/sqoop
test it out with something like
$sqoop help import
Late but i hope this workaround could help you. I've had the same problem but i'm using the cloudera sandbox.
In my case to solve it i just created an empty directory and set $ACCUMULO_HOME env variable as following:
$ sqoop help
Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
...
As said, first create the accumulo dir
$ mkdir /var/lib/accumulo
Next create the var
$ ACCUMULO_HOME='/var/lib/accumulo'
$ export ACCUMULO_HOME
That's it
$ echo $ACCUMULO_HOME
$ sqoop help
usage: sqoop COMMAND [ARGS]
Available commands:
codegen Generate code to interact with database records
create-hive-table Import a table definition into Hive
eval Evaluate a SQL statement and display the results
....
use the following commands:
1. sudo mkdir /var/lib/accumulo
2. ACCUMULO_HOME='/var/lib/accumulo'
3. export ACCUMULO_HOME
Its just a warning that you could ignore. Sqoop tried to validate the environment and send these warning.
If you are really bothered , You could try with
Set up $ACCUMULO_HOME to some directory which you would not use it.
I have Hadoop(pseudo distributed mode), Hive, sqoop and mysql installed in my local machine.
But when I am trying to run sqoop Its giving me the following error
Error: /usr/lib/hadoop does not exist!
Please set $HADOOP_COMMON_HOME to the root of your Hadoop installation.
Then I set the sqoop-env-template.sh file with all the information. Beneath is the snapshot of the sqoop-env-template.sh file.
Even after providing the hadoop hive path I face the same error.
I've installed
hadoop in /home/hduser/hadoop version 1.0.3
hive in /home/hduser/hive version 0.11.0
sqoop in /home/hduser/sqoop version 1.4.4
and mysql connector jar java-5.1.29
Could anybody please throw some light on what is going wrong
sqoop-env-template.sh is a template, meaning it doesn't by itself get sourced by the configurator. If you want it to have a custom conf and load it, make a copy as $SQOOP_HOME/conf/sqoop-env.sh.
Note: here is the relevant excerpt from bin/configure-sqoop for version 1.4.4:
SQOOP_CONF_DIR=${SQOOP_CONF_DIR:-${SQOOP_HOME}/conf}
if [ -f "${SQOOP_CONF_DIR}/sqoop-env.sh" ]; then
. "${SQOOP_CONF_DIR}/sqoop-env.sh"
fi
Can anyone please tell me that, does HCatalog require installation before using? Or it can be used just as a jar file?
I have Cloudera running on a VM, and I can use HCatalog for my MR job, Pig, Hive with no problem. And I thought the same MR code would work with another hadoop installed platform, but obviously it's not the case, exception thrown on the HCatInputFormat.setInput(). When I use Pig -useHCatalog, I'ved been prompted that the usage was wrong, meaning that it didn't know what's -useHCatalog as a parameter.
Didn't thought about this before as have been using HCatalog on Cloudera...
Yes, you need to install and start HCatalog server. HCatalog should come with the latest Hive tar package.
Check here of Apache Hive documentation for details,
Basically you need to,
Setup MySQL database for HCatalog
Run server install script
share/hcatalog/scripts/hcat_server_install.sh -r root -d dbroot -h
hadoop_home -p portnum
Start the HCatalog server
export HIVE_HOME=hive_home
$HIVE_HOME/sbin/hcat_server.sh start
As pointed out, you do not need to install hcatalog separately if you are working with hive 0.12 or later versions.