Error: /usr/local/Cellar/sqoop/1.4.6/../hadoop does not exist! Please set $HADOOP_COMMON_HOME to the root of your Hadoop installation - macos

I installed Hadoop on Mac using brew and then configured it. Then I installed Sqoop and when I try to run Sqoop I get the following error:
Error: /usr/local/Cellar/sqoop/1.4.6/../hadoop does not exist!
Please set $HADOOP_COMMON_HOME to the root of your Hadoop installation.
My Hadoop is running fine, I have even set the path to HADOOP_COMMON_HOME in both ~/.bash_profile and sqoop-env.sh
Here is my sqoop env file:
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version
2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# included in all the hadoop scripts with source command
# should not be executable directly
# also should not be passed any arguments, since we need original $*
# Set Hadoop-specific environment variables here.
#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME= /usr/local/Cellar/hadoop/3.0.0
export PATH=$PATH:$HADOOP_COMMON_HOME/bin
#Set path to where hadoop-*-core.jar is available
#export HADOOP_MAPRED_HOME=
#set the path to where bin/hbase is available
#export HBASE_HOME=
#Set the path to where bin/hive is available
#export HIVE_HOME=
#Set the path for where zookeper config dir is
#export ZOOCFGDIR=

I have done brew install hadoop previously to do some Spark things. I don't think I had to edit any configurations in Hadoop itself,
but brew install sqoop seemed to work out of the box for finding the Hadoop configuration. Not sure where you got your sqoop-env.sh from, but having just installed it, here is mine.
[jomoore#libexec] $ cat /usr/local/Cellar/sqoop/1.4.6/libexec/conf/sqoop-env.sh
export HADOOP_HOME="/usr/local"
export HBASE_HOME="/usr/local"
export HIVE_HOME="/usr/local"
export ZOOCFGDIR="/usr/local/etc/zookeeper"
And if we grep the location that it wants to find hadoop in... $HADOOP_HOME/bin
[jomoore#libexec] $ ls -l /usr/local/bin/ | grep hadoop
lrwxr-xr-x 1 jomoore admin 45 Mar 13 10:24 container-executor -> ../Cellar/hadoop/3.0.0/bin/container-executor
lrwxr-xr-x 1 jomoore admin 33 Mar 13 10:24 hadoop -> ../Cellar/hadoop/3.0.0/bin/hadoop
lrwxr-xr-x 1 jomoore admin 31 Mar 13 10:24 hdfs -> ../Cellar/hadoop/3.0.0/bin/hdfs
lrwxr-xr-x 1 jomoore admin 33 Mar 13 10:24 mapred -> ../Cellar/hadoop/3.0.0/bin/mapred
lrwxr-xr-x 1 jomoore admin 50 Mar 13 10:24 test-container-executor -> ../Cellar/hadoop/3.0.0/bin/test-container-executor
lrwxr-xr-x 1 jomoore admin 31 Mar 13 10:24 yarn -> ../Cellar/hadoop/3.0.0/bin/yarn
And since I don't have Accumulo, Hive, or Zookeeper installed, these other warnings are just noise.
[jomoore#libexec] $ sqoop
Warning: /usr/local/Cellar/sqoop/1.4.6/libexec/bin/../../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/local/Cellar/sqoop/1.4.6/libexec/bin/../../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /usr/local/Cellar/sqoop/1.4.6/libexec/bin/../../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
ERROR: Cannot execute /usr/local/libexec/hadoop-config.sh.
That error at the end is actually looking for the file here, which probably shouldn't be at libexec/libexec anyway within the Brew package
$ find /usr/local/Cellar -name hadoop-config.sh
/usr/local/Cellar/hadoop/3.0.0/libexec/libexec/hadoop-config.sh

Related

HDFS fails to start with Hadoop 3.2 : bash v3.2+ is required

I'm building a small Hadoop cluster composed of 2 nodes : 1 master + 1 worker. I'm using the latest version of Hadoop (3.2) and everything is executed by the root user. In the installation process, I've been able to hdfs namenode -format. Next step is to start the HDFS daemon with start-dfs.sh.
$ start-dfs.sh
Starting namenodes on [master]
bash v3.2+ is required. Sorry.
Starting datanodes
bash v3.2+ is required. Sorry.
Starting secondary namenodes [master]
bash v3.2+ is required. Sorry.
Here's the generated logs in the journal:
$ journalctl --since "1 min ago"
-- Logs begin at Thu 2019-08-29 11:12:27 CEST, end at Thu 2019-08-29 11:46:40 CEST. --
Aug 29 11:46:40 master su[3329]: (to root) root on pts/0
Aug 29 11:46:40 master su[3329]: pam_unix(su-l:session): session opened for user root by root(uid=0)
Aug 29 11:46:40 master su[3329]: pam_unix(su-l:session): session closed for user root
Aug 29 11:46:40 master su[3334]: (to root) root on pts/0
Aug 29 11:46:40 master su[3334]: pam_unix(su-l:session): session opened for user root by root(uid=0)
Aug 29 11:46:40 master su[3334]: pam_unix(su-l:session): session closed for user root
Aug 29 11:46:40 master su[3389]: (to root) root on pts/0
Aug 29 11:46:40 master su[3389]: pam_unix(su-l:session): session opened for user root by root(uid=0)
Aug 29 11:46:40 master su[3389]: pam_unix(su-l:session): session closed for user root
As I'm using Zsh (with Oh-my-Zsh), I logged into a bash console to give it a try. Sadly, I get the same result. In fact, this error happens for all sbin/start-*.sh scripts. However, the hadoop and yarn commands work like a charm.
Since I didn't find much information on this error on the Internet, here I am. Would be glad to have any advice!
Other technical details
Operating system info:
$ lsb_release -d
Description: Debian GNU/Linux 10 (buster)
$ uname -srm
Linux 4.19.0-5-amd64 x86_64
Available Java versions (tried with both):
$ update-alternatives --config java
There are 2 choices for the alternative java (providing /usr/bin/java).
Selection Path Priority Status
------------------------------------------------------------
0 /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1111 auto mode
* 1 /usr/lib/jvm/adoptopenjdk-8-hotspot-amd64/bin/java 1081 manual mode
2 /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1111 manual mode
Some ENV variables you might be interested in:
$ env
USER=root
LOGNAME=root
HOME=/root
PATH=/root/bin:/usr/local/bin:/usr/local/hadoop/bin:/usr/local/hadoop/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
SHELL=/usr/bin/zsh
TERM=rxvt-unicode
JAVA_HOME=/usr/lib/jvm/adoptopenjdk-8-hotspot-amd64
HADOOP_HOME=/usr/local/hadoop
HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
ZSH=/root/.oh-my-zsh
Output of the Hadoop executable:
$ hadoop version
Hadoop 3.2.0
Source code repository https://github.com/apache/hadoop.git -r e97acb3bd8f3befd27418996fa5d4b50bf2e17bf
Compiled by sunilg on 2019-01-08T06:08Z
Compiled with protoc 2.5.0
From source with checksum d3f0795ed0d9dc378e2c785d3668f39
This command was run using /usr/local/hadoop/share/hadoop/common/hadoop-common-3.2.0.jar
My Zsh and Bash installation:
$ zsh --version
zsh 5.7.1 (x86_64-debian-linux-gnu)
$ bash --version
GNU bash, version 5.0.3(1)-release (x86_64-pc-linux-gnu)
# only available in a console using *bash*
$ echo ${BASH_VERSINFO[#]}
5 0 3 1 release x86_64-pc-linux-gnu
TL;DR: use a different user (e.g. hadoop) instead of root.
I found the solution but not the deep understanding on what is going on. Despite how sad I can be, here's the solution I found:
Running with root user:
$ start-dfs.sh
Starting namenodes on [master]
bash v3.2+ is required. Sorry.
Starting datanodes
bash v3.2+ is required. Sorry.
Starting secondary namenodes [master_bis]
bash v3.2+ is required. Sorry
Then I created a hadoop user and gave this user privileges on the Hadoop installation (R/W access). After logging in with this new user I have the following output for the command that caused me some troubles:
$ start-dfs.sh
Starting namenodes on [master]
Starting datanodes
Starting secondary namenodes [master_bis]
Moreover, I noticed that processes created by start-yarn.sh were not listed in the output of jps while using Java 11. Switching to Java 8 solved my problem (don't forget to update all $JAVA_HOME variables, both in /etc/environment and hadoop-env.sh).
Success \o/. However, I'd be glad to understand why the root user cannot do this. I know it's a bad habit to use root but in an experimental environment this is not of our interest to have a clean "close-to" production environment. Any information about this will be kindly appreciated :).
try
chsh -s /bin/bash
to change the default shell back to bash

Which version of sqoop is compatible with hadoop 3.0

When running sqoop version, getting this error:
hadoopusr#houssein:~$ sqoop version
Warning: /usr/lib/sqoop/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /usr/lib/sqoop/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /usr/lib/sqoop/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
/usr/local/hadoop/libexec/hadoop-functions.sh: line 2326: HADOOP_ORG.APACHE.SQOOP.SQOOP_USER: bad substitution
/usr/local/hadoop/libexec/hadoop-functions.sh: line 2421: HADOOP_ORG.APACHE.SQOOP.SQOOP_OPTS: bad substitution
2019-02-24 04:43:16,533 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
Sqoop 1.4.7
git commit id 2328971411f57f0cb683dfb79d19d4d19d185dd8
Compiled by maugli on Thu Dec 21 15:59:58 STD 2017`
Any help?
First, change the version of Hadoop to 2.6.x.
The environment warnings say that you need put each dependency in the correspondent directory.
I am assuming the below:
SQOOP_HOME and other params are configured in .profile
the HADOOP_COMMON_HOME & HADOOP_MAPRED_HOME setup are already done in $SQOOP_HOME/conf path.
I too faced a similar issue but it resolved after i added the mysql connectors
I downloaded the below from the link given below, the file mysql-connector-java-8.0.15.tar.gz (you can google and download it from a different link as well)
[Link to download][1]
Steps to configure the mysql connector.
Once the mysql-connector-java gz package is downloaded, Run the below commands to unzip it:
tar -xvf mysql-connector-java-8.0.15.tar.gz
mv mysql-connector-java-8.0.15/mysql-connector-java-8.0.15.jar /$SQOOP_HOME/lib
Finally, to verify the installation, run:
sqoop version
This will output version about sqoop
Hope this helps!!

Getting error while running Sqoop in EC2 instance

I installed Sqoop in my EC2 instance with the reference of http://kontext.tech/docs/DataAndBusinessIntelligence/p/configure-sqoop-in-a-edge-node-of-hadoop-cluster My hadoop cluster is also working well and good.
I got the error Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster and I solved it with the mentioned solution given in that particular link. But unfortunately I get another error while running Sqoop Import:
Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class org.apache.hadoop.mapred.YarnChild.
Please suggest me how to over come this error.
This is how my sqoop-env.template.sh looks:
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# included in all the hadoop scripts with source command
# should not be executable directly
# also should not be passed any arguments, since we need original $*
# Set Hadoop-specific environment variables here.
#Set path to where bin/hadoop is available
#export HADOOP_COMMON_HOME=$HOME/hadoop-3.1.0
#Set path to where hadoop-*-core.jar is available
#export HADOOP_MAPRED_HOME=$HOME/hadoop-3.1.0
#set the path to where bin/hbase is available
#export HBASE_HOME=
#Set the path to where bin/hive is available
#export HIVE_HOME=
#Set the path for where zookeper config dir is
#export ZOOCFGDIR=`

Apache Sqoop - addtowar.sh not found

I just now downloaded the Sqoop installation file sqoop-1.99.3-bin-hadoop100.tar.gz. I am not able to find the file addtowar.sh in it. I am following the installation instructions from here - https://sqoop.apache.org/docs/1.99.1/Installation.html . The following is the listing of the bin directory.
hduser#system:~/sqoop-1.99.3-bin-hadoop100/bin$ ls -ltr
total 8
-rwxr-xr-x 1 hduser2 hadoop 1361 Oct 18 2013 sqoop-sys.sh
-rwxr-xr-x 1 hduser2 hadoop 3439 Oct 18 2013 sqoop.sh
Am I missing something here or are the installations instructions not updated properly?
You may refer to the docs of the version you are using.
For 1.99.3 Refer the below link
http://sqoop.apache.org/docs/1.99.3/Installation.html
I don't have a direct answer but I have been tracking this down and it seems addtowar.sh has been removed (I am also using 1.99.3) in favor of adding hadoop jar directories into catalina.properties under common.loader line. However I cannot get this to work.
Definitely follow the 1.99.3 documentation:
http://sqoop.apache.org/docs/1.99.3/Installation.html
But they fail to mention in that documentation that you need to add all of the libraries for Hadoop to the common.loader variable in catalina.properties.
To get the sqoop client working, I had to add the following to catalina.properties:
common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/Users/bone/tools/hadoop/share/hadoop/common/*.jar,/Users/bone/tools/hadoop/share/hadoop/yarn/lib/*.jar,/Users/bone/tools/hadoop/share/hadoop/mapreduce/*.jar,/Users/bone/tools/hadoop/share/hadoop/tools/lib/*.jar,/Users/bone/tools/hadoop/share/hadoop/common/lib/*.jar
In my case, /Users/bone/tools/hadoop was a complete install of hadoop-2.4.0.

hbase 0.90.5 not work after replace hadoop*.jar in hbase/lib/

i have Debian 6.03 and problem with best friends hbase and hadoop
step by step, I want working configuration hbase (standalone for the first step) and hadoop :
wget http://www.sai.msu.su/apache//hbase/hbase-0.90.5/hbase-0.90.5.tar.gz
tar xzfv hbase-0.90.5.tar.gz
sudo mv hbase-0.90.5 /usr/local/
sudo ln -s hbase-0.90.5/ hbase
sudo chown -R hduser:hadoop hbase*
lrwxrwxrwx 1 hduser hadoop 13 Янв 21 10:11 hbase -> hbase-0.90.5/
drwxr-xr-x 8 hduser hadoop 4096 Янв 21 10:11 hbase-0.90.5
dan#master:/usr/local/hbase$ su hduser
hduser#master:/usr/local/hbase$ bin/start-hbase.sh
starting master, logging to /usr/local/hbase/bin/../logs/hbase-hduser-master-master.out
hduser#master:/usr/local/hbase$ bin/hbase shell
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 0.90.5, r1212209, Fri Dec 9 05:40:36 UTC 2011
hbase(main):001:0> list
TABLE
0 row(s) in 0.8560 seconds
But, after unpack hadoop core v 1.0 in hbase lib/ folder - I got:
hduser#master:/usr/local/hbase$ bin/stop-hbase.sh
hduser#master:/usr/local/hbase$ cp ../hadoop/hadoop-core-1.0.0.jar lib/
hduser#master:/usr/local/hbase$ rm lib/hadoop-core-0.20-append-r1056497.jar
hduser#master:/usr/local/hbase$ bin/start-hbase.sh
starting master, logging to /usr/local/hbase/bin/../logs/hbase-hduser-master-master.out
hduser#master:/usr/local/hbase$ bin/hbase shell
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 0.90.5, r1212209, Fri Dec 9 05:40:36 UTC 2011
hbase(main):001:0> list
TABLE
ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to connect to ZooKeeper but the connection closes immediately. This could be a sign that the server has too many connections (30 is the default). Consider inspecting your ZK server logs for that error and then make sure you are reusing HBaseConfiguration as often as you can. See HTable's javadoc for more information.
Why I need zookeeper on standalone after replace hadoop-core*.jar?
how to fix it?
Have you configured hbase-env.sh to manage Zookeeper itself?
Have you configured zookeeper quorums in hbase-site.xml?
I have the same problem, and solved it by configuring yarn and map-reduce.
Try this post.

Resources