Apache Kylin Unable to find HBase common lib - hadoop

Question

I have installed Hadoop version 2.6.0, HBase version 0.99.0 , Hive version 1.2, Kylin version 1.5.0.
I have setup all of the above in Standalone mode while in running Kylin it checks in early stage about Hadoop, HBase and Hive. Each and everything has been installed but when I start Kylin it give an error of HBase common lib not found.
Following is the log of Apache Kylin.
KYLIN_HOME is set to bin/../
16/03/24 18:02:16 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
kylin.security.profile is set to testing
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/kunalgupta/Desktop/kunal/Desktop/Backup/Kunal/Downloads/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/kunalgupta/Downloads/spark-1.6.0-bin-hadoop2.6/lib/spark-assembly-1.6.0-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/kunalgupta/Desktop/kunal/Desktop/Backup/Kunal/Downloads/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/kunalgupta/Downloads/spark-1.6.0-bin-hadoop2.6/lib/spark-assembly-1.6.0-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Logging initialized using configuration in jar:file:/Users/kunalgupta/Desktop/kunal/Desktop/Backup/Kunal/Downloads/apache-hive-1.2.1-bin/lib/hive-common-1.2.1.jar!/hive-log4j.properties
cut: illegal option -- -
usage: cut -b list [-n] [file ...]
cut -c list [file ...]
cut -f list [-s] [-d delim] [file ...]
HIVE_CONF is set to: /Users/kunalgupta/Desktop/kunal/Desktop/Backup/Kunal/Downloads/apache-hive-1.2.1-bin/conf/, use it to locate hive configurations.
HCAT_HOME is set to: /Users/kunalgupta/Desktop/kunal/Desktop/Backup/Kunal/Downloads/apache-hive-1.2.1-bin/hcatalog, use it to find hcatalog path:
usage: dirname path
find: -printf: unknown primary or operator
hive dependency: /Users/kunalgupta/Desktop/kunal/Desktop/Backup/Kunal/Downloads/apache-hive-1.2.1-bin/conf/::/Users/kunalgupta/Desktop/kunal/Desktop/Backup/Kunal/Downloads/apache-hive-1.2.1-bin/hcatalog/share/hcatalog/hive-hcatalog-core-1.2.1.jar
cut: illegal option -- -
usage: cut -b list [-n] [file ...]
cut -c list [file ...]
cut -f list [-s] [-d delim] [file ...]
hbase-common lib not found
Please somebody help me out.

Answer 1

The issue you encountered is that cut command on mac-osx does not support "--output-delimiter" option. I encountered the same error while installing kylin-1.5.1.
The easy to solve it is to use the gnu binaries in your shell instead of osx binaries.
Use brew for installing coreutils (I changed all the commonly used shell utils to their gnu versions)
Use the below command for that.
brew install coreutils findutils gnu-tar gnu-sed gawk gnutls gnu-indent gnu-getopt --default-names
Now to make the shell use these instead of mac binaries add the path to these utils in your shell profile file.
vi ~/.profile
add the following lines to this file
PATH="/usr/local/opt/coreutils/libexec/gnubin:$PATH"
After this open a new terminal window and do
echo $PATH
The result should have the path that we set in the previous step in the front (prepended)
Now start kylin, should work smoothly.
Some references links that helped me:
Mac forum link
Installation guide from apple.se

Answer 2

vi /etc/profile
export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera/
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$CLASSPATH
export KYLIN_HOME=/usr/local/kylin
export CDH_HOME=/opt/cloudera/parcels/CDH
export SPARK_HOME=${CDH_HOME}/lib/spark
export HBASE_HOME=${CDH_HOME}/lib/hbase
export HIVE_HOME=${CDH_HOME}/lib/hive
export HADOOP_HOME=${CDH_HOME}/lib/hadoop
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH
source /etc/profile
added ${HBASE_HOME} to /etc/profile
cat /opt/cloudera/parcels/CDH/lib/hbase/bin/hbase
if [ -n "${shaded_jar}" ] && [ -f "${shaded_jar}" ]; then
CLASSPATH="${CLASSPATH}:${shaded_jar}"
# fall through to grabbing all the lib jars and hope we're in the omnibus tarball
#
# N.B. shell specifically can't rely on the shaded artifacts because RSGroups is only
# available as non-shaded
#
# N.B. pe and ltt can't easily rely on shaded artifacts because they live in hbase-mapreduce:test-jar
# and need some other jars that haven't been relocated. Currently enumerating that list
# is too hard to be worth it.
#
else
for f in $HBASE_HOME/lib/*.jar; do
CLASSPATH=${CLASSPATH}:$f;
done
# make it easier to check for shaded/not later on.
shaded_jar=""
fi
for f in "${HBASE_HOME}"/lib/client-facing-thirdparty/*.jar; do
if [[ ! "${f}" =~ ^.*/htrace-core-3.*\.jar$ ]] && \
[ "${f}" != "htrace-core.jar$" ] && \
[[ ! "${f}" =~ ^.*/slf4j-log4j.*$ ]]; then
CLASSPATH="${CLASSPATH}:${f}"
fi
done

Answer 3

I use centos7 ,apache-kylin-3.0.0-alpha2-bin-hadoop3 and hbase 2.1.4 .
I found solution in these links.
https://issues.apache.org/jira/browse/KYLIN-3564
http://92072234.wiz03.com/share/s/2i1O8Q1L1k042IDoOy3h7BgH2K4G6J2SoQv42Xc4b01xpCrj
The second link solve my problem.
Edit hbase file
$HBASE_HOME/bin/hbase
change
CLASSPATH=${CLASSPATH}:$JAVA_HOME/lib/tools.jar
to
CLASSPATH=${CLASSPATH}:$JAVA_HOME/lib/tools.jar:/YOUR HBASE FULL PATH or $HBASE_HOME/lib/*

Answer 4

I solved it as the following:
1. export HBASE_CLASSPATH=/opt/cloudera/parcels/CDH-6.2.0-/hbase-common-2.0.0-cdh6.2.0.jar
2. then start kylin again.

Answer 5

You are running on Windows? Sorry Kylin only runs on Linux as of version 1.5