Nutch and HBase configuration error - hadoop

Question

I am trying to get nutch and hbase working based on this docker image: https://hub.docker.com/r/cogfor/nutch/
I am getting an exception that i try to inject a URL file:
InjectorJob: starting at 2017-12-19 20:49:45
InjectorJob: Injecting urlDir: urls
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/hbase/HBaseConfiguration
org.apache.gora.hbase.store.HBaseStore.initialize(HBaseStore.java:114)
at g.apache.gora.store.DataStoreFactory.initializeDataStore(DataStoreFactory.java:102)
at org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:161)
at org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:135)
at org.apache.nutch.storage.StorageUtils.createWebStore(StorageUtils.java:78)
at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:218)
at org.apache.nutch.crawl.InjectorJob.inject(InjectorJob.java:252)
at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:275)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.nutch.crawl.InjectorJob.main(InjectorJob.java:284)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfiguration
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
I know there is some misconfiguration between Nutch/HBase/Hadoop.
My gora.properties has:
gora.datastore.default=org.apache.gora.hbase.store.HBaseStore
My hbase-site.xml has:
<configuration>
<property>
<name>hbase.rootdir</name>
<value>file:///data</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>false</value>
</property>
</configuration>
And my nutch-site.xml has:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>http.agent.name</name>
<value>My Spider</value>
</property>
<property>
<name>storage.data.store.class</name>
<value>org.apache.gora.hbase.store.HBaseStore</value>
<description>Default class for storing data</description>
</property>
<property>
<name>plugin.includes</name>
<value>protocol-httpclient|urlfilter-regex|parse-(text|tika|js)|index-(basic|anchor)|query-(basic|site|url)|response-(json|xml)|summary-basic|scoring-opic|urlnormalizer-(pass|regex|basic)</value>
</property>
<property>
<name>db.ignore.external.links</name>
<value>true</value>
</property>
<property>
<name>parser.character.encoding.default</name>
<value>utf-8</value>
</property>
<property>
<name>http.content.limit</name>
<value>6553600</value>
</property>
This same is error is reported multiple times on S.O. but none of the solutions worked for me. The $HBASE_HOME and $HADOOP_CLASSPATH env variables are set to:
root#a5fb7fefc53e:/nutch_source/runtime/local/bin# echo $HADOOP_CLASSPATH
/opt/hbase-0.98.21-hadoop2/lib/hbase-client-0.98.21-hadoop2.jar:
/opt/hbase-0.98.21-hadoop2/lib/hbase-common-0.98.12-hadoop2.jar:
/opt/hbase-0.98.21-hadoop2/lib/protobuf-java-2.5.0.jar: /opt/hbase-
0.98.21-hadoop2/lib/guava-12.0.1.jar: /opt/hbase-0.98.21-
hadoop2/lib/zookeeper-3.4.6.jar: /opt/hbase-0.98.21-hadoop2/lib/hbase-
protocol-0.98.12-hadoop2.jar
root#a5fb7fefc53e:/nutch_source/runtime/local/bin# echo $HBASE_HOME
/opt/hbase-0.98.21-hadoop2
I verified all those files exist.
Can someone please help me out what i am missing?

Answer 1

The issue is mentioned in the documentation (https://wiki.apache.org/nutch/Nutch2Tutorial)
"N.B. It's possible to encounter the following exception: java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration; this is caused by the fact that sometimes the hbase TEST jar is deployed in the lib dir. To resolve this just copy the lib over from your installed HBase dir into the build lib dir. (This issue is currently in progress)."
All that needs to be done is this:
cp -R /root/hbase/lib/* /root/nutch/lib/
and nutch will start working fine.