MLCP Import java.lang.UnsatisfiedLinkError - hadoop

I am trying to import data into Marklogic server with MLCP. The data is in XML and inside an archive(zip) file.
MLCP is ending with java.lang.UnsatisfiedLinkError
I have tried with MLCP 8.0.6 and MLCP 8.0.7 but the error is same in both cases
Details:
Marklogic server 8.0-6.3
MLCP 8.0.6 and MLCP 8.0.7
mlcp.bat -options_file "options file path"
Options file
import
-host
localhost
-port
8907
-mode
local
-database
dj_test
-username
admin
-password
admin
-input_file_path
"C:\Users\djoshi16\Documents\training-project-data\"
-input_compressed
true
-input_file_type
archive
-thread_count
4
Error StackTrace
c:\windows\system32>"c:\Users\djoshi\Documents\mlcp-8.0.6\bin\mlcp.bat" -options_file c:\Users\djoshi\Documents\training-project-data-import.options
17/08/29 20:49:47 INFO contentpump.LocalJobRunner: Content type: XML
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
at org.apache.hadoop.security.Groups.<init>(Groups.java:70)
at org.apache.hadoop.security.Groups.<init>(Groups.java:66)
at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:280)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:271)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:248)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:763)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:748)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:621)
at org.apache.hadoop.mapreduce.task.JobContextImpl.<init>(JobContextImpl.java:72)
at org.apache.hadoop.mapreduce.Job.<init>(Job.java:144)
at org.apache.hadoop.mapreduce.Job.getInstance(Job.java:187)
at com.marklogic.contentpump.Command$1.createJob(Command.java:349)
at com.marklogic.contentpump.ContentPump.runCommand(ContentPump.java:192)
at com.marklogic.contentpump.ContentPump.main(ContentPump.java:62)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:129)
... 14 more
Caused by: java.lang.UnsatisfiedLinkError: org.apache.hadoop.security.JniBasedUnixGroupsMapping.anchorNative()V
at org.apache.hadoop.security.JniBasedUnixGroupsMapping.anchorNative(Native Method)
at org.apache.hadoop.security.JniBasedUnixGroupsMapping.<clinit>(JniBasedUnixGroupsMapping.java:49)
at org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.<init>(JniBasedUnixGroupsMappingWithFallback.java:39)
... 19 more

Thanks mattsun for the valuable reference.
In my case the actual issue was conflicting class information. After removing .dll file from the bin directory of MLCP everything worked like a charm.
Both dll and jars were shipped into the MLCP package and that is why it was hard to identify the actual issue.

Related

Spark without Hadoop: Failed to Launch

I'm running Spark 2.1.0, Hive 2.1.1 and Hadoop 2.7.3 on Ubuntu 16.04.
I download the Spark project from github and build the "without hadoop" version:
./dev/make-distribution.sh --name "hadoop2-without-hive" --tgz
"-Pyarn,hadoop-provided,hadoop-2.7,parquet-provided"
When I run ./sbin/start-master.sh, I get the following exception:
Spark Command: /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -cp /home/server/spark/conf/:/home/server/spark/jars/*:/home/server/hadoop/etc/hadoop/:/home/server/hadoop/share/hadoop/common/lib/:/home/server/hadoop/share/hadoop/common/:/home/server/hadoop/share/hadoop/mapreduce/:/home/server/hadoop/share/hadoop/mapreduce/lib/:/home/server/hadoop/share/hadoop/yarn/:/home/server/hadoop/share/hadoop/yarn/lib/ -Xmx1g org.apache.spark.deploy.master.Master --host ThinkPad-W550s-Lab --port 7077 --webui-port 8080
========================================
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/Logger
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.slf4j.Logger
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 7 more
I edit SPARK_DIST_CLASSPATH according to the post Where are hadoop jar files in hadoop 2?
export SPARK_DIST_CLASSPATH=~/hadoop/share/hadoop/common/lib:~/hadoop/share/hadoop/common:~/hadoop/share/hadoop/mapreduce:~/hadoop/share/hadoop/mapreduce/lib:~/hadoop/share/hadoop/yarn:~/hadoop/share/hadoop/yarn/lib
But I'm still getting the same error.
I can see the slf4j jar file is under ~/hadoop/share/hadoop/common/lib.
How could I fix this error?
Thank you!
“Hadoop free” builds need to modify SPARK_DIST_CLASSPATH to include Hadoop’s package jars.
The most convenient place to do this is by adding an entry in conf/spark-env.sh :
export SPARK_DIST_CLASSPATH=$(/path/to/hadoop/bin/hadoop classpath)
check this https://spark.apache.org/docs/latest/hadoop-provided.html

Apache Drill 1.6.0 failure in starting embedded Drillbit (windows)

I am unable to start Embedded drillbit on windows machine and getting the following error. I have checked for the jars in 3rd party folder where Jackson-databind-2.7.1.jar is present, still it's saying class not found exception. Can you help me here?
Error: Failure in starting embedded Drillbit: UNSUPPORTED_OPERATION ERROR: Failure while attempting to load instance of the class of type org.apache.drill.exec.store.StoragePluginRegistry requested at path drill.exec.storage.registry.
[Error Id: 4e654256-f63d-434f-8f41-981892a776b5 ] (state=,code=0)
java.sql.SQLException: Failure in starting embedded Drillbit: UNSUPPORTED_OPERATION ERROR: Failure while attempting to load instance of the class of type org.apache.drill.exec.store.StoragePluginRegistry requested at path drill.exec.storage.registry.
[Error Id: 4e654256-f63d-434f-8f41-981892a776b5 ]
at org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:120)
at org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:64)
at org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
at net.hydromatic.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:126)
at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
at sqlline.DatabaseConnection.connect(DatabaseConnection.java:167)
at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:213)
at sqlline.Commands.connect(Commands.java:1083)
at sqlline.Commands.connect(Commands.java:1015)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
at sqlline.SqlLine.dispatch(SqlLine.java:742)
at sqlline.SqlLine.initArgs(SqlLine.java:528)
at sqlline.SqlLine.begin(SqlLine.java:596)
at sqlline.SqlLine.start(SqlLine.java:375)
at sqlline.SqlLine.main(SqlLine.java:268)
Caused by: org.apache.drill.common.exceptions.UserException: UNSUPPORTED_OPERATION ERROR: Failure while attempting to load instance of the class of type org.apache.drill.exec.store.StoragePluginRegistry requested at path drill.exec.storage.registry.
[Error Id: 4e654256-f63d-434f-8f41-981892a776b5 ]
at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
at org.apache.drill.common.config.DrillConfig.getInstance(DrillConfig.java:88)
at org.apache.drill.exec.server.DrillbitContext.(DrillbitContext.java:85)
at org.apache.drill.exec.work.WorkManager.start(WorkManager.java:105)
at org.apache.drill.exec.server.Drillbit.run(Drillbit.java:110)
at org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:118)
... 18 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.drill.common.config.DrillConfig.getInstance(DrillConfig.java:86)
... 22 more
Caused by: java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.ObjectMapper.readerFor(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/ObjectReader;
at org.apache.drill.exec.serialization.JacksonSerializer.(JacksonSerializer.java:32)
at org.apache.drill.exec.store.sys.PersistentStoreConfig.newJacksonBuilder(PersistentStoreConfig.java:81)
at org.apache.drill.exec.store.StoragePluginRegistryImpl.(StoragePluginRegistryImpl.java:90)
... 27 more
apache drill 1.6.0
"this isn't your grandfather's sql"
The issue is related with the HADOOP_HOME environment variable
If it is set, Embedded Drill does not start properly
My HADOOP_HOME was set because I use sometimes Spark or Hadoop MapReduce on my machine.
So, with
set HADOOP_HOME=
and then
sqlline.bat -u "jdbc:drill:zk=local"
The initialization is completed and the Drillbit starts

HiveServer Class Not Found Exception

I'm trying to run hive from the command prompt it is working absolutely fine. But when I try running hiveserver using "hive --service hiveserver" command, I'm getting the following exception.
Starting Hive Thrift Server
Exception in thread "main" java.lang.ClassNotFoundException: org.apache.hadoop.hive.service.HiveServer
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:278)
at org.apache.hadoop.util.RunJar.run(RunJar.java:214)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
So I then tried with the command "hive --service hiveserver2"; still I'm not finding any solution.
Can anybody please suggest a solution for this problem.
May be another process (another hiveserver) already listening on port 10000.
can you check it by :
netstat -ntulp | grep ':10000' to see it and if found then kill the process.
Otherwise start the server on another port.
By the way which version you are using ?
This error occurred to me when it can't find hive-service-*.jar in hadoop classpath. Just copy the hive-service-*.jar to your hadoop lib folder or export classpath in hadoop-env.sh. I have mentioned how to add classpath below.
Add this line in hadoop-env.sh:
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/usr/local/hive/lib/hive-*.jar
I have mentioned the path for hive as /usr/local/hive since i have hive installed at that location. Change it to point to your hive installation.

ERROR in ImportTsv-HBASE BULK LOAD

I have hbase 0.94.0. I tried doing bulk import using the importtsv tool.
Here is the command i gave
./hadoop jar /home/ericsson/Desktop/ProjectFiles/hbase-0.94.0/hbase-0.94.0.jar importtsv -Dimporttsv.columns=HBASE_ROW_KEY,a,b,c,d,e,f,g '-Dimporttsv.separator=,' Test1 /home/ericsson/Desktop/ProjectFiles/inputFiles1/CharginUsage-m-00000
Test1-My table that already exists in Hbase.
/home/ericsson/Desktop/ProjectFiles/inputFiles1/CharginUsage-m-00000- My directory where i have the CSV file.
I got the following error:
Exception in thread "main" java.lang.NoClassDefFoundError: com/google/common/collect/Multimap
at org.apache.hadoop.hbase.mapreduce.Driver.main(Driver.java:43)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.lang.ClassNotFoundException: com.google.common.collect.Multimap
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
... 6 more
importtsv task needs Google's Guava library in order run. This library is present under $HBASE_HOME/lib/guava-.jar
It is matter of telling hadoop to fetch this guava jar during execution. Simply you could copy the jar from hbase lib to hadoop lib. A more decent solution is to add this jar path to hadoop classpath or execute the hadoop task with the below command.
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HBASE_HOME/lib/guava-<version>.jar
OR
export HADOOP_CLASSPATH=`hbase classpath ` /hadoop jar /home/ericsson/Desktop/ProjectFiles/hbase-0.94.0/hbase-0.94.0.jar importtsv -Dimporttsv.columns=HBASE_ROW_KEY,a,b,c,d,e,f,g '-Dimporttsv.separator=,' Test1 /home/ericsson/Desktop/ProjectFiles/inputFiles1/CharginUsage-m-00000*

CDH4 Hbase using Pig ERROR 2998 java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/filter/Filter

I am using CDH4 in a pseudo-distributed mode and I have some trouble working with HBase and Pig together (but both work fine alone).
I am following step by step this nice tutorial:
http://blog.whitepages.com/2011/10/27/hbase-storage-and-pig/
So my Pig-script looks like this
register /usr/lib/zookeeper/zookeeper-3.4.3-cdh4.1.2.jar
register /usr/lib/hbase/hbase-0.92.1-cdh4.1.2-security.jar
register /usr/lib/hbase/lib/guava-11.0.2.jar
raw_data = LOAD 'input.csv' USING PigStorage( ',' ) AS (
listing_id: chararray,
fname: chararray,
lname: chararray );
STORE raw_data INTO 'hbase://sample_names' USING org.apache.pig.backend.hadoop.hbase.HBaseStorage ('info:fname info:lname');
But upon entering following command
pig -x local hbase_sample.pig
I get following error message
ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2998: Unhandled internal error. org/apache/hadoop/hbase/filter/Filter
Main reason I found online would be the classpaths, so here's a list of the current configuration, maybe you find some nonsense in my configuration:
export HADOOP_HOME=/usr/lib/hadoop
export HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce
export HADOOP_CONF_DIR=/etc/hadoop/conf
export HBASE_HOME=/usr/lib/hbase
export HBASE_CONF_DIR=/etc/hbase/conf
export PIG_HOME=/usr/lib/pig
export PIG_CONF_DIR=/etc/pig/conf
export PATH="$HADOOP_HOME/bin:$HBASE_HOME/bin:$HADOOP_MAPRED_HOME/bin:$PIG_HOME/bin:$PATH"
export HADOOP_CLASSPATH="$HBASE_HOME/bin"
export PIG_CLASSPATH="$HBASE_HOME/bin:$PIG_HOME/bin"
If you need more details, here's the complete pig stack trace:
Pig Stack Trace
---------------
ERROR 2998: Unhandled internal error. org/apache/hadoop/hbase/filter/Filter
java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/filter/Filter
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:478)
at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:508)
at org.apache.pig.parser.LogicalPlanBuilder.validateFuncSpec(LogicalPlanBuilder.java:791)
at org.apache.pig.parser.LogicalPlanBuilder.buildFuncSpec(LogicalPlanBuilder.java:780)
at org.apache.pig.parser.LogicalPlanGenerator.func_clause(LogicalPlanGenerator.java:4583)
at org.apache.pig.parser.LogicalPlanGenerator.store_clause(LogicalPlanGenerator.java:6225)
at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1335)
at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:789)
at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:507)
at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:382)
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:175)
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1594)
at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1545)
at org.apache.pig.PigServer.registerQuery(PigServer.java:545)
at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:970)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
at org.apache.pig.Main.run(Main.java:430)
at org.apache.pig.Main.main(Main.java:111)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.filter.Filter
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
... 28 more
================================================================================
Your PIG_CLASSPATH is wrong, it should look like the following:
export PIG_CLASSPATH=”`hbase classpath`:$PIG_CLASSPATH”
This will add your missing hbase-related jars to your classpath for Pig.

Resources