I am copying my external jars to /home/hadoop/lib directoy in emr as a bootstrap process. But it is showing following error during bootstrap process
Exception in thread "main" java.lang.IncompatibleClassChangeError: class com.google.common.cache.CacheBuilder$3 has interface com.google.common.base.Ticker as super class
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:792)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at com.google.common.cache.CacheBuilder.<clinit>(CacheBuilder.java:207)
at org.apache.hadoop.security.ShellBasedUnixGroupsMapping.<clinit>(ShellBasedUnixGroupsMapping.java:46)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:861)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:906)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:932)
at org.apache.hadoop.security.Groups.<init>(Groups.java:48)
at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:140)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:205)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:184)
at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:236)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:477)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:463)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1521)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1422)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:256)
at com.amazon.elasticmapreduce.scriptrunner.ScriptRunner.fetchFile(ScriptRunner.java:39)
at com.amazon.elasticmapreduce.scriptrunner.ScriptRunner.main(ScriptRunner.java:56)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:187)
Can anyone help me why it is happening?
I found the answer. The problem was that Mahout 0.7 is using an older version of Guava. I replaced Mahout 0.7 with 0.8 and it worked.
Related
I am getting this below error while running kafka jdbc connector
ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone:108) java.lang.NoClassDefFoundError: org/apache/calcite/avatica/UnregisteredDriver at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) at java.net.URLClassLoader.access$100(URLClassLoader.java:73) at java.net.URLClassLoader$1.run(URLClassLoader.java:368) at java.net.URLClassLoader$1.run(URLClassLoader.java:362) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:361) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.loadClass(DelegatingClassLoader.java:311) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) at org.apache.kafka.connect.runtime.isolation.PluginClassLoader.loadClass(PluginClassLoader.java:62) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at java.sql.DriverManager.isDriverAllowed(DriverManager.java:556) at java.sql.DriverManager.getConnection(DriverManager.java:661) at java.sql.DriverManager.getConnection(DriverManager.java:247) at io.confluent.connect.jdbc.source.JdbcSourceConnectorConfig$TableRecommender.validValues(JdbcSourceConnectorConfig.java:482) at io.confluent.connect.jdbc.source.JdbcSourceConnectorConfig$CachingRecommender.validValues(JdbcSourceConnectorConfig.java:521) at org.apache.kafka.common.config.ConfigDef.validate(ConfigDef.java:595) at org.apache.kafka.common.config.ConfigDef.validate(ConfigDef.java:616) at org.apache.kafka.common.config.ConfigDef.validate(ConfigDef.java:516) at org.apache.kafka.common.config.ConfigDef.validateAll(ConfigDef.java:499) at org.apache.kafka.common.config.ConfigDef.validate(ConfigDef.java:481) at org.apache.kafka.connect.connector.Connector.validate(Connector.java:133) at org.apache.kafka.connect.runtime.AbstractHerder.validateConnectorConfig(AbstractHerder.java:265) at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.putConnectorConfig(StandaloneHerder.java:163) at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:102) Caused by: java.lang.ClassNotFoundException: org.apache.calcite.avatica.UnregisteredDriver at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 32 more
Before this i have tested for twitter and ftp connector it was working after completed this two connector again i had run the jdbc its showing this error Previously it was working, after i run the twitter and ftp connector it is showing this jar issue
I have main class XML.scala, Project XMLHDFS
I am trying to run spark submit job with command
spark-submit --class com.abc.Import.XMLHDFS.XML /some-path/XMLHDFS.jar
it is giving me class not found exception. Please suggest where I am wrong.
java.lang.ClassNotFoundException: com.abc.Import.XMLHDFS.XML
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:278)
at org.apache.spark.util.Utils$.classForName(Utils.scala:172)
at org.apache.spark.util.Utils$.classForName(Utils.scala:172)
......................
I installed giraph-1.0.0 on my 2-node cluster, built with hadoop version 2.0.4-alpha where as i have hadoop 2.7.1 installed on my cluster. When I tried to run the giraph example for Simple Shortest Path Vertex, I got the following NoClassDefFoundError.
hadoop jar $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.0.0-for-hadoop-2.0.4-alpha-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsVertex -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/hduser/input/tinygraph.txt -of org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/hduser/output/shortestpaths -w 1
The error is
16/08/05 14:26:09 INFO utils.ConfigurationUtils: No edge input format specified. Ensure your InputFormat does not require one.
16/08/05 14:26:09 WARN job.GiraphConfigurationValidator: Output format vertex index type is not known
16/08/05 14:26:09 WARN job.GiraphConfigurationValidator: Output format vertex value type is not known
16/08/05 14:26:09 WARN job.GiraphConfigurationValidator: Output format edge value type is not known
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/client/YarnClientImpl
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:803)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:83)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:126)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.yarn.client.YarnClientImpl
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 21 more
The command I used for building giraph is this.
mvn -Phadoop_yarn -DskipTests -Dhadoop.version=2.0.4-alpha clean package
I am not sure if I need to rebuild apache giraph or it is something else which is the problem. Thanks in advance.
I tried to install the plugin "knapsack" on elasticsearch,
but in my log I see the error
[2015-10-20 09:12:49,739][WARN ][plugins ] [yt1] plugin knapsack, failed to invoke custom onModule method
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.elasticsearch.plugins.PluginsService.processModule(PluginsService.java:200)
at org.elasticsearch.plugins.PluginsModule.processModule(PluginsModule.java:61)
at org.elasticsearch.common.inject.Modules.processModules(Modules.java:64)
at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:58)
at org.elasticsearch.node.internal.InternalNode.<init>(InternalNode.java:197)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:70)
at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:203)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)
Caused by: java.lang.VerifyError: class org.xbib.elasticsearch.action.RestExportAction overrides final method handleRequest.(Lorg/elasticsearch/rest/RestRequest;Lorg/elasticsearch/rest/RestChannel;)V
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.xbib.elasticsearch.knapsack.KnapsackPlugin.onModule(KnapsackPlugin.java:40)
... 13 more
when I try to export a index it states:
{"error":"MapperParsingException[failed to parse, document is empty]","status":400}
Do you have any idea of how to solve this error? Unfortunately I did not find a solution on Google.
Thanks in advance :)
ok, I've found the answer myself.
You need the same version of elasticsearch and knapsack
I am doing sqoop import in MAC OSX 10.9.4 and getting error as:
14/10/24 11:51:41 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5
14/10/24 11:51:41 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override
14/10/24 11:51:41 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.
14/10/24 11:51:41 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
14/10/24 11:51:41 INFO tool.CodeGenTool: Beginning code generation
14/10/24 11:51:42 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `test_log` AS t LIMIT 1
14/10/24 11:51:42 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `test_log` AS t LIMIT 1
14/10/24 11:51:42 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /Users/hadoop/hadoop-2.5.1/share/hadoop/mapreduce
Note: /tmp/sqoop-hadoop/compile/2398ed5be0489f1c76d6fb556f3e0e9d/test_log.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
14/10/24 11:51:45 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hadoop/compile/2398ed5be0489f1c76d6fb556f3e0e9d/test_log.jar
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/InputFormat
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.sqoop.manager.ImportJobContext.<init>(ImportJobContext.java:51)
at com.cloudera.sqoop.manager.ImportJobContext.<init>(ImportJobContext.java:33)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:483)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:601)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.InputFormat
at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 58 more
Hadoop version : 2.5.1
Sqoop version :1.4.5
os : OSX10.9.4
env varibles are set:
export HADOOP_COMMON_HOME=/Users/hadoop/hadoop-2.5.1/
export HADOOP_MAPRED_HOME=/Users/hadoop/hadoop-2.5.1/share/hadoop/mapreduce/
Sqoop jar (sqoop-1.4.5.jar) is placed in /Users/hadoop/hadoop-2.5.1/lib/
and mysql connector in /Users/hadoop/sqoop/lib
also the core.jar exist in
/Users/hadoop/hadoop-2.5.1/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.5.1.jar
for f in ${HADOOP_MAPRED_HOME}/*.jar; do
HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:$f;
done
add this to bin/configure-sqoop
Somehow the Jars were not accessible. After puting the Sqoop jars in HDFS, Sqoop import went fine.
This link helped
Sqoop jar files not found
cd $SQOOP_HOME/bin
vim configure-sqoop
add below line
add_to_classpath ${HADOOP_MAPRED_HOME}
after "add_to_classpath ${SQOOP_JAR_DIR}"
Try to check your if in your $HADOOP_COMMON_LIB_NATIVE_DIR you can find the library.
If not you can put it in the same dir of your mapreduce implementation:
/Users/hadoop/hadoop-2.5.1/share/hadoop/mapreduce/
I had the same problem ! Just copy hadoop-mapreduce-client-core-3.0.0-SNAPSHOT.jar into the lib folder from sqooq. Dirty but it works. You probably just want to make some tests