Below is my pom.xml. I build the jar with maven shade. I am very sure with the org.apache.kafka.clients.consumer.Consumer is included in my uber jar. Also I have put the kafka-clients-0.10.1.0.jar into spark spark.driver.extraLibraryPath. I also tried add --jars option in spark-submit command. But I still get the classNotFoundException.
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-reflect</artifactId>
<version>2.11.8</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.0.2</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.11</artifactId>
<version>2.0.2</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
<version>2.0.2</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.11</artifactId>
<version>0.10.1.0</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
</dependencies>
I just find a bypass solution. Add the jar into SPARK_HOME/jars.
I use spark-submit command. Tried add --jars,--driver-library-path. I am sure the options take effect. But still classNotFound.
I find the bypass solution according to the driver log listed as below.
enter image description here
Basically, you need:
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>0.10.1.0</version>
</dependency>
Related
I am trying to implement an Spark app WordCount, but i get an error when I add hadoop-common dependency:
'Maven Dependencies' references non existing library '/root/.m2/repository/org/apache/hadoop/hadoop-common/2.6.0-cdh5.9.0/hadoop-common-2.6.0-cdh5.9.0.jar'
and this my pom.xml :
<dependencies>
<!-- https://mvnrepository.com/artifact/org.scala-lang/scala-library -->
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.10.6</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.6.0-cdh5.9.0</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
</dependencies>
my haddop version is 2.6.0-cdh5.9.0.
Any help please!!
replace
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.6.0-cdh5.9.0</version>
</dependency>
with
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.6.0</version>
</dependency>
or use external jar and add http://repo.spring.io/libs-milestone/org/apache/hadoop/hadoop-common/2.6.0-cdh5.9.0/hadoop-common-2.6.0-cdh5.9.0.jar
hope this helps
I am using hadoop and hbase. the hbase version is 1.1.1.2.3. I try to use maven to compile, run the project. But it has the org.apache.hadoop.hbase.HBaseConfiguration not find problem.
Here is my pom.xml:
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>0.20.2</version>
</dependency>
<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
<version>3.4.6</version>
</dependency>
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>2.4.0a</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>11.0.2</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-client</artifactId>
<version>1.1.2</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>0.23.1</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-protocol</artifactId>
<version>1.1.2</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase</artifactId>
<version>0.94.27</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-server</artifactId>
<version>1.1.1</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-testing-util</artifactId>
<version>1.1.1</version>
</dependency>
</dependencies>
dependencies are as above, I really do not know what to add more. Could anyone help? with many thanks. I know there are many problems similar but none of them is using the newest version base and I tried there solution, does not work. thanks
Furthermore, I tried to add the dependency: http://mvnrepository.com/artifact/org.apache.hbase/hbase/1.1.1
but get error: Failed to execute goal on project hbaseTable: Could not resolve dependencies for project apache.hbase:hbaseTable:jar:0.0.1-SNAPSHOT: Failure to find org.apache.hbase:hbase:jar:1.1.1 in https://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced
I'm using hadoop to design mapreduce job. I decided to create a custom InputFormat.
When I want to run the job on my Hortonworks Sandbox, I got this error :
Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
According to some topics on forums (like here), it is a version problem : in Hadoop 2.x, TaskAttemptContext is an interface, before it was a class.
Here is my pom.xml file :
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.4</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>1.2.1</version>
</dependency>
<dependency>
<groupId>jdk.tools</groupId>
<artifactId>jdk.tools</artifactId>
<version>${java.version}</version>
<scope>system</scope>
<systemPath>${JAVA_HOME}/lib/tools.jar</systemPath>
</dependency>
<dependency>
<groupId>org.apache.mrunit</groupId>
<artifactId>mrunit</artifactId>
<version>0.9.0-incubating</version>
<classifier>hadoop1</classifier>
</dependency>
</dependencies>
Thanks to help me if you have an idea.
This probably means you compiled your code against an earlier version of Hadoop than you're running on.
This might have something to do with it...
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>1.2.1</version>
</dependency>
You have two different versions of Hadoop here.
i am new to Maven and OSGI, can anyone please help me knowing how to deploy Microsoft Azure API jar on karaf OSGI container?
Following is pom content of my Maven project.
com.microsoft.windowsazure
microsoft-windowsazure-api
0.4.3
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.1</version>
</dependency>
<dependency>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
<version>1.1.1</version>
</dependency>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-core-asl</artifactId>
<version>1.9.2</version>
</dependency>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-jaxrs</artifactId>
<version>1.9.2</version>
</dependency>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mapper-asl</artifactId>
<version>1.9.2</version>
</dependency>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-xc</artifactId>
<version>1.9.2</version>
</dependency>
<dependency>
<groupId>javax.inject</groupId>
<artifactId>javax.inject</artifactId>
<version>1</version>
</dependency>
<dependency>
<groupId>com.sun.xml.bind</groupId>
<artifactId>jaxb-impl</artifactId>
<version>2.2.3-1</version>
</dependency>
<dependency>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-client</artifactId>
<version>1.13</version>
</dependency>
<dependency>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-core</artifactId>
<version>1.13</version>
</dependency>
<dependency>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-json</artifactId>
<version>1.13</version>
</dependency>
<dependency>
<groupId>org.codehaus.jettison</groupId>
<artifactId>jettison</artifactId>
<version>1.1</version>
</dependency>
<dependency>
<groupId>javax.mail</groupId>
<artifactId>mail</artifactId>
<version>1.4.5</version>
</dependency>
i have embedded dependencies like following
<Embed-Dependency>
microsoft-windowsazure-api,
commons-lang3,
commons-logging,
jackson-core-asl,
jackson-jaxrs,
jackson-mapper-asl,
jackson-xc,
javax.inject,
jaxb-impl,
jersey-client,
jersey-core,
jersey-json,
jettison,
mail
</Embed-Dependency>
When i do this it keeps showing missing dependencies for different packages. i have imported all packages (*) in pom.
download bnd.jar (bundle tool)
keep azur and bnd both jar in a directory
go to that directory and run following command
java -jar bnd.jar wrap microsoft-windowsazure-api 0.4.3.jar
tada... your bundle is ready, just deploy it and continue your work :)
Take a look at jcloud feature for Karaf it also contains some bundles for azure. I'm sure this will give you a list of bundles working.
I have a web-app with following dependencies in pom:
<dependency>
<groupId>javax</groupId>
<artifactId>javaee-web-api</artifactId>
<version>6.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.derby</groupId>
<artifactId>derby</artifactId>
<version>10.9.1.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-entitymanager</artifactId>
<version>4.1.9.Final</version>
<scope>test</scope>
</dependency>
With this configuration, I get following error in junit test:
java.lang.ClassFormatError: Absent Code attribute in method that is
not native or abstract in class file javax/persistence/Persistence
I found some info on this error here.
So, I removed 'javaee-web-api' dependency.
After that I get compilation error 'package javax.persistence does not exist', and many more.
So, I changed scope of 'hibernate-entitymanager' to default (compile).
The working pom (dependencies) look like this:
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.derby</groupId>
<artifactId>derby</artifactId>
<version>10.9.1.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-entitymanager</artifactId>
<version>4.1.9.Final</version>
</dependency>
Now everything is OK, except generated war contains hibernate jar, which I do not want.
Any suggestion how to fix this, correct maven pom?
If you are working with an in application server, you can replace the hibernate dependency with a (still vendor specific) JPA API with provided scope like this ( example is for JBoss with hibernate):
<dependency>
<groupId>org.hibernate.javax.persistence</groupId>
<artifactId>hibernate-jpa-2.0-api</artifactId>
<scope>provided</scope>
</dependency>
Finally, I got this working, with same set of dependencies, using profiles.
Here is relevant section of pom:
<profiles>
<profile>
<id>test</id>
<dependencies>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-entitymanager</artifactId>
<version>4.1.9.Final</version>
</dependency>
</dependencies>
</profile>
<profile>
<id>package</id>
<dependencies>
<dependency>
<groupId>javax</groupId>
<artifactId>javaee-web-api</artifactId>
<version>6.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-entitymanager</artifactId>
<version>4.1.9.Final</version>
<scope>test</scope>
</dependency>
</dependencies>
</profile>
</profiles>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.derby</groupId>
<artifactId>derby</artifactId>
<version>10.9.1.0</version>
<scope>test</scope>
</dependency>
</dependencies>
Now, I use following commands to package and test,
mvn -Ptest test
mvn -Ppackage package -DskipTests
-DskipTests is required, as, otherwise, package goal triggers 'test' causing the same error I started with, as it uses same set of dependencies configured for 'package'.
java.lang.ClassFormatError: Absent Code attribute in method that is
not native or abstract in class file javax/persistence/Persistence
I was having the same problem and was able to solve it by adding a couple of dependencies to the Maven pom.xml file. See this answer for more details.