I'm using hadoop to design mapreduce job. I decided to create a custom InputFormat.
When I want to run the job on my Hortonworks Sandbox, I got this error :
Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
According to some topics on forums (like here), it is a version problem : in Hadoop 2.x, TaskAttemptContext is an interface, before it was a class.
Here is my pom.xml file :
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.4</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>1.2.1</version>
</dependency>
<dependency>
<groupId>jdk.tools</groupId>
<artifactId>jdk.tools</artifactId>
<version>${java.version}</version>
<scope>system</scope>
<systemPath>${JAVA_HOME}/lib/tools.jar</systemPath>
</dependency>
<dependency>
<groupId>org.apache.mrunit</groupId>
<artifactId>mrunit</artifactId>
<version>0.9.0-incubating</version>
<classifier>hadoop1</classifier>
</dependency>
</dependencies>
Thanks to help me if you have an idea.
This probably means you compiled your code against an earlier version of Hadoop than you're running on.
This might have something to do with it...
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>1.2.1</version>
</dependency>
You have two different versions of Hadoop here.
Related
I'm a new in Hadoop. Yesterday I followed the book and used JUnit as the apper unit test for meteorological data. But there were some problems.
This is my pom file:
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.9.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>2.9.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>1.2.1</version>
</dependency>
<dependency>
<groupId>org.apache.mrunit</groupId>
<artifactId>mrunit</artifactId>
<version>1.1.0</version>
<classifier>hadoop2</classifier>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-minicluster</artifactId>
<version>2.9.0</version>
<scope>test</scope>
</dependency>
</dependencies>
</project>
and this are problems:
java.lang.IncompatibleClassChangeError: Found class
org.apache.hadoop.mapreduce.TaskInputOutputContext, but interface was
expected
Look forward and thank you for your answer！
I just found out that in Hadoop 2.x, there is not Hadoop-core, so the right pom is:
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.9.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>2.9.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>2.9.0</version>
</dependency>
<dependency>
<groupId>org.apache.mrunit</groupId>
<artifactId>mrunit</artifactId>
<version>1.1.0</version>
<classifier>hadoop2</classifier>
<scope>test</scope>
</dependency>
try adding different jar version.
I changed the from mrunit-1.1.0.jar to mrunit-0.9.0-incubating-hadoop2.jar
OLD JAR:-
public interface Mapper<K1, V1, K2, V2> extends JobConfigurable, Closeable { } NEW JAR:-
public class Mapper<KEYIN, VALUEIN, KEYOUT, VALUEOUT> {}
I just changed the jar from hadoop1 to hadoop2 and after that all the my unit test cases ran successfully. https://issues.apache.org/jira/browse/MRUNIT-156
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.org</groupId>
<artifactId>com.org</artifactId>
<version>0.0.1-SNAPSHOT</version>
<dependencies>
<!-- JDK Tools -->
<dependency>
<groupId>jdk.tools</groupId>
<artifactId>jdk.tools</artifactId>
<scope>system</scope>
<version>1.8.0_73</version>
<systemPath>${JAVA_HOME}/lib/tools.jar</systemPath>
</dependency>
<!-- Hadoop Core -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>1.2.1</version>
</dependency>
<!-- Hadoop Mapreduce Client Core -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>2.7.1</version>
</dependency>
<dependency>
<groupId>org.apache.mrunit</groupId>
<artifactId>mrunit</artifactId>
<version>0.9.0-incubating</version>
<classifier>hadoop1</classifier>
</dependency>
<dependency>
<groupId>org.apache.mrunit</groupId>
<artifactId>mrunit</artifactId>
<version>0.9.0-incubating</version>
<classifier>hadoop2</classifier>
</dependency>
</dependencies>
</project>
I am trying to implement an Spark app WordCount, but i get an error when I add hadoop-common dependency:
'Maven Dependencies' references non existing library '/root/.m2/repository/org/apache/hadoop/hadoop-common/2.6.0-cdh5.9.0/hadoop-common-2.6.0-cdh5.9.0.jar'
and this my pom.xml :
<dependencies>
<!-- https://mvnrepository.com/artifact/org.scala-lang/scala-library -->
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.10.6</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.6.0-cdh5.9.0</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
</dependencies>
my haddop version is 2.6.0-cdh5.9.0.
Any help please!!
replace
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.6.0-cdh5.9.0</version>
</dependency>
with
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.6.0</version>
</dependency>
or use external jar and add http://repo.spring.io/libs-milestone/org/apache/hadoop/hadoop-common/2.6.0-cdh5.9.0/hadoop-common-2.6.0-cdh5.9.0.jar
hope this helps
Below is my pom.xml. I build the jar with maven shade. I am very sure with the org.apache.kafka.clients.consumer.Consumer is included in my uber jar. Also I have put the kafka-clients-0.10.1.0.jar into spark spark.driver.extraLibraryPath. I also tried add --jars option in spark-submit command. But I still get the classNotFoundException.
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-reflect</artifactId>
<version>2.11.8</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.0.2</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.11</artifactId>
<version>2.0.2</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
<version>2.0.2</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.11</artifactId>
<version>0.10.1.0</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
</dependencies>
I just find a bypass solution. Add the jar into SPARK_HOME/jars.
I use spark-submit command. Tried add --jars,--driver-library-path. I am sure the options take effect. But still classNotFound.
I find the bypass solution according to the driver log listed as below.
enter image description here
Basically, you need:
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>0.10.1.0</version>
</dependency>
I am using hadoop and hbase. the hbase version is 1.1.1.2.3. I try to use maven to compile, run the project. But it has the org.apache.hadoop.hbase.HBaseConfiguration not find problem.
Here is my pom.xml:
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>0.20.2</version>
</dependency>
<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
<version>3.4.6</version>
</dependency>
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>2.4.0a</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>11.0.2</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-client</artifactId>
<version>1.1.2</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>0.23.1</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-protocol</artifactId>
<version>1.1.2</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase</artifactId>
<version>0.94.27</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-server</artifactId>
<version>1.1.1</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-testing-util</artifactId>
<version>1.1.1</version>
</dependency>
</dependencies>
dependencies are as above, I really do not know what to add more. Could anyone help? with many thanks. I know there are many problems similar but none of them is using the newest version base and I tried there solution, does not work. thanks
Furthermore, I tried to add the dependency: http://mvnrepository.com/artifact/org.apache.hbase/hbase/1.1.1
but get error: Failed to execute goal on project hbaseTable: Could not resolve dependencies for project apache.hbase:hbaseTable:jar:0.0.1-SNAPSHOT: Failure to find org.apache.hbase:hbase:jar:1.1.1 in https://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced
i am new to Maven and OSGI, can anyone please help me knowing how to deploy Microsoft Azure API jar on karaf OSGI container?
Following is pom content of my Maven project.
com.microsoft.windowsazure
microsoft-windowsazure-api
0.4.3
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.1</version>
</dependency>
<dependency>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
<version>1.1.1</version>
</dependency>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-core-asl</artifactId>
<version>1.9.2</version>
</dependency>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-jaxrs</artifactId>
<version>1.9.2</version>
</dependency>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mapper-asl</artifactId>
<version>1.9.2</version>
</dependency>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-xc</artifactId>
<version>1.9.2</version>
</dependency>
<dependency>
<groupId>javax.inject</groupId>
<artifactId>javax.inject</artifactId>
<version>1</version>
</dependency>
<dependency>
<groupId>com.sun.xml.bind</groupId>
<artifactId>jaxb-impl</artifactId>
<version>2.2.3-1</version>
</dependency>
<dependency>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-client</artifactId>
<version>1.13</version>
</dependency>
<dependency>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-core</artifactId>
<version>1.13</version>
</dependency>
<dependency>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-json</artifactId>
<version>1.13</version>
</dependency>
<dependency>
<groupId>org.codehaus.jettison</groupId>
<artifactId>jettison</artifactId>
<version>1.1</version>
</dependency>
<dependency>
<groupId>javax.mail</groupId>
<artifactId>mail</artifactId>
<version>1.4.5</version>
</dependency>
i have embedded dependencies like following
<Embed-Dependency>
microsoft-windowsazure-api,
commons-lang3,
commons-logging,
jackson-core-asl,
jackson-jaxrs,
jackson-mapper-asl,
jackson-xc,
javax.inject,
jaxb-impl,
jersey-client,
jersey-core,
jersey-json,
jettison,
mail
</Embed-Dependency>
When i do this it keeps showing missing dependencies for different packages. i have imported all packages (*) in pom.
download bnd.jar (bundle tool)
keep azur and bnd both jar in a directory
go to that directory and run following command
java -jar bnd.jar wrap microsoft-windowsazure-api 0.4.3.jar
tada... your bundle is ready, just deploy it and continue your work :)
Take a look at jcloud feature for Karaf it also contains some bundles for azure. I'm sure this will give you a list of bundles working.