Using Maven 3.0.5
I'm trying to get spark-testing-base from com.holdenkarau to work with Hadoop 3.1. holdenkarau's dependency tree includes Hadoop 2.8.3; which is why I think I'm getting errors.
From my mvn dependency:tree I see the following lines:
[INFO] +- org.apache.hadoop:hadoop-common:jar:3.1.0:provided
...
[INFO] | +- org.apache.hadoop:hadoop-common:jar:tests:2.8.3:test
These lines come from these two lines in the pom.xml file:
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>3.1.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.holdenkarau</groupId>
<artifactId>spark-testing-base_${scala.compat.version}</artifactId>
<version>${spark.version}_0.12.0</version>
<scope>test</scope>
<exclusions>
<exclusion>
<groupId>commons-beanutils</groupId>
<artifactId>commons-beanutils-core</artifactId>
</exclusion>
</exclusions>
</dependency>
I basically have two related questions:
What is the difference between org.apache.hadoop:hadoop-common:jar:3.1.0 and org.apache.hadoop:hadoop-common:jar:tests:2.8.3. What is that extra tests in there for; where does it come from and what does it mean?
If I have a dependency that uses an older version of a package in the test scope, how do I force it to use a newer version; i.e., how do I force spark-testing-base to use Hadoop 3.1 in the test scope.
tests is called a classifier, and it contains code that's really only useful in the context of actually testing, such as an embedded HDFS system
You could explicitly try pulling in a new version like so, assuming it exists
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>3.1.0</version>
<scope>test</scope>
<classifier>test</classifier>
</dependency>
You may also want to exclude the same within the other dependency, however you might then run into build issues since that library is only written to test against 2.8.3
Related
I excluded an artifact because it causes conflicts, namely the jsr311-api given below. Yet when I run the generated jar I'm still getting the java.lang.NoSuchMethodError error. I suspect that another dependency is also including this artifact. How can I find out which one? My dependency list is quite large. Which dependencies include the package javax.ws.rs.core?
<dependency>
<groupId>org.apache.axis2</groupId>
<artifactId>axis2-kernel</artifactId>
<version>1.7.3</version>
<exclusions>
<!-- Causes java.lang.NoSuchMethodError: javax.ws.rs.core.Response$Status$Family.familyOf(I)Ljavax/ws/rs/core/Response$Status$Family; -->
<exclusion>
<artifactId>jsr311-api</artifactId>
<groupId>javax.ws.rs</groupId>
</exclusion>
</exclusions>
</dependency>
Go to
http://search.maven.org/#advancedsearch%7Cgav
and use classname search to find
javax.ws.rs.core.Response
If you use a Nexus 2.x in your company, you can use classname search there as well.
If you want to find out where a given artifact (that you e.g. found by classnmae search) comes from, use dependency:tree in Maven.
In my case the mistake was that I had to manually add the javaee api and I set <scope>provided</scope> which was a mistake, fixing this solved the problem.
<dependency>
<groupId>javax</groupId>
<artifactId>javaee-api</artifactId>
<version>7.0</version>
<scope>provided</scope><!-- remove this -->
</dependency>
I would like to use the logback slf4j implementation in my application, but activemq-all is spoiling the classpath by including the log4j implementation classes. I'm not the only one facing that problem, as witnessed by for instance multiple SLF4J bindings Error with activemq-all-5.6.0.jar. According to that post I have to replace activemq-all by
org.apache.activemq:activemq-camel
org.apache.activemq:activemq-core
org.apache.activemq:activemq-console
org.apache.activemq:activemq-jaas
org.apache.activemq:activemq-optional
org.apache.activemq:kahadb
org.apache.geronimo.specs:geronimo-jms_1.1_spec
org.apache.geronimo.specs:geronimo-jta_1.0.1B_spec
org.apache.geronimo.specs:geronimo-j2ee-management_1.1_spec
org.apache.geronimo.specs:geronimo-annotation_1.0_spec.
The problem is that I don't have the complete maven dependencies (group id, artifact id, version) for these artifacts. Can someone provide me with a ready-to-use replacement for
<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-all</artifactId>
<version>5.9.0</version>
</dependency>
You can use active mq core library. Please note that active mq is backward compatible for client.
<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-core</artifactId>
<version>5.4.3</version>
<exclusions>
<exclusion>
<artifactId>org.slf4j</artifactId>
<groupId>slf4j-log4j12</groupId>
</exclusion>
<exclusion>
<artifactId>log4j</artifactId>
<groupId>log4j</groupId>
</exclusion>
</exclusions>
</dependency>
In a nutshell, you have already listed group id/artifact id separated by a colon for the artifact you found. Please note that these satisfy some usecase with ActiveMQ 5.6. For instance activemq-core is not really valid any more - use activemq-client and activemq-broker instead.
Currently, these artifact are bundled in activemq-all. But you may want to check out the pom.xml for your version of choice (this list might change over time). You probably won't need all of them unless you are about to embedd a broker with all transports, plugins and configurations within your applications.
<artifactSet>
<includes>
<include>${project.groupId}:activemq-client</include>
<include>${project.groupId}:activemq-openwire-legacy</include>
<include>${project.groupId}:activemq-camel</include>
<include>${project.groupId}:activemq-jaas</include>
<include>${project.groupId}:activemq-broker</include>
<include>${project.groupId}:activemq-console</include>
<include>${project.groupId}:activemq-shiro</include>
<include>${project.groupId}:activemq-spring</include>
<include>${project.groupId}:activemq-pool</include>
<include>${project.groupId}:activemq-jms-pool</include>
<include>${project.groupId}:activemq-amqp</include>
<include>${project.groupId}:activemq-http</include>
<include>${project.groupId}:activemq-mqtt</include>
<include>${project.groupId}:activemq-stomp</include>
<include>${project.groupId}:activemq-kahadb-store</include>
<include>${project.groupId}:activemq-leveldb-store</include>
<include>${project.groupId}:activemq-jdbc-store</include>
<include>org.apache.activemq.protobuf:activemq-protobuf</include>
<include>org.fusesource.hawtbuf:hawtbuf</include>
<include>org.jasypt:jasypt</include>
<include>org.apache.geronimo.specs:geronimo-jms_1.1_spec</include>
<include>org.apache.geronimo.specs:geronimo-jta_1.0.1B_spec</include>
<include>org.apache.geronimo.specs:geronimo-j2ee-management_1.1_spec</include>
<include>org.apache.geronimo.specs:geronimo-annotation_1.0_spec</include>
<include>org.slf4j:slf4j-api</include>
<include>org.slf4j:slf4j-log4j12</include>
<include>log4j:log4j</include>
</includes>
</artifactSet>
Ok, the version number for org.apache.activemq should simply be the release you want to use. For the geronimo specs, this is not so obvious.
<dependency>
<groupId>org.apache.geronimo.specs</groupId>
<artifactId>geronimo-jms_1.1_spec</artifactId>
<version>1.1.1</version>
</dependency>
<dependency>
<groupId>org.apache.geronimo.specs</groupId>
<artifactId>geronimo-j2ee-management_1.1_spec</artifactId>
<version>1.0.1</version>
</dependency>
<dependency>
<groupId>org.apache.geronimo.specs</groupId>
<artifactId>geronimo-annotation_1.0_spec</artifactId>
<version>1.1.1</version>
</dependency>
I was also facing this same issue with activemq-all API and I replaced this dependency with this below dependency and it worked for me.
<!-- https://mvnrepository.com/artifact/org.apache.activemq/activemq-spring -->
<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-spring</artifactId>
<version>5.14.3</version>
</dependency>
Hope this can help others.
I am a newbie in Solr and maven and i want to make a small application that index all my database tables via SolrJ .
For that i looked up at this tutorial where they are using MAVEN .
I installed the librairies and jars (except maven) but i had this exception:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/http/HttpRequestInterceptor
I looked into the tutorial and i saw that for resolving this problem we need to add this to my maven configuration:
org.slf4j
slf4j-simple
1.5.6
Is there anyway to do that without maven?
Thank you
Use maven. Even with it, it took me a fairly considerable amount of time to get the dependencies right. The tutorials were all a bit lacking. Below is my pom.xml with the relevant dependencies that I had maven bring in. Perhaps it will help you.
<dependency>
<groupId>org.apache.solr</groupId>
<artifactId>solr-core</artifactId>
<version>4.3.0</version>
</dependency>
<dependency>
<artifactId>solr-solrj</artifactId>
<groupId>org.apache.solr</groupId>
<version>4.3.0</version>
<type>jar</type>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
<version>1.1.1</version>
</dependency>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
<version>2.5</version>
</dependency>
Maven is the suggested build technology for the Solrj, because it automates the management of 3rd party dependencies. Without dependency management it's a royal pain to decipher these relationships (Jar hell).
What I could suggest is to use ivy, which has a command-line mode.
First download the ivy jar
http://search.maven.org/remotecontent?filepath=org/apache/ivy/ivy/2.3.0/ivy-2.3.0.jar
To retrieve the following Maven module and all it's dependencies:
<dependency>
<artifactId>solr-solrj</artifactId>
<groupId>org.apache.solr</groupId>
<version>1.4.0</version>
<type>jar</type>
<scope>compile</scope>
</dependency>
Then run it as follows:
java -jar ivy.jar \
-dependency org.apache.solr solr-solrj 1.4.0 \
-retrieve "lib/[artifact]-[revision](-[classifier]).[ext]" \
-confs default
Retrieves into the lib directory:
lib/commons-httpclient-3.1.jar
lib/wstx-asl-3.2.7.jar
lib/slf4j-api-1.5.5.jar
lib/commons-codec-1.3.jar
lib/stax-api-1.0.1.jar
lib/geronimo-stax-api_1.0_spec-1.0.1.jar
lib/commons-logging-1.0.4.jar
lib/solr-solrj-1.4.0.jar
lib/commons-io-1.4.jar
lib/commons-fileupload-1.2.1.jar
Update
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/http/HttpRequestInterceptor
This is due to a missing httpcore.jar file. I found this out by browsing Maven Central:
http://search.maven.org/#search|ga|1|fc%3A%22org.apache.http.HttpRequestInterceptor%22
The recommendation on using the "slf4j-simple" is to provide a logging implementation in case your application doesn't have one.
Finally... This demonstrates what I've tried to say. In the absence of a dependency management tool (ivy, groovy, Maven) you're on your own in deciphering the 3rd party jar dependencies.
maven noob, be patient...
I'm upgrading from cdh3u1 to apache hadoop 0.20.203.0 and pig 0.9.0. I used to have:
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>0.20.2-cdh3u1</version>
</dependency>
<dependency>
<groupId>org.apache.pig</groupId>
<artifactId>pig</artifactId>
<version>0.8.1-cdh3u1</version>
</dependency>
and running them from inside eclipse, with junit run configuration worked great.
Now I have:
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>0.20.203.0</version>
</dependency>
<dependency>
<groupId>org.apache.pig</groupId>
<artifactId>pig</artifactId>
<version>0.9.0</version>
</dependency>
and I got NoClassDefFoundError: jline/ConsoleReaderInputStream on runtime.
I ended with adding all these dependencies manually until it worked:
<dependency>
<groupId>jline</groupId>
<artifactId>jline</artifactId>
<version>0.9.94</version>
</dependency>
<dependency>
<groupId>org.antlr</groupId>
<artifactId>antlr-runtime</artifactId>
<version> 3.2 </version> <- this is 3.0.1 in cdh3u1, but probably changed in pig 0.9.0
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>r06</version>
</dependency>
What gives? why isn't maven automatically pulling my dependencies and putting them in the classpath?
Maven has a feature called Transitive dependencies, so you don´t have to specify the libraries that your own dependencies require.
ConsoleReaderInputStream is in the Jline JAR. When you were using Pig.0.8.1-cdh3u1, you didn´t have to add the Jline dependency because it is declared in Pig.0.8.1-cdh3u1.pom. Pig 0.9.0.pom does not have Jline dependency declared anymore, that´s the reason you had to add it by yourself. As for the reason JLine was removed from Pig, you have to ask the developers of that project.
How can I remove artifacts from the lib folder of the ear? For exemple: activation.jar.
I was try follow the especification about Excluding a module (just change webModule for jarModule) but without success with this error: Artifact[jar:javax.activation:activation] is not a dependency of the project.
Use provided as scope wherever the dependency is defined.
<dependency>
<groupId>javax.activation</groupId>
<artifactId>activation</artifactId>
<version>1.1.1</version>
<scope>provided</scope>
</dependency>
Set the scope to provided. Maven will still add the dependency to the compile time classpath but not to the final WAR.
As others have said, you need to use provided as the scope. If the jars you need to remove are not dependencies of your project directly (ie are transient dependencies), then you need to exclude them from whatever is bringing them into your build . If you actually need those classes at compile-time, you'll have to re-include them as provided:
<dependency>
<!-- this dependency has javax.activation as a dependency -->
<groupId>blah.groupId</groupId>
<artifactId>blah.artifactId</artifactId>
<version>blah.version</version>
<exclusions>
<exclusion>
<groupId>javax.activation</groupId>
<artifactId>activation</artifactId>
<exclusion>
</exclusions>
</dependency>
<!-- only necessary if this jar is needed at compile-time -->
<dependency>
<groupId>javax.activation</groupId>
<artifactId>activation</artifactId>
<version>1.1.1</version>
<scope>provided</scope>
</dependency>