pig-0.9.0.pom does not contain all its runtime dependencies, like pig-0.8.1-cdh3u1.pom - maven

maven noob, be patient...
I'm upgrading from cdh3u1 to apache hadoop 0.20.203.0 and pig 0.9.0. I used to have:
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>0.20.2-cdh3u1</version>
</dependency>
<dependency>
<groupId>org.apache.pig</groupId>
<artifactId>pig</artifactId>
<version>0.8.1-cdh3u1</version>
</dependency>
and running them from inside eclipse, with junit run configuration worked great.
Now I have:
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>0.20.203.0</version>
</dependency>
<dependency>
<groupId>org.apache.pig</groupId>
<artifactId>pig</artifactId>
<version>0.9.0</version>
</dependency>
and I got NoClassDefFoundError: jline/ConsoleReaderInputStream on runtime.
I ended with adding all these dependencies manually until it worked:
<dependency>
<groupId>jline</groupId>
<artifactId>jline</artifactId>
<version>0.9.94</version>
</dependency>
<dependency>
<groupId>org.antlr</groupId>
<artifactId>antlr-runtime</artifactId>
<version> 3.2 </version> <- this is 3.0.1 in cdh3u1, but probably changed in pig 0.9.0
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>r06</version>
</dependency>
What gives? why isn't maven automatically pulling my dependencies and putting them in the classpath?

Maven has a feature called Transitive dependencies, so you don´t have to specify the libraries that your own dependencies require.
ConsoleReaderInputStream is in the Jline JAR. When you were using Pig.0.8.1-cdh3u1, you didn´t have to add the Jline dependency because it is declared in Pig.0.8.1-cdh3u1.pom. Pig 0.9.0.pom does not have Jline dependency declared anymore, that´s the reason you had to add it by yourself. As for the reason JLine was removed from Pig, you have to ask the developers of that project.

Related

Excluding "provided" libraries from MobileFirst Maven builds

I have added a <scope>provided</scope> dependency to my Maven pom.xml for my Java adapter, because the referenced jars are provided by my application server, but Maven install is still including the jars in the .adapter file.
Is this the expected behavior? Is this a Maven issue, or something related to MobileFirst platform foundations use of it? The MFP provided libraries seem to not be included.
Not included in .adapter file:
<dependency>
<groupId>com.ibm.mfp</groupId>
<artifactId>adapter-maven-api</artifactId>
<scope>provided</scope>
<version>8.0.2017012516</version>
</dependency>
Included:
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.3</version>
<scope>provided</scope>
</dependency>

Hadoop/Hbase: java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration

I have hadoop and Hbase installed, both working fine as far as I can tell. When trying to the built jar with hadoop, I get a
java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration
error, using Hbase version 0.90.2 in my maven dependency.
I think this is quite an old version of Hbase and I am unsure if this old version is compatible with hadoop 2.7.2 or even Java 8. Thus I tried using Hbase version 0.99.2 in my maven dependency, but then I get a
Failed to execute goal on project exercise_2: Could not resolve dependencies for project com.company.exercise_2:exercise_2:jar:1.0-SNAPSHOT: Failure to find org.apache.hbase:hbase:jar:0.99.2 in http://repo.maven.apache.org/maven2 was cached in the local repository
error from the maven plugin. What am I doing wrong?
Here is my pom.xml:
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>2.7.2</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>2.7.2</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.7.2</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase</artifactId>
<version>0.99.2</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-client</artifactId>
<version>1.1.2</version>
<scope>provided</scope>
</dependency>
Seems like this is jar caching issue, I think HbaseConfiguration is common class regardless of which version of hbase used.
Can you manually delete local repository file of hbase and try mvn XXXX command once again.
Maven will then try to download and fix the class path.
for cross checking, use mvn ... -X option to see which version of jar its trying to download.
Since scope of this jar is
provided
Cross check the hbase version of this jar in your cluster. by using "hbase classpath" and check whether this jar version is closely matching with your jar file version of maven repository of your pom.xml.
That should fix.

Maven: 1.7.4 openejb dependancy fails

Wanted to make use of openejb on top of tomcat v7 using maven instead of installing tomee. Referring to Apache documentation, 3 dependencies have to be added to the maven project.
<dependency>
<groupId>org.apache.openejb</groupId>
<artifactId>javaee-api</artifactId>
<version>6.0-6</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.openejb</groupId>
<artifactId>openejb-core</artifactId>
<version>4.7.4</version>
</dependency>
<dependency>
<groupId>org.apache.openejb</groupId>
<artifactId>tomee</artifactId>
<version>1.7.4</version>
</dependency>
but the last depency generates following error: Missing artifact org.apache.openejb:tomee:jar:1.7.4
The correct artifactId being visible in:
http://mvnrepository.com/artifact/org.apache.openejb/apache-tomee/1.7.4
is apache-tomee and not tomee
so, replace the last depency by the following one and the problem will be solved:
<dependency>
<groupId>org.apache.openejb</groupId>
<artifactId>apache-tomee</artifactId>
<version>1.7.4</version>
</dependency>

maven artifactId hadoop 2.2.0 for hadoop-core

I am migrating my application from hadoop 1.0.3 to hadoop 2.2.0 and maven build had hadoop-core marked as dependency. Since hadoop-core is not present for hadoop 2.2.0. I tried replacing it with hadoop-client and hadoop-common but I am still getting this error for ant.filter. Can anybody please suggest which artifact to use?
previous config :
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>1.0.3</version>
</dependency>
New Config:
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.2.0</version>
</dependency>
Error:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project event: Compilation failure: Compilation failure:
[ERROR] /opt/teamcity/buildAgent/work/c670ebea1992ec2f/event/src/main/java/com/intel/event/EventContext.java:[27,36] package org.apache.tools.ant.filters does not exist
[ERROR] /opt/teamcity/buildAgent/work/c670ebea1992ec2f/event/src/main/java/com/intel/event/EventContext.java:[27,36] package org.apache.tools.ant.filters does not exist
[ERROR] /opt/teamcity/buildAgent/work/c670ebea1992ec2f/event/src/main/java/com/intel/event/EventContext.java:[180,59] cannot find symbol
[ERROR] symbol: class StringInputStream
[ERROR] location: class com.intel.event.EventContext
We mainly depend on hdfs api for our application. When we migrated to hadoop 2.X, we were surprised to see the changes in dependencies. We started adding dependencies one at a time. Today we depend on the following core libraries.
hadoop-annotations-2.2.0
hadoop-auth-2.2.0
hadoop-common-2.2.0
hadoop-hdfs-2.2.0
hadoop-mapreduce-client-core-2.2.0
In addition to these we depend on test libraries too. Based on your needs, you may want to include hadoop-hdfs and hadoop-mapreduce-client to the dependencies along with hadoop-common.
Try with these artifacts, word fine on my sample project wordcount
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>1.2.1</version>
</dependency>
Maven dependencies can be got from this link.
As far as hadoop-core dependies goes, hadoop-core was the name for hadoop 1.X and just renaming the version to 2.X wont help. Also in a hadoop 2.X project using the hadoop 1.X dependency gives an error like
Caused by: org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate with client version 4
Thus it is suggested not to use it. I have been using the following dependencies in my hadoop
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.7.1</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>2.7.1</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>2.7.1</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-jobclient</artifactId>
<version>2.7.1</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-common</artifactId>
<version>2.7.1</version>
</dependency>
<dependency>
You can try these.

multiple SLF4J bindings Error with activemq-all-5.6.0.jar

When I upgrade to activemq-all-5.6.0
I get this error during server startup
SLF4J: Class path contains multiple SLF4J bindings
I don't have this issue when using activemq-all-5.5.1
On checking I do find that there StaticLoggerBinder.class in both activemq-all-5.6.0.jar and slf4j-log4j12-1.5.10.jar which is causing the issue
Please do help in debugging this issue
My pom.xml is as follows
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.5.10</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>jcl-over-slf4j</artifactId>
<version>1.5.10</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.5.10</version>
<scope>runtime</scope>
</dependency>
The active mq dependency is like this
Old Version 5.5.1 (This works)
<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-all</artifactId>
<version>5.5.1</version>
</dependency>
New Version 5.6.0 (This gives the error)
<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-all</artifactId>
<version>5.6.0</version>
</dependency>
Thanks in advance.
The ActiveMQ guys use the Maven Shade Plugin to create the activemq-all "ueber" jar. Somewhere between version 5.5.1 and 5.6.0 they added the org.slf4j:slf4j-log4j12 dependency - hence your problem.
Unfortunately because they used the shade plugin you can not use exclusions in your activemq-all dependency definition in your POM.
Instead you will need to completely replace the activemq-all dependency with all the required individual dependencies (except of course the org.sl4j-log4j12 one).
The following page details all the required dependencies:
http://activemq.apache.org/initial-configuration.html#InitialConfiguration-RequiredJARs
Alternatively the following is the list of all dependencies (required and optional) included in the activemq-all jar (taken from the configuration of the shade plugin in the activemq-all pom):
org.apache.activemq:activemq-camel
org.apache.activemq:activemq-core
org.apache.activemq:activemq-console
org.apache.activemq:activemq-jaas
org.apache.activemq:activemq-optional
org.apache.activemq:kahadb
org.apache.geronimo.specs:geronimo-jms_1.1_spec
org.apache.geronimo.specs:geronimo-jta_1.0.1B_spec
org.apache.geronimo.specs:geronimo-j2ee-management_1.1_spec
org.apache.geronimo.specs:geronimo-annotation_1.0_spec
org.slf4j:slf4j-api
org.slf4j:slf4j-log4j12
log4j:log4j
Hope that helps.
I had the same problem while using Spring. What helped me was replacing the dependency of activemq-all with:
<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-spring</artifactId>
<version>5.14.3</version>
</dependency>
Hope this will help anyone...

Resources