In my work project, I use spark-submit to launch my application into a yarn cluster. I am quite new to Maven projects and pom.xml use, but the problem I seem to be having is that hadoop is using an older version of google protobuf (2.5.0) than the internal dependencies I'm importing at work (2.6.1).
The error is here:
java.lang.NoSuchMethodError:
com/google/protobuf/LazyStringList.getUnmodifiableView()Lcom/google/protobuf/LazyStringList;
(loaded from file:/usr/hdp/2.6.4.0-91/spark2/jars/protobuf-java-2.5.0.jar
by sun.misc.Launcher$AppClassLoader#8b6f2bf7)
called from class protobuf.com.mycompany.group.otherproject.api.JobProto$Query
Since I'm not quite sure how to approach dependency issues like this, and I can't change the code of the internal dependency that uses 2.6.1, I added the required protobuf version as a dependency to my project, as well:
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>2.6.1</version>
</dependency>
Unfortunately, this hasn't resolved the issue. When the internal dependency (which does import 2.6.1 on its own) tries to use its proto, the conflict occurs.
Any suggestions on how I could force the usage of the newer, correct version would be greatly appreciated.
Ultimately I found the Maven Shade Plugin to be the answer. I shaded my company's version of protobufs, deployed our service as an uber jar, and the conflict was resolved.
Related
I am trying to run Java code from a Maven project that uses both beam-runners-google-cloud-dataflow-java and bigtable-client-core, and I cannot get it to properly reconcile dependencies amongst these two. When I run and attempt to create a BigtableDataClient, I get the following error:
java.lang.NoSuchFieldError: TE_HEADER
at io.grpc.netty.shaded.io.grpc.netty.Utils.<clinit> (Utils.java:74)
at io.grpc.netty.shaded.io.grpc.netty.NettyChannelBuilder.<clinit> (NettyChannelBuilder.java:72)
at io.grpc.netty.shaded.io.grpc.netty.NettyChannelProvider.builderForAddress (NettyChannelProvider.java:37)
at io.grpc.netty.shaded.io.grpc.netty.NettyChannelProvider.builderForAddress (NettyChannelProvider.java:23)
at io.grpc.ManagedChannelBuilder.forAddress (ManagedChannelBuilder.java:39)
at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createSingleChannel (InstantiatingGrpcChannelProvider.java:242)
at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createChannel (InstantiatingGrpcChannelProvider.java:198)
at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.getTransportChannel (InstantiatingGrpcChannelProvider.java:185)
at com.google.api.gax.rpc.ClientContext.create (ClientContext.java:160)
at com.google.cloud.bigtable.data.v2.stub.EnhancedBigtableStub.create (EnhancedBigtableStub.java:151)
at com.google.cloud.bigtable.data.v2.BigtableDataClient.create (BigtableDataClient.java:138)
at com.google.cloud.bigtable.data.v2.BigtableDataClient.create (BigtableDataClient.java:130)
...
I can only conclude this is due to an issue with version conflict on the relevant libraries (either grpc-netty or grpc-netty-shaded); I'm using 1.17 for grpc-netty and 1.23 for grpc-netty-shaded. I've tried using dependencyManagement to force the use of version 1.23.0 for both grpc-netty and grpc-netty-shaded, and then tried 1.17 for both, but this doesn't help. I've also tried using earlier versions of both the Beam runners and bigtable-client-core, and this doesn't help either.
The relevant Maven dependencies are:
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
<version>2.15.0</version>
</dependency>
<dependency>
<groupId>com.google.cloud.bigtable</groupId>
<artifactId>bigtable-client-core</artifactId>
<version>1.12.1</version>
</dependency>
I look at the code for Utils.java (https://github.com/grpc/grpc-java/blame/master/netty/src/main/java/io/grpc/netty/Utils.java), and I don't see any evidence that I'd be using any earlier version that might not have this constant (it's been there since version 1.7).
I'm completely baffled what the issue is here. How do I identify the dependency conflict? Is there another way I can find what version of the class Maven is actually looking at here?
I am trying to transfer a gradle-based build plugin for android to a maven build system.
Up to now I was successful except with the android part.
It seems that I am missing classes from the package com.android.build.api.transform. Although I used this code
<dependency>
<groupId>com.android.tools.build</groupId>
<artifactId>gradle</artifactId>
<version>2.2.0</version>
<type>jar</type>
</dependency>
which is supposed to provide (rather indirectly) the required classes/packages, it seems that this is not working.
I also added
<repositories>
<repository>
<id>android</id>
<name>android</name>
<url>https://plugins.gradle.org/m2/</url>
</repository>
</repositories>
just in case, with no luck. Still the project is not able to compile due to missing classes.
Any idea what I am missing? I am very new to the gradle/maven scene, and I still feel out of my waters with these tools.
For reference here is a link to the javadoc of this artifact.
Thanks for your help
The 'gradle' artifact does not contain the missing package. As you mention, it depends transitively on at least one artifact that contains that package, but only in scope 'runtime'. Hence, it will not be available to your project on the compile classpath.
Anyway, adding the dependency explicitly in your pom.xml is the right thing to do. A search for the missing package on search.maven.org shows that you have the choice between com.android.tools.build:transform-api and com.android.tools.build:gradle-api. Locking at the transform-api artifact reveals that it's deprecated and encourages to use gradle-api instead. (The latest version is '2.0.0-deprecated-use-gradle-api'). I downloaded the gradle-api jar file, which indeed has the missing package. So please try adding that artifact to your pom.xml instead:
<dependency>
<groupId>com.android.tools.build</groupId>
<artifactId>gradle-api</artifactId>
<version>2.3.0</version>
</dependency>
I didn't check version 2.2.0, but if you for some reason want to use the older version, it will probably work too.
the transform API was a Google's Android Gradle plugin technology. but it was absorbed into and expanded by Gradle. so newer Android plugins migrated to the Gradle API and they no longer include the transform engine, which now is part of Gradle. if you use the deprecated transform API from Google that was bundled with older plugins, you will get the engine. but what you plan to do about all the other dependencies on Gradle, i have no idea.
I'm trying the Storm analysis presents here
CallLogCounterBolt.java:4: error: package backtype.storm.tuple does not exist
import backtype.storm.tuple.Fields;
I ran into similar problems with another old Apache Storm tutorial. It turned out to simply be because of the tutorial using deprecated classes from previous versions (0.9.6), while I was using newer ones (1.1.0). Therefore my suggestion is to either look through the newer libraries for corresponding resources in those and changing your library load statements accordingly, or checking that the dependencies that you are using are not masked by similarly named libraries.
The issue is with your Java classpath... which entirely depends on how you have setup your project. Rather than try to fix what you have I'll give you a suggestion.
If you're using Java, then the "normal" way to create storm topologies is using Maven which should work with whatever IDE you're using (Eclipse, Intellij, etc.).
Once you have a skeleton maven project setup, all you need to do is add the storm dependencies. For example:
<dependency>
<groupId>org.apache.storm</groupId>
<artifactId>storm-core</artifactId>
<version>${storm.version}</version>
<scope>provided</scope>
</dependency>
Here is an example POM file.
You should use newer Libraries in order to execute Since backtype is deprecated, Go through the Apache Storm javadocs Apache Storm javadocs
I'm having a slight problem with incorporating the appropriate maven dependencies into my project for sling models.
When I deploy my bundle, I get the following import that can't be resolved:
org.apache.sling.models.annotations,version=[1.1,2) -- Cannot be resolved
I believe I have included this with the following dependencies:
<dependency>
<groupId>org.apache.sling</groupId>
<artifactId>org.apache.sling.models.api</artifactId>
</dependency>
<dependency>
<groupId>org.apache.sling</groupId>
<artifactId>org.apache.sling.models.impl</artifactId>
</dependency>
I tried using:
<Embed-Transitive>true</Embed-Transitive>
<Import-Package>*</Import-Package>
in my bundle compile instructions, but this has just resulted in a ton of other dependencies not being resolved.
Surely I've gone down the garden path here somewhere. Any help would be greatly appreciated.
The org.apache.sling.models.api V1.0.2 bundle does export the following packages:
javax.inject,version=0.0.0
org.apache.sling.models.annotations,version=1.1.0
org.apache.sling.models.annotations.injectorspecific,version=1.0.0
org.apache.sling.models.spi,version=1.0.0
org.apache.sling.models.spi.injectorspecific,version=1.0.0
So if that bundle is active in your Sling instance, the org.apache.sling.models.annotations,version=[1.1,2) import should resolve.
Note that adding bundles to your maven dependencies might not be sufficient to install them in the running instance, what matters is whether the models.api bundle is active as seen from /system/console/bundles
Using Embed-Transitive is almost always a terrible idea. It traverses the entire transitive dependency hierarchy in Maven and pulls each one of those JARs into your own JAR. As a result you inherit all the package dependencies of all that crap you have dragged in.
When you have a bundle such as yours that requires an import -- in this case org.apache.sling.models.annotations -- the best solution is to find another bundle already available that exports the same package.
I have a Maven project that just displays a graph on the xy axis. I want to change that graph to a Javafx 2.0 linechart to display the same data. I tried using the FEST-javafx-maven plugin, but I still cannot compile the code; the compiler cannot find all of the javafx.xxx imports.
Any help would be appreciated.
Update (Oct 6 2015)
Modern JavaFX versions (shipped with Oracle Java 8+) do not require any additional class path to use JavaFX. The JavaFX runtime is on the default classpath Java uses for compilation and execution. This means that a plain maven pom.xml file, with no additional dependencies, will build a JavaFX application.
However, if you wish to use additional packaging features for your application, such as the ability to deploy it as a self-contained application, then I advise using the (third party) JavaFX Maven plugin.
Previous Answer
The following information in this answer is now mostly old and outdated.
The link to the fest maven plugin I found (http://fest.easytesting.org/javafx/maven/) is to a tool for building JavaFX 1.x script projects, which is a completely different and incompatible beast to JavaFX 2.0 - I'm not sure if there is an updated version of the fest maven plugin which supports JavaFX 2.0.
There is currently no official support for Maven from Oracle, nor a version of JavaFX 2.0 in a publicly hosted Maven repository.
However, I have successfully built JavaFX 2.0 projects using maven in the past by using a system scoped dependency on the jfxrt.jar and (optionally) invoking the JavaFX ant tasks from maven.
If you are embedding your graph in an existing Swing application via a JFXPanel, then you don't need to use the JavaFX ant tasks. Add jfxrt.jar from the JavaFX runtime as a system dependency OR manually install it into your maven repository to use a non-system scoped dependency.
An example of the command to manually install the required JavaFX 2.0 runtime jar is:
mvn install:install-file -Dfile="C:\Program Files\Oracle\JavaFX 2.1.0 SDK\rt\lib\jfxrt.jar" -DgroupId=com.oracle.javafx -DartifactId=javafx -Dversion=2.1 -Dpackaging=jar
After running the above command, add the dependency to the jfxrt.jar file to your maven pom and your project compilation should resolve all JavaFX API references:
<dependency>
<groupId>com.oracle.javafx</groupId>
<artifactId>javafx</artifactId>
<version>2.1</version>
</dependency>
If you extend the JavaFX Application class and you want your application packaged for deployment via webstart, browser embedding or as a JavaFX installation aware clickable jar, then you should adapt your pom.xml file to execute the relevant JavaFX 2.0 ant tasks - http://code.google.com/p/willow-browser/source/browse/pom.xml demonstrates such an adaption.
These threads discuss JavaFX 2.0 maven support and provide further background info and samples:
https://forums.oracle.com/forums/thread.jspa?messageID=9970002
http://mail.openjdk.java.net/pipermail/openjfx-dev/2011-December/000076.html
I had the same problem and here is my solution:
If using Java 7u7 (javafx is integrated into jdk/jre):
<dependency>
<groupId>javafx</groupId>
<artifactId>javafx</artifactId>
<version>2.1</version>
<scope>system</scope>
<systemPath>${java.home}/lib/jfxrt.jar</systemPath>
</dependency>
For previous versions of java:
<dependency>
<groupId>javafx</groupId>
<artifactId>javafx</artifactId>
<version>2.1</version>
<scope>system</scope>
<systemPath>${env.JAVAFX_HOME}\rt\lib\jfxrt.jar</systemPath>
</dependency>
And you have to set system variable JAVAFX_HOME to home dir of JavaFx.
You might try the JavaFX Maven plugin. This takes care of adding javafx to the classpath as well as building JavaFX apps and creating Windows/Mac/Linux executables, as well as double-clickable JARs and JNLP files.
Using JDK 1.7 you have to perform the following mvn goal.
mvn com.zenjava:javafx-maven-plugin:2.0:fix-classpath
This command will change the classpath of your JRE and copy the jfxrt.jar to the JAVA_HOME\lib\ext directory.
Take an additional look here for further information:
JavaFX Maven Plugin
Within your pom-file you have to add the following dependency configuration.
<dependency>
<groupId>javafx</groupId>
<artifactId>jfxrt</artifactId>
<version>2.2</version>
<!-- <version>${javafx.min.version}</version> -->
<scope>system</scope>
<systemPath>${java.home}/lib/ext/jfxrt.jar</systemPath>
</dependency>