Missing "Artifacts" from uber jar using IntelliJ / Maven - maven

I am developing a Flink application, and I'm very new to building Java applications.
I am using IntelliJ 2022.2.3 Community Edition, and Maven for dependency management.
I have the following dependencies in my POM file:
<!-- https://mvnrepository.com/artifact/com.amazonaws/amazon-sqs-java-messaging-lib -->
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>amazon-sqs-java-messaging-lib</artifactId>
<version>2.0.1</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.postgresql/postgresql -->
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<version>42.5.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/com.amazonaws/aws-kinesisanalytics-runtime -->
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-kinesisanalytics-runtime</artifactId>
<version>1.2.0</version>
</dependency>
When I build the artifact and view its contents, I notice that some of the dependancies are included, and others are missing. I expect to see the postgressql drivers at org/postgressql/... but that folder does not exist.
I have a copy of the project where the artifacts folder does contain the expected folders and when I look at the project settings/artifacts/output layout view, the postgres jars are in the list, but not in my problem project?
I read How can I create an executable/runnable JAR with dependencies using Maven? and i don't have that section in the POM, but in my case as I mentioned the 2 projects I have seem to have different artifacts missing from the jar?
Sorry for my lack of correct terminology.
UPDATE:
I should add this section is in my POM
<!-- We use the maven-shade plugin to create a fat jar that contains all necessary dependencies. -->
<!-- Change the value of <mainClass>...</mainClass> if your program entry point changes. -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.1.1</version>
<executions>
<!-- Run shade goal on package phase -->
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<artifactSet>
<excludes>
<exclude>org.apache.flink:flink-shaded-force-shading</exclude>
<exclude>com.google.code.findbugs:jsr305</exclude>
<exclude>org.slf4j:*</exclude>
<exclude>org.apache.logging.log4j:*</exclude>
</excludes>
</artifactSet>
<filters>
<filter>
<!-- Do not copy the signatures in the META-INF folder.
Otherwise, this might cause SecurityExceptions when using the JAR. -->
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>vendor.flink.StreamProcessingNoJoin</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>

Related

maven Error creating shaded jar: error in opening zip file

I am trying to build my dropwizard project using the following command :
mvn package
I am getting the following error :
Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:2.3:shade (default) on project rest-api: Error creating shaded jar: error in opening zip file /Users/ldelaney/.m2/repository/mysql/mysql-connector-java/5.1.36/mysql-connector-java-5.1.36.jar
I have checked, the jar is there, right where maven is looking.
I have also tried blowing it away, and running :
mvn clean
mvn package
But the error just will not go away. My IDE is not showing me any errors.
Also, here is my dependency in the POM :
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.36</version>
</dependency>
Here is the entire POM file.
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.company.test</groupId>
<artifactId>rest-api</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
<dropwizard.version>1.0.5</dropwizard.version>
<mongodriver.version>3.4.2</mongodriver.version>
<mysqlConnector.version>5.1.36</mysqlConnector.version>
<log4j.version>1.2.17</log4j.version>
</properties>
<dependencies>
<!-- Drop Wizard -->
<dependency>
<groupId>io.dropwizard</groupId>
<artifactId>dropwizard-core</artifactId>
<version>${dropwizard.version}</version>
</dependency>
<dependency>
<groupId>io.dropwizard</groupId>
<artifactId>dropwizard-auth</artifactId>
<version>${dropwizard.version}</version>
</dependency>
<dependency>
<groupId>io.dropwizard</groupId>
<artifactId>dropwizard-db</artifactId>
<version>${dropwizard.version}</version>
</dependency>
<dependency>
<groupId>io.dropwizard</groupId>
<artifactId>dropwizard-hibernate</artifactId>
<version>${dropwizard.version}</version>
</dependency>
<!-- MYSQL and Hibernate -->
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>${mysqlConnector.version}</version>
</dependency>
<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
<version>1.4.193</version>
</dependency>
<!-- Log4j dependencies -->
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>${log4j.version}</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<configuration>
<createDependencyReducedPom>true</createDependencyReducedPom>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>application.ServerApplication</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.7</source>
<target>1.7</target>
</configuration>
</plugin>
</plugins>
</build>
Anyone have any ideas?
From the OP response, it looks like in .m2/repository/mysql-connector-java/5.1.36 there is only pom.xml file. Please do the following,
if you are using either nexus or artifactory repositories in your organization. delete mysql-connector coordinates and corresponding pom.xml file.
Ensure that your firewall is configured such that you can download file from here. Central repository is by default configured with maven installation.
Upgrade to maven 3 if you are on maven 2.
I just copied your pom and i was able to successfully build. I have no artifactory or nexus as well. Could you try following,
Go to your maven installation and specify a different repository location. You should go to /conf/settings.xml and uncomment and change <localRepository>/path/to/local/repo</localRepository> to a path. Make sure you do chmod +777 to that path.
If the problem persists, then manually download mysql-connector-java-5.1.36.jar and place it there. See if it works.
if 2 does not work, then try step 2 again but run mvn with -o flag. This is offline flag and maven will not download dependency.
If it does not work, can you show your maven output with -X flag,
mvn -U -X -e install
Is had this issue: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:2.1:shade (default) on project xxx: Error creating shaded jar: error in opening zip file /Users/username/.m2/repository/org/codehaus/groovy/groovy/2.2.0/groovy-2.2.0.jar -> [Help 1]
Resolution was rm -rf /Users/username/.m2/repository/org/codehaus/groovy/groovy/2.2.0 then it redownloaded it and started working again. If that helps.

How to build a project with AspectJ plugin which also has dependencies on querydsl plugin

I have Aspects defined in project A which can be referred from project B without any compilations issues because project A has project B as dependency. However project B needs AspectJ plugin so that aspects can be weaved via compilation/build.
Issue : My project B is using com.mysema.querydsl plugin to generate Q files for database entities. When I compile using AspectJ in eclipse (command clean aspectj:compile install) it does NOT auto generate these Q files and thus the compilation fails and weaving is not processed, overall build fails.
I have tried so many combinations of adding this dependency in AspectJ plugin but nothing works.
Please refer the pom part below:
<!-- AspectJ Maven Plugin -->
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>aspectj-maven-plugin</artifactId>
<version>1.5</version>
<dependencies>
<dependency>
<groupId>com.mysema.querydsl</groupId>
<artifactId>querydsl-core</artifactId>
<version>3.6.0</version>
<scope>compile</scope>
</dependency>
</dependencies>
<configuration>
<source>${java.version}</source>
<target>${java.version}</target>
<complianceLevel>1.7</complianceLevel>
<!-- <aspectLibraries>
<aspectLibrary>
<groupId>projectA.groupId</groupId>
<artifactId>projectA.artifactId</artifactId>
</aspectLibrary>
</aspectLibraries> -->
</configuration>
<executions>
<execution>
<!-- <phase>generate-sources</phase> -->
<goals>
<goal>compile</goal>
</goals>
</execution>
</executions>
</plugin>
<!-- AspectJ Maven Plugin -->
What should be done here? I really dont want to create individual aspects in each project. All in all how to build projects with aspectj plugins which have querydsl plugin code references?

Maven shade plugin with Kryo 3.0: ClassNotFoundException

I'm fairly new to Maven and I want to update a project from Kryo 2.22 to 3.0.1. In the project I've got the following dependency.
<dependency>
<groupId>com.esotericsoftware.kryo</groupId>
<artifactId>kryo</artifactId>
<version>2.22</version>
</dependency>
The project uses the Maven shade plugin to create a jar file:
<plugin>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<executions>
<execution>
<id>shade-gremlin-groovy</id>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<createDependencyReducedPom>false</createDependencyReducedPom>
<artifactSet>
<includes>
<!-- As of Kryo 2.22, this artifact includes both minlog and reflectasm -->
<!-- If we upgrade to later a Kryo version, we may have to add includes
for minlog and reflectasm (this is true of 2.24.0, not sure about 3) -->
<include>com.esotericsoftware.kryo:*</include>
</includes>
</artifactSet>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>reflectasm-1.07-shaded.jar</exclude>
<exclude>minlog-1.2.jar</exclude>
<exclude>objenesis-1.2.jar</exclude>
<exclude>META-INF/MANIFEST.MF</exclude>
</excludes>
</filter>
</filters>
<relocations>
<relocation>
<pattern>com.esotericsoftware.kryo</pattern>
<shadedPattern>com.thinkaurelius.shaded.kryo_2_22</shadedPattern>
</relocation>
<relocation>
<pattern>com.esotericsoftware.minlog</pattern>
<shadedPattern>com.thinkaurelius.shaded.minlog_1_2</shadedPattern>
</relocation>
<relocation>
<pattern>com.esotericsoftware.reflectasm</pattern>
<shadedPattern>com.thinkaurelius.shaded.reflectasm_1_07</shadedPattern>
</relocation>
<relocation>
<pattern>com.esotericsoftware.shaded.org.objenesis</pattern>
<shadedPattern>com.thinkaurelius.shaded.objenesis_1_2</shadedPattern>
</relocation>
</relocations>
<!-- false below means the shade plugin overwrites the main project artifact (the one with no classifier).
false does *not* actually detach the main artifact, despite what the option name suggests. -->
<shadedArtifactAttached>false</shadedArtifactAttached>
<minimizeJar>false</minimizeJar>
</configuration>
</execution>
</executions>
</plugin>
However I cannot figure out what I should change except the groupId to: <groupId>com.esotericsoftware.kryo</groupId>
If I change the version number to 3.0.1 it cannot find the shaded classes (throws ClassNotfound exception:
java.lang.NoClassDefFoundError: com/thinkaurelius/shaded/kryo_3_0_1/kryo/Serializer
at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at com.thinkaurelius.titan.graphdb.database.serialize.kryo.KryoSerializer.<init>(KryoSerializer.java:71)
at com.thinkaurelius.titan.graphdb.database.serialize.StandardSerializer.<init>(StandardSerializer.java:29)
at com.thinkaurelius.titan.graphdb.database.serialize.StandardSerializer.<init>(StandardSerializer.java:41)
at com.thinkaurelius.titan.diskstorage.configuration.backend.KCVSConfiguration.<init>(KCVSConfiguration.java:69)
at com.thinkaurelius.titan.diskstorage.configuration.backend.KCVSConfiguration.<init>(KCVSConfiguration.java:57)
at com.thinkaurelius.titan.diskstorage.configuration.KCVSConfigTest.getConfig(KCVSConfigTest.java:31)
I think the problem is that Kryo 2.22 has a different groupId (com.esotericsoftware.kryo) than Kryo 3.0.1 and that Maven cannot connect the dependency. How do I resolve this?
It appeared that the following line in the maven shade plugin:
<include>com.esotericsoftware.kryo:*</include>
had to be changed to:
<include>com.esotericsoftware:*</include>
Since the groupId is different. So the problem was a mismatch of a dependency.
From the maven repo search you linked to, it looks as though the dependency is in the maven central repository. However, the groupId has changed between 2.2.x and 3.x.
Have you tried with:
<dependency>
<groupId>com.esotericsoftware</groupId>
<artifactId>kryo</artifactId>
<version>3.0.1</version>
</dependency>
i.e. the same as you have, but without the trailing .kryo

Linkage failure when running Apache Flink jobs

I have a job developed in Flink 0.9 that is using the graph module (Gelly). The job is running successfully within the IDE (Eclipse) but after exporting it to a JAR using maven (mvn clean install) it fails to execute on the local flink instance with the following error
"The program's entry point class 'myclass' could not be loaded due to a linkage failure"
java.lang.NoClassDefFoundError: org/apache/flink/graph/GraphAlgorithm
Any idea why is this happening and how to solve it?
It looks like the code of flink-gelly did not end up in your jar file.
The most obvious reason for this issue is the missing maven dependency in your project's pom file. But I assume the dependency is present, otherwise developing the job in the IDE would be impossible.
Most likely, the jar file has been created by the maven-jar-plugin, which is not including dependencies.
Try adding the following fragment to your pom.xml:
<build>
<plugins>
<!-- We use the maven-shade plugin to create a fat jar that contains all dependencies
except flink and it's transitive dependencies. The resulting fat-jar can be executed
on a cluster. Change the value of Program-Class if your program entry point changes. -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<executions>
<!-- Run shade goal on package phase -->
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<filters>
<filter>
<artifact>org.apache.flink:*</artifact>
<excludes>
<exclude>org/apache/flink/shaded/**</exclude>
<exclude>web-docs/**</exclude>
</excludes>
</filter>
</filters>
<transformers>
<!-- add Main-Class to manifest file -->
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>YOURMAINCLASS</mainClass>
</transformer>
</transformers>
<createDependencyReducedPom>false</createDependencyReducedPom>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<profiles>
<profile>
<!-- A profile that does everyting correctly:
We set the Flink dependencies to provided -->
<id>build-jar</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-java</artifactId>
<version>0.9-SNAPSHOT</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-core</artifactId>
<version>0.9-SNAPSHOT</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-clients</artifactId>
<version>0.9-SNAPSHOT</version>
<scope>provided</scope>
</dependency>
</dependencies>
</profile>
</profiles>
Now, you can build the jar using mvn clean package -Pbuild-jar.
The jar file will now be located in the target/ directory.
You can manually check whether the jar (zip) file contains class files in /org/apache/flink/graph/

how to exclude GWT dependency code from OSGI bundle generated by MAven+BND?

I have several Maven modules with Vaadin library dependency in the root pom.xml file.
I'm trying to build a set of OSGI bundles (1 per Maven module) using Maven+BND.
I added this to my "root" pom.xml file:
<dependencies>
<dependency>
<groupId>com.vaadin</groupId>
<artifactId>vaadin</artifactId>
<version>6.6.6</version>
</dependency>
<dependency>
<groupId>com.google.gwt</groupId>
<artifactId>gwt-user</artifactId>
<version>2.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.felix</groupId>
<artifactId>org.osgi.core</artifactId>
<version>1.0.0</version>
</dependency>
</dependencies>
unfortunately, the resulting JAR files (bundles) include GWT (com.google.gwt) classes. This
1) makes the bundles huge, with lots of duplicated dependencies.
2) generated thousands of build warnings about "split packages".
QUESTION: how to prevent adding GWT classes into my Jar files?
I tried setting "scope" of GWT to "provided", setting "type" to "bundle", and even optional=true - didn't help.
here's the part of my root pom.xml, which is responsible for Vaadin/GWT stuff:
<plugins>
<plugin>
<groupId>org.apache.felix</groupId>
<artifactId>maven-bundle-plugin</artifactId>
<version>2.3.5</version>
<extensions>true</extensions>
<configuration>
<instructions>
<Export-Package>mycompany.*</Export-Package>
<Private-Package>*.impl.*</Private-Package>
<Bundle-SymbolicName>${project.artifactId}</Bundle-SymbolicName>
<!-- <Bundle-Activator>com.alskor.publicpackage.MyActivator</Bundle-Activator>-->
</instructions>
</configuration>
</plugin>
<!-- Compiles your custom GWT components with the GWT compiler -->
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>gwt-maven-plugin</artifactId>
<!-- Version 2.1.0-1 works at least with Vaadin 6.5 -->
<version>2.3.0-1</version>
<configuration>
<!-- if you don't specify any modules, the plugin will find them -->
<!--modules>
..
</modules-->
<webappDirectory>${project.build.directory}/${project.build.finalName}/VAADIN/widgetsets
</webappDirectory>
<extraJvmArgs>-Xmx512M -Xss1024k</extraJvmArgs>
<runTarget>clean</runTarget>
<hostedWebapp>${project.build.directory}/${project.build.finalName}</hostedWebapp>
<noServer>true</noServer>
<port>8080</port>
</configuration>
<executions>
<execution>
<goals>
<goal>resources</goal>
<goal>compile</goal>
</goals>
</execution>
</executions>
</plugin>
<!-- Updates Vaadin 6.2+ widgetset definitions based on project dependencies -->
<plugin>
<groupId>com.vaadin</groupId>
<artifactId>vaadin-maven-plugin</artifactId>
<version>1.0.1</version>
<executions>
<execution>
<configuration>
<!-- if you don't specify any modules, the plugin will find them -->
<!--
<modules>
<module>${package}.gwt.MyWidgetSet</module>
</modules>
-->
</configuration>
<goals>
<goal>update-widgetset</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
The wildcards in your Export-Package and Private-Package statements strike me as exceedingly dangerous. It's possible that the GWT packages are being dragged in because of the *.impl.* pattern in Private-Package.
Also you should never use wildcards in Export-Package: exports should be tightly controlled and versioned.
use mvn dependency:tree to see where the gwt dependency comes from
Add an <excludes/> element with an appropriate <exclude/> to the dependency in question to suppress it.
I've had similar problem, as final war file exceeded almost 90MB !
One of the culprit was aforementioned jar, so I did this :
<dependencies>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>widgetset</artifactId>
<version>3.2</version>
<exclusions>
<exclusion>
<groupId>com.vaadin.external.gwt</groupId>
<artifactId>gwt-user</artifactId>
</exclusion>
</exclusions>
</dependency>
...
</dependencies>

Resources