Maven shade plugin warning: we have a duplicate - how to fix? - maven

Question

This is my project POM (link to the paste, so you can right click > save as pom.xml)
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.zybnet</groupId>
<artifactId>excel-reporter</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>mvn1</name>
<url>http://maven.apache.org</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.17</version>
</dependency>
<dependency>
<groupId>org.apache.poi</groupId>
<artifactId>poi</artifactId>
<version>3.8</version>
</dependency>
<dependency>
<groupId>net.sf.jasperreports</groupId>
<artifactId>jasperreports</artifactId>
<version>4.6.0</version>
</dependency>
<dependency>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy-all</artifactId>
<version>2.0.0</version>
</dependency>
</dependencies>
<build>
<resources>
<resource>
<directory>${project.build.sourceDirectory}</directory>
</resource>
</resources>
<plugins>
<plugin>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<finalName>${artifactId}-${version}-tmp</finalName>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>1.7.1</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>com.zybnet.Main</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
I followed the advice of configuring the default jar plugin as advertised in the FAQ, but still when I run mvn package about 20K warnings are issued. Running mvn clean does not help either.
According to this answer, I could manually exclude some dependencies. However, I don't know if it's the right way, and the dependency tree is rather complex so it's difficult to argue where to start.
I know that these issues are not harmful, but I'm used to treat warnings as something that must be fixed. Moreover, I'm a beginner with Maven, so I'd like to understand what's wrong with my understanding, and how to troubleshoot problems.
(Using maven assembly plugin is not an option here)

Answer 1

Sometimes it happens that the same class definition is found in two or more JAR files (usually these are dependency JARs). In such a case, there is nothing the developer can do except trying to figure out what to manually exclude from the dependencies or from the final artifact (maybe with the help of mvn dependency:tree -Ddetail=true). I opened an issue and submitted a patch, to help the developer with a slightly prettier output like
[WARNING] xml-apis-1.3.02.jar, xmlbeans-2.3.0.jar define 4 overlappping classes:
[WARNING] - org.w3c.dom.TypeInfo
[WARNING] - org.w3c.dom.DOMConfiguration
[WARNING] - org.w3c.dom.DOMStringList
[WARNING] - org.w3c.dom.UserDataHandler
[WARNING] maven-shade-plugin has detected that some .class files
[WARNING] are present in two or more JARs. When this happens, only
[WARNING] one single version of the class is copied in the uberjar.
[WARNING] Usually this is not harmful and you can skip these
[WARNING] warnings, otherwise try to manually exclude artifacts
[WARNING] based on mvn dependency:tree -Ddetail=true and the above
[WARNING] output
[WARNING] See http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin
Using this output and that from mvn dependency:tree I added sections like
<dependency>
<groupId>net.sf.jasperreports</groupId>
<artifactId>jasperreports</artifactId>
<version>4.7.0</version>
<exclusions>
<exclusion>
<groupId>org.apache.poi</groupId>
<artifactId>poi-ooxml</artifactId>
</exclusion>
<exclusion>
<groupId>bouncycastle</groupId>
<artifactId>bcmail-jdk14</artifactId>
</exclusion>
<exclusion>
<groupId>bouncycastle</groupId>
<artifactId>bcprov-jdk14</artifactId>
</exclusion>
<exclusion>
<groupId>org.bouncycastle</groupId>
<artifactId>bctsp-jdk14</artifactId>
</exclusion>
</exclusions>
</dependency>
and managed to reduce the number of warnings from several thousands to a couple dozens. Still, this is not perfect. Still, they continue to copypaste classes that lead to name clashes (I can't understand why). Still, this solution is specific to this particular project and can't be easily ported on anything else.

Answer 2

As an newer update to this problem is updating to the current maven plugin.
The question is using
<version>1.7.1</version>
which will only tell you that you:
We have a duplicate org/eclipse/persistence/internal/libraries/asm/AnnotationVisitor.class in /Users/.../repository/org/eclipse/persistence/eclipselink/2.7.0/eclipselink-2.7.0.jar
Leaving you to guess or trial and error where the first entry where you have overlapping classes. Not very useful. Fortunatly,
<version>3.1.0</version>
will give you the more usefull output that includes the jars where they came from:
e.g.
annotations-3.0.1.jar, jcip-annotations-1.0.jar define 4 overlapping classes:
- net.jcip.annotations.GuardedBy
- net.jcip.annotations.NotThreadSafe
...
You can then decide to exclude one or the other, or use shade to rename the classes as appropriate to your project.

Answer 3

I have updated to version 3.4.1, and it works now.
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.4.1</version>