Maven shade plugin with Kryo 3.0: ClassNotFoundException - maven

I'm fairly new to Maven and I want to update a project from Kryo 2.22 to 3.0.1. In the project I've got the following dependency.
<dependency>
<groupId>com.esotericsoftware.kryo</groupId>
<artifactId>kryo</artifactId>
<version>2.22</version>
</dependency>
The project uses the Maven shade plugin to create a jar file:
<plugin>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<executions>
<execution>
<id>shade-gremlin-groovy</id>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<createDependencyReducedPom>false</createDependencyReducedPom>
<artifactSet>
<includes>
<!-- As of Kryo 2.22, this artifact includes both minlog and reflectasm -->
<!-- If we upgrade to later a Kryo version, we may have to add includes
for minlog and reflectasm (this is true of 2.24.0, not sure about 3) -->
<include>com.esotericsoftware.kryo:*</include>
</includes>
</artifactSet>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>reflectasm-1.07-shaded.jar</exclude>
<exclude>minlog-1.2.jar</exclude>
<exclude>objenesis-1.2.jar</exclude>
<exclude>META-INF/MANIFEST.MF</exclude>
</excludes>
</filter>
</filters>
<relocations>
<relocation>
<pattern>com.esotericsoftware.kryo</pattern>
<shadedPattern>com.thinkaurelius.shaded.kryo_2_22</shadedPattern>
</relocation>
<relocation>
<pattern>com.esotericsoftware.minlog</pattern>
<shadedPattern>com.thinkaurelius.shaded.minlog_1_2</shadedPattern>
</relocation>
<relocation>
<pattern>com.esotericsoftware.reflectasm</pattern>
<shadedPattern>com.thinkaurelius.shaded.reflectasm_1_07</shadedPattern>
</relocation>
<relocation>
<pattern>com.esotericsoftware.shaded.org.objenesis</pattern>
<shadedPattern>com.thinkaurelius.shaded.objenesis_1_2</shadedPattern>
</relocation>
</relocations>
<!-- false below means the shade plugin overwrites the main project artifact (the one with no classifier).
false does *not* actually detach the main artifact, despite what the option name suggests. -->
<shadedArtifactAttached>false</shadedArtifactAttached>
<minimizeJar>false</minimizeJar>
</configuration>
</execution>
</executions>
</plugin>
However I cannot figure out what I should change except the groupId to: <groupId>com.esotericsoftware.kryo</groupId>
If I change the version number to 3.0.1 it cannot find the shaded classes (throws ClassNotfound exception:
java.lang.NoClassDefFoundError: com/thinkaurelius/shaded/kryo_3_0_1/kryo/Serializer
at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at com.thinkaurelius.titan.graphdb.database.serialize.kryo.KryoSerializer.<init>(KryoSerializer.java:71)
at com.thinkaurelius.titan.graphdb.database.serialize.StandardSerializer.<init>(StandardSerializer.java:29)
at com.thinkaurelius.titan.graphdb.database.serialize.StandardSerializer.<init>(StandardSerializer.java:41)
at com.thinkaurelius.titan.diskstorage.configuration.backend.KCVSConfiguration.<init>(KCVSConfiguration.java:69)
at com.thinkaurelius.titan.diskstorage.configuration.backend.KCVSConfiguration.<init>(KCVSConfiguration.java:57)
at com.thinkaurelius.titan.diskstorage.configuration.KCVSConfigTest.getConfig(KCVSConfigTest.java:31)
I think the problem is that Kryo 2.22 has a different groupId (com.esotericsoftware.kryo) than Kryo 3.0.1 and that Maven cannot connect the dependency. How do I resolve this?

It appeared that the following line in the maven shade plugin:
<include>com.esotericsoftware.kryo:*</include>
had to be changed to:
<include>com.esotericsoftware:*</include>
Since the groupId is different. So the problem was a mismatch of a dependency.

From the maven repo search you linked to, it looks as though the dependency is in the maven central repository. However, the groupId has changed between 2.2.x and 3.x.
Have you tried with:
<dependency>
<groupId>com.esotericsoftware</groupId>
<artifactId>kryo</artifactId>
<version>3.0.1</version>
</dependency>
i.e. the same as you have, but without the trailing .kryo

Related

Missing "Artifacts" from uber jar using IntelliJ / Maven

I am developing a Flink application, and I'm very new to building Java applications.
I am using IntelliJ 2022.2.3 Community Edition, and Maven for dependency management.
I have the following dependencies in my POM file:
<!-- https://mvnrepository.com/artifact/com.amazonaws/amazon-sqs-java-messaging-lib -->
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>amazon-sqs-java-messaging-lib</artifactId>
<version>2.0.1</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.postgresql/postgresql -->
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<version>42.5.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/com.amazonaws/aws-kinesisanalytics-runtime -->
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-kinesisanalytics-runtime</artifactId>
<version>1.2.0</version>
</dependency>
When I build the artifact and view its contents, I notice that some of the dependancies are included, and others are missing. I expect to see the postgressql drivers at org/postgressql/... but that folder does not exist.
I have a copy of the project where the artifacts folder does contain the expected folders and when I look at the project settings/artifacts/output layout view, the postgres jars are in the list, but not in my problem project?
I read How can I create an executable/runnable JAR with dependencies using Maven? and i don't have that section in the POM, but in my case as I mentioned the 2 projects I have seem to have different artifacts missing from the jar?
Sorry for my lack of correct terminology.
UPDATE:
I should add this section is in my POM
<!-- We use the maven-shade plugin to create a fat jar that contains all necessary dependencies. -->
<!-- Change the value of <mainClass>...</mainClass> if your program entry point changes. -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.1.1</version>
<executions>
<!-- Run shade goal on package phase -->
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<artifactSet>
<excludes>
<exclude>org.apache.flink:flink-shaded-force-shading</exclude>
<exclude>com.google.code.findbugs:jsr305</exclude>
<exclude>org.slf4j:*</exclude>
<exclude>org.apache.logging.log4j:*</exclude>
</excludes>
</artifactSet>
<filters>
<filter>
<!-- Do not copy the signatures in the META-INF folder.
Otherwise, this might cause SecurityExceptions when using the JAR. -->
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>vendor.flink.StreamProcessingNoJoin</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>

Can I bundle import-control file for checkstyle for use in all projects using our checkstyle?

I am trying to add import-control to our checkstyle in such a way that the import-control file exists in the project making the checstyle.xml file and not in the projects we build later on.
We have a specific gradle project where we define all our rules and it is in this project our import-control.xml. My issue is that when I try to run mvn clean install on another project that uses this checkstyle it tries to locate import-control.xml in that project.
I did the following configuration in the checkstyle.xml:
<module name="ImportControl">
<property name="file" value="import-control.xml"/>
</module>
and the import-control.xml is placed next to checkstyle.xml.
Can anyone tell me what I need to do so that I can tell maven that this file exists in our checkstyle project and not in the root project that is being built?
Errors I have gotten are:
Cannot initialize module TreeWalker - cannot initialize module ImportControl - illegal value 'import-control.xml' for property 'file' Unable to find: import-control.xml
In v 2.17
Unable to load import-control.xml: unable to find file:/C://import-control.xml: \import-control.xml
What I have tried:
Upgrade checkstyle version to 3.1.0 (we used to have 2.17)
Use import-control.xml but didn't work.
Tried to read documentation and code but to no help.
Thanks for any help
Write you later / Mårten
mvn configuration:
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
<version>3.1.0</version>
<executions>
<execution>
<id>do checkstyle</id>
<phase>process-sources</phase>
<goals>
<goal>check</goal>
</goals>
</execution>
</executions>
<configuration>
<includes>projectA/**/*</includes>
<configLocation>checkstyle.xml</configLocation>
<consoleOutput>true</consoleOutput>
<failOnViolation>false</failOnViolation>
<failsOnError>true</failsOnError>
<includeTestSourceDirectory>true</includeTestSourceDirectory>
</configuration>
<dependencies>
<dependency>
<groupId>company.checkstyle</groupId>
<artifactId>company-checkstyle</artifactId>
<version>0.2-SNAPSHOT</version>
</dependency>
</dependencies>
</plugin>
</plugins>
</pluginManagement>```
Thanks again barfuin, it seemed like ${config_loc} was the answer but we needed one more thing for it to fully work.
So, for adding resources from the checkstyle project, as in this file an import_control.xml I did as follow in my checkstyle.xml:
<module name="ImportControl">
<property name="file" value="${config_loc}/config/import_control.xml"/>
</module>
What I also needed to do was to add:
<propertyExpansion>config_loc=</propertyExpansion>
in my pom.xml configuration, this solved the issue with config_loc not being defined and for checkstyle to find the file as a resource and gave me the following pom.xml configuration:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
<executions>
<execution>
<id>do checkstyle</id>
<phase>process-sources</phase>
<goals>
<goal>check</goal>
</goals>
</execution>
</executions>
<configuration>
<includes>projectA/**/*</includes>
<configLocation>checkstyle.xml</configLocation>
<consoleOutput>true</consoleOutput>
<failOnViolation>false</failOnViolation>
<failsOnError>true</failsOnError>
<includeTestSourceDirectory>true</includeTestSourceDirectory>
<propertyExpansion>config_loc=</propertyExpansion>
</configuration>
<dependencies>
<dependency>
<groupId>company.checkstyle</groupId>
<artifactId>company-checkstyle</artifactId>
<version>0.2-SNAPSHOT</version>
</dependency>
</dependencies>
</plugin>
</plugins>

Which dependencies do I have to include in a Maven Apache Storm Project?

I am pretty new to Apache Storm and Maven projects, so I tried to follow this "tutorial" (which from my point of view is not a tutorial at all):
http://storm.apache.org/releases/current/Creating-a-new-Storm-project.html
There is a huge pom.xml referenced (https://github.com/apache/storm/blob/v1.1.1/examples/storm-starter/pom.xml) which shall be used as basis for a new project. I tried to figure out what to copy into my project pom.xml. So I decided to start with the apache core dependency first. My pom.xml looks like this:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>groupId</groupId>
<artifactId>StormTest</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.storm</groupId>
<artifactId>storm-core</artifactId>
<version>1.1.1</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>1.5.0</version>
<executions>
<execution>
<goals>
<goal>exec</goal>
</goals>
</execution>
</executions>
<configuration>
<executable>java</executable>
<includeProjectDependencies>true</includeProjectDependencies>
<includePluginDependencies>false</includePluginDependencies>
<classpathScope>compile</classpathScope>
<mainClass>${storm.topology}</mainClass>
<cleanupDaemonThreads>false</cleanupDaemonThreads>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.3</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>3.1.0</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<appendAssemblyId>false</appendAssemblyId>
<finalName>StormTest-1.0-SNAPSHOT_dep</finalName>
<archive>
<manifest>
<mainClass>de.arphi.bi.WordCountTopology</mainClass>
</manifest>
</archive>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
<resources>
<resource>
<directory>${basedir}/resources</directory>
<filtering>false</filtering>
<includes>
<include>log4j2.xml</include>
</includes>
</resource>
</resources>
</build>
</project>
This even works when it is about runing maven and building / packaging a jar. The outcome are two jar files (a small one without dependencies and a bigger one with dependencies). I cannot run the smaller one because it says "main manifest not found". But I can run the bigger one by executing the following command:
storm -jar StormTest-1.0-SNAPSHOT_dep.jar
Actually the runs on my locally installed apache storm 1.1.1 (I have some System.out.printlns ...) but I get an exception:
2018-01-02 21:38:31,864 main ERROR Unable to create file C:\Users\Artur\Desktop\Bi\apache-storm-1.1.1\logs/access-web-${sys:daemon.name}.log java.io.IOException: Die Syntax für den Dateinamen, Verzeichnisnamen oder die Datenträgerbezeichnung ist falsch
at java.io.WinNTFileSystem.canonicalizeWithPrefix0(Native Method)
at java.io.WinNTFileSystem.canonicalizeWithPrefix(WinNTFileSystem.java:451)
at java.io.WinNTFileSystem.canonicalize(WinNTFileSystem.java:422)
at java.io.File.getCanonicalPath(File.java:618)
at java.io.File.getCanonicalFile(File.java:643)
at org.apache.logging.log4j.core.util.FileUtils.makeParentDirs(FileUtils.java:134)
at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:573)
at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:554)
at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:112)
at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:114)
at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.getFileManager(RollingFileManager.java:155)
at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:131) at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:60)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:122)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:952)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:892)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:884)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:508) at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:232)
at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:244)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:545)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:617)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:634)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:229)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:152)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:122)
at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:43)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:46)
at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:29)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:358)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383)
at org.apache.storm.topology.BasicBoltExecutor.<clinit>(BasicBoltExecutor.java:28)
at org.apache.storm.topology.TopologyBuilder.setBolt(TopologyBuilder.java:215)
at de.arphi.bi.WordCountTopology.main(WordCountTopology.java:22)
It says something about the syntax for creating a directory is wrong. And I know that it is about logging. I played arroung with different other dependencies (log4j, slf4j) and tried ecen to exclude dependecies without any success. I cannot get rid of this error.
Any ideas? I think that I am missing a dependency or that I have to exclude some parts of my pom.xml. But since I am not an Maven expert it is really hard for me to figure out what I have to adapt here.
I agree that storm-starter has gotten pretty big, and we should maybe have more of a minimal example.
First you should set the storm-core dependency to "provided" scope. When you deploy the topology to Storm, your jar will use the storm-core jar present in the Storm installation, so you shouldn't also put it in your fat jar.
<dependency>
<groupId>org.apache.storm</groupId>
<artifactId>storm-core</artifactId>
<version>1.1.1</version>
<scope>provided</scope>
</dependency>
When your topology runs on Storm, it will use the Log4j2 configuration in the log4j2/worker.xml file in your Storm installation. You shouldn't include your own log4j2.xml. If you need to set specific log levels, you can either modify worker.xml or use the CLI as described at http://storm.apache.org/releases/1.1.1/dynamic-log-level-settings.html.
Other than that your pom looks fine. I don't know why you have exec-maven-plugin in there (Edit: I see it's also in storm-starter, I think it's a leftover from when it was possible to run storm-starter in local mode. You shouldn't need it), and I might replace maven-assembly-plugin with the shade plugin, but I'd expect your topology to work regardless.
Thank Stig Rohde Døssing. Finally I found the origin for my exception base on your hint regarding the log4j2/worker.xml. The issue was not in the worker.xml, but in the cluster.xml which is located in the same directory.
When reading my Exception shown here, you can see that Java complains about creating a system path ("access-web-${sys:daemon.name}.log"). I found the placeholder sys:daemon.name in the cluster.xml and replaced it with something static like "access-web-mysysdaemonname.log". That fixed the issue. I have no idea why this placeholder could not be resolved by the system while there was no trouble with other placeholders.
Thanks for the hints. Topic can be closed.

jaxb2-maven-plugin with external XSD dependencies

We have a common set of XSDs (datatypes, vocabulary, etc.) we're generating with the jaxb2-maven-plugin in its own Maven project. In a second project, I need to refer to one or more of those XSDs at compile time but don't want them included in the resulting artifact. I've created a catalog file, which works fine except I get everything in it.
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>jaxb2-maven-plugin</artifactId>
<version>1.6</version>
<executions>
<execution>
<goals>
<goal>xjc</goal>
</goals>
</execution>
</executions>
<configuration>
<outputDirectory>${basedir}/src/main/java</outputDirectory>
<target>2.1</target>
<catalog>catalog.cat</catalog>
</configuration>
I've pored over the plugin docs, but they're woefully light on detail. Is there any way to get reuse out of common schemas without every project having to take a copy of them?
Thanks
This is something my maven-jaxb2-plugin can do:
Compiling schema from Maven Artifact
The documentation site is currently very unstable so here's snippets of the documentation.
<configuration>
<forceRegenerate>true</forceRegenerate>
<schemas>
<schema>
<dependencyResource>
<groupId>org.jvnet.jaxb2.maven2</groupId>
<artifactId>maven-jaxb2-plugin-tests-po</artifactId>
<!-- Can be defined in project dependencies or dependency management -->
<version>${project.version}</version>
<resource>purchaseorder.xsd</resource>
</dependencyResource>
</schema>
</schemas>
</configuration>
Here's a sample project.

XML maven artifact not on classpath

I have some external configuration (XML files) that are installed in Maven. I need to have them on my test classpath but they aren't appearing.
They must stay as XML, I cannot package them inside a Jar - but I am willing to try anything else for this, custom plugin etc.
(Please don't inform me that Maven is only for Jars - that's simply not true (and if you provide a reference refuting that I can assure you it's out-of-date/misinformation).
The dependencies are specified thus:
<dependency>
<groupId>some.group</groupId>
<artifactId>some.artifact</artifactId>
<version>${project.version}</version>
<scope>test</scope>
<type>xml</type>
<classifier>some.classifier</classifier>
</dependency>
These XML artifacts have been created by the build-helper plugin (so there's no 1-2-1 with their project's POM).
My only current hacky solution is to, check for the M2_HOME property and load the files from there (as they're defined as dependencies Maven does pull them down) - but I'm not happy with this.
EDIT: The next best hack is probably to use the maven-dependency-plugin to copy these to the output directory (target/classes). If my config is fine for Jars then this smells like a Maven bug.
EDIT 2: #khmarbaise asked for the build-helper-plugin config:
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<executions>
<execution>
<id>attach-artifacts</id>
<phase>package</phase>
<goals>
<goal>attach-artifact</goal>
</goals>
<configuration>
<artifacts>
<artifact>
<file>target/classes/ddl-seed.xml</file>
<type>xml</type>
<classifier>ddl-seed</classifier>
</artifact>
<!-- ... more definitions -->
This generates the correct maven-metadata-local.xml data for all the XML artifacts.
Unfortunately I can find no way of forcing maven to add the test dependency specified to the test classpath, other than this stinky hack of copying it to a directory on the test classpath.
This seems the quickest way (it's for a test dependency), avoiding any JAR creation.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<id>unpack</id>
<phase>generate-test-resources</phase>
<goals>
<goal>copy</goal>
</goals>
<configuration>
<artifactItems>
<artifactItem>
<groupId>com.acme.gid</groupId>
<artifactId>com.acme.aid</artifactId>
<version>${project.version}</version>
<classifier>ddl</classifier>
<type>xml</type>
<outputDirectory>${project.build.directory}/test-classes</outputDirectory>
</artifactItem>
</artifactItems>
</configuration>
</execution>
</executions>
</plugin>

Resources