Oozie loading jars provided by user first - hadoop

I have tried oozie.launcher.mapreduce.job.user.classpath.first property in workflow.xml to load my jars first over sharelib jars.
Unable to run workflow when this property is used.
hadoop-hdfs is available in the provided jar and version is not incompatible


Maven relocation strategy can be used to avoid conflicting classes
http://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html
Posting pom.xml shaded plugin config which worked for me
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.2.1</version>
<executions>
<execution>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<relocations>
<relocation>
<pattern>org.apache.http</pattern>
<shadedPattern>shaded.org.apache.http</shadedPattern>
</relocation>
</relocations>
<shadedArtifactAttached>true</shadedArtifactAttached>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
</execution>
</executions>
</plugin>
</plugins>

Related

Exclude specific files from all dependencies in pom

I want to exclude pdf files from all the dependencies. When I use filter property and specify the folder name for a pariticular dependency as below it works.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.2.1</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<filters>
<filter>
<artifact>com.test:myexample</artifact>
<excludes>
<exclude>TESTCASES/VFX/**/**/**/*.pdf</exclude>
<exclude>Test/**/*.pdf</exclude>
</excludes>
</filter>
</filters>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>com.test.CLI</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
However it does'nt work, when I used the configuration below. Reference link: https://maven.apache.org/plugins/maven-jar-plugin/examples/include-exclude.html
Do I need to follow any specific fileset pattern?
<configuration>
<excludes>
<exclude>**/*.pdf</exclude>
</excludes>
</configuration>

After adding the filter property as below it works.
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>**/*.pdf</exclude>
</excludes>
</filter>
</filters>

How to build a jar from a module of maven with dependencies from other module?

I have some modules in maven, and I want to build a jar of a module which has a dependency of just other module.
How can I include in the jar both modules?
Edit:
I have the followings modules:
myproject-core/
myproject-api/
myproject-dependencie/
myproject-api-web/
And I want to build a jar in myproject-api with myproject-dependencie. I have more dependencies in myproject-api and I only need to have in this jar myproject-api and myproject-dependencie.
Thanks!

I guess the maven-shade-plugin will be your friend:
https://maven.apache.org/plugins/maven-shade-plugin/
In you plugins section Something like:
...
<properties>
<shade-main-class>{your-main-class}</shade-main-class>
</properties>
...
...
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.3</version>
<configuration>
<createDependencyReducedPom>false</createDependencyReducedPom>
<filters>
<filter>
<artifact>*:*</artifact>
<includes>
<include>com/yourcompany/**</include>
</includes>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>${shade-main-class}</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
...

How to use shade plugin relocation with 3rdparty dependency injection

Can shade plugin modify resource and byte code such that dependency injection will us relocated packages? I don't think it does.
I shaded and relocated javax.transaction.TransactionManager to org.me.shaded.javax... and my code that use it appears to function but a org.infinispan:infinispan-core attempts dependency injection using the original package name, fails to locate TransactionManager and throws this error:
Caused by: java.lang.NoSuchMethodException: java.lang.Object.injectDependencies(..., javax.transaction.TransactionManager, ...)
When I look in my uber jar I find infinispan-core-component-metadata.dat when contains strings referencing the javax and not the shaded package.
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.1</version>
<configuration>
<minimizeJar>false</minimizeJar>
<createDependencyReducedPom>false</createDependencyReducedPom>
<keepDependenciesWithProvidedScope>false</keepDependenciesWithProvidedScope>
<shadedArtifactAttached>true</shadedArtifactAttached>
<relocations>
<relocation>
<pattern>javax.transaction</pattern>
<shadedPattern>org.me.shaded.javax.transaction</shadedPattern>
</relocation>
</relocations>
<artifactSet>
<excludes>
<exclude>org.apache.hbase:*</exclude>
<exclude>org.apache.ant:*</exclude>
<exclude>org.eclipse.jetty:*</exclude>
<exclude>org.springframework:*</exclude>
<exclude>geronimo-spec</exclude>
</excludes>
</artifactSet>
<filters>
<filter>
<excludes>
<exclude>webapps/**</exclude>
</excludes>
</filter>
<filter>
<artifact>org.infinispan:infinispan-core</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude> <exclude>META-INF/*.DSA</exclude> <exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
<filter>
<artifact>org.jboss.spec.javax.transaction:jboss-transaction-api_1.1_spec</artifact>
<excludes>
<exclude>javax/transaction/**</exclude>
</excludes>
</filter>
<filter>
<artifact>javax.transaction:jta</artifact>
<includes>
<include>javax/transaction/**</include>
</includes>
</filter>
</filters>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
</plugin>

Why maven does not include dependencies in resulting jar

I am trying to build an uber jar with maven-shade-plugin 2.1. I expect it to include all the classes in my jar and dependency jars. But I see that it does not include classes from dependency jars. What could I be doing wrong? Following is my usage of maven-shade-plugin in pom.xml. Could it be because finalName is same as project.artifactid?
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.1</version>
<configuration>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<minimizeJar>false</minimizeJar>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
</transformer>
</transformers>
<createDependencyReducedPom>false</createDependencyReducedPom>
<finalName>${project.artifactId}</finalName>
</configuration>
</execution>
</executions>
</plugin>

I tested your code and it works if it is inside <build> <plugins>, like this :
<build>
<plugins>
<plugin>
<!-- your code here -->
</plugin>
</plugins>
</build>
All dependencies are correctly included in the final jar. Also, make sure the dependencies you want to include in the shaded jar are inside the <dependencies> element, not the <dependencyManagement> element.

Maven Shade Plugin not including resources of dependency

I'm using the Maven Shade Plugin to include all dependencies during package phase.
That works fine for classes, but dependent resources aren't included.
Here's the layout of the dependent jar:
./config.properties <-- this is the missing resource
./META-INF
./META-INF/MANIFEST.MF
./META-INF/maven
./META-INF/maven/com.example
./META-INF/maven/com.example/bar
./META-INF/maven/com.example/bar/pom.properties
./META-INF/maven/com.example/bar/pom.xml
Here's the shade plugin configuration:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>1.6</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<manifestEntries>
<Main-Class>com.example.foo.Foo</Main-Class>
<!-- <X-Compile-Source-JDK>${maven.compile.source}</X-Compile-Source-JDK>.
<X-Compile-Target-JDK>${maven.compile.target}</X-Compile-Target-JDK> -->
</manifestEntries>
</transformer>
</transformers>
<filters>
<filter>
<!--
Exclude files that sign a jar
(one or multiple of the dependencies).
One may not repack a signed jar without
this, or you will get a
SecurityException at program start.
-->
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.RSA</exclude>
<exclude>META-INF/*.INF</exclude> <!-- This one may not be required -->
</excludes>
</filter>
</filters>
</configuration>
</execution>
</executions>
</plugin>

It's embarrasing, but there was a typo in the version of the dependency and that version didn't have the file.

Resources