Mac OS Hadoop: Exception in thread "main" java.io.FileNotFoundException - hadoop

When I run a jar file in Hadoop, I run into a problem.
In the terminal, I get the following exception:
Exception in thread "main" java.io.FileNotFoundException: /var/folders/5_/hxmqt1090j1g1tqm485hr7tw0000gn/T/hadoop-unjar898783490589040837/META-INF/LICENSE (Is a directory)
How can I solve this problem?

I ran into this issue, my code written in scala, and I wanna run the jar on hadoop with command:
like:
./bin/hadoop jar testing/learning-yarn-1.0.0.jar com.learning.yarn.simpleapp.SimpleApp
It report error:
Exception in thread "main" java.io.FileNotFoundException: /var/folders/dy/kgryx_f11g1fdqpcnc8jl
m840000gp/T/hadoop-unjar5812913115154946721/META-INF/LICENSE (Is a directory)
I modify my maven configuration base on Hadoop java.io.IOException: Mkdirs failed to create /some/path
And now soved it.
My maven shade plugin configuration is like this:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<minimizeJar>true</minimizeJar>
<!-- <shadedArtifactAttached>true</shadedArtifactAttached>-->
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ApacheLicenseResourceTransformer">
</transformer>
</transformers>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>log4j.properties</exclude>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
<exclude>META-INF/LICENSE*</exclude>
<exclude>license/*</exclude>
</excludes>
</filter>
<filter>
<artifact>com.typesafe.akka</artifact>
<includes>
<include>reference.conf</include>
</includes>
</filter>
</filters>
</configuration>
</execution>
</executions>
</plugin>
if I add true it will report scala class not found, comment out this config worked.
Good luck.

Related

Oozie loading jars provided by user first

I have tried oozie.launcher.mapreduce.job.user.classpath.first property in workflow.xml to load my jars first over sharelib jars.
Unable to run workflow when this property is used.
hadoop-hdfs is available in the provided jar and version is not incompatible
Maven relocation strategy can be used to avoid conflicting classes
http://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html
Posting pom.xml shaded plugin config which worked for me
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.2.1</version>
<executions>
<execution>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<relocations>
<relocation>
<pattern>org.apache.http</pattern>
<shadedPattern>shaded.org.apache.http</shadedPattern>
</relocation>
</relocations>
<shadedArtifactAttached>true</shadedArtifactAttached>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
</execution>
</executions>
</plugin>
</plugins>

How to build a jar from a module of maven with dependencies from other module?

I have some modules in maven, and I want to build a jar of a module which has a dependency of just other module.
How can I include in the jar both modules?
Edit:
I have the followings modules:
myproject-core/
myproject-api/
myproject-dependencie/
myproject-api-web/
And I want to build a jar in myproject-api with myproject-dependencie. I have more dependencies in myproject-api and I only need to have in this jar myproject-api and myproject-dependencie.
Thanks!
I guess the maven-shade-plugin will be your friend:
https://maven.apache.org/plugins/maven-shade-plugin/
In you plugins section Something like:
...
<properties>
<shade-main-class>{your-main-class}</shade-main-class>
</properties>
...
...
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.3</version>
<configuration>
<createDependencyReducedPom>false</createDependencyReducedPom>
<filters>
<filter>
<artifact>*:*</artifact>
<includes>
<include>com/yourcompany/**</include>
</includes>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>${shade-main-class}</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
...

How to use shade plugin relocation with 3rdparty dependency injection

Can shade plugin modify resource and byte code such that dependency injection will us relocated packages? I don't think it does.
I shaded and relocated javax.transaction.TransactionManager to org.me.shaded.javax... and my code that use it appears to function but a org.infinispan:infinispan-core attempts dependency injection using the original package name, fails to locate TransactionManager and throws this error:
Caused by: java.lang.NoSuchMethodException: java.lang.Object.injectDependencies(..., javax.transaction.TransactionManager, ...)
When I look in my uber jar I find infinispan-core-component-metadata.dat when contains strings referencing the javax and not the shaded package.
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.1</version>
<configuration>
<minimizeJar>false</minimizeJar>
<createDependencyReducedPom>false</createDependencyReducedPom>
<keepDependenciesWithProvidedScope>false</keepDependenciesWithProvidedScope>
<shadedArtifactAttached>true</shadedArtifactAttached>
<relocations>
<relocation>
<pattern>javax.transaction</pattern>
<shadedPattern>org.me.shaded.javax.transaction</shadedPattern>
</relocation>
</relocations>
<artifactSet>
<excludes>
<exclude>org.apache.hbase:*</exclude>
<exclude>org.apache.ant:*</exclude>
<exclude>org.eclipse.jetty:*</exclude>
<exclude>org.springframework:*</exclude>
<exclude>geronimo-spec</exclude>
</excludes>
</artifactSet>
<filters>
<filter>
<excludes>
<exclude>webapps/**</exclude>
</excludes>
</filter>
<filter>
<artifact>org.infinispan:infinispan-core</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude> <exclude>META-INF/*.DSA</exclude> <exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
<filter>
<artifact>org.jboss.spec.javax.transaction:jboss-transaction-api_1.1_spec</artifact>
<excludes>
<exclude>javax/transaction/**</exclude>
</excludes>
</filter>
<filter>
<artifact>javax.transaction:jta</artifact>
<includes>
<include>javax/transaction/**</include>
</includes>
</filter>
</filters>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
</plugin>

Maven Shade Plugin not including resources of dependency

I'm using the Maven Shade Plugin to include all dependencies during package phase.
That works fine for classes, but dependent resources aren't included.
Here's the layout of the dependent jar:
./config.properties <-- this is the missing resource
./META-INF
./META-INF/MANIFEST.MF
./META-INF/maven
./META-INF/maven/com.example
./META-INF/maven/com.example/bar
./META-INF/maven/com.example/bar/pom.properties
./META-INF/maven/com.example/bar/pom.xml
Here's the shade plugin configuration:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>1.6</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<manifestEntries>
<Main-Class>com.example.foo.Foo</Main-Class>
<!-- <X-Compile-Source-JDK>${maven.compile.source}</X-Compile-Source-JDK>.
<X-Compile-Target-JDK>${maven.compile.target}</X-Compile-Target-JDK> -->
</manifestEntries>
</transformer>
</transformers>
<filters>
<filter>
<!--
Exclude files that sign a jar
(one or multiple of the dependencies).
One may not repack a signed jar without
this, or you will get a
SecurityException at program start.
-->
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.RSA</exclude>
<exclude>META-INF/*.INF</exclude> <!-- This one may not be required -->
</excludes>
</filter>
</filters>
</configuration>
</execution>
</executions>
</plugin>
It's embarrasing, but there was a typo in the version of the dependency and that version didn't have the file.

Maven shade jar throw exception

I have the following Exception:
Exception in thread "main" java.lang.SecurityException: no manifiest
section for signature file entry
javax/security/cert/CertificateException.class
at sun.security.util.SignatureFileVerifier.verifySection(SignatureFileVerifier.java:380)
at sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:231)
at sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:176)
at java.util.jar.JarVerifier.processEntry(JarVerifier.java:288)
at java.util.jar.JarVerifier.update(JarVerifier.java:199)
at java.util.jar.JarFile.initializeVerifier(JarFile.java:323)
at java.util.jar.JarFile.getInputStream(JarFile.java:388)
at sun.misc.URLClassPath$JarLoader$2.getInputStream(URLClassPath.java:692)
at sun.misc.Resource.cachedInputStream(Resource.java:61)
at sun.misc.Resource.getByteBuffer(Resource.java:144)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:256)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class: com.mainClass. Program will exit.
My pom:
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>1.5</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<filter>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>com.mainClass</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
The SecurityException comes up because one of your dependencies is a signed jar.
As the shade plugin is repacking this jar, it gets invalid. -> SecurityException at launch
To solve the problem, you have to unsign the dependency jars while repacking them.
This can be done by simply not repacking the files that make the jar signed, using a filter:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>1.5</version>
<executions>
<execution>
<id>stand-alone</id>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<shadedArtifactAttached>true</shadedArtifactAttached>
<shadedClassifierName>stand-alone</shadedClassifierName>
<filters>
<filter>
<!--
Exclude files that sign a jar
(one or multiple of the dependencies).
One may not repack a signed jar without
this, or you will get a
SecurityException at program start.
-->
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.RSA</exclude>
<exclude>META-INF/*.INF</exclude> <!-- This one may not be required -->
</excludes>
</filter>
</filters>
</configuration>
</execution>
</executions>
</plugin>
This solution was extracted from here:
https://issues.apache.org/jira/browse/MSHADE-61
The problem is because of java version.
I didn't notice that my new ide automatically use ibm's java, when I change the jre to sun's java ,it works well:)
The last line of the stacktrace above says
Could not find the main class: com.mainClass.
Perhaps a typo in the classname or the class is not compiled before the plugin in invoked?

Resources