.jar created from maven shade plugin throws error when accessing resources under src/main/resources, but running main from exploded .jar works? - maven

Question

Updated Exec Summary of Solution
Following up from the answer provided by Victor, I implemented a Java class that lists the contents of a folder resource in the classpath. Most critical for me was that this had to work when the class path resource is discovered when executing from the IDE, from an exploded uberjar, or from within an unexploded uberjar (which I typically create with the maven shade plugin.) Class and associated unit test available here.
Original Question
I am seeing strange behavior with the maven-shade-plugin and class path resources when I run very simple
java Test program that access a directory structure in a standard maven project like this:
src/main
Test.java
resources/
resource-directory
spark
junk1
zeppelin
junk2
When run from the IDE or the exploded maven shaded .jar (please see below)
it works correctly, which means it prints this:.
result of directory contents as classpath resource:[spark, zeppelin]
The source is as follows:
import org.apache.commons.io.IOUtils;
import java.io.IOException;
import java.io.InputStream;
public class Tester {
public void test(String resourceName) throws IOException {
InputStream in = this.getClass().getClassLoader().getResourceAsStream(resourceName);
System.out.println("input stream: " + in);
Object result = IOUtils.readLines(in);
System.out.println("result of directory contents as classpath resource:" + result);
}
public static void main(String[] args) throws IOException {
new Tester().test("resource-directory");
}
}
Now, if I run mvn clean install in my project and run the
maven shaded .jar under ${project.dir}target, I see the following exception:
> java -jar target/sample.jar
Exception in thread "main" java.lang.NullPointerException
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:161)
at java.io.BufferedReader.readLine(BufferedReader.java:324)
at java.io.BufferedReader.readLine(BufferedReader.java:389)
at org.apache.commons.io.IOUtils.readLines(IOUtils.java:1030)
at org.apache.commons.io.IOUtils.readLines(IOUtils.java:987)
at org.apache.commons.io.IOUtils.readLines(IOUtils.java:968)
at Tester.test(Tester.java:16)
at Tester.main(Tester.java:24)
Running with Exploded .jar
> mkdir explode/
> cd explode/
> jar xvf ../sample.jar
......
inflated: META-INF/MANIFEST.MF
created: META-INF/
etc etc.
> ls # look at contents of exploded .jar:
logback.xml META-INF org resource-directory Tester.class
#
# now run class with CLASSPATH="."
(master) /tmp/maven-shade-non-working-example/target/explode > java Tester
input stream: java.io.ByteArrayInputStream#70dea4e
result of directory contents as classpath resource:[spark, zeppelin] # <<<- works !
I have the whole project here: https://github.com/buildlackey/maven-shade-non-working-example
but for convenience, here is the pom.xml(below), with two maven shade configs that I tried.
Note: I don't think the IncludeResourceTransformer would be of any use because my resources are appearing
at the appropriate levels in the .jar file.
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.foo.core</groupId>
<artifactId>sample</artifactId>
<packaging>jar</packaging>
<version>1.0-SNAPSHOT</version>
<name>sample</name>
<url>http://maven.apache.org</url>
<properties>
<jdk.version>1.8</jdk.version>
<junit.version>4.11</junit.version>
</properties>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>${junit.version}</version>
<scope>test</scope>
</dependency>
<dependency><!-- commons-io: Easy conversion from stream to string list, etc.-->
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.4</version>
</dependency>
</dependencies>
<build>
<finalName>sample</finalName>
<plugins>
<!-- Set a compiler level -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>2.3.2</version>
<configuration>
<source>${jdk.version}</source>
<target>${jdk.version}</target>
</configuration>
</plugin>
<!-- Maven Shade Plugin -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<executions>
<!-- Run shade goal on package phase -->
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<!-- add Main-Class to manifest file -->
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>Tester</mainClass>
</transformer>
<!-- tried with the stanza below enabled, and also disabled: in both cases, got exceptions from runs -->
<transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>src/main/resources/</resource>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
anyway, thanks in advance for any help you can provide ~
chris
UPDATE
This didn't work for me in Spring when I tried it (but I'd be interested if anyone has success with a Spring approach). I have a working alternative which I will post shortly. But if you care to comment on how to fix this broken Spring attempt, I'd be very interested.
import org.springframework.core.io.Resource;
import org.springframework.core.io.support.PathMatchingResourcePatternResolver;
import org.springframework.core.io.support.ResourcePatternResolver;
import java.io.IOException;
public class Tester {
public void test(String resourceName) throws IOException {
ResourcePatternResolver resourceResolver = new PathMatchingResourcePatternResolver();
Resource[] resources = resourceResolver.getResources("resource-directory/*");
for (Resource resource : resources) {
System.out.println("resource: " + resource.getDescription());
}
}
public static void main(String[] args) throws IOException {
new Tester().test("resource-directory/*");
}
}

Answer 1

The problem is that getResourceAsStream can read only files as a stream, not folders, from a jar file.
To read folder contents from a jar file you might need to use the approach, like described in the accepted answer to this question:
How can I get a resource "Folder" from inside my jar File?

Answer 2

To supplement the answer from my good friend Victor, here is a full code solution. below. The full project is available here
import java.io.File;
import java.io.IOException;
import java.util.*;
import java.util.zip.ZipEntry;
import java.util.zip.ZipException;
import java.util.zip.ZipFile;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* List entries of a subfolder of an entry in the class path, which may consist of file system folders and .jars.
*/
public class ClassPathResourceFolderLister {
private static final Logger LOGGER = LoggerFactory.getLogger(ClassPathResourceFolderLister.class);
/**
* For each entry in the classpath, verify that (a) "folder" exists, and (b) "folder" has child content, and if
* these conditions hold, return the child entries (be they files, or folders). If neither (a) nor (b) are true for
* a particular class path entry, move on to the next entry and try again.
*
* #param folder the folder to match within the class path entry
*
* #return the subfolder items of the first matching class path entry, with a no duplicates guarantee
*/
public static Collection<String> getFolderListing(final String folder) {
final String classPath = System.getProperty("java.class.path", ".");
final String[] classPathElements = classPath.split(System.getProperty("path.separator"));
List<String> classPathElementsList = new ArrayList<String> ( Arrays.asList(classPathElements));
return getFolderListingForFirstMatchInClassPath(folder, classPathElementsList);
}
private static Collection<String>
getFolderListingForFirstMatchInClassPath(final String folder, List<String> classPathElementsList) {
if (LOGGER.isDebugEnabled()) {
LOGGER.debug("getFolderListing for " + folder + " with classpath elements " + classPathElementsList);
}
Collection<String> retval = new HashSet<String>();
String cleanedFolder = stripTrailingAndLeadingSlashes(folder);
for (final String element : classPathElementsList) {
System.out.println("class path element:" + element);
retval = getFolderListing(element, cleanedFolder);
if (retval.size() > 0) {
if (LOGGER.isDebugEnabled()) {
LOGGER.debug("found matching folder in class path list. returning: " + retval);
}
return retval;
}
}
return retval;
}
private static String stripTrailingAndLeadingSlashes(final String folder) {
String stripped = folder;
if (stripped.equals("/")) { // handle degenerate case:
return "";
} else { // handle cases for strings starting or ending with "/", confident that we have at least two characters
if (stripped.endsWith("/")) {
stripped = stripped.substring(0, stripped.length()-1);
}
if (stripped.startsWith("/")) {
stripped = stripped.substring(1, stripped.length());
}
if (stripped.startsWith("/") || stripped.endsWith("/")) {
throw new IllegalArgumentException("too many consecutive slashes in folder specification: " + stripped);
}
}
return stripped;
}
private static Collection<String> getFolderListing( final String element, final String folderName) {
final File file = new File(element);
if (file.isDirectory()) {
return getFolderContentsListingFromSubfolder(file, folderName);
} else {
return getResourcesFromJarFile(file, folderName);
}
}
private static Collection<String> getResourcesFromJarFile(final File file, final String folderName) {
final String leadingPathOfZipEntry = folderName + "/";
final HashSet<String> retval = new HashSet<String>();
ZipFile zf = null;
try {
zf = new ZipFile(file);
final Enumeration e = zf.entries();
while (e.hasMoreElements()) {
final ZipEntry ze = (ZipEntry) e.nextElement();
final String fileName = ze.getName();
if (LOGGER.isTraceEnabled()) {
LOGGER.trace("zip entry fileName:" + fileName);
}
if (fileName.startsWith(leadingPathOfZipEntry)) {
final String justLeafPartOfEntry = fileName.replaceFirst(leadingPathOfZipEntry,"");
final String initSegmentOfPath = justLeafPartOfEntry.replaceFirst("/.*", "");
if (initSegmentOfPath.length() > 0) {
LOGGER.trace(initSegmentOfPath);
retval.add(initSegmentOfPath);
}
}
}
} catch (Exception e) {
throw new RuntimeException("getResourcesFromJarFile failed. file=" + file + " folder=" + folderName, e);
} finally {
if (zf != null) {
try {
zf.close();
} catch (IOException e) {
LOGGER.error("getResourcesFromJarFile close failed. file=" + file + " folder=" + folderName, e);
}
}
}
return retval;
}
private static Collection<String> getFolderContentsListingFromSubfolder(final File directory, String folderName) {
final HashSet<String> retval = new HashSet<String>();
try {
final String fullPath = directory.getCanonicalPath() + "/" + folderName;
final File subFolder = new File(fullPath);
System.out.println("fullPath:" + fullPath);
if (subFolder.isDirectory()) {
final File[] fileList = subFolder.listFiles();
for (final File file : fileList) {
retval .add(file.getName());
}
}
} catch (final IOException e) {
throw new Error(e);
}
return retval;
}
}