Accessing files in a Jar using ClassPathResource - spring

I have a spring application that i must convert to jar. In this application I have a unit test:
#BeforeEach
void setUp() throws IOException {
//facturxHelper = new FacturxHelper();
facturxService = new FacturxService();
// String pdf = "facture.pdf"; // invalid pdfa1
String pdf = "resources/VALID PDFA1.pdf";
// InputStream sourceStream = new FileInputStream(pdf); //
InputStream sourceStream = getClass().getClassLoader().getResourceAsStream(pdf);
byte[] sourceBytes = IOUtils.toByteArray(sourceStream);
this.b64Pdf = Base64.getEncoder().encodeToString(sourceBytes);
}
#Test
void createFacturxMin() throws Exception {
// on va créer une facturX avec l'objet request
FacturxRequestMin request = FacturxRequestMin.builder()
.pdf(this.b64Pdf)
.chorusPro(Boolean.FALSE)
.invoiceNumber("FA-2017-0010")
.issueDate("13/11/2017")
.buyerReference("SERVEXEC")
.seller(TradeParty.builder()
.name("Au bon moulin")
.specifiedLegalOrganization(LegalOrganization.builder()
.id("99999999800010") .scheme(SpecifiedLegalOrganizationScheme.FR_SIRENE.getSpecifiedLegalOrganizationScheme())
.build())
.postalAddress(PostalAddress.builder()
.countryId(CountryIso.FR.name())
.build())
.vatId("FR11999999998")
.build())
.buyer(TradeParty.builder()
.name("Ma jolie boutique")
.specifiedLegalOrganization(LegalOrganization.builder()
.id("78787878400035")
.scheme(SpecifiedLegalOrganizationScheme.FR_SIRENE.getSpecifiedLegalOrganizationScheme())
.build())
.build())
.headerMonetarySummation(HeaderMonetarySummation.builder()
.taxBasisTotalAmount("624.90")
.taxTotalAmount("46.25")
.prepaidAmount("201.00")
.grandTotalAmount("671.15")
.duePayableAmount("470.15")
.build())
.build();
FacturXAppManager facturXAppManager = new FacturXAppManager(facturxService);
FacturxResponse facturxResponse = facturXAppManager.createFacturxMin(request);
Gson gson = new GsonBuilder().setPrettyPrinting().create();
String json = gson.toJson(facturxResponse);
System.out.println(json);
}
The aim of the application is to create an xml and to embed it into the pdf file.
My issue is concerning an xml validation through xsd.
Here is an abstract of the code :
public static boolean xmlValidator(String fxGuideLine, String xmlString) throws Exception {
System.out.println("xmlValidator() called");
File xsdFile = null;
Source source = new StreamSource(new StringReader(xmlString));
// i removed a lot of if else statement concerning files which allow to validate xml
try {
xsdFile = new ClassPathResource(FacturxConstants.FACTUR_X_MINIMUM_XSD).getFile();
} catch (IOException e) {
throw new FacturxException(e.getMessage());
}
// validation du contenu XML
try {
SchemaFactory schemaFactory = SchemaFactory
.newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI);
Schema schema = schemaFactory.newSchema(xsdFile);
Validator validator = schema.newValidator();
validator.validate(source);
return true;
} catch (SAXException | IOException e) {
throw new FacturxException(e.getLocalizedMessage());
}
...
}
In constants class, I added path to the xsd file:
public static final String FACTUR_X_MINIMUM_XSD = "resources/xsd/MINIMUM_XSD/FACTUR-X_MINIMUM.xsd";
In my POM file I do want to put the resources files in the built jar.
<build>
<finalName>${project.artifactId}</finalName>
<resources>
<resource>
<directory>src/main/resources</directory>
<includes>
<include>*</include>
</includes>
</resource>
</resources>
<plugins>
<plugin>
<artifactId>maven-resources-plugin</artifactId>
<version>3.3.0</version>
<configuration>
<outputDirectory> ${project.build.outputDirectory}\resources</outputDirectory>
</configuration>
</plugin>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>3.4.2</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
</plugin>
</plugins>
</build>
When I do a simple maven clean package, everything is running perfectly.
So far so good.
Next step is where my problem comes. Let's consider i want to use this dependency in an another application (a spring boot application). The previous jar compiled is a high level API that i want to integrate.
I launched the following command line :
mvn install:install-file -Dfile=myapi.jar -DgroupId=fr.myapi -DartifactId=graph-api-sharepoint -Dversion=1.0.0-SNAPSHOT -Dpackaging=jar
I do add my dependency correctly in my new project. that's perfect.
To check if my import worked correctly, i created a simple unit test with the same code (I do have a VALID PDFA1 in my resources folder. So far so good.
When running the test I do have the following error:
class path resource [resources/xsd/BASIC-WL_XSD/FACTUR-X_BASIC-WL.xsd] cannot be resolved to absolute file path because it does not reside in the file system: jar:file:/.m2/repository/fr/myapi/1.1.0/myapi-1.1.0.jar!/resources/xsd/BASIC-WL_XSD/FACTUR-X_BASIC-WL.xsd
How can i fix this issue ? I read many post but not fixes solved my issue. I do also think that i will have an issue also while compiling the springboot app as a jar
As mentionned, using a File won't work.
In the current code I updated it using InputStream:
InputStream is = new ClassPathResource(FacturxConstants.FACTUR_X_MINIMUM_XSD).getInputStream();
xsdSource = new StreamSource(is);
if my xsd path doesn't have resources:
public static final String FACTUR_X_MINIMUM_XSD = "xsd/MINIMUM_XSD/FACTUR-X_MINIMUM.xsd";
I have the following exception:
class path resource [xsd/MINIMUM_XSD/FACTUR-X_MINIMUM.xsd] cannot be opened because it does not exist
If i do put
public static final String FACTUR_X_MINIMUM_XSD = "resources/xsd/MINIMUM_XSD/FACTUR-X_MINIMUM.xsd";
the response is the following:
src-resolve: Cannot resolve the name 'ram:ExchangedDocumentContextType' to a(n) 'type definition' component.
I updated also the SchemaFactory and schema implementation:
SchemaFactory schemaFactory =
SchemaFactory.newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI);
Schema schema = schemaFactory.newSchema(xsdSource);
Validator validator = schema.newValidator();
validator.validate(source);
return true;

public static final String FACTUR_X_MINIMUM_XSD = "resources/xsd/MINIMUM_XSD/FACTUR-X_MINIMUM.xsd";
Is wrong it should be (assuming src/main/resources/xsd is the actual location you are using).
public static final String FACTUR_X_MINIMUM_XSD = "/xsd/MINIMUM_XSD/FACTUR-X_MINIMUM.xsd";
Then your code is using a java.io.File which won't work, as a java.io.File needs to be a physical file on the file system. Which this isn't as it is inside a jar file. You need to use an InputStream.
public static boolean xmlValidator(String fxGuideLine, String xmlString) throws Exception {
System.out.println("xmlValidator() called");
Source source = new StreamSource(new StringReader(xmlString));
// i removed a lot of if else statement concerning files which allow to validate xml
try {
InputStream xsd = new ClassPathResource(FacturxConstants.FACTUR_X_MINIMUM_XSD).getInputStream();
StreamSource xsdSource = new StreamSource(xsd);
SchemaFactory schemaFactory = SchemaFactory
.newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI);
Schema schema = schemaFactory.newSchema(xsdSource);
Validator validator = schema.newValidator();
validator.validate(source);
return true;
} catch (SAXException | IOException e) {
throw new FacturxException(e.getLocalizedMessage());
}
...
}
Which loads the schema using an inputstream.

Thanks to M. Deinum, I was able to find out a solution. I had to use indeed StreamSource. This didn't solve the following issue:
src-resolve: Cannot resolve the name 'ram:ExchangedDocumentContextType' to a(n) 'type definition' component.
As I used several xsd files, I implemented a way to retrieve a list of sources using PathMatchingResourcePatternResolver (from spring)
private static Source[] buildSources(String fxGuideLine, String pattern) throws SAXException, IOException {
List<Source> sources = new ArrayList<>();
PathMatchingResourcePatternResolver patternResolver = new PathMatchingResourcePatternResolver();
Resource[] resources = patternResolver.getResources(pattern);
for (Resource resource : resources) {
StreamSource dtd = new StreamSource(resource.getInputStream());
dtd.setSystemId(resource.getURI().toString());
sources.add(dtd);
}
return sources.toArray(new Source[sources.size()]);
}

Related

Java Spring Boot: How to access local WSDL instead of a Public WSDL URL?

I have been using a public WSDL URL to make a call to our customer. Now the customer decided to hide the public WSDL URL and I have been asked to use a local WSDL that I need to deploy on my own server.
I'm using Java Spring Boot and here's my previous code to call the public WSDL URL:
try {
SaajSoapMessageFactory messageFactory= new SaajSoapMessageFactory(MessageFactory.newInstance());
messageFactory.afterPropertiesSet();
WebServiceTemplate webServiceTemplate = new WebServiceTemplate( messageFactory);
Jaxb2Marshaller marshaller = new Jaxb2Marshaller();
marshaller.setContextPath(appConfig.SOAP_PKG);
marshaller.afterPropertiesSet();
webServiceTemplate.setMarshaller(marshaller);
webServiceTemplate.setUnmarshaller(marshaller);
webServiceTemplate.afterPropertiesSet();
WebServiceMessageSender messageSender = this.webServiceMessageSender();
webServiceTemplate.setMessageSender(messageSender);
try {
response = webServiceTemplate.marshalSendAndReceive(soapURL, request, new WebServiceMessageCallback() {
#Override
public void doWithMessage(WebServiceMessage message) {
try {
SoapHeader soapHeader = ((SoapMessage) message).getSoapHeader();
Map mapRequest = new HashMap();
mapRequest.put("loginuser", soapUsername);
mapRequest.put("loginpass", soapPassword);
StrSubstitutor substitutor = new StrSubstitutor(mapRequest, "%(", ")");
String finalXMLRequest = substitutor.replace(appConfig.SOAP_HEADER);
StringSource headerSource = new StringSource(finalXMLRequest);
Transformer transformer = TransformerFactory.newInstance().newTransformer();
transformer.transform(headerSource, soapHeader.getResult());
} catch (Exception e) {
logger.error("Error while invoking session service :", e.getMessage() );
}
}
});
}catch (SoapFaultClientException e){
logger.error("Error while invoking session service : " + e.getMessage());
}
....
How am I supposed now to replace "soapURL" which is the public WSDL URL used in marshalSendAndReceive with the local wsdl?
I used wsd2ljava to generate the sources in eclipse as shown below.
<plugin>
<groupId>org.apache.cxf</groupId>
<artifactId>cxf-codegen-plugin</artifactId>
<version>3.1.12</version>
<executions>
<execution>
<id>generate-sources</id>
<phase>generate-sources</phase>
<configuration>
<sourceRoot>${project.build.directory}/generated/cxf</sourceRoot>
<wsdlOptions>
<wsdlOption>
<wsdl>${basedir}/src/main/resources/wsdl/XXX.wsdl</wsdl>
<wsdlLocation>classpath:wsdl/XXX.wsdl</wsdlLocation>
</wsdlOption>
</wsdlOptions>
<fork>always</fork>
</configuration>
<goals>
<goal>wsdl2java</goal>
</goals>
</execution>
</executions>
</plugin>
Java classes have been created. What is the next step after generating all the classes? Shall I create a CXF client configuration?

Querydsl fetchCount() & fetch() NullPointerException, Connection is closed

My goal is to implement dao method within pagination, sorting and filtering. For pagination I need get firstly count and then set offset & limit and get result (so get just one "page" from database):
#Slf4j
#Repository
public class UserJdbcRepository {
private final SQLQueryFactory queryFactory;
#Autowired
public UserJdbcRepository(DataSource dataSource) {
Configuration configuration = new Configuration(new OracleTemplates());
configuration.setExceptionTranslator(new SpringExceptionTranslator());
this.queryFactory = new SQLQueryFactory(configuration, dataSource);
}
public Page<User> findAll(BooleanExpression predicate, Pageable pageable) {
QUser u = new QUser("u");
SQLQuery<Tuple> sql = queryFactory
.select(u.userId, // omitted)
.from(u)
.where(predicate);
long count = sql.fetchCount();
List<Tuple> results = sql.fetch();
// Conversion List<Tuple> to List<User> omitted
return new PageImpl<>(users, pageable, count);
}
}
fetchCount() is executed correctly, but fetch() is throwing NullPointerException:
java.lang.NullPointerException: null
at com.querydsl.sql.AbstractSQLQuery.fetch(AbstractSQLQuery.java:502) ~[querydsl-sql-4.4.0.jar:na]
From debug I found that root cause is in com.querydsl.sql.AbstractSQLQuery:
java.sql.SQLException: Connection is closed
If I create second (the same as first one) query sql2, then it is working (of course):
SQLQuery<Tuple> sql2 = queryFactory
.select(... // same as first one)
long count = sql.fetchCount();
List<Tuple> results = sql2.fetch();
My question is if connection should be really closed after fetchCount() is called? Or do I have some misconfiguration?
I have SpringBoot 2.4.5; spring-data-commons 2.5.0; Oracle driver ojdbc8 21.1.0.0; QueryDSL 4.4.0
<dependency>
<groupId>com.querydsl</groupId>
<artifactId>querydsl-sql</artifactId>
<version>${querydsl.version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>com.querydsl</groupId>
<artifactId>querydsl-sql-spring</artifactId>
<version>${querydsl.version}</version>
<scope>compile</scope>
</dependency>
<plugin>
<groupId>com.querydsl</groupId>
<artifactId>querydsl-maven-plugin</artifactId>
<version>${querydsl.version}</version>
<executions>
<execution>
<goals>
<goal>export</goal>
</goals>
</execution>
</executions>
<configuration>
<jdbcDriver>oracle.jdbc.OracleDriver</jdbcDriver>
<jdbcUrl>jdbc:oracle:thin:#//localhost:1521/XE</jdbcUrl>
<jdbcUser>user</jdbcUser>
<jdbcPassword>password</jdbcPassword>
<sourceFolder>${project.basedir}/src/main/java</sourceFolder>
<targetFolder>${project.basedir}/src/main/java</targetFolder>
<packageName>org.project.backend.repository.querydsl</packageName>
<schemaToPackage>true</schemaToPackage>
<schemaPattern>project</schemaPattern>
<tableNamePattern>
// omitted
</tableNamePattern>
</configuration>
<plugin>
Issue is caused by missconfiguration. QueryDSL doc contains Spring integration section, where is mentioned that SpringConnectionProvider must be used. So I changed my constructor and it is working now as expected:
#Autowired
public UserJdbcRepository(DataSource dataSource) {
Configuration configuration = new Configuration(new OracleTemplates());
configuration.setExceptionTranslator(new SpringExceptionTranslator());
// wrong: this.queryFactory = new SQLQueryFactory(configuration, dataSource);
Provider<Connection> provider = new SpringConnectionProvider(dataSource);
this.queryFactory = new SQLQueryFactory(configuration, provider);
}
I also found there is useful method fetchResults() containing count for pagination purpose (so not needed to explicitly call fetchCount()):
public Page<User> findAll(BooleanExpression predicate, Pageable pageable) {
QUser u = new QUser("u");
SQLQuery<Tuple> sql = queryFactory
.select(u.userId, // omitted)
.from(u)
.where(predicate);
sql.offset(pageable.getOffset());
sql.limit(pageable.getPageSize());
QueryResults<Tuple> queryResults = sql.fetchResults();
long count = queryResults.getTotal();
List<Tuple> results = queryResults.getResults();
// Conversion List<Tuple> to List<User> omitted
return new PageImpl<>(users, pageable, count);
}

.jar created from maven shade plugin throws error when accessing resources under src/main/resources, but running main from exploded .jar works?

Updated Exec Summary of Solution
Following up from the answer provided by Victor, I implemented a Java class that lists the contents of a folder resource in the classpath. Most critical for me was that this had to work when the class path resource is discovered when executing from the IDE, from an exploded uberjar, or from within an unexploded uberjar (which I typically create with the maven shade plugin.) Class and associated unit test available here.
Original Question
I am seeing strange behavior with the maven-shade-plugin and class path resources when I run very simple
java Test program that access a directory structure in a standard maven project like this:
src/main
Test.java
resources/
resource-directory
spark
junk1
zeppelin
junk2
When run from the IDE or the exploded maven shaded .jar (please see below)
it works correctly, which means it prints this:.
result of directory contents as classpath resource:[spark, zeppelin]
The source is as follows:
import org.apache.commons.io.IOUtils;
import java.io.IOException;
import java.io.InputStream;
public class Tester {
public void test(String resourceName) throws IOException {
InputStream in = this.getClass().getClassLoader().getResourceAsStream(resourceName);
System.out.println("input stream: " + in);
Object result = IOUtils.readLines(in);
System.out.println("result of directory contents as classpath resource:" + result);
}
public static void main(String[] args) throws IOException {
new Tester().test("resource-directory");
}
}
Now, if I run mvn clean install in my project and run the
maven shaded .jar under ${project.dir}target, I see the following exception:
> java -jar target/sample.jar
Exception in thread "main" java.lang.NullPointerException
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:161)
at java.io.BufferedReader.readLine(BufferedReader.java:324)
at java.io.BufferedReader.readLine(BufferedReader.java:389)
at org.apache.commons.io.IOUtils.readLines(IOUtils.java:1030)
at org.apache.commons.io.IOUtils.readLines(IOUtils.java:987)
at org.apache.commons.io.IOUtils.readLines(IOUtils.java:968)
at Tester.test(Tester.java:16)
at Tester.main(Tester.java:24)
Running with Exploded .jar
> mkdir explode/
> cd explode/
> jar xvf ../sample.jar
......
inflated: META-INF/MANIFEST.MF
created: META-INF/
etc etc.
> ls # look at contents of exploded .jar:
logback.xml META-INF org resource-directory Tester.class
#
# now run class with CLASSPATH="."
(master) /tmp/maven-shade-non-working-example/target/explode > java Tester
input stream: java.io.ByteArrayInputStream#70dea4e
result of directory contents as classpath resource:[spark, zeppelin] # <<<- works !
I have the whole project here: https://github.com/buildlackey/maven-shade-non-working-example
but for convenience, here is the pom.xml(below), with two maven shade configs that I tried.
Note: I don't think the IncludeResourceTransformer would be of any use because my resources are appearing
at the appropriate levels in the .jar file.
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.foo.core</groupId>
<artifactId>sample</artifactId>
<packaging>jar</packaging>
<version>1.0-SNAPSHOT</version>
<name>sample</name>
<url>http://maven.apache.org</url>
<properties>
<jdk.version>1.8</jdk.version>
<junit.version>4.11</junit.version>
</properties>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>${junit.version}</version>
<scope>test</scope>
</dependency>
<dependency><!-- commons-io: Easy conversion from stream to string list, etc.-->
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.4</version>
</dependency>
</dependencies>
<build>
<finalName>sample</finalName>
<plugins>
<!-- Set a compiler level -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>2.3.2</version>
<configuration>
<source>${jdk.version}</source>
<target>${jdk.version}</target>
</configuration>
</plugin>
<!-- Maven Shade Plugin -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<executions>
<!-- Run shade goal on package phase -->
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<!-- add Main-Class to manifest file -->
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>Tester</mainClass>
</transformer>
<!-- tried with the stanza below enabled, and also disabled: in both cases, got exceptions from runs -->
<transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>src/main/resources/</resource>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
anyway, thanks in advance for any help you can provide ~
chris
UPDATE
This didn't work for me in Spring when I tried it (but I'd be interested if anyone has success with a Spring approach). I have a working alternative which I will post shortly. But if you care to comment on how to fix this broken Spring attempt, I'd be very interested.
import org.springframework.core.io.Resource;
import org.springframework.core.io.support.PathMatchingResourcePatternResolver;
import org.springframework.core.io.support.ResourcePatternResolver;
import java.io.IOException;
public class Tester {
public void test(String resourceName) throws IOException {
ResourcePatternResolver resourceResolver = new PathMatchingResourcePatternResolver();
Resource[] resources = resourceResolver.getResources("resource-directory/*");
for (Resource resource : resources) {
System.out.println("resource: " + resource.getDescription());
}
}
public static void main(String[] args) throws IOException {
new Tester().test("resource-directory/*");
}
}
The problem is that getResourceAsStream can read only files as a stream, not folders, from a jar file.
To read folder contents from a jar file you might need to use the approach, like described in the accepted answer to this question:
How can I get a resource "Folder" from inside my jar File?
To supplement the answer from my good friend Victor, here is a full code solution. below. The full project is available here
import java.io.File;
import java.io.IOException;
import java.util.*;
import java.util.zip.ZipEntry;
import java.util.zip.ZipException;
import java.util.zip.ZipFile;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* List entries of a subfolder of an entry in the class path, which may consist of file system folders and .jars.
*/
public class ClassPathResourceFolderLister {
private static final Logger LOGGER = LoggerFactory.getLogger(ClassPathResourceFolderLister.class);
/**
* For each entry in the classpath, verify that (a) "folder" exists, and (b) "folder" has child content, and if
* these conditions hold, return the child entries (be they files, or folders). If neither (a) nor (b) are true for
* a particular class path entry, move on to the next entry and try again.
*
* #param folder the folder to match within the class path entry
*
* #return the subfolder items of the first matching class path entry, with a no duplicates guarantee
*/
public static Collection<String> getFolderListing(final String folder) {
final String classPath = System.getProperty("java.class.path", ".");
final String[] classPathElements = classPath.split(System.getProperty("path.separator"));
List<String> classPathElementsList = new ArrayList<String> ( Arrays.asList(classPathElements));
return getFolderListingForFirstMatchInClassPath(folder, classPathElementsList);
}
private static Collection<String>
getFolderListingForFirstMatchInClassPath(final String folder, List<String> classPathElementsList) {
if (LOGGER.isDebugEnabled()) {
LOGGER.debug("getFolderListing for " + folder + " with classpath elements " + classPathElementsList);
}
Collection<String> retval = new HashSet<String>();
String cleanedFolder = stripTrailingAndLeadingSlashes(folder);
for (final String element : classPathElementsList) {
System.out.println("class path element:" + element);
retval = getFolderListing(element, cleanedFolder);
if (retval.size() > 0) {
if (LOGGER.isDebugEnabled()) {
LOGGER.debug("found matching folder in class path list. returning: " + retval);
}
return retval;
}
}
return retval;
}
private static String stripTrailingAndLeadingSlashes(final String folder) {
String stripped = folder;
if (stripped.equals("/")) { // handle degenerate case:
return "";
} else { // handle cases for strings starting or ending with "/", confident that we have at least two characters
if (stripped.endsWith("/")) {
stripped = stripped.substring(0, stripped.length()-1);
}
if (stripped.startsWith("/")) {
stripped = stripped.substring(1, stripped.length());
}
if (stripped.startsWith("/") || stripped.endsWith("/")) {
throw new IllegalArgumentException("too many consecutive slashes in folder specification: " + stripped);
}
}
return stripped;
}
private static Collection<String> getFolderListing( final String element, final String folderName) {
final File file = new File(element);
if (file.isDirectory()) {
return getFolderContentsListingFromSubfolder(file, folderName);
} else {
return getResourcesFromJarFile(file, folderName);
}
}
private static Collection<String> getResourcesFromJarFile(final File file, final String folderName) {
final String leadingPathOfZipEntry = folderName + "/";
final HashSet<String> retval = new HashSet<String>();
ZipFile zf = null;
try {
zf = new ZipFile(file);
final Enumeration e = zf.entries();
while (e.hasMoreElements()) {
final ZipEntry ze = (ZipEntry) e.nextElement();
final String fileName = ze.getName();
if (LOGGER.isTraceEnabled()) {
LOGGER.trace("zip entry fileName:" + fileName);
}
if (fileName.startsWith(leadingPathOfZipEntry)) {
final String justLeafPartOfEntry = fileName.replaceFirst(leadingPathOfZipEntry,"");
final String initSegmentOfPath = justLeafPartOfEntry.replaceFirst("/.*", "");
if (initSegmentOfPath.length() > 0) {
LOGGER.trace(initSegmentOfPath);
retval.add(initSegmentOfPath);
}
}
}
} catch (Exception e) {
throw new RuntimeException("getResourcesFromJarFile failed. file=" + file + " folder=" + folderName, e);
} finally {
if (zf != null) {
try {
zf.close();
} catch (IOException e) {
LOGGER.error("getResourcesFromJarFile close failed. file=" + file + " folder=" + folderName, e);
}
}
}
return retval;
}
private static Collection<String> getFolderContentsListingFromSubfolder(final File directory, String folderName) {
final HashSet<String> retval = new HashSet<String>();
try {
final String fullPath = directory.getCanonicalPath() + "/" + folderName;
final File subFolder = new File(fullPath);
System.out.println("fullPath:" + fullPath);
if (subFolder.isDirectory()) {
final File[] fileList = subFolder.listFiles();
for (final File file : fileList) {
retval .add(file.getName());
}
}
} catch (final IOException e) {
throw new Error(e);
}
return retval;
}
}

How to set up testsuite files for execution across environments

I have a test suite file that needs ability to be executed from both mvn command line (from Jenkins) as well as on-demand from Eclipse.
The test suite file must have ability to support parameters, ie:
<suite name="test run1">
<parameter name="testEnv" value="dev"></parameter>
<parameter name="proxyServer" value="x"></parameter>
<parameter name="proxyPort" value="y"></parameter>
If I leave as is, then mvn command line parameters don't work, as the values in the test suite file will override the parameters. i.e. this will not work:
mvn test ... -dtestEnv=E1QA -dproxyServer= -dproxyPort=
How can I write the test suite file so it supports both ad-hoc execution from Eclipse and mvn command line execution?
If you want configurable test property, use #DataProvider instead of hardcoding suite xml.
Provider class:
public class EnvProvider {
#DataProvider(name = "envProvider")
public static Object[][] createData() {
return new Object[][] { new Object[] {
System.getProperty("testEnv", "eclipse-default") }
};
}
Test method:
#Test(dataProvider = "envProvider", dataProviderClass = EnvProvider.class)
public void myTest(String currentEnv) {
System.out.println("Current env is : " + currentEnv);
}
pom.xml
<properties>
<testEnv>default-pom</testEnv>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19.1</version>
<configuration>
<systemPropertyVariables>
<testEnv>${testEnv}</testEnv>
</systemPropertyVariables>
...
Result from eclipse right click
Current env is : eclipse-default
Result from mvn test
Current env is : default-pom
Result from mvn test -DtestEnv=jenkins
Current env is : jenkins
References: http://testng.org/doc/documentation-main.html#parameters-dataproviders
You can override the suite xml params with system level args if those are available - which would allow you to run from xml as well as from mvn.
All the params are basically assigned to some constant properties to be used across tests. The properties file is initialized in the onBeforeSuite method of the suite listener something to the effect
SuiteList extends SuiteListener{
public void onStart(ISuite suite) {
LProperties.loadProperties(suite);
}
//loadProperties implementation
LProperties{
public static void loadProperties(ISuite suite){
//This can read from a properties file and load properties
//or
// from the suite.getParameter - pick the params you need (reflection or list) and assign to the constants
//For all the properties, do System.getProperty and override values if found
}
Use LProperties constants in your tests.
based on a combination of above answers, I've figured it out.
First remove the hard-coded parameters in the test suite file.
Next, ensure the parameters are supported in the pom file.
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.4.2</version>
<configuration>
<systemPropertyVariables>
<environment>${testEnv}</environment>
<environment>${proxyServer}</environment>
<environment>${proxyPort}</environment>
</systemPropertyVariables>
</configuration>
</plugin>
</plugins>
</build>
Create constants file to contain default values for the parameters
public class Constants {
public static final String DEFAULT_TEST_ENV = "dev";
public static final String DEFAULT_PROXY_SERVER = "a-dev.phx.com";
public static final String DEFAULT_PROXY_PORT = "8585";
}
In the testNG setup() method, use System.getproperty():
#BeforeClass(alwaysRun = true)
protected static void setUp() throws Exception {
String testEnv = System.getProperty("testEnv", Constants.DEFAULT_TEST_ENV);
String proxyServer = System.getProperty("proxyServer", Constants.DEFAULT_PROXY_SERVER);
String proxyPort = System.getProperty("proxyPort", Constants.DEFAULT_PROXY_PORT);
System.out.println("testEnv: " + testEnv);
System.out.println("proxyServer: " + proxyServer);
System.out.println("proxyPort: " + proxyPort);
}
I could put the default values in a config file, but for now, constants file seems easiest in its own class.

configuring the encrypted database password in the spring datasource [duplicate]

I have the task of obfuscating passwords in our configuration files. While I don't think this is the right approach, managers disagree...
So the project I am working on is based on Spring Boot and we are using YAML configuration files. Currently the passwords are in plain text:
spring:
datasource:
url: jdbc:sqlserver://DatabaseServer
driverClassName: com.microsoft.sqlserver.jdbc.SQLServerDriver
username: ele
password: NotTheRealPassword
The idea is to have some special syntax that supports an obfuscated or encrypted password:
spring:
datasource:
url: jdbc:sqlserver://DatabaseServer
driverClassName: com.microsoft.sqlserver.jdbc.SQLServerDriver
username: ele
password: password(Tm90VGhlUmVhbFBhc3N3b3Jk)
In order for this to work I want to parse the property values using a regular expression and if it matches replace the value with the deobfuscated/decrypted value.
But how do I intercept the property value?
If finally got this to work. (Mainly thanks to stephane-deraco on github)
Key to the solution is a class that implements ApplicationContextInitializer<ConfigurableApplicationContext>. I called it PropertyPasswordDecodingContextInitializer.
The main problem was to get spring to use this ApplicationContextInitializer. Important information can be found in the reference. I chose the approach using a META-INF/spring.factories with following content:
org.springframework.context.ApplicationContextInitializer=ch.mycompany.myproject.PropertyPasswordDecodingContextInitializer
The PropertyPasswordDecodingContextInitializer uses a PropertyPasswordDecoder and an implementing class, currently for simplicity a Base64PropertyPasswordDecoder.
PropertyPasswordDecodingContextInitializer.java
package ch.mycompany.myproject;
import java.util.LinkedHashMap;
import java.util.Map;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import org.springframework.context.ApplicationContextInitializer;
import org.springframework.context.ConfigurableApplicationContext;
import org.springframework.core.env.CompositePropertySource;
import org.springframework.core.env.ConfigurableEnvironment;
import org.springframework.core.env.EnumerablePropertySource;
import org.springframework.core.env.MapPropertySource;
import org.springframework.core.env.PropertySource;
import org.springframework.stereotype.Component;
#Component
public class PropertyPasswordDecodingContextInitializer implements ApplicationContextInitializer<ConfigurableApplicationContext> {
private static final Pattern decodePasswordPattern = Pattern.compile("password\\((.*?)\\)");
private PropertyPasswordDecoder passwordDecoder = new Base64PropertyPasswordDecoder();
#Override
public void initialize(ConfigurableApplicationContext applicationContext) {
ConfigurableEnvironment environment = applicationContext.getEnvironment();
for (PropertySource<?> propertySource : environment.getPropertySources()) {
Map<String, Object> propertyOverrides = new LinkedHashMap<>();
decodePasswords(propertySource, propertyOverrides);
if (!propertyOverrides.isEmpty()) {
PropertySource<?> decodedProperties = new MapPropertySource("decoded "+ propertySource.getName(), propertyOverrides);
environment.getPropertySources().addBefore(propertySource.getName(), decodedProperties);
}
}
}
private void decodePasswords(PropertySource<?> source, Map<String, Object> propertyOverrides) {
if (source instanceof EnumerablePropertySource) {
EnumerablePropertySource<?> enumerablePropertySource = (EnumerablePropertySource<?>) source;
for (String key : enumerablePropertySource.getPropertyNames()) {
Object rawValue = source.getProperty(key);
if (rawValue instanceof String) {
String decodedValue = decodePasswordsInString((String) rawValue);
propertyOverrides.put(key, decodedValue);
}
}
}
}
private String decodePasswordsInString(String input) {
if (input == null) return null;
StringBuffer output = new StringBuffer();
Matcher matcher = decodePasswordPattern.matcher(input);
while (matcher.find()) {
String replacement = passwordDecoder.decodePassword(matcher.group(1));
matcher.appendReplacement(output, replacement);
}
matcher.appendTail(output);
return output.toString();
}
}
PropertyPasswordDecoder.java
package ch.mycompany.myproject;
public interface PropertyPasswordDecoder {
public String decodePassword(String encodedPassword);
}
Base64PropertyPasswordDecoder.java
package ch.mycompany.myproject;
import java.io.UnsupportedEncodingException;
import org.apache.commons.codec.binary.Base64;
public class Base64PropertyPasswordDecoder implements PropertyPasswordDecoder {
#Override
public String decodePassword(String encodedPassword) {
try {
byte[] decodedData = Base64.decodeBase64(encodedPassword);
String decodedString = new String(decodedData, "UTF-8");
return decodedString;
} catch (UnsupportedEncodingException e) {
throw new RuntimeException(e);
}
}
}
Mind you, the ApplicationContext has not finished initialized at this stage, so autowiring or any other bean related mechanisms won't work.
Update: Included #jny's suggestions.
I used #Daniele Torino's answer and made several minor changes.
First, thanks to his link to the options on how to make spring recognize Initializer, I chose to do it in the Application:
public static void main(String[] args) throws Exception {
SpringApplication application=new SpringApplication(Application.class);
application.addInitializers(new PropertyPasswordDecodingContextInitializer());
application.run(args);
}
Second, IDEA told me that that else if (source instanceof CompositePropertySource) { is redundant and it is because CompositePropertySource inherits from EnumerablePropertySource.
Third, I beleive there is a minor bug: it messes up the order of property resolution. If you have one encoded property in environment, and another one in application.properties file the environment value will be overwritten with the application.properties value.
I changed the logic to insert the decodedProperties right before encoded:
for (PropertySource<?> propertySource : environment.getPropertySources()) {
Map<String, Object> propertyOverrides = new LinkedHashMap<>();
decodePasswords(propertySource, propertyOverrides);
if (!propertyOverrides.isEmpty()) {
environment.getPropertySources().addBefore(propertySource.getName(), new MapPropertySource("decoded"+propertySource.getName(), propertyOverrides));
}
}
Just use https://github.com/ulisesbocchio/jasypt-spring-boot, works out of the box
Inspired by #gogstad. Here is my major action in the spring boot project to encrypted my username and password and decrypted them in the project to work with tomcat:
1. In pom.xml file
<dependency>
<groupId>com.github.ulisesbocchio</groupId>
<artifactId>jasypt-spring-boot</artifactId>
<version>1.12</version>
</dependency>
…
<build>
<resources>
<resource>
<directory>src/main/java</directory>
<includes>
<include>**/*.properties</include>
<include>**/*.xml</include>
</includes>
<targetPath>${project.build.directory}/classes</targetPath>
</resource>
<resource>
<directory>src/main/resources</directory>
<includes>
<include>**/*.properties</include>
</includes>
<targetPath>${project.build.directory}/classes</targetPath>
</resource>
</resources>
…
</build>
2. In App.java (Note:to deploy the decryted springboot on tomcat, you should add the #ServletComponentScan annotation and extends the SpringBootServletInitializer)
#SpringBootApplication
#ServletComponentScan
#EnableEncryptableProperties
#PropertySource(name="EncryptedProperties", value = "classpath:config/encrypted.properties")
public class App extends SpringBootServletInitializer {
public static void main(String[] args) throws Exception {
SpringApplication.run(App.class, args);
}
}
3. Encrypted your username and password and fill the application.properties file with the result:
java -cp ~/.m2/repository/org/jasypt/jasypt/1.9.2/jasypt-1.9.2.jar org.jasypt.intf.cli.JasyptPBEStringEncryptionCLI input="mypassword" password=mykey algorithm=PBEWithMD5AndDES
output is like the demo below:
java -cp ~/.m2/repository/org/jasypt/jasypt/1.9.2/jasypt-1.9.2.jar org.jasypt.intf.cli.JasyptPBEStringEncryptionCLI input="mypassword" password=mykey algorithm=PBEWithMD5AndDES
----ENVIRONMENT-----------------
Runtime: Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.45-b02
----ARGUMENTS-------------------
algorithm: PBEWithMD5AndDES
input: mypassword
password: mykey
----OUTPUT----------------------
5XNwZF4qoCKTO8M8KUjRprQbivTkmI8H
4. under the directory src/main/resources/config add two properties file:
a. application.properties
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
spring.datasource.url=jdbc:mysql://xxx
spring.datasource.username=ENC(xxx)
spring.datasource.password=ENC(xxx)
mybatis.mapper-locations=classpath:*/mapper/*.xml
mybatis.type-aliases-package=com.xx.xxx.model
logging.level.com.xx.xxx: DEBUG
b. encrypted.properties
jasypt.encryptor.password=mykey
Use spring cloud config server
Define encrypt.key=MySecretKey
Post message to encrypt https://config-server/encrypt
Define password now like
app.password={cipher}encryptedvalue
Use #Value("${app.password}") in code
and spring boot should give you decrypted value

Resources