Heroku DATABASE_URL as a JDBC Url for Maven - maven

My apps on Heroku use a DATABASE_URL. This is simple to parse with Java into a JDBC URL with a user name and password. There's no issue there. However, I have a JOOQ generator and Flyway migrator that have maven plugins and I can't figure out how to get the JDBC URL, User Name, and Password that these plugins require into maven. So currently I do it on app startup which is not ideal. When my app starts I get the DATABASE_URL, parse it, then do the flyway migration and jOOQ code generation. But I would like this to happen during the actual build process, not during application startup.
Basically I need the Heroku Environment variable formatted like (postgres://user:pass#ec2-host:1234/path-to-db) to be accessible in maven as a property like this (jdbc:postgresql://ec2-host:1234?user=user&password=pass).
I think the solution may lie with the maven build helper plugin but I can't get the regex properties specification quite right.
Thanks

I have solved this thanks to heroku support. Lukas and Axel you may want to document this for any of your users that are using heroku, want to run your tools in their build (vs code startup), and do not want to hand maintain different environment variables.
We will be using the GMavin Plus Plugin to run some code to parse the DATABASE_URL environment variable into properties prior to running the flyway migration or jOOQ code generation.
First off you need to add Groovy as a dependency:
<dependency>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy-all</artifactId>
<version>2.3.9</version>
<scope>compile</scope>
</dependency>
Then the GMaven plus plugin:
<plugin>
<groupId>org.codehaus.gmavenplus</groupId>
<artifactId>gmavenplus-plugin</artifactId>
<version>1.2</version>
<executions>
<execution>
<phase>initialize</phase>
<goals>
<goal>execute</goal>
</goals>
</execution>
</executions>
<configuration>
<scripts>
<script><![CDATA[
URI dbUri = new URI(System.getenv("DATABASE_URL"));
String username = dbUri.getUserInfo().split(":")[0];
String password = dbUri.getUserInfo().split(":")[1];
int port = dbUri.getPort();
String dbUrl = "jdbc:postgresql://" + dbUri.getHost() + ":" + port + dbUri.getPath();
project.properties['database.jdbcUrl']=dbUrl
project.properties['database.username']=username
project.properties['database.password']=password
]]></script>
</scripts>
</configuration>
</plugin>
Now you can use ${database.jdbcUrl}, ${database.username}, and ${database.password} in your flyway, jOOQ, or whatever else configurations.

There are probably myriad ways to solve this with Maven, but one option is to keep a single properties file in your /src/main/resources path, load it using the properties-maven-plugin (an example can be seen in the jOOQ-Spring example), and then compose URLs using the individual parts, e.g. jOOQ and Flyway:
<url>jdbc:postgresql://${db.host}:${db.port}/${db.database}</url>
<user>${db.username}</user>
<password>${db.password}</password>
Heroku:
<database_url>postgres://${db.username}:${db.password}#${db.host}:${db.port}/${db.database}</database_url>

Related

Does flyway support sql migrations under test folder?

I am migrating a project from Spring Boot 1.5.21 to 2.2.5. I have followed all instructions in the guide. However, I have an issue with flyway (upgrading from 4.2.0 to 6).
Previously, when I had sql migration files under src/test/resources/db/migration, flyway would run them as part of the mvn clean install command. Now, for some reason, it stopped running these migrations (Just to clarify, I'm talking about maven build and not while running the app).
I'm using maven 3.6.3 and flyway-maven-plugin version 6.0.8 with the following configuration (some values are tokens which are irrelevant for this question):
<plugin>
<groupId>org.flywaydb</groupId>
<artifactId>flyway-maven-plugin</artifactId>
<version>${flyway.version}</version>
<executions>
<execution>
<phase>generate-sources</phase>
<goals>
<goal>migrate</goal>
</goals>
</execution>
</executions>
<configuration>
<skip>${db.skip}</skip>
<url>${db.url}</url>
<user>${db.username}</user>
<password>${db.password}</password>
<locations>
<location>classpath:db/migration</location>
</locations>
<schemas>public,downstream</schemas>
<outOfOrder>true</outOfOrder>
<callbacks>
db.migration.callback.PopulateControlFieldsFlywayCallback,db.migration.callback.UpdateReplicaIdentityFlywayCallback,db.migration.callback.UpdateSchemaHistoryTableFlywayCallback
</callbacks>
</configuration>
<dependencies>
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<version>${postgresql.version}</version>
</dependency>
</dependencies>
</plugin>
Why did the behaviour changed? How to revert to old behaviour? Am I missing something?
I have dug in the flyway and flyway-maven-plugin sorces. I have found that the path from classpath location tag is searched by ClassRealm in class. And there is exist two classpathes: ClassRealm[plugin>org.flywaydb:flyway-maven-plugin:5.0.0, parent: jdk.internal.loader.ClassLoaders$AppClassLoader#1f89ab83]. ClassRealm is a classloader that is used in Maven.
I have executed classLoader.getResources('/db/migration')and found that it returns 2 paths (target/classes/db/migration and target/test-classes/db/migration) for mave-plugin version 4.2.0, but returns only 1 path (target/classes/db/migration) for version5.0.0`.
Also I have found that there was a commit in flyway-maven-plugin repository for using maven 3 instead of 2 between maven-plugin version 4.2.0 and 5.0.0.
I haven't found an exact reason, but, summarizing all, I guess that there were some changes in maven-plugin (I guess some maven core library version) as a result of which the finding path in classpath was changed.
I have tried several options for returning the old behavior, but all of these not so good. These are work:
Stay on 4.2.0 maven-plugin version (I have a pet project where
I specified flyway 5.0.0 in <dependencies> and use maven-plugin
4.2.0 and it works correctly for you);
Specify a path to test-migration via <location>filesystem:target/test-classes/db/migration</location> or <location>filesystem:src/test/resources/db/migration</location>
I hope it will help you.
P.S. By the way, why do you use migration under test folder for generating classes by jooq? It looks very strange to me. You generate classes from one DB model, but in PROD you will have another DB model (since migrations only under main are used). Maybe you should think about it and using only migration under main folder? If you have some exceptions while jooq generatinig, try to fix it without test migrations.

Spring Contract Maven Plugin pushStubsToScm does not work

I tried to use Spring Contract Maven Plugin in a producer side to upload the stubs jar and share it with consumers.
I am using Spring Cloud Contract 2.0.0.
I configured the spring-cloud-contract-maven-plugin in my project, the codes is uploaded to Github.
<plugin>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-contract-maven-plugin</artifactId>
<version>${spring-cloud-contract.version}</version>
<extensions>true</extensions>
<configuration>
<basePackageForTests>com.example.demo</basePackageForTests>
<baseClassMappings>
<baseClassMapping>
<contractPackageRegex>.*rest.*</contractPackageRegex>
<baseClassFQN>com.example.demo.RestVerifierBase</baseClassFQN>
</baseClassMapping>
</baseClassMappings>
<!-- We want to pick contracts from a Git repository -->
<!--<contractsRepositoryUrl>git://file://${project.basedir}/target/contract_git/</contractsRepositoryUrl>-->
<!-- Example of URL via git protocol -->
<!--<contractsRepositoryUrl>git://git#github.com:spring-cloud-samples/spring-cloud-contract-samples.git</contractsRepositoryUrl>-->
<!-- Example of URL via http protocol -->
<!--<contractsRepositoryUrl>git://https://github.com/spring-cloud-samples/spring-cloud-contract-samples.git</contractsRepositoryUrl>-->
<contractsRepositoryUrl>git://https://github.com/hantsy/contracts-git.git</contractsRepositoryUrl>
<!-- We reuse the contract dependency section to set up the path
to the folder that contains the contract definitions. In our case the
path will be /groupId/artifactId/version/contracts -->
<contractDependency>
<groupId>${project.groupId}</groupId>
<artifactId>${project.artifactId}</artifactId>
<version>${project.version}</version>
</contractDependency>
<!-- The mode can't be classpath -->
<contractsMode>REMOTE</contractsMode>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<!-- By default we will not push the stubs back to SCM,
you have to explicitly add it as a goal -->
<goal>pushStubsToScm</goal>
</goals>
</execution>
</executions>
</plugin>
When I ran command mvn clean install -DskipTests -DcontractsRepositoryUsername=hantsy -DcontractsRepositoryPassword=mypassword and got the failure.
Caused by: org.eclipse.jgit.errors.TransportException: https://github.com/hantsy/contracts-git.git: Authentication is required but no CredentialsProvider has been registered
Update: If I set contractsMode to LOCAL, it will fail with downloading stubs jar error, maybe similar with this issue, in the new version 2.0.0, Spring Cloud contracts can not resolve jars from Local maven repos as expected when stubsMode is set to LOCAL.
Update 2: I also tried to add contractsRepositoryUsername and contractsRepositoryPassword in spring contract maven plugin config, it does not work.
contractsMode needs to be REMOTE. I've taken your example and it works well with:
<contractsRepositoryUrl>git://git#github.com:marcingrzejszczak/contracts-git.git</contractsRepositoryUrl>
<contractsMode>REMOTE</contractsMode>
with passing the credentials manually I see sth absolutely bizarre...
I see [INFO] Passed username and password - will set a custom credentials provider which means that username and password got passed and I'm using them. But actually what I get is Authentication is required but no CredentialsProvider has been registered, which makes no sense. Can you file another bug for that? The workaround is to use the agent.
UPDATE:
The issue got fixed here https://github.com/spring-cloud/spring-cloud-contract/issues/678

WSO2, Birt 4.4.1, and Maven

I am deploying a web services on WSO2 Application Server 5.2.1 which loads a reportDesign from a database table and renders a PDF stream, which is then stored as a file in an outgoing ftp server directory. All works well when running from the command line, but I am having a hard time configuring the EngineConfig's EngineHome property in when deploying on WSO2 AS 5.2.1.
My pom.xml file (for Maven) embeds Birt 4.4.1 runtime in my aar file, which is then deployed as part of a larger car file. Birt jar files are correctly picked up by my web service.
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.8</version>
<executions>
<execution>
<id>copy</id>
<phase>package</phase>
<goals>
<goal>copy</goal>
</goals>
<configuration>
....
<artifactItem>
<groupId>org.eclipse.birt.runtime</groupId>
<artifactId>org.eclipse.birt.runtime</artifactId>
<version>4.4.1</version>
<type>jar</type>
<overWrite>false</overWrite>
<outputDirectory>${project.build.directory}/aar/lib</outputDirectory>
</artifactItem>
...
</configuration>
</execution>
</executions>
</plugin>
...
</plugins>
</build>
<dependencies>
...
<dependency>
<groupId>org.eclipse.birt.runtime</groupId>
<artifactId>org.eclipse.birt.runtime</artifactId>
<version>4.4.1</version>
</dependency>
...
</dependencies>
A snippet of my Java code is as follows, which has been stripped down for simplicity:
InputStream theReportDesign = getReportDesignStream();
// The engine will eventually be called just once...
EngineConfig config = new EngineConfig();
config.setEngineHome("");
// Create the report engine itself. This engine can be used to run
// multiple reports.
ReportEngine engine = new ReportEngine(config);
// Create a task to run the report and convert the output to PDF.
IReportRunnable report = engine.openReportDesign(theReportName, theReportDesign);
IRunAndRenderTask task = engine.createRunAndRenderTask(report);
outputStream = new ByteArrayOutputStream();
// Define the report generation options
HTMLRenderOption options = new HTMLRenderOption();
options.setOutputFormat(HTMLRenderOption.OUTPUT_FORMAT_PDF);
options.setOutputStream(outputStream);
task.setRenderOption(options);
task.setErrorHandlingOption(IEngineTask.CONTINUE_ON_ERROR);
// set the parameters to appear in the report
task.setParameterValues(values);
// Generate the report.
task.run();
// Write the report to disk, so ftp clients can pick it up
outputStream.close();
writePdfReportToFile(outputStream);
The error I get when running this from within WSO2 Application Server is as follows:
Can not load the report engine
java.lang.NullPointerException
at org.eclipse.birt.report.engine.api.ReportEngine.openReportDesign(ReportEngine.java:182)
I have seen many proposed solutions on the web, but most within the Tomcat/Axis2 environment (which is what WSO2 uses) require knowledge of the ServletContext. My attempts to get the ServletContext off of the MessageContext have not been successful.
I am hoping to not have to deploy Birt into WSO2, but to instead keep the Birt Runtime contained within my aar file which I deploy to WSO2. But I'll resort to deploying Birt directly to the WSO2 server, as well (just need instructions to do that and still make the above code work).
Thanks in advance for any help.
This line should not be necessary because the BIRT runtime is embedded in your bundle, try to comment it:
config.setEngineHome("");
Furthermore usually we need to start the BIRT platform before creating an engine, check this link. I know you said the java snippet has been stripped, but we don't have any clue to guess if you forgot this or not:
Platform.startup( config );
IReportEngineFactory factory = (IReportEngineFactory) Platform.createFactoryObject(IReportEngineFactory.EXTENSION_REPORT_ENGINE_FACTORY );
However it is strange you don't get any exception when the engine is created, may be it is actually wrapped in a try..catch?
Ok, figured it out.
First of all, it is acceptable for EngineHome and BirtHome to be null if the runtime engine is already in your path.
As such, my issue is that when building with maven, it is not sufficient just define the the birt runtime in your pom.xml. There are a number of jar files which are not automatically included in the build and are otherwise not found in the WSO2 environment.
To this end, I've thus far added tidy to my pom.xml file, and added the following jar files to my WSO2_HOME/repository/components/dropins folder:
com.lowagie.text_2.1.7.v201004222200.jar
icu4j-54_1_1.jar
mysql_connector_java_5.1.26_bin_1.0.0.jar
org.apache.batik.bridge_1.6.0.v201011041432.jar
org.apache.batik.css_1.6.0.v201011041432.jar
org.apache.batik.dom.svg_1.6.0.v201011041432.jar
org.apache.batik.dom_1.6.0.v201011041432.jar
org.apache.batik.ext.awt_1.6.0.v201011041432.jar
org.apache.batik.parser_1.6.0.v201011041432.jar
org.apache.batik.pdf_1.6.0.v201105071520.jar
org.apache.batik.svggen_1.6.0.v201011041432.jar
org.apache.batik.transcoder_1.6.0.v201011041432.jar
org.apache.batik.util.gui_1.6.0.v201011041432.jar
org.apache.batik.util_1.6.0.v201011041432.jar
org.apache.batik.xml_1.6.0.v201011041432.jar
org.eclipse.datatools.connectivity.oda_3.4.3.v201405301249.jar
org.w3c.css.sac_1.3.0.v200805290154.jar
org.w3c.dom.smil_1.0.0.v200806040011.jar
org.w3c.dom.svg_1.1.0.v201011041433.jar
org.wso2.carbon.logging.propfile_1.0.0.jar
I've also found it necessary to put the following line of code prior to the Platform.startup(config) statement:
// The following line is required in wso2:
// http://stackoverflow.com/questions/13241045/why-doesnt-websphere-like-birt
RegistryProviderFactory.releaseDefault();
Platform.startup(config);
I am now successfully generating reports in the WSO2 environment. With that said, I am sure of the following:
Additional missing jar files will likely be found as I build out more reports
Some of the jar files I manually added to the wso2 dropins folder may be replaced with additional pom.xml dependencies
So more trial and error to come. But for now, am successfully generating reports in WSO2.

Why is my ${user.home} variable resoled at build time

I have a JHipster generated application with an YAML property file that looks like this:
storage:
location: ${user.home}/my/folder
My problem is that the variable ${user.home} is resolved at build time, when I run mvn package (on Jenkins). So the property is already resolved in the resulting artifact, hence when I deploy on on my server, that path contains the resolved home of the user Jenkins.
Anybody know who is doing this and why? I was expecting that the variable would be resolved at runtime.
Thanks.
Valentin
I'm not totally sure of how JHipster builds on top of Spring Boot, but my guess would be that it's Maven's resource filtering that's expanding ${user.home} at build time. It's enabled by default by spring-boot-starter-parent for application.properties and application.yaml in src/main/resources.
This Spring Boot issue contains some more information, along with details of a configuration change that you may like to make so that ${…} entries are no longer filtered:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-resources-plugin</artifactId>
<version>2.6</version>
<configuration>
<encoding>UTF-8</encoding>
<delimiters>
<delimiter>#</delimiter>
</delimiters>
<useDefaultDelimiters>false</useDefaultDelimiters>
</configuration>
</plugin>

Configure Maven to use CXF wsdl2java with Basic Authentication

I have an application that needs to integrate with one of SharePoint's web services. This web service cannot be accessed freely and needs authentication.
As such, the standard wsdl2java Maven plugin in my application gives an HTTP 401 error when the generate-sources phase is executed.
Is there a way to setup Maven/POM so that I can provide a user/password that will generate the stubs?
I have come across some answers saying this is not possible but all answers are older than 1 year. I haven't found if Maven have issued an update on this. One option is to save a local copy of the WSDL (as suggested here) but I would like to avoid having local copies.
Because you mentioned CXF then I suppose you meant cxf-codegen-plugin. It's a bit of a hack but it works.
HTTP authentication credentials can be provided using java.net.Authenticator. One need to just define his own Authenticator class which overrides getPasswordAuthentication(..) method. Then it has to be set as default Authenticator. As far as I know it can't be done declaratively (for instance using environment properties) only programatically using Authenticator.setDefault(..).
In order to call Authenticator.setDefault(..) I would use CXF extension mechanism. Create separate maven project with similar class:
public class AuthenticatorReplacer {
public AuthenticatorReplacer(Bus bus) {
java.net.Authenticator.setDefault(new java.net.Authenticator() {
#Override
protected PasswordAuthentication getPasswordAuthentication() {
return new PasswordAuthentication("test", "test123"
.toCharArray());
}
});
}
}
and file src\main\resources\META-INF\cxf\bus-extensions.txt with contents:
org.example.AuthenticatorReplacer::false
Then add newly created project as a dependency to cxf-codegen-plugin:
<plugin>
<groupId>org.apache.cxf</groupId>
<artifactId>cxf-codegen-plugin</artifactId>
<version>${project.version}</version>
<dependencies>
<dependency>
<groupId>org.example</groupId>
<artifactId>cxf-authenticator-replacer</artifactId>
<version>0.0.1-SNAPSHOT</version>
</dependency>
</dependencies>
...
</plugin>
This way AuthenticatorReplacer is initialized by CXF extension mechanism and replaces default Authenticator with ours.
An clean alternative to #Dawid Pytel's solution would be to run this class during lifecycle of wsdl class auto generation:
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>1.4.0</version>
<executions>
<execution>
<phase>generate-sources</phase>
<goals>
<goal>java</goal>
</goals>
</execution>
</executions>
<configuration>
<mainClass>path.to.AuthenticatorReplacer</mainClass>
</configuration>
</plugin>
Important: your AuthenticatorReplacer has to be a main(String[] args) class and running the code inside.
I verified that Dawid's solution works. Alternatively, you can use SoapUI to pull down and cache the wsdl and then use SoapUi code generation support to use cxf to generate the code.
http://java.dzone.com/tips/generating-client-java-code
Dawid's solution works for me too. It is a little tricky though. In Eclipse, the pom.xml keeps complaining that "wsdl2java failed: Could not load extension class AuthenticatorReplacer". You have to ignore this error message and use the command line:
mvn generate-sources
The Java classes will then be generated successfully.

Resources