What is the best approach to have my maven package-deploy project write to a properties file inside of a WAR file?
I currently have three separate maven projects that create their own packages:
a.war, b.zip, and c.tar.gz
Inside of the WAR file (a.war), there is a properties file that contains the following:
buildDate=2018-01-25 16:11:49 PST
aUiNumber=2.1.0-SNAPSHOT.5
buildNumber=2.1.0-SNAPSHOT.${deploy.number}
The file is located here (inside of the WAR file):
WEB-INF/classes/a-version.properties
On a Jenkins server, I have a job that uses maven to do the following:
Pull the latest a.war, b.zip, c.tar.gz from nexus
Package these into app-assets.zip
Deploy app-assets.zip
I would like to have this maven job populate the ${deploy.number} in the a-version.properties file with the Jenkins job number. What is the best approach for this? Is there a way to do it without unpacking the WAR file?
I attempted this by adding the a.war/WEB-INF/classes to the <directory> section of the war file. As expected, the build did not fail; however, it did not populate the variable as well:
mvn -U -f ./PackageDeployPom.xml resources:resources -Ddeploy.number=${BUILD_NUMBER}
[INFO] skip non existing resourceDirectory /home/jenkins/app-assets/apache-tomcat-8.0.41/webapps/a.war/WEB-INF/classes
Not sure if this is the best way to do this but here is how I did it:
I used the exec-maven-plugin.
The maven project now follows this procedure:
Pull the latest a.war, b.zip and c.tar from nexus (place in to-be-packaged-directory).
Use exec-maven-plugin to call a bash script. This script will:
i. Copy the a.war file to a temporary workspace directory, unpack it here.
ii. Populate the variables in temp-workspace/WEB-INF/classes/a-version.properties. (Using sed).
iii. Use jar to update the file in the war file.
Package the app-assets.zip
Deploy app-assets.zip
Here is what I added to the pom file for the packaging job:
<plugin>
<artifactId>exec-maven-plugin</artifactId>
<version>1.6.0</version>
<groupId>org.codehaus.mojo</groupId>
<executions>
<execution><!-- For Adding deploy Number -->
<id>Renaming build artifacts</id>
<phase>package</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<executable>bash</executable>
<commandlineArgs>scripts/BuildNumber.sh -b ${deploy.number}</commandlineArgs>
</configuration>
</execution>
</executions>
</plugin>
Here is the working portion of the Bash script that gets executed with by the maven plugin:
#Cleaning the TempWorkspace
TempWs=${Workspace}/a-war-temp;
if [[ ! -d ${TempWs} ]];then
mkdir ${TempWs};
else
rm -r ${TempWs}/;
mkdir ${TempWs};
fi
#--- end cleaning temp workspace ---
#Copying the war file to the temp workspace
cp ${Workspace}/app-assets/${ApacheTomcat}/webapps/a.war ${TempWs}/a-old.war;
#Unpacking the war file, using sed to make changes to variable(s)
cd ${TempWs}
jar xvf aw-old.war;
sed -i "s/\${deploy.number}/${BuildNumber}/g" ${TempWs}/WEB-INF/classes/am-version.properties
cd ${Workspace}; #Going back to the Workspace
#--- end populating variable(s) ---
#Updating the war file with the new properties file
jar -uf ${Workspace}/aw-emr/${ApacheTomcat}/webapps/aw.war -C ${TempWs} WEB-INF/classes/am-version.properties
Maven is run with this command:
mvn -U -B -Ddeploy.number=${BUILD_NUMBER} -f ./App-Asset-deploy.xml clean package
Related
Testing the cloud-build
Part of my cloudbuild.yaml
- name: 'gcr.io/cloud-builders/mvn'
args: ['dockerfile:build']
dockerfile:build perfectly works in bitbucket pipeline, no problem. I use
<plugin>
<groupId>com.spotify</groupId>
<artifactId>dockerfile-maven-plugin</artifactId>
<version>${dockerfile-maven-version}</version>
<executions>
<execution>
<id>default</id>
<goals>
<goal>build</goal>
<goal>push</goal>
</goals>
</execution>
</executions>
<configuration>
<repository>gcr.io/my-project-id/${project.artifactId}</repository>
<tag>${project.version}</tag>
<buildArgs>
<JAR_FILE>${project.build.finalName}.jar</JAR_FILE>
</buildArgs>
</configuration>
</plugin>
But with the cloud-build for this single step I get the error:
[INFO] Step 14/15 : ARG JAR_FILE
[INFO]
[INFO] ---> Using cache
[INFO] ---> 55793de4bb9f
[INFO] [INFO] Step 15/15 : ADD target/${JAR_FILE} /usr/share/$SERVCE_FOLDER_NAME/app.jar
[INFO]
[ERROR] ADD failed: stat /mnt/cache/docker/tmp/docker-builder589658449/target/myappname-0.0.1-SNAPSHOT.jar: no such file or directory
(the JAR_FILE is passed from the maven dockerfile plugin
no such file or directory
Why?.. In the end of the day I juse call dockerfile:build and expect it to be the same as it is when I build it from another pipeline.
My Dockerfile:
FROM openjdk:8-jdk
ENV GOOGLE_APPLICATION_CREDENTIALS=/app/credentials.json
ARG ACTIVE_PROFILES=dev
ENV ACTIVE_PROFILES=$ACTIVE_PROFILES
ARG CREDENTIALS
ARG SERVCE_FOLDER_NAME=myappname-service
ENV SERVCE_FOLDER_NAME=$SERVCE_FOLDER_NAME
#ENTRYPOINT ["/usr/bin/java", "-jar", "/usr/share/$SERVCE_FOLDER_NAME/app.jar"]
ENTRYPOINT ["./entrypoint.sh" ]
WORKDIR /app
EXPOSE 8080
COPY ./.gcloud/credentials.json credentials.json
COPY entrypoint.sh .
#Add Maven dependencies (not shaded into the artifact; Docker-cached)
#ADD target/lib /usr/share/$SERVCE_FOLDER_NAME/lib
ARG JAR_FILE
ADD target/${JAR_FILE} /usr/share/$SERVCE_FOLDER_NAME/app.jar
EntryPoint script is (that is what is mentioned on step 15/15 in the log):
java -Djava.security.egd=file:/dev/./urandom -jar /usr/share/$SERVCE_FOLDER_NAME/app.jar --spring.profiles.active=$ACTIVE_PROFILES
(I did try to pass hard-coded values to $SERVCE_FOLDER_NAME, $ACTIVE_PROFILES - same [it works in bitbucket pipeline])
A few things come to mind,
how are you triggering the builds?
manually with gcloud or api? or automatically with build triggers or the github app?
it seems that the target/ directory might not be present in the remote workspace-- are you ignoring target/ or .jar files anywhere?
the remote workspace might not be getting the target/ directory or .jar files if they are in your .gitignore or .gcloudignore
try making an empty .gcloudignore or temporarily removing target/ and .jar files from .gitignore
relevant links: https://cloud.google.com/sdk/gcloud/reference/topic/gcloudignore, https://github.com/GoogleCloudPlatform/cloud-builders/issues/40
have you tried debugging with cloud-build-local? it allows you to write and explore the workspace locally
https://cloud.google.com/cloud-build/docs/build-debug-locally
https://github.com/GoogleCloudPlatform/cloud-build-local
I have a Talend studio Enterprise veresion 6.4.1. I have downloaded the CI builder and installed with maven using the below command.
mvn install:install-file -Dfile=ci.builder-6.4.1.jar -DpomFile=ci.builder-6.4.1.pom
I have exported the job from Talend studio put into the Git. It contains the following files.
1) process a) .items b) .properties c).screenshot
2) talend.project
I have created a pom.xml with the below details
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.talend</groupId>
<artifactId>buildsources</artifactId>
<version>0.1</version>
<packaging>pom</packaging>
<build>
<plugins>
<plugin>
<groupId>org.talend</groupId>
<artifactId>ci.builder</artifactId>
<version>6.4.1</version>
<executions>
<execution>
<phase>validate</phase>
<goals>
<goal>generate</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
I have placed pom.xml in the root folder of commandline workspace.
I placed the checked out folder inside command line workspace .
After that I executed the commandline.bat inside Talend studio to start the Talend.
I tried executing the below command to generate the source file.
mvn org.talend:ci.builder:6.4.1:generate -X -Dcommandline.workspace="D:\ci-builder\commandline_workspace\TEST" -Dcommandline.host=localhost -Dcommandline.port=8002 -Dcommandline.user=dummy#talend.com -DprojectsTargetDirectory=D:\buildsources\
Now it is able to find the project but at the end it says no job
[INFO] Preparing for the project: TEST
[INFO] Logon project
[INFO] Generating sources...
[INFO] Logoff project
[WARNING] There's no job in current project!
I am using Windows machine to test this - Windows 10 64 bit.
Maven version is 3.5.3
I have been referred the below links for doing this.
https://community.talend.com/t5/Architecture-Best-Practices-and/Continuous-Integration-with-Talend-CI-Builder-6-3/ta-p/33033
https://akhileshthipparthi.wordpress.com/2017/01/23/talend-ci-using-jenkins/
But when I check the .Java folder in my Talend installation directory command line workspace, I can see some files generated. pom.xml and pom_routines.xml. But there are no job related files getting generated.
And with those pom.xml , if I do a mvn clean install, it is generating a routines.jar file only in the target directory.
I am not able to proceed further.
Any pointers will be helpful.
One thing I wanted to highlight is that, for talend CI builder to work there is no need to writing test cases.
I was able to build the job using the CI builder using the below steps.
Inside the command line work space, you need to keep the complete project structure.
Checkout the source files from Git and copy the files to the project folder.
All the source files will be present inside the process folder of the project folder.
Setup the maven repository for Talend command line. It will be inside the configuration folder inside the Talend folder.
Execute the below command to Generate the source code inside the command line workspace.
Here SayHello is the Job name.
mvn org.talend:ci.builder:6.4.1:generate -Dcommandline.workspace="D:\ci-builder\commandline_workspace\" -Dcommandline.host=localhost -Dcommandline.port=8002 -Dcommandline.user=dummy#talend.com -DprojectsTargetDirectory=D:\ci-builder\target -DitemFilter=(type=process)and(label%SayHello*)
if there is a parent child job, we need to place all the jobs inside a folder and we need to refer that path during generate source.
Here Sample is the folder name
mvn org.talend:ci.builder:6.4.1:generate -Dcommandline.workspace="D:\ci-builder\commandline_workspace\" -Dcommandline.host=localhost -Dcommandline.port=8002 -Dcommandline.user=dummy#talend.com -DprojectsTargetDirectory=D:\ci-builder\target -DitemFilter=(type=process)and(Path%Sample*)
It will generate the source files and it will be kept in "D:\ci-builder\target"
Now navigate to that path and do mvn clean package
The job will be built and it will be placed inside the target folder. It will contain all the related Jar's, shell script, batch script and libraries and routines. Also it will have the entire folder structure.
Note : If you get the message like, there is no job in folder, just check if there is any error in the talend command line shell. Mostly u can encounter, the .Java folder is missing in command line workspace or there will be dependency errors (missing Jars in the maven local repo).
As mentioned in the prerequisites section of the Talend documentation you linked, your jobs must have a test case
Talend Jobs
Jobs containing test cases for Continuous Testing
Otherwise they won't be detected.
Please create a test case for your job, and try again.
So I have the following tar.gz file in my repo with structure as:
> A.tar.gz
> |
> |____ a.tar.gz
> |
> |____ b.tar.gz
> |
> |_____ folderA
> |
> |_____ folderB
> |
> |______ jar1.jar
> |
> |______ jar2.jar
Now in my POM file for another project I would like to add the jar1 and jar2 as dependencies. So far I have the following:
<dependency>
<groupId>com.groupid</groupId>
<artifactId>master</artifactId>
<version>18.1</version>
<type>tar.gz</type>
<classifier>bin</classifier>
</dependency>
This made the tar file available. I then tried to unpack as:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<phase>generate-resources</phase>
<goals>
<goal>unpack-dependencies</goal>
</goals>
<configuration>
<includeTypes>tar.gz</includeTypes>
<includeArtifactIds>master</includeArtifactIds>
<outputDirectory>target/somefolder</outputDirectory>
</configuration>
</execution>
</executions>
</plugin>
However, on running the build, I still don't get the jars as dependencies. I am sure I am missing something , so any help appreciated!
So you have JAR dependencies located inside another tar.gz dependency that you want in your project. So far so good, the problem is you're trying to:
Make the the first .tar.gz dependency available (OK)
Launch the build to unpack the jars (OK)
Add unpacked jar as dependencies during the same build (Not possible)
All in one run. That is not possible in Maven by design (to the best of my knowledge). Even if you did find a way to do it this way, that would over-complexify your build and break Maven design, probably leading to other issues.
You mentioned that you don't control the packaging of the other team and it seems you can't upload said dependencies on a Nexus repo either. What you can do is make the JAR dependencies available in your local repo prior to running your build by doing something like:
Download your tar.gz file and unpack it
Install the jar dependencies in your local Maven repo using commands like mvn install:install-file -Dfile=path/to/jar1.jar -DgroupId=com.mycompany -DartifactId=jar1 -Dversion=1.0.0 -Dpackaging=jar with proper version, groupdId and artifactId for each jars (see the Guide to installing 3rd party JARs for more details)
Now you can run your your original project only by mentioning your jar dependencies as <dependency>
With this you will manually install your jar dependencies in your Maven local repository, rendering them available to your project without needing a Nexus repository or further unpackaging. You can perform step 1 and 2 manually or by creating another Maven project that you will have to run once before running your main project. To do so you can create a new project and use the maven-dependency-plugin as you already did coupled with the maven-exec-plugin to run the mvn install:install-file command.
Note this process must be done for every individual machine on which you will run your project. As #khmarbaise mentioned, it's best to have your dependencies available directly through a repository manager such as Nexus without having to perform additional steps, but this temporary workaround should work just fine.
I am regularly deploying a Maven project to a Tomcat server, using Travis CI. My project is a web app, so I have configured my pom.xml for building a WAR file, instead of a JAR:
...
<packaging>war</packaging>
...
With Maven, I can generate a directory containing all the JavaDoc files for my project; Maven puts them in the target/site/apidocs directory. But then, when I deploy my project, Travis doesn't perform any mvn site phase so I don't have my JavaDocs on the server.
Should I edit my pom.xml so that Maven puts the JavaDoc files somewhere in the src directory (instead of target) or is there a way to package the JavaDoc files together with the WAR file? I thought that I could create a docs/ directory inside src/main/webapp/. Specifically: is it "good practice" to generate my JavaDoc in src instead of target? if not, how can I have a WAR file containing my JavaDoc?
What would you suggest is the best thing to do?
I already know how to generate a standalone JAR containing my JavaDoc files (see here), but this is not what I'm looking for.
Use the site plugin https://maven.apache.org/plugins/maven-site-plugin/ and the javdoc plugin https://maven.apache.org/plugins/maven-javadoc-plugin/usage.html.
Add the following to your pom.xml
<reporting>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<configuration>
<links>
<link>http://commons.apache.org/lang/api</link>
<link>http://java.sun.com/j2se/1.5.0/docs/api</link>
<link>http://this-one-will-not-work</link>
</links>
</configuration>
</plugin>
</plugins>
</reporting>
then mvn site:site your documentation will be in target/site you can also deploy it.
I have a project that uses a JAR with no maven repo.
I made this by myself.
Before build my project, I do this on my console:
mvn install:install-file -Dfile=myownjar-1.5.jar -DgroupId=com.cmabreu -DartifactId=mylocal-lib -Dversion=1.5 -D packaging=jar -DgeneratePom=true
and add the JAR to my maven repo (local).
Then I add the required dependency tag to my POM file and build my project.
But, when I commit to Github, I do not send my custom JAR (is another project).
The question is: how can I tell Travis-CI to build my project using this custom JAR in its repository without send it to Github?
Not a recommended solution but a very useful workaround:
Make a directory inside project's home. Let's call it
$projectBasseDir/lib
Put all your external jars in this folder.
In the pom file add scope and systemPath as follows for your dependency:
< scope>system< /scope >
< systemPath>${project.basedir}/lib/yourJar.jar< /systemPath>
Push this lib/ directory to your project repo on github
Travis builds work fine with this
If the jars are not present locally, then we need to add a before_start script to your repo which basically does this:
mkdir lib/
wget -P "lib/" http://urlForYourJar.jar
and it works great again.
I had a very similar problem and after some research, I did the following.
First of all, add before_install command such as
before_install:
- wget -P somewhere/ http://some.url/awesome.jar
- mvn validate
The line validate is very important, since we need to explicitly install the third-party jar via the maven-install plugin.
The second step is to modify your pom.xml file as follows:
<build>
....
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-install-plugin</artifactId>
<version>2.5.1</version>
<configuration>
<groupId>your.group</groupId>
<artifactId>your_artifact</artifactId>
<version>some.version</version>
<packaging>jar</packaging>
<file>${basedir}/match_this_with_wget/awesome.jar</file>
</configuration>
<executions>
<execution>
<id>install-jar-lib</id>
<goals>
<goal>install-file</goal>
<goals>
<phase>validate</validate>
</execution>
</executions>
</plugin>
....
In your local repository, you can put the jar file in the desired place and use .gitignore to avoid committing it into the git repo.
It is also very easy to replace the wget command with other downloading commands such as git clone. However, you may need to put the jar file within your repository if you install it with maven-install. When you are executing the before_install commands on travis-ci, your pwd should be right at your repo root.
If your company has private repo (Artifactory/Nexus) you may want to deploy it there (deploy:deploy-file). This would be one time manual step. Than you wouldn't need to install it into local repo on every build.
If it's not your case, there is no way how to install it into local repo without checking it into your source control.
You can get Travis to run a custom build script. If you can wget the JAR or if you've checked it into your repo, you can run that mvn install command before running your tests yourself.