Jenkins "Invoke Maven 3" build step does not expose POM_* variables - maven

I am using Jenkins as my CI server and Artifactory as my artifact repository. The Artifactory plugin works in free-style projects and can be used just by using Invoke Maven3 build step to both fetch the dependencies and deploy the artifacts. Furthermore, I have another build step for building a docker image and pushing it to my private docker registry using CloudBees Docker Build and Publish plugin.
The problem is that I want to use POM_* variables in CloudBees configurarion that should be exposed by Maven Plugin as mentioned here. But everytime I build the project, I got the following error:
ERROR: Unrecognized macro 'POM_VERSION' in '${POM_VERSION}'
org.jenkinsci.plugins.tokenmacro.MacroEvaluationException: Unrecognized macro 'POM_VERSION' in '${POM_VERSION}'
at org.jenkinsci.plugins.tokenmacro.TokenMacro.expand(TokenMacro.java:198)
at org.jenkinsci.plugins.tokenmacro.TokenMacro.expandAll(TokenMacro.java:233)
at org.jenkinsci.plugins.tokenmacro.TokenMacro.expandAll(TokenMacro.java:222)
at com.cloudbees.dockerpublish.DockerBuilder$Perform.expandAll(DockerBuilder.java:266)
at com.cloudbees.dockerpublish.DockerBuilder$Perform.getNameAndTag(DockerBuilder.java:277)
at com.cloudbees.dockerpublish.DockerBuilder$Perform.exec(DockerBuilder.java:247)
at com.cloudbees.dockerpublish.DockerBuilder$Perform.access$100(DockerBuilder.java:233)
at com.cloudbees.dockerpublish.DockerBuilder.perform(DockerBuilder.java:208)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779)
at hudson.model.Build$BuildExecution.build(Build.java:205)
at hudson.model.Build$BuildExecution.doRun(Build.java:162)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:537)
at hudson.model.Run.execute(Run.java:1744)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:374)
Build step 'Docker Build and Publish' marked build as failure
I do not know what exactly is the reason of this problem. Maybe the Invoke Maven3 build step is causing this problem by not exposing env variables. Another source of this could be CloudBees Plugin that does not utilize env variables. Any ideas?

The build step you mention, in a freestyle project, is not part of the Maven plugin. The environment variable described is only set for native Maven projects. From another project type, I suppose you would need to do something like
POM_VERSION=$(mvn help:evaluate -Dexpression=project.version | fgrep -v '[INFO]')
at the start of a script which continued to go on to use that variable.
Exposing the variable to subsequent non-shell build steps, such as from the CloudBees Docker Build and Publish plugin, is another matter; there may be some plugin that can do it, but I am not sure what offhand. (EnvInject perhaps.) It may be simpler to switch to a workflow job, using the CloudBees Docker Workflow plugin to drive Docker operations (depending on what exactly you are using from the Artifactory plugin):
node {
stage 'build'
git '…'
sh 'mvn package'
stage 'package'
sh 'mvn help:evaluate -Dexpression=project.version | fgrep -v '[INFO]' > .version'
def version = readFile '.version'
def image = docker.build("my-image:${version}")
stage 'publish'
docker.withRegistry(…) {
image.push()
}
}
By the way, you can add a shell build step just running
env | sort
if you are curious to see what environment variables are available.

Based on #Jesse Glick answer, searching around a little and struggling a bit with jenkins, I ended up with the following solution that works like a charm.
1-Add an "Execute Shell" build step containing following commands:
echo POM_VERSION=$(mvn help:evaluate -Dexpression=project.version | fgrep -v '[INFO]') > pom_file
echo POM_ARTIFACTID=$(mvn help:evaluate -Dexpression=project.groupId | fgrep -v '[INFO]') >> pom_file
echo POM_GROUPID=$(mvn help:evaluate -Dexpression=project.artifactId | fgrep -v '[INFO]') >> pom_file
2-Add an "Inject environment variables" build step (from EnvInject plugin) and enter "pom_file" as "Properties File Path"
3-Add a "Docker Build and Publish" build step and use POM_VERSION, POM_ARTIFACTID, POM_GROUPID in its configuration as you wish.

Related

CircleCI cannot find maven settings when supplied on CLI

[ERROR] Error executing Maven.
[ERROR] The specified user settings file does not exist: /home/circleci/project/ .folder/mvn-settings.xml
I have a script build-project.sh to build a mvn project.
This script takes the environment variable CUSTOM_MVN_OPTS, where I specify the path to the custom settings (e.g. private repo locations, etc.).
When I run the script in CircleCI or any CICD pipeline under docker env it's throwing above error.
#!/bin/sh
# build-project.sh
mvn ${CUSTOM_MVN_OPTS} package
What I expect? build-project.sh to build artifacts.
CUSTOM_MVN_OPTS="-s .folder/mvn-settings.xml" build-project.sh
After debugging found the solution.
Fix: hardcode the -s option within the script, you cannot pass this thru env variable.
#!/bin/sh
# build-project.sh
if [ -z "${CUSTOM_MVN_OPTS}" ]; then
mvn package
else
mvn -s ${CUSTOM_MVN_OPTS} package
fi

Can Datomic dev-local be installed on windows?

I'm trying to install Datomic on a Windows 10 computer, following the official instructions.
I downloaded and unzipped the dev tools as instructed.
I cannot, however, run the install script because it is a bash script.
I opened the script and discovered it requires maven, so I installed maven and tried to run the commands manually.
echo 'Installing: com.cognitect/rebl {:mvn/version "0.9.242"}'
mvn -q org.apache.maven.plugins:maven-install-plugin:3.0.0-M1:install-file -Dfile=rebl-0.9.242/rebl-0.9.242.jar
echo 'Installing: com.datomic/dev-local {:mvn/version "0.9.232"}'
mvn -q org.apache.maven.plugins:maven-install-plugin:3.0.0-M1:install-file -Dfile=dev-local-0.9.232/dev-local-0.9.232.jar
At first this errored with
The goal you specified requires a project to execute but there is no POM in this directory
So I figured out how to create a maven pom.xml.
Then it errors
[ERROR] The specified file 'C:\workspaces\clj-recipe\rebl-0' not exists
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-install-plugin:3.0.0-M1:install-file (default-cli) on project clj-recipe: The specified file 'C:\workspaces\clj-recipe\rebl-0' not exists
Is dev-local not intended for windows?
Update
I did get the maven scripts to run. I created my own install.ps1 in the dev tool directory, which kept paths the same, and quoted the file paths.
# expects to be run from the project (pom.xml) directory, but in a script file in the same directory as the original install script
echo 'Installing: com.cognitect/rebl {:mvn/version "0.9.242"}'
mvn -q org.apache.maven.plugins:maven-install-plugin:3.0.0-M1:install-file -Dfile="rebl-0.9.242/rebl-0.9.242.jar"
echo 'Installing: com.datomic/dev-local {:mvn/version "0.9.232"}'
mvn -q org.apache.maven.plugins:maven-install-plugin:3.0.0-M1:install-file -Dfile="dev-local-0.9.232/dev-local-0.9.232.jar"
I still can't get dev-local to run though. There appears to be no changes to the pom.xml.
I start up a repl for the current lein project and run
(require '[datomic.client.api :as d])
(def client (d/client {:server-type :dev-local
:system "dev"}))
Getting the error No such namespace: d.
My guess is that I don't understand how deps.edn works... Right now I have a single deps.edn under C:/Users/[username here]/documents/.clojure/deps.edn
{
:mvn/repos {"cognitect-dev-tools"
{:url "https://dev-tools.cognitect.com/maven/releases/"}}
:deps
{com.datomic/dev-local {:mvn/version "0.9.225"}}
}
There were two key issues here
The install script is not necessary with leiningen (and is not written for windows)
If you want to use a maven-based approach, then use the install script. Be warned that you need maven installed.
The install script can be tweaked for windows by changing as shown above (quote paths, remove the cd, make it a ps1 file)
The deps.edn, maven, and leiningen paths are not compatible. I must configure the dependency using leiningen to use it in my lein-based project
Configuring for lein is fairly simple
add a repositories configuration section
add a package dependency
(defproject ;;...
:dependencies [
;;...
[com.datomic/dev-local "0.9.225"]
]
:repositories [
["cognitect-dev-tools" {:url "https://dev-tools.cognitect.com/maven/releases/"
:username :env/datomic_username
:password :env/datomic_password}]]
;;...
)
Note that the credentials have to be supplied to the lein project. This can be done with
environment variables as shown above (specify the name as :env/var-name-here)
or using an encrypted password field
or use a profile

How can you deploy a specific artifact from Jenkins into Nexus?

I have a multi-module maven project running in Jenkins. I would like to deploy the final artifact (an RPM from an assembly build) to the Nexus server. I see no reason to deploy intermediate artifacts (hence no "mvn clean deploy") since this will produce extra junk on the server that I don't need. We're trying to set up a continuous delivery pipeline, so we're not deploying SNAPSHOT versions - ever. The various plugins for Jenkins seem focused on deploying all of the artifacts. How can I just deploy the one I choose?
EDIT:
After much consideration, I'd be willing to deploy all the artifacts to nexus if the deploy happens after all of the build has completed. If I was going that route, I'd want to use the "Deploy artifacts to maven repository" post build action. This tool seems broken though because it's missing the functionality of specifying the username/password. I know this can be specified in the settings.xml, but apparently only the one in ".m2". It doesn't seem to be looking at the settings.xml I specified for this build.
This also seems like it's broken for redeploying artifacts. There is some verbage in "Jenkins the Definitive Guide" for this, but it talks about a "local" settings.xml. All my builds are happening on Jenkins slaves, so this isn't an option and doesn't even really make sense with the Jenkins architecture.
If you need still generic way.
Use Execute shell option and use the mvn deploy command manually, You can pass your version and others things such as groupID etc as parameters in job, If you maintain separate job to build and upload this will work out.
Ex:
export M2_HOME=/PATH/TO/softwares/apache-maven-3.0.4
PATH=$M2_HOME/bin:$JAVA_HOME/bin:$PATH
export PATH
mvn -v
mvn deploy:deploy-file -Durl=http://someorg:8081/nexus/content/repositories/t1.snapshot/ -DrepositoryId=t1.snapshot -DartifactId=artifactID -DgroupId=groupID -Dpackaging=zip -Dfile=${WORKSPACE}/filename.zip -Dversion=1.0-TEST-SNAPSHOT -s "/path/to/.m2/settings.xml"
You can make use of 'Deploy artifacts to Maven Repository' under 'Post-Build Actions'. Take look at this answer
Here's the ugly hack I came up with. I'll gladly give someone else the "correct" answer for this if anyone has a better idea:
I realized that I need to deploy both the parent pom.xml and the assembly. I did this in two separate post build steps.
First, I chose "Invoke top-level Maven targets" with a Maven Version of "Maven" (I think this uses Jenkins version of maven. I don't want to put a different version on the system). I used Goals of:
-s svn-admin/settings.xml -N deploy
That deploys just the parent pom to nexus with my specified settings.xml.
The REALLY big hack happens when I want to deploy the rpm. I tried a "deploy-file" target, but without a variable I could expand to the version number, I couldn't specify the exact file and wildcards don't expand. Instead I did an "Execute shell" option and used curl I found here:
env
UPLOAD_FILE=assembly/target/ips-${POM_VERSION}-x.x86_64.rpm
DESTINATION=http://mvnrepo01/nexus/content/repositories/releases/com/bla/ips/assembly/${POM_VERSION}/assembly-${POM_VERSION}.rpm
sha1sum ${UPLOAD_FILE} | awk -F" " '{print $1}' | curl -v -u admin:password --upload-file - ${DESTINATION}.sha1
md5sum ${UPLOAD_FILE} | awk -F" " '{print $1}' | curl -v -u admin:password --upload-file - ${DESTINATION}.md5
curl -v -u admin:password --upload-file ${UPLOAD_FILE} ${DESTINATION}
UPLOAD_FILE=assembly/pom.xml
DESTINATION=http://mvnrepo01/nexus/content/repositories/releases/com/bla/ips/assembly/${POM_VERSION}/assembly-${POM_VERSION}.pom
sha1sum ${UPLOAD_FILE} | awk -F" " '{print $1}' | curl -v -u admin:password --upload-file - ${DESTINATION}.sha1
md5sum ${UPLOAD_FILE} | awk -F" " '{print $1}' | curl -v -u admin:password --upload-file - ${DESTINATION}.md5
curl -v -u admin:password --upload-file ${UPLOAD_FILE} ${DESTINATION}
Like i said, this is an ugly hack. I'm pretty sure there are metadata files that aren't getting updated, but the rpm, it's pom, and their checksums are getting uploaded.
You can use Nexus Jenkins Plugin to deploy a specific artifact from Jenkins into Nexus:
https://support.sonatype.com/hc/en-us/articles/227256688-How-do-I-configure-the-Nexus-Jenkins-Plugin
Example of Jenkins pipeline:
stage('Publish') {
def pom = readMavenPom file: 'pom.xml'
nexusPublisher nexusInstanceId: 'your-nexus-instance-id', \
nexusRepositoryId: 'your-nexus-repository-id', \
packages: [[$class: 'MavenPackage', \
mavenAssetList: [[classifier: '', extension: '', filePath: "target/${pom.artifactId}-${pom.version}.${pom.packaging}"]], \
mavenCoordinate: [artifactId: "${pom.artifactId}", \
groupId: "${pom.groupId}", \
packaging: "${pom.packaging}", \
version: "${pom.version}"]]]
}
In this case Nexus Jenkins Plugin will deploy only your target/${pom.artifactId}-${pom.version}.${pom.packaging} and pom.xml files to Nexus repository.

Jenkins: How to use a variable from a pre-build shell in the Maven "Goals and options"

I have a Maven job in Jenkins. Before the actual build step I have an "Execute shell" pre-build step. In that shell I set a variable:
REVISION=$(cat .build_revision)
I would like to use that variable in the Maven build job in "Goals and options":
clean install -Drevision=${REVISION}
But that does not work! The "Drevision" is set to "${REVISION}" not the actual value of ${REVISION}. Output:
Executing Maven: -B -f /home/gerrit/.jenkins/jobs/<job_name>/workspace/pom.xml clean install -Drevision=${REVISION}
It works with Jenkins environment variables:
clean install -Dbuild=${BUILD_NUMBER}
It sets "Dbuild" to the actual build number. Output:
Executing Maven: -B -f /home/gerrit/.jenkins/jobs/<job_name>/workspace/pom.xml clean install -Dbuild=54
My question: How to use a shell variable in Maven "Goals and options"??
EDIT:
I tried using Jenkins EnvInject Plugin to "Inject environment variables" after the pre-build shell, and my variable is now accessible by e.g. post-build shells, but it is still not available in Maven "Goals and options".
Then it is possible to set "Inject environment variables to the build process" using the EnvInject Plugin, which actually makes those variables available in Maven "Goals and options", but those are set right after SCM checkout, i.e. before pre-build steps, and do not support expression evaluations.
You're on the right track here, but missed a third feature of the EnvInject-Plugin: The "Inject environment variables" build step that can inject variables into following build steps based on the result of a script or properties.
We're using the EnvInject plugin just like that; A script sets up a resource and communicates its parameters using properties that are then propagated by the plugin as environment variables.
i.e. setting up a temporary database for the build:
I had a very similar problem, trying to compute a build version and inject it into the build. After running into all the same issues (not expanding, etc), I used the "Generate environment variables from script" option, which interprets the output as tag=value pairs into Jenkins variables. The script :
#generate a version code that is high enough to surpass previously published clients
val=`expr 150000 + $BUILD_NUMBER`
echo VERSION_CODE=$val
After this, I was able to inject $VERSION_CODE into maven as follows :
-Dbuild.vercode=${VERSION_CODE}
Hope that works for you.
This issue is caused by a bug in the Jenkins Maven Project Plugin as detailed in this bug report opened on 2012-06-22. The plugin has not yet been fixed as of version 2.1.
A fix has been proposed for the Maven Project Plugin, but has not yet been integrated. Here is the link to the pull request: https://github.com/jenkinsci/maven-plugin/pull/14
If you build the plugin yourself with the pull request patch applied, the variables are injected and made available to the "goals and options" field as expected.
I see there is an accepted answer, but for a newbie in Jenkins I found it hard to grasp it all. That's why I would add a bit more detail in this answer and show how I did it.
As #jjungnickel suggested you need to have EnvInject Plugin installed for Jenkins. Then in the Build section > Add build step you'll get option "Inject environment variables".
Basically the idea is:
Add variables you want to access later to a file (might be added by a shell script or it could be file from the file system).
Inject the file with the variables.
Use the variables.
Here a sample setup:
Since I want to use them in maven goal I need to check the Inject Build Variables checkbox.
Then at the end of the build I remove the file just because I want to keep the environment as it was before the build.
I think your best shot is to try the EnvInject plugin for this along with your initial pre-scm step.
You run the pre-scm as you already do.
You use the env inject to load the file for the main job's build steps
Consider loading your file's content (properties format) or execute a script which will load the file as you want and make a variable available for the rest of the job with the "Prepare an environment for the run" option.
I hope this helps.
I needed to resolve the variables before the injection was done so I put this in script content:
Example: (note it doesn't seem possible to simply export variables here so I wrote to files and the help section in jenkins seems to indicate this is expected)
git ls-tree --name-only -r ${sha1} | grep -v -c "*\.md" > diff.bak
git diff origin/master --shortstat | grep "1 files changed" && echo 1 > count.bak || echo 0 > count.bak
I then added this in the groovy script, using the output files I can create a map:
def procDiff = "cat $WORKSPACE/diff.bak".execute()
def procCount = "cat $WORKSPACE/count.bak".execute()
def diff = procDiff.text
def count = procCount.text
print "string val = $diff and count = $count "
if ("0".equals(diff) || !"1".equals(count)){
def map = ["GOAL": "clean verify"]
return map
} else {
def map = ["GOAL": "clean"]
return map
}
Then I could reference $GOAL in my maven build to conditionally trigger a "clean" or a "clean verify" based on the type of PR raised.

Jenkins executing maven from incorrect path

This happens for both: maven projects, and freestyle projects, when maven target is envoked, it tries to execute mvn using absolute path.
[MY-Job] $ tools/Maven/Jenkins_Private_Maven/bin/mvn -f cc/pom.xml -Ddeploy_env=xxx.dev.prv -Dbranch=dev -D-Dsmdist.target=/opt/builds -U clean test -DtestGroups=unit,delegate -Do verride:server=xxx.dev.prv
FATAL: command execution failed
java.io.IOException: Cannot run program "tools/Maven/Jenkins_Private_Maven/bin/mvn" (in directory "workspace/MY-Job"): java.io.IOException: error=2, No such file or directory
I can see that mvn is installed at user's home :
/home/jenkins/tools/Maven/Jenkins_Private_Maven/bin/mvn
but it's trying to run it from the workspace:
/home/jenkins/workspace/MY-Job/tools/Maven/Jenkins_Private_Maven/bin/mvn
Add the default maven installation under (Jenkins -> configuration)
Goto the failing job and make sure you choose the default maven installation from dropdown
Run the job. Success!
Are you sure you have set up Maven in Jenkins -> configuration like this
I have hunch you have a accidental **"."** current directory reference somewhere in your maven set up.
This looks like an auto installed Maven by Jenkins. In which case the previous answers are not correct.
It would seem in this occasion that you have not specified a "remote fs root" for your slave in the salve setup - later versions of Jenkins flag not setting this up correctly as an error.
in Jenkins 2.43:
Manage Jenkins -> Global Tool Configuration -> Maven

Resources