How can you deploy a specific artifact from Jenkins into Nexus? - maven

I have a multi-module maven project running in Jenkins. I would like to deploy the final artifact (an RPM from an assembly build) to the Nexus server. I see no reason to deploy intermediate artifacts (hence no "mvn clean deploy") since this will produce extra junk on the server that I don't need. We're trying to set up a continuous delivery pipeline, so we're not deploying SNAPSHOT versions - ever. The various plugins for Jenkins seem focused on deploying all of the artifacts. How can I just deploy the one I choose?
EDIT:
After much consideration, I'd be willing to deploy all the artifacts to nexus if the deploy happens after all of the build has completed. If I was going that route, I'd want to use the "Deploy artifacts to maven repository" post build action. This tool seems broken though because it's missing the functionality of specifying the username/password. I know this can be specified in the settings.xml, but apparently only the one in ".m2". It doesn't seem to be looking at the settings.xml I specified for this build.
This also seems like it's broken for redeploying artifacts. There is some verbage in "Jenkins the Definitive Guide" for this, but it talks about a "local" settings.xml. All my builds are happening on Jenkins slaves, so this isn't an option and doesn't even really make sense with the Jenkins architecture.

If you need still generic way.
Use Execute shell option and use the mvn deploy command manually, You can pass your version and others things such as groupID etc as parameters in job, If you maintain separate job to build and upload this will work out.
Ex:
export M2_HOME=/PATH/TO/softwares/apache-maven-3.0.4
PATH=$M2_HOME/bin:$JAVA_HOME/bin:$PATH
export PATH
mvn -v
mvn deploy:deploy-file -Durl=http://someorg:8081/nexus/content/repositories/t1.snapshot/ -DrepositoryId=t1.snapshot -DartifactId=artifactID -DgroupId=groupID -Dpackaging=zip -Dfile=${WORKSPACE}/filename.zip -Dversion=1.0-TEST-SNAPSHOT -s "/path/to/.m2/settings.xml"

You can make use of 'Deploy artifacts to Maven Repository' under 'Post-Build Actions'. Take look at this answer

Here's the ugly hack I came up with. I'll gladly give someone else the "correct" answer for this if anyone has a better idea:
I realized that I need to deploy both the parent pom.xml and the assembly. I did this in two separate post build steps.
First, I chose "Invoke top-level Maven targets" with a Maven Version of "Maven" (I think this uses Jenkins version of maven. I don't want to put a different version on the system). I used Goals of:
-s svn-admin/settings.xml -N deploy
That deploys just the parent pom to nexus with my specified settings.xml.
The REALLY big hack happens when I want to deploy the rpm. I tried a "deploy-file" target, but without a variable I could expand to the version number, I couldn't specify the exact file and wildcards don't expand. Instead I did an "Execute shell" option and used curl I found here:
env
UPLOAD_FILE=assembly/target/ips-${POM_VERSION}-x.x86_64.rpm
DESTINATION=http://mvnrepo01/nexus/content/repositories/releases/com/bla/ips/assembly/${POM_VERSION}/assembly-${POM_VERSION}.rpm
sha1sum ${UPLOAD_FILE} | awk -F" " '{print $1}' | curl -v -u admin:password --upload-file - ${DESTINATION}.sha1
md5sum ${UPLOAD_FILE} | awk -F" " '{print $1}' | curl -v -u admin:password --upload-file - ${DESTINATION}.md5
curl -v -u admin:password --upload-file ${UPLOAD_FILE} ${DESTINATION}
UPLOAD_FILE=assembly/pom.xml
DESTINATION=http://mvnrepo01/nexus/content/repositories/releases/com/bla/ips/assembly/${POM_VERSION}/assembly-${POM_VERSION}.pom
sha1sum ${UPLOAD_FILE} | awk -F" " '{print $1}' | curl -v -u admin:password --upload-file - ${DESTINATION}.sha1
md5sum ${UPLOAD_FILE} | awk -F" " '{print $1}' | curl -v -u admin:password --upload-file - ${DESTINATION}.md5
curl -v -u admin:password --upload-file ${UPLOAD_FILE} ${DESTINATION}
Like i said, this is an ugly hack. I'm pretty sure there are metadata files that aren't getting updated, but the rpm, it's pom, and their checksums are getting uploaded.

You can use Nexus Jenkins Plugin to deploy a specific artifact from Jenkins into Nexus:
https://support.sonatype.com/hc/en-us/articles/227256688-How-do-I-configure-the-Nexus-Jenkins-Plugin
Example of Jenkins pipeline:
stage('Publish') {
def pom = readMavenPom file: 'pom.xml'
nexusPublisher nexusInstanceId: 'your-nexus-instance-id', \
nexusRepositoryId: 'your-nexus-repository-id', \
packages: [[$class: 'MavenPackage', \
mavenAssetList: [[classifier: '', extension: '', filePath: "target/${pom.artifactId}-${pom.version}.${pom.packaging}"]], \
mavenCoordinate: [artifactId: "${pom.artifactId}", \
groupId: "${pom.groupId}", \
packaging: "${pom.packaging}", \
version: "${pom.version}"]]]
}
In this case Nexus Jenkins Plugin will deploy only your target/${pom.artifactId}-${pom.version}.${pom.packaging} and pom.xml files to Nexus repository.

Related

Package maven artifacts from local repository

I'm aware of artifactory (which is planned in the future), but it's requested to package the artifacts on our releases.
We currently do on a script:
printf "\nCopying artifacts...\n"
for artifact_dir in "$HOME"/.m2/repository/com/foo/{*-ear,*-ui,*-tool}; do
highest_version=$(find "${artifact_dir}"/* -maxdepth 1 -type d -printf "%f\n" | sort -V | tail -1)
artifact_name=$(basename "${artifact_dir}")
mkdir --parent "${artifacts_out}/com/foo/${artifact_name}/${highest_version}"
cp --archive "${artifact_dir}/${highest_version}"/* \
"${artifacts_out}/com/foo/${artifact_name}/${highest_version}"
done
But as you can see it's not very neat, p.e we can not create a release with older artifacts because the script always takes the newest one.
Is there a maven plugin or something that permits to archive specific artifacts ?
If you want to create bundles of artifacts, run the Maven assembly plugin during the build. It can be used to gather artifacts, zip them and deploy the result to Artifactory/Nexus.

Command line tool to parse XML and run an XPath for CI/CD

For Maven projects running in CD, we often need to reference the current version of the project (I imagine this is a pretty universal problem for anyone doing Java builds)
For example, one may need to find an artifact pattern based on version or copy the artifact to a S3 bucket named after the current version declared in the POM
However, I've found it shockingly difficult to get data out of the POM. Best I could find was:
mvn org.apache.maven.plugins:maven-help-plugin:2.1.1:evaluate -Dexpression=project.version | grep -Ev '(^\[|Download)' > target/version.txt
which seems hacky ...
is there a linux command line utility available through a package manager that can just parse an XML file and run a XPath? If such a thing does not exist, I might consider creating one
I am asking because I am surprised how frequently this feature is needed in builds and how little I was able to find in terms of a popular solution
First you should configure to use maven-help-plugin version 3.1.0 and than you can get the version like this:
RESULT=$(mvn help:evaluate -Dexpression=project.version -q -DforceStdout)
echo $RESULT
Or you can also go the explicit way:
RESULT=$(mvn org.apache.maven.plugins:maven-help-plugin:3.1.0:evaluate -Dexpression=project.version -q -DforceStdout)
echo $RESULT
More details can be found in my blog.

Jenkins "Invoke Maven 3" build step does not expose POM_* variables

I am using Jenkins as my CI server and Artifactory as my artifact repository. The Artifactory plugin works in free-style projects and can be used just by using Invoke Maven3 build step to both fetch the dependencies and deploy the artifacts. Furthermore, I have another build step for building a docker image and pushing it to my private docker registry using CloudBees Docker Build and Publish plugin.
The problem is that I want to use POM_* variables in CloudBees configurarion that should be exposed by Maven Plugin as mentioned here. But everytime I build the project, I got the following error:
ERROR: Unrecognized macro 'POM_VERSION' in '${POM_VERSION}'
org.jenkinsci.plugins.tokenmacro.MacroEvaluationException: Unrecognized macro 'POM_VERSION' in '${POM_VERSION}'
at org.jenkinsci.plugins.tokenmacro.TokenMacro.expand(TokenMacro.java:198)
at org.jenkinsci.plugins.tokenmacro.TokenMacro.expandAll(TokenMacro.java:233)
at org.jenkinsci.plugins.tokenmacro.TokenMacro.expandAll(TokenMacro.java:222)
at com.cloudbees.dockerpublish.DockerBuilder$Perform.expandAll(DockerBuilder.java:266)
at com.cloudbees.dockerpublish.DockerBuilder$Perform.getNameAndTag(DockerBuilder.java:277)
at com.cloudbees.dockerpublish.DockerBuilder$Perform.exec(DockerBuilder.java:247)
at com.cloudbees.dockerpublish.DockerBuilder$Perform.access$100(DockerBuilder.java:233)
at com.cloudbees.dockerpublish.DockerBuilder.perform(DockerBuilder.java:208)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779)
at hudson.model.Build$BuildExecution.build(Build.java:205)
at hudson.model.Build$BuildExecution.doRun(Build.java:162)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:537)
at hudson.model.Run.execute(Run.java:1744)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:374)
Build step 'Docker Build and Publish' marked build as failure
I do not know what exactly is the reason of this problem. Maybe the Invoke Maven3 build step is causing this problem by not exposing env variables. Another source of this could be CloudBees Plugin that does not utilize env variables. Any ideas?
The build step you mention, in a freestyle project, is not part of the Maven plugin. The environment variable described is only set for native Maven projects. From another project type, I suppose you would need to do something like
POM_VERSION=$(mvn help:evaluate -Dexpression=project.version | fgrep -v '[INFO]')
at the start of a script which continued to go on to use that variable.
Exposing the variable to subsequent non-shell build steps, such as from the CloudBees Docker Build and Publish plugin, is another matter; there may be some plugin that can do it, but I am not sure what offhand. (EnvInject perhaps.) It may be simpler to switch to a workflow job, using the CloudBees Docker Workflow plugin to drive Docker operations (depending on what exactly you are using from the Artifactory plugin):
node {
stage 'build'
git '…'
sh 'mvn package'
stage 'package'
sh 'mvn help:evaluate -Dexpression=project.version | fgrep -v '[INFO]' > .version'
def version = readFile '.version'
def image = docker.build("my-image:${version}")
stage 'publish'
docker.withRegistry(…) {
image.push()
}
}
By the way, you can add a shell build step just running
env | sort
if you are curious to see what environment variables are available.
Based on #Jesse Glick answer, searching around a little and struggling a bit with jenkins, I ended up with the following solution that works like a charm.
1-Add an "Execute Shell" build step containing following commands:
echo POM_VERSION=$(mvn help:evaluate -Dexpression=project.version | fgrep -v '[INFO]') > pom_file
echo POM_ARTIFACTID=$(mvn help:evaluate -Dexpression=project.groupId | fgrep -v '[INFO]') >> pom_file
echo POM_GROUPID=$(mvn help:evaluate -Dexpression=project.artifactId | fgrep -v '[INFO]') >> pom_file
2-Add an "Inject environment variables" build step (from EnvInject plugin) and enter "pom_file" as "Properties File Path"
3-Add a "Docker Build and Publish" build step and use POM_VERSION, POM_ARTIFACTID, POM_GROUPID in its configuration as you wish.

Maven build to generate only changed artifact

In our project, Maven build generates artifacts for different modules i.e. jar, console, car etc in corresponding folder structure.
Everytime we check in the code, the build genarates full new artifacts even if there is only change in "console" module.
Is there any Maven plugin or a way to generate only the artifacts which were changed since last successful build?
For instance, if I have changed the code for "console" module, then the artifact generated should only have console file in its corresponding folder.
If you are on command line you can use
mvn -pl moduleToBuild
which can be combined with:
mvn -pl moduleToBuild -am
which will also build the dependencies of moduleToBuild.
If you are in a CI solution like jenkins there is a check box to activate this behaviour. This can be found under the Maven configuration part Incremental build - only build changed modules.
You have to start the maven call on the root of your multi-module build.
You may want to look at using maven reactor plugin's reactor:make-scm-changes goal. This link has example on how to use this.
I was looking for something that would check what files I have changed compared to the "upstream" version, and build all Maven modules which contain the files, and all depending on them.
Since reactor:make-scm-changes doesn't seem to do that, one way to do it (Linux Bash way) is to
list the changed files, using (git diff --name-only master...),
find the nearest pom.xml for all,
deduplicate (... | sort | uniq),
provide it to Maven as a list using --project-list, with --also-make.
The rest is joining it together using pipes and functions.
Of course this assumes that all sources are within the folder with pom.xml, which typically is true.
Here's an example of the approach mentioned by Ondra Žižka, using mvn clean install and bash.
Note, it ignores pom packaging modules (as those are typically roots of subtrees and would usually cause additional, unnecessary modules to be built. It also looks for pom.xml files 3 levels deep (for speed), assuming they're all part of the same reactor, but this can be adjusted to your project.
find . -maxdepth 3 -name pom.xml | xargs -I{} grep -iL -F "<packaging>pom</packaging>" {} | xargs dirname | grep -v target | sed -e 's/^.[/]*//g' | grep . > /tmp/mvn-modules.txt && git diff --name-only #{u}...HEAD | grep -o -F -f /tmp/mvn-modules.txt | xargs | tr ' ' ',' | xargs -I{} mvn clean install -U -pl {} -amd
In the example #{u}...HEAD references changes in current branch compared to upstream, but this can be swapped for another diff (example <branchname> master) if this is more suitable.

How to deploy an arbitrary binary to Nexus with maven

I think I have run in to a maven deploy plugin bug, but I want to confirm that I am not doing it wrong before I open a bug report and start grepping through source.
I am trying to deploy an arbitrary binary (tar.gz,jar,box,etc) to a maven repository.
The user I am using is in the settings.xml file with a plaintext password.
The user is in an LDAP group that is connected to nexus with a Role.
That role has 3 privileges associated with it. Update, Create, Read which are in turn associated with the repository target.
Now using wget the user can pull data from the repo, and using this curl command I can deploy to the repo:
curl --request PUT --user USERNAME:PASSWORD https://NEXUS_URL/nexus/content/repositories/REPO_NAME/path/to/file/with/groupid/version/iamafile.jar \
--data #./iamafile.jar -H Content-Type:application/java-archive --verbose
This works fine, http 201 and the file is added. When I use a similiar mvn deploy command:
mvn deploy:deploy-file --settings /path/to/settings.xml \
-Durl=https://NEXUS_URL/nexus/content/repositories/REPOSITORYID -Dfile=iamafile.jar \
-DrepositoryId=REPOSITORYID -DartifactId=ARTIFACTID -DgroupId=GROUPID \
-Dversion=VERSION
I get an http 401 unauthorized error.
There have been multiple bug reports about this, but they supposedly have been fixed.
With a pom file with the same values for version, artifcatId, etc. mvn deploy works fine.
Have I found a bug, or am I using it improperly?
Edit:
I opened up this bug regarding the issue.
If the dev's end up saying that it isn't a bug, I will add that as an answer if someone hasn't beat me to it.
you need to additionally specify the -DrepositoryId

Resources