Exporting the variables in Gitlab to properties file - spring

I am using a Spring boot based application and a pipeline on gitlab. I need to export the tag and version to my variables in the properties file in the repo.
In before-script i need to take the value of tag from gitlab-ci.yml and put it in my config.properties file in the project.
export APP_VERSION=${CI_COMMIT_TAG:-$(git describe --tags)}
need to pass the values in app version to
version.major=1
version.minor=0
version.patch=0
any other approaches are welcome to do this.
End Goal to pass the version to application configuration before the script runs.

Get major, minor and patch by using sed and git describe
Let's start with getting major,minor and patch from the latest tag. You can use sed to parse the tag. I won't go into detail about the syntax, but this should help you get everything you need.
//fetch latest tag
> VERSION=$(git describe --tags $(git rev-list --tags --max-count=1))
> echo $VERSION
v0.2.2
// remove prefix and suffix from tag
> VERSION=$(echo "$VERSION" | sed "s/^.*[^0-9]\([0-9]*\.[0-9]*\.[0-9]*\).*$/\1/")
> echo $VERSION
0.2.2
// get major, minor, patch
> MAJOR_VERSION=$(echo "$VERSION" | sed "s/^\([0-9]*\).*/\1/")
> echo $MAJOR_VERSION
0
> MINOR_VERSION=$(echo "$VERSION" | sed "s/[0-9]*\.\([0-9]*\).*/\1/")
> echo $MINOR_VERSION
2
> PATCH_VERSION=$(echo "$VERSION" | sed "s/[0-9]*\.[0-9]*\.\([0-9]*\).*/\1/")
> echo $PATCH_VERSION
2
mavens automatic property expansion
As you are using spring boot, you might consider mavens automatic property expansion which allows you to automatically override properties (https://docs.spring.io/spring-boot/docs/current/reference/html/howto.html#howto.properties-and-configuration) .
Add this to the build section of your pom.xml
<build>
...
<resources>
<resource>
<directory>src/main/resources</directory>
<filtering>true</filtering>
</resource>
</resources>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-resources-plugin</artifactId>
<version>2.7</version>
<configuration>
<delimiters>
<delimiter>#</delimiter>
</delimiters>
<useDefaultDelimiters>false</useDefaultDelimiters>
</configuration>
</plugin>
...
</plugins>
</build>
This tells the maven-resources plugin to automatically update values in files in your src/main/resources directory that are delimited by #. So now your change your property file that it looks like this:
major.version=#application.version#
minor.version=#minor.version#
patch.version=#patch.version#
Finally add your desired properties to your pom.xml:
<properties>
...
<version.major>1</version.major>
<version.minor>0</version.minor>
<version.patch>0</version.patch>
</properties>
To dynamically update the version you now only need to pass the variables to maven / JVM when building your application.
build:
stage: build
script:
- mvn clean package -Dversion.major=$MAJOR_VERSION -Dversion.minor=$MINOR_VERSION -Dversion.patch=$PATCH_VERSION
artifacts:
paths:
- pom.xml
- target/*.jar
except:
- tags
optional: use sed to write to properties file
If you are not using maven, you could also directly write your app version to the config file via sed
build:
stage: build
before_script:
- sed -i "s/^\(major\.version\s*=\s*\).*\$/\1$MAJOR_VERSION/" src/main/resources/config.properties
- sed -i "s/^\(minor\.version\s*=\s*\).*\$/\1$MINOR_VERSION/" src/main/resources/config.properties
- sed -i "s/^\(patch\.version\s*=\s*\).*\$/\1$PATCH_VERSION/" src/main/resources/config.properties
script:
- ...
artifacts:
paths:
- pom.xml
- target/*.jar
except:
- tags

Related

MuleSoft Maven plugin not reading variables in Azure pipelines

I have a MuleSoft application that I am trying to deploy from a pipeline.
I am using a Maven plugin and a connected app for credentials. Plugin configuration looks like this:
<configuration>
<armDeployment>
<muleVersion>${app.runtime}</muleVersion>
<uri>https://anypoint.mulesoft.com</uri>
<businessGroupId>${BUSINESSGROUPID}</businessGroupId>
<target>${TARGET}</target>
<targetType>${TARGETGROUP}</targetType>
<connectedAppClientId>${APPCLIENTID}</connectedAppClientId>
<connectedAppClientSecret>${APPCLIENTSECRET}</connectedAppClientSecret>
<connectedAppGrantType>client_credentials</connectedAppGrantType>
<environment>${ENVIRONMENT}</environment>
</armDeployment>
</configuration>
I define variables in Azure pipeline(3 of them are secret credentials) and when I run the pipeline I am getting 401 Unauthorized error.
When I hard-code values in the above configuration it works fine. Only when I try to have the POM file read them from the pipeline variables do I get this error.
Below is my pipeline config too:
trigger:
- master
variables:
APPCLIENTID: $(APPCLIENTID)
APPCLIENTSECRET: $(APPCLIENTSECRET)
ENVIRONMENT: $(ENVIRONMENT)
BUSINESSGROUPID: $(BUSINESSGROUPID)
TARGET: $(TARGET)
TARGETGROUP: $(TARGETGROUP)
pool:
vmImage: ubuntu-latest
steps:
- task: Maven#3
inputs:
mavenPomFile: 'pom.xml'
mavenOptions: '-Xmx3072m'
javaHomeOption: 'JDKVersion'
jdkVersionOption: '1.8'
jdkArchitectureOption: 'x64'
publishJUnitResults: true
testResultsFiles: '**/surefire-reports/TEST-*.xml'
goals: 'clean package deploy -DmuleDeploy'
I am not sure whether I need to define variables here again or not.
How do I make the POM file read variables correctly?
You can't use Azure Pipeline variables directly in the pom. They are not properties in Maven. You have to define them explicitly as such in Maven's command line.
In the goals input you can define Maven properties for the command line and assign them the values from the respective Azure Pipeline variable using the 'goals' input.
I'm guessing that the syntax for referencing variables is $(var) so as an example:
goals: 'clean package deploy -DmuleDeploy -DAPPCLIENTID=$(APPCLIENTID)'
Just try adding the other properties next to APPCLIENTID.
Ok, I found out what the issue is.
Azure Pipelines documentation states that you need to use $() in order to access variables but in the case of a POM file you need to use {}. So the POM file should look like this:
<configuration>
<armDeployment>
<muleVersion>${app.runtime}</muleVersion>
<uri>https://anypoint.mulesoft.com</uri>
<businessGroupId>${BUSINESSGROUPID}</businessGroupId>
<target>${TARGET}</target>
<targetType>${TARGETGROUP}</targetType>
<connectedAppClientId>${APPCLIENTID}</connectedAppClientId>
<connectedAppClientSecret>${APPCLIENTSECRET}</connectedAppClientSecret>
<connectedAppGrantType>client_credentials</connectedAppGrantType>
<environment>${ENVIRONMENT}</environment>
</armDeployment>
</configuration>
One more thing I noticed is that doing this will not allow POM file to read variables set as secret. I am yet to find out how to make this work properly but for now I got most of it to work.

Jib - How to use environmental variables from base image

I have a base java image with some pre defined java_opts as a environmental variable. How can I use them in plugin?
<plugin>
<groupId>com.google.cloud.tools</groupId>
<artifactId>jib-maven-plugin</artifactId>
<version>1.8.0</version>
<configuration>
<from>
<image>${docker.registry}java:11</image>
</from>
<to>
<image>${docker.registry}portal-backend:${dockerfile.tag}</image>
</to>
<container>
<jvmFlags>
# This will fail
<jvmFlag>$JAVA_OPTS</jvmFlag>
</jvmFlags>
</container>
</configuration>
</plugin>
(Before I start: even if $JAVA_OPTS were expanded when running a Maven build at compile time (it isn't expanded, obviously), <jvmFlag>$JAVA_OPTS<jvmFlag> would still fail, because the entire string value of $JAVA_OPTS containing multiple JVM flags would be passed as a single argument to the java binary. For example, -Xms1024m -Xmx2048m should be passed as two separate flags. The entire string including the whitespace as a single argument is not a valid JVM flag.)
If possible, have the base image define JAVA_TOOL_OPTIONS (note not JAVA_TOOL_OPTS nor JAVA_OPTS). Most JVMs will honor JAVA_TOOL_OPTIONS. See https://stackoverflow.com/a/58715040/1701388 for details. (Also note that, container runtimes (docker, Kubernetes, etc.) can always provide environment variables (and/or override whatever variables defined at build time as container configuration) at runtime. That is, you can dynamically set arguments at runtime.)
Another option is to define your own <entrypoint> to use a shell. (Therefore, you need a base image that includes a shell binary (such as /bin/bash). Note that the default base image prior to Jib 3.0 was Distroless and did not include a shell program. OTOH, Jib 3.0+ doesn't use Distroless.)
In this method, you'll need to know the right Java runtime classpath and the main class to use in your JVM launch command. To help this, starting with Jib >= 3.1, Jib creates two JVM argument files inside a built image; they will hold, respectively, the classpath and the main class inside a built image.
Knowing the entrypoint, you can write a shell script (my-entrypoint.sh):
#!/bin/sh
# Assumes `java` is on PATH in the base image.
exec java $JAVA_OPTS \
-cp $( cat /app/jib-classpath-file ) \
$( cat /app/jib-main-class-file )
Alternatively, if you are on Java 9+, you can leverage the #-argument file:
exec java $JAVA_OPTS -cp #/app/jib-classpath-file #/app/jib-main-class-file
Place my-entrypoint.sh under <project root>/src/main/jib. This is the default directory for Jib's <extraDirectories> feature, and Jib will place src/main/jib/my-entrypoint.sh at the root directory in the container image. Then set the default <entrypoint> to this script:
<container>
<!-- Assumes you have /bin/sh as specified at the top of /my-entrypoint.sh. -->
<entrypoint>/my-entrypoint.sh</entrypoint>
</container>
<!-- You also need to make the script executable. -->
<extraDirectories>
<permissions>
<permission>
<file>/my-entrypoint.sh</file>
<mode>755</mode>
</permission>
</permissions>
</extraDirectories>
Alternatively, if you invoke /bin/sh as below, you don't have to configure <extraDirectories> to make the file executable. This may not look customary; you would normally make the script executable and run it directly. But this is perfectly valid, and there is no difference in terms of actual execution (as long as the shebang of /entrypoint.sh is the same #!/bin/sh).
<container>
<entrypoint>
<arg>/bin/sh</arg>
<arg>/my-entrypoint.sh</arg>
</entrypoint>
</container>
It's also possible to do this without creating a script (basically embedding the entire script in pom.xml and passing it to a shell program). In this case, you don't need to configure <extraDirectories>.
<container>
<entrypoint>
<arg>/bin/sh</arg>
<arg>-c</arg>
<arg>exec java $JAVA_OPTS -cp $( cat /app/jib-classpath-file ) $( cat /app/jib-main-class-file )</arg>
</entrypoint>
</container>
Variables defined as below
Option 1: Java System Properties (VM Arguments)
It's important that the -D parameters are before your application.jar otherwise they are not recognized.
java -jar -Dspring.profiles.active=prod application.jar
Option 2: Program arguments
java -jar application.jar --spring.profiles.active=prod --spring.config.location=c:\config
POM changes :
When using jib as maven plugin - to change the loading of spring config file location : then entryPoint to be passed inside container, but seems jib plugin didn't pick that up . so below changes needs to be done in pom for the argument access for the location :
<plugin>
<groupId>com.google.cloud.tools</groupId>
<artifactId>jib-maven-plugin</artifactId>
<version>2.2.0</version>
<configuration>
<to>
<image>image-url</image>
</to>
<container>
<creationTime>${​​​​​​maven.build.timestamp}​​​​​​</creationTime>
<mainClass>com.package.SpringBootMainClass</mainClass>
<args>
<arg>--spring.config.location=/demo/location/application.yml</arg>
</args>
</container>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>build</goal>
</goals>
</execution>
</executions>
</plugin>
Jib-maven plugin in pom how to pass the arguments , just shown a way through pom , jib don't pick up arguments from entrypoint for spring app, that's why thought of providing another way for the same. as above answer doesn't have it.
My solution is to remove JVM memory parameters from jib-maven-plugin configuration at all. Instead I define JAVA_TOOL_OPTIONS environment variable (example: JAVA_TOOL_OPTIONS='-Xss=512k') for the container (for example: in docker-compose configuration file).
Please note that this environment variable DOESN'T NEED to be defined in a base image at all.

Maven Release Plugin - tagBase is ignored

I'm trying to create a release with the Maven Release Plugin, and it seems as if the tagBase setting is being completely ignored.
Here's what my pom looks like:
...
<scm>
<developerConnection>scm:svn:http://svn.server/svn/repos/.../project/trunk</developerConnection>
</scm>
...
<build>
...
<plugins>
...
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-release-plugin</artifactId>
<version>2.5.3</version>
<configuration>
<tagBase>http://svn.server/svn/repos/.../project/branches</tagBase>
</configuration>
</plugin>
</plugins>
</build>
...
And here's the mvn release command I'm running:
mvn release:prepare
I initially tried this with the -DdryRun=true option, and noticed that the pom.xml.tag file that got generated by the dry run was referencing the .../project/tags/ directory instead of the .../projects/branches/ directory specified in the tagBase. I tried this without the dry run, and the results were the same. The release was created in the tags directory in my SVN repository instead of in the branches directory per the tagBase setting in the pom.
Why is tagBase being ignored? How can I get the maven release plugin to create the release under .../project/branches/ instead of .../project/tags/?
Edit - Adding Some Additional Information:
I'm using version 2.5.3 of maven-release-plugin and Maven version 3.3.9.
After running the mvn release:prepare -DdryRun=true command, here's what the generated release.properties and pom.xml.tag files look like:
release.properties:
#release configuration
#Thu Aug 25 09:07:24 EDT 2016
scm.tagNameFormat=#{project.artifactId}-#{project.version}
scm.tag=project-1.0.0
project.scm.group.id\:project.tag=HEAD
pushChanges=true
scm.url=scm\:svn\:http\://svn.server/svn/repos/.../project/trunk
preparationGoals=clean verify
remoteTagging=true
projectVersionPolicyId=default
scm.commentPrefix=[maven-release-plugin]
scm.tagBase=http\://svn.server/svn/repos/.../project/branches
project.scm.group.id\:project.developerConnection=scm\:svn\:http\://svn.server/svn/repos/.../project/trunk
project.dev.group.id\:project=1.0.1-SNAPSHOT
project.rel.group.id\:project=1.0.0
exec.snapshotReleasePluginAllowed=false
exec.additionalArguments=-P my-active-profile
completedPhase=end-release
pom.xml.tag:
...
<scm>
<developerConnection>scm:svn:http://svn.server/svn/repos/.../project/tags/project-1.0.0</developerConnection>
</scm>
...
It looks like the scm.tagBase property is correct in the generated release.properties file, but the scm information in pom.xml.tag still references the tags directory instead of branches.
I started digging through the maven-release-plugin code and the maven-scm-provider-svn-commons code, and I'm pretty sure I've figured out why this is happening.
If the tagBase property matches the standard naming convention for the branches directory in an SVN repository (for example, http://svn.server/svn/repo/project/branches), then the tagBase gets ignored in favor of the standard tags directory. Here's the relevant code from version 1.9.4 of the maven-scm-provider-svn-commons project:
SvnTagBranchUtils.java:
public static String resolveUrl( String repositoryUrl, String tagBase, String subdir, ScmBranch branchTag )
{
...
// User has a tagBase specified so just return the name appended to the tagBase
if ( StringUtils.isNotEmpty( tagBase ) && !tagBase.equals( resolveTagBase( repositoryUrl ) )
&& !tagBase.equals( resolveBranchBase( repositoryUrl ) ) )
{
return appendPath( tagBase, branchTagName );
}
...
return addSuffix( appendPath( appendPath( projectRoot, subdir ), branchTagName ), queryString );
}
The !tagBase.equals( resolveBranchBase( repositoryUrl) ) condition prevents the tagBase from being used if it matches the standard /branches/ directory in the SVN repository layout. I was even able to test this out using a different tagBases that didn't match the /branches/ directory (for example, http://svn.server/svn/repo/.../project/releases) and it worked as expected.
Unfortunately, I'm still not 100% sure about the best way to have the releases written to /branches/ when running mvn release:prepare. I'm thinking I might just modify that bit of code in maven-scm-provider-svn-commons and figure out a way to get the maven-release-plugin to use my modified version of svn-commons.
Anyways, I hope this might help anyone who's trying to do something similar with the maven-release-plugin.

How do I access maven project version in javadoc overview page?

I am using PDFDoclet with maven-javadoc-plugin and I've come quite a long way with it now. I have the maven and javadoc config almost at a point that is good enough but my immediate problem now is that I can't work out how to push the project version number into the PDF title page.
Before you leap to answer my question by telling me to use maven's <resource> filtering, let me outline why that isn't working.
Filtering works by taking the original file from somewhere in the src folder, doing variable substitution and putting the output in the target folder.
Javadoc works by reading files in src/main/java and src/main/javadoc and AFAIK outputting the results into target. This means filtering is useless for Javadoc since it won't read anything from target
My results show that any maven variables in javadoc comments don't get substituted.
What trick can I use to get those variables substituted into the javadoc?
The solution can't involve filtering the javadoc output after the site:site task, unless resource filtering works on PDFs.
This is the configuration, FWIW:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<version>2.10.3</version>
<configuration>
<show>package</show>
<docfilessubdirs>true</docfilessubdirs>
<tags>
<tag>
<name>pdfInclude</name>
<placement>X</placement>
<head></head>
</tag>
</tags>
</configuration>
<reportSets>
<reportSet>
<id>PDF</id>
<reports>
<report>javadoc</report>
</reports>
<configuration>
<name>PDF</name>
<description>PDF doc</description>
<destDir>pdf</destDir>
<doclet>com.tarsec.javadoc.pdfdoclet.PDFDoclet</doclet>
<docletPath>${basedir}/pdfdoclet/pdfdoclet-1.0.3-all.jar</docletPath>
<useStandardDocletOptions>false</useStandardDocletOptions>
<additionalparam>-pdf my_tech_doc-${project.version}.pdf
-config ${basedir}/pdfdoclet/pdfdoclet.properties</additionalparam>
</configuration>
</reportSet>
</reportSets>
</plugin>
and the pdfdoclet.properties:
# http://pdfdoclet.sourceforge.net/configuration.html
#
#Lets the doclet print additional output in the console and to a logfile.
debug=true
#Print "Author" tags
author=false
#Print "Version" tags
version=true
#Print "since" tags
tag.since=true
#Create summary tables
summary.table=false
#Create hyperlinks
create.links=true
#Encrypt the PDF file
encrypted=false
#Allow printing of the PDF file
allow.printing=true
#Create a bookmark frame
create.frame=true
#Print a title page
api.title.page=true
api.copyright=None
api.author=Hansruedi
#Enables the tag-based filter
filter=true
filter.tags=pdfInclude
font.text.name=resources/arial.ttf
page.orientation=portrait
The PDFDoclet-specific api.* properties should result in a title page as the first page of the PDF, but it doesn't work. If there is a trick that I've missed here and I could get that title page produced, then that might also allow a solution for this somehow.
I realised I can do a quick and dirty hack with the maven <resources> functionality:
<resource>
<directory>${basedir}/src/main/resources</directory>
<targetPath>${basedir}/src/main/javadoc</targetPath>
<filtering>true</filtering>
<includes>
<include>**/overview.html</include>
</includes>
</resource>
This copies my overview.html and filters it, outputting it into the javadoc source directory.
The dirtiness is that this filtered version could then accidentally end up under version control, although using svn I can add it to the ignore list.

How to download the latest artifact from Artifactory repository?

I need the latest artifact (for example, a snapshot) from a repository in Artifactory. This artifact needs to be copied to a server (Linux) via a script.
What are my options? Something like Wget / SCP? And how do I get the path of the artifact?
I found some solutions which require Artifactory Pro. But I just have Artifactory, not Artifactory Pro.
Is it possible at all to download from Artifactory without the UI and not having the Pro-Version? What is the experience?
I'm on OpenSUSE 12.1 (x86_64) if that matters.
Something like the following bash script will retrieve the lastest com.company:artifact snapshot from the snapshot repo:
# Artifactory location
server=http://artifactory.company.com/artifactory
repo=snapshot
# Maven artifact location
name=artifact
artifact=com/company/$name
path=$server/$repo/$artifact
version=$(curl -s $path/maven-metadata.xml | grep latest | sed "s/.*<latest>\([^<]*\)<\/latest>.*/\1/")
build=$(curl -s $path/$version/maven-metadata.xml | grep '<value>' | head -1 | sed "s/.*<value>\([^<]*\)<\/value>.*/\1/")
jar=$name-$build.jar
url=$path/$version/$jar
# Download
echo $url
wget -q -N $url
It feels a bit dirty, yes, but it gets the job done.
Artifactory has a good extensive REST-API and almost anything that can be done in the UI (perhaps even more) can also be done using simple HTTP requests.
The feature that you mention - retrieving the latest artifact, does indeed require the Pro edition; but it can also be achieved with a bit of work on your side and a few basic scripts.
Option 1 - Search:
Perform a GAVC search on a set of group ID and artifact ID coordinates to retrieve all existing versions of that set; then you can use any version string comparison algorithm to determine the latest version.
Option 2 - the Maven way:
Artifactory generates a standard XML metadata that is to be consumed by Maven, because Maven is faced with the same problem - determining the latest version; The metadata lists all available versions of an artifact and is generated for every artifact level folder; with a simple GET request and some XML parsing, you can discover the latest version.
Using shell/unix tools
curl 'http://$artiserver/artifactory/api/storage/$repokey/$path/$version/?lastModified'
The above command responds with a JSON with two elements - "uri" and "lastModified"
Fetching the link in the uri returns another JSON which has the "downloadUri" of the artifact.
Fetch the link in the "downloadUri" and you have the latest artefact.
Using Jenkins Artifactory plugin
(Requires Pro) to resolve and download latest artifact, if Jenkins Artifactory plugin was used to publish to artifactory in another job:
Select Generic Artifactory Integration
Use Resolved Artifacts as
${repokey}:**/${component}*.jar;status=${STATUS}#${PUBLISH_BUILDJOB}#LATEST=>${targetDir}
You could also use Artifactory Query Language to get the latest artifact.
The following shell script is just an example. It uses 'items.find()' (which is available in the non-Pro version), e.g. items.find({ "repo": {"$eq":"my-repo"}, "name": {"$match" : "my-file*"}}) that searches for files that have a repository name equal to "my-repo" and match all files that start with "my-file". Then it uses the shell JSON parser ./jq to extract the latest file by sorting by the date field 'updated'. Finally it uses wget to download the artifact.
#!/bin/bash
# Artifactory settings
host="127.0.0.1"
username="downloader"
password="my-artifactory-token"
# Use Artifactory Query Language to get the latest scraper script (https://www.jfrog.com/confluence/display/RTF/Artifactory+Query+Language)
resultAsJson=$(curl -u$username:"$password" -X POST http://$host/artifactory/api/search/aql -H "content-type: text/plain" -d 'items.find({ "repo": {"$eq":"my-repo"}, "name": {"$match" : "my-file*"}})')
# Use ./jq to pars JSON
latestFile=$(echo $resultAsJson | jq -r '.results | sort_by(.updated) [-1].name')
# Download the latest scraper script
wget -N -P ./libs/ --user $username --password $password http://$host/artifactory/my-repo/$latestFile
You can use the REST-API's "Item last modified". From the docs, it retuns something like this:
GET /api/storage/libs-release-local/org/acme?lastModified
{
"uri": "http://localhost:8081/artifactory/api/storage/libs-release-local/org/acme/foo/1.0-SNAPSHOT/foo-1.0-SNAPSHOT.pom",
"lastModified": ISO8601
}
Example:
# Figure out the URL of the last item modified in a given folder/repo combination
url=$(curl \
-H 'X-JFrog-Art-Api: XXXXXXXXXXXXXXXXXXXX' \
'http://<artifactory-base-url>/api/storage/<repo>/<folder>?lastModified' | jq -r '.uri')
# Figure out the name of the downloaded file
downloaded_filename=$(echo "${url}" | sed -e 's|[^/]*/||g')
# Download the file
curl -L -O "${url}"
With recent versions of artifactory, you can query this through the api.
https://www.jfrog.com/confluence/display/RTF/Artifactory+REST+API#ArtifactoryRESTAPI-RetrieveLatestArtifact
If you have a maven artifact with 2 snapshots
name => 'com.acme.derp'
version => 0.1.0
repo name => 'foo'
snapshot 1 => derp-0.1.0-20161121.183847-3.jar
snapshot 2 => derp-0.1.0-20161122.00000-0.jar
Then the full paths would be
https://artifactory.example.com/artifactory/foo/com/acme/derp/0.1.0-SNAPSHOT/derp-0.1.0-20161121.183847-3.jar
and
https://artifactory.example.com/artifactory/foo/com/acme/derp/0.1.0-SNAPSHOT/derp-0.1.0-20161122.00000-0.jar
You would fetch the latest like so:
curl https://artifactory.example.com/artifactory/foo/com/acme/derp/0.1.0-SNAPSHOT/derp-0.1.0-SNAPSHOT.jar
The role of Artifactory is to provide files for Maven (as well as other build tools such as Ivy, Gradle or sbt). You can just use Maven together with the maven-dependency-plugin to copy the artifacts out. Here's a pom outline to start you off...
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>A group id</groupId>
<artifactId>An artifact id</artifactId>
<version>1.0.0-SNAPSHOT</version>
<packaging>pom</packaging>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.3</version>
<executions>
<execution>
<id>copy</id>
<phase>package</phase>
<goals>
<goal>copy</goal>
</goals>
<configuration>
<artifactItems>
<artifactItem>
<groupId>The group id of your artifact</groupId>
<artifactId>The artifact id</artifactId>
<version>The snapshot version</version>
<type>Whatever the type is, for example, JAR</type>
<outputDirectory>Where you want the file to go</outputDirectory>
</artifactItem>
</artifactItems>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
Just run mvn install to do the copy.
You can use the wget --user=USER --password=PASSWORD .. command, but before you can do that, you must allow artifactory to force authentication, which can be done by unchecking the "Hide Existence of Unauthorized Resources" box at Security/General tab in artifactory admin panel. Otherwise artifactory sends a 404 page and wget can not authenticate to artifactory.
For me the easiest way was to read the last versions of the project with a combination of curl, grep, sort and tail.
My format: service-(version: 1.9.23)-(buildnumber)156.tar.gz
versionToDownload=$(curl -u$user:$password 'https://$artifactory/artifactory/$project/' | grep -o 'service-[^"]*.tar.gz' | sort | tail -1)
This may be new:
https://artifactory.example.com/artifactory/repo/com/example/foo/1.0.[RELEASE]/foo-1.0.[RELEASE].tgz
For loading module foo from example.com . Keep the [RELEASE] parts verbatim. This is mentioned in the docs but it's not made abundantly clear that you can actually put [RELEASE] into the URL (as opposed to a substitution pattern for the developer).
If you want to download the latest jar between 2 repositores, you can use this solution. I actually use it within my Jenkins pipeline, it works perfectly. Let's say you have a plugins-release-local and plugins-snapshot-local and you want to download the latest jar between these. Your shell script should look like this
NOTE : I use jfrog cli and it's configured with my Artifactory server.
Use case : Shell script
# your repo, you can change it then or pass an argument to the script
# repo = $1 this get the first arg passed to the script
repo=plugins-snapshot-local
# change this by your artifact path, or pass an argument $2
artifact=kshuttle/ifrs16
path=$repo/$artifact
echo $path
~/jfrog rt download --flat $path/maven-metadata.xml version/
version=$(cat version/maven-metadata.xml | grep latest | sed "s/.*<latest>\([^<]*\)<\/latest>.*/\1/")
echo "VERSION $version"
~/jfrog rt download --flat $path/$version/maven-metadata.xml build/
build=$(cat build/maven-metadata.xml | grep '<value>' | head -1 | sed "s/.*<value>\([^<]*\)<\/value>.*/\1/")
echo "BUILD $build"
# change this by your app name, or pass an argument $3
name=ifrs16
jar=$name-$build.jar
url=$path/$version/$jar
# Download
echo $url
~/jfrog rt download --flat $url
Use case : Jenkins Pipeline
def getLatestArtifact(repo, pkg, appName, configDir){
sh """
~/jfrog rt download --flat $repo/$pkg/maven-metadata.xml $configDir/version/
version=\$(cat $configDir/version/maven-metadata.xml | grep latest | sed "s/.*<latest>\\([^<]*\\)<\\/latest>.*/\\1/")
echo "VERSION \$version"
~/jfrog rt download --flat $repo/$pkg/\$version/maven-metadata.xml $configDir/build/
build=\$(cat $configDir/build/maven-metadata.xml | grep '<value>' | head -1 | sed "s/.*<value>\\([^<]*\\)<\\/value>.*/\\1/")
echo "BUILD \$build"
jar=$appName-\$build.jar
url=$repo/$pkg/\$version/\$jar
# Download
echo \$url
~/jfrog rt download --flat \$url
"""
}
def clearDir(dir){
sh """
rm -rf $dir/*
"""
}
node('mynode'){
stage('mysstage'){
def repos = ["plugins-snapshot-local","plugins-release-local"]
for (String repo in repos) {
getLatestArtifact(repo,"kshuttle/ifrs16","ifrs16","myConfigDir/")
}
//optional
clearDir("myConfigDir/")
}
}
This helps alot when you want to get the latest package between 1 or more repos. Hope it helps u too!
For more Jenkins scripted pipelines info, visit Jenkins docs.
With awk:
curl -sS http://the_repo/com/stackoverflow/the_artifact/maven-metadata.xml | grep latest | awk -F'<latest>' '{print $2}' | awk -F'</latest>' '{print $1}'
With sed:
curl -sS http://the_repo/com/stackoverflow/the_artifact/maven-metadata.xml | grep latest | sed 's:<latest>::' | sed 's:</latest>::'
In case you need to download an artifact in a Dockerfile, instead of using wget or curl or the likes you can simply use the 'ADD' directive:
ADD ${ARTIFACT_URL} /opt/app/app.jar
Of course, the tricky part is determining the ARTIFACT_URL, but there's enough about that in all the other answers.
However, Docker best practises strongly discourage using ADD for this purpose and recommend using wget or curl.
No one mention xmllint aka proper xml parser, install it with:
sudo apt-get update -qq
sudo apt-get install -y libxml2-utils
use it:
ART_URL="https://artifactory.internal.babycorp.com/artifactory/api-snapshot/com/babycorp/baby-app/CD-684-my-branch-SNAPSHOT"
ART_VERSION=`curl -s $ART_URL/maven-metadata.xml | xmllint --xpath '//snapshotVersion[1]/value/text()' -`
and finally:
curl -s -o baby-app.jar ${ART_URL}/baby-app-${ART_VERSION}.jar
or
wget ${ART_URL}/baby-app-${ART_VERSION}.jar
to keep the filename
If using the JFrog CLI, this can be done in a single line as follows:
jf rt dl "path/to/artifacts/-/" --sort-by=created --sort-order=desc --limit=1
(The above is for v2 but think the v1 command should be the same but substituting jf with jfrog.)

Resources