How to download the latest artifact from Artifactory repository? - shell

Question

I need the latest artifact (for example, a snapshot) from a repository in Artifactory. This artifact needs to be copied to a server (Linux) via a script.
What are my options? Something like Wget / SCP? And how do I get the path of the artifact?
I found some solutions which require Artifactory Pro. But I just have Artifactory, not Artifactory Pro.
Is it possible at all to download from Artifactory without the UI and not having the Pro-Version? What is the experience?
I'm on OpenSUSE 12.1 (x86_64) if that matters.

Answer 1

Something like the following bash script will retrieve the lastest com.company:artifact snapshot from the snapshot repo:
# Artifactory location
server=http://artifactory.company.com/artifactory
repo=snapshot
# Maven artifact location
name=artifact
artifact=com/company/$name
path=$server/$repo/$artifact
version=$(curl -s $path/maven-metadata.xml | grep latest | sed "s/.*<latest>\([^<]*\)<\/latest>.*/\1/")
build=$(curl -s $path/$version/maven-metadata.xml | grep '<value>' | head -1 | sed "s/.*<value>\([^<]*\)<\/value>.*/\1/")
jar=$name-$build.jar
url=$path/$version/$jar
# Download
echo $url
wget -q -N $url
It feels a bit dirty, yes, but it gets the job done.

Answer 2

Artifactory has a good extensive REST-API and almost anything that can be done in the UI (perhaps even more) can also be done using simple HTTP requests.
The feature that you mention - retrieving the latest artifact, does indeed require the Pro edition; but it can also be achieved with a bit of work on your side and a few basic scripts.
Option 1 - Search:
Perform a GAVC search on a set of group ID and artifact ID coordinates to retrieve all existing versions of that set; then you can use any version string comparison algorithm to determine the latest version.
Option 2 - the Maven way:
Artifactory generates a standard XML metadata that is to be consumed by Maven, because Maven is faced with the same problem - determining the latest version; The metadata lists all available versions of an artifact and is generated for every artifact level folder; with a simple GET request and some XML parsing, you can discover the latest version.

Answer 3

Using shell/unix tools
curl 'http://$artiserver/artifactory/api/storage/$repokey/$path/$version/?lastModified'
The above command responds with a JSON with two elements - "uri" and "lastModified"
Fetching the link in the uri returns another JSON which has the "downloadUri" of the artifact.
Fetch the link in the "downloadUri" and you have the latest artefact.
Using Jenkins Artifactory plugin
(Requires Pro) to resolve and download latest artifact, if Jenkins Artifactory plugin was used to publish to artifactory in another job:
Select Generic Artifactory Integration
Use Resolved Artifacts as
${repokey}:**/${component}*.jar;status=${STATUS}#${PUBLISH_BUILDJOB}#LATEST=>${targetDir}

Answer 4

You could also use Artifactory Query Language to get the latest artifact.
The following shell script is just an example. It uses 'items.find()' (which is available in the non-Pro version), e.g. items.find({ "repo": {"$eq":"my-repo"}, "name": {"$match" : "my-file*"}}) that searches for files that have a repository name equal to "my-repo" and match all files that start with "my-file". Then it uses the shell JSON parser ./jq to extract the latest file by sorting by the date field 'updated'. Finally it uses wget to download the artifact.
#!/bin/bash
# Artifactory settings
host="127.0.0.1"
username="downloader"
password="my-artifactory-token"
# Use Artifactory Query Language to get the latest scraper script (https://www.jfrog.com/confluence/display/RTF/Artifactory+Query+Language)
resultAsJson=$(curl -u$username:"$password" -X POST http://$host/artifactory/api/search/aql -H "content-type: text/plain" -d 'items.find({ "repo": {"$eq":"my-repo"}, "name": {"$match" : "my-file*"}})')
# Use ./jq to pars JSON
latestFile=$(echo $resultAsJson | jq -r '.results | sort_by(.updated) [-1].name')
# Download the latest scraper script
wget -N -P ./libs/ --user $username --password $password http://$host/artifactory/my-repo/$latestFile

Answer 5

You can use the REST-API's "Item last modified". From the docs, it retuns something like this:
GET /api/storage/libs-release-local/org/acme?lastModified
{
"uri": "http://localhost:8081/artifactory/api/storage/libs-release-local/org/acme/foo/1.0-SNAPSHOT/foo-1.0-SNAPSHOT.pom",
"lastModified": ISO8601
}
Example:
# Figure out the URL of the last item modified in a given folder/repo combination
url=$(curl \
-H 'X-JFrog-Art-Api: XXXXXXXXXXXXXXXXXXXX' \
'http://<artifactory-base-url>/api/storage/<repo>/<folder>?lastModified' | jq -r '.uri')
# Figure out the name of the downloaded file
downloaded_filename=$(echo "${url}" | sed -e 's|[^/]*/||g')
# Download the file
curl -L -O "${url}"

Answer 6

With recent versions of artifactory, you can query this through the api.
https://www.jfrog.com/confluence/display/RTF/Artifactory+REST+API#ArtifactoryRESTAPI-RetrieveLatestArtifact
If you have a maven artifact with 2 snapshots
name => 'com.acme.derp'
version => 0.1.0
repo name => 'foo'
snapshot 1 => derp-0.1.0-20161121.183847-3.jar
snapshot 2 => derp-0.1.0-20161122.00000-0.jar
Then the full paths would be
https://artifactory.example.com/artifactory/foo/com/acme/derp/0.1.0-SNAPSHOT/derp-0.1.0-20161121.183847-3.jar
and
https://artifactory.example.com/artifactory/foo/com/acme/derp/0.1.0-SNAPSHOT/derp-0.1.0-20161122.00000-0.jar
You would fetch the latest like so:
curl https://artifactory.example.com/artifactory/foo/com/acme/derp/0.1.0-SNAPSHOT/derp-0.1.0-SNAPSHOT.jar

Answer 7

The role of Artifactory is to provide files for Maven (as well as other build tools such as Ivy, Gradle or sbt). You can just use Maven together with the maven-dependency-plugin to copy the artifacts out. Here's a pom outline to start you off...
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>A group id</groupId>
<artifactId>An artifact id</artifactId>
<version>1.0.0-SNAPSHOT</version>
<packaging>pom</packaging>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.3</version>
<executions>
<execution>
<id>copy</id>
<phase>package</phase>
<goals>
<goal>copy</goal>
</goals>
<configuration>
<artifactItems>
<artifactItem>
<groupId>The group id of your artifact</groupId>
<artifactId>The artifact id</artifactId>
<version>The snapshot version</version>
<type>Whatever the type is, for example, JAR</type>
<outputDirectory>Where you want the file to go</outputDirectory>
</artifactItem>
</artifactItems>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
Just run mvn install to do the copy.

Answer 8

You can use the wget --user=USER --password=PASSWORD .. command, but before you can do that, you must allow artifactory to force authentication, which can be done by unchecking the "Hide Existence of Unauthorized Resources" box at Security/General tab in artifactory admin panel. Otherwise artifactory sends a 404 page and wget can not authenticate to artifactory.

Answer 9

For me the easiest way was to read the last versions of the project with a combination of curl, grep, sort and tail.
My format: service-(version: 1.9.23)-(buildnumber)156.tar.gz
versionToDownload=$(curl -u$user:$password 'https://$artifactory/artifactory/$project/' | grep -o 'service-[^"]*.tar.gz' | sort | tail -1)

Answer 10

This may be new:
https://artifactory.example.com/artifactory/repo/com/example/foo/1.0.[RELEASE]/foo-1.0.[RELEASE].tgz
For loading module foo from example.com . Keep the [RELEASE] parts verbatim. This is mentioned in the docs but it's not made abundantly clear that you can actually put [RELEASE] into the URL (as opposed to a substitution pattern for the developer).

Answer 11

If you want to download the latest jar between 2 repositores, you can use this solution. I actually use it within my Jenkins pipeline, it works perfectly. Let's say you have a plugins-release-local and plugins-snapshot-local and you want to download the latest jar between these. Your shell script should look like this
NOTE : I use jfrog cli and it's configured with my Artifactory server.
Use case : Shell script
# your repo, you can change it then or pass an argument to the script
# repo = $1 this get the first arg passed to the script
repo=plugins-snapshot-local
# change this by your artifact path, or pass an argument $2
artifact=kshuttle/ifrs16
path=$repo/$artifact
echo $path
~/jfrog rt download --flat $path/maven-metadata.xml version/
version=$(cat version/maven-metadata.xml | grep latest | sed "s/.*<latest>\([^<]*\)<\/latest>.*/\1/")
echo "VERSION $version"
~/jfrog rt download --flat $path/$version/maven-metadata.xml build/
build=$(cat build/maven-metadata.xml | grep '<value>' | head -1 | sed "s/.*<value>\([^<]*\)<\/value>.*/\1/")
echo "BUILD $build"
# change this by your app name, or pass an argument $3
name=ifrs16
jar=$name-$build.jar
url=$path/$version/$jar
# Download
echo $url
~/jfrog rt download --flat $url
Use case : Jenkins Pipeline
def getLatestArtifact(repo, pkg, appName, configDir){
sh """
~/jfrog rt download --flat $repo/$pkg/maven-metadata.xml $configDir/version/
version=\$(cat $configDir/version/maven-metadata.xml | grep latest | sed "s/.*<latest>\\([^<]*\\)<\\/latest>.*/\\1/")
echo "VERSION \$version"
~/jfrog rt download --flat $repo/$pkg/\$version/maven-metadata.xml $configDir/build/
build=\$(cat $configDir/build/maven-metadata.xml | grep '<value>' | head -1 | sed "s/.*<value>\\([^<]*\\)<\\/value>.*/\\1/")
echo "BUILD \$build"
jar=$appName-\$build.jar
url=$repo/$pkg/\$version/\$jar
# Download
echo \$url
~/jfrog rt download --flat \$url
"""
}
def clearDir(dir){
sh """
rm -rf $dir/*
"""
}
node('mynode'){
stage('mysstage'){
def repos = ["plugins-snapshot-local","plugins-release-local"]
for (String repo in repos) {
getLatestArtifact(repo,"kshuttle/ifrs16","ifrs16","myConfigDir/")
}
//optional
clearDir("myConfigDir/")
}
}
This helps alot when you want to get the latest package between 1 or more repos. Hope it helps u too!
For more Jenkins scripted pipelines info, visit Jenkins docs.

Answer 12

With awk:
curl -sS http://the_repo/com/stackoverflow/the_artifact/maven-metadata.xml | grep latest | awk -F'<latest>' '{print $2}' | awk -F'</latest>' '{print $1}'
With sed:
curl -sS http://the_repo/com/stackoverflow/the_artifact/maven-metadata.xml | grep latest | sed 's:<latest>::' | sed 's:</latest>::'

Answer 13

In case you need to download an artifact in a Dockerfile, instead of using wget or curl or the likes you can simply use the 'ADD' directive:
ADD ${ARTIFACT_URL} /opt/app/app.jar
Of course, the tricky part is determining the ARTIFACT_URL, but there's enough about that in all the other answers.
However, Docker best practises strongly discourage using ADD for this purpose and recommend using wget or curl.

Answer 14

No one mention xmllint aka proper xml parser, install it with:
sudo apt-get update -qq
sudo apt-get install -y libxml2-utils
use it:
ART_URL="https://artifactory.internal.babycorp.com/artifactory/api-snapshot/com/babycorp/baby-app/CD-684-my-branch-SNAPSHOT"
ART_VERSION=`curl -s $ART_URL/maven-metadata.xml | xmllint --xpath '//snapshotVersion[1]/value/text()' -`
and finally:
curl -s -o baby-app.jar ${ART_URL}/baby-app-${ART_VERSION}.jar
or
wget ${ART_URL}/baby-app-${ART_VERSION}.jar
to keep the filename

Answer 15

If using the JFrog CLI, this can be done in a single line as follows:
jf rt dl "path/to/artifacts/-/" --sort-by=created --sort-order=desc --limit=1
(The above is for v2 but think the v1 command should be the same but substituting jf with jfrog.)