Get uri of published artifact using maven-publish gradle plugin - gradle

After a publish, I want to know what the url of the published artifact is (to use it in other gradle tasks for automated deployment).
Is there any way to capture this generated url?

#Hollerweger answer is probably the least hackish, but it's wrong in couple of ways:
AbstractPublishToMaven is the superclass of PublishToMavenLocal and PublishToMavenRepository. Extending it and printing a message saying artifact is published to Nexus is wrong, because that message is going to be printed even when publishing to the local Maven repo. The correct task class to use is PublishToMavenRepository that publishes to the remote repo.
There's no need to know the remote repo URL; publication has a repository property.
Putting it all together:
tasks.withType(PublishToMavenRepository) {
doFirst {
println("Publishing ${publication.groupId}:${publication.artifactId}:${publication.version} to ${repository.url}")
}
}

I found a local copy of a file called maven-metadata-remote.xml that gets uploaded. It is under this dir: build/tmp/publishMavenPublicationToNexusRepository/
Example:
$ cat build/tmp/publishMavenPublicationToNexusRepository/com/example/my.package/0.1.0-SNAPSHOT/maven-metadata-remote.xml
<?xml version="1.0" encoding="UTF-8"?>
<metadata>
<groupId>com.example</groupId>
<artifactId>my.package</artifactId>
<version>0.1.0-SNAPSHOT</version>
<versioning>
<snapshot>
<timestamp>20180326.191238</timestamp>
<buildNumber>36</buildNumber>
</snapshot>
<lastUpdated>20180326191238</lastUpdated>
</versioning>
</metadata>
So I was able to ingest that file by adding something like this to the bottom of the build.gradle file:
tasks.withType(PublishToMavenRepository) {
doLast {
def metadata_file_path = "build/tmp/publishMavenPublicationToNexusRepository/com/example/my.package/0.1.0-SNAPSHOT"
def metadata_XML = new File("${metadata_file_path}/maven-metadata-remote.xml").text
def metadata = new groovy.util.XmlSlurper().parseText(metadata_XML)
def BN = metadata.versioning.snapshot.buildNumber
def TS = metadata.versioning.snapshot.timestamp
println "INFO: uploaded version 0.1.0-${TS}-${BN}.pom to Nexus"
...

Sadly this information is unavailable via the gradle build system... What you can do is create a task that finalizes the publish task. Then have that task query the maven repository for the most recent build. Maven will look in the maven-metadata.xml file and return the <release> tag value or the most recent upload if that's missing. You can get the exact download url from the response's Location header.
Here is an example of how you would query a maven repo
$ curl -Is 'http://my.nexus.repo.com:8081/nexus/service/local/artifact/maven/redirect?r=Release&g=com.my.group.id&a=myArtifactId&v=RELEASE&p=war' | grep -Fi Location | cut -d' ' -f2
http://my.nexus.repo.com:8081/nexus/service/local/repositories/Release/content/com/my/group/id/myArtifactId/1.0.012/myArtifactId-1.0.012.war
Explaining the command
curl -Is http://<nexus-url>:<nexus-port>/nexus/service/local/artifact/maven/redirect?r=<nexus-repository>&g=<artifact-group-id>&a=<artifact-name>&v=<artifact-version>
curl
-I # only print return headers
-s # quiet output of curl's downloading progress
url params
r # nexus-repository name, tends to be Release or Snapshot
g # group id for the artifact
a # artifact id
v # artifact version or a link like RELEASE (don't use LATEST it's problematic)
## you can also supply classifier and extension if needed
c # artifact classifier
e # artifact extension

You can extend all maven publish tasks and compute the the url again:
I just needed the folder - For release artifacts you can also add the artifact at the end but I haven't found a good solution for SNAPSHOTs. The nexus base repo URL is expected to be known.
tasks.withType(AbstractPublishToMaven) {
doLast {
String urlString = nexusUrl.toString() + convertPackageNameToPath(publication.groupId) + "/" + publication.artifactId + "/" + publication.version
println "Artifact URL: " + urlString
}
}
static String convertPackageNameToPath(String packageName) {
return packageName.replace(".", "/");
}

Starting from #Abhijit Sarkar's answer, you can write universal solution for both remote and local publish direction:
tasks.withType(AbstractPublishToMaven) {
doLast {
println("Published ${publication.groupId}:${publication.artifactId}:${publication.version} to ${name.contains('Local') ? "local maven repository" : repositories.maven.url}")
}
}
Tasks publishing to local repo, contain 'Local' in its name (letter case matter), so the ternary operator defines a correct direction.

The DefaultMavenPublication class exposes the following method:
#Override
public PublishedFile getPublishedFile(final PublishArtifact source) {
final String publishedUrl = getPublishedUrl(source);
final String publishedName = source.getFile().getName();
return new PublishedFile() {
#Override
public String getName() {
return publishedName;
}
#Override
public String getUri() {
return publishedUrl;
}
};
}
introduced in this commit.
Maybe this could do the trick.

Related

Unable to publish jar to Gitlab package registry with gradle

I am trying to publish some jar artefacts on gitlab package registry but I get this error from the server :
Received status code 415 from server: Unsupported Media Type
Here is the publishing section of my build.gradle.kts :
publishing {
publications {
create<MavenPublication>("maven"){
artifact(tasks["bootJar"])
}
}
repositories {
maven {
url = uri("https://gitlab.com/api/v4/groups/my-group/-/packages/maven")
name = "Gitlab"
credentials(HttpHeaderCredentials::class) {
name = "Token"
value = System.getenv("CI_JOB_TOKEN")
}
authentication {
create<HttpHeaderAuthentication>("header")
}
}
}
}
In my gitlab-ci, I added a task for publish the artefacts :
deploy:
stage: deploy
script: gradle publish
only:
- master
Any help would be appreciated
Quick answer
Replace your publishing url pointing to the group-scope with the one pointing to the specific-package-repository, e.g. on gitlab.com:
https://gitlab.com/api/v4/projects/<your-project-id>/packages/maven
You need to replace <your-project-id> with your specific project-id of course.
Related to this a quote from docs.gitlab:
Note: In all cases, you need a project specific URL for uploading a package in the distributionManagement section.
Or in other words: Only the general repositories section can use your groups-url for searching other already published artifacts! (I also had to understand that). So:
you cannot publish to the group-package-store on gitlab, you can just search there.
Publication goes always to the project-specific package-store, which will then be visible at group-scope too.
Example gradle config (kotlin-dsl)
repositories {
mavenCenter()
jcenter()
// Here you USE the group api/v4 url for SEARCHING packages
maven {
name = "GitLab"
url = uri("https://gitlab.com/api/v4/groups/my-group/-/packages/maven")
credentials(HttpHeaderCredentials::class) {
name = "Job-Token"
value = System.getenv("CI_JOB_TOKEN")
}
authentication {
create<HttpHeaderAuthentication>("header")
}
}
}
publishing {
publications {
create<MavenPublication>("maven"){
artifact(tasks["bootJar"])
}
}
repositories {
maven {
// here your PROVIDE the PROJECT-URI for publishing your package
// in the project-specific package-space which is also visible at
// the group scope above
url = uri("https://gitlab.com/api/v4/projects/<your-project-id>/packages/maven")
name = "Gitlab"
credentials(HttpHeaderCredentials::class) {
name = "Job-Token"
value = System.getenv("CI_JOB_TOKEN")
}
authentication {
create<HttpHeaderAuthentication>("header")
}
}
}
}
More Info
There are multiple scenarios on how you may interact with the maven-repository-space on GitLab. The three switches are:
The place where you want to look for existing published packages
project-scope (https://.../api/v4/projects/<project-id>/packages/maven)
group-scope (https://.../api/v4/groups/<group-id>/-/packages/maven)
instance-scope (https://.../api/v4/packages/maven)
The authorization-method you want to use
PERSONAL_ACCESS_TOKEN
DEPLOY_TOKEN
CI_JOB_TOKEN
The place where your want to publish your package
this must always be a specific project-url (https://.../api/v4/projects/<project-id>/packages/maven)
I think the most important thing is to make sure you've enabled archives in your project:
Go to Project Settings
Expand Permissions
Switch on "Packages"
Apparently, there are also other reasons for the response status 415 Unsupported Media Type:
I ran into the same error message while trying to publish to the project repository. The error that I made was to use the URL-encoded path of the project instead of the project ID in the repository URL.
From the Gitlab documentation (emphasis added):
For retrieving artifacts, use either the URL-encoded path of the project
(like group%2Fproject) or the project's ID (like 42). However, only the
project's ID can be used for publishing.
This is the build.gradle.kts configuration that worked for me on my self-hosted Gitlab instance for publishing a Spring Boot fat JAR:
plugins {
/* ... other stuff ... */
`java-library`
`maven-publish`
}
publishing {
publications {
create<MavenPublication>("bootJava") {
artifact(tasks.getByName("bootJar"))
}
}
repositories {
maven {
val projectId = System.getenv("CI_PROJECT_ID")
name = "Project Name"
url = uri("https://gitlab.example.com/api/v4/projects/${projectId}/packages/maven")
credentials(HttpHeaderCredentials::class) {
name = "Job-Token"
value = System.getenv("CI_JOB_TOKEN")
}
authentication {
create<HttpHeaderAuthentication>("header")
}
}
}
}

How do I replace a Maven repository with a local directory after it is defined in a Gradle build?

I'm trying to write tests for a build process, so my unit test wants to replace the actual repository locations with local locations to avoid poisoning the real server. (Plus, I suppose, the person running the test might not have access to publish anyway.)
In the build itself:
publishing {
repositories {
maven {
name = 'snapshot'
url = "${artifactory_contextUrl}/libs-snapshot-local"
credentials {
username artifactory_user
password artifactory_password
}
}
maven {
name = 'release'
url = "${artifactory_contextUrl}/libs-release-local"
credentials {
username artifactory_user
password artifactory_password
}
}
}
}
In my test build, I'm trying to override it with this:
publishing {
repositories {
getByName('snapshot') {
url = uri('/tmp/local-repo/snapshots')
}
getByName('release') {
url = uri('/tmp/local-repo/release')
}
}
}
When I try to run the build, I get:
Execution failed for task ':publishMavenJavaPublicationToReleaseRepository'.
> Failed to publish publication 'mavenJava' to repository 'release'
> Authentication scheme 'all'(Authentication) is not supported by protocol 'file'
There are a lot of posts out on the web about this specific error, but it always seems to be people who accidentally put a file path in when they should have put a URI. I'm putting in a URI deliberately, though, so is there a way to get this to work?
I have also tried this:
publishing {
repositories {
clear()
maven {
name = 'snapshot'
url = uri('/tmp/local-repo/snapshots')
}
maven {
name = 'release'
url = uri('/tmp/local-repo/release')
}
}
}
That fails with:
A problem occurred configuring root project 'test-common-plugin1913987501683151177'.
> Exception thrown while executing model rule: PublishingPluginRules#publishing(ExtensionContainer)
> Cannot add task 'publishMavenJavaPublicationToSnapshotRepository' as a task with that name already exists.
I was surprised that deleting all the repositories doesn't also delete all the tasks they own. When I try to programmatically delete the task it's complaining about, Gradle claims that it doesn't exist.
Alright, I ended up having to read the source of Gradle (again), but I found a way to do it. Essentially you can directly set credentials back to null, like this:
publishing {
repositories {
getByName('snapshot') {
url = uri('/tmp/local-repo/snapshots')
configuredCredentials = null
}
getByName('release') {
url = uri('/tmp/local-repo/release')
configuredCredentials = null
}
}
}

Release multimodule project using one version and tag in Gradle

I have multimodule project in Gradle, where parent build.gradle is just an aggregator of subprojects, without any sources.
Structure:
parent with version in gradle.properties
- child1 inheriting version
- child2 inheriting version
Both children produce JARs with artifacts, while parent produces empty JAR which I don't want to upload anywhere at all.
Now I want to release such project. There should be one commit with version update and one tag in Git. However, all subprojects should be build and uploaded to a repository. How can I achieve it?
I've tried gradle-release plugin, but I struggle to configure it properly. I get either of the following:
apply plugin on parent, get proper commit and tag but get only root project uploaded
apply plugin on subprojects, get projects properly uploaded, but have separate commit and tag for every subproject
I had the same problem and ended up writing the necessary steps in Gradle with the help of the gradle-svn-tools plugin (https://github.com/martoe/gradle-svntools-plugin). Note that I'm no Groovy expert, so I guess a few things can be done more easily :)
My build.gradle in the parent project:
buildscript {
repositories {
jcenter()
}
dependencies {
classpath 'at.bxm.gradleplugins:gradle-svntools-plugin:1.6.2'
}
}
allprojects {
apply plugin: 'at.bxm.svntools'
//subversion login
svntools {
username = svnUsername
password = credentials.svnpwd
}
}
//task to write the version parameter given via command line into the "gradle.properties" files.
// Call with: gradle writeProjectVersion -PnewVersion=1.0.1-SNAPSHOT
task('writeProjectVersion') << {
if (project.hasProperty('newVersion')) {
//set version in gradle settings
project.version = project.newVersion
project.subprojects?.each {it.version = project.version }
//set version in all the gradle.properties files
new File(".").traverse(type : groovy.io.FileType.FILES, nameFilter: 'gradle.properties') {
project.ant.replaceregexp(file: it, byline: true) {
key = 'version'
version = project.version
regexp(pattern: "^(\\s*)$key((\\s*[=|:]\\s*)|(\\s+)).+\$")
substitution(expression: "\\1$key\\2$version")
}
}
println 'Successfully changed project version in gradle.properties to \'' + project.version + '\''
} else {
println 'No parameter \'newVersion\' provided, so not changing the project version.'
}
}
//task to commit the gradle.properties changes into SVN - also needs the new version as parameter for the commit message
// Call with: gradle svnCommit -PnewVersion=1.0.1-SNAPSHOT
task('svnCommit', type: at.bxm.gradleplugins.svntools.tasks.SvnCommit) {
if (project.hasProperty('newVersion')) {
def files = []
new File(".").traverse(type : groovy.io.FileType.FILES, nameFilter: 'gradle.properties') { files << it } //appends all matching files to the list
source = files
commitMessage = "[Jenkins Release Build] Changing project version to '" + project.newVersion + "'"
} else {
println 'No parameter \'newVersion\' provided, so SVN commit was not executed.'
}
}
//task to tag the current head - also needs the new version as parameter for the commit message
// Call with: gradle svnTag -PnewVersion=1.0.1-SNAPSHOT
task('svnTag', type: at.bxm.gradleplugins.svntools.tasks.SvnTag) {
if (project.hasProperty('newVersion')) {
tagName = "$project.newVersion"
localChanges = true
commitMessage = "[Jenkins Release Build] Tagging Release version $project.newVersion"
} else {
println 'No parameter \'newVersion\' provided, so SVN tagging was not executed.'
}
}
I use Jenkins for execution and run the following shell script:
# execute gradle tests
gradle test
# set version in gradle.properties and tag the current code (gradle.properties changes are commited in the tag)
gradle writeProjectVersion -PnewVersion=${releaseVersion}
gradle svnTag -PnewVersion=${releaseVersion}
# build the artifacts and upload them to Nexus without running the test cases again
gradle clean build uploadArchives -x test
# set version in gradle.properties and commit the files
gradle writeProjectVersion -PnewVersion=${newSnapshotVersion}
gradle svnCommit -PnewVersion=${newSnapshotVersion}
Hope this helps.

How do Gradle Uploads Really Work?

I'm having some real problems understanding how all the pieces in gradle to create and upload an artifact fit together.
My intention in this script is simple: I want to download a source tarball and possibly a bunch of dependencies, run a "build.sh" shellscript which will end up creating a binary tarball and have the gradle script publish it to an artifact repo.
The main idea is that I can use gradle's dependency management, maven artifact knowledge and build parallelization and avoidance to control the execution of the build scripts themselves, mainly to manage my set of third party binary dependencies...
The following script fails with a 400 error, I suspect it's because I'm not linking the artifact with the actual output file.
What's the right and proper way to go?
apply plugin: 'maven'
version 'testarch-4.2'
repositories {
maven {
url "http://nexus/..."
}
}
configurations {
sourceArchive
binaryArchive
}
dependencies {
sourceArchive "org.gnu:bash:4.2:src#tgz"
}
task buildFromSource(type: Exec) {
inputs.files configurations.sourceArchive.files
outputs.file file("${project.name}-${project.version}.tgz")
executable './build.sh'
def myArgs = configurations.sourceArchive.files.path
myArgs.add(0, outputs.files.asPath)
args myArgs
}
artifacts {
// Is this really the only way to transform a singleton collection
// into the singleton?
// def outputFile
// buildFromSource.outputs.files.each { outputFile = it }
// Nope: this is better magic:
def outputFile = buildFromSource.outputs.files.singleFile
println outputFile.path
binaryArchive file: outputFile, name: 'bash'
// binaryArchive file: file(buildFromSource.outputs.files.asPath), name: 'bash'
}
uploadArchives {
configuration = configurations.binaryArchive
repositories.mavenDeployer {
repository(url: "http://nexus/..") {
authentication(userName: "me", password: "secret!")
}
pom.groupId = 'org.gnu'
}
}
uploadArchives.dependsOn buildFromSource
The error I get is:
* What went wrong:
Execution failed for task ':uploadArchives'.
> Could not publish configuration 'binaryArchive'
> Error deploying artifact 'org.gnu:bash:tgz': Error deploying artifact: Failed to transfer file: http://nexus/.../org/gnu/bash/testarch-4.2/bash-testarch-4.2.tgz. Return code is: 400
Updated from comments, same error - trying to get access to the nexus logs for further debugging.
Error from Nexus is "missing entity", see: Missing Request Entity response to a PUT to Nexus
The root cause of my problem was I was testing with an empty file. Nexus doesn't like empty files. As soon as I put content in it, Nexus was happy and the code was working.

how to download external files in gradle?

I have a gradle project which requires some data files available somewhere on the internet using http. The goal is that this immutable remote file is pulled once upon first build. Subsequent build should not download again.
How can I instruct gradle to fetch the given file to a local directory?
I've tried
task fetch(type:Copy) {
from 'http://<myurl>'
into 'data'
}
but it seems that copy task type cannot deal with http.
Bonus question: is there a way to resume a previously aborted/interrupted download just like wget -c does?
How about just:
def f = new File('the file path')
if (!f.exists()) {
new URL('the url').withInputStream{ i -> f.withOutputStream{ it << i }}
}
You could probably use the Ant task Get for this. I believe this Ant task does not support resuming a download.
In order to do so, you can create a custom task with name MyDownload. That can be any class name basically. This custom task defines inputs and outputs that determine whether the task need to be executed. For example if the file was already downloaded to the specified directory then the task is marked UP-TO-DATE. Internally, this custom task uses the Ant task Get via the built-in AntBuilder.
With this custom task in place, you can create a new enhanced task of type MyDownload (your custom task class). This task set the input and output properties. If you want this task to be executed, hook it up to the task you usually run via task dependencies (dependsOn method). The following code snippet should give you the idea:
task downloadSomething(type: MyDownload) {
sourceUrl = 'http://www.someurl.com/my.zip'
target = new File('data')
}
someOtherTask.dependsOn downloadSomething
class MyDownload extends DefaultTask {
#Input
String sourceUrl
#OutputFile
File target
#TaskAction
void download() {
ant.get(src: sourceUrl, dest: target)
}
}
Try like that:
plugins {
id "de.undercouch.download" version "1.2"
}
apply plugin: 'java'
apply plugin: 'de.undercouch.download'
import de.undercouch.gradle.tasks.download.Download
task downloadFile(type: Download) {
src 'http://localhost:8081/example/test-jar-test_1.jar'
dest 'localDir'
}
You can check more here: https://github.com/michel-kraemer/gradle-download-task
For me works fine..
The suggestion in Ben Manes's comment has the advantage that it can take advantage of maven coordinates and maven dependency resolution. For example, for downloading a Derby jar:
Define a new configuration:
configurations {
derby
}
In the dependencies section, add a line for the custom configuration
dependencies {
derby "org.apache.derby:derby:10.12.1.1"
}
Then you can add a task which will pull down the right files when needed (while taking advantage of the maven cache):
task deployDependencies() << {
String derbyDir = "${some.dir}/derby"
new File(derbyDir).mkdirs();
configurations.derby.resolve().each { file ->
//Copy the file to the desired location
copy {
from file
into derbyDir
// Strip off version numbers
rename '(.+)-[\\.0-9]+\\.(.+)', '$1.$2'
}
}
}
(I learned this from https://jiraaya.wordpress.com/2014/06/05/download-non-jar-dependency-in-gradle/).
Using following plugin:
plugins {
id "de.undercouch.download" version "3.4.3"
}
For a task which has the purpose of only downloading
task downloadFile(type: Download) {
src DownloadURL
dest destDir
}
For including download option into your task:
download {
src DownloadURL
dest destDir
}
For including download option with multiple downloads into your task:
task downloadFromURLs(){
download {
src ([
DownloadURL1,
DownloadURL2,
DownloadURL3
])
dest destDir
}
}
Hope it helped :)
just now ran into post on upcoming download task on gradle forum.
Looks like the perfect solution to me.. Not (yet) available in an official gradle release though
Kotlin Version of #Benjamin's Answer
buildscript {
repositories {
jcenter()
google()
}
dependencies {
classpath("com.android.tools.build:gradle:4.0.1")
}
}
tasks.register("downloadPdf"){
val path = "myfile.pdf"
val sourceUrl = "https://file-examples-com.github.io/uploads/2017/10/file-sample_150kB.pdf"
download(sourceUrl,path)
}
fun download(url : String, path : String){
val destFile = File(path)
ant.invokeMethod("get", mapOf("src" to url, "dest" to destFile))
}

Resources