Extract specific JARs from dependencies - gradle

I am new to gradle but learning quickly. I need to get some specific JARs from logback into a new directory in my release task. The dependencies are resolving OK, but I can't figure out how, in the release task, to extract just logback-core-1.0.6.jar and logback-access-1.0.6.jar into a directory called 'lib/ext'. Here are the relevant excerpts from my build.gradle.
dependencies {
...
compile 'org.slf4j:slf4j-api:1.6.4'
compile 'ch.qos.logback:logback-core:1.0.6'
compile 'ch.qos.logback:logback-classic:1.0.6'
runtime 'ch.qos.logback:logback-access:1.0.6'
...
}
...
task release(type: Tar, dependsOn: war) {
extension = "tar.gz"
classifier = project.classifier
compression = Compression.GZIP
into('lib') {
from configurations.release.files
from configurations.providedCompile.files
}
into('lib/ext') {
// TODO: Right here I want to extract just logback-core-1.0.6.jar and logback-access-1.0.6.jar
}
...
}
How do I iterated over the dependencies to locate those specific files and drop them in the lib/ext directory created by into('lib/ext')?

Configurations are just (lazy) collections. You can iterate over them, filter them, etc. Note that you typically only want to do this in the execution phase of the build, not in the configuration phase. The code below achieves this by using the lazy FileCollection.filter() method. Another approach would have been to pass a closure to the Tar.from() method.
task release(type: Tar, dependsOn: war) {
...
into('lib/ext') {
from findJar('logback-core')
from findJar('logback-access')
}
}
def findJar(prefix) {
configurations.runtime.filter { it.name.startsWith(prefix) }
}

It is worth nothing that the accepted answer filters the Configuration as a FileCollection so within the collection you can only access the attributes of a file. If you want to filter on the dependency itself (on group, name, or version) rather than its filename in the cache then you can use something like:
task copyToLib(type: Copy) {
from findJarsByGroup(configurations.compile, 'org.apache.avro')
into "$buildSrc/lib"
}
def findJarsByGroup(Configuration config, groupName) {
configurations.compile.files { it.group.equals(groupName) }
}
files takes a dependencySpecClosure which is just a filter function on a Dependency, see: https://gradle.org/docs/current/javadoc/org/gradle/api/artifacts/Dependency.html

Related

Gradle: process files inside a zip inside a tar

I have the following use case: I want to untar a tar file which contains a zip file. I then want to process files in the zip file (let's say I just want to filter out some file).
What I'm trying to do is something like:
task extractTar(type: Copy) {
from(tarTree("../a.tgz")) {
include '**/*.zip'
}
into "${buildDir}/tarOutput"
}
task unzipAndRezipZip(type: Zip, dependsOn: extractTar) {
archiveFileName = "rezipped.zip"
destinationDirectory
from(fileTree("${buildDir}/tarOutput").find { it.name.endsWith("zip") }) {
exclude '**/b.txt'
}
}
I've spent many many hours on this and every way I try to do it, it fails some way or another.
Expected Behavior
I can use the output files from a CopyTask in my ZipTask.
Current Behavior
path may not be null or empty string. path='null'
or
Circular dependency between the following tasks:
:unzipAndRezipZip
\--- :unzipAndRezipZip (*)
Steps to Reproduce
I have made a repo which has three examples of different things I tried: https://github.com/viktornordling/gradle-unzip-zip-from-tar-issue and clearly shows the repro steps for each example.
(I also put this in https://github.com/gradle/gradle/issues/16413, but I'm not sure how much love it will get there. Cross posting here to see if I can get some traction.)
Update: I would prefer not to use doLast in the solution as I feel that should not be needed. Usage of doLast is explicitly discouraged in the docs and not mentioned in the docs of CopyTask.
In example-4 from your repo https://github.com/viktornordling/gradle-unzip-zip-from-tar-issue you state that it almost works, but fails when you run clean unzipAndRezipZip in one go, with the error message Cannot expand ZIP...:
task unzipAndRezipZip(type: Zip) {
archiveFileName = "rezipped.zip"
destinationDir(file("${buildDir}/zipOutput"))
from(zipTree(tarTree("../a.tgz").find { it.name.endsWith("zip") })) {
exclude '**/b.txt'
}
}
You can fix that by changing the call to from() to a closure from { } to delay the execution of zipTree, i.e.:
task unzipAndRezipZip(type: Zip) {
archiveFileName = "rezipped.zip"
destinationDir(file("${buildDir}/zipOutput"))
from { zipTree(tarTree("../a.tgz").find { it.name.endsWith("zip") }) } {
exclude '**/b.txt'
}
}

Can Gradle generate a POM in such a way that it doesn't unnecessarily touch the output file?

We're creating a POM file like so:
task createPom {
doLast {
pom {
project {
groupId project.group
artifactId project.name
version project.ext.pomVersion
}
}.writeTo("${buildDir}/pom.xml")
}
}
This writeTo eagerly writes to the file even if no changes have occurred, which causes the jar to change, so later expensive tasks in the build (indexing, signing, building installers) all have to run as well.
I thought about writing to a temp file and diffing the two files somehow, but it seems like I'd have to write a lot of boilerplate just to do that, so I'm wondering whether there is a proper way to go about it.
What you need to do is to configure task's inputs and outputs which are used to determine whether your task is up-to-date or not. Here, how it should be done:
apply plugin: 'maven'
ext {
pomVersion = '3.1.4.5'
}
group = 'lolgroup'
task createPom {
outputs.file "${buildDir}/pom.xml"
inputs.property('group', project.group)
inputs.property('name', project.name)
inputs.property('pomVersion', project.ext.pomVersion)
doLast {
pom {
project {
groupId inputs.properties['group']
artifactId inputs.properties['name']
version inputs.properties['pomVersion']
}
}.writeTo("${buildDir}/pom.xml")
}
}
Here you can find a demo.

Gradle copy build script dependency to folder

We are building one of our applications with gradle and part of the distribution I want to include an external jar which is not a run time dependency in a config folder. That jar is needed as part of the application install and it contains some custom ant tasks.
Our build script dependency looks like below:
buildscript {
...
dependencies {
classpath 'org.jfrog.buildinfo:build-info-extractor-gradle:3.1.1'
classpath 'my-group:custom-tasks:1.2.3'
}
}
How would I access and copy the custom-task-1.2.3.jar into a certain folder so I can include it in my distribution? Something like below:
task copyCustomTasks {
doLast {
copy {
// This below is a make up to express what I want
from buildscript.dependencies
include 'custom-tasks*.jar'
into "$buildDir/config"
}
}
}
If this is not the gradle way of doing things please let me know what alternatives I have.
Thank you in advance for your help.
UPDATE
I solved my problem in a different way by creating an extra configuration. However I would still be interested to find out how you can access build script dependencies at run time. Thanks again for your inputs.
configurations {
install {
description = "application install classpath"
transitive = true
}
}
...
dependencies {
...
install('my-group:custom-tasks:1.2.3')
...
}
...
task copyInstallDeps {
doLast {
copy {
from configurations.install
into "$buildDir/config"
}
}
}
You're quite close:
buildscript {
repositories {
mavenCentral()
}
dependencies {
classpath 'com.google.guava:guava:18.0'
}
}
task copyLibs(type: Copy) {
from buildscript.configurations.classpath
into 'lib'
}

How can I define two different 'distribution' tasks in gradle?

The normal behavior of the distTar and distZip tasks from the application plugin in gradle seems to be to copy the contents of src/dist into the zip and tar files, but I have a subfolder in src/dist that I want to exclude from the default distribution, and include it for a new (extended) task, possibly to be called distZipWithJRE.
I have been able to exclude this folder in the default task as follows:
distributions.main {
contents {
from('build/config/main') {
into('config')
}
from('../src/dist') {
exclude('jre')
}
}
}
How can I define a second task that behaves just like the original (unmodified) task?
Using Gradle 4.8 I had to tweak the answer to use 'with' from CopySpec instead
distributions {
zipWithJRE {
baseName = 'zipWithJRE'
contents {
with distributions.main.contents
}
}
}
It seems that what you're looking for is in the docs. You need to leave current settings as is and for zipWithJRE create and configure custom distribution:
distributions {
zipWithJRE {
baseName = 'zipWithJRE'
contents {
from { distributions.main.contents }
}
}
}

how to download external files in gradle?

I have a gradle project which requires some data files available somewhere on the internet using http. The goal is that this immutable remote file is pulled once upon first build. Subsequent build should not download again.
How can I instruct gradle to fetch the given file to a local directory?
I've tried
task fetch(type:Copy) {
from 'http://<myurl>'
into 'data'
}
but it seems that copy task type cannot deal with http.
Bonus question: is there a way to resume a previously aborted/interrupted download just like wget -c does?
How about just:
def f = new File('the file path')
if (!f.exists()) {
new URL('the url').withInputStream{ i -> f.withOutputStream{ it << i }}
}
You could probably use the Ant task Get for this. I believe this Ant task does not support resuming a download.
In order to do so, you can create a custom task with name MyDownload. That can be any class name basically. This custom task defines inputs and outputs that determine whether the task need to be executed. For example if the file was already downloaded to the specified directory then the task is marked UP-TO-DATE. Internally, this custom task uses the Ant task Get via the built-in AntBuilder.
With this custom task in place, you can create a new enhanced task of type MyDownload (your custom task class). This task set the input and output properties. If you want this task to be executed, hook it up to the task you usually run via task dependencies (dependsOn method). The following code snippet should give you the idea:
task downloadSomething(type: MyDownload) {
sourceUrl = 'http://www.someurl.com/my.zip'
target = new File('data')
}
someOtherTask.dependsOn downloadSomething
class MyDownload extends DefaultTask {
#Input
String sourceUrl
#OutputFile
File target
#TaskAction
void download() {
ant.get(src: sourceUrl, dest: target)
}
}
Try like that:
plugins {
id "de.undercouch.download" version "1.2"
}
apply plugin: 'java'
apply plugin: 'de.undercouch.download'
import de.undercouch.gradle.tasks.download.Download
task downloadFile(type: Download) {
src 'http://localhost:8081/example/test-jar-test_1.jar'
dest 'localDir'
}
You can check more here: https://github.com/michel-kraemer/gradle-download-task
For me works fine..
The suggestion in Ben Manes's comment has the advantage that it can take advantage of maven coordinates and maven dependency resolution. For example, for downloading a Derby jar:
Define a new configuration:
configurations {
derby
}
In the dependencies section, add a line for the custom configuration
dependencies {
derby "org.apache.derby:derby:10.12.1.1"
}
Then you can add a task which will pull down the right files when needed (while taking advantage of the maven cache):
task deployDependencies() << {
String derbyDir = "${some.dir}/derby"
new File(derbyDir).mkdirs();
configurations.derby.resolve().each { file ->
//Copy the file to the desired location
copy {
from file
into derbyDir
// Strip off version numbers
rename '(.+)-[\\.0-9]+\\.(.+)', '$1.$2'
}
}
}
(I learned this from https://jiraaya.wordpress.com/2014/06/05/download-non-jar-dependency-in-gradle/).
Using following plugin:
plugins {
id "de.undercouch.download" version "3.4.3"
}
For a task which has the purpose of only downloading
task downloadFile(type: Download) {
src DownloadURL
dest destDir
}
For including download option into your task:
download {
src DownloadURL
dest destDir
}
For including download option with multiple downloads into your task:
task downloadFromURLs(){
download {
src ([
DownloadURL1,
DownloadURL2,
DownloadURL3
])
dest destDir
}
}
Hope it helped :)
just now ran into post on upcoming download task on gradle forum.
Looks like the perfect solution to me.. Not (yet) available in an official gradle release though
Kotlin Version of #Benjamin's Answer
buildscript {
repositories {
jcenter()
google()
}
dependencies {
classpath("com.android.tools.build:gradle:4.0.1")
}
}
tasks.register("downloadPdf"){
val path = "myfile.pdf"
val sourceUrl = "https://file-examples-com.github.io/uploads/2017/10/file-sample_150kB.pdf"
download(sourceUrl,path)
}
fun download(url : String, path : String){
val destFile = File(path)
ant.invokeMethod("get", mapOf("src" to url, "dest" to destFile))
}

Resources