Is It possible to configure inputs of a gradle task at runtime after other tasks have run?
For example I am calculating a SHA of a zip in one step, and then uploading the zip with a path consisting of the SHA from a previous step. But when I got get get the value of the SHA which is contained in a file via: def sha = shaFile.text I get an error: (No such file or directory).
I had always assumed tasks were closures which were run at runtime but I guess that is just the doFirst & doLast, but the inputs need to be configured already before that.
Is It possible to configure inputs of a gradle task at runtime after other tasks have run?
Think of it this way:
In order for task B to run, task A must run first, that is to say, task B has a dependency on task A.
Refer to Adding dependencies to a task for more details on task dependencies.
Ok so now we're at the point where we need the output of task A (SHA value) as an input for task B. Since we have a dependency on task A, Gradle well make sure that task A is executed prior to B's execution.
Here's quick dirty example in Kotlin DSL (should be easily translated to Groovy):
tasks {
val taskA = register("taskA") {
val shaText = File("sha.txt")
if (shaText.exists()) {
shaText.delete()
}
File("sha.txt").writeText("abc");
}
register("taskB") {
dependsOn(taskA)
println(File("sha.txt").readText())
}
}
Ideally, you should create a custom task type specifying the input file and also specifying the output file so that Gradle can cache tasks inputs/outputs. Refer to Incremental tasks
for more details.
Related
I have created a task in Gradle as follows:
task findTaskCreatingSpecificOutput() {
dependsOn testClasses
doLast {
tasks.findAll { task ->
task.outputs.getFiles().getFiles().each { output ->
if (output != null && output.getAbsolutePath().contains('generated')) {
println task
println output
}
}
}
}
}
to try and find which two tasks are writing to the same generated location. However the output is a list of tasks and directories which do not overlap at all so it should be easy for Gradle to know exactly which task created the location.
Example output of ./gradlew clean findTaskCreatingSpecificOutput --info
/Users/mylocation/shared/build/generated/sources/annotationProcessor/java/main
task ':shared:compileJava'
/Users/mylocation/shared/build/generated/sources/headers/java/main
task ':shared:compileTestFixturesJava'
/Users/mylocation/shared/build/generated/sources/annotationProcessor/java/testFixtures
task ':shared:compileTestFixturesJava'
/Users/mylocation/shared/build/generated/sources/headers/java/testFixtures
task ':shared:compileTestJava'
/Users/mylocation/shared/build/generated/sources/annotationProcessor/java/test
task ':shared:compileTestJava'
/Users/mylocation/shared/build/generated/sources/headers/java/test
task ':shared:delombok'
/Users/mylocation/shared/build/generated/sources/delombok/java/main
task ':shared:delombokTest'
/Users/mylocation/shared/build/generated/sources/delombok/java/test
task ':shared:delombokTestFixtures'
/Users/mylocation/shared/build/generated/sources/delombok/java/testFixtures
However I still get the warning Gradle does not know how file 'build/classes/java/main/generated' was created (output property 'destinationDirectory'). Task output caching requires exclusive access to output paths to guarantee correctness (i.e. multiple tasks are not allowed to produce output in the same location).
I don't think there's any way to find out which task actually wrote any particular file. The code I used only lists the declared outputs.
In my case the problem was that I was running a bytecode weaving task in doLast for compileJava.
I've changed compileJava to write to a new directory using destinationDirectory and then the weaving task writes the updated classes to sourceSets.main.output.classesDirs.singleFile
I am trying to do something like this:
jar {
doLast{
from "build/libs/TheJar.jar"
into "."
}
}
So far, I have tried various tutorials including all forms from this answer but non have worked. The only thing that works is calling a separate task but I'd like to know why my construction is wrong and why can't I run something after the jar or shadowJar tasks.
It looks like you took some parts of the answers in the linked post and somehow mixed them without knowing what your final code actually does.
Tasks in Gradle may have a type (e.g. Copy, Jar, ...). This type defines what the task will do once it gets executed (so called task actions). A task without a type won't do anything when its executed, unless you add task actions manually. Using doFirst will add the passed action (also called closure) to the start of the list of task actions, using doLast will add it to the end of the list.
Everything outside of doFirst and doLast closures is not part of the execution of the task, but can be used to configure the task:
task example {
doLast {
println "second action"
}
doFirst {
println "first action"
}
println "configuration"
}
Run the code above with gradle example and you will see the order of the log messages as configuration, first action, second action.
Task configuration will run, even if the task won't be executed later on. Even if you call gradle (without any task names as arguments), the console will still print configuration. This was the actual problem in the linked question.
Lets come to the real question: How to copy a file?
Well, Gradle offers two ways to copy a file. The first one is the task type Copy. You can create a task based on this type, apply your configuration and then either call it directly from the command line or define task dependencies using dependsOn or finalizedBy:
task copySomeFiles(type: Copy) {
from '...'
into '...'
}
However, you don't want to create an additional task. Gradle also has a method called copy that may be called anywhere in your script and will instantly copy the files:
copy {
from '...'
into '...'
}
The code above will copy your files, but it will copy your files every time Gradle executes (which may be often, e.g. when using an IDE). To solve this problem, you may move your code with copy to a task action, e.g. inside a doFirst or doLast closure:
jar {
doLast {
copy {
from "build/libs/TheJar.jar"
into "."
}
}
}
As you can see, your code was missing the copy statement. Now, whenever your task jar gets executed, its last task action will copy the files as intended.
Bonus question: Why is there no error?
The "problem" in your case is that your code is perfectly valid Gradle code. The task jar is of type Jar. Every task of type Jar may be configured using methods called from and into. The method from adds files to the JAR file and the method into sets the destination directory inside the JAR. So instead of copying, your code configures the underlying task jar. However, this has no negative consequences, as this configuration gets applied inside doLast, which only runs once the JAR file has already been created. Something that already happened cannot be configured.
I am using the gradle application plugin.
I noticed that:
distZip task get's run before the assemble task.
And I can see
from the docs (link above), and confirmed during my gradle run, that
the distZip tasks depends on: jar, startScripts tasks
So execution looks something like this:
> Task :my-app:processResources
> Task :my-app:compileJava
> Task :my-app:classes
> Task :my-app:jar
> Task :my-app:startScripts
> Task :my-app:distTar
> Task :my-app:distZip
> Task :my-app:assemble
Looking through the code in ApplicationPlugin.java and/or DistributionPlugin.java, I expected to see the task dependencies defined.
e.g. something like this:
distZipTask.dependsOn(jarTask)
distZipTask.dependsOn(createScriptsTask)
assembleTask.dependsOn(distZipTask)
...but I couldn't find anything like this in the java code.
Would very much appreciate it if someone could point me in the direction of where to look to find out how these tasks are 'linked'.
Gradle can determine implicit task dependencies from task inputs. If (the output of) a task is used as input of another task, Gradle can derive that there must be a dependency on the task. This functionality is called incremental build support, as it also supports checking for changes in the inputs or outputs to skip tasks if the inputs did not change.
Regarding your example, the task dependencies are defined implicitly in line 197 and line 208 of the file ApplicationPlugin.java:
libChildSpec.from(jar);
...
binChildSpec.from(startScripts);
The arguments in these lines are the respective tasks (to be exact: providers of the tasks). The from method can often be used to define sources of a file operation (e.g. copying, zipping ...). So we define the results (outputs) of the tasks jar and startScripts as a source of some operation (CopySpec). The resulting CopySpec objects are then passed to both tasks distZip and distTar. When a task tries to resolve the defined sources it finds the tasks jar and startScripts and defines the dependencies on them on its own.
Please note that the mentioned task assemble is a so-called lifecycle task. It does not run any action but can be used to organize tasks into build phases (just like build and check).
Let's say that I have a task "main" that depends on another task "dependency." I would like "main" to be rerun any time its dependency (or its dependency's dependencies) is rebuilt because "main" relies on the artifacts produced by "dependency" (or the dependencies of "dependency").
A build.gradle file containing an example of what I'm dealing with is the following:
defaultTasks 'main'
task baseDependency {
outputs.file 'deps.out'
outputs.upToDateWhen { false }
doLast {
exec {
commandLine 'bash', '-c', 'echo hello world > deps.out'
}
}
}
task dependency(dependsOn: baseDependency)
task main(dependsOn: dependency) {
outputs.file 'main.out'
doLast {
exec {
commandLine 'bash', '-c', 'echo hello world > main.out'
}
}
}
Executing gradle the first time:
:baseDependency
:dependency
:main
BUILD SUCCESSFUL
Total time: 0.623 secs
Executing it a second time:
:baseDependency
:dependency
:main UP-TO-DATE
BUILD SUCCESSFUL
Total time: 0.709 secs
I would really love if "main" were not marked "UP-TO-DATE" if its dependencies had to be rebuilt. That seems essential. How do you make sure that's the case?
The standard way to specify dependency between tasks is via task's inputs and outputs. This can be any file, fileset or directory.
In your case you should modify main task and add inputs.file 'deps.out' to its definition.
Note that gradle has an optimization that may lead to unexpected behavior in a simplistic example that you provided.
Before a task is executed for the first time, Gradle takes a snapshot
of the inputs. This snapshot contains the set of input files and a
hash of the contents of each file. Gradle then executes the task. If
the task completes successfully, Gradle takes a snapshot of the
outputs. This snapshot contains the set of output files and a hash of
the contents of each file. Gradle persists both snapshots for the next
time the task is executed.
Each time after that, before the task is executed, Gradle takes a new
snapshot of the inputs and outputs. If the new snapshots are the same
as the previous snapshots, Gradle assumes that the outputs are up to
date and skips the task. If they are not the same, Gradle executes the
task. Gradle persists both snapshots for the next time the task is
executed.
So even if you specify correct inputs in a simple example where the same file is generated the dependent task will be marked as up to date on the second and subsequent runs.
If you do not want or cannot hardcode dependency on the file you can override upToDateWhen for dependent task and calculate the condition if the task is up to date based on dependencies of this task and their state like this:
outputs.upToDateWhen { task ->
task.taskDependencies.inject(true) { r, dep ->
r && dep.values.inject(true) { res, v ->
res && (!(v instanceof Task) || v?.state.getSkipped())
}
}
}
upToDateWhen should return true if this task should not be run at all (because its output are already up-to-date). And this is the case when no dependent task was run (the gradle documentation is a bit vague about this I must admit but getSkipped seems work as expected).
I've prepared a very simple script, that illustrates the problem I see using Gradle 1.7 (need to stick with it because of some plugins not yet supporting newer versions).
I'm trying to dynamically create tasks each of which corresponds to a file in the project directory. This works fine, but the tasks I create never get executed as soon as I assign them type 'Copy'.
Here is my problem build.gradle:
file('templates').listFiles().each { File f ->
// THIS LINE DOES NOT WORK
task "myDist-${f.name}" (type: Copy) {
// NEXT LINE WORKS
//task "myDist-${f.name}" {
doLast {
println "MYDIST-" + f.name
}
}
}
task distAll(dependsOn: tasks.matching { Task task -> task.name.startsWith("myDist")}) {
println "MYDISTALL"
}
defaultTasks 'distAll'
in this way my tasks do not get executed when I call default task calling simply gradle:
MYDISTALL
:myDist-template1 UP-TO-DATE
:myDist-template2 UP-TO-DATE
:distAll UP-TO-DATE
BUILD SUCCESSFUL
If I remove type Copy from my dynamic task (uncommenting the line above), my tasks get executed:
MYDISTALL
:myDist-template1
MYDIST-template1
:myDist-template2
MYDIST-template2
:distAll
BUILD SUCCESSFUL
(You'll need to create a folder name templates in the same directory where build.gradle is located and put couple of empty files into there in order to run the test)
According to the debug output:
Skipping task ':myDist-template1' as it has no source files.
Skipping task ':myDist-template2' as it has no source files.
So how can I specify source files and make my Copy tasks execute?
I've tried adding
from( '/absolute/path/to/existing/file' ) {
into 'myfolder'
}
to the task body, I've tried assigning task's inputs.source file('/my/existing/file') with no success.
Could you please advise on how to modify my simple script leaving dynamic task creation and keeping my dynamic tasks of type Copy?
Thank you!
Edit:
All right, this way the task gets called:
file('templates').listFiles().each { File f ->
task "myDist-${f.name}" (type: Copy) {
from f
into 'dist'
doLast {
println "MYDIST-" + f.name
}
}
}
but it looks I must always specify from/into. It doesn't suffice to do that in the doLast{} body.
A Copy task only gets executed if it has something to copy. Telling it what to copy is part of configuring the task, and therefore needs to be done in the configuration phase, rather than the execution phase. These are very important concepts to understand, and you can read up on them in the Gradle User Guide or on the Gradle Forums.
doFirst and doLast blocks get executed in the execution phase, as part of executing the task. Both are too late to tell the task what to copy: doFirst gets executed immediately before the main task action (which in this case is the copying), but (shortly) after the skipped and up-to-date checks (which are based on the task's configuration). doLast gets executed after the main task action, and is therefore clearly too late.
I think the following Gradle User Guide quote answers my question the best:
Secondly, the copy() method can not honor task dependencies when a task is used as a copy source (i.e. as an argument to from()) because it's a method and not a task. As such, if you are using the copy() method as part of a task action, you must explicitly declare all inputs and outputs in order to get the correct behavior.
Having read most of the answers to "UP-TO-DATE" Copy tasks in gradle, it appears that the missing part is 'include' keyword:
task copy3rdPartyLibs(type: Copy) {
from 'src/main/jni/libs/'
into 'src/main/libs/armeabi/'
include '**/*.so'
}
Putting from and into as part of the doLast section does not work. An example of a working task definitions is:
task copyMyFile(type: Copy) {
def dockerFile = 'src/main/docker/Dockerfile'
def copyTo = 'build/docker'
from dockerFile
into copyTo
doLast {
println "Copied Docker file [$dockerFile] to [$copyTo]"
}
}
Not the behavior I was expecting.
Using gradle 3.2.1