To get the incremental build support running correctly in gradle it is required to define the "inputs" and "outputs" in each (custom-) tasks.
That's a very neat way of gradle to check if a task can be skipped cause it is up-to-date or not. Sample:
task myTaskA {
def someOutputDir = file("$buildDir/gen")
// define task outputs (input not required here)
outputs.dir someOutputDir
doLast{
// generate something into the defined output directory
new File(someOutputDir, "dummy.txt").text = "Gradle sample"
}
}
task myTaskB {
// Input of this task is dependent on the output of myTaskA
inputs.files myTaskA
doLast{
// do something
}
}
Yes, that is pretty nice and additionally the good thing is, we do not need to declare a explicit dependency instruction (dependsOn) in task "myTaskB".
dependsOn myTaskA
This directive is not necessary cause we have a implicit dependency defined by the inputs-declaration.
I think it is a good style to provide always a inputs/outputs definition in custom tasks to support incremental builds.
BUT: This also means we can completely ignore the dependsOn.
SO: When should we prefer dependsOn over inputs/outputs?
Maybe if there is no inputs or outputs. Yes, but are there other use cases conceivable? I was always working with dependsOn, and this seams to me as obsolete now. What do you think?
Bye have a great day, Tony
Related
[ Updated 16 April 2021, see reproducible test case below.]
I started converting a small Gradle build to Gradle 7.0. It says:
> Task :junit_xsd
Execution optimizations have been disabled for task ':junit_xsd' to ensure correctness due to the following reasons:
- Gradle detected a problem with the following location: '/Users/ndw/Projects/xproc/test-suite/build'. Reason: Task ':git_log_shorter' uses this output of task ':junit_xsd' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to https://docs.gradle.org/7.0/userguide/validation_problems.html#implicit_dependency for more details about this problem.
I've looked at the documentation link and it wasn't very informative (to me).
The two tasks in question are, AFAICT, unrelated. junit_xsd is a simple copy task:
task junit_xsd(type: Copy) {
from "src/main/schema/"
into "build/"
include "*.xsd"
}
The git_log_shorter task is a bit harder to describe. It's a custom task that runs a process I wrote.
task git_log_shorter(dependsOn: ["git_log_summary"], type: XMLCalabashTask) {
inputs.file "tools/xsl/shorter-log.xsl"
inputs.file "build/git-log-summary.xml"
outputs.file "build/git-log-shorter.xml"
input("source", "build/git-log-summary.xml")
output("result", "build/git-log-shorter.xml")
pipeline "tools/xpl/run-xslt.xpl"
option("stylesheet", "../xsl/shorter-log.xsl")
}
But it doesn't read any xsd files, so I can't work out why Gradle thinks there's an implicit dependency on junit_xsd.
I mean, I suppose it reads out of the build directory, but I have lots of tasks that create and use temporary files from the build directory. I'll be a little shocked if any task that reads from the build directory is now implicitly dependent on every task that writes to it.
If that is the case, what's the proposed solution for temporary files?
The git_log_summary task, for example, is the task that writes build/git-log-summary.xml and I have an explicit dependency for that task.
Am I overlooking something obvious?
Here's a test case that doesn't involve any extension steps. Running the doall task produces the warning:
task doall(dependsOn: ["copyA", "pointless_gzip"]) {
// nop
}
task copyA(type: Copy) {
from "src/main/data/"
into "${buildDir}"
include "*.xsd"
}
task pointless_tar(type: Exec) {
outputs.file "${buildDir}/pointless.tar"
commandLine 'tar', '-cf', "${buildDir}/pointless.tar", 'src'
doFirst {
mkdir("${buildDir}")
}
}
task pointless_gzip(dependsOn: ["pointless_tar"], type: Exec) {
inputs.file "${buildDir}/pointless.tar"
outputs.file "${buildDir}/pointless.tar.gz"
commandLine 'gzip', '-k', "${buildDir}/pointless.tar"
}
Is there any difference between two following pieces of code?
May be there is some difference about eager initialization of a task?
tasks.bootJar {
archiveFileName = "some-name"
}
and
bootJar {
archiveFileName = "some-code"
}
They are the effectively the same just different syntax. Both cause the task to be realized or eager initialized. You can confirm this by simply running with debug output: ./gradlew -d
You will see the following line logged:
[DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Realize task :bootJar' started
For the Groovy DSL, the Gradle team takes advantage of the metaprogramming offered by the Groovy language to dynamically make tasks, extensions, properties, and more "magically" available.
So for this example:
tasks.bootJar {
archiveFileName = "some-name"
}
tasks is of type TaskContainer. If you were to examine all available methods and properties for TaskContainer, you will see that bootJar is not one of them. So Gradle hooks into the Groovy metaprogramming to search for something named bootJar. Since it's on the TaskContainer type, it can be assumed that bootJar is a task.
So if you were to desugar the DSL, the first example is effectively:
tasks.getByName("bootJar") {
}
Now for the second example:
bootJar {
archiveFileName = "some-code"
}
The default scope or this in a Gradle build file is Project. Again, just like before, if you were to examine all available methods and properties, you will not see bootJar as one of them.
Groovy's 'magic' is at play here, but this time around, Gradle will search (my understanding) practically everywhere because there is a lot more available on a Project.
project.extensions.named("bootJar") // exists?
project.hasProperty("bootJar") // exists?
project.tasks.getByName("bootJar") // found it!
To summarize, no there is no difference between the two since both cause the bootJar task to be realized. Use task configuration avoidance wherever possible to make your Gradle builds more efficient:
import org.springframework.boot.gradle.tasks.bundling.BootJar
tasks.named("bootJar", BootJar) {
}
I have a tar file that contains multiple tar.gz files (a Docker image), and I want to extract a single file from it. Doing this is in a shell script is trivial, but it seems a bit tricky when using Gradle.
This is what I have so far:
task extractOuter(type: Copy) {
dependsOn jibBuildTar
from tarTree(file("${buildDir}/jib-image.tar"))
include "*.tar.gz"
into "${buildDir}/tgz"
}
task extractInner(type: Copy) {
dependsOn extractOuter
from (fileTree(dir: "${buildDir}/tgz").collect { tarTree(it) }) {
include "**/filename"
includeEmptyDirs = false
}
into "${buildDir}/files"
}
It seemed to work at first, but it turned out that it fails occasionally: the extractInner task does not find the file. Maybe I don't use Gradle's lazy evaluation correctlty.
How to make it work? Or is there totally different, more elegant way?
Doing this is in a shell script is trivial
You can continue using the shell script by using the Exec task type.
but it seems a bit tricky when using Gradle.
What you have so far is how you do it with Gradle. The advantage with Gradle is that it won't perform work that has already happened. See Build Cache for more details.
It seemed to work at first, but it turned out that it fails occasionally: the extractInner task does not find the file. Maybe I don't use Gradle's lazy evaluation correctlty.
This is called out in the above linked docs (emphasis mine):
Some tasks, like Copy or Jar, usually do not make sense to make cacheable because Gradle is only copying files from one location to another. It also doesn’t make sense to make tasks cacheable that do not produce outputs or have no task actions.
So you've declared your tasks, but you haven't configured them to produce any outputs which may or may not contribute to the problem since you expect the output to be present for a task dependency.
Since Copy extends DefaultTask, you can use the outputs to set the task output.
task extractOuter(type: Copy) {
dependsOn jibBuildTar
outputs.dir(file("$buildDir/tgz")
from tarTree(file("${buildDir}/jib-image.tar"))
include "*.tar.gz"
into "${buildDir}/tgz"
}
task extractInner(type: Copy) {
dependsOn extractOuter
outputs.dir(file("$buildDir/files")
from (fileTree(dir: "${buildDir}/tgz").collect { tarTree(it) }) {
include "**/filename"
includeEmptyDirs = false
}
into "${buildDir}/files"
}
Is It possible to configure inputs of a gradle task at runtime after other tasks have run?
For example I am calculating a SHA of a zip in one step, and then uploading the zip with a path consisting of the SHA from a previous step. But when I got get get the value of the SHA which is contained in a file via: def sha = shaFile.text I get an error: (No such file or directory).
I had always assumed tasks were closures which were run at runtime but I guess that is just the doFirst & doLast, but the inputs need to be configured already before that.
Is It possible to configure inputs of a gradle task at runtime after other tasks have run?
Think of it this way:
In order for task B to run, task A must run first, that is to say, task B has a dependency on task A.
Refer to Adding dependencies to a task for more details on task dependencies.
Ok so now we're at the point where we need the output of task A (SHA value) as an input for task B. Since we have a dependency on task A, Gradle well make sure that task A is executed prior to B's execution.
Here's quick dirty example in Kotlin DSL (should be easily translated to Groovy):
tasks {
val taskA = register("taskA") {
val shaText = File("sha.txt")
if (shaText.exists()) {
shaText.delete()
}
File("sha.txt").writeText("abc");
}
register("taskB") {
dependsOn(taskA)
println(File("sha.txt").readText())
}
}
Ideally, you should create a custom task type specifying the input file and also specifying the output file so that Gradle can cache tasks inputs/outputs. Refer to Incremental tasks
for more details.
I've prepared a very simple script, that illustrates the problem I see using Gradle 1.7 (need to stick with it because of some plugins not yet supporting newer versions).
I'm trying to dynamically create tasks each of which corresponds to a file in the project directory. This works fine, but the tasks I create never get executed as soon as I assign them type 'Copy'.
Here is my problem build.gradle:
file('templates').listFiles().each { File f ->
// THIS LINE DOES NOT WORK
task "myDist-${f.name}" (type: Copy) {
// NEXT LINE WORKS
//task "myDist-${f.name}" {
doLast {
println "MYDIST-" + f.name
}
}
}
task distAll(dependsOn: tasks.matching { Task task -> task.name.startsWith("myDist")}) {
println "MYDISTALL"
}
defaultTasks 'distAll'
in this way my tasks do not get executed when I call default task calling simply gradle:
MYDISTALL
:myDist-template1 UP-TO-DATE
:myDist-template2 UP-TO-DATE
:distAll UP-TO-DATE
BUILD SUCCESSFUL
If I remove type Copy from my dynamic task (uncommenting the line above), my tasks get executed:
MYDISTALL
:myDist-template1
MYDIST-template1
:myDist-template2
MYDIST-template2
:distAll
BUILD SUCCESSFUL
(You'll need to create a folder name templates in the same directory where build.gradle is located and put couple of empty files into there in order to run the test)
According to the debug output:
Skipping task ':myDist-template1' as it has no source files.
Skipping task ':myDist-template2' as it has no source files.
So how can I specify source files and make my Copy tasks execute?
I've tried adding
from( '/absolute/path/to/existing/file' ) {
into 'myfolder'
}
to the task body, I've tried assigning task's inputs.source file('/my/existing/file') with no success.
Could you please advise on how to modify my simple script leaving dynamic task creation and keeping my dynamic tasks of type Copy?
Thank you!
Edit:
All right, this way the task gets called:
file('templates').listFiles().each { File f ->
task "myDist-${f.name}" (type: Copy) {
from f
into 'dist'
doLast {
println "MYDIST-" + f.name
}
}
}
but it looks I must always specify from/into. It doesn't suffice to do that in the doLast{} body.
A Copy task only gets executed if it has something to copy. Telling it what to copy is part of configuring the task, and therefore needs to be done in the configuration phase, rather than the execution phase. These are very important concepts to understand, and you can read up on them in the Gradle User Guide or on the Gradle Forums.
doFirst and doLast blocks get executed in the execution phase, as part of executing the task. Both are too late to tell the task what to copy: doFirst gets executed immediately before the main task action (which in this case is the copying), but (shortly) after the skipped and up-to-date checks (which are based on the task's configuration). doLast gets executed after the main task action, and is therefore clearly too late.
I think the following Gradle User Guide quote answers my question the best:
Secondly, the copy() method can not honor task dependencies when a task is used as a copy source (i.e. as an argument to from()) because it's a method and not a task. As such, if you are using the copy() method as part of a task action, you must explicitly declare all inputs and outputs in order to get the correct behavior.
Having read most of the answers to "UP-TO-DATE" Copy tasks in gradle, it appears that the missing part is 'include' keyword:
task copy3rdPartyLibs(type: Copy) {
from 'src/main/jni/libs/'
into 'src/main/libs/armeabi/'
include '**/*.so'
}
Putting from and into as part of the doLast section does not work. An example of a working task definitions is:
task copyMyFile(type: Copy) {
def dockerFile = 'src/main/docker/Dockerfile'
def copyTo = 'build/docker'
from dockerFile
into copyTo
doLast {
println "Copied Docker file [$dockerFile] to [$copyTo]"
}
}
Not the behavior I was expecting.
Using gradle 3.2.1