Incremental build support for output directory considering added files - gradle

The Gradle documentation tells this:
Note that if a task has an output directory specified, any files added to that directory since the last time it was executed are ignored and will NOT cause the task to be out of date. This is so unrelated tasks may share an output directory without interfering with each other. If this is not the behaviour you want for some reason, consider using TaskOutputs.upToDateWhen(groovy.lang.Closure)
Question: How does the solution with upToDateWhen look like (so that added files are considered). The main problem is that one has to access the build cache to retrieve the output directory content hash the last time the task ran.

Not sure if I understand the question correctly or why you mention the build cache. I assume you are not aware that the predicates added with upToDateWhen() are considered in addition to any other up-to-date checks like the ones added with TaskOutputs.dir()?
Take the following sample task:
task foo {
def outDir = file('out')
outputs.dir(outDir)
outputs.upToDateWhen { outDir.listFiles().length == 1 }
doLast {
new File(outDir, 'foo.txt') << 'whatever'
}
}
As long as there is only a single file in the output directory (as configured via upToDateWhen) and the file produced by the task (out/foo.txt) isn’t changed after the task has run, the task will be up-to-date. If you change/remove the file created by the task in the output directory or if you add further files to the output directory, then the task will run again.
Updated answer as per the updated question from the comments:
task foo {
def outDir = file('out')
/* sample task action: */
doFirst {
def numOutFiles = new Random().nextInt(5)
for (int i = 1; i <= numOutFiles; i++) {
new File(outDir, "foo${i}.txt") << 'whatever'
}
}
/* up-to-date checking configuration: */
def counterFile = new File(buildDir, 'counterFile.txt')
outputs.dir(outDir)
outputs.upToDateWhen {
counterFile.isFile() \
&& counterFile.text as Integer == countFiles(outDir)
}
doLast {
counterFile.text = countFiles(outDir)
}
}
def countFiles(def dir) {
def result = 0
def files = dir.listFiles()
if (files != null) {
files.each {
result++
if (it.isDirectory()) {
result += countFiles(it)
}
}
}
result
}

Related

Grooviest way in Gradle to list subdirectories matching a certain pattern

In my Gradle script, I would like to include directories matching a certain pattern in my source set. I need to search and scan for all subdirectories and include all subdirectories:
Any second level directories that have Java or Groovy files in them (could be deeply nested inside the package folder structure).
And/or any sub/directories that have a certain name e.g. api or source
The problem is that srcDirs under java sourceSet only accepts directories and not files.
What I've tried so far:
def javaSourceDirs = []
project.rootDir.eachFileRecurse(FileType.DIRECTORIES) { file ->
if ((file.getName().equalsIgnoreCase("api") || file.getName().equalsIgnoreCase("sources")))
javaSourceDirs << file
}
sourceSets {
main {
java {
srcDirs = javaSourceDirs
};
}
}
Just wondering if there's a better/cleaner way using fileTree or something similar?
This should work:
def findAllUnder = { File f, String fileOrDir, String pattern ->
def result = [ ]
def recursiveCall
switch ( fileOrDir ) {
case 'file': recursiveCall = f.&eachFileRecurse
break
case 'dir': recursiveCall = f.&eachDirRecurse
break
default: throw new RuntimeException( "Must be file or dir: $fileOrDir" )
}
recursiveCall { File path -> if ( path.name ==~ pattern ) result << path }
return result
}
def hasSources = { File f -> f.directory && findAllUnder( f, 'file', /.*\.(groovy|java)/ ) }
def sourceDirs = findAllUnder( project.rootDir, 'dir', '(?i)api|sources' ).findAll( hasSources )
It will return a List<File> with all dirs matching the names api or sources (case-insensitive) which have at least one groovy or java file under it.
It does not check for nested dirs, so if you're unlucky and have a dir with sources called groovy/api/com/api/, both groovy/api and groovy/api/com/api will be returned.

Call one Gradle Task from another with arguments

I have a gradle task which calls a script and passed the command line arguments to the script using -PARGS.
task taskAll(type: Exec, dependsOn: taskinit) {
environment['PROJECT_ROOT'] = "${projectDir}"
workingDir rootProject.projectDir.path
description = 'Main task'
executable rootProject.projectDir.path + "/execute.me"
if (project.hasProperty('ARGS')) {
args(ARGS.split(','))
}
}
I call this gradle task with any of the below options
./gradlew taskAll
./gradlew taskAll -PARGS="arg1"
./gradlew taskAll -PARGS="arg2"
However, am looking to see if I split taskAll into multiple tasks, say
./gradlew taskA #Calls task taskAll with arg1
./gradlew taskB #Calls task taskAll with arg2
I understand that I will have to replicate the taskAll to create taskA, taskB and remove the "if" condition and hardcode args in each of these.
However, I wonder if it is possible to have a cleaner implementation by having MainTask which only calls the executable, and then have TaskA, TaskB, TaskC call MainTask and pass the arguments arg1, arg2 and arg3.
In most cases, executing one task from another is done by configuration of task dependencies, via providing dependsOn and optionally mustRunAfter properies. In your case, it's not possible to use it, since your main task has to be executed after some configuration task. In that case, you can use finalizedBy property of the task.
For your requirements, you can create a number of tasks, which will set some script variable with predefined arguments, just as you need it. And you could leave your main task, which will call something, relying on this arguments. Only thing you need to do, is to make each custom task finilizedBy your main task. So, every call of custom task will execute the main task after excution.
Here is the short example, how to do it:
//define a variable to store arguments
def ARGS = null
//2 custom tasks, which set arguments during the execution phase
task taskA << {
ARGS = "poperty1,property2"
}
task taskB << {
ARGS = "property3,property4"
}
//your main task
task mainTask (type: Exec) {
environment['PROJECT_ROOT'] = "${projectDir}"
workingDir rootProject.projectDir.path
description = 'Main task'
executable rootProject.projectDir.path + "/execute.me"
//here is the main difference, we moved arguments setting into
//execution phase, before execution of this task
doFirst{
//if you call custom task it will be executed with predefined params
if (ARGS != null) {
args(ARGS)
//if you call mainTask, you are able to pass arguments via command line with -PCOMMAND_LINE_ARGS=123
} else if (project.hasProperty('COMMAND_LINE_ARGS')) {
args(COMMAND_LINE_ARGS)
} else {
throw new GradleException("No arguments found")
}
}
}
//finilization settings for custom tasks
taskA.finalizedBy mainTask
taskB.finalizedBy mainTask
A prettier way to do that with GradleBuild API:
task ('taskA', type: GradleBuild) {
startParameter.projectProperties = ['ARGS':'arg1']
tasks = ['taskAll']
}
task ('taskB', type: GradleBuild) {
startParameter.projectProperties = ['ARGS':'arg2']
tasks = ['taskAll']
}
You can have complex project properties, for example command line argument -Pmyextension.config=true will become :
startParameter.projectProperties = ['myextension.config':true]
Note that this will erase CLI args. If you need to append it :
startParameter.projectProperties << project.getGradle().getStartParameter().getProjectProperties() << ['myextension.config':true]
You can use ext:
task outraTask << {
printf(arg0)
printf(arg1)
}
project(':projetc2').tasks.outraTask {
ext.arg0 = "0"
ext.arg1 = "1"
}.execute()
output:
> Task :projetc2:outraTask
0
1

How do I concatenate multiple files in Gradle?

Is there an easy way to concatenate multiple text files into a single one in Gradle? The build script should look something like this:
FileCollection jsDeps = files(
'file1.js',
'file2.js'
// other files here
)
task concatenate << {
// concatenate the files to file.js
}
I am using Gradle 2.3.
leftShift / "<<" is deprecated in gradle 3.4 You may use something like:
task concatenate {
doLast {
def toConcatenate = files("filename1", "filename2", ...)
def outputFileName = "output.txt"
def output = new File(outputFileName)
output.write('') // truncate output if needed
toConcatenate.each { f -> output << f.text }
}
You can also register the files as inputs/outputs to help with incremental builds. It's especially helpful with larger files.
something like this:
task 'concatenateFiles', {
inputs.files( fileTree( "path/to/dir/with/files" ) ).skipWhenEmpty()
outputs.file( "$project.buildDir/tmp/concatinated.js" )
doLast {
outputs.files.singleFile.withOutputStream { out ->
for ( file in inputs.files ) file.withInputStream { out << it << '\n' }
}
}
}
Instead of the fileTree, it can also be replaced with sourceset/sourceset output, specific files, outputs from a different task, etc.
Gradle doc on task inputs/output
Concatenating files in groovy
The following task should do the job:
task concatenate << {
def toConcatenate = files('f1', 'f2', 'f3')
def output = new File('output')
toConcatenate.each { f -> output << f.text }
}
(new File('test.js')).text = file('test1.js').getText() + file('test2.js').getText()
UPDATE:
For collections.
(new File('test.js')).text = files('test1.js', 'test2.js').collect{it.getText()}.join("\n")

Gradle CustomTask not considering InputFiles for dependencies

I'm trying to write a custom Gradle task that takes a specific set of input files, which may be in different directories, an output file and call a javaexec with the inputs.
I currently have it where it will skip the #TaskAction if it sees that the #OutputFile exists but it pays no consideration to the InputFiles list and if they are newer than the output file.
Here are the relevant parts of the buildfile:
class JavaCPPTask extends DefaultTask {
def className = []
String outputDir = ""
String localClasspath = "bin/classes"
String runIfExists = ""
String javacpp = "libs/javacpp.jar"
String outputName = ""
//#InputDirectory
//File inputFile = new File("bin/classes/com/package/ve")
#InputFiles
List<File> getInputFiles() {
ArrayList<File> inputFiles = new ArrayList<File>()
for ( int i = 0; i < className.size(); i++ ) {
def inputFileStr = localClasspath + '/' + className.get(i).replace('.','/') + '.class'
println("InputFileStr: " + inputFileStr )
File inputFile = new File(inputFileStr)
println("Input file exists? " + inputFile.exists() )
inputFiles.add(inputFile)
}
return inputFiles
}
#OutputFile
File getTargetFile() {
if ( outputName.length() == 0 ) {
def nameWithPackage = className.get(0)
def name = nameWithPackage.substring(nameWithPackage.lastIndexOf(".")+1)
outputName = 'jni' + name;
}
File outputFile = new File( outputDir + outputName + '.cpp');
println("Output Name is: " + outputFile )
println("Output file exists? " + outputFile.exists() )
return outputFile
}
JavaCPPTask() {
}
#TaskAction
def javacpp() {
def haveFile = new File(runIfExists)
if ( runIfExists.length() > 0 && ! haveFile.exists() ) {
println("Skipping class " + className + " makefile does not exist: " + runIfExists );
}
else {
def useOutputName = []
if ( outputName.length() > 0 ) {
useOutputName = [ '-o', outputName ]
}
def optionArray = [ '-nocompile', '-classpath', localClasspath, '-d', outputDir ]
def allOptions = [javacpp] + useOutputName + optionArray + className.findAll()
project.javaexec { main="-jar"; args allOptions }
}
}
}
def codecOpus = ['com.package.ve.CodecOpus']
task javaCPP_CodecOpus(type:JavaCPPTask) {
className = codecOpus
outputDir = 'jni/VECodecOpus/'
runIfExists = outputDir + 'Android.mk'
}
And the output, which may or may not help. The output file has already been created by a previous run, but the .class file has been updated so I'd expect that the target would run again.
> c:\java\gradle-2.2.1\bin\gradle -b build
JavaCPP.gradle javaCPP_CodecOpus
Incremental java compilation is an incubating feature.
InputFileStr: bin/classes/com/package/ve/CodecOpus.class
Input file exists? true
:javaCPP_CodecOpus
InputFileStr: bin/classes/com/package/ve/CodecOpus.class
Input file exists? true
Output Name is: jni\VECodecOpus\jniCodecOpus.cpp
Output file exists? true
Output Name is: jni\VECodecOpus\jniCodecOpus.cpp
Output file exists? true
Output Name is: jni\VECodecOpus\jniCodecOpus.cpp
Output file exists? true
Output Name is: jni\VECodecOpus\jniCodecOpus.cpp
Output file exists? true
InputFileStr: bin/classes/com/package/ve/CodecOpus.class
Input file exists? true
:javaCPP_CodecOpus UP-TO-DATE
BUILD SUCCESSFUL
What am I doing wrong?
Turns out after debugging through the Gradle source, the script is working. I had been forcing an update and a recompile on the .java file which created a new .class file, but Gradle was smart enough to know that the contents of the class file had not changed. Once I changed the source in the .java file and recompiled, Gradle correctly identified the source as changed and rebuilt the target.
Guess I've been using Make for far too long.
There's still and issue with the script since i was forced to run a clean build before I started down this rabbit hole, but the out of date input file wasn't it.

Gradle shortcut notations does not work for copy task

I'm new to gradle and i tried to copy files from one folder to another using a task of "Copy" type, but it does not work. The following is my script:
def dest = 'newfolder'
task copy(type: Copy) << {
println dest
from "src"
into dest
}
But if i remove the "<<" and run gradle copy again, it works. Script like following:
def dest = 'newfolder'
task copy(type: Copy) {
println dest
from "src"
into dest
}
Why doesn't it work if i use "<<" ?
But when i run another following scripts, they all work.
task hello << {
println 'Hello world!'
}
task hello {
println 'Hello world!'
}
<< (short for doLast) adds a task action, which will be executed after the Copy task's main task action (which does the copying). At that point it's too late to configure the task, as the main action has already completed.

Resources