I have a Gradle build file which uses ProtoBuffer plugin and runs some tasks. At some point some tasks are run for some files, which are inputs to tasks.
I want to modify the set of files which is the input to those tasks. Say, I want the tasks to be run with files which are listed, one per line, in a particular file. How can I do that?
EDIT: Here is a part of rather big build.gradle which provides some context.
configure(protobufProjects) {
apply plugin: 'java'
ext {
protobufVersion = '3.9.1'
}
dependencies {
...
}
protobuf {
generatedFilesBaseDir = "$projectDir/gen"
protoc {
if (project.hasProperty('protocPath')) {
path = "$protocPath"
}
else {
artifact = "com.google.protobuf:protoc:$protobufVersion"
}
}
plugins {
...
}
generateProtoTasks {
all().each { task ->
...
}
}
sourceSets {
main {
java {
srcDirs 'gen/main/java'
}
}
}
}
clean {
delete protobuf.generatedFilesBaseDir
}
compileJava {
File generatedSourceDir = project.file("gen")
project.mkdir(generatedSourceDir)
options.annotationProcessorGeneratedSourcesDirectory = generatedSourceDir
}
}
The question is, how to modify the input file set for existing task (which already does something with them), not how to create a new task.
EDIT 2: According to How do I modify a list of files in a Gradle copy task? , it's a bad idea in general, as Gradle makes assumptions about inputs and outputs dependencies, which can be broken by this approach.
If you would have added the gradle file and more specific that would have been very helpful. I will try to give an example from what I have understood:
fun listFiles(fileName: String): List<String> {
val file = file(fileName).absoluteFile
val listOfFiles = mutableListOf<String>()
file.readLines().forEach {
listOfFiles.add(it)
}
return listOfFiles
}
tasks.register("readFiles") {
val inputFile: String by project
val listOfFiles = listFiles(inputFile)
listOfFiles.forEach {
val file = file(it).absoluteFile
file.readLines().forEach { println(it) }
}
}
Then run the gradle like this: gradle -PinputFile=<path_to_the_file_that_contains_list_of_files> readFiles
Related
Consider this toy build.gradle file:
plugins {
id "java"
}
import org.gradle.internal.os.OperatingSystem;
apply plugin: 'java'
repositories {
mavenLocal()
mavenCentral()
}
def pathExists(pathname) {
// Returns true iff pathame is an existing file or directory
try {
// This may throw an error for a Windows pathname, c:/path/to/thing
if (file(pathname).exists()) {
return true;
}
} catch (GradleException e) {
// I don't care
}
if (OperatingSystem.current().isWindows()) {
try {
// If we're on Windows, try to make c:/path/to/thing work
if (file("file:///${pathname}").exists()) {
return true;
}
} catch (GradleException e) {
// I don't care
}
}
return false
}
def someVariable = "absent"
if (pathExists("/tmp")) {
someVariable = "present"
}
task someCommonTask() {
doLast {
println("Did some setup stuff: ${someVariable}")
}
}
task someATask(dependsOn: ["someCommonTask"]) {
doLast {
println("A: ${someVariable}")
}
}
task someBTask(dependsOn: ["someCommonTask"]) {
def otherVariable = someVariable == "absent" || pathExists("/etc")
doLast {
println("B: ${otherVariable}")
}
}
Is it possible to reorganize this build file so that someATask is in a.gradle and someBTask is in b.gradle? I've made some attempts to use apply from: without success and, if the answer is on this page: https://docs.gradle.org/current/userguide/organizing_gradle_projects.html it eludes me.
In the real project, there would be a couple of dozen tasks in each of those subordinate files and the goal of dividing up the build.gradle file isn't to have them be subprojcts, per se, they're just logical groupings of tasks.
This split can be done using apply from: ... to separate the files.
E.g.
build.gradle:
plugins {
id "java"
}
apply plugin: 'java'
repositories {
mavenLocal()
mavenCentral()
}
ext {
someVariable = "absent" // XXX
}
apply from: "setup.gradle" // XXX
apply from: "common.gradle" // XXX
task someATask(dependsOn: ["someCommonTask"]) {
doLast {
println("A: ${someVariable}")
}
}
setup.gradle:
task setup() {
someVariable = "present"
}
common.gradle:
task someCommonTask(dependsOn: ["setup"]) {
doLast {
println("Did some setup stuff: ${someVariable}")
}
}
# gradle someATask
> Task :someCommonTask
Did some setup stuff: present
> Task :someATask
A: present
BUILD SUCCESSFUL in 403ms
My guess, the problems you are facing are with toplevel code and not
using properties.
I am trying to exclude some files from the merged jacoco report. I am using:
(root gradle)
tasks.register<JacocoReport>("codeCoverageReport") {
subprojects {
val subProject = this
subProject.plugins.withType<JacocoPlugin>().configureEach {
subProject.tasks.matching { it.extensions.findByType<JacocoTaskExtension>() != null }.configureEach {
val testTask = this
sourceSets(subProject.sourceSets.main.get())
executionData(testTask)
}
subProject.tasks.matching { it.extensions.findByType<JacocoTaskExtension>() != null }.forEach {
rootProject.tasks["codeCoverageReport"].dependsOn(it)
}
}
}
reports {
xml.isEnabled = false
html.isEnabled = true
csv.isEnabled = false
}
}
And for the every module exclusion jacoco report (e.g. for common module):
tasks.withType<JacocoReport> {
classDirectories.setFrom(
sourceSets.main.get().output.asFileTree.matching {
exclude(JacocoExcludes.commonModule)
}
)
}
For each module this is working but when trying to interact with root gradle task either the gradle sync fails or it only add the files from the last module. Any help ?
Thanks
I had the same problem and used the following code :
tasks.jacocoTestReport {
// tests are required to run before generating the report
dependsOn(tasks.test)
// print the report url for easier access
doLast {
println("file://${project.rootDir}/build/reports/jacoco/test/html/index.html")
}
classDirectories.setFrom(
files(classDirectories.files.map {
fileTree(it) {
exclude("**/generated/**", "**/other-excluded/**")
}
})
)
}
as suggested here : https://github.com/gradle/kotlin-dsl-samples/issues/1176#issuecomment-610643709
I'm trying to create a custom gradle task that will run the different detekt profiles I have setup.
Here is my Detekt config:
detekt {
version = "1.0.0.RC6-4"
profile("main") {
input = "$projectDir/app/src/main/java"
output = "$projectDir/app/build/reports/detekt"
config = "$projectDir/config/detekt-config.yml"
}
profile("app") {
input = "$projectDir/app/src/main/java"
output = "$projectDir/app/build/reports/detekt"
}
profile("database") {
input = "$projectDir/database/src/main/java"
output = "$projectDir/database/build/reports/detekt"
}
profile("logging") {
input = "$projectDir/logging/src/main/java"
output = "$projectDir/logging/build/reports/detekt"
}
profile("network") {
input = "$projectDir/network/src/main/java"
output = "$projectDir/network/build/reports/detekt"
}
}
And here is what I'm trying for the custom gradle task:
task detektAll {
group = 'verification'
dependsOn 'detektCheck'
doLast {
println "\n##################################################" +
"\n# Detekt'ed all the things! Go you! #" +
"\n##################################################"
}
}
I need to add -Ddetekt.profile=app and the others for each profile.
How can I accomplish this?
Inspired by this neat TestNG task, and this SO question I thought I'd whip up something quick for re-running of only failed JUnit tests from Gradle.
But after searching around for awhile, I couldn't find anything analogous which was quite as convenient.
I came up with the following, which seems to work pretty well and adds a <testTaskName>Rerun task for each task of type Test in my project.
import static groovy.io.FileType.FILES
import java.nio.file.Files
import java.nio.file.Paths
// And add a task for each test task to rerun just the failing tests
subprojects {
afterEvaluate { subproject ->
// Need to store tasks in static temp collection, else new tasks will be picked up by live collection leading to StackOverflow
def testTasks = subproject.tasks.withType(Test)
testTasks.each { testTask ->
task "${testTask.name}Rerun"(type: Test) {
group = 'Verification'
description = "Re-run ONLY the failing tests from the previous run of ${testTask.name}."
// Depend on anything the existing test task depended on
dependsOn testTask.dependsOn
// Copy runtime setup from existing test task
testClassesDirs = testTask.testClassesDirs
classpath = testTask.classpath
// Check the output directory for failing tests
File textXMLDir = subproject.file(testTask.reports.junitXml.destination)
logger.info("Scanning: $textXMLDir for failed tests.")
// Find all failed classes
Set<String> allFailedClasses = [] as Set
if (textXMLDir.exists()) {
textXMLDir.eachFileRecurse(FILES) { f ->
// See: http://marxsoftware.blogspot.com/2015/02/determining-file-types-in-java.html
String fileType
try {
fileType = Files.probeContentType(f.toPath())
} catch (IOException e) {
logger.debug("Exception when probing content type of: $f.")
logger.debug(e)
// Couldn't determine this to be an XML file. That's fine, skip this one.
return
}
logger.debug("Filetype of: $f is $fileType.")
if (['text/xml', 'application/xml'].contains(fileType)) {
logger.debug("Found testsuite file: $f.")
def testSuite = new XmlSlurper().parse(f)
def failedTestCases = testSuite.testcase.findAll { testCase ->
testCase.children().find { it.name() == 'failure' }
}
if (!failedTestCases.isEmpty()) {
logger.info("Found failures in file: $f.")
failedTestCases.each { failedTestCase ->
def className = failedTestCase['#classname']
logger.info("Failure: $className")
allFailedClasses << className.toString()
}
}
}
}
}
if (!allFailedClasses.isEmpty()) {
// Re-run all tests in any class with any failures
allFailedClasses.each { c ->
def testPath = c.replaceAll('\\.', '/') + '.class'
include testPath
}
doFirst {
logger.warn('Re-running the following tests:')
allFailedClasses.each { c ->
logger.warn(c)
}
}
}
outputs.upToDateWhen { false } // Always attempt to re-run failing tests
// Only re-run if there were any failing tests, else just print warning
onlyIf {
def shouldRun = !allFailedClasses.isEmpty()
if (!shouldRun) {
logger.warn("No failed tests found for previous run of task: ${subproject.path}:${testTask.name}.")
}
return shouldRun
}
}
}
}
}
Is there any easier way to do this from Gradle? Is there any way to get JUnit to output a consolidated list of failures somehow so I don't have to slurp the XML reports?
I'm using JUnit 4.12 and Gradle 4.5.
Here is one way to do it. The full file will be listed at the end, and is available here.
Part one is to write a small file (called failures) for every failed test:
test {
// `failures` is defined elsewhere, see below
afterTest { desc, result ->
if ("FAILURE" == result.resultType as String) {
failures.withWriterAppend {
it.write("${desc.className},${desc.name}\n")
}
}
}
}
In part two, we use a test filter (doc here) to restrict the tests to any that are present in the failures file:
def failures = new File("${projectDir}/failures.log")
def failedTests = []
if (failures.exists()) {
failures.eachLine { line ->
def tokens = line.split(",")
failedTests << tokens[0]
}
}
failures.delete()
test {
filter {
failedTests.each {
includeTestsMatching "${it}"
}
}
// ...
}
The full file is:
apply plugin: 'java'
repositories {
jcenter()
}
dependencies {
testCompile('junit:junit:4.12')
}
def failures = new File("${projectDir}/failures.log")
def failedTests = []
if (failures.exists()) {
failures.eachLine { line ->
def tokens = line.split(",")
failedTests << tokens[0]
}
}
failures.delete()
test {
filter {
failedTests.each {
includeTestsMatching "${it}"
}
}
afterTest { desc, result ->
if ("FAILURE" == result.resultType as String) {
failures.withWriterAppend {
it.write("${desc.className},${desc.name}\n")
}
}
}
}
The Test Retry Gradle plugin is designed to do exactly this. It will rerun each failed test a certain number of times, with the option of failing the build if too many failures have occurred overall.
plugins {
id 'org.gradle.test-retry' version '1.2.1'
}
test {
retry {
maxRetries = 3
maxFailures = 20 // Optional attribute
}
}
I have the following gradle.properties
version=2.1
paths=['/home/desk/hie', '/home/mydesk1/hai1', '/home/mydesk2/hai2']
sources=['/src/path/impl', '/src/path/src']
I need to access these paths and sources form gradle.properties to build.gradle
sourceSets {
main {
java {
srcDir=${paths}
}
}
}
but ${paths} is not working.
Can you someone help to get out of this issue and how to use those list in build.gradle file
Consider the following, which converts the paths String to an Iterable (that is, a List):
apply plugin: 'java'
def pathsList = Eval.me(project.ext.paths)
sourceSets {
main {
java {
srcDirs = pathsList
}
}
}
You can share the value using gradle object:
// settings.gradle
gradle.ext {
paths = ['/home/desk/hie', '/home/mydesk1/hai1', '/home/mydesk2/hai2']
}
// build.gradle
sourceSets {
main {
java {
srcDir = paths
}
}
}