I'm trying to create an task that deletes all the old versions of my dependency, I got it working but the problem is to my dependency version has '.+' in version for build number, so when I get it all the folders get deleted instead of only the older ones. I currently have this:
task cleanTerraCore(type: Delete) {
doLast {
def dirName = new File("${gradle.gradleUserHomeDir}/caches/minecraft/deobfedDeps/deobf/terrails/terracore/TerraCore")
dirName.eachDir { dir ->
project.configurations.deobfCompile.dependencies.each {
System.out.println(it.version)
if (dir.name.contains("SNAPSHOT") && it.name.contains("TerraCore")) {
if (!dir.name.contains(it.version)) {
delete(dir)
}
}
}
}
}
}
dependencies {
deobfCompile("terrails.terracore:TerraCore:" + getMajorMC() + "-" + "${terracore_version}-SNAPSHOT.+")
}
'it.version' always prints out 'SNAPSHOT.+' so I'm not sure how to handle this, could I somehow efficiently check for the biggest build number?
Figured it out after couple of hours. It now gets the latest version without the '+' symbol
task cleanTerra(type: Delete) {
doLast {
def dirName = new File("${gradle.gradleUserHomeDir}/caches/minecraft/deobfedDeps/deobf/terrails/terracore/TerraCore")
dirName.eachDir { dir ->
configurations.deobfCompile.resolvedConfiguration.firstLevelModuleDependencies.each {
System.out.println(it.moduleVersion + ", ${it.moduleName}")
if (dir.name.contains("SNAPSHOT") && it.moduleName.contains("TerraCore")) {
if (!dir.name.contains(it.moduleVersion)) {
delete(dir)
}
}
}
}
}
}
So the println now prints out the full version (currently "1.12-2.1.10-SNAPSHOT.7") and the name of the dependency "TerraCore"
Related
I have a Gradle build file which uses ProtoBuffer plugin and runs some tasks. At some point some tasks are run for some files, which are inputs to tasks.
I want to modify the set of files which is the input to those tasks. Say, I want the tasks to be run with files which are listed, one per line, in a particular file. How can I do that?
EDIT: Here is a part of rather big build.gradle which provides some context.
configure(protobufProjects) {
apply plugin: 'java'
ext {
protobufVersion = '3.9.1'
}
dependencies {
...
}
protobuf {
generatedFilesBaseDir = "$projectDir/gen"
protoc {
if (project.hasProperty('protocPath')) {
path = "$protocPath"
}
else {
artifact = "com.google.protobuf:protoc:$protobufVersion"
}
}
plugins {
...
}
generateProtoTasks {
all().each { task ->
...
}
}
sourceSets {
main {
java {
srcDirs 'gen/main/java'
}
}
}
}
clean {
delete protobuf.generatedFilesBaseDir
}
compileJava {
File generatedSourceDir = project.file("gen")
project.mkdir(generatedSourceDir)
options.annotationProcessorGeneratedSourcesDirectory = generatedSourceDir
}
}
The question is, how to modify the input file set for existing task (which already does something with them), not how to create a new task.
EDIT 2: According to How do I modify a list of files in a Gradle copy task? , it's a bad idea in general, as Gradle makes assumptions about inputs and outputs dependencies, which can be broken by this approach.
If you would have added the gradle file and more specific that would have been very helpful. I will try to give an example from what I have understood:
fun listFiles(fileName: String): List<String> {
val file = file(fileName).absoluteFile
val listOfFiles = mutableListOf<String>()
file.readLines().forEach {
listOfFiles.add(it)
}
return listOfFiles
}
tasks.register("readFiles") {
val inputFile: String by project
val listOfFiles = listFiles(inputFile)
listOfFiles.forEach {
val file = file(it).absoluteFile
file.readLines().forEach { println(it) }
}
}
Then run the gradle like this: gradle -PinputFile=<path_to_the_file_that_contains_list_of_files> readFiles
Hi I got this zip task which works great:
def dir = new File("${projectDir.parentFile}/test/")
task testZip(type: Zip) {
from dir
destinationDirectory = dir
include 'toast/**'
archiveFileName = 'test.zip'
}
but then when I make the include property lazy (because I need to in my real case)
def dir = new File("${projectDir.parentFile}/test/")
task testZip(type: Zip) {
from dir
destinationDirectory = dir
include {
'toast/**'
}
archiveFileName = 'test.zip'
}
then it creates a zip that includes everything in the folder, (so the generated archive too). In this test case the inner zip is just corrupted (doesn't run infinitely) but in the real world case it does make an infinite zip. (Not sure why, maybe my best case has too few or small files). Either way the test case shows the problem, the generated zip contains a zip even though it should only contain the toast directory and all of its content.
How do I fix this? I need a lazy include because the directory I want to include is computed by other tasks. I get the exact same problem with Tar except it refuses to create the archive since it includes itself.
Using exclude '*.zip' is a dumb workaround which makes the archive include other folders I don't want. I only want to include a specific folder, lazyly.
Here's what the monster looks like in the real world case. I basically need to retrieve the version of the project from Java to then use that version to name the folders I'm packaging. (Making a libGDX game and packaging it with a jre using packr). The problematic tasks are 'makeArchive_' + platform.
String jumpaiVersion;
task fetchVersion(type: JavaExec) {
outputs.upToDateWhen { jumpaiVersion != null }
main = 'net.jumpai.Version'
classpath = sourceSets.main.runtimeClasspath
standardOutput new ByteArrayOutputStream()
doLast {
jumpaiVersion = standardOutput.toString().replaceAll("\\s+", "")
}
}
def names = [
'win64' : "Jumpai-%%VERSION%%-Windows-64Bit",
'win32' : "Jumpai-%%VERSION%%-Windows-32Bit",
'linux64' : "Jumpai-%%VERSION%%-Linux-64Bit",
'linux32' : "Jumpai-%%VERSION%%-Linux-32Bit",
'mac' : "Jumpai-%%VERSION%%-Mac.app"
]
def platforms = names.keySet() as String[]
def jdks = [
'win64' : 'https://cdn.azul.com/zulu/bin/zulu9.0.7.1-jdk9.0.7-win_x64.zip',
'win32' : 'https://cdn.azul.com/zulu/bin/zulu9.0.7.1-jdk9.0.7-win_i686.zip',
'linux64' : 'https://cdn.azul.com/zulu/bin/zulu9.0.7.1-jdk9.0.7-linux_x64.tar.gz',
'linux32' : 'https://cdn.azul.com/zulu/bin/zulu9.0.7.1-jdk9.0.7-linux_i686.tar.gz',
'mac' : 'https://cdn.azul.com/zulu/bin/zulu9.0.7.1-jdk9.0.7-macosx_x64.zip'
]
def formats = [
'win64' : 'ZIP',
'win32' : 'ZIP',
'linux64' : 'TAR_GZ',
'linux32' : 'TAR_GZ',
'mac' : 'ZIP'
]
File jdksDir = new File(project.buildscript.sourceFile.parentFile.parentFile, 'out/jdks')
File gameJar = new File("${projectDir.parentFile}/desktop/build/libs/Jumpai.jar")
File gameData = new File("${projectDir.parentFile}/desktop/build/libs/Jumpai.data")
File packrDir = new File("${projectDir.parentFile}/out/packr/")
File minimalTmpDir = new File("${projectDir.parentFile}/desktop/build/libs/minimal-tmp")
task minimizeGameJar {
dependsOn ':desktop:dist'
doFirst {
minimalTmpDir.mkdirs()
copy {
from zipTree(gameJar)
into minimalTmpDir
}
for(file in minimalTmpDir.listFiles())
if(file.getName().contains("humble"))
file.delete()
}
}
task makeMinimal(type: Zip) {
dependsOn minimizeGameJar
dependsOn fetchVersion
from minimalTmpDir
include '**'
archiveFileName = provider {
"Jumpai-${->jumpaiVersion}-Minimal.jar"
}
destinationDir packrDir
doLast {
minimalTmpDir.deleteDir()
}
}
task copyGameJar(type: Copy) {
outputs.upToDateWhen { gameData.exists() }
dependsOn ':desktop:dist'
from gameJar.getAbsolutePath()
into gameData.getParentFile()
rename("Jumpai.jar", "Jumpai.data")
}
task setWindowsIcons(type: Exec) {
dependsOn fetchVersion
workingDir '.'
commandLine 'cmd', '/c', 'set_windows_icons.bat', "${->jumpaiVersion}"
}
for(platform in platforms) {
task("getJdk_" + platform) {
String url = jdks[platform]
File jdkDir = new File(jdksDir, platform + "-jdk")
File jdkFile = new File(jdkDir, url.split("/").last())
outputs.upToDateWhen { jdkFile.exists() }
doFirst {
if(!jdkDir.exists())
jdkDir.mkdirs()
if(jdkFile.exists())
{
println jdkFile.getName() + " is already present"
return
}
else
{
println "Downloading " + jdkFile.getName()
new URL(url).withInputStream {
i -> jdkFile.withOutputStream { it << i }
}
}
for(file in jdkDir.listFiles()) {
if(file.equals(jdkFile))
continue
if(file.isFile()) {
if (!file.delete())
println "ERROR: could not delete " + file.getAbsoluteFile()
} else if(!file.deleteDir())
println "ERROR: could not delete content of " + file.getAbsoluteFile()
}
if(url.endsWith(".tar.gz"))// don't mix up archive type of what we downloaded vs archive type of what we compress (in formats)
{
copy {
from tarTree(resources.gzip(jdkFile))
into jdkDir
}
}
else if(url.endsWith(".zip"))
{
copy {
from zipTree(jdkFile)
into jdkDir
}
}
}
}
File packrInDir = new File(packrDir, platform)
String platformRawName = names[platform]
task("packr_" + platform, type: JavaExec) {
outputs.upToDateWhen { new File(packrDir, platformRawName.replace("%%VERSION%%", jumpaiVersion)).exists() }
dependsOn fetchVersion
dependsOn copyGameJar
dependsOn 'getJdk_' + platform
main = 'com.badlogicgames.packr.Packr'
classpath = sourceSets.main.runtimeClasspath
args 'tools/res/packr_config/' + platform + '.json'
workingDir = project.buildscript.sourceFile.parentFile.parentFile
doLast {
File packrOutDir = new File(packrDir, platformRawName.replace("%%VERSION%%", jumpaiVersion));
packrOutDir.deleteDir()
if(packrOutDir.exists())
{
println "ERROR Could not delete packr output " + packrOutDir.getAbsolutePath()
return
}
if(!packrInDir.renameTo(packrOutDir))
println "ERROR Could not rename packr output dir for " + packrInDir.getName()
}
}
if(formats[platform] == 'ZIP')
{
task('makeArchive_' + platform, type: Zip) {
if(platform.contains("win"))
dependsOn setWindowsIcons
dependsOn fetchVersion
dependsOn 'packr_' + platform
from packrDir
destinationDirectory = packrDir
include {
platformRawName.replace("%%VERSION%%", jumpaiVersion) + "/"
}
archiveFileName = provider {
platformRawName.replace("%%VERSION%%", jumpaiVersion) + ".zip"
}
}
}
else if(formats[platform] == 'TAR_GZ')
{
task('makeArchive_' + platform, type: Tar) {
dependsOn 'packr_' + platform
from packrDir
destinationDirectory = packrDir
include {
platformRawName.replace("%%VERSION%%", jumpaiVersion) + '/**'
}
archiveFileName = provider {
platformRawName.replace("%%VERSION%%", jumpaiVersion) + ".tar.gz"
}
extension 'tar'
compression = Compression.GZIP
}
}
else
println 'Unsupported format for ' + platform
}
task deploy {
dependsOn makeMinimal
for(platform in platforms)
dependsOn 'makeArchive_' + platform
}
How do I fix this? I need a lazy include because the directory I want to include is computed by other tasks. I get the exact same problem with Tar except it refuses to create the archive since it includes itself.
You can get what you want by using the doFirst method and modifiying the tasks properties with the passed action.
task('makeArchive_' + platform, type: Zip) {
if(platform.contains("win"))
dependsOn setWindowsIcons
dependsOn fetchVersion
dependsOn 'packr_' + platform
from packrDir
destinationDirectory = packrDir
archiveFileName = provider {
platformRawName.replace("%%VERSION%%", jumpaiVersion) + ".zip"
}
doFirst {
def includeDir = platformRawName.replace("%%VERSION%%", jumpaiVersion)
// Include only files and directories from 'includeDir'
include {
it.relativePath.segments[ 0 ].equalsIgnoreCase(includeDir)
}
}
}
Please have also a look at this answer to a similar question. My solution is just a workaround. If you know your version at configuration phase you can achieve what you want more easily. Writing your own custom tasks or plugins can also help to clean up your build script.
Inspired by this neat TestNG task, and this SO question I thought I'd whip up something quick for re-running of only failed JUnit tests from Gradle.
But after searching around for awhile, I couldn't find anything analogous which was quite as convenient.
I came up with the following, which seems to work pretty well and adds a <testTaskName>Rerun task for each task of type Test in my project.
import static groovy.io.FileType.FILES
import java.nio.file.Files
import java.nio.file.Paths
// And add a task for each test task to rerun just the failing tests
subprojects {
afterEvaluate { subproject ->
// Need to store tasks in static temp collection, else new tasks will be picked up by live collection leading to StackOverflow
def testTasks = subproject.tasks.withType(Test)
testTasks.each { testTask ->
task "${testTask.name}Rerun"(type: Test) {
group = 'Verification'
description = "Re-run ONLY the failing tests from the previous run of ${testTask.name}."
// Depend on anything the existing test task depended on
dependsOn testTask.dependsOn
// Copy runtime setup from existing test task
testClassesDirs = testTask.testClassesDirs
classpath = testTask.classpath
// Check the output directory for failing tests
File textXMLDir = subproject.file(testTask.reports.junitXml.destination)
logger.info("Scanning: $textXMLDir for failed tests.")
// Find all failed classes
Set<String> allFailedClasses = [] as Set
if (textXMLDir.exists()) {
textXMLDir.eachFileRecurse(FILES) { f ->
// See: http://marxsoftware.blogspot.com/2015/02/determining-file-types-in-java.html
String fileType
try {
fileType = Files.probeContentType(f.toPath())
} catch (IOException e) {
logger.debug("Exception when probing content type of: $f.")
logger.debug(e)
// Couldn't determine this to be an XML file. That's fine, skip this one.
return
}
logger.debug("Filetype of: $f is $fileType.")
if (['text/xml', 'application/xml'].contains(fileType)) {
logger.debug("Found testsuite file: $f.")
def testSuite = new XmlSlurper().parse(f)
def failedTestCases = testSuite.testcase.findAll { testCase ->
testCase.children().find { it.name() == 'failure' }
}
if (!failedTestCases.isEmpty()) {
logger.info("Found failures in file: $f.")
failedTestCases.each { failedTestCase ->
def className = failedTestCase['#classname']
logger.info("Failure: $className")
allFailedClasses << className.toString()
}
}
}
}
}
if (!allFailedClasses.isEmpty()) {
// Re-run all tests in any class with any failures
allFailedClasses.each { c ->
def testPath = c.replaceAll('\\.', '/') + '.class'
include testPath
}
doFirst {
logger.warn('Re-running the following tests:')
allFailedClasses.each { c ->
logger.warn(c)
}
}
}
outputs.upToDateWhen { false } // Always attempt to re-run failing tests
// Only re-run if there were any failing tests, else just print warning
onlyIf {
def shouldRun = !allFailedClasses.isEmpty()
if (!shouldRun) {
logger.warn("No failed tests found for previous run of task: ${subproject.path}:${testTask.name}.")
}
return shouldRun
}
}
}
}
}
Is there any easier way to do this from Gradle? Is there any way to get JUnit to output a consolidated list of failures somehow so I don't have to slurp the XML reports?
I'm using JUnit 4.12 and Gradle 4.5.
Here is one way to do it. The full file will be listed at the end, and is available here.
Part one is to write a small file (called failures) for every failed test:
test {
// `failures` is defined elsewhere, see below
afterTest { desc, result ->
if ("FAILURE" == result.resultType as String) {
failures.withWriterAppend {
it.write("${desc.className},${desc.name}\n")
}
}
}
}
In part two, we use a test filter (doc here) to restrict the tests to any that are present in the failures file:
def failures = new File("${projectDir}/failures.log")
def failedTests = []
if (failures.exists()) {
failures.eachLine { line ->
def tokens = line.split(",")
failedTests << tokens[0]
}
}
failures.delete()
test {
filter {
failedTests.each {
includeTestsMatching "${it}"
}
}
// ...
}
The full file is:
apply plugin: 'java'
repositories {
jcenter()
}
dependencies {
testCompile('junit:junit:4.12')
}
def failures = new File("${projectDir}/failures.log")
def failedTests = []
if (failures.exists()) {
failures.eachLine { line ->
def tokens = line.split(",")
failedTests << tokens[0]
}
}
failures.delete()
test {
filter {
failedTests.each {
includeTestsMatching "${it}"
}
}
afterTest { desc, result ->
if ("FAILURE" == result.resultType as String) {
failures.withWriterAppend {
it.write("${desc.className},${desc.name}\n")
}
}
}
}
The Test Retry Gradle plugin is designed to do exactly this. It will rerun each failed test a certain number of times, with the option of failing the build if too many failures have occurred overall.
plugins {
id 'org.gradle.test-retry' version '1.2.1'
}
test {
retry {
maxRetries = 3
maxFailures = 20 // Optional attribute
}
}
I want to download the dependency artifacts manually in the future after Gradle has all the dependency artifacts available, hence I would like to get the URLs which Gradle used to download those artifacts.
Is there a way to get the URL of dependencies which artifacts have been downloaded by Gradle?
use gson for a example:
dependencies {
// https://mvnrepository.com/artifact/com.google.code.gson/gson
compile 'com.google.code.gson:gson:2.8.6'
}
create a task to print url:
task getURLofDependencyArtifact() {
doFirst {
project.configurations.compile.dependencies.each { dependency ->
for (ArtifactRepository repository : project.repositories.asList()) {
def url = repository.properties.get('url')
//https://repo.maven.apache.org/maven2/com/google/code/gson/gson/2.8.6/gson-2.8.6.jar
def jarUrl = String.format("%s%s/%s/%s/%s-%s.jar", url.toString(),
dependency.group.replace('.', '/'), dependency.name, dependency.version,
dependency.name, dependency.version)
try {
def jarfile = new URL(jarUrl)
def inStream = jarfile.openStream();
if (inStream != null) {
println(String.format("%s:%s:%s", dependency.group, dependency.name, dependency.version)
+ " -> " + jarUrl)
return
}
} catch (Exception ignored) {
}
}
}
}
}
run ./gradlew getURLofDependencyArtifact
Task :getURLofDependencyArtifact
com.google.code.gson:gson:2.8.6 -> https://jcenter.bintray.com/com/google/code/gson/gson/2.8.6/gson-2.8.6.jar
PS:the result dependency your project's
repositories {
jcenter()
mavenCentral()
}
so, the result maybe:
Task :getURLofDependencyArtifact
com.google.code.gson:gson:2.8.6 -> https://repo.maven.apache.org/maven2/com/google/code/gson/gson/2.8.6/gson-2.8.6.jar
using Gradle version 6.0 or above, another way of outputting the URLs is to mix --refresh-dependencies with --info
// bash/terminal
./gradlew --info --refresh-dependencies
// cmd
gradlew --info --refresh-dependencies
or output to file
// bash/terminal
./gradlew --info --refresh-dependencies > urls.txt
// cmd
gradlew --info --refresh-dependencies > urls.txt
note on --refresh-dependencies
It’s a common misconception to think that using --refresh-dependencies
will force download of dependencies. This is not the case: Gradle will
only perform what is strictly required to refresh the dynamic
dependencies. This may involve downloading new listing or metadata
files, or even artifacts, but if nothing changed, the impact is
minimal.
source: https://docs.gradle.org/current/userguide/dependency_management.html
see also: How can I force gradle to redownload dependencies?
Wanted something similar but on Android and Kotlin DSL so based on #andforce's answer developed this which hopefully will be useful for others also,
import org.jetbrains.kotlin.utils.addToStdlib.firstNotNullResult
import java.net.URL
val dependenciesURLs: Sequence<Pair<String, URL?>>
get() = project.configurations.getByName(
"implementation"
).dependencies.asSequence().mapNotNull {
it.run { "$group:$name:$version" } to project.repositories.mapNotNull { repo ->
(repo as? UrlArtifactRepository)?.url
}.flatMap { repoUrl ->
"%s/%s/%s/%s/%s-%s".format(
repoUrl.toString().trimEnd('/'),
it.group?.replace('.', '/') ?: "", it.name, it.version,
it.name, it.version
).let { x -> listOf("$x.jar", "$x.aar") }
}.firstNotNullResult { url ->
runCatching {
val connection = URL(url).openConnection()
connection.getInputStream() ?: throw Exception()
connection.url
}.getOrNull()
}
}
tasks.register("printDependenciesURLs") {
doLast {
dependenciesURLs.forEach { (dependency: String, url: URL?) -> println("$dependency => $url") }
}
}
Update: It might not able to find indirect dependencies however.
We need to take care about aar also.
project.configurations.getByName(
"implementation"
).dependencies.each { dependency ->
for (ArtifactRepository repository : rootProject.repositories.asList()) {
def url = repository.properties.get('url')
def urlString = url.toString()
if (url.toString().endsWith("/")) {
urlString = url.toString()
} else {
urlString = url.toString() + "/"
}
def jarUrl = String.format("%s%s/%s/%s/%s-%s.jar", urlString,
dependency.group.replace('.', '/'), dependency.name, dependency.version,
dependency.name, dependency.version)
def aarUrl = String.format("%s%s/%s/%s/%s-%s.aar", urlString,
dependency.group.replace('.', '/'), dependency.name, dependency.version,
dependency.name, dependency.version)
try {
def jarfile = new URL(jarUrl)
def inStreamJar = jarfile.openStream();
if (inStreamJar != null) {
println(String.format("%s:%s:%s", dependency.group, dependency.name, dependency.version)
+ " -> " + jarUrl)
return
}
} catch (Exception ignored) {
}
try {
def aarfile = new URL(aarUrl).setURLStreamHandlerFactory()
def inStreamAar = aarfile.openStream();
if (inStreamAar != null) {
println(String.format("%s:%s:%s", dependency.group, dependency.name, dependency.version)
+ " -> " + aarUrl)
return
}
} catch (Exception ignored) {
}
}
}
Let's say I have the following in my gradle build script:
configurations.all {
resolutionStrategy {
failOnVersionConflict()
force 'com.google.guava:guava:18.0'
}
}
This will fail if more than one version of a jar is found, except for guava where it will force to version 18.0.
Now let's imagine that I want to have failOnVersionConflict() for all external jars, and be forced to use the force clause (so I know what I'm doing), but I want to use the default resolutionStrategy (newest version) for some specific group, like com.mycompany.
Is such a thing possible?
I was looking at this documentation page:
https://gradle.org/docs/current/dsl/org.gradle.api.artifacts.ResolutionStrategy.html
I found my own answer... But it involves a bit of "hacking"... But, after all, that's just what gradle offers...
def dependencyErrors = 0
configurations.all {
resolutionStrategy {
def thirdPartyPackages = [:]
def forced = [
'com.google.guava:guava' : '18.0'
//Here all forced dependencies...
]
eachDependency { DependencyResolveDetails details ->
if (!details.requested.group.startsWith('com.mycompany')) {
def key = details.requested.group + ":" + details.requested.name
if(!thirdPartyPackages.containsKey(key)) {
if(forced.containsKey(key)) {
details.useVersion forced.get(key)
}
else {
thirdPartyPackages.put(key, details.requested.version);
}
}
else {
def existing = thirdPartyPackages.get(key);
if(existing != details.requested.version) {
logger.error "Conflicting versions for [$key]"
logger.error " [$existing]"
logger.error " [$details.requested.version]"
dependencyErrors++
}
}
}
}
}
}
myTask.doFirst {
//here it might also be doLast, or whatever you need. I just put it before a war task, but it might depend on each need.
if(dependencyErrors > 0) {
ant.fail 'There are ' + dependencyErrors + ' conflicting jar versions in the build.'
}
}