Weird behaviour of a simple gradle copy task - gradle

I am completely baffled by the gradle behaviour of a Copy task into multiple directories.
I intend to copy all files from src/common into
target/dir1/ps_modules
target/dir2/ps_modules
target/dir3/ps_modules
Below is how my build.gradle looks:
project.ext.dirs = ["dir1", "dir2", "dir3"]
// ensures that ps_modules directory is created before copying
def ensurePsModulesDir() {
dirs.each {
def psModules = file("target/$it/ps_modules")
if (!(psModules.exists())) {
println("Creating ps_modules directory $psModules as it doesn't exist yet")
mkdir psModules
}
}
}
task copyCommons(type: Copy) {
doFirst {
ensurePsModulesDir()
}
from("src/common")
dirs.each {
into "target/$it/ps_modules"
}
}
The result of running the command ./gradlew copyCommons is completely weird.
The folder creation works as expected, however, the contents/files are copied only in the target/dir3/ps_modules directory. The rest two directories remain empty.
Any help would be appreciated.
Below is the screen grab of target directory tree once the job is run:

I think you want to do something like:
task copyCommons(type: Copy) {
dirs.each {
with copySpec {
from "src/common"
into "target/$it/ps_modules"
}
}
}
I think you can get rid of the ensurePsModulesDir() with this change
* edit *
it seems that the copy task is forcing us to set a destination dir. You might think that setting destinationDir = '.' is ok but it's used in up-to-date checking so likely the task will NEVER be considered up-to-date so will always run. I suggest you use project.copy(...) instead of a Copy task. Eg
task copyCommons {
// setup inputs and outputs manually
inputs.dir "src/common"
dirs.each {
outputs.dir "target/$it/ps_modules"
}
doLast {
dirs.each { dir ->
project.copy {
from "src/common"
into "target/$dir/ps_modules"
}
}
}
}

You can configure a single into for a task of type Copy. In this particular example gradle behaves as expected. Since dir3 is the last element on the list it is finally configured as a destination. Please have a look at this question - which you can find helpful. Also this thread might be helpful as well.

Related

Executing task on its own is working but not when executed from another one

I have three Gradle tasks if I execute them one by one on its own then its working. But when I execute them from another task then its not working. Here is how my task looks like
import com.github.gradle.node.npm.task.NpmTask
plugins {
id("com.github.node-gradle.node") version "3.4.0"
}
// Executing this task on its own is working
tasks.register<NpmTask>("buildFrontEnd") {
workingDir.set(file("${projectDir}/frontend"))
args.set(listOf("run", "build"))
}
// Executing this task on its own is working
tasks.register<Delete>("cleanFrontEnd") {
delete(
fileTree("${projectDir}/backend/main/resources/static/js"),
)
}
// Executing this task on its own is working
tasks.register<Copy>("copyFrontEnd") {
into("$projectDir")
copy {
from("${projectDir}/frontend/dist/css")
into("${projectDir}/backend/main/resources/static/css")
}
copy {
from("${projectDir}/frontend/dist/js")
into("${projectDir}/backend/main/resources/static/js")
}
}
// This tasks is not executing "copyFrontEnd"
tasks.register("frontEndBuild") {
dependsOn("buildFrontEnd", "cleanFrontEnd", "copyFrontEnd")
tasks.findByName("copyFrontEnd")?.mustRunAfter("buildFrontEnd", "cleanFrontEnd")
// Tried this too but it is not working
// tasks.findByName("cleanFrontEnd")?.mustRunAfter("buildFrontEnd")
// tasks.findByName("copyFrontEnd")?.mustRunAfter("cleanFrontEnd")
}
This is the output
> Task :cleanFrontEnd
> Task :copyFrontEnd NO-SOURCE
> Task :frontEndBuild
BUILD SUCCESSFUL in 20s
2 actionable tasks: 2 executed
For :copyFrontEnd it is saying NO-SOURCE but if that is the case then why its working when executing on its own? Is there anyway to fix this.
When you register a task, the bit inside the { } block configures the task. This block will always run whenever the task is required.
tasks.register<Copy>("copyFrontEnd") {
into("$projectDir")
// these copy actions will ALWAYS run whenever Gradle configures this task
copy {
from("${projectDir}/frontend/dist/css")
into("${projectDir}/backend/main/resources/static/css")
}
copy {
from("${projectDir}/frontend/dist/js")
into("${projectDir}/backend/main/resources/static/js")
}
}
Because you've used register when creating the task, Gradle won't execute the task configuration block won't unless the task is required. But when it is, those copy actions will run immediately.
To add task actions, which will actually do the work of the task, add them using doLast { } or doFirst { }.
tasks.register<Copy>("copyFrontEnd") {
into("$projectDir")
doLast {
copy {
from("${projectDir}/frontend/dist/css")
into("${projectDir}/backend/main/resources/static/css")
}
copy {
from("${projectDir}/frontend/dist/js")
into("${projectDir}/backend/main/resources/static/js")
}
}
}
Now those copy actions will only run when the task runs.
This still won't work however...
NO-SOURCE
Task did not need to execute its actions.
Task has inputs and outputs, but no sources
https://docs.gradle.org/current/userguide/more_about_tasks.html#sec:task_outcomes
When you register a Copy task, you need to specify what the source is.
tasks.register<Copy>("copyFrontEnd") {
into("$projectDir")
// from(...) // no from, no source - Gradle thinks "this task doesn't need to run"
}
In your case, you can specify the source as "${projectDir}/frontend/dist/css" and get rid of the two separate copy actions.
tasks.register<Copy>("copyFrontEnd") {
from("${projectDir}/frontend/dist/")
into("${projectDir}/backend/main/resources/static/css")
}
You might want to fiddle around with includes to only copy the css/ and js/ directories - https://docs.gradle.org/current/userguide/working_with_files.html#sec:copying_directories_example
Typesafe references
This last bit is unrelated, but I thought I'd include it as it can help make your buildscript clearer.
While you can reference task by their names, it's nicer to reference them by a variable. When you register a task, it returns a handle to that task.
You can use that handle to specify task dependencies.
val buildFrontEndTask: TaskProvider<NpmTask> = tasks.register<NpmTask>("buildFrontEnd") {
// ...
}
val cleanFrontEndTask: TaskProvider<Delete> = tasks.register<Delete>("cleanFrontEnd") {
// ...
}
val copyFrontEndTask: TaskProvider<Copy> = tasks.register<Copy>("copyFrontEnd") {
// ...
}
tasks.register("frontEndBuild") {
// here you can set the task dependencies based on the task providers
dependsOn(buildFrontEndTask, cleanFrontEndTask, copyFrontEndTask)
}

Gradle: process files inside a zip inside a tar

I have the following use case: I want to untar a tar file which contains a zip file. I then want to process files in the zip file (let's say I just want to filter out some file).
What I'm trying to do is something like:
task extractTar(type: Copy) {
from(tarTree("../a.tgz")) {
include '**/*.zip'
}
into "${buildDir}/tarOutput"
}
task unzipAndRezipZip(type: Zip, dependsOn: extractTar) {
archiveFileName = "rezipped.zip"
destinationDirectory
from(fileTree("${buildDir}/tarOutput").find { it.name.endsWith("zip") }) {
exclude '**/b.txt'
}
}
I've spent many many hours on this and every way I try to do it, it fails some way or another.
Expected Behavior
I can use the output files from a CopyTask in my ZipTask.
Current Behavior
path may not be null or empty string. path='null'
or
Circular dependency between the following tasks:
:unzipAndRezipZip
\--- :unzipAndRezipZip (*)
Steps to Reproduce
I have made a repo which has three examples of different things I tried: https://github.com/viktornordling/gradle-unzip-zip-from-tar-issue and clearly shows the repro steps for each example.
(I also put this in https://github.com/gradle/gradle/issues/16413, but I'm not sure how much love it will get there. Cross posting here to see if I can get some traction.)
Update: I would prefer not to use doLast in the solution as I feel that should not be needed. Usage of doLast is explicitly discouraged in the docs and not mentioned in the docs of CopyTask.
In example-4 from your repo https://github.com/viktornordling/gradle-unzip-zip-from-tar-issue you state that it almost works, but fails when you run clean unzipAndRezipZip in one go, with the error message Cannot expand ZIP...:
task unzipAndRezipZip(type: Zip) {
archiveFileName = "rezipped.zip"
destinationDir(file("${buildDir}/zipOutput"))
from(zipTree(tarTree("../a.tgz").find { it.name.endsWith("zip") })) {
exclude '**/b.txt'
}
}
You can fix that by changing the call to from() to a closure from { } to delay the execution of zipTree, i.e.:
task unzipAndRezipZip(type: Zip) {
archiveFileName = "rezipped.zip"
destinationDir(file("${buildDir}/zipOutput"))
from { zipTree(tarTree("../a.tgz").find { it.name.endsWith("zip") }) } {
exclude '**/b.txt'
}
}

Gradle Copy Task Is UP-TO-DATE After Manually Deleting a File

I am forced to use Gradle 2.3 in this project.
I am trying to copy a set of dependencies from a custom configuration to a specific dir.
If I delete one of the files manually, Gradle still marks the task as UP-TO-DATE and I end up with an incomplete set of files.
task copyFiles(type: Copy) {
from configurations.zips
into 'zip-dir'
configurations.zips.allDependencies.each {
rename "-${it.version}", ''
}
}
This works as expected in v4.0.2 though.
To work around it I am counting files in that dir.
task copyFiles(type: Copy) {
outputs.upToDateWhen {
def count = new File('zip-dir').listFiles().count { it.name ==~ /.*zip/ }
count == configurations.zips.files.size()
}
from configurations.zips
into 'zip-dir'
configurations.zips.allDependencies.each {
rename "-${it.version}", ''
}
}
Which issue and version of gradle was this fixed in and is there a better workaround than what I have so far?
You can just run it always with
outputs.upToDateWhen { false }
or not use a type Copy for your task and
task copyFiles {
doLast {
copy {
from configurations.zips
into 'zip-dir'
configurations.zips.allDependencies.each {
rename "-${it.version}", ''
}
}
}
}
Note that this is a workaround not the solution

Gradle copy without overwrite

Is there a way to avoid overwriting files, when using task type:Copy?
This is my task:
task unpack1(type:Copy)
{
duplicatesStrategy= DuplicatesStrategy.EXCLUDE
delete(rootDir.getPath()+"/tmp")
from zipTree(rootDir.getPath()+"/app-war/app.war")
into rootDir.getPath()+"/tmp"
duplicatesStrategy= DuplicatesStrategy.EXCLUDE
from rootDir.getPath()+"/tmp"
into "WebContent"
}
I want to avoid to specify all the files using exclude 'file/file*'.
It looks like that duplicatesStrategy= DuplicatesStrategy.EXCLUDE doesn't work. I read about an issue on gradle 0.9 but I'm using Gradle 2.1.
Is this problem still there?
Or am I misunderstanding how this task should be used properly?
Thanks
A further refinement of BugOrFeature's answer. It's using simple strings for the from and into parameters, uses the CopySpec's destinationDir property to resolve the destination file's relative path to a File:
task ensureLocalTestProperties(type: Copy) {
from zipTree('/app-war/app.war')
into 'WebContent'
eachFile {
if (it.relativePath.getFile(destinationDir).exists()) {
it.exclude()
}
}
}
You can always check first if the file exists in the destination directory:
task copyFileIfNotExists << {
if (!file('destination/directory/myFile').exists())
copy {
from("source/directory")
into("destination/directory")
include("myFile")
}
}
Sample based on Peter's comment:
task unpack1(type: Copy) {
def destination = project.file("WebContent")
from rootDir.getPath() + "/tmp"
into destination
eachFile {
if (it.getRelativePath().getFile(destination).exists()) {
it.exclude()
}
}
}

Remove a directory when unzipping

I am unpacking a zip file into a directory. The zip file has an extra top level directory that I don't want in the unzipped destination.
task unpackDojoSource(type: Copy) {
new File("build/dojo/src").mkdirs()
from(zipTree(dojoSource)) {
eachFile { details -> details.path =
details.path.substring(details.relativePath.segments[0].length()) }
} into "build/dojo/src"
}
The task produces the following output
/dijit
/dojo
/dojo-release-1.7.2
/dijit
/dojo
/dojox
/util
/dojox
/util
Is there a way I can prevent the dojo-release directory from being created?
Ref:
http://gradle.markmail.org/thread/x6gmbrhhen63rybe#query:+page:1+mid:lws7nlqcncjumnvs+state:results
I just stumble upon the same problem. Only thing you need to do is to add includeEmptyDirs = false into your copy spec like this:
task unpackDojoSource(type: Copy) {
new File("build/dojo/src").mkdirs()
from(zipTree(dojoSource)) {
eachFile { details -> details.path =
details.path.substring(details.relativePath.segments[0].length()) }
} into "build/dojo/src"
includeEmptyDirs = false
}
The resulting structure will not contain the empty directories left by flattening.
I've just ran across this myself. I worked around it by doing a delete afterwards. Less than ideal but still effective.
delete "dojo-release-1.7.2"

Resources