I have a directory structure like this:
file1.txt
file2.txt
dir1/
file3.txt
file4.txt
I want to use Gradle to copy that entire structure into another directory. I tried this:
task mytest << {
copy {
from "file1.txt"
from "file2.txt"
from "dir1"
into "mytest"
}
}
But this results in the following:
mytest/
file1.txt
file2.txt
file3.txt
file4.txt
See, the copy from dir1 copied the files in dir1, whereas I want to copy dir1 itself.
Is it possible to do this directly with Gradle copy?
So far, I have only been able to come up with this solution:
task mytest << {
copy {
from "file1.txt"
from "file2.txt"
into "mytest"
}
copy {
from "dir1"
into "mytest/dir1"
}
}
For my simple example there's not much to it, but in my actual case there are many directories I want to copy, and I'd like to not have to repeat so much.
You can use . as the directory path and include to specify, which files and directories you want to copy:
copy {
from '.'
into 'mytest'
include 'file*.txt'
include 'dir1/**'
}
If both from and into are directories, you'll end up with the full copy of the source directory in the destination directory.
I know this is a bit late, but I tried #Andrew solution above and it copied everything inside the directory.
"." is not required nowadays to represent a direct in gradle.
So I did some research
and found this
and created the following code (with Up-to-date check) based on it:
task resourcesCopy() {
doLast {
copy {
from "src/main/resources"
into "./target/dist/WEB-INF/classes"
}
copy {
from "GeoIP2.conf"
into "./target/dist/WEB-INF"
}
}
}
I don't know how long this syntax has been around, but it seems a little clearer.
task copyToRelease(dependsOn: [deletePreviousRelease], type: Copy) {
from('build/dist') {
include '**/*.*'
}
destinationDir(new File('../htmlrelease/src/main/webapp/canvas'))
}
Maybe also helpful: Usage of fileTree to copy an entire directory recursively, e.g.,
task mytest << {
copy {
from fileTree('.')
into "mytest"
}
}
Related
I have a huge maven project, lot of people are using it. I'm currently working on converting it to gradle. One of the last steps will be that I merge the gradle files, and delete the pom.xml files. But I'd like to add a gradle task to clean the maven target directories (of all the sub-projects). In shell I would do something like:
find . -type d -name target -exec rm -rf "{}" \;
But I prefer this to be a gradle task. How do I add it? This is what I tried but it doesn't delete anything:
task cleanMaven(type: Delete) {
delete fileTree('.').matching { include '**/target/**' }
}
below will handle all modules of root project and prints true if a target dir existed and is deleted
allprojects {
task mvnClean {
doFirst {
def targetPath = project.projectDir.toString() + '/target'
println "target dir exists: ${Files.deleteIfExists(Paths.get(targetPath))}"
}
}
}
Based on #PrasadU's answer, but this also deletes all the contents of the target/ directories:
allprojects {
task mvnClean {
doFirst {
def targetPath = project.projectDir.toString() + '/target'
project.delete(files("${targetPath}"))
}
}
}
I have a task:
task myCopy(type: Copy) {
from 'from.txt'
into 'into.txt'
doLast {
//Read write 'into.txt'
}
}
It always print a FileNotFoundException, I find out that doLast block is executed before the copy operation has complete, so how can I read (or write) with into.file when my copy operation is completed?
Here you are not using Copy task in a proper way: the "into" property is the target directory where files will be copied, not the target file name.
When executing your script you should have a directory "input.txt" created at the root directory of your project, and the file "from.txt" copied into it , which is not what you expect, and I guess this is the cause for you FileNotFoundException .
If you want to copy/rename file, you should use the 'rename' method from CopyTask, described here : https://docs.gradle.org/current/dsl/org.gradle.api.tasks.Copy.html#org.gradle.api.tasks.Copy:rename(java.lang.String,%20java.lang.String)
Exemple:
task myCopy(type: Copy) {
from './source-file.txt'
into './output-dir'
rename ( 'source-file.txt', 'input.txt')
doLast{
println "Is file already copied when executing doLast ? ==> "+ project.file("./output-dir/input.txt").exists()
}
}
This should output:
$ gradle myCopy
> Task :myCopy
Is file already copied when executing doLast ? ==> true
The problem
Suppose that I have written a lengthy script
in some language "lang",
and now want to convert this single-file script
into a directory tree with a project consisting of many files. I want to insert some kind of separators and file-paths into this file, and process it in some way so that in the end I obtain:
a proper project directory layout (sth. like this),
build-definition file,
readme's,
separate subdirectories for main/src and test/src etc.
For example, given the following script (pseudocode):
// required dependencies, should be moved
// into the build definition build.foo
require "org.foo" % "foo-core" % "1.2.3"
require "org.bar" % "bar-gui" % "3.2.1"
// A longer comment that should be converted
// into a text file and moved into a 'notes'
// subdirectory
/*
#README
Another lengthy comment that should go into
a readme.md
*/
/** A class that should
* go to src/main/lang/proj/A.lang
*/
class A {
def a = "foo"
}
/** Another class
* that should go to src/main/lang/proj/B.lang
*/
class B {
def b = "bar"
}
/** Some tests,
* should end up in
* src/test/lang/proj/MyTest.lang
#Test def testFoo() {
assert(2 + 2 == 5)
}
and assuming that I can insert arbitrary separators, commands, escape-sequences and file paths into this file, I would like to obtain the following project:
project/
|-- build.txt
|-- notes
| `-- note_01.txt
|-- readme.md
`-- src
|-- main
| `-- lang
| `-- proj
| |-- A.lang
| `-- B.lang
`-- test
`-- lang
`-- proj
`-- MySpec.lang
Edit:
What follows is a less-sophisticated version of my own answer below
What I've tried
Here is one naive way to do it:
Convert the original script into a bash script by prepending #!/bin/bash
split the source code into HEREDOCS
insert package declarations where necessary
add bunch of mkdir -p and cd between the HEREDOC-pieces
cat the HEREDOC pieces into appropriately named files
test the script on empty directories until it works as expected
For the above script, it might look somehow like this:
#!/bin/bash
mkdir project
cd project
cat <<'EOF' > build.txt
// required dependencies, should be moved
// into the build definition build.foo
require "org.foo" % "foo-core" % "1.2.3"
require "org.bar" % "bar-gui" % "3.2.1"
EOF
mkdir notes
cd notes
cat <<'EOF' > note_01.txt
// A longer comment that should be converted
// into a text file and moved into a 'notes'
// subdirectory
EOF
cd ..
cat <<'EOF' > readme.md
/*
#README
Another lengthy comment that should go into
a readme.md
*/
EOF
mkdir -p src/main/lang/proj
cd src/main/lang/proj
cat <<'EOF' > A.lang
package proj
/** A class
* that should go to src/main/lang/proj/A.lang
*/
class A {
def a = "foo"
}
EOF
cat <<'EOF' > B.lang
package proj
/** Another class
* that should go to src/main/lang/proj/B.lang
*/
class B {
def b = "bar"
}
EOF
cd ../../..
mkdir -p test/lang/proj
cd test/lang/proj
cat <<'EOF' > MySpec.lang
package proj
/** Some tests,
* should end up in
* src/test/lang/proj/MyTest.lang
#Test def testFoo() {
// this should end up in test
assert(2 + 2 == 5)
}
EOF
cd ../../..
What's wrong with this approach
It does generate the correct tree, but this approach seems rather error-prone:
it's too easy to cd ../../.. to the wrong nesting level
too easy to mkdir with a wrong name, and then fail to cd into it.
There is no way to handle the entire tree construction as a single
transaction, that is, if something fails later in the script,
there is no simple way to clean up the mess generated before
the error occurred.
I certainly could try to make it a bit less brittle by defining
special functions that mkdir and cd in one go, and
then wrap invocations of those functions together with cats into
(mkdirAndCd d ; cat) etc.
But it just doesn't feel quite right. Isn't there a much simpler
way to do it? Could one somehow combine the standard bash/linux utilities
into a tiny & very restricted domain specific language for
generating directory trees with text files? Maybe some newer version of split where one could specify where to split and where to put the pieces?
Related questions:
mkdir and touch in single command
The reverse of tree - reconstruct file and directory structure from text file contents?
Other interesting proposals that don't seem to work:
Use tar. That would mean that one would have to convert the text file manually into a valid tar-archive. While a tar archive indeed is a single plain-text file, its internal format does not look like the most comfortable DSL for such a simple task. It was never intended to be used by humans directly in that way.
Similar argument holds for shar. Since shar uses the bash itself to extract the archive, my above proposal is, in principle, a manually generated shar-archive in a very uncommon format, therefore shar seems to share all the drawbacks with the above proposal. I'd rather prefer something more restricted, that allows to do fewer things, but provides more guarantees about the quality of the outcome.
Maybe I should emphasize again that I don't have a tree to begin with, so there is nothing to compress. I have only the single script file and a rough idea of what the tree should look like in the end.
Seems to me that you are trying to write a custom parser. Provided that all blocks mentioned by you are ended by double line endings, this could help you
#!/bin/bash
gawk 'BEGIN{RS="\n\n([/][*]|[/]{2,2})"}
{
if ($0 ~ /#README/){
system("echo -e \"\nThis is a Readme.md\n--------\n" $0 "\"")
}else if ($0 ~ /class /){
system("echo -e \"\nThis is a class\n---------\n/*" $0 "\"")
}else if ($0 ~ /require /){
system("echo -e \"\nthis is a conf\n-----------\n" $0 "\"")
}else if($0 ~ /[/]{2,2}.*\n[/]{2,2}/){
system("echo -e \"\nthis is a note\n-----------\n" $0 "\"")
}
}' your_script.lang
The key part is the record separator RS that splits block of code that start with '\n\n//' or '\n\n/*'.
Instead of echo -e you could write custom scripts for each type of block.
Please note that the record separator will not be present on $0 so you have to add the missing characters, as in the /class / example above.
The output of the above code is
this is a conf
-----------
// required dependencies, should be moved
// into the build definition build.foo
require org.foo % foo-core % 1.2.3
require org.bar % bar-gui % 3.2.1
this is a note
-----------
A longer comment that should be converted
// into a text file and moved into a 'notes'
// subdirectory
This is a Readme.md
--------
#README
Another lengthy comment that should go into
a readme.md
*/
This is a class
---------
/** A class that should
* go to src/main/lang/proj/A.lang
*/
class A {
def a = foo
}
This is a class
---------
/** Another class
* that should go to src/main/lang/proj/B.lang
*/
class B {
def b = bar
}
About your concerns:
it's too easy to cd ../../.. to the wrong nesting level
-> define a variable with root path and cd to it.
too easy to mkdir with a wrong name, and then fail to cd into it.
-> define variables with directory names and check if they already exists.
path1=src/main/lang/some
if [ -d $path1 ]; then
do_something
fi
There is no way to handle the entire tree construction as a single transaction ...
-> write to file paths of every NEW directory/file that you create and use it to revert if necessary.
(my own answer)
Consider the following definition of a tiny embedded domain specific language for defining directory trees with text files:
#!/bin/bash
enter() {
local subdir="$1"
if [ ! -d "$subdir" ]
then
mkdir -p "$subdir"
fi
pushd "$subdir" > /dev/null
}
leave() {
popd > /dev/null
}
save() {
local fileName="$1"
cut -d'|' -f2- > "$fileName"
}
The enter command creates a directory if necessary, and cds into this directory, it works with arbitrary relative paths. The save command saves the text content of a here-document to file. The leave command changes to previous directory.
When a file is saved, the margin (empty space followed by '|') is stripped from each line. This is to ensure that the indentation of the script does not interfere with the indentation of the written files.
If these definitions are sourced, then the tree-generation script can be written as follows:
#!/bin/bash
source explode.sh
mkdir project
cd project
enter "src"
enter "main/lang/proj"
save "A.lang" <<'____EOF'
|package proj
|
|/** A totally useful class
| * that should go to src/main/lang/proj/A.lang
| */
|class A {
| def a = "foo"
|}
____EOF
save "B.lang" <<'____EOF'
|package proj
|/** Another very useful class
| * that should go to src/main/lang/proj/B.lang
| */
|class B {
| def b = "bar"
|}
____EOF
leave
enter "test/lang/proj"
save "MyTest.lang" <<'____EOF'
|package proj
|
|/** A test that should end up in
| * src/test/lang/proj/MyTest.lang
|#Test def testFoo() {
| assert(2 + 2 == 5)
|}
____EOF
leave
leave
save "build.txt" <<'EOF'
|require "org.foo" % "foo-core" % "1.2.3"
|require "org.bar" % "bar-gui" % "3.2.1"
EOF
enter "notes"
save "note_01.txt" <<'__EOF'
|A longer comment that should be converted
|into a text file and moved into a 'notes'
|subdirectory. This is a very long comment
|about the purpose of the project. Blah
|blah blah.
__EOF
leave
save "README.md" <<'EOF'
|#README
|
|This is a readme file for my awesome project.
|It ends with this line. Bye.
EOF
When executed, the script generates the following directory tree:
project/
├── build.txt
├── notes
│ └── note_01.txt
├── README.md
└── src
├── main
│ └── lang
│ └── proj
│ ├── A.lang
│ └── B.lang
└── test
└── lang
└── proj
└── MyTest.lang
The bash-script mirrors the tree structure very closely, and it's impossible to mess up the cd ../../../../../.. commands. It still lacks various desirable properties though (not transactional, no dry-run capability).
I just observed a very weird behaviour from a Gradle Tar task.
Let's take a simple example, 2 files :
/tmp/test$ ls
test1.txt ##test2##
Here is a simple Tar task :
task('testHash', type: Tar) {
from "/tmp/test"
extension = 'tar.gz'
compression = Compression.GZIP
}
The file ##test2## is skipped for some reason, after running gradle testHash :
/path/to/gradle/project/foo$ tar tvf build/distributions/foo-1.0.tar.gz
test1.txt
It seems to happen when the filename is containing # character both at the beginning and the end.
A regular tar is working well :
/tmp/test$ tar czvf test.tar.gz *
test1.txt
##test2##
/tmp/test$ tar tf test.tar.gz
test1.txt
##test2##
I am using Gradle 4.1. Any explanation ?
Thanks to Opal's comments, I adjusted my searches and found a workaround. There is maybe a cleaner way but this one works for me
task('testHash', type: Tar) {
doFirst {
org.apache.tools.ant.DirectoryScanner.defaultExcludes.each {
DirectoryScanner.removeDefaultExclude it
}
}
from "/tmp/test"
extension = 'tar.gz'
compression = Compression.GZIP
}
FYI, here are default excludes
There are a set of definitions that are excluded by default from all
directory-based tasks. As of Ant 1.8.1 they are:
**/*~
**/#*#
**/.#*
**/%*%
**/._*
**/CVS
**/CVS/**
**/.cvsignore
**/SCCS
**/SCCS/**
**/vssver.scc
**/.svn
**/.svn/**
**/.DS_Store
Ant 1.8.2 adds the following default excludes:
**/.git
**/.git/**
**/.gitattributes
**/.gitignore
**/.gitmodules
**/.hg
**/.hg/**
**/.hgignore
**/.hgsub
**/.hgsubstate
**/.hgtags
**/.bzr
**/.bzr/**
**/.bzrignore
I have a gradle build script in which I'm using the 'Zip' task to generate a zip archive directly from a source directory.
In addition to copying over all the files inside the source directory structure into the zip archive, I'm looking for a way to include a dynamically generated file that's not in the source directory.
Do you guys know how I can do this?
Here's pseudo code of what I want to do:
task('archive', type: Zip) {
...
from 'source'
newFile('dynamically-generated.txt', 'dynamic file content')
}
And here are the source and destination structures:
[i71178#bddev01-shell01 ~]$ tree source/
source/
├── file1.txt
├── file2.txt
└── file3.txt
0 directories, 3 files
[i71178#bddev01-shell01 ~]$ unzip -l destination.zip
Archive: destination.zip
Length Date Time Name
--------- ---------- ----- ----
0 02-26-2016 16:56 source/
0 02-26-2016 16:56 source/file2.txt
0 02-26-2016 16:56 source/dynamically-generated.txt
0 02-26-2016 16:56 source/file1.txt
0 02-26-2016 16:56 source/file3.txt
--------- -------
0 5 files
Combining my comments above and your getTemporaryDir comment:
task generateThenZip()<<{
def tempDir = getTemporaryDir()
newFile("$tempDir/dynamically-generated.txt", 'dynamic file content')
zip{
from 'source'
from tempDir
}
}
This is what I ended up doing:
subprojects {
apply plugin: 'myPlugin'
//The 'build' task is set up by the plugin
build {
//Customization to keep consistent with artifacts being currently generated.
doFirst {
new File(getTemporaryDir(), 'artifact-fullname.txt').text = "${project.name}-${project.version}\n"
}
archiveName = "${project.name}.${project.build.extension}"
from getTemporaryDir()
}
}
Thanks!
I have managed to achieve it quite easyly on Gradle 6.6.1
by just creating the file in the from section.
distributions {
main {
contents {
into("/") {
from {
if (!buildDir.exists()) {
buildDir.mkdir()
}
def f = new File("$buildDir/all")
f.text = "project_version: ${project.version}\n"
f
}
}
...
}
}
}
btw. I think $buildDir is safer than /tmp