I have the following folder structure under build folder (which you get during a Gradle build):
CDROM/disk1
CDROM/disk1/disk1file1a.txt
CDROM/disk1/disk1file1b.txt
CDROM/disk2/disk2file2a.txt
CDROM/disk2/disk2file2btxt
CDROM/disk2/disk2folder2x
CDROM/disk2/disk2folder2y
CDROM/disk3
CDROM/disk3/disk3
CDROM/disk3/disk33
CDROM/disk3/disk33/disk3
CDROM/folder1
CDROM/file1.txt
How can I tell Gradle to show me the following:
Print only the top level / direct child folders (only) in folder "CDROM"
i.e. it should print only disk1, disk2, disk3 and folder1
Print only the top level / direct child folders (only) which has a pattern of disk[0-9] i.e. diskX where X is a number.
i.e. it should print only disk1, disk2 and disk3
The following will do it, but I think there should be an efficient way to achieve the same and where one can define patterns and DON'T have to use "IF" statements that I have used below.
FileTree dirs = fileTree (dir: "$buildDir/CDROM", include: "disk*/**")
dirs.visit { FileVisitDetails fd ->
if (fd.directory && fd.name.startsWith('disk')){
println "------ $buildDir/CDROM_Installers/${fd.name} ---------------"
}
}
By top level if you mean only direct children of CDROM then this should be as easy as:
new File("${buildDir}/CDROM").eachDir{ if(it.name ==~/disk.*/) println it}
If you want more control on depth and other things, then you can try variations of following code:
new File("${buildDir}/CDROM").traverse( [maxDepth: 2, filter: ~/.*disk\d/,
type: groovy.io.FileType.DIRECTORIES]){
println it // or do whatever
}
see traverse for more details.
Related
Good day. I have some directories (shown in bold below) each having some .fastq files for different lanes.
CND1/ UD_LOO3_R1.fastq.gz UD_LOO4_R1.fastq.gz
CND2/ XD_L001_R1.fastq.gz XD_L004_R1.fastq.gz
Inside each directory, i want to create a merged fastq file that will be named as : sample_R1.fastq.gz. For instance, CND1/UD_R1.fastq.gz and CND2/XD_R1.fastq.gz etc and so on. To this end, i created the following snakemake workflow.
from collections import defaultdict
dirs,samp,lane = glob_wildcards("dir}/{sample}_L{lane}_R1.fastq.gz")
dirs, fls = glob_wildcards("{dir}/{files}_R1.fastq.gz")
D = defaultdict(list)
for x,y in zip(dirs,fls):
D[x].append(y+'_R1.fastq.gz')
rule ML:
message:
"Merge all Lanes for Fragment R1"
input:
expand( "{dir}/{files}",zip,dir=D.keys(),files=D.values() )
output:
expand( "{dir}/{s}_R1.fastq.gz",zip,dir=dirs,s=set(samp) )
shell:
"echo {input} && echo {output} "
#"zcat {input} >> {output}"
In the code above, dict D contains directories as keys and list of fastq as values.
{ 'CND2': ['XD_L001_R1.fastq.gz', 'XD_L004_R1.fastq.gz'], 'CND1': ['UD_LOO3_R1.fastq.gz', 'UD_LOO4_R1.fastq.gz'] }
Doing a dry-run, snakemake complains me of missing files as follows
Missing input files for rule ML:
CND2/['XD_L001_R1.fastq.gz', 'XD_L004_R1.fastq.gz']
CND1/['UD_LOO3_R1.fastq.gz', 'UD_LOO4_R1.fastq.gz']
I want to understand what is the correct way to provide both a directory and a list of files as input together. Any help is greatly appreciated.
Thanks.
The error clearly explains the issue. As a result of the expand function, your input is two files:
CND2/['XD_L001_R1.fastq.gz', 'XD_L004_R1.fastq.gz']
CND1/['UD_LOO3_R1.fastq.gz', 'UD_LOO4_R1.fastq.gz']
If you need the input to be four files like that:
CND1/UD_LOO3_R1.fastq.gz
CND1/UD_LOO4_R1.fastq.gz
CND2/XD_L001_R1.fastq.gz
CND2/XD_L004_R1.fastq.gz
you need to flatten your dictionary:
inputs = [(dir, file) for dir, files in D.items() for file in files]
rule ML:
input:
expand( "{dir}/{files}", zip, dir=[row[0] for row in inputs], files=[row[1] for row in inputs])
or alternatively:
inputs = [(dir, file) for dir, files in D.items() for file in files]
rule ML:
input:
expand( "{filename}", filename=[f"{dir}/{file}" for dir, file in inputs])
Overall you are overcomplicating the problem. The same can be done without this ugly juggling the lists of tuples. glob_wildcards("{filename_base}_R1.fastq.gz") should return you more convenient representation.
I have a custom Gradle plugin that will generate Java files from a template file. I have several such template files in different locations, and I need to "compile" all of them to generate the Java files I need. Once I have the files, I want to package them into a .jar.
One way I thought I could do this was to call the "compile template" task multiple times from within the same build file. I'd call it once in a task that compiles template files in location A, again from a task that compiles template files from location B... etc., until I have all the Java files I need.
Something like this:
task compileFromLocationA <<{
compileTemplate.execute(A)...
}
task compileFromLocationB
compileTemplate.execute(B)...
...
packageJar(depends: compileFromLocationA, compileFromLocationB, ...)
...
However, you can't programmatically call a task from within another task. I suppose I could break each compileFromLocation_ task into it's own build.gradle file, but that seems like overkill. What's the "best practice" in a case like this?
This code seems to work in build.gradle by using tasks.register() - e.g. to perform multiple source code generating steps - in my case I needed to load different pairs of files (XML schema and generation options) in two different steps:
plugins {
id 'java'
id "com.plugin" version "1.0"
}
sourceSets.main.java.srcDirs += file("${buildDir}/genSrc")
sourceSets.test.java.srcDirs += file("${buildDir}/testGenSrc")
tasks.compileJava {
dependsOn tasks.named("genMessage")
}
genMessage {
codesFile = "${projectDir}/src/main/resources/file.xml"
}
def testGenModel1 = tasks.register("testGenModel1", com.plugin.TestGenModelTask.class) {
schema = "${projectDir}/src/test/resources/file.xsd"
options = "${projectDir}/src/test/resources/file.xml"
}
def testGenModel2 = tasks.register("testGenModel2", com.plugin.TestGenModelTask.class) {
schema = "${projectDir}/src/test/resources/file2.xsd"
options = "${projectDir}/src/test/resources/file2.xml"
}
tasks.compileTestJava {
dependsOn tasks.named("testGenModel1"), tasks.named("testGenModel2")
}
I have parent and child project. Parent's build.gradle is empty, settings.xml contains include 'child' and
in build.gradle of child I have a task
task('executionPath') << {
println projectDir
}
This task is supposed to print the path to project on which the build was started.
If I invoke it in root by ./gradlew executionPath I expect it to show path of the root project, e.g. C:\projects\parent.
If I invoke it in root by ./gradlew child:executionPath I expect it to show path of the child project, e.g. C:\projects\parent\child.
I've tried the following:
projectDir always path to child
new File('.') always path to parent
System.getProperty("user.dir") always path to parent
Answer Gradle: get folder from which "gradle" was executed is not helpful in my case. How can I achieve the above?
It's not very straightforward solution, but you can use start parameters to find out, whether the task was called for the root project or for the current. Something like this:
task('executionPath') << {
//find the argument representing current task
String calledTaskName = null;
for (String taskArgument : project.getGradle().startParameter.taskRequests.get(0).args) {
if (taskArgument.equals(name) || taskArgument.endsWith(':'+name)) {
calledTaskName = taskArgument;
}
}
if (calledTaskName == null) {
println 'Task was not called via arguments'
return;
}
//check, whether task was called on root project or for subproject only
if (calledTaskName.startsWith(project.getPath())) {
println projectDir
} else {
println System.getProperty("user.dir")
}
}
This task is looking within start parameters for the current task name. If it was called via start parameters, it checks, whether task name contains current project name as a prefix and according to it prints out current project path or root project path.
Unfortunately, I don't know any other solution for your case. Sure, you may need to modify it for your exact purposes.
I need to copy files under the list of directories to its corresponding directories in destination list. Say I have a list of source directories like 'A','B','C' and a list of target directories like 'X','Y','Z'. What I need to do is to copy files under A directory to be copied to X directory and from B to Y and C to Z. I have created a gradle task for this purpose. But I get an error
task copyDirs( ) {
def targetDirList = ['/target1', '/target2', '/target3'].toArray()
def sourceDirList = ['/source1', '/source2', '/source3'].toArray()
[sourceDirList,targetDirList].transpose().each {
copy{
from it[0].toString()
into it[1].toString()
}
}
}
And below is the exception I get when I try to execute it
No signature of method: org.gradle.api.internal.file.copy.CopySpecWrapper_Decorated.getAt() is applicable for argument types: (java.lang.Integer) values: [0]
Possible solutions: getAt(java.lang.String), putAt(java.lang.String, java.lang.Object), wait(), grep(), getClass(), wait(long)
It's because the it you're using is relating to the copy closure, not the values you're iterating through. Just name your element:
[sourceDirList,targetDirList].transpose().each { d ->
copy{
from d[0].toString()
into d[1].toString()
}
}
Can gradle alter the structure of the tree while copying?
original
mod/a/src
mod/b/src
desired
dest/mod-a/source
dest/mod-b/source
dest/mod-c/source
I'm not sure where I should create a closure and override the copy tree logic
I'd like to do the gradle equivalent of ant's globmapper functionality
<property name="from.dir" location=".."/>
<property name="to.dir" location="dbutil"/>
<copy>
<fileset dir="${from.dir}" ... />
<globmapper from="${from.dir}/*/db" to="${to.dir}"/>
</copy>
Thanks
Peter
When changing file name, rename seems a good approach. When changing path you can override eachFile and modify the destination path.
This works pretty well.
copy {
from("${sourceDir}") {
include 'modules/**/**'
}
into(destDir)
eachFile {details ->
// Top Level Modules
def targetPath = rawPathToModulesPath(details.path)
details.path = targetPath
}
}
....
def rawPathToModulesPath(def path) {
// Standard case modules/name/src -> module-name/src
def modified=path.replaceAll('modules/([^/]+)/.*src/(java/)?(.*)', {"module-${it[1]}/src/${it[3]}"})
return modified
}
Please see sample below. Gradle 4.3 does not have rename/move methods, so we can do renaming on the fly.
What was happened:
Load file tree into the memory. I used zip file from dependencies in my example
Filter items, which are in the target folder
All result items will have the same prefix: if we filter files from directory "A/B/C/", then all files will be like "A/B/C/file.txt" or "A/B/C/D/file.txt". E.g. all of them will start with the same words
In the last statement eachFile we will change final name by cutting the directory prefix (e.g. we will cut "A/B/C").
Important: use type of task "Copy", which has optimizations for incremental compilation. Gradle will not do file copy if all of items below are true:
Input is the same (for my case - all dependencies of scope "nativeDependenciesScope") with previous build
Your function returned the same items with the previous build
Destination folder has the same file hashes, with the previous build
task copyNativeDependencies(type: Copy) {
includeEmptyDirs = false
def subfolderToUse = "win32Subfolder"
def nativePack = configurations.nativeDependenciesScope.singleFile // result - single dependency file
def nativeFiles = zipTree(nativePack).matching { include subfolderToUse + "/*" } // result - filtered file tree
from nativeFiles
into 'build/native_libs'
eachFile {
print(it.path)
// we filtered this folder above, e.g. all files will start from the same folder name
it.path = it.path.replaceFirst("$subfolderToUse/", "")
}
}
// and don't forget to link this task for something mandatory
test.dependsOn(copyNativeDependencies)
run.dependsOn(copyNativeDependencies)
The following works, but is there a more gradle-ish way to do this?
ant.copy(todir: destDir) {
fileset( dir: "${srcDir}/module", includes: '**/src/**')
regexpmapper(from: '^(.*)/src/(.*)$', to: /module-\1\/src\/\2/)
}