Suppose I have a folder (group var folder) in which there are multiple sub folders. The folder names are based on version number such as 1.0.1 and 1.1.1 etc. If I define the same variable in multiple sub folders, which one will Ansible pick?
All files in your group var folder and subfolders will be applied recursively.
Files and folders are sorted alphabetically on every level before processing.
The last processed variable assignment wins.
Here is the example of processing order:
./group_vars/testit/1.yml
./group_vars/testit/v0
./group_vars/testit/v0/1.yml
./group_vars/testit/v1
./group_vars/testit/v1/1.yml
./group_vars/testit/z.yml
In this case if testvar is a in v0/1.yml and b in v1/1.yml, testvar will have b value in the end.
Related
Objective: Copy all files from multiple subfolders into one folder with same filenames.
E.g.
Source Root Folder
20221110/
AppID1
File1.csv
File2.csv
/AppID2
File3.csv
File4.csv
20221114
AppID3
File5.csv
File6.csv
and so on
Destination Root Folder
File1.csv
File2.csv
File3.csv
File4.csv
File5.csv
File6.csv
Approach 1 Azure Data Factory V2 All datasets selected as binary
GET METADATA - CHILDITEMS
FOR EACH - Childitem
COPY ACTIVITY(RECURSIVE : TRUE, COPY BEHAVIOUR: FLATTEN)
This config renames the files with autogenerated names.
If I change the copy behaviour to preserve hierarchy, Both file name and folder structure remains intact.
Approach 2
GET METADATA - CHILDITEMS
FOR EACH - Childitems
Execute PL2 (Pipeline level parameter: #item.name)
Get Metadata2 (Parameterised from dataset, invoked at pipeline level)
For EACH2- Childitems
Copy (Source: FolderName - Pipeline level, File name - ForEach2)
Both approaches not giving the desired output. Any help/Workaround would be appreciated.
My understanding is in Option 2
Step 3 & 5 :is done as to iterate through the folder and subfolder correct ?
6 . Copy (Source: FolderName - Pipeline level, File name - ForEach2)
I think since in step 6 you already have the filename . On the SINK side , add an dynamic expression and add #Filename and that should do the trick .
If all of your files are in the same directory level, you can try the below approach.
First use Get Meta data activity to get all files list and then use copy inside ForEach to copy to a target folder.
These are my source files with directory structure:
Source dataset:
Based on your directory level use the wildcard placeholder(*/*) in the source dataset.
The above error is only a warning, and we can ignore it while debug.
Get meta data activity:
This will give all the files list inside subfolders.
Give this array to a ForEach activity and inside ForEach use copy activity.
Copy activity source:
In the above also, the */* should be same as we gave in Get Meta data.
For sink dataset create a dataset parameter and use in the file path of dataset.
Copy activity sink:
Files copied to target folder:
If your source files are not in same directory level then you can try the recursive approach mentioned in this article by #Richard Swinbank.
I have received multiple fastq.gz files from Illumina Sequencing for 100 samples. But all the fastq.gz files for the respective samples are in separate folders according to the sample ID. Moreover, I have multiple (8-16) R1.fastq.gz and R2.fastq.gz files for one sample. So, I used the following code for concatenating all the R1.fastq.gz and R2.fastq.gz into a single R1.fastq.gz and R2.fastq.gz.
cat V350043117_L04_some_digits-525_1.fq.gz V350043117_L04_some_digits-525_1.fq.gz V350043117_L04_some_digits-525_1.fq.gz > sample_R1.fq.gz
So in the sequencing file, the structure is like the above in the code. For each sample, the string with V has different number then L with different number and then another string of digits before the _1 and _2. For each sample, the numbers keep changing.
My questing is, how can I create a loop that will go over all the folders at once taking the different file numbering of sequence files into consideration for concatenating the multiple fq.gz files and combine them into a single R1 and R2 file?
Surely, I cannot just concatenate one by one by going into each sample folder.
Please give some helpful tips. Thank you.
The folder structure is the following:
/data/Sample_1/....._525_1_fq.gz /....._525_2_fq.gz /....._526_1_fq.gz /....._526_2_fq.gz
/data/Sample_2/....._580_1_fq.gz /....._580_2_fq.gz /....._589_1_fq.gz /....._589_2_fq.gz
/data/Sample_3/....._690_1_fq.gz /....._690_2_fq.gz /....._645_1_fq.gz /....._645_2_fq.gz
Below I have attached a screenshot of the folder structure.
Folder structure
Based on the provided file structure, would you please try:
#!/bin/bash
for d in Raw2/C*/; do
(
cd "$d"
id=${d%/}; id=${id##*/} # extract ID from the directory name
cat V*_1.fq.gz > "${id}_R1.fq.gz"
cat V*_2.fq.gz > "${id}_R2.fq.gz"
)
done
The syntax for d in Raw2/C*/ loops over the subdirectories starting with C.
The parentheses make the inner commands executed in a subshell so we don't have to care about returning from cd "$d" (at the expense of small extra execution time).
The variable id is assigned to the ID extracted from the directory name.
cat V*_1.fq.gz, for example, will be expanded as V350028825_L04_581_1.fq.gz V350028825_L04_582_1.fq.gz V350028825_L04_583_1.fq.gz ... according to the files in the directory and are concatenated into ${id}_R1.fastq.gz. Same for ${id}_R2.fastq.gz.
Suppose folder A contains 2000 files of same type.
I would like to copy 1000 random files from folder A to folder B.
what is the easiest way to do that in Ruby!!
Thanks!
S
Hints :
Dir.entries(<dir>) gives the list of filenames in the given directory as an array.
Array#sample(n) gives you n random elements taken from the array.
FileUtils.cp(<src>, <dest>) helps you copy a file from a dir to another (you need to require fileutils for this though.
I wrote the below condition to get all *.js files from a folder and its subfolder. I wrote the below command to get the list:
c:/svn/myfolder/**/*.js
"myfolder" has 4 sub folders a, b, c, d in it and i want to get list of .js files only from three of the subfolders and skip checking the subfolder 'c'.
Currently it returns all .js files as i put myfolder/**/*.js
Ruby's glob syntax borrows from regex. You can specify a range of characters using brackets.
Dir['c:/svn/myfolder/[abd]/*.js']
This will include a, b, and d but not c. You can also negate:
Dir['c:/svn/myfolder/[^c]/*.js']
This will include all but c
See http://ruby-doc.org/core-2.1.3/Dir.html#method-c-glob for more info on Ruby globs.
You can use Dir glob to get all the files and directories as a list. To accomplish this with regex do:
Dir.glob(File.join(['c:/svn/myfolder', '*[^excluded_folder]', '**', '*.js']))
The *[^excluded_folder] includes everything but excluded_folder.
Keeping the ** as part of your path will grab files from the current directory path it's placed at as well as all subdirectories.
I'm trying to create a directory tree for a specific set of folders at a certain level. I found how to limit the Tree results using Regular Expressions but I'm unsure as to how I can limit the resulting tree to only say, directories at the first level instead of it finding all sub-directories recursively.
In Linux it's possible to do tree -L 1 to limit results to the first level. How can I do this in Windows cmd?
Example - If my folder directory is as follows:
Folder A
Folder B
Folder C
Folder D
Folder E
I want my tree result to be:
--Folder A
----Folder B
--Folder D
----Folder E
(Only showing 2 levels)
The Tree help is as follows:
Graphically displays the folder structure of a drive or path.
TREE [drive:][path] [/F] [/A]
/F Display the names of the files in each folder.
/A Use ASCII instead of extended characters.
You can't without using some other tool. In an app in C# for instance this would be a few lines of code.
The batch file CDS.bat that is here claims to do what you want. I haven't tested it.