Creating complex path structure in linux - bash

I am having trouble creating the following directory structure. It is to test a scanner program to see if it can handle it.
Start inside /data/
Inside /data/ there are 2 sub-directories. /data/sub1 and /data/sub2
Now, this gets complex :)
In each of the sub-directories there are 2 more sub-directories. For first sub: /data/sub1/sub1/ and /data/sub1/sub2 and for second sub /data/sub2/sub1/ and /data/sub2/sub2
... this goes on to a depth of 16 levels, with a total directory of 65,536.
What I have so far, this only creates 1 subdirectory inside /data/, 16 folders deep. I am thinking of using the following script recursively for the subdirectories created along the way. And for the subdirectories it then creates. And so on. But I can't imagine scripting it.
count=0;
folder=$(printf "Folder%02dB" $count);
mkdir -p $folder;
while [ $count -lt 16 ]; do
((count=$count+1));
folder=$(printf "Folder%02dB" $count);
mkdir -p $folder;
cd $folder;
done
Anyone has any ideas on how to create such directory structure? How to proceed? Thanks!

Recursion is a good way to do this.
#! /bin/bash
godeep()
{
if [ $1 -lt 16 ]
then
((level = $1 + 1))
mkdir sub1
cd sub1
godeep $level
cd ..
mkdir sub2
cd sub2
godeep $level
cd ..
fi
}
godeep 1
~

You shouldn't be incorporating $count in the folder name. That's just used to count the recursion depth.
Simplest would be to define a function to create a single level, and call it recursively.
create_folders() {
if (( $1 > 16 )); then
return
fi
depth=$(($1 + 1))
mkdir sub1; (cd sub1; create_folders $depth)
mkdir sub2; (cd sub2; create_folders $depth)
}
create_folders 1
Using () around the cd and recursive call keeps them from affecting the working directory and variables in the current call.

Related

I want to name a file with the name of a directory in bash

I'm trying to create files with the name of it's antepenultimate directory:
Example:
Directory: a/b/c/d/e/f/g/h/i/j
The name of folder h is different for each case.
So I created an array
array=(/ a / b / c / d / e / f / g / * / * / *)
len=${#array[#]}
for (( q=0; q<$len; q++ ));
do
cd ${array[$q]}
sleep 1
mri_convert 0001*.dcm RAW.nii.gz #--->this line is just converting the format of file 0001*.dcm in to file RAW.nii.gz
done
This code is working but I want the file RAW.nii.gz to be named h_RAW.nii.gz
I tried doing this:
s1="${array%/*/*}"
$ echo "${s1##*/}"
and then:
mri_convert 0001*.dcm ${s1##*/}_RAW.nii.gz
but it's not working.
How about
cd /a/b/c/d/e/f/g
for dir in *; do
[[ -d $dir ]] || continue
for subdir in "$dir"/*/*/; do (
# doing this in a subshell so we don't need to "undo" this cd
cd "$subdir"
mri_convert 0001*.dcm "${dir}_RAW.nii.gz"
); done
done
Let's see if I can help. I'm not exactly sure of the details of what you're trying to do (mainly because the code you posted:
for (( q=0; q do cd ${array[$q]} sleep 1 mri_convert 0001*.dcm RAW.nii.gz
is not syntactically correct. So, that can't be what you're actually doing.
Just a hint of how i would approach a problem like this:
for path6 in /a/b/c/*/*/*
do
path5="${path6##*/}"
path4="${path5##*/}"
name4="${path4%/*}"
echo "Processing ${path4}"
mriconvert "${path6}"/0001*.dcm "${path6}/${name4}_RAW.nii.gz"
done

Recursively create directories for all letters

I would like to create a folder structure based on a brace expansion such as {a-z}. Each string generated by the brace-expansion should be a new folder. Furthermore, each of these folders should contain the same set of subfolders similarly generated. And this up to a given level.
An example for the range a-z and depth 16
a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/
a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/b/
a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/c/
...
d/a/h/r/y/d/s/b/e/y/k/f/o/o/q/c/
...
z/z/z/z/z/z/z/z/z/z/z/z/z/z/z/y/
z/z/z/z/z/z/z/z/z/z/z/z/z/z/z/z/
The following code allows me to go upto depth 2:
for x in {a..z} ; do mkdir -p $x/{a..z} ; done
But how do I go further?
A recursive solution. Job is called with 2 params: max_depth and base_path
#!/bin/bash
function job()
{
local depth=$(($1-1))
local path=$2
local x
for x in a b c # reduced for test
do
mkdir -p "$path/$x"
((depth>0)) && job $depth "$path/$x"
done
}
job 3 ./test
Proof it with:
find test -type d
The simplest form would be to use any of the following lines:
mkdir -p {a..c}/{a..c} # depth 2
mkdir -p {a..c}/{a..c}/{a..c} # depth 3
mkdir -p {a..c}/{a..c}/{a..c}/{a..c} # depth 4
...
The brace-expansion will make all combinations and mkdir -p will take care of the rest.
Of course, you do not want to type this over and over for various sets. So you could generate the full brace-expansion-string with bash and use exec to process the brace-expansion-string before passing it to mkdir -p:
depth=3
set={a..c}
dirs=$(printf "/${set}%.0s" $(seq $depth))
mkdir -p $(eval echo .${dirs})
Be aware however that if your set has length m, and you want a depth of n, you are creating m^n directories. This number could conflict with the number of arguments you can pass on to a program.
Related information:
What is the maximum allowed depth of sub-folders?
https://www.in-ulm.de/~mascheck/various/argmax/
A recursive funcion may solve your problem. Take care with inodes generation when using high directory levels...
#!/bin/bash
function createDir {
mkdir -p $1 && cd $1;
for x in {a..z} ; do
local i=$(($2-1))
[ $i -lt 0 ] && continue;
createDir $x $i
done
cd ..
}
createDir $1 $2
Save into a file, like mkdir.sh, and call it: ./mkdir.sh <main_folder> <level>.

Shell script: Copy file and folder N times

I've two documents:
an .json
an folder with random content
where <transaction> is id+sequancial (id1, id2... idn)
I'd like to populate this structure (.json + folder) to n. I mean:
I'd like to have id1.json and id1 folder, an id2.json and id2 folder... idn.json and idn folder.
Is there anyway (shell script) to populate this content?
It would be something like:
for (i=0,i<n,i++) {
copy "id" file to "id+i" file
copy "id" folder to "id+i" folder
}
Any ideas?
Your shell syntax is off but after that, this should be trivial.
#!/bin/bash
for((i=0;i<$1;i++)); do
cp "id".json "id$i".json
cp -r "id" "id$i"
done
This expects the value of n as the sole argument to the script (which is visible inside the script in $1).
The C-style for((...)) loop is Bash only, and will not work with sh.
A proper production script would also check that it received the expected parameter in the expected format (a single positive number) but you will probably want to tackle such complications when you learn more.
Additionaly, here is a version working with sh:
#!/bin/sh
test -e id.json || { (>&2 echo "id.json not found") ; exit 1 ; }
{
seq 1 "$1" 2> /dev/null ||
(>&2 echo "usage: $0 transaction-count") && exit 1
} |
while read i
do
cp "id".json "id$i".json
cp -r "id" "id$i"
done

batch processing : File name comparison error

I have written a program (Cifti_subject_fmri) which compares whether file name matches in two folders and essentially executes a set of instructions
#!/bin/bash -- fix_mni_paths
source activate ciftify_v1.0.0
export SUBJECTS_DIR=/scratch/m/mchakrav/dev/functional_data
export HCP_DATA=/scratch/m/mchakrav/dev/tCDS_ciftify
## make the $SUBJECTS_DIR if it does not already exist
mkdir -p ${HCP_DATA}
SUBJECTS=`cd $SUBJECTS_DIR; ls -1d *` ## list of my subjects
HCP=`cd $HCP_DATA; ls -1d *` ## List of HCP Subjects
cd $HCP_DATA
## submit the files to the queue
for i in $SUBJECTS;do
for j in $HCP ; do
if [[ $i == $j ]];then
parallel "echo ciftify_subject_fmri $i/filtered_func_data.nii.gz $j fMRI " ::: $SUBJECTS |qbatch --walltime '05:00:00' --ppj 8 -c 4 -j 4 -N ciftify_subject_fmri -
fi
done
done
When i run this code in the cluster i am getting an error which says
./Cifti_subject_fmri: [[AS1: command not found
The query ciftify_subject_fmri is part of toolbox ciftify, for it to execute it requires following instructions
ciftify_subject_fmri <func.nii.gz> <Subject> <NameOffMRI>
I have 33 subjects [AS1 -AS33] each with its own func.nii.gz files located SUBJECTS directory,the results need to be populated in HCP directory, fMRI is name of file format .
Could some one kindly let me know why i am getting an error in loop

Nested for loops in bash

In the directory from where I am running a bash script,
directory saved in the variable:
ScriptDir=`pwd`
I have the following files:
B3LYP_BOTTOM_FRAGMENT
B3LYP-D3_BOTTOM_FRAGMENT
PBE_BOTTOM_FRAGMENT
LDA_BOTTOM_FRAGMENT
PBE-D3_BOTTOM_FRAGMENT
PBE0_BOTTOM_FRAGMENT
PBE0-DC_BOTTOM_FRAGMENT
254.186305_TOP_FRAGMENT.d12
252.050453_TOP_FRAGMENT.d12
249.921810_TOP_FRAGMENT.d12
247.812353_TOP_FRAGMENT.d12
245.699603_TOP_FRAGMENT.d12
243.644688_TOP_FRAGMENT.d12
241.581529_TOP_FRAGMENT.d12
239.554134_TOP_FRAGMENT.d12
237.467646_TOP_FRAGMENT.d12
235.473555_TOP_FRAGMENT.d12
These files can be classified into two different variables: DIRS and FOLDERS
DIRS="
PBE-D3
PBE
B3LYP
B3LYP-D3
PBE0
PBE0-DC
LDA
"
FOLDERS="
237.467646
239.554134
241.581529
243.644688
245.699603
247.812353
249.921810
252.050453
254.186305
235.473555
"
Given this path: /path/to/target, If I loop over DIRS ($i) and FOLDERS ($j), I would like to end up with the following:
ls -lrth /path/to/target/PBE-D3/scaling_volumes/237.467646
237.467646_TOP_FRAGMENT.d12 # j = 1 on FOLDERS
PBE-D3_BOTTOM_FRAGMENT # i = 1 on DIRS
237.467646.d12
# where `237.467646.d12` is the result of doing:
# cat 237.467646_TOP_FRAGMENT.d12 PBE-D3_BOTTOM_FRAGMENT > 237.467646.d12
ls -lrth /path/to/target/PBE-D3/scaling_volumes/239.554134
239.554134_TOP_FRAGMENT.d12 # j = 2 on FOLDERS
PBE-D3_BOTTOM_FRAGMENT # i = 1 on DIRS
239.554134.d12
ls -lrth /path/to/target/PBE-D3/scaling_volumes/241.581529
241.581529_TOP_FRAGMENT.d12 # j = 3 on FOLDERS
PBE-D3_BOTTOM_FRAGMENT # i = 1 on DIRS
241.581529.d12
# and so on...
# In other words, in this iteration, all the `j`th `FOLDERS` for a given `j`th `DIR`
# For the second `DIR`, again the 1st `FOLDER`:
ls -lrth /path/to/target/PBE/scaling_volumes/237.467646
237.467646_TOP_FRAGMENT.d12 # j = 1 on FOLDERS
PBE_BOTTOM_FRAGMENT # i = 2 on DIRS
237.467646.d12
# and so on
I have written the following script:
DIRS="
PBE-D3
PBE
B3LYP
B3LYP-D3
PBE0
PBE0-DC
LDA
"
FOLDERS="
237.467646
239.554134
241.581529
243.644688
245.699603
247.812353
249.921810
252.050453
254.186305
235.473555
"
ScriptDir=`pwd`
for i in ${DIRS}; do
cd /path/to/target/$i
rm -Rf scaling_volumes
mkdir scaling_volumes
cd scaling_volumes
for j in ${FOLDERS}; do
rm -Rf ${j}
mkdir ${j}
cd $ScriptDir
cp -avr ${j}_TOP_FRAGMENT.d12 /path/to/target/$i/scaling_volumes/${j}
cp -avr ${i}_BOTTOM_FRAGMENT /path/to/target/$i/scaling_volumes/${j}
cd /path/to/target/$i/scaling_volumes/${j}
cat ${j}_TOP_FRAGMENT.d12 ${i}_BOTTOM_FRAGMENT > ${j}.d12
cd $ScriptDir
done
done
For some reason what I receive is:
ls -lrth /path/to/target/PBE-D3/scaling_volumes
235.473555 # Only the last FOLDER has been created
or:
ls -lrth /path/to/target/PBE/scaling_volumes
235.473555 # Only the last FOLDER has been created
where only last jth FOLDER was created
Fail fast, add || exit 1 after cd and mkdir commands
Avoid cd use it only when necessary because paths are absolute
Also script directory can be different from pwd (current working directory), for example if script is called from another directory.
Following #choroba 's great advice, I managed to solve the problem by creating a
scaling=`pwd`
variable, and place it just before the for j in ${FOLDERS} loop, and end this loop with
cd $scaling
However, I am very interested in the || exit 1 approach suggested by #Nahuel Fouilleul but I am afraid I do not know where to start with.
ScriptDir=`pwd`
for i in ${DIRS}; do
cd /home/david/Trabajo/structures/Trial_for_double_for_loop_in_bash/pob_TZVP/Calcite_I/$i
rm -Rf scaling_volumes_from_117.743646
mkdir scaling_volumes_from_117.743646
cd scaling_volumes_from_117.743646
scaling=`pwd`
for j in ${FOLDERS}; do
rm -Rf ${j}
mkdir ${j}
cd $ScriptDir
cp -avr ${j}_TOP_FRAGMENT.d12 /home/david/Trabajo/structures/Trial_for_double_for_loop_in_bash/pob_TZVP/Calcite_I/$i/scaling_volumes_from_117.743646/${j}
cp -avr ${i}_BOTTOM_FRAGMENT /home/david/Trabajo/structures/Trial_for_double_for_loop_in_bash/pob_TZVP/Calcite_I/$i/scaling_volumes_from_117.743646/${j}
cd /home/david/Trabajo/structures/Trial_for_double_for_loop_in_bash/pob_TZVP/Calcite_I/$i/scaling_volumes_from_117.743646/${j}
cat ${j}_TOP_FRAGMENT.d12 ${i}_BOTTOM_FRAGMENT > ${j}.d12
cd $scaling
done
done

Resources