In the directory from where I am running a bash script,
directory saved in the variable:
ScriptDir=`pwd`
I have the following files:
B3LYP_BOTTOM_FRAGMENT
B3LYP-D3_BOTTOM_FRAGMENT
PBE_BOTTOM_FRAGMENT
LDA_BOTTOM_FRAGMENT
PBE-D3_BOTTOM_FRAGMENT
PBE0_BOTTOM_FRAGMENT
PBE0-DC_BOTTOM_FRAGMENT
254.186305_TOP_FRAGMENT.d12
252.050453_TOP_FRAGMENT.d12
249.921810_TOP_FRAGMENT.d12
247.812353_TOP_FRAGMENT.d12
245.699603_TOP_FRAGMENT.d12
243.644688_TOP_FRAGMENT.d12
241.581529_TOP_FRAGMENT.d12
239.554134_TOP_FRAGMENT.d12
237.467646_TOP_FRAGMENT.d12
235.473555_TOP_FRAGMENT.d12
These files can be classified into two different variables: DIRS and FOLDERS
DIRS="
PBE-D3
PBE
B3LYP
B3LYP-D3
PBE0
PBE0-DC
LDA
"
FOLDERS="
237.467646
239.554134
241.581529
243.644688
245.699603
247.812353
249.921810
252.050453
254.186305
235.473555
"
Given this path: /path/to/target, If I loop over DIRS ($i) and FOLDERS ($j), I would like to end up with the following:
ls -lrth /path/to/target/PBE-D3/scaling_volumes/237.467646
237.467646_TOP_FRAGMENT.d12 # j = 1 on FOLDERS
PBE-D3_BOTTOM_FRAGMENT # i = 1 on DIRS
237.467646.d12
# where `237.467646.d12` is the result of doing:
# cat 237.467646_TOP_FRAGMENT.d12 PBE-D3_BOTTOM_FRAGMENT > 237.467646.d12
ls -lrth /path/to/target/PBE-D3/scaling_volumes/239.554134
239.554134_TOP_FRAGMENT.d12 # j = 2 on FOLDERS
PBE-D3_BOTTOM_FRAGMENT # i = 1 on DIRS
239.554134.d12
ls -lrth /path/to/target/PBE-D3/scaling_volumes/241.581529
241.581529_TOP_FRAGMENT.d12 # j = 3 on FOLDERS
PBE-D3_BOTTOM_FRAGMENT # i = 1 on DIRS
241.581529.d12
# and so on...
# In other words, in this iteration, all the `j`th `FOLDERS` for a given `j`th `DIR`
# For the second `DIR`, again the 1st `FOLDER`:
ls -lrth /path/to/target/PBE/scaling_volumes/237.467646
237.467646_TOP_FRAGMENT.d12 # j = 1 on FOLDERS
PBE_BOTTOM_FRAGMENT # i = 2 on DIRS
237.467646.d12
# and so on
I have written the following script:
DIRS="
PBE-D3
PBE
B3LYP
B3LYP-D3
PBE0
PBE0-DC
LDA
"
FOLDERS="
237.467646
239.554134
241.581529
243.644688
245.699603
247.812353
249.921810
252.050453
254.186305
235.473555
"
ScriptDir=`pwd`
for i in ${DIRS}; do
cd /path/to/target/$i
rm -Rf scaling_volumes
mkdir scaling_volumes
cd scaling_volumes
for j in ${FOLDERS}; do
rm -Rf ${j}
mkdir ${j}
cd $ScriptDir
cp -avr ${j}_TOP_FRAGMENT.d12 /path/to/target/$i/scaling_volumes/${j}
cp -avr ${i}_BOTTOM_FRAGMENT /path/to/target/$i/scaling_volumes/${j}
cd /path/to/target/$i/scaling_volumes/${j}
cat ${j}_TOP_FRAGMENT.d12 ${i}_BOTTOM_FRAGMENT > ${j}.d12
cd $ScriptDir
done
done
For some reason what I receive is:
ls -lrth /path/to/target/PBE-D3/scaling_volumes
235.473555 # Only the last FOLDER has been created
or:
ls -lrth /path/to/target/PBE/scaling_volumes
235.473555 # Only the last FOLDER has been created
where only last jth FOLDER was created
Fail fast, add || exit 1 after cd and mkdir commands
Avoid cd use it only when necessary because paths are absolute
Also script directory can be different from pwd (current working directory), for example if script is called from another directory.
Following #choroba 's great advice, I managed to solve the problem by creating a
scaling=`pwd`
variable, and place it just before the for j in ${FOLDERS} loop, and end this loop with
cd $scaling
However, I am very interested in the || exit 1 approach suggested by #Nahuel Fouilleul but I am afraid I do not know where to start with.
ScriptDir=`pwd`
for i in ${DIRS}; do
cd /home/david/Trabajo/structures/Trial_for_double_for_loop_in_bash/pob_TZVP/Calcite_I/$i
rm -Rf scaling_volumes_from_117.743646
mkdir scaling_volumes_from_117.743646
cd scaling_volumes_from_117.743646
scaling=`pwd`
for j in ${FOLDERS}; do
rm -Rf ${j}
mkdir ${j}
cd $ScriptDir
cp -avr ${j}_TOP_FRAGMENT.d12 /home/david/Trabajo/structures/Trial_for_double_for_loop_in_bash/pob_TZVP/Calcite_I/$i/scaling_volumes_from_117.743646/${j}
cp -avr ${i}_BOTTOM_FRAGMENT /home/david/Trabajo/structures/Trial_for_double_for_loop_in_bash/pob_TZVP/Calcite_I/$i/scaling_volumes_from_117.743646/${j}
cd /home/david/Trabajo/structures/Trial_for_double_for_loop_in_bash/pob_TZVP/Calcite_I/$i/scaling_volumes_from_117.743646/${j}
cat ${j}_TOP_FRAGMENT.d12 ${i}_BOTTOM_FRAGMENT > ${j}.d12
cd $scaling
done
done
Related
I have a script that submits processing jobs to a queue. Before I submit the jobs, I assign the string variables to each respective data point so I can use them as the arguments before I submit the jobs through qsub.
I had to fix up the module I'm loading first by putting in a -v variable to set up my working environment. I got the error message that is in the title however, and looking around there is very limited resources to debugging it. One resource I found seems to have led me in the direction of the potential likelihood of an extraneous space in the qsub command itself. Has anyone run into this?
I also did echo on my qsub command to make sure it was being inputted correctly, as it was.
Here's my script:
#!/bin/bash
# This script is for submitting the initial registration subjects for Greedy registration.
# It can serve as a template for later studies when multiple submissions could be handy
# GO_HOME = Origin diqrectory for all niftis of interest
GO_NIFTI="/gpfs/fs001/medorg/comp_space/myname/Test-Retest/Nifti/"
GO_B0="/gpfs/fs001/medorg/comp_space/myname/Test-Retest/Protocols/ants_SyNBaseline/W_Registration_antsSyN_Baseline7/"
GO_FM="/gpfs/fs001/medorg/comp_space/myname/Test-Retest/Protocols/brainmage_batch_t1/"
FINAL_DESTINATION="/gpfs/fs001/cbica/comp_space/wingerti/Test-Retest/Protocols/Registration_greedy_Rigid/"
cd $GO_NIFTI
nii_directories=($(find . -type d -name "*t1*" -o -name "*t0*" -o -name "*t2*" -maxdepth 1 ))
module load greedy
# Will look at these subjects individually, taking them out list to not run DTI_Preprocess
unset nii_directories[27] # 1000009_t0_test
unset nii_directories[17] # 1000001_t0
unset nii_directories[4] # 1000009_t2
# With directories, navigate into each, and find where the suitable niis are (31dir and 33dir)
for g in "${nii_directories[#]}";
do
# Subject ID argument
subjid=${g:2:9}
echo "$subjid is the subject ID..."
# -i argument (T1 NIFTI File and DTI)
cd $GO_NIFTI
nii_is=$(find $subjid -type f -name ${subjid}_T1.nii.gz)
nii_i=${GO_NIFTI}${nii_is}
cd $GO_B0
cd $subjid
GO_B0_2=$PWD
b0_is=$(find . -type f -name b0.nii.gz)
b0_i=${GO_B0_2}${b0_is}
echo "-i arguments for $subjid is $nii_i and $b0_i"
# -m argument (Mask File)
#cd $GO_DTI
#mask_ms=$(find $subjid -type f -name ${subjid}_tensor_mask.nii.gz)
#mask_m=${GO_DTI}${mask_ms}
#echo "-m argument for $subjid is $mask_m"
# -fm argument (T1 mask)
cd $GO_FM
mask_fms=$(find $subjid -type f -name ${subjid}_t1_brain_mask.nii.gz)
mask_fm=${GO_FM}${mask_fms}
echo "-fm argument for $subjid is $mask_fm"
# -o argument (Working Directory for possible debugging and tmp dir organization among experiments)
cd $FINAL_DESTINATION
g=${FINAL_DESTINATION:73:-1}
experiment_name="${subjid}_${g}"
mkdir $experiment_name
output_o=${FINAL_DESTINATION}${experiment_name}/${experiment_name}_rigid.txt
echo "-o argument for $g is $output_o"
#
printf "\nSubmitting the following command: \n
qsub -m beas -M myname#medschool.edu -N Registration_${experiment_name} "$(which greedy)" -d3 -i $nii_i $b0_i -o $output_o -a -m MI -n 100x100 -fm $mask_fm dof 6 -ia-identity\n
as JobID: Registration_${experiment_name}\n\n"
qsub -v /medorg/software/external/greedy/centos7/c6dca2e -m beas -M myname#medschool.edu -N Registration_${experiment_name} "$(which greedy)" -d3 -i $nii_i $b0_i -o $output_o -a -m MI -n 100x100 -fm $mask_fm dof 6 -ia-identity
# --- Above line submits Greedy Rigid jobs (dof 6) with
# --- "-m" for emailing updates on jobs, inbox sorts job submission emails
# --- "-N" names the job for book-keeping
cd $GO_NIFTI
done
I am having trouble creating the following directory structure. It is to test a scanner program to see if it can handle it.
Start inside /data/
Inside /data/ there are 2 sub-directories. /data/sub1 and /data/sub2
Now, this gets complex :)
In each of the sub-directories there are 2 more sub-directories. For first sub: /data/sub1/sub1/ and /data/sub1/sub2 and for second sub /data/sub2/sub1/ and /data/sub2/sub2
... this goes on to a depth of 16 levels, with a total directory of 65,536.
What I have so far, this only creates 1 subdirectory inside /data/, 16 folders deep. I am thinking of using the following script recursively for the subdirectories created along the way. And for the subdirectories it then creates. And so on. But I can't imagine scripting it.
count=0;
folder=$(printf "Folder%02dB" $count);
mkdir -p $folder;
while [ $count -lt 16 ]; do
((count=$count+1));
folder=$(printf "Folder%02dB" $count);
mkdir -p $folder;
cd $folder;
done
Anyone has any ideas on how to create such directory structure? How to proceed? Thanks!
Recursion is a good way to do this.
#! /bin/bash
godeep()
{
if [ $1 -lt 16 ]
then
((level = $1 + 1))
mkdir sub1
cd sub1
godeep $level
cd ..
mkdir sub2
cd sub2
godeep $level
cd ..
fi
}
godeep 1
~
You shouldn't be incorporating $count in the folder name. That's just used to count the recursion depth.
Simplest would be to define a function to create a single level, and call it recursively.
create_folders() {
if (( $1 > 16 )); then
return
fi
depth=$(($1 + 1))
mkdir sub1; (cd sub1; create_folders $depth)
mkdir sub2; (cd sub2; create_folders $depth)
}
create_folders 1
Using () around the cd and recursive call keeps them from affecting the working directory and variables in the current call.
I'm trying to create files with the name of it's antepenultimate directory:
Example:
Directory: a/b/c/d/e/f/g/h/i/j
The name of folder h is different for each case.
So I created an array
array=(/ a / b / c / d / e / f / g / * / * / *)
len=${#array[#]}
for (( q=0; q<$len; q++ ));
do
cd ${array[$q]}
sleep 1
mri_convert 0001*.dcm RAW.nii.gz #--->this line is just converting the format of file 0001*.dcm in to file RAW.nii.gz
done
This code is working but I want the file RAW.nii.gz to be named h_RAW.nii.gz
I tried doing this:
s1="${array%/*/*}"
$ echo "${s1##*/}"
and then:
mri_convert 0001*.dcm ${s1##*/}_RAW.nii.gz
but it's not working.
How about
cd /a/b/c/d/e/f/g
for dir in *; do
[[ -d $dir ]] || continue
for subdir in "$dir"/*/*/; do (
# doing this in a subshell so we don't need to "undo" this cd
cd "$subdir"
mri_convert 0001*.dcm "${dir}_RAW.nii.gz"
); done
done
Let's see if I can help. I'm not exactly sure of the details of what you're trying to do (mainly because the code you posted:
for (( q=0; q do cd ${array[$q]} sleep 1 mri_convert 0001*.dcm RAW.nii.gz
is not syntactically correct. So, that can't be what you're actually doing.
Just a hint of how i would approach a problem like this:
for path6 in /a/b/c/*/*/*
do
path5="${path6##*/}"
path4="${path5##*/}"
name4="${path4%/*}"
echo "Processing ${path4}"
mriconvert "${path6}"/0001*.dcm "${path6}/${name4}_RAW.nii.gz"
done
I would like to create a folder structure based on a brace expansion such as {a-z}. Each string generated by the brace-expansion should be a new folder. Furthermore, each of these folders should contain the same set of subfolders similarly generated. And this up to a given level.
An example for the range a-z and depth 16
a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/
a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/b/
a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/c/
...
d/a/h/r/y/d/s/b/e/y/k/f/o/o/q/c/
...
z/z/z/z/z/z/z/z/z/z/z/z/z/z/z/y/
z/z/z/z/z/z/z/z/z/z/z/z/z/z/z/z/
The following code allows me to go upto depth 2:
for x in {a..z} ; do mkdir -p $x/{a..z} ; done
But how do I go further?
A recursive solution. Job is called with 2 params: max_depth and base_path
#!/bin/bash
function job()
{
local depth=$(($1-1))
local path=$2
local x
for x in a b c # reduced for test
do
mkdir -p "$path/$x"
((depth>0)) && job $depth "$path/$x"
done
}
job 3 ./test
Proof it with:
find test -type d
The simplest form would be to use any of the following lines:
mkdir -p {a..c}/{a..c} # depth 2
mkdir -p {a..c}/{a..c}/{a..c} # depth 3
mkdir -p {a..c}/{a..c}/{a..c}/{a..c} # depth 4
...
The brace-expansion will make all combinations and mkdir -p will take care of the rest.
Of course, you do not want to type this over and over for various sets. So you could generate the full brace-expansion-string with bash and use exec to process the brace-expansion-string before passing it to mkdir -p:
depth=3
set={a..c}
dirs=$(printf "/${set}%.0s" $(seq $depth))
mkdir -p $(eval echo .${dirs})
Be aware however that if your set has length m, and you want a depth of n, you are creating m^n directories. This number could conflict with the number of arguments you can pass on to a program.
Related information:
What is the maximum allowed depth of sub-folders?
https://www.in-ulm.de/~mascheck/various/argmax/
A recursive funcion may solve your problem. Take care with inodes generation when using high directory levels...
#!/bin/bash
function createDir {
mkdir -p $1 && cd $1;
for x in {a..z} ; do
local i=$(($2-1))
[ $i -lt 0 ] && continue;
createDir $x $i
done
cd ..
}
createDir $1 $2
Save into a file, like mkdir.sh, and call it: ./mkdir.sh <main_folder> <level>.
I have this code which scans folders and moves all files in each folder to a new one.
How do I make it so only every Nth file is moved?
#!/bin/bash
# Save this file in the directory containing the folders (bb in this case)
# Then to run it, type:
# ./rencp.sh
# The first output frame number
let "frame=1"
# this is where files will go. A new directory will be created if it doesn't exist
outFolder="collected"
# print info every so many files.
feedbackFreq=250
# prefix for new files
namePrefix="ben_timelapse"
#new extension (uppercase is so ugly)
ext="jpg"
# this will make sure we only get files from camera directories
srcPattern="ND850"
mkdir -p $outFolder
for f in *${srcPattern}/*
do
mv $f `printf "$outFolder/$namePrefix.%05d.$ext" $frame`
if ! ((frame % $feedbackFreq)); then
echo "moved and renamed $frame files to $outFolder"
fi
let "frame++"
done
Pretty sure I need to edit the line for f in *${srcPattern}/* but not sure of the correct syntax
If files in the ND850 folders are sequential when listed (i.e. padded frame numbers), and the folders themselves are in order, then the following code should work.
#!/bin/bash
# Maintain a counter, and the output frame number
let "frame=1"
let "outframe=1"
outFolder="collected"
# frequency
gap=5
namePrefix="ben_timelapse"
#new extension (uppercase is so ugly)
ext="jpg"
srcPattern="ND850"
echo "Copying and renaming 1 in every $gap files"
mkdir -p "$outFolder"
for f in *${srcPattern}/*
do
if ! ((frame % $gap)); then
outfile=`printf "$outFolder/$namePrefix.%05d.$ext" $outframe`
cp $f "$outfile"
echo "copied $f to $outfile"
let "outframe++"
fi
let "frame++"
done
Try this instead of your mv command after do:
if ! ((frame % 5)); then
a=$((frame / 5));
mv $f `printf "$outFolder/$namePrefix.%05d.$ext" $a`
fi
It will move frame=5,10, and so on, to $outFolder/$namePrefix.00001.$ext,$outFolder/$namePrefix.00002.$ext, and so on