I want to loop through a list of directories (a subset of the directories in a folder) and do operations with them. However, for some reason it is not working. Here is the code where I am just echoing all of them:
#!/bin/bash
cd images
array=$(ls -d *)
selection=(${array[#]:1:12})
cd ..
for sub in ${selection[#]}
do
echo $sub
mkdir $HOME/Projects/PhD/0_project/fMRI/scans/temp/temp_$sub
done
The ouptut I get for the echo command is:
04
306
307
3
And the folders: temp_3, temp_04, temp_306, temp_307
HOWEVER, if I run each single line in bash in the termiinal (interactive mode, no script) I do get the correct output for the echo command:
306
307
308
309
310
311
312
314
317
318
323
324
And for the mkdir command: temp_306, temp_307... temp_324
Any idea about this strange and inconsistent behaviour? What am I missing?
Thanks for your help.
result of $(ls -d *) is a string.
you are slicing a string not an array.
remove :1:12.
#!/bin/bash
cd images || exit
array=(*/) # result is array of subdirectories with / on end of each
selection=("${array[#]%/}") # remove trailing slashes from that array.
selection_sliced=("${selection[#]:1:12}") # if you want to slice.
cd .. || exit
for sub in "${selection_sliced[#]}"
do
echo "$sub"
mkdir "$HOME/Projects/PhD/0_project/fMRI/scans/temp/temp_$sub"
done
Related
I am having trouble creating the following directory structure. It is to test a scanner program to see if it can handle it.
Start inside /data/
Inside /data/ there are 2 sub-directories. /data/sub1 and /data/sub2
Now, this gets complex :)
In each of the sub-directories there are 2 more sub-directories. For first sub: /data/sub1/sub1/ and /data/sub1/sub2 and for second sub /data/sub2/sub1/ and /data/sub2/sub2
... this goes on to a depth of 16 levels, with a total directory of 65,536.
What I have so far, this only creates 1 subdirectory inside /data/, 16 folders deep. I am thinking of using the following script recursively for the subdirectories created along the way. And for the subdirectories it then creates. And so on. But I can't imagine scripting it.
count=0;
folder=$(printf "Folder%02dB" $count);
mkdir -p $folder;
while [ $count -lt 16 ]; do
((count=$count+1));
folder=$(printf "Folder%02dB" $count);
mkdir -p $folder;
cd $folder;
done
Anyone has any ideas on how to create such directory structure? How to proceed? Thanks!
Recursion is a good way to do this.
#! /bin/bash
godeep()
{
if [ $1 -lt 16 ]
then
((level = $1 + 1))
mkdir sub1
cd sub1
godeep $level
cd ..
mkdir sub2
cd sub2
godeep $level
cd ..
fi
}
godeep 1
~
You shouldn't be incorporating $count in the folder name. That's just used to count the recursion depth.
Simplest would be to define a function to create a single level, and call it recursively.
create_folders() {
if (( $1 > 16 )); then
return
fi
depth=$(($1 + 1))
mkdir sub1; (cd sub1; create_folders $depth)
mkdir sub2; (cd sub2; create_folders $depth)
}
create_folders 1
Using () around the cd and recursive call keeps them from affecting the working directory and variables in the current call.
I have to write a simple bash script, which does the compilation of multiple sql scripts which can contain recursive references to the other sql scripts with Oracle SQLPlus convention, so in the end the result will be only one SQL file, containing all statements from all (possibly also recursively) referenced subscripts.
I have created a simple script called oracle-script-compressor.sh with following bash code:
#!/bin/bash
#
#
function err () {
echo $* > "/dev/stderr"
}
function replace_subscripts_placeholders() {
local PROCESSING_FILENAME=$1
echo "-- processing file $PROCESSING_FILENAME"
while read script_line; do
local script_line_NO_LEAD_SPACE="$(echo -e "${script_line}" | sed -e 's/^[[:space:]]*//')"
if [[ $script_line_NO_LEAD_SPACE == \#\#* ]] ; then
local file_name="./${script_line_NO_LEAD_SPACE#\#\#}"
echo "-- found reference to file $file_name"
echo "-- starting with processing, pwd=$PWD"
# for debug purposes:
ls -la
# but this returns always false:
if [ -f "$file_name" ]; then
# so this part is never started:
. replace_subscripts_placeholders $file_name
else
err_msg="WARNING: Could not find the referenced file $file_name"
echo "-- $err_msg"
err $err_msg
fi
else
echo "$script_line"
fi
done < $PROCESSING_FILENAME
}
if test -z "$1" ; then
err "Usage: oracle-script-compressor.sh {file_name_to_process} [> output_file]"
err " Be aware, if the referenced files within {file_name_to_process} are not specified by absolute paths, you have to start the script from corresponding directory."
err " If the part [> output_file] is omitted, then this script writes to standard output"
err " If the part [>> output_file] is used, then the result will be appended to output_file it it exists before the processing"
else
if [ -f "$1" ]; then
replace_subscripts_placeholders $1
else
echo "file $1 does not exist"
fi
fi
and I am stuck on interesting problem - it seems, inside of the function replace_subscripts_placeholders() the file check
if [ -f "$file_name" ]; then
. replace_subscripts_placeholders $file_name
else
err_msg="WARNING: Could not find the referenced file $file_name"
echo "-- $err_msg"
err $err_msg
fi
never goes to the recursion call and even, if I remove the if statement and really call the function in recursion with the correct referenced file name, which exists, it is still not recognized to be found and not passed into the loop over all lines of the referenced file in the recursive call (then the error file not found comes and the loop cannot be executed).
I have added the debug messages into script like mentioned above in the script, but still I am unable to find, why the hell should bash not find the file, if it is in the same directory. The scripts are placed in
user#mycomputer:/tmp/test$ ls -la
celkem 52
drwxr-xr-x 2 user user 4096 zář 14 21:47 .
drwxrwxrwt 38 root root 36864 zář 14 21:48 ..
-rw-r--r-- 1 user user 51 zář 14 21:45 a.sql
-rw-r--r-- 1 user user 51 zář 14 21:45 b.sql
-rw-r--r-- 1 user user 590 zář 14 21:46 start.sql
user#mycomputer:/tmp/test$
and the content of the file start.sql looks like this:
spool output__UpdSch.log
whenever sqlerror exit sql.sqlcode
--
--
PROMPT a.sql - starting
select to_char(current_timestamp,'YYYY-MM-DD HH24:MI:SS.FF3') as START_TIMESTAMP from dual;
##a.sql
PROMPT a.sql - finished
select to_char(current_timestamp,'YYYY-MM-DD HH24:MI:SS.FF3') as FINISH_TIMESTAMP from dual;
--
--
PROMPT b.sql - starting
select to_char(current_timestamp,'YYYY-MM-DD HH24:MI:SS.FF3') as START_TIMESTAMP from dual;
##b.sql
PROMPT b.sql - finished
select to_char(current_timestamp,'YYYY-MM-DD HH24:MI:SS.FF3') as FINISH_TIMESTAMP from dual;
--
--
spool off
and if I execute the script, it seems it decoded the filenames correctly, but there is still the problem in bash - the testing of the file existence returns always false:
user#mycomputer:/tmp/test$ ~/tmp/oracle-script-compressor.sh start.sql
-- processing file start.sql
spool output__UpdSch.log
whenever sqlerror exit sql.sqlcode
--
--
PROMPT a.sql - starting
select to_char(current_timestamp,'YYYY-MM-DD HH24:MI:SS.FF3') as START_TIMESTAMP from dual;
-- found reference to file ./a.sql
-- starting with processing, pwd=/tmp/test
celkem 52
drwxr-xr-x 2 user user 4096 zář 14 21:47 .
drwxrwxrwt 38 root root 36864 zář 14 21:48 ..
-rw-r--r-- 1 user user 51 zář 14 21:45 a.sql
-rw-r--r-- 1 user user 51 zář 14 21:45 b.sql
-rw-r--r-- 1 user user 590 zář 14 21:46 start.sql
-- WARNING: Could not find the referenced file ./a.sql
WARNING: Could not find the referenced file ./a.sql
PROMPT a.sql - finished
select to_char(current_timestamp,'YYYY-MM-DD HH24:MI:SS.FF3') as FINISH_TIMESTAMP from dual;
--
--
PROMPT b.sql - starting
select to_char(current_timestamp,'YYYY-MM-DD HH24:MI:SS.FF3') as START_TIMESTAMP from dual;
-- found reference to file ./b.sql
-- starting with processing, pwd=/tmp/test
celkem 52
drwxr-xr-x 2 user user 4096 zář 14 21:47 .
drwxrwxrwt 38 root root 36864 zář 14 21:48 ..
-rw-r--r-- 1 user user 51 zář 14 21:45 a.sql
-rw-r--r-- 1 user user 51 zář 14 21:45 b.sql
-rw-r--r-- 1 user user 590 zář 14 21:46 start.sql
-- WARNING: Could not find the referenced file ./b.sql
WARNING: Could not find the referenced file ./b.sql
PROMPT b.sql - finished
select to_char(current_timestamp,'YYYY-MM-DD HH24:MI:SS.FF3') as FINISH_TIMESTAMP from dual;
--
--
spool off
user#mycomputer:/tmp/test$
The script has read access to files, everything is in the same folder, where I start the script, It also does not matter, if I reference the file with the current directory prefix "./" or not, it is just never found. Interesting is, the check during the start of the script passes correctly, the only problem is the searching within the function... I have also tried to call the function with preceded "." and without it, it makes no difference... There are also no trailing spaces in the referenced file names... It also makes no difference if I declare the variables inside of the function as local or not.
So I have no idea, what could be the problem, maybe I just looked too long on it and cannot see something - seems it has to be something trivial, like the function is started in some different current directory - (but pwd and ls shows the directory is always correct...???) - Any help or pointers will be appreciated.
Thank you for the comments, it brought me to the solution. I found out, this problem was related to the fact, the test sql files were created on Windows, so they had in the end of line always the <CR><LF> characters. This leads to the fact in the original bash script I posted, the script line
while read script_line
puts into the variable not only the given file name from the line, preceded by the characters ##
##a.sql
but also the <CR> character - in the variable is then the value
##a.sql<CR>
what was the cause the file a.sql etc. could never be found. Of course the character is invisible, so therefore it has not been shown on any debug echoing I made here - I had to put the content of $file_name between some another characters and then i could see it in the echoed test... I made also some other corrections and the final working script looks in the following way, if somebody needs to join referenced SQL scripts into one, this does it:
#!/bin/bash
#
#
function err () {
echo $* > "/dev/stderr"
}
function replace_subscripts_placeholders() {
local PROCESSING_FILENAME=$1
echo "-- processing file $PROCESSING_FILENAME"
while read script_line || [ -n "$script_line" ]; do
local script_line_NO_LEAD_SPACE="$(echo -e "${script_line}" | sed -e 's/^[[:space:]]*//' | sed -e 's/^M$//')"
if [[ $script_line_NO_LEAD_SPACE == \#\#* ]] ; then
local file_name="${script_line_NO_LEAD_SPACE#\#\#}"
echo "-- found reference to file $file_name"
if [ -f "$file_name" ]; then
replace_subscripts_placeholders $file_name
else
err_msg="WARNING: Could not find the referenced file $file_name"
echo "-- $err_msg"
err $err_msg
fi
else
echo "$script_line"
fi
done < $PROCESSING_FILENAME
}
if test -z "$1" ; then
err "Usage: oracle-script-compressor.sh {file_name_to_process} [> output_file]"
err " Be aware, if the referenced files within {file_name_to_process} are not specified by absolute paths, you have to start the script from corresponding directory."
err " If the part [> output_file] is omitted, then this script writes to standard output"
err " If the part [>> output_file] is used, then the result will be appended to output_file it it exists before the processing"
else
if [ -f "$1" ]; then
replace_subscripts_placeholders $1
else
echo "file $1 does not exist"
fi
fi
I've two documents:
an .json
an folder with random content
where <transaction> is id+sequancial (id1, id2... idn)
I'd like to populate this structure (.json + folder) to n. I mean:
I'd like to have id1.json and id1 folder, an id2.json and id2 folder... idn.json and idn folder.
Is there anyway (shell script) to populate this content?
It would be something like:
for (i=0,i<n,i++) {
copy "id" file to "id+i" file
copy "id" folder to "id+i" folder
}
Any ideas?
Your shell syntax is off but after that, this should be trivial.
#!/bin/bash
for((i=0;i<$1;i++)); do
cp "id".json "id$i".json
cp -r "id" "id$i"
done
This expects the value of n as the sole argument to the script (which is visible inside the script in $1).
The C-style for((...)) loop is Bash only, and will not work with sh.
A proper production script would also check that it received the expected parameter in the expected format (a single positive number) but you will probably want to tackle such complications when you learn more.
Additionaly, here is a version working with sh:
#!/bin/sh
test -e id.json || { (>&2 echo "id.json not found") ; exit 1 ; }
{
seq 1 "$1" 2> /dev/null ||
(>&2 echo "usage: $0 transaction-count") && exit 1
} |
while read i
do
cp "id".json "id$i".json
cp -r "id" "id$i"
done
I have written a program (Cifti_subject_fmri) which compares whether file name matches in two folders and essentially executes a set of instructions
#!/bin/bash -- fix_mni_paths
source activate ciftify_v1.0.0
export SUBJECTS_DIR=/scratch/m/mchakrav/dev/functional_data
export HCP_DATA=/scratch/m/mchakrav/dev/tCDS_ciftify
## make the $SUBJECTS_DIR if it does not already exist
mkdir -p ${HCP_DATA}
SUBJECTS=`cd $SUBJECTS_DIR; ls -1d *` ## list of my subjects
HCP=`cd $HCP_DATA; ls -1d *` ## List of HCP Subjects
cd $HCP_DATA
## submit the files to the queue
for i in $SUBJECTS;do
for j in $HCP ; do
if [[ $i == $j ]];then
parallel "echo ciftify_subject_fmri $i/filtered_func_data.nii.gz $j fMRI " ::: $SUBJECTS |qbatch --walltime '05:00:00' --ppj 8 -c 4 -j 4 -N ciftify_subject_fmri -
fi
done
done
When i run this code in the cluster i am getting an error which says
./Cifti_subject_fmri: [[AS1: command not found
The query ciftify_subject_fmri is part of toolbox ciftify, for it to execute it requires following instructions
ciftify_subject_fmri <func.nii.gz> <Subject> <NameOffMRI>
I have 33 subjects [AS1 -AS33] each with its own func.nii.gz files located SUBJECTS directory,the results need to be populated in HCP directory, fMRI is name of file format .
Could some one kindly let me know why i am getting an error in loop
I would like to build a shell script to automatically archive object files into a static library, and copy all the headers and .a file to desired directory.
However, as the number of obj files grows, the following mess gets worse:
8 CWD=$(pwd)
9
10 FILE1="SDL_Logger.o"
11 HEADER1="SDL_Logger.h"
12 FILE2="SDL_Initializer.o"
13 HEADER2="SDL_Logger.h"
14 ARC="libsdlhelper.a"
15
16 INCLUDE=~/include
17 LIB=~/lib
18
19 if [ ! -f $FILE1 ];
20 then
21 echo " error: file $FILE1 does not exist. Abort."
22 else
23 if [ ! -f $FILE2 ];
24 then
25 echo " error: file $FILE2 does not exist. Abort."
26 else
27 echo " building archive... "
28 ar rs $ARC $FILE1 $FILE2
29
31 # lib
32 cp $ARC $LIB
34
35 # include
36 cp $HEADER1 $INCLUDE
37 cp $HEADER2 $INCLUDE
39
41 fi
42 fi
So, if I were to group all files into ONE variable like:
FILE="obj1.o obj2.o ... "
How would I check the existence of each file, and copy them(headers)? I can only do this one by one, which will be soon unacceptable.
Here is what I came up with to help you.
The best way to do this is using an array for FILE
short example to implement what you are trying to do:
# !/bin/bash
#define an array of files to look for
FILES=("SDL_Logger.h" "SDL_Initializer.h" "libsdlhelpere.o")
#define your directories
INCLUDE=~/include
LIB=~/lib
#set the amount of entries in our array
SIZE=${#FILES[#]}
# use for loop read all FILES
for (( i=0; i<${SIZE}; i++ ));
do
echo "checking for file: "${FILES[$i]}
if [ ! -f ${FILES[$i]} ];
then
echo " error: file"${FILES[$i]}" does not exist. Abort."
else
echo " building archive... "
fi
done
Here we define FILES as our array
FILES=("SDL_Logger.h" "SDL_Initializer.h" "libsdlhelpere.a")
then we Find the size of files
SIZE=${#FILES[#]}
Then we go through each file and execute a command which is echo and then our if statement:
for (( i=0; i<${SIZE}; i++ ));
do
Here is how we call our item in the array...
${FILES[$i]}
so in what we are doing in plain english is:
echo FILES[1]....
IF FILES[1] doesn't exist then error....
else execute building archive...
echo FILES[2]....
IF FILES[2] doesn't exist then error...
etc...
This will repeat until the SIZE of FILES in met.
Using this method you can also just define the beginning part of the file name such as
"SDL_Initializer" and have your loop add in the .o .h. .a etc... Maybe another array FILETYPES ;)
hope this helped...
Sorry I didn't complete the code for you :)
Here is an answer to the question as asked:
#!/bin/bash
FILES="SDL_Logger.o SDL_Initializer.o"
HEADERS="${FILES//.o/.h}"
ARC="libsdlhelper.a"
INCLUDE=~/include
LIB=~/lib
for f in ${FILES}; do
if [ ! -f "${f}" ]; then
echo "error: file ${f} does not exist. Abort."
exit 1
fi
done
ar rs ${ARC} ${FILES}
cp ${ARC} ${LIB}
cp ${HEADERS} ${INCLUDE}
However, as the comments point out, the right tool for this job is make. For example GNU make. If you plan on developing software on a unix platform, investing time learning make is time well spent.