I have written a program (Cifti_subject_fmri) which compares whether file name matches in two folders and essentially executes a set of instructions
#!/bin/bash -- fix_mni_paths
source activate ciftify_v1.0.0
export SUBJECTS_DIR=/scratch/m/mchakrav/dev/functional_data
export HCP_DATA=/scratch/m/mchakrav/dev/tCDS_ciftify
## make the $SUBJECTS_DIR if it does not already exist
mkdir -p ${HCP_DATA}
SUBJECTS=`cd $SUBJECTS_DIR; ls -1d *` ## list of my subjects
HCP=`cd $HCP_DATA; ls -1d *` ## List of HCP Subjects
cd $HCP_DATA
## submit the files to the queue
for i in $SUBJECTS;do
for j in $HCP ; do
if [[ $i == $j ]];then
parallel "echo ciftify_subject_fmri $i/filtered_func_data.nii.gz $j fMRI " ::: $SUBJECTS |qbatch --walltime '05:00:00' --ppj 8 -c 4 -j 4 -N ciftify_subject_fmri -
fi
done
done
When i run this code in the cluster i am getting an error which says
./Cifti_subject_fmri: [[AS1: command not found
The query ciftify_subject_fmri is part of toolbox ciftify, for it to execute it requires following instructions
ciftify_subject_fmri <func.nii.gz> <Subject> <NameOffMRI>
I have 33 subjects [AS1 -AS33] each with its own func.nii.gz files located SUBJECTS directory,the results need to be populated in HCP directory, fMRI is name of file format .
Could some one kindly let me know why i am getting an error in loop
Related
This question already has answers here:
Extract filename and extension in Bash
(38 answers)
Closed 1 year ago.
I'm trying to create a bash script that will move all files recursively from a source folder to a target folder, and simply rename files if they already exist. Similar to the way M$ Windows does, when a file exists it auto-renames it with "<filemame> (X).<ext>", etc. except for ALL files.
I've create the below, which works fine for almost all scenarios except when a folder has a (.) period in its name and a file within that folder has no extension (no period in its name).
eg a folder-path-file such as: "./oldfolder/this.folder/filenamewithoutextension"
I get (incorrectly):
"./newfolder/this (1).folder/filenamewithoutextension"
if "./newfolder/this.folder/filenamewithoutextension" already exist in the target location (./newfolder),
instead of correctly naming the new file: "./oldfolder/this.folder/filenamewithoutextension (1)"
#!/bin/bash
source=$1 ; target=$2 ;
if [ "$source" != "" ] && [ "$target" != "" ] ; then
#recursive file search
find "$source" -type f -exec bash -c '
#setup variables
oldfile="$1" ; osource='"${source}"' ; otarget='"${target}"' ;
#set new target filename with target path
newfile="${oldfile/${osource}/${otarget}}" ;
#check if file already exists at target
[ -f "${newfile}" ] && {
#get the filename and fileextension for numbering - ISSUE HERE?
filename="${newfile%/}" ; newfileext="${newfile##*.}" ;
#compare filename and file extension for missing extension
if [ "$filename" == "$newfileext" ] ; then
#filename has no ext - perhaps fix the folder with a period issue here?
newfileext="" ;
else
newfileext=".$newfileext" ;
fi
#existing files counter
cnt=1 ; while [ -f "${newfile%.*} (${cnt})${newfileext}" ] ; do ((cnt+=1)); done
#set new filename with counter - New Name created here *** Needs re-work, as folder with a period = fail
newfile="${newfile%.*} (${cnt})${newfileext}";
}
#show mv command
echo "mv \"$oldfile\" \"${newfile}\""
' _ {} \;
else
echo "Requires source and target folders";
fi
I suspect the issue is, how to properly identify the filename and extension, found in this line:
filename="${newfile%/}" ; newfileext="${newfile##*.}" which doesn't identify a filename properly (files are always after the last /).
Any suggestion on how to make it work properly?
UPDATED: Just some completion notes - Issues fixes with:
Initially Splitting each full path filename: path - filename - (optional ext)
Reconstructing the full path filename: path - filename - counter - (optional ext)
fixed the file move to ensure directory structure exists with mkdir -p (mv does not create new folders if they do not exist in the target location).
Maybe you could try this instead?
filename="${newfile##*/}" ; newfileext="${filename#*.}"
The first pattern means: remove the longest prefix (in a greedy way) up to the last /.
The second one: remove the prefix up to the first dot (the greedy mode seems unnecessary here) − and as you already noted, in case the filename contains no dot, you will get newfileext == filename…
Example session:
newfile='./oldfolder/this.folder/filenamewithoutextension'
filename="${newfile##*/}"; newfileext="${filename#*.}"
printf "%s\n" "$filename"
#→ filenamewithoutextension
printf "%s\n" "$newfileext"
#→ filenamewithoutextension
newfile='./oldfolder/this.folder/file.tar.gz'
filename="${newfile##*/}"; newfileext="${filename#*.}"
printf "%s\n" "$filename"
#→ file.tar.gz
printf "%s\n" "$newfileext"
#→ tar.gz
I would like to create a folder structure based on a brace expansion such as {a-z}. Each string generated by the brace-expansion should be a new folder. Furthermore, each of these folders should contain the same set of subfolders similarly generated. And this up to a given level.
An example for the range a-z and depth 16
a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/
a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/b/
a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/c/
...
d/a/h/r/y/d/s/b/e/y/k/f/o/o/q/c/
...
z/z/z/z/z/z/z/z/z/z/z/z/z/z/z/y/
z/z/z/z/z/z/z/z/z/z/z/z/z/z/z/z/
The following code allows me to go upto depth 2:
for x in {a..z} ; do mkdir -p $x/{a..z} ; done
But how do I go further?
A recursive solution. Job is called with 2 params: max_depth and base_path
#!/bin/bash
function job()
{
local depth=$(($1-1))
local path=$2
local x
for x in a b c # reduced for test
do
mkdir -p "$path/$x"
((depth>0)) && job $depth "$path/$x"
done
}
job 3 ./test
Proof it with:
find test -type d
The simplest form would be to use any of the following lines:
mkdir -p {a..c}/{a..c} # depth 2
mkdir -p {a..c}/{a..c}/{a..c} # depth 3
mkdir -p {a..c}/{a..c}/{a..c}/{a..c} # depth 4
...
The brace-expansion will make all combinations and mkdir -p will take care of the rest.
Of course, you do not want to type this over and over for various sets. So you could generate the full brace-expansion-string with bash and use exec to process the brace-expansion-string before passing it to mkdir -p:
depth=3
set={a..c}
dirs=$(printf "/${set}%.0s" $(seq $depth))
mkdir -p $(eval echo .${dirs})
Be aware however that if your set has length m, and you want a depth of n, you are creating m^n directories. This number could conflict with the number of arguments you can pass on to a program.
Related information:
What is the maximum allowed depth of sub-folders?
https://www.in-ulm.de/~mascheck/various/argmax/
A recursive funcion may solve your problem. Take care with inodes generation when using high directory levels...
#!/bin/bash
function createDir {
mkdir -p $1 && cd $1;
for x in {a..z} ; do
local i=$(($2-1))
[ $i -lt 0 ] && continue;
createDir $x $i
done
cd ..
}
createDir $1 $2
Save into a file, like mkdir.sh, and call it: ./mkdir.sh <main_folder> <level>.
I am not familiar with shell. I just want to successfully run this piece of code which I downloaded from Github. When I directly ran the code with the command,
make file='example.txt' file_test
it didn't work, and here is the error.
makefile line 1: syntax error: unexpected end of file
I tried to fix the problem by unifying the encoding format of 'Makefile.sh' and 'example.txt' to be Unix, but still it did not work at all.
Then I asked my friend(he is also new to shell), and he told me to delete the "tab" before each line(not including the command lines). The error is different at this time, but I still could not fix it by using Google.
Makefile:75: *** commands commence before first target. Stop.
Please, somebody can help? It is driving me crazy and I have no time left for solving this problem. :( Thanks!!!
And, here is the code after I deleted the 'TAB'.
# State-of-the-art paper: http://www.aclweb.org/anthology/P18-1016
########################################Definitions########################################
SHELL := /bin/bash
NC := \033[0m
RED := \033[0;31m
GREEN := \033[0;32m
CYAN := \033[0;36m
work_dir := $(PWD)
script_dir := $(work_dir)/scripts
NTS_dir := $(work_dir)/NeuralTextSimplification_model
file?=""
filetype := $(shell file $(file) | cut -d: -f2)
article_name := $(basename $(notdir $(file)))
article_simple := $(basename $(file))-simple.txt
NTS_script := $(NTS_dir)/src/scripts/translate.sh
NTS_input := $(NTS_dir)/data/test.en
NTS_result_file = $(NTS_dir)/results_NTS/result_NTS_epoch11_10.19.t7_5
# passage = xml file representing a sentence parsed with TUPA
sentence_dir = $(script_dir)/sentences/$(article_name)
passage_dir = $(script_dir)/passages/$(article_name)
sentences = $(sentence_dir)/*.txt
passages = $(passage_dir)/*.xml
############################################################################################
.SILENT:
.ONESHELL: # To execute all commands in one single bash shell
.PHONY: all file_test NTS DSS tupa_parse split_to_sentences help clean
all: file_test $(article_simple)
printf "\n${GREEN}Article simplified :\n====================${NC}\n\n"
cat $(article_simple)
# evaluate the system results using BLUE & SARI metrics
# The original test.en must be given to evaluate (https://github.com/senisioi/NeuralTextSimplification)
evaluate: all
printf "\n${GREEN}Evaluating: (the original test.en must be given)${NC}\n\n"
python $(NTS_dir)/src/evaluate.py $(NTS_input) $(NTS_dir)/data/references/references.tsv $(NTS_dir)/predictions
############################################################################################
# The following targets are only useful as aliases for testing
# Generates the corresponding NTS model result file.
NTS: file_test $(NTS_result_file)
# Generates the article sentences split using the two semantic rules mentioned in the paper.
DSS: file_test $(NTS_input)
# Parses the article's sentences using TUPA -> output : xml files
tupa_parse: file_test $(passages)
# Splits the article into sentences, each one in a single file
split_to_sentences: file_test $(sentences)
#############################################################################################
# Testing the validity of the file
file_test:
# Whether the file was provided as an argument
if [ $(file) = "" ]; then
printf "${RED}ERROR${NC}: One & only file must be given in argument! Please specify it by:\nmake file=<file_name> <target>\n\n"; exit 1
fi
# Whether the file exists
if [ ! -f $(file) ]; then
printf "$(RED)ERROR${NC}: $(file) File not found!\n\n"; exit 1
fi
# Whether the file is empty
if [ ! -s $(file) ]; then
printf "$(RED)ERROR${NC}: $(file) is empty!\n\n"; exit 1
fi
# Whether it is a text file
if [[ ! "$(filetype)" = *"ASCII"* && ! "$(filetype)" = *"UTF-8"* ]]; then
printf "$(RED)ERROR${NC}: Only text files (ASCII or UTF-8 Unicode text) are accepted!\n\n"; exit 1
fi
# Prints help on the useful targets for the user
help:
printf "\n${CYAN}make file=<file_path>${NC} : Executes the simplification completely and avoids rebuilding if unnecessary.\n\n"
printf "${CYAN}make file=<file_path> file_test${NC} : Tests whether the given file is valid.\n\n"
printf "${CYAN}make file=<file_path> NTS${NC} : Generates the corresponding NTS model result file.\n\n"
printf "${CYAN}make file=<file_path> DSS${NC} : Generates the article's sentences split using the two semantic rules mentioned in the paper.\n\n"
printf "${CYAN}make file=<file_path> tupa_parse${NC} : Parses the article's sentences using TUPA -> output : xml files.\n\n"
printf "${CYAN}make file=<file_path> split_to_sentences${NC} : Splits the article into sentences, each one in a single file.\n\n"
printf "${CYAN}make clean${NC} : Cleans results of previous executions.\n\n"
##############################################################################################
# Produces the simple version of text as an output and writes it to "article_simple" variable
$(article_simple): $(NTS_result_file)
cd $(work_dir)
cat $(NTS_result_file) > $(article_simple)
# Equivalent for target NTS
$(NTS_result_file): $(NTS_script) $(NTS_input)
printf "\n${GREEN}Simplifying sentences using the NTS model : ... ${NC}\n\n"
cd $(NTS_dir)/src/scripts/
source ./translate.sh
# Equivalent for target DSS
$(NTS_input): $(script_dir)/split_sentences.py $(passages)
printf "\n${GREEN}Splitting the article's sentences : ... ${NC}\n"
python $(script_dir)/split_sentences.py $(passage_dir) > $(NTS_input)
# If the output of the splitting is empty, then it failed
if [ ! -s $(NTS_input) ]; then
printf "$(RED)ERROR${NC}: Splitting failed!\n\n"; exit 1
fi
# Equivalent for target tupa_parse
$(passages): $(sentences) | $(passage_dir)
printf "\n${GREEN}Parsing the article's sentences : ... ${NC}\n\n"
python -m tupa $(sentences) -m $(work_dir)/TUPA_models/ucca-bilstm
mv *.xml $(passage_dir)
# Equivalent for target split_to_sentences
$(sentences): $(file) $(script_dir)/article_to_sentences.py | $(sentence_dir)
python $(script_dir)/article_to_sentences.py $(file)
# Creates the passage directory
$(passage_dir): $(file)
if [ -d $(passage_dir) ]; then
rm -f $(passage_dir)/*.xml
else
mkdir -p $(script_dir)/passages/
mkdir $(passage_dir)/
fi
# Creates the sentence directory
$(sentence_dir): $(file)
if [ -d $(sentence_dir) ]; then
rm -f $(sentence_dir)/*.txt
else
mkdir -p $(script_dir)/sentences/
mkdir $(sentence_dir)
fi
# Cleans the residues from previous executions
clean:
rm -rf $(passage_dir)* $(sentence_dir)*
echo > $(NTS_result_file)
I've two documents:
an .json
an folder with random content
where <transaction> is id+sequancial (id1, id2... idn)
I'd like to populate this structure (.json + folder) to n. I mean:
I'd like to have id1.json and id1 folder, an id2.json and id2 folder... idn.json and idn folder.
Is there anyway (shell script) to populate this content?
It would be something like:
for (i=0,i<n,i++) {
copy "id" file to "id+i" file
copy "id" folder to "id+i" folder
}
Any ideas?
Your shell syntax is off but after that, this should be trivial.
#!/bin/bash
for((i=0;i<$1;i++)); do
cp "id".json "id$i".json
cp -r "id" "id$i"
done
This expects the value of n as the sole argument to the script (which is visible inside the script in $1).
The C-style for((...)) loop is Bash only, and will not work with sh.
A proper production script would also check that it received the expected parameter in the expected format (a single positive number) but you will probably want to tackle such complications when you learn more.
Additionaly, here is a version working with sh:
#!/bin/sh
test -e id.json || { (>&2 echo "id.json not found") ; exit 1 ; }
{
seq 1 "$1" 2> /dev/null ||
(>&2 echo "usage: $0 transaction-count") && exit 1
} |
while read i
do
cp "id".json "id$i".json
cp -r "id" "id$i"
done
I have a large data file that contains many joint files.
It has an separate index file has that file name, start + end byte of each file within the data file.
I'm needing help in creating a bash script to split the large file into it's 1000's of sub files.
Data File : fileafilebfilec etc
Index File:
filename.png<0>3049
folder\filename2.png<3049>6136.
I guess this needs to loop through each line of the index file, then using dd to extract the relevant bytes into a file. Maybe a fiddly part might be the folder structure bracket being windows style rather than linux style.
Any help much appreciated.
while read p; do
q=${p#*<}
startbyte=${q%>*}
endbyte=${q#*>}
filename=${p%<*}
count=$(($endbyte - $startbyte))
toprint="processing $filename startbyte: $startbyte endbyte: $endbyte count: $c$
echo $toprint
done <indexfile
Worked it out :-) FYI:
while read p; do
#sort out variables
q=${p#*<}
startbyte=${q%>*}
endbyte=${q#*>}
filename=${p%<*}
count=$(($endbyte - $startbyte))
#let it know we're working
toprint="processing $filename startbyte: $startbyte endbyte: $endbyte count: $c$
echo $toprint
if [[ $filename == *"/"* ]]; then
echo "have found /"
directory=${filename%/*}
#if no directory exists, create it
if [ ! -d "$directory" ]; then
# Control will enter here if $directory doesn't exist.
echo "directory not found - creating one"
mkdir ~/etg/$directory
fi
fi
dd skip=$startbyte count=$count if=~/etg/largefile of=~/etg/$filename bs=1
done <indexfile