Recursively create directories for all letters - bash

I would like to create a folder structure based on a brace expansion such as {a-z}. Each string generated by the brace-expansion should be a new folder. Furthermore, each of these folders should contain the same set of subfolders similarly generated. And this up to a given level.
An example for the range a-z and depth 16
a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/
a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/b/
a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/c/
...
d/a/h/r/y/d/s/b/e/y/k/f/o/o/q/c/
...
z/z/z/z/z/z/z/z/z/z/z/z/z/z/z/y/
z/z/z/z/z/z/z/z/z/z/z/z/z/z/z/z/
The following code allows me to go upto depth 2:
for x in {a..z} ; do mkdir -p $x/{a..z} ; done
But how do I go further?

A recursive solution. Job is called with 2 params: max_depth and base_path
#!/bin/bash
function job()
{
local depth=$(($1-1))
local path=$2
local x
for x in a b c # reduced for test
do
mkdir -p "$path/$x"
((depth>0)) && job $depth "$path/$x"
done
}
job 3 ./test
Proof it with:
find test -type d

The simplest form would be to use any of the following lines:
mkdir -p {a..c}/{a..c} # depth 2
mkdir -p {a..c}/{a..c}/{a..c} # depth 3
mkdir -p {a..c}/{a..c}/{a..c}/{a..c} # depth 4
...
The brace-expansion will make all combinations and mkdir -p will take care of the rest.
Of course, you do not want to type this over and over for various sets. So you could generate the full brace-expansion-string with bash and use exec to process the brace-expansion-string before passing it to mkdir -p:
depth=3
set={a..c}
dirs=$(printf "/${set}%.0s" $(seq $depth))
mkdir -p $(eval echo .${dirs})
Be aware however that if your set has length m, and you want a depth of n, you are creating m^n directories. This number could conflict with the number of arguments you can pass on to a program.
Related information:
What is the maximum allowed depth of sub-folders?
https://www.in-ulm.de/~mascheck/various/argmax/

A recursive funcion may solve your problem. Take care with inodes generation when using high directory levels...
#!/bin/bash
function createDir {
mkdir -p $1 && cd $1;
for x in {a..z} ; do
local i=$(($2-1))
[ $i -lt 0 ] && continue;
createDir $x $i
done
cd ..
}
createDir $1 $2
Save into a file, like mkdir.sh, and call it: ./mkdir.sh <main_folder> <level>.

Related

Creating complex path structure in linux

I am having trouble creating the following directory structure. It is to test a scanner program to see if it can handle it.
Start inside /data/
Inside /data/ there are 2 sub-directories. /data/sub1 and /data/sub2
Now, this gets complex :)
In each of the sub-directories there are 2 more sub-directories. For first sub: /data/sub1/sub1/ and /data/sub1/sub2 and for second sub /data/sub2/sub1/ and /data/sub2/sub2
... this goes on to a depth of 16 levels, with a total directory of 65,536.
What I have so far, this only creates 1 subdirectory inside /data/, 16 folders deep. I am thinking of using the following script recursively for the subdirectories created along the way. And for the subdirectories it then creates. And so on. But I can't imagine scripting it.
count=0;
folder=$(printf "Folder%02dB" $count);
mkdir -p $folder;
while [ $count -lt 16 ]; do
((count=$count+1));
folder=$(printf "Folder%02dB" $count);
mkdir -p $folder;
cd $folder;
done
Anyone has any ideas on how to create such directory structure? How to proceed? Thanks!
Recursion is a good way to do this.
#! /bin/bash
godeep()
{
if [ $1 -lt 16 ]
then
((level = $1 + 1))
mkdir sub1
cd sub1
godeep $level
cd ..
mkdir sub2
cd sub2
godeep $level
cd ..
fi
}
godeep 1
~
You shouldn't be incorporating $count in the folder name. That's just used to count the recursion depth.
Simplest would be to define a function to create a single level, and call it recursively.
create_folders() {
if (( $1 > 16 )); then
return
fi
depth=$(($1 + 1))
mkdir sub1; (cd sub1; create_folders $depth)
mkdir sub2; (cd sub2; create_folders $depth)
}
create_folders 1
Using () around the cd and recursive call keeps them from affecting the working directory and variables in the current call.

How can I deal with Makefile syntax error made by "tab"?

I am not familiar with shell. I just want to successfully run this piece of code which I downloaded from Github. When I directly ran the code with the command,
make file='example.txt' file_test
it didn't work, and here is the error.
makefile line 1: syntax error: unexpected end of file
I tried to fix the problem by unifying the encoding format of 'Makefile.sh' and 'example.txt' to be Unix, but still it did not work at all.
Then I asked my friend(he is also new to shell), and he told me to delete the "tab" before each line(not including the command lines). The error is different at this time, but I still could not fix it by using Google.
Makefile:75: *** commands commence before first target. Stop.
Please, somebody can help? It is driving me crazy and I have no time left for solving this problem. :( Thanks!!!
And, here is the code after I deleted the 'TAB'.
# State-of-the-art paper: http://www.aclweb.org/anthology/P18-1016
########################################Definitions########################################
SHELL := /bin/bash
NC := \033[0m
RED := \033[0;31m
GREEN := \033[0;32m
CYAN := \033[0;36m
work_dir := $(PWD)
script_dir := $(work_dir)/scripts
NTS_dir := $(work_dir)/NeuralTextSimplification_model
file?=""
filetype := $(shell file $(file) | cut -d: -f2)
article_name := $(basename $(notdir $(file)))
article_simple := $(basename $(file))-simple.txt
NTS_script := $(NTS_dir)/src/scripts/translate.sh
NTS_input := $(NTS_dir)/data/test.en
NTS_result_file = $(NTS_dir)/results_NTS/result_NTS_epoch11_10.19.t7_5
# passage = xml file representing a sentence parsed with TUPA
sentence_dir = $(script_dir)/sentences/$(article_name)
passage_dir = $(script_dir)/passages/$(article_name)
sentences = $(sentence_dir)/*.txt
passages = $(passage_dir)/*.xml
############################################################################################
.SILENT:
.ONESHELL: # To execute all commands in one single bash shell
.PHONY: all file_test NTS DSS tupa_parse split_to_sentences help clean
all: file_test $(article_simple)
printf "\n${GREEN}Article simplified :\n====================${NC}\n\n"
cat $(article_simple)
# evaluate the system results using BLUE & SARI metrics
# The original test.en must be given to evaluate (https://github.com/senisioi/NeuralTextSimplification)
evaluate: all
printf "\n${GREEN}Evaluating: (the original test.en must be given)${NC}\n\n"
python $(NTS_dir)/src/evaluate.py $(NTS_input) $(NTS_dir)/data/references/references.tsv $(NTS_dir)/predictions
############################################################################################
# The following targets are only useful as aliases for testing
# Generates the corresponding NTS model result file.
NTS: file_test $(NTS_result_file)
# Generates the article sentences split using the two semantic rules mentioned in the paper.
DSS: file_test $(NTS_input)
# Parses the article's sentences using TUPA -> output : xml files
tupa_parse: file_test $(passages)
# Splits the article into sentences, each one in a single file
split_to_sentences: file_test $(sentences)
#############################################################################################
# Testing the validity of the file
file_test:
# Whether the file was provided as an argument
if [ $(file) = "" ]; then
printf "${RED}ERROR${NC}: One & only file must be given in argument! Please specify it by:\nmake file=<file_name> <target>\n\n"; exit 1
fi
# Whether the file exists
if [ ! -f $(file) ]; then
printf "$(RED)ERROR${NC}: $(file) File not found!\n\n"; exit 1
fi
# Whether the file is empty
if [ ! -s $(file) ]; then
printf "$(RED)ERROR${NC}: $(file) is empty!\n\n"; exit 1
fi
# Whether it is a text file
if [[ ! "$(filetype)" = *"ASCII"* && ! "$(filetype)" = *"UTF-8"* ]]; then
printf "$(RED)ERROR${NC}: Only text files (ASCII or UTF-8 Unicode text) are accepted!\n\n"; exit 1
fi
# Prints help on the useful targets for the user
help:
printf "\n${CYAN}make file=<file_path>${NC} : Executes the simplification completely and avoids rebuilding if unnecessary.\n\n"
printf "${CYAN}make file=<file_path> file_test${NC} : Tests whether the given file is valid.\n\n"
printf "${CYAN}make file=<file_path> NTS${NC} : Generates the corresponding NTS model result file.\n\n"
printf "${CYAN}make file=<file_path> DSS${NC} : Generates the article's sentences split using the two semantic rules mentioned in the paper.\n\n"
printf "${CYAN}make file=<file_path> tupa_parse${NC} : Parses the article's sentences using TUPA -> output : xml files.\n\n"
printf "${CYAN}make file=<file_path> split_to_sentences${NC} : Splits the article into sentences, each one in a single file.\n\n"
printf "${CYAN}make clean${NC} : Cleans results of previous executions.\n\n"
##############################################################################################
# Produces the simple version of text as an output and writes it to "article_simple" variable
$(article_simple): $(NTS_result_file)
cd $(work_dir)
cat $(NTS_result_file) > $(article_simple)
# Equivalent for target NTS
$(NTS_result_file): $(NTS_script) $(NTS_input)
printf "\n${GREEN}Simplifying sentences using the NTS model : ... ${NC}\n\n"
cd $(NTS_dir)/src/scripts/
source ./translate.sh
# Equivalent for target DSS
$(NTS_input): $(script_dir)/split_sentences.py $(passages)
printf "\n${GREEN}Splitting the article's sentences : ... ${NC}\n"
python $(script_dir)/split_sentences.py $(passage_dir) > $(NTS_input)
# If the output of the splitting is empty, then it failed
if [ ! -s $(NTS_input) ]; then
printf "$(RED)ERROR${NC}: Splitting failed!\n\n"; exit 1
fi
# Equivalent for target tupa_parse
$(passages): $(sentences) | $(passage_dir)
printf "\n${GREEN}Parsing the article's sentences : ... ${NC}\n\n"
python -m tupa $(sentences) -m $(work_dir)/TUPA_models/ucca-bilstm
mv *.xml $(passage_dir)
# Equivalent for target split_to_sentences
$(sentences): $(file) $(script_dir)/article_to_sentences.py | $(sentence_dir)
python $(script_dir)/article_to_sentences.py $(file)
# Creates the passage directory
$(passage_dir): $(file)
if [ -d $(passage_dir) ]; then
rm -f $(passage_dir)/*.xml
else
mkdir -p $(script_dir)/passages/
mkdir $(passage_dir)/
fi
# Creates the sentence directory
$(sentence_dir): $(file)
if [ -d $(sentence_dir) ]; then
rm -f $(sentence_dir)/*.txt
else
mkdir -p $(script_dir)/sentences/
mkdir $(sentence_dir)
fi
# Cleans the residues from previous executions
clean:
rm -rf $(passage_dir)* $(sentence_dir)*
echo > $(NTS_result_file)

Shell script: Copy file and folder N times

I've two documents:
an .json
an folder with random content
where <transaction> is id+sequancial (id1, id2... idn)
I'd like to populate this structure (.json + folder) to n. I mean:
I'd like to have id1.json and id1 folder, an id2.json and id2 folder... idn.json and idn folder.
Is there anyway (shell script) to populate this content?
It would be something like:
for (i=0,i<n,i++) {
copy "id" file to "id+i" file
copy "id" folder to "id+i" folder
}
Any ideas?
Your shell syntax is off but after that, this should be trivial.
#!/bin/bash
for((i=0;i<$1;i++)); do
cp "id".json "id$i".json
cp -r "id" "id$i"
done
This expects the value of n as the sole argument to the script (which is visible inside the script in $1).
The C-style for((...)) loop is Bash only, and will not work with sh.
A proper production script would also check that it received the expected parameter in the expected format (a single positive number) but you will probably want to tackle such complications when you learn more.
Additionaly, here is a version working with sh:
#!/bin/sh
test -e id.json || { (>&2 echo "id.json not found") ; exit 1 ; }
{
seq 1 "$1" 2> /dev/null ||
(>&2 echo "usage: $0 transaction-count") && exit 1
} |
while read i
do
cp "id".json "id$i".json
cp -r "id" "id$i"
done

Bash - How to for-loop last digit of a variable

I am trying to automate creation of folders. Each folder should have a number at beginning of file name (increased by the for-loop [$i]. Also the rest of the folder name should be build up by variables that are constructed as [folder_x] where [x] is supposed to be stepped up also by the loop.
To be more specific. How do you build a string that is combined by the for-loops [$i] and a variable that is called, but should at the end also use the [$i]?
For more details see below:
#!/bin/bash
# Variables to be used
folder_1=1-folderOne
folder_2=2-folderTwo
folder_3=3-folderThree
# This is the folder names that should be created:
# mkdir /tmp2/1-folderOne
# mkdir /tmp2/2-folderTwo
# mkdir /tmp2/3-folderThree
# The for loop should combine the [$i] and above [folder_x],
# where the [x] should also be increased by the loop.
# Below is what i have right now:
# Note! The text "created-by-forloop" is just dummy text,
# and should be replace by the real solution.
for i in 1 2 3
do
if [ ! -d /tmp2/$i-created-by-forloop ]; then
mkdir -p /tmp2/$i-created-by-forloop;
fi
done
Use an array of names instead of distinct variables for each:
numbers=(One Two Three)
for i in "${!numbers[#]}" ; do
mkdir /tmp2/$((i+1))-folder"${numbers[i]}"
done
The loop iterates $i over the indices of the array. We need to add 1 to the index as arrays are zero based, but we want our files to be numbered from 1, not 0.

batch processing : File name comparison error

I have written a program (Cifti_subject_fmri) which compares whether file name matches in two folders and essentially executes a set of instructions
#!/bin/bash -- fix_mni_paths
source activate ciftify_v1.0.0
export SUBJECTS_DIR=/scratch/m/mchakrav/dev/functional_data
export HCP_DATA=/scratch/m/mchakrav/dev/tCDS_ciftify
## make the $SUBJECTS_DIR if it does not already exist
mkdir -p ${HCP_DATA}
SUBJECTS=`cd $SUBJECTS_DIR; ls -1d *` ## list of my subjects
HCP=`cd $HCP_DATA; ls -1d *` ## List of HCP Subjects
cd $HCP_DATA
## submit the files to the queue
for i in $SUBJECTS;do
for j in $HCP ; do
if [[ $i == $j ]];then
parallel "echo ciftify_subject_fmri $i/filtered_func_data.nii.gz $j fMRI " ::: $SUBJECTS |qbatch --walltime '05:00:00' --ppj 8 -c 4 -j 4 -N ciftify_subject_fmri -
fi
done
done
When i run this code in the cluster i am getting an error which says
./Cifti_subject_fmri: [[AS1: command not found
The query ciftify_subject_fmri is part of toolbox ciftify, for it to execute it requires following instructions
ciftify_subject_fmri <func.nii.gz> <Subject> <NameOffMRI>
I have 33 subjects [AS1 -AS33] each with its own func.nii.gz files located SUBJECTS directory,the results need to be populated in HCP directory, fMRI is name of file format .
Could some one kindly let me know why i am getting an error in loop

Resources