I have a main directory with several subfolders. Each subfolder contains a *.fna file and I want my script to do the command with the fna file and write the output back in the subfolder.
As my script is now, it makes one big file in the main directory, but I want the output per subfolder, in each subfolder.
find * |grep fna$ |while read fna ; do formatdb -i $fna -p F ; blastall -p blastn/
-d $fna -i plasmiddb_genes_renamed.fsa -m 8 -e 1E-30 |while read hit ; do echo/
$fna $hit ; done ; done > $fna.blast_plasmidrefdb.out
Couple of ways
Capture the dirname of the $fna value
find * |grep fna$ |while read fna ; do fdir=dirname $fna; formatdb -i $fna -p F ; blastall -p blastn/
-d $fna -i plasmiddb_genes_renamed.fsa -m 8 -e 1E-30 |while read hit ; do echo/
$fna $hit | tee ${fdir}/out.txt; done ; done > $fna.blast_plasmidrefdb.out
use the $fna path with a suffix : e.g. ${fna}.out
find * |grep fna$ |while read fna ; do formatdb -i $fna -p F ; blastall -p blastn/
-d $fna -i plasmiddb_genes_renamed.fsa -m 8 -e 1E-30 |while read hit ; do echo/
$fna $hit | tee ${fna}.out; done ; done > $fna.blast_plasmidrefdb.out
used | tee in case you wanted to ALSO capture in one file still.
compressing the one liner even more
for fna in $(find . -type f -name "*.fna"); do formatdb -i $fna -p F ; blastall -p blastn/ -d $fna -i plasmiddb_genes_renamed.fsa -m 8 -e 1E-30 | tee ${fna}.out; done
If your find supports it, use -execdir:
find . -name '*.fna' -execdir sh -c '{
formatdb -i $0 -p F; blastall -p blastn -d $0 \
-i /path/to/plasmiddb_genes_renaed.fsa -m 8 -e 1E-30;
} > $0.output' {} \;
That will generate an output-file for each *.fna file in the directory containing the *.fna file. Note that you need to use a full path to plasmiddb_genes_renaed.fsa.
Related
I have files in this format:
2022-03-5344-REQUEST.jpg
2022-03-5344-IMAGE.jpg
2022-03-5344-00imgtest.jpg
2022-03-5344-anotherone.JPG
2022-03-5343-kdijffj.JPG
2022-03-5343-zslkjfs.jpg
2022-03-5343-myimage-2010.jpg
2022-03-5343-anotherone.png
2022-03-5342-ebee5654.jpeg
2022-03-5342-dec.jpg
2022-03-5341-att.jpg
2022-03-5341-timephoto_december.jpeg
....
about 13k images like these.
I want to create folders like:
2022-03-5344/
2022-03-5343/
2022-03-5342/
2022-03-5341/
....
I started manually moving them like:
mkdir name
mv name-* name/
But of course I'm not gonna repeat this process for 13k files.
So I want to do this using bash scripting, and since I am new to bash, and I am working on a production environment, I want to play it safe, but it doesn't give me my results. This is what I did so far:
#!/bin/bash
name = $1
mkdir "$name"
mv "${name}-*" $name/
and all I can do is: ./move.sh name for every folder, I didn't know how to automate this using loops.
With bash and a regex. I assume that the files are all in the current directory.
for name in *; do
if [[ "$name" =~ (^....-..-....)- ]]; then
dir="${BASH_REMATCH[1]}"; # dir contains 2022-03-5344, e.g.
echo mkdir -p "$dir" || exit 1;
echo mv -v "$name" "$dir";
fi;
done
If output looks okay, remove both echo.
Try this
xargs -i sh -c 'mkdir -p {}; mv {}-* {}' < <(ls *-*-*-*|awk -F- -vOFS=- '{print $1,$2,$3}'|uniq)
Or:
find . -maxdepth 1 -type f -name "*-*-*-*" | \
awk -F- -vOFS=- '{print $1,$2,$3}' | \
sort -u | \
xargs -i sh -c 'mkdir -p {}; mv {}-* {}'
Or find with regex:
find . -maxdepth 1 -type f -regextype posix-extended -regex ".*/[0-9]{4}-[0-9]{2}-[0-9]{4}.*"
You could use awk
$ cat awk.script
/^[[:digit:]-]/ && ! a[$1]++ {
dir=$1
} /^[[:digit:]-]/ {
system("sudo mkdir " dir )
system("sudo mv " $0" "dir"/"$0)
}
To call the script and use for your purposes;
$ awk -F"-([0-9]+)?[[:alpha:]]+.*" -f awk.script <(ls)
You will see some errors such as;
mkdir: cannot create directory ‘2022-03-5341’: File exists
after the initial dir has been created, you can safely ignore these as the dir now exist.
The content of each directory will now have the relevant files
$ ls 2022-03-5344
2022-03-5344-00imgtest.jpg 2022-03-5344-IMAGE.jpg 2022-03-5344-REQUEST.jpg 2022-03-5344-anotherone.JPG
what I am trying to achieve is, to delete same filenames(filename+modfiedtimestamp)exisitng in Src_Dir1 and Src_Dir2
So first i have tried to deploy all the filenames to tempa(Src_Dir1) and tempb(Src_Dir2) respectively.
Below is the screenshot of the source directory.
Files inside archive be like this and few files outside too..
So, initially I am want to deal with the files inside Archive(SRC_Dir1) and later outside Archive(SRC_Dir2) what I am trying to do is to use a while loop to read each and every filename and string concat with the modified timestamp(mtime) and input to tempc(like for example it should be like AirTimeActs_2018-12-03.csv+2019-01-24 14:41:53.000000000 -0500 = AirTimeActs_2018-12-03.csv_2019-01-24 14:41:53.000000000 -0500 this is how it should be generating into tempc file for each and every filename inside Archive(SRC_Dir1). This is where I am stuck under string concat variable section on how to proceed. Please help me with the code, hope I am comprehensible.
IMPORTANT
(Really appreciate it, if you help me out with the extension of the code which i haven't mentioned here and yet to achieve which is - >
Have to implement the same code(which I am trying to do for tempa, I'd like to do it for tempb too and name it as tempd) and then do a file data compare between tempc and tempd) if there is any kind of same data filename, then delete the file existing in Src_Dir2, if there is no same data filename, then do nothing.)
#!/bin/bash
Src_Dir1=path/Airtime_Activation/Archive
Src_Dir2=path/Airtime_Activation/
find "$Src_Dir1" -maxdepth 1 -name "*.xlsx" -o -name "*.csv" | sed "s/.*\///" > -print>path/Airtime_Activation/temp_a
find "$Src_Dir2" -maxdepth 1 -name "*.xlsx" -o -name "*.csv" | sed "s/.*\///" > -print>path/Airtime_Activation/temp_b
echo 'phase1'
cat path/Airtime_Activation/temp_a | while read file;
do
echo 'phase1.5'
echo "$file"
echo 'phase2'
mtime=$(stat -c '%y' $file)
Full_name=${file}_${mtime}
echo "$Full_name" >> path/Airtime_Activation/temp_c
echo 'phase3'
done
#!/bin/bash
Src_Dir1=path/Airtime_Activation/Archive
Src_Dir2=path/Airtime_Activation/
find "$Src_Dir1" -maxdepth 1 -name "*.xlsx" -o -name "*.csv" | sed "s/.*\///" > -print>path/Airtime_Activation/temp_a
find "$Src_Dir2" -maxdepth 1 -name "*.xlsx" -o -name "*.csv" | sed "s/.*\///" > -print>path/Airtime_Activation/temp_b
echo 'phase1'
cat path/Airtime_Activation/temp_a | while read file;
do
echo 'phase1.5'
echo "$file"
echo 'phase2'
mtime=$(stat -c '%y' $file)
Full_name=${file}_${mtime}
echo "$Full_name" >> path/Airtime_Activation/temp_c
echo 'phase3'
done
cat /path/Airtime_Activation/temp_b | while read file
#while IFS="" read -r -d $'\0' file;
do
#echo "$file"
echo 'phase2'
mtime=$(stat -c '%y' $Src_Dir2/$file)
Full_name=${file}_${mtime}
echo "$Full_name" >> path/temp_d
echo 'phase3'
done
#file compare and delete old files from outisde archive
grep -Ff temp_d temp_c > path/Airtime_Activation/temp_e
cat path/Airtime_Activation/temp_e | while read file
#while IFS="" read -r -d $'\0' file;
do
#echo "$file"
echo 'phase2'
echo "${file%_*}"
rm $Src_Dir2/${file%_*}
echo 'phase3'
done
#!/bin/bash
for x in ontwikkelkaart
do
echo "***";
echo ${x};
ssh ${x}#localhost "
find ~/public_html/wp-content/themes/ -type f -name "*.webp" | awk '{ gsub(".webp$", "") ; print $0 }' | xargs -i sh -c 'if [ ! -f "{}" ]; then echo {}.webp; fi' \;
"
done
I have the above script that connects to a server via SSH, it checks wether there are webp files with no jpg/png as source file; and echo's rm "filename".
The command:
find ~/public_html/wp-content/themes/ -type f -name "*.webp" | awk '{ gsub(".webp$", "") ; print $0 }' | xargs -i sh -c 'if [ ! -f "{}" ]; then echo {}.webp; fi' \;
Works when i run it on the command line of the server (via SSH), but when i try to do it in the for loop, it does not work because of the "".
Can someone (try to) explain why the above code does not work?
Try using a heredoc :
ssh -q -T ${x}#localhost 2> /dev/null <<'EOF'
find ~/public_html/wp-content/themes/ -type f -name "*.webp" | awk '{ gsub(".webp$", "") ; print $0 }' | xargs -i sh -c 'if [ ! -f "{}" ]; then echo {}.webp; fi' \;
EOF
I am having trouble looping and searching. It seems that the loop is not waiting for the find to finish. What am I doing wrong?
I made a loop the reads a file line by line. I then want to use that "name" to search a directory looking to see if a folder has that name. If it exists copy it to a drive.
#!/bin/bash
DIRFIND="$2"
DIRCOPY="$3"
if [ -d $DIRFIND ]; then
while IFS='' read -r line || [[ -n "$line" ]]; do
echo "$line"
FILE=`find "$DIRFIND" -type d -name "$line"`
if [ -n "$FILE" ]; then
echo "Found $FILE"
cp -a "$FILE" "$DIRCOPY"
else
echo "$line not found."
fi
done < "$1"
else
echo "No such file or directory"
fi
Have you tried xargs...
Proposed Solution
cat filenamelist | xargs -n1 -I {} find . -type d -name {} -print | xargs -n1 -I {} mv {} .
what the above does is pipe a list of filenames into find (one at a time), when found find prints the name and passes to xarg which moves the file...
Expansion
file = yogo
yogo -> | xargs -n1 -I yogo find . -type d -name yogo -print | xargs -n1 -I {} mv ./<path>/yogo .
I hope the above helps, note that xargs has the advantage that you do not run out of command line buffer.
The below code snippet is for searching files recursively and iterating them.
find . -type f -not -name '*.ini' -print0 | while IFS= read -r -d '' filename; do
echo "$filename"
done
It gives this resut:
1.jpg
2.jpg
3.jpg
But if I want to process the file somehow like this
find . -type f -not -name '*.ini' -print0 | while IFS= read -r -d '' filename; do
echo "$filename"
echo "$(${ExternalApp} -someparams $filename 2> /dev/null| cut -f 2- -d: | cut -f 2- -d ' ' )"
done
The loop terminates after the first iteration and result become like this:
1.jpg
I have recently updated bash (I'm on windows with MSYS). What is the problem here?
find's output is read by the command. This is an especially common problem when using ssh, ffmpeg or mplayer.
You can redirect from /dev/null if it doesn't need input at all:
find . -type f -not -name '*.ini' -print0 | while IFS= read -r -d '' filename; do
echo "$filename"
# v-- here
echo "$(${ExternalApp} -someparams $filename < /dev/null 2> /dev/null |
cut -f 2- -d: | cut -f 2- -d ' ' )"
done