How to iterate through multiple directories with multiple ifs in bash? - bash

unfortunately I'm quite new at bash, and I want to write a script that will start in a main directory, and check all subdirectories one by one for the presence of certain files, and if those files are present, perform an operation on them. For now, I have written a simplified version to test whether I can do the first part (checking for the files in each directory). This code runs without any errors that I can tell, but it does not echo anything to say that it has successfully found the files which I know are there.
#!/bin/bash
runlist=(1 2 3 4 5 6 7 8 9)
for f in *; do
if [[ -d {$f} ]]; then
#if f is a directory then cd into it
cd "{$f}"
for b in $runlist; do
if [[ -e "{$b}.png" ]]; then
echo "Found {$b}"
#if the file exists then say so
fi
done
cd -
fi
done
'''

Welcome to stackoverflow.
The following will do the trick (a combination of find, array, and if then else):
# list of files we are looking for
runlist=(1 2 4 8 16 32 64 128)
#find each of above anywhere below current directory
# using -maxdepth 1 because, based on on your exam you want to look one level only
# if that's not what you want then take out -maxdepth 1 from the find command
for b in ${runlist[#]}; do
echo
PATH_TO_FOUND_FILE=`find . -name $b.png`
if [ -z "$PATH_TO_FOUND_FILE" ]
then
echo "nothing found" >> /dev/null
else
# You wanted a postive confirmation, so
echo found $b.png
# Now do something with the found file. Let's say ls -l: change that to whatever
ls -l $PATH_TO_FOUND_FILE
fi
done
Here is an example run:
mamuns-mac:stack foo$ ls -lR
total 8
drwxr-xr-x 4 foo 1951595366 128 Apr 11 18:03 dir1
drwxr-xr-x 3 foo 1951595366 96 Apr 11 18:03 dir2
-rwxr--r-- 1 foo 1951595366 652 Apr 11 18:15 find_file_and_do_something.sh
./dir1:
total 0
-rw-r--r-- 1 foo 1951595366 0 Apr 11 17:58 1.png
-rw-r--r-- 1 foo 1951595366 0 Apr 11 17:58 8.png
./dir2:
total 0
-rw-r--r-- 1 foo 1951595366 0 Apr 11 18:03 64.png
mamuns-mac:stack foo$ ./find_file_and_do_something.sh
found 1.png
-rw-r--r-- 1 foo 1951595366 0 Apr 11 17:58 ./dir1/1.png
found 8.png
-rw-r--r-- 1 foo 1951595366 0 Apr 11 17:58 ./dir1/8.png
found 64.png
-rw-r--r-- 1 foo 1951595366 0 Apr 11 18:03 ./dir2/64.png

Related

Calling a bash script function from another bash script doesn't display my shell command

I have 2 bash scripts, one calling another but depending on how I call it, it does or does not display my ls command.
script2.sh
#!/bin/bash
function test() {
i=0
while IFS= read -r line; do
IFS=',' read -ra ITEM <<<"$line"
printf "\n[${i}] ${ITEM}"
((i = i + 1))
done <<<$(ls $1)
printf "\nPress any other keys to abort.\n\n"
read -p "Please enter your selection: " ANSWER
echo $ANSWER
}
script1a.sh WORKS
#!/bin/bash
. ./scripts/bash/script2.sh
PARAM='-lag'
(test $PARAM)
Returns:
[0] total 5
[1] drwxr-xr-x 1 1049089 0 Oct 29 09:10 .
[2] drwxr-xr-x 1 1049089 0 Oct 9 23:11 ..
[3] -rw-r--r-- 1 1049089 87 Jul 6 14:19 .eslintignore
[4] -rw-r--r-- 1 1049089 449 Jul 10 13:56 .forceignore
[5] drwxr-xr-x 1 1049089 0 Oct 29 09:11 .git
Press any other keys to abort.
Please enter your selection:
script1b.sh FAILS
#!/bin/bash
. ./scripts/bash/script2.sh
PARAM='-lag'
myanswer=$(test $PARAM)
Returns:
Please enter your selection:
Anyone knows why this odd behavior and how to get around it? Thanks in advance.

Does anybody have a script that counts the number of consecutive files which contain a specific word?

Any resources or advice would help, since I am pretty rubbish at scripting
So, I need to go to this path: /home/client/data/storage/customer/data/2020/09/15
And check to see if there are 5 or more consecutive files that contain the word "REJECTED":
ls -ltr
-rw-rw-r-- 1 root root 5059 Sep 15 00:05 customer_rlt_20200915000514737_20200915000547948_8206b49d-b585-4360-8da0-e90b8081a399.zip
-rw-rw-r-- 1 root root 5023 Sep 15 00:06 customer_rlt_20200915000547619_20200915000635576_900b44dc-1cf4-4b1b-a04f-0fd963591e5f.zip
-rw-rw-r-- 1 root root 39856 Sep 15 00:09 customer_rlt_20200915000824108_20200915000908982_b87b01b3-a5dc-4a80-b19d-14f31ff667bc.zip
-rw-rw-r-- 1 root root 39719 Sep 15 00:09 customer_rlt_20200915000901688_20200915000938206_38261b59-8ebc-4f9f-9e2d-3e32eca3fd4d.zip
-rw-rw-r-- 1 root root 12829 Sep 15 00:13 customer_rlt_20200915001229811_20200915001334327_1667be2f-f1a7-41ae-b9ca-e7103d9abbf8.zip
-rw-rw-r-- 1 root root 12706 Sep 15 00:13 customer_rlt_20200915001333922_20200915001357405_609195c9-f23a-4984-936f-1a0903a35c07.zip
Example of rejected file:
customer_rlt_20200513202515792_20200513202705506_5b8deae0-0405-413c-9a81-d1cc2171fa51REJECTED.zip
What I have so far:
!/bin/bash
YYYY=$(date +%Y);
MM=$(date +%m)
DD=$(date +%d)
#Set constants
CODE_OK=0
CODE_WARN=1
CODE_CRITICAL=2
CODE_UNKNOWN=3
#Set Default Values
FILE="/home/client/data/storage/customer/data/${YYYY}/${MM}/{DD}"
if [ ! -f $FILE ]
then
echo "NO TRANSACTIONS FOUND"
exit $CODE_CRITICAL
fi
You can do something quick in AWK:
$ cat consec.awk
/REJECTED/ {
if (match_line == NR - 1) {
consecutives++
} else {
consecutives = 1
}
if (consecutives == 5) {
print "5 REJECTED"
exit
}
match_line = NR
}
$ touch 1 2REJECTED 3REJECTED 5REJECTED 6REJECTED 7REJECTED 8
$ ls -1 | awk -f consec.awk
5 REJECTED
$ rm 3REJECTED; touch 3
$ ls -1 | awk -f consec.awk
$
This works by matching line containing REJECTED, counting consecutive lines (checked with match_line == NR - 1, which means "the last matching line was the previous line") and printing "5 REJECTED" if the number of consecutive lines is 5.
I've used ls -1 (note digit 1, not letter l) to sort by filename in this example. You could use ls -1rt (digit 1 again) to sort by file modification time, as in your original post.

For loop with if statements isn't working as expected in bash

It only prints the "else" statement for everything but I know for a fact the files exist that it's looking for. I've tried adapting some of the other answers but I thought this should definitely work.
Does anyone know what's wrong with my syntax?
# Contents of script
for ID_SAMPLE in $(cut -f1 metadata.tsv | tail -n +2);
do if [ -f ./output/${ID_SAMPLE} ]; then
echo Skipping ${ID_SAMPLE};
else
echo Processing ${ID_SAMPLE};
fi
done
Additional information
# Output directory
(base) -bash-4.1$ ls -lhS output/
total 170K
drwxr-xr-x 8 jespinoz tigr 185 Jan 3 16:16 ERR1701760
drwxr-xr-x 8 jespinoz tigr 185 Jan 17 18:03 ERR315863
drwxr-xr-x 8 jespinoz tigr 185 Jan 16 23:23 ERR599042
drwxr-xr-x 8 jespinoz tigr 185 Jan 17 00:10 ERR599072
drwxr-xr-x 8 jespinoz tigr 185 Jan 16 13:00 ERR599078
# Example of inputs
(base) -bash-4.1$ cut -f1 metadata.tsv | tail -n +2 | head -n 10
ERR1701760
ERR599078
ERR599079
ERR599070
ERR599071
ERR599072
ERR599073
ERR599074
ERR599075
ERR599076
# Output of script
(base) -bash-4.1$ bash test.sh | head -n 10
Processing ERR1701760
Processing ERR599078
Processing ERR599079
Processing ERR599070
Processing ERR599071
Processing ERR599072
Processing ERR599073
Processing ERR599074
Processing ERR599075
Processing ERR599076
# Checking a directory
(base) -bash-4.1$ ls -l ./output/ERR1701760
total 294
drwxr-xr-x 2 jespinoz tigr 386 Jan 15 21:00 checkpoints
drwxr-xr-x 2 jespinoz tigr 0 Jan 10 01:36 tmp
-f is for checking whether the name is a file, but all your names are directories. Use -d to check that.
if [ -d "./output/$ID_SAMPLE" ]
then
If you want to check whether the name exists with any type, use -e.

Get list of files group by Date

I have a directory with files coming for every day. Now I want to zip those files group by dates. Is there anyway to group/list the files which landed in same date.
Suppose there are below files in a directory
-rw-r--r--. 1 anirban anirban 1598 Oct 14 07:19 hello.txt
-rw-r--r--. 1 anirban anirban 1248 Oct 14 07:21 world.txt
-rw-rw-r--. 1 anirban anirban 659758 Oct 14 11:55 a
-rw-rw-r--. 1 anirban anirban 9121 Oct 18 07:37 b.csv
-rw-r--r--. 1 anirban anirban 196 Oct 20 08:46 go.xls
-rw-r--r--. 1 anirban anirban 1698 Oct 20 08:52 purge.sh
-rw-r--r--. 1 anirban anirban 47838 Oct 21 08:05 code.java
-rw-rw-r--. 1 anirban anirban 9446406 Oct 24 05:51 cron
-rw-rw-r--. 1 anirban anirban 532570 Oct 24 05:57 my.txt
drwxrwsr-x. 2 anirban anirban 67 Oct 25 05:05 look_around.py
-rw-rw-r--. 1 anirban anirban 44525 Oct 26 17:23 failed.log
So there are no way to group the files with any suffix/prefix, since all are unique. Now when I will run the command I am seeking I will get a set of lines like below based on group by dates.
[ [hello.txt world.txt a] [b.csv] [go.xls purge.sh] [code.java] ... ] and so on.
With that list I will loop through and make archive
tar -zvcf Oct_14.tar.gz hello.txt world.txt a
If you have the GNU version of the date command, you can get the date of modification of a file with the -r flag, which can be very useful.
For example, given the file list in your question, date +%b_%d -r hello.txt will output Oct_14.
Using this, you could loop over the files, and build up tar files:
If the tar file doesn't exist, create it with a single file
If the tar file exists, add the file to it
After the loop, zip the tar files
Like this:
#!/usr/bin/env bash
tarfiles=()
for file; do
tarfile=$(date +%b_%d.tar -r "$file")
if ! [ -f "$tarfile" ]; then
tar cf "$tarfile" "$file"
tarfiles+=("$tarfile")
else
tar uf "$tarfile" "$file"
fi
done
for tarfile in "${tarfiles[#]}"; do
gzip "$tarfile"
done
Pass the list of files you want to archive as command line parameters, for example if /path/to/files is the directory where you want to archive files (listed in your question), and you save this script in ~/bin/tar-by-dates.sh, then you can use like this:
cd /path/to/files
~/bin/tar-by-dates.sh *
Create zero-seperated list of (Month_Day.tar FILENAME) pairs and use xargs to add each file to the corresponding archive:
find . -maxdepth 1 -mindepth 1 -type f -printf "%Tb%Td.tar\0%f\0"|xargs -n 2 -0 tar uf

/dev/stdin with herestring

I would like a Bash script that can take input from a file or stdin, much like grep, for example
$ cat hw.txt
Hello world
$ grep wor hw.txt
Hello world
$ echo 'Hello world' | grep wor
Hello world
$ grep wor <<< 'Hello world'
Hello world
all works beautifully. However with the following script
read b < "${1-/dev/stdin}"
echo $b
It fails if using a herestring
$ hw.sh hw.txt
Hello world
$ echo 'Hello world' | hw.sh
Hello world
$ hw.sh <<< 'Hello world'
/opt/a/hw.sh: line 1: /dev/stdin: No such file or directory
Using /dev/stdin in this manner can be problematic because you are attempting to get a handle to stdin using a name in the filesystem (/dev/stdin) rather than using the file descriptor which bash has already handed you as stdin (file descriptor 0).
Here's a small script for you to test:
#!/bin/bash
echo "INFO: Listing of /dev"
ls -al /dev/stdin
echo "INFO: Listing of /proc/self/fd"
ls -al /proc/self/fd
echo "INFO: Contents of /tmp/sh-thd*"
cat /tmp/sh-thd*
read b < "${1-/dev/stdin}"
echo "b: $b"
On my cygwin installation this produces the following:
./s <<< 'Hello world'
$ ./s <<< 'Hello world'
INFO: Listing of /dev
lrwxrwxrwx 1 austin None 15 Jan 23 2012 /dev/stdin -> /proc/self/fd/0
INFO: Listing of /proc/self/fd
total 0
dr-xr-xr-x 2 austin None 0 Mar 11 14:27 .
dr-xr-xr-x 3 austin None 0 Mar 11 14:27 ..
lrwxrwxrwx 1 austin None 0 Mar 11 14:27 0 -> /tmp/sh-thd-1362969584
lrwxrwxrwx 1 austin None 0 Mar 11 14:27 1 -> /dev/tty0
lrwxrwxrwx 1 austin None 0 Mar 11 14:27 2 -> /dev/tty0
lrwxrwxrwx 1 austin None 0 Mar 11 14:27 3 -> /proc/5736/fd
INFO: Contents of /tmp/sh-thd*
cat: /tmp/sh-thd*: No such file or directory
./s: line 12: /dev/stdin: No such file or directory
b:
What this output shows is that bash is creating a temporary file to hold your HERE document (/tmp/sh-thd-1362969584) and making it available on file descriptor 0, stdin. However, the temporary file has already been unlinked from the file system and so is not accessible by reference through a file system name such as /dev/stdin. You can get the contents by reading file descriptor 0, but not by trying to open /dev/stdin.
On Linux, the ./s script above gives the following, showing that the file has been unlinked:
INFO: Listing of /dev
lrwxrwxrwx 1 root root 15 Mar 11 09:26 /dev/stdin -> /proc/self/fd/0
INFO: Listing of /proc/self/fd
total 0
dr-x------ 2 austin austin 0 Mar 11 14:30 .
dr-xr-xr-x 7 austin austin 0 Mar 11 14:30 ..
lr-x------ 1 austin austin 64 Mar 11 14:30 0 -> /tmp/sh-thd-1362965400 (deleted) <---- /dev/stdin not found
lrwx------ 1 austin austin 64 Mar 11 14:30 1 -> /dev/pts/12
lrwx------ 1 austin austin 64 Mar 11 14:30 2 -> /dev/pts/12
lr-x------ 1 austin austin 64 Mar 11 14:30 3 -> /proc/10659/fd
INFO: Contents of /tmp/sh-thd*
cat: /tmp/sh-thd*: No such file or directory
b: Hello world
Change your script to use the stdin supplied, rather than trying to reference through /dev/stdin.
if [ -n "$1" ]; then
read b < "$1"
else
read b
fi
bash parses some file names (like /dev/stdin) specially, so that they are recognized even if they are not actually present in the file system. If your script doesn't have #!/bin/bash at the top, and /dev/stdin isn't in your file system, your script may be run using /bin/sh, which would expect /dev/stdin to actually be a file.
(This should, perhaps, not be an answer, but rather a comment to Austin's answer.)
$ cat ts.sh
read b < "${1-/dev/stdin}"
echo $b
$ ./ts.sh <<< 'hello world'
hello world
No problem for me. I'm using bash 4.2.42 on Mac OS X.
You got a typo here
read b < "${1-/dev/stdin}"
Try
read b < "${1:-/dev/stdin}"

Resources