I'm trying to do a simple script that if the "grep" comes with results, send the email with the results. But I want also to send an email if there is a case when there are no rejects
#! /bin/bash
FILE=$(find . /opt/FIXLOGS/l51prdsrv* -iname "TRADX_*.log" -type f -exec grep -F 103=16 {} /dev/null \; )>> Rejects.txt
if [ "$FILE" == true ]
then
mailx -s "Logs $(date +%d-%m-%y)" "email" < Rejects.txt
rm -f Rejects.txt
elif [ "$FILE" == null ]
then
echo "No Rejects" >> Rejects.txt
mailx -s "Logs $(date +%d-%m-%y)" "email" < Rejects.txt
rm -f Rejects.txt
fi
In bash, everything is pretty much just a string. null is a string, not a null reference like in other languages. true is also a string (unless it's the command true, but that's not the case in your comparison).
If you want to test that a file exists, you'd use [[ -f "$FILE" ]]. However, the file is going to exist in your case even if grep matches nothing because bash automatically creates the file when you set it as the destination for your output. What you really need is -s which tests if the file exists and has size greater than 0.
#!/bin/bash
find . /opt/FIXLOGS/l51prdsrv* -iname "TRADX_*.log" -type f -exec grep -F 103=16 {} /dev/null \; >> Rejects.txt
if [[ -s Rejects.txt ]] ; then
: # grep wrote stuff into the file
else
: # file is empty so grep found nothing
fi
Related
I am i need to find & remove empty files. The definition of empty files in my use case is a file which has zero lines.
I did try testing the file to see if it's empty However, this behaves strangely as in even though the file is empty it doesn't detect it so.
Hence, the best thing I could write up is the below script which i way too slow given it has to test several hundred thousand files
#!/bin/bash
LOOKUP_DIR="/path/to/source/directory"
cd ${LOOKUP_DIR} || { echo "cd failed"; exit 0; }
for fname in $(realpath */*)
do
if [[ $(wc -l "${fname}" | awk '{print $1}') -eq 0 ]]
then
echo "${fname}" is empty
rm -f "${fname}"
fi
done
Is there a better way to do what I'm after or alternatively, can the above logic be re-written in a way that brings better performance please?
Your script is slow beacuse wc reads every file to the end, which is not needed for your purpose. This might be what you're looking for:
#!/bin/bash
lookup_dir='/path/to/source/directory'
cd "$lookup_dir" || exit
for file in *; do
if [[ -f "$file" && -r "$file" && ! -L "$file" ]]; then
read < "$file" || echo rm -f -- "$file"
fi
done
Drop the echo after making sure it works as intended.
Another version, calling the rm only once, could be:
#!/bin/bash
lookup_dir='/path/to/source/directory'
cd "$lookup_dir" || exit
for file in *; do
if [[ -f "$file" && -r "$file" && ! -L "$file" ]]; then
read < "$file" || files_to_be_deleted+=("$file")
fi
done
rm -f -- "${files_to_be_deleted[#]}"
Explanation:
The core logic is in the line
read < "$file" || rm -f -- "$file"
The read < "$file" command attempts to read a line from the $file. If it succeeds, that is, a line is read, then the rm command on the right-hand side of the || won't be executed (that's how the || works). If it fails then the rm command will be executed. In any case, at most one line will be read. This has great advantage over the wc command because wc would read the whole file.
if ! read < "$file"; then rm -f -- "$file"; fi
could be used instead. The two lines are equivalent.
To check a "$fname" is a file and is empty or not, use [ -s "$fname" ]:
#!/usr/bin/env sh
LOOKUP_DIR="/path/to/source/directory"
for fname in "$LOOKUP_DIR"*/*; do
if ! [ -s "$fname" ]; then
echo "${fname}" is empty
# remove echo when output is what you want
echo rm -f "${fname}"
fi
done
See: help test:
File operators:
...
-s FILE True if file exists and is not empty.
Yet another method
wc -l ~/tmp/* 2>/dev/null | awk '$1 == 0 {print $2}' | xargs echo rm
This will break if any of your files have whitespace in the name.
To work around that, with awk still
wc -l ~/tmp/* 2>/dev/null \
| awk 'sub(/^[[:blank:]]+0[[:blank:]]+/, "")' \
| xargs echo rm
This works because the sub function returns the number of substitutions made, which can be treated as a boolean zero/not-zero condition.
Remove the echo to actually delete the files.
I currently use a bash script and PDFgrep to rename files to a certain structure. However, in order to stop overriding if the new file has a duplicate name, I want to add a number at the end of the name. Keep in mind that there may be 3 or 4 duplicate names. What's the best way to do this?
#!/bin/bash
if [ $# -ne 1 ]; then
echo Usage: Renamer file
exit 1
fi
f="$1"
id1=$(pdfgrep -m 1 -i "MR# : " "$f" | grep -oE "[M][0-9][0-9]+") || continue
id2=$(pdfgrep -m 1 -i "Visit#" "$f" | grep -oE "[V][0-9][0-9]+") || continue
{ read today; read dob; read dop; } < <(pdfgrep -i " " "$f" | grep -oE "[0-9][0-9]/[0-9][0-9]/[0-9][0-9][0-9][0-9]")
dobsi=$(echo $dob | sed -e 's/\//-/g')
dopsi=$(echo $dop | sed -e 's/\//-/g')
mv -- "$f" "${id1}_${id2}_$(printf "$dobsi")_$(printf "$dopsi")_1.pdf"
Use a loop that checks if the destination filename exists, and increments a counter if it does. Replace the mv line with this:
prefix="${id1}_{id2}_${dob}_${dop}"
counter=0
while true
do
if [ "$counter" -ne 0 ]
then target="${prefix}_${counter}.pdf"
else target="${prefix}.pdf"
fi
if [ ! -e "$target" ]
then
mv -- "$f" "$target"
break
fi
((counter++))
done
Note that this suffers from a TOCTTOU problem, if the duplicate file is created between the ! -f "$target" test and the mv. I thought it would be possible to replace the existence check with using mv -n; but while this won't overwrite the file, it still treats the mv as successful, so you can't test the result to see if you need to increment the counter.
I have a script that uses nested if-else statements to search for files. I want the script to exit once the conditions are met for any one of the nested statement.
But the script still continues to run through the all the remaining if-else statements.
I have tested using exit 0 and return 0 but neither works.
Here's the script:
#!/bin/sh
PATH1=/filer1_vol1_dir1
PATH2=/filer2_vol1_dir1
PATH3=/filer3_vol1_dir2
PATTERN=fruits
find $PATH1 -type f -name "*$PATTERN*" -exec ls -l {} \; >> /tmp/${PATTERN}_search
if [[ -s /tmp/${PATTERN}_search && `grep -i apples /tmp/${PATTERN}_search` ]]
then
echo "Matching files have been found under $PATH1"
cat /tmp/${PATTERN}_search
return 0
else
echo "No matching files, proceeding to search $PATH2"
find $PATH2 -type f -name "*$PATTERN*" -exec ls -l {} \; >> /tmp/${PATTERN}_search
if [[ -s /tmp/${PATTERN}_search && `grep -i apples /tmp/${PATTERN}_search` ]]
then
echo "Matching files have been found under $PATH2"
cat /tmp/${PATTERN}_search
return 0
else
echo "No matching files, proceeding to search $PATH3"
find $PATH3 -type f -name "*$PATTERN*" -exec ls -l {} \; >> /tmp/${PATTERN}_search
if [[ -s /tmp/${PATTERN}_search && `grep -i apples /tmp/${PATTERN}_search` ]]
then
echo "Matching files have been found under $PATH3"
cat /tmp/${PATTERN}_search
return 0
else
echo "No file matches, please search elsewhere"
return 0
fi
fi
fi
exit 0
I have found that the better way is to use a while loop to iterate through each search. Within each iteration, an if-else condition will test if files matching the find pattern are found. Once this condition is true, a break statement was able to stop the script.
Sample script below:
#!/bin/sh
PATH1=/filer1_vol1_dir1
PATH2=/filer2_vol1_dir1
PATH3=/filer3_vol1_dir2
PATTERN=fruits
echo $PATH1 > /tmp/PATH.list
echo $PATH2 >> /tmp/PATH.list
echo $PATH3 >> /tmp/PATH.list
echo /tmp/PATH.list contains
cat /tmp/PATH.list
echo
cat /dev/null > /tmp/${PATTERN}_search.list
while read PATH
do
echo "Searching under the following parameters"
echo PATTERN = $PATTERN
echo PATH = $PATH
echo
/usr/bin/find $PATH -type f -name "*$PATTERN*" -exec ls -l {} \; >> /tmp/${PATTERN}_search.list
/usr/bin/grep -i apples /tmp/${PATTERN}_search.list
if [ $? -eq 0 ]
then
echo "All matching files have been found."
break
else
echo "No matches found, continuing search in next directory."
fi
done < /tmp/PATH.list
exit 0
#!/usr/bin/env bash
FILETYPES=( "*.html" "*.css" "*.js" "*.xml" "*.json" )
DIRECTORIES=`pwd`
MIN_SIZE=1024
for currentdir in $DIRECTORIES
do
for i in "${FILETYPES[#]}"
do
find $currentdir -iname "$i" -exec bash -c 'PLAINFILE={};GZIPPEDFILE={}.gz; \
if [ -e $GZIPPEDFILE ]; \
then if [ `stat --printf=%Y $PLAINFILE` -gt `stat --printf=%Y $GZIPPEDFILE` ]; \
then gzip -k -4 -f -c $PLAINFILE > $GZIPPEDFILE; \
fi; \
elif [ `stat --printf=%s $PLAINFILE` -gt $MIN_SIZE ]; \
then gzip -k -4 -c $PLAINFILE > $GZIPPEDFILE; \
fi' \;
done
done
This script compresses all web static files using gzip. When I try to run it, I get this error bash: line 5: [: 93107: unary operator expected. What is going wrong in this script?
You need to export the MIN_SIZE variable. The bash you are having find spawn doesn't have a value for it so the script runs (as I just mentioned in my comment on #ooga's answer) [ $result_from_stat -gt ] which is an error and (when the result is 93107) gets you [ 93107 -gt ] which (if you run that in your shell) gets you output of:
$ [ 93107 -gt ]
-bash: [: 93107: unary operator expected
This could be simpler:
#!/usr/bin/env bash
FILETYPES=(html css js xml json)
DIRECTORIES=("$PWD")
MIN_SIZE=1024
IFS='|' eval 'FILTER="^.*[.](${FILETYPES[*]})\$"'
for DIR in "${DIRECTORIES[#]}"; do
while IFS= read -ru 4 FILE; do
GZ_FILE=$FILE.gz
if [[ -e $GZ_FILE ]]; then
[[ $GZ_FILE -ot "$FILE" ]] && gzip -k -4 -c "$FILE" > "$GZ_FILE"
elif [[ $(exec stat -c '%s' "$FILE") -ge MIN_SIZE ]]; then
gzip -k -4 -c "$FILE" > "$GZ_FILE"
fi
done 4< <(exec find "$DIR" -mindepth 1 -type f -regextype egrep -iregex "$FILTER")
done
There's no need to use pwd. You can just have $PWD. And probably what you needed was an array variable as well.
Instead of calling bash multiple times as an argument to find with static string commands, just read input from a pipe or better yet from a named pipe through process substitution.
Instead of comparing stats, you can just use -ot or -nt.
You don't need -f if you're writing the output through redirection (>) as that form of redirection overwrites the target by default.
You can just call find against multiple files once by making a pattern as it's more efficient. You can check how I made the filter and used -iregex. Probably doing \( -iname one_ext_pat -or -iname another_ext_pat \) can also be applicable but it's more difficult.
exec is optional to prevent unnecessary use of another process.
Always prefer [[ ]] over [ ].
4< opens input with file descriptor 4 and -u 4 makes read read from that file descriptor, not stdin (0).
What you probably need is -ge MIN_SIZE (greater than or equal) not -gt.
Come to think of it, readarray is a cleaner option if your bash is version 4.0 or newer:
for DIR in "${DIRECTORIES[#]}"; do
readarray -t FILES < <(exec find "$DIR" -mindepth 1 -type f -regextype egrep -iregex "$FILTER")
for FILE in "${FILES[#]}"; do
...
done
done
I need to write a bash script that will iterate through the contents of a directory (including subdirectories) and perform the following replacements:
replace 'foo' in any file names with 'bar'
replace 'foo' in the contents of any files with 'bar'
So far all I've got is
find . -name '*' -exec {} \;
:-)
With RH rename:
find -f \( -exec sed -i s/foo/bar/g \; , -name \*foo\* -exec rename foo bar {} \; \)
find "$#" -depth -exec sed -i -e s/foo/bar/g {} \; , -name '*foo*' -print0 |
while read -d '' file; do
base=$(basename "$file")
mv "$file" "$(dirname "$file")/${base//foo/bar}"
done
UPDATED: 1632 EST
Now handles whitespace but 'while read item' never terminates. Better,
but still not right. Will keep
working on this.
aj#mmdev0:~/foo_to_bar$ cat script.sh
#!/bin/bash
dirty=true
while ${dirty}
do
find ./ -name "*" |sed -s 's/ /\ /g'|while read item
do
if [[ ${item} == "./script.sh" ]]
then
continue
fi
echo "working on: ${item}"
if [[ ${item} == *foo* ]]
then
rename 's/foo/bar/' "${item}"
dirty=true
break
fi
if [[ ! -d ${item} ]]
then
cat "${item}" |sed -e 's/foo/bar/g' > "${item}".sed; mv "${item}".sed "${item}"
fi
dirty=false
done
done
#!/bin/bash
function RecurseDirs
{
oldIFS=$IFS
IFS=$'\n'
for f in *
do
if [[ -f "${f}" ]]; then
newf=`echo "${f}" | sed -e 's/foo/bar/g'`
sed -e 's/foo/bar/g' < "${f}" > "${newf}"
fi
if [[ -d "${f}" && "${f}" != '.' && "${f}" != '..' && ! -L "${f}" ]]; then
cd "${f}"
RecurseDirs .
cd ..
fi
done
IFS=$oldIFS
}
RecurseDirs .
bash 4.0
#!/bin/bash
shopt -s globstar
path="/path"
cd $path
for file in **
do
if [ -d "$file" ] && [[ "$file" =~ ".*foo.*" ]];then
echo mv "$file" "${file//foo/bar}"
elif [ -f "$file" ];then
while read -r line
do
case "$line" in
*foo*) line="${line//foo/bar}";;
esac
echo "$line"
done < "$file" > temp
echo mv temp "$file"
fi
done
remove the 'echo' to commit changes
for f in `tree -fi | grep foo`; do sed -i -e 's/foo/bar/g' $f ; done
Yet another find-exec solution:
find . -type f -exec bash -c '
path="{}";
dirName="${path%/*}";
baseName="${path##*/}";
nbaseName="${baseName/foo/bar}";
#nbaseName="${baseName//foo/bar}";
# cf. http://www.bash-hackers.org/wiki/doku.php?id=howto:edit-ed
ed -s "${path}" <<< $'H\ng/foo/s/foo/bar/g\nwq';
#sed -i "" -e 's/foo/bar/g' "${path}"; # alternative for large files
exec mv -iv "{}" "${dirName}/${nbaseName}"
' \;
correction to find-exec approach by gregb (adding quotes):
# compare
bash -c '
echo $'a\nb\nc'
'
bash -c '
echo $'"'a\nb\nc'"'
'
# therefore we need
find . -type f -exec bash -c '
...
ed -s "${path}" <<< $'"'H\ng/foo/s/foo/bar/g\nwq'"';
...
' \;