I am trying to write script that monitors log file and on specific error it performs certain function.
I got the main code working however stuck with one problem.
Application create new log file when system date changes (log file name pattern is LOG.NODE1.DDMMYYYY), how do I get my code to automatically switch to new file that is created. Following is my script so far,
#!/bin/sh
logfile=$(ls -t $DIR"/env/log/LOG"* | head -n 1)
echo $logfile
tail -f $logfile | while read LOGLINE
do
if [[ "${LOGLINE}" == *";A database exception has occurred: FATAL DBERR: SQL_ERROR: ORA-00001: unique constraint (IX_TEST1) violated"* ]];
then
#Do something
fi
done
#!/bin/bash
# ^^^^ -- **NOT** /bin/sh
substring=";A database exception has occurred: FATAL DBERR: SQL_ERROR: ORA-00001: unique constraint (IX_TEST1) violated"
newest=
timeout=10 # number of seconds of no input after which to look for a newer file
# sets a global shell variable called "newest" when run
# to keep overhead down, this avoids invoking any external commands
find_newest() {
set -- "${DIR?The variable DIR is required}"/env/log/LOG*
[[ -e $1 || -L $1 ]] || return 1
newest=$1; shift
while (( $# )); do
[[ $1 -nt $newest ]] && newest=$1
shift
done
}
while :; do
find_newest # check for newer files
# if the newest file isn't the one we're already following...
if [[ $tailing_from_file != "$newest" ]]; then
exec < <(tail -f -- "$newest") # start a new copy of tail following the newer one
tailing_from_file=$newest # and record that file's name
fi
if read -t "$timeout" -r line && [[ $line = *"$substring"* ]]; then
echo "Do something here"
fi
done
Related
I want to create a utility function for bash to remove duplicate lines. I am using function
function remove_empty_lines() {
if ! command -v awk &> /dev/null
then
echo '[x] ERR: "awk" command not found'
return
fi
if [[ -z "$1" ]]
then
echo "usage: remove_empty_lines <file-name> [--replace]"
echo
echo "Arguments:"
echo -e "\t--replace\t (Optional) If not passed, the result will be redirected to stdout"
return
fi
if [[ ! -f "$1" ]]
then
echo "[x] ERR: \"$1\" file not found"
return
fi
echo $0
local CMD="awk '!seen[$0]++' $1"
if [[ "$2" = '--reload' ]]
then
CMD+=" > $1"
fi
echo $CMD
}
If I am running the main awk command directly, it is working. But when i execute the same $CMD in the function, I am getting this error
$ remove_empty_lines app.js
/bin/bash
awk '!x[/bin/bash]++' app.js
The original code is broken in several ways:
When used with --reload, it would truncate the output file's contents before awk could ever read those contents (see How can I use a file in a command and redirect output to the same file without truncating it?)
It didn't ever actually run the command, and for the reasons described in BashFAQ #50, storing a shell command in a string is inherently buggy (one can work around some of those issues with eval; BashFAQ #48 describes why doing so introduces security bugs).
It wrote error messages (and other "diagnostic content") to stdout instead of stderr; this means that if your function's output was redirected to a file, you could never see its errors -- they'd end up jumbled into the output.
Error cases were handled with a return even in cases where $? would be zero; this means that return itself would return a zero/successful/truthy status, not revealing to the caller that any error had taken place.
Presumably the reason you were storing your output in CMD was to be able to perform a redirection conditionally, but that can be done other ways: Below, we always create a file descriptor out_fd, but point it to either stdout (when called without --reload), or to a temporary file (if called with --reload); if-and-only-if awk succeeds, we then move the temporary file over the output file, thus replacing it as an atomic operation.
remove_empty_lines() {
local out_fd rc=0 tempfile=
command -v awk &>/dev/null || { echo '[x] ERR: "awk" command not found' >&2; return 1; }
if [[ -z "$1" ]]; then
printf '%b\n' >&2 \
'usage: remove_empty_lines <file-name> [--replace]' \
'' \
'Arguments:' \
'\t--replace\t(Optional) If not passed, the result will be redirected to stdout'
return 1
fi
[[ -f "$1" ]] || { echo "[x] ERR: \"$1\" file not found" >&2; return 1; }
if [ "$2" = --reload ]; then
tempfile=$(mktemp -t "$1.XXXXXX") || return
exec {out_fd}>"$tempfile" || { rc=$?; rm -f "$tempfile"; return "$rc"; }
else
exec {out_fd}>&1
fi
awk '!seen[$0]++' <"$1" >&$out_fd || { rc=$?; rm -f "$tempfile"; return "$rc"; }
exec {out_fd}>&- # close our file descriptor
if [[ $tempfile ]]; then
mv -- "$tempfile" "$1" || return
fi
}
First off the output from your function call is not an error but rather the output of two echo commands (echo $0 and echo $CMD).
And as Charles Duffy has pointed out ... at no point is the function actually running the $CMD.
As for the inclusion of /bin/bash in your function's echo output ... the main problem is the reference to $0; by definition $0 is the name of the running process, which in the case of a function is the shell under which the function is being called. Consider the following when run from a bash command prompt:
$ echo $0
-bash
As you can see from your output this generates /bin/bash in your environment. See this and this for more details.
On a related note, the reference to $0 within double quotes causes the $0 to be evaluated, so this:
local CMD="awk '!seen[$0]++' $1"
becomes
local CMD="awk '!seen[/bin/bash]++' app.js"
I'm thinking what you want is something like:
echo $1 # the name of the file to be processed
local CMD="awk '!seen[\$0]++' $1" # escape the '$' in '$0'
becomes
local CMD="awk '!seen[$0]++' app.js"
That should fix the issues shown in your function's output; as for the other issues ... you're getting a good bit of feedback in the various comments ...
I am a beginner and trying to write a script that takes a config file (example below) and sets the rights for the users, if that user or group doesn´t exist, they get added.
For every line in the file, I am cutting out the user or the group and check if they exist.
Right now I only check for users.
#!/bin/bash
function SetRights()
{
if [[ $# -eq 1 && -f $1 ]]
then
for line in $1
do
var1=$(cut -d: -f2 $line)
var2=$(cat /etc/passwd | grep $var1 | wc -l)
if [[ $var2 -eq 0 ]]
then
sudo useradd $var1
else
setfacl -m $line
fi
done
else
echo Enter the correct path of the configuration file.
fi
}
SetRights $1
The config file looks like this:
u:TestUser:- /home/temp
g:TestGroup:rw /home/temp/testFolder
u:TestUser2:r /home/temp/1234.txt
The output:
grep: TestGroup: No such file or directory
grep: TestUser: No such file or directory
"The useradd help menu"
If you could give me a hint what I should look for in my research, I would be very grateful.
Is it possible to reset var1 and var2? Using unset didn´t work for me and I couldn´t find variables could only be set once.
It's not clear how you are looping over the contents of the file -- if $1 contains the file name, you should not be seeing the errors you report.
But anyway, here is a refactored version which hopefully avoids your problems.
# Avoid Bash-only syntax for function definition
SetRights() {
# Indent function body
# Properly quote "$1"
if [[ $# -eq 1 && -f "$1" ]]
then
# Read lines in file
while read -r acl file
do
# Parse out user
user=${acl#*:}
user=${user%:*}
# Avoid useless use of cat
# Anchor regex correctly
if ! grep -q "^$user:" /etc/passwd
then
# Quote user
sudo useradd "$user"
else
setfacl -m "$acl" "$file"
fi
done <"$1"
else
# Error message to stderr
echo Enter the correct path of the configuration file. >&2
# Signal failure to the caller
return 1
fi
}
# Properly quote argument
SetRights "$1"
I have a tracked foo. Now, since I'm absent-minded, I've run:
mv foo bar
now, when I do hg st, I get:
! foo
? bar
I want to fix this retroactively - as though I'd done an hg mv foo bar.
Now, I could write a bash script which does that for me - but is there something better/simpler/smarter I could do?
Use the --after option: hg mv --after foo bar
$ hg mv --help
hg rename [OPTION]... SOURCE... DEST
aliases: move, mv
rename files; equivalent of copy + remove
Mark dest as copies of sources; mark sources for deletion. If dest is a
directory, copies are put in that directory. If dest is a file, there can
only be one source.
By default, this command copies the contents of files as they exist in the
working directory. If invoked with -A/--after, the operation is recorded,
but no copying is performed.
This command takes effect at the next commit. To undo a rename before
that, see 'hg revert'.
Returns 0 on success, 1 if errors are encountered.
options ([+] can be repeated):
-A --after record a rename that has already occurred
-f --force forcibly copy over an existing managed file
-I --include PATTERN [+] include names matching the given patterns
-X --exclude PATTERN [+] exclude names matching the given patterns
-n --dry-run do not perform actions, just print output
--mq operate on patch repository
(some details hidden, use --verbose to show complete help)
Here's what I'm doing right now;
#!/bin/bash
function die {
echo "$1" >&2
exit -1
}
(( $# == 2 )) || die "Usage: $0 <moved filename> <original filename>"
[[ -e "$1" ]] || die "Not an existing file: $1"
[[ ! -e "$2" ]] || die "Not a missing file: $2"
hg_st_lines_1=$(hg st "$1" 2>/dev/null | wc -l)
hg_st_lines_2=$(hg st "$2" 2>/dev/null | wc -l)
(( ${hg_st_lines_1} == 1 )) || die "Expected exactly one line in hg status for $1, but got ${hg_st_lines_1}"
(( ${hg_st_lines_2} == 1 )) || die "Expected exactly one line in hg status for $2, but got ${hg_st_lines_2}"
[[ "$(hg st "$1" 2>/dev/null)" == \?* ]] || die "Mercurial does not consider $1 to be an unknown (untracked) file"
[[ "$(hg st "$2" 2>/dev/null)" =~ !.* ]] || die "Mercurial does not consider $2 to be a missing file"
mv $1 $2
hg mv $2 $1
Some weeks ago I found in this site a very useful bash script that downloads images from google image results (download images from google with command line)
Although the script is quite complicate for me, I did some simple modifications so as not to rename the results so as to keep the original names.
However, since the last week, the script stopped working... probably Google updated the code or something, and the regexes of the script don't parse the results any more. I don't know enough about google's codes, web programing or regexing to see what is wrong, although I did some educated guesses, but still didn't work.
My (unworking) tweaked script is this
#! /bin/bash
# function to create all dirs til file can be made
function mkdirs {
file="$1"
dir="/"
# convert to full path
if [ "${file##/*}" ]; then
file="${PWD}/${file}"
fi
# dir name of following dir
next="${file#/}"
# while not filename
while [ "${next//[^\/]/}" ]; do
# create dir if doesn't exist
[ -d "${dir}" ] || mkdir "${dir}"
dir="${dir}/${next%%/*}"
next="${next#*/}"
done
# last directory to make
[ -d "${dir}" ] || mkdir "${dir}"
}
# get optional 'o' flag, this will open the image after download
getopts 'o' option
[[ $option = 'o' ]] && shift
# parse arguments
count=${1}
shift
query="$#"
[ -z "$query" ] && exit 1 # insufficient arguments
# set user agent, customize this by visiting http://whatsmyuseragent.com/
useragent='Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:31.0) Gecko/20100101 Firefox/31.0'
# construct google link
link="www.google.cz/search?q=${query}\&tbm=isch"
# fetch link for download
imagelink=$(wget -e robots=off --user-agent "$useragent" -qO - "$link" | sed 's/</\n</g' | grep '<a href.*\(png\|jpg\|jpeg\)' | sed 's/.*imgurl=\([^&]*\)\&.*/\1/' | head -n $count | tail -n1)
imagelink="${imagelink%\%*}"
# get file extention (.png, .jpg, .jpeg)
ext=$(echo $imagelink | sed "s/.*\(\.[^\.]*\)$/\1/")
# set default save location and file name change this!!
dir="$PWD"
file="google image"
# get optional second argument, which defines the file name or dir
if [[ $# -eq 2 ]]; then
if [ -d "$2" ]; then
dir="$2"
else
file="${2}"
mkdirs "${dir}"
dir=""
fi
fi
# construct image link: add 'echo "${google_image}"'
# after this line for debug output
google_image="${dir}/${file}"
# construct name, append number if file exists
if [[ -e "${google_image}${ext}" ]] ; then
i=0
while [[ -e "${google_image}(${i})${ext}" ]] ; do
((i++))
done
google_image="${google_image}(${i})${ext}"
else
google_image="${google_image}${ext}"
fi
# get actual picture and store in google_image.$ext
wget --max-redirect 0 -q "${imagelink}"
# if 'o' flag supplied: open image
[[ $option = "o" ]] && gnome-open "${google_image}"
# successful execution, exit code 0
exit 0
one way to invetigate : provide -x option to bash so to have the trace of your script; that is change /bin/bash to /bin/bash -x in your script -or- simply invoke your script with
bash -x <yourscript>
You can also annotate your script with echo commands to track some variables.
I'm trying to figure out a way to compare an existing file with the result of a process (a heavy one, not to be repeated) and clobber the existing file with the result of that process without having to write it in a temp file (it would be a large temp file, about the same size of the existing file: let's try to be efficient and not take twice space it should).
I would like to replace the normal file /tmp/replace_with_that (see below) with a fifo, but of course doing so with the code below would just lock up the script, since the /tmp/replace_with_that fifo cannot be read before comparing the existing file with the named pipe /tmp/test_against_this
#!/bin/bash
mkfifo /tmp/test_against_this
: > /tmp/replace_with_that
echo 'A B C D' >/some/existing/file
{
#A very heavy process not to repeat;
#Solved: we used a named pipe.
#Its large output should not be sent to a file
#To solve: using this code, we write the output to a regular file
for LETTER in "A B C D E"
do
echo $LETTER
done
} | tee /tmp/test_against_this /tmp/replace_with_that >/dev/null &
if cmp -s /some/existing/file /tmp/test_against_this
then
echo Exact copy
#Don't do a thing to /some/existing/file
else
echo Differs
#Clobber /some/existing/file with /tmp/replace_with_that
cat /tmp/replace_with_that >/some/existing/file
fi
rm -f /tmp/test_against_this
rm -f /tmp/replace_with_that
I think I would recommend a different approach:
Generate an MD5/SHA1/SHA256/whatever hash of the existing file
Run your heavy process and replace the output file
Generate a hash of the new file
If the hashes match, the files were the same; if not, the new file is different
Just for completeness, my answer (wanted to explore the use of pipes):
Was trying to find a way to compare on the fly a stream and an existing file, without overwriting the existing file unnecessarily (leaving it as is if stream and file are exact copies), and without creating sometimes big temp files (the product of a a heavy process like mysqldump for instance). The solution had to rely on pipes only (named and anonymous), and maybe a few very small temp files.
The checksum solution suggested by twalberg is just fine, but md5sum calls on large files are processor intensive (and processing time lengthens linearly with file size). cmp is faster.
Example call of the function listed below:
#!/bin/bash
mkfifo /tmp/fifo
mysqldump --skip-comments $HOST $USER $PASSWORD $DB >/tmp/fifo &
create_or_replace /some/existing/dump /tmp/fifo
#This also works, but depending on the anonymous fifo setup, seems less robust
create_or_replace /some/existing/dump <(mysqldump --skip-comments $HOST $USER $PASSWORD $DB)
The functions:
#!/bin/bash
checkdiff(){
local originalfilepath="$1"
local differs="$2"
local streamsize="$3"
local timeoutseconds="$4"
local originalfilesize=$(stat -c '%s' "$originalfilepath")
local starttime
local stoptime
#Hackish: we can't know for sure when the wc subprocess will have produced the streamsize file
starttime=$(date +%s)
stoptime=$(( $starttime + $timeoutseconds ))
while ([[ ! -f "$streamsize" ]] && (( $stoptime > $(date +%s) ))); do :; done;
if ([[ ! -f "$streamsize" ]] || (( $originalfilesize == $(cat "$streamsize" | head -1) )))
then
#Using streams that were exact copies of files to compare with,
#on average, with just a few test runs:
#diff slowest, md5sum 2% faster than diff, and cmp method 5% faster than md5sum
#Did not test, but on large unequal files,
#cmp method should be way ahead of the 2 other methods
#since equal files is the worst case scenario for cmp
#diff -q --speed-large-files <(sort "$originalfilepath") <(sort -) >"$differs"
#( [[ $(md5sum "$originalfilepath" | cut -b-32) = $(md5sum - | cut -b-32) ]] && : || echo -n '1' ) >"$differs"
( cmp -s "$originalfilepath" - && : || echo -n '1' ) >"$differs"
else
echo -n '1' >"$differs"
fi
}
create_or_replace(){
local originalfilepath="$1"
local newfilepath="$2" #Should be a pipe, but could be a regular file
local differs="$originalfilepath.differs"
local streamsize="$originalfilepath.size"
local timeoutseconds=30
local starttime
local stoptime
if [[ -f "$originalfilepath" ]]
then
#Cleanup
[[ -f "$differs" ]] && rm -f "$differs"
[[ -f "$streamsize" ]] && rm -f "$streamsize"
#cat the pipe, get its size, check for differences between the stream and the file and pipe the stream into the original file if all checks show a diff
cat "$newfilepath" |
tee >(wc -m - | cut -f1 -d' ' >"$streamsize") >(checkdiff "$originalfilepath" "$differs" "$streamsize" "$timeoutseconds") | {
#Hackish: we can't know for sure when the checkdiff subprocess will have produced the differs file
starttime=$(date +%s)
stoptime=$(( $starttime + $timeoutseconds ))
while ([[ ! -f "$differs" ]] && (( $stoptime > $(date +%s) ))); do :; done;
[[ ! -f "$differs" ]] || [[ ! -z $(cat "$differs" | head -1) ]] && cat - >"$originalfilepath"
}
#Cleanup
[[ -f "$differs" ]] && rm -f "$differs"
[[ -f "$streamsize" ]] && rm -f "$streamsize"
else
cat "$newfilepath" >"$originalfilepath"
fi
}