Check if a file has updated - bash

I am trying to check to see if a file (in this case /var/log/messages) has been updated using 'stat' command in a loop. However the code never exits the loop and moves on for some reason.
#!/bin/bash
check='/var/log/messages'
THEN="stat -c %z ${check}"
NOW="stat -c %z ${check}"
while [ $"NOW" == $"THEN" ]
do
echo "$NOW"
if [ $"NOW" != $"THEN" ]; then
echo "${check} has been updated."
if
done
Thoughts on this? Is there an easier way to see if /var/log/messages has changed?

The dollar signs need to be inside the quotes. $"..." is a special quoting mechanism for doing translations, so unless you are using a locale in which NOW and THEN translate to the same string, the condition will never be true.
if [ "$NOW" == "$THEN" ]; then

Firstly the version below in the least executes the stat command plus all the changes that are explained above. One thing that you would have to think about is the then and now would almost always be the same unless the messages is being updated in less then the time it takes to execute
#!/bin/bash
check='/var/log/messages'
THEN=`stat -c %z ${check}`
NOW=`stat -c %z ${check}`
if [ "$NOW" == "$THEN" ]; then
echo "$NOW"
elif [ "$NOW" != "$THEN" ]; then
echo "$check has been updated."
fi
~
Note the fact that the stat is being executed.

The piece of code provided by in the OP never finishes because of the following reason:
the variables NOW and THEN are never updated inside of the while-loop. As they never update, the loop continues to run as both values are identical.
the variables NOW and THEN represent the status change and not the modification change. The time of status-change is the time when the meta-data of the file has changed (permissions, groups, ...). The user might be more interested in looking at the content change (modification change)
the variables NOW and THEN actually are not the output of the stat command, but just the command itself since they are not executed using $(stat ...).
An update to the code would be:
#!/usr/bin/env bash
check_file="/var/log/messages"
THEN="$(stat -c "%Y" "${check_file}")"
NOW="$(stat -c "%Y" "${check_file}")"
while [ "$NOW" = "$THEN" ]; do
# Sleep a second to have some waiting time between checks
sleep 1
# Update now
NOW="$(stat -c "%Y" ${check_file}")"
done
echo "${check_file} has been modified"
This can be simplified to
#!/usr/bin/env bash
check_file="/var/log/messages"
THEN="$(stat -c "%Y" "${check_file}")"
while [ "$(stat -c "%Y" ${check_file}")" = "$THEN" ]; do
# Sleep a second to have some waiting time between checks
sleep 1
done
echo "${check_file} has been modified"
However, it might be easier to use bash-interals and check modification by comparing files-dates
#!/usr/bin/env bash
touch tmpfile
while [ tmpfile -nt /var/log/messages ]; do sleep 1; done
echo "/var/log/messages has been modified"
rm tmpfile
This method is unfortunately fairly impractical. If you want to monitor if a file has been updated, especially in case of log-file where lines are appended, you can just use tail. The command will print out the new lines which are appended the moment a file is updated.
$ tail -f -- /var/log/messages
If you don't want to monitor updates to log-files but just want to check if a file is updated, you can use inotify-wait:
$ inotifywait -e modify /var/log/messages

Related

How to detect a non-rolling log file and pattern match in a shell script which is using tail, while, read, and?

I am monitoring a log file and if PATTERN didn't appear in it within THRESHOLD seconds, the script should print "error", otherwise, it should print "clear". The script is working fine, but only if the log is rolling.
I've tried reading 'timeout' but didn't work.
log_file=/tmp/app.log
threshold=120
tail -Fn0 ${log_file} | \
while read line ; do
echo "${line}" | awk '/PATTERN/ { system("touch pattern.tmp") }'
code to calculate how long ago pattern.tmp touched and same is assigned to DIFF
if [ ${diff} -gt ${threshold} ]; then
echo "Error"
else
echo "Clear"
done
It is working as expected only when there is 'any' line printed in the app.log.
If the application got hung for any reason and the log stopped rolling, there won't be any output by the script.
Is there a way to detect the 'no output' of tail and do some command at that time?
It looks like the problem you're having is that the timing calculations inside your while loop never get a chance to run when read is blocking on input. In that case, you can pipe the tail output into a while true loop, inside of which you can do if read -t $timeout:
log_file=/tmp/app.log
threshold=120
timeout=10
tail -Fn0 "$log_file" | while true; do
if read -t $timeout line; then
echo "${line}" | awk '/PATTERN/ { system("touch pattern.tmp") }'
fi
# code to calculate how long ago pattern.tmp touched and same is assigned to diff
if [ ${diff} -gt ${threshold} ]; then
echo "Error"
else
echo "Clear"
fi
done
As Ed Morton pointed out, all caps variable names are not a good idea in bash scripts, so I used lowercase variable names.
How about something simple like:
sleep "$threshold"
grep -q 'PATTERN' "$log_file" && { echo "Clear"; exit; }
echo "Error"
If that's not all you need then edit your question to clarify your requirements. Don't use all upper case for non exported shell variable names btw - google it.
To build further on your idea, it might be beneficial to run the awk part in the background and a continuous loop to do the checking.
#!/usr/bin/env bash
log_file="log.txt"
# threshold in seconds
threshold=10
# run the following process in the background
stdbuf -oL tail -f0n "$log_file" \
| awk '/PATTERN/{system("touch "pattern.tmp") }' &
while true; do
match=$(find . -type f -iname "pattern.tmp" -newermt "-${threshold} seconds")
if [[ -z "${match}" ]]; then
echo "Error"
else
echo "Clear"
fi
done
This looks to me like a watchdog timer. I've implemented something like this by forcing a background process to update my log, so I don't have to worry about read -t. Here's a working example:
#!/usr/bin/env bash
threshold=10
grain=2
errorstate=0
while sleep "$grain"; do
date '+[%F %T] watchdog timer' >> log
done &
trap "kill -HUP $!" 0 HUP INT QUIT TRAP ABRT TERM
printf -v lastseen '%(%s)T'
tail -F log | while read line; do
printf -v now '%(%s)T'
if (( now - lastseen > threshold )); then
echo "ERROR"
errorstate=1
else
if (( errorstate )); then
echo "Recovered, yay"
errorstate=0
fi
fi
if [[ $line =~ .*PATTERN.* ]]; then
lastseen=$now
fi
done
Run this in one window, wait $threshold seconds for it to trigger, then in another window echo PATTERN >> log to see the recovery.
While this can be made as granular as you like (I've set it to 2 seconds in the example), it does pollute your log file.
Oh, and note that printf '%(%s)T' format requires bash version 4 or above.

Bash Logic Check - Repeating While Loop with nested "IF Then" statements

I'm writing a script to monitor my sip trunk and attempt to fix it. If it fails to fix the issue 6 times, then reboot the server. The script is called by cron via #reboot. I first had nested While Loops but that didn't work correctly so I switched to a never ending While Loop with two nested If Loops to perform the functions of the script.
I was wondering if somebody could take a quick look and see if the way I am attacking it makes sense and is logical approach.
Thank You,
Script as it stands:
#!/bin/bash
pwd="/srv/scripts"
count=0
echo "Script Started on $(date -u) Failure.Count=$count" >> "$pwd/failures.count"
start=start
while [ $start = "start" ]; do
sleep 420
var="$(asterisk -rx "pjsip show registrations" | grep -o Registered)"
if [ "$var" != "Registered" ]; then
amportal restart
count=$(( $count + 1 ))
echo "Trunk Failure on $(date -u) Failure.Count=$count" >> "$pwd/failures.count"
fi
if [ "$count" -gt 5 ]; then
echo "Server Reboot due to Failure.Count=$count on $(date -u)" >> "$pwd/reboot.notification"
reboot
fi
done
There is no need to use a variable in the while loop, or to capture the grep output into a variable.
#!/bin/bash
pwd="/srv/scripts"
count=0
echo "Script Started on $(date -u) Failure.Count=$count" >> "$pwd/failures.count"
# No need for a variable here
while true; do
# Fix indentation
sleep 420
# Again, no need for a variable; use grep -q
if ! asterisk -rx "pjsip show registrations" | grep -q Registered
then
amportal restart
count=$(( $count + 1 ))
echo "Trunk Failure on $(date -u) Failure.Count=$count" >> "$pwd/failures.count"
fi
if [ "$count" -gt 5 ]; then
echo "Server Reboot due to Failure.Count=$count on $(date -u)" >> "$pwd/reboot.notification"
reboot
fi
done
I would perhaps also collect all the log notices in a single log file, and use a more traditional log format with a time stamp and the script's name bofore each message.
Should the counter reset to zero if you see a success? Having the server reboot because you disconnected the network cable at the wrong time seems like something you'd want to avoid.

"Argument list too long" for every command [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 8 years ago.
Improve this question
Occasionally, when I have a program that generates large arrays I get this bug where every command throws the error
"Argument list too long"
even if I just type:
$ cp
-bash: /bin/cp: Argument list too long
$
I can't use ls, or even open a new file with vim:
$ vim test.txt
-bash: /usr/bin/vim: Argument list too long
$
I tried using "wait" for all bg processes to finish, but no change. It seems to happen inconsistently, but when it does, the only fix is to restart the shell.
Any ideas what might be going on?
Update: I did some further testing and i got the error to be repeatable. It happens when a recursively defined array reaches 85 elements in length. The first command which throws the error is a bc that doesnt even depend on the array! and then from there on out, almost every other command throws the same error.
Update: The program I'm using has many bash scripts working together, but I've determined the problem always arises in this one:
function MPMDrun_prop()
{
PARDIR=$1
COMPDIR=$2
runSTR=$3
NUMNODES=$4
ForceRun=$5
if [ $# -le 3 ] ; then
echo "USAGE: MPMDrun_prop \$PARDIR \$COMPDIR \$runSTR \$NUMNODES \$ForceRun"
fi
echo "in MPMDrun_Prop"
. $PARDIR/ParameterScan.inp
. $MCTDHBDIR/Scripts/get_NumberOfJobs.sh
if [ "$MPMD" != "T" ]; then
MPMDnodes=1
fi
## If no runscripts in the $PARDIR, copy one and strip of the line which runs the program
if [ -z "$(ls $PARDIR/run*.sh 2> /dev/null)" ] ; then
if [ "$forhost" == "maia" ]; then
cp $MCTDHBDIR/../PBS_Scripts/run-example-maia.sh $PARDIR/run.tmp
sed 's|mpirun.*||' < $PARDIR/run.tmp > $PARDIR/run.sh
jobtime=86400
elif [ "$forhost" == "hermit" ]; then
cp $MCTDHBDIR/../PBS_Scripts/run-example-hermit.sh $PARDIR/run.tmp
sed 's|aprun.*||' < $PARDIR/run.tmp > $PARDIR/run.sh
jobtime=86400
elif [ "$forhost" == "hornet" ]; then
cp $MCTDHBDIR/../PBS_Scripts/run-example-hornet.sh $PARDIR/run.tmp
sed 's|aprun.*||' < $PARDIR/run.tmp > $PARDIR/run.sh
jobtime=86400
elif [ "$forhost" == "bwgrid" ]; then
cp $MCTDHBDIR/../PBS_Scripts/run-example-BWGRID.sh $PARDIR/run.tmp
sed 's|mpirun.*||' < $PARDIR/run.tmp > $PARDIR/run.sh
jobtime=86400
fi
sed 's|nodes=[0-9]*|nodes=0|' < $PARDIR/run.sh > $PARDIR/run.tmp
sed 's|#PBS -N.*|#PBS -N MONSTER_'$MonsterName'|' < $PARDIR/run.tmp > $PARDIR/run.sh_
rm $PARDIR/run.sh
rm $PARDIR/run.tmp
chmod 755 $PARDIR/run.sh_
echo ". $MCTDHBDIR/Scripts/RunFlagSleeper.sh" >> $PARDIR/run.sh_
## Include check_convergence.sh for mixed relax/prop compatibility
echo ". $MCTDHBDIR/Scripts/check_convergence.sh" >> $PARDIR/run.sh_
echo "RunFlagSleeper $jobtime " >> $PARDIR/run.sh_
echo "(" >> $PARDIR/run.sh_
cp $PARDIR/run.sh_ $PARDIR/run1.sh
fi
### Add $runSTR to the most recent runscript
### find runscript$N.sh (run1.sh, run 2.sh, etc) that has numnodes less than $MPMDnodes
for qq in $(ls $PARDIR/run[0-9]*.sh | sort -g ); do
NodesInRun=$(cat $qq | grep -o "nodes *= *[0-9]*" | grep -o "[0-9]*")
if [ "$NodesInRun" -lt "$MPMDnodes" ]; then
## The number of nodes already specified in the runscript doesnt exceed the maximum, so add on another job
NewNodes=$(echo "$NodesInRun+$NUMNODES" | bc)
## Start each aprun command in its own subshell
## wait for 24 hrs after aprun, to guarantee that no subshell finishes before the job is done
sed 's|nodes=[0-9]*|nodes='$NewNodes'|' < $qq > $qq-1
sed 's|\(RunFlagSleeper .*\)|\1 '$COMPDIR'|' <$qq-1 >$qq
rm $qq-1
echo " (" >> $qq
## Sleeps for $jobtime - 5 mins, then removes runflag. in case aprun doesnt finish in $jobtime
echo " cd $COMPDIR" >> $qq
echo " $runSTR" >> $qq
## remove runflag after aprun command has finished
echo " rm $COMPDIR/RunFlag" >> $qq
# echo "sleep $jobtime" >> $qq-1
echo " ) &" >> $qq
# mv $qq-1 $qq
## put a flag in the computation directory so it isnt computed multiple times
touch $COMPDIR/RunFlag
if [[ "$NewNodes" -ge "$MPMDnodes" || "$ForceRun" == "T" ]]; then
## This last process made the nodecount exceed the maximum, or there is a ForceRun flag passed
## So now, exceute the runscript and start another
echo " wait" >> $qq
echo ") &" >> $qq
echo "PID=\$!" >> $qq
echo "wait \$PID" >> $qq
## Ensure the queue has room for the next job, if not, wait for it
Njobs=$(get_NumberOfJobs $runhost)
while [ "$Njobs" -ge "$maxjobs" ]; do
echo "Njobs=$Njobs and maxjobs=$maxjobs"
echo "Waiting 30 minutes for que to clear"
sleep 1800
done
echo "qsub $qq"
# qsub $qq
RunCount=$(echo $qq | grep -o 'run[0-9]*.sh' | grep -o '[0-9]*')
let "RunCount++"
cp $PARDIR/run.sh_ $PARDIR/run$RunCount.sh
fi
fi
done
}
The error typically starts at the 80-90'th call of this function at the first cp or bc. I've commented ALL array manipulations, so there is zero chance this is caused by the array being too large. The environment stays at ~100-200 Kb so that isn't the problem either.
That error message is a bit misleading. It should say something like "Argument list and environment use too much space".
The environment consists of all the environment variables you have exported, plus the environment your shell was started with. Normally, the environment should only be a few kilobytes, but there is nothing stopping you from exporting a million-byte string, and if you do that, you'll use up all the space allowed.
It's not totally obvious how much space the system allows for arguments + environment. You should be able to query the limit with getconf ARG_MAX, and with Gnu xargs you can get more information from xargs --show-limits </dev/null (in both cases, assuming you haven't exceeded the limit :) ), but sometimes the actual space available will turn out to be less than what is indicated.
In any event, it's not a good idea to try to stuff megabytes into the environment. If you're tempted to do that, put the data in a temporary file instead, and just export the name of the file.
Since you stated that when you have a program that generates large arrays you get this bug where every command throws the error "Argument list too long". So, I presume that last command you executed is causing problem for next command. My suggestion is that don't use large argument list for any command. This could cause an overflow in the environment causing problems even for next command. Instead of large arg list, use a file having list of data and use the file redirected for input as in:
command < inputfile

Shell script to poll a directory and stop upon an event

Need shell script to:
1/keep polling a directory "receive_dir" irrespective of having files or no files in it.
2/move the files over to another directory "send_dir".
3/the script should only stop polling upon a file "stopfile" get moved to "receive_dir". Thanks !!
My script:
until [ $i = stopfile ]
do
for i in `ls receive_dir`; do
time=$(date +%m-%d-%Y-%H:%M:%S)
echo $time
mv receive_dir/$i send_dir/;
done
done
This fails on empty directories and also is there any better way ?
If you are running on Linux, you might wish to consider inotifywait
$ declare -f tillStopfile
tillStopfile ()
{
cd receive_dir
[[ -d ../send_dir ]] || mkdir ../send_dir
while true; do
date +%m-%d-%Y-%H:%M:%S
for f in *
do
mv "$f" ../send_dir
[[ $f == "stopfile" ]] && break 2
done
sleep 3
done
}
$
Improvements
while true ... break #
easier to control this loop
cd receive_dir #
why not run in the "receive_dir"
factor date out of inner loop #
unless you need to see each time-stamp?
added suggested "sleep"
# pick a suitable inteval
Run:
$ tillStopfile 2>/dev/null # suppresses ls error messages

Loop script until file has equal size for a minute

I have cronjob to run a script every day in specific time. The script is for conversion a large file (about 2GB) in specific folder. The problem is that not every day my coleague put the file in the folder before the time, written as cronjob.
Please help me to add commands in the script or to write second script for:
Check if the file exists in the folder.
If the previous action is true, check the file size every minute. (I would like to avoid conversion of still incomming large file).
If filesize stays unchanged for 2 minutes, start the script for conversion.
I give you the important lines of the script so far:
cd /path-to-folder
for $i in *.mpg; do avconv -i "$i" "out-$i.mp4" ; done
10x for the help!
NEW CODE AFTER COMMENTS:
There is file in the folder!
#! /bin/bash
cdate=$(date +%Y%m%d)
dump="/path/folder1"
base=$(ls "$dump")
if [ -n "$file"]
then
file="$dump/$base"
size=$(stat -c '%s' "$file")
count=0
while sleep 10
do
size0=$(stat -c '%s' "$file")
if [ $size=$size0 ]
then $((count++))
count=0
fi
if [ $count = 2 ]
then break
fi
done
# file has been stable for two minutes. Start conversion.
CONVERSION CODE
fi
MESSAGE IN TERMINAL: Maybe error???
script.sh: 17: script.sh: arithmetic expression: expecting primary: "count++"
file=/work/daily/dump/name_of_dump_file
if [ -f "$file" ]
then
# size=$(ls -l "$file" | awk '{print $5}')
size=$(stat -c '%s' "$file")
count=0
while sleep 60
do
size0=$(stat -c '%s' "$file")
if [ $size = $size0 ]
then : $((count++))
else size=$size0
count=0
fi
if [ $count = 2 ]
then break
fi
done
# File has been stable for 2 minutes — start conversion
fi
Given the slightly revised requirements (described in the comments), and assuming that the file names do not contain spaces or newlines or other similarly awkward characters, then you can use:
dump="/work/daily/dump" # folder 1
base=$(ls "$dump")
if [ -n "$file" ]
then
file="$dump/$base"
...code as before...
# File has been stable for 2 minutes - start conversion
dir2="/work/daily/conversion" # folder 2
file2="$dir2/$(basename $base .mpg).xyz"
convert -i "$file" -o "$file2"
mv "$file" "/work/daily/originals" # folder 3
ncftpput other.coast.example.com /work/daily/input "$file2"
mv "$file2" "/work/daily/converted" # folder 4
fi
If there's nothing in the folder, the process exits. If you want it to wait until there is a file to convert, then you need a loop around the file test:
while file=$(ls "$dump")
[ -z "$file" ]
do sleep 60
done
This uses a little-known feature of shell loops; you can stack the commands in the control, but it is the exit status of the last one that controls the loop.
Well, I finally made some working code as follows:
#!/bin/bash
cdate=$(date +%Y%m%d)
folder1="/path-to-folder1"
cd $folder1
while file=$(ls "$folder1")
[ -z "$file" ]
do sleep 5 && echo "There in no file in the folder at $cdate."
done
echo "There is a file in folder at $cdate"
size1=$(stat -c '%s' "$file")
echo "The size1 is $size1 at $cdate"
size2=$(stat -c '%s' "$file")
echo "The size2 is $size2 at $cdate"
if [ $size1 = $size2 ]
then
echo "file is stable at $cdate. Do conversion."
Is the next line the right one to loop the same script???
else sh /home/user/bin/exist-stable.sh
fi
The right code after comments below is
else exec /home/user/bin/exist-stable.sh
fi

Resources