Shell script - Log file not updating - bash

I'm trying to get my shell script to alert me if the 2 log files is not updating. I am trying to get the current size... wait 10 seconds and if the file did not change size send an email alert. any help would be great!
My Code:
#! /bin/bash
logFiles="lg.log lg2.log"
logLocation="/user/file/logs"
sleepTime=10
subject="issue - file not updated"
failMessage="::no recent updates "
successMessage="OK"
not_updated_time=0
arr=($logFiles)
arrlen=${#arr[#]}
arrcount=()
date
for ((count=0; count<arrlen; count++)) ; do
arrcount[$count]=`ls -l $logLocation${arr[$count]} |awk '{print $5}'`
echo "${arr[$count]} Original size :: ${arrcount[$count]}"
done
echo
while [ "e" == "e" ] ; do
sleep $sleepTime
date
for ((count=0; count<arrlen; count++)) ; do
nc=`ls -l $logLocation${arr[$count]} |awk '{print $5}'`
echo -n "${arr[$count]} "
if [ $nc == ${arrcount[$count]} ] ; then
echo 'error' | mailx -s "issue" cat#gmail.com
else
arrcount[$count]=$nc
echo $successMessage
not_updated_time=0
fi
done
echo
done
Fixed the issue with comma, script still not working. any ideas?

There appears to be a / missing either on the initialization of logLocation or when you're using that variable as $logLocation${arr[$count]}.
Try one of these two changes:
logLocation="/user/file/logs/"
Or:
`ls -l $logLocation/${arr[$count]} |awk '{print $5}'`

Related

update value in bash script

i have stuck code. i make bash script to send Notification If file Size changed
#!/bin/bash
#File Embeded
test_file=/home/optimus/bot_test/dump.txt
msg_caption=/tmp/telegram_msg_caption.txt
#initialize
initCount=0
#checkLog
#cmd="ls /var/log/snort/* | wc -l"
cmd=$(wc -c "dump.txt" | awk '{print $1}')
#Chat ID and TOKEN Telegram
chat_id="xxxx"
token="xxx"
#Send Alert
function sendAlert
{
curl -s -F chat_id=$chat_id -F caption="$caption" -F document=#$test_file https://api.telegram.org/bot$token/sendDocument > /dev/null 2&>1
}
#Monitoring DoS Server
while true
do
#GetLastCount
echo "Start Execute"
lastCount=$cmd
echo before_last $lastCount #ex 100 #after reset 0
echo before_init $initCount #ex 0
echo "--------------------"
if(($(($lastCount)) > $initCount));
then
echo "Running Script..." #debug
echo -e "Halo Admin\nTerjadi Indikasi Penyerangan DoS!!!" > $msg_caption #set Caption / Pesan
caption=$(<$msg_caption) #set Caption
sendAlert #Panggil Fungsi
#error disini
initCount=$lastCount
lastCount=0 #reset
echo after_last $lastCount
echo after_Init $initCount
echo "==================="
rm -f $msg_caption
sleep 5 #delay Proses jika ada indikasi
fi
sleep 5 #delay proses jika tidak ada indikasi
done
when code first running, send First Notification, and if file size changed , value in code not updated
example i use dump.txt to object size changed
how to get Last Size from dump.txt after
echo "Start Execute"
conclusion every 5 second, lastCount get last size File updated
The problem in your code is that the cmd=$(wc -c "dump.txt" | awk '{print $1}') line executes only once.
Instead, you should use:
#....
do
#...
lastCount=$(wc -c "dump.txt" | awk '{print $1}')
#...
Or make the cmd a function or store in a string and evaluate the string. The way you wrote, it is only evaluated in the beginning of your script and you use the same value afterward.

How to detect a non-rolling log file and pattern match in a shell script which is using tail, while, read, and?

I am monitoring a log file and if PATTERN didn't appear in it within THRESHOLD seconds, the script should print "error", otherwise, it should print "clear". The script is working fine, but only if the log is rolling.
I've tried reading 'timeout' but didn't work.
log_file=/tmp/app.log
threshold=120
tail -Fn0 ${log_file} | \
while read line ; do
echo "${line}" | awk '/PATTERN/ { system("touch pattern.tmp") }'
code to calculate how long ago pattern.tmp touched and same is assigned to DIFF
if [ ${diff} -gt ${threshold} ]; then
echo "Error"
else
echo "Clear"
done
It is working as expected only when there is 'any' line printed in the app.log.
If the application got hung for any reason and the log stopped rolling, there won't be any output by the script.
Is there a way to detect the 'no output' of tail and do some command at that time?
It looks like the problem you're having is that the timing calculations inside your while loop never get a chance to run when read is blocking on input. In that case, you can pipe the tail output into a while true loop, inside of which you can do if read -t $timeout:
log_file=/tmp/app.log
threshold=120
timeout=10
tail -Fn0 "$log_file" | while true; do
if read -t $timeout line; then
echo "${line}" | awk '/PATTERN/ { system("touch pattern.tmp") }'
fi
# code to calculate how long ago pattern.tmp touched and same is assigned to diff
if [ ${diff} -gt ${threshold} ]; then
echo "Error"
else
echo "Clear"
fi
done
As Ed Morton pointed out, all caps variable names are not a good idea in bash scripts, so I used lowercase variable names.
How about something simple like:
sleep "$threshold"
grep -q 'PATTERN' "$log_file" && { echo "Clear"; exit; }
echo "Error"
If that's not all you need then edit your question to clarify your requirements. Don't use all upper case for non exported shell variable names btw - google it.
To build further on your idea, it might be beneficial to run the awk part in the background and a continuous loop to do the checking.
#!/usr/bin/env bash
log_file="log.txt"
# threshold in seconds
threshold=10
# run the following process in the background
stdbuf -oL tail -f0n "$log_file" \
| awk '/PATTERN/{system("touch "pattern.tmp") }' &
while true; do
match=$(find . -type f -iname "pattern.tmp" -newermt "-${threshold} seconds")
if [[ -z "${match}" ]]; then
echo "Error"
else
echo "Clear"
fi
done
This looks to me like a watchdog timer. I've implemented something like this by forcing a background process to update my log, so I don't have to worry about read -t. Here's a working example:
#!/usr/bin/env bash
threshold=10
grain=2
errorstate=0
while sleep "$grain"; do
date '+[%F %T] watchdog timer' >> log
done &
trap "kill -HUP $!" 0 HUP INT QUIT TRAP ABRT TERM
printf -v lastseen '%(%s)T'
tail -F log | while read line; do
printf -v now '%(%s)T'
if (( now - lastseen > threshold )); then
echo "ERROR"
errorstate=1
else
if (( errorstate )); then
echo "Recovered, yay"
errorstate=0
fi
fi
if [[ $line =~ .*PATTERN.* ]]; then
lastseen=$now
fi
done
Run this in one window, wait $threshold seconds for it to trigger, then in another window echo PATTERN >> log to see the recovery.
While this can be made as granular as you like (I've set it to 2 seconds in the example), it does pollute your log file.
Oh, and note that printf '%(%s)T' format requires bash version 4 or above.

how to create incremental files with version in shell

how to create files dynamically when i run shell script.
Intially in the /tmp folder i need to check if file like(CFT_AH-120_v1.txt) exist in tmp folder else create CFT_AH-120_v1.txt. Next time when i run shell script it should create CFT_AH-120_v2.txt and in each run it should increment the version number of the file.
In tmp folder i should have files like
CFT_AH-120_v1.txt
CFT_AH-120_v2.txt
CFT_AH-120_v3.txt
i will get CFT_AH-120 from variable dynamically.
#!/bin/bash
export filename
temp=$(find CFT_AH-120-V* | sort -V | tail -1)
if [ -e $temp ]
then echo "ok"
echo $temp
fname="${temp%.*}"
echo $fname
temp1="${temp%[[:digit:]]*}$((${temp##*[[:alpha:]]} + 1))"
echo $temp1
touch $temp1 ".txt"
else
touch CFT_AH-120-V1.txt
echo "nok"
fi
I am not sure the exact requirement for you. As per my understanding this is a simple way of approach.
#!/bin/bash
file_temp=$(find . -name "CFT_AH-120-V*" -type f | sort -V | tail -1)
echo $temp
if [ -z "$temp" ]; then
echo "File not found!"
else
num_temp=$(echo $temp | cut -d '-' -f3- | sed 's/V//')
num_value_incr=$(expr $new_temp + 1)
touch "CFT_AH-120-V$num_value_incr"
echo "New file created!"
fi
Note: This code search for the largest value of the "V"....number.... (eg.***V120) and increment based on that value and also the intermediate number wont be increment. If you need the missing numbers to be created, then the logic for this code needs to be changed.
Hope this might help you!

Shell script output, how to scroll up?

I have a shell script that runs on Debian. It contains lots of functions and conditions. There is a menu and the user may choose different options and different outputs will be displayed. The user can always go back to the main menu and choose again some options then another output will be displayed. Of course each time the screen is cleared with "clear".
However when then output contains too many lines, I will be able to scroll up a little bit, but it will stop and I won't be able to scroll all the way to the first line I need to see. Being able to scroll up with the mousse wheel is the king of behavior I would like...
It looks like the problem comes from the xterm window, because it is fine with the normal terminal. However xterm is nice because I can setup the height and the width, as well as changing the colors...
Is there a way to increase this limitation from the script itself as I won't have the permission to change anything in the Debian environment...
I read that some people actually pipe the entire script to "less", I tried that, the problem is that I can't use the menu anymore...
Please find below the first script that is used to run the main one:
xterm -fg ivory -bg darkblue -fn 8x13bold -geometry 76x110+1700+0 -T "QC CHECK" -e /tests/SCRIPTS/QC/qc.sh
Below is a little sample of my script, but, it contains much more:
#!/bin/sh
stty erase ^H
function water
{
clear
echo -e "Current Survey : ${proj}"
echo -e "Current Sequence : ${seq}"
echo -e ""
echo -e " [ Trace QC Water Column ]"
echo -e ""
if [ $mincable -eq $maxcable ]
then
echo -e " Cable checked : $maxcable"
else
echo -e " Cables checked : ${mincable}-${maxcable}"
fi
echo -e " Max noise level : ${maxnoise}uB"
if [ ${skiptrace} -eq 0 ]
echo -e " Traces skipped : ${skiptrace}"
else
echo -e " Traces skipped : 1-${skiptrace}"
fi
echo -e ""
#############
water=`awk --field-separator=";" '($4>'$maxnoise') {print int(a=(($1-1)/'$nb_traces')+1) " " ($1-((int(a)-1)*'$nb_traces')) " " $4}' ${seq}_TraceAverages.txt | grep -v "USER_AVRMS_WC1" | grep -v "R32" | awk '{printf $1 " " $2 " " ("%*.*f\n"), 1, 2, $3}' | awk '($2>'$skiptrace')&&($1>='$mincable')&&($1<='$maxcable') {print $1 " - " $2 " - " $3}' | awk '{printf("%16s%6s%8s%6s%10s\n"), $1, $2, $3, $4, $5}' | awk '(NR>1) && (old != $1) {printf("%65s\n"), "'$sep_cable'"} {print; old=$1}'`
#############
count_water=`awk --field-separator=";" '($4>'$maxnoise') {print int(a=(($1-1)/'$nb_traces')+1) " " (b=($1-((int(a)-1)*'$nb_traces'))) " " $3}' ${seq}_TraceAverages.txt | grep -v "USER_AVRMS_WC1" | grep -v "R32" | awk '($2>'$skiptrace')&&($1>='$mincable')&&($1<='$maxcable') {print $3}' | wc -l`
#############
echo -e " ------------------------------------------------------"
echo -e ""
echo -e " Cable - Trace - RMS_WC"
echo -e ""
echo -e " ------------------------------------------------------"
echo -e ""
if [ $count_water -ge 1 ]
then
echo -e "$water"
else
setterm -term linux -back red -fore white
echo -e " Wow! No traces? Maybe decrease your values..."
setterm -term linux -default
fi
echo -e ""
setterm -term linux -back blue -fore white
echo -e " RMS_WC > ${maxnoise}uB = $count_water"
setterm -term linux -default
echo -e ""
echo -e " ------------------------------------------------------"
echo -e ""
}
# check for latest project in /tests
proj_temp
# if config file is missing go to config menu
if [ ! -e /tests/$proj/SCRIPTS/QC/config ]
then
config
fi
# force choice=1 and config_ok=1 to return to main menu when loop has run once (no problem when more than one)
choice=1
config_ok=1
while :
do
# do it all
if [ ${choice} -eq 1 2>/dev/null ]
then
choice=X
config_ok=1
# read configuration file
readconfig
main
# config menu and help
if [ ${seq} = "c" ]
then
config
elif [ ${seq} = "h" ]
then
help
elif [ ${seq} = "q" ]
then
clear
setterm -term linux -back magenta -fore white
echo ""
echo -e "\t Try me next time :*"
sleep 0.65
exit
fi
# config_ok=1 when configuration is done, meaning user returns to main menu after exiting config menu
if [ ${config_ok} -eq 1 ]
then
cd $input_dir
# check if file for requested sequence is valid
testline
# this function updates the awk script for signal QC check
awkscript
doall
choice
fi
# let the user choose what QC is wanted
elif [ ${choice} -eq 2 2>/dev/null ]
then
choice=X
# initialize values so that user can choose its own
init
menu
option
fi
case ${menu} in
1) deep
choice ;;
2) water
choice ;;
3) awkscript
signal
choice ;;
4) readconfig
awkscript
doall
choice ;;
esac
if [ ${choice} = "q" 2>/dev/null ]
then
exit
fi
done
As you can see I have many functions and many variables that are called is some "echo" which makes it hard for me to scroll up when there is too many lines, and, the user got also to scroll up and down to see everything and to choose and action.
Pipe the output thru less or more. There are options (hot keys) to go forth, back, search etc.
I could not find a way to scroll through the length of my output so what I did is a loop that is gradually increasing by increments of 0.1 the value of ${maxnoise} (with a condition on the number of line output) because this variable is actually the one conditioning how big is the output. It works fine this way so I consider my question answered.

Bash Script - Will not completely execute

I am writing a script that will take in 3 outputs and then search all files within a predefined path. However, my grep command seems to be breaking the script with error code 123. I have been staring at it for a while and cannot really seem the error so I was hoping someone could point out my error. Here is the code:
#! /bin/bash -e
#Check if path exists
if [ -z $ARCHIVE ]; then
echo "ARCHIVE NOT SET, PLEASE SET TO PROCEED."
echo "EXITING...."
exit 1
elif [ $# -ne 3 ]; then
echo "Illegal number of arguments"
echo "Please enter the date in yyyy mm dd"
echo "EXITING..."
exit 1
fi
filename=output.txt
#Simple signal handler
signal_handler()
{
echo ""
echo "Process killed or interrupted"
echo "Cleaning up files..."
rm -f out
echo "Finsihed"
exit 1
}
trap 'signal_handler' KILL
trap 'signal_handler' TERM
trap 'signal_handler' INT
echo "line 32"
echo $1 $2 $3
#Search for the TimeStamp field and replace the / and : characters
find $ARCHIVE | xargs grep -l "TimeStamp: $2/$3/$1"
echo "line 35"
fileSize=`wc -c out.txt | cut -f 1 -d ' '`
echo $fileSize
if [ $fileSize -ge 1 ]; then
echo "no"
xargs -n1 basename < $filename
else
echo "NO FILES EXIST"
fi
I added the echo's to know where it was breaking. My program prints out line 32 and the args but never line 35. When I check the exit code I get 123.
Thanks!
Notes:
ARCHIVE is set to a test directory, i.e. /home/'uname'/testDir
$1 $2 $3 == yyyy mm dd (ie a date)
In testDir there are N number of directories. Inside these directories there are data files that have contain data as well as a time tag. The time tag is of the following format: TimeStamp: 02/02/2004 at 20:38:01
The scripts goal is to find all files that have the date tag you are searching for.
Here's a simpler test case that demonstrates your problem:
#!/bin/bash -e
echo "This prints"
true | xargs false
echo "This does not"
The snippet exits with code 123.
The problem is that xargs exits with code 123 if any command fails. When xargs exits with non-zero status, -e causes the script to exit.
The quickest fix is to use || true to effectively ignore xargs' status:
#!/bin/bash -e
echo "This prints"
true | xargs false || true
echo "This now prints too"
The better fix is to not rely on -e, since this option is misleading and unpredictable.
xargs makes the error code 123 when grep returns a nonzero code even just once. Since you're using -e (#!/bin/bash -e), bash would exit the script when one of its commands return a nonzero exit code. Not using -e would allow your code to continue. Just disabling it on that part can be a solution too:
set +e ## Disable
find "$ARCHIVE" | xargs grep -l "TimeStamp: $2/$1/$3" ## If one of the files doesn't match the pattern, `grep` would return a nonzero code.
set -e ## Enable again.
Consider placing your variables around quotes to prevent word splitting as well like "$ARCHIVE".
-d '\n' may also be required if one of your files' filename contain spaces.
find "$ARCHIVE" | xargs -d '\n' grep -l "TimeStamp: $2/$1/$3"

Resources