How to get lines from log file from last 10 minutes with specific string - shell

Tried other solution but not giving correct solutions my time format is [Thu Aug 20 09:28:51 2020]. Most close one was this one
awk -vDate=`date -d'now-2 hours' +[%a %b %d %H:%M:%S %Y]` '$4 > Date {print Date, $0}' $input
my log file are like this
[Thu Aug 20 09:10:51 2020] [error] vendor
[Thu Aug 20 09:23:51 2020] [error] vendor
[Thu Aug 20 09:25:51 2020] [error] vendor
[Thu Aug 20 09:27:51 2020] [error] vendor
[Thu Aug 20 09:28:51 2020] [error] dad
i want result as from current time [Thu Aug 20 09:28:51 2020] to last 10 mins
[Thu Aug 20 09:23:51 2020] [error] vendor
[Thu Aug 20 09:25:51 2020] [error] vendor
[Thu Aug 20 09:27:51 2020] [error] vendor
[Thu Aug 20 09:28:51 2020] [error] dad

well i tried doing directly with grep but i dont know why but the grep wasnt taking this date format and give some wrong output so i did some way around for it .
#!/bin/bash
input="/home/babin/Desktop/code2"
count=0
dateyear=$(date +'%Y')
month=$(date +'%b')
day=$(date +'%a')
#do loop for 10 mins from now
for (( i = 0; i <=9; i++ )) ; do
if grep $(date +%R -d "-$i min") $input | grep -i "error" | grep -wi "$month" | grep -wi "$year" | grep -wi "$day"
then
currentcount=$(grep $(date +%R -d "-$i min") $input | grep -wi "70007" | grep -wi "$month" | grep -wi "$year" | grep -wic "$day")
else
currentcount=0
echo "not found"
fi
count=$(( $count + $currentcount ))
done
echo "$count"
#check no of error found and do task
if(( $count >= 10))
then
echo "more oe equal to 10 finds"
else
echo "less than 10 occurence"
fi
it gives output as currenttime is [Thu Aug 20 09:28:51 2020] also it matches "error" string.
enter [Thu Aug 20 09:23:51 2020] [error] vendor
[Thu Aug 20 09:25:51 2020] [error] vendor
[Thu Aug 20 09:27:51 2020] [error] vendor
[Thu Aug 20 09:28:51 2020] [error] dadcode here

The overall flow is:
Preprocess input to exctract the date part
Convert date to seconds since epoch
Filter seconds since epoch according to the conditions given
Remove seconds since epoch.
Output.
As a overall rule, work in bash using streams. The strptime is from dateutils package. Like so:
# Extract the date+time part from within [..] and put it on the first column with tab
sed 's/ \[\([^]]*\)\]/\1\t&/' "$input" |
# For each line
while IFS=$'\t' read -r date rest; do
# Convert the date to seconds since epoch
date=$(strptime -f "%s" -i "%a %b %d %H:%M:%S %Y" "$date")
# Output the updated line
printf "%s\t%s\n" "$date" "$rest"
done |
# Read it all in awk and compare second since epoch in the first field to given value
awk -v "since=$(date -d'now -2 hours' +%s)" '$1 > since' |
# Remove first field - ie. second since epoch
cut -f2-
Do not use backticks ``. They are discouraged. Use $(...) instead. Remember as a rule of a thumb to quote all variable expansions. Check your scripts for most common mistakes with http://shellcheck.net . I think somewhere between date and strptime you may encounter problems with your timezone (ie difference in the number of hours).

Related

SSH into multiple servers and compare timestamps of each server

I need to add the timestamp of all remote servers as part of output and check & compare whether the timestamp is the same or not,
I am able to print the machine IP and date.
#!/bin/bash
all_ip=(192.168.1.121 192.168.1.122 192.168.1.123)
for ip_addr in "${all_ip[#]}"; do
aws_ip=$"ip route get 1 | sed -n 's/^.*src \([0-9.]*\) .*$/\1/p'"
date=date
sshpass -p "password" ssh root#$ip_addr "$aws_ip & $date"
echo "==================================================="
done
Getting Output as :
Wed 27 Jul 2022 05:48:15 AM PDT
192.168.1.121
===================================================
Wed Jul 27 05:48:15 PDT 2022
192.168.1.122
===================================================
Wed Jul 27 05:48:15 PDT 2022
192.168.1.123
===================================================
How to check whether the timestamp ( ignoring seconds ) of all machines is the same or not ,
eg: (Wed 27 Jul 2022 05:48:15 || Wed 27 Jul 2022 05:48:15 || Wed 27 Jul 2022 05:48:15)
Expected Output:
|| Time are in sync on all machines || # if in sync
|| Time are not in sync on all machines || # if not sync
Wed 27 Jul 2022 05:48:15 AM PDT
192.168.1.121
===================================================
Wed Jul 27 05:48:15 PDT 2022
192.168.1.122
===================================================
Wed Jul 27 05:48:15 PDT 2022
192.168.1.123
===================================================
How to check whether the time ( ignoring seconds )
tmpdir=$(mktemp -d)
trap 'rm -r "$tmpdir"' EXIT
for ip in "${allips[#]}"; do
# Do N connections, in paralllel, each one writes to a separate file.
sshpass -p "password" ssh root#"$ip" "date +%Y-%m-%d_%H:%M" > "$tmpdir/$ip.txt" &
done
wait
times=$(
for i in "$tmpdir"/*.txt; do
# print filename with file contents.
echo "$i $(<$i)"
done |
# Sort them on second column
sort -k2 |
# Uniq on second field
uniq -f 2
)
echo "$times"
timeslines=$(wc -l <<<"$times")
if ((timeslines == 1)); then
echo "YAY! minutes on all servers the same"
fi
First, you may adjust your "date" command as folow in order to exclude the seconds:
date +%Y-%m-%d_%H:%M
Then, simply grep your output and validate that all the timestamps are identical. You may dump in a temporary file or any other way.
Ex:
grep [aPatternSpecificToTheLinewithTheDate] [yourTemporaryFile] | sort | uniq | wc -l
If the result is 1, it means that all the timestamps are identical.
However you will have to deal with the corner case where the minute shift while you are fetching the time form all your servers.

Restart Apache if average server load past minute is higher than X

I wrote a shell script and added it to my cron. It's supposed to run every minute and check for the average server load, past 1 minute, and if it's over 40 it should log the load, date and then restart Apache httpd. Here is my script:
#!/bin/bash
LOGFILE=/home/user/public_html/domain.com/cron/restart.log
function float_to_int() {
echo $1 | cut -d. -f1
}
check=$(uptime | awk -F' *,? *' '{print $12}')
now=$(date)
checkk=$(float_to_int $check)
if [[ $checkk > 40 ]]; then
echo $now $checkk >> $LOGFILE 2>&1
/usr/bin/systemctl restart httpd.service
fi
If I look at the log file I see the following:
Wed Jul 3 20:02:01 EDT 2019 70
Wed Jul 3 23:03:01 EDT 2019 43
Wed Jul 3 23:12:01 EDT 2019 9
Wed Jul 3 23:13:01 EDT 2019 7
Wed Jul 3 23:14:01 EDT 2019 6
Wed Jul 3 23:15:02 EDT 2019 5
Wed Jul 3 23:16:01 EDT 2019 5
Something is clearly wrong as it should only log and restart Apache if the load is over 40 but as you can see from the logs the load was 9, 7, 6, 5 and 5. Could someone point me in the right direction?
From man bash, section CONDITIONAL EXPRESSIONS (emphasis mine) :
string1 > string2
True if string1 sorts after string2 lexicographically.
You will either want to use [['s -gt operator, or use arithmetic evaluation instead of [[ :
if (( chekk > 40 )); then
Here's one in GNU awk (GNU awk due to strftime()):
awk '
$1 > 0.4 { # interval above 0.4
logfile="./log.txt" # my logpath, change it
print strftime("%c"), $1 >> logfile # date and load to log
cmd="/usr/bin/systemctl restart httpd.service" # command to use for restarting
if((ret=(cmd|getline res)) !=0 ) # store return value and result
print "failed: " ret # if failed
else
print "success"
}' /proc/loadavg # getting load avg from /proc

subtracting dates with specific format

i have search for a solution to my shell date subtraction issue with no joy so here goes.
i have a date format like so %m%d%H%M%S which is "0102231203" and the second %Y%m%d%H%M%S, i can take the year off the second one and do a normal subtraction but when it is over a day it becomes an issue with the time being incorrect.
here is what i have tried so far
BTT=0102234500
TPP=0102233635 (after removing the year)
BT=date -d ${BTT}
TP=date -d ${TPP}
and
BT=date -d $BTT +%m%d%H%M%S
TP=date +%m%d%H%M%S -d ${TPP}
date: invalid date `0102234500'
date: invalid date `0102233635'
BT=date -d #${BTT} +%m%d%H%M%S
TP=date +%m%d%H%M%S -d #${TPP}
weird output
0329071355
0329072820
BT=date -d #${BTT}
TP=date -d #${TPP}
Thu Mar 29 07:13:55 BST 1973
Thu Mar 29 07:28:20 BST 1973
even changed it to add the year to both still
BTT=20130102234500
TPP=20130102233635
BT=date -d #${BTT}
TP=date -d #${TPP}
Fri Jul 19 08:53:55 GMT 639867
Fri Jul 19 09:08:20 GMT 639867
how do i resolve this issue.
tnx
The -d option of date accept human readable string so if you can have full length date you can do :
me#server:/tmp$ BTT=`date +"%Y-%m-%d %H:%M:%S"`
me#server:/tmp$ TPP=`date +"%Y-%m-%d %H:%M:%S"`
me#server:/tmp$ echo $((`date -d "$TPP" +%s`-`date -d "$BTT" +%s`))
3
With your datas :
me#server:/tmp$ BTT="2013-01-02 23:45:00"
me#server:/tmp$ TPP="2013-01-02 23:36:35"
me#server:/tmp$ echo $((`date -d "$BTT" +%s`-`date -d "$TPP" +%s`))
505
With the results in seconds.

Converting Month String into Integer Shell

okay so i run an openssl command to get the date of an expired script. Doing so gives me this:
enddate=Jun 26 23:59:59 2012 GMT
Then i cut everything out and just leave the month which is "Jun"
Now the next part of my script is to tell the user if the the certificate is expired or not and to do that i use an if statement in which it looks like this:
if [ $exp_year -lt $cur_year && $exp_month -lt $cur_month ]; then
echo ""
echo "Certificate is still valid until $exp_date"
echo ""
else
echo ""
echo "Certificate has expired on $exp_date, please renew."
echo ""
fi
I can't figure out how to convert the month into an integer to even do the comparison.
I thought of doing the brute force way which is this:
Jan=01
Feb=02
Mar=03
...
Clearly that's a terrible way to do it. Does anyone know what i can do?
well, you can use:
now=$(date +%s)
cert=$(date --date="$enddate" +%s)
if [ $cert -lt $now ]; then
echo "Old!"
fi
i.e. convert the date into the seconds past the epoch and compare those
I would recommend using Petesh's answer, but here's a way to set up an associative array if you have Bash 4:
months=(Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec)
declare -A mlookup
for monthnum in ${!months[#]}
do
mlookup[${months[monthnum]]=$((monthnum + 1))
done
echo "${mlookup["Jun"]}" # outputs 6
If you have Bash before version 4, you can use AWK to help you out:
month=Feb
awk -v "month=$month" 'BEGIN {months = "Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec"; print (index(months, month) + 3) / 4}'
Another way in pure Bash (any version):
months="Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec"
month=Aug
string="${months%$month*}"
echo "$((${#string}/4 + 1))"

How to get epoch time in shell script (for ksh)?

How to get epoch time in shell script (for ksh)?
I am interested in getting epoch time for the start of day (so e.g. now is July 28th, 2011 ~ 14:25:00 EST, I need time at midnight).
If you have GNU date,
epoch=$( date -d 00:00 +%s )
Otherwise, if you have tclsh,
epoch=$( echo 'puts [clock scan 00:00]' | tclsh )
Otherwise,
epoch=$( perl -MTime::Local -le 'print timelocal(0,0,0,(localtime)[3..8])' )
ksh's printf '%(fmt)T' supports time calculating. For example:
$ printf '%T\n' now
Mon Mar 18 15:11:46 CST 2013
$ printf '%T\n' '2 days ago'
Sat Mar 16 15:11:55 CST 2013
$ printf '%T\n' 'midnight today'
Mon Mar 18 00:00:00 CST 2013
$

Resources