calculate start and end of working, grep in logfile - bash

I need to know when I can do a maintance on a frequently used system. All I can check is a logfile, where I can see when the users are starting and ending there work in average.
I need to do this for weekdays, saturday and sunday.
I know how to grep these information but I don't know how to separate weekdays from weekends and how to build an average from the timestamps. Can anyone help me with that please? Kind regards
Edit: More information as requested
Here is my script so far:
i=14
while i >=0
do dow=$(date -d "-$i day" +%A)
if [ $dow = "Saturday" ] || [ $dow = "Sunday" ]
then i=$((i-1))
fi
beginnweek+=(`zgrep T400: logfile|grep -v 'T811:Icinga'|head -n 1|cut -d " " -f2`)
endweek+=(`zgrep T400: logfile|grep -v 'T811:Icinga'|tail -n 1|cut -d " " -f2`)
i=$((i-1))
done
###calculate average beginn and end - Thats what missing
i=14
while i >=0
do dow=$(date -d "-$i day" +%A)
if [ $dow = "Monday" ] || [ $dow = "Tuesday" ] || [ $dow = "Wednesday" ] || [ $dow = "Thursday" ] || [ $dow = "Friday" ] || [ $dow = "Sunday" ]
then i=$((i-1))
fi
beginnSat+=(`zgrep T400: logfile|grep -v 'T811:Icinga'|head -n 1|cut -d " " -f2`)
endSat+=(`zgrep T400: logfile|grep -v 'T811:Icinga'|tail -n 1|cut -d " " -f2`)
i=$((i-1))
done
###calculate average beginn and end - Thats what missing
i=14
while i >=0
do dow=$(date -d "-$i day" +%A)
if [ $dow = "Monday" ] || [ $dow = "Tuesday" ] || [ $dow = "Wednesday" ] || [ $dow = "Thursday" ] || [ $dow = "Friday" ] || [ $dow = "Saturday" ]
then i=$((i-1))
fi
beginnSun+=(`zgrep T400: logfile|grep -v 'T811:Icinga'|head -n 1|cut -d " " -f2`)
endSun+=(`zgrep T400: logfile|grep -v 'T811:Icinga'|tail -n 1|cut -d " " -f2`)
i=$((i-1))
done
###calculate average beginn and end - Thats what missing
I'm working with
GNU bash, version 4.2.46
on SLES and with
GNU bash, version 3.1.17
The logfiles are looking like this:
19/10/2018 04:00:03.175 : [32631] INFO : (8) >>\\\\\\\\\\T090:NOPRINT,NOSAVE|T400:551200015480|T811:Icinga|T8904:001|T8905:001|//////////
19/10/2018 07:17:19.501 : [4935] INFO : >>\\\\\\\\\\T021:datamax|T050:software|T051:V 1.0|T101:|T400:428568605212|T520:00000000|T510:|T500:|T545:19.10.2018||T821:DE|PRINTINFO:|PRINT1:|PRINT0:intermec pf4i.int01|//////////

First of all you should ask yourself if you really want to use an average. An average makes only sense if all users login at morning, stay logged in over noon, and logout at evening. If you have logouts distributed all over the day the average logout time is meaningless.
But even in such an idealized case you shouldn't start maintenance right after the average logout time since around 50% of the users would still be logged in at that time.
I would rather visualize logins as bars and determine a good maintenance time by hand. ranwhen.py (see picture below) is a very nice tool to display when your system was up. Maybe you can find something similar for logins or adapt the tool yourself.
Nevertheless, here's what you asked for:
Parsing The Logs
Instead of parsing the log manually, I would advise you to use the last tool, which prints the last logins in a simpler format. Since you are on Linux, there should be an -F option for last to print dates prefixed with their weekday. With -R we suppress some unneeded information. The output of last -FR looks as follows:
socowi pts/5 Fri Oct 19 17:42:16 2018 still logged in
reboot system boot Fri Oct 19 14:34:44 2018 still running
alice pts/2 Fri Oct 19 10:35:05 2018 - Fri Oct 19 11:51:03 2018 (01:15)
alice tty7 Fri Oct 19 10:24:32 2018 - Fri Oct 19 11:51:52 2018 (01:27)
bob tty7 Fri Oct 19 10:04:21 2018 - Fri Oct 19 10:14:01 2018 (00:09)
reboot system boot Fri Oct 19 12:03:34 2018 - Fri Oct 19 11:51:55 2018 (00:-11)
carol tty7 Fri Oct 19 08:10:49 2018 - down (01:50)
dave tty7 Thu Oct 18 12:48:12 2018 - crash (04:28)
wtmp begins Tue Oct 16 12:38:03 2018
To extract the valid login and logout dates we use the following functions.
onlyUsers() { last -FR | head -n -2 | grep -Ev '^reboot '; }
onlyDates() { grep -F :; }
loginDates() { onlyUsers | cut -c 23-46 | onlyDates; }
logoutDates() { onlyUsers | cut -c 50-73 | onlyDates; }
Filter By Weekday
The functions loginDates and logoutDates print something like
Fri Oct 19 17:42:16 2018
Fri Oct 19 14:34:44 2018
[...]
Thu Oct 18 12:48:12 2018
Filtering out specific weekdays is pretty easy:
workweek() { grep -E 'Mon|Tue|Wed|Thu|Fri'; }
weekend() { grep -E 'Sat|Sun'; }
If you want all login dates on weekends, you would write loginDates | weekend.
Computing An Average Time
To compute the average time from multiple dates, we first extract the time of day from the dates. Then we convert the HH:MM format to minutes since midnight. Computing an average of a list numbers is easy. Afterwards we convert back to HH:MM.
timeOfDay() { cut -c 12-16; }
timeToMins() { awk -F: '{print $1*60 + $2}'; }
minsToTime() { awk '{printf "%02d:%02d", $1/60, $1%60}'; }
avgMins() { awk '{s+=$1}END{printf "%d", s/NR}'; }
avgTime() { timeOfDay | timeToMins | avgMins | minsToTime; }
Putting Everything Together
To get the average times just combine the commands as needed. Some examples:
# Average login times during workweeks
avg="$(loginDates | workweek | avgTime)"
# Average logout times on weekends
avg="$(logoutDates | weekend | avgTime)"

Related

SSH into multiple servers and compare timestamps of each server

I need to add the timestamp of all remote servers as part of output and check & compare whether the timestamp is the same or not,
I am able to print the machine IP and date.
#!/bin/bash
all_ip=(192.168.1.121 192.168.1.122 192.168.1.123)
for ip_addr in "${all_ip[#]}"; do
aws_ip=$"ip route get 1 | sed -n 's/^.*src \([0-9.]*\) .*$/\1/p'"
date=date
sshpass -p "password" ssh root#$ip_addr "$aws_ip & $date"
echo "==================================================="
done
Getting Output as :
Wed 27 Jul 2022 05:48:15 AM PDT
192.168.1.121
===================================================
Wed Jul 27 05:48:15 PDT 2022
192.168.1.122
===================================================
Wed Jul 27 05:48:15 PDT 2022
192.168.1.123
===================================================
How to check whether the timestamp ( ignoring seconds ) of all machines is the same or not ,
eg: (Wed 27 Jul 2022 05:48:15 || Wed 27 Jul 2022 05:48:15 || Wed 27 Jul 2022 05:48:15)
Expected Output:
|| Time are in sync on all machines || # if in sync
|| Time are not in sync on all machines || # if not sync
Wed 27 Jul 2022 05:48:15 AM PDT
192.168.1.121
===================================================
Wed Jul 27 05:48:15 PDT 2022
192.168.1.122
===================================================
Wed Jul 27 05:48:15 PDT 2022
192.168.1.123
===================================================
How to check whether the time ( ignoring seconds )
tmpdir=$(mktemp -d)
trap 'rm -r "$tmpdir"' EXIT
for ip in "${allips[#]}"; do
# Do N connections, in paralllel, each one writes to a separate file.
sshpass -p "password" ssh root#"$ip" "date +%Y-%m-%d_%H:%M" > "$tmpdir/$ip.txt" &
done
wait
times=$(
for i in "$tmpdir"/*.txt; do
# print filename with file contents.
echo "$i $(<$i)"
done |
# Sort them on second column
sort -k2 |
# Uniq on second field
uniq -f 2
)
echo "$times"
timeslines=$(wc -l <<<"$times")
if ((timeslines == 1)); then
echo "YAY! minutes on all servers the same"
fi
First, you may adjust your "date" command as folow in order to exclude the seconds:
date +%Y-%m-%d_%H:%M
Then, simply grep your output and validate that all the timestamps are identical. You may dump in a temporary file or any other way.
Ex:
grep [aPatternSpecificToTheLinewithTheDate] [yourTemporaryFile] | sort | uniq | wc -l
If the result is 1, it means that all the timestamps are identical.
However you will have to deal with the corner case where the minute shift while you are fetching the time form all your servers.

Restart Apache if average server load past minute is higher than X

I wrote a shell script and added it to my cron. It's supposed to run every minute and check for the average server load, past 1 minute, and if it's over 40 it should log the load, date and then restart Apache httpd. Here is my script:
#!/bin/bash
LOGFILE=/home/user/public_html/domain.com/cron/restart.log
function float_to_int() {
echo $1 | cut -d. -f1
}
check=$(uptime | awk -F' *,? *' '{print $12}')
now=$(date)
checkk=$(float_to_int $check)
if [[ $checkk > 40 ]]; then
echo $now $checkk >> $LOGFILE 2>&1
/usr/bin/systemctl restart httpd.service
fi
If I look at the log file I see the following:
Wed Jul 3 20:02:01 EDT 2019 70
Wed Jul 3 23:03:01 EDT 2019 43
Wed Jul 3 23:12:01 EDT 2019 9
Wed Jul 3 23:13:01 EDT 2019 7
Wed Jul 3 23:14:01 EDT 2019 6
Wed Jul 3 23:15:02 EDT 2019 5
Wed Jul 3 23:16:01 EDT 2019 5
Something is clearly wrong as it should only log and restart Apache if the load is over 40 but as you can see from the logs the load was 9, 7, 6, 5 and 5. Could someone point me in the right direction?
From man bash, section CONDITIONAL EXPRESSIONS (emphasis mine) :
string1 > string2
True if string1 sorts after string2 lexicographically.
You will either want to use [['s -gt operator, or use arithmetic evaluation instead of [[ :
if (( chekk > 40 )); then
Here's one in GNU awk (GNU awk due to strftime()):
awk '
$1 > 0.4 { # interval above 0.4
logfile="./log.txt" # my logpath, change it
print strftime("%c"), $1 >> logfile # date and load to log
cmd="/usr/bin/systemctl restart httpd.service" # command to use for restarting
if((ret=(cmd|getline res)) !=0 ) # store return value and result
print "failed: " ret # if failed
else
print "success"
}' /proc/loadavg # getting load avg from /proc

AWK - Getting all columns where first is = $var && Date >= $date

Im new to AWK, and am trying to work out how to get all the results where the first column is equal to a variable and the date is greater than another unix timestamp formatted variable. Im using 'last' as my command. Example output is:
bob pts/2 172.6.14.37 Fri July 24 12:43 - 12:17 (9+23:34)
bob pts/2 172.6.14.37 Fri July 24 10:03 - 12:17 (5+23:34)
bob pts/2 172.6.14.37 Tue June 4 17:55 - 09:42 (8+15:46)
bob pts/2 172.6.14.37 Tue Mar 4 17:55 - 09:42 (8+15:46)
tim pts/1 172.6.14.37 Mon Mar 3 16:22 - 17:30 (1+01:08)
root pts/1 172.6.14.37 Thu Feb 27 09:38 - 09:56 (4+00:18)
and so I want all the results where 'bob' is in the first column. I've got
last -f /var/log/btmp | awk '$1 == "bob"'
Which gives me all bobs failed logins. Now I need to filter again where the date filed is greater than say '20140723145100' something like
last -f /var/log/btmp | awk '$1 == "bob" && $4 >= $DATE'
Assuming $DATE = 20140723145100 , the result I would want would be :
bob pts/2 172.6.14.37 Fri July 24 12:43 - 12:17 (9+23:34)
bob pts/2 172.6.14.37 Fri July 24 10:03 - 12:17 (5+23:34)
bash:
user=bob
since=20140623145100
last -Fa -f /var/log/btmp |
while read line; do
set -- $line # no quotes here
[[ $1 == "$user" ]] || continue
[[ $(date -d "$3 $4 $5 $6 $7" +%Y%m%d%H%M%S) > $since ]] && echo "$line"
done
Use the -s option in last:
last -s 20140723145100
From man last:
-s, --since time
Display the state of logins since specified time. This is useful,
e.g., to determine easily who was logged in at a particular time. The
option is often combined with --until.
And then grep for the user:
last -s 20140723145100 | grep "^bob"
As you do not have the -s option, you can use this workaround: store all the last output and the output until a certain time (using -t option). Then compare the output:
last -f /var/log/btmp | grep "^bob" > everything
last -f /var/log/btmp -t "20140723145100" | grep "^bob" > upto_20140723145100
grep -vf upto_20140723145100 everything
Using GNU Awk:
gawk -v user=bob -v date=20140723145100 -F '[[:space:]]{3,}| - ' '$1 == user { cmd = "exec date -d \"" $4 "\" +%Y%m%d%H%M%S"; cmd | getline d; close(cmd); if (d >= date) print }' sample
Output:
bob pts/2 172.6.14.37 Fri July 24 12:43 - 12:17 (9+23:34)
bob pts/2 172.6.14.37 Fri July 24 10:03 - 12:17 (5+23:34)
Of course actual command is last -f /var/log/btmp | gawk -v user=bob -v date=20140723145100 ....
And here's a script version:
#!/usr/bin/gawk -f
BEGIN {
FS = "[[:space:]]{3,}| - "
}
$1 == user {
cmd = "exec date -d \"" $4 "\" +%Y%m%d%H%M%S"
cmd | getline d
close(cmd)
if (d >= date)
print
}
Usage:
last -f /var/log/btmp | gawk -v user=bob -v date=20140723145100 -f script.awk

subtracting dates with specific format

i have search for a solution to my shell date subtraction issue with no joy so here goes.
i have a date format like so %m%d%H%M%S which is "0102231203" and the second %Y%m%d%H%M%S, i can take the year off the second one and do a normal subtraction but when it is over a day it becomes an issue with the time being incorrect.
here is what i have tried so far
BTT=0102234500
TPP=0102233635 (after removing the year)
BT=date -d ${BTT}
TP=date -d ${TPP}
and
BT=date -d $BTT +%m%d%H%M%S
TP=date +%m%d%H%M%S -d ${TPP}
date: invalid date `0102234500'
date: invalid date `0102233635'
BT=date -d #${BTT} +%m%d%H%M%S
TP=date +%m%d%H%M%S -d #${TPP}
weird output
0329071355
0329072820
BT=date -d #${BTT}
TP=date -d #${TPP}
Thu Mar 29 07:13:55 BST 1973
Thu Mar 29 07:28:20 BST 1973
even changed it to add the year to both still
BTT=20130102234500
TPP=20130102233635
BT=date -d #${BTT}
TP=date -d #${TPP}
Fri Jul 19 08:53:55 GMT 639867
Fri Jul 19 09:08:20 GMT 639867
how do i resolve this issue.
tnx
The -d option of date accept human readable string so if you can have full length date you can do :
me#server:/tmp$ BTT=`date +"%Y-%m-%d %H:%M:%S"`
me#server:/tmp$ TPP=`date +"%Y-%m-%d %H:%M:%S"`
me#server:/tmp$ echo $((`date -d "$TPP" +%s`-`date -d "$BTT" +%s`))
3
With your datas :
me#server:/tmp$ BTT="2013-01-02 23:45:00"
me#server:/tmp$ TPP="2013-01-02 23:36:35"
me#server:/tmp$ echo $((`date -d "$BTT" +%s`-`date -d "$TPP" +%s`))
505
With the results in seconds.

Converting Month String into Integer Shell

okay so i run an openssl command to get the date of an expired script. Doing so gives me this:
enddate=Jun 26 23:59:59 2012 GMT
Then i cut everything out and just leave the month which is "Jun"
Now the next part of my script is to tell the user if the the certificate is expired or not and to do that i use an if statement in which it looks like this:
if [ $exp_year -lt $cur_year && $exp_month -lt $cur_month ]; then
echo ""
echo "Certificate is still valid until $exp_date"
echo ""
else
echo ""
echo "Certificate has expired on $exp_date, please renew."
echo ""
fi
I can't figure out how to convert the month into an integer to even do the comparison.
I thought of doing the brute force way which is this:
Jan=01
Feb=02
Mar=03
...
Clearly that's a terrible way to do it. Does anyone know what i can do?
well, you can use:
now=$(date +%s)
cert=$(date --date="$enddate" +%s)
if [ $cert -lt $now ]; then
echo "Old!"
fi
i.e. convert the date into the seconds past the epoch and compare those
I would recommend using Petesh's answer, but here's a way to set up an associative array if you have Bash 4:
months=(Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec)
declare -A mlookup
for monthnum in ${!months[#]}
do
mlookup[${months[monthnum]]=$((monthnum + 1))
done
echo "${mlookup["Jun"]}" # outputs 6
If you have Bash before version 4, you can use AWK to help you out:
month=Feb
awk -v "month=$month" 'BEGIN {months = "Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec"; print (index(months, month) + 3) / 4}'
Another way in pure Bash (any version):
months="Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec"
month=Aug
string="${months%$month*}"
echo "$((${#string}/4 + 1))"

Resources