how to process last x lines of a file - shell

I want to Analyse a logfile for specific Errors.
Therefore i want to be able to loop through the last x lines of the files and check every line with a specific REGEX Pattern and then define a specific return value.
The logfile Looks in case of success as follows at the Moment when i want to check it.
….
sftp> get blahblah/blahblah
sftp> bye
In case of an Error there is something between the two sftp lines.
What i allready tried is to solve the Problem with a specific regex which worked fine on some online Regex testers but couldn´t get it to work in ksh.
My current Approach is the following
LOG_FIL="test_log"
MODE="${1}"
check_log_file() {
ERRNBR=${1}
REGEX=${2}
TAIL=${3}
RETURN="0"
echo "ERRNBR = ${ERRNBR}"
echo "REGEX = ${REGEX}"
echo "TAIL = ${TAIL}"
while read line; do
echo "${line}"
if [[ "${line}" =~ ${REGEX} ]]; then
RETURN="0"
echo "bin hier"
else
RETURN=${ERRNBR}
echo "bin wo anders"
break
fi
done <<<$(tail -${TAIL} ${LOG_FIL})
echo "${RETURN}"
return ${RETURN}
}
echo "sftp> get cwi/cdk_final*" >> ${LOG_FIL}
if [ "${MODE}" == "1" ]; then
echo "Werner ist der beste" >>${LOG_FIL}
fi
check_log_file "22" "^(sftp> ).*$" "1"
echo "$?"
echo "sftp> bye" >> ${LOG_FIL}
check_log_file "21" "((sftp> ).*|(sftp> bye))" "2"
echo "$?"
The results i get are the following
edv> sh cdk_test4sftp.sh 1
ERRNBR = 22
REGEX = ^(sftp> ).*$
TAIL = 1
Werner ist der beste
bin wo anders
22
22
ERRNBR = 21
REGEX = ((sftp> ).*|(sftp> bye))
TAIL = 2
Werner ist der beste sftp> bye
bin hier
0
0
What i hoped to achieve was that the Output coming from the tail command would be seperated. So that i ccould test each line individually.

Your second regex:
((sftp> ).*|(sftp> bye))
Matches the following line, which is why your function returns 0:
Werner ist der beste sftp> bye
Since you want to match the following pattern on each line:
sftp> get blahblah/blahblah
sftp> bye
Your regex should look more like the first one you used to match:
^(sftp> ).*$

Related

Cannot understand why our function calls return twice?

We have a 15 year (or so) old script we are trying to figure out and document. We have found some errors in it but one specific log file gives us much headache. and I would love some help figuring it out.
First the function that are run with the question:
#=========================================================#
# Define function removeOldBackupFile. #
#=========================================================#
removeOldBackupFile()
{
#set -x
echo "Removing old backups if they exists." >> "${report}"
local RCLOC=0
spaceBefore=$(getAvailableSpace ${backupDirectory})
timesToWait=60 # Wait a maximum of 10 minutes before bailing
cat ${oldDbContainer} | while read fileName
do
echo "Old file exists. Removing ${fileName}." >> "${report}"
removeFileIfExist "${fileName}"
RC=$?
echo "Resultcode for removing old backup is: RC=$RC." >> "${report}"
RCLOC=$(($RC+$RCLOC))
spaceAfter=$(getAvailableSpace ${backupDirectory})
# Wait for the OS to register that the file is removed
cnt=0
while [ $spaceAfter -le $spaceBefore ]; do
cnt=$((cnt+1))
if [ $cnt -gt $timesToWait ]; then
echo "Waited too long for space in ${backupDirectory}" | tee -a "${report}"
RCLOC=$(($RCLOC+1))
return $RCLOC
fi
sleep 10
spaceAfter=$(getAvailableSpace ${backupDirectory})
done
done
return $RCLOC
}
The place where this function is ran looks as follows:
#=========================================================#
# Remove old backupfiles if any exist. #
#=========================================================#
removeOldBackupFile
RC=$?
RCSUM=$(($RC+$RCSUM))
We have identified that the if condition is a bit wrong and the while loops would not work as intended if there are multiple files.
But what bothers us is output from a log file:
...
+ cnt=61
+ '[' 61 -gt 60 ']'
+ echo 'Waited too long for space in /<redacted>/backup'
+ tee -a /tmp/maintenanceBackupMessage.70927
Waited too long for space in /<redacted>/backup
+ RCLOC=1
+ return 1
+ return 0
+ RC=0
+ RCSUM=0
...
As seen in the log output after the inner loop have ran 60 times and ending it returns 1 as expected.. BUT! it also have return 0 after!? Why is it also returning 0?
We are unable to figure out the double returns... Any help appriciated
The first return executes in the subshell started by the pipe cat ${oldDbContainer} | while .... The second return is from return $RCLOC at the end of the function. Get rid of the useless use of cat:
removeOldBackupFile()
{
#set -x
echo "Removing old backups if they exists." >> "${report}"
local RCLOC=0
spaceBefore=$(getAvailableSpace ${backupDirectory})
timesToWait=60 # Wait a maximum of 10 minutes before bailing
while read fileName
do
...
done < ${oldDbContainer}
return $RCLOC
}

What & How does this line of code work: {message:0:1} {message:23:17}

Would love for someone to help educate me on this line of code.
using diff command to compare two files which somehow allows for other commands such as message:0:1 and message:23:17 to access it's results.
How does this work?
message=$(diff previousscan.txt scan.txt | grep 192)
#get first char which indicates if the host came up or went away
iostring="${message:0:1}"
#get first ip-number from the list
computer="${message:23:17}"
#show ip-number in notify if host came up
if [ "$iostring" = \> ]; then
notify-send "$computer online"
fi
#show ip-number in notify if host went away
if [ "$iostring" = \< ]; then
notify-send "$computer offline"
fi
$message is not a command; it is a variable which holds the output of the diff command. The later lines reference substrings; ${message:0:1} is the first character (1 character starting at offset 0) of whatever is stored in $message.
A simple example to show the substring mechanism:
$ message="abcdefghijklmnop"
$ echo ${message:0:1}
a
$ echo ${message:7:3}
hij
The construction foo=$(bar) runs the command bar in a subshell, and places the output you would normally see in your terminal if you simply ran the command bar in the variable $foo.

Sometimes Getting Incorrectly Generated Passwords

I am using the following function to generate passwords:
genpasswd() {
local l=$1
[[ "$l" == "" ]] && l=15
echo $(tr -dc "[:print:]" < /dev/urandom | head -c ${l})
}
This works almost all of the time; however, every now and then I get a password with a strange character or a password that is not the correct length. This can be demonstrated by running the following command:
COUNTER=0; while [ $COUNTER -lt 1000000 ]; do; pass=$(genpasswd); if [ ${#pass} -ne 15 ]; then; echo $pass; fi; done
It generated the following output for me for instance (I cut it off after a little while):
G/bteEpm
U^e^!7oT\hGC3)
S7Dcpio1GlQTM
CYOyMMA
$Ze^li
&EKl}'o]u7[]T5
i|y9#6P7?
I9)y
3HIZ"0S%65X[md
sIiWTHk2&0>k}e
O
iy3WET^1q+|(
Mh'6A:Z^;lP-x`
e
htQ%sGl4LWl2
:hhHeoG(Hd%(ct
lp?[Qw'~Z994iy
{SK"
giUI|L4Vz
d0G]+?x|
baR
G0B[sqBr,sXiZ:
o&vi[D4HA/uRwg
^(a(UX40ubapQJ
wh6ZW)]"=\O_:n
pNUapE`t$k#b1K
A`6^)?|'
Xt$vA
M8t!#T(J2#
a
B]FazJBd6#2iY1
\7d-V$O+LQcn8#
6<f-\%GNXKNqU'
W}kY{,N&WU[Gc1
]GPrZn678Vy0c`
O#"o#0zQh^g.('
1Sy).
]5J#rA-S
nk9g
(0{z][,M[
P5vB9[5A1C#h`o
_PC+r1S9$hg%CS
T1p-M6iB9t`-_F
o8RR-V`(|&M#'8
OD-k=SMu}7N$#C
2#qus0),FNMhIq
z$mzQ>kW9;HZ}!
dn#KA
Q>6,35gu
2Uj]_&|TbG"ZNo
1:U)JFjou{SaR9
vq}O}IV}s9:cT#
*[%(1U:lbB.NR!
w{znznAf.'6#UC
6_1m/E0\XaHv`V
q3#yuQ0mY+\pF&
%=,l^vE+]')[gO
AT4cMA5{p<Q'M:
xC9jR'I'Vx=LFQ
+RyVQ6%j{nTz>u
=oVpz+Uw+#CV?Q
KRkH
s'sl$Bk:$
f[}2*>^|q_5_|)
u>l}%*I.O/1ea!
&+joM"k-0`Dz-u
t
3
#`H[/?X!]A
[(p)*,4P>OB.K]
PVv}w#/RAw4#$M
x.S[GjK2Y?-O|A
f5NoAZ\ofTrkg&
DYcij8)]d3?<Yi
F+d#pK>kc7KtoP
7#EPtAe$QSw#%s
[P*I.k?DdM}lZc
jkuL37]OGu`CW\
q.c2%qm"'7r(se
vFwGk5;pj08>5
.DXE
lHhj%8.~
W<}wun]hMym%/<
!}7<JpP`,-h~%^
9UW?L=sEvymmI^
4#w4/E|=}5bQpb
^C
Why is it doing this? I assume some sort of escape character issue or something but I can't figure it out. How can I fix it?
Thanks to #hek2mgl I now have the following function which does not have the same problem:
genpasswd() {
local len=$1
[[ -z $len ]] && len=15
pwgen -syN 1 "$len"
}

BASH script checking log files for current and previous month

I have been working on this on and off for the last two months, and despite how many times I look at it, I can't make it work.
This script checks daily log files for a user defined variable, so they don't have to look through every one manually. It worked great checking the current month, but if the user wants to check back 20 days, and today is the 12th of this month, I wanted to be able to then go back to the previous month (not look for the log file with a date of 20150399 and so on). I have checked the logic for my date/day computations, and they seem okay (if there is a better way to do that in BASH, I am open to suggestions). What happens when I try to debug is unexpected end of file. I am somewhat new to writing scripts that contain more than 20 or so lines, but I just can't come up with what I am missing.
I have tried various fixes, to no avail, but I think this is the last iteration.
Ideas?
#!/bin/bash
########################################################
# multi_log_chk.sh
# This script will take input from the user and report which
# CyberFusion MFT logs contain what the user is looking for.
# Hopefully this will save the user having to search through every
# stinking log file to find what they are looking for.
# 20150406 pxg007 started typing
# 20150413 pxg007 added && comparison for back out (line 28)
# added message for no entries found (line 32, 38, 48-52)
# Added some further description (line 16)
# 20150424 pxg007 Added logic to calculate previous month and if necessary, year. (Lines 16-24, 60-78 )
#
########################################################
currDate=`date +%d%B%C%y`
currDay=`date +%d`
currMnth=`date +%m`
currYear=`date +%C%y`
case $currMnth in #Let's establish number of days for previous month
05 | 07 | 10 | 12 ) lastMnthD=30;;
01 |02 | 04 | 06 | 09 | 08 | 11 ) lastMnthD=31;;
03 ) lastMnthD=28;; ##and screw leap year
esac
if [ $currMnth -eq 01 ]; then ##accounting for January
lastMnth=12
else
lastMnth=$((currMnth-1))
fi
if [ $lastMnth -eq 12 ]; then ## accounting for Dec of previous year
lastMnthYr=$((currYear-1))
else
lastMnthYr=$currYear
fi
echo "This script will find entries for your query in whatever available MFT logs you request."
echo " "
echo "For instance - how many log files have transfer entries with \"DOG\" in them?"
echo " "
echo "I also will also give an estimate of how many transfers per log file contain your query, give or take a couple."
echo " "
echo "This search is case sensitive, so \"DOG\" is *** NOT *** the same as \"dog\""
echo " "
read -p "What text you are looking for? Punctuation is okay, but no spaces please. " looking ### what we want to find
echo " "
echo "Today's date is: $currDate."
echo " "
read -p "How many days back do you want to search(up to 25)? " daysBack ### How far back we are going to look
if [ "$daysBack" == 0 ] && [ "$daysBack" >> 25 ]; then
echo "I said up to 25 days. We ain't got more than that!"
exit 1
fi
echo " "
echo "I am going to search through the last $daysBack days of log files for:\"$looking\" "
echo " "
read -p "Does this look right? Press N to quit, or any other key to continue: " affirm
if [ "$affirm" = N ] && [ "$affirm" = n ]; then ###Yes, anything other than "N" or "n" is a go
echo "Quitter!"
exit 1
else
nada=0 ### Used to test for finding anything
backDate=$((currDay-daysBack)) ### current month iterator (assuming query covers only current month)
if (("$daysBack" => "$currDay")); then ## If there are more logs requested than days in the month...
lastMnthCnt=$((daysBack-currDay)) ### how many days to check last month
lastMnthStrt=$((lastMnthD-lastMnthCnt)) ## last month start and iterator
backDate=$(currDay-(daysBack-lastMnthCnt)) # Setting the iterator if we have to go back a month
while (("$lastMnthStrt" <= "$lastMnthD" )); do
foundIt=$(grep "$looking" /CyberFusion/log/Log.txt."$lastMnthYr$lastMnth$lastMnthStrt" | parsecflog | wc -l )
howMany=$((foundIt/40+1)) ### Add one in case there are less than 40 lines in the record.
if (("$foundIt" > 0))
then
nada=$((nada+1))
echo "Log.txt.$lastMnthYr$lastMnth$lastMnthStrt contains $looking in approximately $howMany transfer records."
lastMnthStrt=$((lastMnthStrt+1))
echo " "
else
lastMnthStrt=$((lastMnthStrt+1))
fi
fi
backDate=$((currDay-daysBack)) ### current month iterator (assuming query covers only current month)
while (("$backDate" <= "$currDay")); do
foundIt=$(grep "$looking" /CyberFusion/log/Log.txt."$backDate" | parsecflog | wc -l )
howMany=$((foundIt/40+1)) ### Add one in case there are less than 40 lines in the record.
if (("$foundIt" > 0))
then
nada=$((nada+1))
echo "Log.txt.$backDate contains $looking in approximately $howMany transfer records."
backDate=$((backDate+1))
echo " "
else
backDate=$((backDate+1))
fi
if [ "$nada" \< 1 ]
then
echo " "
echo "I found no entries for $looking in any log file."
fi
You are missing the keyword 'done' on lines 81 and 96 and also a final 'fi' keyword on the last line.
Also as others suggested you can do
date -d "20 days ago" +"%d%B%C%y"
to easily get dates in the past

Bash IF not working as expected

I am trying to have a function, called from PS1 which outputs something in a different colour, depending on what that something is.
In this case it's $?, the exit status of a program.
I am trying to get this function to output red text if the exit status is anything other than 0.
I have tried all possible variations of this, ways of representing that variable in the conditions and so forth and it just isn't working.
Instead of outputting what I expect it's just either always $LRED in one variation of this IF, or always $HII in another variation of this IF.
All relevant BASH is posted below, can you guys offer any insight?
...
# Custom Colour Alias
NM="\[\033[0;38m\]" # No background and white lines
HI="\[\033[1;36m\]" # Username colour
HII="\[\033[0;37m\]" # Name colour
SI="\[\033[1;32m\]" # Directory colour
IN="\[\033[0m\]" # Command input color
LRED="\[\033[1;31m\]"
BRW="\[\033[0;33m\]"
...
exitStatus ()
{
if [ $? -ne 0 ]
then
echo "$LRED\$?"
else
echo "\$?"
fi
#echo \$?
}
...
export PS1="\n$HII[ $LRED\u $SI\w$NM $HII]\n[ \! / \# / $(exitStatus) $HII]$LRED $ $IN"
CODE BASED ON SOLUTION
This is what I did based on the accepted answer below.
# Before Prompt
export PROMPT_COMMAND='EXSO=$?;\
if [[ $EXSO != 0 ]];\
then\
ERRMSG="$LRED$EXSO";\
else\
ERRMSG="$EXSO";\
fi;\
PS1="\n$HII[ $LRED\u $SI\W$NM $HII\! / \# / $ERRMSG $HII] $SI$ $IN";'
Problem is that your assignment to PS1 is only evaluated once, thus exitStatus is only called once. As Nirk also mentions you should use PROMPT_COMMAND. Set it to the command you want executed before every new prompt is displayed. An example:
PROMPT_COMMAND='if [ $? -ne 0 ]; then echo -n FAIL:;fi'
Will yell FAIL: before every new prompt if the previous command failed:
mogul#linuxine:~$ date
Sun Sep 29 21:13:53 CEST 2013
mogul#linuxine:~$ rm crappy_on_existent_file
rm: cannot remove ‘crappy_on_existent_file’: No such file or directory
FAIL:mogul#linuxine:~$

Resources