I'm trying to check filed on log file , line by line. For each line, check if specific filed is empty or field have wrong value .
And print line with the error message
#!/bin/bash
LOG=/log/output.log
x=false
while read -r line; do
if
(echo $line | awk -F'|' '{if ($8=="") print "Application is empty"}') ||
(echo $line | awk -F'|' '{if ($9=="") print "Http request method is empty"}') ||
(echo $line | awk -F'|' '{if ($7=="") print "Operation is empty"}')
(echo $line | awk -F'|' '{if ($13 !~ /0|1|2/) print "result(0,1,2) has a wrong value"}')
then
echo $line
fi
done < "$LOG"
Actual results:
9f2b|EDR|V1|0|V1|2019-05-14|7||||2019-05-14T08:00:40.095Z|1|0|14|19|||XXXXX|||||||||897|||||||||5b8689707|||||||
Application is empty
9f2b|EDR|V1|0|V1|2019-05-14|7||||2019-05-14T08:00:40.095Z|18|0|||||XXXXX|||||||||1234|||||||||5b868007|||||||
Application is empty
42e2|EDR|V1|0|V1|2019-05-14|7||||2019-05-14T08:00:42.532Z|22|0|||||XXXXX|||||||||235|||||||||3b6959ae||||||||
Application is empty
83ac|EDR|V1|0|V1|2019-05-14|7||||2019-05-14T08:00:42.758Z|8|0|||||XXXXX|||||||||789|||||||||5945548f|||||
Expected Result:
Application is empty
Operation is empty
Http request method is empty
83ac|EDR|V1|0|V1|2019-05-14|7||||2019-05-14T08:00:42.758Z|8|0|||||XXXXX|||||||||789|||||||||5945548f|||||
awk reads every line of the file, so there's no need for your while loop with echos. Just do:
awk -F\| ' {b=1}
$8=="" { print "Application is empty"; b=0 }
$9=="" { print "Http request method is empty"; b=0 }
$7=="" { print "Operation is empty"; b=0 }
$13 !~ /0|1|2/ {print "result(0,1,2) has a wrong value"; b=0 }
b
' /log/output.log
The main problem with your approach is that the command you are evaluating with if always succeeds, so you always print the line. To make awk fail, you would have to add exit statements to each of the awk statements. (eg echo $line | awk -F'|' '$8=="" {print "Application is empty"; exit 1 }'
This will print multiple error messages if a line fails multiple conditions, which yours will not do if you add the exit statements and short circuit the || operators.
If you want to print only one error message, you could do:
awk -F\| '
$8=="" { print "Application is empty"; next }
$9=="" { print "Http request method is empty"; next }
$7=="" { print "Operation is empty"; next }
$13 !~ /0|1|2/ {print "result(0,1,2) has a wrong value"; next }
' /log/output.log
Or, if you want to print multiple error messages, but all on one line, you could do:
awk -F\| ' {s=""}
$8=="" { printf "Application is empty"; s="; " }
$9=="" { printf "%sHttp request method is empty", s; s="; " }
$7=="" { printf "%sOperation is empty", s; s="; " }
$13 !~ /0|1|2/ {printf "%sresult(0,1,2) has a wrong value", s; s="; " }
s { print ""}
!s
' /log/output.log
Related
i have this script in bash:
#!/bin/bash
dir="/home/dortiz/Prueba"
for i in $dir/*
do
cat $i | awk '{print $1" " $2" " $3" " $4"\n " $5}' | \
awk '/gi/{print ">" $0; getline; print}' | \
awk '$3>20.00 {print $0; getline; print;}' \
> "${i}.outsel"
done
cd /home/dortiz/Prueba
mv *.outsel /home/dortiz/Prueba2
and i would like to set an argument to change the value after ""awk '$3>"" in an easy way from my main program that will call this script.
i have read something about getopts but i dont uderstand it at all
Thanks a lot in advance
The simplest way is to just pass an argument to your script:
yourscript.sh 20.0
Then in your script
#!/bin/bash
value=$1 # store the value passed in as the first parameter.
dir="/home/dortiz/Prueba"
for i in $dir/*; do
awk '{print $1" " $2" " $3" " $4"\n " $5}' "$i" |
awk '/gi/{print ">" $0; getline; print}' |
awk -v val="$value" '$3>val {print $0; getline; print;}' > "${i}.outsel"
# ^^^^^^^^^^^^^^^
done
...
and the cat|awk|awk|awk pipeline can probably be written like this:
awk -v val="$value" '
$3 > val {
prefix = /gi/ ? ">" : ""
print prefix $1 " " $2" " $3" " $4"\n " $5
}
' "$i" > "$i.outsel"
I have been trying for hours on a certain parts of my program, I will be greatful if someone can help me. I still cannot solve the problem of adding a new line called "Book is not found" at the bottom when 0 record is found, how do i do that? How do i make the search to be case insensitive also?
awk -F':' -v search="$title" '{ if($1 == search) {printf "%s, %s,%s,%s,%s\n", $1, $2, $3, $4, $5; i++ } else {printf "Book not found"} }END {printf "%d records found\n", i}' BookDB.txt
This give me things like
Title:dwaf
Author:
Book not found
Book not found
Book not found
Total, 0 record found
instead of
Title:wdawd
Author:
Book not found
Total, 0 record found
If a record exist, it will show
Book not found
Book not found
Book not found
Book not found
Happy Day book
Happy Day book
Happy Day book
Book not found
3 records found
instead of
Happy Day book
Happy Day book
Happy Day book
3 records found
This does not show the "book not found" message
if [ "$title" != "" ] && [ "$author" = "" ] ; then
awk -F':' -v search="$title" '{ if($1 == search) {printf "%s, %s,%s,%s,%s\n", $1, $2, $3, $4, $5; i++ } else {printf "Book not found"} }END {printf "%d records found\n", i}' BookDB.txt
if [ "$i" < 1 ]; then
echo -n "Book not found"
fi
Next i have a error of case insensitive
i try to change the code
awk -F':' -v search="$title" 'IGNORECASE = 1;/^search/;'
BEGIN { i=0; }
{
if($1 == search) {
printf "%s, %s,%s,%s,%s\n", $1, $2, $3, $4, $5
i++
}
}
END {
if(i >= 1) {
printf "%d records found\n", i
} else {
printf "Book not found\n"
}
}' BookDB.txt
i isn't a shell variable; it's only an awk variable. Thus, you can only reference it inside of awk code (and in the same instance of awk in which it was set).
Consider the following:
awk -F':' -v search="$title" '
BEGIN { i=0; }
{
if($1 == search) {
printf "%s, %s,%s,%s,%s\n", $1, $2, $3, $4, $5
i++
}
}
END {
if(i >= 1) {
printf "%d records found\n", i
} else {
printf "Book not found\n"
}
}' BookDB.txt
Now, if you wanted to do this in bash, instead of awk, I might write it like so:
#!/bin/bash
# ^-- must be /bin/bash, not /bin/sh
title=$1 # or get it from somewhere else, but needs to be assigned
i=0
while IFS=: read -r -a fields; do
if [[ ${fields[0]} = "$title" ]]; then
IFS=,; printf '%s\n' "${fields[*]}"
i=$(( i + 1 ))
fi
done <BookDB.txt
if (( i > 0 )); then
echo "$i books found" >&2
else
echo "Book not found" >&2
fi
For case-insensitivity, I'd change the bash version like so:
title=${1,,} # store title in lower-case
# ...and lower-case the first field when comparing it.
if [[ ${fields[0],,} = "$title" ]]; then
i have written awk to print specyfic lines from space-delimited log file.
if [ ! -f $userlog ]
then
awk -F' ' '$7$8 == "Loginitialized" {print $2" "$3" "$4" "$5" process start"}''$7$8$9 == "Applictionsuccessfullyended" {print $2" "$3" "$4" "$5" process end"}' $preplog > $userlog
fi
output sample (userlog)
2015-03-02 13:14:25 (PID 19594) process start
2015-03-02 22:42:29 (PID 30473) process start
2015-03-02 22:53:20 (PID 30473) process end
2015-03-03 07:16:55 (PID 31078) process start
2015-03-03 14:53:15 (PID 16591) process start
2015-03-03 14:54:10 (PID 18292) process start
I need to use same awk on $preplog but from last line i've printed with it.
i was trying with that but i failed:
if [ -f $userlog ]
then
lastpid=`awk -F' ' 'END{print $4}' $userlog` #pid ostatniego procesu
cat $preplog | awk 'BEGIN{ found=0} /PID /$lastpid/ Log initialized/{found=1} {if (found) {awk -F" " "$7$8 == "Loginitialized" {print $2" "$3" "$4" "$5" process start"}''$7$8$9 == "Applictionsuccessfullyended" {print $2" "$3" "$4" "$5" process end"}"} ;}' >> $userlog
fi
but my awk programing is not strong in me. I literaly have no idea how to bite it.
Let's start by cleaning up you current script:
awk -F' ' '$7$8 == "Loginitialized" {print $2" "$3" "$4" "$5" process start"}''$7$8$9 == "Applictionsuccessfullyended" {print $2" "$3" "$4" "$5" process end"}' $preplog > $userlog
Change that to:
awk '
$7$8 == "Loginitialized" { state="start" }
$7$8$9 == "Applictionsuccessfullyended" { state="end" }
state { print $2, $3, $4, $5, "process", state; state="" }
' "$preplog" > "$userlog"
for improved robustness and clarity, and to remove unnecessary constructs and redundancy.
For what you want to do next though - see if reading https://stackoverflow.com/a/18409469/1745001 helps and if not edit your question to show sample input and expected output plus a clearer, more detailed explanation of what you're trying to do.
Hang on, I think I just had a revelation after re-reading your attempted code segment - MAYBE what you want is to only analyze and operate on the last line of the input file instead of every line of it. If so that'd be:
awk '
{ var1=$7$8; var2=$7$8$9 }
END {
if (var1 == "Loginitialized") { state="start" }
if (var2 == "Applictionsuccessfullyended") { state="end" }
if (state) { print $2, $3, $4, $5, "process", state; state="" }
}
' "$preplog" > "$userlog"
If that's not what you wanted you'll need to provide more information.
I want to monitor Disk IO of a server and want to set different threshold for
the warning if %busy above 60% and
error if %busy above 80%.
From sar output need to find error disk with more than 80% busy and on 2nd line need disk more than 60 but less than 80% busy if disk not crossing theresholds then it should print 'ok' for respective category
I've created a script which will do this, but is there a better way to run sar just one time and use the same output for error & warning threshold? My system is AIX 6.1.
sar -d 2 2 |sed -n '/Average/,$p'|
sed -e 's/Average//g'|
awk 'NR> 1 {
if ($2 >80) {
print $1, $2; f=1
}
}
END { if (!f) print "ok" }'
sar -d 2 2 |
sed -n '/Average/,$p'|
sed -e 's/Average//g'|
awk 'NR> 1{ if ($2 >60 && $2 <81) { print $1, $2; f=1; } }
END { if (!f) print "ok" }'
Pls find my updated script below
bash-3.00$ cat diskio.sh
#!/bin/bash
a=`sar -d 2 2 |sed -n '/Average/,$ {s/Average//g;p;}'`
e=`$a|awk 'NR> 1{ if ($2 >40) { print $1, $2; f=1; } }
END { if (!f) print "ok" }'`
w=`$a|awk 'NR> 1{ if ($2 <40 && $2 >20) { print $1, $2; f=1; } }
END { if (!f) print "ok" }'`
echo $e
echo $w
bash-3.00$ sh -v diskio.sh
#!/bin/bash
a=`sar -d 2 2 |sed -n '/Average/,$ {s/Average//g;p;}'`
e=`$a|awk 'NR> 1{ if ($2 >40) { print $1, $2; f=1; } }
END { if (!f) print "ok" }'`
diskio.sh[3]: hdisk18: not found.
w=`$a|awk 'NR> 1{ if ($2 <40 && $2 >20) { print $1, $2; f=1; } }
END { if (!f) print "ok" }'`
diskio.sh[4]: hdisk18: not found.
echo $e
ok
echo $w
ok
bash-3.00$ sar -d 2 2 |
sed -n '/Average/,$ {s/Average//g;p;}'|
awk 'NR> 1{ if ($2 >40) { print $1, $2; f=1; } }
END { if (!f) print "ok" }'
hdisk79 135
hdisk101 85
bash-3.00$
sed -n '/Average/,$p'|sed -e 's/Average//g'
could be replaced by
sed -n '/Average/,$ {s/Average//g;p;}'
but sed does not make any evaluation of value (compare) easily so the awk for all is a better solution
This can be shorten some.
sar -d 2 2 | awk '/Average/ {t=1;next} t && NR>1 && $2>80 {print $1,$2;f=1} END { if (!f) print "ok" }'
This should do what you want
sar -d 2 2 | awk '/Average/ {t=1;next} t && NR>1 {if ($2>80) h=1;if ($2>60) l=1} END {if (h) {print "80+";exit};if (l) print "60+"}'
It will print 60+ if its above 60 and 80+ above 80
Or store it to a variable
status=$(sar -d 2 2 | awk '/Average/ {t=1;next} t && NR>1 {if ($2>80) h=1;if ($2>60) l=1} END {if (h) {print "H";exit};if (l) print "L"}')
Here it set status to L for 60+ and H for 80+
This should also print the status:
sar -d 2 2 | awk '/Average/ {t=1;next} t && NR>1 {if ($2>80) h=1;if ($2>60) {print $1,$2;l=1}} END {if (!l) print "ok"}'
I have an input file that contains:
123,apple,orange
123,pineapple,strawberry
543,grapes,orange
790,strawberry,apple
870,peach,grape
543,almond,tomato
123,orange,apple
i want the output to be:
The following numbers are repeated:
123
543
is there a way to get this output using awk; i'm writing the script in solaris , bash
sed -e 's/,/ , /g' <filename> | awk '{print $1}' | sort | uniq -d
awk -vFS=',' \
'{KEY=$1;if (KEY in KEYS) { DUPS[KEY]; }; KEYS[KEY]; } \
END{print "Repeated Keys:"; for (i in DUPS){print i} }' \
< yourfile
There are solutions with sort/uniq/cut as well (see above).
If you can live without awk, you can use this to get the repeating numbers:
cut -d, -f 1 my_file.txt | sort | uniq -d
Prints
123
543
Edit: (in response to your comment)
You can buffer the output and decide if you want to continue. For example:
out=$(cut -d, -f 1 a.txt | sort | uniq -d | tr '\n' ' ')
if [[ -n $out ]] ; then
echo "The following numbers are repeated: $out"
exit
fi
# continue...
This script will print only the number of the first column that are repeated more than once:
awk -F, '{a[$1]++}END{printf "The following numbers are repeated: ";for (i in a) if (a[i]>1) printf "%s ",i; print ""}' file
Or in a bit shorter form:
awk -F, 'BEGIN{printf "Repeated "}(a[$1]++ == 1){printf "%s ", $1}END{print ""} ' file
If you want to exit your script in case a dup is found, then you can exit a non-zero exit code. For example:
awk -F, 'a[$1]++==1{dup=1}END{if (dup) {printf "The following numbers are repeated: ";for (i in a) if (a[i]>1) printf "%s ",i; print "";exit(1)}}' file
In your main script you can do:
awk -F, 'a[$1]++==1{dup=1}END{if (dup) {printf "The following numbers are repeated: ";for (i in a) if (a[i]>1) printf "%s ",i; print "";exit(-1)}}' file || exit -1
Or in a more readable format:
awk -F, '
a[$1]++==1{
dup=1
}
END{
if (dup) {
printf "The following numbers are repeated: ";
for (i in a)
if (a[i]>1)
printf "%s ",i;
print "";
exit(-1)
}
}
' file || exit -1