shell script + match error words in the log file - shell

please advice how to match the ERROR strings that comes only after "]" char
by awk or sed
[Mon Jan 30 21:14:01 IST 2012] ERROR file /etc/ghy.txt not found
My target is to count all ERROR words that appears only after "]” character in the log file
remark - between “]” and ERROR must be one space or more

My target is to count all ERROR words that appears only after "]”
character in the log file
remark - between “]” and ERROR must be one space or more
then you don't need those nuclear heads like awk, sed even perl. grep does it for you like this:
grep -Pc ']\s+ERROR' yourLogFile
small test:
kent$ echo "[Mon Jan 30 21:14:01 IST 2012] ERROR file /etc/ghy.txt not found "|grep -Pc ']\s+ERROR'
1

AWK:
awk -F"] " '/ERROR/{print $2}' inputfile
Test:
[jaypal:~] echo "[Mon Jan 30 21:14:01 IST 2012] ERROR file /etc/ghy.txt not found " | awk -F"] " '/ERROR/{print $2}'
ERROR file /etc/ghy.txt not found
Perl:
perl -pe 's/.*(?<=] )(.*)/$1/' inputfile
Test:
echo "[Mon Jan 30 21:14:01 IST 2012] ERROR file /etc/ghy.txt not found " | perl -pe 's/.*(?<=] )(.*)/$1/'
ERROR file /etc/ghy.txt not found
Count no. of occurrences:
[jaypal:~/Temp] cat file
[Mon Jan 30 21:14:01 IST 2012] ERROR file /etc/ghy.txt not found
[Mon Jan 30 21:14:01 IST 2012] ERROR file /etc/ghy.txt not found
[Mon Jan 30 21:14:01 IST 2012] ERROR file /etc/ghy.txt not found
[Mon Jan 30 21:14:01 IST 2012] ERROR file /etc/ghy.txt not found
[Mon Jan 30 21:14:01 IST 2012] ERROR file /etc/ghy.txt not found
[Mon Jan 30 21:14:01 IST 2012] ERROR file /etc/ghy.txt not found
[Mon Jan 30 21:14:01 IST 2012] ERROR file /etc/ghy.txt not found
[Mon Jan 30 21:14:01 IST 2012] ERROR file /etc/ghy.txt not found
[Mon Jan 30 21:14:01 IST 2012] ERROR file /etc/ghy.txt not found
[jaypal:~/Temp] awk -F"] " '/ERROR/{a[NR]=$2}END{print "count is " length(a)}' file
count is 9

This might work for you:
grep -c '\] \+ERROR' file
Or
grep -c '\][[:space:]]\+ERROR' file
Or
sed '/\]\s\+ERROR/!d' file | wc -l

Here is a shell only snippet, that should be faster than using any external programs (for small file reads) since it only uses shell builtins. It can be modified to handle errors case by case while running in a daemon mode (by tail-ing the log file to a fifo instead of reading it directly and modifying the case conditionals)
not the intended use of echo, but it does reduce spaces/tabs to 1 space
FILE="logfile"
ERRORS=0
while read LINE || [ "$LINE" ]; do
case "`echo $LINE`" in
*\]" "ERROR*)ERRORS=$(($ERRORS+1));;
esac
done < "${FILE}"
echo $ERRORS

You can do
sed -n '/] ERROR/p' infile

Related

convert multiple variable output to a table in bash

I have 3 variables, $commonName, $expiryDate and $DaysRemInUnixEpoch. Each variable has 3 lines as below output. I want to display output of all 3 variables in to 3 different columns. I tried looking for solution using printf but no luck. Can anyone please advise if they have done this in the past using printf and how? Any help will be much appreciated.
Below are 3 variables output together in one column. I want to split in to 3 columns having 3 rows in each column.
bash-4.1$ echo -e "$commonName\n$expiryDate\n$daysRemInUnixEpoch"
mycertificate_mycert.mycomp.net
PSIN0P551
ROOTROOTCA
Feb 6 2022 11:57:32 GMT
Jan 9 2023 18:51:25 GMT
Mar 12 2035 18:24:54 GMT
682
1020
5465
bash-4.1$
desired output I am looking for is something like below
mycertificate_mycert.mycomp.net Feb 6 2022 11:57:32 GMT 682
PSIN0P551 Jan 9 2023 18:51:25 GMT 1020
ROOTROOTCA Mar 12 2035 18:24:54 GMT 5465
With bash (Process Substitution), paste and column:
paste -d ';' <(echo "$commonName") <(echo "$expiryDate") <(echo "$daysRemInUnixEpoch") | column -s ';' -t
Output:
mycertificate_mycert.mycomp.net Feb 6 2022 11:57:32 GMT 682
PSIN0P551 Jan 9 2023 18:51:25 GMT 1020
ROOTROOTCA Mar 12 2035 18:24:54 GMT 5465
I assume that your variables do not contain ;.
see: man paste and man column

Find and Echo only the date (with format) in String Output on Bash

I am trying to get the date "+%a %b %d %R:%S %Y" in bash.
here's the sample command and output
$ xscreensaver-command --time
XScreenSaver 5.32: screen non-blanked since Thu Oct 29 12:15:05 2015 (hacks: #184, #60)
I am trying to get the the value Thu Oct 29 12:15:05 2015 on the string.
How can I achieve this?
Try to append with GNU grep:
2>&1 | grep -Po 'since \K.*(?= \()'
Output:
Thu Oct 29 12:15:05 2015

cut from stream witing for EOL/EOF char

I need to cut something from stream, but it seems, that cut cant process last line before EOL or EOF appear.
My bash command:
tail -n 5 -F /data/apache/log/error/error-log | cut -d ' ' -f1-5
log file contains for example:
[Wed Jan 14 09:00:00 2015] [error] Error 1
[Wed Jan 14 09:10:00 2015] [error] Error 2
[Wed Jan 14 09:20:00 2015] [error] Error 3
[Wed Jan 14 09:30:00 2015] [error] Error 4
[Wed Jan 14 09:40:00 2015] [error] Error 5
but my output is:
[Wed Jan 14 09:00:00 2015]
[Wed Jan 14 09:10:00 2015]
[Wed Jan 14 09:20:00 2015]
[Wed Jan 14 09:30:00 2015]
is there a way to make cut process (and output) last, unfinished line if there are all needed columns (first five) streamed already?
My desired output should contain last line from stream:
[Wed Jan 14 09:00:00 2015]
[Wed Jan 14 09:10:00 2015]
[Wed Jan 14 09:20:00 2015]
[Wed Jan 14 09:30:00 2015]
[Wed Jan 14 09:40:00 2015]
//edit
Answers "remove -F" are useless, because I need it for live stream. Whenever apache add new entry into log, I need immediately process it with cut. Without user interaction, without running script again etc. Just process stream in real time in the way, that 'cut' don't wait for EOL before it process (and output) line.
If that is not possible, fine, I try to find workaround, but please, don't tell me to not use live stream when I need live stream!
Finally, after some tests, I found that the problem is not the tail but the cut. Replacing cut by awk fixes the problem:
tail -n 5 -F /data/apache/log/error/error-log | awk '{print $1, $2, $3, $4, $5}'
Is it really necessary -F (follow & retry) option?
Remove it, and it'll work.

sed: convert time(3) seconds in a table into printable date (spamdb)

I get the following from spamdb, where the third field represents the time in seconds since the Epoch.
Cns# spamdb | fgrep TRAPPED
TRAPPED|113.163.117.129|1360836903
TRAPPED|113.171.216.201|1360837481
TRAPPED|122.177.159.61|1360844596
TRAPPED|36.231.9.231|1360865649
TRAPPED|37.146.207.209|1360832096
TRAPPED|212.156.98.210|1360837015
TRAPPED|59.99.160.62|1360839785
TRAPPED|86.127.116.162|1360840492
TRAPPED|92.83.139.194|1360843056
TRAPPED|219.71.12.150|1360844704
I want to sort this table by the time, and print the time field with date -r, such that it's presentable and clear when the event has occurred.
How do I do this in tcsh on OpenBSD?
Sorting with sort is easy, and so is editing with sed; but how do I make sed execute date -r or equivalent?
There are indeed a few obstacles here: first, you basically have to separate the data, and then one part of it is presented as-is, whereas another part has to be passed down to date -r for date formatting, prior to being presented to the user.
Another obstacle is making sure the output is aligned: apparently, it's quite difficult to handle the tab character in the shell, possibly only on the BSDs:
sed replace literal TAB
Replacing / with TAB using sed
Also, as we end up piping this to sh for execution, we have to use a different separator for the fields other than the pipe character, |.
So far, this is the best snippet I could come up with, it seems to work great in my tcsh:
Cns# spamdb | fgrep TRAPPED | sort -n -t '|' -k 3 | sed -E -e 's#\|###g' \
-e 's#^([A-Z]+)#([0-9.]+)#([0-9]+)$#"echo -n \2_"; "date -r \3"#g' | \
xargs -n1 sh -c | awk '{gsub("_","\t",$0); print;}'
37.146.207.209 Thu Feb 14 00:54:56 PST 2013
113.163.117.129 Thu Feb 14 02:15:03 PST 2013
212.156.98.210 Thu Feb 14 02:16:55 PST 2013
113.171.216.201 Thu Feb 14 02:24:41 PST 2013
59.99.160.62 Thu Feb 14 03:03:05 PST 2013
86.127.116.162 Thu Feb 14 03:14:52 PST 2013
92.83.139.194 Thu Feb 14 03:57:36 PST 2013
122.177.159.61 Thu Feb 14 04:23:16 PST 2013
219.71.12.150 Thu Feb 14 04:25:04 PST 2013
36.231.9.231 Thu Feb 14 10:14:09 PST 2013

Remove duplicate entries in a Bash script [duplicate]

This question already has answers here:
How to delete duplicate lines in a file without sorting it in Unix
(9 answers)
Closed 7 years ago.
I want to remove duplicate entries from a text file, e.g:
kavitha= Tue Feb 20 14:00 19 IST 2012 (duplicate entry)
sree=Tue Jan 20 14:05 19 IST 2012
divya = Tue Jan 20 14:20 19 IST 2012
anusha=Tue Jan 20 14:45 19 IST 2012
kavitha= Tue Feb 20 14:00 19 IST 2012 (duplicate entry)
Is there any possible way to remove the duplicate entries using a Bash script?
Desired output
kavitha= Tue Feb 20 14:00 19 IST 2012
sree=Tue Jan 20 14:05 19 IST 2012
divya = Tue Jan 20 14:20 19 IST 2012
anusha=Tue Jan 20 14:45 19 IST 2012
You can sort then uniq:
$ sort -u input.txt
Or use awk:
$ awk '!a[$0]++' input.txt
It deletes duplicate, consecutive lines from a file (emulates "uniq").
First line in a set of duplicate lines is kept, rest are deleted.
sed '$!N; /^\(.*\)\n\1$/!P; D'
Perl one-liner similar to #kev's awk solution:
perl -ne 'print if ! $a{$_}++' input
This variation removes trailing whitespace before comparing:
perl -lne 's/\s*$//; print if ! $a{$_}++' input
This variation edits the file in-place:
perl -i -ne 'print if ! $a{$_}++' input
This variation edits the file in-place, and makes a backup input.bak
perl -i.bak -ne 'print if ! $a{$_}++' input
This might work for you:
cat -n file.txt |
sort -u -k2,7 |
sort -n |
sed 's/.*\t/ /;s/\([0-9]\{4\}\).*/\1/'
or this:
awk '{line=substr($0,1,match($0,/[0-9][0-9][0-9][0-9]/)+3);sub(/^/," ",line);if(!dup[line]++)print line}' file.txt

Resources