I am developing script which will grep logs of last one hour and check any exception and send email for solaris platform.
I did following steps
grep -n -h date +'%Y-%m-%d %H:%M' test.logs
above command gives me line number and then i do following
tail +6183313 test.log | grep 'exception'
sample logs
2014-02-17 10:15:02,625 | WARN | m://mEndpoint | oSccMod | 262 - com.sm.sp-client - 0.0.0.R2D03-SNAPSHOT | 1201 or 101 is returned as exception code from SP, but it is ignored
2014-02-17 10:15:02,625 | WARN | m://mEndpoint | oSccMod | 262 - com.sm.sp-client - 0.0.0.R2D03-SNAPSHOT | SP error ignored and mock success returned
2014-02-17 10:15:02,626 | INFO | 354466740-102951 | ServiceFulfill | 183 - org.apache.cxf | Outbound Message
Please suggest any better alternative to perform above task.
With GNU date, one can use:
grep "^$(date -d -1hour +'%Y-%m-%d %H')" test.logs | grep 'exception'| mail -s "exceptions in last hour of test.logs" ImranRazaKhan
The first step above is to select all log entries from the last hour. This is done with grep by looking for all lines beginning with the year-month-day and hour that matches one hour ago:
grep "^$(date -d -1hour +'%Y-%m-%d %H')" test.logs
The next step in the pipeline is to select from those lines the ones that have exceptions:
grep 'exception'
The last step in the pipeline is to send out the mail:
mail -s "exceptions in last hour of test.logs" ImranRazaKhan
The above sends mail to ImranRazaKhan (or whatever email address you chose) with the subject line of "exceptions in last hour of test.logs".
The convenience of having the -d option to date should not be underestimated. It might seem simple to subtract 1 from the current hour but, if the current hour is 12am, then we need to adjust both the day and the hour. If the hour was 12am on the first of the month, we would also have to change the month. And likewise for year. And, of course, February requires special consideration during leap years.
Adapting the above to Solaris:
Consider three cases:
Under Solaris 11 or better, the GNU date utility is available at /usr/gnu/bin/date. Thus, we need simply to specify a path for date:
grep "^$(/usr/gnu/bin/date -d -1hour +'%Y-%m-%d %H')" test.logs | grep 'exception'| mail -s "exceptions in last hour of test.logs" ImranRazaKhan
Under Solaris 10 or earlier, one can download & install GNU date
If GNU date is still not available, we need to find another way to find the date and time for one hour ago. The simplest workaround is likely to select a timezone that is one hour behind your timezone. If that timezone was, say, Hong Kong, then use:
grep "^$(TZ=HongKong date +'%Y-%m-%d %H')" test.logs | grep 'exception'| mail -s "exceptions in last hour of test.logs" ImranRazaKhan
You can do like this:
dt="$(date -d '1 hour ago' "+%m/%d/%Y %H:%M:%S")"
awk -v dt="$dt" '$0 ~ dt && /exceltion/' test.logs
Scanning through millions lines of log sounds terribly inefficient. I would suggest changing log4j (what it looks like) configuration of your application to cut a new log file every hour. This way, tailing the most recent file becomes a breeze.
Related
I saw it posted here that this would work but it is not working for me. I need something short and sweet like this command. using the python version of rsstail.
rsstail -dl -e 1 -U -a -u https://threatpost.com/feed/ -n 10 | grep -A 2 "2021/03/15 20[1-5]"
This should grab the last 5 hours but it doesn't.
Sample line from the feed follows
Updated: 2021/03/12 21:42:59 Title: Critical Security Hole Can Knock Smart Meters Offline Author: Tara Seals Link: https://threatpost.com/critical-security-smart-meter-offline/164753/ Description: Unpatched Schneider Electric PowerLogic ION/PM smart meters are open to dangerous attacks
You were heading on the right track with the grep - however there wasn't going to be a match on the date because 20[1-5] matches two digits and a digit in the range 1-5 in a row after the specific date - that would match
2021/03/15 2022 but not 2021/03/15 20:22.
Assuming you are in daylight hours you don't have to worry about spanning two days - imagine you ran at 2:00am you'd need yesterday 21:XX, 22:XX, 23:XX and today 00:XXam, 01:XXam.
So say you run at 11.am today - previous 5 hours 6/7/8/9/10 .. so you could do something like this.
grep -A 2 -E -e '2021/03/17 (06|07|08|09|10):'
OR even
grep -A 2 -E -e '2021/03/17 (0[6789]|10):'
You can auto-generate some of the query like this (again I've ignored cross-over of hour) NOTE: OSX Date & GNU date are different - this is OSX example -
FROMMIN=$( date -v -4M +'%M' )
TOMIN=$( date +'%M' )
## GIVES like this 2021/03/17 20:(44|45|46|47|48|)
MATCH=$( echo $( date +'%Y/%m/%d %H:(' )$( seq -s "|" $FROMMIN 1 $TOMIN )')' )
grep -A 2 -E -e "$MATCH"
I have a log file with entries like this:
24/09/20 | 11:22:56am | Server1 | Backup Done
28/09/20 | 10:44:05am | Server1 | Error in Config File
10/10/20 | 04:22:10am | Server1 | Error in Config File
How can I extract only the lines which are between
[Today's Date] <-> [Today's Date - 30Days]
After searching on web I found this command to work but it gives a error.
sed -n "/$(date --date='-30 day' '+%d/%m/%y')/,/$(date +'%d/%m/%y')/p"
Error
cat logs.txt | sed -n "/$(date --date='-10 day' '+%d/%m/%y')/,/$(date '+%d/%m/%y')/p"
sed: -e expression #1, char 5: unknown command: `1'
Can anyone please help me to extract only lines which are 30 days old from current date.
Use GNU date to get the date for 30 days ago.
Then reverse the format, to %y%m%d, in order to apply alphabetical comparison with awk.
start=$(date --date='30 days ago' +'%y%m%d')
awk -v start="$start" '{split($1, a, "/")} a[3]a[2]a[1] >= start' file
Note: In general, date formats that can be directly sorted or compared, as strings, like the ISO 8601 date, or as numbers, like the Unix timestamp, should be preferred for timestamped files.
i try to delete all days of a csv file which not matched last days. But I find not the right solution.
date,price
2018-07-02,162.17
2018-06-29,161.94
2018-06-28,162.22
2018-06-27,162.32
2018-06-12,163.01
2018-06-11,163.53
2018-05-31,164.87
2018-05-30,165.59
2018-05-29,165.42
2018-05-25,165.96
2018-05-02,164.94
2018-04-30,166.16
2018-04-27,166.69
The output I want become
date,price
2018-06-29,161.94
2018-05-31,164.87
2018-04-30,166.16
I try it with cut + grep
cut -d, -f1 file.csv | grep -E "28|29|30"
Work but bring nothing when combine -f1,2.
I find csvkit which seem to me the right tool, but I find not the solution for multiple grep.
csvgrep -c 1 -m 30 file.csv
Bring me the right result but how can combine multiple search option? I try -m 28,29,30 and -m 28 -m 29 -m 30 all work not. Best it work with last day of every month.
Maybe one have here a idea.
Thank you and nice Sunday
Silvio
You want to get all records of the LAST day of the month. But months vary in length (28-29-30-31).
I don't see why you used cut to extract the first field (the date part), because the data in the second field does not look like dates at all (xx-xx).
I suggest to use grep directly to display the lines that matches the following pattern mm-dd; where mm is the month number, and dd is the last day of the month.
This command should do the trick:
grep -E "01-31|02-(28|29)|03-31|04-30|05-31|06-30|07-31|08-30|09-31|10-30|11-31|12-30" file.csv
This command will give the following output:
2018-05-31,164.87
2018-04-30,166.16
This question already has answers here:
Filter log file entries based on date range
(5 answers)
Closed 6 years ago.
I have an application which emits logs in this format:
00:00:10,799 ERROR [stderr] (http-prfilrjb08/10.1.29.34:8180-9) {}:return code: 500
I would need to monitor for new ERRORs in the log file, happened in the last hour. Looking at some tutorials I've come up with the following grep:
grep "^$(date -d -1 hour +'%H:%M:%S')" /space/log/server.log | grep 'ERROR'
However nothing is grepped! Can you help me to fix it ?
Thanks!
You need quotes around the -1 hour and also you want to remove the seconds and minutes from the output (your current solution finds data only for the first second 1 hour ago):
grep "^$(date -d '-1 hour' +'%H')" /space/log/server.log | grep 'ERROR'
grep -E "^($(date -d '-1 hour' '+%H')|$(date '+%H')):[0-9]{2}:[0-9]{2}" /space/log/server.log | grep 'ERROR'
Let's take a look at the parts
grep -E tells grep to use extended regular expressions (so we don't need to escape all those brackets)
date -d '-1 hour' '+%H' prints the previous hour. Similarly date '+%H' prints the current hour. These need to be evaluated at runtime and captured in a capture group, that's why we have the (date|date) structure (you'll probably want some data not only from the previous hour, but the current running hour).
Next you need to specify that you are indeed looking at timestamps. We use : to delimit hours, minutes and seconds. A two-digit number group can be matched with the [0-9]{2} regexp (this is basically identical to [0-9][0-9] but shorter)
There you go.
Ps. I'd recommend sed.
I'm running a background script that involves reading a log file every 5 minutes formatted like this
21:25:57 [INFO] event from 5 minutes ago
21:26:54 [INFO] potentially relevant event
21:28:26 [INFO] some event
21:30:06 [INFO] another event
except I only want to look at a the lines that were printed within the last 5 minutes. Essentially I need to find the line where date -s "$logdate" +%s <= date -d "5 min ago" +%s closest to the end of the file and ignore all the lines before it.
Unfortunately there are no dates in the timestamps, so it makes it tricky in that the timestamps will recur every day unless I restart the server and reset the log (I'd prefer not to). Using tac instead of cat might be more effective since I only care about the lines starting from the end of the file.
If you have GNU date:
$ date +%T
04:56:32
$ date -d "5 min ago" +%T
04:51:45
And you can do:
tac logfile | awk -v start=$(date -d "5 min ago" +%T) '$1 < start {exit} 1' | tac