So basically I have my output as the following:
<span id="PlayerCount">134,015 people currently online</span>
What I want is a way to trim it to show:
134,015 - 3:24:20AM - Oct 24
Can anyone help? Also note the number may change so is it possible output everything between ">" and the "c" in currently? And add a timestamp somehow?
Using commands from terminal in Linux, so that's called bash right?
Do you perhaps mean something like:
$ echo '<span id="PlayerCount">134,015 people currently online</span>' | sed
-e 's/^[^>]*>//'
-e "s/currently.*$/$(date '+%r %b %d %Y')/"
which generates:
134,015 people 03:36:30 PM Oct 24 2011
The echo is just for the test data. The first sed command will change everything up to the first > character into nothing (ie, delete it).
The second one will change everything from the currently to the end of the line with the current date in your desired format (although I have added the year since I'm a bit of a stickler for detail).
The relevant arguments for date here are:
%r locale's 12-hour clock time (e.g., 11:11:04 PM)
%b locale's abbreviated month name (e.g., Jan)
%d day of month (e.g., 01)
%Y year
A full list of format specifiers can be obtained from the date man page (execute man date from a shell).
A small script which will give you the desired information from the page you mentioned in the comments is:
#!/usr/bin/bash
wget --output-document=- http://runescape.com/title.ws 2>/dev/null \
| grep PlayerCount \
| head -1l \
| sed 's/^[^>]*>//' \
| sed "s/currently.*$/$(date '+%r %b %d %Y')/"
Running this gives me:
pax$ ./online.sh
132,682 people 04:09:17 PM Oct 24 2011
In detail:
The wget bit pulls down the web page and writes it on standard output. The standard error (progress bar) is thrown away.
The grep extracts only lines with the word PlayerCount in them.
The head throws away all but the first of those.
The first sed strips up to the first > character.
The second sed changes the trailing text to the durrent date and time.
Quickhack(tm):
$ people=$(echo '<span id="PlayerCount">134,015 people currently online</span>' | \
sed -e 's/^.*>\(.*\) people.*$/\1/')
$ echo $people - $(date)
134,015 - Mon Oct 24 09:36:23 CEST 2011
produce_OUTPUT | grep -o '[0-9,]\+' | while read count; do
printf "%s - %s\n" $count "$(date +'%l:%M:%S %p - %b %e')"
done
Related
I have a log file with entries like this:
24/09/20 | 11:22:56am | Server1 | Backup Done
28/09/20 | 10:44:05am | Server1 | Error in Config File
10/10/20 | 04:22:10am | Server1 | Error in Config File
How can I extract only the lines which are between
[Today's Date] <-> [Today's Date - 30Days]
After searching on web I found this command to work but it gives a error.
sed -n "/$(date --date='-30 day' '+%d/%m/%y')/,/$(date +'%d/%m/%y')/p"
Error
cat logs.txt | sed -n "/$(date --date='-10 day' '+%d/%m/%y')/,/$(date '+%d/%m/%y')/p"
sed: -e expression #1, char 5: unknown command: `1'
Can anyone please help me to extract only lines which are 30 days old from current date.
Use GNU date to get the date for 30 days ago.
Then reverse the format, to %y%m%d, in order to apply alphabetical comparison with awk.
start=$(date --date='30 days ago' +'%y%m%d')
awk -v start="$start" '{split($1, a, "/")} a[3]a[2]a[1] >= start' file
Note: In general, date formats that can be directly sorted or compared, as strings, like the ISO 8601 date, or as numbers, like the Unix timestamp, should be preferred for timestamped files.
I am trying to convert timestamps read from a file from string to date format so that I can find the difference of 2 dates/timestamps. most of the threads/discussions on web show usage of date argument '-d' to convert the string to epoch or to find the difference of two timestamps Find difference between two dates in bash
But it looks like my environment/OS doesn't support -d date argument. Below are the details of my env:
bash --version
GNU bash, version 3.2.52(1)-release (i386-pc-solaris2.10)
Copyright (C) 2007 Free Software Foundation, Inc.
uname -a
SunOS s01***** 5.10 Generic_147148-26 i86pc i386 i86pc
Sample dates read from file:
START_TIME="09/03/16 - 01:04:56"
END_TIME="09/03/16 - 05:10:44"
Code that I have tried
I have tried to mimic the below code from Find difference between two dates in bash
!# /usr/bin/sh
date1="Sat Dec 28 03:22:19 2013"
date2="Sun Dec 29 02:22:19 2013"
date -d #$(( $(date -d "$date2" +%s) - $(date -d "$date1" +%s) )) -u +'%H:%M:%S'
bash test.sh
date: illegal option -- d
usage: date [-u] mmddHHMM[[cc]yy][.SS]
date [-u] [+format]
date -a [-]sss[.fff]
date: illegal option -- d
usage: date [-u] mmddHHMM[[cc]yy][.SS]
date [-u] [+format]
date -a [-]sss[.fff]
test.sh: line 5: - : syntax error: operand expected (error token is " ")
I don't think syntax error on line 5 is the main culprit cause I didnot find option -d in my date's man page.
In response to comments:
>>> date --version
date: illegal option -- version
usage: date [-u] mmddHHMM[[cc]yy][.SS]
date [-u] [+format]
date -a [-]sss[.fff]
>>> date --help
date: illegal option -- help
usage: date [-u] mmddHHMM[[cc]yy][.SS]
date [-u] [+format]
date -a [-]sss[.fff]
>>> echo $0
bash
Even these arguments are not supported. Apologies if I am committing any silly mistake.
Could someone please give me the equivalent of -d for the env details shared above or a way to find the difference between two dates without using -d.
Thanks in advance
awk mktime has a decent chance of existing on your system:
#!/bin/bash
START_TIME="09/03/16 - 01:04:56"
END_TIME="09/03/16 - 05:10:44"
echo -e "$START_TIME\n$END_TIME" |
tr '/:-' ' ' |
awk '{print "20"$3" "$2" "$1" "$4" "$5" "$6}' |
awk '{printf "%s ", mktime($0)}' |
awk '{print $2 - $1}'
explanation:
echo both time strings
tr converts 09/03/16 - 01:04:56 to 09 03 16 01:04:56
first awk changes 09 03 16 01 04 56 to 2016 03 09 01 04 56
second awk converts 2016 03 09 01 04 56 to epoch time: 1457514296 and prints both on one line: 1457514296 1457529044
third awk subtracts first from second, giving difference in seconds: 14748
the awks could also easily be merged, but here i kept each separate for clarity.
According to the POSIX standard, date does not do date and time math for you. It gets or sets the system date and time, possibly with timezone adjustments. See date(1) and note the lack of a -d option (or indeed, any interesting options!).
The question becomes "How do we do date and time math without date?" The timestamps you provided do not have time zone information, so the usual behavior is to assume local time. Local time is bad. Seriously. Some time zones have crazy discontinuities or didn't meaningfully exist hundreds of years ago (e.g. most American time zones). So while you might be able to hack together a custom solution that works in your particular part of the world during the recent past, it simply will not be robust.
If you can get your timestamps into Unix time, you can just subtract them, and that will give you a mostly but not entirely correct answer. Unfortunately, to the best of my knowledge, that can't be done at the command line. Unix provides strptime(3) to do this from C (and from there you go on to mktime(3), as shown in this answer), but I don't believe there is any fully standard utility which provides a command-line interface for this. You may have to write and compile your own.
Correcting for leap seconds is difficult in the extreme because, to the best of my knowledge, POSIX has never provided a standard interface for finding out when leap seconds have happened in the past. Such an interface would require internet connectivity in order to remain up-to-date, which is likely a non-starter for a number of implementations. Without knowing more about your system and its capabilities, I simply cannot begin to guess at what will or will not work for your use case.
I am developing script which will grep logs of last one hour and check any exception and send email for solaris platform.
I did following steps
grep -n -h date +'%Y-%m-%d %H:%M' test.logs
above command gives me line number and then i do following
tail +6183313 test.log | grep 'exception'
sample logs
2014-02-17 10:15:02,625 | WARN | m://mEndpoint | oSccMod | 262 - com.sm.sp-client - 0.0.0.R2D03-SNAPSHOT | 1201 or 101 is returned as exception code from SP, but it is ignored
2014-02-17 10:15:02,625 | WARN | m://mEndpoint | oSccMod | 262 - com.sm.sp-client - 0.0.0.R2D03-SNAPSHOT | SP error ignored and mock success returned
2014-02-17 10:15:02,626 | INFO | 354466740-102951 | ServiceFulfill | 183 - org.apache.cxf | Outbound Message
Please suggest any better alternative to perform above task.
With GNU date, one can use:
grep "^$(date -d -1hour +'%Y-%m-%d %H')" test.logs | grep 'exception'| mail -s "exceptions in last hour of test.logs" ImranRazaKhan
The first step above is to select all log entries from the last hour. This is done with grep by looking for all lines beginning with the year-month-day and hour that matches one hour ago:
grep "^$(date -d -1hour +'%Y-%m-%d %H')" test.logs
The next step in the pipeline is to select from those lines the ones that have exceptions:
grep 'exception'
The last step in the pipeline is to send out the mail:
mail -s "exceptions in last hour of test.logs" ImranRazaKhan
The above sends mail to ImranRazaKhan (or whatever email address you chose) with the subject line of "exceptions in last hour of test.logs".
The convenience of having the -d option to date should not be underestimated. It might seem simple to subtract 1 from the current hour but, if the current hour is 12am, then we need to adjust both the day and the hour. If the hour was 12am on the first of the month, we would also have to change the month. And likewise for year. And, of course, February requires special consideration during leap years.
Adapting the above to Solaris:
Consider three cases:
Under Solaris 11 or better, the GNU date utility is available at /usr/gnu/bin/date. Thus, we need simply to specify a path for date:
grep "^$(/usr/gnu/bin/date -d -1hour +'%Y-%m-%d %H')" test.logs | grep 'exception'| mail -s "exceptions in last hour of test.logs" ImranRazaKhan
Under Solaris 10 or earlier, one can download & install GNU date
If GNU date is still not available, we need to find another way to find the date and time for one hour ago. The simplest workaround is likely to select a timezone that is one hour behind your timezone. If that timezone was, say, Hong Kong, then use:
grep "^$(TZ=HongKong date +'%Y-%m-%d %H')" test.logs | grep 'exception'| mail -s "exceptions in last hour of test.logs" ImranRazaKhan
You can do like this:
dt="$(date -d '1 hour ago' "+%m/%d/%Y %H:%M:%S")"
awk -v dt="$dt" '$0 ~ dt && /exceltion/' test.logs
Scanning through millions lines of log sounds terribly inefficient. I would suggest changing log4j (what it looks like) configuration of your application to cut a new log file every hour. This way, tailing the most recent file becomes a breeze.
Centos / Linux
Bash
I have a log file, which has lots of text in and epoch numbers all over the place. I want to replace all epochs whereever they are into readable date/time.
I've been wanting to this via sed, as that seems the tool for the job. I can't seem to get the replacement part of sed to actually parse the variable(epoch) to it for conversion.
Sample of what I'm working with...
echo "Some stuff 1346474454 And not working" \
| sed 's/1[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]/'"`bpdbm -ctime \&`"'/g'
Some stuff 0 = Thu Jan 1 01:00:00 1970 And not working
The bpdbm part will convert a supplied epoch variable into useful date. Like this..
bpdbm -ctime 1346474454
1346474454 = Sat Sep 1 05:40:54 2012
So how do i get the "found" item to be parsed into a command. As i don't seem to be able to get it to work.
Any help would be lovely. If there is another way, that would be cool...but i suspect sed will be quickest.
Thanks for your time!
that seems the tool for the job
No, it is not. sed can use & only itself, there is no way how to make it an argument to a command. You need something more powerful, e.g. Perl:
perl -pe 'if ( ($t) = /(1[0-9]+)/ ) { s/$t/localtime($t)/e }'
You can do it with GNU sed, the input:
infile
Some stuff 1346474454 And not working
GNU sed supports /e parameter which allows for piping command output into pattern space, one way to take advantage of this with bpdbm:
sed 's/(.*)(1[0-9]{9})(.*)/echo \1 $(bpdbm -ctime \2) \3/e' infile
Or with coreutils date:
sed 's/(.*)(1[0-9]{9})(.*)/echo \1 $(date -d #\2) \3/e' infile
output with date
Some stuff Sat Sep 1 06:40:54 CEST 2012 And not working
To get the same output as with bpdbm:
sed 's/(.*)(1[0-9]{9})(.*)/echo "\1$(date -d #\2 +\"%a %b %_d %T %Y\")\3"/e' infile
output
Some stuff Sat Sep 1 06:40:54 2012 And not working
Note, this only replaces the last epoch found on a line. Re-run if there are more.
Edit: Translated
I have a RSS-feed that i want to parse. It's a podcast and I want just the MP3-urls to download them with wget.
This is the podcast: http://feeds.feedburner.com/Film-UndKino-trailerVideopodcast
The title should include an (de) to get just the german episodes.
The publish-date should be today.
Would be great if someone could help me – I came this far:
wget -q -O- view-source:http://feeds.feedburner.com/Film-UndKino-trailerVideopodcast?format=xml| awk 'BEGIN{RS=""}
/(date +'%d %M %Y')/{
gsub(/.*|.*/,"")
print
}
But it doesn't work.
Thanks in advance,
arneb3rt
You need to drop the "view-source:" from the wget command and execute the date command (with %b to print the abbreviated month instead of %M) outside of the awk command. The following bash script uses grep instead of awk to produce the URLs of where wget can fetch the podcasts.
Note that, probably due to the holidays, there have been no podcasts since 24 Dec 2011 at the feed, so I hard-coded the date of the last podcast for testing:
url='http://feeds.feedburner.com/Film-UndKino-trailerVideopodcast?format=xml'
d=$(date +'%d %b %Y')
d="24 Dec 2011"
echo "Checking podcasts for date: ${d}"
wget -q -O- ${url} |\
grep -A6 "(de)" |\
grep -A1 "${d}" |\
egrep -o 'http[^ ]*de.mp4' |\
sort | uniq
The output of the above bash script lists two URLs (one feedburner and the other iTunes):
Checking podcasts for date: 24 Dec 2011
http://feedproxy.google.com/~r/Film-UndKino-trailerVideopodcast/~5/pzeSvkVK-3A/trailer01_de.mp4
http://www.moviemaze-trailer.de/ipod/6841/trailer01_de.mp4
Therefore, you could wget the 24 Dec 2011 podcast from either of the above URLs.