I'm working on a shell script that scrapes a date off a webpage and then checks if it is in the last three days of the current date (running it in Cygwin). If the test passes, it simply echoes "PASS", and if it fails, it echoes "FAIL";
However, when I run my script, I get the following:
integer expression expected: 1317618000
FAIL
Here is the script:
updateStr=$(curl "http://www.mywebsite.com" | grep "Last Update")
dateStr=(`echo $updateStr | sed -e 's/.*Last Update: \([^<]*\)<.*/\1/'`)
update=$(date -d "$dateStr" +%s)
epoch=$(date -d "-3 days ago" +%s)
test "$update" -ge "$epoch" && echo "PASS" || echo "FAIL"
Any ideas on what the issue is?
Edit
Here is the result of running bash -x on the script:
$ bash -x check_date.sh
++ curl http://www.mywebsite.com
++ grep 'Last Update'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 6542 100 6542 0 0 789 0 0:00:08 0:00:08 --:--:-- 1544
+ updateStr=' <span style="float:right">Last Update: 10/3/2011 2:
'8:45 AM</span></p>
++ echo '<span' 'style="float:right">Last' Update: 10/3/2011 2:58:45 'AM</span><
'p>
++ sed -e 's/.*Last Update: \([^<]*\)<.*/\1/'
' dateStr='(10/3/2011 2:58:45 AM)
' +%ste -d '(10/3/2011 2:58:45 AM)
+ update=$'1317618000\r'
++ date -d '-3 days ago' +%s
+ epoch=$'1317938194\r'
+ test $'1317618000\r' -ge $'1317938194\r'
: integer expression expected1317618000
+ echo FAIL
FAIL
Update
I tried removing the carriage returns that appear in the date outputs, but it's still not working. Here is the updated script:
updateStr=$(curl "http://mywebsite.com" | grep "Last Update")
dateStr=$(echo $updateStr | sed -e 's/.*Last Update: \([^<]*\)<.*/\1/')
update=$(date -d "$dateStr" +%s | tr -d '\r')
epoch=$(date -d "3 days ago" +%s | tr -d '\r')
echo "Last Update: $update"
echo "Epoch Date: $epoch"
test "$update" -ge "$epoch" && echo "PASS" || echo "FAIL"
And here is the result:
$ bash -x ./check_date.sh
++ curl http://mywebsite.com
++ grep 'Last Update'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 6542 100 6542 0 0 17398 0 --:--:-- --:--:-- --:--:-- 19704
+ updateStr=' <span style="float:right">Last Update: 10/6/2011 2:
'0:02 AM</span></p>
++ echo '<span' 'style="float:right">Last' Update: 10/6/2011 2:40:02 'AM</span><
'p>
++ sed -e 's/.*Last Update: \([^<]*\)<.*/\1/'
' dateStr='10/6/2011 2:40:02 AM
' +%ste -d '10/6/2011 2:40:02 AM
++ tr -d '\r'
+ update=$'1317886802\r'
++ date -d '-3 days ago' +%s
++ tr -d '\r'
+ epoch=$'1318184767\r'
' echo 'Last Update: 1317886802
Last Update: 1317886802
' echo 'Epoch Date: 1318184767
Epoch Date: 1318184767
+ test $'1317886802\r' -ge $'1318184767\r'
: integer expression expected: 1317886802
+ echo FAIL
FAIL
The carriage returns might be in your script itself. Try running dos2unix on your script.
Try this:
update=$(date -d "$dateStr" +%s | tr -d '\r')
epoch=$(date -d "-3 days ago" +%s | tr -d '\r')
That will get rid of the carriage returns at the end of each number, which might help.
bash is determining from context (the presence of '\r' in this case) that these are strings and not integers. The '-ge' binary operator expects integer arguments and so it's throwing an error. Tom Zych's suggestion will work, as well as using string comparison operators, like '<' or '>', which might give unexpected results if the string is badly formatted for some reason. I'm not sure if there is a way to force date to output as an integer or not...
References:
http://tldp.org/LDP/abs/html/untyped.html
http://tldp.org/LDP/abs/html/comparison-ops.html
Related
I am trying to write a bash script, to merge 24 files in a given day. The requirement changes during Day light saving time changes, where I get 23 or 25 files.
So, with further research I realized that day-light savings begins on the second Sunday of March(23) of every year and ends on first sunday of Novemeber(25).
I need more inputs to get second sunday in a given month to do the check of finding 23 or 25 files for March and November respectively.
Any inputs to help me with this will be really appreciated.
Thank you
Here is the sample code to find 24 files in a day-
if [ -z "$1" ];then
now=$(date -d "-1 days" +%Y-%m-%d);
else now=$1;
fi
load_date='load_date='$now
singlePath="$newPath/$load_date"
fileCount=$(hdfs dfs -ls -R $hdfsPath/$load_date/ | grep -E '^-' | wc -l)
path=$hdfsPath/$load_date
if [ $fileCount -eq 24 ]; then
echo "All files are available for "$load_date;
hadoop fs -cat $path/* | hadoop fs -put - $singlePath/messages.txt
else echo $fileCount" files are available for "$load_date"! Please note, few files are being missed";
fi
I wouldn't hardcode the dates of DST transistions. I would just count "how many hours did today have":
a "normal" day:
$ diff=$(( $(date -d now +%s) - $(date -d yesterday +%s) ))
$ echo $(( diff / 3600 ))
24
"spring forward"
$ diff=$(( $(date -d "2019-03-10 23:59:59" +%s) - $(date -d "2019-03-09 23:59:59" +%s) ))
$ echo $(( diff / 3600 ))
23
"fall back"
$ diff=$(( $(date -d "2019-11-03 23:59:59" +%s) - $(date -d "2019-11-02 23:59:59" +%s) ))
$ echo $(( diff / 3600 ))
25
One thing to note: since bash only does integer arithmetic, if the difference is not 86400 but 86399, you get:
$ echo $((86399 / 3600))
23
So, better to query yesterday's time first in the tiny-but-non-zero chance that the seconds tick over between the 2 date calls:
diff=$(( -$(date -d yesterday +%s) + $(date -d now +%s) ))
Here, $diff will be 86400 or 86401 (for non DST transition days), and dividing by 3600 will give 24 not 23.
I have two dates as follows:
2019-01-06 00:02:10 | END
2019-01-05 23:52:00 | START
How could I calculate and print the difference between START and END dates in seconds?
For above case I would like to get something like:
610
Assuming GNU implementation based OS, you can use date's option %s and -d to calculate the time difference in seconds using command substitution and arithmetic operations.
START="2019-01-05 23:52:00"
END="2019-01-06 00:02:10"
Time_diff_in_secs=$(($(date -d "$END" +%s) - $(date -d "$START" +%s)))
echo $Time_diff_in_secs
Output:
610
Hope this helps!!!
With bash and GNU date:
while read d t x x; do
[[ $x == "END" ]] && end="$d $t"
[[ $x == "START" ]] && start="$d $t"
done < file
end=$(date -u -d "$end" '+%s')
start=$(date -u -d "$start" '+%s')
diff=$(($end-$start))
echo "$diff"
Output:
610
See: man date
What you're asking for is difficult verging on impossible using pure bash. Bash doesn't have any date functions of its own. For date processing, most recommendations you'll get will be to use your operating system's date command, but the usage of this command varies by operating system.
In BSD (including macOS):
start="2019-01-05 23:52:00"; end="2019-01-06 00:02:10"
printf '%d\n' $(( $(date -j -f '%F %T' "$end" '+%s') - $(date -j -f '%F %T' "$start" '+%s') ))
In Linux, or anything using GNU date (possibly also Cygwin):
printf '%d\n' $(( $(date -d "$end" '+%s') - $(date -d "$start" '+%s') ))
And just for the fun of it, if you can't (or would prefer not to) use date for some reason, you might be able to get away with gawk:
gawk 'END{ print mktime(gensub(/[^0-9]/," ","g",end)) - mktime(gensub(/[^0-9]/," ","g",start)) }' start="$start" end="$end" /dev/null
The mktime() option parses a date string in almost exactly the format you're providing, making the math easy.
START="2019-01-05 23:52:00"
END="2019-01-06 00:02:10"
parse () {
local data=(`grep -oP '\d+' <<< "$1"`)
local y=$((${data[0]}*12*30*24*60*60))
local m=$((${data[1]}*30*24*60*60))
local d=$((${data[2]}*24*60*60))
local h=$((${data[3]}*60*60))
local mm=$((${data[4]}*60))
echo $((y+m+d+h+mm+${data[5]}))
}
START=$(parse "$START")
END=$(parse "$END")
echo $((END-START)) // OUTPUT: 610
Was trying to solve the same problem on a non-GNU OS, i.e. macOS. I couldn't apply any of the solutions above, although it inspired me to come up with the following solution. I am using some in-line Ruby from within my shell script, which should work out of the box on macOS.
START="2019-01-05 23:52:00"
END="2019-01-06 00:02:10"
SECONDS=$(ruby << RUBY
require 'date'
puts ((DateTime.parse('${END}') - DateTime.parse('${START}')) * 60 * 60 * 24).to_i
RUBY)
echo ${SECONDS}
# 610
I'm trying to find a pattern "INFO: Server startup in" for last 5 mins in a log file.
Here is the line from which I'm trying to find the pattern: "INFO | jvm 1 | main | 2018/07/09 00:11:29.077 | INFO: Server startup in 221008 ms"
The pattern is coming, but I need to shorten the code or create a loop for it.
I tried to create a loop, but it is not working. Here is my code without loops, which is working:
#!/bin/bash
#Written by Ashutosh
#We will declare variables with date and time of last 5 mins.
touch /tmp/a.txt;
ldt=$(date +"%Y%m%d");
cdt=$(date +"%Y/%m/%d %H:%M");
odtm5=$(date +"%Y/%m/%d %H:%M" --date "-5 min");
odtm4=$(date +"%Y/%m/%d %H:%M" --date "-4 min");
odtm3=$(date +"%Y/%m/%d %H:%M" --date "-3 min");
odtm2=$(date +"%Y/%m/%d %H:%M" --date "-2 min");
odtm1=$(date +"%Y/%m/%d %H:%M" --date "-1 min");
## Finding the pattern and storing it in a file
grep -e "$odtm1" -e "$cdt" -e "$odtm2" -e "$odtm3" -e "$odtm4" -e
"$odtm5" /some/log/path/console-$ldt.log
> /tmp/a.txt;
out=$(grep 'INFO: Server startup in' /tmp/a.txt);
echo "$out"
## remove the file that contains the pattern
rm /tmp/a.txt;
I have tried to use sed also, but date function is not working with it.
Can someone please give me the new changed script with loops?
Adopting your original logic:
time_re='('
for ((count=5; count>0; count--)); do
time_re+="$(date +'%Y/%m/%d %H:%M' --date "-$count min")|"
done
time_re+="$(date +'%Y/%m/%d %H:%M'))"
ldt=$(date +'%Y%m%d')
awk -v time_re="$time_re" '
$0 ~ time_re && /INFO: Server startup in/ { print $0 }
' "/some/log/path/console-$ldt.log"
Performance enhancements are certainly possible -- this could be made much faster by bisecting the log for the start time -- but the above addresses the explicit question (about using a loop to generate the time window). Note that it will get unwieldy -- you wouldn't want to use this to search for the last day, for example, as the regex would become utterly unreasonable.
Sounds like all you need is:
awk -v start="$(date +'%Y/%m/%d %H:%M' --date '-5 min')" -F'[[:space:]]*[|][[:space:]]*' '
($4>=start) && /INFO: Server startup in/
' file
No explicit loops or multiple calls to date required.
Here is a bash script that does the job (thanks to Charles for its improvement):
#!/bin/bash
limit=$(date -d '5 minutes ago' +%s)
today_logs="/some/log/path/console-$(date +'%Y%m%d').log"
yesterday_logs="/some/log/path/console-$(date +'%Y%m%d' -d yesterday).log"
tac "$today_logs" "$yesterday_logs" \
| while IFS='|' read -r prio jvm app date log; do
[ $(date -d "$date" +%s) -lt "$limit" ] && break
echo "|$jvm|$prio|$app|$date|$log"
done \
| grep -F 'INFO: Server startup in' \
| tac
It has the following advantages over your original script:
optimized: it parses log lines starting from the more recent ones and stops at the first line encountered that is more than 5 min old. At 23:59, no need to parse log lines from 0:00 to 23:53
arbitrary time window: you can replace "5 minutes" with "18 hours" and it will still work. A time window of more than one day needs adaptation since each day has it own log file
works correctly when day changes: at 0:00 the original script will never parse the log lines from 23:55:00 to 23:59:59
Mixing the above code with Ed Morton's answer, you get:
#!/bin/bash
limit=$(date -d '5 minutes ago' +'%Y/%m/%d %H:%M')
today_logs="/some/log/path/console-$(date +'%Y%m%d').log"
yesterday_logs="/some/log/path/console-$(date +'%Y%m%d' -d yesterday).log"
tac "$today_logs" "$yesterday_logs" \
| awk -v stop="$limit" -F'[[:space:]]*[|][[:space:]]*' '
($4 < stop) { exit }
/INFO: Server startup in/
' \
| tac
#!/bin/bash
# Program's Purpose: Compute time elapsed between epoch time and current time
# Produce an MD5 hash from that time
# Get code from that hash
# compute time elapsed in seconds between epoch time and current time
#EPOCH=$(date -u -d '2001-02-03 04:05:06' '+%F %H:%M:%S')
#CURRENT=$(date -u -d '2010-06-13 12:55:34' '+%F %H:%M:%S')
# code: dd15
EPOCH=$(date -u -d '1999-12-31 23:59:59' '+%F %H:%M:%S')
CURRENT=$(date -u -d '2013-05-06 07:43:25' '+%F %H:%M:%S')
# interval is time elapsed minus time elapsed % 60
echo $EPOCH
echo $CURRENT
read YEAR1 M1 DAY1 HOUR1 MIN1 SEC1 <<< "${EPOCH//[:-]/ }"
read YEAR2 M2 DAY2 HOUR2 MIN2 SEC2 <<< "${CURRENT//[:-]/ }"
echo $YEAR1 $M1 $DAY1 $HOUR1 $MIN1 $SEC1
# date in seconds from
sec1=$(date -d "$EPOCH" -u +%s)
sec2=$(date -d "$CURRENT" -u +%s)
echo $sec1
echo $sec2
# get the difference from the two times
difference=$((sec2 - sec1))
difference=$((difference - ((difference % 60))))
echo $difference
# get the hash from the time
hash=$(echo -n $difference | md5sum | tr -d '\n')
hash=$(echo -n $hash | md5sum | tr -d '\n')
echo $hash
# creating two strings, one with all of the letters
# the other with all of the numbers
letter=$(echo $hash | sed 's/[0-9]*//g' | cut -c1-2)
echo $letter
num=$(echo $hash | sed 's/[^0-9]*//g')
echo $num
#num=$(echo $num | cut -c1-2)
#echo $num
# getting the last two numbers in reverse order
num1=$(echo ${num: -1})
num=$(echo "${num::-1}")
echo $num
num2=$(echo ${num: -1})
code="$letter$num1$num2"
echo $code
I'm trying to write a program that takes an epoch time and
current time, computes the difference in seconds, and then creates a
code by doing a double md5 hash on the time in seconds. By what times
I have entered in currently, the difference in seconds should be 421,
141, 406, and the 'code' is supposed to be based on 60-second
intervals, so the code I'm trying to generate should come from 421,
141, 380.
The resulting hash should be
042876ca07cb2d993601fb40a1525736, but I am getting
d49904f9e7e62d0ff16e523d89be08eb. Can anyone tell me what I'm doing
wrong exactly?
I read that bash leaves a newline at the end of
strings, so I ran echo with -n option to remove that newline, but I am
still not getting the preferred results.
The output of md5sum on many platforms is not just the MD5 sum. For example, on a GNU/Linux system, you get
debian$ echo -n 401 | md5sum
816b112c6105b3ebd537828a39af4818 -
Notice that the output has two fields: The actual MD5 sum, followed by two spaces, followed by the input file name (where - stands for standard input).
(By contrast, on OSX, and I would expect most *BSDs, you get
yosemite$ echo -n 401 | md5
816b112c6105b3ebd537828a39af4818
Here, you'll notice that the name of the MD5 utility is different.)
The fix should be simple. I have refactored your code to (a) prefer the portable printf over the less portable echo -n; (b) remove the completely superfluous final tr -d '\n' (newlines are trimmed off the end of the captured variable by the shell already); and (c) fold everything into a single pipeline.
hash=$(printf '%s' "$difference" | md5sum | cut -d ' ' -f 1 | tr -d '\n' |
md5sum | cut -d ' ' -f 1)
echo "$hash"
For completeness, this code also has proper quoting; it's not strictly necessary here (but it would have been if you really needed to preserve the spacing in the value you originally obtained from md5sum, for example) but omitting quotes is a common newbie problem which should be avoided.
(Capturing a variable just so you can echo it is also a common newbie antipattern; but your code will want to use the variable hash subsequently.)
Repeated code is always a bad smell; maybe provide a function which performs the same job as the *BSD md5 utility;
md5 () { md5sum "$#" | cut -d ' ' -f 1; }
hash=$(printf '%s' "$difference" | md5 | tr -d '\n' | md5)
I can generate two timestamps like so:
date +"%s" -d "$(curl -s --head http://google.com | grep ^Date: | sed 's/Date: //g')"
// Result: 1417800327
date +"%s"
// Result: 1417800325
How can I subtract them with only one line?
echo "$((1417800327-1417800325))"
// Result: 2
But I want something closer to:
echo "$(( (date +"%s" -d "$(curl -s --head http://google.com | grep ^Date: | sed 's/Date: //g')") - (date +"%s")))"
Try doing this :
LANG=C echo $(( $(date +%s) - $(date -d "$(curl -s --head http://google.com 2>&1 | awk -F'Date: ' '/^Date:/{print $2}')" +%s) ))
or splitted multi-lines for readability :
LANG=C echo $((
$(date +%s) - $(
date -d "$(curl -s --head http://google.com |
awk -F'Date: ' '/^Date:/{print $2}'
)" +%s)
))
#!/bin/bash
##On Linux. start time, do something, find end time.
st=`date +%s`; sleep 10; et=`date +%s`;
##On SunOS instead of using date command, use `truss /usr/bin/date 2>&1 | grep ^time | awk -F"= " '{print $2}'`;
##substract to find elapsed time
elt=$((et-st));
##find elapsed hours, minutes, seconds
eh=$((elt/3600)); em=$(($((elt%3600))/60)); es=$(($((elt%3600))%60));
elt=`printf "%02d:%02d:%02d%s" $((eh)) $((em)) $((es))`;
##show elapsed time in user friendly way
echo - Elapsed time: $elt
Output:
- Elapsed time: 00:00:10
Using a subshell function with one big command bash line
(timestamp=`date +"%s" -d "$(curl -s --head http://google.com | grep ^Date: | sed 's/Date: //g')"`;date=`date +"%s"`;let "time_difference= $timestamp-$date";echo "$time_difference"; )