date bash script works under RHEL but not ubuntu - bash

the script from How to keep: daily backups for a week, weekly for a month, monthly for a year, and yearly after that
works under RHEL7 but reports errors when used on two separate ubuntu machines
I do not find what could be wrong with it besides it has loops within loops and some expressions are not quoted
keepdates ()
{
for i in {0..7};
do
((keep[$(date +%Y%m%d -d "-$i day")]++));
done;
for i in {0..4};
do
((keep[$(date +%Y%m%d -d "sunday-$((i+1)) week")]++));
done;
for i in {0..12};
do
DW=$(($(date +%-W)-$(date -d $(date -d "$(date +%Y-%m-15) -$i month" +%Y-%m-01) +%-W)));
for ((AY=$(date -d "$(date +%Y-%m-15) -$i month" +%Y); AY < $(date +%Y); AY++ ))
do
((DW+=$(date -d $AY-12-31 +%W)));
done;
((keep[$(date +%Y%m%d -d "sunday-$DW weeks")]++));
done;
for i in {0..5};
do
DW=$(date +%-W);
for ((AY=$(($(date +%Y)-i)); AY < $(date +%Y); AY++ ))
do
((DW+=$(date -d $AY-12-31 +%W)));
done;
((keep[$(date +%Y%m%d -d "sunday-$DW weeks")]++));
done;
echo ${!keep[#]}
}
The error is:
-bash: ./test.sh: line 18: syntax error near unexpected token `newline'
-bash: ./test.sh: line 18: ` done;'
any idea?
added output under RHEL7
$ bash --version
GNU bash, version 4.2.46(2)-release (x86_64-redhat-linux-gnu)
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
$ date --version
date (GNU coreutils) 8.22
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Written by David MacKenzie.
$ keepdates; # my custom function but a bash script would do the same
20130106 20140105 20150104 20160103 20170101 20170205 20170305 20170402 20170507 20170604 20170702 20170806 20170903 20171001 20171105 20171203 20171224 20171231 20180107 20180114 20180117 20180118 20180119 20180120 20180121 20180122 20180123 20180124

(I have no fedora but openSUSE 42.3 with GNU bash, Version 4.3.42(1)-release.)
The unnecessary semicolons have already been mentioned above. You call the date command with a fix result in your loops.
I wrote sequencies with $(seq from to) but that's just cosmetics, the {1..3} notation works too, I did not know that before.
The main problem is the nested date call in line 13 and the date call within the double expression evaluation in line 23 of your original post.
I have solved that by pulling out the fixed variables motm (MiddleOfTheMonth) and EY (EndYear).
For Line 13:
in this for condition $(date +%Y-%m-15) is fixed, why should the value be recalculated every time, same for $(date +%Y), try this line alone:
i=5; for ((AY=$(date -d "$(date +%Y-%m-15) -$i month" +%Y); AY < $(date +%Y); AY++ )); do echo $AY; done
-bash: syntax error near unexpected token `newline'
and with the $mdtm:
motm="2018-01-15"; i=5; for ((AY=$(date -d "$motm -$i month" +%Y); AY < $(date +%Y); AY++ )); do echo $AY; done
2017
For Line 23 you can observe the for condition:
i=2; for ((AY=$(($(date +%Y)-i));AY<2018;AY++));do echo $AY; done
-bash: syntax error near unexpected token `newline'
and
i=2; for ((AY=$((2018-i));AY<2018;AY++));do echo $AY; done
2016
2017
I do not know what goes wrong in the bash. I can only tell you how to write it down so it works.
This one produce the same on openSUSE and Ubuntu 16.04.3 LTS
#!/bin/bash
keepdates ()
{
local -a keep
for i in $(seq 0 7)
do
(( keep[$(date +%Y%m%d -d "-$i day")]++ ))
done
for i in $(seq 0 4)
do
(( keep[$(date +%Y%m%d -d "sunday-$((i+1)) week")]++ ))
done
motm=$(date +%Y-%m-15)
for i in $(seq 0 12)
do
DW=$(( $(date +%-W)-$(date -d $(date -d "$motm -$i month" +%Y-%m-01) +%-W) ))
for (( AY=$(date -d "$motm -$i month" +%Y); $AY < $(date +%Y); AY++ ))
do
(( DW+=$(date -d $AY-12-31 +%W) ))
done
(( keep[$(date +%Y%m%d -d "sunday-$DW weeks")]++ ))
done
for i in $(seq 0 5)
do
DW=$(date +%-W)
EY=$(date +%Y)
for (( AY=$(( EY-i )); $AY < $EY; AY++ ))
do
(( DW+=$(date -d $AY-12-31 +%W) ))
done
(( keep[$(date +%Y%m%d -d "sunday-$DW weeks")]++ ))
done
echo ${!keep[#]}
}
keepdates

Related

How to get second sunday in a Month given a date parameter in bash script

I am trying to write a bash script, to merge 24 files in a given day. The requirement changes during Day light saving time changes, where I get 23 or 25 files.
So, with further research I realized that day-light savings begins on the second Sunday of March(23) of every year and ends on first sunday of Novemeber(25).
I need more inputs to get second sunday in a given month to do the check of finding 23 or 25 files for March and November respectively.
Any inputs to help me with this will be really appreciated.
Thank you
Here is the sample code to find 24 files in a day-
if [ -z "$1" ];then
now=$(date -d "-1 days" +%Y-%m-%d);
else now=$1;
fi
load_date='load_date='$now
singlePath="$newPath/$load_date"
fileCount=$(hdfs dfs -ls -R $hdfsPath/$load_date/ | grep -E '^-' | wc -l)
path=$hdfsPath/$load_date
if [ $fileCount -eq 24 ]; then
echo "All files are available for "$load_date;
hadoop fs -cat $path/* | hadoop fs -put - $singlePath/messages.txt
else echo $fileCount" files are available for "$load_date"! Please note, few files are being missed";
fi
I wouldn't hardcode the dates of DST transistions. I would just count "how many hours did today have":
a "normal" day:
$ diff=$(( $(date -d now +%s) - $(date -d yesterday +%s) ))
$ echo $(( diff / 3600 ))
24
"spring forward"
$ diff=$(( $(date -d "2019-03-10 23:59:59" +%s) - $(date -d "2019-03-09 23:59:59" +%s) ))
$ echo $(( diff / 3600 ))
23
"fall back"
$ diff=$(( $(date -d "2019-11-03 23:59:59" +%s) - $(date -d "2019-11-02 23:59:59" +%s) ))
$ echo $(( diff / 3600 ))
25
One thing to note: since bash only does integer arithmetic, if the difference is not 86400 but 86399, you get:
$ echo $((86399 / 3600))
23
So, better to query yesterday's time first in the tiny-but-non-zero chance that the seconds tick over between the 2 date calls:
diff=$(( -$(date -d yesterday +%s) + $(date -d now +%s) ))
Here, $diff will be 86400 or 86401 (for non DST transition days), and dividing by 3600 will give 24 not 23.

Time difference in seconds between given two dates

I have two dates as follows:
2019-01-06 00:02:10 | END
2019-01-05 23:52:00 | START
How could I calculate and print the difference between START and END dates in seconds?
For above case I would like to get something like:
610
Assuming GNU implementation based OS, you can use date's option %s and -d to calculate the time difference in seconds using command substitution and arithmetic operations.
START="2019-01-05 23:52:00"
END="2019-01-06 00:02:10"
Time_diff_in_secs=$(($(date -d "$END" +%s) - $(date -d "$START" +%s)))
echo $Time_diff_in_secs
Output:
610
Hope this helps!!!
With bash and GNU date:
while read d t x x; do
[[ $x == "END" ]] && end="$d $t"
[[ $x == "START" ]] && start="$d $t"
done < file
end=$(date -u -d "$end" '+%s')
start=$(date -u -d "$start" '+%s')
diff=$(($end-$start))
echo "$diff"
Output:
610
See: man date
What you're asking for is difficult verging on impossible using pure bash. Bash doesn't have any date functions of its own. For date processing, most recommendations you'll get will be to use your operating system's date command, but the usage of this command varies by operating system.
In BSD (including macOS):
start="2019-01-05 23:52:00"; end="2019-01-06 00:02:10"
printf '%d\n' $(( $(date -j -f '%F %T' "$end" '+%s') - $(date -j -f '%F %T' "$start" '+%s') ))
In Linux, or anything using GNU date (possibly also Cygwin):
printf '%d\n' $(( $(date -d "$end" '+%s') - $(date -d "$start" '+%s') ))
And just for the fun of it, if you can't (or would prefer not to) use date for some reason, you might be able to get away with gawk:
gawk 'END{ print mktime(gensub(/[^0-9]/," ","g",end)) - mktime(gensub(/[^0-9]/," ","g",start)) }' start="$start" end="$end" /dev/null
The mktime() option parses a date string in almost exactly the format you're providing, making the math easy.
START="2019-01-05 23:52:00"
END="2019-01-06 00:02:10"
parse () {
local data=(`grep -oP '\d+' <<< "$1"`)
local y=$((${data[0]}*12*30*24*60*60))
local m=$((${data[1]}*30*24*60*60))
local d=$((${data[2]}*24*60*60))
local h=$((${data[3]}*60*60))
local mm=$((${data[4]}*60))
echo $((y+m+d+h+mm+${data[5]}))
}
START=$(parse "$START")
END=$(parse "$END")
echo $((END-START)) // OUTPUT: 610
Was trying to solve the same problem on a non-GNU OS, i.e. macOS. I couldn't apply any of the solutions above, although it inspired me to come up with the following solution. I am using some in-line Ruby from within my shell script, which should work out of the box on macOS.
START="2019-01-05 23:52:00"
END="2019-01-06 00:02:10"
SECONDS=$(ruby << RUBY
require 'date'
puts ((DateTime.parse('${END}') - DateTime.parse('${START}')) * 60 * 60 * 24).to_i
RUBY)
echo ${SECONDS}
# 610

How to subtract today's date with a file's modification date in unix?

For example:
echo $(date) - $(date -r sample.txt)
Output:
90 days(for example)
Use %s seconds since 1970-01-01 00:00:00 UTC as in
echo $(expr $(date +%s) - $(date -r sample.txt +%s)) #!/bin/sh
echo $(($(date +%s) - $(date -r sample.txt +%s))) #/bin/bash
One more way
$ ls -l peter.txt
-rwxrw-r--+ 1 pppp qqqq 149 Dec 15 18:39 peter.txt
$ echo "(" $(date +%s) - $(date -r peter.txt +%s) ")/" 86400 | perl -nle ' print eval, " days" '
29.254537037037 days
$

Compute time-difference on millisecond level

Assuming the following time-formats:
MM:DD:YYYY hh:mm:ss:nn
How can I compute the difference between two times? I have tried the
following, but it seems to fail.
% Value1='08:27:2018 23:53:50:08'
% Value2='08:28:2018 00:00:08:89'
% echo "$(($(date -d "$Value2" '+%s') - $(date -d "$Value1" '+%s')))"
Update: as the OP changed his format
$ Value1='08:27:2018 23:53:50:08'
$ Value2='08:28:2018 00:00:08:89'
$ Value1=${Value1/://}; Value1=${Value1/://}; Value1=${Value1%:*}.${Value1##*:}
$ Value2=${Value2/://}; Value2=${Value2/://}; Value2=${Value2%:*}.${Value2##*:}
$ echo $(date -d "$Value2" '+%s.%N') - $(date -d "$Value1" '+%s.%N') | bc -l
378.810000000
So all you need to do is convert it to a format that date knows, this is MM/DD/YYYY
Original answer below:
The problem is that your date-time format is not really recognized.
The date format:
$ date -d "08272018"
date: invalid date ‘08272018’
The date command knows many formats, but it is hard for it to distinguish between YYYYMMDD, DDMMYYYY and MMDDYYYY. To be clear, what does 10021002 represent as a date?
In this simple format—without delimiters—date will recognize YYYYMMDD and YYMMDD
$ date -d "20180708"
Sun 8 Jul 00:00:00 UTC 2018
The time format:
The notation HH:MM:SS:ss is by far from standard. What does ss represent, ss 60th of a second? The normal notation would be more HH:MM:SS.sss This will be recognized.
$ date -d "23:53:50:08"
date: invalid date ‘23:53:50:08’
$ date -d "23:53:50.08" "+%a %d %b %T.%N %Z %Y"
Wed 12 Sep 23:53:50.080000000 UTC 2018
So if you get your date format correct, you already get a long way:
% Value1='20180827 23:53:50.08'
% Value2='20180828 00:00:08.89'
% echo "$(($(date -d "$Value2" '+%s') - $(date -d "$Value1" '+%s')))"
378
The sad thing is that we are missing our milliseconds for this you need floating point arithmetic and bash does not support it. But there are ways around that (How do I use floating-point division in bash?)
$ echo $(date -d "$Value2" '+%s.%N') - $(date -d "$Value1" '+%s.%N') | bc -l
378.810000000
I shortened variable names:
v1='08:27:2018 23:53:50:08'
v2='08:28:2018 00:00:08:89'
With GNU date, just stick to one safe input format, you can convert YYYY-MM-DD HH:MM:SS.NN to... anything another. (side note: I love freebsd date, where you can just specify -f option for strptime. I wish I could do that with GNU date). So we can:
v1_epoch=$(date -d "${v1:6:4}-${v1:0:2}-${v1:3:2} ${v1:11:2}:${v1:14:2}:${v1:17:2}.${v1:20}" +%s.%N)
v2_epoch=$(date -d "${v2:6:4}-${v2:0:2}-${v2:3:2} ${v2:11:2}:${v2:14:2}:${v2:17:2}.${v2:20}" +%s.%N)
It will get us values of seconds with nanosecond resolution since epoch time. Now we need to calc a difference, we need to use a tool like bc, cause bash does not support floating point calculations.
diff=$(printf "scale=9; %s - %s\n" "$v2_epoch" "$v1_epoch" | bc)
Now this represents the difference of time we need to represent in hours, minutes, seconds and miliseconds.
printf "%s.%.3s" $(date -d#"$diff" -u +'%H:%M:%S %N')
That's simple, but it will wrap around at 23 hours, so we can do better with bc. The rounding in bc is sometimes unexpected... you need to just get used to unexpected scale=0 lines:
printf "%02d:%02d:%02d.%03d\n" $(printf 'scale=11; a=%s; scale=0; h=a/3600; m=a%%3600/60; s=a%%60/1; ms=a*1000%%1000/1; h \n m \n s \n ms \n' '$diff' | bc -l)
A "neat" oneliner:
$ v1='08:27:2018 23:53:50:08'
$ v2='08:28:2018 00:00:08:89'
$ printf "%02d:%02d:%02d.%03d\n" $(printf 'scale=11; a=%s; scale=0; h=a/3600; m=a%%3600/60; s=a%%60/1; ms=a*1000%%1000/1; h \n m \n s \n ms \n' "$(printf "scale=9; %s - %s\n" "$(date -d "${v2:6:4}-${v2:0:2}-${v2:3:2} ${v2:11:2}:${v2:14:2}:${v2:17:2}.${v2:20}" +%s.%N)" "$(date -d "${v1:6:4}-${v1:0:2}-${v1:3:2} ${v1:11:2}:${v1:14:2}:${v1:17:2}.${v1:20}" +%s.%N)" | bc)" | bc -l)
I guess this could be even shortened with some here strings, but that just harms readability:
printf "%02d:%02d:%02d.%03d\n" $(<<<"scale=11; a=$(<<< "scale=9; $(date -d "${v2:6:4}-${v2:0:2}-${v2:3:2} ${v2:11:2}:${v2:14:2}:${v2:17:2}.${v2:20}" +%s.%N) - $(date -d "${v1:6:4}-${v1:0:2}-${v1:3:2} ${v1:11:2}:${v1:14:2}:${v1:17:2}.${v1:20}" +%s.%N)"$'\n' bc); scale=0; h=a/3600; m=a%3600/60; s=a%60/1; ms=a*1000%1000/1; h"$'\n'"m"$'\n'"s"$'\n'"ms"$'\n' bc -l)
Or you can create a function for conversion:
mydate_read() { date -d "${1:6:4}-${1:0:2}-${1:3:2} ${1:11:2}:${1:14:2}:${1:17:2}.${1:20}" +%s.%N; };
printf "%02d:%02d:%02d.%03d\n" $(<<<"scale=11; a=$(<<< "scale=9; $(mydate_read "$v2") - $(mydate_read "$v1")"$'\n' bc); scale=0; h=a/3600; m=a%3600/60; s=a%60/1; ms=a*1000%1000/1; h"$'\n'"m"$'\n'"s"$'\n'"ms"$'\n' bc -l)
I forgot, we can merge the two bc calls into one:
mydate_read() { date -d "${1:6:4}-${1:0:2}-${1:3:2} ${1:11:2}:${1:14:2}:${1:17:2}.${1:20}" +%s.%N; };
printf "%02d:%02d:%02d.%03d\n" $(printf 'scale=11; a=%s - %s; scale=0; h=a/3600; m=a%%3600/60; s=a%%60/1; ms=a*1000%%1000/1; h \n m \n s \n ms \n' "$(mydate_read "$v2")" "$(mydate_read "$v1")" | bc -l)

Error on backup bash script: syntax error near unexpected token `newline'

I am having problem finding error on bash script for handling backup:daily, monthly, yearly. Here is the script:
#!/bin/bash
echo > /home/alpha/folder/keep.txt
#writing dates of the backups that should be kept to the array
for i in {0..7}; do ((keep[$(date +%Y%m%d -d "-$i day")]++)); done
for i in {0..4}; do ((keep[$(date +%Y%m%d -d "sunday-$((i+1)) week")]++)); done
for i in {0..12}; do
DW=$(($(date +%-W)-$(date -d $(date -d "$(date +%Y-%m-15) -$i month" +%Y-%m-01) +%-W)))
for (( AY=$(date -d "$(date +%Y-%m-15) -$i month" +%Y); AY < $(date +%Y); AY++ )); do
((DW+=$(date -d $AY-12-31 +%W)))
done
((keep[$(date +%Y%m%d -d "sunday-$DW weeks")]++))
done
for i in {0..30}; do
DW=$(date +%-W)
for (( AY=$(($(date +%Y)-i)); AY < $(date +%Y); AY++ )); do
((DW+=$(date -d $AY-12-31 +%W)))
done
((keep[$(date +%Y%m%d -d "sunday-$DW weeks")]++))
done
#writing the array to file keep.txt line by line
for i in ${!keep[#]}; do echo $i >> /home/alpha/folder/keep.txt; done
#delete all files that not mentioned in keep.txt
cd /home/alpha/folder
ls -1 /home/alpha/folder/ | sort /home/alpha/folder/keep.txt /home/alpha/folder/keep.txt - | uniq -u | xargs rm -rf
rm /home/alpha/folder/keep.txt
When I try to run the script, throws error message:
./back.sh: line 12: syntax error near unexpected token `newline' ./back.sh: line 12: ` done'
Where did I do wrong on the script?
Your date expression seems to misbehave inside the arithmetic context. Adding temporary variables solved your issue for me :
#!/bin/bash
echo > /home/alpha/folder/keep.txt
#writing dates of the backups that should be kept to the array
for i in {0..7}; do ((keep[$(date +%Y%m%d -d "-$i day")]++)); done
for i in {0..4}; do ((keep[$(date +%Y%m%d -d "sunday-$((i+1)) week")]++)); done
for i in {0..12}; do
DW=$(($(date +%-W)-$(date -d $(date -d "$(date +%Y-%m-15) -$i month" +%Y-%m-01) +%-W)))
begin=$(date -d "$(date +%Y-%m-15) -$i month" +%Y)
for (( AY=begin; AY < $(date +%Y); AY++ )); do
((DW+=$(date -d $AY-12-31 +%W)))
done
((keep[$(date +%Y%m%d -d "sunday-$DW weeks")]++))
done
for i in {0..30}; do
DW=$(date +%-W)
begin=$(($(date +%Y)-i))
for (( AY=begin; AY < $(date +%Y); AY++ )); do
((DW+=$(date -d $AY-12-31 +%W)))
done
((keep[$(date +%Y%m%d -d "sunday-$DW weeks")]++))
done
#writing the array to file keep.txt line by line
for i in ${!keep[#]}; do echo $i >> /home/alpha/folder/keep.txt; done
#delete all files that not mentioned in keep.txt
cd /home/alpha/folder
ls -1 /home/alpha/folder/ | sort /home/alpha/folder/keep.txt /home/alpha/folder/keep.txt - | uniq -u | xargs rm -rf
rm /home/alpha/folder/keep.txt
However, I am unsure why the expression misbehaves inside the arithmetic block.

Resources