I am writing a code in a shell script to load data from specific range but it does not stops at the data I want and instead goes past beyond that. Below is my code of shell script.
j=20180329
while [ $j -le 20180404]
do
i have problem that my loop run after the date 20180331 till 20180399 then it go to 20180401.
i want it to go from 20180331 to 20180401. not 20180332 and so on
One simple question, 3+ not so short answer...
As your request stand for shell
1. Compatible answer first
j=20180329
while [ "$j" != "20180405" ] ;do
echo $j
j=`date -d "$j +1 day" +%Y%m%d`
done
Note I used one day after as while condition is based on equality! Of course interpreting YYYYMMDD date as integer will work too:
Note 2 Care about timezone set TZ=UTC see issue further...
j=20180329
while [ $j -le 20180404 ] ;do
echo $j
j=`TZ=UTC date -d "$j +1 day" +%Y%m%d`
done
But I don't like this because if time format change, this could become an issue.
Tested under bash and shell as dash and busybox.
(using date (GNU coreutils) 8.26.
1.2 Minimize fork under POSIX shell
Before using bash bashisms, here is a way of doing this under any POSIX shell:
The power of POSIX shell is that we could use very simple converter like date and do condition over result:
#!/usr/bin/env sh
tempdir=$(mktemp -d)
datein="$tempdir/datein"
dateout="$tempdir/dateout"
mkfifo "$datein" "$dateout"
exec 5<>"$datein"
exec 6<>"$dateout"
stdbuf -i0 -o0 date -f - <"$datein" >"$dateout" +'%Y%m%d' &
datepid=$!
echo "$2" >&5
read -r end <&6
echo "$1" >&5
read -r crt <&6
while [ "$crt" -le "$end" ];do
echo $crt
echo "$crt +1 day" >&5
read -r crt <&6
done
exec 5>&-
exec 6<&-
kill "$datepid"
rm -fR "$tempdir"
Then
daterange.sh 20180329 20180404
20180329
20180330
20180331
20180401
20180402
20180403
20180404
2. bash date via printf
Under bash, you could use so-called bashisms:
Convert date to integer Epoch (Unix time), but two dates via one fork:
{
read start;
read end
} < <(date -f - +%s <<eof
20180329
20180404
eof
)
or
start=20180329
end=20180404
{ read start;read end;} < <(date -f - +%s <<<$start$'\n'$end)
Then using bash builtin printf command (note: there is $[24*60*60] -> 86400 seconds in a regular day)
for (( i=start ; i<=end ; i+=86400 )) ;do
printf "%(%Y%m%d)T\n" $i
done
3. Timezone issue!!
Warning there is an issue around summer vs winter time:
As a function
dayRange() {
local dR_Start dR_End dR_Crt
{
read dR_Start
read dR_End
} < <(date -f - +%s <<<${1:-yesterday}$'\n'${2:-tomorrow})
for ((dR_Crt=dR_Start ; dR_Crt<=dR_End ; dR_Crt+=86400 )) ;do
printf "%(%Y%m%d)T\n" $dR_Crt
done
}
Showing issue:
TZ=CET dayRange 20181026 20181030
20181026
20181027
20181028
20181028
20181029
Replacing printf "%(%Y%m%d)T\n" $dR_Crt by printf "%(%Y%m%dT%H%M)T\n" $dR_Crt could help:
20181026T0000
20181027T0000
20181028T0000
20181028T2300
20181029T2300
In order to avoid this issue, you just have to localize TZ=UTC at begin of function:
local dR_Start dR_End dR_Crt TZ=UTC
Final step for function: Avoiding useless forks
In order to improve performances, I try to reduce forks, avoiding syntax like:
for day in $(dayRange 20180329 20180404);do ...
# or
mapfile range < <(dayRange 20180329 20180404)
I use ability of function to directly set submited variables:
There is my purpose:
dayRange() { # <start> <end> <result varname>
local dR_Start dR_End dR_Crt dR_Day TZ=UTC
declare -a dR_Var='()'
{
read dR_Start
read dR_End
} < <(date -f - +%s <<<${1:-yesterday}$'\n'${2:-tomorrow})
for ((dR_Crt=dR_Start ; dR_Crt<=dR_End ; dR_Crt+=86400 )) ;do
printf -v dR_Day "%(%Y%m%d)T\n" $dR_Crt
dR_Var+=($dR_Day)
done
printf -v ${3:-dRange} "%s" "${dR_Var[*]}"
}
Then quick little bug test:
TZ=CET dayRange 20181026 20181030 bugTest
printf "%s\n" $bugTest
20181026
20181027
20181028
20181029
20181030
Seem fine. This could be used like:
dayRange 20180329 20180405 myrange
for day in $myrange ;do
echo "Doing something with string: '$day'."
done
2.2 Alternative using shell-connector
There is a shell function for adding background command in order to reduce forks.
wget https://f-hauri.ch/vrac/shell_connector.sh
. shell_connector.sh
Initiate background date +%Y%m%d and test: #0 must answer 19700101
newConnector /bin/date '-f - +%Y%m%d' #0 19700101
Then
j=20190329
while [ $j -le 20190404 ] ;do
echo $j; myDate "$j +1 day" j
done
3.++ Little bench
Let's try little 3 years range:
j=20160329
time while [ $j -le 20190328 ] ;do
echo $j;j=`TZ=UTC date -d "$j +1 day" +%Y%m%d`
done | wc
1095 1095 9855
real 0m1.887s
user 0m0.076s
sys 0m0.208s
More than 1 second on my system... Of course, there are 1095 forks!
time { dayRange 20160329 20190328 foo && printf "%s\n" $foo | wc ;}
1095 1095 9855
real 0m0.061s
user 0m0.024s
sys 0m0.012s
Only 1 fork, then bash builtins -> less than 0.1 seconds...
And with newConnector function:
j=20160329
time while [ $j -le 20190328 ] ;do echo $j
myDate "$j +1 day" j
done | wc
1095 1095 9855
real 0m0.109s
user 0m0.084s
sys 0m0.008s
Not as quick than using builtin integer, but very quick anyway.
Store the max and min dates using seconds since epoch. Don't use dates - they are not exact (GMT? UTC? etc.). Use seconds since epoch. Then increment your variable with the number of seconds in a day - ie. 24 * 60 * 60 seconds. In your loop, you can convert the number of seconds since epoch back to human readable date using date --date=#<number>. The following will work with POSIX shell and GNU's date utlity:
from=$(date --date='2018/04/04 00:00:00' +%s)
until=$(date --date='2018/04/07 00:00:00' +%s)
counter="$from"
while [ "$counter" -le "$until" ]; do
j=$(date --date=#"$counter" +%Y%m%d)
# do somth with j
echo $j
counter=$((counter + 24 * 60 * 60))
done
GNU's date is a little strange when parsing it's --date=FORMAT format string. I suggest to always feed it with %Y/%m/%d %H/%M/%S format string so that it always knows how to parse it.
Related
In bash on macOS, I would like to write a small script with dates (or any other program that would do) that gives me a list of dates in the format yyyymmdd of every Saturday of a given year and saves it to a variable.
For example, if I wanted to have a list of dates for all Saturdays of the year 1850, it should somehow look like this:
var = [ 18500105, 18500112, 18500119, …, 18501228 ]
with the below code:
list=()
for month in `seq -w 1 12`; do
for day in `seq -w 1 31`; do
list=( $(gdate -d "1850$month$day" '+%A %Y%m%d' | grep 'Saturday' | egrep -o '[[:digit:]]{4}[[:digit:]]{2}[[:digit:]]{2}' | tee /dev/tty) )
done
done
However, the above command does not write anything in the array list although it gives me the right output with tee.
How can I fix these issues?
Modifying Dennis Williamson's answer slightly to suit your requirement and to add the results into the array. Works on the GNU date and not on FreeBSD's version.
#!/usr/bin/env bash
y=1850
for d in {0..6}
do
# Identify the first day of the year that is a Saturday and break out of
# the loop
if (( $(date -d "$y-1-1 + $d day" '+%u') == 6))
then
break
fi
done
array=()
# Loop until the last day of the year, increment 7 days at a
# time and append the results to the array
for ((w = d; w <= $(date -d "$y-12-31" '+%j'); w += 7))
do
array+=( $(date -d "$y-1-1 + $w day" '+%Y%m%d') )
done
Now you can just print the results as
printf '%s\n' "${array[#]}"
To set up the GNU date on MacOS you need to do brew install coreutils and access the command as gdate to distinguish it from the native version provided.
Argh, just realised you need it for MacOS date.
I will leave the answer for others that do not have that restriction, but it will not work for you.
This is not quite what you want, but close:
year=1850
firstsat=$(date -d $year-01-01-$(date -d $year-01-01 +%w)days+6days +%Y%m%d)
parset a 'date -d '$firstsat'+{=$_*=7=}days +%Y%m%d' ::: {0..52}
echo ${a[#]}
It has the bug, that it finds the next 53 Saturdays, and the last of those may not be in current year.
parset is part of GNU Parallel.
I didn't do much error checking, but here's another implementation.
Takes day of week and target year as arguments.
Gets the julian day of the first matching weekday requested -
gets the epoch seconds of noon on that day -
as long as the year matches what was requested, adds that date to the array and adds a week's worth of seconds to the tracking variable.
Lather, rinse, repeat until no longer in that year.
$: typeset -f alldays
alldays () { local dow=$1 year=$2 julian=1;
until [[ "$dow" == "$( date +%a -d $year-01-0$julian )" ]]; do (( julian++ )); done;
es=$( date +%s -d "12pm $year-01-0$julian" );
allhits=( $( while [[ $year == "$( date +%Y -d #$es )" ]]; do date +%Y%m%d -d #$es; (( es+=604800 )); done; ) )
}
$: time alldays Sat 1850
real 0m9.931s
user 0m1.025s
sys 0m6.695s
$: printf "%s\n" "${allhits[#]}"
18500105
18500112
18500119
18500126
18500202
18500209
18500216
18500223
18500302
18500309
18500316
18500323
18500330
18500406
18500413
18500420
18500427
18500504
18500511
18500518
18500525
18500601
18500608
18500615
18500622
18500629
18500706
18500713
18500720
18500727
18500803
18500810
18500817
18500824
18500831
18500907
18500914
18500921
18500928
18501005
18501012
18501019
18501026
18501102
18501109
18501116
18501123
18501130
18501207
18501214
18501221
18501228
Using BASH I want to loop from a start to end date at ten-minute intervals.
I tried
begin_date="2015-01-01 00:00:00"
end_date="2015-02-20 00:00:00"
d=$begin_date
while [ "$d" != "$end_date" ]; do
echo $d
d=$(date -d "${d} + 10 min" +"%Y-%m-%d %H:%M")
done
But it didn't work. Looking at Bash:Looping thru dates
#This works
d=$(date -I -d "${d} + 1 day")
#This doesn't work
d=$(date -d "${d} + 1 day" +"%Y-%m-%d")
What am I missing in the format string?
The expression date -d "${d} + 10 min" seems not to produce a date with an offset of 10 minutes. In fact, when I run your code, I see a date counter going backwards. (Posting this diagnostic as part of your question would help others see where the problem is; you should not require others to run your code just to see what it does.)
Anyway, the sane way to do this is to convert the dates to Unix epoch, then take it from there.
for ((d=$(date -d "$begin_date" +%s); d <= $(date -d "$end_date" +%s); d += 600))
do
date -d #$d +"%F %H:%M"
done
Doing date arithmetic in the shell is probably going to be rather inefficient; converting this to e.g. Awk or Perl might be worth your time if you find it's too sluggish, or need to run it lots of times.
The example you linked to just needs to be adjusted slightly:
#!/bin/bash
## User-specified dates.
# See [GNU Coreutils: Date] for more info
# [GNU Coreutils: Date]: https://www.gnu.org/software/coreutils/manual/html_node/Combined-date-and-time-of-day-items.html#Combined-date-and-time-of-day-items
begin_date="2015-01-01T00:00"
end_date="2015-01-01T00:40"
# Run through `date` to ensure iso-8601 consistency
startdate=$(date --iso-8601='minutes' --date="${begin_date}")
enddate=$(date --iso-8601='minutes' --date="${end_date}")
# Do loop
d="$startdate"
while [ "$d" != "$enddate" ]; do
echo $d
d=$(date --iso-8601='minutes' --date="$d + 10 minutes")
done
Note that the options -I and -d are equivalent to --iso-8601 and --date respectively.
This question already has answers here:
A variable modified inside a while loop is not remembered
(8 answers)
Closed 4 years ago.
please bear with me and my questions, but I just started programming in bash yesterday effectively.
I have a script that does a lot of stuff so far. And I came to a point where I need to iterate through variables. I created these variables before using a while loop. The $Time variable for example looks like this:
2016-01-29 17:07:00Z
2016-01-29 17:26:20Z
2016-01-29 17:26:20Z
2016-01-29 00:07:00Z
The Grabinterval variable like this:
hour
minute
minute
day
The first step for me is to check if a different variable is not empty. If it is not I go on with checking line per line inside the $Grabinterval variable what kind it is day, hour or minute.
That is done by a while loop as well and works good. But no the problems are rising. Now I want to calculate a time difference between a time that was created earlier outside of the while loop and the time given in the first line of the $Time variable. I tried this using the following code:
while read -r line; do
if [[ ! -z "$Filelocation" ]]
then
if [[ $line = "day" || $line = "days" ]]
then
Interval="Days"
GrabTimeNew=$(date +'%Y-%m-%d 00:0'$UPOFFSET':00Z')
default=$(date --date "-1 day" +'%Y-%m-%d 00:0'$UPOFFSET':00Z')
start=$(date -d"$line$Time" +%s)
end=$(date -d"$GRABtime" +%s)
TimeDiff=$(( $start - $end ))
fi
fi
done <<< $Grabinterval
This is only on part of the bigger while loop, the other two parts looking for hour and minuteare pretty much the same.
The way I tried it here using $line$Time gives me following error message:
date: invalid date ‘day2016-01-29 17:07:00Z\n2016-01-29 17:26:20Z\n2016-01-29 17:26:20Z\n2016-01-29 00:07:00Z’
So it goes over all lines instead of only the dayline I want it to go through. Is there a way to use the first line of the $Timevariable inside the $Grabinterval variable?
I would love to use a for loop, but I have no idea how to use it later in the command block to have the wanted line read in the command block of the if statement.
Thanks,
BallerNacken
EDIT: Tried something like this now, but not working either:
while read -r GI TI; do
if [[ ! -z "$Filelocation" ]]
then
if [[ $GI = "day" || $GI = "days" ]]
then
Interval="Days"
GrabTimeNew=$(date +'%Y-%m-%d 00:0'$UPOFFSET':00Z')
default=$(date --date "-1 day" +'%Y-%m-%d 00:0'$UPOFFSET':00Z')
start=$(date -d"$TI" +%s)
end=$(date -d"$GRABtime" +%s)
TimeDiff=$(( $start - $end ))
fi
if [[ $GI = "hours" || $GI = "hour" ]]
then
Interval="Hours"
GrabTimeNew=$(date +'%Y-%m-%d %H:0'$UPOFFSET':00Z')
default=$(date --date "-1 hour" +'%Y-%m-%d %H:0'$UPOFFSET':00Z')
start=$(date -d"$TI" +%s)
end=$(date -d"$GRABtime" +%s)
TimeDiff2=$(( $start - $end ))
fi
if [[ $GI = "min" || $GI = "minutes" || $GI = "minute" ]]
then
Interval="Minutes"
GrabTimeNew=$(date +'%Y-%m-%d %H:%M:20Z')
default=$(date --date "-1 minute" +'%Y-%m-%d %H:%M:00Z')
start=$(date -d"$TI" +%s)
end=$(date -d"$GRABtime" +%s)
TimeDiff3=$(( $start - $end ))
fi
fi
done < <(paste <(echo "$Grabinterval") <(echo "$Time"))
I don't get any error messages, but no variables were created inside the if statement/while loop.
You might want to try something like this:
while read -r interval datetime; do
#...
done < <(paste <(echo "$Grabinterval") <(echo "$Time"))
That will read one line from Grabinterval and the corresponding line from Time
You need to quote the variable to keep the newlines in the <<< $var.
list=$'a\nb\nc'
while read a ; do echo $a ; done <<< $list
while read a ; do echo $a ; done <<< "$list"
I trying to print date between 2 dates using while loop in a bash script.
But when i execute i am getting below error:
test.sh: line 8: [: 02-12-14: integer expression expected
Below is my code, can anyone help me out
#!/bin/bash
sdate=02-12-14
edate=02-25-14
while [ "$sdate" -le "$edate" ]
do
echo $sdate
sdate=$(date +%m-%d-%y -d "$sdate + 1 day")
done
You should store them as timestamps:
#!/bin/bash
sdate=$(date -d '2014-02-12' +%s)
edate=$(date -d '2014-02-25' +%s)
while [[ sdate -le edate ]]; do
date -d "#$sdate" '+%m-%d-%y'
sdate=$(date -d "$(date -d "#${sdate}" ) + 1 day" +%s)
done
Output:
02-12-14
02-13-14
02-14-14
02-15-14
02-16-14
02-17-14
02-18-14
02-19-14
02-20-14
02-21-14
02-22-14
02-23-14
02-24-14
02-25-14
Always prefer [[ ]] over [ ] when it comes to conditional expressions in Bash. (( )) may also be a preference.
It requires GNU date. e.g. date --version = date (GNU coreutils) 8.21 ...
mm-dd-yy is not a format acceptable by date for input so I used yyyy-mm-dd which is acceptable.
I am trying to test how old ago a file was created (in seconds) with bash in an if statement. I need creation date, not modification.
Do you have any idea how to do this, without using a command like find with grep?
I'm afraid I cann't answer the question for creation time, but for last modification time you can use the following to get the epoch date, in seconds, since filename was last modified:
date --utc --reference=filename +%s
So you could then so something like:
modsecs=$(date --utc --reference=filename +%s)
nowsecs=$(date +%s)
delta=$(($nowsecs-$modsecs))
echo "File $filename was modified $delta secs ago"
if [ $delta -lt 120 ]; then
# do something
fi
etc..
Update
A more elgant way of doing this (again, modified time only): how do I check in bash whether a file was created more than x time ago?
Here is the best answer I found at the time being, but it's only for the modification time :
expr `date +%s` - `stat -c %Y /home/user/my_file`
If your system has stat:
modsecs=$(stat --format '%Y' filename)
And you can do the math as in Joel's answer.
you can use ls with --full-time
file1="file1"
file2="file2"
declare -a array
i=0
ls -go --full-time "$file1" "$file2" | { while read -r a b c d time f
do
time=${time%.*}
IFS=":"
set -- $time
hr=$1;min=$2;sec=$3
hr=$(( hr * 3600 ))
min=$(( min * 60 ))
totalsecs=$(( hr+min+sec ))
array[$i]=$totalsecs
i=$((i+1))
unset IFS
done
echo $(( ${array[0]}-${array[1]} ))
}