I am trying to get the time difference between two dates as given below in Bash script. However I am not successful
head_info: 05-31-2017:04:27:37
tail_info: 05-31-2017:04:30:57
the problem is that after Reformation above time and while trying to calculate in seconds due to space, it is ignoring time.
This is my script:
fm_head_info=(${head_info:6:4}"-"${head_info:0:2}"-"${head_info:3:2}" \
"${head_info:11:8})
fm_tail_info=(${tail_info:6:4}"-"${tail_info:0:2}"-"${tail_info:3:2}" \
"${tail_info:11:8})
$ fm_head_info
-bash: 2017-05-31: command not found
Thank you
Let's define your shell variables:
$ tail_info=05-31-2017:04:30:57
$ head_info=05-31-2017:04:27:37
Now, let's create a function to convert those dates to seconds-since-epoch:
$ date2sec() { date -d "$(sed 's|-|/|g; s|:| |' <<<"$*")" +%s; }
To find the time difference between those two date in seconds:
$ echo $(( $(date2sec "$tail_info") - $(date2sec "$head_info") ))
200
As written above, this requires bash (or similar advanced shell) and GNU date. In other words, this should work on any standard Linux. To make this work on OSX, some changes to the date command will likely be necessary.
How it works
Starting with the innermost command inside the function date2sec, we have:
sed 's|-|/|g; s|:| |' <<<"$*"
In the argumnet to the function, this replaces all - with / and it replaces the first : with a space. This converts the the dates from the format in your input to one that the GNU date function will understand. For example:
$ sed 's|-|/|g; s|:| |' <<<"05-31-2017:04:30:57"
05/31/2017 04:30:57
With this form, we can use date to find seconds-since-epoch:
$ date -d "05/31/2017 04:30:57" +%s
1496230257
And, for the head_info:
$ date -d "05/31/2017 04:27:37" +%s
1496230057
Now that we have that, all that is left is to subtract the times:
$ echo $(( 1496230257 - 1496230057 ))
200
Your immediate issue is the inclusion of erroneous (...) surrounding your string indexed assignment and your questionable quoting. It looks like you intended:
fm_head_info="${head_info:6:4}-${head_info:0:2}-${head_info:3:2} ${head_info:11:8}"
fm_tail_info="${tail_info:6:4}-${tail_info:0:2}-${tail_info:3:2} ${tail_info:11:8}"
Your use of string indexes is correct, e.g.
#!/bin/bash
head_info=05-31-2017:04:27:37
tail_info=05-31-2017:04:30:57
fm_head_info="${head_info:6:4}-${head_info:0:2}-${head_info:3:2} ${head_info:11:8}"
fm_tail_info="${tail_info:6:4}-${tail_info:0:2}-${tail_info:3:2} ${tail_info:11:8}"
echo "fm_head_info: $fm_head_info"
echo "fm_tail_info: $fm_tail_info"
Example Use/Output
$ bash headinfo.sh
fm_head_info: 2017-05-31 04:27:37
fm_tail_info: 2017-05-31 04:30:57
You can then do something similar with the differences in date -d "$var" +%s as John shows in his answer to arrive at the time difference. Note, string indexes are limited to bash, while a sed solution (absent the herestring) would be portable on all POSIX shells.
Related
I have two dates in forms like: YYYYMMDDHH and want to calculate the differences (in hours) between these two dates. For example
start_date=1996010100
end_date=1996010122
which stands for two dates: 1996-01-01 00:00:00 and 1996-01-01 22:00:00. I want to use date to calculate the difference in hours, the result shall be 22 hours. I tried with
START=$(date -d "$start_date" +"%s")
END=$(date -d "$end_date" +"%s")
HOURS=$(bc -l <<< "($END - $START) / 3600")
but it failed...
So how can I do this? Thanks!
For performance reasons we want to limit the number of sub-process calls we need to invoke:
use bash substring functionality to convert inputs into usable date/time strings
use bash math to replace bc call
bash substring functionality to break the inputs into a usable date/time format, eg:
# convert to usable date/time format:
$ start_date=1996010100
$ echo "${start_date:0:4}-${start_date:4:2}-${start_date:6:2} ${start_date:8:2}:00:00"
1996-01-01 00:00:00
# convert to epoch/seconds:
$ start=$(date -d "${start_date:0:4}-${start_date:4:2}-${start_date:6:2} ${start_date:8:2}:00:00" +"%s")
$ echo $start
820476000
Applying to ${end_date} and using bash math:
$ end_date=1996010122
$ end=$(date -d "${end_date:0:4}-${end_date:4:2}-${end_date:6:2} ${end_date:8:2}:00:00" +"%s")
$ echo $end
820555200
$ hours=$(( (end - start) / 3600))
$ echo $hours
22
This leaves us with 2 sub-process calls ($(date ...)). While other languages/tools (awk, perl, etc) can likely speed this up a bit, if you need to store the result in a bash variable then you're looking at needing at least 1 sub-process call (ie, hours=$(awk/perl/??? ...)).
If performance is really important (eg, needing to perform this 1000's of times) take a look at this SO answer that uses a fifo, background date process and io redirection ... yeah, a bit more coding and a bit more convoluted but also a bit faster for large volumes of operations.
busybox date can do the trick
start_date=1996010100
end_date=1996010122
START=$(busybox date -D "%Y%m%d%H" -d "$start_date" +"%s")
END=$(busybox date -D "%Y%m%d%H" -d "$end_date" +"%s")
HOURS=$(bc -l <<< "scale=0;($END - $START) / 3600")
echo $HOURS
If it's possible for you to use a more fully-featured scripting language like Python, it'll provide a much more pleasant and understandable date parsing experience, and is probably installed by default (datetime is also a standard Python library)
Structured with shell vars
start_date=1996010100
end_date=1996010122
python -c "import datetime ; td = datetime.datetime.strptime('${end_date}', '%Y%m%d%H') - datetime.datetime.strptime('${start_date}', '%Y%m%d%H') ; print(int(td.total_seconds() / 3600))"
Structured to read dates and format code from stdin
echo '%Y%m%d%H' 1996010100 1996010122 | python -c "import datetime,sys ; fmt, date_start, date_end = sys.stdin.read().strip().split() ; td = datetime.datetime.strptime(date_end, fmt) - datetime.datetime.strptime(date_start, fmt) ; print(int(td.total_seconds() / 3600))"
Should work with both Python 3 and Python 2.7
format codes available here (1989 C standard)
https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes
which stands for two dates: 1996-01-01 00:00:00
So convert it to that form if it stands for it.
start_date=1996010100
start_date=$(sed -E 's/(....)(..)(..)(..)/\1-\2-\3 \4:00:00/' <<<"$start_date")
start=$(date -d "$start_date" +"%s")
and the same with end.
the most simple way is to install "dateutils" using this command
sudo apt-get install dateutils
Run these commands to get the difference in seconds:
dateutils.ddiff -i '%Y%m%d%H%M%S' 20200817040001 20210817040101
output:
31536060s
next step: Simply divide by 86400 to get the number of days or similarly for hours and minutes :)
I have been trying to do this all afternoon and cannot figure out how to do this. I'm running MXLinux and from the commandline am trying (unsucessfully) to batch edit a bunch of filenames (I've about 500 so don't want to do this by hand) from:
2020-August-15.pdf
2021-October-15.pdf
To:
2020-08-15.pdf
2021-10-15.pdf
I cannot find anything that does this (in a way I understand) so am wondering. Is this possible or am I to do this by hand?
Admittedly I'm not very good with Bash but I can use sed, awk, rename, date, etc. I just can't seem to find a way to combine them to rename my files.
I cannot find anything on here that has been of any help in doing this.
Many thanks.
EDIT:
I'm looking for a way to combine commands and ideally not have to overtly for-loop through the files and the months. What I mean is I would prefer, and was trying to, pipe ls into a command combination to convert as specified above. Sorry for the confusion.
EDIT 2:
Thank you to everyone who came up with answers, and for you patience with my lack of ability. I don't think I'm qualified to make a decision as to the best answer however have settled, for my use-case on the following:
declare -A months=( [January]=01 [February]=02 [March]=03 [April]=04 [May]=05\
[June]=06 [July]=07 [August]=08 [September]=09 [October]=10 [November]=11 [December]=12 )
for oldname in 202[01]-[A-za-z]*-15.pdf
do
IFS=-. read y m d ext <<< "${oldname}"
mv "$oldname" "$y-${months[$m]}-$d.$ext"
done
I think this offer the best flexibility. I would have liked the date command but don't know how to not have the file extension hard coded. I was unaware of the read command or that you could use patterns in the for-loop.
I have learned a lot from this thread so again thank you all. Really my solution is a cross of most of the solutions below as I've taken from them all.
With just Bash built-ins, try
months=(\
January February March April May June \
July August September October November December)
for file in ./*; do
dst=$file
for ((i=1; i<=${#months[#]}; ++i)); do
((i<10)) && i=0$i
dst=${dst//${months[$i]}/$i}
done
mv -- "$file" "$dst"
done
This builds up an array of month names, and loops over it to find the correct substitution.
The line ((i<10)) && i=0$i adds zero padding for single-digit month numbers; remove it if that's undesired.
As an aside, you should basically never use ls in scripts.
The explicit loop could be avoided if you had a command which already knows how to rename files, but this implements that command. If you want to save it in a file, replace the hard-coded ./* with "$#", add a #!/bin/bash shebang up top, save it as monthrenamer somewhere in your PATH, and chmod a+x monthrenamer. Then you can run it like
monthrenamer ./*
to rename all the files in the current directory without an explicit loop, or a more restricted wildcard argument to only select a smaller number of files, like
monthrenamer /path/to/files/2020*.pdf
You could run date twelve times to populate the array, but it's not like hard-coding the month names is going to be a problem. We don't expect them to change (and calling twelve subprocesses at startup just to avoid that seems quite excessive in this context).
As an aside, probably try to fix the process which creates these files to produce machine-readable file names. It's fairly obvious to a human, too, that 2021-07 refers to the month of July, whereas going the other way is always cumbersome (you will need to work around it in every tool or piece of code which wants to order the files by name).
Assuming you have the GNU version of date(1), you could use date -d to map the month names to numbers:
for f in *.pdf; do
IFS=- read y m d <<<"${f%.pdf}"
mv "$f" "$(date -d "$m $d, $y" +%F.pdf)"
done
I doubt it's any more efficient than your sed -e 's/January/01/' -e 's/February/02/' etc, but it does feel less tedious to type. :)
Explanation:
Loop over the .pdf files, setting f to each filename in turn.
The read line is best explained right to left:
a.
"${f%.pdf}" expands to the filename without the .pdf part, e.g. "2020-August-15".
b. <<< turns that value into a here-string, which is a mechanism for feeding a string as standard input to some command. Essentially, x <<<y does the same thing as echo y | x, with the important difference that the x command is run in the current shell instead of a subshell, so it can have side effects like setting variables.
c. read is a shell builtin that by default reads a single line of input and assigns it to one or more shell variables.
d. IFS is a parameter that tells the shell how to split lines up into words. Here we're setting it – only for the duration of the read command – to -. That tells read to split the line it reads on hyphens instead of whitespace; IFS=- read y m d <<<"2020-August-15" assigns "2020" to y, "August" to m, and "15" to d.
The GNU version of date(1) has a -d parameter that tells it to display another date instead of the current one. It accepts a number of different formats itself, sadly not including "yyyy-Mon-dd", which is why I had to split the filename up with read. But it does accept "Mon dd, yyyy", so that's what I pass to it. +%F.pdf tells it that when it prints the date back out it should do so ISO-style as "yyyy-mm-dd", and append ".pdf" to the result. ("%F" is short for "%Y-%m-%d"; I could also have used -I instead of +anything and moved the .pdf outside the command expansion.)
f. The call to date is wrapped in $(...) to capture its output, and that result is used as the second parameter to mv to rename the files.
Another way with POSIX shell:
# Iterate over pattern that will exclude already renamed pdf files
for file in [0-9][0-9][0-9][0-9]-[^0-9]*.pdf
do
# Remove echo if result match expectations
echo mv -- "$file" "$(
# Set field separator to - or . to split filename components
IFS=-.
# Transfer filename components into arguments using IFS
set -- $file
# Format numeric date string
date --date "$3 $2 $1" '+%Y-%m-%d.pdf'
)"
done
If you are using GNU utilities and the Perl version of rename (not the util-linux version), you can build a one-liner quite easily:
rename "$(
seq -w 1 12 |
LC_ALL=C xargs -I# date -d 1970-#-01 +'s/^(\d{4}-)%B(-\d{2}\.pdf)$/$1%m$2/;'
)" *.pdf
You can shorten if you don't care about safety (or legibility)... :-)
rename "$(seq -f%.f/1 12|date -f- +'s/%B/%m/;')" *.pdf
What I mean is I would prefer, and was trying to, pipe ls into a command combination to convert as specified above.
Well, you may need to implement that command combination then. Here’s one consisting of a single “command” and in pure Bash without external processes. Pipe your ls output into that and, once satisfied with the output, remove the final echo…
#!/bin/bash
declare -Ar MONTHS=(
[January]=01
[February]=02
[March]=03
[April]=04
[May]=05
[June]=06
[July]=07
[August]=08
[September]=09
[October]=10
[November]=11
[December]=12)
while IFS= read -r path; do
IFS=- read -ra segments <<<"$path"
segments[-2]="${MONTHS["${segments[-2]}"]}"
IFS=- new_path="${segments[*]}"
echo mv "$path" "$new_path"
done
What is working for me in Mac OS 12.5 with GNU bash, version 3.2.57(1)-release (arm64-apple-darwin21)
is the following :
for f in *.pdf; do mv "$f" "$(echo $f |sed -e 's/Jan/-01-/gi' -e 's/Feb/-02-/gi' -e 's/Mar/-03-/gi' -e 's/Apr/-04-/gi' -e 's/May/-05-/gi' -e 's/jun/-06-/gi' -e 's/Jul/-07-/gi' -e 's/Aug/-08-/gi' -e 's/Sep/-09-/gi' -e 's/Oct/-10-/gi' -e 's/Nov/-11-/gi' -e 's/Dec/-12-/gi' )"; done
Note the original file had the month expressed in three litters in my case :
./04351XXX73435-2021Mar08-2021Apr08.pdf
I have this function in R, which I use to produce a list of dates:
#! usr/bin/env Rscript
date_seq = function(){
args = commandArgs(trailingOnly = TRUE)
library(lubridate)
days = seq(ymd(args[1]),ymd(args[2]),1)
days =format(days, "%Y%m%d")
return(days)
}
date_seq()
I call this function in a bash script to create a vector of dates:
Rscript date_seq.R 20160730 20160801 > dates
I define a couple of other string variables in the bash script:
home_url="https://pando-rgw01.chpc.utah.edu/hrrr/sfc/"
file_name="/hrrr.t{00-23}z.wrfsfcf00.grib2"
The final goal is to create a vector of download links, that incorporates the three variables home_url, date and file_name, like so:
"https://pando-rgw01.chpc.utah.edu/hrrr/sfc/20160730/hrrr.t{00-23}z.wrfsfcf00.grib2"
"https://pando-rgw01.chpc.utah.edu/hrrr/sfc/20160731/hrrr.t{00-23}z.wrfsfcf00.grib2"
"https://pando-rgw01.chpc.utah.edu/hrrr/sfc/20160801/hrrr.t{00-23}z.wrfsfcf00.grib2"
I tried a few lines in bash script:
for date in $dates; do download_url=$home_url$date$hrrr_file; cat
$download_url; done
for date in $dates; do download_url="${home_url}${date}${hrrr_file}"; cat $download_url;
done
for date in $dates; do download_url="$home_url"; download_url+="$date"; download_url+="$hrrr_file"; cat $download_url; done
None of these produce the output I expect. I am not sure if the download_url variable is not being produced, or is being produced and stored somewhere, and I am not able to reproduce it. Can anyone please help me understand?
Edit
Results of trying the suggestions below:
#triplee suggested using
sed "s#.*#$home_url&$hrrr_file#" "dates"
and
while read -r date; do; printf '%s%s%s\n' "$home_url" "$date" "$hrrr_file"; done <dates
Both of these produce this output:
https://pando-rgw01.chpc.utah.edu/hrrr/sfc/[1] "20160730" "20160731" "20160801"/hrrr.t{00-23}z.wrfsfcf00.grib2
#xdhmoore suggested using
for date in $(cat dates); do; echo ${home_url}${date}${hrrr_file}"; done
which produces this output:
https://pando-rgw01.chpc.utah.edu/hrrr/sfc/[1]/hrrr.t{00-23}z.wrfsfcf00.grib2
https://pando-rgw01.chpc.utah.edu/hrrr/sfc/"20160730"/hrrr.t{00-23}z.wrfsfcf00.grib2
https://pando-rgw01.chpc.utah.edu/hrrr/sfc/"20160731"/hrrr.t{00-23}z.wrfsfcf00.grib2
https://pando-rgw01.chpc.utah.edu/hrrr/sfc/"20160801"/hrrr.t{00-23}z.wrfsfcf00.grib2`
Both are not the output I am expecting, though the solution by #xdhmoore is closer. But I see another problem in #xdhmoore's solution: The quotations around the date in output. The output of cat dates looks like this: "20160730" "20160731" "20160801", so I think I have to rework the function or the way I call it in the bash script as well.
I'll keep updating the question to reflect the output of all suggestions, since it is simpler to do so than trying to answer each comment. As always, thanks a lot!
The for statement loops over the tokens you give it as arguments, not the contents of files.
You seem to be looking for
sed "s#.*#$home_url&$hrrr_file#" "dates"
The token & recalls the text which was matched by the regex in a sed substitution.
The same thing could be done vastly more slowly with a shell loop;
while read -r date; do
printf '%s%s%s\n' "$home_url" "$date" "$hrrr_file"
done <dates
which illustrates how to (slowly) iterate over the lines in a file without the use of external utilities.
Either of hese can be piped to xargs curl (or perhaps xargs -n 1 curl); or you could refactor the while loop;
while read -r date; do
curl "$home_url$date$hrrr_file"
done <dates
As noted in comments, cat is a command for copying files, not echoing text; for the latter, use echo or (for any nontrivial formatting) printf.
Update: The above assumes your R output generated one date per line. To split the file into lines and remove quotes around the values, you can preprocess with sed 's/"\([^"]\)" */\1\n/g' "dates" (provided your sed dialects supports \n as an escape for newline); or perhaps do
sed "s#\"\([^\"]*\)\" *#$home_url\\1$frrr_file\\
#g" "dates"
again with some reservation for differences between sed dialects. In the worst case, maybe switch to Perl, which actually brings some relief to the backslashitis, but requires new backslashes in other places:
perl -pe "s#\"(\d+)\" *#$home_url\$1$frrr_file\n#g" "dates"
But probably a better solution is to change your R script so it doesn't produce wacky output. Or just don't use R in the first place. See e.g. https://stackoverflow.com/a/3494814/874188 for how to get dates from Perl. Or if you have GNU date, try
#!/bin/bash
start=$(date -d "$1" +%s)
end=$(date -d "$2" +%s)
for ((i=start; i<=end; i+=60*60*24)); do
date -d "#$i" +%Y%m%d
done
(If you are on a Mac or similar, the date program won't accept a date as an argument to -d and you will have to use slightly different syntax. It's not hard to do but this answer has too many speculations already.)
I'm currently creating a shell script that will run a python code once an hour that collects, processes, and displays data from a radar for the previous hour.
The python code I am using requires a UTC begin time and end time in format "YYYYMMDDHHmm". So far, I have found using the unix command date -u +"%Y%m%d%H%M" will retrieve my current time in the correct format, but I have not been able to find a way to subtract 60 minutes from this first time and have it output the "start" time/
code I have tried:
date +"%Y%m%d%H%M-60" >> out: 201908201833-60
now= date -u +"%Y%m%d%H%M" >> out:201908201834
echo "$now - 60" >> out: - 60
I'm just starting to self teach/learn shell coding and I am more comfortable with python coding which is why my attempts are set up more like how you would write with python. I'm sure there is a way to store the variable and have it subtract 60 from the end time, but I have not been able to find a good online source for this (both on here and via Google).
You can use -d option in date:
date -u +"%Y%m%d%H%M" -d '-60 minutes'
or else subtract 1 hour instead of 60 minutes:
date -u +"%Y%m%d%H%M" -d '-1 hour'
To be able to capture this value in a variable, use command substitution:
now=$(date -u +"%Y%m%d%H%M" -d '-1 hour')
On OSX (BSD) use this date command as -d is not supported:
now=$(date -u -v-1H +"%Y%m%d%H%M")
Your current attempt has some simple shell script errors.
now= date -u +"%Y%m%d%H%M" >> out:201908201834
This assigns an empty string to the variable now and then runs the date command as previously. If the plan is to capture the output to the variable now, the syntax for that is
now=$(date -u +"%Y%m%d%H%M")
Next up, you try to
echo "$now - 60"
which of course will output the literal string
201908201834 - 60
rather than perform arithmetic evaluation. You can say
echo "$((now - 60))"
to subtract 60 from the value and echo that -- but of course, date arithmetic isn't that simple; subtracting 60 from 201908210012 will not produce 201908202312 like you would hope.
If you have GNU date (that's a big if if you really want to target any Unix) you could simply have done
date -u -d "60 minutes ago" +%F%H%M
but if you are doing this from Python anyway, performing the date extraction and manipulation in Python too will be a lot more efficient as well as more portable.
from datetime import datetime, timedelta
dt = datetime.strptime(when,'%Y%m%d%H%M')
print(dt - timedelta(minutes=60))
The shell command substitution $(command) and arithmetic evaluation $((expression)) syntaxes look vaguely similar, but are really unrelated. Both of them have been introduced after the fundamental shell syntax was already stable, so they had to find a way to introduce new syntax which didn't already have a well-established meaning in the original Bourne shell.
I have created a script that adds seconds from 1/1 1970 to a file called datefile.txt. IT looks like this:
echo $(date +%s) > datefile.txt
What i want to do now is to read that value from datefile.txt and subtract the current value (current amount of seconds) from the previous stored in datefile.txt.
I have tried a bunch of things but I lack knowledge of bash syntax and general knowledge.
Most recent attempt:
d<datefile.txt
echo $(( d-date +%s ))
Obviously this is a retarded solution because it doesn't work.
Im running this on Ubuntu.
After doing :
echo $(date +%s) > datefile.txt
You can do this :
oldtime=$(<datefile.txt)
difference=$(( $(date +%s) - oldtime))
I assume you actually want to substract the old value from the new one to get a positive result.
The $(<datefile.txt) expands to the whole content of a file.
Also, note that when you want to use the output of a command inside an arithmetic expression, you have to enclose that command with $() (command substitution) or else the shell will not know this is a command you want to execute, as opposed to arguments that need to be read directly.