I have created a script that adds seconds from 1/1 1970 to a file called datefile.txt. IT looks like this:
echo $(date +%s) > datefile.txt
What i want to do now is to read that value from datefile.txt and subtract the current value (current amount of seconds) from the previous stored in datefile.txt.
I have tried a bunch of things but I lack knowledge of bash syntax and general knowledge.
Most recent attempt:
d<datefile.txt
echo $(( d-date +%s ))
Obviously this is a retarded solution because it doesn't work.
Im running this on Ubuntu.
After doing :
echo $(date +%s) > datefile.txt
You can do this :
oldtime=$(<datefile.txt)
difference=$(( $(date +%s) - oldtime))
I assume you actually want to substract the old value from the new one to get a positive result.
The $(<datefile.txt) expands to the whole content of a file.
Also, note that when you want to use the output of a command inside an arithmetic expression, you have to enclose that command with $() (command substitution) or else the shell will not know this is a command you want to execute, as opposed to arguments that need to be read directly.
Related
I'm currently creating a shell script that will run a python code once an hour that collects, processes, and displays data from a radar for the previous hour.
The python code I am using requires a UTC begin time and end time in format "YYYYMMDDHHmm". So far, I have found using the unix command date -u +"%Y%m%d%H%M" will retrieve my current time in the correct format, but I have not been able to find a way to subtract 60 minutes from this first time and have it output the "start" time/
code I have tried:
date +"%Y%m%d%H%M-60" >> out: 201908201833-60
now= date -u +"%Y%m%d%H%M" >> out:201908201834
echo "$now - 60" >> out: - 60
I'm just starting to self teach/learn shell coding and I am more comfortable with python coding which is why my attempts are set up more like how you would write with python. I'm sure there is a way to store the variable and have it subtract 60 from the end time, but I have not been able to find a good online source for this (both on here and via Google).
You can use -d option in date:
date -u +"%Y%m%d%H%M" -d '-60 minutes'
or else subtract 1 hour instead of 60 minutes:
date -u +"%Y%m%d%H%M" -d '-1 hour'
To be able to capture this value in a variable, use command substitution:
now=$(date -u +"%Y%m%d%H%M" -d '-1 hour')
On OSX (BSD) use this date command as -d is not supported:
now=$(date -u -v-1H +"%Y%m%d%H%M")
Your current attempt has some simple shell script errors.
now= date -u +"%Y%m%d%H%M" >> out:201908201834
This assigns an empty string to the variable now and then runs the date command as previously. If the plan is to capture the output to the variable now, the syntax for that is
now=$(date -u +"%Y%m%d%H%M")
Next up, you try to
echo "$now - 60"
which of course will output the literal string
201908201834 - 60
rather than perform arithmetic evaluation. You can say
echo "$((now - 60))"
to subtract 60 from the value and echo that -- but of course, date arithmetic isn't that simple; subtracting 60 from 201908210012 will not produce 201908202312 like you would hope.
If you have GNU date (that's a big if if you really want to target any Unix) you could simply have done
date -u -d "60 minutes ago" +%F%H%M
but if you are doing this from Python anyway, performing the date extraction and manipulation in Python too will be a lot more efficient as well as more portable.
from datetime import datetime, timedelta
dt = datetime.strptime(when,'%Y%m%d%H%M')
print(dt - timedelta(minutes=60))
The shell command substitution $(command) and arithmetic evaluation $((expression)) syntaxes look vaguely similar, but are really unrelated. Both of them have been introduced after the fundamental shell syntax was already stable, so they had to find a way to introduce new syntax which didn't already have a well-established meaning in the original Bourne shell.
I am trying to get the time difference between two dates as given below in Bash script. However I am not successful
head_info: 05-31-2017:04:27:37
tail_info: 05-31-2017:04:30:57
the problem is that after Reformation above time and while trying to calculate in seconds due to space, it is ignoring time.
This is my script:
fm_head_info=(${head_info:6:4}"-"${head_info:0:2}"-"${head_info:3:2}" \
"${head_info:11:8})
fm_tail_info=(${tail_info:6:4}"-"${tail_info:0:2}"-"${tail_info:3:2}" \
"${tail_info:11:8})
$ fm_head_info
-bash: 2017-05-31: command not found
Thank you
Let's define your shell variables:
$ tail_info=05-31-2017:04:30:57
$ head_info=05-31-2017:04:27:37
Now, let's create a function to convert those dates to seconds-since-epoch:
$ date2sec() { date -d "$(sed 's|-|/|g; s|:| |' <<<"$*")" +%s; }
To find the time difference between those two date in seconds:
$ echo $(( $(date2sec "$tail_info") - $(date2sec "$head_info") ))
200
As written above, this requires bash (or similar advanced shell) and GNU date. In other words, this should work on any standard Linux. To make this work on OSX, some changes to the date command will likely be necessary.
How it works
Starting with the innermost command inside the function date2sec, we have:
sed 's|-|/|g; s|:| |' <<<"$*"
In the argumnet to the function, this replaces all - with / and it replaces the first : with a space. This converts the the dates from the format in your input to one that the GNU date function will understand. For example:
$ sed 's|-|/|g; s|:| |' <<<"05-31-2017:04:30:57"
05/31/2017 04:30:57
With this form, we can use date to find seconds-since-epoch:
$ date -d "05/31/2017 04:30:57" +%s
1496230257
And, for the head_info:
$ date -d "05/31/2017 04:27:37" +%s
1496230057
Now that we have that, all that is left is to subtract the times:
$ echo $(( 1496230257 - 1496230057 ))
200
Your immediate issue is the inclusion of erroneous (...) surrounding your string indexed assignment and your questionable quoting. It looks like you intended:
fm_head_info="${head_info:6:4}-${head_info:0:2}-${head_info:3:2} ${head_info:11:8}"
fm_tail_info="${tail_info:6:4}-${tail_info:0:2}-${tail_info:3:2} ${tail_info:11:8}"
Your use of string indexes is correct, e.g.
#!/bin/bash
head_info=05-31-2017:04:27:37
tail_info=05-31-2017:04:30:57
fm_head_info="${head_info:6:4}-${head_info:0:2}-${head_info:3:2} ${head_info:11:8}"
fm_tail_info="${tail_info:6:4}-${tail_info:0:2}-${tail_info:3:2} ${tail_info:11:8}"
echo "fm_head_info: $fm_head_info"
echo "fm_tail_info: $fm_tail_info"
Example Use/Output
$ bash headinfo.sh
fm_head_info: 2017-05-31 04:27:37
fm_tail_info: 2017-05-31 04:30:57
You can then do something similar with the differences in date -d "$var" +%s as John shows in his answer to arrive at the time difference. Note, string indexes are limited to bash, while a sed solution (absent the herestring) would be portable on all POSIX shells.
It seems that by executing code in PS0 and PS1 variables (which are eval'ed before and after a prompt command is run, as I understand) it should be possible to record time of each running command and display it in the prompt. Something like that:
user#machine ~/tmp
$ sleep 1
user#machine ~/tmp 1.01s
$
However, I quickly got stuck with recording time in PS0, since something like this doesn't work:
PS0='$(START=$(date +%s.%N))'
As I understand, START assignment happens in a sub-shell, so it is not visible in the outer shell. How would you approach this?
I was looking for a solution to a different problem and came upon this question, and decided that sounds like a cool feature to have. Using #Scheff's excellent answer as a base in addition to the solutions I developed for my other problem, I came up with a more elegant and full featured solution.
First, I created a few functions that read/write the time to/from memory. Writing to the shared memory folder prevents disk access and does not persist on reboot if the files are not cleaned for some reason
function roundseconds (){
# rounds a number to 3 decimal places
echo m=$1";h=0.5;scale=4;t=1000;if(m<0) h=-0.5;a=m*t+h;scale=3;a/t;" | bc
}
function bash_getstarttime (){
# places the epoch time in ns into shared memory
date +%s.%N >"/dev/shm/${USER}.bashtime.${1}"
}
function bash_getstoptime (){
# reads stored epoch time and subtracts from current
local endtime=$(date +%s.%N)
local starttime=$(cat /dev/shm/${USER}.bashtime.${1})
roundseconds $(echo $(eval echo "$endtime - $starttime") | bc)
}
The input to the bash_ functions is the bash PID
Those functions and the following are added to the ~/.bashrc file
ROOTPID=$BASHPID
bash_getstarttime $ROOTPID
These create the initial time value and store the bash PID as a different variable that can be passed to a function. Then you add the functions to PS0 and PS1
PS0='$(bash_getstarttime $ROOTPID) etc..'
PS1='\[\033[36m\] Execution time $(bash_getstoptime $ROOTPID)s\n'
PS1="$PS1"'and your normal PS1 here'
Now it will generate the time in PS0 prior to processing terminal input, and generate the time again in PS1 after processing terminal input, then calculate the difference and add to PS1. And finally, this code cleans up the stored time when the terminal exits:
function runonexit (){
rm /dev/shm/${USER}.bashtime.${ROOTPID}
}
trap runonexit EXIT
Putting it all together, plus some additional code being tested, and it looks like this:
The important parts are the execution time in ms, and the user.bashtime files for all active terminal PIDs stored in shared memory. The PID is also shown right after the terminal input, as I added display of it to PS0, and you can see the bashtime files added and removed.
PS0='$(bash_getstarttime $ROOTPID) $ROOTPID experiments \[\033[00m\]\n'
As #tc said, using arithmetic expansion allows you to assign variables during the expansion of PS0 and PS1. Newer bash versions also allow PS* style expansion so you don't even need a subshell to get the current time. With bash 4.4:
# PS0 extracts a substring of length 0 from PS1; as a side-effect it causes
# the current time as epoch seconds to PS0time (no visible output in this case)
PS0='\[${PS1:$((PS0time=\D{%s}, PS1calc=1, 0)):0}\]'
# PS1 uses the same trick to calculate the time elapsed since PS0 was output.
# It also expands the previous command's exit status ($?), the current time
# and directory ($PWD rather than \w, which shortens your home directory path
# prefix to "~") on the next line, and finally the actual prompt: 'user#host> '
PS1='\nSeconds: $((PS1calc ? \D{%s}-$PS0time : 0)) Status: $?\n\D{%T} ${PWD:PS1calc=0}\n\u#\h> '
(The %N date directive does not seem to be implemented as part of \D{...} expansion with bash 4.4. This is a pity since we only have a resolution in single second units.)
Since PS0 is only evaluated and printed if there is a command to execute, the PS1calc flag is set to 1 to do the time difference (following the command) in PS1 expansion or not (PS1calc being 0 means PS0 was not previously expanded and so didn't re-evaluate PS1time). PS1 then resets PS1calc to 0. In this way an empty line (just hitting return) doesn't accumulate seconds between return key presses.
One nice thing about this method is that there is no output when you have set -x active. No subshells or temporary files in sight: everything is done within the bash process itself.
I took this as puzzle and want to show the result of my puzzling:
First I fiddled with time measurement. The date +%s.%N (which I didn't realize before) was where I started from. Unfortunately, it seems that bashs arithmetic evaluation seems not to support floating points. Thus, I chosed something else:
$ START=$(date +%s.%N)
$ awk 'BEGIN { printf("%fs", '$(date +%s.%N)' - '$START') }' /dev/null
8.059526s
$
This is sufficient to compute the time difference.
Next, I confirmed what you already described: sub-shell invocation prevents usage of shell variables. Thus, I thought about where else I could store the start time which is global for sub-shells but local enough to be used in multiple interactive shells concurrently. My solution are temp. files (in /tmp). To provide a unique name I came up with this pattern: /tmp/$USER.START.$BASHPID.
$ date +%s.%N >/tmp/$USER.START.$BASHPID ; \
> awk 'BEGIN { printf("%fs", '$(date +%s.%N)' - '$(cat /tmp/$USER.START.$BASHPID)') }' /dev/null
cat: /tmp/ds32737.START.11756: No such file or directory
awk: cmd. line:1: BEGIN { printf("%fs", 1491297723.111219300 - ) }
awk: cmd. line:1: ^ syntax error
$
Damn! Again I'm trapped in the sub-shell issue. To come around this, I defined another variable:
$ INTERACTIVE_BASHPID=$BASHPID
$ date +%s.%N >/tmp/$USER.START.$INTERACTIVE_BASHPID ; \
> awk 'BEGIN { printf("%fs", '$(date +%s.%N)' - '$(cat /tmp/$USER.START.$INTERACTIVE_BASHPID)') }' /dev/null
0.075319s
$
Next step: fiddle this together with PS0 and PS1. In a similar puzzle (SO: How to change bash prompt color based on exit code of last command?), I already mastered the "quoting hell". Thus, I should be able to do it again:
$ PS0='$(date +%s.%N >"/tmp/${USER}.START.${INTERACTIVE_BASHPID}")'
$ PS1='$(awk "BEGIN { printf(\"%fs\", "$(date +%s.%N)" - "$(cat /tmp/$USER.START.$INTERACTIVE_BASHPID)") }" /dev/null)'"$PS1"
0.118550s
$
Ahh. It starts to work. Thus, there is only one issue - to find the right start-up script for the initialization of INTERACTIVE_BASHPID. I found ~/.bashrc which seems to be the right one for this, and which I already used in the past for some other personal customizations.
So, putting it all together - these are the lines I added to my ~/.bashrc:
# command duration puzzle
INTERACTIVE_BASHPID=$BASHPID
date +%s.%N >"/tmp/${USER}.START.${INTERACTIVE_BASHPID}"
PS0='$(date +%s.%N >"/tmp/${USER}.START.${INTERACTIVE_BASHPID}")'
PS1='$(awk "BEGIN { printf(\"%fs\", "$(date +%s.%N)" - "$(cat /tmp/$USER.START.$INTERACTIVE_BASHPID)") }" /dev/null)'"$PS1"
The 3rd line (the date command) has been added to solve another issue. Comment it out and start a new interactive bash to find out why.
A snapshot of my cygwin xterm with bash where I added the above lines to ./~bashrc:
Notes:
I consider this rather as solution to a puzzle than a "serious productive" solution. I'm sure that this kind of time measurement consumes itself a lot of time. The time command might provide a better solution: SE: How to get execution time of a script effectively?. However, this was a nice lecture for practicing the bash...
Don't forget that this code pollutes your /tmp directory with a growing number of small files. Either clean-up the /tmp from time to time or add the appropriate commands for clean-up (e.g. to ~/.bash_logout).
Arithmetic expansion runs in the current process and can assign to variables. It also produces output, which you can consume with something like \e[$((...,0))m (to output \e[0m) or ${t:0:$((...,0))} (to output nothing, which is presumably better). 64-bit integer support in Bash supports will count POSIX nanoseconds until the year 2262.
$ PS0='${t:0:$((t=$(date +%s%N),0))}'
$ PS1='$((( t )) && printf %d.%09ds $((t=$(date +%s%N)-t,t/1000000000)) $((t%1000000000)))${t:0:$((t=0))}\n$ '
0.053282161s
$ sleep 1
1.064178281s
$
$
PS0 is not evaluated for empty commands, which leaves a blank line (I'm not sure if you can conditionally print the \n without breaking things). You can work around that by switching to PROMPT_COMMAND instead (which also saves a fork):
$ PS0='${t:0:$((t=$(date +%s%N),0))}'
$ PROMPT_COMMAND='(( t )) && printf %d.%09ds\\n $((t=$(date +%s%N)-t,t/1000000000)) $((t%1000000000)); t=0'
0.041584565s
$ sleep 1
1.077152833s
$
$
That said, if you do not require sub-second precision, I would suggest using $SECONDS instead (which is also more likely to return a sensible answer if something sets the time).
As correctly stated in the question, PS0 runs inside a sub-shell which makes it unusable for this purpose of setting the start time.
Instead, one can use the history command with epoch seconds %s and the built-in variable $EPOCHSECONDS to calculate when the command finished by leveraging only $PROMPT_COMMAND.
# Save start time before executing command (does not work due to PS0 sub-shell)
# preexec() {
# STARTTIME=$EPOCHSECONDS
# }
# PS0=preexec
# Save end time, without duplicating commands when pressing Enter on an empty line
precmd() {
local st=$(HISTTIMEFORMAT='%s ' history 1 | awk '{print $2}');
if [[ -z "$STARTTIME" || (-n "$STARTTIME" && "$STARTTIME" -ne "$st") ]]; then
ENDTIME=$EPOCHSECONDS
STARTTIME=$st
else
ENDTIME=0
fi
}
__timeit() {
precmd;
if ((ENDTIME - STARTTIME >= 0)); then
printf 'Command took %d seconds.\n' "$((ENDTIME - STARTTIME))";
fi
# Do not forget your:
# - OSC 0 (set title)
# - OSC 777 (notification in gnome-terminal, urxvt; note, this one has preexec and precmd as OSC 777 features)
# - OSC 99 (notification in kitty)
# - OSC 7 (set url) - out of scope for this question
}
export PROMPT_COMMAND=__timeit
Note: If you have ignoredups in your $HISTCONTROL, then this will not report back for a command that is re-run.
Following #SherylHohman use of variables in PS0 I've come with this complete script. I've seen you don't need a PS0Time flag as PS0Calc doesn't exists on empty prompts so _elapsed funct just exit.
#!/bin/bash
# string preceding ms, use color code or ascii
_ELAPTXT=$'\E[1;33m \uf135 '
# extract time
_printtime () {
local _var=${EPOCHREALTIME/,/};
echo ${_var%???}
}
# get diff time, print it and end color codings if any
_elapsed () {
[[ -v "${1}" ]] || ( local _VAR=$(_printtime);
local _ELAPSED=$(( ${_VAR} - ${1} ));
echo "${_ELAPTXT}$(_formatms ${_ELAPSED})"$'\n\e[0m' )
}
# format _elapsed with simple string substitution
_formatms () {
local _n=$((${1})) && case ${_n} in
? | ?? | ???)
echo $_n"ms"
;;
????)
echo ${_n:0:1}${_n:0,-3}"ms"
;;
?????)
echo ${_n:0:2}","${_n:0,-3}"s"
;;
??????)
printf $((${_n:0:3}/60))m+$((${_n:0:3}%60)),${_n:0,-3}"s"
;;
???????)
printf $((${_n:0:4}/60))m$((${_n:0:4}%60))s${_n:0,-3}"ms"
;;
*)
printf "too much!"
;;
esac
}
# prompts
PS0='${PS1:(PS0time=$(_printtime)):0}'
PS1='$(_elapsed $PS0time)${PS0:(PS0time=0):0}\u#\h:\w\$ '
img of result
Save it as _prompt and source it to try:
source _prompt
Change text, ascii codes and colors in _ELAPTXT
_ELAPTXT='\e[33m Elapsed time: '
I have a file where few parameter is specified like load_start_date, load_end_date etc. like below. I also have a variable in that same file called part_date.
load_start_date=2016-10-03
load_end_date=2016-10-03
part_date=
Now my intention is to read the file (specifically read the load_start_date parameter) , deduct 1 day (i.e. for this example it will be 2016-10-02) and then convert the format as YYYYMMDD (e.g. 20161003) save save the value against the variable part_date like below.
load_start_date=2016-10-03
load_end_date=2016-10-03
part_date=20161002
how easily this can be achieved?
I suggest taking advantage from the fact that the strings in your file are well suited for assignment:
#!/bin/bash -
{
source file.txt
if test "$load_start_date" != ""; then
part_date=$(date -d "$load_start_date - 1 day" "+%Y%m%d")
sed -i -e "s/^part_date=.*/part_date=$part_date/" file.txt
fi
}
The source command evaluates file.txt in the current context, i.e. creates variables $load_start_date, $load_end_date, and $part_date. This would have been a little bit unsafe, if we hadn't used subshell expression (the curly braces) which creates special scope for the block of code.
Then we simply assign the date before $load_start_date to $part_date variable, then replace corresponding line in the source file by means of sed.
Use date command in your script:
part_date=$(date -d #$(($(date -d $load_start_date +%s) - 24*60*60)) +%Y%m%d)
This will convert the start date in seconds and substract 1 day from that value, then print again in the YYYYMMDD format.
If I understand your question, and you have a file say file.txt containing:
load_start_date=2016-10-03
load_end_date=2016-10-03
and your goal is to read (or source) that file and set the variable part_date to one day less that that shown in load_start_date (with the - removed), then in addition to the date approach, you can simply use POSIX parameter substitution and arithmetic. For example:
#!/bin/sh
[ -f "dat/lsd.txt" ] && . file.txt ## source file in script
test -n "$load_start_date" || exit 1 ## validate load_start_date
# part_date=$((${load_start_date//-/} - 1)) ## if your shell supports
## parameter substitution
part_date="${load_start_date%%-*}" ## otherwise, us POSIX
tmp="${load_start_date#*-}" ## parameter expansion
part_date="${part_date}${tmp%-*}"
part_date="${part_date}${load_start_date##*-}"
part_date=$((part_date - 1)) ## subtract 1
printf "part_date : %s\n" "$part_date"
(note: if your shell supports parameter substitution, you can reduce the four lines used to isolate the date without the '-' to a single line shown commented above)
Once the '-' are removed, you have a simple integer (e.g. 20161003), and simply arithmetic is all that is needed to reduce the value by 1.
Example Use/Output
part_date : 20161002
There are many ways you can approach this. Look over all the answers and see which one fits your needs the best.
Within my backup script, I'd like to only keep 7 days worth of backups (tried using logrotate for this and it worked perfectly, but I ran into issues with the timing of cron.daily and how it affected the "dateext"). I'm running into problems using parameter expansion to extract the date from the filenames.
Here are some examples of some of the files
foo.bar.tar.gz-20120904
bar.baz.tar.gz-20120904
...
Here is my bash script:
#!/bin/bash
path="/foo/"
today=$(date +%Y%m%d)
keepDays=7
keepSeconds=$(date -d "-$keepDays day" +%s)
for f in $path"*"; do
fileSeconds=$(date -d ${f##*-} +%s)
if [ $fileSeconds -lt $keepSeconds ]
then
rm $f
fi
done
Here is the error I'm getting:
date: extra operand `/foo/foo.bar.tar.gz-20120904'
Remove the quotes around the *, that prevents globbing:
for f in ${path}*; do
(the { } are not strictly required here, but make it easier to read)
Not part of the question, but the Bourne shell syntax [ $fileSeconds -lt $keepSeconds ] could be written as (( $fileSeconds < $keepSeconds )) which is possibly safer.
As cdarke says, remove the quotes around the * in the for loop:
for f in ${path}/*; do
What happens is that the shell executing date gets '/foo/*' and expands that into a list of file names (more than one) and then uses ${f##*-} on part of the list, and date is called with multiple names, and objects.
You'd see this if you ran with bash -x your-script.sh, for instance. When something mysterious goes on, the first step is to make sure you know what the shell is doing. Adding echo "$f" or echo $f in the loop would help you understand — though you'd get two different answers.