I have a number of files in the form foo_[SECONDS.MILLISECONDS]_bar.tar.gz and for each file I would like to be to get a datetime value (YYYYMMDDHHMMSS) for each file.
So far I have
ls -1 /filestore/*.tar.gz | cut -d _ -f 2 | date -f -
But this errors along the lines of
date: invalid date '1467535262.712041352'
How should a bash pipeline of epoch values be converted into a datetime string?
MWE
mkdir tmpBLAH
touch tmpBLAH/foo_1467483118.640314986_bar.tar.gz
touch tmpBLAH/foo_1467535262.712041352_bar.tar.gz
ls -1 tmpBLAH/*.tar.gz | cut -d _ -f 2 | date -f -
To convert epoch time to datetimem, please try the following command:
date -d #1346338800 +'%Y%m%d%H%M%S'
1346338800 is a epoch time.
About your case, for comand line as following:
echo 1467535262.712041352 | cut -d '.' -f 1 | xargs -I{} date -d #{} +'%Y%m%d%H%M%S'
you will get:
20160703174102
Something like this?
for f in /filestore/*.tar.gz; do
epoch=${f#*_}
date -d #${epoch%%.*} +%Y%m%d%H%M%S
done
The syntax of the date command differs between platforms; I have assumed GNU date, as commonly found on Linux. (You could probably use date -f if you add the # before each timestamp, but I am not in a place where I can test this right now.) Running a loop makes some things easier, such as printing both the input file name and the converted date, while otherwise a pipeline would be the most efficient and idiomatic solution.
As an aside, basically never use ls in scripts.
First, the -1 option to ls is useless, because ls prints its output one file per line by default, it's just that when the output is a terminal (not a pipe), it pretty-prints in columns. You can check that fact by just running ls | cat.
Then, date converts epoch timestamps safely only if prefixed with an #.
% date -d 0
Sun Jul 3 00:00:00 CEST 2016
% LANG=C date -d #0
Thu Jan 1 01:00:00 CET 1970
% date -d 12345
date: invalid date '12345'
% date -d #12345
Thu Jan 1 04:25:45 CET 1970
Which gives:
printf "%s\n" tmpBLAH/foo_*_bar.tar.gz | sed 's/.*foo_/#/; s/_bar.*//' | date -f -
You can do:
for i in foo_*_bar.tar.gz; do date -d "#$(cut -d_ -f2 <<<"$i")" '+%Y%m%d%H%M%S'; done
The epoch time is provided with the -d #<time> and the desired format is '+%Y%m%d%H%M%S'.
Example:
% for i in foo_*_bar.tar.gz; do date -d "#$(cut -d_ -f2 <<<"$i")" '+%Y%m%d%H%M%S'; done
20160703001158
20160703144102
Related
i have time logs in timestamp (epoch unix time) format :
1515365117236
1515365123162
1515365139963
i would like to convert it to a regular date like
2017-01-07 23:48:01
2017-01-07 23:48:02
2017-01-07 23:48:03
any ideas what approach would be the fastest?
cat ff1.csv | while read line ; do echo $line\;$(date -d +"%Y-%m-%d %H:%M:%S") ; done > somefile.csv
this takes awful lot of time and just appends the current time
Another approach that must be much faster , using printf of bash version >4.2 :
$ printf '%(datefmt)T\n' epoch
For datefmt you need a string accepted by strftime(3) - see man 3 strftime
Testing:
$ cat file10
1515365117236
1515365123162
1515365139963
$ printf '%(%F %H:%M:%S)T\n' $(cat file10)
49990-01-04 04:47:16
49990-01-04 06:26:02
49990-01-04 11:06:03
In this case , printf format string is:
%F Equivalent to %Y-%m-%d (the ISO 8601 date format). (C99)
%H The hour as a decimal number using a 24-hour clock (range 00 to 23).(Calculated from tm_hour.)
%M The minute as a decimal number (range 00 to 59). (Calculated from tm_min.)
%S The second as a decimal number (range 00 to 60). (The range is up to 60 to allow for occasional leap seconds.- Calculated from tm_sec.)
Update to remove milliseconds:
$ printf '%(%F %T)T\n' $(printf '%s/1000\n' $(<file10) |bc)
2018-01-08 00:45:17
2018-01-08 00:45:23
2018-01-08 00:45:39
The way to transform epoch to date is date -d #epochtime +format
An alternative way is to use date --file switch to read dates from a file directly.
$ cat file10
1515365117236
1515365123162
1515365139963
In order date to understand that these lines are epoch time you need to add # in the beginning of each line.
This can be done like bellow:
$ sed -i 's/^/#/g' file10 #caution - this will make changes in your file
$ date --file file10 +"%Y-%m-%d %H:%M:%S"
Alternativelly, you can do it on the fly without affecting the original file:
$ sed 's/^/#/g' file10 |date --file - +"%Y-%m-%d %H:%M:%S"
PS: in this case --file reads from - == stdin == pipe
In both cases, the result is
49990-01-04 04:47:16
49990-01-04 06:26:02
49990-01-04 11:06:03
PS: by the way, the timestamps you provide seems invalid, since it seems to refer at year 49990
Your input data aren't epoch unix time, it has miliseconds. If you wish to use any method on bash first you must convert to timestamp:
cat ff1.csv | while read LINE; do echo "#$(expr $LINE \/ 1000)" | date +"%Y-%m-%d %H:%M:%S" --file - ; done
First divide by 1000 to delete miliseconds parts, the rest is the same that explain George Vasiliou
I am having the below outputs and I need to get the time difference in seconds.
------------------------------
Wed Nov 23 15:09:20 2016
------------------------------
Wed Nov 23 15:27:47 2016
------------------------------
Generally month should be the same on all cases so we can escape it, the same for the year, I may get different values for the day of week and the day for sure, the difference for sure will be in seconds and minutes and might be in hours ...
I tried some awks and cut by : but I still having an issue.
Thanks in advance !
Any help appreciated !
My first perl script ever :
# extract two dates and calculate difference in s
# http://stackoverflow.com/questions/40781429/get-the-time-difference-in-seconds/
#
# cat time_diff.txt | grep -e "20[0-2][0-9]" | perl time_difference.pl
use Date::Parse;
$date_str1 = <STDIN>;
$date_str2 = <STDIN>;
$date1 = str2time($date_str1);
$date2 = str2time($date_str2);
print $date2-$date1;
print "\n";
Too bad you cannot use date -d, I was proud of this one-liner :
cat time_diff.txt | grep -e "20[0-2][0-9]" | xargs -i date -d{} +%s | (read -d "\n" t1 t2; echo $t2-$t1 | bc)
Tested with bash and zsh on Linux Mint 17.3
I have a string from conf file (lets call for example date1):
#!/bin/bash
# it is example
date1="201605250925"
datenow="$(date +%Y%m%d%H%M -d "+1hour")"
date2=$(date +%Y%m%d%H%M -d "$date1 + 1hour")
# NOT WORK?
echo "$date1 --> $date2"
# WORK!
echo "$date1 --> $datenow"
I need to add 1 hour. But getting error like this:
date: invalid date `201605250925 + 1hour'
But its work for datenow.
How can I user addhour for custom date format from string?
You need a format that meets the command date expectations, something like:
2016-05-25 09:25
The space denote the start of time and the time format is HH:MM.
That comes from then international ISO 8601, but using an space instead of a T.
If the format is fixed, we can use bash internal capacities (no external command except date used) to change it like this:
#!/bin/bash
d1="201605250925"
dc="${d1:0:8} ${d1:8:2}:${d1:10:2}+0"
d2=$(date +'%Y%m%d %H:%M' -ud "$dc + 1 hour" )
echo "$d2"
Or POSIXly (dash) with no call needed to sed, awk or cut (faster):
#!/bin/dash
d1="201605250925"
dt=${d1##????????}
dc="${d1%%"$dt"} ${dt%%??}:${dt##??}+0"
d2=$(date -ud "$dc + 1 hour" +'%Y%m%d %H:%M')
echo "$d2"
20160525 10:25
The inclusion of a +0 after the time in dc: 20160525 09:25+0
will ensure that date will interpret the time as with offset 0 (UTC).
The use of the option -u to date will ensure that the value read in UTC also change in UTC, avoiding any Daylight correction or local time change.
If you want to keep the same format as your input string you could use cut or sed to split it:
cut
d1="201605250925"
ds=$(echo $d1 | cut --output-delimiter=" " -c 1-8,9-12); \
d2=$(date '+%Y%m%d%H%M' -ud "$ds + 1hour")
sed
d1="201605250925"
ds=$(echo $d1 | sed 's/\(.*\)\(....\)$/\1 \2/g'); \
d2=$(date '+%Y%m%d%H%M' -ud "$ds + 1hour")
This way the date utility can make sense of the date and time.
Result:
$ echo $d2
201605251025
I'm sure this answer is obvious but I'm banging my head on it and getting a headache and my Search Foo is failing me…
I have a log file with this date format:
Sep 1 16:55:00 stuff happening
Sep 1 16:55:01 THIS IS THE LINE YOU WANT at this time stamp
Sep 1 16:55:02 more stuff
Sep 1 16:55:02 THIS IS THE LINE YOU WANT at this time stamp
Sep 1 16:55:03 blah
Sep 1 16:55:04 blah and so on…..
My ultimate goal is to:
Find the last line in the log file with a given string eg: "THIS IS THE LINE…" this is my "magic time" that I will do calculations on later.
Take the date of that line and set a variable that is the date +NN seconds. The time in the future will usually just short of 24hrs in the future from the time in step 1 so crossing into the next day may happen if that is important.
At some point in the script, advance the system clock to the new date/time after which I will be checking for certain events to fire.
I know this is way wrong but so far I have figured out how to:
Grab the last date stamp for my event.
logDate=cat /logdir/my.log | grep "THIS IS THE LINE" | tail -1 | cut -f1,2,3 -d" "
Returns: Sept 1 16:55:02
Convert the date into a more usable format
logDate2="$(date -d "$logDate" +"%m-%d %H:%M:%S")"; echo $logDate2
Returns: 09-17 16:55:02
I'm stuck here - what I want is:
futuredate=$logdate2 + XXXSeconds
Could someone help me with the time calculation or perhaps point out a better way to do all of this?
Thanks.
I'm stuck here - what I want is:
futuredate=$logdate2 + XXXSeconds
You can do it by converting through timestamps:
# convert log date to timestamp
logts="$(date -d "$logDate" '+%s')"
# add timestamp with seconds
futurets=$(( logts + XXXSeconds ))
# get date based from timestamp, optionally you can add a format.
futuredate=$(date -d "#${futurets}")
# Get time in seconds from the epoc (1970-01-01 00:00:00 UTC)
dateinseconds=$(date +"%s" -d "$(tail -1 logfile | grep "THIS IS THE LINE" | awk '{print $1, $2, $3}')")
# You can also use just awk without grep and tail to match and print the last line
dateinseconds=$(date +"%s" -d "$(awk '{/THIS IS THE LINE/}END{print $1, $2, $3}' logfile)")
gotofuture=$(( $dateinseconds + 2345 )) # Add 2345 seconds
newdate=$(date -d "#${gotofuture}")
echo "$newdate"
How would you parse a date in bash, with separate fields (years, months, days, hours, minutes, seconds) into different variables?
The date format is: YYYY-MM-DD hh:mm:ss
Does it have to be bash? You can use the GNU coreutils /bin/date binary for many transformations:
$ date --date="2009-01-02 03:04:05" "+%d %B of %Y at %H:%M and %S seconds"
02 January of 2009 at 03:04 and 05 seconds
This parses the given date and displays it in the chosen format. You can adapt that at will to your needs.
I had a different input time format, so here is a more flexible solution.
Convert dates in BSD/macOS
date -jf in_format [+out_format] in_date
where the formats use strftime (see man strftime).
For the given input format YYYY-MM-DD hh:mm:ss:
$ date -jf '%Y-%m-%d %H:%M:%S' '2017-05-10 13:40:01'
Wed May 10 13:40:01 PDT 2017
To read them into separate variables, I'm taking NVRAM's idea, but allowing you to use any strftime format:
$ date_in='2017-05-10 13:40:01'
$ format='%Y-%m-%d %H:%M:%S'
$ read -r y m d H M S <<< "$(date -jf "$format" '+%Y %m %d %H %M %S' "$date_in")"
$ for var in y m d H M S; do echo "$var=${!var}"; done
y=2017
m=05
d=10
H=13
M=40
S=01
In scripts, always use read -r.
In my case, I wanted to convert between timezones (see your /usr/share/zoneinfo directory for zone names):
$ format=%Y-%m-%dT%H:%M:%S%z
$ TZ=UTC date -jf $format +$format 2017-05-10T02:40:01+0200
2017-05-10T00:40:01+0000
$ TZ=America/Los_Angeles date -jf $format +$format 2017-05-10T02:40:01+0200
2017-05-09T17:40:01-0700
Convert dates in GNU/Linux
On a Mac, you can install the GNU version of date as gdate with brew install coreutils.
date [+out_format] -d in_date
where the out_format uses strftime (see man strftime).
In GNU coreutils' date command, there is no way to explicitly set an input format, since it tries to figure out the input format by itself, and stuff usually just works. (For detail, you can read the manual at coreutils: Date input formats.)
For example:
$ date '+%Y %m %d %H %M %S' -d '2017-05-10 13:40:01'
2017 05 10 13 40 01
To read them into separate variables:
$ read -r y m d H M S <<< "$(date '+%Y %m %d %H %M %S' -d "$date_in")"
To convert between timezones (see your /usr/share/zoneinfo directory for zone names), you can specify TZ="America/Los_Angeles" right in your input string. Note the literal " chars around the zone name, and the space character before in_date:
TZ=out_tz date [+out_format] 'TZ="in_tz" in_date'
For example:
$ format='%Y-%m-%d %H:%M:%S%z'
$ TZ=America/Los_Angeles date +"$format" -d 'TZ="UTC" 2017-05-10 02:40:01'
2017-05-09 19:40:01-0700
$ TZ=UTC date +"$format" -d 'TZ="America/Los_Angeles" 2017-05-09 19:40:01'
2017-05-10 02:40:01+0000
GNU date also understands hour offsets for the time zone:
$ TZ=UTC date +"$format" -d '2017-05-09 19:40:01-0700'
2017-05-10 02:40:01+0000
This is simple, just convert your dashes and colons to a space (no need to change IFS) and use 'read' all on one line:
read Y M D h m s <<< ${date//[-:]/ }
For example:
$ date=$(date +'%Y-%m-%d %H:%M:%S')
$ read Y M D h m s <<< ${date//[-: ]/ }
$ echo "Y=$Y, m=$m"
Y=2009, m=57
$ t='2009-12-03 12:38:15'
$ a=(`echo $t | sed -e 's/[:-]/ /g'`)
$ echo ${a[*]}
2009 12 03 12 38 15
$ echo ${a[3]}
12
The array method is perhaps better, but this is what you were specifically asking for:
IFS=" :-"
read year month day hour minute second < <(echo "YYYY-MM-DD hh:mm:ss")
Pure Bash:
date="2009-12-03 15:35:11"
saveIFS="$IFS"
IFS="- :"
date=($date)
IFS="$saveIFS"
for field in "${date[#]}"
do
echo $field
done
2009
12
03
15
35
11
instead of using the shell scripting,incorporate in your scripting itself like below wheever you need:
a=date +%Y
b=date +%S
c=date +%H
a will be year
b will be seconds
c will be hours. and so on.
Another solution to the OP's problem:
IFS=' -:' read y m d h m s<<<'2014-03-26 16:36:41'
Converting a date to another format with BSD date and GNU date:
$ LC_ALL=C date -jf '%a %b %e %H:%M:%S %Z %Y' 'Wed Mar 26 16:36:41 EET 2014' +%F\ %T
2014-03-26 16:36:41
$ gdate -d 'Wed Mar 26 16:36:41 EET 2014' +%F\ %T
2014-03-26 16:36:41
GNU date recognizes Wed and Mar even in non-English locales but BSD date doesn't.
Converting seconds since epoch to a date and time with GNU date and BSD date:
$ gdate -d #1234567890 '+%F %T'
2009-02-14 01:31:30
$ date -r 1234567890 '+%F %T'
2009-02-14 01:31:30
Converting seconds to hours, minutes, and seconds with a POSIX shell, POSIX awk, GNU date, and BSD date:
$ s=12345;printf '%02d:%02d:%02d\n' $((s/3600)) $((s%3600/60)) $((s%60))
05:25:45
$ echo 12345|awk '{printf "%02d:%02d:%02d\n",$0/3600,$0%3600/60,$0%60}'
05:25:45
$ gdate -d #12345 +%T
05:25:45
$ date -r 12345 +%T
05:25:45
Converting seconds to days, hours, minutes, and seconds:
$ t=12345678
$ printf '%d:%02d:%02d:%02d\n' $((t/86400)) $((t/3600%24)) $((t/60%60)) $((t%60))
142:21:21:18
another pure bash
$ d="2009-12-03 15:35:11"
$ d=${d//[- :]/|}
$ IFS="|"
$ set -- $d
$ echo $1
2009
$ echo $2
12
$ echo $#
2009 12 03 15 35 11
have you tried using cut?
something like this:
dayofweek=date|cut -d" " -f1