i have time logs in timestamp (epoch unix time) format :
1515365117236
1515365123162
1515365139963
i would like to convert it to a regular date like
2017-01-07 23:48:01
2017-01-07 23:48:02
2017-01-07 23:48:03
any ideas what approach would be the fastest?
cat ff1.csv | while read line ; do echo $line\;$(date -d +"%Y-%m-%d %H:%M:%S") ; done > somefile.csv
this takes awful lot of time and just appends the current time
Another approach that must be much faster , using printf of bash version >4.2 :
$ printf '%(datefmt)T\n' epoch
For datefmt you need a string accepted by strftime(3) - see man 3 strftime
Testing:
$ cat file10
1515365117236
1515365123162
1515365139963
$ printf '%(%F %H:%M:%S)T\n' $(cat file10)
49990-01-04 04:47:16
49990-01-04 06:26:02
49990-01-04 11:06:03
In this case , printf format string is:
%F Equivalent to %Y-%m-%d (the ISO 8601 date format). (C99)
%H The hour as a decimal number using a 24-hour clock (range 00 to 23).(Calculated from tm_hour.)
%M The minute as a decimal number (range 00 to 59). (Calculated from tm_min.)
%S The second as a decimal number (range 00 to 60). (The range is up to 60 to allow for occasional leap seconds.- Calculated from tm_sec.)
Update to remove milliseconds:
$ printf '%(%F %T)T\n' $(printf '%s/1000\n' $(<file10) |bc)
2018-01-08 00:45:17
2018-01-08 00:45:23
2018-01-08 00:45:39
The way to transform epoch to date is date -d #epochtime +format
An alternative way is to use date --file switch to read dates from a file directly.
$ cat file10
1515365117236
1515365123162
1515365139963
In order date to understand that these lines are epoch time you need to add # in the beginning of each line.
This can be done like bellow:
$ sed -i 's/^/#/g' file10 #caution - this will make changes in your file
$ date --file file10 +"%Y-%m-%d %H:%M:%S"
Alternativelly, you can do it on the fly without affecting the original file:
$ sed 's/^/#/g' file10 |date --file - +"%Y-%m-%d %H:%M:%S"
PS: in this case --file reads from - == stdin == pipe
In both cases, the result is
49990-01-04 04:47:16
49990-01-04 06:26:02
49990-01-04 11:06:03
PS: by the way, the timestamps you provide seems invalid, since it seems to refer at year 49990
Your input data aren't epoch unix time, it has miliseconds. If you wish to use any method on bash first you must convert to timestamp:
cat ff1.csv | while read LINE; do echo "#$(expr $LINE \/ 1000)" | date +"%Y-%m-%d %H:%M:%S" --file - ; done
First divide by 1000 to delete miliseconds parts, the rest is the same that explain George Vasiliou
Related
I'm migrating old notes from a SQL database based note taking app to separate text files.
I've managed to export the notes and date codes as separate text files.
The files are ordered like this:
$ ls -1
Note0001.txt
Note0001-date.txt
Note0002.txt
Note0002-date.txt
Note0003.txt
Note0003-date.txt
The contents of the date files looks like this:
$ cat Note0001-date.txt
388766121.742373
$ cat Note0002-date.txt
274605766.273638
$ cat Note0003-date.txt
384996285.436197
The dates are seconds since the epoch 2001-01-01. See other question about the format: What type of date format is this? And how to convert it?.
How do I batch change the accessed and modified date of the notes files, NoteNNNN.txt, to the date in the contents of respective date file, NoteNNNN-date.txt?
How to convert the date to UTC+1? Preferably with consideration of DST (daylight saving time).
I am trying to convert the dates with the method described this question:
https://unix.stackexchange.com/questions/2987/
But it outputs an error message in bash 3.2.57 (macOS):
$ date -d '2001-01-01 UTC+1 + 388766121 seconds'
usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ...
[-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format]
I am new to working with the dates and timestamps in the terminal.
Iterate over each file pair, access the timestamp, shift the timestamp so it's something unix tools can understand, then touch files. Ie. big problems are composed of sum of small problems.
# find all files named .txt but not -date.txt
find . -name '*.txt' '!' -name '*-date.txt' |
# remove the .txt suffix
sed 's/\.txt$//' |
{
# the reference point of files content
start=$(date -d "2001-01-01" +%s) # will not work with BSD date
# I guess just precompute the value:
start=978303600
# for each file
while IFS= read -r f; do
# get the timestamp
diff=$(<"$f"-date.txt)
# increment the timestamp to seconds since epoch
ref=$(<<<"scale=6; $start + $diff" bc)
# TODO: use a tool convert the timestamp sinece epoch to BSD touch
# compatible format, ie. to ccyy-mm-ddTHH:MM:SS[.frac][Z]
ref=(TODO "$ref")
# change access and modification times of .txt file
touch -d "#$ref" "$f".txt
done
}
Assuming your OS local timezone is what you want for your output, and you have a version of awk that supports the GNU awk time functions, you could use the
following script. Also:
If the DST daylight-savings flag is positive, the time is assumed to
be daylight savings time; if zero, the time is assumed to be standard
time; and if negative (the default), mktime() attempts to determine
whether daylight savings time is in effect for the specified time.
file tst.awk:
BEGIN {
epoch = mktime("2001 01 01 00 00 00")
}
FNR==1 {
close(out)
out = substr(FILENAME, 1, length(FILENAME)-9) ".txt"
}
{
print strftime("%F %T %Z", epoch+$0) > out
}
Usage:
awk -f tst.awk *-date.txt
Example
Here is an example with the script, without the I/O part, just converting the datetimes.
test file:
> cat file
388766121.742373
274605766.273638
384996285.436197
script tst.awk:
BEGIN { epoch = mktime("2001 01 01 00 00 00") }
{ print strftime("%F %T %Z", epoch+$0) }
Output:
> awk -f tst.awk file
2013-04-27 15:35:21 EEST
2009-09-14 08:22:46 EEST
2013-03-14 23:24:45 EET
The timezone of my box is being used by default (EET). If we 'd like to print to a different timezone, we should define that and set the TZ. Also DST is used by default, notice that some days are printed as EEST (Summer Time).
How would you go about doing a few lines of code in Bash to accomplish the following. I'm trying to build up my skills in Bash and learn how to handle more small tasks directory from the command line.
Steps:
Specify a start date and an end date. Load all the dates in between including the start and end date into a "list"
Loop over the list creating a file like this each time.
(requires date formatting)
2017-11-10.w
2017-11-11.w
2017-11-12.w
You could convert the input dates to Unix timestamps, then add the number of seconds per day and touch a file named after the result until you are past the end date:
#!/bin/bash
startstamp=$(date -d "$1" +'%s')
endstamp=$(date -d "$2" +'%s')
secs_per_day=$(( 24 * 3600 ))
for (( thedate = startstamp; thedate <= endstamp; thedate += secs_per_day )); do
touch "$(date -d "#$thedate" '+%F.w')"
done
The %s formatting string (a GNU extension) prints the number of seconds since the Unix epoch, and # in the argument to the -d option indicates that the date is in that format. %F is short for %Y-%m-%d, which translates to YYYY-MM-DD.
Example usage:
$ ./dates 2017-11-10 2017-11-15
$ ls -1
2017-11-10.w
2017-11-11.w
2017-11-12.w
2017-11-13.w
2017-11-14.w
2017-11-15.w
dates
I have a string from conf file (lets call for example date1):
#!/bin/bash
# it is example
date1="201605250925"
datenow="$(date +%Y%m%d%H%M -d "+1hour")"
date2=$(date +%Y%m%d%H%M -d "$date1 + 1hour")
# NOT WORK?
echo "$date1 --> $date2"
# WORK!
echo "$date1 --> $datenow"
I need to add 1 hour. But getting error like this:
date: invalid date `201605250925 + 1hour'
But its work for datenow.
How can I user addhour for custom date format from string?
You need a format that meets the command date expectations, something like:
2016-05-25 09:25
The space denote the start of time and the time format is HH:MM.
That comes from then international ISO 8601, but using an space instead of a T.
If the format is fixed, we can use bash internal capacities (no external command except date used) to change it like this:
#!/bin/bash
d1="201605250925"
dc="${d1:0:8} ${d1:8:2}:${d1:10:2}+0"
d2=$(date +'%Y%m%d %H:%M' -ud "$dc + 1 hour" )
echo "$d2"
Or POSIXly (dash) with no call needed to sed, awk or cut (faster):
#!/bin/dash
d1="201605250925"
dt=${d1##????????}
dc="${d1%%"$dt"} ${dt%%??}:${dt##??}+0"
d2=$(date -ud "$dc + 1 hour" +'%Y%m%d %H:%M')
echo "$d2"
20160525 10:25
The inclusion of a +0 after the time in dc: 20160525 09:25+0
will ensure that date will interpret the time as with offset 0 (UTC).
The use of the option -u to date will ensure that the value read in UTC also change in UTC, avoiding any Daylight correction or local time change.
If you want to keep the same format as your input string you could use cut or sed to split it:
cut
d1="201605250925"
ds=$(echo $d1 | cut --output-delimiter=" " -c 1-8,9-12); \
d2=$(date '+%Y%m%d%H%M' -ud "$ds + 1hour")
sed
d1="201605250925"
ds=$(echo $d1 | sed 's/\(.*\)\(....\)$/\1 \2/g'); \
d2=$(date '+%Y%m%d%H%M' -ud "$ds + 1hour")
This way the date utility can make sense of the date and time.
Result:
$ echo $d2
201605251025
I would like to convert the current date and time into a hex time stamp, something like:
Tue Feb 2 10:27:46 GMT 2010 converted into 0x6d054a874449e
I would like to do this from a bash script, any idea how I might do that?
Thanks
J
printf '0x%x' $(date +%s)
Without knowing the unit or epoch for your hex timestamp, it's hard to say for sure (and I was slightly confused by your example of "Feb 2" which is not even close to the current date!).
date +%s will convert the current date into a time_t, the number of seconds since the usual Unix epoch (which is midnight on 1st Jan 1970).
printf "0x%x" some_number will convert a value from decimal to hex.
If you need to convert to a different epoch / unit, you will need to do some calculation. You can do arithmetic in bash using $(( expression )):
$ time_t=$(date +%s)
$ echo $(($time_t * 1000))
1284505668000
If you want to convert an arbitrary date (like your "Feb 2 ..." example), rather than the current one, and are happy to assume that you have the GNU version of date, then you can use the -d option along with the +%s output format to do the conversion:
$ date -d 'Tue Feb 2 10:27:46 GMT 2010' +%s
1265106466
An example of putting this all together:
$ time_t=$(date -d 'Tue Feb 2 10:27:46 GMT 2010' +%s)
$ time_t_ms=$(($time_t * 1000))
$ hexstamp=$(printf "0x%x" $time_t_ms)
$ echo $hexstamp
0x1268e38b4d0
Seconds since unix epoch, in hex:
echo "$(date +%s)"|xargs printf "0x%x"
0x59a8de5b
Milliseconds since the epoch:
echo "$(date +%s%N)/1000000"|bc|xargs printf "0x%x"
0x15e3ba702bb
Microseconds:
echo "$(date +%s%N)/1000"|bc|xargs printf "0x%x"
0x55818f6eea775
Nanoseconds:
echo "$(date +%s%N)"|xargs printf "0x%x"
0x14e0219022e3745c
In a bash script, if I have a number that represents a time, in the form hhmmss (or hmmss), what is the best way of subtracting 10 minutes?
ie, 90000 -> 85000
This is a bit tricky. Date can do general manipulations, i.e. you can do:
date --date '-10 min'
Specifying hour-min-seconds (using UTC because otherwise it seems to assume PM):
date --date '11:45:30 UTC -10 min'
To split your date string, the only way I can think of is substring expansion:
a=114530
date --date "${a:0:2}:${a:2:2}:${a:4:2} UTC -10 min"
And if you want to just get back hhmmss:
date +%H%M%S --date "${a:0:2}:${a:2:2}:${a:4:2} UTC -10 min"
why not just use epoch time and then take 600 off of it?
$ echo "`date +%s` - 600"| bc; date
1284050588
Thu Sep 9 11:53:08 CDT 2010
$ date -d '1970-01-01 UTC 1284050588 seconds' +"%Y-%m-%d %T %z"
2010-09-09 11:43:08 -0500
Since you have a 5 or 6 digit number, you have to pad it before doing string manipulation:
$ t=90100
$ while [ ${#t} -lt 6 ]; do t=0$t; done
$ echo $t
090100
$ date +%H%M%S --utc -d"today ${t:0:2}:${t:2:2}:${t:4:2} UTC - 10 minutes"
085100
Note both --utc and UTC are required to make sure the system's timezone doesn't affect the results.
For math within bash (i.e. $(( and ((), leading zeros will cause the number to be interpreted as octal. However, your data is more string-like (with a special format) than number-like, anyway. I've used a while loop above because it sounds like you're treating it as a number and thus might get 100 for 12:01 am.
My version of bash doesn't support -d or --date as used above. However, assuming a correctly 0-padded input, this does work
$ input_time=130503 # meaning "1:05:03 PM"
# next line calculates epoch seconds for today's date at stated time
$ epoch_seconds=$(date -jf '%H%M%S' $input_time '+%s')
# the 600 matches the OP's "subtract 10 minutes" spec. Note: Still relative to "today"
$ calculated_seconds=$(( epoch_seconds - 600 )) # bc would work here but $((...)) is builtin
# +%H%M%S formats the result same as input, but you can do what you like here
$ echo $(date -r $calculated_seconds '+%H%M%S')
# output is 125503: Note that the hour rolled back as expected.
For MacOS users you can do the following:
$(date -v -10M +"%H:%M:%S")
Date time without a specific format:
$(date -v -10M)
For non-macOS users:
Date time without a specific format:
date --date '-10 min'