I'm reverse engineering a query from a kibana board and it has timestamp values in it like '1408884022624'.
In reading over the elastic search date mapping docs I don't see anything in there regarding what (appears to be) some sort of millisecond or tick format. Could someone tell me what the number above represents in my query? (I'm pretty sure we're not using a custom date format.)
It's the number of milliseconds since the beginning of Unix Epoch time, 00:00:00 UTC January 1 1970. Sometimes referred to as Java Epoch time. Technically it's not Unix Epoch time as that's tracked as the number of seconds since the above date, but many tools/converters handle both seconds and milliseconds.
Care should be taken, though, as it's quite easy to accidentally get the time in one format (let's say seconds) and pass it to a function or method expecting it in the other.
http://en.wikipedia.org/wiki/Unix_time
http://www.javaworld.com/article/2074293/core-java/groovy--java--and-the-unix-epoch-time.html
Related
Time Formats
A point in time is often represented as Unix Time, or as a human-readable ISO 8601 date in UTC time string.
For example:
Unix Time
Seconds since Epoch, or Unix timestamp, in seconds or milliseconds:
1529325705
1529325705000
ISO 8601 Date
2018-06-18T15:41:45+00:00
My question
Is there a one-to-one and onto relationship between the two? In other words, is there a point in time with a single representation in one format, and more than one, or zero, representations in the other?
Yes, it is possible to find such a date. From the wiki article on Unix time:
Every day is treated as if it contains exactly 86400 seconds,[2] so leap seconds are not applied to seconds since the Epoch.
That means that the leap seconds themselves cannot be represented in Unix time.
For example, the latest leap second occurred at the end of 2016, so 2016-12-31T23:59:60+00:00 is a valid ISO 8601 time stamp. However, the Unix time stamp for the second before, at 23:59:59, is represented as 1483228799 and the second after, 00:00:00 (on January 1 2017) is 1483228800, so there is no Unix timestamp that represents the leap second.
In practice, this is probably not a problem for you; there has only been 27 leap seconds since they were introduced in 1972.
It might be worthwhile to mention that most software implementations of ISO 8601 does not take leap seconds into account either, but will do something else if asked to parse "2016-12-31T23:59:60+00:00". The System.DateTime class in .NET throws an exception, while it's also conceivable that a library would return 2017-01-01 00:00:00.
No. there is a nice correspondence between the two, but the relationship is 1 to many, and strictly speaking there may not even exist a precise Unix millisecond for a given ISO date-time string. Some issues are:
There are some freedoms in the ISO 8601 format, so the same Unix millisecond may be written in several ways even when we require that the time be in UTC (the offset is zero).
Seconds and fraction of seconds are optional, and there may be a varying number of decimals on the seconds. So a milliseconds value of 1 529 381 160 000, for example, could be written as for example
2018-06-19T04:06:00.000000000Z
2018-06-19T04:06:00.00Z
2018-06-19T04:06:00Z
2018-06-19T04:06Z
The offset of 0 would normally be written as Z, but may also be written as you do in the question, +00:00. I think the forms +00 and +0000 are OK too (forms with a minus are not).
Since there may be more than three decimals on the seconds in ISO 8601, no exact Unix millisecond may match. So you will have to accept truncation (or rounding) to convert to Unix time. Of course the error will be still greater if you convert to Unix seconds rather than milliseconds.
As Thomas Lycken noted, leap seconds can be represented in ISO 8601, but not in Unix time.
In other words, is there a point in time with a single representation in one format, and more than one, or zero, representations in the other?
No. The UTC time depends on your geographic location, ie. your latitude and longitude. However, the UNIX timestamp is a way to track time as a running total of seconds. This count starts at the Unix Epoch on January 1st, 1970 at UTC.
From Unix TimeStamp,
It should also be pointed out that this point in time technically does not change no
matter where you are located on the globe.
So I'm writing some code trying to convert some oracle dates I got into a different time zone. The issue I'm having is that when the time portion is 00:00:00 I don't now how to determine if it is legitimately midnight or if the date was meant to not include a time.
Currently, I'm making the assumption that if the time is 00:00:00 then the value is just a time-free date, because unfortunately that is sometimes the case, but while statistically small, there is a chance that the date is legitimately midnight so I'm trying to find a better approach with no success.
I can't assume all 00:00:00 are midnight, because if the data was intended to only have a date then converting to most other US time zones would change the date.
Any suggestions?
Check for other values in the same column to see what the distribution of 00:00:00's is. If all entries are 00:00:00 then the column is definitely dates, if it's roughly 1 in (24*60*60) occurrences then it's definitely times, if it's somewhere in between then you've got a problem.
You could also look for a check constraint to see if times are constrained to be midnight.
You could look at the semantics of the column also -- what does the name tell you?
I've a epoch time it's seconds or milliseconds format. How do I get time zone and day light saving details from that epoch time? Or do I have to pass as a separate value?
An epoch time in second or millisecond resolution is just a number. And a number cannot hold more than ONE information, here the elapsed time since an epoch.
Time zone or daylight saving details must therefore be transmitted as extra information. Keep also in mind that a time zone is often pretty complex (has an ID, a name, a history of offset transitions etc.) so most time zones cannot just be expressed as simple numbers.
By the way, I cannot give more concrete answer how to transmit extra details like time zones since you have not even told us which programming language or tool you use.
I am planning on starting a project that will need to record timestamps of incoming transactions. I appreciate that Unix Time is an integer value and I can use this type of functionality to my advantage. However, Unix Time only measures in seconds. As a minimal requirement I need to record transaction times at the millisecond level.
I know that there are ways that I could get around this issue, but I was wondering if there was another standardized way of representing time data that also represented milliseconds (or, some factor of sub-milliseconds) in the time value that is fully expressed as an integer value since epoch.
Does such a time format exist? FYI, so long as the date data-type is standardized, I don't care what system this is native in. I can code my own implementation, however, I would like to use an existing date/time format, rather than create my own.
One place where such a standard is used is ECMAScript / Javascript. Javascript date objects use milliseconds since January 1, 1970, midnight UTC for their numerical integer representation. This is detailed here.
You can test this using your browser's console:
var d = new Date();
console.log(d.getTime()); // yields integer milliseconds since epoch
So yes, there is prior art for such a use.
date +%s
outputs timestamp in seconds
date +%s%N
returns timestamp in nanoseconds
To get milliseconds divide the nanoseconds by 1 000 000
UNIX time is not appropriate for time stamping transactions because it does some weird stuff, inserting leap seconds on occasion, thus making it so that you won't be able to add and subtract time stamps reliably, nor sort transactions by timestamp.
A more appropriate standard for timestamps is TAI https://www.nist.gov/pml/time-and-frequency-division/nist-time-frequently-asked-questions-faq#tai . TAI is stored in the same way as UNIX time as a number of seconds and or microseconds and or nanoseconds since the UNIX epoch, however, it is the true number, no leap seconds are added or removed. This means that you can actually add and subtract TAI timestamps to get elapsed time and TAI timestamps are always sortable. Unfortunately, support for TAI timestamps is somewhat limited. For example, linux added support for TAI timestamps only recently in version 3.10 python added this support only in version 3.9/time.html?highlight=time#module-time
Is there any sort of advantage (performance, indexes, size, etc) to storing dates in MongoDB as an ISODate() vs. storing as a regular UNIX timestamp?
The amount of overhead of a ISODate compared to a time_t is trivial compared to the advantages of the former.
An ISO 8601 format date is human readable, it can be used to express dates prior to January 1, 1970, and most importantly, it isn't prey to the Y2038 problem.
This last bit can't be stressed enough. In 1960, it seemed ludicrous that wasting an octet or two on a century number could yield any benefit as the turn of the century was impossibly far off. We know how wrong that turned out to be. The year 2038 will be here sooner than you expect, and time_t are already insufficient for representing – for example – the schedule of payments on a 30-year contract.
MongoDB's built-in Date type is very similar to a unix timestamp stored in time_t. The only difference is that Dates are a 64bit field storing miliseconds since Jan 1 1970, rather than a 32bit fields storing seconds since the same epoch. The only down side is that for current releases it treats the count as unsigned so it can't handle dates before 1970 correctly. This will be fixed in MongoDB 2.0 scheduled for release in about a month.
A possible point of confusion is the name "ISODate". It is just a helper function in the shell to wrap around javascript's horrible Date constructor. If you call either "ISODate()" or "new Date()" you will get back the exact same Date object, we just changed how it prints. You are still free to use normal ISO Date stings or time_t ints without using our constructors, but you won't get nice Date objects back in your language of choice.