I'm trying to convert EST datetime to UTC in a Hive query, but can't see daylight saving taken into account. Do you know how to account for daylight saving in Hive?
For example:
TO_UTC_TIMESTAMP('2014-12-31 00:00:00', 'EST') gives 2014-12-31 05:00:00 i.e. 5 hour difference
TO_UTC_TIMESTAMP('2014-06-30 00:00:00', 'EST') gives 2014-06-30 05:00:00, also 5 hour difference
I'm expecting the June query to give a 4 hour difference.
In June the East Coast observes EDT (Eastern Daylight Savings Time), but Hive doesn't understand EDT at all:
TO_UTC_TIMESTAMP('2014-12-31 00:00:00', 'EDT') gives 2014-12-31 00:00:00 i.e. no difference
Any ideas?
Thanks,
Ilmari
(Running Hadoop 1.0.3 on AWS Elastic MapReduce)
Here is an open ticket from the Hive project that address this issue.
https://issues.apache.org/jira/browse/HIVE-12194
See 2nd comment:
Ben Breakstone added a comment - 16/Oct/15 16:54
It's worth noting the daylight saving time version of US three-letter codes like "PDT" are not included in /lib/zi/ for the Oracle JDK. New identifiers like "PST8PDT" appear to work as expected.
See http://www.oracle.com/technetwork/articles/javase/alertfurtherinfo-139131.html
Perhaps as Ben Breakstone suggests new identifiers will work?
Related
I'm using ConvertTimeZone function in power automate to convert from UTC to a certain time zone. The only conversion that has passed is from UTC to Eastern Standard Time:
convertTimeZone(triggerOutputs()?['body/Maintenance_x0020_Window'],'UTC','Eastern Standard Time','dd-MMM-yyyy hh:mm tt')
Next I need to convert to the time zone in London - Greenwich Mean Time/GMT and to Lyon, Rhône, France time zone Central European Summer Time/CEST but i get this error:
the value provided for the time zone id 'Central European Summer Time' was not valid.
Any idea what I'm doing wrong?
It would seem that the convertTimeZone function expects a Windows time zone identifier. For your requested zones, the IDs are:
"Eastern Standard Time" - for the Eastern time zone of US & Canada inclusive of both EST and EDT
"GMT Standard Time" - for Great Britain, inclusive of both GMT and BST
"Romance Standard Time" - for France and several other European countries, inclusive of both CET and CEST
Yes - the names are confusing. You can read more about this and see ways to get the entire list of zone names by reading the subsection titled "The Microsoft Windows Time Zone Database" within the timezone tag wiki.
How can i filter for certain time ranges which are not using a fixed date ?
e.g.
22:00 yesterday - 06:00 today
06:00 today - 14:00 today
14:00 today - 22:00 today
is there something like a universal date-format which is not fixing the date?
I dont think your question is very clear but are you looking for relative time queries?
Like now-1d etc, if so check Elastic date math here.
select to_utc_timestamp(current_timestamp(),'IST');
is converting to utc from ist.
but,
select to_utc_timestamp(current_timestamp(),'PHT');
PHT time is not converting, it simply returning current timestamp in utc.
select current_timestamp(),to_utc_timestamp(current_timestamp(),'PHT');
| 2019-07-10 07:52:29.795 | 2019-07-10 07:52:29.795 |
returning utc only.
You need to specify the continent/capital of country:
select to_utc_timestamp(current_timestamp(),'Asia/Manila');
'PHT' timezine is not recognized because java.time.ZoneId does not contain such zone, it contains IST though. Use GMT offset notation like this:
hive> select current_timestamp(),to_utc_timestamp(current_timestamp(),'GMT+8:00');
OK
2019-07-10 23:21:24.743 2019-07-10 15:21:24.743
Time taken: 0.096 seconds, Fetched: 1 row(s)
I am not completely sure is it GMT+8:00 in Phillipines or should it be GMT-8:00, you should know better, to_utc_timestamp() will work correctly if you specify correct GMT offset. And there is no daylight saving time in Manilla hence this method should work fine.
Asia/Manila also works fine and as #F.Lazarescu mentioned this is better to use timezone identifier instead of GMT offset because daylight saving will be counted if applicable:
select current_timestamp(),to_utc_timestamp(current_timestamp(),'Asia/Manila');
OK
2019-07-11 23:09:47.257 2019-07-11 15:09:47.257
Time taken: 4.029 seconds, Fetched: 1 row(s)
Have a look at this rather useful article also: Watch out for timezones with Sqoop, Hive, Impala and Spark
Don't use time zone abbreviations as identifiers. IST could be India Standard Time, Israel Standard Time or Ireland Standard Time. (There are lots of other ambiguities.)
If by IST you meant India, then use 'Asia/Kolkata'
If by IST you meant Israel, then use 'Asia/Jerusalem'
If by IST you meant Irelant, then use 'Europe/Dublin'
Instead of PHT, use 'Asia/Manila' for the Philippines
See the list of tz database time zones on Wikipedia for the complete list.
I need to get time difference in minutes for my analysis in Hive query.
I am using unix_timestamp() to convert dates to seconds and then subtracting to get the diff in seconds and the multiplied by 60 for minutes.
My issue is my recent date - older date difference is coming negative.
here is my query and results
Hive query and result screenshot
processed_ts create_ts processed_unix_timestamp create_unix_timestamp miniue Diff
2017-03-12 3:01:06 2017-03-12 2:58:36 1489312865 1489316315 -57.5
2017-03-12 3:01:36 2017-03-12 2:59:06 1489312895 1489316345 -57.5
2017-03-12 3:02:12 2017-03-12 2:59:42 1489312932 1489316382 -57.5
Any help is much appreciated.
USA & Canada Start DST on March 12
Published 17-Feb-2017
Most of the United States, Canada, and Mexico's northern border cities
will begin Daylight Saving Time (DST) on Sunday, March 12, 2017.
People in areas that observe DST will spring forward 1 hour from 02:00
(2 am) to 03:00 (3 am), local time.
Standard time will resume on
Sunday, November 5, 2017.
https://www.timeanddate.com/news/time/usa-canada-start-dst-2017.html
select timestamp '2017-03-12 02:58:36'
2017-03-12 03:58:36
I'm importing SQL data into Spotfire Analyst. All of the date and time fields are in the form of a Unix timestamp. What's the best way to convert this into an actual date format that I can manipulate in Spotfire?
Utilizing a calculated column you can calculate the datetime based on the UNIX epoch.
We simply add our seconds to the DateTime of the UNIX epoch (JAN 01 1970 00:00:00 UTC) to get the result. Below is an example of the UNIX time when I started writing this post.
DateAdd("second",1429733486,DateTime(1970,1,1,0,0,0,0))
The below is what should work for you:
DateAdd("second",[UNIX_TIMESTAMP_COLUMN],DateTime(1970,1,1,0,0,0,0))
Keep in mind these dates produced will be in the UTC timezone as per the JAN 1 1970 epoch. If you need them in your local time zone you may have to adjust accordingly with further DateAdd functions adding/subtracting time as per current conversions. Also, if you observe daylight savings time you may need to add some extra case logic to handle that as well.
Please let me know if you have any questions or need further clarification.
In 7.0 and later you can use
FromEpohTimeSeconds([UNIXDATE])
or
FromEpohTimeMilliseconds([UNIXDATE])