I need to get time difference in minutes for my analysis in Hive query.
I am using unix_timestamp() to convert dates to seconds and then subtracting to get the diff in seconds and the multiplied by 60 for minutes.
My issue is my recent date - older date difference is coming negative.
here is my query and results
Hive query and result screenshot
processed_ts create_ts processed_unix_timestamp create_unix_timestamp miniue Diff
2017-03-12 3:01:06 2017-03-12 2:58:36 1489312865 1489316315 -57.5
2017-03-12 3:01:36 2017-03-12 2:59:06 1489312895 1489316345 -57.5
2017-03-12 3:02:12 2017-03-12 2:59:42 1489312932 1489316382 -57.5
Any help is much appreciated.
USA & Canada Start DST on March 12
Published 17-Feb-2017
Most of the United States, Canada, and Mexico's northern border cities
will begin Daylight Saving Time (DST) on Sunday, March 12, 2017.
People in areas that observe DST will spring forward 1 hour from 02:00
(2 am) to 03:00 (3 am), local time.
Standard time will resume on
Sunday, November 5, 2017.
https://www.timeanddate.com/news/time/usa-canada-start-dst-2017.html
select timestamp '2017-03-12 02:58:36'
2017-03-12 03:58:36
Related
In the Oracle Apex 20 Gantt chart, I would like to display the tasks broken down by hours, working hours from Monday to Friday, from 6:00 to 18:00. I would like to hide Saturday and Sunday, as well as 0:00 to 06:00 and 18:00 to 24:00.
Is this possible?
I want to compare data from the current period (e.g current month) with the previous period (e.g. last month).
Let's say I have the following dataset:
date
orders
2020-01-01
1
2020-01-02
2
2020-01-03
5
2020-02-01
4
2020-02-02
2
And I want to get:
Jan 2020 Orders
Feb 2020 Orders
Delta Orders
8
6
-25%
Period can be selected by user.
the way I reached your desired outcome is by creating a calculated filed "MoM Change":
percentDifference(sum({ orders}), [date ASC], -1, [])
Also, this tutorial might be helpful: https://learnquicksight.workshop.aws/en/business-level-up.html
I was hoping to upvote the example above as it helped me solve this problem. I'm not yet to a reputation of 15 (how embarrassing) so I can't yet do that. But thank you #chrigu for the help.
In researching this, I found that QuickSight has extended this with a few similar functions. I used this one, which works with less parameters:
periodOverPeriodPercentDifference(measure, date, period, offset))
https://docs.aws.amazon.com/quicksight/latest/user/periodOverPeriodPercentDifference-function.html
There are several other variants for periodTo___ and periodOver___ to check out as well.
How can i filter for certain time ranges which are not using a fixed date ?
e.g.
22:00 yesterday - 06:00 today
06:00 today - 14:00 today
14:00 today - 22:00 today
is there something like a universal date-format which is not fixing the date?
I dont think your question is very clear but are you looking for relative time queries?
Like now-1d etc, if so check Elastic date math here.
I have a list of dates which indicates when the job finished running successfully: The list can be millions of dates
YYYY-MM-DD HH:MM:SS
2016-01-01 05:00:00
2016-01-02 05:00:00
2016-01-05 13:00:00
2016-01-06 13:00:00
2016-01-09 05:00:00
2016-01-10 05:00:00
Occasionally, the job might have failed and delay the process by few hour to few days:
2016-01-13 14:00:00
2016-01-15 14:00:00
2016-01-19 06:00:00
2016-01-20 06:00:00
The pattern obviously is 1 and 3 day for this list.
My question is, how can I figure out the pattern of any arbitrary list of dates, able to ignore the delay, and estimate the next date that the job will be finish running?
I need to estimate and produce a date that accurately-ish predicted when will the job finish next time, using the date that occur the most, and ignore delay if possible.
Any help will be appreciated!
I'm trying to convert EST datetime to UTC in a Hive query, but can't see daylight saving taken into account. Do you know how to account for daylight saving in Hive?
For example:
TO_UTC_TIMESTAMP('2014-12-31 00:00:00', 'EST') gives 2014-12-31 05:00:00 i.e. 5 hour difference
TO_UTC_TIMESTAMP('2014-06-30 00:00:00', 'EST') gives 2014-06-30 05:00:00, also 5 hour difference
I'm expecting the June query to give a 4 hour difference.
In June the East Coast observes EDT (Eastern Daylight Savings Time), but Hive doesn't understand EDT at all:
TO_UTC_TIMESTAMP('2014-12-31 00:00:00', 'EDT') gives 2014-12-31 00:00:00 i.e. no difference
Any ideas?
Thanks,
Ilmari
(Running Hadoop 1.0.3 on AWS Elastic MapReduce)
Here is an open ticket from the Hive project that address this issue.
https://issues.apache.org/jira/browse/HIVE-12194
See 2nd comment:
Ben Breakstone added a comment - 16/Oct/15 16:54
It's worth noting the daylight saving time version of US three-letter codes like "PDT" are not included in /lib/zi/ for the Oracle JDK. New identifiers like "PST8PDT" appear to work as expected.
See http://www.oracle.com/technetwork/articles/javase/alertfurtherinfo-139131.html
Perhaps as Ben Breakstone suggests new identifiers will work?