hive rows preceding unexpected behavior - hadoop

Given this ridiculously simple data set:
+--------+-----+
| Bucket | Foo |
+--------+-----+
| 1 | A |
| 1 | B |
| 1 | C |
| 1 | D |
+--------+-----+
I want to see the value of Foo in the previous row:
select
foo,
max(foo) over (partition by bucket order by foo rows between 1 preceding and 1 preceding) as prev_foo
from
...
Which gives me:
+--------+-----+----------+
| Bucket | Foo | Prev_Foo |
+--------+-----+----------+
| 1 | A | A |
| 1 | B | A |
| 1 | C | B |
| 1 | D | C |
+--------+-----+----------+
Why do I get 'A' back for the first row? I would expect it to be be null. It's throwing off calculations where I'm looking for that null. I can work around it by throwing a row_number() in there, but I'd prefer to handle it with fewer calcs.

use the LAG function to get previous row:
LAG(foo) OVER(partition by bucket order by foo) as Prev_Foo

Related

Delete column and Add column with default value Amplify graphql DynamoDB Appsync

I want to delete the whole column at once if there 1000's of data deleting one by one is time consuming so is there any best way to do it
Deleting this column
And my next question is I need to add new column with default value aftering deleting
i.e :
id | x | y | z
| A | B | C
| A | B | C
| A | B | C
| A | B | C
If above is the the table with thousands of data , I want to delete "z" and add "newColumn" with default value of "D" as below
id | x | y | newColumn
| A | B | D
| A | B | D
| A | B | D
| A | B | D

Spring Data Pagination GroupBy

I need the solution for the below question using Spring Data / Spring Boot.
I have a record in my MySQL table which is something like:
---------------------------------
| id | Postcontent | postdate |
------+-------------+------------
| 1 | A | 2013-01-31|
| 2 | B | 2013-01-31|
| 3 | C | 2013-01-30|
| 4 | D | 2013-01-30|
| 5 | E | 2013-01-29|
| 6 | F | 2013-01-29|
and I would like to show it in something like
2013-01-31
A
B
2013-01-30
C
D
2013-01-29
E
F
i can able to show like above with group by, but not able to add pagination in group by
if anyone can help me on how to add pagination in group by.

MDX - filter empty outside of selected range

Cube is populated with data divided into time dimension ( period ) which represents a month.
Following query:
select non empty {[Measures].[a], [Measures].[b], [Measures].[c]} on columns,
{[Period].[Period].ALLMEMEMBERS} on rows
from MyCube
returns:
+--------+----+---+--------+
| Period | a | b | c |
+--------+----+---+--------+
| 2 | 3 | 2 | (null) |
| 3 | 5 | 3 | 1 |
| 5 | 23 | 2 | 2 |
+--------+----+---+--------+
Removing non empty
select {[Measures].[a], [Measures].[b], [Measures].[c]} on columns,
{[Period].[Period].ALLMEMEMBERS} on rows
from MyCube
Renders:
+--------+--------+--------+--------+
| Period | a | b | c |
+--------+--------+--------+--------+
| 1 | (null) | (null) | (null) |
| 2 | 3 | 2 | (null) |
| 3 | 5 | 3 | 1 |
| 4 | (null) | (null) | (null) |
| 5 | 23 | 2 | 2 |
| 6 | (null) | (null) | (null) |
+--------+--------+--------+--------+
What i would like to get, is all records from period 2 to period 5, first occurance of values in measure "a" denotes start of range, last occurance - end of range.
This works - but i need this to be dynamically calculated during runtime by mdx:
select non empty {[Measures].[a], [Measures].[b], [Measures].[c]} on columns,
{[Period].[Period].&[2] :[Period].[Period].&[5]} on rows
from MyCube
desired output:
+--------+--------+--------+--------+
| Period | a | b | c |
+--------+--------+--------+--------+
| 2 | 3 | 2 | (null) |
| 3 | 5 | 3 | 1 |
| 4 | (null) | (null) | (null) |
| 5 | 23 | 2 | 2 |
+--------+--------+--------+--------+
I tried looking for first/last values but just couldn't compose them into the query properly. Anyone has this issue before ? This should be pretty common seeing as I want to get a continuous financial report without skipping months where nothing is going on. Thanks.
Maybe try playing with NonEmpty / Tail function in a WITH clause:
WITH
SET [First] AS
{HEAD(NONEMPTY([Period].[Period].MEMBERS, [Measures].[a]))}
SET [Last] AS
{TAIL(NONEMPTY([Period].[Period].MEMBERS, [Measures].[a]))}
SELECT
{
[Measures].[a]
, [Measures].[b]
, [Measures].[c]
} on columns,
[First].ITEM(0).ITEM(0)
:[Last].ITEM(0).ITEM(0) on rows
FROM MyCube;
to debug a custom set, to see what members it is returning you can do something like this:
WITH
SET [First] AS
{HEAD(NONEMPTY([Period].[Period].MEMBERS, [Measures].[a]))}
SELECT
{
[Measures].[a]
, [Measures].[b]
, [Measures].[c]
} on columns,
[First] on rows
FROM MyCube;
I think reading your comment about Children means that this is also an alternative - to add an extra [Period]:
WITH
SET [First] AS
{HEAD(NONEMPTY([Period].[Period].[Period].MEMBERS
, [Measures].[a]))}
SET [Last] AS
{TAIL(NONEMPTY([Period].[Period].[Period].MEMBERS
, [Measures].[a]))}
SELECT
{
[Measures].[a]
, [Measures].[b]
, [Measures].[c]
} on columns,
[First].ITEM(0).ITEM(0)
:[Last].ITEM(0).ITEM(0) on rows
FROM MyCube;

HIVE Pivot and Sum

I have a table that I am trying to figure out how to pivot and sum based on the values in a second column.
Example input:
|own|pet|qty|
|---|---|---|
|bob|dog| 2 |
|bob|dog| 3 |
|bob|dog| 1 |
|bob|cat| 1 |
|jon|dog| 1 |
|jon|cat| 1 |
|jon|cat| 1 |
|jon|cow| 4 |
|sam|dog| 3 |
|sam|cow| 1 |
|sam|cow| 2 |
Example output:
|own|dog|cat|cow|
|---|---|---|---|
|bob| 6 | 1 | |
|jon| 1 | 2 | 4 |
|sam| 1 | | 3 |
Use case and sum():
select own, sum(case when pet='dog' then qty end) as dog,
sum(case when pet='cat' then qty end) as cat,
sum(case when pet='cow' then qty end) as cow
from your_table
group by own;
For dynamic data you can use MAP
select own
,str_to_map(concat_ws(',',collect_list(concat(pet,':',cast(qty as string))))) as pet_qty
from (select own,pet
,sum(qty) qty
from mytable
group by own,pet
) t
group by own
;
+-----+---------------------------------+
| own | pet_qty |
+-----+---------------------------------+
| bob | {"cat":"1","dog":"6"} |
| jon | {"cat":"2","cow":"4","dog":"1"} |
| sam | {"cow":"3","dog":"3"} |
+-----+---------------------------------+

Sum of the grouped distinct values

This is a bit hard to explain in words ... I'm trying to calculate a sum of grouped distinct values in a matrix. Let's say I have the following data returned by a SQL query:
------------------------------------------------
| Group | ParentID | ChildID | ParentProdCount |
| A | 1 | 1 | 2 |
| A | 1 | 2 | 2 |
| A | 1 | 3 | 2 |
| A | 1 | 4 | 2 |
| A | 2 | 5 | 3 |
| A | 2 | 6 | 3 |
| A | 2 | 7 | 3 |
| A | 2 | 8 | 3 |
| B | 3 | 9 | 1 |
| B | 3 | 10 | 1 |
| B | 3 | 11 | 1 |
------------------------------------------------
There's some other data in the query, but it's irrelevant. ParentProdCount is specific to the ParentID.
Now, I have a matrix in the MS Report Designer in which I'm trying to calculate a sum for ParentProdCount (grouped by "Group"). If I just add the expression
=Sum(Fields!ParentProdCount.Value)
I get a result 20 for Group A and 3 for Group B, which is incorrect. The correct values should be 5 for group A and 1 for group B. This wouldn't happen if there wasn't ChildID involved, but I have to use some other child-specific data in the same matrix.
I tried to nest FIRST() and SUM() aggregate functions but apparently it's not possible to have nested aggregation functions, even when they have scopes defined.
I'm pretty sure there is some way to calculate the grouped distinct sum without needing to create another SQL query. Anyone got an idea how to do that?
Ok I got this sorted out by adding a ROW_NUMBER() function my SQL query:
SELECT Group, ParentID, ROW_NUMBER() OVER (PARTITION BY ParentID ORDER BY ChildID ASC) AS Position, ChildID, ParentProdCount FROM Table
and then I replaced the SSRS SUM function with
=SUM(IIF(Position = 1, ParentProdCount.Value, 0))
Put a grouping over the ParentID and use a summation over that group,
eg:
if group over ParentID = "ParentIDGroup"
then
column sum of ParentPrdCount = SUM(Fields!ParentProdCount.Value,"ParentIDGroup")

Resources