I have a table a show below
Date | Customer | Count | Daily_Count | ITD_Count
d1 | A | 3 | 3 |
d2 | B | 4 | 4 |
d3 | A | 7 | 16 |
d3 | B | 9 | 16 |
d4 | A | 8 | 9 |
d4 | B | 1 | 9 |
Descrption of Fields:
Date : date
customer : name of customer
Count : # of customers
daily_Count : # of customers on daily basis calculated as
SUM(count) OVER (partition BY date )as Daily_Count
Question :
How do I calculate the Running Total or Rolling Total in the ITD_Count ?
The output should look like
Date | Customer | Count | Daily_Count | ITD_Count
d1 | A | 3 | 3 | 3
d2 | B | 4 | 4 | 7
d3 | A | 7 | 16 | 23
d3 | B | 9 | 16 | 23
d4 | A | 8 | 9 | 31
d4 | B | 1 | 9 | 31
I have tried several variations of using the Window functionality.. But hit a road-block in all my attempts.
Attempt 1 ;
SUM(daily_COunt) OVER (partition BY date order by date rows between unbounded preceding and current row ) as ITD_account_linking
Attempt 2 :
SUM(daily_COunt) OVER (partition BY date, daily_count order by date rows between unbounded preceding and current row ) as ITD_account_linking
and several more attempts following this. :(
Any possible suggestions to guide me in the right direction are welcome.
Please let me know if you need more details.
Use Hive Windowing and Analytics functions.
SELECT Date, Customer, Count, Daily_Count,
SUM(Daily_Count) OVER (ORDER BY Date ROWS UNBOUNDED PRECEDING) AS ITD_Count
FROM table;
Related
I was wondering if there is a version of 'Proc Transpose' in SAS Hive SQL (Hadoop) ?
Otherwise I can see the only other (long winded) way is creating a lot of separate tables to then join back together, which I'd rather avoid.
Any assistance most welcome!
Sample table to Transpose > Intention to put Month along the top of the table so the rates are split by month:
+------+-------+----------+----------+-------+
| YEAR | MONTH | Geog | Category | Rates |
+------+-------+----------+----------+-------+
| 2018 | 1 | National | X | 32 |
| 2018 | 1 | National | Y | 43 |
| 2018 | 1 | National | Z | 47 |
| 2018 | 1 | Regional | X | 52 |
| 2018 | 1 | Regional | Y | 38 |
| 2018 | 1 | Regional | Z | 65 |
| 2018 | 2 | National | X | 63 |
| 2018 | 2 | National | Y | 14 |
| 2018 | 2 | National | Z | 34 |
| 2018 | 2 | Regional | X | 90 |
| 2018 | 2 | Regional | Y | 71 |
| 2018 | 2 | Regional | Z | 69 |
+------+-------+----------+----------+-------+
Sample output:
+------+----------+----------+----+----+
| YEAR | Geog | Category | 1 | 2 |
+------+----------+----------+----+----+
| 2018 | National | X | 32 | 63 |
| 2018 | National | Y | 43 | 14 |
| 2018 | National | Z | 47 | 34 |
| 2018 | Regional | X | 52 | 90 |
| 2018 | Regional | Y | 38 | 71 |
| 2018 | Regional | Z | 65 | 69 |
+------+----------+----------+----+----+
The typical wallpaper SQL technique for transposing (or pivoting) is a group+transform to pivot case statements sub-query within a group aggregating query that collapses the sub-query. The group represents a single resultant pivot row.
For example your group is year, geog, category and min is used to collapse:
proc sql;
create view want_pivot as
select year, geog, category
, min(rate_m1) as rate_m1
, min(rate_m2) as rate_m2
from
( select
year, geog, category
, case when month=1 then rates end as rate_m1
, case when month=2 then rates end as rate_m2
from have
)
group by year, geog, category
;
Here is the same concept, a little more generically where data is repeated within the group at the detail level and mean is used to collapse over the repeats.
data have;
input id name $ value;
datalines;
1 a 1
1 a 2
1 a 3
1 b 2
1 c 3
2 a 2
2 d 4
2 b 5
3 e 1
run;
proc sql;
create view have_pivot as
select
id
, mean(a) as a
, mean(b) as b
, mean(c) as c
, mean(d) as d
, mean(e) as e
from
(
select
id
, case when name='a' then value end as a
, case when name='b' then value end as b
, case when name='c' then value end as c
, case when name='d' then value end as d
, case when name='e' then value end as e
from have
)
group by id
;
quit;
When the column names are not known apriori, you will need to write a code generator that passes over all the data to determine the name values, writes the wall paper query which will perform a second pass over the data returning the pivot.
Also, many contemporary data bases have a PIVOT clause that can be leveraged via pass through.
The Hadoop Mania post "TRANSPOSE/PIVOT a Table in Hive" shows the use of collect_list and map in a similar wallpapery manner:
select b.id, b.code, concat_ws('',b.p) as p, concat_ws('',b.q) as q, concat_ws('',b.r) as r, concat_ws('',b.t) as t from
(select id, code,
collect_list(a.group_map['p']) as p,
collect_list(a.group_map['q']) as q,
collect_list(a.group_map['r']) as r,
collect_list(a.group_map['t']) as t
from ( select
id, code,
map(key,value) as group_map
from test_sample
) a group by a.id, a.code) b;
if your sample dataset is representative of real dataset then you can use a simple inner join as shown below. Year geo and categoty makes unique combination below code should work.
select a.YEAR ,
a.Geog ,
a.Category ,
a.Rates ,
a.month as month_1,
b.month as month_2
from have a
inner join
have b
on a.year = b.year
and a.Geog = b.Geog
and a.Category = b.category
where a.month ne b.month;
I have a table with data at hour level. I want to find the count of hours and the values for col1 and col2 for all hours in an array. Input Table
+-----+-----+-----+
| hour| col1| col2|
+-----+-----+-----+
| 00 | 0.0 | a |
| 04 | 0.1 | b |
| 08 | 0.2 | c |
| 12 | 0.0 | d |
+-----+-----+-----+
I am using the below query to get the column values in an array
Query:
select count(hr), map_values(str_to_map(concat_ws(',',collect_set(concat_ws(':',reflect('java.util.UUID','randomUUID'),cast(col1 as string)))))) as col1_arr, map_values(str_to_map(concat_ws(',',collect_set(concat_ws(':',reflect('java.util.UUID','randomUUID'),cast(col2 as string)))))) as col2_arr from table;
Output that i am getting, values in col2_arr are not in the same sequence with col1_arr. Please suggest how can i get the values in array/list for different columns in same sequence.
+----------+-----------------+----------+
| count(hr)| col1_arr | col2_arr |
+----------+-----------------+----------+
| 4 | 0.0,0.1,0.2,0.0 | b,a,c,d |
+----------+----------------+-----------+
Required output:
+----------+-----------------+----------+
| count(hr)| col1_arr | col2_arr |
+----------+-----------------+----------+
| 4 | 0.0,0.1,0.2,0.0 | a,b,c,d |
+----------+----------------+-----------+
Thanks
select count(*) as cnt
,concat_ws(',',sort_array(collect_list(hour))) as hour
,regexp_replace(concat_ws(',',sort_array(collect_list(concat_ws(':',hour,cast(col1 as string))))),'..:','') as col1
,regexp_replace(concat_ws(',',sort_array(collect_list(concat_ws(':',hour,col2)))),'..:','') as col2
from mytable
;
+-----+-------------+-------------+---------+
| cnt | hour | col1 | col2 |
+-----+-------------+-------------+---------+
| 4 | 00,04,08,12 | 0,0.1,0.2,0 | a,b,c,d |
+-----+-------------+-------------+---------+
I have models Book and BookCategory
How do I select the cheapest book in every category?
Book table:
| id | name | price | book_category_id |
| 1 | test | 10 | 1
| 2 | test | 15 | 3
| 3 | test | 75 | 1
| 4 | test | 25 | 2
| 5 | test | 19 | 1
| 6 | test | 11 | 2
| 7 | test | 10 | 1
The selection should be :
| id | name | price | book_category_id |
| 1 | test | 10 | 1
| 2 | test | 15 | 3
| 6 | test | 11 | 2
I've tried:
$books = Book::groupBy("book_category_id")->orderBy("price")->get()
But the output is not the minimum price row.
any idea?
EDIT:
I found this page:
https://www.xaprb.com/blog/2006/12/07/how-to-select-the-firstleastmax-row-per-group-in-sql/
it has 90% of the solution in SQL
SELECT *
FROM books
WHERE price = (SELECT MIN(price) FROM books AS u WHERE u.book_category_id= books.book_category_id)
GROUP BY books.book_category_id
how to convert this to laravel query builder?
You need to perform a subquery like this post. Try this:
$books = Book::from(DB::raw("SELECT * FROM books order by price asc"))
->groupBy("book_category_id")->get();
Please note that this is a mysql only solution because in mysql you're allowed to not aggregate non-group-by columns. If you need to do this for another DB, you need to perform an inner join on the subquery
How do I do the following in Oracle:
I have a (simplified) table:
+-----+-----+-----+
| a | b | ... |
+-----+-----+-----+
| 1 | 7 | ... |
| 2 | 5 | ... |
| 1 | 7 | ... |
+-----+-----+-----+
Where a functions as a unique identifier for a person, and b is the field I am interested in matching across rows. How do I construct a query that basically says "give me the person-ID's where the person has multiple b values (i.e., duplicates)"?
So far I have tried:
SELECT a FROM mytable GROUP BY a HAVING COUNT(DISTINCT b) > 1;
This feels close except it just gives me the user IDs where the user has multiple unique b's, which I suspect is coming from the DISTINCT part, but I'm not sure how to change the query to achieve what I want.
Try
group by a,b having count(b) > 1
Yours would count 7,5,7 as 2 (one 7, one 5). This one one will count total Bs in any grouping, so you'll get 1,7 - > 2 and 1,5 -> 1
SQL Fiddle
Oracle 11g R2 Schema Setup:
CREATE TABLE mytable ( a, b ) AS
SELECT LEVEL, LEVEL FROM DUAL CONNECT BY LEVEL <= 2000
UNION ALL
SELECT LEVEL *2, LEVEL * 2 FROM DUAL CONNECT BY LEVEL <= 1000;
Query 1:
WITH data AS (
SELECT a
FROM mytable
GROUP BY a
HAVING COUNT(b) > COUNT( DISTINCT b )
ORDER BY a
),
numbered AS (
SELECT a,
ROWNUM AS rn
FROM data
)
SELECT a
FROM numbered
WHERE rn <= 20
Results:
| A |
|----|
| 2 |
| 4 |
| 6 |
| 8 |
| 10 |
| 12 |
| 14 |
| 16 |
| 18 |
| 20 |
| 22 |
| 24 |
| 26 |
| 28 |
| 30 |
| 32 |
| 34 |
| 36 |
| 38 |
| 40 |
Assume a range of values inserted in a schema table and in the end of the month i want to apply for these records (i.e. 2500 rows = numeric values) the algorithm: sort the values descending (from the smallest to highest value) and then find the 80% value of the sorted column.
In my example, if each row increases by one starting from 1, the 80% value will be the 2000 row=value (=2500-2500*20/100). This algorithm needs to be implemented in a procedure where the number of rows is not constant, for example it can varries from 2500 to 1,000,000 per month
Hint: You can achieve this using Oracle's cumulative aggregate functions. For example, suppose your table looks like this:
MY_TABLE
+-----+----------+
| ID | QUANTITY |
+-----+----------+
| A | 1 |
| B | 2 |
| C | 3 |
| D | 4 |
| E | 5 |
| F | 6 |
| G | 7 |
| H | 8 |
| I | 9 |
| J | 10 |
+-----+----------+
At each row, you can sum the quantities so far using this:
SELECT
id,
quantity,
SUM(quantity)
OVER (ORDER BY quantity ROWS UNBOUNDED PRECEDING)
AS cumulative_quantity_so_far
FROM
MY_TABLE
Giving you:
+-----+----------+----------------------------+
| ID | QUANTITY | CUMULATIVE_QUANTITY_SO_FAR |
+-----+----------+----------------------------+
| A | 1 | 1 |
| B | 2 | 3 |
| C | 3 | 6 |
| D | 4 | 10 |
| E | 5 | 15 |
| F | 6 | 21 |
| G | 7 | 28 |
| H | 8 | 36 |
| I | 9 | 45 |
| J | 10 | 55 |
+-----+----------+----------------------------+
Hopefully this will help in your work.
Write a query using the percentile_disc function to solve your problem. Sounds like it does what you want.
An example would be
select percentile_disc(0.8) within group (order by the_value)
from my_table