MDX Ordering By Dimension - sorting

Over in the SQL side, my data is looking like this:
Select f.id, f.TimeKey,t.CalendarYearMonth
from FactSubmission f
inner join DimTime t on t.TimeKey = f.TimeKey
order by f.Id asc
Sorting from MDX we have descending
SELECT
NON EMPTY ORDER(
[DimTime.CalendarYearMonth].[CalendarYearMonth].Members,
[DimTime.CalendarYearMonth].CurrentMember.Properties("MEMBER_KEY"),
DESC
) ON COLUMNS
FROM [PSE_FactSubmission]
And Ascending
The January dates aren't at the top of either sort, which suggests I'm sorting by the FactSubmission.ID key instead of DimTime.CalendarYearMonth
Is this how things are supposed to work? I'd like to pull back Jan,Feb,March.
DimTime.CalendarYearMonthNum is a column with data in the form 201501,201502,201503 etc. Here's an attempt at using this column to to sort the CalendarYearMonth data.
Debugging Query to Select Keys
NonEmpty Query

Try ordering using a different property:
SELECT
NON EMPTY ORDER(
[DimTime.CalendarYearMonth].[CalendarYearMonth].Members,
[DimTime.CalendarYearMonth].CurrentMember.Properties("MEMBER_Value"),
DESC
) ON COLUMNS
FROM [PSE_FactSubmission];
or maybe this:
SELECT
NON EMPTY ORDER(
[DimTime.CalendarYearMonth].[CalendarYearMonth].Members,
[DimTime.CalendarYearMonth].CurrentMember.MEMBERValue,
DESC
) ON COLUMNS
FROM [PSE_FactSubmission];
In the above you should be good using DESC - sometimes you need to break the underlying hierarchical ordering by adding a B i.e. BDESC
From here I cannot see MEMBER_VALUE: http://mondrian.pentaho.com/documentation/mdx.php
...but there is a function .VALUE so maybe try the following:
SELECT
NON EMPTY ORDER(
[DimTime.CalendarYearMonth].[CalendarYearMonth].Members,
[DimTime.CalendarYearMonth].CurrentMember.Value,
DESC
) ON COLUMNS
FROM [PSE_FactSubmission];
Strange that the key doesn't work. What values do you get if you run something like this?
WITH MEMBER [KEYcheck] AS
[DimTime.CalendarYearMonth].CurrentMember.Properties("MEMBER_KEY")
//[DimTime.CalendarYearMonth].CurrentMember.MEMBER_KEY
//[DimTime.CalendarYearMonth].[CalendarYearMonth].CurrentMember.MEMBER_KEY
//[DimTime].CurrentMember.MEMBER_KEY
SELECT
[KEYcheck] ON 0,
[DimTime.CalendarYearMonth].[CalendarYearMonth].Members ON 1
FROM [PSE_FactSubmission];

You are doing an alphabetical sort. February->January->March.
For doing a sort based on the month number, there needs to be a field which maps January-1, February-2, March-3.
If you have such a column in cube, use that to to sort. If not create a calculated member like below -
WITH MEMBER Measures.CalendarMonth AS
CASE [DimTime.CalendarYearMonth].CurrentMember
WHEN [DimTime.CalendarYearMonth].&[January] THEN 1
WHEN [DimTime.CalendarYearMonth].&[February] THEN 2
WHEN [DimTime.CalendarYearMonth].&[March] THEN 3
END
SELECT
NON EMPTY ORDER(
[DimTime.CalendarYearMonth].[CalendarYearMonth].Members,
Measures.CalendarMonth,
DESC
) ON COLUMNS
FROM [PSE_FactSubmission]
EDIT for Andrew
with member Measures.[MonthNum] as
NonEmpty
(
[DimTime.CalendarYearMonthNum].[CalendarMonthNum].members,
([DimTime.CalendarYearMonth].[CalendarYearMonth].currentmember, Measures.foo)
).item(0).membervalue
select
non empty
order
(
[DimTime.CalendarYearMonth].[CalendarYearMonth].members,
Measures.[MonthNum],
desc
) on rows
from [PSE_FactSubmission]
EDIT - with EXISTS
with member Measures.[MonthNum] as
EXISTS
(
[DimTime.CalendarYearMonthNum].[CalendarMonthNum].members,
[DimTime.CalendarYearMonth].[CalendarYearMonth].currentmember,
"SomeMeasureGroup"
).item(0).membervalue

Related

Trying to display top 3 amount from a table using sql query in oracle 11g..column is of varchar type

Am trying to list top 3 records from atable based on some amount stored in a column FTE_TMUSD which is of varchar datatype
below is the query i tried
SELECT *FROM
(
SELECT * FROM FSE_TM_ENTRY
ORDER BY FTE_TMUSD desc
)
WHERE rownum <= 3
ORDER BY FTE_TMUSD DESC ;
o/p i got
972,9680,963 -->FTE_TMUSD values which are not displayed in desc
I am expecting an o/p which will display the top 3 records of values
That should work; inline view is ordered by FTE_TMUSD in descending order, and you're selecting values from it.
What looks suspicious are values you specified as the result. It appears that FTE_TMUSD's datatype is VARCHAR2 (ah, yes - it is, you said so). It means that values are sorted as strings, not numbers - and it seems that you expect numbers. So, apply TO_NUMBER to that column. Note that it'll fail if column contains anything but numbers (for example, if there's a value 972C).
Also, an alternative to your query might be use of analytic functions, such as row_number:
with temp as
(select f.*,
row_number() over (order by to_number(f.fte_tmusd) desc) rn
from fse_tm_entry f
)
select *
from temp
where rn <= 3;

MDX - How to select one column and sort the returned data

For a SSRS report, I'm trying to return a list of sorted data from a dimension to use with a parameter.
My dimension is [Radio].[Radio NO].[Radio NO] where the last Radio NO is a string.
I can find examples of returning one column while sorting on another but I can't figure out how to sort and return just one column.
Thanks whytheq! Based on your answer, here's what I came up with that works:
SELECT {} ON COLUMNS,
ORDER(
[Radio].[Radio NO].[Radio NO].MEMBERS
,[Radio].[Radio NO].CURRENTMEMBER.MEMBER_CAPTION
,BASC
) On ROWS
FROM [OurCube]
Without seeing the exact structure of your cube / query an avenue you could explore, if you'd like to order alphabetical, is the following
ORDER(
[Radio].[Radio NO].[Radio NO].MEMBERS
,[Radio].[Radio NO].CURRENTMEMBER.MEMBER_CAPTION
,BDESC
)
If you want to order by a measure in your cube, then something like the following:
ORDER(
[Radio].[Radio NO].[Radio NO].MEMBERS
,[Measures].[Profit]
,BDESC
)
This is a possible if you really need to change the column name before hitting SSRS but it has the disadvantage of changing it to a measure:
WITH
MEMBER [Measures].[thisIsTheNewName] AS
[Radio].[Radio NO].CURRENTMEMBER.MEMBER_CAPTION
SELECT
{[Measures].[thisIsTheNewName]} ON COLUMNS,
ORDER(
[Radio].[Radio NO].[Radio NO].MEMBERS
,[Radio].[Radio NO].CURRENTMEMBER.MEMBER_CAPTION
,BASC
) On ROWS
FROM [OurCube];

Delete duplicate rows from a BigQuery table

I have a table with >1M rows of data and 20+ columns.
Within my table (tableX) I have identified duplicate records (~80k) in one particular column (troubleColumn).
If possible I would like to retain the original table name and remove the duplicate records from my problematic column otherwise I could create a new table (tableXfinal) with the same schema but without the duplicates.
I am not proficient in SQL or any other programming language so please excuse my ignorance.
delete from Accidents.CleanedFilledCombined
where Fixed_Accident_Index
in(select Fixed_Accident_Index from Accidents.CleanedFilledCombined
group by Fixed_Accident_Index
having count(Fixed_Accident_Index) >1);
You can remove duplicates by running a query that rewrites your table (you can use the same table as the destination, or you can create a new table, verify that it has what you want, and then copy it over the old table).
A query that should work is here:
SELECT *
FROM (
SELECT
*,
ROW_NUMBER()
OVER (PARTITION BY Fixed_Accident_Index)
row_number
FROM Accidents.CleanedFilledCombined
)
WHERE row_number = 1
UPDATE 2019: To de-duplicate rows on a single partition with a MERGE, see:
https://stackoverflow.com/a/57900778/132438
An alternative to Jordan's answer - this one scales better when having too many duplicates:
#standardSQL
SELECT event.* FROM (
SELECT ARRAY_AGG(
t ORDER BY t.created_at DESC LIMIT 1
)[OFFSET(0)] event
FROM `githubarchive.month.201706` t
# GROUP BY the id you are de-duplicating by
GROUP BY actor.id
)
Or a shorter version (takes any row, instead of the newest one):
SELECT k.*
FROM (
SELECT ARRAY_AGG(x LIMIT 1)[OFFSET(0)] k
FROM `fh-bigquery.reddit_comments.2017_01` x
GROUP BY id
)
To de-duplicate rows on an existing table:
CREATE OR REPLACE TABLE `deleting.deduplicating_table`
AS
# SELECT id FROM UNNEST([1,1,1,2,2]) id
SELECT k.*
FROM (
SELECT ARRAY_AGG(row LIMIT 1)[OFFSET(0)] k
FROM `deleting.deduplicating_table` row
GROUP BY id
)
Not sure why nobody mentioned DISTINCT query.
Here is the way to clean duplicate rows:
CREATE OR REPLACE TABLE project.dataset.table
AS
SELECT DISTINCT * FROM project.dataset.table
If your schema doesn’t have any records - below variation of Jordan’s answer will work well enough with writing over same table or new one, etc.
SELECT <list of original fields>
FROM (
SELECT *, ROW_NUMBER() OVER (PARTITION BY Fixed_Accident_Index) AS pos,
FROM Accidents.CleanedFilledCombined
)
WHERE pos = 1
In more generic case - with complex schema with records/netsed fields, etc. - above approach can be a challenge.
I would propose to try using Tabledata: insertAll API with rows[].insertId set to respective Fixed_Accident_Index for each row.
In this case duplicate rows will be eliminated by BigQuery
Of course, this will involve some client side coding - so might be not relevant for this particular question.
I havent tried this approach by myself either but feel it might be interesting to try :o)
If you have a large-size partitioned table, and only have duplicates in a certain partition range. You don't want to overscan nor process the whole table. use the MERGE SQL below with predicates on partition range:
-- WARNING: back up the table before this operation
-- FOR large size timestamp partitioned table
-- -------------------------------------------
-- -- To de-duplicate rows of a given range of a partition table, using surrage_key as unique id
-- -------------------------------------------
DECLARE dt_start DEFAULT TIMESTAMP("2019-09-17T00:00:00", "America/Los_Angeles") ;
DECLARE dt_end DEFAULT TIMESTAMP("2019-09-22T00:00:00", "America/Los_Angeles");
MERGE INTO `gcp_project`.`data_set`.`the_table` AS INTERNAL_DEST
USING (
SELECT k.*
FROM (
SELECT ARRAY_AGG(original_data LIMIT 1)[OFFSET(0)] k
FROM `gcp_project`.`data_set`.`the_table` AS original_data
WHERE stamp BETWEEN dt_start AND dt_end
GROUP BY surrogate_key
)
) AS INTERNAL_SOURCE
ON FALSE
WHEN NOT MATCHED BY SOURCE
AND INTERNAL_DEST.stamp BETWEEN dt_start AND dt_end -- remove all data in partiion range
THEN DELETE
WHEN NOT MATCHED THEN INSERT ROW
credit: https://gist.github.com/hui-zheng/f7e972bcbe9cde0c6cb6318f7270b67a
Easier answer, without a subselect
SELECT
*,
ROW_NUMBER()
OVER (PARTITION BY Fixed_Accident_Index)
row_number
FROM Accidents.CleanedFilledCombined
WHERE TRUE
QUALIFY row_number = 1
The Where True is neccesary because qualify needs a where, group by or having clause
Felipe's answer is the best approach for most cases. Here is a more elegant way to accomplish the same:
CREATE OR REPLACE TABLE Accidents.CleanedFilledCombined
AS
SELECT
Fixed_Accident_Index,
ARRAY_AGG(x LIMIT 1)[SAFE_OFFSET(0)].* EXCEPT(Fixed_Accident_Index)
FROM Accidents.CleanedFilledCombined AS x
GROUP BY Fixed_Accident_Index;
To be safe, make sure you backup the original table before you run this ^^
I don't recommend to use ROW NUMBER() OVER() approach if possible since you may run into BigQuery memory limits and get unexpected errors.
Update BigQuery schema with new table column as bq_uuid making it NULLABLE and type STRING

Create duplicate rows by running same command 5 times for example
insert into beginner-290513.917834811114.messages (id, type, flow, updated_at) Values(19999,"hello", "inbound", '2021-06-08T12:09:03.693646')
Check if duplicate entries exist
select * from beginner-290513.917834811114.messages where id = 19999
Use generate uuid function to generate uuid corresponding to each message

UPDATE beginner-290513.917834811114.messages
SET bq_uuid = GENERATE_UUID()
where id>0
Clean duplicate entries
DELETE FROM beginner-290513.917834811114.messages
WHERE bq_uuid IN
(SELECT bq_uuid
FROM
(SELECT bq_uuid,
ROW_NUMBER() OVER( PARTITION BY updated_at
ORDER BY bq_uuid ) AS row_num
FROM beginner-290513.917834811114.messages ) t
WHERE t.row_num > 1 );

MDX rather complicated sorting

I can't find out a way, how to sort my query, this is the simple query:
SELECT {[Measures].[IB]}
ON COLUMNS,
{[Dim_Product_Models_new].[PLA].members } *
{[Dim Dates_new].[Date Full].&[2013-02-01]:[Dim Dates_new].[Date Full].&[2014-01-01]}
ON ROWS
FROM [cub_dashboard_spares]
The think is, I would get a result for 6 PLAs combined across 12 months (72 rows in total), however it is sorted alphabetically upon PLA.
What i need, is to sort the PLAs based on a measure in last month (2014-01-01 in this case).
Is there any way to perform this task so that the groupping (PLAs, Dates from 2013-02 to 2013-12) is perserved, but only the order of my PLAs is different. (PLA with highest measure in last month would be first, and so on)
Thank you very much for any kind of help
Just put the sorted set on the rows, using the Order function. The third parameter of this function is DESC if you want to sort within each hierarchy level, but still want to get parents before children (like ALL before the single attribute members), or BDESC if you want to sort across all levels.
SELECT {[Measures].[IB]}
ON COLUMNS,
Order({[Dim_Product_Models_new].[PLA].members },
([Measures].[IB], [Dim Dates_new].[Date Full].&[2014-01-01]),
DESC)
*
{[Dim Dates_new].[Date Full].&[2013-02-01]:[Dim Dates_new].[Date Full].&[2014-01-01]}
ON ROWS
FROM [cub_dashboard_spares]
The order function over a crossjoin should preserve the initial order of the first set so reversing the order of the tuple will do the job:
SELECT
{
[Measures].[IB]
} ON COLUMNS,
order(
{[Dim Dates_new].[Date Full].&[2013-02-01]:[Dim Dates_new].[Date Full].&[2014-01-01]} *
{[Dim_Product_Models_new].[PLA].members } ,
[Measures].[IB],
desc
) ON ROWS
FROM [cub_dashboard_spares]
If you want to preserve the oder of appearance of the column labels, you can use the generate function like in the following example from the AW cube:
SELECT
{[Measures].[Internet Sales Amount]} ON 0
,Generate
(
{[Customer].[Country].&[Australia]:[Customer].[Country].&[United Kingdom]}
,(
Order
(
[Date].[Calendar Year].[Calendar Year].MEMBERS
,(
[Customer].[Country].CurrentMember
,[Measures].[Internet Sales Amount]
)
,DESC
)
,[Customer].[Country].CurrentMember
)
) ON 1
FROM [Adventure Works];
Philip,

Trying to select all records where two fields are distinct (but the rest don't have to be)

I have a table that has 14 columns in it. These columns are color, type, ft, date, count, etc. What I need is to select all distinct records of id and type with the most recent date. So, for example...
color------type-----------date
red--------work-----------01/01/01
red---------play----------02/02/02
red---------play----------03/03/03
In this case, I want to return red, work, 01/01/01 and red, play 03/03/03. Hopefully this makes sense. I've tried different combinations of select unique and select distinct and group bys, and I haven't been able to come up with anything.
Here is the SQL statement I'm trying:
select distinct
chock_id,
roll_type,
max(chock_service_dt),
chock_id_dt,
chock_seq_num,
chock_service_cmnt,
total_rolled_lineal_ft,
total_rolled_tons,
chock_usage_cnt,
chock_insert_dt,
record_modify_dt,
next_chock_service_dt_act,
previous_alarm_value,
upload_complete_yn
from
tp07_chock_summary_row
group by
chock_id,
roll_type,
chock_service_dt,
chock_id_dt,
chock_seq_num,
chock_service_cmnt,
total_rolled_lineal_ft,
total_rolled_tons,
chock_usage_cnt,
chock_insert_dt,
record_modify_dt,
next_chock_service_dt_act,
previous_alarm_value,
upload_complete_yn;
Here's a screenshot. Like I said in a comment below, like in rows 2 and 4, I can't have multiple records with the same chock_id and roll_type.
Given your new requirements, which you did not explain initially, this should do it:
select
chock_id,
roll_type,
chock_service_dt,
chock_id_dt,
chock_seq_num,
chock_service_cmnt,
total_rolled_lineal_ft,
total_rolled_tons,
chock_usage_cnt,
chock_insert_dt,
record_modify_dt,
next_chock_service_dt_act,
previous_alarm_value,
upload_complete_yn
from (
select
chock_id,
roll_type,
chock_service_dt,
chock_id_dt,
chock_seq_num,
chock_service_cmnt,
total_rolled_lineal_ft,
total_rolled_tons,
chock_usage_cnt,
chock_insert_dt,
record_modify_dt,
next_chock_service_dt_act,
previous_alarm_value,
upload_complete_yn,
row_number() over (
partition by chock_id, roll_type
order by chock_service_dt desc
) rn
from
tp07_chock_summary_row
) where rn = 1
select color, type, max(date)
from ...
group by color, type
select
color,
type,
max(date)
from
yourtable
group by
color,
type

Resources