Generate Calendar Table from Min/Max Seed - powerquery

I'm trying to create a calendar from YYYYMM (Integer) value in my SQL table. Here is the first step - getting the start value = "01 jan 2019" in a Date format. The code below returns an error with a description that it "cannot convert a value of type Record to type Text".
let
mySource=
Sql.Database("sqlServer"
,"DWH"
,[Query=
"SELECT
MIN(mdate) as mdate
FROM BD_plan_AC
UNION
SELECT
MAX(mdate) as mdate
FROM myTable"
]
)-- here I get the mdate range within I want to build my calendar.
-- from the first day of mdate to the end of the last mdate.
,startDateAsNum=Record.Field(Table.First (mySource),"mdate")*100+1
// Table.First (mySource),"mdate")=201901
// startDateAsNum=20190101
,startDateAsText=Text.From(startDateAsNum)
,startDateAsDate=Date.FromText(startDateAsText,[Format="yyyyMMdd"])
in
startDateAsDate
Many thanks in advance.
The idea is to get the mdate range and build a calendar from the first day of min to the last day of max, so, all dates from the range are in.
mySource = -- dates are not constant and comes from the query.
mdate
201901
202212
This, what I want to get finally - a calendar with a Date() data type. (Dates are as dd.mm.yyyy)
Date
01.01.2019
02.01.2019
....
31.12.2022

Given your input data, you can generate a calendar as follows.
let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WMjIwtDQwVIrVATGNjAyNlGJjAQ==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [mdate = _t]),
#"Changed Type" = Table.TransformColumnTypes(Source,{{"mdate", type text}}),
start = #date( Number.From( Text.Range( #"Changed Type"[mdate]{0},0,4 )) ,Number.From( Text.Range( #"Changed Type"[mdate]{0},4,2 )),1 ),
end = #date( Number.From( Text.Range( #"Changed Type"[mdate]{1},0,4 )) ,Number.From( Text.Range( #"Changed Type"[mdate]{1},4,2 )),31 ),
cal = { Number.From( start ) .. Number.From( end ) },
#"Converted to Table" = Table.FromList(cal, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Changed Type1" = Table.TransformColumnTypes(#"Converted to Table",{{"Column1", type date}})
in
#"Changed Type1"

Why do you need to do anything?
01 jan 2019 converts perfectly to a date with Date.From() in powerquery

try this to get the data from SQL in Format yyyymmdd then maybe you have to convert anything...
SELECT
format(cast(MIN(mdate) ,datetime),"YYYYMM") as mdate
FROM BD_plan_AC
UNION
SELECT
format(cast(MAX(mdate) ,datetime),"YYYYMM") as mdate
FROM myTable

Related

Power Query - Can a range be queried using SQL?

I have a range named tDetails. How can I query this using SQL?
I click in my range, and on the Power Query tab, I click "From Table/Range". This opens the Power Query editor.
In the advanced editor, I see something like:
let
Source = Excel.CurrentWorkbook(){[Name="tDetails"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"User ID", type text},
{"Company", type text},
{"State", type text},
{"Billed", type Int65.Type}})
in
#"Changed Type"
How do I go about querying this data? I'm needing to do things like:
SELECT 'UserID' = "User ID"
, 'Discount' = "Billed" * .51
, 'Note' = CASE WHEN State = 'NY' THEN 'Standard'
WHEN State = 'OH' THEN 'Second'
WHEN State = 'CA' THEN 'Third'
ELSE 'Fourth' END
FROM Source
WHERE "State" = "NY"
Also, can temp tables be used here?
SELECT 'UserID' = "User ID"
, 'Discount' = "Billed" * .51
INTO #UserDiscount
FROM Source
WHERE "State" = "NY"
SELECT * FROM #UserDiscount ORDER BY Discount DESC
These are super basic examples. What I really need to do is far more complex, but this would give me a place to start.
Thanks!
I think the equivalent Power Query would be
let
Source = Excel.CurrentWorkbook(){[Name="tDetails"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"User ID", type text},
{"Company", type text},
{"State", type text},
{"Billed", type Int65.Type}})
#"Filtered Rows" = Table.SelectRows(#"Changed Type", each ([State] = "NY")),
#"Added Custom" = Table.AddColumn(#"Filtered Rows", "Discount", each [Billed]*0.51),
#"Added Custom1" = Table.AddColumn(#"Added Custom", "UserID", each [UserID]),
#"Removed Other Columns" = Table.SelectColumns(#"Added Custom1",{"UserID", "Discount", "Billed"})
in
#"Removed Other Columns"
Also, the equivalent of SQL #temp table in Power Query would be use of table variable. In power query each step is table variable, e.g. Source, #"Changed Type", #"Filtered Rows", #"Added Custom".... which contains the table that was evaluated in that step. You can keep on utilizing the tables contained in each table variable in succeeding steps.
e.g.
let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WMlTSUcrPS1WK1YlWMgKyS8rzwWxjEDujKBUoEwsA", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Column1 = _t, Column2 = _t]),
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Column1", Int64.Type}, {"Column2", type text}}),
Custom0 = Table.SelectRows(#"Changed Type", each ([Column2] = "one")),
Custom1 = Table.SelectRows(#"Changed Type", each ([Column2] = "two")),
Custom2 = Table.SelectRows(#"Changed Type", each ([Column2] = "three")),
Custom3 = Custom0&Custom1&Custom2
in
Custom3
Yes, but you have to use a reserved function SqlExpression.ToExpression, and it doesn't look like your SQL is "valid" according to Power Query which likely needs in T-SQL format, so you'll have to experiment with how quotes are handled and columns renamed.
See this page for details: https://bengribaudo.com/blog/2021/07/13/5868/m-mysteries-sqlexpression-toexpression
Queries:
// tDetails
// Input from Excel range. This example uses embedded data so you can replicate my result.
let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WcspPUtJRcs7PK8kvzgey/CKBhKGRsYlSrE60kldqWhqKtL8HSNoADJRiYwE=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [#"User ID" = _t, Company = _t, State = _t, Billed = _t]),
#"Changed Type" = Table.TransformColumnTypes(Source,{{"User ID", type text}, {"Company", type text}, {"State", type text}, {"Billed", Int64.Type}})
in
#"Changed Type"
// Result
// Executes translated M code
let
ReferenceTables = [Source = tDetails],
Source = Expression.Evaluate(SQL, #shared)(ReferenceTables)
in
Source
// SQL
// Translates SQL to M
let
ReferenceTables = [Source = tDetails],
SqlQuery = "SELECT [User ID] as UserID, Billed * 0.51 as Discount, 'Standard' as [Note] FROM Source WHERE State = 'NY'",
TranslatedToM = SqlExpression.ToExpression(SqlQuery, ReferenceTables)
in
TranslatedToM
tDetails:
My SQL that works: SELECT [User ID] as UserID, Billed * 0.51 as Discount, 'Standard' as [Note] FROM Source WHERE State = 'NY'
Result:

Keeping the last row in a month and year for a particular group

I have a table which contains roughly 30 columns. One of the columns is "Date" and one is "Project Code". For each project code, there could be multiple entries in the table for a given month and year. For example, for project code "ABC" there could be multiple entries for February, 2020. Each would have a different date (i.e, 20200202, 20200209, 20200216, 20200223, 20200229)
In Power Query, I'd like to perform a transformation step that includes only those rows that represent the last entry for a month and year, for each project code. Using the example above, I want a single row for project "ABC", for February, 2020, and it would be for the date of 20200229.
I immediately looked at Group By but I don't think I'm going to be able to do what I want using Group By.
You need to do something along these lines
Assuming column Date is formatted as a date then ...
Add a column to pull out the month using
= Date.Month([Date])
Add a column to pull out the year using
= Date.Year([Date])
Add a column that finds the maximum date from all the rows with matching (Group, Month, Year)
= Table.AddColumn(#"PriorStep","MaxDate",(i)=>List.Max(Table.SelectRows(#"PriorStep", each [Project Code]=i[Project Code] and [Month]=i[Month] and [Year]=i[Year]) [Date]), type date )
Add a column to compare current date to that maximum date
=if [Date]=[MaxDate] then "keep" else "remove"
Remove the rows that don't match by filtering
Remove the extra columns
Sample code below
let Source = Excel.CurrentWorkbook(){[Name="Table5"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Project Code", type text}, {"Date", type date}}),
#"Added Custom2" = Table.AddColumn(#"Changed Type", "Month", each Date.Month([Date])),
#"Added Custom3" = Table.AddColumn(#"Added Custom2", "Year", each Date.Year([Date])),
#"Added Custom" = Table.AddColumn(#"Added Custom3","MaxDate",(i)=>List.Max(Table.SelectRows(#"Added Custom3", each [Project Code]=i[Project Code] and [Month]=i[Month] and [Year]=i[Year]) [Date]), type date ),
#"Added Custom1" = Table.AddColumn(#"Added Custom", "Match", each if [Date]=[MaxDate] then "keep" else "remove"),
#"Filtered Rows" = Table.SelectRows(#"Added Custom1", each ([Match] = "keep")),
#"Removed Columns" = Table.RemoveColumns(#"Filtered Rows",{"MaxDate", "Match","Month","Year"})
in #"Removed Columns"
With Table.Group:
let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("TcwxCsAwDEPRu3jOIMkxScf2GiH3v0Y9uDRgLY+P1zJByJM1u3MEbLdiEsVRyAvS/Nr4U2mUej8eOEvVzzZSntz0Q8FSDtv7BQ==", BinaryEncoding.Base64), Compression.Deflate)), {"Date","Project","SomeData"}),
types = Table.TransformColumnTypes(Source,{{"Date", Int64.Type}, {"Project", type text}, {"SomeData", Int64.Type}}),
year = Table.AddColumn(types, "year", each Number.RoundDown([Date]/10000)),
month = Table.AddColumn(year, "month", each Number.RoundDown(Number.Mod([Date],10000)/100)),
day = Table.AddColumn(month, "day", each Number.Mod([Date],100)),
group = Table.Group(day, {"Project", "year", "month"}, {"temp", each Table.FirstN(Table.Sort(_,{"day",1}),1)}),
combine = Table.Combine(group[temp])
in
combine
Initial data:
Result:

How to get a calculated column based on 2 different column in Power Query

Appreciate for any helps and suggestions.
I have a table structure as following:-
Date, Product Code ,Result ,Schedule
Day1, A ,0 ,0
Day2, A ,20 ,100
Day3, A ,200 ,100
How can i add a new column [Different] which reset by product code
Date ,Product Code ,Result ,Schedule ,Different
Day1 ,A ,0 ,0 ,0
Day2 ,A ,20 ,100 ,-80
Day3 ,A ,200 ,100 ,20
Where different = previous's different + result - schedule
thank you.
Paste code below in Home ... Advanced Editor, save, and name it fnRunningSum
It will create a function to do a cumultive running total on column named Amount
(MyTable as table) =>
let Source = Table.Buffer(MyTable),
MyColumn="Amount",
TableType = Value.Type(Table.AddColumn(Source, "Cumul", each null, type number)),
Cumulative = List.Skip(List.Accumulate(Table.Column(Source,MyColumn),{0},(cumulative,MyColumn) => cumulative & {List.Last(cumulative) + MyColumn})),
Cumu = Table.FromColumns(Table.ToColumns(Source)&{Cumulative},TableType)
in Cumu
Load your data into powerquery, here assumed to be in range Table1. Paste code below in Home ... Advanced Editor...
What it does is (1) add a new column that is Result-Schedule (2) Group on Product Code and cumulative sum the new column (3) Expand to get the columns back
letSource = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Added Custom" = Table.AddColumn(Source, "Amount", each [Result]-[Schedule]),
#"Grouped Rows" = Table.Group(#"Added Custom" , {"Product Code"}, {{"AllData", fnRunningSum}}),
#"Expanded AllData" = Table.ExpandTableColumn(#"Grouped Rows", "AllData", {"Date", "Result", "Schedule", "Cumul"}, {"Date", "Result", "Schedule", "Different"})
in #"Expanded AllData"
Try to achieve that with a Column From Examples, and type what you want to see in the new column. It usually works fine for me.
Documentation in case you have never used it before:
https://learn.microsoft.com/en-us/power-bi/desktop-add-column-from-example

Customized column power query editor - filters

I'm trying to add a column in the power query editor (using M as I want to show new column in table output) that should calculate following formula:
Value of record Column Market Value / Sum of Column Market Value
BUT this sum should only include rows where two other columns (in my case Date and Account number) are equal to the value in the record/row I want to calculate value for.
From what I can tell, you are trying to achieve the same as doing a SUMIFS() calculation in an excel table but in power query?
This can be done in a couple of ways, but the simplest is with power query's "Group By" function. With this method you use Table.Group() to group the data by your two columns (Date & Account Number) and get the sum for all records. Then all you need to do is merge this value to the original table by referencing the group by step. This is equivalent to using a pivot table then doing a VLOOKUP on both the Date and Account number column.
Here is the M code:
let
// You're previous code here (replace the "Source" & #"Changed Type" lines)
Source = Excel.CurrentWorkbook(){[Name="my_data"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Date", type date}, {"Account number", type text}, {"Market Value", type number}}),
// Group by Date & Account Number to sum each record
#"Calculate Record Value" = Table.Group(#"Changed Type", {"Date", "Account number"}, {{"Value of Record", each List.Sum([Market Value]), type number}}), // Replace #"Changed Type" in this line with the last step reference from your code
// Merge the group by value back to each record in the orginal table
#"Merged Record Value" = Table.NestedJoin(#"Changed Type",{"Date", "Account number"},#"Calculate Record Value",{"Date", "Account number"},"sum",JoinKind.LeftOuter),
// Expand your new column
#"Expanded Record Value" = Table.ExpandTableColumn(#"Merged Record Value", "sum", {"Value of Record"}, {"Value of Record"})
in
#"Expanded Record Value"
Hopefully this is helpful!
Please let me know if I have misunderstood your question or if this solution doesn't work for you.
Best regards
DougC

Add calculated custom column using entire table for calculation power query or dax

I have this table in Power BI with events concerning some objects
|new_state |object_id | created_at |
|new |1 |11/4/2015 1:50:48 PM |
|in_use |3 |11/4/2015 2:31:10 PM |
|in_use |1 |11/4/2015 2:31:22 PM |
|deleted |2 |11/4/2015 3:14:10 PM |
.....
I am trying to add a calculated column either in DAX or power query so that for each row I would have the previous_state of that object. From a logical point of view it's not difficult: you group by id and for each row in that group you look for the closest previous time and get the "new_state" which would represent the previous state for that row.
I have tried doing this by creating a function in power query and use it in a custom column but I am getting a "cyclic reference detected" error and cannot do it. Any ideas on solutions?
It's hard to express comparisons between rows today in Power Query. Most of the functions assume the table is just an unordered set of rows.
To expand on Oğuz's comment, you could add an index column, then add a column PreviousState indexing into the previous row (or null). As an optimization it might be much faster if you buffer the whole table first.
let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WqslLLY8vLkksSVWoyU/KSk0uic9MUahRSC5KBYqlxCeWKNQoxepAFCrUGAKRob6JvpGBoamCoZWpgZWJhUKAL0xNZl58aTHQJGNkZUZWxoZWhgZYlBliKDMyQlKWkpqTCnSDQo0RsjpjK0MThHGxAA==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type text) meta [Serialized.Text = true]) in type table [Column1 = _t]),
#"Split Column by Delimiter" = Table.SplitColumn(Source,"Column1",Splitter.SplitTextByDelimiter("|", QuoteStyle.Csv),{"Column1.1", "Column1.2", "Column1.3", "Column1.4", "Column1.5"}),
#"Removed Columns" = Table.RemoveColumns(#"Split Column by Delimiter",{"Column1.1", "Column1.5"}),
#"Trimmed Text" = Table.TransformColumns(#"Removed Columns",{},Text.Trim),
#"Promoted Headers" = Table.PromoteHeaders(#"Trimmed Text"),
ChangedType = Table.TransformColumnTypes(#"Promoted Headers",{{"object_id", Int64.Type}, {"created_at", type datetime}, {"new_state", type text}}),
#"Added Index" = Table.AddIndexColumn(ChangedType, "Index", 0, 1),
Buffer = Table.Buffer(#"Added Index"),
#"Added Custom" = Table.AddColumn(Buffer, "PreviousState", each try Buffer{[Index] - 1}[created_at] otherwise null),
#"Inserted Time Subtraction" = Table.AddColumn(#"Added Custom", "TimeDifference", each [created_at] - [PreviousState], type duration)
in
#"Inserted Time Subtraction"
There are surely neater solutions than this but in DAX you can create a calculated column (prevdate) to store the datetime of the previous entry:
=
CALCULATE (
MAX ( [created_at] ),
ALL ( table1 ),
Table1[created_at] < EARLIER ( [created_at] ),
Table1[object_id] = EARLIER ( [object_id] ) )
Then you add another calculated column to store the state at that previous time:
=
CALCULATE (
VALUES ( Table1[new_state] ),
ALL ( Table1 ),
Table1[created_at] = EARLIER ( Table1[prevdate] ),
Table1[object_id] = EARLIER ( Table1[object_id] )
)
I've solved it :D
#"Sorted Rows" = Table.Sort(#"Reordered Columns",{{"object_id", Order.Ascending}, {"created_at", Order.Ascending}}),
#"Added Index" = Table.AddIndexColumn(#"Sorted Rows", "Index", 0, 1),
Buffer = Table.Buffer(#"Added Index"),
#"Added Custom" = Table.AddColumn(Buffer, "PreviousState", each try (if Buffer{[Index] - 1}[object_id]=Buffer{[Index]}[object_id] then Buffer{[Index] - 1}[new_state] else null ) otherwise null)
I'm not sure it's not mostly a hack but it seems to be working. Do you see any point where it might fail in the future?

Resources