Duplicate row for each item in column - powerquery

I can do things in PowerQuery but I can't find how to achieve the following result:
Before:
After:
The goal is to duplicate the last row (filtered with Project Code=null) for each item in Project Code column. I think duplicating the row as is is important to keep the Metadata Table and expand it later.
Thank you very much for your help.

Try this
Grab all rows except null row
Get unique values of Project column in a list
Grab null row
Create row with the list and expand it
Put the two tables back together.
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
AllButNull=Table.SelectRows( Source, each ([Project Code] <> null)),
UniqueProjects=List.Distinct(AllButNull[Project Code]),
OnlyNull = Table.SelectRows(#"Added Custom", each ([Project Code] = null)),
#"Replaced Value" = Table.ReplaceValue(OnlyNull,null, UniqueProjects,Replacer.ReplaceValue,{"Project Code"}),
#"Expanded Project Code" = Table.ExpandListColumn(#"Replaced Value" , "Project Code"),
combined = AllButNull & #"Expanded Project Code"
in combined
Alternately, grab the last row instead of the null row:
Grab all rows except last
Get unique values of Project column in a list
Grab last row
Create row with the list and expand it
Put the two tables back together.
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
AllButLast=Table.RemoveLastN(Source ,1),
UniqueProjects=List.Distinct(AllButLast[Project Code]),
OnlyBottomRow = Table.LastN(#"Added Custom", 1),
#"Replaced Value" = Table.ReplaceValue(OnlyBottomRow,null, UniqueProjects,Replacer.ReplaceValue,{"Project Code"}),
#"Expanded Project Code" = Table.ExpandListColumn(#"Replaced Value", "Project Code"),
combined = AllButLast & #"Expanded Project Code"
in combined

Related

Power Query: Extracting value from nested lists

Hello hope someone can assist me in a power query I'm having trouble with. I'm brand new to Power Query and the M language and while I do have some coding background coding is not my day job. I'm pulling data from web page and the data in one column that is a list nested in a list.
This is a clip of what I see at from query initially:
I then drill down on the list and see this for all of the rows:
I then drill down again in that list and get a this:
At this level I get at least one row but there could be many rows.
What I want is to take all of the values and combine them into one cell as a bulleted list like this:
Any assistance on how to do this would be appreciated
I've tried looking at some of the examples in other threads and only get errors when I do this.
You didn't really provide enough detail here, but it looks like a bunch of lists within lists
you can run them through something like this to expand them all. If the results dont look like what you want, provide more information and sample data we can reproduce
let Source = <<copy whatever your source is here>>,
//Marcel Beug 2017
TableSchema = Table.Schema(Source),
ColumnNames = Table.SelectColumns(TableSchema,{"Name"}),
IsListColumn = Table.AddColumn(ColumnNames, "IsListColumn?", each List.AllTrue(List.Transform(Table.Column(Source,[Name]), each _ is list))),
NonListColumns = Table.SelectRows(IsListColumn, each ([#"IsListColumn?"] = false)),
NonListColumnNames = Table.RemoveColumns(NonListColumns,{"IsListColumn?"})[Name],
SelectNonListColumns = Table.SelectColumns(Source,NonListColumnNames),
ListColumns = Table.SelectRows(IsListColumn, each ([#"IsListColumn?"] = true)),
ListColumnNames = Table.RemoveColumns(ListColumns,{"IsListColumn?"})[Name],
SelectListColumns = Table.SelectColumns(Source,ListColumnNames),
TableFromLists = Table.AddColumn(SelectListColumns, "TableFromLists", each Table.FromColumns(Record.FieldValues(_))),
ListTables = Table.SelectColumns(TableFromLists,{"TableFromLists"}),
Custom1 = Table.FromColumns({Table.ToRecords(SelectNonListColumns),Table.ToRecords(ListTables)}),
#"Expanded Column1" = Table.ExpandRecordColumn(Custom1, "Column1", Table.ColumnNames(#table(List.Min({1,List.Count(NonListColumnNames)}),{})), NonListColumnNames),
#"Expanded Column2" = Table.ExpandRecordColumn(#"Expanded Column1", "Column2", {"TableFromLists"}, {"TableFromLists"}),
#"Expanded TableFromLists" = Table.ExpandTableColumn(#"Expanded Column2", "TableFromLists", Table.ColumnNames(#table(List.Count(ListColumnNames),{})), ListColumnNames),
#"Reordered Columns" = Table.ReorderColumns(#"Expanded TableFromLists",ColumnNames[Name])
in #"Reordered Columns"
EDIT for specific website clarification
let Source = Web.Page(Web.Contents("https://ised-isde.canada.ca/site/high-speed-internet-canada/en/universal-broadband-fund/selected-universal-broadband-fund-projects")),
#"Expanded Data" = Table.ExpandTableColumn(Source, "Data", {"Location of project", "Number of Households to be served / Number of kilometers to be covered (mobile projects)", "Funding recipient", "Funding amountFootnote *"}, {"Location of project", "Number of Households to be served / Number of kilometers to be covered (mobile p", "Funding recipient", "Funding amountFootnote *"}),
#"Added Custom" = Table.AddColumn(#"Expanded Data", "Location of Project2", each Text.Combine([Location of project]{1},"#(lf)")),
#"Removed Columns" = Table.RemoveColumns(#"Added Custom",{"Source", "ClassName", "Id", "Location of project"}),
#"Filtered Rows" = Table.SelectRows(#"Removed Columns", each ([Caption] <> "Document")),
#"FundingToAmount" = Table.TransformColumns(#"Filtered Rows",{{"Funding amountFootnote *", each Number.From(Text.Select(_,{"0".."9",".","$"})), type number}})
in #"FundingToAmount"
then in excel format that column as text control [x] wrap text

Filter rows where all matching rows are True

I have a table formatted as follows:
BOM
Imported
COM123
True
COM123
True
COM123
False
COM999
True
COM999
True
COM999
True
I'd like to filter the table to show only rows where all matching BOM rows are True in the Imported column. I.e., in this case, COM999 rows should show, and COM123 rows are filtered out because one entry is False.
FYI this is to produce a list of Production BOMs where all the components already exist in Business Central. The Imported column is the result from a merge of the query with an extract from BC, and sets the value to true where the BOM components exist.
Can anyone please give me a steer?
I've been farting around with this half the day, but I can't find the equivalent to a EXISTS query in SQL...
Algorithm should be clear in the code comments
Group by BOM
Test each BOM subgroup for ALLTRUE
Filter the Grouped table
Re-expand the original
let
//change next line to reflect actual data source
Source = Excel.CurrentWorkbook(){[Name="Table21"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"BOM", type text}, {"Imported", type logical}}),
//Group rows by BOM
//Then determine where all "Imported" = True
#"Grouped Rows" = Table.Group(#"Changed Type", {"BOM"}, {
{"all true", each List.AllTrue([Imported]), type logical},
{"all", each _, type table [BOM=nullable text, Imported=nullable logical]}}),
//Remove the false rows
//then delete the "all true" column
#"Filtered Rows" = Table.SelectRows(#"Grouped Rows", each ([all true] = true)),
#"Removed Columns" = Table.RemoveColumns(#"Filtered Rows",{"all true"}),
//re-expand
#"Expanded all" = Table.ExpandTableColumn(#"Removed Columns", "all", {"Imported"}, {"Imported"})
in
#"Expanded all"

Power query, iterate over the column records to apply a custom cumulative calculation

Using Power Query in Excel. I am trying to implement a custom column that would iteratively calculate the row based on the previous row's value of the same column.
I have a 3 column table and the 4th column will be the calculation column that I am failing to implement.
The calculation is very easy to apply in Excel which goes as follows:
Formula in cell D3 --> = =IF(A3=1,C3+6.4,IF(C3+D2>=12.8,12.8,IF(C3+D2<=1.28,1.28,C3+D2)))
The same formula is applied to the whole column by dragging.
The idea behind it:
For each category, I have an index column starting from 1,
If Index = 1, then Calculation is Value + 6.4,
else if Value + Value(previous row Custom cumulative) >= 12.8 then 12.8
else if Value + Value(previous row Custom cumulative) <= 1.28 then 1.28
else Value + Value(previous row Custom cumulative)
So, the calculation is a cumulative sum with an upper and lower cap built into it.
How can I implement this in Power Query and M-Language?
I really appreciate your help!
I have tried to use List.Generate and List.Accumulate features, however, I was stuck with creating records that has values from multiple columns in it.
Try this
(edited to make more efficient with single pass process)
let Source = Excel.CurrentWorkbook(){[Name="Table15"]}[Content],
process = (zzz as list) => let x= List.Accumulate( zzz,{0},( state, current ) =>
if List.Last(state) =0 then List.Combine ({state,{6.4+current}}) else
if List.Last(state)+current >=12.8 then List.Combine ({state,{12.8}}) else
if List.Last(state)+current <=1.28 then List.Combine ({state,{1.28}}) else
List.Combine ({state,{List.Last(state)+current}})
) in x,
#"Grouped Rows" = Table.Group(Source, {"Category"}, {{"data", each
let a=process(_[Values])
in Table.AddColumn(_, "Custom Cumulative", each a{[Index]}), type table }}),
#"Expanded data" = Table.ExpandTableColumn(#"Grouped Rows", "data", {"Index", "Values", "Custom Cumulative"}, {"Index", "Values", "Custom Cumulative"})
in #"Expanded data"

Power Query - best way to sub select?

Suppose I have a column representing object type and another column representing object color. I want to remove blue and red fruits (example of object type) but keep all other red and blue objects.
How can I acheive this in Power Query ?
Thanks,
Just (un)select (not) matching rows
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
Filtered = Table.SelectRows(Source, each not ([ObjectType] = "Fruit" and ([ObjectColor]="Red" or [ObjectColor]="Blue")))
in
Filtered
Here's one way:
If you start with this:
You can merge the two columns together like this:
Then filter out the "Fruit,Blue" and "Fruit,Red":
Which yields this:
And you can then delete the "Merged" column to get this:
Here's the M code:
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"ObjectType", type text}, {"ObjectColor", type text}}),
#"Inserted Merged Column" = Table.AddColumn(#"Changed Type", "Merged", each Text.Combine({[ObjectType], [ObjectColor]}, ","), type text),
#"Filtered Rows" = Table.SelectRows(#"Inserted Merged Column", each ([Merged] <> "Fruit,Blue" and [Merged] <> "Fruit,Red")),
#"Removed Columns" = Table.RemoveColumns(#"Filtered Rows",{"Merged"})
in
#"Removed Columns"

Power Query Parameter works in one table but not another

So I have two tables (power query), and want to combine them into one. The second table just looks at the first table (power query) and applies a parameter filter to it. When i try to combine the parameter code into the original query the filter doesn't work. I have enabled fast combine to made all queries public to get rid of any firewall issues.
So as not to break the original working set of pq, i duplicated the first pq and modified using advanced by coping the needed code to apply the parameter (third pq)
Second power query code (this looks at first pq an applies a parameter filter) and it works
let
Date_Parameter = Excel.CurrentWorkbook(){[Name="Parameter"]}[Content],
Date_Value = Date_Parameter{0}[Value],
Source = Excel.CurrentWorkbook(){[Name="Timesheet1"]}[Content],
#"Filtered Rows" = Table.SelectRows(Source, each ([Date] = Date_Value))
in
#"Filtered Rows"
Third power query code (this is the one where i duplicated the first pq and added parameter code from second pq) this doesn't work
let
Date_Parameter = Excel.CurrentWorkbook(){[Name="Parameter"]}[Content],
Date_Value = Date_Parameter{0}[Value],
Source = Excel.Workbook(File.Contents("\\192.168.12.31\Project Files\Daily Truck Sheet\TimeTrack\TimeTrack.xlsm")),
Timesheet_Table = Source{[Item="Timesheet",Kind="Table"]}[Data],
Merge = Table.NestedJoin(Timesheet_Table,{"Ref"},Project,{"Ref"},"NewColumn"),
#"Expand NewColumn" = Table.ExpandTableColumn(Merge, "NewColumn", {"Crew"}, {"NewColumn.Crew"}),
#"Renamed Columns" = Table.RenameColumns(#"Expand NewColumn",{{"NewColumn.Crew", "Crew"}}),
#"Removed Duplicates" = Table.Distinct(#"Renamed Columns", {"Ref"}),
#"Removed Columns" = Table.RemoveColumns(#"Removed Duplicates",{"Ref", "Employee Name", "Truck #", "Hours", "Per Diem", "Piecework", "Travel Day", "Timecard Filename", "Paid DT Hrs.", "hours check", "project hours", "Paid Regular Hours", "Paid OT Hrs.", "PayPeriod", "Employee Number", "Lead Hand Employee Number", "Crew Count", "Employee Revenue"}),
#"Reordered Columns"= Table.ReorderColumns(#"Removed Columns",{"Date", "Date Received", "Lead Hand", "Crew", "Project#", "Comments", "Work Performed", "time card hours", "Revenue per hour", "Total Reveneu"}),
Rounding = Table.TransformColumns(#"Reordered Columns",{{"Revenue per hour", each Number.Round(_, 2)}, {"Total Reveneu", each Number.Round(_, 2)}}),
#"Filtered Rows" = Table.SelectRows(Rounding, each ([Date] = Date_Value))
in
#"Filtered Rows"
so i had to insert a transform for pq to treat as a date. Even though in the Parameter pq (that loads the value from the parameter table) it is already transformed. replaced the first three lines before the source line with the following and it worked
Date_Parameter = Excel.CurrentWorkbook(){[Name="Parameter"]}[Content],
#"Changed Type1" = Table.TransformColumnTypes(Date_Parameter,{{"Value", type date}}),
Date_Value = #"Changed Type1"{0}[Value],
So maybe now i can get rid of the parameter pq as it is all built into the final pq but haven't tried yet

Resources