My data has invoiced rental with a start date and end date, which more often than not overlaps our fiscal periods. I used the function List.Dates to create records for each date between the start and end dates, which worked great. When trying to merge the data to get the fiscal periods for each new record, I lose all the listed dates except for the first one. Here is the advanced editor info:
let
Source = Covid19,
#"Removed Columns" = Table.RemoveColumns(Source,{"DTTRANS", "NOPRODUIT", "DSLIGNE", "QTEXP", "PXVENDANT", "MTLIGNE", "DTDEB", "DTFIN", "Location", "Tableau1.Nocardex"}),
#"Reordered Columns" = Table.ReorderColumns(#"Removed Columns",{"NoCardex", "COMLOC", "Facture", "JoursAjustés", "DateDébut", "DateFin", "ParJour"}),
#"Grouped Rows" = Table.Group(#"Reordered Columns", {"NoCardex", "COMLOC", "Facture", "JoursAjustés", "DateDébut", "DateFin"}, {{"LocationParJour", each List.Sum([ParJour]), type number}}),
#"Added Custom" = Table.AddColumn(#"Grouped Rows", "Journee", each List.Dates([DateDébut],[JoursAjustés],#duration(1, 0, 0, 0))),
#"Expanded {0}" = Table.ExpandListColumn(#"Added Custom", "Journee"),
#"Changed Type" = Table.TransformColumnTypes(#"Expanded {0}",{{"Journee", type date}}),
#"Removed Columns1" = Table.RemoveColumns(#"Changed Type",{"JoursAjustés", "DateDébut", "DateFin"}),
#"Merged Queries" = Table.NestedJoin(#"Removed Columns1", {"Journee"}, PériodesFiscales, {"DateTrans"}, "PériodesFiscales", JoinKind.LeftOuter),
#"Expanded {0}1" = Table.ExpandTableColumn(#"Merged Queries", "PériodesFiscales", {"Produit"}, {"PériodesFiscales.Produit"})
in
#"Expanded {0}1"
I am puzzled as to why I lose the dates. I am sure it is triviial. Hoping someone can help me figure this one out
Ok, this is a bit embarrassing. I found out it had nothing to do with the expanded List.Dates. The merge changed the order of records. I found out after pasting a 1000 records onto a spreasheet to recreate the merge in Power Query without the expanded List.Dates. Turns out that the merge changed the sort on the orignal record set. Sorry. :-)
Related
Can Power query do this?
So I have a group of parent IDs. If the parent Ids are the same but the values from the corresponding attributes are different, I want PQ to let me know they can be grouped together.
Here is the example.
So Parent IDs 12345 are the same, and the values are different, I want the output to say SDSKU..Yes Then if the Parent IDs 333 are the same and values are the same, then that will not be a grouping and I want it to say NO. See image link
If you mean by "values" the values of the column "Color", try the M code below :
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Parent ID", Int64.Type}, {"Kitchen Sink", Int64.Type}, {"Color", type text}}),
#"Grouped Rows" = Table.Group(#"Changed Type", {"Parent ID", "Kitchen Sink"}, {{"AllData", each _, type table [Parent ID=nullable number, Kitchen Sink=nullable number, Color=nullable text]}, {"OccuID", each Table.RowCount(_), Int64.Type}}),
#"Added Custom" = Table.AddColumn(#"Grouped Rows", "NumberOfColors", each List.Count(List.Distinct([AllData][Color]))),
#"Added Custom1" = Table.AddColumn(#"Added Custom", "SDSKU", each if [OccuID] = [NumberOfColors] then "Yes" else "No"),
#"Expanded AllData" = Table.ExpandTableColumn(#"Added Custom1", "AllData", {"Kitchen Sink", "Color"}, {"Kitchen Sink.1", "Color"}),
#"Removed Columns" = Table.RemoveColumns(#"Expanded AllData",{"OccuID", "NumberOfColors"})
in
#"Removed Columns"
If "attributes" are the value of every column except the one named Parent ID, try the M code below :
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Grouped Rows" = Table.Group(Source , {"Parent ID"}, {
{"data", each _, type table },
{"check", each if Table.RowCount(_) = Table.RowCount(Table.Distinct(_, List.Difference(Table.ColumnNames(_),{"Parent ID"}))) then "YES" else "NO"}}),
#"Expanded data" = Table.ExpandTableColumn(#"Grouped Rows", "data", List.Difference(Table.ColumnNames(Source),{"Parent ID"}), List.Difference(Table.ColumnNames(Source),{"Parent ID"}))
in #"Expanded data"
I frequently have large datasets in powerquery where I need to remove/filter out the same row, as well as the following 13 whenever a certain value, in this case "Page" occurs. This occurs multiple times throughout the column.
I've tried referring to the next/previous rows by adding an index column and {[Index]+1} shenanigans but that either didn't work or took 15+ minutes to load.
I've tried setting up something with Table.RemoveFirstN(Text.Contains([Column], "Page"), 13) but that just errored out.
Would anyone know how I could filter the row where a value occurs, as well as the next n rows (index?) in Powerquery?
Kind regards,
This seems to work ok
We add an index. Test for "Page". In a new column, if Page is present, copy over the index. Fill down then group on that. Add 2nd index to the grouping. Expand all columns. Filter out anything where 2nd index is <14. Remove extra columns
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Merged Price Country", type text}}),
#"Added Index" = Table.AddIndexColumn(#"Changed Type", "Index", 1, 1),
#"Added Custom" = Table.AddColumn(#"Added Index", "Custom", each try if Text.Contains([Merged Price Country],"Page") then [Index] else null otherwise null),
#"Filled Down" = Table.FillDown(#"Added Custom",{"Custom"}),
mGroup = Table.Group(#"Filled Down", {"Custom"}, {{"Data", each Table.AddIndexColumn(_, "Index2", 1, 1), type table}}),
#"Removed Columns" = Table.RemoveColumns(mGroup,{"Custom"}),
// expand all columns
List = List.Union(List.Transform(#"Removed Columns"[Data], each Table.ColumnNames(_))),
#"Expanded Data" = Table.ExpandTableColumn(#"Removed Columns", "Data", List,List),
#"Filtered Rows" = Table.SelectRows(#"Expanded Data", each [Custom]=null or [Index2] > 14),
#"Removed Columns1" = Table.RemoveColumns(#"Filtered Rows",{"Index", "Custom", "Index2"})
in #"Removed Columns1"
I skipped out on using Table.RemoveFirstN() on the groupings in code above case there are leading rows you want to keep, but you could use that instead of adding the 2nd index and filtering like below
let Source = Excel.CurrentWorkbook(){[Name="Table3"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Merged Price Country", type text}}),
#"Added Index" = Table.AddIndexColumn(#"Changed Type", "Index", 1, 1),
#"Added Custom" = Table.AddColumn(#"Added Index", "Custom", each try if Text.Contains([Merged Price Country],"Page") then [Index] else null otherwise null),
#"Filled Down" = Table.FillDown(#"Added Custom",{"Custom"}),
mGroup = Table.Group(#"Filled Down", {"Custom"}, {{"Data", each Table.RemoveFirstN(_, 13), type table}}),
#"Removed Columns" = Table.RemoveColumns(mGroup,{"Custom"}),
// expand all columns
List = List.Union(List.Transform(#"Removed Columns"[Data], each Table.ColumnNames(_))),
#"Expanded Data" = Table.ExpandTableColumn(#"Removed Columns", "Data", List,List),
#"Removed Columns1" = Table.RemoveColumns(#"Expanded Data",{"Index", "Custom"})
in #"Removed Columns1"
Different approach. Wonder which might be faster:
Create a list of rows to be removed (by row number)
Select the rows not in that list
let
Source = Excel.CurrentWorkbook(){[Name="Table12"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Text", type text}, {"Data", Int64.Type}}),
//Add index column
#"Added Index" = Table.AddIndexColumn(#"Changed Type", "Index", 0, 1, Int64.Type),
//create list rows to be removed
textCol = List.Transform(#"Added Index"[Text], each
if _ = null then null
else if Text.Contains(_,"Page",Comparer.OrdinalIgnoreCase) then "RemoveMe"
else _),
//create list of positions to be removed
removePos = List.Combine(List.Transform(List.PositionOf(textCol,"RemoveMe",Occurrence.All), each {_..List.Min({_+13, List.Count(textCol)})})),
//Filter the table using the "RemoveMe" list
filter = Table.SelectRows(#"Added Index", each not List.Contains(removePos,[Index])),
#"Removed Columns" = Table.RemoveColumns(filter,{"Index"})
in
#"Removed Columns"
I had a formula in a table in excel
=IF([#STATUS]="",[KEY]&"_"&COUNTIF(INDEX([KEY],1):[#KEY],[#KEY]),"")
which showed me how often a value showed in the data. But the same is not working in Power Query
with the formula I use to get if the same value's position in a long data list, and then I use the same in index match formula to find and locate other relevant data
I am trying to achieve:
Date Name Frequency
1/10/2019 Adrian Bartholomeusz 1
1/10/2019 Aditya Tipnis 1
2/10/2019 Abdul Atef 1
2/10/2019 Aditya Tipnis 2
3/10/2019 Abdul Atef 2
In excel I used the formula "=COUNTIF(INDEX([Name],1):[#Name],[#Name])" but when I use the same in Power Query I am getting error
The key steps are:
Add Index
Group Rows
Transform Columns to add a sub-index.
Expand the data back.
The rest are cosmetics.
let
Source = Excel.CurrentWorkbook(),
Table1 = Source{[Name="Table1"]}[Content],
#"Added Index" = Table.AddIndexColumn(Table1, "Index", 0, 1),
#"Grouped Rows" = Table.Group(#"Added Index", {"key"}, {{"Data", each _, type table [key=number, f=text, Index=number]}}),
#"TransformColumns" = Table.TransformColumns(#"Grouped Rows",{"Data", (x) => Table.AddIndexColumn(x, "Index2", 1, 1)}),
#"Expanded Data" = Table.ExpandTableColumn(#"TransformColumns", "Data", {"excel formula", "Index", "Index2"}, {"excel formula", "Index", "Index2"}),
#"Added Custom" = Table.AddColumn(#"Expanded Data", "PQ method", each Text.From([key]) & "_" & Text.From([Index2])),
#"Sorted Rows" = Table.Sort(#"Added Custom",{{"Index", Order.Ascending}}),
#"Removed Columns" = Table.RemoveColumns(#"Sorted Rows",{"Index", "Index2"})
in
#"Removed Columns"
I have power query in MS excel 2016, I order data by name, but I have accented letters š, č, ... which are now sorted to the end of dataset but should be for example š after s or č after c. Is it possible how to make some workaround here? I guess maybe change encoding, but I can't find how.
#"Sorted Rows" = Table.Sort(#"Renamed Columns",{{{"Name", Order.Ascending}})
The best way I can think of to do this is to create a calculated column where you replace those special values and then sort on that column.
#"Added Custom" = Table.AddColumn(#"Renamed Columns", "Custom", each Text.Replace(Text.Replace([Name],"š","sz"),"č","cz")),
#"Sorted Rows" = Table.Sort("Added Custom",{{{"Custom", Order.Ascending}})
Once you've sorted, then you can delete that column.
I used your logic Alexis but add one step - lowercased the column. Also I replaced more values. So I'm posting it in case somebody is interested. Thanks a lot Alexis!
#"Added Custom2" = Table.AddColumn(#"Renamed Columns", "Custom", each [Name]),
#"Lowercased Text" = Table.TransformColumns(#"Added Custom2",{{"Custom", Text.Lower, type text}}),
#"Replaced Value" = Table.ReplaceValue(Table.ReplaceValue(Table.ReplaceValue(Table.ReplaceValue(Table.ReplaceValue(Table.ReplaceValue(Table.ReplaceValue(Table.ReplaceValue(Table.ReplaceValue(Table.ReplaceValue(Table.ReplaceValue(Table.ReplaceValue(Table.ReplaceValue(Table.ReplaceValue(Table.ReplaceValue(#"Lowercased Text","á","az",Replacer.ReplaceText,{"Custom"}),"č","cz",Replacer.ReplaceText,{"Custom"}),"ď","dz",Replacer.ReplaceText,{"Custom"}), "é","ez",Replacer.ReplaceText,{"Custom"}), "ě","ez",Replacer.ReplaceText,{"Custom"}), "í","iz",Replacer.ReplaceText,{"Custom"}), "ň","nz",Replacer.ReplaceText,{"Custom"}), "ó","oz",Replacer.ReplaceText,{"Custom"}), "ř","rz",Replacer.ReplaceText,{"Custom"}), "š","sz",Replacer.ReplaceText,{"Custom"}), "ť","tz",Replacer.ReplaceText,{"Custom"}), "ú","uz",Replacer.ReplaceText,{"Custom"}), "ů","uz",Replacer.ReplaceText,{"Custom"}), "ý","yz",Replacer.ReplaceText,{"Custom"}), "ž","zz",Replacer.ReplaceText,{"Custom"}),
#"Sorted Rows" = Table.Sort(#"Replaced Value",{{"Custom", Order.Ascending}}),
#"Removed Columns" = Table.RemoveColumns(#"Sorted Rows",{"Custom"})
So I have two tables (power query), and want to combine them into one. The second table just looks at the first table (power query) and applies a parameter filter to it. When i try to combine the parameter code into the original query the filter doesn't work. I have enabled fast combine to made all queries public to get rid of any firewall issues.
So as not to break the original working set of pq, i duplicated the first pq and modified using advanced by coping the needed code to apply the parameter (third pq)
Second power query code (this looks at first pq an applies a parameter filter) and it works
let
Date_Parameter = Excel.CurrentWorkbook(){[Name="Parameter"]}[Content],
Date_Value = Date_Parameter{0}[Value],
Source = Excel.CurrentWorkbook(){[Name="Timesheet1"]}[Content],
#"Filtered Rows" = Table.SelectRows(Source, each ([Date] = Date_Value))
in
#"Filtered Rows"
Third power query code (this is the one where i duplicated the first pq and added parameter code from second pq) this doesn't work
let
Date_Parameter = Excel.CurrentWorkbook(){[Name="Parameter"]}[Content],
Date_Value = Date_Parameter{0}[Value],
Source = Excel.Workbook(File.Contents("\\192.168.12.31\Project Files\Daily Truck Sheet\TimeTrack\TimeTrack.xlsm")),
Timesheet_Table = Source{[Item="Timesheet",Kind="Table"]}[Data],
Merge = Table.NestedJoin(Timesheet_Table,{"Ref"},Project,{"Ref"},"NewColumn"),
#"Expand NewColumn" = Table.ExpandTableColumn(Merge, "NewColumn", {"Crew"}, {"NewColumn.Crew"}),
#"Renamed Columns" = Table.RenameColumns(#"Expand NewColumn",{{"NewColumn.Crew", "Crew"}}),
#"Removed Duplicates" = Table.Distinct(#"Renamed Columns", {"Ref"}),
#"Removed Columns" = Table.RemoveColumns(#"Removed Duplicates",{"Ref", "Employee Name", "Truck #", "Hours", "Per Diem", "Piecework", "Travel Day", "Timecard Filename", "Paid DT Hrs.", "hours check", "project hours", "Paid Regular Hours", "Paid OT Hrs.", "PayPeriod", "Employee Number", "Lead Hand Employee Number", "Crew Count", "Employee Revenue"}),
#"Reordered Columns"= Table.ReorderColumns(#"Removed Columns",{"Date", "Date Received", "Lead Hand", "Crew", "Project#", "Comments", "Work Performed", "time card hours", "Revenue per hour", "Total Reveneu"}),
Rounding = Table.TransformColumns(#"Reordered Columns",{{"Revenue per hour", each Number.Round(_, 2)}, {"Total Reveneu", each Number.Round(_, 2)}}),
#"Filtered Rows" = Table.SelectRows(Rounding, each ([Date] = Date_Value))
in
#"Filtered Rows"
so i had to insert a transform for pq to treat as a date. Even though in the Parameter pq (that loads the value from the parameter table) it is already transformed. replaced the first three lines before the source line with the following and it worked
Date_Parameter = Excel.CurrentWorkbook(){[Name="Parameter"]}[Content],
#"Changed Type1" = Table.TransformColumnTypes(Date_Parameter,{{"Value", type date}}),
Date_Value = #"Changed Type1"{0}[Value],
So maybe now i can get rid of the parameter pq as it is all built into the final pq but haven't tried yet