Power Query: Extracting value from nested lists - powerquery

Hello hope someone can assist me in a power query I'm having trouble with. I'm brand new to Power Query and the M language and while I do have some coding background coding is not my day job. I'm pulling data from web page and the data in one column that is a list nested in a list.
This is a clip of what I see at from query initially:
I then drill down on the list and see this for all of the rows:
I then drill down again in that list and get a this:
At this level I get at least one row but there could be many rows.
What I want is to take all of the values and combine them into one cell as a bulleted list like this:
Any assistance on how to do this would be appreciated
I've tried looking at some of the examples in other threads and only get errors when I do this.

You didn't really provide enough detail here, but it looks like a bunch of lists within lists
you can run them through something like this to expand them all. If the results dont look like what you want, provide more information and sample data we can reproduce
let Source = <<copy whatever your source is here>>,
//Marcel Beug 2017
TableSchema = Table.Schema(Source),
ColumnNames = Table.SelectColumns(TableSchema,{"Name"}),
IsListColumn = Table.AddColumn(ColumnNames, "IsListColumn?", each List.AllTrue(List.Transform(Table.Column(Source,[Name]), each _ is list))),
NonListColumns = Table.SelectRows(IsListColumn, each ([#"IsListColumn?"] = false)),
NonListColumnNames = Table.RemoveColumns(NonListColumns,{"IsListColumn?"})[Name],
SelectNonListColumns = Table.SelectColumns(Source,NonListColumnNames),
ListColumns = Table.SelectRows(IsListColumn, each ([#"IsListColumn?"] = true)),
ListColumnNames = Table.RemoveColumns(ListColumns,{"IsListColumn?"})[Name],
SelectListColumns = Table.SelectColumns(Source,ListColumnNames),
TableFromLists = Table.AddColumn(SelectListColumns, "TableFromLists", each Table.FromColumns(Record.FieldValues(_))),
ListTables = Table.SelectColumns(TableFromLists,{"TableFromLists"}),
Custom1 = Table.FromColumns({Table.ToRecords(SelectNonListColumns),Table.ToRecords(ListTables)}),
#"Expanded Column1" = Table.ExpandRecordColumn(Custom1, "Column1", Table.ColumnNames(#table(List.Min({1,List.Count(NonListColumnNames)}),{})), NonListColumnNames),
#"Expanded Column2" = Table.ExpandRecordColumn(#"Expanded Column1", "Column2", {"TableFromLists"}, {"TableFromLists"}),
#"Expanded TableFromLists" = Table.ExpandTableColumn(#"Expanded Column2", "TableFromLists", Table.ColumnNames(#table(List.Count(ListColumnNames),{})), ListColumnNames),
#"Reordered Columns" = Table.ReorderColumns(#"Expanded TableFromLists",ColumnNames[Name])
in #"Reordered Columns"
EDIT for specific website clarification
let Source = Web.Page(Web.Contents("https://ised-isde.canada.ca/site/high-speed-internet-canada/en/universal-broadband-fund/selected-universal-broadband-fund-projects")),
#"Expanded Data" = Table.ExpandTableColumn(Source, "Data", {"Location of project", "Number of Households to be served / Number of kilometers to be covered (mobile projects)", "Funding recipient", "Funding amountFootnote *"}, {"Location of project", "Number of Households to be served / Number of kilometers to be covered (mobile p", "Funding recipient", "Funding amountFootnote *"}),
#"Added Custom" = Table.AddColumn(#"Expanded Data", "Location of Project2", each Text.Combine([Location of project]{1},"#(lf)")),
#"Removed Columns" = Table.RemoveColumns(#"Added Custom",{"Source", "ClassName", "Id", "Location of project"}),
#"Filtered Rows" = Table.SelectRows(#"Removed Columns", each ([Caption] <> "Document")),
#"FundingToAmount" = Table.TransformColumns(#"Filtered Rows",{{"Funding amountFootnote *", each Number.From(Text.Select(_,{"0".."9",".","$"})), type number}})
in #"FundingToAmount"
then in excel format that column as text control [x] wrap text

Related

Expanded column From List.Dates contracts after a subsequent merge

My data has invoiced rental with a start date and end date, which more often than not overlaps our fiscal periods. I used the function List.Dates to create records for each date between the start and end dates, which worked great. When trying to merge the data to get the fiscal periods for each new record, I lose all the listed dates except for the first one. Here is the advanced editor info:
let
Source = Covid19,
#"Removed Columns" = Table.RemoveColumns(Source,{"DTTRANS", "NOPRODUIT", "DSLIGNE", "QTEXP", "PXVENDANT", "MTLIGNE", "DTDEB", "DTFIN", "Location", "Tableau1.Nocardex"}),
#"Reordered Columns" = Table.ReorderColumns(#"Removed Columns",{"NoCardex", "COMLOC", "Facture", "JoursAjustés", "DateDébut", "DateFin", "ParJour"}),
#"Grouped Rows" = Table.Group(#"Reordered Columns", {"NoCardex", "COMLOC", "Facture", "JoursAjustés", "DateDébut", "DateFin"}, {{"LocationParJour", each List.Sum([ParJour]), type number}}),
#"Added Custom" = Table.AddColumn(#"Grouped Rows", "Journee", each List.Dates([DateDébut],[JoursAjustés],#duration(1, 0, 0, 0))),
#"Expanded {0}" = Table.ExpandListColumn(#"Added Custom", "Journee"),
#"Changed Type" = Table.TransformColumnTypes(#"Expanded {0}",{{"Journee", type date}}),
#"Removed Columns1" = Table.RemoveColumns(#"Changed Type",{"JoursAjustés", "DateDébut", "DateFin"}),
#"Merged Queries" = Table.NestedJoin(#"Removed Columns1", {"Journee"}, PériodesFiscales, {"DateTrans"}, "PériodesFiscales", JoinKind.LeftOuter),
#"Expanded {0}1" = Table.ExpandTableColumn(#"Merged Queries", "PériodesFiscales", {"Produit"}, {"PériodesFiscales.Produit"})
in
#"Expanded {0}1"
I am puzzled as to why I lose the dates. I am sure it is triviial. Hoping someone can help me figure this one out
Ok, this is a bit embarrassing. I found out it had nothing to do with the expanded List.Dates. The merge changed the order of records. I found out after pasting a 1000 records onto a spreasheet to recreate the merge in Power Query without the expanded List.Dates. Turns out that the merge changed the sort on the orignal record set. Sorry. :-)

countif formula in power query

I had a formula in a table in excel
=IF([#STATUS]="",[KEY]&"_"&COUNTIF(INDEX([KEY],1):[#KEY],[#KEY]),"")
which showed me how often a value showed in the data. But the same is not working in Power Query
with the formula I use to get if the same value's position in a long data list, and then I use the same in index match formula to find and locate other relevant data
I am trying to achieve:
Date Name Frequency
1/10/2019 Adrian Bartholomeusz 1
1/10/2019 Aditya Tipnis 1
2/10/2019 Abdul Atef 1
2/10/2019 Aditya Tipnis 2
3/10/2019 Abdul Atef 2
In excel I used the formula "=COUNTIF(INDEX([Name],1):[#Name],[#Name])" but when I use the same in Power Query I am getting error
The key steps are:
Add Index
Group Rows
Transform Columns to add a sub-index.
Expand the data back.
The rest are cosmetics.
let
Source = Excel.CurrentWorkbook(),
Table1 = Source{[Name="Table1"]}[Content],
#"Added Index" = Table.AddIndexColumn(Table1, "Index", 0, 1),
#"Grouped Rows" = Table.Group(#"Added Index", {"key"}, {{"Data", each _, type table [key=number, f=text, Index=number]}}),
#"TransformColumns" = Table.TransformColumns(#"Grouped Rows",{"Data", (x) => Table.AddIndexColumn(x, "Index2", 1, 1)}),
#"Expanded Data" = Table.ExpandTableColumn(#"TransformColumns", "Data", {"excel formula", "Index", "Index2"}, {"excel formula", "Index", "Index2"}),
#"Added Custom" = Table.AddColumn(#"Expanded Data", "PQ method", each Text.From([key]) & "_" & Text.From([Index2])),
#"Sorted Rows" = Table.Sort(#"Added Custom",{{"Index", Order.Ascending}}),
#"Removed Columns" = Table.RemoveColumns(#"Sorted Rows",{"Index", "Index2"})
in
#"Removed Columns"

Power Query - best way to sub select?

Suppose I have a column representing object type and another column representing object color. I want to remove blue and red fruits (example of object type) but keep all other red and blue objects.
How can I acheive this in Power Query ?
Thanks,
Just (un)select (not) matching rows
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
Filtered = Table.SelectRows(Source, each not ([ObjectType] = "Fruit" and ([ObjectColor]="Red" or [ObjectColor]="Blue")))
in
Filtered
Here's one way:
If you start with this:
You can merge the two columns together like this:
Then filter out the "Fruit,Blue" and "Fruit,Red":
Which yields this:
And you can then delete the "Merged" column to get this:
Here's the M code:
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"ObjectType", type text}, {"ObjectColor", type text}}),
#"Inserted Merged Column" = Table.AddColumn(#"Changed Type", "Merged", each Text.Combine({[ObjectType], [ObjectColor]}, ","), type text),
#"Filtered Rows" = Table.SelectRows(#"Inserted Merged Column", each ([Merged] <> "Fruit,Blue" and [Merged] <> "Fruit,Red")),
#"Removed Columns" = Table.RemoveColumns(#"Filtered Rows",{"Merged"})
in
#"Removed Columns"

How do you search the Dynamics 365 database for Power BI

I'm having trouble finding things in the database of Dynamics 365 with Power BI, when you connect to the OrganizationData.svc, there is a LOT of table, and sub table.
For exemple, I'm working with the table "OpportunitySet", someone add a custom combobox, I'm able to get the value "176000004", but I can't find a way to get the text value from this.
I search in the "PickListMappingSet", but there is too much stuff.
Also, all my Opportunities have a Team and people in those teams, I think they are link with "Connections", but I have no idea how to obtain them in Power BI, I need they find some opportunities who have missing people in their team.
Is there any way to search everywhere in the DataBase, or to find where each value is store in it.
Thanks
stringmaps is not listed as an entity in the metadata for the api. However https://<your_dynamics_url>/api/data/v9.1/stringmaps works in Chrome for me although you receive a paginated response, which means you can build a reference query to use when looking up optionsets or even state and status codes:
let
DataList = List.Generate(
() => [
SourceURI="https://<your_dynamics_url>/api/data/v9.1/stringmaps"
,Pagecount=0
,Stringmaps = {}
,Source = []
,ErrorTest = try Source = []
]
,each if [ErrorTest][HasError] then false else true
,each [
ErrorTest = try Source = Json.Document(Web.Contents([SourceURI]))
,Source = Json.Document(Web.Contents([SourceURI]))
,SourceURI = Record.Field(Source,"#odata.nextLink")
,Stringmaps = Source[value]
,Pagecount = [Pagecount] + 1
]
),
#"Converted to Table" = Table.FromList(DataList, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Expanded Column1" = Table.ExpandRecordColumn(#"Converted to Table", "Column1", {"Stringmaps"}, {"Column1.Stringmaps"}),
#"Removed Errors" = Table.RemoveRowsWithErrors(#"Expanded Column1", {"Column1.Stringmaps"}),
#"Expanded Column1.Stringmaps" = Table.ExpandListColumn(#"Removed Errors", "Column1.Stringmaps"),
#"Removed Blank Rows" = Table.SelectRows(#"Expanded Column1.Stringmaps", each not List.IsEmpty(List.RemoveMatchingItems(Record.FieldValues(_), {"", null}))),
#"Expanded Column1.Stringmaps1" = Table.ExpandRecordColumn(#"Removed Blank Rows", "Column1.Stringmaps", {"value", "attributename", "objecttypecode", "attributevalue"}, {"value", "attributename", "objecttypecode", "attributevalue"}),
#"Sorted Rows" = Table.Sort(#"Expanded Column1.Stringmaps1",{{"objecttypecode", Order.Ascending},{"attributename", Order.Ascending}}),
#"Grouped Rows" = Table.Group(#"Sorted Rows", {"attributename", "objecttypecode"}, {{"Count", each _, type table [value=text, attributename=text, objecttypecode=text, attributevalue=number]}}),
#"Grouped Rows1" = Table.Group(#"Grouped Rows", {"objecttypecode"}, {{"Count", each _, type table [attributename=text, objecttypecode=text, Count=table]}})
in
#"Grouped Rows1"
Basically for onpremise, you can pull from StringMap entity & see all the optionset (=picklist=combobox) values in Database. For online it's a problem.
This is a known issue with PowerBI query as well as oData endpoint, you cannot get the custom or user created picklist values.
You can ask some CRM developers to get all the picklist values/text from CRM customizations & store as a Source in PowerBI datasets to get the desired results after merge with main dataset.
Edit: Xrmtoolbox PowerBI optionset assistant will be helpful.

Power Query Parameter works in one table but not another

So I have two tables (power query), and want to combine them into one. The second table just looks at the first table (power query) and applies a parameter filter to it. When i try to combine the parameter code into the original query the filter doesn't work. I have enabled fast combine to made all queries public to get rid of any firewall issues.
So as not to break the original working set of pq, i duplicated the first pq and modified using advanced by coping the needed code to apply the parameter (third pq)
Second power query code (this looks at first pq an applies a parameter filter) and it works
let
Date_Parameter = Excel.CurrentWorkbook(){[Name="Parameter"]}[Content],
Date_Value = Date_Parameter{0}[Value],
Source = Excel.CurrentWorkbook(){[Name="Timesheet1"]}[Content],
#"Filtered Rows" = Table.SelectRows(Source, each ([Date] = Date_Value))
in
#"Filtered Rows"
Third power query code (this is the one where i duplicated the first pq and added parameter code from second pq) this doesn't work
let
Date_Parameter = Excel.CurrentWorkbook(){[Name="Parameter"]}[Content],
Date_Value = Date_Parameter{0}[Value],
Source = Excel.Workbook(File.Contents("\\192.168.12.31\Project Files\Daily Truck Sheet\TimeTrack\TimeTrack.xlsm")),
Timesheet_Table = Source{[Item="Timesheet",Kind="Table"]}[Data],
Merge = Table.NestedJoin(Timesheet_Table,{"Ref"},Project,{"Ref"},"NewColumn"),
#"Expand NewColumn" = Table.ExpandTableColumn(Merge, "NewColumn", {"Crew"}, {"NewColumn.Crew"}),
#"Renamed Columns" = Table.RenameColumns(#"Expand NewColumn",{{"NewColumn.Crew", "Crew"}}),
#"Removed Duplicates" = Table.Distinct(#"Renamed Columns", {"Ref"}),
#"Removed Columns" = Table.RemoveColumns(#"Removed Duplicates",{"Ref", "Employee Name", "Truck #", "Hours", "Per Diem", "Piecework", "Travel Day", "Timecard Filename", "Paid DT Hrs.", "hours check", "project hours", "Paid Regular Hours", "Paid OT Hrs.", "PayPeriod", "Employee Number", "Lead Hand Employee Number", "Crew Count", "Employee Revenue"}),
#"Reordered Columns"= Table.ReorderColumns(#"Removed Columns",{"Date", "Date Received", "Lead Hand", "Crew", "Project#", "Comments", "Work Performed", "time card hours", "Revenue per hour", "Total Reveneu"}),
Rounding = Table.TransformColumns(#"Reordered Columns",{{"Revenue per hour", each Number.Round(_, 2)}, {"Total Reveneu", each Number.Round(_, 2)}}),
#"Filtered Rows" = Table.SelectRows(Rounding, each ([Date] = Date_Value))
in
#"Filtered Rows"
so i had to insert a transform for pq to treat as a date. Even though in the Parameter pq (that loads the value from the parameter table) it is already transformed. replaced the first three lines before the source line with the following and it worked
Date_Parameter = Excel.CurrentWorkbook(){[Name="Parameter"]}[Content],
#"Changed Type1" = Table.TransformColumnTypes(Date_Parameter,{{"Value", type date}}),
Date_Value = #"Changed Type1"{0}[Value],
So maybe now i can get rid of the parameter pq as it is all built into the final pq but haven't tried yet

Resources