I've spent hours and hours on this, but no luck. Can anyone help?
Issue: I have two tables; one source data and one lookup data. I need to count the number of times the lookup data appears in the corresponding column on the main table. The column names could be anything, but they always have a counterpart in the other table.
In the attached example, There are four columns (Month/Shape/Animal/Mineral) in the first table, and the same columns in the lookup table. The lookup table contains items that do not belong in the respective columns (i.e. "Triangle" is not a month, and "Kitten" is not a shape).
I'm trying to use Power Query to count the number of invalid entries in each column. I'm thinking that "Group By" is the way to go, but I'm still stuck....
Input Data
-Month- -Shape- -Animal- -Mineral-
January Square Puppy Gold
February Circle Kitten Quartz
Square Triangle October Amber
Circle Kitten October Lead
Square Puppy Horse Gold
Circle Puppy Goldfish Silver
May Hexagon Pig Gold
Invalid Values
-Month- -Shape- -Animal- -Mineral-
Square Kitten July Kitten
Circle Puppy October Puppy
Triangle Pig January Pig
Trying to end-up with this table:
Field Invalid Entry Count
-Month- 4
-Shape- 3
-Animal- 2
-Mineral- 0
I have also attached a screenshot and sample workbook. Any help/guidance/assistance would be ~greatly~ appreciated.
https://drive.google.com/file/d/1VpgwySRra9-gStlmtjJum3w2JySPyrt5/view?usp=sharing
Thank you!
Load invalid Value, add index, click the index and unpivot other columns. Close and Load. Lets call this Table2
Load Input Data, add index, click the index and unpivot other columns. Merge in Table2 by matching Attribute and Value with left outer join. Expand the resulting column to extract index column from Table2. Filter out the nulls. Group and Count on Attribute.
Table2 = Invalid Values code
let Source = Excel.CurrentWorkbook(){[Name="Table2"]}[Content],
#"Added Index" = Table.AddIndexColumn(Source, "Index", 0, 1),
#"Unpivoted Other Columns" = Table.UnpivotOtherColumns(#"Added Index", {"Index"}, "Attribute", "Value")
in #"Unpivoted Other Columns"
Table 1 = Input Data code
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Added Index" = Table.AddIndexColumn(Source, "Index", 0, 1),
#"Unpivoted Other Columns" = Table.UnpivotOtherColumns(#"Added Index", {"Index"}, "Attribute", "Value"),
#"Merged Queries" = Table.NestedJoin(#"Unpivoted Other Columns",{"Value", "Attribute"},Table2,{"Value", "Attribute"},"Table2",JoinKind.LeftOuter),
#"Expanded Table2" = Table.ExpandTableColumn(#"Merged Queries", "Table2", {"Index"}, {"Index.1"}),
#"Filtered Rows" = Table.SelectRows(#"Expanded Table2", each ([Index.1] <> null)),
#"Grouped Rows" = Table.Group(#"Filtered Rows", {"Attribute"}, {{"Invalid Entry Count", each Table.RowCount(Table.Distinct(_)), type number}})
in #"Grouped Rows"
Related
In regular Excel, I can write three index/match formulae to translate three separate columns. (Sometimes I do the match in its own column to aid readability of each of the index formulae).
I can't see how to replicate that in Power Query except by repeating the same merge and expand steps three times, which doesn't feel very efficient. Is that the way to do it or am I missing a simpler method?
In powerquery, unpivot the three columns, then merge once, and if needed, pivot it back
let Source = Excel.CurrentWorkbook(){[Name="Table2"]}[Content],
#"Added Index" = Table.AddIndexColumn(Source, "Index", 0, 1, Int64.Type),
#"Unpivoted Other Columns" = Table.UnpivotOtherColumns(#"Added Index", {"Index"}, "Attribute", "Value"),
#"Merged Queries" = Table.NestedJoin(#"Unpivoted Other Columns", {"Value"}, Table1, {"Column1"}, "Table1", JoinKind.LeftOuter)
in #"Merged Queries"
First of all, sorry for my broken English. If something might feel offence, that is my fault. Sorry (and Thanks) in advance.
As title said, I want to get a way how to multiply many rows with one single row in one step: regardless how many columns and whatever column name is, & ignore blank cells then shift up filled results in POWERQUERY.
want to get a final table in yellow, with merging Ratio table and Total table.
want to know how to multiply Ratio table (many columns) and Total table (a single column) in one step.
From Grey table, want to get a way how to remove null value and shift up filled results.
prefer "Transform" than "Add column".
if you need an example file, can get it via this link : https://www.dropbox.com/s/fs2cymeak1f2w57/powerquery%20practice.xlsx?dl=0
Im 99% certain Im doing homework for someone, but anyway
Unpivot Ratio. Merge in Total, multiply with custom column. Group on Attribute and add index. Pivot
let Source = Ratio,
// I am too lazy to do my own homework and should be ashamed
#"Unpivoted Other Columns" = Table.UnpivotOtherColumns(Source, {"M"}, "Attribute", "Value"),
#"Filtered Rows1" = Table.SelectRows(#"Unpivoted Other Columns", each ([Value] <> 0)),
#"Merged Queries" = Table.NestedJoin(#"Filtered Rows1", {"M"}, Total, {"Month"}, "Total", JoinKind.LeftOuter),
#"Expanded Total" = Table.ExpandTableColumn(#"Merged Queries", "Total", {"Total"}, {"Total"}),
#"Added Custom" = Table.AddColumn(#"Expanded Total", "Custom", each if [Total]=null then 0 else [Total]*[Value]),
#"Removed Columns" = Table.RemoveColumns(#"Added Custom",{"Value", "Total","M"}),
#"Grouped Rows" = Table.Group(#"Removed Columns", {"Attribute"}, {{"data", each Table.AddIndexColumn(_, "Month", 1, 1, Int64.Type), type table }}),
#"Expanded data" = Table.ExpandTableColumn(#"Grouped Rows", "data", { "Custom", "Month"}, { "Custom", "Month"}),
#"Pivoted Column" = Table.Pivot(#"Expanded data", List.Distinct(#"Expanded data"[Attribute]), "Attribute", "Custom")
in #"Pivoted Column"
In power query I would use the Excel formula as shown in the screen shot link below.
Is there an easy way to do this in Power Query M Code? It doesn't seem so!
It looks like you want to add an index by group on Type
Right click Type column, and Group
Use new column name:data, Operation:All Rows and hit ok
Add column .. index column .. custom and put in your starting numbers and increment
Use arrows atop the data column to expand to new rows
Remove the extra column
sample code:
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Grouped Rows" = Table.Group(Source, {"Type"}, {{"data", each _, type table}}),
#"Added Index" = Table.AddIndexColumn(#"Grouped Rows", "Index", 23, 1),
#"Expanded data" = Table.ExpandTableColumn(#"Added Index", "data", {"Type"}, {"Type.1"}),
#"Removed Columns" = Table.RemoveColumns(#"Expanded data",{"Type.1"})
in #"Removed Columns"
another way to do it that requires editing code
Right-click the Type column and remove duplicates (or group, whichever is easier)
Add index
Merge that result back into original data and expand
sample code
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
// remove dupes and create index
#"Removed Duplicates" = Table.Distinct(Source),
#"Added Index" = Table.AddIndexColumn(#"Removed Duplicates", "Index", 23, 1),
//merge the numbered table back into original
#"Merged Queries" = Table.NestedJoin(Source,{"Type"},#"Added Index",{"Type"},"TT",JoinKind.LeftOuter),
#"Expanded Table1" = Table.ExpandTableColumn(#"Merged Queries", "TT", {"Index"}, {"Index"})
in #"Expanded Table1"
I'm trying to add a column in the power query editor (using M as I want to show new column in table output) that should calculate following formula:
Value of record Column Market Value / Sum of Column Market Value
BUT this sum should only include rows where two other columns (in my case Date and Account number) are equal to the value in the record/row I want to calculate value for.
From what I can tell, you are trying to achieve the same as doing a SUMIFS() calculation in an excel table but in power query?
This can be done in a couple of ways, but the simplest is with power query's "Group By" function. With this method you use Table.Group() to group the data by your two columns (Date & Account Number) and get the sum for all records. Then all you need to do is merge this value to the original table by referencing the group by step. This is equivalent to using a pivot table then doing a VLOOKUP on both the Date and Account number column.
Here is the M code:
let
// You're previous code here (replace the "Source" & #"Changed Type" lines)
Source = Excel.CurrentWorkbook(){[Name="my_data"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Date", type date}, {"Account number", type text}, {"Market Value", type number}}),
// Group by Date & Account Number to sum each record
#"Calculate Record Value" = Table.Group(#"Changed Type", {"Date", "Account number"}, {{"Value of Record", each List.Sum([Market Value]), type number}}), // Replace #"Changed Type" in this line with the last step reference from your code
// Merge the group by value back to each record in the orginal table
#"Merged Record Value" = Table.NestedJoin(#"Changed Type",{"Date", "Account number"},#"Calculate Record Value",{"Date", "Account number"},"sum",JoinKind.LeftOuter),
// Expand your new column
#"Expanded Record Value" = Table.ExpandTableColumn(#"Merged Record Value", "sum", {"Value of Record"}, {"Value of Record"})
in
#"Expanded Record Value"
Hopefully this is helpful!
Please let me know if I have misunderstood your question or if this solution doesn't work for you.
Best regards
DougC
I created a table called Table3 with two columns named URL which is empty and Value which contains a list of websites. The following query retrieves data from the websites stored in Table 3.
let
Parameter = Excel.CurrentWorkbook(){[Name="Table3"]}[Content],
URL= Parameter{1}[Value],
Source = Web.Page(Web.Contents(URL)),
Data0 = Source{0}[Data],
#"Changed Type" = Table.TransformColumnTypes(Data0,{{"Date", type date}, {"Open", type number}, {"High", type number}, {"Low", type number}, {"Close", type number}, {"Volume", type number}, {"Market Cap", type number}}),
#"Removed Columns" = Table.RemoveColumns(#"Changed Type",{"Market Cap", "Open", "High", "Low"}),
#"Sorted Rows" = Table.Sort(#"Removed Columns",{{"Date", Order.Ascending}})
in
#"Sorted Rows"
The second line runs the query for the first website in the Value column.
Is it possible to introduce a loop that would run the query for all the websites?
If not is it possible to run the query for all the websites in sequence by manually pasting the above code and changing the number in the brackets for each website?
If it is possible I assume it would load the contents in the same sheet, is there any way to load the content in different sheets for each iteration?
Thanks for reading my question.
Loops aren't really a thing in Power Query, but you can still do what you're after. I don't know what URLs you're pulling from, so let me give you an example using publicly available ones.
Let's suppose my Table3 is the following:
URL
--------
https://finance.yahoo.com/quote/AAPL?p=AAPL
https://finance.yahoo.com/quote/AAPL?p=GOOG
I can load this into the query editor and create a custom column that reads the webpage for each URL.
= Web.Page(Web.Contents([URL])){0}[Data]
(The table I want is the first one (hence the {0} row index) and is in the [Data] column.)
Now I have a table like this where the bottom table is a preview of the cell I have selected.
Click the arrows icon to expand the tables.
From here you can filter Column1 to pick which values you are interested in (let's say Ask, Bid, Open, and Volume) and then pivot that column (Transform > Pivot Column). Choose Column2 as the values column and select "Don't Aggregate" under Advanced options.
The result should be the table you see above the pivot dialogue box.
Here's the full M code for the query that shows up in the Advanced Editor
let
Source = Excel.CurrentWorkbook(){[Name="Table3"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"URL", type text}}),
#"Added Custom" = Table.AddColumn(#"Changed Type", "Custom", each Web.Page(Web.Contents([URL])){0}[Data]),
#"Expanded Custom" = Table.ExpandTableColumn(#"Added Custom", "Custom", {"Column1", "Column2"}, {"Column1", "Column2"}),
#"Filtered Rows" = Table.SelectRows(#"Expanded Custom", each ([Column1] = "Ask" or [Column1] = "Bid" or [Column1] = "Open" or [Column1] = "Volume")),
#"Pivoted Column" = Table.Pivot(#"Filtered Rows", List.Distinct(#"Filtered Rows"[Column1]), "Column1", "Column2")
in
#"Pivoted Column"