Dynamically expand ALL lists and records from json - powerquery

I want to expand all lists and records in a json response.
Columns are like e.g. (this is dynamically, it also can be 10 records and 5 lists):
Text, Text, [List], [List], Text, [Record], [Record], String, [Record]
I wrote a function for getting all columns with the specific type
Cn.GetAllColumnsWithType = (table as table, typ as type) as list =>
let
ColumnNames = Table.ColumnNames(table),
ColumnsOfType = List.Select(ColumnNames, (name) =>
List.AllTrue(List.Transform(Table.Column(table, name), (cell) => Type.Is(Value.Type(cell), typ))))
in
ColumnsOfType;
and a function to expand all lists from a table
Cn.ExpandAllListsFromTable = (table as table, columns as list) =>
let
expandedListTable = List.Accumulate(columns, table, (state, columnToExpand) =>
Table.ExpandListColumn(state, columnToExpand))
in
expandedListTable;
all lists are now records and i want to dynamically expand all these records.
I think i need a foreach to iterate through the list (which are only records cause of Cn.GetAllColumnsWithType),
Table.ExpandRecordColumn each element with it's Table.ColumnNames and add it to the table but i don't know how to do it.
Maybe you can help me out cause it's driving me crazy.
Cheers
Edit:
I recently opened a thread but there i wanted to expand a specific one like
#"SelectItems" = Table.SelectColumns(records,{"$items"}),
#"$items1" = #"SelectItems"{0}[#"$items"],
but now i want to do it all dynamically.

Chris Webb wrote a function to do this for Table-type columns:
http://blog.crossjoin.co.uk/2014/05/21/expanding-all-columns-in-a-table-in-power-query/
I've shared a tweaked version of that that I made for Record-type columns:
https://gist.github.com/Mike-Honey/0a252edf66c3c486b69b

You do not need a function for that. Assuming that the previous step in M was named Removed Other Columns, and that the column to expand is named Data, then make regular Expand step and replace its code of #"Expanded Data" with the following code:
#"Expanded Data"
= Table.ExpandTableColumn(
#"Removed Other Columns",
"Data",
List.Union(List.Transform(#"Removed Other Columns"[Data], each Table.ColumnNames(_)))
)
It expands all columns without referencing their names.

Related

Is there any way how to refer actual column name as variable in Power Query

I am new to Power Query and I would like to ask more experienced people about it.
I am trying to solve problem with Text.Combine where I would like to combine together value in specific column and column name of actual cell.
Do you have any idea?
My idea is formula similar to this:
=Text.Combine({[kod], Column.Name},";")
Thank you very much for answer.
Tomas
Edit 8.12.2021:
#horseyride
I actually try fill columns automatically with data in following format where first part will be value from actual row and actual column name:
For e.g.:
8M0183:F01A0101.B in first row, second column,
8M0182:F01A0102.A in second row, first column
Table example mentioned bellow.
Thank you very much for all answers.
see if this works for you. It combines the KOD column with the column name into each null cell, for every column that is not named KOD
It finds all column not named KOD. It converts those to text. It replaces all nulls with a placeholder, here "xxx". We unpivot to get everything to just three columns. We then combine the column title and the cell contents if the cell contents is equal to the placeholder. Pivot to get back to original format
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
ColumnList=List.Difference(Table.ColumnNames(Source),{"KOD"}),
ConvertToText= Table.TransformColumnTypes (Source,List.Transform(ColumnList, each {_ , type text})),
ReplaceNullsWithPlaceholder = Table.ReplaceValue(ConvertToText,null,"xxx",Replacer.ReplaceValue,ColumnList),
#"Unpivoted Other Columns" = Table.UnpivotOtherColumns(ReplaceNullsWithPlaceholder, {"KOD"}, "Attribute", "Value"),
CombineColumnAndRow = Table.AddColumn(#"Unpivoted Other Columns", "Value2", each if [Value]="xxx" then [Attribute]&":"&[KOD] else [Value]),
#"Removed Columns" = Table.RemoveColumns(CombineColumnAndRow,{"Value"}),
#"Pivoted Column1" = Table.Pivot(#"Removed Columns", List.Distinct(#"Removed Columns"[Attribute]), "Attribute", "Value2")
in #"Pivoted Column1"

Bring Value with Sumifs in Pow.Query language to specified row, and column(location)

Next step? I have brought with sumifs and a lot sumif from other workbook, information to the exact row, column in excel workbook. Now I want to do the same with query language. I can bring two values if condition is met, but then it is unclear how I will bring the total sum to the one row in excel workbook. Can anyone show me the path? I guess I will need Data Model...
= Table.AddColumn(#"Changed Type", "Sumif", each if [Column2] =2 or [Column2]=1 then [Column3]+[Column4] else 0)
let
Source = Folder.Files...
#"C:\Users...
#"Imported Excel" = Excel.Workbook(#"C:\...
SegPL_Chart = #"Imported Excel"{[Name="SegPL_Chart"]}[Data],
#"Removed Top Rows" = Table.Skip(SegPL_Chart,12),
#"Removed Alternate Rows" = Table.AlternateRows(#"Removed Top Rows",1,1,90),
#"Promoted Headers" = Table.PromoteHeaders(#"Removed Alternate Rows"),
#"Filtered Rows" = Table.SelectRows(#"Promoted Headers", each ([Col1]="1" or [Col1]="2")),
#"Table Group = Table.Group(#"Filtered Rows", {}, List.TransformMany(Table.ColumnNames(#"Filtered Rows",(x)=>{each if x = "Names" then "Totals" else List.Sum(Table.Column(_,x))},(x,y)=>{x,y})),
#"append" = Table.Combine({#"Filtered Rows",#"Table Group"})
in
#"append"
It gives an error "in" Token comma needed..? What else I need to do bring total rows?
You can use several steps to create several helper columns with intermediate results of conditional sums. Then you can create a new column, sum up all the intermediate results and the delete the helper columns with the intermediate results.
Keep in mind that unlike Excel, the calculations in Power Query always return constants and you can then delete calculated columns you no longer need. So,
Create helper column 1 with complicated IF and Sum scenario
Create helper column 2 with complicated IF and Sum scenario
Create total column to add column 1 + column 2
Delete helper columns and keep only the total column
That gives me exact result what I was looking for, but it is with DAX formula in PowerPivot:
=SUMX(FILTER('TableName',[ColName] = 1),'TableName'[ColName2])
So would be glad to convert it to Power-Query formula

join two tables in linq with special conditions

I hope one can help me, I am new in linq,
I have 2 tables name tblcart and tblorderdetail:
I just show some fields in these two tables to show whats my problem:
tblCart:
ID,
CartID,
Barcode,
and tblOrderDetail:
ID,
CartID,
IsCompleted
Barcode
when someone save an order, before he confirms his request,one row temporarily enter into the tblCart,
then if he or she confirms his request another row will be inserted into the tblOrderDetail ,
Now I wanna not to show the rows that is inserted into tblOrderDetailed(showing just temporarily rows which there is in tblCart),
In another words, if there is rows in tblCart with cartID=1 and at the same time there is the same row with CartID= 1 in tblOrderDetail, then I dont want that Row.
All in all, Just the rows that there isnt in tblOrderDetail, and the field to realize this is CartID,
I should mention that I make Iscompleted=true, and with that either we can exclude the rows we do not want,
I did this:
var cartItems = context.tblCarts
.Join(context.tblSiteOrderDetails,
w => w.CartID,
orderDetail => orderDetail.cartID,
(w,orderDetail) => new{w,orderDetail})
.Where(a=>a.orderDetail.cartID !=a.w.CartID)
.ToList()
however it doesn't work.
one example:
tblCart:
ID=1
CartID=1213
Barcode=4567
ID=2
CartID=1214
Barcode=4567
ID=3
CartID=1215
Barcode=6576
tblOrderDetail:
ID=2
CartID=1213
Barcode=4567
IsCompleted=true
with these data it should just show the last two Row in tblCart, I mean
ID=2
CartID=1214
Barcode=4567
ID=3
CartID=1215
Barcode=6576
This sounds like a case for WHERE NOT EXISTS in sql.
roughly translated this should be something like this in LINQ:
var cartItems = context.tblCarts.Where(crt => !context.tblSiteOrderDetails.Any(od => od.CartID == crt.cartID));
If you have a navigation property on cart to reference details (I'll assume it's called Details), then:
var results=context.tblCarts.Where(c=>!c.Details.Any(d=>d.IsCompleted));

Is there an ISNUMBER() or ISTEXT() equivalent for Power Query?

I have a column with mixed types of Number and Text and am trying to separate them into different columns using an if ... then ... else conditional. Is there an ISNUMBER() or ISTEXT equivalent for power query?
Here is how to check type in Excel Powerquery
IsNumber
=Value.Is(Value.FromText([ColumnOfMixedValues]), type number)
IsText
=Value.Is(Value.FromText([ColumnOfMixedValues]), type text)
hope it helps!
That depends a bit on the nature of the data and how it is originally encoded. Power Query is more strongly typed than Excel.
For example:
Source = Table.FromRecords({[A=1],[A="1"],[A="a"]})
Creates a table with three rows. The first row's data type is number. The second and third rows are both text. But the second row's text could be interpreted as a number.
The following is a query that creates two new columns showing if each row is a text or number type. The first column checks the data type. The second column attempts to guess the data type based on the value. The guessing code assumes everything that isn't a number is text.
Example Code
Edit: Borrowing from #AlejandroLopez-Lago-MSFT's comment for the interpreted type.
let
Source = Table.FromRecords({[A=1],[A="1"],[A="a"]}),
#"Added Custom" = Table.AddColumn(Source, "Type", each
let
TypeLookup = (inputType as type) as text =>
Table.FromRecords(
{
[Type=type text, Value="Text"],
[Type=type number, Value="Number"]
}
){[Type=inputType]}[Value]
in
TypeLookup(Value.Type([A]))
),
#"Added Custom 2" = Table.AddColumn(#"Added Custom", "Interpreted Type", each
let
result = try Number.From([A]) otherwise "Text",
resultType = if result = "Text" then "Text" else "Number"
in
resultType
)
in
#"Added Custom 2"
Sample output
Put it in logical test format
Value.Type([Column1]) = type number
Value.Type([Column1]) = type text
The function Value.Type returns a type, so by putting it in equation thus return a true / false.
Also, equivalently,
Value.Type([Column1]) = Date.Type
Value.Type([Column1]) = Text.Type
HTH
ISTEXT() doesn't exist in any language I've worked with - typically any numeric or date value can be converted to text so what would be a false result?
For ISNUMBER, I would solve this without any code by changing the Data Type to a number type e.g. Whole Number. Any rows that don't convert will show Error - you can then apply Replace Errors or Remove Errors to handle them.
Use Duplicate Column first if you don't want to disturb the original column.
I agree with Mike Honey.
I have a SKU code that is a mix of Char and Num.
Normally the last 8 Char are Numbers but in some weird circumstances the SKU is repeated with an additional letter but given the same EAN which causes chaos.
by creating a new temp column using Text.End(SKU, 1) I get only the last character. I then convert that column to Whole Number. Any Error rows are then removed to leave only the rows I need. I then delete the temp Column and am left with the Rows I need in the format I started with.

DataTable.RowFilter using List results?

I need to keep the original DataTable and filter it based on list values. I've tried to hide rows that don't match up to the list but it doesn't work instantly. I don't want to copy the list to another dataTable and bind my dataGridView to is cause i have too much tied up to the original DataTable. Filtering only works for a few conditions but the list is holds only the values that are needed. Is there a LINQ solution to this? C# only please. This is how I got my list. There is another dataGridView that holds primary key for the datatable that I need to filter. It's key is held in a comboBox.
DataTable MainTable = MainDataSet.Tables["MyTable"];
DataTable lookupTable = lookupDataSet.Tables["MyLookupTable"];
var List = (from x in lookupTable.AsEnumerable()
where x.Field<string>("kAutoInc") == comboBox.SelectedValue.ToString()
select x.Field<int>("Pct")).ToList();
var a = MainTable.AsEnumerable().Where(r =>
List .Any(id => id == r.Field<int>("Pct")));
This gives me values of int 30, 40, 50, 60...
The DataGridView has a matching column "Pct" that I need to filter rows that only contain these values. I know it seems easy but I just can't seem to get it to work.
Please use DataView instead.
dataview dv=new dataview(datatable);
dv.filter=string.format("column1={0}","value1");
Like that.

Resources