Related
I'm trying to load the text contents of an email from outlook in into PowerBI using Power Query. I want to specifically load only the contents of emails from a specified person, with a specified subject line, and only the most recent email.
Here's my code so far:
let
Source = Exchange.Contents("user#email.com"),
Mail1 = Source{[Name="Mail"]}[Data],
#"Filtered Rows" = Table.SelectRows(Mail1, each ([Folder Path] = "\Inbox\Project\")),
#"Sorted Rows" = Table.Sort(#"Filtered Rows",{{"DateTimeReceived", Order.Descending}}),
#"Filtered Rows1" = Table.SelectRows(#"Sorted Rows", each Text.Contains([Subject], "Speficied Email Subject")),
#"Expanded Sender" = Table.ExpandRecordColumn(#"Filtered Rows1", "Sender", {"Address"}, {"Sender.Address"}),
#"Filtered Rows2" = Table.SelectRows(#"Expanded Sender", each Text.Contains([Sender.Address], "specified email sender")),
#"Filtered Rows3" = Table.SelectRows(#"Filtered Rows2", each Text.Contains([Subject], "specified email subject")),
#"Kept First Rows" = Table.FirstN(#"Filtered Rows3",1),
#"Choose First Result's Content" = #"Kept First Rows"[Body],
TextBody = #"Choose First Result's Content"[TextBody]
in
TextBody
I'm getting an error:
Expression.Error: We cannot apply field access to the type List.
Details:
Value=[List]
Key=TextBody
Can someone help me correct this code?
Thanks!
Change the last step to TextBody = #"Choose First Result's Content"{0}[TextBody]
Here is the script I have based on the steps generated:
let
Source = Sql.Database("BITESTDBSVR1", "DW_FINANCE"),
CORPDB_BTT_SCENARIO_DETAILS_WITH_BACKLOG = Source{[Schema="CORPDB",Item="BTT_SCENARIO_DETAILS_WITH_BACKLOG"]}[Data],
#"Filtered Rows" = Table.SelectRows(CORPDB_BTT_SCENARIO_DETAILS_WITH_BACKLOG, each [ScenarioHeaderID] = FromScenario or [ScenarioHeaderID] = ToScenario),
#"Filtered Rows1" = Table.SelectRows(#"Filtered Rows", each [ValueTypeName] <> "Backlog"),
#"Removed Columns" = Table.RemoveColumns(#"Filtered Rows1",{"ID", "ContractID", "LineNumber", "Ophours", "Prime", "ScenarioHeaderID", "Division", "ORN", "BidNumber", "CustomerContractNumber", "ProjectID", "CustomerOrderID", "CurrentStatus", "CustomerBuyer", "ProgramName", "ProgramNotes", "Notes", "MajorProgram", "ProductFamily", "ProductSubFamily", "ProductGroup", "SeriesGroup", "Turret", "ProgramSeriesID", "OrderQuantity", "ContractValueSource", "SourceCurrencyID", "BookingFXRate", "ContractValueUSDollars", "PODateIn", "AcceptanceDate", "ContractAwardDate", "ContractAwardYear", "Probability", "FactoredValue", "ProductAccountManager", "ProgramManager", "InterIntraDivision", "InterIntraExternal", "Territory", "IDIQ", "Platform", "CustomerBuyerCountry", "EndUser", "LHXRegion", "LHXName", "ProcurementLocation", "CustomerType1", "CustomerType2", "ExternalCustomer", "SalesType", "BidType", "PricingType", "ExportLicenceNumber", "USExportStatus", "CANExportStatus", "ExportComments", "RevenueRecognitionCriteria", "IncoTerms", "ShipToCountry", "PaymentTerms", "EOIRMaster", "EOIRChildID", "BidDivision", "SubProduct", "ProductLine", "ServiceAccountManager", "BusinessUnit", "OffsetCapture", "EndUserBusinessPartner", "GeographicRegion", "GeographicSubRegion", "ContractDuration", "RevRecTiming", "ForecastRevenueMonths", "InvoiceNumber", "WorkOrdernumber", "SerialNumber", "OrderLine", "PKLineIdentifier", "Year1", "Year2", "Year3", "BI1", "BI2", "BI3", "RI1", "RI2", "RI3", "VMProject", "ServiceContractNo", "MainProject", "LNProject", "ProjectContract", "SalesForceRef", "SalesRegion", "FiscalDate", "ScenarioCreated", "ScenarioUpdated", "ScenarioWeek", "Scenario", "ValueType"}),
#"Pivoted Column" = Table.Pivot(#"Removed Columns", List.Distinct(#"Removed Columns"[ScenarioName]), "ScenarioName", "Value", List.Sum),
#"Replaced Value" = Table.ReplaceValue(#"Pivoted Column",null,0,Replacer.ReplaceValue,{ScenarioFrom, ScenarioTo}),
#"Inserted Subtraction" = Table.AddColumn(#"Replaced Value", "Subtraction", each [#"Jan 15, 2021"] - [#"Mar 5, 2021"], type number),
#"Changed Type" = Table.TransformColumnTypes(#"Inserted Subtraction",{{"Subtraction", type number}}),
#"Renamed Columns" = Table.RenameColumns(#"Changed Type",{{"Subtraction", "Variance"}}),
#"Merged Queries" = Table.NestedJoin(#"Renamed Columns", {"InterIntra"}, #"CORPDB INTER_INTRA_MAPPING", {"INTER DIVISION"}, "CORPDB INTER_INTRA_MAPPING", JoinKind.LeftOuter),
#"Expanded CORPDB INTER_INTRA_MAPPING" = Table.ExpandTableColumn(#"Merged Queries", "CORPDB INTER_INTRA_MAPPING", {"INTER DIVISION NAME"}, {"INTER DIVISION NAME"}),
#"Reordered Columns" = Table.ReorderColumns(#"Expanded CORPDB INTER_INTRA_MAPPING",{"Forecast", "ProgramGroupID", "InterIntra", "INTER DIVISION NAME", "EndUserCountry", "ValueTypeName", "Month", "Year", "Quarter", "MonthNumber", "FiscalYearMonth", "QuarterStart", "QuarterEnd", ScenarioFrom, ScenarioTo , "Variance"}),
#"Renamed Columns1" = Table.RenameColumns(#"Reordered Columns",{{"INTER DIVISION NAME", "InterIntraDivision"}}),
#"Removed Columns1" = Table.RemoveColumns(#"Renamed Columns1",{"InterIntra"})
in
#"Removed Columns1"
The part of the code I want to update is
#"Inserted Subtraction" = Table.AddColumn(#"Replaced Value", "Subtraction", each [#"Jan 15, 2021"] - [#"Mar 5, 2021"], type number),
Where I have 2 text variables (ScenarioFrom, ScenarioTo) that I want to dynamically substitute in the definition to say
#"Inserted Subtraction" = Table.AddColumn(#"Replaced Value", "Subtraction", each [Scenariofrom] - [Scenarioto] , type number),
I get that I'm trying to force a literal into a table column which is causing the problem, but wondering if there is a function / easy work-around without transforming the data entirely.
When you are defining a custom column, writing each [ColName] is short for each _[ColName] where _ represents the current row, which is a Record type.
With this in mind, we can define the code like this
each Record.Field(_, ScenarioFrom) - Record.Field(_, ScenarioTo)
instead of
each [#"Jan 15, 2021"] - [#"Mar 5, 2021"]
This question is similar but uses the value in a different column rather than parameters/variables:
PowerQuery choose values based on a key column
I'm scraping a website with different URLs (about 1000) and only able to get 25 query due to website limitation (of course!). I have used Function.InvokeAfter with Number.RandomBetween but no luck.
However, I found that if I wait/pause about an hour or so after the 25th URL, I could query again. Now, I would like to automate that part.
I created a column called Wait with a value on every 25th row on the URL_List table with a number between 60-90 so that every 25th URL, I can have it wait 60 to 90 mins but I don't know how to implement. If I could replace Number.RandomBetween(10,20) with Number.RandomBetween(10,[Wait]). Or also open to other ideas.
Here's what I have done so far:
Function for fGetResults:
let Scrape=(URL) =>
let
Source = Web.Page(Web.Contents(URL)),
Data0 = ()=>Source{0}[Data],
#"Transposed Table" = Table.Transpose(Function.InvokeAfter(Data0, #duration(0,0,0,Number.RandomBetween(10,20))))
in
#"Transposed Table"
in Scrape
Query:
let
Source = Excel.CurrentWorkbook(){[Name="URL_Table"]}[Content],
#"Added Custom" = Table.AddColumn(Source, "Custom", each fGetResults([URL_List]))
in
#"Added Custom"
Merci beaucoup...
#############
UPDATE
#############
I did some more digging and found this and this. Instead of a Wait column with a number from 60-90mins after 25th URL or so, I just place a "yes".
WaitMinutes function:
let
Wait = (minutes as number, action as function) =>
if (List.Count(List.Generate(() => DateTimeZone.LocalNow() + #duration(0,0,minutes,0), (x) => DateTimeZone.LocalNow() < x, (x) => x)) = 0) then null else action()
in
Wait
Now, I tried it but is now giving me an error Token Comma expected at the then statement on the New Query, any thoughts?
New Query:
let
Source = Excel.CurrentWorkbook(){[Name="URL_Table"]}[Content],
#"Added Custom" = Table.AddColumn(Source, "Custom", each [Wait] = "yes" then WaitMinutes(Number.RandomBetween(60,90), () => fGetResults([URL_List])) else fGetResults([URL_List]))
in
#"Added Custom"
#############
UPDATE2
#############
I think I found the issue. Simply forgot the "if" on the if-then-else statement. The only thing is, there maybe a more elegant solution for this but this works for now. Let me know if you have a more efficient solution. Thank you.
let
Source = Excel.CurrentWorkbook(){[Name="URL_Table"]}[Content],
#"Added Custom" = Table.AddColumn(Source, "Custom", each if [Wait] = "yes" then WaitMinutes(Number.RandomBetween(60,90), () => fGetResults([URL_List])) else fGetResults([URL_List]))
in
#"Added Custom"
#############
UPDATE3: SOLUTION
#############
More problems, I think my usage for minutes is not correct but followed exactly what the website said and used seconds and removed the random function on the Query.
WaitSeconds function:
let
Wait = (seconds as number, action as function) =>
if (List.Count(List.Generate(() => DateTimeZone.LocalNow() + #duration(0,0,0,seconds), (x) => DateTimeZone.LocalNow() < x, (x) => x)) = 0) then null else action()
in
Wait
Query (15mins seems to work which is same as 900sec):
let
Source = Excel.CurrentWorkbook(){[Name="URL_Table"]}[Content],
#"Added Custom" = Table.AddColumn(Source, "Custom", each if [Wait] = "yes" then WaitSeconds(900, () => fGetResults([URL_List])) else fGetResults([URL_List]))
in
#"Added Custom"
I have the following calendar table script:
let
StartDate = #date(2016, 1, 1),
EndDate = #date(2018, 12, 31),
CurrentDate = DateTime.Date(DateTime.FixedLocalNow()),
FiscalYearEndMonth = 6,
ListDates = List.Dates(StartDate, Number.From(EndDate - StartDate)+1, #duration(1,0,0,0)),
#"Converted to Table" = Table.FromList(ListDates, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Renamed Columns as Date" = Table.RenameColumns(#"Converted to Table",{{"Column1", "Date"}}),
#"Changed Type to Date" = Table.TransformColumnTypes(#"Renamed Columns as Date",{{"Date", type date}}),
#"Added Calendar MonthNum" = Table.AddColumn(#"Changed Type to Date", "MonthNum", each Date.Month([Date]), Int64.Type),
#"Added Calendar Year" = Table.AddColumn(#"Added Calendar MonthNum", "Year", each Date.Year([Date]), Int64.Type),
#"Added MonthYearNum" = Table.AddColumn(#"Added Calendar Year", "MonthYearNum", each [Year]*100 + [MonthNum] /*e.g. Sep-2016 would become 201609*/, Int64.Type),
#"Added Day Num" = Table.AddColumn(#"Added MonthYearNum", "DayNum", each Date.Day([Date])),
#"Changed Type1" = Table.TransformColumnTypes(#"Added Datekey",{{"DayNum", Int64.Type}}),
#"Added Datekey" = Table.AddColumn(#"Added Day Num", "DateKey", each [Year]*100 + [MonthNum] + [DayNum]),
#"Changed Type" = Table.TransformColumnTypes(#"Changed Type1",{{"DayNum", Int64.Type}, {"DateKey", Int64.Type}})
in
#"Changed Type"
It results in the following:
So the last column is causing me the headache - I think the code for that column is:
each [Year]*100 + [MonthNum] + [DayNum]
It seems like the first two elements of this expression get concatenated as expected but then DayNum is added e.g. for 1st of Jan we get 2016 concatenated with 1 to give 201601 but then the DayNum gets added mathematically to it, to give 201602 but I want it to give 20160101 (format YYYYMMDD)
Why is this happening and what is the correct M/PowerQuery to avoid this behavior?
I believe you are correct as far as what it's doing. How about this instead?
each [Year]*10000 + [MonthNum]*100 + [DayNum]
or you could do
each [MonthYearNum]*100 + [DayNum]
i need to filter a field which contains another field in the same query.
#"Filtered Rows" = Table.SelectRows(#"Filtered Rows1", each Text.Contains([ACIKLAMA], [SANTIYE]))
the error i got is
Expression.Error: The field 'SANTIYE' of the record wasn't found.
full code :
let
Source = Table.NestedJoin(Query1,{"Sicil No", "TARIH"},#"IK Bordro",{"Personel Kodu", "Bordro Tarihi"},"NewColumn",JoinKind.LeftOuter),
#"Expanded NewColumn" = Table.ExpandTableColumn(Source, "NewColumn", {"Santiye", "Taseron", "Turk/Yerel"}, {"Santiye", "Taseron", "Turk/Yerel"}),
#"Changed Type" = Table.TransformColumnTypes(#"Expanded NewColumn",{{"TARIH", type date}}),
#"Added Conditional Column" = Table.AddColumn(#"Changed Type", "Bordro", each if [#"Turk/Yerel"] = "YEREL" then "YEREL BORDRO" else if Text.Contains([ACIKLAMA], "RUBLE") then "TURK RUBLE" else if Text.Contains([ACIKLAMA], "USD") then "TURK USD" else if Text.Contains([ACIKLAMA], "RUB") then "TURK RUBLE" else if Text.Contains([ACIKLAMA], "IZIN") then "TURK IZIN HAKKI" else if Text.Contains([ACIKLAMA], "IHBAR") then "TURK IHBAR HAKKI" else if Text.Contains([ACIKLAMA], "KIDEM") then "TURK KIDEM HAKKI" else "DIGER" ),
#"Reordered Columns" = Table.ReorderColumns(#"Added Conditional Column",{"Sicil No", "HESAP ADI", "TARIH", "ACIKLAMA", "Santiye", "Taseron", "Turk/Yerel", "Bordro", "Ruble Tahakkuk", "USD Tahakkuk"}),
#"Filtered Rows1" = Table.SelectRows(#"Reordered Columns", each [Santiye] <> null),
#"Filtered Rows" = Table.SelectRows(#"Filtered Rows1", each Text.Contains([ACIKLAMA], [Santiye] ))
in
#"Filtered Rows"
any ideas, workarounds ?
Maybe you are mixing up step names and field names?
It works fine with me as you can see here.
Code generated:
let
Source = ..... (table created, code not relevant),
#"Changed Type" = Table.TransformColumnTypes(Source,{{"ACIKLAMA", type text}, {"SANTIYE", type text}}),
#"Filtered Rows" = Table.SelectRows(#"Changed Type", each Text.Contains([ACIKLAMA],[SANTIYE]))
in
#"Filtered Rows"