I'm scraping a website with different URLs (about 1000) and only able to get 25 query due to website limitation (of course!). I have used Function.InvokeAfter with Number.RandomBetween but no luck.
However, I found that if I wait/pause about an hour or so after the 25th URL, I could query again. Now, I would like to automate that part.
I created a column called Wait with a value on every 25th row on the URL_List table with a number between 60-90 so that every 25th URL, I can have it wait 60 to 90 mins but I don't know how to implement. If I could replace Number.RandomBetween(10,20) with Number.RandomBetween(10,[Wait]). Or also open to other ideas.
Here's what I have done so far:
Function for fGetResults:
let Scrape=(URL) =>
let
Source = Web.Page(Web.Contents(URL)),
Data0 = ()=>Source{0}[Data],
#"Transposed Table" = Table.Transpose(Function.InvokeAfter(Data0, #duration(0,0,0,Number.RandomBetween(10,20))))
in
#"Transposed Table"
in Scrape
Query:
let
Source = Excel.CurrentWorkbook(){[Name="URL_Table"]}[Content],
#"Added Custom" = Table.AddColumn(Source, "Custom", each fGetResults([URL_List]))
in
#"Added Custom"
Merci beaucoup...
#############
UPDATE
#############
I did some more digging and found this and this. Instead of a Wait column with a number from 60-90mins after 25th URL or so, I just place a "yes".
WaitMinutes function:
let
Wait = (minutes as number, action as function) =>
if (List.Count(List.Generate(() => DateTimeZone.LocalNow() + #duration(0,0,minutes,0), (x) => DateTimeZone.LocalNow() < x, (x) => x)) = 0) then null else action()
in
Wait
Now, I tried it but is now giving me an error Token Comma expected at the then statement on the New Query, any thoughts?
New Query:
let
Source = Excel.CurrentWorkbook(){[Name="URL_Table"]}[Content],
#"Added Custom" = Table.AddColumn(Source, "Custom", each [Wait] = "yes" then WaitMinutes(Number.RandomBetween(60,90), () => fGetResults([URL_List])) else fGetResults([URL_List]))
in
#"Added Custom"
#############
UPDATE2
#############
I think I found the issue. Simply forgot the "if" on the if-then-else statement. The only thing is, there maybe a more elegant solution for this but this works for now. Let me know if you have a more efficient solution. Thank you.
let
Source = Excel.CurrentWorkbook(){[Name="URL_Table"]}[Content],
#"Added Custom" = Table.AddColumn(Source, "Custom", each if [Wait] = "yes" then WaitMinutes(Number.RandomBetween(60,90), () => fGetResults([URL_List])) else fGetResults([URL_List]))
in
#"Added Custom"
#############
UPDATE3: SOLUTION
#############
More problems, I think my usage for minutes is not correct but followed exactly what the website said and used seconds and removed the random function on the Query.
WaitSeconds function:
let
Wait = (seconds as number, action as function) =>
if (List.Count(List.Generate(() => DateTimeZone.LocalNow() + #duration(0,0,0,seconds), (x) => DateTimeZone.LocalNow() < x, (x) => x)) = 0) then null else action()
in
Wait
Query (15mins seems to work which is same as 900sec):
let
Source = Excel.CurrentWorkbook(){[Name="URL_Table"]}[Content],
#"Added Custom" = Table.AddColumn(Source, "Custom", each if [Wait] = "yes" then WaitSeconds(900, () => fGetResults([URL_List])) else fGetResults([URL_List]))
in
#"Added Custom"
Related
I would like to transform Numbers to Excel Column Letters.
There are several threads on the topic but it seems that it has not been answered yet for M language.
I managed to address it with the following code but I am sure someone will have a much more efficient code to propose:
= Table.AddColumn(Source, "Column Letter", each Text.Combine({ if (Number.IntegerDivide(
Number.IntegerDivide(
Number.IntegerDivide([Column1]-1,1),26)-1,26))<1 then "" else Character.FromNumber( 64+Number.IntegerDivide(Number.IntegerDivide(Number.IntegerDivide([Column1]-1,1),26)-1,26)),
(if Number.IntegerDivide([Column1]-1,26)-
Number.Abs(
Number.IntegerDivide(
Number.IntegerDivide([Column1]-1,26)
-1,26)*26)<1 then "" else
Character.FromNumber(64+
Number.IntegerDivide([Column1]-1,26)-Number.Abs(
Number.IntegerDivide(
Number.IntegerDivide([Column1]-1,26)-1,26)*26))),
(if [Column1]-Number.Abs(
Number.IntegerDivide([Column1]-1,26)*26)<1 then "" else Character.FromNumber(64+[Column1]-Number.Abs(
Number.IntegerDivide([Column1]-1,26)*26)))}))
Thank you!
I think this custom function will work also:
Edit*Formula for b corrected as per #VassilisAgapitos note below
(n as number)=>
let
//make sure it is an integer
a = Number.IntegerDivide(n,1),
b = Number.IntegerDivide(Number.IntegerDivide(a-1,26)-1,26),
L1 = if b = 0 then null else Character.FromNumber(b+64),
c = Number.IntegerDivide(a-b*26*26-1,26),
L2 = if c = 0 then null else Character.FromNumber(c+64),
d = a-b*26*26-c*26,
L3 = Character.FromNumber(d+64)
in
Text.Combine({L1,L2,L3})
You can use it as a Table.AddColumn argument:
#"Added Custom" = Table.AddColumn(#"Changed Type", "Column Letter", each fnNumberToExcelColumn([Column Number]), type text)
An uglier version than Ron's, just for fun
=(if [Column1] <703 then "" else Character.FromNumber(64+Number.IntegerDivide([Column1]-27,26*26)))
&( if [Column1] >26 then Replacer.ReplaceText(Character.FromNumber(64+ Number.Mod( Number.IntegerDivide([Column1]-1,26) ,26) ), "#", "Z") else "" )
&(Character.FromNumber(65+Number.Mod([Column1]-1,26)))
I'm trying to load the text contents of an email from outlook in into PowerBI using Power Query. I want to specifically load only the contents of emails from a specified person, with a specified subject line, and only the most recent email.
Here's my code so far:
let
Source = Exchange.Contents("user#email.com"),
Mail1 = Source{[Name="Mail"]}[Data],
#"Filtered Rows" = Table.SelectRows(Mail1, each ([Folder Path] = "\Inbox\Project\")),
#"Sorted Rows" = Table.Sort(#"Filtered Rows",{{"DateTimeReceived", Order.Descending}}),
#"Filtered Rows1" = Table.SelectRows(#"Sorted Rows", each Text.Contains([Subject], "Speficied Email Subject")),
#"Expanded Sender" = Table.ExpandRecordColumn(#"Filtered Rows1", "Sender", {"Address"}, {"Sender.Address"}),
#"Filtered Rows2" = Table.SelectRows(#"Expanded Sender", each Text.Contains([Sender.Address], "specified email sender")),
#"Filtered Rows3" = Table.SelectRows(#"Filtered Rows2", each Text.Contains([Subject], "specified email subject")),
#"Kept First Rows" = Table.FirstN(#"Filtered Rows3",1),
#"Choose First Result's Content" = #"Kept First Rows"[Body],
TextBody = #"Choose First Result's Content"[TextBody]
in
TextBody
I'm getting an error:
Expression.Error: We cannot apply field access to the type List.
Details:
Value=[List]
Key=TextBody
Can someone help me correct this code?
Thanks!
Change the last step to TextBody = #"Choose First Result's Content"{0}[TextBody]
I have a Date/Time column in format "13.10.2020 06:08:43" and I need to calculate the "age" from current time. Output should be like from given example: "4h" since its 10:08 atm - Date/Time column was 4h ago. I need this to be in M language to use in visual studio 2019.
I used:
let
Source = Sql.Database("xxxx", "xxxxx", [Query="SELECT#(lf)#(tab)storeid as 'Shop'#(lf)#(tab), concat('SCO', posnr) as POS#(lf)#(tab), concat(datediff(hour,[LastTransactionDate]#(lf)#(tab), CURRENT_TIMESTAMP),' h') as 'Age'#(lf)#(tab), POSIPaddress as 'IP'#(lf)#(tab),[PosNr]#(lf) ,[AdapterPosOrGroupId]#(lf) ,[UpdatedOn]#(lf) ,[LastTransactionDate]#(lf) ,[LastStartDate]#(lf) ,[Open]#(lf) ,[Locked]#(lf) ,[CashDisabled]#(lf) ,[IsCashEnabled]#(lf) ,[CashDevicesTraficLight]#(lf) ,[POSIPAddress]#(lf) ,[ScoPosServiceVersion]#(lf) ,[WinScoVersion]#(lf) ,[StoreType]#(lf)FROM [LaneEventDatabase ].[dbo].[POS.LaneCurrentState]"]),
#"Added Conditional Column" = Table.AddColumn(Source, "Custom", each if [CashDevicesTraficLight] = "green" then 1 else if [CashDevicesTraficLight] = "yellow" then 2 else if [CashDevicesTraficLight] = "red" then 3 else 4),
#"Changed Type" = Table.TransformColumnTypes(#"Added Conditional Column",{{"Custom", type number}})
in
#"Changed Type"
which works just great but visual studio doesnt like the way source is handled and while deploying gives me error:
Failed to save modifications to the server. Error returned: 'An M partition uses a data function which results in access to a data source different from those defined in the model.
'.
Just create a new Custom Column using this below code-
= Text.From(
(Duration.Days(DateTime.LocalNow()-[date]) * 24) +
Duration.Hours(DateTime.LocalNow()-[date])
) & "h"
Here is sample output-
I am trying to get data from my email and the data is attached to an email and compressed in a zip folder.Is there a way to extract the contents of the zip folder from my email and into power query?
This is what I have so far
let Source = Exchange.Contents("xxxxx#xxx.com"), Mail1 = Source{[Name="Mail"]}[Data], #"Filtered Rows" = Table.SelectRows(Mail1, each ([Folder Path] = "\xxxx Report\")), #"Filtered Rows1" = Table.SelectRows(#"Filtered Rows", let latest = List.Max(#"Filtered Rows"[DateTimeReceived]) in each [DateTimeReceived] = latest), #"Removed Other Columns" = Table.SelectColumns(#"Filtered Rows1",{"Attachments"}), #"Added Custom" = Table.AddColumn(#"Removed Other Columns", "Custom", each UnzipContent([Attachments])) in #"Added Custom"
This gives me an error.
Please assist,
Vava
I have the following calendar table script:
let
StartDate = #date(2016, 1, 1),
EndDate = #date(2018, 12, 31),
CurrentDate = DateTime.Date(DateTime.FixedLocalNow()),
FiscalYearEndMonth = 6,
ListDates = List.Dates(StartDate, Number.From(EndDate - StartDate)+1, #duration(1,0,0,0)),
#"Converted to Table" = Table.FromList(ListDates, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Renamed Columns as Date" = Table.RenameColumns(#"Converted to Table",{{"Column1", "Date"}}),
#"Changed Type to Date" = Table.TransformColumnTypes(#"Renamed Columns as Date",{{"Date", type date}}),
#"Added Calendar MonthNum" = Table.AddColumn(#"Changed Type to Date", "MonthNum", each Date.Month([Date]), Int64.Type),
#"Added Calendar Year" = Table.AddColumn(#"Added Calendar MonthNum", "Year", each Date.Year([Date]), Int64.Type),
#"Added MonthYearNum" = Table.AddColumn(#"Added Calendar Year", "MonthYearNum", each [Year]*100 + [MonthNum] /*e.g. Sep-2016 would become 201609*/, Int64.Type),
#"Added Day Num" = Table.AddColumn(#"Added MonthYearNum", "DayNum", each Date.Day([Date])),
#"Changed Type1" = Table.TransformColumnTypes(#"Added Datekey",{{"DayNum", Int64.Type}}),
#"Added Datekey" = Table.AddColumn(#"Added Day Num", "DateKey", each [Year]*100 + [MonthNum] + [DayNum]),
#"Changed Type" = Table.TransformColumnTypes(#"Changed Type1",{{"DayNum", Int64.Type}, {"DateKey", Int64.Type}})
in
#"Changed Type"
It results in the following:
So the last column is causing me the headache - I think the code for that column is:
each [Year]*100 + [MonthNum] + [DayNum]
It seems like the first two elements of this expression get concatenated as expected but then DayNum is added e.g. for 1st of Jan we get 2016 concatenated with 1 to give 201601 but then the DayNum gets added mathematically to it, to give 201602 but I want it to give 20160101 (format YYYYMMDD)
Why is this happening and what is the correct M/PowerQuery to avoid this behavior?
I believe you are correct as far as what it's doing. How about this instead?
each [Year]*10000 + [MonthNum]*100 + [DayNum]
or you could do
each [MonthYearNum]*100 + [DayNum]