Fix an empty list in PowerQuery to a RESTFUL API - powerquery

Question

Forgive me I am very novice, I am experimenting with PQ in Excel to pull some sales data from a REST API https://docs.vendhq.com/reference/2/spec/sales/listsales
My query looks like the below
let
MaxVersion = Excel.CurrentWorkbook(){[Name="Versions"]}[Content]{0}[Sales],
Source = Json.Document(Web.Contents("https://*****.vendhq.com/api/2.0/sales?after=" &Text.From(MaxVersion) , [Headers=[Authorization="Bearer **********************************"]])),
data = Source[data],
#"Converted to Table" = Table.FromList(data, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Expanded Column1" = Table.ExpandRecordColumn(#"Converted to Table", "Column1", {"id", "outlet_id", "register_id", "user_id", "customer_id", "invoice_number", "source", "source_id", "status", "note", "short_code", "return_for", "total_price", "total_tax", "total_loyalty", "created_at", "updated_at", "sale_date", "deleted_at", "line_items", "payments", "adjustments", "version", "receipt_number", "total_price_incl", "taxes"}, {"Column1.id", "Column1.outlet_id", "Column1.register_id", "Column1.user_id", "Column1.customer_id", "Column1.invoice_number", "Column1.source", "Column1.source_id", "Column1.status", "Column1.note", "Column1.short_code", "Column1.return_for", "Column1.total_price", "Column1.total_tax", "Column1.total_loyalty", "Column1.created_at", "Column1.updated_at", "Column1.sale_date", "Column1.deleted_at", "Column1.line_items", "Column1.payments", "Column1.adjustments", "Column1.version", "Column1.receipt_number", "Column1.total_price_incl", "Column1.taxes"}),
#"Expanded Column1.line_items" = Table.ExpandListColumn(#"Expanded Column1", "Column1.line_items"),
#"Expanded Column1.line_items1" = Table.ExpandRecordColumn(#"Expanded Column1.line_items", "Column1.line_items", {"id", "product_id", "tax_id", "discount_total", "discount", "price_total", "price", "cost_total", "cost", "tax_total", "tax", "quantity", "loyalty_value", "note", "price_set", "status", "sequence", "gift_card_number", "tax_components", "promotions", "total_tax", "total_cost", "total_discount", "total_loyalty_value", "total_price", "unit_cost", "unit_discount", "unit_loyalty_value", "unit_price", "unit_tax", "is_return"}, {"Column1.line_items.id", "Column1.line_items.product_id", "Column1.line_items.tax_id", "Column1.line_items.discount_total", "Column1.line_items.discount", "Column1.line_items.price_total", "Column1.line_items.price", "Column1.line_items.cost_total", "Column1.line_items.cost", "Column1.line_items.tax_total", "Column1.line_items.tax", "Column1.line_items.quantity", "Column1.line_items.loyalty_value", "Column1.line_items.note", "Column1.line_items.price_set", "Column1.line_items.status", "Column1.line_items.sequence", "Column1.line_items.gift_card_number", "Column1.line_items.tax_components", "Column1.line_items.promotions", "Column1.line_items.total_tax", "Column1.line_items.total_cost", "Column1.line_items.total_discount", "Column1.line_items.total_loyalty_value", "Column1.line_items.total_price", "Column1.line_items.unit_cost", "Column1.line_items.unit_discount", "Column1.line_items.unit_loyalty_value", "Column1.line_items.unit_price", "Column1.line_items.unit_tax", "Column1.line_items.is_return"})
in
#"Expanded Column1.line_items1"
The query works great once, it passes a parameter from the excel workbook to use as the version for pulling in sales, after the last refresh (which gets populated from a second query, not sure if its possible to combine the queries??).
The other problem is if on the refresh the data is empty (no new sales) or if it needs to paginate the data I cant quite workout the syntax of how to iteratively call this as a function and include an escape clause for if the data returned is empty.

Answer 1

Seems appropriate to use List.Generate here.
I can't test the code below (as I don't have the API credentials or know the exact endpoint), so you'll need to check if it returns all/expected pages.
let
getSalesData = (maxVersion as number) =>
let
url = "https://YOUR_DOMAIN_PREFIX.vendhq.com/api/2.0/sales",
requestOptions = [
Query = [after = Number.ToText(maxVersion)],
Headers = [Authorization = "Bearer YOUR_API_KEY"]
],
response = Web.Contents(url, requestOptions),
deserialised = Json.Document(response)
in deserialised,
allPagesOfSalesData = List.Generate(
() => getSalesData(0),
each not List.IsEmpty([data]),
each getSalesData([version][max])
)
in
allPagesOfSalesData
I pass in 0 as an argument for the first API call even though the documentation (https://docs.vendhq.com/reference/introduction/pagination#first-page) states:
"By default, the value of the after parameter will be assumed as equal 0 so it’s not necessary to use it on the first page."
The list allPagesOfSalesData stops being generated when the data field contains an empty list, which I think is what the documentation (https://docs.vendhq.com/reference/introduction/pagination#subsequent-pages) suggests:
"This should be repeated until an empty collection is returned. This will mean that all items of the collection have been returned."
But I've assumed that "empty collection" refers to the data field (in the JSON response).
You should get a list of tables (for allPagesOfSalesData), which you can then combine into a single table (using Table.Combine).
You might want to use Table.FromRecords (instead of the Table.FromList and Table.ExpandRecordColumn) when transforming the response.
Hopefully it works, but I can't test it.