Filter on date in PowerQuery (PowerBI) - filter

Question

I'm currently getting to much data from my cosmosDB, which I want to reduce to the last 8 weeks.
How can I filter in PowerQuery to get the last 8 weeks based on my date column.
This is my powerquery to get the data:
let
Source = DocumentDB.Contents("https://xxx.xxx", "xxx", "xxx"),
#"Expanded Document" = Table.ExpandRecordColumn(Source, "Document", {"$v"}, {"Document.$v"}),
#"Expanded Document.$v" = Table.ExpandRecordColumn(#"Expanded Document", "Document.$v", {"date"}, {"Document.$v.date"}),
#"Expanded Document.$v.date" = Table.ExpandRecordColumn(#"Expanded Document.$v", "Document.$v.date", {"$v"}, {"Document.$v.date.$v"}),
#"Changed Type" = Table.TransformColumnTypes(#"Expanded Document.$v.date",{{"Document.$v.date.$v", type text}})
in
#"Changed Type"
And this is how the data is in my CosmosDB:
{
"_id" : ObjectId("5c6144bdf7ce070001acc213"),
"date" : {
"$date" : 1549792055030
},

Answer 1

If you want to do all the work on your end (maybe the server can do some/all of it):
Assuming the 1549792055030 (shown in example) is a Unix timestamp expressed in milliseconds, to convert to a datetime in Power Query, try something like: #datetime(1970, 1, 1, 0, 0, 0) + #duration(0, 0, 0, 1549792055030/1000)
You seem to expand a record field named $v (which itself was nested within a field named date, which itself was nested within a field named $v) in your M code, but $v is not shown as being present in the structure. I mention this as it's confusing to know whether to follow your M code or the structure. I'm going to assume that you have $v field, which contains a date field, which itself contains a $date field. To get at the nested Unix timestamp, you could try something like: someRecord[#"$v"][date][#"$date"]
Since you're interested in only the last 8 weeks, you could test for something like: Date.IsInPreviousNWeeks(DateTime.AddZone(someDatetime, 0), 8). (You could also do it the other way, by converting 8 weeks ago before now to a Unix timestamp and then filter for timestamps >= to the value you've worked out.)
Putting the above together, we might get some M code that looks like:
let
Source = DocumentDB.Contents("https://xxx.xxx", "xxx", "xxx"),
filterDates = Table.SelectRows(Source, each
let
millisecondsSinceEpoch = Number.From([document][#"$v"][date][#"$date"]),
toDatetime = #datetime(1970, 1, 1, 0, 0, 0) + #duration(0, 0, 0, millisecondsSinceEpoch/1000),
toFilter = Date.IsInPreviousNWeeks(DateTime.AddZone(toDatetime, 0), 8)
in toFilter
)
in filterDates
The code above may be functional (hopefully) but, conceptually, it might not be the right way to do it. I am not familiar with the function DocumentDB.Contents, but this link (https://www.powerquery.io/accessing-data/document-db/documentdb.contents) suggests it has these parameters:
function (url as text, optional database as nullable any, optional
collection as nullable any, optional options as nullable record) as
table
and it goes on to say:
if the field Query is specified in the options record the results of
the query being executed on either the specified database and/or
collection will be returned.
What I understand this to mean is that if you change your first line to something like:
Source = DocumentDB.Contents("https://xxx.xxx", "xxx", "xxx", [Query = "..."])
and the query you specify in "..." is understood by the server (presume the query needs be in Cosmos DB's native query language), only the last 8 weeks' worth of data will be returned to you (meaning less data needs sending and less work for you). As I said, I'm unfamiliar with Azure Cosmos DB, so I can't really comment further. But this seems the better way of doing it.