I created PowerBI report which which is connecting to data source via API service. Returning json contains thousands of entities. API service is called via Web.Content function. API service returns always total record count and so we are able to calculate nr. of pages which has to be called to obtain whole dataset. This report is displaying data from our servicedesk app, which is deployed on many servers and for many customers and use Query parameters to connect to any of these servers.
Detail of Power query is below.
Why am I writing here. This report was working without any issue more than 1,5 year but on August 17th one of servers start causing erros in step Pages where are some random lines (pages) with errors - see attached picture labeled "Errors in step Pages". and this is reason that next step Entities (List.Union) in query is stopping refresh and generate errors with message:
Expression.Error: We cannot apply field access to the type List. Details: Value=[List] Key=requests
What is notable
API service si returning records in the same order but faulty lists are random when calling with same parameters
some times is refresh without any error
The same power query called on another server is working correctly , problem is only with one specific server.
This problem started without notice on the most important server after 1,5 year without any problem.
Here is full text power of query for this main source, which is used later in other queries to extract all necessary data. Json is really complicated and I extract from it list of requests, list of solvers, list of solver groups,.... and this base query and its output is input for many referenced queries.
Errors in step Pages
let
BaseAPIUrl = apiurl&"apiservice?", /*apiurl is parameter - name of server e.g. https://xxxx.xxxxxx.sk/ */
EntitiesPerPage = RecordsPerPage, /*RecordsPerPage is parameter and defines nr. of record per page - we used as optimum 200-400 record per pages, but is working also with 4000 record per page*/
ApiToken = FnApiToken(), /*this function is returning apitoken value which is returning value of another api service apiurl&"api/auth/login", which use username and password in body of call to get apitoken */
GetJson = (QParm) => /*definiton general function to get data from data source*/
let
Options =
[ Query= QParm,
Headers=
[
Accept="application/json",
ApiKeyName="apitoken",
Authorization=ApiToken
]
],
RawData = Web.Contents(BaseAPIUrl, Options),
Json = Json.Document(RawData)
in Json,
GetEntityCount = () => /*one times called function to get nr of records using GetJson, which is returned as a part of each call*/
let
QParm = [pp="1", pg="1" ],
Json = GetJson(QParm),
Count = Json[totalRecord]
in
Count,
GetPage = (Index) => /*repeatadly called function to get each page of json using GetJson*/
let
PageNr = Text.From(Index+1),
PerPage = Text.From(EntitiesPerPage),
QParm = [pg = PageNr, pp=PerPage],
Json = GetJson(QParm),
Value = Json[data][requests]
in Value,
EntityCount = List.Max({ EntitiesPerPage, GetEntityCount() }), /*setup of nr. of records to variable*/
PageCount = Number.RoundUp(EntityCount / EntitiesPerPage), /*setup of nr. of pages */
PageIndices = { 0 .. PageCount - 1 },
Pages = List.Transform(PageIndices, each GetPage(_) /*Function.InvokeAfter(()=>GetPage(_),#duration(0,0,0,1))*/), /*here we call for each page GetJson function to get whole dataset - there is in comment test with delay between getpages but was not neccessary*/
Entities = List.Union(Pages),
Table = Table.FromList(Entities, Splitter.SplitByNothing(), null, null, ExtraValues.Error)
I also tried another way of appending pages to list using List.Generate. This is also bringing random errors in list but
it is bringing possibility to transform to table in contrast with original way with using List.Transform, but other referenced queries are failing and contains on the last row errors
When I am exploring content of faulty page/list extracting it via Add as New Query there are always all record without any fail.....
Source = List.Generate( /*another way to generate list of all pages*/
() => [Page = 0, ReqPageData = GetPage(0) ],
each [Page] < PageCount,
each [ReqPageData = GetPage( [Page] ),
Page = [Page] + 1 ],
each [ReqPageData]
),
#"Converted to Table" = Table.FromList(Source, Splitter.SplitByNothing(), null, null, ExtraValues.Error), /*here i am able to generate table from list in contrast when is used List.Generate*/
#"Expanded Column1" = Table.ExpandListColumn(#"Converted to Table", "Column1"), /*here aj can expand list to column*/
#"Removed Errors" = Table.RemoveRowsWithErrors(#"Expanded Column1", {"Column1"}) /*here i try to exclude errors, but i dont know what happend and which records (if any) are excluded*/
Extracting errored page
and finnaly I am tottaly clueless not able to find the cause of this behavior on this specific server. I tested to call pages which are errored via POSTMAN, I discused this issue with author of API service and He also tried to call this API service with all parameters but server is returning every page OK, only Power query is not able to List.Transform ...
I will be grateful and appreciate any tips or advice or if somebody solved the same issue in the past ....
Kuby
No, each error line of list in step List.Transform coud by extracted as new query and there are all records from one page OK. hmmmm
Finnaly, problem described in this issue was caused by "corrupted" content of returning json. The provider of core system informed me that they found bug and after fixing on the side of servisdesk is everything OK again. I tried to find problem in Power query and problem was in servisdesk. :(
I have aprox 520 classrooms archived in my account, if I try to select them with
var courseList = Classroom.Courses.list({"courseStates":["ARCHIVED"]}).courses;
I get only 300 of them. Is this normal?
How can I select them all? Actually I'm writing a script to delete the oldest, but if I can't retrieve them, I can't delete them.
I understand that you got so many courses that the Courses.list() response is splitted in separate pages. In that case you can very easily navigate them by using tokens. First of all, make sure that you specify the pageSize in your request. That would set the desired amount of responses per page. Please keep in mind that the server may return fewer than the specified number of results, as it declared on the docs. In case that your response got divided into pages, the response would include the nextPageToken field. Then, to obtain the rest of courses, you have to repeat your request including that nextPageToken into the pageToken property. Please don't hesitate to ask me any doubt about this approach.
Thanks a lot Jaques, I found the solution:
var parametri = {"courseStates": "ARCHIVED"};
var page = Classroom.Courses.list(parametri);
var listaClassi = page.courses;
if (page.nextPageToken !== '') {
parametri.pageToken = page.nextPageToken;
page = Classroom.Courses.list(parametri);
listaClassi = listaClassi.concat(page.courses);
}
Anyway, I didn't need to change the pageSize, nor I found any tutorial about it.
I want to get mentions of a any user using linq to twitter api. I have a code which brings back the user time line, but I want to get only mentions of a user, one of the method that is Mention Timeline can only be used to get mentions for authenticatd user but I want to get for any user. How it is possible? Here is my code which return a user timeline, what modification require in it? Please help
var mentions= await _twitterContext.Status
.Where(s => s.ScreenName== ScreenName && s.Type == StatusType.User && s.Count == 200)
.ToListAsync();
The Twitter API doesn't expose an endpoint that allows you to get mentions for any user with a screen name. However, you can get the mentions if you are authorized by the user for whom you want to get mentions for. Here's an example:
var tweets =
await
(from tweet in twitterCtx.Status
where tweet.Type == StatusType.Mentions &&
tweet.ScreenName == "JoeMayo"
select tweet)
.ToListAsync();
PrintTweetsResults(tweets);
The documentation for this is here:
http://linqtotwitter.codeplex.com/wikipage?title=Querying%20the%20Mentions%20Timeline&referringTitle=Making%20Status%20Queries%20and%20Calls
I'm using the koala gem as show in Railscasts episode #361. I'm attempting to get all of the comments of a given Post but Facebook only seems to be giving me back the last 50 comments on the post. Is this a limitation of Facebook's Graph API or am I doing something wrong?
fb = Koala::Facebook::API.new oauth_token
post = fb.get_object(id_of_the_post)
comments = fb.get_object(post['id'])['comments']['data']
puts comments.size # prints 50
Graph API paginates the result when is a larger number of posts than the limit that is set (in your case 50).
In order to access the next page of results, call "next_page" method:
comments = fb.get_object(post['id'])
while comments['comments']['data'].present?
# Make operations with your results
comments = comments.next_page
end
Also, by looking in source one can see that "get_object" method receives 3 parameters:
def get_object(id, args = {}, options = {})
This way, you can raise your posts per page to as many posts as you want:
comments = fb.get_object(post['id'], {:limit => 1000})
Is there a way to use a LINQ expression to request a Count query from the Netflix oData service in Silverlight 4?
The Netflix documentation shows that you can return counts by appending $count to a request for a collection, but a URL like this:
http://netflix.cloudapp.net/Catalog/Genres/$count
Is not generated from an expression like this:
var count = (from g in catalog.Genres select g).Count();
The above code returns an error saying that the Count method is not supported. Is there a way to do this in LINQ, or do I just need to make WebClient request to get the value?
Count and LongCount are not supported in Silverligth because they require a synchornous execution of the query. Since Silverlight requires all network operations to by asynchronous this is not possible.
You can either issue the HTTP query in question programatically not using DataServiceContext (or related classes), since the $count returns a text representation of the number, parsing the response is not that hard.
Or you can use a bit of a trick. You can use IncludeTotalCount() to add $inlinecount=allpages query option to the query which will include the count in the response. Then to not download all the entities from server, you can use Take(0) which will add $top=0 and thus return empty result set. But the inline count will still contain the right number.
You can access the inline count on the QueryOperationResponse.TotalCount property.
Something like this:
NetflixCatalog ctx = new NetflixCatalog(new Uri("http://netflix.cloudapp.net/Catalog"));
var q = (DataServiceQuery<Genre>)ctx.Genres.IncludeTotalCount().Take(0);
q.BeginExecute((ar) =>
{
QueryOperationResponse<Genre> r = (QueryOperationResponse<Genre>)q.EndExecute(ar);
r.TotalCount.ToString(); // Use the count in whatever way you need
}, null);
It works in LinqPad 4 using C# 4.0
var count = (from g in Genres select g).Count();
count.Dump();
Result: 518
In LinqPad 2 using C# 3.0 the error appears.