I created PowerBI report which which is connecting to data source via API service. Returning json contains thousands of entities. API service is called via Web.Content function. API service returns always total record count and so we are able to calculate nr. of pages which has to be called to obtain whole dataset. This report is displaying data from our servicedesk app, which is deployed on many servers and for many customers and use Query parameters to connect to any of these servers.
Detail of Power query is below.
Why am I writing here. This report was working without any issue more than 1,5 year but on August 17th one of servers start causing erros in step Pages where are some random lines (pages) with errors - see attached picture labeled "Errors in step Pages". and this is reason that next step Entities (List.Union) in query is stopping refresh and generate errors with message:
Expression.Error: We cannot apply field access to the type List. Details: Value=[List] Key=requests
What is notable
API service si returning records in the same order but faulty lists are random when calling with same parameters
some times is refresh without any error
The same power query called on another server is working correctly , problem is only with one specific server.
This problem started without notice on the most important server after 1,5 year without any problem.
Here is full text power of query for this main source, which is used later in other queries to extract all necessary data. Json is really complicated and I extract from it list of requests, list of solvers, list of solver groups,.... and this base query and its output is input for many referenced queries.
Errors in step Pages
let
BaseAPIUrl = apiurl&"apiservice?", /*apiurl is parameter - name of server e.g. https://xxxx.xxxxxx.sk/ */
EntitiesPerPage = RecordsPerPage, /*RecordsPerPage is parameter and defines nr. of record per page - we used as optimum 200-400 record per pages, but is working also with 4000 record per page*/
ApiToken = FnApiToken(), /*this function is returning apitoken value which is returning value of another api service apiurl&"api/auth/login", which use username and password in body of call to get apitoken */
GetJson = (QParm) => /*definiton general function to get data from data source*/
let
Options =
[ Query= QParm,
Headers=
[
Accept="application/json",
ApiKeyName="apitoken",
Authorization=ApiToken
]
],
RawData = Web.Contents(BaseAPIUrl, Options),
Json = Json.Document(RawData)
in Json,
GetEntityCount = () => /*one times called function to get nr of records using GetJson, which is returned as a part of each call*/
let
QParm = [pp="1", pg="1" ],
Json = GetJson(QParm),
Count = Json[totalRecord]
in
Count,
GetPage = (Index) => /*repeatadly called function to get each page of json using GetJson*/
let
PageNr = Text.From(Index+1),
PerPage = Text.From(EntitiesPerPage),
QParm = [pg = PageNr, pp=PerPage],
Json = GetJson(QParm),
Value = Json[data][requests]
in Value,
EntityCount = List.Max({ EntitiesPerPage, GetEntityCount() }), /*setup of nr. of records to variable*/
PageCount = Number.RoundUp(EntityCount / EntitiesPerPage), /*setup of nr. of pages */
PageIndices = { 0 .. PageCount - 1 },
Pages = List.Transform(PageIndices, each GetPage(_) /*Function.InvokeAfter(()=>GetPage(_),#duration(0,0,0,1))*/), /*here we call for each page GetJson function to get whole dataset - there is in comment test with delay between getpages but was not neccessary*/
Entities = List.Union(Pages),
Table = Table.FromList(Entities, Splitter.SplitByNothing(), null, null, ExtraValues.Error)
I also tried another way of appending pages to list using List.Generate. This is also bringing random errors in list but
it is bringing possibility to transform to table in contrast with original way with using List.Transform, but other referenced queries are failing and contains on the last row errors
When I am exploring content of faulty page/list extracting it via Add as New Query there are always all record without any fail.....
Source = List.Generate( /*another way to generate list of all pages*/
() => [Page = 0, ReqPageData = GetPage(0) ],
each [Page] < PageCount,
each [ReqPageData = GetPage( [Page] ),
Page = [Page] + 1 ],
each [ReqPageData]
),
#"Converted to Table" = Table.FromList(Source, Splitter.SplitByNothing(), null, null, ExtraValues.Error), /*here i am able to generate table from list in contrast when is used List.Generate*/
#"Expanded Column1" = Table.ExpandListColumn(#"Converted to Table", "Column1"), /*here aj can expand list to column*/
#"Removed Errors" = Table.RemoveRowsWithErrors(#"Expanded Column1", {"Column1"}) /*here i try to exclude errors, but i dont know what happend and which records (if any) are excluded*/
Extracting errored page
and finnaly I am tottaly clueless not able to find the cause of this behavior on this specific server. I tested to call pages which are errored via POSTMAN, I discused this issue with author of API service and He also tried to call this API service with all parameters but server is returning every page OK, only Power query is not able to List.Transform ...
I will be grateful and appreciate any tips or advice or if somebody solved the same issue in the past ....
Kuby
No, each error line of list in step List.Transform coud by extracted as new query and there are all records from one page OK. hmmmm
Finnaly, problem described in this issue was caused by "corrupted" content of returning json. The provider of core system informed me that they found bug and after fixing on the side of servisdesk is everything OK again. I tried to find problem in Power query and problem was in servisdesk. :(
For table cmdb_rel_ci, I want to retrieve unique parent.sys_class_name with count for "type=In Rack::Rack contains". I am doing practice in out of the box instance.
At table level URL is as below:
URL
I want to retrieve result from above URL with my below script.
var count = new GlideAggregate('cmdb_rel_ci');
count.addQuery('type','e76b8c7b0a0a0aa70082c9f7c2f9dc64');// sys_id of type In Rack::Rack contains e76b8c7b0a0a0aa70082c9f7c2f9dc64
count.addAggregate('COUNT', 'parent.sys_class_name');
count.query();
while(count.next()){
var parentClassName = count.parent.sys_class_name.toString();
var parentClassNameCount = count.getAggregate('COUNT','parent.sys_class_name');
gs.log(parentClassName + " : " + parentClassNameCount );
}
The issue is I am getting parentClassName empty.
Try this instead:
var parentClassName = count.getValue("parent.sys_class_name")
Since it's a GlideAggregate query (instead of GlideRecord), the query being issued isn't returning all of the fields on the target table. With GlideRecord, dot-walking through a reference field (e.g. parent.sys_class_name) automatically resolves that referenced record to provide access to its field values. This is made possible by the fact that the driving/original query brought back the value of the parent field. This is not happening with GlideAggregate. The query in this case basically looks like:
SELECT cmdb1.`sys_class_name` AS `parent_sys_class_name`, count(*)
FROM (cmdb_rel_ci cmdb_rel_ci0 LEFT JOIN cmdb cmdb1 ON cmdb_rel_ci0.`parent` = cmdb1.`sys_id` )
WHERE cmdb_rel_ci0.`type` = 'e76b8c7b0a0a0aa70082c9f7c2f9dc64'
GROUP BY cmdb1.`sys_class_name`
ORDER BY cmdb1.`sys_class_name`
So, you actually have access specifically to that dot-walked sys_class_name that's being grouped, but not through the dot-walk. The call to getValue("parent.sys_class_name") is expectedly resolved to the returned column aliased as parent_sys_class_name.
That being said, what you're doing probably should also work, based on user expectations, so you've not done anything incorrect here.
I have two columns one is from user and other is to user. now i am checking if current user is in from user or to user. i am querying like the following.
var matQueryFrom = new Parse.Query(Parse.Object.extend("VRMatches"));
var matQueryTo = new Parse.Query(Parse.Object.extend("VRMatches"));
matQueryFrom.equalTo("FROM_USER_OBJECTID",user.id);
matQueryFrom.include("TO_USER");
matQueryTo.equalTo("TO_USER_OBJECTID", user.id);
matQueryTo.include("FROM_USER");
var mainQuery = Parse.Query.or(matQueryFrom, matQueryTo);
i want to get from user if the current user matched to user. OR i want to get to user if the current user matches from user. How can i achieve that or what i am doing wrong?
You need to read the Parse documentation becuase it clearly says:
Note that we do not, however, support GeoPoint or non-filtering
constraints (e.g. near, withinGeoBox, limit, skip,
ascending/descending, include) in the subqueries of the compound
query.
You might be able to add both of the include constraints to the final compound query. Maybe this will work but obviously you will not be able to distinguish which sub query returned which row as the results of both sub queries were Ored together:
var matQueryFrom = new Parse.Query("VRMatches");
var matQueryTo = new Parse.Query("VRMatches");
matQueryFrom.equalTo("FROM_USER_OBJECTID",user.id);
matQueryTo.equalTo("TO_USER_OBJECTID", user.id);
var mainQuery = Parse.Query.or(matQueryFrom, matQueryTo);
mainQuery.include("TO_USER");
mainQuery.include("FROM_USER");
I have a relatively standard Parse query to obtain the nearest items in the 'checkIn' class to an object's location as follows:
var checkInObject = Parse.Object.extend('checkIn');
var query = new Parse.Query(checkInObject);
query.near('location', request.object.get('location'));
query.limit(Parse.User.current().get('limit'));
This works as expected, however can return checkins of the user carrying out the request. To avoid this I add in an additional constraint so my code now looks like this:
var checkInObject = Parse.Object.extend('checkIn');
var query = new Parse.Query(checkInObject);
query.near('location', request.object.get('location'));
query.notEqualTo('user', Parse.User.current());
query.limit(Parse.User.current().get('limit'));
But with this addition I no longer receive any results from the query. How can I add the constraint and still obtain the required results?
My suggestion:
Add a string column called "ownerID" and assigned each checkin the value of the objectId of the parse user that created it. Then, you can do this:
var checkInObject = Parse.Object.extend('checkIn');
var query = new Parse.Query(checkInObject);
query.near('location', request.object.get('location'));
query.limit(Parse.User.current().get('limit'));
query.notEqualTo('ownerID', Parse.User.current().get('objectId'));
You're also gonna want to make sure that the line where you get 'limit' returns a valid value.
In the table ReservationWorkerPeriods there are records of all workers that are planned to work on a given period on any possible machine.
The additional table WorkerOnMachineOnConstructionSite contains columns workerId, MachineId and ConstructionSiteId.
From the table ReservationWorkerPeriods I would like to retrieve just workers who work on selected machine.
In order to retrieve just relevant records from WorkerOnMachineOnConstructionSite table I have written the following code:
var relevantWorkerOnMachineOnConstructionSite = (from cswm in currentConstructionSiteSchedule.ContrustionSiteWorkerOnMachine
where cswm.MachineId == machineId
select cswm).ToList();
workerOnMachineOnConstructionSite = relevantWorkerOnMachineOnConstructionSite as List<ContrustionSiteWorkerOnMachine>;
These records are also used in the application so I don't want to bypass the above code even if is possible to directly retrieve just workerPeriods for workers who work on selected machine. Anyway I haven't figured out how it is possible to retrieve the relevant workerPeriods once we know which userIDs are relevant.
I have tried the following code:
var userIDs = from w in workerOnMachineOnConstructionSite select new {w.WorkerId};
List<ReservationWorkerPeriods> workerPeriods = currentConstructionSiteSchedule.ReservationWorkerPeriods.ToList();
allocatedWorkers = workerPeriods.Where(wp => userIDs.Contains(wp.WorkerId));
but it seems to be incorrect and don't know how to fix it. Does anyone know what is the problem and how it is possible to retrieve just records which contain userIDs from the list?
Currently, you are constructing an anonymous object on the fly, with one property. You'll want to grab the id directly with (note the missing curly braces):
var userIDs = from w in workerOnMachineOnConstructionSite select w.WorkerId;
Also, in such cases, don't call ToList on it - the variable userIDs just contains the query, not the result. If you use that variable in a further query, the provider can translate it to a single sql query.