I created PowerBI report which which is connecting to data source via API service. Returning json contains thousands of entities. API service is called via Web.Content function. API service returns always total record count and so we are able to calculate nr. of pages which has to be called to obtain whole dataset. This report is displaying data from our servicedesk app, which is deployed on many servers and for many customers and use Query parameters to connect to any of these servers.
Detail of Power query is below.
Why am I writing here. This report was working without any issue more than 1,5 year but on August 17th one of servers start causing erros in step Pages where are some random lines (pages) with errors - see attached picture labeled "Errors in step Pages". and this is reason that next step Entities (List.Union) in query is stopping refresh and generate errors with message:
Expression.Error: We cannot apply field access to the type List. Details: Value=[List] Key=requests
What is notable
API service si returning records in the same order but faulty lists are random when calling with same parameters
some times is refresh without any error
The same power query called on another server is working correctly , problem is only with one specific server.
This problem started without notice on the most important server after 1,5 year without any problem.
Here is full text power of query for this main source, which is used later in other queries to extract all necessary data. Json is really complicated and I extract from it list of requests, list of solvers, list of solver groups,.... and this base query and its output is input for many referenced queries.
Errors in step Pages
let
BaseAPIUrl = apiurl&"apiservice?", /*apiurl is parameter - name of server e.g. https://xxxx.xxxxxx.sk/ */
EntitiesPerPage = RecordsPerPage, /*RecordsPerPage is parameter and defines nr. of record per page - we used as optimum 200-400 record per pages, but is working also with 4000 record per page*/
ApiToken = FnApiToken(), /*this function is returning apitoken value which is returning value of another api service apiurl&"api/auth/login", which use username and password in body of call to get apitoken */
GetJson = (QParm) => /*definiton general function to get data from data source*/
let
Options =
[ Query= QParm,
Headers=
[
Accept="application/json",
ApiKeyName="apitoken",
Authorization=ApiToken
]
],
RawData = Web.Contents(BaseAPIUrl, Options),
Json = Json.Document(RawData)
in Json,
GetEntityCount = () => /*one times called function to get nr of records using GetJson, which is returned as a part of each call*/
let
QParm = [pp="1", pg="1" ],
Json = GetJson(QParm),
Count = Json[totalRecord]
in
Count,
GetPage = (Index) => /*repeatadly called function to get each page of json using GetJson*/
let
PageNr = Text.From(Index+1),
PerPage = Text.From(EntitiesPerPage),
QParm = [pg = PageNr, pp=PerPage],
Json = GetJson(QParm),
Value = Json[data][requests]
in Value,
EntityCount = List.Max({ EntitiesPerPage, GetEntityCount() }), /*setup of nr. of records to variable*/
PageCount = Number.RoundUp(EntityCount / EntitiesPerPage), /*setup of nr. of pages */
PageIndices = { 0 .. PageCount - 1 },
Pages = List.Transform(PageIndices, each GetPage(_) /*Function.InvokeAfter(()=>GetPage(_),#duration(0,0,0,1))*/), /*here we call for each page GetJson function to get whole dataset - there is in comment test with delay between getpages but was not neccessary*/
Entities = List.Union(Pages),
Table = Table.FromList(Entities, Splitter.SplitByNothing(), null, null, ExtraValues.Error)
I also tried another way of appending pages to list using List.Generate. This is also bringing random errors in list but
it is bringing possibility to transform to table in contrast with original way with using List.Transform, but other referenced queries are failing and contains on the last row errors
When I am exploring content of faulty page/list extracting it via Add as New Query there are always all record without any fail.....
Source = List.Generate( /*another way to generate list of all pages*/
() => [Page = 0, ReqPageData = GetPage(0) ],
each [Page] < PageCount,
each [ReqPageData = GetPage( [Page] ),
Page = [Page] + 1 ],
each [ReqPageData]
),
#"Converted to Table" = Table.FromList(Source, Splitter.SplitByNothing(), null, null, ExtraValues.Error), /*here i am able to generate table from list in contrast when is used List.Generate*/
#"Expanded Column1" = Table.ExpandListColumn(#"Converted to Table", "Column1"), /*here aj can expand list to column*/
#"Removed Errors" = Table.RemoveRowsWithErrors(#"Expanded Column1", {"Column1"}) /*here i try to exclude errors, but i dont know what happend and which records (if any) are excluded*/
Extracting errored page
and finnaly I am tottaly clueless not able to find the cause of this behavior on this specific server. I tested to call pages which are errored via POSTMAN, I discused this issue with author of API service and He also tried to call this API service with all parameters but server is returning every page OK, only Power query is not able to List.Transform ...
I will be grateful and appreciate any tips or advice or if somebody solved the same issue in the past ....
Kuby
No, each error line of list in step List.Transform coud by extracted as new query and there are all records from one page OK. hmmmm
Finnaly, problem described in this issue was caused by "corrupted" content of returning json. The provider of core system informed me that they found bug and after fixing on the side of servisdesk is everything OK again. I tried to find problem in Power query and problem was in servisdesk. :(
How I can get ids of checked items in items list in backend. I want add extra functionality to publish function. So what I need is to get that checked ids. In (com_registracijos\controllers\lists.php) I tried add something likes this:
function publish()
{
$id = JRequest::getInt('id');
$value = JRequest::getInt('value');
$boxchecked = JRequest::getInt('boxchecked');
}
To get the ids of a list checked itemsyou should get and array instead of an int:
$arrayIDs = JRequest::getVar ( 'cid', null, 'default', 'array' );
Please, note that cid is the name of the check.
Every component, if any, of the resulting array is a checked item id.
If you need a sample, you may check delete uploaded file from an array of id when they are delted in joomla? It is doing a delete, but it will give you the idea.
Regards,
I have this to select columns from a table and I want to pass this to Jquery ajax fn.
I am using below code but getting invalid json
My table has three column id, name and city but I am not selecting city
This is my json reponse
["{id:1,name\":\"JOHN\",\"city\":\"null\"}"
,"{\"id\":2,\"name\":\"MICHEAL\,\"city\":\"null\"}"]
Sadly the current stable (1.8.1) WanWizard datamapper DMZ_Json class double encode fields when you call all_to_json(). This issue seem to be fixed in the develop branch. You could workaround this multiple ways:
1. Call to_json() on individual model objects and create the json array with string manipulation:
$my_objects = (new Model)->get();
$results = array();
foreach ($my_objects as $o) {
$results[] = $o->to_json();
}
// building a json array from the strings returned by $o->to_json()
print '['.join(',', $results).']';
2. You can use the array extension's all_to_array method and json_encode that result:
$my_object_arrays = (new Model)->get()->all_to_array();
print json_encode($my_object_arrays);
I am trying to access the list of last viewed items using the below code:
$attributes = Mage::getSingleton('catalog/config')->getProductAttributes();
$model = Mage::getModel('reports/product_index_viewed');
//
$_collection = $model->getCollection()->addAttributeToSelect($attributes)
->excludeProductIds($model->getExcludeProductIds())
->addUrlRewrite()
->setPageSize($columnCount)
->setCurPage(1);
//
$_collection->addPriceData();
$_collection->addIndexFilter();
$_collection->setAddedAtOrder();
//
Mage::getSingleton('catalog/product_visibility')->addVisibleInSiteFilterToCollection($_collection);
I copied this from the Mage_Reports_Block_Product_Abstract but this is giving the products in the creation order.
I doubt Prasanth will ever come back to this but I too having been trying to get a simple list of products without using a block which may not always be available. Eventually I found you need this:
$viewedCollection = Mage::getModel('reports/product_index_viewed')
->getCollection()
->addIndexFilter();
The secret is in addIndexFilter(), it uses the current customer or - if not logged in - the current visitor instead. From this you can loop through as with any other collection or extract a single array:
$viewedProductIds = $viewedCollection->getColumnValues('product_id');
I have created a new query like the following
var pressData = from press in dataContext.Releases
select new
{
Heading = press.Heading,
Description = press.Desc,
DatePublished = press.PublishDate.ToString(),
Body = press.BodyContent,
ID=press.ReleaseID,
CreatedBy=press.CreatedBy
};
Later in the code I want to update the entity from a session variable, but not save any data back to the database. Here is the code I am trying to accomplish this with....
var edit = pressData.Where(a => a.Heading == sectionPreview.HeadingContent && a.ID == sectionPreview.tionID).FirstOrDefault();
if (edit != null)
{
//WONT LET ME UPDATE THE Body VALUE
edit.Body = sectionPreview.SectionContent;
}
The code aboves purpose is to look at pressData and replace the body content with the new body from a session variable(not shown here), but NOT save it to the db. I want pressData to be filtered and updated only in the entity. So when I bind it to the control in this case it binds the data stored in my session.
this.rptSections.DataSource = pressData;
this.rptSections.DataBind();
I am getting a complier error stating
Property or indexer 'AnonymousType#1.Body' cannot be assigned to -- it is read only.
I checked the entity model and nothing is read only not any fields not anything. I must be missing something?
Anonymous Types encapsulate a read only property collection - for more information, read here. The compiler rewrites anonymous types as a constructor injections, ie:
select new
{
Heading = press.Heading,
Description = press.Desc,
DatePublished = press.PublishDate.ToString(),
Body = press.BodyContent,
ID=press.ReleaseID,
CreatedBy=press.CreatedBy
};
Is really rewritten as:
new Anonymous`1(press.Heading, press.Desc, press.PublishDate.ToString(), press.BodyContent, press.ReleaseID, press.CreatedBy)
And the properties are read only (public get, private / protected set, to use an easy comparison). If you want to solve your issue, instead of taking the data and making an anonymous object, create a real type and set properties on it.