I have a php program that must read analytics data and save these data to a DB. Because the data is too big, I use the pagination (10000 to be more specific). As I run the code over and over again, sometimes I get the next page token, sometimes not. The problems it is a one-time-run program. I wonder what I'm doing wrong, here's the code:
do {
$body->reportRequests[0]->setPageToken($token);
$data = $analytics->reports->batchGet( $body );
$token = $data->reports[0]->getNextPageToken();
$this->printResults($data);
} while ( $token != '' );
The searching result was right, ScottMcC. I don't understand about analytics, so I forgot to filter in the sales report and check the results there are the same in the api return.
Related
I created PowerBI report which which is connecting to data source via API service. Returning json contains thousands of entities. API service is called via Web.Content function. API service returns always total record count and so we are able to calculate nr. of pages which has to be called to obtain whole dataset. This report is displaying data from our servicedesk app, which is deployed on many servers and for many customers and use Query parameters to connect to any of these servers.
Detail of Power query is below.
Why am I writing here. This report was working without any issue more than 1,5 year but on August 17th one of servers start causing erros in step Pages where are some random lines (pages) with errors - see attached picture labeled "Errors in step Pages". and this is reason that next step Entities (List.Union) in query is stopping refresh and generate errors with message:
Expression.Error: We cannot apply field access to the type List. Details: Value=[List] Key=requests
What is notable
API service si returning records in the same order but faulty lists are random when calling with same parameters
some times is refresh without any error
The same power query called on another server is working correctly , problem is only with one specific server.
This problem started without notice on the most important server after 1,5 year without any problem.
Here is full text power of query for this main source, which is used later in other queries to extract all necessary data. Json is really complicated and I extract from it list of requests, list of solvers, list of solver groups,.... and this base query and its output is input for many referenced queries.
Errors in step Pages
let
BaseAPIUrl = apiurl&"apiservice?", /*apiurl is parameter - name of server e.g. https://xxxx.xxxxxx.sk/ */
EntitiesPerPage = RecordsPerPage, /*RecordsPerPage is parameter and defines nr. of record per page - we used as optimum 200-400 record per pages, but is working also with 4000 record per page*/
ApiToken = FnApiToken(), /*this function is returning apitoken value which is returning value of another api service apiurl&"api/auth/login", which use username and password in body of call to get apitoken */
GetJson = (QParm) => /*definiton general function to get data from data source*/
let
Options =
[ Query= QParm,
Headers=
[
Accept="application/json",
ApiKeyName="apitoken",
Authorization=ApiToken
]
],
RawData = Web.Contents(BaseAPIUrl, Options),
Json = Json.Document(RawData)
in Json,
GetEntityCount = () => /*one times called function to get nr of records using GetJson, which is returned as a part of each call*/
let
QParm = [pp="1", pg="1" ],
Json = GetJson(QParm),
Count = Json[totalRecord]
in
Count,
GetPage = (Index) => /*repeatadly called function to get each page of json using GetJson*/
let
PageNr = Text.From(Index+1),
PerPage = Text.From(EntitiesPerPage),
QParm = [pg = PageNr, pp=PerPage],
Json = GetJson(QParm),
Value = Json[data][requests]
in Value,
EntityCount = List.Max({ EntitiesPerPage, GetEntityCount() }), /*setup of nr. of records to variable*/
PageCount = Number.RoundUp(EntityCount / EntitiesPerPage), /*setup of nr. of pages */
PageIndices = { 0 .. PageCount - 1 },
Pages = List.Transform(PageIndices, each GetPage(_) /*Function.InvokeAfter(()=>GetPage(_),#duration(0,0,0,1))*/), /*here we call for each page GetJson function to get whole dataset - there is in comment test with delay between getpages but was not neccessary*/
Entities = List.Union(Pages),
Table = Table.FromList(Entities, Splitter.SplitByNothing(), null, null, ExtraValues.Error)
I also tried another way of appending pages to list using List.Generate. This is also bringing random errors in list but
it is bringing possibility to transform to table in contrast with original way with using List.Transform, but other referenced queries are failing and contains on the last row errors
When I am exploring content of faulty page/list extracting it via Add as New Query there are always all record without any fail.....
Source = List.Generate( /*another way to generate list of all pages*/
() => [Page = 0, ReqPageData = GetPage(0) ],
each [Page] < PageCount,
each [ReqPageData = GetPage( [Page] ),
Page = [Page] + 1 ],
each [ReqPageData]
),
#"Converted to Table" = Table.FromList(Source, Splitter.SplitByNothing(), null, null, ExtraValues.Error), /*here i am able to generate table from list in contrast when is used List.Generate*/
#"Expanded Column1" = Table.ExpandListColumn(#"Converted to Table", "Column1"), /*here aj can expand list to column*/
#"Removed Errors" = Table.RemoveRowsWithErrors(#"Expanded Column1", {"Column1"}) /*here i try to exclude errors, but i dont know what happend and which records (if any) are excluded*/
Extracting errored page
and finnaly I am tottaly clueless not able to find the cause of this behavior on this specific server. I tested to call pages which are errored via POSTMAN, I discused this issue with author of API service and He also tried to call this API service with all parameters but server is returning every page OK, only Power query is not able to List.Transform ...
I will be grateful and appreciate any tips or advice or if somebody solved the same issue in the past ....
Kuby
No, each error line of list in step List.Transform coud by extracted as new query and there are all records from one page OK. hmmmm
Finnaly, problem described in this issue was caused by "corrupted" content of returning json. The provider of core system informed me that they found bug and after fixing on the side of servisdesk is everything OK again. I tried to find problem in Power query and problem was in servisdesk. :(
I am using TokenGuard for all the database related requests through JQuery in Laravel 5.4
Once the user is logged in...I am fetching the user profile using below method.
public function ViewProfile() {
$user = \Auth::guard("api")->user();
$Data = [
"FirstName" => $user->FirstName,
"LastName" => $user->LastName
];
return \Response::json([
'Status' => true,
'Data' => $Data,
], 200);
}
Below is the POST method Url being sent to Server to fetch the data using above method
http://localhost:1234/public/api/v1/apiviewprofile?api_token=598bbba095bdd1598bbba095bf1
When I printed the select statements being issued on fetching the user details...I saw there were two select statements below.
[2017-08-10 02:37:05] local.INFO: select * from `tbluser` where `UserID` = ? limit 1
[2017-08-10 02:37:05] local.INFO: select * from `tbluser` where `api_token` = ? limit 1
Question
I can understand that when I send POST request using TokenGuard...it first validate token....so that time it goes to database. So second select statement is fine. But, where is Laravel 5.4 using first select statement?
The second statement is indeed run after you call ->user() on the guard returned from Auth::guard(), inside the guard's contained UserProvider implementation.
The first statement is unlikely to be triggered from the code that you provided. I suspect that it has something to do with some middleware that you have on the route using the ViewProfile() action.
Maybe the query is performed even before a redirect to that route, after you've successfully authenticated the user? (the first query is exactly a user authentication query).
Redirects can complete within a fraction of a second, and since the logs show the time in 1-second precision, it is impossible to tell how much time is exactly in between. Unless you are certain that no redirect takes place at all, it is hard to tell if these two queries execute within the same Laravel container lifecycle at all.
I have this chunk of code:
//to-do
public function searchVehicles($terms, $offset=1, $order='ASC')
{
if (trim($terms) == '') {
return array();
}
$query = $this->_getQuery($terms);
$query->setStoreId(1);
if ($query->getId()) {
$query->setPopularity($query->getPopularity()+1);
}
else {
$query->setPopularity(1);
}
$query->prepare();
$query->save();
$collection = Mage::getResourceModel('catalog/product_collection');
$collection->getSelect()->joinInner(
array('search_result' => $collection->getTable('catalogsearch/result')),
$collection->getConnection()->quoteInto(
'search_result.product_id=e.entity_id AND search_result.query_id=?',
$query->getId()
),
array('relevance' => 'relevance')
);
$collection->setStore(1);
//Mage::getSingleton('catalog/product_status')->addVisibleFilterToCollection($collection);
//Mage::getSingleton('catalog/product_visibility')->addVisibleInSearchFilterToCollection($collection);
return $this->_listProductCollection($collection, $offset, $order);
}
Which is inside a Resource class and reachable via SOAP.
Before we start: Yes, I remember to do the cache flushing and recompiling process - I clarify because this is an usual issue to newbies like me xDDD.
Now: I can access such method but it returns [].
SPECIAL NOTE: $this->_listProductCollection($collection, $offset, $order); WORKS since i'm using the same method in other collections fetched from other methods in the same resource, and have no trouble at all.
Let me review the intention of my code since I'm a newbie at Magento (I'm using version 1.6.2).
The code is based on the CatalogSearch/ResultController controller's indexAction() method, and tried to learn about it.
An empty query will yield an empty result and will not bother the Magento search engine.
There's only a Store (id = 1) in the site and the search query is created like this:
private function _getQuery($terms)
{
$query = Mage::getModel('catalogsearch/query')->loadByQuery($terms);
if (!$query->getId()) {
$query->setQueryText($terms);
}
return $query;
}
The query increases it's popularity (I took this code from the controller. I assume this is for statistical purposes only).
The query is prepared (I think this means: the MySQL internal query is prepared) so I can fetch it later.
The query is saved - AFAIK this means that the query results are iterated and cached so a subsequent same query will only fetch the stored results instead of processing the search again.
At this point the query will have an ID.
I get the whole Product collection, and join it with the search result table. SEEMS that the results table has - at least (queryId, matchedProductId). I only keep the products having IDs in the matched results, and from store 1.
I list the products.
Note that the filters are currently commented.
However, the returned list is [] (an empty list) when I hit this API entry point, althought searching in the usual search bar gives me the expected result.
Question: What am I missing? What did I misunderstood in the process?
I use CodeIgniter + Tank Auth. Only the code http://localhost/XXXXXXXX/auth/forgot_password doesn't work.
The result always is:
"Your activation key is incorrect or expired. Please check your email again and follow the instructions."
The registration and activation are all right.
Some likely problems:
Cookies are not being set correctly. Check your cookie settings, and do a test to make sure you can set and read cookies. (this may be invalid if cookies are not used for the reset)
The reset password key is expired or wasn't set correctly. Check the database to see if hte value is correct before following the link, and check your $config['forgot_password_expire'] in Tank Auth.
You may be linking to the wrong URL in your email.
This doesn't look right:
http://localhost/XXXXXXXX/auth/forgot_password
It should be something like:
http://localhost/auth/forgot_password/XXXXXXXX
Not to discourage you from using Tank Auth, but having used it I can recommend trying Ion_Auth if you are still in the early stages. I believe it's used in PyroCMS as well if that adds any credit.
If the XXXXXXXX in your URL is indicating that you have an extra URI segment before /auth/, you should change this:
function reset_password()
{
$user_id = $this->uri->segment(3);
$new_pass_key = $this->uri->segment(4);
to this:
function reset_password()
{
$user_id = $this->uri->segment(4);
$new_pass_key = $this->uri->segment(5);
Note the different numbers in $this->uri->segment(). With an extra segment before /auth/, your user id and activation code will be passed as parameters in the 4th and 5th segment (rather than the 3rd and 4th that Tank Auth assumes).
it could be the time stamp in your database
in user model function "can_reset_password"
uses a UNIX_TIMESTAMP(new_password_requested)
you could echo $user_id and $new_pass_key if they are correct then the problem is with time comparison.
to fix the the url to always get the last two segments
$break =$this->uri->total_segments();
$new_pass_key= $this->uri->segment($break);
$user_id= $this->uri->segment($break-1);
for the time stamp try this for the function reset_password in users model
function reset_password($user_id, $new_pass, $new_pass_key, $expire_period = 900)
{
$this->load->helper('date');
$this->db->set('password', $new_pass);
$this->db->set('new_password_key', NULL);
$this->db->set('new_password_requested', NULL);
$this->db->where('id', $user_id);
$this->db->where('new_password_key', $new_pass_key);
$this->db->where('UNIX_TIMESTAMP(new_password_requested) >=',mysql_to_unix( time() - $expire_period));
$this->db->update($this->table_name);
return $this->db->affected_rows() > 0;
}
In the main index.php, you have to define a time zone. Example:
date_default_timezone_set('Europe/Paris');
this will ensure that the following check has all the dates with the same time zone
$this->db->where('UNIX_TIMESTAMP(new_password_requested) >', time() - $expire_period);
Which is right? notice in the second option, I'm passing the form values using the $_POST variable. Whereas the first option, I call and assign variables for each form field.
I've seen this ...
<validation code> ....
$todo = array(
'name'=>$this->input->post('title'),
'description'=>$this->input->post('description')
);
$this->Todo_model->add($todo);
But I've also seen the following ...
$records['email'] = "trim|required|min_length[4]|xss_clean";
...
...
$this->validation->set_rules($records);
if ($this->validation->run())
{
$this->account_model->saveAccountSettings("sam", $_POST);
$this->session->set_flashdata('message', 'Done!');
redirect('account/settings');
} else {
...
}
I tend to use a mix of your two examples. I'm pretty sure things like trim won't modify the actual post data, so you can only take advantage of it if you go through the validation framework to get the data. I actually never access POST directly anymore using CI.
Plus I'd be worried in your second example about just shoving POST into my model. What happens if someone clever adds "lastname" to the post data sent in and your db column is named the same? Even though you weren't expecting to deal with that data now you've got unvalidated data coming in. That's why I employ part of your first example and manually pull out the items I want to save into an array first.
So I'd recommend a hybrid.
Normally my code looks something like this:
$fields['email'] = "trim|required|valid_email|min_length[4]|xss_clean";
...
...
$this->validation->set_rules($fields);
if ($this->validation->run())
{
$account = new array();
$account['id'] = $accountId; //wherever you get the Id from
$account['email'] = $this->validation->email;
$this->account_model->save($account);
$this->session->set_flashdata('message', 'Done!');
redirect('account/settings');
} else {
...
}
The first option is better easy to read or trace
Pass values using post variables is better option
What the real benefit to use this
$account['email'] = $this->validation->email;
Instead of
$account['email'] = $this->input->post('email');