I want to page through an entire month's worth of Calls usage in a report. Unfortunately, I'm experiencing random 500 errors from the API.
First I initialize the first page call to /Calls.json
// Set initial baseUrl
urlStr := "https://api.twilio.com/2010-04-01/Accounts/" + accountSid + "/Calls.json"
baseUrl, err := url.Parse(urlStr)
if err != nil {
panic(err)
}
// Build query parameters and URL
v := url.Values{}
v.Set("StartTime>", "2017-08-01")
v.Set("StartTime<", "2017-08-02")
v.Set("To", os.Args[1])
v.Set("PageSize", "1000")
v.Set("Page", "0")
baseUrl.RawQuery = v.Encode()
// Begin recursive call
pageNext(baseUrl)
I then determine if there is a "next page" by checking the response property next_page_uri, if the property is not blank then I proceed to recursively call the API with the previous calls next_page_uri query.
Each time I end up with a 500 error, sometimes within the first call and sometimes it is after 100-ish calls.
Is this really the best way to obtain such a report?
Related
I am using the list permissions endpoint (https://developers.google.com/drive/api/v3/reference/permissions/list) to get all the permissions for a file that exists in a shared drive.
I am running into an issue with a file I have that lives in a shared drive. The file has 100 permissions. The max limit on the number of permissions returned for a shared drive file is 100, so I should only need to make one request to get all the permissions for the file and the API should not return a next page token.
But this is not the behaviour I am experiencing, after the first request I continuously get the same next page token back from the API.
So for the following code below (written in go), I get into an infinite loop, since I continuously get the same next page token back.
var permissionService PermissionsService := getPermissionsService()
fileID := "1234abcd"
nextPageToken := ""
anotherPage := true
permissions := []*Permission{}
for anotherPage {
result, err := permissionService.
List(fileID).
SupportsAllDrives(true).
SupportsTeamDrives(false).
Fields("*").
PageSize(100).
Do()
if err != nil {
panic(err)
}
anotherPage = result.NextPageToken != ""
nextPageToken = result.NextPageToken
permissions = append(permissions, result.Permissions...)
}
fmt.Println("Permissions", permissions)
Am I supposed to account for this case in my code? From the documentation, this is never mentioned so I assume this is not supposed to happen.
It seems that you did not add the PageToken parameter in your permissionService. You are actually requesting the same first 100 results of the permission service.
Not familiar with the language but something like this
for anotherPage {
result, err := permissionService.
List(fileID).
SupportsAllDrives(true).
SupportsTeamDrives(false).
Fields("*").
PageSize(100).
PageToken(nextPageToken).
Do()
if err != nil {
panic(err)
}
anotherPage = result.NextPageToken != ""
nextPageToken = result.NextPageToken
permissions = append(permissions, result.Permissions...)
}
Can you also verify first if the result.NextPageToken will return "" if the file has < 100 results, I used API explorer and Apps Script to access this API and the nextPageToken is not part of the response body when permission list is less than the pageSize.
I have to fetch 250K chromebooks from google workspace (Gsuite), I am using Admin Directory API to retrieve JSON data from Google.
The response returns in chunks of 200 records, in the response is included a nextPageToken, I use that next page token to retrieve the next 200 and so on.
After an hour, of using the nextPageToken attached from the previous request, However Google returns with error 400,
{error_code: 400, "message"=>"Invalid Input: CMiJhq7-5ewCEp0BCm737N8GN......"},
Note: This string 'CMiJhq7-5ewCEp0BCm737N8GN......' which google is calling as invalid is the nextPageToken.
Why is this happening? Does nextPageToken expire after 1 hour?
My code snippet:
query_list = {
'maxResults' => 200,
'access_token' => access_token,
'pageToken' => next_page_token
}
HTTParty.get(endpoint_url, query: query_list)
The nextPage token is created when the initial request is sent. This token is used in order to get the next batch of rows from the request.
This token is intended to be used immediately as the data associated with the initial request may be changed if you wait to long.
So yes next page tokens do expire i would actually expect them to expire in a lot less than an hour. I also wonder if the next page token wouldn't just expire after you used it the first time.
If you want to make the same request again i suggest you do that and get new next page tokens built for you after the hour.
I had to change my approach, initially, I fetched 200 chunks from Google API, performed some time-taking processing and then made entries into my database (database-intensive tasks) and then requested the next 200 chunks and so so. After an hour, the last nextpagetoken sent by Google became invalid.
So, now I fetch 200 chunks, save them to my database in JSON format without performing any database-intensive tasks, request the next 200, and so on. I was able to fetch 300K Chromebooks JSON data from google in around 56 Minutes before the nextPageToken became invalid.
I am now processing that JSON data present in my database, without having network overhead or any google API dependency.
I'm encountering error Error 400: Invalid Value, invalid when I tried to use pageToken parameter of the Google Calendar Events API.
The code (in Go) I was using.
call := service.Events.List(calendarID).SingleEvents(true)
events, err := call.Do()
if err != nil {
return err
}
var allEvents []*calendar.Event
allEvents = append(allEvents, events.Items...)
if events.NextPageToken != "" {
nextPageToken := events.NextPageToken
nextCall := service.Events.List(calendarID).SingleEvents(true).PageToken(nextPageToken)
events, err = nextCall.Do()
if err != nil {
return err
}
allEvents = append(allEvents, events.Items...)
}
As illustrated in Paging through lists of resources, the next API call has to be exactly the same as the previous one. Thus, the following code works with the pageToken (where no nextCall is being used).
call := service.Events.List(calendarID).SingleEvents(true)
events, err := call.Do()
if err != nil {
return err
}
var allEvents []*calendar.Event
allEvents = append(allEvents, events.Items...)
if events.NextPageToken != "" {
nextPageToken := events.NextPageToken
call := call.PageToken(nextPageToken)
events, err = call.Do()
if err != nil {
return err
}
allEvents = append(allEvents, events.Items...)
}
I want to dynamically populate my internal struct, for an atomic insert. I am new to go so pointers and referencing them is something that I am still learning. I can not figure out why this for each loop is putting the same fields in twice. I tried removing the '&' then I get a cannot use type as *type error, I checked to make sure my loop was hitting every object in the tradeArray, and it is. It looks like it is overwriting the object before it with the last one it loops over. How can I fix this?
func createTrade(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
var tradeArray []Trade
if err := json.NewDecoder(r.Body).Decode(&tradeArray); err != nil {
e := Error{Message: "Bad Request - Improper Types Passed"}
w.WriteHeader(http.StatusBadRequest)
_ = json.NewEncoder(w).Encode(e)
return
}
for _, trade := range tradeArray {
internal := InternalTrade{
Id: strconv.Itoa(rand.Intn(1000000)),
Trade: &trade,
}
submit := TradeSubmitted{
TradeId: internal.Id,
ClientTradeId: trade.ClientTradeId ,
}
submitArray = append(submitArray, submit)
trades = append(trades, internal)
}
if err := json.NewEncoder(w).Encode(submitArray); err != nil {
e := Error{Message:"Internal Server Error"}
w.WriteHeader(http.StatusInternalServerError)
_ = json.NewEncoder(w).Encode(e)
return
}
}
edit: I was able to fix my problem by creating a new variable to hold the trade and referencing that variable in the struct creation. I am not sure how this is different that what I was doing above with just referencing the "trade" if someone could explain that I would greatly appreciate it.
for _, trade := range tradeArray {
p := trade
internal := InternalTrade{
Id: strconv.Itoa(rand.Intn(1000000)),
Trade: &p,
}
submit := TradeSubmitted{
TradeId: internal.Id,
ClientTradeId: trade.ClientTradeId ,
}
submitArray = append(submitArray, submit)
trades = append(trades, internal)
}
Let's look at just these parts:
var tradeArray []Trade
// code that fills in `tradeArray` -- correct, and omitted here
for _, trade := range tradeArray {
internal := InternalTrade{
Id: strconv.Itoa(rand.Intn(1000000)),
Trade: &trade,
}
submit := TradeSubmitted{
TradeId: internal.Id,
ClientTradeId: trade.ClientTradeId ,
}
submitArray = append(submitArray, submit)
trades = append(trades, internal)
}
This for loop, as you have seen, doesn't work the way you want. Here's a variant of it that's kind of similar, except that the variable trade has scope that extends beyond the for loop:
var trade Trade
for i := range tradeArray {
trade = tradeArray[i]
internal := InternalTrade{
Id: strconv.Itoa(rand.Intn(1000000)),
Trade: &trade,
}
// do correct stuff with `internal`
}
Note that each internal object points to a single, shared trade variable, whose value gets overwritten on each trip through the loop. The result is that they all point to the one from the last trip through the loop.
Your fix is itself OK: each trip through the loop, you make a new (different) p variable, and use &p, so that each internal.Trade has a different pointer to a different copy. You could also just do trade := trade inside the loop, to create a new unique trade variable. However, in this particular case, it may make the most sense to rewrite the loop this way:
for i := range tradeArray {
internal := InternalTrade{
Id: strconv.Itoa(rand.Intn(1000000)),
Trade: &tradeArray[i],
}
// do correct stuff with `internal`
}
That is, you already have len(tradeArray) different Trade objects: the slice header tradeArray gives you access to each tradeArray[i] instance, stored in the underlying array. You can just point to those directly.
There are various advantages and disadvantages to this approach. The big advantage is that you don't re-copy each trade at all: you just use the ones from the array that the slice header covers, that was allocated inside the json Decode function somewhere. The big disadvantage is that this underlying array cannot be garbage-collected as long as you retain any pointer to any of its elements. That disadvantage may have no cost at all, depending on the structure of the remaining code, but if it is a disadvantage, consider declaring tradeArray as:
var tradeArray []*Trade
so that the json Decode function allocates each one separately, and you can point to them one at a time without forcing the retention of the entire collection.
My aim is to create a logging function that lists the name of a function and the list of passed parameters.
An example would be the following:
func MyFunc(a string, b int){
... some code ...
if err != nil{
errorDescription := myLoggingFunction(err)
fmt.Println(errorDescription)
}
}
func main(){
MyFunc("hello", 42)
}
// where MyLoggingFunction should return something like:
// "MyFunc: a = hello, b = 42, receivedError = "dummy error description"
So far it seems that in Go there is no way to get the name of the parameters of a function at runtime, as answered in this question, but I could give up this feature.
I've managed to get the function name and the memory address of the passed parameters by analysing the stack trace, but I'm hitting a wall when it comes to print somehow the parameters starting from their address (I understand that it might not be trivial depending on the type of the parameters, but even something very simple will do for now)
This is an implementation of the logging function I'm building (you can test it on this playground), is there away to print the parameter values?
func MyLoggingFunction(err error) string {
callersPCs := make([]uintptr, 10)
n := runtime.Callers(2, callersPCs) //skip first 2 entries, (Callers, GetStackTrace)
callersPCs = callersPCs[:n]
b := make([]byte, 1000)
runtime.Stack(b, false)
stackString := string(b)
frames := runtime.CallersFrames(callersPCs)
frame, _ := frames.Next()
trimmedString := strings.Split(strings.Split(stackString, "(")[2], ")")[0]
trimmedString = strings.Replace(trimmedString, " ", "", -1)
parametersPointers := strings.Split(trimmedString, ",")
return fmt.Sprintf("Name: %s \nParameters: %s \nReceived Error: %s", frame.Function, parametersPointers, err.Error())
}
If there are other ideas for building such logging function without analysing the stack trace, except the one that consists in passing a map[string]interface{} containing all the passed parameter names as keys and their values as values (that is my current implementation and is tedious since I'd like to log errors very often), I'd be glad to read them.
Problem:
I have created a simple form, where there's an input field "num". After submission I want to show the value of num in the same input field, in other words want to retain the input in that field. If the value was set to 0 then I want to ignore that.
I can do it in several languages but I'm not sure about how to do it in Golang. My current template file has,
<input type="text" placeholder="foo" name="bar" value="{{if gt .N 0 }} {{.N}} {{end}} "/>
Server file contains:
data := &listOfReport {
R: r,
I: i,
N: n
}
listTmpl := template.Must(template.New("list_tmpl").Parse(string(report.Template["xxx.tmpl"])))
if err := listTmpl.Execute(w, data); err != nil {
http.Error(w, fmt.Sprintf("Error rendering template %v", err), 500)
}
Another thought is to make N a string so make it '' or value in the server file. But that actually spoils the variable's name/purpose.
Is there any better way to do it? Is ther any better way to access GET parameters directly from template? Please note that the value of N is originally got from a GET variable.
*This code is not tested
There is no standard/builtin way to get any request parameters from within a template, you'll have to put it into your data. (You could write a function which does this for you, but that will result in an ugly hack.)
I don't see what's wrong with your solution.
I take a similar approach, but use structs.
type SignupForm struct {
Name string
Email string
Etcera bool
}
// Type alias
type M map[string]interface{}
...
// In the handler that accepts your form
err := r.ParseForm()
if err != nil {
// handle error
}
signup := SignupForm{}
err := decoder.Decode(signup, r.PostForm)
if err != nil {
// handle error
}
// Store the 'saved' form contents somewhere temporary -
// e.g.
// - cookies (keep in mind the 4K browser limit)
// - server side sessions (Redis; how I do it)
// - db
// In the handler that renders your form
err := template.ExecuteTemplate(w, "form.html", M{
"form": signup,
"csrfToken": csrfToken,
// and so on...
})
Note that wherever you store the form data, make sure it is temporary. Server side sessions are ideal as you can have them expire (if you don't want to delete them manually).