NextPageToken gives invalid input error, 400 - Google Directory API ChomeDevices - google-api

Question

I have to fetch 250K chromebooks from google workspace (Gsuite), I am using Admin Directory API to retrieve JSON data from Google.
The response returns in chunks of 200 records, in the response is included a nextPageToken, I use that next page token to retrieve the next 200 and so on.
After an hour, of using the nextPageToken attached from the previous request, However Google returns with error 400,
{error_code: 400, "message"=>"Invalid Input: CMiJhq7-5ewCEp0BCm737N8GN......"},
Note: This string 'CMiJhq7-5ewCEp0BCm737N8GN......' which google is calling as invalid is the nextPageToken.
Why is this happening? Does nextPageToken expire after 1 hour?
My code snippet:
query_list = {
'maxResults' => 200,
'access_token' => access_token,
'pageToken' => next_page_token
}
HTTParty.get(endpoint_url, query: query_list)

Answer 1

The nextPage token is created when the initial request is sent. This token is used in order to get the next batch of rows from the request.
This token is intended to be used immediately as the data associated with the initial request may be changed if you wait to long.
So yes next page tokens do expire i would actually expect them to expire in a lot less than an hour. I also wonder if the next page token wouldn't just expire after you used it the first time.
If you want to make the same request again i suggest you do that and get new next page tokens built for you after the hour.

Answer 2

I had to change my approach, initially, I fetched 200 chunks from Google API, performed some time-taking processing and then made entries into my database (database-intensive tasks) and then requested the next 200 chunks and so so. After an hour, the last nextpagetoken sent by Google became invalid.
So, now I fetch 200 chunks, save them to my database in JSON format without performing any database-intensive tasks, request the next 200, and so on. I was able to fetch 300K Chromebooks JSON data from google in around 56 Minutes before the nextPageToken became invalid.
I am now processing that JSON data present in my database, without having network overhead or any google API dependency.

Answer 3

I'm encountering error Error 400: Invalid Value, invalid when I tried to use pageToken parameter of the Google Calendar Events API.
The code (in Go) I was using.
call := service.Events.List(calendarID).SingleEvents(true)
events, err := call.Do()
if err != nil {
return err
}
var allEvents []*calendar.Event
allEvents = append(allEvents, events.Items...)
if events.NextPageToken != "" {
nextPageToken := events.NextPageToken
nextCall := service.Events.List(calendarID).SingleEvents(true).PageToken(nextPageToken)
events, err = nextCall.Do()
if err != nil {
return err
}
allEvents = append(allEvents, events.Items...)
}
As illustrated in Paging through lists of resources, the next API call has to be exactly the same as the previous one. Thus, the following code works with the pageToken (where no nextCall is being used).
call := service.Events.List(calendarID).SingleEvents(true)
events, err := call.Do()
if err != nil {
return err
}
var allEvents []*calendar.Event
allEvents = append(allEvents, events.Items...)
if events.NextPageToken != "" {
nextPageToken := events.NextPageToken
call := call.PageToken(nextPageToken)
events, err = call.Do()
if err != nil {
return err
}
allEvents = append(allEvents, events.Items...)
}