I am using Youtube Data API version 3 and it seems there is an internal threshold (200) to limit the number of results. While the nextPageToken is available, the api returns an empty list. I am trying to get a list of videos from an specific channel ('UCJEy8aBnqDymMIcJCh72gMw'), it contains more than 3K videos and you can only get the first 200 and after that you will getting next tokens but the items are empty.
https://www.googleapis.com/youtube/v3/search?pageToken=XXXXX&order=date&part=snippet&key=XXXXXXXX&channelId=UCJEy8aBnqDymMIcJCh72gMw
Related
From reading the documentation I can see that the maximum number of results per response can be set to 50. See link below.
https://learn.microsoft.com/en-us/rest/api/cognitiveservices-bingsearch/bing-custom-search-api-v7-reference#count
What is the maximum number of total results this API can return?
If a search returned a total of 400 results, would the Bing Custom Search API be able to return them all?
For example the Google Custom Search API will only return a maximum of 100 results per query (10 pages of 10 results).
Accessing Additional Results
By default, each Nearby Search or Text Search returns up to 20 establishment results per query; however, each search can return as many as 60 results, split across three pages. If your search will return more than 20, then the search response will include an additional value — next_page_token. Pass the value of the next_page_token to the pagetoken parameter of a new search to see the next set of results. If the next_page_token is null, or is not returned, then there are no further results.
How is next_page_token paid? Or is it only charged for the first response from placeAPI? Or I will ask a question differently. If I use next_page_token after the first 20 spots, is it taken into account somewhere in the payment?
I am trying to get all documents in an index, I tried the following-
1) getting the total number of records first and then setting /_search?size= parameter -doesn't work as size parameter is restricted to 10000
2)tried paginating by making multiple calls and used the parameters '?size=1000&from=9000'
-worked till 'from' was < 9000 but after it exceeds 9000 i again get this size restriction error-
"Result window is too large, from + size must be less than or equal to: [10000] but was [100000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting"
So how can I retrieve all documents in the index?I read some answers suggesting to use the scroll api and even the documentation states -
"While a search request returns a single “page” of results, the scroll API can be used to retrieve large numbers of results (or even all results) from a single search request, in much the same way as you would use a cursor on a traditional database."
But I couldn't find any sample query to get all records in a single request.
I have a total of 388794 documents in the index.
Also note, this is a one time call so I am not worried about performance concerns.
Figured out the solution-
Scroll api is the proper way to do it- here's how its working-
In the first call to fetch the documents, a size say 1000 can be provided and scroll parameter specifying the time in minutes after which search context times out.
POST /index/type/_search?scroll=1m
{
"size": 1000,
"query": {....
}
}
For all subsequent calls we can use the scroll_id returned in the response of the first call to get the nest chunk of records.
POST /_search/scroll
{
"scroll" : "1m",
"scroll_id" : "DnF1ZXJ5VGhIOLSJJKSVNNZZND344D123RRRBNMBBNNN==="
}
I am trying to fetch more than 10000 doc with jest client.
I used scroll feature and use a query size of 50, but my program goes into an infinite loop and in every iteration returns the same 50 doc results.
I guess it is problem with scroll id which I am not passing can some body help.
Below is the call to be made to retrieve the first 50 records :
POST <host_name>:<port_num>/<index_name>/_search?scroll=1m&size=50
As shown above, the size is mentioned as 50 and scroll is 1m, this means that the scroll api will retrieve 50 records per hit and this scroll is available for 1 minute. Also, this api returns a scroll id, which should be used for further retrieval of records. Please find the sample below:
POST <host_name>:<port_num>/_search?scroll=1m&scroll_id=<scroll_id>
Note : For further scroll api calls, index name need not be mentioned. Only the scroll_id and scroll time is sufficient.
For more information, please refer to the elastic search documentation on scroll api : https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html
I am doing a search for videos using the keyword "woodturning". I have MaxResults set to 50, and use the returned NextPageToken to request more videos. On the 10th request, NextPageToken is null. So I can only get 500 items returned.
If I do a similar search from the youtube web page, I see a figure of more than 47,000 videos.
How do I get around this 500 item return limit?
After more searching, I found the answer already on stackexchange. Youtube is imposing a 500 item limit on search results.