We currently use our applications to download some pins information for analytics purposes
Currently we hit the API on hourly basis but we have some problems with rate limiting, so we need the following questions to be addresses:
The documentation states that the rate limit is 1000 calls per hour per user token per endpoint. By looking at the HTTP headers apparently the actual limit is 200. So, which is the correct one?
Are GET /v1/pins/1234 and GET /v1/pins/5678 considered to be a call to the same endpoint?
Are there any application level (so, despite of the used user token) rate-limit in place?
Are there any IP level rate limits?
Since we use concurrent requests, are there any limitation on the number of CONCURRENT request? (which means: make another call from the same IP and using the same token, even if the previous one hasn't ended yet)
Thanks in advance.
PS: I tried to contact them via support form, but it returns a 422. Which is awesome!
For apps that is still in development stage the limit is 200. You will get 1000 if your apps passed the review.
The documentation states that the rate limit is 1000 calls per hour
per user token per endpoint. By looking at the HTTP headers
apparently the actual limit is 200. So, which is the correct one?
Ans: Its 10 per token per app id per hour for app in development(tested), 1000 per token per hour for production app.
Are there any application level (so, despite of the used user token)
rate-limit in place?
Ans: No, rate limit will be 10(app in development) or 1000(approved app)/hour for each tokens you have
Are there any IP level rate limits?
Ans: No
Since we use concurrent requests, are there any limitation on the number of CONCURRENT request? (which means: make another call from the same IP and using the same token, even if the previous one hasn't ended yet)
Ans: No there is no such limit but they will probably check for bot activity and you may end up with banned account, so please do maintain some delay in between requests for same token IMO 12 seconds should be enough
Related
tl/dr do 100 devices all using the same Client ID count as 100 users, with their own limits, or one user sharing limits?
I have a webpage which reads and writes to a Google Sheet.
Because the webpage needs to know if a cell has changed, it polls the server once every 1000ms:
var pollProcId = window.setInterval(pollForInput, 1000);
where pollForInput does a single:
gapi.client.sheets.spreadsheets.values.get(request).then(callback);
When I tried to use this app with a class of 100 students I got many 429 error codes (more than I got successful reads) in response to google.apps.sheets.v4.SpreadsheetsService.GetValues requests:
Many of my users never got as far as seeing even the first request come back.
As far as I can make out, these are AnalyticsDefaultGroupUSER-100s errors which, according to the error responses page:
Indicates that the requests per 100 seconds per user per project quota
has been exhausted.
But with my app only requesting once per 1000 milliseconds, I wouldn't expect to see this many 429s as I have a limit of 100 requests per 100 seconds (1 per second) so only users whose application didn't complete in 100 seconds should have received a 429.
I know I should implement Exponential Backoff (which I'll do, I promise) but I'm worried I'm misunderstanding what a "user" in this context is.
Each user is using their own device (so presumably has a different IP address) but they are all using my "Client ID".
Does this scenario count as many users making one request per second, or a single user making a hundred requests per second?
Well, the user in the per user quota means that a single user making a request. So let's take the Sheets API, it has a quota of 100 for the Read requests per 100 seconds per user. So meaning only a single user can make a request per second. Note that Read request has a same set of quota as the Write request. But these two sets of quotas have their own set of quota and didn't share the same limit quota.
If you want a higher quota than the default, then you can apply for a higher quota by using this form or by visiting your developer console, then click the pencil icon in the quota that you want to increase.
I also suggest you to do the Exponential Backoff as soon as possible, because it can help you to avoid getting this kind of error.
Hope it helps you.
I'm currently integrate Google+ API to my service.
I'd like to know what's the limitation for this kind of api:
https://www.googleapis.com/plus/v1/people/{user_id}/activities/public
in google develop console, I found this:
Quota summary
Free quota 10,000 requests/day
Remaining 9,998 requests/day
99.98% of total
Per-user limit
5 requests/second/user
I think there will be two kind of limitation:
Application level
For example, how many requests can an app send per day(sum of the number for all users), and what's the max qps?
User level
For example, how many requests can an app send per day for a special user, and what's the max qps?
But I can't find the exactly info, does anyone know?
Can't say particularly about this API, but when I used Google Places api the quota was linked with IP address.So if it expired, we need to use different IP for hit.No user / application quota.
you already have those answers in your question:
10,000 requests per day in total, using that developer console key, thus your total "app" calls.
per user there is no limit, there is a rate quota. a single user could at most make 5*(seconds in a day) requests per day.
in this specific api case this is much bigger than 10,000 thus that rate quota is not that useful (except it prevents users from quickly depeting the 10,000 daily quota).
you can edit that rate quota so its lower or higher, and is used so a single user cant consume all the app quota (maliciously or otherwise)
Does the Google Analytics API throttle requests?
We have a batch script that I have just moved from v2 to v3 of the API and the requests go through quite well for the first bit (50 queries or so) and then they start taking 4s or so each. Is this Google throttling us?
While Matthew is correct, I have another possibility for you. Google analytics API cashes your requests to some extent. Let me try and explain.
I have a customer / site that I request data from. While testing I noticed some strange things.
the first million rows results would come back with in an acceptable amount of time.
after a million rows things started to slow down we where seeing results returning in 5 times as much time instead of 5 minutes we where waiting 20 minutes or more for the results to return.
Example:
Request URL :
https://www.googleapis.com/analytics/v3/data/ga?ids=ga:34896748&dimensions=ga:date,ga:sourceMedium,ga:country,ga:networkDomain,ga:pagePath,ga:exitPagePath,ga:landingPagePath&metrics=ga:entrances,ga:pageviews,ga:exits,ga:bounces,ga:timeOnPage,ga:uniquePageviews&filters=ga:userType%3D%3DReturning+Visitor;ga:deviceCategory%3D%3Ddesktop&start-date=2014-05-12&end-date=2014-05-22&start-index=236001&max-results=2000&oauth_token={OauthToken}
Request Time (seconds:milliseconds): :0:484
Request URL :
https://www.googleapis.com/analytics/v3/data/ga?ids=ga:34896748&dimensions=ga:date,ga:sourceMedium,ga:country,ga:networkDomain,ga:pagePath,ga:exitPagePath,ga:landingPagePath&metrics=ga:entrances,ga:pageviews,ga:exits,ga:bounces,ga:timeOnPage,ga:uniquePageviews&filters=ga:userType%3D%3DReturning+Visitor;ga:deviceCategory%3D%3Ddesktop&start-date=2014-05-12&end-date=2014-05-22&start-index=238001&max-results=2000&oauth_token={OauthToken}
Request Time (seconds:milliseconds): :7:968
I did a lot of testing stopping and starting my application. I couldn't figure out why the data was so fast in the beginning then slow later.
Now I have some contacts on the Google Analytics Development team the guys in charge of the API. So I made a nice test app, logged some results showing my issue and sent it off to them. With the question Are you throttling me?
They where also perplexed, and told me there is no throttle on the API. There is a flood protection limit that Matthew speaks of. My Developer contact forwarded it to the guys in charge of the traffic.
Fast forward a few weeks. It seams that when we make a request for a bunch of data Google cashes the data for us. Its saved on the server incase we request it again. By restarting my application I was accessing the cashed data and it would return fast. When I let the application run longer I would suddenly reach non cashed data and it would take longer for them to return the request.
I asked how long is data cashed for, answer there was no set time. So I don't think you are being throttled. I think your initial speedy requests are cashed data and your slower requests are non cashed data.
Email back from google:
Hi Linda,
I talked to the engineers and they had a look. The response was
basically that they thinks it's because of caching. The response is
below. If you could do some additional queries to confirm the behavior
it might be helpful. However, what they need to determine is if it's
because you are querying and hitting cached results (because you've
already asked for that data). Anyway, take a look at the comments
below and let me know if you have additional questions or results that
you can share.
Summary from talking to engineer: "Items not already in our cache will
exhibit a slower retrieval processing time than items already present
in the cache. The first query loads the response into our cache and
typical query times without using the cache is about 7 seconds and
with using the cache is a few milliseconds. We can also confirm that
you are not hitting any rate limits on our end, as far as we can tell.
To confirm if this is indeed what's happening in your case, you might
want to rerun verified slow queries a second time to see if the next
query speeds up considerably (this could be what you're seeing when
you say you paste the request URL into a browser and results return
instantly)."
-- IMBA Google Analytics API Developer --
Google's Analytics API does have a rate limit per their docs: https://developers.google.com/analytics/devguides/reporting/core/v3/coreErrors
However they should not caused delayed requests, rather the request should be returned with a response of: 403 userRateLimitExceeded
Description of that error:
Indicates that the user rate limit has been exceeded. The maximum rate limit is 10 qps per IP address. The default value set in Google Developers Console is 1 qps per IP address. You can increase this limit in the Google Developers Console to a maximum of 10 qps.
Google's recommended course of action:
Retry using exponential back-off. You need to slow down the rate at which you are sending the requests.
I have an angularjs app that calls a RESTful service at apiary.io
Does apiary.io throttle responses and delay responses after a certain number have been received?
If so what are the parameters?
Currently, Apiary limits you for 120 reqs/minute/IP.
There are no artificial delays, but occasionally someones floods Apiary with production traffic and even when ratelimiting is fairly efficient, it may temporarily degrade service for other users.
You can (and should) check X-Apiary-RateLimit-Limit and X-Apiary-RateLimit-Remaining header. Once you'll hit the limit, Apiary will sent Retry-After header you should obey.
From their docs:
API Call Limit
API calls are subject to the default limit of 15 requests per second and exceeding this limit will result in all endpoints returning an HTTP status code of 429. Limits are per API key. If the limit is exceeding then the API Key will be blocked for the remainder of the sample period. If an API key continually hits the call limit we reserve the right to permanently block the key and to charge a fee to unblock the key.
To determine the API call amount we monitor the traffic over a sample period. If the traffic results in a particular API key reaching 80% of the limit (i.e., 12 if the limit is 15) over the sample period then the responses will start to contain a throttle node which contains useful information on how close you are to reaching the call limit.
Lately, in our app using IPP data services, we have encountered these errors from time to time.
<RestResponse xmlns="http://www.intuit.com/sb/cdm/v2">
<Error RequestId="49f7926a9aa84cfc8289534801dee72d">
<RequestName>ErrorRequest</RequestName>
<ProcessedTime>2012-12-07T10:10:59+00:00</ProcessedTime>
<ErrorCode>3001</ErrorCode>
<ErrorDesc>message=This client has made too many consecutive requests over too short a period of time. Please wait a short amount of time before attempting to submit again; errorCode=003001; statusCode=403; source=Throttling Policy</ErrorDesc>
</Error>
</RestResponse>
Can't find any reference to a "Throttling Policy" or error code "3001" anywhere in the IPP documentation.
Any help in figuring out what the throttling limits are would be appreciated. Are they based around an IP, rate limitation, concurrency limitation, OAuth consumer, OAuth client, something else perhaps?
EDIT: Link to IDN forums about the same issue: https://idnforums.intuit.com/messageview.aspx?catid=69&threadid=18910.
Yes, there is a throttling process in place if > 500 requests per minute by a single user or against a single realm.
You had over 600 requests during the one minute period.
Looks like almost all (all except 32 requests) were individual customer queries… all different customer record ids. Is there a way you can make a single customer list query, filtered if necessary, to get a bunch of customer records in a single request and reduce the number of calls you are making.