I have read the gmail api quota explanation here (https://developers.google.com/gmail/api/v1/reference/quota), but am still having troubles understanding what causes us to go over the limit.
Question 1:
What is a user in a per user quota? I am not sure if the user is an individual gmail user, or a service client using the gmail api.
Question 2:
We've seen the following error a few times, but don't see any obvious limit we've hit.
"error": {
"errors": [
{
"domain": "usageLimits",
"reason": "rateLimitExceeded",
"message": "Rate Limit Exceeded"
}
],
"code": 429,
"message": "Rate Limit Exceeded"
}
We were under 250 units/s and 25,000 units/100s. We're only using history.list and message.get calls no sending or modifications.
Is there some other quota I am missing?
User quota is based upon the account you are accessing. So it would be the GMail account. Sometimes you can trick it by sending a random quotaUser but this doesn't always work Google also uses your IP address to track quota I suspect.
User rate limit is flood protection you are going to fast.
Per User Rate Limit 250 quota units per user per second, moving
average (allows short bursts)
Exceeding a rate limit will cause an HTTP 403 or HTTP 429 Too Many
Requests response and your app should respond by retrying with
exponential backoff.
Googles calculations are not perfect you could be sending more or less and still hit this quota. Just implementexponential backoff.
Exponential backoff
The flow for implementing simple exponential backoff is as follows:
Make a request to the API.
Receive an HTTP 403 rate-limited response, which indicates you should retry the request.
Wait 1 + random_number_milliseconds seconds and retry the request.
Receive an HTTP 403 rate-limited response, which indicates you should retry the request.
Wait 2 + random_number_milliseconds seconds, and retry the request.
Receive an HTTP 403 rate-limited response, which indicates you should retry the request.
Wait 4 + random_number_milliseconds seconds, and retry the request.
Receive an HTTP 403 rate-limited response, which indicates you should retry the request.
Wait 8 + random_number_milliseconds seconds, and retry the request.
Receive an HTTP 403 rate-limited response, which indicates you should retry the request.
Wait 16 + random_number_milliseconds seconds, and retry the request.
Stop. Report or log an error.
For your Question 1
Here are the meaning of the different quota in your Gmail
QPD(quota per day) - meaning the maximum numbers of request over a 24 hour period a client id is able to make to an API
QPS(quota per second) - meaning a global quota per second for the application, meaning how many calls a second an application can make
quota per seconds per user - meaning the number of queries a user, the application can make.
For question number 2
Well, if you check the Quota of Gmail in your developer console, the Gmail has a default quota of:
So what can I suggest you is to use the following tips so that you work with your quota efficiently:
Push notification - it improve the performance of your application. It allows you to eliminate the extra network and compute costs involved with polling resources to determine if they have changed. Whenever a mailbox changes, the Gmail API notifies your backend server application.
Use synchronization to retrieve and store as many of the most recent messages or threads as are necessary for your purpose.
Batching Requests - to reduce the number of HTTP connections your client has to make.
If you notice that you reach this limit and you need more than this, then you can apply for more quota here.
Related
I am getting
"code": 403,
"message": "User Rate Limit Exceeded"
while using Google Drive API in my web app
Although the quota is 10,000 requests per 100 seconds and my average is less than 2:
How can I resolve this error? How to implement exponential backoff as the documents say?
There are sevrail types of quotas with Google apis.
Project based quotas which effect your project itself. These quotas can be extended. If for example you your project can make 10000 requests pre 100 seconds. you could request that this be extended.
Then there is the user based quotas. these quotas limit how much each user can send.
User Rate Limit Exceeded
Means that you are hitting a user rate quota. User rate quotas are flood protection they ensure that a single user of your application can not make to many requests at once.
These quotas can not be extended.
if you are getting a user rate limiting quota then you need to slow down your application and implement exponential backoff.
How you implement exponential backoff is up to you and the language you are using but it basically involves just retrying the same request again only adding wait times each time it fails
the graph
the graph in the google cloud console is an guestimate and it is not by any means accurate. If you are getting the error message you should go by that and not by what the graph says.
After hours of searching and thinking, I found out that,
'User Rate Limit Exceeded' has a spam protection which allow max 10 requests per second.
Thus I found out a lazy trick to do so by delaying the calls using:
usleep(rand(1000000,2000000);
It simply delays the call by a random duration between 1 and two seconds.
tl/dr do 100 devices all using the same Client ID count as 100 users, with their own limits, or one user sharing limits?
I have a webpage which reads and writes to a Google Sheet.
Because the webpage needs to know if a cell has changed, it polls the server once every 1000ms:
var pollProcId = window.setInterval(pollForInput, 1000);
where pollForInput does a single:
gapi.client.sheets.spreadsheets.values.get(request).then(callback);
When I tried to use this app with a class of 100 students I got many 429 error codes (more than I got successful reads) in response to google.apps.sheets.v4.SpreadsheetsService.GetValues requests:
Many of my users never got as far as seeing even the first request come back.
As far as I can make out, these are AnalyticsDefaultGroupUSER-100s errors which, according to the error responses page:
Indicates that the requests per 100 seconds per user per project quota
has been exhausted.
But with my app only requesting once per 1000 milliseconds, I wouldn't expect to see this many 429s as I have a limit of 100 requests per 100 seconds (1 per second) so only users whose application didn't complete in 100 seconds should have received a 429.
I know I should implement Exponential Backoff (which I'll do, I promise) but I'm worried I'm misunderstanding what a "user" in this context is.
Each user is using their own device (so presumably has a different IP address) but they are all using my "Client ID".
Does this scenario count as many users making one request per second, or a single user making a hundred requests per second?
Well, the user in the per user quota means that a single user making a request. So let's take the Sheets API, it has a quota of 100 for the Read requests per 100 seconds per user. So meaning only a single user can make a request per second. Note that Read request has a same set of quota as the Write request. But these two sets of quotas have their own set of quota and didn't share the same limit quota.
If you want a higher quota than the default, then you can apply for a higher quota by using this form or by visiting your developer console, then click the pencil icon in the quota that you want to increase.
I also suggest you to do the Exponential Backoff as soon as possible, because it can help you to avoid getting this kind of error.
Hope it helps you.
When I run requests series like
https://api.sparkpost.com:443/api/v1/suppression-list/he**0#gmail.com
Sometimes, I get error:
name: 'SparkPostError',
errors: [ { message: 'Too many requests' } ],
statusCode: 429
Too many - this is how many?
How long the server keeps track of period? How long the server resets the counter? How to solve this problem?
https://developers.sparkpost.com/api/index.html#header-rate-limiting
After a quick look at the site...
The answer of support
As mentioned before we do limit requests on our endpoints to prevent abuse and while we can’t reveal the actual limits we do recommend that you wait several seconds before making consecutive requests. If you are making these requests via some type of automated process (script, code, etc.), we highly recommend that you add a wait for several seconds upon seeing this 429 error and reattempt after. Decreasing the frequency of your requests and waiting before reattempting is the only way around this error message.
https://developers.sparkpost.com/api/index.html#header-rate-limiting
Rate Limiting
Note: To prevent abuse, our servers enforce request rate limiting, which may trigger responses with HTTP status code 429.
SparkPost implements rate limiting on the following API endpoints:
/api/v1/message-events
/api/v1/metrics/*
The limits imposed here are dynamic but as a general rule, polling these endpoints more than once in 2 minutes may encounter rate limiting and a 429 status code.
I was writing a node application that uses the gmail API when I noticed this error. My understanding of the error is that there are too many concurrent requests. It seems to be prompting me to wait 15 minutes and try again. After the waiting period, I tried to poke the API with the gui over at https://developers.google.com/gmail/api/v1/reference/users/messages/list#response, but the same error appears (with the time incremental 15 minutes). I've looked at my quota usage on the API site in the developer console, but there's no activity other than the errors. Does anyone know why this might be? I'd be extremely appreciative.
{
"error": {
"errors": [
{
"domain": "usageLimits",
"reason": "rateLimitExceeded",
"message": "User-rate limit exceeded. Retry after 2016-07-11T23:51:49.309Z"
}
],
"code": 429,
"message": "User-rate limit exceeded. Retry after 2016-07-11T23:51:49.309Z"
}
}
The Gmail API is subject to a daily usage limit that applies to all requests made from your application, as well as per-user rate limits.
Daily Usage 1,000,000,000 quota units per day Per User Rate
Limit
250 quota units per user per second, moving average (allows
short bursts)
Exceeding a rate limit will cause an HTTP 403 or HTTP 429 Too Many Requests response and your app should respond by retrying with exponential backoff.
Exponential backoff is a standard error handling strategy for network
applications in which the client periodically retries a failed request
over an increasing amount of time. If a high volume of requests or
heavy network traffic causes the server to return errors, exponential
backoff may be a good strategy for handling those errors. Conversely,
it is not a relevant strategy for dealing with errors unrelated to
rate-limiting, network volume or response times, such as invalid
authorization credentials or file not found errors.
Used properly, exponential backoff increases the efficiency of
bandwidth usage, reduces the number of requests required to get a
successful response, and maximizes the throughput of requests in
concurrent environments.
I have an angularjs app that calls a RESTful service at apiary.io
Does apiary.io throttle responses and delay responses after a certain number have been received?
If so what are the parameters?
Currently, Apiary limits you for 120 reqs/minute/IP.
There are no artificial delays, but occasionally someones floods Apiary with production traffic and even when ratelimiting is fairly efficient, it may temporarily degrade service for other users.
You can (and should) check X-Apiary-RateLimit-Limit and X-Apiary-RateLimit-Remaining header. Once you'll hit the limit, Apiary will sent Retry-After header you should obey.
From their docs:
API Call Limit
API calls are subject to the default limit of 15 requests per second and exceeding this limit will result in all endpoints returning an HTTP status code of 429. Limits are per API key. If the limit is exceeding then the API Key will be blocked for the remainder of the sample period. If an API key continually hits the call limit we reserve the right to permanently block the key and to charge a fee to unblock the key.
To determine the API call amount we monitor the traffic over a sample period. If the traffic results in a particular API key reaching 80% of the limit (i.e., 12 if the limit is 15) over the sample period then the responses will start to contain a throttle node which contains useful information on how close you are to reaching the call limit.