tl/dr do 100 devices all using the same Client ID count as 100 users, with their own limits, or one user sharing limits?
I have a webpage which reads and writes to a Google Sheet.
Because the webpage needs to know if a cell has changed, it polls the server once every 1000ms:
var pollProcId = window.setInterval(pollForInput, 1000);
where pollForInput does a single:
gapi.client.sheets.spreadsheets.values.get(request).then(callback);
When I tried to use this app with a class of 100 students I got many 429 error codes (more than I got successful reads) in response to google.apps.sheets.v4.SpreadsheetsService.GetValues requests:
Many of my users never got as far as seeing even the first request come back.
As far as I can make out, these are AnalyticsDefaultGroupUSER-100s errors which, according to the error responses page:
Indicates that the requests per 100 seconds per user per project quota
has been exhausted.
But with my app only requesting once per 1000 milliseconds, I wouldn't expect to see this many 429s as I have a limit of 100 requests per 100 seconds (1 per second) so only users whose application didn't complete in 100 seconds should have received a 429.
I know I should implement Exponential Backoff (which I'll do, I promise) but I'm worried I'm misunderstanding what a "user" in this context is.
Each user is using their own device (so presumably has a different IP address) but they are all using my "Client ID".
Does this scenario count as many users making one request per second, or a single user making a hundred requests per second?
Well, the user in the per user quota means that a single user making a request. So let's take the Sheets API, it has a quota of 100 for the Read requests per 100 seconds per user. So meaning only a single user can make a request per second. Note that Read request has a same set of quota as the Write request. But these two sets of quotas have their own set of quota and didn't share the same limit quota.
If you want a higher quota than the default, then you can apply for a higher quota by using this form or by visiting your developer console, then click the pencil icon in the quota that you want to increase.
I also suggest you to do the Exponential Backoff as soon as possible, because it can help you to avoid getting this kind of error.
Hope it helps you.
Related
I am getting
"code": 403,
"message": "User Rate Limit Exceeded"
while using Google Drive API in my web app
Although the quota is 10,000 requests per 100 seconds and my average is less than 2:
How can I resolve this error? How to implement exponential backoff as the documents say?
There are sevrail types of quotas with Google apis.
Project based quotas which effect your project itself. These quotas can be extended. If for example you your project can make 10000 requests pre 100 seconds. you could request that this be extended.
Then there is the user based quotas. these quotas limit how much each user can send.
User Rate Limit Exceeded
Means that you are hitting a user rate quota. User rate quotas are flood protection they ensure that a single user of your application can not make to many requests at once.
These quotas can not be extended.
if you are getting a user rate limiting quota then you need to slow down your application and implement exponential backoff.
How you implement exponential backoff is up to you and the language you are using but it basically involves just retrying the same request again only adding wait times each time it fails
the graph
the graph in the google cloud console is an guestimate and it is not by any means accurate. If you are getting the error message you should go by that and not by what the graph says.
After hours of searching and thinking, I found out that,
'User Rate Limit Exceeded' has a spam protection which allow max 10 requests per second.
Thus I found out a lazy trick to do so by delaying the calls using:
usleep(rand(1000000,2000000);
It simply delays the call by a random duration between 1 and two seconds.
I keep getting errors from the Google API telling me that I have exceeded the limit of 100 reads per second per user when trying to read from Google Sheets using the API. I'm using CozyRoc's REST Connection and REST Source connected to Google Sheets then connecting to a SQL Server Destination table and trying to populate the table from the Google Sheet.
Please don't try to offer any suggestions about other programs or other ways of accessing the data. I have scores of SSIS packages that use the same set up to import data. In all of those that cycle through about 10 Google sheets with anywhere from one to 20 tabs and as many as 400 rows, I've had to set the Reads per Second to one to avoid the error. That makes my uploads incredibly slow. And YES, I contacted CozyRoc who tells me it's a Google API problem, not theirs, and there's nothing they can or will do about it.
So . . . I made sure we have a billing account . . .
I was able to sign in and look at the Quotas screen. 500 reads per Second and 100 Reads per Second per user.
I was able to request an increased limit for the Reads Per Second to 500 which doesn't require a billing account, but that doesn't change the Reads Per Second Per User.
CozyRoc uses my user id (oAuth token) to access Google Sheets, so every read only comes from one user.
A popup displays when I try to edit the Reads Per Second Per User from the Quotas screen asking me to set the limit to a maximum of 100 or to request a quota limit increase.
I click on that request button and I am immediately returned to the Quotas screen. I'm never asked to set a new limit, I'm never told that the request was sent and received, and I see no notice telling me how long it may take to process the request. It's been about a week now since I first tried. It would be nice if they would tell you SOMETHING!
My thought is that there may be something wrong with their website, perhaps with that popup. I've tried calling Cloud Support, but they refuse to help, saying basically "Not my job to help you with setting quota limits," even though my question is really whether there is a known issue with the quota limit increase website/popup.
SO . . . Is anyone else having a similar problem?
Can anyone tell me if what's happening is normal and that it just takes WEEKS for Google to process a quota limit increase or how long it normally takes for them to process a quota limit rate increase?
Is there anywhere I can reach out to at Google where I can do a screen sharing session and show them what's happening to get an answer or find out whether or not my request was even received?
Any ideas or thoughts as to how I can find out what's going on?
Please let me know.
Thanks,
How to edit API specific quota
The user per minute quota can be edited API and project specific
Go to the GCP console
Chose the project for which you want to icnrease the quota limit
Go onIAM & Admin -> Quotas
Choose Google Sheets API and Read reuquests
Click on the pencil next to Read requests per 100 seconds per user
Choose a quota limit of below 100 - which is the maimum limit allowed by the Sheets API
Click on Save
If you want to increase your limit above 100 requests per 100 seconds per user:
Mind that this limit is the above the normally alloweed quota
Click on apply for higher quota
Click on ALL QUOTA for Read requests per 100 seconds per user
Check the toickbox next to GLOBAL
Confirm your contact details and click on Next
Enter the desire new limit and click on Done
KEEP IN MIND
If you do not have a billing account, you are not eligible to chose a limit of more than 100 read requests for user per 100 seconds and you will get the error message
You can't request more quota because your project is not linked to a billing account.
We currently use our applications to download some pins information for analytics purposes
Currently we hit the API on hourly basis but we have some problems with rate limiting, so we need the following questions to be addresses:
The documentation states that the rate limit is 1000 calls per hour per user token per endpoint. By looking at the HTTP headers apparently the actual limit is 200. So, which is the correct one?
Are GET /v1/pins/1234 and GET /v1/pins/5678 considered to be a call to the same endpoint?
Are there any application level (so, despite of the used user token) rate-limit in place?
Are there any IP level rate limits?
Since we use concurrent requests, are there any limitation on the number of CONCURRENT request? (which means: make another call from the same IP and using the same token, even if the previous one hasn't ended yet)
Thanks in advance.
PS: I tried to contact them via support form, but it returns a 422. Which is awesome!
For apps that is still in development stage the limit is 200. You will get 1000 if your apps passed the review.
The documentation states that the rate limit is 1000 calls per hour
per user token per endpoint. By looking at the HTTP headers
apparently the actual limit is 200. So, which is the correct one?
Ans: Its 10 per token per app id per hour for app in development(tested), 1000 per token per hour for production app.
Are there any application level (so, despite of the used user token)
rate-limit in place?
Ans: No, rate limit will be 10(app in development) or 1000(approved app)/hour for each tokens you have
Are there any IP level rate limits?
Ans: No
Since we use concurrent requests, are there any limitation on the number of CONCURRENT request? (which means: make another call from the same IP and using the same token, even if the previous one hasn't ended yet)
Ans: No there is no such limit but they will probably check for bot activity and you may end up with banned account, so please do maintain some delay in between requests for same token IMO 12 seconds should be enough
I'm currently integrate Google+ API to my service.
I'd like to know what's the limitation for this kind of api:
https://www.googleapis.com/plus/v1/people/{user_id}/activities/public
in google develop console, I found this:
Quota summary
Free quota 10,000 requests/day
Remaining 9,998 requests/day
99.98% of total
Per-user limit
5 requests/second/user
I think there will be two kind of limitation:
Application level
For example, how many requests can an app send per day(sum of the number for all users), and what's the max qps?
User level
For example, how many requests can an app send per day for a special user, and what's the max qps?
But I can't find the exactly info, does anyone know?
Can't say particularly about this API, but when I used Google Places api the quota was linked with IP address.So if it expired, we need to use different IP for hit.No user / application quota.
you already have those answers in your question:
10,000 requests per day in total, using that developer console key, thus your total "app" calls.
per user there is no limit, there is a rate quota. a single user could at most make 5*(seconds in a day) requests per day.
in this specific api case this is much bigger than 10,000 thus that rate quota is not that useful (except it prevents users from quickly depeting the 10,000 daily quota).
you can edit that rate quota so its lower or higher, and is used so a single user cant consume all the app quota (maliciously or otherwise)
Lately, in our app using IPP data services, we have encountered these errors from time to time.
<RestResponse xmlns="http://www.intuit.com/sb/cdm/v2">
<Error RequestId="49f7926a9aa84cfc8289534801dee72d">
<RequestName>ErrorRequest</RequestName>
<ProcessedTime>2012-12-07T10:10:59+00:00</ProcessedTime>
<ErrorCode>3001</ErrorCode>
<ErrorDesc>message=This client has made too many consecutive requests over too short a period of time. Please wait a short amount of time before attempting to submit again; errorCode=003001; statusCode=403; source=Throttling Policy</ErrorDesc>
</Error>
</RestResponse>
Can't find any reference to a "Throttling Policy" or error code "3001" anywhere in the IPP documentation.
Any help in figuring out what the throttling limits are would be appreciated. Are they based around an IP, rate limitation, concurrency limitation, OAuth consumer, OAuth client, something else perhaps?
EDIT: Link to IDN forums about the same issue: https://idnforums.intuit.com/messageview.aspx?catid=69&threadid=18910.
Yes, there is a throttling process in place if > 500 requests per minute by a single user or against a single realm.
You had over 600 requests during the one minute period.
Looks like almost all (all except 32 requests) were individual customer queries… all different customer record ids. Is there a way you can make a single customer list query, filtered if necessary, to get a bunch of customer records in a single request and reduce the number of calls you are making.