Currently my app easily reaches 2-5 requests/min with 5 users. From my understanding of Parse this is rather high.
To give an example of what we do is:
When a user logs on and refreshes the list is filled with all relevant information from the database (query that specific user and gather all of his event/group data from server)
When a user creates an event, create all of the data and pass it to the server.
From my understanding a request/min is only when you make a network call. In this case, I see that when a user refreshes that is 1 network call as well as when a user creates an event it is 1 network call. Is there something I am missreading about request calls?
Thanks!
30 requests per second is the (current) baseline limit (1800 per minute).
1 network request could be multiple requests for a number of reasons (batch processing, calling a cloud function, anything which has a save hook, ...). Depending on the API you use you could be requesting a batched save of multiple objects for instance.
If all of your users are continually averaging 1 request per second then you have a problem. If your users each have a short burst of 5 per second followed by a long period of no requests then you're fine.
Related
I've an application with 10M users. The application has access to the user's Google Health data. I want to periodically read/refresh users' data using Google APIs.
The challenge that I'm facing is the memory-intensive task. Since Google does not provide any callback for new data, I'll be doing background sync (every 30 mins). All users would be picked and added to a queue, which would then be picked sequentially (depending upon the number of worker nodes).
Now for 10M users being refreshed every 30 mins, I need a lot of worker nodes.
Each user request takes around 1 sec including network calls.
In 30 mins, I can process = 1800 users
To process 10M users, I need 10M/1800 nodes = 5.5K nodes
Quite expensive. Both monetary and operationally.
Then thought of using lambdas. However, lambda requires a NAT with an internet gateway to access the public internet. Relatively, it very cheap.
Want to understand if there's any other possible solution wrt the scale?
Without knowing more about your architecture and the google APIs it is difficult to make a recommendation.
Firstly I would see if google offer a bulk export functionality, then batch up the user requests. So instead of making 1 request per user you can make say 1 request for 100k users. This would reduce the overhead associated with connecting and processing/parsing of the message metadata.
Secondly i'd look to see if i could reduce the processing time, for example an interpreted language like python is in a lot of cases much slower than a compiled language like C# or GO. Or maybe a library or algorithm can be replaced with something more optimal.
Without more details of your specific setup its hard to offer more specific advice.
I created a new group through Admin control panel and use Directory API to add new members in bulk, using batch-update, to the tune of 1000 users.
The documentation states that it should be valid.
You're limited to 1000 calls in a single batch request. If you need to make more calls than that, use multiple batch requests.
The script ends without error, however not all users are showing. Sometimes 300 get added on each execution, sometimes 150. I have to refresh it again and then more members are being added in chunks.
I wanted to clarify from experts if there are any restrictions to this approach using API? For example, may be, you can't add more than X number of users to a newly created group and perhaps you have to wait 24 hours.
as the documentation states a batch can contain a max of 1000 request however I recommend going a bit lower say 900
You're limited to 1000 calls in a single batch request. If you need to make more calls than that, use multiple batch requests.
We currently use our applications to download some pins information for analytics purposes
Currently we hit the API on hourly basis but we have some problems with rate limiting, so we need the following questions to be addresses:
The documentation states that the rate limit is 1000 calls per hour per user token per endpoint. By looking at the HTTP headers apparently the actual limit is 200. So, which is the correct one?
Are GET /v1/pins/1234 and GET /v1/pins/5678 considered to be a call to the same endpoint?
Are there any application level (so, despite of the used user token) rate-limit in place?
Are there any IP level rate limits?
Since we use concurrent requests, are there any limitation on the number of CONCURRENT request? (which means: make another call from the same IP and using the same token, even if the previous one hasn't ended yet)
Thanks in advance.
PS: I tried to contact them via support form, but it returns a 422. Which is awesome!
For apps that is still in development stage the limit is 200. You will get 1000 if your apps passed the review.
The documentation states that the rate limit is 1000 calls per hour
per user token per endpoint. By looking at the HTTP headers
apparently the actual limit is 200. So, which is the correct one?
Ans: Its 10 per token per app id per hour for app in development(tested), 1000 per token per hour for production app.
Are there any application level (so, despite of the used user token)
rate-limit in place?
Ans: No, rate limit will be 10(app in development) or 1000(approved app)/hour for each tokens you have
Are there any IP level rate limits?
Ans: No
Since we use concurrent requests, are there any limitation on the number of CONCURRENT request? (which means: make another call from the same IP and using the same token, even if the previous one hasn't ended yet)
Ans: No there is no such limit but they will probably check for bot activity and you may end up with banned account, so please do maintain some delay in between requests for same token IMO 12 seconds should be enough
tl/dr do 100 devices all using the same Client ID count as 100 users, with their own limits, or one user sharing limits?
I have a webpage which reads and writes to a Google Sheet.
Because the webpage needs to know if a cell has changed, it polls the server once every 1000ms:
var pollProcId = window.setInterval(pollForInput, 1000);
where pollForInput does a single:
gapi.client.sheets.spreadsheets.values.get(request).then(callback);
When I tried to use this app with a class of 100 students I got many 429 error codes (more than I got successful reads) in response to google.apps.sheets.v4.SpreadsheetsService.GetValues requests:
Many of my users never got as far as seeing even the first request come back.
As far as I can make out, these are AnalyticsDefaultGroupUSER-100s errors which, according to the error responses page:
Indicates that the requests per 100 seconds per user per project quota
has been exhausted.
But with my app only requesting once per 1000 milliseconds, I wouldn't expect to see this many 429s as I have a limit of 100 requests per 100 seconds (1 per second) so only users whose application didn't complete in 100 seconds should have received a 429.
I know I should implement Exponential Backoff (which I'll do, I promise) but I'm worried I'm misunderstanding what a "user" in this context is.
Each user is using their own device (so presumably has a different IP address) but they are all using my "Client ID".
Does this scenario count as many users making one request per second, or a single user making a hundred requests per second?
Well, the user in the per user quota means that a single user making a request. So let's take the Sheets API, it has a quota of 100 for the Read requests per 100 seconds per user. So meaning only a single user can make a request per second. Note that Read request has a same set of quota as the Write request. But these two sets of quotas have their own set of quota and didn't share the same limit quota.
If you want a higher quota than the default, then you can apply for a higher quota by using this form or by visiting your developer console, then click the pencil icon in the quota that you want to increase.
I also suggest you to do the Exponential Backoff as soon as possible, because it can help you to avoid getting this kind of error.
Hope it helps you.
According to the parse.com pricing page, push notifications are free up to 1 million unique recipients.
API calls are free up to 30 requests / second.
I want to make sure there is no catch here.
An example will clarify: I have 100K subscribed users. I will send weekly push notifications to them. In a month, that will be 4 push "blasts" with 100K recipients each. Is this covered by the free tier? Would this count as 4 API calls, 400K API calls, or some other amount?
100k users is 1/10 the advertised unique recipient limit, so that should be okay.
Remember that there's a 10sec timeout, too. So the only way to blast 100k pushes within the free-tier resource limits is to create a scheduled job that spends about 2 hours (that's a safe rate of 15 req/sec) doing pushes and writing state so you can pick up later where you left off.
Assuming there's no hidden gotcha (you'll probably need to discover those empirically), I think the only gotcha in plain sight is the fact that the free tier allows only one (1) scheduled job. Any other long-running processing -- and there are bound to be some on 100k users -- are going to have to share the job, making the what-should-this-single-job-work-on-now logic pretty complex.
You should take a look at the FAQ for Parse.com:
https://www.parse.com/plans/faq
What is considered an API request?
Anytime you make a network call to
Parse on behalf of your app using one of the Parse SDKs or REST API,
it counts as an API request. This does include things like queries,
saves, logins, amongst other kinds of requests. It also includes
requests to send push notifications, although this is seen as a single
request regardless of how many recipients are targeted. Serving Parse
files counts as an API request, including static assets served from
Parse Hosting. Analytics requests do have a special exemption. You can
send us your analytics events any time without being limited by your
app's request limit.