I'm building a pretty large app for a client that is going to aggregate feeds from various sources. My client estimates around 900 follow-able users will be in this system to start out, with more being added over time. He wants to update the feed data every 15 minutes, so we would need to update one user feed per second, assuming 900 feeds and a 15 minute TTL. As the requests take a few seconds to complete, we would then need to load balance across a few threads to tackle the queue asynchronously.
Should I be worried about quota errors or hitting any kind of limitations? If so, what are our options?
I've already read their help pages and documentation, but it's very vague; I need concrete numbers. It's not feasible to load test their API to figure out the limitation.
Version 3 of the YouTube Data API has concrete quota numbers listed in the Google API Console where you register for your API Key. You can use 10,000 units per day. Projects that had enabled the YouTube Data API before April 20, 2016, have a default quota of 50,000,000 per day.
You can read about what a unit is here:
https://developers.google.com/youtube/v3/getting-started#quota
A simple read operation that only retrieves the ID of each returned resource has a cost of approximately 1 unit.
A write operation has a cost of approximately 50 units.
A video upload has a cost of approximately 1600 units.
If you hit the limits, Google will stop returning results until your quota is reset. You can apply for more than 1,000,000 requests per day, but you will have to pay for those extra requests.
There is a calculator provided by YouTube to check your usage. It is a good tool to estimate your usage.
https://developers.google.com/youtube/v3/determine_quota_cost
If you need to make more requests than allotted, you can request a higher quota here: https://support.google.com/youtube/contact/yt_api_form
Related
Reading the documentation on here I am still not clear of the following points:
Are there any limits on the size of the API request to the Google cloud Storage bucket? (We need to transfer PDFs from a CRM to Google Cloud Storage)
How many files we can send (it mentions a limit of 1000 writes per second) is that the same thing?
Files are the objects you store in Cloud Storage, so the maximum size limit 5 TB should be considered for individual files. There is no limit to write across multiple objects, however buckets support roughly 1000 writes per second and then scale up as needed.
For parallel uploads, please take note that up to 32 objects/files can be composed in a single composition request and a per-project composition rate limit of approximately 1,000 source objects per second.
I also recommend that you take a look at best practices on how to ramp up the request rate
You go with more than 1000 writes per second per bucket - the GCS infrastructure will scale automatically to accommodate it. The only requirement is not to ramp up the load too quickly, namely "double the request rate no faster than every 20 minutes".
https://cloud.google.com/storage/docs/request-rate#ramp-up
Parse now allows us to send 30 requests/second, but it is not straightforward to me.
Quoting some info gathered:
Here they say
At 30 requests/sec, an app can send us 77.76 million API requests in
a month before needing to pay a dime.
So I suppose he meant
send up to 77.76 million
Here, they suggest the rate of requests/second is calculated in a small window, generally a minute. This was answered about 2 years ago.
On their pricing faq page they give an example:
if an app is set to 30 requests/second, your app will hit its request
limit once it makes more than 1,800 requests over a 60 second period.
Suggesting that the window is one minute, even though they didn't clearly state it.
What intrigues me is that they say:
Pricing is pro-rated by the hour.
What does it mean? (sorry if it's obvious, English is not my first language)
Has anyone actually used parse and kept track of those request/second and burst/limits?
The only resource I found was a guy saying he had a web application with 10,000 users/day staying in the website around 4min, and he had under 12r/s.
Moreover, if my app logs users' activities, would that be a good practice to cache this info then send it in random times like between from 3am-7am?
Any help is very appreciated. My company is deciding whether go forward with Parse or not, so it's very important.
They could have worded it better but it basically means the same as "We'll charge you for a minimum of 1 hour based on the request limit you have set".
Here's an example. Assume you are using a 40 rps settings ($100/month which is $100/720 hours). If you keep this for 1 minute, you'll be charged for 1 hour, roughly $1.40.
You can change the request limit as often as you want. So if your app/site receives peak traffic for only 12 hours/day, you can increase the limit just for those 12 hours and end up paying just for those 12 hours.
Check the third question (How frequently can I increase/decrease my request limit?) on the FAQ page at https://parse.com/plans/faq
How frequently can I increase/decrease my request limit?
You can increase/decrease your request/limit as frequently as you would like
within a given month. We will prorate your charges on an hourly basis.
Not really clear what the pro rating means as I understand the setting to be an explicit limit that you pay for. If your limit is exceeded then the requests fail. I don't think there's an option to allow for payment on demand when the limit is exceeded but pro rating would do that.
The one minute is accurate and that is the current limit management.
The point of the pricing model is that your service should be making money before you reach any of the limits. If you have enough users to hit the limits and you aren't making money then you need to reconsider your business plan. As such you shouldn't need to upload at random times of day as your users should naturally spread out a bit.
Here is something that can help you understand Parse's users request per second.
Parse estimates that the average app's active user will issue 10 requests. Thus, if you had a million users on a particular day, and their traffic was evenly spread throughout the day, you could estimate your app would need about 10,000,000 total API requests, or about 120 requests per second. Every app is different, therefore Parse encourages you to measure how many requests your users send.
You can read more this question answered directly from Parse staff here on Parse/F&A link
Hope this helps
We are building a new application in parse and are trying to estimate our requests/second and optimize the application to limit it and keep it below the 30/second. Our app, still in development, makes various calls to parse. Some only use 1 requests, and a few as many as 5 requests. We have tested and verified this in the analytics/events/api requests tab.
However, when I go to the analytics/performance/total requests section, the requests/second rarely go above .2 and are often much lower. I assume this is because this is an average over a minute or more. So I have two questions:
1) does anyone know what the # represents on this total requests/second screen. Is it an average over a certain time period. If so, how much?
2) when parse denies the request due to rate limit, does it deny based on the actual per second, or is it based on an average over a certain time period?
Thanks!
I supposed you have your answer by now but just in case:
You're allowed 30reqs/sec on a free plan, but Parse actually counts it on a per minute basis, or 1800 requests per minute.
So here we are again with the usual bullshit from Google.
They claim their speech-recognition api for developers has a limit of 50 requests per day. But it is evident that there is some other per-minute or per-some-smaller-time-unit limit, because I have been trying requests without reaching 50 per day, but if they are "too close" to one another I randomly get an empty array as a result (and I'm testing with always the same sound files, identical requests).
So please, SOMEBODY FROM GOOGLE (as they say that they monitor SO for questions tagged with google-api), tell me: what are the REAL EXACT usage rate limits for this api?
So that I can adjust my app to those limits instead of wasting my requests per day just to figure out the more strict limits.
Please, only answers from Google staff, and no bullshit.
OR you could just sleep some seconds between requests and/or backoff if you hit that unofficial limit.
Other Google APIs can also fail for seemingly random causes. You just need to retry some time later.
BTW: harsh language is not making yourself any favor.
I'm building a pretty large app for a client that is going to aggregate feeds from various sources. My client estimates around 900 follow-able users will be in this system to start out, with more being added over time. He wants to update the feed data every 15 minutes, so we would need to update one user feed per second, assuming 900 feeds and a 15 minute TTL. As the requests take a few seconds to complete, we would then need to load balance across a few threads to tackle the queue asynchronously.
Should I be worried about quota errors or hitting any kind of limitations? If so, what are our options?
I've already read their help pages and documentation, but it's very vague; I need concrete numbers. It's not feasible to load test their API to figure out the limitation.
Version 3 of the YouTube Data API has concrete quota numbers listed in the Google API Console where you register for your API Key. You can use 10,000 units per day. Projects that had enabled the YouTube Data API before April 20, 2016, have a default quota of 50,000,000 per day.
You can read about what a unit is here:
https://developers.google.com/youtube/v3/getting-started#quota
A simple read operation that only retrieves the ID of each returned resource has a cost of approximately 1 unit.
A write operation has a cost of approximately 50 units.
A video upload has a cost of approximately 1600 units.
If you hit the limits, Google will stop returning results until your quota is reset. You can apply for more than 1,000,000 requests per day, but you will have to pay for those extra requests.
There is a calculator provided by YouTube to check your usage. It is a good tool to estimate your usage.
https://developers.google.com/youtube/v3/determine_quota_cost
If you need to make more requests than allotted, you can request a higher quota here: https://support.google.com/youtube/contact/yt_api_form