I need to get tags value of incoming api key in lambda authorizer.
In github.com/aws/aws-sdk-go-v2/service/apigateway Method GetApiKey in awsclient is throwing 429 Too Many Requests for 100 concurrent request. Is there any aws documentation on throttle limit and how can I increase limit
Since API keys are different Authorizer caching is also not helping much.
Please let me know about documentation on this and how can I overcome this issue.
In scale setup with 50 to 100 simultaneous request I am getting this error
Related
I'm currently using an API Gateway to listen for web-hooks, and then the web-hook's payload is then offloaded to a lambda. The problem is that the web-hook's payload size can vary. I'm aware that API Gateway's allows for request of 10 MBs; while, lambdas allow only up to 6 MBs. Due to that, I don't want accept any payloads that exceed 6 MB. So I'm wondering if there's any way to filter out request prior to hitting my lambda that exceed 6MBs or some desired amount?
Also I've look into AWS' WAF, but that seems to only apply to request under 8KB if you use a size constraint statement. I don't think this would work in my case as I can easily see the payloads exceeding that amount.
I currently have a lambda that is triggered by requests made to an API Gateway route, and I made some research about how to set a payload limit (e.g. 2kb) for this route. My goal is to guarantee that my lambda will never receive a large input to deal with, so it will have a slight execution time and low costs.
I found that the default payload limit for AWS API Gateway is 10 MB, while the default limit to AWS Lambda is 6 MB. Both of them cannot be increased.
However, I did not found any docs or discussion about how to decrease it. Is it possible? Are there any other AWS services that I should use as a middleware between API Gateway and my lambda to limit the received input? Or should I solve it by another approach?
AWS API Gateway does not have the functionality you are describing.
What you are describing I would consider a WAF function, and indeed AWS WAF does have this functionality.
https://docs.aws.amazon.com/waf/latest/developerguide/web-acl-size-conditions.html
I'm hesitant to add the following as it is a complete mis-use of the toolset, but...
A dirty and likely an inexact method to restrict payload size using just API Gateway would be check the length of the body of the incoming request with an IF statement in a body mapping template in the Integration Response, and if it is over a certain length, ignore it.
Example assuming Content-Type is application/json:
#set($allParams = $input.params())
{
#if ($input.body.length() < 5000)
"request" : $input.json('$')
#else
"request" : null
#end
}
The 5000 is just an arbitrary number, I haven't done the maths.
Doing this would still lead to your Lambda function being invoked even if the payload was to big (well long), and the Lambda would need to be able to deal with an empty request, but it will never receive a request with more than 5000 characters.
We currently use our applications to download some pins information for analytics purposes
Currently we hit the API on hourly basis but we have some problems with rate limiting, so we need the following questions to be addresses:
The documentation states that the rate limit is 1000 calls per hour per user token per endpoint. By looking at the HTTP headers apparently the actual limit is 200. So, which is the correct one?
Are GET /v1/pins/1234 and GET /v1/pins/5678 considered to be a call to the same endpoint?
Are there any application level (so, despite of the used user token) rate-limit in place?
Are there any IP level rate limits?
Since we use concurrent requests, are there any limitation on the number of CONCURRENT request? (which means: make another call from the same IP and using the same token, even if the previous one hasn't ended yet)
Thanks in advance.
PS: I tried to contact them via support form, but it returns a 422. Which is awesome!
For apps that is still in development stage the limit is 200. You will get 1000 if your apps passed the review.
The documentation states that the rate limit is 1000 calls per hour
per user token per endpoint. By looking at the HTTP headers
apparently the actual limit is 200. So, which is the correct one?
Ans: Its 10 per token per app id per hour for app in development(tested), 1000 per token per hour for production app.
Are there any application level (so, despite of the used user token)
rate-limit in place?
Ans: No, rate limit will be 10(app in development) or 1000(approved app)/hour for each tokens you have
Are there any IP level rate limits?
Ans: No
Since we use concurrent requests, are there any limitation on the number of CONCURRENT request? (which means: make another call from the same IP and using the same token, even if the previous one hasn't ended yet)
Ans: No there is no such limit but they will probably check for bot activity and you may end up with banned account, so please do maintain some delay in between requests for same token IMO 12 seconds should be enough
I have read the gmail api quota explanation here (https://developers.google.com/gmail/api/v1/reference/quota), but am still having troubles understanding what causes us to go over the limit.
Question 1:
What is a user in a per user quota? I am not sure if the user is an individual gmail user, or a service client using the gmail api.
Question 2:
We've seen the following error a few times, but don't see any obvious limit we've hit.
"error": {
"errors": [
{
"domain": "usageLimits",
"reason": "rateLimitExceeded",
"message": "Rate Limit Exceeded"
}
],
"code": 429,
"message": "Rate Limit Exceeded"
}
We were under 250 units/s and 25,000 units/100s. We're only using history.list and message.get calls no sending or modifications.
Is there some other quota I am missing?
User quota is based upon the account you are accessing. So it would be the GMail account. Sometimes you can trick it by sending a random quotaUser but this doesn't always work Google also uses your IP address to track quota I suspect.
User rate limit is flood protection you are going to fast.
Per User Rate Limit 250 quota units per user per second, moving
average (allows short bursts)
Exceeding a rate limit will cause an HTTP 403 or HTTP 429 Too Many
Requests response and your app should respond by retrying with
exponential backoff.
Googles calculations are not perfect you could be sending more or less and still hit this quota. Just implementexponential backoff.
Exponential backoff
The flow for implementing simple exponential backoff is as follows:
Make a request to the API.
Receive an HTTP 403 rate-limited response, which indicates you should retry the request.
Wait 1 + random_number_milliseconds seconds and retry the request.
Receive an HTTP 403 rate-limited response, which indicates you should retry the request.
Wait 2 + random_number_milliseconds seconds, and retry the request.
Receive an HTTP 403 rate-limited response, which indicates you should retry the request.
Wait 4 + random_number_milliseconds seconds, and retry the request.
Receive an HTTP 403 rate-limited response, which indicates you should retry the request.
Wait 8 + random_number_milliseconds seconds, and retry the request.
Receive an HTTP 403 rate-limited response, which indicates you should retry the request.
Wait 16 + random_number_milliseconds seconds, and retry the request.
Stop. Report or log an error.
For your Question 1
Here are the meaning of the different quota in your Gmail
QPD(quota per day) - meaning the maximum numbers of request over a 24 hour period a client id is able to make to an API
QPS(quota per second) - meaning a global quota per second for the application, meaning how many calls a second an application can make
quota per seconds per user - meaning the number of queries a user, the application can make.
For question number 2
Well, if you check the Quota of Gmail in your developer console, the Gmail has a default quota of:
So what can I suggest you is to use the following tips so that you work with your quota efficiently:
Push notification - it improve the performance of your application. It allows you to eliminate the extra network and compute costs involved with polling resources to determine if they have changed. Whenever a mailbox changes, the Gmail API notifies your backend server application.
Use synchronization to retrieve and store as many of the most recent messages or threads as are necessary for your purpose.
Batching Requests - to reduce the number of HTTP connections your client has to make.
If you notice that you reach this limit and you need more than this, then you can apply for more quota here.
I have an angularjs app that calls a RESTful service at apiary.io
Does apiary.io throttle responses and delay responses after a certain number have been received?
If so what are the parameters?
Currently, Apiary limits you for 120 reqs/minute/IP.
There are no artificial delays, but occasionally someones floods Apiary with production traffic and even when ratelimiting is fairly efficient, it may temporarily degrade service for other users.
You can (and should) check X-Apiary-RateLimit-Limit and X-Apiary-RateLimit-Remaining header. Once you'll hit the limit, Apiary will sent Retry-After header you should obey.
From their docs:
API Call Limit
API calls are subject to the default limit of 15 requests per second and exceeding this limit will result in all endpoints returning an HTTP status code of 429. Limits are per API key. If the limit is exceeding then the API Key will be blocked for the remainder of the sample period. If an API key continually hits the call limit we reserve the right to permanently block the key and to charge a fee to unblock the key.
To determine the API call amount we monitor the traffic over a sample period. If the traffic results in a particular API key reaching 80% of the limit (i.e., 12 if the limit is 15) over the sample period then the responses will start to contain a throttle node which contains useful information on how close you are to reaching the call limit.