AWS lambda can be used to create serverless applications, but it has some limits. One of these is a limit on the payload size: you can only return a maximum if 1MB payload from your function.
Does anyone know why this limit exists?
AWS Lambda limits the amount of compute and storage resources that you can use to run and store functions. AWS has deliberately put several Lambda limits that are either soft or hard to prevent misuse or abuse.
Here is the reference from AWS documentation AWS Hard and Soft Limits
Related
I have a lambda function that uses SSM ParameterStore values. It queries the ParameterStore and stores the fetched values in Lambda env variables so that next time it can use them from env variables instead of making calls to the ParameterStore which is working fine if the lambda is in a hot-state, but during the cold start still, my lambda is making many calls to ParameterStore during peak traffic and getting throttling exceptions.
I'm looking to reduce the num of calls to the parameter store by having a caching layer, I found this article online, but I'm new to Lambda extensions. I just wanted to check if this caching works between the cold starts or not before I create a POC. please advise.
Thanks in advance!
I'm developing an AWS Serverless architecture.
I have a lambda attached to a subnet within a VPC. I have setup a VPC endpoint to reach my DynamoDB table.
When my lambda is cold, it takes up to 200 - 300ms make a simple GetItem command to my dynamodb table. This is just the GetItem command, I have already substracted lambda initialization, DynamoDB client instanciation etc. This is unacceptable for my application.
However, when my lambda is hot, I only take ~8-9msm for the GetItem command, which is acceptable.
Is there some ENIs latencies because my lambda is attached to a subnet ? If so, what can I do to speed it up ? Or is there another problem that I do not see ?
The latencies you're experiencing is due to the metadata caching that DynamoDB utilizes to lower latency.
When requests are made frequently to DynamoDB it makes use of caching certain metadata locally, such as authentication and data locality etc....
When requests are infrequent this cache is stale and results in extra hope by the DynamoDB service in order to serve your requests. This is one of the fundamental reasons that DynamoDB latency decreases as throughput increases.
Some thing's that you can do to avoid the latency hit:
Ensure you reuse TCP connections by setting keep-alice to true
Ensure client is created outside of the Lambda handler
Send dummy traffic to your DynamoDB table to ensure it keeps the metadata cache warm.
I use AWS Simple Email Services (SES) for email. I've configured SES to save incoming email to an S3 bucket, which triggers an AWS Lambda function. This function reads the new object and forwards the object contents to an alternate email address.
I'd like to log some basic info. from my AWS Lambda function during invocation -- who the email is from, to whom it was sent, if it contained any links, etc.
Ideally I'd save this info. to a database, but since AWS Lambda functions are costly (relatively so to other AWS ops.), I'd like to do this as efficiently as possible.
I was thinking I could issue an HTTPS GET request to a private endpoint with a query-string containing the info. I want logged. Since I could fire my request async. at the outset and continue processing, I thought this might be a cheap and efficient approach.
Is this a good method? Are there any alternatives?
My Lambda function fires irregularly so despite Lambda functions being kept alive for 10 minutes or so post-firing, it seems a database connection is likely slow and costly since AWS charges per 100ms of usage.
Since I could conceivable get thousands of emails/month, ensuring my Lambda function is efficient is paramount to cost. I maintain 100s of domain names so my numbers aren't exaggerated. Thanks in advance.
I do not think that thousands per emails per month should be a problem, these cloud services have been developed with scalability in mind and can go way beyond the numbers you are suggesting.
In terms of persisting, I cannot really understand - lack of logs, metrics - why your db connection would be slow. From the moment you use AWS, it will use its own internal infrastructure so speeds will be high and not something you should be worrying about.
I am not an expert on billing but from what you are describing, it seems like using lambdas + S3 + dynamoDB is highly optimised for your use case.
From the type of data you are describing (email data) it doesn't seem that you would have neither a memory issue (lambdas have mem constraints which can be a pain) or an IO bottleneck. If you can share more details on your memory used during invocation and the time taken that would be great. Also how much data you store on each lambda invocation.
I think you could store jsonified strings of your email data in dynamodb easily, it should be pretty seamless and not that costly.
Have not used (SES) but you could put a trigger on DynamoDB whenever you store a record, in case you want to follow up with another lambda.
You could combine S3 + dynamoDB. When you store a record, simply upload a file containing the record to a new S3 key and update the row in DynamoDB with a pointer to the new S3 object
DynamoDB + S3
You can now persist data using AWS EFS.
In our set up, we have lots of AWS Lambda functions, developed by different teams. Some of the them have set a reserved concurrency. This eats out of the total concurrency of the account (1000).
Is there a way to monitor or set an alarm that is triggered when the unreserved currency drops below specific level?
This would be helpful to proactively do something to alleviate the issue and reduce failures.
In AWS there are pre-defined metrics, related to Lambda concurrency, that are exposed in AWS CloudWatch
ConcurrentExecutions: Shows you the concurrent executions that are happening at that moment across the all the Lambda functions in the Account including Reserved and Unreserved.
UnreservedConcurrentExecutions: This shows you the total concurrent executions, that are happening at that moment, that are using the Unreserved Concurrency.
The information I was looking for can be seen when we run the CLI command
ConcurrentExecutions and UnreservedConcurrentExecutions
$ aws lambda get-account-settings
{
"AccountLimit": {
"TotalCodeSize": 1231232132,
"CodeSizeUnzipped": 3242424,
"CodeSizeZipped": 324343434,
"**ConcurrentExecutions**": 10000,
"**UnreservedConcurrentExecutions**": 4000
},
"AccountUsage": {
"TotalCodeSize": 36972950817,
"FunctionCount": 1310
}
}
It is not possible to get these values in a dashboard. As we cannot execute API calls to fetch and display data in the dashboard.
Solution
We can create a lambda function, and, in that function, we can extract, using the API, the account wide values/settings for ConcurrentExecutions and UnreservedConcurrentExecutions. We can then create new metrics that would send the two values to CloudWatch. We can schedule AWS Lambda Functions Using CloudWatch Events.
Once we have the metric, we can set the required alarm for the Unreserved Concurrency.
I would like to know if there is a limit on the number of Lambda functions one can define (either per region or per account). To be clear I am not talking about a limit on the number of instances of a single lambda function, but instead on the number of function definitions that are allowed.
Looking at http://docs.aws.amazon.com/lambda/latest/dg/limits.html I did not find any explicit limit. There is a limit on the "Total size of all the deployment packages that can be uploaded per region" that is set to 75 GB, which indirectly limits the number of defined functions (for instance, if average size - across all of my lambda functions - of the deployment package is 40MB, then there is a limit of 75GB/40MB = 1875 functions per region).
As you found on the AWS Lambda Limits page, there is no current limit on the number of AWS Lambda functions per region or account.
You are correct, that there is a limit on Function and Layer Storage. That is described as "The amount of storage that's available for deployment packages and layer archives in the current Region." By default, this is set to 75 GB per region. However, this limit is adjustable. If you're getting close to this limit, you can request a quota increase and expand your indirect limit on the number of functions or layers.
Check out the Service Quotas service to see your current hard and soft limits for AWS services and request quota increases.
To request an increase in function storage:
In the Service Quotas console, select the "AWS Lambda" service.
Then, select "Function and layer storage" and click "Request quota increase".
On the request form, enter your requested storage amount and submit the request.