I am currently developing an API management platform where it is possible to move every endpoint action to a serverless function (lambda).
My question is: It is cheaper to use a single lambda function which then invokes the complete app and the app makes internally the routing or it is better to use the AWS routing and create a lambda for each endpoint, in my case this could be (100+) lambda instances.
From a technical perspective I think it is better to have multiple lambda functions since then we can also scale each function independently but I am not sure how it looks on the costs side. So please let me know if you have any experiences.
Look here:
https://s3.amazonaws.com/lambda-tools/pricing-calculator.html
The most important thing to keep in mind, is run short time functions at lambda, slow executions can increase your budget! But many lambda fast invokes no!
You need to know your executions time! To maintain a very large set of lambda functions i recommend to you:
https://www.serverless.com/
Related
I have a lambda function that uses SSM ParameterStore values. It queries the ParameterStore and stores the fetched values in Lambda env variables so that next time it can use them from env variables instead of making calls to the ParameterStore which is working fine if the lambda is in a hot-state, but during the cold start still, my lambda is making many calls to ParameterStore during peak traffic and getting throttling exceptions.
I'm looking to reduce the num of calls to the parameter store by having a caching layer, I found this article online, but I'm new to Lambda extensions. I just wanted to check if this caching works between the cold starts or not before I create a POC. please advise.
Thanks in advance!
Is there a good way to orchestrate lambda functionality that changes based on a queue message? I was thinking about taking a similar approach described in the strategy pattern.
The lambda function is polling an SQS queue. The queue message would contain some context that is passed into a lambda telling it what workflow needs to be executed. Based on this message, the lambda would execute some corresponding script.
The idea behind this is that I can write code for different ad hoc jobs and use the same queue + lambda function for these jobs but have it delegate the work. This way, I can track unsuccessful jobs in a dead letter queue. Are there any red flags here or potential pitfalls I should be aware of when you hear this? Any advice would be appreciated. TIA!
EDIT: For some additional context, this different workflows triggered by this lambda will vary in compute resources needed. An example is ingesting a large dataset from an api call and doing some custom schematization on the contents before making an api call.
This is indeed possible, but there's a variety of approaches you may take. These depend on what type of workflow/processing you require.
As you highlight, Lambda could be used for this. It's worth noting that Lambda functions do not work well for computationally-intensive tasks.
If you were looking to perform a workflow with some complexity, you should consider AWS Step Functions. Suppose you had three "tasks" to choose from, you could define a Step Function for each, then use Lambda to (1.) receive the message & work out which task is required, then (2.) start an execution for the desired Step Function.
FYI, you don't need to make your Lambda function poll the SQS queue, instead, you can set up SQS to automatically trigger Lambda once a new message is added to the queue. See AWS Docs - Configuring a queue to trigger an AWS Lambda function.
If you edit your question with more info on what you're looking to do (processing-wise) with each message, people will be able to better help with your use-case.
Best of luck! :)
I have a Lambda function bound to CodeBuild notifications; a Lambda instance writes details of the notification that triggered it to a DynamoDB table (BillingMode PAY_PER_REQUEST)
Each CodeBuild notification spawns an independent Lambda instance. A CodeBuild build can spawn 7-8 separate notifications/Lambda instances, many of which often happen simultaneously.
The Lambda function uses DynamoDB:PutItem to put details of the notification to DynamoDB. What I find is that out of 7-8 notifications in a 30 second period, sometimes all 7-8 get written to DynamoDB, but sometimes it can be as low as 0-1; many calls to DynamoDB:PutItem simply seem to be "ignored".
Why is this happening?
My guess is that DynamoDB simply shouldn't be accessed by multiple Lambda instances in this way; that best practice is to push the updates to a SQS queue bound to a separate Lambda, and have that separate Lambda write many updates to DynamoDB as part of a transaction.
Is that right? Why might parallel independent calls to DynamoDB:PutItem fail silently?
TIA.
DynamoDB uses a web endpoint and for that reason it can handle any number of concurrent connections, so the issue is not with how many Lambdas are writing.
I typically see this happen when users do not allow the Lambda to wait until the API requests are complete and the container gets shut down prematurely. I would first check your code and ensure that your lambda is staying alive for all items to be processed, you can do this by adding some simple logging in your code.
What you are describing is a good use case for Step Functions.
As much as Lambda functions are great to glue between services, they have their overheads and their limitations. With Step Functions, you can call directly to DynamoDB:PutItem, and you can handle various scenarios and flows, such as Async calls. These flows are possible to implement in a Lambda function, however with less visibility and with less traceability.
BTW, you can also call a Lambda function from Step Functions, however, I recommend you to try and use the direct service call to maximize the benefits of the Step Functions service.
My mistake, I had a separate issue which was messing up some of the range keys and causing updates to "fail" silently. But thx for the tip regarding timeouts
I have an application where an initial lambda will spawn several asynchronous lambdas, and some of those lambdas may spawn their own asynchronous lambdas, and so on (although with a generation-counter to prevent runaways). The lambdas all currently write to a DynamoDB table. I'd like to know when the last one finishes, so as to kick off some different processes.
I can think of several ways generally:
writing specific fields to the DB, and each lambda checks if it's the last one running
SQS
SWF (Step Functions/state machines)
I'd like to know the simplest way to do this, and/or the "best" or canonical way, if there is one. Would also like to avoid SQS (although I'm going to experiment with SWF anyway, just because it sounds cool).
This is a good use case for AWS Step Functions.
The Parallel state is exactly what you need.
The Parallel state ("Type": "Parallel") can be used to create parallel branches of execution in your state machine.
Your scenario best fits with AWS Step Functions where you can define the Parallel state for Lambda steps and at the last execution to trigger the final step, which will kick of the different process. However this will simplify the state machine but will incur additional cost for individual states.
Another approach is to use DynamoDB Atomic Counters to keep track of each execution so that after the last execution, the Lambda function attached to the Stream, can identify and kick of the different process.
I have a use case that I need to implement using aws lambda functions. I have two or more functions that make calls to web service to get data with a criteria. I need to use results from all the process, compare it against other input criteria and produce a final result.
My question is:
Is it a good idea to implement the web service function as a lambda step up function executing in parallel?
If so, do I need a another lambda function to process results. And will the final lambda wait for the completion of execution of parallel lambda functions to complete.
And how will the final lambda get the outputs from all the parallel lambda functions as input?
This is the ideal usecase for Step Functions.
If you want to orchestrate multiple lambda and have dependencies among each other, Step Functions is one of the way to accomplish to complete the task in a scalable way.
While there are other technologies such as streams, SQS available to decouple services. All orchestrations need to be done manually.
With Step Functions, you can take output of one lambda and feed as input of another lambda.
Step Functions:
https://aws.amazon.com/step-functions/details/
Hope it helps.