There is an SNS topic that I would like to listen in on and I understand that I can either use SQS with SWF to work on each event or have AWS Lambda subscribe directly to SNS to work on each event when it arrives. For each event all I plan to do is pull out certain information and store it into Elastic Search.
My question is when would I use one method versus the other? Is one better when it comes to handling errors?
For your use case you definitely want to Lambda.
SWF is much more complicated and is designed for longer processes, with multiple steps, that may take days to complete. For SWF I generally think of use cases like a customer placing an order on a website triggering a workflow that takes the order through all the steps of the process of billing, manufacturing, packaging, shipping, etc.
Related
I am relatively new the AWS Lambda, I wanted to run my workflow through people who are more experienced. I am generating alerts(15-20), each alert is unique, they talk to another service using a few API calls. I have turned these API calls into Lambda functions which each alert uses respectively. One of the lambdas iterates over each alert and sends the alert name using SNS to another lambda function which turn the alert names into classes which then call the main of the class. I did this to avoid making a lambda function per alert. My question is whether that was a good strategy, or would it be best to create a lambda function per alert? So, instead of sending the alert name using SNS to invoke the second lambda, it would invoke a lambda with the given alert name. The one downside I see to way I've done it is that I am seeing logs of all alerts inside of one lambda function vs creating a lambda function per alert and muddying up the cdk.
From reading the above it seems like you may have made a good decision keeping it in one Lambda function as having 10/15 lambdas all just generating an alert seems like it can be a bit more complex in terms of management and developer experience when it comes to testing these functions and maintaining code but would segment the logs into different log groups.
Unless the code for generating an alert is dramatically different and cannot be shared following the DRY principle then I would recommend a single lambda function to handle your alerting.
Regarding you noisy logs because of alerts, I would recommend looking into structured logging and using Log Insights for queries and building dashboards around your specific alerts and that way you can filter out each alert type almost like you would if your query a database for specific data.
You can find documentation on structured logging here: Parsing logs and structured logging
newbie here.
Reading the docs I understand we can use an incoming HTTP request as a trigger to wake up a suspended activity.
In my case, the business trigger is the arrival of a message on a bus (from another system)…..
I thought of building out dedicated hosted service that just listens to messages arriving on the bus and invoke / trigger the respective activities....
Would I be following the suggested patterns if I do that ? It feels wrong as I'd be writing some custom external code rather than relying on the declarative approach usually described in the ELSA docs...
Any thoughts welcome..
This is a great question. Both patterns are great and in fact, the declarative approach depends on supporting infrastructure (such as hosted services).
For example, let's take the HttpEndpoint and AzureServiceBusMessageReceived activities.
Both of them require supporting infrastructure:
HttpEndpoint depends on ASP.NET Core middleware to trigger workflows as HTTP requests come in
AzureServiceBusMessageReceived depends on a hosted service that contains message workers to trigger the appropriate workflows.
For your case, you don't have to write your own hosted service if you can use one of the existing messaging activities, since it's already done for you.
At the same time, it's perfectly OK to just have your own hosted service that consumes messages and trigger workflows yourself. You could make it even a bit fancier by having your hosted service trigger business-specific activities.
For example, rather than triggering some low-level "message received" activity, you could trigger a "order created" activity if that is what the message is all about.
More details about implementing these types of activities can be found https://elsa-workflows.github.io/elsa-core/docs/guides/guides-blocking-activities.
As you already discovered, there are also examples in the repository https://github.com/elsa-workflows/elsa-core/tree/master/src/samples.
I was only considering the Elsa Guides, but just discovered a whole list of additional samples in the Elsa-Core project itself. In particular, there are several examples that seem to handle my use case (example Elsa.Samples.RabbitMqWorker)....
We are moving our infrastructure to cloud formation since it's much easier to describe the infrastructure in a nice manner. This works fantastically well for things like security groups, routing, VPCs, transit gateways.
However, we have two issues which we are struggling with and I don't think fit the declarative, infrastructure-as-code paradigm which things like terrafrom and cloud formation are.
(1) We have a business requirement where we run a scheduled batch at specific times in the day. These are very computationally intensive. To save costs, we run these on an EC2 which is brought up at that time, then torn down when the batch is finished. However, this seems to require a temporary change to the terraform/CF files, then a change back. Is there a more native way of doing this?
(2) We dynamically store and allow to be edited by clients their firewalling rules on their load balancer (ALB). This information cannot be stored in the terraform/CF files since it can be changed by clients on demand.
Is there a way of properly doing these things in CF/Terraform?
(1) If you have to use EC2, you could create a Lambda that would start your EC2 instances. Then, create a CloudWatch Event that triggers the Lambda at your specified date / time. For more details you can see https://aws.amazon.com/premiumsupport/knowledge-center/start-stop-lambda-cloudwatch/. Once the job is done, have your EC2 shut itself down using the awssdk or awscli.
Alternatively, you could use AWS Lambda to run your batch job. You only get charged when the Lambda runs. Likewise, create a CloudWatch Event rule that schedules the Lambda.
(2) You could store the firewall rules in your own DB and modify the actual ALB SG rules using the awssdk. I don't think it's a good idea to store these things in Terraform/CF. IMHO Terraform/CF are great for declaring infrastructure but won't be a good solution for resources that are dynamically changing, especially by third parties like your clients.
I am looking to setup an event driven architecture to process messages from SQS and load into AWS S3. The events will be low volume and I was looking at either using Databricks or AWS lambda to process these messages as these are the 2 tools we already have procured.
I wanted to understand which one would be best to use as I'm struggling to differentiate them for this task as the throughput is only up to 1000 messages per day and unlikely to go higher at the moment so both are capable.
I just wanted to see what other people would consider and see as the differentiators between the two of these products so I can make sure this is future proofed as best I can?
We have used lambda more where I work and it may help to keep it consistent as we have more AWS skills in house but we are looking to build out databricks capability and I do personally find it easier to use.
If it was big data then I would have made the decision easier.
Thanks
AWS Lambda seems to be a much better choice in this case. Following are some benefits you will get with Lambda as compared to DataBricks.
Pros:
Free of cost: AWS Lambda is free for 1 Million requests per month and 400,000 GB-seconds of compute time per month, which means your request rate of 1000/day will easily be covered under this. More details here.
Very simple setup: The Lambda function implementation will be very straight-forward. Connect the SQS Queue with your Lambda function using the AWS Console or AWS cli. More details here. The Lambda function code will just be a couple of lines. It receives the message from SQS queue and writes to S3.
Logging and monitoring: You won't need any separate setup to track the performance metrics - How many messages were processed by Lambda, how many were successful, how much time it took. All these metrics are automatically generated by AWS CloudWatch. You also get an in-built retry mechanism, just specify the retry policy and AWS Lambda will take care of the rest.
Cons:
One drawback of this approach would be that each invocation of Lambda will write to a separate file in S3 because S3 doesn't provide APIs to append to existing files. So you will get 1000 files in S3 per day. Maybe you are fine with this (depends on what you want to do with this data in S3). If not, you will either need a separate job to join all files periodically or do a download of existing file from S3, append to it and upload back, which makes your Lambda a bit more complex.
DataBricks on the other hand, is built for different kind of use cases - Loading large datasets from Amazon S3 and performing analytics, SQL-like queries, builing ML models etc. It won't be suitable for this use case.
I didn't find any posts related to this topic. It seems natural to use Lambda as a getstream backend, but I'm not sure if it heavily depends on persistent connections or other architectural choices that would rule it out. Is it a sensible approach? Has anyone made it work? Any advice?
While you can build an entire website only in Lambda, you have to consider the followings:
Lambda behind API Gateway has a timeout limit of 30 seconds and a Payload size limit (both received and sended) of 6MB. While for most of the cases this is fine, if you have some really big operations or you need to send some really big datas (like a high resolution image), you can't do it with this approach, but you need to think about something else (for instance you can send an SNS to another Lambda function with higher timeout that can do all this asynchronously and then send the result to the client when it's done, supposing the client is capable of receiving events)
Lambda has cold starts, which in terms slow down your APIs when a client calls them for the first time in a while. The cold start time depends on the language you are doing your Lambdas, so you might consider this too. If you are using C# or Java for your Lambdas, than this is probably not the best choice. From this point of view, Node.JS and Python seems to be the best choices, with Golang rising. You can find more about it here. And by the way, you can now specify a Provisioned Throughput for your Lambda, which aims to fix the cold start issue, but I haven't used it yet so I can't tell if there is any difference (but I'm sure there is)
If done correctly you'll end up managing hundreds of Lambda functions, while with a standard Docker Container under ECS you'll manage few APIs with multiple endpoints. This point should not be underestimated, as on one side it will make changes easier in the future, since lambda will be small and you'll easily find the bug and fix it, but on the other side you have to move across these functions, which if you don't know exactly which lambda is responsible of what can be a long process
Lambda can't handle sessions as far as I know. Because after some time the Lambda container gets dropped, you can't store any session inside the Lambda itself. You'll always need a structure to store the session so it can be shared across multiple Lambda invocations, such as some records in a DynamoDB table or something else, but this mean that you have to write the code for this, while in a classic API (like a .NET Core one) all of this is handled by the language itself and you only need to store or retrieve items from the session (most of the times)
So... yeah! A backed written entirely in Lambda is possible. The company I work in does it and I must say is a lot better, both in terms of speed and development time. But those benefits comes later, since you need to face all of the reasons I listed above before, and is not as easy as it could seem
Yes, you can use AWS Lambda as backend and integrate with Stream API there.
Building an entire application on Lambda directly is going to be very complex and requires writing lot of boiler plate code just to enforce some basic organization and structure to your project.
My recommendation is use a serverless framework to do this that takes care of keeping your application well organized and to deploy new versions (and environments).
Serverless is a good option for that: https://serverless.com/framework/docs/providers/aws/guide/intro/