Example use case
Send the user a notification 2 hours after signup.
Options considered
setTimeout(() => { /* send notification */ }, 2*60*60*1000); is not an option in serverless environments since the function terminates after execution (so it has to be stateless).
CloudWatch events can schedule lambda invocations using cron expressions - but this was designed for repetitive invocations (there's a limit of 100 rules/region).
I have not seen scheduling options in AWS SNS/SQS or GCP Pub/Sub. Are there alternatives with scheduling?
I want to avoid (if possible) setting up a dedicated message broker (overkill) or stateful/non-serverless instance - is there a serverless way to do this?
I can queue the events in a database and invoke a lambda function every minute to poll the database for events to execute in that minute... is there a more elegant solution?
Use AWS Step functions, they are like serverless functions that don't have the 15 minute limit like AWS Lambda does. You can design a workflow in AWS step that integrates with API Gateway, Lambda and SNS to send email and text notifications as follows:
Create a REST API via API gateway that will invoke a Lambda function passing in for example, the destination address (email, phone #) of the SNS notification, when it should be sent, notification method (e.g. email, text, etc.).
The Lambda function on invocation will invoke the Step function passing in the data (Lambda is needed because API Gateway currently can't invoke Step functions directly).
The Step function is basically a workflow, you can define states for waiting (like waiting for the specified time to send the notification e.g. 30 seconds), and states for invoking other Lambda functions that can use SNS to send out an email and/or text notifications.
A rudimentary example is provided by AWS w/ their Task Timer example.
Things are coming on GCP for doing this, but not very soon. Thereby, today, the solution is to poll a database.
You can to that with Datastore/firestore with the execution datetime indexed (to prevent to read all the documents each minute). But be careful of traffic spike, you could create hotspot.
You can use Cloud Scheduler on Google Cloud Platform. As is is stated in the official documentation :
Cloud Scheduler is a fully managed enterprise-grade cron job scheduler. It allows you to schedule virtually any job, including batch, big data jobs, cloud infrastructure operations, and more. You can automate everything, including retries in case of failure to reduce manual toil and intervention. Cloud Scheduler even acts as a single pane of glass, allowing you to manage all your automation tasks from one place.
Here you can check a quickstart for using it with Pub/Sub and Cloud Functions.
Related
I have a lambda that is triggered whenever an event is dropped in the eventbus to which my lambda is connected and is triggered automatically. How can I performance test it to test how it performs of 500 events are dropped at a time?
Also I know aws has some inbuilt metrics like lambda execution time, xray tracing etc. Can anyone let me know how to use them for my use case?
If by "eventbus" you mean AWS EventBus which is a part of Amazon EventBrigde my expectation is that the easiest would be using PutEvents API endpoint, you can come up with a JSON payload having 500 events or make 500 separate calls with 1 event or any combination you can think of.
Be aware that the AWS API requests need to be signed to the load testing tool you choose must have the possibility to calculate this signature. A guide for Apache JMeter: How to Handle Dynamic AWS SigV4 in JMeter
With regards to metrics - check out AWS CloudWatch
Looking for some help on an application design. I am using spring framework and hosting application in AWS.
I am working on an enterprise Java Web application that is suppose to handle events when their trigger time is reached. For example, consumers can set an event to begin on 12/20/22 at 07:35 AM, and system is suppose to send a notification when that time is reached.
I can store these events in a database along with their trigger time and setup a Spring scheduler (#Scheduler) to run every minute and process events whose trigger time is reached. My only concern with this approach is, there could be hundreds/thousands of event to trigger at any minute, and it cannot be processed within one minute.
Is there any alternate way to design this? I don't know if Spring offers a feature where I could create these Event, and Frameworks trigger these events when trigger time is reached. In that way, I can stay away from managing Scheduling and Triggering part.
I am using AWS to host this applications, so another option I'm thinking towards is creating an AWS lambda for every such Event, and let AWS manage the triggering part. In that way, I can stay away from managing the triggers.
Let me know your views? Or If you came across similar problems and how you resolved that?
You can consider using spring-cloud-dataflow to manage this as tasks and streams.
You create a custom batch application that will use #Scheduled to check the your database when events are dure and then send events to a stream. You can use Spring Integration APIs to interact with RabbitMQ or Kafka topics.
The event should contain enough information needed to process the event.
You then have a stream application that produces the content and send via email or pass it on to a separate stream app that sends the email.
https://dataflow.spring.io/docs/stream-developer-guides/programming-models/
The flow will look something like:
:mail_events | message-processor | message-sender
You will configure property for mail_events to match the topic created and configured for you mail-event-batch application.
You can use Spring Cloud Data Flow to manage the mail-event-batch application as well.
You can scale each application https://dataflow.spring.io/docs/recipes/scaling/
I've been thinking about how to design a system that supports user created scheduled alerts. My problem is once the alerts are created and inserted into a database, I don't know what the best way to go about scheduling those alerts. Polling the database to see which alerts need to go out next doesn't seem entirely right to me.
What are some ways this could be handled on a scale where say a million users could create their own custom alerts like change baby diaper at 3pm everyday?
This problem is very suitable for cloud platforms. For example, you could use GCP Cloud Scheduler to invoke a cloud function when the alert is supposed to be sent out. The cloud function then calls some API to alert the user.
If cloud platforms are not an option, you could have your application spawn a new thread when an alert is created, and sleep that thread for a certain duration. When it wakes up, it sends the alert. Less elegant and less scalable than the first solution, but it would still work.
I have created a alexa smart home function and want to run it asynchronously so plan to use amazon sqs (Simple que service) functionality. I connected amazon sqs trigger output to lambda function and successfully able to send message from sqs to lambda. Now need to connect the alexa to sqs input. When i try to use sqs arn in alexa developer console it does not support it. Is there any way to solve this or will alexa support only lambda function for invocation.
The alexa skill is for smart home service to control switches (Turn on/off), so when try to control the multiple switches because of synchronous nature execution of lambda it turns on switches one after the other. I need to control them at single shot so need asynchronous execution for lambda where requests need to execute without waiting for the response.
Thanks in advance for answers.
It will not work as SQS works asynchronus and just reply that message was put there. But Alexa needs a valid JSON response with speech tag and so on immediately and SQS is not able to fulfill this.
What you could do:
Alexa -> Lambda (new) -> SQS - Lambda
In your new created lambda you could give a valid reply to Alexa and put a message in SQS.
AWS Lambda can work asynchronously. You can have a bunch of back-end processes all working as they need to, triggering various Lambdas as needed.
But the exchange with Alexa opens a session to your backend, sends its request, and the full response is expected to end that session. That response may have directives to download other content to incorporate into the response, like a sound file or lazy loading a list in APL. But it is expecting a full response.
If you go through the basic Cake Time tutorial for building Alexa skills, they actually use async-await for some APIs because that response has to be complete before it's sent.
There are some async APIs like reminders and proactive events, but they're NOT conversational. They're unique one-way messages.
The real questions are why do you feel you need to do it this way and what are you optimizing for by queuing?
I'm beginner of greengrass core application, and finished the demo setup following greengrass developper guide. but i'm still confusing about how lambda functio works.the bellow is the quesitons I want to ask for help.
I want to run a lambda function in my raspberry pi 3 as greengrass core, which can recieve multiple IoT devices' MQTT messages and do some process according to task tpye(i.e various signal filtering or house-hold machine learning algorithms). After proceesing, I need send the information using MQTT to my own server(not AWS IoT cloud) for higher level processing with some topics.
my questions are as follows( I want to use JAVA language):
1 To receive multiple aws iot devices connected to the GGC, should I need to set up a AWSIoTMQTTClient in aws-iot-device-sdk-java?
I also find in aws_greengrass_core_sdk_java, there is “IotDataClient” class,what's it for?and what's the different with AWSIoTMQTTClient. here is really very confusing, even with sdk document description.
2 In GGC, when I deployed my lambda function, will it has an internal MQTT broker to receive messages for AWSIoTMQTTClient ?
3 for lambda functions, after creation and deployment on GGC, will it start to work. I saw there is method to invoke another lambda funciton from a lambda funciton. I don't understand the mechanism how lambda works.
4 Can i have multiple lambda functions for different uage,for instance, one is only to receive MQTT messages, another is to process the received info, other one is to send the processed info out to my own MQTT server? if permitted, how to make the work together to perform all the tasks.
5 I saw there is event input to lambda interface, how can I call a lambda only when some specific topic arriverd to AWSIoTMQTTClient defined in the lambda function?
6 the below is JAVA lambda interface template:
outputType handler-name(inputType input, Context context) {
...
}
i think it should permit user to define input data type as he need. but the quesiton is if I define inputtype is string. how to the lambda handler to receive the string. the development guidence have no clear description.
7 finally, can you please share some demo codes for the above questions?
Thanks for you attention and kind help in advance.
your help is highly expected
AWSIoTMQTTClient from the device SDK is not for Greengrass Lambda functions. Instead use IotDataClient from the Greengrass Java SDK, create a publish request, and then invoke the publish method. There is an example of that here - https://github.com/aws-samples/aws-greengrass-lambda-functions/blob/master/foundation/CDDBaselineJava/src/main/java/com/timmattison/greengrass/cdd/communication/GreengrassCommunication.java
AWSIoTMQTTClient is for devices/applications that run outside of Greengrass.
If you'd like to see some example Greengrass Lambda function code in Java check out at least this skeleton example - https://github.com/aws-samples/aws-greengrass-lambda-functions/tree/master/functions/CDDSkeletonJava. Note this function and other other ones in the repo depend on a framework called CDD (Cloud Device Driver). It is shared in the same repo and does most of the heavy lifting (messaging, startup, etc). That combined with the Greengrass provisioner - https://github.com/awslabs/aws-greengrass-provisioner - gives you a quick way to develop Java functions on Greengrass. Let me know if you try it out.
If you want to see the internals of CDD the root of it is here - https://github.com/aws-samples/aws-greengrass-lambda-functions/tree/master/foundation/CDDBaselineJava
As far as Lambda functions and how they run briefly I'll say that they can run on-demand (when they receive a message) or they can run "pinned" (forever). Pinned functions can receive messages too. Pinned functions are good when you need to track some kind of state. On-demand functions are more efficient for stateless data processing.