Is there a way to mock s3 bucket, sqs queues, and sns topics in the Ruby aws-sdk (similar to Moto in the Python sdk) to where assertions can be made on those uploaded objects and sent messages after the application code is run?
I know I am able to stub the response of the aws sdk so that no real requests are sent using the following code:
Aws.config[:stub_responses] = true
But that doesn't get me the ability to make assertions on uploaded s3 objects and sent sqs/sns messages.
All of the unit tests in the following folder use stubs for Amazon S3 for Ruby. Hope this helps you: https://github.com/awsdocs/aws-doc-sdk-examples/blob/master/ruby/example_code/s3/tests
You can extend this pattern for Amazon SQS and Amazon SNS as well. I hope to add some myself by the end of this year...
Related
I have created a alexa smart home function and want to run it asynchronously so plan to use amazon sqs (Simple que service) functionality. I connected amazon sqs trigger output to lambda function and successfully able to send message from sqs to lambda. Now need to connect the alexa to sqs input. When i try to use sqs arn in alexa developer console it does not support it. Is there any way to solve this or will alexa support only lambda function for invocation.
The alexa skill is for smart home service to control switches (Turn on/off), so when try to control the multiple switches because of synchronous nature execution of lambda it turns on switches one after the other. I need to control them at single shot so need asynchronous execution for lambda where requests need to execute without waiting for the response.
Thanks in advance for answers.
It will not work as SQS works asynchronus and just reply that message was put there. But Alexa needs a valid JSON response with speech tag and so on immediately and SQS is not able to fulfill this.
What you could do:
Alexa -> Lambda (new) -> SQS - Lambda
In your new created lambda you could give a valid reply to Alexa and put a message in SQS.
AWS Lambda can work asynchronously. You can have a bunch of back-end processes all working as they need to, triggering various Lambdas as needed.
But the exchange with Alexa opens a session to your backend, sends its request, and the full response is expected to end that session. That response may have directives to download other content to incorporate into the response, like a sound file or lazy loading a list in APL. But it is expecting a full response.
If you go through the basic Cake Time tutorial for building Alexa skills, they actually use async-await for some APIs because that response has to be complete before it's sent.
There are some async APIs like reminders and proactive events, but they're NOT conversational. They're unique one-way messages.
The real questions are why do you feel you need to do it this way and what are you optimizing for by queuing?
My use case is the following. I have 5 lambdas. They need to talk to each other. I've heard that it can be done with SNS but also SNS and SQS. What is the difference, why not call lambdas only from one another directly?
It's possible to design durable and scalable applications using SNS-SQS AWS pattern. You can do this by having an SNS topic to which lambda A posts then the SNS triggers directly SQS which is a queue. In that way if you have high volume messages they will be processed sequentially.
Take care that the SNS and SQS can trigger more than once.
For more info check the article here:
https://aws.amazon.com/blogs/compute/designing-durable-serverless-apps-with-dlqs-for-amazon-sns-amazon-sqs-aws-lambda/
You can also use AWS Step Function which is a serverless function orchestrator that makes it easy to sequence AWS Lambda functions and multiple AWS services.
You can check out getting started guide here - https://docs.aws.amazon.com/step-functions/latest/dg/getting-started.html
I have an architecture which looks like as follows:-
Multiple SNS -> (AWS Lambda or SQS with Poller)??? -> Dynamo Db
So, basically multiple SNS have subscribed to AWS Lambda or SQS with Poller and that thing pushes data to Dynamo Db.
But this ? thing do lot of transformation of message in between. So, now for such case, I can either use AWS Lambda or SQS with Poller. With AWS Lambda, I can do transformation in Lambda function and with SQS with Poller, I can do transformation in Poller. With AWS Lambda, I see one problem that code would become quite large as transformation is quite complex(has lot of rules), so I am thinking to use SQS. But before finalising on SQS, I wanted to know of the drawbacks of SQS which AWS Lambda removes?
Please help. Let me know if you need further information.
Your question does not contain much detail, so I shall attempt to interpret your needs.
Option 1: SQS Polling
Information is sent to an Amazon SNS topic
An SQS queue is subscribed to the SNS topic
An application running on Amazon EC2 instance(s) regularly poll the SQS queue to ask for a message
If a message is available, the data in the message is transformed and saved to an Amazon DynamoDB table
This approach is good if the transformation takes a long time to process. The number of EC2 instances can be scaled based upon the amount of work in the queue. Multiple messages can be received at the same time. It is a traditional message-based approach.
Option 2: Using Lambda
Information is sent to an Amazon SNS topic
An AWS Lambda function is subscribed to the SNS topic
A Lambda function is invoked when a message is sent to the SNS topic
The Lambda function transforms the data in the message and saves it to an Amazon DynamoDB table
AWS Lambda functions are limited to five minutes of execution time, so this approach will only work if the transformation process can be completed within that timeframe.
No servers are required because Lambda will automatically run multiple functions in parallel. When no work is to be performed, no Lambda functions execute and there is no compute charge.
Between the two options, using AWS Lambda is much more efficient and scalable but it might vary depending upon your specific workload.
We can now use SQS messages to trigger AWS Lambda Functions.
28 JUN 2018: AWS Lambda Adds Amazon Simple Queue Service to Supported
Event Sources
Moreover, no longer required to run a message polling service or create an SQS to SNS mapping.
AWS Serverless Model supports a new event source as following:
Type: SQS
PropertiesProperties:
QueueQueue: arn:aws:sqs:us-west-2:213455678901:test-queue arn:aws:sqs:us-west-2:123791293
BatchSize: 10
AWS Console also support:
Further details:
https://aws.amazon.com/blogs/aws/aws-lambda-adds-amazon-simple-queue-service-to-supported-event-sources/
https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html
What is the most efficient way to receive messages from a Amazon SQS queue?
I've been using the Peddler Gem to create, register and subscribe to an Amazon SQS queue that captures Amazon Marketplace order changes. All good there, the SQS queue is receiving the messages fine. The next step I'm a bit fuzzy on and need some help before I go down a rabbit hole.
It seems like the SQS queue should just be like a webhook that I can subscribe to, too receive notices. But I'm not seeing that option anywhere.
But then it looks like I can use the Shoryuken Gem or maybe Amazon's own AWS SDK for Ruby to create workers to poll the queue in order to get notified of new messages.
Is the Shoryuken gem the most efficient way to pull messages from SQS? Or is there a better way?
IMO Shoryuken is currently the most efficient way for polling SQS messages in Ruby.
You can go ahead and use only the aws-sdk, that would work - with certain limitations. If you go on that path, you will ended implementing a lot of stuff around the aws-sdk, which Shoryuken already does. With the sdk you can receive messages in a loop, call a Ruby class to consume them etc. Shoryuken is a process for polling messages, which uses multithread for performance wise. Besides that, a single process can receive messages from multiple queues.
It seems like the SQS queue should just be like a webhook that I can subscribe to, too receive notices. But I'm not seeing that option anywhere.
That is not SQS, the service that's like that is AWS SNS. If Amazon Marketplace can also integrate with SNS, you can implement a pub/sub calling webhooks.
PS: Shoryuken author here :)
I am working in the cool but very constrained environment of Tropo (cloud telephony) Ruby scripting. The entire app is a single JRuby file. No gems, no requires.
I need to send simple messages to a single SQS queue. I don't need to do any other SQS operations. Before I start pulling code out of existing gems to do this, I wanted to see if anyone has standalone code for sending SQS messages or code that does the HTTP request signing that SQS requires.
We ended up going with the following super-simple code to post a message to an SQS queue that already exists. No gems required.
super_simple_sqs.rb