Are there any recommendations on how we can use AWS Kinesis Firehose where data/events are published from different applications from customer’s environment (multiple customers or tenants)?
Our application continuously generate events which we want to ingest into cloud solution for processing. We are evaluating if AWS Kinesis Firehose delivery stream as entry point. Since it is a multi tenant solution we are trying to find case studies to understand how data can be partitioned if we use Firehose for data ingestion.
Thanks,
Vishnu.
Related
Is there any way to use MassTransit with AmazonMQ or SNS+SQS and AWS Lambda?
I am new to serverless, AWS Lambda and AWS in general. I am implementing an event-driven architecture that is going to need some event bus for communication between microservices/lambda.
I have an app that needs to publish events, and I want to have many AWS Lambda that react to those events, do their work and go to sleep. I have had a look at AWS EventBridge but the latency (300ms to 600ms) is way too high for real time needs and it's not really designed for a large amount of subscribers. SNS + SQS is another option because I know MassTransit also supports it. Notice that RabbitMQ broker does not seem yet compatible with AWS Lambda as Chris mentioned in the comment below pointing to https://docs.aws.amazon.com/lambda/latest/dg/with-mq.html.
I've had good results with MassTransit in the past and I like the way it abstracts the transport layer away, but I can't find any opinion/sample on how to use it with AWS Lambda. Is it even possible? Any sample/link or reasons why it's not possible would be appreciated.
UPDATE: RabbitMQ cannot be used as broker for AmazonMQ if I want to use AWS Lambda. Question edited and it remains but applied to MassTransit + AmazonMQ (ActiveMQ broker) or SQS/SNS + AWS Lambda
I am using Elasticcloud (hosted elasticsearch) to index my app data. Now I want to start streaming logs from my AWS lambda functions to my Elasticcloud account. I have googled and I can see that there are couple of ways to do this:
Functionbeat
Cloudwatch-> Elasticsearch subscription filter
Cloudwatch-> Lambda subscription filter
My questions are
which is the most cost efficient and performant way to stream logs from AWS cloudwatch to elasticcloud
For functionbeat is it necessary to first send logs to a S3 bucket? (I am referring to this https://www.elastic.co/guide/en/beats/functionbeat/current/configuration-functionbeat-options.html)
First question:
Since Functionbeat is deployed to Lambda in case of AWS, no.1 and no.3 cost the same. No.1 is faster to deploy because you need to create Lambda by yourself in no.3.
As for performance, of course it depends on the implementation, I guess there is no big difference between two methods unless millisecond latency has impact to you.
If you are using Elastic Cloud you can't use no.2, which works with Amazon Elasticsearch Service. These two are completely different services. (see this page, I know it's a bit confusing!)
Second question:
No, you don't have to. Functionbeat directly gets logs from CloudWatch.
S3 bucket is used to store Function beat module itself before being deployed to Lambda.
I am trying the stream cloudwatch logs to elastic search using elastic search subscription filter.
I want to automate with terraform, but didn't find if terraform supports this resource type.
Please let me know if it is feasible.
In AWS REST API nor AWS CLI there is no such thing as a subscription to ElasticSearch. Only the following subscriptions are supported:
An Amazon Kinesis stream belonging to the same account as the subscription filter, for same-account delivery.
A logical destination that belongs to a different account, for cross-account delivery.
An Amazon Kinesis Firehose delivery stream that belongs to the same account as the subscription filter, for same-account delivery.
An AWS Lambda function that belongs to the same account as the subscription filter, for same-account delivery.
What you see in AWS Console, is console only shortcut for that. Basically, when you create a "subscription" to ES, console will just provision a lambda function and create subscription it. The lambda will get the log events and inject into the ES domain.
Therefor, to inject logs into ES in terraform, you have to construct such "subscription" yourself. This is done through actual subscription to a lambda function. To simplify the development, you can take the lambda function that AWS creates and use that, instead of developing your own code for injecting logs to ES.
I am developing an application on AWS and it has regulatory needs to retain the data in a certain geography. I know that with RDS we can achieve the same. But if we use Aurora Server-less, can we define that my data does not leave the geography in which the Amazon data centre is located.
I have gone through the documentation of AWS. It seems to suggest that the data is geographically distributed to improve latency. But this would mean I do not have control over where the data is. My need is the opposite of it, where I want to restrict it to a certain geo location.
Aurora Serverless clusters are similar to Provisioned clusters - they are tied to a region. Provisioned clusters have new features like Global databases which makes the data available in other geographies, but Aurora Serverless does not support those features. Your data in, say, us-east-1 is not leaving that region.
For bandwidth of 400GB per month, what EC2 instance should I use if I want to create a video streaming infrastructure to different regions?
You won't get any specific answers on questions like this. It is totally dependent on your application.
If you stored the videos in Amazon S3 and streamed videos through Amazon CloudFront, then Amazon EC2 would purely be handling user interactions and web pages, without having to serve video content at all.
For any application, the only way to know how much compute is required is to test the application under many different workloads and instance types and measure the performance. Alternatively, an application can be designed to use serverless microservices using AWS Lambda, which can automatically scale without using EC2 instances.