I have turned on database activity events which I think is some kind of log file on AWS Aurora. They are currently being passed through AWS kinesis into s3 via AWS Firehose. The log in s3 looks like this:
{"type":"DatabaseActivityMonitoringRecords","version":"1.0","databaseActivityEvents":"AYADeOC+7S/mFpoYLr17gZCXuq8AXwABABVhd3MtY3J5cHRvLXB1YmxpYy1rZXkAREFvbjhIZ01uQTVpVHlyS0l3NnVIOS9xdXF3OWEza0xZV0c2QXYzQmtWUFI2alpIK2hsczNwalAyTTIzYnpPS2RXUT09AAEAAkJDABtEYXRhS2V5AAAAgAAAAAwzb2YKNe4h6b2CpykAMLzY7gDftUKUr3QxmxSzylw9qCRxnGW9Fn1qL4uKnbDV/PE44WyOQbXKGXv9s8BxEwIAAAAADAAAEAAAAAAAAAAAAAAAAAC+gU55u4hvWxW1RG/FNNSJ/////wAAAAEAAAAAAAAAAAAAAAEAAACtbmBmDwZw2/1rKiwA4Nyl7cm19/RcHhCpMMwbOFFkZHKL/bvsohf5T+yM9vNxCgAi2qTUIEe17VA5bJ0eCcNAA9mb6Ys+PR1w7QhKrQsHHTBC2dhJ4ELwpXamGRmPLga5Dml2rOveA59YefcJ4PhrqztZXfrS8fBYJ3HgBWHY9nPh1jdyinjQAl61hQrz2LPII85zlqAWTNeL2pXwaRdtGdYeIXXoh4VsoV3Q18Hj/uOQzTIbT8EJvwnk0gj8AGcwZQIxAJNuoCJhHPUfbkk0fHF6HYz1STIc4HX2HOl0qSIHqwpgtQK6BMa3YlPI9hNwhB8x+AIwWDY0bMjuLRGQgjjBv5z1xPpZQ+pMZ4K6m9JaNBFVKxZTvqDL1z7lrV0rlbZThad+","key":"AQIDAHhQgnMAiP8TEQ3/r+nxwePP2VOcLmMGvmFXX8om3hCCugE7IUxSH/eJBEKvnkYoNIqFAAAAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMQIX97gE5ioBR1+nnAgEQgDuDX2B2T7nOxjKDyL31+wHJb0pwkCeaU7CwA6BwIkiT7FmhMB71XgvCVrY9C9ABUtc1e5J7QIfsVB214w=="}
I think a KMS key is being used to encrypt that log file. How do I decrypt it? Is there working sample code somewhere? Also, more importantly, the Aurora database I'm using is a test database with no activity (no inserts, selects, updates). Why are there so many logs? Why are there so many databaseActivityEvents. They seem to be getting written to s3 every minute of the day.
Yes it uses RDS Activity stream KMS key (ActivityStreamKmsKeyId) for encrypting the log event and also base64 encoding. You will have to make use of AWS cryptographic SDKs to decrypt the key and the log event.
For reference see below their the sample java and python versions:
Processing a Database Activity Stream using the AWS SDK
In your firehose pipeline you can add transformation with Lambda step and do this decryption in your lambda function.
Why there are so many events in idle postgres RDS cluster? They are heartbeat events.
When you decrypt and take a look at the actual activity event json, it has type field which can be either be record or heartbeat. Events with type record are the user activity generated ones.
Related
I want to implement the following once a file is uploaded to aS3 Bucket
Download the file to a windows server
Run a 3rd party exe to process the file and generate an output file on a Windows Server
What is the best approach to implement this using .Net Core?
Solution 1:
Create a Lambda function to Trigger an API
API will download the file and process
Solution 2:
Create an executable to download the file from s3 bucket
Create a lambda function trigger an executable
Solution 3:
Create a service to check and download files from s3 bucket
The downloaded file will be processed by the service
Solution 4:
Use AWS Lambda to push the file to SQS
Create an application to monitor SQS.
Please let me know the best solution to implement this. Sorry for asking this non-technical question.
The correct architecture approach would be:
Create a trigger on the Amazon S3 bucket that sends a message to an Amazon SQS queue when the object is created
A Windows server is continually polling the Amazon SQS queue waiting for a message to appear
When a message appears, use the information in the message to download the object from S3 and process the file
Upload the result to Amazon S3 and optionally send an SQS message to signal completion (depending on what you wish to do after a file is processed)
This architecture is capable of scaling to large volumes and allows files to be processed in parallel and even across multiple servers. If a processing task fails and does not signal completion, then Amazon SQS will make the message visible again for processing.
i've created all(as i think) as described in this article
building-snowpipe-on-azure-blob
snowflake
azure blob storage
snowpipe
but pipe works only after run "alter pipe myPipe refresh"
data loading correctly, but auto_ingest doesn't work.
please give an advice how to find an issue.
The refreshing pipe command fetches files directly from the stage while the auto-ingest option doesn't take the same route and consume messages from the Azure queue storage. Therefore, even if the Azure blob storage container is correct, the message could be delivered to the queue but not to Snowflake.
Solution Details: https://community.snowflake.com/s/article/Ingesting-new-files-with-Snowpipe-for-Azure-fails-while-refreshing-the-pipe-works
Good day,
Is there any script or any aws cli command to know which IAM user created what resource in AWS. so that we just enter the IAM user name and it shows all the resources created by that particular IAM user.
thanks in advance.
The service that you're looking for is CloudTrail.
By default, it retains 90 days worth of events for the current account and region, and you can access it from either the Console or CLI. You can also configure it to write events to S3, where they're be preserved as long as you want to pay for the storage (this also lets you capture events across all regions, and for every account in an orgnanization).
CloudTrail events can be challenging to search. If you're just looking for events by a specific user, and know that user's access key (here I'm using my access key stored in an environment variable) you can use a query like this:
aws cloudtrail lookup-events --lookup-attributes "AttributeKey=AccessKeyId,AttributeValue=$AWS_ACCESS_KEY_ID" --query 'Events[].[EventTime,EventName,Username,EventId]' --output table
Or, by username:
aws cloudtrail lookup-events --lookup-attributes "AttributeKey=Username,AttributeValue=parsifal" --query 'Events[].[EventTime,EventName,Username,EventId]' --output table
You can then use grep to find the event(s) that interest you, and dig into the details of a specific event with:
aws cloudtrail lookup-events --lookup-attributes "AttributeKey=EventId,AttributeValue=8c5a5d8a-9999-9999-9999-a8e4b5213c3d"
I am beginner in AWS (from Microsoft domain). I want to run a SQL query against Redshift tables to view duplicates in table on daily basis and send results out in email to a Prod Support group.
Please advise, what is right way to proceed on this.
Recommend doing this with either AWS Lambda or AWS Batch. Use one of these services to issue a short query on a schedule and send the results if required.
Lambda is ideally for simple tasks that complete quickly. https://aws.amazon.com/lambda/ Note that Lambda charges by duration has very tight limits on how long a step can run. A basic skeleton for connecting to Redshift in Lambda is provided in this S.O. answer: Using psycopg2 with Lambda to Update Redshift (Python)
Batch is useful for more complex or long running tasks that need to complete in a sequence. https://aws.amazon.com/batch/
There is no in-built capability with Amazon Redshift to do this for you (eg no stored procedures).
The right way is to write a program that queries Redshift and then sends an email.
I see that you tagged your question with aws-lambda. I'd say that a Lambda function would not be suitable here because it can only run for a maximum of 5 minutes and that might be longer than you need your analysis to run.
Instead, you could run the program from an Amazon EC2 instance, or from any computer connected to the Internet.
I am trying to filter the data on specific EC2 instances with a tag called application name to ex:ABCD and send the metrics to Telegraf. Right I have the telegraf taking all the data under the namespace AWS/EC2. Does anyone have any idea on how to filter the data and send it to telegraf? Does it have to do with any changes to be done in telegraf config file?
Generally in all use cases you want clean raw data saved to the db and use indexing and queries to filter the data you are requesting from the database.