We are using Hashicorp vault to store the Database secrets. Our application is based on AWS Lambda microservices.
Whenever there is burst in traffic and all the Lambda services are trying to get database secrets from vault simultaneously, vault is taking huge amount of time(~20 secs) to return the secret.
How can I fix this issue ? Doesn't Vault have concurrency support ?
Related
Recently I've started to use Secrets Manager to read credentials from Lambda, and I noticed that reading a secret from SM takes several seconds. Introducing VPC interface endpoint as described here: https://docs.aws.amazon.com/secretsmanager/latest/userguide/vpc-endpoint-overview.html#vpc-endpoint didn't help, I see the same response time.
In CloudTrail I see that created VPC endpoint is used to call Secrets Manager.
Did anyone have similar issue?
While developing microservice (for e.g. spring boot service), we are storing database username and password in properties file. Microservice would connect to database using these database credentials.
Is there any other best way to store database username and password ?
What are best techniques to store database user and password securely in encrypted format and how to decrypt before connecting to database schema?
We are using Chef Cookbook (DevOps), Chef Client for deployment of microservices in Amazon Web Services(AWS)
One possible solution is to have a Config Server to which your applications connect and get their configurations at startup time. This Config Server might decrypt the encrypted sensitive data.
A simple example: https://spring.io/guides/gs/centralized-configuration/
What are permissions that my lambda function need to be able to retrieve secrets from AWS Secrets Manager and change it also ?
You need the secretsmanager:GetSecretValue policy to retrieve secrets and the secretsmanager:UpdateSecret policy to update secrets.
Note that if you are using a customer-managed AWS KMS key for encryption you will also need some KMS permissions:
kms:Decrypt for retrieving the secret.
kms:Decrypt and kms:GenerateDataKey for updating the secret.
https://docs.aws.amazon.com/cli/latest/reference/secretsmanager/get-secret-value.html
https://docs.aws.amazon.com/cli/latest/reference/secretsmanager/update-secret.html
If you are using the Lambda functions provided by AWS, then (as described in the docs) you will need: DescribeSecret, GetSecretValue, PutSecretValue, UpdateSecretVersionStage and GetRandomPassword. If you are using a Custom KMS Key (CMK) you will also need Decrypt and GenerateDataKey permissions for that CMK (both in the Lambda policy and in the KMS key policy).
If you are seeing Task timed out errors, it is likely your Lambda can not access either the secrets manager endpoint (try using a VPC endpoint), or the Lambda can not connect to the DB (check security group settings).
We want to build a web application and deploy it on AWS.
EC2: Laravel
RDS: MySQL
I will use Laravel's encrypter to encrypt the data of database. Even RDS got hacked, the data have encrypted. Hacker can't know the contents. But if EC2 got hacked, hacker can get the database credential and the encryption key on the source code and decrypt the encrypted data from database.
My Boss (maybe client) think that it is not enough because of the database contains sensitive informations of users. He want to prevent the leakage of the original data of the database even if the web server (EC2) got hacked. Is it possible?
If not, I think we should focus on make the web server more difficult to be hacked:
Set Security Group to limit ssh access by IP address.
Or any other measures?
Here are a few safety measures you can do to reduce your blast radius.
Move your credentials for the RDS database so they are not directly on the instance, use a credential store such as:
AWS Secrets Manager
HashiCorp Vault
Rotate your database credentials frequently, and use IAM roles for your EC2 applications and not IAM users.
Keep your EC2 and RDS within private subnets, add an ELB in front of the EC2 so that public traffic can only access this device only.
Configure security groups to scoped to only what they need, limit inbound access to your AWS VPC to a VPN or direct connect connection.
Restrict access to who can do what in your AWS account, if a user does not need to perform certain actions for their role then just remove those permissions. This will prevent an accidental action on a service the user should not be using.
AWS have a large number of actions you can do in the security pillar too, so make sure to take a read of that.
I have a dev and prod cognito pool, a dev/prod lambda function that pushes to a dev/prod dynamoDb table.
Is there a simple way to have it know when to use the prod credentials (pool id, etc), and when to use the dev credentials?
And same to do with firing the appropriate dev/prod API gateway apis that check the appropriate pools for authentication, and post to the appropriate dynamoDb tables? For now I just manually change the tokens, and in API Gateway, I manually switch out which cognito pool the API gateway authenticates and which tables they post to, which isn't very practical.
If you expose your lambda with API Gateway then just deploy it to two stages - a prod stage which calls the prod lambda which accesses prod Dynamodb & a dev stage which calls dev lambda. In your application, you would just need to change the stage name & you can do so by setting it from Info.plist.
Regarding how to get tokens for prod or dev automatically, it depends on how you get these tokens. For example, you could create a /login resource in API Gateway which takes username + password as parameters and returns tokens. Again, deploy it to two stages which use different Cognito pool in the backend calls. Now, you can use the same variable/property in your application to get the stage name for getting tokens too.
So, by just changing the value of one property you can switch between prod & dev in your app.