I am trying to connect to an RDS Proxy with IAM authentication and getting invalid credentials error.
Error: Proxy authentication with IAM authentication failed for user "lambda_user" with TLS on. Reason: Invalid credentials. If you provide an IAM token, make sure to either use the correct password or enable IAM authentication
I added full RDS permissions to the Lambda and also attached database proxy to it.
def get_db_token():
db_client = rds_client('rds', region_name="us-east-1")
database_token = db_client.generate_db_auth_token(
DBHostname='test-rds.proxy-xxxxxxxx.us-east-1.rds.amazonaws.com',
Port=5432,
DBUsername='lambda_user')
return database_token
db_token = get_db_token()
f"postgresql://lambda_user:{db_token}#test-rds.proxy-xxxxxxx.us-east-1.rds.amazonaws.com:5432/TestDatabase?sslmode=require"
IAM policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "rds-db:connect",
"Resource": "arn:aws:rds-db:*:xxxxxxxxxx:dbuser:*/*"
}
]
}
I tried enhanced logging in RDS proxy but not clear on why IAM token is invalid.
You have to enable iam authentication on your RDS database.
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.Enabling.html
here is also an interresting link you can follow: https://aws.amazon.com/blogs/database/iam-role-based-authentication-to-amazon-aurora-from-serverless-applications/
Edit: changed lambda function and logs output, problem reminas :/
Have following lambda function as step in codepipeline:
import boto3
import json
import sys
import os
import pymysql
import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
DB_HOST = os.environ['DB_HOST']
DB_USER = os.environ['DB_USER']
DB_PASS = os.environ['DB_PASS'],
DB_PORT = int(os.environ['DB_PORT'])
codepipeline = boto3.client('codepipeline')
cursorType = pymysql.cursors.DictCursor
try:
connection = pymysql.connect(
host=DB_HOST,
user=DB_USER,
password=DB_PASS,
port=DB_PORT,
)
except pymysql.MySQLError as err:
logger.error("Error: Could not connect to MySql db")
logger.error(err)
sys.exit()
logger.info("Success: Connected to MySql db")
def lambda_handler(event, context):
cursor = connection.cursor()
try:
logger.info("Dropping db...")
cursor.execute(f"drop database {DB_NAME}")
logger.info("Creating db...")
cursor.execute(f"create database {DB_NAME}")
logger.info("Db created")
connection.close()
logger.info('Conection closed')
job_id = event['CodePipeline.job']['id']
logger.info("Job id `{job_id}`")
response = codepipeline.put_job_success_result(jobId=job_id)
logger.info(response)
except Exception as err:
logger.error(err)
response = codepipeline.put_job_failure_result(
jobId=job_id, failureDetails={'message': message, 'type': 'JobFailed'}
)
return {
"statusCode": 200,
}
Function log from function run:
START RequestId: 02e2f7cb-817d-4e49-90db-5b4cae5c9336 Version: $LATEST
[INFO] 2020-09-23T07:38:34.515Z Found credentials in environment variables.
[INFO] 2020-09-23T07:38:34.598Z Success: Connected to MySql db
{'CodePipeline.job': {'id': '9a8b13ea-d4f8-4aea-8481-60db0b7b5b5d... snip}
Dropping db
Creating db
Db created
Conection closed
[INFO] 2020-09-23T07:38:34.732Z 02e2f7cb-817d-4e49-90db-5b4cae5c9336 Job id 9a8b13ea-d4f8-4aea-8481-60db0b7b5b5d
successfuly done
END RequestId: 02e2f7cb-817d-4e49-90db-5b4cae5c9336
REPORT RequestId: 02e2f7cb-817d-4e49-90db-5b4cae5c9336 Duration: 60060.17 ms Billed Duration: 60000 ms Memory Size: 128 MB Max Memory Used: 76 MB Init Duration: 426.53 ms
2020-09-23T07:39:34.660Z 02e2f7cb-817d-4e49-90db-5b4cae5c9336 Task timed out after 60.06 seconds
[INFO] 2020-09-23T07:39:35.55Z Found credentials in environment variables.
[INFO] 2020-09-23T07:39:35.94Z Success: Connected to MySql db
START RequestId: 02e2f7cb-817d-4e49-90db-5b4cae5c9336 Version: $LATEST
{'CodePipeline.job': {'id': '9a8b13ea-d4f8-4aea-8481-60db0b7b5b5d',... snip}
Dropping db
Creating db
Db created
Conection closed
[INFO] 2020-09-23T07:41:39.974Z 02e2f7cb-817d-4e49-90db-5b4cae5c9336 Job id 9a8b13ea-d4f8-4aea-8481-60db0b7b5b5d
successfuly done
END RequestId: 02e2f7cb-817d-4e49-90db-5b4cae5c9336
REPORT RequestId: 02e2f7cb-817d-4e49-90db-5b4cae5c9336 Duration: 60060.32 ms Billed Duration: 60000 ms Memory Size: 128 MB Max Memory Used: 30 MB
2020-09-23T07:42:39.925Z 02e2f7cb-817d-4e49-90db-5b4cae5c9336 Task timed out after 60.06 seconds
How can I "force" lambda to report back to codepipeline that job is done either ok or not instead of just running in some kind of loop?
Lambda IAM role has policy attached like below:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ssm:Describe*",
"ssm:Get*",
"ssm:List*",
"kms:Decrypt",
"ssm:GetParametersByPath",
"ec2:DescribeNetworkInterfaces",
"ec2:CreateNetworkInterface",
"ec2:DeleteNetworkInterface",
"ec2:DescribeInstances",
"ec2:AttachNetworkInterface",
"codepipeline:PutJobSuccessResult",
"codepipeline:PutJobFailureResult"
],
"Resource": "*"
}
]
}
Help please as I can't find reason why lambda is not "letting know" back to pipeline about job status.
So either create codepipline vpc endpoint or move lambda to private network... case solved :).
I am trying to use the AWS RDS Proxy on my lambda to proxy our database (Aurora MySQL). I wasn't able to find any specific instructions for Sequelize, but it seemed like all I needed for RDS proxy to work is to create a signer, use it to get my token and then pass in the token as my password to the Sequelize constructor:
const signer = new RDS.Signer({
region: process.env.REGION,
hostname: process.env.DB_PROXY_ENDPOINT,
port: 3306,
username: process.env.DB_PROXY_USERNAME,
});
const token = signer.getAuthToken({
username: process.env.DB_PROXY_USERNAME,
});
const connection = new Sequelize(process.env.DB_DATABASE, process.env.DB_PROXY_USERNAME, token, {
dialect: 'mysql',
host: process.env.DB_HOSTNAME,
port: process.env.DB_PORT,
pool: {
acquire: 15000,
idle: 9000,
max: 10
},
});
The RDS proxy is attached to my lambda and I'm able to log the token, but as soon as I make a request against the database, my connection times out. Does anyone know if there is something I could be missing in this setup?
Here's how I connected from AWS Lambda to RDS Proxy using MySql (in typescript)
import { APIGatewayProxyEvent, APIGatewayProxyResult } from "aws-lambda";
import { Signer } from "#aws-sdk/rds-signer";
import { Sequelize } from "sequelize";
//other code
const signer = new Signer({
hostname: host
port: port,
region: region,
username: username,
});
const sequelize = new Sequelize({
username,
host,
port,
dialect: "mysql",
dialectOptions: {
ssl: "Amazon RDS",
authPlugins: {
mysql_clear_password: () => () => signer.getAuthToken(),
},
},
});
// some more code
Your connection timing out may be due to some authentication error, perhaps in the way you're passing in the token. I would double check your RDS Proxy IAM role has secretsmanager:GetSecretValue permission for the Secrets Manager resource of the db user credentials as well as kms:Decrypt on the key used to encrypt the secret. And your lambda (or whatever context your code is running in) has the rds-db:connect permission.
NOTE:
This doesn't include the connection pooling options, I'm still trying to figure out how to optimize that. Check out Using sequelize in AWS Lambda docs for a place to start.
I am using the AWS Node SDK to try and use a launch template to launch an EC2 instance. In the web console for the same user I have access to the Launch Template ID, but in the Node API it tells me the Launch Template with ID does not exist:
version: aws-sdk#2.606.0
Screenshot in AWS Console
Screenshot in my config file
(I've blocked the last few characters for security but they are the same)
This is the Node
AWS = require('aws-sdk');
AWS.config.update({region: global.settings.aws.region});
let instanceParams = {
LaunchTemplate: {
LaunchTemplateId: global.settings.aws.ltId
},
InstanceType: 't2.micro',
MinCount: 1,
MaxCount: 1
};
let instancePromise = new AWS.EC2().runInstances(instanceParams).promise();
instancePromise.then(
function(data) {
console.log(data);
}).catch(
function(err) {
console.error(err, err.stack);
});
This is the error the API returns:
InvalidLaunchTemplateId.NotFound: The specified launch template, with template ID lt-03969d13638b5XXXX, does not exist.
at Request.extractError (/ApplicationPath/node_modules/aws-sdk/lib/services/ec2.js:50:35)
at Request.callListeners (/ApplicationPath/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/ApplicationPath/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/ApplicationPath/node_modules/aws-sdk/lib/request.js:683:14)
at Request.transition (/ApplicationPath/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/ApplicationPath/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /ApplicationPath/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request. (/ApplicationPath/node_modules/aws-sdk/lib/request.js:38:9)
at Request. (/ApplicationPath/node_modules/aws-sdk/lib/request.js:685:12)
at Request.callListeners (/ApplicationPath/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
at Request.emit (/ApplicationPath/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/ApplicationPath/node_modules/aws-sdk/lib/request.js:683:14)
at Request.transition (/ApplicationPath/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/ApplicationPath/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /ApplicationPath/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request. (/ApplicationPath/node_modules/aws-sdk/lib/request.js:38:9)
at Request. (/ApplicationPath/node_modules/aws-sdk/lib/request.js:685:12)
at Request.callListeners (/ApplicationPath/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
at callNextListener (/ApplicationPath/node_modules/aws-sdk/lib/sequential_executor.js:96:12)
at IncomingMessage.onEnd (/ApplicationPath/node_modules/aws-sdk/lib/event_listeners.js:307:13)
at IncomingMessage.emit (events.js:214:15)
at IncomingMessage.EventEmitter.emit (domain.js:476:20) {
message: 'The specified launch template, with template ID lt-03969d13638b5XXXX, does not exist.',
code: 'InvalidLaunchTemplateId.NotFound',
time: 2020-01-22T15:29:18.641Z,
requestId: '16082702-49c0-4451-a3b7-570b930b5238',
statusCode: 400,
retryable: false,
retryDelay: 47.92654090836894
}
The IAM access includes both:
AmazonEC2FullAccess
AmazonEC2FullAccess
(The JSON are super long but I am happy to post those in here if needed)
My thought is that there is some permissioning error for the IAM but I can't determine what that might be. Thank you for the help and please let me know if there is any other clarity I can provide.
When you can see that a given AWS resource exists in the AWS console, but it is not returned by the awscli or SDK calls, (or vice-versa) then one of the following is likely the cause:
you have queried the wrong region
you have queried the wrong AWS account (your effective credentials are for a 2nd, different account)
you have mis-typed the resource name/id
your browser has cached results that no longer exist (refresh your browser)
Today I have a new AWS Lambda question, and can't find anywhere in Google.
I new a Lambda function, there is no question.
But when I input any code in this function[eg. console.log();] and click "Save", error is occured:
"The provided execution role does not have permissions to call DescribeNetworkInterfaces on EC2"
exports.handler = (event, context, callback) => {
callback(null, 'Hello from Lambda');
console.log(); // here is my code
};
I bound the function with Role: lambda_excute_execution(Policy:AmazonElasticTranscoderFullAccess)
And this function is not bound with any triggers now.
And then, I give the role "AdministratorAccess" Policy, I can save my source code correctly.
This role can run Functions successfully before today.
Is anyone know this error?
Thanks Very much!
This error is common if you try to deploy a Lambda in a VPC without giving it the required network interface related permissions ec2:DescribeNetworkInterfaces, ec2:CreateNetworkInterface, and ec2:DeleteNetworkInterface (see AWS Forum).
For example, this a policy that allows to deploy a Lambda into a VPC:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:DescribeNetworkInterfaces",
"ec2:CreateNetworkInterface",
"ec2:DeleteNetworkInterface",
"ec2:DescribeInstances",
"ec2:AttachNetworkInterface"
],
"Resource": "*"
}
]
}
If you are using terraform, just add:
resource "aws_iam_role_policy_attachment" "AWSLambdaVPCAccessExecutionRole" {
role = aws_iam_role.lambda.name
policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole"
}
via Managed Policy
To grant Lambda necessary permissions to dig in to a VPC where a production RDS db resides in a private subnet.
As mentioned by #portatlas above, the AWSLambdaVPCAccessExecutionRole managed policy fits like a glove (and we all know use of IAM Managed Policies is an AWS-recommended best-practice).
This is for Lambdas with a service role already attached.
AWS CLI
1. Get Lambda Service Role
Ask Lambda API for function configuration, query the role from that, output to text for an unquoted return.
aws lambda get-function-configuration \
--function-name <<your function name or ARN here>> \
--query Role \
--output text
return, take your-service-role-name to #2
your-service-role-name
2. Attach Managed Policy AWSLambdaVPCAccessExecutionRole to Service Role
aws iam attach-role-policy \
--role-name your-service-role-name \
--policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole
CDK 2 TypeScript
const lambdaVPCExecutionRole:iam.Role = new iam.Role(this, `createLambdaVPCExecutionRole`, {
roleName : `lambdaVPCExecutionRole`,
assumedBy : new iam.ServicePrincipal(`lambda.amazonaws.com`),
description : `Lambda service role to operate within a VPC`,
managedPolicies : [
iam.ManagedPolicy.fromAwsManagedPolicyName(`service-role/AWSLambdaVPCAccessExecutionRole`),
],
});
const lambdaFunction:lambda.Function = new lambda.Function(this, `createLambdaFunction`, {
runtime : lambda.Runtime.NODEJS_14_X,
handler : `lambda.handler`,
code : lambda.AssetCode.fromAsset(`./src`),
vpc : vpc,
role : lambdaVPCExecutionRole,
});
This is actually such a common issue.
You can resolve this by adding a custom Inline Policy to the Lambda execution role under the Permissions tab.
Just add this:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:DescribeNetworkInterfaces",
"ec2:CreateNetworkInterface",
"ec2:DeleteNetworkInterface",
"ec2:DescribeInstances",
"ec2:AttachNetworkInterface"
],
"Resource": "*"
}
]
}
There's a full tutorial with pictures here if you need more information (Terraform, CloudFormation, and AWS Console) or are confused: https://ao.ms/the-provided-execution-role-does-not-have-permissions-to-call-createnetworkinterface-on-ec2/
Additionally, a more recent sequence of steps follows:
Under your Lambda Function, select "Configuration"
Select "Permissions"
Select the execution role:
Select "Add Permissions"
Create Inline Policy
Select "JSON"
Paste the JSON above and select Review.
It seems like this has been answered many different ways already but as of this posting, AWS has a managed policy. If you just search for the AWSLambdaVPCAccessExecutionRole you will be able to attached that, and this method worked for me.
Here is the arn:
arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole
Just go to execution role -> Attach policy -> Search for 'AWSLambdaVPCAccessExecutionRole' and add it.
An example for Cloudformation and AWS SAM users.
This example lambda role definition adds the managed AWSLambdaVPCAccessExecutionRole and solves the issue:
Type: "AWS::IAM::Role"
Properties:
RoleName: "lambda-with-vpc-access"
ManagedPolicyArns:
- "arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole"
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action:
- sts:AssumeRole
Principal:
Service:
- lambda.amazonaws.com
Just cause there aren't enough answers already ;) I think this is the easiest way. If you're using the web admin console, when you're creating your Lambda function in the first place, down the bottom just expand 'Advanced Settings' and check 'Enable VPC' & choose your vpc... Simple! Before doing this, my connection to my RDS proxy was timing out. After doing this (and nothing else) - works great!
After a bit of experimentation, here is a solution using "least privilege". It's written in Python, for the AWS CDK. However the same could be applied to normal JSON
iam.PolicyDocument(
statements=[
iam.PolicyStatement(
effect=iam.Effect.ALLOW,
actions=["ec2:DescribeNetworkInterfaces"],
resources=["*"],
),
iam.PolicyStatement(
effect=iam.Effect.ALLOW,
actions=["ec2:CreateNetworkInterface"],
resources=[
f"arn:aws:ec2:{region}:{account_id}:subnet/{subnet_id}"
f"arn:aws:ec2:{region}:{account_id}:security-group/{security_group_id}",
f"arn:aws:ec2:{region}:{account_id}:network-interface/*",
],
),
iam.PolicyStatement(
effect=iam.Effect.ALLOW,
actions=["ec2:DeleteNetworkInterface"],
resources=[f"arn:aws:ec2:{region}:{account_id}:*/*"],
),
],
),
Here's a quick and dirty way of resolving the error.
Open IAM on AWS console, select the role that's attached to the Lambda function and give it the EC2FullAccess permission.
This will let you update the Lambda VPC by granting EC2 control access. Be sure to remove the permission from the role, the function still runs.
Is it more or less secure than leaving some permissions attached permanently? Debatable.
If you are using SAM you just need to add to the Globals in the Template, like this:
Globals:
Function:
VpcConfig:
SecurityGroupIds:
- sg-01eeb769XX2d6cc9b
SubnetIds:
- subnet-1a0XX614
- subnet-c6dXXb8b
- subnet-757XX92a
- subnet-8afXX9ab
- subnet-caeXX7ac
- subnet-b09XXd81
(of course, you can put all in variables, or parameters!)
and then, to the Lambda Function, add Policies to the Properties, like this:
BasicFunction:
Type: AWS::Serverless::Function
Properties:
Policies:
- AWSLambdaVPCAccessExecutionRole
- AWSLambdaBasicExecutionRole
It is definitely a strange error, but are you sure the example code you added is the one you're using in your lambda?
Because in your code, you are trying to log something in your lambda after returning control via the callback. In other words, first you told your lambda that you're done. Next, while it is busy shutting down and returning your results, you try to do some logging...
So first, I'd try this:
exports.handler = (event, context, callback) => {
console.log('this is a test');
// do stuff
callback(null, 'Hello from Lambda'); // only do a callback *after* you've run all your code
};
And see if that fixes the problem.