I am trying to connect to an RDS Proxy with IAM authentication and getting invalid credentials error.
Error: Proxy authentication with IAM authentication failed for user "lambda_user" with TLS on. Reason: Invalid credentials. If you provide an IAM token, make sure to either use the correct password or enable IAM authentication
I added full RDS permissions to the Lambda and also attached database proxy to it.
def get_db_token():
db_client = rds_client('rds', region_name="us-east-1")
database_token = db_client.generate_db_auth_token(
DBHostname='test-rds.proxy-xxxxxxxx.us-east-1.rds.amazonaws.com',
Port=5432,
DBUsername='lambda_user')
return database_token
db_token = get_db_token()
f"postgresql://lambda_user:{db_token}#test-rds.proxy-xxxxxxx.us-east-1.rds.amazonaws.com:5432/TestDatabase?sslmode=require"
IAM policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "rds-db:connect",
"Resource": "arn:aws:rds-db:*:xxxxxxxxxx:dbuser:*/*"
}
]
}
I tried enhanced logging in RDS proxy but not clear on why IAM token is invalid.
You have to enable iam authentication on your RDS database.
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.Enabling.html
here is also an interresting link you can follow: https://aws.amazon.com/blogs/database/iam-role-based-authentication-to-amazon-aurora-from-serverless-applications/
I am working with Google Cloud in Go and following this article by John Hanley:
https://www.jhanley.com/google-cloud-improving-security-with-impersonation/
and mashed it with this SO answer:
How to authenticate Google APIs (Google Drive API) from Google Compute Engine and locally without downloading Service Account credentials?
The credentials are successfully saved to, "application_default_credentials.json":
Notice: "type": "impersonated_service_account"
{
"delegates": [],
"service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/[sa#example-2021.iam.gserviceaccount.com]:generateAccessToken",
"source_credentials": {
"client_id": "...apps.googleusercontent.com",
"client_secret": "...",
"refresh_token": "...",
"type": "authorized_user"
},
"type": "impersonated_service_account"
}
My code which produces an unknown credential type: "impersonated_service_account" error:
package main
import (
...
"cloud.google.com/go/storage"
"golang.org/x/oauth2"
"google.golang.org/api/docs/v1"
"google.golang.org/api/drive/v3"
"google.golang.org/api/impersonate"
"google.golang.org/api/option"
...
)
var Config.GoogleServiceAccount string = "sa#example-2021.iam.gserviceaccount.com"
func main(){
_ = getTokenAsImpersonator()
}
// From: https://pkg.go.dev/google.golang.org/api/impersonate#example-CredentialsTokenSource-ServiceAccount
func getTokenAsImpersonator() oauth2.TokenSource {
ctx := context.Background()
// Base credentials sourced from ADC or provided client options.
ts, err := impersonate.CredentialsTokenSource(ctx, impersonate.CredentialsConfig{
TargetPrincipal: Config.GoogleServiceAccount,
Scopes: []string{"https://www.googleapis.com/auth/cloud-platform"},
// Delegates: []string{"bar#project-id.iam.gserviceaccount.com"},
})
if err != nil {
log.Fatal(err)
}
return ts
}
The 'unknown credential type: "impersonated_service_account"' error:
google: error getting credentials using GOOGLE_APPLICATION_CREDENTIALS environment variable: unknown credential type: "impersonated_service_account"
Have I done something wrong or is this a bug?
UPDATE
Answering John's questions from the comments:
1.
a) What is the value of the environment variable GOOGLE_APPLICATION_CREDENTIALS?
GOOGLE_APPLICATION_CREDENTIALS=/Users/x/.config/gcloud/application_default_credentials.json
b) What command did you use to generate application_default_credentials.json?
gcloud auth application-default login --scopes=https://www.googleapis.com/auth/drive,https://www.googleapis.com/auth/userinfo.email,https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/accounts.reauth,openid --impersonate-service-account=[sa#example-2021.iam.gserviceaccount.com]
Response:
Credentials saved to file: [/Users/x/.config/gcloud/application_default_credentials.json]
c)Which OS and version?
MacOS 10.13.6
d)gcloud --version?
Google Cloud SDK 343.0.0
app-engine-go
app-engine-python 1.9.91
bq 2.0.69
cloud-datastore-emulator 2.1.0
core 2021.05.27
gsutil 4.62
If you can create a minimum example ...
I have updated the example code above.
At some point I had used the CLI to impersonate an account:
gcloud config set auth/impersonate_service_account <service account>
Then later on when trying to use the application default credentials command it wraps your credentials with the service account credentials.
gcloud auth application-default login
What you end up with is a file that looks like this:
{
"delegates": [],
"service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/example#example-project.iam.gserviceaccount.com:generateAccessToken",
"source_credentials": {
"client_id": "123abc.apps.googleusercontent.com",
"client_secret": "XXXXXXXXX",
"refresh_token": "XXXXXXXXX",
"type": "authorized_user"
},
"type": "impersonated_service_account"
}
This appears to cause a lot of problems with third party services such as terraform.
What is strange is that Terraform is just making API calls to Google using Google SDKs, so really its something to do with Google.
You need to remove the impersonation:
gcloud config unset auth/impersonate_service_account
And then run the application default credential command again:
gcloud auth application-default login
Now if you check your file it should look like this:
{
"client_id": "XXXXXXXXX",
"client_secret": "XXXXXXXXX",
"quota_project_id": "example-project",
"refresh_token": "XXXXXXXXXX",
"type": "authorized_user"
}
I was hitting the same issue when I was trying to impersonate an account so I could run Terraform commands as a service account instead of my personal account but it doesn't like that.
EDIT: Rereading you question it sounds like you're in the same boat as me. We want to use service accounts without physically downloading the keys. This is even mentioned by Google as best practice. But doing so is causing issues with their own SDKs.
I had the same issue running GCP Terraform provider tests. You can specify the Service Account Terraform have to impersonate setting the env variable GOOGLE_IMPERSONATE_SERVICE_ACCOUNT (documentation).
Configuration steps:
export GOOGLE_IMPERSONATE_SERVICE_ACCOUNT=SERVICE_ACCOUNT#PROJECT_ID.iam.gserviceaccount.com
gcloud auth application-default login
Am picking ECS optimised instance(ami-05958d7635caa4d04) in data plane of ECS in ca-central-1 region.
AWS Systems Manager Agent (SSM Agent) is Amazon software that can be installed and configured on an Amazon EC2 instance, an on-premises server, or a virtual machine (VM). SSM Agent makes it possible for Systems Manager to update, manage, and configure these resources.
In my scenario, Launching a ECS task in ECS optimised instance(ami-05958d7635caa4d04), causes resource:memory error. More on this error, here. Monitoring ECS->cluster->service->events will not work for me, because cloudformation roll back the cluster.
My existing ECS optimised instance is launched as shown below:
"EC2Instance":{
"Type": "AWS::EC2::Instance",
"Properties":{
"ImageId": "ami-05958d7635caa4d04",
"InstanceType": "t2.micro",
"SubnetId": { "Ref": "SubnetId"},
"KeyName": { "Ref": "KeyName"},
"SecurityGroupIds": [ { "Ref": "EC2InstanceSecurityGroup"} ],
"IamInstanceProfile": { "Ref" : "EC2InstanceProfile"},
"UserData":{
"Fn::Base64": { "Fn::Join": ["", [
"#!/bin/bash\n",
"echo ECS_CLUSTER=", { "Ref": "EcsCluster" }, " >> /etc/ecs/ecs.config\n",
"groupadd -g 1000 jenkins\n",
"useradd -u 1000 -g jenkins jenkins\n",
"mkdir -p /ecs/jenkins_home\n",
"chown -R jenkins:jenkins /ecs/jenkins_home\n"
] ] }
},
"Tags": [ { "Key": "Name", "Value": { "Fn::Join": ["", [ { "Ref": "AWS::StackName"}, "-instance" ] ]} }]
}
}
1) Does aws ssm agent installation required on ECS instance(ami-05958d7635caa4d04) to retrieve such cloudwatch events(resource:memory) with aws.ssm cloudwatch event rule filter? or Does aws.ec2 cloudwatch event rule filter suffice?
2) If yes, Do I need to explicitly install SSM agent on ECS instance(ami-05958d7635caa4d04)? through CloudFormation...
You don't need to install SSM agent to monitor something such as memory usage of your instance (whether container instance or not). This is domain of CloudWatch, not SSM.
All you need to install is unified cloud watch agent and configure it accordingly. This is where SSM can help but it is not necessary and you can install it manually (or via script if you want).
If you decide to use SSM then you will need to explicitly install it. It comes preinstalled on some OSes but not on Amazon ECS-Optimized AMI - more about this.
Today I have a new AWS Lambda question, and can't find anywhere in Google.
I new a Lambda function, there is no question.
But when I input any code in this function[eg. console.log();] and click "Save", error is occured:
"The provided execution role does not have permissions to call DescribeNetworkInterfaces on EC2"
exports.handler = (event, context, callback) => {
callback(null, 'Hello from Lambda');
console.log(); // here is my code
};
I bound the function with Role: lambda_excute_execution(Policy:AmazonElasticTranscoderFullAccess)
And this function is not bound with any triggers now.
And then, I give the role "AdministratorAccess" Policy, I can save my source code correctly.
This role can run Functions successfully before today.
Is anyone know this error?
Thanks Very much!
This error is common if you try to deploy a Lambda in a VPC without giving it the required network interface related permissions ec2:DescribeNetworkInterfaces, ec2:CreateNetworkInterface, and ec2:DeleteNetworkInterface (see AWS Forum).
For example, this a policy that allows to deploy a Lambda into a VPC:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:DescribeNetworkInterfaces",
"ec2:CreateNetworkInterface",
"ec2:DeleteNetworkInterface",
"ec2:DescribeInstances",
"ec2:AttachNetworkInterface"
],
"Resource": "*"
}
]
}
If you are using terraform, just add:
resource "aws_iam_role_policy_attachment" "AWSLambdaVPCAccessExecutionRole" {
role = aws_iam_role.lambda.name
policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole"
}
via Managed Policy
To grant Lambda necessary permissions to dig in to a VPC where a production RDS db resides in a private subnet.
As mentioned by #portatlas above, the AWSLambdaVPCAccessExecutionRole managed policy fits like a glove (and we all know use of IAM Managed Policies is an AWS-recommended best-practice).
This is for Lambdas with a service role already attached.
AWS CLI
1. Get Lambda Service Role
Ask Lambda API for function configuration, query the role from that, output to text for an unquoted return.
aws lambda get-function-configuration \
--function-name <<your function name or ARN here>> \
--query Role \
--output text
return, take your-service-role-name to #2
your-service-role-name
2. Attach Managed Policy AWSLambdaVPCAccessExecutionRole to Service Role
aws iam attach-role-policy \
--role-name your-service-role-name \
--policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole
CDK 2 TypeScript
const lambdaVPCExecutionRole:iam.Role = new iam.Role(this, `createLambdaVPCExecutionRole`, {
roleName : `lambdaVPCExecutionRole`,
assumedBy : new iam.ServicePrincipal(`lambda.amazonaws.com`),
description : `Lambda service role to operate within a VPC`,
managedPolicies : [
iam.ManagedPolicy.fromAwsManagedPolicyName(`service-role/AWSLambdaVPCAccessExecutionRole`),
],
});
const lambdaFunction:lambda.Function = new lambda.Function(this, `createLambdaFunction`, {
runtime : lambda.Runtime.NODEJS_14_X,
handler : `lambda.handler`,
code : lambda.AssetCode.fromAsset(`./src`),
vpc : vpc,
role : lambdaVPCExecutionRole,
});
This is actually such a common issue.
You can resolve this by adding a custom Inline Policy to the Lambda execution role under the Permissions tab.
Just add this:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:DescribeNetworkInterfaces",
"ec2:CreateNetworkInterface",
"ec2:DeleteNetworkInterface",
"ec2:DescribeInstances",
"ec2:AttachNetworkInterface"
],
"Resource": "*"
}
]
}
There's a full tutorial with pictures here if you need more information (Terraform, CloudFormation, and AWS Console) or are confused: https://ao.ms/the-provided-execution-role-does-not-have-permissions-to-call-createnetworkinterface-on-ec2/
Additionally, a more recent sequence of steps follows:
Under your Lambda Function, select "Configuration"
Select "Permissions"
Select the execution role:
Select "Add Permissions"
Create Inline Policy
Select "JSON"
Paste the JSON above and select Review.
It seems like this has been answered many different ways already but as of this posting, AWS has a managed policy. If you just search for the AWSLambdaVPCAccessExecutionRole you will be able to attached that, and this method worked for me.
Here is the arn:
arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole
Just go to execution role -> Attach policy -> Search for 'AWSLambdaVPCAccessExecutionRole' and add it.
An example for Cloudformation and AWS SAM users.
This example lambda role definition adds the managed AWSLambdaVPCAccessExecutionRole and solves the issue:
Type: "AWS::IAM::Role"
Properties:
RoleName: "lambda-with-vpc-access"
ManagedPolicyArns:
- "arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole"
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action:
- sts:AssumeRole
Principal:
Service:
- lambda.amazonaws.com
Just cause there aren't enough answers already ;) I think this is the easiest way. If you're using the web admin console, when you're creating your Lambda function in the first place, down the bottom just expand 'Advanced Settings' and check 'Enable VPC' & choose your vpc... Simple! Before doing this, my connection to my RDS proxy was timing out. After doing this (and nothing else) - works great!
After a bit of experimentation, here is a solution using "least privilege". It's written in Python, for the AWS CDK. However the same could be applied to normal JSON
iam.PolicyDocument(
statements=[
iam.PolicyStatement(
effect=iam.Effect.ALLOW,
actions=["ec2:DescribeNetworkInterfaces"],
resources=["*"],
),
iam.PolicyStatement(
effect=iam.Effect.ALLOW,
actions=["ec2:CreateNetworkInterface"],
resources=[
f"arn:aws:ec2:{region}:{account_id}:subnet/{subnet_id}"
f"arn:aws:ec2:{region}:{account_id}:security-group/{security_group_id}",
f"arn:aws:ec2:{region}:{account_id}:network-interface/*",
],
),
iam.PolicyStatement(
effect=iam.Effect.ALLOW,
actions=["ec2:DeleteNetworkInterface"],
resources=[f"arn:aws:ec2:{region}:{account_id}:*/*"],
),
],
),
Here's a quick and dirty way of resolving the error.
Open IAM on AWS console, select the role that's attached to the Lambda function and give it the EC2FullAccess permission.
This will let you update the Lambda VPC by granting EC2 control access. Be sure to remove the permission from the role, the function still runs.
Is it more or less secure than leaving some permissions attached permanently? Debatable.
If you are using SAM you just need to add to the Globals in the Template, like this:
Globals:
Function:
VpcConfig:
SecurityGroupIds:
- sg-01eeb769XX2d6cc9b
SubnetIds:
- subnet-1a0XX614
- subnet-c6dXXb8b
- subnet-757XX92a
- subnet-8afXX9ab
- subnet-caeXX7ac
- subnet-b09XXd81
(of course, you can put all in variables, or parameters!)
and then, to the Lambda Function, add Policies to the Properties, like this:
BasicFunction:
Type: AWS::Serverless::Function
Properties:
Policies:
- AWSLambdaVPCAccessExecutionRole
- AWSLambdaBasicExecutionRole
It is definitely a strange error, but are you sure the example code you added is the one you're using in your lambda?
Because in your code, you are trying to log something in your lambda after returning control via the callback. In other words, first you told your lambda that you're done. Next, while it is busy shutting down and returning your results, you try to do some logging...
So first, I'd try this:
exports.handler = (event, context, callback) => {
console.log('this is a test');
// do stuff
callback(null, 'Hello from Lambda'); // only do a callback *after* you've run all your code
};
And see if that fixes the problem.
I have an Elastic Transcoder pipeline configured, and it has successfully processed jobs created via the AWS Management Console. However, when using the Ruby API, the pipeline doesn't appear to exist:
et = AWS::ElasticTranscoder::Client.new
puts et.list_pipelines.inspect
# {:pipelines=>[], :request_id=>"e9e5ae2b-ca43-11e3-969d-530832cf62dd"}
Similarly, calling create_job with the correct :pipeline_id raises an error, claiming AWS returned a 404 for that pipeline ID.
According to the documentation, this does not indicate a permissions error. A permissions error should return a 403. But just to be sure, I set the IAM user's permissions to superuser as follows:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "*",
"Resource": "*"
}
]
}
Why would the pipeline not be found?
You have to connect to the same AWS region in which your pipeline resides. To find out the pipeline's region:
Go to the list of pipelines in the AWS Management Console.
Click the magnifying glass icon for your pipeline. This should open the pipeline's details.
Find the region in the ARN string. For example, us-west-2.
Then, when you connect to AWS, do it like this:
AWS.config({
:access_key_id => 'abc',
:secret_access_key => '123',
:region => 'us-west-2' # Or whatever your region is
})