I have an Elastic Transcoder pipeline configured, and it has successfully processed jobs created via the AWS Management Console. However, when using the Ruby API, the pipeline doesn't appear to exist:
et = AWS::ElasticTranscoder::Client.new
puts et.list_pipelines.inspect
# {:pipelines=>[], :request_id=>"e9e5ae2b-ca43-11e3-969d-530832cf62dd"}
Similarly, calling create_job with the correct :pipeline_id raises an error, claiming AWS returned a 404 for that pipeline ID.
According to the documentation, this does not indicate a permissions error. A permissions error should return a 403. But just to be sure, I set the IAM user's permissions to superuser as follows:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "*",
"Resource": "*"
}
]
}
Why would the pipeline not be found?
You have to connect to the same AWS region in which your pipeline resides. To find out the pipeline's region:
Go to the list of pipelines in the AWS Management Console.
Click the magnifying glass icon for your pipeline. This should open the pipeline's details.
Find the region in the ARN string. For example, us-west-2.
Then, when you connect to AWS, do it like this:
AWS.config({
:access_key_id => 'abc',
:secret_access_key => '123',
:region => 'us-west-2' # Or whatever your region is
})
Related
I am trying to connect to an RDS Proxy with IAM authentication and getting invalid credentials error.
Error: Proxy authentication with IAM authentication failed for user "lambda_user" with TLS on. Reason: Invalid credentials. If you provide an IAM token, make sure to either use the correct password or enable IAM authentication
I added full RDS permissions to the Lambda and also attached database proxy to it.
def get_db_token():
db_client = rds_client('rds', region_name="us-east-1")
database_token = db_client.generate_db_auth_token(
DBHostname='test-rds.proxy-xxxxxxxx.us-east-1.rds.amazonaws.com',
Port=5432,
DBUsername='lambda_user')
return database_token
db_token = get_db_token()
f"postgresql://lambda_user:{db_token}#test-rds.proxy-xxxxxxx.us-east-1.rds.amazonaws.com:5432/TestDatabase?sslmode=require"
IAM policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "rds-db:connect",
"Resource": "arn:aws:rds-db:*:xxxxxxxxxx:dbuser:*/*"
}
]
}
I tried enhanced logging in RDS proxy but not clear on why IAM token is invalid.
You have to enable iam authentication on your RDS database.
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.Enabling.html
here is also an interresting link you can follow: https://aws.amazon.com/blogs/database/iam-role-based-authentication-to-amazon-aurora-from-serverless-applications/
Today I have a new AWS Lambda question, and can't find anywhere in Google.
I new a Lambda function, there is no question.
But when I input any code in this function[eg. console.log();] and click "Save", error is occured:
"The provided execution role does not have permissions to call DescribeNetworkInterfaces on EC2"
exports.handler = (event, context, callback) => {
callback(null, 'Hello from Lambda');
console.log(); // here is my code
};
I bound the function with Role: lambda_excute_execution(Policy:AmazonElasticTranscoderFullAccess)
And this function is not bound with any triggers now.
And then, I give the role "AdministratorAccess" Policy, I can save my source code correctly.
This role can run Functions successfully before today.
Is anyone know this error?
Thanks Very much!
This error is common if you try to deploy a Lambda in a VPC without giving it the required network interface related permissions ec2:DescribeNetworkInterfaces, ec2:CreateNetworkInterface, and ec2:DeleteNetworkInterface (see AWS Forum).
For example, this a policy that allows to deploy a Lambda into a VPC:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:DescribeNetworkInterfaces",
"ec2:CreateNetworkInterface",
"ec2:DeleteNetworkInterface",
"ec2:DescribeInstances",
"ec2:AttachNetworkInterface"
],
"Resource": "*"
}
]
}
If you are using terraform, just add:
resource "aws_iam_role_policy_attachment" "AWSLambdaVPCAccessExecutionRole" {
role = aws_iam_role.lambda.name
policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole"
}
via Managed Policy
To grant Lambda necessary permissions to dig in to a VPC where a production RDS db resides in a private subnet.
As mentioned by #portatlas above, the AWSLambdaVPCAccessExecutionRole managed policy fits like a glove (and we all know use of IAM Managed Policies is an AWS-recommended best-practice).
This is for Lambdas with a service role already attached.
AWS CLI
1. Get Lambda Service Role
Ask Lambda API for function configuration, query the role from that, output to text for an unquoted return.
aws lambda get-function-configuration \
--function-name <<your function name or ARN here>> \
--query Role \
--output text
return, take your-service-role-name to #2
your-service-role-name
2. Attach Managed Policy AWSLambdaVPCAccessExecutionRole to Service Role
aws iam attach-role-policy \
--role-name your-service-role-name \
--policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole
CDK 2 TypeScript
const lambdaVPCExecutionRole:iam.Role = new iam.Role(this, `createLambdaVPCExecutionRole`, {
roleName : `lambdaVPCExecutionRole`,
assumedBy : new iam.ServicePrincipal(`lambda.amazonaws.com`),
description : `Lambda service role to operate within a VPC`,
managedPolicies : [
iam.ManagedPolicy.fromAwsManagedPolicyName(`service-role/AWSLambdaVPCAccessExecutionRole`),
],
});
const lambdaFunction:lambda.Function = new lambda.Function(this, `createLambdaFunction`, {
runtime : lambda.Runtime.NODEJS_14_X,
handler : `lambda.handler`,
code : lambda.AssetCode.fromAsset(`./src`),
vpc : vpc,
role : lambdaVPCExecutionRole,
});
This is actually such a common issue.
You can resolve this by adding a custom Inline Policy to the Lambda execution role under the Permissions tab.
Just add this:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:DescribeNetworkInterfaces",
"ec2:CreateNetworkInterface",
"ec2:DeleteNetworkInterface",
"ec2:DescribeInstances",
"ec2:AttachNetworkInterface"
],
"Resource": "*"
}
]
}
There's a full tutorial with pictures here if you need more information (Terraform, CloudFormation, and AWS Console) or are confused: https://ao.ms/the-provided-execution-role-does-not-have-permissions-to-call-createnetworkinterface-on-ec2/
Additionally, a more recent sequence of steps follows:
Under your Lambda Function, select "Configuration"
Select "Permissions"
Select the execution role:
Select "Add Permissions"
Create Inline Policy
Select "JSON"
Paste the JSON above and select Review.
It seems like this has been answered many different ways already but as of this posting, AWS has a managed policy. If you just search for the AWSLambdaVPCAccessExecutionRole you will be able to attached that, and this method worked for me.
Here is the arn:
arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole
Just go to execution role -> Attach policy -> Search for 'AWSLambdaVPCAccessExecutionRole' and add it.
An example for Cloudformation and AWS SAM users.
This example lambda role definition adds the managed AWSLambdaVPCAccessExecutionRole and solves the issue:
Type: "AWS::IAM::Role"
Properties:
RoleName: "lambda-with-vpc-access"
ManagedPolicyArns:
- "arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole"
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action:
- sts:AssumeRole
Principal:
Service:
- lambda.amazonaws.com
Just cause there aren't enough answers already ;) I think this is the easiest way. If you're using the web admin console, when you're creating your Lambda function in the first place, down the bottom just expand 'Advanced Settings' and check 'Enable VPC' & choose your vpc... Simple! Before doing this, my connection to my RDS proxy was timing out. After doing this (and nothing else) - works great!
After a bit of experimentation, here is a solution using "least privilege". It's written in Python, for the AWS CDK. However the same could be applied to normal JSON
iam.PolicyDocument(
statements=[
iam.PolicyStatement(
effect=iam.Effect.ALLOW,
actions=["ec2:DescribeNetworkInterfaces"],
resources=["*"],
),
iam.PolicyStatement(
effect=iam.Effect.ALLOW,
actions=["ec2:CreateNetworkInterface"],
resources=[
f"arn:aws:ec2:{region}:{account_id}:subnet/{subnet_id}"
f"arn:aws:ec2:{region}:{account_id}:security-group/{security_group_id}",
f"arn:aws:ec2:{region}:{account_id}:network-interface/*",
],
),
iam.PolicyStatement(
effect=iam.Effect.ALLOW,
actions=["ec2:DeleteNetworkInterface"],
resources=[f"arn:aws:ec2:{region}:{account_id}:*/*"],
),
],
),
Here's a quick and dirty way of resolving the error.
Open IAM on AWS console, select the role that's attached to the Lambda function and give it the EC2FullAccess permission.
This will let you update the Lambda VPC by granting EC2 control access. Be sure to remove the permission from the role, the function still runs.
Is it more or less secure than leaving some permissions attached permanently? Debatable.
If you are using SAM you just need to add to the Globals in the Template, like this:
Globals:
Function:
VpcConfig:
SecurityGroupIds:
- sg-01eeb769XX2d6cc9b
SubnetIds:
- subnet-1a0XX614
- subnet-c6dXXb8b
- subnet-757XX92a
- subnet-8afXX9ab
- subnet-caeXX7ac
- subnet-b09XXd81
(of course, you can put all in variables, or parameters!)
and then, to the Lambda Function, add Policies to the Properties, like this:
BasicFunction:
Type: AWS::Serverless::Function
Properties:
Policies:
- AWSLambdaVPCAccessExecutionRole
- AWSLambdaBasicExecutionRole
It is definitely a strange error, but are you sure the example code you added is the one you're using in your lambda?
Because in your code, you are trying to log something in your lambda after returning control via the callback. In other words, first you told your lambda that you're done. Next, while it is busy shutting down and returning your results, you try to do some logging...
So first, I'd try this:
exports.handler = (event, context, callback) => {
console.log('this is a test');
// do stuff
callback(null, 'Hello from Lambda'); // only do a callback *after* you've run all your code
};
And see if that fixes the problem.
I have Amazon S3 bucket named mysub.domain.com and tryin to put or get data from it thru Cloudflare's CDN (app based on Laravel 5.1 with CodeSleeve/laravel-stapler depends on aws/aws-sdk-php).
My Amazon S3 bucket policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "PublicReadGetObject",
"Effect": "Allow",
"Principal": "*",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::mysub.domain.com/*"
}
]
}
And CORS Configuration:
<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
<AllowedOrigin>*</AllowedOrigin>
<AllowedMethod>PUT</AllowedMethod>
<AllowedMethod>POST</AllowedMethod>
<AllowedMethod>DELETE</AllowedMethod>
<AllowedHeader>*</AllowedHeader>
</CORSRule>
</CORSConfiguration>
`
My Stapler config for s3:
`
's3_client_config' => [
...
'endpoint' => 'https://mysub.domain.com',
...
],
's3_object_config' => [
'Bucket' => 'mysub.domain.com',
...
],
I've created CNAME for my subdomain to Amazon's S3 bucket on Cloudflare as mentioned in documentation:
mysub.domain.com CNAME mysub.domain.com.s3.amazonaws.com
It work's without endpoint, but not going thru CDN, because using urls like s3.amazonaws.com/mysub.domain.com (path-style), but when i added endpoint it uses https://mysub.domain.com/mysub.domain.com (uses endpoint and bucket name). It must anyway add objects in bucket's path /mysub.domain.com/path/to/file.jpg, but it gaves me an error:
Aws\S3\Exception\SignatureDoesNotMatchException: AWS Error Code: SignatureDoesNotMatch, Status Code: 403, AWS Request ID: ABDC27DF1F472901, AWS Error Type: client, AWS Error Message: The request signature we calculated does not match the signature you provided. Check your key and signing method.
But as i said it works without endpoint.
Is there any way to avoid this error and duplicates of bucket name in url (maybe there's any way to switch it to domain-styled url)
Thank you in advance.
It's hard to say for sure with the details provided, but it may be that you have a bucket CNAME setup incorrectly. You may want to look at this Help Center article.
If that doesn't work you should send an email to support[at]cloudflare[dot]com so they can dig deeper.
I can launch ec2-instance with iam-role in management console.
But I have no idea how to launch ec2-instance with iam-role from aws-ruby-sdk
iam-role " test"'s Policy is here
"Effect": "Allow",
"Action": "*",
"Resource": "*"
Here is the result:
/var/lib/gems/1.8/gems/aws-sdk-1.7.1/lib/aws/core/client.rb:318:in `return_or_raise':
You are not authorized to perform iam:PassRole with arn:aws:iam::xxxxxxxxxxx:role/test
(AWS::EC2::Errors::UnauthorizedOperation)
The credentials you are using from your Ruby script do not have permission to launch an instance using the 'test' IAM Role. You need to modify the policy for this user, and grant it the IAM:PassRole permission, e.g.:
{
"Statement": [{
"Effect":"Allow",
"Action":"ec2:RunInstances",
"Resource":"*"
},
{
"Effect":"Allow",
"Action":"iam:PassRole",
"Resource":"arn:aws:iam::xxxxxxxxxxx:role/test"
}]
}
This is a security feature - it is possible to misconfigure IAM to allow privilege escalations, so AWS uses a "secure by default" policy.
You could also use this policy to allow your users to launch instances using any IAM role - but you should consider the security implications before doing this:
{
"Effect":"Allow",
"Action":"iam:PassRole",
"Resource":"*"
}]
Ref: http://docs.amazonwebservices.com/IAM/latest/UserGuide/role-usecase-ec2app.html
We're using Route 53 DNS to point to an EC2 instance. Is there any way to get Route 53 to point to the instance directly, instead of to an Elastic IP or CNAME?
I have multiple reasons for this:
I don't want to burn an IP.
CNAMEs are unreliable, because if an instance goes down and comes back up, the full name, ec2-X-X-X-X.compute-1.amazonaws.com, will change.
In the future, I need to spin up instances programmatically and address them with a subdomain, and I see no easy way to do this with either elastic IPs or CNAMEs.
What's the best approach?
I wrote my own solution to this problem since I was unhappy with other approaches that were presented here. Using Amazon CLI tools is nice, but they IMHO tend to be slower than direct API calls using other Amazon API libraries (Ruby for example).
Here's a link to my AWS Route53 DNS instance update Gist. It contains an IAM policy and a Ruby script. You should create a new user in IAM panel, update it with the attached policy (with your zone id in it) and set the credentials and parameters in the Ruby script. First parameter is the hostname alias for your instance in your hosted zone. Instance's private hostname is aliased to <hostname>.<domain> and instance's public hostname is aliased to <hostname>-public.<domain>
UPDATE: Here's a link to AWS Route53 DNS instance update init.d script registering hostnames when instance boots. Here's another one if want to use AWS Route53 DNS load-balancing in similar fashion.
If you stick to using route53, you can make a script that updates the CNAME record for that instance everytime it reboots.
see this -> http://cantina.co/automated-dns-for-aws-instances-using-route-53/ (disclosure, i did not create this, though i used it as a jumping point for a similar situation)
better yet, because you mentioned being able to spin up instances programmatically, this approach should guide you to that end.
see also -> http://docs.pythonboto.org/en/latest/index.html
Using a combination of Cloudwatch, Route53 and Lambda is also an option if you host at a least part of your dns in Route53. The advantage of this is that you don't need any applications running on the instance itself.
To use this this approach you configure a Cloudwatch rule to trigger a Lambda function whenever the status of an EC2 instance changes to running. The Lambda function can then retrieve the public ip address of the instance and update the dns record in Route53.
The Lambda could look something like this (using Node.js runtime):
var AWS = require('aws-sdk');
var ZONE_ID = 'Z1L432432423';
var RECORD_NAME = 'testaws.domain.tld';
var INSTANCE_ID = 'i-423423ccqq';
exports.handler = (event, context, callback) => {
var retrieveIpAddressOfEc2Instance = function(instanceId, ipAddressCallback) {
var ec2 = new AWS.EC2();
var params = {
InstanceIds: [instanceId]
};
ec2.describeInstances(params, function(err, data) {
if (err) {
callback(err);
} else {
ipAddressCallback(data.Reservations[0].Instances[0].PublicIpAddress);
}
});
}
var updateARecord = function(zoneId, name, ip, updateARecordCallback) {
var route53 = new AWS.Route53();
var dnsParams = {
ChangeBatch: {
Changes: [
{
Action: "UPSERT",
ResourceRecordSet: {
Name: name,
ResourceRecords: [
{
Value: ip
}
],
TTL: 60,
Type: "A"
}
}
],
Comment: "updated by lambda"
},
HostedZoneId: zoneId
};
route53.changeResourceRecordSets(dnsParams, function(err, data) {
if (err) {
callback(err, data);
} else {
updateARecordCallback();
}
});
}
retrieveIpAddressOfEc2Instance(INSTANCE_ID, function(ip) {
updateARecord(ZONE_ID, RECORD_NAME, ip, function() {
callback(null, 'record updated with: ' + ip);
});
});
}
You will need to execute the Lambda with a role that has permissions to describe EC2 instances and update records in Route53.
With Route 53 you can create alias records that map to an Elastic Load Balancer (ELB):
http://docs.amazonwebservices.com/Route53/latest/DeveloperGuide/HowToAliasRRS.html
I've not tried on aws EC2 instance but it should work too.
I've written a small Java program that detect the public IP of the machine and update a certain record on aws route 53.
The only requirement is that you need Java installed on your EC2 instance.
The project is hosted on https://github.com/renatodelgaudio/awsroute53 and you are also free to modify it in case you need it
You could configure it to run at boot time or as a crontab job so that your record get updated with the new public IP following instructions similar to these
Linux manual installation steps
I used this cli53 tool to let an EC2 instance create an A record for itself during startup.
https://github.com/barnybug/cli53
I added file following lines to my rc.local (please check your linux calls this script during startup):
IP=$(curl http://169.254.169.254/latest/meta-data/public-ipv4)
/usr/local/bin/cli53 rrcreate example.com "play 30 A $IP" --wait --replace
It creates an A record play.example.com pointing to the current public IP of the EC2 instance.
You need to assign a IAM role to EC2 instance, which allows the instance to manipulate Route 53. In the simplest case just create a IAM role using a predefined policy AmazonRoute53FullAccess. Then assign this role to the EC2 instance.
Assuming the EC2 instance has the aws command configured with proper permissions, the following shell script does it:
#!/bin/bash
IP=$(curl http://169.254.169.254/latest/meta-data/public-ipv4)
PROFILE="dnsuserprofile"
ZONE="XXXXXXXXXXXXXXXXXXXXX"
DOMAIN="my.domain.name"
TMPFILE="/tmp/updateip.json"
cat << EOF > $TMPFILE
{
"Comment": "Updating instance IP address",
"Changes": [
{
"Action": "UPSERT",
"ResourceRecordSet": {
"Name": "$DOMAIN",
"Type": "A",
"TTL": 300,
"ResourceRecords": [
{
"Value": "$IP"
}
]
}
}
]
}
EOF
aws route53 change-resource-record-sets --profile $PROFILE --hosted-zone-id $ZONE --change-batch file://$TMPFILE > /dev/null && \
rm $TMPFILE
Set that script to run on reboot, for example in cron:
#reboot /home/ec2-user/bin/updateip
The IAM policy can be as narrow as:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "route53:ChangeResourceRecordSets",
"Resource": "arn:aws:route53:::hostedzone/XXXXXXXXXXXXXXXXXXXXX"
}
]
}