I want to reuse my serverless.yml in different environments (dev, test, prod).
In the config I have:
provider:
name: aws
stage: ${opt:stage, 'dev'}
environment:
NODE_ENV: ${self:provider.stage}
Right now the value will be dev, test or prod (all in lower-case).
Is there a way to convert it toUpperCase() in a way that the input and self:provider:stage will stay as it is (i.e. lower-case) but the value of NODE_ENV will be UPPER-CASE?
Update (2022-10-13)
This answer was correct at the time of its writing (circa 2018). A better answer now is to use serverless-plugin-utils as stated in #ShashankRaj's comment below.
varName: ${upper(value)}
AFAIK, there is no such function in YAML.
You can achieve what you want though by using a map between the lowercase and uppercase names.
custom:
environments:
dev: DEV
test: TEST
prod: PROD
provider:
name: aws
stage: ${opt:stage, 'dev'}
environment:
NODE_ENV: ${self:custom.environments.${self:provider.stage}}
You can achieve something to this effect using the reference variables in javascript files functionality provided.
To take your example, this should work (assuming you're running in a node.js environment that supports modern syntax)
serverless.yml
...
provider:
name: aws
stage: ${opt:stage, 'dev'}
environment:
NODE_ENV: ${file(./yml-helpers.js):provider.stage.uppercase}
...
yml-helpers.js (adjacent to serverless.yml)
module.exports.provider = serverless => {
// The `serverless` argument containers all the information in the .yml file
const provider = serverless.service.provider;
return Object.entries(provider).reduce(
(accumulator, [key, value]) => ({
...accumulator,
[key]:
typeof value === 'string'
? {
lowercase: value.toLowerCase(),
uppercase: value.toUpperCase()
}
: value
}),
{}
)
};
I arrived at something that works, via reading some source code and console logging the entire serverless object. This example applies a helper function to title-case some input option values (apply str.toUpperCase() instead, as required). There is a result of parsing the input options already available in the serverless object.
// serverless-helpers.js
function toTitleCase(word) {
console.log("input word: " + word);
let lower = word.toLowerCase();
let title = lower.replace(lower[0], lower[0].toUpperCase());
console.log("output word: " + title);
return title;
}
module.exports.dynamic = function(serverless) {
// The `serverless` argument contains all the information in
// the serverless.yaml file
// serverless.cli.consoleLog('Use Serverless config and methods as well!');
// this is useful for discovery of what is available:
// serverless.cli.consoleLog(serverless);
const input_options = serverless.processedInput.options;
return {
part1Title: toTitleCase(input_options.part1),
part2Title: toTitleCase(input_options.part2)
};
};
# serverless.yaml snippet
custom:
part1: ${opt:part1}
part2: ${opt:part2}
dynamicOpts: ${file(./serverless-helpers.js):dynamic}
combined: prefix${self:custom.dynamicOpts.part1Title}${self:custom.dynamicOpts.part2Title}Suffix
This simple example assumes the input options are --part1={value} and --part2={value}, but the generalization is to traverse the properties of serverless.processedInput.options and apply any custom helpers to those values.
Using Serverless Plugin Utils:
plugins:
- serverless-plugin-utils
provider:
name: aws
stage: ${opt:stage, 'dev'}
environment:
NODE_ENV: ${upper(${self:provider.stage})}
Thanks to #ShashankRaj...
Related
I am using wkhtmltopdf in my code to generate reports data in my application.Which works well at my local machine.
I need to add a wkhtmltopdf layer to below lambda function to run the same on server.
const graphHandler = new lambda.Function(this, "graphqlHandler", {
runtime: lambda.Runtime.GO_1_X,
functionName: `${STAGE}-graphql`,
code: lambda.Code.fromAsset(Path.join("..", "bin")),
handler: "graphql",
tracing: Tracing.ACTIVE,
timeout: Duration.seconds(60),
memorySize: 512,
vpc: vpc,
vpcSubnets: {
subnets: vpc.privateSubnets,
},
securityGroups: [vpcSecurityGroup],
});
Searched various available articles and documents, but couldn't find much with cdk and Go combination. Anyone with answers, who solved the similar thing?
import { LayerVersion } from "aws-cdk-lib/aws-lambda";
const layerWkHtmlToPdf = LayerVersion.fromLayerVersionArn(this, "wkhtmltopdf",ARN)
Add above constant as a layer to lambda function
layers: [layerWkHtmlToPdf],
Note : This solution is valid if you have wkhtmltopdf layer available within your lambda section of your aws account.If not, then you can create a wkhtmltopdf layer using class LayerVersion
I have the following code:
const func = new NodejsFunction(this, <function name>, {
memorySize: 2048,
timeout: Duration.seconds(60),
runtime: Runtime.NODEJS_14_X,
handler: 'handler',
role: <role>,
entry: path.join(__dirname, <filePath>),
currentVersionOptions: {
description: `Version created on ${new Date(Date.now())}`,
},
});
const version = func.currentVersion;
const alias = new Alias(this, 'VersionAlias', {
aliasName: 'current',
version,
});
I do this with a handful of Lambda functions all in the same stack. The first deployment works, however the lambda functions are created with random version numbers (some have v4, some with v5, some with v7).
Subsequent deployments then fail with a vague Internal Failure error message. So I check the CloudTrail logs and find a series of ResourceNotFoundException errors. The "Version" resources are unable to be updated because they have the incorrect version number stemming from the first deploy. How can I force CloudFormation to start at #1 for versioning my lambda functions?
For anyone visiting this later, the problem was with the following code:
currentVersionOptions: {
description: `Version created on ${new Date(Date.now())}`,
},
Apparently you can't have a dynamic description as it is an immutable field
I am using Terraform to publish lambda to AWS. It works fine when I deploy to AWS but stuck on "Refreshing state..." when running against localstack.
Below is my .tf config file as you can see I configured the lambda endpoint to be http://localhost:4567.
provider "aws" {
profile = "default"
region = "ap-southeast-2"
endpoints {
lambda = "http://localhost:4567"
}
}
variable "runtime" {
default = "python3.6"
}
data "archive_file" "zipit" {
type = "zip"
source_dir = "crawler/dist"
output_path = "crawler/dist/deploy.zip"
}
resource "aws_lambda_function" "test_lambda" {
filename = "crawler/dist/deploy.zip"
function_name = "quote-crawler"
role = "arn:aws:iam::773592622512:role/LambdaRole"
handler = "handler.handler"
source_code_hash = "${data.archive_file.zipit.output_base64sha256}"
runtime = "${var.runtime}"
}
Below is docker compose file for localstack:
version: '2.1'
services:
localstack:
image: localstack/localstack
ports:
- "4567-4583:4567-4583"
- '8055:8080'
environment:
- SERVICES=${SERVICES-lambda }
- DEBUG=${DEBUG- }
- DATA_DIR=${DATA_DIR- }
- PORT_WEB_UI=${PORT_WEB_UI- }
- LAMBDA_EXECUTOR=${LAMBDA_EXECUTOR-docker-reuse }
- KINESIS_ERROR_PROBABILITY=${KINESIS_ERROR_PROBABILITY- }
- DOCKER_HOST=unix:///var/run/docker.sock
volumes:
- "/var/run/docker.sock:/var/run/docker.sock"
Does anyone know how to fix the issue?
This is how i fixed similar issue :
Set export TF_LOG=TRACE which is the most verbose logging.
Run terraform plan ....
In the log, I got the root cause of the issue and it was :
dag/walk: vertex "module.kubernetes_apps.provider.helmfile (close)" is waiting for "module.kubernetes_apps.helmfile_release_set.metrics_server"
From logs, I identify the state which is the cause of the issue: module.kubernetes_apps.helmfile_release_set.metrics_server.
I deleted its state :
terraform state rm module.kubernetes_apps.helmfile_release_set.metrics_server
Now run terraform plan again should fix the issue.
This is not the best solution, that's why I contacted the owner of this provider to fix the issue without this workaround.
The reason I failed because terraform tries to check credentials against AWS. Add below two lines in your .tf configuration file solves the issue.
skip_credentials_validation = true
skip_metadata_api_check = true
I ran into the same issue and fixed it by logging into the aws dev profile from the console.
So don't forget to log in.
provider "aws" {
region = "ap-southeast-2"
profile = "dev"
}
I want to hide my secret credential from my yaml, i need to use .env, so how to call .env file from my yaml, so that every I call this YAML, YAML will automatic call .env file. Please help me. thx
Instead of using an .env file, which is a simple properties file if you're following dotenv package, you can do the following:
create additional .yml file, for example .secrets.yml. you can store the secrets per stage:
prod:
MY_SECRET: foo
dev:
MY_SECRET: bar
store your secrets/configurations there
Then in serverless.yml:
load this file into an object:
custom:
secrets: ${file(.secrets.yml):${self:provider.stage}}
load object fields as environment variables:
provider:
environment:
MY_SECRET: ${self:custom.secrets.MY_SECRET}
How to test locally
In your tests you can load the secrets file this way:
const yaml = require('js-yaml');
const fs = require('fs');
const _ = require('lodash');
module.exports.loadSecrets = function (env = 'dev', path = './.secrets.yml') {
const secrets = yaml.load(fs.readFileSync(path));
_.forEach(secrets[env], (value, key) => {
process.env[key] = value;
});
}
Reference: http://www.goingserverless.com/blog/using-environment-variables-with-the-serverless-framework
I want to create an EC2 cloudformation stack which basically can be described in the following steps:
1.- Launch instance
2.- Provision the instance
3.- Stop the instance and create an AMI image out of it
4.- Create an autoscaling group with the created AMI image as source to launch new instances.
Basically I can do 1 and 2 in one cloudformation template and 4 in a second template. What I don't seem able to do is to create an AMI image from an instance inside a cloudformation template, which basically generates the problem of having to manually remove the AMI if I want to remove the stack.
That being said, my questions are:
1.- Is there a way to create an AMI image from an instance INSIDE the cloudformation template?
2.- If the answer to 1 is no, is there a way to add an AMI image (or any other resource for that matter) to make it part of a completed stack?
EDIT:
Just to clarify, I've already solved the problem of creating the AMI and using it in a cloudformation template, I just can't create the AMI INSIDE the cloudformation template or add it somehow to the created stack.
As I commented on Rico's answer, what I do now is use an ansible playbook which basically has 3 steps:
1.- Create a base instance with a cloudformation template
2.- Create, using ansible, an AMI of the instance created on step 1
3.- Create the rest of the stack (ELB, autoscaling groups, etc) with a second cloudformation template that updates the one created on step 1, and that uses the AMI created on step 2 to launch instances.
This is how I manage it now, but I wanted to know if there's any way to create an AMI INSIDE a cloudformation template or if it's possible to add the created AMI to the stack (something like telling the stack, "Hey, this belongs to you as well, so handle it").
Yes, you can create an AMI from an EC2 instance within a CloudFormation template by implementing a Custom Resource that calls the CreateImage API on create (and calls the DeregisterImage and DeleteSnapshot APIs on delete).
Since AMIs can sometimes take a long time to create, a Lambda-backed Custom Resource will need to re-invoke itself if the wait has not completed before the Lambda function times out.
Here's a complete example:
Description: Create an AMI from an EC2 instance.
Parameters:
ImageId:
Description: Image ID for base EC2 instance.
Type: AWS::EC2::Image::Id
# amzn-ami-hvm-2016.09.1.20161221-x86_64-gp2
Default: ami-9be6f38c
InstanceType:
Description: Instance type to launch EC2 instances.
Type: String
Default: m3.medium
AllowedValues: [ m3.medium, m3.large, m3.xlarge, m3.2xlarge ]
Resources:
# Completes when the instance is fully provisioned and ready for AMI creation.
AMICreate:
Type: AWS::CloudFormation::WaitCondition
CreationPolicy:
ResourceSignal:
Timeout: PT10M
Instance:
Type: AWS::EC2::Instance
Properties:
ImageId: !Ref ImageId
InstanceType: !Ref InstanceType
UserData:
"Fn::Base64": !Sub |
#!/bin/bash -x
yum -y install mysql # provisioning example
/opt/aws/bin/cfn-signal \
-e $? \
--stack ${AWS::StackName} \
--region ${AWS::Region} \
--resource AMICreate
shutdown -h now
AMI:
Type: Custom::AMI
DependsOn: AMICreate
Properties:
ServiceToken: !GetAtt AMIFunction.Arn
InstanceId: !Ref Instance
AMIFunction:
Type: AWS::Lambda::Function
Properties:
Handler: index.handler
Role: !GetAtt LambdaExecutionRole.Arn
Code:
ZipFile: !Sub |
var response = require('cfn-response');
var AWS = require('aws-sdk');
exports.handler = function(event, context) {
console.log("Request received:\n", JSON.stringify(event));
var physicalId = event.PhysicalResourceId;
function success(data) {
return response.send(event, context, response.SUCCESS, data, physicalId);
}
function failed(e) {
return response.send(event, context, response.FAILED, e, physicalId);
}
// Call ec2.waitFor, continuing if not finished before Lambda function timeout.
function wait(waiter) {
console.log("Waiting: ", JSON.stringify(waiter));
event.waiter = waiter;
event.PhysicalResourceId = physicalId;
var request = ec2.waitFor(waiter.state, waiter.params);
setTimeout(()=>{
request.abort();
console.log("Timeout reached, continuing function. Params:\n", JSON.stringify(event));
var lambda = new AWS.Lambda();
lambda.invoke({
FunctionName: context.invokedFunctionArn,
InvocationType: 'Event',
Payload: JSON.stringify(event)
}).promise().then((data)=>context.done()).catch((err)=>context.fail(err));
}, context.getRemainingTimeInMillis() - 5000);
return request.promise().catch((err)=>
(err.code == 'RequestAbortedError') ?
new Promise(()=>context.done()) :
Promise.reject(err)
);
}
var ec2 = new AWS.EC2(),
instanceId = event.ResourceProperties.InstanceId;
if (event.waiter) {
wait(event.waiter).then((data)=>success({})).catch((err)=>failed(err));
} else if (event.RequestType == 'Create' || event.RequestType == 'Update') {
if (!instanceId) { failed('InstanceID required'); }
ec2.waitFor('instanceStopped', {InstanceIds: [instanceId]}).promise()
.then((data)=>
ec2.createImage({
InstanceId: instanceId,
Name: event.RequestId
}).promise()
).then((data)=>
wait({
state: 'imageAvailable',
params: {ImageIds: [physicalId = data.ImageId]}
})
).then((data)=>success({})).catch((err)=>failed(err));
} else if (event.RequestType == 'Delete') {
if (physicalId.indexOf('ami-') !== 0) { return success({});}
ec2.describeImages({ImageIds: [physicalId]}).promise()
.then((data)=>
(data.Images.length == 0) ? success({}) :
ec2.deregisterImage({ImageId: physicalId}).promise()
).then((data)=>
ec2.describeSnapshots({Filters: [{
Name: 'description',
Values: ["*" + physicalId + "*"]
}]}).promise()
).then((data)=>
(data.Snapshots.length === 0) ? success({}) :
ec2.deleteSnapshot({SnapshotId: data.Snapshots[0].SnapshotId}).promise()
).then((data)=>success({})).catch((err)=>failed(err));
}
};
Runtime: nodejs4.3
Timeout: 300
LambdaExecutionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal: {Service: [lambda.amazonaws.com]}
Action: ['sts:AssumeRole']
Path: /
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
- arn:aws:iam::aws:policy/service-role/AWSLambdaRole
Policies:
- PolicyName: EC2Policy
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- 'ec2:DescribeInstances'
- 'ec2:DescribeImages'
- 'ec2:CreateImage'
- 'ec2:DeregisterImage'
- 'ec2:DescribeSnapshots'
- 'ec2:DeleteSnapshot'
Resource: ['*']
Outputs:
AMI:
Value: !Ref AMI
For what it's worth, here's Python variant of wjordan's AMIFunction definition in the original answer. All other resources in the original yaml remain unchanged:
AMIFunction:
Type: AWS::Lambda::Function
Properties:
Handler: index.handler
Role: !GetAtt LambdaExecutionRole.Arn
Code:
ZipFile: !Sub |
import logging
import cfnresponse
import json
import boto3
from threading import Timer
from botocore.exceptions import WaiterError
logger = logging.getLogger()
logger.setLevel(logging.INFO)
def handler(event, context):
ec2 = boto3.resource('ec2')
physicalId = event['PhysicalResourceId'] if 'PhysicalResourceId' in event else None
def success(data={}):
cfnresponse.send(event, context, cfnresponse.SUCCESS, data, physicalId)
def failed(e):
cfnresponse.send(event, context, cfnresponse.FAILED, str(e), physicalId)
logger.info('Request received: %s\n' % json.dumps(event))
try:
instanceId = event['ResourceProperties']['InstanceId']
if (not instanceId):
raise 'InstanceID required'
if not 'RequestType' in event:
success({'Data': 'Unhandled request type'})
return
if event['RequestType'] == 'Delete':
if (not physicalId.startswith('ami-')):
raise 'Unknown PhysicalId: %s' % physicalId
ec2client = boto3.client('ec2')
images = ec2client.describe_images(ImageIds=[physicalId])
for image in images['Images']:
ec2.Image(image['ImageId']).deregister()
snapshots = ([bdm['Ebs']['SnapshotId']
for bdm in image['BlockDeviceMappings']
if 'Ebs' in bdm and 'SnapshotId' in bdm['Ebs']])
for snapshot in snapshots:
ec2.Snapshot(snapshot).delete()
success({'Data': 'OK'})
elif event['RequestType'] in set(['Create', 'Update']):
if not physicalId: # AMI creation has not been requested yet
instance = ec2.Instance(instanceId)
instance.wait_until_stopped()
image = instance.create_image(Name="Automatic from CloudFormation stack ${AWS::StackName}")
physicalId = image.image_id
else:
logger.info('Continuing in awaiting image available: %s\n' % physicalId)
ec2client = boto3.client('ec2')
waiter = ec2client.get_waiter('image_available')
try:
waiter.wait(ImageIds=[physicalId], WaiterConfig={'Delay': 30, 'MaxAttempts': 6})
except WaiterError as e:
# Request the same event but set PhysicalResourceId so that the AMI is not created again
event['PhysicalResourceId'] = physicalId
logger.info('Timeout reached, continuing function: %s\n' % json.dumps(event))
lambda_client = boto3.client('lambda')
lambda_client.invoke(FunctionName=context.invoked_function_arn,
InvocationType='Event',
Payload=json.dumps(event))
return
success({'Data': 'OK'})
else:
success({'Data': 'OK'})
except Exception as e:
failed(e)
Runtime: python2.7
Timeout: 300
No.
I suppose Yes. Once the stack you can use the "Update Stack" operation. You need to provide the full JSON template of the initial stack + your changes in that same file (Changed AMI) I would run this in a test environment first (not production), as I'm not really sure what the operation does to the existing instances.
Why not create an AMI initially outside cloudformation and then use that AMI in your final cloudformation template ?
Another option is to write some automation to create two cloudformation stacks and you can delete the first one once the AMI that you've created is finalized.
While #wjdordan's solution is good for simple use cases, updating the User Data will not update the AMI.
(DISCLAIMER: I am the original author) cloudformation-ami aims at allowing you to declare AMIs in CloudFormation that can be reliably created, updated and deleted. Using cloudformation-ami You can declare custom AMIs like this:
MyAMI:
Type: Custom::AMI
Properties:
ServiceToken: !ImportValue AMILambdaFunctionArn
Image:
Name: my-image
Description: some description for the image
TemplateInstance:
ImageId: ami-467ca739
IamInstanceProfile:
Arn: arn:aws:iam::1234567890:instance-profile/MyProfile-ASDNSDLKJ
UserData:
Fn::Base64: !Sub |
#!/bin/bash -x
yum -y install mysql # provisioning example
# Signal that the instance is ready
INSTANCE_ID=`wget -q -O - http://169.254.169.254/latest/meta-data/instance-id`
aws ec2 create-tags --resources $INSTANCE_ID --tags Key=UserDataFinished,Value=true --region ${AWS::Region}
KeyName: my-key
InstanceType: t2.nano
SecurityGroupIds:
- sg-d7bf78b0
SubnetId: subnet-ba03aa91
BlockDeviceMappings:
- DeviceName: "/dev/xvda"
Ebs:
VolumeSize: '10'
VolumeType: gp2