terraform environments and updating a single resource - aws-lambda

I am using Terraform for continuous deployments of lambda functions. The lambda module creates the function and initial aliases [DEV,QA,PROD]. When a change is made the source_code_hash is updated and Terraform updates the code. The challenge is when I want to update the alias from DEV to QA it updates the entire stack. The code is below. Your help is appreciated.
$ cat main.tf
module "sample" {
source = "./lambda"
name = "sample"
runtime = "nodejs6.10"
role = "${aws_iam_role.iam_role_for_lambda.arn}"
filename = "../Archive.zip"
source_code_hash = "${base64sha256(file("../Archive.zip"))}"
source_dir = "../sample"
alias = "${var.env_name}"
}
$ cat module/main.tf
resource "aws_lambda_function" "lambda" {
filename = "${var.filename}"
function_name = "${var.name}"
role = "${var.role}"
handler = "${var.name}.${var.handler}"
runtime = "${var.runtime}"
source_code_hash = "${data.archive_file.lambda_zip.output_base64sha256}"
publish = "true"
}
resource "aws_lambda_alias" "lambda_alias" {
count = "2"
name = "${element(var.alias, count.index)}"
#name = "${var.alias}"
description = "${var.name}"
function_name = "${aws_lambda_function.lambda.arn}"
function_version = "${aws_lambda_function.lambda.version}"
}

You need different tfvars and tfstate files for different environments.
Suppose you use latest terraform version.
create an environment folder (for example, env) and set <env>-backend.tf and <env>.tfvars files for each environment
$ cat main.tf
terraform {
required_version = ">= 0.9.1"
backend "s3" {
encrypt = "true"
}
}
$ cat env/dev-backend.tf
bucket = "terraform-<change-to-s3-global-unique-id>"
key = "terraform/dev/terraform.tfstate"
kms_key_id = "xxxx-xxxx-xxxx-xxxx"
$ cat env/dev.tfvars
env_name = dev
Do the same for qa and prod environments.
So you should be fine to run below commands to get same lambda tf codes work in different environments
rm -rf .terraform
export env="dev"
terraform init -backend=true -backend-config=env/${env}-backend.tf
terraform plan -var-file=./env/${env}.tfvars

Related

Terraform creating lambda before zip is ready

I'm trying to create a Terraform module that will build my JS lambdas, zip them and deploy them. This however proves to be problematic
resource "null_resource" "build_lambda" {
count = length(var.lambdas)
provisioner "local-exec" {
command = "mkdir tmp"
working_dir = path.root
}
provisioner "local-exec" {
command = var.lambdas[count.index].code.build_command
working_dir = var.lambdas[count.index].code.working_dir
}
}
data "archive_file" "lambda_zip" {
count = length(var.lambdas)
type = "zip"
source_dir = var.lambdas[count.index].code.working_dir
output_path = "${path.root}/tmp/${count.index}.zip"
depends_on = [
null_resource.build_lambda
]
}
/*******************************************************
* Lambda definition
*******************************************************/
resource "aws_lambda_function" "lambda" {
count = length(var.lambdas)
filename = data.archive_file.lambda_zip[count.index].output_path
source_code_hash = filebase64sha256(data.archive_file.lambda_zip[count.index].output_path)
function_name = "${var.application_name}-${var.lambdas[count.index].name}"
description = var.lambdas[count.index].description
handler = var.lambdas[count.index].handler
runtime = var.lambdas[count.index].runtime
role = aws_iam_role.iam_for_lambda.arn
memory_size = var.lambdas[count.index].memory_size
depends_on = [aws_iam_role_policy_attachment.lambda_logs, aws_cloudwatch_log_group.log_group, data.archive_file.lambda_zip]
}
The property source_code_hash = filebase64sha256(data.archive_file.lambda_zip[count.index].output_path) , although technically not obligatory, is necessary, or the existing lambda will never get overriden as Terraform will think that it is still the same version of lambda and will skip the deployment altogether. Unfortunately it looks like the method filebase64sha256 is evaluated before the creation of any resource. This means that there is no zip for the hash calculation and so I get the error
Error: Error in function call
on modules\api-gateway-lambda\main.tf line 35, in resource "aws_lambda_function" "lambda":
35: source_code_hash = filebase64sha256(data.archive_file.lambda_zip[count.index].output_path)
|----------------
| count.index is 0
| data.archive_file.lambda_zip is tuple with 1 element
Call to function "filebase64sha256" failed: no file exists at tmp\0.zip.
If i manually place a zip in the right location, I can see that the whole thing starts working and the zip eventually gets overriden by a new one, but the hash in this case must come from the previous zip.
What is the right way to execute the whole thing in the right order?
The archive_file data source has its own output_base64sha256 attribute which can give you that same result without asking Terraform to read a file that doesn't exist yet:
source_code_hash = data.archive_file.lambda_zip[count.index].output_base64sha256
The data source will populate this at the same time it creates the file, and because your lambda function depends on the data source it will therefore always be available before the lambda function configuration is evaluated.

Terraform: Why doesn't this attempt to link resources work?

AWS Lambda uploading requires the generation of a zip archive of required source code and libraries. For use of NodeJS as the language for Lambda, it may be more typically the case that you want a source file and the node_modules directory to be included in the zip archive. The Terraform archive provider gives a file_archive resource which works well when it can be used. It can't be used when you want more than just 1 file or 1 directory. See feature request . To work around this, I came up with this code below. It executes steps but not in the required sequence. Run it once and it updates the zip file, but doesn't upload it to AWS. I run it again and it uploads to AWS.
# This resource checks the state of the node_modules directory, hoping to determine,
# most of the time, when there was a change in that directory. Output
# is a 'mark' file with that data in it. That file can be hashed to
# trigger updates to zip file creation.
resource "null_resource" "get_directory_mark" {
provisioner "local-exec" {
command = "ls -l node_modules > node_modules.mark; find node_modules -type d -ls >> node_modules.mark"
interpreter = ["bash", "-lc"]
}
triggers = {
always = "${timestamp()}" # will trigger each run - small cost.
}
}
resource "null_resource" "make_zip" {
depends_on = ["null_resource.get_directory_mark"]
provisioner "local-exec" {
command = "zip -r ${var.lambda_zip} ${var.lambda_function_name}.js node_modules"
interpreter = ["bash", "-lc"]
}
triggers = {
source_hash = "${sha1("${file("lambda_process_firewall_updates.js")}")}"
node_modules = "${sha1("${file("node_modules.mark")}")}" # see above
}
}
resource "aws_lambda_function" "lambda_process" {
depends_on = ["null_resource.make_zip"]
filename = "${var.lambda_zip}"
function_name = "${var.lambda_function_name}"
description = "process items"
role = "${aws_iam_role.lambda_process.arn}"
handler = "${var.lambda_function_name}.handler"
runtime = "nodejs8.10"
memory_size = "128"
timeout = "60"
source_code_hash = "${base64sha256(file("lambda_process.zip"))}"
}
Other related discussion includes: this question on code hashing, (see my answer) and this GitHub issue.

How to iterate over all aws_instances in terraform?

I'm relatively new to terraform and I'm trying to iterate over all aws_instances to apply a null_resource. Can you use multiple splats to access all instances, regardless of their names?
The EC2 instances are broken down by three types:
aws_instance.web.* (3 instances)
aws_instance.app.* (3 instances)
aws_instance.db.* (2 instances)
Here's my attempt to apply a null_resource to all eight aws_instances:
resource "null_resource" "install_security_package" {
#count = "${length(aws_instance)}" #terraform error: resource count can't reference variable: aws_instance
#count = "${length(aws_instance.*)}" #terraform error: resource variables must be three parts: TYPE.NAME.ATTR
count = "${length(aws_instance.*.*)}" #terraform error: unknown resource 'aws_instance.*'
connection {
type = "ssh"
host = "${element(aws_instance.*.private_ip, count.index)}"
user = "${lookup(var.user, var.platform)}"
private_key = "${file("${var.private_key_path}")}"
timeout = "2m"
}
provisioner "remote-exec" {
inline = [
"sudo rpm -Uvh http://www.example.com/security/repo/security_baseline.rpm",
]
}
}
It is not currently possible to match all resources of a given type. The "splat" syntax, as you've seen, only allows selecting all of the instances created from a particular resource block.
The closest you can get to this with Terraform today is to concatenate together the different resources:
concat(aws_instance.web.*.private_ip, aws_instance.app.*.private_ip, aws_instance.db.*.private_ip)
In the current version of Terraform as of this answer it is necessary to use some of the workarounds shared in github issue #4084 in order to avoid duplicating that complex expression in multiple places. A forthcoming feature called Local Values will make this simpler in the near future, allowing the list to be given an name to be re-used in multiple places:
# Won't work until Terraform PR#15449 is merged and released
locals {
aws_instance_addrs = "${concat(aws_instance.web.*.private_ip, aws_instance.app.*.private_ip, aws_instance.db.*.private_ip)}"
}
resource "null_resource" "install_security_package" {
count = "${length(local.aws_instance_addrs)}"
connection {
type = "ssh"
host = "${local.aws_instance_addrs[count.index]}"
user = "${lookup(var.user, var.platform)}"
private_key = "${file("${var.private_key_path}")}"
timeout = "2m"
}
provisioner "remote-exec" {
inline = [
"sudo rpm -Uvh http://www.example.com/security/repo/security_baseline.rpm",
]
}
}

Unable to pass environment variables to Create_Function AWS SDK method in Ruby

I'm trying to execute the following Ruby code and it constantly fails with "unexpected value at params[:environment]" error. I tried many different options for passing Hash to 'environment' parameter but it triggers the same error.
require 'aws-sdk'
client = Aws::Lambda::Client.new(region: 'us-east-1')
args = {}
args[:role] = "some_role"
args[:function_name] = "function"
args[:handler] = "function_handler"
args[:runtime] = "java8"
code = {}
code[:zip_file] = ::File.open("file.jar", "rb").read
args[:code] = code
environment = {}
environment[:variables] = { "AAA": "BBB" }
args[:environment] = environment
client.create_function(args)
Fixed by upgrading Ruby AWS SDK from 2.6 to 2.9

When provisioning with Terraform, how does code obtain a reference to machine IDs (e.g. database machine address)

Let's say I'm using Terraform to provision two machines inside AWS:
An EC2 Machine running NodeJS
An RDS instance
How does the NodeJS code obtain the address of the RDS instance?
You've got a couple of options here. The simplest one is to create a CNAME record in Route53 for the database and then always point to that CNAME in your application.
A basic example would look something like this:
resource "aws_db_instance" "mydb" {
allocated_storage = 10
engine = "mysql"
engine_version = "5.6.17"
instance_class = "db.t2.micro"
name = "mydb"
username = "foo"
password = "bar"
db_subnet_group_name = "my_database_subnet_group"
parameter_group_name = "default.mysql5.6"
}
resource "aws_route53_record" "database" {
zone_id = "${aws_route53_zone.primary.zone_id}"
name = "database.example.com"
type = "CNAME"
ttl = "300"
records = ["${aws_db_instance.default.endpoint}"]
}
Alternative options include taking the endpoint output from the aws_db_instance and passing that into a user data script when creating the instance or passing it to Consul and using Consul Template to control the config that your application uses.
You may try Sparrowform - a lightweight provision tool for Terraform based instances, it's capable to make an inventory of Terraform resources and provision related hosts, passing all the necessary data:
$ terrafrom apply # bootstrap infrastructure
$ cat sparrowfile # this scenario
# fetches DB address from terraform cache
# and populate configuration file
# at server with node js code:
#!/usr/bin/env perl6
use Sparrowform;
$ sparrowfrom --ssh_private_key=~/.ssh/aws.pem --ssh_user=ec2 # run provision tool
my $rdb-adress;
for tf-resources() -> $r {
my $r-id = $r[0]; # resource id
if ( $r-id 'aws_db_instance.mydb') {
my $r-data = $r[1];
$rdb-address = $r-data<address>;
last;
}
}
# For instance, we can
# Install configuration file
# Next chunk of code will be applied to
# The server with node-js code:
template-create '/path/to/config/app.conf', %(
source => ( slurp 'app.conf.tmpl' ),
variables => %(
rdb-address => $rdb-address
),
);
# sparrowform --ssh_private_key=~/.ssh/aws.pem --ssh_user=ec2 # run provisioning
PS. disclosure - I am the tool author

Resources