When provisioning with Terraform, how does code obtain a reference to machine IDs (e.g. database machine address) - amazon-ec2

Let's say I'm using Terraform to provision two machines inside AWS:
An EC2 Machine running NodeJS
An RDS instance
How does the NodeJS code obtain the address of the RDS instance?

You've got a couple of options here. The simplest one is to create a CNAME record in Route53 for the database and then always point to that CNAME in your application.
A basic example would look something like this:
resource "aws_db_instance" "mydb" {
allocated_storage = 10
engine = "mysql"
engine_version = "5.6.17"
instance_class = "db.t2.micro"
name = "mydb"
username = "foo"
password = "bar"
db_subnet_group_name = "my_database_subnet_group"
parameter_group_name = "default.mysql5.6"
}
resource "aws_route53_record" "database" {
zone_id = "${aws_route53_zone.primary.zone_id}"
name = "database.example.com"
type = "CNAME"
ttl = "300"
records = ["${aws_db_instance.default.endpoint}"]
}
Alternative options include taking the endpoint output from the aws_db_instance and passing that into a user data script when creating the instance or passing it to Consul and using Consul Template to control the config that your application uses.

You may try Sparrowform - a lightweight provision tool for Terraform based instances, it's capable to make an inventory of Terraform resources and provision related hosts, passing all the necessary data:
$ terrafrom apply # bootstrap infrastructure
$ cat sparrowfile # this scenario
# fetches DB address from terraform cache
# and populate configuration file
# at server with node js code:
#!/usr/bin/env perl6
use Sparrowform;
$ sparrowfrom --ssh_private_key=~/.ssh/aws.pem --ssh_user=ec2 # run provision tool
my $rdb-adress;
for tf-resources() -> $r {
my $r-id = $r[0]; # resource id
if ( $r-id 'aws_db_instance.mydb') {
my $r-data = $r[1];
$rdb-address = $r-data<address>;
last;
}
}
# For instance, we can
# Install configuration file
# Next chunk of code will be applied to
# The server with node-js code:
template-create '/path/to/config/app.conf', %(
source => ( slurp 'app.conf.tmpl' ),
variables => %(
rdb-address => $rdb-address
),
);
# sparrowform --ssh_private_key=~/.ssh/aws.pem --ssh_user=ec2 # run provisioning
PS. disclosure - I am the tool author

Related

Executing Powershell script on remote Windows EC2 instance in Terraform

I am starting a Windows EC2 instance in AWS. Now I want to install certain software like OpenSSH and some other tasks like creating user after the server has been created. If I have a PowerShell script, how do I execute on the remote instance?
I have a local PowerShell script - install_sft.ps1 and I want to execute on the remote EC2 instance in AWS.
I know I need to use a "provisioner" but unable to get my head around how to use it for Windows.
resource "aws_instance" "win-master" {
provider = aws.lmedba-dc
ami = data.aws_ssm_parameter.WindowsAmi.value
instance_type = var.instance-type
key_name = "RPNVirginia"
associate_public_ip_address = true
vpc_security_group_ids = [aws_security_group.windows-sg.id]
subnet_id = aws_subnet.dc1.id
tags = {
Name = "Win server"
}
depends_on = [aws_main_route_table_association.set-master-default-rt-assoc]
}
You can do this by making use of the user_data parameter of the aws_instance resource:
resource "aws_instance" "win-master" {
...
user_data_base64 = "${base64encode(file(install_sft.ps1))}"
...
}
Just ensure that install_sft.ps1 is in the same directory as your Terraform code.
An EC2 instance's User Data script executes when it starts up for the first time. See the AWS documentation here for more details.

Vault Error, Server gave HTTP response to HTTPS client

I'm using Hashicorp vault as a secrets store and installed it via apt repository on Ubuntu 20.04.
After that, I added the root key to access the UI and I'm able to add or delete secrets using UI.
Whenever I'm trying to add or get a secret using the command line, I get the following error :
jarvis#saki:~$ vault kv get secret/vault
Get "https://127.0.0.1:8200/v1/sys/internal/ui/mounts/secret/vault": http: server gave HTTP response to HTTPS client
My vault config looks like this :
# Full configuration options can be found at https://www.vaultproject.io/docs/configuration
ui = true
#mlock = true
#disable_mlock = true
storage "file" {
path = "/opt/vault/data"
}
#storage "consul" {
# address = "127.0.0.1:8500"
# path = "vault"
#}
# HTTP listener
#listener "tcp" {
# address = "127.0.0.1:8200"
# tls_disable = 1
#}
# HTTPS listener
listener "tcp" {
address = "0.0.0.0:8200"
tls_cert_file = "/opt/vault/tls/tls.crt"
tls_key_file = "/opt/vault/tls/tls.key"
}
# Example AWS KMS auto unseal
#seal "awskms" {
# region = "us-east-1"
# kms_key_id = "REPLACE-ME"
#}
# Example HSM auto unseal
#seal "pkcs11" {
# lib = "/usr/vault/lib/libCryptoki2_64.so"
# slot = "0"
# pin = "AAAA-BBBB-CCCC-DDDD"
# key_label = "vault-hsm-key"
# hmac_key_label = "vault-hsm-hmac-key"
#}
I fixed the problem. Though the exception can be common to more than one similar problem, I fixed the problem by exporting the root token generated after running this command :
vault server -dev
The output is like this
...
You may need to set the following environment variable:
$ export VAULT_ADDR='http://127.0.0.1:8200'
The unseal key and root token are displayed below in case you want to
seal/unseal the Vault or re-authenticate.
Unseal Key: 1+yv+v5mz+aSCK67X6slL3ECxb4UDL8ujWZU/ONBpn0=
Root Token: s.XmpNPoi9sRhYtdKHaQhkHP6x
Development mode should NOT be used in production installations!
...
Then just export these variables by running the following commands :
export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN="s.XmpNPoi9sRhYtdKHaQhkHP6x"
Note: Replace "s.XmpNPoi9sRhYtdKHaQhkHP6x" with your token received as output from the above command.
Then run the following command to check the status :
vault status
Again, the error message can be similar for many different problems.
In PowerShell on Windows 10, I was able to set it this way:
$Env:VAULT_ADDR='http://127.0.0.1:8200'
Then
vault status
returned correctly. This was on Vault 1.7.3 in dev mode
You can echo VAULT_ADDR by specifying it on the command line and pressing enter - same as the set line above but omitting the = sign and everything after it
$Env:VAULT_ADDR
Output:
Key Value
--- ----- Seal Type shamir Initialized true Sealed false Total Shares 1 Threshold 1 Version
1.7.3 Storage Type inmem Cluster Name vault-cluster-80649ba2 Cluster ID 2a35e304-0836-2896-e927-66722e7ca488 HA Enabled
false
Try using a new terminal window. This worked for me

How to iterate over all aws_instances in terraform?

I'm relatively new to terraform and I'm trying to iterate over all aws_instances to apply a null_resource. Can you use multiple splats to access all instances, regardless of their names?
The EC2 instances are broken down by three types:
aws_instance.web.* (3 instances)
aws_instance.app.* (3 instances)
aws_instance.db.* (2 instances)
Here's my attempt to apply a null_resource to all eight aws_instances:
resource "null_resource" "install_security_package" {
#count = "${length(aws_instance)}" #terraform error: resource count can't reference variable: aws_instance
#count = "${length(aws_instance.*)}" #terraform error: resource variables must be three parts: TYPE.NAME.ATTR
count = "${length(aws_instance.*.*)}" #terraform error: unknown resource 'aws_instance.*'
connection {
type = "ssh"
host = "${element(aws_instance.*.private_ip, count.index)}"
user = "${lookup(var.user, var.platform)}"
private_key = "${file("${var.private_key_path}")}"
timeout = "2m"
}
provisioner "remote-exec" {
inline = [
"sudo rpm -Uvh http://www.example.com/security/repo/security_baseline.rpm",
]
}
}
It is not currently possible to match all resources of a given type. The "splat" syntax, as you've seen, only allows selecting all of the instances created from a particular resource block.
The closest you can get to this with Terraform today is to concatenate together the different resources:
concat(aws_instance.web.*.private_ip, aws_instance.app.*.private_ip, aws_instance.db.*.private_ip)
In the current version of Terraform as of this answer it is necessary to use some of the workarounds shared in github issue #4084 in order to avoid duplicating that complex expression in multiple places. A forthcoming feature called Local Values will make this simpler in the near future, allowing the list to be given an name to be re-used in multiple places:
# Won't work until Terraform PR#15449 is merged and released
locals {
aws_instance_addrs = "${concat(aws_instance.web.*.private_ip, aws_instance.app.*.private_ip, aws_instance.db.*.private_ip)}"
}
resource "null_resource" "install_security_package" {
count = "${length(local.aws_instance_addrs)}"
connection {
type = "ssh"
host = "${local.aws_instance_addrs[count.index]}"
user = "${lookup(var.user, var.platform)}"
private_key = "${file("${var.private_key_path}")}"
timeout = "2m"
}
provisioner "remote-exec" {
inline = [
"sudo rpm -Uvh http://www.example.com/security/repo/security_baseline.rpm",
]
}
}

AWS - get autoscaling names for use with Capistrano

I am trying to setup a remote deployment with Capistrano on the Amazon Cloud.
The idea : I SSH to a random machine of the autoscaling group and I want to deploy to all the other machines from there. In order to do that I need to get the names of the other instances so I can define the Capistrano servers I want to deploy to
I have installed the Ruby sdk but I cannot figure out the best way to retrieve the instances names (taking advantage that I am on the VPN).
I have actually two possibilities : either find the instances by tags (I have tagged them with "production") or by the ID of the autoscaling group.
I don't want to use other "big guns" like Chef, etc.
After reading too much documentation
Two strategies : retrieve the dns names by autoscaling group OR by tags
By Tags
ec2 = Aws::EC2::Client.new
instances_tagged = ec2.describe_instances(
dry_run: false,
filters: [
{
name: 'tag:environment',
values: ['production'],
},
{
name: 'tag:stack',
values: ['rails'],
}
],
)
dns_tagged = instances_tagged.reservations[0].instances.map(&:private_dns_name)
By Autoscaling group
as = Aws::AutoScaling::Client.new
instances_of_as = as.describe_auto_scaling_groups(
auto_scaling_group_names: ['Autoscaling-Group-Name'],
max_records: 1,
).auto_scaling_groups[0].instances
if instances_of_as.empty?
autoscaling_dns = []
else
instances_ids = instances_of_as.map(&:instance_id)
autoscaling_dns = instance_ids.map do |instance_id|
ec2.instances[instance_id].private_dns_name
end
end

External configuration beside app.conf & environment variables for revel go framework

I have read revel app.conf manual for custom configuration and environment variables. however I couldn't find way to use additional external configuration along with app.conf.
My goal is to achieve external configuration file in addition to internal app.conf. Let's say creating a product called example and example product maintains it's sensible defaults with app.conf (not exposing to end user) instead product exposes config attributes via example.conf (default location could be /etc/example/example.conf) for product users.
For example: http config field from app.conf
http.addr =
http.port = 9000
extend it to example.conf
http.addr =
http.port = 9000
[database]
host = "localhost"
port = 8080
user = "username"
password = "password"
# etc...
Then I read example.conf during an application start use values also apply values on top of app.conf (overriding). Finally revel server starts!
How to achieve this goal with revel go framework?
It appears you are working against the design of the app.conf. It is already setup to be sectioned, for example all this is in a single app.conf file
[dev]
results.pretty = true
watch = true
http.addr = 192.168.1.2
[test]
results.pretty = true
watch = true
http.addr = 192.168.1.22
[prod]
results.pretty = false
watch = false
http.addr = 192.168.1.100
you can launch 3 different scenarios by using three different command line options
revel run bitbucket.org/mycorp/my-app dev
revel run bitbucket.org/mycorp/my-app test
revel run bitbucket.org/mycorp/my-app prod
I know this is not exactly what your goal is but you can acheive a similar result.
In github.com/revel/revel/revel.go around line 152 you have something like
Config, err = LoadConfig("app.conf").
Maybe you can try and modify that with this
if len(os.Getenv("SOME ENV VAR")) > 0 {
Config, err = LoadConfig("path/to/your/example.conf")
} else {
Config, err = LoadConfig("app.conf")
}
You just need to set env var on your prod server.
That way you will not be using app.conf but your example.conf.

Resources