Attaching AWS security group to multiple EC2 instances - amazon-ec2

I am spinning up multiple Amazon EC2 instances and need to attach a Security Group. I am able to achieve it for one EC2 instance but looking for solution for multiple EC2s. I am using TerraForm 0.12. Please let me know how can I use data resource :- data "aws_instances" (s).
Here is the code for single EC2 which i am trying to convert for multiple EC2s:
resource "aws_instance" "ec2_instance" {
count = "${var.ec2_instance_count}"
ami = "${data.aws_ami.app_qrm_ami.id}"
...
}
data "aws_instances" "ec2_instances" {
count = "${var.ec2_instance_count}"
filter {
name = "instance-id"
values = ["${aws_instance.ec2_instance.*.id[count.index]}"]
}
}
resource "aws_network_interface_sg_attachment" "sg_attachment" {
security_group_id = "${data.aws_security_group.security_group.id}"
network_interface_id = "${data.aws_instance.ec2_instance[count.index].network_interface_id}" //facing issues here.
}
I want to achieve this using data "aws_instances" #notice the (s). Thanks in advance.

For removing the Hard coding of ec2 AMI, you can use the following data provider:-
data "aws_ami" "amazon_linux" {
count = "${var.ec2_instance_count}"
most_recent = true
owners = ["amazon"]
filter {
name = "name"
values = [
"amzn-ami-hvm-*-x86_64-gp2",
]
}
filter {
name = "owner-alias"
values = [
"amazon",
]
}
}
For rendering the ami id:-
resource "aws_instance" "ec2_instance" {
count = "${var.ec2_instance_count}"
ami = "${data.aws_ami.amazon_linux[count.index].id}"
network_interface =
For getting network_interface_id:-
resource "aws_network_interface" "ec2_nic" {
count = "${var.ec2_instance_count}"
subnet_id = "${aws_subnet.public_a.id}"
private_ips = ["10.0.0.50"]
security_groups = ["${aws_security_group.web.id}"]
attachment {
instance = "${aws_instance.ec2_instance[count.index].id}"
}
}
resource "aws_network_interface_sg_attachment" "sg_attachment" {
security_group_id = "${data.aws_security_group.security_group.id}"
network_interface_id = "${aws_network_interface.ec2_ami[count.index].id}"
}

Thanks Karan, your answer solved the issue for me. Later the infra got fairly complex and I found a different and a smarter way to solve it. I would like to share with other people which might help TF community in future.
multiple internal SG {internal 0-7}and one external to all for creating different groups of swarm which allowed to communicate internal and selectively externally. Majorly used in Microsoft HPC grid.
resource "aws_instance" "ec2_instance" {
count = tonumber(var.mycount)
vpc_security_group_ids = [data.aws_security_group.external_security_group.id, element(data.aws_security_group.internal_security_group.*.id, count.index)]
...
}
resource "aws_security_group" "internal_security_group" {
count = tonumber(var.mycount)
name = "${var.internalSGname}${count.index}"
}
resource "aws_security_group" "external_security_group" {
name = ${var.external_sg_name}"
}

Related

terraform | how to add tag name for multiple instances?

Could u please let me know if is it possible to add tag name to multiple instances.
Below follow my code:
terraform.tfvars:
instance_names = [
"webserver01",
"webserver02",
"webserver03"
]
main.tf:
resource "aws_instance" "web" {
for_each = toset(var.instance_names)
ami = data.aws_ami.ubuntu.id
instance_type = var.instance_size
associate_public_ip_address = "true"
key_name = var.key_name
vpc_security_group_ids = [module.security.sg_id]
subnet_id = module.vpc.subnet_id
tags = {
"Name" = ["${var.instance_names}"]
}
}
The expected result on AWS should be something like this:
Name = webserver01
Name = webserver02
Name = webserver03
Does anyone knows how can achieve this?
I must have to keep use the toset instead of count
Best Regards,

ARM template for Azure Data Bricks Diagnostic settings

I am able to configure diagnostic settings for azure data bricks in the portal,I need a ARM template to automate the creation of diagnostic settings for azure data bricks.
let me know if any additional information required from my side.
Thanks in advance
Diagnostic settings for Azure Databricks Workspace are configured separately from its creation - you can use standard ARM templates for Azure Monitor that you can find in the documentation. You just need to select what category of diagnostic you want to enable, and modify ARM template correspondingly (the full list of categories could be found in the UI, here is the partial list: "dbfs", "clusters", "accounts", "jobs", "notebook", "ssh", "workspace", "secrets", "sqlPermissions", "instancePools").
If you're able to use Terraform, you can enable diagnostic settings via it as well, with something like this:
data "azurerm_databricks_workspace" "example" {
name = var.workspace_name
resource_group_name = var.resource_group
}
data "azurerm_eventhub_namespace_authorization_rule" "test" {
name = "test"
resource_group_name = var.resource_group
namespace_name = var.evhub_ns_name
}
resource "azurerm_monitor_diagnostic_setting" "test" {
name = "test"
target_resource_id = data.azurerm_databricks_workspace.example.id
eventhub_authorization_rule_id = data.azurerm_eventhub_namespace_authorization_rule.test.id
eventhub_name = var.evhub_name
dynamic "log" {
for_each = var.enabled_log_types
content {
category = log.value
enabled = true
}
}
}
variable resource_group {
type = string
description = "Resource group to deploy"
}
variable region {
type = string
description = "Region to deploy"
}
variable evhub_ns_name {
type = string
description = "Name of eventhub namespace"
}
variable evhub_name {
type = string
description = "Name of EventHubs topic"
}
variable workspace_name {
type = string
description = "The name of DB Workspace"
}
variable enabled_log_types {
type = list(string)
description = "List of the log types to enable"
default = ["dbfs", "clusters", "accounts", "jobs", "notebook", "ssh",
"workspace", "secrets", "sqlPermissions", "instancePools"]
}

How to define terraform aws_ami resource for Fedora Atomic Amazon Machine Image (ami)

I am trying to use terraform to get an aws_ami data resource as follows:
data "aws_ami" "fedora_atomic" {
most_recent = true
filter {
name = "name"
values = [
"ubuntu/images/hvm-ssd/ubuntu-trusty-14.04-amd64-server-*"] <==== What to specify here?
}
filter {
name = "virtualization-type"
values = [
"hvm"]
}
owners = [
"099720109477"] <=== What's the owner id?
# Canonical
}
But I want to replace the above with the following image desription, which I found on the AWS console:
Fedora-Atomic-25-20170727.0.x86_64-us-east-1-HVM-standard-0 - ami-00035c7b
Question
How do I find the right values for the fields above i.e. what is the correct code for the above for a Fedora Atomic image?
I am struggling to find this information.
Many Thanks
Fedora Atomic has been EOL since 2019 and you won't find new AMIs but to answer your question, the owner is the Account ID and you can find it from the AWS Console
The name can be part of what is available in the description, ie Fedora-Atomic-25-
Combining them all
data "aws_ami" "fedora_atomic" {
most_recent = true
filter {
name = "name"
values = ["Fedora-Atomic-25-*"]
}
filter {
name = "virtualization-type"
values = [ "hvm"]
}
owners = ["125523088429"]
}
output "ami" {
value = data.aws_ami.fedora_atomic.id
}

Iterating through map of objects in terraform with route 53 records

I am trying to figure out how to read from additional values in Terraform using for / for_each using Terraform 0.12.26
dns.tfvars
mx = {
"mywebsite.org." = {
ttl = "3600"
records = [
"home.mywebsite.org.",
"faq.mywebsite.org."
]
}
"myotherwebsite.org." = {
ttl = "3600"
records = [
"home.myotherwebsite.org."
]
}
}
variables.tf
variable "mx" {
type = map(object({
ttl = string
records = set(string)
}))
}
mx.tf
locals {
mx_records = flatten([
for mx_key, mx in var.mx : [
for record in mx.records : {
mx_key = mx_key
record = record
ttl = mx.ttl
}]
])
}
resource "aws_route53_record" "mx_records" {
for_each = { for mx in local.mx_records : mx.mx_key => mx... }
zone_id = aws_route53_zone.zone.zone_id
name = each.key
type = "MX"
ttl = each.value.ttl
records = [
each.value.record
]
}
In mx.tf, I can comment out the second value, faq.mywebsite.org, and the code works perfectly. I cannot figure out how to set up my for loop and for each statements to get it to "loop" through the second value. The first error I had received stated below:
Error: Duplicate object key
on mx.tf line 13, in resource "aws_route53_record" "mx_records":
13: for_each = { for mx in local.mx_records : mx.mx_key => mx }
|----------------
| mx.mx_key is "mywebsite.org."
Two different items produced the key "mywebsite.org." in this 'for'
expression. If duplicates are expected, use the ellipsis (...) after the value
expression to enable grouping by key.
To my understanding, I do not have two duplicate values helping to form the key so I should not have to use the ellipsis, but I tried using the ellipsis anyway to see if it would apply properly. After adding on the ellipsis after the value expression, I got this error:
Error: Unsupported attribute
on mx.tf line 20, in resource "aws_route53_record" "mx_records":
20: each.value.record
|----------------
| each.value is tuple with 2 elements
This value does not have any attributes.
Any advice on this issue would be appreciated.
UPDATE
Error: [ERR]: Error building changeset: InvalidChangeBatch: [Tried to create resource record set [name='mywebsiteorg.', type='MX'] but it already exists]
status code: 400, request id: dadd6490-efac-47ac-be5d-ab8dad0f4a6c
It's trying to create the record, but it already created because of the first record in the list.
I think you could just construct a map of your objects with key being the index of mx_records list (note the idx being the index):
resource "aws_route53_record" "mx_records" {
for_each = { for idx, mx in local.mx_records : idx => mx }
zone_id = aws_route53_zone.zone.zone_id
name = each.value.mx_key
type = "MX"
ttl = each.value.ttl
records = [
each.value.record
]
}
The above for_each expressions changes your local.mx_records from list(objects) to map(objects), where the map key is idx, and the value is the original object.
Update:
I verified in Route53 and you can't duplicate codes. Thus may try using orginal mx variable:
resource "aws_route53_record" "mx_records" {
for_each = { for idx, mx in var.mx : idx => mx }
zone_id = aws_route53_zone.zone.zone_id
name = each.key
type = "MX"
ttl = each.value.ttl
records = each.value.records
}
Moreover, if you want to avoid flatten function and for loop local variable, you can access the object in the map as:
resource "aws_route53_record" "mx_records" {
for_each = var.mx
zone_id = aws_route53_zone.zone.zone_id
name = each.key
type = "MX"
ttl = each.value["ttl"]
records = each.value["records"]
}

Error creating ElasticSearch domain: ValidationException: Authentication error

I have been getting this error lately while creating a ES domain using Terraform. Nothing has changed in the way I define the ES domain. I did however start using SSL (AWS ACM cert) on the ALB layer but that should not have affected this. Any ideas what it might be complaining about ?
resource "aws_elasticsearch_domain" "es" {
domain_name = "${var.es_domain}"
elasticsearch_version = "6.3"
cluster_config {
instance_type = "r4.large.elasticsearch"
instance_count = 2
zone_awareness_enabled = true
}
vpc_options {
subnet_ids = "${var.private_subnet_ids}"
security_group_ids = [
"${aws_security_group.es_sg.id}"
]
}
ebs_options {
ebs_enabled = true
volume_size = 10
}
access_policies = <<CONFIG
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "es:*",
"Principal": "*",
"Effect": "Allow",
"Resource": "arn:aws:es:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:domain/${var.es_domain}/*"
}
]
}
CONFIG
snapshot_options {
automated_snapshot_start_hour = 23
}
tags = {
Domain = "${var.es_domain}"
}
depends_on = [
"aws_iam_service_linked_role.es",
]
}
resource "aws_iam_service_linked_role" "es" {
aws_service_name = "es.amazonaws.com"
}
EDIT: Oddly enough, when I removed using the ACM cert and moved back to using HTTP (port 80) for my ALB Listener, the ES domain was provisioned.
Not sure what to make of this but clearly the ACM cert is interfering with the ES domain creation. Or I am doing something wrong with the ACM creation. Here is how I do it and use it -
resource "aws_acm_certificate" "ssl_cert" {
domain_name = "api.xxxx.io"
validation_method = "DNS"
tags = {
Environment = "development"
}
lifecycle {
create_before_destroy = true
}
}
resource "aws_alb_listener" "alb_listener" {
load_balancer_arn = "${aws_alb.alb.id}"
port = "443"
protocol = "HTTPS"
ssl_policy = "ELBSecurityPolicy-2016-08"
certificate_arn = "${aws_acm_certificate.ssl_cert.arn}"
default_action {
target_group_arn = "${aws_alb_target_group.default.id}"
type = "forward"
}
}
The cert is validated and issued by AWS pretty fast as far as I can see in the console. And as seen, it has nothing to do with the ES domain per say.
It sometimes occurs that when it creates an ES-domain before enabling a service-linked role, even though using depends_on.
maybe you can try using local-exec provisioner to wait.
resource "aws_iam_service_linked_role" "es" {
aws_service_name = "es.amazonaws.com"
provisioner "local-exec" {
command = "sleep 10"
}
}
Below one is enough for service-linked role creation, also incl the role in the depends_on
resource "aws_iam_service_linked_role" "es" {
aws_service_name = "es.amazonaws.com"
}

Resources