Error creating ElasticSearch domain: ValidationException: Authentication error - elasticsearch

Question

I have been getting this error lately while creating a ES domain using Terraform. Nothing has changed in the way I define the ES domain. I did however start using SSL (AWS ACM cert) on the ALB layer but that should not have affected this. Any ideas what it might be complaining about ?
resource "aws_elasticsearch_domain" "es" {
domain_name = "${var.es_domain}"
elasticsearch_version = "6.3"
cluster_config {
instance_type = "r4.large.elasticsearch"
instance_count = 2
zone_awareness_enabled = true
}
vpc_options {
subnet_ids = "${var.private_subnet_ids}"
security_group_ids = [
"${aws_security_group.es_sg.id}"
]
}
ebs_options {
ebs_enabled = true
volume_size = 10
}
access_policies = <<CONFIG
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "es:*",
"Principal": "*",
"Effect": "Allow",
"Resource": "arn:aws:es:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:domain/${var.es_domain}/*"
}
]
}
CONFIG
snapshot_options {
automated_snapshot_start_hour = 23
}
tags = {
Domain = "${var.es_domain}"
}
depends_on = [
"aws_iam_service_linked_role.es",
]
}
resource "aws_iam_service_linked_role" "es" {
aws_service_name = "es.amazonaws.com"
}
EDIT: Oddly enough, when I removed using the ACM cert and moved back to using HTTP (port 80) for my ALB Listener, the ES domain was provisioned.
Not sure what to make of this but clearly the ACM cert is interfering with the ES domain creation. Or I am doing something wrong with the ACM creation. Here is how I do it and use it -
resource "aws_acm_certificate" "ssl_cert" {
domain_name = "api.xxxx.io"
validation_method = "DNS"
tags = {
Environment = "development"
}
lifecycle {
create_before_destroy = true
}
}
resource "aws_alb_listener" "alb_listener" {
load_balancer_arn = "${aws_alb.alb.id}"
port = "443"
protocol = "HTTPS"
ssl_policy = "ELBSecurityPolicy-2016-08"
certificate_arn = "${aws_acm_certificate.ssl_cert.arn}"
default_action {
target_group_arn = "${aws_alb_target_group.default.id}"
type = "forward"
}
}
The cert is validated and issued by AWS pretty fast as far as I can see in the console. And as seen, it has nothing to do with the ES domain per say.

Answer 1

It sometimes occurs that when it creates an ES-domain before enabling a service-linked role, even though using depends_on.
maybe you can try using local-exec provisioner to wait.
resource "aws_iam_service_linked_role" "es" {
aws_service_name = "es.amazonaws.com"
provisioner "local-exec" {
command = "sleep 10"
}
}

Answer 2

Below one is enough for service-linked role creation, also incl the role in the depends_on
resource "aws_iam_service_linked_role" "es" {
aws_service_name = "es.amazonaws.com"
}