Effective GitLab CI/CD workflow with multiple Terraform configurations? - continuous-integration

Question

My team uses AWS for our infrastructure, across 3 different AWS accounts. We'll call them simply sandbox, staging, and production.
I recently set up Terraform against our AWS infrastructure, and its hierarchy maps against our accounts, then by either application, or AWS service itself. The repo structure looks something like this:
staging
iam
groups
main.tf
users
main.tf
s3
main.tf
sandbox
iam
...
production
applications
gitlab
main.tf
route53
main.tf
...
We're using separate configurations per AWS service (e.g., IAM or S3) or application (e.g., GitLab) so we don't end up with huge .tf files per account that would take a long time to apply updates for any one change. Ideally, we'd like to move away from the service-based configuration approach and move towards more application-based configurations, but the problem at hand remains the same either way.
This approach has been working fine when applying updates manually from the command line, but I'd love to move it to GitLab CI/CD to better automate our workflow, and that's where things have broken down.
In my existing setup, if I make an single change to, say, staging/s3/main.tf, GitLab doesn't seem to have a good way out of the box to only run terraform plan or terraform apply for that specific configuration.
If I instead moved everything into a single main.tf file for an entire AWS account (or multiple files but tied to a single state file), I could simply have GitLab trigger a job to do plan or apply to just that configuration. It might take 15 minutes to run based on the number of AWS resources we have in each account, but it's a potential option I suppose.
It seems like my issue might be ultimately related to how GitLab handles "monorepos" than how Terraform handles its workflow (after all, Terraform will happily plan/apply my changes if I simply tell it what has changed), although I'd also be interested in hearing about how people structure their Terraform environments given -- or in order to avoid entirely -- these limitations.
Has anyone solved an issue like this in their environment?

Answer 1

The nice thing about Terraform is that it's idempotent so you can just apply even if nothing has changed and it will be a no-op action anyway.
If for some reason you really only want to run a plan/apply on a specific directory when things change then you can achieve this by using only.changes so that Gitlab will only run the job if the specified files have changed.
So if you have your existing structure then it's as simple as doing something like this:
stages:
- terraform plan
- terraform apply
.terraform_template:
image: hashicorp/terraform:latest
before_script:
- LOCATION=$(echo ${CI_JOB_NAME} | cut -d":" -f2)
- cd ${LOCATION}
- terraform init
.terraform_plan_template:
stage: terraform plan
extends: .terraform_template
script:
- terraform plan -input=false -refresh=true -module-depth=-1 .
.terraform_apply_template:
stage: terraform apply
extends: .terraform_template
script:
- terraform apply -input=false -refresh=true -auto-approve=true .
terraform-plan:production/applications/gitlab:
extends: .terraform_plan_template
only:
refs:
- master
changes:
- production/applications/gitlab/*
- modules/gitlab/*
terraform-apply:production/applications/gitlab:
extends: .terraform_apply_template
only:
refs:
- master
changes:
- production/applications/gitlab/*
- modules/gitlab/*
I've also assumed the existence of modules that are in a shared location to indicate how this pattern can also look for changes elsewhere in the repo than just the directory you are running Terraform against.
If this isn't the case and you have a flatter structure and you're happy to have a single apply job then you can simplify this to something like:
stages:
- terraform
.terraform_template:
image: hashicorp/terraform:latest
stage: terraform
before_script:
- LOCATION=$(echo ${CI_JOB_NAME} | cut -d":" -f2)
- cd ${LOCATION}
- terraform init
script:
- terraform apply -input=false -refresh=true -auto-approve=true .
only:
refs:
- master
changes:
- ${CI_JOB_NAME}/*
production/applications/gitlab:
extends: .terraform_template
In general though this can just be avoided by allowing Terraform to run against all of the appropriate directories on every push (probably only applying on push to master or other appropriate branch) because, as mentioned, Terraform is idempotent so it won't do anything if nothing has changed. This also has the benefit that if your automation code hasn't changed but something has changed in your provider (such as someone opening up a security group) then Terraform will go put it back to how it should be the next time it is triggered.