Using Makefile with Terraform and split project layout - makefile

Question

I have a Terraform project layout that's similar to
stage
└ Makefile
└ terraform.tfvars
└ vpc
└ services
└ frontend-app
└ backend-app
└ vars.tf
└ outputs.tf
└ main.tf
└ data-storage
└ mysql
└ redis
Where the contents of Makefile are similar to
.PHONY: all plan apply destroy
all: plan
plan:
terraform plan -var-file terraform.tfvars -out terraform.tfplan
apply:
terraform apply -var-file terraform.tfvars
destroy:
terraform plan -destroy -var-file terraform.tfvars -out terraform.tfplan
terraform apply terraform.tfplan
As far as I understand it, Terraform will only run on templates in the current directory. So I would need to cd stage/services/backend-app and run terraform apply there.
However I would like to be able to manage the whole stack from the Makefile. I have not seen a good clean way to pass arguments to make.
My goal is to have targets such as
make s3 plan # verify syntax
make s3 apply # apply plan
Unless there's a better way to run terraform from a parent directory? Is there something similar to:
make all plan # create stage plan
make all apply # apply stage plan

Answer 1

Another solution could be to create a tmp folder on each run and use terraform init ... and terraform get..., like this (the example also shows the remote state management using partial configuration):
readonly orig_path=$(pwd) && \
mkdir tmp && \
cd tmp && \
terraform init -backend=true -backend-config="$tf_backend_config" -backend-config="key=${account}/${envir}/${project}.json" $project_path && \
terraform get $project_path && \
terraform apply && \
cd $orig_path && \
rm -fR tmp
Or maybe wrap the above into a shell script, and call it from make file under "apply" etc.
-- adding this section to address a comment/question from Sam Hammamy --
In general, with the way how the current versions of terraform processes projects, we do want to think ahead of time to how to structure our projects, and how to break them down into manageable still functional pieces. Which is why usually we break them into "foundational" projects like VPC, VPN, SecurityGroups, IAM-Policies, Bastions etc. vs. 'functional" like "db", "web-cluster" etc. We usually run/deploy/modify the "fundamental" pieces once or occasionally, while the "functional" pieces we might re-deploy several times a day.
Which means that with the fragmenting of our IaC code like that, we also will end up of fragmenting of our remote state accordingly, and the execution of our project deployment as well.
For a project structure, which reflects that "philosophy" we usually end up with a project structure similar to this (common modules are not shown):
├── projects
│ └── application-name
│ ├── dev
│ │ ├── bastion
│ │ ├── db
│ │ ├── vpc
│ │ └── web-cluster
│ ├── prod
│ │ ├── bastion
│ │ ├── db
│ │ ├── vpc
│ │ └── web-cluster
│ └── backend.config
└── run-tf.sh
Where each project is a subfolder, and for each application_name/env/component = folder (i.e. dev/vpc) we added a placeholder backend configuration file: backend.tf:
terraform {
backend "s3" {
}
}
Where the folder content for each component will contain files similar to:
│ ├── prod
│ │ ├── vpc
│ │ │ ├── backend.tf
│ │ │ ├── main.tf
│ │ │ ├── outputs.tf
│ │ │ └── variables.tf
At "application_name/" or "application_name/env" level we added a backend.config file, with a content:
bucket = "BUCKET_NAME"
region = "region_name"
lock = true
lock_table = "lock_table_name"
encrypt = true
Our wrapper shell script expects parameters application-name, environment, component, and the actual terraform cmd to run.
The content of run-tf.sh script (simplified):
#!/bin/bash
application=$1
envir=$2
component=$3
cmd=$4
tf_backend_config="root_path/$application/$envir/$component/backend.config"
terraform init -backend=true -backend-config="$tf_backend_config" -backend-config="key=tfstate/${application}/${envir}/${component}.json"
terraform get
terraform $cmd
Here is how a typical run-tf.sh invocation looks like (to be executed from Makefile):
$ run-tf.sh application_name dev vpc plan
$ run-tf.sh application_name prod bastion apply

Answer 2

We use shell scripts to handle this exact use case which more nicely handles cding around.
However you can set Make variables by either using environment variables or setting it directly on the command line following the target like this:
make target FOO=bar
So in your case you might want something like:
ifndef LOCATION
$(error LOCATION is not set)
endif
.PHONY: all plan apply destroy
all: plan
plan:
cd $(LOCATION) && \
terraform plan -var-file terraform.tfvars -out terraform.tfplan
apply:
cd $(LOCATION) && \
terraform apply -var-file terraform.tfvars
destroy:
cd $(LOCATION) && \
terraform plan -destroy -var-file terraform.tfvars -out terraform.tfplan
terraform apply terraform.tfplan
I'd probably be inclined to have a target that runs terraform get and also configures remote state as well but that should be trivial to set now.