I'm using Gradle with the wrapper, and the folder structure by default is like so:
.
├── settings.gradle
├── build.gradle
├── gradle.properties
├── gradle
│ └── wrapper
│ ├── gradle-wrapper.jar
│ └── gradle-wrapper.properties
├── gradlew
└── gradlew.bat
However, I would like to change it to so:
.
├── gradle
| ├── build.gradle
│ ├── settings.gradle
│ ├── gradle.properties
│ └── wrapper
│ ├── gradlew
│ ├── gradlew.bat
│ ├── gradle-wrapper.jar
│ └── gradle-wrapper.properties
└── src
├── main
└── test
Other than the fact that I don't know how to tell IntelliJ about the folder structure, I don't know how to change it for Gradle since the Environment Options related with changing the folder structure are deprecated:
-b, --build-file (deprecated)
Specifies the build file. For example: gradle --build-file=foo.gradle. The default is build.gradle, then build.gradle.kts.
-c, --settings-file (deprecated)
Specifies the settings file. For example: gradle --settings-file=somewhere/else/settings.gradle
You can't tell Gradle and Intellij IDEA that you use a non-standard Gradle build layout. And in all honesty, you shouldn't even consider that unless you have strong reasons to do so. There are mainly two reasons for that:
Developers familiar with one Gradle project feel immediately at home when starting with your Gradle project.
A non-standard build file and directory layout requires additional logic in IDE's (which is not present) and requires to provide extra parameters when building on the command line.
To put things into context, please have look at Gradle issue #16402.
Deprecate command-line options that describe the build layout
The -b and -c command-line options are effectively used to describe a non-standard build layout to Gradle. This is problematic because it means that a specific combination of options must be used whenever Gradle is used on that build, for example whenever invoked from the IDE, CI, command-line or some other tool. These command-line options also have some potentially surprising behaviours, such as running a settings script present in the target directory.
We don't think there are any use cases that are strong enough to justify keeping these options, and we should remove them (via deprecation). If we discover there are some use cases, we might consider replacing the options with more self-describing contracts, for example conventions for build script names.
I am using swagger to develop a new api written in Go. This is my first swagger project. I installed and used this command to create my project from a swagger.yaml. I aim to make reconfigurations I put into the swagger.yaml file part of my pipeline tasks - putting a task in to execute something like swagger-codegen generate -i ./api/swagger.yaml -l go-server by strategically setting up ignores in my .swagger-codegen-ignore file. There is one thing I don't necessarily like but i can't figure out how to change. Any advice? Do i need to live with it?
the generated directory structure looks like this for go-server
.
├── api
│ └──swagger.yaml
├── go #everyting in this directory is part of the "swagger" package
│ ├── a_handler_function_file.go
│ ├── logger.go
│ ├── model_struct_file.go
│ ├── routers.go
│ └── ...
├── Dockerfile
└── main.go
I am not keen on the directory called go or the package it produces called swagger. I want something more meaningful to the project.
Does it go against conventions to rename the directory?
Is there a way to configure the swagger-codegen to rename these what I want? - I am doing research to see if there is a way but I can't find one.
It seems that SEO magic has not really crawled in a way to effectively land on this page in the swagger-codegen git repo https://github.com/swagger-api/swagger-codegen#customizing-the-generator . maybe this Q and A will help.
One can either use add a -D<configParameterName> to the generate command or one can create a config.json file and add it to the generate command using -c config.yaml.
for go-server there are only two parameters available, packageName and hideGenerationTimestamp.
So I tried swagger-codegen generate -i ./swagger.yaml -l go-server -DpackageName="myPackageName" and it worked!!!
I also tried creating a config.json file that looks like this
{
"packageName": "myPackageName"
}
and then generate command that looks like this swagger-codegen generate -i ./swagger.yaml -l go-server -c config.json
and that works too.
As far as changing the go directory - it looks like I will have to live with it
my working tree is like this:
/opt/go/src/tb-to-composer/
├── apis
│ └── rtb.go
├── config.yaml
├── jsondef
│ └── structures.go
├── LICENSE.md
├── README.md
├── tb-to-composer
└── thingsToComposer.go
when I do go build inside /opt/go/src/tb-to-composer/ the build doesn't recompile rtb.go and structures.go even though there was changes in them. In order to achieve build I need to run go build -a every time I do a change to rtb.go or structures.go, is that the expected behavior from go build? How to I recompile only custom libs inside my package folder without recompile the whole /opt/go/src tree?
You can try the -i flag, or (this does not work, sorry) specify the files in the directories explicitly as arguments to go build, i.e. go build thingsToComposer.go apis/rtb.go jsondef/structures.go
I have a Terraform project layout that's similar to
stage
└ Makefile
└ terraform.tfvars
└ vpc
└ services
└ frontend-app
└ backend-app
└ vars.tf
└ outputs.tf
└ main.tf
└ data-storage
└ mysql
└ redis
Where the contents of Makefile are similar to
.PHONY: all plan apply destroy
all: plan
plan:
terraform plan -var-file terraform.tfvars -out terraform.tfplan
apply:
terraform apply -var-file terraform.tfvars
destroy:
terraform plan -destroy -var-file terraform.tfvars -out terraform.tfplan
terraform apply terraform.tfplan
As far as I understand it, Terraform will only run on templates in the current directory. So I would need to cd stage/services/backend-app and run terraform apply there.
However I would like to be able to manage the whole stack from the Makefile. I have not seen a good clean way to pass arguments to make.
My goal is to have targets such as
make s3 plan # verify syntax
make s3 apply # apply plan
Unless there's a better way to run terraform from a parent directory? Is there something similar to:
make all plan # create stage plan
make all apply # apply stage plan
Another solution could be to create a tmp folder on each run and use terraform init ... and terraform get..., like this (the example also shows the remote state management using partial configuration):
readonly orig_path=$(pwd) && \
mkdir tmp && \
cd tmp && \
terraform init -backend=true -backend-config="$tf_backend_config" -backend-config="key=${account}/${envir}/${project}.json" $project_path && \
terraform get $project_path && \
terraform apply && \
cd $orig_path && \
rm -fR tmp
Or maybe wrap the above into a shell script, and call it from make file under "apply" etc.
-- adding this section to address a comment/question from Sam Hammamy --
In general, with the way how the current versions of terraform processes projects, we do want to think ahead of time to how to structure our projects, and how to break them down into manageable still functional pieces. Which is why usually we break them into "foundational" projects like VPC, VPN, SecurityGroups, IAM-Policies, Bastions etc. vs. 'functional" like "db", "web-cluster" etc. We usually run/deploy/modify the "fundamental" pieces once or occasionally, while the "functional" pieces we might re-deploy several times a day.
Which means that with the fragmenting of our IaC code like that, we also will end up of fragmenting of our remote state accordingly, and the execution of our project deployment as well.
For a project structure, which reflects that "philosophy" we usually end up with a project structure similar to this (common modules are not shown):
├── projects
│ └── application-name
│ ├── dev
│ │ ├── bastion
│ │ ├── db
│ │ ├── vpc
│ │ └── web-cluster
│ ├── prod
│ │ ├── bastion
│ │ ├── db
│ │ ├── vpc
│ │ └── web-cluster
│ └── backend.config
└── run-tf.sh
Where each project is a subfolder, and for each application_name/env/component = folder (i.e. dev/vpc) we added a placeholder backend configuration file: backend.tf:
terraform {
backend "s3" {
}
}
Where the folder content for each component will contain files similar to:
│ ├── prod
│ │ ├── vpc
│ │ │ ├── backend.tf
│ │ │ ├── main.tf
│ │ │ ├── outputs.tf
│ │ │ └── variables.tf
At "application_name/" or "application_name/env" level we added a backend.config file, with a content:
bucket = "BUCKET_NAME"
region = "region_name"
lock = true
lock_table = "lock_table_name"
encrypt = true
Our wrapper shell script expects parameters application-name, environment, component, and the actual terraform cmd to run.
The content of run-tf.sh script (simplified):
#!/bin/bash
application=$1
envir=$2
component=$3
cmd=$4
tf_backend_config="root_path/$application/$envir/$component/backend.config"
terraform init -backend=true -backend-config="$tf_backend_config" -backend-config="key=tfstate/${application}/${envir}/${component}.json"
terraform get
terraform $cmd
Here is how a typical run-tf.sh invocation looks like (to be executed from Makefile):
$ run-tf.sh application_name dev vpc plan
$ run-tf.sh application_name prod bastion apply
We use shell scripts to handle this exact use case which more nicely handles cding around.
However you can set Make variables by either using environment variables or setting it directly on the command line following the target like this:
make target FOO=bar
So in your case you might want something like:
ifndef LOCATION
$(error LOCATION is not set)
endif
.PHONY: all plan apply destroy
all: plan
plan:
cd $(LOCATION) && \
terraform plan -var-file terraform.tfvars -out terraform.tfplan
apply:
cd $(LOCATION) && \
terraform apply -var-file terraform.tfvars
destroy:
cd $(LOCATION) && \
terraform plan -destroy -var-file terraform.tfvars -out terraform.tfplan
terraform apply terraform.tfplan
I'd probably be inclined to have a target that runs terraform get and also configures remote state as well but that should be trivial to set now.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
Improve this question
I'm in the process of swapping over our infrastructure into terraform.
What's the best practice for actually managing the terraform files and state?
I realize it's infrastructure as code, and i'll commit my .tf files into git, but do I commit tfstate as well? Should that reside somewhere like S3 ? I would like eventually for CI to manage all of this, but that's far stretched and requires me to figure out the moving pieces for the files.
I'm really just looking to see how people out there actually utilize this type of stuff in production
I am also in a state of migrating existing AWS infrastructure to Terraform so shall aim to update the answer as I develop.
I have been relying heavily on the official Terraform examples and multiple trial and error to flesh out areas that I have been uncertain in.
.tfstate files
Terraform config can be used to provision many boxes on different infrastructure, each of which could have a different state. As it can also be run by multiple people this state should be in a centralised location (like S3) but not git.
This can be confirmed looking at the Terraform .gitignore.
Developer control
Our aim is to provide more control of the infrastructure to developers whilst maintaining a full audit (git log) and the ability to sanity check changes (pull requests). With that in mind the new infrastructure workflow I am aiming towards is:
Base foundation of common AMI's that include reusable modules e.g. puppet.
Core infrastructure provisioned by DevOps using Terraform.
Developers change Terraform configuration in Git as needed (number of instances; new VPC; addition of region/availability zone etc).
Git configuration pushed and a pull request submitted to be sanity checked by a member of DevOps squad.
If approved, calls webhook to CI to build and deploy (unsure how to partition multiple environments at this time)
Edit 1 - Update on current state
Since starting this answer I have written a lot of TF code and feel more comfortable in our state of affairs. We have hit bugs and restrictions along the way but I accept this is a characteristic of using new, rapidly changing software.
Layout
We have a complicated AWS infrastructure with multiple VPC's each with multiple subnets. Key to easily managing this was to define a flexible taxonomy that encompasses region, environment, service and owner which we can use to organise our infrastructure code (both terraform and puppet).
Modules
Next step was to create a single git repository to store our terraform modules. Our top level dir structure for the modules looks like this:
tree -L 1 .
Result:
├── README.md
├── aws-asg
├── aws-ec2
├── aws-elb
├── aws-rds
├── aws-sg
├── aws-vpc
└── templates
Each one sets some sane defaults but exposes them as variables that can be overwritten by our "glue".
Glue
We have a second repository with our glue that makes use of the modules mentioned above. It is laid out in line with our taxonomy document:
.
├── README.md
├── clientA
│ ├── eu-west-1
│ │ └── dev
│ └── us-east-1
│ └── dev
├── clientB
│ ├── eu-west-1
│ │ ├── dev
│ │ ├── ec2-keys.tf
│ │ ├── prod
│ │ └── terraform.tfstate
│ ├── iam.tf
│ ├── terraform.tfstate
│ └── terraform.tfstate.backup
└── clientC
├── eu-west-1
│ ├── aws.tf
│ ├── dev
│ ├── iam-roles.tf
│ ├── ec2-keys.tf
│ ├── prod
│ ├── stg
│ └── terraform.tfstate
└── iam.tf
Inside the client level we have AWS account specific .tf files that provision global resources (like IAM roles); next is region level with EC2 SSH public keys; Finally in our environment (dev, stg, prod etc) are our VPC setups, instance creation and peering connections etc. are stored.
Side Note: As you can see I'm going against my own advice above keeping terraform.tfstate in git. This is a temporary measure until I move to S3 but suits me as I'm currently the only developer.
Next Steps
This is still a manual process and not in Jenkins yet but we're porting a rather large, complicated infrastructure and so far so good. Like I said, few bugs but going well!
Edit 2 - Changes
It's been almost a year since I wrote this initial answer and the state of both Terraform and myself have changed significantly. I am now at a new position using Terraform to manage an Azure cluster and Terraform is now v0.10.7.
State
People have repeatedly told me state should not go in Git - and they are correct. We used this as an interim measure with a two person team that relied on developer communication and discipline. With a larger, distributed team we are now fully leveraging remote state in S3 with locking provided by DynamoDB. Ideally this will be migrated to consul now it is v1.0 to cut cross cloud providers.
Modules
Previously we created and used internal modules. This is still the case but with the advent and growth of the Terraform registry we try to use these as at least a base.
File structure
The new position has a much simpler taxonomy with only two infx environments - dev and prod. Each has their own variables and outputs, reusing our modules created above. The remote_state provider also helps in sharing outputs of created resources between environments. Our scenario is subdomains in different Azure resource groups to a globally managed TLD.
├── main.tf
├── dev
│ ├── main.tf
│ ├── output.tf
│ └── variables.tf
└── prod
├── main.tf
├── output.tf
└── variables.tf
Planning
Again with extra challenges of a distributed team, we now always save our output of the terraform plan command. We can inspect and know what will be run without the risk of some changes between the plan and apply stage (although locking helps with this). Remember to delete this plan file as it could potentially contain plain text "secret" variables.
Overall we are very happy with Terraform and continue to learn and improve with the new features added.
We use Terraform heavily and our recommended setup is as follows:
File layout
We highly recommend storing the Terraform code for each of your environments (e.g. stage, prod, qa) in separate sets of templates (and therefore, separate .tfstate files). This is important so that your separate environments are actually isolated from each other while making changes. Otherwise, while messing around with some code in staging, it's too easy to blow up something in prod too. See Terraform, VPC, and why you want a tfstate file per env for a colorful discussion of why.
Therefore, our typical file layout looks like this:
stage
└ main.tf
└ vars.tf
└ outputs.tf
prod
└ main.tf
└ vars.tf
└ outputs.tf
global
└ main.tf
└ vars.tf
└ outputs.tf
All the Terraform code for the stage VPC goes into the stage folder, all the code for the prod VPC goes into the prod folder, and all the code that lives outside of a VPC (e.g. IAM users, SNS topics, S3 buckets) goes into the global folder.
Note that, by convention, we typically break our Terraform code down into 3 files:
vars.tf: Input variables.
outputs.tf: Output variables.
main.tf: The actual resources.
Modules
Typically, we define our infrastructure in two folders:
infrastructure-modules: This folder contains small, reusable, versioned modules. Think of each module as a blueprint for how to create a single piece of infrastructure, such as a VPC or a database.
infrastructure-live: This folder contains the actual live, running infrastructure, which it creates by combining the modules in infrastructure-modules. Think of the code in this folder as the actual houses you built from your blueprints.
A Terraform module is just any set of Terraform templates in a folder. For example, we might have a folder called vpc in infrastructure-modules that defines all the route tables, subnets, gateways, ACLs, etc for a single VPC:
infrastructure-modules
└ vpc
└ main.tf
└ vars.tf
└ outputs.tf
We can then use that module in infrastructure-live/stage and infrastructure-live/prod to create the stage and prod VPCs. For example, here is what infrastructure-live/stage/main.tf might look like:
module "stage_vpc" {
source = "git::git#github.com:gruntwork-io/module-vpc.git//modules/vpc-app?ref=v0.0.4"
vpc_name = "stage"
aws_region = "us-east-1"
num_nat_gateways = 3
cidr_block = "10.2.0.0/18"
}
To use a module, you use the module resource and point its source field to either a local path on your hard drive (e.g. source = "../infrastructure-modules/vpc") or, as in the example above, a Git URL (see module sources). The advantage of the Git URL is that we can specify a specific git sha1 or tag (ref=v0.0.4). Now, not only do we define our infrastructure as a bunch of small modules, but we can version those modules and carefully update or rollback as needed.
We've created a number of reusable, tested, and documented Infrastructure Packages for creating VPCs, Docker clusters, databases, and so on, and under the hood, most of them are just versioned Terraform modules.
State
When you use Terraform to create resources (e.g. EC2 instances, databases, VPCs), it records information on what it created in a .tfstate file. To make changes to those resources, everyone on your team needs access to this same .tfstate file, but you should NOT check it into Git (see here for an explanation why).
Instead, we recommend storing .tfstate files in S3 by enabling Terraform Remote State, which will automatically push/pull the latest files every time you run Terraform. Make sure to enable versioning in your S3 bucket so you can roll back to older .tfstate files in case you somehow corrupt the latest version. However, an important note: Terraform doesn't provide locking. So if two team members run terraform apply at the same time on the same .tfstate file, they may end up overwriting each other's changes.
Edit 2020: Terraform now supports locking: https://www.terraform.io/docs/state/locking.html
To solve this problem, we created an open source tool called Terragrunt, which is a thin wrapper for Terraform that uses Amazon DynamoDB to provide locking (which should be completely free for most teams). Check out Add Automatic Remote State Locking and Configuration to Terraform with Terragrunt for more info.
Further reading
We've just started a series of blog posts called A Comprehensive Guide to Terraform that describes in detail all the best practices we've learned for using Terraform in the real world.
Update: the Comprehensive Guide to Terraform blog post series got so popular that we expanded it into a book called Terraform: Up & Running!
Previously remote config allowed this but now has been replaced by "backends", so terraform remote is not anymore available.
terraform remote config -backend-config="bucket=<s3_bucket_to_store_tfstate>" -backend-config="key=terraform.tfstate" -backend=s3
terraform remote pull
terraform apply
terraform remote push
See the docs for details.
Covered in more depth by #Yevgeny Brikman but specifically answering the OP's questions:
What's the best practice for actually managing the terraform files and state?
Use git for TF files. But don't check State files in (i.e. tfstate). Instead use Terragrunt for sync / locking of state files to S3.
but do I commit tfstate as well?
No.
Should that reside somewhere like S3?
Yes
I know there’s a lot of answers here but my approach is quite different.
⁃ Modules
⁃ Environment management
⁃ Separation of duties
Modules
Create modules for logical collections of resources.
Example: If your goal is to deploy an API, which requires a DB, HA VMs, autoscaling, DNS, PubSub and object storage then all of these resources should be templated in a single module.
Avoid creating modules that utilise a single resource. This can and has been done and a lot of the modules in the registry do this but it’s a practice that helps with resource accessibility rather than infrastructure orchestration.
Example: A module for AWS EC2 helps the user access the EC2 by making complex configurations more simple to invoke but a module like the example in 1. assists the user when orchestrating application, component or service driven infrastructure.
Avoid resource declarations in your workspace. This is more about keeping your code tidy and organised. As modules are easily versioned, you have more control over your releases.
Environment management
IaC has made SDLC process relevant to infrastructure management and it’s not normal to expect to have development infrastructure as well as development application environments.
Don’t use folders to manage your IaC environments. This leads to drift as there’s no common template for your infrastructure.
Do use a single workspace and variables to control environment specifications.
Example: Write your modules so that when you change the environment variable (var.stage is popular) the plan alters to fit your requirements. Typically the environments should vary as little as possible with quantity, exposure and capacity usually being the variable configurations. Dev might deploy 1 VM with 1 core and 1GB RAM in private topology but production may be 3 VMs with 2 cores and 4GB RAM with additional public topology. You can of course have more variation: dev may run database process on the same server as the application to save cost but production may have a dedicated DB instance. All of this can be managed by changing a single variable, ternary statements and interpolation.
Separation of duties
If you’re in a small organisation or running personal infrastructure this doesn’t really apply but it will help you manage your operations.
Break down your infrastructure by duties, responsibilities or teams.
Example: Central IT control underlying shared services (virtual networks, subnets, public IP addresses, log groups, governance resources, multi tenanted DBs, shared keys, etc.) whilst the API team only control the resources needed for their service (VMs, LBs, PubSub etc) and consume Central ITs services through data source and remote state lookups.
Govern team access.
Example: Central IT may have admin rights but the API team only have access to a restricted set of public cloud APIs.
This also helps with release concerns as you will find some resources rarely change whilst others change all the time. Separation removes risk and complexity.
This strategy draws parallels with AWS’ multi account strategy. Have a read for more info.
CI/CD
This is a topic of its own but Terraform works very well within a good pipeline. The most common error here is to treat CI as a silver bullet. Technically Terraform should only be provisioning infrastructure during stages of an assembly pipeline. This would be separate to what happens in CI stages where one typically validates and tests the templates.
N.B. Written on mobile so please excuse any errors.
Before answers have been very solid and informative, I will try to add
my 2 cents here
Common recommendations for structuring code
It is easier and faster to work with smaller number of resources:
Cmdsterraform plan and terraform apply both make cloud API calls to verify the status of resources.
If you have your entire infrastructure in a single composition this can take many minutes (even if you have several files in the same folder).
Blast radius is smaller with fewer resources:
Insulating unrelated resources from each other by placing them in separate compositions (folders) reduces the risk if something goes wrong.
Start your project using remote state:
Your laptop is no place for your infrastructure source of truth.
Managing a tfstate file in git is a nightmare.
Later when infrastructure layers starts to grow in any direction (number of dependencies or resources).
example module: https://github.com/cloudposse/terraform-aws-tfstate-backend
ref tool: https://github.com/camptocamp/terraboard
Try to practice a consistent structure and naming convention:
Like procedural code, Terraform code should be written for people to read first, consistency will help when changes happen six months from now.
It is possible to move resources in Terraform state file but it may be harder to do if you have inconsistent structure and naming.
Keep resource modules as plain as possible.
Don't hard-code values which can be passed as variables or discovered using data sources.
Use data sources and terraform_remote_state specifically as a glue between infrastructure modules within composition.
(ref article: https://www.terraform-best-practices.com/code-structure)
Example:
It is easier and faster to work with smaller number of resources so
below we present a recommended code layout.
NOTE: just as reference not to be strictly follow since each project has it's own specific characteristics
.
├── 1_tf-backend #remote AWS S3 + Dynamo Lock tfstate
│ ├── main.tf
│ ├── ...
├── 2_secrets
│ ├── main.tf
│ ├── ...
├── 3_identities
│ ├── account.tf
│ ├── roles.tf
│ ├── group.tf
│ ├── users.tf
│ ├── ...
├── 4_security
│ ├── awscloudtrail.tf
│ ├── awsconfig.tf
│ ├── awsinspector.tf
│ ├── awsguarduty.tf
│ ├── awswaf.tf
│ └── ...
├── 5_network
│ ├── account.tf
│ ├── dns_remote_zone_auth.tf
│ ├── dns.tf
│ ├── network.tf
│ ├── network_vpc_peering_dev.tf
│ ├── ...
├── 6_notifications
│ ├── ...
├── 7_containers
│ ├── account.tf
│ ├── container_registry.tf
│ ├── ...
├── config
│ ├── backend.config
│ └── main.config
└── readme.md
I believe there are few best practices need to follow while using terraform for orchestrating the infrastructure
Don't write the same code again ( Reusability)
Keep environment configuration separate to maintain it easily.
Use remote backend s3(encrypted) and dynamo DB to handle the concurrency locking
Create a module and use that module in main infrastructure multiple time, its like a reusable function which can be called multiple time by passing different parameter.
Handle multiple environments
Most of the time recommended way is to use terraform 'workspace' to handle the multiple environments but I believe the usage of workspace could vary based on way of work in an organization.
Other is storing the Terraform code for each of your environments (e.g. stage, prod, QA) to separate the environment states. However, in this case we are just copying the same code at many places.
├── main.tf
├── dev
│ ├── main.tf
│ ├── output.tf
│ └── variables.tf
└── prod
├── main.tf
├── output.tf
└── variables.tf
I followed some different approach to handle and avoid the duplication of the same terraform code by keeping in each environment folder since I believe most of the time all environment would be 90% same.
├── deployment
│ ├── 01-network.tf
│ ├── 02-ecs_cluster.tf
│ ├── 03-ecs_service.tf
│ ├── 04-eks_infra.tf
│ ├── 05-db_infra.tf
│ ├── 06-codebuild-k8s.tf
│ ├── 07-aws-secret.tf
│ ├── backend.tf
│ ├── provider.tf
│ └── variables.tf
├── env
│ ├── dev
│ │ ├── dev.backend.tfvar
│ │ └── dev.variables.tfvar
│ └── prod
│ ├── prod.backend.tfvar
│ └── prod.variables.tfvar
├── modules
│ └── aws
│ ├── compute
│ │ ├── alb_loadbalancer
│ │ ├── alb_target_grp
│ │ ├── ecs_cluster
│ │ ├── ecs_service
│ │ └── launch_configuration
│ ├── database
│ │ ├── db_main
│ │ ├── db_option_group
│ │ ├── db_parameter_group
│ │ └── db_subnet_group
│ ├── developertools
│ ├── network
│ │ ├── internet_gateway
│ │ ├── nat_gateway
│ │ ├── route_table
│ │ ├── security_group
│ │ ├── subnet
│ │ ├── vpc
│ └── security
│ ├── iam_role
│ └── secret-manager
└── templates
Configuration related to environments
Keep environment related configuration and parameters separate in a variable file and pass that value to configure the infrastructure. e.g as below
dev.backend.tfvar
region = "ap-southeast-2"
bucket = "dev-samplebackendterraform"
key = "dev/state.tfstate"
dynamo_db_lock = "dev-terraform-state-lock"
dev.variable.tfvar
environment = "dev"
vpc_name = "demo"
vpc_cidr_block = "10.20.0.0/19"
private_subnet_1a_cidr_block = "10.20.0.0/21"
private_subnet_1b_cidr_block = "10.20.8.0/21"
public_subnet_1a_cidr_block = "10.20.16.0/21"
public_subnet_1b_cidr_block = "10.20.24.0/21"
Conditional skipping of infrastructure part
Create a configuration in env specific variable file and based on that variable decide to create or skipping that part. In this way based on need the specific part of the infrastructure can be skipped.
variable vpc_create {
default = "true"
}
module "vpc" {
source = "../modules/aws/network/vpc"
enable = "${var.vpc_create}"
vpc_cidr_block = "${var.vpc_cidr_block}"
name = "${var.vpc_name}"
}
resource "aws_vpc" "vpc" {
count = "${var.enable == "true" ? 1 : 0}"
cidr_block = "${var.vpc_cidr_block}"
enable_dns_support = "true"
enable_dns_hostnames = "true"
}
below command is required to initialize and execute the infra changes for each environment, cd to the required environment folder.
terraform init -var-file=dev.variables.tfvar -backend-config=dev.backend.tfvar ../../deployment/
terraform apply -var-file=dev.variables.tfvar ../../deployment
For reference: https://github.com/mattyait/devops_terraform
I don't like the idea of subfolders because this will result in different sources per environment and this tends to drift.
The better approach is to have a single stack for all environments (lets say dev, preprod and prod). To work on a single environment use terraform workspace.
terraform workspace new dev
This creates a new workspace. This includs a dedicated state file and the variable terraform.workspace you can use in your code.
resource "aws_s3_bucket" "bucket" {
bucket = "my-tf-test-bucket-${terraform.workspace}"
}
In this way you will get buckets called
my-tf-test-bucket-dev
my-tf-test-bucket-preprod
my-tf-test-bucket-prod
after applying to the workspaces above (use terraform workspace select <WORKSPACE> to change environments).
To make the code even multi-region-proof do it like this:
data "aws_region" "current" {}
resource "aws_s3_bucket" "bucket" {
bucket = "my-tf-test-bucket-${data.aws_region.current.name}-${terraform.workspace}"
}
to get (for us-east-1 region)
my-tf-test-bucket-us-east-1-dev
my-tf-test-bucket-us-east-1-preprod
my-tf-test-bucket-us-east-1-prod
Some Terraform Best Practices to Follow:
Avoid hard coding:
Sometimes developers manually created resources directly. You need to mark these resource and use terraform import to include them in codes.
A sample:
account_number=“123456789012"
account_alias="mycompany"
Run Terraform from a docker container:
Terraform releases an official Docker container that allows you to easily control which version you can run.
It is recommended to run the Terraform Docker container when you set your build job in the CI/CD pipeline.
TERRAFORM_IMAGE=hashicorp/terraform:0.11.7
TERRAFORM_CMD="docker run -ti --rm -w /app -v ${HOME}/.aws:/root/.aws -v ${HOME}/.ssh:/root/.ssh -v `pwd`:/app $TERRAFORM_IMAGE"
For more, please refer to my blog: https://medium.com/tech-darwinbox/how-darwinbox-manages-infrastructure-at-scale-with-terraform-371e2c5f04d3
I'd like to contribute to this thread.
This will most likely be AWS S3+DynamoDB unless you are using Terraform Cloud.
Separate infrastructure (network + RBAC) of production and non-prod backends.
Plan to disable access to state files (network access and RBAC) from outside of a designated network (e.g. deployment agent pool).
Do not keep Terraform backend infrastructure with the run-time environment. Use separate
account.
Enable object versioning on your Terraform backends to avoid losing changes and state-files, and in order to maintain Terraform state history.
In some special cases, manual access to Terraform state files will be required. Things like refactoring, breaking changes or fixing defects will require running Terraform state operations by operations personnel. For such occasions, plan extraordinary controlled access to the Terraform state using bastion host, VPN etc.
Check a longer best practices blog that covers this in details including guidelines for CI/CD pipelines.
If you are still looking for the better solution, take a look at workspaces which can replace maintaining different environment folder structure can have workspace specific variables.
As Yevgeniy Brikman mentioned it's better to have a modules structure.
Use terraform cloud for manage and save states, together with advises above.