I am trying to figure out the best practice folder structure to organize my vault variables.
Currently it looks like this:
.
├── group_vars/
│ ├── group1.yml
│ └── group2.yml
├── host_vars/
│ ├── host1.yml
│ └── host2.yml
├── roles/
│ └── .../
└── vault/
├── enc-file1.yml
└── enc-file2.yml
However, this way I always have to use include_vars inside my role to source a specific encrypted file.
Is there any naming convention and folder structure I can apply that Ansible will automatically source the correct vaulted variable just as it does with host_vars and group_vars?
I had something like this in mind:
.
└── group_vars/
├── group1/
│ ├── main.yml
│ └── vault.yml
└── group2/
├── main.yml
└── vault.yml
Is there anything I can do, so I do not have to explicitly include vault variables?
Is there anything I can do, so I do not have to explicitly include vault variables?
In recent Ansible versions (since 2.3) you don't need to include separate files for vault-protected variables. Instead, you can encrypt individual variables in regular vars-files - see Single Encrypted Variable.
Is there any naming convention and folder structure [] that Ansible will automatically source the correct vaulted variable []?
No, there is no convention nor automatic mechanism.
#cytopia, what you have in mind works.
You can split encrypted/clear inventory variables in separate files. Ansible will sort out what is crypted and what is not.
I use the following layout and had no problems with it so far (1.9.x - 2.2.1.0):
group_vars/
all/
clear
secret
group1/
clear
secret
group2/
clear
secret
Related
I was wondering if anyone could advise on how I can manage multi environment variables using ansible. Imagine i have development and staging. I have the following structure:
.
├── inventories
│ ├── development
│ │ ├── group_vars
│ │ │ └── all.yml
│ │ └── hosts
│ └── staging
│ ├─── group_vars
│ │ └── all.yml
│ │ └── hosts
├── roles
└── test.yml
In each all.yml file, i have one variable like this:
vars:
name:"development"
I was wondering how is the best approach to load the values depending on a parameter to set for example on command line (-i env=development).
In test.yml, I have this:
- name: test playbook
hosts: localhost
connection: local
tasks:
- name:
debug:
var: name
When I execute, I get this:
ansible-playbook -i inventories/development/ test.yml -vvv
....
ok: [localhost] => {
"name": "VARIABLE IS NOT DEFINED!: 'name' is undefined"
}
I'm kind of new to ansible and I've found thousands of different ways and I was wondering what is the best approach? Because I'm kind of lost :( :(
Regards,
We have something similar to your needs on our project, I guess.
Organized in below maner:
.
group_vars folder
├── all.yml # variables common for all envs (creds for example)
├── development.yml # variables for development env
└── staging.yml # variables for staging env
|
invenory_file.yml # file which stoers your managed hosts list devided by groups
├── [development]
│ ├── dev.host1.name # development host#1 with vars applied from development.yml
│ ├── dev.host2.name # development host#2 with vars applied from development.yml
│ └── dev.host3.name # development host#3 with vars applied from development.yml
├── [staging]
│ ├── staging.host1.name # staging host#1 with vars applied from staging.yml
│ ├── staging.host2.name # staging host#2 with vars applied from staging.yml
│ └── staging.host3.name # staging host#3 with vars applied from staging.yml
|
roles
Main idea that you have separate files with vars for different envs, and you have your hosts organized in environments groups in inventory file. So variables from files in group_vars will applied to correct hosts grouped in Inventory.
Here is usuful link to check on how Ansible groups are organized:
https://docs.ansible.com/ansible/latest/user_guide/intro_inventory.html
I have installed few roles using ansible-galaxy and have included the following line in my .ansible.cfg:
roles_path = /Users/idyllic/ansible/roles
Now, for a specific project I have defined a requirement.yml where I have listed all the ansible roles that need to be installed (and they are all stored in /Users/idyllic/ansible/roles).
ansible-galaxy install -r requirement.yml
Now, even project has a roles directory so is it wise to download the roles in the project roles directory instead of the one defining it on directory define by (.ansible_cfg)
What is the best practice for storing the roles in ansible (version 2.1.1.0)?
i have this line in my project root folder
roles_path = ./roles:..:/etc/ansible/roles
and then the requirements.yml installation places then within the project.
This ensures that if a role is part of two projects, its not updated by accident
project tree structure:
.
├── Makefile
├── Vagrantfile
├── ansible.cfg
├── defaults
│ └── main.yml
├── elasticsearch.yml
├── handlers
│ └── main.yml
├── meta
│ └── main.yml
├── requirements.yml
├── tasks
│ ├── main.yml
├── templates
├── roles # this is where the requirements.yml go
└── vars
└── main.yml
I'm using the project structure from https://github.com/enginyoyen/ansible-best-practises.
It contains a bash script which downloads and updates in a dedicated directory Ansible external roles which are declared in a requirements file:
I find it very easy and very useful to create many different and separated project structures.
The Ansible best practices documentation recommends to separate inventories:
inventories/
production/
hosts.ini # inventory file for production servers
group_vars/
group1 # here we assign variables to particular groups
group2 # ""
host_vars/
hostname1 # if systems need specific variables, put them here
hostname2 # ""
staging/
hosts.ini # inventory file for staging environment
group_vars/
group1 # here we assign variables to particular groups
group2 # ""
host_vars/
stagehost1 # if systems need specific variables, put them here
stagehost2 # ""
My staging and production environments are structured in the same way. I have in both environments the same groups. And it turns out that I have also the same group_vars for the same groups. This means redundancy I would like to wipe out.
Is there a way to share some group_vars between different inventories?
As a work-around I started to put shared group_vars into the roles.
my_var:
my_group:
- { var1: 1, var2: 2 }
This makes it possible to iterate over some vars by intersecting the groups of a host with the defined var:
with_items: "{{group_names | intersect(my_var.keys())}}"
But this is a bit complicate to understand and I think roles should not know anything about groups.
I would like to separate most of the inventories but share some of the group_vars in an easy to understand way. Is it possible to merge global group_vars with inventory specific group_vars?
I scrapped the idea of following Ansible's recommendation. Now one year later, I am convinced that Ansible's recommendation is not useful for my requirements. Instead I think it is important to share as much as possible among different stages.
Now I put all inventories in the same directory:
production.ini
reference.ini
And I take care that each inventory defines a group including all hosts with the name of the stage.
The file production.ini has the group production:
[production:children]
all_production_hosts
And the file reference.ini has the group reference:
[reference:children]
all_reference_hosts
I have just one group_vars directory in which I define a file for every staging group:
group_vars/production.yml
group_vars/reference.yml
And each file defines a stage variable. The file production.yml defines this:
---
stage: production
And the file reference.yml defines that:
---
stage: reference
This makes it possible to share everything else between production and reference. But the hosts are completely different. By using the right inventory the playbook runs either on production or on reference hosts:
ansible-playbook -i production.ini site.yml
ansible-playbook -i reference.ini site.yml
If it is necessary for the site.yml or the roles to behave slightly different in the production and reference environment, they can use conditions using the stage variable. But I try to avoid even that. Because it is better to move all differences into equivalent definitions in the staging files production.yml and reference.yml.
For example, if the group_vars/all.yml defines some users:
users:
- alice
- bob
- mallory
And I want to create the users in both environments, but I want to exclude mallory from the production environment, I can define a new group called effective_users. In the reference.yml it is identical to the users list:
effective_users: >-
{{ users }}
But in the production.yml I can exclude mallory:
effective_users: >-
{{ users | difference(['mallory']) }}
The playbook or the roles do not need to distinguish between the two stages, they can simply use the group effective_users. The group contains automatically the right list of users simply by selecting the inventory.
The simple option here (and what we do) is simply symlink generic group vars files around.
For instance we might have a generic role for something like NGINX and then a few concrete use cases for that role. In this case we create a group vars file that uses the NGINX role for each concrete use case and then simply symlink those group vars files into the appropriate folders.
Our project folder structure then might look something like this (drastically simplified):
.
├── inventories
│ ├── bar-dev
│ │ ├── group_vars
│ │ │ ├── bar.yml -> ../../shared/bar.yml
│ │ │ └── dev.yml -> ../../shared/dev.yml
│ │ └── inventory
│ ├── bar-prod
│ │ ├── group_vars
│ │ │ ├── bar.yml -> ../../shared/bar.yml
│ │ │ └── prod.yml -> ../../shared/prod.yml
│ │ └── inventory
│ ├── bar-test
│ │ ├── group_vars
│ │ │ ├── bar.yml -> ../../shared/bar.yml
│ │ │ └── test.yml -> ../../shared/test.yml
│ │ └── inventory
│ ├── foo-dev
│ │ ├── group_vars
│ │ │ ├── dev.yml -> ../../shared/dev.yml
│ │ │ └── foo.yml -> ../../shared/foo.yml
│ │ └── inventory
│ ├── foo-prod
│ │ ├── group_vars
│ │ │ ├── foo.yml -> ../../shared/foo.yml
│ │ │ └── prod.yml -> ../../shared/prod.yml
│ │ └── inventory
│ ├── foo-test
│ │ ├── group_vars
│ │ │ ├── foo.yml -> ../../shared/foo.yml
│ │ │ └── test.yml -> ../../shared/test.yml
│ │ └── inventory
│ └── shared
│ ├── bar.yml
│ ├── dev.yml
│ ├── foo.yml
│ ├── prod.yml
│ └── test.yml
└── roles
└── nginx
├── defaults
│ └── main.yml
├── meta
│ └── main.yml
├── tasks
│ └── main.yml
└── templates
└── main.yml
Now our inventory files can have the hosts use these shared group vars simply by putting the hosts in the correct groups.
You can place group_vars in playbook directory as well. More info.
Ansible will pick them up for all inventories.
I'm coming from a Puppet background using Vagrant and have some trouble making sense of Ansible and its differences.
My Puppet structure looked like this:
puppet
├── servers
│ └── Backend
│ └── Vagrantfile
└── src
├── manifests
│ └── nodes
│ └── development
│ └── backend.pp
└── modules
└── mysql
Setup was simple, cd to the Vagrantfile and fire up the VM with Vagrant.
Now this is my first draft of an Ansible folder structure:
ansible
├── servers
│ └── Backend
│ ├── Vagrantfile
│ └── ansible.cfg
└── sources
├── backend.yml
├── site.yml
├── inventories
│ └── development
│ ├── group_vars
│ │ ├── all
│ │ └── backend
│ └── hosts
├── playbooks
└── roles
└── mysql
Following questions now:
Is this best practise for Ansible or too close to Puppet?
Is it correct to treat backend.yml like a Puppet node manifest?
Where should I put site.yml and backend.yml? This example has them in the main directory while here it's in the 'plays' directory. What's the difference?
I think my group_vars in group_vars/backend aren't being used correctly, how do I access them?
Sources:
http://leucos.github.io/ansible-files-layout/
https://github.com/ansible/ansible-examples
https://github.com/enginyoyen/ansible-best-practises
In my case, I use following structures according environment complexity (check directory-layout) :
Simple environment
I use this structure when there is one environment or when I use playbooks in provision mode
ansible
├── inventory
│ ├── hosts
│ └── group_vars
│ └── my_group.yml
├── roles
│ └── mysql
├── ansible.cfg
├── README.md
├── playbook1.yml
└── playbook2.yml
In ansible.cfg, I use variable inventory = ./inventory in [default] in order to avoid setting inventory path with commands ansible-*.
Medium/Complex environment
I use this structure when there are more than one environment
ansible
├── inventories
│ ├── production
│ │ ├── hosts
│ │ └── group_vars
│ │ └── my_group.yml
│ └── development
│ ├── hosts
│ └── group_vars
│ └── my_group.yml
├── playbooks
│ ├── playbook1
│ │ ├── group_vars
│ │ │ └── my_group.yml
│ │ ├── roles
│ │ │ └── mysql
│ │ ├── README.md
│ │ └── site.yml
│ ...
├── README.md
└── ansible.cfg
In this case, there is a folder for each environments in ./inventories.
I prefer also to use a specific folder for each playbooks in order to be able to use easily a folder group_vars at playbook level as defined in variable precedence section. As environment become more complex, there are much more variables. A group_vars (and host_vars) in playbooks allows to defines common variables for all environment which makes there are less inventories variables.
Ultimate level environment
I already used Ansible to adresse systems with more than 5000 servers, here under some tips for adressing more complex environments :
Split inventory file
Use multiple files to define your inventory servers instead of a single hosts file. In this case hosts file contains only server's names, and other files contains groups with different perspectives :
└── production
├── hosts
├── middleware
└── trigram
middleware: Groups with mapping to used middlewares or other stuf. I use this file to map, for example, servers to tomcat, java, postgresql, etc. And I use it, for example, with playbooks that deploys monitoring agents : How to retrieve metrics, logs from tomcat, java, postgresql, etc.
trigram: On my project, I usually use codes with a fixed length (3 or 4) to identify my business components (ex. : 'STK' for stock management) then I create a group file to map a business component to my servers (which servers are used to deploy 'STK')
When you create a new playbook, choose your perspective to address different environments.
Caution : I think ansible load files with alphabetical name order, you can't define a group that refers a group not yet loaded.
Use folders for group_vars
In group_vars, instead of using files, you can use a folder with subfiles :
└── production
└── group_vars
└── my_group
├── vars1.yml
└── vars2.yml
This is usefull to split huge files or if you have tools that generate variables, in this case you have vars1.yml under git and vars2.yml is generated
Split git repo
When you are using ansible for huge system, there is a lot of commits and a question comes up often : How to split my huge git repo ?
I my case, I use one git repo for each folder in ./inventories with differents access rules. And a git repo for each folder in ./playbooks also with different access rules.
You found several examples of the recommended layout, but not the official one. That should hopefully answer many of your questions, but I'll try to address them here as well.
Is this best practise for Ansible or too close to Puppet?
It's definitely not the recommend layout for Ansible. In the best practices layout, there is no servers or sources - the things inside those all live at the top level (what does "servers" mean, anyways?).
Is it correct to treat backend.yml like a Puppet node manifest?
I'm not familiar with Puppet, and so can't answer this question.
Where should I put site.yml and backend.yml? This example has them in the main directory while here it's in the 'plays' directory. What's the difference?
The official recommendation has all playbooks splattered across the root directory. However, this gets a bit messy, so some people put them in a subdirectory (in your examples, plays). This works reasonably well, but you'll need to adjust the paths in your playbooks accordingly.
I think my group_vars in group_vars/backend aren't being used correctly, how do I access them?
You shouldn't put them in a subdirectory of the inventory folder, as they're not part of inventory. There are a number of places to define variables, and you should read the documentation to learn what they are and when you should use them, but group_vars should be at the root of the directory with everything else.
I need to set up quite few host specific variables. Up until now I was adding them to the hosts file but the list of variables is getting ridiculously long.
What I'd like to do is to tell ansible to load a host specific file that will contain all the necessary variables. Let's say my hosts are:
[webservers]
hydrogen
helium
lithium
And that variables for them are structured like this:
.
├── variables
│ ├── hydrogen.yml
│ ├── helium.yml
│ └── lithium.yml
└── blahblah
How can I load those variables for each host automatically?
Ansible has the concept of group and host vars. No need to manually tell it to load vars from a file, it's all built in. Simply create files relative to your playbook (or inventory file) in the folders group_vars and/or host_vars:
.
├── group_vars
│ └── webservers.yml
├── host_vars
│ ├── hydrogen.yml
│ ├── helium.yml
│ └── lithium.yml
└── playbook.yml
Also see Best Practices: Group And Host Variables