I have installed few roles using ansible-galaxy and have included the following line in my .ansible.cfg:
roles_path = /Users/idyllic/ansible/roles
Now, for a specific project I have defined a requirement.yml where I have listed all the ansible roles that need to be installed (and they are all stored in /Users/idyllic/ansible/roles).
ansible-galaxy install -r requirement.yml
Now, even project has a roles directory so is it wise to download the roles in the project roles directory instead of the one defining it on directory define by (.ansible_cfg)
What is the best practice for storing the roles in ansible (version 2.1.1.0)?
i have this line in my project root folder
roles_path = ./roles:..:/etc/ansible/roles
and then the requirements.yml installation places then within the project.
This ensures that if a role is part of two projects, its not updated by accident
project tree structure:
.
├── Makefile
├── Vagrantfile
├── ansible.cfg
├── defaults
│ └── main.yml
├── elasticsearch.yml
├── handlers
│ └── main.yml
├── meta
│ └── main.yml
├── requirements.yml
├── tasks
│ ├── main.yml
├── templates
├── roles # this is where the requirements.yml go
└── vars
└── main.yml
I'm using the project structure from https://github.com/enginyoyen/ansible-best-practises.
It contains a bash script which downloads and updates in a dedicated directory Ansible external roles which are declared in a requirements file:
I find it very easy and very useful to create many different and separated project structures.
Related
Lets say I have numerous ansible playbooks for various personal projects. They all do wildly different things, but they have a few repeated roles in common.
For example one of the roles clones my personal dot-files repository so I can have my personal .bash_profile, .vimrc, etc... on every server.
All my playbooks are in separate git repositories. Right now each of them duplicates the "dotfiles" role separately.
Is there a good way I can move that re-usable role to a separate repository and have each of the invidiual playbooks import that as needed?
Prerequisite
Have each role in its own git repository.
Typical project structure
This can be challenged. that's the one I like to use on different of my projects:
├── inventories
│ ├── dev
│ │ ├── group_vars/
│ │ └── hosts.ini
│ └── prod
│ ├── group_vars/
│ └── hosts.ini
├── group_vars/
├── host_vars/
├── files/
├── templates/
├── roles
│ ├── localy_versionned_role1/
│ ├── localy_versionned_role2/
│ ├── requirements.yml
│ ├── .gitignore
├── ansible.cfg
├── README.md
├── some_playbook.yml
├── other_playbook.yml
roles/.gitignore
# Ignore everything in dir...
/*
# ... but current file...
!.gitignore
# ... external role requirement file
!requirements.yml
# ... and configured custom/local roles
!localy_versionned_role*/
roles/requirements.yml
# Classic galaxy role
- src: galaxy_user.role_name
# Git available role
- src: git#git.service.com:path/to/repo.git
scm: git
version: master
name: local_role_name
You only need to list "top" roles, dependencies (listed in meta/main.yml of the role) will be downloaded also.
Ansible.cfg
We make sure that roles are searched and downloaded in our local folder
roles_path = roles
Workflow to deploy from your project:
Clone your project repository
Download the external roles:
ansible-galaxy install -r roles/requirements
Launch your playbook:
ansible-playbook -i inventories/dev some_playbook.yml
Going further
Downloading the roles as git workspaces
By default, ansible-galaxy downloads from git and removes the local repository structure (i.e. the .git directory). If you want to download the roles and keep working on them (change, commit, push...), you can keep the git structure with:
ansible-galaxy install -g -r roles/requirements
Note that this will write a meta/.galaxy_install_info file inside your role that git will see as new if you did not ignore already.
Updating roles after first install
Roles will not be updated with a new version (especially from galaxy) if they are already installed. To force updating use:
ansible-galaxy -f -r roles/requirements.yml
of course you do this also to switch to git workspace version (or reset to version given in requirements file)
ansible-galaxy -f -g -r roles/requirements.yml
I am trying to figure out the best practice folder structure to organize my vault variables.
Currently it looks like this:
.
├── group_vars/
│ ├── group1.yml
│ └── group2.yml
├── host_vars/
│ ├── host1.yml
│ └── host2.yml
├── roles/
│ └── .../
└── vault/
├── enc-file1.yml
└── enc-file2.yml
However, this way I always have to use include_vars inside my role to source a specific encrypted file.
Is there any naming convention and folder structure I can apply that Ansible will automatically source the correct vaulted variable just as it does with host_vars and group_vars?
I had something like this in mind:
.
└── group_vars/
├── group1/
│ ├── main.yml
│ └── vault.yml
└── group2/
├── main.yml
└── vault.yml
Is there anything I can do, so I do not have to explicitly include vault variables?
Is there anything I can do, so I do not have to explicitly include vault variables?
In recent Ansible versions (since 2.3) you don't need to include separate files for vault-protected variables. Instead, you can encrypt individual variables in regular vars-files - see Single Encrypted Variable.
Is there any naming convention and folder structure [] that Ansible will automatically source the correct vaulted variable []?
No, there is no convention nor automatic mechanism.
#cytopia, what you have in mind works.
You can split encrypted/clear inventory variables in separate files. Ansible will sort out what is crypted and what is not.
I use the following layout and had no problems with it so far (1.9.x - 2.2.1.0):
group_vars/
all/
clear
secret
group1/
clear
secret
group2/
clear
secret
Below is my directory structure for ansibel role called webserver
localhost roles # tree
.
├── readme.md
├── site.yml
└── webserver
├── files
│ ├── createswap.sh
│ └── nginxkeyadd.sh
├── handlers
│ └── main.yml
├── tasks
│ └── main.yml
├── templates
│ ├── helloworld.conf.j2
│ └── index.html.j2
└── vars
└── main.yml
my tasks/main.yml looks like
- name: Create swap file 50MB
script: /etc/ansible/roles/webserver/files/createswap.sh
- name: add GPG key for nginx
script: /etc/ansible/roles/webserver/files/nginxkeyadd.sh
- name: Install nginx on target
apt: name={{ item }} state=latest
with_items:
- rsync
- git
- nginx
in the task/main.yml im specifying absolute path to local scriptfile like
script: /etc/ansible/roles/webserver/files/nginxkeyadd.sh and script: /etc/ansible/roles/webserver/files/createswap.sh . The scripts don't have any ansible variables.
is it good practice in ansible ?
is it good practice in ansible ?
No. Excerpt from the docs:
Any copy, script, template or include tasks (in the role) can reference files in roles/x/{files,templates,tasks}/ (dir depends on task) without having to path them relatively or absolutely
Also using shell-scripts instead of native Ansible modules is an antipattern.
The Ansible best practices documentation recommends to separate inventories:
inventories/
production/
hosts.ini # inventory file for production servers
group_vars/
group1 # here we assign variables to particular groups
group2 # ""
host_vars/
hostname1 # if systems need specific variables, put them here
hostname2 # ""
staging/
hosts.ini # inventory file for staging environment
group_vars/
group1 # here we assign variables to particular groups
group2 # ""
host_vars/
stagehost1 # if systems need specific variables, put them here
stagehost2 # ""
My staging and production environments are structured in the same way. I have in both environments the same groups. And it turns out that I have also the same group_vars for the same groups. This means redundancy I would like to wipe out.
Is there a way to share some group_vars between different inventories?
As a work-around I started to put shared group_vars into the roles.
my_var:
my_group:
- { var1: 1, var2: 2 }
This makes it possible to iterate over some vars by intersecting the groups of a host with the defined var:
with_items: "{{group_names | intersect(my_var.keys())}}"
But this is a bit complicate to understand and I think roles should not know anything about groups.
I would like to separate most of the inventories but share some of the group_vars in an easy to understand way. Is it possible to merge global group_vars with inventory specific group_vars?
I scrapped the idea of following Ansible's recommendation. Now one year later, I am convinced that Ansible's recommendation is not useful for my requirements. Instead I think it is important to share as much as possible among different stages.
Now I put all inventories in the same directory:
production.ini
reference.ini
And I take care that each inventory defines a group including all hosts with the name of the stage.
The file production.ini has the group production:
[production:children]
all_production_hosts
And the file reference.ini has the group reference:
[reference:children]
all_reference_hosts
I have just one group_vars directory in which I define a file for every staging group:
group_vars/production.yml
group_vars/reference.yml
And each file defines a stage variable. The file production.yml defines this:
---
stage: production
And the file reference.yml defines that:
---
stage: reference
This makes it possible to share everything else between production and reference. But the hosts are completely different. By using the right inventory the playbook runs either on production or on reference hosts:
ansible-playbook -i production.ini site.yml
ansible-playbook -i reference.ini site.yml
If it is necessary for the site.yml or the roles to behave slightly different in the production and reference environment, they can use conditions using the stage variable. But I try to avoid even that. Because it is better to move all differences into equivalent definitions in the staging files production.yml and reference.yml.
For example, if the group_vars/all.yml defines some users:
users:
- alice
- bob
- mallory
And I want to create the users in both environments, but I want to exclude mallory from the production environment, I can define a new group called effective_users. In the reference.yml it is identical to the users list:
effective_users: >-
{{ users }}
But in the production.yml I can exclude mallory:
effective_users: >-
{{ users | difference(['mallory']) }}
The playbook or the roles do not need to distinguish between the two stages, they can simply use the group effective_users. The group contains automatically the right list of users simply by selecting the inventory.
The simple option here (and what we do) is simply symlink generic group vars files around.
For instance we might have a generic role for something like NGINX and then a few concrete use cases for that role. In this case we create a group vars file that uses the NGINX role for each concrete use case and then simply symlink those group vars files into the appropriate folders.
Our project folder structure then might look something like this (drastically simplified):
.
├── inventories
│ ├── bar-dev
│ │ ├── group_vars
│ │ │ ├── bar.yml -> ../../shared/bar.yml
│ │ │ └── dev.yml -> ../../shared/dev.yml
│ │ └── inventory
│ ├── bar-prod
│ │ ├── group_vars
│ │ │ ├── bar.yml -> ../../shared/bar.yml
│ │ │ └── prod.yml -> ../../shared/prod.yml
│ │ └── inventory
│ ├── bar-test
│ │ ├── group_vars
│ │ │ ├── bar.yml -> ../../shared/bar.yml
│ │ │ └── test.yml -> ../../shared/test.yml
│ │ └── inventory
│ ├── foo-dev
│ │ ├── group_vars
│ │ │ ├── dev.yml -> ../../shared/dev.yml
│ │ │ └── foo.yml -> ../../shared/foo.yml
│ │ └── inventory
│ ├── foo-prod
│ │ ├── group_vars
│ │ │ ├── foo.yml -> ../../shared/foo.yml
│ │ │ └── prod.yml -> ../../shared/prod.yml
│ │ └── inventory
│ ├── foo-test
│ │ ├── group_vars
│ │ │ ├── foo.yml -> ../../shared/foo.yml
│ │ │ └── test.yml -> ../../shared/test.yml
│ │ └── inventory
│ └── shared
│ ├── bar.yml
│ ├── dev.yml
│ ├── foo.yml
│ ├── prod.yml
│ └── test.yml
└── roles
└── nginx
├── defaults
│ └── main.yml
├── meta
│ └── main.yml
├── tasks
│ └── main.yml
└── templates
└── main.yml
Now our inventory files can have the hosts use these shared group vars simply by putting the hosts in the correct groups.
You can place group_vars in playbook directory as well. More info.
Ansible will pick them up for all inventories.
I'm coming from a Puppet background using Vagrant and have some trouble making sense of Ansible and its differences.
My Puppet structure looked like this:
puppet
├── servers
│ └── Backend
│ └── Vagrantfile
└── src
├── manifests
│ └── nodes
│ └── development
│ └── backend.pp
└── modules
└── mysql
Setup was simple, cd to the Vagrantfile and fire up the VM with Vagrant.
Now this is my first draft of an Ansible folder structure:
ansible
├── servers
│ └── Backend
│ ├── Vagrantfile
│ └── ansible.cfg
└── sources
├── backend.yml
├── site.yml
├── inventories
│ └── development
│ ├── group_vars
│ │ ├── all
│ │ └── backend
│ └── hosts
├── playbooks
└── roles
└── mysql
Following questions now:
Is this best practise for Ansible or too close to Puppet?
Is it correct to treat backend.yml like a Puppet node manifest?
Where should I put site.yml and backend.yml? This example has them in the main directory while here it's in the 'plays' directory. What's the difference?
I think my group_vars in group_vars/backend aren't being used correctly, how do I access them?
Sources:
http://leucos.github.io/ansible-files-layout/
https://github.com/ansible/ansible-examples
https://github.com/enginyoyen/ansible-best-practises
In my case, I use following structures according environment complexity (check directory-layout) :
Simple environment
I use this structure when there is one environment or when I use playbooks in provision mode
ansible
├── inventory
│ ├── hosts
│ └── group_vars
│ └── my_group.yml
├── roles
│ └── mysql
├── ansible.cfg
├── README.md
├── playbook1.yml
└── playbook2.yml
In ansible.cfg, I use variable inventory = ./inventory in [default] in order to avoid setting inventory path with commands ansible-*.
Medium/Complex environment
I use this structure when there are more than one environment
ansible
├── inventories
│ ├── production
│ │ ├── hosts
│ │ └── group_vars
│ │ └── my_group.yml
│ └── development
│ ├── hosts
│ └── group_vars
│ └── my_group.yml
├── playbooks
│ ├── playbook1
│ │ ├── group_vars
│ │ │ └── my_group.yml
│ │ ├── roles
│ │ │ └── mysql
│ │ ├── README.md
│ │ └── site.yml
│ ...
├── README.md
└── ansible.cfg
In this case, there is a folder for each environments in ./inventories.
I prefer also to use a specific folder for each playbooks in order to be able to use easily a folder group_vars at playbook level as defined in variable precedence section. As environment become more complex, there are much more variables. A group_vars (and host_vars) in playbooks allows to defines common variables for all environment which makes there are less inventories variables.
Ultimate level environment
I already used Ansible to adresse systems with more than 5000 servers, here under some tips for adressing more complex environments :
Split inventory file
Use multiple files to define your inventory servers instead of a single hosts file. In this case hosts file contains only server's names, and other files contains groups with different perspectives :
└── production
├── hosts
├── middleware
└── trigram
middleware: Groups with mapping to used middlewares or other stuf. I use this file to map, for example, servers to tomcat, java, postgresql, etc. And I use it, for example, with playbooks that deploys monitoring agents : How to retrieve metrics, logs from tomcat, java, postgresql, etc.
trigram: On my project, I usually use codes with a fixed length (3 or 4) to identify my business components (ex. : 'STK' for stock management) then I create a group file to map a business component to my servers (which servers are used to deploy 'STK')
When you create a new playbook, choose your perspective to address different environments.
Caution : I think ansible load files with alphabetical name order, you can't define a group that refers a group not yet loaded.
Use folders for group_vars
In group_vars, instead of using files, you can use a folder with subfiles :
└── production
└── group_vars
└── my_group
├── vars1.yml
└── vars2.yml
This is usefull to split huge files or if you have tools that generate variables, in this case you have vars1.yml under git and vars2.yml is generated
Split git repo
When you are using ansible for huge system, there is a lot of commits and a question comes up often : How to split my huge git repo ?
I my case, I use one git repo for each folder in ./inventories with differents access rules. And a git repo for each folder in ./playbooks also with different access rules.
You found several examples of the recommended layout, but not the official one. That should hopefully answer many of your questions, but I'll try to address them here as well.
Is this best practise for Ansible or too close to Puppet?
It's definitely not the recommend layout for Ansible. In the best practices layout, there is no servers or sources - the things inside those all live at the top level (what does "servers" mean, anyways?).
Is it correct to treat backend.yml like a Puppet node manifest?
I'm not familiar with Puppet, and so can't answer this question.
Where should I put site.yml and backend.yml? This example has them in the main directory while here it's in the 'plays' directory. What's the difference?
The official recommendation has all playbooks splattered across the root directory. However, this gets a bit messy, so some people put them in a subdirectory (in your examples, plays). This works reasonably well, but you'll need to adjust the paths in your playbooks accordingly.
I think my group_vars in group_vars/backend aren't being used correctly, how do I access them?
You shouldn't put them in a subdirectory of the inventory folder, as they're not part of inventory. There are a number of places to define variables, and you should read the documentation to learn what they are and when you should use them, but group_vars should be at the root of the directory with everything else.