Lets say I have numerous ansible playbooks for various personal projects. They all do wildly different things, but they have a few repeated roles in common.
For example one of the roles clones my personal dot-files repository so I can have my personal .bash_profile, .vimrc, etc... on every server.
All my playbooks are in separate git repositories. Right now each of them duplicates the "dotfiles" role separately.
Is there a good way I can move that re-usable role to a separate repository and have each of the invidiual playbooks import that as needed?
Prerequisite
Have each role in its own git repository.
Typical project structure
This can be challenged. that's the one I like to use on different of my projects:
├── inventories
│ ├── dev
│ │ ├── group_vars/
│ │ └── hosts.ini
│ └── prod
│ ├── group_vars/
│ └── hosts.ini
├── group_vars/
├── host_vars/
├── files/
├── templates/
├── roles
│ ├── localy_versionned_role1/
│ ├── localy_versionned_role2/
│ ├── requirements.yml
│ ├── .gitignore
├── ansible.cfg
├── README.md
├── some_playbook.yml
├── other_playbook.yml
roles/.gitignore
# Ignore everything in dir...
/*
# ... but current file...
!.gitignore
# ... external role requirement file
!requirements.yml
# ... and configured custom/local roles
!localy_versionned_role*/
roles/requirements.yml
# Classic galaxy role
- src: galaxy_user.role_name
# Git available role
- src: git#git.service.com:path/to/repo.git
scm: git
version: master
name: local_role_name
You only need to list "top" roles, dependencies (listed in meta/main.yml of the role) will be downloaded also.
Ansible.cfg
We make sure that roles are searched and downloaded in our local folder
roles_path = roles
Workflow to deploy from your project:
Clone your project repository
Download the external roles:
ansible-galaxy install -r roles/requirements
Launch your playbook:
ansible-playbook -i inventories/dev some_playbook.yml
Going further
Downloading the roles as git workspaces
By default, ansible-galaxy downloads from git and removes the local repository structure (i.e. the .git directory). If you want to download the roles and keep working on them (change, commit, push...), you can keep the git structure with:
ansible-galaxy install -g -r roles/requirements
Note that this will write a meta/.galaxy_install_info file inside your role that git will see as new if you did not ignore already.
Updating roles after first install
Roles will not be updated with a new version (especially from galaxy) if they are already installed. To force updating use:
ansible-galaxy -f -r roles/requirements.yml
of course you do this also to switch to git workspace version (or reset to version given in requirements file)
ansible-galaxy -f -g -r roles/requirements.yml
Related
I'm trying to follow the the README.md for grpc-java's TLS example, https://github.com/grpc/grpc-java/tree/master/examples/example-tls. It states that running
../gradlew installDist
This creates the scripts hello-world-tls-server, hello-world-tls-client, in the build/install/example-tls/bin/ directory that run the example. The example requires the server to be running before starting the client.
However, after replacing the gRPC version 1.36.0-SNAPSHOT by 1.35.0 everywhere in the examples directory and running that command, the build/install/examples/bin directory contains no such scripts:
~/D/S/g/examples (master)> tree build/install/examples/bin
build/install/examples/bin
├── compressing-hello-world-client
├── compressing-hello-world-client.bat
├── hedging-hello-world-client
├── hedging-hello-world-client.bat
├── hedging-hello-world-server
├── hedging-hello-world-server.bat
├── hello-world-client
├── hello-world-client.bat
├── hello-world-server
├── hello-world-server.bat
├── retrying-hello-world-client
├── retrying-hello-world-client.bat
├── retrying-hello-world-server
├── retrying-hello-world-server.bat
├── route-guide-client
├── route-guide-client.bat
├── route-guide-server
└── route-guide-server.bat
0 directories, 18 files
In my local repository, I've changed this line, https://github.com/grpc/grpc-java/blob/1b23cf4f39ab26728336edbda8bb6af22dfe0a01/examples/example-tls/build.gradle#L58, to
startScripts.enabled = true
However, after running ./gradlew installDist again, the expected start scripts are still nowhere to be found. I've perused the documentation on CreateStartScripts, https://docs.gradle.org/current/dsl/org.gradle.jvm.application.tasks.CreateStartScripts.html, but I don't immediately see what's amiss. Any idea why no start scripts are created for the TLS example?
The documentation recommends to checkout a git tag:
You are strongly encouraged to check out a git release tag, since there will already be a build of grpc available
❯ git checkout v1.35.0
❯ cd examples/example-tls
❯ ../gradlew installDist
Then you'll be able to find the binaries in build/install/example-tls/bin:
❯ tree build/install/example-tls/bin
build/install/example-tls/bin
├── hello-world-tls-client
├── hello-world-tls-client.bat
├── hello-world-tls-server
└── hello-world-tls-server.bat
I am trying to figure out the best practice folder structure to organize my vault variables.
Currently it looks like this:
.
├── group_vars/
│ ├── group1.yml
│ └── group2.yml
├── host_vars/
│ ├── host1.yml
│ └── host2.yml
├── roles/
│ └── .../
└── vault/
├── enc-file1.yml
└── enc-file2.yml
However, this way I always have to use include_vars inside my role to source a specific encrypted file.
Is there any naming convention and folder structure I can apply that Ansible will automatically source the correct vaulted variable just as it does with host_vars and group_vars?
I had something like this in mind:
.
└── group_vars/
├── group1/
│ ├── main.yml
│ └── vault.yml
└── group2/
├── main.yml
└── vault.yml
Is there anything I can do, so I do not have to explicitly include vault variables?
Is there anything I can do, so I do not have to explicitly include vault variables?
In recent Ansible versions (since 2.3) you don't need to include separate files for vault-protected variables. Instead, you can encrypt individual variables in regular vars-files - see Single Encrypted Variable.
Is there any naming convention and folder structure [] that Ansible will automatically source the correct vaulted variable []?
No, there is no convention nor automatic mechanism.
#cytopia, what you have in mind works.
You can split encrypted/clear inventory variables in separate files. Ansible will sort out what is crypted and what is not.
I use the following layout and had no problems with it so far (1.9.x - 2.2.1.0):
group_vars/
all/
clear
secret
group1/
clear
secret
group2/
clear
secret
Below is my directory structure for ansibel role called webserver
localhost roles # tree
.
├── readme.md
├── site.yml
└── webserver
├── files
│ ├── createswap.sh
│ └── nginxkeyadd.sh
├── handlers
│ └── main.yml
├── tasks
│ └── main.yml
├── templates
│ ├── helloworld.conf.j2
│ └── index.html.j2
└── vars
└── main.yml
my tasks/main.yml looks like
- name: Create swap file 50MB
script: /etc/ansible/roles/webserver/files/createswap.sh
- name: add GPG key for nginx
script: /etc/ansible/roles/webserver/files/nginxkeyadd.sh
- name: Install nginx on target
apt: name={{ item }} state=latest
with_items:
- rsync
- git
- nginx
in the task/main.yml im specifying absolute path to local scriptfile like
script: /etc/ansible/roles/webserver/files/nginxkeyadd.sh and script: /etc/ansible/roles/webserver/files/createswap.sh . The scripts don't have any ansible variables.
is it good practice in ansible ?
is it good practice in ansible ?
No. Excerpt from the docs:
Any copy, script, template or include tasks (in the role) can reference files in roles/x/{files,templates,tasks}/ (dir depends on task) without having to path them relatively or absolutely
Also using shell-scripts instead of native Ansible modules is an antipattern.
I have installed few roles using ansible-galaxy and have included the following line in my .ansible.cfg:
roles_path = /Users/idyllic/ansible/roles
Now, for a specific project I have defined a requirement.yml where I have listed all the ansible roles that need to be installed (and they are all stored in /Users/idyllic/ansible/roles).
ansible-galaxy install -r requirement.yml
Now, even project has a roles directory so is it wise to download the roles in the project roles directory instead of the one defining it on directory define by (.ansible_cfg)
What is the best practice for storing the roles in ansible (version 2.1.1.0)?
i have this line in my project root folder
roles_path = ./roles:..:/etc/ansible/roles
and then the requirements.yml installation places then within the project.
This ensures that if a role is part of two projects, its not updated by accident
project tree structure:
.
├── Makefile
├── Vagrantfile
├── ansible.cfg
├── defaults
│ └── main.yml
├── elasticsearch.yml
├── handlers
│ └── main.yml
├── meta
│ └── main.yml
├── requirements.yml
├── tasks
│ ├── main.yml
├── templates
├── roles # this is where the requirements.yml go
└── vars
└── main.yml
I'm using the project structure from https://github.com/enginyoyen/ansible-best-practises.
It contains a bash script which downloads and updates in a dedicated directory Ansible external roles which are declared in a requirements file:
I find it very easy and very useful to create many different and separated project structures.
I'm coming from a Puppet background using Vagrant and have some trouble making sense of Ansible and its differences.
My Puppet structure looked like this:
puppet
├── servers
│ └── Backend
│ └── Vagrantfile
└── src
├── manifests
│ └── nodes
│ └── development
│ └── backend.pp
└── modules
└── mysql
Setup was simple, cd to the Vagrantfile and fire up the VM with Vagrant.
Now this is my first draft of an Ansible folder structure:
ansible
├── servers
│ └── Backend
│ ├── Vagrantfile
│ └── ansible.cfg
└── sources
├── backend.yml
├── site.yml
├── inventories
│ └── development
│ ├── group_vars
│ │ ├── all
│ │ └── backend
│ └── hosts
├── playbooks
└── roles
└── mysql
Following questions now:
Is this best practise for Ansible or too close to Puppet?
Is it correct to treat backend.yml like a Puppet node manifest?
Where should I put site.yml and backend.yml? This example has them in the main directory while here it's in the 'plays' directory. What's the difference?
I think my group_vars in group_vars/backend aren't being used correctly, how do I access them?
Sources:
http://leucos.github.io/ansible-files-layout/
https://github.com/ansible/ansible-examples
https://github.com/enginyoyen/ansible-best-practises
In my case, I use following structures according environment complexity (check directory-layout) :
Simple environment
I use this structure when there is one environment or when I use playbooks in provision mode
ansible
├── inventory
│ ├── hosts
│ └── group_vars
│ └── my_group.yml
├── roles
│ └── mysql
├── ansible.cfg
├── README.md
├── playbook1.yml
└── playbook2.yml
In ansible.cfg, I use variable inventory = ./inventory in [default] in order to avoid setting inventory path with commands ansible-*.
Medium/Complex environment
I use this structure when there are more than one environment
ansible
├── inventories
│ ├── production
│ │ ├── hosts
│ │ └── group_vars
│ │ └── my_group.yml
│ └── development
│ ├── hosts
│ └── group_vars
│ └── my_group.yml
├── playbooks
│ ├── playbook1
│ │ ├── group_vars
│ │ │ └── my_group.yml
│ │ ├── roles
│ │ │ └── mysql
│ │ ├── README.md
│ │ └── site.yml
│ ...
├── README.md
└── ansible.cfg
In this case, there is a folder for each environments in ./inventories.
I prefer also to use a specific folder for each playbooks in order to be able to use easily a folder group_vars at playbook level as defined in variable precedence section. As environment become more complex, there are much more variables. A group_vars (and host_vars) in playbooks allows to defines common variables for all environment which makes there are less inventories variables.
Ultimate level environment
I already used Ansible to adresse systems with more than 5000 servers, here under some tips for adressing more complex environments :
Split inventory file
Use multiple files to define your inventory servers instead of a single hosts file. In this case hosts file contains only server's names, and other files contains groups with different perspectives :
└── production
├── hosts
├── middleware
└── trigram
middleware: Groups with mapping to used middlewares or other stuf. I use this file to map, for example, servers to tomcat, java, postgresql, etc. And I use it, for example, with playbooks that deploys monitoring agents : How to retrieve metrics, logs from tomcat, java, postgresql, etc.
trigram: On my project, I usually use codes with a fixed length (3 or 4) to identify my business components (ex. : 'STK' for stock management) then I create a group file to map a business component to my servers (which servers are used to deploy 'STK')
When you create a new playbook, choose your perspective to address different environments.
Caution : I think ansible load files with alphabetical name order, you can't define a group that refers a group not yet loaded.
Use folders for group_vars
In group_vars, instead of using files, you can use a folder with subfiles :
└── production
└── group_vars
└── my_group
├── vars1.yml
└── vars2.yml
This is usefull to split huge files or if you have tools that generate variables, in this case you have vars1.yml under git and vars2.yml is generated
Split git repo
When you are using ansible for huge system, there is a lot of commits and a question comes up often : How to split my huge git repo ?
I my case, I use one git repo for each folder in ./inventories with differents access rules. And a git repo for each folder in ./playbooks also with different access rules.
You found several examples of the recommended layout, but not the official one. That should hopefully answer many of your questions, but I'll try to address them here as well.
Is this best practise for Ansible or too close to Puppet?
It's definitely not the recommend layout for Ansible. In the best practices layout, there is no servers or sources - the things inside those all live at the top level (what does "servers" mean, anyways?).
Is it correct to treat backend.yml like a Puppet node manifest?
I'm not familiar with Puppet, and so can't answer this question.
Where should I put site.yml and backend.yml? This example has them in the main directory while here it's in the 'plays' directory. What's the difference?
The official recommendation has all playbooks splattered across the root directory. However, this gets a bit messy, so some people put them in a subdirectory (in your examples, plays). This works reasonably well, but you'll need to adjust the paths in your playbooks accordingly.
I think my group_vars in group_vars/backend aren't being used correctly, how do I access them?
You shouldn't put them in a subdirectory of the inventory folder, as they're not part of inventory. There are a number of places to define variables, and you should read the documentation to learn what they are and when you should use them, but group_vars should be at the root of the directory with everything else.