Configure Ansible roles with dependent roles - ansible

The problem is best described with an example:
There are two roles:
mailserver: a basic mail server configuration
mailinglist: mailing list application
The mailing list software needs the mailserver to transport incoming mails to the mailing list software's "virtual inbox". This requires some configuration of the mail server. But the mailserver does not know about the mailing list role, nor other roles with similar configuration requirements.
What I would like to do is this:
mailinglist (and other similar roles) stores the transport configuration in a variable transport_config. This could be a "transport map" like $email => $spool.
mailinglist depends on the mailserver role.
mailserver configures it's "transport" using the variable transport_config.
Is there a way to do something like this in Ansible? Or another solution to this problem? It's not possible to use role variables like {role: mailserver, transport_config: ...}, as there may be more than one role depending on the mailserver.
What I can think of is a workaround: The mailserver reads/parses a configuration directory where transport maps are defined. mailinglist and other roles add files to this directory. The problem here is that this often requires a "configuration builder" which reads such configuration directories and generates the main configuration file.

You can accomplish this using role dependencies.
In the mailinglist role under roles/mailinglist/meta/main.yml, add something like this:
---
dependencies:
- { role: mailserver, transport_config: ... }
Do the same for any other similar roles.

In response to your comment about a "configuration builder", see the assemble ansible module. The tricky part might be getting all the files into one place, before you run the assemble module.
Otherwise, tima's suggestion of host_ and group_vars makes sense.

I know this is long answered, but I only just found a workable solution, and it's a little sneaky.
I use a third settings role, which has no tasks, only variables in a defaults/main.yml. The docs are a bit vague on this, but values here get rippled to all dependent roles, so if both roles depend on settings through their meta/main.yml files, both get a common set of values. These are overridable in the usual ways, through a group_vars file.
The surprise to me was that since, yes, the settings role is used more than once, it doesn't matter because there are no tasks in it, and that data can flow from there up the chain of dependencies.
It's an entirely new form of data flow in Ansible that I didn't know was possible.

Consider managing the mailserver configuration file with something like dotdee:
Original Post by Dustin Kirkland describing the concept
Dustin's dotdee repository
A simpler implementation with less features in Python
With dotdee, you assemble your final configuration file from a series of files placed in a .d directory.
In your case, the mailinglist role and others depending on mailserver would drop configuration snippets in a .d directory (created by the mailserver role) and run a command to update the mailserver configuration file like dotdee --update /path/to/mailserver/conf.

I'm not familiar enough with your apps here to be sure, but I'm thinking that you may be better served using host or group variables to pass your configuration in to your roles. The roles can hold your defaults et al and expect to retrieve host/group variables values that you set on an individual basis. This may mean treating each app instance as a host rather than psychical host. How you model it depends on many finer points in your workflow, configuration and app.
This recent thread touches on some of this sort of thing: https://groups.google.com/forum/#!topic/ansible-project/yrnsx2Mw6rc

Related

Ansible specific log details collection

Ansible has logging plugins to send the data to log stash, log DNA, etc. But is it possible to log specific details like person who ran the playbook, on what servers playbook was executed , IP addresses. I am trying to understand whether is there any module specifically for logging.
I bet the thing you want is ARA from the OpenStack folks. I have been using it for a while and find it a ton easier to read than wading through a sea of log output.
That said, you may also be happier with AWX, or "Tower" (their commercial verion). Using AWX would have the benefit of enforcing access to the playbooks, versus asking people to correctly configure their ansible.cfg to use ARA.

How can I speedup ansible by not running entire sections too often?

Assume that a normal deployment script does a lot of things and many of them are related to preparing the OS itself.
These tasks are taking a LOT of time to run even if there is nothing new to do and I want to prevent running them more often than, let's say once a day.
I know that I can use tags to filter what I am running but that's not the point: I need to make ansible aware that these "sections" executed successfully one hour ago and due to this, it would skip the entire block now.
I was expecting that caching of facts was supposed to do this but somehow I wasnt able to see any read case.
You need to figure out how to determine what "executed successfully" means. Is is just that a given playbook ran to completion? Certain roles ran to completion? Certain indicators exist that allow you determine success?
As you mention, I don't think fact caching is going to help you here unless you want to introduce custom facts into each host (http://docs.ansible.com/ansible/playbooks_variables.html#local-facts-facts-d)
I typically come up with a set of variables/facts that indicate a role has already been run. Sometimes this involves making shell calls and registering vars, looking at gathered facts and determining if certain files exist. My typical pattern for a role looks something like this
roles/my_role/main.yml
- name: load facts
include: facts.yml
- name: run config tasks if needed
include: config.yml
when: fact_var1 and fact_vars2 and inv_var1 and reg_var1
You could also dynamically write a yaml variable file that get's included in your playbooks and contains variables about the configured state of your environment. This is a more global option and doesn't really work if you need to look at the configured status of individual machines. An extension of this would be to write status variables to host_vars or group_vars inventory variable files. These would get loaded automatically on a host by host basis.
Unfortunately, as far as I know, fact caching only caches host based facts such as those created by the setup module so wouldn't allow you to use set_fact to register a fact that, for example, a role had been completed and then conditionally check for that at the start of the role.
Instead you might want to consider using Ansible with another product to orchestrate it in a more complex fashion.

Prevent usage of wrong Ansible inventory

Imagine a server setup with Ansible with a production and a reference system/cluster and a separate server running Ansible (with ssh-keys). The different clusters are identified in two inventory files.
Every playbook usage will somehow look like ansible-playbook -i production ... or ansible-playbook -i reference....
How do I prevent an accidental usage of the production inventory?
This could happen easily by either using an history entry in the shell or copy the command from some documentation.
Some ideas:
As a start every documentation is referring to the reference inventory and is also using --check.
Use two different Ansible instances and the common part is mirrored via Git. But this will result in some overhead.
Prompt once for i.e. database passwords and use different on production and reference. But not all task/tags have such a password requirement.
Actually I'm looking for something like a master password when using a specific inventory? Or a task that is always executed even if a tag is used? What is the best practice here? Do you have other ideas? Or am I somehow totally wrong here and there is a better way for my scenario?
Your production inventory could either be vaulted or even just include a vaulted file (which doesn't actually have to contain anything).
This will mean that when you attempt to run:
ansible-playbook -i production playbook.yml
It will fail with a message telling you to pass a vault password to it.
The vault password could be something simple such as "pr0duct10n" so it's entire purpose is to remind people that they are about to run it in production and so to think things through (much like how many people use a sudo password).

Ansible Playbooks vs Roles

According to the Ansible docs, a Playbook
is:
...the basis for a really simple configuration management and multi-machine deployment system, unlike any that already exist, and one that is very well suited to deploying complex applications.
And, again, according to those same docs, a Role
are:
...ways of automatically loading certain vars_files, tasks, and handlers based on a known file structure. Grouping content by roles also allows easy sharing of roles with other users.
However the distinction between these and their different use cases is not immediately obvious to me. For instance, if I configure my /etc/ansible/hosts file to look like:
[databases]
mydb01.example.org
mydb02.example.org
[mail_servers]
mymail01.example.org
mymail_dr.example.org
...then what is this "[databases]" entry...a role? Or the name of a playbook YAML file somewhere? Or something else?!?
If someone could explain to me the differences on these, my understanding of Ansible would be greatly enhance!
Playbook vs Role vs [databases] and similar entries in /etc/ansible/hosts
If Playbooks are defined inside of YAML files, then where are Roles defined?
Aside from the ansible.cfg living on the Ansible server, how do I add/configure Ansible with available Playbooks/Roles? For instance, when I run ansible-playbook someplaybook.yaml, how does Ansible know where to find that playbook?
Playbook vs Role vs [databases] and similar entries in /etc/ansible/hosts
[databases] is a single name for a group of hosts. It allows you to reference multiple hosts by a single name.
Role is a set of tasks and additional files to configure host to serve for a certain role.
Playbook is a mapping between hosts and roles.
Example from documentation describes example project. It contains two things:
Playbooks. site.yml, webservers.yml, fooservers.yml are playbooks.
Roles: roles/common/ and roles/webservers/ contain definitions of common and webservers roles accordingly.
Inside playbook (webservers.yml) you have something like:
---
- hosts: webservers <- this group of hosts defined in /etc/ansible/hosts, databases and mail_servers in example from your question
roles: <- this is list of roles to assign to these hosts
- common
- webservers
If Playbooks are defined inside of YAML files, then where are Roles defined?
They are defined inside roles/* directories. Roles are defined mostly using YAML files, but can also contain resources of any types (files/, templates/). According to documentation role definition is structured this way:
If roles/x/tasks/main.yml exists, tasks listed therein will be added to the play
If roles/x/handlers/main.yml exists, handlers listed therein will be added to the play
If roles/x/vars/main.yml exists, variables listed therein will be added to the play
If roles/x/meta/main.yml exists, any role dependencies listed therein will be added to the list of roles (1.3 and later)
Any copy tasks can reference files in roles/x/files/ without having to path them relatively or absolutely
Any script tasks can reference scripts in roles/x/files/ without having to path them relatively or absolutely
Any template tasks can reference files in roles/x/templates/ without having to path them relatively or absolutely
Any include tasks can reference files in roles/x/tasks/ without having to path them relatively or absolutely
The most important file is roles/x/tasks/main.yml, here you define tasks, which will be executed, when role is executed.
Aside from the ansible.cfg living on the Ansible server, how do I add/configure Ansible with available Playbooks/Roles? For instance, when I run ansible-playbook someplaybook.yaml, how does Ansible know where to find that playbook?
$ ansible-playbook someplaybook.yaml
Will look for a playbook inside current directory.
$ ansible-playbook somedir/somedir/someplaybook.yaml
Will look for a playbook inside somedir/somedir/ directory.
It's your responsibility to put your project with all playbooks and roles on server. Ansible has nothing to do with that.
Playbook vs Role vs [databases] and similar entries in /etc/ansible/hosts
Roles are a way to group tasks together into one container. You could have a role for setting up MySQL, another one for setting up Postfix etc.
A playbook defines what is happening where. This is the place where you define the hosts (hostgroups, see below) and the roles which will be applied to those hosts.
[databases] and the other entries in your inventory are hostgroups. Hostgroups define a set of hosts a play will run on.
A play is a set of tasks or roles (or both) inside a playbook. In most cases (and examples) a playbook will contain only one single play. But you can have as many as you like. That means you could have a playbook which will run the role postfix on the hostgroup mail_servers and the role mysql on the hostgroup databases:
- hosts: mail_servers
roles:
- postfix
- hosts: databases
roles:
- mysql
If Playbooks are defined inside of YAML files, then where are Roles defined?
In Ansible pretty much everything is defined in YAML, that counts for roles and playbooks.
Aside from the ansible.cfg living on the Ansible server, how do I add/configure Ansible with available Playbooks/Roles? For instance, when I run ansible-playbook someplaybook.yaml, how does Ansible know where to find that playbook?
AFAIK you have to provide the path to the playbook when invoking ansible-playbook. So ansible-playbook someplaybook.yaml would expect someplaybook.yaml to be in you current directory. But you can provide the full path: ansible-playbook /path/to/someplaybook.yaml
It's a terminology/semantic question. It can be subjective, even though there is a baseline definition.
My view is as follows:
Any configuration management/deployment system has:
source data - data used to create target host's configuration
target data - data used to identify target hosts
config changes - list/set of rules/actions we apply with source data over target host based on target data
In Ansible terms:
source data - is the various places we can put data - group_vars, playbook vars, role vars, etc., These places affect precedence (if a variable named the same is re-defined in different locations, there are very specific rules of what would be the value of the variable during ansible/ansible-playbook execution
target data - is the inventory (And, It's also possible to define inventory/hostgroup variables inside inventory!)
config changes - ansible has 4 levels of abstraction for it:
task - single action
task list - list of actions
role - list of actions (or list of lists) grouped by the same 'subject', usually all targets are operating on the same host/hostgroup
playbook - list of plays, each operating on possibly different hostgroup, applying several roles/tasks/tasklists (and special tasks like handlers)
From 'software' aspect - role should be generic enough to be reused.
Also in some (rather big) organizations, 'roles' are shipped by group A, while used in playbooks maintained by group B.
summary
All the above allows grouping of similar configurations - into a role.
grouping related subsystems/components into one playbook.
Also, worth mentioning, 1 YAML item in a playbook (including hosts: and either or tasks, pre_tasks, post_tasks, roles) is called a play
Now for your question:
Yes, it is confusing at first.
You usually connect your source data to your role's semantics, so when you see that role setup_db is applied in a play onto related hostgroup (e.g. db_hosts)
But a play can be running over a union of several hostgroups.
It's just a matter of convention vs flexibility.
P.S.
Please write me back whether this added to the confusion, or clarified.
Thanks.
Simply put:
A playbook is like the main program, it contents complete instructions to finish the job. However, for big projects, it is not desirable to actually put every detail in it. So you need role.
A role is a subroutine and usually achieves one goal, e.g. setup a database server. You can put it in roles/ directory, or download 3rd party roles by providing URIs in rolesfile.yml and ask ansible-galaxy to download them for you.
The [database] is a host group defined in inventory file that lists hosts that belong to the database group. You can also specify a group of web servers by specifying something like
[web]
web1.example.com
web2.example.com
Group web or database can then be used in playbooks or roles to specify the hosts to apply.
The groups can also be used in command ansible to run ad-hoc commands.
Also keep in mind a playbook can call more than one role if a meta file is used that is intended to affect the different roles.
Example Playbook: dual_role-playbook.yml
- name: Some Action for two roles
hosts: localhost
vars_files:
- roles/dual_role/meta/main.yml
roles:
- dual_role/container-1
- dual_role/container-2
The role folder and files scheme will look like this:
dual_role-playbook.yml
-- roles
-- dual_role
-- meta/main.yml
-- container-1
-- tasks/main.yml
-- templates/template.j2
-- container-2
-- tasks/main.yml
-- templates/template.j2

ansible design help (servers, teams, roles, playbooks and more)

We are trying to design an Ansible system for our crew.
We have some open questions that cause us to stop and think and maybe hear other ideas.
The details:
4 development teams.
We hold CI servers, DB servers, and a personal virtual machine for each programer.
A new programer receives a clean VM and we would like to use Ansible to "prepare" it for him according to team he is about to join.
We also want to use Ansible for weekly updates (when needed) on some VMs - it might be for a whole team or for all our VMs.
Team A and Team B shares some of their needs (for example, they both use Django) but there are naturally applications that Team A uses and Team B does not.
What we have done:
We had old "maintenance" bash scripts that we translate to YAML scripts.
We grouped them into Ansible roles
We have an inventory file which contains group for each team and our servers:
`
[ALL:children]
Team A
Team B
...
[Team A]
...
[Team B]
...
[CIservers]
...
[DBservers]
...
We have large playbook that contains all our roles (with tag to each):
- hosts: ALL
roles:
- { role x, tags: 'x' }
- { role y, tags: 'y' }
...
We invoke Ansible like that:
ansible-playbook -i inventory -t TAG1,TAG2 -l TeamA play.yml
The Problems:
We have a feeling we are not using roles as we should. We ended up with roles like "mercurial" or "eclipse" that install and configure (add aliases, edit PATH, creates symbolic links, etc) and role for apt_packages (using apt module to install the packages we need) and role for pip_packages (using pip module to install the packages we need).
Some of our roles depends on other roles (we used the meta folder to declare those dependencies). Because our playbook contains all the roles we have, when we run it without tags (on a new programer VM for example) the roles that other roles depends on are running twice (or more) and it is a waste of time. We taught to remove the roles that other depends on from our playbook, but it is not a good solution because in this way we loose the ability to run that role by itself.
We are not sure how to continue from this point. Whether to yield roles dependencies and create playbooks that implement those dependencies by specify the roles in the right order.
Should we change our roles into something like TeamA or DBserver that will unite many of our current roles (in such case, how do we handle the common tasks between TeamA and TeamB and how do we handle the tasks that relevant only for TeamA?)
Well, that is about everything.
Thanks in advance!
Sorry for the late answer and I guess your team has probably figured out the solution by now. I suspect you'll have the standard ansible structure with group_vars, hosts_vars, a roles folder and a site.yml as outlines below
site.yml
group_vars
host_vars
roles
common
dbserver
ciserver
I suspect your team is attempting to link everything into a single site.yml file. This is fine for the common tasks which operate based on roles and tags. I suggest for those edge cases, you create a second or third playbook at the root level, which can be specific to a team or a weekly deployment. In this case, you can still keep the common tasks in the standard structure, but you are not complicating your tasks with all the meta stuff.
site.yml // the standard ansible way
teamb.yml // we need to do something slightly different
Again, as you spot better ways of implementing a task, the playbooks can be refactored and tasks moved from the specific files to the standard roles
Seems you are still trying to see whats the best way to use ansible when you have multiple teams which will work on the same and don't want to affect others task. Have a look at this boilerplate it might help.
If you look in that repo. You will see there are multiple roles and you can design the playbook as per your requirement.
Example:
- common.yml (This will be common between all the team)
- Else you can create using by teamname.yml or project.yml
If you use any of the above you just need to define the proper role in the playbook & it should associate with the right host & group vars.

Resources