I am trying to query a list of ec2 instances via ansible using the ec2 plugin for dynamic inventories.
I can see the utility of using dynamic inventories. If new machines are added then ansible will automatically execute a play against them. But I also saw on the net that it is possible to spawn instances with ansible, and manually add the new hosts to the static list of hosts.
So my question is : What would be the use cases where we would use dynamic inventories vs static inventories ? i'm new to the realm of devops so i don't know how often we need to spawn instances automatically Vs doing it manually via the AWS console for example.
Thanks !
In case you use autoscaling groups you have to use dynamic inventories.
If you launch ec2s temporary as part of a build pipelines use dynamic inventories. e.g. you just want to test the deployment of your software and terminate the machine after that test.
If you want to disable ansible plays on some machines you can create dynamic inventories based on ec2 tags. e.g. you have a security play that runs an all web server each hour but a developer wants to test something on his machine. So he can tag his machine to be skipped. He doesn't need access to the inventory file (and you can run another play once at midnight to enable the security play again. so it won't be forgotten).
By the way: you can use ec2_instance_facts with filter options and add_host to create dynamic inventories during the playbooks run time.
e.g. you have three types of server "web", "app", "db". you tag the ec2s during launch with servertype: [web|app|db]. You can filter these ec2s with:
- name: collect ec2s
ec2_instance_facts:
region: "{{ region }}"
filters:
"tag:servertype": "{{ servertype_list }}"
register: ec2_list
and run your play selectively on a server group with an external variable ansible-playbook test.yml -e servertype_list=['web','app'] or ansible-playbook test.yml -e servertype_list=['db'].
So by tagging the machine you avoid to take care of a static inventory.
I have 2 VPCs, each of which has a VPN instance, whose configuration I would like to share. Both instances have a class tag, whose value is now vpn_dev, however, they should differ by the environment tags - one would have environment: default, and the other environment: london.
I also have 2 playbooks, one for each VPC. I would like to run my vpn role on exactly one of the instances based on the combination of the class and environment tags, i.e. select the instance that has class: vpn_dev and environment: london. E.g.
- name: Deploy developer VPN in AWS
hosts:
- tag_class_vpn_dev
- tag_environment_london
roles:
- vpn
However, this would of course install the role on all instances that have class: vpn_dev (regardless of environment), and on all instances that have environment: london (regardless of class).
Is there a way to achieve this? Currently it seems like the only way is to have a single tag that uniquely identifies an instance.
The lists of hosts in a playbook is an OR operation by default. You can use AND over to host groups with this syntax:
- hosts:
- tag_class_vpn_dev:&tag_environment_london
More about host patterns here
It is possible to apply some ansible role to one host for each tag combination, although I would not recommend it, because this approach IMHO is not idempotent.
One of the possible risk with this scenario is that when you re-run this playbook, selected host for some tag combination can be different from previous run, and therefore you will end up with two instances per this unique tag combination group.
My recommended approach would be to use some tag to mark host which is supposed to be used as VPN role target, and then use ansible to apply VPN role only to hosts marked by this tag.
I am trying to follow best pratices in designing my ansible playbook. So the playbook file itself looks like this:
- hosts: ...
roles:
- dbservers
- webservers
- ...
Now there is a task check_proxy_and_firewalls that I want to execute in the dbserers as well as in webservers roles.
How would I properly structure that? All the "best_practice" guides I find do not seem to consider this case.
I could simple add a file common_tasks/check_proxy_and_firewalls.yml and include it in the roles where I need this. Is that how to do it?
The proper way to reuse tasks from other playbooks is as you stated, by using an include.
Suppose you want to reuse lists of tasks between plays or playbooks. You can use include files to do this. Use of included task lists is a great way to define a role that system is going to fulfill.
From: Ansible Playbook Roles
To include a file simply add this to your playbook:
- include: path/to/task.yml
Structuring your roles largely depends on what you're using Ansible for. For example, I manage Windows and Linux servers so my roles are structured like this:
roles
windows
<some_function>
tasks
vars
linux
<some_function>
tasks
vars
Yours will be different depending on what your goals with ansible are.
According to the Ansible docs, a Playbook
is:
...the basis for a really simple configuration management and multi-machine deployment system, unlike any that already exist, and one that is very well suited to deploying complex applications.
And, again, according to those same docs, a Role
are:
...ways of automatically loading certain vars_files, tasks, and handlers based on a known file structure. Grouping content by roles also allows easy sharing of roles with other users.
However the distinction between these and their different use cases is not immediately obvious to me. For instance, if I configure my /etc/ansible/hosts file to look like:
[databases]
mydb01.example.org
mydb02.example.org
[mail_servers]
mymail01.example.org
mymail_dr.example.org
...then what is this "[databases]" entry...a role? Or the name of a playbook YAML file somewhere? Or something else?!?
If someone could explain to me the differences on these, my understanding of Ansible would be greatly enhance!
Playbook vs Role vs [databases] and similar entries in /etc/ansible/hosts
If Playbooks are defined inside of YAML files, then where are Roles defined?
Aside from the ansible.cfg living on the Ansible server, how do I add/configure Ansible with available Playbooks/Roles? For instance, when I run ansible-playbook someplaybook.yaml, how does Ansible know where to find that playbook?
Playbook vs Role vs [databases] and similar entries in /etc/ansible/hosts
[databases] is a single name for a group of hosts. It allows you to reference multiple hosts by a single name.
Role is a set of tasks and additional files to configure host to serve for a certain role.
Playbook is a mapping between hosts and roles.
Example from documentation describes example project. It contains two things:
Playbooks. site.yml, webservers.yml, fooservers.yml are playbooks.
Roles: roles/common/ and roles/webservers/ contain definitions of common and webservers roles accordingly.
Inside playbook (webservers.yml) you have something like:
---
- hosts: webservers <- this group of hosts defined in /etc/ansible/hosts, databases and mail_servers in example from your question
roles: <- this is list of roles to assign to these hosts
- common
- webservers
If Playbooks are defined inside of YAML files, then where are Roles defined?
They are defined inside roles/* directories. Roles are defined mostly using YAML files, but can also contain resources of any types (files/, templates/). According to documentation role definition is structured this way:
If roles/x/tasks/main.yml exists, tasks listed therein will be added to the play
If roles/x/handlers/main.yml exists, handlers listed therein will be added to the play
If roles/x/vars/main.yml exists, variables listed therein will be added to the play
If roles/x/meta/main.yml exists, any role dependencies listed therein will be added to the list of roles (1.3 and later)
Any copy tasks can reference files in roles/x/files/ without having to path them relatively or absolutely
Any script tasks can reference scripts in roles/x/files/ without having to path them relatively or absolutely
Any template tasks can reference files in roles/x/templates/ without having to path them relatively or absolutely
Any include tasks can reference files in roles/x/tasks/ without having to path them relatively or absolutely
The most important file is roles/x/tasks/main.yml, here you define tasks, which will be executed, when role is executed.
Aside from the ansible.cfg living on the Ansible server, how do I add/configure Ansible with available Playbooks/Roles? For instance, when I run ansible-playbook someplaybook.yaml, how does Ansible know where to find that playbook?
$ ansible-playbook someplaybook.yaml
Will look for a playbook inside current directory.
$ ansible-playbook somedir/somedir/someplaybook.yaml
Will look for a playbook inside somedir/somedir/ directory.
It's your responsibility to put your project with all playbooks and roles on server. Ansible has nothing to do with that.
Playbook vs Role vs [databases] and similar entries in /etc/ansible/hosts
Roles are a way to group tasks together into one container. You could have a role for setting up MySQL, another one for setting up Postfix etc.
A playbook defines what is happening where. This is the place where you define the hosts (hostgroups, see below) and the roles which will be applied to those hosts.
[databases] and the other entries in your inventory are hostgroups. Hostgroups define a set of hosts a play will run on.
A play is a set of tasks or roles (or both) inside a playbook. In most cases (and examples) a playbook will contain only one single play. But you can have as many as you like. That means you could have a playbook which will run the role postfix on the hostgroup mail_servers and the role mysql on the hostgroup databases:
- hosts: mail_servers
roles:
- postfix
- hosts: databases
roles:
- mysql
If Playbooks are defined inside of YAML files, then where are Roles defined?
In Ansible pretty much everything is defined in YAML, that counts for roles and playbooks.
Aside from the ansible.cfg living on the Ansible server, how do I add/configure Ansible with available Playbooks/Roles? For instance, when I run ansible-playbook someplaybook.yaml, how does Ansible know where to find that playbook?
AFAIK you have to provide the path to the playbook when invoking ansible-playbook. So ansible-playbook someplaybook.yaml would expect someplaybook.yaml to be in you current directory. But you can provide the full path: ansible-playbook /path/to/someplaybook.yaml
It's a terminology/semantic question. It can be subjective, even though there is a baseline definition.
My view is as follows:
Any configuration management/deployment system has:
source data - data used to create target host's configuration
target data - data used to identify target hosts
config changes - list/set of rules/actions we apply with source data over target host based on target data
In Ansible terms:
source data - is the various places we can put data - group_vars, playbook vars, role vars, etc., These places affect precedence (if a variable named the same is re-defined in different locations, there are very specific rules of what would be the value of the variable during ansible/ansible-playbook execution
target data - is the inventory (And, It's also possible to define inventory/hostgroup variables inside inventory!)
config changes - ansible has 4 levels of abstraction for it:
task - single action
task list - list of actions
role - list of actions (or list of lists) grouped by the same 'subject', usually all targets are operating on the same host/hostgroup
playbook - list of plays, each operating on possibly different hostgroup, applying several roles/tasks/tasklists (and special tasks like handlers)
From 'software' aspect - role should be generic enough to be reused.
Also in some (rather big) organizations, 'roles' are shipped by group A, while used in playbooks maintained by group B.
summary
All the above allows grouping of similar configurations - into a role.
grouping related subsystems/components into one playbook.
Also, worth mentioning, 1 YAML item in a playbook (including hosts: and either or tasks, pre_tasks, post_tasks, roles) is called a play
Now for your question:
Yes, it is confusing at first.
You usually connect your source data to your role's semantics, so when you see that role setup_db is applied in a play onto related hostgroup (e.g. db_hosts)
But a play can be running over a union of several hostgroups.
It's just a matter of convention vs flexibility.
P.S.
Please write me back whether this added to the confusion, or clarified.
Thanks.
Simply put:
A playbook is like the main program, it contents complete instructions to finish the job. However, for big projects, it is not desirable to actually put every detail in it. So you need role.
A role is a subroutine and usually achieves one goal, e.g. setup a database server. You can put it in roles/ directory, or download 3rd party roles by providing URIs in rolesfile.yml and ask ansible-galaxy to download them for you.
The [database] is a host group defined in inventory file that lists hosts that belong to the database group. You can also specify a group of web servers by specifying something like
[web]
web1.example.com
web2.example.com
Group web or database can then be used in playbooks or roles to specify the hosts to apply.
The groups can also be used in command ansible to run ad-hoc commands.
Also keep in mind a playbook can call more than one role if a meta file is used that is intended to affect the different roles.
Example Playbook: dual_role-playbook.yml
- name: Some Action for two roles
hosts: localhost
vars_files:
- roles/dual_role/meta/main.yml
roles:
- dual_role/container-1
- dual_role/container-2
The role folder and files scheme will look like this:
dual_role-playbook.yml
-- roles
-- dual_role
-- meta/main.yml
-- container-1
-- tasks/main.yml
-- templates/template.j2
-- container-2
-- tasks/main.yml
-- templates/template.j2
Is there a way to return a value from one included playbook to use in another? Consider the following playbook:
---
- include: provisionPostgres.yml store=09123
- include: provisionWebserver.yml store=09123
Each of these included playbooks provisions an AWS instance, and then installs software on that instance. Postgres and my web app respectively.
The use case I'm trying to handle is to have the first included playbook return the IP address of my database server so that I can use it to point my web server to in the 2nd included playbook.
I know that I can register values from tasks to use in other tasks, but I can't find out how to return a value from a playbook.
How can I accomplish this in ansible?