We have dozens of Ansible roles with tasks which have been tagged with pretty standard labels along with other labels which are broadly in line with the facility or function the task affects.
deploy
configure
restart
dependancies
hubspot
database
nginx
rabbitmq
Based on the --limit syntax I really wish to do things like --tags '(hubspot,nginx)&restart'.
The --limit sytnax has Common patterns and over time it has proved very useful.
Any reason we cant we have the same for tags?
Or is there some other way to do this?
Any reason we cant we have the same for tags?
According the documentation about Selecting or skipping tags when you run a playbook it is just implemented in that documented way and without patterns.
Issue Links
... considerations, discussions, workarounds, etc.
Ansible Issue #11185 - Add a feature to run only tasks that have all of the given tags
Ansible Issue #11396 - Feature request: Provide capability to combine tags using logic AND or OR conditions
Ansible Issue #16273 - Feature request: More flexible tags
Ansible Issue #18401 - Support patterns logic for tags
Ansible Issue #31971 - Restrict task execution by tag intersection
Ansible Issue #70036 - Allow regex for "tags"
Ansible Pull #70043 - Support simple (fnmatch) wildcarding for tags
Ansible Issue #71733 - Allow combining tags with AND
Ansible Proposal #88 - Wildcards/regex in Ansible tags
I'm using AWX as a task runner to run a variety of Ansible modules. Some of the Ansible modules are third-party modules whose parameters I can't control without forking the module, which is undesirable for a variety of reasons.
AWX supplies ansible_user as one variable that is used by some of the modules I'm using, and I'm trying to allow a user to some hosts by setting another variable, user_override.
I first thought to simply add the line ansible_user: "{{ user_override | default(ansible_user) }}" to the task's parameters, which would work... but the modules in question don't accept credentials via parameters. My next thought was to add a vars: entry to the playbook and supply the override there via the same markup as above. This unfortunately results in the error recursive loop detected in template string, which has been the bane of my existence while working through this problem.
I've also tried using the if/else syntax and intermediate variables, but neither appear to solve this problem.
How can I achieve this override functionality without forking AWX or the module in question?
Mods: This is distinct from the pile of questions asking about simple variable defaulting because the existing questions aren't in the context of AWX or can be solved by simply using default() or default(lookup()).
At this point, I'm pretty sure that the thing I (the asker) was trying to do is intended to be done by just sometimes setting a value, but some challenging-to-anonymize constraints in our software make that somewhere between challenging and not an option.
A sideways solution to the particular problem of overriding the user is that AWX sets the username in remote_user as well as ansible_user, which is then used by networkcli. So you can use the line `remote_user: "{{ override_user | default(ansible_user) }}". This doesn't help generally, but does answer the question.
I'm stuck. Googled the hell out of the Web and couldn't find an answer.
I've been using Ansible for years, but always with static inventories. To differentiate between different environments like staging and production, I used different static inventory files, staging and production, respectively. When I needed to provision staging servers, I'd do:
ansible-playbook site.yml -i staging
When I wanted to do the same for production, I'd do:
ansible-playbook site.yml -i production
Both staging and production need variables with different values, so I have group_vars/staging and group_vars/production. All good and according to best practices.
Now, I need to provision EC2 instances in AWS. I'm using this AWS guide. I have a playbook with two plays. The first is run against localhost, creates/finds required EC2 instances in AWS, and populates a group with add_host. The second play uses that group to run against the EC2 instances discovered in the first play. All according to that guide.
It all works great except one thing. I have no idea how to specify which environment to provision and hence the required variables are not being loaded from group_vars/(staging|production). Basically, what I want is something similar to -i (staging|production) I used all these years with static inventories, but it seems that using -i doesn't make sense now since the inventory is dynamic. I want a way to be able to load variables from either group_vars/staging or group_vars/production based on an argument I pass to ansible-playbook when I run it.
How do I do that? What's the best practice?
While I am not sure how to do it with ansible EC2 moduel as we don't use it to build boxes from ansible level, there is a simple way to get what you want with ec2 external inventory script and simple settings in your inventories/main. What you need to do is set up the ec2.py and ec2.ini inside of your inventories so it will be used as source of instances. Make sure to uncomment group_by_tag_keys = True inside of ec2.ini.
Next step is to differentiate which instance goes where. While there are many selection methods available in ec2.py, I prefer to specifically tag each instance accordingly. So all my instances have a tag called environment which is filled accordingly (in your case it would be either staging or production). Then all is left is to handle it inside of your inventories/main, and here is a small example how to do it.
First you must define empty group for tags you want to use:
[tag_environment_staging]
[tag_environment_production]
so we can later reference to them. After that all there is left to do is specify those groups as children for appropriate stages. So after that our minimal file will look like that:
[tag_environment_staging]
[tag_environment_production]
[staging:children]
tag_environment_staging
[production:children]
tag_environment_production
And there you go. From now on every instance pulled from ec2 via dynamic inventory script that comes with environment tag will be matched to appropriate config in group_vars. All you have to remember that when dealing with dynamic inventories you want your -i point at inventories directory rather than specific file for it to work right.
I have a similar problem with dynamic inventories but for Openstack. The solution I've come up with so far is to use an environment variable to specify whether I want to target the staging or production environment. It should be applicable to your case as well. In our setup $OS_PROJECT_NAME is either stage or prod. In ansible.cfg set
inventory = ./inventories/${OS_PROJECT_NAME}/openstack.py
Then we have environment specific group variables under
inventories/(stage|prod)/group_vars/
The drawback is you have to have the inventory script in two places or have it symlinked. Beware also that group_vars found relative to the playbook directory will still override the inventory group_vars.
According to the Ansible docs, a Playbook
is:
...the basis for a really simple configuration management and multi-machine deployment system, unlike any that already exist, and one that is very well suited to deploying complex applications.
And, again, according to those same docs, a Role
are:
...ways of automatically loading certain vars_files, tasks, and handlers based on a known file structure. Grouping content by roles also allows easy sharing of roles with other users.
However the distinction between these and their different use cases is not immediately obvious to me. For instance, if I configure my /etc/ansible/hosts file to look like:
[databases]
mydb01.example.org
mydb02.example.org
[mail_servers]
mymail01.example.org
mymail_dr.example.org
...then what is this "[databases]" entry...a role? Or the name of a playbook YAML file somewhere? Or something else?!?
If someone could explain to me the differences on these, my understanding of Ansible would be greatly enhance!
Playbook vs Role vs [databases] and similar entries in /etc/ansible/hosts
If Playbooks are defined inside of YAML files, then where are Roles defined?
Aside from the ansible.cfg living on the Ansible server, how do I add/configure Ansible with available Playbooks/Roles? For instance, when I run ansible-playbook someplaybook.yaml, how does Ansible know where to find that playbook?
Playbook vs Role vs [databases] and similar entries in /etc/ansible/hosts
[databases] is a single name for a group of hosts. It allows you to reference multiple hosts by a single name.
Role is a set of tasks and additional files to configure host to serve for a certain role.
Playbook is a mapping between hosts and roles.
Example from documentation describes example project. It contains two things:
Playbooks. site.yml, webservers.yml, fooservers.yml are playbooks.
Roles: roles/common/ and roles/webservers/ contain definitions of common and webservers roles accordingly.
Inside playbook (webservers.yml) you have something like:
---
- hosts: webservers <- this group of hosts defined in /etc/ansible/hosts, databases and mail_servers in example from your question
roles: <- this is list of roles to assign to these hosts
- common
- webservers
If Playbooks are defined inside of YAML files, then where are Roles defined?
They are defined inside roles/* directories. Roles are defined mostly using YAML files, but can also contain resources of any types (files/, templates/). According to documentation role definition is structured this way:
If roles/x/tasks/main.yml exists, tasks listed therein will be added to the play
If roles/x/handlers/main.yml exists, handlers listed therein will be added to the play
If roles/x/vars/main.yml exists, variables listed therein will be added to the play
If roles/x/meta/main.yml exists, any role dependencies listed therein will be added to the list of roles (1.3 and later)
Any copy tasks can reference files in roles/x/files/ without having to path them relatively or absolutely
Any script tasks can reference scripts in roles/x/files/ without having to path them relatively or absolutely
Any template tasks can reference files in roles/x/templates/ without having to path them relatively or absolutely
Any include tasks can reference files in roles/x/tasks/ without having to path them relatively or absolutely
The most important file is roles/x/tasks/main.yml, here you define tasks, which will be executed, when role is executed.
Aside from the ansible.cfg living on the Ansible server, how do I add/configure Ansible with available Playbooks/Roles? For instance, when I run ansible-playbook someplaybook.yaml, how does Ansible know where to find that playbook?
$ ansible-playbook someplaybook.yaml
Will look for a playbook inside current directory.
$ ansible-playbook somedir/somedir/someplaybook.yaml
Will look for a playbook inside somedir/somedir/ directory.
It's your responsibility to put your project with all playbooks and roles on server. Ansible has nothing to do with that.
Playbook vs Role vs [databases] and similar entries in /etc/ansible/hosts
Roles are a way to group tasks together into one container. You could have a role for setting up MySQL, another one for setting up Postfix etc.
A playbook defines what is happening where. This is the place where you define the hosts (hostgroups, see below) and the roles which will be applied to those hosts.
[databases] and the other entries in your inventory are hostgroups. Hostgroups define a set of hosts a play will run on.
A play is a set of tasks or roles (or both) inside a playbook. In most cases (and examples) a playbook will contain only one single play. But you can have as many as you like. That means you could have a playbook which will run the role postfix on the hostgroup mail_servers and the role mysql on the hostgroup databases:
- hosts: mail_servers
roles:
- postfix
- hosts: databases
roles:
- mysql
If Playbooks are defined inside of YAML files, then where are Roles defined?
In Ansible pretty much everything is defined in YAML, that counts for roles and playbooks.
Aside from the ansible.cfg living on the Ansible server, how do I add/configure Ansible with available Playbooks/Roles? For instance, when I run ansible-playbook someplaybook.yaml, how does Ansible know where to find that playbook?
AFAIK you have to provide the path to the playbook when invoking ansible-playbook. So ansible-playbook someplaybook.yaml would expect someplaybook.yaml to be in you current directory. But you can provide the full path: ansible-playbook /path/to/someplaybook.yaml
It's a terminology/semantic question. It can be subjective, even though there is a baseline definition.
My view is as follows:
Any configuration management/deployment system has:
source data - data used to create target host's configuration
target data - data used to identify target hosts
config changes - list/set of rules/actions we apply with source data over target host based on target data
In Ansible terms:
source data - is the various places we can put data - group_vars, playbook vars, role vars, etc., These places affect precedence (if a variable named the same is re-defined in different locations, there are very specific rules of what would be the value of the variable during ansible/ansible-playbook execution
target data - is the inventory (And, It's also possible to define inventory/hostgroup variables inside inventory!)
config changes - ansible has 4 levels of abstraction for it:
task - single action
task list - list of actions
role - list of actions (or list of lists) grouped by the same 'subject', usually all targets are operating on the same host/hostgroup
playbook - list of plays, each operating on possibly different hostgroup, applying several roles/tasks/tasklists (and special tasks like handlers)
From 'software' aspect - role should be generic enough to be reused.
Also in some (rather big) organizations, 'roles' are shipped by group A, while used in playbooks maintained by group B.
summary
All the above allows grouping of similar configurations - into a role.
grouping related subsystems/components into one playbook.
Also, worth mentioning, 1 YAML item in a playbook (including hosts: and either or tasks, pre_tasks, post_tasks, roles) is called a play
Now for your question:
Yes, it is confusing at first.
You usually connect your source data to your role's semantics, so when you see that role setup_db is applied in a play onto related hostgroup (e.g. db_hosts)
But a play can be running over a union of several hostgroups.
It's just a matter of convention vs flexibility.
P.S.
Please write me back whether this added to the confusion, or clarified.
Thanks.
Simply put:
A playbook is like the main program, it contents complete instructions to finish the job. However, for big projects, it is not desirable to actually put every detail in it. So you need role.
A role is a subroutine and usually achieves one goal, e.g. setup a database server. You can put it in roles/ directory, or download 3rd party roles by providing URIs in rolesfile.yml and ask ansible-galaxy to download them for you.
The [database] is a host group defined in inventory file that lists hosts that belong to the database group. You can also specify a group of web servers by specifying something like
[web]
web1.example.com
web2.example.com
Group web or database can then be used in playbooks or roles to specify the hosts to apply.
The groups can also be used in command ansible to run ad-hoc commands.
Also keep in mind a playbook can call more than one role if a meta file is used that is intended to affect the different roles.
Example Playbook: dual_role-playbook.yml
- name: Some Action for two roles
hosts: localhost
vars_files:
- roles/dual_role/meta/main.yml
roles:
- dual_role/container-1
- dual_role/container-2
The role folder and files scheme will look like this:
dual_role-playbook.yml
-- roles
-- dual_role
-- meta/main.yml
-- container-1
-- tasks/main.yml
-- templates/template.j2
-- container-2
-- tasks/main.yml
-- templates/template.j2
My current Ansible project is setup like so:
backup-gitlab.yml
roles/
aws_backups/
tasks/
main.yml
backup-vm.yml
gitlab/
tasks/
main.yml
start.yml
stop.yml
backup-gitlab.yml needs to do the following:
Invoke stop.yml on the gitlab host.
Invoke backup-gitlab.yml on a different host.
Invoke start.yml on the gitlab host.
The problem I'm running into is Ansible doesn't seem to support a way of choosing which task files to run within the same role in the same playbook. Before I was using tags to control what Ansible would do, but in this case tagging the include statements for start.yml and stop.yml doesn't work because Ansible doesn't appear to have a way to dynamically change the applied tags that are run once they are set through the command line.
I can't come up with an elegant way to achieve this.
Some options are:
Have each task file be contained within its own role. This is annoying because I will end up with a million roles that are not grouped in any way. It's essentially abandoning the whole 'role' concept.
Use include with hard coded paths. This is prone to error as things move around. Also, since Ansible deprecated combining with_items with include (or using any sort of dynamic looping with include), I can no longer quickly change up the task files being run. Any minor change in my workflow requires lots of coding changes. I would really like to stick with using tags from the command line to control exactly what Ansible does.
Use shell scripts to invoke separate Ansible playbooks.
Use conditionals (when clause) on every single Ansible action, and control what gets run by setting variables. While several people have recommended this on SO, it sounds awful. I will have to add the conditional to hundreds of actions and every time I run a playbook the output will be cluttered by hundred's of 'skip' statements.
Leverage Jinja templates and ansible's local_connection to dynamically build static main.yml files with all the required task files included in the proper order (using computed relative paths). Then invoke that computed main.yml file. This is dangerous and convoluted.
Use top level Ansible plays to invoke lower level plays. Seems messy, also this brings in problems when I need to pass variables between plays. Using Ansible's Python Api may help this.
Ansible strives to bring VMs into idempotent states but this isn't very helpful and is a dated way of thinking in my opinion (I would have stuck with Chef if that is all I wanted). I want to leverage Ansible to actually do things such as: actively change configuration states, kick off processes, monitor events, react to events, etc. Essentially I want it to automate as much of my job as possible. The current 'role' structure (with static configurations) that Ansible recommends doesn't fit this paradigm very well even though their usage of remote command execution via SSH gets us so close to the dream.
Just use a playbook for these types of management tasks.
Granted the skip statements do somewhat clutter the output. If you wish to fix that you can further breakdown the roles into something like aws_backups-setup and aws_backups-managment.
Also the roles documentation has some information on how you can run pre_tasks and post_tasks on roles.