Ansible: applying role templates hierarchically - ansible

I'm planning to use Ansible to manage a fairly large set of Linux servers with a lot of variation among them.
The hosts will be grouped into roles with a hierarchy between them, there's a common role, on which every other depends, child-roles which depend on on common, and there may be grandchild-roles which depend on them, with no limit on dependency depth.
common
|___ role1
|___ role2
|___ role3
|___ role3.1
Each role has its own subset of templates for system configuration files.
roles/
common/
templates/
etc/
hosts
resolv.conf
tasks/
main.yml
system-settings.yml # imported in main.yml
role1/
templates/
etc/
hosts # role-specific version, overrides 'common'
httpd/
conf/
httpd.conf
tasks/
main.yml
system-settings.yml # imported in main.yml
role2/
role3/
role3.1/
The tricky part is that I'd like to apply these templates hierarchically, similarly to the way Ansible resolves variable scopes.
So, for a given child-role, the most specific version of a template file is applied, if a template does not exist in that role then apply the parent's version, searching up to the root level.
I have (sort of) achieved this by putting the following into the system-settings.yml for each role:
- name: Create directories
file:
path: /{{ item.path }}
state: directory
mode: '{{ item.mode }}'
with_filetree: templates/
when: item.state == 'directory'
- name: Template files
template:
src: '{{ item.src }}'
dest: /{{ item.path }}
mode: '{{ item.mode }}'
with_filetree: templates/
when: item.state == 'file'
- name: Recreate symlinks
file:
src: '{{ item.src }}'
dest: /{{ item.path }}
state: link
force: yes
mode: '{{ item.mode }}'
with_filetree: templates/
when: item.state == 'link'
It works, but unfortunately it applies all of the parent role's templates, then all of the child role's templates.
This is a waste of work and dangerous in case a playbook execution aborts for some reason, since the hosts may be left with generic versions of the system files.
So, for the above example tree, when applying role1 to a group of hosts Ansible:
applies /etc/hosts and /etc/resolv.conf from the common role
applies /etc/hosts and /etc/httpd/conf/httpd.conf from the role1 role
But I want it to do:
apply /etc/resolv.conf from the common role
apply /etc/hosts from the role1 role (the most specific version)
apply /etc/httpd/conf/httpd.conf from the role1 role
The template files should not be specifically listed in the tasks/playbooks due to management overhead, when adding a new file, or a role-specific version to override a generic one we don't want to have to change any playbooks or other configurations.
Is this achievable with Ansible?
I can see a possible solution using external scripts (as described in https://docs.ansible.com/ansible/2.4/playbooks_loops.html#iterating-over-the-results-of-a-program-execution), but I'm not sure how to pass down the role hierarchy into the script.

My thoughts:
First Choice:
Change the "common" file tasks to "common" handlers. The reason I picked handlers is we can conditionally include handlers. Also we have roles pre/post handlers. Notify what all tasks that are not done in role and what must be done can be in pre role handlers.
Second Choice:
Conditionally include tasks, I think lot of effort is needed to attach tags to included files.
E.g:
---
-name: task inclusion
Hosts: localhost
Gether_facts: false
Tasks:
Include: common.yaml
When: item | bool
A_list:
-true
-false
Hope this helps.
Regards
Sudhakar

Related

It's possible to parse template file on ansible role having the role itself as destination

I'm trying to parse a template file with Ansible, but I don't want this file to be created in any of my remote hosts, but instead I just want to create in my role_path.
I have the following in my role.
---
- name: Create configuration.yml on ansible role
ansible.builtin.template:
src: configuration.j2
dest: "{{role_path | default('')}}{{stack_name | mandatory}}/configuration.yml"
vars:
stack_env: "dev"
app_network: "my_network"
- name: Run tasks/main.yml from compose role
ansible.builtin.include_role:
name: compose
vars:
stack_name: "logging"
stack_path: "{{ ansible_base_path }}/"
When I run, my pipeline says that the directory doesn't exist, which is correct, because this directory exists outside my host, and not inside.
I basically want to parse this template file into my role, to be used by another role dependency.
Anyone knows if this is possible?
I found by myself the solution. It's possible to make use of local_action.
This is how my final playbook looks like.
- name: Create configuration.yml parsing variables
local_action:
module: template
src: configuration.j2
dest: "{{ role_path }}/logging/configuration.yml"
- name: Run tasks/main.yml from compose role
ansible.builtin.include_role:
name: compose
vars:
stack_name: "logging"
stack_path: "{{ ansible_base_path }}/"

Ansible: how to achieve idempotence with tasks that append files on host (w/o reverting to initial state)

I am having a hard time getting to know how to create Ansible roles that are following the best practices according to documentation. The following use-case which I am looking at is e.g. enabling Filebeat on host. Filebeat can be configured by placing a module definition in /etc/filebeat/modules.d folder.
It works fine when I am adding modules. Idempotence is working, everytime, on each run of the role (playbook), a given set of modules is enabled.
But what I should do when I decide that a given module is not longer needed? I remove it from role, rerun a playbook, so that all other modules are enabled. But: the previous run enabled a module that I am not installing directly with role after changes. So my server state is still altered in a way that is different than the role is imposing itself.
My question is: should I take care of removing modules before I apply them so I always start from, let's say, fresh state?
E.g.:
- name: Remove modules
file:
dest: "/etc/filebeat/modules.d/{{ item }}"
state: absent
loop:
- "module1.yml"
- "module2.yml"
- "module3.yml" # It was being installed in previous role, but not now
- name: Enable modules via 'modules.d' directory
template:
src: "modules.d/{{ item }}"
dest: "/etc/filebeat/modules.d/{{ item }}"
mode: '0644'
loop:
- "module1.yml"
- "module2.yml"
So I remove module3.yml, because I remember that I've installed it before, and install module1.yml and module2.yml.
Instead of just installing what I need, no matter what has been installed before:
- name: Enable modules via 'modules.d' directory
template:
src: "modules.d/{{ item }}"
dest: "/etc/filebeat/modules.d/{{ item }}"
mode: '0644'
loop:
- "module1.yml"
- "module2.yml"
Leaving me with module1.yml and module2.yml (desired) and, unfortunately: module3.yml (from previous role).
How to manage that to avoid such situations? And avoid treating server as one big stateful machine that even if I run a role, the output is different than desired, because something has been done before that I cannot see in current Ansible role code.
Do you code revert playbooks in your Ansible workflow to revert to initial state when needed?
I am curious. Thanks in advance for your reply.
In a nutshell:
- name: Configure filebeat modules
hosts: all
vars:
fb_modules_d:
- file: module1.yml
state: present
- file: module2.yml
state: present
- file: module3.yml
state: absent
tasks:
- name: Make sure all needed module files are present
template:
src: "modules.d/{{ item.file }}"
dest: "/etc/filebeat/modules.d/{{ item.file }}"
mode: '0644'
loop: "{{ fb_modules_d | selectattr('state', '==', 'present') }}"
notifiy: restart_filebeat
- name: Make sure all disabled modules are removed
file:
dest: "/etc/filebeat/modules.d/{{ item.file }}"
state: "{{ item.state }}"
loop: loop: "{{ fb_modules_d | selectattr('state', '==', 'absent') }}"
notify: restart_filebeat
handlers:
- name: Restart filebeat service
listen: restart_filebeat
systemd:
name: filebeat
state: restarted
Note: I declared the variable inside the playbook for the example but that one one should most probably go inside your inventory (group or host level), and certainly not in a role (except in defaults for documentation)
A simpler and more robust solution is to remove all files not listed. This would be especially useful when you don't know what is the current state of the directory. For example, given the directory (relative to PWD for testing)
shell> tree etc/filebeat/modules.d
etc/filebeat/modules.d
└── module99.yml
The playbook below
shell> cat playbook.yml
- name: Configure filebeat modules
hosts: localhost
vars:
fb_modules_path: etc/filebeat/modules.d
fb_modules_d:
- module1.yml
- module2.yml
- module3.yml
tasks:
- name: Create modules
template:
src: "modules.d/{{ item }}"
dest: "{{ fb_modules_path }}/{{ item }}"
mode: '0644'
loop: "{{ fb_modules_d }}"
notify: restart_filebeat
- name: Remove modules
file:
dest: "{{ fb_modules_path }}/{{ item }}"
state: absent
loop: "{{ query('fileglob', fb_modules_path ~ '/*')|
map('basename')|
difference(fb_modules_d) }}"
notify: restart_filebeat
handlers:
- name: Restart filebeat service
listen: restart_filebeat
# systemd:
# name: filebeat
# state: restarted
debug:
msg: Restart filebeat
creates the files from the list and removes others
TASK [Create modules] ***************************************************
changed: [localhost] => (item=module1.yml)
changed: [localhost] => (item=module2.yml)
changed: [localhost] => (item=module3.yml)
TASK [Remove modules] ***************************************************
changed: [localhost] => (item=module99.yml)
RUNNING HANDLER [Restart filebeat service] ******************************
ok: [localhost] =>
msg: Restart filebeat
shell> tree etc/filebeat/modules.d
etc/filebeat/modules.d
├── module1.yml
├── module2.yml
└── module3.yml
The playbook is idempotent.

Ansible role dependencies and facts with delegate_to

The scenario is: I have several services running on several hosts. There is one special service - the reverseproxy/loadbalancer. Any service needs to configure that special service on the host, that runs the rp/lp service. During installation/update/deletion of a random service with an Ansible role, I need to call the ReverseProxy role on the specific host to configure the corresponding vhost.
At the moment I call a specific task file in the reverse proxy role to add or remove a vhost by the service with include_role and set some vars (very easy example without service and inventory specific vars).
- name: "Configure ReverseProxy"
include_role:
name: reverseproxy
tasks_from: vhost_add
apply:
delegate_to: "{{ groups['reverseproxy'][0] }}"
vars:
reverse_proxy_url: "http://{{ ansible_fqdn }}:{{ service_port }}/"
reverse_proxy_domain: "sub.domain.tld"
I have three problems.
I know, it's not a good idea to build such dependencies between roles and different hosts. I don't know a better way, especially if you think about the situation, where you need to do some extra stuff after creating the vhost (f.e. configure the service via REST API, which needs the external fqdn). In case of two separate playbooks with "backend"-service and "reverseproxy"-service - then I need a third playbook for configuring "hanging" services. Also I'm not sure, if I can retrieve the correct backend URL in the reverse proxy role (only think about the HTTP scheme or paths). That sounds not easy, or?
Earlier I had separate roles for adding/removing vhosts to a reverseproxy. This roles didn't have dependencies, but I needed to duplicate several defaults and templates and vars etc. which isn't nice too. Then I've changed that to a single role. Of course - in my opinion, this isn't really that, what a "role" should be. A role is something like "webserver" or "reverseproxy" (a state). But not something like "add_vhost_to_reverseproxy" (a verb). This would be something like a playbook - but is calling a parameterized playbook via a role a good idea/possible? The main problem is, that the state of reverseproxy is the sum of all services in the inventory.
In case of that single included role, including it, starts also all dependent roles (configure custom, firewall, etc.). Nevertheless in that case I found out, that the delegation did not use the facts of the delegated host.
I tested that with the following example - the inventory:
all:
hosts:
server1:
my_var: a
server2:
my_var: b
children:
service:
hosts:
server1:
reverseproxy:
hosts:
server2:
And playbook which assigns a role-a to the group webserver. The role-a has a task like:
- block:
- setup:
- name: "Include role b on delegated {{ groups['reverseproxy'][0] }}"
include_role:
name: role-b
delegate_to: "{{ groups['reverseproxy'][0] }}"
delegate_facts: true # or false or omit - it has no effect on Ansible 2.9 and 2.10
And in role-b only outputing the my_var of the inventory will output
TASK [role-b : My_Var on server1] *******************
ok: [server1 -> <ip-of-server2>] =>
my_var: a
Which says me, that role-b that should be run on server2 has the facts of server1. So - configuring the "reverseproxy" service is done in context of the "backend"-service. Which would have several other issues - when you think about firewall-dependencies etc. I can avoid that, by using tags - but then I need to run the playbook not just with the tag of the service, but also with all tags I want to configure, and I cannot use include_tasks with args-apply-tags anymore inside a role that also includes other roles (the tags will applied to all subtasks...). I miss something like include_role but only that specific tags or ignore dependencies. This isn't a bug, but has possible side effects in case of delegate_to.
I'm not really sure, what is the question? The question is - what is a good way to handle dependencies between hosts and roles in Ansible - especially when they are not on the same host?
I am sure I do not fully understand your exact problem, but when I was dealing with load balancers I used a template. So this was my disable_workers playbook:
---
- hosts: "{{ ip_list | default( 'jboss' ) }}"
tasks:
- name: Tag JBoss service as 'disabled'
ec2_tag:
resource: "{{ ec2_id }}"
region: "{{ region }}"
state: present
tags:
State: 'disabled'
delegate_to: localhost
- action: setup
- hosts: httpd
become: yes
become_user: root
vars:
uriworkermap_file: "{{ httpd_conf_dir }}/uriworkermap.properties"
tasks:
- name: Refresh inventory cache
ec2_remote_facts:
region: "{{ region }}"
delegate_to: localhost
- name: Update uriworkermap.properties
template:
backup: yes
dest: "{{ uriworkermap_file }}"
mode: 0644
src: ./J2/uriworkermap.properties.j2
Do not expect this to work as-is. It was v1.8 on AWS hosts, and things may have changed.
But the point is to set user-defined facts, on each host, for that host's desired state (enabled, disabled, stopped), reload the facts, and then run the Jinja template that uses those facts.

How to override some specific variables values in Ansible playbook

I have some multiple war files which I am downloading from nexus and creating the respective app folder on my remote host and copying the files.
Now there are 2 files which have a different name as war file but need to create a directory with different name. What I am doing right now is this -
- name: Ensuring the web deployment folder is created on tomcat nodes for each web application
file:
path: "{{tomcat_server_release_location}}/Release{{my_release_version}}/{{item}}"
state: directory
mode: 0777
with_items:
- "{{ apps }}"
- name: Copying the release from Admin server to tomcat nodes
copy: src={{admin_server_release_location}}/{{my_release_version}}/{{item}}-{{my_release_version}}.war dest={{tomcat_server_release_location}}/Release{{my_release_version}}/{{item}}
with_items:
- "{{ apps }}"
apps variable is defined like this -
webapps: ['test1','test2','test3','test4'].
Now test2 has a different name on the nexus it says test2-web.war but on the remote node I have to create a folder called just test2 and copy the war file in there.
Is there a way to override some variables at run time of playbook by using some condition or anything
If you want to override at runtime use extra vars.
From the docs --extra-vars "version=1.23.45"
Don't use conditions if you know a specific host should always be overwritten. I think placing it as a host var is best.
hosts:
myhost:
apps:
- test1
- trst2
See https://docs.ansible.com/ansible/latest/user_guide/playbooks_variables.html#variable-precedence-where-should-i-put-a-variable for all of the various locations you can put variables and how to determine which will be used
It is possible to override variables by doing something like this:
- name: copy war
copy:
src: "{{ src_war | default('/path/to/my/war') }}"
dest: "{{ dest_war | default(src_war) }}"
Then at runtime you can set the variables src_war and dest_war either with extra vars, host/group_vars or any other way documented here: https://docs.ansible.com/ansible/latest/user_guide/playbooks_variables.html
In your case I think it would be helpful to set a src & dest mapping:
- set_fact:
apps:
- {src: "test1"}
- {src: "test2", dest: "test2-web"}
- {src: "test3"}
- {src: "test4"}
Then you can use this mapping to achieve your goal:
- name: Ensuring the web deployment...
file:
path: {{ tomcat_server_release_location }}/Release{{ my_release_version }}/{{ item.src }}
state: directory
mode: 0777
with_items: "{{ apps }}"
- name: Copying the release...
copy:
src: "{{ admin_server_release_location }}/{{ my_release_version }}/{{ item.src }}-{{ my_release_version }}.war"
dest: "{{ tomcat_server_release_location }}/Release{{ my_release_version }}/{{ item.dest | default(item.src) }}"
with_items: "{{ apps }}"

How to configure attribute precedence to group_vars with Ansible

I have a file in my Ansible role vars/sonarqube.yml with content
---
lvm_roles:
sonarqube:
size: '10g'
path: '{{ sonar_home }}'
And a file group_vars/all/lvm.yml with content
lvm_roles:
sonarqube:
size: '20g'
In ansible.cfg I have a line
hash_behaviour = merge
Without merge the resulting fact will be
lvm_roles:
sonarqube:
size: '20g'
With other words I loose the path var.
With merge the result is
lvm_roles:
sonarqube:
size: '10g'
path: '/opt/sonarqube'
The result I want and expected is however
lvm_roles:
sonarqube:
size: '20g'
path: '/opt/sonarqube'
So the desired behavior is that
Ansible merges vars
config in group_vars takes precedence to config in my role.
Can I configure this behavior in Ansible? How?
Precedence is not configurable in Ansible, you cannot give configuration in group_vars a higher precedence than configuration in the vars directory in a role.
What you want are conditional default vars which Ansible does not support . As shown for example here Set Ansible role defaults conditionally and here use conditionals in vars ansible.
This is an area where Ansible is definitely lacking as an IaC tool. It is for example a very commonly used feature with Chef. Snippet from attributes/default.rb from the Chef Apache cookbook demonstrates this common pattern
....
case node['platform_family']
when 'rhel', 'fedora', 'amazon'
if node['platform'] == 'amazon'
default['apache']['package'] = 'httpd24'
default['apache']['devel_package'] = 'httpd24-devel'
else
default['apache']['package'] = 'httpd'
default['apache']['devel_package'] = 'httpd-devel'
end
default['apache']['service_name'] = 'httpd'
default['apache']['perl_pkg'] = 'perl'
default['apache']['apachectl'] = '/usr/sbin/apachectl'
default['apache']['dir'] = '/etc/httpd'
default['apache']['log_dir'] = '/var/log/httpd'
Configuration in the vars directory can be compared to "override" attributes in Chef. Configuration in a Chef cookbook has a low precedence but you can use "override" attributes to give a very high precedence. The use of "override" attributes in cookbooks is however very uncommon. They are of very limited practical use. Vice versa the Ansible vars directory for its intended use of creating high precedence configuration that overrides almost all other configuration, is of very limited practical use.
If you disagree please share examples of roles where we absolutely need high precedence configuration in a role. You can for example share a link to a Ansible role that demonstrates practical use.
The vars role directory is useful but not for its intended use. In practice the directory is used to store conditional configuration. The fact that configuration gets a high precedence is a more a problem than a desired or intended result.
This is demonstrated by the geerlingguy.posttgresql role. In this role geerlingguy uses "pseudo variables" to work around the fact that Ansible does not have conditional defaults vars.
For example in vars/Debian-7.yml a variable __postgresql_data_dir is introduced. This variable gets a high precedence.
__postgresql_data_dir: "/var/lib/postgresql/{{ __postgresql_version }}/main"
It is of no practical use other than that it can be used to mimic a conditional default var postgresql_data_dir as show in tasks/variables.yml
- name: Define postgresql_data_dir.
set_fact:
postgresql_data_dir: "{{ __postgresql_data_dir }}"
when: postgresql_data_dir is not defined
It would make sense if precedence rules could be configured because the vars directory in a Ansible role is typically of limited practical use because of its high precedence. To make practical use of the vars directory trickery is needed as demonstrated by the postgresql_data_dir in the geerlingguy.posttgresql role to lower precedence of configuration in this directory.
If you don't like this trickery you can alternatively use workaround set_fact as described in Set Ansible role defaults conditionally or unholy inline coding as described in use conditionals in vars ansible.
The Ansible community would be well advised to change the intended use of the vars directory from "override" to "conditional" configuration. Giving high precedence to configuration in a role is very uncommon requirement. Conditional configuration is however very very common.
Q: "The desired behavior is that"
1) Ansible merges vars
2) config in group_vars takes precedence over config in my role.
"Can I configure this behavior in Ansible?"
A: No. The precedence can't be changed. "role vars"(15) override "group_vars"(7). Use "group_vars" to override "role defaults"(2).
Q: "What you want are conditional default vars which Ansible does not support."
A: If Ansible would not support conditional default it would not be possible to write roles for multiple systems. It is possible to create conditional defaults, e.g.
# Defaults variables
- name: "os_vars_playbook_dir: Vars from {{ playbook_dir }}/vars/defaults"
include_vars: "{{ item }}"
with_first_found:
- files:
- "{{ ansible_distribution }}-{{ ansible_distribution_release }}.yml"
- "{{ ansible_distribution }}.yml"
- "{{ ansible_os_family }}.yml"
- "default.yml"
- "defaults.yml"
paths: "{{ playbook_dir }}/vars/defaults"
# Custom variables
- name: "os_vars_playbook_dir: Vars from {{ playbook_dir }}/vars"
include_vars: "{{ item }}"
with_first_found:
- files:
- "{{ ansible_distribution }}-{{ ansible_distribution_release }}.yml"
- "{{ ansible_distribution }}.yml"
- "{{ ansible_os_family }}.yml"
- "default.yml"
- "defaults.yml"
paths: "{{ playbook_dir }}/vars"
(available at GitHub)
Notes
OS-specific variables can be included after a play setup found out what system is running on the host.
Role's vars/defaults have very high precedence "include_vars"(18) (because of 1.). But this directory should comprise system variables only. The user does not want to change these variables under standard circumstances.
If necessary, the role may be customized in vars/. Custom configuration files in vars/ will survive the role's update while vars/defaults may be updated.

Resources