How to configure attribute precedence to group_vars with Ansible - ansible

I have a file in my Ansible role vars/sonarqube.yml with content
---
lvm_roles:
sonarqube:
size: '10g'
path: '{{ sonar_home }}'
And a file group_vars/all/lvm.yml with content
lvm_roles:
sonarqube:
size: '20g'
In ansible.cfg I have a line
hash_behaviour = merge
Without merge the resulting fact will be
lvm_roles:
sonarqube:
size: '20g'
With other words I loose the path var.
With merge the result is
lvm_roles:
sonarqube:
size: '10g'
path: '/opt/sonarqube'
The result I want and expected is however
lvm_roles:
sonarqube:
size: '20g'
path: '/opt/sonarqube'
So the desired behavior is that
Ansible merges vars
config in group_vars takes precedence to config in my role.
Can I configure this behavior in Ansible? How?

Precedence is not configurable in Ansible, you cannot give configuration in group_vars a higher precedence than configuration in the vars directory in a role.
What you want are conditional default vars which Ansible does not support . As shown for example here Set Ansible role defaults conditionally and here use conditionals in vars ansible.
This is an area where Ansible is definitely lacking as an IaC tool. It is for example a very commonly used feature with Chef. Snippet from attributes/default.rb from the Chef Apache cookbook demonstrates this common pattern
....
case node['platform_family']
when 'rhel', 'fedora', 'amazon'
if node['platform'] == 'amazon'
default['apache']['package'] = 'httpd24'
default['apache']['devel_package'] = 'httpd24-devel'
else
default['apache']['package'] = 'httpd'
default['apache']['devel_package'] = 'httpd-devel'
end
default['apache']['service_name'] = 'httpd'
default['apache']['perl_pkg'] = 'perl'
default['apache']['apachectl'] = '/usr/sbin/apachectl'
default['apache']['dir'] = '/etc/httpd'
default['apache']['log_dir'] = '/var/log/httpd'
Configuration in the vars directory can be compared to "override" attributes in Chef. Configuration in a Chef cookbook has a low precedence but you can use "override" attributes to give a very high precedence. The use of "override" attributes in cookbooks is however very uncommon. They are of very limited practical use. Vice versa the Ansible vars directory for its intended use of creating high precedence configuration that overrides almost all other configuration, is of very limited practical use.
If you disagree please share examples of roles where we absolutely need high precedence configuration in a role. You can for example share a link to a Ansible role that demonstrates practical use.
The vars role directory is useful but not for its intended use. In practice the directory is used to store conditional configuration. The fact that configuration gets a high precedence is a more a problem than a desired or intended result.
This is demonstrated by the geerlingguy.posttgresql role. In this role geerlingguy uses "pseudo variables" to work around the fact that Ansible does not have conditional defaults vars.
For example in vars/Debian-7.yml a variable __postgresql_data_dir is introduced. This variable gets a high precedence.
__postgresql_data_dir: "/var/lib/postgresql/{{ __postgresql_version }}/main"
It is of no practical use other than that it can be used to mimic a conditional default var postgresql_data_dir as show in tasks/variables.yml
- name: Define postgresql_data_dir.
set_fact:
postgresql_data_dir: "{{ __postgresql_data_dir }}"
when: postgresql_data_dir is not defined
It would make sense if precedence rules could be configured because the vars directory in a Ansible role is typically of limited practical use because of its high precedence. To make practical use of the vars directory trickery is needed as demonstrated by the postgresql_data_dir in the geerlingguy.posttgresql role to lower precedence of configuration in this directory.
If you don't like this trickery you can alternatively use workaround set_fact as described in Set Ansible role defaults conditionally or unholy inline coding as described in use conditionals in vars ansible.
The Ansible community would be well advised to change the intended use of the vars directory from "override" to "conditional" configuration. Giving high precedence to configuration in a role is very uncommon requirement. Conditional configuration is however very very common.

Q: "The desired behavior is that"
1) Ansible merges vars
2) config in group_vars takes precedence over config in my role.
"Can I configure this behavior in Ansible?"
A: No. The precedence can't be changed. "role vars"(15) override "group_vars"(7). Use "group_vars" to override "role defaults"(2).
Q: "What you want are conditional default vars which Ansible does not support."
A: If Ansible would not support conditional default it would not be possible to write roles for multiple systems. It is possible to create conditional defaults, e.g.
# Defaults variables
- name: "os_vars_playbook_dir: Vars from {{ playbook_dir }}/vars/defaults"
include_vars: "{{ item }}"
with_first_found:
- files:
- "{{ ansible_distribution }}-{{ ansible_distribution_release }}.yml"
- "{{ ansible_distribution }}.yml"
- "{{ ansible_os_family }}.yml"
- "default.yml"
- "defaults.yml"
paths: "{{ playbook_dir }}/vars/defaults"
# Custom variables
- name: "os_vars_playbook_dir: Vars from {{ playbook_dir }}/vars"
include_vars: "{{ item }}"
with_first_found:
- files:
- "{{ ansible_distribution }}-{{ ansible_distribution_release }}.yml"
- "{{ ansible_distribution }}.yml"
- "{{ ansible_os_family }}.yml"
- "default.yml"
- "defaults.yml"
paths: "{{ playbook_dir }}/vars"
(available at GitHub)
Notes
OS-specific variables can be included after a play setup found out what system is running on the host.
Role's vars/defaults have very high precedence "include_vars"(18) (because of 1.). But this directory should comprise system variables only. The user does not want to change these variables under standard circumstances.
If necessary, the role may be customized in vars/. Custom configuration files in vars/ will survive the role's update while vars/defaults may be updated.

Related

Ansible role dependencies and facts with delegate_to

The scenario is: I have several services running on several hosts. There is one special service - the reverseproxy/loadbalancer. Any service needs to configure that special service on the host, that runs the rp/lp service. During installation/update/deletion of a random service with an Ansible role, I need to call the ReverseProxy role on the specific host to configure the corresponding vhost.
At the moment I call a specific task file in the reverse proxy role to add or remove a vhost by the service with include_role and set some vars (very easy example without service and inventory specific vars).
- name: "Configure ReverseProxy"
include_role:
name: reverseproxy
tasks_from: vhost_add
apply:
delegate_to: "{{ groups['reverseproxy'][0] }}"
vars:
reverse_proxy_url: "http://{{ ansible_fqdn }}:{{ service_port }}/"
reverse_proxy_domain: "sub.domain.tld"
I have three problems.
I know, it's not a good idea to build such dependencies between roles and different hosts. I don't know a better way, especially if you think about the situation, where you need to do some extra stuff after creating the vhost (f.e. configure the service via REST API, which needs the external fqdn). In case of two separate playbooks with "backend"-service and "reverseproxy"-service - then I need a third playbook for configuring "hanging" services. Also I'm not sure, if I can retrieve the correct backend URL in the reverse proxy role (only think about the HTTP scheme or paths). That sounds not easy, or?
Earlier I had separate roles for adding/removing vhosts to a reverseproxy. This roles didn't have dependencies, but I needed to duplicate several defaults and templates and vars etc. which isn't nice too. Then I've changed that to a single role. Of course - in my opinion, this isn't really that, what a "role" should be. A role is something like "webserver" or "reverseproxy" (a state). But not something like "add_vhost_to_reverseproxy" (a verb). This would be something like a playbook - but is calling a parameterized playbook via a role a good idea/possible? The main problem is, that the state of reverseproxy is the sum of all services in the inventory.
In case of that single included role, including it, starts also all dependent roles (configure custom, firewall, etc.). Nevertheless in that case I found out, that the delegation did not use the facts of the delegated host.
I tested that with the following example - the inventory:
all:
hosts:
server1:
my_var: a
server2:
my_var: b
children:
service:
hosts:
server1:
reverseproxy:
hosts:
server2:
And playbook which assigns a role-a to the group webserver. The role-a has a task like:
- block:
- setup:
- name: "Include role b on delegated {{ groups['reverseproxy'][0] }}"
include_role:
name: role-b
delegate_to: "{{ groups['reverseproxy'][0] }}"
delegate_facts: true # or false or omit - it has no effect on Ansible 2.9 and 2.10
And in role-b only outputing the my_var of the inventory will output
TASK [role-b : My_Var on server1] *******************
ok: [server1 -> <ip-of-server2>] =>
my_var: a
Which says me, that role-b that should be run on server2 has the facts of server1. So - configuring the "reverseproxy" service is done in context of the "backend"-service. Which would have several other issues - when you think about firewall-dependencies etc. I can avoid that, by using tags - but then I need to run the playbook not just with the tag of the service, but also with all tags I want to configure, and I cannot use include_tasks with args-apply-tags anymore inside a role that also includes other roles (the tags will applied to all subtasks...). I miss something like include_role but only that specific tags or ignore dependencies. This isn't a bug, but has possible side effects in case of delegate_to.
I'm not really sure, what is the question? The question is - what is a good way to handle dependencies between hosts and roles in Ansible - especially when they are not on the same host?
I am sure I do not fully understand your exact problem, but when I was dealing with load balancers I used a template. So this was my disable_workers playbook:
---
- hosts: "{{ ip_list | default( 'jboss' ) }}"
tasks:
- name: Tag JBoss service as 'disabled'
ec2_tag:
resource: "{{ ec2_id }}"
region: "{{ region }}"
state: present
tags:
State: 'disabled'
delegate_to: localhost
- action: setup
- hosts: httpd
become: yes
become_user: root
vars:
uriworkermap_file: "{{ httpd_conf_dir }}/uriworkermap.properties"
tasks:
- name: Refresh inventory cache
ec2_remote_facts:
region: "{{ region }}"
delegate_to: localhost
- name: Update uriworkermap.properties
template:
backup: yes
dest: "{{ uriworkermap_file }}"
mode: 0644
src: ./J2/uriworkermap.properties.j2
Do not expect this to work as-is. It was v1.8 on AWS hosts, and things may have changed.
But the point is to set user-defined facts, on each host, for that host's desired state (enabled, disabled, stopped), reload the facts, and then run the Jinja template that uses those facts.

Ansible - vars are not correctly propagated to handlers when role run in loop

I am asking for help with a problem of deploying multiple versions (different variables) of the app with the same role run from one playbook.
We have an app with multiple product families, which are different code versions. Each version has separate uWSGI vassal config and Nginx virtualhost config(/api/v2, /api/v3, ...).
The desired state would be to run playbook and configure the server with all versions specified.
Sadly, ansible's import_role/import_tasks can't be used with with_items, so include_role/include_tasks must be used (pitty because they do not honor role tags).
The include_role method would not be the biggest problem, but we use handlers to notify uWSGI touch to reload - on a code change, link change, virtualenv change, app_config change, ...).
But when using loop (with_items), the variables passed from the loop does not correctly propagate to handlers.
I tried this scenarios
playbook.yml - with_items loop inside the playbook
PROBLEM: Handler is run only for the first iteration of the loop.
#!/usr/bin/env ansible-playbook
# HAndler is run only once, from first notifier
- hosts: localhost
gather_facts: no
vars:
app_root: "/tmp/test_ansible"
app_versions:
- app_product_family: 1
app_release: "v1.0.2"
- app_product_family: 3
app_release: "v4.0.7"
tasks:
- name: Deploy multiple versions of app
include_role:
name: app
with_items: "{{ app_versions }}"
loop_control:
loop_var: app_version
vars:
app_product_family: "{{ app_version.app_product_family }}"
app_release: "{{ app_version.app_release }}"
tags:
- app
- app_debug
playbook_v2.yml - with_items loop inside role task
PROBLEM: Handler is run with the default value from "Defaults"
#!/usr/bin/env ansible-playbook
- hosts: localhost
gather_facts: no
roles:
- app_v2
vars:
app_v2_root: "/tmp/test_ansible_v2"
app_v2_versions:
- app_v2_product_family: 1
app_v2_release: "v1.0.2"
- app_v2_product_family: 3
app_v2_release: "v4.0.7"
Tasks roles/app_v2/main.yml
---
# Workaround because import_tasks can't be run with_items
- include_tasks: deploy.yml
when: app_v2_versions
with_items: "{{ app_v2_versions }}"
loop_control:
loop_var: app_v2
vars:
app_v2_product_family: "{{ app_v2.app_v2_product_family }}"
app_v2_release: "{{ app_v2.app_v2_release }}"
tags:
- app_v2
- app_v2_deploy
...
One idea was about writing a separate role for each product family, but they share nginx and uWSGI, so it will be lots of copy-pasting and sharing tasks (so tags would not work properly).
For now, I solved it with shell script wrapper, but this is not an ideal solution and does not work from Ansible tower.
Sample repo with tasks to reproduce problem (tested with ansible 2.4, 2.5, 2.6)
Any ideas & recommendations are very welcome.
The order of overrides for variables is broken for includes in Ansible. F.e. even set_fact in the included role will be shadowed by role defaults.
See this bug: https://github.com/ansible/ansible/issues/22025
It's closed but not fixed. My advice: use include and variables really carefully.
In practice I never use role includes with loop. If you need loop, include a tasklist in this loop (and that tasklist, in turn, may import_role).
Ok, It is a bug as #George Shuklin posted.
I will use my shell wrapper, which reads group_vars yaml and then runs the playbook multiple times according to the variable list length.
Sadly I hit multiple annoying bugs in ansible in last few weeks, kinda losing my trust in it ):
And probably everybody is using microservices and kubernetes, so need to speed up our migration (:

Ansible: applying role templates hierarchically

I'm planning to use Ansible to manage a fairly large set of Linux servers with a lot of variation among them.
The hosts will be grouped into roles with a hierarchy between them, there's a common role, on which every other depends, child-roles which depend on on common, and there may be grandchild-roles which depend on them, with no limit on dependency depth.
common
|___ role1
|___ role2
|___ role3
|___ role3.1
Each role has its own subset of templates for system configuration files.
roles/
common/
templates/
etc/
hosts
resolv.conf
tasks/
main.yml
system-settings.yml # imported in main.yml
role1/
templates/
etc/
hosts # role-specific version, overrides 'common'
httpd/
conf/
httpd.conf
tasks/
main.yml
system-settings.yml # imported in main.yml
role2/
role3/
role3.1/
The tricky part is that I'd like to apply these templates hierarchically, similarly to the way Ansible resolves variable scopes.
So, for a given child-role, the most specific version of a template file is applied, if a template does not exist in that role then apply the parent's version, searching up to the root level.
I have (sort of) achieved this by putting the following into the system-settings.yml for each role:
- name: Create directories
file:
path: /{{ item.path }}
state: directory
mode: '{{ item.mode }}'
with_filetree: templates/
when: item.state == 'directory'
- name: Template files
template:
src: '{{ item.src }}'
dest: /{{ item.path }}
mode: '{{ item.mode }}'
with_filetree: templates/
when: item.state == 'file'
- name: Recreate symlinks
file:
src: '{{ item.src }}'
dest: /{{ item.path }}
state: link
force: yes
mode: '{{ item.mode }}'
with_filetree: templates/
when: item.state == 'link'
It works, but unfortunately it applies all of the parent role's templates, then all of the child role's templates.
This is a waste of work and dangerous in case a playbook execution aborts for some reason, since the hosts may be left with generic versions of the system files.
So, for the above example tree, when applying role1 to a group of hosts Ansible:
applies /etc/hosts and /etc/resolv.conf from the common role
applies /etc/hosts and /etc/httpd/conf/httpd.conf from the role1 role
But I want it to do:
apply /etc/resolv.conf from the common role
apply /etc/hosts from the role1 role (the most specific version)
apply /etc/httpd/conf/httpd.conf from the role1 role
The template files should not be specifically listed in the tasks/playbooks due to management overhead, when adding a new file, or a role-specific version to override a generic one we don't want to have to change any playbooks or other configurations.
Is this achievable with Ansible?
I can see a possible solution using external scripts (as described in https://docs.ansible.com/ansible/2.4/playbooks_loops.html#iterating-over-the-results-of-a-program-execution), but I'm not sure how to pass down the role hierarchy into the script.
My thoughts:
First Choice:
Change the "common" file tasks to "common" handlers. The reason I picked handlers is we can conditionally include handlers. Also we have roles pre/post handlers. Notify what all tasks that are not done in role and what must be done can be in pre role handlers.
Second Choice:
Conditionally include tasks, I think lot of effort is needed to attach tags to included files.
E.g:
---
-name: task inclusion
Hosts: localhost
Gether_facts: false
Tasks:
Include: common.yaml
When: item | bool
A_list:
-true
-false
Hope this helps.
Regards
Sudhakar

Usage of variable and role in openstack-ansible

I am a newbie to Ansible and I started to study deploying Openstack with Ansible recently.
Here is the git of openstack-ansible:
https://github.com/openstack/openstack-ansible
From the tutorial:
https://docs.openstack.org/project-deploy-guide/openstack-ansible/newton/run-playbooks.html
there is one command running playbook:
openstack-ansible setup-hosts.yml
I read this playbook, it includes another playbook at line 16:
include: openstack-hosts-setup.yml
Then I read the included openstack-hosts-setup.yml and following is the snippet from it:
- name: Basic host setup
hosts: "{{ openstack_host_group|default('hosts') }}"
gather_facts: "{{ gather_facts | default(True) }}"
max_fail_percentage: 20
user: root
pre_tasks:
- name: Check for a supported Operating System
assert:
that:
- (ansible_distribution == 'Ubuntu' and ansible_distribution_release == 'xenial') or
(ansible_os_family == 'RedHat' and ansible_distribution_major_version == '7')
msg: "The only supported platforms for this release are Ubuntu 16.04 LTS (Xenial) and CentOS 7 (WIP)"
roles:
- role: "openstack_hosts"
tags:
- openstack-hosts
I was confused about roles here
roles:
- role: "openstack_hosts"
As far as I know, there should be directory structure when using roles like:
roles/
openstack_hosts/
files/
templates/
tasks/
handlers/
vars/
defaults/
meta/
However, I can't find this structure in this openstack-ansible project.
So I have no idea where is the role(openstack_hosts) in this playbook.
When running this playbook , an error occurred just expected:
ERROR! the role 'openstack_hosts' was not found in /opt/openstack-ansible/playbooks/roles:/root/ansible/roles:/opt/openstack-ansible/playbooks
Also, I was wondering where the variablescome from:
hosts: "{{ openstack_host_group|default('hosts') }}"
I thought the variable will be found from the directory vars or defaults
under roles directory.
Since there is no such directory, I have no idea where this variable comes from.
Could any tell me where should I find the variable openstack_host_group
and explain how the role is used here?
Or did I miss something or I was wrong with some concepts about Ansible?
Thanks!!
By the way, this is the first time I ask question.
If the question is not clear enough or something not proper, please let me know :). Thanks again.
I guess you didn't complete bootstrap-ansible.sh from this guide. So you don't have required roles installed.
As for openstack_host_group, it seems that it is optional variable, so you can override default host pattern (which is hosts) if you want with extra variable (like -e openstack_host_group=my_other_group).

Does Ansible duplicate role variables for role dependencies that enable "allow_duplicates"?

Does Ansible duplicate role variables for role dependencies that have enabled allow_duplicates?
For example, given a playbook that includes more than once role application-environment that allows duplicates, will Ansible create multiple copies of its variables?
meta/main.yml:
---
allow_duplicates: yes
dependencies:
- src: git+http://javasource/git/ansible/roles/organization
version: 1.1.0
vars/main.yml:
---
application_directory: "{{ organization.directory }}/{{ application_name }}"
application_component_directory: "{{ application_directory }}/{{ application_component_name }}"
If Ansible does not create multiple copies of these variables, how could I rework the role so that it can support multiple variables?
You may find some helpful information here:
About vars:
Anything in the vars directory of the role overrides previous versions of that variable in namespace.
About defaults:
Tasks in each role will see their own role’s defaults. Tasks defined outside of a role will see the last role’s defaults.

Resources