I have a string containing a jinja expression. How can I parse/evaluate/expand/ that string so that the expression is replaced the host variable running the role/task?
Detailed explanation:
I do have a proxmox cluster with multiple hosts. I want to manage their iptables with a dedicated role, but with separate data sources. The role should collect all variables that start with a specific prefix, collect their rules for filter and nat tables, sort them and write them into /etc/iptables/rules.v4via ansible.builtin.template.
Later I want to define multiple lxc/ and VMs inside group_vars, e.g. inventory/group_vars/proxmox/lxc_nginx.yml: ```
nginx_reverse_proxy:
host: '<some_cluster_node>'
network:
ip: 10.1.0.3
cidr: 24
iprules_nginx_reverse_proxy:
nat:
weight: 20
entries: |
-A PREROUTING -i {{ network_external_ip.external_device }} -p tcp --dport 80 -j DNAT --to-destination 10.1.0.3
with network_external_ip beeing defined for each node, in inventory/host_vars/some_cluster_node.yml:
network_external_ip:
ip: 1.1.1.1
gateway: 0.0.0.0
mask: 255.255.252.0
cidr: 22
external_device: ens3 # <<< this differs for each cluster node
now I collect all iprules, which might have been defined in multiple files, within my role and try to get them all converted into a sorted dict:
- name: basics/iptables | collect all rules
set_fact:
helper: |
{%- set filter = [] %}
{%- set nat = [] %}
{%- for entry in vars.items() %}
{%- if entry[0].startswith('iprules_') %}
{%- if entry[1].filter is defined %}
{%- set _ = filter.append(entry[1].filter) %}
{%- endif %}
{%- if entry[1].nat is defined %}
{%- set _ = nat.append(entry[1].nat) %}
{%- endif %}
{%- endif %}
{%- endfor %}
{%- set filter = filter | sort(attribute='weight') | map(attribute='entries') | join('\\n') %}
{%- set nat = nat | sort(attribute='weight') | map(attribute='entries') | join('\\n') -%}
filter: "{{ filter }}"
nat: "{{ nat }}"
- name: basics/iptables | create iptables dict
set_fact:
iptables_rules: "{{ helper | from_yaml }}"
- name: debug
debug:
var: iptables_rules
- name: basics/iptables | write ip rules
ansible.builtin.template:
src: rules.v4.j2
dest: /etc/iptables/rules.v4
which results in:
TASK [basics/iptables : basics/iptables | collect all rules] *****************************************************************************************************************************************************************************************************
ok: [<some_cluster_node>]
TASK [basics/iptables : basics/iptables | create iptables dict] **************************************************************************************************************************************************************************************************
ok: [<some_cluster_node>]
TASK [basics/iptables : debug] ***********************************************************************************************************************************************************************************************************************************
ok: [<some_cluster_node>] =>
iptables_rules:
filter: ''
nat: '-A PREROUTING -i {{ network_external_ip.external_device }} -p tcp --dport 80 -j DNAT --to-destination 10.1.0.3 '
Meaning, {{ network_external_ip.external_device }} never gets evaluated to "ens3". It stays the same, as in the inventory. How can I get this expression being parsed?
Information: The shown dicts here are WIP and do not fulfill certain criteria (like I would write all iprules_* into all hosts, regardless if they even host the lxc), but without the basics working this is no a concern right now.
PS: ansible version used
~ $ ansible --version
ansible [core 2.13.5]
config file = /home/apit/.ansible.cfg
configured module search path = ['/home/apit/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python3.10/site-packages/ansible
ansible collection location = /home/apit/.ansible/collections:/usr/share/ansible/collections
executable location = /usr/lib/python-exec/python3.10/ansible
python version = 3.10.8 (main, Nov 27 2022, 20:55:16) [GCC 11.3.0]
jinja version = 3.1.2
libyaml = True
Related
I try to build elasticsearch cluster with ansible and I have a problem with jinja2.
How can I set in jinja2 IP addresses of other hosts?
I have in my inventory.ini:
[elasticsearch]
192.168.0.1
192.168.0.2
192.168.0.3
and I want in jinja2 template to pass two addresses 192.168.0.2,192.168.0.3 when there is more hosts than 1, to look something like that:
- discovery.seeds_hosts=192.168.0.2,192.168.0.3
but when is one host in inventory.ini
it should look like this:
- discovery.seeds_hosts=192.168.0.1
I tried with something like this(hostnames):
{% for host in groups['elasticsearch'] %}{% if host == ansible_host %}{% else %}{%if loop.index0 > 0 %}{% endif %}elasticsearch-{{ loop.index }}{% if not loop.last %},{% endif %}{% endif %}{% endfor %}
and it works only for more than 1 host in elasticsearch group. If in the inventory.ini is only one host, the variable discovery.seeds_hosts will be empty.
Ansible version:
ansible [core 2.14.1]
config file = /etc/ansible/ansible.cfg
configured module search path = ['/home/ansible/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/lib/python3.10/site-packages/ansible
ansible collection location = /home/ansible/.ansible/collections:/usr/share/ansible/collections
executable location = /usr/local/bin/ansible
python version = 3.10.9 (main, Dec 8 2022, 01:46:27) [GCC 10.2.1 20210110] (/usr/local/bin/python)
jinja version = 3.1.2
libyaml = True
You could use a conditional that checks the number of hosts:
{% if groups.elasticsearch|length > 1 %}
- discovery.seeds_hosts={{ groups.elasticsearch[1:]|join(',') }}
{% else %}
- discovery.seeds_hosts={{ groups.elasticsearch.0 }}
{% endif %}
I've tested this locally, and with multiple hosts in the inventory:
[elasticsearch]
192.168.0.1
192.168.0.2
192.168.0.3
This produces:
- discovery.seeds_hosts=192.168.0.2,192.168.0.3
With a single host in the inventory:
[elasticsearch]
192.168.0.1
This produces:
- discovery.seeds_hosts=192.168.0.1
I want to define a dictionary variable that various host groups can add their own keys to in group_vars (not using set_fact). E.g. something like this:
group_vars\ftp_servers.yml:
important_ports:
ftp: 21
group_vars\web_servers.yml:
important_ports:
http: 80
so that when run on a server with both of these roles the dictionary is combined, i.e. important_ports =
{
ftp: 21,
http: 80
}
This is exactly what hash_behaviour = merge does, but it's deprecated and will be removed in Ansible 2.13. How do I achieve the same thing without that?
The only solution I've seen recommended is to use the combine filter:
set_fact:
important_ports: "{{ important_ports | combine({ http: 80 }) }}"
This works in a set_fact task, but fails in group_vars with "recursive loop detected in template string: {{ important_ports | combine({ http: 80 }) }}"
I even tried initialising the variable to empty dictionary (important_ports: {}) in group_vars/all, which is supposed to be evaluated before other group_vars, but it still gives the same error.
A (not so clean...) possible solution to this specific problem. This is based on convention where you use the group name in the var name for specific group ports.
Note that if you redefine the same port name with a different value, the latest loaded group will win.
Given the following all-in-one inventory placed in inventories/mergegroups/hosts.yml
---
all:
vars:
ansible_connection: local
important_ports: >-
{%- set result={} -%}
{%- for group in groups -%}
{{ result.update(lookup('vars', group + '_important_ports', default={})) }}
{%- endfor -%}
{{ result }}
ftp_servers:
vars:
ftp_servers_important_ports:
ftp: 21
hosts:
a:
b:
web_servers:
vars:
web_servers_important_ports:
http: 80
hosts:
a:
other_group:
vars:
other_group_important_ports:
mysql: 3306
hosts:
a:
b:
no_port_group:
# as you can see no port definition here.
hosts:
a:
b:
c:
We get the following results for the 3 different host:
$ ansible -i inventories/mergegroups/ a -m debug -a msg="{{ important_ports }}"
a | SUCCESS => {
"msg": {
"ftp": 21,
"http": 80,
"mysql": 3306
}
}
$ ansible -i inventories/mergegroups/ b -m debug -a msg="{{ important_ports }}"
b | SUCCESS => {
"msg": {
"ftp": 21,
"mysql": 3306
}
}
$ ansible -i inventories/mergegroups/ c -m debug -a msg="{{ important_ports }}"
c | SUCCESS => {
"msg": {}
}
This is a playbook that connects with all the servers in my inventory file, and makes a note of the server ip and mount point information of hosts where mount point usage exceeds 80% and it writes to a text file on the localhost (ansible-controller).
- hosts: all
tasks:
- shell:
cmd: df -h | sed 's/%//g' | awk '$5 > 80 {if (NR > 1) print $5"%",$6}'
register: disk_stat
- debug:
var: disk_stat
- file:
path: /home/app/space_report_{{ td }}.txt
state: touch
run_once: true
delegate_to: localhost
- shell: echo -e "{{ ansible_host }} '\n' {{ disk_stat.stdout_lines| to_nice_yaml }}" >> /home/thor/space_report_{{ td }}.txt
args:
executable: /bin/bash
delegate_to: localhost
I was wondering if I could create a jinja2 template and bring the playbook down to one task. I am stuck at integrating a shell command inside the jinja2 template and I am not sure if it is possible. Please advise.
- hosts: all
tasks:
- template:
src: monitor.txt.j2
dest: /home/app/playbooks/monitor.txt
delegate_to: localhost
monitor.txt.j2
{% for host in groups['all'] %}
{{ hostvars[host].ansible_host }}
--shell command--
{% endfor %}
As I say in my comment under your question, while it is possible to use shell or command modules, Ansible is a configuration / automation tool, so it's better to forget your shell coding / logic to use native ansible functionalities, that'll ease the tasks / playbook writing.
For instance, it's no needed to do a df because ansible when connecting will gather facts about the target, including devices and their capacity and current usage so you can use that directly.
For the jinja question, you can use the module copy and pass directly jinja code in the option content on this module:
- name: Trigger a tasks on hosts to gather facts
hosts: all
tasks:
- copy:
dest: /home/app/playbooks/monitor.txt
content: |
{% for host in groups['all'] %}
{% for dev in hostvars[host].ansible_mounts %}
{% if (dev.block_used / dev.block_total * 100 ) > 80 %} {{ dev.block_used / dev.block_total * 100 }} {{ dev.mount }} {% endif %}
{% endfor %}
{% endfor %}
run_once: true
delegate_to: localhost
I've got a playbook which needs to run against my entire inventory, with a list of hostnames as an extra variable (target_hosts).
The hosts in target_hosts all have a group_id hostvar defined on them. I use the whole inventory because some ancillary hosts which correspond to the group_id var need per-group configuration to match in one section.
There will often be multiple group_id values associated with the hosts in the target_hosts list. I need to select the correct inventory group of ancillary hosts and import/run a playbook to configure both sets of servers partway through the main playbook.
This is what I currently do:
include_playbook: group-configure.yaml
vars:
src_hosts: "group-{{ group_id }}-ancillary-1"
dest_hosts: "{{ target_hosts }}"
I currently have to manually separate the target_hosts by group_id manually, then run the main playbook once for each. This has tons of unnecessary overhead.
What I really want to execute is this:
for each group of hosts from `target_hosts` with the same `group_id` hostvar:
import and run group-configure.yaml with:
src_hosts: "ancillary-{{ group_id }}"
target_hosts: restricted to those with that value of `group_id`'
How can I do this? If the current way this is structured won't work, what's the best alternative approach?
I am pretty sure the add_host: combined with groupby is what you are looking for, which will allow you to roll up those hosts by their attribute, and then run the playbook against them as if that group was defined already:
- hosts: localhost
connection: local
gather_facts: no
become: no
vars:
list_of_name_groups: >-
{%- set results = [] -%}
{%- for g_id, items in (dict(hostvars) | dict2items | groupby("value.group_id")) -%}
{%- for hostname in (items | map(attribute="key") | list) -%}
{%- set _ = results.append({"group_id": g_id, "hostname": hostname}) -%}
{%- endfor -%}
{%- endfor -%}
{{ results }}
tasks:
- add_host:
name: '{{ item.hostname }}'
groups: ancillary-{{ item.group_id }}
with_items: '{{ list_of_name_groups }}'
- hosts: ancillary-my-awesome-groupid
# etc etc
I have a static variable that holds IP addresses of all hosts in my inventory (how to obtain this dynamically is a separate question) like this:
server_ips:
www1.example.com:
ipv4:
- 192.168.0.10
- 192.168.0.11
ipv6:
- '2a00:abcd:1234::100'
- '2a00:abcd:1234::101'
www2.example.com:
ipv4:
ipv6:
- '2a00:abcd:1234::200'
- '2a00:abcd:1234::201'
db1.example.com:
ipv4:
- 192.168.1.2
ipv6:
These names align with hosts in my inventory:
[webservers]
www1.example.com
www2.example.com
[dbservers]
db1.example.com
On a task that's run on the dbservers group, I need a list of all IPs from the webserver group (this makes querying facts directly tricky as facts may not have been gathered for those hosts) - in this case it would need to extract:
- 192.168.0.10
- 192.168.0.11
- '2a00:abcd:1234::100'
- '2a00:abcd:1234::101'
- '2a00:abcd:1234::200'
- '2a00:abcd:1234::201'
The tasks will do things like configure firewall and DB access, along the lines of:
- name: Allow web server access to DB server
ufw:
rule: allow
name: mysql
from_ip: "{{ item }}"
loop: "{{ <loop expression goes here> }}"
It's what's in the loop expression that I'm having trouble with.
There are two parts to the query: extract the list of hosts, and then gather the ip addresses - doing it separately for ipv4 and ipv6 is fine.
I can get part way there with expressions like:
{{ server_ips | map('extract', groups['webservers']) }}
{{ server_ips | intersect(groups['webservers']) }}
However, both of these appear to flatten the result so though they find the right items, the ipv4 and ipv6 list elements are not there, so I can't proceed to the next step. Swapping the lists in these didn't help either.
The subelements lookup seems a good way to get the IPs parts (though I actually need sub-subelements) and skip empty entries, but I can't see how to get to that point.
How should I do this lookup?
You try to reinvent functionality, Ansible provides already. You define your DIY inventory although Ansible has already an inventory. And you define your DIY inventory iteration although Ansible knows how to iterate over its inventory.
If you want to assign data to individual hosts use the host_vars directory as shown in the Best Practices.
host_vars/www1.example.com.yml:
ipv4:
- 192.168.0.10
- 192.168.0.11
ipv6:
- '2a00:abcd:1234::100'
- '2a00:abcd:1234::101'
host_vars/www2.example.com.yml:
ipv4:
ipv6:
- '2a00:abcd:1234::200'
- '2a00:abcd:1234::201'
Then you define a task for each host and use the {{ipv4}} or {{ipv6}} lists for anything you want to do.
If you need to execute actions on a different host like a firewall, use Ansible's delegation.
This extracts all IP addresses from your server_ips dictionary:
- hosts: localhost
connection: local
gather_facts: no
vars:
server_ips:
www1.example.com:
ipv4:
- 192.168.0.10
- 192.168.0.11
ipv6:
- '2a00:abcd:1234::100'
- '2a00:abcd:1234::101'
www2.example.com:
ipv4:
ipv6:
- '2a00:abcd:1234::200'
- '2a00:abcd:1234::201'
xyz.example.com:
ipv4:
- 192.168.1.2
ipv6:
ipv4: >-
{% set ipv4 = [] -%}
{% for ips in server_ips.values() | selectattr ('ipv4') | map (attribute='ipv4') | list -%}
{% for ip in ips -%}
{% set _ = ipv4.append(ip) -%}
{% endfor -%}
{% endfor -%}
{{ ipv4 }}
ipv6: >-
{% set ipv6 = [] -%}
{% for ips in server_ips.values() | selectattr ('ipv6') | map (attribute='ipv6') | list -%}
{% for ip in ips -%}
{% set _ = ipv6.append(ip) -%}
{% endfor -%}
{% endfor -%}
{{ ipv6 }}
ips: >-
{{ ipv4 + ipv6 }}
tasks:
- debug: var=server_ips
- debug: var=ipv4
- debug: var=ipv6
- debug: var=ips
But in order to build firewall rules you have to build a cross product. You have to iterate for each destination over all sources to get all rules.