Loop over nested dict - ansible

I need to iterate over a complex dict and apply properties from one key to another
I have tried many possibilities, such as with_nested, and with_subelements
I have an object as such
group_hosts:
- group:
name: a
hosts:
- host1
- host2
ports:
- 22
- 80
- group:
name: b
hosts:
- host3
- host4
ports:
- 22
- 80
And, given a host, I need to associate all the ports to it (host1 will have ports 22 and 80 associated to is, for example), so that in a later moment, while iterating I can use the wait_for module to check if the ports are open
The only workaround I found was to repeat the hostname for how many ports it had (by so doing I removed an extra list to loop)
More explicitly my var object became this
group_hosts:
- group:
name: a
hosts:
- name: host1
port: 80
- name: host1
port: 22
- name: host2
port: 22
- name: host2
port: 80
- group:
name: b
hosts:
- name: host3
port: 20
- name: host4
port: 2222
And my play this:
- name: traverse dict
debug:
msg: "group: {{item.0.group.name}} host is: {{item.1.name}} port is: {{item.1.port}}"
loop: "{{ group_hosts | subelements('group.hosts') | list }}"
But I don't like this workaround as I've had to modify the dict object by writing it in a less efficient way.
So, given the first dict object, how can I loop hosts and associate ports to them?
Meaning:
I want that given host1, I check its 22 and 80 ports, host 2 the same.
So:
hosts group a:
host1 : check ports 22, 80
host2: check ports 22, 80
hosts group b:
same as above
I already know how to "check" the ports on some host, my question is how to iterate such an object

Let's simplify the dictionary. The tasks below
- set_fact:
hosts2: "{{ hosts2|default({})|
combine({item.1:
{'name': item.0.group.name,
'ports': item.0.group.ports}}) }}"
loop: "{{ lookup('subelements', group_hosts, 'group.hosts') }}"
- debug:
var: hosts2
give a dictionary that can be easily iterated.
"hosts2": {
"host1": {
"name": "a",
"ports": [
22,
80
]
},
"host2": {
"name": "a",
"ports": [
22,
80
]
},
"host3": {
"name": "b",
"ports": [
22,
80
]
},
"host4": {
"name": "b",
"ports": [
22,
80
]
}
}

Related

Use variable group in playbook like target hosts

I hava this situation:
A play that run in localhost use include_task for create, on the fly with add_host, two sub-group extracting only one host from two group that are present in the inventory file.
Another play in the same yaml file use this group as hosts (host:sub-group).
This is the inventory file:
all:
children:
group_one:
hosts:
hostA01:
ansible_host: host1a
hostA02:
ansible_host: host2a
hostA03:
ansible_host: host3a
vars:
cluster: hosta
vip: 192.168.10.10
home: /cluster/hosta
user: usr_hosta
pass: pass_hosta
group_two:
hosts:
hostB01:
ansible_host: host1b
hostB02:
ansible_host: host2b
hostB03:
ansible_host: host3b
vars:
cluster: hostb
vip: 192.168.10.20
home: /cluster/hostb
user: usr_hostb
pass: pass_hostb
other groups...
I have created the sub groups, with add_host, for each group in inventoty file. The name the sub-groups add the prefix "sub-" to inventory original groups, like sub-(one/two/etc..)
In hostvars i retrieve this situation:
"groups": {
"all": [
"hostA01",
"hostA02",
"hostA03",
"hostB01",
"hostB02",
"hostB03",
"other_host_from _other groups"
],
"group_one": [
"hostA01",
"hostA02",
"hostA03"
],
"group_two": [
"hostB01",
"hostB02",
"hostB03"
],
"other_group": [
"other_host",
.....
],
"sub-group_one": [
"hostA01"
],
"sub_group_two": [
"hostB01"
],
"sub-other_group": [
"other_first_host"
],
"ungrouped": []
},
"vars_for_group": {
"group_one": {
"cluster: hosta
"vip: 192.168.10.10
"home: /cluster/hosta
"user: usr_hosta
"pass: pass_hosta,
"ansible_host": "host1a",
"host": "hostA01"
},
"group_two": {
"cluster: hostb
"vip: 192.168.10.20
"home: /cluster/hostb
"user: usr_hostb
"pass: pass_hostb,
"ansible_host": "host1b",
"host": "hostB01"
},
"otehr_groups": {
.......
},
},
"inventory_hostname": "127.0.0.1",
"inventory_hostname_short": "127",
"module_setup": true,
"playbook_dir": ""/home/foo/playbook",
"choice": "'sub-two'"
}
}
The environment variable "choice" in the last line of hostvars derives from other tool and indicates the group on which the final user wants to operate (one, two, ..., all).
Now, my playbook is:
---
- hosts: 127.0.0.1
become: yes
gather_facts: yes
remote_user: root
tasks:
- name: include news_groups
ansible.builtin.include_tasks:
newsgroups.yaml
vars:
choice: "{{ lookup('env','CHOICE') }}"
- hosts: "{{ hostvars['localhost']['groups']['{{ hostvars['localhost']['choice'] }}'] }}"
name: second_play
become: yes
gather_facts: no
remote_user: root
tasks:
- hosts: all
name: other play
gather_facts: no
vars:
other_vars:...
tasks:
.....
Unfortunately this line don't work.
- hosts: "{{ hostvars['localhost']['groups']['{{ hostvars['localhost']['choice'] }}'] }}"
I have made several attempts with many different configuration and different sintax (without {{...}}, with or without "double quotes" and 'single quote' but it always seems syntactically wrong or still cannot find the group indicated by the variable. Or, maybe could it also be that the approach is wrong?
Any suggestions?
Thanks in advance

Create a list of dictionaries from another list and a static value, using Jinja2

I need to create a list of dictionaries using only Jinja2 from another list, as input.
One key/value pair is static and always the same, the other changes value.
Input:
targets: ["abc", "qwe", "def"]
I know that the server will always be xyz.
Final
connections:
- { "target": "abc", "server": "xyz" }
- { "target": "qwe", "server": "xyz" }
- { "target": "def", "server": "xyz" }
I tried this:
"{{ dict(targets | zip_longest([], fillvalue='xyz')) }}"
But, that just takes one for key and the other for value.
You were quite close.
But you'll need a product rather than a zip_longest to have the same element repeating for all the element of your targets list.
You were also missing a dict2items to close the gap and have the resulting list out of your dictionary.
Which gives the task:
- set_fact:
connections: >-
{{
dict(targets | product(['xyz']))
| dict2items(key_name='target', value_name='server')
}}
Given the playbook:
- hosts: localhost
gather_facts: no
tasks:
- set_fact:
connections: >-
{{
dict(targets | product(['xyz']))
| dict2items(key_name='target', value_name='server')
}}
vars:
targets:
- abc
- qwe
- def
- debug:
var: connections
This yields:
ok: [localhost] =>
connections:
- server: xyz
target: abc
- server: xyz
target: qwe
- server: xyz
target: def
just use set_fact with the right loop:
- name: testplaybook jinja2
hosts: localhost
gather_facts: no
vars:
targets: ["abc", "qwe", "def"]
tasks:
- name: DEFINE VARIABLE SPINE
set_fact:
connections: "{{ connections | d([]) + [ {'target': item, 'server': _server} ] }}"
loop: "{{ targets }}"
vars:
_server: xyz
- name: display
debug:
var: connections
result:
connections:
- server: xyz
target: abc
- server: xyz
target: qwe
- server: xyz
target: def

how to use ansible with_subelements

Here is my database yaml file:
apn_operation:
action: create
epg_version: 4G
apn_name: internet_jack
access-restrictions selection-mode: public
routing-instance: sgi_nat
pgw-enabled: yes
uplink-dscp-map: test
allow-rule-space:
- RS_Mobile
- RS_Mobile_OCC
name-server:
- ip_address: 10.0.0.1
priority: 10
- ip_address: 10.0.0.2
priority: 20
Here is my playbook:
- hosts: all
gather_facts: no
vars_files:
- apn_operation.yaml
tasks:
- name: show content
debug:
msg: "{{item}}"
with_subelements:
- "{{apn_operation}}"
- name-server
when I ran the playbook. I got the following error:
TASK [show content] ******************************************************************************************************************************************************************************************
fatal: [11RRvEPG01]: FAILED! => {"msg": "subelements lookup expects a dictionary, got 'OPTUS-QCIDSCP'"}
I don't know what the problem is. can anyone help?
apn_operation is dictionary. It's not possible to iterate a dictionary. It's possible to use dict2items but this is not what you're looking for.
with_subelements is able to iterate a list where each item is a dictionary that comprises another list. For example
apn_operation:
- action: create
name-server:
- ip_address: 10.0.0.1
priority: 10
- ip_address: 10.0.0.2
priority: 20
works as expected and gives
"msg": [
{
"action": "create"
},
{
"ip_address": "10.0.0.1",
"priority": 10
}
]
"msg": [
{
"action": "create"
},
{
"ip_address": "10.0.0.2",
"priority": 20
}
]

passing numeric parameter on ansible variable array

I got this ansible variable array, weblogic[1].name which will give me the name of the second array "manageServer1".
weblogic: [
{
name: "adminServer"
address: "1.1.1.1"
port: 1701
ssl: 1702
},
{
name: "manageServer1"
address: "1.1.1.2"
port: 1703
ssl: 1704
},
]
How can I pass parameter x=1 on the array, this one won't work, weblogic[x].name or weblogic['x'].name?
I'm working on Ansible 2.6-2.7.
Commas are missing in the lists. See example below.
> cat test.yml
---
- hosts: localhost
gather_facts: no
vars:
weblogic:
- { name: "adminServer", address: "1.1.1.1", port: 1701, ssl: 1702 }
- { name: "manageServer1", address: "1.1.1.2", port: 1703, ssl: 1704 }
tasks:
- debug: var=weblogic[item].name
loop:
- 0
- 1
> ansible-playbook test.yml | grep weblogic
"weblogic[item].name": "adminServer"
"weblogic[item].name": "manageServer1"

Ansible: Updating values in a list of dictionaries

In my payload, I have a variable that is actually a list of dictionaries, such as this one:
myvar:
- name: name1
ip_addresses:
- 10.10.10.10
- 11.11.11.11
nat_destination_addresses:
- host: 12.12.12.12
destination: 13.13.13.13
- host: 14.14.14.14
destination: 15.15.15.15
nat_source_address: 16.16.16.16
applications:
- protocol: tcp
port: 8302
- protocol: udp
port: 2000
- protocol: tcp
port: 2000-5600
- name: name2
ip_addresses:
- 17.17.17.17
- name: name3
ip_addresses:
- 18.18.18.18
- 19.19.19.19
All the values for each element in myvar are optional, except for the name, which is mandatory.
I am trying to pad the ip addresses (ip_addresses, nat_destination_addresses and nat_source_address) and ports. The ports should have a length of five characters with zeroes at the beginning (2000 becomes 02000 and 2000-5600 becomes 02000-05600) and the ip addresses should have three characters for each subsection (18.18.18.18 becomes 018.018.018.018).
The problem that I have is that I am not able to change only subsections of myvar.
I have read other questions here, such as:
merging dictionaries in ansible
Using set_facts and with_items together in Ansible
But to no avail. No matter what I do, I am not able to keep the original dictionary, I end up with a list of ip_addresses if I use the combine filter from the second StackOverflow link.
The expected result is the original myvar variable with updated ip addresses and ports.
This seems like a good time to throw your logic into a custom Ansible module. It doesn't have to be anything fancy, for example:
from ansible.module_utils.basic import AnsibleModule
def pad_addr(addr):
return '.'.join('%03d' % int(x) for x in addr.split('.'))
def main():
module_args = dict(
data=dict(type='list', required=True),
)
module = AnsibleModule(
argument_spec=module_args,
supports_check_mode=True
)
data = module.params['data']
for d in data:
if 'ip_addresses' in d:
d['ip_addresses'] = [pad_addr(x) for x in d['ip_addresses']]
if 'nat_destination_addresses' in d:
for dest in d['nat_destination_addresses']:
dest['host'] = pad_addr(dest['host'])
dest['destination'] = pad_addr(dest['destination'])
if 'nat_source_address' in d:
d['nat_source_address'] = pad_addr(d['nat_source_address'])
if 'applications' in d:
for service in d['applications']:
service['port'] = '%05d' % service['port']
module.exit_json(changed=False,
result=data)
if __name__ == '__main__':
main()
If I drop the above into library/pad_data.py and then run the following playbook:
- hosts: localhost
gather_facts: false
vars:
myvar:
- name: name1
ip_addresses:
- 10.10.10.10
- 11.11.11.11
nat_destination_addresses:
- host: 12.12.12.12
destination: 13.13.13.13
- host: 14.14.14.14
destination: 15.15.15.15
nat_source_address: 16.16.16.16
applications:
- protocol: tcp
port: 8302
- protocol: udp
port: 2000
- protocol: tcp
port: 2000
- name: name2
ip_addresses:
- 17.17.17.17
- name: name3
ip_addresses:
- 18.18.18.18
- 19.19.19.19
tasks:
- pad_data:
data: "{{ myvar }}"
register: padded
- debug:
var: padded.result
I get as the result:
TASK [debug] *******************************************************************
ok: [localhost] => {
"padded.result": [
{
"applications": [
{
"port": "08302",
"protocol": "tcp"
},
{
"port": "02000",
"protocol": "udp"
},
{
"port": "02000",
"protocol": "tcp"
}
],
"ip_addresses": [
"010.010.010.010",
"011.011.011.011"
],
"name": "name1",
"nat_destination_addresses": [
{
"destination": "013.013.013.013",
"host": "012.012.012.012"
},
{
"destination": "015.015.015.015",
"host": "014.014.014.014"
}
],
"nat_source_address": "016.016.016.016"
},
{
"ip_addresses": [
"017.017.017.017"
],
"name": "name2"
},
{
"ip_addresses": [
"018.018.018.018",
"019.019.019.019"
],
"name": "name3"
}
]
}
Larsks' answer was on point and is probably the best solution for most people, but my requirements are to limit the number of modules created with Python for this project, so here is my workaround for reference purposes.
Basically, what I do in this sample is:
Locally:
I take myvar, I output it to a yml file (with '---' at the top of the file and making sure that myvar is still set as the key.
Using regexp and the replace module, I replace the parts of the file that I want to replace.
On all of my hosts:
I reload the (now) properly formatted myvar and replace the old myvar variable using include_vars
---
- name: Customer {{ customer_id }} - Format the ip addresses and ports
hosts: localhost
gather_facts: no
connection: local
tags: [format_vars]
tasks:
- name: Copy the 'myvar' content to a local file to allow ip addresses
and ports formatting
copy:
content: "---\n{{ { 'myvar': myvar} | to_nice_yaml(indent=2) }}"
dest: "{{ formatted_myvar_file }}"
- name: Pad all ip addresses parts with two zeroes to ensure that all parts have at least three numbers
replace:
path: "{{ formatted_myvar_file }}"
regexp: '(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})'
replace: '00\1.00\2.00\3.00\4'
- name: Remove extra zeroes from ip addresses to ensure that all of their parts have exactly three numbers
replace:
path: "{{ formatted_myvar_file }}"
regexp: '\d{0,2}(\d{3})\.\d{0,2}(\d{3})\.\d{0,2}(\d{3})\.\d{0,2}(\d{3})'
replace: '\1.\2.\3.\4'
- name: Pad all ports with four zeroes to ensure that they all have at least five numbers
replace:
path: "{{ formatted_myvar_file }}"
regexp: 'port: (\d{1,5})'
replace: 'port: 0000\1'
- name: Remove extra zeroes from ports to ensure that they all have exactly five numbers
replace:
path: "{{ formatted_myvar_file }}"
regexp: 'port: \d{0,4}(\d{5})'
replace: 'port: \1'
- name: Pad all second parts of port ranges with four zeroes to ensure that they all have at least five numbers
replace:
path: "{{ formatted_myvar_file }}"
regexp: 'port: (\d{5})-(\d{1,5})'
replace: 'port: \1-0000\2'
- name: Remove extra zeroes from second parts of port ranges to ensure that they all have exactly five numbers
replace:
path: "{{ formatted_myvar_file }}"
regexp: 'port: (\d{5})-\d{0,4}(\d{5})'
replace: 'port: \1-\2'
- name: Customer {{ customer_id }} - Load the properly formatted ip addresses and ports
hosts: localhost:all-n7k:srx-clu:all-mx80:all-vsrx
gather_facts: no
connection: local
tags: [format_vars]
tasks:
- include_vars:
file: "{{ formatted_myvar_file }}"
ignore_errors: yes

Resources