Iterate dictionary in dictionary - ansible

I have a json structure like this:
{
"group_a": {
"setting_one": "text one",
"setting_two": "text two"
},
"group_b": {
"setting_three": "text three",
"setting_four": "text four"
}
}
And I need to do iterate over the dicts inside the dicts and create tasks, kinda like this:
- name: Task group_a setting_one
...
- name: Task group_a setting_two
...
- name: Task group_b setting_three
...
- name: Task group_b setting_four
...
I was hoping to make something like this:
- name: Insert {{ group }} {{ setting }}
ini_file:
path: settings.ini
section: "{{ group }}"
option: "{{ setting }}"
value: "{{ setting_value }}"
with_nested:
- "{{ data.keys() }}" # Can I get this key and use in the next loop?
- "{{ data.key.keys() }}"
How can I do this? I have been messing about with loop, with_items, with_dict etc, but I can't manage to get it exactly the way I need. I also tried with_nested, but that does not seem to do what I need either.
Kind regards, Jonas

Let's reduce the structure to one with the same piece of information
my_data:
group_a:
- one
- two
group_b:
- three
- four
Then the play below
- debug:
msg: "{{ item.0.key }} setting_{{ item.1 }}"
with_subelements:
- "{{ my_data|dict2items }}"
- value
gives:
"msg": "group_b setting_three"
"msg": "group_b setting_four"
"msg": "group_a setting_one"
"msg": "group_a setting_two"
It's not straightforward to use the original data. The play below separates the keys and values
tasks:
- set_fact:
my_groups: "{{ my_data.keys()}}"
- debug:
msg: "{{ item }} {{ my_data|dict2items|json_query(my_query)|to_yaml }}"
loop: "{{ my_groups }}"
vars:
my_query: "[?key=='{{ item }}'].value"
and gives:
"msg": "group_b - - {setting_three: text three}\n - {setting_four: text four}\n"
"msg": "group_a - - {setting_one: text one}\n - {setting_two: text two}\n"
Multi-step loops are needed to proceed. For example the play below
- set_fact:
my_data1: |
{% for group, values in my_data.items() %}
{% for value in values %}
{% for item_key, item_val in value.items() %}
Task {{ group }} {{ item_key }},
{% endfor %}{% endfor %}{% endfor %}
- debug:
msg: "{{ my_data1.split('\n') }}"
gives:
"msg": [
"Task group_b setting_three,",
"Task group_b setting_four,",
"Task group_a setting_one,",
"Task group_a setting_two,",
""
]

Related

list of dictionary iteration

I have local.yml file where I am passing variable as list
roles:
- role: book
vars:
data:
- 'std1'
- 'std2'
vars.yml mentioned as below
book_data:
"std1": "std1"
"std2": "std2"
"std1":
- '1a'
- '1b'
"std2":
- '2a'
- '2b'
I want to iterate data over list mentioned in local.yml and print value from vars.yml. I have written code which print only list(std1,std2)
- name: set fact
set_fact:
standard: "{{ lookup('list', data) }}"
- name: get standard
debug: msg="{{ item }}"
with_items: "{{ standard }}"
Expected output would be:
list mentioned in local.yml would call vars/main.yml content and fetch content.
Example: whenever I would call str1 it would fetch 1a and 1b,
str2 it would fetch 2a and 2b
Below sample code is working:
- name: get standard data
debug:
msg="{{ item.key }}"
with_items: "{{ lookup('dict', book_data) }}"
but I want specific the iteration call from local.yaml to vars/main.yml. Dont want to use vars/main.yml variable directly into playbook
Given the variables
data: [x, y]
book_data:
x: std1
y: std2
std1: [1a, 1b]
std2: [2a, 2b]
Iterate data, for example
- debug:
msg: "{{ lookup('vars', book_data[item]) }}"
loop: "{{ data }}"
gives (abridged)
msg:
- 1a
- 1b
msg:
- 2a
- 2b
, or create the structure
book_data_lists: |
{% for i in data %}
{{ i }}: {{ lookup('vars', book_data[i]) }}
{% endfor %}
gives
book_data_lists: |-
x: ['1a', '1b']
y: ['2a', '2b']
Iterate the dictionary with subelements
- debug:
msg: "{{ item.0.key }} {{ item.1 }}"
with_subelements:
- "{{ book_data_lists|from_yaml|dict2items }}"
- value
gives (abridged)
msg: x 1a
msg: x 1b
msg: y 2a
msg: y 2b
Example of a complete playbook for testing
- hosts: localhost
vars:
data: [x, y]
book_data:
x: std1
y: std2
std1: [1a, 1b]
std2: [2a, 2b]
book_data_lists: |
{% for i in data %}
{{ i }}: {{ lookup('vars', book_data[i]) }}
{% endfor %}
tasks:
- debug:
msg: "{{ item.key }}"
loop: "{{ lookup('dict', book_data) }}"
- debug:
msg: "{{ item.key }}"
loop: "{{ book_data|dict2items }}"
- debug:
msg: "{{ lookup('vars', book_data[item]) }}"
loop: "{{ data }}"
- debug:
var: book_data_lists
- debug:
msg: "{{ item.0.key }} {{ item.1 }}"
with_subelements:
- "{{ book_data_lists|from_yaml|dict2items }}"
- value

Is there any way to loop through Ansible list of dictionary registered variable in combination with Jinja2?

In my inventory I have 3 servers in a group. I want to be able to increase that size in the future so I can add more nodes into the network and generate the template with Jinja2.
- name: Gathering API results
shell:
cmd: "curl {{ groups['nodes'][node_index] }}/whatever/api/result "
loop: "{{ groups['nodes'] }}"
loop_control:
index_var: node_index
register: api_value
If I run some debug tasks hardcoding which list I want to use everyhing works fine
- debug: "msg={{ api_value.results.0.stdout }}"
- debug: "msg={{ api_value.results.1.stdout }}"
- debug: "msg={{ api_value.results.2.stdout }}"
output:
ok: [server-1] => {
"msg": "random-value-a"
ok: [server-2] => {
"msg": "random-value-b"
ok: [server-3] => {
"msg": "random-value-c"
The problem is when I try to increase the list number in Jinja template. I tried several for loops combination, nested for loops and many other things but nothing seems to be working.
For example I want my Jinja template look similar like this:
{% for vm in groups['nodes'] %}
NODE_{{ loop.index }}={{ api_value.results.{loop.index}.stdout }}
{% endfor %}
This way I want to achieve this output:
NODE_0=random-value-a
NODE_1=random-value-b
NODE_2=random-value-c
Is there any other way to workaround this? Or maybe is something I could do better in the "Gathering API results" task?
Given the inventory
shell> cat hosts
[nodes]
server-1
server-2
server-3
Either run it in the loop at a single host, e.g.
- hosts: localhost
gather_facts: false
vars:
whatever_api_result:
server-1: random-value-a
server-2: random-value-b
server-3: random-value-c
tasks:
- command: "echo {{ whatever_api_result[item] }}"
register: api_value
loop: "{{ groups.nodes }}"
- debug:
msg: "{{ api_value.results|json_query('[].[item, stdout]') }}"
gives
msg:
- - server-1
- random-value-a
- - server-2
- random-value-b
- - server-3
- random-value-c
Then, in the Jinja template, fix the index variable
- debug:
msg: |-
{% for vm in groups.nodes %}
NODE_{{ loop.index0 }}={{ api_value.results[loop.index0].stdout }}
{% endfor %}
gives what you want
msg: |-
NODE_0=random-value-a
NODE_1=random-value-b
NODE_2=random-value-c
Optionally, iterate api_value.results. This gives the same result
- debug:
msg: |-
{% for v in api_value.results %}
NODE_{{ loop.index0 }}={{ v.stdout }}
{% endfor %}
Or run it in the group, e.g.
- hosts: nodes
gather_facts: false
vars:
whatever_api_result:
server-1: random-value-a
server-2: random-value-b
server-3: random-value-c
tasks:
- command: "echo {{ whatever_api_result[inventory_hostname] }}"
register: api_value
delegate_to: localhost
- debug:
msg: "{{ api_value.stdout }}"
(delegate to localhost for testing)
gives
ok: [server-1] =>
msg: random-value-a
ok: [server-2] =>
msg: random-value-b
ok: [server-3] =>
msg: random-value-c
Then, in the Jinja template, use hostvars
- debug:
msg: |-
{% for vm in groups.nodes %}
NODE_{{ loop.index0 }}={{ hostvars[vm].api_value.stdout }}
{% endfor %}
run_once: true
gives also what you want
msg: |-
NODE_0=random-value-a
NODE_1=random-value-b
NODE_2=random-value-c
Optionally, iterate hostvars. This gives the same result
- debug:
msg: |-
{% for k,v in hostvars.items() %}
NODE_{{ loop.index0 }}={{ v.api_value.stdout }}
{% endfor %}
run_once: true

Ansible merge dictionaries using with_items and vars stores only last item

Trying to create a dictionary per item and merge them
---
- name: TestingLab
hosts: localhost
gather_facts: False
tasks:
- name: Hello Vars
set_fact:
two_nums:
- 1
- 2
- name: create empty dict
set_fact:
ids: {}
- name: Merge all
vars:
single_entry: "{ '{{ item }}': {{ item }} }"
set_fact:
ids: "{{ ids | combine(single_entry) }}"
with_items: "{{ two_nums }}"
- name: Print Result
debug:
msg: "{{ ids }}"
I thought I followed the right guidelines but I seem to be getting only the last item afterwards:
ok: [localhost] => {
"msg": {
"2": 2
} }
I tried replacing the single_entry with the expression in vars but it does not run.
Is there a different syntax to get this done?
EDIT: version info
ansible-playbook 2.5.1
python version = 2.7.17 [GCC 7.5.0]
Try the filters dict and zip. The zip is available since 2.3, e.g.
- set_fact:
d2: "{{ dict(two_nums|zip(two_nums)) }}"
- debug:
var: d2
- debug:
var: d2|type_debug
gives
d2:
1: 1
2: 2
d2|type_debug: dict
If this does not work try Jinja and the filter from_yaml, e.g.
- hosts: localhost
vars:
two_nums:
- 1
- 2
l1: |-
{% for i in two_nums %}
{{ i }}: {{ i }}
{% endfor %}
tasks:
- set_fact:
d1: "{{ l1|from_yaml }}"
- debug:
var: d1
- debug:
var: d1|type_debug
gives the same result
d1:
1: 1
2: 2
d1|type_debug: dict
If you need the keys to be strings quote it, e.g.
l1: |-
{% for i in two_nums %}
"{{ i }}": {{ i }}
{% endfor %}
gives
d1:
'1': 1
'2': 2
In the first case, map the list's items to string, e.g.
- set_fact:
d2: "{{ dict(two_nums|map('string')|zip(two_nums)) }}"
gives the same result
d2:
'1': 1
'2': 2
I can't reproduce the behavior you're describing. Running your
playbook verbatim, I get as output:
TASK [Print Result] **************************************************************************************************************************************************************************
ok: [localhost] => {
"msg": {
"1": 1,
"2": 2
}
}
I'm using Ansible core 2.11.2, but I've also tested your playbook with Ansible 2.9.20 and I get the same output.
I would probably drop the set_fact task, and also change how you're
setting single_entry:
- name: TestingLab
hosts: localhost
gather_facts: False
tasks:
- name: Hello Vars
set_fact:
two_nums:
- 1
- 2
- name: Merge all
vars:
ids: {}
single_entry: "{{ {item: item} }}"
set_fact:
ids: "{{ ids | combine(single_entry) }}"
with_items: "{{ two_nums }}"
- name: Print Result
debug:
msg: "{{ ids }}"
In this version, the template expression is returning a dictionary,
and only requires a single set of Jinja template markers. I'm curious
if this version behaves any differently for you.

Ansible : double loop into json file

I have as a source a json file that contains a list of blocks and data. from which i would like to extract information to create security rules, using a double loop in ansible.
Below an example from my json file :
[
{
"Name":"Some_name",
"NetworkFlow":[
{
"GroupName":"Test1",
"Type":"Ingress",
"Env":"dev",
"Server":[
"192.168.1.1",
"192.168.1.2",
...
],
"Service":[
{
"Protocol":"TCP",
"Port":"443"
},
{
"Protocol":"UDP",
"Port":"21"
},
....
]
},
....
]
}
]
This is for a generic deployment, and for each "NetworkFlow" section, i have to loop in the list of servers and also in the list of protocols and ports to get a simular parsing like the below:
#rule= Server,Protocol,Port,Type,Env,GroupName
192.168.1.1,TCP,443,Ingress,Dev,Test1
192.168.1.2,TCP,443,Ingress,Dev,Test1
192.168.1.1,UDP,21,Ingress,Dev,Test1
192.168.1.2,UDP,21,Ingress,Dev,Test1
I tried with_nested but it doesn't work, Any idea to deal with that please?
Create a file with the nested loop, for example
shell> cat rules.yml
- debug:
msg: "{{ item.0 }},{{ item.1.Protocol }},{{ item.1.Port }},{{ outer_item.Type }},{{ outer_item.Env }},{{ outer_item.GroupName }}"
with_nested:
- "{{ outer_item.Server }}"
- "{{ outer_item.Service }}"
and include it
- include_tasks: rules.yml
loop: "{{ NetworkFlow }}"
loop_control:
loop_var: outer_item
gives
msg: 192.168.1.1,TCP,443,Ingress,dev,Test1
msg: 192.168.1.1,UDP,21,Ingress,dev,Test1
msg: 192.168.1.2,TCP,443,Ingress,dev,Test1
msg: 192.168.1.2,UDP,21,Ingress,dev,Test1
Q: "... have a list of ports separated by a comma and not just one port."
A: Convert the data. For example
shell> cat rules.yml
- set_fact:
Services: "{{ Services|from_yaml }}"
vars:
Services: |
{% for service in oi.Service %}
{% for port in service.Port.split(',') %}
- Protocol: {{ service.Protocol }}
Port: {{ port }}
{% endfor %}
{% endfor %}
- debug:
msg: "{{ i.0 }},{{ i.1.Protocol }},{{ i.1.Port }},{{ oi.Type }},{{ oi.Env }},{{ oi.GroupName }}"
with_nested:
- "{{ oi.Server }}"
- "{{ Services }}"
loop_control:
loop_var: I
gives
msg: 192.168.1.1,TCP,443,Ingress,dev,Test1
msg: 192.168.1.1,TCP,22,Ingress,dev,Test1
msg: 192.168.1.1,TCP,53,Ingress,dev,Test1
msg: 192.168.1.1,UDP,21,Ingress,dev,Test1
msg: 192.168.1.2,TCP,443,Ingress,dev,Test1
msg: 192.168.1.2,TCP,22,Ingress,dev,Test1
msg: 192.168.1.2,TCP,53,Ingress,dev,Test1
msg: 192.168.1.2,UDP,21,Ingress,dev,Test1

jinja2 turning lists into strings even when from_json is being used

I am trying to run a nested for loop in order to retrieve a nested value.
I would like to retrieve some_value_4 when some_value_3 matches a criteria that's predefined.
{
"some_dict": {
"some_key_0": "value_0",
"some_key_1": "value_1"
},
"testval_dict": {
"test_key_0": "some_value_0",
"test_key_1": "some_value_1",
"test_key_2": "some_value_2",
"test_key_3": "some_value_3",
"test_key_4": "some_value_4"
}
}
The playbook:
- hosts: localhost
tasks:
- set_fact:
mydict: "{{ lookup('file', '/tmp/file.json' ) | from_json }}"
- debug:
msg: |
"{% for item in mydict %}
{{ item }}
{% endfor %}"
when run, it alreay looks like dict names are treated as string and nothing more:
ansible-playbook /tmp/test_playbook.yml -c local -i ', localhost'
TASK [debug] ******************************************************************
ok: [localhost] => {}
MSG:
" somee_dict
testval_dict
"
Then when I add an itme.key to the debug task, the playbook fails:
MSG:
The task includes an option with an undefined variable. The error was: 'str object' has no attribute 'value'
Thank you.
edit for clarification
In the real example, I will not know the names of the dicts, so I cannot use some_dict or testval_dict, that is why I'm trying to go over this data source in an item.key or item.value form.
Q: "{% for item in mydict %} ... dict names are treated as string and nothing more."
A: This is correct. A dictionary, when evaluated as a list, returns the list of its keys. See some examples below
- debug:
msg: "{{ mydict.keys()|list }}"
- debug:
msg: "{{ mydict[item] }}"
loop: "{{ mydict.keys()|list }}"
- debug:
msg: "{{ mydict|difference(['testval_dict']) }}"
give
msg:
- some_dict
- testval_dict
msg:
some_key_0: value_0
some_key_1: value_1
msg:
test_key_0: some_value_0
test_key_1: some_value_1
test_key_2: some_value_2
test_key_3: some_value_3
test_key_4: some_value_4
msg:
- some_dict
See How to iterate through a list of dictionaries in Jinja template?
If you need to loop over the dictionary, you can use with_dict loop functionality. This way, if you loop over mydict and get item.key you will get somee_dict and testval_dict.
tasks:
- set_fact:
mydict: "{{ lookup('file', '/tmp/file.json')|from_json }}"
# This will get the top level dictionaries somee_dict and testval_dict
- debug:
var: item.key
with_dict: "{{ mydict }}"
And if you get item.value of mydict you will get the sub-dictionary:
- debug:
var: item.value
with_dict: "{{ mydict }}"
Will produce (showing output only for testval_dict):
"item.value": {
"test_key_0": "some_value_0",
"test_key_1": "some_value_1",
"test_key_2": "some_value_2",
"test_key_3": "some_value_3",
"test_key_4": "some_value_4"
}

Resources