I have as a source a json file that contains a list of blocks and data. from which i would like to extract information to create security rules, using a double loop in ansible.
Below an example from my json file :
[
{
"Name":"Some_name",
"NetworkFlow":[
{
"GroupName":"Test1",
"Type":"Ingress",
"Env":"dev",
"Server":[
"192.168.1.1",
"192.168.1.2",
...
],
"Service":[
{
"Protocol":"TCP",
"Port":"443"
},
{
"Protocol":"UDP",
"Port":"21"
},
....
]
},
....
]
}
]
This is for a generic deployment, and for each "NetworkFlow" section, i have to loop in the list of servers and also in the list of protocols and ports to get a simular parsing like the below:
#rule= Server,Protocol,Port,Type,Env,GroupName
192.168.1.1,TCP,443,Ingress,Dev,Test1
192.168.1.2,TCP,443,Ingress,Dev,Test1
192.168.1.1,UDP,21,Ingress,Dev,Test1
192.168.1.2,UDP,21,Ingress,Dev,Test1
I tried with_nested but it doesn't work, Any idea to deal with that please?
Create a file with the nested loop, for example
shell> cat rules.yml
- debug:
msg: "{{ item.0 }},{{ item.1.Protocol }},{{ item.1.Port }},{{ outer_item.Type }},{{ outer_item.Env }},{{ outer_item.GroupName }}"
with_nested:
- "{{ outer_item.Server }}"
- "{{ outer_item.Service }}"
and include it
- include_tasks: rules.yml
loop: "{{ NetworkFlow }}"
loop_control:
loop_var: outer_item
gives
msg: 192.168.1.1,TCP,443,Ingress,dev,Test1
msg: 192.168.1.1,UDP,21,Ingress,dev,Test1
msg: 192.168.1.2,TCP,443,Ingress,dev,Test1
msg: 192.168.1.2,UDP,21,Ingress,dev,Test1
Q: "... have a list of ports separated by a comma and not just one port."
A: Convert the data. For example
shell> cat rules.yml
- set_fact:
Services: "{{ Services|from_yaml }}"
vars:
Services: |
{% for service in oi.Service %}
{% for port in service.Port.split(',') %}
- Protocol: {{ service.Protocol }}
Port: {{ port }}
{% endfor %}
{% endfor %}
- debug:
msg: "{{ i.0 }},{{ i.1.Protocol }},{{ i.1.Port }},{{ oi.Type }},{{ oi.Env }},{{ oi.GroupName }}"
with_nested:
- "{{ oi.Server }}"
- "{{ Services }}"
loop_control:
loop_var: I
gives
msg: 192.168.1.1,TCP,443,Ingress,dev,Test1
msg: 192.168.1.1,TCP,22,Ingress,dev,Test1
msg: 192.168.1.1,TCP,53,Ingress,dev,Test1
msg: 192.168.1.1,UDP,21,Ingress,dev,Test1
msg: 192.168.1.2,TCP,443,Ingress,dev,Test1
msg: 192.168.1.2,TCP,22,Ingress,dev,Test1
msg: 192.168.1.2,TCP,53,Ingress,dev,Test1
msg: 192.168.1.2,UDP,21,Ingress,dev,Test1
Related
I have local.yml file where I am passing variable as list
roles:
- role: book
vars:
data:
- 'std1'
- 'std2'
vars.yml mentioned as below
book_data:
"std1": "std1"
"std2": "std2"
"std1":
- '1a'
- '1b'
"std2":
- '2a'
- '2b'
I want to iterate data over list mentioned in local.yml and print value from vars.yml. I have written code which print only list(std1,std2)
- name: set fact
set_fact:
standard: "{{ lookup('list', data) }}"
- name: get standard
debug: msg="{{ item }}"
with_items: "{{ standard }}"
Expected output would be:
list mentioned in local.yml would call vars/main.yml content and fetch content.
Example: whenever I would call str1 it would fetch 1a and 1b,
str2 it would fetch 2a and 2b
Below sample code is working:
- name: get standard data
debug:
msg="{{ item.key }}"
with_items: "{{ lookup('dict', book_data) }}"
but I want specific the iteration call from local.yaml to vars/main.yml. Dont want to use vars/main.yml variable directly into playbook
Given the variables
data: [x, y]
book_data:
x: std1
y: std2
std1: [1a, 1b]
std2: [2a, 2b]
Iterate data, for example
- debug:
msg: "{{ lookup('vars', book_data[item]) }}"
loop: "{{ data }}"
gives (abridged)
msg:
- 1a
- 1b
msg:
- 2a
- 2b
, or create the structure
book_data_lists: |
{% for i in data %}
{{ i }}: {{ lookup('vars', book_data[i]) }}
{% endfor %}
gives
book_data_lists: |-
x: ['1a', '1b']
y: ['2a', '2b']
Iterate the dictionary with subelements
- debug:
msg: "{{ item.0.key }} {{ item.1 }}"
with_subelements:
- "{{ book_data_lists|from_yaml|dict2items }}"
- value
gives (abridged)
msg: x 1a
msg: x 1b
msg: y 2a
msg: y 2b
Example of a complete playbook for testing
- hosts: localhost
vars:
data: [x, y]
book_data:
x: std1
y: std2
std1: [1a, 1b]
std2: [2a, 2b]
book_data_lists: |
{% for i in data %}
{{ i }}: {{ lookup('vars', book_data[i]) }}
{% endfor %}
tasks:
- debug:
msg: "{{ item.key }}"
loop: "{{ lookup('dict', book_data) }}"
- debug:
msg: "{{ item.key }}"
loop: "{{ book_data|dict2items }}"
- debug:
msg: "{{ lookup('vars', book_data[item]) }}"
loop: "{{ data }}"
- debug:
var: book_data_lists
- debug:
msg: "{{ item.0.key }} {{ item.1 }}"
with_subelements:
- "{{ book_data_lists|from_yaml|dict2items }}"
- value
I goal is to replace all the {{ and }} with {% raw %}{{ and }}{% endraw %} in all my markdown notes so that my hexo plugin won't attempt to template when rendering.
My code is:
- name: Get file list of post files
find:
paths: "{{ hexo_data_post_file_dir }}"
register: hexo_data_post_file_list
- name: Replace special char
replace:
path: "{{ item[0]['path'] }}"
regexp: "{{ item[1]['pattern'] }}"
replace: "{{ item[1]['string'] }}"
with_nested:
- "{{ hexo_data_post_file_list['files'] }}"
- - pattern: "{{ '{{' }}"
string: "{{ '{% raw %}{{' }}"
- pattern: "{{ '}}' }}"
string: "{{ '}}{% endraw %}' }}"
Despite I used "{{ '{{' }}" to ask jinja not to render {{, I still get error:
TASK [setup_docker_hexo : Replace special char] ********************************************************************************************************************************************
Tuesday 27 December 2022 11:58:56 +0800 (0:00:00.491) 0:00:13.572 ******
fatal: [uranus-debian]: FAILED! => {"msg": "template error while templating string: unexpected 'end of template'. String: {{. unexpected 'end of template'"}
Given the file
shell> cat test.txt.j2
{{ test_var }}
Declare the strings unsafe to block templating. For example,
my_files:
- "{{ playbook_dir }}/test.txt.j2"
my_regex:
- pattern: !unsafe '{{ '
string: !unsafe '{% raw %}{{ {% endraw %}'
- pattern: !unsafe ' }}'
string: !unsafe '{% raw %} }}{% endraw %}'
Test the expansion
- debug:
msg: "{{ item }}"
with_nested:
- "{{ my_files }}"
- "{{ my_regex }}"
gives
TASK [debug] *********************************************************************************
ok: [localhost] => (item=['/export/scratch/tmp7/test-117/test.txt.j2', {'pattern': '{{ ', 'string': '{% raw %}{{ {% endraw %}'}]) =>
msg:
- /export/scratch/tmp7/test-117/test.txt.j2
- pattern: '{{ '
string: '{% raw %}{{ {% endraw %}'
ok: [localhost] => (item=['/export/scratch/tmp7/test-117/test.txt.j2', {'pattern': ' }}', 'string': '{% raw %} }}{% endraw %}'}]) =>
msg:
- /export/scratch/tmp7/test-117/test.txt.j2
- pattern: ' }}'
string: '{% raw %} }}{% endraw %}'
Replace the patterns
- replace:
path: "{{ item.0 }}"
regexp: "{{ item.1.pattern }}"
replace: "{{ item.1.string }}"
with_nested:
- "{{ my_files }}"
- "{{ my_regex }}"
- debug:
msg: "{{ lookup('file', my_files.0) }}"
- debug:
msg: "{{ lookup('template', my_files.0) }}"
gives
TASK [debug] *********************************************************************************
ok: [localhost] =>
msg: '{% raw %}{{ {% endraw %}test_var{% raw %} }}{% endraw %}'
TASK [debug] *********************************************************************************
ok: [localhost] =>
msg: |-
{{ test_var }}
Declare the variable
my_file: "{{ playbook_dir }}/test.txt"
test_var: foo bar
and use the template
- template:
src: "{{ my_files.0 }}"
dest: "{{ my_file }}"
- debug:
msg: "{{ lookup('file', my_file) }}"
- debug:
msg: "{{ lookup('template', my_file) }}"
gives
TASK [debug] *********************************************************************************
ok: [localhost] =>
msg: '{{ test_var }}'
TASK [debug] *********************************************************************************
ok: [localhost] =>
msg: |-
foo bar
Notes
Example of a complete playbook for testing
- hosts: localhost
vars:
my_files:
- "{{ playbook_dir }}/test.txt.j2"
my_regex:
- pattern: !unsafe '{{'
string: !unsafe '{% raw %}{{{% endraw %}'
- pattern: !unsafe '}}'
string: !unsafe '{% raw %}}}{% endraw %}'
my_file: "{{ playbook_dir }}/test.txt"
test_var: foo bar
tasks:
- debug:
msg: "{{ item }}"
with_nested:
- "{{ my_files }}"
- "{{ my_regex }}"
- replace:
path: "{{ item.0 }}"
regexp: "{{ item.1.pattern }}"
replace: "{{ item.1.string }}"
with_nested:
- "{{ my_files }}"
- "{{ my_regex }}"
- debug:
msg: "{{ lookup('file', my_files.0) }}"
- debug:
msg: "{{ lookup('template', my_files.0) }}"
- template:
src: "{{ my_files.0 }}"
dest: "{{ my_file }}"
- debug:
msg: "{{ lookup('file', my_file) }}"
- debug:
msg: "{{ lookup('template', my_file) }}"
The playbook is not idempotent!
In my inventory I have 3 servers in a group. I want to be able to increase that size in the future so I can add more nodes into the network and generate the template with Jinja2.
- name: Gathering API results
shell:
cmd: "curl {{ groups['nodes'][node_index] }}/whatever/api/result "
loop: "{{ groups['nodes'] }}"
loop_control:
index_var: node_index
register: api_value
If I run some debug tasks hardcoding which list I want to use everyhing works fine
- debug: "msg={{ api_value.results.0.stdout }}"
- debug: "msg={{ api_value.results.1.stdout }}"
- debug: "msg={{ api_value.results.2.stdout }}"
output:
ok: [server-1] => {
"msg": "random-value-a"
ok: [server-2] => {
"msg": "random-value-b"
ok: [server-3] => {
"msg": "random-value-c"
The problem is when I try to increase the list number in Jinja template. I tried several for loops combination, nested for loops and many other things but nothing seems to be working.
For example I want my Jinja template look similar like this:
{% for vm in groups['nodes'] %}
NODE_{{ loop.index }}={{ api_value.results.{loop.index}.stdout }}
{% endfor %}
This way I want to achieve this output:
NODE_0=random-value-a
NODE_1=random-value-b
NODE_2=random-value-c
Is there any other way to workaround this? Or maybe is something I could do better in the "Gathering API results" task?
Given the inventory
shell> cat hosts
[nodes]
server-1
server-2
server-3
Either run it in the loop at a single host, e.g.
- hosts: localhost
gather_facts: false
vars:
whatever_api_result:
server-1: random-value-a
server-2: random-value-b
server-3: random-value-c
tasks:
- command: "echo {{ whatever_api_result[item] }}"
register: api_value
loop: "{{ groups.nodes }}"
- debug:
msg: "{{ api_value.results|json_query('[].[item, stdout]') }}"
gives
msg:
- - server-1
- random-value-a
- - server-2
- random-value-b
- - server-3
- random-value-c
Then, in the Jinja template, fix the index variable
- debug:
msg: |-
{% for vm in groups.nodes %}
NODE_{{ loop.index0 }}={{ api_value.results[loop.index0].stdout }}
{% endfor %}
gives what you want
msg: |-
NODE_0=random-value-a
NODE_1=random-value-b
NODE_2=random-value-c
Optionally, iterate api_value.results. This gives the same result
- debug:
msg: |-
{% for v in api_value.results %}
NODE_{{ loop.index0 }}={{ v.stdout }}
{% endfor %}
Or run it in the group, e.g.
- hosts: nodes
gather_facts: false
vars:
whatever_api_result:
server-1: random-value-a
server-2: random-value-b
server-3: random-value-c
tasks:
- command: "echo {{ whatever_api_result[inventory_hostname] }}"
register: api_value
delegate_to: localhost
- debug:
msg: "{{ api_value.stdout }}"
(delegate to localhost for testing)
gives
ok: [server-1] =>
msg: random-value-a
ok: [server-2] =>
msg: random-value-b
ok: [server-3] =>
msg: random-value-c
Then, in the Jinja template, use hostvars
- debug:
msg: |-
{% for vm in groups.nodes %}
NODE_{{ loop.index0 }}={{ hostvars[vm].api_value.stdout }}
{% endfor %}
run_once: true
gives also what you want
msg: |-
NODE_0=random-value-a
NODE_1=random-value-b
NODE_2=random-value-c
Optionally, iterate hostvars. This gives the same result
- debug:
msg: |-
{% for k,v in hostvars.items() %}
NODE_{{ loop.index0 }}={{ v.api_value.stdout }}
{% endfor %}
run_once: true
I have a json structure like this:
{
"group_a": {
"setting_one": "text one",
"setting_two": "text two"
},
"group_b": {
"setting_three": "text three",
"setting_four": "text four"
}
}
And I need to do iterate over the dicts inside the dicts and create tasks, kinda like this:
- name: Task group_a setting_one
...
- name: Task group_a setting_two
...
- name: Task group_b setting_three
...
- name: Task group_b setting_four
...
I was hoping to make something like this:
- name: Insert {{ group }} {{ setting }}
ini_file:
path: settings.ini
section: "{{ group }}"
option: "{{ setting }}"
value: "{{ setting_value }}"
with_nested:
- "{{ data.keys() }}" # Can I get this key and use in the next loop?
- "{{ data.key.keys() }}"
How can I do this? I have been messing about with loop, with_items, with_dict etc, but I can't manage to get it exactly the way I need. I also tried with_nested, but that does not seem to do what I need either.
Kind regards, Jonas
Let's reduce the structure to one with the same piece of information
my_data:
group_a:
- one
- two
group_b:
- three
- four
Then the play below
- debug:
msg: "{{ item.0.key }} setting_{{ item.1 }}"
with_subelements:
- "{{ my_data|dict2items }}"
- value
gives:
"msg": "group_b setting_three"
"msg": "group_b setting_four"
"msg": "group_a setting_one"
"msg": "group_a setting_two"
It's not straightforward to use the original data. The play below separates the keys and values
tasks:
- set_fact:
my_groups: "{{ my_data.keys()}}"
- debug:
msg: "{{ item }} {{ my_data|dict2items|json_query(my_query)|to_yaml }}"
loop: "{{ my_groups }}"
vars:
my_query: "[?key=='{{ item }}'].value"
and gives:
"msg": "group_b - - {setting_three: text three}\n - {setting_four: text four}\n"
"msg": "group_a - - {setting_one: text one}\n - {setting_two: text two}\n"
Multi-step loops are needed to proceed. For example the play below
- set_fact:
my_data1: |
{% for group, values in my_data.items() %}
{% for value in values %}
{% for item_key, item_val in value.items() %}
Task {{ group }} {{ item_key }},
{% endfor %}{% endfor %}{% endfor %}
- debug:
msg: "{{ my_data1.split('\n') }}"
gives:
"msg": [
"Task group_b setting_three,",
"Task group_b setting_four,",
"Task group_a setting_one,",
"Task group_a setting_two,",
""
]
I have the following data being returned from an API call
"napalm_interfaces_ip": {
"Vlan5": {
"ipv4": {
"10.45.230.250": {
"prefix_length": 24
}
}
}
}
How do I debug print the Vlan info, the IP and the prefix length?
This is what I have
- debug:
msg: "Interface: {{ item.key }}, IP: {{ item.value.ipv4 }}"
with_dict:
- "{{ napalm_interfaces_ip }}"
It produces the below output which shows the IP pointing to another dictionary
TASK [validate_device_ips : debug]
ok: [] => (item={'value': {u'ipv4': {u'10.45.230.250':
{u'prefix_length': 24}}}, 'key': u'Vlan5'}) => {
"msg": "Interface: Vlan5, IP: {u'10.45.230.250': {u'prefix_length': 24}}" }
dict2items should help:
- debug:
msg: "IP: {{ item.key }} prefix_length: {{ item.value.prefix_length }}"
loop: "{{ napalm_interfaces_ip.Vlan5.ipv4 | dict2items }}"
with_dict works too
- debug:
msg: "IP: {{ item.key }} prefix_length: {{ item.value.prefix_length }}"
with_dict: "{{ napalm_interfaces_ip.Vlan5.ipv4 }}"
Example how to loop the interfaces is below
- hosts: localhost
gather_facts: no
vars:
interfaces:
- Vlan0:
ipv4:
10.45.230.250:
prefix_length: 24
- Vlan1:
ipv4:
10.45.230.251:
prefix_length: 24
- Vlan2:
ipv4:
10.45.230.252:
prefix_length: 24
tasks:
- name: List selected variables
vars:
msg: |
vlan {{ item|dict2items|json_query('[].key') }}
ip {{ item|dict2items|json_query('[].value.ipv4|[0]')|dict2items|json_query('[].key') }}
prefix {{ item|dict2items|json_query('[].value.ipv4|[0]')|dict2items|json_query('[].value.prefix_length') }}
debug:
msg: "{{ msg.split('\n') }}"
loop: "{{ interfaces }}"
Once you decide to make your live easier below is an option
- hosts: localhost
vars:
interfaces:
- Vlan0:
ipv4: "10.45.230.250"
prefix_length: "24"
- Vlan1:
ipv4: "10.45.230.251"
prefix_length: "24"
- Vlan2:
ipv4: "10.45.230.252"
prefix_length: "24"
tasks:
- name: List selected variables
vars:
msg: |
vlan {{ item|dict2items|json_query('[].key') }}
ip {{ item|dict2items|json_query('[].value.ipv4') }}
prefix {{ item|dict2items|json_query('[].value.prefix_length') }}
debug:
msg: "{{ msg.split('\n') }}"
loop: "{{ interfaces }}"