I have local.yml file where I am passing variable as list
roles:
- role: book
vars:
data:
- 'std1'
- 'std2'
vars.yml mentioned as below
book_data:
"std1": "std1"
"std2": "std2"
"std1":
- '1a'
- '1b'
"std2":
- '2a'
- '2b'
I want to iterate data over list mentioned in local.yml and print value from vars.yml. I have written code which print only list(std1,std2)
- name: set fact
set_fact:
standard: "{{ lookup('list', data) }}"
- name: get standard
debug: msg="{{ item }}"
with_items: "{{ standard }}"
Expected output would be:
list mentioned in local.yml would call vars/main.yml content and fetch content.
Example: whenever I would call str1 it would fetch 1a and 1b,
str2 it would fetch 2a and 2b
Below sample code is working:
- name: get standard data
debug:
msg="{{ item.key }}"
with_items: "{{ lookup('dict', book_data) }}"
but I want specific the iteration call from local.yaml to vars/main.yml. Dont want to use vars/main.yml variable directly into playbook
Given the variables
data: [x, y]
book_data:
x: std1
y: std2
std1: [1a, 1b]
std2: [2a, 2b]
Iterate data, for example
- debug:
msg: "{{ lookup('vars', book_data[item]) }}"
loop: "{{ data }}"
gives (abridged)
msg:
- 1a
- 1b
msg:
- 2a
- 2b
, or create the structure
book_data_lists: |
{% for i in data %}
{{ i }}: {{ lookup('vars', book_data[i]) }}
{% endfor %}
gives
book_data_lists: |-
x: ['1a', '1b']
y: ['2a', '2b']
Iterate the dictionary with subelements
- debug:
msg: "{{ item.0.key }} {{ item.1 }}"
with_subelements:
- "{{ book_data_lists|from_yaml|dict2items }}"
- value
gives (abridged)
msg: x 1a
msg: x 1b
msg: y 2a
msg: y 2b
Example of a complete playbook for testing
- hosts: localhost
vars:
data: [x, y]
book_data:
x: std1
y: std2
std1: [1a, 1b]
std2: [2a, 2b]
book_data_lists: |
{% for i in data %}
{{ i }}: {{ lookup('vars', book_data[i]) }}
{% endfor %}
tasks:
- debug:
msg: "{{ item.key }}"
loop: "{{ lookup('dict', book_data) }}"
- debug:
msg: "{{ item.key }}"
loop: "{{ book_data|dict2items }}"
- debug:
msg: "{{ lookup('vars', book_data[item]) }}"
loop: "{{ data }}"
- debug:
var: book_data_lists
- debug:
msg: "{{ item.0.key }} {{ item.1 }}"
with_subelements:
- "{{ book_data_lists|from_yaml|dict2items }}"
- value
In my inventory I have 3 servers in a group. I want to be able to increase that size in the future so I can add more nodes into the network and generate the template with Jinja2.
- name: Gathering API results
shell:
cmd: "curl {{ groups['nodes'][node_index] }}/whatever/api/result "
loop: "{{ groups['nodes'] }}"
loop_control:
index_var: node_index
register: api_value
If I run some debug tasks hardcoding which list I want to use everyhing works fine
- debug: "msg={{ api_value.results.0.stdout }}"
- debug: "msg={{ api_value.results.1.stdout }}"
- debug: "msg={{ api_value.results.2.stdout }}"
output:
ok: [server-1] => {
"msg": "random-value-a"
ok: [server-2] => {
"msg": "random-value-b"
ok: [server-3] => {
"msg": "random-value-c"
The problem is when I try to increase the list number in Jinja template. I tried several for loops combination, nested for loops and many other things but nothing seems to be working.
For example I want my Jinja template look similar like this:
{% for vm in groups['nodes'] %}
NODE_{{ loop.index }}={{ api_value.results.{loop.index}.stdout }}
{% endfor %}
This way I want to achieve this output:
NODE_0=random-value-a
NODE_1=random-value-b
NODE_2=random-value-c
Is there any other way to workaround this? Or maybe is something I could do better in the "Gathering API results" task?
Given the inventory
shell> cat hosts
[nodes]
server-1
server-2
server-3
Either run it in the loop at a single host, e.g.
- hosts: localhost
gather_facts: false
vars:
whatever_api_result:
server-1: random-value-a
server-2: random-value-b
server-3: random-value-c
tasks:
- command: "echo {{ whatever_api_result[item] }}"
register: api_value
loop: "{{ groups.nodes }}"
- debug:
msg: "{{ api_value.results|json_query('[].[item, stdout]') }}"
gives
msg:
- - server-1
- random-value-a
- - server-2
- random-value-b
- - server-3
- random-value-c
Then, in the Jinja template, fix the index variable
- debug:
msg: |-
{% for vm in groups.nodes %}
NODE_{{ loop.index0 }}={{ api_value.results[loop.index0].stdout }}
{% endfor %}
gives what you want
msg: |-
NODE_0=random-value-a
NODE_1=random-value-b
NODE_2=random-value-c
Optionally, iterate api_value.results. This gives the same result
- debug:
msg: |-
{% for v in api_value.results %}
NODE_{{ loop.index0 }}={{ v.stdout }}
{% endfor %}
Or run it in the group, e.g.
- hosts: nodes
gather_facts: false
vars:
whatever_api_result:
server-1: random-value-a
server-2: random-value-b
server-3: random-value-c
tasks:
- command: "echo {{ whatever_api_result[inventory_hostname] }}"
register: api_value
delegate_to: localhost
- debug:
msg: "{{ api_value.stdout }}"
(delegate to localhost for testing)
gives
ok: [server-1] =>
msg: random-value-a
ok: [server-2] =>
msg: random-value-b
ok: [server-3] =>
msg: random-value-c
Then, in the Jinja template, use hostvars
- debug:
msg: |-
{% for vm in groups.nodes %}
NODE_{{ loop.index0 }}={{ hostvars[vm].api_value.stdout }}
{% endfor %}
run_once: true
gives also what you want
msg: |-
NODE_0=random-value-a
NODE_1=random-value-b
NODE_2=random-value-c
Optionally, iterate hostvars. This gives the same result
- debug:
msg: |-
{% for k,v in hostvars.items() %}
NODE_{{ loop.index0 }}={{ v.api_value.stdout }}
{% endfor %}
run_once: true
Practice:
I have the file files.yml with a list of files and their respective md5_sum hash, like:
files:
- name: /opt/file_compare1.tar
hash: 9cd599a3523898e6a12e13ec787da50a /opt/file_compare1.tar
- name: /opt/file_compare2tar.gz
hash: d41d8cd98f00b204e9800998ecf8427e /opt/file_compare2.tar.gz
I need to create a playbook to check this list of files if the current hash is the same or if it was changed, the playbook should have a debug message like below:
---
- hosts: localhost
connection: local
vars_files:
- files.yml
tasks:
- name: Use md5 to calculate checksum
stat:
path: "{{ item.name }}"
checksum_algorithm: md5
register: hash_check
with_items:
- "{{ files }}"
- name: Debug files - Different
debug:
msg: |
"Hash changed: {{ item.name }}"
when:
- item.hash != hash_check
with_items:
- "{{ files }}"
- name: Debug files - Equal
debug:
msg: |
"Hash NOT changed: {{ item.name }}"
when:
- item.hash == hash_check
with_items:
- "{{ files }}"
- debug:
msg: |
- "{{ hash_check }} {{ item.name }}"
with_items:
- "{{ files }}"
For example, given the files
files:
- name: /scratch/file_compare1.tar
hash: 4f8805b4b64dcc575547ec1c63793aec /scratch/file_compare1.tar
- name: /scratch/file_compare2.tar.gz
hash: 2dc4f1e9ca4081cc49d25195627982ef /scratch/file_compare2.tar.gz
the tasks below
- name: Use md5 to calculate checksum
stat:
path: "{{ item.name }}"
checksum_algorithm: md5
register: hash_check
loop: "{{ files }}"
- name: Debug files - Different
debug:
msg: |
Hash NOT changed: {{ item.0.name }}
{{ item.0.hash.split()|first }}
{{ item.1 }}
with_together:
- "{{ files }}"
- "{{ hash_check.results|map(attribute='stat.checksum')|list }}"
when: item.0.hash.split()|first == item.1
give
msg: |-
Hash NOT changed: /scratch/file_compare1.tar
4f8805b4b64dcc575547ec1c63793aec
4f8805b4b64dcc575547ec1c63793aec
msg: |-
Hash NOT changed: /scratch/file_compare2.tar.gz
2dc4f1e9ca4081cc49d25195627982ef
2dc4f1e9ca4081cc49d25195627982ef
A more robust option would be to create a dictionary with the calculated hashes
- name: Use md5 to calculate checksum
stat:
path: "{{ item.name }}"
checksum_algorithm: md5
register: hash_check
loop: "{{ files }}"
- set_fact:
path_hash: "{{ dict(_path|zip(_hash)) }}"
vars:
_path: "{{ hash_check.results|map(attribute='stat.path')|list }}"
_hash: "{{ hash_check.results|map(attribute='stat.checksum')|list }}"
gives
path_hash:
/scratch/file_compare1.tar: 4f8805b4b64dcc575547ec1c63793aec
/scratch/file_compare2.tar.gz: 2dc4f1e9ca4081cc49d25195627982ef
Then use this dictionary to compare the hashes. For example, the task below gives the same results
- name: Debug files - Different
debug:
msg: |
Hash NOT changed: {{ item.name }}
{{ item.hash.split()|first }}
{{ path_hash[item.name] }}
loop: "{{ files }}"
when: item.hash.split()|first == path_hash[item.name]
The next option is to create a dictionary with the original hashes and both lists of original and calculated hashes
- name: Use md5 to calculate checksum
stat:
path: "{{ item.name }}"
checksum_algorithm: md5
register: hash_check
loop: "{{ files }}"
- set_fact:
hash_name: "{{ dict(_hash|zip(_name)) }}"
hash_orig: "{{ _hash }}"
hash_stat: "{{ hash_check.results|map(attribute='stat.checksum')|list }}"
vars:
_hash: "{{ files|map(attribute='hash')|map('split')|map('first')|list }}"
_name: "{{ files|map(attribute='name')|list }}"
gives
hash_name:
2dc4f1e9ca4081cc49d25195627982ef: /scratch/file_compare2.tar.gz
4f8805b4b64dcc575547ec1c63793aec: /scratch/file_compare1.tar
hash_orig:
- 4f8805b4b64dcc575547ec1c63793aec
- 2dc4f1e9ca4081cc49d25195627982ef
hash_stat:
- 4f8805b4b64dcc575547ec1c63793aec
- 2dc4f1e9ca4081cc49d25195627982ef
Then calculate the difference of the lists and use it to extract both lists of changed and unchanged files
- set_fact:
files_diff: "{{ _diff|map('extract', hash_name)|list }}"
files_orig: "{{ _orig|map('extract', hash_name)|list }}"
vars:
_diff: "{{ hash_orig|difference(hash_stat) }}"
_orig: "{{ hash_orig|difference(_diff) }}"
- name: Debug files changed
debug:
var: files_diff
- name: Debug files NOT changed
debug:
var: files_orig
gives
files_diff: []
files_orig:
- /scratch/file_compare1.tar
- /scratch/file_compare2.tar.gz
I used your suggestion to complement the playbook, it's working now.
The idea is to get a list of files, read each one and compare with both hash, file, and current hash.
---
- hosts: localhost
connection: local
gather_facts: false
vars_files:
- files3.yml
tasks:
- stat:
path: "{{ item.file }}"
checksum_algorithm: md5
loop: "{{ files }}"
register: stat_results
- name: NOT changed files
debug:
msg: "NOT changed: {{ item.stat.path }}"
when: item.stat.checksum == item.item.checksum.split()|first
loop: "{{ stat_results.results }}"
loop_control:
label: "{{ item.stat.path }}"
- name: Changed files
debug:
msg: "CHANGED: {{ item.stat.path }}"
when: item.stat.checksum != item.item.checksum.split()|first
loop: "{{ stat_results.results }}"
loop_control:
label: "{{ item.stat.path }}"
Result:
>> ansible-playbook playbooks/check-file3.yml
PLAY [localhost] ********************************************************************************************************************************************************************************************************************
TASK [stat] *************************************************************************************************************************************************************************************************************************
ok: [localhost] => (item={'file': '/opt/file_compare1.tar', 'checksum': '9cd599a3523898e6a12e13ec787da50a /opt/file_compare1.tar'})
ok: [localhost] => (item={'file': '/opt/file_compare2.tar.gz', 'checksum': 'd41d8cd98f00b204e9800998ecf8427e /opt/file_compare2.tar.gz'})
TASK [NOT changed files] ************************************************************************************************************************************************************************************************************
skipping: [localhost] => (item=/opt/file_compare1.tar)
ok: [localhost] => (item=/opt/file_compare2.tar.gz) => {
"msg": "NOT changed: /opt/file_compare2.tar.gz"
}
TASK [Changed files] ****************************************************************************************************************************************************************************************************************
ok: [localhost] => (item=/opt/file_compare1.tar) => {
"msg": "CHANGED: /opt/file_compare1.tar"
}
skipping: [localhost] => (item=/opt/file_compare2.tar.gz)
PLAY RECAP **************************************************************************************************************************************************************************************************************************
localhost : ok=3 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
I have as a source a json file that contains a list of blocks and data. from which i would like to extract information to create security rules, using a double loop in ansible.
Below an example from my json file :
[
{
"Name":"Some_name",
"NetworkFlow":[
{
"GroupName":"Test1",
"Type":"Ingress",
"Env":"dev",
"Server":[
"192.168.1.1",
"192.168.1.2",
...
],
"Service":[
{
"Protocol":"TCP",
"Port":"443"
},
{
"Protocol":"UDP",
"Port":"21"
},
....
]
},
....
]
}
]
This is for a generic deployment, and for each "NetworkFlow" section, i have to loop in the list of servers and also in the list of protocols and ports to get a simular parsing like the below:
#rule= Server,Protocol,Port,Type,Env,GroupName
192.168.1.1,TCP,443,Ingress,Dev,Test1
192.168.1.2,TCP,443,Ingress,Dev,Test1
192.168.1.1,UDP,21,Ingress,Dev,Test1
192.168.1.2,UDP,21,Ingress,Dev,Test1
I tried with_nested but it doesn't work, Any idea to deal with that please?
Create a file with the nested loop, for example
shell> cat rules.yml
- debug:
msg: "{{ item.0 }},{{ item.1.Protocol }},{{ item.1.Port }},{{ outer_item.Type }},{{ outer_item.Env }},{{ outer_item.GroupName }}"
with_nested:
- "{{ outer_item.Server }}"
- "{{ outer_item.Service }}"
and include it
- include_tasks: rules.yml
loop: "{{ NetworkFlow }}"
loop_control:
loop_var: outer_item
gives
msg: 192.168.1.1,TCP,443,Ingress,dev,Test1
msg: 192.168.1.1,UDP,21,Ingress,dev,Test1
msg: 192.168.1.2,TCP,443,Ingress,dev,Test1
msg: 192.168.1.2,UDP,21,Ingress,dev,Test1
Q: "... have a list of ports separated by a comma and not just one port."
A: Convert the data. For example
shell> cat rules.yml
- set_fact:
Services: "{{ Services|from_yaml }}"
vars:
Services: |
{% for service in oi.Service %}
{% for port in service.Port.split(',') %}
- Protocol: {{ service.Protocol }}
Port: {{ port }}
{% endfor %}
{% endfor %}
- debug:
msg: "{{ i.0 }},{{ i.1.Protocol }},{{ i.1.Port }},{{ oi.Type }},{{ oi.Env }},{{ oi.GroupName }}"
with_nested:
- "{{ oi.Server }}"
- "{{ Services }}"
loop_control:
loop_var: I
gives
msg: 192.168.1.1,TCP,443,Ingress,dev,Test1
msg: 192.168.1.1,TCP,22,Ingress,dev,Test1
msg: 192.168.1.1,TCP,53,Ingress,dev,Test1
msg: 192.168.1.1,UDP,21,Ingress,dev,Test1
msg: 192.168.1.2,TCP,443,Ingress,dev,Test1
msg: 192.168.1.2,TCP,22,Ingress,dev,Test1
msg: 192.168.1.2,TCP,53,Ingress,dev,Test1
msg: 192.168.1.2,UDP,21,Ingress,dev,Test1
I have a dict playbook which looks like this:
x_php_versions_installed:
ea-php71:
- ea-php71-php-bcmath
- ea-php71-php-xmlrpc
- ea-php71-php-zip
- pecl-memcached
- pecl-imagick
ea-php72:
- ea-php72-php-cli
- ea-php72-php-common
- ea-php72-php-curl
- pecl-imagick
I would like to filter them, to write me each item.value which contains 'ea' string but not everything else. My task looks like this:
- name: Write out only the ea packages
debug:
msg: '{{ item.value }}'
when: item.value | selectattr(item.value, 'contains', 'ea')
loop: '{{ x_php_versions_installed | dict2items }}
But it does not work, because it will list all of the packages, not only the ea ones. The expected answer should look like this:
...
"msg": [
"ea-php71-php-bcmath",
"ea-php71-php-xmlrpc",
"ea-php71-php-zip"
]
...
"msg": [
"ea-php72-php-cli",
"ea-php72-php-common",
"ea-php72-php-curl"
]
...
Another possibility is to filter out the 'pecl' string, it will gave me the same result and it also works fine.
Q: "Filter item.value which contains ea string."
A: The task below does the job
- debug:
msg: "{{ item.value|select('match','^ea-(.*)$')|list }}"
loop: "{{ x_php_versions_installed|dict2items }}"
gives (abridged)
msg:
- ea-php71-php-bcmath
- ea-php71-php-xmlrpc
- ea-php71-php-zip
msg:
- ea-php72-php-cli
- ea-php72-php-common
- ea-php72-php-curl
Note: The test match by default "succeeds if it finds the pattern at the beginning of the string". The task below gives the same result
- debug:
msg: "{{ item.value|select('match', 'ea-')|list }}"
loop: "{{ x_php_versions_installed|dict2items }}"
Q: "Filter out the pecl string."
A: Change the filter to reject and fit the regex. For example, the task below gives the same result
- debug:
msg: "{{ item.value|reject('match','^pecl-(.*)$')|list }}"
loop: "{{ x_php_versions_installed|dict2items }}"
Notes:
Select the lists without iteration. Declare the variables
x_php_versions_installed_keys: "{{ x_php_versions_installed.keys()|list }}"
x_php_versions_installed_ea_vals: "{{ x_php_versions_installed|dict2items|
map(attribute='value')|
map('select', 'match', 'ea-')|list }}"
x_php_versions_installed_ea: "{{ dict(x_php_versions_installed_keys|
zip(x_php_versions_installed_ea_vals)) }}"
gives
x_php_versions_installed_ea:
ea-php71:
- ea-php71-php-bcmath
- ea-php71-php-xmlrpc
- ea-php71-php-zip
ea-php72:
- ea-php72-php-cli
- ea-php72-php-common
- ea-php72-php-curl
Example of a complete playbook for testing
- hosts: localhost
vars:
x_php_versions_installed:
ea-php71:
- ea-php71-php-bcmath
- ea-php71-php-xmlrpc
- ea-php71-php-zip
- pecl-memcached
- pecl-imagick
ea-php72:
- ea-php72-php-cli
- ea-php72-php-common
- ea-php72-php-curl
- pecl-imagick
x_php_versions_installed_keys: "{{ x_php_versions_installed.keys()|list }}"
x_php_versions_installed_ea_vals: "{{ x_php_versions_installed|dict2items|
map(attribute='value')|
map('select', 'match', 'ea-')|list }}"
x_php_versions_installed_ea: "{{ dict(x_php_versions_installed_keys|
zip(x_php_versions_installed_ea_vals)) }}"
tasks:
- debug:
msg: "{{ item.value|select('match','^ea-(.*)$')|list }}"
loop: "{{ x_php_versions_installed|dict2items }}"
- debug:
msg: "{{ item.value|select('match', 'ea-')|list }}"
loop: "{{ x_php_versions_installed|dict2items }}"
- debug:
msg: "{{ item.value|reject('match','^pecl-(.*)$')|list }}"
loop: "{{ x_php_versions_installed|dict2items }}"
- debug:
msg: "{{ item.value|reject('match','pecl-')|list }}"
loop: "{{ x_php_versions_installed|dict2items }}"
- debug:
var: x_php_versions_installed_ea