Extracting nested object in Jinja2 - ansible

I need to extract IPv4 addresses for specific interface from hostvars in Ansible Jinja2 template without loops using.
Trying to extract 'ansible_eth1' object from hostvars is successful and Ansible provides me all info about `eth1' interface:
- debug:
msg: "{{ groups['my_hosts'] | map('extract', hostvars, 'ansible_eth1') | join(', ') }}"
TASK [my_task : debug] *****************************************************************************************************************************************************************************************************
ok: [server1] => {
"msg": [
{
"active": true,
"device": "eth1",
...
"ipv4": {
"address": "192.168.56.15",
"broadcast": "192.168.56.255",
"netmask": "255.255.255.0",
"network": "192.168.56.0"
},
...
But if I try to extract nested object - 'ipv4.address' - it returns empty list:
- debug:
msg: "{{ groups['my_hosts'] | map('extract', hostvars, 'ansible_eth1.ipv4.address') | join(', ') }}"
TASK [my_task : debug] *****************************************************************************************************************************************************************************************************
ok: [server1] => {
"msg": ", "
}
ok: [server2] => {
"msg": ", "
}
Is it possible?

The behavior you are after is specified by the final line in the extract section, where it says "The third argument to the filter can also be a list, for a recursive lookup inside the container"
Thus, in your case:
msg: "{{ groups['my_hosts']
| map('extract', hostvars, ['ansible_eth1', 'ipv4', 'address'])
| join(', ') }}"

Related

How do I map metadata to a variable?

I have an ansible playbook in which I need to pass 2 metadata elements to 2 different variables.
My relevent code in my yml is:
- debug:
var: result
- name: convert
set_fact:
var1: "{{ result | map(attribute='appname') }}"
var2: "{{ result | map(attribute='vipport') }}"
My metadata output looks like this:
"result": {
"changed": true,
"failed": false,
"meta": {
"appname": " testserver4",
"serverquerytype": "A",
"servicemonitor": "http-ecv",
"serviceport": 4433,
"vipmethod": "LEASTCONNECTION",
"vipport": 80,
"viptype": "HTTP"
}
I need to be able to create a variable of appname and vipport, the code I tried above does not work. Any idea what I am missing?
Below I put examples of how you can manipulate the result variable. Set_fact is redundant but I have added and example of taking the variable directly from the result variable.
- hosts: localhost
vars:
"result": {
"changed": true,
"failed": false,
"meta": {
"appname": " testserver4",
"serverquerytype": "A",
"servicemonitor": "http-ecv",
"serviceport": 4433,
"vipmethod": "LEASTCONNECTION",
"vipport": 80,
"viptype": "HTTP"
}
}
tasks:
- name: set var1 and var2 from result var
set_fact:
var1: "{{ result.meta.appname }}"
var2: "{{ result.meta.vipport }}"
- name: output of set_fact
debug:
msg:
- "{{ var1 }}"
- "{{ var2 }}"
- name: Output directly from result variable. Just an example that set_fact is not really needed.
debug:
msg:
- "{{ result.meta.appname }}"
- "{{ result.meta.vipport }}"
Output:
TASK [set var1 and var2 from result var] ***************************************************************************************************************************************************
ok: [localhost]
TASK [output of set_fact] ******************************************************************************************************************************************************************
ok: [localhost] => {
"msg": [
" testserver4",
"80"
]
}
TASK [Output directly from result variable. Just an example that set_fact is not really needed.] *******************************************************************************************
ok: [localhost] => {
"msg": [
" testserver4",
"80"
]
}

ansible compare DNS list from F5

I would like to create a playbook which logs into a big-ip device, downloads a list of certificates and compares it with a list of existing certificates to find matches.
My current playbook is:
---
- name: App Collection f5_modules
hosts: localhost
connection: local
gather_facts: false
vars:
from_ex_var: ["test1.com","test.com","httpsN-dep.san-dc.yu.com"]
from_F5: [
{
"createDateTime": "2022-11-27T09:23:39.000Z",
"expirationDate": 1681379023,
"expirationDateTime": "2023-04-13T09:43:40.000Z",
"subject": "CN=test.com,O=chicago,L=los,ST=Nord,C=PE",
"subjectAlternativeName": "DNS:httpsN-dep.san-dc.yu.com, DNS:test1.com, DNS:test.com",
},
{
"createDateTime": "2023-11-27T09:23:39.000Z",
"expirationDate": 1781379023,
"expirationDateTime": "2023-04-13T09:43:40.000Z",
"subject": "CN=test.com,O=chicago,L=los,ST=Nord,C=PE",
"subjectAlternativeName": "DNS:test.com, DNS:test1.com, DNS:httpsN-dep.san-dc.yu.com",
}
]
tasks:
- name: debug 0
set_fact:
alt_name: "{{ from_ex_var | map('regex_replace', '^(.*)$', 'DNS:\\1') | list | sort }}"
- name: works
debug:
msg: "{{ item.subjectAlternativeName.split(',') | map('regex_replace', '.*DNS:(.*)', 'DNS:\\1') | list | sort }}"
loop: "{{ from_F5 }}"
- name: does not work
debug:
var: item.subjectAlternativeName.split(',') | map('regex_replace', '.*DNS:(.*)', 'DNS:\\1') | list | sort
loop: "{{ from_F5 }}"
- name: checking compare two list
debug:
msg: "works"
loop: "{{ from_F5 }}"
when: item.subjectAlternativeName.split(',') | map('regex_replace', '.*DNS:(.*)', 'DNS:\\1') | list | sort == alt_name
I think the problem is that DNS:\\1 gives me a wrong value in my regex_replace expression in the last when: line as I tried to debug in the following task which gives:
TASK [does not work] *****************************************************************************************************************************************************************************************
ok: [localhost] => (item={'createDateTime': '2022-11-27T09:23:39.000Z', 'expirationDate': 1681379023, 'expirationDateTime': '2023-04-13T09:43:40.000Z', 'subject': 'CN=test.com,O=chicago,L=los,ST=Nord,C=PE', 'subjectAlternativeName': 'DNS:httpsN-dep.san-dc.yu.com, DNS:test1.com, DNS:test.com'}) => {
"ansible_loop_var": "item",
"item": {
"createDateTime": "2022-11-27T09:23:39.000Z",
"expirationDate": 1681379023,
"expirationDateTime": "2023-04-13T09:43:40.000Z",
"subject": "CN=test.com,O=chicago,L=los,ST=Nord,C=PE",
"subjectAlternativeName": "DNS:httpsN-dep.san-dc.yu.com, DNS:test1.com, DNS:test.com"
},
"item.subjectAlternativeName.split(',') | map('regex_replace', '.*DNS:(.*)', 'DNS:\\\\1') | list | sort": [
"DNS:\\1",
"DNS:\\1",
"DNS:\\1"
]
}
When I use the same expression in a debug msg it works as expected.
ok: [localhost] => (item={'createDateTime': '2023-11-27T09:23:39.000Z', 'expirationDate': 1781379023, 'expirationDateTime': '2023-04-13T09:43:40.000Z', 'subject': 'CN=test.com,O=chicago,L=los,ST=Nord,C=PE', 'subjectAlternativeName': 'DNS:test.com, DNS:test1.com, DNS:httpsN-dep.san-dc.yu.com'}) => {
"msg": [
"DNS:httpsN-dep.san-dc.yu.com",
"DNS:test.com",
"DNS:test1.com"
]
}
How can I troubleshoot this condition ?
when: item.subjectAlternativeName.split(',') | map('regex_replace', '.*DNS:(.*)', 'DNS:\\1') | list | sort == alt_name

Create dictionary from list of strings using delimiter with Ansible

I have a nested list of strings, which I am trying to convert into a dictionary. The list seems to be in a reasonable format, but my dictionary is getting overwritten each time I append to it.
Initial list:
TASK [test first_list] **********************************************************************************************************************************************************************************************************************************************************************************************************
ok: [node11] => {
"first_list": [
[
"DEVNAME: /dev/sanstorage/node11-data103-2fd31b74e1ef5a8006c9ce9000b10399f",
"UUID: 2751719f-d1a2-49b0-9e42-3d686c90d2e6",
"TYPE: xfs"
],
[
"DEVNAME: /dev/sanstorage/node11-data104-2e81f1279171dce656c9ce9000b10399f",
"UUID: 6a6265d6-b103-471e-9e25-e8cc5f5585a8",
"TYPE: xfs"
],
[
"DEVNAME: /dev/sanstorage/node11-data102-2e385a327788d866e6c9ce9000b10399f",
"UUID: 8c52d974-8584-4aa6-89b8-f1e1db016118",
"TYPE: xfs"
],
[
"DEVNAME: /dev/sanstorage/node11-data101-241afe5ab93b5c0ee6c9ce9000b10399f",
"UUID: 94b56164-6717-4b82-8f11-86fd94a39672",
"TYPE: xfs"
],
[
"DEVNAME: /dev/sanstorage/node11-data100-25626d38d4c4239456c9ce9000b10399f",
"UUID: 55a1a388-fe0a-4dd0-980a-a10c5317952e",
"TYPE: xfs"
],
[
"DEVNAME: /dev/sanstorage/node11-data114-231d2661b7ab88f8f6c9ce9000b10399f",
"UUID: f87ad708-1d12-41a5-9441-d32a97b5318c",
"TYPE: ext4"
],
[
"DEVNAME: /dev/sanstorage/node11-data115-2d3a824975e90550f6c9ce9000b10399f",
"UUID: b8b79886-9710-4205-900b-7b9d7d4ad933",
"TYPE: ext4"
],
[
"DEVNAME: /dev/sanstorage/oneview-FA-data8165-284392eae1ad17d846c9ce9000b10399f",
"UUID: c5a43057-676e-49b7-b2a9-b53a40f7010b",
"TYPE: xfs"
],
[
"DEVNAME: /dev/sanstorage/oneview-FA-data3420-2e262703236f9b7046c9ce9000b10399f",
"UUID: 1a70c187-9364-4f48-92f8-f9b9dec9824f",
"TYPE: xfs"
],
[
"DEVNAME: /dev/sanstorage/node11-data112-2464954fd324508e66c9ce9000b10399f",
"UUID: 2238a12e-2ca1-466e-8617-11051e1d612e",
"TYPE: ext4"
],
[
"DEVNAME: /dev/sanstorage/node11-data111-25bb14827531149456c9ce9000b10399f",
"UUID: db37479f-80b0-46b2-85e0-aef679bea164",
"TYPE: ext4"
],
[
"DEVNAME: /dev/sanstorage/node11-data105-28ac6825aa80520d46c9ce9000b10399f",
"UUID: e987fc7c-9a4f-46ea-91f0-4d5f37d3421e",
"TYPE: xfs"
],
[
"DEVNAME: /dev/sanstorage/node11-data113-2177c50c4dd5063506c9ce9000b10399f",
"UUID: 36c1194d-ac4f-4cee-8688-353974cb6be0",
"TYPE: ext4"
]
]
}
From here, I try to form a dictionary, but only the last list entry is stored at the end:
TASK [test final_list] **********************************************************************************************************************************************************************************************************************************************************************************************************
ok: [node11] => {
"final_list": {
"DEVNAME": "/dev/sanstorage/node11-data113-2177c50c4dd5063506c9ce9000b10399f",
"TYPE": "ext4",
"UUID": "36c1194d-ac4f-4cee-8688-353974cb6be0"
}
}
Ideally, it would look something like this, I'm guessing I need to nest the dictionary within a list?
"final list": [
{
"DEVNAME: /dev/sanstorage/node11-data103-2fd31b74e1ef5a8006c9ce9000b10399f",
"UUID: 2751719f-d1a2-49b0-9e42-3d686c90d2e6",
"TYPE: xfs"
},
{
"DEVNAME: /dev/sanstorage/node11-data104-2e81f1279171dce656c9ce9000b10399f",
"UUID: 6a6265d6-b103-471e-9e25-e8cc5f5585a8",
"TYPE: xfs"
},
{
"DEVNAME: /dev/sanstorage/node11-data102-2e385a327788d866e6c9ce9000b10399f",
"UUID: 8c52d974-8584-4aa6-89b8-f1e1db016118",
"TYPE: xfs"
},
... snip ...
Below is the important part of the playbook:
- set_fact:
first_list: "{{ first_list | default([]) + [disk_data.split('\n')] }}"
vars:
disk_data: '{{ item.stdout }}'
with_items: "{{ disk_check.results }}"
- name: test first_list
debug:
var: first_list
- set_fact: final_list:{}
- set_fact:
final_list: "{{ final_list | default([]) | combine(dict([ item.partition(':')[::2]|map('trim')])) }}"
with_items: "{{ first_list }}"
- name: test final_list
debug:
var: final_list
Thoughts?
TL;DR
- debug:
msg: "{{ disk_check.results | map(attribute='stdout') | map('from_yaml') | list }}"
You did not provide the initial task which returns disk_check. From your usage in your example, I'll take for granted it is a shell or command task used in a loop.
Meanwhile you still provided enough information so we can avoid falling into an x/y problem. Your question is basically "how can I split a string on delimiters and use the result to create a dict?" where what you really want is more "How can I parse a string representing a yaml dict into an actual dict?"
To start with
- set_fact:
first_list: "{{ first_list | default([]) + [disk_data.split('\n')] }}"
vars:
disk_data: '{{ item.stdout }}'
with_items: "{{ disk_check.results }}"
This is going through all your command results, spliting stdout on a new line and adding that list to a top level list.
Since the output of shell/command contains a stdout_lines attribute and that we can extract attributes from a list of dicts with the jinja2 map filter, this could be replaced in one go without having to run a task at all with the following jinja expression:
"{{ disk_check.results | map(attribute='stdout_lines') | list }}"
But we would still be walking the wrong path.
Let's have a look at one of your (reconstructed) individual stdout from results
"stdout": "DEVNAME: /dev/sanstorage/node11-data102-2e385a327788d866e6c9ce9000b10399f\nUUID: 8c52d974-8584-4aa6-89b8-f1e1db016118\nTYPE: xfs"
This is a string representation of a yaml dict. Ansible has a from_yaml filter. And we can use the map filter to apply a filter to each element in a list.
The below playbook tries to reproduce your original data to display it in one go
---
- name: Parse a yaml dict in each element of a list
hosts: localhost
gather_facts: false
vars:
example_disks: 3
tasks:
- name: Faking your disk check supposed command
shell: |-
cat <<EOF
DEVNAME: /dev/sanstorage/node{{ (item | string)*2 }}-data{{ (item | string)*3 }}-$(uuidgen)
UUID: $(uuidgen)
TYPE: xfs
EOF
loop: "{{ range(1, example_disks+1) }}"
register: disk_check
- name: Show the original data (use -v to trigger)
debug:
var: disk_check
verbosity: 1
- name: Display a list of dicts from the above result
debug:
msg: "{{ disk_check.results | map(attribute='stdout') | map('from_yaml') | list }}"
and gives (run with -v to show the intermediate debug):
PLAY [Parse a yaml dict in each element of a list] *************************************************************************************************************************************************************************************
TASK [Faking your disk check supposed command] *****************************************************************************************************************************************************************************************
changed: [localhost] => (item=1)
changed: [localhost] => (item=2)
changed: [localhost] => (item=3)
TASK [Show the original data (use -v to trigger)] **********************************************************************************************************************************************************************************************************
skipping: [localhost]
TASK [Display a list of dicts from the above result] ***********************************************************************************************************************************************************************************
ok: [localhost] => {
"msg": [
{
"DEVNAME": "/dev/sanstorage/node11-data111-10e7a318-62af-46db-895e-5d94b0c2cf88",
"TYPE": "xfs",
"UUID": "5a87ae1c-c312-4325-88ac-b9cc1edfa69b"
},
{
"DEVNAME": "/dev/sanstorage/node22-data222-18247d1d-87b2-4c7d-8d48-cd333d7530f9",
"TYPE": "xfs",
"UUID": "bc0d7f1a-16e4-4694-b3e2-904e69cc208d"
},
{
"DEVNAME": "/dev/sanstorage/node33-data333-21bc7fde-645f-4cf2-9e18-72d004700085",
"TYPE": "xfs",
"UUID": "58985028-3eb1-4f34-90e9-f0e4c8257607"
}
]
}
PLAY RECAP *****************************************************************************************************************************************************************************************************************************
localhost : ok=2 changed=1 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0
Let's simplify the data, e.g.
first_list:
- - a: 1
- b: 1
- - a: 2
- b: 2
The expected result is
final_list:
- a: 1
b: 1
- a: 2
b: 2
The task below does the job
- set_fact:
final_list: "{{ final_list|default([]) + [_dict|from_yaml] }}"
loop: "{{ first_list }}"
vars:
_dict: |
{% for i in item %}
{{ (i|to_yaml)[1:-2] }}
{% endfor %}
There are other options on how to format the item, e.g.
{{ i.keys()|first }}: {{ i.values()|first }}
Write a filter if you use such conversions frequently, e.g.
shell> cat filter_plugins/list_filters.py
def list2dict(l):
out = []
for i in l:
item = {}
for j in range(0, len(i)):
item.update(i[j])
out.append(item)
return out
class FilterModule(object):
''' List filters. '''
def filters(self):
return {
'list2dict': list2dict,
}
Then the simple filter below gives the same result
- set_fact:
final_list: "{{ first_list|list2dict }}"

problems with nested json_query

I have the following json structure.
"results": [
{
"ltm_pools": [
{
"members": [
{
"full_path": "/Common/10.49.128.185:8080",
},
{
"full_path": "/Common/10.49.128.186:8080",
}
"name": "Staging-1-stresslab",
},
{
"members": [
{
"full_path": "/Common/10.49.128.187:0",
},
{
"full_path": "/Common/10.49.128.188:0",
}
],
"name": "Staging-2-lab",
},
I get an error when trying to do something like this
- debug:
msg: "{{item[0].host}} --- {{ item[1] }} --- {{ item[2] }}"
with_nested:
- "{{F5_hosts}}"
- "{{bigip_facts | json_query('[results[0].ltm_pools[*].name]') | flatten }}"
- "{{bigip_facts | json_query('[results[0].ltm_pools[?name.contains(#,'Staging'].members[::2].full_path]') | flatten }}"
I am unable to get the third array working.
I want to print the even members full_path variable from all objects where name contains staging.
I hope someone can help me I've been struggling with this for days.
From what I see/read/tried myself, you fell in this bug: https://github.com/ansible/ansible/issues/27299
This is the problem of "contains" JMESPath function as it is run by Ansible, to quote:
https://github.com/ansible/ansible/issues/27299#issuecomment-331068246
The problem is related to the fact that Ansible uses own types for strings: AnsibleUnicode and AnsibleUnsafeText.
And as long as jmespath library has very strict type-checking, it fails to accept this types as string literals.
There is also a suggested workaround, if you convert the variable to json and back, the strings in there have the correct type. Making long story short, this doesn't work:
"{{bigip_facts | json_query('results[0].ltm_pools[?name.contains(#,`Staging`)==`true`].members[::2].full_path') }}"
but this does:
"{{bigip_facts | to_json | from_json | json_query('results[0].ltm_pools[?name.contains(#,`Staging`)==`true`].members[::2].full_path') }}"
I've managed to run such a code:
- hosts: all
gather_facts: no
tasks:
- set_fact:
bigip_facts:
results:
- ltm_pools:
- members:
- full_path: "/Common/10.49.128.185:8080"
- full_path: "/Common/10.49.128.186:8080"
name: "Staging-1-stresslab"
- members:
- full_path: "/Common/10.49.128.187:0"
- full_path: "/Common/10.49.128.188:0"
name: "Staging-2-stresslab"
- name: "Debug ltm-pools"
debug:
msg: "{{ item }}"
with_items:
- "{{bigip_facts | to_json | from_json | json_query('results[0].ltm_pools[?name.contains(#,`Staging`)==`true`].members[::2].full_path') }}"
And it works as you wanted:
PLAY [all] *****************************************************************************************
TASK [set_fact] ************************************************************************************
ok: [localhost]
TASK [Debug ltm-pools] *****************************************************************************
ok: [localhost] => (item=[u'/Common/10.49.128.185:8080']) => {
"msg": [
"/Common/10.49.128.185:8080"
]
}
ok: [localhost] => (item=[u'/Common/10.49.128.187:0']) => {
"msg": [
"/Common/10.49.128.187:0"
]
}
PLAY RECAP *****************************************************************************************
localhost : ok=2 changed=0 unreachable=0 failed=0

How to create list for with_item from included list

I have this list:
"mylist": [
{ "files": [{"path": "path11"}, {"path": "path12"}] },
{ "files": [{"path": "path21"}, {"path": "path22"}] }
]
and I need to run role with with_items, where items should be path elements from my list.
For example:
- debug: msg="{{ item }}"
with_items: "{{ mylist | some_magic }}"
Needed output:
TASK [test : debug] **********************************
ok: [host] => (item=path11 ) => {
"msg": "path11"
}
ok: [host] => (item=path12 ) => {
"msg": "path12"
}
ok: [host] => (item=path21 ) => {
"msg": "path21"
}
...
Is it possible?
This is what I have already tried:
Constructions look like this:
- debug: msg="{{ item }}"
with_items: "{% for files in mylist | map(attribute='files') %}{% for file in files %}{{file.path}}{% endfor %}{% endfor %}"
Returned value as expected is not a list.
Wrong constructions look like this:
- debug: msg="{{ item }}"
with_items: "{{ mylist | map(attribute='files') | map(attribute='path') | list }}"
It is a legacy with_subelements loop pattern.
Using the loop keyword (Ansible 2.5 and later) you can iterate with:
- debug:
msg: "{{ item.1.path }}"
loop: "{{ mylist | subelements('files') }}"
Or the same using JMESPath (this shows how to create a list out of the whole data structure):
- debug:
msg: "{{ item }}"
loop: "{{ mylist | json_query('[].files[].path') }}"

Resources