Create dictionary from list of strings using delimiter with Ansible - ansible

Question

I have a nested list of strings, which I am trying to convert into a dictionary. The list seems to be in a reasonable format, but my dictionary is getting overwritten each time I append to it.
Initial list:
TASK [test first_list] **********************************************************************************************************************************************************************************************************************************************************************************************************
ok: [node11] => {
"first_list": [
[
"DEVNAME: /dev/sanstorage/node11-data103-2fd31b74e1ef5a8006c9ce9000b10399f",
"UUID: 2751719f-d1a2-49b0-9e42-3d686c90d2e6",
"TYPE: xfs"
],
[
"DEVNAME: /dev/sanstorage/node11-data104-2e81f1279171dce656c9ce9000b10399f",
"UUID: 6a6265d6-b103-471e-9e25-e8cc5f5585a8",
"TYPE: xfs"
],
[
"DEVNAME: /dev/sanstorage/node11-data102-2e385a327788d866e6c9ce9000b10399f",
"UUID: 8c52d974-8584-4aa6-89b8-f1e1db016118",
"TYPE: xfs"
],
[
"DEVNAME: /dev/sanstorage/node11-data101-241afe5ab93b5c0ee6c9ce9000b10399f",
"UUID: 94b56164-6717-4b82-8f11-86fd94a39672",
"TYPE: xfs"
],
[
"DEVNAME: /dev/sanstorage/node11-data100-25626d38d4c4239456c9ce9000b10399f",
"UUID: 55a1a388-fe0a-4dd0-980a-a10c5317952e",
"TYPE: xfs"
],
[
"DEVNAME: /dev/sanstorage/node11-data114-231d2661b7ab88f8f6c9ce9000b10399f",
"UUID: f87ad708-1d12-41a5-9441-d32a97b5318c",
"TYPE: ext4"
],
[
"DEVNAME: /dev/sanstorage/node11-data115-2d3a824975e90550f6c9ce9000b10399f",
"UUID: b8b79886-9710-4205-900b-7b9d7d4ad933",
"TYPE: ext4"
],
[
"DEVNAME: /dev/sanstorage/oneview-FA-data8165-284392eae1ad17d846c9ce9000b10399f",
"UUID: c5a43057-676e-49b7-b2a9-b53a40f7010b",
"TYPE: xfs"
],
[
"DEVNAME: /dev/sanstorage/oneview-FA-data3420-2e262703236f9b7046c9ce9000b10399f",
"UUID: 1a70c187-9364-4f48-92f8-f9b9dec9824f",
"TYPE: xfs"
],
[
"DEVNAME: /dev/sanstorage/node11-data112-2464954fd324508e66c9ce9000b10399f",
"UUID: 2238a12e-2ca1-466e-8617-11051e1d612e",
"TYPE: ext4"
],
[
"DEVNAME: /dev/sanstorage/node11-data111-25bb14827531149456c9ce9000b10399f",
"UUID: db37479f-80b0-46b2-85e0-aef679bea164",
"TYPE: ext4"
],
[
"DEVNAME: /dev/sanstorage/node11-data105-28ac6825aa80520d46c9ce9000b10399f",
"UUID: e987fc7c-9a4f-46ea-91f0-4d5f37d3421e",
"TYPE: xfs"
],
[
"DEVNAME: /dev/sanstorage/node11-data113-2177c50c4dd5063506c9ce9000b10399f",
"UUID: 36c1194d-ac4f-4cee-8688-353974cb6be0",
"TYPE: ext4"
]
]
}
From here, I try to form a dictionary, but only the last list entry is stored at the end:
TASK [test final_list] **********************************************************************************************************************************************************************************************************************************************************************************************************
ok: [node11] => {
"final_list": {
"DEVNAME": "/dev/sanstorage/node11-data113-2177c50c4dd5063506c9ce9000b10399f",
"TYPE": "ext4",
"UUID": "36c1194d-ac4f-4cee-8688-353974cb6be0"
}
}
Ideally, it would look something like this, I'm guessing I need to nest the dictionary within a list?
"final list": [
{
"DEVNAME: /dev/sanstorage/node11-data103-2fd31b74e1ef5a8006c9ce9000b10399f",
"UUID: 2751719f-d1a2-49b0-9e42-3d686c90d2e6",
"TYPE: xfs"
},
{
"DEVNAME: /dev/sanstorage/node11-data104-2e81f1279171dce656c9ce9000b10399f",
"UUID: 6a6265d6-b103-471e-9e25-e8cc5f5585a8",
"TYPE: xfs"
},
{
"DEVNAME: /dev/sanstorage/node11-data102-2e385a327788d866e6c9ce9000b10399f",
"UUID: 8c52d974-8584-4aa6-89b8-f1e1db016118",
"TYPE: xfs"
},
... snip ...
Below is the important part of the playbook:
- set_fact:
first_list: "{{ first_list | default([]) + [disk_data.split('\n')] }}"
vars:
disk_data: '{{ item.stdout }}'
with_items: "{{ disk_check.results }}"
- name: test first_list
debug:
var: first_list
- set_fact: final_list:{}
- set_fact:
final_list: "{{ final_list | default([]) | combine(dict([ item.partition(':')[::2]|map('trim')])) }}"
with_items: "{{ first_list }}"
- name: test final_list
debug:
var: final_list
Thoughts?

Answer 1

TL;DR
- debug:
msg: "{{ disk_check.results | map(attribute='stdout') | map('from_yaml') | list }}"
You did not provide the initial task which returns disk_check. From your usage in your example, I'll take for granted it is a shell or command task used in a loop.
Meanwhile you still provided enough information so we can avoid falling into an x/y problem. Your question is basically "how can I split a string on delimiters and use the result to create a dict?" where what you really want is more "How can I parse a string representing a yaml dict into an actual dict?"
To start with
- set_fact:
first_list: "{{ first_list | default([]) + [disk_data.split('\n')] }}"
vars:
disk_data: '{{ item.stdout }}'
with_items: "{{ disk_check.results }}"
This is going through all your command results, spliting stdout on a new line and adding that list to a top level list.
Since the output of shell/command contains a stdout_lines attribute and that we can extract attributes from a list of dicts with the jinja2 map filter, this could be replaced in one go without having to run a task at all with the following jinja expression:
"{{ disk_check.results | map(attribute='stdout_lines') | list }}"
But we would still be walking the wrong path.
Let's have a look at one of your (reconstructed) individual stdout from results
"stdout": "DEVNAME: /dev/sanstorage/node11-data102-2e385a327788d866e6c9ce9000b10399f\nUUID: 8c52d974-8584-4aa6-89b8-f1e1db016118\nTYPE: xfs"
This is a string representation of a yaml dict. Ansible has a from_yaml filter. And we can use the map filter to apply a filter to each element in a list.
The below playbook tries to reproduce your original data to display it in one go
---
- name: Parse a yaml dict in each element of a list
hosts: localhost
gather_facts: false
vars:
example_disks: 3
tasks:
- name: Faking your disk check supposed command
shell: |-
cat <<EOF
DEVNAME: /dev/sanstorage/node{{ (item | string)*2 }}-data{{ (item | string)*3 }}-$(uuidgen)
UUID: $(uuidgen)
TYPE: xfs
EOF
loop: "{{ range(1, example_disks+1) }}"
register: disk_check
- name: Show the original data (use -v to trigger)
debug:
var: disk_check
verbosity: 1
- name: Display a list of dicts from the above result
debug:
msg: "{{ disk_check.results | map(attribute='stdout') | map('from_yaml') | list }}"
and gives (run with -v to show the intermediate debug):
PLAY [Parse a yaml dict in each element of a list] *************************************************************************************************************************************************************************************
TASK [Faking your disk check supposed command] *****************************************************************************************************************************************************************************************
changed: [localhost] => (item=1)
changed: [localhost] => (item=2)
changed: [localhost] => (item=3)
TASK [Show the original data (use -v to trigger)] **********************************************************************************************************************************************************************************************************
skipping: [localhost]
TASK [Display a list of dicts from the above result] ***********************************************************************************************************************************************************************************
ok: [localhost] => {
"msg": [
{
"DEVNAME": "/dev/sanstorage/node11-data111-10e7a318-62af-46db-895e-5d94b0c2cf88",
"TYPE": "xfs",
"UUID": "5a87ae1c-c312-4325-88ac-b9cc1edfa69b"
},
{
"DEVNAME": "/dev/sanstorage/node22-data222-18247d1d-87b2-4c7d-8d48-cd333d7530f9",
"TYPE": "xfs",
"UUID": "bc0d7f1a-16e4-4694-b3e2-904e69cc208d"
},
{
"DEVNAME": "/dev/sanstorage/node33-data333-21bc7fde-645f-4cf2-9e18-72d004700085",
"TYPE": "xfs",
"UUID": "58985028-3eb1-4f34-90e9-f0e4c8257607"
}
]
}
PLAY RECAP *****************************************************************************************************************************************************************************************************************************
localhost : ok=2 changed=1 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0

Answer 2

Let's simplify the data, e.g.
first_list:
- - a: 1
- b: 1
- - a: 2
- b: 2
The expected result is
final_list:
- a: 1
b: 1
- a: 2
b: 2
The task below does the job
- set_fact:
final_list: "{{ final_list|default([]) + [_dict|from_yaml] }}"
loop: "{{ first_list }}"
vars:
_dict: |
{% for i in item %}
{{ (i|to_yaml)[1:-2] }}
{% endfor %}
There are other options on how to format the item, e.g.
{{ i.keys()|first }}: {{ i.values()|first }}
Write a filter if you use such conversions frequently, e.g.
shell> cat filter_plugins/list_filters.py
def list2dict(l):
out = []
for i in l:
item = {}
for j in range(0, len(i)):
item.update(i[j])
out.append(item)
return out
class FilterModule(object):
''' List filters. '''
def filters(self):
return {
'list2dict': list2dict,
}
Then the simple filter below gives the same result
- set_fact:
final_list: "{{ first_list|list2dict }}"