Ansible - How to combine list attributes? - ansible

Question

I have two separate lists. The first is a list (base_list) with basic parameters, and the second is a list (dev_list) with parameters for a specific stand.
"base_list": [
{
"name": "kibana",
"path": "kibana/conf/kibana.xml",
"src": "/Users/ansible/inventories/_base/group_vars/kibana/conf/kibana.xml"
},
{
"name": "logstash",
"path": "logstash/conf/logstash.yml",
"src": "/Users/ansible/inventories/_base/group_vars/logstash/conf/logstash.yml"
},
{
"name": "grafana",
"path": "grafana/conf/grafana.json",
"src": "/Users/ansible/inventories/_base/group_vars/grafana/conf/grafana.json"
},
{
"name": "grafana",
"path": "grafana/conf/nginx.json",
"src": "/Users/ansible/inventories/_base/group_vars/grafana/conf/nginx.json"
},
{
"name": "grafana",
"path": "grafana/conf/config.json",
"src": "/Users/ansible/inventories/_base/group_vars/grafana/conf/config.json"
},
]
"dev_list": [
{
"name": "kibana",
"path": "kibana/conf/kibana.xml",
"src": "/Users/ansible/inventories/dev-st/group_vars/kibana/conf/kibana.xml"
},
{
"name": "logstash",
"path": "logstash/conf/jvm.options",
"src": "/Users/ansible/inventories/dev-st/group_vars/logstash/conf/jvm.options"
}
]
My goal is to combine these two lists to get one item.name with several item.path and item.src. Paths that look like this:
"end_list": [
{
"name": "kibana",
"path": "kibana/conf/kibana.xml",
"src": "/Users/ansible/inventories/dev-st/group_vars/kibana/conf/kibana.xml"
},
{
"name": "logstash",
"path": [
"logstash/conf/logstash.yml",
"logstash/conf/jvm.options"
],
"src": [
"/Users/ansible/inventories/_base/group_vars/logstash/conf/logstash.yml",
"/Users/ansible/inventories/dev-st/group_vars/logstash/conf/jvm.options"
]
},
{
"name": "grafana",
"path": [
"grafana/conf/grafana.json",
"grafana/conf/nginx.json",
"grafana/conf/config.json"
]
"src": [
"/Users/ansible/inventories/_base/group_vars/grafana/conf/grafana.json",
"/Users/ansible/inventories/_base/group_vars/grafana/conf/nginx.json",
"/Users/ansible/inventories/_base/group_vars/grafana/conf/config.json"
]
},
]
What would be the best way to do this?

Answer 1

This would probably be easier with a custom Python filter, but here's a solution using Ansible's built-in filters:
---
- hosts: localhost
gather_facts: false
vars:
"base_list": [
{
"name": "kibana",
"path": "kibana/conf/kibana.xml",
"src": "/Users/ansible/inventories/_base/group_vars/kibana/conf/kibana.xml"
},
{
"name": "logstash",
"path": "logstash/conf/logstash.yml",
"src": "/Users/ansible/inventories/_base/group_vars/logstash/conf/logstash.yml"
},
{
"name": "grafana",
"path": "grafana/conf/grafana.json",
"src": "/Users/ansible/inventories/_base/group_vars/grafana/conf/grafana.json"
},
]
"dev_list": [
{
"name": "kibana",
"path": "kibana/conf/kibana.xml",
"src": "/Users/ansible/inventories/dev-st/group_vars/kibana/conf/kibana.xml"
},
{
"name": "logstash",
"path": "logstash/conf/jvm.options",
"src": "/Users/ansible/inventories/dev-st/group_vars/logstash/conf/jvm.options"
}
]
tasks:
- set_fact:
end_list: >-
{{ end_list|default([]) + [
{
'name': item.0.name,
'path': item.1.path|ternary([item.0.path, item.1.path], item.0.path),
'src': item.1.src|ternary([item.0.src, item.1.src], item.1.src)
}
]}}
loop: >-
{{ base_list|zip_longest(dev_list,
fillvalue={'path': false, 'src': false})|list }}
- debug:
var: end_list
This was a little tricky to put together, so I'll try to describe the various parts:
The loop uses the zip_longest filter. Given the lists list1=[1, 2, 3] and list2=[11, 12], list1|zip_longest(list2) would produce [[1,11], [2,12], [3,None]] (that is, by default, zip_longest will use None as a fill value if one list is shorter than the other). By setting the fillvalue parameter, we can use a value other than None. In this case...
loop: >-
{{ base_list|zip_longest(dev_list,
fillvalue={'path': false, 'src': false})|list }}
...We're setting the fill value to a dictionary with stub values for path and src, since this makes the rest of the expression easier.
The meat of the solution is of course the set_fact action, which in simplified form looks like:
end_list: "{{ end_list|default([]) + [{...a dictionary...}] }}"
In other words, for each iteration of the loop, this will append a new dictionary to end_list.
We create the dictionary like this:
{
'name': item.0.name,
'path': item.1.path|ternary([item.0.path, item.1.path], item.0.path),
'src': item.1.src|ternary([item.0.src, item.1.src], item.1.src)
}
We're using the ternary filter here, which evaluates it's input as a boolean; if it's true, it selects the first argument, otherwise the second. Here we're taking advantage of the fillvalue we passed to the zip_longest filter: if dev_list is shorter than base_list, we'll have some items for which item.1.path and item.1.src are false, causing the ternary filter to select the second value (either item.0.path or item.1.src). In other cases, we build a list by combining the values from each of base_list and dev_list.
The result of running this playbook looks like:
ok: [localhost] => {
"end_list": [
{
"name": "kibana",
"path": [
"kibana/conf/kibana.xml",
"kibana/conf/kibana.xml"
],
"src": [
"/Users/ansible/inventories/_base/group_vars/kibana/conf/kibana.xml",
"/Users/ansible/inventories/dev-st/group_vars/kibana/conf/kibana.xml"
]
},
{
"name": "logstash",
"path": [
"logstash/conf/logstash.yml",
"logstash/conf/jvm.options"
],
"src": [
"/Users/ansible/inventories/_base/group_vars/logstash/conf/logstash.yml",
"/Users/ansible/inventories/dev-st/group_vars/logstash/conf/jvm.options"
]
},
{
"name": "grafana",
"path": "grafana/conf/grafana.json",
"src": false
}
]
}
Let me know if that helps, and whether or not the resulting data structure is what you were looking for. I had to make a few assumptions since your example end_list contained invalid syntax, so I took a guess at what you wanted.

Answer 2

Assuming you had well-formed json and those are properties on the root object, jq is perfectly suited for this. Group the contents of the arrays by name then generate the appropriate result objects.
$ jq '{
end_combine: (
.base_list + .dev_list
| group_by(.name)
| map({ name: .[0].name, path: map(.path), src: map(.src) })
)
}' input.json
{
"end_combine": [
{
"name": "grafana",
"path": [
"grafana/conf/grafana.json",
"grafana/conf/nginx.json",
"grafana/conf/config.json"
],
"src": [
"/Users/ansible/inventories/_base/group_vars/grafana/conf/grafana.json",
"/Users/ansible/inventories/_base/group_vars/grafana/conf/nginx.json",
"/Users/ansible/inventories/_base/group_vars/grafana/conf/config.json"
]
},
{
"name": "kibana",
"path": [
"kibana/conf/kibana.xml",
"kibana/conf/kibana.xml"
],
"src": [
"/Users/ansible/inventories/_base/group_vars/kibana/conf/kibana.xml",
"/Users/ansible/inventories/dev-st/group_vars/kibana/conf/kibana.xml"
]
},
{
"name": "logstash",
"path": [
"logstash/conf/logstash.yml",
"logstash/conf/jvm.options"
],
"src": [
"/Users/ansible/inventories/_base/group_vars/logstash/conf/logstash.yml",
"/Users/ansible/inventories/dev-st/group_vars/logstash/conf/jvm.options"
]
}
]
}