I have a use case where I need to merge 2 lists on common key name using Ansible.
List1:
{
"poc-cu2": [
"40:A6:B7:5E:22:11",
"40:A6:B7:5E:22:22"
],
"test2211": [
"40:A6:B7:5E:33:11",
"40:A6:B7:5E:33:22"
],
"test2244": [
"40:A6:B7:5E:22:45",
"40:A6:B7:5E:22:46"
]
}
List2:
{
"poc-cu2": [
"root",
"9WKA3KK3XN39",
"9.3.13.44"
],
"test2211": [
"root2211",
"221122112211",
"9.3.13.82"
]
}
Expected:
List3:
{
"poc-cu2": [
"root",
"9WKA3KK3XN39",
"9.3.13.44",
"40:A6:B7:5E:22:11",
"40:A6:B7:5E:22:22"
],
"test2211": [
"root2211",
"221122112211",
"9.3.13.82",
"40:A6:B7:5E:33:11",
"40:A6:B7:5E:33:22"
]
}
I got how to merge 2 lists using unique key but on my case I need to merge only on common key, please suggest.
You can get most of what you want using the combine filter, like this:
- hosts: localhost
gather_facts: false
tasks:
- set_fact:
dict3: "{{ dict1|combine(dict2, list_merge='append') }}"
- debug:
var: dict3
This will produce:
TASK [debug] *******************************************************************
ok: [localhost] => {
"dict3": {
"poc-cu2": [
"40:A6:B7:5E:22:11",
"40:A6:B7:5E:22:22",
"root",
"9WKA3KK3XN39",
"9.3.13.44"
],
"test2211": [
"40:A6:B7:5E:33:11",
"40:A6:B7:5E:33:22",
"root2211",
"221122112211",
"9.3.13.82"
],
"test2244": [
"40:A6:B7:5E:22:45",
"40:A6:B7:5E:22:46"
]
}
}
If you want the final result to consist of only the keys common to both
dictionaries it gets a little trickier, but this seems to work:
- hosts: localhost
gather_facts: false
tasks:
- set_fact:
dict3: "{{ dict3|combine({item: dict1[item] + dict2[item]}) }}"
when: item in dict2
loop: "{{ dict1.keys() }}"
vars:
dict3: {}
- debug:
var: dict3
Which produces:
TASK [debug] *******************************************************************
ok: [localhost] => {
"dict3": {
"poc-cu2": [
"40:A6:B7:5E:22:11",
"40:A6:B7:5E:22:22",
"root",
"9WKA3KK3XN39",
"9.3.13.44"
],
"test2211": [
"40:A6:B7:5E:33:11",
"40:A6:B7:5E:33:22",
"root2211",
"221122112211",
"9.3.13.82"
]
}
}
The above works by iterating over the keys in dict1, and for each
key from dict1 that also exists in dict2, we synthesize a new
dictionary containing the corresponding values from both dict1 and dict2 and then merge it into our final dictionary using the combine filter.
Given the data
dict1:
poc-cu2:
- 40:A6:B7:5E:22:11
- 40:A6:B7:5E:22:22
test2211:
- 40:A6:B7:5E:33:11
- 40:A6:B7:5E:33:22
test2244:
- 40:A6:B7:5E:22:45
- 40:A6:B7:5E:22:46
dict2:
poc-cu2:
- root
- 9WKA3KK3XN39
- 9.3.13.44
test2211:
- root2211
- '221122112211'
- 9.3.13.82
Iteration is not needed. Put the declarations below as appropriate. The dictionary dict_cmn keeps the merged common attributes of dict1 and dict2
dict_1_2: "{{ dict1|combine(dict2, list_merge='append') }}"
keys_cmn: "{{ dict1.keys()|intersect(dict2.keys()) }}"
vals_cmn: "{{ keys_cmn|map('extract', dict_1_2) }}"
dict_cmn: "{{ dict(keys_cmn|zip(vals_cmn)) }}"
give
dict_1_2:
poc-cu2:
- 40:A6:B7:5E:22:11
- 40:A6:B7:5E:22:22
- root
- 9WKA3KK3XN39
- 9.3.13.44
test2211:
- 40:A6:B7:5E:33:11
- 40:A6:B7:5E:33:22
- root2211
- '221122112211'
- 9.3.13.82
test2244:
- 40:A6:B7:5E:22:45
- 40:A6:B7:5E:22:46
keys_cmn:
- poc-cu2
- test2211
vals_cmn:
- - 40:A6:B7:5E:22:11
- 40:A6:B7:5E:22:22
- root
- 9WKA3KK3XN39
- 9.3.13.44
- - 40:A6:B7:5E:33:11
- 40:A6:B7:5E:33:22
- root2211
- '221122112211'
- 9.3.13.82
dict_cmn:
poc-cu2:
- 40:A6:B7:5E:22:11
- 40:A6:B7:5E:22:22
- root
- 9WKA3KK3XN39
- 9.3.13.44
test2211:
- 40:A6:B7:5E:33:11
- 40:A6:B7:5E:33:22
- root2211
- '221122112211'
- 9.3.13.82
Related
I have some servers with a lot of wordpress instances, who I ask them what versions they have.
- name: CONTADOR WP VERSIONES
shell: mycommand
register: wp_versions
- debug: msg: "{{ wp_versions.stdout_lines }}"
For example:
TASK [debug] *********************************************************************
ok: [server1] => {
"msg": [
"5.1.13"
]
}
ok: [server2] => {
"msg": [
"5.1.12",
"5.1.13"
]
}
ok: [server3] => {
"msg": [
"5.1.10",
"5.1.13",
]
}
I need to list a unique values like this:
"msg": [
"5.1.10",
"5.1.12",
"5.1.13",
]
I have tried all that i found but nothing works as I want.
Thanks
Use special variable ansible_play_hosts and extract the variables from the hostvars
- set_fact:
all_vers: "{{ ansible_play_hosts|
map('extract', hostvars, ['wp_versions', 'stdout_lines'])|
flatten|unique }}"
run_once: true
gives
all_vers:
- 5.1.13
- 5.1.12
- 5.1.10
You could do something like this:
- hosts: all
gather_facts: false
tasks:
- name: CONTADOR WP VERSIONES
shell: mycommand
register: wp_versions
- hosts: localhost
gather_facts: false
tasks:
# This tasks builds a flattened list of all the
# wp_versions.stdout_lines values collected from your hosts.
- name: Collect wp_versions information
set_fact:
all_wp_versions_pre: "{{ all_wp_versions_pre + hostvars[item].wp_versions.stdout_lines }}"
loop: "{{ groups.all }}"
vars:
all_wp_versions_pre: []
# Here we use the `unique` filter to produce a list of
# unique versions.
- name: Set all_wp_versions fact
set_fact:
all_wp_versions: "{{ all_wp_versions_pre|unique }}"
- debug:
var: all_wp_versions
Given you examples, this would produce the following output:
TASK [debug] ********************************************************************************************
ok: [localhost] => {
"all_wp_versions": [
"5.1.13",
"5.1.12",
"5.1.10"
]
}
I have a list of lists of hosts:
[['host-0', 'host-1'], ['host-2', 'host-3'], ['host-4', 'host-5', 'host-6']]
How can I add a port number, e.g., 8000, to each host using ansible/ jinja2 to get:
[['host-0:8000', 'host-1:8000'], ['host-2:8000', 'host-3:8000'], ['host-4:8000', 'host-5:8000', 'host-6:8000']]
this task shall do it:
- name: convert items in list
set_fact:
my_new_list: "{{ my_new_list | default([])+ [ my_var ] }}"
vars:
my_var: "{{ item | map('regex_replace', '$', ':8000') | list }}"
with_items:
- "{{ my_list }}"
full playbook to run as demo:
---
- hosts: localhost
gather_facts: false
vars:
my_list:
- ['host-0', 'host-1']
- ['host-2', 'host-3']
- ['host-4', 'host-5', 'host-6']
tasks:
- name: print original variable
debug:
var: my_list
- name: convert items in list
set_fact:
my_new_list: "{{ my_new_list | default([])+ [ my_var ] }}"
vars:
my_var: "{{ item | map('regex_replace', '$', ':8000') | list }}"
with_items:
- "{{ my_list }}"
- name: print new variable
debug:
var: my_new_list
result:
TASK [print new variable] **********************************************************************************************************************************************************************************************
ok: [localhost] => {
"my_new_list": [
[
"host-0:8000",
"host-1:8000"
],
[
"host-2:8000",
"host-3:8000"
],
[
"host-4:8000",
"host-5:8000",
"host-6:8000"
]
]
}
PLAY RECAP
Use map + regex_replace.
- debug:
msg: "{{ foo | map('map', 'regex_replace', '$', ':8000') }}"
vars:
foo: [['host-0', 'host-1'], ['host-2', 'host-3'], ['host-4', 'host-5', 'host-6']]
"msg": [
[
"host-0:8000",
"host-1:8000"
],
[
"host-2:8000",
"host-3:8000"
],
[
"host-4:8000",
"host-5:8000",
"host-6:8000"
]
]
I need to create a variable of ip addresses that are derived from the run state of all hosts. I expect to insert a new string value and have it appended to the list.
When I run the following ansible-playbook, it creates what looks to be a new list instance for each host instead of modifying the playbook level vars.
My understanding is the set_fact below should concatenate the lists and assign them back to the play scoped var ip_list_from_terraform. Why am I getting host scoped results?
---
- name: Bootstrap nodes
hosts: all
become: true
gather_facts: true
vars:
ip_list_from_terraform: ['Verify']
tasks:
- name: assign a list of all the physical network private IPs
set_fact:
ip_list_from_terraform: "{{ ip_list_from_terraform + [ item ] }}"
with_items: " {{ hostvars[inventory_hostname]['ansible_' + ansible_default_ipv4['interface']]['ipv4']['address'] }} "
register: ip_list_from_terraform_list
- name: Debug global var
debug:
var: ip_list_from_terraform
- name: Debug register result
debug:
var: ip_list_from_terraform_list
Expected:
ok: [shrimp-master-0] => {
"ip_list_from_terraform": [
"Verify",
"10.0.2.41",
"10.0.2.172",
"10.0.2.33",
"10.0.2.215",
"10.0.2.131",
"10.0.2.168",
"10.0.2.118"
]
}
Actual:
TASK [Debug global var] ********************************************************************************************************************************************************************************************************************
ok: [shrimp-master-0] => {
"ip_list_from_terraform": [
"Verify",
"10.0.2.12"
]
}
ok: [shrimp-master-1] => {
"ip_list_from_terraform": [
"Verify",
"10.0.2.33"
]
}
ok: [shrimp-master-2] => {
"ip_list_from_terraform": [
"Verify",
"10.0.2.215"
]
}
ok: [shrimp-worker-0-super-wallaby] => {
"ip_list_from_terraform": [
"Verify",
"10.0.2.131"
]
}
ok: [shrimp-gpu-worker-0-settled-wombat] => {
"ip_list_from_terraform": [
"Verify",
"10.0.2.151"
]
}
Let's simplify the case. Given the inventory
shell> cat hosts
host1 test_ip=10.0.2.41
host2 test_ip=10.0.2.172
host3 test_ip=10.0.2.33
the playbook
- hosts: host1,host2,host3
vars:
ip_list_from_terraform: ['Verify']
tasks:
- set_fact:
ip_list_from_terraform: "{{ ip_list_from_terraform + [ item ] }}"
with_items: "{{ hostvars[inventory_hostname]['test_ip'] }}"
- debug:
var: ip_list_from_terraform
gives
ok: [host2] =>
ip_list_from_terraform:
- Verify
- 10.0.2.172
ok: [host1] =>
ip_list_from_terraform:
- Verify
- 10.0.2.41
ok: [host3] =>
ip_list_from_terraform:
- Verify
- 10.0.2.33
As a side-note, with_items is needed because the argument is a string. loop would crash with the error 'Invalid data passed to ''loop'', it requires a list,.
Q: "set_fact should concatenate the lists"
A: Run once and loop ansible_play_hosts. For example
- set_fact:
ip_list_from_terraform: "{{ ip_list_from_terraform +
[hostvars[item]['test_ip']] }}"
loop: "{{ ansible_play_hosts }}"
run_once: true
gives
ip_list_from_terraform:
- Verify
- 10.0.2.41
- 10.0.2.172
- 10.0.2.33
I am attempting to build a dictionary but cannot grasp how jinja2 interpolates variables.
I want to set a specific item in the array (for example item[0]) to a specific key-value dictionary item.
- set_fact:
nodes:
- node1
- node2
- set_fact:
list_one:
- f-one
- f-two
- set_fact:
list_two:
- n-one
- n-two
what I want:
- set_fact:
**node_dict:
node1:
labels:
f-one: n-one
node2:
labels:
f-two: n-two**
When I run :
- name: check loop1
debug:
msg: '{{item[0]}} - {{item[1]}} - {{ item[2]}} '
with_nested:
- '{{ nodes }}'
- '{{ list_one }}'
- '{{ list_two }}'
item variable is availble. But doing this:
- set_fact:
final:
'{{item[0]}}':
labels:
"{{item[1] }}" : "{{item[2]}}"
with_nested:
- '{{ nodes }}'
- '{{ list_one }}'
- '{{ list_two }}'
results in an error.
Can someone explain why? How do I end up with my desired result?
Although your last piece of code above does not meet your requirement, It's perfectly valid: I'm not getting any error when running it.
As your are using it right now, set_fact is overwriting your final variable on each loop. To append element to a dict like your are trying to do, you need to initialize the var to an empty dict and combine it with the values you are calculating for each iteration. Since your calculated values are a dict themselves, you will need to use recursive=True if you have to write expressions deep inside the dict.
If I take into account your original data and your expected result, you want to relate the Nth element of each lists together. This is not what nested does (loop over nodes with a sub-loop on list_one sub-sub-loop on list_two....). In your case, you simply need to loop over an index of the length of your lists and combine the elements of same index together. My take below.
---
- name: test for SO
hosts: localhost
vars:
nodes:
- node1
- node2
list_one:
- f-one
- f-two
list_two:
- n-one
- n-two
tasks:
- name: Make my config
set_fact:
final: >-
{{
final
| default({})
| combine ({
nodes[item]: {
'labels': {
list_one[item]: list_two[item]
}
}
}, recursive=True)
}}
loop: "{{ range(nodes | length) | list }}"
- name: debug
debug:
var: final
which gives the following result
$ ansible-playbook test.yml
PLAY [test for SO] ******************************************************************
TASK [Gathering Facts] **************************************************************
ok: [localhost]
TASK [Make my config] ***************************************************************
ok: [localhost] => (item=0)
ok: [localhost] => (item=1)
TASK [debug] ************************************************************************
ok: [localhost] => {
"final": {
"node1": {
"labels": {
"f-one": "n-one"
}
},
"node2": {
"labels": {
"f-two": "n-two"
}
}
}
}
PLAY RECAP **************************************************************************
localhost : ok=3 changed=0 unreachable=0 failed=0
Edit: The same result can be acheived using the zip filter (which I (re)discovered today reading an other contribution).
- name: Make my config
set_fact:
final: >-
{{
final
| default({})
| combine ({
item.0: {
'labels': {
item.1: item.2
}
}
}, recursive=True)
}}
loop: "{{ nodes | zip(list_one, list_two) | list }}"
An option would be to create the list of labels and then combine the dictionary. The play below
- hosts: localhost
vars:
nodes:
- node1
- node2
list_one:
- f-one
- f-two
list_two:
- n-one
- n-two
node_dict: {}
my_labels: []
tasks:
- set_fact:
my_labels: "{{ my_labels + [ {list_one[my_idx]:list_two[my_idx]} ] }}"
loop: "{{ nodes }}"
loop_control:
index_var: my_idx
- set_fact:
node_dict: "{{ node_dict | combine({item:{'labels':my_labels[my_idx]}}) }}"
loop: "{{ nodes }}"
loop_control:
index_var: my_idx
- debug:
var: node_dict
gives:
"node_dict": {
"node1": {
"labels": {
"f-one": "n-one"
}
},
"node2": {
"labels": {
"f-two": "n-two"
}
}
}
I have a role in my playbook that generates two lists using set_fact.
The two facts are used in different tasks.
I then need to merge them for a final task.
list1:
- name: alice
roles: ['role1', 'role2']
- name: bob
roles: ['role1']
list2:
- name: alice
roles: ['role3']
- name: charlie
roles: ['role2']
For my final task I need the output to be:
list3:
- name: alice
roles: ['role1', 'role2', 'role3']
- name: bob
roles: ['role1']
- name: charlie
roles: ['role2']
I asked about lists vs dictionaries in the comment because of the impact it will have on the solution. If you were to restructure your data like this:
dict1:
alice: ['role1', 'role2']
bob: ['role1']
dict2:
alice: ['role3']
charlie: ['role2']
Then your solution becomes:
- set_fact:
dict3: >-
{{
dict3|default([])|combine({
item: (dict1[item]|default([]) + dict2[item]|default([]))|unique
})
}}
loop: "{{ (dict1.keys()|list + dict2.keys()|list)|unique }}"
- debug:
var: dict3
Which outputs:
TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
"dict3": {
"alice": [
"role1",
"role2",
"role3"
],
"bob": [
"role1"
],
"charlie": [
"role2"
]
}
}
If you're stuck with using lists, we can improve upon the json_query solution that Zeitounator suggested:
- set_fact:
list3: >-
{{
list3|default([]) + [{
'name': item,
'roles': (list1|json_query('[?name==`' + item + '`].roles[]') + list2|json_query('[?name==`' + item + '`].roles[]'))|unique
}]
}}
loop: "{{ (list1|json_query('[].name') + list2|json_query('[].name'))|unique }}"
- debug:
var: list3
This produces your desired output:
TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
"list3": [
{
"name": "alice",
"roles": [
"role1",
"role2",
"role3"
]
},
{
"name": "bob",
"roles": [
"role1"
]
},
{
"name": "charlie",
"roles": [
"role2"
]
}
]
}
This is a solution in plain ansible. If it becomes out of control because of your datastructure growth, you should consider writing your own filter (example)
---
- name: demo playbook for deeps dictionary merge
hosts: localhost
gather_facts: false
vars:
list1:
- name: alice
roles: ['role1', 'role2']
- name: bob
roles: ['role1']
list2:
- name: alice
roles: ['role3']
- name: charlie
roles: ['role2']
tasks:
- debug:
msg: >-
roles for {{ item }}:
{{
(list1 | json_query("[?name == '" + item +"'].roles")).0 | default([])
+
(list2 | json_query("[?name == '" + item +"'].roles")).0 | default([])
}}
loop: >-
{{
(
list1 | json_query("[].name")
+
list2 | json_query("[].name")
)
| unique
}}
Which gives:
PLAY [demo playbook for deeps dictionary merge] ************************************************************************************************************************************************************************************************************************
TASK [debug] ************************************************************************************************************************************************************************************************************************************************************
Wednesday 24 April 2019 16:23:07 +0200 (0:00:00.046) 0:00:00.046 *******
ok: [localhost] => (item=alice) => {
"msg": "roles for alice: ['role1', 'role2', 'role3']"
}
ok: [localhost] => (item=bob) => {
"msg": "roles for bob: ['role1']"
}
ok: [localhost] => (item=charlie) => {
"msg": "roles for charlie: ['role2']"
}