Replace the key value in a list with the relevant matching values from another list in Ansible - ansible

I am trying to replace the key values in a dict inside a list(here only for the values inside Objectids) with the matching values from another list. Somehow i am only able to replace the 1st value but not iterating the whole list. Here the same key in the input.json will have multiple matching values from the finallist.json and all those values needs to be matched and added to the get the final expected output.
Input.json
[
{
"name": "DNS_One",
"objectIds": [
"DNS_One",
"DNS_One-HO",
"NTP"
]
},
{
"name": "DNS_Two",
"objectIds": [
"NTP"
]
}
]
finallist.json
[
{
"id": "123456",
"net_name": "DNS_One"
},
{
"id": "789101112",
"net_name": "DNS_One"
},
{
"id": "131415161718",
"net_name": "DNS_One-HO"
},
{
"id": "23897845",
"net_name": "NTP"
},
{
"id": "9879879546",
"net_name": "NTP"
}
]
Playbook
- name: Id mapping
vars:
objectid: >-
{{
finallist
| selectattr('net_name', 'in', item.1)
| map(attribute = 'id')
| first
| default([])
}}
set_fact:
obj: >-
{{
obj | default([]) +
[item.0 | combine({'objectIds': [objectid]})]
}}
with_subelements:
- "{{ input }}"
- objectIds
ignore_errors: yes
Expected Output
[
{
"name": "DNS_One",
"objectIds": [
"123456","789101112"
"131415161718",
"23897845","9879879546"
]
},
{
"name": "DNS_Two",
"objectIds": [
"23897845","9879879546"
]
}
]

Read the data and create the variables finallist and input
- set_fact:
finallist: "{{ lookup('file', 'finallist.json')|from_yaml }}"
- set_fact:
input: "{{ lookup('file', 'input.json')|from_yaml }}"
gives
finallist:
- id: '123456'
net_name: DNS_One
- id: '789101112'
net_name: DNS_One
- id: '131415161718'
net_name: DNS_One-HO
- id: '23897845'
net_name: NTP
- id: '9879879546'
net_name: NTP
input:
- name: DNS_One
objectIds:
- DNS_One
- DNS_One-HO
- NTP
- name: DNS_Two
objectIds:
- NTP
In finallist, group the items by net_name and convert the list to a dictionary
- set_fact:
finaldict: "{{ finaldict|d({})|
combine({item.0: item.1|map(attribute='id')}) }}"
loop: "{{ finallist|groupby('net_name') }}"
gives
finaldict:
DNS_One:
- '123456'
- '789101112'
DNS_One-HO:
- '131415161718'
NTP:
- '23897845'
- '9879879546'
Iterate input and substitute the IDs
- set_fact:
new: "{{ new|d([]) + [item|combine({'objectIds': _ids})] }}"
loop: "{{ input }}"
vars:
_ids: "{{ item.objectIds|map('extract', finaldict)|list }}"
gives the expected result
new:
- name: DNS_One
objectIds:
- ['123456', '789101112']
- ['131415161718']
- ['23897845', '9879879546']
- name: DNS_Two
objectIds:
- ['23897845', '9879879546']
You can flatten the lists if you want to
_ids: "{{ item.objectIds|map('extract', finaldict)|flatten }}"
gives
new:
- name: DNS_One
objectIds: ['123456', '789101112', '131415161718', '23897845', '9879879546']
- name: DNS_Two
objectIds: ['23897845', '9879879546']

Related

Create dictionary keys from variable and add list of IPs as values

I would like to create a dictionary and populated from variables both the keys and values. The values will be a list of IPs.
If the key is already present in the dictionary, then, I need to add the IP to the existing list.
Here is the input data, which is a list of list.
parse1.json:
[
[
"/RAD33G4099/mobile_app/pl_crad.la-dc.vivo.com_80",
"properties",
"members",
"/Common/10.50.100.25:80",
"session"
],
[
"/RAD33G4099/mobile_app/pl_crad.la-dc.vivo.com_80",
"properties",
"members",
"/Common/10.50.100.39:80",
"session"
],
[
"/RAD33G4099/mobile_app/pl_crad.la-dc.vivo.com_80",
"properties",
"members",
"/Common/10.50.100.49:80",
"session"
],
[
"/RAD33G4099/mobile_app/pl_crad.la-dc.vivo.com_443",
"properties",
"members",
"/Common/10.50.100.18:80",
"session"
],
[
"/RAD33G4099/mobile_app/pl_crad.la-dc.vivo.com_443",
"properties",
"members",
"/Common/10.50.100.28:80",
"session"
],
[
"/RAD33G4099/mobile_app/pl_crad.la-dc.vivo.com_443",
"properties",
"members",
"/Common/10.50.100.48:80",
"session"
]
]
Here is my trial at solving it:
- set_fact:
my_dic: {}
disabled_host: "{{ lookup('file', 'parse1.json') }}"
- name: add new key / value pairs to dict
set_fact:
my_dict_var: "{{ my_dict_var + [ my_dic | combine ( { name + '_' + app_name : host } , recursive=True) ] }}"
loop: "{{ disabled_host }}"
vars:
name: "{{ item[0] | regex_replace('^\/.*\/(.*)$', '\\1') }}"
app_name: "{{ item[0] | regex_replace('^\/.*\/(.*)\/.*$', '\\1') }}"
host: "{{ item[3] | regex_replace('^\/.*\/(.*):.*$', '\\1') }}"
The actual output is
ok: [localhost] => {
"msg": [
{
"pl_crad.la-dc.vivo.com_80_mobile_app": "10.50.100.25"
},
{
"pl_crad.la-dc.vivo.com_80_mobile_app": "10.50.100.39"
},
{
"pl_crad.la-dc.vivo.com_80_mobile_app": "10.50.100.49"
},
{
"pl_crad.la-dc.vivo.com_443_mobile_app": "10.50.100.18"
},
{
"pl_crad.la-dc.vivo.com_443_mobile_app": "10.50.100.28"
},
{
"pl_crad.la-dc.vivo.com_443_mobile_app": "10.50.100.48"
}
]
}
But, I would like to create a list of IPs if the key is same.
My desired output is
{
"pl_crad.la-dc.vivo.com_80_mobile_app": [
"10.50.100.25",
"10.50.100.39",
"10.50.100.49"
],
"pl_crad.la-dc.vivo.com_443_mobile_app": [
"10.50.100.18",
"10.50.100.28",
"10.50.100.48"
]
}
You don't have to go the hard way and use regex, you can just split those strings, and / or, use basename and dirname.
Using dirname and basename:
- set_fact:
disabled_hosts_ips: >-
{{
disabled_hosts_ips | default({})
| combine({
name: (disabled_hosts_ips | default({}))[name] | default([])
+ [ip]
})
}}
loop: "{{ disabled_hosts }}"
vars:
name: "{{ item[0] | basename ~ '_' ~ item[0] | dirname | basename }}"
ip: "{{ (item[3] | basename).split(':')[0] }}"
Using split:
- set_fact:
disabled_hosts_ips: >-
{{
disabled_hosts_ips | default({})
| combine({
name: (disabled_hosts_ips | default({}))[name] | default([])
+ [ip]
})
}}
loop: "{{ disabled_hosts }}"
vars:
name: "{{ item[0].split('/')[-2:] | reverse | join('_') }}"
ip: "{{ item[3].split('/')[-1].split(':')[0] }}"
Those both give:
disabled_hosts_ips:
pl_crad.la-dc.vivo.com_443_mobile_app:
- 10.50.100.18
- 10.50.100.28
- 10.50.100.48
pl_crad.la-dc.vivo.com_80_mobile_app:
- 10.50.100.25
- 10.50.100.39
- 10.50.100.49
For example, using the variable my_dict_var
my_dict_groups: "{{ dict(my_dict_keys|zip(my_dict_vals)) }}"
my_dict_keys: "{{ dh_groups|map('first')|list }}"
my_dict_vals: "{{ dh_groups|map('last')|map('map', attribute='value')|list }}"
dh_groups: "{{ my_dict_var|map('dict2items')|flatten|groupby('key') }}"
gives
my_dict_groups:
pl_crad.la-dc.vivo.com_443_mobile_app:
- 10.50.100.18
- 10.50.100.28
- 10.50.100.48
pl_crad.la-dc.vivo.com_80_mobile_app:
- 10.50.100.25
- 10.50.100.39
- 10.50.100.49
The complete set of the declarations is below
my_dict_groups: "{{ dict(my_dict_keys|zip(my_dict_vals)) }}"
my_dict_keys: "{{ dh_groups|map('first')|list }}"
my_dict_vals: "{{ dh_groups|map('last')|map('map', attribute='value')|list }}"
dh_groups: "{{ dh_keys|zip(dh_vals)|map('combine')|groupby('key') }}"
dh_vals: "{{ disabled_host|
map(attribute=3)|
map('split', '/')|map('last')|
map('split', ':')|map('first')|
map('community.general.dict_kv', 'value')|
list }}"
dh_keys: "{{ disabled_host|
map(attribute=0)|
map('regex_replace', dh_regex, dh_replace)|
map('community.general.dict_kv', 'key')|
list }}"
dh_regex: '^/.*?/(.*?)/(.*)$'
dh_replace: '\2_\1'
disabled_host: "{{ lookup('file', 'parse1.json') }}"
Details
In both cases use the filter groupby and create the list
dh_groups:
- - pl_crad.la-dc.vivo.com_443_mobile_app
- - key: pl_crad.la-dc.vivo.com_443_mobile_app
value: 10.50.100.18
- key: pl_crad.la-dc.vivo.com_443_mobile_app
value: 10.50.100.28
- key: pl_crad.la-dc.vivo.com_443_mobile_app
value: 10.50.100.48
- - pl_crad.la-dc.vivo.com_80_mobile_app
- - key: pl_crad.la-dc.vivo.com_80_mobile_app
value: 10.50.100.25
- key: pl_crad.la-dc.vivo.com_80_mobile_app
value: 10.50.100.39
- key: pl_crad.la-dc.vivo.com_80_mobile_app
value: 10.50.100.49
Create the lists of the keys my_dict_keys and values my_dict_vals, and create the dictionary my_dict_groups
my_dict_keys:
- pl_crad.la-dc.vivo.com_443_mobile_app
- pl_crad.la-dc.vivo.com_80_mobile_app
my_dict_vals:
- - 10.50.100.18
- 10.50.100.28
- 10.50.100.48
- - 10.50.100.25
- 10.50.100.39
- 10.50.100.49
For example, put the declarations into the playbook vars
- hosts: localhost
vars:
my_dict_groups: "{{ dict(my_dict_keys|zip(my_dict_vals)) }}"
my_dict_keys: "{{ dh_groups|map('first')|list }}"
my_dict_vals: "{{ dh_groups|map('last')|map('map', attribute='value')|list }}"
# dh_groups: "{{ my_dict_var|map('dict2items')|flatten|groupby('key') }}"
dh_groups: "{{ dh_keys|zip(dh_vals)|map('combine')|groupby('key') }}"
dh_vals: "{{ disabled_host|
map(attribute=3)|
map('split', '/')|
map('last')|map('split', ':')|map('first')|
map('community.general.dict_kv', 'value')|
list }}"
dh_keys: "{{ disabled_host|
map(attribute=0)|
map('regex_replace', dh_regex, dh_replace)|
map('community.general.dict_kv', 'key')|
list }}"
dh_regex: '^/.*?/(.*?)/(.*)$'
dh_replace: '\2_\1'
disabled_host: "{{ lookup('file', 'parse1.json') }}"
# my_dict_var:
# - pl_crad.la-dc.vivo.com_80_mobile_app: 10.50.100.25
# - pl_crad.la-dc.vivo.com_80_mobile_app: 10.50.100.39
# - pl_crad.la-dc.vivo.com_80_mobile_app: 10.50.100.49
# - pl_crad.la-dc.vivo.com_443_mobile_app: 10.50.100.18
# - pl_crad.la-dc.vivo.com_443_mobile_app: 10.50.100.28
# - pl_crad.la-dc.vivo.com_443_mobile_app: 10.50.100.48
tasks:
- debug:
var: my_dict_groups

ansible json_query pasing CN name subject

I would like parsing DNS with json_query.
Here is example.
{
"items": [
{
"subject": "CN=DevTest22.aa.com,OU=IT Services,O=Deutsche,L=Berlin,ST=Nordrhein,C=DE",
"version": 3
},
{
"subject": "CN=Devops.aa.com,OU=IT Services,O=Deutsche,L=Berlin,ST=Nordrhein,C=DE",
"version": 3
},
{
"version": 3
},
{
"subject": "CN=Devops22.aa.com,OU=IT Services,O=Deutsche,L=Berlin,ST=Nordrhein,C=DE",
"version": 3
}
]
}
I have create playbook which is work but for more data loop is too slow.
- name: search if Common Name certificate exists in BIG-IQ
set_fact:
CNs_exists: "{{ CNs_exists | d([]) + [ item ] }}"
loop: "{{ json_file | json_query('items') }}"
when: item.subject is defined and item.subject is search(CN_find)
vars:
CN_find: "CN=Devops22.aa.com"
problem is that subject are list and I need first value.
here is what I would like but this does not work
- name: search if Common Name certificate exists in BIG-IQ
set_fact:
CNs_exists: "{{ CNs_exists | d([]) + [ item ] }}"
loop: "{{ list_of_cert['json']['items'] | selectattr('subject') | selectattr('subject', '==', CN_find ) }}"
vars:
CN_find: "CN=Devops22.aa.com"
this code is what I am looking for.
- name: search if Common Name certificate exists in BIG-IQ
set_fact:
CNs_exists: "{{ CNs_exists | d([]) + [ item ] }}"
loop: "{{ json_file['items'] | selectattr('subject', 'defined') | selectattr('subject', 'search', CN_find ) }}"
vars:
CN_find: "CN=Devops22.aa.com"

how to use json_query filter to extract all items equals to a value

Here is my json output:
{
"kind": [
{
"inventory": "",
"inventory_sources": "",
"job_templates": "",
"workflow_job_templates": "104"
},
{
"inventory": "",
"inventory_sources": "",
"job_templates": "114",
"workflow_job_templates": ""
},
{
"inventory": "24",
"inventory_sources": "",
"job_templates": "",
"workflow_job_templates": ""
},
{
"inventory": "",
"inventory_sources": "108",
"job_templates": "",
"workflow_job_templates": ""
}
]
}
I'd like to display all items name that contain a specific value. For example, for a search value of 104 I want to get the key name workflow_job_templates
I tested some syntaxes without any success:
- debug:
msg: "104 is {{kind|json_query(query)}}"
vars:
query: "[?*==`104`].workflow_job_templates"
I know it is wrong but can someone tell me how he'd do for himself?
json_query could be part of the equation for your solution but is really not needed here.
Explanation of the below piece of code:
Apply the dict2items filter to each element of your list. This transforms each mapping to a list of {key: "key", value: "value"} pairs
Flatten the given list so we get all those elements to a single top level
Select elements having a value of '104' only
Extract the key attribute of each element in a list
Make that list unique and sort it.
- name: Display all element having a value of 104
debug:
msg: "{{ kind | map('dict2items') | flatten
| selectattr('value', '==', '104')
| map(attribute='key') | unique | sort }}"
Please note that this solution will give you a result if the same key name has different values but one of them is `104. With your above data the result is:
TASK [Display all element having a value of 104] ***************************************************************************************************************************************************************************************
ok: [localhost] => {
"msg": [
"workflow_job_templates"
]
}
(Update)
The task below
- debug:
msg: "{{ item }} {{ kind|
map('dict2items')|
map('json_query', query)|
flatten }}"
loop: [104, 114, 108, 24]
vars:
query: "[?to_string(value) == to_string('{{ item }}')].key"
gives
msg: 104 ['workflow_job_templates']
msg: 114 ['job_templates']
msg: 108 ['inventory_sources']
msg: 24 ['inventory']
(For the record. Brute-force approach)
Create a unique list of the keys
- set_fact:
my_keys: "{{ my_keys|default([]) + item.keys()|list }}"
loop: "{{ kind }}"
- set_fact:
my_keys: "{{ my_keys|unique }}"
gives
my_keys:
- inventory
- inventory_sources
- job_templates
- workflow_job_templates
Create a dictionary with all values
- set_fact:
my_dict: "{{ my_dict|default({})|combine({item: values}) }}"
loop: "{{ my_keys }}"
vars:
query: "[].{{ item }}"
values: "{{ kind|json_query(query) }}"
gives
my_dict:
inventory:
- ''
- ''
- '24'
- ''
inventory_sources:
- ''
- ''
- ''
- '108'
job_templates:
- ''
- '114'
- ''
- ''
workflow_job_templates:
- '104'
- ''
- ''
- ''
Then search the dictionary. For example
- debug:
msg: "{{ item }} {{ my_dict|dict2items|json_query(query) }}"
loop: [104, 114, 108, 24]
vars:
query: "[?value[?contains(#, '{{ item }}')]].key"
gives
msg: 104 ['workflow_job_templates']
msg: 114 ['job_templates']
msg: 108 ['inventory_sources']
msg: 24 ['inventory']
The correct alternative of selectattr with json_query is:
- debug:
msg: "{{ kind | map('dict2items') | flatten | json_query(query)}}"
vars:
- query: "[?value == `\"104\"`].key"
If you really just want to use json_query()
---
- name: PLAYBOOK Filtering
hosts: localhost # run locally
tasks:
- name: Create json
set_fact:
kind: '{{ lookup("file", "kind.json") }}'
- name: check the var was created properly
debug:
var: kind
- name: output the element that matches 104
debug:
msg: "{{ kind | json_query(\"kind[?workflow_job_templates=='104'].workflow_job_templates\") }}"
- name:
set_fact:
output: "{{ kind | json_query(\"kind[?workflow_job_templates=='104'].workflow_job_templates\") }}"
Output
TASK [output the element that matches 104] *************************************************************************************************************
ok: [localhost] => {
"msg": [
"104"
]
}

json_query - Get object key of current node

I am trying to access an object key of the current node
pkgs:
pkg1:
package_description: xzy
owner: node1
pkg2:
package_description: desc2
owner: node2
What I need is an object that looks like this
- name: pkgs1
package_description: xyz
owner: node1
- name: pkg
package_description: desc2
owner: node2
This is the query I created but am not sure how to access the "key"
- debug:
msg: "{{ clusters.json | json_query(qry) | flatten }}"
verbosity: 2
vars:
qry: '[*].pkgs.*.{"package_description": package_description, "ower": owner, "name": ??}'
The query below does the job
- set_fact:
pkgl: "{{ pkgs|
dict2items|
json_query('[].{name: key,
package_description: value.package_description,
owner: value.owner}') }}"
- debug:
var: pkgl
gives
"pkgl": [
{
"name": "pkg2",
"owner": "node2",
"package_description": "desc2"
},
{
"name": "pkg1",
"owner": "node1",
"package_description": "xzy"
}
]
The same result gives the task without json_query
- set_fact:
pkgl: "{{ pkgl|default([]) +
[{'name': item.key,
'package_description': item.value.package_description,
'owner': item.value.owner}] }}"
loop: "{{ pkgs|dict2items }}"

Ansible: Remove item from dict

I have a situation where i am trying to remove the item from a list. but i am not getting expected result. Please help me what I am doing wrong here?
here is the list :
"get_ec2_id.instances[0].tags": {
"Name": "test-db-system-2",
"aws:cloudformation:logical-id": "DBInstance",
"aws:cloudformation:stack-id": "arn:aws:cloudformation:us-east-1:123456789012:stack/test-db-system-2/0115v0a0-5d44-17e8-a024-503ama4a5qd1",
"aws:cloudformation:stack-name": "test-db-system-2",
"dbsystem:stack": "test-db-system-2",
"dbsystem:type": "db"
}
}
I am trying to remove the all "aws:cloudformation" tags from a list using below filter:
"{{ get_ec2_id.instances[0].tags | reject('search','aws:') | list }}"
I am getting the below result:
ok: [10.52.8.101] => {
"instances_tags": [
"dbsystem:type",
"dbsystem:stack",
"Name"
]
}
but I am expected below result :
"instances_tags": [
"dbsystem:stack": "test-db-system-2",
"dbsystem:type": "db"
"Name" : "test-db-system-2",
]
}
Help me to solve the issue.
More generic solution where input is a dict and blacklist is a list:
---
- set_fact:
blacklist:
- bad1
- bad2
- set_fact:
output: {}
- name: remove blacklisted items from input
set_fact:
output: "{{ output | combine({item.key: item.value}) }}"
when: item.key not in blacklist
loop: "{{ input | dict2items }}"
Given the data
get_ec2_id:
instances:
- tags:
Name: test-db-system-2
aws:cloudformation:logical-id: DBInstance
aws:cloudformation:stack-id: arn:aws:cloudformation:us-east-1:123456789012:stack/test-db-system-2/0115v0a0-5d44-17e8-a024-503ama4a5qd1
aws:cloudformation:stack-name: test-db-system-2
dbsystem:stack: test-db-system-2
dbsystem:type: db
Use rejectattr. For example
dict2: "{{ get_ec2_id.instances.0.tags|
dict2items|
rejectattr('key', 'search', 'aws:')|
items2dict }}"
gives
dict2:
Name: test-db-system-2
dbsystem:stack: test-db-system-2
dbsystem:type: db
Then, convert the dictionary into a list of dictionaries
instances_tags: "{{ dict2|
dict2items|
json_query('[].[[key, value]]')|
map('community.general.dict')|
list }}"
gives
instances_tags:
- Name: test-db-system-2
- dbsystem:stack: test-db-system-2
- dbsystem:type: db
Use this:
---
- name: dictionary
hosts: localhost
gather_facts: False
connection: local
vars:
get_ec2_id:
instances:
tags:
Name: "test-db-system-2"
"aws:cloudformation:logical-id": "DBInstance"
"aws:cloudformation:stack-id": "arn:aws:cloudformation:us-east-1:123456789012:stack/test-db-system-2/0115v0a0-5d44-17e8-a024-503ama4a5qd1"
"aws:cloudformation:stack-name": "test-db-system-2"
"dbsystem:stack": "test-db-system-2"
"dbsystem:type": "db"
dict2: {}
tasks:
- name: Fact1
set_fact:
dict: "{{ get_ec2_id.instances.tags }}"
- name: Debug1
debug:
var: dict
- name: Fact2
set_fact:
dict2: "{{ dict2 | combine({item.key: item.value}) }}"
when: "{{ item.key.find('aws:') }}"
with_dict: "{{ dict }}"
- name: Debug2
debug:
var: dict2
Output:
TASK [Debug2] ******************************************************************
ok: [localhost] => {
"dict2": {
"Name": "test-db-system-2",
"dbsystem:stack": "test-db-system-2",
"dbsystem:type": "db"
}
}

Resources