I have a list of dictionaries
member:
- name: test2
orig: test2
- name: test1
orig: test1
and would like to remove the orig key and value from all dictionaries in the list.
- name: Print Firewall Group Member
debug:
msg: "{{ item }}"
loop: "{{ member }}"
TASK [Print Firewall Group Member] *****************************************************************************************
ok: [fortigate01] => (item={'name': 'test2', 'orig': 'test2'}) => {
"msg": {
"name": "test2",
"orig": "test2"
}
}
ok: [fortigate01] => (item={'name': 'test1', 'orig': 'test1'}) => {
"msg": {
"name": "test1",
"orig": "test1"
}
}
Use the filter ansible.utils.remove_keys. For example, given simplified data
member:
- name: test2
orig: test2
- name: test1
orig: test1
Declare the variable
names: "{{ member|ansible.utils.remove_keys(target=['orig']) }}"
gives
names:
- name: test2
- name: test1
Optionally, use the filter ansible.utils.keep_keys. The declaration below gives the same result
names: "{{ member|ansible.utils.keep_keys(target=['name']) }}"
The next option is json_query
names: "{{ member|json_query('[].{name: name}') }}"
Example of a complete playbook for testing
- hosts: localhost
vars:
member:
- name: test2
orig: test2
- name: test1
orig: test1
names: "{{ member|ansible.utils.remove_keys(target=['orig']) }}"
name2: "{{ member|ansible.utils.keep_keys(target=['name']) }}"
name3: "{{ member|json_query('[].{name: name}') }}"
tasks:
- debug:
var: names
- debug:
var: name2
- debug:
var: name3
Related
Is there find a key deep inside a nested dictionaries and show the path to that key?
Example:
---
isbn: 123-456-222
author:
lastname: Doe
firstname:
name: John
initial: D
I want to find the initial variable and print the path to it like author.fistname.initial
I would do something like this using a custom filter. E.g.:
def _findkey_helper(data, want, path=None):
if path is None:
path = []
if not data:
return False, None, None
if want in data:
return True, path + [want], data[want]
for k, v in data.items():
if isinstance(v, dict):
found, val, foundat = _findkey_helper(v, want, path=path + [k])
if found:
return (found, foundat, val)
return False, None, None
def filter_findkey(data, want):
"""Search a nested dictionary for a given key.
This filter recursively searches the nested dictionary structure `data`
for the key `want`. It returns the 3-tuple (found, path, value), where:
- `found` is True if the named key was found, False otherwise
- `path` is a dotted path to the named key, or `None` if `found` is False
- `value` is the corresponding value, or `None` if `found` is False
"""
found, path, val = _findkey_helper(data, want)
if found:
return True, ".".join(path), val
return False, None, None
class FilterModule:
def filters(self):
return {"findkey": filter_findkey}
Drop this into filter_plugins/findkey.py adjacent to your playbook, and use it like this:
- hosts: localhost
gather_facts: false
vars:
data:
isbn: 123-456-222
author:
lastname: Doe
firstname:
name: John
initial: D
tasks:
- debug:
msg: "{{ data | findkey('initial') }}"
The above outputs:
TASK [debug] ********************************************************************************************
ok: [localhost] => {
"msg": "(True, 'author.firstname.initial', 'D')"
}
This only works with nested dictionary structures (that is, it won't search through lists), but necessary changes to support lists would be pretty simple.
The following playbook does the job using the to_paths lookup
---
- hosts: localhost
gather_facts: false
vars:
isbn: 123-456-222
author:
lastname: Doe
firstname:
name: John
initial: D
needle: initial
tasks:
# Note: this is a necessary step to "fix" the value from the lookup.
# If you set this value in play/task/host/group variable,
# you will get a recursive loop error any time you use the resulting variable
- name: extract the info
set_fact:
matches: "{{ lookup('ansible.utils.to_paths', vars) | dict2items | selectattr('key', 'search', needle) }}"
- name: Overall info
debug:
msg: "I found {{ matches | length }} entries having '{{ needle }}' in their path"
- name: details
debug:
msg: "item at path {{ item.key }} has value: {{ item.value }}"
loop: "{{ matches }}"
Example run:
$ ansible-playbook playbook.yml
PLAY [localhost] ***********************************************************************************************************************************************************************************************************************
TASK [extract the info] ****************************************************************************************************************************************************************************************************************
ok: [localhost]
TASK [Overall info] ********************************************************************************************************************************************************************************************************************
ok: [localhost] => {
"msg": "I found 1 entries having 'initial' in their path"
}
TASK [details] *************************************************************************************************************************************************************************************************************************
ok: [localhost] => (item={'key': 'author.firstname.initial', 'value': 'D'}) => {
"msg": "item at path author.firstname.initial has value: D"
}
PLAY RECAP *****************************************************************************************************************************************************************************************************************************
localhost : ok=3 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
And to be ahead of some possible comments: this works with lists too. Replacing the var author above with:
authors:
- lastname: Doe
firstname:
name: John
initial: D
- lastname: Perez
firstname:
name: Juan
initial: J
gives with the exact same playbook:
TASK [Overall info] ********************************************************************************************************************************************************************************************************************
ok: [localhost] => {
"msg": "I found 2 entries having 'initial' in their path"
}
TASK [details] *************************************************************************************************************************************************************************************************************************
ok: [localhost] => (item={'key': 'authors[0].firstname.initial', 'value': 'D'}) => {
"msg": "item at path authors[0].firstname.initial has value: D"
}
ok: [localhost] => (item={'key': 'authors[1].firstname.initial', 'value': 'J'}) => {
"msg": "item at path authors[1].firstname.initial has value: J"
}
Q: "Find the initial variable and print the path to it like author.fistname.initial"
A: Use the dictionaries below. There might be more paths. For example, given the data
data:
- isbn: 123-456-222
author:
lastname: Doe
firstname:
name: John
initial: D
- isbn: 123-456-333
author:
lastname: Doe
firstname:
name: Alice
initial: C
- isbn: 123-456-444
author:
lastname: Doe
firstname:
name: Bob
initial: D
The task creates the dictionary initials
- name: Create dictionary initials
set_fact:
initials: "{{ initials|d({})|combine({item: authors}) }}"
loop: "{{ data|json_query('[].*.*.initial')|flatten|unique }}"
vars:
authors: "{{ data|
selectattr('author.firstname.initial', 'eq', item)|
json_query('[].join(`,`,[author.lastname,
author.firstname.name,
author.firstname.initial])') }}"
gives
initials:
C:
- Doe,Alice,C
D:
- Doe,John,D
- Doe,Bob,D
The task creates the dictionary paths
- name: Create dictionary paths
set_fact:
paths: "{{ paths|d({})|combine(_item, list_merge='append') }}"
loop: "{{ data }}"
vars:
_paths: "{{ lookup('ansible.utils.to_paths', item) }}"
_item: "{{ dict(_paths|dict2items|json_query('[].[value, [key]]')) }}"
gives
paths:
123-456-222:
- isbn
123-456-333:
- isbn
123-456-444:
- isbn
Alice:
- author.firstname.name
Bob:
- author.firstname.name
C:
- author.firstname.initial
D:
- author.firstname.initial
- author.firstname.initial
Doe:
- author.lastname
- author.lastname
- author.lastname
John:
- author.firstname.name
Filter unique paths. Create dictionary paths_unique
paths_keys: "{{ paths.keys()|list }}"
paths_vals: "{{ paths.values()|map('unique')|list }}"
paths_unique: "{{ dict(paths_keys|zip(paths_vals)) }}"
gives
paths_unique:
123-456-222:
- isbn
123-456-333:
- isbn
123-456-444:
- isbn
Alice:
- author.firstname.name
Bob:
- author.firstname.name
C:
- author.firstname.initial
D:
- author.firstname.initial
Doe:
- author.lastname
John:
- author.firstname.name
Example of a complete playbook for testing
- hosts: localhost
vars:
data:
- isbn: 123-456-222
author:
lastname: Doe
firstname:
name: John
initial: D
- isbn: 123-456-333
author:
lastname: Doe
firstname:
name: Alice
initial: C
- isbn: 123-456-444
author:
lastname: Doe
firstname:
name: Bob
initial: D
paths_keys: "{{ paths.keys()|list }}"
paths_vals: "{{ paths.values()|map('unique')|list }}"
paths_unique: "{{ dict(paths_keys|zip(paths_vals)) }}"
tasks:
- name: Create dictionary initials
set_fact:
initials: "{{ initials|d({})|combine({item: authors}) }}"
loop: "{{ data|json_query('[].*.*.initial')|flatten|unique }}"
vars:
authors: "{{ data|
selectattr('author.firstname.initial', 'eq', item)|
json_query('[].join(`,`,[author.lastname,
author.firstname.name,
author.firstname.initial])') }}"
- debug:
var: initials
- name: Create dictionary paths
set_fact:
paths: "{{ paths|d({})|combine(_item, list_merge='append') }}"
loop: "{{ data }}"
vars:
_paths: "{{ lookup('ansible.utils.to_paths', item) }}"
_item: "{{ dict(_paths|dict2items|json_query('[].[value, [key]]')) }}"
- debug:
var: paths
- debug:
var: paths_unique
Here is my playbook:
- hosts: localhost
vars:
{
"result": [
{
"_ref": "vlan/ZG5zLnZsYW4kLmNvbS5pbmZvYmxveC5kbnMudmxhbl92aWV3JElORlJBTEFCLjEuNDA5NC4xMQ:LAB1/test1/11",
"id": 11,
"name": "test1",
"parent": {
"_ref": "vlanview/ZG5zLnZsYW5fdmlldyRJTkZSQUxBQi4xLjQwOTQ:LAB1/1/4094"
}
},
{
"_ref": "vlan/ZG5zLnZsYW4kLmNvbS5pbmZvYmxveC5kbnMudmxhbl92aWV3JFNDTEFCLU9PQi4xLjQwOTQuMTE:LAB2/test1/11",
"id": 11,
"name": "test1,
"parent": {
"_ref": "vlanview/ZG5zLnZsYW5fdmlldyRTQ0xBQi1PT0IuMS40MDk0:LAB2/1/4094"
}
}
]
}
tasks:
- set_fact:
var1: "{{result|json_query(jquery)}}"
vars:
jquery: "[].{vlan_view: _ref|regex_search('(?<=:)[^/]*'), vlan_id: id, vlan_name: name}"
- debug: msg={{var1}}
Which errors with:
fatal: [localhost]: FAILED! => {"msg": "JMESPathError in json_query filter plugin:\nUnknown function: regex_search()"}
My desired output
[
{
"vlan_view": LAB1,
"vlan_id": 11,
"vlan_name": "test1"
},
{
"vlan_id": 11,
"vlan_name": "test1",
"vlan_view": "LAB2"
}
]
You cannot do regex operation in JMESPath, as per this issue on their tracker.
And you surely cannot use a Jinja filter as a JMESPath function, as the error is pointing out.
So, you will have to achieve this with Jinja filters and Ansible alone.
And with a loop, it is definitely possible to create a list corresponding to your desired output:
- set_fact:
var1: "{{ var1 | default([]) + [_vlan] }}"
loop: "{{ result }}"
loop_control:
label: "{{ item.id }}"
vars:
_vlan:
vlan_id: "{{ item.name }}"
vlan_name: "{{ item.id }}"
vlan_view: >-
{{
item.parent._ref
| regex_search(':(.*?)\/', '\1')
| first
}}
Given the two tasks:
- set_fact:
var1: "{{ var1 | default([]) + [_vlan] }}"
loop: "{{ result }}"
loop_control:
label: "{{ item.id }}"
vars:
_vlan:
vlan_id: "{{ item.name }}"
vlan_name: "{{ item.id }}"
vlan_view: >-
{{
item.parent._ref
| regex_search(':(.*?)\/', '\1')
| first
}}
- debug:
var: var1
This will yield:
TASK [set_fact] ***************************************************************
ok: [localhost] => (item=11)
ok: [localhost] => (item=11)
TASK [debug] ******************************************************************
ok: [localhost] =>
var1:
- vlan_id: test1
vlan_name: '11'
vlan_view: LAB1
- vlan_id: test1
vlan_name: '11'
vlan_view: LAB2
Get the attributes vlan_view
vlan_view: "{{ result|map(attribute='_ref')|
map('split', ':')|map('last')|
map('split', '/')|map('first')|
map('community.general.dict_kv', 'vlan_view')|
list }}"
gives
vlan_view:
- vlan_view: LAB1
- vlan_view: LAB2
Then use json_query to get the other attributes and combine the dictionaries
var1: "{{ result|json_query('[].{vlan_id: id, vlan_name: name}')|
zip(vlan_view)|map('combine')|list }}"
gives the expected result
var1:
- vlan_id: 11
vlan_name: test1
vlan_view: LAB1
- vlan_id: 11
vlan_name: test1
vlan_view: LAB2
Example of a complete playbook (simplified for testing)
- hosts: localhost
vars:
result:
- _ref: vlan/ZG5z...4xMQ:LAB1/test1/11
id: 11
name: test1
parent:
_ref: vlanview/ZG5zL...wOTQ:LAB1/1/4094
- _ref: vlan/ZG5zL...uMTE:LAB2/test1/11
id: 11
name: test1
parent:
_ref: vlanview/ZG5zL...MDk0:LAB2/1/4094
vlan_view: "{{ result|map(attribute='_ref')|
map('split', ':')|map('last')|
map('split', '/')|map('first')|
map('community.general.dict_kv', 'vlan_view')|
list }}"
var1: "{{ result|json_query('[].{vlan_id: id, vlan_name: name}')|
zip(vlan_view)|map('combine')|list }}"
tasks:
- debug:
var: vlan_view
- debug:
var: var1
I have a list of lists of hosts:
[['host-0', 'host-1'], ['host-2', 'host-3'], ['host-4', 'host-5', 'host-6']]
How can I add a port number, e.g., 8000, to each host using ansible/ jinja2 to get:
[['host-0:8000', 'host-1:8000'], ['host-2:8000', 'host-3:8000'], ['host-4:8000', 'host-5:8000', 'host-6:8000']]
this task shall do it:
- name: convert items in list
set_fact:
my_new_list: "{{ my_new_list | default([])+ [ my_var ] }}"
vars:
my_var: "{{ item | map('regex_replace', '$', ':8000') | list }}"
with_items:
- "{{ my_list }}"
full playbook to run as demo:
---
- hosts: localhost
gather_facts: false
vars:
my_list:
- ['host-0', 'host-1']
- ['host-2', 'host-3']
- ['host-4', 'host-5', 'host-6']
tasks:
- name: print original variable
debug:
var: my_list
- name: convert items in list
set_fact:
my_new_list: "{{ my_new_list | default([])+ [ my_var ] }}"
vars:
my_var: "{{ item | map('regex_replace', '$', ':8000') | list }}"
with_items:
- "{{ my_list }}"
- name: print new variable
debug:
var: my_new_list
result:
TASK [print new variable] **********************************************************************************************************************************************************************************************
ok: [localhost] => {
"my_new_list": [
[
"host-0:8000",
"host-1:8000"
],
[
"host-2:8000",
"host-3:8000"
],
[
"host-4:8000",
"host-5:8000",
"host-6:8000"
]
]
}
PLAY RECAP
Use map + regex_replace.
- debug:
msg: "{{ foo | map('map', 'regex_replace', '$', ':8000') }}"
vars:
foo: [['host-0', 'host-1'], ['host-2', 'host-3'], ['host-4', 'host-5', 'host-6']]
"msg": [
[
"host-0:8000",
"host-1:8000"
],
[
"host-2:8000",
"host-3:8000"
],
[
"host-4:8000",
"host-5:8000",
"host-6:8000"
]
]
I have got multiple yaml files on remote machine. I would like to parse those files in order to get information about names for each kind (Deployment, Configmap, Secret) of object, For example:
...
kind: Deployment
metadata:
name: pr1-dep
...
kind: Secret
metadata:
name: pr1
...
....
kind: ConfigMap
metadata:
name: cm-pr1
....
Ecpected result:
3 variables:
deployments = [pr1-dep]
secrets = [pr1]
configmaps = [cm-pr1]
I started with:
- shell: cat "{{ item.desc }}"
with_items:
- "{{ templating_register.results }}"
register: objs
but i have no idea how to correctly parse item.stdout from objs
Ansible has a from_yaml filter that takes YAML text as input and outputs an Ansible data structure. So for example you can write something like this:
- hosts: localhost
gather_facts: false
tasks:
- name: Read objects
command: "cat {{ item }}"
register: objs
loop:
- deployment.yaml
- configmap.yaml
- secret.yaml
- debug:
msg:
- "kind: {{ obj.kind }}"
- "name: {{ obj.metadata.name }}"
vars:
obj: "{{ item.stdout | from_yaml }}"
loop: "{{ objs.results }}"
loop_control:
label: "{{ item.item }}"
Given your example files, this playbook would output:
PLAY [localhost] ***************************************************************
TASK [Read objects] ************************************************************
changed: [localhost] => (item=deployment.yaml)
changed: [localhost] => (item=configmap.yaml)
changed: [localhost] => (item=secret.yaml)
TASK [debug] *******************************************************************
ok: [localhost] => (item=deployment.yaml) => {
"msg": [
"kind: Deployment",
"name: pr1-dep"
]
}
ok: [localhost] => (item=configmap.yaml) => {
"msg": [
"kind: ConfigMap",
"name: pr1-cm"
]
}
ok: [localhost] => (item=secret.yaml) => {
"msg": [
"kind: Secret",
"name: pr1"
]
}
PLAY RECAP *********************************************************************
localhost : ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
Creating the variables you've asked for is a little trickier. Here's
one option:
- hosts: localhost
gather_facts: false
tasks:
- name: Read objects
command: "cat {{ item }}"
register: objs
loop:
- deployment.yaml
- configmap.yaml
- secret.yaml
- name: Create variables
set_fact:
names: >-
{{
names|combine({
obj.kind.lower(): [obj.metadata.name]
}, list_merge='append')
}}
vars:
names: {}
obj: "{{ item.stdout | from_yaml }}"
loop: "{{ objs.results }}"
loop_control:
label: "{{ item.item }}"
- debug:
var: names
This creates a single variable named names that at the end of the
playbook will contain:
{
"configmap": [
"pr1-cm"
],
"deployment": [
"pr1-dep"
],
"secret": [
"pr1"
]
}
The key to the above playbook is our use of the combine filter, which can be used to merge dictionaries and, when we add list_merge='append', handles keys that resolve to lists by appending to the existing list, rather than replacing the existing key.
Include the dictionaries from the files into the new variables, e.g.
- include_vars:
file: "{{ item }}"
name: "objs_{{ item|splitext|first }}"
register: result
loop:
- deployment.yaml
- configmap.yaml
- secret.yaml
This will create dictionaries objs_deployment, objs_configmap, and objs_secret. Next, you can either use the dictionaries
- set_fact:
objs: "{{ objs|d({})|combine({_key: _val}) }}"
loop: "{{ query('varnames', 'objs_') }}"
vars:
_obj: "{{ lookup('vars', item) }}"
_key: "{{ _obj.kind }}"
_val: "{{ _obj.metadata.name }}"
, or the registered data
- set_fact:
objs: "{{ dict(_keys|zip(_vals)) }}"
vars:
_query1: '[].ansible_facts.*.kind'
_keys: "{{ result.results|json_query(_query1)|flatten }}"
_query2: '[].ansible_facts.*.metadata[].name'
_vals: "{{ result.results|json_query(_query2)|flatten }}"
Both options give
objs:
ConfigMap: cm-pr1
Deployment: pr1-dep
Secret: pr1
I have a situation where i am trying to remove the item from a list. but i am not getting expected result. Please help me what I am doing wrong here?
here is the list :
"get_ec2_id.instances[0].tags": {
"Name": "test-db-system-2",
"aws:cloudformation:logical-id": "DBInstance",
"aws:cloudformation:stack-id": "arn:aws:cloudformation:us-east-1:123456789012:stack/test-db-system-2/0115v0a0-5d44-17e8-a024-503ama4a5qd1",
"aws:cloudformation:stack-name": "test-db-system-2",
"dbsystem:stack": "test-db-system-2",
"dbsystem:type": "db"
}
}
I am trying to remove the all "aws:cloudformation" tags from a list using below filter:
"{{ get_ec2_id.instances[0].tags | reject('search','aws:') | list }}"
I am getting the below result:
ok: [10.52.8.101] => {
"instances_tags": [
"dbsystem:type",
"dbsystem:stack",
"Name"
]
}
but I am expected below result :
"instances_tags": [
"dbsystem:stack": "test-db-system-2",
"dbsystem:type": "db"
"Name" : "test-db-system-2",
]
}
Help me to solve the issue.
More generic solution where input is a dict and blacklist is a list:
---
- set_fact:
blacklist:
- bad1
- bad2
- set_fact:
output: {}
- name: remove blacklisted items from input
set_fact:
output: "{{ output | combine({item.key: item.value}) }}"
when: item.key not in blacklist
loop: "{{ input | dict2items }}"
Given the data
get_ec2_id:
instances:
- tags:
Name: test-db-system-2
aws:cloudformation:logical-id: DBInstance
aws:cloudformation:stack-id: arn:aws:cloudformation:us-east-1:123456789012:stack/test-db-system-2/0115v0a0-5d44-17e8-a024-503ama4a5qd1
aws:cloudformation:stack-name: test-db-system-2
dbsystem:stack: test-db-system-2
dbsystem:type: db
Use rejectattr. For example
dict2: "{{ get_ec2_id.instances.0.tags|
dict2items|
rejectattr('key', 'search', 'aws:')|
items2dict }}"
gives
dict2:
Name: test-db-system-2
dbsystem:stack: test-db-system-2
dbsystem:type: db
Then, convert the dictionary into a list of dictionaries
instances_tags: "{{ dict2|
dict2items|
json_query('[].[[key, value]]')|
map('community.general.dict')|
list }}"
gives
instances_tags:
- Name: test-db-system-2
- dbsystem:stack: test-db-system-2
- dbsystem:type: db
Use this:
---
- name: dictionary
hosts: localhost
gather_facts: False
connection: local
vars:
get_ec2_id:
instances:
tags:
Name: "test-db-system-2"
"aws:cloudformation:logical-id": "DBInstance"
"aws:cloudformation:stack-id": "arn:aws:cloudformation:us-east-1:123456789012:stack/test-db-system-2/0115v0a0-5d44-17e8-a024-503ama4a5qd1"
"aws:cloudformation:stack-name": "test-db-system-2"
"dbsystem:stack": "test-db-system-2"
"dbsystem:type": "db"
dict2: {}
tasks:
- name: Fact1
set_fact:
dict: "{{ get_ec2_id.instances.tags }}"
- name: Debug1
debug:
var: dict
- name: Fact2
set_fact:
dict2: "{{ dict2 | combine({item.key: item.value}) }}"
when: "{{ item.key.find('aws:') }}"
with_dict: "{{ dict }}"
- name: Debug2
debug:
var: dict2
Output:
TASK [Debug2] ******************************************************************
ok: [localhost] => {
"dict2": {
"Name": "test-db-system-2",
"dbsystem:stack": "test-db-system-2",
"dbsystem:type": "db"
}
}