Ansible pick dictionary out of a list of dictionaries - ansible

Variable mule_runtimes has a list of dictionaries:
- id: N-Newton
version: 4.3.0
- id: N-Galileo
version: 3.9.0-hf4
- id: N-Einstein
version: 3.8.5-hf4
I want the dictionary with id = N-Einstein.
I have tried using this:
- debug:
msg: "{{ mule_runtimes | selectattr('id', 'equalto', 'N-Einstein') | to_json }}"
And got error: Unexpected templating type error occurred on ({{ mule_runtimes | selectattr('id', 'equalto', 'N-Einstein') | to_json }}): Object of type 'generator' is not JSON serializable.
What's the correct way to pick a dictionary from a list?

The first problem is that mule_runtimes | selectattr('id', 'equalto', 'N-Einstein') returns a generator. Think of it like d for d in mule_runtimes if d['id'] == 'N-Einstein' in Python. You'll need to convert it to something JSON serializable (like a list) before using the to_json filter.
The second problem is that it doesn't select only a single dictionary from the list. The predicate id == 'N-Einstein' could be true for multiple dictionaries. If you know it will only match one dictionary, you'll need to convert the list to a single dictionary.
Putting that all together:
{{ mule_runtimes | selectattr('id', 'equalto', 'N-Einstein') | list | last | to_json }}

I suggest json_query:
- name: dictionaries
vars:
mule_runtimes:
- id: N-Newton
version: 4.3.0
- id: N-Galileo
version: 3.9.0-hf4
- id: N-Einstein
version: 3.8.5-hf4
json: "{{ mule_runtimes }}"
query: "[?id=='{{ want }}'].version"
want: N-Einstein
debug:
msg: "{{ json | json_query(query) }}"
Gives the output:
TASK [test : dictionaries] *****************************************************
ok: [127.0.0.1] => {
"msg": [
"3.8.5-hf4"
]
}

Related

Filter webhook payload for modified repositories

I'm building a pipeline with Gitlab and AWX and need to filter the payload of the webhook. What I need is basically parsing only project1 and project2 (which is between two /) into a variable which I can use in other roles.
tower_webhook_payload:
after:
before:
checkout_sha:
commits:
- id:
message:
title:
timestamp:
url: >-
author:
name:
email:
added: []
modified:
- repository/project1/file1
removed: []
- id:
message:
title:
timestamp:
url: >-
author:
name:
email:
added:
modified:
- repository/project2/file2
removed: []
This was my rough idea but I don't have much experience working with lists or regex.
- debug: msg="{{ tower_webhook_payload.commits | select('match', 'modified') | list }}"
"msg": "Unexpected templating type error occurred on ({{ tower_webhook_payload.commits | select('match', 'modified') | list }}): expected string or bytes-like object"}
One way to achieve this:
extract the modified list attribute of each commits.
flatten the result so you get a single list of all gathered elements.
map the regex_replace filter to each element to extract only the project name. The below regex capture everything after the initial repository/ that does not contain a / and replaces the entire match only with that capture
apply the unique filter so you eliminate duplicates if any
---
- debug:
msg: "{{ tower_webhook_payload.commits | map(attribute='modified') | flatten
| map('regex_replace', 'repository/([^/]*)/.*', '\\1') | unique }}"
Edit: with older versions of ansible, you might have to explicitely cast to list after maping
---
- debug:
msg: "{{ tower_webhook_payload.commits | map(attribute='modified')
| list | flatten
| map('regex_replace', 'repository/([^/]*)/.*', '\\1')
| list | unique }}"
how about using json_query like:
- debug: msg="{{ tower_webhook_payload | json_query(\"commits[].modified[0]\") }}"

Get the newest dictionary from a list in ansible

I have a list of dictionaries and I want to get the latest one.
reviewing jinja2 docs it seems I should be able to do this:
- set_fact:
t:
- a: 1
b: 2
- a: 3
b: 1
- debug:
msg: "{{ t | max(attribute='a') }}"
But that fails with
fatal: [localhost]: FAILED! => {
"msg": "Unexpected templating type error occurred on ({{ t | max(attribute='a') }}): max() > got an unexpected keyword argument 'attribute'"
}
what is the best whay to do this? Of course my use case is harder than that small demo.
My think looks something like this:
check_mode: no
set_fact:
tgs_info: "{{ tgs_info | default({}) | combine({ item: all_tg_info | to_json | from_json | json_query(query) | max(attribute='target_group_name') }) }}"
vars:
query: "target_groups[?contains(target_group_name, `{{ product }}`) == `true`] | [?ends_with(target_group_name, `{{ tg_suffix }}{{ item }}`) == `true`]"
loop: "{{ projects | selectattr('protocol', 'match', '^HTTP$') | map(attribute='port') | list }}"
The idea is that all_tg_info contains all the autoscaling of my aws account. I filter them and I want to get the latest one based on the name or any other parameter.
I'm kind of stuk here.
Update: As reported in #Hellseher comment below, from ansible 2.11 release, max|min(attribute='someattr') will be available. The answer below will therefore become obsolete for this version and onward. See the corresponding pull request.
reviewing jinja2 docs it seems I should be able to do this...
Even though the jinja2 builtin filters documentation mentions possible params for the max filter, there is an open bug report on ansible side to support those.
In this case you can easily acheive your requirement with the json_query filter. Here is a demo playbook with your simplistic data (as I don't have your more elaborate one...). You can probably adapt this to your actual json_query.
---
- hosts: localhost
gather_facts: false
vars:
t:
- a: 1
b: 2
- a: 3
b: 1
tasks:
- debug:
msg: "{{ t | json_query('max_by(#, &a)') }}"

Ansible: how to filter using varible

i have ansible-playbook that gives list of lines in debug output.
I am able to filter debug OUTPUT using a string (exp: CUST) but I am struggling to filter the list using a variable.
- debug:
msg: "{{ List.msg | select('match', '^(CUST)[0-9]+') | list }}"
List msg output:
CUST1
CUST2
NEW1
NEW2
from the above debug command, i get CUST1, CUST2 in filtered output.
- set_fact:
filter: "{{ fileout.results[0].content }}"
above filter generates "CUST" and i want to use this filter variable in above debug command.
using below syntax i get nothing, may be ansible is NOT taking it as appropriate variable.
- debug:
msg: "{{ List.msg | select('match', '^("{{ filter }}")[0-9]+') | list }}"
Please help.
thanks in advance.
It's possible to isolate the declaration of the regex filter and simplify the quotation. For example
vars:
List:
msg: ['CUST1','CUST2','NEW1','NEW2']
Patterns: ['CUST','NEW']
tasks:
- debug:
msg: "{{ List.msg | select('match', filter) | list }}"
vars:
filter: '^{{ item }}[0-9]+'
loop: "{{ Patterns }}"
gives
"msg": [
"CUST1",
"CUST2"
]
"msg": [
"NEW1",
"NEW2"
]

In Ansible, how to filter a dictionary based on regex pattern on the keys?

I need to extract a subset of a dictionary based on a pattern on the key names.
For example, in the v below I need to extract the key->values section1*.
The code below assigns the list of values, but I still haven't found a way to preserve the key->map setting.
- set_fact:
v:
section1_1: true
section1_2: false
section2_1: true
section2_2: false
section3: true
- set_fact:
v2: "{{ v | select('match','^section1_.*') | map('extract', v) | list }}"
- debug:
var: v2
Any help, please?
Thanks.
Combine dict2items and items2dict filters:
- debug:
msg: "{{ v | dict2items | selectattr('key', 'match', '^section1') | list | items2dict }}"
fatal: [localhost]: FAILED! => {"msg": "template error while templating string: no filter named 'items2dict'. String: {{ v | dict2items | selectattr('key', 'match', '^section1_') | list | items2dict }}"}
If I look at the sources in /usr/lib/python2.7/dist-packages I see that there are references to it, but not real function definitions
ansible/plugins/filter/core.py: raise AnsibleFilterError("items2dict requires a list, got %s instead." % type(mylist))
ansible/plugins/filter/core.py: 'items2dict': list_of_dict_key_value_elements_to_dict,
I'm runing 2.5.1. Do I need a later version?

When to use from_json filter in Ansible?

When should I use the from_json filter in Ansible?
I found out that using it sometimes has and sometimes have no effect.
Please consider the following example which illustrates the inconsistency I am getting.
Included in reverse order are: the questions - expected result - actual result - the playbook - the data. The data is taken from this question and the playbook is based on this answer.
The question(s):
Why storing the left part (before json_query) of the following expression in a variable and then using json_query on the variable causes the expression to be evaluated differently?
"{{ lookup('file','test.json') | json_query(query) }}"
Why does adding from_json filter alter the results (but does not if processing a variable):
"{{ lookup('file','test.json') | from_json | json_query(query) }}"
Expected result:
Last four tasks should give the same result. Alternatively, last two tasks should give the same result as previous two tasks.
Actual result (last four tasks only):
One task result differs.
TASK [This query is run against lookup value with from_json stored in a variable] ***
ok: [localhost] => {
"msg": [
678
]
}
TASK [This query is run against lookup value without from_json stored in a variable] ***
ok: [localhost] => {
"msg": [
678
]
}
TASK [This query is run directly against lookup value with from_json] **********
ok: [localhost] => {
"msg": [
678
]
}
TASK [This query is run directly against lookup value without from_json - the result is empty - why?] ***
ok: [localhost] => {
"msg": ""
}
The playbook:
---
- hosts: localhost
gather_facts: no
connection: local
tasks:
- set_fact:
from_lookup_with_from_json: "{{ lookup('file','test.json') | from_json }}"
- set_fact:
from_lookup_without_from_json: "{{ lookup('file','test.json') }}"
- name: Save the lookup value stored in a variable in a file for comparison
copy: content="{{ from_lookup_with_from_json }}" dest=./from_lookup_with_from_json.txt
- name: Save the lookup value stored in a variable in a file for comparison (they are the same)
copy: content="{{ from_lookup_without_from_json }}" dest=./from_lookup_without_from_json.txt
- name: This query is run against lookup value with from_json stored in a variable
debug: msg="{{ from_lookup_with_from_json | json_query(query) }}"
vars:
query: "Foods[].{id: Id, for: (Tags[?Key=='For'].Value)[0]} | [?for=='Tigger'].id"
- name: This query is run against lookup value without from_json stored in a variable
debug: msg="{{ from_lookup_without_from_json | json_query(query) }}"
vars:
query: "Foods[].{id: Id, for: (Tags[?Key=='For'].Value)[0]} | [?for=='Tigger'].id"
- name: This query is run directly against lookup value with from_json
debug: msg="{{ lookup('file','test.json') | from_json | json_query(query) }}"
vars:
query: "Foods[].{id: Id, for: (Tags[?Key=='For'].Value)[0]} | [?for=='Tigger'].id"
- name: This query is run directly against lookup value without from_json - the result is empty - why?
debug: msg="{{ lookup('file','test.json') | json_query(query) }}"
vars:
query: "Foods[].{id: Id, for: (Tags[?Key=='For'].Value)[0]} | [?for=='Tigger'].id"
The data (test.json):
{ "Foods" :
[ { "Id": 456
, "Tags":
[ {"Key":"For", "Value":"Heffalump"}
, {"Key":"Purpose", "Value":"Food"}
]
}
, { "Id": 678
, "Tags":
[ {"Key":"For", "Value":"Tigger"}
, {"Key":"Purpose", "Value":"Food"}
]
}
, { "Id": 911
, "Tags":
[ {"Key":"For", "Value":"Roo"}
, {"Key":"Purpose", "Value":"Food"}
]
}
]
}
json_query requires Python object (dict) as input, if you feed it with string, it gives empty string as result.
You get different result because of Ansible templating engine tricky work.
I should definitely write a post about it on my site...
After evaluating jijna2 expression Ansible try to cast complex types to Python objects (like dict or list). See my other answer.
In your case:
1.
- set_fact:
from_lookup_with_from_json: "{{ lookup('file','test.json') | from_json }}"
from_lookup_with_from_json is a dict, because you manually convert JSON-string from file to dict with from_json filter.
2.
- set_fact:
from_lookup_without_from_json: "{{ lookup('file','test.json') }}"
from_lookup_with_from_json becomes dict, because Ansible converts it when jinja2 expression ends with }}. So from_json is actually unnecessary as the last filter in chain.
3.
debug: msg="{{ lookup('file','test.json') | from_json | json_query(query) }}"
Again, you manually convert JSON-string here. So json_query get dict as input.
4.
debug: msg="{{ lookup('file','test.json') | json_query(query) }}"
In this case you feed JSON-string (not dict) as input to json_query filter. As everything happens inside one jinja2 expression, Ansible doesn't attempt to convert anything in between.
You can also get empty string result with a variable this way:
- set_fact:
from_lookup_force_string: "{{ lookup('file','test.json') | string }}"
In this case from_lookup_force_string will not be converted by Ansible tempating engine, and json_query will give you empty response on it.

Resources