I want to combine two attribute into single string separated by delimiter using the json_query in ansible
Sample data
{
"locations": [
{"name": "Seattle", "state": "WA"},
{"name": "New York", "state": "NY"},
{"name": "Bellevue", "state": "WA"},
{"name": "Olympia", "state": "WA"}
]
}
As shown in above data set i'm trying to filter the state "WA" and execpted output is:
[
"Seattle-WA",
"Bellevue-WA",
"Olympia-WA"
]
What i have tried as of now:
- debug:
msg: "{{ chart_list.HELM_CHARTS | json_query(\"[?state == 'WA'].{name:name,state:state}\") }}"
Output:
[
{
"name": "Seattle",
"state": "WA"
},
{
"name": "Bellevue",
"state": "WA"
},
{
"name": "Olympia",
"state": "WA"
}
]
Updated :
I was able to get the expected result by trial and error method and these are my findings:
[?state == 'WA'].[join('-',[name,state])][]
Output:
[
"Seattle-WA",
"Bellevue-WA",
"Olympia-WA"
]
Also if the input which you give is in unicode format, i suggest you to add to_json | from_json expressions as mentioned below:
selected_cities: "{{ test.locations| to_json | from_json | json_query(\"[?state == 'WA'].[join('-',[name,state])][]\") }}"
Using above expression will eliminate unicode error whil using the values or in any condition.
Check JMESPath site for more detail on the json_query, it was really helpful in resolving the issue.
For example
- debug:
msg: "{{ locations|
json_query('[?state == `WA`].[name,state]')|
map('join', '-')|list }}"
gives
msg:
- Seattle-WA
- Bellevue-WA
- Olympia-WA
The same result gives the task below using Jinja2 filters only
- debug:
msg: "{{ _names|zip(_states)|map('join', '-')|list }}"
vars:
_locations: "{{ locations|selectattr('state', 'eq', 'WA')|list }}"
_names: "{{ _locations|map(attribute='name')|list }}"
_states: "{{ _locations|map(attribute='state')|list }}"
json_query issue (fixed in 2.10 and later)
There is JMESPath join. Unfortunately
- debug:
msg: "{{ locations|
json_query('[].join(`-`, [name,state])') }}"
fails
msg: |-
JMESPathError in json_query filter plugin:
In function join(), invalid type for value: Seattle, expected one of: ['array-string'], received: "AnsibleUnicode"
to_json|from_json workaround
Quoting from json_query: Add examples for starts_with and contains #72821
data structure returned from register variables needs to be parsed using to_json | from_json in order to get a correct result. Fixes: ansible-collections/community.general#320
- debug:
msg: "{{ locations|to_json|from_json|
json_query('[].join(`-`, [name,state])') }}"
gives
msg:
- Seattle-WA
- New York-NY
- Bellevue-WA
- Olympia-WA
Just for the sake of a pure JMESPath way of doing it, as your trial and error solution still have an unneeded extra layer of complexity.
When you are doing
[?state == 'WA'].[join('-', [name, state])][]
You are creating an array [join('-', [name, state])] then flattening it [] for no reason.
You can just go to the solution with a shorter approach:
[?state == `WA`].join(`-`, [name, state])
Also mind that you can overcome the quotes in quotes (simple or double) complication for JMESPath queries using:
YAML multilines string: How do I break a string in YAML over multiple lines?
Backticks in your JMESPath query, as pointed in the documentation:
In the example above, quoting literals using backticks avoids escaping quotes and maintains readability.
Source: https://docs.ansible.com/ansible/latest/user_guide/playbooks_filters.html#selecting-json-data-json-queries
So you end up with (see note below if you are on an Ansible version < 2.10):
- debug:
msg: >-
{{ test.locations
| json_query('[?state == `WA`].join(`-`, [name, state])') }}
Please note: as raised by #Vladimir Botka on the versions prior to 2.10, you will be affected by this issue: https://github.com/ansible/ansible/issues/27299#issuecomment-331068246, forcing you to add a | to_json | from_json filter on the list.
Given the playbook:
- hosts: all
gather_facts: yes
tasks:
- debug:
msg: >-
{{ test.locations
| json_query('[?state == `WA`].join(`-`, [name, state])')
}}
vars:
test:
locations:
- name: Seattle
state: WA
- name: New York
state: NY
- name: Bellevue
state: WA
- name: Olympia
state: WA
This yields:
[
"Seattle-WA",
"Bellevue-WA",
"Olympia-WA"
]
Related
I am trying to query the following Infoblox data with Ansible and JMESPath json_query:
{
"ip_records.json": {
"result": [
{
"_ref": "fixedaddress/blabla",
"ipv4addr": "10.10.10.10",
"network_view": "Bla"
},
{
"_ref": "record:host/blabla",
"ipv4addrs": [
{
"_ref": "record:host_ipv4addr/blabla",
"host": "bla.bla.com",
"ipv4addr": "10.10.10.10"
}
],
"name": "bla.bla.com",
"view": " "
},
{
"_ref": "record:a/blabla",
"ipv4addr": "10.10.10.10",
"name": "bla.bla.com",
"view": "bla"
}
]
}
}
I want to get only the _ref value for the item with fixedaddress in the _ref value.
Forgot to add that there might also be multiple records with fixedaddress but different IP's. So I also want to filter on a specific IP as the same time.
I have created queries to filter
only on IP address given as input
the string fixedaddress
a combination of both
The first two work as expected. But, I want to combine both conditions and would expect to get the single item as output, but I get nothing. I tried using && and | to combine both, as showed below.
- name: "Search IP Record: Task 2.2: Filter Results."
vars:
jmesquery: "[] | [?ipv4addr==`{{ infoblox_ip }}`]._ref"
set_fact:
ip_records_refs: "{{ ip_records.json.result | json_query(jmesquery) }}"
- name: "Search IP Record: Task 2.4: Filter Results."
vars:
jmesquery: "[] | [?_ref.contains(#,`fixedaddress`)]._ref"
set_fact:
ip_records_refs: "{{ ip_records.json.result | to_json | from_json | json_query(jmesquery) }}"
- name: "Search IP Record: Task 2.6: Filter Results."
vars:
# jmesquery: "[] | ([?ipv4addr==`{{ infoblox_ip }}` && _ref.contains(#,`fixedaddress`)])._ref"
jmesquery: "[] | [?ipv4addr==`{{ infoblox_ip }}`].ref | [?_ref.contains(#,`fixedaddress`)]._ref"
set_fact:
ip_records_refs: "{{ ip_records.json.result | to_json | from_json | json_query(jmesquery) }}"
Output:
TASK [Search IP Record: Task 2.3 Dump variable Content] ***********
ok: [localhost] => {
"ip_records_refs": [
"fixedaddress/blabla",
"record:a/blabla"
]
}
TASK [Search IP Record: Task 2.5 Dump variable Content] ***********
ok: [localhost] => {
"ip_records_refs": [
"fixedaddress/blabla"
]
}
TASK [Search IP Record: Task 2.7 Dump variable Content] ***********
ok: [localhost] => {
"ip_records_refs": []
}
You are misusing the pipe expression.
From your trial, it is hard to tell exactly what you think it does, but here is a simple explanation: you might not see it, but a JMESPath filter on an array does not return you a JSON array, rather it returns you a projection.
You cannot chain a filter on top of projection, you need to reset it first, in order to get the resulting JSON array, and this is what the pipe expression is meant for.
In your case, you do not want to have a filter on top of a projection, you want a filter with multiple conditions, so, your last set_fact query should read:
jmesquery: >-
[?
_ref.contains(#,`fixedaddress`)
&& ipv4addr == `{{ infoblox_ip }}`
]._ref
And your two first queries should be simplified to:
jmesquery: "[?_ref.contains(#,`fixedaddress`)]._ref"
and
jmesquery: "[?ipv4addr == `{{ infoblox_ip }}`]._ref"
Q: "Get _ref value for the item with 'fixedaddress' in the _ref key."
A: The query below
jmesquery: "[?_ref.contains(#,`fixedaddress`)]._ref"
ip_records_refs: "{{ ip_records.json.result|json_query(jmesquery) }}"
gives the expected result
ip_records_refs:
- fixedaddress/blabla
Example of a complete playbook for testing
- hosts: localhost
vars:
ip_records:
json:
result:
- _ref: fixedaddress/blabla
ipv4addr: 10.10.10.10
network_view: Bla
- _ref: record:host/blabla
ipv4addrs:
- _ref: record:host_ipv4addr/blabla
host: bla.bla.com
ipv4addr: 10.10.10.10
name: bla.bla.com
view: ' '
- _ref: record:a/blabla
ipv4addr: 10.10.10.10
name: bla.bla.com
view: bla
# Get _ref value for the item with 'fixedaddress' in the _ref key
jmesquery: "[?_ref.contains(#,`fixedaddress`)]._ref"
ip_records_refs: "{{ ip_records.json.result|json_query(jmesquery) }}"
tasks:
- debug:
var: ip_records
- debug:
var: ip_records_refs
I am trying to match certain key from an ansible output/vars, and return the value of another key.
Here is my vars.
{
"_meta": {
"hostvars": {
"ansibletower1_422c3aed-780c-8c33-3054-d32e330c9285": {
"guest.hostName": "ansibletower1.rh.pro",
"name": "ansibletower1"
},
"child domain_422c4cd1-d644-7eeb-df7c-c32a2a05c030": {
"guest.hostName": null,
"name": "child domain"
}
}
}
}
My non-working playbook
- hosts: ansibletower1.rh.pro
tasks:
- debug:
msg: "{{ _meta | json_query(querystr) }}"
vars:
querystr: "[?hostvars.*.\"guest.hostName\"=='{{inventory_hostname}}'].name"
I am trying to follow the method here, https://blog.networktocode.com/post/ansible-filtering-json-query/. However in my scenario is not array, which is different than the example in the link.
My end goal is to match guest.hostName with {{ inventory_hostname }}, and return the value of the name by using - set_fact: to register it to another variable.
I'd try with:
tasks:
- name: Loop over data and continue if string was found.
debug:
msg: "{{ _meta | json_query(querystr) }}"
vars:
querystr: "hostvars.* | [?\"guest.hostName\"==`{{inventory_hostname}}`].name"
when you select all keys with .* you get an array back, so it is piped into another query that filters and returns name.
I need to Fetch the ipv4Address value from the Below JSON Output using set_fact. Finally ending up with some errors. Could you please suggest me some best method.
JSON Output:
"result_dns": {
"ansible_facts": {
"azure_dnsrecordset": [
{
"etag": "440922d5-b234-488a-8cbc-97b77f0fef8f",
"id": "2",
"name": "test2",
"properties": {
"ARecords": [
{
"ipv4Address": "10.30.23.5"
}
],
"TTL": 3600,
"fqdn": "test2.testzone.com."
},
"type": "Microsoft.Network/dnszones/A"
I am using the set_fact as below to retrieve ipv4Address.
- name: name
set_fact:
host_name: "{{ result_dns.ansible_facts.azure_dnsrecordset map(attribute='ipv4Address') | list }}"
I am not able to filter the value by above method. Can you suggest me some best method to filter the Value.
I think you are getting snared by the fact that 'azure_dnsrecordset' contains a list of dictionaries. There is only one, but I guess the presumption is that there could in other conditions be multiple. This will do what you have asked, but once you have more than one record, you might find your requirements are more detailed than in this question:
# Presumes that you have a 'result_dns' variable set in Ansible containing the JSON
- set_fact:
ip_s: "{{ ( ip_s | default([]) ) + [ item.1.ipv4Address ] }}"
with_subelements:
- "{{ result_dns.ansible_facts.azure_dnsrecordset }}"
- properties.ARecords
- debug:
msg: "{{ ip_s }}"
with_subelements, extracts keys from a list of dictionaries.
This is just going to produce a list of IP addresses, which might not be useful in the case you have several record sets returned. If you do something like:
ip_s: "{{ ( ip_s | default([]) ) + [ { 'etag': item.0.etag, 'ip_address': item.1.ipv4Address } ] }}"
that will give you a list of dictionaries containing an 'etag' and 'ip_address' key, giving you a way to identify the recordsets they came from.
Hopefully enough to get you started.
When should I use the from_json filter in Ansible?
I found out that using it sometimes has and sometimes have no effect.
Please consider the following example which illustrates the inconsistency I am getting.
Included in reverse order are: the questions - expected result - actual result - the playbook - the data. The data is taken from this question and the playbook is based on this answer.
The question(s):
Why storing the left part (before json_query) of the following expression in a variable and then using json_query on the variable causes the expression to be evaluated differently?
"{{ lookup('file','test.json') | json_query(query) }}"
Why does adding from_json filter alter the results (but does not if processing a variable):
"{{ lookup('file','test.json') | from_json | json_query(query) }}"
Expected result:
Last four tasks should give the same result. Alternatively, last two tasks should give the same result as previous two tasks.
Actual result (last four tasks only):
One task result differs.
TASK [This query is run against lookup value with from_json stored in a variable] ***
ok: [localhost] => {
"msg": [
678
]
}
TASK [This query is run against lookup value without from_json stored in a variable] ***
ok: [localhost] => {
"msg": [
678
]
}
TASK [This query is run directly against lookup value with from_json] **********
ok: [localhost] => {
"msg": [
678
]
}
TASK [This query is run directly against lookup value without from_json - the result is empty - why?] ***
ok: [localhost] => {
"msg": ""
}
The playbook:
---
- hosts: localhost
gather_facts: no
connection: local
tasks:
- set_fact:
from_lookup_with_from_json: "{{ lookup('file','test.json') | from_json }}"
- set_fact:
from_lookup_without_from_json: "{{ lookup('file','test.json') }}"
- name: Save the lookup value stored in a variable in a file for comparison
copy: content="{{ from_lookup_with_from_json }}" dest=./from_lookup_with_from_json.txt
- name: Save the lookup value stored in a variable in a file for comparison (they are the same)
copy: content="{{ from_lookup_without_from_json }}" dest=./from_lookup_without_from_json.txt
- name: This query is run against lookup value with from_json stored in a variable
debug: msg="{{ from_lookup_with_from_json | json_query(query) }}"
vars:
query: "Foods[].{id: Id, for: (Tags[?Key=='For'].Value)[0]} | [?for=='Tigger'].id"
- name: This query is run against lookup value without from_json stored in a variable
debug: msg="{{ from_lookup_without_from_json | json_query(query) }}"
vars:
query: "Foods[].{id: Id, for: (Tags[?Key=='For'].Value)[0]} | [?for=='Tigger'].id"
- name: This query is run directly against lookup value with from_json
debug: msg="{{ lookup('file','test.json') | from_json | json_query(query) }}"
vars:
query: "Foods[].{id: Id, for: (Tags[?Key=='For'].Value)[0]} | [?for=='Tigger'].id"
- name: This query is run directly against lookup value without from_json - the result is empty - why?
debug: msg="{{ lookup('file','test.json') | json_query(query) }}"
vars:
query: "Foods[].{id: Id, for: (Tags[?Key=='For'].Value)[0]} | [?for=='Tigger'].id"
The data (test.json):
{ "Foods" :
[ { "Id": 456
, "Tags":
[ {"Key":"For", "Value":"Heffalump"}
, {"Key":"Purpose", "Value":"Food"}
]
}
, { "Id": 678
, "Tags":
[ {"Key":"For", "Value":"Tigger"}
, {"Key":"Purpose", "Value":"Food"}
]
}
, { "Id": 911
, "Tags":
[ {"Key":"For", "Value":"Roo"}
, {"Key":"Purpose", "Value":"Food"}
]
}
]
}
json_query requires Python object (dict) as input, if you feed it with string, it gives empty string as result.
You get different result because of Ansible templating engine tricky work.
I should definitely write a post about it on my site...
After evaluating jijna2 expression Ansible try to cast complex types to Python objects (like dict or list). See my other answer.
In your case:
1.
- set_fact:
from_lookup_with_from_json: "{{ lookup('file','test.json') | from_json }}"
from_lookup_with_from_json is a dict, because you manually convert JSON-string from file to dict with from_json filter.
2.
- set_fact:
from_lookup_without_from_json: "{{ lookup('file','test.json') }}"
from_lookup_with_from_json becomes dict, because Ansible converts it when jinja2 expression ends with }}. So from_json is actually unnecessary as the last filter in chain.
3.
debug: msg="{{ lookup('file','test.json') | from_json | json_query(query) }}"
Again, you manually convert JSON-string here. So json_query get dict as input.
4.
debug: msg="{{ lookup('file','test.json') | json_query(query) }}"
In this case you feed JSON-string (not dict) as input to json_query filter. As everything happens inside one jinja2 expression, Ansible doesn't attempt to convert anything in between.
You can also get empty string result with a variable this way:
- set_fact:
from_lookup_force_string: "{{ lookup('file','test.json') | string }}"
In this case from_lookup_force_string will not be converted by Ansible tempating engine, and json_query will give you empty response on it.
I have two lists:
the_list:
- { name: foo }
- { name: bar }
- { name: baz }
and I use a task which gets some value for its every element:
- name: Get values
shell:
magic_command {{ item.name }}
with_items: the_list
register: spells
from now on I can use the_list and its correspondig values together:
- name: Use both
shell:
do_something {{ item.0.name }} {{ item.1.stdout }}
with_together:
- "{{ the_list }}"
- "{{ spells.results }}"
All works fine but it's uncomfortable to use with_together for many tasks and it'll be hard to read that code in a future so I would be more than happy to build merged_list from that which I can use in a simple way. Let say something like this:
merged_list:
- { name: foo, value: jigsaw }
- { name: bar, value: crossword }
- { name: baz, value: monkey }
which makes the puzzle. Anyone can help ?
I wrote two ansible filters to tackle this problem: zip and todict which are available in my repo at https://github.com/ahes/ansible-filter-plugins
Sample ansible playbook:
- hosts: localhost
vars:
users:
- { name: user1 }
- { name: user2 }
tasks:
- name: Get uids for users
command: id -u {{ item.name }}
register: uid_results
with_items: users
- set_fact:
uids: "{{ uid_results.results | map(attribute='stdout') | todict('uid') }}"
- set_fact:
users: "{{ users | zip(uids) }}"
- name: Show users with uids
debug: var=users
Result would be:
TASK [Show users with uids] ****************************************************
ok: [localhost] => {
"users": [
{
"name": "user1",
"uid": "1000"
},
{
"name": "user2",
"uid": "2000"
}
]
}
It may be an overkill but you should try to write a custom filter plugin.
Each time you iterates the_list you simple wants to add value to that dict {name: 'foo'} right?
After the update you just want that the new dict has the value like: {name: 'foo', value: 'jigsaw'}
The filter plugin for that it's pretty simple:
def foo(my_list, spells):
try:
aux = my_list
for i in xrange(len(aux)):
my_list[i].update(spells[i])
return my_list
except Exception, e:
raise errors.AnsibleFilterError('Foo plugin error: %s, arguments=%s' % str(e), (my_list,spells) )
class FilterModule(object):
def filters(self):
return {
'foo' : foo
}
After adding this python code inside your plugins directory, you can easily call the foo plugin passing the spells list as a parameter:
- name: Get values
shell:
magic_command {{ item.name }}
with_items: the_list
register: spells
- name: Use both
shell:
do_something {{ item.name }} {{ item.value }}
with_items:
- "{{ the_list | foo(spells.results) }}"
NOTE: The python code it's just an example. Read the ansible documentations about developing filter plugins.
I think I've found a cleaner, easier way to deal with these kind of things. Ansible runs all strings through jinja and then tries to load the result as yaml. This is because jinja only outputs strings so that allows it to load a data structure from a variable if there is one.
So any valid yaml in a string is loaded as a data structure -- so if you template valid yaml it will get loaded as data.
Trying to template correct yaml in the conventional, human form is tricky. But yaml loads all json. So, json is easier because there is no need to worry about whitespace. One bonus though, yaml does not care about extra commas, so that makes templating it easier.
In this case here is the playbook from the top answer rewritten to use this method.
- hosts: localhost
vars:
users:
- { name: "user1" }
- { name: "user2" }
tasks:
- name: Get uids for users
command: id -u {{ item.name }}
register: uid_results
loop: "{{ users }}"
- name: Show users with uids
debug: var=users_with_uids
vars:
users_with_uids: |
[
{% for user_dict, uid in users | zip(uids) %}
{
"name": {{ user_dict['name'] | to_json }},
"uid": {{ uid | to_json }},
},
{% endfor %}
]
uids: "{{ uid_results.results | map(attribute='stdout') }}"
Notes
The | character tells yaml to load a multi-line string. Instead of putting the variables in quotes I use the to_json filter which will quote it and, more importantly, automatically escape anything in the variable that needs escaping. Also, remember commas after list or dictionary elements.
The results should be the same:
TASK [Show users with uids] ************************************************************
ok: [localhost] => {
"users_with_uids": [
{
"name": "user1",
"uid": "1000"
},
{
"name": "user2",
"uid": "1001"
}
]
}
One more thing
I like to use the yaml callback especially for testing this. That way if my json-looking yaml doesn't get loaded I'll see a json-like structure. Otherwise it will come back in normal looking yaml if it was loaded. You can enable this by environment variable -- export ANSIBLE_STDOUT_CALLBACK=community.general.yaml.