I always struggle with iterations in Ansible and this time is no different.
Anyway, I have a custom fact being returned as a list of dictionaries my_list which I need to iterate to build a new variables named after the key = 'name'
[localhost] => {
"ansible_facts": {
"my_list": [
{
"name": "asd-ASX030-vc-0",
"key_2": [
"A",
"B"
]
},
{
"name": "asd-ASX030-vc-1",
"key_2": [
"C",
"D"
]
}
]
}
...
}
I can run a single list index.
tasks:
- set_fact: set_fact: {"{{item[1]['name'].split('-')[2:] | join('_')}}":"{{item[1]['key_2']}}"}
loop: "{{ my_list }}"
What I'm after is.
vc_0 = ["A", "B"]
vc_1 = ["C", "D"]
Simple:
set_fact: {"{{item['name'].split('-')[2:] | `enter code here`join('_')}}":"{{item['key_2']}}"}
loop: "{{ my_list }}"
Doesn't work:
"The task includes an option with an undefined variable. The error was: 'list object' has no attribute 'key_2'
I would like to avoid hard coding list indexes, how can I do this dynamically?
Is this the code (without formatting) that you're looking for?
shell> cat play.yml
- hosts: localhost
vars:
my_list:
- name: "asd-ASX030-vc-0"
key_2: ["A", "B"]
- name: "asd-ASX030-vc-1"
key_2: ["C", "D"]
tasks:
- debug:
msg: "{{ item.name }} = {{ item.key_2 }}"
loop: "{{ my_list }}"
shell> ansible-playbook play.yml | grep msg
msg: asd-ASX030-vc-0 = [u'A', u'B']
msg: asd-ASX030-vc-1 = [u'C', u'D']
I think you use Jinja expressions to construct the variables:
set_fact:
var2: {% for item in ansible_facts %}\
{{ item.name }}\
{% if not loop.last %},\
{% endif %}\
{% endfor %}
Related
I am trying to find a way of creating the following result with the Ansible debug module.
Desired Results
{
"myserver01": {
"host_var_one": "foo",
"host_var_two": "bar"
},
"myserver02": {
"host_var_one": "biz",
"host_var_two": "baz"
},
"myserver03": {
"host_var_one": "fizz",
"host_var_two": "buzz"
}
}
Example Inventory
[my_servers]
myserver01 host_var_one=foo host_var_two=bar
myserver02 host_var_one=biz host_var_two=baz
myserver03 host_var_one=fizz host_var_two=buzz
I would like to be able to provide a list of hostvars and have them displayed in a dict under each host in the inventory, where the key is the hostvar name, and the value is the hostvar value. Including another hostvar in the results should ideally just require adding another variable name to a list.
For example, in the task I would list that I want ["host_var_one", "host_var_two"] for each host in the inventory and get the above desired results.
I have the following task that gets somewhat close to what I want. I just can't figure out a way of listing all desired variables that I want for each host in the format described above. The following only works with one variable, and it doesn't list the variable name along with the value.
myplaybook.yml
---
- name: Test
hosts: all
gather_facts: yes
user: ansible
become: yes
tasks:
- name: Debug
debug:
msg: "{{ dict(query('inventory_hostnames', 'all') | zip(query('inventory_hostnames', 'all') | map('extract', hostvars, 'host_var_one'))) }}"
run_once: yes
Results of myplaybook.yml
$ ansible-playbook -i inventory test.yml --limit "myserver01"
PLAY [Test] **************************************************************************************************************************************************
TASK [Gathering Facts] ***************************************************************************************************************************************
ok: [myserver01]
TASK [Debug] *************************************************************************************************************************************************
ok: [myserver01] => {
"msg": {
"myserver01": "foo",
"myserver02": "biz",
"myserver03": "fizz"
}
}
My solution is as follows:
- name: Dict of host_vars
debug:
msg: "{{ dict( keys | zip(values) ) }}"
vars:
keys: "{{ hostvars | dict2items | map(attribute='key') }}"
values: "{{ hostvars | dict2items | map(attribute='value') | map('dict2items')
| map('selectattr', 'key', 'match', 'host_var_') | map('items2dict') }}"
run_once: yes
The result looks like:
TASK [Host vars dict] **************************************************
ok: [myserver01] => {
"msg": {
"myserver01": {
"host_var_one": "foo",
"host_var_two": "bar"
},
"myserver02": {
"host_var_one": "biz",
"host_var_two": "baz"
},
"myserver03": {
"host_var_one": "fizz",
"host_var_two": "buzz"
}
}
}
Step by step explanation
Two separate lists are created: keys and values.
The list for the keys is created by converting the dict with dict2items, so that the attribute key can be extracted by map. This way you get the list:
[ "myserver01", "myserver02", "myserver03" ]
For the values it works the same way, that a list of the value attributes is generated. These lists are in each case dicts and must be converted therefore by map('dict2items') again into a list. Now the key can be filtered over the list via selectattr. Here host_var_ is filtered out by match (the beginning of a string). Afterwards the result (list in list) must be converted by map('items2dict') back into a dict.
[
{
"host_var_one": "foo",
"host_var_two": "bar"
},
{
"host_var_one": "biz",
"host_var_two": "baz"
},
{
"host_var_one": "fizz",
"host_var_two": "buzz"
}
]
Finally, the two lists are merged via zip and converted back into one via dict.
Variables filtered by prefix
The variables are filtered by the prefix of their name, so several different ones (with the same prefix) can be defined for different hosts and will be filtered out completely.
E.g. the Inventory:
[my_servers]
myserver01 host_var_one=foo host_var_two=bar
myserver02 host_var_one=biz host_var_two=baz host_var_three=boz host_var_four=buz
myserver03 host_var_one=fizz host_var_two=buzz host_var_six=dazz
returns the following result:
TASK [Dict of host_vars] **********************************************
ok: [myserver01] => {
"msg": {
"myserver01": {
"host_var_one": "foo",
"host_var_two": "bar"
},
"myserver02": {
"host_var_four": "buz",
"host_var_one": "biz",
"host_var_three": "boz",
"host_var_two": "baz"
},
"myserver03": {
"host_var_one": "fizz",
"host_var_six": "dazz",
"host_var_two": "buzz"
}
}
}
I always enjoy seeing the competing ways of solving these problems, but here's mine: because json_query is rotten at dealing with top-level dicts, a little sprinkling of dict2items rotates it into an array of dicts which JMESPath is much better at dealing with, project the items into the same [{key: .key, value: ...}] shape for the things you want, then rotate it back into a dict with the opposite filter (items2dict)
- debug:
msg: '{{ hostvars | dict2items | json_query(jq) | items2dict }}'
delegate_to: localhost
run_once: true
vars:
jq: '[*].{key: key, value: {one: value.host_var_one, two: value.host_var_two}}'
produces
ok: [alpha -> localhost] => {
"msg": {
"alpha": {
"one": "a1",
"two": "a2"
},
"beta": {
"one": "b1",
"two": "b2"
}
}
}
I didn't follow what you were doing with the zip but to the best of my knowledge hostvars contains all hostvars, and thus no zip trickery required
Or, if you have either a huge list, or an unknown list, of those hostvars to extract, I gravely lament the lack of dict comprehensions in ansible's Jinja2 but the iterative version works just fine:
- debug:
msg: >-
{%- set r = {} -%}
{%- for hv in hostvars.keys() -%}
{%- set d = {} -%}
{%- for f in the_facts -%}
{%- set _ = d.update({f: hostvars[hv][f]}) -%}
{%- endfor -%}
{%- set _ = r.update({hv: d}) -%}
{%- endfor -%}
{{ r }}
vars:
the_facts:
- host_var_one
- host_var_two
The simplest solution is putting the below declarations into the vars
selection: "{{ dict(ansible_play_hosts_all|
zip(hostvars|json_query(_query))) }}"
_query: '*.{host_var_one: host_var_one, host_var_two: host_var_two}'
Given the above inventory, the result is
selection:
myserver01:
host_var_one: foo
host_var_two: bar
myserver02:
host_var_one: biz
host_var_two: baz
myserver03:
host_var_one: fizz
host_var_two: buzz
In the dictionary, all hosts keep the same variables selected by the common _query.
Example of a complete playbook for testing
- hosts: my_servers
gather_facts: false
vars:
selection: "{{ dict(ansible_play_hosts_all|
zip(hostvars|json_query(_query))) }}"
_query: '*.{host_var_one: host_var_one, host_var_two: host_var_two}'
tasks:
- debug:
var: selection
run_once: true
Q: "List the variables for each host in the inventory."
A: For example, given the inventory
shell> cat hosts
all:
children:
my_servers:
hosts:
myserver01:
host_var_one: foo
host_var_two: bar
host_var_three: baz
sel_vars: [host_var_one, host_var_two, host_var_three]
myserver02:
host_var_one: biz
host_var_two: baz
sel_vars: [host_var_one, host_var_two]
myserver03:
host_var_one: fiz
host_var_two: buz
sel_vars: [host_var_two]
The playbook below
- hosts: my_servers
gather_facts: false
vars:
sel_vals: "{{ sel_vars|map('extract', vars)|list }}"
sel_dict: "{{ dict(sel_vars|zip(sel_vals)) }}"
sel_all: "{{ ansible_play_hosts_all|map('extract', hostvars, 'sel_dict') }}"
selection: "{{ dict(ansible_play_hosts_all|zip(sel_all)) }}"
tasks:
- set_fact:
sel_dict: "{{ sel_dict }}"
- debug:
var: selection
run_once: true
gives (abridged)
selection:
myserver01:
host_var_one: foo
host_var_three: baz
host_var_two: bar
myserver02:
host_var_one: biz
host_var_two: baz
myserver03:
host_var_two: buz
The task set_fact is necessary to "instantiate" (create in hostvars) the variables sel_dict for all hosts.
You can select the variables dynamically. For example, put the below declaration into the vars
sel_vars: "{{ query('varnames', '.*_one') }}"
The result will be
selection:
myserver01:
host_var_one: foo
myserver02:
host_var_one: biz
myserver03:
host_var_one: fiz
I have a playbook which takes a input list of comma-separated values from the user:
a,b,c,d
And turns it into a list using split (extra_hosts is the variable that stores the input from the user)
- name: Generate hosts list
set_fact:
hosts_list: []
- name: Build hosts list
set_fact:
san_list: "{{ hosts_list + [host] }}"
loop_control:
loop_var: host
with_items: "{{ extra_hosts | split(',') }}"
Now this works fine so far. If I debug at this point, I get:
ok: [localhost] => {
"msg": [
"a",
"b",
"c",
"d"
]
}
What I want to do now is to output into a file, the list in the format:
Host 1: a
Host 2: b
Host 3: c
Host 4: d
I can easily iterate over the list and output using this:
- name: Append hosts to file
lineinfile:
path: "hosts.conf"
line: "Host: {{ host }}"
loop_control:
loop_var: host
with_items: "{{ hosts_list }}"
However, this outputs (obviously) as:
Host: a
Host: b
Host: c
Host: d
But I don't think I can iterate with two variables (the second being the sequence module), so I can't do something like:
- name: Append hosts to file
lineinfile:
path: "hosts.conf"
line: "Host: {{ host }}"
loop_control:
loop_var: host
with_items: "{{ hosts_list }}"
loop_control:
loop_var: id
with_sequence: start=1
I was thinking maybe somehow converting the list into a dict, so I'd end up with something like:
{id: 1, host: "a"},
{id: 2, host: "b"},
{id: 3, host: "c"},
{id: 4, host: "d"}
Then I can then just use something like {{ item.id }} and {{ item.host }} to write those out -- but to build the dict, I would still need to iterate with two pointers -- an incremental value, and the pointer within the list.
Is there way I can do this, and what is the best/correct way of doing it?
For example, the task below will be skipped if extra_hosts is not defined
- copy:
dest: hosts.conf
content: |-
{% for i in extra_hosts.split(',') %}
Host {{ loop.index }}: {{ i }}
{% endfor %}
when: extra_hosts|d('')|length > 0
If the variable extra_hosts is defined (e.g. -e extra_hosts='a,b,c,d') the task will create the file hosts.conf
shell> cat hosts.conf
Host 1: a
Host 2: b
Host 3: c
Host 4: d
Regarding your question
... the list in the format ...
you may take advantage from Extended loop variables.
---
- hosts: localhost
become: no
gather_facts: no
vars:
LIST: ['a', 'b', 'c', 'd']
tasks:
- name: Show result
debug:
msg: "Host {{ ansible_loop.index }}: {{ LIST[ansible_loop.index0] }}"
loop: "{{ LIST }}"
loop_control:
label: "{{ item }}"
extended: yes
results into an output of
TASK [Show result] ***********
ok: [localhost] => (item=a) =>
msg: 'Host 1: a'
ok: [localhost] => (item=b) =>
msg: 'Host 2: b'
ok: [localhost] => (item=c) =>
msg: 'Host 3: c'
ok: [localhost] => (item=d) =>
msg: 'Host 4: d'
Here's something I think should be straight-forward in Ansible, but I'm having trouble finding a canonical best practice.
In Ansible, I have a list of dicts in YAML:
config:
- id: x
value: X
- id: y
value: Y
I want to generate a list of strings that can be passed as a whole to the mysql_query module.
I've tried various elegant ways of doing this using Ansible filters, but that best I've come up with is to generate a list of newline-separate strings in a single template that iterates over config and then trim | split('\n') the resulting single string.
Template:
{% for item in config %}
{{ item.id }} => {{ item.value }}
{% endfor %}
Playbook task:
set_fact:
converted_items: "{{ lookup('template', './template.j2') | trim | split('\n') }}"
But this feels like a kludge.
What am I missing here?
[Note this is a canned example, to keep things simple.]
if you want to transform list of dicts to list of strings, just use loop and union:
- name: vartest
hosts: localhost
vars:
config:
- id: x
value: X
- id: y
value: Y
tasks:
- name: transform value
set_fact:
result: "{{ result | default([]) | union([item.id ~ ' => ' ~ item.value]) }}"
loop: "{{ config }}"
- name: display result
debug:
var: result
equivalent of union: result | default([]) + [item.id ~ ' => ' ~ item.value]
result:
ok: [localhost] => {
"result": [
"x => X",
"y => Y"
]
}
You don't have to use a file. Put the Jinja template into the local vars, e.g.
- set_fact:
converted_items: "{{ _converted_items|trim|split('\n') }}"
vars:
_converted_items: |
{% for item in config %}
{{ item.id }} => {{ item.value }}
{% endfor %}
gives
converted_items:
- x => X
- y => Y
There are many options of how to transform the data, e.g. the task below gives the same result
- set_fact:
converted_items: "{{ _keys|zip(_vals)|map('join', ' => ')|list }}"
vars:
_keys: "{{ config|map(attribute='id')|list }}"
_vals: "{{ config|map(attribute='value')|list }}"
I have a given vars as a list of kafka topics and possible configurations, like:
kafka_topics:
foo:
retentiontime: 3600
deletepolicy: delete
bar:
retentiontime: 3600
compression: gzip
I have figured multiple ways with dict2items to set them explicit like:
- name: Set RetentionTime for Topics
debug:
msg: "Topic {{ item.key }} get {{ item.value.retentiontime }} for retentiontime "
loop: "{{ lookup('dict', kafka_topics) }}"
when: item.value.retentiontime is defined
But is it possible to get a output like
Topic foo get 3600 for retentiontime
Topic foo get delete for deletepolicy
Topic bar get 3600 for retentiontime
Topic bar get gzip for compression
without defining the value name by name with ansible?
I also tried with
- name: Loop over subelements of the dictionary
debug:
msg: "Key={{ item.0.key }} value={{ item.1 }}"
loop: "{{ lookup('dict', kafka_topics) | list | subelements('value') }}"
which prints Key=bar value={'compression': gzip} but now i'm stuck at seperating those two. Is there a way to extract from item.1 the key and the value?
The simplest option is Jinja. For example
- debug:
msg: |-
{% for k1,v1 in kafka_topics.items() %}
{% for k2,v2 in v1.items() %}
Topic {{ k1 }} get {{ v2 }} for {{ k2 }}
{% endfor %}
{% endfor %}
gives
msg: |-
Topic foo get 3600 for retentiontime
Topic foo get delete for deletepolicy
Topic bar get 3600 for retentiontime
Topic bar get gzip for compression
Create a dictionary of lists first if the iteration is needed. For example the template
shell> cat templates/kafka_facts_list.j2
{% for k1,v1 in kafka_topics.items() %}
{{ k1 }}:
{% for k2,v2 in v1.items() %}
- {key: {{ k2 }}, val: {{ v2 }}}
{% endfor %}
{% endfor %}
- set_fact:
dict2: "{{ lookup('template', 'kafka_facts_list.j2')|from_yaml }}"
- debug:
var: dict2
gives
dict2:
bar:
- key: retentiontime
val: 3600
- key: compression
val: gzip
foo:
- key: retentiontime
val: 3600
- key: deletepolicy
val: delete
Then use subelements
- debug:
msg: "Topic {{ item.0.key }} get {{ item.1.val }} for {{ item.1.key }}"
with_subelements:
- "{{ dict2|dict2items }}"
- value
gives
msg: Topic foo get 3600 for retentiontime
msg: Topic foo get delete for deletepolicy
msg: Topic bar get 3600 for retentiontime
msg: Topic bar get gzip for compression
#Vladimir's answer is indeed the most straightforward one if you need to write some debugging info as above or create a configuration file from a template.
But this might get a little trickier if you actually need to loop over this data in a classic task where you have to pass each value separately to the corresponding moldule's option.
In such cases, here is an alternative that will transform your current dict to list.
The goal is to go from:
"first topic name":
"option 1 name": "option 1 value"
"option 2 name": "option 2 value"
"second topic name": ...
to
- topic_name: "first topic name"
topic_options:
- option_name: "option 1 name"
option_value: "option 1 value"
- option_name: "option 2 name"
option_value: "option 2 value"
- topic_name: "second topic name"
...
This transformed data is then easily loopable with subelements as demonstrated in the following playbook:
---
- hosts: localhost
gather_facts: false
vars:
kafka_topics: {"foo": {"retentiontime": 3600, "deletepolicy": "delete"}, "bar": {"retentiontime": 3600, "compression": "gzip"}}
tasks:
- name: Transform our list to something easier to loop with subelements
vars:
current_topic:
topic_name: "{{ item.topic_name }}"
topic_options: "{{ item.topic_options | dict2items(key_name='option_name', value_name='option_value') }}"
set_fact:
my_topic_list: "{{ my_topic_list | default([]) + [current_topic] }}"
loop: "{{ kafka_topics | dict2items(key_name='topic_name', value_name='topic_options') }}"
- name: Show the transformed var
debug:
var: my_topic_list
- name: Loop over topics and their options
debug:
msg: "Topic `{{ topic.0.topic_name }}` has option `{{ topic.1.option_name }}` with value `{{ topic.1.option_value }}`"
loop: "{{ my_topic_list | subelements('topic_options') }}"
loop_control:
label: "{{ topic.0.topic_name }} - {{ topic.1.option_name }} - {{ topic.1.option_value }}"
loop_var: topic
Which gives:
PLAY [localhost] ***********************************************************************************************************************************************************************************************************************
TASK [Transform our list to something easier to loop with subelements] *****************************************************************************************************************************************************************
ok: [localhost] => (item={'topic_name': 'foo', 'topic_options': {'retentiontime': 3600, 'deletepolicy': 'delete'}})
ok: [localhost] => (item={'topic_name': 'bar', 'topic_options': {'retentiontime': 3600, 'compression': 'gzip'}})
TASK [Show the transformed var] ********************************************************************************************************************************************************************************************************
ok: [localhost] => {
"my_topic_list": [
{
"topic_name": "foo",
"topic_options": [
{
"option_name": "retentiontime",
"option_value": 3600
},
{
"option_name": "deletepolicy",
"option_value": "delete"
}
]
},
{
"topic_name": "bar",
"topic_options": [
{
"option_name": "retentiontime",
"option_value": 3600
},
{
"option_name": "compression",
"option_value": "gzip"
}
]
}
]
}
TASK [Loop over topics and their options] **********************************************************************************************************************************************************************************************
ok: [localhost] => (item=foo - retentiontime - 3600) => {
"msg": "Topic `foo` has option `retentiontime` with value `3600`"
}
ok: [localhost] => (item=foo - deletepolicy - delete) => {
"msg": "Topic `foo` has option `deletepolicy` with value `delete`"
}
ok: [localhost] => (item=bar - retentiontime - 3600) => {
"msg": "Topic `bar` has option `retentiontime` with value `3600`"
}
ok: [localhost] => (item=bar - compression - gzip) => {
"msg": "Topic `bar` has option `compression` with value `gzip`"
}
PLAY RECAP *****************************************************************************************************************************************************************************************************************************
localhost : ok=3 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
I have a json structure like this:
{
"group_a": {
"setting_one": "text one",
"setting_two": "text two"
},
"group_b": {
"setting_three": "text three",
"setting_four": "text four"
}
}
And I need to do iterate over the dicts inside the dicts and create tasks, kinda like this:
- name: Task group_a setting_one
...
- name: Task group_a setting_two
...
- name: Task group_b setting_three
...
- name: Task group_b setting_four
...
I was hoping to make something like this:
- name: Insert {{ group }} {{ setting }}
ini_file:
path: settings.ini
section: "{{ group }}"
option: "{{ setting }}"
value: "{{ setting_value }}"
with_nested:
- "{{ data.keys() }}" # Can I get this key and use in the next loop?
- "{{ data.key.keys() }}"
How can I do this? I have been messing about with loop, with_items, with_dict etc, but I can't manage to get it exactly the way I need. I also tried with_nested, but that does not seem to do what I need either.
Kind regards, Jonas
Let's reduce the structure to one with the same piece of information
my_data:
group_a:
- one
- two
group_b:
- three
- four
Then the play below
- debug:
msg: "{{ item.0.key }} setting_{{ item.1 }}"
with_subelements:
- "{{ my_data|dict2items }}"
- value
gives:
"msg": "group_b setting_three"
"msg": "group_b setting_four"
"msg": "group_a setting_one"
"msg": "group_a setting_two"
It's not straightforward to use the original data. The play below separates the keys and values
tasks:
- set_fact:
my_groups: "{{ my_data.keys()}}"
- debug:
msg: "{{ item }} {{ my_data|dict2items|json_query(my_query)|to_yaml }}"
loop: "{{ my_groups }}"
vars:
my_query: "[?key=='{{ item }}'].value"
and gives:
"msg": "group_b - - {setting_three: text three}\n - {setting_four: text four}\n"
"msg": "group_a - - {setting_one: text one}\n - {setting_two: text two}\n"
Multi-step loops are needed to proceed. For example the play below
- set_fact:
my_data1: |
{% for group, values in my_data.items() %}
{% for value in values %}
{% for item_key, item_val in value.items() %}
Task {{ group }} {{ item_key }},
{% endfor %}{% endfor %}{% endfor %}
- debug:
msg: "{{ my_data1.split('\n') }}"
gives:
"msg": [
"Task group_b setting_three,",
"Task group_b setting_four,",
"Task group_a setting_one,",
"Task group_a setting_two,",
""
]