Ansible: approach to loop through a complex variable in order to access different layers under the same task - ansible

What would be the correct approach to loop through a complex dictionary variable in an Ansible play in order to be able to call items from different layers of the variable in the same module/task?
The variable in question looks like this (but it could be any number of layers down, so far I only have about 3 layers down):
installation:
v10:
installs: yes
das_username: dasusr
das_group: dasadm
package_name: v10.tar.gz
binaries_folder: binaries
install_path: /opt/V10
instances:
inst1:
enabled: yes
username: inst1
group: dbiadm
port: 50000
databases:
jts:
name: 'JTS'
state: present
pagesize: 32768
rtc:
name: 'CCM'
state: absent
pagesize: 16384
rqm:
name: 'QM'
state: absent
pagesize: 32768
inst2:
enabled: no
username: inst2
group: dbiadm
port: 50005
databases:
warehouse:
name: 'DW'
state: present
pagesize: 32768
v9:
installs: no
instances:
inst3:
enabled: no
databases:
rqm:
name: 'RM'
state: absent
pagesize: 32768
lqe:
name: 'LQE'
state: absent
pagesize: 16384
Tasks include creating databases using the install_path (layer 1) and databases parameters (layer 3) under the same shell module or in a template.
For example (the task should look something like this) but with the correct method of looping:
- name: Creating the databases
become_user: "{{ item.0.instances.username }}"
shell: "./db2 create database {{ item.1.name }} using pagesize {{ item.1.pagesize }}"
args:
chdir: "{{ item.0.install_path }}/bin"
with_subelements:
- "{{ installation }}"
- instances.databases.clm
Thank you!

It's not a trivial feat to do such thing in ansible, becase loops in ansible are one-dimensional. There is a hackish way though, you can use nested includes with loop control. So let's say i put all of your variables inside vars.yml and have a following files:
playbook-nested.yml
- hosts: localhost
tasks:
- name: pass version forward
include: instances.yml v="{{ version_item }}"
with_items: "{{ installation }}"
loop_control:
loop_var: version_item
instances.yml
- name: Pass instance forward
include: databases.yml i={{ instance_item }}
with_items: "{{ installation[v].instances }}"
loop_control:
loop_var: instance_item
databases.yml
- name: echo command
debug: msg="./db2 create database {{ item.name }} using pagesize {{ item.pagesize }}"
with_items: "{{ installation[v].instances[i].databases.clm }}"
when: installation[v].instances[i].databases.clm is defined
So the output of command ansible-playbook playbook-nested.yml -e #vars.yml will be:
PLAY [localhost] ***************************************************************
TASK [setup] *******************************************************************
ok: [localhost]
TASK [pass version forward] ****************************************************
included: /home/user/instances.yml for localhost
included: /home/user/instances.yml for localhost
TASK [Pass instance forward] ***************************************************
included: /home/user/databases.yml for localhost
TASK [echo command] ************************************************************
ok: [localhost] => (item={u'state': u'absent', u'name': u'RQM', u'pagesize': 32768}) => {
"item": {
"name": "RQM",
"pagesize": 32768,
"state": "absent"
},
"msg": "./db2 create database RQM using pagesize 32768"
}
ok: [localhost] => (item={u'state': u'absent', u'name': u'LQE', u'pagesize': 16384}) => {
"item": {
"name": "LQE",
"pagesize": 16384,
"state": "absent"
},
"msg": "./db2 create database LQE using pagesize 16384"
}
TASK [Pass instance forward] ***************************************************
included: /home/user/databases.yml for localhost
included: /home/user/databases.yml for localhost
TASK [echo command] ************************************************************
ok: [localhost] => (item={u'state': u'present', u'name': u'JTS', u'pagesize': 32768}) => {
"item": {
"name": "JTS",
"pagesize": 32768,
"state": "present"
},
"msg": "./db2 create database JTS using pagesize 32768"
}
ok: [localhost] => (item={u'state': u'absent', u'name': u'CCM', u'pagesize': 16384}) => {
"item": {
"name": "CCM",
"pagesize": 16384,
"state": "absent"
},
"msg": "./db2 create database CCM using pagesize 16384"
}
ok: [localhost] => (item={u'state': u'absent', u'name': u'QM', u'pagesize': 32768}) => {
"item": {
"name": "QM",
"pagesize": 32768,
"state": "absent"
},
"msg": "./db2 create database QM using pagesize 32768"
}
TASK [echo command] ************************************************************
skipping: [localhost]
PLAY RECAP *********************************************************************
localhost : ok=8 changed=0 unreachable=0 failed=0
I'm not sure what does clm mean and do i have to loop over databases as well, but i think the concept is clear to you now.
You can also use the nested_loops on the lower level i guess.

Related

add_host - add all host

I made a playbook that devid the hosts to windows and linux groups. this is the script (for now its just Linux and print the items that in the group):
---
- name: firstPlaybook
hosts: all
gather_facts: true
tasks:
- name: Linux Group
add_host:
name: "{{ item }}"
group: LinuxGroup
when: hostvars[item].ansible_system == 'Linux'
with_items: "{{ ansible_play_hosts }} "
run_once: yes
- name: dubug
debug:
msg: "{{ item }}"
with_items: LinuxGroup
and when I run the script with -vvv this is what I saw in the tasks itself:
TASK [Linux Group] *************************************************************
task path: /home/opc/playbooks/playbook.yml:7
skipping: [windowsserver] => (item=windowsserver) => {
"ansible_loop_var": "item",
"changed": false,
"item": "windowsserver",
"skip_reason": "Conditional result was False"
}
creating host via 'add_host': hostname=linuxserver
changed: [windowsserver] => (item=linuxserver) => {
"add_host": {
"groups": [
"LinuxGroup"
],
"host_name": "linuxserver",
"host_vars": {
"group": "LinuxGroup"
}
},
"ansible_loop_var": "item",
"changed": true,
"item": "linuxserver"
}
Perform task: TASK: dubug (N)o/(y)es/(c)ontinue: y
Perform task: TASK: dubug (N)o/(y)es/(c)ontinue: *******************************
TASK [dubug] *******************************************************************
task path: /home/opc/playbooks/playbook.yml:15
ok: [windowsserver] => (item=LinuxGroup) => {
"msg": "LinuxGroup"
}
ok: [linuxserver] => (item=LinuxGroup) => {
"msg": "LinuxGroup"
}
why its adding the windows server the the linux group even thought it suppose to be skipped (I changed the ips to windowsserver and linuxserver that I wont publish them)?
All is working as expected. Take a look at
- debug:
msg: "{{ groups.LinuxGroup }}"

ansible loop with failed_when on register variable list output

Team,
I have below task working fine when i only have one node or only one item but I need to modify it work for all items returned in the list that is stored in register variable.
- name: "Fetch all CPU nodes from clusters using K8s beta.kubernetes.io/instance-type"
k8s_info:
kind: Node
label_selectors:
- "beta.kubernetes.io/instance-type={{ kube_cpu_node_class }}"
verify_ssl: no
register: cpu_class_list
failed_when: not cpu_class_list.resources
How can i do this for all nodes in cpu_class_list variable with loop or with_items?
proposed solution but does not work
- name: "Fetch all CPU nodes from clusters using K8s beta.kubernetes.io/instance-type"
k8s_info:
kind: Node
label_selectors:
- "beta.kubernetes.io/instance-type={{ kube_cpu_node_class }}"
verify_ssl: no
register: cpu_class_list
failed_when: not {{ item }}
with_items: cpu_class_list.resources
sample output with two nodes is below
services-pre-install-checks : debug]
ok: [localhost] => {
"cpu_class_list": {
"changed": false,
"deprecations": [
{
"msg": "The 'k8s_facts' module has been renamed to 'k8s_info'",
"version": "2.13"
}
],
"failed": false,
"failed_when_result": false,
"resources": [
{
"apiVersion": "v1",
"kind": "Node",
"metadata": {
"annotations": {
"volumes.kubernetes.io/controller-managed-attach-detach": "true"
},
"creationTimestamp": "2019-07-16T00:23:27Z",
"labels": {
"nodeType": "cpu"
},
"name": "node1",
"nodeInfo": {
"architecture": "amd64",
}
}
},
{
{
"apiVersion": "v1",
"kind": "Node",
"metadata": {
"annotations": {
"volumes.kubernetes.io/controller-managed-attach-detach": "true"
},
"creationTimestamp": "2019-07-16T00:23:27Z",
"labels": {
"nodeType": "cpu"
},
"name": "node2",
"nodeInfo": {
"architecture": "amd64",
}
}
}
]
}
}
Proposed Solution:
- name: "Fetch all CPU nodes from clusters using K8s beta.kubernetes.io/instance-type"
k8s_facts:
kind: Node
label_selectors:
- "beta.kubernetes.io/instance-type={{ kube_cpu_node_class }}"
verify_ssl: no
register: cpu_class_list
failed_when: not cpu_class_list.resources
#above to fail when none of the nodes has label, that is resources list is empty.
#below to fail when any of the nodes has no label
- debug:
msg: "{{ item.metadata.labels.nodeType }}"
loop: "{{ cpu_class_list.resources }}"
loop_control:
label: "{{ item.metadata.name }}"
failed_when: not item.metadata.labels.nodeType
I would propose to split it up into two distinct tasks. First, register the variable, then use the fail ansible module (docs) to inspect the var and fail if a condition is met.
See this snippet for outlining the logic:
- hosts: localhost
vars:
test: # test array
- fail: false
- fail: false
- fail: true
- fail: false
tasks:
- name: iterate and fail
fail:
msg: "Fail as requested"
with_items: "{{ test }}"
when: item.fail
Running this, will output the following:
$ ansible-playbook failing.yml
PLAY [localhost] ***********************
TASK [Gathering Facts] *************************
ok: [localhost]
TASK [iterate and fail] **************************
skipping: [localhost] => (item={u'fail': False})
skipping: [localhost] => (item={u'fail': False})
failed: [localhost] (item={u'fail': True}) => {"ansible_loop_var": "item", "changed": false, "item": {"fail": true}, "msg": "Failed as requested"}
skipping: [localhost] => (item={u'fail': False})
PLAY RECAP *****************
localhost : ok=1 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
Hope this helps!
#to fail when none of the nodes has label, that is resources list is empty.
- name: "Fetch all CPU nodes from clusters using K8s beta.kubernetes.io/instance-type"
k8s_facts:
kind: Node
label_selectors:
- "beta.kubernetes.io/instance-type={{ kube_cpu_node_class }}"
verify_ssl: no
register: cpu_class_list
failed_when: not cpu_class_list.resources
#below to fail when any of the nodes has no label
- debug:
msg: "{{ item.metadata.labels.nodeType }}"
loop: "{{ cpu_class_list.resources }}"
loop_control:
label: "{{ item.metadata.name }}"
failed_when: not item.metadata.labels.nodeType

I'm trying find datasore of vm guest with ansible

I'm trying to extract value of vm datastore with ansible snippet:
- name: Gather disk facts from virtual machine using name
vmware_guest_disk_facts:
hostname: "{{ vcenter_server }}"
username: "{{ vcenter_user }}"
password: "{{ vcenter_pass }}"
datacenter: "{{ datacenter }}"
validate_certs: no
name: "{{ item }}"
delegate_to: localhost
register: disk_facts
with_items: "{{ vm_list }}"
- name: Get disk info
debug:
var: disk_facts.results.guest_disk_facts.backing_datastore
Output:
[root#sysmgttl1 lvm]# ansible-playbook -i hosts vmfacts.yml -e vcenter_server=vmimgtpw002 -e vm_list=sndprfql8
PLAY [sndprfql8] ***********************************************************************************************************************************************************
TASK [Gathering Facts] *****************************************************************************************************************************************************
ok: [sndprfql8]
TASK [Gather disk facts from virtual machine using name] *******************************************************************************************************************
ok: [sndprfql8 -> localhost] => (item=sndprfql8)
TASK [Get disk info] *******************************************************************************************************************************************************
ok: [sndprfql8] => {
"disk_facts.results.guest_disk_facts.backing_datastore": "VARIABLE IS NOT DEFINED!"
}
PLAY RECAP *****************************************************************************************************************************************************************
sndprfql8 : ok=3 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
This is what my disk_facts looks like.
ok: [sndprfql8] => {
"disk_facts": {
"changed": false,
"msg": "All items completed",
"results": [
{
"ansible_loop_var": "item",
"changed": false,
"failed": false,
"guest_disk_facts": {
"0": {
"backing_datastore": "QADEV07",
"backing_eagerlyscrub": false,
"backing_filename": "[QADEV07] sndprfql8/sndprfql8.vmdk",
"backing_thinprovisioned": true,
"backing_type": "FlatVer2",
"backing_uuid": "6000C292-7716-6296-de04-69bac9186661",
"backing_writethrough": false,
"capacity_in_bytes": 68719476736,
"capacity_in_kb": 67108864,
"controller_key": 1000,
"key": 2000,
"label": "Hard disk 1",
"summary": "67,108,864 KB",
"unit_number": 0
}
},
"invocation": {
"module_args": {
"datacenter": "RDC",
"folder": null,
"hostname": "vmimgtpw002",
"name": "sndprfql8",
"password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
"port": 443,
"use_instance_uuid": false,
"username": "svcvread#chop.edu",
"uuid": null,
"validate_certs": false
}
},
"item": "sndprfql8"
}
]
}
}
If this is really for a one shot use, the following should do:
- name: Get disk info
debug:
var: disk_facts.results[0].guest_disk_facts['0'].backing_datastore
If you goal is to get all values for a list of vms in the result possibly having several disks, the following will give you a flattened list (that you can de-duplicate with the unique filter if needed)
- debug:
msg: "{{ disk_facts | json_query('results[].guest_disk_facts.*[].backing_datastore') }}"

Print only one item in list in ansible

I have a variable that contains a list
- debug:
var: plugin_versions
Output
ok: [localhost] => {
"plugin_versions": [
{
"name": "ace-editor",
"version": "1.1"
},
{
"name": "analysis-core",
"version": "1.95"
},
{
"name": "ant",
"version": "1.9"
}
]
Now I want to print out only the name
What Ive tried is
- debug:
var: plugin_versions.name
- debug:
var: plugin_versions[name]
But in both cases I get
TASK [plugins : debug] ***************************************************************************************************************
ok: [localhost] => {
"plugin_versions.name": "VARIABLE IS NOT DEFINED!"
}
TASK [plugins : debug] ***************************************************************************************************************
ok: [localhost] => {
"plugin_versions[name]": "VARIABLE IS NOT DEFINED!"
}
Im kinda clueless what else can I do here to print out the name only.
you could do it in a few ways. The plugin_versions is a list of dictionaries, you can print the name attribute of each dictionary using loops, here are 2 loop examples you could use:
---
- hosts: localhost
gather_facts: false
vars:
plugin_versions:
- name: ace-editor
version: '1.1'
- name: analysis-core
version: '1.95'
- name: ant
version: '1.9'
tasks:
- name: print variable - with_items
debug:
msg: "{{ item.name }}"
with_items:
- "{{ plugin_versions }}"
- name: print variable - with map filter
debug:
var: item
with_items:
- "{{ plugin_versions | map(attribute='name') | list }}"
output:
[http_offline#greenhat-29 tests]$ ansible-playbook -i hosts test.yml
PLAY [localhost] *******************************************************************************************************************************************************************************************************
TASK [print variable - with_items] *************************************************************************************************************************************************************************************
ok: [localhost] => (item={'name': 'ace-editor', 'version': '1.1'}) => {
"msg": "ace-editor"
}
ok: [localhost] => (item={'name': 'analysis-core', 'version': '1.95'}) => {
"msg": "analysis-core"
}
ok: [localhost] => (item={'name': 'ant', 'version': '1.9'}) => {
"msg": "ant"
}
TASK [print variable - with map filter] ********************************************************************************************************************************************************************************
ok: [localhost] => (item=ace-editor) => {
"item": "ace-editor"
}
ok: [localhost] => (item=analysis-core) => {
"item": "analysis-core"
}
ok: [localhost] => (item=ant) => {
"item": "ant"
}
PLAY RECAP *************************************************************************************************************************************************************************************************************
localhost : ok=2 changed=0 unreachable=0 failed=0
[http_offline#greenhat-29 tests]$
hope it helps
Since you have a list (plugin_versions) you'll need to access each element by index:
---
- hosts: all
vars:
plugin_versions:
- { name: "ace-editor", version: "1.1"}
- { name: "analysis-core", version: "1.95"}
- { name: "ant", version: "1.9"}
tasks:
- debug: var=plugin_versions[0].name
also, if you want to loop through all the items in the list you can do something like this:
- debug: "var=item['name']"
with_items: "{{ plugin_versions }}"
You can iterate over the items of your variable with:
- debug:
var: item.name
with_items: "{{plugin_versions}}"

How to loop over this dictionary in Ansible?

Say I have this dictionary
war_files:
server1:
- file1.war
- file2.war
server2:
- file1.war
- file2.war
- file3.war
and for now I just want to loop over each item (key), and then over each item in the key (value). I did this
- name: Loop over the dictionary
debug: msg="Key={{ item.key }} value={{ item.value }}"
with_dict: "{{ war_files }}"
And I get this. It is of course correct, but is NOT what I want.
ok: [localhost] => (item={'value': [u'file1.war', u'file2.war'], 'key': u'server1'}) => {
"item": {
"key": "server1",
"value": [
"file1.war",
"file2.war"
]
},
"msg": "Server=server1, WAR=[u'file1.war', u'file2.war']"
}
ok: [localhost] => (item={'value': [u'file1.war', u'file2.war', u'file3.war'], 'key': u'server2'}) => {
"item": {
"key": "server2",
"value": [
"file1.war",
"file2.war",
"file3.war"
]
},
"msg": "Server=server2, WAR=[u'file1.war', u'file2.war', u'file3.war']"
}
I want to get an output that says
"msg": "Server=server1, WAR=file1.war"
"msg": "Server=server1, WAR=file2.war"
"msg": "Server=server2, WAR=file1.war"
"msg": "Server=server2, WAR=file2.war"
"msg": "Server=server2, WAR=file3.war"
IOW, how can I write a task to iterates over the dictionary so it goes through each key, and then the items within each key? In essence, I have a nested array and want to iterate over it?
Hows this
- hosts: localhost
vars:
war_files:
server1:
- file1.war
- file2.war
server2:
- file1.war
- file2.war
- file3.war
tasks:
- name: Loop over subelements of the dictionary
debug:
msg: "Key={{ item.0.key }} value={{ item.1 }}"
loop: "{{ war_files | dict2items | subelements('value') }}"
dict2items, subelements filters are coming in Ansible 2.6.
FYI, if a filter for your objective doesn't exist, you can write your own in python without having to resort to jinja2 hacks. Ansible is easily extendable; filters in filter_plugins/*.py are searched by default adjacent to your plays/roles and are automatically included - see Developing Plugins for details.
Now Ansible allows this
- name: add several users
user:
name: "{{ item.name }}"
state: present
groups: "{{ item.groups }}"
with_items:
- { name: 'testuser1', groups: 'wheel' }
- { name: 'testuser2', groups: 'root' }
EDIT: At the time of writing this answer, Ansible 2.6 wasn't out. Please read the answer provided by #tmoschou, as it is much better.
Well, I couldn't find a very easy way to do it, however, with a little bit of jinja2, we can achieve something of this sort:
/tmp ❯❯❯ cat example.yml
---
- hosts: 127.0.0.1
vars:
war_files:
server1:
- file1.war
- file2.war
server2:
- file1.war
- file2.war
- file3.war
tasks:
- set_fact:
war_files_list_of_dicts: |
{% set res = [] -%}
{% for key in war_files.keys() -%}
{% for value in war_files[key] -%}
{% set ignored = res.extend([{'Server': key, 'WAR':value}]) -%}
{%- endfor %}
{%- endfor %}
{{ res }}
- name: let's debug the crap out of this
debug: var=war_files_list_of_dicts
- name: Servers and their WARs!!!
debug:
msg: "Server={{ item.Server }}, WAR={{ item.WAR }}"
with_items: "{{ war_files_list_of_dicts }}"
And, when the playbook is run:
/tmp ❯❯❯ ansible-playbook example.yml
[WARNING]: provided hosts list is empty, only localhost is available
PLAY [127.0.0.1] ***************************************************************
TASK [setup] *******************************************************************
ok: [127.0.0.1]
TASK [set_fact] ****************************************************************
ok: [127.0.0.1]
TASK [let's debug the crap out of this] ****************************************
ok: [127.0.0.1] => {
"war_files_list_of_dicts": [
{
"Server": "server1",
"WAR": "file1.war"
},
{
"Server": "server1",
"WAR": "file2.war"
},
{
"Server": "server2",
"WAR": "file1.war"
},
{
"Server": "server2",
"WAR": "file2.war"
},
{
"Server": "server2",
"WAR": "file3.war"
}
]
}
TASK [Servers and their WARs!!!] ***********************************************
ok: [127.0.0.1] => (item={'WAR': u'file1.war', 'Server': u'server1'}) => {
"item": {
"Server": "server1",
"WAR": "file1.war"
},
"msg": "Server=server1, WAR=file1.war"
}
ok: [127.0.0.1] => (item={'WAR': u'file2.war', 'Server': u'server1'}) => {
"item": {
"Server": "server1",
"WAR": "file2.war"
},
"msg": "Server=server1, WAR=file2.war"
}
ok: [127.0.0.1] => (item={'WAR': u'file1.war', 'Server': u'server2'}) => {
"item": {
"Server": "server2",
"WAR": "file1.war"
},
"msg": "Server=server2, WAR=file1.war"
}
ok: [127.0.0.1] => (item={'WAR': u'file2.war', 'Server': u'server2'}) => {
"item": {
"Server": "server2",
"WAR": "file2.war"
},
"msg": "Server=server2, WAR=file2.war"
}
ok: [127.0.0.1] => (item={'WAR': u'file3.war', 'Server': u'server2'}) => {
"item": {
"Server": "server2",
"WAR": "file3.war"
},
"msg": "Server=server2, WAR=file3.war"
}
PLAY RECAP *********************************************************************
127.0.0.1 : ok=4 changed=0 unreachable=0 failed=0
Here is my preferred way to loop over dictionaries:
input_data.yml contains the following:
----
input_data:
item_1:
id: 1
info: "Info field number 1"
item_2:
id: 2
info: "Info field number 2"
I then use a data structure like the above in a play using the keys() function and iterate over the data using with_items:
---
- hosts: localhost
gather_facts: false
connection: local
tasks:
- name: Include dictionary data
include_vars:
file: data.yml
- name: Show info field from data.yml
debug:
msg: "Id: {{ input_data[item]['id'] }} - info: {{ input_data[item]['info'] }}"
with_items: "{{ input_data.keys() | list }}"
The above playbook produces the following output:
PLAY [localhost] ***********************************************************
TASK [Include dictionary data] *********************************************
ok: [localhost]
TASK [Show info field from data.yml] ***************************************
ok: [localhost] => (item=item_2) => {
"msg": "Id: 2 - info: Info field item 2"
}
ok: [localhost] => (item=item_3) => {
"msg": "Id: 3 - info: Info field item 3"
}
ok: [localhost] => (item=item_1) => {
"msg": "Id: 1 - info: Info field item 1"
}
PLAY RECAP *****************************************************************
localhost : ok=2 changed=0 unreachable=0 failed=0
dict2items
I found myself wanting to iterate over a heterogeneous set of keys and their associated values and use the key-value pair in a task. The dict2items filter is the least painful way I've found. You can find dict2items in Ansible 2.6
Example Dict
systemsetup:
remotelogin: "On"
timezone: "Europe/Oslo"
usingnetworktime: "On"
sleep: 0
computersleep: 0
displaysleep: 0
harddisksleep: 0
allowpowerbuttontosleepcomputer: "Off"
wakeonnetworkaccess: "On"
restartfreeze: "On"
restartpowerfailure: "On"
Example Task
---
- debug:
msg: "KEY: {{ item.key }}, VALUE: {{ item.value }}"
loop: "{{ systemsetup | dict2items }}"
One way of doing it that worked for me was using with_dict. Note the dict should not be named. Just the key value pairs.
- name: ssh config
lineinfile:
dest: /etc/ssh/sshd_config
regexp: '^#?\s*{{item.key}}\s'
line: '{{item.key}} {{item.value}}'
state: present
with_dict:
LoginGraceTime: "1m"
PermitRootLogin: "yes"
PubkeyAuthentication: "yes"
PasswordAuthentication: "no"
PermitEmptyPasswords: "no"
IgnoreRhosts: "yes"
Protocol: 2

Resources