Loops within loops - ansible

I've set up some application information in my Ansible group_vars like this:
applications:
- name: app1
- name: app2
- name: app3
- name: app4
settings:
log_dir: /var/logs/app4
associated_files:
- auth/key.json
- name: app5
settings:
log_dir: /var/logs/app5
repo_path: new_apps/app5
I'm struggling to get my head around how I can use these "sub loops".
My tasks for each application are:
Create some folders based purely on the name value
Create a log folder if a settings/log_dir value exists
Copy associated files over, if specified
The syntax for these tasks isn't the problem here, I'm comfortable with those - I just need to know how to access the information from this applications variable. Number 3 in particular seems troublesome to me - I need to loop within a loop.
To debug this, I've been trying to run the following task:
- debug:
msg: "{{ item }}"
with_subelements:
- "{{ applications }}"
- settings
Here's the output:
with_items: I get the error with_items expects a list or a set
with_nested: I can see the top level information (e.g. msg: {{ item }} outputs an array of app1, app2 etc)
with_subelements: I get the error subelements lookup expects a dictionary, got 'None'
It's possible/probable that the way I've set the variable up in the first instance is wrong. If there's a better way to do this, it's not a problem to change it.

You can't use with_subelements because settings is a dictionary, not a list. If you were to restructure your data so that settings is a list, like this:
applications:
- name: app1
- name: app2
- name: app3
- name: app4
settings:
- name: log_dir
value: /var/logs/app4
- name: associated_files
value:
- auth/key.json
- name: app5
settings:
- name: log_dir
value: /var/logs/app5
- name: repo_path
value: new_apps/app5
You could then write something like the following to iterate over each setting for each application:
---
- hosts: localhost
gather_facts: false
vars_files:
- applications.yml
tasks:
- debug:
msg: "set {{ item.1.name }} to {{ item.1.value }} for {{ item.0.name }}"
loop: "{{ applications|subelements('settings', skip_missing=true) }}"
loop_control:
label: "{{ item.0.name }}.{{ item.1.name }} = {{ item.1.value }}"
(I'm using loop_control here just to make the output nicer.)
Using the sample data you posted in applications.yml, this will produce as output:
PLAY [localhost] *********************************************************************
TASK [debug] *************************************************************************
ok: [localhost] => (item=app4.log_dir = /var/logs/app4) => {
"msg": "set log_dir to /var/logs/app4 for app4"
}
ok: [localhost] => (item=app4.associated_files = ['auth/key.json']) => {
"msg": "set associated_files to ['auth/key.json'] for app4"
}
ok: [localhost] => (item=app5.log_dir = /var/logs/app5) => {
"msg": "set log_dir to /var/logs/app5 for app5"
}
ok: [localhost] => (item=app5.repo_path = new_apps/app5) => {
"msg": "set repo_path to new_apps/app5 for app5"
}
PLAY RECAP ***************************************************************************
localhost : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

Related

Items are overwritten in Task

We're trying to fill an Palo Alto Object Group with multiple objects.
Our current script does run through all the items in the following line:
static_value: "{{ item[1].addressobjectname|join }}"
But only registers the last object.
For Example:
We have 3 objects in the list. So we expect to add object 1, 2, 3.
Instead the script only registers the 3 and 1 and 2 are overwritten in the process.
How should we fix this?
This is our code:
- name: Create object group
with_nested:
- "{{ tag_firewall# }}"
- "{{ addressobjects }}"
panos_address_group:
provider: "{{ palo_provider }}"
name: "Prisma-Unsecure"
static_value: "{{ item[1].addressobjectname|join }}"
tag: ["ansible_test_tag"]
device_group: "{{ item[0] }}"
description: "Created by ansible automation"
commit: 'no'
Thanks in advance!
As mentioned before we expect all items to be added to the group in stead of the last one in the list.
Hard to say without the precision asked by Vladimir in comments. Meanwhile it looks like you are creating an address group for each addressojects for each firewall definition. My guess at this stage is that the last one wins with the latest address.
Guess again: I believe you want to create one group with all adresses, something like the following (to be tested and adapted with your actual input data):
- name: Create object group
loop: "{{ tag_firewall# }}"
panos_address_group:
provider: "{{ palo_provider }}"
name: "Prisma-Unsecure"
static_value: "{{ addressobjects | map(attribute='addressobjectname') }}"
tag: ["ansible_test_tag"]
device_group: "{{ item }}"
description: "Created by ansible automation"
commit: 'no'
My guess is to remove the join filter. For example,
shell> cat pb.yml
- hosts: localhost
vars:
tag_firewall: [Prod1, Prod2]
addressobjects:
- addressobjectname: [Test-One, Test-Three]
- addressobjectname: [Test-Two, Test-Four]
tasks:
- debug:
msg: |
panos_address_group:
provider: Palo_provider_dict
name: Prisma-Unsecure
static_value: "{{ item.1.addressobjectname }}"
tag: Ansible_test_tag
device_group: "{{ item.0 }}"
description: Created by Ansible automation
commit: false
with_nested:
- "{{ tag_firewall }}"
- "{{ addressobjects }}"
shows the iteration
shell> ansible-playbook pb.yml
PLAY [localhost] *****************************************************************************
TASK [debug] *********************************************************************************
ok: [localhost] => (item=['Prod1', {'addressobjectname': ['Test-One', 'Test-Three']}]) =>
msg: |-
panos_address_group:
provider: Palo_provider_dict
name: Prisma-Unsecure
static_value: "['Test-One', 'Test-Three']"
tag: Ansible_test_tag
device_group: "Prod1"
description: Created by Ansible automation
commit: false
ok: [localhost] => (item=['Prod1', {'addressobjectname': ['Test-Two', 'Test-Four']}]) =>
msg: |-
panos_address_group:
provider: Palo_provider_dict
name: Prisma-Unsecure
static_value: "['Test-Two', 'Test-Four']"
tag: Ansible_test_tag
device_group: "Prod1"
description: Created by Ansible automation
commit: false
ok: [localhost] => (item=['Prod2', {'addressobjectname': ['Test-One', 'Test-Three']}]) =>
msg: |-
panos_address_group:
provider: Palo_provider_dict
name: Prisma-Unsecure
static_value: "['Test-One', 'Test-Three']"
tag: Ansible_test_tag
device_group: "Prod2"
description: Created by Ansible automation
commit: false
ok: [localhost] => (item=['Prod2', {'addressobjectname': ['Test-Two', 'Test-Four']}]) =>
msg: |-
panos_address_group:
provider: Palo_provider_dict
name: Prisma-Unsecure
static_value: "['Test-Two', 'Test-Four']"
tag: Ansible_test_tag
device_group: "Prod2"
description: Created by Ansible automation
commit: false
PLAY RECAP ***********************************************************************************
localhost: ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

Parse yaml files in Ansible

I have got multiple yaml files on remote machine. I would like to parse those files in order to get information about names for each kind (Deployment, Configmap, Secret) of object, For example:
...
kind: Deployment
metadata:
name: pr1-dep
...
kind: Secret
metadata:
name: pr1
...
....
kind: ConfigMap
metadata:
name: cm-pr1
....
Ecpected result:
3 variables:
deployments = [pr1-dep]
secrets = [pr1]
configmaps = [cm-pr1]
I started with:
- shell: cat "{{ item.desc }}"
with_items:
- "{{ templating_register.results }}"
register: objs
but i have no idea how to correctly parse item.stdout from objs
Ansible has a from_yaml filter that takes YAML text as input and outputs an Ansible data structure. So for example you can write something like this:
- hosts: localhost
gather_facts: false
tasks:
- name: Read objects
command: "cat {{ item }}"
register: objs
loop:
- deployment.yaml
- configmap.yaml
- secret.yaml
- debug:
msg:
- "kind: {{ obj.kind }}"
- "name: {{ obj.metadata.name }}"
vars:
obj: "{{ item.stdout | from_yaml }}"
loop: "{{ objs.results }}"
loop_control:
label: "{{ item.item }}"
Given your example files, this playbook would output:
PLAY [localhost] ***************************************************************
TASK [Read objects] ************************************************************
changed: [localhost] => (item=deployment.yaml)
changed: [localhost] => (item=configmap.yaml)
changed: [localhost] => (item=secret.yaml)
TASK [debug] *******************************************************************
ok: [localhost] => (item=deployment.yaml) => {
"msg": [
"kind: Deployment",
"name: pr1-dep"
]
}
ok: [localhost] => (item=configmap.yaml) => {
"msg": [
"kind: ConfigMap",
"name: pr1-cm"
]
}
ok: [localhost] => (item=secret.yaml) => {
"msg": [
"kind: Secret",
"name: pr1"
]
}
PLAY RECAP *********************************************************************
localhost : ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
Creating the variables you've asked for is a little trickier. Here's
one option:
- hosts: localhost
gather_facts: false
tasks:
- name: Read objects
command: "cat {{ item }}"
register: objs
loop:
- deployment.yaml
- configmap.yaml
- secret.yaml
- name: Create variables
set_fact:
names: >-
{{
names|combine({
obj.kind.lower(): [obj.metadata.name]
}, list_merge='append')
}}
vars:
names: {}
obj: "{{ item.stdout | from_yaml }}"
loop: "{{ objs.results }}"
loop_control:
label: "{{ item.item }}"
- debug:
var: names
This creates a single variable named names that at the end of the
playbook will contain:
{
"configmap": [
"pr1-cm"
],
"deployment": [
"pr1-dep"
],
"secret": [
"pr1"
]
}
The key to the above playbook is our use of the combine filter, which can be used to merge dictionaries and, when we add list_merge='append', handles keys that resolve to lists by appending to the existing list, rather than replacing the existing key.
Include the dictionaries from the files into the new variables, e.g.
- include_vars:
file: "{{ item }}"
name: "objs_{{ item|splitext|first }}"
register: result
loop:
- deployment.yaml
- configmap.yaml
- secret.yaml
This will create dictionaries objs_deployment, objs_configmap, and objs_secret. Next, you can either use the dictionaries
- set_fact:
objs: "{{ objs|d({})|combine({_key: _val}) }}"
loop: "{{ query('varnames', 'objs_') }}"
vars:
_obj: "{{ lookup('vars', item) }}"
_key: "{{ _obj.kind }}"
_val: "{{ _obj.metadata.name }}"
, or the registered data
- set_fact:
objs: "{{ dict(_keys|zip(_vals)) }}"
vars:
_query1: '[].ansible_facts.*.kind'
_keys: "{{ result.results|json_query(_query1)|flatten }}"
_query2: '[].ansible_facts.*.metadata[].name'
_vals: "{{ result.results|json_query(_query2)|flatten }}"
Both options give
objs:
ConfigMap: cm-pr1
Deployment: pr1-dep
Secret: pr1

Errors when use "json_query" in Ansible

When use json_query in Ansible, got the error along with my code as shown below.
The Python used in Ansible is 3.6.8, and used the same version installed jmespath: pip install jmespath, so, this should not be the issue. The Ansible code should be fine as well.
fatal: [localhost]: FAILED! => {"msg": "template error while templating string: no filter named 'json_query'. String: {{ jsondata | json_query(jmesquery) }}"
The following is the Ansible codes:
---
- name: ReadJsonfile
hosts: localhost
tasks:
- name: Display the JSON file content
shell: cat config.json
register: result
- name: save the JSON data to a Variable as a Fact
set_fact:
jsondata: "{{ result.stdout | from_json }}"
- name: setDomainName
set_fact:
domain_name: "{{ jsondata | json_query(jmesquery) }}"
vars:
jmesquery: 'domain.name'
- name: setDomainUsername
set_fact:
domain_username: "{{ jsondata | json_query(jmesquery) }}"
vars:
jmesquery: 'domain.user'
- name: setDomainPassword
set_fact:
domain_password: "{{ jsondata | json_query(jmesquery) }}"
vars:
jmesquery: 'domain.password'
- name: setadmin_Listenport
set_fact:
admin_ListenPort: "{{ jsondata | json_query(jmesquery) }}"
vars:
jmesquery: 'domain.admin.listenport'
- name: Debug the values
debug: msg=" Admin Listen Port => {{ admin_ListenPort }}, DomainName => {{ domain_name }}, DomainUserName => {{ domain_username }} , Domain Password => {{ domain_password }}"
From the error message, I suspect you are using ansible 2.10
The json_query filter is part of the community.general collection which needs to be installed separately starting from this version (this is clearly stated in the documentation)
ansible-galaxy collection install community.general
That being said:
your actual jmespath queries will not return what you expect
using shell to get a file content from remote is a bad practice
that's lots of set_facts and json_query to simply get data that is readily available
Since yaml is a strict superset of json, any valid json is also a valid yaml. In your case, you only have to load the content of the file and you have the data.
One solution is to fetch the file locally and then use include_vars.
In this particular case, using slurp looks like the best option.
This is what (I believe, since you did not provide an example) your config file looks like. I pushed that file in /tmp/config.json for my example.
{
"domain": {
"name": "toto",
"user": "doejohn",
"password": "sosecret",
"admin": {
"listenport": 5501
}
}
}
This is the demo playbook
---
- name: Load remote json content demo
hosts: localhost
gather_facts: false
vars:
config_file: /tmp/config.json
# Of course the below vars will fire errors
# if you call them before slurping data from remote.
# Note: loading `data` in a jinja expression on its own
# forces to transform it back rom string to json data.
# You can accomplish the same result in one go
# using the `from_json` filter
# i.e. => domain: "{{ (slurped_config.content | b64decode | from_json).domain }}"
data: "{{ slurped_config.content | b64decode }}"
domain: "{{ data.domain }}"
# Note2: the above will work and adapt to N hosts in your play: you will get
# the correct data for each host in every task.
tasks:
- name: Slurp config file content
slurp:
src: "{{ config_file }}"
register: slurped_config
- name: Show result
vars:
message: |-
name is: {{ domain.name }}
user is: {{ domain.user }}
password is: {{ domain.password }}
port is: {{ domain.admin.listenport }}
debug:
msg: "{{ message.split('\n') }}"
which gives:
PLAY [Load remote json content demo] ******************************************************
TASK [Slurp config file content] **********************************************************
Wednesday 21 April 2021 17:57:47 +0200 (0:00:00.010) 0:00:00.010 *******
ok: [localhost]
TASK [Show result] ************************************************************************
Wednesday 21 April 2021 17:57:48 +0200 (0:00:00.199) 0:00:00.209 *******
ok: [localhost] => {
"msg": [
"name is: toto",
"user is: doejohn",
"password is: sosecret",
"port is: 5501"
]
}
PLAY RECAP ********************************************************************************
localhost : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
Wednesday 21 April 2021 17:57:48 +0200 (0:00:00.027) 0:00:00.237 *******
===============================================================================
Slurp config file content ---------------------------------------------------------- 0.20s
Show result ------------------------------------------------------------------------ 0.03s

How to split a string into a list with Ansible/Jinja2?

I have the following variables:
domain_names:
- app1.example.com
- app2.example.com
customers:
- name: customer1
- name: customer2
And I'm trying to generate the following list of domain names:
- customer1.app1.example.com
- customer2.app1.example.com
- customer1.app2.example.com
- customer2.app2.example.com
By using the following Ansible/Jinja2 code:
- name: check which certificates exist
stat:
path: '/etc/nginx/{{item}}.crt'
register: cert_file
loop: '{% for d in domain_names %}{{ d }} {% for customer in customers %}{{ customer.name }}.{{ d }} {% endfor %}{% endfor %}'
However, I'm getting the following error:
failed | msg: Invalid data passed to 'loop', it requires a list, got this instead: customer1.app1.example.com customer2.app1.example.com customer1.app2.example.com customer2.app2.example.com. Hint: If you passed a list/dict of just one element, try adding wantlist=True to your lookup invocation or use q/query instead of lookup.
How can I fix this?
Simply use the right tools :). In this case your friends are:
the map filter to extract the name attribute as a list from your customers variable
the product filter to mix-up your two lists
e.g. test.yml playbook:
---
- name: product and map filters demo
hosts: localhost
gather_facts: false
vars:
domain_names:
- app1.example.com
- app2.example.com
customers:
- name: customer1
- name: customer2
tasks:
- name: Demonstrate product and map filters use
debug:
msg: "{{ item.0 }}.{{ item.1 }}"
loop: "{{ customers | map(attribute='name') | product(domain_names) | list }}"
Which gives:
$ ansible-playbook test.yml
PLAY [product and map filters demo] *******************************************************************************************************************************************************************************
TASK [Demonstrate product and map filters use] ********************************************************************************************************************************************************************
ok: [localhost] => (item=['customer1', 'app1.example.com']) => {
"msg": "customer1.app1.example.com"
}
ok: [localhost] => (item=['customer1', 'app2.example.com']) => {
"msg": "customer1.app2.example.com"
}
ok: [localhost] => (item=['customer2', 'app1.example.com']) => {
"msg": "customer2.app1.example.com"
}
ok: [localhost] => (item=['customer2', 'app2.example.com']) => {
"msg": "customer2.app2.example.com"
}
PLAY RECAP ********************************************************************************************************************************************************************************************************
localhost : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
Applied to your task, this gives:
- name: Check which certificates exist
stat:
path: '/etc/nginx/{{ item.0 }}.{{ item.1 }}.crt'
register: cert_file
loop: "{{ customers | map(attribute='name') | product(domain_names) | list }}"
If you really want to re-use that list you can build it into an other var. One easy way to understand is to build it in a set_fact task, e.g.
- name: Create my application names list
vars:
current_name: "{{ item.0 }}.{{ item.1 }}"
set_fact:
application_names_list: "{{ application_names_list | default([]) + [current_name] }}"
loop: "{{ customers | map(attribute='name') | product(domain_names) | list }}"
- name: Check which certificates exist
stat:
path: '/etc/nginx/{{ item }}.crt'
register: cert_file
loop: "{{ application_names_list }}"
But you can also declare it "statically" in your vars with a little more complex expression (see the map filter possibilities and the join filter)
---
- name: product and map filters demo
hosts: localhost
gather_facts: false
vars:
domain_names:
- app1.example.com
- app2.example.com
customers:
- name: customer1
- name: customer2
application_names_list: "{{ customers | map(attribute='name') | product(domain_names) | map('join', '.') | list }}"
tasks:
- name: Demonstrate product and map filters use
debug:
var: application_names_list
=>
PLAY [product and map filters demo] ****************************************************************************************************************************************************************************************************
TASK [Demonstrate product and map filters use] *****************************************************************************************************************************************************************************************
ok: [localhost] => {
"all_domains": [
"customer1.app1.example.com",
"customer1.app2.example.com",
"customer2.app1.example.com",
"customer2.app2.example.com"
]
}
PLAY RECAP *****************************************************************************************************************************************************************************************************************************
localhost : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

Create VLANs only if they don't exist on a Nexus switch

I'm trying to create an Ansible playbook which should create VLANs defined in file vlans.dat on a Cisco Nexus switch only when they don't exist on device.
File vlans.dat contains:
---
vlans:
- { vlan_id: 2, name: TEST }
And Ansible file:
---
- name: Verify and create VLANs
hosts: switches_group
gather_facts: no
vars_files:
- vlans.dat
tasks:
- name: Get Nexus facts
nxos_facts:
register: data
- name: Create new VLANs only
nxos_vlan:
vlan_id: "{{ item.vlan_id }}"
name: "{{ item.name }}"
state: "{{item.state | default('present') }}"
with_items: "{{ vlans }}"
when: item.vlan_id not in data.ansible_facts.vlan_list
In the when statement I'm trying to limit execution only to the case when vlan_id (defined in the file) doesn't exist in the vlan_list gathered by nxos_facts module. Unfortunately it gets executed even when the vlan_id already exists in the vlan_list and I don't know why?
PLAY [Verify and create VLANs]
TASK [Get Nexus facts]
ok: [10.1.1.1]
TASK [Create new VLANs only]
ok: [10.1.1.1] => (item={u'name': u'TEST', u'vlan_id': 2})
TASK [debug]
skipping: [10.1.1.1]
PLAY RECAP
10.1.1.1 : ok=2 changed=0 unreachable=0 failed=0
Can you help me with that or provide some solution what I'm doing wrong here?
It appears you have stumbled upon a side-effect of YAML having actual types. Because in {vlan_id: 2} the 2 is an int but the list is strings. As you might imagine {{ 1 in ["1"] }} is False.
There are two ways out of that situation: make the vlan_id a string via - { vlan_id: "2" } or coerce the vlan_id to a string just for testing the list membership:
when: (item.vlan_id|string) not in data.ansible_facts.vlan_list

Resources