How to loop through similar vars files in an Ansible playbook - ansible

I need to create a single ansible playbook that will loop through multiple vars files in a single directory and use each configuration individually in a rest API POST.
Ideally, with the following vars files...
/my/vars/dir
- my_pen.yml
pen:
color: "blue"
- her_pen.yml
pen:
color: "red"
- his_pen.yml
pen:
color: "green"
...my playbook would execute a POST for each pen. Unfortunately, all files contain a configuration for the same object type so an include_vars task would only retain the configuration for "his_pen".
I've been able to get a list of all files in the directory using find:
- name: "find config files"
find:
paths: /path/to/config/files
patterns: '*.yml'
register: config_files
I have a task that can do the POST:
- name: Do post
uri:
url: "{{ rest_api_endpoint }}"
method: POST
user: "{{ username }}"
password: "{{ password }}"
body: "{{ pen }}"
body_format: json
return_content: yes
status_code: 200
register: post_result
Now I just need to fuse the two. Is this possible? I can't change the file structure, so I have to use what's in place.
Thoughts?

Let's create the list of pens first and then loop the list. The play below
- set_fact:
pens: "{{ pens|default([]) + [ lookup('file', item)|from_yaml ] }}"
loop: "{{ lookup('fileglob', 'vars/*.yml', wantlist=True) }}"
- debug:
msg: "{{ item }}"
loop: "{{ pens }}"
gives (abridged):
ok: [localhost] => (item={'pen': {'color': u'blue'}}) => {
"msg": {
"pen": {
"color": "blue"
}
}
}
ok: [localhost] => (item={'pen': {'color': u'green'}}) => {
"msg": {
"pen": {
"color": "green"
}
}
}
ok: [localhost] => (item={'pen': {'color': u'red'}}) => {
"msg": {
"pen": {
"color": "red"
}
}
}

Related

Create Dictionary from Ansible inventory file

I am trying to create a dictionary with below ansible inventory file.
"myfile": {
"all": {
"hosts": null,
"vars": {
"ansible_connection": "local",
"ansible_python_interpreter": "{{ ansible_playbook_python }}"
}
},
"USA": {
"children": {
"MYDC1": {
"children": {
"MYDC1-CLU01": {
"hosts": {
"MYDC2-VM1": null,
"MYDC2-VM2": null,
"MYDC2-VM3": null,
"MYDC2-VM4": null,
"MYDC2-VM5": null,
"MYDC2-VM6": null
}
}
}
},
"MYDC2": {
"children": {
"MYDC2-CLU01": {
"hosts": {
"MYDC1-VM1": null
"MYDC1-VM2": null
}
}
}
},
"MYDC3": {
"children": {
"MYDC3-CLU01": {
"hosts": {
"MYDC3-VM1": null
"MYDC3-VM2": null
"MYDC3-VM3": null
}
}
}
}
},
"vars": {
"vcenter_hostname": "myvcenter.esxihost.com"
}
}
}
}
From this file, expected output is:
{
'MYDC1-CLU01': ['MYDC1-VM1.myhost.com','MYDC1-VM2.myhost.com'],
'MYDC2-CLU01':['MYDC2-VM1.myhost.com','MYDC2-VM2.myhost.com','MYDC2-VM3.myhost.com','MYDC2-VM4.myhost.com','MYDC2-VM5.myhost.com','MYDC2-VM6.myhost.com'],
'MYDC3-CLU01':['MYDC3-VM1.myhost.com','MYDC3-VM2.myhost.com','MYDC3-VM3.myhost.com'],
}
The code given below works fine:
my_domain: myhost.com
my_groups: [MYDC1-CLU01, MYDC2-CLU01, MYDC3-CLU01]
cluster_dict: "{{ dict(my_groups|
zip(my_groups|
map('extract', groups)|
map('product', [my_domain])|
map('map', 'join', '.')|
list)) }}"
Output:
$ ansible-playbook main.yml -i hosts.yml
[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details
PLAY [Check Service status] ***********************************************************************************************************************************************************************************************
TASK [Gathering Facts] ****************************************************************************************************************************************************************************************************ok: [localhost]
TASK [my_checks : Show host inventory file] *************************************************************************************************************************************************************************ok: [localhost] => {
"cluster_dict": {
"MYDC1-CLU01": [
"MYDC1-VM1.myhost.com",
"MYDC1-VM2.myhost.com"
],
"MYDC2-CLU01": [
"MYDC2-VM1.myhost.com",
"MYDC2-VM2.myhost.com",
"MYDC2-VM3.myhost.com",
"MYDC2-VM4.myhost.com",
"MYDC2-VM5.myhost.com",
"MYDC2-VM6.myhost.com"
],
"MYDC3-CLU01": [
"MYDC3-VM1.myhost.com",
"MYDC3-VM2.myhost.com"
]
}
}
I have 2 questions:
1: Is there anyway I can store variable in "cluster_dict" without passing the hosts.yml file in the command line. Like converting hosts file into below
host_intent_data: "{{ lookup('file', 'hosts.yml')|from_yaml }}"
and process the host_intent_data.
The CLUSTER/DC values will be growing each day and we can't use static fields as mentioned in my_groups.Is there any way we can get the desired dictionary dynamically?
Given the inventory
shell> cat hosts
all:
hosts:
vars:
ansible_connection: local
ansible_python_interpreter: '{{ ansible_playbook_python }}'
USA:
children:
MYDC1:
children:
MYDC_CLU01:
hosts:
MYDC1-VM1:
MYDC1-VM2:
MYDC2:
children:
MYDC2_CLU01:
hosts:
MYDC2-VM1:
MYDC2-VM2:
MYDC2-VM3:
MYDC2-VM4:
MYDC2-VM5:
MYDC2-VM6:
vars:
vcenter_hostname: myvcenter.esxihost.com
and the variables
my_domain: myhost.com
my_groups: [MYDC_CLU01, MYDC2_CLU01]
Put the below declaration into the vars
my_dict: "{{ dict(my_groups|
zip(my_groups|
map('extract', groups)|
map('product', [my_domain])|
map('map', 'join', '.')|
list)) }}"
gives what you want
my_dict:
MYDC2_CLU01:
- MYDC2-VM1.myhost.com
- MYDC2-VM2.myhost.com
- MYDC2-VM3.myhost.com
- MYDC2-VM4.myhost.com
- MYDC2-VM5.myhost.com
- MYDC2-VM6.myhost.com
MYDC_CLU01:
- MYDC1-VM1.myhost.com
- MYDC1-VM2.myhost.com
Example of a complete playbook for testing
- hosts: all
gather_facts: false
vars:
my_domain: myhost.com
my_groups: [MYDC_CLU01, MYDC2_CLU01]
my_dict: "{{ dict(my_groups|
zip(my_groups|
map('extract', groups)|
map('product', [my_domain])|
map('map', 'join', '.')|
list)) }}"
tasks:
- debug:
var: my_dict
run_once: true
Q: "Convert hosts file to 'cluster_dict'."
A: Use ansible-inventory and convert the file to yaml. Select the region (e.g. USA) and create the dictionary. For example, if you provide a valid inventory file
cluster: "{{ lookup('pipe', 'ansible-inventory -i hosts.json --list --yaml')|from_yaml }}"
cluster_USA: "{{ cluster.all.children.USA.children|json_query('*.children') }}"
cluster_dict_raw: |
{% for i in cluster_USA %}
{{ i.keys()|first }}:
{{ (i.values()|first).hosts.keys() }}
{% endfor %}
cluster_dict: "{{ cluster_dict_raw|from_yaml }}"
gives
cluster_dict:
MYDC1-CLU01: [MYDC2-VM1, MYDC2-VM2, MYDC2-VM3, MYDC2-VM4, MYDC2-VM5, MYDC2-VM6]
MYDC2-CLU01: [MYDC1-VM1, MYDC1-VM2]
MYDC3-CLU01: [MYDC3-VM1, MYDC3-VM2, MYDC3-VM3]
Notes:
Test the inventory file before complaining!
shell> ansible-inventory -i hosts.json --list --yaml
The names of the groups are not valid. You must have seen the warning. Fix it before complaining!
[WARNING]: Invalid characters were found in group names ...
Fit the template to your needs. For example, add domain and create FQDN
cluster_dict_raw: |
{% for i in cluster_USA %}
{{ i.keys()|first }}:
{{ (i.values()|first).hosts.keys()|product(['myhost.com'])|
map('join', '.')|list }}
{% endfor %}

Loop with subelements filter

Variable:
customers:
- name: CompanyX
destination_addresses:
- 192.168.0.0/24
- 192.168.1.0/24
- name: CompanyY
destination_addresses:
- 192.168.2.0/24
- 192.168.3.0/24
I'm trying to create address objects for each address in destination_addresses, and create an address group object that stores all addresses per customer.
The creation of each address works as expected like this:
- name: add address object for destination networks
fortinet.fortios.fortios_firewall_address:
state: present
firewall_address:
name: "{{ item.0.name }}-{{ item.1 }}"
subnet: "{{ item.1 }}"
loop: "{{ customers | subelements('destination_addresses') }}"
This creates:
CompanyX-192.168.0.0/24
CompanyX-192.168.1.0/24
CompanyY-192.168.2.0/24
CompanyY-192.168.3.0/24
But I'm struggling how to group the address objects.
This is what I use now:
- set_fact:
grp_members: "{{ grp_members | default([]) + [{ 'name': item.0.name ~ '-' ~ item.1 }] }}"
loop: "{{ customers | subelements('destination_addresses') }}"
loop_control:
extended: yes
- name: create address group
fortinet.fortios.fortios_firewall_addrgrp:
state: present
firewall_addrgrp:
name: "{{ item.name }}"
member: "{{ grp_members }}"
loop: "{{ customers }}"
Which creates the group CompanyX and CompanyY but with all addresses in each group, because the grp_members variable contains all addresses.
How can I limit the group members to only contain the addresses for CompanyX & CompanyY separately?
Current output:
- debug:
var: grp_members
"grp_members": [
{
"name": "CompanyX-192.168.0.0/24"
},
{
"name": "CompanyX-192.168.1.0/24"
},
{
"name": "CompanyY-192.168.2.0/24"
},
{
"name": "CompanyY-192.168.3.0/24"
}
]
Desired result for each customer:
"grp_members": [
{
"name": "CompanyX-192.168.0.0/24"
},
{
"name": "CompanyX-192.168.1.0/24"
}
]
"grp_members": [
{
"name": "CompanyY-192.168.2.0/24"
},
{
"name": "CompanyY-192.168.3.0/24"
}
]
The fortinet.fortios.fortios_firewall_addrgrp module expects a dictionary in the above syntax.
With the changed conditions you need the following:
- name: create address group
fortinet.fortios.fortios_firewall_addrgrp:
state: present
firewall_addrgrp:
name: "{{ item.name }}"
member: "{{ grp_members }}"
vars:
grp_members: "{{ [item.name] | product(item.destination_addresses) | map('join', '-') | map('community.general.dict_kv', 'name') }}"
loop: "{{ customers }}"
You continue iterating over customers. The variable grp_members is generated locally for each iteration.
via product a cross product of the customer with each IP is created
via join the two elements customer name and IP are connected.
dict_kv creates a dict from the list with the key name.
You don't need your task with set_fact anymore.
Here you can see the sample output of the joined addresses.
This task
- name: generate kv dict with customer-address
debug:
msg: "{{ grp_members }}"
vars:
grp_members: "{{ [item.name] | product(item.destination_addresses) | map('join', '-') | map('community.general.dict_kv', 'name') }}"
loop: "{{ customers }}"
produces this output
TASK [generate kv dict with customer-address] *******************************************************************************
ok: [localhost] => (item={'name': 'CompanyX', 'destination_addresses': ['192.168.0.0/24', '192.168.1.0/24']}) => {
"msg": [
{
"name": "CompanyX-192.168.0.0/24"
},
{
"name": "CompanyX-192.168.1.0/24"
}
]
}
ok: [localhost] => (item={'name': 'CompanyY', 'destination_addresses': ['192.168.2.0/24', '192.168.3.0/24']}) => {
"msg": [
{
"name": "CompanyY-192.168.2.0/24"
},
{
"name": "CompanyY-192.168.3.0/24"
}
]
}
old post
I think that's what you're looking for.
- name: create address group
fortinet.fortios.fortios_firewall_addrgrp:
state: present
firewall_addrgrp:
name: "{{ item.name }}"
member: "{{ item.destination_addresses | join(',') }}"
loop: "{{ customers }}"
I guess that member should be all addresses of the respective customer? These are in a list for the respective customer and you can join them to a string via join function.
So the task set_fact for grp_members would not be necessary.
If this is not the result you need, you have to describe it exactly.
Here you can see the sample output of the joined addresses.
This task
- name: Iterate over customers.
debug:
msg: "{{ item.name }}: {{ item.destination_addresses | join(',') }}"
loop: "{{ customers }}"
produces this output
TASK [Iterate over customers.] ***************************************************************************************
ok: [localhost] => (item={'name': 'CompanyX', 'destination_addresses': ['192.168.0.0/24', '192.168.1.0/24']}) => {
"msg": "CompanyX: 192.168.0.0/24,192.168.1.0/24"
}
ok: [localhost] => (item={'name': 'CompanyY', 'destination_addresses': ['192.168.2.0/24', '192.168.3.0/24']}) => {
"msg": "CompanyY: 192.168.2.0/24,192.168.3.0/24"
}

How do I work with Ansible list of dicts?

I'm so confused with this. If I have a file containing:
users:
- name: jconnor
first: john
last: connor
uid: 3003
- name: sconnor
first: sarah
last: connor
uid: 3001
How do I get the details of each user? With this simple playbook:
- name: create users
hosts: localhost
gather_facts: false
tasks:
- name: Include vars
include_vars:
file: user_list.yml
name: users
- name: debug
debug:
msg: "{{ item }}"
with_dict: "{{ users }}"
I get the following which I can't use:
ok: [localhost] => (item={'value': [{u'last': u'connor', u'uid': 3003, u'name': u'jconnor', u'first': u'john'}, {u'last': u'connor', u'uid': 3001, u'name': u'sconnor', u'first': u'sarah'}], 'key': u'users'}) => {
"msg": {
"key": "users",
"value": [
{
"first": "john",
"last": "connor",
"name": "jconnor",
"uid": 3003
},
{
"first": "sarah",
"last": "connor",
"name": "sconnor",
"uid": 3001
}
]
}
}
I want to create user accounts with this but I simply don't understand how to use this structure.
Note that this is part of a larger structure and I can't change it.
Thanks
Since the users variable is a list of dicts, you should loop with loop or with_items. Then we can access the key of each dict with item.key. E.g.: item.name, item.uid, etc.
Note that you are importing the variables from the file with the name users. So this variable now contains the users hash of that file. If you skip name: users in include_var, then you can directly access the users dict while looping.
tasks:
- include_vars:
file: user_list.yml
name: users
- debug:
msg: "Username is {{ item.name }}, full name is {{ item.first }} {{ item.last }}, userid is {{ item.uid }}"
with_items: "{{ users.users }}"
This outputs message (showing 1 item):
ok: [localhost] => (item={u'last': u'connor', u'uid': 3003, u'name': u'jconnor', u'first': u'john'}) => {
"msg": "Username is jconnor, full name is john connor, userid is 3003"
}

Tell Ansible to iterate through a (network camera) configuration json file and update every value via a new api call

I've got a .json file filled with hundreds of configuration values for an Axis network camera. The contents look like this:
{
"Name": "HTTPS.Port",
"Value": "443"
},
{
"Name": "Image.DateFormat",
"Value": "YYYY-MM-DD"
},
{
"Name": "Image.MaxViewers",
"Value": "20"
},
The Axis API, called Vapix, only provides an update function that updates a single value, so I have to circle through the values and trigger a new API call with every iteration:
name: update parameters
local_action:
module: uri
user: x
password: y
url: "{{ axis_snmp_role.server_url }}?action=update&{{ item }}"
with_items:
- "SNMP.V2c=yes"
- "SNMP.Enabled=yes"
- "ImageSource.I0.Sensor.ExposureValue=100"
Now the above example requires me to hardcode hundreds of config values into the loop. Is there a way to tell Ansible to go through the camera configuration json, update every value via a new api call and stop when there are no more values left within the json file?
Given the configuration's values is a list. For example
shell> cat data.yml
config: [
{"Name": "HTTPS.Port", "Value": "443"},
{"Name": "Image.DateFormat", "Value": "YYYY-MM-DD"},
{"Name": "Image.MaxViewers", "Value": "20"}]
The play
- hosts: localhost
tasks:
- include_vars:
file: data.json
- debug:
msg: "?action=update&{{ item.Name }}={{ item.Value }}"
loop: "{{ config }}"
gives
ok: [localhost] => (item={u'Name': u'HTTPS.Port', u'Value': u'443'}) => {
"msg": "?action=update&HTTPS.Port=443"
}
ok: [localhost] => (item={u'Name': u'Image.DateFormat', u'Value': u'YYYY-MM-DD'}) => {
"msg": "?action=update&Image.DateFormat=YYYY-MM-DD"
}
ok: [localhost] => (item={u'Name': u'Image.MaxViewers', u'Value': u'20'}) => {
"msg": "?action=update&Image.MaxViewers=20"
}
If this is what you want loop the uri module. For example
- local_action:
module: uri
user: x
password: y
url: "{{ axis_snmp_role.server_url }}?action=update&{{ item.Name }}={{ item.Value }}"
loop: "{{ config }}"

Variable won't be filtered correctly

what am I doing wrong?
I use below task to get all defined log_dirs of a host. Those information are stored in a fact, which is a dict and this works like a charm.
- name: get all_log_dirs
set_fact:
all_log_dirs="{{ (all_log_dirs|default({})) | combine( { item.key:vars[item.key] } ) }}"
with_dict: "{{ vars }}"
when: item.key is search('^((?!splunk).)*_log_dir')
Here the appropriate output:
"ansible_facts": {
"all_log_dirs": {
"springboot_server_log_dir": "{{ server_deployment_dir }}/logs"}
But the problem is, if I now want to use the new dict for e. g.:
- name: create symlink for splunk if not present
file:
src: "{{ item.value }}"
dest: "{{ splunk_log_dir }}/{{ item.key | regex_replace('_server_log_dir|_log_dir') | regex_replace('eap','jboss-eap') }}"
state: link
with_dict: "{{ all_log_dirs }}"
I only get:
failed: [...] (item={'value': u'{{ server_deployment_dir }}/logs', 'key': u'springboot_server_log_dir'}) => {
"changed": false,
"invocation": {
"module_args": {
"dest": "/somedir/springboot",
"path": "/somedir/springboot",
"src": "{{ server_deployment_dir }}/logs",
"state": "link",
}
},
"msg": "src file does not exist, use \"force=yes\" if you really want to create the link: /somedir/{{ server_deployment_dir }}/logs",
"path": "/somedir/springboot",
"src": "{{ server_deployment_dir }}/logs",
"state": "absent"
}
Why isn't {{ server_deployment_dir }} filtered correctly by Ansible?
Even I change src to dest and the way around, it won't work, because the variable isn't being filtered.
The value of {{ server_deployment_dir }} is of course host specific and is sth like /opt/applicationXY/appDeployDir
Don't use vars object. Period.
It is internal variable storage intended for under-the-hood usage.
When Ansible template engine detects vars access, it stops further templating chain!
Example:
---
- hosts: localhost
connection: local
gather_facts: no
vars:
myvar1: hello
myvar2: world
myvar3: "{{ myvar2 }}"
tasks:
- debug:
msg: "{{ myvar1 }} {{ myvar3 }}"
- debug:
msg: "{{ vars['myvar1'] }} {{ vars['myvar3'] }}"
Result:
TASK [debug] ***************************
ok: [localhost] => {
"msg": "hello world"
}
TASK [debug] ***************************
ok: [localhost] => {
"msg": "hello {{ myvar2 }}"
}
Update: if you utterly need to access variable trough vars object, there's vars lookup available in Ansible 2.5; and it templates values as usual:
E.g.
- debug:
msg: "{{ lookup('vars','myvar1') }} {{ lookup('vars','myvar3') }}"
results to hello world in the context of my previous example.

Resources