Use list of dictionaries variable on Ansible Tower textare survey - ansible

I'm trying to develop a playbook were I have the following variable.
disk_vars:
- { Unit: C, Size: 50 }
- { Unit: D, Size: 50 }
With the variables defined on the playbook there is no problem but when I try to use a texarea survey on Ansible Tower I cannot manage to parse them as list of dictionaries.
I tried adding to the survey the following two lines which are already on yaml format.
- { Unit: C, Size: 50 }
- { Unit: D, Size: 50 }
And on my vars section I use test_var: "{{ test_var1.split('\n') }} which converts the output into a two line string. Without the split is just a single line string.
I could make my playbook work with a simple dictionary like
dict1: {{ Unit: C, Size: 50 }}
but I'm having issues parsing it as well.
EDIT
Changing it to the following as suggested by mdaniels works.
- set_fact:
test_var: "{{ test_var1 | from_yaml }}"
- name: test
debug: msg=" hostname is {{ item.Unit }} and {{ item.Size }}"
with_items:
- "{{ test_var }}"
I'm trying to figure a way to clear-up the data input as asking users to respect the format is not a very good idea.
tried changing the input date to the following but I could not figure out how to format that into a list of dictionaries.
disk_vars:
Unit: C, Size: 50
Unit: D, Size: 50
I tried with the following piece of code
- set_fact:
db_list: >-
{{ test_var1.split("\n") | select |
map("regex_replace", "^", "- {") |
map("regex_replace", "$", "}") |
join("\n") }}
But is putting it all on a single line.
"db_list": "- {dbid: 1, dbname: abc\ndbid: 2, dbname: xyz} "
I have tried to play with it but could not manage to make it work.

I believe you were very close; instead of "{{ test_var1.split('\n') }}" I believe you can just feed it to the from_yaml filter:
- set_fact:
test_var1: '{{ test_var1 | from_yaml }}'
# this is just to simulate the **str** that you will receive from the textarea
vars:
test_var1: "- { Unit: C, Size: 50 }\n- { Unit: D, Size: 50 }\n"
- debug:
msg: and now test_var1[0].Unit is {{ test_var1[0].Unit }}

I faced a similar dilemma, i.e. that I was bound to the survey format(s) available, and I was forced to use mdaniels suggested solution above with sending the data as text and then later parse it from YAML . Problem was however that controlling the format of the input (i.e. a YAML-string inside the text) would probably cause a lot of headache/errors, just like you describe.
Maybe you really need to use the Survey, but in my case I was more interested of calling the Job Template using the Tower REST API. For some reason I thought I then had to have a survey with all parameters defined. But it turned out I was wrong, when having a survey I was not able to provide dictionaries as input data (in the extra_vars). However, when removing the Survey, and also (not sure if required or not) enabling "Extra Variables -> prompt on launch", then things started to work!! Now I can provide lists / dictionaries as input to my Templates when calling them using REST API POST calls, see example below:
{
"extra_vars": {
"p_db_name": "MYSUPERDB",
"p_appl_id": "MYD32",
"p_admin_user": "myadmin",
"p_admin_pass": "mysuperpwd",
"p_db_state": "present",
"p_tablespaces": [
{
"name": "tomahawk",
"size": "10M",
"bigfile": true,
"autoextend": true,
"next": "1M",
"maxsize": "20M",
"content": "permanent",
"state": "present"
}
],
"p_users": [
{
"schema": "myschema",
"password": "Mypass123456#",
"default_tablespace": "tomahawk",
"state": "present",
"grants": "'create session', 'create any table'"
}
]
}
}

Related

Using parse_xml in an Ansible playbook

I've been trying to parse XML data in Ansible. I can get it to work using the xml module but I think that using parse_xml would better suit my needs.
I don't seem to be able to match any of the data in the xml with my specs file.
Here is the xml data:
<data xmlns=\"urn:ietf:params:xml:ns:netconf:base:1.0\" xmlns:nc=\"urn:ietf:params:xml:ns:netconf:base:1.0\">
<ntp xmlns=\"http://cisco.com/ns/yang/Cisco-IOS-XR-ip-ntp-oper\">
<nodes>
<node>
<node>0/0/CPU0</node>
<associations>
<is-ntp-enabled>true</is-ntp-enabled>
<sys-leap>ntp-leap-no-warning</sys-leap>
<peer-summary-info>
<peer-info-common>
<host-mode>ntp-mode-client</host-mode>
<is-configured>true</is-configured>
<address>10.1.1.1</address>
<reachability>0</reachability>
</peer-info-common>
<time-since>-1</time-since>
</peer-summary-info>
<peer-summary-info>
<peer-info-common>
<host-mode>ntp-mode-client</host-mode>
<is-configured>true</is-configured>
<address>172.16.252.29</address>
<reachability>255</reachability>
</peer-info-common>
<time-since>991</time-since>
</peer-summary-info>
</associations>
</node>
</nodes>
</ntp>
</data>
This is what the spec file looks like:
---
vars:
ntp_peers:
address: "{{ item.address }}"
reachability: "{{ item.reachability}}"
keys:
result:
value: "{{ ntp_peers }}"
top: data/ntp/nodes/node/associations
items:
address: peer-summary-info/peer-info-common/address
reachability: peer-summary-info/peer-info-common/reachability
and the task in the yaml file:
- name: parse ntp reply
set_fact:
parsed_ntp_data: "{{ NTP_STATUS.stdout | parse_xml('specs/iosxr_ntp.yaml') }}"
but the data does not return any results:
TASK [debug parsed_ntp_data] **************************************************************************************************************************************************************************
ok: [core-rtr01] => {
"parsed_ntp_data": {
"result": []
}
}
ok: [dist-rtr01] => {
"parsed_ntp_data": {
"result": []
}
}
I had never seen parse_xml before, so that was a fun adventure
There appear to be two things conspiring against you: the top: key is evaluated from the root Element, and your XML (unlike the rest of the examples) uses XML namespaces (the xmlns= bit) which means your XPaths have to be encoded in the Element.findall manner
For the first part, since Element.findall is run while sitting on the <data> Element, that means one cannot reference data/... in an XPath because that would be applicable to a structure <data><data>. I tried being sneaky by just making the XPath absolute /data/... but Python's XPath library throws up in that circumstance. So, at the very least your top: key needs to not start with data anything
Then, the xmlns= in your snippet stood out to me because that means those element's names are actually NS+":"+localName for every element, and thus an XPath of ntp does NOT match ns0:ntp because they're considered completely separate names (that being the point of the namespace, after all). It may very well be possible to use enough //*[localname() = "ntp"] silliness to avoid having to specify the namespace over and over, but I didn't try it
Again, as a concession to Python's XPath library, they encode the fully qualified name in an xpath as {the-namespace}local-name and there does not seem to be any way short of modifying network.py to pass in namespaces :-(
Thus, the "hello world" version that I used to confirm my theory:
vars:
ntp_peers:
address: "{{ item.address }}"
keys:
result:
value: "{{ ntp_peers }}"
top: '{http://cisco.com/ns/yang/Cisco-IOS-XR-ip-ntp-oper}ntp/{http://cisco.com/ns/yang/Cisco-IOS-XR-ip-ntp-oper}nodes/{http://cisco.com/ns/yang/Cisco-IOS-XR-ip-ntp-oper}node'
items:
address: '{http://cisco.com/ns/yang/Cisco-IOS-XR-ip-ntp-oper}node'
cheerfully produced
ok: [localhost] => {
"msg": {
"result": [
{
"address": "0/0/CPU0"
}
]
}
}

Override group_vars by external source

I have a pilote project keeping many common variables in group_vars.
group_vars/
group1.yml
group2.yml
group3.yml
For different implementations (usually per client), I'd like to maintain reserved file which overrides the content of group_vars, where the content of that file could have following format, i.e. client1.yml :
group1:
var11_to_override: "foo"
var12_to_override: "bar"
group2:
var21_to_override: "foo"
var22_to_override: "bar"
Is there a simple possibility to say to Ansible that file client1.yml overrides group_vars content?
The module include_vars could be certainly the first step together with set_facts within a loop, but it requires probably complicated jinja2 filter expressions ...
Have I to write a new module or filter updating hostvars?
Finally resolved by custom filter updating a dict by another:
filter_plugins/vars_update.py
import copy
import collections
class FilterModule(object):
def update_hostvars(self, _origin, overlay):
origin = copy.deepcopy(_origin)
for k, v in overlay.items():
if isinstance(v, collections.Mapping):
origin[k] = self.update_hostvars(origin.get(k, {}), v)
else:
origin[k] = v
return origin
def filters(self):
return {"update_hostvars": self.update_hostvars}
.. and using this filter when updating all variables:
- name: Include client file
include_vars:
file: "{{ client_file_path }}"
name: client_overlay
- name: Update group_vars by template client
set_fact:
"{{ item.key }}": "{{ hostvars[inventory_hostname][item.key] | update_hostvars(item.value) }}"
with_dict: "{{ client_overlay }}"
Using the examples given in this thread i made my own solution:
The "external source" feeds in an inventory item using --extra-vars "#". The file content itself is uploaded as base64 encoded content and then decoded/written to fs.
The external file has a list of overrides per role/group like so:
role_overrides: [{
"groups": [
"my-group"
],
"overrides": {
"foo": "value",
"bar": "value",
}
},
but then jsonified obviously...
The filter module
#!/usr/bin/env python
class FilterModule(object):
def filters(self):
return {
"filter_hostvars_overrides": self.filter_hostvars_overrides,
}
def filter_hostvars_overrides(self, role_overrides, group_names):
"""
filter the overrides for the ones to apply for this host
[
{
"groups": [
"my-group"
],
"overrides": {
"foo: 42,
}
},
:param group_names: List of groups this host is member of
:param role_overrides: document with all overrides; to be filtered using groups_names
:return: items to be set
"""
overrides = {}
for idx, per_group_overrides in enumerate(role_overrides):
groups = per_group_overrides.get("groups", [])
if set(groups).intersection(set(group_names)):
overrides.update(per_group_overrides.get("overrides", {}))
return overrides
The play code:
- name: Apply group overrides
set_fact:
"{{ item.key }}": "{{ item.value }}"
with_dict: "{{ role_overrides | filter_hostvars_overrides(group_names) }}"

Ansible - Add some calculated properties to list of dict

I am using a list of dict to declare some website to configure on web server.
There are some calculated properties I don't want to redeclare each time I need it, so before using it, I made a loop declaring all calculated/missing properties to get a proper list of websites (dict).
Here is what I am doing for now
- name: Set server vhosts
set_fact:
websites: "{{ websites|default([]) + [item | combine({'vhost': '200-' + item.name, 'path': path_vhosts + '/' + item.name, 'domain': app_hosts[item.name]})] }}"
with_items: "{{ vhosts }}"
But this is very limited, and will be unreadable if there is too much property to set.
How could I improve it to build it properly please ?
In a the best way, there is no variable vhosts, we just use websites and replace it.
Q: "Will be unreadable if there is too much property to set. How could I improve it to build it properly?"
A: It's built properly. The formatting might help. See below
- name: Set server vhosts
set_fact:
websites: "{{ websites|default([]) + [item|
combine({'vhost': '200-' + item.name,
'path': path_vhosts + '/' + item.name,
'domain': app_hosts[item.name]
}) ] }}"
loop: "{{ vhosts }}"

Listing more than 100 records in Route 53 using Ansible

I'm currently using the route53_facts module on a project. I have 250 record sets in one hosted zone. I'm having difficulty with listing all record sets in that zone. The Route 53 API works by returning pages of maximum 100 records at a time. In order to retrieve the next page, you must pass the NextRecordName response value to the route53_facts module's start_record_name: field (pretty straightforward).
The issue I'm having specifically is getting Ansible to do this. Presumably one would do this using a loop, e.g. in pseudocode:
start
get 100 records
do until response does not contain NextRecordName:
get 100 records (start_record_name=NextRecordName)
end
In Ansible, I have written the below task to do this:
- block:
- name: List record sets in a given hosted zone
route53_facts:
query: record_sets
hosted_zone_id: "/hostedzone/ZZZ1111112222"
max_items: 100
start_record_name: "{{ record_sets.NextRecordName | default(omit) }}"
register: record_sets
until: record_sets.NextRecordName is not defined
when: "'{{ hosted_zone['Name'] }}' == 'test.example.com.'"
...however, this does not work as expected. Instead of continuously paging through responses until no more records are left, it repeatedly returns the first 100 records ("the first page").
As I can see from the Ansible debug output, start_record_name: is repeatedly null:
"attempts": 2,
"changed": false,
"invocation": {
"module_args": {
"aws_access_key": null,
"aws_secret_key": null,
"change_id": null,
"delegation_set_id": null,
"dns_name": null,
"ec2_url": null,
"health_check_id": null,
"health_check_method": "list",
"hosted_zone_id": "/hostedzone/ZZZ1111112222",
"hosted_zone_method": "list",
"max_items": "100",
"next_marker": null,
"profile": null,
"query": "record_sets",
"region": null,
"resource_id": null,
"security_token": null,
"start_record_name": null,
"type": null,
"validate_certs": true
}
},
...my guess is that the | default(omit) filter is always being executed. In other words, record_sets.NextRecordName is never initialized at this point in the task.
I'm hoping somebody can assist me in getting Ansible to return all records from a zone in Route 53. I think I've gotten tangled up in Ansible's looping behavior. Thanks!
Caveat this with "as best I can tell:"
To answer your question, it actually seems that until: and register: do not interact in the same way that when: and register: do. The best explanation I have is that until: behaves like a database transaction: it rolls back the register: assignment if the conditional is false, meaning that when the body of the until: task is tried again, it uses the same parameters as the first time. The only thing which keeps an until: block from being an infinite loop is the retries: value.
So, in your specific case, I think this will do the job:
- name: initial record_set
route53_facts:
# bootstrap so the upcoming "when:" will evaluate correctly
register: record_facts
- set_fact:
# capture the initial answer
records0: '{{ record_facts.ResourceRecordSets }}'
- name: rest of them
route53_facts:
start_record_name: '{{ record_facts.NextRecordName }}'
register: record_facts
when: record_facts.NextRecordName | default("")
with_sequence: count=10
- set_fact:
all_records: >-
{{ record0 + (record_facts.results |
selectattr("ResourceRecordSets", "defined") |
map(attribute="ResourceRecordSets") | list) }}
The with_sequence: is a hack because loop: (for which with_* is syntatic sugar) needs a list of items over which to iterate, but given that the responses that come back without NextRecordName will cause the when: to fail, skipping them, makes the (in your case) 3 through 10 items resolve almost immediately.
Then you just need to pull out the actual response data from the now list of route53_facts: replies, and glue them to the initial one to get the complete list.
Having said all of that, I am now convinced that route53_facts: (and any other AWS module that pushes the burden of that iteration into the playbook) behavior is a bug. The module caller already has a max_items: available to them, but it's an implementation detail that that any value can't be larger than some random pagination cut-off.

Ansible Dict and Tags

I have a playbook creating EC2 by using a dictionary declared in vars: then registering the IPs into a group to be used later on.
The dict looks like this:
servers:
serv1:
name: tag1
type: t2.small
region: us-west-1
image: ami-****
serv2:
name: tag2
type: t2.medium
region: us-east-1
image: ami-****
serv3:
[...]
I would like to apply tags to this playbook in the simplest way so I can create just some of them using tags. For example, running the playbook with --tags tag1,tag3 would only start EC2 matching serv1 and serv3.
Applying tags on the dictionary doesn't seem possible and I would like to avoid doing multiplying tasks like:
Creatinge EC2
Register infos
Getting private IP from previously registered infos
adding host to group
While I already have a working loop for the case I want to create all EC2 at once, is there any way to achieve that (without relying on --extra-vars, which would need key=value) ? For example, filtering out the dictionary by keeping only what is tagged before running the EC2 loop ?
I doubt you can do this out of the box. And not sure this is good idea at all.
Because tags are used to filter tasks in Ansible, so you will have to mark all tasks with tags: always.
You can accomplish this with custom filter plugin, for example (./filter_plugins/apply_tags.py):
try:
from __main__ import cli
except ImportError:
cli = False
def apply_tags(src):
if cli:
tags = cli.options.tags.split(',')
res = {}
for k,v in src.iteritems():
keep = True
if 'name' in v:
if v['name'] not in tags:
keep = False
if keep:
res[k] = v
return res
else:
return src
class FilterModule(object):
def filters(self):
return {
'apply_tags': apply_tags
}
And in your playbook:
- debug: msg="{{ servers | apply_tags }}"
tags: always
I found a way to match my needs without touching to the rest so I'm sharing it in case other might have a similar need.
I needed to combine dictionaries depending on tags, so my "main" dictionary wouldn't be static.
Variables became :
- serv1:
- name: tag1
type: t2.small
region: us-west-1
image: ami-****
- serv2:
- name: tag2
type: t2.medium
region: us-east-1
image: ami-****
- serv3:
[...]
So instead of duplicating my tasks, I used set_fact with tags like this:
- name: Combined dict
# Declaring empty dict
set_fact:
servers: []
tags: ['always']
- name: Add Server 1
set_fact:
servers: "{{ servers + serv1 }}"
tags: ['tag1']
- name: Add Server 2
set_fact:
servers: "{{ servers + serv2 }}"
tags: ['tag2']
[..]
20 lines instead of multiply tasks for each server, change vars from dictionary to lists, a few tags and all good :) Now if I add a new server it will only take a few lines.

Resources