Select and concatenate dict value lists - ansible

I've got a dict of lists like this:
packages:
server:
- foo
- bar
- baz
client:
- spam
- ham
- eggs
runtime:
- corge
- grault
- garply
and I want to generate a list by defining keys to select in another dict, so e.g.:
enable:
server: true
client: false
runtime: true
would result in:
output:
- foo
- bar
- baz
- corge
- grault
- garply
Any ideas how to do this in ansible/jinja?? Ideally without using the loop construct but I could live with that. I suspect it'll need an intermediate variable and so far all I've got is something to extract a list of the keys from enable which have a true value:
- set_fact:
enabled: "{{ (enable | dict2items | selectattr('value') | list | items2dict).keys() }}"
If this looks a bit convoluted its because both packages and enable are existing role variables which I'd prefer not to change. But open to suggestions (especially on packages) if reshaping them makes this much easier.

Let's create the list of enabled hosts first. For example
- set_fact:
my_hosts: "{{ enable|dict2items|
selectattr('value')|
map(attribute='key')|list }}"
- debug:
var: my_hosts
give
my_hosts:
- server
- runtime
Then use this list to extract the packages. For example
- set_fact:
my_list: "{{ my_hosts|map('extract', packages)|list|flatten }}"
- debug:
var: my_list
give
my_list:
- foo
- bar
- baz
- corge
- grault
- garply
The single task below comprises both steps
- set_fact:
my_list: "{{ my_hosts|map('extract', packages)|list|flatten }}"
vars:
my_hosts: "{{ enable|dict2items|
selectattr('value')|
map(attribute='key')|list }}"

Related

Iterate dictionary Ansible for certain keys

a doubt on how to iterate a certain dictionary
---
- name: main playbook
hosts: localhost
tasks:
- name: Set fact
var:
tower_objects_organizations:
security:
org: ORG_I-SSO-ES-ADM-SSMM
teams:
- I-SSO-ES-ADM-SSMM
- I-SSO-ES-ADM-SSMM2
monit:
org: ORG_I-SSO-ES-TS-MONIT
teams:
- I-SSO-ES-TS-MONIT
- name: "[{{ productname }}][parallel set organization] Set organizations fact"
debug:
msg: "XXXXX"
with_items: "{{ tower_objects_organizations| dict2items | subelements('value.teams') }}"
Does anyone know how could I iterate through the teams in each subdirectionary (security, monit ...) and to fill a dictionary with the team as key and its org as value, i.e. I-SSO-ES-ADM-SSMM: ORG_I-SSO-ES-ADM-SSMM, I-SSO-ES-ADM-SSMM2: ORG_I-SSO-ES-ADM-SSMM, I-SSO-ES-TS-MONIT:ORG_I-SSO-ES-TS-MONIT
Using subelements I am capable of reaching the teams list but the output is not the expected one
A solution with and without custom filter:
when i have to do some complex actions on dict i prefer to use a custom filter:
you create a folder filter_plugins in your playbook folder (i have named the file myfilters.py and the filter customfilter)
#!/usr/bin/python
class FilterModule(object):
def filters(self):
return {
'customfilter': self.customfilter
}
def customfilter(self, obj):
result = {}
for k in obj:
org = obj[k]['org']
for l in obj[k]['teams']:
result[l] = org
return result
playbook sample:
tasks:
- name: Set fact
set_fact:
result: "{{ tower_objects_organizations | customfilter}}"
vars:
tower_objects_organizations:
security:
org: ORG_I-SSO-ES-ADM-SSMM
teams:
- I-SSO-ES-ADM-SSMM
- I-SSO-ES-ADM-SSMM2
monit:
org: ORG_I-SSO-ES-TS-MONIT
teams:
- I-SSO-ES-TS-MONIT
others:
org: ORG_I-SSO-ES-TS-OTHER
teams:
- I-SSO-ES-TS-OTHER0
- I-SSO-ES-TS-OTHER1
- name: display result
debug:
var: result
result:
ok: [localhost] =>
result:
I-SSO-ES-ADM-SSMM: ORG_I-SSO-ES-ADM-SSMM
I-SSO-ES-ADM-SSMM2: ORG_I-SSO-ES-ADM-SSMM
I-SSO-ES-TS-MONIT: ORG_I-SSO-ES-TS-MONIT
I-SSO-ES-TS-OTHER0: ORG_I-SSO-ES-TS-OTHER
I-SSO-ES-TS-OTHER1: ORG_I-SSO-ES-TS-OTHER
EDIT
another solution without custom filter but less readable i think:
- set_fact:
result: "{{ result | d({}) | combine(dict(item.value.teams|product([item.value.org]))) }}"
loop: "{{ tower_objects_organizations|dict2items }}"
- name: debug result
debug:
var: result

A method for listing Ansible modules used by playbooks

I'm creating a requirements.yml in an Ansible project, and I want to identify all of the modules that need to be installed from ansible-galaxy that are used by project playbooks. ansible-doc --list --playbook-dir foo seems like the right tool for the job, but it lists all locally available modules, not just the ones which are actually used in the foo directory. ansible-galaxy list doesn't account for any which are needed but not installed.
Is there a way to do this where I don't end up writing a shell script to sed|awk|grep the info I want?
The best approach I've been able to come up with so far is to ansible-playbook --syntax-check each of the playbooks. This will throw errors such as
ERROR! the role 'foo' was not found ...
ERROR! couldn't resolve module/action 'bar'. This often indicates a misspelling, missing collection, or incorrect module path.
but this is not ideal because it exits as soon as any error occurs. I have to fix each one and run the syntax check again.
FWIW, as a concept, the playbook below lists the modules used in a role
- hosts: localhost
vars:
keywords:
- always
- become
- block
- loop
- loop_control
- name
- notify
- register
- tags
- vars
- when
tasks:
- name: The variable my_role_path is mandatory
assert:
that: my_role_path|d('')|length > 0
- name: Find tasks files
find:
path: "{{ my_role_path }}/tasks"
patterns: '*.yml,*.yaml'
recurse: true
register: result
- name: Create list of tasks
set_fact:
lft: "{{ lft|d([]) + lookup('file', item)|from_yaml }}"
loop: "{{ result.files|map(attribute='path')|list }}"
- name: Get list of keys
set_fact:
lfk: "{{ lfk|d([]) + item.keys()|list }}"
loop: "{{ lft }}"
- name: Get list of keys from block/rescue/always
set_fact:
lfk: "{{ lfk|d([]) + item.keys()|list }}"
loop: "{{ lft|json_query('[].[block, rescue, always]')|flatten }}"
- name: Display list of modules
debug:
var: lfk|unique|sort|difference(keywords)
For example, analyze the role systemd
shell> ansible-playbook pb.yml -e my_role_path=roles/ansible-role-systemd
...
lfk|unique|sort|difference(keywords):
- command
- file
- include_tasks
- meta
- systemd
- template
Complete the list of the keywords.
Use the tasks below to analyze a playbook
tasks:
- name: The variable my_playbook_path is mandatory
assert:
that: my_playbook_path|d('')|length > 0
- name: Create list of tasks
set_fact:
lft: "{{ _playbook|map(attribute='tasks')|flatten }}"
vars:
_playbook: "{{ lookup('file', my_playbook_path)|from_yaml }}"
- name: Get list of keys
set_fact:
lfk: "{{ lfk|d([]) + item.keys()|list }}"
loop: "{{ lft }}"
- name: Get list of keys from block/rescue/always
set_fact:
lfk: "{{ lfk|d([]) + item.keys()|list }}"
loop: "{{ lft|json_query('[].[block, rescue, always]')|flatten }}"
- name: Display list of modules
debug:
var: lfk|unique|sort|difference(keywords)
For example, analyzing the first playbook gives
lfk|unique|sort|difference(keywords):
- assert
- debug
- find
- set_fact

Ansibe: concatenation of items from with_items

I'm trying to get a variable which will contain comma separated items from with_itmes loop as follow:
- hosts: localhost
connection: local
gather_facts: no
tasks:
- name: set_fact
set_fact:
foo: "{{ foo }}{{ item }},"
with_items:
- "one"
- "two"
- "three"
vars:
foo: ""
- name: Print the var
debug:
var: foo
It works as expected but what I'm getting at the end is trailing comma.
Is there any way to remove it?
There is a join filter that we can use with lists to concatenate list elements with a given character.
If we are passing the list directly to with_items or loop then we can use loop_control to "extend" some more loop information to get ansible_loop.allitems. Then this can be joined with the join filter.
Example:
- set_fact:
foo: "{{ ansible_loop.allitems|join(',') }}"
loop:
- one
- two
- three
loop_control:
extended: true
Otherwise a more straightforward way is to define a variable with list and use join filter on elements of that variable.
Example:
- set_fact:
foo: "{{ mylist|join(',') }}"
vars:
mylist:
- one
- two
- three
No clue if this is correct way to do but it does the job:
- name: Print the var
debug:
msg: "LIST: {{ foo | regex_replace(',$','') }}"

ansible get result from role and append to a list

I have a play like this
---
- name: List images in ACRs
any_errors_fatal: true
hosts:
- localhost
gather_facts: false
vars:
acrs: ["registry1", "registry2"]
tasks:
- name: list repos
with_items: "{{ acrs }}"
include_role:
name: list_docker_image_repos
vars:
registry_name: "{{ item }}"
list_docker_image_repos will do set_fact which a list of dicts.
How can I append all the facts (from every iteration) to a list?
Or is there are different way to do this?
thanks
In each iteration put the list into a dictionary. For example, given the role
shell> cat roles/list_docker_image_repos/tasks/main.yml
- set_fact:
docker_image_repos: "{{ ['repo1', 'repo2', 'repo3']|
product([registry_name])|
map('join', '-')|
list }}"
- set_fact:
my_lists: "{{ my_lists|
combine({registry_name: docker_image_repos}) }}"
the playbook
- hosts: localhost
vars:
acrs: [reg1, reg2]
my_lists: {}
tasks:
- name: list repos
include_role:
name: list_docker_image_repos
loop: "{{ acrs }}"
vars:
registry_name: "{{ item }}"
- debug:
var: my_lists
gives the dictionary
my_lists:
reg1:
- repo1-reg1
- repo2-reg1
- repo3-reg1
reg2:
- repo1-reg2
- repo2-reg2
- repo3-reg2
A dictionary is a more suitable structure for this purpose compared with a list. There are many ways how to use it. For example, you can
extract the lists
- debug:
msg: "{{ acrs|map('extract', my_lists)|list }}"
gives
msg:
- - repo1-reg1
- repo2-reg1
- repo3-reg1
- - repo1-reg2
- repo2-reg2
- repo3-reg2
Use the filter flatten to put all items into a single list
- debug:
msg: "{{ acrs|map('extract', my_lists)|flatten }}"
gives
msg:
- repo1-reg1
- repo2-reg1
- repo3-reg1
- repo1-reg2
- repo2-reg2
- repo3-reg2
Use the filter dict2items to iterate items
- debug:
var: item
loop: "{{ my_lists|dict2items }}"
gives (abridged)
item:
key: reg1
value:
- repo1-reg1
- repo2-reg1
- repo3-reg1
item:
key: reg2
value:
- repo1-reg2
- repo2-reg2
- repo3-reg2
Or, use the lookup plugin subelements to iterate the items of the lists as well
- debug:
var: item
with_subelements:
- "{{ my_lists|dict2items }}"
- value
gives (abridged)
item:
- key: reg1
- repo1-reg1
item:
- key: reg1
- repo2-reg1
item:
- key: reg1
- repo3-reg1
item:
- key: reg2
- repo1-reg2
item:
- key: reg2
- repo2-reg2
item:
- key: reg2
- repo3-reg2
Example of a complete playbook for testing
- hosts: localhost
vars:
acrs: [reg1, reg2]
my_lists: {}
tasks:
- name: list repos
include_role:
name: list_docker_image_repos
loop: "{{ acrs }}"
vars:
registry_name: "{{ item }}"
- debug:
var: my_lists
- debug:
msg: "{{ acrs|map('extract', my_lists)|list }}"
- debug:
msg: "{{ acrs|map('extract', my_lists)|flatten }}"
- debug:
var: item
loop: "{{ my_lists|dict2items }}"
- debug:
var: item
with_subelements:
- "{{ my_lists|dict2items }}"
- value

How to fallback to a default value when ansible lookup fails?

I was a little bit surprised to discover that his piece of code fails with an IOError exception instead of defaulting to omitting the value.
#!/usr/bin/env ansible-playbook -i localhost,
---
- hosts: localhost
tasks:
- debug: msg="{{ lookup('ini', 'foo section=DEFAULT file=missing-file.conf') | default(omit) }}"
How can I load a value without raising an exception?
Please note that the lookup module supports a default value parameter but this one is useless to me because it works only when it can open the file.
I need a default value that works even when the it fails to open the file.
As far as I know Jinja2 unfortunately doesn't support any try/catch mechanism.
So you either patch ini lookup plugin / file issue to Ansible team, or use this ugly workaround:
---
- hosts: localhost
gather_facts: no
tasks:
- debug: msg="{{ lookup('first_found', dict(files=['test-ini.conf'], skip=true)) | ternary(lookup('ini', 'foo section=DEFAULT file=test-ini.conf'), omit) }}"
In this example first_found lookup return file name if file exists or empty list otherwise. If file exists, ternary filter calls ini lookup, otherwise omit placeholder is returned.
In case people like me stumble upon this question in 2022,
Ansible now supports rescue blocks, which is similar to try-catch-finally in programming languages.
Examples can be found in the official documentation Error handling with blocks.
You can use block/rescue as follows:
- hosts: localhost
tasks:
- block:
- debug: msg="{{ lookup('ini', 'foo section=DEFAULT file=missing-file.conf') }}"
rescue:
- debug: msg="omit"
You can also convert your input file with a from_yaml filter before using the default filter
- name: "load a yaml file or a default value"
set_fact:
myvar: "{{ lookup('file', 'myfile.yml', errors='ignore') | from_yaml | default(mydefaultObject, true) }}"
To avoid the error when the path doesn't exist, use a condition to check for the path before attempting the lookup:
---
- hosts: localhost
tasks:
- debug: msg="{{ lookup('ini', 'foo section=DEFAULT file=missing-file.conf') }}"
when: missing-file.conf | exists
You can use this with set_fact as well, then omit the undefined var when using it if required:
- hosts: localhost
tasks:
- set_fact:
foo: "{{ lookup('ini', 'foo section=DEFAULT file=missing-file.conf') }}"
when: missing-file.conf | exists
- debug:
var: foo # undefined
msg: "{{ foo | default(omit) }}" # omitted
Note that lookups and Jinja2 tests run on the controller. If you need to check the path on the host, use the stat and either slurp or fetch modules:
- stat:
file: missing-remote-file-with-text-i-want
register: file
- slurp:
src: missing-remote-file-with-text-i-want
register: slurp
when: file.stat.exists
- set_fact:
foo: "{{ slurp.content | b64decode }}"
when: file.stat.exists
- fetch:
src: missing-file.conf
dest: /tmp/fetched
fail_on_missing: False
- set_fact:
bar: "{{ lookup('ini', 'foo section=DEFAULT file=/tmp/fetched/' + inventory_hostname + '/missing-file.conf') }}"
when: ('/tmp/fetched/' + inventory_hostname + '/missing-file.conf') | exists
Second note, in Ansible v2.5 the grammar for using the path tests was changed, the format is now:
- set_fact:
foo: "{{ lookup('ini', 'foo section=DEFAULT file=missing-file.conf') }}"
when: missing-file.conf is exists

Resources