Check if a file has certain strings - ansible

I have some files (file1), in some servers (group: myservers), which should look like this:
search www.mysebsite.com
nameserver 1.2.3.4
nameserver 1.2.3.5
This is an example of what this file should look like:
The first line is mandatory ("search www.mysebsite.com").
The second and the third lines are mandatory as well, but the ips can change (although they should all be like this: ...).
I've being researching to implement some tasks using Ansible to check if the files are properly configured. I don't want to change any file, only check and output if the files are not ok or not.
I know I can use ansible.builtin.lineinfile to check it, but I still haven't managed to find out how to achieve this.
Can you help please?

For example, given the inventory
shell> cat hosts
[myservers]
test_11
test_13
Create a dictionary of what you want to audit
audit:
files:
/etc/resolv.conf:
patterns:
- '^search example.com$'
- '^nameserver \d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$'
/etc/rc.conf:
patterns:
- '^sshd_enable="YES"$'
- '^syslogd_flags="-ss"$'
Declare the directory at the controller where the files will be stored
my_dest: /tmp/ansible/myservers
fetch the files
- fetch:
src: "{{ item.key }}"
dest: "{{ my_dest }}"
loop: "{{ audit.files|dict2items }}"
Take a look at the fetched files
shell> tree /tmp/ansible/myservers
/tmp/ansible/myservers
├── test_11
│   └── etc
│   ├── rc.conf
│   └── resolv.conf
└── test_13
└── etc
├── rc.conf
└── resolv.conf
4 directories, 4 files
Audit the files. Create the dictionary host_files_results in the loop
- set_fact:
host_files_results: "{{ host_files_results|default({})|
combine(host_file_dict|from_yaml) }}"
loop: "{{ audit.files|dict2items }}"
loop_control:
label: "{{ item.key }}"
vars:
host_file_path: "{{ my_dest }}/{{ inventory_hostname }}/{{ item.key }}"
host_file_lines: "{{ lookup('file', host_file_path).splitlines() }}"
host_file_result: |
[{% for pattern in item.value.patterns %}
{{ host_file_lines[loop.index0] is regex pattern }},
{% endfor %}]
host_file_dict: "{ {{ item.key }}: {{ host_file_result|from_yaml is all }} }"
gives
ok: [test_11] =>
host_files_results:
/etc/rc.conf: true
/etc/resolv.conf: true
ok: [test_13] =>
host_files_results:
/etc/rc.conf: true
/etc/resolv.conf: true
Declare the dictionary audit_files that aggregates host_files_results
audit_files: "{{ dict(ansible_play_hosts|
zip(ansible_play_hosts|
map('extract', hostvars, 'host_files_results'))) }}"
gives
audit_files:
test_11:
/etc/rc.conf: true
/etc/resolv.conf: true
test_13:
/etc/rc.conf: true
/etc/resolv.conf: true
Evaluate the audit results
- block:
- debug:
var: audit_files
- assert:
that: "{{ audit_files|json_query('*.*')|flatten is all }}"
fail_msg: "[ERR] Audit of files failed. [TODO: list failed]"
success_msg: "[OK] Audit of files passed."
run_once: true
gives
msg: '[OK] Audit of files passed.'
Example of a complete playbook for testing
- hosts: myservers
vars:
my_dest: /tmp/ansible/myservers
audit:
files:
/etc/resolv.conf:
patterns:
- '^search example.com$'
- '^nameserver \d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$'
/etc/rc.conf:
patterns:
- '^sshd_enable="YES"$'
- '^syslogd_flags="-ss"$'
audit_files: "{{ dict(ansible_play_hosts|
zip(ansible_play_hosts|
map('extract', hostvars, 'host_files_results'))) }}"
tasks:
- fetch:
src: "{{ item.key }}"
dest: "{{ my_dest }}"
loop: "{{ audit.files|dict2items }}"
loop_control:
label: "{{ item.key }}"
- set_fact:
host_files_results: "{{ host_files_results|default({})|
combine(host_file_dict|from_yaml) }}"
loop: "{{ audit.files|dict2items }}"
loop_control:
label: "{{ item.key }}"
vars:
host_file_path: "{{ my_dest }}/{{ inventory_hostname }}/{{ item.key }}"
host_file_lines: "{{ lookup('file', host_file_path).splitlines() }}"
host_file_result: |
[{% for pattern in item.value.patterns %}
{{ host_file_lines[loop.index0] is regex pattern }},
{% endfor %}]
host_file_dict: "{ {{ item.key }}: {{ host_file_result|from_yaml is all }} }"
- debug:
var: host_files_results
- block:
- debug:
var: audit_files
- assert:
that: "{{ audit_files|json_query('*.*')|flatten is all }}"
fail_msg: "[ERR] Audit of files failed. [TODO: list failed]"
success_msg: "[OK] Audit of files passed."
run_once: true

... implement some tasks using Ansible to check if the files are properly configured. I don't want to change any file, only check and output if the files are not ok or not.
Since Ansible is mostly used as Configuration Management Tool there is no need to check (before) if a file is properly configured. Just declare the Desired State and make sure that the file is in that state. As this is approach is working with Validating: check_mode too, if interested in a Configuration Check or an Audit it could be implemented simply as follow:
resolv.conf as is it should be
# Generated by NetworkManager
search example.com
nameserver 192.0.2.1
hosts.ini
[test]
test.example.com NS_IP=192.0.2.1
resolv.conf.j2 template
# Generated by NetworkManager
search {{ DOMAIN }}
nameserver {{ NS_IP }}
A minimal example playbook for Configuration Check in order to audit the config
---
- hosts: test
become: false
gather_facts: false
vars:
# Ansible v2.9 and later
DOMAIN: "{{ inventory_hostname.split('.', 1) | last }}"
tasks:
- name: Check configuration (file)
template:
src: resolv.conf.j2
dest: resolv.conf
check_mode: true # will never change existing config
register: result
- name: Config change
debug:
msg: "{{ result.changed }}"
will result for no changes into an output of
TASK [Check configuration (file)] ******
ok: [test.example.com]
TASK [Config change] *******************
ok: [test.example.com] =>
msg: false
or for changes into
TASK [Check configuration (file)] ******
changed: [test.example.com]
TASK [Config change] *******************
ok: [test.example.com] =>
msg: true
and depending on what's in the config file.
If one is interested in an other message text and need to invert the output therefore, just use msg: "{{ not result.changed }}" as it will report an false if true and true if false.
Further Reading
Using Ansible inventory, variables in inventory, the template module (to) Template a file out to a target host and Enforcing check_mode on tasks makes it extremely simply to prevent Configuration Drift.
And as a reference for getting the search domain, Ansible: How to get hostname without domain name?.

Related

How to include variables with include_vars with the same name without overwriting previous

I am having this let's call it include.yaml
#- name: "Playing with Ansible and Include files"
- hosts: localhost
connection: local
tasks:
- find: paths="./" recurse=yes patterns="test.yaml"
register: file_to_exclude
- debug: var=file_to_exclude.stdout_lines
- name: shell
shell: "find \"$(pwd)\" -name 'test.yaml'"
register: files_from_dirs
- debug: var=files_from_dirs.stdout_lines
- name: Include variable files
include_vars: "{{ item }}"
with_items:
- "{{ files_from_dirs.stdout_lines }}"
- debug: var=files
and 2 ore more test files
./dir1/test.yaml
that contains
files:
- file1
- file2
./dir2/test.yaml
that contains
files:
- file3
- file4
the result is
TASK [Include variable files] ******************************************************************************************
ok: [localhost] => (item=/mnt/c/Users/GFlorinescu/ansible_scripts/ansible/1st/test.yaml)
ok: [localhost] => (item=/mnt/c/Users/GFlorinescu/ansible_scripts/ansible/2nd/test.yaml)
TASK [debug] ***********************************************************************************************************
ok: [localhost] => {
"files": [
"file3",
"file4"
]
}
How can I get all the values in files, at the moment the last included files variable from last file overrides the files from the previous files? Of course without changing the variables names in files test.yaml?
In other words I want files to be:
ok: [localhost] => {
"files": [
"file1",
"file2",
"file3",
"file4"
]
}
To be more specific, I ask for any kind of solution or module, even not official or some github module, I don't want a specific include_vars module solution.
Put the included variables into the dictionaries with unique names. For example, create the names from the index of the loop. Then, iterate the names and concatenate the lists
- command: "find {{ playbook_dir }} -name test.yaml"
register: files_from_dirs
- include_vars:
file: "{{ item }}"
name: "{{ name }}"
loop: "{{ files_from_dirs.stdout_lines }}"
loop_control:
extended: true
vars:
name: "files_{{ ansible_loop.index }}"
- set_fact:
files: "{{ files|d([]) + lookup('vars', item).files }}"
with_varnames: "files_[0-9]+"
- debug:
var: files
give
files:
- file1
- file2
- file3
- file4
Notes:
You have to provide either a path relative to the home directory or an absolute path. See the example below
- command: "echo $PWD"
register: out
- debug:
var: out.stdout
give
out.stdout: /home/admin
For example, when you want to find the files relative to the directory of the playbook
- command: "find {{ playbook_dir }} -name test.yaml"
register: files_from_dirs
- debug:
var: files_from_dirs.stdout_lines
give
files_from_dirs.stdout_lines:
- /export/scratch/tmp8/test-987/dir1/test.yaml
- /export/scratch/tmp8/test-987/dir2/test.yaml
The same is valid for the module find. For example,
- find:
paths: "{{ playbook_dir }}"
recurse: true
patterns: test.yaml
register: files_from_dirs
- debug:
var: files_from_dirs.files|map(attribute='path')|list
give the same result
files_from_dirs.files|map(attribute='path')|list:
- /export/scratch/tmp8/test-987/dir1/test.yaml
- /export/scratch/tmp8/test-987/dir2/test.yaml
Simplify the code and put the declaration of files into the vars. For example, the below declaration gives the same result
files: "{{ query('varnames', 'files_[0-9]+')|
map('extract', hostvars.localhost, 'files')|
flatten }}"
Example of a complete playbook for testing
- hosts: localhost
vars:
files: "{{ query('varnames', 'files_[0-9]+')|
map('extract', hostvars.localhost, 'files')|
flatten }}"
tasks:
- find:
paths: "{{ playbook_dir }}"
recurse: true
patterns: test.yaml
register: files_from_dirs
- include_vars:
file: "{{ item }}"
name: "{{ name }}"
loop: "{{ files_from_dirs.files|map(attribute='path')|list }}"
loop_control:
extended: true
vars:
name: "files_{{ ansible_loop.index }}"
- debug:
var: files
(maybe off-topic, see comments)
Q: "Is there a way to write the path where it was found?"
A: Yes, it is. See the self-explaining example below. Given the inventory
shell> cat hosts
host_1 file_1=alice
host_2 file_2=bob
host_3
the playbook
- hosts: host_1,host_2,host_3
vars:
file_1_list: "{{ hostvars|json_query('*.file_1') }}"
file_2_list: "{{ hostvars|json_query('*.file_2') }}"
file_1_dict: "{{ dict(hostvars|dict2items|
selectattr('value.file_1', 'defined')|
json_query('[].[key, value.file_1]')) }}"
file_1_lis2: "{{ hostvars|dict2items|
selectattr('value.file_1', 'defined')|
json_query('[].{key: key, file_1: value.file_1}') }}"
tasks:
- debug:
msg: |-
file_1_list: {{ file_1_list }}
file_2_list: {{ file_2_list }}
file_1_dict:
{{ file_1_dict|to_nice_yaml|indent(2) }}
file_1_lis2:
{{ file_1_lis2|to_nice_yaml|indent(2) }}
run_once: true
gives
msg: |-
file_1_list: ['alice']
file_2_list: ['bob']
file_1_dict:
host_1: alice
file_1_lis2:
- file_1: alice
key: host_1

Ansible - Getting List Values from within a Dictionary (Regsiter Variable)

So I'm working on some audit points using Ansible for many of the servers we support. In most cases, I have had to use the shell modules to get the data I want and then write some files based on pass/fail cases. In a lot of situations, this has been the easier way to work with the output data. First, I realize this isn't necessarily Ansible's forte. I guess at some point it was pitched to the company that it could do this pretty easily, and I would agree - it's easier in many ways than just writing a custom python/BASH script to do the same. So - I do realize I'm bending the concept of Ansible a bit here for reporting, rather than configuration/state management. However; I like the tool and want to show the company we can get a lot of value from it.
While I could do this section easily using the shell module, I would like to learn Ansible a bit better. So thought I would post this question.
I'm using the Yum module to just get a repolist on the target hosts. But I've been running into confusion on just how to extract the list values nested in the output dictionary. I have done some checking on the types and as far as I can tell - the 'results' variable is a dictionary, with the output in in a list. What I want to do, is get the key/values from the list and then perform some other tasks based on that output. But for the life of me - I can't figure out how to do this!
Ideally - I would like to either use some 'when' module statements based on the output (When the repo ID is.. do this.. for example) or at least be able to store them in a variable to work with the data. So from this output, I just want to get the repoid and if it's enabled. How can I get these values from the nested list?
Simple Playbook:
---
- hosts: localhost
become: yes
tasks:
- name: Section 1.1 - Check Yum Repos
yum:
list: repos
register: section1_1
- name: Debug
debug:
var: section1_1
Here is my output from the debug task in this playbook:
TASK [Debug] ****************************************************************************************************************************************************
ok: [localhost] => {
"section1_1": {
"changed": false,
"failed": false,
"results": [
{
"repoid": "ansible",
"state": "enabled"
},
{
"repoid": "epel",
"state": "enabled"
},
{
"repoid": "ol7_UEKR6",
"state": "enabled"
},
{
"repoid": "ol7_latest",
"state": "enabled"
}
]
}
}
I suspect this might be easy for someone out there. I've been trying this and that's for quite a while now and finally got to the point where I thought I would just ask :)
As the output of registered in section1_1 is a list of dictionaries. We can loop through each item, to get the dictionary keys.
Example:
- name: Get the first repo's repoid and state
debug:
msg: "Repo ID: {{ results[0]['repoid'] }}, is {{ results[0]['state'] }}"
# This will show -- Repo ID: ansible, is enabled
Similarly we can access other elements with their number.
Or we can loop through each element of array:
- name: loop through array and conditionally do something
debug:
msg: "Repo ID is {{ item.repoid }}, so I am going to write a playbook."
when: item.repoid == 'ansible'
loop: "{{ results }}"
Q: "Get the key/values from the list."
A: There are more options. Given the data below
section1_1:
changed: false
failed: false
results:
- repoid: ansible
state: enabled
- repoid: epel
state: enabled
- repoid: ol7_UEKR6
state: enabled
- repoid: ol7_latest
state: enabled
- repoid: test
state: disabled
1a) Get the keys and values, and create a dictionary
_keys1: "{{ section1_1.results|map(attribute='repoid')|list }}"
_vals1: "{{ section1_1.results|map(attribute='state')|list }}"
repos1: "{{ dict(_keys1|zip(_vals1)) }}"
gives
_keys1: [ansible, epel, ol7_UEKR6, ol7_latest, test]
_vals1: [enabled, enabled, enabled, enabled, disabled]
repos1:
ansible: enabled
epel: enabled
ol7_UEKR6: enabled
ol7_latest: enabled
test: disabled
1b) The filter items2dict gives the same result
repos2: "{{ section1_1.results|
items2dict(key_name='repoid', value_name='state') }}"
1c) The filter json_query gives also the same result
repos3: "{{ dict(section1_1.results|
json_query('[].[repoid, state]')) }}"
Iterate the dictionary
- debug:
msg: "Repo {{ item.key }} is {{ item.value }}"
loop: "{{ repos1|dict2items }}"
gives (abridged)
msg: Repo ansible is enabled
msg: Repo epel is enabled
msg: Repo ol7_UEKR6 is enabled
msg: Repo ol7_latest is enabled
msg: Repo test is disabled
The next option is the conversion of the values to boolean
_vals4: "{{ section1_1.results|
json_query('[].state.contains(#, `enabled`)') }}"
repos4: "{{ dict(_keys1|zip(_vals4)) }}"
gives
_vals4: [true, true, true, true, false]
repos4:
ansible: true
epel: true
ol7_UEKR6: true
ol7_latest: true
test: false
Iterate the dictionary
- debug:
msg: "Repo {{ item.key }} is enabled: {{ item.value }}"
loop: "{{ repos4|dict2items }}"
gives (abridged)
msg: 'Repo ansible is enabled: True'
msg: 'Repo epel is enabled: True'
msg: 'Repo ol7_UEKR6 is enabled: True'
msg: 'Repo ol7_latest is enabled: True'
msg: 'Repo test is enabled: False'
3a) The list of the enabled repos can be easily selected
- debug:
msg: "Repo {{ item.key }} is enabled"
loop: "{{ repos4|dict2items|selectattr('value') }}"
gives (abridged)
msg: Repo ansible is enabled
msg: Repo epel is enabled
msg: Repo ol7_UEKR6 is enabled
msg: Repo ol7_latest is enabled
3b), or rejected
- debug:
msg: "Repo {{ item.key }} is disabled"
loop: "{{ repos4|dict2items|rejectattr('value') }}"
gives (abridged)
msg: Repo test is disabled
Example of a complete playbook for testing
- hosts: localhost
vars:
section1_1:
changed: false
failed: false
results:
- {repoid: ansible, state: enabled}
- {repoid: epel, state: enabled}
- {repoid: ol7_UEKR6, state: enabled}
- {repoid: ol7_latest, state: enabled}
- {repoid: test, state: disabled}
_keys1: "{{ section1_1.results|map(attribute='repoid')|list }}"
_vals1: "{{ section1_1.results|map(attribute='state')|list }}"
repos1: "{{ dict(_keys1|zip(_vals1)) }}"
repos2: "{{ section1_1.results|
items2dict(key_name='repoid', value_name='state') }}"
repos3: "{{ dict(section1_1.results|
json_query('[].[repoid, state]')) }}"
_vals4: "{{ section1_1.results|
json_query('[].state.contains(#, `enabled`)') }}"
repos4: "{{ dict(_keys1|zip(_vals4)) }}"
tasks:
- debug:
var: section1_1
- debug:
var: _keys1|to_yaml
- debug:
var: _vals1|to_yaml
- debug:
var: repos1
- debug:
var: repos2
- debug:
var: repos3
- debug:
msg: "Repo {{ item.key }} is {{ item.value }}"
loop: "{{ repos1|dict2items }}"
- debug:
var: _vals4|to_yaml
- debug:
var: repos4
- debug:
msg: "Repo {{ item.key }} is enabled: {{ item.value }}"
loop: "{{ repos4|dict2items }}"
- debug:
msg: "Repo {{ item.key }} is enabled"
loop: "{{ repos4|dict2items|selectattr('value') }}"
- debug:
msg: "Repo {{ item.key }} is disabled"
loop: "{{ repos4|dict2items|rejectattr('value') }}"

Ansible search sublists for value

A webhook triggers an AWX job and I want to run the deployment on a certain host depending on the service, since they run on different servers. I need to know which server uses that service to set is as a var so it can be used as a host in the following play.
My variable inside vars.yaml looks like this:
staging_hosts:
server1: ['service1', 'service2', 'service3']
server2: ['service4', 'service5', 'service6']
server3: ['service7', 'service8', 'service9']
Playbook:
- name: write deployment hosts
hosts: localhost
vars:
deployment_hosts: absent
vars_files:
- ./group_vars/vars.yaml
tasks:
- set_fact:
modified_repos: (small regex filter to find modified repository)
- set_fact:
deployment_hosts: "{{ item }}"
when: '{{ modified_repos }} in {{ item }}'
with_list:
- "{{ staging_hosts }}"
- name: connect to Cluster
hosts: "{{ hostvars['localhost']['deployment_hosts'] }}"
What can I do against this warning and error?
[WARNING]: conditional statements should not include jinja2 templating
delimiters such as {{ }} or {% %}. Found: {{ modified_repos }} in {{ item }}
fatal: [localhost]: FAILED! => {"msg": "The conditional check '{{ modified_repos }} in {{ item }}' failed. True {% else %} False {% endif %}): unhashable type: 'list'
Oh I forgot to mention. It is important, that deployment_hosts could also contain two hosts if modified repos include for example service1 and service4.
Q: "deployment_hosts could also contain two hosts if modified repos include for example service1 and service4."
A: Use intersect filter. For example, the playbook
- hosts: localhost
vars:
staging_hosts:
server1: ['service1', 'service2', 'service3']
server2: ['service4', 'service5', 'service6']
server3: ['service7', 'service8', 'service9']
modified_repos: ['service1', 'service4']
tasks:
- set_fact:
deployment_hosts: "{{ deployment_hosts|default([]) + [item.key] }}"
loop: "{{ staging_hosts|dict2items }}"
when: modified_repos|intersect(item.value)|length > 0
- debug:
var: deployment_hosts
gives
deployment_hosts:
- server1
- server2

Sync files between 2 hosts using ansible

I'm looking for a way to sync files between 2 hosts in ansible. The scenario is as follows. I have a CSV file which contains 3 columns indicating directories which needs to be synced between 2 servers. the first 2 columns indicate the source and target servers and the third column indicates the directory
source, target, directory
src01, tgt02, dir0003
src02, tgt05, dir0004
src10, tgt68, dir1022
I found this answer for syncing files between 2 hosts - How to copy files between two nodes using ansible
Is there any way to parameterize this using a csv config file?
Yes. It's possible. In the first play read the CSV file and create group of targets. Use the new group in the second play and loop the synchronize module. For example the playbook
- hosts: localhost
tasks:
- read_csv:
path: db.csv
register: my_db
- add_host:
hostname: "{{ item.target }}"
groups: my_targets
my_list: "{{ my_db.list }}"
loop: "{{ my_db.list }}"
- hosts: my_targets
tasks:
- debug:
msg: "Copy {{ item.directory }} from {{ item.source }} to {{ inventory_hostname }}"
loop: "{{ my_list|json_query(query) }}"
vars:
query: "[?target == '{{ inventory_hostname }}']"
- name: Copy
synchronize:
src: "{{ item.directory }}"
dest: "{{ item.directory }}"
delegate_to: "{{ item.source }}"
loop: "{{ my_list|json_query(query) }}"
vars:
query: "[?target == '{{ inventory_hostname }}']"
(not tested)
gives
"msg": "Copy dir0004 from src02 to tgt05"
"msg": "Copy dir0003 from src01 to tgt02"
"msg": "Copy dir1022 from src10 to tgt68"

Adding field to dict items

Consider the following play. What I am trying to do is add a field, tmp_path which is basically the key and revision appended together to each element in the scripts dict.
---
- hosts: localhost
connection: local
gather_facts: no
vars:
scripts:
a.pl:
revision: 123
b.pl:
revision: 456
tasks:
- with_dict: "{{ scripts }}"
debug:
msg: "{{ item.key }}_{{ item.value.revision }}"
# - with_items: "{{ scripts }}"
# set_fact: {{item.value.tmp_path}}="{{item.key}}_{{item.value.revision}}"
# - with_items: "{{ scripts }}"
# debug:
# msg: "{{ item.value.tmp_path }}"
...
Obviously the commented code doesn't work, any idea how I can get this working? Is it possible to alter the scripts dict directly, or should I somehow be creating a new dict to reference instead?
By the way welcome to correct the terminology for what I am trying to do.
OK, I think I got a solution (below), at least to let me move forwards with this. Disadvantages are it has removed the structure of my dict and also seems a bit redundant having to redefine all the fields and use a new variable, If anyone can provide a better solution I will accept that instead.
---
- hosts: localhost
connection: local
gather_facts: no
vars:
scripts:
a.pl:
revision: 123
b.pl:
revision: 456
tasks:
- with_dict: "{{ scripts }}"
debug:
msg: "{{ item.key }}_{{ item.value.revision }}"
- with_dict: "{{ scripts }}"
set_fact:
new_scripts: "{{ (new_scripts | default([])) + [ {'name': item.key, 'revision': item.value.revision, 'tmp_path': item.key ~ '_' ~ item.value.revision}] }}"
# - debug:
# var: x
# - with_dict: "{{ scripts }}"
- with_items: "{{ new_scripts }}"
debug:
msg: "{{ item.tmp_path }}"
...
BTW credit to the following question which pointed me in the right direction:
Using Ansible set_fact to create a dictionary from register results

Resources