Inconsistent list indent in Ansible/Yaml file - yaml

I have an Ansible YAML file formatted like below. Notice the - name and - when parts.
- name: populate package facts
package_facts:
- name: install auditd
apt:
name: auditd
state: present
register: apt_status
until: apt_status is success
retries: 12
delay: 10
- name: touch stig.rules
file:
path: /etc/audit/rules.d/stig.rules
state: touch
mode: '0640'
modification_time: preserve
access_time: preserve
# R-238197 UBTU-20-010002
- name: stigrule_238197__etc_gdm3_greeter_dconf_defaults_enable
ini_file:
path: /etc/gdm3/greeter.dconf-defaults
section: org/gnome/login-screen
option: banner-message-enable
value: "{{ ubuntu2004STIG_stigrule_238197__etc_gdm3_greeter_dconf_defaults_enable_Value }}"
no_extra_spaces: yes
notify: dconf_update
when:
- ubuntu2004STIG_stigrule_238197_Manage
- "'gdm3' in packages"
I process it and remove some elements then dump to a file using the method below:
def dump(path: str, export: list) -> None:
yaml: YAML = YAML()
yaml.default_flow_style = False
yaml.preserve_quotes = True
yaml.width = 4096 # Prevent line breaks
yaml.indent(mapping=2, sequence=2, offset=0) # Default values
with open(path, "w", encoding=ENCODING) as fp:
yaml.dump(export, fp)
With default indentation the result will be like below:
- name: populate package facts
package_facts:
- name: install auditd
apt:
name: auditd
state: present
register: apt_status
until: apt_status is success
retries: 12
delay: 10
- name: touch stig.rules
file:
path: /etc/audit/rules.d/stig.rules
state: touch
mode: '0640'
modification_time: preserve
access_time: preserve
# R-238197 UBTU-20-010002
- name: stigrule_238197__etc_gdm3_greeter_dconf_defaults_enable
ini_file:
path: /etc/gdm3/greeter.dconf-defaults
section: org/gnome/login-screen
option: banner-message-enable
value: "{{ ubuntu2004STIG_stigrule_238197__etc_gdm3_greeter_dconf_defaults_enable_Value }}"
no_extra_spaces: yes
notify: dconf_update
when:
- ubuntu2004STIG_stigrule_238197_Manage
- "'gdm3' in packages"
As you can see, the Ansible file is a list of elements already, starting from offset 0. Then, the list items after - when will also use that 0 offset value. How can I preserve the original indentation -offset 0 for - name but offset 2 for all others?

With ruamel.yaml you can only have one indent for all mappings and one for all sequences.
If you want to preserve the inconsistency, you need to indent with 4 and an offset 2 and remove the
extra two spaces that start every line (effectively dedenting the root level sequences):
import sys
import ruamel.yaml
YAML = ruamel.yaml.YAML
from pathlib import Path
def dump(path: str, export: list) -> None:
def strip_first_two(s):
res = []
for x in s.splitlines(True):
xl = x.lstrip()
# do not dedent full comment lines
if xl and xl[0] == '#' or not x.startswith(' '):
res.append(x)
else:
res.append(x[2:])
return ''.join(res)
yaml: YAML = YAML()
yaml.default_flow_style = False # default
yaml.width = 4096 # Prevent line breaks
yaml.indent(mapping=2, sequence=4, offset=2)
yaml.dump(export, Path(path), transform=strip_first_two)
yaml = ruamel.yaml.YAML()
yaml.preserve_quotes = True
data = yaml.load(Path('input.yaml'))
dump('output.yaml', data)
print(Path('output.yaml').read_text())
which gives:
- name: populate package facts
package_facts:
- name: install auditd
apt:
name: auditd
state: present
register: apt_status
until: apt_status is success
retries: 12
delay: 10
- name: touch stig.rules
file:
path: /etc/audit/rules.d/stig.rules
state: touch
mode: '0640'
modification_time: preserve
access_time: preserve
# R-238197 UBTU-20-010002
- name: stigrule_238197__etc_gdm3_greeter_dconf_defaults_enable
ini_file:
path: /etc/gdm3/greeter.dconf-defaults
section: org/gnome/login-screen
option: banner-message-enable
value: "{{ ubuntu2004STIG_stigrule_238197__etc_gdm3_greeter_dconf_defaults_enable_Value }}"
no_extra_spaces: yes
notify: dconf_update
when:
- ubuntu2004STIG_stigrule_238197_Manage
- "'gdm3' in packages"
If you have any end-of-line comments than these will shift two positions to the left.
If you don't have multi-line scalars that have embedded comment markers you can easily handled
that in strip_first_two by replacing ' #' with ' #' in each line. If you have both this is not so trivial
and you would have to walk the data structure before dumping and adjust each start column of each EOL comment.
Setting yaml.default_flow_style = False is not necessary as that is the default for round-trip dumping (on the other hand explicit
is better than implicit). But you shouldn't set yaml.preserve_quotes on the YAML instance in your dump function as that only
works during loading of a YAML document, so that could actually mislead someone.
I also tend to use pathlib.Path instances and let ruamel.yaml do the right thing with regards to encoding and opening ('w' vs 'wb').

Related

How to search for a string in a remote file using Ansible?

Based on a questions
How to search for a string in a file using Ansible?
Ansible: How to pull a specific string out of the contents of a file?
Can slurp be used as a direct replacement for lookup?
and considerations like
By using the slurp module one is going to transfer the whole file from the Remote Node to the Control Node over the network just in order to process it and looking up a string. For log files these can be several MB and whereby one is mostly interested only in the information if the file on the Remote Node contains a specific string and therefore one would only need to transfer that kind of information, true or false.
How to execute a script on a Remote Node using Ansible?
I was wondering how this can be solved instead of using the shell module?
---
- hosts: localhost
become: false
gather_facts: false
vars:
SEARCH_STRING: "test"
SEARCH_FILE: "test.file"
tasks:
- name: Search for string in file
command:
cmd: "grep '{{ SEARCH_STRING }}' {{ SEARCH_FILE }}"
register: result
# Since it is a reporting task
# which needs to deliver a result in any case
failed_when: result.rc != 0 and result.rc != 1
check_mode: false
changed_when: false
Or instead of using a workaround with the lineinfile module?
---
- hosts: localhost
become: false
gather_facts: false
vars:
SEARCH_STRING: "test"
SEARCH_FILE: "test.file"
tasks:
- name: Search for string
lineinfile:
path: "{{ SEARCH_FILE }}"
regexp: "{{ SEARCH_STRING }}"
line: "SEARCH_STRING FOUND"
state: present
register: result
# Since it is a reporting task
changed_when: false
failed_when: "'replaced' not in result.msg" # as it means SEARCH_STRING NOT FOUND
check_mode: true # to prevent changes and to do a dry-run only
- name: Show result, if not found
debug:
var: result
when: "'added' in result.msg" # as it means SEARCH_STRING NOT FOUND
Since I am looking for a more generic approach, could it be a feasible case for Should you develop a module?
Following Developing modules and Creating a module I've found the following simple solution with
Custom Module library/pygrep.py
#!/usr/bin/python
from __future__ import (absolute_import, division, print_function)
__metaclass__ = type
from ansible.module_utils.basic import AnsibleModule
def run_module():
module_args = dict(
path=dict(type='str', required=True),
search_string=dict(type='str', required=True)
)
result = dict(
changed=False,
found_lines='',
found=False
)
module = AnsibleModule(
argument_spec=module_args,
supports_check_mode=True
)
with open(module.params['path'], 'r') as f:
for line in f.readlines():
if module.params['search_string'] in line:
result['found_lines'] = result['found_lines'] + line
result['found'] = True
result['changed'] = False
module.exit_json(**result)
def main():
run_module()
if __name__ == '__main__':
main()
Playbook pygrep.yml
---
- hosts: localhost
become: false
gather_facts: false
vars:
SEARCH_FILE: "test.file"
SEARCH_STRING: "test"
tasks:
- name: Grep string from file
pygrep:
path: "{{ SEARCH_FILE }}"
search_string: "{{ SEARCH_STRING }}"
register: search
- name: Show search
debug:
var: search
when: search.found
For a simple test.file
NOTEST
This is a test file.
It contains several test lines.
321tset
123test
cbatset
abctest
testabc
test123
END OF TEST
it will result into an output of
TASK [Show search] ******************
ok: [localhost] =>
search:
changed: false
failed: false
found: true
found_lines: |-
This is a test file.
It contains several test lines.
123test
abctest
testabc
test123
Some Links
What is the grep equivalent in Python?
cat, grep and cut - translated to Python
What is the Python code equivalent to Linux command grep -A?

ansible updating a YAML file using set_fact/update_fact

My ansible project consists of 6 roles (app, db, init, ror, tick, web). Each role represents the installation of one component of the system. Each device keeps track of which roles have been installed using the file "/etc/ansible/ansible-install-state.yml". It's initial contents are:
# ansible-install-state.yml
#
#
# This file is maintained by Ansible
# Please leave it alone, or accept the consequences!
#
roles:
app:
installed: false
db:
installed: false
init:
installed: false
ror:
installed: false
tick:
installed: false
web:
installed: false
At the completion of the 'init' role, I want to process the file to update the installed state to 'true'. I do that with the following call. (For testing, I'm also setting 'ror: true' and adding a new role 'new: unknown'
- name: System Init | Process ansible_install_state
ansible.builtin.include_tasks: process_ansible_install_state.yml
with_items:
- role: { key: init, value: true }
- role: { key: ror, value: true }
- role: { key: new, value: unknown}
Updating existing roles went fairly smoothly, but adding a new role did not. I finally got it to work by first adding the role itself, essentially 'new: {}', then modifying it again to add the 'installed: unknown' part. Although this does work, I can't help but believe there's a better way - a way to do it all at once. Following is the process_enable_install_state.yml file.
---
# process_ansible_install_state.yml
#
#------------------------- Process Ansible Install State File
#
- name: display variable
debug:
var: item
# See https://stackoverflow.com/questions/32994002/is-there-a-yaml-editing-module-for-ansible
- name: Ansible Install State | Read ansible-install-state.yml
ansible.builtin.slurp:
path: "/etc/ansible/ansible-install-state.yml"
register: install_state
- name: Ansible Install State | Decode and save yaml as fact
ansible.builtin.set_fact:
my_roles: "{{ install_state['content'] | b64decode | from_yaml }}"
register: its_a_fact
- name: Ansible Install State | Save the new ROLE block
block:
- name: Ansible Install State | Save the new role
ansible.utils.update_fact:
updates:
- path: my_roles.roles.{{ item.role.key }}
value: {}
register: updated
# Update_fact does not update 'in place', so we need to save it as a fact
- name: Ansible Install State | Update the saved 'fact'
ansible.builtin.set_fact:
my_roles: "{{ updated.my_roles}}"
when: my_roles.roles[item.role.key] is undefined
- name: debug
debug:
var: updated
- name: Ansible Install State | Update the role's installed state
ansible.utils.update_fact:
updates:
- path: my_roles.roles.{{ item.role.key }}.installed
value: "{{item.role.value}}"
register: updated
- name: debug
debug:
var: updated
# Update_fact does not update 'in place', so we need to save it as a fact
- name: Ansible Install State | Save modified as 'fact'
ansible.builtin.set_fact:
my_roles: "{{ updated.my_roles}}"
- name: Ansible Install State | Write Yaml File
ansible.builtin.copy:
content: '{{ my_roles | to_nice_yaml }}'
dest: "/etc/ansible/ansible-install-state.yml"
- name: Ansible Install State | Insert Header Comments
ansible.builtin.blockinfile:
path: "/etc/ansible/ansible-install-state.yml"
insertbefore: BOF
marker: "#{mark}"
marker_begin: "-------------------------------------------"
marker_end: "-------------------------------------------"
block: |
# ansible-install-state.yml
#
#
# This file is maintained by Ansible
# Please leave it alone, or accept the consequences!
#
#
This successfully produced...
# ansible-install-state.yml
#
#
# This file is maintained by Ansible
# Please leave it alone, or accept the consequences!
#
#
#-------------------------------------------
roles:
app:
installed: false
db:
installed: false
init:
installed: true
new:
installed: unknown
ror:
installed: true
tick:
installed: false
web:
installed: false
deploy#rpi-tick2:/etc/ansible $

Ansible: Modify cmdline.txt on Raspberry Pi

I am modifying /boot/cmdline.txt to add container features to a Raspberry Pi, so I need to add cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory into the file, within the same line.
I am trying to do it with the lineinfile module without much success:
- hosts: mypi
become: yes
tasks:
- name: Enable container features
lineinfile:
path: /boot/cmdline.txt
regex: " cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory"
line: " cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory"
insertafter: EOF
state: present
I have been trying modifying the insertafter to BOF, using insertbefore too, using a regex to match the last word... But it ends up adding a carriage return. I have been unable to find some way to not add a new line.
As Vladimir pointed out, Jack's answer unfortunately is not sufficient for empty files and also fails if the desired argument already exists at the beginning of the line.
The following suggested solution should address those issues. In particular, it is supposed to
support empty files,
support existing arguments at any position within the string,
be robust even with multi-line files (just in case...),
be idempotent, and
optionally update existing keys with the desired value.
# cmdline.yml
- name: read cmdline.txt
become: true
slurp: "src={{ cmdline_txt_path }}"
register: result_cmdline
- name: generate regular expression for existing arguments
set_fact:
regex_existing: '{{ "\b" + key|string + "=" + ("[\w]*" if update else value|string) + "\b" }}'
key_value_pair: '{{ key|string + "=" + value|string }}'
- name: generate regular expression for new arguments
set_fact:
regex_add_missing: '{{ "^((?!(?:.|\n)*" + regex_existing + ")((?:.|\n)*))$" }}'
- name: update cmdline.txt
become: true
copy:
content: '{{ result_cmdline.content
| b64decode
| regex_replace(regex_existing, key_value_pair)
| regex_replace(regex_add_missing, key_value_pair + " \1")
}}'
dest: "{{ cmdline_txt_path }}"
Usage:
- set_fact:
cmdline_txt_path: /boot/cmdline.txt
- include_tasks: cmdline.yml
vars:
key: cgroup_enable
value: memory
update: false
# will add the argument if the key-value-pair doesn't exist
- include_tasks: cmdline.yml
vars:
key: cgroup_enable
value: cpu
update: false
- include_tasks: cmdline.yml
vars:
key: cgroup_memory
value: 1
update: true
# will replace the value of the first matching key, if found;
# will add it if it's not found
However, I might have missed some edge cases - please let me know if you find any issues.
Since you only have the one line in the file, you can do that with either replace or lineinfile. Here is the replace version:
- name: Enable container features
replace:
path: cmdline.txt
regexp: '^([\w](?!.*\b{{ item }}\b).*)$'
replace: '\1 {{ item }}'
with_items:
- "cgroup_enable=cpuset"
- "cgroup_memory=1"
- "cgroup_enable=memory"
Stole the answer from here
I followed all of the different strategies laid out above, but in the end, I wanted something simple, as this is my first playbook, and I need to understand it now, and when I pick it up again later,.
My cmdline.txt contained multiple lines:
cat /boot/cmdline.txt -E
console=serial0,115200 console=tty1 rootfstype=ext4 fsck.repair=yes rootwait cgroup_enable=memory cgroup_enable=cpuset cgroup_memory=1$
dtoverlay=pi3-disable-bt$
dtoverlay=pi3-disable-wifi$
So, the approach I was looking for:
Would ignore the other configurations in the cmdline.txt
Would only add the add a specific key=value if it was missing
It had to be idempotent
I settled on a simple regex to decide if this was the row I wanted to edit:
If the row contaied console= (as this is the row I'm after)
AND.. If the row does not contain cgroup_memory=1
- name: Adding cgroup_enable=memory to boot parameters for k3s
lineinfile:
path: /boot/cmdline.txt
state: present
regexp: '^((?!.*cgroup_enable=memory).*console.*)$'
line: '\1 cgroup_enable=memory'
backrefs: yes
notify: reboot
- name: Adding cgroup_enable=cpuset to boot parameters for K3s
lineinfile:
path: /boot/cmdline.txt
state: present
regexp: '^((?!.*cgroup_enable=cpuset).*console.*)$'
line: '\1 cgroup_enable=cpuset'
backrefs: yes
notify: reboot
- name: Adding cgroup_memory=1 to boot parameters for K3s
lineinfile:
path: /boot/cmdline.txt
state: present
regexp: '^((?!.*cgroup_memory=1).*console.*)$'
line: '\1 cgroup_memory=1'
backrefs: yes
notify: reboot
And at some point in future, I'll probably condense all three of these into a single loop task. But not today.
Q: "Ansible lineinfile module: Do not add new line. Find some way to not add a new line."
A: It's not possible. New line will be always added by module lineinfile. See source for example
b_lines.insert(index[1], b_line + b_linesep)
This is how a new line is added. Such additions will be terminated with b_linesep. See how the variable is defined
b_linesep = to_bytes(os.linesep, errors='surrogate_or_strict')
The os.linesep is used when you want to iterate through the lines of a text file. The internal scanner recognizes the os.linesep and replaces it with a single "\n".
See What is os.linesep for?.
The task with the module replace doesn't solve this problem either. Neither it creates the line without a newline, nor it modifies existing one this way. In addition to this it's not idempotent.
- name: Enable container features
replace:
path: cmdline.txt
regexp: '^([\w](?!.*\b{{ item }}\b).*)$'
replace: '\1 {{ item }}'
loop:
- "cgroup_enable=cpuset"
- "cgroup_memory=1"
- "cgroup_enable=memory"
It will do nothing if the file is empty
TASK [Enable container features]
ok: [localhost] => (item=cgroup_enable=cpuset)
ok: [localhost] => (item=cgroup_memory=1)
ok: [localhost] => (item=cgroup_enable=memory)
If the line is present in the file this task will change it
shell> cat cmdline.txt
cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory
ASK [Enable container features] *****************************************
ok: [localhost] => (item=cgroup_enable=cpuset)
--- before: cmdline.txt
+++ after: cmdline.txt
## -1 +1 ##
-cgroup_memory=1 cgroup_enable=memory cgroup_enable=cpuset
+cgroup_memory=1 cgroup_enable=memory cgroup_enable=cpuset cgroup_memory=1

How can I copy a multi-line string to a file with literal newlines?

So I know I can do something like this:
copy:
dest: /etc/issue
content: |
Hello
World
But this doesn't work:
vars:
login_banner_text: !!str |-
"Hello\nWorld"
tasks:
- name: Set TTY login banner
copy:
dest: /etc/issue
content: "{{ login_banner_text }}"
The newline character is printed straight to the file, without parsing, i.e. it's a single line populated with \n strings. I'd like to do this without copying a file into place, because I need to copy this text to two files.
For one file, the \n strings need to remain unparsed, so it's written to file as a single line. For the other, I want the \n to be interpreted, so that the text is expanded into multiple lines.
The first file is being modified using the ini_file module. This task works as expected using the explicit \n in the variable declaration.
- name: "Set message"
ini_file:
dest: /etc/dconf/db/gdm.d/00-security-settings
section: org/gnome/login-screen
option: banner-message-text
value: string '{{ login_banner_text }}'
create: yes
tags:
- always
However, other modules behave this way as well.
If I copy a file into place, the rather long text (not "Hello World") has to be maintained in two places.
Update
I found, what I think is, a better way of doing this, based on this post. It stores the banner in a separate file, and then uses that to modify both configuration files. So the value is only stored in one place. However, the answer given by #larsks does answer the question as it was originally posed.
- hosts: 127.0.0.1
connection: local
vars:
login_banner_text: "{{ lookup('file', 'login_banner.txt') }}"
tasks:
- name: "Set the GNOME3 Login Warning Banner Text"
ini_file:
dest: /etc/dconf/db/gdm.d/00-security-settings
section: org/gnome/login-screen
option: banner-message-text
value: '{{ login_banner_text | to_json }}'
create: yes
tags:
- always
- name: "Set the TTY Login Warning Banner Text"
copy:
dest: '/etc/issue'
content: "{{ '\n' + login_banner_text + '\n' }}"
tags:
- always
You already know how to specify a value with literal newlines; you're doing it when setting the content key in your first example. You can set the value of a variable the same way:
---
- hosts: localhost
gather_facts: false
vars:
mytext: |
Hello
World
tasks:
- copy:
dest: ./output.txt
content: "{{ mytext }}"
This would create output.txt with the content:
Hello
World
If instead your goal is to have content like this...
[org/gnome/login-screen]
banner-message-text = "Hello\nWorld"
...then you don't want literal newlines, you want a literal \n, in which case this would work:
---
- hosts: localhost
gather_facts: false
vars:
mytext: "Hello\\nWorld"
tasks:
- ini_file:
dest: ./example.ini
section: org/gnome/login-screen
option: banner-message-text
value: "{{ mytext }}"
create: true
Which would result in:
[org/gnome/login-screen]
banner-message-text = Hello\nWorld
If you wanted the value in the config file quoted, then:
- ini_file:
dest: ./example.ini
section: org/gnome/login-screen
option: banner-message-text
value: '"{{ mytext }}"'
create: true
Which gets you:
[org/gnome/login-screen]
banner-message-text = "Hello\nWorld"
You could also do it like this:
---
- hosts: localhost
gather_facts: false
vars:
mytext: |-
Hello
World
tasks:
- ini_file:
dest: ./example.ini
section: org/gnome/login-screen
option: banner-message-text
value: '{{ mytext|to_json }}'
create: true
This gets you the same output as the previous example.

Registering multiple variables in a loop

I have a variable yaml file:
---
apps:
- client
- node
- gateway
- ic
- consul
and this task:
- name: register if apps exists
stat: path="/etc/init.d/{{ item }}"
with_items: apps
register: "{{ item.stat.exists }}exists"
I need to end up with a variable for each app with a value of true or false whether the file exists or not:
clientexists = true
nodeexists = true
gatewayexists = false
icexists = true
consulexists = false
For some reason, the item and exists concat is not working.
How can I achieve that??
Try this hope this will help you out. While looping in Stats.yml then msg field will contain your desired output.
Variables.yml Here we are defining variables
---
apps:
- client
- node
- gateway
- ic
- consul
Stats.yml
---
- hosts: localhost
name: Gathering facts
vars_files:
- /path/to/variables.yml
tasks:
- name: "Here we are printing variables"
debug:
msg: "{{apps}}"
- name: "Here we are gathering stats and registering it"
stat:
path: "/etc/init.d/{{item}}"
register: folder_stats
with_items:
- "{{apps}}"
- name: "Looping over registered variables, Its msg field will contain desired output"
debug:
msg: "{{item.invocation.module_args.path}}: {{item.stat.exists}}"
with_items:
- "{{folder_stats.results}}"
...
folder_stats will contain whole result, for individually referring to single - single result You can use it like this in you playbook.
folder_stats.results[0].stat.exists
folder_stats.results[1].stat.exists
folder_stats.results[2].stat.exists
folder_stats.results[3].stat.exists

Resources