SaltStack: Reverse engineering where a file comes from - debugging

If you look at a host which was set up be SaltStack, then it is sometimes like looking at a binary file with vi.
You have no clue how the config/file was created.
This makes trouble shooting errors hard. Reverse engineering where a file comes from takes too much time.
My goal: Make it easy to find the way from looking at the unix config file on the minion (created by salt) to the source where this configuration came from. Like $Id$ in svn and cvs.
One idea a friend and I had:
The state file.managed should (optionally) add the source of the file.
Example:
My sls file contains this:
file_foo_bar:
file.managed:
- source:
- salt://foo/bar
Then the created file should contain this comment.
# Source: salt://foo/bar
Of course this is not simple, since there are different ways to put comments into configuration files.
Is this feasible? Or is there a better solution to my goal.
Update
Usually I know what I did wrong and can find the root easily. The problem arises if several people work on a state tree.

This is a starting point where you can get the date and time of the modified file when its managed by Salt by using Salt Pillar.
Lets call our variable salt_managed. Create a pillar file like the following:
{% set managed_text = 'Salt managed: File modified on ' + salt.cmd.run('date "+%Y-%m-%d %H:%M:%S"') %}
salt_managed: {{ managed_text | yaml_dquote }}
Then on the minion when you call the pillar you will get the following result:
$ salt-call pillar.get salt_managed
local:
Salt managed: File modified on 2016-10-18 11:12:40
And you can use this by adding it on the top of your config files for example like this:
{{ pillar.get('salt_managed') }}
Update:
I found a work around that might be useful for someone. Lets say we have a multiple states that could modify the same file. How can we know that State X is the responsible for modifying that file ? by doing the following steps:
1- I have created a state like this one:
Create a File:
file.managed:
- name: /path/to/foofile
- source: salt://statedir/barfile
Add file header:
file.prepend:
- name: /path/to/foofile
- text: "This file was managed by using this salt state {{ sls }}"
The contents of barfile is:
This is a new file
2- Call the state from the minion and this will be the result:
$ salt-call state.sls statedir.test
local:
----------
ID: Create a File
Function: file.managed
Name: /path/to/foofile
Result: True
Comment: File /path/to/foofile updated
Started: 07:50:45.254994
Duration: 1034.585 ms
Changes:
----------
diff:
New file
mode:
0644
----------
ID: Add file header
Function: file.prepend
Name: /path/to/foofile
Result: True
Comment: Prepended 1 lines
Started: 07:50:46.289766
Duration: 3.69 ms
Changes:
----------
diff:
---
+++
## -1,1 +1,2 ##
+This file was managed by using this salt state statedir.test
This is a new file
Summary for local
------------
Succeeded: 2 (changed=2)
Failed: 0
------------
Total states run: 2
Currently the content of foofile is:
This file was managed by using this salt state statedir.test
This is a new file

Related

Pre-commit - /dev/tty block all output text defined in hook (ex: through echo) before entering user input

I'm trying to create my own hook (defined in terraform_plan.sh, can refer in terraform_plan.sh and .pre-commit-config.yaml below) and require user input to determine if this hook success or fail (This hook is about some checking by the user before commit). To activate user input function, I add exec < /dev/tty according to How do I prompt the user from within a commit-msg hook?.
The snippet code looks like this (terraform_plan.sh).
#!/bin/sh
location=$(pwd)
echo "location: ${location}"
cd ./tf_build/
project_id=$(gcloud config get-value project)
credentials=$(gcloud secrets versions access latest --secret="application-default-credentials")
echo "PROJECT_ID: ${project_id}"
echo "CREDENTIALS: ${credentials}"
terraform plan -var "PROJECT_ID=${project_id}" -var "APPLICATION_DEFAULT_CREDENTIALS=${credentials}"
exec < /dev/tty
read -p "Do yu agree this plan? (Y/N): " answer
echo "answer: ${answer}"
# for testing purpose, assign 0 directly
exit 0
I expect that the prompt Do you agree this plan? (Y/N) should appear before I can enter my answer. But actually nothing shown and it just hangs there waiting for the input.
(.venv) ➜ ✗ git commit -m "test"
sqlfluff-lint........................................(no files to check)Skipped
sqlfluff-fix.........................................(no files to check)Skipped
black................................................(no files to check)Skipped
isort................................................(no files to check)Skipped
docformatter.........................................(no files to check)Skipped
flake8...............................................(no files to check)Skipped
blackdoc.............................................(no files to check)Skipped
Terraform plan...........................................................
Only after I give an input "Y", all output strings defined in this hook (ex: output string through echo, terraform plan) comes out.
Terraform plan...........................................................Y
Passed
- hook id: terraform_plan
- duration: 222.33s
location: [remove due to privacy]
PROJECT_ID: [remove due to privacy]
CREDENTIALS: [remove due to privacy]
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# google_bigquery_table.bar will be created
+ resource "google_bigquery_table" "bar" {
+ creation_time = (known after apply)
+ dataset_id = "default_dataset"
+ deletion_protection = true
...
I also try read -p "Do yu agree this plan? (Y/N): " answer < /dev/tty, but get the same issue.
Here is my part of .pre-commit-config.yaml config file.
repos:
- repo: local
hooks:
# there are other hooks above, remove them for easy to read
- id: terraform_plan
name: Terraform plan
entry: hooks/terraform_plan.sh
language: script
verbose: true
files: (\.tf|\.tfvars)$
exclude: \.terraform\/.*$
- repo: ../pre-commit-terraform
rev: d7e049d0b72eebcb09b719bb296589a47f4fa806
hooks:
- id: terraform_fmt
- id: terraform_tflint
- id: terraform_validate
args: [--args=-no-color]
So far I do not know what the root cause is and how to solve it. Seek for someone's help and suggestion. Thanks.
pre-commit intentionally does not allow interactivity so there is no working solution within the framework. you can sidestep the framework and utilize a legacy shell script (.git/hooks/pre-commit.legacy) but I would also not recommend that:
terraform plan is also kind of a poor choice for a pre-commit hook -- especially if you are blocking the user to confirm it. they are likely to be either surprised by such a hook or fatigued by it and will ignore the output (mash yes) or ignore the entire process (SKIP / --no-verify).
disclaimer: I wrote pre-commit

why does sysctl show failed to access a file when using Ansbile

I have hit this a few times and never figured it out/ it resolved itself.
Running below playbook gives an error but it does make the change requested. . . .
If I run the same play again it does show the message but that is cause it is not updating the sysctl.
---
- hosts: "{{ target }}"
gather_facts: yes
become: yes
become_user: root
tasks:
- name: add a vm.overcommit_memory setting at the end of the sysctl.conf
sysctl: name=vm.overcommit_memory value=0 state=present reload=yes
The error is:
fatal: [testbox.local]: FAILED! => {"changed": false, "msg": "Failed to reload sysctl: net.ipv4.tcp_syncookies = 1\nnet.ipv4.tcp_synack_retries = 2\nnet.ipv4.conf.all.accept_redirects = 0\nnet.ipv4.conf.default.accept_redirects = 0\nnet.ipv6.conf.all.accept_ra = 0\nnet.ipv6.conf.default.accept_ra = 0\nnet.ipv4.icmp_ignore_bogus_error_responses = 1\nnet.ipv6.conf.all.disable_ipv6 = 1\nnet.ipv6.conf.default.disable_ipv6 = 1\nkernel.randomize_va_space = 0\nvm.swappiness = 5\nvm.overcommit_memory = 0\nkernel.shmmni = 15872\nkernel.shmmax = 67546587136\nkernel.shmall = 32981732\nkernel.sem = 250 256000 32 15872\nkernel.msgmni = 64417\nkernel.msgmax = 65536\nkernel.msgmnb = 65536\nsysctl: setting key \"kernel.msgmni\": Invalid argument\nsysctl: cannot stat /proc/sys/randomize_va_space: No such file or directory\nsysctl: cannot stat /proc/sys/“vm/overcommit_memory”: No such file or directory\n"}
There is likely problem with /etc/sysctl.conf prior the change applied by play. If you look at error message there is typo in kernel.msgmni (which should be really kernel.msgmin) and also double quotes around the path to vm.overcommit_memory. From that I suspect there are bad lines in file from previous attempts? Try to comment out these or try again with vanilla file obtained from your distribution.
On reload, good lines are still applied by sysctl; but there are some wrong lines in file which sysctl fails to apply, report and also why it exits with non-zero exit code - which makes play to fail.
According the error message
No such file or directory\nsysctl: cannot stat /proc/sys/“vm/overcommit_memory”: No such file or directory\n"
it seems you are running into a barely documented issue. The file path isn't constructed correctly. For possible reasons you may a look into #blami's answer, since there is also a correct entry in the message with vm.overcommit_memory = 0.
Furthermore may need to use use the YAML notation like
- name: Add a 'vm.overcommit_memory' setting at the end of the 'sysctl.conf'
sysctl:
name: vm.overcommit_memory
value: 0
state: present
reload: yes
which is also used in linux-system-roles/kernel_settings for vm. settings.
Further Q&A
Using Ansible, can we edit kernel level setting?

Backuping files before change

Since Ansible backup feature is questionable a little with lack of configuration. I'm looking into some solution.
Normally in script I would have backup function that you can call with file name and it would copy the file to separate location with changed name.. for example bkp_location = /tmp/backup//
Lets say I want to backup /etc/systemconf/network I pass it to function and it would copy it to backup directory under etc_systemconf_network ( it replace / with _ so we can tell where it come from )
What would be the best solution in Ansible for something like that ? That I could call it in every role etc...
Maybe one backup.yml in root directory and have it include and pass variable ( file name ) to it, would that work ?
Edit:
Backup feature I speak of:
there is an option backup: yes for some modules ( this is shared function between them as far as I know ) but does not offer any modification to what it does.
Like what would be the backup file name, where it would be located... ? so I have to handle that externally... kind of mid-step between.. but seems like include backup.yml and pass variable to it will do the trick.
cat backup.yml
- name: creating backup
copy: src="{{ path_of_file }}" dest="{{ bkp_location }}/backup{{ path_of_file }}{{ contenttoaddwhilebackingup }}" remote_src=true
in running playbook
include: backup.yml
So if you run a playbook like this
ansible-playbook random.yml -e 'bkp_location=/tmp/backup/ path_of_file=/etc/systemconf/network contenttoaddwhilebackingup=26march2021'
It will create backup like this
ls -lrt /etc/systemconf/
-rw-r--r-- 1 root root 2 Mar 25 15:22 network
ls -lrt /tmp/backup/
-rw-r--r-- 1 root root 2 Mar 25 15:22 backupnetwork26march2021

SaltStack: edit yaml file on minion host based on salt pillar data

Say the minion host has a default yaml configuration named myconf.yaml. What I want to do is to edit parts of those yaml entries using values from a pillar. I can't even begin to think how to do this on Salt. The only think I can think of is to run a custom python script on the host via cmd.run and feed it with input via arguments, but this seems overcomplicated.
I want to avoid file.managed. I cannot use a template, since the .yaml file is big, and can change by external means. I just want to edit a few parameters in it. I suppose a python script could do it but I thought salt could do it without writing s/w
I have found salt.states.file.serialize with the merge_if_exists option, I will try this and report.
You want file.serialize with the merge_if_exists option.
# states/my_app.sls
something_conf_file:
file.serialize:
- name: /etc/my_app.yaml
- dataset_pillar: my_app:mergeconf
- formatter: yaml
- merge_if_exists: true
# pillar/my_app.sls
my_app:
mergeconf:
options:
opt3: 100
opt4: 200
On the target, /etc/my_app.yaml might start out looking like this (before the state is applied):
# /etc/my_app.yaml
creds:
user: a
pass: b
options:
opt1: 1
opt2: 2
opt3: 3
opt4: 4
And would look like this after the state is applied:
creds:
user: a
pass: b
options:
opt1: 1
opt2: 2
opt3: 100
opt4: 200
As far as I can tell this uses the same algorithm as pillar merges, so e.g. you can merge or partially overwrite dictionaries, but not lists; lists can only be replaced whole.
This can be done for both json and yaml with file.serialize. Input can be inline on the state or come from a pillar. A short excerpt follows:
state:
cassandra_yaml:
file:
- serialize
# - dataset:
# concurrent_reads: 8
- dataset_pillar: cassandra_yaml
- name: /etc/cassandra/conf/cassandra.yaml
- formatter: yaml
- merge_if_exists: True
- require:
- pkg: cassandra-pkgs
pillar:
cassandra_yaml:
concurrent_reads: "8"

Proper syntax of write_files directive in cloud config?

I'm trying to get a cloud config script working properly with my DigitalOcean droplet, but I'm testing on local lxc containers in the interim.
One consistent problem I have is that I can never get the write_files directive working properly for more than one file. It seems to behave in weird ways that I cannot understand.
For example, this configuration is incorrect, and only outputs a single file (.tarsnaprc) in /tmp:
#cloud-config
users:
- name: julian
shell: /bin/bash
ssh_authorized_keys:
- ssh-rsa myrsakeygoeshere julian#hostname
write_files:
- path: /tmp/.tarsnaprc
permissions: "0644"
content: |
cachedir /home/julian/tarsnap-cache
keyfile /home/julian/tarsnap.key
nodump
print-stats
checkpoint-bytes 1G
owner: julian:julian
- path: /tmp/lxc
content: |
lxc.id_map = u 0 100000 65536
lxc.id_map = g 0 100000 65536
lxc.network.type = veth
lxc.network.link = lxcbr0
permissions: "0644"
However, if I swap the two items in the write_files array, it magically works, and creates both files, .tarsnaprc and lxc. What am I doing wrong, do I have a syntax error?
It may be too late, as it was posted 1 year ago. The problem is setting the owner in /tmp/.tarsnaprc as the user does not exist when the file is created.
Check cloud-init: What is the execution order of cloud-config directives? answer that clearly explains the order of cloud-config directives.
Do not write files under /tmp during boot because of a race with systemd-tmpfiles-clean that can cause temp files to get cleaned during the early boot process. Use /run/somedir instead to avoid race LP:1707222.
ref: https://cloudinit.readthedocs.io/en/latest/topics/modules.html#write-files
Came here because of using canonicals multipass. Nowadays the answers of #rvelaz and #Christian still hint to the right direction. The corrected example whould look like this:
#cloud-config
users:
- name: julian
shell: /bin/bash
ssh_authorized_keys:
- ssh-rsa myrsakeygoeshere julian#hostname
write_files:
# not writing to /tmp
- path: /data/.tarsnaprc
permissions: "0644"
content: |
cachedir /home/julian/tarsnap-cache
keyfile /home/julian/tarsnap.key
nodump
print-stats
checkpoint-bytes 1G
# at execution time, this owner does not yet exist (see runcmd)
# owner: julian:julian
- path: /data/lxc
content: |
lxc.id_map = u 0 100000 65536
lxc.id_map = g 0 100000 65536
lxc.network.type = veth
lxc.network.link = lxcbr0
permissions: "0644"
runcmd:
- "chown julian:julian /data/lxc /data/.tarsnaprc"

Resources