I am trying to use bitwise operation inside my ansible playbooks. As sugegsted in this post (How to perform bitwise operations in Ansible?), I have created the .py file for "bitwise or" filter and placed under default filterplugin dir
/usr/share/ansible/plugins/filter
1)The ansible ansible.cfg file is edited to specify the filter plugin path.
filter_plugins = /usr/share/ansible/plugins/filter
2)Ansible version on controller node is: 3.6.10
But unfortunately playbook is not able to find the new defined filter. I am not sure if I am missing anything. Is there anything else I need to do?
Any help with this is appreciate.
The filter plugin I tried is working now. There was a small coding issue inside my filter python file, which was reported by playbook run logs on the console, that I didn't spot.
After carefully looking into the logs, I could fix the coding issue and, after that, the filter operation was found from the filter_plugin path that was specified in the ansible.cfg file.
filter_plugins = /usr/share/ansible/plugins/filter
There was no more setting needed apart from specifying the filter path in ansible.cfg.
I'm getting an error with Ansible, "ERROR! No action detected in task." According to Why does Ansible show "ERROR! no action detected in task" error? , it looks like my error should be that I tried to write a role, and put a playbook in my role/tasks/main.yml. And the solution is to specify only a list of tasks in the tasks/main.yml file. Like this:
---
- name: My first task
my_module:
parameter1: value1
So my file looks like this:
---
- name: using a module
ansible.builtin.command: echo hello there
So I am using a module, just like in the example shown elsewhere on StackOverflow, but still I get the error message. What am I missing?
This is for Ansible 2.9 on RHEL 7.7. Thanks.
You are using ansible v2.9.
The (optional) full <namespace>.<collection>.<module_name> you are using in your task is only valid starting from ansible v2.10.
Use the short name: ansible.builtin.command => command.
You can also switch to the ansible v2.9 version on the documentation on the site to prevent this kind of problem when copy/pasting. See the dropdown on the top left
From the information you provided here it doesn't appear that you're using an actual module, but instead you've placed a task list file in a module search path. Modules are implemented as programming language code that gets bundled and piped over the connection plugin to be executed in order to carry out task work on the managed host in the inventory.
SUMMARY
As per documentation for Ansible Configuration Settings, we can place ansible.cfg in current directory of the project we are working on and Ansible will search for a config file in the order specified in link above.
However, it appears that ansible is unable to correctly parse ansible.cfg file within my project directory. I am not sure but I think it has to be with the Ini ConfigParser
ANSIBLE VERSION
ansible 2.6.3
config file = /Users/pnotes/Code/Terraform/Blog/ansible/ansible.cfg
configured module search path = ['/Users/pnotes/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /Users/pnotes/.pyenv/versions/3.6.4/lib/python3.6/site-packages/ansible
executable location = /Users/pnotes/.pyenv/versions/3.6.4/bin/ansible
python version = 3.6.4 (default, Feb 26 2018, 21:07:35) [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)]
CONFIGURATION
output of "ansible-config dump --only-changed"
Error reading config file (/Users/pnotes/Code/Terraform/Blog/terraform/ansible.cfg): File contains no section headers.
file: '/Users/pnotes/Code/Terraform/Blog/terraform/ansible.cfg', line: 3
'vault_password_file = ~/.vault_pass.txt\n'
OS / ENVIRONMENT
macOS High Sierra
contents of terraform/ansible.cfg:
# If set, configures the path to the Vault password file as an
# alternative to specifying --vault-password-file on the command
# line.
vault_password_file = ~/.vault_pass.txt
n
Note: when project_directory/terraform/ansible.cfg is commented out, ansible uses ~/.ansible.cfg and is able to decrypt encrypted files.
EXPECTED RESULTS
I expect playbook to decrypt all encrypted vars using the ansible vault password saved in the path provide in the configuration file (ansible.cfg) provided in the project directory.
ACTUAL RESULTS
I get the error below:
Error: Error applying plan:
1 error(s) occurred:
* linode_linode.base: Error running command 'sleep 30; ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -u root --ask-pass -i '172.104.29.185,' ../ansible/provision.yml --extra-vars 'ip=000.111.22.185' -vvvv': exit status 5. Output: Error reading config file (/Users/pnotes/Code/Terraform/Test/terraform/ansible.cfg): File contains no section headers.
file: '/Users/pnotes/Code/Terraform/Test/terraform/ansible.cfg', line: 3
'vault_password_file = ~/.vault_pass.txt\n'
Can someone please explain why I keep getting the error **exit status 5. Output: Error reading config file (/Users/pnotes/Code/Terraform/Test/terraform/ansible.cfg): File contains no section headers.
I also tried looking for info on how ansible generates the default /etc/ansible.cfg file but can't seem to track it down (I use python in a limited capacity so this might be why I am having a hard time with this)
Would really appreciate it.
ansible.cfg is supposed to be divided into sections with headers of the form [section_name]. The vault_password_file option, like the majority of the configuration options, belongs to the [defaults] section, so your ansible.cfg needs to look like:
[defaults]
vault_password_file = ~/.vault_pass.txt
Sometimes, ansible doesn't do what you want. And increasing verbosity doesn't help. For example, I'm now trying to start coturn server, which comes with init script on systemd OS (Debian Jessie). Ansible considers it running, but it's not. How do I look into what's happening under the hood? Which commands are executed, and what output/exit code?
Debugging modules
The most basic way is to run ansible/ansible-playbook with an increased verbosity level by adding -vvv to the execution line.
The most thorough way for the modules written in Python (Linux/Unix) is to run ansible/ansible-playbook with an environment variable ANSIBLE_KEEP_REMOTE_FILES set to 1 (on the control machine).
It causes Ansible to leave the exact copy of the Python scripts it executed (either successfully or not) on the target machine.
The path to the scripts is printed in the Ansible log and for regular tasks they are stored under the SSH user's home directory: ~/.ansible/tmp/.
The exact logic is embedded in the scripts and depends on each module. Some are using Python with standard or external libraries, some are calling external commands.
Debugging playbooks
Similarly to debugging modules increasing verbosity level with -vvv parameter causes more data to be printed to the Ansible log
Since Ansible 2.1 a Playbook Debugger allows to debug interactively failed tasks: check, modify the data; re-run the task.
Debugging connections
Adding -vvvv parameter to the ansible/ansible-playbook call causes the log to include the debugging information for the connections.
Debugging Ansible tasks can be almost impossible if the tasks are not your own. Contrary to what Ansible website states.
No special coding skills needed
Ansible requires highly specialized programming skills because it is not YAML or Python, it is a messy mix of both.
The idea of using markup languages for programming has been tried before. XML was very popular in Java community at one time. XSLT is also a fine example.
As Ansible projects grow, the complexity grows exponentially as result. Take for example the OpenShift Ansible project which has the following task:
- name: Create the master server certificate
command: >
{{ hostvars[openshift_ca_host]['first_master_client_binary'] }} adm ca create-server-cert
{% for named_ca_certificate in openshift.master.named_certificates | default([]) | lib_utils_oo_collect('cafile') %}
--certificate-authority {{ named_ca_certificate }}
{% endfor %}
{% for legacy_ca_certificate in g_master_legacy_ca_result.files | default([]) | lib_utils_oo_collect('path') %}
--certificate-authority {{ legacy_ca_certificate }}
{% endfor %}
--hostnames={{ hostvars[item].openshift.common.all_hostnames | join(',') }}
--cert={{ openshift_generated_configs_dir }}/master-{{ hostvars[item].openshift.common.hostname }}/master.server.crt
--key={{ openshift_generated_configs_dir }}/master-{{ hostvars[item].openshift.common.hostname }}/master.server.key
--expire-days={{ openshift_master_cert_expire_days }}
--signer-cert={{ openshift_ca_cert }}
--signer-key={{ openshift_ca_key }}
--signer-serial={{ openshift_ca_serial }}
--overwrite=false
when: item != openshift_ca_host
with_items: "{{ hostvars
| lib_utils_oo_select_keys(groups['oo_masters_to_config'])
| lib_utils_oo_collect(attribute='inventory_hostname', filters={'master_certs_missing':True}) }}"
delegate_to: "{{ openshift_ca_host }}"
run_once: true
I think we can all agree that this is programming in YAML. Not a very good idea. This specific snippet could fail with a message like
fatal: [master0]: FAILED! => {"msg": "The conditional check 'item !=
openshift_ca_host' failed. The error was: error while evaluating
conditional (item != openshift_ca_host): 'item' is undefined\n\nThe
error appears to have been in
'/home/user/openshift-ansible/roles/openshift_master_certificates/tasks/main.yml':
line 39, column 3, but may\nbe elsewhere in the file depending on the
exact syntax problem.\n\nThe offending line appears to be:\n\n\n-
name: Create the master server certificate\n ^ here\n"}
If you hit a message like that you are doomed. But we have the debugger right? Okay, let's take a look what is going on.
master0] TASK: openshift_master_certificates : Create the master server certificate (debug)> p task.args
{u'_raw_params': u"{{ hostvars[openshift_ca_host]['first_master_client_binary'] }} adm ca create-server-cert {% for named_ca_certificate in openshift.master.named_certificates | default([]) | lib_utils_oo_collect('cafile') %} --certificate-authority {{ named_ca_certificate }} {% endfor %} {% for legacy_ca_certificate in g_master_legacy_ca_result.files | default([]) | lib_utils_oo_collect('path') %} --certificate-authority {{ legacy_ca_certificate }} {% endfor %} --hostnames={{ hostvars[item].openshift.common.all_hostnames | join(',') }} --cert={{ openshift_generated_configs_dir }}/master-{{ hostvars[item].openshift.common.hostname }}/master.server.crt --key={{ openshift_generated_configs_dir }}/master-{{ hostvars[item].openshift.common.hostname }}/master.server.key --expire-days={{ openshift_master_cert_expire_days }} --signer-cert={{ openshift_ca_cert }} --signer-key={{ openshift_ca_key }} --signer-serial={{ openshift_ca_serial }} --overwrite=false"}
[master0] TASK: openshift_master_certificates : Create the master server certificate (debug)> exit
How does that help? It doesn't.
The point here is that it is an incredibly bad idea to use YAML as a programming language. It is a mess. And the symptoms of the mess we are creating are everywhere.
Some additional facts. Provision of prerequisites phase on Azure of Openshift Ansible takes on +50 minutes. Deploy phase takes more than +70 minutes. Each time! First run or subsequent runs. And there is no way to limit provision to a single node. This limit problem was part of Ansible in 2012 and it is still part of Ansible today. This fact tells us something.
The point here is that Ansible should be used as was intended. For simple tasks without the YAML programming. Fine for lots of servers but it should not be used for complex configuration management tasks.
Ansible is a not Infrastructure as Code ( IaC ) tool.
If you ask how to debug Ansible issues, you are using it in a way it was not intended to be used. Don't use it as a IaC tool.
Here's what I came up with.
Ansible sends modules to the target system and executes them there. Therefore, if you change module locally, your changes will take effect when running playbook. On my machine modules are at /usr/lib/python2.7/site-packages/ansible/modules (ansible-2.1.2.0). And service module is at core/system/service.py. Anisble modules (instances of AnsibleModule class declared at module_utils/basic.py) has log method, which sends messages to systemd journal if available, or falls back to syslog. So, run journalctl -f on target system, add debug statements (module.log(msg='test')) to module locally, and run your playbook. You'll see debug statements under ansible-basic.py unit name.
Additionally, when you run ansible-playbook with -vvv, you can see some debug output in systemd journal, at least invocation messages, and error messages if any.
One more thing, if you try to debug code that's running locally with pdb (import pdb; pdb.set_trace()), you'll most likely run into BdbQuit exception. That's because python closes stdin when creating a thread (ansible worker). The solution here is to reopen stdin before running pdb.set_trace() as suggested here:
sys.stdin = open('/dev/tty')
import pdb; pdb.set_trace()
Debugging roles/playbooks
Basically debugging ansible automation over big inventory across large networks is none the other than debugging a distributed network application. It can be very tedious and delicate, and there are not enough user friendly tools.
Thus I believe the also answer to your question is a union of all the answers before mine + small addition. So here:
absolutely mandatory: you have to want to know what's going on, i.e. what you're automating, what you are expecting to happen. e.g. ansible failing to detect service with systemd unit as running or as stopped usually means a bug in service unit file or service module, so you need to 1. identify the bug, 2. Report the bug to vendor/community, 3. Provide your workaround with TODO and link to bug. 4. When bug is fixed - delete your workaround
to make your code easier to debug use modules, as much as you can
give all tasks and variables meaningful names.
use static code analysis tools like ansible-lint. This saves you from really stupid small mistakes.
utilize verbosity flags and log path
use debug module wisely
"Know thy facts" - sometimes it is useful to dump target machine facts into file and pull it to ansible master
use strategy: debugin some cases you can fall into a task debugger at error. You then can eval all the params the task is using, and decide what to do next
the last resort would be using Python debugger, attaching it to local ansible run and/or to remote Python executing the modules. This is usually tricky: you need to allow additional port on machine to be open, and if the code opening the port is the one causing the problem?
Also, sometimes it is useful to "look aside" - connect to your target hosts and increase their debuggability (more verbose logging)
Of course log collection makes it easier to track changes happening as a result of ansible operations.
As you can see, like any other distributed applications and frameworks - debug-ability is still not as we'd wish for.
Filters/plugins
This is basically Python development, debug as any Python app
Modules
Depending on technology, and complicated by the fact you need to see both what happens locally and remotely, you better choose language easy enough to debug remotely.
You could use register module, and debug module to print the return values. For example, I want to know what is the return code of my script execution called "somescript.sh", so I will have my tasks inside the play such as:
- name: my task
shell: "bash somescript.sh"
register: output
- debug:
msg: "{{ output.rc }}"
For full return values you can access in Ansible, you can check this page: http://docs.ansible.com/ansible/latest/common_return_values.html
There are multiple levels of debugging that you might need but the easiest one is to add ANSIBLE_STRATEGY=debug environment variable, which will enable the debugger on the first error.
1st approach: Debugging Ansible module via q module and print the debug logs via the q module as q('Debug statement'). Please check q module page to check where in tmp directory the logs would get generated in the majority of the case either it'll be generated either at: $TMPDIR\q or \tmp\q, so one can do tail -f $TMPDIR\q to check the logs generated once the Ansible module play runs (ref: q module).
2nd Approach: If the play is running on localhost one can use pdb module to debug the play following respective doc: https://docs.ansible.com/ansible/latest/dev_guide/debugging.html
3rd Approach: Using Ansible debug module to print the play result and debug the module(ref: Debug module).
I would like to add debug logging to my custom Ansible module such that when the -vvv option is supplied to the ansible-playbook command I see the log messages but otherwise do not. I don't want the results of the module to be affected by this logging. In other words, I don't want to write the log messages to stdout or stderr directly.
Modules normally are executed remotely, so there actually is no way to directly output anything.
You can return additional data in your module in the exit_json call.
if module._verbosity >= 3:
module.exit_json(changed=True, debug="wooha!")
else:
module.exit_json(changed=True)
module._verbosity corresponds to the verbosity level (-v = 1, -vvv = 3) and is available since Ansible 2.1.
Source: Ansible Devel-list and github
Ansible module are hard to debug because:
modules are normally executed remotely
modules are executed by python subprocess (so we can not use pdb to debug)
So, you can debug the Ansible module in this way:
Make the module execute locally. This can be implemented either by setting the hosts to localhost or by using Ansible local_action module.
Write/Logging the variables you want to debug into a specific file. You can do this by using Python q library.