Today I noticed that Ansible won't merge vars.
For example when I have something like
---
lvm_roles:
postgresql:
size: '10g'
path: '/var/lib/postgresql'
And in another place I have for example
---
lvm_roles:
sonarqube:
size: '10g'
path: '/opt/sonarqube'
Ansible won't merge these facts. I am not sure about precedence but I think the first one wins. Without errors or warnings. IMHO a dangerous feature for a configuration management tool.
Are there any vars that Ansible can merge? Lists and hash won't work. Is there a workaround of some sort for this?
This is a significant shortcoming of Ansible. Because "facts" can be dependent on what you are provisioning. The inability to merge "facts" make it necessary to hard code and duplicate the stuff that you wan't to be configurable.
For example when I create one file with
lvm_roles:
postgresql:
size: '10g'
path: '{{ postgresql_home }}'
sonarqube:
size: '10g'
path: '{{ sonar_home }}'
This will not work because sonar_home is not defined on de postgresql node. On the the sonarqube node, postgresql_home is not defined. The ability to flexibly use vars is greatly impacted if merging is not possible.
Extract of a default ansible.cfg file:
# if inventory variables overlap, does the higher precedence one win
# or are hash values merged together? The default is 'replace' but
# this can also be set to 'merge'.
#hash_behaviour = replace
You can therefore change this behavior by setting hash_behaviour = merge.
I would not change that on a system wide basis as it might break other projects/roles that would rely on a default behavior. You can distribute the ansible.cfg at the root of your specific project that really needs this.
Meanwhile, as #dgw pointed out with a specific example, I've always been able to keep the default behavior by carefully choosing where to place my variables (group or host in inventory, included file, playbook...) and eventually merge them myself if needed.
I'm trying to write my first ansible playbook to setup my Arch Linux workstations and servers. So far everything worked fine but now I've run into a problem I can't really wrap my head around.
I'm trying to have multiple roles that require changes to the mkinitcpio.conf file, often in the same line.
At the moment I have a role mkinitcpio which generates the /etc/mkinitcpio.conf file based on a template.
group_vars/all.yml
encryption
enabled: true
roles/mkinitcpio/tasks/main.yml
- name: Generate mkinitcpio.conf
template:
src: mkinitcpio.conf.j2
dest: /etc/mkinitcpio.conf
notify:
- generate mkinitcpio
roles/mkinitcpio/handlers/main.yml
- name: generate mkinitcpio
comand: mkinitcpio -p linux
roles/mkinitcpio/templates/mkinitcpio.conf.j2
HOOKS=(base systemd autodetect keyboard sd-vconsole modconf block {% if encryption.enabled == true %}sdencrypt {% endif %}sd-lvm2 filesystems fsck)
Now I have a second role plymouth that also has to update the mkinitcpio.conf, dependend if it will be installed or not.
It will have to add sd-plymouth between systemd and autodetect and notify generate mkinitcpio after that.
The solutions I came up with so far are:
1.)
Make the plymouth role dependend on the mkinitcpio role.
The plymouth role installs plymouth, than modifies the HOOKS line via the lineinfile module. After that, it notifies generate mkinitcpio if needed.
This solution will get very complicated, if more roles will be added, that may update the mkinitcpio.conf file.
2.) Add another conditional to roles/mkinitcpio/templates/mkinitcpio.conf.j2:
HOOKS=(base systemd {% if plymouth == true %}sd-plymouth {% endif %}autodetect keyboard sd-vconsole modconf block {% if encryption.enabled == true %}sdencrypt {% endif %}sd-lvm2 filesystems fsck)
Make the mkinitcpio role dependend on the plymouth role, so that it will install plymouth at first. After that, the mkinitcpio role will update the complete mkinitcpio.conf file and notify generate mkinitcpio if needed.
This solution does not seem right to me, because it has a "reversed dependency".
I hope I described my problem understandable and you can give me some tips about the best way to solve it. Thank you in advance!
In Ansible 2.4, I'm getting this deprecation warning:
[DEPRECATION WARNING]: [defaults]hostfile option, The key is misleading as it can also be a list of hosts, a directory or a list of
paths . This feature will be removed in version 2.8. Deprecation warnings can be disabled by setting deprecation_warnings=False in
ansible.cfg.
For the life of me, I cannot figure out what this means. Anybody know?
In my ansible.cfg file, I had a line like
hostfile = ./inventory
That needs to be changed to
inventory = ./inventory
Sometimes, ansible doesn't do what you want. And increasing verbosity doesn't help. For example, I'm now trying to start coturn server, which comes with init script on systemd OS (Debian Jessie). Ansible considers it running, but it's not. How do I look into what's happening under the hood? Which commands are executed, and what output/exit code?
Debugging modules
The most basic way is to run ansible/ansible-playbook with an increased verbosity level by adding -vvv to the execution line.
The most thorough way for the modules written in Python (Linux/Unix) is to run ansible/ansible-playbook with an environment variable ANSIBLE_KEEP_REMOTE_FILES set to 1 (on the control machine).
It causes Ansible to leave the exact copy of the Python scripts it executed (either successfully or not) on the target machine.
The path to the scripts is printed in the Ansible log and for regular tasks they are stored under the SSH user's home directory: ~/.ansible/tmp/.
The exact logic is embedded in the scripts and depends on each module. Some are using Python with standard or external libraries, some are calling external commands.
Debugging playbooks
Similarly to debugging modules increasing verbosity level with -vvv parameter causes more data to be printed to the Ansible log
Since Ansible 2.1 a Playbook Debugger allows to debug interactively failed tasks: check, modify the data; re-run the task.
Debugging connections
Adding -vvvv parameter to the ansible/ansible-playbook call causes the log to include the debugging information for the connections.
Debugging Ansible tasks can be almost impossible if the tasks are not your own. Contrary to what Ansible website states.
No special coding skills needed
Ansible requires highly specialized programming skills because it is not YAML or Python, it is a messy mix of both.
The idea of using markup languages for programming has been tried before. XML was very popular in Java community at one time. XSLT is also a fine example.
As Ansible projects grow, the complexity grows exponentially as result. Take for example the OpenShift Ansible project which has the following task:
- name: Create the master server certificate
command: >
{{ hostvars[openshift_ca_host]['first_master_client_binary'] }} adm ca create-server-cert
{% for named_ca_certificate in openshift.master.named_certificates | default([]) | lib_utils_oo_collect('cafile') %}
--certificate-authority {{ named_ca_certificate }}
{% endfor %}
{% for legacy_ca_certificate in g_master_legacy_ca_result.files | default([]) | lib_utils_oo_collect('path') %}
--certificate-authority {{ legacy_ca_certificate }}
{% endfor %}
--hostnames={{ hostvars[item].openshift.common.all_hostnames | join(',') }}
--cert={{ openshift_generated_configs_dir }}/master-{{ hostvars[item].openshift.common.hostname }}/master.server.crt
--key={{ openshift_generated_configs_dir }}/master-{{ hostvars[item].openshift.common.hostname }}/master.server.key
--expire-days={{ openshift_master_cert_expire_days }}
--signer-cert={{ openshift_ca_cert }}
--signer-key={{ openshift_ca_key }}
--signer-serial={{ openshift_ca_serial }}
--overwrite=false
when: item != openshift_ca_host
with_items: "{{ hostvars
| lib_utils_oo_select_keys(groups['oo_masters_to_config'])
| lib_utils_oo_collect(attribute='inventory_hostname', filters={'master_certs_missing':True}) }}"
delegate_to: "{{ openshift_ca_host }}"
run_once: true
I think we can all agree that this is programming in YAML. Not a very good idea. This specific snippet could fail with a message like
fatal: [master0]: FAILED! => {"msg": "The conditional check 'item !=
openshift_ca_host' failed. The error was: error while evaluating
conditional (item != openshift_ca_host): 'item' is undefined\n\nThe
error appears to have been in
'/home/user/openshift-ansible/roles/openshift_master_certificates/tasks/main.yml':
line 39, column 3, but may\nbe elsewhere in the file depending on the
exact syntax problem.\n\nThe offending line appears to be:\n\n\n-
name: Create the master server certificate\n ^ here\n"}
If you hit a message like that you are doomed. But we have the debugger right? Okay, let's take a look what is going on.
master0] TASK: openshift_master_certificates : Create the master server certificate (debug)> p task.args
{u'_raw_params': u"{{ hostvars[openshift_ca_host]['first_master_client_binary'] }} adm ca create-server-cert {% for named_ca_certificate in openshift.master.named_certificates | default([]) | lib_utils_oo_collect('cafile') %} --certificate-authority {{ named_ca_certificate }} {% endfor %} {% for legacy_ca_certificate in g_master_legacy_ca_result.files | default([]) | lib_utils_oo_collect('path') %} --certificate-authority {{ legacy_ca_certificate }} {% endfor %} --hostnames={{ hostvars[item].openshift.common.all_hostnames | join(',') }} --cert={{ openshift_generated_configs_dir }}/master-{{ hostvars[item].openshift.common.hostname }}/master.server.crt --key={{ openshift_generated_configs_dir }}/master-{{ hostvars[item].openshift.common.hostname }}/master.server.key --expire-days={{ openshift_master_cert_expire_days }} --signer-cert={{ openshift_ca_cert }} --signer-key={{ openshift_ca_key }} --signer-serial={{ openshift_ca_serial }} --overwrite=false"}
[master0] TASK: openshift_master_certificates : Create the master server certificate (debug)> exit
How does that help? It doesn't.
The point here is that it is an incredibly bad idea to use YAML as a programming language. It is a mess. And the symptoms of the mess we are creating are everywhere.
Some additional facts. Provision of prerequisites phase on Azure of Openshift Ansible takes on +50 minutes. Deploy phase takes more than +70 minutes. Each time! First run or subsequent runs. And there is no way to limit provision to a single node. This limit problem was part of Ansible in 2012 and it is still part of Ansible today. This fact tells us something.
The point here is that Ansible should be used as was intended. For simple tasks without the YAML programming. Fine for lots of servers but it should not be used for complex configuration management tasks.
Ansible is a not Infrastructure as Code ( IaC ) tool.
If you ask how to debug Ansible issues, you are using it in a way it was not intended to be used. Don't use it as a IaC tool.
Here's what I came up with.
Ansible sends modules to the target system and executes them there. Therefore, if you change module locally, your changes will take effect when running playbook. On my machine modules are at /usr/lib/python2.7/site-packages/ansible/modules (ansible-2.1.2.0). And service module is at core/system/service.py. Anisble modules (instances of AnsibleModule class declared at module_utils/basic.py) has log method, which sends messages to systemd journal if available, or falls back to syslog. So, run journalctl -f on target system, add debug statements (module.log(msg='test')) to module locally, and run your playbook. You'll see debug statements under ansible-basic.py unit name.
Additionally, when you run ansible-playbook with -vvv, you can see some debug output in systemd journal, at least invocation messages, and error messages if any.
One more thing, if you try to debug code that's running locally with pdb (import pdb; pdb.set_trace()), you'll most likely run into BdbQuit exception. That's because python closes stdin when creating a thread (ansible worker). The solution here is to reopen stdin before running pdb.set_trace() as suggested here:
sys.stdin = open('/dev/tty')
import pdb; pdb.set_trace()
Debugging roles/playbooks
Basically debugging ansible automation over big inventory across large networks is none the other than debugging a distributed network application. It can be very tedious and delicate, and there are not enough user friendly tools.
Thus I believe the also answer to your question is a union of all the answers before mine + small addition. So here:
absolutely mandatory: you have to want to know what's going on, i.e. what you're automating, what you are expecting to happen. e.g. ansible failing to detect service with systemd unit as running or as stopped usually means a bug in service unit file or service module, so you need to 1. identify the bug, 2. Report the bug to vendor/community, 3. Provide your workaround with TODO and link to bug. 4. When bug is fixed - delete your workaround
to make your code easier to debug use modules, as much as you can
give all tasks and variables meaningful names.
use static code analysis tools like ansible-lint. This saves you from really stupid small mistakes.
utilize verbosity flags and log path
use debug module wisely
"Know thy facts" - sometimes it is useful to dump target machine facts into file and pull it to ansible master
use strategy: debugin some cases you can fall into a task debugger at error. You then can eval all the params the task is using, and decide what to do next
the last resort would be using Python debugger, attaching it to local ansible run and/or to remote Python executing the modules. This is usually tricky: you need to allow additional port on machine to be open, and if the code opening the port is the one causing the problem?
Also, sometimes it is useful to "look aside" - connect to your target hosts and increase their debuggability (more verbose logging)
Of course log collection makes it easier to track changes happening as a result of ansible operations.
As you can see, like any other distributed applications and frameworks - debug-ability is still not as we'd wish for.
Filters/plugins
This is basically Python development, debug as any Python app
Modules
Depending on technology, and complicated by the fact you need to see both what happens locally and remotely, you better choose language easy enough to debug remotely.
You could use register module, and debug module to print the return values. For example, I want to know what is the return code of my script execution called "somescript.sh", so I will have my tasks inside the play such as:
- name: my task
shell: "bash somescript.sh"
register: output
- debug:
msg: "{{ output.rc }}"
For full return values you can access in Ansible, you can check this page: http://docs.ansible.com/ansible/latest/common_return_values.html
There are multiple levels of debugging that you might need but the easiest one is to add ANSIBLE_STRATEGY=debug environment variable, which will enable the debugger on the first error.
1st approach: Debugging Ansible module via q module and print the debug logs via the q module as q('Debug statement'). Please check q module page to check where in tmp directory the logs would get generated in the majority of the case either it'll be generated either at: $TMPDIR\q or \tmp\q, so one can do tail -f $TMPDIR\q to check the logs generated once the Ansible module play runs (ref: q module).
2nd Approach: If the play is running on localhost one can use pdb module to debug the play following respective doc: https://docs.ansible.com/ansible/latest/dev_guide/debugging.html
3rd Approach: Using Ansible debug module to print the play result and debug the module(ref: Debug module).
I want to use the archive module of ansible but it is unfortunately not working. I have the following version installed: ansible 2.3.0 (devel 2131eaba0c)
my playbook looks like this:
- archive: path="{{path_dir}}" dest="{{dest_dir}}/foo.zip" format=zip
The output looks like this:
"failed": true, "reason": "no action detected in task. This often indicates a misspelled module name, or incorrect module path.
The error appears to have been in '/prj/sndbox1/app/jenkins/jobs/release/workspace/tasks/build_rpclient.yml': line 125, column 3, but may be elsewhere in the file depending on the exact syntax problem.
The offending line appears to be:
- archive: path="{{path_dir}}" dest="{{dest_dir}}/foo.zip" format=zip
^ here
We could be wrong, but this one looks like it might be an issue with missing quotes. Always quote template expression brackets when they start a value. For instance:
with_items:
- {{ foo }}
Should be written as:
with_items:
- "{{ foo }}"
As far as I understood the doc, the extra modules are shipped within ansible, so I assume I don't need to separately install this module.
However, what am I doing wrong? Is there any configuration I need to change in order to tell ansible where to look for the extra modules?
Thanks in advance!
Edit: Included the the full log message
Edit 2:
I tried to put the archive.py directly into my working directory --> [library]/archive.py
Now I get the following error:
"failed": true, "msg": "Could not find imported module support code for archive. Looked for either get_exception or pycompat24"
I have the same use case here, I'm using Ansible version 2.2.0.0. Installed via brew on MacOS sierra.
I've managed to solve the issue by adding the following into my local ansible.cfg file.
On defautls section, you must change the library entry:
[defaults]
library = /usr/local/Cellar/ansible/2.2.0.0_2/libexec/lib/python2.7:library
If you have a different setup, check:
pip show ansible
/usr/share/ansible if you are on Linux machine.