I'm interested to learn if Ansible can run an included playbook asynchronously?
Basically what I'm trying to do is run a task "Fire and forget, check on it later." When I check on it later I also want to send a slack notification with the result.
However I've notice the included playbook for slack notification takes a little longer than expected to complete and hence it holds up the rest of the playbook.
What I want is to async the included playbook for slack notification so that the current playbook continues.
For instance I have a playbook.yml file that looks like:
- hosts: localhost
tasks:
- name: Fire and forget task
shell: some_task.sh
chdir=/tmp/
register: fire_and_forget_task
async: 3600
poll: 0
- name: Check on fire and forget task
async_status: jid={{ fire_and_forget_task.ansible_job_id }}
register: task_status
until: task_status.finished
retries: 100
ignore_errors: yes
- name: Send slack success msg
include: slack.yml msg="Fire and forget task SUCCESS"
when: task_status.stdout is defined and
'SUCCESS' in fire_and_forget_task.stdout
async: 3600
poll: 0
- name: Send slack failed msg
include: slack.yml msg="Fire and forget task FAILED"
when: task_status.stdout is defined and
'FAILED' in fire_and_forget_task.stdout
async: 3600
poll: 0
My slack.yml file looks like:
- name: Send notification message via Slack
local_action:
module: slack
token: <REDACTED>
attachments:
- text: "{{ msg }}"
color: "#83F52C"
title: "Ansible Status {{ lookup('pipe','date') }}"
With the above playbook, the "Send slack success msg" task takes an awfully long time to execute for a simple task like that. It seems its not running asynchronously even though I have explicitly stated it should.
What is the best way to achieve the desired outcome?
Thank you.
include can't use async keyword.
If your slack.yml is that simple, just replace your include stuff with a single call:
- name: Send notification message via Slack
local_action:
module: slack
token: <REDACTED>
attachments:
- text: "Task result {{ (task_status.stdout is defined and 'SUCCESS' in task_status.stdout) | ternary('SUCCESS','FAILURE') }}"
color: "#83F52C"
title: "Ansible Status {{ lookup('pipe','date') }}"
async: 3600
poll: 0
P.S. but I don't understand how is single Slack HTTP-call slowing down your playbook if you have a long-running task (with high async and retries numbers) just before it...
Related
I have the following playbook
---
- name: Test Playbook
hosts: localhost
vars:
hostinfo_input: "{{ lookup('file','hostinfo.json') | from_json }}"
tasks:
- name: Check Server
command: python3 check.py {{item}}
register: async_out
async: 100
poll: 0
with_items: "{{hostinfo_input.hosts}}"
I am using callback plugin to print something at the start and end of "Check Server" task. I have the following questions.
How do I access the {{item}} variable inside the callback function ?
Which callback functions can I use to print something at the start of the task and then at the end of the task ?
For start of task, I guess, I can use def v2_playbook_on_task_start(self, task, is_conditional): , right ?
I have a users.yaml file with information regarding 400+ users. I need Ansible to create these users during provisioning. I tried with the async keyword (if that's the right word to use, tell me if I'm wrong) and poll: 15 but it takes ~10minutes.
- name: Add FTP users asynchronously
ansible.builtin.user:
name: "{{ item.name }}"
home: "{{ item.home }}"
shell: /sbin/nologin
groups: ftp-users
create_home: yes
append: no
loop: "{{ ftp_users }}"
async: 60
poll: 15
tags: users
I also tried using poll:0 but many users aren't created.
Your actual use of async is adapted to a single long running task use case where you want to minimize the chance of getting your connection kicked because of a timeout. You are asking ansible to start a job, disconnect from the target and then reconnect every 15 seconds to check if the job is done (or until you reach the 60 seconds timeout). Nothing will be launched in parallel: the next iteration in the loop will only start when the current is done.
What you want to do instead is run those tasks in parallel as fast as possible and then check back later if they are done. In this case, you have to use poll: 0 on your task and later check for completion with the async_status module as described on the ansible async guide. Note that you also need to cleanup the async job cache as ansible will not do it automagically for you in that case.
In your case, this would give:
- name: Add FTP users asynchronously
ansible.builtin.user:
name: "{{ item.name }}"
home: "{{ item.home }}"
shell: /sbin/nologin
groups: ftp-users
create_home: yes
append: no
loop: "{{ ftp_users }}"
async: 60
poll: 0
register: add_user
- name: Wait until all commands are done
async_status:
jid: "{{ item.ansible_job_id }}"
register: async_poll_result
until: async_poll_result.finished
retries: 60
delay: 1
loop: "{{ add_user.results }}"
- name: clean async job cache
async_status:
jid: "{{ item.ansible_job_id }}"
mode: cleanup
loop: "{{ add_user.results }}"
Meanwhile, although this is a direct answer on how to use async for parallel jobs, I'm not entirely sure this will fix your actual performance problem which could come from other issues (like slow dns, slow network, pipelining not enabled if that is possible, master ssh connection not configured...)
I have an ansible task that fails about 20% of the time. It almost always succeeds if retried a couple of times. I'd like to use until to loop until the task succeeds and store the output of each attempt to a separate log file on the local machine. Is there a good way to achieve this?
For example, my task currently looks like this:
- name: Provision
register: prov_ret
until: prov_ret is succeeded
retries: 2
command: provision_cmd
I can see how to store the log output from the last retry when it succeeds, but I'd like to store it from each retry. To store from the last attempt to run the command I use:
- name: Write Log
local_action: copy content={{ prov_ret | to_nice_json }} dest="/tmp/ansible_logs/provision.log"
It's not possible as of 2.9. The until loop doesn't preserve results as loop does. Once a task terminates all variables inside this task will be gone except the register one.
To see what's going on in the loop write a log inside the command at the remote host. For example, the command provision_cmd writes a log to /scratch/provision_cmd.log. Run it in the block and display the log in the rescue section.
- block:
- name: Provision
command: provision_cmd
register: prov_ret
until: prov_ret is succeeded
retries: 2
rescue:
- name: Display registered variable
debug:
var: prov_ret
- name: Read the log
slurp:
src: /scratch/provision_cmd.log
register: provision_cmd_log
- name: Display log
debug:
msg: "{{ msg.split('\n') }}"
vars:
msg: "{{ provision_cmd_log.content|b64decode }}"
It seems I'm not able to register a variable whilst using async in the same task. What I'm trying to do is running a check in parallel to see if multiple services have finished completed before executing the next task. However if I have async mode on, the register variable contains information on the async job status rather than the json I'm expecting. If I take off async then it works fine.
- name: Get infos on container
docker_container_info:
name: "{{ item }}"
with_items:
- service1
- service2
- service3
async: 30
poll: 0
until: result.stdout.find("completed") != -1
retries: 5
delay: 10
register: result
- name: Run next task
command: echo done
.....
.....
I don't really need to check the job status but just the json of each item that is returned from docker_container_info but want async on. How do I get them to work together?
How can I see realtime output from a shell script run by ansible?
I recently refactored a wait script to use multiprocessing and provide realtime status of the various service wait checks for multiple services.
As a stand alone script, it works as expecting providing status for each thread as they wait in parallel for various services to get stable.
In ansible, the output pauses until the python script completes (or terminates) and then provides the output. While, OK, it I'd rather find a way to display output sooner. I've tried setting PYTHONUNBUFFERED prior to running ansible-playbook via jenkins withEnv but that doesn't seem to accomplish the goal either
- name: Wait up to 30m for service stability
shell: "{{ venv_dir }}/bin/python3 -u wait_service_state.py"
args:
chdir: "{{ script_dir }}"
What's the standard ansible pattern for displaying output for a long running script?
My guess is that I could follow one of these routes
Not use ansible
execute in a docker container and report output via ansible provided this doesn't hit the identical class of problem
Output to a file from the script and have either ansible thread or Jenkins pipeline thread watch and tail the file (both seem kludgy as this blurs the separation of concerns coupling my build server to the deploy scripts a little too tightly)
You can use - https://docs.ansible.com/ansible/latest/user_guide/playbooks_async.html
main.yml
- name: Run items asynchronously in batch of two items
vars:
sleep_durations:
- 1
- 2
- 3
- 4
- 5
durations: "{{ item }}"
include_tasks: execute_batch.yml
loop: "{{ sleep_durations | batch(2) | list }}"
execute_batch.yml
- name: Async sleeping for batched_items
command: sleep {{ async_item }}
async: 45
poll: 0
loop: "{{ durations }}"
loop_control:
loop_var: "async_item"
register: async_results
- name: Check sync status
async_status:
jid: "{{ async_result_item.ansible_job_id }}"
loop: "{{ async_results.results }}"
loop_control:
loop_var: "async_result_item"
register: async_poll_results
until: async_poll_results.finished
retries: 30
"What's the standard ansible pattern for displaying output for a long running script?"
Standard ansible pattern for displaying output for a long-running script is polling async and loop until async_status finishes. The customization of the until loop's output is limited. See Feature request: until for blocks #16621.
ansible-runner is another route that might be followed.