Ansible - run two tasks in parallel - ansible

My playbook is as following... this is related to one host not several at once:
---------------- Deregister from TG's
- name: DeRegister instance ID from a Public TG
elb_target:
target_group_name: "{{ TG_public }}"
target_id: "{{ Instance_ID }}"
state: absent
target_status: unused ## wait until this status
target_status_timeout: 300 ## in seconds
profile: "{{ profile_ID }}" ## specify AWS profile
delegate_to: 127.0.0.1 ## Execute this stage locally
- name: DeRegister instance ID from a Private TG
elb_target:
target_group_name: "{{ TG_private }}"
target_id: "{{ Instance_ID }}"
state: absent
target_status: unused ## wait until this status
target_status_timeout: 300 ## in seconds
profile: "{{ profile_ID }}" ## specify AWS profile
delegate_to: 127.0.0.1 ## Execute this stage locally
----------------------------------
The question how to execute above tasks in parallel... Please help

Use async and set
poll: 0
For example, in the playbook below, 2 commands sleep (10 seconds each) are used instead of the 2 commands from the question. The output shows that both tasks started the commands and terminated. The commands ran in parallel. The first task async_status waited for 3 iterations (3 seconds each) until the first command completed. The second task async_status completed immediately. The whole playbook ran for 11 seconds.
- hosts: localhost
tasks:
- debug:
msg: "{{ lookup('pipe', 'date') }}"
- command: /usr/bin/sleep 10 # Public TG
async: 45
poll: 0
register: public
- command: /usr/bin/sleep 10 # Private TG
async: 45
poll: 0
register: private
- debug:
msg: "{{ lookup('pipe', 'date') }}"
- async_status:
jid: "{{ public.ansible_job_id }}"
register: job_result
until: job_result.finished
retries: 100
delay: 3
- debug:
msg: "{{ lookup('pipe', 'date') }}"
- async_status:
jid: "{{ private.ansible_job_id }}"
register: job_result
until: job_result.finished
retries: 100
delay: 3
- debug:
msg: "{{ lookup('pipe', 'date') }}"
gives (abridged)
TASK [debug] ****
msg: Mon 12 Oct 2020 09:42:21 PM CEST
TASK [command] ****
TASK [command] ****
TASK [debug] ****
msg: Mon 12 Oct 2020 09:42:21 PM CEST
TASK [async_status] ****
FAILED - RETRYING: async_status (100 retries left).
FAILED - RETRYING: async_status (99 retries left).
FAILED - RETRYING: async_status (98 retries left).
TASK [debug] ****
msg: Mon 12 Oct 2020 09:42:32 PM CEST
TASK [async_status] ****
TASK [debug] ****
msg: Mon 12 Oct 2020 09:42:32 PM CEST
It is possible to wait for more jobs in a single loop. For example
- async_status:
jid: "{{ item.ansible_job_id }}"
loop:
- "{{ public }}"
- "{{ private }}"
register: job_result
until: job_result.finished
retries: 100
delay: 3

Related

Copy file and place it to different directories based on the month created of the file

I want to copy the patch files of the current month into the assigned quarter directories as the destination. If the month is:
MONTHS
QUARTER DIRECTORIES
January to March
{{patch_oracle_qdir}}/Q1
April to June
{{patch_oracle_qdir}}/Q2
July to Sept
{{patch_oracle_qdir}}/Q3
October to December
{{patch_oracle_qdir}}/Q4
So far I have this script using ternary to identify the quarter directory
- name: DETERMINE QUARTER DIRECTORY
set_fact:
patch_oracle_qdir: "{{ansible_date_time.month == 08 or ansible_date_time.month == 09 or ansible_date_time.month == 10 | ternary(patch_oracle_base/Q3, patch_oracle_base/Q4) }}"
#with_together:
# - "{{Q1}}"
# - "{{Q2}}"
# - "{{Q3}}"
# - "{{Q4}}"
- name: COPY THE PATCH FILE INTO TARGET SERVER ORACLE
synchronize:
src: {{files_base_dir}}"
dest: "{{patch_oracle_qdir}}"
owner: yes
group: yes
mode: push
register: extract_patch_oracle
Not tested this yet, but is it okay to use ternary or just use with_together loop?
Declare the variable below
patch_oracle_qdir_index: "{{ '%m'|strftime()|int // 3 + 1 }}"
gives the index
patch_oracle_qdir_index: '4'
Notes
Example of a complete playbook for testing
shell> cat pb.yml
- hosts: localhost
vars:
patch_oracle_qdir: /tmp
patch_oracle_qdir_index: "{{ '%m'|strftime()|int // 3 + 1 }}"
tasks:
- debug:
var: patch_oracle_qdir_index
- debug:
msg: |
dest: {{ patch_oracle_qdir }}/Q{{ patch_oracle_qdir_index }}
gives (abridged)
shell> date +%B
November
shell> ansible-playbook pb.yml
PLAY [localhost] *****************************************************************************
TASK [debug] *********************************************************************************
ok: [localhost] =>
patch_oracle_qdir_index: '4'
TASK [debug] *********************************************************************************
ok: [localhost] =>
msg: |-
dest: /tmp/Q4
Q: "What does this line mean?"
{{ '%m'|strftime()|int // 3 + 1 }}
A: See the self-explaining tasks below
- debug:
msg: "{{ '%Y-%m-%d'|strftime() }}"
- debug:
msg: "{{ '%m'|strftime() }}"
- debug:
msg: "{{ '%m'|strftime()|int }}"
- debug:
msg: "{{ '%m'|strftime()|int // 3 }}"
- debug:
msg: "{{ '%m'|strftime()|int // 3 + 1 }}"
give
msg: '2022-11-11'
msg: '11'
msg: '11'
msg: '3'
msg: '4'

Parallel execution of localhost tasks in Ansible

I'm using community.vmware.vmware_guest_powerstate collection for Ansible to start VMs.
The problem is the time it takes for 1 VM can be 2-5 sec, which makes its very inefficient when I want to start 50 VMs ...
Is there any way to make it in parallel?
The playbook:
- hosts: localhost
gather_facts: false
collections:
- community.vmware
vars:
certvalidate: "no"
server_url: "vc01.x.com"
username: "{{ lookup('ansible.builtin.env', 'API_USER', default=Undefined) }}"
password: "{{ lookup('ansible.builtin.env', 'API_PASS', default=Undefined) }}"
tasks:
- name: "setting state={{ requested_state }} in vcenter"
community.vmware.vmware_guest_powerstate:
username: "{{ lookup('ansible.builtin.env', 'API_USER', default=Undefined) }}"
password: "{{ lookup('ansible.builtin.env', 'API_PASS', default=Undefined) }}"
hostname: "{{ server_url }}"
datacenter: "DC1"
validate_certs: no
name: "{{ item }}"
state: "powered-on"
loop: "{{ hostlist }}"
This is Ansible's output: (every line can take 2-5 sec ...)
TASK [setting state=powered-on in vcenter] ************************************************************************************************************
Monday 19 September 2022 11:17:59 +0000 (0:00:00.029) 0:00:08.157 ******
changed: [localhost] => (item=x1.com)
changed: [localhost] => (item=x2.com)
changed: [localhost] => (item=x3.com)
changed: [localhost] => (item=x4.com)
changed: [localhost] => (item=x5.com)
changed: [localhost] => (item=x6.com)
changed: [localhost] => (item=x7.com)
try this instead...
- hosts: all
gather_facts: false
collections:
- community.vmware
vars:
certvalidate: "no"
server_url: "vc01.x.com"
username: "{{ lookup('ansible.builtin.env', 'API_USER', default=Undefined) }}"
password: "{{ lookup('ansible.builtin.env', 'API_PASS', default=Undefined) }}"
tasks:
- name: "setting state={{ requested_state }} in vcenter"
community.vmware.vmware_guest_powerstate:
username: "{{ username }}"
password: "{{ password }}"
hostname: "{{ server_url }}"
datacenter: "DC1"
validate_certs: no
name: "{{ inventory_hostname }}"
state: "powered-on"
delegate_to: localhost
Then run it with your hostlist as the inventory and use forks:
ansible-playbook -i x1.com,x2.com,x3.com,... --forks 10 play.yml
... the time it takes for 1 VM can be 2-5 sec, which makes its very inefficient when I want to start 50 VMs ...
Right, this is the usual behavior.
Is there any way to make it in parallel?
As already mentioned within the comments by Vladimir Botka, asynchronous actions and polling is worth a try since
By default Ansible runs tasks synchronously, holding the connection to the remote node open until the action is completed. This means within a playbook, each task blocks the next task by default, meaning subsequent tasks will not run until the current task completes. This behavior can create challenges.
You see it in your case in the task and in a loop.
Probably the Best Practice to address the use case and to eliminate the cause is to enhance the module code.
According the documentation vmware_guest_powerstate module – Manages power states of virtual machines in vCenter and source ansible-collections/community.vmware/blob/main/plugins/modules/vmware_guest_powerstate.py, the parameter name: takes one name for one VM only. If it would be possible to provide a list of VM names "{{ hostlist }}" to the module directly, there would be one connection attempt only and the loop happening one the Remote Node instead of the Controller Node (... even if this is running localhost for both cases).
To do so one would need to start with name=dict(type='list') instead of str and implement all other logic, error handling and responses.
Further Documentation
Since the community vmware_guest_powerstate module is importing and utilizing additional libraries
pyVmomi library
pyVmomi Community Samples
Meanwhile and based on
Further Q&A and Tests
How do I optimize performance of Ansible playbook with regards to SSH connections?
I've setup another short performance test to simulate the behavior you are observing
---
- hosts: localhost
become: false
gather_facts: false
tasks:
- name: Gather subdirectories
shell:
cmd: "ls -d /home/{{ ansible_user }}/*/"
warn: false
register: subdirs
- name: Gather stats (loop) async
shell: "stat {{ item }}"
loop: "{{ subdirs.stdout_lines }}"
loop_control:
label: "{{ item }}"
async: 5
poll: 0
- name: Gather stats (loop) serial
shell: "stat {{ item }}"
loop: "{{ subdirs.stdout_lines }}"
loop_control:
label: "{{ item }}"
- name: Gather stats (list)
shell: "stat {% raw %}{{% endraw %}{{ subdirs.stdout_lines | join(',') }}{% raw %}}{% endraw %}"
register: result
- name: Show result
debug:
var: result.stdout
and found that adding async will add some additional overhead resulting into even longer execution time.
Gather subdirectories ------------------------ 0.57s
Gather stats (loop) async -------------------- 3.99s
Gather stats (loop) serial ------------------- 3.79s
Gather stats (list) -------------------------- 0.45s
Show result ---------------------------------- 0.07s
This is because of the "short" runtime of the executed task in comparison to "long" time establishing a connection. As the documentation pointed out
For example, a task may take longer to complete than the SSH session allows for, causing a timeout. Or you may want a long-running process to execute in the background while you perform other tasks concurrently. Asynchronous mode lets you control how long-running tasks execute.
one may take advantage from async in case of long running processes and tasks.
In respect the given answer from #Sonclay I've performed another test with
---
- hosts: all
become: false
gather_facts: false
tasks:
- name: Gather subdirectories
shell:
cmd: "ls -d /home/{{ ansible_user }}/*/"
warn: false
register: subdirs
delegate_to: localhost
- name: Gather stats (loop) serial
shell: "stat {{ item }}"
loop: "{{ subdirs.stdout_lines }}"
loop_control:
label: "{{ item }}"
delegate_to: localhost
whereby a call with
ansible-playbook -i "test1.example.com,test2.example.com,test3.example.com" --forks 3 test.yml
will result into an execution time of
Gather subdirectories ------------------------ 0.72s
Gather stats (loop) -------------------------- 0.39s
so it seems to be worth a try.

Registering variables from a looped task and conditionals for skipped tasks

I'm trying to figure out a port test task in a playbook. I have two different sets of ports that need to be checked, but only checked if the server hostname has dev/test/stg/eng in it. Then everything else will be considered a production host and the production ports will be checked.
Here is my playbook:
- name: Filebeat Firewall port check | Dev
wait_for:
host: "{{ item.name }}"
port: "{{ item.port }}"
state: started
delay: 0
timeout: 5
loop: "{{ dev_ports }}"
register: devresults
when: ansible_facts['hostname'] | regex_search('dev|tst|test|eng')
- name: Filebeat Firewall port check | Prod
wait_for:
host: "{{ item.name }}"
port: "{{ item.port }}"
state: started
delay: 0
timeout: 5
loop: "{{ prod_ports }}"
when: devresults is skipped
The first task runs as expected when ran against a production server (is skipped) and a development server (ports are checked). However, the second task is skipped when ran against a production server, as well as a development server (this should happen).
Basically, if the first task is skipped the second task should run, and if the first ran the second task should be skipped, this isn't happening. I'm not sure what I'm missing, I'm thinking its the when statement in the second task, or if the loop is causing my issue, any help would be welcomed.
Regarding your question
Basically, if the first task is skipped the second task should run, and if the first ran the second task should be skipped, this isn't happening.
I've setup a short first example
---
- hosts: localhost
become: false
gather_facts: false
tasks:
- name: Create result set
shell:
cmd: echo 'I am not skipped.'
register: result
when: not ansible_check_mode
- name: Show result when not skipped
debug:
var: result
when: not result.skipped | default('false') | bool
- name: Show result when skipped
debug:
var: result
when: result.skipped | default('false') | bool
and found it working when running with ansible-playbook skip.yml and ansible-playbook skip.yml --check.
Depending on the task and used modules, the reason seems to be when a task becomes skipped the result set is
TASK [Show result when skipped] **************************
ok: [localhost] =>
result:
changed: false
skip_reason: Conditional result was False
skipped: true
whereby when not skipped the result set is
TASK [Show result when not skipped] *********************
ok: [localhost] =>
result:
changed: true
cmd: echo 'I am not skipped.'
delta: '0:00:00.123456'
end: '2022-02-16 20:00:00.123456'
failed: false
rc: 0
start: '2022-02-16 20:00:00.000000'
stderr: ''
stderr_lines: []
stdout: I am not skipped.
stdout_lines:
- I am not skipped.
and don't has the skipped flag. Therefore you may introduce filter to Provide default values and to Force the data type.
Regarding your objection
"the example doesn't really applies as it isn't doing any loops"
you may have a look into the result set of the following second example with loop and Extended loop variables.
---
- hosts: localhost
become: false
gather_facts: false
tasks:
- name: Create result set
shell:
cmd: echo 'I am not skipped.'
register: result
when: item is even
loop: "{{ [1,2] }}"
loop_control:
extended: true
label: "{{ item }}"
- name: Show result
debug:
var: result
which produce an output of
TASK [Create result set] *******
changed: [localhost] => (item=2)
TASK [Show result] ******************************
ok: [localhost] =>
result:
changed: true
msg: All items completed
results:
- ansible_loop:
allitems:
- 1
- 2
first: true
index: 1
index0: 0
last: false
length: 2
nextitem: 2
revindex: 2
revindex0: 1
ansible_loop_var: item
changed: false
item: 1
skip_reason: Conditional result was False
skipped: true
- ansible_loop:
allitems:
- 1
- 2
first: false
index: 2
index0: 1
last: true
length: 2
previtem: 1
revindex: 1
revindex0: 0
ansible_loop_var: item
changed: true
cmd: echo 'I am not skipped.'
delta: '0:00:00.123456'
end: '2022-02-16 22:15:00.123456'
failed: false
invocation:
module_args:
_raw_params: echo 'I am not skipped.'
_uses_shell: true
argv: null
chdir: null
creates: null
executable: null
removes: null
stdin: null
stdin_add_newline: true
strip_empty_ends: true
warn: true
item: 2
rc: 0
start: '2022-02-16 22:15:00.000000'
stderr: ''
stderr_lines: []
stdout: I am not skipped.
stdout_lines:
- I am not skipped.
As you can see it is a data structure containing a list of items and it does contain the information about the first skipped item. To access it you may enhance the second example with
- name: Show result when skipped
debug:
msg: "{{ item.item }}"
when: item.skipped | default('false') | bool
loop: "{{ result.results }}"
loop_control:
extended: true
label: "{{ item }}"
and have a look into the output. It also works according the examples of Conditions based on registered variables.
- name: Show result when skipped
debug:
msg: "{{ item.item }}"
when: item is skipped
loop: "{{ result.results }}"
Further Reading
Troubleshoot Ansible Playbooks
Ansible - Check variable type
Discovering the data type

How to parallize the execution by hostgroup in ansible

I am dynamically creating the hostgroup by site and it did well with below recipe.
- name: "generate batches from batch_processor"
batch_processor:
inventory: "{{ inventory_url }}"
ignore_applist: "{{ ignore_applist | default('None') }}"
ignore_podlist: "{{ ignore_podlist | default('None') }}"
register: batch_output
- name: "Creating batches by site"
include: batch_formatter.yml hostkey="{{ 'batch_' + item.key }}" hostlist="{{ item.value | list | join(',') }}"
with_dict: "{{ batch_output.result['assignments'] }}"
In my playbook, i have like this
- hosts: batch_*
serial: 10
gather_facts: False
tasks:
- include: roles/deployment/tasks/dotask.yml
I initially had strategy: free but in some reason it didn't pickup parallel. Currently i am using all batch hosts 10 at time to deploy.
I am thinking of below items
batch_site1 - 10 in parallel
batch_site2 - 10 in parallel
batch_site3 - 10 in parallel
But in the playbook, i don't want to specify the hostgroup execution by site as they are dynamic. Sometime, we will have siteX and sometime it wont be there. Please suggest the best approach.
- hosts: batch_*
gather_facts: False
strategy: free
tasks:
- include: roles/deployment/tasks/dotask.yml
when: "'batch_site1' not in group_names"
async: 600
poll: 0
register: job1
- name: Wait for asynchronous job to end
async_status:
jid: '{{ job1.ansible_job_id }}'
register: job_result1
until: job_result1.finished
retries: 30
- include: roles/deployment/tasks/dotask.yml
when: "'batch_site2' not in group_names"
register: job2
async: 600
poll: 0
- name: Wait for asynchronous job to end
async_status:
jid: '{{ job2.ansible_job_id }}'
register: job_result2
until: job_result2.finished
retries: 30
- include: roles/deployment/tasks/dotask.yml
when: "'batch_site3' not in group_names"
register: job3
async: 600
poll: 0
- name: Wait for asynchronous job to end
async_status:
jid: '{{ job3.ansible_job_id }}'
register: job_result3
until: job_result3.finished
retries: 30
This config slightly working on my purpose, however i couldn't pull the async_status to show better results in screen.
{"msg": "The task includes an option with an undefined variable. The error was: 'job1' is undefined\n\nThe error appears to have been in '/../../../../play.yml': line 12, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n - name: Wait for asynchronous job to end\n ^ here\n"}
not sure why ?

How to do waiting elements in playbook

i have playbook which creating vm's. Me need to waiting when will appear network. How to do it if element don't exist?
Part my playbook:
- ovirt_nics_facts:
vm: "{{ vm }}"
auth: "{{ ovirt_auth }}"
register: nics_facts
until: (ovirt_nics[0].reported_devices[0].ips[0].address is search("10.10.20.")) or
(ovirt_nics[0].reported_devices[0].ips[0].address is search("192.168.250."))
retries: 20
delay: 60
Error:
fatal: [localhost]: FAILED! => {"msg": "The conditional check 'ovirt_nics[0].reported_devices[0].ips[0].address is defined' failed. The error was: error while evaluating conditional (ovirt_nics[0].reported_devices[0].ips[0].address is defined): list object has no element 0"}
I decided trouble:
- name: add vm to temporary inventory
block:
- ovirt_nics_facts:
vm: "{{ vm }}"
auth: "{{ ovirt_auth }}"
register: nics_facts
until: ovirt_nics[0].reported_devices|length > 0
retries: 20
delay: 60

Resources