I'm provisioning a new server via Terraform and using Ansible as the provisioner on my local system.
Terraform provisions a system on EC2, and then it runs the Ansible playbook providing the IP of the newly built system as the inventory.
I want to use Ansible to wait for the system to finish booting and prevent further tasks from being attempted up until a connection can be established. Up until this point I have been using a manual pause which is inconvenient and imprecise.
Ansible doesn't seem to do what the documentation says it will (unless I'm wrong, a very possible scenario). Here's my code:
- name: waiting for server to be alive
wait_for:
state: started
port: 22
host: "{{ ansible_ssh_host | default(inventory_hostname) }}"
delay: 10
timeout: 300
connect_timeout: 300
search_regex: OpenSSH
delegate_to: localhost
What happens in this step is that the connection doesn't wait any more than 10 seconds to make the connection, and it fails. If the server has booted and I try the playbook again, it works fine and performs as expected.
I've also tried do_until style loops which never seem to work. All examples given in documentation use shell output, and I don't see any way that it would work for non-shell modules.
I also can't seem to get any debug information if I try to register a result and print it out using the debug module.
Anyone have any suggestions as to what I'm doing wrong?
When you use delegate_to or local_action module, {{ ansible_ssh_host }} resolves to localhost, so your task is always running with the following parameter:
host: localhost
It waits for 10 seconds, checks the SSH connection to local host and proceeds (because most likely it is open).
If you use gather_facts: false (which I believe you do) you can add a set_fact task before, to store the target host name value in a variable:
- set_fact:
host_to_wait_for: "{{ ansible_ssh_host | default(inventory_hostname) }}"
and change the line to:
host: "{{ host_to_wait_for }}"
You can proof-test the variables with the following playbook:
---
- hosts: all
gather_facts: false
tasks:
- set_fact:
host_to_wait_for: "{{ ansible_ssh_host | default(inventory_hostname) }}"
- debug: msg="ansible_ssh_host={{ ansible_ssh_host }}, inventory_hostname={{ inventory_hostname }}, host_to_wait_for={{ host_to_wait_for }}"
delegate_to: localhost
Alternatively you can try to find a way to provide the IP address of the EC2 instance to Ansible as a variable and use it as a value for host: parameter. For example, you run Ansible from CLI, then pass ${aws_instance.example.public_ip} to --extra-vars argument.
As techraf indicates, your inventory lookup is actually grabbing the localhost address because of the delegation, so it's not running against the correct machine.
I think your best solution might be to have terraform pass a variable to the playbook containing the instance's IP address. Example:
terraform passes -e "new_ec2_host=<IP_ADDR>"
Ansible task:
- name: waiting for server to be alive
wait_for:
state: started
port: 22
host: "{{ new_ec2_host }}"
delay: 10
timeout: 300
connect_timeout: 300
search_regex: OpenSSH
delegate_to: localhost
Related
I'm trying to create a playbook which basically consists 2 hosts init; (don't ask why)
---
- hosts: all
tasks:
- name: get the hostname of machine and save it as a variable
shell: hostname
register: host_name
when: ansible_host == "x.x.x.x" *(will be filled by my application)*
- hosts: "{{ host_name.stdout }}"
tasks:
- name: use the variable as hostname
shell: whoami
I don't have any hostname information in my application so I need to trigger my playbook with an IP address, then i should get the hostname of that machine and save it to a variable to use in my other tasks to avoid "when" command for each task.
The problem is that I'm able to use "host_name" variable in all other fields except "hosts", it gives me an Error like this when i try to run;
ERROR! The field 'hosts' has an invalid value, which includes an undefined variable. The error was: 'host_name' is undefined
Screenshot of the error
By default, Ansible itself gathers some information about a host. This happens at the beginning of a playbook's execution right after PLAY in TASK [Gathering Facts].
This automatic gathering of information about a system can be turned off via gather_facts: no, by default this is active.
This collected information is called Ansible Facts. An example of the collected facts is shown in the Ansible Docs, for your host you can print out all Ansible Facts:
either in the playbook as a task:
- name: Print all available facts
debug:
var: ansible_facts
or via CLI as an adhoc command:
ansible <hostname> -m setup
The Ansible Facts contain values like: ansible_hostname, ansible_fqdn, ansible_domain or even ansible_all_ipv4_addresses. This is the simplest way to act with the hostname of the client.
If you want to output the hostname and IP addresses that Ansible has collected, you can do it with the following tasks for example:
- name: Print hostname
debug:
var: ansible_hostname
- name: Print IP addresses
debug:
var: ansible_all_ipv4_addresses
If you start your playbook for all hosts, you can check the IP address and also stop it directly for the "wrong" clients.
---
- hosts: all
tasks:
- name: terminate execution for wrong hosts
assert:
that: '"x.x.x.x" is in ansible_all_ipv4_addresses'
fail_msg: Terminating because IP did not match
success_msg: "Host matched. Hostname: {{ ansible_hostname }}"
# your task for desired host
I'm using Ansible with a dynamic inventory plugin to build some virtual machines in libvirt. After creating the machines, I need to wait for them to acquire an ip address. I can't simply do this:
- wait_for_connection:
Because immediately after the virtual machines are created, they won't have an ip address. What I need to do is this:
- name: wait until node has acquired an address
meta: refresh_inventory
until: ansible_host|ipaddr
retries: 30
delay: 1
- name: wait until node has finished booting
wait_for_connection:
That is, I need to wait until the inventory information for each host includes an address in ansible_host. Unfortunately, the above task doesn't work: it simply executes once and continues.
I could just hardcode a delay:
- pause:
seconds: 30
But I would love to have a more active check here to avoid unnecessary delays (and problems if something takes longer than expected).
After fiddling around with it a bit, this is what I ended up doing:
- hosts: ovn
gather_facts: false
tasks:
- name: wait for nodes to acquire addresses
delegate_to: localhost
command: >-
ansible-inventory --host {{ inventory_hostname }}
register: nodecheck
changed_when: false
until: >-
(nodecheck.stdout|from_json).ansible_host|default('')|ipaddr
retries: 30
delay: 1
- meta: refresh_inventory
This repeatedly calls ansible-inventory --host <host>, which outputs inventory information for <host> in JSON format. We parse that, look for ansible_host, and verity that it's an ip address.
Once we know that the inventory source is able to report an ip address for all the nodes, we then call refresh_inventory.
I am trying to use ansible to modify the hostnames of a dozen newly created Virtual Machines, and I am failing to understand how to loop correctly.
Here is the playbook I've written:
---
- hosts: k8s_kvm_vms
tasks:
- name: "update hostnames"
hostname:
name: "{{ item }}"
with_items:
- centos01
- centos02
...
The result is that it updates each host with each hostname. So if I have 12 machines, each hostname would be "centos12" at the end of the playbook.
I expect this behavior to essentially produce the same result as:
num=0
for ip in ${list_of_ips[*]}; do
ssh $ip hostnamectl set-hostname centos${num}
num=$((num+1))
done
If I were to write it myself in bash
The answer on this page leads me to believe I would have to include all of the IP addresses in my playbook. In my opinion, the advantage of scripting would be that I could have the same hostnames even if their IP changes (I just have to copy the ip addresses into /etc/ansible/hosts) which I could reuse with other playbooks. I have read the ansible page on the hostname module, but their example leads me to believe I would, instead of using a loop, have to define a task for each individual IP address. If this is the case, why use ansible over a bash script?
ansible hostname module
You can create a new variable for each of the servers in the inventory like
[k8s_kvm_vms]
server1 new_hostname=centos1
server2 new_hostname=centos2
Playbook:
---
- hosts: k8s_kvm_vms
tasks:
- name: "update hostnames"
hostname:
name: "{{ new_hostname }}"
I think you need to sudo to change the hostname thus you should add "become: yes"
---
- hosts: k8s_kvm_vms
become: yes
tasks:
- name: "update hostnames"
hostname:
name: "{{ new_hostname }}"
I am using the following Ansible playbook to shut down a list of remote Ubuntu hosts all at once:
- hosts: my_hosts
become: yes
remote_user: my_user
tasks:
- name: Confirm shutdown
pause:
prompt: >-
Do you really want to shutdown machine(s) "{{play_hosts}}"? Press
Enter to continue or Ctrl+C, then A, then Enter to abort ...
- name: Cancel existing shutdown calls
command: /sbin/shutdown -c
ignore_errors: yes
- name: Shutdown machine
command: /sbin/shutdown -h now
Two questions on this:
Is there any module available which can handle the shutdown in a more elegant way than having to run two custom commands?
Is there any way to check that the machines are really down? Or is it an anti-pattern to check this from the same playbook?
I tried something with the net_ping module but I am not sure if this is its real purpose:
- name: Check that machine is down
become: no
net_ping:
dest: "{{ ansible_host }}"
count: 5
state: absent
This, however, fails with
FAILED! => {"changed": false, "msg": "invalid connection specified, expected connection=local, got ssh"}
In more restricted environments, where ping messages are blocked you can listen on ssh port until it goes down. In my case I have set timeout to 60 seconds.
- name: Save target host IP
set_fact:
target_host: "{{ ansible_host }}"
- name: wait for ssh to stop
wait_for: "port=22 host={{ target_host }} delay=10 state=stopped timeout=60"
delegate_to: 127.0.0.1
There is no shutdown module. You can use single fire-and-forget call:
- name: Shutdown server
become: yes
shell: sleep 2 && /sbin/shutdown -c && /sbin/shutdown -h now
async: 1
poll: 0
As for net_ping, it is for network appliances such as switches and routers. If you rely on ICMP messages to test shutdown process, you can use something like this:
- name: Store actual host to be used with local_action
set_fact:
original_host: "{{ ansible_host }}"
- name: Wait for ping loss
local_action: shell ping -q -c 1 -W 1 {{ original_host }}
register: res
retries: 5
until: ('100.0% packet loss' in res.stdout)
failed_when: ('100.0% packet loss' not in res.stdout)
changed_when: no
This will wait for 100% packet loss or fail after 5 retries.
Here you want to use local_action because otherwise commands are executed on remote host (which is supposed to be down).
And you want to use trick to store ansible_host into temp fact, because ansible_host is replaced with 127.0.0.1 when delegated to local host.
I have 3 remote VMs and 1 ansible node.
I am getting the hostname of some VMs by running hostname command on those remote VMs through ansible shell module and registering that output in hostname_output variable.
Then I want to add those VM's IP (collected using gather_facts: True, {{ ansible_default_ipv4.address }} ) with their hostname and append it to a file temp_hostname on localhost, hence I am delegating the task to localhost.
But the issue is, when I see on console, the lineinfile module says that line has been added when the module executed for each node and delegated to localhost, but when I check the file on the localhost, only 1 entry is shown on localhost instead of 3.
---
- name: get hostnames of dynamically created VMs
hosts: all
remote_user: "{{ remote_user }}"
gather_facts: True
tasks:
- name: save hostname in variable, as this command is executed remotely, and we want the value on the ansible node
shell: hostname
register: hostname_output
- name: writing hostname_output in ansible node in file on ansible node
lineinfile:
line: "{{ ansible_default_ipv4.address }} {{ hostname_output.stdout }}"
dest: temp_hostname
state: present
delegate_to: 127.0.0.1
I even tried with copy module as specified in Ansible writing output from multiple task to a single file , but that also gave same result i.e 1 entry only.
---
- name: get hostnames of dynamically created VMs
hosts: all
remote_user: "{{ remote_user }}"
gather_facts: True
tasks:
- name: save hostname in variable, as this command is executed remotely, and we want the value on the ansible node
shell: hostname
register: hostname_output
- name: writing hostname_output in ansible node in file on ansible node
copy:
content: "{{ ansible_default_ipv4.address }} {{ hostname_output.stdout }}"
dest: /volume200gb/sushil/test/code_hostname/temp_hostname
delegate_to: 127.0.0.1
Finally when I used shell module with redirection operator, it worked as I wanted i.e 3 entries in file on localhost.
---
- name: get hostnames of dynamically created VMs
hosts: all
remote_user: "{{ remote_user }}"
gather_facts: True
tasks:
- name: save hostname in variable, as this command is executed remotely, and we want the value on the ansible node
shell: hostname
register: hostname_output
- name: writing hostname_output in ansible node in file on ansible node
shell: echo -e "{{ ansible_default_ipv4.address }} {{ hostname_output.stdout }}" >> temp_hostname
delegate_to: 127.0.0.1
I am calling this ansible-playbook get_hostname.yml using command:
ansible-playbook -i hosts get_hostname.yml --ssh-extra-args="-o StrictHostKeyChecking=no" --extra-vars "remote_user=cloud-user" -vvv
My hosts file is:
10.194.11.86 private_key_file=/root/.ssh/id_rsa
10.194.11.87 private_key_file=/root/.ssh/id_rsa
10.194.11.88 private_key_file=/root/.ssh/id_rsa
I am using ansible 2.1.0.0
I am using default ansible.cfg only, no modications
My question is why lineinfile and copy module didn't work? Did I miss anything or wrote something wrongly
I tried to reproduce your issue and it did not happen for me, I suspect this is a problem with your version of ansible, try with the latest.
That being said, I think you might be able to make it work using serial: 1, it is probably an issue with file locking that I don't see happening in ansible 2.3. I also think that instead of using a shell task to gather the hostname you could use the ansible_hostname variable which is provided as an ansible fact, and you can also avoid gathering ALL facts if all you want is the hostname by adding a task for that specifically. In the end, it would look like this:
---
- name: get hostnames of dynamically created VMs
hosts: all
serial: 1
remote_user: "{{ remote_user }}"
tasks:
- name: Get hostnames
setup:
filter: ansible_hostname
- name: writing hostname_output in ansible node in file on ansible node
lineinfile:
line: "{{ ansible_default_ipv4.address }} {{ ansible_hostname }}"
dest: temp_hostname
state: present
delegate_to: 127.0.0.1
I get inconsistent results using your first code block with lineinfile. Sometimes I get all 3 IPs and hostnames in the destination file and sometimes I only get 2. I'm not sure why this is happening but my guess is that Ansible is trying to save changes to the file at the same time and only one change gets picked up.
The second code block won't work since copy will overwrite the file unless content matches what is already there. The last host that runs will be the only IP/hostname in the destination file.
To work around this, you can loop over your play_hosts (the active hosts in the current play) and reference their variables using hostvars.
- name: writing hostname_output in ansible node in file on ansible node
lineinfile:
line: "{{ hostvars[item]['ansible_default_ipv4'].address }} {{ hostvars[item]['hostname_output'].stdout }}"
dest: temp_hostname
state: present
delegate_to: 127.0.0.1
run_once: True
with_items: "{{ play_hosts }}"
Or you can use a template with the same logic
- name: writing hostname_output in ansible node in file on ansible node
template:
src: IP_hostname.j2
dest: temp_hostname
delegate_to: 127.0.0.1
run_once: True
IP_hostname.j2
{% for host in play_hosts %}
{{ hostvars[host]['ansible_default_ipv4'].address }} {{ hostvars[host]['hostname_output'].stdout }}
{% endfor %}
The problem is here that there is multiple concurrent writes to only one file. That leads to unexpected results:
A solution for that is to use serial: 1 on your play, which forces non-parallel execution among your hosts.
But it can be a performance killer depending on the number of hosts.
I would suggest using another solution: instead of writing to only one file, each host delegation could write on its own file (here using the inventory_hostname value). Therefore, it will have no more concurrent writes.
After that, you can use the module assemble to merge all the file in one. Here is an example (untested):
---
- name: get hostnames of dynamically created VMs
hosts: all
remote_user: "{{ remote_user }}"
gather_facts: True
tasks:
- name: save hostname in variable, as this command is executed remotely, and we want the value on the ansible node
shell: hostname
register: hostname_output
- name: deleting tmp folder
file: path=/tmp/temp_hostname state=absent
delegate_to: 127.0.0.1
run_once: true
- name: create tmp folder
file: path=/tmp/temp_hostname state=directory
delegate_to: 127.0.0.1
run_once: true
- name: writing hostname_output in ansible node in file on ansible node
template: path=tpl.j2 dest=/tmp/temp_hostname/{{ inventory_hostname }}
delegate_to: 127.0.0.1
- name: assemble hostnames
assemble: src=/tmp/temp_hostname/ dest=temp_hostname
delegate_to: '{{ base_rundeck_server }}'
run_once: true
Obviously you have to create the tpl.j2 file.