How can I run local-exec provisionner AFTER cloud-init / user_data? - ansible

Question

I'm experiencing a race condition issue on Terraform when running an Ansible playbook with the local-exec provisioner. At one point, that playbook has to install an APT package.
But first, I'm running a cloud-config file init.yml specified in the user_data argument that installs a package as well.
Consequently, I'm getting the following error :
Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?
How can I prevent this?
# init.yml
runcmd:
- sudo apt-get update
- sudo apt-get -y install python python3
# main.tf
resource "digitalocean_droplet" "hotdog" {
image = "ubuntu-18-04-x64"
name = "my_droplet"
region = "FRA1"
size = "s-1vcpu-1gb"
user_data = file("init.yml")
provisioner "local-exec" {
command = "ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -i '${self.ipv4_address},' ./playbook.yml"
}
}

Answer 1

Disclaimer: my terraform knowledge is quite sparse compared to my ansible one. The below should work but there might be terraform centric options I totally missed
A very easy solution is to use an until loop so as to retry the task until it succeeds.
- name: retry apt task every 5s during 1mn until it succeeds (e.g. lock is released)
apt:
name: my_package
register: apt_install
until: apt_install is success
delay: 5
retries: 12
A better approach would be to make sure there is no lock in place on the different dpkg lock files. I did not make the exercise to implement this in ansible and you might need a specific script or custom module to succeed. If you want to give it a try, there is a question with a solution on serverfault
In your context and since this problem seems quite common actually, I searched a bit and I came across this issue on github. I think that my adaptation as below will meet your requirements and might help for any other possible race condition inside your init phase.
Modify your user_data as:
---
runcmd:
- touch /tmp/user-init-running
- sudo apt-get update
- sudo apt-get -y install python python3
- rm /tmp/user-init-running
And in your playbook:
- name: wait for init phase to end. Error after 1mn
wait_for:
path: /tmp/user-init-running
state: absent
delay: 60
- name: install package
apt:
name: mypkg
state: present