I am using ansible to deploy my django App
using
- name: Upgrade the virtualenv.
pip: requirements={{project_root}}/www/requirements.txt virtualenv={{project_root}}/www/virtualenv
But i only want to run that if requirements.txt changed since last run
We need to determine if any of the requirement files have changed. The steps are as follows:
Touch the temp requirement files. (If they didn't exist, the md5 will be different for the new blank file)
Calculate the md5 hash of the previous requirement files
Caclulate the md5 hash of the current requirement files (the ones just pulled down from GIT)
Iterate through the results of these stat commands in-step, comparing the md5 hash, register the output of the comparison
Only if ANY of the results in #4 changed will we install the pip packages
Copy the current requirement files to the tmp location.
Here's my playbook, {{virtualenv.requirements}} is a list of requirement files, eg: ['/work/project/requirements.txt', '/work/project/requirements-prod.txt']:
- name: Assures temp requirements directory exists
file: path="/tmp{{ virtualenv.path }}" state=directory
sudo: yes
when: install_pip_packages
- name: Assures temp requirements files exists
file: path="/tmp{{ item }}" state=touch
sudo: yes
with_items: virtualenv.requirements_files
when: install_pip_packages
- name: Calculate md5 of temp requirements
stat: path="/tmp{{ item }}"
with_items: virtualenv.requirements_files
register: temp_requirements_stat
when: install_pip_packages
- name: Calculate md5 of current requirements
stat: path="{{ item }}"
with_items: virtualenv.requirements_files
register: current_requirements_stat
when: install_pip_packages
- name: Check requirement files for changes
command: test {{ temp_requirements_stat.results[item.0].stat.md5 }} = {{ current_requirements_stat.results[item.0].stat.md5 }}
changed_when: "requirements_check.rc != 0"
failed_when: requirements_check.stderr
with_indexed_items: virtualenv.requirements_files
register: requirements_check
when: install_pip_packages
- name: Install packages required by the Django app inside virtualenv
pip: virtualenv={{ virtualenv.path }} extra_args='-U' requirements="{{ virtualenv.requirements_files | join(' -r ') }}"
when: install_pip_packages and requirements_check.changed
- name: Copy requirements to /tmp
command: cp "{{ item }}" "/tmp{{ item }}"
sudo: yes
with_items: virtualenv.requirements_files
when: install_pip_packages
Here are two options:
put your requirements.txt under Ansible control and use 'copy' or 'template' module, then invoke 'pip' module with 'notify:' statement
second way is more complex:
retrieve md5 sum of requirements.txt on each Ansible run and compare it with saved md5 somewhere on the server ('stat' module could be used)
retrieve pre-saved md5 sum of requirements.txt
if current md5 is not equal to presaved, then invoke pip task ('when:' statement)
save new md5 somewhere on the server for next Ansible run
I use this pretty short workaround for git repository.
- name: get requirements changes since last pull
shell: "cd {{ project_root }}; git log --name-status --oneline origin/master
{{ git_result.before }}..{{ git_result.after }} | grep requirements.txt"
register: pip_check
failed_when: false
- name: update pip requirements
pip: requirements={{ project_root }}/requirements.txt
virtualenv=~/.virtualenvs/www/
when: pip_check.stdout_lines
It's not universal and not cross-platform recipe, but works well for many situations.
Related
I have 2 remote servers (Prod and Demo) and I would like to copy the latest file from a particular folder in Prod to another folder in Demo. Only one file is to be copied.
I can find the latest file in Prod using:
- name: Get files in folder
find:
paths: "/path_in_prod/arch/"
register: found_files
become: true
become_user: root
delegate_to: "{{ prod_server }}"
when: copy_content_from_prod is defined
- name: Get latest file
set_fact:
latest_file: "{{ found_files.files | sort(attribute='mtime', reverse=true) | first }}"
become: true
become_user: root
delegate_to: "{{ prod_server }}"
when: copy_content_from_prod is defined
I can check I have the correct file (debug).
When I try to copy the file with
- name: Fetch the file from prod
fetch: src= {{ latest_file.path }} dest=buffer/ flat=yes
delegate_to: "{{ prod_server }}"
- name: Copy the file to demo
copy: src=buffer/{{ latest_file.path | basename }} dest=/path_in_demo/in
I get a "File not found" error. But if I look for the file it is there (latest_file.path on Prod).
this is the error message
fatal: [demoServerHost -> ProdServerHost ]: FAILED! => {"changed": false, "msg": "file not found: "}
I do not know if I am interpreting the error message correctly but it seems to be looking in Demo in order to copy onto Prod?
In such case the synchronize_module might be the solution.
- name: Synchronize file from PROD to DEMO
synchronize:
src: "/tmp/test.txt"
dest: "/tmp/test.txt"
mode: push
delegate_to: "{{ prod_server }}"
when: "{{ demo_server }}"
which is "copying" a file from the production node to the demo node.
There are also a lot of answers under How to copy files between two nodes using Ansible.
I have faced a similar issue, where the copy task hangs indefinitely. Here is my example which is not site specific (will identify the site and user using the options).
The easiest solution I have found is to scp directly using the shell module:
- name: scp files onto '{{ target_destination }}' looping for each file on '{{ target_source }}'
shell: 'scp {{ hostvars[target_source].ansible_user }}#{{ hostvars[target_source].ansible_host }}:/opt/{{ hostvars[target_source].ansible_user }}/{{ item }} /opt/{{ destuser.stdout }}'
loop: '{{ diffout.stdout_lines }}'
when: diffout.stdout != ""
Some notes:
"target_source" and "target_destination" are defined using the extra-vars option
diffout is an earlier task comparing the folders on "Prod" and "Demo" and shows any new files to copy
this task is run on the "target_destination" (in my case Prod)
hostvars[target_source] will look at the variables for the "target_source" host in the inventory
this serves as a "pull" from Demo to Prod in my case, if your "Demo" doesn't have permissions, then you could delegate the task to "Prod" and rearrange the scp to look for "Demo" vars to push from "Prod"
I am struggling to find out how to compare two files. Tried several methods including this one which errors out with:
FAILED! => {"msg": "The module diff was not found in configured module paths. Additionally, core modules are missing. If this is a
checkout, run 'git pull --rebase' to correct this problem."}
Is this the best practice to compare two files and ensure the contents are the same or is there a better way?
Thanks in advance.
My playbook:
- name: Find out if cluster management protocol is in use
ios_command:
commands:
- show running-config | include ^line vty|transport input
register: showcmpstatus
- local_action: copy content="{{ showcmpstatus.stdout_lines[0] }}" dest=/poc/files/{{ inventory_hostname }}.result
- local_action: diff /poc/files/{{ inventory_hostname }}.result /poc/files/transport.results
failed_when: "diff.rc > 1"
register: diff
- name: debug output
debug: msg="{{ diff.stdout }}"
Why not using stat to compare the two files?
Just a simple example:
- name: Get cksum of my First file
stat:
path : "/poc/files/{{ inventory_hostname }}.result"
register: myfirstfile
- name: Current SHA1
set_fact:
mf1sha1: "{{ myfirstfile.stat.checksum }}"
- name: Get cksum of my Second File (If needed you can jump this)
stat:
path : "/poc/files/transport.results"
register: mysecondfile
- name: Current SHA1
set_fact:
mf2sha1: "{{ mysecondfile.stat.checksum }}"
- name: Compilation Changed
debug:
msg: "File Compare"
failed_when: mf2sha1 != mf1sha1
your "diff" task is missing the shell keyword, Ansible thinks you want to use the diff module instead.
also i think diff (as name of the variable to register the tasks result) leads ansible to confusion, change to diff_result or something.
code (example):
tasks:
- local_action: shell diff /etc/hosts /etc/fstab
failed_when: "diff_output.rc > 1"
register: diff_output
- debug:
var: diff_output
hope it helps
From Ansible User Guide: https://docs.ansible.com/ansible/latest/user_guide/playbooks_error_handling.html
- name: Fail task when both files are identical
ansible.builtin.raw: diff foo/file1 bar/file2
register: diff_cmd
failed_when: diff_cmd.rc == 0 or diff_cmd.rc >= 2
A slightly shortened version of 'imjoseangel' answer which avoids setting facts:
vars:
file_1: cats.txt
file_2: dogs.txt
tasks:
- name: register the first file
stat:
path: "{{ file_1 }}"
checksum: sha1
get_checksum: yes
register: file_1_checksum
- name: register the second file
stat:
path: "{{ file_2 }}"
checksum: sha1
get_checksum: yes
register: file_2_checksum
- name: Check if the files are the same
debug: msg="The {{ file_1 }} and {{ file_2 }} are identical"
failed_when: file_1_checksum.stat.checksum != file_2_checksum.stat.checksum
ignore_errors: true
Is there with_fileglob that works remotely in ansible?
Mainly I do want to use something similar with the with_fileglob but that will glob the files on the remote/target machine, not on the one that is running ansible.
Use find module to filter the files and then process the resulting list:
- name: Get files on remote machine
find:
paths: /path/on/remote
register: my_find
- debug:
var: item.path
with_items: "{{ my_find.files }}"
All of the with_* looping mechanisms are local lookups unfortunately so there's no really clean way to do this in Ansible. Remote operations by design must be enclosed in tasks as it would need to deal with connections and inventory etc.
What you can do is generate your fileglob by shelling out to the host and then registering the output and looping over the stdout_lines part of the output.
So a trivial example may be something like this:
- name : get files in /path/
shell : ls /path/*
register: path_files
- name: fetch these back to the local Ansible host for backup purposes
fetch:
src : /path/"{{item}}"
dest: /path/to/backups/
with_items: "{{ path_files.stdout_lines }}"
This would connect to the remote host (e.g., host.example.com), get all the file names under /path/ and then copy them back to the Ansible host to the path: /path/host.example.com/.
Using ls /path/* didn't work for me, so here's an example that uses find and some simple regex to delete all nginx managed virtual hosts:
- name: get all managed vhosts
shell: find /etc/nginx/sites-enabled/ -type f -name \*-managed.conf
register: nginx_managed_virtual_hosts
- name: delete all managed nginx virtual hosts
file:
path: "{{ item }}"
state: absent
with_items: "{{ nginx_managed_virtual_hosts.stdout_lines }}"
You could use it to find all files with a specific extension or any other mix. For instance to simply get all files in a directory: find /etc/nginx/sites-enabled/ -type f.
Here's a way to do it so that you can loop through all found. In my example, i had to look for all instances of pip to wipe out awscli in preparation to install awscli v2.0. I've done similar with lineinfile to strip out vars in /etc/skel dotfiles
- name: search for pip
find:
paths: [ /usr/local/bin, /usr/bin ]
file_type: any
pattern: pip*
register: foundpip
- name: Parse out pip paths (say that 3 times fast)
set_fact:
pips: "{{ foundpip | json_query('files[*].path') }}"
- name: List all the found versions of pip
debug:
msg: "{{ pips }}"
#upgrading pip often leaves broken symlinks or older wrappers behind which doesn't affect pip but breaks playbooks so ignore!
- name: remove awscli with found versions of pip
pip:
name: awscli
state: absent
executable: "{{ item }}"
loop: "{{ pips }}"
ignore_errors: yes
I'm trying to write an Ansible role that moves a number of files on the remote system. I found a Stack Overflow post about how to do this, which essentially says "just use the command module with 'mv'". I have a single task defined with a with_items statement like this where each item in dirs is a dictionary with src and dest keys:
- name: Move directories
command: mv {{ item.src }} {{ item.dest }}
with_items: dirs
This is good and it works, but I run into problems if the destination directory already exists. I don't want to overwrite it, so I thought about trying to stat each dest directory first. I wanted to update the dirs variable with the stat info, but as far as I know, there isn't a good way to set or update variables once they're defined. So I used stat to get the info on each directory and then saved the data with register:
- name: Check if directories already exist
stat: path={{ item.dest }}
with_items: dirs
register: dirs_stat
Is there a way to tie the registered stat info to the mv commands? This would be easy if it were a single directory. The looping is what makes this tricky. Is there a way to do this without unrolling this loop into two tasks per directory?
This is not the simplest solution by any means, but if you wanted to use Ansible and not "unroll":
---
- hosts: all
vars:
dirs:
- src: /home/ubuntu/src/test/src1
dest: /home/ubuntu/src/test/dest1
- src: /home/ubuntu/src/test/src2
dest: /home/ubuntu/src/test/dest2
tasks:
- stat:
path: "{{item.dest}}"
with_items: dirs
register: dirs_stat
- debug:
msg: "should not copy {{ item.0.src }}"
with_together:
- dirs
- dirs_stat.results
when: item.1.stat.exists
Simply adapt the debug task to run the appropriate command task instead and when: to when: not ....
You can use stat keyword in your playbook to check if it exists or not if it doesn't then move.
---
- name: Demo Playbook
hosts: all
become: yes
tasks:
- name: check destination
stat:
path: /path/to/dest
register: p
- name: copy file if not exists
command: mv /path/to/src /path/to/src
when: p.stat.exists == False
So I have an Ansible playbook and I'm trying to call a command for each item in a list, but also run that command over a fileglob. There is a "with_nested" in Ansible, and it can take variable names, but if I add a "with_fileglob," it just inserts "with_fileglob" as the filename instead of actually doing a glob.
vars:
repo_versions:
- version: trusty
distribution: Ubuntu
- version: wheezy
distribution: Debian
...
- command: reprepro -b /var/www/html includedeb {{ item[0].version }} {{ item[1] }}
with_nested:
- repo_versions
with_fileglob: /home/repoman/debs/*.deb
when: debs_available.stat.exists == True
I've tried a couple of different combinations and I can't seem to get it to process the command in a double for loop (for each .version, for each .deb file)
This should be what you are trying to accomplish.
I used the shell module to register the output of the fileglob, and then the stdout_lines property of the registerd variable in the loop. I have converted the task from my test to your actual commands and paths, so you might need to double check:
vars:
repo_versions:
- version: trusty
distribution: Ubuntu
- version: wheezy
distribution: Debian
tasks:
- shell: ls -1 /home/repoman/debs/*.deb
register: repo_list
- command: reprepro -b /var/www/html includedeb {{ item[0].version }} {{ item[1] }}
with_nested:
- repo_versions
- repo_list.stdout_lines