Is possible to edit find module command to find X latest folders inside remotebackup dir? Thanks
- find: paths="{{ remotebackupdir }}" age="**find_oldest_folders_except_X_newer**" recurse=no
register: result
- debug: var=result.files
- file: path: "{{ remotebackupdir }}" state: absent
with_items: result.files
Ive done something similar.
Try something like this: (to be executed in the directory where the files are that you want to delete)
- name: Get first 4 files
command: ls -trd -1 {{path}}/* | head -4
register: filenames
- name: actually delete files
file: {{path}}/{{ item }} state=absent
with_items: filenames.stdout_lines
Basically -tr lists things in reverse date order, and -1 ensures everything is on one line, and -d says directories only
Related
How to delete the oldest directory with ansible.
suppose I have the following tree structure
Parent Directory
-Dir2020-05-20
-Dir2020-05-21
-Dir2020-05-22
-Dir2020-05-23
now every time an ansible playbook is run, it should delete the oldest directory, For e.g it should delete Dir2020-05-20 in its first run if we consider its creation date to be 2020-05-20.
age attribute of file module does not seen helpful as i have to run this playbook very randomly and i want to keep limited no. of these directories.
Just assign dirpath to the path of your "Parent Directory" where all these directories are present
---
- hosts: localhost
vars:
dir_path: "/home/harshit/ansible/test/" ##parent directory path, make sure it ends with a slash
tasks:
- name: find oldest directory
shell:
cmd: "ls `ls -tdr | head -n 1 `"
chdir: "{{dir_path}}"
register: dir_name_to_delete
- name: "delete oldest directory: {{dir_path}}{{dir_name_to_delete.stdout}}"
file:
state: absent
path: "{{dir_path}}{{dir_name_to_delete.stdout}}"
Considering a recommended practice is not to use shell or command modules wherever possible I suggest a pure ansible solution for this case:
- name: Get directory list
find:
paths: "{{ target_directory }}"
file_type: directory
register: found_dirs
- name: Get the oldest dir
set_fact:
oldest_dir: "{{ found_dirs.files | sort(attribute='mtime') | first }}"
- name: Delete oldest dir
file:
state: absent
path: "{{ oldest_dir.path }}"
when:
- found_dirs.files | count > 3
There are two ways to know how many files were found with find module - either using its return value matched like this when: found_dirs.matched > 3 or using count filter. I prefer the latter method because I just use this filter in a lot of other cases so this is just a habit.
For your reference, ansible has a whole bunch of useful filters (e.g. I used count and sort here, but there are dozens of them). One does not need to remember those filter names, of course, just keep in mind they exist and might be useful in many cases.
In my Ansible Playbook I'd like to have a task that removes old files and folders from the application's directory. The twist to this otherwise simple task is that a few files or folders need to remain. Imagine something like this:
/opt/application
- /config
- *.properties
- special.yml
- /logs
- /bin
- /var
- /data
- /templates
Let's assume I'd like to keep /logs completely, /var/data and from /config I want to keep special.yml.
(I cannot provide exact code at the moment because I left work frustrated by this and, after cooling down, I am now writing up this question at home)
My idea was to have two lists of exclusions, one holding the folders and one the file. Then I use the find module to first get the folders in the application's directory into a variable and the same for the remaining files into another variable. Afterwards I wanted to remove every folder and file that are not in the lists of exclusions using the file module.
(Pseudo-YML because I'm not yet fluent enough in Ansible that I can whip up a properly structured example; it should be close enough though)
file:
path: "{{ item.path }}"
state: absent
with_items: "{{ found_files_list.files }}"
when: well, that is the big question
What I can't figure out is how to properly construct the when clause. Is it even possible like this?
I don't believe there is a when clause with the file module.
But you can probably achieve what you need as follows:
- name: Find /opt/application all directories, exclude logs, data, and config
find:
paths: /opt/application
excludes: 'logs,data,config'
register: files_to_delete
- name: Ansible remove file glob
file:
path: "{{ item.path }}"
state: absent
with_items: "{{ files_to_delete.files }}"
I hope this is what you need.
First use the find module like you said to get a total list of all files and directories. Register to a variable like all_objects.
- name: Get list of all files recursively
find:
path: /opt/application/
recurse: yes
register: all_objects
Then manually make a list of things you want to keep.
vars:
keep_these:
- /logs
- /var/data
- /config/special.yml
Then this task should delete everything except things in your list:
- name: Delete all files and directories except exclusions
file:
path: "{{ item.path }}"
state: absent
recurse: true
with_items: "{{ all_objects.files }}"
when: item.path not in keep_these
I think this general strategy should work... only thing I'm not sure about is the exact nesting heiararchy of the registered variable from the find module. You might have to play around with the debug module to get it exactly right.
I have an ansible playbook that I need to add a task to. Essentially I have a directory that on each deploy adds another subdirectory. When there are more than 3 subdirectories I want to delete all of the older directories past the 3 most recent directories. I am having a very hard time trying to write a play for it.
I first try to get all the subdirectories
- name: Get all install artifacts
find:
paths: "{{ home_path }}/install/"
file_type: directory
recurse: no
register: install_artifacts
But after this I am trying to check if there are more than 3 items, and if so get the 3rd items modified date, and delete everything with a modified date earlier than this.
- block:
- name: Determine old directories
set_fact:
old_dirs: "{{ (install_artifacts.files|sort(attribute='mtime', reverse=True))[3:] }}"
- name: Remove old directories
file:
path: "{{ item.path }}"
state: absent
with_items: "{{ old_dirs }}"
when: install_artifacts.matched > 3
First of all, install_artifacts.files|sort(attribute='mtime', reverse=True) will sort the list of dicts by mtime in descending order. [3:] means removing the first 3 items from the list, which are the 3 most recent directories. So old_dirs now contains all the old directories.
I'm trying to turn these lines into something I can put in an ansible playbook:
# Install Prezto files
shopt -s extglob
shopt -s nullglob
files=( "${ZDOTDIR:-$HOME}"/.zprezto/runcoms/!(README.md) )
for rcfile in "${files[#]}"; do
[[ -f $rcfile ]] && ln -s "$rcfile" "${ZDOTDIR:-$HOME}/.${rcfile##*/}"
done
So far I've got the following:
- name: Link Prezto files
file: src={{ item }} dest=~ state=link
with_fileglob:
- ~/.zprezto/runcoms/z*
I know it isn't the same, but it would select the same files: except with_fileglob looks on the host machine, and I want it to look on the remote machine.
Is there any way to do this, or should I just use a shell script?
A clean Ansible way of purging unwanted files matching a glob is:
- name: List all tmp files
find:
paths: /tmp/foo
patterns: "*.tmp"
register: tmp_glob
- name: Cleanup tmp files
file:
path: "{{ item.path }}"
state: absent
with_items:
- "{{ tmp_glob.files }}"
Bruce P's solution works, but it requires an addition file and gets a little messy. Below is a pure ansible solution.
The first task grabs a list of filenames and stores it in files_to_copy. The second task appends each filename to the path you provide and creates symlinks.
- name: grab file list
shell: ls /path/to/src
register: files_to_copy
- name: create symbolic links
file:
src: "/path/to/src/{{ item }}"
dest: "path/to/dest/{{ item }}"
state: link
with_items: files_to_copy.stdout_lines
The file module does indeed look on the server where ansible is running for files when using with_fileglob, etc. Since you want to work with files that exist solely on the remote machine then you could do a couple things. One approach would be to copy over a shell script in one task then invoke it in the next task. You could even use the fact that the file was copied as a way to only run the script if it didn't already exist:
- name: Copy link script
copy: src=/path/to/foo.sh
dest=/target/path/to/foo.sh
mode=0755
register: copied_script
- name: Invoke link script
command: /target/path/to/foo.sh
when: copied_script.changed
Another approach would be to create an entire command line that does what you want and invoke it using the shell module:
- name: Generate links
shell: find ~/.zprezto/runcoms/z* -exec ln -s {} ~ \;
You can use with_lines to accomplish this:
- name: Link Prezto files
file: src={{ item }} dest=~ state=link
with_lines: ls ~/.zprezto/runcoms/z*
I want to recursively copy over a directory and render all .j2 files in there as templates. For this I am currently using the following lines:
- template: >
src=/src/conf.d/{{ item }}
dest=/dest/conf.d/{{ item|replace('.j2','') }}
with_lines: find /src/conf.d/ -type f -printf "%P\n"
Now I'm looking for a way to remove unmanaged files from this directory. For example if I remove a file/template from /src/conf.d/ I want Ansible to remove it from /dest/conf.d/ as well.
Is there some way to do this? I tried fiddling around with rsync --delete, but there I got a problem with the templates which get their suffix .j2 removed.
I'd do it like this, assuming a variable defined as 'managed_files' up top that is a list.
- shell: ls -1 /some/dir
register: contents
- file: path=/some/dir/{{ item }} state=absent
with_items: contents.stdout_lines
when: item not in managed_files
We do this with our nginx files, since we want them to be in a special order, come from templates, but remove unmanaged ones this works:
# loop through the nginx sites array and create a conf for each file in order
# file will be name 01_file.conf, 02_file.conf etc
- name: nginx_sites conf
template: >
src=templates/nginx/{{ item.1.template }}
dest={{ nginx_conf_dir }}/{{ '%02d' % item.0 }}_{{ item.1.conf_name|default(item.1.template) }}
owner={{ user }}
group={{ group }}
mode=0660
with_indexed_items: nginx_sites
notify:
- restart nginx
register: nginx_sites_confs
# flatten and map the results into simple list
# unchanged files have attribute dest, changed have attribute path
- set_fact:
nginx_confs: "{{ nginx_sites_confs.results|selectattr('dest', 'string')|map(attribute='dest')|list + nginx_sites_confs.results|selectattr('path', 'string')|map(attribute='path')|select|list }}"
when: nginx_sites
# get contents of conf dir
- shell: ls -1 {{ nginx_conf_dir }}/*.conf
register: contents
when: nginx_sites
# so we can delete the ones we don't manage
- name: empty old confs
file: path="{{ item }}" state=absent
with_items: contents.stdout_lines
when: nginx_sites and item not in nginx_confs
The trick (as you can see) is that template and with_items have different attributes in the register results. Then you turn them into a list of files you manage and then get a list of the the directory and removed the ones not in that list.
Could be done with less code if you already have a list of files. But in this case I'm creating an indexed list so need to create the list as well with map.
I want to share my experience with this case.
Ansible from 2.2 is had with_filetree loop provides simple way to upload dirs, links, static files and even (!) templates. It's best way to keep my config dir synchronized.
- name: etc config - Create directories
file:
path: "{{ nginx_conf_dir }}/{{ item.path }}"
state: directory
mode: 0755
with_filetree: etc/nginx
when: item.state == 'directory'
- name: etc config - Creating configuration files from templates
template:
src: "{{ item.src }}"
dest: "{{ nginx_conf_dir }}/{{ item.path | regex_replace('\\.j2$', '') }}"
mode: 0644
with_filetree: etc/nginx
when:
- item.state == "file"
- item.path | match('.+\.j2$') | bool
- name: etc config - Creating staic configuration files
copy:
src: "{{ item.src }}"
dest: "{{ nginx_conf_dir }}/{{ item.path }}"
mode: 0644
with_filetree: etc/nginx
when:
- item.state == "file"
- not (item.path | match('.+\.j2$') | bool)
- name: etc config - Recreate symlinks
file:
src: "{{ item.src }}"
dest: "{{ nginx_conf_dir }}/{{ item.path }}"
state: link
force: yes
mode: "{{ item.mode }}"
with_filetree: etc/nginx
when: item.state == "link"
Next we may want delete unused files from config dir. It's simple.
We gather list of uploaded files and files exist on remote server, next remove diffrence.
But we may want to have unmanaged files in config dir.
I've used -prune functionality of find to avoid clearing folders with unmanaged files.
PS _(Y)_ sure after I have deleted some unmanaged files
- name: etc config - Gathering managed files
set_fact:
__managed_file_path: "{{ nginx_conf_dir }}/{{ item.path | regex_replace('\\.j2$', '') }}"
with_filetree: etc/nginx
register: __managed_files
- name: etc config - Convert managed files to list
set_fact: managed_files="{{ __managed_files.results | map(attribute='ansible_facts.__managed_file_path') | list }}"
- name: etc config - Gathering exist files (excluding .ansible_keep-content dirs)
shell: find /etc/nginx -mindepth 1 -type d -exec test -e '{}/.ansible_keep-content' \; -prune -o -print
register: exist_files
changed_when: False
- name: etc config - Delete unmanaged files
file: path="{{ item }}" state=absent
with_items: "{{ exist_files.stdout_lines }}"
when:
- item not in managed_files
Here's something I came up with:
- template: src=/source/directory{{ item }}.j2 dest=/target/directory/{{ item }}
register: template_results
with_items:
- a_list.txt
- of_all.txt
- templates.txt
- set_fact:
managed_files: "{{ template_results.results|selectattr('invocation', 'defined')|map(attribute='invocation.module_args.dest')|list }}"
- debug:
var: managed_files
verbosity: 0
- find:
paths: "/target/directory/"
patterns: "*.txt"
register: all_files
- set_fact:
files_to_delete: "{{ all_files.files|map(attribute='path')|difference(managed_files) }}"
- debug:
var: all_files
verbosity: 0
- debug:
var: files_to_delete
verbosity: 0
- file: path={{ item }} state=absent
with_items: "{{ files_to_delete }}"
This generates the templates (however which way you want), and records the results in 'template_results'
The the results are mangled to get a simple list of the "dest" of each template. Skipped templates (due to a when condition, not shown) have no "invocation" attribute, so they're filtered out.
"find" is then used to get a list of all files that should be absent unless specifically written.
this is then mangled to get a raw list of files present, and then the "supposed to be there" files are removed.
The remaining "files_to_delete" are then removed.
Pros: You avoid multiple 'skipped' entries showing up during deletes.
Cons: You'll need to concatenate each template_results.results if you want to do multiple template tasks before doing the find/delete.
There might be a couple of ways to handle this, but would it be possible to entirely empty the target directory in a task before the template step? Or maybe drop the templated files into a temporary directory and then delete+rename in a subsequent step?
Usually I do not remove files but I add -unmanaged suffix to its name.
Sample ansible tasks:
- name: Get sources.list.d files
shell: grep -r --include=\*.list -L '^# Ansible' /etc/apt/sources.list.d || true
register: grep_unmanaged
changed_when: grep_unmanaged.stdout_lines
- name: Add '-unmanaged' suffix
shell: rename 's/$/-unmanaged/' {{ item }}
with_items: grep_unmanaged.stdout_lines
EXPLANATION
Grep command uses:
-r to do recursive search
--include=\*.list - only take files
with .list extension during recursive search
-L '^# Ansible' - display file names that are not having line starting with '# Ansible'
|| true - this is used to ignore errors. Ansible's ignore_errors also works but before ignoring the error ansible will show it in red color during ansible-playbook run
which is undesired (at least for me).
Then I register output of grep command as a variable. When grep displays any output I set this task as changed (the line changed_when is responsible for this).
In next task I iterate grep output (i.e. file names returned by grep) and run rename command to add suffix to each file.
That's all. Next time you run the command first task should be green and second skipped.
I am using Ansible version 2.9.20
---
# tasks file for delete_unmanaged_files
- name: list files in dest
shell: ls -1 dest/conf.d
register: files_in_dest
- name: list files in src
shell: ls -1 src/conf.d
register: files_in_src
- name: Managed files - dest
command: echo {{ item|replace('.j2','') }}
with_items: "{{ files_in_dest.stdout_lines }}"
register: managed_files_dest
- name: Managed files - src
command: echo {{ item|replace('.j2','') }}
with_items: "{{ files_in_src.stdout_lines }}"
register: managed_files_src
- name: Convert src managed files to list
set_fact: managed_files_src_list="{{ managed_files_src.results | map(attribute='stdout') | list }}"
- name: Delete unmanaged files in dest
file: path=dest/conf.d/{{ item.stdout }} state=absent
with_items: "{{ managed_files_dest.results }}"
when: item.stdout not in managed_files_src_list
I think depending on the usecase of this issue, I found above solution might help you. Here, I have created 6 tasks.
Explanation:
Task-1 & Task-2 will help storing file names in variable "files_in_dest" and "files_in_src" in it.
Task-3 & Task-4 will inherit the output coming from Task-1 & Task-2 and then replace the j2 file (required for the usecase). Then these tasks will store the output in "managed_files_dest" and "managed_files_src" variables.
Task-5 will convert the output of "managed_files_src" to list, so that we can have all the present files (at current state) stored in src directory in a proper or single list and then we can use this list in next task to know the unmanaged files in dest directory.
Task-6 will delete the unmanaged files in dest.
Apparently this isn't possible with ansible at the moment. I had a conversation with mdehaan on IRC and it boils down to ansible not having a directed acyclic graph for resources, making things like this very hard.
Asking mdehaan for an example e.g. authoritatively managing a sudoers.d directory he came up with these things:
14:17 < mdehaan> Robe: http://pastebin.com/yrdCZB0y
14:19 < Robe> mdehaan: HM
14:19 < Robe> mdehaan: that actually looks relatively sane
14:19 < mdehaan> thanks :)
14:19 < Robe> the problem I'm seeing is that I'd have to gather the managed files myself
14:19 < mdehaan> you would yes
14:19 < mdehaan> ALMOST
14:20 < mdehaan> you could do a fileglob and ... well, it would be a little gross
[..]
14:32 < mdehaan> eh, theoretical syntax, nm
14:33 < mdehaan> I could do it by writing a lookup plugin that filtered a list
14:34 < mdehaan> http://pastebin.com/rjF7QR24
14:34 < mdehaan> if that plugin existed, for instance, and iterated across lists in A that were also in B
Building on #user2645850's answer I came up with this improved version, in this case it manages the vhost configuration of Apache. It doesn't use shell and thus works also in --check mode.
# Remove unmanged vhost configs left over from renaming or removing apps
# all managed configs need to be added to "managed_sites" in advance
- find:
paths: /etc/apache2/sites-available
patterns: '*.conf'
register: sites_available_contents
- name: Remove unmanaged vhost config files
file:
path: /etc/apache2/sites-available/{{ item }}
state: absent
with_items: "{{ sites_available_contents.files | map(attribute='path') | map('basename') | list }}"
when: item not in managed_sites
# links may differ from files, therefore we need our own find task for them
- find:
paths: /etc/apache2/sites-enabled
file_type: any
register: sites_enabled_contents
- name: Remove unmanaged vhost config links
file:
path: /etc/apache2/sites-enabled/{{ item }}
state: absent
with_items: "{{ sites_enabled_contents.files | map(attribute='path') | map('basename') | list }}"
when: item not in managed_sites
Examples on how to build managed_sites:
# Add single conf and handle managed_sites being unset
- set_fact:
managed_sites: "{{ (managed_sites | default([])) + [ '000-default.conf' ] }}"
# Add a list of vhosts appending ".conf" to each entry of vhosts
- set_fact:
managed_sites: "{{ managed_sites + ( vhosts | map(attribute='app') | product(['.conf']) | map('join') | list ) }}"