Find files in a loop and delete them - ansible

I would like to delete file in some sub directory which contains certain format. However, I am getting the error
'dict object' has no attribute 'files'.
Below is my code. The file pattern would be file_name.file_extension.processID.YYYY-MM-DD#HH:MM:SS~
My variables
fileToFindInAllSubDirecotry
- "home/usr/file1"
- "home/usr/file2"
- "home/usr/file3/file4"
- "home/usr/file5"
My playbook role
- name: Find file
find:
paths: "{{ item }}"
use_regex: yes
patterns:
- '.*\.\d+\.\d{4}-\d{2}-\d{2}#\d{2}:\d{2}:\d{2}~$'
age: 1d
recurse: yes
register: fileToDelete
loop: "{{ fileToFindInAllSubDirecotry }}"
- name: Delete file
file:
path: "{{ item.path }}"
state: absent
loop: "{{ fileToDelete.files }}"
This is the sample file and directory
home
|-------usr
|-------file1
|-------configFile.xml
|-------file2
|-------propertiesFile.txt.2012.2020-07-13#23:08:10~
|-------file3
|-------file4
|-------content.yml.2012.2020-04-04#23:08:10~
|-------file5
|-------configFile.xml.2012.2020-03-05#13:08:10~

This is happening because you are populating the find with a loop, so you end up with a result that would be a dictionary having a list or results.
Something like:
ok: [localhost] => {
"msg": {
"changed": false,
"msg": "All items completed",
"results": [
{
...
"files": [ ... ],
...
"item": "/home/usr/file1",
...
},
{
...
"files": [ ... ],
...
"item": "/home/usr/file2",
...
},
...
]
}
}
There is two ways to fix this:
The nicest one, because, as pointed by the documentation, the paths parameter of the module find can accept lists of paths, just pass it your whole fileToFindInAllSubDirecotry variables instead of using a loop, this way your deletion works as is:
- name: Find file
find:
paths: "{{ fileToFindInAllSubDirecotry }}"
use_regex: yes
patterns:
- '.*\.\d+\.\d{4}-\d{2}-\d{2}#\d{2}:\d{2}:\d{2}~$'
age: 1d
recurse: yes
register: fileToDelete
- name: Delete file
file:
path: "{{ item.path }}"
state: absent
loop: "{{ fileToDelete.files }}"
Use json_query to fetch the result[*].files then flatten the resulting list of list
- name: Find file
find:
paths: "{{ item }}"
use_regex: yes
patterns:
- '.*\.\d+\.\d{4}-\d{2}-\d{2}#\d{2}:\d{2}:\d{2}~$'
age: 1d
recurse: yes
register: fileToDelete
loop: "{{ fileToFindInAllSubDirecotry }}"
- name: Delete file
file:
path: "{{ item.path }}"
state: absent
loop: "{{ fileToDelete | json_query('results[*].files') | flatten }}"

Related

How to include variables with include_vars with the same name without overwriting previous

I am having this let's call it include.yaml
#- name: "Playing with Ansible and Include files"
- hosts: localhost
connection: local
tasks:
- find: paths="./" recurse=yes patterns="test.yaml"
register: file_to_exclude
- debug: var=file_to_exclude.stdout_lines
- name: shell
shell: "find \"$(pwd)\" -name 'test.yaml'"
register: files_from_dirs
- debug: var=files_from_dirs.stdout_lines
- name: Include variable files
include_vars: "{{ item }}"
with_items:
- "{{ files_from_dirs.stdout_lines }}"
- debug: var=files
and 2 ore more test files
./dir1/test.yaml
that contains
files:
- file1
- file2
./dir2/test.yaml
that contains
files:
- file3
- file4
the result is
TASK [Include variable files] ******************************************************************************************
ok: [localhost] => (item=/mnt/c/Users/GFlorinescu/ansible_scripts/ansible/1st/test.yaml)
ok: [localhost] => (item=/mnt/c/Users/GFlorinescu/ansible_scripts/ansible/2nd/test.yaml)
TASK [debug] ***********************************************************************************************************
ok: [localhost] => {
"files": [
"file3",
"file4"
]
}
How can I get all the values in files, at the moment the last included files variable from last file overrides the files from the previous files? Of course without changing the variables names in files test.yaml?
In other words I want files to be:
ok: [localhost] => {
"files": [
"file1",
"file2",
"file3",
"file4"
]
}
To be more specific, I ask for any kind of solution or module, even not official or some github module, I don't want a specific include_vars module solution.
Put the included variables into the dictionaries with unique names. For example, create the names from the index of the loop. Then, iterate the names and concatenate the lists
- command: "find {{ playbook_dir }} -name test.yaml"
register: files_from_dirs
- include_vars:
file: "{{ item }}"
name: "{{ name }}"
loop: "{{ files_from_dirs.stdout_lines }}"
loop_control:
extended: true
vars:
name: "files_{{ ansible_loop.index }}"
- set_fact:
files: "{{ files|d([]) + lookup('vars', item).files }}"
with_varnames: "files_[0-9]+"
- debug:
var: files
give
files:
- file1
- file2
- file3
- file4
Notes:
You have to provide either a path relative to the home directory or an absolute path. See the example below
- command: "echo $PWD"
register: out
- debug:
var: out.stdout
give
out.stdout: /home/admin
For example, when you want to find the files relative to the directory of the playbook
- command: "find {{ playbook_dir }} -name test.yaml"
register: files_from_dirs
- debug:
var: files_from_dirs.stdout_lines
give
files_from_dirs.stdout_lines:
- /export/scratch/tmp8/test-987/dir1/test.yaml
- /export/scratch/tmp8/test-987/dir2/test.yaml
The same is valid for the module find. For example,
- find:
paths: "{{ playbook_dir }}"
recurse: true
patterns: test.yaml
register: files_from_dirs
- debug:
var: files_from_dirs.files|map(attribute='path')|list
give the same result
files_from_dirs.files|map(attribute='path')|list:
- /export/scratch/tmp8/test-987/dir1/test.yaml
- /export/scratch/tmp8/test-987/dir2/test.yaml
Simplify the code and put the declaration of files into the vars. For example, the below declaration gives the same result
files: "{{ query('varnames', 'files_[0-9]+')|
map('extract', hostvars.localhost, 'files')|
flatten }}"
Example of a complete playbook for testing
- hosts: localhost
vars:
files: "{{ query('varnames', 'files_[0-9]+')|
map('extract', hostvars.localhost, 'files')|
flatten }}"
tasks:
- find:
paths: "{{ playbook_dir }}"
recurse: true
patterns: test.yaml
register: files_from_dirs
- include_vars:
file: "{{ item }}"
name: "{{ name }}"
loop: "{{ files_from_dirs.files|map(attribute='path')|list }}"
loop_control:
extended: true
vars:
name: "files_{{ ansible_loop.index }}"
- debug:
var: files
(maybe off-topic, see comments)
Q: "Is there a way to write the path where it was found?"
A: Yes, it is. See the self-explaining example below. Given the inventory
shell> cat hosts
host_1 file_1=alice
host_2 file_2=bob
host_3
the playbook
- hosts: host_1,host_2,host_3
vars:
file_1_list: "{{ hostvars|json_query('*.file_1') }}"
file_2_list: "{{ hostvars|json_query('*.file_2') }}"
file_1_dict: "{{ dict(hostvars|dict2items|
selectattr('value.file_1', 'defined')|
json_query('[].[key, value.file_1]')) }}"
file_1_lis2: "{{ hostvars|dict2items|
selectattr('value.file_1', 'defined')|
json_query('[].{key: key, file_1: value.file_1}') }}"
tasks:
- debug:
msg: |-
file_1_list: {{ file_1_list }}
file_2_list: {{ file_2_list }}
file_1_dict:
{{ file_1_dict|to_nice_yaml|indent(2) }}
file_1_lis2:
{{ file_1_lis2|to_nice_yaml|indent(2) }}
run_once: true
gives
msg: |-
file_1_list: ['alice']
file_2_list: ['bob']
file_1_dict:
host_1: alice
file_1_lis2:
- file_1: alice
key: host_1

Loop through a list of folders to delete old ones

I want to implement some sort of rotation on subdirectories of folders in a list. Say I have dir1 and dir2, I need to go inside each of them and delete all folders older than X days in dir1 and Y days in dir2.
vars:
backups:
dir1:
name: dir1
days: 10d
dir2:
name: dir2
days: 3d
I've tried to create a task like this
- name: find all folders
find:
paths: "/home/user1/{{ item.value.name }}"
age: "{{ item.value.days }}"
file_type: directory
loop: "{{ lookup('dict', backups) }}"
register: dirsOlderThanXd
But, dirsOlderThanXd has a strange format. If there were no loop and just a single directory next step would be something like
- name: remove old folders
file:
path: "{{ item.path }}"
state: absent
with_items: "{{ dirsOlderThanXd.files }}"
Documentation says
When you use register with a loop, the data structure placed in the variable will contain a results attribute that is a list of all responses from the module. This differs from the data structure returned when using register without a loop.
So, it's expected, but, how exactly do I work with this output? Or am I doing it all completely wrong?
You can access the list — results — of list — files — using a map filter, then flatten the resulting list of list:
- name: remove old folders
file:
path: "{{ item.path }}"
state: absent
loop: "{{ dirsOlderThanXd.results | map(attribute='files') | flatten }}"
loop_control:
label: "{{ item.path }}"
Given the two tasks:
- name: find all folders
find:
paths: "/home/user1/{{ item.value.name }}"
age: "{{ item.value.days }}"
file_type: directory
loop: "{{ backups | dict2items }}"
loop_control:
label: "{{ item.key }}"
register: dirsOlderThanXd
vars:
backups:
dir1:
name: dir1
days: 10d
dir2:
name: dir2
days: 3d
- name: remove old folders
file:
path: "{{ item.path }}"
state: absent
loop: "{{ dirsOlderThanXd.results | map(attribute='files') | flatten }}"
loop_control:
label: "{{ item.path }}"
This would yields something along the line of:
TASK [find all folders] **************************************************
ok: [localhost] => (item=dir1)
ok: [localhost] => (item=dir2)
TASK [remove old folders] ************************************************
changed: [localhost] => (item=/home/user1/dir1/foo)
changed: [localhost] => (item=/home/user1/dir1/bar)
changed: [localhost] => (item=/home/user1/dir2/qux)
changed: [localhost] => (item=/home/user1/dir2/baz)

Ansible delete Files with wildcard/regex/glob with exception

I want to delete files based on a wildcard but also add exceptions to the rule.
- hosts: all
tasks:
- name: Ansible delete file wildcard
find:
paths: /etc/wild_card/example
patterns: "*.txt"
use_regex: true
register: wildcard_files_to_delete
- name: Ansible remove file wildcard
file:
path: "{{ item.path }}"
state: absent
with_items: "{{ wildcard_files_to_delete.files }}"
For example I want to except a file named "important.txt". How can I do that?
Just add a when condition to the task that deletes files. E.g., something like:
- name: Ansible remove file wildcard
file:
path: "{{ item.path }}"
state: absent
when: item.path != '/etc/wild_card/example/important.txt'
with_items: "{{ wildcard_files_to_delete.files }}"
This will skip a specific file. If you have a list of files to skip you could do this instead:
- name: Ansible remove file wildcard
file:
path: "{{ item.path }}"
state: absent
when: item.path not in files_to_skip
with_items: "{{ wildcard_files_to_delete.files }}"
vars:
files_to_skip:
- /etc/wild_card/example/important.txt
- /etc/wild_card/example/saveme.txt
And if you want to preserve files based on some sort of pattern, you could make use of ansible's match or search tests:
- name: Ansible remove file wildcard
file:
path: "{{ item.path }}"
state: absent
when: item.path is not search('important.txt')
with_items: "{{ wildcard_files_to_delete.files }}"

Ansible: Copy multiple files from multiple sources to multiple destinations in single command

I am trying to copy some configuration files from /tmp to /opt.
Here first I am recursively searching for the files in /tmp and /opt directories and storing it in the variable tmp_file_path and code_file_path respectively, which has an attribute files.path that I need to use in the source and destination for copy
- name: Find files in tmp
find:
paths: /tmp/
file_type: file
recurse: yes
patterns:
- file1
- file2
- file3
register: tmp_file_path
- debug:
var: tmp_file_path
- name: Find files in code
find:
paths: /opt/
file_type: file
recurse: yes
patterns:
- file1
- file2
- file3
register: code_file_path
- debug:
var: code_file_path
Here the source file paths can be /tmp/folder1/file1, /tmp/folder2/file2, /tmp/folder13/file3.
Destinations can be /opt/folderA/file1, /opt/folderB/file3, /opt/folderC/file3
As of now I have managed to write the task as below
- name: Copy files from tmp to code directory
copy:
src: "{{item.path}}"
dest: "{{item.path}}"
remote_src: yes
with_items:
- { "{{ tmp_file_path.files }}", "{{ code_file_path.files }}" }
The copy has to be done in a single command so that I do not end up hardcoding the paths for source and destination. Can anyone help me with achieving this?
Below piece of code worked for recursively copying files from code_file_path to /tmp
- name: Copy files from code directory to tmp
copy:
src: "{{item.path}}"
dest: /tmp/
remote_src: yes
with_items: "{{code_file_path.files}}"
Try this
- name: Copy files from tmp to code directory
copy:
src: "{{ item.0 }}"
dest: "{{ item.1 }}"
remote_src: yes
with_together:
- "{{ tmp_file_path.files|map(attribute='path')|list|sort }}"
- "{{ code_file_path.files|map(attribute='path')|list|sort }}"
Try with debug first
- debug:
msg:
- "src: {{ item.0 }}"
- "dest: {{ item.1 }}"
with_together:
- "{{ tmp_file_path.files|map(attribute='path')|list|sort }}"
- "{{ code_file_path.files|map(attribute='path')|list|sort }}"
It might be useful to test the sanity first
- debug:
msg: The numbers of files do not match
when: tmp_file_path.files|map(attribute='path')|list|length !=
code_file_path.files|map(attribute='path')|list|length
Q: "This did not work as expected because the files which I am searching in code_file_path has multiple sub-directories. It is sorting on the entire file path returned by code_file_path.files rather than just the file name."
A: Create a list with both paths and names. Then sort the list by the name. For example
- set_fact:
code_files: "{{ code_files|default([]) +
[{'path': item, 'name': item|basename}] }}"
loop: "{{ code_file_path.files|map(attribute='path')|list }}"
- debug:
msg:
- "src: {{ item.0 }}"
- "dest: {{ item.1.path }}"
with_together:
- "{{ tmp_file_path.files|map(attribute='path')|list|sort }}"
- "{{ code_files|sort(attribute='name') }}"
Example
shell> tree tmp
tmp
├── file1
├── file2
└── file3
shell> tree opt
opt
├── bar
│   └── file2
├── baz
│   └── file3
└── foo
└── file1
The tasks below
- set_fact:
code_files: "{{ code_files|default([]) +
[{'path': item, 'name': item|basename}] }}"
loop: "{{ code_file_path.files|map(attribute='path')|list }}"
- debug:
var: code_files|sort(attribute='name')
- debug:
msg:
- "src: {{ item.0 }}"
- "dest: {{ item.1.path }}"
with_together:
- "{{ tmp_file_path.files|map(attribute='path')|list|sort }}"
- "{{ code_files|sort(attribute='name') }}"
give
"code_files|sort(attribute='name')": [
{
"name": "file1",
"path": "/export/test/opt/foo/file1"
},
{
"name": "file2",
"path": "/export/test/opt/bar/file2"
},
{
"name": "file3",
"path": "/export/test/opt/baz/file3"
}
]
"msg": [
"src: /export/test/tmp/file1",
"dest: /export/test/opt/foo/file1"
]
"msg": [
"src: /export/test/tmp/file2",
"dest: /export/test/opt/bar/file2"
]
"msg": [
"src: /export/test/tmp/file3",
"dest: /export/test/opt/baz/file3"
]
You may achieve this with this condition : every filenames are unique in the way that if 2 files have the same name in your list from /opt directory, then their content should be the same.
If it's the case, then you may use the with_nested loop and using a conditionnal when on filename.
For example:
- name: Copy files from tmp to code directory
copy:
src: "{{ item.0 }}"
dest: "{{ item.1 }}"
remote_src: yes
when:
- item.0|basename == item.1|basename
with_nested:
- "{{ tmp_file_path.files|map(attribute='path')|list }}"
- "{{ code_file_path.files|map(attribute='path')|list }}"
The only "problem" with this solution is that you'll run the loop a lot of times...
You may also want to use the loop synthax over with_ synthax and use some loop_control in order to choose what is printed: https://docs.ansible.com/ansible/latest/user_guide/playbooks_loops.html
If you have the jinja.ext.do extension loaded (in your ansible.cfg: jinja2_extensions = jinja2.ext.do) you may then contruct a dict with your paths:
- name: Make a dict
set_fact:
files_dict: |
{%- set out_dict = dict() -%}
{%- for tmp_file in tmp_file_path.files|map(attribute='path')|list -%}
{%- do out_dict.update({tmp_file|basename: {'tmp': tmp_file}}) -%}
{%- endfor -%}
{%- for opt_file in code_file_path.files|map(attribute='path')|list -%}
{%- do out_dict[opt_file|basename].update({'opt': opt_file}) -%}
{%- endfor -%}
{{ out_dict }}
- name: Copy from dict
copy:
src: "{{ item.value.tmp }}"
dest: "{{ item.value.opt }}"
remote_src: yes
with_dict: "{{ files_dict }}"

win_find FAILED! => {"msg": "'dict object' has no attribute 'files'"}

I am trying to run win_find with variables for "paths" to find temp files (e.g. c:\users\public\appdata\local\temp) but when I pass it to debug I get the error message
"FAILED! => {"msg": "'dict object' has no attribute 'files'"}"
I have tried swapping the "register" and "with_items" lines but that seems to make no difference at all.
- hosts: windows-02
vars:
dir: \AppData\Local\Temp\
tasks:
- name: find user directories
win_find:
paths: c:\users
recurce: yes
file_type: directory
get_checksum: no
register: user_dir
- name: debug 1
debug:
msg: "{{ item.path + dir }}"
loop: "{{ user_dir.files }}"
- name: find temp files
win_find:
paths: "{{ item.path + dir }}"
recurce: yes
hidden: yes
get_checksum: no
register: files_to_delete
with_items: "{{ user_dir.files }}"
- name: debug
debug:
msg: "{{ item }}"
loop: "{{ files_to_delete.files }}"
- name: remove
win_file:
path: "{{ item.path }}"
state: absent
with_items: "{{ files_to_delete.files }}"
I expect to get a list of files to be deleted which will be passed to the "win_files" module. Instead I'm getting the error message
"FAILED! => {"msg": "'dict object' has no attribute 'files'"}"
There is a typo in win_find module, it is recurse: yes
Please find the below code whcih worked for me
---
- name: Find files
win_find:
paths: "{{paths}}"
age: "{{duration}}"
register: log_files
- name: Delete the files
win_file:
path: "{{item.path}}"
state: absent
with_items: "{{log_files.files}}"
Also files_to_delete will not have files_to_delete.files as it is in loop. It will have files_to_delete.results
So, it appears that win_find, at least in my case, does not return files when using "when_items". I got around this by creating a list of paths and passing it into the "paths" parameter of the module.
Here is my code:
- hosts: "{{ host }}"
vars:
dir: \AppData\Local\Temp\
temp_paths: []
line_break: \n
tasks:
- name: find user directories
win_find:
paths: c:\users
recurse: no
file_type: directory
get_checksum: no
register: user_dir
- name: debug 1
debug:
msg: "{{ item.path + dir }}"
loop: "{{ user_dir.files }}"
- name: set temp path
set_fact: temp_paths="{{temp_paths + [ item.path + dir ] }}"
when: item.path != 'C:\\users\\Public'
with_items: "{{ user_dir.files }}"
- name: find temp files
win_find:
paths: "{{ temp_paths }}"
recurse: True
patterns: '*'
hidden: False
get_checksum: False
register: files_to_delete
# ignore_errors: yes
- name: debug
debug:
msg: "{{ item.path }}"
loop: "{{ files_to_delete.files }}"
- name: remove
win_file:
path: "{{ item.path }}"
state: absent
with_items: "{{ files_to_delete.files }}"

Resources