I want to implement some sort of rotation on subdirectories of folders in a list. Say I have dir1 and dir2, I need to go inside each of them and delete all folders older than X days in dir1 and Y days in dir2.
vars:
backups:
dir1:
name: dir1
days: 10d
dir2:
name: dir2
days: 3d
I've tried to create a task like this
- name: find all folders
find:
paths: "/home/user1/{{ item.value.name }}"
age: "{{ item.value.days }}"
file_type: directory
loop: "{{ lookup('dict', backups) }}"
register: dirsOlderThanXd
But, dirsOlderThanXd has a strange format. If there were no loop and just a single directory next step would be something like
- name: remove old folders
file:
path: "{{ item.path }}"
state: absent
with_items: "{{ dirsOlderThanXd.files }}"
Documentation says
When you use register with a loop, the data structure placed in the variable will contain a results attribute that is a list of all responses from the module. This differs from the data structure returned when using register without a loop.
So, it's expected, but, how exactly do I work with this output? Or am I doing it all completely wrong?
You can access the list — results — of list — files — using a map filter, then flatten the resulting list of list:
- name: remove old folders
file:
path: "{{ item.path }}"
state: absent
loop: "{{ dirsOlderThanXd.results | map(attribute='files') | flatten }}"
loop_control:
label: "{{ item.path }}"
Given the two tasks:
- name: find all folders
find:
paths: "/home/user1/{{ item.value.name }}"
age: "{{ item.value.days }}"
file_type: directory
loop: "{{ backups | dict2items }}"
loop_control:
label: "{{ item.key }}"
register: dirsOlderThanXd
vars:
backups:
dir1:
name: dir1
days: 10d
dir2:
name: dir2
days: 3d
- name: remove old folders
file:
path: "{{ item.path }}"
state: absent
loop: "{{ dirsOlderThanXd.results | map(attribute='files') | flatten }}"
loop_control:
label: "{{ item.path }}"
This would yields something along the line of:
TASK [find all folders] **************************************************
ok: [localhost] => (item=dir1)
ok: [localhost] => (item=dir2)
TASK [remove old folders] ************************************************
changed: [localhost] => (item=/home/user1/dir1/foo)
changed: [localhost] => (item=/home/user1/dir1/bar)
changed: [localhost] => (item=/home/user1/dir2/qux)
changed: [localhost] => (item=/home/user1/dir2/baz)
Related
I am having this let's call it include.yaml
#- name: "Playing with Ansible and Include files"
- hosts: localhost
connection: local
tasks:
- find: paths="./" recurse=yes patterns="test.yaml"
register: file_to_exclude
- debug: var=file_to_exclude.stdout_lines
- name: shell
shell: "find \"$(pwd)\" -name 'test.yaml'"
register: files_from_dirs
- debug: var=files_from_dirs.stdout_lines
- name: Include variable files
include_vars: "{{ item }}"
with_items:
- "{{ files_from_dirs.stdout_lines }}"
- debug: var=files
and 2 ore more test files
./dir1/test.yaml
that contains
files:
- file1
- file2
./dir2/test.yaml
that contains
files:
- file3
- file4
the result is
TASK [Include variable files] ******************************************************************************************
ok: [localhost] => (item=/mnt/c/Users/GFlorinescu/ansible_scripts/ansible/1st/test.yaml)
ok: [localhost] => (item=/mnt/c/Users/GFlorinescu/ansible_scripts/ansible/2nd/test.yaml)
TASK [debug] ***********************************************************************************************************
ok: [localhost] => {
"files": [
"file3",
"file4"
]
}
How can I get all the values in files, at the moment the last included files variable from last file overrides the files from the previous files? Of course without changing the variables names in files test.yaml?
In other words I want files to be:
ok: [localhost] => {
"files": [
"file1",
"file2",
"file3",
"file4"
]
}
To be more specific, I ask for any kind of solution or module, even not official or some github module, I don't want a specific include_vars module solution.
Put the included variables into the dictionaries with unique names. For example, create the names from the index of the loop. Then, iterate the names and concatenate the lists
- command: "find {{ playbook_dir }} -name test.yaml"
register: files_from_dirs
- include_vars:
file: "{{ item }}"
name: "{{ name }}"
loop: "{{ files_from_dirs.stdout_lines }}"
loop_control:
extended: true
vars:
name: "files_{{ ansible_loop.index }}"
- set_fact:
files: "{{ files|d([]) + lookup('vars', item).files }}"
with_varnames: "files_[0-9]+"
- debug:
var: files
give
files:
- file1
- file2
- file3
- file4
Notes:
You have to provide either a path relative to the home directory or an absolute path. See the example below
- command: "echo $PWD"
register: out
- debug:
var: out.stdout
give
out.stdout: /home/admin
For example, when you want to find the files relative to the directory of the playbook
- command: "find {{ playbook_dir }} -name test.yaml"
register: files_from_dirs
- debug:
var: files_from_dirs.stdout_lines
give
files_from_dirs.stdout_lines:
- /export/scratch/tmp8/test-987/dir1/test.yaml
- /export/scratch/tmp8/test-987/dir2/test.yaml
The same is valid for the module find. For example,
- find:
paths: "{{ playbook_dir }}"
recurse: true
patterns: test.yaml
register: files_from_dirs
- debug:
var: files_from_dirs.files|map(attribute='path')|list
give the same result
files_from_dirs.files|map(attribute='path')|list:
- /export/scratch/tmp8/test-987/dir1/test.yaml
- /export/scratch/tmp8/test-987/dir2/test.yaml
Simplify the code and put the declaration of files into the vars. For example, the below declaration gives the same result
files: "{{ query('varnames', 'files_[0-9]+')|
map('extract', hostvars.localhost, 'files')|
flatten }}"
Example of a complete playbook for testing
- hosts: localhost
vars:
files: "{{ query('varnames', 'files_[0-9]+')|
map('extract', hostvars.localhost, 'files')|
flatten }}"
tasks:
- find:
paths: "{{ playbook_dir }}"
recurse: true
patterns: test.yaml
register: files_from_dirs
- include_vars:
file: "{{ item }}"
name: "{{ name }}"
loop: "{{ files_from_dirs.files|map(attribute='path')|list }}"
loop_control:
extended: true
vars:
name: "files_{{ ansible_loop.index }}"
- debug:
var: files
(maybe off-topic, see comments)
Q: "Is there a way to write the path where it was found?"
A: Yes, it is. See the self-explaining example below. Given the inventory
shell> cat hosts
host_1 file_1=alice
host_2 file_2=bob
host_3
the playbook
- hosts: host_1,host_2,host_3
vars:
file_1_list: "{{ hostvars|json_query('*.file_1') }}"
file_2_list: "{{ hostvars|json_query('*.file_2') }}"
file_1_dict: "{{ dict(hostvars|dict2items|
selectattr('value.file_1', 'defined')|
json_query('[].[key, value.file_1]')) }}"
file_1_lis2: "{{ hostvars|dict2items|
selectattr('value.file_1', 'defined')|
json_query('[].{key: key, file_1: value.file_1}') }}"
tasks:
- debug:
msg: |-
file_1_list: {{ file_1_list }}
file_2_list: {{ file_2_list }}
file_1_dict:
{{ file_1_dict|to_nice_yaml|indent(2) }}
file_1_lis2:
{{ file_1_lis2|to_nice_yaml|indent(2) }}
run_once: true
gives
msg: |-
file_1_list: ['alice']
file_2_list: ['bob']
file_1_dict:
host_1: alice
file_1_lis2:
- file_1: alice
key: host_1
I have the following task:
- name: Create required folders.
become: true
ansible.builtin.file:
owner: "{{ item.key }}"
group: ftp
mode: '0755'
path: '/data/{{ item.key }}/in'
state: directory
loop: "{{ query('dict', ftp) | list }}"
when: "'state' not in item.value or item.value.state == 'present'"
And the following host variables with different users:
ftp:
test:
ssh_public_key: "XXXX"
password: "XXX"
home: /data/test
test2:
ssh_public_key: "XXXX"
password: "XXX"
home: /data/test2
What I want is to create two directories for every user :
path: '/data/{{ user }}/in' # item.key, in the code above
path: '/data/{{ user }}/out' # item.key, in the code above
But I already need the loop for iterating over the users itself:
loop: "{{ query('dict', ftp) | list }}"
How can I handle this, for example, with nested loop?
Use a product filter to generate every possible combination of user/folder.
loop: "{{ ftp.keys() | product(['in', 'out']) }}"
Then, respectively,
item.0 contains the users dictionary keys
item.1 contains the folders
It is not fully clear what your condition when does actually, in order to adapt it too, but I guess that you do have an absent or present state in those use dictionaries.
So, the resulting task should be something along the lines of
- name: Create required folders
ansible.builtin.file:
owner: "{{ item.0 }}"
group: ftp
mode: '0755'
path: "/data/{{ item.0 }}/{{ item.1 }}"
state: directory
loop: "{{ ftp.keys() | product(['in', 'out']) }}"
loop_control:
label: "/data/{{ item.0 }}/{{ item.1 }}"
when: "ftp[item.0].state | default('absent') == 'present'"
become: true
Given the task above, when run on those data:
ftp:
test:
state: present
test1:
test2:
state: present
It will yield:
TASK [Create required folders] ***************************************
ok: [localhost] => (item=/data/test/in)
ok: [localhost] => (item=/data/test/out)
skipping: [localhost] => (item=/data/test1/in)
skipping: [localhost] => (item=/data/test1/out)
ok: [localhost] => (item=/data/test2/in)
ok: [localhost] => (item=/data/test2/out)
Test it first, for example
- debug:
msg: "Create /data/{{ item.0.key }}/{{ item.1 }}"
with_nested:
- "{{ ftp|dict2items }}"
- [in, out]
when: item.0.value.state|d('present') == 'present'
gives (abridged)
msg: Create /data/test/in
msg: Create /data/test/out
msg: Create /data/test2/in
msg: Create /data/test2/out
Then try to create the dictionaries
- file:
owner: "{{ item.0.key }}"
group: ftp
mode: '0755'
path: "/data/{{ item.0.key }}/{{ item.1 }}"
state: directory
with_nested:
- "{{ ftp|dict2items }}"
- [in, out]
when: item.0.value.state|d('present') == 'present'
(not tested)
I am trying to list a set of directories with their creation times in an ansible playbook on MacOS.
The problem I have is that the file and stat modules give you the "ctime" of a file, but on MacOS this is the time of last metadata change, not the creation date.
So for example this playbook:
- name: Get dirs and create dates
hosts: localhost
tasks:
- name: make a list of dirs
find:
paths:
- "/Users/me/Work/Customers"
file_type: directory
recurse: false
register: projectdirs
- name: Dump found paths
debug:
msg: "{{ shortname }} {{ date }}"
loop: "{{ projectdirs.files }}"
loop_control:
label: "{{ item.path }}"
vars:
shortname: "{{ item.path | basename }}"
date: "{{ '%Y-%m-%d' | strftime(item.ctime) }}"
This just lists todays date as "creation" date for the directory.
ok: [localhost] => (item=/Users/me/Work/Customers/Base) => {
"msg": "basefarm 2021-08-23 1629699678.021843"
}
ok: [localhost] => (item=/Users/me/Work/Customers/Acme) => {
"msg": "Orange 2021-08-23 1629699678.5415485"
}
ok: [localhost] => (item=/Users/me/Work/Customers/Foo) => {
"msg": "VKB 2021-08-23 1629699679.0579438"
}
ok: [localhost] => (item=/Users/me/Work/Customers/Bar) => {
"msg": "LombardOdier 2021-08-23 1629699679.5856457"
}
...
I found out that the reason is of course that ctime is not the creation time on a mac...
Is there a way around this? I want the real creation date, but Ansible's stat module does not give me this. The creation date is available, as I can get it with ls -ldU for example.
So how do I solve this?
As far as I know, there is no way to do so without using shell command.
- name: make a list of dirs
find:
paths:
- "/Users/me/Work/Customers"
file_type: directory
recurse: false
register: projectdirs
- name: Get creation time
shell: GetFileInfo {{ item.path }} | grep 'created:'
loop: "{{ projectdirs.files }}"
loop_control:
label: "{{ item.path }}"
register: directories
- name: Dump found paths
debug:
msg: "{{ item.item.path }} {{ item.stdout.split('created: ')[1] }}"
loop: "{{ directories.results }}"
loop_control:
label: "{{ item.item.path }}"
Gives me this result :
ok: [localhost] => (item=/Users/jeromeverdoni/stackoverflow/68888321) => {
"msg": "/Users/jeromeverdoni/stackoverflow/68888321 08/23/2021 08:01:44"
}
Hope this is close enough to your expected behaviour.
For reference, GetFileInfo output :
╰─ GetFileInfo /Users/jeromeverdoni/stackoverflow/68888321
directory: "/Users/jeromeverdoni/stackoverflow/68888321"
attributes: avbstclinmedz
created: 08/23/2021 08:01:44
modified: 08/23/2021 08:03:18
I have folders like on my target host:
/home/admin/stream
/home/admin/STB/stream/data_en
/home/admin/STB_1/stream
I search based on stream in /home/admin but need only folders that does not have data_en in it..
What i tried:
Find directories where stream is present with ansbile code
- name: Find all folders with stream subfolder in it
find:
paths: /home/admin/stream
file_type: directory
pattern: 'stream'
use_regex: no
recurse: yes
register: files_to_change
- name: rename folders
command: mv "{{ item['path'] }}" "{{ item['path'] }}.disable"
with_items: "{{ files_to_change.files }}"
when:
- item['path'] is not regex(item['path']/data_en)
I am trying to rename only folder 1 and 3 from the below list, because folder 2 has data_en inside it.
/home/admin/stream
/home/admin/STB/stream
/home/admin/STB_1/stream
I could not figure out a way to skip that..
Fix and test the condition, e.g.
when: item.path is not regex('^(.*)/data_en(.*)$')
You might want to test the code first, e.g.
- debug:
msg: "mv {{ item.path }} {{ item.path }}.disable"
loop: "{{ files_to_change.files }}"
loop_control:
label: "{{ item.path }}"
when: item.path is not regex('^(.*)/data_en(.*)$')
should give
TASK [debug] ********************************************************************
ok: [localhost] => (item=/home/admin/stream) =>
msg: mv /home/admin/stream /home/admin/stream.disable
skipping: [localhost] => (item=/home/admin/STB/stream/data_en)
ok: [localhost] => (item=/home/admin/STB_1/stream) =>
msg: mv /home/admin/STB_1/stream /home/admin/STB_1/stream.disable
Hi can i know how can i keep the latest 3 files based on file name and path. For instance i have
fileAbackup1.war, fileAbackup2.war, fileAbackup3.war, fileAbackup4.war
fileBbackup1.war, fileBbackup2.war, fileBbackup3.war, fileBbackup4.war
So i should keep
fileAbackup1.war, fileAbackup2.war, fileAbackup3.war
fileBbackup1.war, fileBbackup2.war, fileBbackup3.war
I know the part of finding files and deleting files older than x days. I already done the coding part. But i need have some filter to keep 3 backup files of same name of different path
Updated
Below is my ansible code
- name: Find files
find:
paths: "{{ remove_file_path }}"
use_regex: yes
patterns:
- '*.war.*.backup*'
- '{{ rar_file_name }}.rar.*.bakup*'
- '.*\..*\.\d+\.\d{4}-\d{2}-\d{2}#\d{2}:\d{2}:\d{2}~$'
age: 5d
recurse: yes
register: removeFile
- name: Delete files
file:
path: "{{ item.path }}"
state: absent
loop: "{{ removeFile.files }}"
The tasks below
- find:
paths: dir1
patterns: '*.war'
register: result
- set_fact:
my_files: "{{ result.files|
map(attribute='path')|
sort|
list }}"
- debug:
var: my_files[0:3]
give
my_files[0:3]:
- dir1/fileAbackup1.war
- dir1/fileAbackup2.war
- dir1/fileAbackup3.war
If you need the filenames only map the filter basename. For example
- set_fact:
my_files: "{{ result.files|
map(attribute='path')|
map('basename')|
sort|
list }}"
- debug:
var: my_files[0:3]
give
my_files[0:3]:
- fileAbackup1.war
- fileAbackup2.war
- fileAbackup3.war
Q: "Will it have file fileBbackup1.war, fileBbackup2.war, and fileBbackup3.war as well?"
A: Yes. It will. Create a dictionary of the lists. For example
- set_fact:
my_files: "{{ my_files|default({})|
combine({item: result.files|
map(attribute='path')|
map('basename')|
select('search', item)|
sort|
list}) }}"
loop:
- fileAbackup
- fileBbackup
- debug:
msg: "{{ item.value[0:3] }}"
loop: "{{ my_files|dict2items }}"
loop_control:
label: "{{ item.key }}"
give
ok: [localhost] => (item=fileAbackup) =>
msg:
- fileAbackup1.war
- fileAbackup2.war
- fileAbackup3.war
ok: [localhost] => (item=fileBbackup) =>
msg:
- fileBbackup1.war
- fileBbackup2.war
- fileBbackup3.war
This command will provide the information you need.
ls -1 *.war | sort -r | head -n -3
Explanation
ls -1 *.war | sort -r Performs a reverse sort and displays just the filenames
head -n -3 Displays all except the first 3
You can pipe this to the command that deletes the files or you can run the following
rm $(ls -1 | sort -r | head -n -3)