I have a playbook that copy files from different local folders to different remote paths and it works, configured to overwrite the files with backup (with copy module).
How to get the list of files that will be copied and overwrite before doing it?
I'd like to prevent to copy wrong files.
Example of the copy:
name: Copy all
copy:
src: scenarios/all_machines/
dest: /path/to/dest
backup: yes
force: yes
owner: root
group: root
mode: 0644
when: ansible_facts.services['crowdsec.service'] is defined
UPDATE
With this local folder:
scenarios/all_machines/
├── wg-sshd-more-logs.yaml
└── wg-x00.yaml
and this remote dir:
test_ansible/
└── wg-x00.yaml
where the wg-x00.yaml file has the same content.
With this playbook (where I changed the '>' char with '<'):
- name: TEST
synchronize:
src: scenarios/all_machines/
dest: test_ansible/
rsync_opts:
- --dry-run
register: result
- debug:
var: result.stdout_lines
- name: TEST 2
set_fact:
overwrite: "{{ file_stat|
selectattr('stat', 'match', '<f.*')|
selectattr('stat', 'ne', '<f+++++++++')|
map(attribute='file')|
list }}"
vars:
file_stat: "{{ result.stdout_lines|
map('regex_replace', regex, replace)|
map('from_yaml')|
list }}"
regex: '^(.*?) (.*)$'
replace: '{file: "\2", stat: "\1"}'
- debug:
var: overwrite
I have this output:
"result.stdout_lines": [
".d..tp..... ./",
"<f+++++++++ wg-sshd-more-logs.yaml",
"<f..tp..... wg-x00.yaml"
]
"overwrite": [
"wg-x00.yaml"
]
So:
I changed the the '>' char with '<' because my output is different from your
The overwrite variable tells me that the file will be overwritten but they are equal...
Given the directories below for testing
shell> tree scenarios/all_machines/
scenarios/all_machines/
├── file1
├── file2
└── file3
0 directories, 3 files
shell> tree dest/
dest/
└── file1
0 directories, 1 file
the file file1 at dest/ will be replaced. The question is how to find it.
Use synchronize and set 'rsync_opts: --dry-run'
- synchronize:
src: scenarios/all_machines/
dest: dest
rsync_opts:
- --dry-run
register: result
The result will display the status of the files
result.stdout_lines:
- .d..t...... ./
- '>f..t...... file1'
- '>f+++++++++ file2'
- '>f+++++++++ file3'
The task below
- set_fact:
overwrite: "{{ file_stat|
selectattr('stat', 'match', '>f.*')|
selectattr('stat', 'ne', '>f+++++++++')|
map(attribute='file')|
list }}"
vars:
file_stat: "{{ result.stdout_lines|
map('regex_replace', regex, replace)|
map('from_yaml')|
list }}"
regex: '^(.*?) (.*)$'
replace: '{file: "\2", stat: "\1"}'
will create the list
file_stat:
- {file: ./, stat: .d..t......}
- {file: file1, stat: '>f..t......'}
- {file: file2, stat: '>f+++++++++'}
- {file: file3, stat: '>f+++++++++'}
and will select names of the existing files only
overwrite:
- file1
See Understanding the output of rsync --itemize-changes.
Notes
Use debug to display the variables. For example, use to_yaml/to_nice_yaml filters
- debug:
var: file_stat|to_yaml
- debug:
var: overwrite|to_nice_yaml
Use yaml callback.
See DEFAULT_STDOUT_CALLBACK on how to configure the callback.
Related
I need to find folder names in /home/test directory. Once I have the list of folder names I need to loop over a task. I am able to retrieve the complete path, but not able to retrieve the folder name. is my approach wrong, or is there any easy way to retrieve folder names?
--
- hosts: localhost
gather_facts: no
vars:
Files: []
namespaces: []
tasks:
- name: Recursively find folders under /home/test
ansible.builtin.find:
paths: /home/test
file_type: directory
recurse: no
register: output
- name: Adding Files to the LIST
set_fact:
Files: "{{ Files + [item.path] }}"
with_items: "{{ output.files }}"
- name: Remove the path to only and retrieve folder name only
set_fact:
namespaces: "{{ namespaces + item | split('/') | last}}"
with_items: "{{ Files }}"
Error:
fatal: [localhost]: FAILED! => {"msg": "Unexpected templating type error occurred on ({{ namespaces + item | split('/') | last}}): can only concatenate list (not \"str\") to list"}
Put the below declaration into the vars
namespaces: "{{ output.files|map(attribute='path')|map('basename')|list }}"
Example of a complete playbook for testing
Given the tree
shell> tree /tmp/test/
/tmp/test/
├── dir_A
├── dir_B
├── file_1
└── file_2
the playbook
- hosts: localhost
vars:
namespaces: "{{ output.files|map(attribute='path')|map('basename')|list }}"
tasks:
- find:
paths: /tmp/test
file_type: directory
register: output
- debug:
var: namespaces
gives (abridged)
namespaces:
- dir_B
- dir_A
Your expression item | split('/') | last currently returns a string that you try to concatenate to the namespaces list. However, this is not allowed and thus you need to cast it to a list as well by simply put it into braces []:
- name: Remove the path to only and retrieve folder name only
set_fact:
namespaces: "{{namespaces + [item | split('/') | last]}}"
with_items: "{{ Files }}"
I want to list all files of a directory with the owner of the file her name start with 'a',
i have a ansible playbook like this
- name: Recursively find
find:
paths: "/tmp"
age: "6d"
age_stamp: ctime
file_type: any
patterns: "*"
recurse: no
register: output
Q: "List all files of a directory with the owner of the file her name starts with 'a'."
A: For example, given the tree
shell> tree -u tmp
tmp
├── [admin ] file1
├── [admin ] file2
└── [user1 ] file3
Find all files in the first task and select the owner in the second task
- find:
paths: tmp
register: output
- debug:
msg: "{{ item.path }}"
loop: "{{ output.files|selectattr('pw_name', 'match', '^a.*$')|list }}"
give
msg: tmp/file2
msg: tmp/file1
I'm trying to pull in a file with the same name on multiple servers and I would like to just concatenate the results, but I don't think fetch module will allow me to do this. Can someone advise on another module that I could use for this task?
Current non-working code:
- hosts: '{{ target }}'
gather_facts: false
tasks:
- name: Pull in file.log contents from servers, concatenating results
fetch:
src: '/tmp/file.log'
dest: /tmp/fetched
flat: yes
fail_on_missing: no
For example, given the files
shell> ssh admin#test_11 cat /tmp/file.log
test_11
shell> ssh admin#test_12 cat /tmp/file.log
test_12
shell> ssh admin#test_13 cat /tmp/file.log
test_13
Throttle the task and time-stamp the fetched files, e.g.
- hosts: test_11,test_12,test_13
tasks:
- fetch:
src: /tmp/file.log
dest: /tmp/fetched/file-{{ time_stamp }}.log
flat: true
fail_on_missing: false
throttle: 1
vars:
time_stamp: "{{ lookup('pipe', 'date +%Y-%m-%d_%H-%M-%S') }}"
gives
shell> tree /tmp/fetched/
/tmp/fetched/
├── file-2021-03-22_21-16-54.log
├── file-2021-03-22_21-16-58.log
└── file-2021-03-22_21-17-02.log
Then assemble the content of the files, e.g.
- assemble:
src: /tmp/fetched
regexp: '^file-.*log$'
dest: /tmp/fetched/assemble-{{ time_stamp }}.log
vars:
time_stamp: "{{ lookup('pipe', 'date +%Y-%m-%d_%H-%M-%S') }}"
delegate_to: localhost
run_once: true
gives
shell> cat /tmp/fetched/assemble-2021-03-22_21-17-07.log
test_11
test_12
test_13
If you want to speed up the transfer from many hosts (e.g. ~100) increase the number of the parallel tasks (e.g. throttle: 10). Put the name of the host into the name of the file. Otherwise, the task would overwrite the files with the same timestamp, e.g.
- fetch:
src: /tmp/file.log
dest: /tmp/fetched/file-{{ inventory_hostname }}-{{ time_stamp }}.log
flat: true
fail_on_missing: false
throttle: 3
vars:
time_stamp: "{{ lookup('pipe', 'date +%Y-%m-%d_%H-%M-%S') }}"
What I am trying to achieve here is as below
Unarchive the code from tar.gz - working
Find file names in unarchived directory - working
Find file names in code directory based on file names fetched in step 2 - failed
Copy files from source:(Step 2) destination:(Step3) - working if I use the hardcoded file names in the pattern section of step 3
Below mentioned is the Ansible role I have used:
- name: Unarchive config files to server
unarchive:
src: "{{ config_dir }}/config.tar.gz"
dest: /tmp
list_files: yes
register: tar_path
- name: Find file names in unarchived config files
find:
paths: "{{ tar_path.dest }}"
file_type: file
recurse: yes
register: tmp_file_path
- name: Find file names in code base
find:
paths: /opt
file_type: file
recurse: yes
patterns:
#Search for file names with the values in tmp_file_path
register: code_file_path
- set_fact:
code_files: "{{ code_files|default([]) +
[{'path': item, 'name': item|basename}] }}"
loop: "{{ code_file_path.files|map(attribute='path')|list }}"
- name: copy files
command: cp "{{ item.0 }}" "{{ item.1.path }}"
with_together:
- "{{ tmp_file_path.files|map(attribute='path')|list|sort }}"
- "{{ code_files|sort(attribute='name') }}"
Here I need to use find to locate files in /opt directory based on pattern(filename) exactly which I have unarchived in /tmp
And finally, replace files from /tmp to /opt based on the file names and paths(This I am able to do). The directory structure is as follows:
shell> tree tmp
tmp
├── file1
├── file2
└── file3
shell> tree opt
opt
├── bar
│ └── file2
├── baz
│ └── file3
└── foo
└── file1
Here if I use the below code wherein I manually mention the file names, then it works. However, I don't want to do that
- name: Find file names in code base
find:
paths: /opt
file_type: file
recurse: yes
patterns:
- file1
- file2
- file3
register: code_file_path
I need a solution to replace the hardcoding for patterns: file1, file2 and file3 and use some variable to do that. The file names in /tmp and /opt where I need to replace is exactly the same
If I understood correctly, here is a possible way to handle what you're trying to do. In the below example, I took away the unarchive job as it's not on the critical path.
Playbook walkthrough
I created two sample directories. The first two tasks are only there to further show you this test structure:
an archive directory containing 4 files in random directories. One of them is not present in the target
a code directory containing several files random directories. 3 files have the same basenames as other files found in archive.
The first find task is identical to yours and registers a result with details of all files in the archive dir.
For the second find task in the code directory, the key point is to pass as patterns parameter the list of basenames from the first search which you can get with the expression:
{{ search_archive.files | map(attribute='path') | map('basename') | list }}
We can detail this one as: get files list from our archive find result, extract only the path attribute, apply the basename filter on each list element and return a list.
For the last task, I used the copy module. My example runs on localhost but since yours will probably run on a remote target, the remote_src has to be set (or files will be fetched from the controller).
The loop is done on the result of the previous task so we only get the matching files in code directory as dest. To select the src, we look for corresponding files in the archive folder with the following expression:
{{ search_archive.files | map(attribute='path') | select('match', '^.*/' + item | basename + '$') | first }}
The select filter select is applying the match test to each path in the list selecting only the elements ending with the current code path basename. The first filter get only the first (and only in your case) matching element. loop_control.label is used to get a better output of task result.
Demo playbook
The first two task are only for debugging/demo purpose.
---
- name: Update files from package in code wherever they are
hosts: localhost
gather_facts: false
tasks:
- name: Capture sample data structure
command: tree archive code
register: structure
changed_when: false
- name: Show sample data structure
debug:
msg: "{{ structure.stdout_lines}}"
- name: Find files in archive
find:
paths: archive
file_type: file
recurse: yes
register: search_archive
- name: Find files in code matching names in archive
find:
paths: code
file_type: file
recurse: yes
patterns: >-
{{
search_archive.files |
map(attribute='path') |
map('basename') |
list
}}
register: search_code
- name: Copy files from archive to code
vars:
archive_source: >-
{{
search_archive.files |
map(attribute='path') |
select('match', '^.*/' + item | basename + '$') |
first
}}
copy:
remote_src: yes
src: "{{ archive_source }}"
dest: "{{ item }}"
loop: "{{ search_code.files | map(attribute='path') | list }}"
loop_control:
label:
Source: "{{ archive_source }}"
Destination: "{{ item }}"
Result
PLAY [Update files from package in code wherever they are] *****************************************************************************************************************************************************************************
TASK [Capture sample data structure] ***************************************************************************************************************************************************************************************************
ok: [localhost]
TASK [Show sample data structure] ******************************************************************************************************************************************************************************************************
ok: [localhost] => {
"msg": [
"archive",
"├── a_dir",
"│ └── file2",
"├── file1.txt",
"├── file3",
"└── other_dir",
" └── bla",
" └── fileX",
"code",
"├── dir1",
"│ └── file1.txt",
"├── dir2",
"│ ├── file2",
"│ ├── pipo",
"│ └── toto",
"└── dir3",
" └── subdir",
" └── file3",
"",
"7 directories, 9 files"
]
}
TASK [Find files in archive] ***********************************************************************************************************************************************************************************************************
ok: [localhost]
TASK [Find files in code matching names in archive] ************************************************************************************************************************************************************************************
ok: [localhost]
TASK [Copy files from archive to code] *************************************************************************************************************************************************************************************************
changed: [localhost] => (item={'Source': 'archive/file1.txt', 'Destination': 'code/dir1/file1.txt'})
changed: [localhost] => (item={'Source': 'archive/a_dir/file2', 'Destination': 'code/dir2/file2'})
changed: [localhost] => (item={'Source': 'archive/file3', 'Destination': 'code/dir3/subdir/file3'})
PLAY RECAP *****************************************************************************************************************************************************************************************************************************
localhost : ok=5 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
I can replace a string in the file if the file exists in a directory.
Similarly, I want to replace the string in another 25 files (test2.conf .. test25.conf) if the files exist in the same directory. Using with_items I can able to replace a string in all files but unable to use the register to check the file exists in the loop.
Below playbook is a success for one file, how can I achieve the same for 25 files?
- hosts: webserver
vars:
strings:
- user1
- user2
tasks:
- name: Check /home/user/test1.conf exists
stat:
path: /home/user/test1.conf
register: stat_result
- name: Replace string in test1.conf
replace:
dest: /home/user/test1.conf
backup: yes
regexp: '^Hello'
replace: "Hello {{ strings | join(' ')}}"
when: stat_result.stat.exists == True
Use find module instead of stat and iterate over the results:
- hosts: webserver
vars:
strings:
- user1
- user2
tasks:
- name: Get a list of test*.conf in /home/user
find:
paths: /home/user
patterns: test*.conf
register: my_find
- name: Replace strings in files found
replace:
dest: "{{ item.path }}"
backup: yes
regexp: '^Hello'
replace: "Hello {{ strings | join(' ')}}"
with_items: "{{ my_find.files }}"
Your replace task is not idempotent, it will add strings infinitely on subsequent runs.