I am working on small playbook that will search for files in specified directories and then delete them if they met certain conditions. I have following playbook so far
- hosts:
- localhost
gather_facts: false
tasks:
- name: find logs
find:
paths: "{{ item.0 }}"
file_type: file
patterns: "{{ item.1 }}"
register: find_logs
with_nested:
- ["/var/log/apache2", "/var/log/nginx"]
- ["access.log", "error.log"]
- debug:
var: item
loop:
- "{{ find_logs }}"
So this will obviously look into /var/log/apache2 and /var/log/nginx directories and search for access.log and error.log files. Now, following ansible's documentation I want to access files return values and their paths. The issue I'm having right now is with nested loop and registered find_logs variable which holds list of dictionaries in results key. If I do find_logs.results then I will get a list of dictionaries and each of these dictionaries will have another list of files kept in files section. How can I 'flatten' this list even more to be able to retrieve files.path for every element produced by nested loop? To be honest I also tried find_logs | json_query('results[*].files[*]') but that gives me another list and I can't seem to iterate over it to get what I want (which is path of file). Any idea on how to make this work?
I completely misunderstood documentation for find module, Nested loop can be excluded in this case and replaced with following syntax
- hosts:
- localhost
gather_facts: false
tasks:
- name: find logs
find:
paths:
- /var/log/apache2
- /var/log/nginx
file_type: file
patterns:
- "access.log"
- "error.log"
- "other_vhosts_access.log"
register: find_logs
- name: check what was registered
debug:
msg: "my path -> {{ item.path }}"
with_items:
- "{{ find_logs.files }}"
Related
Hoping with some help on the following. I'm trying to make reference to an xml file within the work_dir location:
---
- include_vars:
file: var_file.yml
- name: Locate audit results file
find:
paths: "{{ item }}"
recurse: no
patterns: '*.xml'
with_items: "{{ work_dir }}"
register: audit_file
- name: Copy audit results file to local destination
fetch:
src: "{{ item }}/{{ audit_file }}"
dest: /home/bob/audit_results/
flat: yes
validate_checksum: no
with_items: "{{ work_dir }}"
var_file.yml:
---
work_dir:
- /var/tmp/audit
However the above code keeps erroring with:
"msg": "unable to calculate the checksum of the remote file"}
You find files on a loop of paths and register the result. Therefore your registered var audit_file contains a results attribute which is a list.
Each element of the results list contains a files attribute which is again a list. Each files element contains a result of your find run with all the info about the found file. In those info, there is a path attribute pointing to the exact path of the file on the remote server.
Before going further, I strongly suggest your read the above documentation and issue a debug of your var to understand its structure and content
- debug:
var: audit_file
If you want to fetch each found file, you need to make a loop on each results with a sub-loop on each files element. This can be done with a subelements lookup although we will only use here the sub element (i.e. item.1) of the loop.
This is how you can fix your second task:
- name: Copy audit results file to local destination
fetch:
src: "{{ item.1.path }}"
dest: /tmp/test/
flat: yes
validate_checksum: no
with_subelements:
- "{{ audit_file.results }}"
- files
Note: this is not the only solution, I went to the easiest and most obvious.
How to register a variable when using loop with stat module?
I am working on a project where I wish to run comparisons against the known value of a collection of files (checksum), which I will then take action if a change is detected (EG: notify someone, have not written this part yet).
If this were purely a CLI matter, I would have this sorted with some easy SH scripting.
That said, I have Ansible (2.7.5) available within my ENV and am keen to use it!
In reading the vendor documents, using the stat module felt the "Ansible way" to go on this one.
Currently just *NIX servers (Linux, Solaris, and possibly AIX) are in scope, but eventually this might also apply to Windows, where I expect I would use win_stat instead with suitable parameters.
At present I plan to dump the results of the scan to a file (EG: CSV), which I would then iterate / match against, for the purposes of a comparison (to detect if a file has been somehow changed).
This is another part I have not written yet (the read a file and compare portions), but expect to hit those once I get this present matter sorted.
My current challenge, is that I can get "one-off" stat checks to work fine.
However, I expect to be targeting a whole directory worth of files, and thus want to presumably:
"discover" the contents of the target directory, and retain this in memory
iterate (loop) through the list in memory
performing a stat check upon each file
retaining the checksum of each file
building some sort of dict or list?
write the collective results (or one line at a time) out to a log file of sorts (CSV.log: file_path,file_checksum)
I would welcome your feedback on what I might be missing (aside from some hair at this point)?
I have tried a few different approaches to looping within the playbook (loop, with_items, etc.), however the challenge remains the same.
The stat loop runs fine, but the trailing register statement fails to commit the output to memory (resulting in a variety of "undefined variable" errors).
Am I somehow missing something in my loop definition?
Looking at the vendor docs on "Using register with a loop", it would appear I am doing this correctly (in my view anyway).
Simple "target files" I am checking against within a directory.
/tmp/app/targets/file1.txt
Some text.
/tmp/app/targets/file2.cfg
cluster=0
cluster_id=app_pool_00
/tmp/app/targets/file3.sh
#!/bin/sh
printf "Hello world\n"
exit 0
My prototyping playbook as it exists currently.
---
- name: check file integrity
hosts: localhost
become: no
vars:
TARGET: /tmp/app/targets
LOG: /tmp/app/archive/scan_results.log
tasks:
- name: discover target files
find:
paths: "{{ TARGET }}"
recurse: yes
file_type: file
register: TARGET_FILES
- name: scan target
stat:
path: "{{ item.path }}"
get_checksum: yes
loop: "{{ TARGET_FILES.files }}"
register: TARGET_RESULTS
- name: DEBUG
debug:
var: "{{ TARGET_RESULTS }}"
- name: write findings to log
copy:
content: "{{ TARGET_RESULTS.stat.path }},{{ TARGET_RESULTS.stat.checksum }}"
dest: "{{ LOG }}"
...
My "one-off" playbook that worked.
---
- name: check file integrity
hosts: localhost
become: no
vars:
TARGET: /tmp/app/targets/file1.txt
LOG: /tmp/app/archive/scan_results.log
tasks:
- name: scan target
stat:
path: '{{ TARGET }}'
checksum_algorithm: sha1
follow: no
get_attributes: yes
get_checksum: yes
get_md5: no
get_mime: yes
register: result
- name: write findings to log
copy:
content: "{{ result.stat.path }},{{ result.stat.checksum }}"
dest: "{{ LOG }}"
...
The output was not exciting, but useful.
Would expect to build this up with multi-line output (one line per file stat checked) if I could figure out how to loop / register loop output correctly.
/tmp/app/archive/scan_results.log
/tmp/app/targets/file1.txt,8d06cea05d408d70c59b1dbc5df3bda374d869a4
You can use the set_fact module to register a variable like you want.
I don't use it in my test for you, it maybe useless in your case :
---
- name: check file integrity
hosts: localhost
vars:
TARGET: /tmp/app/targets
LOG: /tmp/app/archive/scan_results.log
tasks:
- name: 'discover target files'
find:
paths: "{{ TARGET }}"
recurse: yes
file_type: file
register: TARGET_FILES
- debug:
var: TARGET_FILES
- name: 'scan target'
stat:
path: "{{ item.path }}"
get_checksum: yes
loop: "{{ TARGET_FILES.files }}"
register: TARGET_RESULTS
- debug:
var: TARGET_RESULTS
- name: 'write findings to log'
lineinfile:
line: "{{ item.stat.path }},{{ item.stat.checksum }}"
path: "{{ LOG }}"
create: yes
loop: '{{ TARGET_RESULTS.results }}'
result:
# cat /tmp/app/archive/scan_results.log
/tmp/app/targets/file3.sh,bb4b0ffe4b5d26551785b250c38592b6f482cab4
/tmp/app/targets/file1.txt,8d06cea05d408d70c59b1dbc5df3bda374d869a4
/tmp/app/targets/file2.cfg,fb23292e06f91a0e0345f819fdee34fac8a53e59
Best Regards
Using an Ansible role. I would like to loop over a list of file paths, but I get an error:
template error while templating string: unexpected '/'.
String: {{/home/xyz/download.log}}
This is the main.yml for the "list_log_files" role:
- name: "find logs"
find:
paths: /
patterns: 'download.log'
recurse: yes
register: find_logs
- name: "list log files"
debug: var="{{ item.path }}"
with_items: "{{ find_logs.files }}"
The find returns an array "files", each is a dictionary. The dictionary contains a path entry, which is what I am interested in.
I have faced same issue and above issue is same where I want list of path of each file to insert line. I use Jinja2 filter:
- name: fetch files
find: paths=/var/tmp/ patterns='*.log'
register: find_logs
- name: insert line
lineinfile: dest={{ item }} line='my line' insertafter=EOF
with_items: "{{ find_logs.files | map(attribute='path') | list }}"
{{ find_logs.files | map(attribute='path') | list }}
Helpful Link
map()
Applies a filter on a sequence of objects or looks up an attribute.
This is useful when dealing with lists of objects but you are really
only interested in a certain value of it.
The correct syntax for var argument of debug module (with the value for your use case) is:
In Ansible notation:
debug: var=item.path
In YAML notation:
debug:
var: item.path
Ansible modules' usage is fairy well documented and examples cover most users' needs. This is also true for the debug module, so refer to the examples to check the basic syntax.
I'm trying to write a playbook that will rsync the folders from source to target after a database refresh. Our Peoplesoft HR application also requires a filesystem refresh along with database. I'm new to ansible and not an expert with python. I've written this but my playbook fails if any of the with_items doesn't exist. I'd like to use this playbook for all apps and the folders may differ between apps. How can I skip the folders that doesn't exist in source. I'm passing {{ target }} at command line.
---
- hosts: '<hostname>'
remote_user: <user>
tasks:
- shell: ls -l /opt/custhome/prod/
register: folders
- name: "Copy PROD filesystem to target"
synchronize:
src: "/opt/custhome/prod/{{ item }}"
dest: "/opt/custhome/dev/"
delete: yes
when: "{{ folders == item }}"
with_items:
- 'src/cbl/'
- 'sqr/'
- 'bin/'
- 'NVISION/'
In this case, NVISION doesn't exist in HR app but it does in FIN app. But the playbook is failing coz that folder doesn't exist in source.
You can use find module to find and store paths to source folders and then to iterate over results. Example playbook:
- hosts: '<hostname>'
remote_user: <user>
tasks:
- name: find all directories
find:
file_type: directory
paths: /opt/custhome/prod/
patterns:
- "src"
- "sqr"
- "bin"
register: folders
#debug to understand contents of {{ folders }} variable
# - debug: msg="{{ folders }}"
- name: "Copy PROD filesystem to target"
synchronize:
src: "{{ item.path }}"
dest: "/opt/custhome/dev/"
delete: yes
with_items: "{{ folders.files }}"
You may want to use recurse to descend into subdirectories and use_regex to use the power of python regex instead of shell globbing.
Im creating a deployment playbook for our web services. Each web service is in its own directory such as:
/webapps/service-one/
/webapps/service-two/
/webapps/service-three/
I want to check to see if the service directory exists, and if so, I want to run a shell script that stops the service gracefully. Currently, I am able to complete this step by using ignore_errors: yes.
- name: Stop services
with_items: services_to_stop
shell: "/webapps/scripts/stopService.sh {{item}}"
ignore_errors: yes
While this works, the output is very messy if one of the directories doesnt exist or a service is being deployed for the first time. I effectively want to something like one of these:
This:
- name: Stop services
with_items: services_to_stop
shell: "/webapps/scripts/stopService.sh {{item}}"
when: shell: [ -d /webapps/{{item}} ]
or this:
- name: Stop services
with_items: services_to_stop
shell: "/webapps/scripts/stopService.sh {{item}}"
stat:
path: /webapps/{{item}}
register: path
when: path.stat.exists == True
I'd collect facts first and then do only necessary things.
- name: Check existing services
stat:
path: "/tmp/{{ item }}"
with_items: "{{ services_to_stop }}"
register: services_stat
- name: Stop existing services
with_items: "{{ services_stat.results | selectattr('stat.exists') | map(attribute='item') | list }}"
shell: "/webapps/scripts/stopService.sh {{ item }}"
Also note, that bare variables in with_items don't work since Ansible 2.2, so you should template them.
This will let you get a list of existing directory names into the list variable dir_names (use recurse: no to read only the first level under webapps):
---
- hosts: localhost
connection: local
vars:
dir_names: []
tasks:
- find:
paths: "/webapps"
file_type: directory
recurse: no
register: tmp_dirs
- set_fact: dir_names="{{ dir_names+ [item['path']] }}"
no_log: True
with_items:
- "{{ tmp_dirs['files'] }}"
- debug: var=dir_names
You can then use dir_names in your "Stop services" task via a with_items. It looks like you're intending to use only the name of the directory under "webapps" so you probably want to use the | basename jinja2 filter to get that, so something like this:
- name: Stop services
with_items: "{{ dir_names }}"
shell: "/webapps/scripts/stopService.sh {{item | basename }}"