I have an integration where I download one or more ZIP files. Within those ZIP files, there are dozens of GZ files that also need to be uncompressed. Below is an example of the file structure:
metrics.zip
-> 239238923323.gz
-> 839389239232.gz
-> 928392892839.gz
metrics-001.zip
-> 29389238923.gz
-> 39828393822.gz
-> 09320930323.gz
(etc)
I was struggling to write the playbook needed to loop through the ZIP file(s), then all of the GZ files and uncompress them all.
Created an answer from the author's original post:
- hosts: localhost
gather_facts: no
tasks:
# Download the report into a temporary directory on the Ansible Playbook host
- name: Create temporary directory
ansible.builtin.tempfile:
state: directory
suffix: unique_suffix
register: temp_dir
- name: Download File
ansible.builtin.get_url:
url: "https://path.to/file/download.zip"
dest: "{{ temp_dir.path }}/download_file_name.zip"
# Unzip the ZIP files
- name: Extract all ZIP files
ansible.builtin.unarchive:
src: "{{ item }}"
dest: "{{ temp_dir.path }}"
with_fileglob:
- "{{ temp_dir.path }}/*.zip"
# Unzip the GZ files
- name: Extract all of the GZ files
ansible.builtin.command: find "{{ temp_dir.path }}" -name '*.gz' -exec gzip -d {} \;
- name: Merge CSVs into a single file
ansible.builtin.assemble:
src: "{{ temp_dir.path }}"
dest: "{{ temp_dir.path }}/extract.x"
regexp: '\.csv$'
# Use a Windows host to copy the file over to a file share (copying from Linux to a Windows file share requires mounting the FS. Using a Windows host is easier)
- hosts: "{{ windows_host_name_in_inventory }}"
tasks:
- name: Copy Extract file to File Share
win_copy:
src: "{{ hostvars['localhost']['temp_dir'].path }}/extract.x"
dest: "{{ Extract_To }}\\unique_name.csv"
# Remove the Temporary folder
- hosts: localhost
tasks:
- name: Remove Temporary Directory
ansible.builtin.file:
path: "{{ hostvars['localhost']['temp_dir'].path }}"
state: absent
when: hostvars['localhost']['temp_dir'].path is defined
Related
I am using unarchive module in ansible to extract tar.gz files to a folder. After that, i want to copy the extracted folder to another location. Is there a ansible way of doing it? I tried below , it does not work
- name : Extract Files
unarchive:
src: "/home/app/{{ item }}"
dest: "/home/app/"
mode: 0777
copy: no
with_items:
- "{{ sharef }}"
delegate_to: localhost
register: extractedfiles
- name : Copy Files
copy:
src: "{{extractedfiles.results}}"
dest: "{{ location }}"
delegate_to: localhost
How do i extract the filie names from the previously executed task extractedfiles.results does not work. Any help would be appreciated.
I'm trying to find files older than one day in Ansible and after that, create a tar.gz of those files, I've tried archive module, although it is creating only a tar of the last element in the list. Is there any way to create tar.gz including all files?
Below is my script:
- name: Check all files
find: paths=/myfiles
file_type=file
age=1
age_stamp=mtime
register: files
failed_when: files.matched < 10
- name: Remove previous tarFile
file: path=/tmp/test.tar.gz
state=absent
- name: compress all the files in tar.gz
archive: path="{{ item.path }}"
dest=/tmp/test.tar.gz
format=gz
with_items:
"{{ files.files }}"
it is creating only a tar of the last element in the list
It is creating and overwriting /tmp/test.tar.gz for each file in the loop. When you check, you see only the last one.
If you look at the archive module docs, you will see:
path Remote absolute path, glob, or list of paths or globs for the file or files to compress or archive.
So you can provide the list of files as a value for the path parameter:
- name: compress all the files in tar.gz
archive:
path: "{{ files_to_archive }}"
dest: /tmp/test.tar.gz
format: gz
vars:
files_to_archive: "{{ files.files | map(attribute='path') | list }}"
i have created multiple zip files using below method.
- name: 'Create zip archive of {{ date_input }} NMON HTML files'
archive:
path: /tmp/{{ inventory_hostname }}_*.html
dest: /tmp/NMON-{{ inventory_hostname }}.zip
format: zip
when: option == "5"
delegate_to: localhost
I have a task , where should I copy a file from its source , to its destination while renaming it in the destination .
My task looks like this :
- name: Go to the target folder
shell: ls
args:
chdir: "{{pathTest}}/target"
register: resultLS
- debug:
msg: "{{resultLS}}"
- name: copy jar file
copy:
src: "{{resultLS.stdout}}"
dest: "{{pathTest}}"
mode: 0777
But, like this it copies the jar file with its same name , my purpose is how to rename it in the dest (ideally with the copy action)
Ideas ?
rename it to: renamed.jar
Here you are:
- name: Ensure the first matched file from {{ pathTest }}/target is present on the target
copy:
src: "{{ lookup('fileglob', pathTest + '/target/*') | first }}"
dest: "{{ pathTest }}/renamed.jar"
mode: 0777
Remarks:
Don't parse ls output!
Think how you should handle multiple files.
in the example above - copy only the first one
How can I copy more than a single file into remote nodes by Ansible in a task?
I've tried to duplicate the copy module line in my task to define files but it only copies the first file.
You can use the with_fileglob loop for this:
- copy:
src: "{{ item }}"
dest: /etc/fooapp/
owner: root
mode: 600
with_fileglob:
- "/playbooks/files/fooapp/*"
- name: copy multiple items
copy:
src: "{{ item.src }}"
dest: "{{ item.dest }}"
loop:
- src: containerizers
dest: /etc/mesos/containerizers
- src: another_file
dest: /etc/somewhere
- src: dynamic
dest: "{{ var_path }}"
Since Ansible 2.5 the with_* constructs are not recommended, and loop syntax should be used. A simple practical example:
- name: Copy CA files
copy:
src: '{{item}}'
dest: '/etc/pki/ca-trust/source/anchors'
owner: root
group: root
mode: 0644
loop:
- symantec-private.crt
- verisignclass3g2.crt
You can use with_together for this purpose:
- name: Copy multiple files to multiple directories
copy: src={{ item.0 }} dest={{ item.1 }}
with_together:
- [ 'file1', 'file2', 'file3' ]
- [ '/dir1/', '/dir2/', '/dir3/' ]
If you need more than one location, you need more than one task. One copy task can copy only from one location (including multiple files) to another one on the node.
- copy: src=/file1 dest=/destination/file1
- copy: src=/file2 dest=/destination/file2
# copy each file over that matches the given pattern
- copy: src={{ item }} dest=/destination/
with_fileglob:
- /files/*
You can use a find, and then copy those files.
---
- hosts: lnx
tasks:
- find:
paths: /appl/scripts/inq
recurse: true
patterns: "inq.Linux*"
register: file_to_copy
- copy:
src: "{{ item.path }}"
dest: /usr/local/sbin/
owner: root
mode: 0775
loop: "{{ files_to_copy.files }}"
Or you can use with_items:
- copy:
src: "{{ item }}"
dest: /etc/fooapp/
owner: root
mode: 600
with_items:
- dest_dir
- name: find inq.Linux*
find: paths="/appl/scripts/inq" recurse=yes patterns="inq.Linux*"
register: find_files
- name: set fact
set_fact:
all_files:
- "{{ find_files.files | map(attribute='path') | list }}"
when: find_files > 0
- name: copy files
copy:
src: "{{ item }}"
dest: /destination/
with_items: "{{ all_files }}"
when: find_files > 0
copy module is a wrong tool for copying many files and/or directory structure, use synchronize module instead which uses rsync as backend. Mind you, it requires rsync installed on both controller and target host. It's really powerful, check ansible documentation.
Example - copy files from build directory (with subdirectories) of controller to /var/www/html directory on target host:
synchronize:
src: ./my-static-web-page/build/
dest: /var/www/html
rsync_opts:
- "--chmod=D2755,F644" # copy from windows - force permissions
You can loop through variable with list of directories:
- name: Copy files from several directories
copy:
src: "{{ item }}"
dest: "/etc/fooapp/"
owner: root
mode: "0600"
loop: "{{ files }}"
vars:
files:
- "dir1/"
- "dir2/"
Use the following source code for copy multiple files on your client machine.
- name: Copy data to the client machine
hosts: hostname
become_method: sudo
become_user: root
become: true
tasks:
# Copy twice as sometimes files get skipped (mostly only one file skipped from a folder if the folder does not exist)
- name: Copy UFO-Server
copy:
src: "source files path"
dest: "destination file path"
owner: root
group: root
mode: 0644
backup: yes
ignore_errors: true
Note:
If you are passing multiple paths by using variable then
src: "/root/{{ item }}"
If you are passing path by using a variable for different items then
src: "/root/{{ item.source_path }}"
Copy files from multiple directories to multiple directories with Ansible
I found the guenhter answer helpful but needed to change also the remote files' mode. I don't have enough reputation to put this as a comment, which would be a more appropriate place for this. In the example, I copy two files from two directories into /tmp and /tmp/bin, which I create first and modify remote files mode.
- name: cpldupd
hosts: test
remote_user: root
become: true
vars:
- rpth: /tmp
tasks:
- name: Create '{{rpth}}/bin'
file:
path: '{{rpth}}/bin'
state: directory
- name: Transfer
copy: src={{ item.src }} dest={{ item.dest }} mode=0775
with_items:
- { src: '../utils/cpldupd', dest: '{{rpth}}/cpldupd' }
- { src: '../utils/bin/cpldupd', dest: '{{rpth}}/bin/cpldupd' }
Here is a generic solution for copying files:
...
- name: Find files you want to move
ansible.builtin.find:
paths: /path/to/files/
file_type: file
excludes: "*.txt" # Whatever pattern you want to exclude
register: files_output
- name: Copy the files
ansible.builtin.copy:
src: "{{ item.path }}"
dest: /destination/directory/
loop: "{{ files_output.files }}"
...
This is more powerful than using with_fileglob as you can match using regexes. Here is this play in action:
$ ls /path/to/files
demo.yaml test.sh ignore.txt
$ ls /destination/directory
file.h
$ ansible-playbook playbook.yaml
...[some output]...
$ ls /destination/directory
file.h demo.yaml test.sh
As you can see from the above example, ignore.txt was not copied over to the destination directory because of the excludes regex in the playbook. Ignoring files like this is not possible as simply using with_fileglob.
Additionally, you can move files from multiple directories with relative ease:
...
- name: Find files you want to move
ansible.builtin.find:
paths: /path/to/files/
# ... the rest of the task
register: list1
- name: Find more files you want to move
ansible.builtin.find:
paths: /different/path/
# ... the rest of the task
register: list2
- name: Copy the files
ansible.builtin.copy:
src: "{{ item.path }}"
dest: /destination/directory/
loop: "{{ list1.files + list2.files }}"
...
Here is a sample Ansible Script to copy multiple Files on remote Hosts
- name: Copy Multiple Files on remote Hosts
ansible.windows.win_copy:
src: "{{ srcPath }}/{{ item }}" # Remeber to us {{item}}
# as a postfix to source path
dest: "{{ destPath }}"
remote_src: yes # if source path is available on remote Host
with_items:
- abc.txt
- abc.properties
hosts: test
gather_facts: false
become: true
vars:
path: '/home/ansibm/playbooks'
remote_path: '/home/{{ansible_ssh_user}}'
dir: 'yml_files'
tasks:
name: "creating directory for backup file"
file:
path: '{{ remote_path }}/{{ dir }}'
state: directory
owner: '{{ansible_ssh_user}}'
group: '{{ansible_ssh_user}}'
mode: 0700
name: "copying yml files"
copy:
src: '{{item}}'
dest: '{{ remote_path }}/{{ dir }}'
owner: '{{ansible_ssh_user}}'
group: '{{ansible_ssh_user}}'
mode: 0644
loop:
- '{{ path }}/ab.html'
- '{{ path }}/cp.yml'
I am trying to copy the content of dist directory to nginx directory.
- name: copy html file
copy: src=/home/vagrant/dist/ dest=/usr/share/nginx/html/
But when I execute the playbook it throws an error:
TASK [NGINX : copy html file] **************************************************
fatal: [172.16.8.200]: FAILED! => {"changed": false, "failed": true, "msg": "attempted to take checksum of directory:/home/vagrant/dist/"}
How can I copy a directory that has another directory and a file inside?
You could use the synchronize module. The example from the documentation:
# Synchronize two directories on one remote host.
- synchronize:
src: /first/absolute/path
dest: /second/absolute/path
delegate_to: "{{ inventory_hostname }}"
This has the added benefit that it will be more efficient for large/many files.
EDIT: This solution worked when the question was posted. Later Ansible deprecated recursive copying with remote_src
Ansible Copy module by default copies files/dirs from control machine to remote machine. If you want to copy files/dirs in remote machine and if you have Ansible 2.0, set remote_src to yes
- name: copy html file
copy: src=/home/vagrant/dist/ dest=/usr/share/nginx/html/ remote_src=yes directory_mode=yes
To copy a directory's content to another directory you CAN use ansibles copy module:
- name: Copy content of directory 'files'
copy:
src: files/ # note the '/' <-- !!!
dest: /tmp/files/
From the docs about the src parameter:
If (src!) path is a directory, it is copied recursively...
... if path ends with "/", only inside contents of that directory are copied to destination.
... if it does not end with "/", the directory itself with all contents is copied.
Resolved answer:
To copy a directory's content to another directory I use the next:
- name: copy consul_ui files
command: cp -r /home/{{ user }}/dist/{{ item }} /usr/share/nginx/html
with_items:
- "index.html"
- "static/"
It copies both items to the other directory. In the example, one of the items is a directory and the other is not. It works perfectly.
The simplest solution I've found to copy the contents of a folder without copying the folder itself is to use the following:
- name: Move directory contents
command: cp -r /<source_path>/. /<dest_path>/
This resolves #surfer190's follow-up question:
Hmmm what if you want to copy the entire contents? I noticed that * doesn't work – surfer190 Jul 23 '16 at 7:29
* is a shell glob, in that it relies on your shell to enumerate all the files within the folder before running cp, while the . directly instructs cp to get the directory contents (see https://askubuntu.com/questions/86822/how-can-i-copy-the-contents-of-a-folder-to-another-folder-in-a-different-directo)
Ansible remote_src does not support recursive copying.See remote_src description in Ansible copy docs
To recursively copy the contents of a folder and to make sure the task stays idempotent I usually do it this way:
- name: get file names to copy
command: "find /home/vagrant/dist -type f"
register: files_to_copy
- name: copy files
copy:
src: "{{ item }}"
dest: "/usr/share/nginx/html"
owner: nginx
group: nginx
remote_src: True
mode: 0644
with_items:
- "{{ files_to_copy.stdout_lines }}"
Downside is that the find command still shows up as 'changed'
the ansible doc is quite clear https://docs.ansible.com/ansible/latest/collections/ansible/builtin/copy_module.html for parameter src it says the following:
Local path to a file to copy to the remote server.
This can be absolute or relative.
If path is a directory, it is copied recursively. In this case, if path ends with "/",
only inside contents of that directory are copied to destination. Otherwise, if it
does not end with "/", the directory itself with all contents is copied. This behavior
is similar to the rsync command line tool.
So what you need is skip the / at the end of your src path.
- name: copy html file
copy: src=/home/vagrant/dist dest=/usr/share/nginx/html/
I found a workaround for recursive copying from remote to remote :
- name: List files in /usr/share/easy-rsa
find:
path: /usr/share/easy-rsa
recurse: yes
file_type: any
register: find_result
- name: Create the directories
file:
path: "{{ item.path | regex_replace('/usr/share/easy-rsa','/etc/easy-rsa') }}"
state: directory
mode: "{{ item.mode }}"
with_items:
- "{{ find_result.files }}"
when:
- item.isdir
- name: Copy the files
copy:
src: "{{ item.path }}"
dest: "{{ item.path | regex_replace('/usr/share/easy-rsa','/etc/easy-rsa') }}"
remote_src: yes
mode: "{{ item.mode }}"
with_items:
- "{{ find_result.files }}"
when:
- item.isdir == False
I got involved whole a day, too! and finally found the solution in shell command instead of copy: or command: as below:
- hosts: remote-server-name
gather_facts: no
vars:
src_path: "/path/to/source/"
des_path: "/path/to/dest/"
tasks:
- name: Ansible copy files remote to remote
shell: 'cp -r {{ src_path }}/. {{ des_path }}'
strictly notice to:
1. src_path and des_path end by / symbol
2. in shell command src_path ends by . which shows all content of directory
3. I used my remote-server-name both in hosts: and execute shell
section of jenkins, instead of remote_src: specifier in playbook.
I guess it is a good advice to run below command in Execute Shell section in jenkins:
ansible-playbook copy-payment.yml -i remote-server-name
Below worked for me,
-name: Upload html app directory to Deployment host
copy: src=/var/lib/jenkins/workspace/Demoapp/html dest=/var/www/ directory_mode=yes
This I found an ideal solution for copying file from Ansible server to remote.
copying yaml file
- hosts: localhost
user: {{ user }}
connection: ssh
become: yes
gather_facts: no
tasks:
- name: Creation of directory on remote server
file:
path: /var/lib/jenkins/.aws
state: directory
mode: 0755
register: result
- debug:
var: result
- name: get file names to copy
command: "find conf/.aws -type f"
register: files_to_copy
- name: copy files
copy:
src: "{{ item }}"
dest: "/var/lib/jenkins/.aws"
owner: {{ user }}
group: {{ group }}
remote_src: True
mode: 0644
with_items:
- "{{ files_to_copy.stdout_lines }}"
How to copy directory and sub dirs's and files from ansible server to remote host
- name: copy nmonchart39 directory to {{ inventory_hostname }}
copy:
src: /home/ansib.usr.srv/automation/monitoring/nmonchart39
dest: /var/nmon/data
Where:
copy entire directory: src: /automation/monitoring/nmonchart39
copy directory contents src: nmonchart39/