I've just started to use Ansible to automate binary deployments.
When downloading the zip files and trying to unzip it by passing the downloaded zip files as variable to be unzipped/unarchived but an error is always thrown.
Snippet of the YML below:
- name: Download binaries
get_url:
url={{ download_server }}
url_username={{ username }}
url_password={{ passwd }}
dest={{ base_dir }}
register: bin_files
- set_fact:
my_unzipped_file: "{{ bin_files[0].stdout }}"
- name: UNZIPPING the files
unarchive: src={{ base_dir }}/{{ item }} dest={{ base_dir }} copy=no
with_items: my_unzipped_file
If it wasn't a user/pass protected URL you could erase the 'get_url' module and place the URL in the src: of Unarchive module.
Check the examples:
http://docs.ansible.com/ansible/latest/modules/unarchive_module.html
another way is to download all your files into a directory {{ bin_dir }} for example and use within the unarchive module 'with_fileglob' to unzip all .zip/.tar.gz and such
Example:
- name: UNZIPPING the files
unarchive:
src: "{{ item }}"
dest: "{{ base_dir }}/"
copy: no
with_fileglob:
- "{{ base_dir }}/*.zip"
- "{{ base_dir }}/*.tar.gz"
another tip for you IMHO you should drop the '=' code style in modules and move to ':' as you can see above, it's more human-readable
You corrected SNIPPET:
- name: Download binaries
get_url:
url: {{ download_server }}
url_username: {{ username }}
url_passwor: {{ passwd }}
dest: {{ base_dir }}
register: bin_files
- name: UNZIPPING the files
unarchive:
src: {{ base_dir }}/{{ item }}
dest: {{ base_dir }}
copy: no
with_items:
- "{{ bin_files.stdout }}"
Related
I have a list of items in my playbook.
x_ssl_certs:
- 'host1.domain.tld'
- 'host2.domain.tld'
- 'host3.domain.tld'
- 'anotherhost.domain.tld'
Each of it has it's own SSL files, like host1.domain.tld.key host1.domain.tld.crt host1.domain.tld.fullchain ... and so on. Now I using this playbook to send out this files:
- name: Copy the {{ item }} key files
copy:
src: "{{ item }}/{{ item }}.key"
dest: '/etc/ssl/{{ item }}.key'
owner: 'root'
group: 'root'
mode: '0644'
loop: "{{ x_ssl_certs }}"
- name: Copy the {{ item }} crt files
copy:
src: "{{ item }}/{{ item }}.crt"
dest: '/etc/ssl/{{ item }}.crt'
owner: 'root'
group: 'root'
mode: '0644'
loop: "{{ x_ssl_certs }}"
- name: Copy the {{ item }} fullchain files
copy:
src: "{{ item }}/{{ item }}.fullchain"
dest: '/etc/ssl/{{ item }}.fullchain'
owner: 'root'
group: 'root'
mode: '0644'
loop: "{{ x_ssl_certs }}"
and so on, 1 task for every file. I would like to integrate this into one or two task, so it should lookup the x_ssl_certs list and send out each file which belongs to them. This files are the same naming convention for each item in the list.
This should be a loop in the loop, nested loop, or something like that, but based on the documentation it is not exactly clear for me how to make that.
Use with_nested. For example,
- debug:
msg: "Copy {{ item.0 }}.{{ item.1 }}"
with_nested:
- "{{ x_ssl_certs }}"
- [key, crt, fullchain]
gives
msg: Copy host1.domain.tld.key
msg: Copy host1.domain.tld.crt
msg: Copy host1.domain.tld.fullchain
msg: Copy host2.domain.tld.key
msg: Copy host2.domain.tld.crt
msg: Copy host2.domain.tld.fullchain
msg: Copy host3.domain.tld.key
msg: Copy host3.domain.tld.crt
msg: Copy host3.domain.tld.fullchain
msg: Copy anotherhost.domain.tld.key
msg: Copy anotherhost.domain.tld.crt
msg: Copy anotherhost.domain.tld.fullchain
I faced problems when I don't know how to pass many variables in with_items:
I have these vars:
- vars:
paths:
- /tmp/tecom/python3/
- /tmp/tecom/pip/
- /tmp/tecom/psycopg2/
- /tmp/tecom/docker/
- /tmp/tecom/postgresql/
files:
- python3.tar
- pip.tar
- psycopg2.tar
- docker.tar
- postgresql.tar
And I have the task that should extract the archives:
- name: unarchive "{{ item }}"
unarchive:
src: "{{ item }}"
dest: # should take the paths above. Every path should match it's own `.tar` file.
with_items: "{{ files }}"
Every path should match its own .tar file.
For example,
- debug:
msg: "unarchive {{ item }} to {{ _path }}"
loop: "{{ files }}"
vars:
_path: "{{ path }}/{{ item|splitext|first }}/"
path: /tmp/tecom
files:
- python3.tar
- pip.tar
- psycopg2.tar
- docker.tar
- postgresql.tar
gives (abridged)
msg: unarchive python3.tar to /tmp/tecom/python3/
msg: unarchive pip.tar to /tmp/tecom/pip/
msg: unarchive psycopg2.tar to /tmp/tecom/psycopg2/
msg: unarchive docker.tar to /tmp/tecom/docker/
msg: unarchive postgresql.tar to /tmp/tecom/postgresql/
Q: "How do I unarchive those files to the folders?"
A: Use the expressions as appropriate, e.g.
- name: "unarchive {{ item }}"
unarchive:
src: "{{ item }}"
dest: "{{ path }}/{{ item|splitext|first }}/"
loop: "{{ files }}"
(not tested)
It's up to you where you put the variables files and path. See Variable precedence: Where should I put a variable?
I am trying to copy some configuration files from /tmp to /opt.
Here first I am recursively searching for the files in /tmp and /opt directories and storing it in the variable tmp_file_path and code_file_path respectively, which has an attribute files.path that I need to use in the source and destination for copy
- name: Find files in tmp
find:
paths: /tmp/
file_type: file
recurse: yes
patterns:
- file1
- file2
- file3
register: tmp_file_path
- debug:
var: tmp_file_path
- name: Find files in code
find:
paths: /opt/
file_type: file
recurse: yes
patterns:
- file1
- file2
- file3
register: code_file_path
- debug:
var: code_file_path
Here the source file paths can be /tmp/folder1/file1, /tmp/folder2/file2, /tmp/folder13/file3.
Destinations can be /opt/folderA/file1, /opt/folderB/file3, /opt/folderC/file3
As of now I have managed to write the task as below
- name: Copy files from tmp to code directory
copy:
src: "{{item.path}}"
dest: "{{item.path}}"
remote_src: yes
with_items:
- { "{{ tmp_file_path.files }}", "{{ code_file_path.files }}" }
The copy has to be done in a single command so that I do not end up hardcoding the paths for source and destination. Can anyone help me with achieving this?
Below piece of code worked for recursively copying files from code_file_path to /tmp
- name: Copy files from code directory to tmp
copy:
src: "{{item.path}}"
dest: /tmp/
remote_src: yes
with_items: "{{code_file_path.files}}"
Try this
- name: Copy files from tmp to code directory
copy:
src: "{{ item.0 }}"
dest: "{{ item.1 }}"
remote_src: yes
with_together:
- "{{ tmp_file_path.files|map(attribute='path')|list|sort }}"
- "{{ code_file_path.files|map(attribute='path')|list|sort }}"
Try with debug first
- debug:
msg:
- "src: {{ item.0 }}"
- "dest: {{ item.1 }}"
with_together:
- "{{ tmp_file_path.files|map(attribute='path')|list|sort }}"
- "{{ code_file_path.files|map(attribute='path')|list|sort }}"
It might be useful to test the sanity first
- debug:
msg: The numbers of files do not match
when: tmp_file_path.files|map(attribute='path')|list|length !=
code_file_path.files|map(attribute='path')|list|length
Q: "This did not work as expected because the files which I am searching in code_file_path has multiple sub-directories. It is sorting on the entire file path returned by code_file_path.files rather than just the file name."
A: Create a list with both paths and names. Then sort the list by the name. For example
- set_fact:
code_files: "{{ code_files|default([]) +
[{'path': item, 'name': item|basename}] }}"
loop: "{{ code_file_path.files|map(attribute='path')|list }}"
- debug:
msg:
- "src: {{ item.0 }}"
- "dest: {{ item.1.path }}"
with_together:
- "{{ tmp_file_path.files|map(attribute='path')|list|sort }}"
- "{{ code_files|sort(attribute='name') }}"
Example
shell> tree tmp
tmp
├── file1
├── file2
└── file3
shell> tree opt
opt
├── bar
│ └── file2
├── baz
│ └── file3
└── foo
└── file1
The tasks below
- set_fact:
code_files: "{{ code_files|default([]) +
[{'path': item, 'name': item|basename}] }}"
loop: "{{ code_file_path.files|map(attribute='path')|list }}"
- debug:
var: code_files|sort(attribute='name')
- debug:
msg:
- "src: {{ item.0 }}"
- "dest: {{ item.1.path }}"
with_together:
- "{{ tmp_file_path.files|map(attribute='path')|list|sort }}"
- "{{ code_files|sort(attribute='name') }}"
give
"code_files|sort(attribute='name')": [
{
"name": "file1",
"path": "/export/test/opt/foo/file1"
},
{
"name": "file2",
"path": "/export/test/opt/bar/file2"
},
{
"name": "file3",
"path": "/export/test/opt/baz/file3"
}
]
"msg": [
"src: /export/test/tmp/file1",
"dest: /export/test/opt/foo/file1"
]
"msg": [
"src: /export/test/tmp/file2",
"dest: /export/test/opt/bar/file2"
]
"msg": [
"src: /export/test/tmp/file3",
"dest: /export/test/opt/baz/file3"
]
You may achieve this with this condition : every filenames are unique in the way that if 2 files have the same name in your list from /opt directory, then their content should be the same.
If it's the case, then you may use the with_nested loop and using a conditionnal when on filename.
For example:
- name: Copy files from tmp to code directory
copy:
src: "{{ item.0 }}"
dest: "{{ item.1 }}"
remote_src: yes
when:
- item.0|basename == item.1|basename
with_nested:
- "{{ tmp_file_path.files|map(attribute='path')|list }}"
- "{{ code_file_path.files|map(attribute='path')|list }}"
The only "problem" with this solution is that you'll run the loop a lot of times...
You may also want to use the loop synthax over with_ synthax and use some loop_control in order to choose what is printed: https://docs.ansible.com/ansible/latest/user_guide/playbooks_loops.html
If you have the jinja.ext.do extension loaded (in your ansible.cfg: jinja2_extensions = jinja2.ext.do) you may then contruct a dict with your paths:
- name: Make a dict
set_fact:
files_dict: |
{%- set out_dict = dict() -%}
{%- for tmp_file in tmp_file_path.files|map(attribute='path')|list -%}
{%- do out_dict.update({tmp_file|basename: {'tmp': tmp_file}}) -%}
{%- endfor -%}
{%- for opt_file in code_file_path.files|map(attribute='path')|list -%}
{%- do out_dict[opt_file|basename].update({'opt': opt_file}) -%}
{%- endfor -%}
{{ out_dict }}
- name: Copy from dict
copy:
src: "{{ item.value.tmp }}"
dest: "{{ item.value.opt }}"
remote_src: yes
with_dict: "{{ files_dict }}"
I want to archive the following directory:
temp_build_directory: /tmp/mdr-upgrade
Where
$ ls -1 /tmp/mdr-upgrade
ansible
atr
composefiles
data
images
packs
wheelhouse
and the task is:
- name: archive_artifacts.yml --> Archive artifacts
archive:
path: "{{ temp_build_directory }}/*"
dest: "{{ target_tmp_dir }}/{{ artifacts_file_name }}"
exclude_path: "{{ target_tmp_dir }}/{{ ansible_dir }}"
And ansible_dir: ansible
Tarball ends up always containing the ansible folder.
Why is that?
edit: I am using target_tmp_dir: "/tmp"
exclude_path needs an absolute path (see docs).
Try again with:
- name: archive_artifacts.yml --> Archive artifacts
archive:
path: "{{ temp_build_directory }}/*"
dest: "{{ target_tmp_dir }}/{{ artifacts_file_name }}"
exclude_path: "{{ temp_build_directory }}/{{ ansible_dir }}"
I believe the Ansible copy module can take a whole bunch of "files" and copy them in one hit. This I believe can be achieved by copying a directory recursively.
Can the Ansible template module take a whole bunch of "templates" and deploy them in one hit? Is there such a thing as deploying a folder of templates and applying them recursively?
The template module itself runs the action on a single file, but you can use with_filetree to loop recursively over a specified path:
- name: Ensure directory structure exists
ansible.builtin.file:
path: '{{ templates_destination }}/{{ item.path }}'
state: directory
with_community.general.filetree: '{{ templates_source }}'
when: item.state == 'directory'
- name: Ensure files are populated from templates
ansible.builtin.template:
src: '{{ item.src }}'
dest: '{{ templates_destination }}/{{ item.path }}'
with_community.general.filetree: '{{ templates_source }}'
when: item.state == 'file'
And for templates in a single directory you can use with_fileglob.
This answer provides a working example of the approach laid down by #techraf
with_fileglob expects only files to live within the templates folder - see https://serverfault.com/questions/578544/deploying-a-folder-of-template-files-using-ansible
with_fileglob will only parse files in the templates folder
with_filetree maintains the directory structure when moving the template files to dest. It auto creates those directories for you at dest.
with_filetree will parse all files in the templates folder and nested directories
- name: Approve certs server directories
file:
state: directory
dest: '~/{{ item.path }}'
with_filetree: '../templates'
when: item.state == 'directory'
- name: Approve certs server files
template:
src: '{{ item.src }}'
dest: '~/{{ item.path }}'
with_filetree: '../templates'
when: item.state == 'file'
Essentially, think of this approach as copying and pasting a directory and all its contents from A to B and whilst doing so, parsing all templates.
I could not manage to do it with the other answers. This is what worked for me:
- name: Template all the templates and place them in the corresponding path
template:
src: "{{ item.src }}"
dest: "{{ destination_path }}/{{ item.path | regex_replace('\\.j2$', '') }}"
force: yes
with_filetree: '{{ role_path }}/templates'
when: item.state == 'file'
In my case folder contain both files and jinja2 templates.
- name: copy all directories recursively
file: dest={{templates_dest_path}}/{{ item|replace(templates_src_path+'/', '') }} state=directory
with_items: "{{ lookup('pipe', 'find '+ templates_src_path +'/ -type d').split('\n') }}"
- name: copy all files recursively
copy: src={{ item }} dest={{templates_dest_path}}/{{ item|replace(templates_src_path+'/', '') }}
with_items: "{{ lookup('pipe', 'find '+ templates_src_path +'/ -type f -not -name *.j2 ').split('\n') }}"
- name: copy templates files recursively
template: src={{ item }} dest={{templates_dest_path}}/{{ item|replace(templates_src_path+'/', '')|replace('.j2', '') }}
with_items: "{{ lookup('pipe', 'find '+ templates_src_path +'/*.j2 -type f').split('\n') }}"
I did it and it worked. \o/
- name: "Create file template"
template:
src: "{{ item.src }}"
dest: "{{ your_dir_remoto }}/{{ item.dest }}"
loop:
- { src: '../templates/file1.yaml.j2', dest: 'file1.yaml' }
- { src: '../templates/file2.yaml.j2', dest: 'file2.yaml' }