I'm trying to append a string to each value of a list in ansible, so basically am trying to install multiple pip modules offline using .whl files.
I have two files in /opt/tmp/ path
vars:
DIR: /opt/
pymongo_modules:
- pip-19.1.1-py2.py3-none-any.whl
- pymongo-3.8.0-cp27-cp27mu-manylinux1_x86_64.whl
- name: Install the latest pymongo package
pip:
name: "{{DIR}}/tmp/{{ pymongo_modules | join(' ') }}"
executable: "{{pip_path}}"
The above is not working because it's formating like below
"name": ["/opt/tmp/pip-19.1.1-py2.py3-none-any.whl pymongo-3.8.0-cp27-cp27mu-manylinux1_x86_64.whl"]
I can achieve the same with below syntax but I'm getting deprecation warning
- name: Install the latest pymongo package
pip:
name: "{{DIR}}/tmp/{{ module }}"
executable: "{{pip_path}}"
with_items:
- "{{ pymongo_modules }}"
loop_control:
loop_var: module
Expecting value:
"name": ["/opt/tmp/pip-19.1.1-py2.py3-none-any.whl", "/opt/tmp/pymongo-3.8.0-cp27-cp27mu-manylinux1_x86_64.whl"]
Use product filter like below. BTW, DIR variable already ends with a / so do not need an additional / before tmp.
- debug:
msg: "{{ item }}"
loop: "{{ [(DIR + 'tmp')] | product(pymongo_modules) | map('join', '/') | list }}"
Gives:
ok: [localhost] => (item=/opt/tmp/pip-19.1.1-py2.py3-none-any.whl) => {
"msg": "/opt/tmp/pip-19.1.1-py2.py3-none-any.whl"
}
ok: [localhost] => (item=/opt/tmp/pymongo-3.8.0-cp27-cp27mu-manylinux1_x86_64.whl) => {
"msg": "/opt/tmp/pymongo-3.8.0-cp27-cp27mu-manylinux1_x86_64.whl"
}
I've just answered a similar question over here. In summary, you can achieve this with regex_replace and map. It's quite a tidy method and is also in the official Ansible docs, so it seems to be recommended too.
Related
Below ansible tasks display pip packages in dictionary format.
- name: Ganther pip packages
pip_package_info:
clients: pip3
register: pip_pkgs
And the output is:
{
"packages": {
pip3: {
"xxx": [
{
name: "xxx"
source: "pip3"
version: "1.0.0"
}
],
"yyy": [
{
name: "yyy"
source: "pip3"
version: "2.0.0"
}
]
}
}
}
I tried to loop through the registered variable, but, I am getting errors about undefined variable. How to get the name and version from the above dictionary?
You can use the values method of the Python dictionary in order to have something that you could loop on:
- debug:
msg: "{{ item.name }} is in version {{ item.version }}"
loop: "{{ pip_pkgs.packages.pip3.values() | flatten }}"
Alternatively, it can also be achieved with dict2items, but makes the loop syntax a little bit longer, since you then have to map the attribute you are interested in — here, the value:
- debug:
msg: "{{ item.name }} is in version {{ item.version }}"
loop: >-
{{
pip_pkgs.packages.pip3
| dict2items
| map(attribute='value')
| flatten
}}
If you want to create a dictionary, this would be the easiest:
- set_fact:
pip_packages: >-
{{
dict(
_pip_pkg | map(attribute='name')
| zip(_pip_pkg | map(attribute='version'))
)
}}
vars:
_pip_pkg: "{{ pip_pkgs.packages.pip3.values() | flatten }}"
The dictionary would look like
pip_packages:
ansible: 5.6.0
ansible-compat: 2.0.2
ansible-core: 2.12.4
ansible-lint: 6.0.2
Given the two tasks:
- pip_package_info:
clients: pip3
register: pip_pkgs
- debug:
msg: "{{ item.name }} is in version {{ item.version }}"
loop: "{{ pip_pkgs.packages.pip3.values() | flatten }}"
loop_control:
label: "{{ item.name }}"
This gives:
ok: [localhost] => (item=ansible) =>
msg: ansible is in version 5.6.0
ok: [localhost] => (item=ansible-compat) =>
msg: ansible-compat is in version 2.0.2
ok: [localhost] => (item=ansible-core) =>
msg: ansible-core is in version 2.12.5
ok: [localhost] => (item=ansible-lint) =>
msg: ansible-lint is in version 6.0.2
... list goes on
SOLUTION:
I Don't know what is the exact difference between python -m pip install ansible or apt install ansible but when I installed python -m pip ansible-core==2.10.4 it works fine.
I have CSV file which looks like:
id;env;credentials;path
1;tst;userA;/tmpA
2;dev;userB;/tmpB
3;dev;userB;/tmpC
4;acc;userB;/tmpD
5;prd;userC;/tmpE
I read this file using read_csv module and then I'm filtering using selectattr:
- name: Read CSV
read_csv:
path: "/tmp/example.csv"
delimiter: ';'
register: csv_output
- name: Filter rows
set_fact:
new_fact: "{{ csv_output.list | selectattr('env', 'equalto', tst) }}"
In the past I was able to just use these results as a dict so for example:
- debug:
msg: "{{ new_fact }}"
ok: [ansible_main] => {
"msg": [
{
"id": "1",
"env": "tst",
"credentials": "userA",
"path": "/tmpA"
}
]
}
but when I try to print new_fact on my local machine I see only generator:
ok: [ansible_main] => {
"msg": "<generator object select_or_reject at 0x7f2e4e8847b0>"
}
and I cannot use new_fact.credentials variable... Do you know how can I fix it? I know I can add | list at the end of my filter but then I also cannot use new_fact.credentials
Details of my installation:
ansible 2.9.6
config file = /etc/ansible/ansible.cfg
configured module search path = ['/home/userA/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python3/dist-packages/ansible
executable location = /usr/bin/ansible
python version = 3.8.10 (default, Nov 26 2021, 20:14:08) [GCC 9.3.0]
Regarding
In the past I was able to just use these results as a dict ...
was it a much newer version of Ansible than v2.9.6?
I know I can add | list at the end of my filter but then I also cannot user new_fact.credentials then
Since you will get a list too, it will be necessary to specify the element
- name: Filter rows
set_fact:
new_fact: "{{ csv_output.list | selectattr('env', 'contains', 'tst') | list }}"
- debug:
msg: "{{ new_fact[0].credentials }}"
to loop over the result, or pickup one element only.
- name: Filter rows
set_fact:
new_fact: "{{ csv_output.list | selectattr('env', 'contains', 'tst') | first }}"
- debug:
msg: "{{ new_fact.credentials }}"
I'm building an Ansible playbook in which I want to make a backup of a database in case I need to upgrade the software. For this I want to compare the highest version number that is available to the version number that is installed. In case the latest version is hight than the installed version I'll back up the database.
The problem however is that I cannot find a good way to sort version numbers in Ansible. The standard sort filter sorts on strings instead of numbers/versions.
This is what I'm doing right now:
- name: Get package version
yum:
list: package
register: software_version
- name: Read which version is installed and available
set_fact:
software_version_installed: "{{ software_version | json_query(\"results[?yumstate=='installed'].version\") | sort | last }}"
software_version_available: "{{ software_version | json_query(\"results[?yumstate=='available'].version\") | sort | last }}"
- name: Backup old database file on remote host
copy:
src: "{{ software.database_path }}"
dest: "{{ software.database_path }}_{{ ansible_date_time.date }}_v{{ software_version_installed }}"
remote_src: yes
when: software_version_installed is version(software_version_available, "<")
The playbook above works, as long as version numbers stay underneath the number 10 (e.g. 1.2.3, but not 1.10.1) since sorting is performed like a string. When the version number has to sort e.g. 1.2.3 and 1.10.1, it will take 1.2.3 as latest version.
To show the issue:
- name: Read which version is installed and available
set_fact:
software_versions: [ "2.5.0", "2.9.0", "2.10.0", "2.11.0" ]
- name: Debug
debug:
var: software_versions | sort
TASK [grafana : Debug]**********************************
ok: [localhost] => {
"software_versions | sort": [
"2.10.0",
"2.11.0",
"2.5.0",
"2.9.0"
]
}
Does anyone know a good way to sort version numbers in Ansible?
Q: Does anyone know a good way to sort version numbers in Ansible?
A: Use filter_plugin. For example the filter
shell> cat filter_plugins/version_sort.py
from distutils.version import LooseVersion
def version_sort(l):
return sorted(l, key=LooseVersion)
class FilterModule(object):
def filters(self):
return {
'version_sort' : version_sort
}
with the playbook
shell> cat test-versions.yml
- name: Sort versions
hosts: localhost
vars:
versions:
- "0.1.0"
- "0.1.5"
- "0.11.11"
- "0.9.11"
- "0.9.3"
- "0.10.2"
- "0.6.1"
- "0.6.0"
- "0.11.0"
- "0.6.5"
tasks:
- debug:
msg: "{{ versions|version_sort }}"
gives
"msg": [
"0.1.0",
"0.1.5",
"0.6.0",
"0.6.1",
"0.6.5",
"0.9.3",
"0.9.11",
"0.10.2",
"0.11.0",
"0.11.11"
]
For your convenience, the filter is available at Github ansible-plugins.
Version comparison does the job to iterate the list and compare items. See the example below
shell> cat test-versions.yml
- hosts: localhost
vars:
version_installed: "1.10.1"
versions:
- "1.1.3"
- "1.2.3"
- "1.7.5"
- "1.10.7"
tasks:
- debug: msg="{{ item }} is newer than {{ version_installed }}"
loop: "{{ versions }}"
when: item is version(version_installed, '>')
shell> ansible-playbook test-versions.yml | grep msg
"msg": "1.10.7 is newer than 1.10.1"
It's now solved in another way. Instead of sorting the versions I compared the current version to all available versions.
I've started by setting an update variable to false
Next I compared the installed version to every available version
If installed version > current version, set the update variable to true
The task performing the backup will only be performed when the update variable is true.
- name: Get package version
yum:
list: package
register: software_version
- name: Read which version is installed and available
set_fact:
software_update: false
software_version_installed: "{{ software_version | json_query(\"results[?yumstate=='installed'].version\") | last }}"
software_version_available: "{{ software_version | json_query(\"results[?yumstate=='available'].version\") }}"
- name: Check if upgrade is needed
set_fact:
software_update: true
when: software_version_installed is version(item, "<")
with_items: "{{ software_version_available }}"
- name: Backup old database file on remote host
copy:
src: "{{ software.database_path }}"
dest: "{{ software.database_path }}_{{ ansible_date_time.date }}_v{{ software_version_installed }}"
remote_src: yes
when: software_update
All the answers here provide a custom way of sorting versions, I'd like to point out that stock ansible can do this already:
- name: Sort list by version number
debug:
var: ansible_versions | community.general.version_sort
vars:
ansible_versions:
- '2.8.0'
- '2.11.0'
- '2.7.0'
- '2.10.0'
- '2.9.0'
https://docs.ansible.com/ansible/latest/collections/community/general/docsite/filter_guide_working_with_versions.html
The current accepted answer offers a very nice solution, using a filter_plugin. Unfortunately, the distutils Python package that is uses appears to be deprecated. Some googling led me to the packaging package, which offers a similar Version class. Here's an updated filter_plugin that doesn't use distutils:
from packaging.version import Version
def version_sort(l):
return sorted(l, key=Version)
class FilterModule(object):
def filters(self):
return {
'version_sort': version_sort
}
It's working well for us, but I don't want to promise that the behavior will be exactly the same in every situation.
I'm building an Ansible playbook in which I want to make a backup of a database in case I need to upgrade the software. For this I want to compare the highest version number that is available to the version number that is installed. In case the latest version is hight than the installed version I'll back up the database.
The problem however is that I cannot find a good way to sort version numbers in Ansible. The standard sort filter sorts on strings instead of numbers/versions.
This is what I'm doing right now:
- name: Get package version
yum:
list: package
register: software_version
- name: Read which version is installed and available
set_fact:
software_version_installed: "{{ software_version | json_query(\"results[?yumstate=='installed'].version\") | sort | last }}"
software_version_available: "{{ software_version | json_query(\"results[?yumstate=='available'].version\") | sort | last }}"
- name: Backup old database file on remote host
copy:
src: "{{ software.database_path }}"
dest: "{{ software.database_path }}_{{ ansible_date_time.date }}_v{{ software_version_installed }}"
remote_src: yes
when: software_version_installed is version(software_version_available, "<")
The playbook above works, as long as version numbers stay underneath the number 10 (e.g. 1.2.3, but not 1.10.1) since sorting is performed like a string. When the version number has to sort e.g. 1.2.3 and 1.10.1, it will take 1.2.3 as latest version.
To show the issue:
- name: Read which version is installed and available
set_fact:
software_versions: [ "2.5.0", "2.9.0", "2.10.0", "2.11.0" ]
- name: Debug
debug:
var: software_versions | sort
TASK [grafana : Debug]**********************************
ok: [localhost] => {
"software_versions | sort": [
"2.10.0",
"2.11.0",
"2.5.0",
"2.9.0"
]
}
Does anyone know a good way to sort version numbers in Ansible?
Q: Does anyone know a good way to sort version numbers in Ansible?
A: Use filter_plugin. For example the filter
shell> cat filter_plugins/version_sort.py
from distutils.version import LooseVersion
def version_sort(l):
return sorted(l, key=LooseVersion)
class FilterModule(object):
def filters(self):
return {
'version_sort' : version_sort
}
with the playbook
shell> cat test-versions.yml
- name: Sort versions
hosts: localhost
vars:
versions:
- "0.1.0"
- "0.1.5"
- "0.11.11"
- "0.9.11"
- "0.9.3"
- "0.10.2"
- "0.6.1"
- "0.6.0"
- "0.11.0"
- "0.6.5"
tasks:
- debug:
msg: "{{ versions|version_sort }}"
gives
"msg": [
"0.1.0",
"0.1.5",
"0.6.0",
"0.6.1",
"0.6.5",
"0.9.3",
"0.9.11",
"0.10.2",
"0.11.0",
"0.11.11"
]
For your convenience, the filter is available at Github ansible-plugins.
Version comparison does the job to iterate the list and compare items. See the example below
shell> cat test-versions.yml
- hosts: localhost
vars:
version_installed: "1.10.1"
versions:
- "1.1.3"
- "1.2.3"
- "1.7.5"
- "1.10.7"
tasks:
- debug: msg="{{ item }} is newer than {{ version_installed }}"
loop: "{{ versions }}"
when: item is version(version_installed, '>')
shell> ansible-playbook test-versions.yml | grep msg
"msg": "1.10.7 is newer than 1.10.1"
It's now solved in another way. Instead of sorting the versions I compared the current version to all available versions.
I've started by setting an update variable to false
Next I compared the installed version to every available version
If installed version > current version, set the update variable to true
The task performing the backup will only be performed when the update variable is true.
- name: Get package version
yum:
list: package
register: software_version
- name: Read which version is installed and available
set_fact:
software_update: false
software_version_installed: "{{ software_version | json_query(\"results[?yumstate=='installed'].version\") | last }}"
software_version_available: "{{ software_version | json_query(\"results[?yumstate=='available'].version\") }}"
- name: Check if upgrade is needed
set_fact:
software_update: true
when: software_version_installed is version(item, "<")
with_items: "{{ software_version_available }}"
- name: Backup old database file on remote host
copy:
src: "{{ software.database_path }}"
dest: "{{ software.database_path }}_{{ ansible_date_time.date }}_v{{ software_version_installed }}"
remote_src: yes
when: software_update
All the answers here provide a custom way of sorting versions, I'd like to point out that stock ansible can do this already:
- name: Sort list by version number
debug:
var: ansible_versions | community.general.version_sort
vars:
ansible_versions:
- '2.8.0'
- '2.11.0'
- '2.7.0'
- '2.10.0'
- '2.9.0'
https://docs.ansible.com/ansible/latest/collections/community/general/docsite/filter_guide_working_with_versions.html
The current accepted answer offers a very nice solution, using a filter_plugin. Unfortunately, the distutils Python package that is uses appears to be deprecated. Some googling led me to the packaging package, which offers a similar Version class. Here's an updated filter_plugin that doesn't use distutils:
from packaging.version import Version
def version_sort(l):
return sorted(l, key=Version)
class FilterModule(object):
def filters(self):
return {
'version_sort': version_sort
}
It's working well for us, but I don't want to promise that the behavior will be exactly the same in every situation.
I'm using Ansible to install packages on a new deployment. I have a pre-defined list of dicts in a variable.
I want to open an interface to update this list using Jenkins.
My list looks like this:
package_list: [
{'name': 'python-devel', 'apt': 'python-dev'},
{'name': 'python-pip'},
{'name': 'postgresql-devel'},
...
]
The way I communicate Jenkins input to Ansible is using environment variables. I can pass a list of additional packages to be installed and read it as part of my Ansible configuration.
Question is: How I convert a list of strings, to a list of dictionaries matches the structure of my package_list?
For example:
ENV:
PACKAGES=gcc,vim,ntp
ANSIBLE:
additional_packages = [
{'name': 'gcc'},
{'name': 'vim'},
{'name': 'ntp'}
]
Is it even possible?
i believe this playbook will get you where you want. it assumes you have the env variable: PACKAGES=gcc,vim,ntp
it converts the string variable to a list (split by ,), and then in another loop it converts to a list of dictionaries:
playbook:
- hosts: localhost
gather_facts: false
vars:
tasks:
- name: pick up env variable, convert to list
set_fact:
PACKAGES: "{{ lookup('env', 'PACKAGES').split(',') }}"
- name: create dict list variable
set_fact:
PACKAGES_DICT: "{{ PACKAGES_DICT|default([]) + [{'name': item}] }}"
with_items:
- "{{ PACKAGES }}"
- name: print results
debug:
var: PACKAGES_DICT
results:
TASK [print results] **************************************************************************************************************************************************************************************************
ok: [localhost] => {
"PACKAGES_DICT": [
{
"name": "gcc"
},
{
"name": "vim"
},
{
"name": "ntp"
}
]
}
hope this helps
EDIT
refining the code, removing the set_fact task, declaring the PACKAGES variable in vars section:
- hosts: localhost
gather_facts: false
vars:
PACKAGES: "{{ lookup('env', 'PACKAGES').split(',') }}"
tasks:
- name: create dict list variable
set_fact:
PACKAGES_DICT: "{{ PACKAGES_DICT|default([]) + [{'name': item}] }}"
with_items:
- "{{ PACKAGES }}"
- name: print results
debug:
var: PACKAGES_DICT
Using Ansible, list should be written that way:
package_list:
- name: "gcc"
- name: "vim"
- name: "ntp
so to get that list from the string you can do that way:
vars:
package_list: "{{ packages.split(',').values() | list }}"