Ansible / jmespath - 1d list to zipped list of dictionaries - ansible

I'm attempting to transform a 1d list of AWS EC2 IDs into a list of dictionaries suitable for usage as the targets param to the Ansible elb_target_group module.
Sample input:
TASK [debug]
ok: [localhost] => {
"instance_ids": [
"i-1111",
"i-2222",
"i-3333"
]
}
Desired output:
TASK [debug]
ok: [localhost] => {
"targets": [
{"Id": "i-1111", "Port": 6443},
{"Id": "i-2222", "Port": 6443},
{"Id": "i-3333", "Port": 6443},
]
}
What I've tried: a json_query / JMESPath expression which confusingly returns null values:
---
- name: test
hosts: localhost
connection: local
vars:
instance_ids:
- i-1111
- i-2222
- i-3333
keys_list:
- Id
- Port
tasks:
- debug: var=instance_ids
- debug:
msg: "{{ instance_ids | zip_longest([6443], fillvalue=6443) | list }}" # Halfway there...
- debug:
msg: "{{ instance_ids | zip_longest([6443], fillvalue=6443) | list | map('json_query', '{Id: [0], Port: [1]}') | list }}"
Output:
TASK [debug]
ok: [localhost] => {
"msg": [
[
"i-1111",
6443
],
[
"i-2222",
6443
],
[
"i-3333",
6443
]
]
}
TASK [debug]
ok: [localhost] => {
"msg": [
{
"Id": null,
"Port": null
},
{
"Id": null,
"Port": null
},
{
"Id": null,
"Port": null
}
]
}
How should I adjust the '{Id: [0], Port: [1]}' part ?

Stumbled my way to a working solution with json_query and a JMESPath int literal:
"{{ instance_ids | json_query('[*].{Id: #, Port: `6443`}') }}"
Example:
- name: test
hosts: localhost
connection: local
vars:
instance_ids:
- i-1111
- i-2222
- i-3333
tasks:
- debug:
msg: "{{ instance_ids | json_query('[*].{Id: #, Port: `6443`}') }}"
Output:
TASK [debug]
ok: [localhost] => {
"msg": [
{
"Id": "i-1111",
"Port": 6443
},
{
"Id": "i-2222",
"Port": 6443
},
{
"Id": "i-3333",
"Port": 6443
}
]
}

Related

Select specific cluster with prefix in VMware cluster from Ansible

I'm using the below playbook to list all the VMware cluster belonging to my datacenter, but I need to select a specific cluster containing LAB in the cluster name.
- hosts: localhost
vars_files: 1credentials.yml
tasks:
- name: Gather cluster info from given datacenter
community.vmware.vmware_cluster_info:
hostname: '{{ vcenter_hostname }}'
username: '{{ vcenter_username }}'
password: '{{ vcenter_password }}'
datacenter: SDx-CoE-T7
validate_certs: no
delegate_to: localhost
register: cluster_info
- debug:
msg: "{{ cluster_info }}"
- debug:
msg: "{{ cluster_info.clusters.keys() | list | to_yaml }}"
I'm trying to use selectattr(), but, I am not able to get the exact syntax. Can you please suggest the best option to filter the cluster name containing LAB.
Below is the playbook output:
PLAY [localhost] *******************************************************************************************************************************************************
TASK [Gathering Facts] *************************************************************************************************************************************************
ok: [localhost]
TASK [Gather cluster info from given datacenter] ***********************************************************************************************************************
ok: [localhost]
TASK [debug] ***********************************************************************************************************************************************************
ok: [localhost] => {
"msg": {
"changed": false,
"clusters": {
"TEST_LAB": {
"datacenter": "TEST_CoE-T7",
"drs_default_vm_behavior": "fullyAutomated",
"drs_enable_vm_behavior_overrides": true,
"drs_vmotion_rate": 3,
"enable_ha": false,
"enabled_drs": false,
"enabled_vsan": false,
"ha_admission_control_enabled": true,
"ha_failover_level": 1,
"ha_host_monitoring": "enabled",
"ha_restart_priority": [
"medium"
],
"ha_vm_failure_interval": [
30
],
"ha_vm_max_failure_window": [
-1
],
"ha_vm_max_failures": [
3
],
"ha_vm_min_up_time": [
120
],
"ha_vm_monitoring": "vmMonitoringDisabled",
"ha_vm_tools_monitoring": [
"vmMonitoringDisabled"
],
"hosts": [],
"moid": "domain-c28615",
"resource_summary": {
"cpuCapacityMHz": 0,
"cpuUsedMHz": 0,
"memCapacityMB": 0,
"memUsedMB": 0,
"pMemAvailableMB": 0,
"pMemCapacityMB": 0,
"storageCapacityMB": 0,
"storageUsedMB": 0
},
"tags": [],
"vsan_auto_claim_storage": false
},
"TEST_LaaS": {
"datacenter": "TEST_CoE-T7",
"drs_default_vm_behavior": "fullyAutomated",
"drs_enable_vm_behavior_overrides": true,
"drs_vmotion_rate": 3,
"enable_ha": true,
"enabled_drs": true,
"enabled_vsan": false,
"ha_admission_control_enabled": true,
"ha_failover_level": 1,
"ha_host_monitoring": "enabled",
"ha_restart_priority": [
"medium"
],
"ha_vm_failure_interval": [
30
],
"ha_vm_max_failure_window": [
-1
],
"ha_vm_max_failures": [
3
],
"ha_vm_min_up_time": [
120
],
"ha_vm_monitoring": "vmMonitoringDisabled",
"ha_vm_tools_monitoring": [
"vmMonitoringDisabled"
],
"hosts": [
{
"folder": "/TEST_CoE-T7/host/TEST_LaaS",
"name": "172.17.65.84"
},
{
"folder": "/TEST_CoE-T7/host/TEST_LaaS",
"name": "172.17.65.85"
},
{
"folder": "/TEST_CoE-T7/host/TEST_LaaS",
"name": "172.17.168.202"
}
],
"moid": "domain-c861",
"resource_summary": {
"cpuCapacityMHz": 158040,
"cpuUsedMHz": 17494,
"memCapacityMB": 786073,
"memUsedMB": 361592,
"pMemAvailableMB": 0,
"pMemCapacityMB": 0,
"storageCapacityMB": 22707456,
"storageUsedMB": 14326118
},
"tags": [],
"vsan_auto_claim_storage": false
}
},
"failed": false
}
}
TASK [debug] ***********************************************************************************************************************************************************
ok: [localhost] => {
"msg": "[TEST_Automation, TEST_LAB, TEST_LaaS,]\n"
}
If I use ansible version 2.9.27, I'm getting below error:
fatal: [localhost]: FAILED! => {"msg": "Unexpected templating type error occurred on ({{ cluster_info.clusters | dict2items | selectattr('key','contains','Common')| items2dict | list }}): items2dict requires a list, got <class 'generator'> instead."}
I tried with below one removing items2dict:
- debug:
msg: "{{ cluster_info.clusters | dict2items | selectattr('key','contains','TEST') | list }}"
It is giving the below output but I need only the key: TEST_Lab
ok: [localhost] => {
"msg": [
{
"key": "TEST_LAB",
"value": {
"drs_default_vm_behavior": "partiallyAutomated",
"drs_enable_vm_behavior_overrides": true,
"drs_vmotion_rate": 3,
"enable_ha": false,
"enabled_drs": true,
"enabled_vsan": false,
"ha_admission_control_enabled": true,
"ha_failover_level": 1,
"ha_host_monitoring": "enabled",
"ha_restart_priority": [
"medium"
],
"ha_vm_failure_interval": [
30
],
"ha_vm_max_failure_window": [
-1
],
"ha_vm_max_failures": [
3
],
"ha_vm_min_up_time": [
120
],
"ha_vm_monitoring": "vmMonitoringDisabled",
"ha_vm_tools_monitoring": [
"vmMonitoringDisabled"
],
"tags": [],
"vsan_auto_claim_storage": false
}
}
]
}
In order to use selectattr, you need to have a value to filter on, not a key. But transforming a dictionary into a list, containing the key and value is as simple as using the dict2items filter, in Ansible.
Then, to come back to your dictionary, just use the "antonym" filter, items2dict.
So, your debug task ends up being:
- debug:
var: >-
cluster_info.clusters
| dict2items
| selectattr('key','contains','LAB')
| items2dict
Given the task, with a reduced dictionary for readability:
- debug:
var: |-
cluster_info.clusters
| dict2items
| selectattr('key','contains','LAB')
| items2dict
vars:
cluster_info:
clusters:
TEST_LAB:
datacenter: TEST_CoE-T7
drs_default_vm_behavior: fullyAutomated
TEST_LaaS:
datacenter: TEST_CoE-T7
drs_default_vm_behavior: fullyAutomated
This will yield:
ok: [localhost] =>
? |-
cluster_info.clusters
| dict2items
| selectattr('key','contains','LAB')
| items2dict
: TEST_LAB:
datacenter: TEST_CoE-T7
drs_default_vm_behavior: fullyAutomated

Ansible - Extracting a variable from a large JSON output

The gcp_sql_instance_info module does not seem to allow you to specify the name of the database you are trying to get info for. All it does is every detail of every database in that project.
What would be the best way to extract the "ipAddress" of a database based on it's name and register that value to a variable?
{
"resources": [
{
"kind": "sql#instance",
"state": "RUNNABLE",
"databaseVersion": "POSTGRES_13",
"settings": {
"authorizedGaeApplications": [],
"tier": "db-custom-1-3840",
"kind": "sql#settings",
"availabilityType": "ZONAL",
"pricingPlan": "PER_USE",
"replicationType": "SYNCHRONOUS",
"activationPolicy": "ALWAYS",
"ipConfiguration": {
"privateNetwork": "example-network",
"authorizedNetworks": [],
"ipv4Enabled": false
},
"locationPreference": {
"zone": "us-west1-b",
"kind": "sql#locationPreference"
},
"dataDiskType": "PD_SSD",
"maintenanceWindow": {
"kind": "sql#maintenanceWindow",
"hour": 0,
"day": 0
},
"backupConfiguration": {
"startTime": "02:00",
"kind": "sql#backupConfiguration",
"backupRetentionSettings": {
"retentionUnit": "COUNT",
"retainedBackups": 7
},
"enabled": false,
"replicationLogArchivingEnabled": false,
"pointInTimeRecoveryEnabled": false,
"transactionLogRetentionDays": 7
},
"settingsVersion": "3",
"storageAutoResizeLimit": "0",
"storageAutoResize": true,
"dataDiskSizeGb": "100"
},
"ipAddresses": [
{
"type": "PRIVATE",
"ipAddress": "10.10.10.1"
}
],
"instanceType": "CLOUD_SQL_INSTANCE",
"backendType": "SECOND_GEN",
"name": "database-1",
"region": "us-west1",
"gceZone": "us-west1-b",
"databaseInstalledVersion": "POSTGRES_13_7",
"createTime": "2022-07-22T20:20:32.274Z"
},
{
"kind": "sql#instance",
"state": "RUNNABLE",
"databaseVersion": "MYSQL_8_0",
"settings": {
"authorizedGaeApplications": [],
"tier": "db-n1-standard-1",
"kind": "sql#settings",
"availabilityType": "ZONAL",
"pricingPlan": "PER_USE",
"replicationType": "SYNCHRONOUS",
"activationPolicy": "ALWAYS",
"ipConfiguration": {
"privateNetwork": "example-network",
"authorizedNetworks": [],
"ipv4Enabled": false
},
"locationPreference": {
"zone": "us-west1-c",
"kind": "sql#locationPreference"
},
"dataDiskType": "PD_SSD",
"backupConfiguration": {
"startTime": "21:00",
"kind": "sql#backupConfiguration",
"backupRetentionSettings": {
"retentionUnit": "COUNT",
"retainedBackups": 7
},
"enabled": false,
"transactionLogRetentionDays": 7
},
"settingsVersion": "1",
"storageAutoResizeLimit": "0",
"storageAutoResize": true,
"dataDiskSizeGb": "10"
},
"ipAddresses": [
{
"type": "PRIVATE",
"ipAddress": "10.10.10.2"
}
],
"instanceType": "CLOUD_SQL_INSTANCE",
"backendType": "SECOND_GEN",
"name": "database-2",
"region": "us-west1",
"gceZone": "us-west1-c",
"databaseInstalledVersion": "MYSQL_8_0_26",
"createTime": "2022-07-27T19:27:59.235Z"
}
]
Q: "Extract the ipAddress of a database based on its name."
A: Crete a dictionary. For example, select the first item from the list ipAddresses
name_ip: "{{ dict(resources|json_query(name_ip_query)) }}"
name_ip_query: '[].[name, ipAddresses[0].ipAddress]'
give
name_ip:
database-1: 10.10.10.1
database-2: 10.10.10.2
Notes
Example of a complete playbook for testing
- hosts: localhost
vars_files:
- resources.yml
vars:
name_ip: "{{ dict(resources|json_query(name_ip_query)) }}"
name_ip_query: '[].[name, ipAddresses[0].ipAddress]'
name_ver: "{{ dict(resources|json_query(name_ver_query)) }}"
name_ver_query: '[].[name, databaseVersion]'
ip_of_db1: "{{ name_ip['database-1'] }}"
tasks:
- debug:
var: name_ip
- debug:
var: name_ver
- debug:
var: ip_of_db1
gives
TASK [debug] *********************************************************************************
ok: [localhost] =>
name_ip:
database-1: 10.10.10.1
database-2: 10.10.10.2
TASK [debug] *********************************************************************************
ok: [localhost] =>
name_ver:
database-1: POSTGRES_13
database-2: MYSQL_8_0
TASK [debug] *********************************************************************************
ok: [localhost] =>
ip_of_db1: 10.10.10.1
There might be more IP addresses. The variable ipAddresses is a list. An option would be to create a list of all IP addresses. For example,
name_ips: "{{ dict(resources|json_query(name_ips_query)) }}"
name_ips_query: '[].[name, ipAddresses[].ipAddress]'
give
name_ips:
database-1: [10.10.10.1]
database-2: [10.10.10.2]
This is a working playbook to achieve the desired output:
- hosts: localhost
vars:
db_instance_name: instance-1
tasks:
- name: get info on an instance
gcp_sql_instance_info:
project: project_id
auth_kind: serviceaccount
service_account_file: "/path/to/sa/example.json"
register: cloud_sql_resources
- set_fact:
query: "resources[?contains(name,'{{ db_instance_name }}')].ipAddresses[].ipAddress"
- set_fact:
ipAddress: "{{ cloud_sql_resources | to_json | from_json | json_query(query) }}"
- debug:
msg: "{{ ipAddress }}"
You can run your playbook like this: ansible-playbook -v main.yaml --extra-vars "db_instance_name=instance-1", that would print:
TASK [debug] ********************************************************************************************************************
ok: [localhost] => {
"msg": [
"35.224.46.127"
]
}

Extracting json from ansible playbook

I am fairly new to Ansible, so apologies if this a naïve question.
When I execute Ansible Playbook using AnsibleRunner, I get output in the following format:
PLAY [MyPlay] ******************************************************************************************************************************************************************************************************************
TASK [Gathering Facts] *********************************************************************************************************************************************************************************************************************************************
ok: [localhost]
TASK [Get list of volumes] *****************************************************************************************************************************************************************************************************************************************
ok: [localhost]
TASK [debug] *******************************************************************************************************************************************************************************************************************************************************
ok: [localhost] => {
"volumes": {
"Array_Software_Version": "2.0.0.0",
"Cluster": [
{
"id": "0",
"name": "111"
}
],
"Volumes": [
{
"id": "003ec9af-7a6d-473b-942f-3418da25d88c",
"name": "server1"
},
...
...
...
and my task
tasks:
- name: Get list of volumes
task: execute
register: volumes
- debug:
var: volumes
I'd like to extract just JSON payload, can I do this using Ansible?
Q: "Retrieve the actual JSON payload from ansible."
A: An option would be to use an intermediate file, for example
shell> cat test-280.yml
- hosts: localhost
become: true
vars:
volumes:
Array_Software_Version: "2.0.0.0"
Cluster:
- id: 0
name: 111
Volumes:
- id: 003ec9af-7a6d-473b-942f-3418da25d88c
name: server1
tasks:
- copy:
dest: /tmp/my_volumes.json
content: |
{{ {'volumes': volumes}|to_nice_json }}
shell> cat test-280.sh
#!/bin/bash
ansible-playbook test-280.yml > /dev/null
my_volumes=`cat /tmp/my_volumes.json`
echo $my_volumes
shell> ./test-280.sh
{ "volumes": { "Array_Software_Version": "2.0.0.0", "Cluster": [ { "id": 0, "name": 111 } ], "Volumes": [ { "id": "003ec9af-7a6d-473b-942f-3418da25d88c", "name": "server1" } ] } }

set_fact create a list with items

I have task which calls an API and I register the o/p in a varaible;
- name: Get Object storage account ID
uri:
url: 'https://api.softlayer.com/rest/v3.1/SoftLayer_Network_Storage_Hub_Cleversafe_Account/getAllObjects.json?objectFilter={"username":{"operation":"{{ item }}"}}'
method: GET
user: abxc
password: 66c94c447a6ed8a0cf058774fe38
validate_certs: no
register: old_existing_access_keys_sl
with_items: '{{ info["personal"].sl_cos_accounts }}'
old_existing_access_keys_sl holds:
"old_existing_access_keys_sl.results": [
{
"json": [
{
"accountId": 12345,
"id": 70825621,
"username": "xyz-11"
}
]
},
{
"json": [
{
"accountId": 12345,
"id": 70825621,
"username": "abc-12"
}
]
}
I want to make a list of id's for further processing an tried the following task but this did not work:
- name: Create a list of account ids
set_fact:
admin_usernames = "{{ item.json[0].id | list }}"
with_items: old_existing_access_keys_sl.results
I am not sure if that's even possible. I also tried this:
- name: create a list
set_fact:
foo: "{% set foo = [] %}{% for i in old_existing_access_keys_sl.results %}{{ foo.append(i) }}{% endfor %}"
foo always comes as blank and as a string:
TASK [result] *****************************************************************************************************************************************
ok: [localhost] => {
"foo": ""
}
Given your example data, you can extract a list of ids using the json_query filter, like this:
---
- hosts: localhost
gather_facts: false
vars:
old_existing_access_keys_sl:
results:
[
{
"json": [
{
"accountId": 12345,
"id": 70825621,
"username": "xyz-11"
}
]
},
{
"json": [
{
"accountId": 12345,
"id": 70825621,
"username": "abc-12"
}
]
}
]
tasks:
- debug:
var: old_existing_access_keys_sl|json_query('results[*].json[0].id')
This will output:
TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
"old_existing_access_keys_sl|json_query('results[*].json[0].id')": [
70825621,
70825621
]
}
If you want to store these in a new variable, you can replace that debug task with set_fact:
- set_fact:
admin_ids: "{{ old_existing_access_keys_sl|json_query('results[*].json[0].id') }}"
Update
For a list of dictionaries, just change the json_query expression:
- debug:
var: "old_existing_access_keys_sl|json_query('results[*].json[0].{id: id, username: username}')"
For more information, see the jmespath website for documentation and examples.

Ansible: Trying to parse out Loopback0 IP Address

I am trying to store the IP address of my loopback0 as variable. I can call the ipv4 information, but it doesn't let me call the address information. [Scrubbed Info for Privacy]
Code:
- name: Configure IPSLA on Americas Router
gather_facts: false
hosts: IP_SLA
connection: local
serial: 1
tasks:
- name: Gather Switch Info
ios_facts:
- name: Debug
debug:
var: ansible_facts["net_interfaces"]["Loopback0"]["ipv4"]
...
Output:
PLAY [Configure IPSLA on Americas Router] ***************************************************************************************************************************************************************************************************
TASK [Gather Switch Info] *******************************************************************************************************************************************************************************************************************
ok: [host] => {"ansible_facts": {"ansible_net_interfaces": {"Loopback0": {"bandwidth": 8000000, "description": null, "duplex": null, "ipv4": [{"address": "10.x.x.x", "subnet": "32"}], "lineprotocol": "up ", "macaddress": null, "mediatype": null, "mtu": 1514, "operstatus": "up", "type": null}
TASK [Debug] ********************************************************************************************************************************************************************************************************************************
ok: [host] => {
"ansible_facts[\"net_interfaces\"][\"Loopback0\"][\"ipv4\"]": [
{
"address": "10.x.x.x",
"subnet": "32"
}
]
}
But when I try and call the address:
- name: Debug
debug:
var: ansible_facts["net_interfaces"]["Loopback0"]["ipv4"]["address"]
I end up with this error:
TASK [Debug] ****************************************************************************************************************************************
ok: [host] => {
"ansible_facts[\"net_interfaces\"][\"Loopback0\"][\"ipv4\"][\"address\"]": "VARIABLE IS NOT DEFINED!: 'list object' has no attribute 'address'"
How do I get the address stored as a variable so I can use it in future tasks?
The square brackets in:
ok: [host] => {
"ansible_facts[\"net_interfaces\"][\"Loopback0\"][\"ipv4\"]": [
{
"address": "10.x.x.x",
"subnet": "32"
}
]
}
Indicate this ["net_interfaces"]["Loopback0"]["ipv4"] is an array with a single element.
Use [0] to indicate the first element of the array, as demonstrated by the following code:
ansible_facts["net_interfaces"]["Loopback0"]["ipv4"][0]["address"]

Resources