join multiple arrays in complex data structure with jmespath - ansible

I'm trying to transform NFS exports, described in complex data structure, to config option accepted by nfs-server daemon which later be used in ansible.
I have:
nfs_exports:
- path: /export/home
state: present
options:
- clients: "192.168.0.0/24"
permissions:
- "rw"
- "sync"
- "no_root_squash"
- "fsid=0"
- path: /export/public
state: present
options:
- clients: "192.168.0.0/24"
permissions:
- "rw"
- "sync"
- "root_squash"
- "fsid=0"
- clients: "*"
permissions:
- "ro"
- "async"
- "all_squash"
- "fsid=1"
which must become:
[
{
"options": "192.168.0.0/24(rw,sync,no_root_squash,fsid=0)",
"path": "/export/home",
"state": "present"
},
{
"options": "192.168.0.0/24(rw,sync,root_squash,fsid=0) *(ro,async,all_squash,fsid=1)",
"path": "/export/public",
"state": "present"
}
]
So far I was able, using {{ nfs_exports | json_query(query) }}
query: "[].{path:path,state:state,options:options.join(` `,[].join(``,[clients,`(`,join(`,`,permissions),`)`]))}"
get
{
"options": "192.168.0.0/24(rw,sync,no_root_squash,fsid=0)",
"path": "/export/home",
"state": "present"
},
{
"options": "192.168.0.0/24(rw,sync,root_squash,fsid=0)*(ro,async,all_squash,fsid=1)",
"path": "/export/public",
"state": "present"
}
It's probably simple but I can't get pass that last options join, space ' ' gets removed.
So if someone knows the correct query additional explanation will be much appreciated.

Given the query:
[].{ path: path, state: state, options: join(' ', options[].join('', [clients, '(', join(',', permissions), ')'])) }
On the JSON
{
"nfs_exports": [
{
"path": "/export/home",
"state": "present",
"options": [
{
"clients": "192.168.0.0/24",
"permissions": [
"rw",
"sync",
"no_root_squash",
"fsid=0"
]
}
]
},
{
"path": "/export/public",
"state": "present",
"options": [
{
"clients": "192.168.0.0/24",
"permissions": [
"rw",
"sync",
"root_squash",
"fsid=0"
]
},
{
"clients": "*",
"permissions": [
"ro",
"async",
"all_squash",
"fsid=1"
]
}
]
}
]
}
It would give you your expected output:
[
{
"path": "/export/home",
"state": "present",
"options": "192.168.0.0/24(rw,sync,no_root_squash,fsid=0)"
},
{
"path": "/export/public",
"state": "present",
"options": "192.168.0.0/24(rw,sync,root_squash,fsid=0) *(ro,async,all_squash,fsid=1)"
}
]
Please mind: the string litteral `` wont work on a space character string, because, as pointed in the documentation, it will be parsed as JSON:
A literal expression is an expression that allows arbitrary JSON objects to be specified
Source: https://jmespath.org/specification.html#literal-expressions
This is quite easy when you get to the point of:
[].{ path: path, state: state, options: options[].join('', [clients, '(', join(',', permissions), ')']) }
Which is something you seems to have achived, that gives
[
{
"path": "/export/home",
"state": "present",
"options": [
"192.168.0.0/24(rw,sync,no_root_squash,fsid=0)"
]
},
{
"path": "/export/public",
"state": "present",
"options": [
"192.168.0.0/24(rw,sync,root_squash,fsid=0)",
"*(ro,async,all_squash,fsid=1)"
]
}
]
Because you are just left with joining the whole array in options with a space as glue character.

Related

Ansible search and query

Updated with suggestions from larsks.
With the following structure
"intf_output_ios": {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python3"
},
"changed": false,
"failed": false,
"gathered": [
{
"name": "GigabitEthernet0/0"
},
{
"mode": "trunk",
"name": "GigabitEthernet0/1",
"trunk": {
"allowed_vlans": [
"10",
"20",
"30",
"99",
"100"
],
"encapsulation": "dot1q"
}
},
{
"mode": "trunk",
"name": "GigabitEthernet0/2",
"trunk": {
"allowed_vlans": [
"10",
"20",
"30",
"99",
"100"
],
"encapsulation": "dot1q"
}
},
{
"access": {
"vlan": 30
},
"mode": "access",
"name": "GigabitEthernet0/3"
},
{
"name": "GigabitEthernet1/0"
},
{
"name": "GigabitEthernet1/1"
},
{
"name": "GigabitEthernet1/2"
},
{
"name": "GigabitEthernet1/3"
},
{
"name": "GigabitEthernet2/0"
},
{
"name": "GigabitEthernet2/1"
},
{
"name": "GigabitEthernet2/2"
},
{
"name": "GigabitEthernet2/3"
},
{
"name": "GigabitEthernet3/0"
},
{
"name": "GigabitEthernet3/1"
},
{
"name": "GigabitEthernet3/2"
},
{
"access": {
"vlan": 99
},
"mode": "access",
"name": "GigabitEthernet3/3"
}
]
}
To print only the ports in VLAN 30 use the following?
- name: "P901T6: Set fact to include only access ports - IOS"
set_fact:
access_ports_ios_2: "{{ intf_output_ios | json_query(query) }}"
vars:
query: >-
gathered[?access.vlan==`30`]
- name: "P901T7: Dump list of access ports - IOS"
debug:
var=access_ports_ios_2
NOTE: It is important to use 30 (with backticks) and not '30'
I have gone through https://docs.ansible.com/ansible/latest/user_guide/playbooks_filters.html#managing-list-variables without really understanding how to fix this. If someone has some good link that would be very useful
With a structure like
ok: [access01] => {
"access_ports_ios": [
{
"access": {
"vlan": 30
},
"mode": "access",
"name": "GigabitEthernet0/3"
},
{
"access": {
"vlan": 99
},
"mode": "access",
"name": "GigabitEthernet3/3"
}
]
}
To get ports in vlan 30 use:
- debug:
var: access_ports_ios|json_query(query)
vars:
query: >-
[?access.vlan==`30`]
Note:
If you want to use a variable for vlan instead of hard-coding it. I had to do as follows:
- name: Debug 4
debug:
var: access_ports_ios|json_query('[?access.vlan==`{{ src_vlan | int}}`]')
You're asking for gathered.access, but gathered is a list and does not have an access attribute. You want "all items from gathered for which access.vlan is 30 (and note that the value of access.vlan is an integer, not a string):
- debug:
var: intf_output_ios|json_query(query)
vars:
query: >-
gathered[?access.vlan==`30`]
Which given you example input produces:
TASK [debug] *******************************************************************
ok: [localhost] => {
"intf_output_ios|json_query(query)": [
{
"access": {
"vlan": 30
},
"mode": "access",
"name": "GigabitEthernet0/3"
}
]
}
I'm going to reiterate advice I often give for json_query questions: use something like jpterm or the JMESPath website to test JMESPath expressions against your actual data. This makes it much easier to figure out where an expression might be going wrong.

Heroku: There was an issue setting up your app environment. Name is invalid

When I try to use this app.json:
{
"stack": "heroku-18",
"repository": "https://github.com/OpenHumans/oh-data-source-template",
"logo": "https://avatars.githubusercontent.com/u/3341265?s=280&v=4",
"scripts": {
"postdeploy": "python manage.py init_proj_config"
},
"env": {
"SECRET_KEY": {
"description": "This is set for you and is used to encrypt data.",
"generator": "secret"
},
"OH_CLIENT_ID": {
"description": "See http://openhumans.org/direct-sharing/projects/manage",
"value": ""
},
"OH_CLIENT_SECRET": {
"description": "See http://openhumans.org/direct-sharing/projects/manage",
"value": ""
},
"OH_ACTIVITY_PAGE": {
"description": "See http://openhumans.org/direct-sharing/projects/manage",
"value": ""
},
"APP_BASE_URL ": {
"description": "e.g. https://your-app-name.herokuapp.com - no trailing slash!",
"value": "https://your-app-name.herokuapp.com"
},
"DEBUG": {
"description": "Displays detailed error info for web requests. Set False in production.",
"value": "false"
},
"HEROKU_APP": {
"description": "If true, ALLOWED_HOSTS is set to all.",
"value": "true"
}
},
"addons": [
"coralogix:free-30mbday",
"heroku-redis:hobby-dev",
{
"plan": "heroku-postgresql",
"options": {
"version": "9.5"
}
}
]
}
to deploy from:
https://dashboard.heroku.com/new?template=https%3A%2F%2Fgithub.com%2Fmikepsinn%2Foh-quantimodo-source
I get the error no matter what I set the name to:
I've tried adding the optional name field to the app.json but that doesn't help either.

Remove double quotes from Ansible fact

There is a json output which I am trying to parse. I registered the output into variable named instance_ip.
Here is the json output:
{
"msg": {
"instances": [
{
"root_device_type": "ebs",
"private_dns_name": "",
"cpu_options": {
"core_count": 2,
"threads_per_core": 1
},
"security_groups": [],
"state_reason": {
"message": "Client.UserInitiatedShutdown: User initiated shutdown",
"code": "Client.UserInitiatedShutdown"
},
"monitoring": {
"state": "disabled"
},
"ebs_optimized": false,
"state": {
"code": 48,
"name": "terminated"
},
"client_token": "test-Logst-14O6L4IETB05E",
"virtualization_type": "hvm",
"architecture": "x86_64",
"tags": {
"sg:environment": "TST",
"Name": "logstash1",
"aws:cloudformation:logical-id": "Logstash1A1594E87",
"sg:owner": "Platforms#paparapa.com",
"aws:cloudformation:stack-name": "test-three-ec2-instances-elk-demo",
"elastic_role": "logstash",
"sg:function": "Storage"
},
"key_name": "AWS_key",
"image_id": "ami-09f765d333a8ebb4b",
"state_transition_reason": "User initiated (2021-01-31 09:46:23 GMT)",
"hibernation_options": {
"configured": false
},
"capacity_reservation_specification": {
"capacity_reservation_preference": "open"
},
"public_dns_name": "",
"block_device_mappings": [],
"metadata_options": {
"http_endpoint": "enabled",
"state": "pending",
"http_tokens": "optional",
"http_put_response_hop_limit": 1
},
"placement": {
"group_name": "",
"tenancy": "default",
"availability_zone": "ap-southeast-2a"
},
"enclave_options": {
"enabled": false
},
"ami_launch_index": 0,
"ena_support": true,
"network_interfaces": [],
"launch_time": "2021-01-31T09:44:51+00:00",
"instance_id": "i-0fa5dbb869833d7c6",
"instance_type": "t2.medium",
"root_device_name": "/dev/xvda",
"hypervisor": "xen",
"product_codes": []
},
{
"root_device_type": "ebs",
"private_dns_name": "ip-10-x-x-x.ap-southeast-2.compute.internal",
"cpu_options": {
"core_count": 2,
"threads_per_core": 1
},
"source_dest_check": true,
"monitoring": {
"state": "disabled"
},
"subnet_id": "subnet-0d5f856afab8f0eec",
"ebs_optimized": false,
"iam_instance_profile": {
"id": "AIPARWXXVHXJWC2FL4AI6",
"arn": "arn:aws:iam::instance-profile/test-three-ec2-instances-elk-demo-Logstash1InstanceProfileC3035819-1F2LI7JM16FVM"
},
"state": {
"code": 16,
"name": "running"
},
"security_groups": [
{
"group_id": "sg-0e5dffa834a036fab",
"group_name": "Ansible_sec_group"
}
],
"client_token": "test-Logst-8UF6RX33BH06",
"virtualization_type": "hvm",
"architecture": "x86_64",
"public_ip_address": "3.x.x.x",
"tags": {
"Name": "logstash1",
"aws:cloudformation:logical-id": "Logstash1A1594E87",
"srg:environment": "TST",
"aws:cloudformation:stack-id": "arn:aws:cloudformation:ap-southeast-2:117557247443:stack/test-three-ec2-instances-elk-demo/ca8ef2b0-63ad-11eb-805f-02630ffccc8c",
"sg:function": "Storage",
"aws:cloudformation:stack-name": "test-three-ec2-instances-elk-demo",
"elastic_role": "logstash",
"sg:owner": "Platforms#paparapa.com"
},
"key_name": "AWS_SRG_key",
"image_id": "ami-09f765d333a8ebb4b",
"ena_support": true,
"hibernation_options": {
"configured": false
},
"capacity_reservation_specification": {
"capacity_reservation_preference": "open"
},
"public_dns_name": "ec2-3-x-x-x.ap-southeast-2.compute.amazonaws.com",
"block_device_mappings": [
{
"device_name": "/dev/xvda",
"ebs": {
"status": "attached",
"delete_on_termination": true,
"attach_time": "2021-01-31T10:22:21+00:00",
"volume_id": "vol-058662934ffba3a68"
}
}
],
"metadata_options": {
"http_endpoint": "enabled",
"state": "applied",
"http_tokens": "optional",
"http_put_response_hop_limit": 1
},
"placement": {
"group_name": "",
"tenancy": "default",
"availability_zone": "ap-southeast-2a"
},
"enclave_options": {
"enabled": false
},
"ami_launch_index": 0,
"hypervisor": "xen",
"network_interfaces": [
{
"status": "in-use",
"description": "",
"subnet_id": "subnet-0d5f856afab8f0eec",
"source_dest_check": true,
"interface_type": "interface",
"ipv6_addresses": [],
"network_interface_id": "eni-09b045668ac59990c",
"private_dns_name": "ip-10-x-x-x.ap-southeast-2.compute.internal",
"attachment": {
"status": "attached",
"device_index": 0,
"attachment_id": "eni-attach-0700cd11dfb27e2dc",
"delete_on_termination": true,
"attach_time": "2021-01-31T10:22:20+00:00"
},
"private_ip_addresses": [
{
"private_ip_address": "10.x.x.x",
"private_dns_name": "ip-10-x-x-x.ap-southeast-2.compute.internal",
"association": {
"public_ip": "3.x.x.x",
"public_dns_name": "ec2-3-x-x-x.ap-southeast-2.compute.amazonaws.com",
"ip_owner_id": "amazon"
},
"primary": true
}
],
"mac_address": "02:d1:13:01:59:b2",
"private_ip_address": "10.x.x.x",
"vpc_id": "vpc-0016dcdf5abe4fef0",
"groups": [
{
"group_id": "sg-0e5dffa834a036fab",
"group_name": "Ansible_sec_group"
}
],
"association": {
"public_ip": "3.x.x.x",
"public_dns_name": "ec2-3-x-x-x.ap-southeast-2.compute.amazonaws.com",
"ip_owner_id": "amazon"
},
"owner_id": "117557247443"
}
],
"launch_time": "2021-01-31T10:22:20+00:00",
"instance_id": "i-0482bb8ca1bef6006",
"instance_type": "t2.medium",
"root_device_name": "/dev/xvda",
"state_transition_reason": "",
"private_ip_address": "10.x.x.x",
"vpc_id": "vpc-0016dcdf5abe4fef0",
"product_codes": []
}
],
"failed": false,
"changed": false
},
"_ansible_verbose_always": true,
"_ansible_no_log": false,
"changed": false
}
The goal is to get the private ip address and append the port number.
With the following task I got the list with node ip address ["10.x.x.x"]
- name: Getting EC2 instance ip address
set_fact:
instance_ip: "{{ logstash_instance | json_query('instances[*].network_interfaces[*].private_ip_address') | flatten }}"
With next task in a play I am trying to append the port number but I am keep getting
"['10.x.x.x:5044']"
- name: Get everything between quotes and append port 5044
set_fact:
logstash_hosts: "{{ instance_ip | map('regex_replace', '^(.*)$', '\\1:5044') | list }}"
Here is the template output:
# ------------------------------ Logstash Output -------------------------------
output.logstash:
hosts: "['10.x.x.x:5044']"
I need to get rid of the double quotes and pass the clean variable ['10.x.x.x:5044'] to my template file.
You can try creating a new list variable with the port number appended to each element, using this approach:
- set_fact:
logstash_hosts: "{{ logstash_hosts|default([]) + [ item ~ ':5044' ] }}"
with_items: "{{ instance_ip }}"
Then in template:
output.logstash:
hosts: {{ logstash_hosts|to_yaml }}
Also since the Logstash configuration is a YAML formatted file, you use YAML list syntax and directly use the instance_ip variable (and avoid set_fact). Then the template will look like this:
output.logstash:
hosts:
{% for ip in instance_ip %}
- {{ ip }}:5044
{% endfor %}

Ansible: How to modify a list of dicts

I want to modify dicts in a list within hostvars with a new entry for the IP address I get from the IPAM.
{
"vm_guest_networks": [
{
"device_type": "vmxnet3",
"state": "present",
"subnet": "10.91.1.0/24"
},
{
"device_type": "vmxnet3",
"state": "present",
"subnet": "10.91.0.0/24"
}
]
}
Within a loop I have the subnet to identify the right dict and the IP address I want to add with the ipv4_address key so the result should look like:
{
"vm_guest_networks": [
{
"device_type": "vmxnet3",
"state": "present",
"subnet": "10.91.1.0/24",
"ipv4_address": "10.91.1.216"
},
{
"device_type": "vmxnet3",
"state": "present",
"subnet": "10.91.0.0/24",
"ipv4_address": "10.91.0.21"
}
]
}
The current WIP ansible code is at https://pastebin.com/bFc1Ww2K
Let's assume the list of IPs is available. For example
ip4: [10.91.1.216, 10.91.0.21]
Let's use combine filter and Extended loop variables to create a new list where each dictionary will be updated. For example
- set_fact:
mydata: "{{ mydata|default([]) +
[item|combine({'ipv4_address': ip4[ansible_loop.index0]})] }}"
loop: "{{ vm_guest_networks }}"
loop_control:
extended: yes
- set_fact:
vm_guest_networks: "{{ mydata }}"
- debug:
var: vm_guest_networks
give
"vm_guest_networks": [
{
"device_type": "vmxnet3",
"ipv4_address": "10.91.1.216",
"state": "present",
"subnet": "10.91.1.0/24"
},
{
"device_type": "vmxnet3",
"ipv4_address": "10.91.0.21",
"state": "present",
"subnet": "10.91.0.0/24"
}
]

How to be make ansible regularly probing for a fact value and only proceed if the value changes?

I have a the following snippet where I install OS on a virtual machine using ansible, and after it finishes it stops the VM so I can continue the rest of the tasks, I am collecting facts from the red hat virtualization manager regarding the state the vm, and I want to keep waiting until the status of the VM changes from up to down so I can proceed, how can I code this?:
# I am kickstarting the VM
- name: Installing OS
ovirt_vms:
state: running
name: "{{ vm_name }}"
initrd_path: iso://initrd.img
kernel_path: iso://vmlinuz
kernel_params: initrd=initrd.img inst.stage2=cdrom inst.ks=ftp://10.0.1.2/pub/ks.cfg net.ifnames=0 biosdevname=0 BOOT_IMAGE=vmlinuz
# Getting facts about the VM
- name: Gather VM Status
ovirt_vms_facts:
pattern: name={{ vm_name}}
- name: Register VM Status
debug:
msg: "{{ ovirt_vms[0].status }}"
register: vm_status
#Should Keep probing the value of vm_status until it changes from up to down.
????????????? --> What should I do here?
#When Status change continue the work book
I tried to parse the ovirt_vms I gathered from ovirt_vms_facts, and I got the following:
{
"_ansible_parsed": true,
"invocation": {
"module_args": {
"all_content": false,
"pattern": "name=as-vm-type1",
"nested_attributes": [],
"case_sensitive": true,
"fetch_nested": false,
"max": null
}
},
"changed": false,
"_ansible_no_log": false,
"ansible_facts": {
"ovirt_vms": [
{
"disk_attachments": [],
"origin": "ovirt",
"sso": {
"methods": []
},
"affinity_labels": [],
"placement_policy": {
"affinity": "migratable"
},
"watchdogs": [],
"creation_time": "2018-07-15 13:54:10.565000+02:00",
"snapshots": [],
"graphics_consoles": [],
"cluster": {
"href": "/ovirt-engine/api/clusters/a5272863-38a8-469d-998e-c1e1f26f4f5a",
"id": "a5272863-38a8-469d-998e-c1e1f26f4f5a"
},
"href": "/ovirt-engine/api/vms/08406dad-5173-4241-8d42-904ddf3d096a",
"migration": {
"auto_converge": "inherit",
"compressed": "inherit"
},
"io": {
"threads": 0
},
"migration_downtime": -1,
"id": "08406dad-5173-4241-8d42-904ddf3d096a",
"high_availability": {
"priority": 0,
"enabled": false
},
"cdroms": [],
"statistics": [],
"usb": {
"enabled": false
},
"display": {
"allow_override": false,
"disconnect_action": "LOCK_SCREEN",
"file_transfer_enabled": true,
"copy_paste_enabled": true,
"secure_port": 5900,
"smartcard_enabled": false,
"single_qxl_pci": false,
"type": "spice",
"monitors": 1,
"address": "10.254.148.74"
},
"nics": [],
"tags": [],
"name": "as-vm-type1",
"bios": {
"boot_menu": {
"enabled": false
}
},
"stop_time": "2018-07-15 13:54:10.569000+02:00",
"template": {
"href": "/ovirt-engine/api/templates/00000000-0000-0000-0000-000000000000",
"id": "00000000-0000-0000-0000-000000000000"
},
"memory": 42949672960,
"type": "server",
"katello_errata": [],
"numa_tune_mode": "interleave",
"status": "up",
"next_run_configuration_exists": false,
"delete_protected": false,
"sessions": [],
"start_time": "2018-07-15 13:54:14.079000+02:00",
"quota": {
"id": "ad014a63-fd76-42da-8369-57dae2dd5979"
},
"applications": [],
"host": {
"href": "/ovirt-engine/api/hosts/56a65d3b-1c0a-4b2a-9c6c-aa96262d9502",
"id": "56a65d3b-1c0a-4b2a-9c6c-aa96262d9502"
},
"memory_policy": {
"max": 171798691840,
"guaranteed": 42949672960
},
"numa_nodes": [],
"permissions": [],
"stateless": false,
"reported_devices": [],
"large_icon": {
"href": "/ovirt-engine/api/icons/2971ddbe-1dbf-4af8-b86a-078cbbe66419",
"id": "2971ddbe-1dbf-4af8-b86a-078cbbe66419"
},
"storage_error_resume_behaviour": "auto_resume",
"cpu_profile": {
"href": "/ovirt-engine/api/cpuprofiles/34000c79-d669-41ef-8d2a-d37d7f925c3c",
"id": "34000c79-d669-41ef-8d2a-d37d7f925c3c"
},
"time_zone": {
"name": "Etc/GMT"
},
"run_once": true,
"original_template": {
"href": "/ovirt-engine/api/templates/00000000-0000-0000-0000-000000000000",
"id": "00000000-0000-0000-0000-000000000000"
},
"start_paused": false,
"host_devices": [],
"small_icon": {
"href": "/ovirt-engine/api/icons/28054380-4723-42db-a8e5-fed8a3778199",
"id": "28054380-4723-42db-a8e5-fed8a3778199"
},
"os": {
"boot": {
"devices": [
"hd",
"cdrom"
]
},
"type": "rhel_7x64"
},
"cpu": {
"architecture": "x86_64",
"topology": {
"cores": 1,
"threads": 1,
"sockets": 8
}
},
"cpu_shares": 1024
}
]
}
}
You can do it as follows:
- name: Wait for VMs to be down
ovirt_vms_facts:
auth: "{{ ovirt_auth }}"
pattern: "name={{ vm_name }}"
until: "ovirt_vms[0].status == 'down'"
retries: 5
delay: 10
You can do it as below::
- name: Register VM Status
debug:
msg: "{{ ovirt_vms[0].status }}"
register: vm_status
until: vm_status.stdout.find("down") != -1
retries: 10
delay: 5
Here it is retrying for 10 times with a delay of 5 seconds.

Resources