Using url module with jinja2 templates - ansible

I know how to process jinja2 templates files and let them create files. I also know how to POST to webservices using the url module.
For now I use some code like this, which successfully posts hardcoded JSON to my remote service:
tasks:
- name: GSA app definition
uri:
url: "http://localhost:8764/api/apps?relatedObjects=false"
method: POST
force_basic_auth: yes
user: "{{ admin_name }}"
password: "{{ admin_pass }}"
body_format: json
body: "{\"name\":\"My new app\", \"description\":\"A really great new app\" }"
follow_redirects: all
status_code: 200
timeout: 15
register: app_gsa_cfg
But the JSON is static, how can I process a jinja2 template and POST its content ? I would prefer not having to create temporary files on disk and POST them, what I am looking for is a direct connection or perhaps an approach that puts the template processing result into a string.
For starters a jinja2 template could look like this, later I will add variables too:
{#
This file creates the basic GSA app in Fusion. See https://doc.lucidworks.com/fusion-server/4.2/reference-guides/api/apps-api.html#create-a-new-app for details
#}
{
"name": "GSA",
"description": "Contains all configuration specific to the migrated GSA legacy searches"
}
(I know that this has little advantage over a static json included into the playbook. But is is easier to edit and offers me the opportunity to have (jinja style) comments in Json, which is normally not possible)

In my case, what I do is the following:
I have an API, so I do the following:
- name: Change API Status
uri:
url: "{{ enpoint }}/v1/requests/{{ whatever }}"
method: PATCH
user: "{{ tokenid }}"
password: x
headers:
X-4me-Account: "myaccount"
body: '{ "status":"{{ reqstatus }}" }'
body_format: json
status_code:
- 201
- 200
force_basic_auth: true
validate_certs: false
return_content: true
Then your reqstatus var will change.
Even you can add your whole text as yaml, import into a variable and convert with filters {{ some_variable | to_json }}
Note: Have a look to the formatting without escaping quotes. That will help.
It makes no sense creating a file with jinja2 if you are not going to copy it remotely. Ansible supports jinja natively but its strength is the possibility to have plugins for better maintainability. There is no difference between template (or win_template) modules unless (as said) you copy the file somewhere. Look this example:
---
- name: Adhoc Jinja
hosts: localhost
connection: local
gather_facts: false
vars:
mytemplate:
- name: "GSA"
description: "Contains all configuration specific to the migrated GSA legacy searches"
- name: "Another Name"
description: "Contains Another Var"
tasks:
- name: Read Vars Loop
debug:
msg: "{{ item | to_json }}"
with_items: "{{ mytemplate }}"
- name: Include Vars
include_vars: adhocjinja2.yml
- name: Read Vars Loop
debug:
msg: "{{ item | to_json }}"
with_items: "{{ mytemplate }}"
And adhocjinja2.yml:
mytemplate:
- name: "GSA2"
description: "Contains all configuration specific to the migrated GSA legacy searches"
- name: "Another Name 2"
description: "Contains Another Var"
The output is:
TASK [Read Vars Loop] **************************************************************************************
ok: [localhost] => (item={'name': 'GSA', 'description': 'Contains all configuration specific to the migrated GSA legacy searches'}) => {
"msg": "{\"name\": \"GSA\", \"description\": \"Contains all configuration specific to the migrated GSA legacy searches\"}"
}
ok: [localhost] => (item={'name': 'Another Name', 'description': 'Contains Another Var'}) => {
"msg": "{\"name\": \"Another Name\", \"description\": \"Contains Another Var\"}"
}
TASK [Include Vars] ****************************************************************************************
ok: [localhost]
TASK [Read Vars Loop] **************************************************************************************
ok: [localhost] => (item={'name': 'GSA2', 'description': 'Contains all configuration specific to the migrated GSA legacy searches'}) => {
"msg": "{\"name\": \"GSA2\", \"description\": \"Contains all configuration specific to the migrated GSA legacy searches\"}"
}
ok: [localhost] => (item={'name': 'Another Name 2', 'description': 'Contains Another Var'}) => {
"msg": "{\"name\": \"Another Name 2\", \"description\": \"Contains Another Var\"}"
}
You can manage your variables as you want and create your json on the fly as Ansible has jinja and json it its heart.

Related

Ansible template error while templating string : unexpected char '#' in email

I'm using Ansible to get a list of user emails from an API and I want to loop over them.
This is the json response I get from the API:
"users": [
{
"email": "email1#email.com",
"id": 1,
"is_admin": true
},
{
"email": "email2#email.com",
"id": 2,
"is_admin": false
},
]
edit:
the task after that which I need the email for:
- name: Send emails
register: result
uri:
url: http://api
method: GET
body_format: json
return_content: yes
body:
email: "{{ item.email }}"
scope: SCOPE
loop: "{{ users.json['users'] }}"
- name: Print result
debug:
var: result
the error I get:
fatal: [localhost]: FAILED! => {"msg": "template error while templating string: unexpected char '#' at 16. String: {{ email#email.com }}"}
If I use email: item.email the json request body will be "body": {"email": "item.email"} instead of the email value
How Can I get the full email of each user?
You need to remove the {{ and }} markers from your var: directive. The value of var is implicitly evaluated in a Jinja template context so you don't need those markers:
- name: Print returned json dictionary
debug:
var: item.email
loop: "{{ users.json['users'] }}"
(Updated based on edits to your question)
The example you've shown doesn't generate the error you're asking about. Here's a complete reproducer with just the uri changed:
- hosts: localhost
gather_facts: false
vars:
users:
json:
users:
- email: email1#email.com
id: 1
is_admin: true
- email: email2#email.com
id: 2
is_admin: false
tasks:
- name: Send emails
register: result
uri:
url: https://eny1tdcqj6ghp.x.pipedream.net
method: GET
body_format: json
return_content: true
body:
email: "{{ item.email }}"
scope: SCOPE
loop: "{{ users.json['users'] }}"
This runs without errors (although note that I'm suspicious about the
use of a GET request with a JSON body; typically you would expect
that to be a POST request).
You can see the result of running this playbook here, which shows that the values being sent in the request are exactly what we expect from the data.

How to loop over a list of dict elements deep 3 in Ansible

I have the following variables in my var file:
repo_type:
hosted:
data:
- name: hosted_repo1
online: true
storage:
blobstarage: default
write_policy: allow_once
- name: hosted_repo2
online: true
storage:
blobstarage: default
write_policy: allow_once
proxy:
data:
- name: proxy_repo1
online: true
storage:
blobstarage: default
write_policy: allow_once
- name: proxy_repo2
online: true
storage:
blobstarage: default
write_policy: allow_once
group:
data:
- name: group_repo1
online: true
storage:
blobstarage: default
write_policy: allow_once
- name: group_repo2
online: true
storage:
blobstarage: default
write_policy: allow_once
I want to configure a task to to loop over (hosted,proxy and group) and body over data dict.
Here is the task:
- name: Create pypi hosted Repos
uri:
url: "{{ nexus_api_scheme }}://{{ nexus_api_hostname }}:{{ nexus_api_port }}\
{{ nexus_api_context_path }}{{ nexus_rest_api_endpoint }}/repositories/pypi/{{ item.key}}"
user: "{{ nexus_api_user }}"
password: "{{ nexus_default_admin_password }}"
headers:
accept: "application/json"
Content-Type: "application/json"
body_format: json
method: POST
force_basic_auth: yes
validate_certs: "{{ nexus_api_validate_certs }}"
body: "{{ item }}"
status_code: 201
no_log: no
with_dict: "{{ repo_type}}"
I have tried with_items, with_dict and with_nested but nothing helped.
The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'data'
Any help would be appreciated!
If your goal is to loop over the contents of the data keys as a flat list, you could do it like this:
- debug:
msg: "repo {{ item.name }} write_policy {{ item.storage.write_policy }}"
loop_control:
label: "{{ item.name }}"
loop: "{{ repo_type | json_query('*.data[]') }}"
That uses a JMESPath expression to get the data key from each
top-level dictionary, and then flatten the resulting nested list. In
other words, it transforms you original structure into:
- name: hosted_repo1
online: true
storage:
blobstarage: default
write_policy: allow_once
- name: hosted_repo2
online: true
storage:
blobstarage: default
write_policy: allow_once
- name: proxy_repo1
online: true
storage:
blobstarage: default
write_policy: allow_once
- name: proxy_repo2
online: true
storage:
blobstarage: default
write_policy: allow_once
- name: group_repo1
online: true
storage:
blobstarage: default
write_policy: allow_once
- name: group_repo2
online: true
storage:
blobstarage: default
write_policy: allow_once
When run using your example data, this produces as output:
TASK [debug] *********************************************************************************************************
ok: [localhost] => (item=hosted_repo1) => {
"msg": "repo hosted_repo1 write_policy allow_once"
}
ok: [localhost] => (item=hosted_repo2) => {
"msg": "repo hosted_repo2 write_policy allow_once"
}
ok: [localhost] => (item=proxy_repo1) => {
"msg": "repo proxy_repo1 write_policy allow_once"
}
ok: [localhost] => (item=proxy_repo2) => {
"msg": "repo proxy_repo2 write_policy allow_once"
}
ok: [localhost] => (item=group_repo1) => {
"msg": "repo group_repo1 write_policy allow_once"
}
ok: [localhost] => (item=group_repo2) => {
"msg": "repo group_repo2 write_policy allow_once"
}
If you're trying to do something else, please update your question so
that it clearly shows the values you expect for each iteration of your
loop.
As reported by #larsk, you actually didn't manage to explain clearly how you are trying to loop over your data and what your api call is actually expecting.
But you are in luck this time since I have messed around with Nexus quite a bit (and I think I actually recognize those variable names and overall task layout)
The Nexus Repository POST /v1/repositories/pypi/[hosted|proxy|group] API endpoints are expecting one call for each of your repos in data. To implement your requirement you need to loop over the keys in repo_type to select the appropriate endpoint and then loop again over each element in data to send the repo definition to create.
This is actually possible combining a dict2items and subelements filters in your loop as in the below playbook (not directly tested).
The basics of the transformation are as follow:
transform your dict to a key/value list using dict2items e.g. (shortened example)
- key: hosted
value:
data:
- <repo definition 1>
- <repo definition 2>
[...]
use the subelements filter to combine each top element with each element in value.data e.g.:
- # <- This is the entry for first repo i.e. `item` in your loop
- key: hosted # <- this is the start of the corresponding top element i.e. `item.0` in your loop
value:
data:
- <repo definition 1>
- <repo definition 2>
- <repo definition 1> # <- this is the sub-element i.e. `item.1` in your loop
- # <- this is the entry for second repo
- key: hosted
value:
data:
- <repo definition 1>
- <repo definition 2>
- <repo definition 2>
[...]
Following one of your comments and from my experience, I added to my example an explicit json serialization of the repo definition using the to_json filter.
Putting it all together this gives:
- name: Create pypi hosted Repos
uri:
url: "{{ nexus_api_scheme }}://{{ nexus_api_hostname }}:{{ nexus_api_port }}\
{{ nexus_api_context_path }}{{ nexus_rest_api_endpoint }}/repositories/pypi/{{ item.0.key }}"
user: "{{ nexus_api_user }}"
password: "{{ nexus_default_admin_password }}"
headers:
accept: "application/json"
Content-Type: "application/json"
body_format: json
method: POST
force_basic_auth: yes
validate_certs: "{{ nexus_api_validate_certs }}"
body: "{{ item.1 | to_json }}"
status_code: 201
no_log: no
loop: "{{ repo_type | dict2items | subelements('value.data') }}"

Ansible error: with_items expects a list or a set

Here is the Ansible source code where with_items is linked to a dictionary type variable named "grafana_datasource_body". I always get an error message "ith_items expects a list or a set". Here is the source code and related output.
Ansible source code :
- name: Configure datasource creation http body
set_fact:
grafana_datasource_body:
name: "{{Ass_grafana_Datasource_Name}}"
type: "{{Ass_grafana_Datasource_Type}}"
isDefault: "{{Ass_grafana_Datasource_IsDefault}}"
access: "{{Ass_grafana_Datasource_Access}}"
basicAuth: "{{Ass_grafana_Datasource_BasicAuth}}"
url: "{{InfluxDB_Server}}:{{Ass_grafana_Datasource_Http_Port}}"
database: "{{Ass_grafana_Datasource_Db}}"
username: "{{Ass_grafana_Datasource_DbUser}}"
password: "{{Ass_grafana_Datasource_DbPassword}}"
when: Ass_grafana_Datasource_Name not in grafana_datasources_output.json|map(attribute='name')|list
- debug: msg="Datasource creation http body = {{grafana_datasource_body}}"
when: Ass_grafana_Datasource_Name not in grafana_datasources_output.json|map(attribute='name')|list
# Create non existing datasources
- name: datasource > Create datasource
register: grafana_datasources_create_output
failed_when: "'Datasource added' not in grafana_datasources_create_output|to_json"
uri:
url: "http://127.0.0.1:{{Ass_grafana_Listen_Http_Port}}/api/datasources"
method: POST
HEADER_Content-Type: application/json
body: '{{ item|to_json }}'
force_basic_auth: true
user: "{{Ass_grafana_Api_User}}"
password: "{{Ass_grafana_Api_Password}}"
status_code: 200
with_items: "{{grafana_datasource_body}}"
when: item.name not in grafana_datasources_output.json|map(attribute='name')|list
Output
TASK: [./grafana/grafana_Role | Configure datasource creation http body] ******
<10.0.20.7> ESTABLISH CONNECTION FOR USER: centos
ok: [10.0.20.7] => {"ansible_facts": {"grafana_datasource_body": {"access": "proxy", "basicAuth": "false", "database": "webcom-int", "isDefault": "true", "name": "MonTonton", "password": "webcom", "type": "influxdb", "url": "http://localhost:8086", "username": "webcom"}}}
TASK: [./grafana/grafana_Role | debug msg="Datasource creation http body = {{grafana_datasource_body}}"] ***
<10.0.20.7> ESTABLISH CONNECTION FOR USER: centos
ok: [10.0.20.7] => {
"msg": "Datasource creation http body = {'username': u'webcom', 'name': u'MonTonton', 'database': u'webcom-int', 'url': u'http://localhost:8086', 'basicAuth': u'false', 'access': u'proxy', 'password': u'webcom', 'type': u'influxdb', 'isDefault': u'true'}"
}
TASK: [./grafana/grafana_Role | datasource > Create datasource] ***************
fatal: [10.0.20.7] => with_items expects a list or a set
FATAL: all hosts have already failed -- aborting
Any real reason to use with_items loop here?
Replace
with_items: "{{grafana_datasource_body}}"
with
with_items: "{{ [grafana_datasource_body] }}"
this will work.
But you can use body: '{{ grafana_datasource_body|to_json }}' and don't use with_items.

How to Parse the XML response of a URI in ansible

There is a code which call the web services of an application.
- uri:
url: http://10.164.52.61:8080/ems/v74/ws/customer.ws?customerRefId=f4XXXb15d69c3
method: GET
content_as_json: true
password: admin
user: admin
validate_certs: no
return_content: yes
HEADER_Cookie: "{{login.set_cookie}}"
register: customerId
- debug:
var: customerId.content
The sample response is
"customerId.content": "<listResponse type=\"customer\" count=\"1\"><instance customerId=\"28\" name=\"abc\" customerRefId=\"xyz\" refId1=\"12\" type=\"org\" enabled=\"true\" phone=\"\" fax=\"\" billingZip=\"\" billingAddress=\"\" billingCity=\"\" billingCountry=\"\" billingState=\"\" vendor=\"1\" defaultEmail=\"test\" defaultContactName=\"test\"/></listResponse>"
I want to access the list response in my next block of code. Like i need just the value of "customerId". How this can be achieved using anisble
This is similar to sebaszw's answer, however doesn't require the xml response to be written to a file, instead storing it in a variable.
- uri:
url: http://10.164.52.61:8080/ems/v74/ws/customer.ws?customerRefId=f4XXXb15d69c3
method: GET
content_as_json: true
password: admin
user: admin
validate_certs: no
return_content: yes
HEADER_Cookie: "{{login.set_cookie}}"
register: customerId
- xml:
xmlstring: "{{customerId.content}}"
xpath: /listResponse/instance
content: attribute
register: instance_attributes
- debug:
var: instance_attributes.matches.0.instance.customerId
There is no XML support in Ansible out of the box.
You can use regex filters to make some simple searches in your XML data.
In your example:
- debug: msg="{{ customerId.content | regex_findall('customerId=\"(\d+)\"') }}"
Will search for customerId=\"<number>\" strings and return a list of numbers in quotes.
Update: fully working example:
- hosts: localhost
gather_facts: no
vars:
myvar: "<listResponse type=\"customer\" count=\"1\"><instance customerId=\"28\" name=\"abc\" customerRefId=\"xyz\" refId1=\"12\" type=\"org\" enabled=\"true\" phone=\"\" fax=\"\" billingZip=\"\" billingAddress=\"\" billingCity=\"\" billingCountry=\"\" billingState=\"\" vendor=\"1\" defaultEmail=\"test\" defaultContactName=\"test\"/></listResponse>"
tasks:
- debug: msg="{{ myvar | regex_findall('customerId=\"(\d+)\"') }}"
I use ansible 2.1.1.0.
From Ansible 2.4 you can do this:
- get_url:
url: http://10.164.52.61:8080/ems/v74/ws/customer.ws?customerRefId=f4XXXb15d69c3
dest: ./response.xml
url_password: admin
url_user: admin
validate_certs: no
headers:
Cookie: "{{login.set_cookie}}"
- xml:
path: ./response.xml
xpath: /listResponse/instance
content: attribute
register: instance_attributes
- debug:
var: instance_attributes.matches.0.instance.customerId
I just added XML parsing support in the uri module because I needed it too.
https://github.com/ansible/ansible/pull/53045
Just like the JSON support, it will return an 'xml' key with a dictionary consisting of the XML content for the convenience of accessing the data in the payload.
Your example would look like this:
- uri:
url: http://10.164.52.61:8080/ems/v74/ws/customer.ws?customerRefId=f4XXXb15d69c3
method: GET
password: admin
user: admin
validate_certs: no
HEADER_Cookie: "{{login.set_cookie}}"
register: customerId
- debug:
var: customerId.xml.listResponse.instance['#customerId']
The output in customerId.xml would be:
{
'listResponse': {
'#type': 'customer',
'#count', '1',
'instance': {
'#customerId': '28',
'#name': 'abc'
'#customerRefId': 'xyz',
'#refId1': '12',
'#type': 'org',
'#enabled': 'true',
'#phone': '',
'#fax': '',
'#billingZip': '',
'#billingAddress': '',
'#billingCity': '',
'#billingCountry': '',
'#billingState': '',
'#vendor': '1',
'#defaultEmail': 'test',
'#defaultContactName': 'test'
}
}
}
Its worth looking at ansible-xml for this. It's not part of Ansible but its a super useful module for parsing xml, it uses the lxml Python library under the hood.
As this was reasonably complex getting this to work I thought I'd post my Ansible - XML parameters for parsing a SOAP response:
<S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">
<SOAP-ENV:Header/>
<S:Body xmlns:ns2="http://ws.coverity.com/v9">
<ns2:getLdapServerDomainsResponse>
<return>
<name>soapui_test</name>
</return>
</ns2:getLdapServerDomainsResponse>
</S:Body>
</S:Envelope>
The trick was getting namespaces and xpath right. Also as I wanted the actual name I needed content: text
- name: Extract the name from the SOAP
xml:
xmlstring: "{{ ldap_response.content }}"
xpath: "/S:Envelope/S:Body/ns2:getLdapServerDomainsResponse/return/name"
namespaces:
S: "http://schemas.xmlsoap.org/soap/envelope/"
SOAP-ENV: "http://schemas.xmlsoap.org/soap/envelope/"
ns2: "http://ws.coverity.com/v9"
content: text
register: ldap_name
I still don't understand why the xpath section at the end doesn't work with .../ns2:return/ns2:name as the Ansible - XML docs example suggests. The above fills ldap_name with (simplified):
"ldap_name": {
"matches": [
{
"name": "soapui_test"
}
],
}

Post Json to API via Ansible

I want to make a POST request to an API endpoint via Ansible where some of the items inside the post data are dynamic, here is what I try and fail:
My body_content.json:
{
apiKey: '{{ KEY_FROM_VARS }}',
data1: 'foo',
data2: 'bar'
}
And here is my Ansible task:
# Create an item via API
- uri: url="http://www.myapi.com/create"
method=POST return_content=yes HEADER_Content-Type="application/json"
body="{{ lookup('file','create_body.json') | to_json }}"
Sadly this doesn't work:
failed: [localhost] => {"failed": true}
msg: this module requires key=value arguments
....
FATAL: all hosts have already failed -- aborting
My ansible version is 1.9.1
You can't use newlines like this in yaml. Try this instead (the ">" indicates that the next lines are to be concatenated):
# Create an item via API
- uri: >
url="http://www.myapi.com/create"
method=POST return_content=yes HEADER_Content-Type="application/json"
body="{{ lookup('file','create_body.json') | to_json }}"
But I find this much better:
# Create an item via API
- uri:
url: "http://www.myapi.com/create"
method: POST
return_content: yes
HEADER_Content-Type: "application/json"
body: "{{ lookup('file','create_body.json') | to_json }}"
I'm posting below what I ended up using for my usecase (Ansible 2.0). This is useful if your json payload is stated inline (and not in a file).
this task expects 204 as its success return code.
And since the body_format is json, the header is inferred automatically
- name: add user to virtual host
uri:
url: http://0.0.0.0:15672/api/permissions/{{ rabbit_virtualhost }}/{{ rabbit_username }}
method: PUT
user: "{{ rabbit_username }}"
password: "{{ rabbit_password }}"
return_content: yes
body: {"configure":".*","write":".*","read":".*"}
body_format: json
status_code: 204
it is basically equivalent to:
curl -i -u user:pass -H "content-type:application/json" -XPUT http://0.0.0.0:15672/api/permissions/my_vhost/my_user -d '{"configure":".*","write":".*","read":".*"}'

Resources