I have an Ansible play-book for working with EC2 instances. I'm using dynamic inventory (ec2.py) to get the group of instances that I want to work with (hosts: tag_Service_Foo). When I run it, it produces output like:
GATHERING FACTS ***************************************************************
ok: [54.149.9.198]
ok: [52.11.22.29]
ok: [52.11.0.3]
However, I can fetch the "Name" tag for a particular instance from Amazon (I do this and store it in a variable for use in a couple parts of the playbook).
Is there a way to get Ansible to use this string for the hostname when displaying progress? I'd like to see something more descriptive (since I don't have the IPs memorized):
GATHERING FACTS ***************************************************************
ok: [main-server]
ok: [extra-server]
ok: [my-cool-server]
The output of the ec2.py inventory script looks like this (truncated; it's very long).
{
"_meta": {
"hostvars": {
"54.149.9.198": {
"ec2__in_monitoring_element": false,
"ec2_ami_launch_index": "0",
"ec2_architecture": "x86_64",
"ec2_client_token": "xxx",
"ec2_dns_name": "xxx",
"ec2_ebs_optimized": false,
"ec2_eventsSet": "",
"ec2_group_name": "",
"ec2_hypervisor": "xen",
"ec2_id": "i-xxx",
"ec2_image_id": "ami-xxx",
"ec2_instance_type": "xxx",
"ec2_ip_address": "xxx",
"ec2_item": "",
"ec2_kernel": "",
"ec2_key_name": "xxx",
"ec2_launch_time": "xxx",
"ec2_monitored": xxx,
"ec2_monitoring": "",
"ec2_monitoring_state": "xxx",
"ec2_persistent": false,
"ec2_placement": "xxx",
"ec2_platform": "",
"ec2_previous_state": "",
"ec2_previous_state_code": 0,
"ec2_private_dns_name": "xxx",
"ec2_private_ip_address": "xxx",
"ec2_public_dns_name": "xxx",
"ec2_ramdisk": "",
"ec2_reason": "",
"ec2_region": "xxx",
"ec2_requester_id": "",
"ec2_root_device_name": "/dev/xvda",
"ec2_root_device_type": "ebs",
"ec2_security_group_ids": "xxx",
"ec2_security_group_names": "xxx",
"ec2_sourceDestCheck": "true",
"ec2_spot_instance_request_id": "",
"ec2_state": "running",
"ec2_state_code": 16,
"ec2_state_reason": "",
"ec2_subnet_id": "subnet-xxx",
"ec2_tag_Name": "main-server",
"ec2_tag_aws_autoscaling_groupName": "xxx",
"ec2_virtualization_type": "hvm",
"ec2_vpc_id": "vpc-xxx"
}
}
}
"tag_Service_Foo": [
"54.149.9.198",
"52.11.22.29",
"52.11.0.3"
],
}
What you need to do is create your own wrapper (say my_ec2.py) over the ec2.py that would post process the output. Idea is to use the behavioral hostvar ansible_ssh_host. You can use any language not only python. As long as it prints valid json on stdout you're good to go. Reference if needed.
It'll be a tiny bit of work. But hope the sudo code would help:
output_json_map = new map
for each group in <ec2_output>: # e.g. tag_Service_Foo, I think there would be another
# key in the output that contains list of group names.
for each ip_address in group:
hname = ec2_output._meta.hostvars.find(ip_address).find(ec2_tag_Name)
# Add new host to the group member list
output_json_map.add(key=group, value=hname)
copy all vars from ec2_output._meta.hostvars.<ip_address>
to output_json_map._meta.hostvars.<hname>
# Assign the IP address of this host to the ansible_ssh_host
# in hostvars for this host
output_json_map.add(key=_meta.hostvars.<hname>.ansible_ssh_host,
value=ip_address)
output_json_map.add(key=_meta.hostvars.find(ip_address).ansible_ssh_host,
value=ip_address)
print output_json_map to stdout
E.g. for your example the output of my_ec2.py should be:
{
"_meta": {
"hostvars": {
"main-server": {
"ansible_ssh_host": "54.149.9.198"
--- snip ---
"ec2_tag_Name": "main-server",
--- snip ---
},
"extra-server": {
"ansible_ssh_host": "52.11.22.29"
--- snip ---
"ec2_tag_Name": "extra-server",
--- snip ---
},
<other hosts from all groups>
}
}
"tag_Service_Foo": [
"main-server",
"extra-server",
<other hosts in this group>
],
"some other group": [
<hosts in this group>,
...
],
}
and obviously, use this my_ec2.py instead of ec2.py as the inventory file. :-)
-- edit --
1) In the groups, can I only refer to things by one name? 2) There's
no notion of an alias? 3) I'm wondering if I could use the IP addr in
the groups and just modify the _meta part or if I need to do it all?
Yes*, No and no.
* Technically first yes should be no. Let me explain.
What we are doing here can be done with static inventory file like this:
Original ec2.py was returning json equivalent of following inventory file:
[tag_Service_Foo]
54.149.9.198 ec2_tag_Name="main-server" ec2_previous_state_code="0" ...
52.11.22.29 ec2_tag_Name="extra-server" ec2_previous_state_code="0" ...
our new my_ec2.py returns this:
[tag_Service_Foo]
main-server ansible_ssh_host="54.149.9.198" ec2_tag_Name="main-server" ec2_previous_state_code="0" ...
extra-server ansible_ssh_host="52.11.22.29" ec2_tag_Name="extra-server" ec2_previous_state_code="0" ...
# Technically it's possible to create "an alias" for main-server like this:
main-server-alias ansible_ssh_host="54.149.9.198" ec2_tag_Name="main-server" ec2_previous_state_code="0" ...
Now you would be able to run a play with main-server-alias in the host list and ansible would execute it on 54.149.9.198.
BUT, and this is a big BUT, when you run a play with 'all' as the host pattern ansible would run the task on main-server-alias as well as main-server. So what you created is an alias in one context and a new host in another. I've not tested this BUT part so do come back and correct me if you find out otherwise.
HTH
If you put
vpc_destination_variable = Name
in your ec2.ini file, that should work too.
For a more recent version, working since version 2.9 of Ansible, at least, the aws_ec2 plugin now allows this as a simple configuration, without the need to create any warper around ec2.py.
This can be done using a combination of the parameters hostnames and compose.
So the trick is to change the name of the EC2 instance via the hostnames parameter to have something human readable — for example from an AWS tag on the instance — and then to use the compose parameter to set the ansible_host to the private_ip_address, public_ip_address, private_dns_name or the public_dns_name to allow the connection, still.
Here would be the minimal configuration of the plugin to allow this:
plugin: aws_ec2
hostnames:
- tag:Name
# ^-- Given that you indeed have a tag named "Name" on your EC2 instances
compose:
ansible_host: public_dns_address
Mind, that, in AWS, the tags are not uniques, so you can have multiple instances tagged with the same name, but in Ansible, the hostname is something unique. This means that you will probably be better of composing the name with the tag:Name postfixing with something truly unique, like the instance ID.
Something like:
plugin: aws_ec2
hostnames:
- name: 'instance-id'
separator: '_'
prefix: 'tag:Name'
compose:
ansible_host: public_dns_address
Related
I am trying to extract specific value from kv2 hashicorp vault in ansible playbook using hashi_vault module
- name: Return specific value from vault
ansible.builtin.set_fact:
secret: "{{ lookup('hashi_vault', 'secret=my.secrets/data/dev/heslo:value token=vault-plaintext-root-token url=http://10.47.0.235:8200/')}}" register: secret
I am getting
{"msg": ""An unhandled exception occurred while running the lookup plugin 'hashi_vault'. Error was a <class 'ansible.errors.AnsibleError'>, original message: The secret my.secrets/data/dev/heslo doesn't seem to exist for hashi_vault lookup"}
Query works for all of the secrets in path using
secret=my.secrets/data/dev/
"heslo" record exists in the path
"ansible_facts": {
"secret": {
"data": {
"heslo": "heslo",
"password": "test",
"username": "ahoj"
},
Thank you in advance
The syntax for your lookup is for the KV1 engine. We can update it for the KV2 secrets engine:
- name: Return specific value from vault
ansible.builtin.set_fact:
secret: "{{ lookup('hashi_vault', 'secret=my.secrets/data/dev token=vault-plaintext-root-token url=http://10.47.0.235:8200/') }}"
The secret fact will then be a dictionary containing all of the key value pairs at the specified secrets path my.secrets/data/dev. You can access the value of the key heslo with the normal syntax secret['heslo'].
Finally, you may also want to update to the Vault collection for Ansible with all of its newer features.
I'm trying to enable RDP on a Windows 11 machine with Ansible and its win_regedit module. The idea is to set a value that is to change a value from its default of 1 to 0, which enables RDP.
The task in my playbook looks like this:
- name: Set Registry key
ansible.windows.win_regedit:
path: 'HKLM:\System\CurrentControlSet\Control\Terminal Server'
name: 'fDenyTSConnections'
value: 0
type: dword
However, when I run it, Ansible doesn't change anything and reports it as OK. -vvv output for ansible-playbook looks like this (I have verified the value is set to '1', i.e. the undesired state):
ok: [win11test] => {
"changed": false,
"data_changed": false,
"data_type_changed": false,
"invocation": {
"module_args": {
"data": null,
"delete_key": true,
"hive": null,
"name": "0",
"path": "HKLM:\\System\\CurrentControlSet\\Control\\Terminal Server",
"state": "present",
"type": "dword",
"value": 0
}
}
}
I'm a bit stumped here. There's nothing in the docs to suggest I'm doing anything wrong.
The WinRM connection to the host seems to allow this just fine - I can run a PSSession from a Windows host, using the same credentials, and run the equivalent Powershell just fine. The code is:
Set-ItemProperty -Path 'HKLM:\System\CurrentControlSet\Control\Terminal Server' -name "fDenyTSConnections" -value 0
Anyone got an idea why Ansible thinks there's nothing to do here, and what I need to change about my task?
Not an answer to this specific question, but i found this on Google while researching my similar problem, as might others.
If the name value is surrounded by { and } like a lot of keys are, its essential to double-quote the name, otherwise Ansible tries to expand it as a variable, and it hits the wrong key.
- name: Show My Computer (1)
ansible.windows.win_regedit:
path: HKCU:\Software\Microsoft\Windows\CurrentVersion\Explorer\HideDesktopIcons\ClassicStartMenu
name: "{20D04FE0-3AEA-1069-A2D8-08002B30309D}"
data: 0
type: dword
is good.
- name: Show My Computer (1)
ansible.windows.win_regedit:
path: HKCU:\Software\Microsoft\Windows\CurrentVersion\Explorer\HideDesktopIcons\ClassicStartMenu
name: {20D04FE0-3AEA-1069-A2D8-08002B30309D}
data: 0
type: dword
is bad.
I've been trying to parse XML data in Ansible. I can get it to work using the xml module but I think that using parse_xml would better suit my needs.
I don't seem to be able to match any of the data in the xml with my specs file.
Here is the xml data:
<data xmlns=\"urn:ietf:params:xml:ns:netconf:base:1.0\" xmlns:nc=\"urn:ietf:params:xml:ns:netconf:base:1.0\">
<ntp xmlns=\"http://cisco.com/ns/yang/Cisco-IOS-XR-ip-ntp-oper\">
<nodes>
<node>
<node>0/0/CPU0</node>
<associations>
<is-ntp-enabled>true</is-ntp-enabled>
<sys-leap>ntp-leap-no-warning</sys-leap>
<peer-summary-info>
<peer-info-common>
<host-mode>ntp-mode-client</host-mode>
<is-configured>true</is-configured>
<address>10.1.1.1</address>
<reachability>0</reachability>
</peer-info-common>
<time-since>-1</time-since>
</peer-summary-info>
<peer-summary-info>
<peer-info-common>
<host-mode>ntp-mode-client</host-mode>
<is-configured>true</is-configured>
<address>172.16.252.29</address>
<reachability>255</reachability>
</peer-info-common>
<time-since>991</time-since>
</peer-summary-info>
</associations>
</node>
</nodes>
</ntp>
</data>
This is what the spec file looks like:
---
vars:
ntp_peers:
address: "{{ item.address }}"
reachability: "{{ item.reachability}}"
keys:
result:
value: "{{ ntp_peers }}"
top: data/ntp/nodes/node/associations
items:
address: peer-summary-info/peer-info-common/address
reachability: peer-summary-info/peer-info-common/reachability
and the task in the yaml file:
- name: parse ntp reply
set_fact:
parsed_ntp_data: "{{ NTP_STATUS.stdout | parse_xml('specs/iosxr_ntp.yaml') }}"
but the data does not return any results:
TASK [debug parsed_ntp_data] **************************************************************************************************************************************************************************
ok: [core-rtr01] => {
"parsed_ntp_data": {
"result": []
}
}
ok: [dist-rtr01] => {
"parsed_ntp_data": {
"result": []
}
}
I had never seen parse_xml before, so that was a fun adventure
There appear to be two things conspiring against you: the top: key is evaluated from the root Element, and your XML (unlike the rest of the examples) uses XML namespaces (the xmlns= bit) which means your XPaths have to be encoded in the Element.findall manner
For the first part, since Element.findall is run while sitting on the <data> Element, that means one cannot reference data/... in an XPath because that would be applicable to a structure <data><data>. I tried being sneaky by just making the XPath absolute /data/... but Python's XPath library throws up in that circumstance. So, at the very least your top: key needs to not start with data anything
Then, the xmlns= in your snippet stood out to me because that means those element's names are actually NS+":"+localName for every element, and thus an XPath of ntp does NOT match ns0:ntp because they're considered completely separate names (that being the point of the namespace, after all). It may very well be possible to use enough //*[localname() = "ntp"] silliness to avoid having to specify the namespace over and over, but I didn't try it
Again, as a concession to Python's XPath library, they encode the fully qualified name in an xpath as {the-namespace}local-name and there does not seem to be any way short of modifying network.py to pass in namespaces :-(
Thus, the "hello world" version that I used to confirm my theory:
vars:
ntp_peers:
address: "{{ item.address }}"
keys:
result:
value: "{{ ntp_peers }}"
top: '{http://cisco.com/ns/yang/Cisco-IOS-XR-ip-ntp-oper}ntp/{http://cisco.com/ns/yang/Cisco-IOS-XR-ip-ntp-oper}nodes/{http://cisco.com/ns/yang/Cisco-IOS-XR-ip-ntp-oper}node'
items:
address: '{http://cisco.com/ns/yang/Cisco-IOS-XR-ip-ntp-oper}node'
cheerfully produced
ok: [localhost] => {
"msg": {
"result": [
{
"address": "0/0/CPU0"
}
]
}
}
I went through the documentation provided here
https://docs.ansible.com/ansible/latest/collections/community/aws/ecs_taskdefinition_module.html
It gives me nice examples of setting of Fargate task definition. However it showcases example with only one port mapping and there is no mount point shown here.
I want to dynamically add port mappings ( depending on my app) and volume/mount points
For that I am defining my host_var for app as below ( there can be many such apps with different mount points and ports)
---
task_count: 4
task_cpu: 1028
task_memory: 2056
app_port: 8080
My Task definition yaml file looks like below
- name: Create/Update Task Definition
ecs_taskdefinition:
aws_access_key: "{{....}}"
aws_secret_key: "{{....}}"
security_token: "{{....}}"
region: "{{....}}"
launch_type: FARGATE
network_mode: awsvpc
execution_role_arn: "{{ ... }}"
task_role_arn: "{{ ...}}"
containers:
- name: "{{...}}"
environment: "{{...}}"
essential: true
image: "{{ ....}}"
logConfiguration: "{{....}}"
portMappings:
- containerPort: "{{app_port}}"
hostPort: "{{app_port}}"
cpu: "{{task_cpu}}"
memory: "{{task_memory}}"
state: present
I am able to create/update the task definition.
New requirements are that
Instead of one port, now we can have multiple(or none) port mappings.
We will have multiple (or none) mount points and volumes as well
Here is what I think the modified ansible host_var should look like below for ports
[container_port1:host_port1, container_port2:host_port2, container_port3:host_port3]
task_count: 4
task_cpu: 1028
task_memory: 2056
#[container_port1:host_port1, container_port2:host_port2, container_port3:host_port3]
app_ports: [8080:80, 8081:8081, 5703:5703]
I am not sure what to do in ansible playbook to run through this list of ports.
Another part of the problem is that, although I was able to achieve creating volume and mouting in container thorough aws console, I was not able to do same using ansible.
here is the snippet of json for the AWS fargate looks like ( for volume part). There can be many such mounts depending on the application. I want to achieve that dynamically by defining mount points and volumes in host_vars
-
-
-
"mountPoints": [
{
"readOnly": null,
"containerPath": "/mnt/downloads",
"sourceVolume": "downloads"
}
-
-
-
-
-
-
"volumes": [
{
"efsVolumeConfiguration": {
"transitEncryptionPort": ENABLED,
"fileSystemId": "fs-ecdg222d",
"authorizationConfig": {
"iam": "ENABLED",
"accessPointId": null
},
"transitEncryption": "ENABLED",
"rootDirectory": "/vol/downloads"
},
"name": "downloads",
"host": null,
"dockerVolumeConfiguration": null
}
I am not sure how to do that.
Official documentation offers very little help.
I have this (in the example shown I reduced it by removing many lines) non-trivial JSON retrieved from a Spark server:
{
"spark.worker.cleanup.enabled": true,
"spark.worker.ui.retainedDrivers": 50,
"spark.worker.cleanup.appDataTtl": 7200,
"fusion.spark.worker.webui.port": 8082,
"fusion.spark.worker.memory": "4g",
"fusion.spark.worker.port": 8769,
"spark.worker.timeout": 30
}
I try to read fusion.spark.worker.memory but fail to do so. In my debug statements I can see that the information is there:
msg: "Spark memory: {{spark_worker_cfg.json}} shows this:
ok: [process1] => {
"msg": "Spark memory: {u'spark.worker.ui.retainedDrivers': 50, u'spark.worker.cleanup.enabled': True, u'fusion.spark.worker.port': 8769, u'spark.worker.cleanup.appDataTtl': 7200, u'spark.worker.timeout': 30, u'fusion.spark.worker.memory': u'4g', u'fusion.spark.worker.webui.port': 8082}"
}
The dump using var: spark_worker_cfg shows this:
ok: [process1] => {
"spark_worker_cfg": {
"changed": false,
"connection": "close",
"content_length": "279",
"content_type": "application/json",
"cookies": {},
"cookies_string": "",
"failed": false,
"fusion_request_id": "Pj2zeWThLw",
"json": {
"fusion.spark.worker.memory": "4g",
"fusion.spark.worker.port": 8769,
"fusion.spark.worker.webui.port": 8082,
"spark.worker.cleanup.appDataTtl": 7200,
"spark.worker.cleanup.enabled": true,
"spark.worker.timeout": 30,
"spark.worker.ui.retainedDrivers": 50
},
"msg": "OK (279 bytes)",
"redirected": false,
"server": "Jetty(9.4.12.v20180830)",
"status": 200,
"url": "http://localhost:8765/api/v1/configurations?prefix=spark.worker"
}
}
I can't access the value using {{spark_worker_cfg.json.fusion.spark.worker.memory}}, my problem seems to be caused by the names containing dots:
The task includes an option with an undefined variable. The error was:
'dict object' has no attribute 'fusion'
I have had a look at two SO posts (1 and 2) that look like duplicates of my question but could not derive from them how to solve my current issue.
The keys in the 'json' element of the data structure, contain literal dots, rather than represent a structure. This will causes issues, because Ansible will not know to treat them as literal if dotted notation is used. Therefore, use square bracket notation to reference them, rather than dotted:
- debug:
msg: "{{ spark_worker_cfg['json']['fusion.spark.worker.memory'] }}"
(At first glance this looked like an issue with a JSON encoded string that needed decoding, which could have been handled:"{{ spark_worker_cfg.json | from_json }}")
You could use the json_query filter to get your results. https://docs.ansible.com/ansible/latest/user_guide/playbooks_filters.html
msg="{{ spark_worker_cfg.json | json_query('fusion.spark.worker.memory') }}
edit:
In response to your comment, the fact that we get an empty string returned leads me to believe that the query isn't correct. It can be frustrating to find the exact query while using the json_query filter so I usually use a jsonpath tool beforehand. I've linking one in my comment below but I, personally, use the jsonUtils addon in intelliJ to find my path (which still needs adjustment because the paths are handled a bit differently between the two).
If your json looked like this:
{
value: "theValue"
}
then
json_query('value')
would work.
The path you're passing to json_query isn't correct for what you're trying to do.
If your top level object was named fusion_spark_worker_memory (without the periods), then your query should work. The dots are throwing things off, I believe. There may be a way to escape those in the query...
edit 2: clockworknet for the win! He beat me to it both times. :bow: