When using cloud init's #cloud-config to create configuration files, how would I go about using variables to populate values?
In my specific case I'd like to autostart EC2 instances as preconfigured salt minions. Example of salt minion cloud config
Say I'd like to get the specific EC2 instances id and set that as the salt minion's id.
How would I go about it setting the value dynamically for each instance?
in boot command bootcmd can have environment variable $INSTANCE_ID, you can save it for later usage. see http://cloudinit.readthedocs.org/en/latest/topics/examples.html#run-commands-on-first-boot
I for example do like below
#cloud-config
bootcmd:
- echo $INSTANCE_ID > /hello.txt
The closest I've seen to configurable variables is [Instance Metadata].(https://cloudinit.readthedocs.io/en/latest/topics/instancedata.html#)
It says you can use:
userdata provided at instance creation
You can use data created in /run/cloud-init/instance-data.json.
You can use import this instance data using Jinja templates in your YAML cloud-config to pull in this data:
## template: jinja
#cloud-config
runcmd:
- echo 'EC2 public hostname allocated to instance: {{
ds.meta_data.public_hostname }}' > /tmp/instance_metadata
- echo 'EC2 availability zone: {{ v1.availability_zone }}' >>
/tmp/instance_metadata
- curl -X POST -d '{"hostname": "{{ds.meta_data.public_hostname }}",
"availability-zone": "{{ v1.availability_zone }}"}'
https://example.com
But I'm not exactly sure how you create the /run/cloud-init/instance-data.json file.
This CoreOS issue suggests that if you put variables into /etc/environment then you can use those.
I know for example that there are a few variables used such as $MIRROR $RELEASE, $INSTANCE_ID for the phone_home module.
Try the ec2metadata tool (just queries the EC2 metadata). Say put the following in your instances userdata:
wget http://s3.amazonaws.com/ec2metadata/ec2-metadata
chmod u+x ec2-metadata
# The following gives you the instance id and you can pass it to your salt minion config
ec2-metadata -i
More info on the ec2-metadata script here
Related
I'm using aws_rds plugin to create an inventory with my rds cluster/instances and create groups_vars based on the environment tags.
plugin: aws_rds
regions:
- xx-xxxx-x
include_clusters: true
keyed_groups:
- key: tags.Environment
prefix: "tag_Environment_"
separator: ""
https://docs.ansible.com/ansible/2.9/plugins/inventory/aws_rds.html
The problem is that this plugin doesn't allow to take the Endpoint name so I was wondering if it's possible to add the last part of this cluster-xxxxxxxxxx.xx-xxxx-x.rds.amazonaws.com on each group_var created, like ansible_host= ${ansible_host}.cluster-xxxxxxxxxx.xx-xxxx-x.rds.amazonaws.com or something like that.
Those AWS inventory plugin are gathering a lot of information from the API, I had quite a similar requirement lately on aws_ec2 inventory plugin and here is how I proceeded to find the information I needed under the compose parameter of the inventory's configuration.
To fetch what I needed to put under the compose parameter, I first configured the inventory, like you did.
Then I displayed all the collected hosts, using
ansible-inventory --graph
This yields something like
#all:
|--#nodes:
| |--node1
| |--node2
| |--node3
|--#ungrouped:
With this, I targeted one node, let's say node1, here, and I displayed all the information Ansible had on them, doing:
ansible -m setup node1
and
ansible -m debug -a "var=vars" node1
In all this, in searched for the information needed.
In your case that could be achieved by doing:
ansible -m setup <one-of-the-hosts> | grep "cluster-" | grep ".rds.amazonaws.com"
and
ansible -m debug -a <one-of-the-hosts> | grep "cluster-" | grep ".rds.amazonaws.com"
Last but not least, configured the variable I found out in the compose parameter of the inventory's configuration
compose:
ansible_host: <variable-name-found-at-step-4>
e.g., in my EC2 configuration, it ended up being public_dns_address
plugin: aws_ec2
hostnames:
- tag:Name
# ^-- Given that you indeed have a tag named "Name" on your EC2 instances
compose:
ansible_host: public_dns_address
In the end, I achieved what I wanted by defining this variable in groups_vars (so I could define one for each environment).
ansible_host: "{{ inventory_hostname }}.cluster-xxxxxxxx.xx-xxxx-x.rds.amazonaws.com"
I'm just getting started with ansible and have successfully been able to configure ansible to get dynamic inventory from GCP.
I am able to successfully run the ping module against all instances:
ansible -i ~/git/ansible/inventory all -m ping
I am also able to successfully run the ping module against a single instance based on hostname:
ansible -i ~/git/ansible/inventory instance-2 -m ping
I would now like to utilize tags to group instances. For example, I have set of instances that are labeled 'env:dev'
https://www.evernote.com/l/AfcLWLkermxMyIK7GvGpQXjXdIDFVAiT_z0
I have attempted multiple variations of the command below with no luck
ansible -i ~/git/ansible/inventory tag_env:dev -m ping
How can I filter and group my dynamic inventory on GCP?
So you need to add network tag in instance settings not labels i don't know why but gce.py doesn't return GCP labels so you can only use network tags wich is limited (i mean not key=value but just value)
For example add network tag just 'dev' and then run ansible -i ~/git/ansible/inventory tag_dev -m ping
also if you need to filter by few tags only way i found it's
- name: test stuff
hosts: tag_api:&tag_{{ environment }}
var_files:
vars/{{ environment }}
vars/api
tasks:
- name: test
command: echo "test"
run playbook like this ansible-playbook -i inventory/ -u user playbook/test.yml -e environment=dev
maybe someone know better way, with aws ec2.py i could filter in ec2.ini config but gce.py very limited
also i noticed that sometimes you need to clear cache gce.py --refresh-cache
I am provisioning AWS infrastructure using terraform and want to pass variables such as aws_subnet_id and aws_security_id into ansible playbook using vars_file (don't know if there is any other way though). How can I do that?
I use Terraform local_file to create an Ansible vars_file. I add a tf_ prefix to the variable names to make it clear that they originate in Terraform:
# Export Terraform variable values to an Ansible var_file
resource "local_file" "tf_ansible_vars_file_new" {
content = <<-DOC
# Ansible vars_file containing variable values from Terraform.
# Generated by Terraform mgmt configuration.
tf_environment: ${var.environment}
tf_gitlab_backup_bucket_name: ${aws_s3_bucket.gitlab_backup.bucket}
DOC
filename = "./tf_ansible_vars_file.yml"
}
Run terraform apply to create Ansible var_file tf_ansible_vars_file.yml containing Terraform variable values:
# Ansible vars_file containing variable values from Terraform.
# Generated by Terraform mgmt configuration.
tf_environment: "mgmt"
tf_gitlab_backup_bucket_name: "project-mgmt-gitlab-backup"
Add tf_ansible_vars_file.yml to your Ansible playbook:
vars_files:
- ../terraform/mgmt/tf_ansible_vars_file.yml
Now, in Ansible the variables defined in this file will contain values from Terraform.
Obviously, this means that you must run Terraform before Ansible. But it won't be so obvious to all your Ansible users. Add assertions to your Ansible playbook to help the user figure out what to do if a tf_ variable is missing:
- name: Check mandatory variables imported from Terraform
assert:
that:
- tf_environment is defined
- tf_gitlab_backup_bucket_name is defined
fail_msg: "tf_* variable usually defined in '../terraform/mgmt/tf_ansible_vars_file.yml' is missing"
UPDATE: An earlier version of this answer used a Terraform template. Experience shows that the template file is error prone and adds unnecessarily complexity. So I moved the template file to the content of the local_file.
terraform outputs are an option, or you can just use something like:
provisioner "local-exec" {
command = "ANSIBLE_HOST_KEY_CHECKING=\"False\" ansible-playbook -u ${var.ssh_user} --private-key=\"~/.ssh/id_rsa\" --extra-vars='{"aws_subnet_id": ${aws_terraform_variable_here}, "aws_security_id": ${aws_terraform_variable_here} }' -i '${azurerm_public_ip.pnic.ip_address},' ansible/deploy-with-ansible.yml"
}
or you can do a sed thing ... as a local provisioner to update the var file..
or you can use terraform outputs.... your preference....
I highly recommend this script. It works well and is maintained by Cisco and will give you more flexibility.
https://github.com/CiscoCloud/terraform.py
Since I want to run both terraform plan and ansible --check in a pull request with a CI, I've decided to go with terraform output.
Basically this is how I run my Ansible now and it gets all outputs from terraform :
With the following output
// output.tf
output "tf_gh_deployment_status_token" {
value = var.GH_DEPLOYMENT_STATUS_TOKEN
sensitive = true
}
Running a bit modified terraform output parsed with jq :
$ terraform output --json | jq 'with_entries(.value |= .value)'
{
"tf_gh_deployment_status_token": "<some token>"
}
This makes it great to run with --extra-args with Ansible :
$ ansible-playbook \
-i ./inventory/production.yaml \
./playbook.yaml \
--extra-vars "$(terraform output --json | jq 'with_entries(.value |= .value)')"
Now I can use {{ tf_gh_deployment_status_token }} anywhere in my playbook.
Use terraform outputs - https://www.terraform.io/intro/getting-started/outputs.html (it is not clear if you are using it already)
Then using command like terraform output ip, you can then use those values in your scripts to generate or populate other files like inventory files or vars_file.
Another option is to use terraform templates and render your files like inventory files from terraform itself and then use it from Ansible.
How do I get the value of a variable from the inventory if the variable could be either in the inventory file or group_vars directory?
For example, region=place-a could be in the inventory file or in a file in the group_vars somewhere. I would like a command to be able to retrieve that value using ansible or something retrieve that value. like:
$ ansible -i /somewhere/production/web --get-value region
place-a
That would help me with deployments and knowing which region is being deployed to.
.
A longer explanation to clarify, my inventory structure looks like this:
/somewhere/production/web
/somewhere/production/group_vars/web
The contents with the variables of the inventory file, /somewhere/production/web looks like this:
[web:children]
web1 ansible_ssh_host=10.0.0.1
web2 ansible_ssh_host=10.0.0.2
[web:vars]
region=place-a
I could get the value from the inventory file by simply parsing the file. like so:
$ awk -F "=" '/^region/ {print $2}' /somewhere/production/web
place-a
But that variable could be in the group_vars file, too. For example:
$ cat /somewhere/production/group_vars/web
region: place-a
Or it could look like an array:
$ cat /somewhere/production/group_vars/web
region:
- place-a
I don't want to look for and parse all the possible files.
Does Ansible have a way to get the values? Kind of like how it does with --list-hosts?
$ ansible web -i /somewhere/production/web --list-hosts
web1
web2
Short Answer
NO there is no CLI option to get value of variables.
Long Answer
First of all, you must look into how variable precedence works in Ansible.
The exact order is defined in the documentation. Here is an summary excerpt:
Basically, anything that goes into “role defaults” (the defaults
folder inside the role) is the most malleable and easily overridden.
Anything in the vars directory of the role overrides previous versions
of that variable in namespace. The idea here to follow is that the
more explicit you get in scope, the more precedence it takes with
command line -e extra vars always winning. Host and/or inventory
variables can win over role defaults, but not explicit includes like
the vars directory or an include_vars task.
With that cleared out, the only way to see what value is a variable taking is to use the debug task as follows:
- name: debug a variable
debug:
var: region
This will print out the value of the variable region.
My suggestion would be to maintain a single value type, either a string or a list to prevent confusion across tasks in different playbooks.
You could use something like this:
- name: deploy to regions
<task_type>:
<task_options>: <task_option_value>
// use the region variable as you wish
region: {{ item }}
...
with_items: "{{ regions.split(',') }}"
In this case you could have a variable regions=us-west,us-east comma-separated. The with_items statement would split it over the , and repeat the task for all the regions.
The CLI version of this is important for people who are trying to connect ansible to other systems. Building on the usage of the copy module elsewhere, if you have a POSIX mktemp and a copy of jq locally, then here's a bash one-liner that does the trick from the CLI:
export TMP=`mktemp` && ansible localhost -c local -i inventory.yml -m copy -a "content={{hostvars['SOME_HOSTNAME']}} dest=${TMP}" >/dev/null && cat ${TMP}|jq -r .SOME_VAR_NAME && rm ${TMP}
Breaking it down
# create a tempfile
export TMP=`mktemp`
# quietly use Ansible in local only mode, loading inventory.yml
# digging up the already-merged global/host/group vars
# for SOME_HOSTNAME, copying them to our tempfile from before
ansible localhost --connection local \
--inventory inventory.yml --module-name copy \
--args "content={{hostvars['SOME_HOSTNAME']}} dest=${TMP}" \
> /dev/null
# CLI-embedded ansible is a bit cumbersome. After all the data
# is exported to JSON, we can use `jq` to get a lot more flexibility
# out of queries/filters on this data. Here we just want a single
# value though, so we parse out SOME_VAR_NAME from all the host variables
# we saved before
cat ${TMP}|jq -r .SOME_VAR_NAME
rm ${TMP}
Another more easy solution get a variable through cli from ansible could be this:
export tmp_file=/tmp/ansible.$RANDOM
ansible -i <inventory> localhost -m copy -a "content={{ $VARIABLE_NAME }} dest=$tmp_file"
export VARIBALE_VALUE=$(cat $tmp_file)
rm -f $tmp_file
Looks quite ugly but is really helpful.
I have a harness to build VMs using Packer that in turn calls Ansible (in local mode) to do the heavy lifting.
I'd like to be able to parameters to Packer (got that), which is passes to Ansible as extra vars.
I can pass an external variables files and also a simple variable such as the example below.
ansible-playbook -v -c local something.yml --extra-vars "deploy_loc=custom"
Thats okay, but I really need to pass more complex array of variables, such as the examples below.
I've tried a number of formatting such as the one below and usually get some kind of delimiter error.
ansible-playbook -v -c local something.yml --extra-vars 'deploy_loc=custom deploy_scen: [custom][ip=1.2.34]}'
Role variable file
# Which location
deploy_loc: 'external-dhcp'
# location defaults
deploy_scen:
custom:
ipv4: yes
net_type: dhcp
ip: '1.1.1.1'
prefix: '24'
gw: '1.1.1.1.254'
host: 'custom'
domain: 'domain.com'
dns1: '1.1.1.2'
standard-eng:
ipv4: yes
net_type: none
ip: '12.12.12.5'
prefix: '24'
external-dhcp:
ipv4: yes
net_type: dhcp
I think it's more robust and readable to generate a yaml file and use that with vars_files.
Alternatively you can generate a json file and read and parse it using a file lookup and the from_json filter. Something like this:
- name: Read objects
set_fact: deploy_scen={{lookup('file', 'deploy_scen.json') | from_json}}
However, if you really want --extra-var you can use the dict() function:
-e 'var={{dict(key1=dict(subkey1="value"),key2=dict(subkey1="value2"))}}'
You can pass a json structure in to ansible via the extra-vars parameter. You have to be a little careful to make sure it's set up properly though. You want to have the parameter look something like this:
--extra-vars {"param1":"foo","param2":"bar","file_list":[ "file1", "file2", "file3" ]}
When I do things like this I typically write a small bash wrapper script that will ensure extra-vars is set up properly and then invokes ansible.