I want to use Hashicorp Vault with Ansible to retrieve username/password which I will use in Ansible playbook.
Vault is setup - I created a secret. What are the steps to integrate both? the documentation around plugins isn't that great. I tried the file lookup from ansible and this works but how to use 3rd party plugins? Can somebody help me with the steps to follow?
Install the plugin, pip install ansible-modules-hashivault
What is the difference with https://github.com/jhaals/ansible-vault
2.a The environment variables (VAULT ADDR & VAULT TOKEN) I put where?
Change ansible.cfg to point to vault.py which is located in "plugin" folder of my Ansible Project
To test basic integration, can I use the following playbook?
https://pypi.python.org/pypi/ansible-modules-hashivault
- hosts: localhost
-tasks:
- hashivault_status:
register: 'vault_status'
Tried this but I get:
An exception occurred during task execution. The full traceback is:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 119, in run
res = self._execute()
File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 431, in _execute
self._task.post_validate(templar=templar)
File "/usr/lib/python2.7/site-packages/ansible/playbook/task.py", line 248, in post_validate
super(Task, self).post_validate(templar)
File "/usr/lib/python2.7/site-packages/ansible/playbook/base.py", line 371, in post_validate
value = templar.template(getattr(self, name))
File "/usr/lib/python2.7/site-packages/ansible/template/__init__.py", line 359, in template
d[k] = self.template(variable[k], preserve_trailing_newlines=preserve_trailing_newlines, fail_on_undefined=fail_on_undefined, overrides=overrides)
File "/usr/lib/python2.7/site-packages/ansible/template/__init__.py", line 331, in template
result = self._do_template(variable, preserve_trailing_newlines=preserve_trailing_newlines, escape_backslashes=escape_backslashes, fail_on_undefined=fail_on_undefined, overrides=overrides)
File "/usr/lib/python2.7/site-packages/ansible/template/__init__.py", line 507, in _do_template
res = j2_concat(rf)
File "<template>", line 8, in root
File "/usr/lib/python2.7/site-packages/jinja2/runtime.py", line 193, in call
return __obj(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/ansible/template/__init__.py", line 420, in _lookup
instance = self._lookup_loader.get(name.lower(), loader=self._loader, templar=self)
File "/usr/lib/python2.7/site-packages/ansible/plugins/__init__.py", line 339, in get
self._module_cache[path] = self._load_module_source('.'.join([self.package, name]), path)
File "/usr/lib/python2.7/site-packages/ansible/plugins/__init__.py", line 324, in _load_module_source
module = imp.load_source(name, path, module_file)
File "/etc/ansible/ProjectA/lookup_plugins/vault.py", line 5
<!DOCTYPE html>
^
SyntaxError: invalid syntax
fatal: [win01]: FAILED! => {
"failed": true,
"msg": "Unexpected failure during module execution.",
"stdout": ""
Since you put so many eggs into the post, that I have no clue what the question is really about, here's something to get you going with the native lookup plugin and jhaals/ansible-vault.
you can create lookup_plugins in the current directory and save vault.py inside;
the VAULT_ADDR and VAULT_TOKEN environment variables are as you see them in the script;
The Bash script below (it uses screen and jq, you might need to install them) runs Vault in dev mode, sets the secret, and runs Ansible playbook which queries the secret with two lookup plugins:
#!/bin/bash
set -euo pipefail
export VAULT_ADDR=http://127.0.0.1:8200
if [[ ! $(pgrep -f "vault server -dev") ]]; then
echo \"vault server -dev\" not running, starting...
screen -S vault -d -m vault server -dev
printf "sleeping for 3 seconds\n"
sleep 3
else
echo \"vault server -dev\" already running, leaving as is...
fi
vault write secret/hello value=world excited=yes
export VAULT_TOKEN=$(vault token-create -format=json | jq -r .auth.client_token)
ansible-playbook playbook.yml --extra-vars="vault_token=${VAULT_TOKEN}"
and playbook.yml:
---
- hosts: localhost
connection: local
tasks:
- name: Retrieve secret/hello using native hashi_vault plugin
debug: msg="{{ lookup('hashi_vault', 'secret=secret/hello token={{ vault_token }} url=http://127.0.0.1:8200') }}"
- name: Retrieve secret/hello using jhaals vault lookup
debug: msg="{{ lookup('vault', 'secret/hello') }}"
In the end you should get:
TASK [Retrieve secret/hello using native hashi_vault plugin] *******************
ok: [localhost] => {
"msg": "world"
}
TASK [Retrieve secret/hello using jhaals vault lookup] *************************
ok: [localhost] => {
"msg": {
"excited": "yes",
"value": "world"
}
}
The word world was fetched from Vault.
Related
I have been creating different connectors in our confluent installation by using the curl command to PUT a configuration to a url. The set-up is normally done by adding the configuration and curl call in a shell script. We either write separate shell scripts for each environment or pass the server names - broker and connect - as parameters.
I want to write an ansible script to define new connectors which will basically be same script that can be run in all environments and will only depend on the hosts file in each environment. This will help in eliminating user error / typos in server names when running it in different environments.
My current plan is to just convert the shell script and use ansible shell or command module to call 'curl' and pass the parameters. But I am not sure, if that is the correct way, even if it may get the job done.
Does anyone have any suggestion to do it correctly / properly.
Thank you
UPDATE
Based on #Zeitounator's recommendation, I tried to use the uri module. But keep on getting error
---
- name: Create Kafka Connector
hosts: kafka_connect
vars:
connect_url: https://{{ kafka_connect[0] }}:8083
connector_name: CamelTestConnector
cert_path: /tmp/test/
tasks:
- name: prepare connector config
template:
src: camelconnector.json
dest: /tmp/camelconnector.json
delegate_to: localhost
run_once: yes
- name: Create Connector
uri:
url: "https://kafka-connect1-dev:8083/connectors/{{ connector_name }}/config"
client_cert: "{{ cert_path }}certificate.pem"
client_key: "{{ cert_path }}priv.key"
method: PUT
body: "{{ lookup('file', '/tmp/camelconnector.json' ) }}"
body_format: json
validate_certs: no
status_code: [201, 201, 204]
headers:
Content-Type: application/json
# If you're interested in the response
return_content: yes
register: api_result
run_once: yes
- debug:
var=api_result
#curl --cert /software/scripts/clientcerts/certificate.pem --key /software/scripts/clientcerts/priv.key -k -X PUT -H "${HEADER}" --data "${DATA}" https://"${1}":8083/connectors/CamelTestConnector/config
#curl --cert /software/scripts/clientcerts/certificate.pem --key /software/scripts/clientcerts/priv.key -k https://"${1}":8083/connectors/CamelTestConnector/status
This is the error I keep getting
TASK [Create Connector] ***********************************************************************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: IOError: [Errno 2] No such file or directory
fatal: [kafka-connect1-dev]: FAILED! => {"changed": false, "content": "", "elapsed": 0, "msg": "Status code was -1 and not [201, 201, 204]: An unknown error occurred: [Errno 2] No such file or directory", "redirected": false, "status": -1, "url": "https://kafka-connect1-dev/connectors/"}
I have verified that the json file is present. I am unable to identify which file it is complaining about. Tried various iterations for connect-url, but all give the same error.
How do I identify which file it is complaining about?
Thanks
UPDATE 2
I ran the script in verbose mode and I get below stack trace. If I comment out the cert and key parts, then I get connection refused, so it seems the cert and key are required. I tried to search for the cause of the error without success. The certificate and key files do exist at the location. What else do I need to check?
The full traceback is:
Traceback (most recent call last):
File "/tmp/ansible_uri_payload_6FB8tM/ansible_uri_payload.zip/ansible/module_utils/urls.py", line 1494, in fetch_url
unix_socket=unix_socket, ca_path=ca_path)
File "/tmp/ansible_uri_payload_6FB8tM/ansible_uri_payload.zip/ansible/module_utils/urls.py", line 1390, in open_url
unredirected_headers=unredirected_headers)
File "/tmp/ansible_uri_payload_6FB8tM/ansible_uri_payload.zip/ansible/module_utils/urls.py", line 1294, in open
r = urllib_request.urlopen(*urlopen_args)
File "/usr/lib64/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib64/python2.7/urllib2.py", line 431, in open
response = self._open(req, data)
File "/usr/lib64/python2.7/urllib2.py", line 449, in _open
'_open', req)
File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
result = func(*args)
File "/tmp/ansible_uri_payload_6FB8tM/ansible_uri_payload.zip/ansible/module_utils/urls.py", line 467, in https_open
return self.do_open(self._build_https_connection, req)
File "/usr/lib64/python2.7/urllib2.py", line 1183, in do_open
h = http_class(host, timeout=req.timeout, **http_conn_args)
File "/tmp/ansible_uri_payload_6FB8tM/ansible_uri_payload.zip/ansible/module_utils/urls.py", line 480, in _build_https_connection
return httplib.HTTPSConnection(host, **kwargs)
File "/usr/lib64/python2.7/httplib.py", line 1259, in __init__
context.load_cert_chain(cert_file, key_file)
IOError: [Errno 2] No such file or directory
Using Terraform code I have created Other type of secrets in AWS Secrets Manager.
I need to use these AWS secrets in Ansible code. I found this below link but I am unable to proceed it.
https://docs.ansible.com/ansible/2.8/plugins/lookup/aws_secret.html
I have below Ansible code:-
database.yml
- name: Airflow | DB | Create MySQL DB
mysql_db:
login_user: "{{ mysql_user }}"
# login_password: "{{ mysql_root_password }}"
login_password: "{{ lookup('ca_dev', 'mysql_root_password') }}"
# config_file: /etc/my.cnf
# login_unix_socket: /var/lib/mysql/mysql.sock
# encrypted: yes
name: "airflow"
state: "present"
How can I incorporate AWS secret Manager in my ansible code?
Error message:-
TASK [../../roles/airflow : Airflow | DB | Create MySQL DB] **************************************************************************************************************************************************************************
task path: /home/ec2-user/cng-ansible/roles/airflow/tasks/database.yml:25
The full traceback is:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 140, in run
res = self._execute()
File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 539, in _execute
self._task.post_validate(templar=templar)
File "/usr/lib/python2.7/site-packages/ansible/playbook/task.py", line 267, in post_validate
super(Task, self).post_validate(templar)
File "/usr/lib/python2.7/site-packages/ansible/playbook/base.py", line 364, in post_validate
value = templar.template(getattr(self, name))
File "/usr/lib/python2.7/site-packages/ansible/template/__init__.py", line 540, in template
disable_lookups=disable_lookups,
File "/usr/lib/python2.7/site-packages/ansible/template/__init__.py", line 495, in template
disable_lookups=disable_lookups,
File "/usr/lib/python2.7/site-packages/ansible/template/__init__.py", line 746, in do_template
res = j2_concat(rf)
File "<template>", line 8, in root
File "/usr/lib/python2.7/site-packages/jinja2/runtime.py", line 193, in call
return __obj(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/ansible/template/__init__.py", line 631, in _lookup
instance = self._lookup_loader.get(name.lower(), loader=self._loader, templar=self)
File "/usr/lib/python2.7/site-packages/ansible/plugins/loader.py", line 381, in get
obj = getattr(self._module_cache[path], self.class_name)
AttributeError: 'module' object has no attribute 'LookupModule'
fatal: [127.0.0.1]: FAILED! => {
"msg": "Unexpected failure during module execution.",
"stdout": ""
}
RUNNING HANDLER [../../roles/airflow : restart rabbitmq-server]
task path: /home/ec2-user/cng-ansible/roles/airflow/handlers/main.yml:28
to retry, use: --limit #/home/ec2-user/cng-ansible/plays/airflow/installAirflow.retry
PLAY RECAP
127.0.0.1 : ok=39 changed=7 unreachable=0 failed=1
ansible-doc -t lookup -l output
The error {"msg": "lookup plugin (ca_dev) not found"} suggests your issue is the misuse of the lookup command.
The following line:
login_password: "{{ lookup('ca_dev', 'mysql_root_password') }}"
Should look something like
login_password: "{{ lookup('aws_secret', 'mysql_root_password') }}"
ca_dev is not a valid lookup type, whereas aws_secret is.
You can see a list of supported lookup plugins for Ansible 2.8 in the Lookup Plugins section of the official documentation.
If you are using a custom lookup plugin, or backporting a plugin from a future version of ansible to an older version, you must make sure that it is in a directory visible to ansible.
You can either place the custom file in the default location ansible looks in ~/.ansible/plugins/lookup:/usr/share/ansible/plugins/lookup or configure your ansible.cfg to look in a different place using the following lookup_plugins ini key under the defaults section.
DEFAULT_LOOKUP_PLUGIN_PATH
Description: Colon separated paths in which Ansible will search for Lookup Plugins.
Type: pathspec
Default: ~/.ansible/plugins/lookup:/usr/share/ansible/plugins/lookup
Ini Section: defaults
Ini Key: lookup_plugins
Environment: ANSIBLE_LOOKUP_PLUGINS
Documentation for this can be found in the Ansible Configuration section of the official documentation
Running ansible 2.6
This error happens when I try to run the command:
verify /md5 flash:/{ios_file}
This is the output of the command:
TASK [IOS - MD5 CHECK - PASS1] **************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ansible.module_utils.connection.ConnectionError: timeout trying to send command: verify /md5 c2960x-universalk9-mz.152-2.E8.bin
fatal: [pdctestisesw7]: FAILED! => changed=false
module_stderr: |-
Traceback (most recent call last):
File "/tmp/ansible_cr7Tgd/ansible_module_ios_command.py", line 247, in <module>
main()
File "/tmp/ansible_cr7Tgd/ansible_module_ios_command.py", line 217, in main
responses = run_commands(module, commands)
File "/tmp/ansible_cr7Tgd/ansible_modlib.zip/ansible/module_utils/network/ios/ios.py", line 148, in run_commands
File "/tmp/ansible_cr7Tgd/ansible_modlib.zip/ansible/module_utils/connection.py", line 174, in __rpc__
ansible.module_utils.connection.ConnectionError: timeout trying to send command: verify /md5 c2960x-universalk9-mz.152-2.E8.bin
module_stdout: ''
msg: MODULE FAILURE
rc: 1
to retry, use: --limit #/export/home/e130885/playbooks/ios-switch-upgrade/upgrade_ios_switch_v1.retry
Here is the task being executed:
- name: IOS - MD5 CHECK - PASS1
ios_command:
commands:
- command: "verify /md5 {{ compliant_ios_file }}"
register: md5_response
vars:
ansible_command_timeout: 3000
when: 'compliant_ios_file in dir_response.stdout[0]'
This only seems to happen on commands that take more than a second to execute.
There are different ways to achieve it, the following works fine for me. The timeout values varies depending on router type.
> - name: VERIFY_IMAGE
> ios_command:
> commands:
> - "verify /md5 flash:c800-universalk9-mz.SPA.154-3.M10.bin"
> wait_for:
> - result[0] contains "a8216179d49e598579e21b7e5abc9046"
> retries: 1
> vars:
> ansible_command_timeout: 120
I am using Ansible 2.4.
I can't get following ansible-playbook to run:
test.yml
---
- hosts: "localhost"
become: no
vars:
foo_withsinglequote: !vault |
$ANSIBLE_VAULT;1.1;AES256
39313737636336313832376165636465346162366333663137373165363662316263336166393666
3566643732663063386333303638633962363863306463610a643931396636613361353165653265
38376630313939626637623538613432373336646663636563623062636238313731326263336263
3138643931323662620a336534383964663562353162393930613965386465616630363335326138
3431
foo_withdoublequote: !vault |
$ANSIBLE_VAULT;1.1;AES256
64633863363838326664323238313866616161313937323563636430326432393638336334303336
3533653339663438356238613937336466623834666537630a646139643033653237353262616662
30643732313861373130633036346361663130326332303932616433643761633739306137333237
6263653365386132620a633738663336313532366637613533313361646339623137393461383363
3332
tasks:
- name: Echo foo_withsinglequote
command: echo "{{ foo_withsinglequote }}"
- name: Echo foo_withdoublequote
command: echo "{{ foo_withdoublequote }}"
To generate the vault variables I used following:
$ echo 123 > vlt.txt
$ ansible-vault --vault-password-file=vlt.txt encrypt_string "abc\"def"
$ ansible-vault --vault-password-file=vlt.txt encrypt_string "abc\'def"
To run the playbook:
$ ansible-playbook --vault-password-file=vlt.txt test.yml
This gives following error:
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ValueError: No closing quotation
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "module_stderr": "Traceback (most recent call last):\n File \"/tmp/ansible_8uz23O/ansible_module_command.py\", line 213, in \n main()\n File \"/tmp/ansible_8uz23O/ansible_module_command.py\", line 182, in main\n args = shlex.split(args)\n File \"/usr/lib/python2.7/shlex.py\", line 279, in split\n return list(lex)\n File \"/usr/lib/python2.7/shlex.py\", line 269, in next\n token = self.get_token()\n File \"/usr/lib/python2.7/shlex.py\", line 96, in get_token\n raw = self.read_token()\n File \"/usr/lib/python2.7/shlex.py\", line 172, in read_token\n raise ValueError, \"No closing quotation\"\nValueError: No closing quotation\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 0}
How can I quote the vaulted variables correctly? Because I don't know in advance, if the vaulted variables will contain single or double quotes.
Your problem description, despite being well written, unfortunately wrongly attributes the problem to Ansible Vault.
In fact, the problem you reported, comes simply from trying to execute the task which effectively becomes:
- command: echo abc"def
Ansible Vault plays no role in causing this problem -- if you defined the variable directly with foo: abc\"def you'd get the same error message.
The solution is simply to quote the string in the echo command:
- command: echo '{{ foo }}'
Other than that you can use quote filter, but for Vault-protected variable you need to first set a static fact:
- set_fact:
bar: "{{ foo }}"
- command: echo {{ bar|quote }}
Finally, the simplest solution to the underlying problem is: do not use special characters in passwords. Increase the length instead.
When I have this role:
# playbooks/roles/ansible/tasks/main.yml
- name: Install Ansible
pip:
state: present
name: ansible
version: "{{ ansible_version }}"
# playbooks/roles/ansible/defaults/main.yml
ansible_version: 1.9.4
I get this error while running ansible-playbook version 1.9.4 or 2.0.0.2:
TASK: [ansible | Install Ansible] *********************************************
failed: [localhost] => {"cmd": "/usr/local/bin/pip install ansible=={'major': 1, 'full': '1.9.4', 'string': '1.9.4\\n configured module search path = None', 'minor': 9, 'revision': 4}", "failed": true}
msg:
:stderr: Invalid requirement: 'ansible=={major:'
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/pip/req/req_install.py", line 73, in __init__
req = pkg_resources.Requirement.parse(req)
File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 3036, in parse
req, = parse_requirements(s)
File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 2980, in parse_requirements
"version spec")
File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 2945, in scan_list
raise RequirementParseError(msg, line, "at", line[p:])
RequirementParseError: Expected version spec in ansible=={major: at =={major:
This is the playbook:
- name: Install Sensu
serial: "100%"
hosts: all
sudo: yes
roles:
- role: "ansible-pull"
server_type: "sensu"
ansible_version: "2"
Where the ansible-pull role depends on the ansible role in meta/main.yml.
Am I injecting the variable incorrectly in this case? Is there some problem with setting the variable in the dependent ansible-pull roll rather than directly in the ansible role?
It turns out that ansible_version is a magic variable set by Ansible.
Who knew?
Using an arbitrarily different but unused variable name does the trick.