Ansible script to create new kafka connector - ansible

I have been creating different connectors in our confluent installation by using the curl command to PUT a configuration to a url. The set-up is normally done by adding the configuration and curl call in a shell script. We either write separate shell scripts for each environment or pass the server names - broker and connect - as parameters.
I want to write an ansible script to define new connectors which will basically be same script that can be run in all environments and will only depend on the hosts file in each environment. This will help in eliminating user error / typos in server names when running it in different environments.
My current plan is to just convert the shell script and use ansible shell or command module to call 'curl' and pass the parameters. But I am not sure, if that is the correct way, even if it may get the job done.
Does anyone have any suggestion to do it correctly / properly.
Thank you
UPDATE
Based on #Zeitounator's recommendation, I tried to use the uri module. But keep on getting error
---
- name: Create Kafka Connector
hosts: kafka_connect
vars:
connect_url: https://{{ kafka_connect[0] }}:8083
connector_name: CamelTestConnector
cert_path: /tmp/test/
tasks:
- name: prepare connector config
template:
src: camelconnector.json
dest: /tmp/camelconnector.json
delegate_to: localhost
run_once: yes
- name: Create Connector
uri:
url: "https://kafka-connect1-dev:8083/connectors/{{ connector_name }}/config"
client_cert: "{{ cert_path }}certificate.pem"
client_key: "{{ cert_path }}priv.key"
method: PUT
body: "{{ lookup('file', '/tmp/camelconnector.json' ) }}"
body_format: json
validate_certs: no
status_code: [201, 201, 204]
headers:
Content-Type: application/json
# If you're interested in the response
return_content: yes
register: api_result
run_once: yes
- debug:
var=api_result
#curl --cert /software/scripts/clientcerts/certificate.pem --key /software/scripts/clientcerts/priv.key -k -X PUT -H "${HEADER}" --data "${DATA}" https://"${1}":8083/connectors/CamelTestConnector/config
#curl --cert /software/scripts/clientcerts/certificate.pem --key /software/scripts/clientcerts/priv.key -k https://"${1}":8083/connectors/CamelTestConnector/status
This is the error I keep getting
TASK [Create Connector] ***********************************************************************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: IOError: [Errno 2] No such file or directory
fatal: [kafka-connect1-dev]: FAILED! => {"changed": false, "content": "", "elapsed": 0, "msg": "Status code was -1 and not [201, 201, 204]: An unknown error occurred: [Errno 2] No such file or directory", "redirected": false, "status": -1, "url": "https://kafka-connect1-dev/connectors/"}
I have verified that the json file is present. I am unable to identify which file it is complaining about. Tried various iterations for connect-url, but all give the same error.
How do I identify which file it is complaining about?
Thanks
UPDATE 2
I ran the script in verbose mode and I get below stack trace. If I comment out the cert and key parts, then I get connection refused, so it seems the cert and key are required. I tried to search for the cause of the error without success. The certificate and key files do exist at the location. What else do I need to check?
The full traceback is:
Traceback (most recent call last):
File "/tmp/ansible_uri_payload_6FB8tM/ansible_uri_payload.zip/ansible/module_utils/urls.py", line 1494, in fetch_url
unix_socket=unix_socket, ca_path=ca_path)
File "/tmp/ansible_uri_payload_6FB8tM/ansible_uri_payload.zip/ansible/module_utils/urls.py", line 1390, in open_url
unredirected_headers=unredirected_headers)
File "/tmp/ansible_uri_payload_6FB8tM/ansible_uri_payload.zip/ansible/module_utils/urls.py", line 1294, in open
r = urllib_request.urlopen(*urlopen_args)
File "/usr/lib64/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib64/python2.7/urllib2.py", line 431, in open
response = self._open(req, data)
File "/usr/lib64/python2.7/urllib2.py", line 449, in _open
'_open', req)
File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
result = func(*args)
File "/tmp/ansible_uri_payload_6FB8tM/ansible_uri_payload.zip/ansible/module_utils/urls.py", line 467, in https_open
return self.do_open(self._build_https_connection, req)
File "/usr/lib64/python2.7/urllib2.py", line 1183, in do_open
h = http_class(host, timeout=req.timeout, **http_conn_args)
File "/tmp/ansible_uri_payload_6FB8tM/ansible_uri_payload.zip/ansible/module_utils/urls.py", line 480, in _build_https_connection
return httplib.HTTPSConnection(host, **kwargs)
File "/usr/lib64/python2.7/httplib.py", line 1259, in __init__
context.load_cert_chain(cert_file, key_file)
IOError: [Errno 2] No such file or directory

Related

Use ansible template for Helm's value files with multiline quoted strings

I'm trying to write an ansible role to install helm-charts.
The role structure, simplified for posting, is something like this:
my_role
└───defaults
└───main.yml
└───tasks
└───main.yml
└───templates
└───main.yml
The role's default value file includes a variable chart_values used to pass the chart's values.
It is used only in the template/main.yml file, inside it there is simply the variable expansion, like this:
"{{ chart_values }}"
In tasks/main.yml the template is used as follows:
- name: Deploy chart.
kubernetes.core.helm:
release_name: "{{ chart_release_name }}"
release_namespace: "{{ chart_release_namespace }}"
release_state: present
chart_ref: "{{ chart_chart_ref }}"
chart_version: "{{ chart_chart_version }}"
update_repo_cache: true
release_values: "{{ lookup('template', 'chart_values.yml') | from_yaml }}"
Values seem to be rendered correctly, except when there is a multiline object containing quotes.
For instance, while deploying gitlab runners, a toml configuration snippet is required.
When I try to run the playbook with
chart_values:
replicas: 5
runners:
config: |
[[runners]]
[runners.kubernetes]
namespace = "gitlab-runners"
image = "ubuntu:20.04"
Ansible returns the following error:
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: in "<unicode string>", line 2, column 770
fatal: [127.0.0.1]: FAILED! => {"changed": false}
The more verbose output is:
The full traceback is:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/ansible/executor/task_executor.py", line 504, in _execute
self._task.post_validate(templar=templar)
File "/usr/lib/python3/dist-packages/ansible/playbook/task.py", line 285, in post_validate
super(Task, self).post_validate(templar)
File "/usr/lib/python3/dist-packages/ansible/playbook/base.py", line 650, in post_validate
value = templar.template(getattr(self, name))
File "/usr/lib/python3/dist-packages/ansible/template/__init__.py", line 913, in template
d[k] = self.template(
File "/usr/lib/python3/dist-packages/ansible/template/__init__.py", line 869, in template
result = self.do_template(
File "/usr/lib/python3/dist-packages/ansible/template/__init__.py", line 1139, in do_template
res = j2_concat(rf)
File "<template>", line 12, in root
File "/usr/lib/python3/dist-packages/ansible/template/__init__.py", line 298, in wrapper
ret = func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/ansible/plugins/filter/core.py", line 207, in from_yaml
return yaml_load(text_type(to_text(data, errors='surrogate_or_strict')))
File "/usr/local/lib/python3.8/dist-packages/yaml/__init__.py", line 81, in load
return loader.get_single_data()
File "/usr/local/lib/python3.8/dist-packages/yaml/constructor.py", line 49, in get_single_data
node = self.get_single_node()
File "yaml/_yaml.pyx", line 674, in yaml._yaml.CParser.get_single_node
File "yaml/_yaml.pyx", line 860, in yaml._yaml.CParser._parse_next_event
yaml.parser.ParserError: did not find expected <document start>
in "<unicode string>", line 2, column 770
fatal: [127.0.0.1]: FAILED! => {
"changed": false
}

Lookup secrets from AWS secret manager | Ansible

Using Terraform code I have created Other type of secrets in AWS Secrets Manager.
I need to use these AWS secrets in Ansible code. I found this below link but I am unable to proceed it.
https://docs.ansible.com/ansible/2.8/plugins/lookup/aws_secret.html
I have below Ansible code:-
database.yml
- name: Airflow | DB | Create MySQL DB
mysql_db:
login_user: "{{ mysql_user }}"
# login_password: "{{ mysql_root_password }}"
login_password: "{{ lookup('ca_dev', 'mysql_root_password') }}"
# config_file: /etc/my.cnf
# login_unix_socket: /var/lib/mysql/mysql.sock
# encrypted: yes
name: "airflow"
state: "present"
How can I incorporate AWS secret Manager in my ansible code?
Error message:-
TASK [../../roles/airflow : Airflow | DB | Create MySQL DB] **************************************************************************************************************************************************************************
task path: /home/ec2-user/cng-ansible/roles/airflow/tasks/database.yml:25
The full traceback is:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 140, in run
res = self._execute()
File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 539, in _execute
self._task.post_validate(templar=templar)
File "/usr/lib/python2.7/site-packages/ansible/playbook/task.py", line 267, in post_validate
super(Task, self).post_validate(templar)
File "/usr/lib/python2.7/site-packages/ansible/playbook/base.py", line 364, in post_validate
value = templar.template(getattr(self, name))
File "/usr/lib/python2.7/site-packages/ansible/template/__init__.py", line 540, in template
disable_lookups=disable_lookups,
File "/usr/lib/python2.7/site-packages/ansible/template/__init__.py", line 495, in template
disable_lookups=disable_lookups,
File "/usr/lib/python2.7/site-packages/ansible/template/__init__.py", line 746, in do_template
res = j2_concat(rf)
File "<template>", line 8, in root
File "/usr/lib/python2.7/site-packages/jinja2/runtime.py", line 193, in call
return __obj(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/ansible/template/__init__.py", line 631, in _lookup
instance = self._lookup_loader.get(name.lower(), loader=self._loader, templar=self)
File "/usr/lib/python2.7/site-packages/ansible/plugins/loader.py", line 381, in get
obj = getattr(self._module_cache[path], self.class_name)
AttributeError: 'module' object has no attribute 'LookupModule'
fatal: [127.0.0.1]: FAILED! => {
"msg": "Unexpected failure during module execution.",
"stdout": ""
}
RUNNING HANDLER [../../roles/airflow : restart rabbitmq-server]
task path: /home/ec2-user/cng-ansible/roles/airflow/handlers/main.yml:28
to retry, use: --limit #/home/ec2-user/cng-ansible/plays/airflow/installAirflow.retry
PLAY RECAP
127.0.0.1 : ok=39 changed=7 unreachable=0 failed=1
ansible-doc -t lookup -l output
The error {"msg": "lookup plugin (ca_dev) not found"} suggests your issue is the misuse of the lookup command.
The following line:
login_password: "{{ lookup('ca_dev', 'mysql_root_password') }}"
Should look something like
login_password: "{{ lookup('aws_secret', 'mysql_root_password') }}"
ca_dev is not a valid lookup type, whereas aws_secret is.
You can see a list of supported lookup plugins for Ansible 2.8 in the Lookup Plugins section of the official documentation.
If you are using a custom lookup plugin, or backporting a plugin from a future version of ansible to an older version, you must make sure that it is in a directory visible to ansible.
You can either place the custom file in the default location ansible looks in ~/.ansible/plugins/lookup:/usr/share/ansible/plugins/lookup or configure your ansible.cfg to look in a different place using the following lookup_plugins ini key under the defaults section.
DEFAULT_LOOKUP_PLUGIN_PATH
Description: Colon separated paths in which Ansible will search for Lookup Plugins.
Type: pathspec
Default: ~/.ansible/plugins/lookup:/usr/share/ansible/plugins/lookup
Ini Section: defaults
Ini Key: lookup_plugins
Environment: ANSIBLE_LOOKUP_PLUGINS
Documentation for this can be found in the Ansible Configuration section of the official documentation

Uploading File to Nexus 9000 with Ansible Playbook

Trying to automate file uploading process to the Cisco Nexus 9000 device with Ansible Playbook.
- name: Upload File to Nexus Device
gather_facts: no
hosts: localhost
connection: local
tasks:
- name: Uploading file
nxos_file_copy:
provider:
username: admin
ssh_keyfile: "files/id_rsa"
host: 192.168.0.100
validate_certs: no
transport: cli
local_file: "config.txt"
remote_file: "config.txt"
When running the playbook I am getting following error
The full traceback is:
Traceback (most recent call last):
File "/tmp/ansible_4UEm25/ansible_module_nxos_file_copy.py", line 255, in <module>
main()
File "/tmp/ansible_4UEm25/ansible_module_nxos_file_copy.py", line 245, in main
transfer_file(module, dest)
File "/tmp/ansible_4UEm25/ansible_module_nxos_file_copy.py", line 174, in transfer_file
port=port)
File "/home/user/venv/ansible-cox/local/lib/python2.7/site-packages/paramiko/client.py", line 424, in connect
passphrase,
File "/home/user/venv/ansible-cox/local/lib/python2.7/site-packages/paramiko/client.py", line 714, in _auth
raise saved_exception
paramiko.ssh_exception.PasswordRequiredException: Private key file is encrypted
fatal: [localhost]: FAILED! => {
"changed": false,
"module_stderr": "Traceback (most recent call last):\n File \"/tmp/ansible_4UEm25/ansible_module_nxos_file_copy.py\", line 255, in <module>\n main()\n File \"/tmp/ansible_4UEm25/ansible_module_nxos_file_copy.py\", line 245, in main\n transfer_file(module, dest)\n File \"/tmp/ansible_4UEm25/ansible_module_nxos_file_copy.py\", line 174, in transfer_file\n port=port)\n File \"/home/user/venv/ansible-cox/local/lib/python2.7/site-packages/paramiko/client.py\", line 424, in connect\n passphrase,\n File \"/home/user/venv/ansible-cox/local/lib/python2.7/site-packages/paramiko/client.py\", line 714, in _auth\n raise saved_exception\nparamiko.ssh_exception.PasswordRequiredException: Private key file is encrypted\n",
"module_stdout": "",
"msg": "MODULE FAILURE",
"rc": 1
}
When SSH-ing from local terminal with the same key it works
(ansible) ~/B/c/Ansible on master ⨯ ssh admin#192.168.0.100 -i files/id_rsa
User Access Verification
NX9K#
I am using Ansible 2.5.4 version.
Any help is greatly appreciated!

Ansible bigip_command module

Is there a way you can ignore an error "wait_for" throws when a conditional statement hasn't been satisfied?
In my play, I have a task to see which LTM in the pair is active, and it fails when it hits the standby (which makes sense).
But it would be nice if you could ignore this error.
Traceback (most recent call last):
File "/tmp/ansible_yIW5Ex/ansible_module_bigip_command.py", line 691, in <module>
main()
File "/tmp/ansible_yIW5Ex/ansible_module_bigip_command.py", line 680, in main
results = mm.exec_module()
File "/tmp/ansible_yIW5Ex/ansible_module_bigip_command.py", line 617, in exec_module
result = manager.exec_module()
File "/tmp/ansible_yIW5Ex/ansible_module_bigip_command.py", line 409, in exec_module
changed = self.execute()
File "/tmp/ansible_yIW5Ex/ansible_module_bigip_command.py", line 498, in execute
raise FailedConditionsError(errmsg, failed_conditions)
ansible.module_utils.network.common.parsing.FailedConditionsError: One or more conditional statements have not been satisfied.
fatal: [x.x.x.x -> localhost]: FAILED! => {
"changed": false,
"module_stderr": "Traceback (most recent call last):\n File \"/tmp/ansible_yIW5Ex/ansible_module_bigip_command.py\", line 691, in <module>\n main()\n File \"/tmp/ansible_yIW5Ex/ansible_module_bigip_command.py\", line 680, in main\n results = mm.exec_module()\n File \"/tmp/ansible_yIW5Ex/ansible_module_bigip_command.py\", line 617, in exec_module\n result = manager.exec_module()\n File \"/tmp/ansible_yIW5Ex/ansible_module_bigip_command.py\", line 409, in exec_module\n changed = self.execute()\n File \"/tmp/ansible_yIW5Ex/ansible_module_bigip_command.py\", line 498, in execute\n raise FailedConditionsError(errmsg, failed_conditions)\nansible.module_utils.network.common.parsing.FailedConditionsError: One or more conditional statements have not been satisfied.\n",
"module_stdout": "",
"msg": "MODULE FAILURE",
"rc": 1
And here is the task I'm working with: (Note: the host inventory file contains 2 IPs only)
- name : Checking which LTM is active....
bigip_command:
server: "{{ inventory_hostname }}"
user: "{{ remote_username }}"
password: "{{ remote_passwd }}"
commands:
- "tmsh show sys failover"
- "tmsh list /sys management-ip | grep -o x.x.x.x"
wait_for:
- result[0] contains active
validate_certs: no
delegate_to: localhost
Please check the module documentation here -- There is a "match" directive that defaults to "all" which implies that in the "wait_for" section all checks should pan out as true. You can explicitly set this to "any" which would make your run pass if at least 1 condition in "wait_for" is met.
You also have a "retries" and "interval" flag which will tell the module how many retries to make and how far apart they should be, respectively.
I hope this helps.
Cheers!

Use Hashicorp Vault with Ansible - plugin setup

I want to use Hashicorp Vault with Ansible to retrieve username/password which I will use in Ansible playbook.
Vault is setup - I created a secret. What are the steps to integrate both? the documentation around plugins isn't that great. I tried the file lookup from ansible and this works but how to use 3rd party plugins? Can somebody help me with the steps to follow?
Install the plugin, pip install ansible-modules-hashivault
What is the difference with https://github.com/jhaals/ansible-vault
2.a The environment variables (VAULT ADDR & VAULT TOKEN) I put where?
Change ansible.cfg to point to vault.py which is located in "plugin" folder of my Ansible Project
To test basic integration, can I use the following playbook?
https://pypi.python.org/pypi/ansible-modules-hashivault
- hosts: localhost
-tasks:
- hashivault_status:
register: 'vault_status'
Tried this but I get:
An exception occurred during task execution. The full traceback is:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 119, in run
res = self._execute()
File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 431, in _execute
self._task.post_validate(templar=templar)
File "/usr/lib/python2.7/site-packages/ansible/playbook/task.py", line 248, in post_validate
super(Task, self).post_validate(templar)
File "/usr/lib/python2.7/site-packages/ansible/playbook/base.py", line 371, in post_validate
value = templar.template(getattr(self, name))
File "/usr/lib/python2.7/site-packages/ansible/template/__init__.py", line 359, in template
d[k] = self.template(variable[k], preserve_trailing_newlines=preserve_trailing_newlines, fail_on_undefined=fail_on_undefined, overrides=overrides)
File "/usr/lib/python2.7/site-packages/ansible/template/__init__.py", line 331, in template
result = self._do_template(variable, preserve_trailing_newlines=preserve_trailing_newlines, escape_backslashes=escape_backslashes, fail_on_undefined=fail_on_undefined, overrides=overrides)
File "/usr/lib/python2.7/site-packages/ansible/template/__init__.py", line 507, in _do_template
res = j2_concat(rf)
File "<template>", line 8, in root
File "/usr/lib/python2.7/site-packages/jinja2/runtime.py", line 193, in call
return __obj(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/ansible/template/__init__.py", line 420, in _lookup
instance = self._lookup_loader.get(name.lower(), loader=self._loader, templar=self)
File "/usr/lib/python2.7/site-packages/ansible/plugins/__init__.py", line 339, in get
self._module_cache[path] = self._load_module_source('.'.join([self.package, name]), path)
File "/usr/lib/python2.7/site-packages/ansible/plugins/__init__.py", line 324, in _load_module_source
module = imp.load_source(name, path, module_file)
File "/etc/ansible/ProjectA/lookup_plugins/vault.py", line 5
<!DOCTYPE html>
^
SyntaxError: invalid syntax
fatal: [win01]: FAILED! => {
"failed": true,
"msg": "Unexpected failure during module execution.",
"stdout": ""
Since you put so many eggs into the post, that I have no clue what the question is really about, here's something to get you going with the native lookup plugin and jhaals/ansible-vault.
you can create lookup_plugins in the current directory and save vault.py inside;
the VAULT_ADDR and VAULT_TOKEN environment variables are as you see them in the script;
The Bash script below (it uses screen and jq, you might need to install them) runs Vault in dev mode, sets the secret, and runs Ansible playbook which queries the secret with two lookup plugins:
#!/bin/bash
set -euo pipefail
export VAULT_ADDR=http://127.0.0.1:8200
if [[ ! $(pgrep -f "vault server -dev") ]]; then
echo \"vault server -dev\" not running, starting...
screen -S vault -d -m vault server -dev
printf "sleeping for 3 seconds\n"
sleep 3
else
echo \"vault server -dev\" already running, leaving as is...
fi
vault write secret/hello value=world excited=yes
export VAULT_TOKEN=$(vault token-create -format=json | jq -r .auth.client_token)
ansible-playbook playbook.yml --extra-vars="vault_token=${VAULT_TOKEN}"
and playbook.yml:
---
- hosts: localhost
connection: local
tasks:
- name: Retrieve secret/hello using native hashi_vault plugin
debug: msg="{{ lookup('hashi_vault', 'secret=secret/hello token={{ vault_token }} url=http://127.0.0.1:8200') }}"
- name: Retrieve secret/hello using jhaals vault lookup
debug: msg="{{ lookup('vault', 'secret/hello') }}"
In the end you should get:
TASK [Retrieve secret/hello using native hashi_vault plugin] *******************
ok: [localhost] => {
"msg": "world"
}
TASK [Retrieve secret/hello using jhaals vault lookup] *************************
ok: [localhost] => {
"msg": {
"excited": "yes",
"value": "world"
}
}
The word world was fetched from Vault.

Resources