I have my environment variables stored in a YAML file. The YAML file is used by a third party service for deployment.
I was wondering if there is a way to source the YAML file I am using, so that I can get access to my database credentials to run a migration once the app has been deployed?
example YAML:
env_variables:
DATABASE_CONNECTION_ADDRESS: 'localhost'
DATABASE_PORT: '5432'
DATABASE_NAME: 'a-db'
DATABASE_USERNAME: 'user'
DATABASE_PASSWORD: 'password'
IS_DEBUG: 'false'
GS_BUCKET_NAME: image-bucket
My main motivation is that this deployment is running in a pipeline and I do not want to maintain the duplication of each of these environment variables in their own secret, and storing this YAML file as a secret so the third party service has access to it.
If you have Python installed in your environment and can install ruamel.yaml in there you can source the output of the following one-liner:
python -c 'from pathlib import Path; from ruamel.yaml import YAML; print("".join([f"{k}={v}\n" for k, v in YAML().load(Path("example.yaml"))["env_variables"].items()]))'
Its output is:
DATABASE_CONNECTION_ADDRESS=localhost
DATABASE_PORT=5432
DATABASE_NAME=a-db
DATABASE_USERNAME=user
DATABASE_PASSWORD=password
IS_DEBUG=false
GS_BUCKET_NAME=image-bucket
As Jeff Schaller suggested you probably want to quote the values and escape any single quotes that might occur in the string. This can easily be achieved by changing {v} into {v!r} in the one-liner.
As program:
#!/usr/bin/env python3
from pathlib import Path
from ruamel.yaml import YAML
file_in = Path("example.yaml")
yaml = YAML()
env_data = yaml.load(file_in)["env_variables"]
print("".join([f"{k}={v!r}\n" for k, v in env_data.items()]))
Related
I'm doing one project for puppet, however currently stuck in one logic.
Thus, want to know can we fetch variable from .yaml, .json or plain text file in puppet manifest file.
For example,
My puppet manifest want to create user but the variable exist in the .yaml or any configuration file, hence need to fetch the varibale from the outside file. The puppet manifest also can do looping if it exist multiple users in .yaml file.
I read about hiera but let say we are not using hiera is there any possible way.
There are a number of ways you can do this using a combination of built-in and stdlib functions, at least for YAML and JSON.
Using the built-in file function and the parseyaml or parsejson stdlib functions:
Create a file at mymodule/files/myfile.yaml:
▶ cat files/myfile.yaml
---
foo: bar
Then in your manifests read it into a string and parse it:
$myhash = parseyaml(file('mymodule/myfile.yaml'))
notice($myhash)
That will output:
Notice: Scope(Class[mymodule]): {foo => bar}
Or, using the loadyaml or loadjson stdlib functions:
$myhash = loadyaml('/etc/puppet/data/myfile.yaml')
notice($myhash)
The problem with that approach is that you need to know the path to file on the Puppet master. Or, you could use a Puppet 6 deferred function and read the data from a file on the agent node.
(Whether or not you should do this is another matter entirely - hint: answer is you almost certainly should be using Hiera - but that isn't the question you asked.)
I wanted to print a YAML file (with keys and values) and with some values between double quotes.
I wanted to use the solution provide here: How to print a value with double quotes and spaces in YAML?
Unfortunately when I installed ruamel.yaml for python 3.5 (sudo apt install python3-ruamel.yaml) I was not able to find the function DoubleQuotedScalarString() in the script scalarstring.py .
Here is what it looks like:
from __future__ import absolute_import
from __future__ import print_function
__all__ = ["ScalarString", "PreservedScalarString"]
try:
from .compat import text_type
except (ImportError, ValueError): # for Jython
from ruamel.yaml.compat import text_type
class ScalarString(text_type):
def __new__(cls, *args, **kw):
return text_type.__new__(cls, *args, **kw)
class PreservedScalarString(ScalarString):
def __new__(cls, value):
return ScalarString.__new__(cls, value)
def preserve_literal(s):
return PreservedScalarString(s.replace('\r\n', '\n').replace('\r', '\n'))
def walk_tree(base):
"""
the routine here walks over a simple yaml tree (recursing in
dict values and list items) and converts strings that
have multiple lines to literal scalars
"""
from ruamel.yaml.compat import string_types
if isinstance(base, dict):
for k in base:
v = base[k]
if isinstance(v, string_types) and '\n' in v:
base[k] = preserve_literal(v)
else:
walk_tree(v)
elif isinstance(base, list):
for idx, elem in enumerate(base):
if isinstance(elem, string_types) and '\n' in elem:
print(elem)
base[idx] = preserve_literal(elem)
else:
walk_tree(elem)
Currently this is what I obtain when using ruamel.yamp.dump() to get my yaml file:
key1: 0,0,0,0
key2: 0,0,0,0
And here is what I would like in my yaml file:
key1: "0,0,0,0"
key2: "0,0,0,0"
How am I suppose to solve this?
The class DoubleQuotedScalarString was added 2016-07-06.
You should update your version of ruamel.yaml e.g. using pip install -U ruamel.yaml. You can see what version you have by looking at the __init__.py in the yaml directory. The current version (June 2018) has:
version_info=(0, 15, 37),
In general it is best not to install additional python packages under Linux, using the package manager. Instead create a virtualenv for each program you want to develop and install all packages necessary for that program in such a virtualenv.
So I found a quick and dirty solution to complete Anthon's answer:
First of all, Anthon is right I should work in a virtual environment. With my current app, I was lazy, so I didn't, that's a mistake.
I found that sudo apt install python3-ruamel.yaml was not installing the package in the same dir than sudo pip3 install ruamel.yaml.
The package versions are also not the same which is the reason why I got confused.
Here is the quick and dirty solution that worked for me:
copy all ".py" files from the source (https://pypi.org/project/ruamel.yaml/#files) inside the dir where your ruamel.yaml package is installed (/my/default/python3-lib/ruamel/yaml/) with e.g. mv /home/user/Downloads/ruamel.yaml-0.15.38/*.py /my/default/python3-lib/ruamel/yaml/
It solved the problem for me.
As Anthon already answered, the clean solution is to use a virtual env.
Trying to find a way to turn off the red lines temporarily for that file only.
maybe try to disable the yaml.schemaStore ?
go in in settings.json and add :
"yaml.schemaStore.enable": false
Since this is not valid YAML at all, but you want to edit this as YAML,
you should make it into valid YAML. If you turn of the errors,
instead you probably would not have all of the advantage of the YAML
editing mode.
If saltstate allows you to change the block_start_string and
variable_start_string jinja2 uses you can change {% into #% (or
###% if #% and ###% naturally occur in your source), and also
change {{ into <{ (or <<{, you get the idea). If you would call
jinja2 directly you would then then pass to the FireSystemLoader:
block_start_string='<{' and variable_start_string='#%' If the
above is possible, then you have to change your input file only once,
do that with an editor.
If you cannot control saltstate to do the sane thing, your still not
stuck but you have to do a bit more using Python,
ruamel.yaml and some
support packages (disclaimer: I am the author of those packages).
Install with:
pip install ruamel.yaml[jinja2] ruamel.std.pathlib
Then before editing run the program:
from ruamel.yaml import YAML
from ruamel.std.pathlib import Path
yamlj2 = YAML(typ='jinja2')
yamlrt = YAML()
yaml_flow_style = YAML()
yaml_flow_style.default_flow_style = True
in_file = Path('init.sls')
backup_file = Path('init.sls.org')
in_file.copy(backup_file)
data = yamlj2.load(in_file)
with in_file.open('w') as fp:
# write the header with info needed for revers
fp.write('# ruamel.yaml.jinja2: ') # no EOL
yaml_flow_style.dump(yamlj2._plug_in_jinja2, fp)
yamlrt.dump(data, fp)
which changes the offending jinja2 sequences and add a one-line header comment with the actual patterns used to the file. You should then be able
to edit the init.sls file without getting all those errors.
Before calling saltstate, do run the following:
from ruamel.yaml import YAML
from ruamel.std.pathlib import Path
in_file = Path('init.sls')
yamlj2 = YAML(typ='jinja2')
yamlrt = YAML()
yamlnort = YAML(typ='safe')
with in_file.open() as fp:
yamlj2._plug_in_jinja2 = yamlnort.load(fp.readline().split(':', 1)[1])
data = yamlrt.load(fp)
yamlj2.dump(data, in_file)
If you have multiple of these files, you probably want to take your
filename from sys.argv[1]. You might actually call the salstate program from this second Python program (i.e. decode and run).
I want to control the verbosity of ansible playbooks using an environment variable or a global configuration item. This is because ansible is called from multiple places in multiple ways and I want to change the logging level for all further executions from the same shell session.
I observed that if I configure ANSIBLE_DEBUG=true ansible will run in debug mode but debug more is extremely verbose and I am only looking for something similar to the -vvv option (DEBUG is more verbose than even -vvvv option)
I tried to look for a variable inside https://github.com/ansible/ansible/blob/devel/lib/ansible/constants.py but I wasn't able to find one this fits the bill.
I see two ways to do this:
alias
alias ansible-playbook="echo 'This is -vv alias'; ansible-playbook -vv"
This way your shell will call ansible-playbook -vv when you type ansible-playbook (and print friendly reminder about alias).
callback plugin
Drop this code as verbosity_env.py file into callback_plugins directory:
from ansible.plugins.callback import CallbackBase
import os
try:
from __main__ import display
except ImportError:
display = None
class CallbackModule(CallbackBase):
def v2_playbook_on_start(self, playbook):
v = os.environ.get('ANSIBLE_VERBOSITY')
if v and display:
display.display('Verbosity is set to {} with environment variable.'.format(v), color='blue')
display.verbosity = int(v)
It is not production quality code, but does the job. This will check ANSIBLE_VERBOSITY environment variable and set display verbosity with its value.
Not sure why I missed to answer this, since a long time ago ansible fully supports ANSIBLE_VERBOSITY=[0|1|2|3|4].
For reference, ansible documentation
You can create/edit ansible.cfg file in the local folder and add in section [defaults]:
[defaults]
verbosity=4
Alternatively you can add the same option in /etc/ansible/ansible.cfg file.
I can't find it documented anywhere other than sorin's answer, but if you set ANSIBLE_VERBOSITY=[0|1|2|3|4] environment variable, ansible-playbook will pick this up and use it, unless you specify the verbosity on the command line.
E.g. in Unix-type shells:
export ANSIBLE_VERBOSITY=2
ansible-playbook my-playbook.yml
I only stumbled upon it because I tried setting ANSIBLE_VERBOSITY='-vv' in a pipeline and Ansible started moaning about it not being an integer!
I have had no success in locating a good precommit hook I can use to validate that a Jinja2 formatted file is well-formed without attempting to substitute variables. The goal is something that will return a shell code of zero if the file is well-formed without regard to whether variable are available, 1 otherwise.
You can do this within Jinja itself, you'd just need to write a script to read and parse the template.
Since you only care about well-formed templates, and not whether or not the variables are available, it should be fairly easy to do:
#!/usr/bin/env python
# filename: check_my_jinja.py
import sys
from jinja2 import Environment
env = Environment()
with open(sys.argv[1]) as template:
env.parse(template.read())
or something that iterates over all templates
#!/usr/bin/env python
# filename: check_my_jinja_recursive.py
import sys
import os
from jinja2 import Environment, FileSystemLoader
env = Environment(loader=FileSystemLoader('./mytemplates'))
templates = [x for x in env.list_templates() if x.endswith('.jinja2')]
for template in templates:
t = env.get_template(template)
env.parse(t)
If you have incorrect syntax, you will get a TemplateSyntaxError
So your precommit hook might look like
python check_my_jinja.py template.jinja2
python check_my_jinja_recursive.py /dir/templates_folder