I'm trying to come up with a way of parameterizing some config files for a docker image using env vars. A sample input file could look something like:
<database>
<host>${DB_HOST}</host>
<name>${DB_NAME}</name>
<user>${DB_USER}</user>
<pass>${DB_PASSWORD}</pass>
</database>
and I'd want the output to basically be exactly the same, but with the fields filled in with the corresponding environment variable value.
Is there a simple way to do this directly in bash? The system I'm using doesn't natively support env vars, so I need to write these config files by hand, but I need to inject different vars for each environment I'm in so I don't particularly want to actually maintain real files for each env.
This is that simple as :
cat<<EOF > new_config_file
<database>
<host>${DB_HOST}</host>
<name>${DB_NAME}</name>
<user>${DB_USER}</user>
<pass>${DB_PASSWORD}</pass>
</database>
EOF
ls -l new_config_file
you're looking for envsubst
$ envsubst < config_template > config_instance
Just another option.
My generated config files often require just a bit of logic + embedding environment variable contents. At some point I got tired of reinventing the wheel every time and stitched together a simple tool in Go called Templater (https://github.com/reertech/templater).
It's just a standalone zero dependency binary for most of the systems, so you can just download it and use Go templating language (https://golang.org/pkg/text/template/) like that:
#!/bin/bash
YEAR=1997 \
NAME=John \
SURNAME=Connor \
./templater -t - -o /tmp/hello.txt <<EOF
Hello, {{env "NAME"}} {{env "SURNAME"}}!
{{ if env "YEAR" | parseInt | eq 1997 }}
Have a nice Judgement Day!
{{ end }}
EOF
In case you're not comfortable with prebuilt binaries you can use Go and build one yourself.
Hope this helps you or someone else who cannot cut it with just environment substitution and doesn't feel like Bashing.
Related
The real case:
Every time I set up my environment, I'd like to check and – create if are not existing – certain environment variables. So, instead of doing it manually all the time, I thought it would be great if I can have a file which stores the environment name env_N and environment value env_V pairs. Amongst all text file formats, the yaml looks the simplest and the more natural to store that info.
So what I thought it would be great if I suck in the yaml file with my environmental variables using yq and create associative array ready to be iterated over by zsh foreach loop:
foreach entry in my_assoc_arr
do
check_and_create(entry.env_N, entry.env_V)
done
with the final result of:
$ echo $env_N1
env_V1
$ echo $env_N2
env_V2
$ echo $env_N3
env_V3
...
The problem I'm having is to get my yaml to associative array using yq in shell zsh script.
After applying each suggestion from the comments, I was unable to create my associative array from yaml with yq. I had errors from yq like bad syntax or script worked or not depends on whether I have #!/bin/zsh switched on or commented out.
I got impression that my task is simple, but somehow I cant achieve this.
What I'm doing wrong here?
PS: I'm using zsh on macOS
Why would you want to use Yaml for this, when it’s much easier to write a simple shell script for it?
Simply create a text file with the following line for each env var:
# Set $FOO to 'bar' if $FOO does not yet exist.
export ${FOO=bar}
Then source this file from your shell (or another script).
See https://zsh.sourceforge.io/Doc/Release/Expansion.html#Parameter-Expansion
i'm working on my cicd in gitlab.
I have set up many environment variables, most of which are standard string/numbers. What I've done is to prefix them with "APP_" so that I can properly export to my project during cicd only the required variables. I do it this way:
export | grep APP_ | sed -e 's/APP_//g' | sed -e 's/declare -x //g' > ./app/settings/.env
This will basically take all the environement variables with APP_, remove the APP_, and store all of them in a file in ./app/settings/.env
This works like a charm
Now I'd like to do something similar for my file environment variables. What I've done is creating with a "FILE_" prefix, so I'd like to:
create one file per environement variable starting with "FILE_", naming the file as the environment variable name (but without the prefix FILE_)
store the files in .app/settings/files
How should I do so ?
At the moment i'm doing one by one but this isn't what I'd like:
echo "$FILE_MY_CERTIFICATE" > "./app/settings/files/my_certificate"
P.S. For those experts in gitlab env variables, I'm doing like this because I'm unable to use the standard "file" environement variable feature integrated in gitlab. The variables aren't in my build project so I'd like to find a workaround to suit my needs.
I do it this way:
Forget your code. It's just:
declare -p "${!APP_#}" > ./app/settings/.env
If you really specifically want to really parse something like export, use env -0.
How should I do so ?
declare -p "${!FILE_#}" > ./app/settings/files
I believe this may be a XY question and you should instead rethink your algorithm to use arrays or associative arrays instead.
I have to introduce some templating over text configuration files (yaml, xml, json) that already contain bash-like syntax variables. I need to preserve existing bash-like variables untouched but substitute my ones. List is dynamic, variables should come from environment. Something like simple processor taking "$${MY_VAR}}" pattern but ignoring $MY_VAR. Preferably pure Bash or as small number of tooling required as possible.
Pattern could be $$(VAR) or anything that can be easily separated from ${VAR} and $VAR. The key limitation - it is intended for a docker container startup procedure injecting environment variables into provided service configuration templates and this way building this configuration. So something like Java or even Perl processing is not an option.
Does anybody have a simple approach?
I was using the following bash processing for such variable substitution where original files had no variables. But now I need something one step smarter.
# process input file ($1) placing output into ($2) with shell variables substitution.
process_file() {
set -e
eval "cat <<EOF
$(<$1)
EOF
" | cat > $2
}
Obvious clean solution that is too complex for Docker file because of number of packages needed:
perl -p -i -e 's/\$\{\{([^}]+)\}\}/defined $ENV{$1} ? $ENV{$1} : $&/eg' < test.json
This filters out ${{VAR}}, even better - only set ones.
I'm trying to get going with some more advanced Ansible playbooks and have hit a wall. I'm trying to get Ansible to do what this /bin/bash 'for' loop does;
for i in $(</filename.txt);do '/some/command options=1 user=usera server=$i';done
filesnames.txt contains 50-100 hostnames.
I can't use jinja templates as the command has to be run, not just the config file updated.
Any help would be greatly appreciated.
Thanks in advance,
Jeremy
you can use jinja templates, but differently
your specific code is not doing something that is most advisable
for multi-line code you should use shell module.
example of multi-code piece of call:
- name: run multiline stuff
shell: |
for x in "${envvar}"; do
echo "${x}"
done
args:
executable: /bin/bash
note I'm explicitly setting executable, which will ensure bash-isms would work.
I just used envvar as an example, of arbitrary environment variable available.
if you need to pass specific env variables, you should use environment clause of the call to shell module, refer to: http://docs.ansible.com/ansible/playbooks_environment.html
For simple variables you can just use their value in shell: echo "myvar: {{myvar}}"
If you wish to use an ansible list/tuple variable inside bash code, you can make it bash variable first. e.g. if you have a list of stuff in mylist, you can expand it and assign into a bash array, and then iterate over it. the shell code of the call to shell would be:
mylist_for_bash=({{mylist|join(" ")}})
for myitem in "${mylist_for_bash[#]}"; do
echo "my current item: ${myitem}"
done
Another approach would be to pass it as string env variable, and convert it into an array later in the code.
NOTE:
of course all this works correctly only with SPACELESS values
I've never had to pass array with space containing items
Is there a way to check playbook syntax and variables?
I'm trying to dry-run(--check) but for some reasons it works really slow. It looks like it tries to perform an action instead of just check the syntax
I want to omit en errors like this:
..."msg": "AnsibleUndefinedVariable: ERROR! 'application_name' is undefined"}
This is expected behaviour according to the documentation:
When ansible-playbook is executed with --check it will not make any
changes on remote systems. Instead, any module instrumented to support
‘check mode’ (which contains most of the primary core modules, but it
is not required that all modules do this) will report what changes
they would have made rather than making them. Other modules that do
not support check mode will also take no action, but just will not
report what changes they might have made.
Old link (does not work anymore): http://docs.ansible.com/ansible/playbooks_checkmode.html
New link: https://docs.ansible.com/ansible/latest/user_guide/playbooks_checkmode.html#using-check-mode
If you would like to check the YAML syntax you can use syntax-check.
ansible-playbook rds_prod.yml --syntax-check
playbook: rds_prod.yml
I was looking for the same, but was not satisfied by the --syntax-check option, since it does not work its way down to the roles. A more complete check can be performed with ansible-lint which also includes style-checks. But if you turn off all style-checks, then you have a pretty complete syntax-check.
So do something like
ansible-lint -x $(echo $(ansible-lint -L | awk -F':' '{print $1}' | grep '^[^ ]') | tr ' ' ',') my_playbook.yml
Add a task to fail the playbook when variables aren't defined. This should be the first task run.
Another option is to ensure that all variables have a default value in the /defaults/ directory so that it never fails, but the variables can still be overwritten at other levels.
My preferd way is
pip install yamllint
yamllint -d "{extends: default, rules: {quoted-strings: enable}}" .
Since I really want to catch quote errors, e.g.
validate: bash -c ' ' \""
This is valid yaml, since yaml will just quote the string and turn it into:
validate: "bash -c ' ' \\\"\""
Whilst there was just clearly a quote missing at the beginning of the validate comand.
So a normal yaml checker will not detect this, yamllint wil not even detect this in it's default configuration, so turn on quoted-strings checker.