Ansible lookup env with $ and inject into jj2 template not working - ansible

We're using Ansible playbook with GitLab CI in this project, where we'd pass some variables from ENV_FILE through Ansible playbook, then rendering JJ2 template with them.
Now the problem occurs when some variable has $ in its value, which seems interpreted as shell variable at some point, and the final value is rendered incorrect.
For example, in ENV_FILE
(set via GitLab CI Settings > CI/CD > Variables menu):
export FIRST_VAR=...
export SOME_VAR='123$abc#xyz'
export SOME_OTHER_VAR=...
And the final result in docker-compose.yaml becomes 123#xyz
EDIT: We just tried changing to export SOME_VAR='123''$''abc#xyz', the final result becomes 123abc#xyz, still missing the $.
gitlab-ci.yaml
deploy:
stage: deploy
environment:
name: dev
script:
- source $ENV_FILE
- cd ansible && ansible-playbook -i inventory/dev.ini runapp.yaml --vault-password-file=${ANSIBLE_VAULT_FILE}
runapp.yaml
- hosts: app
become: yes
roles:
- { role: some_app }
vars:
SOME_VAR: "{{ lookup('env', 'SOME_VAR') }}"
Task File:
- name: "Templating docker-compose file"
become: yes
template:
src: app-docker-compose.yaml.j2
dest: /opt/someapp/docker-compose.yaml
app-docker-compose.yaml.j2
someapp-svc:
image: someapp:version
restart: always
ports:
- ####:####
environment:
SOME_VAR: {{ SOME_VAR }}
Any hint about this?
Thanks!

I can reproduce that behavior when setting a CI/CD variable containing $; the docs kind of hint at it, although the docs are written as if the problem only applies when setting variables inside .gitlab-ci.yml which is demonstrably false
If you want a CI/CD variable to contain a literal $, it needs to be doubled, so SOME_VAR would need to be written as 123$$abc#xyz in the CI/CD configuration page in order for it to materialize as 123$abc#xyz inside the pipeline (although as the comments correctly point out, one will want to be exceedingly careful about the use of source to avoid further interpolation)

Related

Using gitlab-ci vars inside an ansible playbook

I want to set a remote environment inside a docker container using an Ansible playbook. This playbook will run from gitlab-ci with variables I set in in the Gitlab CI/CD confituration. How can I acheive that?
Here is the template I want to use. How do I set the user_id and password from the CI/CD variables?
tasks:
- name: Run XYZ Container
docker_container:
name: XYZ
restart_policy: on-failure
image: xxxxxxxxxxx
container_default_behavior: "compatibility"
env:
USER_ID= $USER_ID
PASSWORD= $PASSWORD
Since gitlab-ci variables are just environment variables inside your job, and since your ansible controller runs inside that job, you can use the env lookup to read them from the controller.
Please note that:
the docker_container module's env parameter expects a dict and not a new line separated string of bash like env vars definition like in your example.
as a security measure, you should either check that the vars are defined prior to using them (with an assert or fail task) or use a default value in case they're not. My example uses a default value. For more on providing default value, you can see the ansible documentation (and the original jinja2 documentation to understand that d is a an alias to default)
tasks:
- name: Run XYZ Container
docker_container:
name: XYZ
restart_policy: on-failure
image: xxxxxxxxxxx
container_default_behavior: "compatibility"
env:
USER_ID: "{{ lookup('env', 'USER_ID') | d('defaultuser', true) }}"
PASSWORD: "{{ lookup('env', 'PASSWORD') | d('defaultpass', true) }}"
i wanted to use the CI_JOB_TOKEN so i used:
tasks:
- include_role: role_name
vars:
ci_job_token: "{{ lookup('env', 'CI_JOB_TOKEN') }}"

Conditionals with custom variables

In my playbook i have tasks that use the hostname of a server and extrapolate data to set location and environment based on that. But some servers have unique names and I'm not sure how to set variables on those. I'd prefer not to use ansible facts since i would like to share the playbook with a team. One way I was thinking is to do what's listed below but i'm running into issues. Could someone please guide me.
Create vars_file inventory
---
customservers
customhostname1:
env: test
location: hawaii
customhostname2:
env: prod
location: alaska
In Playbook.
---
task
tasks:
- name: set hostname
shell: echo "customhostname1"
register: my_hostname
- name: setting env var
set_fact:
env: "{{ item.value.env }}"
when: my_hostname == "{{ item.key }}"
with_dict: "{{ customservers }}"
- name: outputing env var
debug:
msg: the output is {{ env }}
Expected output should be test.
Thank you.
In my playbook i have tasks that use the hostname of a server and
extrapolate data to set location and environment based on that.
Bad Idea.
But some servers have unique names and I'm not sure how to set variables on those
And that is why.
The second Bad Idea is to have TEST and PROD in the same inventory. That's just begging for a disaster. They should be two completely separate inventories, though perhaps under the same parent directory:
inventories/
inventories/test/
inventories/test/hosts
inventories/test/host_vars/
inventories/test/host_vars/customhostname1.yml
inventories/prod/
inventories/prod/hosts
inventories/prod/host_vars/
inventories/prod/host_vars/customhostname2.yml
So inventories/prod/hosts could look like this (I prefer the ini format):
[customservers]
customhostname2 location=hawaii
Or:
[customservers]
customhostname2
[customhostname2:vars]
location=hawaii
But in any case, DO NOT combine test and prod inventories.
If you still need that env variable, you can either put it in group_vars/all.yml or right in the hosts file like so:
[all:vars]
env=prod

Retrieving contents of j2 template file on stdout

I'm attempting to use Ansible to better manage my Kubernetes configmaps in a multienvironment project (dev, stage, and prod). I've generalized each of the config maps as j2 templates, and I'll override the variables depending on how they might change in different environments (so that they aren't duplicated three times for basically the same file).
My playbook currently looks something like this:
---
- hosts: localhost
vars_files:
- "vars/{{ env }}.yml"
tasks:
- name: Generate YAML from j2 template
template:
src: templates/foo.j2
dest: output/foo.yml
And this has been working great for testing so far. However, I'm at the point where I want to incorporate this into my already existing Jenkins CI/CD, but I'm having trouble understanding how it might work with what I am doing currently.
After generating what is basically a Kuberenets ConfigMap from the j2, I'll somehow do this within Jenkins:
kubectl apply -f <yaml>
However, the playbook is creating a YAML file every time I run it, and I am wondering if there is an alternative that would allow me to pipe the contents of the YAML file or somehow retrieve it from stdout.
Basically, I want to evaluate the template and retrieve it without necessarily creating a file.
If I do this, I could do something like the following:
echo result | kubectl apply -f -
where result of course is the contents of the YAML file that results after the templating, and the short dash after the f flag specifies Kubernetes to use the process' stdout.
Sorry for so much explaining, I can clarify anything if needed.
I would like to retrieve the result of the template, and pipe it into that command, such as "echo result | kubectl apply -f -"
In which case, you'd use the stdin: parameter of the command: module:
- name: generate kubernetes yaml
command: echo "run your command that generates yaml here"
register: k8s_yaml
- name: feed the yaml to kubectl apply
command: kubectl apply -f -
stdin: '{{ k8s_yaml.stdout }}'
It isn't super clear what the relationship is in your question between the top part, dealing with the template:, and the bottom part about apply -f -, but if you mean "how can I render a template to a variable, instead of a file?" the the answer is the template lookup plugin:
- name: render the yaml
set_fact:
k8s_yaml: '{{ lookup("template", "templates/foo.j2") }}'
- name: now feed it to apply
command: kubectl apply -f -
stdin: '{{ k8s_yaml }}'
You've got a couple options here. I usually try to stay away from shelling out to command wherever possible. Check out the k8s module in ansible. Note that as long as state is present ansible will patch your object.
- name: Apply your previously generated configmap if you so choose.
k8s:
state: present
definition: "{{ lookup('file', '/output/foo.yml') }}"
Or even better, you could just directly create the configmap
- name: Create the configmap for {{ env }}
k8s:
state: present
definition:
apiVersion: v1
kind: ConfigMap
metadata:
name: ConfigMap
namespace: "{{ foo_namespace }}"
labels:
app: bar
environment: "{{ bizzbang }}"

How to assign a value to variable dictionary from another dictionary variable

Hi I have a template in ansible yaml file:
env:
"{{ env }}"
this value is assigned when I executed:
ansible-playbook template.yaml --extra-vars vars.yaml
vars.yml has:
env:
CONSUL_HOST: 'x.x.x.x:8500'
SERVICE_NAME: 'backoffice-fe'
that works fine , now I want to add another variable in the template.yaml with the value of CONSUL_HOST
I tried without lucky:
env:
"{{ env }}"
NEW_VAR: env.CONSULT_HOST
I need to do this in template.yaml because it is used for a lot of modules , I don't want to modify all the modules because there are more than 100!
that is possible?
thanks in advance!

Ansible use task return varible in variable templates

I am trying to craft a list of environment variables to use in tasks that may have slightly different path on each host due to version differences.
For example, /some/common/path/v_123/rest/of/path
I created a list of these variables in variables.yml file that gets imported via roles.
roles/somerole/varables/main.yml contains the following
somename:
somevar: 'coolvar'
env:
SOME_LIB_PATH: /some/common/path/{{ unique_part.stdout }}/rest/of/path
I then have a task that runs something like this
- name: Get unique path part
shell: 'ls /some/common/path/'
register: unique_part
tags: workflow
- name: Perform some actions that need some paths
shell: 'binary argument argument'
environment: somename.env
But I get some Ansible errors about variables not being defined.
Alternatively I tried to predefine the unique_part.stdout in hopes of register overwriting predefined variable, but then I got other ansible errors - failure to template.
Is there another way to craft these variables based on command returns?
You can also use facts:
http://docs.ansible.com/set_fact_module.html
# Prepare unique variables
- hosts: myservers
tasks:
- name: Get unique path part
shell: 'ls /some/common/path/'
register: unique_part
tags: workflow
- name: Add as Fact per for each hosts
set_fact:
library_path: "{{ unique_part.stdout }}"
# launch roles that use those unique variables
- hosts: myservers
roles:
- somerole
This way you can dynamicaly add variable to your hosts before using them.
The vars files gets evaluated when it is read by Ansible. Your only chance would be to include a placeholder which you then later have to replace yourself, like this:
somename:
somevar: 'coolvar'
env:
SOME_LIB_PATH: '/some/common/path/[[ unique_part.stdout ]]/rest/of/path'
And then later in your playbook you can replace that placeholder:
- name: Perform some actions that need some paths
shell: 'binary argument argument'
environment: '{{ somename.env | replace("[[ unique_part.stdout ]]", unique_part.stdout) }}'

Resources