Say I have something like:
Stage one:
stage: one
extends: .build
variables:
DOCKERFILE: "one.Dockerfile"
Stage two:
stage: two
extends: .build
variables:
DOCKERFILE: "two.Dockerfile"
Does YAML have the capability to do something like:
names = ["one", "two"]
for name in names:
Stage name:
stage: name
extends: .build
variables:
DOCKERFILE: "name.Dockerfile"
I know there exist e.g. anchors, but I don't think they'd allow for this. I am pretty sure YAML doesn't have anything like a for loop specifically, but I'm hoping it has something that'd allow me to parameterize sections like this. Does it?
Related
How do I specify or change an environment variable as part of a job matrix?
For example, here I want my job to echo "Two" and "Three" instead of "One", but github completely ignores the definition or change of the environment variable in the matrix:
name: test-test-test
on:
push:
branches: [ main, develop ]
pull_request:
branches: [ main, develop ]
env:
MY_VAR: One
jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
include:
- env:
MY_VAR: Two
- env:
MY_VAR: Three
steps:
- run: echo "$MY_VAR2"
yields echo "One" and echo "One" in both jobs.
Use this context to access information about environment variables that have been set in a workflow, step, or job. Note that you cannot use the env context in the value of the id and uses keys within a step. An example is env.env_var.
GitHub provides extensive documentation about contexts at
https://docs.github.com/en/actions/learn-github-actions/contexts.
The syntax you used won't work for matrix. As stated in the official documentation:
The variables that you define become properties in the matrix context, and you can reference the property in other areas of your workflow file.
In your case, to access the matrix env variable you set in the include field list, you would need to use ${{ matrix.env.MY_VAR }}.
Something like this:
env:
MY_VAR: One
jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
include:
- env:
MY_VAR: Two
- env:
MY_VAR: Three
steps:
- run: |
echo "$MY_VAR" # is equal to ${{ env.MY_VAR }}
echo ${{ matrix.env.MY_VAR }}
This will generate 2 test jobs: test (Two) and test (Three).
I made a test here if you want to have a look.
Conclusion:
Basically, the env you set at the workflow level isn't the same env set in the matrix strategy, so you can't substitute the value there.
Alternative
What you could do eventually (as #Adam stated in the other answer) is to set the env value at the workflow level for it to work, in the environment context. (Reference in the documentation + another workflow example).
Example:
env:
WORKFLOW_VARIABLE: WORKFLOW
jobs:
test:
runs-on: ubuntu-latest
env:
JOB_VARIABLE: JOB
steps:
- name: Example
env:
STEP_VARIABLE: STEP
run: |
echo "Hello World"
echo "This is the $WORKFLOW_VARIABLE environment variable"
echo "This is the $JOB_VARIABLE environment variable"
echo "This is the $STEP_VARIABLE environment variable"
But depending on what you plan to achieve, a matrix strategy you be more efficient.
A YAML file as follows:
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
- job_name: 'node-exporter'
relabel_configs:
- source_labels: [__address__]
target_label: instance
regex: '([^:]+)(:[0-9]+)?'
replacement: '${1}'
static_configs:
- labels:
type: 'YYY'
group: 'Bar'
targets:
- '192.168.1.134:80'
- '192.168.1.146:80'
- labels:
type: 'YYY'
group: 'Foo'
targets:
- '192.168.2.136:80'
- labels:
type: 'ZZZ'
group: 'Foo'
targets:
- '192.168.2.100:80'
I'd like to use a bash script to add the "targets" to the YAML based on "type" and "group". For example:
./add_host.sh 192.168.1.17:80 YYY Bar
Added bonus for deleting entries :)
Things I've tried (and failed):
Using jq;
Tried using sed to insert values, but I have trouble in finding the right section to insert it to (Block parsing).
You don't mean that you want to parse the YAML file with bash, do you? That would be tedious and error-prone, not to mention slow to execute. You must mean you want a command-line utility to manipulate the file.
What you want is something that parses the YAML into a data structure. If I were doing it, I'd try the Python PyYAML module. It represents the whole file as a nested data structure. Read it in, change the parts you want to change, and write it out.
We're using Ansible playbook with GitLab CI in this project, where we'd pass some variables from ENV_FILE through Ansible playbook, then rendering JJ2 template with them.
Now the problem occurs when some variable has $ in its value, which seems interpreted as shell variable at some point, and the final value is rendered incorrect.
For example, in ENV_FILE
(set via GitLab CI Settings > CI/CD > Variables menu):
export FIRST_VAR=...
export SOME_VAR='123$abc#xyz'
export SOME_OTHER_VAR=...
And the final result in docker-compose.yaml becomes 123#xyz
EDIT: We just tried changing to export SOME_VAR='123''$''abc#xyz', the final result becomes 123abc#xyz, still missing the $.
gitlab-ci.yaml
deploy:
stage: deploy
environment:
name: dev
script:
- source $ENV_FILE
- cd ansible && ansible-playbook -i inventory/dev.ini runapp.yaml --vault-password-file=${ANSIBLE_VAULT_FILE}
runapp.yaml
- hosts: app
become: yes
roles:
- { role: some_app }
vars:
SOME_VAR: "{{ lookup('env', 'SOME_VAR') }}"
Task File:
- name: "Templating docker-compose file"
become: yes
template:
src: app-docker-compose.yaml.j2
dest: /opt/someapp/docker-compose.yaml
app-docker-compose.yaml.j2
someapp-svc:
image: someapp:version
restart: always
ports:
- ####:####
environment:
SOME_VAR: {{ SOME_VAR }}
Any hint about this?
Thanks!
I can reproduce that behavior when setting a CI/CD variable containing $; the docs kind of hint at it, although the docs are written as if the problem only applies when setting variables inside .gitlab-ci.yml which is demonstrably false
If you want a CI/CD variable to contain a literal $, it needs to be doubled, so SOME_VAR would need to be written as 123$$abc#xyz in the CI/CD configuration page in order for it to materialize as 123$abc#xyz inside the pipeline (although as the comments correctly point out, one will want to be exceedingly careful about the use of source to avoid further interpolation)
Using GitHub Actions, I would like to invoke a shell script with a list of directories.
(Essentially equivalent to passing an Ansible vars list to the shell script)
I don't really know how, is this even possible? Here's what I have until now, how could one improve this?
name: CI
on:
push:
branches:
- master
tags:
- v*
pull_request:
jobs:
run-script:
runs-on: ubuntu-20.04
steps:
- uses: actions/checkout#v2
- name: Run script on targets
run: ./.github/workflows/script.sh {{ targets }}
env:
targets:
- FolderA/SubfolderA/
- FolderB/SubfolderB/
Today I was able to do this with the following YAML (truncated):
...
with:
targets: |
FolderA/SubfolderA
FolderB/SubfolderB
The actual GitHub Action passes this as an argument like the following:
runs:
using: docker
image: Dockerfile
args:
- "${{ inputs.targets }}"
What this does is simply sends the parameters as a string with the newline characters embedded, which can then be iterated over similar to an array in a POSIX-compliant manner via the following shell code:
#!/bin/sh -l
targets="${1}"
for target in $targets
do
echo "Proof that this code works: $target"
done
Which should be capable of accomplishing your desired task, if I understand the question correctly. You can always run something like sh ./script.sh $target in the loop if your use case requires it.
I'm not able to access environment variables defined at the top-level of a GitHub Action configuration file from within a script run by the action.
For example, given the following config file:
name: x-pull-request
on: pull_request
env:
FOO: bar
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v1
- name: does a thing
run: ./scripts/do-a-thing.sh
... and the following script:
X=${FOO:default}
echo "X: $X" # X: default
The FOO environment variable defined in the config file is not available to the script and the default value is being used.
So, how can I access the environment variable from a Bash script run by the build step? Am I missing a prefix or something? (I know values defined in the input hash require you to use an INPUT_ prefix when referencing them.)
You can use env at any level also in jobs and steps.
I wrote a test action and a test script to validate it:
The action file:
name: Env tests
on: push
env:
FOO_ROOT: bar on root
jobs:
test:
runs-on: ubuntu-latest
env:
FOO_JOB: bar on job
steps:
- uses: actions/checkout#v1
- name: Test envs
run: ./env-test.sh
env:
FOO_STEP: bar on step
The script file:
#!/usr/bin/env bash
echo "FOO_ROOT: $FOO_ROOT"
echo "FOO_JOB: $FOO_JOB"
echo "FOO_STEP: $FOO_STEP"
echo " "
printenv
The results:
FOO_ROOT: bar on root
FOO_JOB: bar on job
FOO_STEP: bar on step
LEIN_HOME=/usr/local/lib/lein
M2_HOME=/usr/share/apache-maven-3.6.3
...
Check my results and, in fact, I don't know why it didn't work on your side because it must work.
If some one else is searching for a solution you have to explicitly pass the environment variables to the bash script. Otherwise they are not available:
name: x-pull-request
on: pull_request
env:
FOO: bar
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v1
- name: does a thing
run: ./scripts/do-a-thing.sh $FOO
and in the script you have to map the parameter to a variable.
X=$1
echo "X: $X" # X: default