Using GitHub Actions, I would like to invoke a shell script with a list of directories.
(Essentially equivalent to passing an Ansible vars list to the shell script)
I don't really know how, is this even possible? Here's what I have until now, how could one improve this?
name: CI
on:
push:
branches:
- master
tags:
- v*
pull_request:
jobs:
run-script:
runs-on: ubuntu-20.04
steps:
- uses: actions/checkout#v2
- name: Run script on targets
run: ./.github/workflows/script.sh {{ targets }}
env:
targets:
- FolderA/SubfolderA/
- FolderB/SubfolderB/
Today I was able to do this with the following YAML (truncated):
...
with:
targets: |
FolderA/SubfolderA
FolderB/SubfolderB
The actual GitHub Action passes this as an argument like the following:
runs:
using: docker
image: Dockerfile
args:
- "${{ inputs.targets }}"
What this does is simply sends the parameters as a string with the newline characters embedded, which can then be iterated over similar to an array in a POSIX-compliant manner via the following shell code:
#!/bin/sh -l
targets="${1}"
for target in $targets
do
echo "Proof that this code works: $target"
done
Which should be capable of accomplishing your desired task, if I understand the question correctly. You can always run something like sh ./script.sh $target in the loop if your use case requires it.
Related
How do I specify or change an environment variable as part of a job matrix?
For example, here I want my job to echo "Two" and "Three" instead of "One", but github completely ignores the definition or change of the environment variable in the matrix:
name: test-test-test
on:
push:
branches: [ main, develop ]
pull_request:
branches: [ main, develop ]
env:
MY_VAR: One
jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
include:
- env:
MY_VAR: Two
- env:
MY_VAR: Three
steps:
- run: echo "$MY_VAR2"
yields echo "One" and echo "One" in both jobs.
Use this context to access information about environment variables that have been set in a workflow, step, or job. Note that you cannot use the env context in the value of the id and uses keys within a step. An example is env.env_var.
GitHub provides extensive documentation about contexts at
https://docs.github.com/en/actions/learn-github-actions/contexts.
The syntax you used won't work for matrix. As stated in the official documentation:
The variables that you define become properties in the matrix context, and you can reference the property in other areas of your workflow file.
In your case, to access the matrix env variable you set in the include field list, you would need to use ${{ matrix.env.MY_VAR }}.
Something like this:
env:
MY_VAR: One
jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
include:
- env:
MY_VAR: Two
- env:
MY_VAR: Three
steps:
- run: |
echo "$MY_VAR" # is equal to ${{ env.MY_VAR }}
echo ${{ matrix.env.MY_VAR }}
This will generate 2 test jobs: test (Two) and test (Three).
I made a test here if you want to have a look.
Conclusion:
Basically, the env you set at the workflow level isn't the same env set in the matrix strategy, so you can't substitute the value there.
Alternative
What you could do eventually (as #Adam stated in the other answer) is to set the env value at the workflow level for it to work, in the environment context. (Reference in the documentation + another workflow example).
Example:
env:
WORKFLOW_VARIABLE: WORKFLOW
jobs:
test:
runs-on: ubuntu-latest
env:
JOB_VARIABLE: JOB
steps:
- name: Example
env:
STEP_VARIABLE: STEP
run: |
echo "Hello World"
echo "This is the $WORKFLOW_VARIABLE environment variable"
echo "This is the $JOB_VARIABLE environment variable"
echo "This is the $STEP_VARIABLE environment variable"
But depending on what you plan to achieve, a matrix strategy you be more efficient.
I am unable to generate a summary for github actions on windows. The following script encapsulates the script.
Using the following script:
name: Testing windows summaries
on:
pull_request:
branches: [master]
types: [synchronize, opened, reopened, ready_for_review]
jobs:
build-and-test:
name: Test
runs-on: "windows-latest"
steps:
- name: Adding markdown
run: |
echo '### Hello world! ' >> $GITHUB_STEP_SUMMARY
cat $GITHUB_STEP_SUMMARY
Produces the following error:
Get-Content: D:\a\_temp\e90ee597-2b32-4d33-ba5c-0ae6be835be0.ps1:7
Line |
7 | cat $GITHUB_STEP_SUMMARY
| ~~~~~~~~~~~~~~~~~~~~
| Cannot bind argument to parameter 'Path' because it is null.
Error: Process completed with exit code 1.
So the problem seems that the variable $GITHUB_STEP_SUMMARY never gets a value.
After looking for some time I found the solution to the problem. On windows the default shell does not seem to work well with the >> operator. The solution is to change the script above for:
name: Testing windows summaries
on:
pull_request:
branches: [master]
types: [synchronize, opened, reopened, ready_for_review]
jobs:
build-and-test:
name: Test
runs-on: "windows-latest"
steps:
- name: Adding markdown
run: |
echo '### Hello world! ' >> $GITHUB_STEP_SUMMARY
cat $GITHUB_STEP_SUMMARY
shell: bash
Where we have added a explicit shell instruction at the end.
We're using Ansible playbook with GitLab CI in this project, where we'd pass some variables from ENV_FILE through Ansible playbook, then rendering JJ2 template with them.
Now the problem occurs when some variable has $ in its value, which seems interpreted as shell variable at some point, and the final value is rendered incorrect.
For example, in ENV_FILE
(set via GitLab CI Settings > CI/CD > Variables menu):
export FIRST_VAR=...
export SOME_VAR='123$abc#xyz'
export SOME_OTHER_VAR=...
And the final result in docker-compose.yaml becomes 123#xyz
EDIT: We just tried changing to export SOME_VAR='123''$''abc#xyz', the final result becomes 123abc#xyz, still missing the $.
gitlab-ci.yaml
deploy:
stage: deploy
environment:
name: dev
script:
- source $ENV_FILE
- cd ansible && ansible-playbook -i inventory/dev.ini runapp.yaml --vault-password-file=${ANSIBLE_VAULT_FILE}
runapp.yaml
- hosts: app
become: yes
roles:
- { role: some_app }
vars:
SOME_VAR: "{{ lookup('env', 'SOME_VAR') }}"
Task File:
- name: "Templating docker-compose file"
become: yes
template:
src: app-docker-compose.yaml.j2
dest: /opt/someapp/docker-compose.yaml
app-docker-compose.yaml.j2
someapp-svc:
image: someapp:version
restart: always
ports:
- ####:####
environment:
SOME_VAR: {{ SOME_VAR }}
Any hint about this?
Thanks!
I can reproduce that behavior when setting a CI/CD variable containing $; the docs kind of hint at it, although the docs are written as if the problem only applies when setting variables inside .gitlab-ci.yml which is demonstrably false
If you want a CI/CD variable to contain a literal $, it needs to be doubled, so SOME_VAR would need to be written as 123$$abc#xyz in the CI/CD configuration page in order for it to materialize as 123$abc#xyz inside the pipeline (although as the comments correctly point out, one will want to be exceedingly careful about the use of source to avoid further interpolation)
I'm not able to access environment variables defined at the top-level of a GitHub Action configuration file from within a script run by the action.
For example, given the following config file:
name: x-pull-request
on: pull_request
env:
FOO: bar
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v1
- name: does a thing
run: ./scripts/do-a-thing.sh
... and the following script:
X=${FOO:default}
echo "X: $X" # X: default
The FOO environment variable defined in the config file is not available to the script and the default value is being used.
So, how can I access the environment variable from a Bash script run by the build step? Am I missing a prefix or something? (I know values defined in the input hash require you to use an INPUT_ prefix when referencing them.)
You can use env at any level also in jobs and steps.
I wrote a test action and a test script to validate it:
The action file:
name: Env tests
on: push
env:
FOO_ROOT: bar on root
jobs:
test:
runs-on: ubuntu-latest
env:
FOO_JOB: bar on job
steps:
- uses: actions/checkout#v1
- name: Test envs
run: ./env-test.sh
env:
FOO_STEP: bar on step
The script file:
#!/usr/bin/env bash
echo "FOO_ROOT: $FOO_ROOT"
echo "FOO_JOB: $FOO_JOB"
echo "FOO_STEP: $FOO_STEP"
echo " "
printenv
The results:
FOO_ROOT: bar on root
FOO_JOB: bar on job
FOO_STEP: bar on step
LEIN_HOME=/usr/local/lib/lein
M2_HOME=/usr/share/apache-maven-3.6.3
...
Check my results and, in fact, I don't know why it didn't work on your side because it must work.
If some one else is searching for a solution you have to explicitly pass the environment variables to the bash script. Otherwise they are not available:
name: x-pull-request
on: pull_request
env:
FOO: bar
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v1
- name: does a thing
run: ./scripts/do-a-thing.sh $FOO
and in the script you have to map the parameter to a variable.
X=$1
echo "X: $X" # X: default
I am trying to create a GitHub Actions workflow which would collect specific paths changed in last commit and run a step for each of collected paths, if any.
Currently, in my workflow I'm creating an array of paths, but I'm not sure how to proceed with my array:
name: Test
on:
push
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v1
# This step will create "an array" of strings, e.g. "path1 path2 path3"
- name: array
id: arr
run: |
arr=()
for i in "$(git diff-tree --no-commit-id --name-only -r ${{ github.sha }})"
do
if [[ $i == *"path1"* ]]; then
arr+=("path1")
fi
if [[ $i == *"path2"* ]]; then
arr+=("path2")
fi
done
echo ::set-output name=arr::${arr[#]}
# How to run this step by iterating the `${{ steps.arr.outputs.arr }}`?
- name: reviewdog-lint
uses: reviewdog/action-eslint#v1
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
reporter: github-pr-review
eslint_flags: 'my_project/some_folder/${{ SINGLE_PATH }}/' # `SINGLE_PATH` would be a path from the array
Is something like this even possible in the first place? If not, what would be recommended way to loop through some values and use them as arguments in other workflow steps?
There is some support for this in Github Actions. There is a very good tutorial here that explains how to do it in detail, but essentially what you'll do is split the steps into two jobs. The first job will output a JSON object that will serve as the input to the matrix of the second job.
Here's a simple example:
jobs:
setup:
runs-on: ubuntu-latest
outputs:
matrix: ${{ steps.matrix.outputs.value }}
steps:
- id: matrix
run: |
echo '::set-output name=value::[\"a\", \"b\", \"c\"]'
build:
needs: [ setup ]
runs-on: ubuntu-latest
strategy:
matrix:
value: ${{fromJson(needs.setup.outputs.matrix)}}
steps:
- run: |
echo "${{ matrix.value }}"
Difficult to say without running it, but I would say you need to use the output in the second step by assigning it to a variable, something like:
env:
OUTPUT: ${{ steps.id.outputs.arr }}
Then you use $OUTPUT as an environment variable inside the action.
The problem with that particular action is that it takes one commit at a time. But you can check out the code, it's a shell script. You can fork it from line 15 and make it split input and run a loop over it, applying eslint to every one of them.