I am trying to list all the changed files on file push. However my List all changed files command does not return anything even though I change the file and commit
name: test on: push:
branches: [ main ]
jobs: build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
with:
fetch-depth: 0 # OR "2" -> To retrieve the preceding commit.
- name: Get changed files using defaults
id: changed-files
uses: actions/checkout#v2
- name: List all added files
run: |
for file in ${{ steps.changed-files.outputs.added_files }}; do
echo "$file was added"
done
- name: List all changed files
run: |
for file in ${{ steps.changed-files.outputs.modified_files }}; do
echo "$file was modified"
done
Try using the one available in marketplace https://github.com/jitterbit/get-changed-files#get-all-changed-files
Related
I have scoured the forums and couldn't find a solution.
I have a config.yml file which contains a set of key/value pairs. One of those pairs I want to set as a GITHUB_ENV to be used in my workflow. But I am running into issues as the logs say
"reusable workflows should be referenced at the top-level
`jobs.*.uses' key, not within steps"
How do I get around this?
name: "Deploy The Kraken"
on:
push:
branches:
- dev
pull_request:
branches:
- dev
workflow_dispatch:
jobs:
call-build:
steps:
- name: Set env
run: |
echo "FEATURE_BRANCH=$(cat config.yml | awk -F: '/^branch:/ { print $2 }'
| sed 's/ //g')" >> $GITHUB_ENV
- name: Test
run: echo $FEATURE_BRANCH
- uses: ######/#####/.github/workflows/kraken.yml#$FEATURE_BRANCH
with:
environment: #####
workspace: #####
contract: #####
production-ref: #####
I have tried multiple variations of placing the shell command in different parts. But I still can't get it to point to my chosen branch.
In my GitHub workflow, I have a finish job that uploads the coverage report to Coveralls once all other jobs are finished.
jobs:
foo: ...
bar: ...
finish:
needs:
- foo
- bar
If either foo or bar fails though, or it is skipped (why a job may be skipped is not relevant to this question), the finish job won't be run.
Is there a way I could make finish run if at least one of the jobs provided in the needs field is run successfully?
It's currently not possible to get job.status or job.conclusion natively on the workflow to check them on other jobs through conditional.
A workaround could be to use outputs or artifacts to always save each job status.
Here is an example using artifacts with 3 jobs, where the third job would check the previous 2 jobs status before executing some operation:
jobs:
JOB_01:
name: Job 01
. . .
steps:
- name: Some steps of job 01
. . .
- name: Create file status_job01.txt and write the job status into it
if: always()
run: |
echo ${{ job.status }} > status_job01.txt
- name: Upload file status_job01.txt as an artifact
if: always()
uses: actions/upload-artifact#v1
with:
name: pass_status_job01
path: status_job01.txt
JOB_02:
name: Job 02
. . .
steps:
- name: Some steps of job 02
. . .
- name: Create file status_job02.txt and write the job status into it
if: always()
run: |
echo ${{ job.status }} > status_job02.txt
- name: Upload file status_job02.txt as an artifact
if: always()
uses: actions/upload-artifact#v1
with:
name: pass_status_job02
path: status_job02.txt
JOB_03:
needs: [JOB_01, JOB_02]
if: always()
name: Job 03
. . .
steps:
- name: Download artifact pass_status_job01
uses: actions/download-artifact#v1
with:
name: pass_status_job01
- name: Download artifact pass_status_job02
uses: actions/download-artifact#v1
with:
name: pass_status_job02
- name: Set the statuses of Job 01 and Job 02 as output parameters
id: set_outputs
run: |
echo "::set-output name=status_job01::$(<pass_status_job01/status_job01.txt)"
echo "::set-output name=status_job02::$(<pass_status_job02/status_job02.txt)"
- name: Show the values of the outputs
run: |
# using the syntax steps.<step_id>.outputs.<output_name> to access the output parameters
echo "status_job01 = ${{ steps.set_outputs.outputs.status_job01 }}"
echo "status_job02 = ${{ steps.set_outputs.outputs.status_job02 }}"
- name: Some other steps of job 03
. . .
Where the other steps of job03 would depend on the outputs results you got from the other jobs, to perform an action or not.
You can find more references about this example on the links below:
Get status of multiple jobs in the same workflow
How to get status of previous job
I also wrote an example here:
workflow file
workflow run
Given a repository structure with two packages like this:
$ tree
.
└── packages
├── foo
└── bar
$ cat pnpm-workspace.yaml
packages:
- 'packages/**'
$ pnpm -s m ls --depth -1
monorepo /monorepo
#mono/foo#0.0.0 /monorepo/packages/foo
#mono/baz#0.0.0 /monorepo/packages/bar
I'd like to run GitHub Actions CI such that it automatically runs each project as separate job. Here I've set up a job that manually does that parallelization:
name: CI
on:
push:
jobs:
build:
strategy:
matrix:
package: ["#mono/foo", "#mono/bar" ]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- uses: pnpm/action-setup#v2
with:
version: 6.9.1
- run: pnpm run --filter ${{ matrix.package }} test
That runs fast because each project is run as parallel jobs, but I don't want to manually maintain that matrix.package list. How can I utilize pnpm to provide a list of workspace projects that gets fed into GitHub Actions CI?
Hmmm... I hit my head on this a bit more and I've found a solution.
I first made a package.json script to turn pnpm output into a json array-fragment:
$ cat package.json
{
"scripts": {
"list-packages": "echo [$(pnpm -s m ls --depth -1 | tr \" \" \"\n\" | grep -o \"#.*#\" | rev | cut -c 2- | rev | sed -e 's/\\(.*\\)/\"\\1\"/' | paste -sd, - )]",
}
}
$ pnpm -s list-packages
["#mono/foo","#mono/bar"]
(I'm not good enough with shell to know if there's a much easier way to express this transformation so I'd be happy to learn!)
I then followed GitHub documentation on dynamically setting matrix variables and created this workflow:
name: CI
on:
push:
workflow_dispatch:
jobs:
packages:
runs-on: ubuntu-latest
outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}
steps:
- uses: actions/checkout#v2
- uses: pnpm/action-setup#v2
with:
version: 6.9.1
- run: $(echo pnpm -s list-packages2)
- id: set-matrix
run: echo "::set-output name=matrix::{\"packages\":$(pnpm -s list-packages)}"
build:
needs: packages
strategy:
matrix: ${{ fromJson(needs.packages.outputs.matrix) }}
runs-on: ubuntu-latest
steps:
- run: echo ${{ matrix.package }}
The packages job now takes the output of $(pnpm -s list-packages) and puts it into the matrix variable, and that makes GitHub Actions run them all in parallel:
I have a cross-platform project which is to be built on 2 platforms: mac and linux(ubuntu).
My pipeline contains 3 jobs:
prepare docker image with all nessesary too to build the project.
build on ubuntu in prepared docker container, depends on step 1
build on MacOS, needs nothing
Steps for linux and macos are definitely the same. But matrixes differs much, and linux build is
run inside container.
Is there a way to share steps between two different jobs?
I tried YAML anchors but GitHub does not support them.
Full workflow
on:
push:
branches: [ main, support/1.2.x ]
pull_request:
branches: [ main, support/1.2.x ]
jobs:
Docker-iroha-builder:
runs-on: ubuntu-latest
steps:
-
name: Checkout
uses: actions/checkout#v2
-
name: Set up Docker Buildx
uses: docker/setup-buildx-action#v1
-
name: Cache Docker layers
uses: actions/cache#v2
with:
path: /tmp/.buildx-cache
key: ${{ runner.os }}-buildx-${{ github.sha }}
restore-keys: |
${{ runner.os }}-buildx-
-
name: Login to DockerHub
uses: docker/login-action#v1
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
-
name: Build and push
uses: docker/build-push-action#v2
with:
file: docker/develop/Dockerfile.builder
# context: .
push: true
tags: ${{ secrets.DOCKERHUB_ORG }}/iroha:builder
cache-from: type=local,src=/tmp/.buildx-cache
cache-to: type=local,dest=/tmp/.buildx-cache-new
-
# Temp fix
# https://github.com/docker/build-push-action/issues/252
# https://github.com/moby/buildkit/issues/1896
name: Move cache
run: |
rm -rf /tmp/.buildx-cache
mv /tmp/.buildx-cache-new /tmp/.buildx-cache
build-iroha-ubuntu:
needs: Docker-iroha-builder
runs-on: ubuntu-latest
container: ikyb/iroha:builder
strategy:
fail-fast: false
matrix:
cc: [ gcc-9, gcc-10, clang ] ##todo g++-10
USE_BURROW: [ -DUSE_BURROW=OFF ]
debrel: [ Debug ] #,Release, RelWithDebInfo
steps:
- ## Takes 22 seconds with default github runner
name: Homebrew
run: brew install cmake ninja coreutils
if: ${{ runner.os == 'MacOS' }}
-
name: Checkout
uses: actions/checkout#v2
-
name: Cache vcpkg
uses: actions/cache#v2
with:
path: |
build-vcpkg
build/vcpkg_installed
$HOME/.cache/vcpkg
key: ${{ runner.os }}-vcpkg-${{ github.sha }}
restore-keys: ${{ runner.os }}-vcpkg-
-
name: Build Iroha vcpkg dependancies
run: ./vcpkg/build_iroha_deps.sh $PWD/build-vcpkg
-
name: CMake configure
run: |
export CC=${{ matrix.cc }} CXX=$(echo ${{ matrix.cc }} | sed -es,gcc,g++, -es,clang,clang++,)
cmake -B build -DCMAKE_TOOLCHAIN_FILE=$PWD/build-vcpkg/scripts/buildsystems/vcpkg.cmake \
${{ matrix.USE_BURROW }} -GNinja #-DCMAKE_VERBOSE_MAKEFILE=ON
-
name: CMake build
run: cmake --build build --config ${{ matrix.debrel }}
build-iroha-macos:
runs-on: macos-latest
strategy:
fail-fast: false
matrix:
USE_BURROW: [ -DUSE_BURROW=OFF ]
debrel: [ Debug,Release ]
steps:
- ## Takes 22 seconds with default github runner
name: Homebrew
run: brew install cmake ninja coreutils
if: ${{ runner.os == 'MacOS' }}
-
name: Checkout
uses: actions/checkout#v2
-
name: Cache vcpkg
uses: actions/cache#v2
with:
path: |
build-vcpkg
build/vcpkg_installed
$HOME/.cache/vcpkg
key: ${{ runner.os }}-vcpkg-${{ github.sha }}
restore-keys: ${{ runner.os }}-vcpkg-
-
name: Build Iroha vcpkg dependancies
run: ./vcpkg/build_iroha_deps.sh $PWD/build-vcpkg
-
name: CMake configure
run: |
export CC=${{ matrix.cc }} CXX=$(echo ${{ matrix.cc }} | sed -es,gcc,g++, -es,clang,clang++,)
cmake -B build -DCMAKE_TOOLCHAIN_FILE=$PWD/build-vcpkg/scripts/buildsystems/vcpkg.cmake \
${{ matrix.USE_BURROW }} -GNinja #-DCMAKE_VERBOSE_MAKEFILE=ON
-
name: CMake build
run: cmake --build build --config ${{ matrix.debrel }}
TL;DR
I solved my problem with shell tool yq
yq eval 'explode(.)' file.yml
The repository with example usage and detailed description https://github.com/kuvaldini/make-workflows.sh may be helpful to easy start. It was make from this answer. Pay attention to Actions tab.
Long answer
GitHub Workflow description in YAML does not support anchors.
There are several workarounds => anyway they come to building-editing workflow yaml from source.
So I suggest yet another one make-workflows.sh based on YAML tool yq.
USAGE
Move your workflows to .github/*.src.yml
Put make-workflows.sh to directory .github/
(optional) Copy or link pre-commit.sh to .git/hooks/pre-commit
Like ln -s ../../.github/pre-commit.sh .git/hooks/pre-commit
File make-workflows.sh
#!/usr/bin/env bash
set -euo pipefail
## The script expands '*.src.yml' from $1(default: script's directory)
## to $2 (default:subdirectory 'workflows') with corresponding name '*.yml'
## Main goal is to dereference YAML anchors.
## Deals only with Git cached/indexed files
## Set -x to debug
script_dir=$(dirname $(realpath "$0"))
dir_from=${1:-${script_dir}}
dir_to=${2:-workflows}
cd $dir_from
edited=
for f in $(git status -s -- \*.src.yml | sed 's,^.. ,,') ;do
readonly out=$(echo $f | sed s,.src.yml\$,.yml,)
readonly wout=$dir_to/$out
readonly tempout=$(mktemp)
trap "rm -f $tempout" EXIT
echo >>$tempout "## DO NOT EDIT"
echo >>$tempout "## Generated from $f with $(basename $0)"
echo >>$tempout ""
yq eval 'explode(.)' $f >>$tempout
if ! diff -q $wout $tempout &>/dev/null ;then
mv $tempout $wout
edited+="'$out' "
fi
done
if [[ -n "$edited" ]]
then echo >&2 "make-workflows: these files were edited: $edited"
else echo >&2 "make-workflows: everything is up to date"
fi
File pre-commit.sh
#!/usr/bin/env bash
set -euo pipefail
gitroot=$(git rev-parse --show-toplevel)
cd $gitroot
./.github/make-workflows.sh
git add .github/workflows
Links
ready to use solution with detailed description https://github.com/kuvaldini/make-workflows.sh
Share same steps for different GitHub Actions jobs
https://github.community/t/support-for-yaml-anchors/16128/60
https://github.com/mithro/actions-includes
https://github.com/allejo/gha-workflows
While github actions does not support YAML anchors directly, one can expand those e.g. by converting from YAML to JSON and then back to YAML. I am doing this here (Makefile in .github/workflows): https://github.com/agda/agda/blob/557681d04aae2100ccde2e045a8afcf30528c3a5/.github/workflows/Makefile
srcpath=../../src/github/workflows
sources=$(wildcard $(srcpath)/*.yml $(srcpath)/*.yaml)
targets=$(sort $(notdir $(sources)))
all : $(targets)
# Normalize YAML files by going via JSON.
# This expands anchors which are not understood by github workflows.
% : $(srcpath)/%
yaml2json $< | json2yaml - > $#
An example for a workflow file with anchors is here: https://github.com/agda/agda/blob/557681d04aae2100ccde2e045a8afcf30528c3a5/src/github/workflows/test.yml
jobs:
build:
runs-on: &runs_on ubuntu-22.04
steps:
- &checkout
uses: actions/checkout#v3
- &haskell_setup
uses: haskell/actions/setup#v2
with:
ghc-version: ${{ env.GHC_VER }}
...
test:
needs: build
runs-on: *runs_on
steps:
- *checkout
- *haskell_setup
...
I am trying to create a GitHub Actions workflow which would collect specific paths changed in last commit and run a step for each of collected paths, if any.
Currently, in my workflow I'm creating an array of paths, but I'm not sure how to proceed with my array:
name: Test
on:
push
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v1
# This step will create "an array" of strings, e.g. "path1 path2 path3"
- name: array
id: arr
run: |
arr=()
for i in "$(git diff-tree --no-commit-id --name-only -r ${{ github.sha }})"
do
if [[ $i == *"path1"* ]]; then
arr+=("path1")
fi
if [[ $i == *"path2"* ]]; then
arr+=("path2")
fi
done
echo ::set-output name=arr::${arr[#]}
# How to run this step by iterating the `${{ steps.arr.outputs.arr }}`?
- name: reviewdog-lint
uses: reviewdog/action-eslint#v1
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
reporter: github-pr-review
eslint_flags: 'my_project/some_folder/${{ SINGLE_PATH }}/' # `SINGLE_PATH` would be a path from the array
Is something like this even possible in the first place? If not, what would be recommended way to loop through some values and use them as arguments in other workflow steps?
There is some support for this in Github Actions. There is a very good tutorial here that explains how to do it in detail, but essentially what you'll do is split the steps into two jobs. The first job will output a JSON object that will serve as the input to the matrix of the second job.
Here's a simple example:
jobs:
setup:
runs-on: ubuntu-latest
outputs:
matrix: ${{ steps.matrix.outputs.value }}
steps:
- id: matrix
run: |
echo '::set-output name=value::[\"a\", \"b\", \"c\"]'
build:
needs: [ setup ]
runs-on: ubuntu-latest
strategy:
matrix:
value: ${{fromJson(needs.setup.outputs.matrix)}}
steps:
- run: |
echo "${{ matrix.value }}"
Difficult to say without running it, but I would say you need to use the output in the second step by assigning it to a variable, something like:
env:
OUTPUT: ${{ steps.id.outputs.arr }}
Then you use $OUTPUT as an environment variable inside the action.
The problem with that particular action is that it takes one commit at a time. But you can check out the code, it's a shell script. You can fork it from line 15 and make it split input and run a loop over it, applying eslint to every one of them.