I have a cross-platform project which is to be built on 2 platforms: mac and linux(ubuntu).
My pipeline contains 3 jobs:
prepare docker image with all nessesary too to build the project.
build on ubuntu in prepared docker container, depends on step 1
build on MacOS, needs nothing
Steps for linux and macos are definitely the same. But matrixes differs much, and linux build is
run inside container.
Is there a way to share steps between two different jobs?
I tried YAML anchors but GitHub does not support them.
Full workflow
on:
push:
branches: [ main, support/1.2.x ]
pull_request:
branches: [ main, support/1.2.x ]
jobs:
Docker-iroha-builder:
runs-on: ubuntu-latest
steps:
-
name: Checkout
uses: actions/checkout#v2
-
name: Set up Docker Buildx
uses: docker/setup-buildx-action#v1
-
name: Cache Docker layers
uses: actions/cache#v2
with:
path: /tmp/.buildx-cache
key: ${{ runner.os }}-buildx-${{ github.sha }}
restore-keys: |
${{ runner.os }}-buildx-
-
name: Login to DockerHub
uses: docker/login-action#v1
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
-
name: Build and push
uses: docker/build-push-action#v2
with:
file: docker/develop/Dockerfile.builder
# context: .
push: true
tags: ${{ secrets.DOCKERHUB_ORG }}/iroha:builder
cache-from: type=local,src=/tmp/.buildx-cache
cache-to: type=local,dest=/tmp/.buildx-cache-new
-
# Temp fix
# https://github.com/docker/build-push-action/issues/252
# https://github.com/moby/buildkit/issues/1896
name: Move cache
run: |
rm -rf /tmp/.buildx-cache
mv /tmp/.buildx-cache-new /tmp/.buildx-cache
build-iroha-ubuntu:
needs: Docker-iroha-builder
runs-on: ubuntu-latest
container: ikyb/iroha:builder
strategy:
fail-fast: false
matrix:
cc: [ gcc-9, gcc-10, clang ] ##todo g++-10
USE_BURROW: [ -DUSE_BURROW=OFF ]
debrel: [ Debug ] #,Release, RelWithDebInfo
steps:
- ## Takes 22 seconds with default github runner
name: Homebrew
run: brew install cmake ninja coreutils
if: ${{ runner.os == 'MacOS' }}
-
name: Checkout
uses: actions/checkout#v2
-
name: Cache vcpkg
uses: actions/cache#v2
with:
path: |
build-vcpkg
build/vcpkg_installed
$HOME/.cache/vcpkg
key: ${{ runner.os }}-vcpkg-${{ github.sha }}
restore-keys: ${{ runner.os }}-vcpkg-
-
name: Build Iroha vcpkg dependancies
run: ./vcpkg/build_iroha_deps.sh $PWD/build-vcpkg
-
name: CMake configure
run: |
export CC=${{ matrix.cc }} CXX=$(echo ${{ matrix.cc }} | sed -es,gcc,g++, -es,clang,clang++,)
cmake -B build -DCMAKE_TOOLCHAIN_FILE=$PWD/build-vcpkg/scripts/buildsystems/vcpkg.cmake \
${{ matrix.USE_BURROW }} -GNinja #-DCMAKE_VERBOSE_MAKEFILE=ON
-
name: CMake build
run: cmake --build build --config ${{ matrix.debrel }}
build-iroha-macos:
runs-on: macos-latest
strategy:
fail-fast: false
matrix:
USE_BURROW: [ -DUSE_BURROW=OFF ]
debrel: [ Debug,Release ]
steps:
- ## Takes 22 seconds with default github runner
name: Homebrew
run: brew install cmake ninja coreutils
if: ${{ runner.os == 'MacOS' }}
-
name: Checkout
uses: actions/checkout#v2
-
name: Cache vcpkg
uses: actions/cache#v2
with:
path: |
build-vcpkg
build/vcpkg_installed
$HOME/.cache/vcpkg
key: ${{ runner.os }}-vcpkg-${{ github.sha }}
restore-keys: ${{ runner.os }}-vcpkg-
-
name: Build Iroha vcpkg dependancies
run: ./vcpkg/build_iroha_deps.sh $PWD/build-vcpkg
-
name: CMake configure
run: |
export CC=${{ matrix.cc }} CXX=$(echo ${{ matrix.cc }} | sed -es,gcc,g++, -es,clang,clang++,)
cmake -B build -DCMAKE_TOOLCHAIN_FILE=$PWD/build-vcpkg/scripts/buildsystems/vcpkg.cmake \
${{ matrix.USE_BURROW }} -GNinja #-DCMAKE_VERBOSE_MAKEFILE=ON
-
name: CMake build
run: cmake --build build --config ${{ matrix.debrel }}
TL;DR
I solved my problem with shell tool yq
yq eval 'explode(.)' file.yml
The repository with example usage and detailed description https://github.com/kuvaldini/make-workflows.sh may be helpful to easy start. It was make from this answer. Pay attention to Actions tab.
Long answer
GitHub Workflow description in YAML does not support anchors.
There are several workarounds => anyway they come to building-editing workflow yaml from source.
So I suggest yet another one make-workflows.sh based on YAML tool yq.
USAGE
Move your workflows to .github/*.src.yml
Put make-workflows.sh to directory .github/
(optional) Copy or link pre-commit.sh to .git/hooks/pre-commit
Like ln -s ../../.github/pre-commit.sh .git/hooks/pre-commit
File make-workflows.sh
#!/usr/bin/env bash
set -euo pipefail
## The script expands '*.src.yml' from $1(default: script's directory)
## to $2 (default:subdirectory 'workflows') with corresponding name '*.yml'
## Main goal is to dereference YAML anchors.
## Deals only with Git cached/indexed files
## Set -x to debug
script_dir=$(dirname $(realpath "$0"))
dir_from=${1:-${script_dir}}
dir_to=${2:-workflows}
cd $dir_from
edited=
for f in $(git status -s -- \*.src.yml | sed 's,^.. ,,') ;do
readonly out=$(echo $f | sed s,.src.yml\$,.yml,)
readonly wout=$dir_to/$out
readonly tempout=$(mktemp)
trap "rm -f $tempout" EXIT
echo >>$tempout "## DO NOT EDIT"
echo >>$tempout "## Generated from $f with $(basename $0)"
echo >>$tempout ""
yq eval 'explode(.)' $f >>$tempout
if ! diff -q $wout $tempout &>/dev/null ;then
mv $tempout $wout
edited+="'$out' "
fi
done
if [[ -n "$edited" ]]
then echo >&2 "make-workflows: these files were edited: $edited"
else echo >&2 "make-workflows: everything is up to date"
fi
File pre-commit.sh
#!/usr/bin/env bash
set -euo pipefail
gitroot=$(git rev-parse --show-toplevel)
cd $gitroot
./.github/make-workflows.sh
git add .github/workflows
Links
ready to use solution with detailed description https://github.com/kuvaldini/make-workflows.sh
Share same steps for different GitHub Actions jobs
https://github.community/t/support-for-yaml-anchors/16128/60
https://github.com/mithro/actions-includes
https://github.com/allejo/gha-workflows
While github actions does not support YAML anchors directly, one can expand those e.g. by converting from YAML to JSON and then back to YAML. I am doing this here (Makefile in .github/workflows): https://github.com/agda/agda/blob/557681d04aae2100ccde2e045a8afcf30528c3a5/.github/workflows/Makefile
srcpath=../../src/github/workflows
sources=$(wildcard $(srcpath)/*.yml $(srcpath)/*.yaml)
targets=$(sort $(notdir $(sources)))
all : $(targets)
# Normalize YAML files by going via JSON.
# This expands anchors which are not understood by github workflows.
% : $(srcpath)/%
yaml2json $< | json2yaml - > $#
An example for a workflow file with anchors is here: https://github.com/agda/agda/blob/557681d04aae2100ccde2e045a8afcf30528c3a5/src/github/workflows/test.yml
jobs:
build:
runs-on: &runs_on ubuntu-22.04
steps:
- &checkout
uses: actions/checkout#v3
- &haskell_setup
uses: haskell/actions/setup#v2
with:
ghc-version: ${{ env.GHC_VER }}
...
test:
needs: build
runs-on: *runs_on
steps:
- *checkout
- *haskell_setup
...
Related
So I have two secrets: DEV_SERVER_IP and MASTER_SERVER_IP.
in main.yml I need something like this
run: echo "::set-env name=BRANCH_NAME::$(echo ${GITHUB_REF#refs/heads/} | sed 's/\//_/g')"
run: ssh-keyscan -H ${{ secrets.BRANCH_NAME_SERVER_IP }} >> ~/.ssh/known_hosts
but am getting error
env:
BRANCH_NAME: dev
Error: Input required and not supplied: key
I need here something like this ssh-keyscan -H ${{ secrets.${BRANCH_NAME}_SERVER_IP }}
how can I fix this?
You're trying to use shell style logic inside a Github context
expansion (${{ ... }}) which won't work. Just move all your logic
into your shell script instead:
name: Example
on:
push:
jobs:
example:
runs-on: ubuntu-latest
steps:
- name: get target ip
env:
DEV_SERVER_IP: ${{ secrets.DEV_SERVER_IP }}
MAIN_SERVER_IP: ${{ secrets.MAIN_SERVER_IP }}
run: |
branch_name=$(sed 's|/|_|g' <<< ${GITHUB_REF#refs/heads/})
target="${branch_name^^}_SERVER_IP"
mkdir -p ~/.ssh
ssh-keyscan -H ${!target} >> ~/.ssh/known_hosts
cat ~/.ssh/known_hosts
In the above workflow, the expression ${branch_name^^} is a bash expression that returns the value of $branch_name in uppercase, and ${!target} is a bash expression that returns the value of the variable who name is stored in $target.
Note that I'm not using your "set the BRANCH_NAME environment variable"
task because the ::set-env command is disabled by default for
security reasons.
I am trying to list all the changed files on file push. However my List all changed files command does not return anything even though I change the file and commit
name: test on: push:
branches: [ main ]
jobs: build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
with:
fetch-depth: 0 # OR "2" -> To retrieve the preceding commit.
- name: Get changed files using defaults
id: changed-files
uses: actions/checkout#v2
- name: List all added files
run: |
for file in ${{ steps.changed-files.outputs.added_files }}; do
echo "$file was added"
done
- name: List all changed files
run: |
for file in ${{ steps.changed-files.outputs.modified_files }}; do
echo "$file was modified"
done
Try using the one available in marketplace https://github.com/jitterbit/get-changed-files#get-all-changed-files
Given a repository structure with two packages like this:
$ tree
.
└── packages
├── foo
└── bar
$ cat pnpm-workspace.yaml
packages:
- 'packages/**'
$ pnpm -s m ls --depth -1
monorepo /monorepo
#mono/foo#0.0.0 /monorepo/packages/foo
#mono/baz#0.0.0 /monorepo/packages/bar
I'd like to run GitHub Actions CI such that it automatically runs each project as separate job. Here I've set up a job that manually does that parallelization:
name: CI
on:
push:
jobs:
build:
strategy:
matrix:
package: ["#mono/foo", "#mono/bar" ]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- uses: pnpm/action-setup#v2
with:
version: 6.9.1
- run: pnpm run --filter ${{ matrix.package }} test
That runs fast because each project is run as parallel jobs, but I don't want to manually maintain that matrix.package list. How can I utilize pnpm to provide a list of workspace projects that gets fed into GitHub Actions CI?
Hmmm... I hit my head on this a bit more and I've found a solution.
I first made a package.json script to turn pnpm output into a json array-fragment:
$ cat package.json
{
"scripts": {
"list-packages": "echo [$(pnpm -s m ls --depth -1 | tr \" \" \"\n\" | grep -o \"#.*#\" | rev | cut -c 2- | rev | sed -e 's/\\(.*\\)/\"\\1\"/' | paste -sd, - )]",
}
}
$ pnpm -s list-packages
["#mono/foo","#mono/bar"]
(I'm not good enough with shell to know if there's a much easier way to express this transformation so I'd be happy to learn!)
I then followed GitHub documentation on dynamically setting matrix variables and created this workflow:
name: CI
on:
push:
workflow_dispatch:
jobs:
packages:
runs-on: ubuntu-latest
outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}
steps:
- uses: actions/checkout#v2
- uses: pnpm/action-setup#v2
with:
version: 6.9.1
- run: $(echo pnpm -s list-packages2)
- id: set-matrix
run: echo "::set-output name=matrix::{\"packages\":$(pnpm -s list-packages)}"
build:
needs: packages
strategy:
matrix: ${{ fromJson(needs.packages.outputs.matrix) }}
runs-on: ubuntu-latest
steps:
- run: echo ${{ matrix.package }}
The packages job now takes the output of $(pnpm -s list-packages) and puts it into the matrix variable, and that makes GitHub Actions run them all in parallel:
I have a workflow where after a push to master I want to create a release and upload an asset to it.
I'm using actions/create-release#v1 and actions/upload-release-asset#v1.
I would like to pass the outputs of a bash commands to the action parameters. However I found out the syntax of "$(command)" does not work.
How can I pass the output of a bash command to an action's parameter.
For example I'd like to do something like this:
- name: Create Release
id: create_release
uses: actions/create-release#v1
env:
GITHUB_TOKEN: ${{ secrets.token }}
with:
tag_name: $(cat projectFile | grep -Po '(?<=Version>).*(?=</Version>)')
Update: set-output is being deprecated as well "Starting 1st June 2023 workflows using save-state or set-output commands via stdout will fail with an error."
Now that set-env is deprecated and set-output is soon to be deprecated, you can use GITHUB_OUTPUT environment files: to accomplish the same thing in this answer
- name: Retrieve version
run: |
echo "TAG_NAME=$(cat projectFile | grep -Po '(?<=Version>).*(?=</Version>)')" >> $GITHUB_OUTPUT
id: version
- name: Create Release
id: create_release
uses: actions/create-release#v1
env:
GITHUB_TOKEN: ${{ secrets.token }}
with:
tag_name: ${{ steps.version.outputs.TAG_NAME }}
References:
https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#environment-files
How to save the output of a bash command to output parameter in github actions
Use environment files
steps:
- name: Set the value
id: step_one
run: |
echo "FOO=$(git status)" >> $GITHUB_ENV
- name: Use the value
id: step_two
run: |
echo "${{ env.FOO }}"
UPDATE: This answer will not work as GitHub as disabled this syntax for security reasons. You should use environment files instead.
I would create an environment variable based of your command output:
- name: Retrieve version
run: |
echo ::set-env name=TAG_NAME::$(cat projectFile | grep -Po '(?<=Version>).*(?=</Version>)')
And then access it like the following:
- name: Create Release
id: create_release
uses: actions/create-release#v1
env:
GITHUB_TOKEN: ${{ secrets.token }}
with:
tag_name: ${{ env.TAG_NAME }}
I am trying to create a GitHub Actions workflow which would collect specific paths changed in last commit and run a step for each of collected paths, if any.
Currently, in my workflow I'm creating an array of paths, but I'm not sure how to proceed with my array:
name: Test
on:
push
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v1
# This step will create "an array" of strings, e.g. "path1 path2 path3"
- name: array
id: arr
run: |
arr=()
for i in "$(git diff-tree --no-commit-id --name-only -r ${{ github.sha }})"
do
if [[ $i == *"path1"* ]]; then
arr+=("path1")
fi
if [[ $i == *"path2"* ]]; then
arr+=("path2")
fi
done
echo ::set-output name=arr::${arr[#]}
# How to run this step by iterating the `${{ steps.arr.outputs.arr }}`?
- name: reviewdog-lint
uses: reviewdog/action-eslint#v1
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
reporter: github-pr-review
eslint_flags: 'my_project/some_folder/${{ SINGLE_PATH }}/' # `SINGLE_PATH` would be a path from the array
Is something like this even possible in the first place? If not, what would be recommended way to loop through some values and use them as arguments in other workflow steps?
There is some support for this in Github Actions. There is a very good tutorial here that explains how to do it in detail, but essentially what you'll do is split the steps into two jobs. The first job will output a JSON object that will serve as the input to the matrix of the second job.
Here's a simple example:
jobs:
setup:
runs-on: ubuntu-latest
outputs:
matrix: ${{ steps.matrix.outputs.value }}
steps:
- id: matrix
run: |
echo '::set-output name=value::[\"a\", \"b\", \"c\"]'
build:
needs: [ setup ]
runs-on: ubuntu-latest
strategy:
matrix:
value: ${{fromJson(needs.setup.outputs.matrix)}}
steps:
- run: |
echo "${{ matrix.value }}"
Difficult to say without running it, but I would say you need to use the output in the second step by assigning it to a variable, something like:
env:
OUTPUT: ${{ steps.id.outputs.arr }}
Then you use $OUTPUT as an environment variable inside the action.
The problem with that particular action is that it takes one commit at a time. But you can check out the code, it's a shell script. You can fork it from line 15 and make it split input and run a loop over it, applying eslint to every one of them.