GitActions workflow: cache is not passing between jobs - caching

What I am trying to do is basically an assembly line: every job must build/maniuplate the provided files, the cache, that is passed to the following job.
This is my workflow:
jobs:
Job1_-_Prepearing_enviornment:
runs-on: ubuntu-latest
steps:
- name: Get date for artifacts
id: date
run: echo "::set-output name=date::$(date +'%Y%m%d_%H%M%S')"
- name: Extract branch name
shell: bash
run: echo "##[set-output name=branch;]$(echo ${GITHUB_REF#refs/heads/})"
id: extract_branch
- name: Generating build ID
run: echo "${{ steps.extract_branch.outputs.branch }}-${{ steps.date.outputs.date }}"
outputs:
buildID: "${{ steps.extract_branch.outputs.branch }}-${{ steps.date.outputs.date }}"
# In the last 3 steps I am generating a build specific key in order to not be confused with other running workflows
Job2_-_Downloading_Sources:
runs-on: ubuntu-latest
needs: [Job1_-_Prepearing_enviornment]
steps:
# Here I download a global cache if it's existing (I am generating the global cache in a job not copypasted here)
- uses: actions/download-artifact#v2
with:
name: global-cache
continue-on-error: true
- uses: actions/checkout#v3
- name: Downloading sources
run: |
git pull
git submodule init
git submodule update
sudo add-apt-repository ppa:alexlarsson/flatpak
sudo apt-get update
sudo apt install flatpak flatpak-builder p7zip-full
sudo flatpak remote-add --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo
sudo flatpak install -y org.kde.Sdk//5.15-21.08 org.kde.Platform//5.15-21.08 io.qt.qtwebengine.BaseApp/x86_64/5.15-21.08 org.freedesktop.Sdk.Extension.llvm13
sudo flatpak-builder --download-only --user --repo=local retrodeck-flatpak net.retrodeck.retrodeck.yml
# Here I am using the key generated in the Job1 to save the cache
- name: Generating build specific cache
id: intenral-cache
uses: actions/cache#v2
with:
path: /home/runner/work/RetroDECK
key: ${{ needs.Job1_-_Prepearing_enviornment.outputs.buildID }}
Job3_-_Building_part_1:
runs-on: ubuntu-latest
needs: [Job1_-_Prepearing_enviornment, Job2_-_Downloading_Sources]
steps:
# Here I am supposed to get the internal cache calling its key but this process is not working and the build stops
- name: cache-cooker
id: internal-cache
uses: actions/cache#v2
with:
path: /home/runner/work/RetroDECK
key: ${{ needs.Job1_-_Prepearing_enviornment.outputs.buildID }}
- name: Initializing enviornment
run: |
git submodule init
git submodule update
sudo add-apt-repository ppa:alexlarsson/flatpak
sudo apt-get update
sudo apt install flatpak flatpak-builder p7zip-full
sudo flatpak remote-add --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo
sudo flatpak install -y org.kde.Sdk//5.15-21.08 org.kde.Platform//5.15-21.08 io.qt.qtwebengine.BaseApp/x86_64/5.15-21.08 org.freedesktop.Sdk.Extension.llvm13
- name: Bulding part 1 - Until melonds
run: |
cd /home/runner/work/RetroDECK/RetroDECK
sudo flatpak-builder --build-only --stop-at=melonds --user --repo=local retrodeck-flatpak net.retrodeck.retrodeck.yml
As you can see from the comments in the code, the job3 cannot get the cache and I get this output from the GitAction log:
Run actions/cache#v2
with:
path: /home/runner/work/RetroDECK
key: cooker-20220502_115338
Cache not found for input keys: cooker-20220502_115338
Then the job fails because it cannot find the needed files.
I am trying to use the Artifacts instead of the cache but it's too slow by zipping, uploading, downloading and unzipping, so maybe the cache it's a more straightforward way.
What am I doing wrong?
How can I fix this?
Thanks in advance.

Related

CI/CD using Github Actions and AWS EC2 instance

I have a dockerised fastapi app whith depends on mysql and redis which are all configured in docker-compose.yml. Want to implement a CI/CD using github actions and AWS EC2 instance. My EC2 instance has docker and docker-compose installed. Here are my questions.
What do I do to run the tests that depends on the test db?
How do I implement CD from github actions and AWS EC2 instance?
I might not be clear so please ask some questions for clarification. Thank you.
name: backend-api
on:
push:
branches: ["main"]
pull_request:
branches: ["main"]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v3
- name: Set up Python 3.10
uses: actions/setup-python#v3
with:
python-version: "3.10"
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install flake8 pytest
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
- name: Lint with flake8
run: |
# stop the build if there are Python syntax errors or undefined names
flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
# exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
- name: Create .env file for configuration settings.
uses: SpicyPizza/create-envfile#v1.3
with:
envkey_APP_ENV: ${{secrets.APP_ENV}}
envkey_APP_HOST: ${{secrets.APP_HOST}}
envkey_MYSQL_USER: ${{secrets.APP_ENV}}
envkey_PROD_BASE_URL: ${{secrets.MYSQL_USER}}
envkey_DEV_BASE_URL: ${{secrets.APP_ENV}}
envkey_MYSQL_ROOT_PASSWORD: ${{secrets.MYSQL_ROOT_PASSWORD}}
envkey_MYSQL_DATABASE: ${{secrets.MYSQL_DATABASE}}
envkey_PRODUCTION_DB_URI: ${{secrets.PRODUCTION_DB_URI}}
envkey_TEST_DB_URI: ${{secrets.TEST_DB_URI}}
envkey_BASE_URL: ${{secrets.BASE_URL}}
envkey_WALLET_PROVIDER_ACCESS_TOKEN: ${{secrets.WALLET_PROVIDER_ACCESS_TOKEN}}
envkey_S3_BUCKET_NAME: ${{secrets.S3_BUCKET_NAME}}
envkey_S3_ACCESS_SECRET: ${{secrets.S3_ACCESS_SECRET}}
envkey_S3_ACCESS_KEY: ${{secrets.S3_ACCESS_KEY}}
envkey_S3_BUCKET_REGION: ${{secrets.S3_BUCKET_REGION}}
envkey_JWT_SECRET_KEY: ${{secrets.JWT_SECRET_KEY}}
envkey_ETHERSCAN_API_URL: ${{secrets.ETHERSCAN_API_URL}}
envkey_BLOCKCHAIN_API_URL: ${{secrets.BLOCKCHAIN_API_URL}}
envkey_WALLET_PROVIDER_BASE_URL: ${{secrets.WALLET_PROVIDER_BASE_URL}}
envkey_STRATEGY_PROVIDER_BASE_URL: ${{secrets.STRATEGY_PROVIDER_BASE_URL}}
envkey_INDEX_PROVIDER_BASE_URL: ${{secrets.INDEX_PROVIDER_BASE_URL}}
- name: Running Tests with pytest
run: |
pytest
Deploy:
needs: build
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: Git pull
env:
AWS_EC2_PEM: ${{ secrets.AWS_EC2_PEM }}
AWS_EC2_PUBLIC_IP: ${{ secrets.AWS_EC2_PUBLIC_IP }}
AWS_EC2_USERNAME: ${{ secrets.AWS_EC2_USERNAME }}
run: |
pwd
echo "$AWS_EC2_PEM" > private_key && chmod 600 private_key
ssh -o StrictHostKeyChecking=no -i private_key ${AWS_EC2_USERNAME}#${AWS_EC2_PUBLIC_IP}
git checkout main &&
git fetch --all &&
git reset --hard origin/main &&
git pull origin main &&
touch .env
docker-compose up -d --build

How to bump version with GitHub Actions

I want to bump the version by running my shell script. I am manually triggering the GitHub Actions with inputs. I am using GitHub API to get the latest version and I want to run my shell script with the version taken from GitHub API.
I am able to achieve everything. It's just that I am not able to get the version output.
This is my GitHub Workflow:
#############################################################################
# GitHub Action to bump release version
#
#############################################################################
name: "Bump version and Update Milestone"
on:
workflow_dispatch:
inputs:
version:
description: 'New Version'
required: true
default: 'warning'
permissions:
contents: write
pull-requests: write
jobs:
printInputs:
runs-on: ubuntu-latest
steps:
- run: |
echo "Latest Version: ${{ github.event.inputs.version }}"
bump:
name: Bump version
runs-on: ubuntu-latest
steps:
- name: Checkout the latest code
uses: actions/checkout#v2
with:
token: ${{ secrets.GITHUB_TOKEN }}
- name: Get the version
id: get_version
run: |
VERSION=$(curl -s https://api.github.com/repos/checkstyle/checkstyle/releases/latest \
| jq ".tag_name")
echo VERSION="$VERSION"
- name: Modify File
run: |
./.ci/bump-version.sh ${{ steps.get_version.VERSION }}
- name: Push commit
run: |
git config --global user.name 'github-actions[bot]'
git config --global user.email 'github-actions[bot]#users.noreply.github.com'
git commit -am "minor: Bump release version to ${{ steps.get_version.VERSION }}"
git push
- name: GitHub Milestone
run: |
.ci/update-github-milestone.sh
This is my shell script:
#!/bin/bash
set -e
VERSION=$1
echo VERSION="$VERSION"
echo "bump version in pom.xml"
mvn versions:set -DnewVersion="$VERSION" && mvn versions:commit
And these is my tested actions where I want help:
Action link: https://github.com/Rahulkhinchi03/checkstyle/runs/7785606554?check_suite_focus=true
To transfer an output from one step to another, you need to use a workflow command:
echo "::set-output name=VERSION::$VERSION"
and then reference it like this in a later step via the outputs object:
run: |
./.ci/bump-version.sh ${{ steps.get_version.outputs.VERSION }}

Using github secrets in another non-workflow yaml file

Is it possible to access a github secret in a yaml file that's not a workflow or an action yaml file?
For example, I've saved in github the environment secret INFURA_RINKEBY_WSS and I attempt to access it in the following yaml config file for my program.
type: EndpointList
endpoints:
- type: RPCEndpoint
chain_id: 1
network: rinkeby
provider: Infura
url: ${{ secrets.INFURA_RINKEBY_WSS}}
explorer: https://etherscan.io
However, the INFURA_RINKEBY_WSS environment variable I've set in github isn't accessed yet by my yaml config file.
The following is my main.yaml github workflow:
name: Report to eth/usd on rinkeby w/ pytelliot
on: push
jobs:
build:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
python-version: ["3.9"]
steps:
- uses: actions/checkout#v2
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python#v2
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install telliot-feed-examples
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
- name: Move pre-made pytelliot configs to home directory
run: |
cp -r ./config ~/
- name: report :)
run: telliot-examples --legacy-id 1 report --submit-once
env:
PK: ${{ secrets.PK }}
INFURA_RINKEBY_WSS: ${{ secrets.INFURA_RINKEBY_WSS }}
Thanks!

Set up github actions to run cypress tests on localhost from 2 different repositories

I have my front-end (repUI) and cypress tests (repTest) in separate repositories., I am trying to run the tests by checkout both from repTest. Once the repUI is set up (install and build), the localhost:8000 is ready for tests to run.
So far the following workflow is designed:
name: Cypress E2E Tests on Dev
on:
workflow_dispatch:
push:
branches-ignore: [ main ]
schedule:
- cron: '0 0 * * 1-5'
jobs:
setup:
runs-on: ubuntu-20.04
strategy:
matrix:
node-version: [14.x]
steps:
- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node#v1
with:
node-version: ${{ matrix.node-version }}
# Checkout UI repo
- name: Checkout voy-frontend repo
uses: actions/checkout#v2
with:
repository: orgName/xyz-frontend
path: xyz-frontend
ssh-key: ${{ secrets.GIT_SSH_KEY }}
# Install voy-f dependancies
- name: Install dependancies for application
run: |
cd voy-frontend
pwd
npm install
- name: Get npm cache directory
id: npm-cache-dir
run: |
echo "::set-output name=dir::$(npm config get cache)"
- uses: actions/cache#v2
id: npm-cache # use this to check for `cache-hit` ==> if: steps.npm-cache.outputs.cache-hit != 'true'
with:
path: ${{ steps.npm-cache-dir.outputs.dir }}
key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
restore-keys: |
${{ runner.os }}-node-
# Run application
- name: Run application
run: |
pwd
cd voy-frontend
npm run serve
cypress-test:
name: Test
needs: setup
runs-on: ubuntu-latest
steps:
# Checkout test automation repo
- name: git checkout
uses: actions/checkout#v2
with:
ref: main
# Install cypress dependancies
- name: Cypress install
uses: cypress-io/github-action#v2
# Run cypress tests
- name: Cypress run - localhost
run: npm run cy:r -- --env TENANT=${{ secrets.TENANT }},CLIENT_ID=${{ secrets.CLIENT_ID }},CLIENT_SECRET=${{ secrets.CLIENT_SECRET }},ORG_SCOPE=${{ secrets.ORG_SCOPE }},G_SCOPE=${{ secrets.G_SCOPE }} --spec "cypress/integration/specs/Search.feature" --headless --browser chrome
- uses: actions/upload-artifact#v2
if: failure()
with:
name: cypress-videos
path: |
cypress/reports
cypress/videos
cypress/screenshots
retention-days: 1
The problem is if I run the build command npm run serve for repUI, the applications successfully getting started but it is not proceeding further instead it just waits. If I comment out the npm run serve, all other jobs are running as it should.
Here is the screenshot:
For building, it only took ~60 seconds, and then it waits until I cancel it. Same case if I trigger the job again and always.
What needs to be done in order to proceed to the cypress-test job once the build is ready.?

Installing private Github Package using yarn on Github Actions is Unauthorized with yarn.lock

There are a lot of similar issues already floating around:
Install private github package from package.json on Github Actions
Download private module from Github Package Registry via Yarn within a Github Action? Publishing works, but installing is met with '401 Unauthorized'
Installing private package from Github Package registry using Yarn fails with not authorized
However, our issue seems different, because:
yarn install runs fine on a local machine
the issue is only when using Github Actions
yarn install succeeds on GH Actions if we delete yarn.lock
Has anyone run into this before? Specifically with it not working with a yarn.lock file?
In case it matters, here's the setup:
build.yml:
steps:
- uses: actions/checkout#v2
- uses: actions/setup-node#v1
with:
node-version: '10.x'
registry-url: 'https://npm.pkg.github.com'
- name: Install
run: yarn install
env:
# GITHUB_TOKEN can't access packages hosted in private repos,
# even within the same organisation
NODE_AUTH_TOKEN: ${{ secrets.PERSONAL_ACCESS_TOKEN }}
- name: Build
run: yarn build
- name: Test
run: yarn test --forbid-only
We also have a .npmrc file for local installs:
#<org>:registry=https://npm.pkg.github.com
But no .yarnrc file.
I'm create a file .npmrc and .yarnrc.
Type:
name: Test
on: push
jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
node-version: [12.x]
steps:
- uses: actions/checkout#v2
- name: Node ${{ matrix.node-version }}
uses: actions/setup-node#v1
with:
node-version: ${{ matrix.node-version }}
- name: Create NPMRC
run: |
echo "//npm.pkg.github.com/:_authToken=${{ secrets.PACKAGES_TOKEN }}" >> ~/.npmrc
echo "#you-scope:registry=https://npm.pkg.github.com" >> ~/.npmrc
echo 'registry "https://registry.yarnpkg.com"' >> ~/.yarnrc
- run: yarn install
Replace #you-scope for you user of github or of your org in github in LowerCase.
Create a PACKAGES_TOKEN secrete token of your github access for this repository.
We managed to solve this by explicitly duplicating the .npmrc config in the build.yml config:
- uses: actions/setup-node#v1
with:
node-version: '10.x'
registry-url: 'https://npm.pkg.github.com'
# These following two lines are the key:
always-auth: true
scope: '#reedsy'

Resources