I am working on an R package and using GitHub Action (GHA) as a Continuous Integration (CI) provider. I cache R packages (dependencies) by using actions/cache. And now I want to clear all cache. How can I do that?
A part of GHA Workflow I use:
on: push
name: R-CMD-check
jobs:
R-CMD-check:
runs-on: ${{ matrix.config.os }}
name: ${{ matrix.config.os }} (${{ matrix.config.r }})
strategy:
fail-fast: false
matrix:
config:
# - {os: windows-latest, r: 'devel'}
- {os: macOS-latest, r: 'release'}
env:
R_REMOTES_NO_ERRORS_FROM_WARNINGS: true
RSPM: ${{ matrix.config.rspm }}
GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }}
steps:
- uses: actions/checkout#v2
- uses: r-lib/actions/setup-r#master
- name: Query dependencies
run: |
repos <- c("https://r-hyperspec.github.io/hySpc.pkgs/", getOption("repos"))
saveRDS("remotes::dev_package_deps(dependencies = TRUE)", ".github/depends.Rds", version = 2)
writeLines(sprintf("R-%i.%i", getRversion()$major, getRversion()$minor), ".github/R-version")
shell: Rscript {0}
- name: Cache R packages
if: runner.os != 'Windows'
uses: actions/cache#v1
with:
path: ${{ env.R_LIBS_USER }}
key: ${{ runner.os }}-${{ hashFiles('.github/R-version') }}-1-${{ hashFiles('.github/depends.Rds') }}
restore-keys: ${{ runner.os }}-${{ hashFiles('.github/R-version') }}-1-
- name: Install dependencies
run: remotes::install_deps(dependencies = TRUE)
shell: Rscript {0}
- name: Session info
run: |
options(width = 100)
pkgs <- installed.packages()[, "Package"]
sessioninfo::session_info(pkgs, include_base = TRUE)
shell: Rscript {0}
Update (October 20, 2022)
You can now manage caches via the UI:
https://github.com/<OWNER>/<REPO>/actions/caches
Update (June 27, 2022)
You can now manage caches via the GitHub Actions Cache API:
GET list of caches for a repository:
$ curl \
-H "Accept: application/vnd.github.v3+json" \
-H "Authorization: token <TOKEN>" \
https://api.github.com/repos/OWNER/REPO/actions/caches
DELETE cache for a repository using a cache ID:
$ curl \
-X DELETE \
-H "Accept: application/vnd.github.v3+json" \
-H "Authorization: token <TOKEN>" \
https://api.github.com/repos/OWNER/REPO/actions/caches/CACHE_ID
Alternatively, you can also use the GitHub CLI to interact with the API, using the gh-actions-cache extension.
Original Post (November 13, 2020)
As pointed out in the corresponding issue, two practical workarounds can be used to force the use of a new cache. This is not exactly the same as clearing the current cache (with regards to the cache usage limits), but it does the job.
In order to do so, you have to change the cache key (and any restore-keys). Because if the key(s) is/are different, this is considered a cache miss and you start with a new one.
You can change the cache key either by modifying the workflow file directly, e.g., by adding a version number:
key: ${{ runner.os }}-mycache-v1-${{ hashFiles(...) }}
If you now want to use a new cache, all you have to do is to commit a different version number:
key: ${{ runner.os }}-mycache-v2-${{ hashFiles(...) }}
If you don't want to modify the workflow file and prefer using the UI, you can abuse secrets:
key: ${{ runner.os }}-mycache-${{ secrets.CACHE_VERSION }}-${{ hashFiles(...) }}
Whenever the secret changes, a new cache will be used.
⚠️ WARNING: Secrets used for cache keys are "revealed" in the UI.
You cannot force a clear cache currently and it seems there is an open feature request for it at the moment https://github.com/actions/cache/issues/2. If I were you, I would post the request there as well so that they know that more people want the feature implemented.
A few things to note about the action:
There are not parameters in the action and not even in the toolkit package that this action is build on top of.
Getting deep into the toolkit code they use a cache api url to do all the good stuff. This means we don't even know if that api supports it with the assumption that we try to test it out and see what else it provides by hitting it directly. Here is the line for the api call for which the base url is taken from env ACTIONS_CACHE_URL
https://github.com/actions/toolkit/blob/c2bc747506bf562195a02bd4fdb1ff2a95d8b7ed/packages/cache/src/internal/cacheHttpClient.ts#L44
npm package as reference https://www.npmjs.com/package/#actions/cache
If we take a step back for a moment and go back to the github docs now that we have looked deep into the action/cache code and how it works,
Per the github docs https://docs.github.com/en/actions/configuring-and-managing-workflows/caching-dependencies-to-speed-up-workflows
There are two things to note,
Once you create a cache, you cannot change the contents of an existing
cache but you can create a new cache with a new key.
GitHub will remove any cache entries that have not been accessed in over 7 days.
This can be done from GH Actions as well.
name: Clear cache
on:
workflow_dispatch:
permissions:
actions: write
jobs:
clear-cache:
runs-on: ubuntu-latest
steps:
- name: Clear cache
uses: actions/github-script#v6
with:
script: |
console.log("About to clear")
const caches = await github.rest.actions.getActionsCacheList({
owner: context.repo.owner,
repo: context.repo.repo,
})
for (const cache of caches.data.actions_caches) {
console.log(cache)
github.rest.actions.deleteActionsCacheById({
owner: context.repo.owner,
repo: context.repo.repo,
cache_id: cache.id,
})
}
console.log("Clear completed")
Make sure it's merged to HEAD branch and trigger it manually from Actions.
Based #beatngu13 answer I've created Makefile target that remove all cache from Github Actions:
GH_REPO ?= ALLATRA-IT/ephyr # replace with your `org/repo`
# Clear Github Acions usage cache.
#
# Need to install github cli first [https://cli.github.com/] and authorize
#
# Usage:
# make gh.clear
gh.clear:
gh api -H "Accept: application/vnd.github+json" \
/repos/$(GH_REPO)/actions/caches \
| for ID in `jq '.actions_caches[].id'`; \
do echo "Deleting $$ID"; \
gh api --method DELETE /repos/$(GH_REPO)/actions/caches/$$ID | echo; done
Need to install Github CLI and jq tool tools first. After install Github CLI you need to authorize with:
gh auth login
#GegznaV you can use something like tmate, and manually clear cache by sshing into a runner.
For anyone looking for a simple commandline solution, the following shell command will also do it. It requires the Github CLI + JQ:
gh api -H 'Accept: application/vnd.github+json' /repos/realm/realm-kotlin/actions/caches --paginate | jq -r '.actions_caches | .[].key' | sed -e 's/|/%7c/g' -e 's/\[/%5b/g' -e 's/\]/%5d/g' | xargs -I {} sh -c 'gh api --method DELETE -H "Accept: application/vnd.github+json" /repos/<OWNER>/<REPO>/actions/caches?key={} --silent'
In this case, I'm using sed to URL encode special characters in the key. I just needed to handle |, [ and ]. Others can be added as needed. See e.g. https://gist.github.com/jaytaylor/5a90c49e0976aadfe0726a847ce58736#file-url_encode-sh-L11
One more easy and straightforward way to remove Actions' cache - Purge cache GH Action.
This action allows the cache of GitHub Actions to be automatically purged based on the last usage and configured TTL (default - 7 days).
Example:
name: Clear Cache
on:
schedule:
- cron: '0 0 * * *' # Runs once a day (https://crontab.guru/once-a-day)
jobs:
clear:
name: Clear cache
runs-on: ubuntu-latest
steps:
- uses: MyAlbum/purge-cache#v1
with:
max-age: 604800 # Cache max 7 days since last use (this is the default)
The workflow above will run once a day and remove all the cache that was last used more than 7 days ago.
Here's another automated solution similar to #beatngu13's cache key invalidation method. This one uses a generated timestamp file that gets committed and then used in a hashFiles(...) prefix to the cache key.
Included is a set of GNU Makefile targets to make using this method very easy: make clear-github-cache
Note: Make sure each Makefile "recipe" section is indented with 1 tab character! StackOverflow converts tabs to spaces in the code block.
.github/clear_github_actions_cache:
date +%s > .github/clear_github_actions_cache
.PHONY: clean-github-cache-file clear-github-cache
clear-github-cache: clean-github-cache-file .github/clear_github_actions_cache ## Force GitHub Actions Cache key to change for fresh CI run
git add .github/clear_github_actions_cache
git commit -m "Clear GitHub Actions Cache # $$(cat .github/clear_github_actions_cache)"
clean-github-cache-file: ## Remove GitHub Actions Cache timestamp invalidator file.
[ -f '.github/clear_github_actions_cache' ] && rm -f '.github/clear_github_actions_cache' || true
Then, inside the GitHub Actions workflow YAML, add a cache key prefix that uses the hash of this file:
- name: Cache R packages
if: runner.os != 'Windows'
uses: actions/cache#v2
with:
path: ${{ env.R_LIBS_USER }}
key: ${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('.github/clear_github_actions_cache') }}-${{ hashFiles('.github/R-version') }}-${{ hashFiles('.github/depends.Rds') }}
restore-keys: |
${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('.github/clear_github_actions_cache') }}-${{ hashFiles('.github/R-version') }}-${{ hashFiles('.github/depends.Rds') }}
${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('.github/clear_github_actions_cache') }}-${{ hashFiles('.github/R-version') }}-
${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('.github/clear_github_actions_cache') }}-
Now just run:
make clear-github-cache
This generates and commits a file, .github/clear_github_actions_cache, with Unix timestamp as contents. The hashFiles(...) cache key prefix uses this and will generate a unique hash of this file as a prefix to the rest of the cache / restore-key.
Next time you use git push, the cache key will be invalidated... effectively running GitHub Actions with an empty cache.
Related
Is that possible to deploy the Laravel web application to shared hosting using GitHub Action & GitHub FTP Deploy? If possible how should I change the.github\workflows\master.yml?
on:
push:
branches:
- master
name: 🚀 Deploy website on push
jobs:
web-deploy:
name: 🎉 Deploy
runs-on: ubuntu-latest
steps:
- name: 🚚 Get latest code
uses: actions/checkout#v2
- name: 📂 Sync files
uses: SamKirkland/FTP-Deploy-Action#4.2.0
with:
server: ${{ secrets.ftp_server }}
username: ${{ secrets.ftp_username }}
password: ${{ secrets.ftp_password }}
server-dir: /
Looks like you're very close but are missing 2 important steps: set up a temporary PHP environment, and use that environment to install your dependencies (Composer).
GitHub Actions Setup
This guide assumes you have a working Laravel installation, a GitHub account, and a shared hosting account that you can access via FTP using a username/password.
I found this video https://www.youtube.com/watch?v=UNWIXYSZfZY helpful to get a basic understanding of how to deploy a simple application. To make this answer helpful to a wider range of people, I'll give a quick outline of my setup. There really aren't any Laravel specific steps.
Workflow directory set up
Create the directories .github\workflows at the root of your project. In the workflows directory, create a yml file named after the branch you want to push to your shared hosting account. Ex. master.yml, staging.yml, development.yml etc. If you only have a single branch then just create one file. The name is important and should match the name of the branch.
Design your workflow
This is very dependent on your project but assuming you have a basic Laravel application without the need for additional components such as Node, then this is a basic GitHub Action that works for me on a variety of projects.
A basic action file consists of 2 sections, the workflow, and the jobs. A workflow triggers the jobs.
Workflow
Lines 1-4 say this will run each time we push to the master branch.
on:
push:
branches:
- master
Line 5 is the name of this workflow and will show up on your Actions page. Set this to something descriptive.
name: 🚀 Deploy website on push (Master)
Setting up jobs
In this action, there are 5 jobs. Some take parameters, others don't. I'm not going to explain all the details here but have linked to the corresponding repositories if you need details.
Checkout your code so the workflow has access to it,
https://github.com/actions/checkout
name: 🚚 Get latest code
uses: actions/checkout#v2
Sets up a temporary PHP environment so you can run things like
Composer, https://github.com/shivammathur/setup-php. Make sure to set your PHP version here otherwise you could run into issues when installing Composer packages with an unexpected PHP version.
name: Setup PHP
uses: shivammathur/setup-php#v2
with:
php-version: 7.2
Caches your dependencies for faster deploys,
https://github.com/actions/cache
name: Cache Composer packages
id: composer-cache
uses: actions/cache#v2
with:
path: vendor
key: ${{ runner.os }}-php-${{ hashFiles('**/composer.lock') }}
restore-keys: |
${{ runner.os }}-php-
Install your dependencies from composer.json and composer.lock
files.
name: Install dependencies
run: composer install --prefer-dist --no-progress
Deploys your code to your remote shared hosting site,
https://github.com/SamKirkland/FTP-Deploy-Action. Note the use of ${{ secrets.ftp_username }} and ${{ secrets.ftp_password }}. These are set up in your repository's secrets section. See https://docs.github.com/en/actions/security-guides/encrypted-secrets
name: 📂 Sync files
uses: SamKirkland/FTP-Deploy-Action#4.0.0
with:
server: name_of_server.com
username: ${{ secrets.ftp_username }}
password: ${{ secrets.ftp_password }}
server-dir: public_html/
Final file
on:
push:
branches:
- master
name: 🚀 Deploy website on push (Master)
jobs:
web-deploy:
name: 🎉 Deploy
runs-on: ubuntu-latest
steps:
- name: 🚚 Get latest code
uses: actions/checkout#v2
- name: Setup PHP
uses: shivammathur/setup-php#v2
with:
php-version: 7.2
- name: Cache Composer packages
id: composer-cache
uses: actions/cache#v2
with:
path: vendor
key: ${{ runner.os }}-php-${{ hashFiles('**/composer.lock') }}
restore-keys: |
${{ runner.os }}-php-
- name: Install dependencies
run: composer install --prefer-dist --no-progress
- name: 📂 Sync files
uses: SamKirkland/FTP-Deploy-Action#4.0.0
with:
server: name_of_server.com
username: ${{ secrets.ftp_username }}
password: ${{ secrets.ftp_password }}
server-dir: public_html/
Running the workflow
Check-in .github\workflows\master.yml, and others if appropriate,
into your GitHub repository. Without these files checked in nothing
will happen when you push a change to the branch.
Go to your Actions tab and ensure the workflow shows up there.
Push a change to your branch and watch the Actions tab. Click into
the running action to see details about the run.
Fix any errors that show up in the console.
Finally, you mentioned in a comment something about NPM. If you have Node as a component in your project you can simply run two extra steps that will bundle your assets and will get deployed along with the rest of the code.
Good luck!
I'm using GitHub Actions.
I'm trying to create a self-hosted runner that will run on one of several cloud servers, but be able to control which specific server it runs on. This is to do rolling updates on machines that have local resources.
I've setup tags with the self-hosted runners, but so far am having to make 1..N separate YML scripts:
on:
workflow_dispatch:
jobs:
post-to-server1:
runs-on: [self-hosted, server1 ]
Then I have to repeat this script n number of times.
What I'd like to do is this:
name: 'Server Patching'
on:
workflow_dispatch:
inputs:
server-target:
description: 'Target Server:'
required: true
default: 'dev'
jobs:
test-params:
runs-on: [ self-hosted, ${{ github.event.inputs.server-target }}]
steps:
- name: Start cloning of ${{ github.event.inputs.server-target }}
run: |
echo "Starting patching of: '${{ github.event.inputs.server-target }}'"
Check failure on line 12 in .github/workflows/test_workflow.yml
GitHub Actions / .github/workflows/test_workflow.yml
Invalid workflow file
You have an error in your yaml syntax on line 12
Is there a way to do this?
I tagged this YAML also, because the variable substitution seems to be an issue with the YAML syntax as well (although it may be specific for GitHub Actions)
Update: Yes, you can, if you change this line:
runs-on: [ self-hosted, ${{ github.event.inputs.server-target }}]
to:
runs-on: ${{ github.event.inputs.server-target }}
The error actually happens on test-params: not the line below it.
My theory is that the syntax checker fires before the variable substitution?
I'm not sure what makes difference, but it works for me:
runs-on: [ self-hosted, "${{ github.event.inputs.server-target }}" ]
Creating a list like that does not seem to work with dynamic values.
As long as you choose labels that are not already in use by GitHub (e.g. ubuntu-latest, macos-11, ...) you should not have a problem with dropping the self-hosted label. So your current naming pattern should be fine by itself
That way you can just use this:
runs-on: ${{ github.event.inputs.serverToPatch }}
i'm setting up the Github Action, AWS EC2, CodeDeploy. All the configuration seems working well. But excepts one thing. I can not understand and how can i solve it. If someone have experiences about this please help me.
I'm using:
EC2 Rhel 8
Node project (VueJs framework)
This is my cicd.yml file
on:
push:
branches:
- paymentV2
name: Deploy VueJS to Amazon ECS
#on: [push]
jobs:
deploy:
name: Deploy
runs-on: ubuntu-latest
strategy:
matrix:
node-version: ['12.x']
appname: ['staging-aws-codedeploy']
deploy-group: ['staging']
repo: ['project/MyProject']
steps:
- uses: actions/checkout#v2
# Configure AWS credentials
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials#v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ap-southeast-1
# Deploy to AWS
- name: Deploy to AWS
run: |
aws deploy create-deployment \
--application-name ${{ matrix.appname }} \
--deployment-config-name CodeDeployDefault.OneAtATime \
--deployment-group-name ${{ matrix.deploy-group }} \
--file-exists-behavior OVERWRITE \
--description "GitHub Deployment for the ${{ matrix.appname }}-${{ github.sha }}" \
--github-location repository=${{ matrix.repo }},commitId=${{ github.sha }}
This is my appspec.yml
version: 0.0
os: linux
files:
- source: /
destination: /var/www/MyProject
hooks:
ApplicationStart:
- location: scripts/application_start.sh
timeout: 300
runas: root
#scripts/application_start.sh
#cd /var/www/MyProject
#npm run build
This is the log from Github action & CodeDeploy AWS
I've tried editing the Vision.vue file and created the pull request on Github. Everything was working well. But one thing i'm confusing is why the modified file is existed. Please refer the image below
=> What am i expected is the modified file shouldn't have existed. I thought that Github should be automatically run git pull to get all new source code.
I've some more research and found out --file-exists-behavior with OVERWRITE but it seems not working as i want.
https://docs.aws.amazon.com/cli/latest/reference/deploy/create-deployment.html
==> Once again, i have no experience about CD by Github action & CodeDeploy. Everyone please help me and advice me the right thing. Thank you so much.
After a period of learning, I understood that appspect and buildspec.yaml were just the way to build and deploy, but for the pull code, I used webhook (aws codebuild, AWS Code Pipeline, Github webhook) or schedule (crontab). And i've decided to user crontab for my project, scheduling to pull new source code every hours. Hope this sharing can help anyone. Tks
I have a Github action on pull_request, and am attempting to cache node_modules between jobs. I can only see cache working on a job re-run, but not when opening new PRs, even when there have been no changes to my node_modules.
Is this by design?
Each time it runs in a new PR, I see this output in the job:
Run actions/cache#v2
with:
path: ~/.npm
key: Linux-build-cache-node-modules-83fc6365b85dd061416fccd5993d48c2003c388bb2184fd57f28d1041d9d261e
restore-keys: Linux-build-cache-node-modules-
Linux-build-
Linux-
env:
cache-name: cache-node-modules
Cache not found for input keys: Linux-build-cache-node-modules-83fc6365b85dd061416fccd5993d48c2003c388bb2184fd57f28d1041d9d261e, Linux-build-cache-node-modules-, Linux-build-, Linux-
Yet, re-running the job will enjoy the cache hit:
Run actions/cache#v2
Received 54525952 of 109978126 (49.6%), 51.8 MBs/sec
Received 109978126 of 109978126 (100.0%), 67.8 MBs/sec
Cache Size: ~105 MB (109978126 B)
/usr/bin/tar --use-compress-program zstd -d -xf /home/runner/work/_temp/0bdf436d-9d4d-4397-b8ed-5910ff503949/cache.tzst -P -C /home/runner/work/argutopia/argutopia
Cache restored successfully
Cache restored from key: Linux-build-cache-node-modules-83fc6365b85dd061416fccd5993d48c2003c388bb2184fd57f28d1041d9d261e
Here is my action yml for reference:
name: PR CI
on:
pull_request:
branches: [main, develop]
jobs:
build:
runs-on: ubuntu-latest
strategy:
matrix:
node-version: [14.x]
steps:
- uses: actions/checkout#v2
with:
fetch-depth: 0
- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node#v1
with:
node-version: ${{ matrix.node-version }}
- name: Cache node modules
uses: actions/cache#v2
id: cache
env:
cache-name: cache-node-modules
with:
path: ~/.npm
key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('**/package-lock.json') }}
restore-keys: |
${{ runner.os }}-build-${{ env.cache-name }}-
${{ runner.os }}-build-
${{ runner.os }}-
- name: Install Dependencies
if: steps.cache.outputs.cache-hit != 'true'
run: |
echo '::debug::deps cache miss - installing deps'
npm install
Caching ~/.npm wont allow you to skip running npm install. It will save the global npm cache on this machine so that running npm install doesn't incur as much download time, but you'll then still need to build the local node_modules for your project.
If you want to be able to skip your install, you can cache the project node_modules folder as well and skip install on cache hits there, though this could present some issues down the road.
Use advanced alternative caching technique from this article
- name: Cache dependencies
id: cache
uses: actions/cache#v3
with:
path: ./node_modules
key: modules-${{ hashFiles('package-lock.json') }}
- name: Install dependencies
if: steps.cache.outputs.cache-hit != 'true'
run: npm ci --ignore-scripts
Hello there and thank you for reading my question, its my first one here.
I am working with CI/CD pipelines for a year now and I think they are pretty nice and convinient for developing Websites and Stuff. But in the last months I have more and more problems creating fast, efficient and smart pipelines without redundant dependency installs or similar. So I want to use as less computation ressources as possible while still have fast builds. I want to parallelize steps and use theire artifacts in another final step. For example the following GitHub Actions workflow:
My goal with this workflow is to just build a VueJS Single Page App and deploy it to the IBM Cloud. For that I need to install the npm dependencies and build the Vue App and also install the IBM Cloud CLI. After these two steps are finished the builded App should be pushed to the IBM Cloud.
I could just simply run all steps sequentially like this:
name: Deploy
on:
push:
branches: [main]
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- name: Checkout Code
uses: actions/checkout#v2
- name: Use Node.js 10.X
uses: actions/setup-node#v1
with:
node-version: '10.x'
- name: Cache Node Modules
uses: actions/cache#v2
env:
cache-name: cache-node-modules
with:
# npm cache files are stored in `~/.npm` on Linux/macOS
path: ~/.npm
key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('**/package-lock.json') }}
restore-keys: |
${{ runner.os }}-build-${{ env.cache-name }}-
${{ runner.os }}-build-
${{ runner.os }}-
- name: Install Dependencies
run: npm ci
- name: Build Page
run: npm run build
- name: Install IBM Cloud CLI
run: curl -fsSL https://clis.cloud.ibm.com/install/linux | sh
shell: bash
- name: Install Cloud Foundry CLI
run: ibmcloud cf install
shell: bash
- name: Authenticate with IBM Cloud CLI
run: ibmcloud login --apikey "${{ secrets.IBM_CLOUD_API_KEY }}" --no-region -g Default
shell: bash
- name: Target a Cloud Foundry org and space
run: ibmcloud target --cf-api "${{ secrets.IBM_CLOUD_CF_API }}" -o "${{ secrets.IBM_CLOUD_CF_ORG }}" -s "${{ secrets.IBM_CLOUD_CF_SPACE }}"
shell: bash
- name: Deploy to Cloud Foundry
run: ibmcloud cf push
shell: bash
But in my opinion this is very ugly and can be improved. So I tried to split the job into 3 parts: build, predeploy and deploy. The build job installs and builds the Vue App. The Predeploy job install the IBM CLI. These two jobs doesn't depend on each other so they can be parallized. But the last job, deploy, depends on both so I added the needs: [build, predeploy] value to it. So I have the following workflow to archive this:
### This will not work!
name: Deploy
on:
push:
branches: [main]
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout Code
uses: actions/checkout#v2
- name: Use Node.js 10.X
uses: actions/setup-node#v1
with:
node-version: '10.x'
- name: Cache Node Modules
uses: actions/cache#v2
env:
cache-name: cache-node-modules
with:
# npm cache files are stored in `~/.npm` on Linux/macOS
path: ~/.npm
key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('**/package-lock.json') }}
restore-keys: |
${{ runner.os }}-build-${{ env.cache-name }}-
${{ runner.os }}-build-
${{ runner.os }}-
- name: Install Dependencies
run: npm ci
- name: Build Page
run: npm run build
predeploy:
runs-on: ubuntu-latest
defaults:
run:
shell: bash
steps:
- name: Install IBM Cloud CLI
run: curl -fsSL https://clis.cloud.ibm.com/install/linux | sh
- name: Install Cloud Foundry CLI
run: ibmcloud cf install
- name: Authenticate with IBM Cloud CLI
run: ibmcloud login --apikey "${{ secrets.IBM_CLOUD_API_KEY }}" --no-region -g Default
- name: Target a Cloud Foundry org and space
run: ibmcloud target --cf-api "${{ secrets.IBM_CLOUD_CF_API }}" -o "${{ secrets.IBM_CLOUD_CF_ORG }}" -s "${{ secrets.IBM_CLOUD_CF_SPACE }}"
deploy:
needs: [build, predeploy]
runs-on: ubuntu-latest
steps:
- name: Deploy to Cloud Foundry
# Error: 'ibmcloud: command not found'
run: ibmcloud cf push
shell: bash
Which looks on the GUI like:
[![My GitHub Workflow on the GUI][1]][1]
But this workflow will error since the last job doesn't share the same environment as the other jobs. I am aware that I could use the up/download Artifact feature of GitHub Actions but this seems to me like using a lot of resources. But I dont want to use a lot of ressources for my pipeline, I dont need a lot of different virtual environments or build matrixes. (I know they are very good for large projects, but they seem a little overkill for my little site)
So here are my two final Questions:
Why is parallelism in CI/CD often complication and not straight forward?
How can I improve my current pipeline with parallelism and without redundant executions?
I am glad about every helpful advice or link. Thank you. :)
[1]: https://i.stack.imgur.com/qEqLs.png
I think your original workflow was already pretty efficient. As you mentioned, different jobs are executed on different runners and sometime the additional complexity and effort put into the synchronization/logic between workflows outweighs the benefits of parallelism. In your case I don't think it would make much sense to run your jobs in parallel.
For your first question, I don't think it's an issue specific to CI/CD pipelines. I am getting a bit out of scope here but you have similar issues in any code that does work in parallel or as a matter of fact in any work in general that is done in parallel anywhere. Being factories, teams, code, CI pipelines, as soon as the work is split up, there will be some sort of mechanism to manage the allocation of work and track its progress. Which will make it more complex.
Why GH workflows might seem less straightforward than other systems seem to be a better question and I think it comes does to how long it has been around. It's a pretty recent addition to github and as new features are progressively being added it gets easier and easier to work with.
Regarding other optimizations for your workflow, I would recommend trying to avoid redoing the same work every time the workflow run if it's not needed. You already do this with the cache action for npm. But you could, for example build a docker image, or even better an action ,with your IBM CLI in it and remove the pre-deploy stage entirely. Simply having:
name: Deploy
on:
push:
branches: [main]
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- name: Checkout Code
uses: actions/checkout#v2
- name: Use Node.js 10.X
uses: actions/setup-node#v1
with:
node-version: '10.x'
- name: Cache Node Modules
uses: actions/cache#v2
env:
cache-name: cache-node-modules
with:
# npm cache files are stored in `~/.npm` on Linux/macOS
path: ~/.npm
key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('**/package-lock.json') }}
restore-keys: |
${{ runner.os }}-build-${{ env.cache-name }}-
${{ runner.os }}-build-
${{ runner.os }}-
- name: Install Dependencies
run: npm ci
- name: Build Page
run: npm run build
- name: Deploy to Cloud Foundry
uses: my-action:v1
with:
api-key: ${{ secrets.IBM_CLOUD_API_KEY }}
cf-api: ${{ secrets.IBM_CLOUD_CF_API }}
cf-org: ${{ secrets.IBM_CLOUD_CF_ORG }}
cf-space: ${{ secrets.IBM_CLOUD_CF_SPACE }}