I'm trying to implement caching for my workflow using actions/cache#v2 (https://github.com/actions/cache) action.
I have a workflow which runs on pull requests.My project has tons of dependencies so it would be a great build time improvement if I could re-use artifacts, downloaded previously. After successful run artifacts got cached within this pull request/branch and can be re-used on next runs for this pull request:
Cache restored successfully
Cache restored from key: Linux-823bfbee4d05185dedf13d718e8e4ff4933074565b84dd636e19434372154c6a-8331f72d4c267e28baf5311f561dc9f09f1813899db5fd4929c70715b85b2934-45275122110c339513af5e3da3a953eed4a843d91a5d03cd48b8a26988749289
From this description: https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows
I got an impression, that the cache (including gradle dependencies, like libraries etc), once downloaded, should be available for other branches based on my develop, after one of the PRs got merged:
A workflow can access and restore a cache created in the current branch, the base branch (including base branches of forked repositories), or the default branch (usually main). For example, a cache created on the default branch would be accessible from any pull request. Also, if the branch feature-b has the base branch feature-a, a workflow triggered on feature-b would have access to caches created in the default branch (main), feature-a, and feature-b.
But apparently, it doesn't work for me, I'm getting this for all the new PRs:
Run actions/cache#v2
Cache not found for input keys: Linux-823bfbee4d05185dedf13d718e8e4ff4933074565b84dd636e19434372154c6a-8331f72d4c267e28baf5311f561dc9f09f1813899db5fd4929c70715b85b2934-45275122110c339513af5e3da3a953eed4a843d91a5d03cd48b8a26988749289
So I'm getting all the gradle dependencies downloaded, over and over again.
So my question is: is it possible to have the dependencies, once downloaded, be available from cache for all other branches/pull requests. If yes, can this be done using actions/cache#v2 or shall I look to another tasks/tools?
Here's simplified version of my workflow:
name: build-pull-request
on:
pull_request:
types:
- opened
- synchronize
- reopened
branches:
- '**'
workflow_dispatch:
jobs:
verify-pull-request-quality:
name: Verify pull request
runs-on: ubuntu-latest
steps:
- name: Cancel previous runs of this workflow
uses: styfle/cancel-workflow-action#0.9.1
with:
access_token: ${{ github.token }}
- name: Check out ${{ github.ref }}
uses: actions/checkout#v2
- name: Cache gradle dependencies
uses: actions/cache#v2
env:
cache-name: cache-gradle-depedencies
with:
path: |
~/.gradle/caches
~/.gradle/wrapper
key: ${{ runner.os }}-${{ hashFiles('**/*.gradle*') }}-${{
hashFiles('**/gradle/wrapper/gradle-wrapper.properties') }}-${{
hashFiles('**/buildSrc/**/*.kt') }}
- name: Run unit test
uses: gradle/gradle-build-action#v2
with:
build-root-directory: ${{env.working-directory}}
arguments: assemble
Related
I have developed an application where the backend is developed using Java language (with maven) and the frontend is using Angular language. I host both parts in one project in github.
Now I am interested on the use of SonarQube on SonarCloud. For this purpose, I am following the information gathered from the community sonarsource and the standard documentation from sonarcloud. The idea is to use GiHub Actions for analyzing the projects.
What I have created is a .github/workflows/build.yml on the root folder with the content:
name: Build
on:
push:
branches:
- master
pull_request:
types: [opened, synchronize, reopened]
jobs:
sonarcloud:
name: SonarCloud
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
with:
fetch-depth: 0 # Shallow clones should be disabled for a better relevancy of analysis
- name: Set up JDK 11
uses: actions/setup-java#v1
with:
java-version: 11
- name: Cache SonarCloud packages
uses: actions/cache#v1
with:
path: ~/.sonar/cache
key: ${{ runner.os }}-sonar
restore-keys: ${{ runner.os }}-sonar
- name: Cache Maven packages
uses: actions/cache#v1
with:
path: ~/.m2
key: ${{ runner.os }}-m2-${{ hashFiles('**/pom.xml') }}
restore-keys: ${{ runner.os }}-m2
- name: Build and analyze
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} # Needed to get PR information, if any
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
run: mvn -B verify org.sonarsource.scanner.maven:sonar-maven-plugin:sonar -Dsonar.projectKey=kendo-tournament-backend
with:
projectBaseDir: ./backend/
- name: SonarCloud Frontend Scan
uses: SonarSource/sonarcloud-github-action#master
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} # Needed to get PR information, if any
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
with:
projectBaseDir: ./frontend/
Where basically, I create some steps for executing the maven project, and another extra step to execute the frontend. On both of them, I included the projectBaseDir with the path to both projects' folders as specified here.
Also, as suggested on the documentation, I have included a sonar-project.properties on the root folder of the frontend folder with:
sonar.projectKey=kendo-tournament-frontend
sonar.organization=softwaremagico
# This is the name and version displayed in the SonarCloud UI.
#sonar.projectName=Kendo Tournament Manager Frontend
#sonar.projectVersion=1.0
# Path is relative to the sonar-project.properties file. Replace "\" by "/" on Windows.
#sonar.sources=.
# Encoding of the source code. Default is default system encoding
#sonar.sourceEncoding=UTF-8
And for the backend, I have updated the root pom.xml with the:
<properties>
<sonar.organization>softwaremagico</sonar.organization>
<sonar.host.url>https://sonarcloud.io</sonar.host.url>
</properties>
As required.
But, no analyses scan is launch for any of both projects. And SonarCloud looks like is ignoring the configuration.
Probably, something is missing but I cannot imagine what. What steps are needed to set up a monorepository correctly using Java and Angular in Github?
Ok, after the example obtained from here. The changes I have made are:
Two different workflows on github, one for backend and one for frontend. Not one workflow with all steps together.
Include two different sonar-project.properties. One inside the backend folder, and one inside the frontend folder. Now I have added the sonar.sources line as follows:
sonar.projectKey=kendo-tournament-backend
sonar.organization=softwaremagico
sonar.sources=.
That ensures that is only for this folder.
For launching CircleCi with Sonar (for backend) edit file .circleci/config.yml:
version: 2.1
jobs:
build:
docker:
- image: 'circleci/openjdk:11-jdk'
working_directory: ~/KendoTournamentManager/backend
steps:
- checkout:
path: ~/KendoTournamentManager
- run:
name: Analyze on SonarCloud
command: mvn verify sonar:sonar -Dsonar.projectKey=kendo-tournament-backend
workflows:
main:
jobs:
- build:
context: SonarCloud
And now seems working fine:
Is that possible to deploy the Laravel web application to shared hosting using GitHub Action & GitHub FTP Deploy? If possible how should I change the.github\workflows\master.yml?
on:
push:
branches:
- master
name: 🚀 Deploy website on push
jobs:
web-deploy:
name: 🎉 Deploy
runs-on: ubuntu-latest
steps:
- name: 🚚 Get latest code
uses: actions/checkout#v2
- name: 📂 Sync files
uses: SamKirkland/FTP-Deploy-Action#4.2.0
with:
server: ${{ secrets.ftp_server }}
username: ${{ secrets.ftp_username }}
password: ${{ secrets.ftp_password }}
server-dir: /
Looks like you're very close but are missing 2 important steps: set up a temporary PHP environment, and use that environment to install your dependencies (Composer).
GitHub Actions Setup
This guide assumes you have a working Laravel installation, a GitHub account, and a shared hosting account that you can access via FTP using a username/password.
I found this video https://www.youtube.com/watch?v=UNWIXYSZfZY helpful to get a basic understanding of how to deploy a simple application. To make this answer helpful to a wider range of people, I'll give a quick outline of my setup. There really aren't any Laravel specific steps.
Workflow directory set up
Create the directories .github\workflows at the root of your project. In the workflows directory, create a yml file named after the branch you want to push to your shared hosting account. Ex. master.yml, staging.yml, development.yml etc. If you only have a single branch then just create one file. The name is important and should match the name of the branch.
Design your workflow
This is very dependent on your project but assuming you have a basic Laravel application without the need for additional components such as Node, then this is a basic GitHub Action that works for me on a variety of projects.
A basic action file consists of 2 sections, the workflow, and the jobs. A workflow triggers the jobs.
Workflow
Lines 1-4 say this will run each time we push to the master branch.
on:
push:
branches:
- master
Line 5 is the name of this workflow and will show up on your Actions page. Set this to something descriptive.
name: 🚀 Deploy website on push (Master)
Setting up jobs
In this action, there are 5 jobs. Some take parameters, others don't. I'm not going to explain all the details here but have linked to the corresponding repositories if you need details.
Checkout your code so the workflow has access to it,
https://github.com/actions/checkout
name: 🚚 Get latest code
uses: actions/checkout#v2
Sets up a temporary PHP environment so you can run things like
Composer, https://github.com/shivammathur/setup-php. Make sure to set your PHP version here otherwise you could run into issues when installing Composer packages with an unexpected PHP version.
name: Setup PHP
uses: shivammathur/setup-php#v2
with:
php-version: 7.2
Caches your dependencies for faster deploys,
https://github.com/actions/cache
name: Cache Composer packages
id: composer-cache
uses: actions/cache#v2
with:
path: vendor
key: ${{ runner.os }}-php-${{ hashFiles('**/composer.lock') }}
restore-keys: |
${{ runner.os }}-php-
Install your dependencies from composer.json and composer.lock
files.
name: Install dependencies
run: composer install --prefer-dist --no-progress
Deploys your code to your remote shared hosting site,
https://github.com/SamKirkland/FTP-Deploy-Action. Note the use of ${{ secrets.ftp_username }} and ${{ secrets.ftp_password }}. These are set up in your repository's secrets section. See https://docs.github.com/en/actions/security-guides/encrypted-secrets
name: 📂 Sync files
uses: SamKirkland/FTP-Deploy-Action#4.0.0
with:
server: name_of_server.com
username: ${{ secrets.ftp_username }}
password: ${{ secrets.ftp_password }}
server-dir: public_html/
Final file
on:
push:
branches:
- master
name: 🚀 Deploy website on push (Master)
jobs:
web-deploy:
name: 🎉 Deploy
runs-on: ubuntu-latest
steps:
- name: 🚚 Get latest code
uses: actions/checkout#v2
- name: Setup PHP
uses: shivammathur/setup-php#v2
with:
php-version: 7.2
- name: Cache Composer packages
id: composer-cache
uses: actions/cache#v2
with:
path: vendor
key: ${{ runner.os }}-php-${{ hashFiles('**/composer.lock') }}
restore-keys: |
${{ runner.os }}-php-
- name: Install dependencies
run: composer install --prefer-dist --no-progress
- name: 📂 Sync files
uses: SamKirkland/FTP-Deploy-Action#4.0.0
with:
server: name_of_server.com
username: ${{ secrets.ftp_username }}
password: ${{ secrets.ftp_password }}
server-dir: public_html/
Running the workflow
Check-in .github\workflows\master.yml, and others if appropriate,
into your GitHub repository. Without these files checked in nothing
will happen when you push a change to the branch.
Go to your Actions tab and ensure the workflow shows up there.
Push a change to your branch and watch the Actions tab. Click into
the running action to see details about the run.
Fix any errors that show up in the console.
Finally, you mentioned in a comment something about NPM. If you have Node as a component in your project you can simply run two extra steps that will bundle your assets and will get deployed along with the rest of the code.
Good luck!
I have a scan step built into my GitHub Action build and that is working fine. I reach out to my company's SonarQub instance and the scan is initiated. The problem I am having is trying to stop a build if there is a failure. For the life of me I can't seem to find a way to do that. Also, when I watch the scan it appears as though the next steps might be happening before it finishes (not positive on that but thought I would mention it). Any ideas??
name: Build, test, & deploy
on: [push]
jobs:
sonarqube:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
with:
# Disabling shallow clone is recommended for improving relevancy of reporting
fetch-depth: 0
# Triggering SonarQube analysis as results of it are required by Quality Gate check
- name: SonarQube Scan
uses: sonarsource/sonarqube-scan-action#master
env:
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
SONAR_HOST_URL: ${{ secrets.SONAR_HOST_URL }}
- name: SonarQube Quality Gate check
uses: sonarsource/sonarqube-quality-gate-action#master
# Force to fail step after specific time
timeout-minutes: 5
env:
SONAR_TOKEN: ${{ secrets.ADAM_SONAR_TOKEN }}
build:
name: Project build & package
if: "!contains(github.even.head_commit.message, '[skip-ci]')"
runs-on: ubuntu-latest
env:
#environment var for this job
#### the rest of the build is below this area - I didn't think it was necessary to include
You should use needs in your build job:
build:
needs: sonarqube
name: Project build & package
You can find information here: https://docs.github.com/en/actions/using-jobs/using-jobs-in-a-workflow
The answer is using "needs": https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idneeds
I'm trying to put together a CI / CD pipeline for GitHub and strugglign with attaching my build artifact to the release. Here's my ci.yml
name: CI
on:
push:
branches: [ main ]
pull_request:
branches: [ main ]
workflow_dispatch:
jobs:
Build:
runs-on: windows-latest
env:
BuildPath: ${{ github.workspace }}\BuildTesting\bin\Release\net5.0-windows
steps:
- uses: actions/checkout#v2
- name: Setup .NET SDK
uses: actions/setup-dotnet#v1.8.1
with:
dotnet-version: 5.0.x
- name: Install dependencies
run: dotnet restore
- name: Build
run: dotnet build --configuration Release --no-restore
- name: Upload a Build Artifact
uses: actions/upload-artifact#v2.2.4
with:
name: thingy
path: ${{ github.workspace }}\BuildTesting\bin\Release\net5.0-windows
retention-days: 1
That runs well and I get a release artifact:
I was under the impression I'd be able to download that existing artifact but I couldn't get my head around why actions/download-artifact isn't downloading anything. So I found another article and in their cd step they were re-building, so I figured that in doing that at least I'd have a fresh build in the cd workflow to pull from. So I create a release triggered on tag push events. I can't use most zip utilities becuase they don't run on windows. I have to user windows-latest as the target framework for WPF desktop applications has to be net5.0-windows and using ubuntu-latest it fails. I tried papeloto/action-zip#v1 and in one case I managed to get a zip file which then attached to the release successfully but was only 22 bytes, so empty once I downloaded it. Here's my cd.yml:
name: CD
on:
push:
tags:
- '*'
jobs:
Release:
runs-on: windows-latest
env:
BuildPath: ${{ github.workspace }}\BuildTesting\bin\Release\net5.0-windows
ZipName: TheThing.zip
steps:
# Build the solution
- uses: actions/checkout#v2
- name: Setup .NET SDK
uses: actions/setup-dotnet#v1.8.1
with:
dotnet-version: 5.0.x
- name: Install dependencies
run: dotnet restore
- name: Build
run: dotnet build --configuration Release --no-restore
- name: Zip the release
uses: << What should I use to zip ${{ env.BuildPath }} ? >>
# Create a Release on the GitHub project
- name: Create release
id: create_release
uses: actions/create-release#v1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} # This token is provided by Actions, you do not need to create your own token
with:
tag_name: ${{ github.ref }}
release_name: ${{ github.ref }}
draft: false
prerelease: false
# Upload the Build Artifact to the Release
- name: Update release asset
id: upload-release-asset
uses: actions/upload-release-asset#v1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
upload_url: ${{ steps.create_release.outputs.upload_url }} # This pulls from the CREATE RELEASE step above, referencing it's ID to get its outputs object, which include a `upload_url`. See this blog post for more info: https://jasonet.co/posts/new-features-of-github-actions/#passing-data-to-future-steps
asset_path: .\${{ env.ZipName }}
asset_name: ${{ env.ZipName }}
asset_content_type: application/zip
Can anyone please recommend a GitHub action for zipping a folder that will work for windows-latest ... or another approach?
Long term I want to go with semantic versioning but GitVersion overwhelmed me last time I tried, granted I'd never working with yml builds before. I'd like to keep this as simple as possible as I'm starting to grok the basics of what's going on and once I get past this I'll start looking into GitVersion.
I'm using a test repository located here so you can see the whole thing.
You may use PowerShell to zip your artifacts using 7Zip. Please check my repository in GitHub. That compile a WPF app, compress the artifacts, create a prerelease and upload files under that release.
Hello there and thank you for reading my question, its my first one here.
I am working with CI/CD pipelines for a year now and I think they are pretty nice and convinient for developing Websites and Stuff. But in the last months I have more and more problems creating fast, efficient and smart pipelines without redundant dependency installs or similar. So I want to use as less computation ressources as possible while still have fast builds. I want to parallelize steps and use theire artifacts in another final step. For example the following GitHub Actions workflow:
My goal with this workflow is to just build a VueJS Single Page App and deploy it to the IBM Cloud. For that I need to install the npm dependencies and build the Vue App and also install the IBM Cloud CLI. After these two steps are finished the builded App should be pushed to the IBM Cloud.
I could just simply run all steps sequentially like this:
name: Deploy
on:
push:
branches: [main]
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- name: Checkout Code
uses: actions/checkout#v2
- name: Use Node.js 10.X
uses: actions/setup-node#v1
with:
node-version: '10.x'
- name: Cache Node Modules
uses: actions/cache#v2
env:
cache-name: cache-node-modules
with:
# npm cache files are stored in `~/.npm` on Linux/macOS
path: ~/.npm
key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('**/package-lock.json') }}
restore-keys: |
${{ runner.os }}-build-${{ env.cache-name }}-
${{ runner.os }}-build-
${{ runner.os }}-
- name: Install Dependencies
run: npm ci
- name: Build Page
run: npm run build
- name: Install IBM Cloud CLI
run: curl -fsSL https://clis.cloud.ibm.com/install/linux | sh
shell: bash
- name: Install Cloud Foundry CLI
run: ibmcloud cf install
shell: bash
- name: Authenticate with IBM Cloud CLI
run: ibmcloud login --apikey "${{ secrets.IBM_CLOUD_API_KEY }}" --no-region -g Default
shell: bash
- name: Target a Cloud Foundry org and space
run: ibmcloud target --cf-api "${{ secrets.IBM_CLOUD_CF_API }}" -o "${{ secrets.IBM_CLOUD_CF_ORG }}" -s "${{ secrets.IBM_CLOUD_CF_SPACE }}"
shell: bash
- name: Deploy to Cloud Foundry
run: ibmcloud cf push
shell: bash
But in my opinion this is very ugly and can be improved. So I tried to split the job into 3 parts: build, predeploy and deploy. The build job installs and builds the Vue App. The Predeploy job install the IBM CLI. These two jobs doesn't depend on each other so they can be parallized. But the last job, deploy, depends on both so I added the needs: [build, predeploy] value to it. So I have the following workflow to archive this:
### This will not work!
name: Deploy
on:
push:
branches: [main]
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout Code
uses: actions/checkout#v2
- name: Use Node.js 10.X
uses: actions/setup-node#v1
with:
node-version: '10.x'
- name: Cache Node Modules
uses: actions/cache#v2
env:
cache-name: cache-node-modules
with:
# npm cache files are stored in `~/.npm` on Linux/macOS
path: ~/.npm
key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('**/package-lock.json') }}
restore-keys: |
${{ runner.os }}-build-${{ env.cache-name }}-
${{ runner.os }}-build-
${{ runner.os }}-
- name: Install Dependencies
run: npm ci
- name: Build Page
run: npm run build
predeploy:
runs-on: ubuntu-latest
defaults:
run:
shell: bash
steps:
- name: Install IBM Cloud CLI
run: curl -fsSL https://clis.cloud.ibm.com/install/linux | sh
- name: Install Cloud Foundry CLI
run: ibmcloud cf install
- name: Authenticate with IBM Cloud CLI
run: ibmcloud login --apikey "${{ secrets.IBM_CLOUD_API_KEY }}" --no-region -g Default
- name: Target a Cloud Foundry org and space
run: ibmcloud target --cf-api "${{ secrets.IBM_CLOUD_CF_API }}" -o "${{ secrets.IBM_CLOUD_CF_ORG }}" -s "${{ secrets.IBM_CLOUD_CF_SPACE }}"
deploy:
needs: [build, predeploy]
runs-on: ubuntu-latest
steps:
- name: Deploy to Cloud Foundry
# Error: 'ibmcloud: command not found'
run: ibmcloud cf push
shell: bash
Which looks on the GUI like:
[![My GitHub Workflow on the GUI][1]][1]
But this workflow will error since the last job doesn't share the same environment as the other jobs. I am aware that I could use the up/download Artifact feature of GitHub Actions but this seems to me like using a lot of resources. But I dont want to use a lot of ressources for my pipeline, I dont need a lot of different virtual environments or build matrixes. (I know they are very good for large projects, but they seem a little overkill for my little site)
So here are my two final Questions:
Why is parallelism in CI/CD often complication and not straight forward?
How can I improve my current pipeline with parallelism and without redundant executions?
I am glad about every helpful advice or link. Thank you. :)
[1]: https://i.stack.imgur.com/qEqLs.png
I think your original workflow was already pretty efficient. As you mentioned, different jobs are executed on different runners and sometime the additional complexity and effort put into the synchronization/logic between workflows outweighs the benefits of parallelism. In your case I don't think it would make much sense to run your jobs in parallel.
For your first question, I don't think it's an issue specific to CI/CD pipelines. I am getting a bit out of scope here but you have similar issues in any code that does work in parallel or as a matter of fact in any work in general that is done in parallel anywhere. Being factories, teams, code, CI pipelines, as soon as the work is split up, there will be some sort of mechanism to manage the allocation of work and track its progress. Which will make it more complex.
Why GH workflows might seem less straightforward than other systems seem to be a better question and I think it comes does to how long it has been around. It's a pretty recent addition to github and as new features are progressively being added it gets easier and easier to work with.
Regarding other optimizations for your workflow, I would recommend trying to avoid redoing the same work every time the workflow run if it's not needed. You already do this with the cache action for npm. But you could, for example build a docker image, or even better an action ,with your IBM CLI in it and remove the pre-deploy stage entirely. Simply having:
name: Deploy
on:
push:
branches: [main]
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- name: Checkout Code
uses: actions/checkout#v2
- name: Use Node.js 10.X
uses: actions/setup-node#v1
with:
node-version: '10.x'
- name: Cache Node Modules
uses: actions/cache#v2
env:
cache-name: cache-node-modules
with:
# npm cache files are stored in `~/.npm` on Linux/macOS
path: ~/.npm
key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('**/package-lock.json') }}
restore-keys: |
${{ runner.os }}-build-${{ env.cache-name }}-
${{ runner.os }}-build-
${{ runner.os }}-
- name: Install Dependencies
run: npm ci
- name: Build Page
run: npm run build
- name: Deploy to Cloud Foundry
uses: my-action:v1
with:
api-key: ${{ secrets.IBM_CLOUD_API_KEY }}
cf-api: ${{ secrets.IBM_CLOUD_CF_API }}
cf-org: ${{ secrets.IBM_CLOUD_CF_ORG }}
cf-space: ${{ secrets.IBM_CLOUD_CF_SPACE }}