I have a Spring boot application which I want to auto-deploy to App Engine. I don't want to create the docker image then deploy it. The build is failing due to 'Cloud SDK not found error'
[ERROR] Failed to execute goal com.google.cloud.tools:appengine-maven-plugin:1.3.2:deploy (default-cli) on project location-finder-rest-api: Execution default-cli of goal com.google.cloud.tools:appengine-maven-plugin:1.3.2:deploy failed: The Google Cloud SDK could not be found in the customary locations and no path was provided.
I followed all the guidelines at https://cloud.google.com/source-repositories/docs/quickstart-triggering-builds-with-source-repositories.
As per the documentation, app.yaml file is created at src/main/appengine. The contents of app.yaml is
# [START runtime]
runtime: java
env: flex
handlers:
- url: /.*
script: this field is required, but ignored
runtime_config: # Optional
jdk: openjdk8
manual_scaling:
instances: 1
# [END runtime]
In order to trigger the build, I have to specify the cloudbuild.yaml file. The contents of this file are:
steps:
- name: 'gcr.io/cloud-builders/mvn'
args: ['appengine:deploy','-Pprod']
The official document for cloud-builder suggest using 'install' as an argument to the mvn step. But this step does not deploy the application.
Am I missing any configuration?
Under the hood, the appengine:deploy goal uses the Cloud SDK to actually deploy your app. It isn't provided by the gcr.io/cloud-builders/mvn image (each Cloud Build step runs in its own container).
You could use separate build steps to install and deploy your app, something like:
steps:
- name: 'gcr.io/cloud-builders/mvn'
args: ['install']
- name: 'gcr.io/cloud-builders/gcloud'
args: ['app', 'deploy']
It worked by making slight modifications to the solution suggested above by LundinCast. Moreover, appengine maven plugin needs to be updated to 2.0.0+. This version automatically downloads the necessary dependencies.
steps:
- id: 'Stage app using mvn appengine plugin on mvn cloud build image'
name: 'gcr.io/cloud-builders/mvn'
args: ['package', 'appengine:stage', '-Pprod']
- id: "Deploy to app engine using gcloud image"
name: 'gcr.io/cloud-builders/gcloud'
args: ['app', 'deploy', 'target/appengine-staging/app.yaml']
Related
What is the best practice when you run Spring Boot Tests that depend on Testcontainers on Google Cloud Build?
Currently we have the following setup (unrelevant steps ommited):
steps:
# Git
- name: 'gcr.io/cloud-builders/git'
args: ['-c', 'http.sslVerify=false', 'clone', '<my source repository>',
'--branch', 'my-tag', '--single-branch']
# Build
- name: 'gcr.io/cloud-builders/mvn'
args: ['package', '-DskipTests=true']
dir: src-dir
We have added some tests that use Testcontainers, and we want to enable the tests for the build on the cloud, but to run them on the cloud we need a cloud builder that have both Maven and docker.
I can't run the tests, on different step, separately from the build, as far as I know.
Is there a cloud builder that combines maven and docker or the only way is to create a custom one?
Thanks!
I'm new to github actions (comming from gitlab-ci) I'm trying to run a integration test with testcontainers in the pipeline and I'm stucked. Here is my current definition.
name: Run Gradle
on: push
jobs:
gradle:
strategy:
matrix:
os: [ ubuntu-18.04 ]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout#v1
- uses: actions/setup-java#v1
with:
java-version: 11
- uses: eskatos/gradle-command-action#v1
with:
build-root-directory: backend
wrapper-directory: backend
arguments: check assemble
How can I ensure that the docker deamon for testcontainers project is available during my run?
You can check the installed packages/software for each GitHub Actions Runner as part of the virtual-environment GitHub repository.
For ubuntu-18.04 you can find the list here. Docker and Docker Compose are already installed on the runner and you can use them without any additional configuration for Testcontainers.
I'm using GitHub Actions for a lot of projects that make heavy use of Testcontainers without any problems.
I have created SPFX library module in which there is code to be shared with multiple webparts similar to the one here. I used npm link <lib name> command to link it. It works pretty well in the local environment as well as when I deploy it manually in SharePoint online app catalog. But if I deploy it using Azure Pipelines (YAML script) it always throws and error for library not found.
I have made it sure that the library is built first and then other webparts are built in pipeline (by introducting stages) but still it doesn't find the library. Is there a way to run npm link as a pipeline task?
If the library is first built in a different stage or job, the library will not be located in the same agent machine with the other webparts.
Let's say the library is built in Stage A and the other webparts are built in Stage B. As workaround, you will need to publish the library as artifacts to azure devops server using Publish Build Artifacts task in stage A. And then download the library to the agent which builds the other webparts in Stage B using Download Build Artifacts task.
For below example:
The library is built in stage A and published to azure devops server as artifacts named library.
In Stage B, the library artifacts is downloaded from azure devops server to folder $(Build.ArtifactStagingDirectory). Then you can refer to the library by path $(Build.ArtifactStagingDirectory)/library for the following tasks in Stage B.
If multiple webparts are built in different jobs in Stage B, each job in Stage B will need to download the library artifacts, for different jobs run on different agent virtual machines.
stages:
- stage: A
jobs:
- job: Library
pool:
vmImage: "windows-latest"
steps:
...
- task: PublishBuildArtifacts#1
inputs:
PathtoPublish: {path to the library}
ArtifactName: library
- stage: B
dependsOn: A
jobs:
- job: WebParts1
pool:
vmImage: "windows-latest"
steps:
- task: DownloadBuildArtifacts#0
inputs:
downloadPath: $(Build.ArtifactStagingDirectory)
buildType: current
artifactName: library
...
- job: WebParts2
pool:
vmImage: "windows-latest"
steps:
- task: DownloadBuildArtifacts#0
inputs:
downloadPath: $(Build.ArtifactStagingDirectory)
buildType: current
artifactName: library
...
After trying the solutions posted above it still didn't work so I published it as NPM package using npm publish command. This solved the problem.
We can also use this command in YAML to automatically publish the package everytime we build but that was not required in my case so I didn't use it.
I have setup CI for a .NET Core solution using GitHub Actions. When code is pushed to the master branche, the solution is build, the unit tests are run and code analysis is run with SonarCloud.
The code analysis step is actually performed by sonarcloud-github-action.
The quality gate in SonarCloud does not pass because the coverage percentage is 0.0% (for both new as existing code). I'm generating code coverage reports using Coverlet. The coverage.opencover.xml file is succesfully generated after test execution for each unit test project.
In the sonar-project.properties file I'm referencing these files as follows:
sonar.cs.opencover.reportsPaths=**\coverage.opencover.xml
But apparently the code coverage reports are recognized but not processed by the SonarCloud scanner.
In the log of my GitHub Actions workflow, I do see these warnings:
INFO: Parsing the OpenCover report <path>/coverage.opencover.xml
INFO: Adding this code coverage report to the cache for later reuse: <path>/coverage.opencover.xml
...
WARN: Missing blame information for the following files:
WARN: * <path>/coverage.opencover.xml
WARN: This may lead to missing/broken features in SonarQube
In trying to solve the 'Missing blame information' warning, I added the coverage files to the exclusions in my SonarCloud project: **/coverage.opencover.xml but that didn't solve the issue. The warning still appears and code coverage is still 0.0%.
Any hints to get this going?
[edit]:
My workflow in GitHub Actions looks like this:
name: .NET Core
on: [push]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v1
- name: Setup .NET Core
uses: actions/setup-dotnet#v1
with:
dotnet-version: 2.2.108
- name: Build with dotnet
run: dotnet build src/<solution>.sln --configuration Release
- name: Unit Tests
run: dotnet test src/<solution>.sln /p:CollectCoverage=true /p:CoverletOutputFormat=opencover
- name: SonarCloud Scan
uses: sonarsource/sonarcloud-github-action#master
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
`
I had a similar problem getting the coverage of a Typescript project to work. Without your sonar logs I just can guess but the problem was that the paths inside the lcov.info where absolute path from github something like SF:/home/runner/work/YoutRepoName.. and Sonar was starting a Docker container and set the workdir to /github/workdir and therefore could not locate the files from the lcov.info.
Check your logs if you find something like
2019-11-28T15:36:34.9243068Z WARN: Could not resolve 2 file paths in [/github/workspace/test/unit/coverage/lcov.info], first unresolved path: /home/runner/work/jobreporter/jobreporter/dispatcher/index.ts
2019-11-28T15:36:34.9243445Z INFO: Sensor SonarJS Coverage [javascript] (done) | time=8ms
So for the time being i had to replace all folder namens in the locv.info with /github/workdir.
In my case i used
- name: 'Run npm lint and test'
shell: bash
run: |
pushd .
npm ci
npm run lint:ci
npm run test --if-present
sed -i 's+/home/runner/work/jobreporter/jobreporter+/github/workspace+g' test/unit/coverage/lcov.info
sed -i 's+/home/runner/work/jobreporter/jobreporter+/github/workspace+g' eslint-report.json
After that the coverage was reported correctly.
Maybe that helps
Regards Mathias
I had the same problem with a Node build, where the paths in the lcov.info are not the same as the one in the Github Action docker container.
To work around it, I do my builds not by setting up Node directly in the worker, but by using a Docker Action, so that my paths stay the same in all Actions. If you dig in the logs, you can see precisely how the docker actions are run, and the available environment.
For reference, my actions look like this
- name: 'yarn install'
uses: docker://node:10.16.3-buster
with:
args: yarn install
env:
NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
CI: true
The downside is that my builds are a bit slower, but all my actions are run in Docker, which I find cleaner.
To get past this error you need to run your tests with the --blame parameter.
Here is my GitHub action for building and pushing to SonarCloud.
name: Build and run tests
on:
push:
branches: [ master ]
pull_request:
branches: [ master ]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
with:
# Disabling shallow clone is recommended for improving relevancy of reporting for sonarcloud
fetch-depth: 0
- name: Setup .Net SDK (v5.0)
uses: actions/setup-dotnet#v1
with:
dotnet-version: '5.0.100'
- name: Install dependencies
run: dotnet restore
- name: Build
run: dotnet build --configuration Release --no-restore
- name: Test
run: dotnet test --blame --no-restore --verbosity normal /p:CollectCoverage=true /p:CoverletOutputFormat=opencover /p:CoverletOutput=opencover.xml
- name: SonarCloud Scan
uses: sonarsource/sonarcloud-github-action#master
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
(Please note i'm a total beginner in all Docker and CI in general)
I am trying to set up a simple CI environment with GitLab. I'm using a local GitLab runner which is configured to build and test in Docker. In registering the runner I chose the 'shell' option. This installation succeeded.
After pushing my code I got an email saying 'build failed'. In the build log I found the following:
on desktop-docker-runner cDD_yf4V
Using Shell executor...
Running on b567d1ba4654...
DEPRECATION: this GitLab server doesn't support refspecs, gitlab-runner 12.0 will no longer work with this version of GitLab
Fetching changes...
Checking out fd20ca86 as dev...
Skipping object checkout, Git LFS is not installed.
Skipping Git submodules setup
Checking cache for default...
Runtime platform arch=amd64 os=linux pid=722 revision=1f513601 version=11.10.1
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted.
Successfully extracted cache
$ mvn $MAVEN_CLI_OPTS test-compile
bash: line 74: mvn: command not found
ERROR: Job failed: exit status 1
How can I add Maven to resolve this?
(EDIT: added gitlab-ci.yml file)
# Build JAVA applications using Apache Maven (http://maven.apache.org)
# For docker image tags see https://hub.docker.com/_/maven/
#
# For general lifecycle information see https://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html
#
# This template will build and test your projects as well as create the documentation.
#
# * Caches downloaded dependencies and plugins between invocation.
# * Verify but don't deploy merge requests.
# * Deploy built artifacts from master branch only.
# * Shows how to use multiple jobs in test stage for verifying functionality
# with multiple JDKs.
# * Uses site:stage to collect the documentation for multi-module projects.
# * Publishes the documentation for `master` branch.
variables:
# This will suppress any download for dependencies and plugins or upload messages which would clutter the console log.
# `showDateTime` will show the passed time in milliseconds. You need to specify `--batch-mode` to make this work.
MAVEN_OPTS: "-Dhttps.protocols=TLSv1.2 -Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=WARN -Dorg.slf4j.simpleLogger.showDateTime=true -Djava.awt.headless=true"
# As of Maven 3.3.0 instead of this you may define these options in `.mvn/maven.config` so the same config is used
# when running from the command line.
# `installAtEnd` and `deployAtEnd` are only effective with recent version of the corresponding plugins.
MAVEN_CLI_OPTS: "--batch-mode --errors --fail-at-end --show-version -DinstallAtEnd=true -DdeployAtEnd=true"
# Cache downloaded dependencies and plugins between builds.
# To keep cache across branches add 'key: "$CI_JOB_NAME"'
cache:
paths:
- .m2/repository
# This will only validate and compile stuff and run e.g. maven-enforcer-plugin.
# Because some enforcer rules might check dependency convergence and class duplications
# we use `test-compile` here instead of `validate`, so the correct classpath is picked up.
.validate: &validate
stage: build
script:
- 'mvn $MAVEN_CLI_OPTS test-compile'
# For merge requests do not `deploy` but only run `verify`.
# See https://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html
.verify: &verify
stage: test
script:
- 'mvn $MAVEN_CLI_OPTS verify site site:stage'
except:
- master
# Validate merge requests using JDK7
validate:jdk7:
<<: *validate
image: maven:3.3.9-jdk-7
# Validate merge requests using JDK8
validate:jdk8:
<<: *validate
image: maven:3.3.9-jdk-8
# Verify merge requests using JDK7
verify:jdk7:
<<: *verify
image: maven:3.3.9-jdk-7
# Verify merge requests using JDK8
verify:jdk8:
<<: *verify
image: maven:3.3.9-jdk-8
# For `master` branch run `mvn deploy` automatically.
# Here you need to decide whether you want to use JDK7 or 8.
# To get this working you need to define a volume while configuring your gitlab-ci-multi-runner.
# Mount your `settings.xml` as `/root/.m2/settings.xml` which holds your secrets.
# See https://maven.apache.org/settings.html
deploy:jdk8:
# Use stage test here, so the pages job may later pickup the created site.
stage: test
script:
- 'mvn $MAVEN_CLI_OPTS deploy site site:stage'
only:
- master
# Archive up the built documentation site.
artifacts:
paths:
- target/staging
image: maven:3.3.9-jdk-8
pages:
image: busybox:latest
stage: deploy
script:
# Because Maven appends the artifactId automatically to the staging path if you did define a parent pom,
# you might need to use `mv target/staging/YOUR_ARTIFACT_ID public` instead.
- mv target/staging public
dependencies:
- deploy:jdk8
artifacts:
paths:
- public
only:
- master
I'm using a local GitLab runner which is configured to build and test in Docker. In registering the runner I chose the 'shell' option
From the sounds of it, you have registered the gitlab-runner incorrectly for the mode you are after. You said you want to use a runner to build and test in Docker, however you have registered the runner in shell mode.
To use the runner in Docker, follow the instructions set in the documentation. Make sure to set the runner executor to Docker when registering.
You'd also ideally remove the shell runner you have created.
If you have issues registering the runner, see this answer, which may also help with setting up your environment.