Im building Spring boot app using gradle. Integration tests (Spock) need to access code/src/resouces/docker-compose.yml file to prepare TestContainers container:
static DockerComposeContainer postgresContainer = new DockerComposeContainer(
ResourceUtils.getFile("classpath:docker-compose.yml"))
The git file structure is:
- code
- src
- main
- test
- resources
- docker-compose.yml
This is working fine on my local machine, but once I run it in Azure pipeline, it gives
No such file or directory: '/__w/1/s/code/build/resources/test/docker-compose.yml'
My pipeline yaml is like bellow. I use Ubuntu container with Java17 as I need to build with 17 but Azure's latest is 11 (maybe this plays any role in the error I get?)
trigger: none
stages:
- stage: Test
displayName: Test Stage
jobs:
- job: Test
pool:
vmImage: 'ubuntu-22.04'
container: gradle:7.6.0-jdk17
variables:
- name: JAVA_HOME_11_X64
value: /opt/java/openjdk
displayName: Test
steps:
- script: java --version
displayName: Java version
- script: |
echo "Build reason is: $(Build.Reason)"
displayName: Build reason
- checkout: self
clean: true
- task: Gradle#2
displayName: 'Gradle Build'
enabled: True
inputs:
javaHomeSelection: 'path'
jdkDirectory: '/opt/java/openjdk'
wrapperScript: code/gradlew
cwd: code
tasks: clean build
publishJUnitResults: true
jdkVersionOption: 1.17
Thanks for help!
I've solved it by "workarround" - I realied that I dont need to use the container with jdk17 that was causing the problem (could not access files on host machine of course)
The true is the Azure silently supports Jkd17 by directive: jdkVersionOption: 1.17
But once someone will need to use the container to build the code and access the repository files that are not on classpath, the problem will raise again
trigger: none
stages:
- stage: Test
displayName: Test Stage
jobs:
- job: Test
pool:
vmImage: 'ubuntu-22.04'
displayName: Test
steps:
- script: java --version
displayName: Java version
- script: |
echo "Build reason is: $(Build.Reason)"
displayName: Build reason
- checkout: self
clean: true
- task: Gradle#2
displayName: 'Gradle Build'
enabled: True
inputs:
wrapperScript: server/code/gradlew
cwd: server/code
tasks: test
publishJUnitResults: true
jdkVersionOption: 1.17
Please follow Azure pipeline issue for more details
....
jobs:
setup:
name: Setup
runs-on: [self-hosted, Linux, X64]
steps:
- name: Set up Go 1.17
uses: actions/setup-go#v2
with:
go-version: 1.17
id: Go
- name: work around permission issue
run: git config --global --add safe.directory /__w/****-***/****-***
- name: Checkout code into go module directory
uses: actions/checkout#v2
- name: Make Directory
run: mkdir build
- name: Build
run: |
CGO_ENABLED=0 go build -ldflags "-linkmode external" -o main
- name: Upload Build
uses: actions/upload-artifact#v2
with:
name: binary
path: main
scans :
needs: setup
name: all scans
runs-on: [self-hosted, Linux, X64]
steps:
- name: Download build for build and test
uses: actions/download-artifact#v2
with:
name: build
- name: Coverity Scan
env:
AUTH_DATA: ${{ secrets.COVERITY_KEY_FILE }}
run: |
export PATH=$PATH:/opt/coverity/coverity-base/bin
mkdir coverity
cov-build --dir coverity go build main.go
cov-analyze --dir coverity --strip-path=`pwd`
touch auth_key_file
chmod 600 auth_key_file
echo $( printf "%s" "$AUTH_DATA" ) > auth_key_file
cov-commit-defects --dir coverity --url `suspicious_url` --auth-key-file auth_key_file --stream ****-overrides
- name: Upload Coverity
uses: actions/upload-artifact#v2
with:
name: coverity
path: coverity
I have this workflow file for my Golang project. I want to run the Coverity scan. But when I am running this it's throwing an error :
I have set up go in setup step. I don't know why it is telling that it did not found the go command. Please help me with this.
Go may not be in the path of the process executing that command. You may need to modify the $PATH variable to include the path to your go binary.
Looks like you're only updating the $PATH in scans, but this ought to be in setup I think:
export PATH=$PATH:/opt/coverity/coverity-base/bin
I'm working with github actions and in my tests I need to make myt build fail when my code coverage percentage is less than 80%. I looked up some github actions in github marketplace but dont find anything. Can I do it ? I'm linking my workflow file if it migth help
---
name: lint build and test
on:
push:
branches: [master]
pull_request:
branches: [master]
jobs:
build-and-test:
runs-on: ubuntu-latest
steps:
- name: Checkout Code
uses: actions/checkout#v2
- name: Set up Go
uses: actions/setup-go#v2
with:
go-version: 1.15
env:
GITHUB_TOKEN: ${{secrets.GITHUB_TOKEN}}
- name: Super-Linter
uses: github/super-linter#v3.14.0
env:
VALIDATE_GO: false
VALIDATE_JSCPD: false
VALIDATE_ALL_CODEBASE: true
DEFAULT_BRANCH: master
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: golangci-lint
uses: golangci/golangci-lint-action#v2
with:
version: v1.29
- name: Build
run: go build -o apiDateTime -v ./...
- name: Test
run: go test ./... -coverprofile cover.out -covermode atomic
- name: Coverage result
run: go tool cover -func cover.out
I would replace invocation of go test with an invocation of a shell script as explained here.
The shell script would look something like this
!#/bin/sh
set -e -u
go test ./... -coverprofile cover.out -covermode atomic
perc=`go tool cover -func=cover.out | tail -n 1 | sed -Ee 's!^[^[:digit:]]+([[:digit:]]+(\.[[:digit:]]+)?)%$!\1!'`
res=`echo "$perc >= 80.0" | bc`
test "$res" -eq 1 && exit 0
echo "Insufficient coverage: $perc" >&2
exit 1
Where:
The encantation involving sed extracts the coverage percentage (see here).
The next line asks the calculator to compare the percentage with your configured threshold.
Then the next line makes the script exit successfully if the test passed.
The rest of the script blows up if the coverage requirements were not met.
This script expects the bc tool is installed in that ubuntu-latest package (which I don't know).
If it isn't, the whole thing can be scripted in any language available in the image—such as Perl or Python.
Try to use GitHub Actions but getting error when trying to run executable build. I can find executable when doing ls and also changing mod of file executable but didn't got anything. Attached log/output of GitHub action.
Here is my github action workflow file
on:
push:
branches:
- master
name: Build For platforms
jobs:
build:
# We want to define a strategy for our job
strategy:
# this will contain a matrix of all of the combinations
# we wish to test again:
matrix:
go-version: [1.14.x]
platform: [ubuntu-latest]
# Defines the platform for each test run
runs-on: ${{ matrix.platform }}
# the steps that will be run through for each version and platform
# combination
steps:
# sets up go based on the version
- name: Install Go
uses: actions/setup-go#v2
with:
go-version: ${{ matrix.go-version }}
# checks out our code locally so we can work with the files
- name: Checkout code
uses: actions/checkout#v2
- name: Download modules
run: go mod download
- name: Build
run: go build -o executable main.go
- name: Chmod
run: chmod +x executable
- name: List Files
run: ls && pwd
- name: Run
run: ./executable
We would like to optimize our pipelines to avoid running steps that are not needed for the particular case. I have created the variable shouldTriggerAnyBuild but it always seems to be true (or ignored), as the specified steps always runs, even though the none of the steps later, where the conditions are combined from, are run.
What is the problem with the script, or how can I debug it?
trigger:
- master
- stage
- release/*
pool:
vmImage: 'macOS-latest'
variables:
shouldTriggerAnyBuild: $[ or(and(eq(variables['Build.SourceBranch'], 'refs/heads/master'), not(startsWith(variables['Build.SourceVersionMessage'], 'release:'))), eq(variables['Build.SourceBranch'], 'refs/heads/stage'), startsWith(variables['Build.SourceBranch'], 'refs/heads/release/')) ]
steps:
- task: UseRubyVersion#0
inputs:
versionSpec: '~> 2.6'
- script: echo "y" | $ANDROID_HOME/tools/bin/sdkmanager --install 'system-images;android-29;google_apis;x86'
displayName: 'install/setup android sdkmanager'
condition: variables.shouldTriggerAnyBuild
- script: gem install bundler
displayName: 'gem install bundler'
condition: variables.shouldTriggerAnyBuild
- script: bundle install
displayName: 'bundle install'
condition: variables.shouldTriggerAnyBuild
- script: bundle exec fastlane ciBuildDev
displayName: 'build dev'
condition: and(eq(variables['Build.SourceBranch'], 'refs/heads/master'), not(startsWith(variables['Build.SourceVersionMessage'], 'release:')))
- script: bundle exec fastlane ciDeployToTest
displayName: 'build stage'
condition: eq(variables['Build.SourceBranch'], 'refs/heads/stage')
- script: bundle exec fastlane ciDeployToGooglePlay
displayName: 'build release'
condition: startsWith(variables['Build.SourceBranch'], 'refs/heads/release/')
- task: PublishBuildArtifacts#1
displayName: "Publish artifacts .apk"
condition: startsWith(variables['Build.SourceBranch'], 'refs/heads/release/')
inputs:
PathtoPublish: ./app/build/outputs/apk/prod/app-prod-unsigned.apk
ArtifactName: Prod_app_apk
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/expressions?view=azure-devops
What is the problem with the script?
Conditions are written as expressions. condition: variables.shouldTriggerAnyBuild won't take effect.
Instead you can use condition: eq(variables['SHOULDTRIGGERANYBUILD'], 'True') for conditional step. I think it's the direct cause of your issue. Also, feel free to use condition: eq(variables['shouldTriggerAnyBuild'], 'True') if you want, it also works.
How to debug it?
Here's a quick way to debug the value when necessary:
1.Change the second value to a impossible variable, then you can check the real value for your custom variable when the step is skipped:
2.If you want to make all thing clear, then you can do something like:
- task: CmdLine#2
inputs:
script: |
echo Hello world
condition: eq(and(eq(variables['Build.SourceBranch'], 'refs/heads/master'), not(startsWith(variables['Build.SourceVersionMessage'], 'release:'))),'True')
- task: CmdLine#2
inputs:
script: |
echo Hello world
condition: eq(eq(variables['Build.SourceBranch'], 'refs/heads/stage'),'True')
- task: CmdLine#2
inputs:
script: |
echo Hello world
condition: eq(startsWith(variables['Build.SourceBranch'], 'refs/heads/release/'),'True')
Since your variable is the combination of and(eq(variables['Build.SourceBranch'], 'refs/heads/master'), not(startsWith(variables['Build.SourceVersionMessage'], 'release:'))), eq(variables['Build.SourceBranch'], 'refs/heads/stage') and startsWith(variables['Build.SourceBranch'], 'refs/heads/release/') via Or function, you can divide them to debug how the shouldTriggerAnyBuild is expanded as always True.
In this way, you can debug them easily to find how the variable is expanded:
Note:
1.Or function: Evaluates True if any parameter is true.
2.Most of the time we choose to skip current task if one of previous tasks have failed, so you can consider combining succeeded() and your variable shouldTriggerAnyBuild. Examples here.
After a lot of trial and error we found that there was Build.SourceVersionMessage that meant we could not use it as a variable, so we ended up with a hack as shown below:
trigger:
- master
- stage
- release/*
pool:
vmImage: "macOS-latest"
variables:
# we can't use Build.SourceVersionMessage up here because it's not defined when the variables is set here
isMaster: ${{eq(variables['Build.SourceBranch'], 'refs/heads/master')}}
isStage: ${{eq(variables['Build.SourceBranch'], 'refs/heads/stage')}}
isRelease: ${{startsWith(variables['Build.SourceBranch'], 'refs/heads/release/')}}
releaseMessagePrefix: "release:"
jobs:
- job:
steps:
- task: UseRubyVersion#0
displayName: "set ruby version"
inputs:
versionSpec: "~> 2.6"
- task: Bash#3
displayName: "set commitMessage variable"
inputs:
targetType: inline
script: echo '##vso[task.setvariable variable=commitMessage]$(Build.SourceVersionMessage)'
- script: echo "y" | $ANDROID_HOME/tools/bin/sdkmanager --install 'system-images;android-29;google_apis;x86'
displayName: "install/setup android sdkmanager"
condition: or(and(variables.isMaster, not(startsWith(variables.commitMessage, variables.releaseMessagePrefix))), ${{variables.isStage}}, ${{variables.isRelease}})
- script: gem install bundler
displayName: "gem install bundler"
condition: or(and(variables.isMaster, not(startsWith(variables.commitMessage, variables.releaseMessagePrefix))), ${{variables.isStage}}, ${{variables.isRelease}})
- script: bundle install
displayName: "bundle install"
condition: or(and(variables.isMaster, not(startsWith(variables.commitMessage, variables.releaseMessagePrefix))), ${{variables.isStage}}, ${{variables.isRelease}})
- script: bundle exec fastlane ciBuildDev
displayName: "Build: dev"
condition: and(variables.isMaster, not(startsWith(variables.commitMessage, variables.releaseMessagePrefix)))
- script: bundle exec fastlane ciDeployToTest
displayName: "Build: stage"
condition: ${{variables.isStage}}
- script: bundle exec fastlane ciDeployToGooglePlay
displayName: "Build: release"
condition: ${{variables.isRelease}}
- task: PublishBuildArtifacts#1
displayName: "Publish: .apk"
condition: ${{variables.isRelease}}
inputs:
PathtoPublish: ./app/build/outputs/apk/prod/app-prod-unsigned.apk
ArtifactName: Prod_app_apk