Azure Spring Gradle: files in src/test/resouces not accessible - spring

Im building Spring boot app using gradle. Integration tests (Spock) need to access code/src/resouces/docker-compose.yml file to prepare TestContainers container:
static DockerComposeContainer postgresContainer = new DockerComposeContainer(
ResourceUtils.getFile("classpath:docker-compose.yml"))
The git file structure is:
- code
- src
- main
- test
- resources
- docker-compose.yml
This is working fine on my local machine, but once I run it in Azure pipeline, it gives
No such file or directory: '/__w/1/s/code/build/resources/test/docker-compose.yml'
My pipeline yaml is like bellow. I use Ubuntu container with Java17 as I need to build with 17 but Azure's latest is 11 (maybe this plays any role in the error I get?)
trigger: none
stages:
- stage: Test
displayName: Test Stage
jobs:
- job: Test
pool:
vmImage: 'ubuntu-22.04'
container: gradle:7.6.0-jdk17
variables:
- name: JAVA_HOME_11_X64
value: /opt/java/openjdk
displayName: Test
steps:
- script: java --version
displayName: Java version
- script: |
echo "Build reason is: $(Build.Reason)"
displayName: Build reason
- checkout: self
clean: true
- task: Gradle#2
displayName: 'Gradle Build'
enabled: True
inputs:
javaHomeSelection: 'path'
jdkDirectory: '/opt/java/openjdk'
wrapperScript: code/gradlew
cwd: code
tasks: clean build
publishJUnitResults: true
jdkVersionOption: 1.17
Thanks for help!

I've solved it by "workarround" - I realied that I dont need to use the container with jdk17 that was causing the problem (could not access files on host machine of course)
The true is the Azure silently supports Jkd17 by directive: jdkVersionOption: 1.17
But once someone will need to use the container to build the code and access the repository files that are not on classpath, the problem will raise again
trigger: none
stages:
- stage: Test
displayName: Test Stage
jobs:
- job: Test
pool:
vmImage: 'ubuntu-22.04'
displayName: Test
steps:
- script: java --version
displayName: Java version
- script: |
echo "Build reason is: $(Build.Reason)"
displayName: Build reason
- checkout: self
clean: true
- task: Gradle#2
displayName: 'Gradle Build'
enabled: True
inputs:
wrapperScript: server/code/gradlew
cwd: server/code
tasks: test
publishJUnitResults: true
jdkVersionOption: 1.17
Please follow Azure pipeline issue for more details

Related

Can't checkout the same repo multiple times in a pipeline

I have self-hosted agents on multiple environments that I am trying to run the same build/deploy processes on. I would like to be able to deploy the same code from a single repo to multiple systems concurrently. Thus, I have created an "overhead" pipeline, and several "processes" pipeline templates. Everything seems to be going very well, except for when I try to perform checkouts of the same repo twice in the same pipeline execution. I get the following error:
An error occurred while loading the YAML build pipeline. An item with the same key has already been added.
I would really like to be able to just click ONE button to trigger a main pipeline that calls all the templates requires and gives the parameters needed to get all my jobs done at once. I could of course define this "overhead" pipeline and then queue up as many instances as I need of it per systems that I need to deploy to, but I'm lazy, hence why I'm using pipelines!
As soon as I remove the checkout from Common.yml, the validation succeeds without any issues. If I keep the checkout in there but only call the Common.yml once for the entire Overhead pipeline, then it succeeds without any issues as well. But the problem is: I need to pull the contents of the repo to EACH of my agents that are running on completely separate environments that are in no way ever able to talk to each other (can't pull the information to one agent and have it do some sort of a "copy" to all the other agent locations.....).
Any assistance is very much welcomed, thank you!
The following is my "overhead" pipeline:
# azure-pipelines.yml
trigger:
none
parameters:
- name: vLAN
type: string
default: 851
values:
- 851
- 1105
stages:
- stage: vLAN851
condition: eq('${{ parameters.vLAN }}', '851')
pool:
name: xxxxx
demands:
- vLAN -equals 851
jobs:
- job: Common_851
steps:
- template: Procedures/Common.yml
- job: Export_851
dependsOn: Common_851
steps:
- template: Procedures/Export.yml
parameters:
Server: ABTS-01
- stage: vLAN1105
condition: eq('${{ parameters.vLAN }}', '1105')
pool:
name: xxxxx
demands:
- vLAN -equals 1105
jobs:
- job: Common_1105
steps:
- template: Procedures/Common.yml
- job: Export_1105
dependsOn: Common_1105
steps:
- template: Procedures/Export.yml
parameters:
Server: OTS-01
And here is the "Procedures/Common.yml":
steps:
- checkout: git://xxxxx/yyyyy#$(Build.SourceBranchName)
clean: true
enabled: true
timeoutInMinutes: 1
- task: UsePythonVersion#0
enabled: true
timeoutInMinutes: 1
displayName: Select correct version of Python
inputs:
versionSpec: '3.8'
addToPath: true
architecture: 'x64'
- task: CmdLine#2
enabled: true
timeoutInMinutes: 5
displayName: Ensure Python Requirements Installed
inputs:
script: |
python -m pip install GitPython
And here is the "Procedures/Export.yml":
parameters:
- name: Server
type: string
steps:
- task: PythonScript#0
enabled: true
timeoutInMinutes: 3
displayName: xxxxx
inputs:
arguments: --name "xxxxx" --mode True --Server ${{ parameters.Server }}
scriptSource: 'filePath'
scriptPath: 'xxxxx/main.py'
I managed to make checkout work with variable branch names by using template expression variables ${{ ... }} instead of macro syntax $(...) variables.
The difference is that, template expressions are processed at compile time while macros are processed at runtime.
So in my case I have something like:
- checkout: git://xxx/yyy#${{ variables.BRANCH_NAME }}
For more information about variables syntax :
Understand variable syntax
I couldn't get it to work with expressions but I was able to get it to work using repository resources following the documentation at: https://learn.microsoft.com/en-us/azure/devops/pipelines/repos/multi-repo-checkout?view=azure-devops
resources:
repositories:
- repository: MyGitHubRepo # The name used to reference this repository in the checkout step
type: git
name: MyAzureProjectName/MyGitRepo
ref: $(Build.SourceBranch)
trigger:
- main
pool:
vmImage: 'ubuntu-latest'
#some job
steps:
- checkout: MyGitHubRepo
#some other job
steps:
- checkout: MyGitHubRepo
- script: dir $(Build.SourcesDirectory)

Running Java GUI tests on GitHub using xvfb

I've got some basic automated GUI tests for my Java desktop application that work when running on my Windows desktop. On GitHub they fail with
java.awt.AWTError at X11GraphicsEnvironment.java:-2
I started from the default Java CI with Gradle workflow, and have added some steps attempting to set up xvfb so it has a display to work with, but it's still failing. Is there a way I can modify the workflow so that X11 is available for the tests?
name: Java CI with Gradle
on:
push:
branches: [ main ]
pull_request:
branches: [ main ]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: Set up JDK 11
uses: actions/setup-java#v2
with:
java-version: '11'
distribution: 'adopt'
- name: Set up virtual X11
run: sudo apt-get install xvfb
- name: Start virtual frame buffer
run: Xvfb :19 -screen 0 1024x768x16 &
- name: set display
run: export DISPLAY=:19
- name: Grant execute permission for gradlew
run: chmod +x gradlew
- name: Build with Gradle
run: ./gradlew build
I ended up splitting the GUI tests off using JUnit #Tags, and then adding separate gradle tasks:
task nonGuiTest(type: Test) {
useJUnitPlatform {
excludeTags 'gui'
}
}
task guiTest(type: Test) {
useJUnitPlatform {
includeTags 'gui'
}
}
then in the workflow
I run build -x test initially
then run the non-GUI tests
then use the GabrielBB/xvfb-action#v1 action to set up xvfb
then run the GUI tests
name: Java CI with Gradle
on:
push:
branches: [ main ]
pull_request:
branches: [ main ]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: Set up JDK 11
uses: actions/setup-java#v2
with:
java-version: '11'
distribution: 'adopt'
- name: Grant execute permission for gradlew
run: chmod +x gradlew
- name: Build with Gradle
run: ./gradlew build -x test
- name: headless tests
run: ./gradlew nonGuiTest
- name: GUI tests
uses: GabrielBB/xvfb-action#v1
with:
run: ./gradlew guiTest

Migrating Azure Pipelines Deployments to GitHub Actions

I have an Azure Pipeline build which builds NuGet packages on Windows, MacOS and Linux and a deployment job that then takes the NuGet packages built from the Windows image and publishes them to Azure Artefacts, GitHub packages and NuGet.
With GitHub actions, I have managed to build NuGet packages on Windows, MacOS and Linux but I don't know how to then create a deployment job as that feature doesn't exist. I think I need to create a new deployment job that is triggered from a successful build job, pick up the NuGet packages from the successful build and then push them. However, I don't see a trigger type that can do that.
GitHub Actions Build YAML
name: Build
on:
push:
branches:
- '*'
tags:
- '*'
env:
DOTNET_SKIP_FIRST_TIME_EXPERIENCE: true
DOTNET_CLI_TELEMETRY_OPTOUT: true
MINVERBUILDMETADATA: build.$(Build.BuildId)
jobs:
build:
name: Build-${{matrix.os}}
runs-on: ${{matrix.os}}
strategy:
matrix:
os: [ubuntu-latest, windows-latest, macOS-latest]
steps:
- name: Checkout
uses: actions/checkout#v2
with:
lfs: true
fetch-depth: 0
- name: 'Git Fetch Tags'
run: git fetch --tags
shell: pwsh
- name: 'Install .NET Core SDK'
uses: actions/setup-dotnet#v1
with:
dotnet-version: 3.1.301
- name: 'Dotnet Tool Restore'
run: dotnet tool restore
shell: pwsh
- name: 'Dotnet Cake Build'
run: dotnet cake --target=Build
shell: pwsh
- name: 'Dotnet Cake Test'
run: dotnet cake --target=Test
shell: pwsh
- name: 'Dotnet Cake Pack'
run: dotnet cake --target=Pack
shell: pwsh
- name: 'Publish Artefacts'
uses: actions/upload-artifact#v1.0.0
with:
name: ${{matrix.os}}
path: './Artefacts'
Azure Pipeline Build & Deployment YAML
trigger:
branches:
include:
- '*'
tags:
include:
- '*'
variables:
DOTNET_SKIP_FIRST_TIME_EXPERIENCE: true
DOTNET_CLI_TELEMETRY_OPTOUT: true
MINVERBUILDMETADATA: build.$(Build.BuildId)
stages:
- stage: Build
jobs:
- job: Build
strategy:
matrix:
Linux:
matrixName: Ubuntu
vmImageName: ubuntu-latest
Mac:
matrixName: Mac
vmImageName: macos-latest
Windows:
matrixName: Windows
vmImageName: windows-latest
pool:
vmImage: $(vmImageName)
timeoutInMinutes: 10
steps:
- checkout: self
lfs: true
- task: UseDotNet#2
displayName: 'Install .NET Core SDK'
inputs:
packageType: 'sdk'
useGlobalJson: true
- pwsh: 'dotnet tool restore'
displayName: 'Dotnet Tool Restore'
failOnStderr: true
- pwsh: 'dotnet cake --target=Build'
displayName: 'Dotnet Cake Build'
failOnStderr: true
- pwsh: 'dotnet cake --target=Test'
displayName: 'Dotnet Cake Test'
failOnStderr: true
- pwsh: 'dotnet cake --target=Pack'
displayName: 'Dotnet Cake Pack'
failOnStderr: true
- task: PublishTestResults#2
displayName: 'Publish Test Results'
inputs:
testResultsFormat: 'VSTest'
testResultsFiles: '**/*.trx'
- task: PublishCodeCoverageResults#1
inputs:
codeCoverageTool: cobertura
summaryFileLocation: '**/*.cobertura.xml'
- publish: './Artefacts'
artifact: $(matrixName)
displayName: 'Publish Artefacts'
- stage: Deploy
jobs:
- deployment: AzureArtefacts
displayName: 'Azure Artefacts'
condition: ne(variables['Build.Reason'], 'PullRequest')
pool:
vmImage: windows-latest
environment: 'Azure Artefacts'
strategy:
runOnce:
deploy:
steps:
- task: NuGetToolInstaller#1
displayName: 'NuGet Install'
- task: NuGetAuthenticate#0
displayName: 'NuGet Authenticate'
- pwsh: nuget push $(Agent.BuildDirectory)\Windows\*.nupkg -Source $(AzureArtefactsSource) -ApiKey AzureArtifacts -SkipDuplicate
displayName: 'NuGet Push'
failOnStderr: true
- deployment: GitHub
condition: ne(variables['Build.Reason'], 'PullRequest')
pool:
vmImage: windows-latest
environment: 'GitHub'
strategy:
runOnce:
deploy:
steps:
- task: NuGetToolInstaller#1
displayName: 'NuGet Install'
- pwsh: nuget source Add -Name GitHub -Source https://nuget.pkg.github.com/GITHUB-USERNAME -UserName GITHUB-USERNAME -Password $(GitHubPersonalAccessToken)
displayName: 'NuGet Add Source'
failOnStderr: true
- pwsh: nuget push $(Agent.BuildDirectory)\Windows\*.nupkg -Source GitHub -SkipDuplicate
displayName: 'NuGet Push'
failOnStderr: true
- deployment: NuGet
condition: and(ne(variables['Build.Reason'], 'PullRequest'), startsWith(variables['Build.sourceBranch'], 'refs/tags/'))
pool:
vmImage: windows-latest
environment: 'NuGet'
strategy:
runOnce:
deploy:
steps:
- task: NuGetToolInstaller#1
displayName: 'Install NuGet'
- pwsh: |
Get-ChildItem $(Agent.BuildDirectory)\Windows -Filter *.nupkg |
Where-Object { !$_.Name.Contains('preview') } |
ForEach-Object { nuget push $_ -Source https://api.nuget.org/v3/index.json -ApiKey $(NuGetApiKey) -SkipDuplicate }
displayName: 'NuGet Push'
failOnStderr: true
Migrating Azure Pipelines Deployments to GitHub Actions
Indeed, there is no such multi-stage feature for Github action at this moment.
Just as you suspect, we could create a deployment job to deploy the artifact. We could try to create a new job which needs the existing build job, in the new job, download the artifacts and push them to azure artifact, github packages, nuget:
jobs:
job_1:
name: Build
job_2:
name: Deploy
needs: job_1
runs-on: windows-latest
steps:
- name: Download math result for job 1
uses: actions/download-artifact#v1
with:
name: xxx
You could check the Github action jobs.<job_id>.needs and the sample for some more details.
Hope this helps.

Artifact dependencies (destination) using Bamboo YAML specs

I'm trying to set up the Bamboo build plan configuration using Bamboo YAML specs (.yml file below). In the last stage (Create deployment artifacts) I want to use the shared artifacts from the previous stage. By specifying the artifacts of the jobs as "shared: true" I can use them in the second stage. However, they are in the same destination folder. Using the UI this can be easily edited.
Artifact dependencies
But how can I specify the destination folder of the two artifacts in the Bamboo YAML specs, e.g. from "Root of working directory" to "./app" and "./wwwroot", respectively?
---
version: 2
plan:
project-key: COCKPIT
key: BE
name: Cockpit - Continuous Build - Windows
stages:
- Build Stage:
- Build Backend
- Build Frontend
- Build Artifact:
- Create Deployment Artifact
Build Backend:
requirements:
- Visual Studio Build Tools (32-bit)
tasks:
- checkout:
repository: cockpit_backend
path: 'cockpit_backend'
force-clean-build: false
- script:
- dotnet publish .\cockpit_backend\src\Cockpit.WebApi\ --configuration Release
artifacts:
-
name: BackendBuild
location: cockpit_backend/src/Cockpit.WebApi/bin/Release/netcoreapp3.1/publish
pattern: '**/*.*'
required: true
shared: true
Build Frontend:
requirements:
- os_linux
tasks:
- checkout:
repository: 'Cockpit / cockpit_frontend'
path: 'cockpit_frontend'
force-clean-build: false
- script:
- cd cockpit_frontend
- npm install
- script:
- cd cockpit_frontend
- npm run build-prod
docker:
image: node:alpine
artifacts:
-
name: FrontendBuild
location: cockpit_frontend/dist
pattern: '**/*.*'
required: true
shared: true
Create Deployment Artifact:
requirements:
- os_windows
tasks:
- script:
interpreter: powershell
scripts:
- $buildDir = "Cockpit"
- $dest = "Cockpit_${bamboo.buildNumber}.zip"
- Add-Type -assembly "system.io.compression.filesystem"
- '[io.compression.zipfile]::CreateFromDirectory($buildDir, $dest)'
artifacts:
-
name: Completebuild
pattern: 'Cockpit_${bamboo.buildNumber}.zip'
required: true
YAML specs doesn't support artifact dependency management and you need to have script task at "Create Deployment Artifact" job to put them into separate folders from root before compressing

How do you run xcuitests in parallel using azure dev Ops?

I'd like to execute my xcuitest suite on azure dev ops in parallel using xcode's out of the box capabilities. Locally, it's as simple as checking the box to enable parallel testing on my test target.
Parallel executions toggle
Locally, xcode opens multiple simulators and the tests run as expected. On azure, they run and pass as expected but they take the same amount of time as they usually do, indicating to me that they are not running in parallel. What am I missing here? Are there extra steps I need to take to get them running in parallel via azure dev ops?
Azure-Pipleline.yml snippet
- stage: uiTests
displayName: UI Tests
dependsOn: []
condition: | # don't run for release builds to save Azure bandwith
not(
or(
contains(variables['Build.SourceBranch'], 'master'),
contains(variables['Build.SourceBranch'], 'release')
)
)
jobs:
- template: azure-templates/job-tests.yml
parameters:
configuration: "$(UI_TEST_CONFIGURATION)"
sdk: "$(UI_TEST_SDK)"
scheme: "$(UI_TEST_SCHEME)"
artifactName: "UI Tests Results - Attempt #$(System.StageAttempt).$(System.JobAttempt)" # include attempt number in suffix because reruns can not overwrite artifacts"
shouldRunWiremock: true
azure-templates/job-test.yml snippet
- job: Test
pool: Hosted macOS
variables:
- template: variables.yml
- group: blah-Keys
steps:
- template: steps-install-code-signing.yml
parameters:
certFile: "$(UAT_SIGNING_FILE_ID)"
signingIdentity: "$(UAT_SIGNING_IDENTITY)"
provisioningFile: "$(UAT_PROVISIONING_FILE_ID)"
- script: "./setBuildVersion.sh"
displayName: "Update Build Number"
- script: "./xcconfig.sh"
displayName: "Populate xcconfig files"
- bash: "./runWiremock.sh &"
continueOnError: true
displayName: "Start Mock Server"
enabled: ${{ parameters.shouldRunWiremock }}
- task: Xcode#5
displayName: "Run Tests"
inputs:
actions: test
configuration: "${{ parameters.configuration }}"
sdk: "${{ parameters.sdk }}"
xcWorkspacePath: "$(WORKSPACE_PATH)"
scheme: "${{ parameters.scheme }}"
xcodeVersion: specifyPath
xcodeDeveloperDir: "$(XCODE_PATH)"
signingOption: default
args: "-resultBundlePath $(build.SourcesDirectory)/testResults"
destinationPlatformOption: iOS
destinationSimulators: "$(TEST_SIMULATOR)"
publishJUnitResults: true
continueOnError: false # report test failures as errors in Azure DevOps
- publish: testResults
artifact: "${{ parameters.artifactName }}.xcresult"
condition: not(canceled()) # run if there are test failures
You can consider to use build matrix which is one feature of YAML to achieve the test ran with multiple simulators:
jobs:
- job: Build
strategy:
matrix:
simulators1:
destinationSimulators: 'iPhone X'
simulators2:
destinationSimulators: 'iPhone 7'
..
..
- task: Xcode#5
displayName: "Run Tests"
..
..
destinationSimulators: "$(destinationSimulators)"
..
..

Resources