I am new to YAML. I have environments and variables specified per environment. However, I have couple of items where I could use help.
I am trying to understand how do I specify variable groups and environments based on trigger or target branch? Here is what I am looking for.
How do I trigger deployment to two environments based on one trigger?
If trigger is "master" then deploy to "Prod" and "Dev" environments
If trigger is "feature/*" then deploy to "QA" environment
Follows the psudo code. I understand that "Build.TargetBranchName" is not a variable, but I am looking for something similar to achieve selective deployment.
trigger:
branches:
include:
- master
- feature/*
- bundle/*
variables:
- ${{if eq(variables['Build.TargetBranchName'], 'feature/123-mybranch')}}:
- group: dw-qa
- name: 'sql-server'
value: 'QA001.SEMPRA.COM'
- name: 'environment'
value: 'QA'
- ${{if eq(variables['Build.TargetBranchName'], 'master')}}:
- group: dw-dev
- name: 'sql-server'
value: 'DV001.SEMPRA.COM'
- name: 'environment'
value: 'DEV'
- ${{if eq(variables['Build.TargetBranchName'], 'master')}}:
- group: dw-prod
- name: 'sql-server'
value: 'PD001.SEMPRA.COM'
- name: 'environment'
value: 'PROD'
stages:
- stage: Build
jobs:
- job: Build_Solution
displayName: 'Build Solution'
pool:
name: 'Shared Server Pool'
steps:
- script: md BuildOutput
displayName: 'Create BuildOutput folder'
- task: VSBuild#1
displayName: 'Visual Studio Build'
inputs:
solution: 'DataWarehouse/DataWarehouse.sln'
platform: 'Any CPU'
configuration: 'Debug'
- task: PublishPipelineArtifact#1
displayName: 'Publish artifacts from BuildOutput folder'
inputs:
targetPath: 'BuildOutput'
artifact: 'DacpacAndConfig'
publishLocation: 'pipeline'
- stage: Deploy
jobs:
- deployment:
pool:
name: 'SEU WinTel VX Shared'
environment: '$(environment)'
strategy:
runOnce:
deploy:
steps:
- script: md DeploymentSource
displayName: 'Create DeploymentSource folder'
- task: DownloadPipelineArtifact#2
displayName: 'Download artifacts to DeploymentSource folder'
inputs:
buildType: 'current'
artifactName: 'DacpacAndConfig'
targetPath: 'DeploymentSource'
- task: SqlAzureDacpacDeployment#1
displayName: 'Deploy STAGE'
inputs:
azureSubscription: 'My-MSSQL-Subscription'
ServerName: '$(sql-server)'
SqlUsername: 'ado-dw'
SqlPassword: '$(sql-password)'
DatabaseName: 'STAGE'
PublishProfile: 'DeploymentSource\STAGE.publish.xml'
DacpacFile: 'DeploymentSource\STAGE.dacpac'
Thanks in advance.
Palak
Related
Im building Spring boot app using gradle. Integration tests (Spock) need to access code/src/resouces/docker-compose.yml file to prepare TestContainers container:
static DockerComposeContainer postgresContainer = new DockerComposeContainer(
ResourceUtils.getFile("classpath:docker-compose.yml"))
The git file structure is:
- code
- src
- main
- test
- resources
- docker-compose.yml
This is working fine on my local machine, but once I run it in Azure pipeline, it gives
No such file or directory: '/__w/1/s/code/build/resources/test/docker-compose.yml'
My pipeline yaml is like bellow. I use Ubuntu container with Java17 as I need to build with 17 but Azure's latest is 11 (maybe this plays any role in the error I get?)
trigger: none
stages:
- stage: Test
displayName: Test Stage
jobs:
- job: Test
pool:
vmImage: 'ubuntu-22.04'
container: gradle:7.6.0-jdk17
variables:
- name: JAVA_HOME_11_X64
value: /opt/java/openjdk
displayName: Test
steps:
- script: java --version
displayName: Java version
- script: |
echo "Build reason is: $(Build.Reason)"
displayName: Build reason
- checkout: self
clean: true
- task: Gradle#2
displayName: 'Gradle Build'
enabled: True
inputs:
javaHomeSelection: 'path'
jdkDirectory: '/opt/java/openjdk'
wrapperScript: code/gradlew
cwd: code
tasks: clean build
publishJUnitResults: true
jdkVersionOption: 1.17
Thanks for help!
I've solved it by "workarround" - I realied that I dont need to use the container with jdk17 that was causing the problem (could not access files on host machine of course)
The true is the Azure silently supports Jkd17 by directive: jdkVersionOption: 1.17
But once someone will need to use the container to build the code and access the repository files that are not on classpath, the problem will raise again
trigger: none
stages:
- stage: Test
displayName: Test Stage
jobs:
- job: Test
pool:
vmImage: 'ubuntu-22.04'
displayName: Test
steps:
- script: java --version
displayName: Java version
- script: |
echo "Build reason is: $(Build.Reason)"
displayName: Build reason
- checkout: self
clean: true
- task: Gradle#2
displayName: 'Gradle Build'
enabled: True
inputs:
wrapperScript: server/code/gradlew
cwd: server/code
tasks: test
publishJUnitResults: true
jdkVersionOption: 1.17
Please follow Azure pipeline issue for more details
I have a base template that accepts a stageList parameter. I don't do anything with the jobs in those stages:
parameters:
- name: stages
type: stageList
default: []
stages:
- ${{ parameters.stages }}
I'm passing into that a stage that contains a deployment job. I have hardcoded the environment for testing purposes, but even so it inserts the key "name: environment" under environment:
resources:
repositories:
- repository: templates
type: git
name: basePipelineTemplatesHost/basePipelineTemplatesHost
extends:
template: templateExtendedByDeployment/template.yml#templates
parameters:
stages:
- stage: buildStage1
jobs:
- deployment:
displayName: Deploy to demo environment
environment: DTL-Demo-Env
strategy:
runOnce:
deploy:
steps:
- script: echo test
Resulting in the following rendered yaml:
environment: {
name: DTL-Demo-Env
}
This causes the job to run on a hosted vm instead of my on-prem environment agent. Is this a bug?
Just a suggestion, you should add resourceType under environment.
jobs:
- deployment:
displayName: Deploy to demo environment
environment:
name: DTL-Demo-Env
resourceType: VirtualMachine
strategy:
runOnce:
deploy:
steps:
- script: echo test
If not, the new created environment will always be created under hosted agent when you use your private agent. You should add it to let the environment variable under your private agent.
these are the error that I can't fix image
trigger:
- master
schedules:
- cron: "0 17 * * * 6"
displayName: ogni giorno alle 17
branches:
include:
- master
always: false
pool:
vmImage: 'windows-2019'
variables:
buildConfiguration: 'Release'
parameters:
RestoreBuildProjects: '**/*.csproj'
steps:
- task: DotNetCoreCLI#2
displayName: 'Install .NET Core SDK'
inputs:
version: 5.0.x
performMultiLevelLookup: true
- task: DotNetCoreCLI#2
displayName: Restore
inputs:
command: 'restore'
projects: '$(Parameters.RestoreBuildProjects)'
vstsFeed:'f52d4c22-658b-47a0-a62f-134d5435cc85'
- task: DotNetCoreCLI#2
displayName: Build
inputs:
command: 'build'
projects: '**/*.csproj'
arguments: '--configuration $(buildConfiguration)'
- task: DotNetCoreCLI#2
displayName: Publish
inputs:
command: publish
publishWebProjects: True
arguments: '-c Release --output $(Build.ArtifactStagingDirectory)'
zipAfterPublish: True
- task: PublishBuildArtifacts#1
displayName: 'Publish Artifact'
inputs:
pathtoPublish: '$(Build.ArtifactStagingDirectory)'
artifactName: 'myWebsiteName'
I created this pipeline for an API ASP.NET Core based on the restful model but I can't make it work, as I try to run it it breaks, I've looked for the errors that Azuredevops gave me but I can't for any reason fix them. see attached email. Hope someone can help me. this is my first approach with YAML pipelines as I used to make them with the graphic tool integrated in Azure DevOPs.
I have helped to modify your yaml file and it shows no error on my test. Here is the yaml:
parameters:
- name: RestoreBuildProjects
type: string
default: '**/*.csproj'
trigger:
- master
schedules:
- cron: "0 17 * * * 6"
displayName: ogni giorno alle 17
branches:
include:
- master
always: false
pool:
vmImage: 'windows-2019'
variables:
buildConfiguration: 'Release'
steps:
- task: UseDotNet#2
inputs:
packageType: 'sdk'
version: '5.0.X'
performMultiLevelLookup: true
- task: DotNetCoreCLI#2
displayName: Restore
inputs:
command: 'restore'
projects: '$(Parameters.RestoreBuildProjects)'
feedsToUse: 'select'
# vstsFeed:'f52d4c22-658b-47a0-a62f-134d5435cc85'
- task: DotNetCoreCLI#2
displayName: Build
inputs:
command: 'build'
projects: '**/*.csproj'
arguments: '--configuration $(buildConfiguration)'
- task: DotNetCoreCLI#2
displayName: Publish
inputs:
command: publish
publishWebProjects: True
arguments: '-c Release --output $(Build.ArtifactStagingDirectory)'
zipAfterPublish: True
- task: PublishBuildArtifacts#1
displayName: 'Publish Artifact'
inputs:
pathtoPublish: '$(Build.ArtifactStagingDirectory)'
artifactName: 'myWebsiteName'
And please follow this step to use the yaml:
Create a new pipeline with the yaml edit.
Select the 'Starter Pipeline', this will help you create a new yaml file.
Copy the script to the pipeline.
Click the 'settings' on your restore task and select the feed you want to use. Then click the Add button. This step will help you input the full feed ID.
I have an Azure Pipeline build which builds NuGet packages on Windows, MacOS and Linux and a deployment job that then takes the NuGet packages built from the Windows image and publishes them to Azure Artefacts, GitHub packages and NuGet.
With GitHub actions, I have managed to build NuGet packages on Windows, MacOS and Linux but I don't know how to then create a deployment job as that feature doesn't exist. I think I need to create a new deployment job that is triggered from a successful build job, pick up the NuGet packages from the successful build and then push them. However, I don't see a trigger type that can do that.
GitHub Actions Build YAML
name: Build
on:
push:
branches:
- '*'
tags:
- '*'
env:
DOTNET_SKIP_FIRST_TIME_EXPERIENCE: true
DOTNET_CLI_TELEMETRY_OPTOUT: true
MINVERBUILDMETADATA: build.$(Build.BuildId)
jobs:
build:
name: Build-${{matrix.os}}
runs-on: ${{matrix.os}}
strategy:
matrix:
os: [ubuntu-latest, windows-latest, macOS-latest]
steps:
- name: Checkout
uses: actions/checkout#v2
with:
lfs: true
fetch-depth: 0
- name: 'Git Fetch Tags'
run: git fetch --tags
shell: pwsh
- name: 'Install .NET Core SDK'
uses: actions/setup-dotnet#v1
with:
dotnet-version: 3.1.301
- name: 'Dotnet Tool Restore'
run: dotnet tool restore
shell: pwsh
- name: 'Dotnet Cake Build'
run: dotnet cake --target=Build
shell: pwsh
- name: 'Dotnet Cake Test'
run: dotnet cake --target=Test
shell: pwsh
- name: 'Dotnet Cake Pack'
run: dotnet cake --target=Pack
shell: pwsh
- name: 'Publish Artefacts'
uses: actions/upload-artifact#v1.0.0
with:
name: ${{matrix.os}}
path: './Artefacts'
Azure Pipeline Build & Deployment YAML
trigger:
branches:
include:
- '*'
tags:
include:
- '*'
variables:
DOTNET_SKIP_FIRST_TIME_EXPERIENCE: true
DOTNET_CLI_TELEMETRY_OPTOUT: true
MINVERBUILDMETADATA: build.$(Build.BuildId)
stages:
- stage: Build
jobs:
- job: Build
strategy:
matrix:
Linux:
matrixName: Ubuntu
vmImageName: ubuntu-latest
Mac:
matrixName: Mac
vmImageName: macos-latest
Windows:
matrixName: Windows
vmImageName: windows-latest
pool:
vmImage: $(vmImageName)
timeoutInMinutes: 10
steps:
- checkout: self
lfs: true
- task: UseDotNet#2
displayName: 'Install .NET Core SDK'
inputs:
packageType: 'sdk'
useGlobalJson: true
- pwsh: 'dotnet tool restore'
displayName: 'Dotnet Tool Restore'
failOnStderr: true
- pwsh: 'dotnet cake --target=Build'
displayName: 'Dotnet Cake Build'
failOnStderr: true
- pwsh: 'dotnet cake --target=Test'
displayName: 'Dotnet Cake Test'
failOnStderr: true
- pwsh: 'dotnet cake --target=Pack'
displayName: 'Dotnet Cake Pack'
failOnStderr: true
- task: PublishTestResults#2
displayName: 'Publish Test Results'
inputs:
testResultsFormat: 'VSTest'
testResultsFiles: '**/*.trx'
- task: PublishCodeCoverageResults#1
inputs:
codeCoverageTool: cobertura
summaryFileLocation: '**/*.cobertura.xml'
- publish: './Artefacts'
artifact: $(matrixName)
displayName: 'Publish Artefacts'
- stage: Deploy
jobs:
- deployment: AzureArtefacts
displayName: 'Azure Artefacts'
condition: ne(variables['Build.Reason'], 'PullRequest')
pool:
vmImage: windows-latest
environment: 'Azure Artefacts'
strategy:
runOnce:
deploy:
steps:
- task: NuGetToolInstaller#1
displayName: 'NuGet Install'
- task: NuGetAuthenticate#0
displayName: 'NuGet Authenticate'
- pwsh: nuget push $(Agent.BuildDirectory)\Windows\*.nupkg -Source $(AzureArtefactsSource) -ApiKey AzureArtifacts -SkipDuplicate
displayName: 'NuGet Push'
failOnStderr: true
- deployment: GitHub
condition: ne(variables['Build.Reason'], 'PullRequest')
pool:
vmImage: windows-latest
environment: 'GitHub'
strategy:
runOnce:
deploy:
steps:
- task: NuGetToolInstaller#1
displayName: 'NuGet Install'
- pwsh: nuget source Add -Name GitHub -Source https://nuget.pkg.github.com/GITHUB-USERNAME -UserName GITHUB-USERNAME -Password $(GitHubPersonalAccessToken)
displayName: 'NuGet Add Source'
failOnStderr: true
- pwsh: nuget push $(Agent.BuildDirectory)\Windows\*.nupkg -Source GitHub -SkipDuplicate
displayName: 'NuGet Push'
failOnStderr: true
- deployment: NuGet
condition: and(ne(variables['Build.Reason'], 'PullRequest'), startsWith(variables['Build.sourceBranch'], 'refs/tags/'))
pool:
vmImage: windows-latest
environment: 'NuGet'
strategy:
runOnce:
deploy:
steps:
- task: NuGetToolInstaller#1
displayName: 'Install NuGet'
- pwsh: |
Get-ChildItem $(Agent.BuildDirectory)\Windows -Filter *.nupkg |
Where-Object { !$_.Name.Contains('preview') } |
ForEach-Object { nuget push $_ -Source https://api.nuget.org/v3/index.json -ApiKey $(NuGetApiKey) -SkipDuplicate }
displayName: 'NuGet Push'
failOnStderr: true
Migrating Azure Pipelines Deployments to GitHub Actions
Indeed, there is no such multi-stage feature for Github action at this moment.
Just as you suspect, we could create a deployment job to deploy the artifact. We could try to create a new job which needs the existing build job, in the new job, download the artifacts and push them to azure artifact, github packages, nuget:
jobs:
job_1:
name: Build
job_2:
name: Deploy
needs: job_1
runs-on: windows-latest
steps:
- name: Download math result for job 1
uses: actions/download-artifact#v1
with:
name: xxx
You could check the Github action jobs.<job_id>.needs and the sample for some more details.
Hope this helps.
I'm trying here to restore the nuget packages for my solution. For that, I have written below GitHub Action:
name: CI
on:
push:
pull_request:
branches:
- '*'
env:
solution: '**/*.sln'
buildPlatform: 'Any CPU'
buildConfiguration: 'Release'
jobs:
ci_build:
name: Build
runs-on: windows-latest
steps:
- name: NPM Authentication
uses: actions/checkout#v1
- name: Use Node.js
uses: actions/setup-node#v1
- name: Nuget Command
uses: actions/checkout#master
- uses: nuget/setup-nuget#v1
with:
nuget-api-key: ${{ secrets.NuGetAPIKey }}
- run: nuget restore MyProject.sln
- name: NuGet Tool Installer
run: NuGetToolInstaller#0
- name: NuGet Commad
run: NuGetCommand#2
env:
restoreSolution: '$(solution)'
selectOrConfig: 'config'
nugetConfigPath: 'Build/NuGet.config'
- name: VS Build
run: VSBuild#1
env:
solution: '$(solution)'
platform: '$(buildPlatform)'
configuration: '$(buildConfiguration)'
msbuildArgs: /p:AuthenticateWithRegistry=false
- name: VS Test
run: VSTest#2
env:
platform: '$(buildPlatform)'
configuration: '$(buildConfiguration)'
testSelector: 'testAssemblies'
testAssemblyVer2: '**\*test*.dll!**\*IntegrationTests.dll!**\*UiTests.dll!**\*TestAdapter.dll!**\obj\**'
- name: Copy Files to - $(build.artifactstagingdirectory)
run: CopyFiles#2
env:
content: |
**\bin\Project*.zip
**\bin\**\$(buildConfiguration)\*.msi
targetFolder: $(build.artifactstagingdirectory)
flattenFolders: true
And below is the error I'm getting:
Run nuget restore Advanced.OpenLink365.sln
nuget restore MyProject.sln
shell: C:\Program Files\PowerShell\6\pwsh.EXE -command ". '{0}'"
env:
solution: **/*.sln
buildPlatform: Any CPU
buildConfiguration: Release
NUGET: C:\hostedtoolcache\windows\nuget.exe\5.4.0\x64/nuget.exe
Input file does not exist: MyProject.sln.
##[error]Process completed with exit code 1.
Additionally, I have found that there is no file availble at "C:\Program Files\Powershell\6\pwsh.EXE" &C:\hostedtoolcache\windows\nuget.exe\5.4.0\x64/nuget.exe
Also, FYR - folder structure is as below:
Project Repo
.github
workflows
CI.yml
Projects
MyProject.sln
...other project files
Please let me know what I have missed here.
Thanx.
Change the command to be:
- run: nuget restore Projects/MyProject.sln