Can't checkout the same repo multiple times in a pipeline - yaml

I have self-hosted agents on multiple environments that I am trying to run the same build/deploy processes on. I would like to be able to deploy the same code from a single repo to multiple systems concurrently. Thus, I have created an "overhead" pipeline, and several "processes" pipeline templates. Everything seems to be going very well, except for when I try to perform checkouts of the same repo twice in the same pipeline execution. I get the following error:
An error occurred while loading the YAML build pipeline. An item with the same key has already been added.
I would really like to be able to just click ONE button to trigger a main pipeline that calls all the templates requires and gives the parameters needed to get all my jobs done at once. I could of course define this "overhead" pipeline and then queue up as many instances as I need of it per systems that I need to deploy to, but I'm lazy, hence why I'm using pipelines!
As soon as I remove the checkout from Common.yml, the validation succeeds without any issues. If I keep the checkout in there but only call the Common.yml once for the entire Overhead pipeline, then it succeeds without any issues as well. But the problem is: I need to pull the contents of the repo to EACH of my agents that are running on completely separate environments that are in no way ever able to talk to each other (can't pull the information to one agent and have it do some sort of a "copy" to all the other agent locations.....).
Any assistance is very much welcomed, thank you!
The following is my "overhead" pipeline:
# azure-pipelines.yml
trigger:
none
parameters:
- name: vLAN
type: string
default: 851
values:
- 851
- 1105
stages:
- stage: vLAN851
condition: eq('${{ parameters.vLAN }}', '851')
pool:
name: xxxxx
demands:
- vLAN -equals 851
jobs:
- job: Common_851
steps:
- template: Procedures/Common.yml
- job: Export_851
dependsOn: Common_851
steps:
- template: Procedures/Export.yml
parameters:
Server: ABTS-01
- stage: vLAN1105
condition: eq('${{ parameters.vLAN }}', '1105')
pool:
name: xxxxx
demands:
- vLAN -equals 1105
jobs:
- job: Common_1105
steps:
- template: Procedures/Common.yml
- job: Export_1105
dependsOn: Common_1105
steps:
- template: Procedures/Export.yml
parameters:
Server: OTS-01
And here is the "Procedures/Common.yml":
steps:
- checkout: git://xxxxx/yyyyy#$(Build.SourceBranchName)
clean: true
enabled: true
timeoutInMinutes: 1
- task: UsePythonVersion#0
enabled: true
timeoutInMinutes: 1
displayName: Select correct version of Python
inputs:
versionSpec: '3.8'
addToPath: true
architecture: 'x64'
- task: CmdLine#2
enabled: true
timeoutInMinutes: 5
displayName: Ensure Python Requirements Installed
inputs:
script: |
python -m pip install GitPython
And here is the "Procedures/Export.yml":
parameters:
- name: Server
type: string
steps:
- task: PythonScript#0
enabled: true
timeoutInMinutes: 3
displayName: xxxxx
inputs:
arguments: --name "xxxxx" --mode True --Server ${{ parameters.Server }}
scriptSource: 'filePath'
scriptPath: 'xxxxx/main.py'

I managed to make checkout work with variable branch names by using template expression variables ${{ ... }} instead of macro syntax $(...) variables.
The difference is that, template expressions are processed at compile time while macros are processed at runtime.
So in my case I have something like:
- checkout: git://xxx/yyy#${{ variables.BRANCH_NAME }}
For more information about variables syntax :
Understand variable syntax

I couldn't get it to work with expressions but I was able to get it to work using repository resources following the documentation at: https://learn.microsoft.com/en-us/azure/devops/pipelines/repos/multi-repo-checkout?view=azure-devops
resources:
repositories:
- repository: MyGitHubRepo # The name used to reference this repository in the checkout step
type: git
name: MyAzureProjectName/MyGitRepo
ref: $(Build.SourceBranch)
trigger:
- main
pool:
vmImage: 'ubuntu-latest'
#some job
steps:
- checkout: MyGitHubRepo
#some other job
steps:
- checkout: MyGitHubRepo
- script: dir $(Build.SourcesDirectory)

Related

Azure Spring Gradle: files in src/test/resouces not accessible

Im building Spring boot app using gradle. Integration tests (Spock) need to access code/src/resouces/docker-compose.yml file to prepare TestContainers container:
static DockerComposeContainer postgresContainer = new DockerComposeContainer(
ResourceUtils.getFile("classpath:docker-compose.yml"))
The git file structure is:
- code
- src
- main
- test
- resources
- docker-compose.yml
This is working fine on my local machine, but once I run it in Azure pipeline, it gives
No such file or directory: '/__w/1/s/code/build/resources/test/docker-compose.yml'
My pipeline yaml is like bellow. I use Ubuntu container with Java17 as I need to build with 17 but Azure's latest is 11 (maybe this plays any role in the error I get?)
trigger: none
stages:
- stage: Test
displayName: Test Stage
jobs:
- job: Test
pool:
vmImage: 'ubuntu-22.04'
container: gradle:7.6.0-jdk17
variables:
- name: JAVA_HOME_11_X64
value: /opt/java/openjdk
displayName: Test
steps:
- script: java --version
displayName: Java version
- script: |
echo "Build reason is: $(Build.Reason)"
displayName: Build reason
- checkout: self
clean: true
- task: Gradle#2
displayName: 'Gradle Build'
enabled: True
inputs:
javaHomeSelection: 'path'
jdkDirectory: '/opt/java/openjdk'
wrapperScript: code/gradlew
cwd: code
tasks: clean build
publishJUnitResults: true
jdkVersionOption: 1.17
Thanks for help!
I've solved it by "workarround" - I realied that I dont need to use the container with jdk17 that was causing the problem (could not access files on host machine of course)
The true is the Azure silently supports Jkd17 by directive: jdkVersionOption: 1.17
But once someone will need to use the container to build the code and access the repository files that are not on classpath, the problem will raise again
trigger: none
stages:
- stage: Test
displayName: Test Stage
jobs:
- job: Test
pool:
vmImage: 'ubuntu-22.04'
displayName: Test
steps:
- script: java --version
displayName: Java version
- script: |
echo "Build reason is: $(Build.Reason)"
displayName: Build reason
- checkout: self
clean: true
- task: Gradle#2
displayName: 'Gradle Build'
enabled: True
inputs:
wrapperScript: server/code/gradlew
cwd: server/code
tasks: test
publishJUnitResults: true
jdkVersionOption: 1.17
Please follow Azure pipeline issue for more details

Why am I not able to trigger my pipeline upon PR merge into master?

I have a pipeline which I want to trigger when PR is merged into master. I have tried different things, but this did not work. Furthermore, I am neither getting any error nor pipeline is triggering.
I do not want this to be triggered on PR creation. I want this to be triggered when PR is merged into master. That is the reason I have not added pr in my yml.
What am I missing here?
Approaches:
Enabled "Continuous Integration" option
Following trigger syntax per Microsoft recommendation
trigger:
batch: True
branches:
include:
- master
paths:
include:
- cosmos
Setting up valid YML file
Pipeline:
# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml
trigger:
batch: True
branches:
include:
- master
paths:
include:
- cosmos
stages:
- stage: SOME_PATH_dev
displayName: SOME_PATH_dev
jobs:
- deployment: 'DeployToDev'
environment: Dev
cancelTimeoutInMinutes: 1
pool:
vmImage: ubuntu-latest
strategy:
runOnce:
deploy:
steps:
- checkout: self
- task: AzureResourceGroupDeployment#2
displayName: Azure Deployment:Create Or Update Resource Group action on SOME_PATH_dev
inputs:
ConnectedServiceName: SOME_KEY
resourceGroupName: SOME_PATH_dev
location: West US
csmFile: cosmos/deploy.json
csmParametersFile: cosmos/parameters-dev.json
deploymentName: SOME_PATH.Cosmos.DEV
Repo Structure:
References:
"Configuring the trigger failed, edit and save the pipeline again" with no noticeable error and no further details
Azure Devops build pipeline: CI triggers not working on PR merge to master when PR set to none
https://medium.com/#aksharsri/add-approval-gates-in-azure-devops-yaml-based-pipelines-a06d5b16b7f4
https://erwinstaal.nl/posts/manual-approval-in-an-azure-devops-yaml-pipeline/
I was changing YML (pipeline) files instead of files inside cosmos folder and expecting trigger.

Difference between PUT and OUTPUT steps in Concourse

Could someone tell me the difference between the PUT step and the OUTPUT step in Concourse? For example, in the following type of YAML files why do we need a put step after a get? Can't we use output instead of put? If not what are the purposes of each two?
jobs:
- name: PR-Test
plan:
- get: some-git-pull-request
trigger: true
- put: some-git-pull-request
params:
context: tests
path: some-git-pull-request
status: pending
....
<- some more code to build ->
....
The purpose of a PUT step is to push to the given resource while an OUTPUT is the result of TASK step.
A task can configure outputs to produce artifacts that can then be propagated to either a put step or to another task step in the same plan.
This means that you send the resource that you are specifying on the GET step to the task as an input, to perform wherever build or scripts executions and the output of that task
is a modified resource that you can later pass to your put step or to another TASK if you don't want to use PUT.
It would also depend on the nature of the defined resource in your pipeline. I'm assuming that you have a git type resource like this:
resources:
- name: some-git-pull-request
type: git
source:
branch: ((credentials.git.branch))
uri: ((credentials.git.uri))
username: ((credentials.git.username))
password: ((credentials.git.pass))
If this is true, the GET step will pull that repo so you can use it as an input for your tasks and if you use PUT against that same resource as you are describing in your sample code, that will push changes to your repo.
Really it depends on the workflow that you want to write but to give an idea it would look something like this:
jobs:
- name: PR-Test
plan:
- get: some-git-pull-request
trigger: true
- task: test-code
config:
platform: linux
image_resource:
type: docker-image
source:
repository: yourRepo/yourImage
tag: latest
inputs:
- name: some-git-pull-request
run:
path: bash
args:
- -exc
- |
cd theNameOfYourRepo
npm install -g mocha
npm test
outputs:
- name: some-git-pull-request-output
Then you can use it on either a PUT
- put: myCloud
params:
manifest: some-git-pull-request-output/manifest.yml
path: some-git-pull-request-output
or a another task whitin the same plan
- task: build-code
config:
platform: linux
image_resource:
type: docker-image
source:
repository: yourRepo/yourImage
tag: latest
inputs:
- name: some-git-pull-request-output
run:
path: bash
args:
- -exc
- |
cd some-git-pull-request-output/
npm install
gulp build
outputs:
- name: your-code-build-output
Hope it helps!

Azure devops yaml: release to multiple web nodes

I'm trying to figure out how I can use yaml pipelines to deploy an application to a multi web-node environment.
I'm trying to create a pipeline with 2 stages. Stage 1 will build the project, and stage 2 will deploy it to a staging environment. This staging environment (currently) has 2 web nodes. I don't want to hardcode the actual machines to deploy to into my pipeline. So I thought I'd add a variable group with a variable containing the web nodes to deploy to, and use a "each" statement to generate a job per node.
However, this doesn't work for several reasons:
The jobs are generated before the variable group is read, so it shows an error it cannot find the variable
Apparently an array variable is not supported at all, except for the built in variables
So my question is, how do other people solve this? I'd like to define the servers to deploy to in a central place, and not in my pipeline definition.
My initial attempt is printed below. This doesn't work, but it does describe what I'm trying to accomplish.
Main yaml:
variables:
- group: LicenseServerVariables #this contains StagingWebNodes variable
stages:
- stage: Build
displayName: Build
<some build steps>
- stage: DeployTest
displayName: Deploy on test
condition: and(succeeded(), eq(variables['DeployToTest'], 'true'))
jobs:
- template: Templates\Deploy.yaml
parameters:
nodes: $(StagingWebNodes)
Deploy.yaml:
parameters:
nodes: []
jobs:
- ${{ each node in parameters.nodes }}:
- job: ${{ node }}
displayname: deploy to ${{ node }}
pool: Saas Staging
demands: ${{ node }}
steps:
- template: DeployToNode.yaml
edit:
I'm a bit closer to a solution. I was able to get the pipeline to work with the "each" construct using the following adjustment to the Deploy.yaml:
parameters:
nodes:
- name: 'Node1'
pool:
name: StagingPool
demands: 'Node1'
- name: 'Node2'
pool:
name: StagingPool
demands: 'Node2'
jobs:
- ${{ each node in parameters.nodes }}:
- job: ${{ node.name }}
displayName: deploy to ${{ node.name }}
pool: ${{ node.pool }}
steps:
- template: DeployToNode.yaml
This makes it a bit better. However, I still don't want to define the "nodes" parameter in my pipeline yaml source, but in a variable group (or some other place if anyone has a good suggestion)
With the addition of virtual machines as resource for environments this issue has gone away. I can now use a rolling deployment task to deploy the apppicatio to all web nodes

Concourse: Resource not found?

I am trying to built a concourse pipeline which is triggered by git, and then runs a script in that git repository.
This is what I have so far:
resources:
- name: component_structure_git
type: git
source:
branch: master
uri: git#bitbucket.org:foo/bar.git
jobs:
- name: component_structure-docker
serial: true
plan:
- aggregate:
- get: component_structure_git
trigger: true
- task: do-something
config:
platform: linux
image_resource:
type: docker-image
source: { repository: ubuntu }
inputs:
- name: component_structure_git
outputs:
- name: updated-gist
run:
path: component_structure_git/run.sh
- put: component_structure-docker
params:
build: component_structure/concourse
- name: component_structure-deploy-test
serial: true
plan:
- aggregate:
- get: component_structure-docker
passed: [component_structure-docker]
- name: component_structure-deploy-prod
serial: true
plan:
- aggregate:
- get: component_structure-docker
passed: [component_structure-docker]
When I apply this code with fly, everything is ok. When I try to run the build. it fails with the following error:
missing inputs: component_structure_git
Any idea what I am missing here?
Agree with first answer. When running things in parallel (aggregate blocks) there are a few things to consider
How many inputs do I have? I have more than one, let's run these get steps in an aggregate block
If I have two tasks, is there a dependency between the tasks that can change the outcome of a task run, eg. Do I have an output from one task that is required in the next task
I have a sequence of put statements, let's run these steps in an aggregate block
just a guess but aggregate is causing the issue. You can't have an input from something that is executing at the same time? Why do you have aggregate anyway? This is usually used for "get" to speed up the process.

Resources