Situation
I'm working on an AzureDevops Server 2020 with only one agent.
I have 2 build pipelines:
Build pipeline (yaml)
Merge pipeline (yaml)
Each pipeline contains multiple stages that contains only one job (because each task of the stage must run on same agent).
Current behavior
If I run the two pipelines at the same time, the agent run the two in a "fake" parallel that make the two builds very slow.
Exemple of agent process order:
build-stage1, build-stage2, merge-stage1, merge-stage2, merge-stage3, merge-stage4, build-stage3...
Wanted behavior
This is not something unexpected if we have more agents than build executions. But this will never be my case.
So I will prefer to lock the agent for the current build (like built-in in Jenkins).
Exemple of agent wanted process order:
build-stage1, build-stage2, build-stage3, build-stage4, build-stage5(latest), merge-stage1, merge-stage2, merge-stage3, merge-stage4, merge-stage5(latest)
Is it possible to set the agent work attribution policy ?
This occurs because a job is not added to agent queue if it depends on something (and by default it depends on previous stage).
Using dependsOn: [] let Azure Devops know that it depends on nothing so each jobs are added to the queue and are executed in FIFO order.
Agree with Dom.
Based on my test, I could reproduce this behavior. But I am afraid that there is no method to lock the agent to complete the pipeline before taking another one.
For a workaround:
You could use Pipeline trigger to trigger the merge Pipeline.
Build Pipeline:
pool:
name: Default
stages:
- stage: Build_Stage1
displayName: Stage1
....
- stage: Build_Stage2
displayName: Stage2
dependsOn: Build_Stage1
....
- stage: Build_Stage3
displayName: Stage3
dependsOn: Build_Stage2
....
Merge Pipeline:
resources:
pipelines:
- pipeline: TestTrigger
source: ABC
trigger:
branches:
- '*'
pool:
name: Default
stages:
- stage: Merge_Stage1
displayName: Merge1
...
- stage: Merge_Stage2
displayName: Merge2
dependsOn: Merge_Stage1
...
- stage: Merge_Stage3
displayName: Merge3
dependsOn: Merge_Stage2
...
In this case, you could queue the Build Pipeline alone. Then the Merge Pipeline will be triggered after completing the Build Pipeline.
The Process: build-stage1, build-stage2, build-stage3, build-stage4, build-stage5(latest), -> Tiggeer -> merge-stage1, merge-stage2, merge-stage3, merge-stage4, merge-stage5(latest)
On the other hand, this requirement is valuable.
To get this feature, you could add your request for this feature on our UserVoice site, which is our main forum for product suggestions. Thank you for helping us build a better Azure DevOps.
Related
I would like to make a given stage fail to test my conditions
- stage: EnvironmentDeploy
condition: and(succeeded()...)
is it possible to make - stage to fail purposefully?
The easiest way I can think of is adding a job with a PowerShell script, and using the throw keyword to exit the script with an error:
stages:
- stage: StageToFail
jobs:
- job: JobToFail
steps:
- pwsh: throw "Throwing error for debugging purposes"
I assume you don't want this fail to happen consistently as that would make deployment impossible. Assuming it's a one-off thing, why not try to upload code with compile errors so the build fails or - if your code base has them and your pipeline checks them as a prerequisite for deployment - add a unit test that fails.
I have tried to use ManualValidation task in my pipeline but it did not passed because it is not installed from the market place but, when I go on the page on the market place I can not find any extension for manual validation on pipeline. Can someone provide me with information about this?
My azure-pipeline.yml looks like this:
trigger:
main
jobs:
job: waitForValidation
displayName: Wait for external validation
pool: server
timeoutInMinutes: 4320
steps:
task: ManualValidation#0
timeoutInMinutes: 1440
inputs:
notifyUsers: |
myemail#name.com
instructions: 'Please validate the build configuration and resume'
onTimeout: 'resume'
A task is missing. The pipeline references a task called 'ManualValidation'. This usually indicates the task isn't installed, and you may be able to install it from the Marketplace: https://marketplace.visualstudio.com.
I have a list of jobs A, B, C, D... which run no matter if the previous job succeed.
The last of them is the artifact publish, that will be used by the Release pipeline. I would like to not run it if none of the previous jobs succeed.
I need to deploy any service which does not fail (so I need to pass the failed services creations) but prevent the automatic launch of the release pipeline linked if all failed.
Here is the YAML code given by Azure
pool:
name: Default
demands:
- msbuild
- visualstudio
steps:
- task: VSBuild#1
displayName: 'Generate Solution'
inputs:
solution: LisaMES.sln
platform: '$(PlatformName)'
configuration: '$(ConfigurationName)'
- task: ArchiveFiles#2
displayName: 'Create Service A'
inputs:
rootFolderOrFile: '$(ServiceName)\bin\$(ConfigurationName)'
includeRootFolder: false
archiveFile: 'Bin_Services/$(ServiceName)_$(ConfigurationName).zip'
condition: succeededOrFailed()
- task: ArchiveFiles#2
displayName: 'Create Service B'
inputs:
rootFolderOrFile: '$(ServiceName)\bin\$(ConfigurationName)'
includeRootFolder: false
archiveFile: 'Bin_Services/$(ServiceName)_$(ConfigurationName).zip'
condition: succeededOrFailed()
... C D E F like this
- task: PublishPipelineArtifact#0
displayName: 'Publish Pipeline Artifact'
inputs:
artifactName: Services
targetPath: 'Bin_Services'
condition: succeededOrFailed()
remove the condition on the PublishPipelineArtifact#0 As it currently reads this task will run irregardless if the preceding tasks succeeded or published.
Even if a previous dependency has failed, unless the run was canceled. Use succeededOrFailed() in the YAML for this condition.
Unless there is a reason you'd want to publish a subset of artifacts that succeeded (this could lead to an inconsistent build state as the same build could produce different artificats) would recommend removing the condition from all the tasks. The default behavior is to run each task only if the preceding task was successful.
Context: performing Android instrumentation (UI) tests with Azure Pipelines.
There are 2 jobs: one does the testing (launches an emulator and runs the tests), and the other job reports an error, if the previous job fails for some reason.
I have the following simple setup in my Azure Pipelines:
jobs:
- job: SmokeTesting
displayName: Smoke testing
timeoutInMinutes: 60
pool:
vmImage: 'macOS-latest'
steps:
- script: meta/scripts.sh launch_avd
displayName: Launch AVD
workingDirectory: ''
- task: Gradle#2
displayName: Run smoke tests
inputs:
workingDirectory: ''
gradleWrapperFile: 'gradlew'
publishJUnitResults: true
tasks: ':app:connectedAndroidTest'
- job: ReportFailure
displayName: Report failure
dependsOn:
- SmokeTesting
condition: or(failed(), canceled())
steps:
- script: meta/scripts.sh report_smoke_tests_error
workingDirectory: ''
env:
BUILD_ID: $(Build.BuildId)
It all works as expected: if there is an error, the second job is run. In this case, the log in Azure Pipelines Web contains very useful information, that I would like to have access to in the second job:
* What went wrong:
Execution failed for task ':app:stripDebugDebugSymbols'.
> No version of NDK matched the requested version 22.0.7026061. Versions available locally: 18.1.5063045, 21.3.6528147, 21.3.6528147
How do I get "What went wrong" message in my second job?
My idea is to use multi-stage variable to record the message in the first job, and then use it in the second one. Unfortunately, I haven't figured out how to get this message in the first place.
As a workaround, you can use the Build Timeline api to get detailed build information. The api response contains the property issues, you can check the results there if there is an errors.
https://dev.azure.com/{org}/{pro}/_apis/build/builds/3838/timeline/?api-version=6.0
If the issues does not contain the error message you want, you can retrieve the content related to Gradle#2 task in the response body. Obtain the log url according to the property log.
By calling this log url, you can get the log of Gradle#2 task, and then parse the log to get the desired message.
we have a CI pipeline on our repository, hosted in gitlab
we setup gitlab-runner on our local machine
the pipeline running 4 steps
build
unit tests
integration test
quality tests
all this pipeline takes almost 20 min
and the pipeline trigger on each push to a branch
is there a way to configure the gitlab-runner that if the HEAD of a branch that the runner currently running on changes the pipe
will auto cancel the run? because the latest version is what matters
for example in this run the lower run is unnecessary
gitlab-ci.yml
stages:
- build
- unit_tests
- unit_and_integration_tests
- quality_tests
build:
stage: build
before_script:
- cd projects/ideology-synapse
script:
- mvn compile
unit_and_integration_tests:
variables:
GIT_STRATEGY: clone
stage: unit_and_integration_tests
only:
- /^milestone-.*$/
script:
- export RUN_ENVIORMENT=GITLAB_CI
- export MAVEN_OPTS="-Xmx32g"
- mvn test
- "cat */target/site/jacoco/index.html"
cache: {}
artifacts:
reports:
junit:
- "*/*/*/target/surefire-reports/TEST-*.xml"
unit_tests:
variables:
GIT_STRATEGY: clone
stage: unit_tests
except:
- /^milestone-.*$/
script:
- export MAVEN_OPTS="-Xmx32g"
- mvn test
- "cat */target/site/jacoco/index.html"
cache: {}
artifacts:
reports:
junit:
- "*/*/*/target/surefire-reports/TEST-*.xml"
quality_tests:
variables:
GIT_STRATEGY: clone
stage: quality_tests
only:
- /^milestone-.*$/
script:
- export RUN_ENVIORMENT_EVAL=GITLAB_CI
- export MAVEN_OPTS="-Xmx32g"
- mvn test
cache: {}
edit after #siloko comment:
I already try using
the auto-cancel redundant, pending pipelines in the setting menu
I want to cancel running pipelines and not pending
after forther investigation, I found that I had 2 active runners
on one of my machines
one shared runner , and another specific runner then if I push a 2 commit one after another to the same branch both of the runners take the jobs and execute them.
that also explains why
Auto-cancel redundant, pending pipelines
options, didn't work because it works only when the same runner have pending jobs
actions that been taken to fix this problem: unregister the specific runner and leave the machine only with the shared runner