I'm defining a github action script that's referencing to another yaml file, hoping to put the configuration into a more organised way.
Here is my job file, named as deploy.yml in the path of ./.github/workflows/, where the first . is the root folder of my project.
....
jobs:
UnitTest:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v3
- uses: ./.github/workflows/unittest.yml
In the same ./.github/workflows/ folder, I created another file called unittest.yml as below:
name: "UnitTest"
description: "Perform Unit Test"
runs:
# using: "composite"
- name: Dependency
run: |
echo "Dependency setup commands go here"
- name: UnitTest
run: make test.unit
However, when I tried to test the script locally using act with command act --secret-file .secrets --container-architecture linux/amd64, I received the following error:
[Deploy/UnitTest] ✅ Success - Main actions/checkout#v3
[Deploy/UnitTest] ⭐ Run Main ./.github/workflows/unittest.yml
[Deploy/UnitTest] ❌ Failure - Main ./.github/workflows/unittest.yml
[Deploy/UnitTest] file does not exist
[Deploy/UnitTest] 🏁 Job failed
I have tried to put just the file name unittest.yml or ./unittest.yml or myrepo_name/.github/workflows/unittest.yml or put the file into a subfolder like step 2 of this document illustrated, but all no luck.
Based on examples of runs for composition actions, I would imagine this should work.
Would anyone please advise?
P.S. You might have noticed the commented line of using: "composite" in the unittest.yml. If I uncomment the line, I'll receive error:
Error: yaml: line 3: did not find expected key
Composite actions are not referenced by YAML file, but a folder. In that folder, you are expected to have an action.yml describing the action.
This is why you're getting the error with using: composite, you're defining a workflow (because it's in ./github/workflows), but you are using action syntax.
I would advise this folder structure:
.github/
|-- workflows/
| -- deploy.yml
unittest-action/
|-- action.yml
With this structure, you should be able to reference the action with
- uses: actions/checkout#v3
- uses: ./unittest-action
Please see the docs for more information.
Depending on your use-case and setup, you might also want to consider reusable workflows.
You can define a reusable workflow in your .github/workflows directory like so:
# unittest.yml
on: workflow_call
jobs:
deploy:
# ...
and then call it like so:
jobs:
UnitTest:
uses: ./.github/workflows/unittest.yml
Note how the reusable workflow is an entire job. This means, you can't do the checkout from the outside and then just run the unit test in the reusable job. The reusable job (unittest.yml) needs to do the checkout first.
Which one to pick?
Here's a blog post summarising some of the differences between composite actions and reusable workflows, like:
reusable workflows can contain several jobs, composite actions only contain steps
reusable workflows have better support for using secrets
composite actions can be nested, but as of Jul '22, reusable workflows can't call other reusable workflows
I have a CircleCi's workflow that has 2 jobs. The second job (gradle/test) is dependent on the first one creating some files for it.
The problem is with the first job running inside a docker, and the second job (gradle/test) is not. Hence, the gradle/test is failing since it cannot find the files the first job created. How can I set gradle/test to work on the same space?
Here is a code of the workflow:
version: 2.1
orbs:
gradle: circleci/gradle#2.2.0
executors:
daml-executor:
docker:
- image: cimg/openjdk:11.0-node
...
workflows:
checkout-build-test:
jobs:
- daml_test:
daml_sdk_version: "2.2.0"
context: refapps
- gradle/test:
app_src_directory: prototype
executor: daml-executor
requires:
- daml_test
Can anyone help me configure gradle/test correctly?
CircleCI has a mechanism to share artifacts between jobs called "workspace" (well, they have multiple ones, but workspace is what you want here).
Concretely, you would add this at the end of your daml_test job definition, as an additional step:
- persist_to_workspace:
root: /path/to/folder
paths:
- "*"
and that would add all the files from /path/to/folder to the workspace. On the other side, you can "mount" the workspace in your gradle/test job by adding something like this before the step where you need the files:
- attach_workspace:
at: /whatever/mountpoint
I like to use /tmp/workspace for the path on both sides, but that's just personal preference.
So, I have a yaml pipeline that has an array storing a set of versions in bash, let's say
arrayVersions=(3.0.1 3.0.2 ....).
Now, I want to set up the pipeline that splits each of these versions in one single job in the yaml pipeline, then run them in multi-agent paradigm.
CONTEXT -
I have set up the pipeline that iterates over the array and runs, however, it is very slow since it runs sequentially. So, I tried multithreaded parallel programming in bash, but it did not work out. In ideal solution, I am thinking to split up all the versions and run them as a new job in the pipeline. It would be something like this:
jobs:
# get all the versions
# split up each version into 1 single job and run the jobs in parallel
job: 3.0.1
...
job: 3.0.2
...
Is there any way I can set it up?
Have you tried using a template and calling it from the jobs section? Here's an example:
# azure-pipelines.yml
trigger:
- none
jobs:
- job: Build
steps:
- template: build-specific-version.yml
parameters:
appVersion:
- '3.0.1'
- '3.0.2'
- '3.0.3'
# build-specific-version.yml
parameters:
- name: 'appVersion'
type: object
default:
- '1.0'
- '1.1'
steps:
- ${{ each v in parameters.appVersion }}:
- script: echo ${{ v }}
Docs: Microsoft technical documentation|Template types & usage
Also see: Loops and arrays in Azure Devops Pipelines
I have a variable called validate in the first pipeline and if it is set to true skip the trigger of the second pipeline .
The yaml sample pipeline
trigger: none // add this trigger value to none
resources:
pipelines:
- pipeline: RELEASE_PIPELINE // any arbitrary name
source: PIPELINE_NAME. // name of the pipeline shown on azure UI portal
trigger:
branches:
include:
- dummy_branch
I have a variable(as global) set on the pipeline called validate as true/false.
The second pipeline should not be triggered if its set to True.
Your requirement can not be achieved using variable in a triggering pipeline. Instead of using triggering pipeline, you could use a task to queue the second pipeline and condition whether the task would run. For example:
parameters:
- name: 'validateTrigger'
type: boolean
default: false
steps:
- task: #first pipeline tasks
- ${{ if eq(parameters.validateTrigger, true) }}:
- script: #queue second pipeline
I'm setting up an Azure Pipelines build that needs to package a C# .NET class library into a NuGet package.
In this documentation, it lists a couple different ways to automatically generate SemVer strings. In particular, I want to implement this one:
$(Major).$(Minor).$(rev:.r), where Major and Minor are two variables
defined in the build pipeline. This format will automatically
increment the build number and the package version with a new patch
number. It will keep the major and minor versions constant, until you
change them manually in the build pipeline.
But that's all they say about it, no example is provided. A link to learn more takes you to this documentation, where it says this:
For byBuildNumber, the version will be set to the build number, ensure
that your build number is a proper SemVer e.g. 1.0.$(Rev:r). If you
select byBuildNumber, the task will extract a dotted version, 1.2.3.4
and use only that, dropping any label. To use the build number as is,
you should use byEnvVar as described above, and set the environment
variable to BUILD_BUILDNUMBER.
Again, no example is provided. It looks like I want to use versioningScheme: byBuildNumber, but I'm not quite sure how to set the build number, I think it pulls it from the BUILD_BUILDNUMBER environment variable, but I can't find a way to set environment variables, only script variables. Furthermore, am I suppose to just set that to 1.0.$(Rev:r), or to $(Major).$(Minor).$(rev:.r)? I'm afraid that would just interpret it literally.
Googling for the literal string "versioningScheme: byBuildNumber" returns a single result... Does anyone have a working azure-pipelines.yml with this versioning scheme?
Working YAML example for Packaging/Versioning using byBuildNumber
NOTE the second parameter of the counter - it is a seed value, really useful when migrating builds from other build systems like TeamCity; It allows you to set the next build version explicitly upon migration. After the migration and initial build in Azure DevOps, the seed value can be set back to zero or whatever start value (like 100) you may prefer every time majorMinorVersion is changed:
reference: counter expression
name: $(majorMinorVersion).$(semanticVersion) # $(rev:r) # NOTE: rev resets when the default retention period expires
pool:
vmImage: 'vs2017-win2016'
# pipeline variables
variables:
majorMinorVersion: 1.0
# semanticVersion counter is automatically incremented by one in each execution of pipeline
# second parameter is seed value to reset to every time the referenced majorMinorVersion is changed
semanticVersion: $[counter(variables['majorMinorVersion'], 0)]
projectName: 'MyProjectName'
buildConfiguration: 'Release'
# Only run against master
trigger:
- master
# Build
- task: DotNetCoreCLI#2
displayName: Build
inputs:
projects: '**/*.csproj'
arguments: '--configuration $(BuildConfiguration)'
# Package
- task: DotNetCoreCLI#2
displayName: 'NuGet pack'
inputs:
command: 'pack'
configuration: $(BuildConfiguration)
packagesToPack: '**/$(ProjectName)*.csproj'
packDirectory: '$(build.artifactStagingDirectory)'
versioningScheme: byBuildNumber # https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/build/dotnet-core-cli?view=azure-devops#yaml-snippet
# Publish
- task: DotNetCoreCLI#2
displayName: 'Publish'
inputs:
command: 'push'
nuGetFeedType: 'internal'
packagesToPush: '$(build.artifactStagingDirectory)/$(ProjectName)*.nupkg'
publishVstsFeed: 'MyPackageFeedName'
byBuildNumber uses the build number you define in your YAML with the name field.
Ex: name: $(Build.DefinitionName)-$(date:yyyyMMdd)$(rev:.r)
So if you set your build format to name: 1.0.$(rev:.r), it should work as you expect.
I had the similar issue and now let me make it clear.
Firstly, what is the definition of Build Number?
By the official document of Azure Pipeline YAML scheme, it is
name: string # build numbering format
resources:
containers: [ containerResource ]
repositories: [ repositoryResource ]
variables: { string: string } | [ variable | templateReference ]
trigger: trigger
pr: pr
stages: [ stage | templateReference ]
Look at the first line:
name: string # build numbering format
Yes, that's it!
So you could define it like
name: 1.0.$(Rev:r)
if you prefer to Semantic Versioning. Then
Secondly, what's the meaning of versioningScheme: 'byBuildNumber' in task NuGetCommand#2?
It's really straightforward: just use the format defined by name!
Last but not least
The official document on Package Versioning and Pack NuGet packages don't make it clear that what a build number really is and how to define it. It's really misleading. And I'm so sad as an MS employee as I'd to resort to external resource to make all that clear.
Azure Pipeline Nuget Package Versioning Scheme, How to Get “1.0.$(Rev:r)”
This should be a issue in the documentation. I reproduced this issue when I set $(Major).$(Minor).$(rev:.r) in the Build number format in the Options of build pipeline:
However, I suddenly noticed that the build number is not correct with that format after many build tests:
There are two points . between 0 and 2 (Open above image in a new tab). Obviously this is very strange. So, I changed the Build number format to:
$(Major).$(Minor)$(rev:.r)
Or
$(Major).$(Minor).$(rev:r)
Now, everything is working fine.
As test, I just set the Build number format to $(rev:.r), and the build number is .x. So, we could confirm that the value of $(rev:.r) including the . by default.
Note: Since where Major and Minor are two variables defined in the build pipeline, so we need defined them in the variables manually.
Hope this helps.
Issues
My issues:
when trying the answer by #Emil, my first package started at 2.0 (I did no further testing to investigate)
when trying the answer by #Leo Liu-MSFT, I was unable to find the matching "Options" tab.
I therefore used this solution by #LanceMcCarthy.
Fix
Set the variables:
variables:
major: '1'
minor: '0'
revision: $[counter(variables['minor'], 1)] # This will get reset every time minor gets bumped.
nugetVersion: '$(major).$(minor).$(revision)'
then use nugetVersion as an environment variable when packing:
- task: NuGetCommand#2
inputs:
command: 'pack'
packagesToPack: '**/*.csproj'
packDestination: '$(Build.ArtifactStagingDirectory)'
versionEnvVar: 'nugetVersion'
versioningScheme: 'byEnvVar'
I think the issue many of use have is that there is no option menu. I upvoted SHarpC's post as this worked for me.
I have a workaround to add a suffix (i.e. '-beta'), since it's for some reason ignored by the Nuget pack command when using the Classic editor and setting auto versioning by build number:
Set a new environment variable, set the value as the predefined $(Build.BuildNumber) variable:
Set the build number:
Set NuGet pack command to auto-name by environment variable and specify newly added variable name:
If you're interested in the whole build/release pipeline design and YAML, have a look at my article here
The core of the problem is solved in the approved answer and refined in #Emils answer, so this is just another approach to the azure-pipelines.yml that works for us with DevOps artifacts.
name: $(majorMinorVersion).$(semanticVersion)
trigger:
- main
variables:
majorMinorVersion: 1.0
semanticVersion: $[counter(variables['majorMinorVersion'], 0)]
pool:
vmImage: windows-latest
steps:
- task: DotNetCoreCLI#2
displayName: 'Create Packages'
inputs:
command: pack
configuration: 'Release'
packagesToPack: '**/<VS projectname>.csproj'
versioningScheme: byBuildNumber
- task: NuGetAuthenticate#0
displayName: 'NuGet Authenticate'
- task: NuGetCommand#2
displayName: 'NuGet Push to feed'
inputs:
command: push
publishVstsFeed: '<DevOps projectname>/<feed name>'
BTW: Don't forget this little hinch