How do you run xcuitests in parallel using azure dev Ops? - xcode

Question

I'd like to execute my xcuitest suite on azure dev ops in parallel using xcode's out of the box capabilities. Locally, it's as simple as checking the box to enable parallel testing on my test target.
Parallel executions toggle
Locally, xcode opens multiple simulators and the tests run as expected. On azure, they run and pass as expected but they take the same amount of time as they usually do, indicating to me that they are not running in parallel. What am I missing here? Are there extra steps I need to take to get them running in parallel via azure dev ops?
Azure-Pipleline.yml snippet
- stage: uiTests
displayName: UI Tests
dependsOn: []
condition: | # don't run for release builds to save Azure bandwith
not(
or(
contains(variables['Build.SourceBranch'], 'master'),
contains(variables['Build.SourceBranch'], 'release')
)
)
jobs:
- template: azure-templates/job-tests.yml
parameters:
configuration: "$(UI_TEST_CONFIGURATION)"
sdk: "$(UI_TEST_SDK)"
scheme: "$(UI_TEST_SCHEME)"
artifactName: "UI Tests Results - Attempt #$(System.StageAttempt).$(System.JobAttempt)" # include attempt number in suffix because reruns can not overwrite artifacts"
shouldRunWiremock: true
azure-templates/job-test.yml snippet
- job: Test
pool: Hosted macOS
variables:
- template: variables.yml
- group: blah-Keys
steps:
- template: steps-install-code-signing.yml
parameters:
certFile: "$(UAT_SIGNING_FILE_ID)"
signingIdentity: "$(UAT_SIGNING_IDENTITY)"
provisioningFile: "$(UAT_PROVISIONING_FILE_ID)"
- script: "./setBuildVersion.sh"
displayName: "Update Build Number"
- script: "./xcconfig.sh"
displayName: "Populate xcconfig files"
- bash: "./runWiremock.sh &"
continueOnError: true
displayName: "Start Mock Server"
enabled: ${{ parameters.shouldRunWiremock }}
- task: Xcode#5
displayName: "Run Tests"
inputs:
actions: test
configuration: "${{ parameters.configuration }}"
sdk: "${{ parameters.sdk }}"
xcWorkspacePath: "$(WORKSPACE_PATH)"
scheme: "${{ parameters.scheme }}"
xcodeVersion: specifyPath
xcodeDeveloperDir: "$(XCODE_PATH)"
signingOption: default
args: "-resultBundlePath $(build.SourcesDirectory)/testResults"
destinationPlatformOption: iOS
destinationSimulators: "$(TEST_SIMULATOR)"
publishJUnitResults: true
continueOnError: false # report test failures as errors in Azure DevOps
- publish: testResults
artifact: "${{ parameters.artifactName }}.xcresult"
condition: not(canceled()) # run if there are test failures

Answer 1

You can consider to use build matrix which is one feature of YAML to achieve the test ran with multiple simulators:
jobs:
- job: Build
strategy:
matrix:
simulators1:
destinationSimulators: 'iPhone X'
simulators2:
destinationSimulators: 'iPhone 7'
..
..
- task: Xcode#5
displayName: "Run Tests"
..
..
destinationSimulators: "$(destinationSimulators)"
..
..