azure devops pipeline | bash script to run dotnet test on multiple projects - bash

Question

I'm trying to run the dotnet test command on defined test projects in a MS Azure devops pipeline.
This is working:
- script: |
dotnet test "./Project One/Project One Unit Tests.fsproj" -c Release --no-build --filter "TestCategory!=SKIP_ON_DEPLOY & TestCategory!=REQUIRES_API_KEY"
dotnet test "./Project Two/Project Two Unit Tests.fsproj" -c Release --no-build --filter "TestCategory!=SKIP_ON_DEPLOY & TestCategory!=REQUIRES_API_KEY"
displayName: 'run tests in cascade'
I don't want to specify the project names, only some rule (project name has to end with "UnitTests", "Unit Tests" or "Unit_Tests") but dotnet test <PROJECT> does not allows wildcards.
Seems like wildcards works with dotnet test <DLL> (./**/*Unit?Tests.dll) but it fails because it does not find the deps.json file on the obj folder.
My solution is to loop trough a filtered list of project files:
for proj in ./**/*Unit?Tests.*proj
do
dotnet test "$proj" -c Release --no-build --filter "TestCategory!=SKIP_ON_DEPLOY & TestCategory!=REQUIRES_API_KEY"
done
It's running the tests but differently from the cascade calls, here when a project test fails it does not fail the step so the pipeline does not stop!
I tried to get the result form the run, but unsuccessfully (this does not work):
for proj in ./**/*Unit?Tests.*proj
do
result=$(dotnet test "$proj" -c Release --no-build --filter "TestCategory!=SKIP_ON_DEPLOY & TestCategory!=REQUIRES_API_KEY")
if result = 1
then
exit 1
fi
done
Any suggestion?
Why when dotnet test fails (exit 1) the step does not fail?
I tried to put in the loop only the failing test project and it fails in that case.
[UPDATE]
Full pipeline.yaml:
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
variables:
project file: "Alex75.MySolution/Alex75.MyProject.fsproj"
steps:
- script: dotnet build -c Release
displayName: 'Build'
- script: |
dotnet test "./ProjectTwo Unit Tests/ProjectTwo Unit Tests.fsproj" -c Release --no-build --filter "TestCategory!=SKIP_ON_DEPLOY & TestCategory!=REQUIRES_API_KEY"
dotnet test "./ProjectOne Tests/ProjectOne Unit Tests.fsproj" -c Release --no-build --filter "TestCategory!=SKIP_ON_DEPLOY & TestCategory!=REQUIRES_API_KEY"
displayName: 'Test'
condition: false
- script: |
for proj in ./**/*Unit?Tests.*proj
do
echo "run tests in $proj"
dotnet test "$proj" -c Release --no-build --filter "TestCategory!=SKIP_ON_DEPLOY & TestCategory!=REQUIRES_API_KEY"
done
displayName: 'Test (Unit Test projects)'
condition: true
- powershell: |
$URL = "$(System.CollectionUri)/$(System.TeamProject)/_apis/build/builds/$(Build.BuildId)/logs?api-version=5.1"
Write-Host "URL = $URL"
$logs = Invoke-RestMethod -Uri $URL -Headers #{authorization = "Basic $(PAT)"} -Method Get
$lastLogId = $Logs.value[$Logs.value.count-1].id
Write-Host "lastLogId = $lastLogId"
$URL = "$(System.CollectionUri)/$(System.TeamProject)/_apis/build/builds/$(Build.BuildId)/logs/$lastLogId?api-version=5.1"
$result = Invoke-RestMethod -Uri $URL -Headers #{authorization = "Basic $(PAT)"} -Method Get
Write-Host $result
Write-Host "Start Check result..."
$lines = $result.Split([Environment]::NewLine)
foreach($line in $lines) {
if($line -match "Failed!")
{
throw 'dotnet test fails ($line)'
}
}
Write-Host "Test result check completed."
displayName: 'Check tests result'
PowerShell script
Using the example of #vito-liu-msft I tried to check the test logs to check the error.
Here the powershell script alone:
$URL = "$(System.CollectionUri)/$(System.TeamProject)/_apis/build/builds/$(Build.BuildId)/logs?api-version=5.1"
Write-Host "URL = $URL"
$logs = Invoke-RestMethod -Uri $URL -Headers #{authorization = "Basic $(PAT)"} -Method Get
$lastLogId = $Logs.value[$Logs.value.count-1].id
Write-Host "lastLogId = $lastLogId"
$URL = "$(System.CollectionUri)/$(System.TeamProject)/_apis/build/builds/$(Build.BuildId)/logs/$lastLogId?api-version=5.1"
$result = Invoke-RestMethod -Uri $URL -Headers #{authorization = "Basic $(PAT)"} -Method Get
Write-Host $result
Write-Host "Start Check result..."
$lines = $result.Split([Environment]::NewLine)
foreach($line in $lines) {
if($line -match "Failed!")
{
throw 'dotnet test fails ($line)'
}
}
Write-Host "Test result check completed."
PAT is the Personal Access Token as it is created, no need to be transformed
before using it in the HTTP request.
The LogId is not available so there is a request to get the logs collection and than a second request to get the specific last log (the response seems ordered by date, maybe it's worth to double check).
Note that if the second request returns a 404, 500 or other result (not a failure) the error check will not find a evenctual errors! A proper check of the response is required.
I tested the -match "Failed!" in PowerShell but I haven't tested the throw command in the pipeline because...
dotnet test command in the loop is working!
After spending so much time trying to figure out how to read and check the logs throw a separate PowerShell script I find out that the initial simple solution works!
Yes, the build fails because of the test failure (it also shows the precise error pointing to the problem) and the step fails.
(so the next step, the check of the test results, is skipped!)
I think I should have made some error before on filtering the test projects so that the failing project was not running (and not raising any error) so I thoutght it was not "intercepting" the error, and the build didn't stop.
It could be that I mixed 2 different pipelines files while I was changing it.
Anyway this is the incriminated step (script):
for proj in ./**/*Unit?Tests.*proj
do
echo "run tests in $proj"
dotnet test "$proj" -c Release --no-build --filter "TestCategory!=SKIP_ON_DEPLOY & TestCategory!=REQUIRES_API_KEY"
done
At list with the echo it is possible to see which test projects are used.

Answer 1

when a project test fails it does not fail the step so the pipeline does not stop!
Why when dotnet test fails (exit 1) the step does not fail?
As a workaround, we could get the dotnet test task log id via this rest api, add the task power shell and enter below script to analyze dotnet test log. we need to enter match code, such as fails (exit 1), If dotnet test fails it will stop the pipeline.
We should add PAT to variable and set it to secret, then use it in the script
$connectionToken="{PAT}"
$base64AuthInfo= [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($connectionToken)"))
$URL = "https://dev.azure.com/{organization}/{project}/_apis/build/builds/{buildId}/logs/{logId}?api-version=6.1-preview.2"
$Result = Invoke-RestMethod -Uri $URL -Headers #{authorization = "Basic $base64AuthInfo"} -Method Get
Write-Host $result
$lines = $result.Split([Environment]::NewLine)
$passed = 0;
$failed = 0;
foreach($line in $lines) {
if ($line -match "{match sentence}") {
throw 'dotnet test fails (exit 1)'
}
}
Update1
how can I find the "logId" ?
We could use this REST API to check logId
GET https://dev.azure.com/{organization}/{project}/_apis/build/builds/{buildId}/logs?api-version=6.1-preview.2
Result:
And I really have to use a api "preview" version?
We could also use other version, such as 5.1, we could switch the REST API version in the doc, you could check the pic below.