In Jenkins I create a Visual studio project using MSBuild. After the build I execute a windows batch command that executes the tests en sets the output file.
Post build I publish the .trx test report file. This all works perfect, the only problem is that if a test fails the build fails. With other types of projects if a test fails the build only becomes unstable.
How can I do the same for my Visual studio project?
Below my batch command:
"C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\MSTest.exe" /testcontainer:TestAppTests\bin\Debug\TestAppTests.dll /resultsfile:TestResults.trx
I know it's pretty old but here are something as I've encountered the same type of problem.
Using Batch:
At the end of that batch command step where you run the tests, add one more exit command, such as exit /b 0 or exit 0.
Using Powershell:
At the end of the powershell command step where you run the tests, same, manually set the exit code $LastExitCode = 0.
Final:
The result will always be set to success no matter how many tests failed so you need to parse and analyze the result yourself. There are tools such as: https://plugins.jenkins.io/mstest and https://plugins.jenkins.io/xunit.
Related
I have a solution in Visual Studio with a bunch of projects under it. A number of these projects are test projects, containing only nunit tests and code relating to tests. For these projects, I want to set up a post-build event that runs the tests for each such test project. I have researched this but haven't found any questions exactly asking to do what I'm asking- see below. I already have the nunit framework installed on Visual Studio via nuget and I can run nunit tests already from within the VS test explorer.
Notes
Why am I doing this? Well, we don't have time to set up continuous integration or a build server, yet. I just wanted something quick that would fail the build of the solution if any tests fail. In particular, there is a Wix project in our solution that builds last and this depends on all the other projects. So if post-build in any of them fails, it fails the Wix build. This is what we want: we don't want the Wix to build out an MSI unless all tests pass.
I have already figured this out, but want to share the knowledge with others (https://stackoverflow.blog/2011/07/01/its-ok-to-ask-and-answer-your-own-questions/) & also see if there's a better way to do this.
Tried and tested in VS2019, but probably applies to other versions too
Links
This asks how to run mstest as a post-build, rather than nunit: Using Post-Build Event To Execute Unit Tests With MS Test in .NET 2.0+
This asks how to run nunit from within VS, but not as a post-build: How to run NUnit tests in Visual Studio 2017?
The first thing you need to do is install nunit.consolerunner if you don't have it already. Install this via NuGet in VS. Once this is done, you can proceed to writing a post-build script for each test project.
The post build script thankfully doesn't need to vary per project. In fact, it doesn't need to vary at all assuming NuGet puts nunit.consolerunner into the expected place. The script will work in both debug and release configurations because the TargetDir and TargetFileName properties from VS are passed down to the console runner as params. Here is the script, you just need to paste it into the post-build section of build events in the properties pane of the respective project:
cd %HOMEPATH%
cd ".nuget\packages\nunit.consolerunner\3.10.0\tools"
nunit3-console.exe $(TargetDir)$(TargetFileName) --where "cat == Unit" --work=$(TargetDir)\..
The various parts of this can be explained as follows:
cd %HOMEPATH% is done because nuget usually installs to your user directory
cd ".nuget\packages\nunit.consolerunner\3.10.0\tools Assuming .nuget exists in your user directory, this will move into the location of the nuget console runner .exe file
nunit3-console.exe This is the exe file that runs your tests
$(TargetDir)$(TargetFileName) This is the full path to the DLL containing your tests, created using VS variables in the properties window
--where "cat == Unit" This is an optional restriction we used to restrict our tests run to a category of unit tests. You can omit this - it's just to show you how you can restrict to categories if you want to. See the docs (link below) for more. Note: the category "Unit" was our own annotation we added. It's not something out of the box. So if you actually add this restriction without annotating your tests, nothing will run!
--work=$(TargetDir)\.. This puts the TestResult.xml file in the parent folder of the DLL for the project you're building. For VS2019, at the moment anyway, this directory is the bin folder under the main project folder. We used this because we needed to redirect test output. Otherwise multiple projects in the same solution were trying to write to the same default file and were failing due to concurrency issues.
Links
Nunit console runner docs: https://github.com/nunit/docs/wiki/Console-Command-Line
Update: A Less Annoying Way
The above is pretty annoying, because if any of your tests fail, and you then try to run them from VS's integrated test runner, you can't, because you end up in a vicious cycle: running the tests kicks off a build, which kicks off the tests on the console, which fail, which cause your tests in VS not to run. So in order to debug, you have to go rooting around for the results.XML file. Ugh.
A better way is to do the following:
Create a new project (I called it TestMaster)
Make this project depend on all other projects you care about
Add a post-build script which calls all the nunit tests for all your tests projects
We have five test projects. Let's say they're called TestProject1, TestProject2, ... etc. Then this batch file will serve as the post-build script, if it is put in any sub-directory of the main solution directory. It assumes you have Nuget v3.11.1 installed. If not, just change that for whatever version you do have:
#echo off
rem Debug or release
set CONFIG=%1
set HOMEPATH=%2
cd ../
echo Running unit tests for soln. in root dir %cd%
set currentTest=TestProject1
call :RUN_TESTS_FOR_PROJECT
set currentTest=TestProject2
call :RUN_TESTS_FOR_PROJECT
set currentTest=TestProject3
call :RUN_TESTS_FOR_PROJECT
set currentTest=TestProject4
call :RUN_TESTS_FOR_PROJECT
set currentTest=TestProject5
call :RUN_TESTS_FOR_PROJECT
echo Finished running tests
goto :EOF
:RUN_TESTS_FOR_PROJECT
cd %currentTest%
cd bin/%CONFIG%
echo --- Running tests in directory %cd%
set targetDir=%cd%
cd %HOMEPATH%
cd ".nuget\packages\nunit.consolerunner\3.11.1\tools"
set targetFileName=%currentTest%.dll
nunit3-console.exe %targetDir%\%targetFileName% --where "cat == Unit" --work=%targetDir%\..
if errorlevel 1 (
echo Tests Failed for %currentTest%
echo Exiting test run early
exit %errorlevel%
)
cd %targetDir%
cd ../../../
goto :EOF
:EOF
Let's say you put the above script in a folder called Util. The post-build script of your TestMaster project is then:
cd $(SolutionDir)
cd Util
RunTests $(ConfigurationName) %HOMEPATH%
Good evening,
I am using the .Net Core 2.0 version from here https://docs.sonarqube.org/display/SCAN/Analyzing+with+SonarQube+Scanner+for+MSBuild on a 2.1 project in Jenkins with:
withSonarQubeEnv('SonarQubeMain') {
bat "dotnet ${globals.SONAR_QUBE_MSBUILD_PATH}\\SonarScanner.MSBuild.dll begin /k:\"${globals.SONAR_QUBE_PROJECT}\" /d:sonar.host.url=${globals.SONAR_HOST_URL} /d:sonar.cs.xunit.reportsPaths=\"XUnit.xml\" /d:sonar.cs.opencover.reportsPaths=\"coverage.xml\"
}
bat "dotnet build --version-suffix ${env.BUILD_NUMBER}"
dir('test/mytestprojecthere') {
bat 'D:\\OpenCover\\OpenCover.Console.exe -target:"c:\\Program Files\\dotnet\\dotnet.exe" -targetargs:"xunit --no-build -xml XUnit.xml" -output:coverage.xml -oldStyle -filter:"-[*Tests*]*" -register:user'
}
withSonarQubeEnv('SonarQubeMain') {
bat "dotnet ${globals.SONAR_QUBE_MSBUILD_PATH}\\SonarScanner.MSBuild.dll end"
}
It works the first build but on the next build it fails with:
Failed to create an empty directory 'D:\Jenkins\workspace\xxxxxxxx\.sonarqube'.
Please check that there are no open or read-only files in the directory and that you have the necessary read/write permissions.
Detailed error message: Access to the path 'SonarScanner.MSBuild.Common.dll' is denied.
and checking my windows server I can see multiple .Net Core Host Background process. If I kill these I can build again..
I readed about msbuild /nodereuse:false for MSBuild but seems is not working for the dotnet core version?
We were just faced with this issue, and found out that it's related to dotnet's and msbuild's reuse of nodes that were left running by a previous multi-threaded build.
To avoid the problem, use either /nodereuse:false or /nr:false on your command line as the following:
msbuild /m /nr:false myproject.proj
msbuild /m /nodereuse:false myproject.proj
dotnet restore myproject.sln /nodereuse:false
FYI #Nauzet opened an issue open for this in the Scanner for MSBuild repo: #535.
To summarise:
the begin and end steps seem to run fine, and dotnet.exe shuts down as expected for those processes
when building a complex solution, multiple instances of dotnet.exe are started and are do not shut down immediately when the build completes. The issue does not seem to occur on simpler solutions.
the issue occurs if you trigger the build phase using dotnet build or dotnet msbuild
workarounds: build using msbuild directly, or build using dotnet build /nodereuse:false
FYI the Scanner for MSBuild has a couple of custom tasks that are called during the build phase. These use the assembly that is being locked. There's nothing unusual about the custom tasks; they just read data from a file on disk. At this point, I'm not convinced it's an issue with the Scanner for MSBuild.
I faced same issue and running following command solved the issue.
dotnet build-server shutdown
This happens due to msbuild node re-use. we leave msbuild nodes alive with the intent of saving on startup time when doing consecutive builds.
It can turned off by setting an following environment variable too.
MSBUILDDISABLENODEREUSE=1
Please edit your pipeline script as shown below and it should work properly:
withSonarQubeEnv('SonarQubeMain') {
bat "dotnet ${globals.SONAR_QUBE_MSBUILD_PATH}\\SonarScanner.MSBuild.dll begin /k:\"${globals.SONAR_QUBE_PROJECT}\" /d:sonar.host.url=${globals.SONAR_HOST_URL} /d:sonar.cs.xunit.reportsPaths=\"XUnit.xml\" /d:sonar.cs.opencover.reportsPaths=\"coverage.xml\"
bat "dotnet build --version-suffix ${env.BUILD_NUMBER}"
dir('test/mytestprojecthere') {
bat 'D:\\OpenCover\\OpenCover.Console.exe -target:"c:\\Program Files\\dotnet\\dotnet.exe" -targetargs:"xunit --no-build -xml XUnit.xml" -output:coverage.xml -oldStyle -filter:"-[*Tests*]*" -register:user'
}
bat "dotnet ${globals.SONAR_QUBE_MSBUILD_PATH}\\SonarScanner.MSBuild.dll end"
}
UPDATE
Here is a dotnet core app pipeline build script that I use, which works well without any issues:
bat "dotnet ${sqScannerMsBuildHome}\\SonarScanner.MSBuild.dll begin /k:yoursonarprojectkey /n:yoursonarprojectname /v:1.0 /d:sonar.host.url=%SONAR_HOST_URL%"
bat 'dotnet build'
bat "dotnet ${sqScannerMsBuildHome}\\SonarScanner.MSBuild.dll end"
I just ran into it myself and "solved" it by running dotnet build-server shutdown as the first task in my build plan. That's not ideal for multiple reasons, one big one being that it'll likely cause issues if I attempt to run multiple .NET Core builds at once on the same machine. This does seem to be a bug in the scanner -- hopefully it'll be fixed soon.
In our solution we have more than 150 projects, I need build one of project before others, but I did not want to set project reference or project dependencies for more than 100 projects one by one. This is a waste of time. So I ask following question before this one:
Set build order without using project reference
And as #Leo Liu suggested that I could use batch file to build the project first, it is should be a valid workaround. As optimization, I want set this bat file in the build event of that project so that I only need to build that project, do not need to manually run the bat file from outside.
However, when I build that project with below build event:
call "$(ProjectDir)\MyTest.bat"
I got this error:
C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\15.0\bin\Microsoft.Common.CurrentVersion.targets(5074,5): error MSB3073: The command "call "C:\Users\Myname\Source\repos\MysolutionName\GenerateFile\MyTest.bat"" exited with code 1. [C:\Users\\Source\repos\MysolutionName\GenerateFile\GenerateFile.csproj]
Update:
The scripts in the bat file is:
#echo OFF
call "C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\Tools\VsDevCmd.bat"
echo "Starting Build for all Projects with proposed changes"
MSBuild.exe "C:\Users\MyName\Source\repos\MySolutionName\MySolutionName.sln"
pause
echo "All builds completed."
Any suggestion? Thanks in advance.
Execute bat file in the build event got the error “error MSB3073: The command ”call “MyTest.bat” exited with code 1."
According to your question, I created a test sample and got more than 30 errors without any other detailed info:
error MSB3073: The command "call "D:\TestSample.bat"" exited with code 1.
Further study found the reason for those errors, that because you build the solution in the bat file, then you set this bat file in the build event of one project in the solution, which called in the bat file.
So an infinite loop will generated, that is the reason why we got more than 30 errors for one command line in the build event.
To resolve this issue, we should not build the specify project when we build the solution with bat file, so you could open Configuration Manager, uncheck the build checkbox for generate project:
With this setting, this project will be ignore when you build the solution with that bat file.
I'm using Visual Studio Online for CI. I have a Release build set up. One of my projects has a pre build step which should only be executed for Debug builds.
if $(ConfigurationName) == Debug copy "$(ProjectDir)Config\web.$(Username).config" "$(ProjectDir)\runtime.config"
However VSO fails the build with an error that the command exited with a status of 1.
The same build runs fine when executed using Visual Studio on a developers machine.
The issue was down to needing brackets on the condition check. Therefore the following works on VSO.
if ($(ConfigurationName) == Debug) copy "$(ProjectDir)Config\web.$(Username).config" "$(ProjectDir)\runtime.config"
The brackets are not required for VS2013.
The command must be between parentheses, because is parsed by CMD.EXE
if $(ConfigurationName) == Debug (
copy "$(ProjectDir)Config\web.$(Username).config" "$(ProjectDir)\runtime.config"
)
See "if".
I have set my Visual Studio to start Nunit as an external program to run all the tests written in a module.
Now what I am trying to do is to create a batch file which will call Myproj.exe. What I am expecting is that it will run Nunit as I have set it to run an external program and execute all my tests in nunit.exe, but when I run that batch file it starts running from Visual Studio instead of opening NUnit.
Can any one please give me a clear idea as how to accomplish it?
I am too much stuck.
Now I am trying to run the following commands in shell
nunit-x86.exe
Can you please tell how should I load my visualbasic project file (exe) here and then run all the tests from here
as unable to execute following command
nunit nunit.tests.vbproj /config:release
You can make NUnit start everytime you debug your "NUnit tests".
You can attach the debugger in Visual Studio Express doing it that way.
If you use a "full version" of VS do it that way:
Note that if you’re using the full and
not the express version of Visual
Studio 2005, you can do this by
opening up the project’s properties,
and in the Debug tab select Start
External Program: and navigate to the
NUnit executable, and set
YourCompanyname.YourProject.Test.dll as the
Command Line Arguments.
I got that ideas from this tutorial(Page 4/5) and love it.
You can also run NUnit after every successful build in Visual Studio with a Post-Build Event.
In VS2005, right-click on the project that has your tests and select Properties. Then on the Build Events tab, in the "Post-build event command line", put this* to use the console:
nunit-console /xml:$(ProjectName).xml $(TargetPath)
or this to use the GUI::
nunit $(TargetPath) /run
In "Run the post-build event:", leave the default: "On successful build"
If you use the GUI, know that your build will appear to be hung up until you close the GUI app.
*NOTE: The nunit console command line docs say "By default the nunit-console program is not added to your path."
you can just shell nunit.exe with the command line to your assembly to run tests in.
You can load nUnit.exe (nUnit-Console.exe for command line execution) using external tool features in Visual studio. Once you add the command via external tools feature (as explained in this blog), you can use it for any project to execute the tests. (Other is to add through project properties but that needs to be done for every project). Also in the arguments you can pass /include or /exclude to include or exclude categories of the tests.
The advantage of this method is you need not worry about giving path to DLL file and it works for any project you are on and gets executed with few clicks