How do I output NUnit formatted xml with XUnit console? - xunit

I need NUnit output from XUnit tests for a report. With NUnit I can do:
nunit-console.exe /xml=c:\TestResultN.xml MyDll.dll
I've tried:
xunit.console MyDll.dll /nunit TestResults.xml
but I get:
unknown output transform: nunit
Any good docs on xunit console? I can't find any info.

Type xunit.console.exe /? in a console and look at the end of the option list.
> xunit.console.exe /?
xUnit.net console test runner (64-bit .NET 2.0.50727.4952)
Copyright (C) 2007-10 Microsoft Corporation.
usage: xunit.console <xunitProjectFile> [options]
usage: xunit.console <assemblyFile> [configFile] [options]
Valid options:
/silent : do not output running test count
/teamcity : forces TeamCity mode (normally auto-detected)
/wait : wait for input after completion
Valid options for assemblies only:
/noshadow : do not shadow copy assemblies
/xml <filename> : output results to Xunit-style XML file
/html <filename> : output results to HTML file
/nunit <filename> : output results to NUnit-style XML file
The last two options are extracted from the config file and consist of a xslt file applied to the output of xunit. If /nunit is not present, make sure that you have the NUnitXml.xslt file in the xunit directory and the entry nunit is declared in the xunit.console.exe.config file.
My xunit.console.exe.config file :
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<configSections>
<section name="xunit" type="Xunit.ConsoleClient.XunitConsoleConfigurationSection, xunit.console"/>
</configSections>
<xunit>
<transforms>
<add
commandline="html"
xslfile="HTML.xslt"
description="output results to HTML file"/>
<add
commandline="nunit"
xslfile="NUnitXml.xslt"
description="output results to NUnit-style XML file"/>
</transforms>
</xunit>
</configuration>
If you're not able to find theses files, you might need to reinstall xunit.

First follow this link.
https://xunit.net/docs/getting-started/netfx/visual-studio
then open cmd from your main project folder as described in above link.
open file location of xunit.console.exe
run below command in cmd :
....\packages\xunit.runner.console.2.4.1\tools\net472\xunit.console.exe Project.UnitTests\bin\Debug\Project.UnitTests.dll -xml test_ouput.xml
Output result "test_ouput.xml" will be saved in project directory.

Related

How do I force msbuild to create SARIF Files on CodeAnalysis

If I run the code analysis in Visual Studio 2022 (on a c++ project) I get a XML and a SARIF file for every code file.
No I try to run the code analysis with MSBuild 2022:
MSBuild.exe solution.sln -p:Configuration=Release /p:RunCodeAnalysis=true
But with this call I only get the code analysis XML files and no SARIF files.
Any idea how to force MSBuild to create the SARIF files?
Try to use following command line:
cl.exe <file/project path> /analyze:autolog:ext .nativecodeanalysis.sarif
Or
cl.exe <file/project path> /analyze:autolog:ext .sarif
Though MSBuild.exe invokes cl.exe to compile, it seems creating a .sarif file is only available for directly using cl.exe and its command.
Here’s the related document: Analysis log options
/analyze:autolog:ext extension
Overrides the default extension of the analysis log files, and uses extension instead. If you use the .sarif extension, the log file uses the SARIF format instead of the default XML format.
https://docs.microsoft.com/en-us/answers/questions/512275/what-to-do-with-static-code-analysis-result-xml-fi.html describes a solution:
Add a Directory.build.props file to your Visual Studio solution:
<?xml version="1.0" encoding="utf-8"?>
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<ItemDefinitionGroup>
<ClCompile>
<AdditionalOptions>$(ClOptions) %(AdditionalOptions)</AdditionalOptions>
</ClCompile>
</ItemDefinitionGroup>
</Project>
Now I can extend my MSBuild Command line on my CI-Server (TeamCity):
/p:RunCodeAnalysis=true /p:ClOptions="/analyze:log%20MyApp.nativecodeanalysis.combined.sarif" (I had to replace the whitespace with %20).
And one SARIF file is generated, or if you want one SARIF file for every code file:
/p:RunCodeAnalysis=true /p:CaOptions="/analyze:log:format:sarif"
If you want to add additional command line switches you have to separate it with %20:
/p:CaOptions=/analyze:log:format:sarif%20/analyze:log:compilerwarnings
BUT: If I activate Clang-Tidy in my Visual Studio project I get the error CLANGTIDY : error : no such file or directory: '/analyze:log' [clang-diagnostic-error] and CLANGTIDY : error : unable to handle compilation, expected exactly one compiler job in ... - Does someone has an idea about that (except disabling Clang-Tidy)?

Running dotCover Console Runner with MsTest

I partially found a solution to mstest with arguments:
https://devnet.jetbrains.com/thread/471980
Command Prompt (Works ok):
dotcover analyse /TargetExecutable="c:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\MsTest.exe" /TargetArguments="/testcontainer:c:\MyApps\Modules\UnitTests\bin\Debug\UnitTests.dll" /Output="C:\temp\AppCoverageReport.html" /ReportType="HTML"
However when using XML I get parse error:
<?xml version="1.0" encoding="utf-8"?>
<AnalyseParams>
<TargetExecutable>c:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\MsTest.exe</TargetExecutable>
<TargetArguments>/testcontainer:c:\MyApp\Modules\UnitTests\bin\Debug\UnitTests.dll</TargetArguments>
<Output>C:\Temp\AppCoverageReport.html</Output>
<ReportType>html</ReportType>
</AnalyseParams>
What am i doing wrong?
Turns out I was mistyping the name of the xml file (type)
dotcover analyse c:\Temp\coverangeconfig.xml => extra 'n'
instead of:
dotcover analyse c:\Temp\coverageconfig.xml
So, to use dotCover with MsText.exe the following work:
From command prompt
From XML file (coverageconfig.xml)
for both: see code above.
Make sure MsTest attribute "/testcontainer" is under /TargetArguments tag

How to set dll search path of MSTest used with opencover?

My unit tests have dependencies which are from a pool of dlls in a directory outside of the project. We have the policy to generally place links to these dlls (from the pool) in a directory called "System" inside the project directory and set their CopyToOutputDirectory to PreserveNewest. Afterwards, in the project we add a reference to the dll in the pool but set it to be not a private copy which means it will not be output to the build dir. In the InitAssembly() method we have setup that the runtime searches also in the AppDomain.CurrentDomain.BaseDirectory, "System") by the means of AppDomain.CurrentDomain.AssemblyResolve.
[TestClass]
public class SomeUnitTest : OurUnitTestBase
{
[AssemblyInitialize]
public static void AssemblyInit(TestContext context)
{
Debug.WriteLine("in AssemblyInit");
base.InitAssembly();
}
}
This works nice when running our code from inside VisualStudio and also when running the unit tests from inside VisualStudio's Test-Explorer (vstest.executionengine.x86.exe). However MSTest fails to find the dlls in the System subdirectory when run from the following batch script which executes a coverage analysis with opencover:
REM Bring dev tools into the PATH.
call "C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\Tools\VsDevCmd.bat"
mkdir .\CoverageAnalysisReport
REM Run unit tests through OpenCover
REM This allows OpenCover to gather code coverage results
.\..\..\Tools\CoverageAnalysis\opencover\OpenCover.Console.exe^
-register:user^
-target:MSTest.exe^
-targetargs:"/noresults /noisolation /testcontainer:..\bin\Debug\OurUnitTests.dll"^
-filter:+[*]*^
-output:.\CoverageAnalysisReport\output.xml
REM Generate the report
.\..\..\Tools\CoverageAnalysis\ReportGenerator\bin\ReportGenerator.exe^
-reports:.\CoverageAnalysisReport\output.xml^
-targetdir:.\CoverageAnalysisReport^
-reporttypes:HTML^
-filters:-OurUnitTests*
REM Open the report
start .\CoverageAnalysisReport\index.htm
The resulting log file says, that none of the unit tests could be run and failed therefore. Furthermore it states
Warning: The assembly "BaseLib" as a direct or indirect dependency of the test container "C:\MyProject\bin\debug\ourunittests.dll" was not found.
Yet I know [AssemblyInitialize] was called because a System.Windows.Form.MessageBox.Show(..) I placed there temporarily actually got displayed during the execution of the bat-file.
Is there a way to tell MSTest to search also for dlls in the System subdirectory also?
I solved it by placing the following as the content of a file named Local.testsettings in the same directory like the batch file (in which I modified the line -targetargs:"/testcontainer:\"..\bin\x86\Debug\OurUnitTests.dll\" /testSettings:\".\Local.testsettings\"):
<?xml version="1.0" encoding="UTF-8"?>
<TestSettings name="Lokal" id="2fa4344c-1f2f-4a04-86f3-41d223b10333" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
<Description>Dies sind die standardmäßigen Testeinstellungen für einen lokalen Testlauf.</Description>
<Deployment>
<DeploymentItem filename="..\bin\Debug\System\" />
</Deployment>
<Execution>
<TestTypeSpecific>
<UnitTestRunConfig testTypeId="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b">
<AssemblyResolution>
<TestDirectory useLoadContext="true" />
</AssemblyResolution>
</UnitTestRunConfig>
</TestTypeSpecific>
<AgentRule name="Execution Agents">
</AgentRule>
</Execution>
</TestSettings>

Running MsTest from the command line with a custom assembly base directory

I did quite a lot of research on the web and tried a few settings, but I couldn't reproduce the behavior of running MsTest in Visual Studio 2012 on the command line.
Our solution consists of many projects that build to the same bin folder residing at the solution level (e.g. C:\MySolution\bin) - this is the code-under-test (CUT). The tests are grouped in a separate project that resides in its own solution and builds in its own bin folder (e.g. C:\MySolution\Tests\bin). There are really a lot of plugins thus we want MsTest to reference the CUT bin folder when running the test intead of copying everything to the TestResults folder. We did achieve this in Visual Studio 2012 by editing the .testrunconfig and specifying ".\bin" as "root folder for the assemblies to be loaded" (in tab "Unit Test" when editing the testrunconfig). So we can load the test solution in VS2012 and run the tests there without having to copy the bin folder contents to the TestResults directory.
Now I wanted to create a .bat file that would run MsTest the same way as in VS2012 so that we can omit launching Visual Studio just for running the tests. I'm now working on how to execute MsTest on the command line, but have been quite frustrated. This is what I tried (command executed on the solution level in a VS command prompt):
MsTest /testcontainer:Tests\bin\Tests.dll
This did not work at all, it couldn't even find the referenced dlls that Tests.dll need to run. So I re-used the configuration and ran it again:
MsTest /runconfig:LocalTestRun.testrunconfig /testcontainer:Tests\bin\Tests.dll
Still it did not work. It could start the tests, but all of them failed. I got a lot of warnings of the kind
Warning: Test Run deployment issue: The assembly or module '....' directly or indirectly referenced by the test container 'C:\MySolution\Tests\bin\tests.dll' was not found.
and in the end it said:
The configured application base directory 'C:\MySolution\TestResults\User_Machine 2013-07-28 13_16_59\Out\bin' does not exist. The test directory will be used instead.
When I changed the option applicationBaseDirectory in the testrunconfig to an absolute path (C:\MySolution\bin), it worked. Still I get many warnings such as:
Warning: Test Run deployment issue: The assembly or module '....' directly or indirectly referenced by the test container 'C:\MySolution\Tests\bin\tests.dll' was not found.
But anyway, it's not really a feasible solution to specify an absolute path. How can I run MsTest on the command line with a different, but relative assembly base directory?
My LocalTestRun.testrunconfig is as follows:
<?xml version="1.0" encoding="UTF-8"?>
<TestSettings name="Local Test Run" id="...." xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
<Description>This is a default test run configuration for a local test run.</Description>
<Deployment>
<DeploymentItem filename="Tests\....\Resources\" />
</Deployment>
<Execution hostProcessPlatform="MSIL">
<TestTypeSpecific>
<UnitTestRunConfig testTypeId="....">
<AssemblyResolution applicationBaseDirectory=".\bin">
<TestDirectory useLoadContext="true" />
</AssemblyResolution>
</UnitTestRunConfig>
<WebTestRunConfiguration testTypeId="....">
<Browser name="Internet Explorer 7.0">
<Headers>
<Header name="User-Agent" value="Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)" />
<Header name="Accept" value="*/*" />
<Header name="Accept-Language" value="{{$IEAcceptLanguage}}" />
<Header name="Accept-Encoding" value="GZIP" />
</Headers>
</Browser>
</WebTestRunConfiguration>
</TestTypeSpecific>
<AgentRule name="LocalMachineDefaultRole">
</AgentRule>
</Execution>
</TestSettings>
After more searching we changed to use the test console runner that is included with VS2012:
VSTest.Console.exe Tests\bin\Tests.dll /Framework:framework40 /Settings:LocalTestRun.testrunconfig
This works with a relative path as applicationBaseDirectory.
This is due to an MSTest bug that sets the current directory to its own working directory, rather than the test project's bin (or deployment) folder. The workaround is to execute the following code in the constructor of your test class:
Environment.CurrentDirectory(AppDomain.CurrentDomain.BaseDirectory);
I got the idea from http://www.ademiller.com/blogs/tech/2008/01/gotchas-mstest-appdomain-changes-in-vs-2008/; however, note that, in my case at least, it required setting Environment.CurrentDirectory, rather than the reverse, as suggested in the article.

Automate VS 2010 "Publish" Config File Substitutions

I'm using the config file replacement feature of Visual Studio 2010's "Publish" functionality, as described in this article. I want to automate this using MSBuild/Hudson. Does anybody know how to do this?
I like how it works but if I cannot automate it I'll have to switch to XmlMassUpdate or similar.
Explanation
To transform your config file you'll have to execute the TransformWebConfig target.
This target takes two files Web.config and Web.$(Configuration).config and generates a Web.config. The generated file is the transformed version of the original one for the current configuration.
This file is generated in folder : obj\$(Configuration)\TransformWebConfig
Usage
You don't really explain what you want to achieve, so here a basic usage, a job that generates a transformed config file in a given folder.
Add the following piece in the end of your project file *.csproj after the import of Microsoft.WebApplication.targets
<PropertyGroup>
<!-- Directory where your web.config will be copied -->
<TransformedWebConfigDestination>$(MSBuildProjectDirectory)</TransformedWebConfigDestination>
</PropertyGroup>
<!--
This target transforms the web.config based on current configuration and
put the transformed files in $(TransformedWebConfigDestination) folder
-->
<Target Name="ConfigSubstitution">
<CallTarget Targets="TransformWebConfig"/>
<ItemGroup>
<TransformedWebConfig Include="obj\$(Configuration)\TransformWebConfig\Web.config"/>
</ItemGroup>
<!-- Copy the transformed web.config to the configured destination -->
<Copy SourceFiles="#(TransformedWebConfig)"
DestinationFolder="$(TransformedWebConfigDestination)"/>
</Target>
In Hudson you could add a Build step in your build, or create a dedicated job configured as follow:
MsBuild Build File : Your csproj file.
Command Line Arguments : /t:ConfigSubstitution /p:Platform=AnyCpu;Configuration=Test;TransformedWebConfigDestination=DestinationFolder
Edit your web project.csproj
under
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|AnyCPU' ">
Add -
<UseMsDeployExe>True</UseMsDeployExe>
Look at the Build output (make sure VS Tools - Options - Project & Solutions -Build & Run - MSBuild Output Verbosity - Detailed)
You should be able to see the msdeploy commands VS uses to produce the package. It's my understanding that VS actually uses Web Platform Pipeline API's and .target files to actually produce the deploy packages when building using MSBuild, and this command changes to use MsDeploy instead.
This stuff is so in need of documentation, its very frustrating.
I am using this in Hudson to target Release:
/Property:Configuration=Release
The exact settings are:
Build
MSBuild Version: msbuild-4 (configured to point to v4 msbuild)
MsBuild Build File: project_name.sln
Command Line Arguments: /Property:Configuration=Release
You can test this in your project directory by running something similar (as your .NET framework version may differ) to this:
%SYSTEMROOT%\Microsoft.NET\Framework\v4.0.30319\msbuild project.sln /Property:Configuration=Release

Resources