DeploymentEnabled false is not working in my runsettings file - visual-studio

I currently updated to visual studio 2017 enterprise edition version 15.9.24 .
The issue that i'm running across is i'm not able to debug my test using the
runsettings file. When I debug the test its access any dlls and deployed files from my TestResults Out folder instead of bin/debug folder since i'm in debug mode. When i was running Visual studio 2015 I was using .testsettings file
and was able to run my test from the bin/debug folder. Since .testsettings is
going to be deprecated i decided to move to runsettings.
Searching the web and stack someone stated that i needed to have the settings located in the runsettings file that is labeled DeploymentEnabled to false to achieve this. I did this and still getting error because the application is running and trying to access the files from my TestResults Out folder. I also changed the run settings to forced legacy mode to true and added directory of my old test settings file but still not working.
Is there another setting that i'm not taken into consideration
or am i missing something or is something wrong with my run settings.
Please see my example of my current runsettings file below
Thanks for any help or direction given.
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
<!-- Configurations that affect the Test Framework -->
<RunConfiguration>
<MaxCpuCount>1</MaxCpuCount>
<!-- Path relative to directory that contains .runsettings file
<ResultsDirectory>.\TestResults</ResultsDirectory> -->
<!-- x86 or x64 -->
<!-- You can also change it from the Test menu; choose "Processor Architecture for AnyCPU Projects" -->
<TargetPlatform>x86</TargetPlatform>
<!-- Framework35 | [Framework40] | Framework45 -->
<TargetFrameworkVersion>Framework45</TargetFrameworkVersion>
<!-- Path to Test Adapters -->
<!--<TestAdaptersPaths>%SystemDrive%\Temp\foo;%SystemDrive%\Temp\bar</TestAdaptersPaths>-->
<!-- TestSessionTimeout was introduced in Visual Studio 2017 version 15.5 -->
<!-- Specify timeout in milliseconds. A valid value should be greater than 0 -->
<TestSessionTimeout>15000000</TestSessionTimeout>
</RunConfiguration>
<!-- Configurations for data collectors -->
<DataCollectionRunSettings>
<DataCollectors>
<DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
<Configuration>
<CodeCoverage>
<ModulePaths>
<Exclude>
<ModulePath>.*CPPUnitTestFramework.*</ModulePath>
</Exclude>
</ModulePaths>
<!-- We recommend you do not change the following values: -->
<UseVerifiableInstrumentation>True</UseVerifiableInstrumentation>
<AllowLowIntegrityProcesses>True</AllowLowIntegrityProcesses>
<CollectFromChildProcesses>True</CollectFromChildProcesses>
<CollectAspDotNet>False</CollectAspDotNet>
</CodeCoverage>
</Configuration>
</DataCollector>
</DataCollectors>
</DataCollectionRunSettings>
<!-- Parameters used by tests at run time -->
<TestRunParameters>
<Parameter name="webAppUrl" value="http://localhost" />
<Parameter name="xxxxUser" value="xxxxm,xxxxtAuto01,xxxxtAuto02,xxxxxtAuto03,xxxxtAuto04" />
<Parameter name="xxxxxxxxxnbr" value="xx86,xx90,xx91,xx92,xx93" />
</TestRunParameters>
<!-- MSTest adapter -->
<MSTest>
<MapInconclusiveToFailed>true</MapInconclusiveToFailed>
<CaptureTraceOutput>false</CaptureTraceOutput>
<DeleteDeploymentDirectoryAfterTestRunIsComplete>true</DeleteDeploymentDirectoryAfterTestRunIsComplete>
<DeploymentEnabled>false</DeploymentEnabled>
<!---->
<ForcedLegacyMode>true</ForcedLegacyMode>
<SettingsFile>%xxxx_Automation_Test\xxxx_Automation_Test\xxx_Automation_Test\xxxxAutomation.testsettings;</SettingsFile>
<InProcMode>true</InProcMode>
<KeepExecutorAliveAfterLegacyRun>False</KeepExecutorAliveAfterLegacyRun>
<AssemblyResolution>
<!--
<Directory path="C:\xxx_Automation\xxxx_Automation_Test\xxx_Automation_Test\bin\" includeSubDirectories="false"/>-->
</AssemblyResolution>
</MSTest>
</RunSettings>

I updated my runsettings to have the file name of of run settings and to
update the ForcedLegacyMode to true.
<ForcedLegacyMode>true</ForcedLegacyMode>
<SettingsFile>xxxxxAutomation.testsettings</SettingsFile>
<KeepExecutorAliveAfterLegacyRun>False</KeepExecutorAliveAfterLegacyRun>
<AssemblyResolution>

Related

Brackets are not included as Code Coverage for Unit Tests c# in VSO

There are some code coverage inconsistencies between the VSO build pipeline and Visual Studio local. We found that the code coverage in Visual Studio is higher because it counts brackets as part of the code coverage. But it doesnt count in VSO build pipeline for some reason. Anyone knows why? Is this a known bug in VSO?
We use Xunit and c# for the tests and have test.runsettings to exclude code. Below is the test.runsettings file
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
<!-- Configurations that affect the Test Framework -->
<RunConfiguration>
<!-- Path relative to directory that contains .runsettings file-->
<ResultsDirectory>.\TestResults</ResultsDirectory>
<TargetPlatform>x64</TargetPlatform>
<MaxCpuCount>10000</MaxCpuCount>
</RunConfiguration>
<!-- Configurations for data collectors -->
<DataCollectionRunSettings>
<DataCollectors>
<DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
<Configuration>
<CodeCoverage>
<Attributes>
<Exclude>
<Attribute>^System.Diagnostics.CodeAnalysis.ExcludeFromCodeCoverageAttribute$</Attribute>
</Exclude>
</Attributes>
<ModulePaths>
<Exclude>
<ModulePath>.*unittests.dll$</ModulePath>
<ModulePath>.*integrationtests.dll$</ModulePath>
<ModulePath>.*xunit.*.dll</ModulePath>
<ModulePath>.*fluentassertions.*$</ModulePath>
<ModulePath>.*moq.*$</ModulePath>
<ModulePath>.*telemetry.extensions.ai.netcore.*$</ModulePath>
<ModulePath>.*newtonsoft.*$</ModulePath>
<ModulePath>.*codeanalysis.*$</ModulePath>
</Exclude>
</ModulePaths>
<!-- We recommend you do not change the following values: -->
<UseVerifiableInstrumentation>True</UseVerifiableInstrumentation>
<AllowLowIntegrityProcesses>True</AllowLowIntegrityProcesses>
<CollectFromChildProcesses>True</CollectFromChildProcesses>
<CollectAspDotNet>False</CollectAspDotNet>
</CodeCoverage>
</Configuration>
</DataCollector>
</DataCollectors>
</DataCollectionRunSettings>
</RunSettings>
The reason why it's showing inconsistent code coverage is because in Visual Studio, I was running Debug mode instead of Release mode. After changing it to Release mode, the number is consistent to the VSO number.

MS Test settings > TeamCity > 64bit

I've configured my .testsettings file as per instructions to run Hosts in 64bit.
But I don't see anything actually written to the settings file.
<?xml version="1.0" encoding="UTF-8"?>
<TestSettings name="accounting" id="907240cb-5a90-45f4-a41f-2e73533c8c25" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
<Description>These are default test settings for a local test run.</Description>
<Deployment enabled="false" />
<Execution>
<Timeouts runTimeout="1800000" testTimeout="40000" />
<TestTypeSpecific>
<UnitTestRunConfig testTypeId="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b">
<AssemblyResolution>
<TestDirectory useLoadContext="true" />
</AssemblyResolution>
</UnitTestRunConfig>
</TestTypeSpecific>
<AgentRule name="LocalMachineDefaultRole">
</AgentRule>
</Execution>
<Properties />
</TestSettings>
And so my build server (Teamcity) continues to build in 32bit.
i've also tried setting it in VS but that definitely doesn't get persisted anywhere for Teamcity to know
how do i force Teamcity to use 64bit ?

VS 2015 cannot open .coveragexml generated by CodeCoverage.exe

I am following Code Coverage Results Import (C#, VB.NET) documentation for SonarQube.
For my project, I am able to generate test coverage binary file as "SonarQubeTests.csproj.coverage"
Visual Studio can open this file with no problem. I can see all the details.
Then when I try to generate ".coveragexml" file with following command;
"C:\Program Files (x86)\Microsoft Visual Studio 14.0\Team Tools\Dynamic Code Coverage Tools\CodeCoverage.exe" "analyze /output:C:\SonarQubeTests.csproj.coveragexml C:\SonarQubeTests.csproj.coverage"
It does generate ".coveragexml" file, but when I try to open Visual Studio it gives me following error:
I can open this file with text editor and content seems all fine. But neither Visual Studio 2015 nor sonarqube does not read this file.
What I am missing here?
P.S I have also runner settings for Ms Test runner as below.
Not sure if this will effect something;
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
<!-- Configurations that affect the Test Framework -->
<RunConfiguration>
<MaxCpuCount>1</MaxCpuCount>
<!-- Path relative to solution directory -->
<ResultsDirectory>C:\</ResultsDirectory>
<!-- [x86] | x64
- You can also change it from menu Test, Test Settings, Default Processor Architecture -->
<TargetPlatform>x86</TargetPlatform>
<!-- Framework35 | [Framework40] | Framework45 -->
<TargetFrameworkVersion>Framework45</TargetFrameworkVersion>
<!-- Path to Test Adapters -->
<TestAdaptersPaths>%SystemDrive%\Temp\foo;%SystemDrive%\Temp\bar</TestAdaptersPaths>
</RunConfiguration>
<!-- Configurations for data collectors -->
<DataCollectionRunSettings>
<DataCollectors>
<DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
<Configuration>
<CodeCoverage>
<ModulePaths>
<Exclude>
<ModulePath>.*CPPUnitTestFramework.*</ModulePath>
</Exclude>
</ModulePaths>
<!-- We recommend you do not change the following values: -->
<UseVerifiableInstrumentation>True</UseVerifiableInstrumentation>
<AllowLowIntegrityProcesses>True</AllowLowIntegrityProcesses>
<CollectFromChildProcesses>True</CollectFromChildProcesses>
<CollectAspDotNet>False</CollectAspDotNet>
</CodeCoverage>
</Configuration>
</DataCollector>
</DataCollectors>
</DataCollectionRunSettings>
<!-- Parameters used by tests at runtime -->
<TestRunParameters>
<Parameter name="webAppUrl" value="http://localhost" />
<Parameter name="webAppUserName" value="Admin" />
<Parameter name="webAppPassword" value="Password" />
</TestRunParameters>
<!-- Adapter Specific sections -->
<!-- MSTest adapter -->
<MSTest>
<MapInconclusiveToFailed>True</MapInconclusiveToFailed>
<CaptureTraceOutput>false</CaptureTraceOutput>
<DeleteDeploymentDirectoryAfterTestRunIsComplete>False</DeleteDeploymentDirectoryAfterTestRunIsComplete>
<DeploymentEnabled>False</DeploymentEnabled>
<AssemblyResolution>
<Directory Path="D:\myfolder\bin\" includeSubDirectories="false"/>
</AssemblyResolution>
</MSTest>
</RunSettings>
I think you could use vstest.console to get code coverage directly. We do not sure whether VS could open the code coverage that generated by third party tools.
I think if we could vstest.console to get the code coverage will has not problems. please try it to see the results.

Starting NUnit runner (2.6.3) from Visual Studio

I followed the instructions here to use "start external program" to start NUnit on my test project's dll. NUnit starts, but I get an error "This assembly was not built with any known testing framework"
I can run the same assembly with the same NUnit.exe from the command line just fine...
nunit.exe version 2.6.3
nunit.exe.config is as follows:
<configuration>
<!--
The GUI only runs under .NET 2.0 or higher. The
useLegacyV2RuntimeActivationPolicy setting only
applies under .NET 4.0 and permits use of mixed
mode assemblies, which would otherwise not load
correctly.
-->
<startup useLegacyV2RuntimeActivationPolicy="true">
<!-- Comment out the next line to force use of .NET 4.0 -->
</startup>
<runtime>
<!-- Ensure that test exceptions don't crash NUnit -->
<legacyUnhandledExceptionPolicy enabled="1"/>
<!-- Run partial trust V2 assemblies in full trust under .NET 4.0 -->
<loadFromRemoteSources enabled="true"/>
<!-- Look for addins in the addins directory for now -->
<assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
<probing privatePath="lib;addins"/>
</assemblyBinding>
</runtime>
</configuration>
OK, the answer is that I had to put the Command line argument in quotes.
e.g.
"C:\SVN\_branches\BUG-2261\Product\ProductTests\bin\Debug\ProductTests.dll"
I thought at first that this might be because there were spaces in the folder path for the dll but there aren't. It still need the quotes to make it work.

Is there any .runsettings documentation?

I'm looking for documentation for .runsettings files as used with vstest. Is there an xsd?
I can't seem to find much except for a few example files in the msdn documentation.
Runsettings (VS2012) are similar to testsettings (VS2010) where testsettings is specific to tests written for MSTest. VS2012 supports settings for different adapters. As such, the schema isn't a closed system so an XSD would not be comprehensive.
This MSDN article lists some high level details (https://msdn.microsoft.com/en-us/library/jj635153.aspx) for the elements in the runsettings file.
Here's a snippet from that article.
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
<!-- Configurations that affect the Test Framework -->
<RunConfiguration>
<MaxCpuCount>1</MaxCpuCount>
<!-- Path relative to solution directory -->
<ResultsDirectory>.\TestResults</ResultsDirectory>
<!-- [x86] | x64
- You can also change it from menu Test, Test Settings, Default
Processor Architecture -->
<TargetPlatform>x86</TargetPlatform>
<!-- Framework35 | [Framework40] | Framework45 -->
<TargetFrameworkVersion>Framework40</TargetFrameworkVersion>
<!-- Path to Test Adapters -->
<TestAdaptersPaths>%SystemDrive%\Temp\foo;%SystemDrive%\Temp\bar</TestAdaptersPaths>
</RunConfiguration>
<!-- Configurations for data collectors -->
<DataCollectionRunSettings>
<DataCollectors>
<DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
<Configuration>
<CodeCoverage>
<ModulePaths>
<Exclude>
<ModulePath>.*CPPUnitTestFramework.*</ModulePath>
</Exclude>
</ModulePaths>
</CodeCoverage>
</Configuration>
</DataCollector>
</DataCollectors>
</DataCollectionRunSettings>
<!-- Parameters used by tests at runtime -->
<TestRunParameters>
<Parameter name="webAppUrl" value="http://localhost" />
<Parameter name="webAppUserName" value="Admin" />
<Parameter name="webAppPassword" value="Password" />
</TestRunParameters>
<!-- Adapter Specific sections -->
<!-- MSTest adapter -->
<MSTest>
<MapInconclusiveToFailed>True</MapInconclusiveToFailed>
<CaptureTraceOutput>false</CaptureTraceOutput>
<DeleteDeploymentDirectoryAfterTestRunIsComplete>False</DeleteDeploymentDirectoryAfterTestRunIsComplete>
<DeploymentEnabled>False</DeploymentEnabled>
<AssemblyResolution>
<Directory Path>"D:\myfolder\bin\" includeSubDirectories="false"/>
</AssemblyResolution>
</MSTest>
</RunSettings>
I found further information on runsettings (specific to code coverage) here:
http://msdn.microsoft.com/en-us/library/jj159530%28v=vs.110%29.aspx

Resources