Recently I migrated all my C# projects from VS2010 to VS2013. There were barely any code changes required but immediately after I checked in the projects, I noticed that CI started failing. On checking the issue, I realized that the coverage file generated after running unit tests was not being generated in the location specified in the .testsettings file. In the Test Results Names section of Test Settings in VS2010, I configured a user defined naming scheme and unchecked the Append date-time stamp checkbox. This would generate the coverage file in \\data.coverage. After migrating to VS2013, the coverage file is getting generated in a Test Settings folder which believe is the default location when the Default naming scheme is chosen.
On researching a bit, I found an MSDN post and a few other blogs posts suggesting doing away with the .testsettings file and adopting the new .runsettings file. The weird thing is Microsoft is recommending usage of the .runsettings file but don't seem to deliver a default one which I can build upon. In any case, I did create one based on the sample provided on the same MSDN post. It did work for me as far as creation of the coverage file is concerned but I again faced the same problem that it is getting generated in a Test Results folder. The bigger problem is that a sub-folder with some GUID type name is created into which the coverage file is dumped and it changes every time a test is run with code coverage ON.
I wonder if there's a way make my VS2010 settings work as expected in VS2013 or using the .runsettings file, can I provide the complete path where the coverage file should be created?
Go to Test->Test Settings->Select Test Settings File and select the .testsettings file associated with your test project. It's a hack but always works.
Related
I wish I could provide a better description, but this is hard to describe succinctly.
Let me start with some background. This is a C++ solution using Visual Studio 2019 Community Edition. It has 12 projects of different types (DLLs, static libraries, and executable tests) and 4 build configurations ((Release, Debug)x(Win32, x64)). Keeping all the properties the same across projects and configurations has been a massive pain, so I decided to try using property sheets so that I could have the properties shared across projects. Given the structure of my solution, the property sheets are themselves somewhat complicated, but it's an improvement over what I had.
However about halfway through this migration I noticed some odd behavior. When I inspect the properties for one of my projects (a static library), the properties page is almost totally empty. Not only the properties I have changed with my sheets are missing, but even the default properties that you would expect.
However when I build this project, I can tell that most of the custom properties are in fact set correctly despite being missing, so it would seem like a UI bug, except some properties are not set correctly, for example the build log is not written to the correct location despite being set in the property sheet. This makes me worried that other more important properties may not be set correctly as well, but it's hard to know which properties are getting set and which are not.
I have another static library project that is set up in the same way with the same property sheets, and it behaves normally. It shows the expected properties in the project properties, and the build log is even in the correct location.
It gets even stranger. When I run the build from the command line using msbuild or devenv, the build log is written to the correct location. This suggests that building from the command line is working correctly, and that building from Visual Studio is not the same as building from the command line.
I ran msbuild -pp to see the expanded project file, and it looks correct. I diffed this project file with another static library from my solution that is working normally, and the diff is as expected, only the project names, paths, and source files are different. I did this diff on the expanded projects files too and they were the same except the expected differences.
I have closed and reopened Visual Studio countless times. I have also unloaded and reloaded the project. I have even tried deleting the .vs folder. None of these had any affect.
Here is a link to my code.
Screenshots demonstrating what I mean.
Is there any way to fix this? Have I done something wrong? I would like all of the properties from my property sheets to show up in the project properties, and I especially need all of them to apply when building the project.
Moving the <ItemGroup>s containing the source files below the <ImportGroup>s in my project file (.vcxproj) fixed the problem. I cannot explain why this works, or why this was ever a problem in the first place, but it fixed it.
We noticed that on a certain dev machine a Visual Studio (2015 update 3) debug build of a C# solution was generating a $RANDOM_SEED$ file alongside every built DLL.
The content of the file is just a single number e.g.
1443972318
Deleting the file(s) then rebuilding resulted in the file being regenerated, with a different number.
This behaviour was also observed when rebuilding a single project in the solution (one which has only the standard C# project refs/dependencies + System.Management).
Note that running a command line build e.g.
msbuild <sln-file>
did not regenerate the file (for build of complete solution or single project).
After a restart of VS, the file is no longer regenerated.
As far as we know this file name is not used in any of our source code, post build steps or internal dependencies.
There are quite a few dependencies on .NET framework classes, including Random and RNGCryptoServiceProvider, and also external dependencies. We don't have complete source code for all these so it's not possible to check exhaustively which if any of the dependencies are responsible.
This is a bit of a shot in the dark but the question is has anyone seen anything similar to this?
EDIT
I'm not surprised this has been downvoted - I appreciate it is pretty open ended, but as I'm currently not able to reproduce this and as it could have potentially serious consequences (random number generator attack?) I have posted it anyway. If I am able to repro I will of course update here.
I have the same file.
After a short investigation I found guilty:
this file is created by NUnit 3.x test adapter.
(You can check it in AdapterSettings.cs from NUnit adapter source code).
The file is used by NUnit to ensure that we use the same random seed value for generating random test cases in both the discovery and execution processes. This is required because the IDE uses two different processes to execute the adapter. It's not actually required (or created) when running the adapter under vstest.console.exe.
Having upgraded a large project from VS2008 to VS2013, a large number of unit tests are now failing because the associated data file cannot be found. The original DataSource attributes were created by the test connection string wizard that VS2008 provided, but this is no longer available in VS2013 Pro. The data files are definitely there, in exactly the same place in the solution, and all have the properties set to Copy Always. I suspect that the required arguments to the DataSource attribute have subtly changed but the MSDN documentation offers little help in this respect.
The error is:
Result Message: The unit test adapter failed to connect to the data
source or to read the data. For more information on troubleshooting
this error, see "Troubleshooting Data-Driven Unit Tests"
(http://go.microsoft.com/fwlink/?LinkId=62412) in the MSDN Library.
Error details: The Microsoft Jet database engine could not find the
object 'MatrixSampleResultGrid_ExcludeHiddenResults.csv'. Make sure
the object exists and that you spell its name and the path name
correctly.
Previously this error has always been reported because the data file has been moved or renamed without the attribute being updated, but that is definitely not the case here.
This is a typical current DataSource attribute definition:
[DataSource("Microsoft.VisualStudio.TestTools.DataSource.CSV", "|DataDirectory|\\MatrixSampleResultGrid_ExcludeHiddenResults.csv", "MatrixSampleResultGrid_ExcludeHiddenResults#csv", DataAccessMethod.Sequential)]
In the VS solution (ie on disk) the actual path to the datafile is
[theProjectRootFolder]\TestData\MatrixSampleResultGrid_ExcludeHiddenResults.csv
The test results are published to
[theProjectRootFolder]\TestResults\[testrun_datetimestamp]\In and ...\Out
although I notice that none of the data files have been copied to the In or Out folders. Is that significant?
With VS2008 these attributes have worked unchanged every day for years, so I can only conclude that for VS2013 the data is no longer appropriate, but what has changed? Without that wizard I can't even reconstruct the attribute so I am at a loss.
Also, I don't know what location "|DataDirectory|" represents in the context of a test run.
Can anyone help?
TIA.
I have figured out the solution to why all our VS2008 data-driven tests fail to find their data files in VS2013. Having spent hours reading all the MSDN documentation I could find on unit testing and TDD in VS with absolutely no illumination (useless!), I am posting the solution here to save others in a similar situation from all the pain:
The VS2013 test framework seems to have different rules on where to look for the data file (different to VS2008, that is). Either we were inadvertently employing an ‘undocumented feature’ in VS2008 which no longer works, or MS have simply changed it. I don’t know which, but it has changed.
However the fix is simple once you have tumbled to the cause. Assuming the data file is in a subfolder of the test project folder (ex [projectfolder]\TestData) , the old VS2008 test attributes:
(ex)
[DataSource("Microsoft.VisualStudio.TestTools.DataSource.CSV", "|DataDirectory|\\MyDataFile.csv", " MyDataFile#csv", DataAccessMethod.Sequential)][DeploymentItem("Test Projects\\Project1Tests\\TestData\\MyDataFile.csv"), TestMethod()]
need to be amended to
[DataSource("Microsoft.VisualStudio.TestTools.DataSource.CSV", "MyDataFile.csv", "MyDataFile#csv", DataAccessMethod.Sequential), DeploymentItem("TestData\\MyDataFile.csv"), TestMethod]
The changes are
The second DataSource argument is just the filename (not a relative path), and
The DeploymentItem argument is a path relative to the project folder that contains the tests.
Also (this has been documented elsewhere on this forum)
The datafile properties must be set to BuildAction=none (or the default blank), Copy To Output Directory=Copy Always.
and you have to have a TestSettings configuration in the solution with Deployment checked on.
Do all that and the old VS2008 data driven tests will magically all start finding their datafiles.
I am forcing myself to learn test-driven development, and so far I'm enjoying myself. There's a few quirks that Visual Studio Unit Testing has that is driving me batty though. A bit of background information, my project folder looks like this:
[Root] BitFlex
BitFlex\Code
BitFlex\Debug
BitFlex\Documents
BitFlex\Release
Now of course all the source code is stored in the code folder, and on a build the project output either goes to the debug or release folders depending on the current configuration. Now for my unit testing, I have it setup so the test project is output to either:
BitFlex\Debug\Unit Tests\
BitFlex\Release\Unit Tests\
1) At this point, everything is fine and dandy. There are two problems with this, the first being that when I run a test it cannot find the assembly, as it gives me this error:
Error AssignDefaultProgramTest BitFlex.UnitTests The test assembly 'D:\src\DCOM Productions\BitFlex\Code\TestResults\David Anderson_DCOMPRODUCTIONS 2009-07-31 23_21_00\Out\BitFlex.UnitTests.dll' cannot be loaded. Error details: Could not find file 'D:\src\DCOM Productions\BitFlex\Code\TestResults\David Anderson_DCOMPRODUCTIONS 2009-07-31 23_21_00\Out\BitFlex.UnitTests.dll'.
I cannot seem to find information on this error, or how to resolve it so I suppose that's where everyone's experise around here comes in to play.
2) My other beef is that Visual Studio generates the "Test Results" folder in my code directory, I would prefer to move that to my Unit tests folder in either output configuration. Is there a way to do this, or a better practice to setting up a well organized Unit Test using my folder hierarchy?
By default MSTesting framework runs all tests in an 'isolated' location and not from the binaries directory.
To fix this you can do one of these two:
1. go to the test configuration file and under deployment uncheck deploy the tests.
2. don't use path when looking for external files instead use deploy attribute or the test config to deploy the needed files along with your tests.
For doing TDD with MSTest, turn off deployment. You shouldn't need it for "unit testing."
Also, never ever EVER have VS automatically generate tests for you. What's generated may be fine for some types of functional testing, but are usually very poor unit tests.
If I open a solution in Visual Studio 2008 and run a unit test then VS creates a new .vsmdi file in the Solution Items folder and gives it the next number available e.g. My Solution2.vsmdi.
Any idea why VS is doing this and how I can get it to stop doing this?
It appears that the VSMDI problem is a known bug and has been around since VS2005 Team System but it has no clear fix as yet. Another reason to NOT use MS Test.
An MSDN blog details how to run unit tests without VSMDI files.
Assuming that the VSMDI file is under source control, here's a Microsoft Support article about this issue: Multiple vsmdi Files after Running Team Test with VSMDI file under Source Control
Which says:
Someone ran a test while someone else was modifying the vsmdi file.
Team Test detects that the VSMDI file is out of sync;therefore, Team
Test it makes one and thus you see the incrementing vsmdi files.
And:
Going forward you want to make sure the file is not marked for auto
checkout when it is modified. When the current tester has the VSMDI
file checked out you, do not want other users to be able to check it
out. You want your developers to checkout the file, run a test, and
check it back in.
I work around this by always checking out the .vsmdi.
It seems that this only happens when the .vsmdi file is read-only, e.g. not checked-out in a version control system that uses that kind of lock-local-files behavior (Perforce etc).
An old post but vsmdi is a meta data file created by the test system.