I have two unit test projects in my VS 2010 solution. Each project has a Data directory with input data needed for the unit tests.
I annotated the test classes that need the data with
[DeploymentItem("Data")]
When I run tests individually, the run fine. However, when I run (or debug) all tests in the solution, I find that only one of the two Data directories is copied to TestResults\MyTestDir-YYYY-MM-DD HH_mm_SS\Out, causing unit tests that rely on the other data directory to fail.
Interestingly if I then use the Test Results window to re-run checked (as in failed) tests, the tests still fail (they do not recognize that the correct Data directory's files are missing). If I then navigate directly to a failed test and select Run Tests in Current Context, the test run gets the correct Data directory and the test succeeds.
My question: How can I cause DeploymentItems from two separate test projects all to be copied to the Out directory where the tests are executed?
For reference, I have reviewed these questions without noting a solution
Problems with DeploymentItem attribute
Do MSTest deployment items only work when present in the project test settings file?
I found that giving each "Data" directory a unique name (e.g. "TestProjectAData") resolved the issue. There seems to be some sort of bug when multiple directories in different paths all have the same relative path to different test projects (i.e. if every test project has a subdirectory called "Data").
Related
Pact merges pacts at the file level, this is great for merging pacts from multiple tests, but not so great when you want to modify and re-run a test without cleaning the target/pacts folder.
The default junit run config in intellij doesn't clean the target folder before running the tests; I know I can use maven clean/remove the files manually, but this means anyone else who runs these tests locally needs to know to run them a specific way.
I want to merge pacts from multiple tests so I don't want to turn off merging.
I tried implementing a before method that deletes files from the pact folder if they exist, but it was janky.
I'm considering setting the pact folder to a temporary directory that removes itself after the tests are run, but that might interfere with pushing new pacts to the broker, and I don't want to remove the folder too soon/often and end up with missing pacts. Also it's useful to be able to see the files at the end, so auto-removing them isn't ideal.
Is there a nice way to stop old pacts merging with new ones, without relying on people to just know they need to remove old pact files before running a modified test?
Why is it an actual problem for you? As in, yes, the pact file is temporarily bigger than it should be, but what is the actual impact?
You shouldn't be publishing from your local machine anyway, that is a CI concern (I usually restrict by not providing write credentials to local environments) so if all you need is to be able to rerun a unit test then I wouldn't worry.
Alternatively, if you are all using the same IDE you could create an IDE specific config and check it in to the repo that cleans the dir before any target / test is run.
I am trying to run multiple feature files through maven goal(command line) but after 2 feature files that run successfully, the other feature files (3rd one onwards) fails in some test cases which when ran independently passes all the test cases.
So f I run each feature file individually I get proper results but running them all together gives wrong results.
We are using serenity framework with cucumber jvm. Please help how can we resolve this issue.
Your failing tests fail to fully setup the context. Some state is leaking from the previous ones. Look for what has changed during the first runs (database/mocks/whatever state) that has to be reset before running the third and following.
I have recently added a TFS build to our project, and configured it to run the automated tests that are in the project, and for some reason the build is ignoring the tests as if they don't exist!
In order to figure this out I created another solution with only a basic project from the origin code and the test project as well.
I've added these to the TFS and configured an almost equal build to this solution, and guess what? it executed the tests! it's the same tests as the original. just a copy of it.
The main difference between these 2 solutions is that the original code is a big solution with many projects, most of the projects are in some kind of a solution directory (and so does the tests project - it's inside a solution directory as well).
The difference between the TFS build definition is only that the output location of the build is set to single directory (I also tried PerProject and it worked) and in the original code it's defined to "AsConfigured" because we have some build tasks for copying dlls and such.
Has anyone encountered this problem?
Any ideas?
Thanks
TFS will look for the test DLLs in the BINARIES output folder location. You need to configure your 'build tasks for copying DLLs' to ensure your test DLLs are copied to this location.
Test assembly file specification
Specify the binary files that contain the tests that you want to run. Leave the default value (**\**test*.dll) if you want the build agent to search recursively for any .dll files that match *test*.dll in the binaries subdirectory of the build agent's working directory.
TFS2013 — http://msdn.microsoft.com/en-us/library/ms253138.aspx
TFS2012 — http://msdn.microsoft.com/en-us/library/ms253138(v=vs.110).aspx
I work with Coded UI Tests and Visual Studio 2013. Now I have to test if images from a folder are shown correctly inside the application. That’s why I have created a folder containing the images and set build action to none and deployment to always.
Unfortunately all tests are executed in an own test results folder and my images are not deployed correctly. I know I could do this by using the DeploymentItem attribute or a testsettings file but I don't want to do this. I want to avoid the "test results" folder and run the tests from within the output folder of my test project.
I do this with the unit tests for instance. They are written with XUnit.Net which works just fine. I thought it would also work with MS Test but this only seems to work for Unit Tests but not for Coded UI Tests.
So, to sum it up again: How can I get rid of the “Test Results” folder when using Coded UI Tests with Visual Studio 2013 and run my tests simply from within the output directory of the project?
You can do this using a custom entry in the .runsettings file I think.
DeploymentItemAttribute Class
Consider running your unit tests directly in the build output
directory, so that testing runs more rapidly. This is especially
useful on the build server after you have checked in your tests. To do
this, add a .runsettings file to your solution, include
False, and select the file in
the Test, Test Settings menu. The same effect occurs in any test run
in which DeploymentItemAttribute is not used at all.
However, you
might prefer not to do this if you want to be able to inspect the data
files after a failed run.
You cannot avoid using a deployment folder
if you are using a .testsettings file, which is required for web and
load tests, coded UI tests, and any test in which you deploy an
application to remote machines.
As for DeploymentItemAttribue itself I would steer clear of it unless the following issues have been fixed, Gotchas: MSTest’s [DeploymentItem] attribute. +1 for xUnit in TestDriven.NET.
You can try deleting the LocalTestRun.testrunconfig file from the Solution Items folder (directly under the solution). When we did that it started using the \bin\debug\ folder instead of the TestResults folder when running our unit tests using MSTest.
I'm new to load testing in Visual Studio/MSTest, and I created a new Load Test recently to validate some high-traffic scenarios for a WCF service. I want to add this to the tests project for the service, but I don't want the test to be executed whenever I "Run All Tests in Solution" nor as part of our Continuous Integration build-verification process because a) it takes 5 minutes to run, and b) the service call that it is testing generates many thousands of email messages. Basically, I'd like to do the equivalent of adding the [Ignore] attribute to a unit test so that the load test is only executed when I explicitly choose to run it.
This MSDN Article ("How to: Disable and Enable Tests") suggests that the only to disable the test is to use Test Lists (.vsmdi files), but I don't have much experience with them, they seem like a hassle to manage, I don't want to have to modify our CI Build Definition, and this blog post says that Test Lists are deprecated in VS2012. Any other ideas?
Edit: I accepted Mauricio's answer, which was to put the load tests into a separate project and maintain separate solutions, one with the load tests and one without. This enables you to run the (faster-running) unit tests during development and also include the (slower-running) load tests during build verification without using test lists.
This should not be an issue for your CI Build Definition. Why?
To run unit tests as part of your build process you need to configure the build definition to point to a test container (usually a .dll file containint your test classes and methods). Load tests do not work this way, they are defined within .loadtest files (which are just xml files) that are consumed by the MSTest engine.
If you do not make any further changes to your CI Build definition the load test will be ignored.
If you want to run the test as part of a build, then you need to configure the build definition to use the .loadtest file.
Stay away from testlists. Like you said, they are being deprecated in VS11.
Edit: The simplest way to avoid running the load test as part of Visual Studio "Run All" tests is to create a different solution for your load tests.
Why don't you want to use Test Lists. I think is the best way to do that. Create different Test Lists for each test type (unit test, load test...) and then in your MSTest command run the Test List(s) you want:
MSTest \testmetadata:testlists.vsmdi \testlist:UnitTests (only UnitTests)
MSTest \testmetadata:testlists.vsmdi \testlist:LoadTests (only LoadTests)
MSTest \testmetadata:testlists.vsmdi \testlist:UnitTests \testlist:LoadTests (UnitTests & LoadTests)