I have a big test suite written in TestNG that takes several hours to complete.
I'd like for maven/surefire to create a new suite that is a copy of the first but with just the failed tests in it, a suite that should be much faster to run.
Is it possible to create such a suite?
As a fall back I could create it on my own from a test report that is easy to parse, if there is such report.
Thank you.
On completion of run, testng generates a testng-failed.xml (in the same output folder as your reports), which basically is your initial suite file with the listeners, but the tests part contains only the failed testcases.
In case you are using Jenkins, you might consider adding a postbuild step that triggers another build that works on the same workspace as the current build and uses this failed xml. Or depending on how you are triggering your tests, you might look at writing a script to run the failed xml.
Related
I am writing test cases in Junit, and when i build my application, i want to see in my logs which test cases has failed, succeeded , skipped and number of test cases run.
So how can we implement this.
When you build an application you probably run some build tool (Maven, Gradle) to do so.
Maven for example has a plugin (called surefire) for running the tests.
And it also produces a report with all the failures/ successfully run tests.
Since maven also establishes a well-known layout of directories, you'll find this report in the target/surefire-reports folder.
There are even tools that can parse the information about the tests execution and show it in a graphical way in CI tools for example
I am trying to run multiple feature files through maven goal(command line) but after 2 feature files that run successfully, the other feature files (3rd one onwards) fails in some test cases which when ran independently passes all the test cases.
So f I run each feature file individually I get proper results but running them all together gives wrong results.
We are using serenity framework with cucumber jvm. Please help how can we resolve this issue.
Your failing tests fail to fully setup the context. Some state is leaking from the previous ones. Look for what has changed during the first runs (database/mocks/whatever state) that has to be reset before running the third and following.
I created a build definition that runs automated tests using MTM build environments and test suites. I recently created a Visual Studio Load Test, which can be added to a test suite just like any test method marked with the [TestMethod] attribute. However, when I run the build, I get no errors and it appears the aggregate tests don't run. Is there a way to make this work?
I found this article: https://blogs.msdn.microsoft.com/testingspot/2013/01/22/how-to-automatically-run-a-load-test-as-part-of-a-build/ which describes a way to do it, but I can't find a build template that matches what he describes, and it appears this only allows you to run a single load test.
Also, when you configure a test controller, there is an option to configure it for load testing, but to do this, you must unregister it from the Team Project Collection. If this is done, it appears the controller can no longer be used in an environments to run project automated tests. This defeats the purpose of what I want to do and makes it seem that Load Tests and Team Projects are mutually exclusive. Is this the case? If so, this is a big oversight. Load tests are the kind of thing you would like to run automatically. Thanks for the help.
You are unfortunately right. A test controller used for load testing cannot be used for other automated test execution 'at the same time'. In your scenario I would recommend that you setup a different test controller and agent for load testing and you would be able to queue it as a part of your build to achieve what you are looking for.
There is no special build process template for this case.
I have been using the Build-Deploy-Test build workflow for TFS 2010 (see here http://msdn.microsoft.com/en-us/vstudio/gg131922.aspx) and have come up against a very annoying limitation.
Here is what I have done:
Setup a build to be queued
Configured it to restore to a test-ready snapshot
Added some deployment steps; in this case starting mongo db
I then came to the step where you configure which tests to run and hit an issue. Firstly you need configure a test plan with some test suites; a test suite consisting of test cases. The problem is that each test case can only be associated with a single unit test.
With having to create a test case per unit test, it means that every time a unit test is added a new test case must be created. Is there any way you can associate many unit tests with a test suite that can be ran in the Build-Deploy-Test build workflow.
One way around this problem is to update your test suite using the tcm.exe testcase /import /syncsuite: command. This command has to be run before test execution begins and a logical place to add it would be into the build template.
I suggest you to use Test List Editor in Test Tab on Visual Studio, create your differents lists of tests, organize your tests based on functionnal after that you can execute on your build list of tests by entering name of list.
link : http://msdn.microsoft.com/en-us/library/ms182463(v=vs.100).aspx
For configuring Build Definition, i suggets you to edit & select Automated Test Tab and insert list.
link : http://support.smartbear.com/articles/testcomplete/testcomplete-and-team-build/
I'm new to load testing in Visual Studio/MSTest, and I created a new Load Test recently to validate some high-traffic scenarios for a WCF service. I want to add this to the tests project for the service, but I don't want the test to be executed whenever I "Run All Tests in Solution" nor as part of our Continuous Integration build-verification process because a) it takes 5 minutes to run, and b) the service call that it is testing generates many thousands of email messages. Basically, I'd like to do the equivalent of adding the [Ignore] attribute to a unit test so that the load test is only executed when I explicitly choose to run it.
This MSDN Article ("How to: Disable and Enable Tests") suggests that the only to disable the test is to use Test Lists (.vsmdi files), but I don't have much experience with them, they seem like a hassle to manage, I don't want to have to modify our CI Build Definition, and this blog post says that Test Lists are deprecated in VS2012. Any other ideas?
Edit: I accepted Mauricio's answer, which was to put the load tests into a separate project and maintain separate solutions, one with the load tests and one without. This enables you to run the (faster-running) unit tests during development and also include the (slower-running) load tests during build verification without using test lists.
This should not be an issue for your CI Build Definition. Why?
To run unit tests as part of your build process you need to configure the build definition to point to a test container (usually a .dll file containint your test classes and methods). Load tests do not work this way, they are defined within .loadtest files (which are just xml files) that are consumed by the MSTest engine.
If you do not make any further changes to your CI Build definition the load test will be ignored.
If you want to run the test as part of a build, then you need to configure the build definition to use the .loadtest file.
Stay away from testlists. Like you said, they are being deprecated in VS11.
Edit: The simplest way to avoid running the load test as part of Visual Studio "Run All" tests is to create a different solution for your load tests.
Why don't you want to use Test Lists. I think is the best way to do that. Create different Test Lists for each test type (unit test, load test...) and then in your MSTest command run the Test List(s) you want:
MSTest \testmetadata:testlists.vsmdi \testlist:UnitTests (only UnitTests)
MSTest \testmetadata:testlists.vsmdi \testlist:LoadTests (only LoadTests)
MSTest \testmetadata:testlists.vsmdi \testlist:UnitTests \testlist:LoadTests (UnitTests & LoadTests)