I have many pending tests in my application, and they make it hard to view a failed test in Rubymine.
Is there a way, either through rspec command line options or Rubymine settings, to disable running pending tests entirely? Or at least not show them in the view? I know of the button to not show passing tests, but that doesn't help with pending tests.
Rspec 3.1? Pending's meaning has changed... try skipped instead?
From: https://www.relishapp.com/rspec/rspec-core/v/3-1/docs/pending-and-skipped-examples
An example can either be marked as skipped, in which is it not
executed, or pending in which it is executed but failure will not
cause a failure of the entire suite. When a pending example passes
(i.e. the underlying reasons for it being marked pending is no longer
present) it will be marked as failed in order to communicate to you
that it should no longer be marked as pending.
Related
When I run tests in Xcode, as long as I run all the tests collectively they execute normally. However, if I run only one individual test class or one individual test function within a test class, it doesn't execute any tests at all and prints:
Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.001) seconds
Upon debugging I've found that it does execute individual tests if I choose a particular other scheme I have, but my other two schemes don't execute the individual tests. I also noticed that if I edit one of the two problematic schemes to use the same Build Configuration as the working scheme, the individual tests do work (each of my schemes uses a different build configuration).
How do I make the problematic build configurations work to let me execute individual tests?
I was able to figure it out!
If I click on my project in the Project navigator, then click on my project in the left sidebar of the page that appears, and then go to the Info tab, I can see at the bottom of the Configurations section it says:
"Use ConfigurationName for command-line builds"
If I click the dropdown and change it to a different build configuration, it makes the scheme that uses that build configuration able to execute individual tests! And, as one might expect, it also makes the scheme previously able to execute individual tests no longer able to.
It would be nice if I could dynamically change the build configuration for command-line builds based on which scheme is active in Xcode, but as of yet I have not figured out how to do so.
So for now, just know that you can manually change this build configuration value whenever you want to run individual tests for a scheme.
Upgraded Resharper to 2016.1 and I'm not able to run NUnit tests marked with the Ignore attribute. This was possible before by right clicking the test and run. Is this a change or am I missing something?
It is very frustrating, I have many tests which I ignore on the build machine which I would like to run locally.
Resharper has been using NUnit itself to run it's tests for a long time, so it seems you must have done a really big upgrade, from an early version of Resharper that executed the test methods itself. NUnit simply will not run an ignored test, even if you programmatically tell it to do so. Or, to put it differently, NUnit "runs" ignored tests by reporting that they are ignored.
This is actually the definition of "Ignored" in NUnit. It was designed a long time ago to deal with tests that should not be run, generally for a very short period, while the developer is doing other work. It shows up as a warning in any test runs because, in the ideal world, Ignored tests should not make it into your CI runs. It may be the wrong attribute for you to use for your purposes, especially if you want to be able to run it some of the time. If you want a test that is only run when expressly selected, we have the "Explicit" attribute instead. Other behavior is possible, but it would take a feature request.
We've recently found that an acceptance test project fails occasionally on our build server- due to our web drivers Type is not resolved error. I'm trying an experiment to see if its a question of timing of build steps. To this end I've tried to create a separate build step which launches the webdriver executable separately and then proceed onto the unit tests - the issue I have is when I launch the process it blocks the next step after it has successfully started.
eg. Type is not resolved for member 'OpenQA.Selenium.WebDriverException,WebDriver, Version=2.41.0.0, Culture=neutral, PublicKeyToken=null'..
Is there a way I can progress to the next build step after I call an exe?
Thanks
Do you have multiple build agents? Have you considered splitting this into separate build configurations and having triggers on previous build configurations?
You could have configurations:
Compile,
Webdriver, triggered on successful compile completion
Unit Tests, triggered on successful compile completion.
It's not nice, but it should give you the non-blocking behavior you're after. I can't see any other way of making it happen, build configurations are supposed to run as a linear sequence.
We designed some tests in Microsoft Test Manager for our first sprint (we are using agile) and executed them. Now we are on our second sprint and we need run these tests again as regression. Is there way we can reset the statuses of the tests without destroying the test execution history of the previous sprint?
There isn't a way. We ended up cloning the test cases into a different plan each time.
You can reset your test back to the active state:
This way you know you have to run this tests again in current sprint.
I would only clone test cases if you are going to change them in the next sprint (e.g. add additional steps).
As long as they remain unchanged there is no reason to clone them, from my point of view.
Create a new test suite and add your cases to it. Each suite keeps the final status of each case within the suite - you can see this in the Test tab. The overall status of the case (that is, outside of any suite) is whatever the last run was.
I agree that you don't want to clone the cases unless you are changing the actual steps of the cases between sprints.
Is it possible to stop a TeamCity build (the entire build, i.e. it won't execute subsequent steps) when a unit test fails? Ideally I'd also like it to terminate the currently executing step which in my case would be the Nunit Test Runner. In my circumstance there is no point in continuing the build if a single unit test fails. I've looked at Failure Conditions but I don't think they are applicable as the build continues to run.
Feature requested: http://youtrack.jetbrains.com/issue/YTF-3275
As you noted, TeamCity can run no further build steps on test failures in NUnit tests with "Only if build status is successful" step execution condition. However, that does not make the tests run stop until the step finish.
A related feature request is TW-23766.
The only workaround I can consider currently is not to use NUnit test runner and implement the logic inside the build script. For example, with nunit-console.exe like Manuel noted. If you choose to follow this route, consider using TeamCity Addin for NUNit.
You can do it using nunit-console.exe.
accordingly to the official documentation (http://nunit.org/index.php?p=consoleCommandLine&r=2.6.2) it provides a /stoponerror switch that does exactly what you need.
it can also generate an XML output that can be parsed by teamcity (there is a build feature for that) in order to populate the "test" tab.
on the build step after the unit tests, change the setting to Only if Build status is successful instead of if all previous steps finished successfully
see screenshot
also make sure that under failure conditions menu item the at least one test failed option is also ticked.