I'm running tests in cucumber using the --retries N option to reattempt failed tests N times to catch some tests which are failing inconsistently. Currently the summary after running these tests in the terminal is something like this:
100 scenarios (2 failed, 5 flaky, 1 skipped, 98 passed)
588 steps (9 failed, 24 skipped, 555 passed)
11m45.859s
Failing Scenarios:
cucumber features/some_feature.feature:13 # Scenario: AC.1 Some scenario
cucumber features/some_feature.feature:54 # Scenario: AC.6 Some other scenario
This lets me know what's failing, however I'd like to also have a list of the flakey scenarios to help me diagnose what is failing inconsistently. Is there a way to set up Cucumber such that this is the case?
The scenarios listed are the scenarios that fail the build (making the exit code non-zero), if you use the option "--strict" or "--strict-flaky" the flaky scenarios will also be listed in the summary ("--strict" will also list the pending and undefined scenarios).
It's currently not possible to see Flaky scenarios in the summary.
In order to change this, someone would have to submit a pull request changing console_issues.rb, and possibly associated tests.
Related
I have a test suite of rspec tests which are divided into different files.
Every file represents one test scenario with some number of test steps.
Now, on some particular tests, it can happen that specific step fails but it is too time consuming and not needed to run rest of the steps in that scenario.
I know there is an option --fail-fast in rspec but if I'm running tests like: rspec spec/* that will mean that when first step fails in any script, it will abort complete execution.
I'm just looking for mechanism to abort execution of that specific test scenario (test script) when failure happens but to continue execution of other test scenarios.
Thanks for the help,
Bakir
Use the RSpec-instafail gem.
According to its documentation, it:
Show failing specs instantly. Show passing spec as green dots as usual.
I'm looking for the way how I can manage ruby cucumber scenarios in runtime. I'd like to filter some scenarios out having some information about SUT which I can gather in runtime.
For instance, I've the following scenarios
#automated
Scenario: As a customer I want to run ... scenario 1
Given ...
#automated #debug
Scenario: As a debugger I want to run ... scenario 2
Given ...
#automated
Scenario: As a customer I want to run ... scenario 3
Given ...
#automated #release
Scenario: As a releaser I want to run ... scenario 4
Given ...
I'm able to determine whether a debug or release application is testing now. And for the debug one I want to see scenarios 1,2,3 to be run but for the release app I want to see 1,3,4 to be run.
I know how to do it using rake or any other wrapper script but it'd be better to find solution without such wrapper scripts.
Also, cucumber profiles might not be a good choice here because there are several parameters each with a number of values. So it might require some crazy number of their combinations.
I have a set of cucumber tests that use Capybara to access a website and perform certain tasks. The tests run fine and at the end they output accurate information about whether or not the tests and steps failed or passed. For example,
1 scenario (1 failed)
3 steps (1 failed, 2 passed)
However, if I try to write a customer formatter or even use one of the built-in custom formatters (such as progress or pretty), it shows that all of the steps are being skipped.
Does anyone know why this could be? Again, I think that all of the steps are executing properly and cucumber is reporting back to me at the end if they failed or passed (as I would expect), but the formatters seem to always think that the steps are being skipped.
If you're using a scenario outline, there's a limitation in the parser that causes them to be reported as skipped: https://github.com/cucumber/cucumber/issues/316
You can run Cucumber with the --expand flag (or -x for short) to output each step in the scenario outline for every row in the the example table. Then they should report as passed or failed as expected.
Well i am trying to achieve one thing. Something fancy but worth trying..
When i run bundle exec rspec spec/
My test cases pass and i get
Finished in 5.12 seconds
38 examples, 0 failures, 1 pending
I want to use the MAC OS say command (Read here about it). Which will say after the test suite has completed, that you have 38 examples, 0 failures and 1 pending example.
eg command
say you have example_count examples, failures_count failures and pending_count pending examples
The requirement is, I need to capture Examples, Failures and pending in a variable which i can use with say command.
You could write custom RSpec formatter and do your magic there. You can also have multiple formatters.
Check this resources for more information:
Custom formatters (Relish)
How to Use Multiple Formatters in RSpec 2
Source code base_text_formatter.rb
Background: I have a relatively large Cucumber test suite. The problem is that there are several test cases that will fail due to known bugs that probably won't be fixed for a month or two. This means that whenever myself or someone else needs to run the test suite, we get several failures and then have to spend time digging through the test results and figuring out which ones were expected and which ones are new.
The quick and dirty solution is to simply comment the test cases out. The problem I have with that is that when the bugs are fixed there is no guarantee that the commented out test case will be uncommented.
Question: Is there a simple method in Cucumber to separate the expected failures from the unexpected ones?
You can tag them as #wip
The default cucumber call will ignore the #wip scenarios
#wip
Scenario: Something
Btw
rake cucumber:ok #will run all the scenarions except the #wip ones
rake cucumber:wip #will run just the #wip tagged scenarios
rake cucumber #same behavior as rake cucumber:ok