I have a scenario outline. I want to skip certain steps for certain scenarios. I used Cucumber.wants_to_quit but it doesn't seem to work. Steps mentioned after this method are run anyway for all scenarios.
I'm using Ruby, selenium, cucumber.
Here is an example Directly from the Docs on how to create a feature that would allow you to skip:
Given a file named "feature/test.feature"
Feature: test
Scenario: test
Given this step says to skip
And this step passes
Using standard step defintions and with a file such as "features/step_definitions/skippy.rb" with:
Given /skip/ do
skip_this_scenario
end
if you run cucmber -q
then the previous would pass as:
Feature: test
Scenario: test
Given this step says to skip
And this step passes
1 scenario (1 skipped)
2 steps (2 skipped)
0m0.012s
Related
This is a how-to/best-practice question.
I have a code base with a suite of unit tests run with pytest
I have a set of *.rst files which provide explanation of each test, along with a table of results and images of some mathematical plots
Each time the pytest suite runs, it dynamically updates the *.rst files with the results of the latest test data, updating numerical values, time-stamping the tests, etc
I would like to integrate this with the project docs. I could
Build these rst files separately with sphinx-build whenever I want to view the test results [this seems bad, since it's labor intensive and not automated]
tell Sphinx to render these pages separately and include them in the project docs [better, but I'm not sure how to configure this]
have a separate set of sphinx docs for the test results which I can build after each run of the test suite
Which approach (or another approach) is most effective? Is there a best practice for doing this type of thing?
Maybe take a look into Sphinx-Test-Reports, which reads in all information from junit-based xml-files (pytest supports this) and generates the output during the normal sphinx build phase.
So you are free to add custom information around the test results.
Example from webpage:
.. test-report:: My Report
:id: REPORT
:file: ../tests/data/pytest_sphinx_data_short.xml
So complete answer to your question: Take none of the given approaches and let a sphinx-extension do it during build-time.
I have two cucumber feature ( DeleteAccountingYear.feature and AddAccountingYear.feature).
How can i do to make that the second feature(AddAccountingYear.feature) run before the first one (AddAccountingYear.feature).
I concur with #alannichols about tests being independent of each other. Thats a fundamental aspect of automation suite. Otherwise we will end up with a unmaintainable, flaky test suite.
To run a certain feature file before running another feature appears to me like a test design issue.
Cucumber provides few options to solve issues like this:
a) Is DeleteAccountingYear.feature really a feature of its own? If not you can use the cucumber Background: option. The steps provided in the background will be run for each scenario in that feature file. So your AddAccountingYear.feature will look like this:
Feature: AddingAccountingYear
Background:
Given I have deleted accounting year
Scenario Outline: add new accounting year
Then I add new account year
b) If DeleteAccountingYear.feature is indeed a feature of its own and needs to be in its own feature file, then you can use setup and teardown functions. In cucumber this can be achieved using hooks. You can tag AddDeleteAccountingYear.feature with a certain tag say #doAfterDeleteAccountYear. Now from the Before hooks you can do the required setup for this specific tag. The before hooks(for ruby) will look like:
Before('#doAfterDeleteAccountYear') do
#Call the function to delete the account year
end
If the delete account year is written as a function, then the only thing required is to call this method in the before hook. This way the code will be DRY compliant as well.
If these options doesn't work for you, one another way of forcing the order of execution is by using a batch/shell script. You can add individual cucumber commands for each feature in the order you would like to execute and then just execute the script. The downside of it is different reports will be generated for each feature file. But this is something that I wouldn't recommend for the reasons mentioned above.
From Justin Ko's website - https://jkotests.wordpress.com/2013/08/22/specify-execution-order-of-cucumber-features/ the run order is determined in the following way:
Alphabetically by feature file directory
Alphabetically by feature file name
Order of scenarios within the feature file
So to run one feature before the other you could change the name of the feature file or put it in a separate feature folder with a name that alphabetically first.
However, it is good practice to make all of your tests independent of one another. One of the easiest way to do this is to use mocks to create your data (i.e. the date you want to delete), but that isn't always an option. Another way would be to create the data you want to delete in the set up of the delete tests. The downside to doing this is that it's a duplication of effort, but it won't matter what order the tests run in. This may not be an issue now, but with a larger test suite and/or multiple coders using the test repo it may be difficult to maintain the test ordering based solely on alphabetical sorting.
Another option would be to combine the add and delete tests. This goes against the general rule that one test should test one thing but is often a pragmatic approach if your tests take a long time to run and adding the add data step to the set up for delete would add a lot of time to your test suite.
Edit: After reading that link to Justin Ko's site you can specify the features that are run when you run cucumber, and it will run them in the order that you give. For any that you don't care about the order for you can just put the whole feature folder at the end and cucumber will run through them, skipping any that have already been run. Copy paste example from the link above -
cucumber features\folder2\another.feature features\folder1\some.feature features
I want to be able to change my feature file from outside visual studio and have the updated feature file picked up, for subsequent test execution, without compilation of my test project. Is it possible to do this? Can someone help to specify the exact steps needed to do this? I am using MsTest.
Here are the steps I followed, but I get the message "No tests to execute." every time:
Change Test Project file (.csproj) as mentioned here
Build the Test DLL from Visual Studio
Kept the feature file in a folder FeatureFiles, under the Test release folder
Changed the feature file in Notepad
Used the Specflow generate all command, to regenerate the tests:
Specflow generateall TestProject.csproj /force /verbose
Create the report:
mstest /testcontainer:Test.Dll /resultsfile:TestResult.trx
A similar question was asked earlier, and I am following the same steps as mentioned by Marcus there.
Update
Here is what I would like to do. Considering the following .feature file:
Feature: Score Calculation
As a player
I want the system to calculate my total score
So that I know my performance
Scenario: Another beginners game
Given a new bowling game
When I roll the following series: 2,7,3,4,1,1,5,1,1,1,1,1,1,1,1,1,1,1,5,1
Then my total score should be 40
In the above feature file, I would like to change the data series of numbers and change the total score and run the same test again to check if it runs fine and I get a correct score
If you only want to change the data for a test, then go with a CSV file that you save inside your VS project. Then in a Given statement open that file, parse it and save it to the ScenarioContext.Current object for use in subsequent steps.
I can't see how what you want is possible. SpecFlow generates unit test classes from the feature file. These unit test classes MUST be compiled before they can be run, the same as ANY code change.
You are getting a 'no tests to execute' because you are still running against the old version of the dll, which doesn't contain any tests because you have not rebuilt it with the new code that specflow has generated.
From what is stated in the linked question it seems they are saying that some changes may be able to be run without recompiling, but honestly I cannot see how that is possible, and even if some feature file changes were possible I imagine that the solution would be specific to certain unit test frameworks and unsupported.
EDIT
When I look at the generated feature.cs file you have this:
[Microsoft.VisualStudio.TestTools.UnitTesting.TestMethodAttribute()]
[Microsoft.VisualStudio.TestTools.UnitTesting.DescriptionAttribute("Another beginners game")]
[Microsoft.VisualStudio.TestTools.UnitTesting.TestPropertyAttribute("FeatureTitle", "Score Calculation")]
public virtual void AnotherBeginnersGame()
{
TechTalk.SpecFlow.ScenarioInfo scenarioInfo = new TechTalk.SpecFlow.ScenarioInfo("Another beginners game", ((string[])(null)));
#line 5
this.ScenarioSetup(scenarioInfo);
#line 6
testRunner.Given("a new bowling game", ((string)(null)), ((TechTalk.SpecFlow.Table)(null)), "Given ");
#line 7
testRunner.When("I roll the following series: 2,7,3,4,1,1,5,1,1,1,1,1,1,1,1,1,1,1,5,1", ((string)(null)), ((TechTalk.SpecFlow.Table)(null)), "When ");
#line 8
testRunner.Then("my total score should be 40", ((string)(null)), ((TechTalk.SpecFlow.Table)(null)), "Then ");
#line hidden
this.ScenarioCleanup();
}
As you can see this contains your actual list of values and expected result in the code
This surely implies that you can't change the feature file and have the test update without recompiling, as otherwise any changes to the cs file will not be reflected in the actual dll that you are running.
How do I conditionally skip a scenario?
For example, I wish to continue a scenario only if certain conditions are met, but I do not want it to register as a failure if it's not present.
This is an issue I had. The tests I write are against a UI that has a constantly changing BE database that I am currently unable to have static data in.
This means that some times it is possible that there is no data for the test.
Not a pass not a fail, just unable to run.
The way that I found to work best was to invoke a cucumber pending.
example test:
Scenario: Test the application
Given my application has data
When I test something
Then I get a result
example step def:
Given /^my application has data$/ do
pending unless application.has_data?
end
These are the kind of results I can see:
201 scenarios (15 pending, 186 passed)
1151 steps (15 pending, 1136 passed)
It's worth noting that I have extra debugging and have these tests tagged so that at any time I can run these pending tests again.
Hope this helps,
Ben.
For anyone still looking for an answer to this:
Apart from using pending, or a specific profile to skip scenarios with certain tags, there are at least 2 more ways to achieve this.
I can understand why you would need this, as I had a similar problem and got a solution, hence worth sharing.
In my case, I had a piece of functionality expected to be available on 3/10 devices, and expected to be not available on the remaining 7.
Caveats with using 'pending' to skip:
Since the tests and code were implemented, it didn't feel right to mark steps as pending.
It caused confusion, as it was difficult to distinguish really pending scenarios from skipped but marked pending scenarios at the end of a run
Some CI jobs (Jenkins/Hudson) might be configured to fail for pending scenarios, hence causing more trouble.
So, I rather wanted to just skip them during execution depending on the condition of which browser is being used. I also didn't want to have too many profiles specific to certain browsers/devices
Solution:
Use cucumber.yml to skip tagged scenarios conditionally
Here's a known ignored interesting fact about cucumber (from https://github.com/cucumber/cucumber/wiki/cucumber.yml):
The cucumber.yml file is preprocessed by ERb; this allows you to use ruby code to generate values in the cucumber.yml file
Building on this, tag your scenarios with something unique, say #conditional
At the beginning of your cucumber config (cucumber.yml), apply your conditional logic outside of any profiles mentioned:
<% included = (ENV['BROWSER'] =~ /chrome/) ? "-t #conditional" : "-t ~#conditional" %>
included is just a variable, which will have a value of tags to include/exclude depending on the condition
Now use this conditional variable in the default profile
default: <%= included %>
So now your default profile will use the included/excluded tests as identified by your conditional logic.
(More complicated and not elegant) Use rake tasks for cucumber execution:
Conditionally choose tags to include/exclude within your rake task, and pass them to cucumber execution.
Hope this helps.
You could check the condition before you start cucumber, then use a profile that would skip the scenarios with certain tags. Put this in your cucumber.yml:
default: --tags ~#wip --tags ~#broken --no-source --color
limited: --tags #core --tags ~#wip --tags ~#broken --no-source --color
Replace #core with whatever tag you use for the cukes you want to run (or use ~ to exclude cukes). Then run the limited profile from a shell script that checks the conditions:
cucumber -p limited
Please see this solution which truly skips the scenario instead of trowing a pending error:
Before do |scenario|
scenario.skip_invoke!
end
I am tagging my scenarios, and then in my "step_definitions/hooks.rb" file, I have something like this:
Before('#proxy') do
skip_this_scenario unless proxy_running?
end
scenario.skip_invoke! which was mentioned in another answer seems to be deprecated.
When i run cucumber it displays the
possible steps that i should define, an example from the RSpec book:
1 scenario (1 undefined)
4 steps (4 undefined)
0m0.001s
You can implement step definitions for undefined steps with these snippets:
Given /^I am not yet playing$/ do
pending
end
When /^I start a new game$/ do
pending
end
Then /^the game should say “Welcome to CodeBreaker”$/ do
pending
end
Then /^the game should say “Enter guess:”$/ do
pending
end
Is there a way that it will automaticly create the step definitions file, so i don't have to
rewrite or copy paste by hand but i can just to customize them to be more generic?
Cucumber doesn't offer this feature. Probably because you would have to tell it where to put the step definitions file, and what to name it.
Like Kevin said, Cucumber would have to know the name of the file to put it in, and there are no good defaults to go with, other than using the same file name as the feature file. And that is something I consider an antipattern: http://wiki.github.com/aslakhellesoy/cucumber/feature-coupled-steps-antipattern
Intellij Idea or RubyIDE does exactly what you are asking for:
Detects missing step definitions
Creates missing step definitions in a new file (you choose the name of the file) or in one of the existing step definition files
Highlights matched step parameters
see http://i48.tinypic.com/10r63o4.gif for a step by step picture
Enjoy
There is a possibility this kind of feature could be useful, but as Kevin says, it doesn't exist at present. But it could also get quite messy, quite quickly.
Maybe you already do this, but there's nothing stopping you cut and pasting the output direct into your text editor, or even piping the output direct to your text editor if you're so inclined. Then at least you're getting pretty much most of the way there, bar creating the file and naming.
try this https://github.com/unxusr/kiwi it auto generate your feature file and make the step definitions file for you and you just fill in the steps with code.
In later version will write the code of the steps and run the test all of that automagically
You can use a work around way to generate steps file
all you have to do is to run the Cucumber on a feature doesn't have defined steps by identify a specific feature as the following command:
1) using path
bundle exec cucumber {PATH}
note path would start with features/....
for example
features/users/login.feature
1) using tags
bundle exec cucumber --tags=#{TAG}
note tag should be above your scenario in the steps file
for example
#TAG
Scenario:
And you will have the suggested steps in the console with pending status