I want to be able to change my feature file from outside visual studio and have the updated feature file picked up, for subsequent test execution, without compilation of my test project. Is it possible to do this? Can someone help to specify the exact steps needed to do this? I am using MsTest.
Here are the steps I followed, but I get the message "No tests to execute." every time:
Change Test Project file (.csproj) as mentioned here
Build the Test DLL from Visual Studio
Kept the feature file in a folder FeatureFiles, under the Test release folder
Changed the feature file in Notepad
Used the Specflow generate all command, to regenerate the tests:
Specflow generateall TestProject.csproj /force /verbose
Create the report:
mstest /testcontainer:Test.Dll /resultsfile:TestResult.trx
A similar question was asked earlier, and I am following the same steps as mentioned by Marcus there.
Update
Here is what I would like to do. Considering the following .feature file:
Feature: Score Calculation
As a player
I want the system to calculate my total score
So that I know my performance
Scenario: Another beginners game
Given a new bowling game
When I roll the following series: 2,7,3,4,1,1,5,1,1,1,1,1,1,1,1,1,1,1,5,1
Then my total score should be 40
In the above feature file, I would like to change the data series of numbers and change the total score and run the same test again to check if it runs fine and I get a correct score
If you only want to change the data for a test, then go with a CSV file that you save inside your VS project. Then in a Given statement open that file, parse it and save it to the ScenarioContext.Current object for use in subsequent steps.
I can't see how what you want is possible. SpecFlow generates unit test classes from the feature file. These unit test classes MUST be compiled before they can be run, the same as ANY code change.
You are getting a 'no tests to execute' because you are still running against the old version of the dll, which doesn't contain any tests because you have not rebuilt it with the new code that specflow has generated.
From what is stated in the linked question it seems they are saying that some changes may be able to be run without recompiling, but honestly I cannot see how that is possible, and even if some feature file changes were possible I imagine that the solution would be specific to certain unit test frameworks and unsupported.
EDIT
When I look at the generated feature.cs file you have this:
[Microsoft.VisualStudio.TestTools.UnitTesting.TestMethodAttribute()]
[Microsoft.VisualStudio.TestTools.UnitTesting.DescriptionAttribute("Another beginners game")]
[Microsoft.VisualStudio.TestTools.UnitTesting.TestPropertyAttribute("FeatureTitle", "Score Calculation")]
public virtual void AnotherBeginnersGame()
{
TechTalk.SpecFlow.ScenarioInfo scenarioInfo = new TechTalk.SpecFlow.ScenarioInfo("Another beginners game", ((string[])(null)));
#line 5
this.ScenarioSetup(scenarioInfo);
#line 6
testRunner.Given("a new bowling game", ((string)(null)), ((TechTalk.SpecFlow.Table)(null)), "Given ");
#line 7
testRunner.When("I roll the following series: 2,7,3,4,1,1,5,1,1,1,1,1,1,1,1,1,1,1,5,1", ((string)(null)), ((TechTalk.SpecFlow.Table)(null)), "When ");
#line 8
testRunner.Then("my total score should be 40", ((string)(null)), ((TechTalk.SpecFlow.Table)(null)), "Then ");
#line hidden
this.ScenarioCleanup();
}
As you can see this contains your actual list of values and expected result in the code
This surely implies that you can't change the feature file and have the test update without recompiling, as otherwise any changes to the cs file will not be reflected in the actual dll that you are running.
Related
This is a how-to/best-practice question.
I have a code base with a suite of unit tests run with pytest
I have a set of *.rst files which provide explanation of each test, along with a table of results and images of some mathematical plots
Each time the pytest suite runs, it dynamically updates the *.rst files with the results of the latest test data, updating numerical values, time-stamping the tests, etc
I would like to integrate this with the project docs. I could
Build these rst files separately with sphinx-build whenever I want to view the test results [this seems bad, since it's labor intensive and not automated]
tell Sphinx to render these pages separately and include them in the project docs [better, but I'm not sure how to configure this]
have a separate set of sphinx docs for the test results which I can build after each run of the test suite
Which approach (or another approach) is most effective? Is there a best practice for doing this type of thing?
Maybe take a look into Sphinx-Test-Reports, which reads in all information from junit-based xml-files (pytest supports this) and generates the output during the normal sphinx build phase.
So you are free to add custom information around the test results.
Example from webpage:
.. test-report:: My Report
:id: REPORT
:file: ../tests/data/pytest_sphinx_data_short.xml
So complete answer to your question: Take none of the given approaches and let a sphinx-extension do it during build-time.
I have two cucumber feature ( DeleteAccountingYear.feature and AddAccountingYear.feature).
How can i do to make that the second feature(AddAccountingYear.feature) run before the first one (AddAccountingYear.feature).
I concur with #alannichols about tests being independent of each other. Thats a fundamental aspect of automation suite. Otherwise we will end up with a unmaintainable, flaky test suite.
To run a certain feature file before running another feature appears to me like a test design issue.
Cucumber provides few options to solve issues like this:
a) Is DeleteAccountingYear.feature really a feature of its own? If not you can use the cucumber Background: option. The steps provided in the background will be run for each scenario in that feature file. So your AddAccountingYear.feature will look like this:
Feature: AddingAccountingYear
Background:
Given I have deleted accounting year
Scenario Outline: add new accounting year
Then I add new account year
b) If DeleteAccountingYear.feature is indeed a feature of its own and needs to be in its own feature file, then you can use setup and teardown functions. In cucumber this can be achieved using hooks. You can tag AddDeleteAccountingYear.feature with a certain tag say #doAfterDeleteAccountYear. Now from the Before hooks you can do the required setup for this specific tag. The before hooks(for ruby) will look like:
Before('#doAfterDeleteAccountYear') do
#Call the function to delete the account year
end
If the delete account year is written as a function, then the only thing required is to call this method in the before hook. This way the code will be DRY compliant as well.
If these options doesn't work for you, one another way of forcing the order of execution is by using a batch/shell script. You can add individual cucumber commands for each feature in the order you would like to execute and then just execute the script. The downside of it is different reports will be generated for each feature file. But this is something that I wouldn't recommend for the reasons mentioned above.
From Justin Ko's website - https://jkotests.wordpress.com/2013/08/22/specify-execution-order-of-cucumber-features/ the run order is determined in the following way:
Alphabetically by feature file directory
Alphabetically by feature file name
Order of scenarios within the feature file
So to run one feature before the other you could change the name of the feature file or put it in a separate feature folder with a name that alphabetically first.
However, it is good practice to make all of your tests independent of one another. One of the easiest way to do this is to use mocks to create your data (i.e. the date you want to delete), but that isn't always an option. Another way would be to create the data you want to delete in the set up of the delete tests. The downside to doing this is that it's a duplication of effort, but it won't matter what order the tests run in. This may not be an issue now, but with a larger test suite and/or multiple coders using the test repo it may be difficult to maintain the test ordering based solely on alphabetical sorting.
Another option would be to combine the add and delete tests. This goes against the general rule that one test should test one thing but is often a pragmatic approach if your tests take a long time to run and adding the add data step to the set up for delete would add a lot of time to your test suite.
Edit: After reading that link to Justin Ko's site you can specify the features that are run when you run cucumber, and it will run them in the order that you give. For any that you don't care about the order for you can just put the whole feature folder at the end and cucumber will run through them, skipping any that have already been run. Copy paste example from the link above -
cucumber features\folder2\another.feature features\folder1\some.feature features
We are working with some huge Visual Studio 2010 solutions with a great many static library projects shared between the two. After building either of these solutions the other solution will complain that many (but not all) of the projects are out of date, though actually building naturally does next to nothing and is almost instant.
After following the steps on this question on debugging MSBuild project dependencies I can see many lines of the following messages indicating that the projects are considered out of date because of a "different evaluation fingerprint":
[8444] Project not up to date because the last build has different evaluation fingerprint.
[8444] devenv.exe Information: 0 :
[8444] Project not up to date because the last build has different evaluation fingerprint.
[8444] devenv.exe Information: 0 :
I have come up completely blank while trying to find out what an MSBuild evaluation fingerprint is, where they come from, or what could cause them to be off like this.
Creating new project files is a non-starter given their shear size, the complexity of their configuration requirements, and the lack of enough time in our schedule for cleaning up small annoyances like this.
What are MSBuild evaluation fingerprints and how are they determined?
I assume you are using C++?
The issue is that the solution directory which was used to build the project is saved to to "*.lastbuildstate" file.
It's just a plain text file which look like:
#v4.0:v100:false
Debug|Win32|C:\MyPath\ToSolution\|
The second line is used as project evaluation fingerprint.
First line consist of following information (see line 246 in Microsoft.CppBuild.targets)
#$(TargetFrameworkVersion):$(PlatformToolSet):$(EnableManagedIncrementalBuild)
Second line (see line 38 in Microsoft.CppBuild.targets)
$(Configuration)|$(Platform)|$(SolutionDir)|
Microsoft.CppBuild.targets can be found at %ProgramFiles (x86)%\MSBuild\Microsoft.Cpp\v4.0\Microsoft.CppBuild.targets
Maybe its possible to modify / extend the MSBuild scripts not to use the SolutionDir in the lastbuildstate file.
I am currently engaged in a large C#.NET 4.0 project and am doing so with a TDD aproach.
In our unit tests we have adopted a naming pattern based on the one in Roy Overshore's "The Art of Unit Testing" book. Essentially for each class "XXX" we have a coresponding "XXXFacts" test class and each [TestMethod] method in that class is named with a pattern of "[Method/Prop name][sate/result][preconditions]" for example "AccessLevel_IsInvalid_WhenNotAuthenticated"
Now intially to see the test results I just configured the test results window to add the ClassName column which looks OK, but takes up much horizonal screen real estate as you can see;
I then discovered the "Group By" option in the window. This does as the name suggests and grouped the output by class name, so I can remove the repeating coulmn and gain more room for any eror message text.
however every time I run my tests the view I am given has the group by contents collapsed like this;
What I would like to do is somehow configure the Visual Stuidio 2010 Test Results Window to automatically expand the "Group by" sections after a test run so it looks like this immediatley after a test run;
At the very least if it could expand a group that contais a FAILED test that would be a massive plus.
I know this is just plain lazy but opening all those groups after every run on a TDD project is already getting old!
I have looked but failed to find a configuration option for this in the tools dialog, but perhaps I i've been looking in the wrong place. I do hope so.
When i run cucumber it displays the
possible steps that i should define, an example from the RSpec book:
1 scenario (1 undefined)
4 steps (4 undefined)
0m0.001s
You can implement step definitions for undefined steps with these snippets:
Given /^I am not yet playing$/ do
pending
end
When /^I start a new game$/ do
pending
end
Then /^the game should say “Welcome to CodeBreaker”$/ do
pending
end
Then /^the game should say “Enter guess:”$/ do
pending
end
Is there a way that it will automaticly create the step definitions file, so i don't have to
rewrite or copy paste by hand but i can just to customize them to be more generic?
Cucumber doesn't offer this feature. Probably because you would have to tell it where to put the step definitions file, and what to name it.
Like Kevin said, Cucumber would have to know the name of the file to put it in, and there are no good defaults to go with, other than using the same file name as the feature file. And that is something I consider an antipattern: http://wiki.github.com/aslakhellesoy/cucumber/feature-coupled-steps-antipattern
Intellij Idea or RubyIDE does exactly what you are asking for:
Detects missing step definitions
Creates missing step definitions in a new file (you choose the name of the file) or in one of the existing step definition files
Highlights matched step parameters
see http://i48.tinypic.com/10r63o4.gif for a step by step picture
Enjoy
There is a possibility this kind of feature could be useful, but as Kevin says, it doesn't exist at present. But it could also get quite messy, quite quickly.
Maybe you already do this, but there's nothing stopping you cut and pasting the output direct into your text editor, or even piping the output direct to your text editor if you're so inclined. Then at least you're getting pretty much most of the way there, bar creating the file and naming.
try this https://github.com/unxusr/kiwi it auto generate your feature file and make the step definitions file for you and you just fill in the steps with code.
In later version will write the code of the steps and run the test all of that automagically
You can use a work around way to generate steps file
all you have to do is to run the Cucumber on a feature doesn't have defined steps by identify a specific feature as the following command:
1) using path
bundle exec cucumber {PATH}
note path would start with features/....
for example
features/users/login.feature
1) using tags
bundle exec cucumber --tags=#{TAG}
note tag should be above your scenario in the steps file
for example
#TAG
Scenario:
And you will have the suggested steps in the console with pending status