I've been working with Cucumber and Watir for the past few months locally to run some regression tests on our application. I have several Cucumber feature files in one repository with one step file and one environment file. Recently I have moved from running this locally to running it on Jenkins along with the cucumber-jvm reports plugin.
I set the build steps up with the following:
cucumber --format json all/features/publish.feature > result_publish.json
cucumber --format json all/features/signin.feature > result_signin.json
cucumber --format json all/features/reports.feature > result_reports.json
When there are no failures with the tests all feature files run successfully one after the other. However, if there is a failure with the first test the build will fail and the subsequent tests will not run.
Is there any way I can force all feature files to run even if one of them fails?
Put the features in the same folder and run
cucumber --format json all/features/integration -o results.json
This will create one single report with all the tests and will run all the features regarding if they fail or not
Related
I'm trying to get our automation environment set up to run headless through Jenkins on a linux server. The automation is set up through Ruby and uses Cucumber. I've run into an issue trying to get the feature files to run.
I have several plug ins for Jenkins, Xfvb, Cucumber reports, Cucumber-plugin.
I have setup the code in Ruby set up correctly (at least I believe so) for one of the feature files to run as headless.
But I can't quite get to test that specific file. When I try to run the build after enable the Cucumber-plugin it tries to run a different feature file. It seems to want to run the entire project. I've tried to add a new profile in my cucumber.yml (name jenkins) file and specify a tag like below:
jenkins: --tags #DisposalFeeService --no-source --format pretty --format html --out results.html --format json --out reports.json
I only have 1 feature file that is tagged with that tag and my build runs a completely different feature file. The feature is failing for good reasons, nothing to do with the Jenkins set-up, it would fail anyways. I also haven't set up all the features to run as headless, nor do I want to at this time. I'm just trying to show that I can get a feature file to run in through Jenkins.
If it is helpful, here is my cucumber-plug in in Jenkins: Cucumber Jenkins plug in
I'm running protractor flake on Jenkins for automation testing using the framework protractor cucumber. Each time the tests run, a json is outputted. Is there a way to delete the old json output or a bash command that can detect if protractor flake is rerun. I have protractor flake set to have 3 attempts. So if protractor flake reruns the 3rd time, I don't want the 1st and 2nd json output from protractor cucumber to be in the folder.
That way the cucumber report won't have flake tests in it.
This is the command I have right now on Jenkins when I do a build:
rm -rf e2e/reports
mkdir e2e/reports
npm install
node flake e2e/staging.protractor.conf.js
There is an onCleanup field/function in the protractor config. I recommend using that to write yourself a function that will delete the files in /e2e/reports. You can use the exitcode to delete the file if the first or second attempt fails. This will be a little tricky as you will need to find a way to keep track of which iteration you are on so that the report doesn't get deleted if the third run fails also.
See the comments in the protractor config for more details.
I would like to skip the build process and directly run tests when I do
./gradlew connectedDevDebug
I could also use adb command but it will run test on only one deivce at a time.
There is currently no gradle tasks that just runs the tests.
But once you have installed (and run) the tests, you can (re-)run them directly with the am instrument command. To start them from your development machine's command line just run:
adb shell am instrument -w <test_package_name>/<runner_class>
You can copy the actual command from AndroidStudio's output. Just run the tests from AndroidStudio, and then scroll to the top of the test log view.
There is one caveat, am instrument does not create any test-reports. All test results are written to stdout. However you could pipe stdout into a file and create a report yourself. I.e. this tool can create an xml JUnit test report from the output of am instrument.
If you have made changes to the tests you can rebuild and install them with:
./gradlew installDebugAndroidTest
For detailed information and instructions about starting tests from the command line you can refer to the official article Test from the Command Line
I am building a CI pipeline with Bamboo and I run automated tests using Appium. I have successfully run the Appium server and test scripts but I would like to have results in for example xml file so I could pass it to Bamboo. Do you know any way to achieve this?
it might be a bit late to respond on your question, but I'm using Appium with node.js and mocha to write my tests. To create a proper xml with the results I'm doing:
mocha --recursive -R xunit test/ > test-reports.xml
(test/ is the folder containing all my test files). This is generating a proper xml. I haven't tested it with Bamboo but I assume it's capable of reading XUnit test reports.
I have a suite of tests that are launched in continuous integration using parallel test. After the deploys the environment is a little unstable, so when the suite is finished, the failed tests are relaunched (by this moment the environment is stable again).
Basically, there is only one jenkins build, and it will execute this:
task :jenkins1_with_rerun do
selenium_successful = system "bundle exec parallel_cucumber features/web/ -o \"-p jenkins1\" -n 3"
p 'start rerun'
rerun_successful = run_rake_task("features:jenkins_rerun")
unless selenium_successful || rerun_successful
raise 'Cucumber tests failed'
end
end
The first execution launches the suite with parallel tests. Here is the cucumber profile:
jenkins1:
<%= std_opts %>
HEADLESS=true
--tags #part1
--tags ~#wip
--format junit
--out log/junit
--format ParallelTests::Cucumber::FailuresLogger
--out parallel_cucumber_failures.log
After this is finished. The second execution starts, launching the failed tests recorded in a file. The cucumber profile for this execution is this:
jenkins_rerun:
ENVIRONMENT=master
SCREENSHOT=true
HEADLESS=true
--format pretty
--no-source
--format junit
--out log/junit_rerun
#parallel_cucumber_failures.log
Now. Everything works. The only problem is that I use the junit report in jenkins to make graphs of failed and successful tests. I may have reports like this:
-------------------Tests---KO---OK
First execution---75----10----65
Rerun-------------10-------0----10
This is 100% green, because all the problems were caused by instability after the deploy. So I want the junit report in jenkins to said that 75 tests have been launched, 75 OK, 0 KO, or something equivalent.
Right now, the junit report of the first execution says that of 75 tests, I have 10 KOs; and the second junit says that of 10 tests, there are 0 KOs.
What may be a good solution to this? It possible to mix the results of both Junit reports?
I will also accept to be able to display in jenkins both junit reports, each one with a graph. But I think that jenkins only allows to show one junit report graph.
I finally resolved this problem.
The solution was to create a "parser" that searches (with RegEx) through the Junit files generated by the rerun, and then goes to the Jnuit files of the original/first run and rewrites the results of those tests that have passed during the rerun.
In the end, if everything is ok in the rerun, the junit graph is 100% green and there are not alerts of failures.