I am trying to run multiple feature files through maven goal(command line) but after 2 feature files that run successfully, the other feature files (3rd one onwards) fails in some test cases which when ran independently passes all the test cases.
So f I run each feature file individually I get proper results but running them all together gives wrong results.
We are using serenity framework with cucumber jvm. Please help how can we resolve this issue.
Your failing tests fail to fully setup the context. Some state is leaking from the previous ones. Look for what has changed during the first runs (database/mocks/whatever state) that has to be reset before running the third and following.
Related
I am writing test cases in Junit, and when i build my application, i want to see in my logs which test cases has failed, succeeded , skipped and number of test cases run.
So how can we implement this.
When you build an application you probably run some build tool (Maven, Gradle) to do so.
Maven for example has a plugin (called surefire) for running the tests.
And it also produces a report with all the failures/ successfully run tests.
Since maven also establishes a well-known layout of directories, you'll find this report in the target/surefire-reports folder.
There are even tools that can parse the information about the tests execution and show it in a graphical way in CI tools for example
Some of the steps that i have done are described below:
Setup Jenkins on remote linux server.
Used own mac as a slave to
run Xcode build
I have setup a Web hook on bitbucket that runs a
build on Jenkins server.
Build gets triggered when i push code to
repo.
I have shared scheme in the Xcode project.
But whenever i push failing tests on my repo the build passes..Shouldn't CI server fail the build..I dont know where am i missing..I have posted some screenshots for clear reference
EDIT: with more research i came to know that build fails but you need to see the test results/reports that fails.How can i see the reports? I cant see any xml files under reports section
Even I have faced the same issue where build was successful even though test cases were getting failed. You can use Log Parser plugin to parse the console output and mark your build as either stable or failed based on the output.
The jenkins runs in a step by step fashion.Build succeeds means your build step is correct and there's no issue with that but it does not mean your test cases have also passed.You need to check that with Junit if your test cases pass or not.
In your execute shell phase, you might need to add set -exo pipefail at the top.
I've been trying to get Jmeter load tests to run in VSTS thus far without avail. I've been back and forth (very slowly!) with the Microsoft support team about this, but as the issues are ironed out I would like to at least run a small set of load tests on our build machine using Jmeter and then have the results uploaded somehow to VSTS so they are easier to track. I have part 1 of this working: From the VSTS release definition I run a batch file that runs the load tests locally, and then generates an aggregate spreadsheet with results.
The question is - how can I get those results loaded into VSTS?
In our case we had to export the results to xml using the jmeter.test.xmlouput configuration. Then we had a script to transform the xml in a proper Xunit result file and we finally used a publish test results to gather this file and add the results to the release. (this approach would work with build definitions too).
It's a little bit complicated, requires some scripting and surely would be easier if a dedicated task was available.
I have a big test suite written in TestNG that takes several hours to complete.
I'd like for maven/surefire to create a new suite that is a copy of the first but with just the failed tests in it, a suite that should be much faster to run.
Is it possible to create such a suite?
As a fall back I could create it on my own from a test report that is easy to parse, if there is such report.
Thank you.
On completion of run, testng generates a testng-failed.xml (in the same output folder as your reports), which basically is your initial suite file with the listeners, but the tests part contains only the failed testcases.
In case you are using Jenkins, you might consider adding a postbuild step that triggers another build that works on the same workspace as the current build and uses this failed xml. Or depending on how you are triggering your tests, you might look at writing a script to run the failed xml.
We've recently found that an acceptance test project fails occasionally on our build server- due to our web drivers Type is not resolved error. I'm trying an experiment to see if its a question of timing of build steps. To this end I've tried to create a separate build step which launches the webdriver executable separately and then proceed onto the unit tests - the issue I have is when I launch the process it blocks the next step after it has successfully started.
eg. Type is not resolved for member 'OpenQA.Selenium.WebDriverException,WebDriver, Version=2.41.0.0, Culture=neutral, PublicKeyToken=null'..
Is there a way I can progress to the next build step after I call an exe?
Thanks
Do you have multiple build agents? Have you considered splitting this into separate build configurations and having triggers on previous build configurations?
You could have configurations:
Compile,
Webdriver, triggered on successful compile completion
Unit Tests, triggered on successful compile completion.
It's not nice, but it should give you the non-blocking behavior you're after. I can't see any other way of making it happen, build configurations are supposed to run as a linear sequence.