Currently I'm aware allure report have status like passed, failed, broken, skip and unknown
I need to add new custom status like info to my report. How do I achieve in Ruby
Allure doesn't provide way to add additional statuses to the report. But instead, you can use Categories feature that allows you add additional classification to each test failure.
Related
I am currently seeking information around the available reporting capabilities for OpenTest. I would need information regarding the following:
Portability of reports/logging - can these results be published in various formats
Granularity of reports/logging - is there a way of getting very detailed in what is reported on and/or strategies to ensure that enough information is logged to allow for debugging of automated tests and System Under Test(SUT)
Screenshots - is there current functionality to allow for screenshots to be taken and published/posted to external systems?
Portability of reports/logging
You can obtain the test session results by using the API, either in JSON format (which contains a lot of detail) or the JUnit XML format:
http://localhost:3000/api/session/<SESSION_ID>?format=json
http://localhost:3000/api/session/<SESSION_ID>?format=junit
The detailed log of the test session can be retrieved in: JSON or human-readable format:
http://localhost:3000/api/session/<SESSION_ID>/log?format=json
http://localhost:3000/api/session/<SESSION_ID>/log?format=pretty
Granularity of reports/logging
The test results in JSON format will tell you everything you need to know about the pass/fail status for each test and each individual test action within a test, arguments that were used for test actions, the name of the screenshot captured for each test action, execution times and many other useful information.
When you want to troubleshoot a failed test, most of the time you'll need the detailed log information, which can be retrieved using the APIs I mentioned above. Aside from the log information generated by OpenTest itself, you can always log additional information that is specific to your application or test scenario using the $log JavaScript API.
Screenshots
Screenshots are automatically captured for Web and UI tests, whenever a test action fails. If you need to capture additional screenshots during your test, you can do so using the TakeScreenshot keyword for either Web testing or mobile testing. You can also capture a screenshot after any test action by using the $screenshot global test action argument:
- description: Click product 1 and capture a screenshot
action: org.getopentest.selenium.Click
args:
locator: { id: product1 }
$screenshot: true
You can download screenshots using this API:
https://localhost:3000/api/screenshot/SID1554380072_WEB_T05_SG01_ST01_after_03.png
SID1554380072_WEB_T05_SG01_ST01_after_03.png is the name of the screenshot file, which you can find in the test execution results in JSON format.
Integrating with custom reporting solutions
At some point you will need to integrate with a dedicated reporting product that can give you all the nice features that OpenTest is not be able to provide out-of-the-box. This is possible to do using the APIs I described. In order to notify interested parties of the current status of test sessions, OpenTest also provides a WebSocket API. You can use that to be notified when a test session has finished, then you can extract all the information you need via APIs. You can find a Java project that does all that here: https://github.com/adrianth/opentest-monitor. This project is intended as a starting point for your own custom integration.
Went through the serenity documented for extracting the test outcomes
below isthe code, it didn't work
OutcomeFormat format = OutcomeFormat.XML; TestOutcomes outcomes =
TestOutcomeLoader.loadTestOutcomes().inFormat(format)
Tried with below code and its working,
OutcomeFormat format = OutcomeFormat.JSON; TestOutcomeLoaderBuilder
outcomes= TestOutcomeLoader.loadTestOutcomes().inFormat(format);
TestOutcomes out =outcomes.from(new File(""));
Issue is i need the test outcomes in #AfterScenario, but the thing is serenity reports gets generated after the entire execution tried changing the pom but didn't help. Is there any other way using which we can extract the test results?
Serenity uses JSON format by default now. Why are you trying to obtain the test outcomes? (i.e. what problem are you trying to solve?)
Created a separate java class for report extracting and added that in maven plugin which will get executed after serenity report is generated.
As #John smart has mentioned JSON and HTML are default output formats.
Still if you want to access Outcomes after test execution.
You can create a custom listener and listen to serenity event bus.
TestRunFinished event will be published with Outcome as a parameter.
You can use the outcome for getting required details.
For creating custom listener you can follow this page
in version 2.0.0 of maven-cucumber-reporting plugin is it possible to hide #Before and #After hooks from the report?
so for example in the html report I see:
Before BusinessStepDefs.prepare()
Scenario: Business scenario
Given business specific input
When performing action
Then result is success
After HelperStepDefs.cleanUp()
I am using the version 2.0.0, using an older version did not have those lines. Any ideas ?
I would like this to appear as
Scenario: Business scenario
Given business specific input
When performing action
Then result is success
See this issue for a discussion and possible resolution.
I am trying to figure out how to run unit tests, using Google Test, and send the results to TeamCity.
I have run my tests, and output the results to an xml, using a command-line argument --gtest_output="xml:test_results.xml".
I am trying to get this xml to be read in TeamCity. I don't see how I can get XML Reports passed to TeamCity during build/run...
Except through XML report Processing:
I added XML Report Processing, added Google Test, then...
it asks me to specify monitoring rules, and I added the path to the xml file... I don't understand what monitoring rules are, or how to create them...
[Still, I can see nowhere in the generated xml, the fact that it intends to talk to TeamCity...]
In the log, I have:
Google Test report watcher
[13:06:03][Google Test report watcher] No reports found for paths:
[13:06:03][Google Test report watcher] C:\path\test_results.xml
[13:06:03]Publishing internal artifacts
And, of course, no report results.
Can anyone please direct me to a proper way to import the xml test results file into TeamCity ? Thank you so much !
Edit: is it possible that XML Report Processing only processes reports that were created during build ? (which Google Test doesn't do?) And is ignoring the previously generated reports, as "out of date", while simply saying that it can't find them - or are in the wrong format, or... however I should read the message above ?
I found a bug report that shows that xml reports that are not generated during the build are ignored, making a newbie like me believe that they may not be generated correctly.
Two simple solutions:
1) Create a post build script
2) Add a build step that calls the command line executable with the command-line argument. Example:
Add build step
Add build feature - XML report processing
I had similar problems getting it to work. This is how I got it working.
When you call your google test executable from the command line, prepend %teamcity.build.checkoutDir% to the name of your xml file to set the path to it like this:
--gtest_output=xml:%teamcity.build.checkoutDir%test_results.xml
Then when configuring your additional build features on the build steps page, add this line to your monitoring rules:
%teamcity.build.checkoutDir%test_results.xml
Now the paths match and are in the build directory.
We're going to trigger hudson builds from powershell using curl and the build url:
http://hudson:port/job/build?parameters...
now I'm wondering if we can somehow pass a parameter that will add to the "triggered by" description for the builds started by our scripts?
I don't know how to edit the "triggered by" info on the build summary page without writing your own Hudson plugin. However, I like to add my own sections and badges to the build summary page with the Groovy Postbuild Plugin. In your case, you could use curl to pass an extra parameter string when starting your Hudson job. Then, you could have the Groovy Postbuild Plugin read that parameter and create a new icon + text string on your job's build summary page. That info could be the custom "triggered by" description that you'd like.