How to add more result matrix in our summary report - maven

As part of the continuous integration task I'm owning, I have been able to add lots of selenium WebDriver scripts in Java and Eclipse. Basically, using a maven project, I have written various java scripts using Selenium and am now able to run the automated tests using Jenkins after the nightly build is deployed. In addition, I am able to successfully create a surefire-report.html. Howoever, this report only contains the following matrix:
Tests Errors Failures Skipped Success Rate Time
38 4 2 18 36.842% 746.477
The development manager has requested to see the following matrix as well:
TRR = (Number of tests run) / (Total number of tests available)
TPR = (Number of test passed) / (number of tests run)
Is there any way I can modify the surefire report in order to add the above matrix results as well?

Look for PerfPublisher plugin in Jenkins.
https://wiki.jenkins-ci.org/display/JENKINS/PerfPublisher+Plugin
This will by default give you the following things:
1. Number of tests run
2. Number of total tests.
3. Number of tests passed and failed.
So you have all required values there :)
The only change you would need to do is to create an XML file of particular format:
<report name="{REPORT_NAME}" categ="{CATEGORY_NAME}">
<test name="{TEST_NAME1}" executed="{EXECUTED_STATUS}">
<result>
<success passed="{RESULT_STATUS}" state="{RESULT_STATE}"/>
</result>
</test>
<test name="{TEST_NAME2}" executed="{EXECUTED_STATUS}">
<result>
<success passed="{RESULT_STATUS}" state="{RESULT_STATE}"/>
</result>
</test>
</report>
Hope this is useful.

Related

Get jacoco report from running acceptance test on real application

I'm new to sonar and jacoco and I wasn't able to find an ansewer to following question
We are going to use sonar and jacoco to analyze our test coverage.
We are going to have three kind of test: unit test, integration test (use spring boot test) and acceptance test that we will run on real application instance. We want to merge test results. To generating jacoco files for unit and integration tests is not a problem as they have access to source code. And we can merge this reports in sonar.
My question is it possible to generate jacoco file for acceptance test which interacts with real application? And maybe you will have a link to how to do it?
My best idea for now is to run acceptance tests twice on both real application and embedded one, and get report from embedded. But maybe there is better solution.
Thanks,
JaCoCo is able to record any kind of execution of Java application - you just need to start this application with agent, for example:
java -javaagent:jacoco-0.8.0/lib/jacocoagent.jar -cp classes Main
By default at termination of application this will produce file jacoco.exec with execution data. There are also ways to get execution data from running application.
And there are plenty of examples all over the internet of using agent in various cases - with Spring Boot, Tomcat, Weblogic, etc.
After that report can be generated using this execution data jacoco.exec and exactly the same class files that were used at runtime for generation of execution data. For example using JaCoCo command line interface (there are also Ant tasks, Gradle plugin, Maven plugin):
java -jar jacoco-0.8.0/lib/jacococli.jar \
report \
jacoco.exec \
--classfiles classes \
--html report
Report can be generated without source files, but in this case you won't be able to drill down lower than method level granularity:
Report with source files:
jacoco-0.8.0/lib/jacococli.jar \
report \
jacoco.exec \
--classfiles classes \
--html report \
--sourcefiles src
Now about SonarQube: it performs analysis of exec and class files, and so also requires that class files are the same as where used at runtime for generation of exec file. So you need to guarantee that SonarQube analysis is performed for exactly the same class files as under testing.

Go Coverage Report Integration with Jenkins

I wonder if there's any tool that can convert go test -coverprofile=cover.out into the formats that Jenkins can accept? I found some tools like go-junit-report and go2xunit, but they actually just convert output from go test -v, which is not the coverage report.
I want to know the detailed test coverage in Jenkins directly. Basically, I want to see the output from go tool cover -func=cover.out and go tool cover -html=cover.out in Jenkins webpage.
There are several go tools for converting coverage data from go test to Cobertura for Jenkins: gocover-cobertura, or gocov with gocov-xml.
You can use gocover-cobertura as follows:
$ go get github.com/t-yuki/gocover-cobertura
$ go test -coverprofile=cover.out example.com/demo/...
ok example.com/demo 0.008s coverage: 0.0% of statements
ok example.com/demo/cmd/demo 0.020s coverage: 23.4% of statements
$ gocover-cobertura < cover.out > coverage.xml
$ head coverage.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE coverage SYSTEM "http://cobertura.sourceforge.net/xml/coverage-04.dtd">
<coverage line-rate="0.35415787" branch-rate="0" version="" timestamp="1520609235359" lines-covered="839" lines-valid="2369" branches-covered="0" branches-valid="0" complexity="0">
<sources>
<source>/usr/local/go/src</source>
<source>/Users/wilfred/workspace/go-scratch/src</source>
</sources>
<packages>
<package name="example.com/demo/cmd/demo" line-rate="0.4848485" branch-rate="0" complexity="0">
<classes>
Note that you'll need Go 1.10+ to run -coverprofile against multiple packages in one run.
https://github.com/AlekSi/gocov-xml
This is what we need. Use Coberuta plugin to generate the coverage profile.
There isn't a dedicated plugin for Go coverage reports, nor really for generic code coverage.
For reports like this, I use the HTML Publisher Plugin to publish .html files created during a build.

JavaScript Unit Tests not working on SonarQube

I have below problems with SonarRunner.
SonarQube along with Sonar runner unable to pull junit format xml reports
Unit Tests or Test Coverage widget doesn't show up. It says No Data.
I am following instructions described here
I manually created report file in XML format as described, but still no luck.
Below is the XML file - TEST-Firefox_210_Mac_OS.com.company.BarTest.xml,
<testsuite name="Firefox_210_Mac_OS.com.company.BarTest" errors="0" failures="0" tests="3" time="0.0">
<testcase classname="Firefox_210_Mac_OS.com.company.BarTest" name="testfullName" time="0.0"/>
</testsuite>
To pull the Unit Test execution report to show on sonar dashboard. I have used sample git project from here
and below is my sonar-project.properties,
# project metadata (required)
sonar.projectKey=org.codehaus.sonar:javascript-sonar-runner-jstestdriver
sonar.projectName=JavaScript project with Sonar Runner reusing reports generated by JsTestDriver
sonar.projectVersion=1.0
# path to source directories (required)
sonar.sources=C:/Sonar/sonar-runner-dist-2.4/sonar-runner-2.4/projects/sources
# path to tests source directories (required)
sonar.tests=C:/Sonar/sonar-runner-dist-2.4/sonar-runner-2.4/projects/tests
sonar.javascript.jstestdriver.reportsPath=C:/Sonar/sonar-runner-dist-2.4/sonar-runner-2.4/projects/target/TEST-Firefox_210_Mac_OS.com.company.BarTest.xml
sonar.sourceEncoding=UTF-8
below is my jsTestDriver.conf
server: http://localhost:9876
load:
- C:/Sonar/sonar-runner-dist-2.4/sonar-runner-2.4/projects/sources/*.js
- C:/Sonar/sonar-runner-dist-2.4/sonar-runner-2.4/projects/sources/com/company/*.js
test:
- C:/Sonar/sonar-runner-dist-2.4/sonar-runner-2.4/projects/tests/*.js
- C:/Sonar/sonar-runner-dist-2.4/sonar-runner-2.4/projects/tests/com/company/*.js
plugin:
- name: "coverage"
jar: "coverage-1.3.5.jar"
module: "com.google.jstestdriver.coverage.CoverageModule"
My sonarqube is running on port : 9000 and below is the screen shot. As you see SonarRunner and jsTestDriver just doing a code analysis and not showing any unit tests.
SonarQube doc website says jsTestDriver will run the javascript unit
tests and copy the results in target folder in XML format
SonarQube doesnt run your Unit Tests, it just gathers the reports generated from your manuall run or other tools automatic run (like: Jenkins).
From SonarQube doc:
Prior to the SonarQube analysis, execute your unit tests in order to
generate XML report. The JUnit like XML format supported is the one
generated by js-test-driver
Then I manually created report file in XML format as described, but
still no luck.
If you created manually the report files and reports are still not showing, check your paths if ok.Check if sonar is really reading data from C:/Sonar/sonar-runner-dist-2.4/sonar-runner-2.4/projects/ path

TestNG, running different tests depending on version of application

Question: how to execute different tests from the same test suite, depending on parameter sent from Jenkins (version)?
Background: have application and 5 versions of it.
The difference between them is small, but exist and it is implemented in testcases.
So I have group of testcases which are the same from version to version and I have groups of test cases which are different from version to version.
What by Boss wants: we should be able to select application version number in Jenkins right in the same place where we selecting against which server to execute and what browser to use.
So I have in my text.xml
<suite name="aaaa">
<parameter name="SiteAddress" value="${SiteAddress}"/>
<parameter name="Version" value="${version}"/>
in POM.xml
<systemPropertyVariables>
<SiteAddress>${serv_url}</SiteAddress>
<browser>${browser}</browser>
<Version>${version}</Password>
</systemPropertyVariables>
Question again:
what is pattern to switch test cases depending of parameter value?
I just can't figure it out how to implement it.
Thank you!
This how I implemented a worflow in Jenkins by creating testng xml programmatically and use the command line args for filtering tests for execution.
Jenkins calls Ant build.xml which in turn calls the Java class, where I create testng xml programmatically and the command line args are passed from jenkins > ant > java class. Let me know if you need more help on this.

cruisecontrol mstest problem wenn tests fail

i use MSTest with CruiseControl. When I run my compile and tests and all tests pass then I can see the mstest results in my dashboard.
for example: 5 tests ran. 5 tests passed. 0 tests failed.
when tests fail I would expect to see something like this:
5 tests ran. 1 failed. 4 passed.
but if one test fails I get this:
0 tests ran. 0 passed. 0 failed.
It looks like cruisecontrol has a display problem if tests fail.
bye bembi
solved it.
i had to move the file merge block out of the task block and into a publishers block:
<project>
...
<publishers>
<merge>
<files>
<file>C:\cc\mstestResult.trx</file>
</files>
</merge>
<xmllogger />
</publishers>
</project>

Resources