Is there a way to include deselected tests in the pytest generated html report, similarly to skipped tests?
generated html file: file://///////reports/report.html -
================ 13 passed, 41 skipped, 6 deselected in 35.54s =================
I found a solution to have them printed in the terminal by adding this to conftest.py
def pytest_deselected(items):
if not items:
return
config = items[0].session.config
reporter = config.pluginmanager.getplugin("terminalreporter")
reporter.ensure_newline()
for item in items:
reporter.line(f"deselected: {item.nodeid}", yellow=True, bold=True)
but can't figure out how to get these tests listed in the report
I ended up marking them as skipped instead of deselecting them, that way they are easily listed on the report.
Related
This question already has an answer here:
TestNG shows 0 Test run
(1 answer)
Closed 4 years ago.
Default suite
Total tests run: 0, Failures: 0, Skips: 0
Configuration Failures: 1, Skips: 0
===============================================
[TestNG] No tests found. Nothing was run
Usage: <main class> [options] The XML suite files to run
Options:
-configfailurepolicy
Configuration failure policy (skip or continue)
-d
Output directory
-dataproviderthreadcount
Number of threads to use when running data providers
-excludegroups
Comma-separated list of group names to exclude
-groups
Comma-separated list of group names to be run
-junit
JUnit mode
Default: false
-listener
List of .class files or list of class names implementing ITestListener
or ISuiteListener
-methods
Comma separated of test methods
Default: []
-methodselectors
List of .class files or list of class names implementing IMethodSelector
-mixed
Mixed mode - autodetect the type of current test and run it with
appropriate runner
Default: false
-objectfactory
List of .class files or list of class names implementing
ITestRunnerFactory
-parallel
Parallel mode (methods, tests or classes)
Possible Values: [tests, methods, classes, instances, none, true, false]
-port
The port
-reporter
Extended configuration for custom report listener
-suitename
Default name of test suite, if not specified in suite definition file or
source code
-suitethreadpoolsize
Size of the thread pool to use to run suites
Default: 1
-testclass
The list of test classes
-testjar
A jar file containing the tests
-testname
Default name of test, if not specified in suitedefinition file or source
code
-testnames
The list of test names to run
-testrunfactory, -testRunFactory
The factory used to create tests
-threadcount
Number of threads to use when running tests in parallel
-usedefaultlisteners
Whether to use the default listeners
Default: true
-log, -verbose
Level of verbosity
-xmlpathinjar
The full path to the xml file inside the jar file (only valid if
-testjar was specified)
Default: testng.xml
As I see it is an configuration issue. As You have not provided pom.xml details here I would like to point out few check points which would help you!
Add testng.xml to your project pointing to correct classes that needs to be executed.
Add pom.xml to project with accurate information described below.
In pom.xml file include maven-compiler-plugin.
In pom.xml file include maven-surefire-plugin.
In pom.xml under maven-surefire-plugin don't forget to add suitexmlfiles tag which will specify testng.xml file to be executed.
Thanks and regards,
Sandeep Jaju
I am working on a framework using webdriverIO and mocha. Recently I have installed the Allure reporter to generate HTML reports using jenkins
I am facing a problem with skipped tests though. I have a lot of tests that consist of a header without any code, that still need to be written.
In mocha I add "it.skip" to skip these tests.
While the tests are skipped, the Allure report only recognizes 1 skipped test per file.
When running the below code, Allure returns 1 passed test, 1 failed test and 1 skipped test
describe('Allure test', function() {
it.skip('1. this is a skipped test without any code', function () {
})
it.skip('2. this is another skipped test without any code', function () {
});
it('3. this is an enabled test that has a successfull assert', function () {
chai.expect("foo", "foo should equal foo").to.contain("foo")
});
it('4. this is an enabled test that has a failed assert', function () {
chai.expect("foo", "foo should equal foo").to.contain("bar")
});
});
I would really like my allure report to show how many tests are skipped, to be able to show how much work is left.
The default mocha logging handles this just fine, it shows this:
Number of specs: 1
1 passing (4.00s)
2 skipped
1 failing
I also use the wdio spec reporter which shows it like this, which is also fine:
1 passing (2s)
2 pending
1 failing
I have tried inplementing a categories.json file to manipulate the Allure categories, but I can't get anything to change.
I tried this as a test, but adding it to my allure results folder changed nothing:
[
{
"name": "Ignored tests",
"matchedStatuses": ["skipped", "Skipped", "pending", "Pending", "failed", "Failed", "broken", "Broken", "skip", "Skip", "failing", "Failing", "passes", "Passes"]
}
]
The tools and versions I use are:
`-- wdio-mocha-framework#0.6.2
`-- wdio-allure-reporter#0.6.3
`-- webdriverio#4.13.1
Can anyone tell me how I can get Allure to see all skipped tests?
It is a bug. I've fixed it in https://github.com/webdriverio/wdio-allure-reporter/pull/127
Thanks for reporting this. For the future if you run into such bug, please, file an issue on github.
I'm not getting any results from opencover. My nunit tests all run and pass, just no coverage results. the problem seems to be opencover filters, but we aren't setting any. Any suggestions?
The CodeCoverage.xml file contains a group of lines like the following which indicate that the plugin is telling opencover to filter out the DLLs we are trying to measure.
<Module hash="A3-F0-3A-1A-FF-38-D7-EF-A2-55-C9-8B-84-37-CF-CF-00-80-70-23" skippedDueTo="Filter">
<FullName>C:\gitlab-runner\builds\83ebc972\0\active\scrpt\output\Scrpt.Core.dll</FullName>
<ModuleName>Scrpt.Core</ModuleName>
<Classes/></Module>
which has the correct path for the dll file, but I don't see why it is skipped due to filtering. The unit tests are contained in a dll called Scrpt.Test.dll and the rest of the code is in other DLLs all of which are being filtered out.
I'm using the following plugins
plugins {
id 'com.ullink.msbuild' version '2.15'
id 'com.ullink.nunit' version '1.8'
id 'com.ullink.opencover-nunit' version '1.6'
}
and the plugin definitions for nunit and opencover are:
nunit {
testAssemblies = [file('output/Scrpt.Tests.dll')]
shadowCopy = false
useX86 = true
ignoreFailures = false
}
opencover {
targetAssemblies = [file('output/Scrpt.dll'),file('output/Scrpt.Core.dll'),file('output/Scrpt.SourceCitations.dll'),file('output/ScrptUtilLib.dll')]
ignoreFailures = false
}
Thank you for your help,
-herb
You need to set an opencover filter. Open cover filter works off of inclusive and exclusive filters.
The first filter should always be something like :
+[*]*
meaning include every assembly and every class.
then add in your exclusive filters:
+[*]* -[AssemblyName]* -[*AnotherName]*
It's very simple. just add in the universal inclusive filter first, get your results, then start to incrementally exclude stuff in your filter one by one.
This was answered thru the help of Francois Valdy over at https://github.com/Ullink/gradle-opencover-plugin/issues/17
The problem turned out to be that the opencover plugin was not generating the coverage.xml results. There was an error message in the xml saying that the files I was interested in were skipped because of filters, but I couldn't find what filters were causing that.
I ended up replacing the opencover plugin with a task that does the same thing. Not sure why the plugin didn't work, but the replacement is essentially the same thing in that it calls nunit, creates the output coverage xml which sonarqube then uploads. Plus doing it this way gives you more control over where the files end up.
A bit awkward, but it work. The following replaced the opencover block in my original question.
task opencover(type:Exec) {
executable 'C:\\Program Files (x86)\\OpenCover\\OpenCover.Console.exe'
workingDir file('output')
args "-target:c:\\Program Files (x86)\\NUnit 2.6.4\\bin\\nunit-console-x86.exe", "-targetargs:c:\\gitlab-runner\\builds\\83ebc972\\0\\active\\scrpt\\output\\scrpt.tests.dll /noshadow", '-output:coverage.xml', '-filter:+[*]*'
}
btw, I did try replacing the file() uses in the targetAssemblies for the opencover gradle plugin, but that had no effect. From what I've read, the following should've worked.
opencover {
targetAssemblies = ['output/Scrpt.dll','output/Scrpt.Core.dll','output/Scrpt.SourceCitations.dll','output/ScrptUtilLib.dll']
ignoreFailures = false
}
I have a maven java project in some folder (it has some unit tests), and tests for the same code in another different project (different directory). Both source code and test share the same parent pom. Now I want to generate the code coverage report using JaCoCo.
How to instrument sources? How to run tests on instrumented code? And how to integrate and get the result report?
Say Project ABC contains the code and project XYZ contains the test cases.
Note:
Project ABC and Project XYZ are independent projects
Both ABC and XYZ contains multiple sub projects(Need to integrate everything).
I had a similar problem. I found a solution by changing the path of the jacoco report path:
<sonar.surefire.reportsPath>${project.basedir}/../target/surefire-reports</sonar.surefire.reportsPath>
<sonar.jacoco.reportPath>${project.basedir}/../target/jacoco.exec</sonar.jacoco.reportPath>
I added these properties and configured the jacoco plugin to append reports and not overwrite them by:
<configuration>
<append>true</append>
</configuration>
This way jacoco write the reports to the base directory of the multi module project. The sonar plugin finds these and analyses them.
I had a problem when building a "reference application" with multiple sub-modules and trying to generate test coverage for the sub-modules and have them push to sonarqube. The issue I was having was that since the sub-modules referenced each other, the resulting jacoco reports were getting overwritten and I'd end up with results for a single module. This may not be the same issue as posted above, but I did solve it by using "append" in gradle, so just want to show how to do that.
In the build.gradle file for each module, I have:
testOptions {
unitTests.returnDefaultValues = true
unitTests.includeAndroidResources = true
unitTests.all {
jacoco {
append = true
includeNoLocationClasses = true
}
systemProperty 'robolectric.enabledSdks', '28'
}
}
s
I'm using:
Sonar version: 2.10
Emma version: 2.1.5320
Sonar Emma plugin version: 1.2
I'm able to generate an Emma report showing coverage of the tests themselves (ideally this would be 100% but in practice it's not always), but Sonar shows only the coverage of the src files. How do I get it to show the coverage of the test files, too?
Would switching to Cobertura help?
AFAICT from:
public final class NewCoverageFileAnalyzer {
public boolean shouldDecorate(Resource resource) {
return Scopes.isFile(resource) && !Qualifiers.UNIT_TEST_FILE.equals(resource.getQualifier());
}
}
it looks like coverage of test files can't be shown in Sonar without changing the Sonar code.
Comment:
Counting test classes as coverage might inflate coverage ratio.
Using more test files would allow coverage > 100 % (lines covered / lines of production code).
It might still be usefull to see, if there is some dead test code somewhere.