I have an Xcode 4.5 project for iOS 6.0 and I am trying to set up test coverage for Jenkins with Cobertura.
I have set up a test target and have configured both the main target and the test target correctly (Instrument Program Flow and Generate Test Coverage Files both on YES).
When the coverage report is generated (.gcda file), I translate it into XML using the gcovr tool (version 2.4).
When I run the test on Jenkins, everything works fine and I can see the report through the Cobertura plugin. The only problem is that in the Cobertura report all the values for "Conditionals" are "N/A", while I see that there are conditionals (if-constructions) in the covered files.
My guess is that something happens in the translation to XML resulting in a faulty conversion of branches/conditionals, but I cannot find what it is.
The strange thing is that for some if constructions it does recognise the conditional. For example if I use the "STAssertEquals()" macro from the "SenTestingKit" library, the if constructions there are recognised.
I hope someone can help me with a fix for this problem or at least knows the cause. Please do not hesitate to ask me if you need more information.
Nicolas
Related
Since we updated to SonarQube 6.2 it seems code coverage plugin got merged in the core.
It shows red flags everywhere and I can’t find how to turn it off, we do not use code coverage.
You don't specify what language(s) you're analyzing. I'll assume Java and/or JavaScript. Starting from 6.2, SonarQube supports "force coverage to 0", which marks as uncovered executable lines in files that don't show up in any coverage reports. (That's assuming the underlying code analyzers support the feature, and Java and JavaScript already do.) The purpose is to have a more accurate picture of what's missing when you actually are using unit tests. Without this feature, it's impossible to tell whether a file that's omitted from coverage reports is missing because it has no executable code or because there are no tests on it - even when there should be.
Since you're not using unit tests (really?) you can exclude all the source files in your project from coverage calculations via the UI: Administration > Analysis Scope > Coverage Exclusions. A pattern value of **/*.* ought to do it for you.
Code coverage feature is in SonarQube misleading. To turn this off:
Under Quality gates create new or copy existing profile.
There delete Coverage metric
SonarQube Version: 9.2
Here's a screenshot so you better understand what I'm talking about:
Shouldn't all directories have coverage?
I have to say that those directories (without any number) are not covered by any tests but doesn't this mean that the coverage is 0%?
Yes, they should. But they don't because when no coverage engine reports any coverage on a file, is that because the file is executable but there are no tests on it, or because the file is not executable?
SonarQube v6.2 will begin to address this by automatically forcing to 0 the coverage metrics on files not covered in the unit test reports. However, this behavior will only be fully enabled when each of the language plugins reports the "executable lines" for each file.
So in practice the new behavior enabled in 6.2 (not released at this writing, but "soon") won't be truly available until the language plugins start supporting it. Probably over the next few months.
I'm testing SonarQube with a .Net solution file. I'm trying to figure out which files exactly get uploaded/imported into the SonarQube database. Based on my tests with the SonarQube MSBuild runner, it analysis and uploads all files which are stated in the .csproj project file(s) (only those that can be associated with a language plugin because I have set "Import unknown files" to false).
Is there a way to specify additional files which should be analyzed and imported? I had some hope for the "sonar.inclusions" setting, but it's not exactly what I need because the description says Patterns used to include some source files and only these ones in analysis.
I'd like to just specify additional files.
My use-case: at the moment, SonarQube does not create issues for compiler warnings (see Displaying C# compilation warnings in Sonar?) (which in my opinion is really sad/weird for a tool that wants to improve code quality). So my idea is to use msbuild with an xml-logger, import the generated xml log file and use the XML plugin with custom XML path rules to extract the compiler messages.
Similar issue for analyzing Delphi code. I found an old SonarQube Delphi plugin which I could not yet get to work with SonarQube 5.x, but I'd like to try to get at least the compiler messages into SonarQube (convert Delphi compiler output into XML, get SonarQube to import this file, use custom XML path rules).
Is there a way to get SonarQube to analyze/import such additional files?
I would like TeamCity to run my mSpec tests and report on the code covered by the tests.
I would also like TeamCity to report that a build has failed if code coverage in certain namespaces doesn't meet a threshold (e.g. MyProduct.ImportantStuff must be 100%, but MyProduct.LegacyStuff must be [23% or whatever it currently is to ensure we don't add new stuff without covering tests].
I initially looked at dotCover as it's integrated into TeamCity. I have since been looking at OpenCover as I couldn't get TC to fail the build on low coverage.
I got OpenCover working but I would still like to know (as I'm sure a lot of people would) how to get TC to fail a build if code coverage is too low.
Are you using the latest TeamCity, ie version 7?
When setting up a build configuration you can specify this:
There are lots more options in the dropdown related to code coverage. You can also force your build to fail if you're using some other code coverage tool.
For example you can echo a line to the console that will then be picked up by teamcity :
##teamcity[buildStatus status='FAILURE' text='something failed']
see official docs on this here
Okay, Emma is killing me. I spent two days on this already.
There two issues with Emma now
It crashes partly through unit testing
It complaints about class stamp mismatch. : com.vladium.emma.EMMARuntimeException: [CLASS_STAMP_MISMATCH] runtime version of class xxx in the coverage data is not consistent with the version of this class in the metadata, possibly because stale metadata is being used for report generation.
While I can live with the crashes, it only often but I don't intent to fix it,
INSTRUMENTATION_RESULT: shortMsg=Process crashed.
[exec] INSTRUMENTATION_CODE: 0
However, I do like to see my coverage result.
I have tried:
clean the device
wiped out entire workspace (in Jenkins)
using the command prompt (bypass jenkins, ant clean, ant emma debug install test)
My understanding of this error is the apk installed and under unit test contains different binaries and hence its metadata info is out of date with the coverage.em generated.
I can confirm this is IMPOSSIBLE in my case, as I clead, ant clean and even wiped out entire work space and the phone memory. It just doesn't make any sense to me now.
Please HELP~
I am using Android sdk r16, NDK 5c, and the default settings from Ant and Emma.
I'm seeing the exact same problem with my Android project (which uses a native code third-party library via JNI). It turns out that the EMMA error message is slightly misleading in this case. It's triggered because the statically generated coverage meta-data on the host (coverage.em) does not match the coverage data collected during run-time of the application on the device (coverage.ec). In fact, it's not only a "mismatch", but the coverage.ec is completely missing. This is due to the crash in native code: On Android, a crash in native code will shut down the associated Java process, too, and in fact even restart the VM. This means EMMA is killed, too, and it does not have chance to flush the coverage.ec file to disk any more, which is why it's missing.
I've wondered whether it would make sense to make EMMA flush the coverage.ec file more often before the crash, but then again it's questionable how valuable an incomplete coverage info would be as
the file format probably is just not build in a way that makes sense to be parsed incrementally,
a partial coverage.ec still would mismatch a complete coverage.em,
coverage generated from a partial coverage.ec would report the wrong coverage with respect to all your tests in the suite.
So, long story short, the best way to address this is to fix your native code to not crash.
I met with this problem too when using EMMA for android. It's because the coverage.ec you pull from sdcard is not consistent with coverage.em. Try rebuild the instrumented version of the project.
Try the following steps:
ant clean release
ant instrument
ant installi
adb pull /mnt/sdcard/coverage.ec
use the coverage.em ( newly generated under the bin folder of the project through step 2 ) and coverage.ec to generate the coverage report
You can refer to EMMA for Android for detailed information.