We have an api in .net5, we were using coverlet.msbuild to validate our code coverage but our solution has 50 csprojs and we were hitting a bug where some projects returned 0% of coverage, so we changed the package to coverlet.collector, our problem now is that coverlet.collector does not support threshold validation.
At the end of we have an xml file with cobertura format and we just want to run a command to see if the code coverage reported is OK and let the PR continue its process.
I've also faced that problem. The solution is to use coverlet directly. You can install it as a global tool for dotnet. Note that, I wasn't able to properly gather coverage without flag --no-build.
My working command is:
"C:\Program Files\dotnet\dotnet.exe" C:\Users\dalec\.nuget\packages\coverlet.console\3.1.2\tools\net5.0\any\coverlet.console.dll \
C:\Users\dalec\source\repos\ZTR.AI\ZTR.AI.Common\ZTR.AI.Common.Core.Tests\bin\Debug\net5.0\ZTR.AI.Common.Core.Tests.dll \
--target "C:\Program Files\dotnet\dotnet.exe" \
--targetargs "test C:\Users\dalec\source\repos\ZTR.AI\ZTR.AI.Common\ZTR.AI.Common.Core.Tests\ZTR.AI.Common.Core.Tests.csproj \
--configuration Debug --no-build" --output C:\Users\dalec\source\repos\ZTR.AI\testrestults\ZTR.AI.Common.Core.Tests/ \
--format cobertura --format json --threshold 69
Unfortunately, I still cannot gather coverage on Ubuntu.
Related
In the steps of setting up CI builds on our gitlab server, I can't seem to find information on how to set up the detection of compiler warnings. Example build output:
[100%] Building CXX object somefile.cpp.o
/home/gitlab-runner/builds/XXXXXXX/0/group/project/src/somefile.cpp:14:2: warning: #warning ("This is a warning to test gitlab") [-Wcpp]
#warning("This is a warning to test gitlab")
^
However the build result is success instead of warning or something similar. Ideally the results wuold also be visible on the merge request on the feature (and block the merge if possible).
I can't imagine I'm the only one trying to achieve this, so I am probably looking in the wrong direction. The 'best' solution I found is to somehow manually parse the build output and generate a JUnit report.
How would I go about doing this without allowing the build job to fail, since I would like the job to fail when compiler errors occur.
Update
For anyone stumbling across this question later, and in lieu of a best practice, this is how I solved it:
stages:
- build
- check-warnings
shellinspector:
stage: build
script:
- cmake -Bcmake-build -S.
- make -C cmake-build > >(tee make.output) 2> >(tee make.error)
artifacts:
paths:
- make.output
- make.error
expire_in: 1 week
analyse build:
stage: check-warnings
script:
- "if [[ $(cat make.error | grep warning -i) ]]; then cat make.error; exit 1; fi"
allow_failure: true
This stores the build output errors in make.error in the first stage, the next stage then queries that file for warnings and fails that stage with allow_failure: true to create the passed with warning pipeline status I was looking for.
It seems that the solution to such need (e.g., see the issue "Add new CI state: has-warnings" https://gitlab.com/gitlab-org/gitlab-runner/issues/1224) has been to introduce the allow_failure option so that one job can be the compilation itself, which is not allowed to fail (if it does, then the pipeline fails) and another job can be the detection of such warnings which is allowed to fail (if one is found then the pipeline will not fail).
Also the possibility of defining warning regex in the .gitlab-ci.yml has been requested but it does not exist yet.
I'm running the following command to run my .net Core tests:
dotnet test
This runs fine. I want to now generate code coverage stats, so after following this article, I ran this:
dotnet test AI.Core.Tests.csproj
/p:CollectCoverage=true
/p:CoverletOutputFormat=cobertura
/p:CoverletOutput=TestResults\Coverage
I get the following output from this command:
C:\Users\sp4_rm\.nuget\packages\coverlet.msbuild\2.2.1\build\netstandard2.0\coverlet.msbuild.targets(23,5): error :
Index was out of range. Must be non-negative and less than the size of the collection.
[C:\Users\sp4_rm\Desktop\EVO\AI.Core\src\Tests\AI.Core.Tests.csproj]
C:\Users\sp4_rm\.nuget\packages\coverlet.msbuild\2.2.1\build\netstandard2.0\coverlet.msbuild.targets(23,5): error :
Parameter name: index
[C:\Users\sp4_rm\Desktop\EVO\AI.Core\src\Tests\AI.Core.Tests.csproj]
See screen shot below:
Has anyone got this command running? What am I doing wrong?
Ok so this was due to a school boy error in not actually have any code to test (or test case to test it) in my sample project!! Adding a couple of classes into the main projects and a couple of tests in the test project does away with this problem! (just in case anyway does the same thing as me!)
This seems to be a classic dilemma, but I haven't figured it out. Maven works fine from outside intellij, but intellij uses its own maven instance which seems to work in general.
However, when trying to debug it doesn't get as far as finding the launcher, which definitely sounds like a classpath issue. Looking at the output of 'ps' (edited for readability), this is what intellij is running:
/Library/Java/JavaVirtualMachines/jdk1.8.0_25.jdk/Contents/Home/bin/java \
-agentlib:jdwp=transport=dt_socket,address=127.0.0.1:63700,suspend=y,server=n \
-Dmaven.home='/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3' \
-Dclassworlds.conf='/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3/bin/m2.conf' \
-Dfile.encoding=UTF-8 \
-classpath \
'/Applications/IntelliJ IDEA 15 CE.app/Contents/plugins/maven/lib/maven3/boot/plexus-classworlds-2.4.jar:\
/Applications/IntelliJ IDEA 15 CE.app/Contents/lib/idea_rt.jar' \
org.codehaus.classworlds.Launcher \
-Didea.version=15.0.2 \
scala:run \
-DmainClass=redangus.nce.Inserter
For some reason that is lost on me, when I remove the 2nd line, (the debug info), I can run it fine from the command line and it has no issue with finding the launcher.
I would be very pleased to have someone help get my mind right.
Hmm, I have a hunch that server option should be 'y'. I wonder how I change that ...
Well, I built a Hello World maven app using the standard net.alchim31.maven:scala-archetype-simple archetype and after fixing the imports so it compiles I find that it doesn't actually work in the debugger, either. Sigh. I give up. Yay, print statements
Look this question, just try
Looks like you have an exception breakpoint on ClassNotFoundException. Please try to open Run | View Breakpoints... and uncheck the breakpoint under "Java Exception Breakpoints".
I check the breakpoint by misktake, and cause the problem.
I'm trying to add Coverity Scan static analysis to my Qt Mac project but I'm not able to submit the build using travis.
Here is my coverity specific travis setup:
addons:
coverity_scan:
project:
name: Phonations/TravisTest
description: Build submitted via Travis CI
notification_email: martin#phonations.com
build_command_prepend: cov-configure --comptype clangcxx --compiler clang++ --template
build_command_prepend: qmake QtTest.pro
build_command: make -j 4
branch_pattern: coverity
And here is the result I got by mail:
Your request for analysis of Phonations/TravisTest is failed.
Analysis status: Failure
Please fix the error and upload the build again.
Error details:
Build uploaded has not been compiled fully. Please fix any compilation error. You may have to run bin/cov-configure as described in the article on Coverity Community. Last few lines of cov-int/build-log.txt should indicate 85% or more compilation units ready for analysis
For more detail explanation on the error, please check: https://communities.coverity.com/message/4820
If your build process isn't going smoothly, email us at scan-admin#coverity.com
with your cov-int/build-log.txt file attached for assistance, or post your issue
to the Coverity Community at https://communities.coverity.com/community/scan-(open-source)/content
There is not really MacOS specific explanation in the documentation. Anyone has an idea how to submit it?
I found a command-line tool called "xctest" that apparently can run the unit tests in your project. This executable lives here:
/Applications/Xcode.app/Contents/Developer/usr/bin/xctest
When I try to run this executable on my xctest bundle, I'm using:
$ ./xctest /Users/myusername/Library/Developer/Xcode/DerivedData/MyApp-abcdefghijklmnop/Build/Products/Debug/MyAppTests.xctest
However, I get the following output:
Test Suite '(null)' started at 2013-11-14 21:16:45 +0000
Test Suite '(null)' finished at 2013-11-14 21:16:45 +0000.
Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.001) seconds
There's no man page for xctest, as far as I can tell, but entering just ./xctest at the command-line yields:
Usage: xctest [--test Self | All | None | <TestCaseClassName/testMethodName>] <path of unit to be tested>
In particular, I'd like to be able to test just a particular method in a test class, which is why I'd like to use this xctest command.
I do see that there is a way to run all the tests from the command line like:
$ xcodebuild test -scheme MyApp
This runs all the unit tests and works properly (I see my unit test results, unlike when using xctest). But I'm interested in being able to run a single test method from the command-line, such as:
$ ./xctest --test MyAppTests/testExample /Users/myusername/Library/Developer/Xcode/DerivedData/MyApp-abcdefghijklmnop/Build/Products/Debug/MyAppTests.xctest
Despite what the usage message says -XCTest is the argument you need:
xctest -XCTest MyAppTests/testExample testbundle.xctest
For a direct invocation of xctest to work you may also need to set DYLD_FRAMEWORK_PATH and DYLD_LIBRARY_PATH to your built products directory. In general you need to use the same arguments and environment as Xcode does, you can see this by putting a breakpoint in one of your tests, running them through Xcode, then printing out the values of arguments and environment for [NSProcessInfo processInfo].
To avoid messing with all that note you can also modify the scheme in Xcode to run only specific tests. Under Product > Scheme > Edit Scheme select the Test action and expand the test bundle. You can use the check boxes to select the tests to run and xcodebuild's test action will then run only these tests.