I am trying to figure out how to run unit tests, using Google Test, and send the results to TeamCity.
I have run my tests, and output the results to an xml, using a command-line argument --gtest_output="xml:test_results.xml".
I am trying to get this xml to be read in TeamCity. I don't see how I can get XML Reports passed to TeamCity during build/run...
Except through XML report Processing:
I added XML Report Processing, added Google Test, then...
it asks me to specify monitoring rules, and I added the path to the xml file... I don't understand what monitoring rules are, or how to create them...
[Still, I can see nowhere in the generated xml, the fact that it intends to talk to TeamCity...]
In the log, I have:
Google Test report watcher
[13:06:03][Google Test report watcher] No reports found for paths:
[13:06:03][Google Test report watcher] C:\path\test_results.xml
[13:06:03]Publishing internal artifacts
And, of course, no report results.
Can anyone please direct me to a proper way to import the xml test results file into TeamCity ? Thank you so much !
Edit: is it possible that XML Report Processing only processes reports that were created during build ? (which Google Test doesn't do?) And is ignoring the previously generated reports, as "out of date", while simply saying that it can't find them - or are in the wrong format, or... however I should read the message above ?
I found a bug report that shows that xml reports that are not generated during the build are ignored, making a newbie like me believe that they may not be generated correctly.
Two simple solutions:
1) Create a post build script
2) Add a build step that calls the command line executable with the command-line argument. Example:
Add build step
Add build feature - XML report processing
I had similar problems getting it to work. This is how I got it working.
When you call your google test executable from the command line, prepend %teamcity.build.checkoutDir% to the name of your xml file to set the path to it like this:
--gtest_output=xml:%teamcity.build.checkoutDir%test_results.xml
Then when configuring your additional build features on the build steps page, add this line to your monitoring rules:
%teamcity.build.checkoutDir%test_results.xml
Now the paths match and are in the build directory.
Related
We run our protractor regression tests in gitlab CI and we have jasmine HTML reports. Right now it is only the QA team that monitors and checks failure if any.
But we would like to make it more visible. The devs have also asked us if we can make it visible in a single place instead of having to go to gitlab job and browse for artifacts.Also would it be possible to have an overview of pass/fail tests over time.
I'm not sure how and where to start. Any pointers would be appreciated.
You're looking for the expose_as keyword for artifacts. The full docs are here: https://docs.gitlab.com/ee/ci/yaml/#artifactsexpose_as.
If you use expose_as with your artifacts, Gitlab CI will link them to any application Merge Request with the name you give in this field.
For example (from the docs):
test:
script: ["echo 'test' > file.txt"]
artifacts:
expose_as: 'artifact 1'
paths: ['file.txt']
In this example, a Merge Request for this pipeline will have a link called "artifact 1" that opens the file "file.txt".
This also works for directories, but if there's more than one file it will open in the job's artifacts browser (like you currently do).
There are some caveats, like:
If you use a variable in the artifacts path field, expose_as won't work
Max of 10 artifacts can be exposed
Glob patters won't work
If Gitlab Pages is enabled, some file extensions will be automatically rendered using Pages (html, xml, txt, etc.).
I am trying to find a portable way to produce code annotations for GitHub in a way that would avoid a vendor-lockin.
Mainly I want to dump annotations inside a file (yaml, json,...) during build process and have a task at the end that does transform this file into github annotations.
The main goal here is to avoid hardcoding support for github-annotation into the tools that produce them, so other CI/CD systems could also consume the annotation-reports and display them in their UI.
linters -> annotations.report -> github-upload
Tools like flake8 are able to produce output in parsable format file:line:column: message, but I need to know if there is any attempt to standardize annotations so we can collect and combine them from multiple tools and feed them to the CI/CD engine.
Today I googled up what the heck those "Github Action Annotations" are all, and this was among the hits:
https://github.com/marketplace/actions/annotations-action
GitHub action for creating annotations from JSON file
As of now that page also contains:
This repository uses npm packages from #attest scope on github; we are working hard to open source these packages.
Annotations Action is not certified by GitHub. It is provided by a third-party and is governed by separate terms of service, privacy policy, and support documentation.
I didn't try it, again, just a random google hit.
I am currently using https://github.com/yuzutech/annotations-action
Sample action code:
- name: Annotate
uses: yuzutech/annotations-action#v0.3.0
with:
repo-token: ${{secrets.GITHUB_TOKEN}}
input: ./annotations.json
title: 'Findings'
ignore-missing-file: true
It does its job well but with one minor defect. If you have a findings on a commit/PR you get to see the finding with a beautiful annotation right where you need it. If you re-push changes, even if the finding persists, the annotation is not displayed on later commits. I have opened an issue but I have not yet received an answer.
The annotations-action mentioned above has not been updated and it does not work with me at all (deprecated calls).
I haven't found anything else that worked exactly as I wanted it to.
Update: I found that you can use reviewdog to annotate based on findings. I also created a GitHub action that can be used for Static Code Analysis here https://github.com/tsigouris007/action-semgrep-reviewdog. You can visit the entrypoint.sh file and check how I piped the custom output to reviewdog utilizing jq.
I've got a Java/Maven project that uses RestDocs to generate documentation of our endpoints.
For the uninitiated of RestDocs, what it does is monitor tests during the build and use information from the tests to generate "snippets" that get used to create API documenation using asciidoc during the mvn package phase.
The problem I'm facing is that if the asciidoc (.adoc) file is referencing a non-existent snippet, the doc gets generated and will say something like:
"Unresolved directive in myDoc.adoc - include::{snippets}/error-example/response-fields.adoc[]"
Since this happens after the tests, I don't know how to test for something like this so that the build can stop and let the developer know that they need to update either the .adoc or the snippet.
Any guidance would be much appreciated.
Adding this to the configuration of the asciidoctor plugin fails the build when a snippet is not found that the .adoc is expecting:
<logHandler>
<outputToConsole>true</outputToConsole>
<failIf>
<containsText>include file not found</containsText>
</failIf>
</logHandler>
Went through the serenity documented for extracting the test outcomes
below isthe code, it didn't work
OutcomeFormat format = OutcomeFormat.XML; TestOutcomes outcomes =
TestOutcomeLoader.loadTestOutcomes().inFormat(format)
Tried with below code and its working,
OutcomeFormat format = OutcomeFormat.JSON; TestOutcomeLoaderBuilder
outcomes= TestOutcomeLoader.loadTestOutcomes().inFormat(format);
TestOutcomes out =outcomes.from(new File(""));
Issue is i need the test outcomes in #AfterScenario, but the thing is serenity reports gets generated after the entire execution tried changing the pom but didn't help. Is there any other way using which we can extract the test results?
Serenity uses JSON format by default now. Why are you trying to obtain the test outcomes? (i.e. what problem are you trying to solve?)
Created a separate java class for report extracting and added that in maven plugin which will get executed after serenity report is generated.
As #John smart has mentioned JSON and HTML are default output formats.
Still if you want to access Outcomes after test execution.
You can create a custom listener and listen to serenity event bus.
TestRunFinished event will be published with Outcome as a parameter.
You can use the outcome for getting required details.
For creating custom listener you can follow this page
I am using guard-jasmine create coverage reports for my javascript app written with backbone js. I would like to exclude the template files from being included in the coverage. Is there a way to do this currently? I have also tried looking through the source and passing the -x option to the intrument command in the coverage.rb file but that doesn't seem to help at all. Any pointers would be appreciated.
Thanks!
There is currently no way of configuring Guard::Jasmine to skip specific files from generating the coverage.
A possible way to add this would be to add something like a coverage_skip option that contains a regex to check it as a preconodition in the coverage tilt template:
return data if file =~ ENV['COVERAGE_SKIP']
Since we do not have access to the Guard::Jasmine options, we need to set it as an environment variable in the server process.
A pull request is heartly welcome ;)