I have a GO project which I build in Travis CI.
I have implemented a few tests with Ginkgo, and I am getting code coverage when I run it locally, however I get no coverage when I run it on Travis.
My .travis.yml
language: go
# safelist
branches:
only:
- master
- travis
before_install:
- go get github.com/onsi/gomega
- go get github.com/onsi/ginkgo/ginkgo
- go get github.com/modocache/gover
script:
- ginkgo -r --randomizeAllSpecs --randomizeSuites --failOnPending --cover --trace --race --compilers=2
after_success:
- gover . coverage.txt
- ls -al
- cat coverage.txt
- bash <(curl -s https://codecov.io/bash)
When I run the script command on my own machine I get the following result
$ ginkgo -r --randomizeAllSpecs --randomizeSuites --failOnPending --cover --trace --race --compilers=2
Running Suite: Gitserver Suite
==============================
Random Seed: 1470431018 - Will randomize all specs
Will run 4 of 4 specs
++++
Ran 4 of 4 Specs in 0.000 seconds
SUCCESS! -- 4 Passed | 0 Failed | 0 Pending | 0 Skipped PASS
coverage: 25.9% of statements
Ginkgo ran 1 suite in 4.411023s
Test Suite Passed
But on travis CI the coverage says "0.0% of statement"
I have tried to setup a new GOPATH on my local machine to get a clean setup and only run the commands that occurs in the Travis log, and I still get a reported 25% coverage. My machine is running windows where as Travis is linux, that is the only difference I can think of right now.
I have just tried GoCover.io on my package, and that also gives me the 25% coverage that I get locally.
I finally got it to work after I ran the travis build locally through their docker image. For some reason I need to specify which package to cover, so the ginkgo command has been changed into
ginkgo -r --randomizeAllSpecs --randomizeSuites --failOnPending --coverpkg gitserver --trace --race --compilers=2
The following command seems to work for me.
ginkgo -r --randomizeAllSpecs --randomizeSuites --cover --race --trace
If you like to know more, look at the documentation here.
Related
I created the automated testing plugin for Godot named WAT. It has a command line interface that outputs 0 (success) or 1 (failure) on the last line when run.
I'm looking for a way to pass that number onto CircleCI so that the step fails if it was 1.
I'm working in a bash environment with the following config.yml
version: 2
jobs:
build:
docker:
- image: barichello/godot-ci:3.1.1
steps:
- checkout
- run:
name: Run Tests
command: godot -s addons/WAT/CLI.gd -run_all
You can set OS.exit_code to a non-zero number and the step should fail.
If that is not acceptable, you can parse your output and fail manually from Bash with exit 1.
I've started to run my ui tests in parallel, in order to improve performance. However, when using xcpretty, I don't get now which tests have passed nor what made the tests fail (I only get which tests have failed). Is there any way to solve this? Or an alternative to xcpretty that works with the outputs in the parallel tests. I want to have a nice terminal output, as when using sequential testing.
This is my script:
xcodebuild \
-workspace './code/ios/myApp/myApp.workspace' \
-scheme 'myApp' \
-destination 'platform=iOS Simulator,name=iPhone 6' \
test | xcpretty -c
This is the output I got when running tests sequentially (And the output I want to continue having when running them in parallel):
Selected tests
[15:35:45]: ▸ Test Suite UITests.xctest started
[15:35:45]: ▸ RegisterTest
[15:36:48]: ▸ ✗ testRegisterBrazil, failed - Couldn't find:
"homeBottomBar_myAccountButton" Button
[15:42:50]: ▸ ✓ testRegisterUSA (61.241 seconds)
[15:42:50]: ▸ Executed 4 tests, with 1 failures (1 unexpected) in 425.314 (425.319) seconds
This is the output I get now:
Failing tests:
UITests:
RegisterTest.testRegisterBrazil()
** TEST FAILED **
we have a CI pipeline on our repository, hosted in gitlab
we setup gitlab-runner on our local machine
the pipeline running 4 steps
build
unit tests
integration test
quality tests
all this pipeline takes almost 20 min
and the pipeline trigger on each push to a branch
is there a way to configure the gitlab-runner that if the HEAD of a branch that the runner currently running on changes the pipe
will auto cancel the run? because the latest version is what matters
for example in this run the lower run is unnecessary
gitlab-ci.yml
stages:
- build
- unit_tests
- unit_and_integration_tests
- quality_tests
build:
stage: build
before_script:
- cd projects/ideology-synapse
script:
- mvn compile
unit_and_integration_tests:
variables:
GIT_STRATEGY: clone
stage: unit_and_integration_tests
only:
- /^milestone-.*$/
script:
- export RUN_ENVIORMENT=GITLAB_CI
- export MAVEN_OPTS="-Xmx32g"
- mvn test
- "cat */target/site/jacoco/index.html"
cache: {}
artifacts:
reports:
junit:
- "*/*/*/target/surefire-reports/TEST-*.xml"
unit_tests:
variables:
GIT_STRATEGY: clone
stage: unit_tests
except:
- /^milestone-.*$/
script:
- export MAVEN_OPTS="-Xmx32g"
- mvn test
- "cat */target/site/jacoco/index.html"
cache: {}
artifacts:
reports:
junit:
- "*/*/*/target/surefire-reports/TEST-*.xml"
quality_tests:
variables:
GIT_STRATEGY: clone
stage: quality_tests
only:
- /^milestone-.*$/
script:
- export RUN_ENVIORMENT_EVAL=GITLAB_CI
- export MAVEN_OPTS="-Xmx32g"
- mvn test
cache: {}
edit after #siloko comment:
I already try using
the auto-cancel redundant, pending pipelines in the setting menu
I want to cancel running pipelines and not pending
after forther investigation, I found that I had 2 active runners
on one of my machines
one shared runner , and another specific runner then if I push a 2 commit one after another to the same branch both of the runners take the jobs and execute them.
that also explains why
Auto-cancel redundant, pending pipelines
options, didn't work because it works only when the same runner have pending jobs
actions that been taken to fix this problem: unregister the specific runner and leave the machine only with the shared runner
I have 2 Gitlab repos:
Project A
Integration tests for Project A
I want to stop the pipeline / build of Project A if the integration tests fail but currently the Project A pipeline passes even if the integration tests fail.
My .gitlab-ci.yml for Project A defines these 7 stages:
stages:
- build
- test
- publish
- dev-deployment
- staging-deployment
- trigger-integration-tests
- prod-deployment
The second last stage (trigger-integration-tests) kicks off the integration tests project by using the Gitlab API call with curl:
trigger-integration-tests:
stage: trigger-integration-tests
image: ubuntu:16.04
script:
- apt-get update && apt-get install -y curl
- "curl -X POST -F token=$INTEGRATION_TESTS_TOKEN -F variables[PROJECT_ID]=$CI_PROJECT_ID -F variables[BRANCH_NAME]=$CI_COMMIT_REF_NAME -F ref=master https://gitlab.mycompany.com/api/v4/projects/123/trigger/pipeline"
allow_failure: false
only:
- master
I tried adding the allow_failure: false flag but that didn't help so I'm looking for more ideas.
I found the trigger-and-wait technique but wasn't sure if there's a more simple solution.
As answered on a previous question, you could do the following:
From the main project, using a Python/Bash script:
Trigger the integration tests pipeline (and capture the pipeline ID)
Poll the status of the pipeline, using the captured ID (which can be running, pending, failed, canceled or skipped)
Raise an exception / error if it has failed...
See here for an example python script to achieve this.
I have a Travis CI setup on GitHub. I use it to check my commits for iOS app. The problem is, I very often and randomly get an error 65. I have yet to find a solution.
When I restart the job 2-3 times after it has failed it passes in 90% of times.
I previously also had a problem with logs being too verbose for Travis (>4MB) but I added xcpretty to fix that.
Errors I took from log:
...
Generating 'XYZ.app.dSYM'
❌ error: couldn't remove '/Users/travis/Library/Developer/Xcode/DerivedData/XYZ-aaltcjvmshpmlufpmzdsgbernspl/Build/Products/Debug-iphonesimulator/XYZ.app/SomeName.storyboardc' after command failed: Directory not empty
...
And then at the end of Travis log:
Testing failed:
The file “056-Jj-FAu-view-XmS-Ro-0cO.nib” couldn’t be opened because there is no such file.
error: couldn't remove '/Users/travis/Library/Developer/Xcode/DerivedData/XYZ-aaltcjvmshpmlufpmzdsgbernspl/Build/Products/Debug-iphonesimulator/XYZ.app/SomeName.storyboardc' after command failed: Directory not empty
error: lipo: can't move temporary file: /Users/travis/Library/Developer/Xcode/DerivedData/XYZ-aaltcjvmshpmlufpmzdsgbernspl/Build/Products/Debug-iphonesimulator/XYZ.app.dSYM/Contents/Resources/DWARF/XYZ to file: /Users/travis/Library/Developer/Xcode/DerivedData/XYZ-aaltcjvmshpmlufpmzdsgbernspl/Build/Products/Debug-iphonesimulator/XYZ.app.dSYM/Contents/Resources/DWARF/XYZ.lipo (No such file or directory)
Command /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/dsymutil emitted errors but did not return a nonzero exit code to indicate failure
** TEST FAILED **
The following build commands failed:
LinkStoryboards
LinkStoryboards
(2 failures)
The command "./scripts/build.sh" exited with 65.
I am using Xcode 8 both in Xcode and Travis settings.
Ah, good question. Occasionally, xcodebuild steps that are failing during the codesigning step can be addressed using travis_retry - Travis will retry the step 3 times for any non-zero exit status, which should reduce the need for you to restart it manually. There are some suggested code snippets in the travis-ci/travis-ci GitHub issue on this as well. Good luck!
If you're running into the error code 65 (from random failures) here's a command you can pipe on the end of your xcodebuild (assuming you're running tests) command to get back more consistent results:
(XCODEBUILD_COMMAND_HERE) | awk 'BEGIN {success=0} $0 ~ /.* tests, with 0 failures \(.*/ {success=1} {print $0} END {if(success==0){exit 1}}
This looks for tests, with 0 failures ( in your output text, thus using the text output of xcodebuild instead of the status code of xcodebuild to determine success.
Note: Keep in mind, if you do something like NSLog(' tests, with 0 failures ('); in your code you make get a false positive, it's very unlikely to happen by accident. You may have to update tests, with 0 failures ( in the awk script between updates of xcodebuild. But, having consistent results with xcodebuild is definitely worth that price.