I have a problem that when I run 3 tests that every test visit other domain I got this error in the third test:
it always happen in the third change.
for example:
test 1 domain: https://example.cypress.io
test 2 domain: https://www.walla.co.il/
test 3 domain: https://example.cypress.io - here it will fail
Related
Im new in Ansible. Im using playbooks/roles in Openstack and im searching for a Global variable or similar that count the number of errors in the playbook or at least that an error ocurred in some moment.
This is why (let say that this variable is called GLOB):
In Task:
---
# tasks file
- name: testing block
block:
# List of tests to run in this test suite:
- include: ../tests/DoThings.yml #API calls, http 50x possible errors: YES
always: #Final task always executed
- include: ../tests/Clear.yml #API calls, http 50x possible errors: YES
- include: ../tests/Report.yml #NOT API calls, http 50x possible errors: NO, just log checker.
So if some error ocurred in DoThings.yml i want to clear all the thing with Clear.yml and AFTER execute Report.yml, inside this i will check if GLOBAL var has 1 failed (one or more failed). This is because a failed can be
A "50x HTTP", that is impossible to predict, you have to try again (this is important to detect for me and differentiate from other errors)
A normal error for code mistake or similar, easy to fix (not important in this case)
Any help sooner would be greatly appreciated
I am using PIPE to connect to Jenkins pipeline from BB and using the below code in my BB.yml
- step: &functionalTest
name: functional test
image: python:3.9
script:
- pipe: atlassian/jenkins-job-trigger:0.1.1
variables:
JENKINS_URL: '<<myJenkinsURL>>'
JENKINS_USER: '<<myJenkinsUser>>'
JENKINS_TOKEN: $JENKINS_USER_TOKEN
JOB_NAME: '<<myJenkinsJob>>'
WAIT: 'true'
WAIT_MAX_TIMEOUT: 500
It was working fine until last week. However, since Friday I can see number failures in BB pipeline with the timeout. Though the Jenkins job is successful and took only 4 mins and 56 secs to execute all test cases. Also I have WAIT_MAX_TIMEOUT: 500 (almost 8 mins max timeout)
Exception:
✖ Timeout while waiting for jenkins job with build number 254 to be completed
PS: Jenkins job for this build ID is successful in 5 mins (including the Sonar report generation)
I created the automated testing plugin for Godot named WAT. It has a command line interface that outputs 0 (success) or 1 (failure) on the last line when run.
I'm looking for a way to pass that number onto CircleCI so that the step fails if it was 1.
I'm working in a bash environment with the following config.yml
version: 2
jobs:
build:
docker:
- image: barichello/godot-ci:3.1.1
steps:
- checkout
- run:
name: Run Tests
command: godot -s addons/WAT/CLI.gd -run_all
You can set OS.exit_code to a non-zero number and the step should fail.
If that is not acceptable, you can parse your output and fail manually from Bash with exit 1.
The documentation for --num-flaky-test-attempts parameter of gcloud firebase test android run says the following:
Specifies the number of times a test execution should be reattempted if one or more of its test cases fail for any reason.
This means it reruns only the failed tests but not the whole suite, right? In other words, as soon as a test passes it won't be retried, right?
The command line parameter --num-flaky-test-attempts of gcloud firebase test android run appears to rerun all of the tests instead of just the failed tests.
I ran a suite of tests using --num-flaky-test-attempts 10 and here timestamps from the logs for one test in the suite:
04-27 03:41:51.225 passed
04-27 03:41:50.519 passed
04-27 03:41:43.533 failed
04-27 03:41:48.625 failed
04-27 03:42:13.886 failed
04-27 03:41:33.749 failed
04-27 03:41:43.694 failed
04-27 03:41:42.101 failed
04-27 03:41:20.310 passed
04-27 03:40:17.819 passed
04-27 03:33:14.154 failed
It appears to have executed the entire test suite each time. In some cases the test mentioned above passed and in some cases it failed. It passed and failed multiple times, so clearly it's rerunning the test no matter whether it passed or failed.
I believe there were 11 total tests because I specified --num-flaky-test-attempts 10 which means it attempted to run the suite once and since that failed it ran 10 more times for a total of 11.
Here is the full command in case that's helpful to anyone:
gcloud firebase test android run \
--project locuslabs-android-sdk \
--app app/build/outputs/apk/debug/app-debug.apk \
--test app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk \
--device model=walleye,version=28,locale=en_US,orientation=portrait \
--test-targets "class com.locuslabs.android.sdk.TestUITest" \
--use-orchestrator \
--num-flaky-test-attempts 10 \
--timeout 30m \
--environment-variables numShards=10,shardIndex=2 \
--verbosity debug
The documentation states the following for --num-flaky-test-attempts:
Specifies the number of times a test execution should be reattempted if one or more of its test cases fail for any reason. An execution that initially fails but succeeds on any reattempt is reported as FLAKY.
I.e. if one test case in a test execution fails, Test Lab will re-run the whole test execution again. A test execution is comprised of running the whole test suite on one device.
Example: You execute your test suite on two devices, lets call them A and B. The whole test suite succeeds on A, but one test case fails on B. In this case only the test suite on device B will be re-attempted.
I am running the protractor Suite (spec file having multiple test cases), If any test case fails, protractor does not continue with the next test case execution and all the rest of test cases also fail.
EXPECTED BEHAVIOR:
Upon failure on any test case, protractor should continue with next test case execution.
I used "Protractor-Fail-Fast" Npm package to stop the rest test case execution if any test case fail. But ideally I am not looking for the same.
But this will not help me!
Just for reference: In Visual Studio MS test, If I created ordered test (same as Spec file in protractor having multiple test cases) and then set test setting like "continue on failure", ordered test execution will continue even if some test case failed.
I am looking for a similar test setting or any solution for protractor.
If you dont't want to stop all tests run just stop using Protractor-Fail-Fast library? Protractor tests run till the end by default even if some of the tests are failed.
set ignoreUncaughtExceptions: true in config file as following:
/**
* If set, Protractor will ignore uncaught exceptions instead of exiting
* without an error code. The exceptions will still be logged as warnings.
*/
ignoreUncaughtExceptions?: boolean;
you can get above description from here
export.config = {
...
ignoreUncaughtExceptions: true
}