When a before hook fails; the tests within that suite are skipped; how could i reliably get the list of those skipped tests ?
I tried to listen to the EVENT_TEST_FAIL on the Mocha runner and extract the information from there:
const Mocha = require('mocha');
const {
EVENT_TEST_FAIL,
} = Mocha.Runner.constants;
class MyReporter {
constructor(runner) {
runner
.on(EVENT_TEST_FAIL, (test, err) => {
console.log(err, test);
console.log("failed tests", test.parent.tests.map(t => t.fullTitle()));
});
}
}
module.exports = MyReporter;
But only issue is that, this code cannot distinguish between a failure in the before hook and a failure in a particular test case.
As a result, this code reports all tests in the suite as failed, even when only one test is failed.
I want to add screenshots for passing tests in the mochawesome html report.
I have managed to add screenshots for a failing test in the html report via the following and was wondering if this could be updated to support screenshots for passing tests?
index.js
import addContext from 'mochawesome/addContext'
Cypress.on('test:after:run', (test, runnable) => {
if (test.state === 'failed') {
addContext({test}, { title: "Screenshot", value:`${Cypress.config('screenshotsFolder')}/${Cypress.spec.name}/${runnable.parent.title} -- ${test.title} (failed).png` })
}
})
I am facing an issue with cleanup action after test run, What I need to do is after each test run, I need to call a certain API to store the result , here's my simplified code,
Cypress.on('test:after:run', (test) => {
Cypress.log({
name: 'Reporting..',
message: 'Sending Test result..',
})
...
add_result(test..) //async request
...
Cypress.log({
name: 'Reporting..',
message: 'Test Result Send Succesfully',
})
})
And this works for most of the test, but it doesnt work for the last test run of spec file, i am assuming it is because the browser is closed after the last test run but i am not sure, What can i do here?
I'm trying to add test step information into an allure report, i use cypress and generate report with allure, report are correctly generate but no details appears in testcases.
I try to use the on('task') without success...
my plugins/index.js contain:
...
on('task', {
allureTestStep () {
const testStep = allure.createStep("Initial", () => {
console.log("First Test")
});
testStep()
return null
}
})
...
and i call it with:
cy.task('allureTestStep')
in my test.
No log in console only two error:
allure-js-commons: Unexpected startStep() of initial. There is no parent step
First Test
allure-js-commons: Unexpected endStep(). There are no any steps running
and in the report nothing is displayed(no error, no step detail).
Any help is welcome :)
I'm using the cypress-allure plugin and you can set it to show cypress commands in the test results. If you call cy.log those also get placed there. It's pretty nice.
it(`Should have title of ${treeListTitle}`, () => {
cy.log('title should be set');
...
}
Notice in the allure results where that is printed out.
I'll sometimes have 1 or 2 tests that fail in CI, and rerunning the build causes them to pass.
How can I automatically re-run these flaky tests so my build will pass the first time? Is there something similar to mocha's this.retries?
For example, I have a test that fails with "The element has an effective height of 0x0" about 10% of the time:
cy.visit('/')
cy.get('.my-element').click() // sometimes fails with not visible error
Update (v5.0.0)
Cypress now has built-in retry support.
You can set test retries in Cypress 5.0 via configuration in cypress.json
{
"retries": 1
}
or specify different options for runMode and openMode:
{
"retries": {
"runMode": 1,
"openMode": 3
}
}
runMode allows you to define the number of test retries when running cypress run
openMode allows you to define the number of test retries when running cypress open
You can turn on test retries for just a single test or suite via test options:
it('my test', {
retries: 2
}, () => {
// ...
})
// or
describe('my suite', {
retries: 2
}, () => {
// ...
})
If a test fails in a beforeEach, afterEach, or in the test body, it will be retried. Failures in beforeAll and afterAll hooks will not retry.
Old answer:
Official Support for test retries is on the way, but there's a plugin for that. cypress-plugin-retries
Disclosure: I'm the creator of the plugin.
Installation
Add the plugin to devDependencies
npm install -D cypress-plugin-retries
At the top of cypress/support/index.js:
require('cypress-plugin-retries')
Usage
Use the environment variable CYPRESS_RETRIES to set the retry number:
CYPRESS_RETRIES=2 npm run cypress
or use Cypress.env('RETRIES') in your spec file:
Cypress.env('RETRIES', 2)
or on a per-test or per-hook basis, set the retry number:
Note: this plugin adds Cypress.currentTest and you should only access it in the context of this plugin.
it('test', () => {
Cypress.currentTest.retries(2)
})
Note: Please refer to this issue for updates about official cypress retry support
Cypress 5 now has a native support for retries. Check: https://cypress.io/blog/2020/08/19/introducing-test-retries-in-cypress-5-0
For retry, you can add it in your config as below
{
"retries": {
// Configure retry attempts for `cypress run`
// Default is 0
"runMode": 2,
// Configure retry attempts for `cypress open`
// Default is 0
"openMode": 0
}
}
but instead of adding above I prefer to add (Which works for both run and open mode)
"retries": 1
Points to consider
You need cypress version > 5
As of now in cypress, you can retry test, not entire spec which will be a great ability to make tests more robust. I have already voted this as a critical feature (Also this one of the most voted for now)
Ref:
https://portal.productboard.com/cypress-io/1-cypress-dashboard/tabs/1-under-consideration
Cypress supports test retries as of version 5.0.0, released on 8/19/2020. These are present to reduce test flakiness and continuous integration build failures. This feature is documented in the online Cypress documentation under the Test Retries guide.
By default, tests will not retry when they fail. To retry failing tests, test retries need to be enabled in the configuration.
Retries can be configured separately for run mode (cypress run) vs. open mode (cypress open), as these will typically be different.
There are two ways to configure retries: globally, and per test or test suite.
Global Configuration
The Cypress configuration file (cypress.json by default) allows configuring the retry attempt either per mode or for all modes.
To use a different value per mode:
{
"retries": {
"runMode": 2,
"openMode": 0
}
}
To use the same value for both modes:
{
"retries": 1
}
Single test configuration
A test's configuration can specify the number of retry attempts specific to that test:
// Customize retry attempts for an individual test
describe('User sign-up and login', () => {
// `it` test block with no custom configuration
it('should redirect unauthenticated user to sign-in page', () => {
// ...
})
// `it` test block with custom configuration
it(
'allows user to login',
{
retries: {
runMode: 2,
openMode: 1,
},
},
() => {
// ...
}
)
})
Single test suite configuration
A test suite's configuration can specify the number of retry attempts for each test within that suite:
// Customizing retry attempts for a suite of tests
describe('User bank accounts', {
retries: {
runMode: 2,
openMode: 1,
}
}, () => {
// The per-suite configuration is applied to each test
// If a test fails, it will be retried
it('allows a user to view their transactions', () => {
// ...
}
it('allows a user to edit their transactions', () => {
// ...
}
})