How can I retry a failed test? - continuous-integration

Question

I'll sometimes have 1 or 2 tests that fail in CI, and rerunning the build causes them to pass.
How can I automatically re-run these flaky tests so my build will pass the first time? Is there something similar to mocha's this.retries?
For example, I have a test that fails with "The element has an effective height of 0x0" about 10% of the time:
cy.visit('/')
cy.get('.my-element').click() // sometimes fails with not visible error

Answer 1

Update (v5.0.0)
Cypress now has built-in retry support.
You can set test retries in Cypress 5.0 via configuration in cypress.json
{
"retries": 1
}
or specify different options for runMode and openMode:
{
"retries": {
"runMode": 1,
"openMode": 3
}
}
runMode allows you to define the number of test retries when running cypress run
openMode allows you to define the number of test retries when running cypress open
You can turn on test retries for just a single test or suite via test options:
it('my test', {
retries: 2
}, () => {
// ...
})
// or
describe('my suite', {
retries: 2
}, () => {
// ...
})
If a test fails in a beforeEach, afterEach, or in the test body, it will be retried. Failures in beforeAll and afterAll hooks will not retry.
Old answer:
Official Support for test retries is on the way, but there's a plugin for that. cypress-plugin-retries
Disclosure: I'm the creator of the plugin.
Installation
Add the plugin to devDependencies
npm install -D cypress-plugin-retries
At the top of cypress/support/index.js:
require('cypress-plugin-retries')
Usage
Use the environment variable CYPRESS_RETRIES to set the retry number:
CYPRESS_RETRIES=2 npm run cypress
or use Cypress.env('RETRIES') in your spec file:
Cypress.env('RETRIES', 2)
or on a per-test or per-hook basis, set the retry number:
Note: this plugin adds Cypress.currentTest and you should only access it in the context of this plugin.
it('test', () => {
Cypress.currentTest.retries(2)
})
Note: Please refer to this issue for updates about official cypress retry support

Answer 2

Cypress 5 now has a native support for retries. Check: https://cypress.io/blog/2020/08/19/introducing-test-retries-in-cypress-5-0

Answer 3

For retry, you can add it in your config as below
{
"retries": {
// Configure retry attempts for `cypress run`
// Default is 0
"runMode": 2,
// Configure retry attempts for `cypress open`
// Default is 0
"openMode": 0
}
}
but instead of adding above I prefer to add (Which works for both run and open mode)
"retries": 1
Points to consider
You need cypress version > 5
As of now in cypress, you can retry test, not entire spec which will be a great ability to make tests more robust. I have already voted this as a critical feature (Also this one of the most voted for now)
Ref:
https://portal.productboard.com/cypress-io/1-cypress-dashboard/tabs/1-under-consideration

Answer 4

Cypress supports test retries as of version 5.0.0, released on 8/19/2020. These are present to reduce test flakiness and continuous integration build failures. This feature is documented in the online Cypress documentation under the Test Retries guide.
By default, tests will not retry when they fail. To retry failing tests, test retries need to be enabled in the configuration.
Retries can be configured separately for run mode (cypress run) vs. open mode (cypress open), as these will typically be different.
There are two ways to configure retries: globally, and per test or test suite.
Global Configuration
The Cypress configuration file (cypress.json by default) allows configuring the retry attempt either per mode or for all modes.
To use a different value per mode:
{
"retries": {
"runMode": 2,
"openMode": 0
}
}
To use the same value for both modes:
{
"retries": 1
}
Single test configuration
A test's configuration can specify the number of retry attempts specific to that test:
// Customize retry attempts for an individual test
describe('User sign-up and login', () => {
// `it` test block with no custom configuration
it('should redirect unauthenticated user to sign-in page', () => {
// ...
})
// `it` test block with custom configuration
it(
'allows user to login',
{
retries: {
runMode: 2,
openMode: 1,
},
},
() => {
// ...
}
)
})
Single test suite configuration
A test suite's configuration can specify the number of retry attempts for each test within that suite:
// Customizing retry attempts for a suite of tests
describe('User bank accounts', {
retries: {
runMode: 2,
openMode: 1,
}
}, () => {
// The per-suite configuration is applied to each test
// If a test fails, it will be retried
it('allows a user to view their transactions', () => {
// ...
}
it('allows a user to edit their transactions', () => {
// ...
}
})