I have a cypress workflow in Github and it runs nicely. But, when the e2e tests fail for some reason and I want to re-run them using the re-run all jobs button (below), the following message appears:
The run you are attempting to access is already complete and will not accept new groups.
The existing run is: https://dashboard.cypress.io/projects/abcdef/runs
When a run finishes all of its groups, it waits for a configurable set of time before finally completing. You must add more groups during that time period.
The --tag flag you passed was:
The --group flag you passed was: core
What should I change in my configuration to make these possible? Sometimes the e2e fails because of a backend error that is fixed later.
I'd like to do this instead of a force e2e commit.
I was facing the same issue before.
I think you can try to pass GITHUB_TOKEN or add a custom build id. It fixed my issue. Hoep it helps.
https://github.com/cypress-io/github-action#custom-build-id
Check your Cypress Dashboard subscription plan. Mine got the free plan full (500 test for free and I was running in 3 different browsers 57 tests, so it got full pretty quick since this is 171 tests in one run) and after that it didn't allowed me to keep running or re running more parallel tests. Test kept running but in 1 machine out of 4 in the first browser and stages for the other 2 browsers started failing, I was able to allow the pipeline to not be failing by passing continueOnError: true in the configuration.
Quick edit, I don't remember where but I read that you could also add a delay to your pipeline and/or reduce the default wait on the Dashboard which is 60s(https://docs.cypress.io/guides/guides/parallelization#Run-completion-delay)
Related
Our automation tests run in gitlab CI environment. We have a regression suite of around 80 tests.
If a test fails due to some intermittent issue, the CI job fails and since the next stage is dependent on the Regression one, the pipeline gets blocked.
We retry the job to rerun regression suite expecting this time it will pass, but some other test fails this time.
So, my question is:
Is there any capability using which on retrying the failed CI job, only the failed tests run (Not the whole suite)?
You can use the retry keyword when you specify the parameters for a job, to define how many times the job can be automatically retried: https://docs.gitlab.com/ee/ci/yaml/#configuration-parameters
[Retry Only Failed Scenarios]
Yes, but it depends. let me explain. I'll mention the psuedo-steps which can be performed to retry only failed scenarios. The steps are specific to pytest, but can be modified depending on the test-runner.
Execute the test scenarios with --last-failed. At first, all 80 scenarios will be executed.
The test-runner creates a metadata file containing a list of failed tests. for example, pytest creates a folder .pytest_cache containing lastfailed file with the list of failed scenarios.
We now have to add the .pytest_cache folder in the GitLab cache with the key=<gitlab-pipeline-id>.
User checks that there are 5 failures and reruns the failed job.
When the job is retried it will see that now .pytest_cache folder exists in the GitLab cache and will copy the folder to your test-running directory. (shouldn't fail if the cache doesn't exist to handle the 1st execution)
you execute the same test cases with the same parameter --last-failed to execute the tests which were failed earlier.
In the rerun, 5 test cases will be executed.
Assumptions:
The test runner you are using creates a metadata file like pytest.
POC Required:
I have not done POC for this but in theory, it looks possible. The only doubt I have is how Gitlab parses the results. Ideally in the final result, all 80 scenarios should be pass. If it doesn't work out this way, then we have to have 2 jobs. execute tests -> [manual] execute failed tests to get 2 parsed results. I am sure with 2 stages, it will definitely work.
You can use Retry Analyser. This will help you definitely.
We have a build pipeline for our automated scripts (selenium) in Azure-Devops. And we use browserstack to run all our scripts. Sometimes we get timeout issue, no matter what we implement additional options to browser settings we still get timeout. So we decided to re run the tests to certain limit and then the pass percentage went high and scripts were passed without any issues.
So we have different data rows for each test method. Now when a test method for a particular data row is failed then the entire test (including all the passed data row) are getting executed again in the re-run which is unnecessary since some are passed.
Is there any way to run only failed data row test ?
As seen in below screenshot on the regular attempt, data row 0 is failed and others passed. So it goes for re-run and running all the passed test again.
Test result screenshot
Note: Using Batch option to re-run the failed tests. Also tried with Automatic batch option where it failed to re-run because of test name issue of vstest.console.exe (There is formatting issue if the test name has spaces or round braces)
i want to log each and every step of the process ex: sign up done, user dashboard opened , user profile edited. How can this be achieved by cypress. I tried this using node.js but as cypress runs in browser i cant create file using node.js and also i tried by cy.writefile where it replace the previous this i have written in the file. Is there a way to do this ???
After this i do want to send this log to an email.
Welcome to Stack Overflow. If you run Cypress on a CI Server like CircleCI, it can record your output.
There is also a --record option, where it saves the details of the session for you.
Cypress can record your tests running and make them available in our
Dashboard.
Recorded tests allow you to: See the number of failed, pending and
passing tests. Get the entire stack trace of failed tests. View
screenshots taken when tests fail and when using cy.screenshot().
Watch a video of your entire test run or a clip at the point of test
failure.
You can read more about recording sessions here https://docs.cypress.io/guides/guides/continuous-integration.html#Recording-tests-in-CI
I have around 20 transactions in my jmeter script, I need to run a load test for 2000 Users.
I have added test checks and Assertions to validate if the transactions are passing
I want to introduce error handling , where the script start from the beginning if any of the assertions/transactions fail
I am not sure if I have understood your question correctly. If your aim is to stop the workflow for the current user and start from the beginning, then you could use until controller.
More information on this can be found here.
Currently, I am initiating a build by posting a few parameters to Jenkins from a shell script. I need to check whether the build succeeded or failed and I was wanting to avoid using the post build Jenkins script calls (I don't want Jenkins to initiate the running of any scripts on my server), so the idea was to post to Jenkins every 10 seconds or so (while building != false) in order to get the JSON object with the various build parameters. While this is working fine if I know the build number of the build I want to check on, I can't seem to see a good way to dynamically keep track of the current build number and make sure my script is checking on the build it just initiated and not some other build currently running.
Potentially, there could be multiple builds initiated within a short period of time, so posting to jenkins/job/my_build_job/lastBuild/api/json just after starting the build and checking the number that way doesn't seem appropriate given problems with race situations.
How can I keep track of a particular build dynamically from a script on my server in order to check the build success or failure of a build initiated from a post called by cron? Is there perhaps a way to name a build so I could initiate it with BUILD_NAME and then post to jenkins/job/my_build_job/BUILD_NAME/api/json?
There are a couple of different API calls you can make:
jenkins/job/my_build_job/api/json?tree=lastBuild[number]
will give you either the last completed build or the current build in progress
jenkins/job/my_build_job/api/json?tree=nextBuildNumber
will give you the next build number - this includes builds that are queued up waiting for resources.
There is already an issue filed in Jenkins to return the build number in the Jenkins remote API call: https://issues.jenkins-ci.org/browse/JENKINS-12827. Please add comments there so it can be worked on as soon as possible.