I have two scenarios, scenario 1 should run at 5:00 PM everyday. Gather the DB results from it and save it on yaml file. I have to use these results to run scenario 2 which needs to be run next day at 8AM. How can I implement this on Jenkins.
Write now I am trying to implement through rake task. My logic is if the yaml file is present run scenario 2 or else run scenario 1.
Is there a better way to do it?
I need these two scenarios on the same jenkins job.
Related
Our automation tests run in gitlab CI environment. We have a regression suite of around 80 tests.
If a test fails due to some intermittent issue, the CI job fails and since the next stage is dependent on the Regression one, the pipeline gets blocked.
We retry the job to rerun regression suite expecting this time it will pass, but some other test fails this time.
So, my question is:
Is there any capability using which on retrying the failed CI job, only the failed tests run (Not the whole suite)?
You can use the retry keyword when you specify the parameters for a job, to define how many times the job can be automatically retried: https://docs.gitlab.com/ee/ci/yaml/#configuration-parameters
[Retry Only Failed Scenarios]
Yes, but it depends. let me explain. I'll mention the psuedo-steps which can be performed to retry only failed scenarios. The steps are specific to pytest, but can be modified depending on the test-runner.
Execute the test scenarios with --last-failed. At first, all 80 scenarios will be executed.
The test-runner creates a metadata file containing a list of failed tests. for example, pytest creates a folder .pytest_cache containing lastfailed file with the list of failed scenarios.
We now have to add the .pytest_cache folder in the GitLab cache with the key=<gitlab-pipeline-id>.
User checks that there are 5 failures and reruns the failed job.
When the job is retried it will see that now .pytest_cache folder exists in the GitLab cache and will copy the folder to your test-running directory. (shouldn't fail if the cache doesn't exist to handle the 1st execution)
you execute the same test cases with the same parameter --last-failed to execute the tests which were failed earlier.
In the rerun, 5 test cases will be executed.
Assumptions:
The test runner you are using creates a metadata file like pytest.
POC Required:
I have not done POC for this but in theory, it looks possible. The only doubt I have is how Gitlab parses the results. Ideally in the final result, all 80 scenarios should be pass. If it doesn't work out this way, then we have to have 2 jobs. execute tests -> [manual] execute failed tests to get 2 parsed results. I am sure with 2 stages, it will definitely work.
You can use Retry Analyser. This will help you definitely.
I want to run a load test on my server. I have built couple of different test plans and I need to execute all the test plans at the same time.
I'm using a few instances of JMeter on three different machines and I'm trying to figure out whether I can schedule my JMeter scripts to be started at a certain time?
One way would be to run, on each of my machines, a script in which I run my JMeter test plans at a certain time
The reason I decided to not use JMeter in master/slave mode is that each of my test plans would generate a load from different parts of my application. So I need each of my JMeter instances to run a different Thread Group.
If you don't want to install software as Jenkins
You can schedule jmeter execution in exact time using crontab, example:
Schedule a cron to execute at 2am daily.
This will be useful for scheduling database backup on a daily basis.
0 2 * * * /bin/sh backup.sh
Depending on your operating system:
Windows: Task Scheduler
Linux: crontab
MacOS: launchd
You can also consider installing a CI/CD solution like Jenkins, BuiltBot or CruiseControl which provide VCS system integration as well as possibility to schedule builds at the specific time or run basing on various triggers.
a solution to my question is to use "at" command which allows us to schedule a job at a certain time. Since I needed my test script to be executed only once, I decided to take this approach.
For example, following command would schedule a task at 12:01 PM (you need to make sure that your user is assigned all the required permissions to run this job)
echo "JMETER_HOME/jmeter -n -t MT_TEST_PLAN.jmx -l MY_TEST_RESULT.jtl" | at 12:01 PM
Although, I do not support the usage of Jmeter GUI for executing load tests, it depends with how many users you are executing. If the user load is small, you can use Start Time/End Time in the Threadgroup.
If you are using windows machines to execute load test, you can use windows task scheduler to run load tests at some specific time.
I want to compare 2 different jmeter runs.
I have a jenkins job to do that which triggers jmx which in turn call Rest APIs.
Lets say, I executed Jmeter run for one time. I will refer this as run 1
Now, after 30 minutes, I will again run the same jenkins job and will again run the jmeter test. I will refer to it as run 2.
So, now i have 2 runs and i want to compare run1 and run2, specifically the response time.
How can I automate that so that everytime this happens, i can have the difference in response time of APIs ?
I tried searching, found few articles (Compare results from a previous test in jmeter) but it did not really help :(
Please let me know how can this be achieved ?
You can use MergeResults plugin in order to compare 2 test executions and plot execution chart and/or have differences in the CSV file.
If you want to run it in non-GUI mode from Jenkins job - go for JMeterPluginsCMD Command Line Tool which has MergeResults plugin to run it in command-line mode.
You can install both the plugins using JMeter Plugins Manager
I have a number of different projects, with Jenkins CI jobs configured for each of them to run tests. When I create a new release, I have a second job that coordinates between a number of different jobs that go over each of the modules in the projects and updates the versions and the dependencies in the pom.xml's. I would like to make the "update" job conditional on the status of all the CI jobs - meaning that if one of the CI jobs is not green, then the update job will not run at all.
I had a look at the Run Condition Plugin as well as the Conditional BuildStep Plugin, however it does not seem possible do configure them to be dependent on the status of another Jenkins job.
you could hit the other jobs via the API at [JOB_URL]/lastCompletedBuild/api/json and verify the result for each.
to mess around with this:
curl `[JOB_URL]/lastCompletedBuild/api/json` | jq '.result'
you probably want result to say SUCCESS.
this is not fancy, but you don't want fancy in CI; you want something that is not likely to break when you upgrade jenkins. :)
Have a [https://wiki.jenkins.io/display/JENKINS/Multijob+Plugin] ["Multijob Plugin"] ,
In your case, you can add a job in first step and configure in that step, at which result condition of first step, you want to run second step.
Again, in second step, you can configure one/many jobs and can also configure if you want to run them in parallel.
I have almost 50 freestyle Jenkins builds that run as many performance centre tests everyday morning and evening. Currently I am getting status of these individual runs on email but would like to consolidate the results in one e-mail. Now problem with this is output of Jenkins build is always pass when it is able to run pc tests. To find actual result I need to see the artifact that contains HTML result. Is there a way I can read these individual HTML output and group them in one report. Like
Dev test prod
Test1 pass fails pass
Test2 fail pass pass
Test3 pass pass pass
I have little programming or scripting exp so pls forgive me for not using much resources on my own
You may use build result trigger plugin to monitor as many jobs required. For consolidating use tutorial HTML Agility Pack