I would like to have Teamcity build configuration that currently has 3 build steps:
Build an artifact to perform tests on & install on remote server
Kick off long running test job on remote server
Pause build awaiting external event (i.e. remote job finishing)
Retrieve results and record the report
I have had a look through the documentation and I can see how I can pause (step 3) the entire build configuration (which stops any additional builds running) ... but not just a single running build.
The Step 2 script that is running the external job has the various parameters passed to it, so that it can issue a REST call back to the teamcity server to resume the build job.
Basically I don't want to tie up a build agent waiting the entire hour the test takes to run.
I have googled and everything I can find points me at pausing the build configuration.
I am currently having to look at splitting the build configuration into two. The first will kick of the test job and finish. Then when the external test job finishes it will call teamcity to start a second job to retrieve and store the reports. But that feels disconnected to me in that I will not be able to show a single job with build/test/report.
At the moment (TeamCity v 2018.1) there is no direct way to pause the build, release the build agent, and later resume the execution.
What you described is the recommended workaround.
Also, please watch/vote for related issue: https://youtrack.jetbrains.com/issue/TW-30777
Related
I have a scenario where I would like a build to start running on one agent (Job 1), and then after doing some work, I'd like it to run a step on a special agent (pool) of machines with specially licensed software. (Job 2). When that is done I'd like the rest of the build to complete on the original agent (Job 3).
I have been able to use "Variable Tools for Azure DevOps Services" to successfully pass any number of variables between agent jobs, even when they are running on different machines. It is no problem for me to pass a UNC path from Job1 to Job2 / Job3, etc.
However, what I am seeing is that no matter what I do, agent jobs are always running in parallel, and there is no way to get them to run serially, unless they are locked to the same agent on the same machine, which defeats the whole purpose.
Does anyone know of a means to accomplish this? Right now in tests, I have to use "Start-Sleep" or something similar, and repeatedly monitor an external event. A terribly inelegant work-around.
I found the answer. A job properties contains a field called "dependencies". You can make it serial by setting a dependency on the previous job.
In Azure Devops for the agent job you will get below options
You can select any option based on your requirements.
I am trying to setup selenium grid to achieve parallel execution of my tests. First, I'll explain the my current scenario.
I have my fully functional test suite written in cucumber with watir webdriver
I need to execute all my tests in multiple environments.
I created a setup for selenium hub and node
I can run my tests on a single node through hub
My goal is to run my tests on multiple vm's simultaneously.
I missing a part where I need to configure my tests to run in parallel. there are some examples about grid setup in the web, as I am using different framework I couldn't relate to my scenario.
Thanks in advance
I was able to do this by leveraging Jenkins with Selenium Grid... Why Jenkins? a) Jenkins is a build tool and is built to run parallel jobs by default. I leverage that ability to send tests to Selenium Grid in parallel, the Grid manages the flow from that point on. b) Jenkins is part of many development build processes. Dev's could make calls to your QA jenkins to kick off tests as they commit/build. c) Jenkins provides a nice UI to see pass/failures of your tests (as well as sending email notifications of failures) and d) Jenkins has a great Cucumber reporting plugin.
You could avoid Jenkins, but you'll need to send your cucumber features in parallel to the Grid. If you just run cucumber, it will farm jobs to the grid, but they will run sequentially. You would need something to kick off each feature async.
Below is my full set up. Jenkins is basically being used here to start multiple/simultaneous cucumber jobs. the details are below:
Grid Setup
I had 10 VM's. I took VM1 as my primary. It was a windows server box, so I put the selenium grid standalone on it and wrote a batch file like so:
#echo off
“C:\[Add your Java Path Here]\java.exe” -jar “C:\[Add your Selenium Grid Jar Path]\selenium-server-standalone-2.31.0.jar” -role hub
Then used the windows server task to auto run that batch file in case of VM restart.
On each of the other VM's I made them part of the grid by registering them with VM1 (hub):
java -jar selenium-server-standalone-2.31.0.jar -role node -hub http://[the server name of your Selenium Hub]:4444/grid/register -browser browserName=chrome,maxInstances=5
Cucumber Set UP
In Cucumber, I set up a env.rb file in the features/support folder. This allows me to specify command line arguments before the tests run, and what happens when they stop. I added a begin statement that set the a value for using a browser, as well as using the Grid...
Browser & Environment Config
In the env.rb file I add:
def browser_name
(ENV['BROWSER'] ||= ‘firefox’).downcase.to_sym
end
def environment
(ENV['ENVI'] ||= ‘int’).downcase.to_sym
end
Grid Config
Then I add:
Before do |scenario|
p "Starting #{scenario}"
if environment == :int
#browser = Watir::Browser.new(:remote, :url=>"http://[Your Selenium Grid Hub]:4444/wd/hub", :desired_capabilities=> browser_name)
#Optional: in the case of setting your default start page #browser.goto "http://[your start page of your test site]:8080"
elsif environment == :local
#browser = Watir::Browser.new browser_name
#browser.goto "http://[some other environment]:8080"
end
end
Now you could pass a argument like: cucumber feature/login.feature BROWSER=firefox ENV=int and it would farm all the work to the Grid HUB - which should pass it to the Grid NODES it's connected to, with the browser support (i.e send the test for login.feature to a firefox compatible node - maybe not all your nodes have firefox, if they all do, then it will go to any one of them.)
At this point, you get one job at a time going through. So how to run more then one?
You would have a script that kicks off all feature files (or sans-Cucumber, your tests) through that same browser profile to use the Grid HUB. If you use a script it needs to make these calls asynchronously - so all the features/tests are sent to the Grid at the same time and the Grid manages the jobs.
How I did that was with...
Jenkins
Jenkins is used for builds/deploys of code - but in my case I use it to trigger QA jobs. Jenkins is a Java JAR, that you just run... i.e. java jenkins.jar It launches a local UI on some port, and you can start adding jobs.
From a high level
I built Jenkins jobs and had a parent job that ran all jobs - sending them to the Grid. The Selenium Grid Hub would then manage the flow of the jobs.
I wanted to be able to kick off individual feature tests, individual feature tests by browser, and all tests by browser. To do that I started with individual jobs for each feature by browser.
Details
In Jenkins I created "A New Job" and choose "Free style software component" and then filled out the description with "[my feature name] [browser name]" i.e. Login Tests via IE
Down in the Build section of this Jenkins job, I choose to use a BATCH Command. Since it's a windows box, I picked Windows Batch command and input something like this:
bundle exec cucumber BROWSER=ie ENVI=int features/login.feature --format json -o login-results/login.json
That stuff from --format on, that's all making use of a Cucumber reports plugin for Jenkins. It makes these nice looking graph driven reports on the features tests that passed/failed. You don't need it, it's optional.
If you "build" the Jenkins job, it will execute that windows batch file and it does the following:
Starts the Job
Runs a cucumber command to use a specific browser (IE in this case)
Runs all the tests of that feature file (i.e. login.feature could have 20 tests)
All tests run through the grid, which farms them off to nodes
It's not yet running jobs in parallel though.
Running Jobs in Parallel
Now that Jenkins can kick off the tests by feature and browser, via a Grid - we can now finally run jobs in parallel, we just need more jobs. So create a few more jobs... like:
Jenkins job for registration.feature, and jenkins job for subscription.feature. Each will have it's own windows batch command like:
bundle exec cucumber BROWSER=ie ENVI=int features/registration.feature --format json -o registration/registration.json
Or
bundle exec cucumber BROWSER=ff ENVI=int features/registration.feature --format json -o registration/registrationff.json
That last one is a duplicate of the registration test, just calling a different browser.
Jenkins by default limits how many simultaneous jobs you can run.. I think it's 10. You can change that in the Jenkins config. I changed mine to 40.
So now, you can click on your first Jenkins job:
Login Test via IE
and click BUILD
As it starts, click "BUILD" on your second job:
Registration Test via IE
and the same for your other jobs...
Jenkins will start each job in parallel, farming them to the Grid HUB, which sends the jobs to your appropriate nodes! All in parallel. By default Jenkins limits the jobs to 5 or 10 parallel jobs. You can change that. I modified mine to be 25 to 40, depending on my needs.
Jenkins will also update the UI with pass/fail and has details of the failures in the logs of each job. You can see the history of the failures... and another Jenkins repo (i.e. the dev repo) can make rest calls to your repo to auto trigger these test as they build.
Launching All Tests From One Job
Rather then manually running your individual jobs, you can build a parent job. In Jenkins the parent job would be a NEW Job that is the same type, "Free style software component." You go down to the very bottom and it should have a field called, "Build Other Projects" In that field you can put each project for deploying: Login, Registration, etc.
Now when you "Build" this parent job, you'll see Jenkins starts all jobs/Projects to kick off at the same time. Basically your entire set of tests could start with one click - all being sent to Selenium Grid, which would then manage the flow.
Group Tests
In Jenkins I create tabs for my tests... IE Tests, FF Tests, Chrome Tests, etc.
In each tab I put the appropriate features.
Then I create a new Parent job to kick off all jobs of a type like:
All IE Tests
All FF Tests
etc.
Conclusion
You can probably avoid Cucumber if you like. You just need something to kick off the jobs in parallel, for me I used Jenkins to do that. You could use a script that runs async jobs (kicking off your tests.)
Hopefully something in post is useful to your needs.
I have this documented at my site, along with some pictures of the Jenkins flow...
URLs
My tutorial on setting up Cucumber and Selenium Grid:
http://sdet.us/selenium-grid-with-watir-and-cucumber-browser-automation/
My tutorial on setting up Cucumber Reports with Jenkins:
http://sdet.us/jenkins-and-cucumber-reports/
look at MBUnit as this can run tests in parallel, This should help. It will only run tests in parallel in one assembly and will not coordinate across multiple assemblies.
Currently, I am initiating a build by posting a few parameters to Jenkins from a shell script. I need to check whether the build succeeded or failed and I was wanting to avoid using the post build Jenkins script calls (I don't want Jenkins to initiate the running of any scripts on my server), so the idea was to post to Jenkins every 10 seconds or so (while building != false) in order to get the JSON object with the various build parameters. While this is working fine if I know the build number of the build I want to check on, I can't seem to see a good way to dynamically keep track of the current build number and make sure my script is checking on the build it just initiated and not some other build currently running.
Potentially, there could be multiple builds initiated within a short period of time, so posting to jenkins/job/my_build_job/lastBuild/api/json just after starting the build and checking the number that way doesn't seem appropriate given problems with race situations.
How can I keep track of a particular build dynamically from a script on my server in order to check the build success or failure of a build initiated from a post called by cron? Is there perhaps a way to name a build so I could initiate it with BUILD_NAME and then post to jenkins/job/my_build_job/BUILD_NAME/api/json?
There are a couple of different API calls you can make:
jenkins/job/my_build_job/api/json?tree=lastBuild[number]
will give you either the last completed build or the current build in progress
jenkins/job/my_build_job/api/json?tree=nextBuildNumber
will give you the next build number - this includes builds that are queued up waiting for resources.
There is already an issue filed in Jenkins to return the build number in the Jenkins remote API call: https://issues.jenkins-ci.org/browse/JENKINS-12827. Please add comments there so it can be worked on as soon as possible.
I have a UIcoded test that works in VS2010.
I want automated it with MTM. But when I associated it with test case and run it I get this error in ViewResult :
Microsoft.VisualStudio.TestTools.UITest.Extension.FailedToLaunchApplicationException: The application failed to start.
I am monitoring it in test agent status and every thing is ok.
how I can pass this test.
Sincerely you M.Bagheri
If you are running UI Coded test in the build process using TFS Build Service, make sure you run the build service as an interactive process.
The answer here also tells you how to make the interactive build service start when the build machine is started.
See here for how to run the build service as an interactive process.
Usually you will get a specific error if you cannot interact with the desktop.
I recently received this error. I performed the following tasks which solved the issue for me:
clear the controler temporary files
restart the agent and controller.
I have a job upstream that executes 4 downstream jobs.
If the upstream job finish successfully the downstream jobs start their execution.
The upstream job, since it finish successfully, gets a blue ball (build result=stable), but even tough the downstream jobs fail (red ball) or are unstable (yellow ball), the upstream job maintain its blue color.
Is there anyway to get the result of the upstream job dependent on the downstream jobs?, i mean, if three downstream jobs get a stable build but one of them get an unstable build, the upstream build result should be unstable.
I found the solution. There is a plugin called Groovy Postbuild pluging that let you execute a Groovy script in the post build phase.
Addind a simple code to the downstream jobs you can modify the upstream overall status.
This is the code you need to add:
upstreamBuilds = manager.build.getUpstreamBuilds();
upstreamJob = upstreamBuilds.keySet().iterator().next();
lastUpstreamBuild = upstreamJob.getLastBuild();
if(lastUpstreamBuild.getResult().isBetterThan(manager.build.result)) {
lastUpstreamBuild.setResult(manager.build.result);
}
You can find more info in the entry of my blog here.
Another option that might work for you is to use the parametrised build plugin. It allows you to have your 4 "downstream" builds as build steps. This means that your "parent" build can fail if any of the child builds do.
We do this when we want to hide complexity for the build-pipeline plugin view.
We had a similar sort of issue and haven't found a perfect solution. A partial solution is to use the Promoted Builds Plugin. Configure it for your upstream project to include some visual indicator when the downstream job finishes. It doesn't change the overall job status, but it does notify us when the downstream job fails.
Perhaps this plugin does what you are looking for?
Jenkins Prerequisite build step Plugin
the work around for my project is to create a new job, which is the down stream of the down streams. We set a post build step "Trigger parameterized build on other projects " in all three of the original downstream jobs. The parameter that parse into the new job depends on the three jobs' status and the parameter will causes the new job react accordingly.
1. Create new job which contains one simple class and one simple test. Both parameters dependens, i.e. class fail if parameter "status" = fail, class pass but test fail if parameter "status"=unstable, etc.
2. Set Trigger parameterized build on other projects for the three original downstream jobs with relevant configurations.
3. Set notification of the new job accordingly.