I need to produce a number of component integration tests to fit into an existing framework composed of Jmeter and run via Bamboo.
My problem is that I know nothing of Bamboo.
Working in a Linux environment.
Does anyone have any recommendations for either basic tutorials for Bamboo, or links to existing questions on here which would form a good start point for me?
Any help greatly appreciated.
The main idea of adding a JMeter test under continuous integration system control is having unattended JMeter test executions basing on triggers (on version control system commit, periodically, on-demand, whatever).
There are several ways of running a JMeter test via Bamboo, the easiest would be either running a shell Script or Custom Command Executable.
The results can be viewed and analysed using JMeter Aggregator plugin.
See How to Run JMeter in a Continuous Integration Environment With Bamboo for step-by-step configuration instructions and examples.
Related
When I run my selenium test (mvn test) from jenkins (windows) I see only the console output. I don't see the real browsers getting opened. How can I configure jenkins so that I can see the browsers running the test?
I think you're mis-understanding the reason you would use Jenkins, i believe it's designed to do exactly what it is doing, you need to install a different test-framework along with Maven and Selenium.
Jenkins is an open source DevOps tool which is basically use to automate repetitive tasks like code deployment on server.
SERVER being the main operating word here, you are trying to run your automation scripts from your desktop.
https://www.youtube.com/watch?v=peDWh9q_d0s
Above is a video tutorial of how to setup Maven and a very popular automation test framework, testNG.
Give me a shout if you need any more from me,
All the best,
Jack
Background
I am currently in the process of writing test scripts in Java with Testng, Maven, selenium and Jenkins. I have a plan to write hundreds of scripts. At this current time I have about 80 scripts written. Only 8 scripts have been uploaded to Bitbucket. Note that each script can have anywhere between 5-25 tests in it based on complexity, for example the 8 scripts currently on the server run 100 tests.
Problem
The issue I can see arising here very very quickly is that a huge amount of test scripts running.Jenkins runs the entire Maven project that sits on Bitbucket. Currently with only 8 scripts Jenkins takes a total of 20 mins to run. By the time I have the more complex ones up this could take hours to run even days with all of the scripts I plan on uploading.
Research
So far I've looked around for some way to break up the testing process so for instance I would have separate Maven projects in my Bitbucket repository for different areas. Then I would have several different builds on Jenkins one for each area of the site. I'm not sure how that would work though since Jenkins seems to just go in and read all of the tests on my repo.
I'm almost certain that having all of the tests in a single build is bad practice but I just cant find information on how to handle a huge test suit, I'm hoping someone with real experience can clarify this for me.
Software
Added as a side note in-case anyone wants to know what I'm using
Maven: 3.3.3
Java: 1.7.0_79
Selenium: 2.46 & 2.47(currently 2.47)
Jenkins: 1.622
Conclusion
I believe there must be a way of breaking the test suit up without having separate bit-bucket repos for each section.
One possible solution is using selenium grid. This will allow you to distribute tests over several vm's. Its fairly well documented.
I'm no experience with Jenkins, I'm currently researching different options for PHP & JS automated unit testing with Jenkins.
I've come across Chutzpah (which uses PhantomJS's headless WebKit browser) but:
Is it possible to use Chutzpah with Jenkins?
There's very little documentation on Chutzpah. Although it does state on the Chutzpah homepage that it can be integrated into the TeamCity continuous integration server.
What's the minimum requirements for something to be compatible with Jenkins?
It is possible to use Chutzpah with Jenkins and with the 2.1 release of Chutzpah it is easier. Chutzpah's command line client can now take /junit argument that lets you specify a file name to output a junit-xml compatible file to. You can use Jenkins to pick this file up and report the test results.
I am not the downvoter, but I agree it is difficult to give a good answer to this question.
I believe the minimum requirement for something to be compatible with Jenkins is: It can be executed from a shell or cmd script. (If it's not, you need to find or write a plugin.)
Additionally, the thing should exit with code 0 for success and anything else for failure. (If it doesn't, you need to find or write a plugin.)
If you are interested in having Jenkins publish test results, the results must be in xml files using junit compatible notation. (If they are not, you need find or write a plugin.)
Additional requirements might be imposed by the tool you want to execute: It might need to draw windows or access the mouse or other parts of a graphical UI desktop/session. If that's the case, you need to run Jenkins in a context/session where it has access to those. (Windows, Mac and Linux all restrict background daemon/service access to the GUI desktop.)
Also, if your tool needs to access resources which are accessible by only certain user, you need to run Jenkins as that user.
This is a very open-ended question. Please try it out and come back with more concrete questions.
Which tool/server can be used to perform continuous integration that helps running PHP UNIT tests, SELENIUM tests and Custom tests such as bash script ideally. I have got following options:
1. Hudson
2. PHP Under COntrol (plugin of cruise control)
3. jenkins
4. Bamboo (Atlassian)
5. TeamCity (JetBrains)
6. Apache Continuum
I need to know which is the best in my case, with some reasons. I appreciate your valuable thoughts.
Also maybe look at Sismo (from the Symfony folks) ?
Jenkins is best option for your problem. Currently we are also using jenkins as our CI server and we are running unit tests, selenium tests and other tests including load testing etc. with jenkins. Also it has support for multiple plugins.
I've just installed Hudson and it is running beautifully. It builds, runs JUnit-tests and also CheckStyle analysis.
Next step for us would be to create an installation, install it and then run automated tests on the actual installation. I would then like to fail the build if the tests fail or at least publish the results somehow. I think we would set it up so that part runs periodically or manually triggered.
We use InstallAnywhere for installation and IBM Rational Functional Tester for automated tests.
So questions are: anyone created a similar setup? are there any plugins, tutorials or other resource that could help me along. Or do you have any tips or advice in general.
The command line reference for Rational Functional Tester:
http://publib.boulder.ibm.com/infocenter/rfthelp/v8r0m0/index.jsp?topic=/com.ibm.rational.test.ft.doc/topics/RobotJCommandLine.html
Sample command for running a test:
java -classpath "C:\IBM\RFT\FunctionalTester\bin\rational_ft.jar"
com.rational.test.ft.rational_ft -datastore \\My_project\AUser\RobotJProjects -user admin -project
\\My_project\AUser\TestManagerProjects\Test.rsp -build "Build 1" -logfolder "Default" -log
"Al_SimpleClassicsA#1" -rt.log_format "TestManager" -rt.bring_up_logviewer true -playback
basetests.SimpleClassicsA_01
An additional note, you'll want to configure windows properly on your agent machine which will be running the tests. This is not advice specific to Hudson or RFT, but rather all GUI automation tools on Windows. RFT will require an interactive desktop environment for it to be able to click buttons, etc. If you have your Hudson agent running as a Windows service, there will be no desktop. See the following: Silverlight tests not working unless RDP connection open
We have run a fairly complicated distributed build on Hudson, it is a process that basically follows:
Test on Windows.
Test on OSX, run code coverage & push results to server.
Test on OSX Tiger.
Package for OSX Leopard & push build to server.
Package for Windows & push build to server.
Update product website.
We don't use InstallAnywhere or Rational Functional Tester, but have similar sorts of mechanisms in their place. The key we found to making it all sing in Hudson was being able run our various steps from the command line. Maven and appropriate plugins made short work of this task. So my advice would be just that, using whatever build tool you are using (ant, maven, ?) configure them so that you can run your rational functional tester and install anywhere from the command line with a simple goal passed to your build tool (i.e. mvn test or mvn assembly:assembly).
After that, make sure whatever machine Hudson is running on has everything installed (i.e. Rational Functional Tester) and configured, so that you can open up the command line and type in the goal and have your tests correctly execute.
Hooking it up in Hudson from that point on is fairly simple - just pass in the goal when you configure the build.
I believe the best answer is that integrating RFT with Hudson/Jenkins is a useless endeavor.
As this IBM FAQ says, to make RFT work you must:
be logged in the machine;
the screen can't be locked;
if you are remotely connected, you can't minimize the connection screen.
So you can't run Jenkins/Hudson as a service, making it not very useful. You must run it from your logged account. If you are in a corporate computer (very probable if you are using RFT), you probably must use a hack to prevent the screen saver to start. If the screen is locked, your tests will always fails.
It isn't very difficult to configure your tests to run from the command line, you just have to take care of the return codes when the tests fail and succeed.
Jenkins/Hudson would also give you some advantages, like integrating the tests with your version control, probably automatically running the tests when a commit is made. It would also help sending emails when the tests fail.
But you still would have to integrate the RFT logs with some kind of JUnit plugin to have a nice report. You also would have to have script to run the tests using the command line.
I think it is not worth the trouble to use an continuous integration server with RFT. Better just have your tests running every day in Windows Task Scheduler. It is a simpler solution with less failure points.
Or use my final solution: quit RFT and use the free Selenium with a headless web driver.
I have some general advice on this because I have not yet implemented this myself.
I am assuming you want to have Hudson run the RFT scripts automatically for you via a build or Hudson process?
I want to implement something similar in my organisation as well.
I have not yet been able to implement this because of organisational constraints but here is what I have thought out/done so far:
Downloaded Windows process viewer, got the command for running the tests.
Made shell Script out of it, separated out the variables etc
The future plan is to setup a Windows Slave machine which would have all the tools in it that would be required once the Tests are kicked off, for eg. the correct versions of browsers, and environment variables, and other tools that are required.
Hudson would kick off a process which runs the shell scripts created which runs all the RFT Scripts and performs necessary operations on the slave machine.