I've just installed Hudson and it is running beautifully. It builds, runs JUnit-tests and also CheckStyle analysis.
Next step for us would be to create an installation, install it and then run automated tests on the actual installation. I would then like to fail the build if the tests fail or at least publish the results somehow. I think we would set it up so that part runs periodically or manually triggered.
We use InstallAnywhere for installation and IBM Rational Functional Tester for automated tests.
So questions are: anyone created a similar setup? are there any plugins, tutorials or other resource that could help me along. Or do you have any tips or advice in general.
The command line reference for Rational Functional Tester:
http://publib.boulder.ibm.com/infocenter/rfthelp/v8r0m0/index.jsp?topic=/com.ibm.rational.test.ft.doc/topics/RobotJCommandLine.html
Sample command for running a test:
java -classpath "C:\IBM\RFT\FunctionalTester\bin\rational_ft.jar"
com.rational.test.ft.rational_ft -datastore \\My_project\AUser\RobotJProjects -user admin -project
\\My_project\AUser\TestManagerProjects\Test.rsp -build "Build 1" -logfolder "Default" -log
"Al_SimpleClassicsA#1" -rt.log_format "TestManager" -rt.bring_up_logviewer true -playback
basetests.SimpleClassicsA_01
An additional note, you'll want to configure windows properly on your agent machine which will be running the tests. This is not advice specific to Hudson or RFT, but rather all GUI automation tools on Windows. RFT will require an interactive desktop environment for it to be able to click buttons, etc. If you have your Hudson agent running as a Windows service, there will be no desktop. See the following: Silverlight tests not working unless RDP connection open
We have run a fairly complicated distributed build on Hudson, it is a process that basically follows:
Test on Windows.
Test on OSX, run code coverage & push results to server.
Test on OSX Tiger.
Package for OSX Leopard & push build to server.
Package for Windows & push build to server.
Update product website.
We don't use InstallAnywhere or Rational Functional Tester, but have similar sorts of mechanisms in their place. The key we found to making it all sing in Hudson was being able run our various steps from the command line. Maven and appropriate plugins made short work of this task. So my advice would be just that, using whatever build tool you are using (ant, maven, ?) configure them so that you can run your rational functional tester and install anywhere from the command line with a simple goal passed to your build tool (i.e. mvn test or mvn assembly:assembly).
After that, make sure whatever machine Hudson is running on has everything installed (i.e. Rational Functional Tester) and configured, so that you can open up the command line and type in the goal and have your tests correctly execute.
Hooking it up in Hudson from that point on is fairly simple - just pass in the goal when you configure the build.
I believe the best answer is that integrating RFT with Hudson/Jenkins is a useless endeavor.
As this IBM FAQ says, to make RFT work you must:
be logged in the machine;
the screen can't be locked;
if you are remotely connected, you can't minimize the connection screen.
So you can't run Jenkins/Hudson as a service, making it not very useful. You must run it from your logged account. If you are in a corporate computer (very probable if you are using RFT), you probably must use a hack to prevent the screen saver to start. If the screen is locked, your tests will always fails.
It isn't very difficult to configure your tests to run from the command line, you just have to take care of the return codes when the tests fail and succeed.
Jenkins/Hudson would also give you some advantages, like integrating the tests with your version control, probably automatically running the tests when a commit is made. It would also help sending emails when the tests fail.
But you still would have to integrate the RFT logs with some kind of JUnit plugin to have a nice report. You also would have to have script to run the tests using the command line.
I think it is not worth the trouble to use an continuous integration server with RFT. Better just have your tests running every day in Windows Task Scheduler. It is a simpler solution with less failure points.
Or use my final solution: quit RFT and use the free Selenium with a headless web driver.
I have some general advice on this because I have not yet implemented this myself.
I am assuming you want to have Hudson run the RFT scripts automatically for you via a build or Hudson process?
I want to implement something similar in my organisation as well.
I have not yet been able to implement this because of organisational constraints but here is what I have thought out/done so far:
Downloaded Windows process viewer, got the command for running the tests.
Made shell Script out of it, separated out the variables etc
The future plan is to setup a Windows Slave machine which would have all the tools in it that would be required once the Tests are kicked off, for eg. the correct versions of browsers, and environment variables, and other tools that are required.
Hudson would kick off a process which runs the shell scripts created which runs all the RFT Scripts and performs necessary operations on the slave machine.
Related
When I run my selenium test (mvn test) from jenkins (windows) I see only the console output. I don't see the real browsers getting opened. How can I configure jenkins so that I can see the browsers running the test?
I think you're mis-understanding the reason you would use Jenkins, i believe it's designed to do exactly what it is doing, you need to install a different test-framework along with Maven and Selenium.
Jenkins is an open source DevOps tool which is basically use to automate repetitive tasks like code deployment on server.
SERVER being the main operating word here, you are trying to run your automation scripts from your desktop.
https://www.youtube.com/watch?v=peDWh9q_d0s
Above is a video tutorial of how to setup Maven and a very popular automation test framework, testNG.
Give me a shout if you need any more from me,
All the best,
Jack
I am an intern now, new to automation test.My goal here is to help my company set up CI for client side.
Right now I have a maven project contains several tests using Appium java-client lib, under Eclipse IDE, which could run the UI tests locally. My goal next step is to hook my tests with the gitlab repo(which is already there, created by the android developers), but I am stuck here. Could somebody help me out?
Please try to be specific:
how should I set up the .gitlab.yaml?
can we just have the script in yaml to download Appium and maven?
or we could just download Appium, but import all the Appium java-client jars to libs in main?
If either of above is true, how? if neither, what and how should I
do?
Where should I put my test in gitlab in that repo? Or I don't have to
put my tests in the existing repo. Instead, I could have another one
and tell yaml where to reach? Again, how?
It will be helpful if you could help me go through the workflow.
Like, when I developers check in code, gitlab read the yaml, then
build, then find my test suits in where(Q3), then execute etc.
Many thanks in advance!
Since finally someone is also interested in this question, let me share my solution to this.
So, if you are looking at this question, I assume you already have your test suite and you could test it locally in your machine, either have your app installed in a simulator or a real device. Now you need to read more about gitlab pipeline and gitlab CI :
pipeline: https://docs.gitlab.com/ee/ci/pipelines.html
gitlab CI: https://docs.gitlab.com/ee/ci/quick_start/
And you should have noticed that, one of the advantages of Appium is that you don't need to change a thing about the App you are testing, you are testing exactly the same App which is going into production. To learn more about Apppium:
http://appium.io/docs/en/about-appium/intro/
Now, to run the automation test, you need your test suite, the app, and Appium server. What we need to do is adding another stage in .gitlab-ci.yml, tell it to
take the newly compiled App, compile your test suite
install the App in simulator/real device
compile your test suite and run it.
To make things easier to understand, we start with question 4, workflow:
So when the code is checked in to gitlab, the gitlab runner runs the jobs of each stage in your .gitlab-ci.yml, and when it runs to your stage, it does the automation test, and note that it is running on your server, so it means you need to have Appium installed on your server and have it up and running when try to run your automation test suite. Now the problem is that, is your server capable to do so? If you wanna do the automation test in your server, you need to install Appium on it, simulator probably(and which might need your server to equip with GPU), etc, these are the concerns of maintaining server. The alternative would be using the third-party service ,which is what I did. Turns out our(when I was in that company) server isn't capable of running automation UI test, so we turned to AWS-ADF(Amazon Device Farm), there are many other service providers you could choose, see the link for references:
https://adtmag.com/blogs/dev-watch/2017/05/device-clouds.aspx
So I basically have a python script in my functional test stage, and it will grab the newly complied App, the automation test suite, upload them to AWS ADF, and then schedule a run, yields result when the run is finished.
so, to answer question 1:
we need to create one more stage for our functional test in .gitlab.yaml, in my case, I have a stage functionalTest_project stage after the stage which compiles the Android App. And then you script the necessary cmd in your stage, or if its too lengthy, your script in another file(put it in your repo) and then execute it. In my case, I put my script in python_ci.py, and then I execute it in my stage use “python python_ci.py” .(here you need a docker with these requirement, see below too)
You don’t download Appium, you set up Appium on your or if you use a cloud service, that service should set up Appium for you.
What I did it is that I use maven built and package the test suite locally and then push it to gitlab repo, which now I believe the better way would be compile and package it in the your functionalTest stage in .gitlab.yml. now it comes back to first point of question 1, how to get maven, my understanding is that its a dependency of the server, like python, so they could both be obtained by telling gitlab to execute your script with a docker that has python and maven dependency.
answer to question 3:
put it in the same repo, but out of the Android project(i.e. they will under the same directory).
how to tell yml to reach the test suite? remember they are in the same server, so you could the relative path in your yml script to tell yml where to get your test suite.
Hope this helps!
I know this may be overkill for a single developer solution (personal project and not yet enterprise software), but I was wondering how to better respond to my needs.
I would be needing to accomplish the following:
Run integration tests (none UI for the moment) at least daily in order to see if any of my commits breaks the build.
Build the entire solution daily to see if any of my commits are incomplete and would cause problems when checked out on another folder.
Be run on my personal computer at least once a day (using another computer to automate the build process is not an option for the moment)
I know that automated build software such as Jenkins are easily capable of doing the previous (even on the same machine as committed?), but I was wondering if lighter solutions are available. Ex: Post-commit actions on the repository?, scripts?, planned tasks etc...
Edit
Forgot to mention that I was using a Windows machine with a c# project running nunit tests. I use visual studio 2012 to compile solution and run tests with nunit. I use tortoise svn and Ank svn as repository browser.
You might make a crontab(5) entry to periodically (e.g. daily) run your build or tests.
I have a crontab entry invoking some shell script to fetch the source tree by svn or git version control in a fresh place and build it daily.
You could consider using inotify(7) facilities, perhaps thru incron, to have a test run as soon as you modify some file (e.g. an executable).
Look also at D.Moreno's garlic project (which I never used).
You could also simply have some Makefile targets for tests, and run them from emacs. I have
(load-library "compile")
(global-set-key [f8] 'recompile)
in my ~/.emacs so I just compile things by pressing the F8 key in my emacs editor.
Use Jenkins - no reason not to, considering its reasonably lightweight itself (despite being a java app). Its very self-contained too, backup involves stopping the Jenkins service and copying the installation directory so it's not going to pollute your OS.
Anything else you come up with is going to be too complex (in terms of maintaining a bundle of scripts, scheduled tasks and so on) or just as 'heavyweight'. You might as well save your time and use the tool that fits from the start.
I'm no experience with Jenkins, I'm currently researching different options for PHP & JS automated unit testing with Jenkins.
I've come across Chutzpah (which uses PhantomJS's headless WebKit browser) but:
Is it possible to use Chutzpah with Jenkins?
There's very little documentation on Chutzpah. Although it does state on the Chutzpah homepage that it can be integrated into the TeamCity continuous integration server.
What's the minimum requirements for something to be compatible with Jenkins?
It is possible to use Chutzpah with Jenkins and with the 2.1 release of Chutzpah it is easier. Chutzpah's command line client can now take /junit argument that lets you specify a file name to output a junit-xml compatible file to. You can use Jenkins to pick this file up and report the test results.
I am not the downvoter, but I agree it is difficult to give a good answer to this question.
I believe the minimum requirement for something to be compatible with Jenkins is: It can be executed from a shell or cmd script. (If it's not, you need to find or write a plugin.)
Additionally, the thing should exit with code 0 for success and anything else for failure. (If it doesn't, you need to find or write a plugin.)
If you are interested in having Jenkins publish test results, the results must be in xml files using junit compatible notation. (If they are not, you need find or write a plugin.)
Additional requirements might be imposed by the tool you want to execute: It might need to draw windows or access the mouse or other parts of a graphical UI desktop/session. If that's the case, you need to run Jenkins in a context/session where it has access to those. (Windows, Mac and Linux all restrict background daemon/service access to the GUI desktop.)
Also, if your tool needs to access resources which are accessible by only certain user, you need to run Jenkins as that user.
This is a very open-ended question. Please try it out and come back with more concrete questions.
my team is currently using buildbot to automate overnight and continuous-integration builds and regression tests. For builds and unit tests, the builder just invokes a script which syncs the sources from p4 and then runs xcodebuild. The regression tests are also launched by a shell script, and are themselves combinations of shell scripts and AppleScripts. The builds are all performed on one system but then buildbot triggers tests to happen on multiple Macs, with different architecture and OS combinations. The things I like about buildbot are the automatic triggers (so the tests run only if and when the build succeeds), and the reporting including the waterfall view to see the overall status.
However, I see a number of problems too. The "master" process which coordinates the builds is either leaky or just has a huge working set which means that it consumes a couple of gigabytes of memory. Occasionally network problems mean that a slave will be lost; rather than retrying later it will just fail the build. In fact if the slave is supposed to be triggered for a dependent build, it will fail the first build after it has otherwise successfully completed.
So what are the rest of you using to automate your Xcode builds and unit tests? How do those solutions work for you? Anything you would recommend?
I use TeamCity in a windows environment but I believe it will work on Mac and has xcode build agents.
Another similar question here.
Our team use Hudson with a script which converts the Unit Test output from XCode to NUnit format.
I know this is an older question, but for those using TFS to hold their Xcode projects, I created a custom build activity to make automating Xcode builds via TFS easier. The code is hosted on Codeplex here: http://tfsxcodebuild.codeplex.com/.
Hope someone finds it useful!
We worked out what was using all the memory - log files - older buildbot keeps all logs forever and keeps them in memory (possibly until a restart).
Newer buildbot can be configured to keep a fixed amount of history.
Buildbot build log files should be limited in size to avoid the problem.
I use the XcodeBuilder that's part of CruiseControl. Of course it helps that I wrote it. :)
But I did use it on a real multiple person project for an iPhone app (Surf) that's for sale in the app store.
Now that it's three years later I have my own answer to provide to this question. I'm using Jenkins, mainly for the plug-ins that it provides. It has a plug-in for building targets in Xcode projects (or schemes in workspaces). There's a plug-in to run the Clang static analyzer. It interfaces with my bug-tracker system, it'll automatically push builds to Testflight too.