I've created a batch script which run tests automatically (have a bin folder including all tests and dll's)
The whole software builds without problems on the CI.
the Tests run locally without errors (running the batch from my machine).
But if I run the tests with the gitlab runner on the CI-Server some tests fail.
Where is the difference? I'm new to die GitLab CI and don't understand the difference right now.
I am working with windows 7 :)
I'm happy with every suggestion.
Wish you a nice weekend :)
I'm trying to run a set of cucumber selenium tests as part of a build in TFS2015, but it doesn't appear to be running them at all. I only get a timeout error. What I can conclude is that it is having difficulty starting up the browsers. I suspect that it is not running them properly from the script. Any idea how to fix?
We include Selenium tests in our feature test build by wrapping each Selenium test with MSTest and/or CodedUI attributes which works perfectly. You do have to make sure the build/test agent is configured to run in Interactive mode otherwise the test run won't have access to the desktop to automate the browser.
http://donovanbrown.com/post/2015/08/28/auto-start-build-agent-in-interactive-mode
We don't use Cucumber however so that may be the issue for you, not sure.
I have setup Jenkins with git and Xcode plugins so that I would be able to build an iOS project on CI machine. However - I don't seem to get any unit test output in the console. I see the compilation messages (for implementation files as well as xibs), I see the build of test bundle, but no "test passed" or "test failed" messages. Running unit tests from Xcode produces all the needed output. What am I missing?
Your problem might be caused by several things. To approach the solution there are some things you need to know:
How do you run Jenkins? Did you use the Jenkins Mac installer from http://jenkins-ci.org?
Is Jenkins running under your user account? Or is it running as someone else? Is it running as a launch daemon?
Do your tests require access to WindowServer or are they completely happy only the commandline?
What is going on in your machine?
When Jenkins runs the build and the unit tests do not run, open Activity Monitor, show the process tree, check the sub processes of Xcode and if you find a process that looks like your unit test and then sample it (either with the Sample button in Activity Monitor or the "sample" command line utility). What is the process doing based on the samples?
If your unit tests require access to WindowServer and you are running Jenkins as a launch daemon, you might be interested in https://github.com/stisti/jenkins-app
Normally output like "Test x passed" or "Test x Failed" is due to the testing suite outputting to the console. When I call MSTest, I am calling via the cmd and the output is relayed back through jenkins. It sounds like you need something to read the output xcode is making and display it back to you. Sounds like you may not be able to do this in real time unless there is some sort of output goes to console flag you are not setting.
I am using the Gallio MSBuildTasks library to run MbUnit tests inside JetBrains TeamCity. When tests pass they report as having passed once, however when they fail they appear as having failed twice. Once with the Namespace and once without. This looks something like:
http://imageshack.us/photo/my-images/191/capturegv.png/
There are two tests here (Test and Test2) and they both fail. However TeamCity reckons they have each failed twice, displaying each one with a namespace (UnitTests) and then without. This is not just the report in the TeamCity website interface, I have pre-test commit enabled and they show up as failing twice in there too.
Has anyone else experienced this and if so how have you solved it?
I've just installed Hudson and it is running beautifully. It builds, runs JUnit-tests and also CheckStyle analysis.
Next step for us would be to create an installation, install it and then run automated tests on the actual installation. I would then like to fail the build if the tests fail or at least publish the results somehow. I think we would set it up so that part runs periodically or manually triggered.
We use InstallAnywhere for installation and IBM Rational Functional Tester for automated tests.
So questions are: anyone created a similar setup? are there any plugins, tutorials or other resource that could help me along. Or do you have any tips or advice in general.
The command line reference for Rational Functional Tester:
http://publib.boulder.ibm.com/infocenter/rfthelp/v8r0m0/index.jsp?topic=/com.ibm.rational.test.ft.doc/topics/RobotJCommandLine.html
Sample command for running a test:
java -classpath "C:\IBM\RFT\FunctionalTester\bin\rational_ft.jar"
com.rational.test.ft.rational_ft -datastore \\My_project\AUser\RobotJProjects -user admin -project
\\My_project\AUser\TestManagerProjects\Test.rsp -build "Build 1" -logfolder "Default" -log
"Al_SimpleClassicsA#1" -rt.log_format "TestManager" -rt.bring_up_logviewer true -playback
basetests.SimpleClassicsA_01
An additional note, you'll want to configure windows properly on your agent machine which will be running the tests. This is not advice specific to Hudson or RFT, but rather all GUI automation tools on Windows. RFT will require an interactive desktop environment for it to be able to click buttons, etc. If you have your Hudson agent running as a Windows service, there will be no desktop. See the following: Silverlight tests not working unless RDP connection open
We have run a fairly complicated distributed build on Hudson, it is a process that basically follows:
Test on Windows.
Test on OSX, run code coverage & push results to server.
Test on OSX Tiger.
Package for OSX Leopard & push build to server.
Package for Windows & push build to server.
Update product website.
We don't use InstallAnywhere or Rational Functional Tester, but have similar sorts of mechanisms in their place. The key we found to making it all sing in Hudson was being able run our various steps from the command line. Maven and appropriate plugins made short work of this task. So my advice would be just that, using whatever build tool you are using (ant, maven, ?) configure them so that you can run your rational functional tester and install anywhere from the command line with a simple goal passed to your build tool (i.e. mvn test or mvn assembly:assembly).
After that, make sure whatever machine Hudson is running on has everything installed (i.e. Rational Functional Tester) and configured, so that you can open up the command line and type in the goal and have your tests correctly execute.
Hooking it up in Hudson from that point on is fairly simple - just pass in the goal when you configure the build.
I believe the best answer is that integrating RFT with Hudson/Jenkins is a useless endeavor.
As this IBM FAQ says, to make RFT work you must:
be logged in the machine;
the screen can't be locked;
if you are remotely connected, you can't minimize the connection screen.
So you can't run Jenkins/Hudson as a service, making it not very useful. You must run it from your logged account. If you are in a corporate computer (very probable if you are using RFT), you probably must use a hack to prevent the screen saver to start. If the screen is locked, your tests will always fails.
It isn't very difficult to configure your tests to run from the command line, you just have to take care of the return codes when the tests fail and succeed.
Jenkins/Hudson would also give you some advantages, like integrating the tests with your version control, probably automatically running the tests when a commit is made. It would also help sending emails when the tests fail.
But you still would have to integrate the RFT logs with some kind of JUnit plugin to have a nice report. You also would have to have script to run the tests using the command line.
I think it is not worth the trouble to use an continuous integration server with RFT. Better just have your tests running every day in Windows Task Scheduler. It is a simpler solution with less failure points.
Or use my final solution: quit RFT and use the free Selenium with a headless web driver.
I have some general advice on this because I have not yet implemented this myself.
I am assuming you want to have Hudson run the RFT scripts automatically for you via a build or Hudson process?
I want to implement something similar in my organisation as well.
I have not yet been able to implement this because of organisational constraints but here is what I have thought out/done so far:
Downloaded Windows process viewer, got the command for running the tests.
Made shell Script out of it, separated out the variables etc
The future plan is to setup a Windows Slave machine which would have all the tools in it that would be required once the Tests are kicked off, for eg. the correct versions of browsers, and environment variables, and other tools that are required.
Hudson would kick off a process which runs the shell scripts created which runs all the RFT Scripts and performs necessary operations on the slave machine.