Is there a facility similar to SeleniumGrid that I can use to run webrat (or other, similar framework) browser automation tests in parallel across a farm of coordinated agents?
Coordinated via TeamCity with rake?
Edit: We're looking at using cucumber+webrat to do functional and acceptance testing as described in Testing ASP.NET Web Applications
I've worked on just this project actually. If you're working on rails, check out http://github.com/sgrove/spec_storm . It's only setup to run rspec + selenium tests in parallel, but it can be extended to others depending on the demand. And of course if you have any questions, I'm more than happy to help out. The more people using it, the happier I am :D
Related
We are looking at cucumber for our automation test framework because everyone including business people can understand it.
We use Angualr JS frontend and Java REST backend. Our team that is going to write the step definitions likes Ruby so we want to stick with Ruby for that.
Also we would like to use Maven to tie this process into our build process.
Will cucumber be a good fit given that story above ?
Hui Peztherez, from my prospective cucumber is a great choice, using it with the same architecture expect for Angular.
We are using Maven too, and it's so useful to orchestrate them with Jenkins, using maven to run the tags..
mvn test -Dcucumber.options="--tags #smoke"
ref: https://cucumber.io/docs/reference/jvm
Also Jenkins have several plugin to report the Cucumber Analysis, so useful for testers, and in the end, we are now working about the HPQ server integration with a plugin called Bumblebee (this part is still under development for both sides, our and bumblebee)
Another good choice is Ruby, you can take the step definition so easily defined with Ruby...
We also have a integration with Selenium for the front end side, and it works as well...
So go further!
We are using Cucumber in Java with gradle in past, It was in Maven and It works fine. We have framework for UI and API, In UI we used WebDriver to write step definition and In API, We used RestAssured to write step definition. You can do same thing in Java what you can do in Ruby.
Maven for Java Cucumber :
http://mvnrepository.com/artifact/info.cukes/cucumber-java/1.2.4 - Please add other dependency as per requirement.
Jenkin Plugin : https://wiki.jenkins-ci.org/display/JENKINS/Cucumber+Reports+Plugin
Will cucumber be a good fit given that story above ?
- Yes It is good fit. I will request you to show POC(Proof of concept) to management. I had experience in past that management have no clue about BDD and they have very hard to time to understand coverage. We did very deep dive to provide all information to them. It is very important to answer following question to management
BDD report is providing accurate test converage idea to management ?
Everyone in team is able to write feature file and able to provide same quality of feature file
Feature file and BDD report will be starting tool for check test converage
Thank you.
Please be aware that Cucumber is a BDD framework that can be used on top of a browser automation framework like Selenium WebDriver/Watir/Protractor they are two distinct things. Most of them implements Selenium WebDriver's protocol.
My only concern is for you using Maven in that project setup, I know that you can run ruby code in a JVM by using JRuby. But I'm not sure which plugin you'd use to trigger that from Maven.
I am using tools like Puppet/Chef/Ansible to set up and config development environments and production servers.
Whenever I update the configuration, I run the tool against my development environment and log in to check manually if things works as expected.
But this is tedious to do, and I can't test everything every time, so is there any way I can automate the testing?
There are Infrastructure Testing Frameworks for this:
ServerSpec / InSpec - ruby-based. Famous, big community, nice looking and best in his class.
BATS - Bash Automated Testing System, which is a bit easier.
TestInfra - Python-based infra testing framework. Still pretty young, very small community. Intro.
Goss - Fast (written in Go), small tool for validating server/infra configuration. Test scenarios are written in yaml.
Automation:
There is interesting Molecule project - some automation for testing Ansible roles, designed by Cisco. Never tried it yet.
Step further would be using TestKitchen which handles automation to spin up Vagrant or Docker or even AWS instance and test Puppet/Chef/Ansible with Rspec/BATS against just spinned up machines.
So what you need - pick up framework, write tests and run your playbooks/recipes & tests against mock VMs.
Ideally is to keep your "infra as code" in vcs and configure ci like TravisCI to run your tests for every PR once you bring new changes in your repository.
You can even follow tdd here: write tests first, make them fail, then write actual implementation in your favorite configuration management tool and see if that change makes tests green/passed.
MOAR Infrastructure Testing & Automation!
If you can let us know, what you want to test. We can help better.
But,
Did you check the dry-run mode? I think, Puppet and Ansible supports it, you can have a cron or some automated script which runs all the puppet/ansible modules against a single(test) node.
More info:
1. http://docs.ansible.com/ansible/playbooks_checkmode.html
2. Check the noop mode in https://docs.puppet.com/puppet/latest/reference/man/agent.html
We have an automation framework that uses Cucumber + Appium to test an iOS application.
Appium is fine, but one of its limitations is that we need to run every test from the beginning, which slows down development. XCTest seems to offer more breakpoint type flexibility, i.e. we can just re-run a small part of the test, as we already do with browser-based applications in Selenium.
XCTest seems easy to use and powerful, and we would like to start using it as soon as possible. Has anyone managed to integrate Cucumber with XCTest? We use Ruby for our framework and would like to keep using that also if possible.
Thanks in advance!
I have recently integrated Quick(a Swift BDD framework) with the new XCTest and it all seems to be coming along fine.
As far as I know Cucumber don't have a Swift implementation yet so it is not possible to integrate Cucumber with XCTest.
I have a system thats highly reliant on various web APIs. I would like to run my API specific tests at least once per day to make sure all API's are still playing nicely and alive. I have a set of unit tests (just plain rb files that test API calls for expected data) and would like to run these every 24 hours. If something breaks, I would like to take an action (e.g. email or sms me).
How best to setup automated Ruby tests and parse the result? Can I just setup a cron job to handle the .rb files? How would I take an action and detect programmatically if the tests are failing? Maybe there is some kind of continuous integration solution for RUby that can handle this?
I've just gone through the process of setting up Hudson CI as my integration server, using this amazing tutorial from Dr. Nic. It installs through a gem, coming pretty much preconfigured, and was extremely simple to get working.
I'm using rspec and cucumber, and Hudson runs all tests when it sees a new commit on my git repository. If all tests pass, it merges the code into my master branch. If any test fails, it holds its horses and sends me an email.
EDIT:
I also want to give ten thumbs up to the ChuckNorris plugin for Hudson. Agile doesn't get better than pair programming with Walker, Texas Ranger.
Ruby has Test::Unit built-in, RSpec, ZenTest, shoulda, cucumber and probably many more tools to help test. Being built-in, Test::Unit is used a lot and is the target to be beaten by the other tools.
ZenTest and RSpec can do continuous testing: You make a change and save a file and they'll see it and run the test suite. I like that because then I know the state of things right away.
I haven't used cucumber, but have used the rest. I heard cucumber's emphasis is on integration testing but that might have been the commenter's feelings rather than by design of the developers. The list of tutorials for cucumber is interesting browsing. In particular there's webrat: Automated Acceptance Testing with RSpec or Cucumber.
Any of these could be wired up with cron to run periodically; Just treat them as you would any other set of command-line apps.
It should be easy to tie in web-testing too, but you'll have to identify the gems/modules needed and write the glue code. I haven't had need for such a beast as I'd go at it using Mechanize and/or one of the other HTTP gems plus Nokogiri to ransack the pages.
I'm interested in using the Lightweight Testing Automation Framework (LTAF) to create integration tests for my web application. However, I need it to be run on the build server. Does anyone know if this can set up to do this?
There does not seem to be a whole lot of information on the web on this right now :-)
There's a blog post (Lightweight Test Automation Framework – Automated Build Support) that describes how to implement the runner as a console application, it can then be integrated into a build server fairly simply by redirecting the build output and setting the return code appropriately.
The author posted the code used in the article, you can download it from here.
I don't have experience with LTAF, but found this nice article:
First steps with Lightweight Test Automation Framework
Quoting the author (Steve Sanderson, from the comments):
Lightweight Test Automation Framework
can be invoked with query string
parameters (to specify which tests to
run) and can emit a log of the results
to a text file, so it would be
possible to integrate it. However,
this is certainly not as easy as using
Selenium RC which as you say works
through a traditional test runner.