I have been learning Fitnesse+RestFixture and it appears to me that JMeter could do everything that can be done using Fitnesse (eps. perfoming tests using URIs through http verbs)? Could someone shed some light on this please?
FitNesse is an acceptance testing tool, where JMeter mainly is an performance testing tool. In acceptance testing, the recipient of the test and test result might not be a technical person. Therefore, how you define your tests and how the results are presented are important matters.
Snipped from Wikipedia on Acceptance testing
Acceptance Tests/Criteria (in Agile Software Development) are usually created by business customers and expressed in a business domain language. [...] These tests are created ideally through collaboration between business customers, business analysts, testers, and developers. [...] The business customers (product owners) are the primary project stakeholder of these tests. As the user stories pass their acceptance criteria, the business owners can be reassured the developers are progressing in the right direction.
A non technical person might have problem to interpret raw HTTP/XML/JSON/etc., whereas the business language can be defined together with testers and developers. But hey, let's see an acceptance test example for a fictive Lottery, using FitNesse with SLiM and with RestFixture, a fictive request builder fixture, and a fictive database fixture.
#Content of the actual test page, it could be named after the user story:
#IfNoActiveDrawsIsAvailableAnEmptyListShouldBeReturned
!|script|Database Fixture|
|given that no active draws exists|
!|script|Request Builder Fixture|
|Build a request for active draws|
!|script|Rest Script Fixture | ${SERVER_URL}|
|when the request is sent to the get active draw service|
|check| that the response is successful|
|check| that the response contains an draw list|
|check| that the response contains| 0| active draws on the lottery|
In this way, the tester and the developer has hidden all of the technical implementation and transformed it into an easy read test.
Related
Can anyone briefly explain what are the common asserts to consider in any automation project please. Whether it might be an in-house or public web application. For example presently i am using selenium (java) to automate an eCommerce web application. As this is my first website to automate, i am running out of ideas where i can verify things expect few which i know mentioned below:
1.Verify each page Title
2.Verify a button, text, link, image, custom text etc
Apart from these is there any thing else i can verify? please feel free to correct my question and if you have worked on various automation projects which areas did you add asserts to verify or validate something on a webpage.
basically, you do automation to decrease the execution time of regression cycles by automating the Test Cases relate to the functionality of the application. so, first develop test cases, using test design techniques like ECP, BVA etc.
Each test case must have an Assertion called expected result or functionality (otherwise it won't be called a Test case).
This assertion can be anything like,
Whether login successful after giving valid credentials
Showing an error message after entering wrong credentials etc.
Selenium helps us to automate web interactions (navigations, clicks, enter texts etc.) and don't perform any assertions for you.
Assertions are available by frameworks like JUnit, TestNG (in Java) with Assertions class. There is built-in support from programming languages like assert keyword in python & Java (http://docs.oracle.com/javase/7/docs/technotes/guides/language/assert.html)
So, whatever you mentioned in your question like common assertions (Verify each page Title etc.), those are just web interactions. they don't decide whether a Test is PASS or FAIL. It is you who define the criteria whether a Test is PASS/FAIL.
For example, there is a test case related to successful login.
here, you automate web interactions like navigate to login page, enter credentials, click Submit button.
Then to validate whether you successfully logged in or not, you look for a web element in the Home Page of the user logged in (like, welcome user) in normal scenario. In Automation, you try to find the text welcome user using webelement. Then you use Assertions provided by frameworks, to assert whether the expected message is present in the webpage like
Assertions.assertEqual(expected_message, actual_message); // just an example.
If expected_message and actual_message is same, then the method don't throw any exception, which results in marking the testcase as PASS by the framework
If expected_message and actual_message is NOT same, then AssertionError is raised by the method assertEqual, which results in marking the test case as FAIL by the framework.
I just completed writing a detailed rspec capybara integration and unit tests for Rails app, which includes mocking Omniauth (twitter) login, filling in forms, data validations, etc. However, I am wondering whether there is a need to write a separate controller or functional test.
Would appreciate your input and any links to further readings etc.
I'll play devil's advocate here, since I know I'm probably in the minority with this opinion: I actually prefer to do exceedingly thorough controller testing. A few reasons:
1) I find it easier to systematically test every path and outcome at the controller level than at the integration test level. My integration tests are primarily just happy-paths, and some of the more common error paths.
2) A lot of potential security issues occur at the controller level. Thorough testing helps me ensure that nothing malicious can get through to my model logic.
3) This is subjective, but it really forces me to think about some of the long-tail paths that my application might go through. What if someone tries to for an invalid password reset token into the URL? Controller testing ensures that I consider all options.
4) Unlike integration tests, they're fairly straight-forward to test. Each action is just a ruby method!
Personally, I think if your request (integration) spec is exercising all code paths you're covered. Ryan Bates has a great Railscast about how he tests here: http://railscasts.com/episodes/275-how-i-test?autoplay=true and about 5:05 in he says a similar thing. Like you I like to write integration tests rather than controller specs. Most of the time controllers simply front CRUD type operations anyway (especially if you're careful about keeping domain logic out of the controller), so all you're testing is the scaffolding.
We have a isolated test automation team responsible for automating only watir+cucumber functional test cases. Their code base is not attached with the rails app that other developers are working on, but kept separate. We have automated several test cases so far, and now what problem we have is, some (watir/cucumber specs)test cases require some data to be preexist into db, so it(testcase) should focus only on the problem stmt, and not creating any data-require itself.
Example, say if it has to check whether rating is working for a post, it requires a post object should preexist and it just checks rating. And not creating 1st post object and then checking its rating.
What are the best approaches here? Like we have fixtures and factory-girl for rails unit testing, what is there for cucumber specs? Or Shall we make use of features only here? These testers may not have idea of all models that exist, do they be aware of them so to make use of fixtures by calling Rails-Model interface.
My idea was, when we write feature file, it should not point or talk about any Model which looks meta stuff. Watir/specs test cases should only be aware of "Web-application"/browser only as the interface to talk/deal with the application. They should not know any other interface(fixture/Models). Hence they should create their own data on their own, by making use of the single interface they know.
Again, what I want to know that, is there any ruby lib/code, given table names, column names, and values(all most like fixtures yml), along with db parameters. It will simply insert them into db, without context of rails environment. And so testers those are having their environment isolated from rails web developers would able to work on their own. Rails fixtures, or factory girls seem to be well coupled with rails. Or am I incorrect?
Like Chirantan said you could use Factory girl with cucumber.
As require your factories in test unit or RSpec, you can do the same in the cucumber's env.rb file or any custom config file.
http://robots.thoughtbot.com/post/284805810/gimme-three-steps
http://www.claytonlz.com/2010/03/zero-to-tested-with-cucumber-and-factory-girl/
http://www.andhapp.com/blog/2009/11/07/using-factory_girl-with-cucumber/
When using cucumber, the Given statement sets the test situation up:
Given I have a basic user with a password
and the When statement triggers the test:
When the user logs in
and the Then statement checks the test results
Then they see the basic menu
The data gets loaded in the Given statement.
I feel like I'm stuck in this political/religious battle. From the research I've done I've found that many people believe very strongly that UI testing is silly to do via unit testing and is something much better done by an actual person. I agree.
However it seems the higher-ups have told me to unit test the UI. (Which is already strange since they had previously told me I only had to write a test plan and not actual test). In all sincerity I'm not very familiar with unit testing. I'm trying to use MOQ but all the tutorials I find for it use repositories and not services and I'm structuring my web application around services.
As an example here is (a part of) a service:
public class ProductService : Service<Product, EntitiesDbContext>
{
...
public Product GetProduct(int id)
{
return All().FirstOrDefault(p => p.ProductId == id);
}
...
}
My assumption is that I need to create let's say, a new Product using Moq, and then someone validate the hard-coded input using my model. I'm not sure how to do this.
Testing the UI is the bulk of my work as the website relies heavily on user input via forms. Again I find it quite silly to test with unit testing since my models are very strict and only allow certain inputs.
To my question:
How would I go about testing form input using MOQ or possibly another testing framework?
I've heard that there are programs out there that can track your actions and then replicate them, but I've been unsuccessful in finding them and I also believe that such a technology would cost money.
Try Selenium or Watin for UI testing.
Visual Studio 2010 Ultimate has something called Coded UI test. Basically, you record your actions in Internet Explorer, which translate to C# code where you add assertions, and then you replay the actions in different browsers.
However, it is very difficult to test against a database backed web application, since you can hardly figure out which is the newly added row. To simplify matters, you might want to set up a sample database which will be reset to its initial state before running your test suite in sequence.
Take a look at the various TechEd videos on how to use Coded UI Test. E.g. http://www.youtube.com/watch?v=rZ8Q5EJ2bQM
Are you testing the Actions for your UI or are you testing Actual UI.
If you are testing the Actions and what data they are sending to the view then doing unit tests using something like NUnit, xUnit, etc and using a moq framework like MOQ or Rhino Mocks would be the right thing to do.
if you are testing the Actual UI(the HTML returned by the web server) then using an automated testing framework like Selenium would be the right tool
You can also try Ivonna (http://ivonna.biz) for MVC testing -- doesn't test your client side, but lets you test your Asp.Net server side code.
But first of all you should ask yourself (or your management) a question: what do I want to test? Client/server side validation? Your Service? Action Method?
There is a lot of opposition by developers to automate testing of the UI. What I did years ago was use "Selenium Core" This allowed me to fix do the testing in Firefox, then save out as C# , and then have a blend of C# and Html files in which I could have it do the following.
Forgot Password -- Automation random usernames from database would be entered and then encrypted passwords were emailed to various gmail accounts I set up and I had C# API to login to gmail and retrieve new reset password, then redirect and have application enter new password , and I could put this on a CI Server Hudson and automate this testing all day long, and logging of success and errors etc...
New Registration... similar to #1
So for your UI testing, lets call that Functional or UAT testing ...
Then for true Unit Testing, this is for NUnit / Xunit, MSTest etc... , and then MOQ etc... can assist.
Integration testing is truly testing of the entire application with connected systems... like a file system or WCF or Database , not being "mocked"
They all have their pros and cons and none of it is perfect, but most people end up saying Unit Testing is most bang for the buck.
Test driven development on wikipedia says first develop a test that will fail because the feature does not exist. Then build the code to pass the test. What does this test look like?
How do you figure out what test will best represent the feature you want to create?
Can someone give an example?
Like if I make a logout button feature to a web application then would the test be hitting the page looking for the button? or what?
I heard test driven is nice for regression testing, I just don't know how to start integrating it with my work.
Well obviously there are areas that are more suited for TDD than others, and running frontend development is one of the areas that I find difficult to do TDD on. But you can.
You can use WATIN or WebAii to do that kind of test. You could then:
Write a test that checks if a button exists on the page ... fail it, then implement it, and pass
Write a test that clicks the button, and checks for something to change on the frontend, fail it, implement feature and pass the test.
But normally you would test the logic behind the actions that you do. You would test the logout functionality on your authenticationservice, that is called by your eventhandler in webforms, or the controller actions in MVC.
What does this test look like?
A test has 3 parts.
it sets up a context
it performs an action
it makes an assertion that the action did what it was supposed to do
How do you figure out what test will best represent the feature you want to create?
Tests are not based on features (unless you are talking about a high level framework like cucumber), they are based on "units" of code. Typically a unit is a function, and you will write multiple tests to assert all possible behaviors of that function are working correctly.
Can someone give an example?
It really varies based on the framework you use. Personally, my favorite is shoulda, which is an extension to the ruby Test::Unit framework
Here is a shoulda example from the readme. In the case of a BDD framework like this, contextual setup happens in its own block
class UserTest < Test::Unit::TestCase
context "A User instance" do
setup do
#user = User.find(:first)
end
should "return its full name" do
assert_equal 'John Doe', #user.full_name
end
context "with a profile" do
setup do
#user.profile = Profile.find(:first)
end
should "return true when sent #has_profile?" do
assert #user.has_profile?
end
end
end
end
Like if I make a logout button feature to a web application then would the test be hitting the page looking for the button? or what?
There are 3 main types of tests.
First you have unit tests (which is what people usually assume you are talking about when you talk about TDD testing). A unit test tests a single unit of work and nothing else. This means that if your method usually hits a database, you make sure that it doesn't actually hit that database for the duration of the test (using a technique called "mocking").
Next, you have integration tests. An integration test usually involves interaction with the infrastructure, and are more "full stack" testing. So from your top level API, if you have an insert method, you would go through the full insert, and then test the resulting data in the database. Because there is more setup in these sorts of tests, they shouldn't really be run from developer machines (it is better to automate these on your build server)
Finally, you have UI testing. This is the most unreliable, and requires a UI scripting framework like Selenium or Waitr to automate clicking around your UI. Don't go crazy with this sort of testing, because these tests are notoriously fragile (a small change can break them), and they wont catch whole classes of issues anyways (like styling).
the unit test would be calling the logout function and verifying that the expected results occurred (user login record ended, for example)
clicking the logout button would be more like an acceptance test - which is also a good thing to do, and (in my opinion) well within the scope of TDD, but it tests TWO features: the button, and the resulting action
It depends on what platform you are using as to how your tests would appear. TDD is much harder in ASP.NET WebForms than ASP.NET MVC because it's very difficult to mock up the HTTP environment in WebForms to get the expected state of Session, Application, ViewState etc. as opposed to ASP.NET MVC.
A typical test is built around Arrange Act Assert.
// Arrange
... setup needed elements for this atomic test
// Act
... set values and/or call methods
// Assert
... test a single expected outcome
It's very difficult to give deeper examples unless you let us know the platform you plan to code with. Please give us more information.
Say I want to make a function that will add one to a number (really simple example).
First off, write a test that says f(10) == 11, then do one that says f(10) != 10. Then write a function that passes those tests. If you realise the function needs more capabilities, add more tests.
The test would be making sure that when the logout function was executed, the user was successfully logged out. Generally a unit testing framework such as NUnit or MSTest (for .Net stuff) would be used.
Web applications are notoriously hard to unit test because of all the contextual information generally required for the execution of server code on a web server. However, a typical example would mock up that information and call the logout logic, and then verify that the correct result was returned. A loose example is an MVC type test using NUnit and Moq:
[Test]
public void LogoutActionShouldLogTheUserOut()
{
var mockController = new Mock<HomeController>() { CallBase = true };
var result = mockController.Object.Logout() as ViewResult;
Assert.That(result.ViewName == "LogoutSuccess",
"Logout function did not return logout view!");
}
This is a loose example because really it's just testing that the "LogoutSuccess" view was returned, and not that any logout logic was executed. In a real test I would mock an HttpContext and ensure the session was cleared or whatever, but I just copied this ;)
Unit tests would not be testing that a UI element was properly wired up to an event handler. If you wanted to ensure that the whole application was working from top to bottom, this would be called integration testing, and you would use something besides unit tests for this. Tools such as Selenium are commonly used for web integration tests, whereas macro recording programs are often used for desktop applications.