Related
I have taken as an example for learning and gathered some information about tools, objectives,scenarios, but I need your inputs. Please assist me.
I am new to Performance testing and would like to test the following website www.volkswagen.co.nz
Can you tell me, what are need to be tested? What are the scenarios and activities for each scenario? What metrics do I need to add? Which is the best and free tool for testing it? How to test if it is deployed in cloud like AWS?
Please let me know, Thanks in advance.
Performance testing needs,
Identify critical/heavy/important scenario in your webapp (irrespective of deployment cloud/standalone)
Identify service level agreements in terms of response times, throughput, latency etc.
Identify workload model i.e. how much user load application is expecting. this should be as fine grained as possible (avg users per transaction/workflow at a point of time)
Identify tools (JMeter is freeware and best but if you can afford paid then look at loadrunner, neoload etc.)
Record the script for workflows and parameterise and correlate.
generate test setup for load test and execute the load test.
monitor system utilization, collect metrics like response time, throughput, error rate, latency etc.
This all comes in load testing. For more you can read http://www.guru99.com/performance-testing.html
I am new to Performance testing and would like to test the following website www.volkswagen.co.nz
That is a recipe for disaster. No one new should be allowed to work on their own without a full period of training and internship with a master in the field. This is true of stone masons, electricians, plumbers, barbers, accountants, engineers and physicians. And it is most certainly true of performance testers/engineers.
There are dozens of foundation skills you need to master before you touch any tool, open source or otherwise. Until you show mastery of those items along with tool mechanics for your tool you should not be allowed to test any website, particularly a production website. And, if you don't work for this company what you are engaging in is a denial of service attack and could leave you with exposed legal liability.
I strongly agree with James on this one.
Do not touched the site if:
it's not yours
not sure what you are doing
the owner gave you explicit (and sounds like irresponsible) permission
don't know or don't have the support to restore the environment into a working state
If you do work for the company then you need to have a test environment first, a playground where you can mess around and nobody would mind if you take it down.
Firstly get information from the business on which use cases needs to
be tested.
Get response times target for user actions and for environments utilisation.
Get response time targets for environments utilisation: define environment monitoring tactics.
Found a tool that can fit for purpose: Jmeter, Gattling,etc, lot's of free ones available.
Get a test environment, preferably similar scale to production
Create scripts to cover critical use cases
Comply scripts into scenarios
Create a reporting framework
Kick off monitoring
Kick off scenario
Collect and analyse results
Be mindful of the free editions of load testing tools: they tend to be easy to use at first but soon as you start to outgrow it it can cost a fortune and more often then not it's hard to port scripts/scenarios to another tool.
End to end testing means exercising an application from the outer boundaries to verify its behavior. This far I've only done written tests for a single executable artifact. How should I test systems made up of multiple artifacts that is deployed on different hosts?
I see two alternatives.
The tests set up the whole system and exercise it from the very outer edges.
Each artifact is end to end tested in isolation, relying on the test content to enforce the protocol between them.
Is there a clear case for only adhering to one of these, or are one of them preferred, or are they interchangeable? If interchangeable, then what are some advantages and disadvantages between them?
Even though I think it depends on the context, I prefer the first alternative. Here are my random thoughts:
I like my tests to be as closely mapped to use cases as possible (BDD style) (with the disclaimer that I misuse the term use case). These use cases may span several applications and sub-systems.
Example: A back office administrator can view a transaction made by a user from the public interface.
Here, the back office admin interface and the public interface are different applications, but they are included in the same use case.
Mapping these thoughts to your problem where you have sub-systems deployed on different hosts, I would say it depends on how it is used, from the user/actor perspective. Do the use cases span several sub-systems?
Also, perhaps the fact that the system is deployed on several hosts isn't important to the tests. You could replace the inter-process communication with method calls in your tests and have the whole system within the same process during tests, reducing the complexity. Supplement this with tests that only verify the inter-process communication.
Edit:
I realise that I forgot to include why I prefer to test the whole system.
Your asset is features, that is, behaviour, while the code is a liability. Therefore you'd like to test the behaviour, not the code (BDD style).
If you are testing each sub-system separately you are testing the code, not the features. Why? When you divided your system into sub-systems you did so based on some technical reasons. When you learn more you might discover that the chosen seam is sub-optimal and would like to move some responsibility from one sub-system to another. And you would have to modify test and production code at the same time, leaving you without a safety net. That's a typical symptom of testing implementation details.
That said, these kind of tests are too blunt to test everything. So you need to have complementary tests for details as well, where necessary.
Testing each artifact end-to-end separately would be highly desiderable in any case. This will ensure that every artifact is sound.
In addition, you might want to test a composition of artifacts. That would catch problems in the interactions between artifacts. I don't know about your situation, but one thing that is important to have is a test environment that is a copy of production. Testing the system in the test environment is a very good idea. You might also want to test the system in the production environment; this might be feasible or not. For instance, if your system processes credit card payments, you may want to avoid test payments on the production system.
In any case, testing each system separately is imho more important than testing the composition. Once you know that your artifacts are sound in isolation, catching interaction tests will be much easier. If you only have the end-to-end test of the whole system, it's much more difficult to understand where is the error when the tests fail.
I've inherited a project which will be connecting to a CRM system via a SOAP service, written by another dev. My question is: to what level should I be testing the interface with the Soap services?
I set up a test case and wrote some methods to test a Soap update method, and confirmed it failed with a suitable error code for invalid customers or order numbers.
I next tested an invalid order status value (not within in a set of expected parameters) and the service returned a success code, which wasn't expected.
I believe I should report this to the developer, but should I now remove this test from my test suite? Or leave it showing as a fail?
If the soap service chooses not to validate its imput parameters I think it's a poor design, but it's not a fault in MY code, I just need to ensure I clean the input before passing values to the other system, and that validation should be covered under another set of tests anyway.
Should I even be talking to the SOAP service through the Unit Tests in the first place?
You should write at least one test per atomic functional requirement. Now, if you are writing using minimal interface guidelines, then you should have at most one interface per atomic functional requirement. But you can write more than one test for each requirement, since there may be a variety of invariants that can be tested.
In general, you should think of invariants and functional requirements when writing tests, not interface methods. Tests >= Atomic Functional Requirements.
Think of the service "contract", what are it's pre-conditions (i.e. legal inputs), it's post-conditions (i.e. legal outputs) and invariants (legal service state). If these are not clear by the developer, or there's a chance another developer is misusing the service, this should be reported and handled.
One exception though - these is all nice and good in theory, but if there are no other customers to the service (besides maybe the original developer) sometimes excess checking is obsolete. It is quite reasonable to assume that in such cases invalid inputs are checked and eliminated by the customer code.
If the library (or 3rd party code) has a good reputation (for example, Apache Commons, Guava...), I don't retest the API.
However, when I am not sure of the quality of the code, I tend to write a couple of tests verifying my assumptions about the library/API (but I don't retest all the library).
If those simple assumptions fail, it is a very bad sign for me. In this case, I tend to write more tests to check more aspects of the library. In your case, I would write more tests and signal errors to the developer.
These tests are not lost, because if a new version of the library is delivered, you can still use them to check for regressions.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 9 years ago.
Improve this question
I have a good grasp of unit testing, DI, mocks, and all the design principal goodness required to have as close to full code coverage as humanly possible (single responsibility principal, think 'how will i test this' as I code, etc...).
My most recent app, I did not code doing true TDD. I kept unit-testing in mind as I coded, and wrote my tests after writing the code, refactoring, etc.. I did TDD when it was 'easy' to do... however I did not have as good of a grasp as I do now... That was the first project I made full use of DI, mocking frameworks, etc, and the first which had full code coverage - and I learned a lot from it as I went along. I'm itching to get assigned to my next project so I can code it completely doing TDD from scratch.
I know this is a broad question, and I've already ordered TDD by example and XP Unleashed, but I'm hoping for a brief overview of how you all design / write a large application doing TDD.
Do you write the entire application, using nothing but stubbed out code? (e.g., write all the function signatures, interfaces, structures, and write the entire application but without writing any actual implementation)? I could picture it working on small-mid sized, but is this even possible on large applications?
If not, how the heck would you write your first unit test for the highest level function in your system? Lets say for example - on a web service where you have a function called DoSomethingComplicated(param1,...,param6) exposed to the world. Obviously, writing the test first for a simple function like AddNumbers() is trivial - but when the function is at the top of the call stack such as this?
Do you still do design up-front? Obviously you still want to do 'architecture' design - e.g., a flow chart showing IE talking to IIS which talks to a windows service via WCF which talks to the SQL Database... an ERD which shows all your SQL tables and their fields, etc... but what about class design? Interactions between the classes, etc? Do you design this up-front, or just keep writing stub code, refactoring the interactions as you go along, until the whole thing connects and looks like it will work?
Any advice is much appreciated
Do you do design up front?
Of course you do. You've got a big application in front of you. You've got to have some idea of the structure it will have before you start writing tests and code. You don't have to have it all worked out in detail, but you should have some basic idea of the layers, components, and interfaces. For example, if you are working on a web services system, you ought to know what the top level services are, and have a good first approximation of their signatures.
Do you write the entire application using nothing but stubbed out code?
No. You stub things out only if they are really difficult to control in a test. For example, I like to stub out the database, and the UI. I will also stub out third party interfaces. Sometimes I will stub out one of my own components if it vastly increases the test time, or it forces me to create test data that is too complicated. But most of the time I let my tests work on a pretty well integrated system.
I have to say I really dislike the style of testing that relies heavily on mocks and stubs. Don't get me wrong, I think mocks and stubs are very useful for decoupling from things that are hard to test. But I don't like writing things that are hard to test, and so I don't use a lot of mocks and stubs.
How do you write your first unit test for a high level function?
Most high level functions have degenerate behavior. For example, login is a pretty high level function and can be very complicated. But if you try to log in with no user name and no password, the response from the system is going to be pretty simple. Writing that tests will also be very simple. So you start with the degenerate cases. Once you have exhausted them, you move on to the next level of complexity. For example, what if a user tries to log in with a username but no password? Bit by bit you climb the ladder of complexity, never tackling the more complex aspects until the less complex aspects are all passing.
It is remarkable how well this strategy works. You might think that you'd just be climbing around the edges all the time and never getting to the meat; but that's not what happens. Instead you find yourself designing the internal structure of the code based on all the degenerate and exceptional cases. When you finally get around to the primary flow, you find that the structure of the code you are working on has a nice hole of just the right shape to plug the main flow in.
Please don't create your UI first.
UIs are misleading things. They make you focus on the wrong aspects of the system. Instead, imagine that your system must have many different UIs. Some will be web, some will be thick client, some will be pure text. Design your system to work properly irrespective of the UI. Get all the business rules working first, with all tests passing. Then plug the UI in later. I know this flies in the face of a lot of conventional wisdom, but I wouldn't do it any other way.
Please don't design the database first.
Databases are details. Save the details for later. Rather, design your system as though you had no idea what kind of database you were using, Keep any notion of schema, tables, rows, and columns out of the core of the system. Implement your business rules as though all the data were kept in memory all the time. Then add the database later, once you've gotten all the business rules working. Again, I know this flies in the face of some conventional wisdom, but coupling systems to databases too early is a source of a lot of badly warped designs.
Do I write the entire application, using nothing but stubbed out code?
No, not in the slightest sense - that sounds like a very wasteful approach. We must always keep in mind that the underlying reason for doing TDD is rapid feedback. An automated test suite can tell us if we broke anything much faster than a manual test can. If we wait wiring things together until the last moment, we don't get rapid feedback - while we may get rapid feedback from our unit tests, we wouldn't know if the application works as a whole. Unit tests are only one form of test we need to perform to verify the application.
A better approach is to start with the most important feature and work your way in from there, using an outside-in approach. This often means starting with some UI.
The way I do it is by creating the desired UI. Since we normally can't develop UI with TDD, I simply create the View with the technology of choice. No tests there, but I wire up the UI to some API (preferrably using declarative databinding), and that's when the testing begins.
In the beginning, I would then TDD my ViewModels/Presentation Models and corresponding Controllers, possibly hard-coding some responses to see that the UI works. As soon as I have something that doesn't explode when you run it, I check in the code (remember, many small incremental check-ins).
I subsequently work my way vertically down that feature and ensure that this particular piece of UI can go all the way to the data source (or whatever), ignoring all other features.
When the feature is done, I can start on the next feature. The way I picture this process is that I fill out the application by doing one vertical slice at a time until all features are done.
Kick-starting a greenfield app this way always takes extra long time for the first feature since this is where you have to wire up everything, so pick something simple (like the initial View of the app) to keep things as simple as possible. Once the first feature is done, the next ones become much easier because the foundations are now in place.
Do I still design up-front?
Not much, no. I normally have an overall design in mind before I start, and when I work in a team, we sketch this overall architecture on a whiteboard or a slide deck before we start.
This is more or less limited to
The number and names of layers (UI, Presentation Logic, Domain Model, Data Access, etc).
The technologies used (WPF, ASP.NET MVC, SQL Server, .NET 3.5 or whatnot)
How we structure production code and test code, and which test technologies we use
Quality requirements for the code (pair programming, static code analysis, coding standards, etc.)
The rest we figure out as we go, but we use many ad-hoc design sessions at the whiteboard as we go along.
+1 Good question
I truly don't know the answer, but I would start with building blocks of classes that I could test then build into the application, not with the top-level stuff. And yes I would have a rough up-front design of the interfaces, otherwise I think you would find those interfaces changing so often as you refactor that it would be a real hinderance.
TDD By Example won't help I don't think. IIRC it goes through a simple example. I am reading Roy Osherove's The Art of Unit Testing and while it seems to comprehensively cover tools and techniques like mocks and stubs, the example so far seem also pretty simple and I don't see that it tells you how to approach a large project.
Do you write the entire application, using nothing but stubbed out code?
To test our systems we mainly do unit, integration and remote services testing. In unit tests we stub out all long running, time consuming, and external services, i.e. database operations, web services connection or any connection to external services. This is to make sure that our tests are fast, independent and not relying on the response of any external service to provide us quick feedback. We have learnt this the hard way because we do have some tests that do database operations which makes it really slow that goes against the principle "Unit tests must be fast to run"
In integration tests, we test the database operations but still not the web services and external services because that can make the test brittle depending on their availability and we use autotest to run the tests in the background all the while we are coding.
However, to test any kind of remote services, we have tests that connect to the external services, do the operation on them and get the response. What matters to the test is their response and their end state if it is important for the test. The important thing here is, we keep these kind of tests in another directory called remote (that's a convention we created and follow) and these remote tests are only run by our CI (continuous integration) server when we merge any code to the master/trunk branch and push/commit it to the repo so that we know quickly if there has been any changes in those external services that can affect our application.
Do I still design up-front?
Yes but we don't do big design up front basically what uncle Bob (Robert C. Martin) said.
In addition, we get to the whiteboard before immersing ourself into coding and create some Class Collaboration Diagrams just to make it clear and sure that everyone in the team is on the same page and this also helps us to divide the work amongst the team members.
I routinely run into this problem, and I'm not sure how to get past this hurdle. I really want to start learning and applying Test-Driven-Development (or BDD, or whatever) but it seems like every application I do where I want to apply is it pretty much only standard database CRUD stuff, and I'm not sure how to go about applying it. The objects pretty much don't do anything apart from being persisted to a database; there is no complex logic that needs to be tested. There is a gateway that I'll eventually need to test for a 3rd-party service, but I want to get the core of the app done first.
Whenever I try to write tests, I only end up testing basic stuff that I probably shouldn't be testing in the first place (e.g. getters/setters) but it doesn't look like the objects have anything else. I guess I could test persistence but this never seems right to me because you aren't supposed to actually hit a database, but if you mock it out then you really aren't testing anything because you control the data that's spit back; like I've seen a lot of examples where there is a mock repository that simulates a database by looping and creating a list of known values, and the test verifies that the "repository" can pull back a certain value... I'm not seeing the point of a test like this because of course the "repository" is going to return that value; it's hard-coded in the class! Well, I see it from a pure TDD standpoint (i.e. you need to have a test saying that your repository needs a GetCustomerByName method or whatever before you can write the method itself), but that seems like following dogma for no reason other than its "the way" - the test doesn't seem to be doing anything useful apart from justifying a method.
Am I thinking of this the wrong way?
For example take a run of the mill contact management application. We have contacts, and let's say that we can send messages to contacts. We therefore have two entities: Contact and Message, each with common properties (e.g. First Name, Last Name, Email for Contact, and Subject and Body and Date for Message). If neither of these objects have any real behavior or need to perform any logic, then how do you apply TDD when designing an app like this? The only purpose of the app is basically to pull a list of contacts and display them on a page, display a form to send a message, and the like. I'm not seeing any sort of useful tests here - I could think of some tests but they would pretty much be tests for the sake of saying "See, I have tests!" instead of actually testing some kind of logic (While Ruby on Rails makes good use of it, I don't really consider testing validation to be a "useful" test because it should be something the framework takes care of for you)
"The only purpose of the app is basically to pull a list of contacts"
Okay. Test that. What does "pull" mean? That sounds like "logic".
" display them on a page"
Okay. Test that. Right ones displayed? Everything there?
" display a form to send a message,"
Okay. Test that. Right fields? Validations of inputs all work?
" and the like."
Okay. Test that. Do the queries work? Find the right data? Display the right data? Validate the inputs? Produce the right error messages for the invalid inputs?
I am working on a pure CRUD application right now
But I see lots of benefits of Unit test cases (note- I didn't say TDD)
I write code first and then the test cases- but never too apart- soon enough though
And I test the CRUD operations - persistence to the database as well.
When I am done with the persistence - and move on to the UI layer- I will have fair amount of confidence that my service\persistence layer is good- and I can then concentrate on the UI alone at that moment.
So IMHO- there is always benefit of TDD\Unit testing (whatever you call it depending on how extreme you feel about it)- even for CRUD application
You just need to find the right strategy for- your application
Just use common sense....and you will be fine.
I feel like we are confusing TDD with Unit Testing.
Unit Testing are specific tests which tests units of behaviors. These tests are often included in the integration build. S.Lott described some excellent candidates for just those types of tests.
TDD is for design. I find more often then not that my tests I write when using TDD will either be discarded or evolve into a Unit Test. Reason behind this is when I'm doing TDD I'm testing my design while I'm designing my application, class, method, domain, etc...
In response to your scenario I agree with what S.Lott implied is that what you are needing is a suite of Unit tests to test specific behaviors in your application.
TDDing a simple CRUD application is in my opinion kind of like practicing scales on a guitar- you may think that it's boring and tedious only to discover how much your playing improves. In development terms - you would be likely to write code that's less coupled - more testable. Additionally you're more likely to see things from the code consumer's perspective - you'll actually be using it. This can have a lot of interesting side effects like more intuitive API's, better segregation of concerns etc. Granted there are scaffold generators that can do basic CRUD for you and they do have a place especially for prototyping, however they are usually tied to a framework of sorts. Why not focus on the core domain first, deferring the Framework / UI / Database decisions until you have a better idea of the core functionality needed - TDD can help you do that as well.
In your example: Do you want messages to be a queue or a hierarchical tree etc?
Do you want them to be loaded in real time? What about sorting / searching? do you need to support JSON or just html? it's much easier to see these kinds of questions with BDD / TDD. If you're doing TDD you may be able to test your core logic without even using a framework (and waiting a minute for it to load / run)
Skip it. All will be just fine. I'm sure you have a deadline to meet. (/sarcasm)
Next month, we can go back and optimize the queries based on user feedback. And break things that we didn't know we weren't supposed to break.
If you think the project will last 2 weeks and then never be reopened, automated testing probably is a waste of time. Otherwise, if you have a vested interest in "owning" this code for a few months, and its active, build some tests. Use your judgement as to where the most risk is. Worse, if you plan on being with the company for a few years, and have other teammates who take turns whacking on various pieces of a system, and it may be your turn again a year from now, build some tests.
Don't over do it, but do "stick a few pins in it", so that if things start to "move around", you have some alarms to call attention to things.
Most of my testing has been JUnit or batch "diff" type tests, and a rudimentaryy screen scraper type tool I wrote a few years ago (scripting some regex + wget/curl type stuff). I hear Selenium is supposed to be a good tool for web app UI testing, but have not tried it. Anybody have available tools for local GUI apps???
Just an idea...
Take the requirements for the CRUD, use tools like watij or watir or AutoIt to create test cases. Start creating the UI to pass the test cases. Once you have the UI up and passing maybe just one test, start writing the logic layer for that test, and then the db layer.
For most users, the UI is the system. Remember to write test cases for each new layer that you are building. So instead of starting from the db to app to ui layer, start in the reverse direction.
At the end of the day, you would probably have a accumulated a powerful set of regression test set, to give you some confidence in doing refactoring safely.
this is just an idea...
I see what you are saying, but eventually your models will become sufficiently advanced that they will require (or be greatly augmented by) automated testing. If not, what you are essentially developing is a spreadsheet which somebody has already developed for you.
Since you mentioned Rails, I would say doing a standard create/read/update/delete test is a good idea for each property, especially because your test should note permissions (this is huge I think). This also ensures that your migrations work as you expected them to.
I am working on a CRUD application now. What I am doing at this point is writing unit tests on my Repository objects and test that the CRUD features are working as they should. I have found that this has inherently unit tested the actual database code as well. We have found quite a few bugs in the database code this way. So I would suggest you push ahead and keep going with unit tests. I know applying TDD on CRUD apps is not as glamorous as things you might read about in blogs or magazines, but it is serving its purpose and you will be that much better when you work on a more complex application.
These days you should not need much hand written code for a CRUD app apart from the UI, as there are a 101 frameworks that will generate the database and data access code.
So I would look at reducing the amount of hand written code, and automating the testing of the UI. Then I would use TDD of the odd bits of logic that need to be written by hand.