Since application tests can now be run on the simulator from Xcode, what would the advantage be, apart from possibly a small saving in execution time, of still separating your tests into logic and application tests?
The differentiation as per the Apple docs:
Logic tests. These tests check the correct functionality of your code in a clean-room environment; that is, your code is not run inside an application. Logic tests let you put together very specific test cases to exercise your code at a very granular level (a single method in class) or as part of a workflow (several methods in one or more classes). You can use logic tests to perform stress-testing of your code to ensure that it behaves correctly in extreme situations that are unlikely in a running application. These tests help you produce robust code that works correctly when used in ways that you did not anticipate. Logic tests are iOS Simulator SDK–based; however, the application is not run in iOS Simulator: The code being tested is run during the corresponding target’s build phase.
Application tests. These tests check the functionality of your code in a running application. You can use application tests to ensure that the connections of your user-interface controls (outlets and actions) remain in place, and that your controls and controller objects work correctly with your object model as you work on your application. Because application tests run only on a device, you can also use these tests to perform hardware testing, such as getting the location of the device.
Application tests compared to logic tests are really used for two different things:
Logic tests/unit tests are used to test very small behavior for one or a few methods, e.g. "Given that I create my object like this, is the value of a certain property what I expect it to be?"
Application tests however are used to test the big picture, e.g. "Do I get the right data in my detail view when I tap on a certain table view cell?"
Related
The problem which I am encountering is related to E2E tests which will run all the time for new app builds (maybe even every few hours on CircleCi). I have ( and will have much more in the future ) features that contain a lot of setups ( necessary the same setup for each scenario to run). For example, before the scenario will run ( many in the feature ) need some users, contents, configuration etc. After the scenario runs probably the best practice is to delete/remove all that users, content etc (or at least after all the scenarios had run for the feature ). I am struggling to understand what is the best practice.
If I add a background then it will run before each scenario, but then I head to remove all that data from the background ( I could add a cleanup function in the last scenario step but that seems bad, correctly if I am wrong). I could add hooks that will clean up after each scenario and keep adding more hooks for new features ( maybe use tags for the scenarios to distinguish for which they should run ).
There are options but it does feel so inefficient... Those tests will be running in a live environment ( not integration or unit tests which are fast, but E2E ). Very often the setup/background will take much more time than one scenario to run and then it will run over and over for each tinny scenario. For example, had to run in e.g. background bunch of endpoints to create users, some content and in many cases ( when we don't have an endpoint for it yet ) I will have to write an automated journey through the UI to add something or change specific settings and then same way add the end delete everything and also through UI change the setting to the state before the feature had run. It feels so slow and inefficient...
The only other thing which comes to my mind ( but will not probably work for all the cases ). Is to create a huge hooks script where I will be adding all the necessary "stuff" before the whole suite run and after the whole thing run I clean the whole stack/instance DB ( or reset to some preset DB snapshot ) to make it state as before the whole suite run.
Please help me to understand what are the best practices in such a cases
Regards
Generally with Cuking the idea is that the database is reset after every scenario. This is done by things like:
running the scenario in a transaction (which is then rolled back)
emptying the database after every scenario
Which you do depends on which flavour of cuke you are using.
The inefficiencies you talk about can be mitigated in a number of ways without compromising the idea that the database should be reset after every scenario. Basically you can think of Cukes as setting up state (Givens) doing something (When) and validating (Thens). Only Whens have to use the UI.
So with Givens you can set up state by either
writing directly to the database using factories or fixtures
calling services (the same ones your UI controllers use) to create things
The second one is much preferred.
With most of the work being done outside the UI this means that you can get rapid cukes that do complex setup quickly.
This really is the way to go when cuking. Setup everything in Givens (using background when appropriate) without using the UI, then login, then do your When using the UI, and validate the result in the UI with your Thens.
Using this approach my current project has approx 450 scenarios that run in about 5 mins on my Mac mini, and that includes
several scenarios that step through UI wizards using a browser (super slow)
many scenarios with complex setup of multiple entities
This idea what you have to work around standard practices to make your suite efficient and fast is common and almost always wrong.
You can actually go way faster than I am going (though it takes quite a bit of work)
I have a number of feature files in my cucumber scenario test suite.
I run the tests by launching Cucumber using the CLI.
These are the steps which occur when the test process is running:
We create a static instance of a class which manages the lifecycle of testcontainers for my cucumber tests.
This currently involves three containers: (i) Postgres DB (with our schema applied), (ii) Axon Server (event store), (iii) a separate application container.
We use spring's new #DynamicPropertySource to set the values of our data source, event store, etc. so that the cucumber process can connect to the containers.
#Before each scenario we perform some clean up on the testcontainers.
This is so that each scenario has a clean slate.
It involves truncating data in tables (postgres container), resetting all events in our event store (Axon Server container), and some other work for our application (resetting relevant tracking event processors), etc.
Although the tests pass fine, the problem is by default it takes far too long for the test suite to run. So I am looking for a way to increase parallelism to speed it up.
Adding the arguments --threads <n> will not work because the static containers will be in contention (and I have tried this and as expected it fails).
The way I see it there is are different options for parallelism which would work:
Each scenario launches its own spring application context (essentially forking a JVM), gets its own containers deployed and runs tests that way.
Each feature file launches its own spring application context (essetially forking a JVM), gets its own containers deployed and runs each scenario serially (as it would normally).
I think in an ideal world we would go for 1 (see *). But this would require a machine with a lot of memory and CPUs (which I do not have access to). And so option 2 would probably make the most sense for me.
My questions are:
is it possible to configure cucumber to fork JVMs which run assigned feature files (which matches option 2 above?)
what is the best way to parallelise this situation (with testcontainers)?
* Having each scenario deployed and tested independently agrees with the cucumber docs which state: "Each scenario should be independent; you should be able to run them in any order or in parallel without one scenario interfering with another. Each scenario should test exactly one thing so that when it fails, it fails for a clear reason. This means you wouldn’t reuse one scenario inside another scenario."
This isn't really a question for stack overflow. There isn't a single correct answer - mostly it depends. You may want to try https://softwareengineering.stackexchange.com/ in the future.
No. This is not possible. Cucumber does not support forking the JVM. Surefire however does support forking and you may be able to utilize this by creating a runner for each feature file.
However I would reconsider the testing strategy and possibly the application design too.
To execute tests in parallel your system has to support parallel invocations. So I would not consider resetting your database and event store for each test a good practice.
Instead consider writing your tests in such a way that each test uses its own isolated set of resources. So for example if you are testing users, you create randomized users for each test. If these users are part of an organization, you create a random organization, ect.
This isn't always possible. Some applications are designed with implicit singleton resources in the code. In this case you'll have to refactor the application to make these resources explicit.
Alternatively consider pushing your Cucumber tests down the stack. You can test business logic at any abstraction level. It doesn't have to be an integration test. Then you can use JUnit with Surefire instead and use Surefire to create multiple forks.
I plan to reuse existing Specflow scenarios (Currently is using for acceptance and automated test) for VS Load Test as well, to avoid duplication and extra work. Specflow works fine for those test since it runs them once but in context of Load test when it executes each Specflow scenario more than one time and parallel it runs into issues and errors and with higher number of user it gets more
These errors can fail some of the test which at end creates incorrect test result, for instance using one Specflow scenario as test scenario with load test of 20 users and time period of 2 minutes can caused 50 errors similar to below. So test result shows that particular scenario is executed 200 times where 150 passed and 50 failed test and failure is caused by Specflow errors. In context of Load test this result is totally wrong and incorrect since the test itself has issues.
Error message:
ScenarioTearDown threw exception. System.NullReferenceException: System.NullReferenceException: Object reference not set to an instance of an object.
TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.HandleBlockSwitch(ScenarioBlock block)
TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.ExecuteStep(StepInstance stepInstance) TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.Step(StepDefinitionKeyword stepDefinitionKeyword, String keyword, String text, String multilineTextArg, Table tableArg)
TechTalk.SpecFlow.TestRunner.Then(String text, String multilineTextArg, Table tableArg, String keyword)
After some investigation it seems Specflow cannot generate and run same scenario parallel which caused this conflict and fails some test but I also have some doubt about that and seeking to see if there is any workaround about this or if I am missing anything and wondering if Specflow scenarios can be used for Load Test at all?
I understand wanting to reuse your tests for load testing (Don't Repeat Yourself), however a load test has a very different purpose than acceptance tests. Load tests should take realistic every day usage scenarios, and throw increasing numbers of users at them. For this reason I would urge you to keep your load tests separate from your acceptance and automated tests. They really are testing different things.
Load tests should test the performance of the application under high usage for every day scenarios, and acceptance and automated tests make sure the application is functioning according to spec.
Load testing is the process of putting demand on a system or device and measuring its response. Load testing is performed to determine a system’s behavior under both normal and anticipated peak load conditions. It helps to identify the maximum operating capacity of an application as well as any bottlenecks and determine which element is causing degradation.
Source: Wikipedia: Load testing
An acceptance test is a formal description of the behaviour of a software product, generally expressed as an example or a usage scenario. ... For many Agile teams acceptance tests are the main form of functional specification; sometimes the only formal expression of business requirements. In other cases, they merely complement a specification document resulting from a less specifically Agile technique or formalism, such as uses cases or more narrative documents.
Source: Agile Aliance: Acceptance Testing
They are different things, so the tests, and the test frameworks, should be different as well. You aren't really "repeating yourself" by keeping a separate suite for load tests.
As for the technical reason why this is failing? SpecFlow, when run using the normal Visual Studio test runner, was not built to run tests in parallel. There are parallel test runners available, but most are paid software.
I have a Visual Studio 2010 Load test, which contains a number of web performance tests. Running the web performance tests requires you to be logged in to the website under test. Accordingly, the load test contains an initialization step - a small web performance test which does the log in, and which uses a plug-in to cache the cookie so obtained. The 'real' web performance tests - the ones that actually do the work also each have a plug-in that reads the cached cookie and adds it to the test, so that each test functions correctly:
public override void PreWebTest(object sender, PreWebTestEventArgs e)
{
if (CookieCache.Cookies != null) // CookieCache is a static class of mine
e.WebTest.Context.CookieContainer.Add(CookieCache.Cookies);
The problem is that while this all works absolutely fine when I run the load test, it means I can't run any of the web performance tests in isolation because if the load test initializer hasn't run then there's no cookie, so the web performance test won't be logged in and will fail.
Is there any recommended solution for this situation? In other words, if a web performance test needs to have logged in, is there any way to get it to run both in isolation and when it's part of a load test?
The obvious way to run each web performance test in isolation would be to have it call the login test first, but I can't do that because that'll be incorrect behaviour for the load test (where logging in should happen only once per user, right at the beginning of the load test).
The solution is to add the Login test to your individual web performance tests (via "Insert Call to Web Test"), but gated by a "Context Parameter Exists" Conditional Rule that looks for the absence of the context parameter $LoadTestUserContext. That parameter only exists if the web test is running in a load test.
This way you get just one Login whether in or outside of a load test.
Why not try and use the PreRequest Function instead of the PreWebTestFunction
Public Overrides Sub PreRequest(sender As Object, e As PreRequestEventArgs)
MyBase.PreRequest(sender, e)
Dim cookie As System.Net.Cookie = New System.Net.Cookie(...)
e.Request.Cookies.Add(cookie)
That way both the Load test and the Web Test will work.
I'm not familiar with Visual Studio 2010 Load Testing, but it sounds like you need the equivalent of NUnit's SetUp and TearDown methods which run once for all tests, whether you have selected a single test or all the tests in an assembly.
A bit of searching implies that the equivalent is the Init and Term tests.
1) Right click on a scenario node in load test and select Edit Test
Mix...
2) In the edit test mix dialog, Look at the bottom the Form. You will
see 2 check boxes. One for an init test and one for a term test.
The init test will run prior to each user and term test will run when
user completes. To make sure the term test runs, you also need to set
the cooldown time for a load test. The cooldown time is a property on
the run setting node. This setting gives tests a chance to bleed out
when duration completes. You can set this to 5 minutes. The cooldown
period does not necessarily run for 5 minutes. It will end when all
term tests have completed. If that takes 20 seconds, then that is
when load test will complete.
I'm currently in an environment where we are parsing data off of the client's website. I want to use my tests to ensure that when the client changes their site, I know when we are no longer receiving the information.
My first approach was to do pure integration tests where my tests hit the client's site and assert that the data was found. However half way through and 500 tests in, the test run has become unbearable and in some cases started timing out. So I cleared out as many tests that I could without loosing the core protection they are providing and I'm down to 350 or so. I'm left with a fear to add more tests to only break all the tests. I also find myself not running the 5+ minute duration (some clients will be longer as this is based on speed of communication with their site) when I make changes anymore. I consider this a complete failure.
I've been putting a lot of thought into this and asking around the office, my thoughts for my next attempt at this is to pull down the client's pages and write tests against these embedded resources in my projects. This will give me my higher test coverage and allow me to go back to testing in isolation. However I would need to be notified when they make changes and then re-pull down the pages to test against. I don't think the clients will adhere to this.
A suggestion was made to me to augment this with a suite of 'random' integration tests that serve the same function as my failed tests (hit the clients site) but in a lot less number than before. I really don't like the idea of random testing, where the possibility of sometimes getting red lights and some times getting green lights with the same code. But this so far sounds like the best idea I've heard to still gain the awareness of when the client's site has changed and my code no longer finds the data.
Has anyone found themselves testing an environment like this? Any suggestions from the testing community for me?
When you say the big test has become unbearable, it suggests that you are running this test suite manually. You shouldn't have to. It should just be running constantly in the background, at whatever speed it takes to complete the suite - and then start over again (perhaps after a delay if there are associated costs). Only when something goes wrong should you get an alert.
If there is something about your tests that causes them to get slower as their number grows - find it and fix it. Tests should be independent of one another, so simply having more of them shouldn't cause individual tests to time out.
My recommendation would be to try to isolate as much as possible the part of code that deals with the uncertainty. This part should be an API that works as a service used by all the other code. This way you would be protecting most of your code against changes.
The stable parts of the code should be unit-tested. With that part being independent from the connection to client's site running the tests should be way quicker and it would also make those tests more reliable.
The part that has to deal with the changes on the client's websites can be reduced. This way you are not solving the problem but at least you're minimising it and centralising it in only one module of your code.
Suggesting to the clients to expose the data as a web service would be the best for you. But I guess that doesn't depend on you :P.
You should look at dividing your tests up, maybe into separate assemblies that can be run independently. I typically have a unit tests assembly and a slower running integration tests assembly.
My unit tests assembly is very fast (because the code is tested in isolation using mocks) and gets run very frequently as I develop. The integration tests are slower and I only run them when I finish a feature / check in or if I have a bad feeling about breaking something.
Maybe you could do something similar or even take the idea further and have 3 test suites with the third containing even slower client UI polling tests.
If you don't have a continuous integration server / process you should look at setting one up. This would continuously build you software and execute the tests. This could be set up to monitor check-ins and work in the background, sending out a notification if anything fails. With this in place you wouldn't care how long your client UI polling tests take because you wouldn't ever have to run them yourself.
Definitely split the tests out - separate unit tests from integration tests as a minimum.
As Martyn said, get a Continuous Integration system in place. I use Teamcity, which is excellent, easy to use, free for the first 20 builds, and you can happily run it on your own machine if you don't have a server at your disposal - http://www.jetbrains.com/teamcity/
Set up one build to run on every check in, and make that build run your unit tests, or fast-running tests if you will.
Set up a second build to run at midnight every night (or some other convenient time), and include in this the longer running client-calling integration tests. With this in place, it won't matter how long the tests take, and you'll get a big red flag first thing in the morning if your client has broken your stuff. You can also run these manually on demand, if you suspect there might be a problem.