I am learning Laravel Dusk and I am surprised by the slowness of this tool.
I read that the trait database migration could generate these delays. But in my case I don't use this trait.
Example:
Any suggestions to speed up Dusk? I am working on Windows.
Each page you are testing probably loads in 1-2 seconds, but if your test clicks to another page or interacts with something that uses transitions or animations and has to wait before it can interact, these all add up. There's also probably some test setup overhead that occurs.
But on average, 38 seconds for 6 tests is 6.7 seconds per test, and given the above, I would actually say that's not terribly bad.
How to speed things up:
Directly seed the database with as much test-related data as possible before running the test rather than creating them via your app. (Unless of course you are testing the ability to create that data via the app.)
Minimize the amount of waiting your test has to do by removing transitions. If you have a lot of tests, you might implement a website mode where your frontend looks for a query parameter in the URL and disables them as necessary.
Reduce the amount of setup/overhead for each test. For instance, if each test first hits a login page, that's an extra page load and input that needs to run each time. You can log in once and then perform a series of other inputs. Or better yet, test the login functionality in only one test, then configure the other tests to be logged in already so it presumes the user is already authenticated.
Related
The problem which I am encountering is related to E2E tests which will run all the time for new app builds (maybe even every few hours on CircleCi). I have ( and will have much more in the future ) features that contain a lot of setups ( necessary the same setup for each scenario to run). For example, before the scenario will run ( many in the feature ) need some users, contents, configuration etc. After the scenario runs probably the best practice is to delete/remove all that users, content etc (or at least after all the scenarios had run for the feature ). I am struggling to understand what is the best practice.
If I add a background then it will run before each scenario, but then I head to remove all that data from the background ( I could add a cleanup function in the last scenario step but that seems bad, correctly if I am wrong). I could add hooks that will clean up after each scenario and keep adding more hooks for new features ( maybe use tags for the scenarios to distinguish for which they should run ).
There are options but it does feel so inefficient... Those tests will be running in a live environment ( not integration or unit tests which are fast, but E2E ). Very often the setup/background will take much more time than one scenario to run and then it will run over and over for each tinny scenario. For example, had to run in e.g. background bunch of endpoints to create users, some content and in many cases ( when we don't have an endpoint for it yet ) I will have to write an automated journey through the UI to add something or change specific settings and then same way add the end delete everything and also through UI change the setting to the state before the feature had run. It feels so slow and inefficient...
The only other thing which comes to my mind ( but will not probably work for all the cases ). Is to create a huge hooks script where I will be adding all the necessary "stuff" before the whole suite run and after the whole thing run I clean the whole stack/instance DB ( or reset to some preset DB snapshot ) to make it state as before the whole suite run.
Please help me to understand what are the best practices in such a cases
Regards
Generally with Cuking the idea is that the database is reset after every scenario. This is done by things like:
running the scenario in a transaction (which is then rolled back)
emptying the database after every scenario
Which you do depends on which flavour of cuke you are using.
The inefficiencies you talk about can be mitigated in a number of ways without compromising the idea that the database should be reset after every scenario. Basically you can think of Cukes as setting up state (Givens) doing something (When) and validating (Thens). Only Whens have to use the UI.
So with Givens you can set up state by either
writing directly to the database using factories or fixtures
calling services (the same ones your UI controllers use) to create things
The second one is much preferred.
With most of the work being done outside the UI this means that you can get rapid cukes that do complex setup quickly.
This really is the way to go when cuking. Setup everything in Givens (using background when appropriate) without using the UI, then login, then do your When using the UI, and validate the result in the UI with your Thens.
Using this approach my current project has approx 450 scenarios that run in about 5 mins on my Mac mini, and that includes
several scenarios that step through UI wizards using a browser (super slow)
many scenarios with complex setup of multiple entities
This idea what you have to work around standard practices to make your suite efficient and fast is common and almost always wrong.
You can actually go way faster than I am going (though it takes quite a bit of work)
I use JMeter for checking load testing.
I note a time with stopwatch when i check load time personally it was
8.5 seconds
when i run same case with JMeter it gave load time of 2 seconds
There is huge difference between them, How can i verify the actual time?
e.g : if one user taking 9 seconds to load the form while in JMeter it is given load time 2 seconds
Client time is a complex item, as you can see from the clip from the Chrome Developer tools, performance tab, above. There's lots going on at the client which does lead to a difference between the time you see with an HTTP protocol test tool, such as JMETER (and most of the other performance test tools on the planet) and the actual client render.
You can address this Delta in a number of ways:
Run a single GUI Virtual user. Name your timing records such as "Login" and "login_GUI." The delta between the two is your client weight. Make sure to run the GUI virtual user on a dedicated host to avoid resource contention
Run a test with all browsers. This was state of the art in 1995. Because of the resource cost and the skew imposed on trying to figure out the cost of the server response the entire industry shifted to protocol level virtual users. Some are trying to bring back this model as "state of the art." It is not
Ask a performance question earlier, also known as "shift left..." Every developer has these developer tools at their disposal, as does every functional tester. If you find that a client is slow for one user, be curious and use the developer tools to identify, "why?" If you are waiting to multi user performance testing to answer questions related to client weight, then you have waited too long and often will not have the time or resources to change the page architecture in meaningful ways to reduce the client page cost. This is where understanding earlier has tremendous advantages for making changes.
I picked the graphic above deliberately to illustrate the precise challenge you have. Notice, the loading of the components takes less than a tenth of a second. These are the requests that JMETER would be making. But the page takes almost five seconds to "render." Jmeter is not broken, it is working as designed. It is your understanding that needs to change on which tools can be used to pull particular stats for analysis.
You can't compare JMeter load time to browser as is, also because your browser will load JavaScript files and can call JavaScript functions on page load while JMeter doesn't execute JavaScript.
JMeter is not a browser, it works at protocol level. As far as
web-services and remote services are concerned, JMeter looks like a
browser (or rather, multiple browsers); however JMeter does not
perform all the actions supported by browsers. In particular, JMeter
does not execute the Javascript found in HTML pages. Nor does it
render the HTML pages as a browser does (it's possible to view the
response as HTML etc., but the timings are not included in any
samples, and only one sample in one thread is ever displayed at a
time).
Just a side note - you can use plugin to check exact load time in chrome.
Well-behaved JMeter test timing should be equal or similar to real user timing, if there is a 4x times difference - most probably your JMeter configuration is not correct.
Probably the most important. Make sure your HTTP Request samplers are configured to retrieve so called "embedded resources" (images, scripts, styles) which are referenced in the web page
If your application is using AJAX technology make sure you execute AJAX-driven requests as well and add their elapsed time to main sampler using i.e. Transaction Controller.
Make sure you mimic browser's:
Cookies via HTTP Cookie Manager
Headers via HTTP Header Manager
Cache via HTTP Cache Manager
Assuming all above you should be receiving similar to real user experience page load time. See How to make JMeter behave more like a real browser article for more detailed information on the above tips.
In addition to the answers provided by James and user7294900, please find these images to help you understand the reason behind the difference in time given by your stop watch and JMeter.
Below image gives the ideology behind how JMeter provides the time.
Below image gives the ideology behind how you have measured the time with
your stop watch.
Notice that there are additional actions performed by the browser when you are taking the time using your stop watch. This is the reason behind the huge difference in time between JMeter and your stop watch.
In addition to this, ensure that you are using the same test environmental conditions for both the tests (like same network conditions, same LG etc.)
Hope this helps!
Across all browsers/devices, I find random different pages, at random times, are very slow to load/don't load. The browser is stuck on 'Waiting for website.com'. I will wait 20 seconds and nothing will happen until I manually refresh the page. As I realise this is very vague, can you suggest a) most likely issues to look for first or b) some diagnostic tools that I could use to try and de-bug the issue as a starting point, so that my hosts/developers can solve the issue. Here are some results of recent speed tests.
One thing to also add is that, it seems it more often gets stuck on particular pages. Namely the pages where users take practice tests. After each time the user clicks 'Next', their selected answer is inserted into the database. My speculation is that potentially it's an issue with the DB itself, or the process which inserts into the database. It's when clicking 'Next', that the whole website sometimes just dies as described above.
Results from Google Speed Test
Waterfall image
A wait time of 20secs at random times and random pages could possibly be due to stop-the-world garbage collection. So GC logs are probably a good starting point.
A thread sampler such as Djigger a colleague of mine wrote might probably also help you figuring out what the machine is doing during the 20 seconds.
If that doesn't help I suggest to use a Profiler or better an APM tool to monitor whats going on on your system. Those tools give a you a broader insight of the internals.
You need to run a few page speed tests and look at the waterfall images.
It is very common on shared servers for the server to be too busy to get to your request. 20 seconds would indicate a serious issue with the server.
Another common reason is the page has a link to a third party resource and that resource is too often unavailable.
In your case the culprit is website.com and I assume that is your site.
Use something like webpagetest.org to run the tests.
In the waterfall image below
Dark Green is DNS lookup time.
Orange is the time for Browser to connect to server.
Green is the wait time for server to put image in output buffer.
Blue is the time for the server to transmit to the Browser.
The problem with the sample waterfall page is the index page took 4 seconds to be generated or retrieved. Most likely this is a Word Press site with plugins.
I suspect yours may be 20 seconds. But due to the randomness, is is also a good possibility it is a page resource that is stalled.
If it is the index page, then you likely have a poor ISP and or one of the other users of the server is hogging the CPU.
Keep running the tests until you see the problem occur.
It will be very obvious where the problem is located.
You can post the waterfall image and send me a message if you have any questions.
Waterfall from webpagetest.org
I have a web performance test that begins with a webforms login, executes a few steps and then finishes.
Mostly this runs without errors but if I extend the load test run beyond 15 minutes I start to get load test failures which fail because some tests send a Session and Auth cookie on the initial Get to the root url.
Clearly the test recording does not have cookies on the initial request. Additionally, I have set the "Percentage of New Users" on the scenarios to 100% to ensure that all tests are running as a new user.
The test is databound to a list of 600 users in a User Pace scenario. Nothing very heavy.
However, I cannot identify why after a period of time (12 minutes) some of the tests begin to send the cookies on the initial request!
Can anyone give me any pointers please?
This is an old question and on re-reading it not very clear.
The scenario reflects more my lack of knowledge of the web testing features I was using.
I am fairly certain it was caused by a missing "log out" test step combined with the configuration of the load test probably re-using connections.
After much prodding around I achieved some clean runs
I'm currently in an environment where we are parsing data off of the client's website. I want to use my tests to ensure that when the client changes their site, I know when we are no longer receiving the information.
My first approach was to do pure integration tests where my tests hit the client's site and assert that the data was found. However half way through and 500 tests in, the test run has become unbearable and in some cases started timing out. So I cleared out as many tests that I could without loosing the core protection they are providing and I'm down to 350 or so. I'm left with a fear to add more tests to only break all the tests. I also find myself not running the 5+ minute duration (some clients will be longer as this is based on speed of communication with their site) when I make changes anymore. I consider this a complete failure.
I've been putting a lot of thought into this and asking around the office, my thoughts for my next attempt at this is to pull down the client's pages and write tests against these embedded resources in my projects. This will give me my higher test coverage and allow me to go back to testing in isolation. However I would need to be notified when they make changes and then re-pull down the pages to test against. I don't think the clients will adhere to this.
A suggestion was made to me to augment this with a suite of 'random' integration tests that serve the same function as my failed tests (hit the clients site) but in a lot less number than before. I really don't like the idea of random testing, where the possibility of sometimes getting red lights and some times getting green lights with the same code. But this so far sounds like the best idea I've heard to still gain the awareness of when the client's site has changed and my code no longer finds the data.
Has anyone found themselves testing an environment like this? Any suggestions from the testing community for me?
When you say the big test has become unbearable, it suggests that you are running this test suite manually. You shouldn't have to. It should just be running constantly in the background, at whatever speed it takes to complete the suite - and then start over again (perhaps after a delay if there are associated costs). Only when something goes wrong should you get an alert.
If there is something about your tests that causes them to get slower as their number grows - find it and fix it. Tests should be independent of one another, so simply having more of them shouldn't cause individual tests to time out.
My recommendation would be to try to isolate as much as possible the part of code that deals with the uncertainty. This part should be an API that works as a service used by all the other code. This way you would be protecting most of your code against changes.
The stable parts of the code should be unit-tested. With that part being independent from the connection to client's site running the tests should be way quicker and it would also make those tests more reliable.
The part that has to deal with the changes on the client's websites can be reduced. This way you are not solving the problem but at least you're minimising it and centralising it in only one module of your code.
Suggesting to the clients to expose the data as a web service would be the best for you. But I guess that doesn't depend on you :P.
You should look at dividing your tests up, maybe into separate assemblies that can be run independently. I typically have a unit tests assembly and a slower running integration tests assembly.
My unit tests assembly is very fast (because the code is tested in isolation using mocks) and gets run very frequently as I develop. The integration tests are slower and I only run them when I finish a feature / check in or if I have a bad feeling about breaking something.
Maybe you could do something similar or even take the idea further and have 3 test suites with the third containing even slower client UI polling tests.
If you don't have a continuous integration server / process you should look at setting one up. This would continuously build you software and execute the tests. This could be set up to monitor check-ins and work in the background, sending out a notification if anything fails. With this in place you wouldn't care how long your client UI polling tests take because you wouldn't ever have to run them yourself.
Definitely split the tests out - separate unit tests from integration tests as a minimum.
As Martyn said, get a Continuous Integration system in place. I use Teamcity, which is excellent, easy to use, free for the first 20 builds, and you can happily run it on your own machine if you don't have a server at your disposal - http://www.jetbrains.com/teamcity/
Set up one build to run on every check in, and make that build run your unit tests, or fast-running tests if you will.
Set up a second build to run at midnight every night (or some other convenient time), and include in this the longer running client-calling integration tests. With this in place, it won't matter how long the tests take, and you'll get a big red flag first thing in the morning if your client has broken your stuff. You can also run these manually on demand, if you suspect there might be a problem.