I'm need to monitor my sites render times for common tasks (Login, Search etc.). I need something automated that can mimic a users actions on I.E. and be able time how long a page takes to render.
Example automated execution:
1) open headless IE browser
2) go to http://google.com
3) type "stackoverflow"
4) press submit button
5) start timer
6) wait for results page to fully
render
7) stop timer
8) Close IE
9) record results
I need this to run as a scheduled task while the server, without the user logged in.
I have been searching for something to help me do so. Anyone have any experience with this type of thing or know of anything that can accomplish this?
It depends on what you focus, functionality or performance.
Functionality
When monitoring functionality, you aim at automatically ensuring that a web application still works correctly. Usually, this is more part of the continous integration process - and less part of production monitoring. It can be well done with HtmlUnit, Selenium or WebDriver. HttpUnit is no longer recommended (API more low-level, JavaScript not so well supported, less widely adopted, fewer bug fixes and enhancements).
HtmlUnit simulates a browser. So you can never be sure, that your application behaves exactly identical in a real browser. This is especially important for sophisticated Ajax applications. This is comparable to all the small incompatibilities between FireFox and Internet Explorer. Pros: Headless, easy to understand. Cons: Risk of undetected incompatibilities.
Selenium remote controls a real browser. In our setup, we could not use it headlessly, especially with Internet Explorer. But if you embed it into a virtual machine, it runs headlessly. If your application is reachable through public internet, you might even use Selenium Grid and a preconfigured virtual machine from the Amazon Elastic Cloud EC2. Pros of Selenium: Real world compatibility, easy scripting. Cons: Headless only in virtual machine, performance overhead, more complex runtime setup, stress simulation of concurrent users only in the cloud.
Up to version 1.5, Selenium uses a JavaScript part called Selenium Core to control the browser. If your application has security restrictions for JavaScript, Selenium might not work correctly.
WebDriver uses for each browser the specific interface, e.g. for FireFox an extension and for internet explorer Automation Controls. Additionally it uses the operation system, e.g. for simulating keystrokes. This is more powerful, robust and reliable than Selenium Core. As of Selenium version 2.0, WebDriver is integrated into Selenium. But Selenium 2.0 is still beta.
Performance
You mention measuring with a timer and you mention rendering times. When monitoring performance of a web application, you want to be alerted when real world usage of a application is no longer possible due to overlong answering times.
In this scenario, you are normally not interested in exact results in milliseconds. You can still use one of the tools mentioned above. For example, a browser with Selenium Core is slower than a real world browser - but this is of little relevance for continuous monitoring.
If you absolutely need exact measurements, none of the above is suitable. You should differentiate between client-side duration and network plus server-side duration.
Client-side duration is needed for rendering the HTML and for executing the JavaScript. It does not depend on the number of concurrent users. You can measure it once, e.g. with Firebug. You do not need to monitor it permanently.
Network plus server-side duration is needed for transferring the request to the server, handling the request and generating the response and transferring the response to the client. They vary according to network usage and number of concurrent users. You can exactly measure and monitor them for example with JMeter. But in case of sophisticated Ajax functionality, the simulation of the right client requests in JMeter is a complex task. Pros of JMeter: Exact measurement, possibility to stress an application with many concurrent users. Cons: Limited for Ajax, much effort for request building.
Another option might be Selenium Remote Control (or Selenium in general).
One option for headless automation is to use HtmlUnit. Have a look at this link for more information: Using HtmlUnit on .NET for Headless Browser Automation
The following Headless IE port for PhantomJS is currently in Beta (v0.2):
http://triflejs.org/
Here is a quick intro:
The API is the same as PhantomJs so eventually you'll be able to do the following:
// 1. Create Page Object and navigate to Google
page = require("webpage").create();
page.open("http://www.google.com", function(status) {
if ( status === "success" ) {
// 2. Inject jQuery for DOM operations
page.includeJs("http://ajax.googleapis.com/ajax/libs/jquery/1.6.1/jquery.min.js", function() {
// 3. Start Timer
console.log('Start Timer: ' + (new Date()).getTime());
// 4. Type string and click search
page.evaluate(function() {
$(("input[type=text")[0]).val("stackoverflow");
$("button:contains('Google Search')).click();
});
// 5. Wait for loading and end timer.
page.onLoadFinished = function() {
console.log('Load Finished. End Timer:' + (new Date()).getTime());
phantom.exit();
};
});
}
});
Related
I wonder if some one solved the issue of browser multi thread with a request response script for load test
If you are going to use real Chrome and FF browsers for load test you can consider the following options:
Selenium Grid
Selenium-Grid allows you run your tests on different machines against different browsers in parallel. That is, running multiple tests at the same time against different machines running different browsers and operating systems. Essentially, Selenium-Grid support distributed test execution. It allows for running your tests in a distributed test execution environment.
Apache JMeter with the WebDriver Sampler plugin. This way you will be able to control concurrency and get performance metrics in form of HTML Reporting Dashboard.
Simulating web browser concurrency is something most load testing tools do very badly, if at all. We have tried to do this in k6 by letting each VU (virtual user) use multiple, concurrent TCP connections to fetch things in parallel. There is a special API function - http.batch() - that enables this functionality. http.batch() accepts multiple URLs as input parameters, and will fetch as many as possible in parallel.
Like Dmitri writes, Jmeter has a plugin that provides concurrency - sort of. However, what it actually does (unless I'm misinformed) is to spread out requests over multple VUs. Each VU will still only use one concurrent connection, which means that if you e.g. want to simulate 100 real browser users, you may need to fire up 1,000 VUs to do so realistically. This is not very different from what you would get with any other load testing tool, in my opinion: all of them allow you to start more VUs to create more concurrency and more traffic.
I'd say that apart from k6 and maybe one or perhaps two other tools (and Jmeter is not one of them), your only option if you want to really simulate the way browsers behave, is to use Selenium Grid or something similar to fire up a large number of real, actual browsers to do the load testing. The drawback is that real browsers are very expensive to run: they want lots of CPU and memory. But they provide the by far best browser "simulation".
I use JMeter for checking load testing.
I note a time with stopwatch when i check load time personally it was
8.5 seconds
when i run same case with JMeter it gave load time of 2 seconds
There is huge difference between them, How can i verify the actual time?
e.g : if one user taking 9 seconds to load the form while in JMeter it is given load time 2 seconds
Client time is a complex item, as you can see from the clip from the Chrome Developer tools, performance tab, above. There's lots going on at the client which does lead to a difference between the time you see with an HTTP protocol test tool, such as JMETER (and most of the other performance test tools on the planet) and the actual client render.
You can address this Delta in a number of ways:
Run a single GUI Virtual user. Name your timing records such as "Login" and "login_GUI." The delta between the two is your client weight. Make sure to run the GUI virtual user on a dedicated host to avoid resource contention
Run a test with all browsers. This was state of the art in 1995. Because of the resource cost and the skew imposed on trying to figure out the cost of the server response the entire industry shifted to protocol level virtual users. Some are trying to bring back this model as "state of the art." It is not
Ask a performance question earlier, also known as "shift left..." Every developer has these developer tools at their disposal, as does every functional tester. If you find that a client is slow for one user, be curious and use the developer tools to identify, "why?" If you are waiting to multi user performance testing to answer questions related to client weight, then you have waited too long and often will not have the time or resources to change the page architecture in meaningful ways to reduce the client page cost. This is where understanding earlier has tremendous advantages for making changes.
I picked the graphic above deliberately to illustrate the precise challenge you have. Notice, the loading of the components takes less than a tenth of a second. These are the requests that JMETER would be making. But the page takes almost five seconds to "render." Jmeter is not broken, it is working as designed. It is your understanding that needs to change on which tools can be used to pull particular stats for analysis.
You can't compare JMeter load time to browser as is, also because your browser will load JavaScript files and can call JavaScript functions on page load while JMeter doesn't execute JavaScript.
JMeter is not a browser, it works at protocol level. As far as
web-services and remote services are concerned, JMeter looks like a
browser (or rather, multiple browsers); however JMeter does not
perform all the actions supported by browsers. In particular, JMeter
does not execute the Javascript found in HTML pages. Nor does it
render the HTML pages as a browser does (it's possible to view the
response as HTML etc., but the timings are not included in any
samples, and only one sample in one thread is ever displayed at a
time).
Just a side note - you can use plugin to check exact load time in chrome.
Well-behaved JMeter test timing should be equal or similar to real user timing, if there is a 4x times difference - most probably your JMeter configuration is not correct.
Probably the most important. Make sure your HTTP Request samplers are configured to retrieve so called "embedded resources" (images, scripts, styles) which are referenced in the web page
If your application is using AJAX technology make sure you execute AJAX-driven requests as well and add their elapsed time to main sampler using i.e. Transaction Controller.
Make sure you mimic browser's:
Cookies via HTTP Cookie Manager
Headers via HTTP Header Manager
Cache via HTTP Cache Manager
Assuming all above you should be receiving similar to real user experience page load time. See How to make JMeter behave more like a real browser article for more detailed information on the above tips.
In addition to the answers provided by James and user7294900, please find these images to help you understand the reason behind the difference in time given by your stop watch and JMeter.
Below image gives the ideology behind how JMeter provides the time.
Below image gives the ideology behind how you have measured the time with
your stop watch.
Notice that there are additional actions performed by the browser when you are taking the time using your stop watch. This is the reason behind the huge difference in time between JMeter and your stop watch.
In addition to this, ensure that you are using the same test environmental conditions for both the tests (like same network conditions, same LG etc.)
Hope this helps!
I have a hunch that it may not be possible but what does it take for me to ask? Hey, "the fool didn't know it was impossible, so he did it."
This question is env/tool/version agnostic. I target any industry standard tools for performance testing like HP LR, Apache JMeter, SilkPerformer etc.
The scenario:
A Web(HTML/HTTP) script is being executed in LR Vugen.
As the script execution progresses, the vuser follows the scripted steps/journey. Each action function calls the SUT hosts/services for responses while maintaining local sessions, managing cookies and remembering headers, emulating browser caches so on.
Now, we can pause the execution any time in the tool and that will stop the vuser action. Question is, can we resume the session somewhere else or in the tool to manually interact with the web page or response while maintaining the same session?
This will help users reproduce a case with plenty of steps which can be executed by the tool and the user can take over at a certain point to carry with a different path.
The only way to accomplish interaction with the session is using Looadrunner's TrucClient protocol.
This protocol actually uses a real browser (Firefox or IE emulator) for every VUser being executed. With some runtime setting options, The browser can be visible while replaying the load scenario.
Of course, TruClient protocol is only used for testing web sites.
Hope this helps.
I am trying to figure out how best to track & trend end to end performance between releases. By end to end I mean, what is the experience form the client visiting this app via a browser. This includes download time, dom rendering, javascript rendering, etc.
Currently I am running load tests using Jmeter, which is great to prove application and database capacity. Unfortunately, Jmeter will never allow me to show a full picture of the user experience. Jmeter is not a browser therefore will never simulate javascript and dom rendering impact. IE: if time to first byte is 100ms, but it takes the browser 10 seconds to download assets and render the dom, we have problems.
I need a tool to help me with this. My initial idea is to leverage Selenium. It could run a set of tests (login, view this, create that) and somehow record timings for each. We would need to run the same scenario multiple times and likely through a set of browsers. This would be done before every release and would allow me to identify changes in the experience to the user.
For example, this is what I would like to generate:
action | v1.5 | v1.6 | v1.7
----------------------------------------
login | 2.3s | 3.1s | 1.2s
create user | 2.9s | 2.7s | 1.5s
The problem with selenium is that 1. I am not sure if it is designed for this and 2. it appears that DOM ready or javascript rendering is realllly hard to detect.
Is this the right path? Does anyone have any pointers? Are there tools out there that I could leverage for this?
I think you have good goals, but I would split them:
Measuring DOM rendering, javascript rendering etc. are not really part of "experience from the client visiting this app via a browser", because your clients are usually unaware that you are "rendering dom" or "running javasript" - and they don't care. But they are something I'd want to address after every committed change, not just release to release, because it could be hard to trace degradation back to particular change if such test is not running all the time. So I would put it in continuous integration on build level. See a good discussion here
Then you probably would want to know if server side performance is the same or worsened (or is better). For that JMeter is ideal. Such testing could be done on some schedule (e.g. nightly or on each release) and can be automated using for example JMeter plug-in for Jenkins. If server side performance got worse, you don't really need end-to-end testing, since you already know what will happen.
But if server is doing well, then "end user experience" test using a real browser has a real value, so Selenium actually fits well to do this, and since it can be integrated with any of the testing frameworks (junit, nunit, etc), it also fits into automated process, and can generate some report, including duration (JUnit for instance has a TestWatcher which allows you to add consistent duration measurement to every test).
After all this automation, I would also do a "real end user experience" test, while JMeter performance test is running at the same time against the same server: get a real person to experience the app while it's under load. Because people, unlike automation, are unpredictable, which is good for finding bugs.
Regarding "JMeter is not a browser". It is really not a browser, but it may act like a browser given proper configuration, so make sure you:
add HTTP Cookie Manager to your Test Plan to represent browser cookies and deal with cookie-based authentication
add HTTP Header Manager to send the appropriate headers
configure HTTP Request samplers via HTTP Request Defaults to
Retrieve all embedded resources
Use thread pool of around 5 concurrent threads to do it
Add HTTP Cache Manager to represent browser cache (i.e. embedded resources retrieved only once per virtual user per iteration)
if your application is build on AJAX - you need to mimic AJAX requests with JMeter as well
Regarding "rendering", for example you detect that your application renders slowly on a certain browser and there is nothing you can do by tuning the application. What's next? You will be developing a patch or raising an issue to browser developers? I would recommend focus on areas you can control, and rendering DOM by a browser is not something you can.
If you still need these client-side metrics for any reason you can consider using WebDriver Sampler along with main JMeter load test so real browser metrics can also be added to the final report. You can even use Navigation API to collect the exact timings and add them to the load test report
See Using Selenium with JMeter's WebDriver Sampler to get started.
There are multiple options for tracking your application performance between builds (and JMeter tests executions), i.e.
JChav - JMeter Chart History And Visualisation - a standalone tool
Jenkins Performance Plugin - a Continuous Integration solution
I'm trying to decide what the best solution would be for my web application. I have a page that will fire an arbitrary number of ajax requests to retrieve data from the server. For example a page on load may fire 10 ajax requests to the server and each request may take 10 seconds (+-) to return content.
In view of this being a web app in a multi user and multi concurrency environment is it a good idea to use a traditional ajax approach or would you opt for long polling, such as SignalR.
What are the pros/cons of both approaches (Pull vs Push)? Ultimately i'm after the most resource efficient approach.
Thanks
In your stated example you are talking about a pure 'Pull' scenario. ie 'When the page loads I want X, Y, Z to happen and then I want to see the results'.
Long polling/websockets (SignalR) is useful for a Push scenario - ie 'Oh look I have finished running this super long process... I better tell any users currently connected'.
You can use SignalR to run those normal style AJAX requests... but you wont get any performance enhancements. The AJAX will run asynchronously and in parallel and once the server side process is complete you will a callback that executes. Perhaps you might get a slight increase in performance as signalR will have a continuous connection running, so you will loose the slight delay in creating a connection. On the flip side the server will have a large number of open connections running which may degrade performance (especially if you are hitting it with 10 X 10 sec computations)