Newbie to Visual Studio load testing
When I record a web performance test I get a list of requests, when I run this test some of the 'requests' expand to show more.
Q: why can I not see all these requests in the recorded list?
If I run the same actions with a tool like LoadUIWeb I see all these requests in the recording, none hidden.
I want to be able to test the website with and without calls to external sites like google-analytics, etc.
I found from googling that it appears that the only way around this is to write a plugin.
I'm surprised that I'd have to do this as other tools will show all requests on the recorded script. I want to ensure that there isn't something I'm missing...
Thanks
The "expanded request" or "child requests" are actually dependent requests. If your site has dependencies to other resources (e.g. images, scripts, css files, etc...) it will result on additional HTTP requests made to obtain these resources (assuming they are not cached locally). VS will simulate this behavior by analyzing the body and issuing dependent requests.
The reason they are not recorded is because they may change.
The following question provides very useful information about the behavior of dependent requests: How can I turn off request caching in a Visual Studio load test
Related
I use JMeter for checking load testing.
I note a time with stopwatch when i check load time personally it was
8.5 seconds
when i run same case with JMeter it gave load time of 2 seconds
There is huge difference between them, How can i verify the actual time?
e.g : if one user taking 9 seconds to load the form while in JMeter it is given load time 2 seconds
Client time is a complex item, as you can see from the clip from the Chrome Developer tools, performance tab, above. There's lots going on at the client which does lead to a difference between the time you see with an HTTP protocol test tool, such as JMETER (and most of the other performance test tools on the planet) and the actual client render.
You can address this Delta in a number of ways:
Run a single GUI Virtual user. Name your timing records such as "Login" and "login_GUI." The delta between the two is your client weight. Make sure to run the GUI virtual user on a dedicated host to avoid resource contention
Run a test with all browsers. This was state of the art in 1995. Because of the resource cost and the skew imposed on trying to figure out the cost of the server response the entire industry shifted to protocol level virtual users. Some are trying to bring back this model as "state of the art." It is not
Ask a performance question earlier, also known as "shift left..." Every developer has these developer tools at their disposal, as does every functional tester. If you find that a client is slow for one user, be curious and use the developer tools to identify, "why?" If you are waiting to multi user performance testing to answer questions related to client weight, then you have waited too long and often will not have the time or resources to change the page architecture in meaningful ways to reduce the client page cost. This is where understanding earlier has tremendous advantages for making changes.
I picked the graphic above deliberately to illustrate the precise challenge you have. Notice, the loading of the components takes less than a tenth of a second. These are the requests that JMETER would be making. But the page takes almost five seconds to "render." Jmeter is not broken, it is working as designed. It is your understanding that needs to change on which tools can be used to pull particular stats for analysis.
You can't compare JMeter load time to browser as is, also because your browser will load JavaScript files and can call JavaScript functions on page load while JMeter doesn't execute JavaScript.
JMeter is not a browser, it works at protocol level. As far as
web-services and remote services are concerned, JMeter looks like a
browser (or rather, multiple browsers); however JMeter does not
perform all the actions supported by browsers. In particular, JMeter
does not execute the Javascript found in HTML pages. Nor does it
render the HTML pages as a browser does (it's possible to view the
response as HTML etc., but the timings are not included in any
samples, and only one sample in one thread is ever displayed at a
time).
Just a side note - you can use plugin to check exact load time in chrome.
Well-behaved JMeter test timing should be equal or similar to real user timing, if there is a 4x times difference - most probably your JMeter configuration is not correct.
Probably the most important. Make sure your HTTP Request samplers are configured to retrieve so called "embedded resources" (images, scripts, styles) which are referenced in the web page
If your application is using AJAX technology make sure you execute AJAX-driven requests as well and add their elapsed time to main sampler using i.e. Transaction Controller.
Make sure you mimic browser's:
Cookies via HTTP Cookie Manager
Headers via HTTP Header Manager
Cache via HTTP Cache Manager
Assuming all above you should be receiving similar to real user experience page load time. See How to make JMeter behave more like a real browser article for more detailed information on the above tips.
In addition to the answers provided by James and user7294900, please find these images to help you understand the reason behind the difference in time given by your stop watch and JMeter.
Below image gives the ideology behind how JMeter provides the time.
Below image gives the ideology behind how you have measured the time with
your stop watch.
Notice that there are additional actions performed by the browser when you are taking the time using your stop watch. This is the reason behind the huge difference in time between JMeter and your stop watch.
In addition to this, ensure that you are using the same test environmental conditions for both the tests (like same network conditions, same LG etc.)
Hope this helps!
I am trying to figure out how best to track & trend end to end performance between releases. By end to end I mean, what is the experience form the client visiting this app via a browser. This includes download time, dom rendering, javascript rendering, etc.
Currently I am running load tests using Jmeter, which is great to prove application and database capacity. Unfortunately, Jmeter will never allow me to show a full picture of the user experience. Jmeter is not a browser therefore will never simulate javascript and dom rendering impact. IE: if time to first byte is 100ms, but it takes the browser 10 seconds to download assets and render the dom, we have problems.
I need a tool to help me with this. My initial idea is to leverage Selenium. It could run a set of tests (login, view this, create that) and somehow record timings for each. We would need to run the same scenario multiple times and likely through a set of browsers. This would be done before every release and would allow me to identify changes in the experience to the user.
For example, this is what I would like to generate:
action | v1.5 | v1.6 | v1.7
----------------------------------------
login | 2.3s | 3.1s | 1.2s
create user | 2.9s | 2.7s | 1.5s
The problem with selenium is that 1. I am not sure if it is designed for this and 2. it appears that DOM ready or javascript rendering is realllly hard to detect.
Is this the right path? Does anyone have any pointers? Are there tools out there that I could leverage for this?
I think you have good goals, but I would split them:
Measuring DOM rendering, javascript rendering etc. are not really part of "experience from the client visiting this app via a browser", because your clients are usually unaware that you are "rendering dom" or "running javasript" - and they don't care. But they are something I'd want to address after every committed change, not just release to release, because it could be hard to trace degradation back to particular change if such test is not running all the time. So I would put it in continuous integration on build level. See a good discussion here
Then you probably would want to know if server side performance is the same or worsened (or is better). For that JMeter is ideal. Such testing could be done on some schedule (e.g. nightly or on each release) and can be automated using for example JMeter plug-in for Jenkins. If server side performance got worse, you don't really need end-to-end testing, since you already know what will happen.
But if server is doing well, then "end user experience" test using a real browser has a real value, so Selenium actually fits well to do this, and since it can be integrated with any of the testing frameworks (junit, nunit, etc), it also fits into automated process, and can generate some report, including duration (JUnit for instance has a TestWatcher which allows you to add consistent duration measurement to every test).
After all this automation, I would also do a "real end user experience" test, while JMeter performance test is running at the same time against the same server: get a real person to experience the app while it's under load. Because people, unlike automation, are unpredictable, which is good for finding bugs.
Regarding "JMeter is not a browser". It is really not a browser, but it may act like a browser given proper configuration, so make sure you:
add HTTP Cookie Manager to your Test Plan to represent browser cookies and deal with cookie-based authentication
add HTTP Header Manager to send the appropriate headers
configure HTTP Request samplers via HTTP Request Defaults to
Retrieve all embedded resources
Use thread pool of around 5 concurrent threads to do it
Add HTTP Cache Manager to represent browser cache (i.e. embedded resources retrieved only once per virtual user per iteration)
if your application is build on AJAX - you need to mimic AJAX requests with JMeter as well
Regarding "rendering", for example you detect that your application renders slowly on a certain browser and there is nothing you can do by tuning the application. What's next? You will be developing a patch or raising an issue to browser developers? I would recommend focus on areas you can control, and rendering DOM by a browser is not something you can.
If you still need these client-side metrics for any reason you can consider using WebDriver Sampler along with main JMeter load test so real browser metrics can also be added to the final report. You can even use Navigation API to collect the exact timings and add them to the load test report
See Using Selenium with JMeter's WebDriver Sampler to get started.
There are multiple options for tracking your application performance between builds (and JMeter tests executions), i.e.
JChav - JMeter Chart History And Visualisation - a standalone tool
Jenkins Performance Plugin - a Continuous Integration solution
Whether the each sample of jmeter will use or increase data usage?, Is there any solution available.
So as I understand, your problem is that while doing load tests whole day you have exceeded your network data limit and you want to know if there any ways to minimize the data usage.
Well, you can definitely minimize the data usage. But that is only possible by removing the samplers/requests. 10 GB is a lot of data and it seems that you are also performing static content requests in your JMeter scripts. So, you can remove these requests to reduce data usage.
But before doing so, you should ask the question if this approach is fine for your scenario. Normally, if we want to mimic real-time scenario then we include static requests. But if our load tests are to identify issues in specific area only then we can exclude static content requests.
Please check below question for further thoughts on this topic include static resources like images, css, js etc in tests
Consider the following:
Real browsers download embedded resources like images, styles, scripts, etc. But they do it only once, on subsequent requests the resources are being returned from the browser's cache. Add HTTP Cache Manager to your test plan to replicate this behaviour in your JMeter test
Focus solely on your application. If it uses content from 3rd-party websites - you don't need to include it into JMeter test as you don't have control of external websites content and won't be able to do anything if it becomes unavailable or will be loading slowly. So use "Embedded URLs must match" input of the HTTP Request Defaults to limit JMeter's scope to your domain(s) only. See Excluding Domains from the Load Test guide for more details.
I have set up a test plan using Apache JMeter, but when testing a remote server I am seeing much slower times than it takes the browser to run the same test, it's on order of 5-10x slower. For example chrome takes about 300ms to load a simple static page with some embedded assets while jmeter reports 2000+ms for the same page from the same machine.
I've tried tweaking the thread count to make sure this is not the bottleneck as well as not too much load slowing down the server, but nothing seems to change these slow numbers.
What else can I look at to get more realistic response times out of this tool?
I figured this one out, the issue was that "Retrieve all embedded resources" was checked, which I want, but I was also not using the "concurrent pool size" option, so it was cycling through each of the ~10 embedded HTTP requests on the page serially and reporting slow overall load times. Checking this and adding a realistic browser concurrency number (Chrome uses 6 these days so that's what I went with: http://sgdev-blog.blogspot.com/2014/01/maximum-concurrent-connection-to-same.html)
give me numbers that are very close to real browser testing.
Make sure you add HTTP Cache Manager as browsers download embedded stuff like images, scripts, styles, etc. but do it only once, on subsequent requests aforementioned resources are being returned from browser's cache, no actual request is being made.
See How to make JMeter behave more like a real browser guide for other recommendations on how to make your JMeter test more realistic.
I'm trying to simulate a connection to a website. The goal of the simulation is to collect statistics on page loading time on browser side.
I configured JMeter Flagging the option Retrieve Embedded Resources in order to simulate the real time to load the whole page. The issue is that while from a real Browser i have a response time (let's assume for the page A the response time is 10 seconds) in JMeter I found i response time 20 times higher.
It seems JMeter takes a much longer time to gather embedded resources (e.g. js, images, ...)
Do you have any suggestion for this issue?
Kind Regards
Update 31/07
I discovered some resources are not completely downloaded. Using Firebug i see some components with 0 bytes downloaded that the browsere keep trying to download (but the user do not percieve since the page is loaded). Therefore i suspect JMeter keeps trying downloading it. Is there any chance to set a timeout to overcome this kind of situation?
Update_1 31/07
I figured out that the issue is related with nested iframes. setting httpsampler.max_frame_depth=0 i get the correct time. however i would like to understand the reason of this issue. Do I have to set other paramters?
Disable browser cache and re-run your test in browser.
Jmeter will not store cache, unless otherwise specified.
Hope this will help.
Add a HTTP Cache Manager to your test plan.
Real browsers retrieve images, scripts, styles, etc. but do it only once. In order to simulate browser behavior you need to configure JMeter appropriately.
See How to make JMeter behave more like a real browser guide for more test elements which can be used for this.