I am looking at load testing of Progressive download video files with 100 user load. The testing tool I am looking at is Jmeter, Load Runner and NeoLoad. Though the script required for creating the load is very simpler, it consist of couple of request and it is able to make the connection with server and start the downloading of the file. Though I understand that the progressive technology is pretty old, but still it is used in many website. The question I have is around the strategy.
Do we need to download the complete file(i.e. 1.3 GB in my case)?
Even we looked at saving the response as file, the resources such as Network and disk I/O are at max? Does this strategy suits here?
Can we have some another strategy where we can engage the server for the duration and test for issues underlying with connection issues and transmission speed?
Depending on your use case, there is Seeking feature so theoretically you should be able to specify start offset so you will not have to get the whole file. Also you can consider using HTTP Header Manager to send Range header
If your target is to verify that the file has been downloaded fully and it is not broken you can tick "Save Response as MD5 Hash" box on "Advanced" tab of the HTTP Request sampler - this way you will save at least 130 GB of disk space. MD5 checksum can be verified using i.e. MD5Hex Assertion
The main idea of the load testing is simulating real application usage with 100% accuracy. Not knowing the requirements of your product it is impossible to come up with suggestions, however JMeter can be configured to behave pretty much like real browser does so it is a viable option.
See Load Testing Video Streaming with JMeter: Learn How article for more information if needed.
Related
I use JMeter for checking load testing.
I note a time with stopwatch when i check load time personally it was
8.5 seconds
when i run same case with JMeter it gave load time of 2 seconds
There is huge difference between them, How can i verify the actual time?
e.g : if one user taking 9 seconds to load the form while in JMeter it is given load time 2 seconds
Client time is a complex item, as you can see from the clip from the Chrome Developer tools, performance tab, above. There's lots going on at the client which does lead to a difference between the time you see with an HTTP protocol test tool, such as JMETER (and most of the other performance test tools on the planet) and the actual client render.
You can address this Delta in a number of ways:
Run a single GUI Virtual user. Name your timing records such as "Login" and "login_GUI." The delta between the two is your client weight. Make sure to run the GUI virtual user on a dedicated host to avoid resource contention
Run a test with all browsers. This was state of the art in 1995. Because of the resource cost and the skew imposed on trying to figure out the cost of the server response the entire industry shifted to protocol level virtual users. Some are trying to bring back this model as "state of the art." It is not
Ask a performance question earlier, also known as "shift left..." Every developer has these developer tools at their disposal, as does every functional tester. If you find that a client is slow for one user, be curious and use the developer tools to identify, "why?" If you are waiting to multi user performance testing to answer questions related to client weight, then you have waited too long and often will not have the time or resources to change the page architecture in meaningful ways to reduce the client page cost. This is where understanding earlier has tremendous advantages for making changes.
I picked the graphic above deliberately to illustrate the precise challenge you have. Notice, the loading of the components takes less than a tenth of a second. These are the requests that JMETER would be making. But the page takes almost five seconds to "render." Jmeter is not broken, it is working as designed. It is your understanding that needs to change on which tools can be used to pull particular stats for analysis.
You can't compare JMeter load time to browser as is, also because your browser will load JavaScript files and can call JavaScript functions on page load while JMeter doesn't execute JavaScript.
JMeter is not a browser, it works at protocol level. As far as
web-services and remote services are concerned, JMeter looks like a
browser (or rather, multiple browsers); however JMeter does not
perform all the actions supported by browsers. In particular, JMeter
does not execute the Javascript found in HTML pages. Nor does it
render the HTML pages as a browser does (it's possible to view the
response as HTML etc., but the timings are not included in any
samples, and only one sample in one thread is ever displayed at a
time).
Just a side note - you can use plugin to check exact load time in chrome.
Well-behaved JMeter test timing should be equal or similar to real user timing, if there is a 4x times difference - most probably your JMeter configuration is not correct.
Probably the most important. Make sure your HTTP Request samplers are configured to retrieve so called "embedded resources" (images, scripts, styles) which are referenced in the web page
If your application is using AJAX technology make sure you execute AJAX-driven requests as well and add their elapsed time to main sampler using i.e. Transaction Controller.
Make sure you mimic browser's:
Cookies via HTTP Cookie Manager
Headers via HTTP Header Manager
Cache via HTTP Cache Manager
Assuming all above you should be receiving similar to real user experience page load time. See How to make JMeter behave more like a real browser article for more detailed information on the above tips.
In addition to the answers provided by James and user7294900, please find these images to help you understand the reason behind the difference in time given by your stop watch and JMeter.
Below image gives the ideology behind how JMeter provides the time.
Below image gives the ideology behind how you have measured the time with
your stop watch.
Notice that there are additional actions performed by the browser when you are taking the time using your stop watch. This is the reason behind the huge difference in time between JMeter and your stop watch.
In addition to this, ensure that you are using the same test environmental conditions for both the tests (like same network conditions, same LG etc.)
Hope this helps!
Whether the each sample of jmeter will use or increase data usage?, Is there any solution available.
So as I understand, your problem is that while doing load tests whole day you have exceeded your network data limit and you want to know if there any ways to minimize the data usage.
Well, you can definitely minimize the data usage. But that is only possible by removing the samplers/requests. 10 GB is a lot of data and it seems that you are also performing static content requests in your JMeter scripts. So, you can remove these requests to reduce data usage.
But before doing so, you should ask the question if this approach is fine for your scenario. Normally, if we want to mimic real-time scenario then we include static requests. But if our load tests are to identify issues in specific area only then we can exclude static content requests.
Please check below question for further thoughts on this topic include static resources like images, css, js etc in tests
Consider the following:
Real browsers download embedded resources like images, styles, scripts, etc. But they do it only once, on subsequent requests the resources are being returned from the browser's cache. Add HTTP Cache Manager to your test plan to replicate this behaviour in your JMeter test
Focus solely on your application. If it uses content from 3rd-party websites - you don't need to include it into JMeter test as you don't have control of external websites content and won't be able to do anything if it becomes unavailable or will be loading slowly. So use "Embedded URLs must match" input of the HTTP Request Defaults to limit JMeter's scope to your domain(s) only. See Excluding Domains from the Load Test guide for more details.
I would like to measure loading time of a document in my testing web app. I have used JMeter for this, but I am getting different values for each run. I am measuring average time in the summary report.
I am not sure, that the value is proper or not.Is this approach is correct or Is there any plugin JMeter available?
I have used HTTP watch to get rendering time, but I can't use that tool for more than 1 user (Load Testing). I am using JMeter 2.13. Could you please help me in this?
With the help of aggregate report or csv / xml results you get required information regarding response times BUT
In Jmeter, Response time = Processing time + Latency(time taken by network while transferring data)
In Browser, Response time = Processing time + Latency + Rendering time
Hence you will found a difference between http watch response times and jmeter response times.
If you need to include rendering times also in your response times, then use tools, like loadrunner (commercial), selenium (open source) and so on. Personally in my opinion client side rendering is not a measurable value, unless all of the users accessing the application are having same configuration of hardware, software and network access. However, while JMeter test running with peak load to the system, manually browse the site using various browsers and with the help of developer tools you can find rendering times.
I am getting different values for each run - this will depends upon test data you are using, server health status, network delays and so on.
I doubt you'll be able to get 2 fully identical test run results, there always will be some form of fluctuation caused by underlying hardware and software implementations. You should be receiving similar results with some statistical noise.
If this is not the case, your JMeter test might be misconfigured. From "realness" perspective mind the following configuration:
Make sure you have Retrieve All Embedded Resources from HTML Files box checked and you Use concurrent pool. The easiest way to configure it for all the samplers is using HTTP Request Defaults
Add HTTP Cache Manager to your test plan. Previous setting "tells" JMeter to fetch embedded resources like scripts, styles, etc. from the pages. Real browsers do it as well but they do it only once, on subsequent requests these resources are being returned from browser's cache.
Add HTTP Cookie Manager to your test plan. It represents browser cookies, enables cookie-based authentication and maintains sessions.
Add HTTP Header Manager to represent browser headers like User-Agent, Content-Type, encoding, etc.
When you use a straight HTTP Protocol layer virtual user, independent of the tool (Jmeter, LoadRunner, SOASTA, Grinder, ...) then what you will be timing will be the request/response information coming from the server with very low coloration from the local processing on the client for JavaScript and the final "drawing on the screen" which is rendering.
Up until the point where the server is degraded due to number of requests or network limitations the only area where you can tune is in the page architecture, which if you are waiting to the last 100 yards before deployment to address then you are likely in trouble.
Steve Souders has written quite a bit on the subject of page architecture in his books "High Performance Websites" and related works. In short, the rule of thumb comes down to making fewer requests, smaller responses and serving the data from the closest possible location to the client. These have the effect of minimizing the most expensive finite resource to a web client, the network. For instance, a browser sprite reduces the number of calls for images, minification and compression reduce the size of the transmission and a CDN changes the number of hops to the requested item to a location closer to the end client.
In order to affect changes to page architecture you need to move upstream into your development cycle and your functional testing cycle. You will need to work with development to implement hard gates where code/pages cannot be submitted to the project without first passing performance gates related to design. Your development team and functional testing members will need to respect those gates. As to what the gates should be, I refer you back to the works of Mr Souders as a great source of data for construction of your gate rules.
This gets you to the level of "works for one: Performant for one." Then you can use that as a known good to answer the questions related to server scalability and at which point the service to the client from requests begins to degrade. If you have a CDN in your organization, be sure to take that into account in your test model, for if you do not then you will overload your server vs production.
As far as actual speeding of the "rendering" or drawing on the screen? You need to purchase a faster video card barring changes from the browser manufacturer. Speeding up JavaScript? Make sure that all of your JavaScript is as small and as lean as possible. Have your functional test team test on very dirty browsers with lots of add-ins as well as lower powered hardware for a view of maximum out of spec response. If you need a view of what your standard hardware model looks like from your clients (Browser/OS/some hardware into) then you can process the data in your HTTP request logs and specifically the user agent related to client configuration information.
I'm trying to simulate a connection to a website. The goal of the simulation is to collect statistics on page loading time on browser side.
I configured JMeter Flagging the option Retrieve Embedded Resources in order to simulate the real time to load the whole page. The issue is that while from a real Browser i have a response time (let's assume for the page A the response time is 10 seconds) in JMeter I found i response time 20 times higher.
It seems JMeter takes a much longer time to gather embedded resources (e.g. js, images, ...)
Do you have any suggestion for this issue?
Kind Regards
Update 31/07
I discovered some resources are not completely downloaded. Using Firebug i see some components with 0 bytes downloaded that the browsere keep trying to download (but the user do not percieve since the page is loaded). Therefore i suspect JMeter keeps trying downloading it. Is there any chance to set a timeout to overcome this kind of situation?
Update_1 31/07
I figured out that the issue is related with nested iframes. setting httpsampler.max_frame_depth=0 i get the correct time. however i would like to understand the reason of this issue. Do I have to set other paramters?
Disable browser cache and re-run your test in browser.
Jmeter will not store cache, unless otherwise specified.
Hope this will help.
Add a HTTP Cache Manager to your test plan.
Real browsers retrieve images, scripts, styles, etc. but do it only once. In order to simulate browser behavior you need to configure JMeter appropriately.
See How to make JMeter behave more like a real browser guide for more test elements which can be used for this.
I'm using JMeter to test a Java application written by a 3rd party vendor using Versata Logic Studio.
I've got some steps in my test plan that submit a request using some post data and then receive a response back:
Response too large to be displayed. Size: 445817 > Max: 204800, Start of message:
{"header":{"action":"300","arguments":{"tabid":"Header","divid":"ActgDisb,Vendor,BusinessType...ETC
This seems fine (I'm guessing that's 400K?), except that the step is taking far longer than it does to click through the pages in a browser. In the browser it takes 5 seconds at most. In JMeter it's taking 2 minutes. The CPU is also at 60% for just one thread during these steps.
Any ideas on speeding this up? We're struggling to get enough slaves going and this certainly isn't helping.
The message that is displayed tells that you are using View Result Tree during your load test. Jmeter sets a limit that can be changed on the size of pages displayed in this component by adding to user.properties file:
view.results.tree.max_size which defaults to 200 ko
BUT during a load test never ever use this component as it requires a lot of resources (memory and cpu). This component must only be used during scripting phase.
You can read this article that gives tips on JMeter configuration and tuning:
http://www.ubik-ingenierie.com/blog/jmeter_performance_tuning_tips/
Disclaimer : I wrote it and it's my company but IMHO I think it's worth reading :-)
Also read this:
http://jmeter.apache.org/usermanual/component_reference.html
If your JMeter script is using a lot of file I/O, then putting those files in RAM will significantly improve the speed. You can use any app such as IMDisk (freeware) to create a virtual disk in RAM. Make sure that you have more than 4GB RAM.
In our case, we are sending around 8000 small files per user. With 200 users on each system, Jmeter is reading 16 Lack files. This was the bottleneck. With the RAM Disk, the file read speed was increased by 20 times and it helped Jmeter to run at full speed.
If you're ready to move to TCP level, there is HTTP Raw Request that allows memory-efficient operation for huge uploads/downloads. Read its manual carefully, there is some JMeter properties for tuning its performance.
However, my experience is that you possibly have a situation where Java itself is a bad technology to perform load tests. I suggest you to take a pair of tries for Raw Request and in case of failure to seek for some C/C++ tool for performance tests.