I need to perform load test on our web application that involves user interaction to the web pages. So i have written selenium scripts to
handle the click events.For,example we have functionalities like some users has to perform signup/registering to our site,some has to login and perform click actions and logout, and a set of users has to visit our home page, click on multiple URL links available on the home page.
In JMeter,i have added one threadgroup each for above mentioned functionalities.Each threadgroup has "JUnit Request Sampler" which calls the selenium methods that has code for performing click actions.
I have setup the "Thread Properties" for the threadgroups to run 200 threads per minute over a span of 5 to 10 mins[target is for 15 mins]
Default browser i am using is "Firefox".I have also set the JMeter properties such that i wouldn't landup into any memory issues.
I am running the scripts in non-gui mode and collecting the result into a jtl file.
The problem i am facing here is, while scripts are executing multiple browsers openup and as each page involves click events,some of the clicks are not happening correctly resulting in increase in error count. If i use very minimal number of threads with time delay then no errors are seen.
I have tried with distributed mode testing as well but there was no major reduce in error count.
I am looking forward for a solution or rather suggestions that can help me out to have minimal or zero errors while running JMeter scripts that involves user interactions and achieve intended load or thereafter increase the load.
Regards
Praveena
As per WebDriver Tutorial it is not recommended to use WebDriver to conduct the load onto the web application
From experience, the number of browser (threads) that the reader creates should be limited by the following formula:
C = B + 1
where C = Number of Cores of the host running the test
and N = Number of Browser (threads).
Looking into Firefox browser requirements I strongly doubt you have a computer with 200 cores and 400 GBs of RAM therefore my expectation is that you should use JMeter's HTTP Request samplers to create the load.
Browsers don't do any magic, they send HTTP Requests and render the responses, so well-behaved protocol-based JMeter tests will look just like a real browser for the application under test. So I would recommend converting your Selenium tests into "pure" JMeter tests, you can leave 1-2 browsers to measure end-user experience when the application is under the load.
Related
I want to do browser based load testing with JMeter.This I can achieve by using web diver sampler and a thread group with multiple concurrent users. This limits me to test only with about 10 users as opening more browser instances eats away my system resources.My objective is to see how the GUI behaves when the server is under heavy load (say 1000) users.
As per the Web Driver Tutorial
Note: It is NOT the intention of this project to replace the HTTP Samplers included in JMeter. Rather it is meant to compliment them by measuring the end user load time.
So you should not be creating the main load using the WebDriver Sampler, you can use it in another Thread Group in parallel with the main load conducted by the HTTP Request samplers to measure page rendering speed, catch any markup or JavaScript errors, etc.
Well-behaved JMeter virtual user looks just like a real user using the real browser from backend perspective so make sure you properly mimic real user behavior, to wit:
"embedded resources" handling (images, scripts, styles, fonts, etc.)
cache
headers
cookies
"think times"
Check out How to make JMeter behave more like a real browser for more details.
This is the case to work with Remove/Distributed Testing
Note: The same test plan is run by all the servers. JMeter does not distribute the load between servers, each runs the full test plan. So if you set 1000 Threads and have 6 JMeter server, you end up injecting 6000 Threads.
So in your case if you have 100 JMeter servers each will execute 10 users opening browser.
I use JMeter for checking load testing.
I note a time with stopwatch when i check load time personally it was
8.5 seconds
when i run same case with JMeter it gave load time of 2 seconds
There is huge difference between them, How can i verify the actual time?
e.g : if one user taking 9 seconds to load the form while in JMeter it is given load time 2 seconds
Client time is a complex item, as you can see from the clip from the Chrome Developer tools, performance tab, above. There's lots going on at the client which does lead to a difference between the time you see with an HTTP protocol test tool, such as JMETER (and most of the other performance test tools on the planet) and the actual client render.
You can address this Delta in a number of ways:
Run a single GUI Virtual user. Name your timing records such as "Login" and "login_GUI." The delta between the two is your client weight. Make sure to run the GUI virtual user on a dedicated host to avoid resource contention
Run a test with all browsers. This was state of the art in 1995. Because of the resource cost and the skew imposed on trying to figure out the cost of the server response the entire industry shifted to protocol level virtual users. Some are trying to bring back this model as "state of the art." It is not
Ask a performance question earlier, also known as "shift left..." Every developer has these developer tools at their disposal, as does every functional tester. If you find that a client is slow for one user, be curious and use the developer tools to identify, "why?" If you are waiting to multi user performance testing to answer questions related to client weight, then you have waited too long and often will not have the time or resources to change the page architecture in meaningful ways to reduce the client page cost. This is where understanding earlier has tremendous advantages for making changes.
I picked the graphic above deliberately to illustrate the precise challenge you have. Notice, the loading of the components takes less than a tenth of a second. These are the requests that JMETER would be making. But the page takes almost five seconds to "render." Jmeter is not broken, it is working as designed. It is your understanding that needs to change on which tools can be used to pull particular stats for analysis.
You can't compare JMeter load time to browser as is, also because your browser will load JavaScript files and can call JavaScript functions on page load while JMeter doesn't execute JavaScript.
JMeter is not a browser, it works at protocol level. As far as
web-services and remote services are concerned, JMeter looks like a
browser (or rather, multiple browsers); however JMeter does not
perform all the actions supported by browsers. In particular, JMeter
does not execute the Javascript found in HTML pages. Nor does it
render the HTML pages as a browser does (it's possible to view the
response as HTML etc., but the timings are not included in any
samples, and only one sample in one thread is ever displayed at a
time).
Just a side note - you can use plugin to check exact load time in chrome.
Well-behaved JMeter test timing should be equal or similar to real user timing, if there is a 4x times difference - most probably your JMeter configuration is not correct.
Probably the most important. Make sure your HTTP Request samplers are configured to retrieve so called "embedded resources" (images, scripts, styles) which are referenced in the web page
If your application is using AJAX technology make sure you execute AJAX-driven requests as well and add their elapsed time to main sampler using i.e. Transaction Controller.
Make sure you mimic browser's:
Cookies via HTTP Cookie Manager
Headers via HTTP Header Manager
Cache via HTTP Cache Manager
Assuming all above you should be receiving similar to real user experience page load time. See How to make JMeter behave more like a real browser article for more detailed information on the above tips.
In addition to the answers provided by James and user7294900, please find these images to help you understand the reason behind the difference in time given by your stop watch and JMeter.
Below image gives the ideology behind how JMeter provides the time.
Below image gives the ideology behind how you have measured the time with
your stop watch.
Notice that there are additional actions performed by the browser when you are taking the time using your stop watch. This is the reason behind the huge difference in time between JMeter and your stop watch.
In addition to this, ensure that you are using the same test environmental conditions for both the tests (like same network conditions, same LG etc.)
Hope this helps!
how can we find rendering time from using values such as Latency, Connect time, Elapsed Time, Idle Time available to us in jtl file generated at end of test?
Is the below approach correct?
Rendering Time = Elapsed Time - Connect Time - Latency
This way (excluding network transport metrics) you will be able to get server response time, however as per JMeter project main page
JMeter is not a browser, it works at protocol level. As far as web-services and remote services are concerned, JMeter looks like a browser (or rather, multiple browsers); however JMeter does not perform all the actions supported by browsers. In particular, JMeter does not execute the Javascript found in HTML pages. Nor does it render the HTML pages as a browser does (it's possible to view the response as HTML etc., but the timings are not included in any samples, and only one sample in one thread is ever displayed at a time).
So JMeter will not actually load DOM, execute scripts, and display the generated page, it just sends a request, receives "plain" response and measures time in-between.
If you need to measure page rendering page while your application is under the load I would recommend going for WebDriver Sampler which provides JMeter integration with Selenium browser automation framework. This way you will be able to measure real user experience and you will have a possibility to use Navigation Timing API to get some extra information regarding page loading events and their duration.
We are using Apache JMeter for the performance testing of web application. Apparently response time is too high in comparison to loading page in browser during load. When we open the page during load it opens in 2 seconds however JMeter reports 70 seconds. I understand browser memory cache and disk cache are used in browser however isn't JMeter cache manager does same. How to assert it, comparing response header is one option. Any thoughts on this will be appreciated.
It may be wrong configuration in the script. There won't be much difference between web browser and Jmeter response times (browser rendering time is ignored in Jmeter, not a big factor but must be considered)
If you are using single Http Sampler for a web page and retrieving all resources in tha page, then select "Parallel Downloads" option to '6' in Http sampler advanced section. So, you are simulating the browser behaviour of parallel downloading of the resources like .js, .css, images etc.
If you recorded the script using Test Script Recorder, then there will be an Http Sampler for each resource requet for that page which will be sent sequentially, hence increase in response time. You might be in this case, as of now, there is no feature/option to send the http samplers in parallel. so I suggest using the approach of adding one sampler, use parallel download of resources option in Http Sampler advanced section.
Also, caching is an important factor which decides the response time. Adding Http Cache manager can solve the issue in jmeter. This simulates browser behaviour of caching. I don't think there will be huge difference b/w browser and jmeter in implementing caching althought may not be perfect.
The reason could be the lack of resources on JMeter machine or JMeter not being properly configured for producing high loads. JMeter default configuration is good for load tests development and debugging, you can run load tests up to certain amount of virtual users, but if you need to conduct a really high loads - you need to perform some configuration changes.
First of all, double check my guess using jvisualvm and your operating system monitoring tools (there is PerfMon JMeter Plugin which can be used for monitoring different metrics on application under test and JMeter load generators sides). If it is the case - take the next steps to get the most performance of your JMeter installation:
Increase JVM Heap Size, NewSize, switch to ConcurrentMarkSweep Garbage Collector.
Run your test in non-GUI mode.
Disable all the listeners during the test.
Follow other recommendations from 9 Easy Solutions for a JMeter Load Test “Out of Memory” Failure article.
I have created a Thread Group which has Firefox Driver Config , a Web driver Sampler and a View Results Tree Listener.
I have also added CSV Data set Config and passing username and Passwords through it to the application.
I have around 10 Credentials.
The scenario i want to do is, I want all the 10 users to login at the same time i.e 10 instances of the firefox browser to be invoked and do the job. i have given 10 in the thread group, but the users login 1 after the other.
Basically i am doing a performance testing, So i need all the 10 users to login at the same time(Simultaneously). So that 10 instances of firefox can be invoked at the same time.
Can anyone tell me, how can i achieve this in Jmeter?
Just add Synchronizing Timer to your test plan, if you need to test simultaneous login - put the timer as a child of login webdriver sampler.
I assume that you aware of recommendations from WebDriver Sampler Tutorial like
It is NOT the intention of this project to replace the HTTP Samplers included in JMeter. Rather it is meant to compliment them by measuring the end user load time.
and
JMeter allows the creation of multiple threads, and each thread is responsible for creating load on the server. However, for the Web Driver use case, the reader should be prudent in the number of threads they will create as each thread will have a single browser instance associated with it. Each browser consumes a significant amount of resources, and a limit should be placed on how many browsers the reader should create.
So it is recommended to create main load using JMeter's HTTP Request samplers and using one thread executing WebDriver Sampler to measure real-life user experience. If you still intend to create the main load with the WebDriver Sampler remember that browsers are resource-intensive so you'll need to provide something like 1 core and 1 Gb of RAM to each browser instance so make sure that machine(s) running JMeter have some capacity.