We are planning to do load testing in browser so would like to know if you suggest any tools to achieve the 10k+ load requests.
Also would like to know if it can be achieved through jmeter for browser test.
Thanks In Advance
JMeter per se works on HTTP protocol level, however given you properly configure JMeter to behave like a real browser from the application under test perspective it will be no difference at all, it wouldn't recognise whether it's being hit by JMeter with 10k of virtual users or 10k real browsers.
The most problematic point of using real browsers is an immense amount of hardware you will need for the test, if you look at Firefox 71 browser requirements you will see that for normal operation of browser you will need a CPU core and 2 GB of RAM so for 10k real browsers you will need 10K processors and 20 terabytes of RAM.
If you still want to proceed with real browsers be aware that JMeter can be integrated with Selenium browser automation framework using WebDriver Sampler but most probably you will have to go for Distributed Testing
Related
We are using JMeter to perform load testing of apis. If want to perform load or stress testing on web application which has JavaScript rendering, would JMeter with Selenium only option or any other options can be utilized like Selenium functional tests integrate with any performance tool.
Please suggest.
Have gone through/referred to this questions:
How to approach "end-client" performance testing on single-page (web) applications?
You can choose to use Selenium but it's hard to generate high load using selenium based test. You can choose to run two performance tests in parallel
Back-end Performance test: Use this to inject load on the back-end. Record Network traffic for the journey steps using JMeter proxy. Take out third-party and static urls, which are not relevant to load test.
Front-end performance test: Use this to measure end-user impact. Use between 1 - 5 virtual users in this test. Generating load is not important in this case, so you can use low amount of virtual users in this case. You can either use selenium based load test, webpagetest.org or similar tools.
This approach is more efficient than the selenium based performance test because selenium based load test will require very high amount of compute resources to scale.
If you use taurus to run your perf test then it should make it easier to run two JMeter tests in parallel using bzt scenario1.jmx scenario2.jmx
Hope this helps.
Depending on what you're trying to achieve and how many machine power you have you can consider:
Running Selenium tests in parallel using Selenium Grid but in this case:
you will need to ensure to have enough CPU, RAM, etc., i.e Firefox 72 needs 1 CPU and 2 GB of RAM per single browser instance
you will not have HTTP protocol specific metrics (Connect time, time to first byte, time to last byte, etc.) and load test specific metrics (active threads over time, transactions per second, etc)
Conducting the main load using JMeter HTTP Request samplers (just make sure to properly configure JMeter to behave like a real browser) and have 1-2 threads running WebDriver Sampler to collect client-side performance metrics i.e from Performance object
I run test in Jmeter with Selenium WebDriver Sampler
on Linux X86 and java SDK 11
The test run with 50 users.
I run it from the command line with non Gui mode and with Chrome headless mode.
but after 5 minutes the CPU going up to 100% and the memory almost full (8G).
What can I do to improve it?, I need run the test with 200 users and up.
Thanks,
Izik
You're expecting too much from your machine.
Although there are no specific RAM requirements defined at the Chrome System Requirements page, on my machine a single Chrome instance with a single tab in "porno" mode consumes almost 1GB of RAM. And this is given http://example.com page open, not modern web application with tons of JavaScript
I bet if you run the following command on your machine - you will get at least 3GB
(Get-Process chrome | Measure-Object WorkingSet -sum).sum
As per WebDriver Sampler tutorial
Note: It is NOT the intention of this project to replace the HTTP Samplers included in JMeter. Rather it is meant to compliment them by measuring the end user load time.
So my expectation is that you should be conducting the load using HTTP Request samplers and forget about using real browsers (or use 1 instance to collect client-side performance metrics). Just consider following recommendations from How to make JMeter behave more like a real browser article to ensure that protocol-based JMeter test has the same network footprint as real browser produces.
If you have to use real browsers for performance testing you won't be able to launch 200 browser instances on a machine with 8GB of RAM, you will have to find another 30-40 machines of this specifications and go for Distributed Testing.
This is expected, Since you are using Selenium it will use the JVM and browser which will consume a lot of memory. I would suggest you to distribute the test in multiple machine if you are going the Selenium route for load testing. This way you will be able to load test for more more number of users.
The best would be to stick to HHTP sampler as suggested above. You could also record and make necessary changes.
I wonder if some one solved the issue of browser multi thread with a request response script for load test
If you are going to use real Chrome and FF browsers for load test you can consider the following options:
Selenium Grid
Selenium-Grid allows you run your tests on different machines against different browsers in parallel. That is, running multiple tests at the same time against different machines running different browsers and operating systems. Essentially, Selenium-Grid support distributed test execution. It allows for running your tests in a distributed test execution environment.
Apache JMeter with the WebDriver Sampler plugin. This way you will be able to control concurrency and get performance metrics in form of HTML Reporting Dashboard.
Simulating web browser concurrency is something most load testing tools do very badly, if at all. We have tried to do this in k6 by letting each VU (virtual user) use multiple, concurrent TCP connections to fetch things in parallel. There is a special API function - http.batch() - that enables this functionality. http.batch() accepts multiple URLs as input parameters, and will fetch as many as possible in parallel.
Like Dmitri writes, Jmeter has a plugin that provides concurrency - sort of. However, what it actually does (unless I'm misinformed) is to spread out requests over multple VUs. Each VU will still only use one concurrent connection, which means that if you e.g. want to simulate 100 real browser users, you may need to fire up 1,000 VUs to do so realistically. This is not very different from what you would get with any other load testing tool, in my opinion: all of them allow you to start more VUs to create more concurrency and more traffic.
I'd say that apart from k6 and maybe one or perhaps two other tools (and Jmeter is not one of them), your only option if you want to really simulate the way browsers behave, is to use Selenium Grid or something similar to fire up a large number of real, actual browsers to do the load testing. The drawback is that real browsers are very expensive to run: they want lots of CPU and memory. But they provide the by far best browser "simulation".
I am performing baseline performance test on a project. Average load time reported by jmeter is much higher than actual load time in browser(fresh- no cache and cookies).
What will be the issue?
I suggest to check the following:
Load generator overload. Re-run JMeter test with one user/thread and compare with Firefox. If the results will be comparable then the response time in JMeter may be excessive due to its overload. Try to address it by adding more load generators.
Inaccurate browser emulation. If even with one user the response time in the load test is higher, then it can be caused by inaccurate emulation of browser paralel connections. To troubleshoot it, compare waterfall diagrams. To get it from Firefox, use Firebug. Route JMeter traffic through Fiddler which displays the waterfall on the Timeline tab. If the waterfalls are different, you may have the following issue: a web browser downloads resources in parallel, while by default JMeter replays recorded traffic sequentially. To fix it, add these settings: Simulating browsers using JMeter.
Are you checking with browser during the load test ? or at another time ?
In the latter case, you would be comparing apples and oranges.
Are you using JMeter GUI mode ? if yes, it's a bad practice, GUI mode is for scripting, NON GUI mode for load testing:
http://www.ubik-ingenierie.com/blog/jmeter_performance_tuning_tips/
How much threads are you using ? and which version of Jmeter ?
For embedded resources testing, 3.0 is the most realistic and performing:
https://jmeter.apache.org/changes.html
Whenever you run performance tests instead of Average response times, always consider 90th percentile. In some cases avg. response time is skewed even if one request takes long to respond. So please check 90th Percentile.
If you are running the test with multiple users, try to hit the application from browser while load test is going on and check the response time on browser. This will tell you if your observation is correct.
The load generator may not be able to establish enough connections due to which you might see higher response time. Check load generator utilization in case if you doubt. In some cases load generator itself can't generate enough load.
Check your sever utilization when you run the performance test. This will give you an idea if the application is not able to handle the load or if it's the issue with load generator.
If you are running tests from UI mode, please try to run tests with non-UI mode. (Can you specify with how many users you are running these tests?)
Increase JMeter memory if you see issue with load generator and keep eye on load generator CPU usage too.
Check if load generator and browser from which you are hitting the application are on the same network and check network latency to check if there is network problem.
I have set up a test plan using Apache JMeter, but when testing a remote server I am seeing much slower times than it takes the browser to run the same test, it's on order of 5-10x slower. For example chrome takes about 300ms to load a simple static page with some embedded assets while jmeter reports 2000+ms for the same page from the same machine.
I've tried tweaking the thread count to make sure this is not the bottleneck as well as not too much load slowing down the server, but nothing seems to change these slow numbers.
What else can I look at to get more realistic response times out of this tool?
I figured this one out, the issue was that "Retrieve all embedded resources" was checked, which I want, but I was also not using the "concurrent pool size" option, so it was cycling through each of the ~10 embedded HTTP requests on the page serially and reporting slow overall load times. Checking this and adding a realistic browser concurrency number (Chrome uses 6 these days so that's what I went with: http://sgdev-blog.blogspot.com/2014/01/maximum-concurrent-connection-to-same.html)
give me numbers that are very close to real browser testing.
Make sure you add HTTP Cache Manager as browsers download embedded stuff like images, scripts, styles, etc. but do it only once, on subsequent requests aforementioned resources are being returned from browser's cache, no actual request is being made.
See How to make JMeter behave more like a real browser guide for other recommendations on how to make your JMeter test more realistic.