I am load testing a scenario in JMeter. I'm trying to achieve a certain amount of r/s and then sustain that for x amount of time. However, with the testing tool, it looks like falling short of r/s for each endpoint. Do I just need more virtual users from the system? or is an additional fix needed for the test plan below?
HTTP Header Manager
User Defined Variables
DNS Cache Manager
HTTP Authorization Manager
HTTP Cache Manager
Thread Group
HTTP Cookie Manager
CSV Data Set Config # 1
HTTP Request #1
HTTP Header Manager
HTTP Request #2
HTTP Header Manager
JSON Extractor
HTTP Request #3
HTTP Header Manager
CSV Data Set Config # 2
HTTP Request #4
JSR2223 PreProcessor
HTTP Header Manager
The CSV Data Set Config # 1 is CSV of username and passwords that are passed to HTTP Requests #1 and #2. The JSON Extractor extracts values from HTTP Request #2 to the HTTP Header Manager of HTTP Request #3.
I can think of 2 situations:
JMeter isn't capable of sending requests fast enough. If this is the case:
Make sure to follow JMeter Best Practices
Try increasing the number of threads/ virtual users
If you cannot conduct the required load from a single load generator - consider switching to Distributed Testing
Your application under test isn't capable of responding fast enough. The reasons could be:
Your application lacks resources (CPU, RAM, Network, Disk, Swap, etc.). Consider monitoring its baseline health metrics using i.e. JMeter PerfMon Plugin
Your application middleware configuration isn't suitable for high loads. Inspect configuration of load balancer, application server, database, whatever to see if they're set up good enough for the anticipated load. Check out Web Performance Optimization tips to get initial idea. Normally the vendors of application servers, databases, etc. provide scaling instructions.
Your application code is not efficient enough, i.e. uses bad algorithms. You can use profiling tools to check what are the slowest and/or frequently called functions, largest objects, etc.
Related
I am working on migration of scripts from performance center to Jmeter5.2.1.
As part of this migration , we are using same functional flow which we did in performance center.
My scenario consists of users logging in to the web application perform 10-15 iterations and then logout.
This is my Testplan.
TestPlan
--ThreadGroup1
--Once Only Controller (login of users)
--Loop Controller (10 Iterations)
HTTP1
HTTP2
HTTP3
.
.
--Once only Controller (logout of users)
--csv Config data ( username/password)
--csv config data( unique data for the loop controller)
With this approach I am noticing that the time taken to complete the test in Jmeter is much more than what we have in performance center ( I took care of think times and added the similar values)
Why is my test run slow in Jmeter?
Is loop controller sequential? Meaning at a given time it can run only one request?
If not loop controller what other options we have to satisfy my scenario.
If I include different thread groups , carrying JSESSIONIDs needs to be done across thread groups which is not a best practice to do so.
Update:
Comparison between performance center and Jmeter settings
Below are the settings in Jmeter.
Thread Group settings:
TestPlan :
HTTP Cookie manager in Thread Group
CSV data files in Test plan
Once Only counters for Login and Logout
Loop Controller for Iterations.
HTTP request Defaults: ( Even with out checking retrieve all embedded and parallel downloads its taking more than an hour for 3 users)
TestPlan
Performance Center results :
Every Sampler has HTTP Header manager
Entire Test Plan
Given you send the same requests you should have the same response times, no matter which tool is being used under the hood.
It's hard to say what the differences are without seeing the full scripts from the both tools so generic advice is to use a third-party sniffer tool like Wireshark or Fiddler in order to identify the differences and configure JMeter to behave exactly like the "performance center" (whatever it is)
For example I fail to see HTTP Cache Manager and it will cause JMeter to download embedded resources (images, scripts, styles, sounds, fonts) for each and every HTTP request while real browser does it only once.
I also don't see HTTP Header Manager which might be very important, for example if you send Accept header the server will be aware that the client can understand gzipped responses which will greatly reduce network traffic.
More information: How to make JMeter behave more like a real browser
We need to test our web application with 100+ users with JMeter tool.
Should we use JMeter webdriver plugin to launch the real browsers or can we use the blazemeter plugin to generate the script and run in JMeter? Will the script created with blazemeter simulate the real browser load?
Theoretically you can use real browsers, but be aware that browsers are very resource intensive so my expectation is that you will need at least one CPU core and at 1 GB of RAM per browser instance in order to have enough resources.
This means you will have to go for distributed testing and on average you will have one slave per 5 browsers. If this is something you can afford - go for it. If don't - be aware that you can configure JMeter to behave like a real browser, to wit:
Tell JMeter to Retrieve All Embedded Resources from HTML Files and Use concurrent pool of 6 threads. This can be done using HTTP Request Defaults, the settings live under "Advanced" tab
Add HTTP Cache Manager to simulate browser cache
Add HTTP Cookie Manager to mimic browser cookies
Add HTTP Header Manager to send browser-like headers (i.e. User-Agent, Accept-Encoding, etc.)
If your application uses AJAX requests - put the relevant HTTP Request samplers under the Parallel Controller
I am using jMeter to load test logging in and logging out of my application. I have a CSV with 500 uname and pwords, and I have also set the active threads to 500.
Is this the correct way to stimulate a load of 500 users?
From technical perspective - yes, JMeter will pick up credentials from CSV and send them along with the request.
However if you think about what happens in reality logging in and immediately logging out doesn't seem a realistic use case for me. The whole idea of load testing a web application is simulating real users as close as possible, it includes:
Application usage scenario(s): how many users will be accessing your application at the same time and what users will be doing what things? For example given Stack OverFlow the vast majority of users are searching for answers by given criteria, some users are typing questions, some of them providing answers, and some of them are commenting. These all are different behavioral patterns which need to be accurately simulated by your load test
Each user should have a "user session" which identifies him so application would distinguish different users so make sure you have HTTP Cookie Manager in your Test Plan
Real users use real browsers which are downloading images, CSS and JavaScript files using parallel thread pool (around 5 concurrent threads) to speed up page loading process. In order to mimic this behavior you need to configure HTTP Request samplers to download embedded resources and do it in parallel. You can apply this configuration to all the HTTP Request samplers via HTTP Request Defaults.
Don't forget to add HTTP Cache Manager to mimic browser cache as real browsers normally cache embedded resources to disk and don't re-request them on subsequent calls.
So I would recommend reviewing your test scenario and/or load pattern and get familiarized with How to make JMeter behave more like a real browser article
There is Jmeter response time difference between Jmeter run results and manually captured the response time from the local system using stop watch on the web application.
Browse the web app from local windows system and use stop watch to see the response time to load the page.
Run the Jmeter in non-gui/gui mode and observe the response time (used Listeners only to debug, when run the script no listener was added)
Can see there is a difference in both. Please suggest how to know if Jmeter has captured the correct response time.
Given you properly configure JMeter you should get the same or similar response time for the same request. "Proper" configuration stands for:
You should configure JMeter to retrieve embedded resources and use parallel thread pool of ~5 threads for this
This options "tells" JMeter to fetch images, styles and scripts referenced in the main HTML page and do it in parallel like real browsers do
Add HTTP Cache Manager to your Test Plan. Real browsers download embedded resources but do it only once, on subsequent requests the resources are being returned from disk cache, no actual request is being made. HTTP Cache Manager enables cache simulation and cache-control headers handling.
Add HTTP Cookie Manager to represent cookies/sessions and deal with cookie-based authentication
Add HTTP Header Manager to your test plan to represent browser headers. It might be important as i.e. missing Accept-Encoding header may disable server-side compression therefore client will receive much more data and it will take more time.
Assuming "good" JMeter configuration you should see more or less same behavior
If there are still differences - capture the requests sent by JMeter and the browser using a sniffer tool like Wireshark and amend JMeter configuration to eliminate the differences
JMeter have three basic measurements it captures per request:
Elapsed Time (which is overall timespan from the point when it just starts sending request to the last bit received)
Latency (which starts the same point in time and ends when server starts responding)
And Connect time (which is included in latency and is basically the time for handshakes with server, including SSL/TLS negotiations)
So if you set a data writer among your listeners (e.g SimpleDataWriter, although AggregateReport & SummaryReport can do it as well), you'll see these metrics in your data file (while standard listeners/visualisers/aggregators stuck to elapsed time only).
But mind that these metrics doesn't include response rendering, and especially any code to be executed by browser.
JMeter just doesn't do it at all: obviously, it measures just the combined performance of Server + Network, skipping everything on the client side (except for bare necessities, like protocol negotiations).
That might explain the difference you've experienced.
As well as the difference between logged server processing times & the response times measured by JMeter: server just doesn't count what the network brings in.
PS And you don't have to sit and click on stopwatch with your browser: modern ones have a Dev Tools capable of showing you the precision timings divided by stages. E.g., just call Ctrl+Shift+I in Chrome, switch to network tab & behold the timings right there as you doing your requests.
How do you test your caching layer? (be it Varnish, Nginx/Apache, Fastly or any other product/service). Given the fact that caching is hard I've found surprisingly small amount of information on the topic on Internet. Currently I'm using PhpUnit to write 'unit' tests that follow certain scenarios:
set up initial parameters like HTTP headers, cookies, HTTP referrer, user agent,
execute (possibly multiple) HTTP requests,
check those requests for expected result (HTTP headers, cookies, presence or absence of certain HTML code),
check for expected response times,
verify that state of client doing the HTTP request is in sync with application (testing session object contents, database etc.)
I constantly add new tests to my test case and test caching layer on each change.
I've seen people using varnishtest to test Varnish but it seems to be greatly limited in what it can do (for example you can't test backend database to confirm that some state has been set)?
What's your strategy to test caching layer?
I use tsung, that is originally a distributed load testing tool but it can do what you listed.
Tsung are piloted by xml files that describe scenario where you can set initial parameters, launch multiple clients in parallel and check for specific results.