We have a system to simulate 5000 users accessing the system concurrently in the 30-minute time span. It uses Microsoft online login accounts and authenticated users are redirected to the system.
Note: We have a limited number of user accounts for the testing. (20 user account)
User logins throughput = 5000/(30*60) = 2.78 user logins/second
Considering the limited number of user account following is planned
Use 20 user accounts to simulate 3-5 logins/second for a period of 30 minutes with throughput controlling plugins
Simulate active user in the system with direct API calls
Questions
Is this a correct way to handle the required throughput with a limited number of user accounts?
What are the other options available
In general the best practice is to have 1 to 1 mapping of JMeter thread (virtual user) to a real user. Depending on how does your application treat new logins it might not show the full picture, especially if some background activities are being triggered for each new login. Moreover, libraries, databases, operating systems, application servers, and other middleware tend to cache requests results so 1 user making 5000 calls is not the same as 5000 users making 5000 calls.
Well-behaved load test needs to simulate real usage of the website (or application or service) as close as possible so if you're testing a website that communicates with the backend using API - it's only a part of the traffic from the browser to the destination, in reality, things could be much more complicated like sending relevant headers, downloading embedded resources (remember about client-side caching as well), executing AJAX calls (when different API calls are being invoked by the same user in parallel) and so on. See How to make JMeter behave more like a real browser article for more details.
So recommendations are:
generate another 4980 test users
make sure that each virtual user's network footprint exactly matches the real browser (or other application which you're simulating) one
Related
I want to test how many users can concurrently login to the application. For that, can I create one user name and password and increase the thread to 100? or should I create 100 username and passwords and load it somehow?
Treat each JMeter thread (virtual user) as the real user using a real browser, it means:
Different credentials (can be provided via CSV Data Set Config)
Properly simulating browser network footprint including (but not limited to)
Sending and receiving headers (and cookies as special header sub-type)
Downloading embedded resources (images, scripts, styles, fonts, sounds)
Representing browser cache
Simulating AJAX requests
Mimicking real user "think time" as real users don't hammer application non-stop, they need some time to "think" between operations so consider adding appropriate Timers
Login itself won't tell you full story, you need to produce realistic workload, i.e. after logging in user should start doing something which he's normally doing
Usually concurrent users = different users, because it should be based on real case scenario
During a load test you try to emulate real-world conditions
If you want to have concurrent users in your application you should use different users because most servers won't allow same user to use your application with multiple sessions, also your application wants to (test) support multiple users so you need to load test the common scenario first.
If your application allows user to use your application with multiple sessions
, then add a second load test which test it also
Please explain the difference between concurrent user vs active Thread. Is both are same or have any differences in both.
In JMeter world a thread is a virtual user so ideally the number of active threads should be equal to the number of concurrent users.
Just keep in mind that you need to properly simulate the real user by the virtual user to wit:
Add HTTP Cookie Manager to represent browser cookies
Configure HTTP Request Defaults to download embedded resources and use concurrent pool of 6 threads for doing this
Add HTTP Cache Manager to simulate browser's cache
Add HTTP Header Manager to simulate the browser's headers
Add Timers to simulate "think times" (real users need some time to think between operations, type something into inputs, read text, etc.)
More information: How to make JMeter behave more like a real browser
This is for e-commerce project where the number of users login will be more. I have been given a benchmark 8000 concurrent users need to login and response time should be 3 minutes
#abi , hi .
Let me provide couple notes here.
Depending upon Your connection bandwidth , from my experience as performance test engineer, I'd say jmeter single instance usually holds up to 1k(1000)- 2k(2000) in best case users load.
Considering You have a requirement for 8k (8000 users) load, You need to launch jmeter in distributed mode ( master <-> slaves).
For this config setup I'd recommend to go with 1 master node and 4slaves. For that - You will need 5 machines (aws/azure, whatever) in the same sub-network.
Re more technical details on distributed setup, please take a look:
in public jmeter documentation
please also look into this step-by-step setup manual
Also, when i've been doing set-up for 10k load for one of my recent projects - I did couple notes for myself in g-doc . Let me know if it opens fine for You.
Last note, If You need to do some load/performance tests on APIs that require AUTHZ, I'd recommend to split authorize (IDP bypass) and performance scenario itself - in different thread groups. As usually IDP in DEVs/Stagings does not hold much load .
So at first You need to authorize w/o any load (1st Thread group).
And in 2nd Thread group - start calling target APIs under the test.
It depends on:
Your machine specifications (CPU, RAM, NIC card, hard drive, etc.)
The nature of your Test Plan (number of requests, size of requests/responses, number of pre/post processors, assertions, timers, etc.)
Response time of your application
So if your test is a simple GET request which returns small text response - you might simulate 10 000 of users on a mid-range modern laptop. And if your test is connected with heavy requests, large responses, file uploads, etc. - it might be 1000 users.
Make sure to follow recommendations from JMeter Best Practices
Make sure to have monitoring of resources usage of your system (CPU, RAM, Swap, etc.). You can use JMeter PerfMon Plugin for this.
Make sure that your test behaves like a real browser
Start with 1 virtual user and gradually increase the load until you reach 8000 virtual users or JMeter starts lacking resources, whatever comes the first. If you can simulate 8000 users from a single machine - you're good to go. If not - you will have to consider Distributed Testing.
Note: This question is about something that I do not understand in the documentation here:
https://developers.google.com/gmail/api/v1/reference/quota#concurrent_requests
Concurrent Requests
The Gmail API enforces a per-user concurrent request limit (in
addition to the per-user rate limit). This limit is shared by all
Gmail API clients accessing a given user and ensures that no API
client is overloading a Gmail user mailbox or their backend server.
enforces a per-user concurrent request limit (in addition to the per-user rate limit).
I do not find what is the 'per-user concurrent request' anywhere in their documentation. Whereas the per-user rate limit is found at the top in the same page.
https://developers.google.com/gmail/api/v1/reference/quota#top_of_page
The Gmail API enforces a per-user concurrent request limit (in addition to the per-user rate limit). This limit is shared by all Gmail API clients accessing a given user and ensures that no API client is overloading a Gmail user mailbox or their backend server.
The confusion here is the difference between per-user concurrent request limit and per-user rate limit
Lets say I make an app that lets users read from their Gmail account. I am going to be limited by the number of request each user can make though MY app the limit is per-user rate limit
Now lets say the user installs your app which also allows them to access their Gmail account. You are also limited to how fast the user can access the api via the per-user rate limit.
However both of our APPs and the gmail and inbox apps are are all running with the same per-user concurrent request limit concurrent meaning across all of the apps the user is using.
the per-user concurrent request limit is probably there to ensure that a developer doesn't create a number of different projects and rip data using all of them.
To my knowledge per-user concurrent request limit is not documented its a stealth limit and i have never seen anyone who has been able to nail down exactly what the numbers are for the concurrent limits in Google APIs. With the exception of the Google Analytics API which is 10000 not including the google analytics website and the official mobile apps
Example:
per-user rate limit
User number one logs in and lists all of his emails he has 10 user gets no error.
User number two logs in, he has 1000 emails and your application tries
to select them all out in 1 second. You are going to get a rate limit error for this user. You are flooding google.
However apps by other developers will be able to access the user inbox via the API
per-user concurrent
user number one is only running your application. logs in and lists all of his emails he has 10 user gets no error.
User number two is running your application and 20 other applications by other developers. Tries lists all of his emails assuming he has done this on all the applications at the same time he may end up getting an error.
These errors are user based due to the name Per-user in the name of the limit.
Example 2:
Lets look at the Google analytics API because i know the hard numbers for this api.
A user using your app can max make 100 requests over 90 seconds. (User-app based)
An application can make max 50000 requests a day. (App Based)
An all Applications can max make 10000 requests a day against a view (concurrent app based)
I want to know how much users the system I am testing can handle and I am confused which test to use in JMeter, user-based where I user specific number of threads or throughput based where I use the throughput timer and from there I calculate the number of concurrent users.
You have answered your question yourself, if you need to know how many users your system is able to handle you need to simulate real users as close as possible, it includes:
Cookies (HTTP Cookie Manager)
Cache (HTTP Cache Manager)
Headers (HTTP Header Manager)
Think Time (Timers)
Your application usage scenarios (user groups distributions)
Start with one and gradually increase the load until response time will start exceeding acceptable or errors start occurring, whatever comes the first.
See How To Make JMeter Behave More Like A Real Browser article for more information on above hints.