I am analyzing a web application and want to predict the maximum users that application can support. Now i have the below numbers out of my load test execution
1. Response Time
2. Throughput
3. CPU
I have the application use case SLA
Response Time - 4 Secs
CPU - 65%
When i execute load test of 10 concurrent users (without Think Time) for a particular use case the average response time reaches 3.5 Seconds and CPU touches 50%. Next I execute load test of 20 concurrent users and response time reaches 6 seconds and CPU 70% thus surpassing the SLA.
The application server configuration is 4 core 7 GB RAM.
Going by the data does this suggests that the web application can support only 10 user at a time? Is there any formula or procedure which can suggest what is the maximum users the application can support.
TIA
"Concurrent users" is not a meaningful measurement, unless you also model "think time" and a couple of other things.
Think about the case of people reading books on a Kindle. An average reader will turn the page every 60 seconds, sending a little ping to a central server. If the system can support 10,000 of those pings per second, how many "concurrent users" is that? About 10,000 * 60, or 600,000. Now imagine that people read faster, turning pages every 30 seconds. The same system will only be able to support half as many "concurrent users". Now imagine a game like Halo online. Each user will be emitting multiple transactions / requests per second. In other words, user behavior matters a lot, and you can't control it. You can only model it.
So, for your application, you have to make a reasonable guess at the "think time" between requests, and add that to your benchmark. Only then will you start to approach a reasonable simulation. Other things to think about are session time, variability, time of day, etc.
Chapter 4 of the "Mature Optimization Handbook" discusses a lot of these issues: http://carlos.bueno.org/optimization/mature-optimization.pdf
Related
I am trying to simulate 5000 virtual users using Locust, with each user sending a message every 5 seconds. What are the resources needed in terms of EC2 specifications in order to achieve this with some level of concurrency.
Number of users is not so important (in my experience, at least when talking about less than a couple of Users per worker), the only thing that matters is the number of requests per second.
Because the performance depends on exactly what your tests do, it is impossible to give a hard number. But the manual gives some best-case figures on what you can do with FastHttpUser/HttpUser
https://docs.locust.io/en/stable/increase-performance.html#increase-locust-s-performance-with-a-faster-http-client
It is impossible to say what your particular hardware can handle, but in a best case scenario you should be able to do close to 5000 requests per second per core, instead of around 850 for the normal HttpUser (tested on a 2018 MacBook Pro i7 2.6GHz)
If your test plan is reasonably simple then you should be fine running at ~50% of that load.
Could any team assist me for sharing standard all J-Meter performance mark criteria while performing J-Meter Performance testing for min 1000 user .it means that how we can decide what parameter(or threshold) caused for deciding load /performance testing with route cause and proposed solution after generating html report with non-GUI with completed scenario.
Thanks
Amit G
There is no "standard", the acceptance criteria normally are being dictated by the business requirements which might differ depending on the nature of the application
for real-time trading a couple of milliseconds delay is critical, companies invest into locating their highly performing servers physically close to exchange servers because even light speed matters for their scenarios
for normal applications used by people (i.e. news portals, e-commerce websites, etc.) the accepted load time would be 2-3 seconds, if people have to wait more - most probably they will switch somewhere else and never return. Moreover, search engines tend to rank slow websites lower
for internal applications which are being used inside the company response time doesn't really matter cause people will have to use particular this application, but you still might report large response times like: "if a person who earns $18 per hour has to wait for 10 seconds for each operation and the number of operations per day is 100 and number of personnel is 3000 the organization looses $15 000 every single day or $5 475 000 a year"
So I would recommend taking the following steps:
Check for existing SLAs or NFRs, it might be the case the document, you got these 1000 users contains some information regarding maximum response time or minimum throughput (requests per unit of time)
If there are no formal acceptance criteria defined you could go for other performance testing types:
Soak testing - putting your application under the prolonged load to see whether it performs consistently and there are no obvious memory leaks
Stress testing - starting with low number of users and gradually increasing the load until errors start occurring. This way you will be able to report correlation between increasing number of users and increasing response time/number of errors, identify saturation point (the maximum number of users which application can efficiently support), bottleneck (breaking point), etc.
Do we need to adjust Throughput given by jmeter, to find out the actual tps of the system
For eg : I am getting 100 tps for concurrent 250 users. This ran for 10 hrs. Can I come to a conclusion like my software can handle 100 transactions per second. Or else do I need to do some adjustment and need to get a value. Why i am asking this because when load started, system will take sometime to perform in adequate level (warm up time). If so how to do this. Please help me to understand this.
By default JMeter sends requests as fast as it can, the main factor which are affecting TPS rate are:
number of threads (virtual users) - this you can define in Thread Group
your application response time - this is not something you can control
Ideally when you increase number of threads the number of TPS should increase by the same factor, i.e. if you have 250 users and getting 100 tps you should get 200 tps for 500 users. If this is not the case - these 500 users are beyond saturation point and your application bottleneck is somewhere between 250 and 500 users (if not earlier).
With regards to "warm up" time - the recommended approach of conducting the load is doing it gradually, this way you will allow your application to get prepared to increasing load, warm up caches, let JIT compiler/optimizer to go their work, etc. Moreover this way you will be able to correlate the increasing load with increasing/decreasing throughput, response time, number of errors, etc. while having 250 users released at once doesn't tell the full story. See
The system warmup period varies from one system to the other. Warm up period is where configurations are cached, different libraries are initialized (eg. Builder.init()) and other initial functions that usually don't happen for subsequent calls. If you study results of the load test, there is a slow period at the very beginning. For most systems, it could be as small as 5 to 10 minutes. These values could be even negligible if the test is as long as 10 hours. But then again, average calculation can be effected if the results give extremely low values at the start (it always depend on the jump from initial warming up period to normal operations).
As per jmeter configurations this thread may explain the configuration. How to exclude warmup time from JMeter summary?
I need to provide the business with a report estimating number of users (devices in this case) the system can cope with without extensive delays and errors.
Assuming each device polls-communicates with the server every 5 seconds or so would it be acceptable to multiple the number of concurrent users I stress test with by 5 to get the figure required by the business?
In general what are the best means of answering such a question considering the above factors?
I am guessing that the collision rate (making them concurrent) may well be over the ratio of 5 (the seconds it takes for the device before it asks to communicate with the server).
Any advice?
I am using JMeter to produce concurrent user/device throughput.
Edit as requested to explain further:
From an analytics point of view if each device will attempt to connect and communicate with the server every 5 seconds and we wish to receive a response within the time it is ready to re-communicate (in other words in next 4 seconds), the collision chances literally for other devices running the same software is calculated on the elapsed time between the two calls no?
I am looking for statistical analysis methodology really to find a percent to multiply the concurrent test results to a real environment.
I know it is a general question without a specific / explicit answer but more the methodology, if there is one, of how can one project the number of "active" users the system can cope with from the known "concurrent" users. I would have though that given the frequency of calls is known and that each call takes 300ms in average one could somehow project the actual users (maybe by an industry standard multiplier?)
Hellow
Which is the maximun number of virtual users that can be testet at a Jmeter distributed test? Is it possible to reach one million of virtual users?
Thak you.
It depends on may factors, technically the limit on JMeter end is very high (I think it should be 231 − 1 or 2 147 483 647 virtual users)
Nature of your application: use cases, is it more about consuming ore creating content, average request and response size, response time, etc.
Nature of your test: again, request and response size, need to use pre/post processors and assertions
Hardware specifications of your load generators
Number of load generators
So I would recommend the following approach:
Start with a single JMeter instance
Make sure you have optimal JMeter configuration and amended your test according to JMeter best practices
Make sure you have monitoring of baseline OS health metrics on that machine
Start with 1 virtual users and gradually increase the number of running users until you start running out of hardware resources (CPU or RAM or Network or Disk IO will be close to maximum)
Mind the number of active users at this stage (you can use i.e. Active Threads Over Time listener) - this is how many users you can simulate for particularly that test scenario. Note, the number might be different for other application or other test scenario.
Multiply the number you get by the number of the load generators you have - if there is > 1M - you are good to go.
If you won't be able to simulate that many users there is a workaround, but personally I don't really like it. The idea is that real users don't hammer application non-stop, they need some time to "think" between actions. Normally you should be simulating these "think times"using JMeter Timers. But if you lack load generators you can consider the following:
Given 1 virtual user needs 15 seconds to think between operations and response time of your application is 5 seconds, it means that each user will be able to execute only 3 requests per minute. So 1M of users will execute 3M requests per minute which gives us 50 000 requests per second which is also high, but more likely to be achievable.