We have some outside consultants whom are using JMeter to load test our application.
Our application consists of a web site and two mobile apps. Our usual usage pattern is having 20 users and x number of mobile users whom we track in our application. Our pattern is usually steady but we have some times of the year where our load will be heavier. We do not get spikes from no activity to, say, 300 mobile users. It will rise fairly steadily and reliably. We want to make sure our scaling settings will be able to handle the next time when we start adding load.
Our partner has written scripts that go from 0 load (zero CPU %, zero bandwith use, zero SQL etc) to a massive load that then ends in about 15 minutes. THIS IS NOT OUR USAGE PATTERN. Our system cannot scale out without a little bit of time.
I told them our usage pattern but they are having trouble using JMeter to do a more steady and slow ramp up. I know very little about the tool but I imagine this is a pretty typical use case.
Any tips or ideas that could send us in the correct direction, or is JMeter just not suited to this task?
It shouldn't be rocket science, for normal JMeter Thread Group ramping up from 0 to 300 users in 15 minutes looks like:
So it's sufficient to change Ramp-up period to 1800 in order to decrease users arrival rate by factor of 2x.
For Custom Thread Groups it's even easier because you can see visually users arrival chart:
So it's a matter of simple arithmetic, it shouldn't be that hard.
Check out JMeter Ramp-Up - The Ultimate Guide for more details if needed.
I like to use the Throughput Shaping Timer JMeter plugin for this. You can specify a duration of the rampup, full load, rampdown, etc. It's pretty flexible. I like to make these values parameters so I can pass in different values from the commandline at for flexibility without having to open and edit the script between differing test runs.
https://jmeter-plugins.org/wiki/ThroughputShapingTimer/
Related
I need to load test my website with 10k req/sec for 1 hour using JMeter. I am confused with the values of loop count, number of thread, ramp-up period and duration.
Also will my laptop (i5 8GB) be able to do that? If not what is the alternative.
PS: I checked every question/answer on stackoverflow for this but I couldn't find any help. Please dont mark it repeated question.
You can use "Constant Throughput Timer" and define target throughput and select throughput based on "all active threads".
Define maximum number of users count in your script so that it will be enough for 10K req/sec.
Also if you are using windows machine then I think you will face this issue "https://www.baselogic.com/2011/11/23/solved-java-net-bindexception-address-use-connect-issue-windows/"
I will recommend to use distributed testing or use more than 1 machine.
The easiest way of configuring JMeter to send X requests per second is using either Precise Troughput Timer or Throughput Shaping Timer in combination with the Concurrency Thread Group. The number of threads needs to be sufficient, the exact number mainly depends on your application response time, if response time is 1 second - you will need 10k threads, if it's 500ms - you will need 5k threads, if it is 2 seconds - you will need 20k threads, etc.
Only you can answer whether your laptop can kick off the required number of virtual users as there are too many factors to consider: nature of the test, the size of the requests/responses, number of pre/post processors and assertions, etc. Make sure to follow JMeter Best Practices and monitor CPU, RAM, Network, etc. usage using i.e. JMeter PerfMon Plugin as if your laptop will be overloaded - JMeter won't be able to send requests fast enough and you will not be able to conduct 10k requests per second even if the server supports it. If your laptop hardware specifications are too low for the test scenario - you will have to go for Distributed Testing
You have a number of issues in play
test design. Use more than one load generator. In fact, use no fewer than three, evenly matched in hardware. Take one and load only one user of each type. This is your control set. If this set degrades at the same rate as your other load generators then you have a common issue, likely the site. If the control set does not degrade, but the other load generators do, then you likely have an overloaded generator. On the commercial test tool side of the fence, generating all load from one host have never been considered a good practice in performance testing.
10K requests per second. This is substantial. I have worked on some top 20 eCommerce sites and I can tell you that even they do not receive this type of traffic to the origin servers. Why? Cache! Either this his a Content Delivery Network where the load is spread across the county, OR there is a cache node directly in front of the load balancer(S) for the site (thing varnishcache of equivalent), OR both for a multi-staged cache. You might want to look for an objective reference in production to pin this to as a validation poinnt, if and only if (IFF) your goal is to represent end user behavior. Running a count of requests grouped by second from the HTTP access logs should be able to validate this number. Also, check the cache plan for fixed assets - it could be poorly managed and load would drop significantly just by better managing the sites cache settings to the client. If your goal is simply to saturate a SOAP/REST interface to the point of destruction then you might have a better path.
If you are looking to take a particular SOAP or REST set of remote procedure calls to the point of destruction, consider a classical stress test. Start your test at zero load, increase with the smallest step interval possible over the longest possible period of time. The physical analogy to this would be the classical hospital style stress test where a nurse comes around every minute and increases the speed OR the incline on the treadmill OR both until some end of test condition is achieved. For a hospital style test that is moving into Oxygen debt, an inability to keep pace, etc... For your application/interface it could be the doubling of response times from what is acceptable, a saturation of resources in the finite resource pool (CPU, DISK, MEMORY, NETWORK) on the back end hosts, etc...
Using JMeter I want to simulate 10000 users that will connect to a webservice. The users will not connect at the same time but they will access the service from 08:00am to 08:00pm.
I didn't find how to do that in JMeter.
In the real life the number of concurrent users will be random (we can't predict that)
So if i'll use Ramp-up period it will not work because the number of users will be increasing gradually, which i don't want to have.
Is there a way to do this scenario in JMeter ?
And another question, i'm using JMeter v5.2 and i can't find the option Scheduler in the ThreadGroup, while in many tutorials they show that option.
Given this statement:
In the real life the number of concurrent users will be random (we can't predict that)
I can only think about implementing a Stress Test.
"Normal" Load Testing is the process of putting the system under anticipated load and if you cannot figure out the current or expected load pattern you can only try to evenly distribute 10 000 users across 12 hours which gives 833 users per hour. There could be some "spikes" at the beginning of the day and after the lunch, but without knowing the business logic of your application it's hard to give a recommendation apart from generic Little's Law approach.
So you can try to identify the limits of your system you can start with 1 virtual user and gradually increase the load up to 10 000 or even more, until the errors start occurring or response time starts exceeding acceptable thresholds, whatever comes the first.
This way you will be able to tell what is the saturation point and where is the first bottleneck
I'm using jmeter to generate a performance test, to keep things short and straight i read the initial data from a json file, i have a single thread group in which after reading the data i randomize certain values to prevent data duplication when i need it, then i'm passing the final data to the endpoint using variables, this will end up in a json body that is recieved by the endpoint and it will basically generate a new transaction in the database. Also i added a constant timer to add a 7 seconds delay between requests, with a test duration of 10 minutes and no ramp up, i calculated the requests per second like this:
1 minute has 60 seconds and i have a delay of 7 seconds per request then it's logical to say that every minute i'm sending approximately 8.5 requests per minute, this is my calculation (60/7) = 8.5 now if the test lasts for 10 minutes then i multiply (8.5*10) = 85 giving me a total of 85 transactions in 10 minutes, so i should be able to see that exact same amount of transactions created in the database after the test completes.
This is true when i'm running 10-20-40 users, after the load test run i query the db and i get the exact same number of transaction however, as i increase the users in the thread group this doesn't happen anymore, for example if i set 1000 users i should be able to generate 8500 transactions in 10 minutes, but this is not the case, the db only creates around 5.1k transactions.
What is happening, what is wrong? Why it initially works as expected and as i increase the users it doesn't? I can provide more information if needed. Please help.
There could be 2 possible reasons for this:
You discovered your application bottleneck. When you add more users the application response time increases therefore throughput decreases. There is a term called saturation point which stands for the maximum performance of the system, if you go beyond this point - the system will respond slower and you will get less TPS than initially. From the application under test side you should take a look into the following areas:
It might be the case your application simply lacks resources (CPU, RAM, Network, etc.), make sure that it has enough headroom to operate using i.e. JMeter PerfMon Plugin
Your application middleware (application server, database, load balancer, etc.) are not properly set up for the high loads. Identify your application infrastructure stack and make sure to follow performance tuning guidelines for each component
It is also possible that your application code needs optimization, you can detect the most time/resource consuming functions, largest objects, slowest DB queries, idle times, etc. using profiling tools
JMeter is not sending requests fast enough
Just like for the application under test check that JMeter machine(s) have enough resources (CPU, RAM, etc.)
Make sure to follow JMeter Best Practices
Consider going for Distributed Testing
Can you please check once CPU and Memory utilization(RAM and java heap utilization) of jmeter load generator while running jemter for 1000 users? If it is higher or reaching to max then it may affect requests/sec. Also just to confirm requests/sec from Jmeter side, can you please add listener in Jmeter script to track Hit/sec or TPS?
This will also be true(8.5K requests in 10 mins test duration) if your API response time is 1 second and also you have provided enough ramp-up time for those 1000 users.
So possible reason is:
You did not provide enough ramp-up time for 1000 users.
Your API average response time is more than 1 second while you performing tests for 1000 users.
Possible workarounds:
First, try to measure the API response time for 1 user.
Then calculate accordingly that how many users you need to reach 8500 requests in 10 mins. Use this formula:
TPS* max response time in second
Give proper ramp-up time for 1000 users. Check this thread to understand how you should calculate ramp-up time.
Check that your load generator is able to generate 1000 users without any memory or health (i.e CPU usage) issues. If requires, try to use distributed architecture.
Hellow
Which is the maximun number of virtual users that can be testet at a Jmeter distributed test? Is it possible to reach one million of virtual users?
Thak you.
It depends on may factors, technically the limit on JMeter end is very high (I think it should be 231 − 1 or 2 147 483 647 virtual users)
Nature of your application: use cases, is it more about consuming ore creating content, average request and response size, response time, etc.
Nature of your test: again, request and response size, need to use pre/post processors and assertions
Hardware specifications of your load generators
Number of load generators
So I would recommend the following approach:
Start with a single JMeter instance
Make sure you have optimal JMeter configuration and amended your test according to JMeter best practices
Make sure you have monitoring of baseline OS health metrics on that machine
Start with 1 virtual users and gradually increase the number of running users until you start running out of hardware resources (CPU or RAM or Network or Disk IO will be close to maximum)
Mind the number of active users at this stage (you can use i.e. Active Threads Over Time listener) - this is how many users you can simulate for particularly that test scenario. Note, the number might be different for other application or other test scenario.
Multiply the number you get by the number of the load generators you have - if there is > 1M - you are good to go.
If you won't be able to simulate that many users there is a workaround, but personally I don't really like it. The idea is that real users don't hammer application non-stop, they need some time to "think" between actions. Normally you should be simulating these "think times"using JMeter Timers. But if you lack load generators you can consider the following:
Given 1 virtual user needs 15 seconds to think between operations and response time of your application is 5 seconds, it means that each user will be able to execute only 3 requests per minute. So 1M of users will execute 3M requests per minute which gives us 50 000 requests per second which is also high, but more likely to be achievable.
while performance testing an application, i was unable to proceed further of handling large number of threads using JMeter, so, i would like to know the max number of threads that are allowed in Jmeter, Is jmeter capable of handling 1,50,000 threads?
There is no upper limit, it strongly depends on what your test is doing, what is response size, etc.
Also keep in mind that real users don't hammer the application nonstop, they need some time to "think" between operations plus they have to wait for response before they start "thinking" about next action.
For example, given users "think" for 10 seconds and response time is 2 seconds it means that each virtual user will execute 5 requests per minute.
In above scenario 1 50 000 users will execute 7 50 000 requests per minute - which is 12 500 requests per second - > 10x times less users to simulate.
So:
first of all make sure that your JMeter configuration is optimal, default settings are good for tests development and debugging but not very good for the load test execution. You need to
tune Java parameters (Heap size, GC, etc.)
disable all listeners
make sure that you have only those assertions and post processors which are absolutely required
you store only those metrics you need and you don't save any excessive results, especially response data
See 9 Easy Solutions for a JMeter Load Test “Out of Memory” Failure article for above points comprehensive explanation and some more tips
Even given you apply above tweaks I don't think you'll be able to conduct the load of 1 50 000 users from a single host (unless you have a supercomputer in your QA Lab) so I expect you'll need to consider JMeter Distributed Testing when one master machine orchestrates multiple load generators aka "slaves" acting as a single instance - this way you will be able to increase the load by factor equal to number of slaves
Yes Of course, it depends a lot on the machine running Jmeter, but if mileage counts I can give you some hints.
JMeter allows you to run multiple processes in the same box, and it's usually pretty reliable generating up to 200-300 threads per JMeter instance. If you need more than that, I'd recommend using multiple JMeter instances
Use below link for better description how Jmeter can handle 1,50,000 threads of multiple instances
https://blazemeter.com/blog/how-run-load-test-50k-concurrent-users