What is the difference between open-ended and close-ended load testing?
Thread-based load-testing tool comes in which category?
Which are some examples of load testing tools in each category?
Looks like you are speaking about closed vs. open workload models.
This classification based on closed/open systems separation:
in case of closed system model a new request is only triggered by the completion of a previous request, following by a think time. System has negative feedback that makes it impossible to bury-out the service, so users wait for the responses before making new requests;
in case of open system model new requests arrival independently of completions, e.g. according to a stochastic process or fixed trace. System has no negative feedback.
Based on this classification we can divide load-testing tools into following categories based on workload model used:
thread-based model where each thread will wait for a response before sending another request - so request rate depends on response rate (load generator drives the load as fast as system can take it):
thread-based approach is VU [virtual user]-oriented while trying to simulate N users working together;
thread-based tools require a separate thread to simulate a user;
examples: JMeter;
hit-based model
hit-based approach is RPS [request per second]-oriented while trying to produce N hits per second load;
hit-based tools use async approach and will always attempt to create defined request rate, queuing requests if necessary;
examples: tsung, Yandex.Tank.
I basically agree with the explanation of closed/open system from Aliaksandr and give you an example.
For example: you have 3 users, each iterating over a sequence of requests. Before a new iteration can start, the previous iteration has to finish. The previous iteration finishes, if all requests are finished. If the System-under-Test takes longer to respond, the request/s rate drops. So load-generator and system-under-test are in a closed loop.
In open loops, the request rate is constant, regardless of the response times - there is no feedback.
Both models reveal different performance characteristics of the system under test, i.e. capacity/throughput limits with closed-loop, queue sizes with open loops. It's more easy to overload a system with an open-loop
Regarding threads, usually thread-based tools define a close-loop, but, you could model open loops as well.
Take JMeter for example, if you want a constant rate of 2 users/s over a period of 3600 seconds, you use a rampup time of 3600 seconds and use a thread number of 7200, without looping.
Gatling is another free tool that is not thread-based but event based. You can model closed-loops (repeat, during) or open-loops (constantUsersPerSecs)
Related
How to create proper workload model in JMeter if I have only number of concurrent users and response time as input/requirement? Do I need any additional information in order to load test app?
If these are only info I can get, how to approach load testing in context of using JMeter as load testing tool, any ideas - suggestions - advices?
A "proper workload" would be simulating real life usage of the system under test by "the number of concurrent users".
Just replicate real user behaviour at HTTP protocol level paying attention to pretty much everything i.e.
JMeter's network footprint must be exact replica of the browser's network footprint when it comes to HTTP requests so cross-check the number and the nature of the requests from browser developer tools and JMeter's View Results Tree listener
Make sure to send the same/proper HTTP Headers
As a subset of point 2 pay attention to Cookies
Make sure to download embedded resources like browser does
In addition to point 4 make sure to properly handle cache
Use Timers wisely to simulate think times
If your users are doing different actions configure your JMeter test to distribute them like real users are distributed
Once done just run your test with the anticipated amount of users for the desired duration and compare the real response time against the anticipated
The Avg RT and other KPIs (such as Throughput) are the result of generating or executing the workload with a determined # of VUsers. IMO, the best approach is to generate the load by varying the VUsers as shown in the graph:
This graph also shows a key concept: perfomance (as measure by KPIs) is not linear (although it might appear linear at a small # of VUsers).
While understanding concept of Concurrent thread and ultimate thread group, i am confused to understand result of summary/aggregate report while running concurrent thread or ultimate thread group .For example if i have 200 user and ramp up time 60 sec then i didn't see all sampler request as 200 sample after completing execution successfully but only few sampler request have 200 sample. when i use normal thread group then i always got thread count same for each sampler request after completing execution .
for realistic load testing with more user , could you please suggest me which thread group should prefer.
Could you please provide valuable guidance with some valuable link/book and also share me standard performance bench mark criteria or key parameter detail while doing load testing.(if any bench load parameter value does not meet standard then we can say that there is a performance issue)
Thanks for giving valuable time in advance.
Thanks
amit
This is due to the fact that:
Your application response time is too high
Your test duration is too low
For example I can see response times > 80 seconds:
it means that if a single virtual user has cumulative response time for 2 samplers > 160 seconds and the test duration is 120 seconds it will not be able to execute all the requests. Just increase your test duration to be i.e. 10 minutes and you should see more virtual users capable of executing all the Samplers you defined in the test plan.
Also given first user is capable of executing all the requests successfully and in time it looks like that your application gets overloaded hence cannot respond fast enough when the number of concurrent users reaches some "critical threshold", you can add listeners like Active Threads Over Time and Response Times vs Threads, this way you will be able to correlate increasing load with the increasing response time.
If also makes sense to collect:
Baseline health metrics of your application (CPU, RAM, Network, Disk usage, etc.), it can be done using JMeter PerfMon Plugin.
Lower level details like slowest methods, largest objects, heaviest database queries, etc. This form of information can be obtained using profiling tools specific to your application programming language(s).
I am running jmeter tests with 100 to 1k users successfully,
But I suspect if response time is higher than expected.
My test plan include: get login and post login, get test page , post test page, get search page, post search page....
in total 5 get and 5 post forms.
For same number of users if I reduce number of pages the overall response time decreases
so should I run test for pages separately?
Should I reduce number of users for realistic numbers (5 pages*20 users=100 concurrent users)--for performance test?
Or should I be using distributed system?
What is the best practice?
Current Setup: i5, 8GB Ram, One windows machine with Jmeter 3.1
You load testing should be realistic, otherwise it does not make sense. You need to simulate anticipated users behaviour as closely as possible, only this way you will get the real life picture.
So you need to implement your test scenario to match the way, your site will be used in the real world. If it is alive already you can use Access Log Sampler to replay the traffic. If it is not, you can think about "user groups", like:
50% of not authenticated users will be browsing the site
20% of authenticated users will be browsing the site
5% of users will be in login process
3% of users will search for something
etc.
You can use different Thread Groups to represent different groups of users and Throughput Controller to control the frequency of samplers execution inside the Thread Group. See Running JMeter Samplers with Defined Percentage Probability article for more information on how to distribute the load in a realistic way.
There are N number of factors that will affect the overall response times. number of users / pages etc are also few of them. Do you have any specific requirement for performance test? If you are not sure of non functional requirements, then I think all your questions will be clarified in these posts -
http://www.testautomationguru.com/jmeter-performance-testing-application-of-littles-law-to-workload-models/
http://www.testautomationguru.com/jmeter-tips-tricks-for-beginners/
I think you could be running into jmeter issues. What is the JVM heap and GC algorithm that you are using? How are you measuring the performance of each page? If you are using "Transaction Controller" to measure the performance, any slowness in jmeter will affect your transaction controller values.
Have you correlated with the application logs to really understand that the application is indeed slowing down when you decrease the # of pages? If your application response time is constant, I think jmeter is the culprit here. Move all your measurements to the sampler level instead of transaction controllers and update the results
I am trying to measure time for next button one page to another. To do this I start transaction before to press button, I press the next button , when the next page loaded I end the transaction. Between this transaction process I use web_reg_find() and check specific text to verify that page.
When I use controller that transaction measured 5 sec , then I modified transaction content and delete web_reg_find() after I measured that transaction it will be 3 sec. Is that normal ?
Because I do load test , functionality is important so transaction are also important. Is there any alternative way to check content and save the performance ?
web_reg_find() does some logic based on the response sent from the server and therefore takes time. LoadRunner is aware that this is not actual time that will be perceived by the real user and therefore reports it as "wasted time" for the transaction. If you check the log for this transaction you will see something like this:
Notify: Transaction "login" ended with "Pass" status (Duration: 4.6360 Wasted Time: 0.0062).
The time the transaction took and out of that time how much time was wasted on LoadRunner internal operations.
Note that when you will open the result in Analysis the transaction times will be reported without the wasted time (i.e. Analysis will report the time as it is perceived by the real user).
The amount of time taken for the processing of web_reg_find() also seems unusually long. As web_reg_find() is both memory and CPU bound (holding the page in ram and running string comparisons) I would look at other possibilities as to why it takes an additional two seconds. My hypothesis is that you have a resource constricted, or over subscribed load generator. Look at the performance of a control group for this type of user, 1 user loaded by itself on a load generator. Compare your control group to the behavior of the global group. If you see a deviation then this is due to a local resource constriction which is showing as slowed virtual users. This would have an impact on your measurement of response time as well.
I deliberately underload my load generators to avoid any possibility of load generator coloration plus employing a control generator in the group to measure any possible coloration.
the time which is taken by web_reg_find is calculated as waste time...
I'm trying to understand the basics of JMeter. I've got a "plus1" Java servlet that adds one to a request parameter and returns the result, so it's a fast test servlet just so I can understand load testing.
Here's my test plan:
Thread Group: 1 thread, ramp up 1 s, loop count 10000
HTTP Request to localhost
Graph Results
Summary Report
When I run this, the summary report shows a throughput number of 200/sec or so.
The key question, with no controllers in the test plan, is JMeter running the test plan (sending a single request) and waiting for the response before looping?
When I introduce a more computationally intensive page for the request, the throughput number goes down as I would expect.
In short, yes.
There is an argument for having a sampler that would make a request and not wait for the response but it's an edge case. In most cases you would want a testing tool to wait to see what happens and verify things. It's also more realistic, most users will wait for a response, in fact they generally have to, before making subsequent calls.
If you want to run a capacity test then the best approach, I think, is to spread the load over multiple threads and to actually throttle the throughput of each one - you can do this using a Constant Throughput Controller. Eg. You could have 500 threads each running at 60 requests per minute, this would give a total load of 500 reqs/sec. This way, your test load is predictable and stable - it won't be linked to the speed of response from the server. Note. with multiple threads you'll want a ramp up period and you might find you have to spread the test over multiple machines (known as 'distributed' testing if you're going to google it).