I am load testing (baseline, capacity, longevity) a bunch of APIs (eg. user service, player service, etc) using JMeter. Each of these services have several endpoints (eg. create, update, delete, etc). I am trying to figure out a good way to organize my test plans in JMeter so that I can load test all of these services.
1) Is it a good idea to create a separate JMeter Test Plan (jmx) for each of the APIs rather than creating one JMeter test plan and adding thread groups like "Thread Group for User Service", "Thread Group for Player Service", etc? I was thinking about adding one test plan per API, and then adding several Thread Groups for different types of load testing (baseline, capacity, longevity, etc).
2) When JMeter calculates the Sample Time (Response Time), does it also include the time taken by the BeanShell Processors?
3) Is it a good idea to put a Listener inside of each Simple Controller? I am using JMeter Plugins for reporting. I wanted to view the reports for each endpoint.
Answers to any or all of the questions would be much appreciated :)
I am using a structure like below for creating a test plan in JMeter.
1) I like a test plan to look like a test suite. JMeter has several ways of separating components and test requirements, so it can be hard to set a rule. One test plan is likely to be more efficient than several, and can be configured to satisfy most requirements. I find there can be alot of repetition between plans, which often means maintaining the same code in different places. Better to use modules and includes on the same plan to reduce code duplication, but includes are equivalent and can be used with test fragments to reduce duplication.
Threadgroups are best used as user groups, but can be used to separate tests any way you please. Consider the scaling you need for different pages/sites. ie User/Administrator tests can be done in different Thread Groups, so you can simulate say 50 users and 2 admins testing concurrently. Or you may distinguish front-end/back-end or even pages/sites.
2) It does not include beanshell pre- and post-processing times. (But if you use a beanshell sampler, it depends on the code)
3) listeners are expensive, so fewer is better. To separate the results, you can give each sampler a different title, and the listeners/graphs can then group these as required. You can include timestamps or indexes as part of your sampler title using variables, properties and ${__javaScript}, etc. This will cause more or less grouping depending on the implementation you choose.
Related
I am using JMeter to simulate a user running our Android app. This app has several background threads that poll different services at different intervals. I'd like to test a large number of these types of users. Is there a way to do that in JMeter?
If "different intervals" come into play the only solution I can think of is using different Thread Groups for representing different services consumers as due to JMeter's threads model all Samplers inside a single Thread Group will act at a speed of the "slowest" sampler and given the application has "background" services these calls need to be non-blocking.
There are other approaches like using Throughput Controller, Weighted Switch Controller, etc. but they fall under the aforementioned limitation.
I may have figured it out. I use a combination of Parallel Controller, and nested within those are Loop Controllers with a Flow Control Action that pauses for the right number of seconds. I'll leave this question open for a bit in case anyone has a better recommendation.
See: https://github.com/Blazemeter/jmeter-bzm-plugins/blob/master/parallel/Parallel.md
I have created a test plan in Jmeter and ran it for 10 users, it has run successfully without any error, as in the below screenshot of the listeners which I have added in my test plan.
In the above listeners, how may I come to know that the values of these fields Standard Deviation, Throughput, Median, Error% calculated as expected Or is there any ideal/expected/benchmark values of the above fields through which I compare and found that my test plan work as standard. Moreover how may I able to explain that the performance of my test plan is fine/good/better or best
Please suggest me thanks
It sounds like you don't really understand what you're doing so I would recommend starting with i.e. Performance Testing Guidance for Web Applications e-book.
With regards to the "values" - we have no any idea whether the "values" match your expectations. There are no any reference "values", normally your project should have non-functional requirements or SLAs which should define maximum response time or minimum number of hits per unit of time.
Check out JMeter Glossary to learn what the "values" mean.
If you don't have NFRs or SLAs defined you still can perform a stress test like:
Make sure that your JMeter test behaves like a real browser, at least I fail to see:
HTTP Cookie Manager
HTTP Cache Manager
HTTP Header Manager
You should be running your test in command-line non-GUI mode
Start with 1 virtual user and gradually increase the load until
you see the saturation point
you start seeing performance degradation
This way you will be able to state what is the maximum number of users you system can support without issues
I have recorded my web application through template & just to confirm that load test result which i am getting is correct? Just by increasing No of users does it give proper results? Is it enough for load testing of web application?
First of all you need to ensure that your test does what it is supposed to be doing. Recorded tests can rarely be successfully replayed, so normally you should be acting as follows:
Add View Results Tree listener and run your test with 1 user. Inspect request and response details to verify your test steps.
Perform correlation and parametrization if required.
Correlation: the process of identifying and handling any dynamic parameters. Most often people use Regular Expression Extractor for it.
Parametrization: the process of making your test data driven. For example, if your application assumes multiple authenticated users you need to store the credentials somewhere. Most commonly used test element for this is CSV Data Set Config
Make your test realistic. Virtual users simulated by JMeter need to represent real users using real browsers as close as possible with all the related stuff: cookies, headers, cache, etc. See How To Make JMeter Behave More Like A Real Browser to learn how to configure JMeter to act closer to real users. Also real users need some time to "think" between operations so make sure you are using Timers to simulate this behaviour as well.
Only after you apply the above points you should add more virtual users. Again, run your test with 2-3 users and iterations to ensure your test funcitons as designed. Once you are happy with it you can increase the load, but don't overkill your server, increase the load gradually and check the impact of the increasing load on your application, i.e. how response time, throughput and number of errors change as you increase the load. The same is applicable for decreasing the load, don't turn it off at once, decrease the number of virtual users gradually.
Building a Web Test Plan
Building an Advanced Web Test Plan
I have created seven thread groups which execute different scenarios in one application. I'm trying to optimize my scripts in order to be more maintainable and easy to master when someone else uses them.
The thing that i cannot figure out is how can i combine those thread groups into one or two and to still have the seven different execution paths and the possibility to control them, by control i mean to set how many users to execute scenario 1, how many to execute scenario 2 etc. till 7.
Currently the test plan looks like this
If you don't want several thread groups for some reason the alternative options are in:
Throughput Controller - with different global executions or execution percentages
Switch Controller - which provides random weighted values (in some cases Throughput Controller doesn't guarantee that samplers in scope will ever be executed)
See Running JMeter Samplers with Defined Percentage Probability guide for more information on configuration and implementation.
Well i just figured out how to do that i have added a Loop controller a Random Order Controller as child of the loop controller. And i have put seven throughput controllers as child of the Random Order Controller so now everything looks fine
I've a simple scenario written in JMeter. Now I want to use SmartMeter instead of JMeter, but I don't know if I have to create a new scenario/test or if can reuse the old one?
I speak about http://www.smartmeter.io/
In an environment of SmartMeter Editor we can run the Test 1:1, then it is virtually identical run as in JMeter 2.12 and it does not use distributed mode. But we can watch the test in the Runner tab and after the test to generate the report (if the listener "et#sm - Controller Summary Report" is included the test).
For a distributed mode we recommend using SmartMeter Thread Group "et#sm - Distributed Lazy Stepping Thread Group", which creates users at the moment of their involvement in a testing process and they are also automatically distributed to generator servers with exact deviation between them and keeping the number of VU from the settings.
You also need to add a component of listener "et#sm - Controller Summary Report" to store the results and display informations in SmartMeter runner.
Further adjustments are voluntary, but I can only recommend them:
To use assert "et#sm - Better Response Assertion" which works much more efficiently and faster ; To retrieve values from the response exploit "et#sm - Boundary Points Extractor" etc.