I've got a distributed JMeter setup and I want to create a Java service wrapper around it.
The test plan (JMX file) is provided by the user. Hence, I don't know the properties and config used in there. The user test plan can have multiple thread groups and potentially custom thread group plugins like Ultimate Thread Group, Arrival Thread Group, etc.
I want to get the overall maximum #threads or maximum concurrency (active threads) that can be achieved by the user test plan. Is it possible to get this value before executing the test plan? This will help me to provision servers for this test run accordingly.
Is it possible to cap the overall max concurrency achieved by the test plan? (similar to what we can do with RPS using timers) (Also, please mention options for custom thread groups like Ultimate thread group)
This is not something you can efficiently control because:
It is possible to configure JMeter to kick off extra threads if current amount is not sufficient to reach/maintain the target throughput by using i.e. Throughput Shaping Timer and Concurrency Thread Group with feedback function
Users can use properties or functions/variables to parameterize/calculate the concurrency
Users can always use JSR223 Test Elements and do something like:
ctx.getThreadGroup().addNewThread(0, ctx.getEngine())
and this is not something you can detect because there are numerous ways of invoking this function including reflection
So the maximum you could do is to rely on JVM limits, by default you have 1 megabyte of stack size per thread which gives your 1024 threads for each gigabyte of memory allocated to Java so you can either increase stack size or decrease the memory allocated to the JVM according to your "caps"
Related
We've a distributed JMeter setup as described here - How to Change JMeter´s Load During Runtime
The test plan (JMX file) is provided by the user. Hence, we don't know the property names used in there. During runtime, the user can provide property names and values that we will directly pass to JMeter setup through beanshell script.
In this setup, can we put a limit/cap on values of certain JMeter properties (which can potentially affect our provided resources) that can be changed by the user at runtime?
For eg. we don't want the total RPS of the system to cross say, 300 RPS at anytime. Or if the user has provided runtime change capability of #threads, we don't want them to exceed say, 100 on any machine at anytime.
We want to refrain from storing any user-defined property names in our system to provide such validation.
You can inject a Constant Throughput Timer or a Precise Throughput Timer or a Throughput Shaping Timer into the user-provided .jmx script and put your maximum allowed concurrency there.
Even if there will be multiple timers in the test plan JMeter will apply the throughput of the slowest one so you won't make the test too fast in case if original RPS is lower than your maximum and vice versa, no matter what RPS user will want it will never get higher than 300 RPS which you will define.
The same approach applies to the number of threads in thread group.
I have a performance test in JMeter and would like to test maximum system performance/throughput with it. So number of active threads should be increased for example while Error rate is under 2 %. I found Constant Throughput Timer, put it into Thread Group but it only pause or slow down threads. I tried define it as follows, with a property: ${__P(throughput,)}, but not sure what should be correct value for this property. I can't see how JMeter could measure system maximal performance.
There is no Out of the box solution as of JMeter 3.3, see this enhancement request:
https://bz.apache.org/bugzilla/show_bug.cgi?id=57640
Still it is possible to dynamically add threads since JMeter 3.2 (see https://bz.apache.org/bugzilla/show_bug.cgi?id=60530) in a JSR223 Test Element using JMeterContext:
ctx.getThreadGroup().addNewThread(delay, ctx.getEngine());
So based on this, you could in a JSR223 Test Element (Pre/Post Processor or Sampler) check the presence of a file in a folder of your choice named :
NUMBER_OF_THREAD_TO_ADD.txt
If present, use its name to compute number of threads and call this method.
There is no such functionality in the "vanilla" JMeter howere it is possible with plugins, check out:
Concurrency Thread Group
Throughput Shaping Timer
They can be connected together via feedback loop so Concurrency Thread Group will add more threads if needed to reach the desired number of requests per second.
You can install both the plugins (and keep them up-to-date) using JMeter Plugins Manager
I have a test plan where I have to pick 100 users from a CSV file. If I give threads count as 500 and loop count as 1, then I can see 490 are failed and only 10 are passed. But at the same time, if I give threads as 5 and loop count as 100, all the tests are passed. Aren't these same where the total number of requests are 500?
Is this because 500 threads and 5 threads?
If you start 500 threads, By default, the file is only opened once, and each thread will use a different line from the file. However the order in which lines are passed to threads depends on the order in which you execute, which may vary between iterations.
Change Your Jmeter CSV-DataSet-Config property "Recycle on EOF - True"
; Your Issue will solve
Hope This Helps!
You get different results because you apply different load pattern. JMeter acts as follows:
Each Thread Group kicks off threads (virtual users) within the ramp-up period
Each Thread starts executing Samplers upside down (or according to the Logic Controllers)
When the thread doesn't have any more samplers to execute or loops to interate it is being shut down
So reasons could be in:
Your application isn't capable of handling 500 users. Checks its logs for error details. If there are no specific errors it might be lack of hardware resources, re-run your test with JMeter PerfMon Plugin Telemetry to check the impact of the increasing load onto hardware resources consumption
Your application underlying components is not suitable for high loads. Some application and database servers come with connection limitations, low memory allocation, etc. in other words configuration suitable for development and debugging. Production deployment assumes totally different configuration therefore it needs to be inspected and amended
Your JMeter instance cannot create the required load. Like in point 2 JMeter default configuration is good for tests development, however when it comes to running load tests you need to mind some important points:
Increase JVM Heap size for JMeter
Run your test in non-GUI mode
Disable all the listeners during test run
See Reducing resource requirements chapter of JMeter's User Manual for more information
I have entered 500 to be the number of thread group and ramp up time to be 120 seconds but when the report is generated,the virtual users count is only 15 or in composite graph-active threads over time is raised to near about 12. So I am bit confuse active threads counts. Because the data(threads) or numbers that I filled in test plan before test is different and after result is different. What about scaled values in graph?and x10?Something related to threads?
Each JMeter thread representing a virtual user after initialization starts executing samplers upside down (or according to the Logic Controllers).
If thread doesn't have more samplers to execute and no more loops to iterate it's being shut down. It looks just like your case. See Max Users is Lower than Expected article for more detailed explanation and workaround.
Usually people set Loop Count to "Forever" and use Runtime Controller to so test could finish in designed time. Another option is using i.e. Ultimate Thread Group available via JMeter Plugins which provides convenient way of defining a load scenario.
My Test Plan have 3 Scenarios,
Submit Quote
Check Status
If Status Complete get Quote Details.
If I keep all these 3 scenarios in one thread then CTT is not putting constant throughput, when I remove create 2 thread 1 for Submit and other for Status and Get Details I am able to maintain CTT, but my 2nd and 3rd Scenario volume drops almost 50%. As I have dependency on 1st step I am using Thread communication and not sure how I can either get CTT in one thread or get same volume from 2 different threads.
Constant Throughput Timer is useful to shape the load where there is > 1 test iteration. If your thread group has only one iteration consider using Throughput Shaping Timer instead or play with ramp-up and thread number values. It is recommended to have only one instance of the CTT in your Test Plan as it's rather resource consuming by nature.
If your scenario assumes different throughput for different thread groups you can add a CTT per Thread Group (make sure that you select All active threads in current thread group (shared) in "Calculate Throughput based on" dropdown
If ITC test elements are throttling your test below desired throughput value consider using
__setProperty()
__property()
__threadNum()
functions to implement inter-thread communication instead. JMeter Properties are global, can be accessed from different thread groups and persist all the time while JMeter is running.
See How to use JMeter's Throughput Constant Timer for details.
If above information is not enough try to elaborate your query and include i.e. thread numbers, target throughput and what kind of information needs to be shared across thread groups.