The default behavior of JMeter definitely seems to just duplicate your test plan across servers. So, if the test plan has 10 "threads", running it against X servers will yield 10x threads.
Is there any way to make this more intelligent? For example, maybe I only want one copy of some HTTP thread running even though I have 5 servers to distribute a more intense load.
Another example...I want to ensure that my sampler uses unique IDs for each thread, but my service requires that the usernames be pre-provisioned so they can't be preprovisioned...I haven't been able to find a straightforward way to coordinate this (statelessly) across my distributed servers.
A "simple" implementation might be if JMeter had distributed testing aware variables built in so the client sent the server something like ServerID and ServerCount so that the test plan could use the numeric serverId as a prefix or mod by the server count. Alternatively, JMeter could have an option to shard thread_num so that if you say 10,000 threads and have 10 servers, it will run 1,000 threads on each server with thread_num never being duplicated across the distributed test for a given sampler (Example, skip thread_num if thread_num % serverCount != serverId).
Any thoughts on the best way to accomplish this?
One approach to have distributed test-aware variable is to start each jmeter-server with different variable value:
bin/jmeter-server -Jvariable=valuehost1
And then in your test script just use:
${__P(variable)}
Related
Jmeter tests are run in master slave fashion with around 8 slave machines. However with the remote batching mode set to MODE_STRIPPED_BATCH, I am not able to run tests for more than 64 hours. Throughput is around 450 requests per minute, and per slave machine it results in the creation of jtl files that are around 1.5 gb. All 8 slaves are going to send this to the master (1.5 gb x 8) and probably the I/O gets too much for the master to handle. The master machines memory is at 16 gb ram and has disk storage of around 250 gb. I was wondering if the jmeter distributed architecture has any provision to make long running soak tests possible without any un explained stress on the master machine. Obviously I have the option to abandon master slave setup and go for 8 independent nodes, however I'll in that case run into complications with respect to serving data csv files ( which I currently serve using simple table server plugin from the master m) and also around aggregating result files. Any suggestions please. It would be great to be able to run tests atleast for around 4 days (96 hours or so).
I would suggest to go for an independent JMeter workers + external data collector setup.
Actually, the JMeter right-out-of-the-box "distributed scaling" abilities are weak, way outdated & overall pretty ridiculous. As well as it's data collection/agregation/processing abilities.
This situation actually puzzles me a lot - mind you, rivals are even worse, so there's literally NOTHING in the field (except for, perhaps, some SaaS solutions trying to monetize on this gap).
But is is what it is...
So that's about why-s, now to how-s.
If I were you, I would:
Containerize the JMeter worker
Equip each container with a watchdog to quickly restart the worker if things go south locally (or probably even on schedule to refresh it ultimately). Be that an internal one, or external like cloud services have - doesn't matter.
Set up a timeseries database - I recommend InfluxDB, it's an excellent product & it's free in basic version (which is going to be enough for your purposes).
Flow your test results/metrics into that DB - do not collect them locally! You can do it right from your tests with pretty simple custom listener (Influx line protocol is ridiculously simple & fast), or you can have external agent watching the result files as they flow. I just suggest you not to use so called Backend Listner to do the job - it's garbage, it won't shape your data right, so you'd have to do additional ops to bring them to order.
If you shape your test result/metrics data properly, you've get 'em already time-synced into a single set - and the further processing options are amazingly powerful!
My expectation is that you're looking for the StrippedAsynch sampler sender mode.
As per the documentation:
Asynch
samples are temporarily stored in a local queue. A separate worker thread sends the samples. This allows the test thread to continue without waiting for the result to be sent back to the client. However, if samples are being created faster than they can be sent, the queue will eventually fill up, and the sampler thread will block until some samples can be drained from the queue. This mode is useful for smoothing out peaks in sample generation. The queue size can be adjusted by setting the JMeter property asynch.batch.queue.size (default 100) on the server node.
StrippedAsynch
remove responseData from successful samples, and use Async sender to send them.
So on slave node add the following line to user.properties file:
mode=StrippedAsynch
and on the master node define asynch.batch.queue.size, to be as high to not to have impact onto JMeter's throughput (won't slow it down) and as low to not to overwhelm the master. I would start with 1000.
Another option is using StrippedDiskStore but you will have to manually collect serialized results after test completion (make sure that slave processes will not shut down because the results will be deleted when slave process finishes)
You could use JMeter PerfMon Plugin to monitor memory and network usage on master and slaves.
We are creating a new hosted server for one of our APIs on managed containers (Kubernetes) and we're trying to validate that it can handle at least the same amount of traffic load requests.
We've started with one of the APIs, where we would need to handle at least 140k requests per minute, all endpoints combined.
To verify this, I created a simple JMeter test as follows:
-Test Plan
---Thread Group Endpoint1
-----HTTP Request -> a GET request with query params for /path1
---Thread Group Endpoint2
-----HTTP Request -> a GET request with query params for /path2
For a local test, I used the following setup:
Thread Groups Endpoint1 and Endpoint2 are set to 200 threads (users), ramp-up period of 1s, loop count = forever and duration 60s.
Using a Summary Report listener when running the test gets me a total of ~9300 # Samples.
Using this approach, is it safe to just increase the number of threads (users) for the Thread Groups until I reach the desired 140k requests per minute?
Note: I only used JMeter a little before, so I'm aware that the entire approach may be wrong, therefore any suggestions and steering to the right path are more than welcomed.
Your approach is viable as long as it represents real-life application usage. If it has 2 endpoints with equally/evenly distributed load - your setup is just fine. If there are more endpoints and some of them are used more than the others - consider defining the workload correspondingly either using different Thread Groups or other distribution mechanism such as Throughput Controller
Increasing the number of threads is also fine, however consider increasing the load gradually, to wit increase ramp-up time so your test could have:
Arrivals phase
Time to hold the load
Ramp-down phase
This way you will be able to correlate various metrics like increasing response time, throughput, number of errors, etc. with the increasing load. Also you will be able to state what was the number of threads/requests per second when the system reached saturation point/breaking point and does it recover when the load gets back.
Also make sure you're following JMeter Best Practices as 2300/2500 requests per second is not something JMeter can support out of the box and you will need to do some tuning, at least increase JVM Heap size allocated to JMeter.
You may not be able to achieve the desired 140k requests per minute using a single Jmeter Machine, in that case you'll need Distributed Load Testing approach here.
refer: http://jmeter.apache.org/usermanual/jmeter_distributed_testing_step_by_step.html
Also keeping the ramp-up period of 1 second will lead to spike and unrealistic load in the system which will not give proper result unless you've pre-warmed your server, you should gradually increase the load as per real/estimated traffic pattern.
I was using non-GUI mode to perform a remote load testing with Jmeter from master server (Linux) to 5 slave servers (Linux). 5x "n" users have been run, "n" users on each server.
The results have been written to master server.
There are samples from all servers in results file but they relate to the number of active users from particular servers ("n") and not from all servers (5x "n").
There are no information in the result file about the real number of active users on all the servers.
As a result, a maximum number of active users is "n" on generated graphs which does not reflect the real load (5x "n" users).
Has anyone got a similar problem?
Is there anything I can do to correct the results already gathered?
Should I change any JMeter parameter to get the correct results in the next run?
Short Answer:
This is normal and no, there's nothing in JMeter you can do to fix it.
Long Answer:
Each Load Generator creates a number of threads n: the threads will be numbered 1-n. When the Controller collects all of the information, it sees 5 results for Thread 1, 5 results for Thread 2, ... Thread n. The Controller has no way of knowing that each of them are 5 separate concurrent threads and not just the same thread 5 sequential times.
Fixing it:
It depends on what you mean by a maximum number of active users is "n" on generated graphs. If this is something inside JMeter, then no, you can't fix it.
If it's a report-generator that you have created yourself, then yes, you can fix it by passing in the number of load generators.
I'm running a distributed test using 5 JMeter clients (slaves). Each client is set to run 50 users. At the conclusion of the test I generate a series of graphs from the resulting JTL along with a SynthesisReport. The SynthesisReport details 250 samples for each request, as you'd expect, however the TimeVsThreads and the ThreadsStateOverTime peak at 50 users, as if they were showing the results from just one of the clients.
I've confirmed that the jmeter.properties files for each client are the same as I suspected that it was possibly an issue with the each clients results file configuration and settings for saveservice.
I can't imagine this is by design, has anyone experienced something similar and if so how did was it solved?
As per documentation:
http://jmeter-plugins.org/wiki/ActiveThreadsOverTime/
Just name your thread groups using a unique id for each generator ( hostname or a property you pass to injector and use with function __P if you have more than 1 injector per host) and it will work fine.
This is normal for Jmeter Distributed Testing.
The reason this occurs is that each load generator separately starts User Threads 1-50, so when cmdrunner runs, it sees 1 responses from each User Thread 1 (5 total), but can't differentiate between them.
If you're using a custom reporter tool (that wraps cmdrunner), you can multiply your peak users by your load generators to display a more accurate number at the top of your Report. But as long as you're calling cmdrunner, you won't be able to see the actual number of users on your graphs.
This is normal behavior of JMeter. 5 clients will run each 50 threads.
Open JMeter and run on all remote hosts, and check Active threads each will run 50 threads.
I have a system that should be able to handle millions of users requests concurrently. In order to check how the system handles the load, I setup a cluster of JMeter servers (slaves), and one controller (client).
I have a database of all users (~10M), and I need each request sent to be from a different user.
I am wondering how I can implement such a thing in JMeter. Basically, I thought about dividing a range of users (let's say 100,000) per each slave, and then within a given slave, each request should read a new user from the local 100,000 list, and delete it. Thus, I will eventually send a request from every user.
The thing is while this idea sounds logical theoretically, I do not exactly know how to implement it using the JMeter terms. Also, I am not sure how to read from database in the test, although I could theoretically read it in advance into a text file, and have each slave contain the text file with its 100,000 users portion.
I can setup a very large cluster of machines, so scale will not be the issue here. Just how to set it all up.
The best way to provide Jmeter with a list of parameters is to use a CSV file:
http://jmeter.apache.org/usermanual/component_reference.html#CSV_Data_Set_Config
You can configure the CSV dataset config to do make every thread use a different line in the CSV. Each engine will need to have it’s own unique CSV file, because the sharing mode does nto work between engines in distributed testing (you can try to automate this part, this can be interesting to do :) ).
This is how your script should look like:
1. Thread Group
1.1 HTTP sampler (login)
1.1.1 CSV dataset config
1.2 second http sampler
etc...
The login sampler will use the parameters loaded by from the CSV file, so for every ’login’ it will use a different line.
Distributed testing is pretty simple:
http://jmeter.apache.org/usermanual/remote-test.html
Keep in mind that running 100K concurrent users on a single Jmeter load engine will be hard (Jmeter consumes resources on the server, so you will need lots of CPU and memory). So you should also monitor the engines.
Also 1M users will cause a lot of data that the engines will send back to the console, so you might need to start a bunch of distributed tests in parallel, and at the end aggregate the results.
Cheers,
This is can be implemented by doing the following steps:
Taking the user credential dump and saving it to a csv file
split the csv file. copy 1 file to each Jmeter slave, in the same location on all the
machines e.g. "C:\Loadtest\"
from the controller, give the path of your csv file in "CSV Data Set Config".
Run the Test.
By doing the above steps, Jmeter controller will start execution of the test by pointing all the Jmeter Slave nodes to use the CSV file in the same location "C:\Loadtest\".
But the trick here is that all the machine will be using different set of users.
Hope this will help.