I am running jmeter with only one thread and it is still eventually consuming all available memory. The test is running gets, heads and puts from a large csv file >3mil rows. Some of these files are large (> 1gb) but most are average sizes, regardless of that, it seems to die well before it gets to those huge files anyway.
I have all listeners disabled and running from the console, which should also reduce the overhead. I have also given the heap 10G of memory which I would think is plenty?
I have attached the jmx file if that helps:
jmx runfile
Is there some process i can run to cleanup/purge as its processing, or am I doing something wrong in the plan its self.
sample csv:
PUT,500,path_name_here,filename_1,replication=false
GET,1500,path_name_here,filename_2,allowredirect=true
GET,500,path_name_here,filename_3,allowredirect=true
You seem to be suffering from the problem described here, try switching to the HTTP Raw Request which can upload large files without loading them fully into memory.
You can install HTTP Raw Request sampler using JMeter Plugins Manager:
Related
I am testing REST API, each of which returns a 10 MB response body.
Now during load test, JMeter gives Out of Memory exception at 20 threads.
Concluded that this is due to APIs have huge response body. When other APIs with fairly low size Response Body are tested I am able to scale up to 500 Threads
Have tried all options shared under hacks to avoid Out of Memory exception:
Running in Non GUI mode
All Listeners Off - generating JTL from command line
Disabled all assertions, relying on application logs
Groovy used as scripting language in JSR223, using to emulate Pacing between requests
Heap Size increased to 12 GB
JMeter 5.4.1 and JDK used
Going for distributed load testing from multiple JMeter host machines also seems to have problem, as when I reduced No. of Threads to 10 for same APIs, Out of Memory Exception still came up.
How to effectively handle huge response body to a request made from JMeter ?
If you don't need to read or assert the response then you can reduce disk space usage, check Save response as MD5 hash in your HTTP Request Advanced tab
Save response as MD5 hash? If this is selected, then the response is not stored in the sample result. Instead, the 32 character MD5 hash of the data is calculated and stored instead. This is intended for testing large amounts of data.
I have a spring boot application which has a post endpoint which takes the request and send it to another service and get the response back and save it to mongo database and returned the response back to user. The application is deployed on embedded tomcat of spring boot. I am using jmeter to see the max response time, throughput etc.
When i ran a test from jmeter with 500 threads for 10 minutes, i got maximum time as around 3500ms.
When i repeat the test from jmeter the maximum time gets reduced to 900ms.
Again, if i run the test after a long time, the maximum again goes upto 3500ms.
I am not able to get any information regarding this behavior of tomcat.
Could you please help me with understanding this behavior of tomcat?
What do you mean by "unexpected"? Lower response time when you repeat the test can be explained by either your application implementation, like when you start load test against the application which is just deployed it's performance might not be optimal and when you repeat the test the cache is "warmed up" so you're getting better performance.
Another explanation could be JIT optimization as JVM is analyzing the pattern of your application usage and does inner improvements of the bytecode to better serve the given load pattern.
Third possible explanation is MongoDB caching, if 500 users are sending the same responses it might be the case the database stores the result sets in memory and when you repeat the test it doesn't actually access the storage but returns the results directly from the memory which is fast and cheap. Consider properly parameterizing your JMeter test so each thread (virtual user) would use its own credentials and perform different query than the other thread(s), but keep in mind that the test needs to be repeatable so don't use unique data each time, it's better to have sufficient set of pre-defined test data
Jmeter tests are run in master slave fashion with around 8 slave machines. However with the remote batching mode set to MODE_STRIPPED_BATCH, I am not able to run tests for more than 64 hours. Throughput is around 450 requests per minute, and per slave machine it results in the creation of jtl files that are around 1.5 gb. All 8 slaves are going to send this to the master (1.5 gb x 8) and probably the I/O gets too much for the master to handle. The master machines memory is at 16 gb ram and has disk storage of around 250 gb. I was wondering if the jmeter distributed architecture has any provision to make long running soak tests possible without any un explained stress on the master machine. Obviously I have the option to abandon master slave setup and go for 8 independent nodes, however I'll in that case run into complications with respect to serving data csv files ( which I currently serve using simple table server plugin from the master m) and also around aggregating result files. Any suggestions please. It would be great to be able to run tests atleast for around 4 days (96 hours or so).
I would suggest to go for an independent JMeter workers + external data collector setup.
Actually, the JMeter right-out-of-the-box "distributed scaling" abilities are weak, way outdated & overall pretty ridiculous. As well as it's data collection/agregation/processing abilities.
This situation actually puzzles me a lot - mind you, rivals are even worse, so there's literally NOTHING in the field (except for, perhaps, some SaaS solutions trying to monetize on this gap).
But is is what it is...
So that's about why-s, now to how-s.
If I were you, I would:
Containerize the JMeter worker
Equip each container with a watchdog to quickly restart the worker if things go south locally (or probably even on schedule to refresh it ultimately). Be that an internal one, or external like cloud services have - doesn't matter.
Set up a timeseries database - I recommend InfluxDB, it's an excellent product & it's free in basic version (which is going to be enough for your purposes).
Flow your test results/metrics into that DB - do not collect them locally! You can do it right from your tests with pretty simple custom listener (Influx line protocol is ridiculously simple & fast), or you can have external agent watching the result files as they flow. I just suggest you not to use so called Backend Listner to do the job - it's garbage, it won't shape your data right, so you'd have to do additional ops to bring them to order.
If you shape your test result/metrics data properly, you've get 'em already time-synced into a single set - and the further processing options are amazingly powerful!
My expectation is that you're looking for the StrippedAsynch sampler sender mode.
As per the documentation:
Asynch
samples are temporarily stored in a local queue. A separate worker thread sends the samples. This allows the test thread to continue without waiting for the result to be sent back to the client. However, if samples are being created faster than they can be sent, the queue will eventually fill up, and the sampler thread will block until some samples can be drained from the queue. This mode is useful for smoothing out peaks in sample generation. The queue size can be adjusted by setting the JMeter property asynch.batch.queue.size (default 100) on the server node.
StrippedAsynch
remove responseData from successful samples, and use Async sender to send them.
So on slave node add the following line to user.properties file:
mode=StrippedAsynch
and on the master node define asynch.batch.queue.size, to be as high to not to have impact onto JMeter's throughput (won't slow it down) and as low to not to overwhelm the master. I would start with 1000.
Another option is using StrippedDiskStore but you will have to manually collect serialized results after test completion (make sure that slave processes will not shut down because the results will be deleted when slave process finishes)
You could use JMeter PerfMon Plugin to monitor memory and network usage on master and slaves.
JMeter machines with versions: 2.13 r13365067, 2.11.20140918 |
Java: OpenJDK 1.7.0_79 |
OS: Debian 8.1
I'm having a problem where some HTTP requests seem to be processed far too long on a load injector that isn't really under load.
Examples from result files from tests with 20 vUs (with caching, on weaker load injector, JMeter v2.11) and 40vUs (without caching, on much higher spec'd load injector, JMeter v2.13):
<time_stamp>,3257,<request_name>,200,<thread_name>,true,28537,20,20,437
<time_stamp>,5158,<request_name>,304,<thread_name>,true,138,40,40,0
Memory is at 75% in the first case, and below 50% in the second. CPU doesn't seem to spike (measured in 1 sec intervals) and goes up to 20% max in both examples.
Checked the JVM's garbage collection, and it doesn't seem like the GC is at its limits at the time of the requests either (actually at no point during the test).
I noticed this in the case where I had caching (via Cache Manager with "Use Cache-Control/Expires headers..." checked) enabled, and, like in the second example above, get the unrealistic response time of 5158 ms.
This only happens at some steps during an iteration and to more than one thread, but not all.
It seems like JMeter is somehow processing the result too long, but I can't really see that my load injectors are under heavy load, to cause processing times of seconds.
Clearly this is messing up the performance statistics so I would like to know how this is happening.
Hope someone can help.
EDIT:
#First example: Case where ResponseTime >> Latecy > 0, happens on both JMeter machines (JMeter v2.11, JMeter v2.13).
#Second example: Case where ResponseTime >> Latecy = 0 happens only on the machine with JMeter v2.13.
2nd EDIT:
Turns out it doesn't matter what JMeter version I run (or on which node).
Regex'd my result file:
Of the same requested resources, cached (latency=0), with header check, about 10% took 1 second or multiple seconds. Without header check it is 6%.
You should run same JMeter version on all nodes. If this won't solve the problem, monitor your JMeter instance resource utilisation with jconsole.
I have a system that should be able to handle millions of users requests concurrently. In order to check how the system handles the load, I setup a cluster of JMeter servers (slaves), and one controller (client).
I have a database of all users (~10M), and I need each request sent to be from a different user.
I am wondering how I can implement such a thing in JMeter. Basically, I thought about dividing a range of users (let's say 100,000) per each slave, and then within a given slave, each request should read a new user from the local 100,000 list, and delete it. Thus, I will eventually send a request from every user.
The thing is while this idea sounds logical theoretically, I do not exactly know how to implement it using the JMeter terms. Also, I am not sure how to read from database in the test, although I could theoretically read it in advance into a text file, and have each slave contain the text file with its 100,000 users portion.
I can setup a very large cluster of machines, so scale will not be the issue here. Just how to set it all up.
The best way to provide Jmeter with a list of parameters is to use a CSV file:
http://jmeter.apache.org/usermanual/component_reference.html#CSV_Data_Set_Config
You can configure the CSV dataset config to do make every thread use a different line in the CSV. Each engine will need to have it’s own unique CSV file, because the sharing mode does nto work between engines in distributed testing (you can try to automate this part, this can be interesting to do :) ).
This is how your script should look like:
1. Thread Group
1.1 HTTP sampler (login)
1.1.1 CSV dataset config
1.2 second http sampler
etc...
The login sampler will use the parameters loaded by from the CSV file, so for every ’login’ it will use a different line.
Distributed testing is pretty simple:
http://jmeter.apache.org/usermanual/remote-test.html
Keep in mind that running 100K concurrent users on a single Jmeter load engine will be hard (Jmeter consumes resources on the server, so you will need lots of CPU and memory). So you should also monitor the engines.
Also 1M users will cause a lot of data that the engines will send back to the console, so you might need to start a bunch of distributed tests in parallel, and at the end aggregate the results.
Cheers,
This is can be implemented by doing the following steps:
Taking the user credential dump and saving it to a csv file
split the csv file. copy 1 file to each Jmeter slave, in the same location on all the
machines e.g. "C:\Loadtest\"
from the controller, give the path of your csv file in "CSV Data Set Config".
Run the Test.
By doing the above steps, Jmeter controller will start execution of the test by pointing all the Jmeter Slave nodes to use the CSV file in the same location "C:\Loadtest\".
But the trick here is that all the machine will be using different set of users.
Hope this will help.