I am running performance tests on an API endpoint, and I've noticed that when I use the GUI version of JMeter, I get lower response times than when I run the tests in non-GUI mode. For example, a simple request with 10 threads will generate the following:
GUI:
Samples (threads): 10
Min: 765
Max: 853
NON-GUI
Samples (threads): 10
Min: 1114
Max: 1226
I've repeated the requests several times, but the results are consistent even with higher numbers of threads (100-150). I have disabled all other threads/listeners for the non-gui mode, and in GUI mode I only use Aggregate report.
Did I miss something?
There are following possible explanations:
In Non GUI mode, injection is not parasitized by handling of GUI by Awt-thread or listeners management. So JMeter would be injecting correctly expected load and leading to degraded response times from your server
Your are load testing from a different machine in non gui mode, in this case check you are really in same configuration (jvm, jmeter, machine, network)
Related
When I run a JMeter Test Plan (jmx file) which comprises a single Thread Group that makes a SOAP request running 50 threads using a LoadRunner Controller the test runs perfectly, and I can achive 45 TPS consistently. This validates that the JMeter test plan and associated property files is uses are working perfectly under LoadRunner.
When I run the same test with JMeter the throughput drops over the course of the test run, reaching maybe 5 TPS after 15 minutes.
JMeter has maximum resources, and the developers have confirmend that the backend is processing as expected, the only difference seems to be how JMeter handles the requests, and how LoadRunner does as they use the same JMeter jmf file, data files, and property files for each test.
Can anybody suggest what the limiting factor can be for JMeter as LoadRunner runs the same test without suffering from decreasing throughut when everything else is identical test wise.
Thanks in advance,
NWC
Well-behaved JMeter doesn't have any limitations, just make sure to:
Follow JMeter Best Practices
Make sure that JMeter has enough headroom to operate in terms of CPU, RAM, etc. as if there is a lack of hardware/software resources JMeter will not be able to send requests fast enough. If you don't have any better monitoring solution you can consider using JMeter PerfMon Plugin
Pay attention to JVM metrics as well, ideally heap occupation should be between 40 and 70% otherwise you will suffer from too frequent/long GC events/chunking
Check out Concurrent, High Throughput Performance Testing with JMeter article for example JMeter-side performance bottlenecks analysis and JMeter tuning tips
We are using Apache JMeter (latest version) for load tests of a high performance API. JMeter response time is too high in comparison to SoapUI (free version). SoapUI gets an average response time of 200 milliseconds while JMeter reports at least 1200 milliseconds for the same API, testing from the same machine.
This affects the calculated throughput in each application. On SoapUI I can get up to 850 tps for said API, no errors. On JMeter I'm getting 190 tps at most, no matter what.
Tests are running with 200 threads, for 120 seconds.
I have already tweaked jvm/jmeter heap mmemory, garbage collector, and so on to no avail. Nothing changes. Can anyone help with this? Thank you.
Edit: Added test setup sample images
JMeter Thread Group and Test Plan Overall Setup
SoapUI TestSuite and Overall Setup
Edit 2: test results
Jmeter test results 1105 ms
SoapUi test results 263 ms
I can only think of one possible reason: you're running JMeter in GUI mode and this is not something you should be doing as JMeter isn't capable of sending requests fast enough, JMeter's GUI mode is designed for tests development and debugging.
Your images don't tell the full story as JMeter test results are missing. It would be also beneficial to generate a HTML Reporting Dashboard and see the correlation between increasing number of virtual users and response time/throughput.
And last but not the least, JMeter can be executed in Distributed Mode and in SoapUI it's available only for LoadUI Pro
I know load tests should be run in non-GUI mode.
But when I run the tests with the following command:
jmeter -n -t server_load_test.jmx -l log_100u_5s_01.jtl
In non-GUI mode:
I get an average response time between 3 or 4 seconds which of course is not acceptable.
In GUI mode:
I get an average response time of 100ms.
The test is really simple, it is just an HTTP request (GET) with 100 users in 5 seconds.
I would not have said anything if it was the other way around.
Which one should I trust?
My question is more: what is going on and how do I find the problem?
Non-GUI mode consumes a way less resources that GUI mode so I would recommend looking not only into Average Response Time, but also keep in mind other important metrics, for example check delivered load in both scenarios, i.e:
Active Threads Over Time and Transactions per Second listeners output (both are available via JMeter Plugins project
Generate HTML Reporting Dashboard and compare the output for both scenarios
Use 3rd-party analysis solution like JAnalyser or BM.Sense
My expectation is that in GUI mode you have much slower ramp-up hence you can run into the situation when some threads already had already finished their work while others were not yet started. In non-GUI ramp-up is faster so you have more online users so the load delivered to your application under test is much higher.
Try increasing loop count and test duration to see how it goes.
I am using JMeter 2.11 for simulate 10000 thread users. I use CSV data set config to simulate 10000 user load and take 2-3 HTTP request in my test plan. All 10000 user load applied successfully by Jmeter but main problem is when my last request goes idle (as I put my Jmeter http request on home page of my website) all sessions goes timed out after some time.
I also configure "app pool's Idle time out=0" and my application's web config's "session state timeout is 20" (I also increase this time from 20 min but no way). Also Keep Alive is checked on all http request.
Please suggest me what to do for keep my last request's session alive.
JMeter threads are never idle, if thread doesn't have any more samplers to execute or no more loops to iterate it's being shut down. 10 000 users is quite a high load, it might be a problem with your JMeter configuration, i.e. it is not capable of generating and sustaining 10 000 threads. In 99% of cases jmeter.log file has enough troubleshooting information.
In the meantime few recommendations:
Upgrade to the latest version of JMeter (for the moment it's Apache JMeter 2.13)
Make sure you use 64-bit server JRE or JDK
Provide JMeter enough Java Heap space, by default it comes with 512Mb only which is not enough to simulate 10k users
Run JMeter test in non-GUI mode
Disable all Listeners during test run
See 9 Easy Solutions for a JMeter Load Test “Out of Memory” Failure for above points explained and few more tips and tricks.
I'm using Jmeter 2.13 and something interesting is happening for which I need some help. I did two test on website with settings
test 1
users: 500
ramp up time:60
result:smooth connections
test 2
users: 500
ramp up time:120
result: crashes java
All I know is the Apache java GUI crashes. I don't know how to troubleshoot what's causing the crash. I know there are some elements with GUI you can configure to observe Jmeter server health stats i.e threads, load etc.
Also, on the settings where it crashes, a single request is made every 0.24 seconds.
On test case 1 where it worked this equates to 0.12 seconds a single request is made.
If the calculations are correct, theoretically speaking, it shouldn't crash right? (because the difference in negligible)
The answer is simple: don't use GUI mode to run a JMeter test. Ever. Use GUI only for test development and debugging.
Running JMeter in non-GUI mode is fairly simple:
jmeter -n -t /path/to/your/testplan.jmx -l /path/to/resultsfile.jtl
Once test finished you can open resultsfile.jtl with your favourite listener and analyse test results.
For more JMeter-side performance tips and tricks see JMeter Performance and Tuning Tips guide.