We are using Apache JMeter (latest version) for load tests of a high performance API. JMeter response time is too high in comparison to SoapUI (free version). SoapUI gets an average response time of 200 milliseconds while JMeter reports at least 1200 milliseconds for the same API, testing from the same machine.
This affects the calculated throughput in each application. On SoapUI I can get up to 850 tps for said API, no errors. On JMeter I'm getting 190 tps at most, no matter what.
Tests are running with 200 threads, for 120 seconds.
I have already tweaked jvm/jmeter heap mmemory, garbage collector, and so on to no avail. Nothing changes. Can anyone help with this? Thank you.
Edit: Added test setup sample images
JMeter Thread Group and Test Plan Overall Setup
SoapUI TestSuite and Overall Setup
Edit 2: test results
Jmeter test results 1105 ms
SoapUi test results 263 ms
I can only think of one possible reason: you're running JMeter in GUI mode and this is not something you should be doing as JMeter isn't capable of sending requests fast enough, JMeter's GUI mode is designed for tests development and debugging.
Your images don't tell the full story as JMeter test results are missing. It would be also beneficial to generate a HTML Reporting Dashboard and see the correlation between increasing number of virtual users and response time/throughput.
And last but not the least, JMeter can be executed in Distributed Mode and in SoapUI it's available only for LoadUI Pro
Related
When I run a JMeter Test Plan (jmx file) which comprises a single Thread Group that makes a SOAP request running 50 threads using a LoadRunner Controller the test runs perfectly, and I can achive 45 TPS consistently. This validates that the JMeter test plan and associated property files is uses are working perfectly under LoadRunner.
When I run the same test with JMeter the throughput drops over the course of the test run, reaching maybe 5 TPS after 15 minutes.
JMeter has maximum resources, and the developers have confirmend that the backend is processing as expected, the only difference seems to be how JMeter handles the requests, and how LoadRunner does as they use the same JMeter jmf file, data files, and property files for each test.
Can anybody suggest what the limiting factor can be for JMeter as LoadRunner runs the same test without suffering from decreasing throughut when everything else is identical test wise.
Thanks in advance,
NWC
Well-behaved JMeter doesn't have any limitations, just make sure to:
Follow JMeter Best Practices
Make sure that JMeter has enough headroom to operate in terms of CPU, RAM, etc. as if there is a lack of hardware/software resources JMeter will not be able to send requests fast enough. If you don't have any better monitoring solution you can consider using JMeter PerfMon Plugin
Pay attention to JVM metrics as well, ideally heap occupation should be between 40 and 70% otherwise you will suffer from too frequent/long GC events/chunking
Check out Concurrent, High Throughput Performance Testing with JMeter article for example JMeter-side performance bottlenecks analysis and JMeter tuning tips
Jmeter average response time is high compared to the load runner for the same test scenario , response time gradually increases for some samplers.
I am using multiple thread groups for different test scenarios.
all scenarios are executing successfully but average response time is not accurate(40 secs in Jmeter, 8 secs in load runner and manual execution)
how to fix this issue.
I suppose you get all embedded resources in JMeter scenario and does not get them in Load Runner. Could you check this settings?
I tried 500 users with 100 Ramp up/Hatch rate per seconds. The load generated was monitored for tomcat using JConsole.
For same set of request JMeter requests were using 100-80% CPU usage where as locust was around 60-30% CPU usage.
I wanted to get the bottle-neck for the server how many maximum request it can server.
Can i simulate same using locust I tried setting min-max-wait to 0 but no use.
I tried 1000 concurrent users in locust but the RPS was same. Using Jmeter i was able to find that the RPS more that this is supported by my server. RPS value by locust gives 160 where as JMeter gives 250 which is huge difference.
Locust can't generate load on my tomcat server.
Can any one please help me to explain what i am missing to simulate the load similar to JMeter.
Thanks in Advance
Looking into Locust Issue # 464 my expectation is that Locust doesn't consider so called "embedded resources", to wit images, scripts, styles, fonts, etc. which actually generate lots of extra asynchronous requests and produce the main load.
Check what exact requests are being sent by Locust and JMeter using either Tomcat access log or a sniffer tool like Wireshark - the requests should be the same in terms of amount/nature. Ideally they also should match the requests which are being sent by real user using real browser.
Follow #aldenpeterson-wf suggestion of setting min_wait and max_wait with 1000 ms to get rate per second
Have you looked at min and max wait? Setting them to both be 1000 (ms) would result in locust making 1 request/sec per client which seems to be exactly what you are looking to do.
I am using JMeter 2.11 for simulate 10000 thread users. I use CSV data set config to simulate 10000 user load and take 2-3 HTTP request in my test plan. All 10000 user load applied successfully by Jmeter but main problem is when my last request goes idle (as I put my Jmeter http request on home page of my website) all sessions goes timed out after some time.
I also configure "app pool's Idle time out=0" and my application's web config's "session state timeout is 20" (I also increase this time from 20 min but no way). Also Keep Alive is checked on all http request.
Please suggest me what to do for keep my last request's session alive.
JMeter threads are never idle, if thread doesn't have any more samplers to execute or no more loops to iterate it's being shut down. 10 000 users is quite a high load, it might be a problem with your JMeter configuration, i.e. it is not capable of generating and sustaining 10 000 threads. In 99% of cases jmeter.log file has enough troubleshooting information.
In the meantime few recommendations:
Upgrade to the latest version of JMeter (for the moment it's Apache JMeter 2.13)
Make sure you use 64-bit server JRE or JDK
Provide JMeter enough Java Heap space, by default it comes with 512Mb only which is not enough to simulate 10k users
Run JMeter test in non-GUI mode
Disable all Listeners during test run
See 9 Easy Solutions for a JMeter Load Test “Out of Memory” Failure for above points explained and few more tips and tricks.
I'm using Jmeter 2.13 and something interesting is happening for which I need some help. I did two test on website with settings
test 1
users: 500
ramp up time:60
result:smooth connections
test 2
users: 500
ramp up time:120
result: crashes java
All I know is the Apache java GUI crashes. I don't know how to troubleshoot what's causing the crash. I know there are some elements with GUI you can configure to observe Jmeter server health stats i.e threads, load etc.
Also, on the settings where it crashes, a single request is made every 0.24 seconds.
On test case 1 where it worked this equates to 0.12 seconds a single request is made.
If the calculations are correct, theoretically speaking, it shouldn't crash right? (because the difference in negligible)
The answer is simple: don't use GUI mode to run a JMeter test. Ever. Use GUI only for test development and debugging.
Running JMeter in non-GUI mode is fairly simple:
jmeter -n -t /path/to/your/testplan.jmx -l /path/to/resultsfile.jtl
Once test finished you can open resultsfile.jtl with your favourite listener and analyse test results.
For more JMeter-side performance tips and tricks see JMeter Performance and Tuning Tips guide.