I am hitting two straightforward application URLs from jmeter using 100 threads via HTTP requests and using response assertion (200 OK). No cookie handler or nothing else at all used in jmeter.
Then using the same machines, I use apache wrk tool (developer asked so) with parameters set to -c100 and -t100. I get throughput difference. The throughput of my application is almost 60 to 80% shown higher via apache wrk compared to jmeter. How do I troubleshoot the difference? Developer is sticking to wrk but I want to resolve the difference. I can provide more info if required but I am not using anything special in jmeter as such.
From my experience wrk is faster, it can deliver as much as up to million requests from single node, quite beefy one though (32 cores with 2 network cards), and it's not apache BTW (while jmeter is). -t100 is probably overkill since your test machine probably has not that much cores. And regarding both throughput and latencies there is wrk2 tool which gives more correct results for general purpose web services (where the load is not coordinated)
Related
We have an application which has HTTP Triggers associated with Eventhubs which means it has the capability to accept multiple HTTP requests /sec => more than 1 million/sec requests. It can take request from multiple API HTTP post requests. We are looking to test this application. Jmeter open supports multiple APIs but its not scalable to have 1 million requests/sec.
Are there any tools which support the event hub testing for such a Load?
Even the fastest load testing tool can achieve ~65K RPS because of the PORT limit. You can use a commercial solution or you can build your own distributed load testing infra. Locust and JMeter have distributed load testing support. But they are the poorest ones in terms of performance. So you will need lots of machines.
What do you mean by "not scalable to have 1 million requests/sec."? JMeter can kick off as many as 2147483647 threads and whether you can have million request per second will depend mainly on your application response time (see article for more details)
Of course you need to follow JMeter Best Practices.
If you don't have a machine which is powerful enough to generate 1 million requests per second be aware that you can run JMeter in distributed mode so the load would come from several machines.
When I run a JMeter Test Plan (jmx file) which comprises a single Thread Group that makes a SOAP request running 50 threads using a LoadRunner Controller the test runs perfectly, and I can achive 45 TPS consistently. This validates that the JMeter test plan and associated property files is uses are working perfectly under LoadRunner.
When I run the same test with JMeter the throughput drops over the course of the test run, reaching maybe 5 TPS after 15 minutes.
JMeter has maximum resources, and the developers have confirmend that the backend is processing as expected, the only difference seems to be how JMeter handles the requests, and how LoadRunner does as they use the same JMeter jmf file, data files, and property files for each test.
Can anybody suggest what the limiting factor can be for JMeter as LoadRunner runs the same test without suffering from decreasing throughut when everything else is identical test wise.
Thanks in advance,
NWC
Well-behaved JMeter doesn't have any limitations, just make sure to:
Follow JMeter Best Practices
Make sure that JMeter has enough headroom to operate in terms of CPU, RAM, etc. as if there is a lack of hardware/software resources JMeter will not be able to send requests fast enough. If you don't have any better monitoring solution you can consider using JMeter PerfMon Plugin
Pay attention to JVM metrics as well, ideally heap occupation should be between 40 and 70% otherwise you will suffer from too frequent/long GC events/chunking
Check out Concurrent, High Throughput Performance Testing with JMeter article for example JMeter-side performance bottlenecks analysis and JMeter tuning tips
I need to test 200.000 VU hitting an app in 10 seconds, so I started to make a test of 10.000 VU, running Jmeter in Non-GUI mode, to see the response of my computer, my internet connection and the site response, but I got 83.50% of Errors.
95% of the errors were these:
Non HTTP response code: java.net.ConnectException/Non HTTP response message: Connection timed out: connect
This means that the internet connection was not enough for the short time of the test?
Thanks.
Running 200K users
Generally speaking in traditional HTTP running 200.000 users from one machine is impossible: there isn't that many ports. I.e. if you maximize your port usage (and it's likely you need to change OS settings to do that, since usually OS will limit number of open ports to somehwere between 1000 and 10000), JMeter will have about 64500 ports to run requests on. Each JMeter HTTP sampler needs a separate port, so you need 200K ports. Thus you need to have at least 4 machines to run 200K requests concurrently.
But that may not be enough: if you have more than one request sequentially (like most performance tests do), you will be able to run even less concurrent requests, since ports are usually not closed right away after request is done, so next request has to use a different port.
Don't forget that server also must be able to receive similar load.
But even that may not be enough: JMeter needs to have enough memory to accommodate 10-30K threads. Size of thread in memory will depend on a few things, and how your script is designed among them.
Bottom line: with all the tweaking, realistically, port availability limits number of concurrent requests JMeter can run from one machine to 10-30K concurrent users. Thus to test 200K users, you need about 7-20 JMeter machines.
Running 10K users
If you were testing in a designated environment (where clients and servers are next to each other with optimized network between them), you should be able to run 10K users from one machine, if other limits, e.g. memory and max ports were properly tweaked. But sounds like you are trying to test them over the internet connection?
Well, 2 problems here:
Performance testing over internet connection is absolutely pointless. You don't know what is between you and servers, and how those things in between are changing the shape of the load. You won't know if it was 10K concurrent requests, or 10K sequential requests. And results will only tell you how fast your internet is.
Any ISP will have a limit on number of connections from one IP, and it will be well below 10K. Not to mention that some ISPs may flag / temporary ban your IP for such flood.
Bottom line: whoever asked you to test 10K or 200K concurrent users, should also provide a set of JMeter machines to run this test from. Those machines should be close to tested servers, preferably without any extra routing in between (or with well known and well configured routing)
I don't think that stressing your application by kicking off 200k users at once is a good idea (same applies to 10k users) as the results, even in case of success, won't tell the full story. Moreover, in case of error you will be able to state only that 10k users in 10 seconds is not possible, however you won't have the information like:
What was the number of users when errors start occurring
What is the correlation between number of concurrent users and response time and/or throughput
What is the saturation point (the maximum system performance)
So I would recommend re-running your test and increasing the load gradually from one virtual user to 10 000 and see when it breaks. The breaking point is called bottleneck and the cause can be determined like:
First of all make sure you're following JMeter Best Practices as default JMeter configuration is not suitable for high loads and if JMeter is not capable of sending requests fast enough you will not get accurate results. Most probably you will have to run JMeter in Distributed mode, it is highly unlikely you will be able to mimic 20k requests per second from a single machine (or it has to be a very powerful one)
Set up monitoring of the application under test in order to ensure that it has enough headroom in terms of CPU, RAM, Disk, etc. You can use JMeter PerfMon Plugin for this
Check your application infrastructure, like JMeter the majority of middleware components like web/application servers, load balancers, databases, etc. default configurations are suitable for development and debugging, they need to be tuned for high throughput.
Check your application code using profiler tools telemetry, the reason could be in i.e. slow DB query, inefficient algorithm, large object, heavy function, etc.
I would like to know which are the factors i need to consider when using jmeter, Most of the time internet speed will be varying and due to which i don't get accurate response time, and operations on server side [CPU utilization, etc].
Do I need to consider all this points when calculating performance of the application.
In regards to "internet speed vary", JMeter is smart enough to detect it and report as Latency metric. As per JMeter glossary:
Latency.
JMeter measures the latency from just before sending the request to just after the first response has been received. Thus the time includes all the processing needed to assemble the request as well as assembling the first part of the response, which in general will be longer than one byte. Protocol analysers (such as Wireshark) measure the time when bytes are actually sent/received over the interface. The JMeter time should be closer to that which is experienced by a browser or other application client
So you should be able to subtract latency from the overall response time and calculate the time, required to process your request on the server side. However it will be much better if JMeter load generators will live in the same intranet. If you need to test your application behavior when virtual users are sitting on different network types it can also be simulated
In regards to other factors that matter:
Application under test health. You should be monitoring baseline server-side health metrics to identify whether application server(s) gets overloaded during the load test as i.e. if you see high response times the reason could be as simple as a lack of free RAM or slow hard-drive or whatever.
JMeter load generator(s) health. The same approach should be applicable to JMeter engine(s). If JMeter hosts are overloaded they cannot generate the requests and send them fast enough which will be reported as reduced throughput.
You can use PerfMon JMeter Plugin for both. See How to Monitor Your Server Health & Performance During a JMeter Load Test article for detailed description of the plugin installation and usage
Your tests need to be realistic and represent virtual user as close to real one as possible. So make sure you:
Add Timers to your test plan to represent "think time"
If you are testing web-based application consider adding and properly configuring HTTP Cookie Manager, Header Manager and Cache Manager. Also don't forget to configure HTTP Request Defaults to "Retrieve all embedded resources" and use "Parallel pool" for this.
Our client asked us to run a stress test on their Web application simulating 62,000 users (threads), the test consists of 13-15 HTTP requests, with 1 second delay between every HTTP request, the test should run for 10.5 hours continuously.
I had previous experience with JMeter running up to 10,000 users, but have not tried for larger number.
Is there a limit for the number of threads that JMeter can handle, or is this limited by the hardware of test server?
agreed with Ray when you run jmeter for this much amount of thread the distributed testing is the best option, and the hardware and network shall be capable to handle this kind of load, and in case you want to do that in short go for http://blazemeter.com, they are Scalable from 1,000 to 100,000 concurrent users.
In principle it is limited by several factors including the configuration (CPU/Mem) of your JMeter machine. That said, it is a VERY large number of threads for just one JMeter. To be honest: I wouldn't run even 10000 threads on one machine. You might want to look into using JMeter distributed, see the manual (http://jmeter.apache.org/usermanual/jmeter_distributed_testing_step_by_step.pdf).
Except running script remotely, you'll need to configure your script with minimal usage of CPU and memory.
Use JMeter best practices and JMeter tuning tips to reach it