Using the siege load testing tool, it sends a number of transactions to a server.
The output is typically something like:
** SIEGE 4.0.4
** Preparing 1 concurrent users for battle.
The server is now under siege...
Transactions: 10 hits
Availability: 100.00 %
Elapsed time: 14.31 secs
Data transferred: 0.09 MB
Response time: 1.43 secs
Transaction rate: 0.70 trans/sec
Throughput: 0.01 MB/sec
Concurrency: 1.00
Successful transactions: 10
Failed transactions: 0
Longest transaction: 4.54
Shortest transaction: 0.00
Sometimes in testing I see that the number of successful transactions plus the number of failed transactions does not equal the total number of transactions (e.g. 1000 transactions, 992 successful, 4 failed). Sometimes the total is lower, sometimes the total is higher (e.g. 1000 transactions, 998 success, 4 fail). What does this mean? What happened to these non-success and non-fail transactions?
Related
I've set up a thread group with JMS Point-to-Point sampler and it works ok with our application.
I send an xml message, using ${__UUID()} function for some fields in order to guarantee the message is unique, and expect a response with a timeout of 60s and the content should contain a simple pattern (Response Assertion).
I tried to do a simple load test sending 1000 messages but I'm confused about how I should configure the threads. I get different results with different configurations.
Case 1:
- Threads: 1000
- Ramp-up: 1
- Loop: 1
- Avg time/sample: ~80s
- Total time: 02:41
Case 2:
- Threads: 1000
- Ramp-up: 10
- Loop: 1
- Avg time/sample: ~60s
- Total time: 01:43
- Errors: 3%
Case 3:
- Threads: 1000
- Ramp-up: 100
- Loop: 1
- Avg time/sample: ~12s
- Total time: 02:13
Case 4:
- Threads: 1
- Ramp-up: 1
- Loop: 1000
- Avg time/sample: ~1.2s
- Total time: >16min
Case 5:
- Threads: 10
- Ramp-up: 1
- Loop: 100
- Avg time/sample: ~1.1s
- Total time: 02:12
Case 6:
- Threads: 100
- Ramp-up: 1
- Loop: 10
- Avg time/sample: ~7.3s
- Total time: 01:30
How I'm supposed to interpret those results? Which configuration should I use?
It depends on what you're trying to achieve, the main performance testing types are:
Load Testing - when you put your system under anticipated load and see if response times/number of transactions per second are expected in terms of non-functional requirements or SLAs. If this is the case - just configure JMeter to exactly replicate the expected usage of your application and that would be your "configuration"
Stress Testing - when you're identifying the limits of your system and looking for the bottleneck. In this case I would recommend starting from 1 thread and gradually increasing the load at the same time looking into the following charts:
Active Threads Over Time
Response Times Over Time
Transactions Per Second
Response Codes Per Second
Ideally the number of transactions per second should increase as you increase the number of users and response time should remain the same, however I expect that at some point you will see that response time will go up and transactions per second will go down - that would indicate the saturation point - the point of maximum system performance. You can record the number of active threads or requests per second at this stage and report it. Additionally you can look for the root cause of the performance problem and try to fix it.
I am doing ab testing of my application, Where I send 3000 concurrent requests and total of 10000 requests. My application is a spring boot application with actuator and I use kubernetes and docker for containerization. During the testing my actuator end points take longer time than expected, due to this my pods restart and requests start failing.
Now I have stopped liveness probe and during the test if I manually hit the actuator endpoint, I can see that it takes a lot of time to respond back and sometimes it does not even returns the result and just stucks.
I can see that each request is served within 10 millisecond by my application, as per the logs. But the AB test results are completely different. Below are the results from AB test:
Concurrency Level: 3000
Time taken for tests: 874.973 seconds
Complete requests: 10000
Failed requests: 6
(Connect: 0, Receive: 0, Length: 6, Exceptions: 0)
Non-2xx responses: 4
Total transferred: 1210342 bytes
Total body sent: 4950000
HTML transferred: 20580 bytes
Requests per second: 11.43 [#/sec] (mean)
Time per request: 262491.958 [ms] (mean)
Time per request: 87.497 [ms] (mean, across all concurrent requests)
Transfer rate: 1.35 [Kbytes/sec] received
5.52 kb/s sent
6.88 kb/s total
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 372 772.5 0 3051
Processing: 1152 226664 145414.1 188502 867403
Waiting: 1150 226682 145404.6 188523 867402
Total: 2171 227036 145372.2 188792 868447
Percentage of the requests served within a certain time (ms)
50% 188792
66% 249585
75% 295993
80% 330934
90% 427890
95% 516809
98% 635143
99% 716399
100% 868447 (longest request)
Not able to understand this behaviour, as it shows approximately only 11.43 requests are served within a second, which is very low. What could be the reason ? Also What should be the way to keep the liveness probe ?
I have below properties set in my application.properties:
server.tomcat.max-connections=10000
server.tomcat.max-threads=2000
Please look at the output of JMeter run:
TestA 20 0 0.00% 45423.30 26988 62228 60189.40 62130.85 62228.00 0.24 1.21 3.07
TestB 20 0 0.00% 245530.50 225405 260410 259775.40 260401.20 260410.00 0.06 0.29 0.51
It is all from the same test run (same period), and one throughput is 0.24 and the other is 0.06. Something wrong with JMeter?
Thanks for the input,
John
My expectation is that you're using numbers from JMeter Reporting Dashboard therefore we're looking at:
Label | #Samples| KO| %Errors| Average |Min |Max |90% |95% |99% |Throughput | Received |Sent
TestA | 20 | 0 | 0.00% | 45423.30| 26988 | 62228 |60189.40 | 62130.85 |62228.00 | 0.24 | 1.21 | 3.07
TestB | 20 | 0 |0.00% |245530.50| 225405| 260410|259775.40| 260401.20|260410.00| 0.06 |0.29 | 0.51
According to JMeter Glossary
Throughput is calculated as requests/unit of time. The time is calculated from the start of the first sample to the end of the last sample. This includes any intervals between samples, as it is supposed to represent the load on the server.
The formula is: Throughput = (number of requests) / (total time).
Looking into Average column you have 45 seconds of average response time for Test A and 245 seconds for Test B. It means that Test B was 6x times longer than the Test A therefore you have 6x times lower Throughput.
So I would recommend looking into server-side logs, version control system commits, APM tools, profiling tools, JMeter PerfMon Plugin results, etc. in order to identify why response time for Test B became 6x times worse than for Test A.
I'm benchmarking a simple hello world example in scala play framework 2.5 and in golang. Golang seems to be out performing play by a significant margin and I would like to know how I could optimize play framework to improve performance.
I'm using the following to benchmark
ab -r -k -n 100000 -c 100 http://localhost:9000/
I'm running play 2.5 in prod mode using the default configuration everywhere in my project. Can someone help me with performance tuning the play server in order to get the most performance? I read up on the default-dispatcher thread pool, but I'm not sure what settings to use for my pc. Also are there any other areas I could check that would help with performance?
here are my marchine specs
Intel(R) Xeon(R) W3670 # 3.20GHz 3.19GHz, 12.0 GM RAM, running windows 7 64-bit
Please note that I'm using sbt (clean and stage) to run the server in place in prod mode and executing the bat file found at target/universal/stage/bin/. Here is the source code for play
package controllers
import play.api.mvc._
class Application extends Controller {
def index = Action {
Ok("Hello, world!")
}
}
here are the result from the ab benchmark
ab -r -k -n 100000 -c 100 http://localhost:9000/
This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests
Server Software:
Server Hostname: localhost
Server Port: 9000
Document Path: /
Document Length: 13 bytes
Concurrency Level: 100
Time taken for tests: 1.537 seconds
Complete requests: 100000
Failed requests: 0
Keep-Alive requests: 100000
Total transferred: 15400000 bytes
HTML transferred: 1300000 bytes
Requests per second: 65061.81 [#/sec] (mean)
Time per request: 1.537 [ms] (mean)
Time per request: 0.015 [ms] (mean, across all concurrent requests)
Transfer rate: 9784.69 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 1
Processing: 0 2 1.9 1 72
Waiting: 0 2 1.9 1 72
Total: 0 2 1.9 1 72
Percentage of the requests served within a certain time (ms)
50% 1
66% 2
75% 2
80% 2
90% 3
95% 3
98% 5
99% 8
100% 72 (longest request)
here is the source code for golang
package main
import (
"fmt"
"net/http"
)
func handler(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Hello, world!")
}
func main() {
http.HandleFunc("/", handler)
http.ListenAndServe(":8080", nil)
}
here are the result from ab benchmark for golang
ab -r -k -n 100000 -c 100 http://localhost:8080/
This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests
Server Software:
Server Hostname: localhost
Server Port: 8080
Document Path: /
Document Length: 13 bytes
Concurrency Level: 100
Time taken for tests: 0.914 seconds
Complete requests: 100000
Failed requests: 0
Keep-Alive requests: 100000
Total transferred: 15400000 bytes
HTML transferred: 1300000 bytes
Requests per second: 109398.30 [#/sec] (mean)
Time per request: 0.914 [ms] (mean)
Time per request: 0.009 [ms] (mean, across all concurrent requests)
Transfer rate: 16452.48 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 1
Processing: 0 1 1.5 1 52
Waiting: 0 1 1.5 1 52
Total: 0 1 1.5 1 52
Percentage of the requests served within a certain time (ms)
50% 1
66% 1
75% 1
80% 1
90% 1
95% 2
98% 5
99% 7
100% 52 (longest request)
Thanking you in advance
Francis
UPDATE!
the following results in improved performance, but I'm still interested in other ideas that could improve performance
package controllers
import play.api.mvc._
import scala.concurrent.Future
import play.api.libs.concurrent.Execution.Implicits.defaultContext
class Application extends Controller {
def index = Action.async {
Future.successful(Ok("Hello, world!"))
}
}
benchmark results
ab -r -k -n 100000 -c 100 http://localhost:9000/
This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests
Server Software:
Server Hostname: localhost
Server Port: 9000
Document Path: /
Document Length: 13 bytes
Concurrency Level: 100
Time taken for tests: 1.230 seconds
Complete requests: 100000
Failed requests: 0
Keep-Alive requests: 100000
Total transferred: 15400000 bytes
HTML transferred: 1300000 bytes
Requests per second: 81292.68 [#/sec] (mean)
Time per request: 1.230 [ms] (mean)
Time per request: 0.012 [ms] (mean, across all concurrent requests)
Transfer rate: 12225.66 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 1
Processing: 0 1 2.2 1 131
Waiting: 0 1 2.2 1 131
Total: 0 1 2.2 1 131
Percentage of the requests served within a certain time (ms)
50% 1
66% 1
75% 1
80% 2
90% 2
95% 3
98% 5
99% 7
100% 131 (longest request)
As #marcospereira has said, Play is a relatively high-level framework that focuses on exploiting the much more advanced type-system in Scala to give you a lot of features and safety, which in turn helps you write code that is refactorable and scalable to your needs. Never the less, I've gotten great performance from it in production.
Asides from suggesting that you try running your benchmark on Linux with native socket transport, I'll repeat what #marcospereira said, to run your benchmark a couple of times without stopping your Play server. The standard deviation in your Play benchmark results seems abnormally high (averages of 1 with a SD's of 2.2), which suggests that perhaps JIT hasn't fully finished optimising your code for you yet.
Test node_redis benchmark, it show incr has more than 100000 ops/s
$ node multi_bench.js
Client count: 5, node version: 0.10.15, server version: 2.6.4, parser: hiredis
INCR, 1/5 min/max/avg/p95: 0/ 2/ 0.06/ 1.00 1233ms total, 16220.60 ops/sec
INCR, 50/5 min/max/avg/p95: 0/ 4/ 1.61/ 3.00 648ms total, 30864.20 ops/sec
INCR, 200/5 min/max/avg/p95: 0/ 14/ 5.28/ 9.00 529ms total, 37807.18 ops/sec
INCR, 20000/5 min/max/avg/p95: 42/ 508/ 302.22/ 467.00 519ms total, 38535.65 ops/sec
Then I add redis in nodejs with http server
var http = require("http"), server,
redis_client = require("redis").createClient();
server = http.createServer(function (request, response) {
response.writeHead(200, {
"Content-Type": "text/plain"
});
redis_client.incr("requests", function (err, reply) {
response.write(reply+'\n');
response.end();
});
}).listen(6666);
server.on('error', function(err){
console.log(err);
process.exit(1);
});
Use ab command to test, it only has 6000 req/s
$ ab -n 10000 -c 100 localhost:6666/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests
Server Software:
Server Hostname: localhost
Server Port: 6666
Document Path: /
Document Length: 7 bytes
Concurrency Level: 100
Time taken for tests: 1.667 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1080000 bytes
HTML transferred: 70000 bytes
Requests per second: 6000.38 [#/sec] (mean)
Time per request: 16.666 [ms] (mean)
Time per request: 0.167 [ms] (mean, across all concurrent requests)
Transfer rate: 632.85 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.3 0 2
Processing: 12 16 3.2 15 37
Waiting: 12 16 3.1 15 37
Total: 13 17 3.2 16 37
Percentage of the requests served within a certain time (ms)
50% 16
66% 16
75% 16
80% 17
90% 20
95% 23
98% 28
99% 34
100% 37 (longest request)
Last I test 'hello world', it reached 7k req/s
Requests per second: 7201.18 [#/sec] (mean)
How to profile and figure out the reason why redis in http lose some performance?
I think you have misinterpreted the result of multi_bench benchmark.
First, this benchmark spreads the load over 5 connections, while you have only one in your node.js program. More connections mean more communication buffers (allocated on a per socket basis) and better performance.
Then, while a Redis server is able to sustain 100K op/s (provided you open several connections, and/or use pipelining), node.js and node_redis are not able to reach this level. The result of your run of multi_bench shows that when pipelining is not used, only 16K op/s are achieved.
Client count: 5, node version: 0.10.15, server version: 2.6.4, parser: hiredis
INCR, 1/5 min/max/avg/p95: 0/ 2/ 0.06/ 1.00 1233ms total, 16220.60 ops/sec
This result means that with no pipelining, and with 5 concurrent connections, node_redis is able to process 16K op/s globally. Please note that measuring a throughput of 16K op/s while only sending 20K ops (default value of multi_bench) is not very accurate. You should increase num_requests for better accuracy.
The result of your second benchmark is not so surprising: you add an http layer (which is more expensive to parse than Redis protocol itself), use only 1 connection to Redis while ab tries to open 100 concurrent connections to node.js, and finally get 6K op/s, resulting in a 1.2K op/s throughput overhead compared to a "Hello world" HTTP server. What did you expect?
You could try to squeeze out a bit more performance by leveraging node.js clustering capabilities, as described in this answer.