I'm trying to perform benchmark blocking vs non-blocking io.
As a blocking, I use spring-boot.
As a non-blocking - play framework.
I Call endpoint which makes 4 remote calls (sequentially)
Here are results:
Spring boot
Running 5m test # http://localhost:8080/remote-multiple
4 threads and 20000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 713.90ms 429.81ms 2.00s 82.16%
Req/Sec 33.04 22.55 340.00 68.84%
9602 requests in 5.00m, 201.85MB read
Socket errors: connect 15145, read 21942, write 0, timeout 2401
Requests/sec: 32.00
Transfer/sec: 688.83KB
Play framework
Running 5m test # http://localhost:9000/remote-multiple
4 threads and 20000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.40s 395.00ms 2.00s 54.73%
Req/Sec 37.97 21.21 230.00 70.71%
39792 requests in 5.00m, 846.41MB read
Socket errors: connect 15145, read 36185, write 60, timeout 35944
Requests/sec: 132.61
Transfer/sec: 2.82MB
Though Play shows higher Requests/sec, it has more errors, timeout, latency.
Can anybody pls explain, what do all those params in result mean?
Are Requests/sec - succesfull requests per second? etc
P.S.:
I run this benchmark on MBP 2013, Intel Core i7, 2.3 GHz, 16GB
If you post benchmarks : Start with a link to the actual benchmark code. It has no value without. Second : In general, testing code on the same machine is considered bad practice.
Related
We had a problem with our WCF endpoint performance, so we did a stress test to get more information regarding this.
Our environment:
LOAD BALANCER;
4 HOSTS(4vCPUS and 4Gb RAM) each;
Application Configuration
CPU Limit Application Pool: 0 (default – NO LIMIT)
Threads Per Processor Limit: 25 (default)
ASP Queue Length property: 3000
MaxPoolThreads registry entry: Not Set
Connection TImeout: 120 seconds (2 minutes)
STRESS TEST DATA
- 5000 Requests in 10 minutes
STRESS TEST RESULT
The tests results were observed using Dynatrace.
- Initially we have a response time of 24,6(which is too much time)
- As soon more requests were received, more our response time increased. Reaching up to 55,3 minutes.
WCF CONFIGURATIONS
Result Object:
WebConfig Segments configurations:
I have weird performance issues with fetch.max.message.bytes parameter in librdkafka consumer implementation (version 0.11). I run some tests using kafkacat over slow speed network link (4 Mbps) and received following results:
1024 bytes = 1.740s
65536 bytes = 2.670s
131072 bytes = 7.070s
When I started debugging protocol messages I noticed a way to high RTT values.
|SEND|rdkafka| Sent FetchRequest (v4, 68 bytes # 0, CorrId 8)
|RECV|rdkafka| Received FetchResponse (v4, 131120 bytes, CorrId 8, rtt 607.68ms)
It seems that increase of fetch.max.message.bytes value causes very high network saturation, but it carries only single message per request.
On the other hand when I try kafka-console-consumer everything runs as expected (I get throughput 500 messages per second over the same network link).
Any ideas or suggestions where to look at?
You are most likely hitting issue #1384 which is a bug with the new v0.11.0 consumer. The bug is particularly evident on slow links or with MessageSets/batches with few messages.
A fix is on the way.
I am learning Lift framework. I used project template from git://github.com/lift/lift_25_sbt.git and started server with container:start sbt command.
This template application displays just simple menu. If i use ab from apache to measure performance, its pretty bad. I am missing something fundamental to improve performance?
C:\Program Files\Apache Software Foundation\httpd-2.0.64\Apache2\bin>ab -n 30 -c
10 http://127.0.0.1:8080/
Benchmarking 127.0.0.1 (be patient).....done
Server Software: Jetty(8.1.7.v20120910)
Server Hostname: 127.0.0.1
Server Port: 8080
Document Path: /
Document Length: 2877 bytes
Concurrency Level: 10
Time taken for tests: 8.15625 seconds
Complete requests: 30
Failed requests: 0
Write errors: 0
Total transferred: 96275 bytes
HTML transferred: 86310 bytes
Requests per second: 3.74 [#/sec] (mean)
Time per request: 2671.875 [ms] (mean)
Time per request: 267.188 [ms] (mean, across all concurrent requests)
Transfer rate: 11.73 [Kbytes/sec] received
Are you running it in production mode? I found i had like 30 rps in devel, but over 250 in production mode. ( https://www.assembla.com/spaces/liftweb/wiki/Run_Modes )
as mentioned earlier, you should run Lift in production mode. This is the main key to get good performance. All templates are cached this way, and other optimizations apply.
if you want to measure something not abstract and theoretical, then you should give the JVM time to "warm up", apply it's JIT optimizations. So, you should apply ~thousand requests first and totally ignore them (must be a couple of seconds). After that, measure the real performance of an already-started server
there are some slight JVM optimizations, altrough they seem more like a hack to me, and give a boost not more than around 20%
other small hacks include serving static content with nginx, starting the application in a dedicated server instead of Simple Build Tool and such.
I'm somewhat newbie for WEB development based on JVM stack, but future project will require specifically some JVM-based WEB engine. So I started looking on some ground to make things quickly and turned to try Grails. Things looked good from the book, but beeing impressed by really long startup time (grails run-app) I decided to test how this works under load. Here it is:
test app: follow few instruction here to make it from ground (takes 2 mins assuming you already have Grails and Tomcat installed):
_http://grails.org/Quick+Start
test case (with Apache benchmark - comes with Apache httpd - _http://httpd.apache.org):
ab.exe -n 500 -c _http://localhost:8080/my-project/book/create
(Note: this is just displays 2 input fields within styled container)
hardware: Intel i5 650 (4Core*3.2GHz) 8GB Ram & Win Server 2003 x64
The result is ..
Grails: 32 Req/Sec
Total transferred: 1380500 bytes
HTML transferred: 1297500 bytes
Requests per second: 32.45 [#/sec] (mean)
Time per request: 308.129 [ms] (mean)
Time per request: 30.813 [ms] (mean, across all concurrent requests)
Transfer rate: 87.51 [Kbytes/sec] received
(Only 32 Req/Sec with 100% of CPU saturation, this is a way too below my expectations for such hardware)
... Next - i tried to compare it for example with similar dummy JSF application (i took one here: _http://www.ibm.com/developerworks/library/j-jsf2/ - look for "Source code with JAR files", there is \jsf-example2\target\jsf-example2-1.0.war inside),
test case: ab.exe -n 500 -c 10 _http://localhost:8080/jsf/backend/listing.jsp
The result is ..
JSF: 400 Req/Sec
Total transferred: 5178234 bytes
HTML transferred: 5065734 bytes
Requests per second: 405.06 [#/sec] (mean)
Time per request: 24.688 [ms] (mean)
Time per request: 2.469 [ms] (mean, across all concurrent requests)
Transfer rate: 4096.65 [Kbytes/sec] received
... And finally goes raw dummy JSP (just for reference)
Jsp: 8000 req/sec:
<html>
<body>
<% for( int i = 0; i < 100; i ++ ) { %>
Dummy Jsp <%= i %> </br>
<% } %>
</body>
</html>
Result:
Total transferred: 12365000 bytes
HTML transferred: 11120000 bytes
Requests per second: 7999.90 [#/sec] (mean)
Time per request: 1.250 [ms] (mean)
Time per request: 0.125 [ms] (mean, across all concurrent requests)
Transfer rate: 19320.07 [Kbytes/sec] received
...
Am i missing something? ... and Grails app can run much better?
PS: I tried profiling my running Grails app with VisualVM, but got endless loop of messages like ...
Profiler Agent: Redefining 100 classes at idx 0, out of total 413
...
Profiler Agent: Redefining 100 classes at idx 0, out of total 769
...
And finally app just stopped working after few mins - so, looks like profiling Grails is no the choice for good diagnose.
Update - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
First of all I have to admin, yes i need to RTFM - i.e. 'grails run-app' is not the correct way to run Grails for performance measurement. After compiling WAR and deploying it to Tomcat performance is not that awfully low - it is just low. The metrics below are for concurrency of 1 user (I just wanted to check what is MAX performance of the framework in one thread and with no heavy load) and while reading other related posts here i came to "http://stackoverflow.com/questions/819684/jsf-and-spring-performance-vs-poor-jsp-performance" and decided to check Apache Wicket mentioned there - its performance is also included.
Use case is:
- ab.exe -n 500 -c 1 _http://localhost:8080/...
- server is Tomcat7 in vFabric tcServer Dev edition with 'insight' running on background
---------------------- tcServer Plain Tomcat 7 -c 10
/Grails/book/create 77 req/sec 130 req/sec 410 req/sec
/jsf/backend/listing.jsp 133 req/sec 194 req/sec 395 req/sec
/wicket/library/ 870 req/sec 1400 req/sec 5300 req/sec
So ... anyway there is something wrong with Grails. I have made some profiling using tcServer (thanks Karthick) - it looks like it is able only to trace 'Spring-based' actions and internal stacktrace for Grails is like following (for 2 requests - note: metrics are not stable - i bet accuracy of tcServer far from beeing perfect, but can be used just for inforamtion)
Total (81ms)
Filter: urlMapping (32ms)
-> SimpleGrailsController#handleRequest (26ms)
-> Render view "/book/create" (4ms)
Render view "/layouts/main.gsp" (47ms)
Total (79ms)
Filter: urlMapping (56ms) ->
-> SimpleGrailsController#handleRequest (4ms)
-> Render view "/book/create" (38ms)
Render view "/layouts/main.gsp" (22ms)
PS: it may happen that the root cause for bad performance in Grails are underlying 'Spring' libs, will check this in more details.
Are you running it with run-app?
http://grails.org/Deployment states:
"Grails should never be deployed using the grails run-app command as this sets Grails up in "development" mode which has additional overheads. "
try deploying your sample app to tomcat. grails run-app is for development only.
in which environment did you start the app? prod? dev?
do you use scaffolding?
i've tried it on my machine (core i7-2600k). a login page with 4 input fields, dynamic layouts and some other things. i've got 525 requests per second in the slower dev environment.
Yeah this is a benchmark by someone who doesn't know alot about grails or his environment; first he is running on Windows know for being bad at resource management which is why most web serves/app serves run in Linux environments.
Second, if he is using 'ab' to benchmark, then he doesn't have his proxy cache setup because after the first hit, the remainder of the hits would be cached and the he is now benchmarking his cache from my understanding of his setup.
So this all just looks like the benchmarking of a bad setup and a poor understanding of Grails. No offense intended.
I've been developing a web-site that uses Django and MySQL; what I want to know is how many HTTP requests my server can handle serving certain pages.
I have been using siege but I'm not sure if that's a good benchmarking tool.
ab, the Apache HTTP server benchmarking tool. Many options. An example of use with ten concurrent requests:
% ab -n 20 -c 10 http://www.bortzmeyer.org/
...
Benchmarking www.bortzmeyer.org (be patient).....done
Server Software: Apache/2.2.9
Server Hostname: www.bortzmeyer.org
Server Port: 80
Document Path: /
Document Length: 208025 bytes
Concurrency Level: 10
Time taken for tests: 9.535 seconds
Complete requests: 20
Failed requests: 0
Write errors: 0
Total transferred: 4557691 bytes
HTML transferred: 4551113 bytes
Requests per second: 2.10 [#/sec] (mean)
Time per request: 4767.540 [ms] (mean)
Time per request: 476.754 [ms] (mean, across all concurrent requests)
Transfer rate: 466.79 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 22 107 254.6 24 854
Processing: 996 3301 1837.9 3236 8139
Waiting: 23 25 1.3 25 27
Total: 1018 3408 1795.9 3269 8164
Percentage of the requests served within a certain time (ms)
50% 3269
66% 4219
...
(In that case, network latency was the main slowness factor.)
ab reports itself in the User-Agent field so, in the log of the HTTP server, you'll see something like:
2001:660:3003:8::4:69 - - [28/Jul/2009:12:22:45 +0200] "GET / HTTP/1.0" 200 208025 "-" "ApacheBench/2.3" www.bortzmeyer.org
ab is a widely used benchmarking tool that comes with apache httpd
Grinder is pretty good. It lets you simulate coordinated load from several client machines, which is more meaningful than from a single machine.
There's also JMeter.
I've used httperf and it's quite easy to use. There's a peepcode screencast on how to use it as well.