I'm somewhat newbie for WEB development based on JVM stack, but future project will require specifically some JVM-based WEB engine. So I started looking on some ground to make things quickly and turned to try Grails. Things looked good from the book, but beeing impressed by really long startup time (grails run-app) I decided to test how this works under load. Here it is:
test app: follow few instruction here to make it from ground (takes 2 mins assuming you already have Grails and Tomcat installed):
_http://grails.org/Quick+Start
test case (with Apache benchmark - comes with Apache httpd - _http://httpd.apache.org):
ab.exe -n 500 -c _http://localhost:8080/my-project/book/create
(Note: this is just displays 2 input fields within styled container)
hardware: Intel i5 650 (4Core*3.2GHz) 8GB Ram & Win Server 2003 x64
The result is ..
Grails: 32 Req/Sec
Total transferred: 1380500 bytes
HTML transferred: 1297500 bytes
Requests per second: 32.45 [#/sec] (mean)
Time per request: 308.129 [ms] (mean)
Time per request: 30.813 [ms] (mean, across all concurrent requests)
Transfer rate: 87.51 [Kbytes/sec] received
(Only 32 Req/Sec with 100% of CPU saturation, this is a way too below my expectations for such hardware)
... Next - i tried to compare it for example with similar dummy JSF application (i took one here: _http://www.ibm.com/developerworks/library/j-jsf2/ - look for "Source code with JAR files", there is \jsf-example2\target\jsf-example2-1.0.war inside),
test case: ab.exe -n 500 -c 10 _http://localhost:8080/jsf/backend/listing.jsp
The result is ..
JSF: 400 Req/Sec
Total transferred: 5178234 bytes
HTML transferred: 5065734 bytes
Requests per second: 405.06 [#/sec] (mean)
Time per request: 24.688 [ms] (mean)
Time per request: 2.469 [ms] (mean, across all concurrent requests)
Transfer rate: 4096.65 [Kbytes/sec] received
... And finally goes raw dummy JSP (just for reference)
Jsp: 8000 req/sec:
<html>
<body>
<% for( int i = 0; i < 100; i ++ ) { %>
Dummy Jsp <%= i %> </br>
<% } %>
</body>
</html>
Result:
Total transferred: 12365000 bytes
HTML transferred: 11120000 bytes
Requests per second: 7999.90 [#/sec] (mean)
Time per request: 1.250 [ms] (mean)
Time per request: 0.125 [ms] (mean, across all concurrent requests)
Transfer rate: 19320.07 [Kbytes/sec] received
...
Am i missing something? ... and Grails app can run much better?
PS: I tried profiling my running Grails app with VisualVM, but got endless loop of messages like ...
Profiler Agent: Redefining 100 classes at idx 0, out of total 413
...
Profiler Agent: Redefining 100 classes at idx 0, out of total 769
...
And finally app just stopped working after few mins - so, looks like profiling Grails is no the choice for good diagnose.
Update - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
First of all I have to admin, yes i need to RTFM - i.e. 'grails run-app' is not the correct way to run Grails for performance measurement. After compiling WAR and deploying it to Tomcat performance is not that awfully low - it is just low. The metrics below are for concurrency of 1 user (I just wanted to check what is MAX performance of the framework in one thread and with no heavy load) and while reading other related posts here i came to "http://stackoverflow.com/questions/819684/jsf-and-spring-performance-vs-poor-jsp-performance" and decided to check Apache Wicket mentioned there - its performance is also included.
Use case is:
- ab.exe -n 500 -c 1 _http://localhost:8080/...
- server is Tomcat7 in vFabric tcServer Dev edition with 'insight' running on background
---------------------- tcServer Plain Tomcat 7 -c 10
/Grails/book/create 77 req/sec 130 req/sec 410 req/sec
/jsf/backend/listing.jsp 133 req/sec 194 req/sec 395 req/sec
/wicket/library/ 870 req/sec 1400 req/sec 5300 req/sec
So ... anyway there is something wrong with Grails. I have made some profiling using tcServer (thanks Karthick) - it looks like it is able only to trace 'Spring-based' actions and internal stacktrace for Grails is like following (for 2 requests - note: metrics are not stable - i bet accuracy of tcServer far from beeing perfect, but can be used just for inforamtion)
Total (81ms)
Filter: urlMapping (32ms)
-> SimpleGrailsController#handleRequest (26ms)
-> Render view "/book/create" (4ms)
Render view "/layouts/main.gsp" (47ms)
Total (79ms)
Filter: urlMapping (56ms) ->
-> SimpleGrailsController#handleRequest (4ms)
-> Render view "/book/create" (38ms)
Render view "/layouts/main.gsp" (22ms)
PS: it may happen that the root cause for bad performance in Grails are underlying 'Spring' libs, will check this in more details.
Are you running it with run-app?
http://grails.org/Deployment states:
"Grails should never be deployed using the grails run-app command as this sets Grails up in "development" mode which has additional overheads. "
try deploying your sample app to tomcat. grails run-app is for development only.
in which environment did you start the app? prod? dev?
do you use scaffolding?
i've tried it on my machine (core i7-2600k). a login page with 4 input fields, dynamic layouts and some other things. i've got 525 requests per second in the slower dev environment.
Yeah this is a benchmark by someone who doesn't know alot about grails or his environment; first he is running on Windows know for being bad at resource management which is why most web serves/app serves run in Linux environments.
Second, if he is using 'ab' to benchmark, then he doesn't have his proxy cache setup because after the first hit, the remainder of the hits would be cached and the he is now benchmarking his cache from my understanding of his setup.
So this all just looks like the benchmarking of a bad setup and a poor understanding of Grails. No offense intended.
Related
We had a problem with our WCF endpoint performance, so we did a stress test to get more information regarding this.
Our environment:
LOAD BALANCER;
4 HOSTS(4vCPUS and 4Gb RAM) each;
Application Configuration
CPU Limit Application Pool: 0 (default – NO LIMIT)
Threads Per Processor Limit: 25 (default)
ASP Queue Length property: 3000
MaxPoolThreads registry entry: Not Set
Connection TImeout: 120 seconds (2 minutes)
STRESS TEST DATA
- 5000 Requests in 10 minutes
STRESS TEST RESULT
The tests results were observed using Dynatrace.
- Initially we have a response time of 24,6(which is too much time)
- As soon more requests were received, more our response time increased. Reaching up to 55,3 minutes.
WCF CONFIGURATIONS
Result Object:
WebConfig Segments configurations:
I have the next test plan in JMeter:
on the screenshot you can see the settings for the 1st ThreadGroup, wich has 50% of common amout of request in test plan (in each Thread Group are 10 different subrequests placed).
So, +1 request per second is added in average using these settings.
Then I ran this test and saw this picture (Error % column):
I save errors in file and all these errors have the same text:
<sample t="30129" lt="0" ts="1356710138314" s="false" lb="WebService(SOAP) Request 1" rc="000" rm="**Connection reset**" tn="jp#gc - Stepping Thread Group1 3-247" dt="text" by="0"/>
Server's cpu screenshot:
and for database:
After the errors have appeared my comp started work slowly and slowly (although the errors stopped to appear further)...
And in the same time the server's cpu progressively dropped to 0.
Could you tell me, please,
What is the reason of this error?
Have I reached the server timeout? (Because Max is more than 30s in the table).
UPD. I have rerun test with next settings: 1000 users per 02:46:40 (+1 Thread Group per 10 second and 10 requests inside each new Thread in the Loop).
I.e. I have reduced the time of test and total Thread Groups by 2 times, but save intensivity of Thead's adding.
The results are the same (including cpu usage on the server).
I've received the error «Connection reset» after 990 thread started. There are screenshots:
Any idea?
First, WebService(SOAP) Request is not the best way to test Webservices in JMeter, it will be deprecated in upcoming 2.9 version.
HTTP Sampler is the one to choose as it performs much better.
Second, Connection Reset means your server has cut connection. It could be coming from the CPU which seems high but it's not sure.
If what you call "my comp" is the computer hosting JMeter started working slowly then your JMeter instance is overwhelmed by the number of threads (2003 or more?) you've configured. It can come from a lot of factors, read this:
http://www.dzone.com/links/see_how_to_make_jmeter_run_thousands_of_threads_w.html
I am learning Lift framework. I used project template from git://github.com/lift/lift_25_sbt.git and started server with container:start sbt command.
This template application displays just simple menu. If i use ab from apache to measure performance, its pretty bad. I am missing something fundamental to improve performance?
C:\Program Files\Apache Software Foundation\httpd-2.0.64\Apache2\bin>ab -n 30 -c
10 http://127.0.0.1:8080/
Benchmarking 127.0.0.1 (be patient).....done
Server Software: Jetty(8.1.7.v20120910)
Server Hostname: 127.0.0.1
Server Port: 8080
Document Path: /
Document Length: 2877 bytes
Concurrency Level: 10
Time taken for tests: 8.15625 seconds
Complete requests: 30
Failed requests: 0
Write errors: 0
Total transferred: 96275 bytes
HTML transferred: 86310 bytes
Requests per second: 3.74 [#/sec] (mean)
Time per request: 2671.875 [ms] (mean)
Time per request: 267.188 [ms] (mean, across all concurrent requests)
Transfer rate: 11.73 [Kbytes/sec] received
Are you running it in production mode? I found i had like 30 rps in devel, but over 250 in production mode. ( https://www.assembla.com/spaces/liftweb/wiki/Run_Modes )
as mentioned earlier, you should run Lift in production mode. This is the main key to get good performance. All templates are cached this way, and other optimizations apply.
if you want to measure something not abstract and theoretical, then you should give the JVM time to "warm up", apply it's JIT optimizations. So, you should apply ~thousand requests first and totally ignore them (must be a couple of seconds). After that, measure the real performance of an already-started server
there are some slight JVM optimizations, altrough they seem more like a hack to me, and give a boost not more than around 20%
other small hacks include serving static content with nginx, starting the application in a dedicated server instead of Simple Build Tool and such.
Im wondering how the performance of th ESI module is nowadays? I've read some posts on the web that ESI performance on varnish were actually slower than the real thing.
Say i had a page with over 3500 esi includes, how would this perform? is esi designed for such usage?
We're using Varnish and ESI to embed sub-documents into JSON documents. Basically a response from our app-server looks like this:
[
<esi:include src="/station/best_of_80s" />,
<esi:include src="/station/herrmerktradio" />,
<esi:include src="/station/bluesclub" />,
<esi:include src="/station/jazzloft" />,
<esi:include src="/station/jahfari" />,
<esi:include src="/station/maximix" />,
<esi:include src="/station/ondalatina" />,
<esi:include src="/station/deepgroove" />,
<esi:include src="/station/germanyfm" />,
<esi:include src="/station/alternativeworld" />
]
The included resources are complete and valid JSON responses on their own. The complete list of all stations is about 1070. So when the cache is cold and a complete station list is the first request varnish issues 1000 requests on our backend. When the cache is hot ab looks like this:
$ ab -c 100 -n 1000 http://127.0.0.1/stations
[...]
Document Path: /stations
Document Length: 2207910 bytes
Concurrency Level: 100
Time taken for tests: 10.075 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 2208412000 bytes
HTML transferred: 2207910000 bytes
Requests per second: 99.26 [#/sec] (mean)
Time per request: 1007.470 [ms] (mean)
Time per request: 10.075 [ms] (mean, across all concurrent requests)
Transfer rate: 214066.18 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 1 11 7.3 9 37
Processing: 466 971 97.4 951 1226
Waiting: 0 20 16.6 12 86
Total: 471 982 98.0 960 1230
Percentage of the requests served within a certain time (ms)
50% 960
66% 985
75% 986
80% 988
90% 1141
95% 1163
98% 1221
99% 1229
100% 1230 (longest request)
$
100 rec/sec doesn't look that good but consider the size of the document. 214066Kbytes/sec oversaturates a 1Gbit interface well.
A single request with warm cache ab (ab -c 1 -n 1 ...) shows 83ms/req.
The backend itself is redis based. We're measuring a mean response time of 0.9ms [sic] in NewRelic. After restarting Varnish the first request with a cold cache (ab -c 1 -n 1 ...) shows 3158ms/rec. This means it takes Varnish and our backend about 3ms per ESI include when generating the response. This is a standard core i7 pizza box with 8 cores. I measured this while being under full load. We're serving about 150mio req/month this way with a hitrate of 0.9. These numbers suggest indeed that the ESI-includes are resolved in serial.
What you have to consider when designing a system like this is 1) that your backend is able to take the load after a Varnish restart when the cache is cold and 2) that usually your resources don't expire all at once. In case of our stations they expire every full hour but we're adding a random value of up to 120 seconds to the expiration header.
Hope that helps.
This isn't first-hand, but I'm led to believe that Varnish's current ESI implementation serialises include requests; i.e., they're not concurrent.
If that's the case, it would indeed suck for performance in the case you mention.
I'll try to get someone with first-hand experience to comment.
Parallel ESI requests are available in the **commercial** version of varnish: https://www.varnish-software.com/plus/parallel-esi/. The parallel nature of the fragment requests apparently makes the assembly of a page comprised of multiple fragments faster.
(this would be a comment but I have insufficient reputation to do that)
I've been developing a web-site that uses Django and MySQL; what I want to know is how many HTTP requests my server can handle serving certain pages.
I have been using siege but I'm not sure if that's a good benchmarking tool.
ab, the Apache HTTP server benchmarking tool. Many options. An example of use with ten concurrent requests:
% ab -n 20 -c 10 http://www.bortzmeyer.org/
...
Benchmarking www.bortzmeyer.org (be patient).....done
Server Software: Apache/2.2.9
Server Hostname: www.bortzmeyer.org
Server Port: 80
Document Path: /
Document Length: 208025 bytes
Concurrency Level: 10
Time taken for tests: 9.535 seconds
Complete requests: 20
Failed requests: 0
Write errors: 0
Total transferred: 4557691 bytes
HTML transferred: 4551113 bytes
Requests per second: 2.10 [#/sec] (mean)
Time per request: 4767.540 [ms] (mean)
Time per request: 476.754 [ms] (mean, across all concurrent requests)
Transfer rate: 466.79 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 22 107 254.6 24 854
Processing: 996 3301 1837.9 3236 8139
Waiting: 23 25 1.3 25 27
Total: 1018 3408 1795.9 3269 8164
Percentage of the requests served within a certain time (ms)
50% 3269
66% 4219
...
(In that case, network latency was the main slowness factor.)
ab reports itself in the User-Agent field so, in the log of the HTTP server, you'll see something like:
2001:660:3003:8::4:69 - - [28/Jul/2009:12:22:45 +0200] "GET / HTTP/1.0" 200 208025 "-" "ApacheBench/2.3" www.bortzmeyer.org
ab is a widely used benchmarking tool that comes with apache httpd
Grinder is pretty good. It lets you simulate coordinated load from several client machines, which is more meaningful than from a single machine.
There's also JMeter.
I've used httperf and it's quite easy to use. There's a peepcode screencast on how to use it as well.