g-wan - reproducing the performance claims - caching

Question

Using gwan_linux64-bit.tar.bz2 under Ubuntu 12.04 LTS unpacking and running gwan
then pointing wrk at it (using a null file null.html)
wrk --timeout 10 -t 2 -c 100 -d20s http://127.0.0.1:8080/null.html
Running 20s test # http://127.0.0.1:8080/null.html
2 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 11.65s 5.10s 13.89s 83.91%
Req/Sec 3.33k 3.65k 12.33k 75.19%
125067 requests in 20.01s, 32.08MB read
Socket errors: connect 0, read 37, write 0, timeout 49
Requests/sec: 6251.46
Transfer/sec: 1.60MB
.. very poor performance, in fact there seems to be some kind of huge latency issue.
During the test gwan is 200% busy and wrk is 67% busy.
Pointing at nginx, wrk is 200% busy and nginx is 45% busy:
wrk --timeout 10 -t 2 -c 100 -d20s http://127.0.0.1/null.html
Thread Stats Avg Stdev Max +/- Stdev
Latency 371.81us 134.05us 24.04ms 91.26%
Req/Sec 72.75k 7.38k 109.22k 68.21%
2740883 requests in 20.00s, 540.95MB read
Requests/sec: 137046.70
Transfer/sec: 27.05MB
Pointing weighttpd at nginx gives even faster results:
/usr/local/bin/weighttp -k -n 2000000 -c 500 -t 3 http://127.0.0.1/null.html
weighttp - a lightweight and simple webserver benchmarking tool
starting benchmark...
spawning thread #1: 167 concurrent requests, 666667 total requests
spawning thread #2: 167 concurrent requests, 666667 total requests
spawning thread #3: 166 concurrent requests, 666666 total requests
progress: 9% done
progress: 19% done
progress: 29% done
progress: 39% done
progress: 49% done
progress: 59% done
progress: 69% done
progress: 79% done
progress: 89% done
progress: 99% done
finished in 7 sec, 13 millisec and 293 microsec, 285172 req/s, 57633 kbyte/s
requests: 2000000 total, 2000000 started, 2000000 done, 2000000 succeeded, 0 failed, 0 errored
status codes: 2000000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 413901205 bytes total, 413901205 bytes http, 0 bytes data
The server is a virtual 8 core dedicated server (bare metal), under KVM
Where do I start looking to identify the problem gwan is having on this platform ?
I have tested lighttpd, nginx and node.js on this same OS, and the results are all as one would expect. The server has been tuned in the usual way with expanded ephemeral ports, increased ulimits, adjusted time wait recycling etc.

Answer 1

Nov. 7 UPDATE: We have fixed the empty-file issue in G-WAN v4.11.7 and G-WAN is now twice faster (with the www cache disabled) than Nginx at this game too.
Recent releases of G-WAN are faster than Nginx with small and large files, and the G-WAN caches are disabled by default in order to make it easier for people to compare G-WAN with other servers like Nginx.
Nginx has a few caching features (a fd cahe to skip stat() calls and a memcached-based module) but both are necessarily much slower than G-WAN's local cache.
Disabling caching was also desirable for certain applications like CDNs. Other applications like AJAX applications greatly benefit from G-WAN caching capabilities so caching can be re-enabled at will, even on a per-request basis.
Hope this clarifies this question.
"reproducing the performance claims"
First, the title is misleading as the poorly documented* test above does not use the same tools nor the HTTP resources fetched by G-WAN tests.
[*] where is your nginx.conf file? what are the HTTP response headers of the two servers? what is your "bare metal" 8-Core CPU?
G-WAN tests are based on ab.c, a wrapper written by the G-WAN Team for weighttp (a test tool made by the Lighttpd server Team) because the information disclosed by ab.c is much more informative.
Second, the tested file "null.html" is... an empty file.
We won't waste time to discuss the irrelevance of such a test (how many empty HTML files your Web site is serving?) but it is likely to be the reason of the observed "poor performance".
G-WAN was not created to serve empty files (and we never tried nor ewre ever asked to do this). But we will surely add this feature to avoid the confusion created by such a test.
As you want to "check the claims" I would encourage you to use weighttp (the fastest HTTP load tool in your test) with a 100.bin file (a 100-byte file with an uncompressible MIME type: no Gzip will be involved here).
With a non-null file Nginx is massively slower than G-WAN, even in independent tests.
We did not know about wrk so far but it seems to be a tool made by the Nginx team:
"wrk was written specifically to try and push nginx to it's limits,
and in it's first round of tests was pushed up to 0.5Mr/s."
UPDATE (a day later)
Since you did not bother to publish any more data, we did it:
wrk weighttp
----------------------- -----------------------
Web Server 0.html RPS 100.html RPS 0.html RPS 100.html RPS
---------- ---------- ------------ ---------- ------------
G-WAN 80,783.03 649,367.11 175,515 717,813
Nginx 198,800.93 179,939.40 184,046 199,075
Like in your test, we can see that wrk is slightly slower than weighttp.
We can also see that G-WAN is faster than Nginx with both HTTP load tools.
Here are the detailled results:
G-WAN
./wrk -c300 -d3 -t6 "http://127.0.0.1:8080/0.html"
Running 3s test # http://127.0.0.1:8080/0.html
6 threads and 300 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 3.87ms 5.30ms 80.97ms 99.53%
Req/Sec 14.73k 1.60k 16.33k 94.67%
248455 requests in 3.08s, 55.68MB read
Socket errors: connect 0, read 248448, write 0, timeout 0
Requests/sec: 80783.03
Transfer/sec: 18.10MB
./wrk -c300 -d3 -t6 "http://127.0.0.1:8080/100.html"
Running 3s test # http://127.0.0.1:8080/100.html
6 threads and 300 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 263.15us 381.82us 16.50ms 99.60%
Req/Sec 115.55k 14.38k 154.55k 82.70%
1946700 requests in 3.00s, 655.35MB read
Requests/sec: 649367.11
Transfer/sec: 218.61MB
weighttp -kn300000 -c300 -t6 "http://127.0.0.1:8080/0.html"
progress: 100% done
finished in 1 sec, 709 millisec and 252 microsec, 175515 req/s, 20159 kbyte/s
requests: 300000 total, 300000 started, 300000 done, 150147 succeeded, 149853 failed, 0 errored
status codes: 150147 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 35284545 bytes total, 35284545 bytes http, 0 bytes data
weighttp -kn300000 -c300 -t6 "http://127.0.0.1:8080/100.html"
progress: 100% done
finished in 0 sec, 417 millisec and 935 microsec, 717813 req/s, 247449 kbyte/s
requests: 300000 total, 300000 started, 300000 done, 300000 succeeded, 0 failed, 0 errored
status codes: 300000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 105900000 bytes total, 75900000 bytes http, 30000000 bytes data
Nginx
./wrk -c300 -d3 -t6 "http://127.0.0.1:8080/100.html"
Running 3s test # http://127.0.0.1:8080/100.html
6 threads and 300 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.54ms 1.16ms 11.67ms 72.91%
Req/Sec 34.47k 6.02k 56.31k 70.65%
539743 requests in 3.00s, 180.42MB read
Requests/sec: 179939.40
Transfer/sec: 60.15MB
./wrk -c300 -d3 -t6 "http://127.0.0.1:8080/0.html"
Running 3s test # http://127.0.0.1:8080/0.html
6 threads and 300 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.44ms 1.15ms 9.37ms 75.93%
Req/Sec 38.16k 8.57k 62.20k 69.98%
596070 requests in 3.00s, 140.69MB read
Requests/sec: 198800.93
Transfer/sec: 46.92MB
weighttp -kn300000 -c300 -t6 "http://127.0.0.1:8080/0.html"
progress: 100% done
finished in 1 sec, 630 millisec and 19 microsec, 184046 req/s, 44484 kbyte/s
requests: 300000 total, 300000 started, 300000 done, 300000 succeeded, 0 failed, 0 errored
status codes: 300000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 74250375 bytes total, 74250375 bytes http, 0 bytes data
weighttp -kn300000 -c300 -t6 "http://127.0.0.1:8080/100.html"
progress: 100% done
finished in 1 sec, 506 millisec and 968 microsec, 199075 req/s, 68140 kbyte/s
requests: 300000 total, 300000 started, 300000 done, 300000 succeeded, 0 failed, 0 errored
status codes: 300000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 105150400 bytes total, 75150400 bytes http, 30000000 bytes data
Nginx configuration file trying to match G-WAN's behavior
# ./configure --without-http_charset_module --without-http_ssi_module
# --without-http_userid_module --without-http_rewrite_module
# --without-http_limit_zone_module --without-http_limit_req_module
user www-data;
worker_processes 6;
worker_rlimit_nofile 500000;
pid /var/run/nginx.pid;
events {
# tried other values up to 100000 without better results
worker_connections 4096;
# multi_accept on; seems to be slower
multi_accept off;
use epoll;
}
http {
charset utf-8; # HTTP "Content-Type:" header
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 10;
keepalive_requests 10; # 1000+ slows-down nginx enormously...
types_hash_max_size 2048;
include /usr/local/nginx/conf/mime.types;
default_type application/octet-stream;
gzip off; # adjust for your tests
gzip_min_length 500;
gzip_vary on; # HTTP "Vary: Accept-Encoding" header
gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
# cache metadata (file time, size, existence, etc) to prevent syscalls
# this does not cache file contents. It should helps in benchmarks where
# a limited number of files is accessed more often than others (this is
# our case as we serve one single file fetched repeatedly)
# THIS IS ACTUALY SLOWING-DOWN THE TEST...
#
# open_file_cache max=1000 inactive=20s;
# open_file_cache_errors on;
# open_file_cache_min_uses 2;
# open_file_cache_valid 300s;
server {
listen 127.0.0.1:8080;
access_log off;
# only log critical errors
#error_log /usr/local/nginx/logs/error.log crit;
error_log /dev/null crit;
location / {
root /usr/local/nginx/html;
index index.html;
}
location = /nop.gif {
empty_gif;
}
location /imgs {
autoindex on;
}
}
}
Comments are welcome - especially from Nginx experts - to have a discussion based on this fully-documented test.