Lighttpd slow downloads - performance

Question

I have a dedicated server with 1GB/s dedicated, 4GB ram and 4cpus. I have static files for download (from 300mb to 900mb). I was testing over Apache, Nginx and Lighttpd.
Apache makes too many threats and after 200 connections it goes very high so apache it's a NO GO...
Nginx after 100 connections it goes very high so it's a NO GO either.
Lighttpd so far is very good as is a single-threaded server. With 500 concurrent connections the load stays at 0.90 - 1.10 (very good) but I'm facing a download speed problem, it goes slower even when I have 1GBps dedicated port, I see the iptraf and with 500 concurrent connections it goes no more than 250000 KB/s. With apache and nginx sometimes it went to 700000 KB/s the upstream in the server. I switched between sendfile and writev in the config and it has the same result.
I'm not using any php or fast-cgi, just straight download directly to the file, for example: http://www.myserver.com/file.zip and it downloads the file.
I will attach some info here for you to help me figure it out.
Kernel 2.6
lighttpd.conf
# lighttpd configuration file
#
# use it as a base for lighttpd 1.0.0 and above
#
# $Id: lighttpd.conf,v 1.7 2004/11/03 22:26:05 weigon Exp $
############ Options you really have to take care of ####################
## modules to load
# at least mod_access and mod_accesslog should be loaded
# all other module should only be loaded if really neccesary
# - saves some time
# - saves memory
server.modules = (
# "mod_rewrite",
# "mod_redirect",
# "mod_alias",
"mod_access",
# "mod_cml",
# "mod_trigger_b4_dl",
# "mod_auth",
# "mod_status",
# "mod_setenv",
# "mod_proxy_core",
# "mod_proxy_backend_http",
# "mod_proxy_backend_fastcgi",
# "mod_proxy_backend_scgi",
# "mod_proxy_backend_ajp13",
# "mod_simple_vhost",
# "mod_evhost",
# "mod_userdir",
# "mod_cgi",
# "mod_compress",
# "mod_ssi",
# "mod_usertrack",
# "mod_expire",
# "mod_secdownload",
# "mod_rrdtool",
"mod_accesslog" )
## a static document-root, for virtual-hosting take look at the
## server.virtual-* options
server.document-root = "/usr/share/nginx/html/"
## where to send error-messages to
server.errorlog = "/www/logs/lighttpd.error.log"
# files to check for if .../ is requested
index-file.names = ( "index.php", "index.html",
"index.htm", "default.htm" )
## set the event-handler (read the performance section in the manual)
# server.event-handler = "freebsd-kqueue" # needed on OS X
server.event-handler = "linux-sysepoll"
#server.network-backend = "linux-sendfile"
server.network-backend = "writev"
# mimetype mapping
mimetype.assign = (
".pdf" => "application/pdf",
".sig" => "application/pgp-signature",
".spl" => "application/futuresplash",
".class" => "application/octet-stream",
".ps" => "application/postscript",
".torrent" => "application/x-bittorrent",
".dvi" => "application/x-dvi",
".gz" => "application/x-gzip",
".pac" => "application/x-ns-proxy-autoconfig",
".swf" => "application/x-shockwave-flash",
".tar.gz" => "application/x-tgz",
".tgz" => "application/x-tgz",
".tar" => "application/x-tar",
".zip" => "application/zip",
".mp3" => "audio/mpeg",
".m3u" => "audio/x-mpegurl",
".wma" => "audio/x-ms-wma",
".wax" => "audio/x-ms-wax",
".ogg" => "application/ogg",
".wav" => "audio/x-wav",
".gif" => "image/gif",
".jpg" => "image/jpeg",
".jpeg" => "image/jpeg",
".png" => "image/png",
".xbm" => "image/x-xbitmap",
".xpm" => "image/x-xpixmap",
".xwd" => "image/x-xwindowdump",
".css" => "text/css",
".html" => "text/html",
".htm" => "text/html",
".js" => "text/javascript",
".asc" => "text/plain",
".c" => "text/plain",
".cpp" => "text/plain",
".log" => "text/plain",
".conf" => "text/plain",
".text" => "text/plain",
".txt" => "text/plain",
".dtd" => "text/xml",
".xml" => "text/xml",
".mpeg" => "video/mpeg",
".mpg" => "video/mpeg",
".mov" => "video/quicktime",
".qt" => "video/quicktime",
".avi" => "video/x-msvideo",
".asf" => "video/x-ms-asf",
".asx" => "video/x-ms-asf",
".wmv" => "video/x-ms-wmv",
".bz2" => "application/x-bzip",
".tbz" => "application/x-bzip-compressed-tar",
".tar.bz2" => "application/x-bzip-compressed-tar"
)
# Use the "Content-Type" extended attribute to obtain mime type if possible
#mimetype.use-xattr = "enable"
## send a different Server: header
## be nice and keep it at lighttpd
# server.tag = "lighttpd"
#### accesslog module
accesslog.filename = "/www/logs/access.log"
## deny access the file-extensions
#
# ~ is for backupfiles from vi, emacs, joe, ...
# .inc is often used for code includes which should in general not be part
# of the document-root
url.access-deny = ( "~", ".inc" )
$HTTP["url"] =~ "\.pdf$" {
server.range-requests = "disable"
}
##
# which extensions should not be handle via static-file transfer
#
# .php, .pl, .fcgi are most often handled by mod_fastcgi or mod_cgi
static-file.exclude-extensions = ( ".php", ".pl", ".fcgi" )
######### Options that are good to be but not neccesary to be changed #######
## bind to port (default: 80)
#server.port = 81
## bind to localhost (default: all interfaces)
#server.bind = "grisu.home.kneschke.de"
## error-handler for status 404
#server.error-handler-404 = "/error-handler.html"
#server.error-handler-404 = "/error-handler.php"
## to help the rc.scripts
#server.pid-file = "/var/run/lighttpd.pid"
###### virtual hosts
##
## If you want name-based virtual hosting add the next three settings and load
## mod_simple_vhost
##
## document-root =
## virtual-server-root + virtual-server-default-host + virtual-server-docroot
## or
## virtual-server-root + http-host + virtual-server-docroot
##
#simple-vhost.server-root = "/home/weigon/wwwroot/servers/"
#simple-vhost.default-host = "grisu.home.kneschke.de"
#simple-vhost.document-root = "/pages/"
##
## Format: <errorfile-prefix><status-code>.html
## -> ..../status-404.html for 'File not found'
#server.errorfile-prefix = "/home/weigon/projects/lighttpd/doc/status-"
## virtual directory listings
#dir-listing.activate = "enable"
## enable debugging
#debug.log-request-header = "enable"
#debug.log-response-header = "enable"
#debug.log-request-handling = "enable"
#debug.log-file-not-found = "enable"
#debug.log-condition-handling = "enable"
### only root can use these options
#
# chroot() to directory (default: no chroot() )
#server.chroot = "/"
## change uid to <uid> (default: don't care)
#server.username = "wwwrun"
## change uid to <uid> (default: don't care)
#server.groupname = "wwwrun"
#### compress module
#compress.cache-dir = "/tmp/lighttpd/cache/compress/"
#compress.filetype = ("text/plain", "text/html")
#### proxy module
## read proxy.txt for more info
#$HTTP["url"] =~ "\.php$" {
# proxy-core.balancer = "round-robin"
# proxy-core.allow-x-sendfile = "enable"
# proxy-core.protocol = "http"
# proxy-core.backends = ( "192.168.0.101:80" )
# proxy-core.max-pool-size = 16
#}
#### fastcgi module
## read fastcgi.txt for more info
## for PHP don't forget to set cgi.fix_pathinfo = 1 in the php.ini
#$HTTP["url"] =~ "\.php$" {
# proxy-core.balancer = "round-robin"
# proxy-core.allow-x-sendfile = "enable"
# proxy-core.check-local = "enable"
# proxy-core.protocol = "fastcgi"
# proxy-core.backends = ( "unix:/tmp/php-fastcgi.sock" )
# proxy-core.max-pool-size = 16
#}
#### CGI module
#cgi.assign = ( ".pl" => "/usr/bin/perl",
# ".cgi" => "/usr/bin/perl" )
#
#### SSL engine
#ssl.engine = "enable"
#ssl.pemfile = "server.pem"
#### status module
#status.status-url = "/server-status"
#status.config-url = "/server-config"
#### auth module
## read authentication.txt for more info
#auth.backend = "plain"
#auth.backend.plain.userfile = "lighttpd.user"
#auth.backend.plain.groupfile = "lighttpd.group"
#auth.backend.ldap.hostname = "localhost"
#auth.backend.ldap.base-dn = "dc=my-domain,dc=com"
#auth.backend.ldap.filter = "(uid=$)"
#auth.require = ( "/server-status" =>
# (
# "method" => "digest",
# "realm" => "download archiv",
# "require" => "user=jan"
# ),
# "/server-config" =>
# (
# "method" => "digest",
# "realm" => "download archiv",
# "require" => "valid-user"
# )
# )
#### url handling modules (rewrite, redirect, access)
#url.rewrite = ( "^/$" => "/server-status" )
#url.redirect = ( "^/wishlist/(.+)" => "http://www.123.org/$1" )
#### both rewrite/redirect support back reference to regex conditional using %n
#$HTTP["host"] =~ "^www\.(.*)" {
# url.redirect = ( "^/(.*)" => "http://%1/$1" )
#}
#
# define a pattern for the host url finding
# %% => % sign
# %0 => domain name + tld
# %1 => tld
# %2 => domain name without tld
# %3 => subdomain 1 name
# %4 => subdomain 2 name
#
#evhost.path-pattern = "/home/storage/dev/www/%3/htdocs/"
#### expire module
#expire.url = ( "/buggy/" => "access 2 hours", "/asdhas/" => "access plus 1 seconds 2 minutes")
#### ssi
#ssi.extension = ( ".shtml" )
#### rrdtool
#rrdtool.binary = "/usr/bin/rrdtool"
#rrdtool.db-name = "/var/www/lighttpd.rrd"
#### setenv
#setenv.add-request-header = ( "TRAV_ENV" => "mysql://user#host/db" )
#setenv.add-response-header = ( "X-Secret-Message" => "42" )
## for mod_trigger_b4_dl
# trigger-before-download.gdbm-filename = "/home/weigon/testbase/trigger.db"
# trigger-before-download.memcache-hosts = ( "127.0.0.1:11211" )
# trigger-before-download.trigger-url = "^/trigger/"
# trigger-before-download.download-url = "^/download/"
# trigger-before-download.deny-url = "http://127.0.0.1/index.html"
# trigger-before-download.trigger-timeout = 10
## for mod_cml
## don't forget to add index.cml to server.indexfiles
# cml.extension = ".cml"
# cml.memcache-hosts = ( "127.0.0.1:11211" )
#### variable usage:
## variable name without "." is auto prefixed by "var." and becomes "var.bar"
#bar = 1
#var.mystring = "foo"
## integer add
#bar += 1
## string concat, with integer cast as string, result: "www.foo1.com"
#server.name = "www." + mystring + var.bar + ".com"
## array merge
#index-file.names = (foo + ".php") + index-file.names
#index-file.names += (foo + ".php")
#### include
#include /etc/lighttpd/lighttpd-inc.conf
## same as above if you run: "lighttpd -f /etc/lighttpd/lighttpd.conf"
#include "lighttpd-inc.conf"
#### include_shell
#include_shell "echo var.a=1"
## the above is same as:
#var.a=1
sysctl.conf
# Kernel sysctl configuration file for Red Hat Linux
#
# For binary values, 0 is disabled, 1 is enabled. See sysctl(8) and
# sysctl.conf(5) for more details.
# Controls IP packet forwarding
net.ipv4.ip_forward = 0
# Controls source route verification
net.ipv4.conf.default.rp_filter = 1
# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0
# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0
# Controls whether core dumps will append the PID to the core filename
# Useful for debugging multi-threaded applications
kernel.core_uses_pid = 1
# Controls the use of TCP syncookies
net.ipv4.tcp_syncookies = 1
# Controls the maximum size of a message, in bytes
kernel.msgmnb = 65536
# Controls the default maxmimum size of a mesage queue
kernel.msgmax = 65536
# Controls the maximum shared segment size, in bytes
kernel.shmmax = 68719476736
# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 4294967296
# These ensure that TIME_WAIT ports either get reused or closed fast.
net.ipv4.tcp_fin_timeout = 1
net.ipv4.tcp_tw_recycle = 1
# TCP memory
net.core.rmem_max = 16777216
net.core.rmem_default = 16777216
net.core.netdev_max_backlog = 262144
net.core.somaxconn = 262144
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_orphans = 262144
net.ipv4.tcp_max_syn_backlog = 262144
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 2
# For Large File Hosting Servers
net.core.wmem_max = 1048576
#net.ipv4.tcp_wmem = 4096 87380 524288
net.ipv4.tcp_wmem = 4096 524288 16777216
Actual top command
top - 16:15:57 up 6 days, 19:30, 2 users, load average: 1.05, 0.85, 0.83
Tasks: 143 total, 1 running, 142 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.6%us, 2.8%sy, 0.0%ni, 64.7%id, 30.8%wa, 0.0%hi, 1.1%si, 0.0%st
Mem: 3914664k total, 3729404k used, 185260k free, 1676k buffers
Swap: 8388600k total, 9984k used, 8378616k free, 3340832k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
28590 root 20 0 518m 75m 71m D 13.1 2.0 1:12.24 lighttpd
28660 root 20 0 15016 1104 812 R 1.9 0.0 0:00.02 top
1 root 20 0 19328 620 396 S 0.0 0.0 0:03.74 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.02 kthreadd
3 root RT 0 0 0 0 S 0.0 0.0 0:00.14 migration/0
4 root 20 0 0 0 0 S 0.0 0.0 0:00.12 ksoftirqd/0
5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
6 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
7 root RT 0 0 0 0 S 0.0 0.0 0:00.32 migration/1
8 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/1
9 root 20 0 0 0 0 S 0.0 0.0 0:01.96 ksoftirqd/1
10 root RT 0 0 0 0 S 0.0 0.0 0:00.19 watchdog/1
11 root RT 0 0 0 0 S 0.0 0.0 0:01.00 migration/2
12 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/2
13 root 20 0 0 0 0 S 0.0 0.0 5:04.44 ksoftirqd/2
14 root RT 0 0 0 0 S 0.0 0.0 0:00.23 watchdog/2
15 root RT 0 0 0 0 S 0.0 0.0 0:00.50 migration/3
16 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/3
17 root 20 0 0 0 0 S 0.0 0.0 0:01.84 ksoftirqd/3
18 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/3
iostat
Linux 2.6.32-220.7.1.el6.x86_64 (zlin) 05/01/2012 _x86_64_ (4 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
0.57 0.00 3.95 30.76 0.00 64.72
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 109.58 38551.74 149.33 22695425748 87908220
netstat -an |grep :80 |wc -l
259
iptraf
247270.0 kbits/sec
What should I change to make the clients download faster, they said sometimes it downloads slower than 10 KB/s

Answer 1

So it appears disk I/O is your problem from looking at top and iostat. You've used up all the disk cache and the system is waiting for data to be read in from disk before it can send it out the NIC.
The first thing I would try is to change to:
server.network-backend = "linux-sendfile"
as that will improve buffer usage (a bit, not as much as it should).
You can do a back of the envelope calculation of how much memory you would need to cache your typical work load (simplistically just add together the sizes of your most 100 popular files). I'm guessing that it's going to be a lot more than the 4GB of memory that you have so the next thing to do would be to either get faster disk drives or more memory.
This is why people use Content Delivery Networks (CDNs) to deliver large amounts of bandwidth (which often comes in the form of large files).

Answer 2

The problem here isn't your web server, it's that HTTP is really not designed as a file download protocol. It's the Hyper Text Transport Protocol, and many of the decisions around HTTP focus on the Hyperlinked Text aspect- file sizes are expected to be small, under a few dozen Kb and certainly under a Mb. The web infrastructure takes advantage of this fact in a lot of their approaches to data caching, etc. Instead of using HTTP for something it really isn't designed for, I would recommend looking at using a different transport mechanism.
FTP: File Transfer Protocol. FTP was designed specifically to transfer files of arbitrary size, and doesn't make the same assumptions as HTTP software. If all you are doing is static downloads, your web page HTML can link to the static files with an ftp:// link, and configuring an FTP server to allow anonymous download is usually straightforward. Check your FTP server's docs for details. Browsers since IE6/FF2 have supported basic FTP natively- the average user will have no different workflow than usual. This is probably not the best approach, as FTP was designed long before HTTP, and as Perry mentioned, long before we had half a gig files.
CDN: Using a content delivery network like Amazon's S3 doesn't technically get around using HTTP, but it lets you not have to worry about your users overloading your server like you're seeing.
BitTorrent: If your users are a bit more tech savy, consider setting your server up to seed the static file indefinitely, then publish magnet links on your site. In the worst case, a single user will experience a direct download from your server, using a protocol that actually knows how to handle large files. In the best case, your hundreds of users will both leech and seed eachother, drastically reducing your server's load. Yes, this required your users to know how to run and configure bittorrent, which is probably not the case, but it's an interesting paradigm for file downloads none the less.