Haproxy 1.5 performance issues - performance

Question

I need to improve the performance of an haproxy 1.5 running as a load balancer in an Ubuntu 14.04 instance. I have an analytics like code on many sites and for every pageview the client asks between 2-5 diferent scrips of ours. The other day we received more than 1k requests per second on the load balancer and it started to run really slow. It reached the active sessions limit 2000 at a rate for 1000 per second. On the configuration we use option http-keep-alive 100 to maintain the connection open for 100 ms until it is closed. How can we improve this? What is the best config for this use case? I may be loosing many details here please ask for them is there is info missing.
EDIT
Here are some details:
I'm running an AWS ubuntu 14.04 server in a c3.xlarge virtual machine. There we use haproxy 1.5 to load balance web traffic between several app instances. (Every app has its own haproxy to load balance between its own app instances - deployed one per core).
The server only has haproxy and no other software installed.
The bottleneck as per haproxy stat page is the front end load balancer as at that moment it had a session rate of 258 and current sessions of 2000 (being 2000 the max), and all the apps had a 96 sessions rate and 0/1 as the current sessions. I would post image but because of my reputation points I can't do that.
This was the configuration at that point in time:
global
maxconn 18000
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL). This list is from:
# https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
ssl-default-bind-options no-sslv3
defaults
mode http
retries 2
option redispatch
timeout connect 5s
timeout client 15s
timeout server 15s
timeout http-keep-alive 1
frontend public
log 127.0.0.1 local0 notice
option dontlognull
option httplog
bind *:80
bind *:443 ssl crt /etc/ssl/private/server.pem
default_backend rely_apps
frontend private
bind 127.0.0.1:80
stats enable
stats auth xxx:xxx
stats admin if LOCALHOST
stats uri /haproxy?stats
stats show-legends
stats realm Haproxy\ Statistics
backend rely_apps
option forwardfor
balance roundrobin
option httpchk
server app1 10.0.0.1:80 check
server app2 10.0.0.2:80 check
server app3 10.0.0.3:80 check
The connections were very high, it seems like it was closing them (or closing at a really low rate).
CPU and memory usage was really low.
Now we changed that config for the following one and it's working without problems:
global
maxconn 64000
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon
tune.bufsize 16384
tune.maxrewrite 1024
nbproc 4
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL). This list is from:
# https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
ssl-default-bind-options no-sslv3
defaults
mode http
retries 2
option redispatch
option forceclose
option http-pretend-keepalive
timeout connect 5s
timeout client 15s
timeout server 15s
timeout http-keep-alive 1s
frontend public
log 127.0.0.1 local0 notice
option dontlognull
option httplog
maxconn 18000
bind *:80
bind *:443 ssl crt /etc/ssl/private/server.pem
default_backend rely_apps
#frontend private
# bind 127.0.0.1:80
stats enable
stats auth xxx:xxx
stats admin if LOCALHOST
stats uri /haproxy?stats
stats show-legends
stats realm Haproxy\ Statistics
backend rely_apps
option forwardfor
balance leastconn
option httpchk
server app1 10.0.0.1:80 check maxconn 100
server app2 10.0.0.2:80 check maxconn 100
server app3 10.0.0.3:80 check maxconn 100
However all connections are being closed on the return (and we have the same rate of sessions and requests).
This is not good also because we have to open a new connection for every client request (and we have 3/4 requests per client).
How can we achieve a good keep-alive (like 100ms I think could work), without hitting the max connections limit?
Thanks.

Answer 1

The number you give are very very low.
Please give more details about your architecture, type of server, third party software running on it (such as iptables), also share your configuration.
Baptiste