Poor performance while using nginx as proxy - performance

Question

The current setup of our backend uses Route53 to route requests to tomcat servers which run on ec2 instances.
I am trying to setup nginx as a load balancer(proxy) to route requests to our tomcat servers.
Here are the instance types,
Tomcat server instance type = m3.2xlarge
nginx server instance type
= c3.large
When I run ab (apache benchmark) with 100 concurrent connections without keep alive, I see that the performance of a single tomcat instance is better than 2 tomcat servers in front of an nginx server. I am now wondering if there is something wrong with my nginx config. I checked the error.log file on nginx instance and there are no errors. Also, the CPU on nginx instance does not cross 30% while running benchmark tool. Here is my nginx config,
user nginx;
pid /run/nginx.pid;
worker_processes auto;
worker_rlimit_nofile 32768;
events {
worker_connections 8192;
multi_accept on;
use epoll;
}
http {
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;
upstream backend {
server x.x.x.x:443;
server x.x.x.x:443;
keepalive 1024;
}
server {
listen 443;
server_name localhost;
ssl on;
ssl_certificate /etc/nginx/certs/ssl-bundle_2015_2018.crt;
ssl_certificate_key /etc/nginx/certs/chewie.key;
ssl_dhparam /etc/nginx/certs/dhparam.pem;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_session_cache shared:SSL:10m;
ssl_prefer_server_ciphers on;
ssl_session_timeout 10m;
ssl_ciphers "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS !RC4";
location / {
proxy_pass https://backend;
proxy_cache_bypass true;
proxy_no_cache true;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
gzip on;
gzip_disable "msie6";
gzip_vary on;
gzip_proxied any;
gzip_comp_level 6;
gzip_buffers 16 8k;
gzip_http_version 1.1;
gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
Here are apache benchmark results without nginx.
Concurrency Level: 100
Time taken for tests: 8.393 seconds
Complete requests: 800
Failed requests: 0
Total transferred: 368000 bytes
HTML transferred: 16800 bytes
Requests per second: 95.32 [#/sec] (mean)
Time per request: 1049.083 [ms] (mean)
Time per request: 10.491 [ms] (mean, across all concurrent requests)
Transfer rate: 42.82 [Kbytes/sec] received
These are results with nginx in front of 2 tomcat servers:
Concurrency Level: 100
Time taken for tests: 23.494 seconds
Complete requests: 800
Failed requests: 0
Total transferred: 381600 bytes
HTML transferred: 16800 bytes
Requests per second: 34.05 [#/sec] (mean)
Time per request: 2936.768 [ms] (mean)
Time per request: 29.368 [ms] (mean, across all concurrent requests)
Transfer rate: 15.86 [Kbytes/sec] received
Any thoughts on where I should be looking to optimize are appreciated!

Answer 1

Here are some things done to improve performance,
Convert the traffic between nginx and the upstream server to http
form https
Use the right ssl ciphers for your nginx. Make sure to run
the ssl test to make sure ciphers used are secure (www.ssllabs.com)
Increase file descriptor limits for nginx server as well the tomcat
instances to a high number.
Will keep updating as I find more things.