nginx upstream connection timed out with AWS ELB - caching

We have nginx webservers which serves as reverse proxy and forwards request to AWS ELB. AWS ELB then forwards request to set of back-end servers which respond to requests. We used it successfully for last couple of years with nginx proxy_pass directive with nginx version 1.8. We recently installed a new cluster where nginx version is 1.11.9 and can see upstream connection time out errors in logs after couple of days.I can understand that it is happening when AWS ELB is changing its IP but don't understand why nginx is caching IP.
Logs:
2017/03/07 06:12:01 [warn] 5322#5322: *41004 upstream server temporarily disabled while connecting to upstream, client: 127.0.0.1, server: localhost, request: "GET/TT11617689 HTTP/1.1", upstream: "http://172.21.107.90:80/TT11617689", host: "localhost:8008"
2017/03/07 06:12:01 [error] 5322#5322: *41004 upstream timed out (110: Connection timed out) while connecting to upstream, client: 127.0.0.1
Nginx configuration
user nginx;
worker_processes 1;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
#gzip on;
include /etc/nginx/conf.d/*.conf;
}
server {
listen 8008;
server_name localhost;
set $elb "<<my elb url>>";
#charset koi8-r;
#access_log /var/log/nginx/log/host.access.log main;
if ($request_method !~ ^(GET|POST)$ )
{
return 405;
}
location / {
resolver 172.16.0.2 valid=10s;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass $elb;
}
location /login {
deny all;
return 404;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
}

Related

nginx keycloak spring boot reverse proxy redirect loop

I have dockerized spring boot application and keycloak for authorization. So, i tried to use nginx as reverse proxy(nginx not dockerized). When i use nginx without ssl it works perfectly. But when i try enable ssl with https to http redirect, keycloak fall into redirect loop after enter credentials.
My nginx config
# For more information on configuration, see:
# * Official English Documentation: http://nginx.org/en/docs/
# * Official Russian Documentation: http://nginx.org/ru/docs/
user docker-user;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 5m;
ssl_prefer_server_ciphers on;
ssl_stapling on;
resolver 8.8.8.8;
include /etc/nginx/mime.types;
default_type application/octet-stream;
include /etc/nginx/conf.d/*.conf;
server {
server_name myhost.com;
listen 443 ssl;
ssl_certificate /etc/nginx/ssl/certs/bundle.crt;
ssl_certificate_key /etc/nginx/ssl/certs/cert.key;
ssl_dhparam /etc/nginx/ssl/certs/dhparam.pem;
ssl_session_cache shared:le_nginx_SSL:10m;
ssl_session_timeout 1440m;
ssl_session_tickets off;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers off;
ssl_ciphers "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384";
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
client_max_body_size 500M;
proxy_read_timeout 3600;
location /auth {
proxy_http_version 1.1;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $server_name;
proxy_pass http://myhost.com:8080;
proxy_redirect off;
}
location / {
proxy_http_version 1.1;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $server_name;
proxy_pass http://myhost.com:9010;
proxy_redirect off;
}
error_page 404 /404.html;
location = /usr/share/nginx/html/50x.html/40x.html {
}
error_page 500 502 503 504 /50x.html;
location = /usr/share/nginx/html/50x.html {
}
}
server {
if ($host = myhost.com) {
return 301 https://$host$request_uri;
}
server_name myhost.com;
listen 80;
return 404;
}
}
spring boot log on redirect loop
2021-05-13 10:45:25.756 ERROR 1 --- [qtp276869158-20] o.k.adapters.OAuthRequestAuthenticator : failed to turn code into token
2021-05-13 10:45:25.757 ERROR 1 --- [qtp276869158-20] o.k.adapters.OAuthRequestAuthenticator : status from server: 301
2021-05-13 10:45:25.757 ERROR 1 --- [qtp276869158-20] o.k.adapters.OAuthRequestAuthenticator : <html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.14.1</center>
</body>
</html>
I solved my problem.
In keycloak docker compose file need to add reverse proxy location in KEYCLOAK_FRONTEND_URL
Like KEYCLOAK_FRONTEND_URL: "https://myhost.com/auth"

How to configure Nginx to redirect http 80 traffic to https 8443 port

I have a Springboot application running in a cloud machine on port 8443. In the same machine I have a Nginx server.
Today I access https://www.example.com and it works fine but if I type www.example.com and try to access it is not redirecting to https://www.example.com.
In other words, all http 80 traffic should be redirected to https 8443
Here are my configuration (Springboot app + Nginx)
Springboot application.properties
server.port=8443
security.require-ssl=true
server.ssl.key-store=/etc/letsencrypt/live/www.example.com/keystore.p12
server.ssl.key-store-password=www.example.com
server.ssl.keyStoreType=PKCS12
server.ssl.keyAlias=myAlias
Nginx /etc/nginx/nginx.conf
pid /run/nginx.pid;
events {
worker_connections 768;
}
http {
log_format formatWithUpstreamLogging '[$time_local] $remote_addr - $remote_user - $server_name to: $upstream_addr: $request';
#main log format
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
error_log /var/log/nginx/error.log;
server {
listen 80;
server_name www.example.com example.com;
ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
# managed by Certbot
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_set_header X-NginX-Proxy true;
proxy_pass https://localhost:8443/;
proxy_redirect http://localhost:8443/ https://localhost:8443/;
}
}
}
Could anyone help me on this?
Thanks in advance
I think what you should do is set up a redirect server to https, and then add in the ssl for the main server (create a backup of the Nginx config file in case something goes wrong):
In Nginx /etc/nginx/nginx.conf:
pid /run/nginx.pid;
events {
worker_connections 768;
}
http {
log_format formatWithUpstreamLogging '[$time_local] $remote_addr - $remote_user - $server_name to: $upstream_addr: $request';
#main log format
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
error_log /var/log/nginx/error.log;
server {
listen 80 default_server;
server_name www.example.com example.com;
return 301 https://$server_name$request_uri;
}
server {
listen 8443 ssl default_server;
server_name www.example.com example.com;
ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
# managed by Certbot
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_set_header X-NginX-Proxy true;
proxy_pass https://localhost:8443/;
proxy_redirect http://localhost:8443/ https://localhost:8443/;
}
}
}

Configuring Nginx as a proxy for a Vapor API

'm pretty stuck after a few days of trying to get this working and I could use some help.
I have a vapor API that works fine. I created a route and can access it from http://localhost:8080/backend/returnA in a browser on the server. It returns some JSON.
Where I'm stuck is in trying to configure Nginx to server as a proxy. Can anyone help me understand how the http://localhost:8080/backend/returnA URL translates into a working URL accessible from the LAN?
I'm pretty confused as the Nginx.conf asks for a root URL but I don't know what to put in. If I leave it blank it defaults to /usr/local/Cellar/nginx/1.15.6/html/backend/returnA/index.html which obviously won't work. If I set it to the public folder in the Vapor app directory this also doesn't work. In both instances I get a "No such file or directory".
I've gone through countless Nginx conf settings found online, tried adding a proxy location, nothing works. Trying http://172.16.1.25/backend/returnA/ always returns a 404 from the Nginx server.
How do I point Nginx to my Vapor route when it's not serving a static file like index.html, and instead retuning JSON?
Any help is much appreciated.
Here's the config, edited to include Thanh's code, old location commented out:
#user nobody;
worker_processes 1;
#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;
#pid logs/nginx.pid;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
#log_format main '$remote_addr - $remote_user [$time_local] "$request" '
# '$status $body_bytes_sent "$http_referer" '
# '"$http_user_agent" "$http_x_forwarded_for"';
#access_log logs/access.log main;
sendfile on;
#tcp_nopush on;
#keepalive_timeout 0;
keepalive_timeout 65;
#gzip on;
server {
server_name 172.16.1.25;
listen 80 default_server;
root /Users/localadmin/Developer/server/MedicapAPI/Public/;
# location #proxy {
# proxy_pass http://127.0.0.1:8080;
# proxy_pass_header Server;
# proxy_set_header Host $host;
# proxy_set_header X-Real-IP $remote_addr;
# proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# proxy_pass_header Server;
# proxy_connect_timeout 3s;
# proxy_read_timeout 10s;
# }
location / {
proxy_ignore_client_abort on;
proxy_pass http://localhost:8080/;
proxy_redirect off;
}
}
include servers/*;
}
Using this :
listen 80 default_server;
server_name 172.16.1.25; #ip address of server
it will catch all server blokck and:
location / {
proxy_ignore_client_abort on;
proxy_pass http://localhost:8080/;
proxy_redirect off;
}
It will be proxy_pass to application which is running in port 8080
server {
#server_name mysite.com;
listen 80;
error_log /var/log/mysite.com_error.log warn;
access_log /var/log/mysite.com.ru_access.log;
large_client_header_buffers 8 32k;
client_max_body_size 10M;
location / {
# redirect all traffic to localhost:8080;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-NginX-Proxy true;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://127.0.0.1:8080/;
proxy_redirect off;
proxy_read_timeout 86400;
# enables WS support
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
# prevents 502 bad gateway error
proxy_buffers 8 32k;
proxy_buffer_size 64k;
reset_timedout_connection on;
tcp_nodelay on;
}
# Give direct access to Public files of your app instead of using FileMiddleware
location ~* ^.+.(jpg|jpeg|gif|css|png|js|ico|xml|html|mp4|pdf)$ {
access_log off;
expires 30d;
root /path/to/your/app/Public;
}
}
This is my working example.
For production I suggest you to use SSL certificate e.g. from LetsEncrypt, replace listen port to 443 and add the following configuration lines after listen line:
ssl on;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
ssl_prefer_server_ciphers on;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_certificate /etc/letsencrypt/live/mysite.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/mysite.com/privkey.pem;
ssl_ciphers 'HIGH:!aNULL:!MD5:!kEDH';
add_header Strict-Transport-Security "max-age=31536000; includeSubdomains;";
ssl_stapling on;
ssl_stapling_verify on;
That's it, now you're ready for production!

use Nginx as proxy, ajax request get 405 every other request

I'm using nginx to separate the static and dynamic content,
this is my nginx config:
#user nobody;
worker_processes 1;
#error_log logs/error.log;
error_log /usr/local/var/logs/nginx/error.log notice;
#error_log logs/error.log info;
pid /usr/local/var/logs/nginx/nginx.pid;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
log_format main '$remote_addr:$remote_port - [$request_method] [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for" '
'rt=$request_time uct="$upstream_connect_time" uht="$upstream_header_time" urt="$upstream_response_time"';
access_log /usr/local/var/logs/nginx/access.log main;
sendfile on;
#tcp_nopush on;
#keepalive_timeout 0;
keepalive_timeout 65;
upstream tomcat_backend {
server localhost:8090;
}
server {
listen 9797;
server_name localhost;
server_name_in_redirect off;
access_log /usr/local/var/logs/nginx/test.access.log main;
error_log /usr/local/var/logs/nginx/test-error.access.log debug;
location ~ \.(jpg|png|js|ico|html|gif|css|map)$ {
root /path/to/static;
}
location ^~ /api/ {
access_log /usr/local/var/logs/nginx/nginx-location.access.log main;
error_log /usr/local/var/logs/nginx/nginx-location-error.access.log debug;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-NginX-Proxy true;
proxy_set_header X-Request-Time $msec;
proxy_redirect off;
proxy_set_header Host $host;
proxy_pass http://tomcat_backend;
#proxy_set_header X-Request-Id $request_id;
}
location / {
root /path/to/static;
index index.html;
if (!-e $request_filename) {
rewrite ^/(.*) /index.html last;
}
}
}
include servers/*;
}
sometimes tomcat can get the request and response it correctly, sometimes nginx just return 405 directly, not event forward to tomcat
this is the console of chrome, it always follow that pattern, 405, 401, 405, 401..., it shows 401 is because I input wrong password in purpose, that mean the request has been forward to tomcat:
I found that if I use node.js as backend(just a proxy server), everything is ok, no 405 anymore.
OK, finally I found the reason.
there is another process is monitoring the same port.
but it's wired that two process are using the same port

NGINX in front of TOMCAT to serve static files

I'm working on a project with grails 2.2.2 on a local machine Mac OSX Lion 10.7.5 I have installed NGINX with brew and modified the nginx.conf as following :
worker_processes 1;
error_log logs/error.log info;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log logs/access.log main;
sendfile on;
keepalive_timeout 65;
server {
listen 8081;
server_name localhost;
root /;
access_log /Users/lorenzo/grails/projects/logs/myproject_access.log;
location / {
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP $remote_addr;
proxy_pass http://localhost:8081;
}
#images folders
location /posters {
root /Users/lorenzo/grails/projects/posters/;
}
#images folders
location /avatars {
root /Users/lorenzo/grails/projects/avatars/;
}
#images folders
location /waveforms {
root /Users/lorenzo/grails/projects/waveforms/;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
When I access http://localhost:8081 my site is running but I want to be sure the images are served by nginx and not by tomcat so I look at myproject_access.log but nothing is happening.
ngnix is writing into the log ONLY when tomcat is NOT running.
Is there a way to "monitor" the static files served by nginx ?
Thank you
EDIT
Executing curl -I http://localhost:8081
when tomcat is running the output is:
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1 //TOMCAT
...
when tomcat is NOT running the output is:
HTTP/1.1 500 Internal Server Error
Server: nginx/1.4.1 //NGINX
Date: Tue, 08 Apr 2014 09:30:00 GMT
Content-Type: text/html
Content-Length: 192
Connection: keep-alive
Your problem is that your are making the both servers listen on the same port, you need to move tomcat to another port like 8082 and let nginx listen to the main port ( which is 8081 in your case ), and then tell nginx to proxy to 8082 when the request isn't an image ( or any asset ).
also here's a refinement to your server block
server {
server_name localhost;
listen 8081;
root /Users/lorenzo/grails/projects;
location #tomcat {
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP $remote_addr;
proxy_pass http://localhost:8082;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
location / {
try_files $uri $uri/ #tomcat;
}
}

Resources