I have an app providing RESTFull api in google kubernetes cluster.
In front of application i have an nginx working as a proxy_pass.
The problem is that one request of few thousands (1000, 2000) has bad data in response (other users data). Analysing logs showed that request of the bad response doesn't come to the application at all.
But it comes to nginx:
2019/05/08 13:48:03 [warn] 5#5: *28350 delaying request, excess: 0.664, by zone "one", client: 10.240.0.23, server: myportal.com, request: "GET /api/myresource?testId=10 HTTP/1.1"
In the same time there's no logs in the app for testId=10 (but there are for testId=9 and testId=11 when i make sequential test 1..1000)
Nginx configuration is almost default
limit_req_zone $binary_remote_addr zone=one:10m rate=4r/s;
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name myportal.com;
if ($http_x_forwarded_proto = "http") {
return 308 https://$server_name;
}
charset utf-8;
access_log on;
server_tokens off;
location /api {
proxy_pass http://backend-service:8000;
limit_req zone=one burst=10;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
There is no caching configured (or maybe it's on by default?).
Application is working in google kubernetes environement, so the request chain looks like this
(k8s ingress, nginx-service) -> nginx -> (k8s backend-service) -> backend
Backend app is written in spring and using jetty to run.
Nginx version was updated from 1.13.X to 1.15.12 but both has the same issue.
I have no idea what and where should i check to find the cause of the problem.
Error you see comes from Nginx because of configs limit_req_zone $binary_remote_addr zone=one:10m rate=4r/s; and limit_req zone=one burst=10;
Read more here: http://nginx.org/ru/docs/http/ngx_http_limit_req_module.html
Did you put it for reason?
Related
I have a React application running with NGINX which handles traffic on one port (www.domain.com - https) and I also have a back-end Spring Boot application which runs on a different port (www.domain.com:7080 - http).
Now NGINX serves 80, 443 ports and loads up my React application. My react application is hard-coded to send requests to www.domain.com:7080, however all requests fail. In the browser's console I can see the following error:
The page at 'https:// domain.com/' was loaded over HTTPS, but requested an insecure XMLHttpRequest endpoint 'http:// domain.com:7080/auth/login'. This request has been blocked; the content must be served over HTTPS.
My NGINX configuration:
server {
listen 443 ssl; # managed by Certbot
root /var/www/ui;
server_name www.domain.com domain.com;
ssl_certificate /etc/letsencrypt/live/domain.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/domain.com/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
location / {
index index.html;
}
}
server {
listen 80;
if ($host = domain.com) {
return 301 $host$request_uri;
} # managed by Certbot
server_name www.domain.com domain.com;
return 301 https://$host$request_uri; # managed by Certbot
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header REMOTE-HOST $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
My back-end application is served over http and I'd like to permit the front-end to talk to the back-end service.
I couldn't locate a similar question or tutorial on how I would go about solving this therefore I'm hoping to get some answers here :3
create api endpoint in your domain i.e. www.domain.com/api and configure nginx to pass traffic from that endpoint to your backend with proxy_pass directive. You'll have secure connection from your users and won't need to change anything in your backend server.
I want to create a high availability with Oracle Weblogic. First, I create a cluster called MyCluster and add two servers (Server1 and Server2) to MyCluster. I use Nginx as a load balancer.
I follow the tutorial from https://www.oracle.com/webfolder/technetwork/tutorials/obe/fmw/wls/12c/12-ManageSessions--4478/session.htm#t1 to replicate session in memory.
Here is my nginx config:
upstream myweb {
server server1:38080 weight=1;
server server2:38080 weight=1;
server server3:38080 weight=1;
}
server {
listen 80;
server_name nginxHost;
access_log /var/log/nginx/nginxHost.access.log main;
error_log /var/log/nginx/nginxHost.error.log warn;
location / {
proxy_pass http://myweb/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
}
When I test session replication, I have a problem. If Server1 is running and Server2 is shutdown now, I connect my application which is on Server1. I power on Server2 and wait it complete startup. Then I shutdown Server1 and refresh browser. The session disappear.
Finally, I find I have to refresh browser after Server2 is running. Is there any way to replicate session when servers start?
I have a Nginx reverse proxy redirecting to a lighttpd server in the same machine. This reverse proxy works on HTTPS so I want to tell to lighttpd that HTTPS is being used as protocol instead of HTTP. Here is my Nginx confuguration.
server {
server_name mydomain.com;
merge_slashes off;
rewrite ^(.*?)//+(.*?)$ $1/$2 permanent;
location / {
proxy_pass http://localhost:8088/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-Ssl on;
}
listen [::]:443 ssl ipv6only=on; # managed by Certbot
listen 443 ssl; # managed by Certbot
# SSL settings
}
server {
if ($host = mydomain.com) {
return 301 https://$host$request_uri;
}
listen 80;
listen [::]:80;
}
The lighttpd server is running a python application that uses web.py module but the returned value by web.ctx.protocol still is HTTP when it should be HTTPS. It looks like lighttpd ignores the X-Forwarded-Proto header sent by Nginx.
What am I doing wrong? Is there any additional configuration to be done?
Thanks.
You have to configure lighttpd to trust headers from upstream. Use mod_extforward in lighttpd. See https://redmine.lighttpd.net/projects/lighttpd/wiki/Docs_ModExtForward
Better than your many headers above, both nginx and lighttpd (via mod_extforward) support RFC 7239 Forwarded header.
https://www.nginx.com/resources/wiki/start/topics/examples/forwarded/
Use of the "Forwarded" header should be preferred.
I have set up an nginx reverse proxy to node essentially using this set up reproduced below:
upstream nodejs {
server localhost:3000;
}
server {
listen 8080;
server_name localhost;
root ~/workspace/test/app;
location / {
try_files $uri $uri/ #nodejs;
}
location #nodejs {
proxy_redirect off;
proxy_http_version 1.1;
proxy_pass http://nodejs;
proxy_set_header Host $host ;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
Now all my AJAX POST requests travel just fine to the node with this set up, but I am polling for files afterward that I cannot find when I make a clientside AJAX GET request to the node server (via this nginx proxy).
For example, for a clientside javascript request like .get('Users/myfile.txt') the browser will look for the file on localhost:8080 but won't find it because it's actually written to localhost:3000
http://localhost:8080/Users/myfile.txt // what the browser searches for
http://localhost:3000/Users/myfile.txt // where the file really is
How do I set up the proxy to navigate through to this file?
Okay, I got it working. The set up in the nginx.conf file posted above is just fine. This problem was never an nginx problem. The problem was in my index.js file over on the node server.
When I got nginx to serve all the static files, I commented out the following line from index.js
app.use(express.static('Users')); // please don't comment this out thank you
It took me a while to troubleshoot my way back to this as I was pretty wrapped up in understanding nginx. My thinking at the time was that if nginx is serving static files why would I need express to serve them? Without this line however, express won't serve any files at all obviously.
Now with express serving static files properly, nginx handles all static files from the web app and node handles all the files from the backend and all is good.
Thanks to Keenan Lawrence for the guidance and AR7 for the config!
I've got an Nginx/Gunicorn/Django server deployed on a Centos 6 machine with only the SSL port (443) visible to the outside world. So unless the server is called with the https://, you won't get any response. If you call it with an http://domain:443, you'll merely get a 400 Bad Request message. Port 443 is the only way to hit the server.
I'm using Nginx to serve my static files (CSS, etc.) and all other requests are handled by Gunicorn, which is running Django at http://localhost:8000. So, navigating to https://domain.com works just fine, as do links within the admin site, but when I submit a form in the Django admin, the https is lost on the redirect and I'm sent to http://domain.com/request_uri which fails to reach the server. The POST action does work properly even so and the database is updated.
My configuration file is listed below. The location location / section is where I feel like the solution should be found. But it doesn't seem like the proxy_set_header X-* directives have any effect. Am I missing a module or something? I'm running nginx/1.0.15.
Everything I can find on the internet points to the X-Forwarded-Protocol https like it should do something, but I get no change. I'm also unable to get the debugging working on the remote server, though my next step may have to be compiling locally with debugging enabled to get some more clues. The last resort is to expose port 80 and redirect everything...but that requires some paperwork.
[http://pastebin.com/Rcg3p6vQ](My nginx configure arguments)
server {
listen 443 ssl;
ssl on;
ssl_certificate /path/to/cert.crt;
ssl_certificate_key /path/to/key.key;
ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers HIGH:!aNULL:!MD5;
server_name example.com;
root /home/gunicorn/project/app;
access_log /home/gunicorn/logs/access.log;
error_log /home/gunicorn/logs/error.log debug;
location /static/ {
autoindex on;
root /home/gunicorn;
}
location / {
proxy_pass http://localhost:8000/;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Scheme $scheme;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Protocol https;
}
}
Haven't had time yet to understand exactly what these two lines do, but removing them solved my problems:
proxy_redirect off;
proxy_set_header Host $host;