I have a stateful spring application and I want to deploy it to kubernetes cluster. There will be more than one instance of the application so i need to enable sticy session using ingress-nginx controller. I made the following configuration:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-nginx
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/affinity: "cookie"
nginx.ingress.kubernetes.io/session-cookie-name: "JSESSIONID"
nginx.ingress.kubernetes.io/session-cookie-expires: "172800"
nginx.ingress.kubernetes.io/session-cookie-max-age: "172800"
nginx.ingress.kubernetes.io/session-cookie-path: /ingress-test
# UPDATE THIS LINE ABOVE
spec:
rules:
- http:
paths:
- path: /ingress-test
backend:
serviceName: ingress-test
servicePort: 31080
ingress-nginx redirect subsequent request to correct pod if login is successful. However, it sometimes switches to other pod just after JSESSIONID is changed (JSESSIONID cookie is changed by spring-security afer successful login) and frontend redirects back to login page even user credentials are correct. Is there anyone that tried ingress-nginx with spring-security?
Best Regards
Following change fixed the problem. Without a host definition in rules, ingress-nginx doesn't set session cookie.
There is an open issue: https://github.com/kubernetes/ingress-nginx/issues/3989
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-nginx
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/affinity: "cookie"
nginx.ingress.kubernetes.io/session-cookie-name: "route"
nginx.ingress.kubernetes.io/session-cookie-expires: "172800"
nginx.ingress.kubernetes.io/session-cookie-max-age: "172800"
nginx.ingress.kubernetes.io/session-cookie-path: /ingress-test
# UPDATE THIS LINE ABOVE
spec:
rules:
- host: www.domainname.com
http:
paths:
- path: /ingress-test
backend:
serviceName: ingress-test
servicePort: 31080
The reason spring changes the cookie is to prevent session fixation (more information can be found here: https://www.owasp.org/index.php/Session_fixation).
In your case you are using the same cookie for the sticky routing policy that is used by spring for session handling.
I suggest to use a different cookie name - it will be created by nginx and there is no need to use a cookie that is used by the application.
Related
I'm trying to create HTTPS ingress for my node.js authentication (auth) REST service in AKS, but I'm getting a 502 Bad Gateway response.
Here's my deployment and service definitions:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: auth
namespace: auth
labels:
app: auth
spec:
selector:
matchLabels:
app: auth
replicas: 1
template:
metadata:
labels:
app: auth
spec:
imagePullSecrets:
- name: docker-hub-creds
containers:
- name: auth
image: ***image***
ports:
- containerPort: 80
name: auth
---
apiVersion: v1
kind: Service
metadata:
name: auth
namespace: auth
spec:
selector:
app: auth
ports:
- protocol: TCP
port: 80
targetPort: 80
I think that's all pretty basic and it seems to work ok. I can see the service running and if I expose a node-port then I can access it with no problems. The service responds to well-formed POST requests on the /auth path with a JWT.
I have configured an Azure Application Gateway following Microsoft's instructions, and following the troubleshooting guide leads me to believe that the installation has worked ok. I have also checked through the web-ui and there appear to be no errors. Finally, I worked through the support options and the automated analysis of my cluster found no major configuration issues.
Next, I tried to create an HTTPS ingress route for my service, and this is where it goes wrong. This is made more complicated by the dynamic generation of certificates for TLS.
The ingress definition looks like this:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: auth-in
namespace: auth
annotations:
kubernetes.io/ingress.class: azure/application-gateway
cert-manager.io/cluster-issuer: letsencrypt-staging
cert-manager.io/acme-challenge-type: http01
ingress.kubernetes.io/rewrite-target: /$1
nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
tls:
- hosts:
- ***hostname***
secretName: ***secret***
rules:
- host: ***same hostname***
http:
paths:
- backend:
serviceName: auth
servicePort: 80
path: /api/(auth/.*)
I have two rewrite-targets in there because I can't determine which one this ingress controller uses. All the example from the web use the nginx. prefix so I added it in desperation, despite thinking that it's probably not necessary.
Accessing the service through: ***hostname***/api/auth results in a Bad Gateway error.
I have checked through the portal and I can see the route is registered, listeners and rules are there, and my service is listed in the backend pools, but there is nothing in the 'rewrite' tabs. I expected to see something in the rewrite tabs.
I've tooled my service to log all access, and the logs show this, repeatedly:
{"level":30,"time":1611739355140,"pid":17,"hostname":"auth-6c7757bb89-d72td","msg":"Req-URL: /api/(auth/.*)"}
Describing the ingress gives me this:
Name: auth-in
Namespace: auth
Address: **redacted***
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
***redacted cert name** terminates **hostname***
Rules:
Host Path Backends
---- ---- --------
***hostname***
/api/(auth/.*) auth:80 10.0.0.69:80)
Annotations: cert-manager.io/acme-challenge-type: http01
cert-manager.io/cluster-issuer: letsencrypt-staging
ingress.kubernetes.io/rewrite-target: /$1
kubernetes.io/ingress.class: azure/application-gateway
nginx.ingress.kubernetes.io/rewrite-target: /$1
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CreateCertificate 43m cert-manager Successfully created Certificate "***cert-name***"
Two things to note. 1st that the logs show that the URL isn't being rewritten -- it's being passed exactly as the path shows, including the regex part. 2nd, that the Default Backend entry in the ingress description shows an error. I'm not sure that the 2nd one matters, but the 1st is clearly wrong.
I am keen to discover how to diagnose the problem and then fix it.
Since you are using AGIC you can include Backend Path Prefix annotation appgw.ingress.kubernetes.io/backend-path-prefix: "/"
The Ingress will be like this:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: auth-in
namespace: auth
annotations:
kubernetes.io/ingress.class: azure/application-gateway
cert-manager.io/cluster-issuer: letsencrypt-staging
cert-manager.io/acme-challenge-type: http01
appgw.ingress.kubernetes.io/backend-path-prefix: "/"
spec:
tls:
- hosts:
- ***hostname***
secretName: ***secret***
rules:
- host: ***same hostname***
http:
paths:
- backend:
serviceName: auth
servicePort: 80
path: /api/auth/*
AGIC on Nov 12 '21 has also included a rewrite-rule-set as part of this PR. For rewrite-rule, you can use the rewrite-rule annotation.
appgw.ingress.kubernetes.io/rewrite-rule-set: <rewrite rule set>
I have this spring boot app container in a pod and a service mapped to access the app, inside a minikube cluster. when I use an exec command and try to access API endpoints it does work fine. But after I exposed it using an Nginx ingress controller it shows Whitelabel 404 error for every request.
I did add the ingress minikube addon and configured the ingress controller using a yaml file.
here's the ingress.yaml file.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: example-ingress
annotations:
nginx.ingress.kubernetes.io/use-regex: "true"
ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- http:
paths:
- path: /path/.*
backend:
serviceName: spring-app
servicePort: 8080
Any tips on how to solve this?? Thanks in advance
This should fix it:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: example-ingress
annotations:
nginx.ingress.kubernetes.io/use-regex: "true"
ingress.kubernetes.io/rewrite-target: /$1
spec:
rules:
- http:
paths:
- path: /path/(.*)
backend:
serviceName: spring-app
servicePort: 8080
For more information, please check here.
I have an GKE ingress with both Http and Https. I want to redirect the traffic from port 80 to port 443.
I found this:
https://github.com/kubernetes/ingress-gce/issues/1075
which let to this:
https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#https_redirect.
The proposed solution adds a FrontendConfig with a RedirectToHttps flag which uses some LoadBalancer functionality. Yet when I try to add the FrontendEndConfig, I get the following error:
error: unable to recognize "ssl.yaml": no matches for kind "FrontendConfig" in version "networking.gke.io/v1beta1"
I have also tried 'networking.gke.io/v1' and 'v1beta2'.
The latest GKE version available in my zone is 1.17.13-gke.2001. I have recently launched the cluster so although I don't know how to check the GKE version, I reckon it's running on the latest version.
Anyone has a clue why my kubectl doesn't recognize this kind?
Ingress yaml:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: basic-ingress
annotations:
FrontendConfig: my-frontend-config
kubernetes.io/ingress.global-static-ip-name: 'web-static-ip'
networking.gke.io/managed-certificates: mycertificate
# kubernetes.io/ingress.allow-http: "false"
spec:
rules:
- http:
paths:
- path: /*
backend:
serviceName: frontend
servicePort: 80
- path: /api/*
backend:
serviceName: backend
servicePort: 80
Redirect yaml:
apiVersion: networking.gke.io/v1beta1
kind: FrontendConfig
metadata:
name: my-frontend-config
spec:
redirectToHttps:
enabled: true
Thank you for pointing me in the right direction!
I had to upgrade the cluster as MrKoopaKiller indicated and also changed the annotation:
FrontendConfig: my-frontend-config
to:
networking.gke.io/v1beta1.FrontendConfig: "my-frontend-config"
and it worked!
also: make sure you have:
kubernetes.io/ingress.allow-http: "true"
I am trying to make the websocket service work on Azure Kubernetes Cluster on our organization environment.
My existing environment also have REST api and Angular application working on ingress with ssl.
But when I added the websocket service on the ingress it is not working.
So, I tried to use Azure Free Subscription to first implement the same WITHOUT SSL. For my applications I enabled Http Routing and using the annotation addon-http-application-routing.
I am getting below error.
'ws://40.119.7.246/ws' failed: Error during WebSocket handshake: Unexpected response code: 404
Please help in validating where I am doing wrong?
Below are the details of the configuration.
Dockerfile
FROM node:alpine
WORKDIR /app
COPY package*.json /app/
RUN npm install
COPY ./ /app/
RUN npm run build
CMD ["node","./dist/server.js"]
EXPOSE 8010
socketserver.yaml - Contains Demployment & Service.
apiVersion: apps/v1
kind: Deployment
metadata:
name: socketserver
spec:
replicas: 1
selector:
matchLabels:
app: socketserver
template:
metadata:
labels:
app: socketserver
spec:
containers:
- name: socketserver
image: regkompella.azurecr.io/socketserver:1.0.0
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
memory: "128Mi"
cpu: "500m"
ports:
- containerPort: 8010
imagePullSecrets:
- name: regkompella-azurecr-io
---
apiVersion: v1
kind: Service
metadata:
name: socketserver-svc
spec:
ports:
- port: 80
protocol: TCP
targetPort: 8010
selector:
app: socketserver
type: ClusterIP
---
ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: demo-ingress
annotations:
kubernetes.io/ingress.class: addon-http-application-routing
nginx.ingress.kubernetes.io/cors-allow-methods: "GET, PUT, POST, DELETE, OPTIONS"
nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/proxy-body-size: 10m
nginx.ingress.kubernetes.io/websocket-services: socketserver-svc
nginx.org/websocket-services: socketserver-svc
spec:
rules:
- host: demosocket.com
- http:
paths:
- path: /
backend:
serviceName: angular-application-svc
servicePort: 80
- path: /ws
backend:
serviceName: socketserver-svc
servicePort: 80
After reading through a lot of articles and referring some of the github forums (Added referenced articles below). I come to a point where my websocket implementation started working after doing the two things. I am not sure yet if, this is the right way to do it or not. I achieved to this solution purely on trail and error method. Hence, I request everyone who have good grasp, kindly suggest if there is a better way to solve my problem. Always take my steps with a pinch of salt.
Installed the NGINX Ingress controller from the link.
As I am using Azure Kubernetes Services, I applied the below yaml from the document.
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud/deploy.yaml
Next I made necessary changes to my demo-ingress configuration for my services.
I come to know that kubernetes.io/ingress.class: addon-http-application-routing annotation doesn't support websocketing. So, had to disable it.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: demo-ingress
annotations:
# this one annotation is making the websocket work.
nginx.ingress.kubernetes.io/websocket-services: socketserver-svc
# this one I left as-is. And not playing any role for this websocket
# implementation to work
nginx.ingress.kubernetes.io/cors-allow-methods: "GET, PUT, POST, DELETE, OPTIONS"
nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/proxy-body-size: 10m
# I thought sticky session is also required for websocket to work. But seems
# this has no effect after I installed nginx ingress controller.
# so disabled all the below annotations also.
#nginx.org/websocket-services: socketserver-svc
#nginx.ingress.kubernetes.io/affinity: cookie
#nginx.ingress.kubernetes.io/affinity-mode: balanced
#nginx.ingress.kubernetes.io/session-cookie-samesite: Strict
#kubernetes.io/ingress.class: nginx
#kubernetes.io/ingress.class: addon-http-application-routing
spec:
rules:
- host: demosocket.com
- http:
paths:
- path: /ws
backend:
serviceName: socketserver-svc
servicePort: 80
I tried to access through the public IP address. And I can be able to successfully send and receive messages.
ws://52.188.38.118/ws
Now, what if I want to make the websocket implementation work without installing NGINX Ingress Controller ( indicated on step 1) and want to try to use default ingress controller coming with AKS/minikube. The answer is below.
From the steps above,
a) Avoid Step 1: Installing NGINX Ingress Controller.
b) Only change that need to be made on ingress is below. Use the below annotations instead of the annotations indicated on Step 2 on the ingress yaml file. Things will start working.
# this annotation is making my web application also work if I plan to configure something in future.
nginx.ingress.kubernetes.io/ingress.class: nginx
# this one annotation is making the websocket work.
nginx.ingress.kubernetes.io/websocket-services: socketserver-svc
# by default ssl is true - as I am trying locally and want to disable ssl-# redirect. So set this to false.
nginx.ingress.kubernetes.io/ssl-redirect: "false"
# Below are just additional annotation to allow CORS etc.
nginx.ingress.kubernetes.io/cors-allow-methods: "GET, PUT, POST, DELETE, OPTIONS"
nginx.ingress.kubernetes.io/proxy-body-size: 10m
Referenced Articles:
https://medium.com/flant-com/comparing-ingress-controllers-for-kubernetes-9b397483b46b
https://kubernetes.github.io/ingress-nginx/deploy/#azure
Mr. dstrebel's comments -> https://github.com/Azure/AKS/issues/768
I typically recommend just setting up a Ingress Controller on the cluster and not enabling "http-application-routing", as there's a lot of limitations to it. The goal with HTTP Application ROuting was for users to get setup quickly with Ingress, but not really for production deployments due to the limitations of the configuration.
DenisBiondic commented on Oct 2, 2018 -> https://github.com/Azure/AKS/issues/672
I am not 100% certain, since I don't use application routing feature, but >I think it does not use the https://github.com/nginxinc/kubernetes-ingress/tree/master/examples/websocket controller but rather the https://github.com/kubernetes/ingress-nginx. In case of the latter, I think enabling session affinity with cookies might be enough: https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/annotations.md#session-affinity
In your case you are using the wrong annotation which does not work with application routing ingress controller under the hood.
I welcome suggestions and best practices.
I am using kubectl to run Kubernetes on a Kops controlled cluster on AWS. I want to insert the Strict-Transport-Security header into the pages that are served from our site. My ingress currently forces all traffic to HTTPS, but ignores the annotations I have in my
ingress.yaml:
nginx.ingress.kubernetes.io/hsts: "true"
When I run kubectl get ingress <ingressname> -o yaml, I can see {"annotations":{"nginx.ingress.kubernetes.io/hsts":"true", but as far as I can tell, there is no sign of HSTS in the headers.
I have tried to make this happen from the configmap, but it also doesn't work.
I am using the quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.12.0 for the ingresscontroller, and my kubectl server version is v1.8.6.
The ingress deployment, service, and ingress itself all respond to changes, though putting gibberish in to the annotations in the ingress.yaml doesn't seem to break anything.
What am I doing wrong?
It may depend if you have actually enabled HTTPS on ingress itself. In my case I'm offloading SSL on AWS ELB thus seems to have to force the HSTS header. You may want to first try shorter max-age and drop out includeSubDomains. Use if you know what you are doing ;-) Check out this issue
You can force it by using config map similar to one below:
---
apiVersion: v1
data:
Strict-Transport-Security: "max-age=15768000 ; includeSubDomains"
kind: ConfigMap
metadata:
name: custom-headers-external-sts
namespace: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-configuration
namespace: ingress-nginx
labels:
app: ingress-nginx
data:
add-headers: "ingress-nginx/custom-headers-external-sts"
You can not directly add hsts header in annotations section in ingress-nginx yml. Because, hsts header is not available in ingress-nginx annotations.
You can use configuration-snippet to add additional headers in ingress-nginx annotations.
Just add it as mentioned below,
annotations:
nginx.ingress.kubernetes.io/configuration-snippet: |
more_set_headers "Strict-Transport-Security: max-age=31536000; includeSubDomains; preload";
Above solution is tried and tested!
Here is the link I had referred to: Configuration snippet
I seem to be experiencing the same problem, although I'm applying the changes via configmap.
Kubernetes: 1.8.6
Image: gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.15
Here's the config I've set:
kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-ingress
namespace: kube-ingress
labels:
k8s-addon: nginx-ingress.addons.k8s.io
data:
allow-backend-server-header: "true"
hsts: "true"
hsts-include-subdomains: "true"
hsts-max-age: "31536000"
hsts-preload: "true"
use-proxy-protocol: "true"