Kubernetes ingress not enforcing inserting hsts into headers - hsts

I am using kubectl to run Kubernetes on a Kops controlled cluster on AWS. I want to insert the Strict-Transport-Security header into the pages that are served from our site. My ingress currently forces all traffic to HTTPS, but ignores the annotations I have in my
ingress.yaml:
nginx.ingress.kubernetes.io/hsts: "true"
When I run kubectl get ingress <ingressname> -o yaml, I can see {"annotations":{"nginx.ingress.kubernetes.io/hsts":"true", but as far as I can tell, there is no sign of HSTS in the headers.
I have tried to make this happen from the configmap, but it also doesn't work.
I am using the quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.12.0 for the ingresscontroller, and my kubectl server version is v1.8.6.
The ingress deployment, service, and ingress itself all respond to changes, though putting gibberish in to the annotations in the ingress.yaml doesn't seem to break anything.
What am I doing wrong?

It may depend if you have actually enabled HTTPS on ingress itself. In my case I'm offloading SSL on AWS ELB thus seems to have to force the HSTS header. You may want to first try shorter max-age and drop out includeSubDomains. Use if you know what you are doing ;-) Check out this issue
You can force it by using config map similar to one below:
---
apiVersion: v1
data:
Strict-Transport-Security: "max-age=15768000 ; includeSubDomains"
kind: ConfigMap
metadata:
name: custom-headers-external-sts
namespace: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-configuration
namespace: ingress-nginx
labels:
app: ingress-nginx
data:
add-headers: "ingress-nginx/custom-headers-external-sts"

You can not directly add hsts header in annotations section in ingress-nginx yml. Because, hsts header is not available in ingress-nginx annotations.
You can use configuration-snippet to add additional headers in ingress-nginx annotations.
Just add it as mentioned below,
annotations:
nginx.ingress.kubernetes.io/configuration-snippet: |
more_set_headers "Strict-Transport-Security: max-age=31536000; includeSubDomains; preload";
Above solution is tried and tested!
Here is the link I had referred to: Configuration snippet

I seem to be experiencing the same problem, although I'm applying the changes via configmap.
Kubernetes: 1.8.6
Image: gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.15
Here's the config I've set:
kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-ingress
namespace: kube-ingress
labels:
k8s-addon: nginx-ingress.addons.k8s.io
data:
allow-backend-server-header: "true"
hsts: "true"
hsts-include-subdomains: "true"
hsts-max-age: "31536000"
hsts-preload: "true"
use-proxy-protocol: "true"

Related

GKE ingress Https Redirect - FrontendConfig not recognized

I have an GKE ingress with both Http and Https. I want to redirect the traffic from port 80 to port 443.
I found this:
https://github.com/kubernetes/ingress-gce/issues/1075
which let to this:
https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#https_redirect.
The proposed solution adds a FrontendConfig with a RedirectToHttps flag which uses some LoadBalancer functionality. Yet when I try to add the FrontendEndConfig, I get the following error:
error: unable to recognize "ssl.yaml": no matches for kind "FrontendConfig" in version "networking.gke.io/v1beta1"
I have also tried 'networking.gke.io/v1' and 'v1beta2'.
The latest GKE version available in my zone is 1.17.13-gke.2001. I have recently launched the cluster so although I don't know how to check the GKE version, I reckon it's running on the latest version.
Anyone has a clue why my kubectl doesn't recognize this kind?
Ingress yaml:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: basic-ingress
annotations:
FrontendConfig: my-frontend-config
kubernetes.io/ingress.global-static-ip-name: 'web-static-ip'
networking.gke.io/managed-certificates: mycertificate
# kubernetes.io/ingress.allow-http: "false"
spec:
rules:
- http:
paths:
- path: /*
backend:
serviceName: frontend
servicePort: 80
- path: /api/*
backend:
serviceName: backend
servicePort: 80
Redirect yaml:
apiVersion: networking.gke.io/v1beta1
kind: FrontendConfig
metadata:
name: my-frontend-config
spec:
redirectToHttps:
enabled: true
Thank you for pointing me in the right direction!
I had to upgrade the cluster as MrKoopaKiller indicated and also changed the annotation:
FrontendConfig: my-frontend-config
to:
networking.gke.io/v1beta1.FrontendConfig: "my-frontend-config"
and it worked!
also: make sure you have:
kubernetes.io/ingress.allow-http: "true"

upstream connect error or disconnect/reset before headers. reset reason: connection failure. Spring Boot and java 11

I'm having a problem migrating my pure Kubernetes app to an Istio managed. I'm using Google Cloud Platform (GCP), Istio 1.4, Google Kubernetes Engine (GKE), Spring Boot and JAVA 11.
I had the containers running in a pure GKE environment without a problem. Now I started the migration of my Kubernetes cluster to use Istio. Since then I'm getting the following message when I try to access the exposed service.
upstream connect error or disconnect/reset before headers. reset reason: connection failure
This error message looks like a really generic. I found a lot of different problems, with the same error message, but no one was related to my problem.
Bellow the version of the Istio:
client version: 1.4.10
control plane version: 1.4.10-gke.5
data plane version: 1.4.10-gke.5 (2 proxies)
Bellow my yaml files:
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
account: tree-guest
name: tree-guest-service-account
---
apiVersion: v1
kind: Service
metadata:
labels:
app: tree-guest
service: tree-guest
name: tree-guest
spec:
ports:
- name: http
port: 8080
targetPort: 8080
selector:
app: tree-guest
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: tree-guest
version: v1
name: tree-guest-v1
spec:
replicas: 1
selector:
matchLabels:
app: tree-guest
version: v1
template:
metadata:
labels:
app: tree-guestaz
version: v1
spec:
containers:
- image: registry.hub.docker.com/victorsens/tree-quest:circle_ci_build_00923285-3c44-4955-8de1-ed578e23c5cf
imagePullPolicy: IfNotPresent
name: tree-guest
ports:
- containerPort: 8080
serviceAccount: tree-guest-service-account
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: tree-guest-gateway
spec:
selector:
istio: ingressgateway # use istio default controller
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: tree-guest-virtual-service
spec:
hosts:
- "*"
gateways:
- tree-guest-gateway
http:
- match:
- uri:
prefix: /v1
route:
- destination:
host: tree-guest
port:
number: 8080
To apply the yaml file I used the following argument:
kubectl apply -f <(istioctl kube-inject -f ./tree-guest.yaml)
Below the result of the Istio proxy argument, after deploying the application:
istio-ingressgateway-6674cc989b-vwzqg.istio-system SYNCED SYNCED SYNCED SYNCED
istio-pilot-ff4489db8-2hx5f 1.4.10-gke.5 tree-guest-v1-774bf84ddd-jkhsh.default SYNCED SYNCED SYNCED SYNCED istio-pilot-ff4489db8-2hx5f 1.4.10-gke.5
If someone have a tip about what is going wrong, please let me know. I'm stuck in this problem for a couple of days.
Thanks.
As #Victor mentioned the problem here was the wrong yaml file.
I solve it. In my case the yaml file was wrong. I reviewed it and the problem now is solved. Thank you guys., – Victor
If you're looking for yaml samples I would suggest to take a look at istio github samples.
As 503 upstream connect error or disconnect/reset before headers. reset reason: connection failure occurs very often I set up little troubleshooting answer, there are another questions with 503 error which I encountered for several months with answers, useful informations from istio documentation and things I would check.
Examples with 503 error:
Istio 503:s between (Public) Gateway and Service
IstIO egress gateway gives HTTP 503 error
Istio Ingress Gateway with TLS termination returning 503 service unavailable
how to terminate ssl at ingress-gateway in istio?
Accessing service using istio ingress gives 503 error when mTLS is enabled
Common cause of 503 errors from istio documentation:
https://istio.io/docs/ops/best-practices/traffic-management/#avoid-503-errors-while-reconfiguring-service-routes
https://istio.io/docs/ops/common-problems/network-issues/#503-errors-after-setting-destination-rule
https://istio.io/latest/docs/concepts/traffic-management/#working-with-your-applications
Few things I would check first:
Check services ports name, Istio can route correctly the traffic if it knows the protocol. It should be <protocol>[-<suffix>] as mentioned in istio
documentation.
Check mTLS, if there are any problems caused by mTLS, usually those problems would result in error 503.
Check if istio works, I would recommend to apply bookinfo application example and check if it works as expected.
Check if your namespace is injected with kubectl get namespace -L istio-injection
If the VirtualService using the subsets arrives before the DestinationRule where the subsets are defined, the Envoy configuration generated by Pilot would refer to non-existent upstream pools. This results in HTTP 503 errors until all configuration objects are available to Pilot.
I landed exactly here with exactly similar symptoms.
But in my case I had to
switch pod listen address from 172.0.0.1 to 0.0.0.0
which solved my issue

Spring Boot HTTP made secure (HTTPS) with Kubernetes Ingress

I am new at Kubernetes and GKE. I have some microservices written in Spring Boot 2 and deployed from GitHub to GKE. I would like to make these services secure and I want to know if it's possible to use ingress on my gateway microservice to make the entry point secure just like that. I created an ingress with HTTPS but it seems all my health checks are failing.
Is it possible to make my architecture secure just by using ingress and not change the spring boot apps?
Yes, It would be possible to use a GKE ingress given your scenario, there is an official guide on how to do this step by step.
Additionally, here's a step by step guide on how to implement Google Managed certs.
Also, I understand that my response is somewhat general, but I can only help you so much without knowing your GKE infrastructure (like your DNS name for said certificate among other things).
Remember that you must implement this directly on your GKE infrastructure and not on your GCP side, if you modify or create something new outside GKE but that it's linked to GKE, you might see that either your deployment rolled back after a certain time or that stopped working after a certain time.
Edit:
I will assume several things here, and since I don't have your Spring Boot 2 deployment yaml file, I will replace that with an nginx deployment.
cert.yaml
apiVersion: networking.gke.io/v1beta1
kind: ManagedCertificate
metadata:
name: ssl-cert
spec:
domains:
- example.com
nginx.yaml
apiVersion: "apps/v1"
kind: "Deployment"
metadata:
name: "nginx"
namespace: "default"
labels:
app: "nginx"
spec:
replicas: 1
selector:
matchLabels:
app: "nginx"
template:
metadata:
labels:
app: "nginx"
spec:
containers:
- name: "nginx-1"
image: "nginx:latest"
nodeport.yaml (please modify "targetPort: 80" to your needs)
apiVersion: "v1"
kind: "Service"
metadata:
name: "nginx-service"
namespace: "default"
labels:
app: "nginx"
spec:
ports:
- protocol: "TCP"
port: 80
targetPort: 80
selector:
app: "nginx"
type: "NodePort"
ingress-cert.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: nginx-ingress
annotations:
networking.gke.io/managed-certificates: ssl-cert
spec:
backend:
serviceName: nginx-service
servicePort: 80
Keep in mind that assuming your DNS name "example.com" is pointing into your Load Balancer external IP, it could take a while to your SSL certificate to be created and applied.

Error on Websocket Implementation on Azure Kubernetes Cluster Not working

I am trying to make the websocket service work on Azure Kubernetes Cluster on our organization environment.
My existing environment also have REST api and Angular application working on ingress with ssl.
But when I added the websocket service on the ingress it is not working.
So, I tried to use Azure Free Subscription to first implement the same WITHOUT SSL. For my applications I enabled Http Routing and using the annotation addon-http-application-routing.
I am getting below error.
'ws://40.119.7.246/ws' failed: Error during WebSocket handshake: Unexpected response code: 404
Please help in validating where I am doing wrong?
Below are the details of the configuration.
Dockerfile
FROM node:alpine
WORKDIR /app
COPY package*.json /app/
RUN npm install
COPY ./ /app/
RUN npm run build
CMD ["node","./dist/server.js"]
EXPOSE 8010
socketserver.yaml - Contains Demployment & Service.
apiVersion: apps/v1
kind: Deployment
metadata:
name: socketserver
spec:
replicas: 1
selector:
matchLabels:
app: socketserver
template:
metadata:
labels:
app: socketserver
spec:
containers:
- name: socketserver
image: regkompella.azurecr.io/socketserver:1.0.0
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
memory: "128Mi"
cpu: "500m"
ports:
- containerPort: 8010
imagePullSecrets:
- name: regkompella-azurecr-io
---
apiVersion: v1
kind: Service
metadata:
name: socketserver-svc
spec:
ports:
- port: 80
protocol: TCP
targetPort: 8010
selector:
app: socketserver
type: ClusterIP
---
ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: demo-ingress
annotations:
kubernetes.io/ingress.class: addon-http-application-routing
nginx.ingress.kubernetes.io/cors-allow-methods: "GET, PUT, POST, DELETE, OPTIONS"
nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/proxy-body-size: 10m
nginx.ingress.kubernetes.io/websocket-services: socketserver-svc
nginx.org/websocket-services: socketserver-svc
spec:
rules:
- host: demosocket.com
- http:
paths:
- path: /
backend:
serviceName: angular-application-svc
servicePort: 80
- path: /ws
backend:
serviceName: socketserver-svc
servicePort: 80
After reading through a lot of articles and referring some of the github forums (Added referenced articles below). I come to a point where my websocket implementation started working after doing the two things. I am not sure yet if, this is the right way to do it or not. I achieved to this solution purely on trail and error method. Hence, I request everyone who have good grasp, kindly suggest if there is a better way to solve my problem. Always take my steps with a pinch of salt.
Installed the NGINX Ingress controller from the link.
As I am using Azure Kubernetes Services, I applied the below yaml from the document.
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud/deploy.yaml
Next I made necessary changes to my demo-ingress configuration for my services.
I come to know that kubernetes.io/ingress.class: addon-http-application-routing annotation doesn't support websocketing. So, had to disable it.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: demo-ingress
annotations:
# this one annotation is making the websocket work.
nginx.ingress.kubernetes.io/websocket-services: socketserver-svc
# this one I left as-is. And not playing any role for this websocket
# implementation to work
nginx.ingress.kubernetes.io/cors-allow-methods: "GET, PUT, POST, DELETE, OPTIONS"
nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/proxy-body-size: 10m
# I thought sticky session is also required for websocket to work. But seems
# this has no effect after I installed nginx ingress controller.
# so disabled all the below annotations also.
#nginx.org/websocket-services: socketserver-svc
#nginx.ingress.kubernetes.io/affinity: cookie
#nginx.ingress.kubernetes.io/affinity-mode: balanced
#nginx.ingress.kubernetes.io/session-cookie-samesite: Strict
#kubernetes.io/ingress.class: nginx
#kubernetes.io/ingress.class: addon-http-application-routing
spec:
rules:
- host: demosocket.com
- http:
paths:
- path: /ws
backend:
serviceName: socketserver-svc
servicePort: 80
I tried to access through the public IP address. And I can be able to successfully send and receive messages.
ws://52.188.38.118/ws
Now, what if I want to make the websocket implementation work without installing NGINX Ingress Controller ( indicated on step 1) and want to try to use default ingress controller coming with AKS/minikube. The answer is below.
From the steps above,
a) Avoid Step 1: Installing NGINX Ingress Controller.
b) Only change that need to be made on ingress is below. Use the below annotations instead of the annotations indicated on Step 2 on the ingress yaml file. Things will start working.
# this annotation is making my web application also work if I plan to configure something in future.
nginx.ingress.kubernetes.io/ingress.class: nginx
# this one annotation is making the websocket work.
nginx.ingress.kubernetes.io/websocket-services: socketserver-svc
# by default ssl is true - as I am trying locally and want to disable ssl-# redirect. So set this to false.
nginx.ingress.kubernetes.io/ssl-redirect: "false"
# Below are just additional annotation to allow CORS etc.
nginx.ingress.kubernetes.io/cors-allow-methods: "GET, PUT, POST, DELETE, OPTIONS"
nginx.ingress.kubernetes.io/proxy-body-size: 10m
Referenced Articles:
https://medium.com/flant-com/comparing-ingress-controllers-for-kubernetes-9b397483b46b
https://kubernetes.github.io/ingress-nginx/deploy/#azure
Mr. dstrebel's comments -> https://github.com/Azure/AKS/issues/768
I typically recommend just setting up a Ingress Controller on the cluster and not enabling "http-application-routing", as there's a lot of limitations to it. The goal with HTTP Application ROuting was for users to get setup quickly with Ingress, but not really for production deployments due to the limitations of the configuration.
DenisBiondic commented on Oct 2, 2018 -> https://github.com/Azure/AKS/issues/672
I am not 100% certain, since I don't use application routing feature, but >I think it does not use the https://github.com/nginxinc/kubernetes-ingress/tree/master/examples/websocket controller but rather the https://github.com/kubernetes/ingress-nginx. In case of the latter, I think enabling session affinity with cookies might be enough: https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/annotations.md#session-affinity
In your case you are using the wrong annotation which does not work with application routing ingress controller under the hood.
I welcome suggestions and best practices.

Kubernetes ingress-nginx sticky session isn't working with spring security

I have a stateful spring application and I want to deploy it to kubernetes cluster. There will be more than one instance of the application so i need to enable sticy session using ingress-nginx controller. I made the following configuration:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-nginx
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/affinity: "cookie"
nginx.ingress.kubernetes.io/session-cookie-name: "JSESSIONID"
nginx.ingress.kubernetes.io/session-cookie-expires: "172800"
nginx.ingress.kubernetes.io/session-cookie-max-age: "172800"
nginx.ingress.kubernetes.io/session-cookie-path: /ingress-test
# UPDATE THIS LINE ABOVE
spec:
rules:
- http:
paths:
- path: /ingress-test
backend:
serviceName: ingress-test
servicePort: 31080
ingress-nginx redirect subsequent request to correct pod if login is successful. However, it sometimes switches to other pod just after JSESSIONID is changed (JSESSIONID cookie is changed by spring-security afer successful login) and frontend redirects back to login page even user credentials are correct. Is there anyone that tried ingress-nginx with spring-security?
Best Regards
Following change fixed the problem. Without a host definition in rules, ingress-nginx doesn't set session cookie.
There is an open issue: https://github.com/kubernetes/ingress-nginx/issues/3989
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-nginx
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/affinity: "cookie"
nginx.ingress.kubernetes.io/session-cookie-name: "route"
nginx.ingress.kubernetes.io/session-cookie-expires: "172800"
nginx.ingress.kubernetes.io/session-cookie-max-age: "172800"
nginx.ingress.kubernetes.io/session-cookie-path: /ingress-test
# UPDATE THIS LINE ABOVE
spec:
rules:
- host: www.domainname.com
http:
paths:
- path: /ingress-test
backend:
serviceName: ingress-test
servicePort: 31080
The reason spring changes the cookie is to prevent session fixation (more information can be found here: https://www.owasp.org/index.php/Session_fixation).
In your case you are using the same cookie for the sticky routing policy that is used by spring for session handling.
I suggest to use a different cookie name - it will be created by nginx and there is no need to use a cookie that is used by the application.

Resources