Error on Websocket Implementation on Azure Kubernetes Cluster Not working - websocket

Question

I am trying to make the websocket service work on Azure Kubernetes Cluster on our organization environment.
My existing environment also have REST api and Angular application working on ingress with ssl.
But when I added the websocket service on the ingress it is not working.
So, I tried to use Azure Free Subscription to first implement the same WITHOUT SSL. For my applications I enabled Http Routing and using the annotation addon-http-application-routing.
I am getting below error.
'ws://40.119.7.246/ws' failed: Error during WebSocket handshake: Unexpected response code: 404
Please help in validating where I am doing wrong?
Below are the details of the configuration.
Dockerfile
FROM node:alpine
WORKDIR /app
COPY package*.json /app/
RUN npm install
COPY ./ /app/
RUN npm run build
CMD ["node","./dist/server.js"]
EXPOSE 8010
socketserver.yaml - Contains Demployment & Service.
apiVersion: apps/v1
kind: Deployment
metadata:
name: socketserver
spec:
replicas: 1
selector:
matchLabels:
app: socketserver
template:
metadata:
labels:
app: socketserver
spec:
containers:
- name: socketserver
image: regkompella.azurecr.io/socketserver:1.0.0
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
memory: "128Mi"
cpu: "500m"
ports:
- containerPort: 8010
imagePullSecrets:
- name: regkompella-azurecr-io
---
apiVersion: v1
kind: Service
metadata:
name: socketserver-svc
spec:
ports:
- port: 80
protocol: TCP
targetPort: 8010
selector:
app: socketserver
type: ClusterIP
---
ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: demo-ingress
annotations:
kubernetes.io/ingress.class: addon-http-application-routing
nginx.ingress.kubernetes.io/cors-allow-methods: "GET, PUT, POST, DELETE, OPTIONS"
nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/proxy-body-size: 10m
nginx.ingress.kubernetes.io/websocket-services: socketserver-svc
nginx.org/websocket-services: socketserver-svc
spec:
rules:
- host: demosocket.com
- http:
paths:
- path: /
backend:
serviceName: angular-application-svc
servicePort: 80
- path: /ws
backend:
serviceName: socketserver-svc
servicePort: 80

Answer 1

After reading through a lot of articles and referring some of the github forums (Added referenced articles below). I come to a point where my websocket implementation started working after doing the two things. I am not sure yet if, this is the right way to do it or not. I achieved to this solution purely on trail and error method. Hence, I request everyone who have good grasp, kindly suggest if there is a better way to solve my problem. Always take my steps with a pinch of salt.
Installed the NGINX Ingress controller from the link.
As I am using Azure Kubernetes Services, I applied the below yaml from the document.
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud/deploy.yaml
Next I made necessary changes to my demo-ingress configuration for my services.
I come to know that kubernetes.io/ingress.class: addon-http-application-routing annotation doesn't support websocketing. So, had to disable it.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: demo-ingress
annotations:
# this one annotation is making the websocket work.
nginx.ingress.kubernetes.io/websocket-services: socketserver-svc
# this one I left as-is. And not playing any role for this websocket
# implementation to work
nginx.ingress.kubernetes.io/cors-allow-methods: "GET, PUT, POST, DELETE, OPTIONS"
nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/proxy-body-size: 10m
# I thought sticky session is also required for websocket to work. But seems
# this has no effect after I installed nginx ingress controller.
# so disabled all the below annotations also.
#nginx.org/websocket-services: socketserver-svc
#nginx.ingress.kubernetes.io/affinity: cookie
#nginx.ingress.kubernetes.io/affinity-mode: balanced
#nginx.ingress.kubernetes.io/session-cookie-samesite: Strict
#kubernetes.io/ingress.class: nginx
#kubernetes.io/ingress.class: addon-http-application-routing
spec:
rules:
- host: demosocket.com
- http:
paths:
- path: /ws
backend:
serviceName: socketserver-svc
servicePort: 80
I tried to access through the public IP address. And I can be able to successfully send and receive messages.
ws://52.188.38.118/ws
Now, what if I want to make the websocket implementation work without installing NGINX Ingress Controller ( indicated on step 1) and want to try to use default ingress controller coming with AKS/minikube. The answer is below.
From the steps above,
a) Avoid Step 1: Installing NGINX Ingress Controller.
b) Only change that need to be made on ingress is below. Use the below annotations instead of the annotations indicated on Step 2 on the ingress yaml file. Things will start working.
# this annotation is making my web application also work if I plan to configure something in future.
nginx.ingress.kubernetes.io/ingress.class: nginx
# this one annotation is making the websocket work.
nginx.ingress.kubernetes.io/websocket-services: socketserver-svc
# by default ssl is true - as I am trying locally and want to disable ssl-# redirect. So set this to false.
nginx.ingress.kubernetes.io/ssl-redirect: "false"
# Below are just additional annotation to allow CORS etc.
nginx.ingress.kubernetes.io/cors-allow-methods: "GET, PUT, POST, DELETE, OPTIONS"
nginx.ingress.kubernetes.io/proxy-body-size: 10m
Referenced Articles:
https://medium.com/flant-com/comparing-ingress-controllers-for-kubernetes-9b397483b46b
https://kubernetes.github.io/ingress-nginx/deploy/#azure
Mr. dstrebel's comments -> https://github.com/Azure/AKS/issues/768
I typically recommend just setting up a Ingress Controller on the cluster and not enabling "http-application-routing", as there's a lot of limitations to it. The goal with HTTP Application ROuting was for users to get setup quickly with Ingress, but not really for production deployments due to the limitations of the configuration.
DenisBiondic commented on Oct 2, 2018 -> https://github.com/Azure/AKS/issues/672
I am not 100% certain, since I don't use application routing feature, but >I think it does not use the https://github.com/nginxinc/kubernetes-ingress/tree/master/examples/websocket controller but rather the https://github.com/kubernetes/ingress-nginx. In case of the latter, I think enabling session affinity with cookies might be enough: https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/annotations.md#session-affinity
In your case you are using the wrong annotation which does not work with application routing ingress controller under the hood.
I welcome suggestions and best practices.