I am working with a 3D simulator web site and it has options to import model as zip files. I was able to import upto 70MB of zip files into my development and production environments. But recently it was migrated to kubernetes using nginx reverse proxy. Afterwards I am unable to load zip files larger than 1MB .I tried to configure nginx config file, setting up limit for multer but doesn't make any difference. I am using ajax call but the error case with error "Request entity too large" is getting executed.
Thanks in advance
Nginx supports changing the max body size. If you're using a Kubernetes Ingress object you can do so in the following way:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: myapp
namespace: default
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer: "letsencrypt-prod"
nginx.ingress.kubernetes.io/proxy-body-size: 512m # Max body size for ingress.
spec:
tls:
- hosts:
- "somehost.com"
secretName: somehost-tls
rules:
- host: somehost.com
http:
paths:
- path: /
backend:
serviceName: myapp
servicePort: 80
I don't know how you're running Nginx. It may be that you're running it in a non-ingress way. For that you can edit the Nginx config: Default nginx client_max_body_size
Related
I have an GKE ingress with both Http and Https. I want to redirect the traffic from port 80 to port 443.
I found this:
https://github.com/kubernetes/ingress-gce/issues/1075
which let to this:
https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#https_redirect.
The proposed solution adds a FrontendConfig with a RedirectToHttps flag which uses some LoadBalancer functionality. Yet when I try to add the FrontendEndConfig, I get the following error:
error: unable to recognize "ssl.yaml": no matches for kind "FrontendConfig" in version "networking.gke.io/v1beta1"
I have also tried 'networking.gke.io/v1' and 'v1beta2'.
The latest GKE version available in my zone is 1.17.13-gke.2001. I have recently launched the cluster so although I don't know how to check the GKE version, I reckon it's running on the latest version.
Anyone has a clue why my kubectl doesn't recognize this kind?
Ingress yaml:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: basic-ingress
annotations:
FrontendConfig: my-frontend-config
kubernetes.io/ingress.global-static-ip-name: 'web-static-ip'
networking.gke.io/managed-certificates: mycertificate
# kubernetes.io/ingress.allow-http: "false"
spec:
rules:
- http:
paths:
- path: /*
backend:
serviceName: frontend
servicePort: 80
- path: /api/*
backend:
serviceName: backend
servicePort: 80
Redirect yaml:
apiVersion: networking.gke.io/v1beta1
kind: FrontendConfig
metadata:
name: my-frontend-config
spec:
redirectToHttps:
enabled: true
Thank you for pointing me in the right direction!
I had to upgrade the cluster as MrKoopaKiller indicated and also changed the annotation:
FrontendConfig: my-frontend-config
to:
networking.gke.io/v1beta1.FrontendConfig: "my-frontend-config"
and it worked!
also: make sure you have:
kubernetes.io/ingress.allow-http: "true"
I'm having a problem migrating my pure Kubernetes app to an Istio managed. I'm using Google Cloud Platform (GCP), Istio 1.4, Google Kubernetes Engine (GKE), Spring Boot and JAVA 11.
I had the containers running in a pure GKE environment without a problem. Now I started the migration of my Kubernetes cluster to use Istio. Since then I'm getting the following message when I try to access the exposed service.
upstream connect error or disconnect/reset before headers. reset reason: connection failure
This error message looks like a really generic. I found a lot of different problems, with the same error message, but no one was related to my problem.
Bellow the version of the Istio:
client version: 1.4.10
control plane version: 1.4.10-gke.5
data plane version: 1.4.10-gke.5 (2 proxies)
Bellow my yaml files:
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
account: tree-guest
name: tree-guest-service-account
---
apiVersion: v1
kind: Service
metadata:
labels:
app: tree-guest
service: tree-guest
name: tree-guest
spec:
ports:
- name: http
port: 8080
targetPort: 8080
selector:
app: tree-guest
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: tree-guest
version: v1
name: tree-guest-v1
spec:
replicas: 1
selector:
matchLabels:
app: tree-guest
version: v1
template:
metadata:
labels:
app: tree-guestaz
version: v1
spec:
containers:
- image: registry.hub.docker.com/victorsens/tree-quest:circle_ci_build_00923285-3c44-4955-8de1-ed578e23c5cf
imagePullPolicy: IfNotPresent
name: tree-guest
ports:
- containerPort: 8080
serviceAccount: tree-guest-service-account
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: tree-guest-gateway
spec:
selector:
istio: ingressgateway # use istio default controller
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: tree-guest-virtual-service
spec:
hosts:
- "*"
gateways:
- tree-guest-gateway
http:
- match:
- uri:
prefix: /v1
route:
- destination:
host: tree-guest
port:
number: 8080
To apply the yaml file I used the following argument:
kubectl apply -f <(istioctl kube-inject -f ./tree-guest.yaml)
Below the result of the Istio proxy argument, after deploying the application:
istio-ingressgateway-6674cc989b-vwzqg.istio-system SYNCED SYNCED SYNCED SYNCED
istio-pilot-ff4489db8-2hx5f 1.4.10-gke.5 tree-guest-v1-774bf84ddd-jkhsh.default SYNCED SYNCED SYNCED SYNCED istio-pilot-ff4489db8-2hx5f 1.4.10-gke.5
If someone have a tip about what is going wrong, please let me know. I'm stuck in this problem for a couple of days.
Thanks.
As #Victor mentioned the problem here was the wrong yaml file.
I solve it. In my case the yaml file was wrong. I reviewed it and the problem now is solved. Thank you guys., – Victor
If you're looking for yaml samples I would suggest to take a look at istio github samples.
As 503 upstream connect error or disconnect/reset before headers. reset reason: connection failure occurs very often I set up little troubleshooting answer, there are another questions with 503 error which I encountered for several months with answers, useful informations from istio documentation and things I would check.
Examples with 503 error:
Istio 503:s between (Public) Gateway and Service
IstIO egress gateway gives HTTP 503 error
Istio Ingress Gateway with TLS termination returning 503 service unavailable
how to terminate ssl at ingress-gateway in istio?
Accessing service using istio ingress gives 503 error when mTLS is enabled
Common cause of 503 errors from istio documentation:
https://istio.io/docs/ops/best-practices/traffic-management/#avoid-503-errors-while-reconfiguring-service-routes
https://istio.io/docs/ops/common-problems/network-issues/#503-errors-after-setting-destination-rule
https://istio.io/latest/docs/concepts/traffic-management/#working-with-your-applications
Few things I would check first:
Check services ports name, Istio can route correctly the traffic if it knows the protocol. It should be <protocol>[-<suffix>] as mentioned in istio
documentation.
Check mTLS, if there are any problems caused by mTLS, usually those problems would result in error 503.
Check if istio works, I would recommend to apply bookinfo application example and check if it works as expected.
Check if your namespace is injected with kubectl get namespace -L istio-injection
If the VirtualService using the subsets arrives before the DestinationRule where the subsets are defined, the Envoy configuration generated by Pilot would refer to non-existent upstream pools. This results in HTTP 503 errors until all configuration objects are available to Pilot.
I landed exactly here with exactly similar symptoms.
But in my case I had to
switch pod listen address from 172.0.0.1 to 0.0.0.0
which solved my issue
I am trying to make the websocket service work on Azure Kubernetes Cluster on our organization environment.
My existing environment also have REST api and Angular application working on ingress with ssl.
But when I added the websocket service on the ingress it is not working.
So, I tried to use Azure Free Subscription to first implement the same WITHOUT SSL. For my applications I enabled Http Routing and using the annotation addon-http-application-routing.
I am getting below error.
'ws://40.119.7.246/ws' failed: Error during WebSocket handshake: Unexpected response code: 404
Please help in validating where I am doing wrong?
Below are the details of the configuration.
Dockerfile
FROM node:alpine
WORKDIR /app
COPY package*.json /app/
RUN npm install
COPY ./ /app/
RUN npm run build
CMD ["node","./dist/server.js"]
EXPOSE 8010
socketserver.yaml - Contains Demployment & Service.
apiVersion: apps/v1
kind: Deployment
metadata:
name: socketserver
spec:
replicas: 1
selector:
matchLabels:
app: socketserver
template:
metadata:
labels:
app: socketserver
spec:
containers:
- name: socketserver
image: regkompella.azurecr.io/socketserver:1.0.0
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
memory: "128Mi"
cpu: "500m"
ports:
- containerPort: 8010
imagePullSecrets:
- name: regkompella-azurecr-io
---
apiVersion: v1
kind: Service
metadata:
name: socketserver-svc
spec:
ports:
- port: 80
protocol: TCP
targetPort: 8010
selector:
app: socketserver
type: ClusterIP
---
ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: demo-ingress
annotations:
kubernetes.io/ingress.class: addon-http-application-routing
nginx.ingress.kubernetes.io/cors-allow-methods: "GET, PUT, POST, DELETE, OPTIONS"
nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/proxy-body-size: 10m
nginx.ingress.kubernetes.io/websocket-services: socketserver-svc
nginx.org/websocket-services: socketserver-svc
spec:
rules:
- host: demosocket.com
- http:
paths:
- path: /
backend:
serviceName: angular-application-svc
servicePort: 80
- path: /ws
backend:
serviceName: socketserver-svc
servicePort: 80
After reading through a lot of articles and referring some of the github forums (Added referenced articles below). I come to a point where my websocket implementation started working after doing the two things. I am not sure yet if, this is the right way to do it or not. I achieved to this solution purely on trail and error method. Hence, I request everyone who have good grasp, kindly suggest if there is a better way to solve my problem. Always take my steps with a pinch of salt.
Installed the NGINX Ingress controller from the link.
As I am using Azure Kubernetes Services, I applied the below yaml from the document.
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud/deploy.yaml
Next I made necessary changes to my demo-ingress configuration for my services.
I come to know that kubernetes.io/ingress.class: addon-http-application-routing annotation doesn't support websocketing. So, had to disable it.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: demo-ingress
annotations:
# this one annotation is making the websocket work.
nginx.ingress.kubernetes.io/websocket-services: socketserver-svc
# this one I left as-is. And not playing any role for this websocket
# implementation to work
nginx.ingress.kubernetes.io/cors-allow-methods: "GET, PUT, POST, DELETE, OPTIONS"
nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/proxy-body-size: 10m
# I thought sticky session is also required for websocket to work. But seems
# this has no effect after I installed nginx ingress controller.
# so disabled all the below annotations also.
#nginx.org/websocket-services: socketserver-svc
#nginx.ingress.kubernetes.io/affinity: cookie
#nginx.ingress.kubernetes.io/affinity-mode: balanced
#nginx.ingress.kubernetes.io/session-cookie-samesite: Strict
#kubernetes.io/ingress.class: nginx
#kubernetes.io/ingress.class: addon-http-application-routing
spec:
rules:
- host: demosocket.com
- http:
paths:
- path: /ws
backend:
serviceName: socketserver-svc
servicePort: 80
I tried to access through the public IP address. And I can be able to successfully send and receive messages.
ws://52.188.38.118/ws
Now, what if I want to make the websocket implementation work without installing NGINX Ingress Controller ( indicated on step 1) and want to try to use default ingress controller coming with AKS/minikube. The answer is below.
From the steps above,
a) Avoid Step 1: Installing NGINX Ingress Controller.
b) Only change that need to be made on ingress is below. Use the below annotations instead of the annotations indicated on Step 2 on the ingress yaml file. Things will start working.
# this annotation is making my web application also work if I plan to configure something in future.
nginx.ingress.kubernetes.io/ingress.class: nginx
# this one annotation is making the websocket work.
nginx.ingress.kubernetes.io/websocket-services: socketserver-svc
# by default ssl is true - as I am trying locally and want to disable ssl-# redirect. So set this to false.
nginx.ingress.kubernetes.io/ssl-redirect: "false"
# Below are just additional annotation to allow CORS etc.
nginx.ingress.kubernetes.io/cors-allow-methods: "GET, PUT, POST, DELETE, OPTIONS"
nginx.ingress.kubernetes.io/proxy-body-size: 10m
Referenced Articles:
https://medium.com/flant-com/comparing-ingress-controllers-for-kubernetes-9b397483b46b
https://kubernetes.github.io/ingress-nginx/deploy/#azure
Mr. dstrebel's comments -> https://github.com/Azure/AKS/issues/768
I typically recommend just setting up a Ingress Controller on the cluster and not enabling "http-application-routing", as there's a lot of limitations to it. The goal with HTTP Application ROuting was for users to get setup quickly with Ingress, but not really for production deployments due to the limitations of the configuration.
DenisBiondic commented on Oct 2, 2018 -> https://github.com/Azure/AKS/issues/672
I am not 100% certain, since I don't use application routing feature, but >I think it does not use the https://github.com/nginxinc/kubernetes-ingress/tree/master/examples/websocket controller but rather the https://github.com/kubernetes/ingress-nginx. In case of the latter, I think enabling session affinity with cookies might be enough: https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/annotations.md#session-affinity
In your case you are using the wrong annotation which does not work with application routing ingress controller under the hood.
I welcome suggestions and best practices.
I have a bare-metal kubernetes cluster (1.13) and am running nginx ingress controller (deployed via helm into the default namespace, v0.22.0).
I have an ingress in a different namespace that attempts to use the nginx controller.
#ingress.yaml
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: myapp
annotations:
kubernetes.io/backend-protocol: https
nginx.ingress.kubernetes.io/enable-rewrite-log: "true"
nginx.ingress.kubernetes.io/rewrite-target: "/$1"
spec:
tls:
- hosts:
- my-host
secretName: tls-cert
rules:
- host: my-host
paths:
- backend:
servicename: my-service
servicePort: https
path: "/api/(.*)"
The nginx controller successfully finds the ingress, and says that there are endpoints. If I hit the endpoint, I get a 400, with no content. If I turn on custom-http-headers then I get a 404 from nginx; my service is not being hit. According to re-write logging, the url is being re-written correctly.
I have also hit the service directly from inside the pod, and that works as well.
#service.yaml
kind: Service
apiVersion: v1
metadata:
name: my-service
spec:
ports:
- name: https
protocol: TCP
port: 5000
targetPort: https
selector:
app: my-app
clusterIP: <redacted>
type: ClusterIP
sessionAffinity: None
What could be going wrong?
EDIT: Disabling https all over still gives the same 400 error. However, if my app is expecting HTTPS requests, and nginx is sending HTTP requests, then the requests get to the app (but it can't processes them)
Nginx will silently fail with 400 if request headers are invalid (like special characters in it). You can debug that using tcpdump.
I'm trying to configure an HAProxy ingress controller to load-balance properly connections to websocket. I tried to raise the value timeout-client, timeout-server and timeout-connect but without success.
ingress.yaml
kind: Ingress
metadata:
namespace: test-deploy
name: app-test
labels:
app: app-test
annotations:
ingress.kubernetes.io/rewrite-target: /
ingress.kubernetes.io/timeout-connect: "5000"
ingress.kubernetes.io/timeout-client: "5000"
ingress.kubernetes.io/timeout-server: "5000"
ingress.kubernetes.io/timeout-tunnel: "3600"
spec:
rules:
- host: k8s-test.local.lan
http:
paths:
- path: /app-test
backend:
serviceName: app-test
servicePort: 9000
I haven't found confirmation about websockets support in the HAProxy documentation, but this post on Quora stated that it works great. You may need to adjust client/server/tulnnel timeouts and sometimes match and route websockets traffic to the correct backend destination.
You can check actual haproxy-ingress confgiration using the following command:
kubectl exec -ti haproxy-ingress-pod-name -n ingress-controller -- cat /etc/haproxy/haproxy.cfg
If you have more than one ingress in the cluster you may need to specify a proxy class in the annotation for every Ingress object that should be used by HAProxy ingress:
kubernetes.io/ingress.class: "haproxy"
HAProxy ingress is pretty much the same HAProxy with the capability to use Kubernetes Ingress objects to update it's configuration.
You can find more information about configuring HAProxy and HAProxy Ingress in the articles:
Websockets Load Balancing with HAProxy
Using HAProxy as an API Gateway, Part 1 [Introduction]
Haproxy Ingres
Voyager
Hope it would be helpful to you.