Changing Kubernetes' node-proxy tcp keepalive time - proxy

Question

How do I properly change the TCP keepalive time for node-proxy?
I am running Kubernetes in Google Container Engine and have set up an ingress backed by HTTP(S) Google Load Balancer. When I continuously make POST requests to the ingress, I get a 502 error exactly once every 80 seconds or so. backend_connection_closed_before_data_sent_to_client error in Cloud Logging, which is because GLB's tcp keepalive (600 seconds) is larger than node-proxy's keepalive (no clue what it is).
The logged error is detailed in https://cloud.google.com/compute/docs/load-balancing/http/.
Thanks!

Answer 1

You can use the custom resource BackendConfig that exist on each GKE cluster to configure timeouts and other parameters like CDN here is the documentacion
An example from here shows how to configure on the ingress
That is the BackendConfig definition:
apiVersion: cloud.google.com/v1beta1
kind: BackendConfig
metadata:
name: my-bsc-backendconfig
spec:
timeoutSec: 40
connectionDraining:
drainingTimeoutSec: 60
And this is how to use on the ingress definition through annotations
apiVersion: v1
kind: Service
metadata:
name: my-bsc-service
labels:
purpose: bsc-config-demo
annotations:
beta.cloud.google.com/backend-config: '{"ports": {"80":"my-bsc-backendconfig"}}'
spec:
type: NodePort
selector:
purpose: bsc-config-demo
ports:
- port: 80
protocol: TCP
targetPort: 8080

Answer 2

just for the sake of understanding, when you use Google solution to load-balance and manage your Kubernetes Ingress, you will have GLBC pods running in kube-system namespace.
You can check it out with :
kubectl -n kube-system get po
These pods are intended to route the incoming traffic from the actual Google Load Balancer.
I think that the timeouts should be configured there, on GLBC. You should check what annotations or ConfigMap GLBC can take to be configured, if any.
You can find details there :
https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/cluster-loadbalancing/glbc
https://github.com/kubernetes/ingress/blob/master/controllers/gce/README.md
https://github.com/kubernetes/ingress/blob/master/controllers/gce/rc.yaml#L64
Personally I prefer to use the Nginx Ingress Controller for now, and it has necessary annotations and ConfigMap support.
See :
https://github.com/kubernetes/ingress/blob/master/controllers/nginx/README.md