Websocket Error: Invalid HTTP upgrade", code: 404 - spring-boot

We have deployed an ALB for our spingboot application which consists of both Rest and Websocket services hosted in EKS cluster. We have added health check, ssl redirect etc. annotations in the ingress.yaml file. The listener is HTTPS: 443 and the SSL certificate is from ACM.
When we hit the ALB endpoint, rest service is working fine but receiving the following error with the websocket service.
Websocket Optional(wss://xxx.xx.xx.com:443/xx/xx/xx/xx) disconnected with error Optional(Starscream.WSError(type: Starscream.ErrorType.upgradeError, message: "Invalid HTTP upgrade", code: 404))
Versions:
aws-load-balancer-controller: v2.3.0,
Kubernetes version: 1.19,
Platform version: eks.7.
Appreciate your inputs in resolving this issue.

Related

Spring Authorization Server Redirects to HTTP when behind proxy

I have a standalone spring authorization server at https://auth.foo.com with certificate configured in the spring boot application directly (no Proxy).
I have a client spring boot app using http 8080 behind a K8s Nginx Proxy. The certificate terminates on the proxy and then delivers http upstream to the spring boot app. https://client.foo.com
I have enabled the following in the client application.yaml
server:
port: 8080
forward-headers-strategy: native
tomcat:
use-relative-redirects: true
remoteip:
remote-ip-header: x-forwarded-for
protocol-header: x-forwarded-proto
When navigating to the https://client.foo.com app, the relative redirects work fine, all links redirect to https.
However on successful login via https://auth.foo.com, the auth server redirects to https://client.foo.com/login/oauth2/code/foo-client-oidc?code=XXX then the client responds with a 302 but the Location header is insecure http://client.foo.com
Is there a way to force the redirect to use https in Spring Authorisation Server
Thanks in advance

Spring cloud gateway does not forward HTTPS correctly

I've recently migrated a microservices backend to Spring boot v2.6.1 + spring cloud v2021.0.0 (old version was v2.2.1.RELEASE).
Previously, the setup included a discovery server (Eureka), a gateway (Zuul) and various microservices accessible externally via the gateway. All of these were TLS/secure port enabled so all requests (discovery, registration, gateway forwarding etc) required SSL trust store configuration.
Since Zuul is no longer supported in this version, I am switching to Spring cloud gateway, with the following application.properties setup:
server.port = 8080
eureka.instance.securePortEnabled=false
eureka.client.serviceUrl.defaultZone = http://localhost:8761/eureka/
spring.application.name= gateway-service
spring.cloud.gateway.routes[0].id=first-service
spring.cloud.gateway.routes[0].uri=lb://FIRST-SERVICE
spring.cloud.gateway.routes[0].predicates[0].name=Path
spring.cloud.gateway.routes[0].predicates[0].args.pattern=/firstService/**
spring.cloud.gateway.routes[0].filters[0]=StripPrefix=1
security.require-ssl=true
server.ssl.key-store=keystore.p12
server.ssl.key-store-password=password
server.ssl.keyStoreType=PKCS12
server.ssl.keyAlias=localhost
spring.cloud.gateway.httpclient.ssl.trustedX509Certificates[0]=cert.pem
Without ssl universally disabled, I can access a first service URL via that gateway without issue. With SSL enabled, I can access the first service directly and again no issue:
https://localhost:8100/test
{ "message": "Hello World!" }
However, the gateway is not correctly forwarding the requests either via the correct protocol or with the correct client certificate. I am getting the below error response instead of "Hello World" message:
https://localhost:8080/firstService/test
Bad Request This combination of host and port requires TLS.
(It's worth noting that the Eureka secure port is disabled, and registration/registry fetch is done via HTTP due to issues logged at Error setting custom trust store for Eureka Discovery client by overriding DiscoveryClient.DiscoveryClientOptionalArgs
Ok, this comes down to how the microservices register with Eureka.
Although FIRST-SERVICE was also setup with security.require-ssl=true for direct access it was also necessary to register with Eureka with SSL enabled.
Adding the settings below to application.properties fixed the issue without any changes needed for the gateway configuration:
eureka.client.tls.enabled=true
eureka.client.tls.key-store=file:<path-to-key-store>
eureka.client.tls.key-store-password=<password>
eureka.client.tls.keyStoreType=PKCS12
eureka.client.tls.keyPassword=<password>
eureka.client.tls.trust-store=file:<path-to-trust-store>
eureka.client.tls.trust-store-password=<password>

Spring Boot Admin unable to access pods

I'm trying to run a Spring Boot Admin application on Kubernetes using Spring Cloud Discovery (without Eureka or Zuul. It directly scrapes from Kubernetes API)
I've setup necessary RBAC and Ingress/Egress for the application to access Kubernetes API and the relevant services in the cluster.
The application is initially fetching all the information regarding the services but it's failing with below error when trying to communicate with individual pods in the namespace
reactor.netty.http.client.PrematureCloseException: Connection prematurely closed
When I try to curl the particular pods from the pod with Spring Boot Admin app:
When I try with FQDN of the service it returns fine with the response
curl {service}.{namespace}.svc.cluster.local/actuator/info
When I do the same with the PodIp ( the one which Spring Boot Admin is struggling to connect to ):
curl 10.x.x.x:8080/actuator/info
I get this error
curl: (56) Recv failure: Connection reset by peer
Is there any particular netpol for pods to be accessed directly rather through the service's cluster IP? Because Spring Boot Admin tries to monitor all the individual pods in the services.
Or is there a workaround/approach where it isn't required by the Spring Boot Admin app to send requests to all the individual pods?

Keycloak Kubernetes 401 Unauthorized

I have a keycloak server and a spring boot app running on Kubernetes cluster and I'm facing 2 scenarios the first one is working and the other one I'm getting 401 Unauthorized.
First scenario:
Enter a test container with kubectl exec command inside my k8s cluster and ask for a token with a curl POST request :
http://keycloak-service-name:8080/auth/realms/lisa/protocol/openid-connect/token
I get a token
I send a request from postman using that token to the secured endpoint of my app and
It's working, access is granted
Second scenario:
Ask for a token by sending a POST request to: http://keycloak-external-ip:port/auth/realms/lisa/protocol/openid-connect/token with the necessary parameters from postman
I receive an access token
I send a request from postman to the secured endpoint using that token of my app I get 401 unauthorized
The Keycloak is exposed using NodePort service.
The app is exposed using istio ingress gateway.
Keycloak server version:9.0.0.
keycloak version in spring boot app: 10.0.2
Fixed this by changing the keycloak.auth-server-url value in the application.properties file.
The keycloak deployment has 2 services, one of type clusterIP, the second is of type nodePort
before
keycloak.auth-server-url=http://keycloak-clusterIP-service-name:8080/auth
after
keycloak.auth-server-url=http://node-ip-adress:nodePort/auth

Spring Boot Microservice com.netflix.zuul.exception.ZuulException: Forwarding error

I have created micro service using spring boot and implment zuul proxy so when I call url via zuul proxy and somehow my service is down I get com.netflix.zuul.exception.ZuulException: Forwarding error I have also implement errorFilter and send custom message from errorFilter but it's not working for me. I get below response:
{
"timestamp": 1537261710692,
"status": 500,
"error": "Internal Server Error",
"exception": "com.netflix.zuul.exception.ZuulException",
"message": "GENERAL"}
This problem occurs when your back-end service is not available but your gateway make proxy calls to unavaiable back-end service.
If you read a log file, you can check it.
DynamicServerListLoadBalancer for client BACKEND-YOUR-SERVICE initialized:
DynamicServerListLoadBalancer:{NFLoadBalancer:name=BACKEND-YOUR-SERVICE,
current list of Servers=[]
It means that there are not available servers of your back-end service.
Please check in the following order.
Check whether your back-end service is running
(If you use Eureka Server) Check whether your back-end service is registered on Eureka server on Eureka webpage.
Check whether your zuul proxy is configured with correct back-end service name.

Resources