I'm trying to run a Spring Boot Admin application on Kubernetes using Spring Cloud Discovery (without Eureka or Zuul. It directly scrapes from Kubernetes API)
I've setup necessary RBAC and Ingress/Egress for the application to access Kubernetes API and the relevant services in the cluster.
The application is initially fetching all the information regarding the services but it's failing with below error when trying to communicate with individual pods in the namespace
reactor.netty.http.client.PrematureCloseException: Connection prematurely closed
When I try to curl the particular pods from the pod with Spring Boot Admin app:
When I try with FQDN of the service it returns fine with the response
curl {service}.{namespace}.svc.cluster.local/actuator/info
When I do the same with the PodIp ( the one which Spring Boot Admin is struggling to connect to ):
curl 10.x.x.x:8080/actuator/info
I get this error
curl: (56) Recv failure: Connection reset by peer
Is there any particular netpol for pods to be accessed directly rather through the service's cluster IP? Because Spring Boot Admin tries to monitor all the individual pods in the services.
Or is there a workaround/approach where it isn't required by the Spring Boot Admin app to send requests to all the individual pods?
Related
We have a few cloud run services deployed on Cloud run with Ingress control set to "All" and Authentication set to "IAM". These are REST APIs created using Spring boot framework. We have one more Cloud Run service which is deployed as Spring boot admin server with Ingress control set to "All" and Authentication set to "Allow Unauthenticated Invocations".
Now the backend team have configured the Cloud run REST API services with the following cloud run endpoints in application.properties to communicate with Spring boot admin server:-
spring.boot.admin.client.url=${sm://cloud_run_rest_api_endpoint}
spring.boot.admin.client.instance.service-url=${sm://spring_boot_admin_server_cloud_run_endpoint}
This obviously fails on cloud run with 403 Unauthorized as the endpoints (Eg: /health, /metrics, /trace etc) required by Spring boot admin server from its clients need an Authorization Bearer token.
Is it possible to pass JWT token when accessing those endpoints ? Has anyone had success in setting up the same ? Or is there any possibility to have some of our cloud run service endpoints to be public on which we can later apply some security provided by spring ?
We are in the middle of setting up free API monitoring tool for our all APIs. Any recommendations are much appreciated.
Trying to use Spring Cloud Kubernetes Discovery server with discovery client as described in https://docs.spring.io/spring-cloud-kubernetes/docs/current/reference/html/index.html#spring-cloud-kubernetes-discoveryserver
and the client couldn't fetch service information from other namespaces. There is already an issue raised in Spring Cloud Kubernetes in GitHub - https://github.com/spring-cloud/spring-cloud-kubernetes/issues/824
Tried Fabric8 client also, encountering error as below:
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://xx.xx.xx.xx/api/v1/services. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. services is forbidden: User "system:serviceaccount:abcd01:xyzssvc" cannot list resource "services" in API group "" at the cluster scope.
Did anyone manage to integrate Spring Cloud Kubernetes Discovery Server with Discovery Client? Integrating Discovery Server with Discovery client will help to prevent assigning clusterrole permission to services.
The microservices are getting registered to Eureka with the pod name as hostname, this causing UnknownHostException, when the Zull API gateway trying to forward the request to the service.
The complete setup working fine with the docker-compose, the issues are happening when I am trying to use the Kubernetes.
For example, the order microservice running with the pod name as "oc-order-6b64576f4b-7gp85" and the order microservice getting register to to Eureka with "oc-order-6b64576f4b-7gp85" as the hostname. Which is causing
java.net.UnknownHostException: oc-order-6b64576f4b-7gp85: Name does not resolve
I am running just one instance of each micro-services as a POC and one instance of the Eureka server. How to fix, how micro-service can register them self with the right hostname.
I want to use the Eureka as discovery service as it is well integrated with spring boot and I do not want to do code changes for Kubernetes deployment.
Add the below property to your spring properties file in each service project:
eureka.instance.hostName=${spring.application.name}
or (if you want a different name than the spring application name)
eureka.instance.hostName=<custom-host-name>
Spring boot will now register with the provided hostname in Eureka
Problem
Trying to use Spring boot admin to do a deep monitoring of spring boot micro services running in Kubernetes.
Spring boot admin listing the micro services but pointing to the internal IPs.
Spring boot admin application listing page showing the internal IP
The application details page has almost zero info
Details
Kubernetes 1.15
Spring boot applications are getting discovered by Spring boot admin using Spring cloud discovery
spring-cloud-kubernetes version 1.1.0.RELEASE
The problem is that the IPs are of internal pod network and would not be accessible to the users in any real world scenario.
Any hints on how to approach this scenario ? Any alternatives ?
Also I was wondering how spring boot admin would behave in case of pods with more than one replica. I think it is close to impossible to point to a unique pod replica through ingress or node port.
Hack I am working on
If I can start another pod which exposes the Linux desktop to the end user. From a browser of this desktop, user may be able to access the pod network ips. It is just a wild thought as a hack.
Spring Boot Admin register each application/client based on its name by below property.
spring.boot.admin.client.instance.name=${spring.application.name}
If all your pods have same name it can register based on individual ips by enabling perfer-ip property (which is false by default):
spring.boot.admin.client.instance.prefer-ip=true
In your case, you want to SBA to register based on the Kubernetes load balanced url, then service-base-url property should be set the corresponding application's url.
spring.boot.admin.client.instance.service-base-url=http://myapp.com
I am trying to setup a Spring Boot Admin server on a Cloud foundry. I am using the client Spring Cloud Discovery with SimpleDiscoveryClient configuration. We are not having any Thrid Party service discovery client like eureka. I can see the service getting registered to the spring boot admin server. But when i scale up any service, i see only one instance of that service and the actual number of instances are not reflected. I would like to know if that is possible without Eureka or any other service discovery, if yes how to achieve that without them.
Thanks