There are two microservices (microservice-A & microservice-B), written in Spring boot, are talking to each other through the Feign Client and the services are registered in K8S Native service registry
Whenever a fresh deployment of a microservice-A is happening, the microservice-B that is already running in kubertnetes fails to make HTTP call to the freshly deployed service-A and below is the exception
feign.RetryableException: No route to host (Host unreachable) executing GET http://microserice-a/api/v1/myresources
This issue is getting resolved immediately after restarting the microservice-B.
When we googled for solutions, we got to see this link https://github.com/spring-cloud/spring-cloud-netflix/issues/769 and an user had given the below comment there
I suspect the root cause is that FeignClient keeps an old list of service providers and the Ribbon cannot move correctly to the next node if one node has been destroyed
Not sure if that is correct root cause. Please comment If anyone has faced similar issue and solved it?
Related
I have created a spring boot integration app and deployed it to Pivotal Cloud Foundry (PCF) environment. It works for couple of days and then it starts to crash randomly afterwards. I checked the PCF logs and found this information about the crash.
OUTApp instance exited with guid 3c348d47-48c4-403f-950a-29af1efa551d
payload: {"instance"=>"e2122543-214f-4806-62c7-00e1", "index"=>2,
"reason"=>"CRASHED", "exit_description"=>"Instance became unhealthy: Failed
to make HTTP request to '/health' on port 8080: timed out after 1.00
seconds", "crash_count"=>1, "crash_timestamp"=>1511959503256098495,
"version"=>"10cea919-d490-460d-83d6-5132c96ef781"}
My CPU utilization is not much. My memory is also not leaking.
Information about the application deployed in PCF:
Spring boot integration app connects to IBM MQ queues and polls for messages and then calls couple of web services.
There is also another application Service Bus, which makes the health check call on PCF application to check if the PCF app is available or not. If Service Bus finds that PCF app is available then the requests are routed to PCF else they are processed at Service Bus end itself.
Please let me know, how to find the root cause of the CRASH and fix it.
Thanks in advance. Please let me know, if you need further details.
I have changed the health check type to port type from http in manifest.yml file.
configuration change in manifest file is as follows:
health-check-type: port
Now the app is not crashing. It is working fine. Hope this helps.
I am trying to register a Wildfly Swarm REST service to a running Consule agent, but it's not working correctly.
I am able to register a service (I can see it in the Consul ui), but somehow the health checks are not working.
The Swarm Server keeps frequently telling me, that "sending the check" failed due to "HTTP 405 Method not allowed". I can see simular logs in the Consule console, that GET method is not allowed.
I am at a dead end: My application is not working, nor does the Wildfly Swarm example (same exception). I also configured a CORS filter on both sides just to be sure, but thats not working either.
I am using Wildfly Swarm 2017.10.1 and Consul 1.0.0.
I hope you guys can help.
Best regards
I figured it out myself. Obviously, it wasn't that hard ^^
I checked the version of the Consul Client API which is used for my Wildfly Swarm version: It's 0.9.16. I've downloaded all Consul versions and checked which one are compatible. I can verify that all versions up to 0.9.3 are working.
Consul 1.0.0 has some very critical breaking changes and I really don't understand why they were not implemented in a HTTP API v2, but thats not the point here.
I highly recommend to upgrade the Consul Client API used by the topology-consul fraction to a newer version like 0.16.5 or 0.17.0.
At least, please add a note in the README for the ribbon-consul example what Consul versions can be used.
We are in the process of standing up a new microservices architecture with Zuul at the front-end and a bunch of tomcat enabled microservices at the backend. Each service as it starts up, will register itself with Eureka and any client that wants to call those service will do so through Zuul. We've got this all wired in and everything is working fine.
However, I have a couple questions as to how we can make this architecture much more dynamic.
One thing that we assumed was there out of the box with Ribbon/Eureka, but have yet to find a solution for is that as we add more services to the backend, that somehow (via Archiaus and update to Zuul's eureka-client.properties file) Zuul's Ribbon client would update itself with the new service details (e.g. vipaddress, load balancing algorithm, etc). So far, the only thing that works is to update the properties file and restart Zuul (ughhh).
For example, let's say today we have 2 microservices at the backend, therefore, Zuul's eureka/ribbon client configuration would include the below:
ribbon.client.niws.clientlist=service1|service2
zuul.ribbon.namespace=zuul.client
service1.zuul.client.DeploymentContextBasedVipAddresses=myService1
service1.zuul.client.NIWSServerListClassName=com.netflix.niws.loadbalancer.DiscoveryEnabledNIWSServerList
service2.zuul.client.DeploymentContextBasedVipAddresses=myService2
service2.zuul.client.NIWSServerListClassName=com.netflix.niws.loadbalancer.DiscoveryEnabledNIWSServerList
Now tomorrow, let's assume we need to add service3. What we have observed is that if we add those details to the same configuration (see below), they only become available to Zuul after a restart. Is there some other configuration parameter we are missing that would allow us to dynamically introduce the new service details or do we have to roll our own Eureka/Ribbon client to do this?
ribbon.client.niws.clientlist=service1|service2|service3
zuul.ribbon.namespace=zuul.client
service1.zuul.client.DeploymentContextBasedVipAddresses=myService1
service1.zuul.client.NIWSServerListClassName=com.netflix.niws.loadbalancer.DiscoveryEnabledNIWSServerList
service2.zuul.client.DeploymentContextBasedVipAddresses=myService2
service2.zuul.client.NIWSServerListClassName=com.netflix.niws.loadbalancer.DiscoveryEnabledNIWSServerList
service3.zuul.client.DeploymentContextBasedVipAddresses=myService3
service3.zuul.client.NIWSServerListClassName=com.netflix.niws.loadbalancer.DiscoveryEnabledNIWSServerList
My other question is related and that is do we really need to add a client configuration (in eureka-client.properties) for every service that Zuul could possibly route to? At some point, we may have 100's of services running and trying to maintain all the related client configurations in Zuul seems a bit clumsy. Is there a way to globally configure Zuul to load all services into its client list from Eureka (or based on some service metadata in Eureka) and dynamically update this list as new services register themselves with Eureka?
Thanks!
The issue is with namespaces.If we use the default namespace it should be able to pick up the new properties addedd by default.
We've three Spring Boot applications:
Eureka Service
Config Server
Simple Web Service making use of Eureka and Config Server
I've set up the services so that we use a Eureka First Discovery, i.e. the simple web application finds out about the config server from the eureka service.
When started separately (either locally or by starting them as individual docker images) everything is ok, i.e. start config server after discovery service is running, and the Simple web service is started once the config server is running.
When docker-compose is used to start the services, they obviously start at the same time and essentially race to get up and running. This isn't an issue as we've added failFast: true and retry values to the simple web service and also have the docker container restarting so that the simple web service will eventually restart at a time when the discovery service and config server are both running but this doesn't feel optimal.
The unexpected behaviour we noticed was the following:
The simple web service reattempts a number of times to connect to the discovery service. This is sensible and expected
At the same time the simple web service attempts to contact the config server. Because it cannot contact the discovery service, it retries to connect to a config server on localhost, e.g. logs show retries going to http://localhost:8888. This wasn't expected.
The simple web service will eventually successfully connect to the discovery service but the logs show it stills tries to establish communication to the config server by going to http://localhost:8888. Again, this wasn't ideal.
Three questions/observations:
Is it a sensible strategy for the config client to fall back to trying localhost:8888 when it has been configured to use discovery to find the config server?
When the eureka connections is established, should the retry mechanism not now switch to trying the config server endpoint as indicated by Eureka? Essentially putting in higher/longer retry intervals and periods for the config server connection is pointless in this case as it's never going to connect to it if it's looking at localhost so we're better just failing fast.
Are there any properties that can override this behaviour?
I've created a sample github repo that demonstrates this behaviour:
https://github.com/KramKroc/eurekafirstdiscovery/tree/master
I use spring cloud and the FeignRibbonClient to access remote services. The problem is, that this client ignores the retry configuration given by the properties:
example-client.ribbon.MaxAutoRetries=5
example-client.ribbon.MaxAutoRetriesNextServer=5
example-client.ribbon.OkToRetryOnAllOperations=true.
The retryHandlers are created without any configuration. What I want to get is to retry the next server after ConnectException. What I get is a RetryableException caused by a ConnectException.
Does anybody knows how to get the client call to the next server in case of a ConnectException?
Thanx
Lutz