Loadbalancing fails when a server is down - spring

Question

I have written a simple set of micro-services with the following architecture:
For all, I have added spring-boot-starter-actuator in order to add /health endpoint.
In Zuul/Ribbon configuration I have added :
zuul:
ignoredServices: "*"
routes:
home-service:
path: /service/**
serviceId: home-service
retryable: true
home-service:
ribbon:
listOfServers: localhost:8080,localhost:8081
eureka.enabled: false
ServerListRefreshInterval: 1
So that, each time client will call GET http://localhost:7070/service/home, loadbalancer will choose one of two HomeService which runs on 8080 or 8081 port and will call its endpoint /home.
But, when one of HomeService is shutdown, the loadbalancer does not seem to be aware (in spite of ServerListRefreshInterval configuration) and will fail with error=500 if it tries to call the shutdown instance.
How could I fix it?

Answer 1

I have received and tested a solution from spring-cloud team.
Solution is here in github
To summarize:
I have added org.springframework.retry.spring-retry to my zuul classpath
I have added #EnableRetry to my zuul application
I have put the following properties in my zuul configuration
application.yml
server:
port: ${PORT:7070}
spring:
application:
name: gateway
endpoints:
health:
enabled: true
sensitive: true
restart:
enabled: true
shutdown:
enabled: true
zuul:
ignoredServices: "*"
routes:
home-service:
path: /service/**
serviceId: home-service
retryable: true
retryable: true
home-service:
ribbon:
listOfServers: localhost:8080,localhost:8081
eureka.enabled: false
ServerListRefreshInterval: 100
retryableStatusCodes: 500
MaxAutoRetries: 2
MaxAutoRetriesNextServer: 1
OkToRetryOnAllOperations: true
ReadTimeout: 10000
ConnectTimeout: 10000
EnablePrimeConnections: true
ribbon:
eureka:
enabled: false
hystrix:
command:
default:
execution:
isolation:
thread:
timeoutInMilliseconds: 30000

Answer 2

Debugging timeouts may be tricky, considering there are 3 levels of routing alone (Zuul→Hystrix→Ribbon), not including async execution layers and the retry engine. The following scheme is valid for Spring Cloud releases Camden.SR6 and newer (I've checked this on Dalston.SR1):
Zuul routes the request through RibbonRoutingFilter, which creates a Ribbon command with the request context. Ribbon command then creates a LoadBalancer command, which uses spring-retry for command execution, choosing retry policy for the RetryTemplate according to Zuul settings. #EnableRetry does nothing in this case, because this annotation enables wrapping methods with #Retryable annotation in retrying proxies.
This means, your command duration is limited to the lesser value of these two (see this post):
[HystrixTimeout], which is a timeout for invoked Hystrix command
[RibbonTimeout * MaxAutoRetries * MaxAutoRetriesNextServer] (retries kick in only if Zuul has them enabled in its configuration), where [RibbonTimeout = ConnectTimeout + ReadTimeout] on the http client.
For debugging, it's convenient to create a breakpoint in RetryableRibbonLoadBalancingHttpClient#executeWithRetry or RetryableRibbonLoadBalancingHttpClient#execute method. At this point, you have:
ContextAwareRequest instance (e.g. RibbonApacheHttpRequest or OkHttpRibbonRequest) with request context, which containes Zuul's retryable property;
LoadBalancedRetryPolicy intsance with load balancer context, which contains Ribbon's maxAutoRetries, maxAutoRetriesNextServer and okToRetryOnAllOperations properties;
RetryCallback instance with a requestConfig, which contains HttpClient's connectTimeout and socketTimeout properties;
RetryTemplate instance with chosen retry policy.
If the breakpoint is not hit, it means that org.springframework.cloud.netflix.ribbon.apache.RetryableRibbonLoadBalancingHttpClient bean was not instantiated. This happenes when the spring-retry library is not in the classpath.