I have below architecture in my project
My UI Service(Port 8080) making Feign call to Gateway Service(Port 8085).
My Get call from UI service is " http://localhost:8080/invoice-list?startDate=2018-08-05&endDate=2018-10-05 "
Similar call from Gateway Service "http://localhost:8085/invoice-download-service/invoice-list?startDate=2018-08-05&endDate=2018-10-05"
When i make this GET call from UI service i get below error within minute
is feign.RetryableException: Read timed out executing GET http://localhost:8085/invoice-download-service/invoice-list?startDate=2018-08-05&endDate=2018-10-05] with root cause
java.net.SocketTimeoutException: Read timed out
But when i make direct call from Gateway Server to microservice, i dont get error.
Application.properties file of Gateway service
hystrix.command.default.execution.isolation.thread.timeoutInMilliseconds=160000000
ribbon.OkToRetryOnAllOperations=true
ribbon.ReadTimeout=5000000
ribbon.ConnectTimeout=5000000
ribbon.MaxAutoRetries=3
ribbon.MaxAutoRetriesNextServer=3
zuul.host.socket-timeout-millis= 5000000
zuul.host.connect-timeout-millis= 5000000
Here i have set readtimeout and connecttimeout property around 8 to 10 min, hence i am not getting error.
Application.properties file of UI service
spring.application.name=external-ui-service
server.port=8080
Here in UI service i dont have timeout property. I tried above properties here but not working.
Obviously this UI service is not using ribbon,zuul etc. This is just an making Feign call to gateway.
So what should i do to increase timeout in UI service?
Added below properties in UI Service's application.propeties file.
feign.client.config.default.connectTimeout: 160000000
feign.client.config.default.readTimeout: 160000000
This issue might also be caused by default laodbalancer implementation of Spring Cloud Gateway in case you make use of Eureka Server and run your microservices undockerized on windows. Services are running on localhost, but Eureka says to the loadbalancer of the gateway to route the request to host.docker.internal.
The links down below give a couple of solutions:
https://localcoder.org/spring-boot-cloud-eurka-windows-10-eurkea-returns-host-docker-internal-for-clien
https://dimitr.im/fix-eureka-localhost
Related
I have configured zuul with 2 instances using ribbon (without eureka) as below:
zuul.retryable=true
zuul.routes.simple-ms-app.serviceId: client
client.ribbon.listOfServers=http://localhost:7788,http://localhost:8877
When both the instances 7788 & 8877 are up and running, everything goes fine.
When the first instance in the listOfServers is down, then the request ends up in the below error:
com.netflix.zuul.exception.ZuulException: Forwarding error
I am using the below version configuration:
spring-boot : 2.0.7.RELEASE
spring-cloud: Finchley.SR2
If anyone had faced similar issue and managed to figure out a solution, please share it here.
Thank you.
By default, Zuul throws exception (instead of throwing 503/404) when upstream service is not available. This behavior has been discussed in detail in Zuul swallows 503 exceptions from upstream microservices GitHub thread.
To handle this case and configure Zuul to retry on (current and next ) available instances, you need to do two things:
Extend ErrorFilter and handle the exception with custom behavior
Configure retry for Zuul
Extend ErrorFilter and provide custom logic to return 404 or 503 status code. Some of the approaches to deal with this exception is explain in this SO thread: Customizing Zuul Exception.
Retry in Zuul can be configured using following application properties:
zuul:
retryable: true
ribbon:
MaxAutoRetries: 1
MaxAutoRetriesNextServer: 3
OkToRetryOnAllOperations: true
yourApplication:
ribbon:
listOfServers: instance-1-url, instance-2-url
Please note that Spring retry is a dependency for retry in Zuul.
My feign call is getting read timeout, even after adding below property to my application.properties.
hystrix.command.default.execution.timeout.enabled=false
I have two more services, which got started successfully and got registered to my eureka.
But my third (A) service is not starting due to readtime out issue. Below is the case
A calls B (For authorization during application start up - Quartz Job)
A calls C (flow ends) and application should start
I have to start B & C before i start A. This is how it is been designed and i dont know the business case for this.
After adding hystrix timeout, the service (A) is still throwing feign.RetryableException: Read timed out executing POST.
I am not able to trouble shoot , since the request are routed through eureka services, and all services are running locally , there should not be any network latency.
My eureka configuration
spring.application.name=Foo-service
eureka.client.registerWithEureka=true eureka.client.fetchRegistry=true
eureka.client.serviceUrl.defaultZone=http://localhost:9094/eureka
hystrix.command.default.execution.timeout.enabled=false
I have a Zuul proxy, and it works fine until it redirects to a server that takes longer time than normal (more than 10 seconds) to response.
I can modify the read-timeout through properties (zuul.host.socket-timeout-millis) so it can work properly.
ribbon.eureka.enabled=false
zuul.host.connect-timeout-millis=100000
zuul.host.socket-timeout-millis=100000
But I need a different timeout based on the service that the proxy is reaching.
I don't have Eureka in my architecture so I disabled it in the properties. (ribbon.eureka.enabled=false)
I've tried with
hystrix.command.default.execution.timeout.enabled=false
hystrix.command.messages.execution.isolation.strategy=THREAD
hystrix.command.messages.execution.isolation.thread.timeoutInMilliseconds= 100000
But it doesn't work. It always use the value on zuul.host.socket-timeout-millis and if I don't set this property Zuul, uses a 10 seconds timeout.
I've also tried
messages.ribbon.ReadTimeout=120000
ribbon.ReadTimeout=60000
But it always use what is on zuul.host.socket-timeout-millis
I don't know how to set the timeout by service.
So I can set for service1 30 seconds, for service2 45 seconds and for the other services a default of 60.
How can I do this ?
I supose you have but I'll ask: Have you tried configuring both properties at the same time with a higher value?
It should work, we use this feature in Zuul and it works well but you need to configure booth timeouts (Zuul and Ribbon)
You have info about how this works here: Ribbon, Hystrix
Note that depending on how you use Zuul the name of your client could be different for Ribbon and for Hystrix.
We've three Spring Boot applications:
Eureka Service
Config Server
Simple Web Service making use of Eureka and Config Server
I've set up the services so that we use a Eureka First Discovery, i.e. the simple web application finds out about the config server from the eureka service.
When started separately (either locally or by starting them as individual docker images) everything is ok, i.e. start config server after discovery service is running, and the Simple web service is started once the config server is running.
When docker-compose is used to start the services, they obviously start at the same time and essentially race to get up and running. This isn't an issue as we've added failFast: true and retry values to the simple web service and also have the docker container restarting so that the simple web service will eventually restart at a time when the discovery service and config server are both running but this doesn't feel optimal.
The unexpected behaviour we noticed was the following:
The simple web service reattempts a number of times to connect to the discovery service. This is sensible and expected
At the same time the simple web service attempts to contact the config server. Because it cannot contact the discovery service, it retries to connect to a config server on localhost, e.g. logs show retries going to http://localhost:8888. This wasn't expected.
The simple web service will eventually successfully connect to the discovery service but the logs show it stills tries to establish communication to the config server by going to http://localhost:8888. Again, this wasn't ideal.
Three questions/observations:
Is it a sensible strategy for the config client to fall back to trying localhost:8888 when it has been configured to use discovery to find the config server?
When the eureka connections is established, should the retry mechanism not now switch to trying the config server endpoint as indicated by Eureka? Essentially putting in higher/longer retry intervals and periods for the config server connection is pointless in this case as it's never going to connect to it if it's looking at localhost so we're better just failing fast.
Are there any properties that can override this behaviour?
I've created a sample github repo that demonstrates this behaviour:
https://github.com/KramKroc/eurekafirstdiscovery/tree/master
I use spring cloud and the FeignRibbonClient to access remote services. The problem is, that this client ignores the retry configuration given by the properties:
example-client.ribbon.MaxAutoRetries=5
example-client.ribbon.MaxAutoRetriesNextServer=5
example-client.ribbon.OkToRetryOnAllOperations=true.
The retryHandlers are created without any configuration. What I want to get is to retry the next server after ConnectException. What I get is a RetryableException caused by a ConnectException.
Does anybody knows how to get the client call to the next server in case of a ConnectException?
Thanx
Lutz