My feign call is getting read timeout, even after adding below property to my application.properties.
hystrix.command.default.execution.timeout.enabled=false
I have two more services, which got started successfully and got registered to my eureka.
But my third (A) service is not starting due to readtime out issue. Below is the case
A calls B (For authorization during application start up - Quartz Job)
A calls C (flow ends) and application should start
I have to start B & C before i start A. This is how it is been designed and i dont know the business case for this.
After adding hystrix timeout, the service (A) is still throwing feign.RetryableException: Read timed out executing POST.
I am not able to trouble shoot , since the request are routed through eureka services, and all services are running locally , there should not be any network latency.
My eureka configuration
spring.application.name=Foo-service
eureka.client.registerWithEureka=true eureka.client.fetchRegistry=true
eureka.client.serviceUrl.defaultZone=http://localhost:9094/eureka
hystrix.command.default.execution.timeout.enabled=false
Related
There are two microservices (microservice-A & microservice-B), written in Spring boot, are talking to each other through the Feign Client and the services are registered in K8S Native service registry
Whenever a fresh deployment of a microservice-A is happening, the microservice-B that is already running in kubertnetes fails to make HTTP call to the freshly deployed service-A and below is the exception
feign.RetryableException: No route to host (Host unreachable) executing GET http://microserice-a/api/v1/myresources
This issue is getting resolved immediately after restarting the microservice-B.
When we googled for solutions, we got to see this link https://github.com/spring-cloud/spring-cloud-netflix/issues/769 and an user had given the below comment there
I suspect the root cause is that FeignClient keeps an old list of service providers and the Ribbon cannot move correctly to the next node if one node has been destroyed
Not sure if that is correct root cause. Please comment If anyone has faced similar issue and solved it?
I have below architecture in my project
My UI Service(Port 8080) making Feign call to Gateway Service(Port 8085).
My Get call from UI service is " http://localhost:8080/invoice-list?startDate=2018-08-05&endDate=2018-10-05 "
Similar call from Gateway Service "http://localhost:8085/invoice-download-service/invoice-list?startDate=2018-08-05&endDate=2018-10-05"
When i make this GET call from UI service i get below error within minute
is feign.RetryableException: Read timed out executing GET http://localhost:8085/invoice-download-service/invoice-list?startDate=2018-08-05&endDate=2018-10-05] with root cause
java.net.SocketTimeoutException: Read timed out
But when i make direct call from Gateway Server to microservice, i dont get error.
Application.properties file of Gateway service
hystrix.command.default.execution.isolation.thread.timeoutInMilliseconds=160000000
ribbon.OkToRetryOnAllOperations=true
ribbon.ReadTimeout=5000000
ribbon.ConnectTimeout=5000000
ribbon.MaxAutoRetries=3
ribbon.MaxAutoRetriesNextServer=3
zuul.host.socket-timeout-millis= 5000000
zuul.host.connect-timeout-millis= 5000000
Here i have set readtimeout and connecttimeout property around 8 to 10 min, hence i am not getting error.
Application.properties file of UI service
spring.application.name=external-ui-service
server.port=8080
Here in UI service i dont have timeout property. I tried above properties here but not working.
Obviously this UI service is not using ribbon,zuul etc. This is just an making Feign call to gateway.
So what should i do to increase timeout in UI service?
Added below properties in UI Service's application.propeties file.
feign.client.config.default.connectTimeout: 160000000
feign.client.config.default.readTimeout: 160000000
This issue might also be caused by default laodbalancer implementation of Spring Cloud Gateway in case you make use of Eureka Server and run your microservices undockerized on windows. Services are running on localhost, but Eureka says to the loadbalancer of the gateway to route the request to host.docker.internal.
The links down below give a couple of solutions:
https://localcoder.org/spring-boot-cloud-eurka-windows-10-eurkea-returns-host-docker-internal-for-clien
https://dimitr.im/fix-eureka-localhost
I have created a spring boot integration app and deployed it to Pivotal Cloud Foundry (PCF) environment. It works for couple of days and then it starts to crash randomly afterwards. I checked the PCF logs and found this information about the crash.
OUTApp instance exited with guid 3c348d47-48c4-403f-950a-29af1efa551d
payload: {"instance"=>"e2122543-214f-4806-62c7-00e1", "index"=>2,
"reason"=>"CRASHED", "exit_description"=>"Instance became unhealthy: Failed
to make HTTP request to '/health' on port 8080: timed out after 1.00
seconds", "crash_count"=>1, "crash_timestamp"=>1511959503256098495,
"version"=>"10cea919-d490-460d-83d6-5132c96ef781"}
My CPU utilization is not much. My memory is also not leaking.
Information about the application deployed in PCF:
Spring boot integration app connects to IBM MQ queues and polls for messages and then calls couple of web services.
There is also another application Service Bus, which makes the health check call on PCF application to check if the PCF app is available or not. If Service Bus finds that PCF app is available then the requests are routed to PCF else they are processed at Service Bus end itself.
Please let me know, how to find the root cause of the CRASH and fix it.
Thanks in advance. Please let me know, if you need further details.
I have changed the health check type to port type from http in manifest.yml file.
configuration change in manifest file is as follows:
health-check-type: port
Now the app is not crashing. It is working fine. Hope this helps.
We've three Spring Boot applications:
Eureka Service
Config Server
Simple Web Service making use of Eureka and Config Server
I've set up the services so that we use a Eureka First Discovery, i.e. the simple web application finds out about the config server from the eureka service.
When started separately (either locally or by starting them as individual docker images) everything is ok, i.e. start config server after discovery service is running, and the Simple web service is started once the config server is running.
When docker-compose is used to start the services, they obviously start at the same time and essentially race to get up and running. This isn't an issue as we've added failFast: true and retry values to the simple web service and also have the docker container restarting so that the simple web service will eventually restart at a time when the discovery service and config server are both running but this doesn't feel optimal.
The unexpected behaviour we noticed was the following:
The simple web service reattempts a number of times to connect to the discovery service. This is sensible and expected
At the same time the simple web service attempts to contact the config server. Because it cannot contact the discovery service, it retries to connect to a config server on localhost, e.g. logs show retries going to http://localhost:8888. This wasn't expected.
The simple web service will eventually successfully connect to the discovery service but the logs show it stills tries to establish communication to the config server by going to http://localhost:8888. Again, this wasn't ideal.
Three questions/observations:
Is it a sensible strategy for the config client to fall back to trying localhost:8888 when it has been configured to use discovery to find the config server?
When the eureka connections is established, should the retry mechanism not now switch to trying the config server endpoint as indicated by Eureka? Essentially putting in higher/longer retry intervals and periods for the config server connection is pointless in this case as it's never going to connect to it if it's looking at localhost so we're better just failing fast.
Are there any properties that can override this behaviour?
I've created a sample github repo that demonstrates this behaviour:
https://github.com/KramKroc/eurekafirstdiscovery/tree/master
I use spring cloud and the FeignRibbonClient to access remote services. The problem is, that this client ignores the retry configuration given by the properties:
example-client.ribbon.MaxAutoRetries=5
example-client.ribbon.MaxAutoRetriesNextServer=5
example-client.ribbon.OkToRetryOnAllOperations=true.
The retryHandlers are created without any configuration. What I want to get is to retry the next server after ConnectException. What I get is a RetryableException caused by a ConnectException.
Does anybody knows how to get the client call to the next server in case of a ConnectException?
Thanx
Lutz