Spring + Eureka Connection refused to registered service - spring

Question

I have a setup structured of Spring Boot & Eureka running on Docker.
I have two Spring Boot applications: discussion-service and a cairo-service.
cairo-service needs to make a call to discussion-service.
Request made using Eureka via #LoadBalanced RestTemplate bean
RestTemplate attempts to reach connection to discussion-service is attempted:
Prepared URL: http://discussion-service/discussion/c3f3e424-bc67-4984-b808-e14ddf493766
The issue is that a ConnectException is thrown. Shelling into the Cairo container, pinging Discussion container is successful, endpoints are responding as expected via cURL. The URL fed into curl was http://discussion/discussion/c3f3e424-bc67-4984-b808-e14ddf493766 as that's the name of the service generated by Docker Compose. Externally, no port is 80 or 8080 - each port is some iteration greater than 8080.
I have hope my Spring implementation is perfectly functional. For some reason when I run the three Spring Boot apps (discussion-service (port 8080), cairo-service (port 8081), and nile-service (port 8761) - local eureka server) it works work excellently.
Lastly, it's worth noting I've only just begun learning this setup for a few days now.
2020-04-14 19:38:47.850 DEBUG 1 --- [nio-8080-exec-2] o.s.web.client.RestTemplate : Accept=[application/json, application/*+json]
2020-04-14 19:38:48.088 INFO 1 --- [nio-8080-exec-2] c.netflix.config.ChainedDynamicProperty : Flipping property: discussion-service.ribbon.ActiveConnectionsLimit to use NEXT property: niws.loadbalancer.availabilityFilteringRule.ac
tiveConnectionsLimit = 2147483647 2020-04-14 19:38:48.142 INFO 1 --- [nio-8080-exec-2] c.n.u.concurrent.ShutdownEnabledTimer : Shutdown hook installed for: NFLoadBalancer-PingTimer-discussion-service
2020-04-14 19:38:48.143 INFO 1 --- [nio-8080-exec-2] c.netflix.loadbalancer.BaseLoadBalancer : Client: discussion-service instantiated a LoadBalancer: DynamicServerListLoadBalancer:{NFLoadBalancer:name=discussion-service,current li
st of Servers=[],Load balancer stats=Zone stats: {},Server stats: []}ServerList:null
2020-04-14 19:38:48.154 INFO 1 --- [nio-8080-exec-2] c.n.l.DynamicServerListLoadBalancer : Using serverListUpdater PollingServerListUpdater 2020-04-14 19:38:48.195 INFO 1 --- [nio-8080-exec-2] c.netflix.config.ChainedDynamicProperty : Flipping property: discussion-service.ribbon.ActiveConnectionsLimit to use NEXT property: niws.loadbalancer.availabilityFilteringRule.ac
tiveConnectionsLimit = 2147483647
2020-04-14 19:38:48.198 INFO 1 --- [nio-8080-exec-2] c.n.l.DynamicServerListLoadBalancer : DynamicServerListLoadBalancer for client discussion-service initialized: DynamicServerListLoadBalancer:{NFLoadBalancer:name=discussion-s
ervice,current list of Servers=[nile:8080],Load balancer stats=Zone stats: {defaultzone=[Zone:defaultzone; Instance count:1; Active connections count: 0; Circuit breaker tripped count: 0; Active connections per se
rver: 0.0;]
},Server stats: [[Server:nile:8080; Zone:defaultZone; Total Requests:0; Successive connection failure:0; Total blackout seconds:0; Last connection made:Thu Jan 01 00:00:00 GMT 1970; First connection
made: Thu Jan 01 00:00:00 GMT 1970; Active Connections:0; total failure count in last (1000) msecs:0; average resp time:0.0; 90 percentile resp time:0.0; 95 percentile resp time:0.0; min resp time:0.0; max resp
time:0.0; stddev resp time:0.0]
]}ServerList:org.springframework.cloud.netflix.ribbon.eureka.DomainExtractingServerList#283933a4
2020-04-14 19:38:48.234 DEBUG 1 --- [nio-8080-exec-2] o.s.web.servlet.DispatcherServlet : Failed to complete request: org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://discussion-serv
ice/discussion/c3f3e424-bc67-4984-b808-e14ddf493766": Connection refused; nested exception is java.net.ConnectException: Connection refused
2020-04-14 19:38:48.242 ERROR 1 --- [nio-8080-exec-2] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception i
s org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://discussion-service/discussion/c3f3e424-bc67-4984-b808-e14ddf493766": Connection refused; nested exception is java.net.ConnectException: Co
nnection refused] with root cause
java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.Net.connect0(Native Method) ~[na:na]
at java.base/sun.nio.ch.Net.connect(Net.java:503) ~[na:na]
at java.base/sun.nio.ch.Net.connect(Net.java:492) ~[na:na]
at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:588) ~[na:na]
at java.base/java.net.Socket.connect(Socket.java:648) ~[na:na]
at java.base/java.net.Socket.connect(Socket.java:597) ~[na:na]
at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:182) ~[na:na]
at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:474) ~[na:na]
at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569) ~[na:na]
at java.base/sun.net.www.http.HttpClient.<init>(HttpClient.java:242) ~[na:na]
at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:341) ~[na:na]
at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:362) ~[na:na]
at java.base/sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1256) ~[na:na]
at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1194) ~[na:na]
at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1082) ~[na:na]
at java.base/sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1016) ~[na:na]
at org.springframework.http.client.SimpleBufferingClientHttpRequest.executeInternal(SimpleBufferingClientHttpRequest.java:76) ~[spring-web-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48) ~[spring-web-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53) ~[spring-web-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
at org.springframework.http.client.InterceptingClientHttpRequest$InterceptingRequestExecution.execute(InterceptingClientHttpRequest.java:109) ~[spring-web-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
at org.springframework.cloud.client.loadbalancer.LoadBalancerRequestFactory.lambda$createRequest$0(LoadBalancerRequestFactory.java:61) ~[spring-cloud-commons-2.2.2.RELEASE.jar!/:2.2.2.RELEASE]
at org.springframework.cloud.netflix.ribbon.RibbonLoadBalancerClient.execute(RibbonLoadBalancerClient.java:144) ~[spring-cloud-netflix-ribbon-2.2.2.RELEASE.jar!/:2.2.2.RELEASE]
at org.springframework.cloud.netflix.ribbon.RibbonLoadBalancerClient.execute(RibbonLoadBalancerClient.java:125) ~[spring-cloud-netflix-ribbon-2.2.2.RELEASE.jar!/:2.2.2.RELEASE]
at org.springframework.cloud.netflix.ribbon.RibbonLoadBalancerClient.execute(RibbonLoadBalancerClient.java:99) ~[spring-cloud-netflix-ribbon-2.2.2.RELEASE.jar!/:2.2.2.RELEASE]
at org.springframework.cloud.client.loadbalancer.LoadBalancerInterceptor.intercept(LoadBalancerInterceptor.java:58) ~[spring-cloud-commons-2.2.2.RELEASE.jar!/:2.2.2.RELEASE]
at org.springframework.http.client.InterceptingClientHttpRequest$InterceptingRequestExecution.execute(InterceptingClientHttpRequest.java:93) ~[spring-web-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
at org.springframework.http.client.InterceptingClientHttpRequest.executeInternal(InterceptingClientHttpRequest.java:77) ~[spring-web-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48) ~[spring-web-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53) ~[spring-web-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:739) ~[spring-web-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:674) ~[spring-web-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
at org.springframework.web.client.RestTemplate.getForObject(RestTemplate.java:315) ~[spring-web-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
at com.supaham.learn.controller.QuestionController.discussion(QuestionController.kt:53) ~[classes!/:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:564) ~[na:na]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) ~[spring-web-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) ~[spring-web-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105) ~[spring-webmvc-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:879) ~[spring-webmvc-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:793) ~[spring-webmvc-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040) ~[spring-webmvc-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943) ~[spring-webmvc-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) ~[spring-webmvc-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:634) ~[tomcat-embed-core-9.0.33.jar!/:9.0.33]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) ~[tomcat-embed-core-9.0.33.jar!/:9.0.33]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-9.0.33.jar!/:9.0.33]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.33.jar!/:9.0.33]
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.33.jar!/:9.0.33]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.33.jar!/:9.0.33]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.33.jar!/:9.0.33]
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.33.jar!/:9.0.33]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.33.jar!/:9.0.33]
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.33.jar!/:9.0.33]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.33.jar!/:9.0.33]
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:109) ~[spring-boot-actuator-2.2.6.RELEASE.jar!/:2.2.6.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.33.jar!/:9.0.33]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.33.jar!/:9.0.33]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.33.jar!/:9.0.33]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.33.jar!/:9.0.33]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) ~[tomcat-embed-core-9.0.33.jar!/:9.0.33]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) ~[tomcat-embed-core-9.0.33.jar!/:9.0.33]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) ~[tomcat-embed-core-9.0.33.jar!/:9.0.33]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) ~[tomcat-embed-core-9.0.33.jar!/:9.0.33]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) ~[tomcat-embed-core-9.0.33.jar!/:9.0.33]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) ~[tomcat-embed-core-9.0.33.jar!/:9.0.33]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) ~[tomcat-embed-core-9.0.33.jar!/:9.0.33]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:373) ~[tomcat-embed-core-9.0.33.jar!/:9.0.33]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) ~[tomcat-embed-core-9.0.33.jar!/:9.0.33]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) ~[tomcat-embed-core-9.0.33.jar!/:9.0.33]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1594) ~[tomcat-embed-core-9.0.33.jar!/:9.0.33]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-9.0.33.jar!/:9.0.33]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[na:na]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.33.jar!/:9.0.33]
at java.base/java.lang.Thread.run(Thread.java:832) ~[na:na]
2020-04-14 19:38:48.249 DEBUG 1 --- [nio-8080-exec-2] o.s.web.servlet.DispatcherServlet : "ERROR" dispatch for GET "/error", parameters={}
2020-04-14 19:38:48.254 DEBUG 1 --- [nio-8080-exec-2] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController#errorHtml(HttpServletRequest, HttpServletRespons
e)
2020-04-14 19:38:48.276 DEBUG 1 --- [nio-8080-exec-2] o.s.w.s.v.ContentNegotiatingViewResolver : Selected 'text/html' given [text/html, text/html;q=0.8]
2020-04-14 19:38:48.282 DEBUG 1 --- [nio-8080-exec-2] o.s.web.servlet.DispatcherServlet : Exiting from "ERROR" dispatch, status 500
2020-04-14 19:38:49.158 INFO 1 --- [erListUpdater-0] c.netflix.config.ChainedDynamicProperty : Flipping property: discussion-service.ribbon.ActiveConnectionsLimit to use NEXT property: niws.loadbalancer.availabilityFilteringRule.ac
tiveConnectionsLimit = 2147483647

Answer 1

After some faffing around and a lot of water it turned out to be the server implicitly being registered on port 8080 of the container. 🙉
I was baffled by what could be causing this issue considering that the containers are able to communicate via cURL. I #Autowired EurekaDiscoveryClient and began digging through its properties, specifically searching for anything regarding services.
client.services: returns [cairo-service, discussion-service]. Great!
first instance of discussion-service has an id of fc8feb8bb803:discussion-service, exactly like the Eureka UI says!
Check out the metadata from that same instance: returns a map of {management.port=8080}. Mmm that sounds relevant.
Somewhere in the middle of all this, my mind registered the slightest possibility that maybe the aforementioned "id" should be representing its port too, similar to. Lo and behold, ports are involved somehow.
On my local development machine, Eureka UI states discussion-service is registered to MY-PC:discussion-service. One important point to make here is that cairo-service is registered to MY-PC:cairo-service:8081, using an explicitly defined port of 8081. Note that I am not currently using docker for development, instead I just run services as needed (on peaceful ports), hence why MY-PC is being used as opposed to a docker container hostname. The system is functional as mentioned earlier, cairo-service can reach discussion-service. Why? Well, it turns out that Eureka works a lot better at reaching the right address and port when on its own network (or host?).
The moment the services became isolated, Eureka couldn't represent the right connection to discussion-service because it fails to include the default implied port of 8080 as part of the URL, causing the consumers to reference the URL using the default HTTP port 80, e.g. http://discussion-service:80/discussion/c3f3e424-bc67-4984-b808-e14ddf493766.
How do we solve this problem?
Turns out explicitly setting the server port causes Eureka to pick it up as part of the service id. The problem can be resolved in various ways, however my preference is the least constraining. I use the server.port property in the application.properties file. In my case I'm using application.yml
server:
port: 8080
And just like that, an entire day's worth of headache resolved in few characters.
I want to thank the Spring team for making this tool so easy to use. However, it is also the culprit in that there are so many tools being used within the framework, but also my environment, that makes it hard to understand which point is failing exactly