spring consul services are all healthy but can't reach each other when on different hosts - spring

Running Consul and spring-cloud-consul I have five micro services all registered and showing as healthy. Each of these services is running on a different EC2 instance.
Using the name of the micro service that is registered in consul I can't get a connection through to any of them using a RestTemplate
If Service A was to call Service B with the following code
private final static String SERVICE = "http://instance-service";
#LoadBalanced
#Autowired
private RestTemplate restTemplate;
#GetMapping("/instances")
public ResponseEntity<String> instances() {
ParameterizedTypeReference<String> reference = new ParameterizedTypeReference<String>() {
};
return restTemplate.exchange(SERVICE, HttpMethod.GET, null, reference);
}
Service A throws
2017-12-19 17:30:36.000 ERROR 1 --- [nio-8443-exec-7] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://instance-service": Connection refused (Connection refused); nested exception is java.net.ConnectException: Connection refused (Connection refused)] with root cause
and even with debugging set to TRACE I can see nothing coming in at all on Service B.
However if I take my identical setup and run it on a single EC2 instance everything works as expected.
All services are running in docker, but the consul agent is running on the host of each EC2 instance.
Each micro service has the following bootstrap.yml set
spring:
cloud:
consul:
config:
enabled: true
discovery:
instance-id: ${spring.application.name}
prefer-ip-address: true
If I give the RestTemplate a random endpoint it returns Unknown host so it knows the micro service is there, it's just being blocked somewhere before it hits the codebase.
How can I debug this further to find the real reason the services are getting rejected when running on different hosts?
I have tried opening all ports on the security config just to rule that out but it made no difference.
Edit: Looking at the Consul UI - I see my services are registered under the a private IP (172.18.0.2).
Should this be a public facing address?

172.18.0.2 is an ip in a a docker bridged network, you definitely want the ip address that is registered in consul to be accessible by clients that use the service discovery..
I don't know your network setup or needs but you can (some ideas, there are many) -
if you expose ports to the docker host (-p), make sure the ip of the docker host is registered for each service instance (I'm not familiar with bootstrap.yml or spring cloud)
run docker containers with --net=host
register the services in consul in different methods and outside of spring cloud's scope (API call to the consul agent, service definition file, https://github.com/gliderlabs/registrator, ...)

Related

Eureka Client Registered with Container IP in ECS Fargate

We deployed Spring Boot services in the ECS Fargate with Eureka Registry Service, Gateway Service and a few business services. In the Eureka registry page all the client services registered were showing same IP (could be container IP).
Please share what configuration should be provided to register with ECS task private ip instead of container IP. Thanks in advance.
Eureka Service Application Properties:
server.port=8080
eureka.client.register-with-eureka=false
eureka.client.fetch-registry=false
Eureka Client Service Application Properties
server.port=8766
spring.application.name= client-service
eureka.client.register-with-eureka=true
eureka.client.fetch-registry=true
eureka.instance.prefer-ip-address= true
eureka.client.service-url.defaultZone= <eureka-ecs-ip>:8080
eureka.instance.lease-renewal-interval-in-seconds=30
Problem solved by adding the below code to SpringBootApplication source. This block picks update host IP address (not container) and overrides the config property/yaml settings.
#Bean
public EurekaInstanceConfigBean eurekaInstanceConfig(InetUtils inetUtils){
EurekaInstanceConfigBean config = new EurekaInstanceConfigBean(inetUtils);
String ip = null;
try {
ip = InetAddress.getLocalHost().getHostAddress();
} catch (UnknownHostException e) {
e.printStackTrace();
}
config.setIpAddress(ip);
config.setPreferIpAddress(true);
return config;
}
To anyone that still can't fix it with the accepted answer:
I also had to set secure/non-secure port in config (and enable it).

How to find IPs of Eureka registered service on the clients

I have the following services:
EurekaServer - hosts the eureka discovery server
Client-service - registers to EurekaServer
Finder-service - registers to EurekaServer
Is there a way to get Client-service's ip so I can make requests to it from the Finder-service.
I know there is a way to find InstanceInfo from EurekaServer and I was thinking of making a controller in eureka server where you pass service id and get service's instance ip. This way Finder-service would only need to know service id and eureka ip which it knows because it is registered there.
Is there another solution which is cleaner than this?
To use client discovery you first need to enable by adding either #EnableEurekaClient or #EnableDiscoveryClient to your #SpringBootApplication annotated class (or your specialized #Configuration).
Next to make use of the resolution of the actual service instance to use you need to create a RestTemplate which is load-balanced. This will add an interceptor which translates the service name into the ip-address / DNS name to send the request to (if multiple instances are found it will distribute the load between those instances).
To create a load balanced RestTemplate add the #LoadBalanced annotation to the configured RestTemplate.
#Bean
#LoadBalanced
public RestTemplate restTemplate(RestTemplateBuilder rtb) {
return rtb.build();
}
Now when doing a request through the RestTemplate it will resolve the service name client-service to the actual service instance to use. Without you having to do anything.

Getting No instance available for a microservice hosted in PCF

I have hosted 3 microservices in PCF. One is a eureka server and the other 2 are client and service microservices. The client is supposed to call the service through a rest template call and service will return a string.
I have 1 instance each of the eureka server and the client and 2 instances of the service.
I can see both my client and service registered in the Eureka dashboard. But when i try to access the service from the client[Using a rest template call] I get - 'No instances available for the [service name]'
But if i access my service directly from browser then its works fine and returns the string. But the same URL if called from a rest template returns the exception i mentioned above.
Any suggestions will help
Did you add the #LoadBalance on restTemplate bean.
#Bean
#LoadBalanced
public RestTemplate restTemplate() {
return new RestTemplate();
}
And when calling service you have to use the service name as below.
https://SERVICENAME/restpath

Feign Client Prioritizing URL's in yaml over Eureka

I have a Spring Boot application which serves as a Eureka client. The application has the need to call another micro-service through REST, and I wish to make this call using Feign. The issue I am having is, my application is trying to lookup the service name in Eureka, when it is only defined in my applications yaml file.
I apologize for the hard to follow explanation, hopefully the following code snippets will help clarify.
Feign client:
#FeignClient("foo")
#Component
public interface FooServiceProxy{
#RequestMapping(value = "/balance", method = RequestMethod.POST, produces = "application/json")
ServiceResponse execute(ServiceRequest serviceRequest);
}
In my controller who calls this Feign client, the FooServiceProxy is defined using #AutoWired:
#Autowired
private FooServiceProxy fooServiceProxy;
My yaml file is as follows:
spring:
application:
name: app-name
server:
port: 8080
foo:
ribbon:
listOfServers: http://hostname:8081/balance
eureka:
client:
fetchRegistry: false
serviceUrl:
defaultZone: http://eurekasrver:8761/eureka/
My issue is, during run-time, the following error is thrown:
java.lang.RuntimeException: com.netflix.client.ClientException: Load balancer does not have available server for client: foo
Interestingly, if I remove the #EnableEurekaClient annotation from the application, everything works. I believe I understand the problem which is that instead of looking up the server for foo in my yaml file, because the application is a Eureka client, Feign is going straight to Eureka to lookup a server ip, then failing as none can be found. Despite seeming to understand the problem, I have been unable to find a solution online or to think of one myself.
Any help will be appreciated.
Thank you!
Concerning this question, you should take in account that when eureka is on your classpath, all ribbon configuration are charged by eureka, so it'll use eureka server's list.
Spring Cloud uses #RibbonClient to configure the types used by ribbon, like server list. If you have eureka on the classpath, by default it uses the eureka server list (hence your need for the flag to disable eureka).
Commented by spencergibb https://github.com/spring-cloud/spring-cloud-netflix/issues/564
You can try either by adding the NIWSServerListClassName configuration:
`someservice.ribbon:
NIWSServerListClassName:com.netflix.loadbalancer.ConfigurationBasedServerList
listOfServers: server1:80`
Or try the solution proposed in this issue https://github.com/spring-cloud/spring-cloud-netflix/issues/564

java.net.UnknownHostException during Eureka service discovery

According to this blog https://spring.io/blog/2015/07/14/microservices-with-spring
Was able to run the application without any issues. In this order:
java -jar microservice-demo-0.0.1-SNAPSHOT.jar registration 1111
java -jar microservice-demo-0.0.1-SNAPSHOT.jar accounts 2222
java -jar microservice-demo-0.0.1-SNAPSHOT.jar web 3333
But when trying to hit any service through the web application (http://localhost:3333/) which uses the http://ACCOUNTS-SERVICE url to access any accounts service endpoints like http://ACCOUNTS-SERVICE/accounts/123456789 I'm getting an error response:
Response Status: 500 (Internal Server Error)
Cause: org.springframework.web.client.ResourceAccessException I/O error on GET request for "http://ACCOUNTS-SERVICE/accounts/123456789": ACCOUNTS-SERVICE; nested exception is java.net.UnknownHostException: ACCOUNTS-SERVICE
When I provide the real address (http://localhost:2223/) of the accounts service to the web server instead of the http://ACCOUNTS-SERVICE everything works properly but there is no service discovery in this case.
The source code is stored at: https://github.com/paulc4/microservices-demo
This issue was due to the RestTemplate was no longer auto-created in the Brixton release-train (Spring Cloud 1.1.0.RELEASE), so the RestTemplate could not resolve properly the http://ACCOUNTS-SERVICE url using the service discovery server.
Was able to fix this issue after declaring a RestTemplate bean with #LoadBalanced as follows:
#Bean
#LoadBalanced
public RestTemplate restTemplate() {
return new RestTemplate();
}

Resources