Like a few others before me, I cannot get hystrix streams reported by my services to be aggregated by turbine (local, not amqp). I have read all the questions and answers here on SO, applied their advice and have got nowhere. Here's my setup.
Version: Brixton M4
Services running on localhost: eureka, zuul, myservice, myservice2, turbine
myservice and myservice2 are microservices that expose hystrix.stream. As with zuul, I can connect directly to these hystrix.stream endpoints and see data. turbine.stream is always empty.
turbine application.yml
The turbine part of this configuration is taken almost directly from the example in the spring cloud docs.
spring:
application:
name: turbine
server:
port: 8989
management:
port: 8990
turbine:
aggregator:
clusterNameExpression: metadata['cluster']
clusterConfig: LOCAL
appConfig: myservice,myservice2,zuul
InstanceMonitor:
eventStream:
skipLineLogic:
enabled: false
eureka:
instance:
leaseRenewalIntervalInSeconds: 10
client:
serviceUrl:
defaultZone: http://localhost:8761/eureka/
info:
component: Turbine!
Each of the 3 services includes:
eureka:
instance:
leaseRenewalIntervalInSeconds: 10
metadataMap:
cluster: LOCAL
client:
serviceUrl:
defaultZone: http://localhost:8761/eureka/
Verifying that the metadata map is correct in Eureka from http://localhost:8761/eureka/apps
<applications>
<versions__delta>1</versions__delta>
<apps__hashcode>UP_4_</apps__hashcode>
<application>
<name>MYSERVICE</name>
<instance>
<instanceId>192.168.43.128:myservice:2222</instanceId>
<hostName>192.168.43.128</hostName>
<app>MYSERVICE</app>
<ipAddr>192.168.43.128</ipAddr>
<status>UP</status>
<overriddenstatus>UNKNOWN</overriddenstatus>
<port enabled="true">2222</port>
<securePort enabled="false">443</securePort>
<countryId>1</countryId>
<dataCenterInfo class="com.netflix.appinfo.InstanceInfo$DefaultDataCenterInfo">
<name>MyOwn</name>
</dataCenterInfo>
<leaseInfo>
<renewalIntervalInSecs>10</renewalIntervalInSecs>
<durationInSecs>90</durationInSecs>
<registrationTimestamp>1453382031096</registrationTimestamp>
<lastRenewalTimestamp>1453382630966</lastRenewalTimestamp>
<evictionTimestamp>0</evictionTimestamp>
<serviceUpTimestamp>1453382031096</serviceUpTimestamp>
</leaseInfo>
<metadata>
<cluster>LOCAL</cluster>
</metadata>
When I run turbine it is finding the service instances in Eureka but is not choosing them to report data:
o.s.c.n.t.CommonsInstanceDiscovery : Fetching instance list for apps: [myservice, myservice2, zuul]
o.s.c.n.turbine.EurekaInstanceDiscovery : Fetching instances for app: myservice
o.s.c.n.turbine.EurekaInstanceDiscovery : Received instance list for app: myservice, size=1
o.s.c.n.turbine.EurekaInstanceDiscovery : Fetching instances for app: myservice2
o.s.c.n.turbine.EurekaInstanceDiscovery : Received instance list for app: myservice2, size=1
o.s.c.n.turbine.EurekaInstanceDiscovery : Fetching instances for app: zuul
o.s.c.n.turbine.EurekaInstanceDiscovery : Received instance list for app: zuul, size=1
c.n.t.discovery.InstanceObservable : Retrieved hosts from InstanceDiscovery: 3
c.n.t.discovery.InstanceObservable : Found hosts that have been previously terminated: 0
And another (potentially useful?) log line that appears when logging is raised to DEBUG:
c.n.t.discovery.InstanceObservable : Retrieved hosts from InstanceDiscovery: [StatsInstance [hostname=192.168.43.128, cluster: MYSERVICE, isUp: true, attrs={cluster=LOCAL, port=2222}], StatsInstance [hostname=192.168.43.128, cluster: MYSERVICE2, isUp: true, attrs={cluster=LOCAL, port=2223}], StatsInstance [hostname=192.168.43.128, cluster: ZUUL, isUp: true, attrs={cluster=LOCAL, port=8765}]]
OK never mind, it was a silly mistake on my part that I spotted while running turbine in the debugger. clusterNameExpression is a child of turbine and not aggregator.
With that error corrected I can see the first service in the comma separated list reporting data in the turbine stream but not the others. Is this expected? i.e. Is turbine designed for monitoring the streams from multiple microservices that make up the same logical application or is it purely for multiple instances of the same microservice?
Related
Spring Boot version: 2.1.6.RELEASE
Spring Cloud Version: Greenwich.SR1
Objective is to set up two instances of Eureka Servers in a cluster and have all microservices registered to both, in order to achieve HA. Currently I am testing this on my local machine running Ubuntu 18.04.
So as mentioned in the official docs, I have set up the peer to peer awareness of Eureka Instances. On bringing both the Eureka Server Instances up, on the dashboard of Instance-1(Port 8080) under available replicas showing Instance-2 name and on dashboard of Instance-2(Port 8081) it is showing Instance-1 name. So far so good.
I have deployed them on external tomcat server as war and not running them as jar. War name is "eureka-naming-server". So my access URL becomes: http://localhost:8080/eureka-naming-server/eureka and http://localhost:8081/eureka-naming-server/eureka.
I have set up a load balancer using Apache 2 and mod_jk. So instead of accessing them using their individual URLs i.e. http://localhost:8080/eureka-naming-server/eureka, http://localhost:8081/eureka-naming-server/eureka, I can access them using Apache URL as http://localhost:80/eureka-naming-server/eureka and I can see load-balancing happening properly.
Now the issue is that once I give Eureka URL as http://localhost:80/eureka-naming-server/eureka in the microservice, it registers only on one instance depending on which Eureka instance the load-balancer has redirected the request. But even though peer to peer awareness is setup, microservice is not getting registered on the other eureka instance.
In Eureak servers, I'm using the below dependency:
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-netflix-eureka-server</artifactId>
</dependency>
Eureka-1 props:
spring:
application:
name: eureka-naming-server
jmx:
default-domain: eureka-naming-server
eureka:
client:
service-url:
defaultZone: http://eureka-server-2:8081/eureka-naming-server/eureka
#register-with-eureka: false
#fetch-registry: false
instance:
hostname: eureka-server-1
Eureka-2 props:
spring:
application:
name: eureka-naming-server
jmx:
default-domain: eureka-naming-server
eureka:
client:
service-url:
defaultZone: http://eureka-server-1:8080/eureka-naming-server/eureka
#register-with-eureka: false
#fetch-registry: false
instance:
hostname: eureka-server-2
In microservices, I'm using below dependencies:
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-netflix-ribbon</artifactId>
</dependency>
Microservice props:
eureka:
instance:
metadata-map:
configPath: /config
client:
service-url:
defaultZone: http://localhost/eureka-naming-server/eureka
In the machine's /etc/hosts file I have set it up as:
127.0.0.1 eureka-server-1
127.0.0.1 eureka-server-2
The eureka instance on which it is getting registered I can see the below in the logs:
WARN 13748 --- [nio-8009-exec-3] Registered instance CONFIG-SERVER/192.168.1.16:config-server:8888 with status UP (replication=false)
The other Eureka instance where it is not registered, when it receives the heart-beat because of the load-balancer, shows this in the logs:
WARN 13748 --- [nio-8009-exec-3] c.n.e.registry.AbstractInstanceRegistry : DS: Registry: lease doesn't exist, registering resource: CONFIG-SERVER - 192.168.1.16:config-server:8888
2019-12-26 19:00:34.995 WARN 13748 --- [nio-8009-exec-3] c.n.eureka.resources.InstanceResource : Not Found (Renew): CONFIG-SERVER - 192.168.1.16:config-server:8888
Eureka-1 Dashboard
Eureka-2 Dashboard
I have already tried suggestions mentioned here and here. But nothing seems to work.
So what could I be missing here?
Issue:
spring:
application:
name: eureka-naming-server
You put the same name for an application so it's displayed like this.
you need to change it to,
Eureka-1 props:
spring:
application:
name: eureka-naming-server-1
Eureka-2 props:
spring:
application:
name: eureka-naming-server-2
It may also be worth checking your security configuration for your eureka servers. If you do have spring boot security enabled, you will need to be sure that you include the corresponding username/password in the defaultZone portion of your config.
i.e. in your eureka server 1 config: eureka.client.serviceUrl.defaultZone=http://:#eureka-server-2:8761/eureka
(and visa versa for your eureka server 2 config)
I only mention this because I had left out my own username/password values in that config value and it was not registering my microservices on all my eureka instances. After make this change, it worked as I was expecting.
I have a couple of microservices on Heroku - eureka-server, Zuul server and some app services.
While I am trying to reach any of my services, for example, "service1" via Zuul gateway, Zuul is unable to forward the request to the respective service (when I'm trying to run them locally everything works fine).
I've found the following errors in Zuul logs:
com.netflix.zuul.exception.ZuulException: Forwarding error
Caused by: com.netflix.client.ClientException: Load balancer does not have the available server for the client: service1
Below are configurations of my services:
1) "zuul server" application.yml
server:
port: ${PORT:8000}
zuul:
prefix: /api
ignoredServices: '*'
routes:
service1:
path: /path_for_service1/**
serviceId: service1
strip-prefix: false
...
management:
endpoints:
web:
exposure:
include: "*"
eureka:
client:
serviceUrl:
defaultZone: ${EUREKA_URL:http://localhost:5000}/eureka/
2) "eureka server" application.yml
server:
port: ${PORT:5000}
eureka:
instance:
hostname: localhost
client:
registerWithEureka: false
fetchRegistry: false
serviceUrl:
defaultZone: http://${eureka.instance.hostname}:${server.port}/eureka/
3.1) "service1" application.yml
server.port=${PORT:8081}
eureka.client.service-url.defaultZone=${EUREKA_URL:http://localhost:5000}/eureka/
3.2) "service1" bootstrap.yml
spring:
application:
name: service1
All microservices are visible in eureka dashboard.
If I change zuul routes to hard-coded url it works well but it is not what I'm looking for.
zuul:
prefix: /api
ignoredServices: '*'
routes:
service1:
path: /path_for_service1/**
url: http://url_of_service_1
strip-prefix: false
Could you please help me with this issue?
Finally, I've found root cause :)
All services are registered in eureka using heroku hostname by default (like "085930c7-b893-4b34-88a7-6e37fbe7fa0f") which is not accessible outside.
But services are accessible by domain names.
So I just added domain name settings to application.properties of each service (https://blog.heroku.com/managing_your_microservices_on_heroku_with_netflix_s_eureka)
eureka:
instance:
non-secure-port: 80
hostname: ${DOMAIN_NAME}
and now it works.
Currently I am running Spring Boot Services with Discovery First.
Starting the services one after another works just fine, but I am having issues starting the services all at once via docker-compose.
The discovery first bootstrap seems to not work as I expected. I would assume that the service would halt / retry till he receives the configuration from the configuration service, which has been discovered via eureka. But currently it will try to resolve the configuration once and then the service will start immediately after - no matter if a configuration has been brought or not.
The service bootstrap looks as following:
spring:
application:
name: my-service
cloud:
config:
fail-fast: false
discovery:
enabled: true
service-id: configserver
retry:
initialInterval: 2000
multiplier: 1.5
maxInterval: 60000
maxAttempts: 10
server:
port: ${APPLICATION_PORT:16000}
eureka:
client:
serviceUrl:
defaultZone: http://${EUREKA_HOSTNAME:localhost}:15000/eureka/
The eureka config looks as following:
spring:
application:
name: manager
server:
port: 15000
eureka:
client:
registerWithEureka: true
fetchRegistry: false
serviceUrl:
defaultZone: http://${EUREKA_HOSTNAME:localhost}:15000/eureka/
Last but not least - the configserver config looks as following:
spring:
application:
name: configserver
cloud:
config:
fail-fast: true
server:
port: 15001
eureka:
client:
serviceUrl:
defaultZone: http://${EUREKA_HOSTNAME:localhost}:15000/eureka/
Anyone able to give any advice here?
I found the issue:
Actually I found two issues on our side. The first one is not having a suitable lease renewal interval. The client just retried receiving the configuration with the cached services.
Since the cached services did not contain a configuration service, he wasn't able to receive a proper configuration.
The second problem was to not wait for enough time. This has been fixed in raising the maxAttempts higher. So the config and discovery services do have enough time to come up.
The client Configuration looks like the following now:
spring:
application:
name: service
cloud:
config:
fail-fast: true
discovery:
enabled: true
service-id: CONFIGSERVER
retry:
initialInterval: 2000
multiplier: 1.5
maxInterval: 60000
maxAttempts: 100
server:
port: ${APPLICATION_PORT:16000}
eureka:
instance:
lease-renewal-interval-in-seconds: 10
client:
fetch-registry: true
serviceUrl:
defaultZone: http://${EUREKA_HOSTNAME:localhost}:15000/eureka/
I'm trying to use docker-compose to run 2 simple services locally (Ubuntu): a eureka server, and config server (which is also a eureka client). Both of these have simple dockerfiles that run java -jar, expose their ports, and individually work fine. I also tried to add eureka.client.service-url.defaultZone=http://company-service-discovery:8761/eureka to see if it would register itself, and it worked.
My config server cannot successfully register to the eureka server, and I've googled it and nothing I've seen helped me solve this.
According to docker-compose documentation at https://docs.docker.com/compose/networking/ :
By default Compose sets up a single network for your app. Each container for a service joins the default network and is both reachable by other containers on that network, and discoverable by them at a hostname identical to the container name.
With the following example web should be able to use postgres://db:5432 to communicate with the database.
version: "3"
services:
web:
build: .
ports:
- "8000:8000"
db:
image: postgres
ports:
- "8001:5432"
I have used the same method to configure my services, but my config server gets connection refused when trying to register:
docker-compose.yml
version: '3.3'
services:
company-service-discovery:
build: company-service-discovery/
ports:
- "8761:8761"
company-config-server:
build: company-config-server/
ports:
- "8888:8888"
links:
- company-service-discovery
config server bootstrap.yml
server:
port: 8888
management:
security:
enabled: false
spring:
application:
name: company-config-server
cloud:
config:
server:
native:
search-locations: classpath:/shared
profiles:
include: native
eureka:
client:
service-url:
defaultZone: http://company-service-discovery:8761/eureka
eureka server bootstrap.yml
spring:
application:
name: company-service-discovery
server:
port: 8761
management:
security:
enabled: false
exception
2017-07-26 14:25:05.738 WARN 1 --- [nfoReplicator-0] c.n.d.s.t.d.RetryableEurekaHttpClient : Request execution failed with message: java.net.ConnectException: Connection refused (Connection refused)
2017-07-26 14:25:05.739 WARN 1 --- [nfoReplicator-0] com.netflix.discovery.DiscoveryClient : DiscoveryClient_COMPANY-CONFIG-SERVER/365d20275ab0:company-config-server:8888 - registration failed Cannot execute request on any known server
question
Is there anything wrong with my configuration ? How can I make it work ?
Let me know if there's any info missing, I'll gladly give any info I can.
Add the defaultZone configuration to the Eureka server's properties too (and change the service-url to serviceUrl in your config server bootstrap.yml).
eureka server bootstrap.yml
spring:
application:
name: company-service-discovery
eureka:
client:
serviceUrl:
defaultZone: http://company-service-discovery:8761/eureka
server:
port: 8761
management:
security:
enabled: false
Thank you sooo much. this post resolved my problem - connecting eureka client to eureka server with docker. After 2 days of searching it work. I am in tears.
So basically -
you should use following in eureka client's application.properties/.yml
eureka.client.service-url.defaultZone=http://eureka-server:8761/eureka
and in docker-compose.yml your eureka service name should match with - the url host name, in my case it is - eureka-server
According to this blog https://spring.io/blog/2015/07/14/microservices-with-spring which is based on eureka service discovery and where the service discovery is working properly.
But when have switched to use Consul instead Eureka the service discovery is not working and getting this error:
java.lang.IllegalStateException: No instances available for ACCOUNTS-SERVICE
at org.springframework.cloud.netflix.ribbon.RibbonLoadBalancerClient.execute(RibbonLoadBalancerClient.java:79)
at org.springframework.cloud.client.loadbalancer.LoadBalancerInterceptor.intercept(LoadBalancerInterceptor.java:46) ...
UPDATED: After have fixed the previous error by providing the correct health-check endpoint (see the answer below), when deploying the services to Cloud Foundry with properly provided host and port of the Consul server in bootstrap.yml (Consul based PropertySource loaded during the 'bootstrap' phase):
---
spring:
profiles: cloud
cloud:
consul:
host: <consul host or ip>
port: 8500
Consul is registering the service, but with critical state (failing)!
Would appreciate any help or guidance.
Thanks
The issue was related to the Consul health check default path which is set to the /health endpoint.
Thus after have enabled the spring-actuator in all clients applications (web-server and micro-service) this issue was resolved.
Or you may change the default Consul health-check endpoint in the bootstrap.yml file:
cloud:
consul:
discovery:
healthCheckPath: /test
NB. To enable spring-actuator in maven the following dependency was added to the pom.xml file:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
For more information see: http://cloud.spring.io/spring-cloud-consul/spring-cloud-consul.html
When deploying (pushing) to CF (CloudFoundry) the URI of the deployed application should be provided to Consul for service discovery process (CF provides application's URIs in vcap.application.uris environment variable), thus the following configuration should be added to the bootsrap.yml file:
---
spring:
profiles: cloud
cloud:
consul:
host: <consul host or ip>
port: 8500
discovery:
instanceId: ${spring.application.name}:${vcap.application.application_name}:${vcap.application.instance_id}
hostname: ${vcap.application.uris[0]}
port: 80
NB. instanceId is used by Consul to register the application
(microservice) instance.