I’m having a bit of trouble getting Turbine to work in Spring Cloud. In a nutshell, I can’t determine how to configure it to aggregate circuits from more than one application at a time.
I have 6 separate services, a eureka server, and a turbine server running in standalone mode. I can see from my Eureka server that all of the services are registered, including turbine. My turbine server is up and running, and I can see its /hystrix page without issue. But when I try to use it to examine turbine.stream, I only see the FIRST server that is listed in turbine.appConfig, the rest are ignored.
This is my Turbine server’s application.yml, or at least the relevant parts:
---
eureka:
client:
serviceUrl:
defaultZone: http://localhost:8010/eureka/
server:
port: 8030
info:
component: Turbine
turbine:
clusterNameExpression: new String(“default”)
appConfig: sentence,subject,verb,article,adjective,noun
management:
port: 8990
When I run this and access the hystrix dashboard on my turbine instance, asking for the turbine.stream, the ONLY circuit breakers listed in the output are for the first service listed in appConfig, the “sentence” service in this case. Curiously, if I re-arrange the order of these services and put another one first (like “noun”), I see only the circuits for THAT service. Only the first service in the list is displayed.
I’ll admit to being a little confused on some of the terminology, like streams, clusters, etc., so I could be missing some basic concept here, but my understanding is that Turbine could digest streams from more than one service and aggregate them in a single display. Suggestions would be appreciated.
I don't have enough reputation to comment, so I have to write this in an answer :)
I had the exactly same problem:
There are two services "test-service" and "other-service", each with it's own working hystrix-stream
and there is one Turbine-Application, which is configured like this:
turbine:
clusterNameExpression: new String("default")
appConfig: test-service,other-service
All of my services are running on my local machine.
Result is: My Hystrix-Dashboard just shows the metrics from "test-service".
Reason:
It seems to be, that a Turbine-Client which is configured the described way doesn't handle multiple services when they are running at the same host.
This is explained here:
https://github.com/Netflix/Hystrix/issues/117#issuecomment-14262713
Turbine maintains state of all these instances in order to maintain persistent connections to them and it does rely on the "hostname" and if the host name is the same then it won't instantiate a new connection to that same server (on a different port).
So the main point is, that all of your services must be registered with different hostnames. How you could do this on your local machine is described below.
UPDATE 2015-06-12/2016-01-23: Workaround for local testing
Change your hostfile:
# ...
127.0.0.1 localhost
127.0.0.1 localdomain1
127.0.0.1 localdomain2
# ...
127.0.0.1 localdomainx
And then set the hostname for your clients each to a different domain-entry like this:
application.yml:
eureka:
instance:
hostname: localdomainx
Related
(It is basically two Eureka server and three Eureka client microservices)
I want to remove the following message:
EMERGENCY! EUREKA MAY BE INCORRECTLY CLAIMING INSTANCES ARE UP WHEN THEY'RE NOT. RENEWALS ARE LESSER THAN THRESHOLD AND HENCE THE INSTANCES ARE NOT BEING EXPIRED JUST TO BE SAFE.
error image
Eureka Server1:
spring.application.name=ms-service-discovery-1
server.port=8761
eureka.client.register-with-eureka=false
eureka.client.fetch-registry=false
eureka.server.enableSelfPreservation= true
eureka.instance.leaseRenewalIntervalInSeconds=1
eviction-interval-timer-in-ms: 1000
eureka.server.wait-time-in-ms-when-sync-empty: 1000
eureka.server.responseCacheUpdateIntervalMs: 1000
Eureka Server2:
spring.application.name=ms-service-discovery-2
server.port=8761
eureka.client.register-with-eureka=false
eureka.client.fetch-registry=false
eureka.server.enableSelfPreservation= true
eureka.instance.leaseRenewalIntervalInSeconds=1
eviction-interval-timer-in-ms: 1000
eureka.server.wait-time-in-ms-when-sync-empty: 1000
It's due to Eureka's self-preservation mode. Eureka servers will stop the eviction of all instances if the number of heartbeats renewals is below the expected threshold. The warning in your Eureka servers shows this situation happened.
Please try to adjust the below property. 85% is the default value. First, just try to lower the below property like 0.50
eureka.renewalPercentThreshold=0.85
Alternatively, you can disable self-preservation mode with the below property.
eureka.enableSelfPreservation=false
Note
Basically, all Eureka client unregister itself when it shuts down. If it unregisters itself successfully, the above problem will not happen.
Unfortunately, sometimes the Spring Cloud based Eureka client doesn't unregister itself when it shuts down and the symptom is different based on the Spring Cloud version. Dalston and Edgware release unregister itself well in most cases. But Finchley releases seem not to unregister itself now.
Also, you start to run MANY instances in your Eureka environment, the above message will disappear because just one or two instances shutdown will not reach the threshold.
You can find more information about self-preservation mode here
I have a development environment in windows where I can access the spring build in both IP Address and Hostname(PC-Name) for my Eureka Config Server / Client. When I move it to the RedHat environment it does not recognize the URL if it is a hostname.
My main goal is to change the eureka client status page to point to the hysterix monitor for the eureka client's hysterix stream. The value of ${spring.cloud.client.hostname} resolves to the hostname. I was wondering what is the way to make it the current IP of the eureka client?
To be exact here is an example of what I want am trying to do.
eureka:
instance:
preferIPAddress: true
statusPageUrlPath: http://${spring.cloud.client.hostname}:${eureka.cloud.config.port}/hystrix/monitor?stream=http%3A%2F%2F${spring.cloud.client.hostname}%3A${server.port}%2Factuator%2Fhystrix.stream
It just so happen that the client and the server are both in the same machine so I am contented in using the client hostname for both the Eureka Config Server path and the Eureka Client hystrix stream.
Note that I already set the preferIPAddress to true but the generated hostname is still the value of "/etc/hostname". I saw some solution that explicitly specifies the IP Address in the Eureka Client Instance. But I prefer to make it dynamic so that the same code can run smoothly on either Development and Deploy environment.
What can I do so that the hostname can also be recognized the same as the ip address?
The answer has dawn upon me just now. I just changed the following to this.
${spring.cloud.client.hostname}
↓↓↓↓↓↓↓↓↓↓↓↓
${spring.cloud.client.ip-address}
I was thrown to a wrong conclusion because other sites would tell me to use this configuration ${spring.cloud.client.ipAddress} which does not work.
Probably there was a change in Finchley / Spring boot 2.0 version. If anyone can give me a link to a documentation or discussion describing the configuration change, it would be helpful.
My microservice with #EnableTurbine config:
turbine:
clusterNameExpression: new String('default')
appConfig: bestallning
bestallning is a #SpringCloudApplication, hystrix.stream is enabled. It registers in eureka and turbine app can find it. But it has management.port set to 8092 and server.port 8082. hystrix.stream binds to management.port
Turbine now tries to fetch hystrix.stream from server.port of bestallning, not management.port that hystrix.stream is bound to.
Fetching instance list for apps: [bestallning]
Fetching instances for app: bestallning
Received instance list for app: bestallning, size=1
Retrieved hosts from InstanceDiscovery: 1
Found hosts that have been previously terminated: 0
Hosts up:1, hosts down: 0
Url for host: http://143.237.21.196:8082/hystrix.stream default
Could not initiate connection to host, giving up: [{"timestamp":1460035761979,"status":404,"error":"Not Found","message":"No message available","path":"/hystrix.stream"}]
Stopping InstanceMonitor for: 143.237.21.196 default
Is it possible to have turbine look for hystrix.stream using the correct port?
I think you'd have to write your own InstanceDiscovery (and create a #Bean of that type). Might be a useful feature in the existing implementations though, so please open an issue in Spring Cloud Netflix.
We've three Spring Boot applications:
Eureka Service
Config Server
Simple Web Service making use of Eureka and Config Server
I've set up the services so that we use a Eureka First Discovery, i.e. the simple web application finds out about the config server from the eureka service.
When started separately (either locally or by starting them as individual docker images) everything is ok, i.e. start config server after discovery service is running, and the Simple web service is started once the config server is running.
When docker-compose is used to start the services, they obviously start at the same time and essentially race to get up and running. This isn't an issue as we've added failFast: true and retry values to the simple web service and also have the docker container restarting so that the simple web service will eventually restart at a time when the discovery service and config server are both running but this doesn't feel optimal.
The unexpected behaviour we noticed was the following:
The simple web service reattempts a number of times to connect to the discovery service. This is sensible and expected
At the same time the simple web service attempts to contact the config server. Because it cannot contact the discovery service, it retries to connect to a config server on localhost, e.g. logs show retries going to http://localhost:8888. This wasn't expected.
The simple web service will eventually successfully connect to the discovery service but the logs show it stills tries to establish communication to the config server by going to http://localhost:8888. Again, this wasn't ideal.
Three questions/observations:
Is it a sensible strategy for the config client to fall back to trying localhost:8888 when it has been configured to use discovery to find the config server?
When the eureka connections is established, should the retry mechanism not now switch to trying the config server endpoint as indicated by Eureka? Essentially putting in higher/longer retry intervals and periods for the config server connection is pointless in this case as it's never going to connect to it if it's looking at localhost so we're better just failing fast.
Are there any properties that can override this behaviour?
I've created a sample github repo that demonstrates this behaviour:
https://github.com/KramKroc/eurekafirstdiscovery/tree/master
I am trying to get a view on 2 different services that are running using Spring Boot. Both services are nicely registered in Eureka and now i want to get some Hystrix data plotted in the Turbine webapplication.
When testing out the single services using Turbine it works great:
http://localhost:2222/hystrix.stream (service1)
http://localhost:3333/hystrix.stream (service2)
I am trying to get a view on the dashboard which plots both of the services but whatever i try only the first service in the Turbine config is plotted.
turbine:
clusterNameExpression: new String('default')
appConfig: service1,service2
I seem to hit a wall when trying to get the data into the turbine.stream for both of the services.
Any hints or tips?
To answer my own question, it seems that Turbine cannot monitor more services on the same host as it is bound to the hostname. See https://github.com/Netflix/Turbine/issues/88
Turbine has 2 operating models:
pull; data from separate service streams are pulled by Turbine
push; data from separate services are pushed to a Turbine server by using an AMQP broker such as RabbitMQ
A workaround is to use TurbineAMQP that is capable of doing exact the stuff that i needed. It involves a bit more infrastructure but works nicely!
First off, I don't think Turbine is host-exclusive as Marco saying
... it seems that Turbine cannot monitor more services on the same host as it is bound to the hostname
I implemented Turbine with the Hystrix Dashboard App (which is not recommended), aggregating 2 services with same spring.application.name.
I found how Turbine collects hystrix stream is that, each Turbine application will look for services in Eureka Registry with the given name by turbine.appConfig in application.properties, which is the spring.application.name by default or eureka.instance.appname explicitly of your services, regardless of the hosts of them. Till now turbine stream is ready.
A list of names as service IDs from your request in the previous step will be returned, and at least one of them should match the list of names you provide in turbine.aggregator.clusterConfig.
When you type http://[turbine-app-host:port]/turbine.stream?cluster=[cluster name] on the home page of Hystrix Dashboard, the cluster name parameter should be one of the matching names from turbine.aggregator.clusterConfig.