I am using Zuul as an api-gateway in a spring-cloud micro-service app, so that every access to api-gateway/some-service/a_route is redirected to /a_route in a generic way (the discovery is backed by consul).
I am trying to use Jaeger to instrument this system. And at this point I am using opentracing-spring-web-autoconfigure, because I cannot upgrade my spring boot/cloud version easily (I am using1.4.5.RELEASE Camden.SR7). So I just added the dependency, created the Jaeger tracer and redirect it to the docker all in one collector.
I have begin by instrumenting the gateway and It somewhat works => It generate span on the gateway, but all the route are marked :
apigateway-service: GET
and there is no information concerning the forwarded route at this level, the full route itself is store in a tag : http.url
"http://localhost:8080/collection-service/collections/projects/"
To be useful I would prefer to have span named :
apigateway-service: GET collection-service/collections/projects/
Can this be configured somewhere ?
No, it cannot, but it wouldn't hurt to open an issue there with this suggestion.
Related
Our spring cloud gateway needs to support services located in non-root URLs. For example:
https://thehost/api/theservice
I can configure the service using SimpleDiscoveryClient and SimpleReactiveDiscoveryClient or I can create my own implementation of the discovery clients. However, no matter what I do I'm still limited by this interface:
ServiceInstance
This interface has a nice method getUri(), which default implementation implements poorly and loses everything except host, port and scheme. I can provide my own implementation, but unfortunately this method:
LoadBalancerUriTools.doReconstructURI()
which the ReactiveLoadBalancerClientFilter relies on to reconstruct URI, ignores all the complexity of the original URI and in my example results in this:
https://thehost:443/theresource
Note that it hardcodes the port for https URI as well, which is not good either.
I'd like this functionality to respect the original configuration and construct the following instead:
https://thehost/api/theservice/theresource
Is there way to achieve a proper service URL handling? Maybe there's a way to customize the behavior somehow?
I have recently changed my Quarkus application from RestEasy to Reactive Routes to implement my HTTP endpoints.
My Quarkus app had OpenTracing enabled and it was working fine. After changing the HTTP resource layer I can not see any trace in Jaeger.
After setting log level in DEBUG I can see my application is registered in Jaeger but I don't see any traceId or spanId in logs neither traces in Jaeger:
15:44:36 DEBUG traceId=, spanId=, sampled= [io.qu.ja.ru.JaegerDeploymentRecorder] (main) Registering tracer to GlobalTracer JaegerTracer(version=Java-0.34.3, serviceName=employee, reporter=RemoteReporter(sender=HttpSender(), closeEnqueueTimeout=1000), sampler=ConstSampler(decision=true, tags={sampler.type=const, sampler.param=true}), tags={hostname=employee-8569585469-tg8wg, jaeger.version=Java-0.34.3, ip=10.244.0.21}, zipkinSharedRpcSpan=false, expandExceptionLogs=false, useTraceId128Bit=false)
15:45:03 INFO traceId=, spanId=, sampled= [or.se.po.re.EmployeeResource] (vert.x-eventloop-thread-0) getEmployees
I'm using the latest version of Quarkus which is 1.9.2.Final.
Is it enabled OpenTracing when I'm using Reactive Routes?
Tracing is enabled by default for JAX-RS endpoints only, not for reactive routes at the moment. You can activate tracing by annotating your route with #org.eclipse.microprofile.opentracing.Traced.
Yes, adding #Traced enable to activate tracing on reactive routes.
Unfortunately, using both JAX-RS reactive and reactive routes bugs the tracing on event-loop threads used by JAX-RS reactive endpoint when they get executed.
I only started Quarkus 2 days ago so i don't really the reason of this behavior (and whether it's normal or it's a bug), but obviously switching between two completely mess up the tracing.
Here is an example to easily reproduce it:
Create a REST Easy reactive endpoint returning an empty Multi
Create a custom reactive route
set up the IO threads to 2 (easier to quickly reproduce it)
Run the application, and request the two endpoints alternatively
Here is a screenshot that show the issue
As you can see, as soon as the JAX-RS resource is it and executed on one of the two threads available, it "corrupts" it, messing the trace_id reported (i don't know if it's the generation or the reporting on logs that is broken) on logs for the next calls of the reactive route.
This does not happen on the JAX-RS resource, as you can notice on the screenshot as well. So it seems to be related to reactive routes only.
Another point here is the fact that JAX-RS Reactive resources are incorrectly reported on Jaeger. (with a mention to a missing root span) Not sure if it's related to the issue but that's also another annoying point.
I'm thinking to completely remove the JAX-RS Reactive endpoint and replace them by normal reactive route to eliminate this bug.
I would appreciate if someone with more experience than me could verify this or tell me what i did wrong :)
EDIT 1: I added a route filter with priority 500 to clear the MDC and the bug is still there, so definitely not coming from MDC.
EDIT 2: I opened a bug report on Quarkus
EDIT 3: It seems related to how both implementations works (thread locals versus context propagation in actor based context)
So, unless JAX-RS reactive resources are marked #Blocking (and get executed in a separated thread pool), JAX-RS reactive and Vertx reactive routes are incompatible when it comes to tracing (but also probably the same for MDC related informations since MDC is also thread related)
I just created simple Zuul Proxy at the front end for our microservices environment but now I wanted to log all the entries into the log file which went through the proxy.
Do any properly which I need to enable to do this.
I assume an implementation of zuul as a regular spring boot driven microservice with a bunch of netflix's beans running under the hood.
In this case it can run on tomcat (probably for other services the idea is the same, but the technical implementation might be different).
So for tomcat:
As a first resort you can take advantage of tomcat feature of "access logs" that logs all the requests anyway. It also allows some level of customizations (what to log). The technical difficulty is that tomcat access log is not by default managed by logback, so you'll have to use some kind of adapter.
Here you can find ideas of how to resolve this technically and integrate access log with logback.
An another approach would be creating a Filter that will extract required pieces and log the request / response / whatever you want to log
Here is an example of creating a custom filter like this.
Of course I you also need to log something from response you should configure the filter type (see the java code example in the link)
One tip / caution: think about performance implications, so that this feature won't slow down the processing if the server is under high load of requests.
We are in the process of standing up a new microservices architecture with Zuul at the front-end and a bunch of tomcat enabled microservices at the backend. Each service as it starts up, will register itself with Eureka and any client that wants to call those service will do so through Zuul. We've got this all wired in and everything is working fine.
However, I have a couple questions as to how we can make this architecture much more dynamic.
One thing that we assumed was there out of the box with Ribbon/Eureka, but have yet to find a solution for is that as we add more services to the backend, that somehow (via Archiaus and update to Zuul's eureka-client.properties file) Zuul's Ribbon client would update itself with the new service details (e.g. vipaddress, load balancing algorithm, etc). So far, the only thing that works is to update the properties file and restart Zuul (ughhh).
For example, let's say today we have 2 microservices at the backend, therefore, Zuul's eureka/ribbon client configuration would include the below:
ribbon.client.niws.clientlist=service1|service2
zuul.ribbon.namespace=zuul.client
service1.zuul.client.DeploymentContextBasedVipAddresses=myService1
service1.zuul.client.NIWSServerListClassName=com.netflix.niws.loadbalancer.DiscoveryEnabledNIWSServerList
service2.zuul.client.DeploymentContextBasedVipAddresses=myService2
service2.zuul.client.NIWSServerListClassName=com.netflix.niws.loadbalancer.DiscoveryEnabledNIWSServerList
Now tomorrow, let's assume we need to add service3. What we have observed is that if we add those details to the same configuration (see below), they only become available to Zuul after a restart. Is there some other configuration parameter we are missing that would allow us to dynamically introduce the new service details or do we have to roll our own Eureka/Ribbon client to do this?
ribbon.client.niws.clientlist=service1|service2|service3
zuul.ribbon.namespace=zuul.client
service1.zuul.client.DeploymentContextBasedVipAddresses=myService1
service1.zuul.client.NIWSServerListClassName=com.netflix.niws.loadbalancer.DiscoveryEnabledNIWSServerList
service2.zuul.client.DeploymentContextBasedVipAddresses=myService2
service2.zuul.client.NIWSServerListClassName=com.netflix.niws.loadbalancer.DiscoveryEnabledNIWSServerList
service3.zuul.client.DeploymentContextBasedVipAddresses=myService3
service3.zuul.client.NIWSServerListClassName=com.netflix.niws.loadbalancer.DiscoveryEnabledNIWSServerList
My other question is related and that is do we really need to add a client configuration (in eureka-client.properties) for every service that Zuul could possibly route to? At some point, we may have 100's of services running and trying to maintain all the related client configurations in Zuul seems a bit clumsy. Is there a way to globally configure Zuul to load all services into its client list from Eureka (or based on some service metadata in Eureka) and dynamically update this list as new services register themselves with Eureka?
Thanks!
The issue is with namespaces.If we use the default namespace it should be able to pick up the new properties addedd by default.
I setup an Eureka cluster composed of 3 replicas.
I got the nice dashboard which is automatically populated with the instances currently registered with Eureka and the DS Replicas.
However the DS Replicas links seems to point to the value I set as eureka.client.serviceUrl.defaultZone. In my case this value is something similar to http://node-01:8761/eureka/ which in reality returns a 404. Is there a way I could configure the dashboard to strip out the /eureka/ part so when I follow the links I end up in the other dashboards or am I misunderstanding the use of those links?
There is no easy way to alter the link as you ask.
However, the UI part of the Eureka server is a standard Spring MVC #Controller. An instance of it is created by org.springframework.cloud.netflix.eureka.server.EurekaServerConfiguration. Have a look at the current implementation... it shouldn't be too hard to provide your own customized version instead.