I tried to get BlockHound working with Ktor like this.
BlockHound.install(CoroutinesBlockHoundIntegration())
But it doesn't detect blocking calls inside routing endpoints. Has anyone had any luck with this? I guess the problem is that Blockhound doesn't recognize Ktor threads as non-blocking threads, but I can't for the life of me figure out how to configure Blockhound without causing other problems.
Related
The following makes the difference of Spring Framework with Play framework. Play framework has the following
Hot reload
Non Blocking I/O
Is it possible to achieve the two items in Spring ?
Tomcat is a blocking I/O and not sure whether any new versions planned with non blocking i/o
Thanks
I am working on the micro-service developed using Spring Boot . I have implemented following layers:
Controller layer: Invoked when user sends API request
Service layer: Processes the request. Either sends request to third-part service or sends request to database
Repository layer: Used to interact with the
database
.
Methods in all of above layers returns the CompletableFuture. I have following questions related to this setup:
Is it good practice to return Completable future from all methods across all layers?
Is it always recommended to use #Async annotation when using CompletableFuture? what happens when I use default fork-join pool to process the requests?
How can I configure the threads for above methods? Will it be a good idea to configure the thread pool per layer? what are other configurations I can consider here?
Which metrics I should focus while optimizing performance for this micro-service?
If the work your application is doing can be done on the request thread without too much latency, I would recommend it. You can always move to an async model if you find that your web server is running out of worker threads.
The #Async annotation is basically helping with scheduling. If you can, use it - it can keep the code free of the references to the thread pool on which the work will be scheduled. As for what thread actually does your async work, that's really up to you. If you can, use your own pool. That will make sure you can add instrumentation and expose configuration options that you may need once your service is running.
Technically you will have two pools in play. One that Spring will use to consume the result of your future, and another that you will use to do the async work. If I recall correctly, Spring Boot will configure its pool if you don't already have one, and will log a warning if you didn't explicitly configure one. As for your worker threads, start simple. Consider using Spring's ThreadPoolTaskExecutor.
Regarding which metrics to monitor, start first by choosing how you will monitor. Using something like Spring Sleuth coupled with Spring Actuator will give you a lot of information out of the box. There are a lot of services that can collect all the metrics actuator generates into time-based databases that you can then use to analyze performance and get some ideas on what to tweak.
One final recommendation is that Spring's Web Flux is designed from the start to be async. It has a learning curve for sure since reactive code is very different from the usual MVC stuff. However, that framework is also thinking about all the questions you are asking so it might be better suited for your application, specially if you want to make everything async by default.
I have recently changed my Quarkus application from RestEasy to Reactive Routes to implement my HTTP endpoints.
My Quarkus app had OpenTracing enabled and it was working fine. After changing the HTTP resource layer I can not see any trace in Jaeger.
After setting log level in DEBUG I can see my application is registered in Jaeger but I don't see any traceId or spanId in logs neither traces in Jaeger:
15:44:36 DEBUG traceId=, spanId=, sampled= [io.qu.ja.ru.JaegerDeploymentRecorder] (main) Registering tracer to GlobalTracer JaegerTracer(version=Java-0.34.3, serviceName=employee, reporter=RemoteReporter(sender=HttpSender(), closeEnqueueTimeout=1000), sampler=ConstSampler(decision=true, tags={sampler.type=const, sampler.param=true}), tags={hostname=employee-8569585469-tg8wg, jaeger.version=Java-0.34.3, ip=10.244.0.21}, zipkinSharedRpcSpan=false, expandExceptionLogs=false, useTraceId128Bit=false)
15:45:03 INFO traceId=, spanId=, sampled= [or.se.po.re.EmployeeResource] (vert.x-eventloop-thread-0) getEmployees
I'm using the latest version of Quarkus which is 1.9.2.Final.
Is it enabled OpenTracing when I'm using Reactive Routes?
Tracing is enabled by default for JAX-RS endpoints only, not for reactive routes at the moment. You can activate tracing by annotating your route with #org.eclipse.microprofile.opentracing.Traced.
Yes, adding #Traced enable to activate tracing on reactive routes.
Unfortunately, using both JAX-RS reactive and reactive routes bugs the tracing on event-loop threads used by JAX-RS reactive endpoint when they get executed.
I only started Quarkus 2 days ago so i don't really the reason of this behavior (and whether it's normal or it's a bug), but obviously switching between two completely mess up the tracing.
Here is an example to easily reproduce it:
Create a REST Easy reactive endpoint returning an empty Multi
Create a custom reactive route
set up the IO threads to 2 (easier to quickly reproduce it)
Run the application, and request the two endpoints alternatively
Here is a screenshot that show the issue
As you can see, as soon as the JAX-RS resource is it and executed on one of the two threads available, it "corrupts" it, messing the trace_id reported (i don't know if it's the generation or the reporting on logs that is broken) on logs for the next calls of the reactive route.
This does not happen on the JAX-RS resource, as you can notice on the screenshot as well. So it seems to be related to reactive routes only.
Another point here is the fact that JAX-RS Reactive resources are incorrectly reported on Jaeger. (with a mention to a missing root span) Not sure if it's related to the issue but that's also another annoying point.
I'm thinking to completely remove the JAX-RS Reactive endpoint and replace them by normal reactive route to eliminate this bug.
I would appreciate if someone with more experience than me could verify this or tell me what i did wrong :)
EDIT 1: I added a route filter with priority 500 to clear the MDC and the bug is still there, so definitely not coming from MDC.
EDIT 2: I opened a bug report on Quarkus
EDIT 3: It seems related to how both implementations works (thread locals versus context propagation in actor based context)
So, unless JAX-RS reactive resources are marked #Blocking (and get executed in a separated thread pool), JAX-RS reactive and Vertx reactive routes are incompatible when it comes to tracing (but also probably the same for MDC related informations since MDC is also thread related)
I have some microservices, which should work on top of WebFlux framework. Each server has own API with Mono or Flux. We are using MongoDB, which is supported by Spring (Spring Data MongoDb Reactive).
The problem is external blocking API, which I have to use in my system.
I have one solution. I can just wrap blocking API calls in dedicated thread pool and use it with CompletableFuture.
Is there anything else to solve my problem? I think, that brand new Rsocket cannot solve my problem.
1.If possible, you can change your blocking API call to the reactive way using the WebClient class.
References:
Reference guide
WebClient API
A simple, complete sample
2.If the blocking API can't be changed to reactive ones, we should have a dedicated, well-tuned thread pool and isolate the blocking code there.
There is also an example here.
I don't see why you cannot wrap a blocking API call in a Flux or a Mono. You can also integrate Akka with Spring if the actor model seems easier to you.
RSocket should be a perfect fit, good tutorials to get you started
https://www.baeldung.com/spring-boot-rsocket
https://spring.io/blog/2020/04/06/getting-started-with-rsocket-spring-boot-channels
Please some help understanding the following:
I am using CachingConnectionFactory in my app and first used it during my jms tests to test my jms config like guaranteed delivery, rollback/commit, etc..
I am using Spring's JmsTemplate for sending and DefaultMessageListenerContainer during delivery.
I noticed that this is hard/impossible when using several test methods run sequential
Example: in test method A I throw exceptions in the Message listener (consumer side) such that retries occur.
Then test B is run and in method A I do a different test, but when I start this test I still get retry messages from test A, which I clearly not want.
I purge the Queue through jmx between tests, but still receive these retries :(...
I searched and debugged... I don't exactly understand why these retries keep comming up, even when I am sure that the purge occur correctly. Maybe it was already cached somewhere in the session... I don't know. Anybody any idea?
I found out that I needed to use the SingleConnectionFactory during testing. With this connection factory the retries disappear, but I don't really understand why. Why?
I understand that it uses only one connection (from the Spring ref), and noticed that it somehow removes the consumer after every send action, but I don't really understand what happen with these retries :(... Any idea?
(It's hard to debug because of the multi threading behavior and difficult to find good information about it on the web)
Also using CachingConnectionFactory with only one session cache size of 1 didn't solve the retry issue.
Thanks
Best bet would probably to use an embedded broker and start/stop it between each test, make sure deleteAllMessagesOnStartup is set to true and the broker should purge the store fore you, which will ensure you've got a clean slate for each test. You might also benefit from having a look at ActiveMQ's unit tests, it's a good source of examples of how the broker can be used in automated tests.
It's not an easy thing to fix: remove the messages between tests.
I tried many thingssss, like mentioned above: stop/start the broker and the class DefaultMessageListenerContainer of Spring that I use to consume my messages.
It all seem to work until I turned I set the cache level in DefaultMessageListenerContainer to Consumer such that the consumer is cached.
That is required such that the redeliveryPolicy works.
However, this messed up everything and messages where cached by DefaultMessageListenerContainer in some way, as it seemed.
At the end, I solved it by simple consuming all messages after a test (just wait a second and consume all Ok), such that the next test can begin.