I am upgrading an application with Kotlin, Webflux to Spring Boot 3.
Now I noticed that our logs are lacking traceIds.
My suspicion is that this is due to coroutines, since I observed that the logs of the controller contain a traceId and in the logs of the service it is not included. The controller method as well as the service method both have the suspend keyword.
#PostMapping("/upload-file")
#Observed
suspend fun uploadFile(
serverWebExchange: ServerWebExchange,
): ResponseEntity<MessageResponse> {
logger.debug("uploadFile")
importService.handle(getMultipartDataPart("file", serverWebExchange))
return ResponseEntity(MessageResponse("Success."), OK)
}
#Service
class ImportService() {
suspend fun handle(fileAsByteArray: ByteArray) {
log.debug("Import file.")
...
}
I followed the documentation and added depdencies
implementation("org.springframework.boot:spring-boot-starter-actuator")
implementation("io.micrometer:micrometer-tracing-bridge-brave")
Bean
#Bean
fun observedAspect(observationRegistry: ObservationRegistry?): ObservedAspect? {
return ObservedAspect(observationRegistry)
}
and Annotation on the controller method
#GetMapping("/keys/{key}")
#Observed
suspend fun isKeyInKeyStore(
Another interesting observation I made is that I don't observe the problem on my local machine, but only on our dev K8s cluster. Could a different JVM (OpenJDK vs AWS Corretto) cause this?
I am quite sure that the context is lost, but not sure how to deal with it in the best way since I am pretty new to coroutines and Webflux.
Related
In intellij idea evaluating (Cmd + F8 or Option + click) suspended functions while debugging works just fine.
The problem arises in combination with webflux and suspended functions.
Once suspended you cannot evaluate the captured variables any longer.
Error states: this#HelloController is not captured
One workaround is to create a coroutineScope around an endpoint.
Not sure why this is working at all and have not tested any performance impacts on this.
Searching for a proper way to be able to debug suspended endpoints in intellij with webflux and coroutines, without having to add this coroutineScope and an explanation why it's not working otherwise.
#RestController
class HelloController(
val helloService: HelloService,
) {
#GetMapping("hello")
suspend fun hello() =
helloService.hello() // Throws error when evaluated
#GetMapping("helloCoroutine")
suspend fun helloCoroutine() = coroutineScope {
helloService.hello() // Works just fine when evaluated
}
}
#Service
class HelloService {
suspend fun hello(): String {
delay(10)
return "hello there!"
}
}
Complete code can be found here: https://github.com/Ch4s3r/webflux_coroutines_test
A simple #RestController is connected with a #MessagingGateway to an IntegrationFlow.
After a load test we saw within the tracing that we lose "a lot of time" before even starting the processing within the flow:
Tracing result
In this example we can see that over 90ms spend befor sending the message to the flow.
Did anyone have some idea what leads to this behavior?
As far as I understood the documentation, everything is handled in the sender thread and therefore no special worker threads are created.
We use the Restcontroller since we need to create the documentation with springdoc-openapi-ui
ExampleCode:
RestController
#RestController
public class DescriptionEndpoint {
HttpMessageGateway httpMessageGateway;
public Result findData(#Valid dataRequest dataRequest) {
final Map<String, Object> headerParams = new HashMap<>();
return httpMessageGateway.basicDataDescriptionFlow(dataRequest, headerParams);
}
}
Gateway
#MessagingGateway
public interface HttpMessageGateway {
#Gateway(requestChannel = "startDataFlow.input")
Result basicDataDescriptionFlow(#Payload dataRequest prDataRequest, #Headers Map<String, Object> map);
}
IntegrationFlow
public class ExampleFlow {
#Bean
public IntegrationFlow startDataFlow() {
return new FlowExtension()
.handle(someHandler1)
.handle(someHandler2)
.handle(someHandler3)
.get();
}
}
After adding some more traces I realized, that this timing issue is caused by my spring security configuration.
Unfortunatelly, i thought, the span is only representing the time after the start of findData(..). But it seems, the tracing starts already in the proxy methods and security chain.
After improving some implementation on our JWTToken filter, the spend times for these endpoints are OK.
Apache Camel has a number of features which make event processing elegant and easy to code. It would be useful to be able to exploit this in an AWS Lambda.
Of course not all features are appropriate, especially anything requiring a long lived process.
Also managing persistant state, for example idempotent repositories and throttling would need thinkng about.
But it would be really useful in simple cases.
It turns out that this is simple using Redhat's Quarkus framework.
I've made a simple example: https://github.com/jcable/SampleCamelLambda
The Camel Route is trivial:
from("direct:input").to("log:input")
.process(new Processor() {
public void process(Exchange exchange) throws Exception {
InputObject input = exchange.getIn().getBody(InputObject.class);
String result = input.getGreeting() + " " + input.getName();
OutputObject out = new OutputObject();
out.setResult(result);
out.setRequestId("aws-request-1");
exchange.getIn().setBody(out);
}
});
Adapting the route to the Lambda makes use of a Quarkus RequestHandler.
public class Lambda implements RequestHandler<InputObject, OutputObject> {
#Inject
CamelContext camelContext;
#Override
public OutputObject handleRequest(InputObject input, Context context) {
return camelContext.createProducerTemplate().requestBody("direct:input", input, OutputObject.class);
}
}
CDI is used to inject the CamelContext into the request handler and then the camelContext object is used to create a
ProducerTemplate which can be used to invoke the Camel route.
The Maven project for the example is derived from the Quarkus lambda example with Apache Camel dependencies from the Camel Quarkus examples.
I have a class which builds multiple RestTemplates using RestTemplateBuilder:
private RestTemplate build(RestTemplateBuilder restTemplateBuilder) {
return restTemplateBuilder
.rootUri("http://localhost:8080/rest")
.build();
}
For my test setup I use #AutoConfigureMockRestServiceServer and mock responses using MockServerRestTemplateCustomizer:
mockServerRestTemplateCustomizer.getServer()
.expect(ExpectedCount.times(2),
requestToUriTemplate("/some/path/{withParameters}", "withParameters"))
.andRespond(withSuccess());
My test passes when I uncomment the spring-boot-actuator dependency in my pom and fails in the other scenario with the following message.
Expected: /some/path/parameter
Actual: http://localhost:8080/rest/pos/some/path/withParameters
I noticed by debugging through MockServerRestTemplateCustomizer that spring-boot-actuator applies a "DelegateHttpClientInterceptor" for supporting their built in metrics for rest templates. However this creates a problem with the following code which I found in RootUriRequestExpectationManager:
public static RequestExpectationManager forRestTemplate(RestTemplate restTemplate,
RequestExpectationManager expectationManager) {
Assert.notNull(restTemplate, "RestTemplate must not be null");
UriTemplateHandler templateHandler = restTemplate.getUriTemplateHandler();
if (templateHandler instanceof RootUriTemplateHandler) {
return new RootUriRequestExpectationManager(((RootUriTemplateHandler) templateHandler).getRootUri(),
expectationManager);
}
return expectationManager;
}
Because as mentioned above spring-boot-actuator registers a "DelegateHttpClientInterceptor" which leads to the above code not recognizing the RootUriTemplateHandler and therefore not matching the request using requestToUriTemplate.
What am I missing here to get this working?
As Andy Wilkinson pointed out, this seems to be a bug in Spring boot. I created an issue with a sample project.
I am using Spring Boot with Undertow and trying to implement some limits on the number of requests Undertow will accept so as not to become overloaded under stress.
I've seen the answer to the question at Spring Boot Undertow add both blocking handler and NIO handler in the same application, and it appears promising, but I'm not clear what HttpHandler should be passed as the argument to the RequestLimitingHandler constructor.
Is there an easy way to add a RequestLimitingHandler to the UndertowEmbeddedServletContainerFactory bean, perhaps using the addDeploymentInfoCustomizers method?
Alternatively, if I look deeper and get into the Xnio code on which Undertow is based, it looks like there is an option to set Options.WORKER_TASK_LIMIT, but upon further investigation, it looks like the XnioWorker class ignores this setting after the 3.0.10.GA release and simply sets taskQueue to an unbounded LinkedBlockingQueue. Am I mistaken and could this also be an option?
Answering my own question in case it helps others in the future. Solution is to create a new Undertow HandlerWrapper and instantiate the new RequestLimitingHandler object within the wrap() method, like so:
#Bean
public UndertowEmbeddedServletContainerFactory embeddedServletContainerFactory(RootHandler rootHandler) {
UndertowEmbeddedServletContainerFactory factory = new UndertowEmbeddedServletContainerFactory();
factory.addDeploymentInfoCustomizers(deploymentInfo -> deploymentInfo.addInitialHandlerChainWrapper(new HandlerWrapper() {
#Override
public HttpHandler wrap(HttpHandler handler) {
return new RequestLimitingHandler(maxConcurrentRequests, queueSize, handler);
}
}));
return factory;
}