About Spring WebClient on external onTerminate event - spring-boot

Question

I'm running a spring-boot v2.0.3 tomcat-embedded webserver 8.5.31, to Serve Spring Webflux REST services.
One of those REST services calls to another, external REST Webservice.
public Mono<ServerResponse> select(ServerRequest request) {
return request.principal().cast(Authentication.class)
.flatMap(principal ->
client.get().uri(f -> buildUri(request, principal, request.queryParams(), f))
.exchange())
.flatMap((ClientResponse mapper) ->
ServerResponse.status(mapper.statusCode())
.headers(c -> mapper.headers().asHttpHeaders().forEach(c::put))
.body(mapper.bodyToFlux(DataBuffer.class)
.delayElements(Duration.ofSeconds(10))
.doOnCancel(() -> log.error("Cancelled client"))
.doOnTerminate(() -> log.error("Terminated client")), DataBuffer.class))
.doOnTerminate(() -> log.error("Termination called"));
}
If a browser calls my REST-Service, and after a short while cancels the connection, I can see the outer "Termination called" event, and that the client was terminated also. But the client termination seems to trigger an error in tomcat:
2018-07-25 12:50:42.860 DEBUG 12084 --- [ elastic-3] org.example.search.security.UserManager : Authorizing org.springframework.security.web.authentication.preauth.PreAuthenticatedAuthenticationToken#809aec11: Principal: cn=dv dbsearch client, ou=dbsearch, o=example, l=eb, st=unknown, c=de; Credentials: [PROTECTED]; Authenticated: false; Details: null; Not granted any authorities
2018-07-25 12:50:42.864 DEBUG 12084 --- [ elastic-3] org.example.search.security.UserManager : Successfully authorized: org.springframework.security.web.authentication.preauth.PreAuthenticatedAuthenticationToken#c03925ec: Principal: org.springframework.security.core.userdetails.User#809aec0e: Username: cn=dv dbsearch client, ou=dbsearch, o=example, l=eb, st=unknown, c=de; Password: [PROTECTED]; Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities: ROLE_ADMIN; Credentials: [PROTECTED]; Authenticated: true; Details: null; Granted Authorities: ROLE_ADMIN
2018-07-25 12:50:45.470 ERROR 12084 --- [ctor-http-nio-4] c.d.s.s.h.SolrSelectRequestHandler : Termination called
2018-07-25 12:51:15.562 ERROR 12084 --- [ parallel-3] c.d.s.s.h.SolrSelectRequestHandler : Terminated client
2018-07-25 12:51:15.625 ERROR 12084 --- [nio-8443-exec-2] o.s.w.s.adapter.HttpWebHandlerAdapter : Unhandled failure: Eine bestehende Verbindung wurde softwaregesteuert durch den Hostcomputer abgebrochen, response already set (status=200)
2018-07-25 12:51:15.628 WARN 12084 --- [nio-8443-exec-2] o.s.h.s.r.ServletHttpHandlerAdapter : Handling completed with error: Eine bestehende Verbindung wurde softwaregesteuert durch den Hostcomputer abgebrochen
2018-07-25 12:51:15.652 ERROR 12084 --- [nio-8443-exec-2] o.a.catalina.connector.CoyoteAdapter : Exception while processing an asynchronous request
java.lang.IllegalStateException: Calling [asyncError()] is not valid for a request with Async state [DISPATCHING]
at org.apache.coyote.AsyncStateMachine.asyncError(AsyncStateMachine.java:424)
at org.apache.coyote.AbstractProcessor.action(AbstractProcessor.java:470)
at org.apache.coyote.Request.action(Request.java:431)
at org.apache.catalina.core.AsyncContextImpl.setErrorState(AsyncContextImpl.java:388)
at org.apache.catalina.connector.CoyoteAdapter.asyncDispatch(CoyoteAdapter.java:176)
at org.apache.coyote.AbstractProcessor.dispatch(AbstractProcessor.java:232)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:53)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1468)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Sorry for the german errormessages, it means "client abortet connection".
I don't really have a problem with this errormessage per se, it's just, that my buffers in spring's Webclient don't seem to be cleared up (the log I did not reproduce locally, so it has diferent timestamps):
2018-07-23 08:44:36.892 ERROR 22707 — [reactor-http-nio-5] io.netty.util.ResourceLeakDetector : LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
Created at:
io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:331) io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:185)
So here the question: How can I cleanly end the WebClient connection, when the request to my REST-Service is cancelled?

Answer 1

I can't really say for sure about that exception message, but I know Tomcat improved this in the 8.5.x generation. Which version are you using? If you can provide a consistent way to reproduce this with a minimal application, you could create a new issue in jira.spring.io on Spring Framework, or Tomcat itself if you managed to reproduce it without Spring (although it should be a hard one to reproduce).
Now about releasing DataBuffer instances - DataBuffer instances can be pooled, depending on the implementation. Here the WebClient is using Netty, which is pooling buffers. So they need to be released when they're no longer used.
Looking at your implementation, I think those unreleased buffers come from this:
the WebClient is fetching data from the remote endpoint and creating DataBuffer instances
various Reactor operators along the way are buffering those using internal queues (depending on the prefetching and the operators used, the amount of queued buffers can vary)
when the subscriber fails or cancels, those buffers sitting in internal queues are not released as they should.
Currently Reactor does not offer a hook point to reach those objects in those error cases. But this is a brand new feature that's been added in Reactor core 3.2.0. This will be leveraged internally by Spring Framework with SPR-17025. Please follow this issue - your use case might be handy when it comes to testing the fix.