I am using spring cloud gateway in my service and using below RequestDecorator as a wrapper in my LoggingFilter.
public class RequestDecorator extends ServerHttpRequestDecorator {
private final List<DataBuffer> dataBuffers = new ArrayList<>();
public RequestDecorator(ServerHttpRequest delegate) {
super(delegate);
super.getBody()
.map(
dataBuffer -> {
dataBuffers.add(dataBuffer);
return dataBuffer;
})
.subscribe();
}
#Override
public Flux<DataBuffer> getBody() {
return copy();
}
private Flux<DataBuffer> copy() {
return Flux.fromIterable(dataBuffers)
.map(dataBuffer -> dataBuffer.factory().wrap(dataBuffer.asByteBuffer()));
}
}
When the service is getting used by Jmeter for performance test, I got below memory Leak errors in the logs.
i.n.u.ResourceLeakDetector : - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
Created at:
io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:403)
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
io.netty.channel.unix.PreferredDirectByteBufAllocator.ioBuffer(PreferredDirectByteBufAllocator.java:53)
io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:120)
io.netty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:75)
io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:785)
io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.base/java.lang.Thread.run(Thread.java:834)
After checking few contents online, I found the following comment -
"If you are using DataBuffer you might get the same error. Spring has DataBufferUtils library to release the resource."
DataBufferUtils.release(dataBuffer);
But I would like to know how do I exactly use this in my decorator class as I am using this wrapper in my LoggingFilter.
Can anyone please advise ?
Related
Is it possible to get PathPattern in the SpringBoot web as a Bean and reuse it in user code?
For example, if the url is : /user/1990/lily, it return the url patten on the Controller: /user/{year}/{name}.
This said:
Patterns are parsed on startup and re-used at runtime for efficient
URL matching
Reactor-netty metrics need a uriTagValue to avoid cardinality explosion,
public class Application {
public static void main(String[] args) {
Metrics.globalRegistry
.config()
.meterFilter(MeterFilter.maximumAllowableTags("reactor.netty.http.server", "URI", 100, MeterFilter.deny()));
DisposableServer server =
HttpServer.create()
.metrics(true, s -> { // HERE is the uriTagValue, it's a smaple of how to handle url mapping.
if (s.startsWith("/stream/")) {
return "/stream/{n}";
}
else if (s.startsWith("/bytes/")) {
return "/bytes/{n}";
}
return s;
})
.route(r ->
r.get("/stream/{n}",
(req, res) -> res.sendString(Mono.just(req.param("n"))))
.get("/bytes/{n}",
(req, res) -> res.sendString(Mono.just(req.param("n")))))
.bindNow();
server.onDispose()
.block();
}
}
Config the Netty to enable metrics in a SpringBoot WebFlux app:
#Configuration
public class NettyWebServerConfig {
#Bean
public ReactiveWebServerFactory reactiveWebServerFactory() {
NettyReactiveWebServerFactory factory = new NettyReactiveWebServerFactory();
factory.addServerCustomizers(httpServer -> httpServer
.wiretap(true)
.metrics(true, s -> "") // enable metrics, ignore all uri, if SpringBoot Web expose URI-Match-Patterns as Bean, we can use it here.
);
return factory;
}
}
My wondering is that is it possible to get PathPattern as a Bean in the SpringBoot web and reuse it in reactor-netty metrics code? As simmple as: bestPattern.matchAndExtract(lookupPath)
I tested PathContainer.parsePath(s);, it seems doesn't work.
With this setup, you are not using Spring WebFlux but actually Reactor Netty directly. PathContainer and PathPattern are then irrevelant here.
I don't think reactor-netty is storing anywhere the matching UriPathTemplate when considering the HttpPredicate.
Software versions in use:
spring-webflux-5.3.4,
reactor-core-3.4.4,
spring-data-mongodb-3.1.6
Am building a spring boot application that uses spring webclient to
invoke an image service that will serve a pdf image back.
The returned pdf is then stored in mongodb using spring's ReactiveGridfsTemplate.
For performance testing am having the service return 120 MB pdf all the
time.
First invocation of the service and storing the returned pdf in mongodb works fine and happens in under 10 seconds.
However, second invocation onward, I start getting the following error while storing the returned pdf in mongodb. Can someone advise on what am doing wrong?
Caused by: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 1056964615, max: 1073741824)
at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:776)
at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:731)
at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:645)
at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:621)
at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:204)
at io.netty.buffer.PoolArena.tcacheAllocateNormal(PoolArena.java:188)
at io.netty.buffer.PoolArena.allocate(PoolArena.java:138)
at io.netty.buffer.PoolArena.allocate(PoolArena.java:128)
at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:378)
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:139)
at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:150)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:825)
Code to build webclient:
WebClient webClient = WebClient.builder().filter(WebClientFilter.logRequest())// for logging request
.filter(WebClientFilter.logResponse()) // for logging response
.exchangeStrategies(ExchangeStrategies.builder()
.codecs(configurer -> configurer.defaultCodecs().maxInMemorySize(5242880)).build())
.build();
Code to invoke image service using webclient:
Flux<DataBuffer> imageFlux = webClient.method(httpmethod).uri(uri)
.bodyValue((payloadBody == null) ? StringUtils.EMPTY : payloadBody.toPayloadBody())
.accept(MediaType.ALL).exchangeToFlux(response -> {
logger.log(Level.DEBUG, "DefaultHttpClient exchangeToFlux got response with status code {}",response.statusCode());
if (response.statusCode().is4xxClientError() || response.statusCode().is5xxServerError()) {
logger.log(Level.ERROR,
"DefaultHttpClient exchangeToFlux encountered error {} throwing service exception",
response.statusCode());
return Flux.error(new ServiceException(response.bodyToMono(String.class).flatMap(body -> {
return Mono.just(body);
}), response.rawStatusCode()));
}
return response.bodyToFlux(DataBuffer.class);
});
Code to store pdf in mongodb returned by image service using spring's ReactiveGridfsTemplate:
imageFlux is what I receive above.
protected Mono<ObjectId> getMono(Flux<DataBuffer> imageFlux , DocumentContext documentContext) {
return reactiveGridFsTmpl.store(imageFlux, new java.util.Date() + ApplicationConstants.PDF_EXTENSION,
<org.bson.Document object with attributes from application>);
}
Here's how am firing the store call by subscribing to Mono returned by getMono(....). Within onComplete and onError have tried to release data buffer
Mono<ObjectId> imageObjectId = getMono(imageFlux, documentContext);
imageObjectId.subscribe(new Subscriber<ObjectId>() {
#Override
public void onComplete() {
logger.log(Level.DEBUG, SUBSCRIPTION_ON_COMPLETE);
DataBufferUtils.release(imageFlux.blockFirst()); --> Attempt to release databuffer
logger.log(Level.DEBUG, SUBSCRIPTION_ON_COMPLETE_RELEASE_DATABUFFER);
}
#Override
public void onError(Throwable t) {
logger.log(Level.ERROR, SUBSCRIPTION_ON_ERROR + t);
if (t instanceof ServiceException) {
logger.log(Level.ERROR, "DocumentDao caught ServiceException.");
flagErrorRecord((ServiceException) t, documentContext);
}
DataBufferUtils.release(imageFlux.blockFirst()); --> Attempt to release databuffer
logger.log(Level.ERROR, SUBSCRIPTION_ON_ERROR_RELEASE_DATABUFFER);
}
#Override
public void onNext(ObjectId t) {
logger.log(Level.DEBUG, SUBSCRIPTION_ON_NEXT + t.toString());
}
#Override
public void onSubscribe(Subscription s) {
logger.log(Level.DEBUG, SUBSCRIPTION_ON_SUBSCRIBE);
s.request(1);
}
});
try to change the directMemory using the JAVA_OPTS Environment variable.
JBP_CONFIG_JAVA_OPTS: '{ java_opts: "-XX:MaxDirectMemorySize=2048m" }'
I see that 1G is not sufficient. so try to set it at 2G
my code:
https://github.com/nikita111100/gRPC-vs-REST
i cant get a trace of all services
I guess grpc-spring-boot-starter can do this without propagate headers
it works for me
#Configuration
public class TracingConfig {
#GrpcGlobalClientInterceptor
TracingClientInterceptor tracingInterceptor(Tracer tracer) {
return TracingClientInterceptor
.newBuilder()
.withTracer(tracer)
.build();
}
https://github.com/opentracing-contrib/java-grpc
Problem
I have a Spring-Boot application in which I am also starting a gRPC server/service. Both the servlet and gRPC code send requests to a common object to process the request. When the request comes in I want to update the logging to display a unique 'ID' so I can track the request through the system.
On the Spring side I have setup a 'Filter' which updates the logging MDC to add some data to the log request (see this example). this works fine
On the gRPC side I have created an 'ServerInterceptor' and added it to the service, while the interceptor gets called the code to update the MDC does not stick, so when a request comes through the gRPC service I do not get the ID printed in the log. I realize this has to do with the fact that I'm intercepting the call in one thread and it's being dispatched by gRPC in another, what I can't seem to figure out is how to either intercept the call in the thread doing the work or add the MDC information so it is properly propagated to the thread doing the work.
What I've tried
I have done a lot of searches and was quite surprised to not find this asked/answered, I can only assume my query skills are lacking :(
I'm fairly new to gRPC and this is the first Interceptor I'm writing. I've tried adding the interceptor several different ways (via ServerInterceptors.intercept, BindableService instance.intercept).
I've looked at LogNet's Spring Boot gRPC Starter, but I'm not sure this would solve the issue.
Here is the code I have added in my interceptor class
#Override
public <ReqT, RespT> ServerCall.Listener<ReqT> interceptCall(final ServerCall<ReqT, RespT> call, final Metadata headers, final ServerCallHandler<ReqT, RespT> next) {
try {
final String mdcData = String.format("[requestID=%s]",
UUID.randomUUID().toString());
MDC.put(MDC_DATA_KEY, mdcData);
return next.startCall(call, headers);
} finally {
MDC.clear();
}
}
Expected Result
When a request comes in via the RESTful API I see log output like this
2019-04-09 10:19:16.331 [requestID=380e28db-c8da-4e35-a097-4b8c90c006f4] INFO 87100 --- [nio-8080-exec-1] c.c.es.xxx: processing request step 1
2019-04-09 10:19:16.800 [requestID=380e28db-c8da-4e35-a097-4b8c90c006f4] INFO 87100 --- [nio-8080-exec-1] c.c.es.xxx: processing request step 2
2019-04-09 10:19:16.803 [requestID=380e28db-c8da-4e35-a097-4b8c90c006f4] INFO 87100 --- [nio-8080-exec-1] c.c.es.xxx: Processing request step 3
...
I'm hoping to get similar output when the request comes through the gRPC service.
Thanks
Since no one replied, I kept trying and came up with the following solution for my interceptCall function. I'm not 100% sure why this works, but it works for my use case.
private class LogInterceptor implements ServerInterceptor {
#Override
public <ReqT, RespT> ServerCall.Listener<ReqT> interceptCall(final ServerCall<ReqT, RespT> call,
final Metadata headers,
final ServerCallHandler<ReqT, RespT> next) {
Context context = Context.current();
final String requestId = UUID.randomUUID().toString();
return Contexts.interceptCall(context, call, headers, new ServerCallHandler<ReqT, RespT>() {
#Override
public ServerCall.Listener<ReqT> startCall(ServerCall<ReqT, RespT> call, Metadata headers) {
return new ForwardingServerCallListener.SimpleForwardingServerCallListener<ReqT>(next.startCall(call, headers)) {
/**
* The actual service call happens during onHalfClose().
*/
#Override
public void onHalfClose() {
try (final CloseableThreadContext.Instance ctc = CloseableThreadContext.put("requestID",
UUID.randomUUID().toString())) {
super.onHalfClose();
}
}
};
}
});
}
}
In my application.properties I added the following (which I already had)
logging.pattern.level=[%X] %-5level
The '%X' tells the logging system to print all of the CloseableThreadContext key/values.
Hopefully this may help someone else.
MDC stores data in ThreadLocal variable and you are right about - "I realize this has to do with the fact that I'm intercepting the call in one thread and it's being dispatched by gRPC in another". Check #Eric Anderson answer about the right way to use ThradLocal in the post -
https://stackoverflow.com/a/56842315/2478531
Here is a working example -
public class GrpcMDCInterceptor implements ServerInterceptor {
private static final String MDC_DATA_KEY = "Key";
#Override
public <R, S> ServerCall.Listener<R> interceptCall(
ServerCall<R, S> serverCall,
Metadata metadata,
ServerCallHandler<R, S> next
) {
log.info("Setting user context, metadata {}", metadata);
final String mdcData = String.format("[requestID=%s]", UUID.randomUUID().toString());
MDC.put(MDC_DATA_KEY, mdcData);
try {
return new WrappingListener<>(next.startCall(serverCall, metadata), mdcData);
} finally {
MDC.clear();
}
}
private static class WrappingListener<R>
extends ForwardingServerCallListener.SimpleForwardingServerCallListener<R> {
private final String mdcData;
public WrappingListener(ServerCall.Listener<R> delegate, String mdcData) {
super(delegate);
this.mdcData = mdcData;
}
#Override
public void onMessage(R message) {
MDC.put(MDC_DATA_KEY, mdcData);
try {
super.onMessage(message);
} finally {
MDC.clear();
}
}
#Override
public void onHalfClose() {
MDC.put(MDC_DATA_KEY, mdcData);
try {
super.onHalfClose();
} finally {
MDC.clear();
}
}
#Override
public void onCancel() {
MDC.put(MDC_DATA_KEY, mdcData);
try {
super.onCancel();
} finally {
MDC.clear();
}
}
#Override
public void onComplete() {
MDC.put(MDC_DATA_KEY, mdcData);
try {
super.onComplete();
} finally {
MDC.clear();
}
}
#Override
public void onReady() {
MDC.put(MDC_DATA_KEY, mdcData);
try {
super.onReady();
} finally {
MDC.clear();
}
}
}
}
Using Springboot 1.5.x, Spring Cloud, and JAX-RS:
I could use a second pair of eyes since it is not clear to me whether the Spring configured, Javanica HystrixCommand works for all use cases or whether I may have an error in my code. Below is an approximation of what I'm doing, the code below will not actually compile.
From below WebService lives in a library with separate package path to the main application(s). Meanwhile MyWebService lives in the application that is in the same context path as the Springboot application. Also MyWebService is functional, no issues there. This just has to do with the visibility of HystrixCommand annotation in regards to Springboot based configuration.
At runtime, what I notice is that when a code like the one below runs, I do see "commandKey=A" in my response. This one I did not quite expect since it's still running while the data is obtained. And since we log the HystrixRequestLog, I also see this command key in my logs.
But all the other Command keys are not visible at all, regardless of where I place them in the file. If I remove CommandKey-A then no commands are visible whatsoever.
Thoughts?
// Example WebService that we use as a shared component for performing a backend call that is the same across different resources
#RequiredArgsConstructor
#Accessors(fluent = true)
#Setter
public abstract class WebService {
private final #Nonnull Supplier<X> backendFactory;
#Setter(AccessLevel.PACKAGE)
private #Nonnull Supplier<BackendComponent> backendComponentSupplier = () -> new BackendComponent();
#GET
#Produces("application/json")
#HystrixCommand(commandKey="A")
public Response mainCall() {
Object obj = new Object();
try {
otherCommandMethod();
} catch (Exception commandException) {
// do nothing (for this example)
}
// get the hystrix request information so that we can determine what was executed
Optional<Collection<HystrixInvokableInfo<?>>> executedCommands = hystrixExecutedCommands();
// set the hystrix data, viewable in the response
obj.setData("hystrix", executedCommands.orElse(Collections.emptyList()));
if(hasError(obj)) {
return Response.serverError()
.entity(obj)
.build();
}
return Response.ok()
.entity(healthObject)
.build();
}
#HystrixCommand(commandKey="B")
private void otherCommandMethod() {
backendComponentSupplier
.get()
.observe()
.toBlocking()
.subscribe();
}
Optional<Collection<HystrixInvokableInfo<?>>> hystrixExecutedCommands() {
Optional<HystrixRequestLog> hystrixRequest = Optional
.ofNullable(HystrixRequestLog.getCurrentRequest());
// get the hystrix executed commands
Optional<Collection<HystrixInvokableInfo<?>>> executedCommands = Optional.empty();
if (hystrixRequest.isPresent()) {
executedCommands = Optional.of(hystrixRequest.get()
.getAllExecutedCommands());
}
return executedCommands;
}
#Setter
#RequiredArgsConstructor
public class BackendComponent implements ObservableCommand<Void> {
#Override
#HystrixCommand(commandKey="Y")
public Observable<Void> observe() {
// make some backend call
return backendFactory.get()
.observe();
}
}
}
// then later this component gets configured in the specific applications with sample configuraiton that looks like this:
#SuppressWarnings({ "unchecked", "rawtypes" })
#Path("resource/somepath")
#Component
public class MyWebService extends WebService {
#Inject
public MyWebService(Supplier<X> backendSupplier) {
super((Supplier)backendSupplier);
}
}
There is an issue with mainCall() calling otherCommandMethod(). Methods with #HystrixCommand can not be called from within the same class.
As discussed in the answers to this question this is a limitation of Spring's AOP.