How to switch back from blocking scheduler to previous scheduler using netty-reactor? - spring-boot

How to switch back from blocking scheduler (blocking-pool) to previous scheduler (reactor-http-nio) using Spring Webflux + Netty + Reactor?
The code:
#RequiredArgsConstructor
#Service
#Slf4j
public class BookService {
private final IBookRepo bookRepo;
private final BlockingPoolConfig blockingPoolConfig;
public Mono<Optional<Book>> getBook(Long id) {
log.debug("getBook() - id: {}", id);
return asyncCallable(() -> {
log.trace("getBook() - invoking bookRepo.findById(id) ...");
return bookRepo.findById(id);
});
}
protected <S> Mono<S> asyncCallable(Callable<S> callable) {
return Mono.fromCallable(callable)
.subscribeOn(blockingPoolConfig.blockingScheduler());
}
}
#RestController
#RequiredArgsConstructor
#Slf4j
public class BookController {
private final BookService bookService;
#GetMapping("/book/{id}")
public Mono<Book> get(#PathVariable Long id) {
log.debug("get() - id: {}", id);
return bookService.getBook(id)
.publishOn(Schedulers.parallel()) //publishOn(... ?)
.map(optionalBook -> {
return optionalBook.map(book -> {
log.debug("get() result: {}", book);
return book;
}).orElseThrow(() -> {
log.debug("book with id: {} is not found.", id);
return new ResponseStatusException(HttpStatus.NOT_FOUND, "Book not found");
});
});
}
#Configuration
#Slf4j
public class BlockingPoolConfig {
#Value("${spring.datasource.maximumPoolSize:8}")
private int connectionPoolSize = 1;
#Scope("singleton")
#Bean
public Scheduler blockingScheduler() {
Scheduler scheduler = Schedulers.newBoundedElastic(connectionPoolSize, connectionPoolSize, "blocking-pool");
return scheduler;
}
}
Above i'm using publishOn(Schedulers.parallel()), but this one creates new thread pool (parallel). Instead of this I prefer to switch reactor-http-nio thread pool.
Log of actual result:
19:17:45.290 [reactor-http-nio-2 ] DEBUG t.a.p.controller.BookController - get() - id: 1
19:17:45.291 [reactor-http-nio-2 ] DEBUG t.a.p.service.BookService - getBook() - id: 1
19:17:45.316 [blocking-pool-1 ] TRACE t.a.p.service.BookService - getBook() - invoking bookRepo.findById(id) ...
19:17:45.427 [parallel-2 ] DEBUG t.a.p.controller.BookController - get() result: Book(id=1, title=Abc)
Log of expected result:
19:17:45.290 [reactor-http-nio-2 ] DEBUG t.a.p.controller.BookController - get() - id: 1
19:17:45.291 [reactor-http-nio-2 ] DEBUG t.a.p.service.BookService - getBook() - id: 1
19:17:45.316 [blocking-pool-1 ] TRACE t.a.p.service.BookService - getBook() - invoking bookRepo.findById(id) ...
19:17:45.427 [reactor-http-nio-2 ] DEBUG t.a.p.controller.BookController - get() result: Book(id=1, title=Abc)

This is currently not possible, because A) these HTTP threads are not controlled by a Reactor Scheduler, but by the underlying Netty event loop itself, and B) there's no generic way in Java to "return execution to an (arbitrary) thread" if that thread doesn't have an Executor/ExecutorService associated with it.
For reactor-netty, once you've switched out of the HTTP threads there should be little reason to want to switch back to the Netty threads anyway. It will be done naturally by reactor-netty once the response is sent.
Assuming blocking pool is something like Schedulers.boundedElastic(), you might indeed want to go to Schedulers.parallel() to limit the life of the blocking threads, and that's a perfectly fine solution.

Related

How to manage tracing with Spring WebClient in Reactive way?

I have a method in EventService that calls an API and handle errors if there is any.
private Mono<ApiResponse> send(String accountId) {
accountIdBaggageField.updateValue(accountId);
logger.info("making the call");
Mono<ApiResponse> res = apiClient.dispatchEvent(accountId);
return res.doOnError(e -> {
logger.error("Could not dispatch batch for events");
});
}
Here is the SleuthConfiguration class that defines accountIdBaggageField bean:
#Configuration
public class SleuthConfiguration {
#Bean
public BaggageField accountIdBaggageField() {
return BaggageField.create(LoggingContextVariables.MDC_ACCOUNT_ID);
}
#Bean
public BaggagePropagationCustomizer baggagePropagationCustomizer(BaggageField accountIdBaggageField) {
return factoryBuilder -> {
factoryBuilder.add(remote(accountIdBaggageField));
};
}
#Bean
public CorrelationScopeCustomizer correlationScopeCustomizer(BaggageField accountIdBaggageField) {
return builder -> {
builder.add(createCorrelationScopeConfig(accountIdBaggageField));
};
}
private CorrelationScopeConfig createCorrelationScopeConfig(BaggageField field) {
return CorrelationScopeConfig.SingleCorrelationField.newBuilder(field)
.flushOnUpdate()
.build();
}
}
Here is the ApiClient's dispatchEvents method:
public Mono<ApiResponse> dispatchEvent(String accountId) {
return webClient
.post()
.uri(properties.getEndpoints().getDispatchEvent(), Map.of("accountId", accountId))
.retrieve()
.onStatus(HttpStatus::isError, this::constructException)
.bodyToMono(ApiResponse.class)
.onErrorMap(WebClientRequestException.class, e -> new CGWException("Error during dispatching event to Connector Gateway", e));
}
Here is how I call the send method:
eventService.send("account1");
eventService.send("account2");
The problem here is that the accountIdBaggageField is first set to "account1" and then the io process is started when apiClient.dispatchEvents is called. Before the end of the io process (before getting a response from the api), the second call takes place and the accountIdBaggageField is set to "account2".
Then, when the response of the first request is returned, the error log in doOnError adds the accountId to the log as "account2" but needs to add it as "account1".
Here are the logs:
2023-01-09 11:50:56.791 INFO [account1] [Thread-1] c.t.e.s.impl.EventServiceImpl making the call
2023-01-09 11:50:56.812 INFO [account2] [Thread-1] c.t.e.s.impl.EventServiceImpl making the call
2023-01-09 11:50:58.241 INFO [account2] [reactor-http-nio-4] c.t.e.s.impl.EventServiceImpl Could not dispatch batch for events
2023-01-09 11:50:58.281 INFO [account2] [reactor-http-nio-6] c.t.e.s.impl.EventServiceImpl Could not dispatch batch for events
As can be seen in the logs, the log on line 3 should have been accountId1 instead of accountId2.
How can I fix this situation?

Spring JPA transaction partially retry

I am trying to use Spring retry for my Spring web application (with JPA, hibernate).
Currently, I have a long transaction chain (marked functions as #Transactional) in my application, intuitively:
start transction -> A() -> B() -> C() -> D() - E() -> commit/rollback
now I want to have D() to be retried if any concurrency failure occurred (marked D() with #retryable), but remain A(), B(), C() in current states
I failed, the function D() is not retried at all and just throwed a concurrency failure error (eg. ObjectOptimisticLockingFailureException)
For me, if I want to do such things in database, I have to make a new transaction try catch block with a cursor while loop to handle retries. I wonder is there a simple way I can handle this "partial" transaction retry in Spring?
An example code would be:
#RestController
public DimensionController()
{
...
#Autowired
private TblDimensionService dimensionService;
...
#PutMapping(...)
public ResponseEntity<TblDimensionDTO> update(#Valid #RequestBody TblDimensionDTO dimensionDTO)
{
...
dimensionService.update(dimensionDTO);
...
}
}
#Transactional //transaction in service level
#Service
public TblDimensionService()
{
...
#Autowired
private RetryService retryService;
...
public TblDimensionDTO update(TblDimensionDTO dimensionDTO) throws InterruptedException
{
if (dimensionDTO.getId() == null)
{
throw new ResponseStatusException(HttpStatus.NOT_FOUND, "not found");
}
TblDimension dimension = findByIdOrElseThrow(dimensionDTO.getId()); //call another function to retrieve entity by id with JPA repository
dimension = retryService.updateEntity(dimension, dimensionDTO);
return tblDimensionMapper.toDto(dimension);
}
...
}
#Transactional //transaction in service level
#Service
public RetryService()
{
...
#Autowired
private TblDimensionRepository dimensionRepository;
...
//I want to only retry this part
//but this retry does not work
#Retryable(value = {ConcurrencyFailureException.class})
public TblDimension updateEntity(TblDimension dimension, TblDimensionDTO dimensionDTO) throws InterruptedException
{
Thread.sleep(3000);
dimension.setHeight(dimension.getHeight() + 1);
Thread.sleep(3000);
return dimensionRepository.save(dimension);
}
...
}

Netty how to test Handler which uses Remote Address of a client

I have a Netty TCP Server with Spring Boot 2.3.1 with the following handler :
#Slf4j
#Component
#RequiredArgsConstructor
#ChannelHandler.Sharable
public class QrReaderProcessingHandler extends ChannelInboundHandlerAdapter {
private final CarParkPermissionService permissionService;
private final Gson gson = new Gson();
private String remoteAddress;
#Override
public void channelActive(ChannelHandlerContext ctx) {
ctx.fireChannelActive();
remoteAddress = ctx.channel().remoteAddress().toString();
if (log.isDebugEnabled()) {
log.debug(remoteAddress);
}
ctx.writeAndFlush("Your remote address is " + remoteAddress + ".\r\n");
}
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg) {
log.info("CLIENT_IP: {}", remoteAddress);
String stringMsg = (String) msg;
log.info("CLIENT_REQUEST: {}", stringMsg);
String lowerCaseMsg = stringMsg.toLowerCase();
if (RequestType.HEARTBEAT.containsName(lowerCaseMsg)) {
HeartbeatRequest heartbeatRequest = gson.fromJson(stringMsg, HeartbeatRequest.class);
log.debug("heartbeat request: {}", heartbeatRequest);
HeartbeatResponse response = HeartbeatResponse.builder()
.responseCode("ok")
.build();
ctx.writeAndFlush(response + "\n\r");
}
}
Request DTO:
#Data
#Builder
#NoArgsConstructor
#AllArgsConstructor
public class HeartbeatRequest {
private String messageID;
}
Response DTO:
#Data
#Builder
#NoArgsConstructor
#AllArgsConstructor
public class HeartbeatResponse {
private String responseCode;
}
Logic is quite simple. Only I have to know the IP address of the client.
I need to test it as well.
I have been looking for many resources for testing handlers for Netty, like
Testing Netty with EmbeddedChannel
How to unit test netty handler
However, it didn't work for me.
For EmbeddedChannel I have following error - Your remote address is embedded.
Here is code:
#ActiveProfiles("test")
#RunWith(MockitoJUnitRunner.class)
public class ProcessingHandlerTest_Embedded {
#Mock
private PermissionService permissionService;
private EmbeddedChannel embeddedChannel;
private final Gson gson = new Gson();
private ProcessingHandler processingHandler;
#Before
public void setUp() {
processingHandler = new ProcessingHandler(permissionService);
embeddedChannel = new EmbeddedChannel(processingHandler);
}
#Test
public void testHeartbeatMessage() {
// given
HeartbeatRequest heartbeatMessage = HeartbeatRequest.builder()
.messageID("heartbeat")
.build();
HeartbeatResponse response = HeartbeatResponse.builder()
.responseCode("ok")
.build();
String request = gson.toJson(heartbeatMessage).concat("\r\n");
String expected = gson.toJson(response).concat("\r\n");
// when
embeddedChannel.writeInbound(request);
// then
Queue<Object> outboundMessages = embeddedChannel.outboundMessages();
assertEquals(expected, outboundMessages.poll());
}
}
Output:
22:21:29.062 [main] INFO handler.ProcessingHandler - CLIENT_IP: embedded
22:21:29.062 [main] INFO handler.ProcessingHandler - CLIENT_REQUEST: {"messageID":"heartbeat"}
22:21:29.067 [main] DEBUG handler.ProcessingHandler - heartbeat request: HeartbeatRequest(messageID=heartbeat)
org.junit.ComparisonFailure:
<Click to see difference>
However, I don't know how to do exact testing for such a case.
Here is a snippet from configuration:
#Bean
#SneakyThrows
public InetSocketAddress tcpSocketAddress() {
// for now, hostname is: localhost/127.0.0.1:9090
return new InetSocketAddress("localhost", nettyProperties.getTcpPort());
// for real client devices: A05264/172.28.1.162:9090
// return new InetSocketAddress(InetAddress.getLocalHost(), nettyProperties.getTcpPort());
}
#Component
#RequiredArgsConstructor
public class QrReaderChannelInitializer extends ChannelInitializer<SocketChannel> {
private final StringEncoder stringEncoder = new StringEncoder();
private final StringDecoder stringDecoder = new StringDecoder();
private final QrReaderProcessingHandler readerServerHandler;
private final NettyProperties nettyProperties;
#Override
protected void initChannel(SocketChannel socketChannel) {
ChannelPipeline pipeline = socketChannel.pipeline();
// Add the text line codec combination first
pipeline.addLast(new DelimiterBasedFrameDecoder(1024 * 1024, Delimiters.lineDelimiter()));
pipeline.addLast(new ReadTimeoutHandler(nettyProperties.getClientTimeout()));
pipeline.addLast(stringDecoder);
pipeline.addLast(stringEncoder);
pipeline.addLast(readerServerHandler);
}
}
How to test handler with IP address of a client?
Two things that could help:
Do not annotate with #ChannelHandler.Sharable if your handler is NOT sharable. This can be misleading. Remove unnecessary state from handlers. In your case you should remove the remoteAddress member variable and ensure that Gson and CarParkPermissionService can be reused and are thread-safe.
"Your remote address is embedded" is NOT an error. It actually is the message written by your handler onto the outbound channel (cf. your channelActive() method)
So it looks like it could work.
EDIT
Following your comments here are some clarifications regarding the second point. I mean that:
your code making use of EmbeddedChannel is almost correct. There is just a misunderstanding on the expected results (assert).
To make the unit test successful, you just have either:
to comment this line in channelActive(): ctx.writeAndFlush("Your remote ...")
or to poll the second message from Queue<Object> outboundMessages in testHeartbeatMessage()
Indeed, when you do this:
// when
embeddedChannel.writeInbound(request);
(1) You actually open the channel once, which fires a channelActive() event. You don't have a log in it but we see that the variable remoteAddress is not null afterwards, meaning that it was assigned in the channelActive() method.
(2) At the end of the channelActive() method, you eventually already send back a message by writing on the channel pipeline, as seen at this line:
ctx.writeAndFlush("Your remote address is " + remoteAddress + ".\r\n");
// In fact, this is the message you see in your failed assertion.
(3) Then the message written by embeddedChannel.writeInbound(request) is received and can be read, which fires a channelRead() event. This time, we see this in your log output:
22:21:29.062 [main] INFO handler.ProcessingHandler - CLIENT_IP: embedded
22:21:29.062 [main] INFO handler.ProcessingHandler - CLIENT_REQUEST: {"messageID":"heartbeat"}
22:21:29.067 [main] DEBUG handler.ProcessingHandler - heartbeat request: HeartbeatRequest(messageID=heartbeat)
(4) At the end of channelRead(ChannelHandlerContext ctx, Object msg), you will then send a second message (the expected one):
HeartbeatResponse response = HeartbeatResponse.builder()
.responseCode("ok")
.build();
ctx.writeAndFlush(response + "\n\r");
Therefore, with the following code of your unit test...
Queue<Object> outboundMessages = embeddedChannel.outboundMessages();
assertEquals(expected, outboundMessages.poll());
... you should be able to poll() two messages:
"Your remote address is embedded"
"{ResponseCode":"ok"}
Does it make sense for you?

Hystrix fallback method returns null

I implemented feign client and hystrix to my spring boot microservice application.
I first tried to test to communicate users service to albums service with feign client,
so I threw an exception at albums service to check if users service Error Decoder can catch the exception and then make the fallback method triggered.
It worked, but the cause is always null only at the first time, and after that I can see the error message that I wanted to see.
Can anyone tell me if something is wrong or not.
This is my code.
Users Service Feign Client
#FeignClient(name = "albums-ws", fallbackFactory = AlbumsFallbackFactory.class)
public interface AlbumServiceClient {
#GetMapping(path = "users/{userId}/albums")
List<AlbumDetailResponse> getAlbums(#PathVariable("userId") String userId);
}
Fallback Factory
#Component
public class AlbumsFallbackFactory implements FallbackFactory<AlbumServiceClient> {
#Override
public AlbumServiceClient create(Throwable cause) {
return new AlbumServiceClientFallback(cause);
}
}
public class AlbumServiceClientFallback implements AlbumServiceClient {
private final Throwable cause;
private final Logger logger = LoggerFactory.getLogger(this.getClass());
public AlbumServiceClientFallback(Throwable cause) {
this.cause = cause;
}
#Override
public List<AlbumDetailResponse> getAlbums(String userId) {
logger.error("An exception took place: " + cause.getMessage());
return new ArrayList<>();
}
}
Feign Error Decoder
#Component
public class FeignErrorDecoder implements ErrorDecoder {
#Override
public Exception decode(String methodKey, Response response) {
switch(response.status()) {
case 400:
break;
case 404:
if(methodKey.contains("getAlbums")) {
return new ResponseStatusException(HttpStatus.valueOf(response.status()), response.reason());
}
break;
default:
return new Exception(response.reason());
}
return null;
}
}
First fallback triggered
2020-08-02 12:42:27.836 ERROR 24772 --- [ HystrixTimer-1] c.a.p.a.u.P.f.AlbumServiceClientFallback : An exception took place: null
After
2020-08-02 12:43:07.672 DEBUG 24772 --- [rix-albums-ws-2] c.a.p.a.u.P.feign.AlbumServiceClient : [AlbumServiceClient#getAlbums] User not found with id: f5b313e2-411f-4fc3-95e7-9aa5c43c286c
Hystrix has class org.springframework.cloud.netflix.feign.HystrixTargeter. There is a comment in targetWithFallbackFactory method:
We take a sample fallback from the fallback factory to check if it
returns a fallback that is compatible with the annotated feign
interface.
and code after:
Object exampleFallback = fallbackFactory.create(new RuntimeException());
It is why you don't have cause in exception.

How to use a gRPC interceptor to attach/update logging MDC in a Spring-Boot app

Problem
I have a Spring-Boot application in which I am also starting a gRPC server/service. Both the servlet and gRPC code send requests to a common object to process the request. When the request comes in I want to update the logging to display a unique 'ID' so I can track the request through the system.
On the Spring side I have setup a 'Filter' which updates the logging MDC to add some data to the log request (see this example). this works fine
On the gRPC side I have created an 'ServerInterceptor' and added it to the service, while the interceptor gets called the code to update the MDC does not stick, so when a request comes through the gRPC service I do not get the ID printed in the log. I realize this has to do with the fact that I'm intercepting the call in one thread and it's being dispatched by gRPC in another, what I can't seem to figure out is how to either intercept the call in the thread doing the work or add the MDC information so it is properly propagated to the thread doing the work.
What I've tried
I have done a lot of searches and was quite surprised to not find this asked/answered, I can only assume my query skills are lacking :(
I'm fairly new to gRPC and this is the first Interceptor I'm writing. I've tried adding the interceptor several different ways (via ServerInterceptors.intercept, BindableService instance.intercept).
I've looked at LogNet's Spring Boot gRPC Starter, but I'm not sure this would solve the issue.
Here is the code I have added in my interceptor class
#Override
public <ReqT, RespT> ServerCall.Listener<ReqT> interceptCall(final ServerCall<ReqT, RespT> call, final Metadata headers, final ServerCallHandler<ReqT, RespT> next) {
try {
final String mdcData = String.format("[requestID=%s]",
UUID.randomUUID().toString());
MDC.put(MDC_DATA_KEY, mdcData);
return next.startCall(call, headers);
} finally {
MDC.clear();
}
}
Expected Result
When a request comes in via the RESTful API I see log output like this
2019-04-09 10:19:16.331 [requestID=380e28db-c8da-4e35-a097-4b8c90c006f4] INFO 87100 --- [nio-8080-exec-1] c.c.es.xxx: processing request step 1
2019-04-09 10:19:16.800 [requestID=380e28db-c8da-4e35-a097-4b8c90c006f4] INFO 87100 --- [nio-8080-exec-1] c.c.es.xxx: processing request step 2
2019-04-09 10:19:16.803 [requestID=380e28db-c8da-4e35-a097-4b8c90c006f4] INFO 87100 --- [nio-8080-exec-1] c.c.es.xxx: Processing request step 3
...
I'm hoping to get similar output when the request comes through the gRPC service.
Thanks
Since no one replied, I kept trying and came up with the following solution for my interceptCall function. I'm not 100% sure why this works, but it works for my use case.
private class LogInterceptor implements ServerInterceptor {
#Override
public <ReqT, RespT> ServerCall.Listener<ReqT> interceptCall(final ServerCall<ReqT, RespT> call,
final Metadata headers,
final ServerCallHandler<ReqT, RespT> next) {
Context context = Context.current();
final String requestId = UUID.randomUUID().toString();
return Contexts.interceptCall(context, call, headers, new ServerCallHandler<ReqT, RespT>() {
#Override
public ServerCall.Listener<ReqT> startCall(ServerCall<ReqT, RespT> call, Metadata headers) {
return new ForwardingServerCallListener.SimpleForwardingServerCallListener<ReqT>(next.startCall(call, headers)) {
/**
* The actual service call happens during onHalfClose().
*/
#Override
public void onHalfClose() {
try (final CloseableThreadContext.Instance ctc = CloseableThreadContext.put("requestID",
UUID.randomUUID().toString())) {
super.onHalfClose();
}
}
};
}
});
}
}
In my application.properties I added the following (which I already had)
logging.pattern.level=[%X] %-5level
The '%X' tells the logging system to print all of the CloseableThreadContext key/values.
Hopefully this may help someone else.
MDC stores data in ThreadLocal variable and you are right about - "I realize this has to do with the fact that I'm intercepting the call in one thread and it's being dispatched by gRPC in another". Check #Eric Anderson answer about the right way to use ThradLocal in the post -
https://stackoverflow.com/a/56842315/2478531
Here is a working example -
public class GrpcMDCInterceptor implements ServerInterceptor {
private static final String MDC_DATA_KEY = "Key";
#Override
public <R, S> ServerCall.Listener<R> interceptCall(
ServerCall<R, S> serverCall,
Metadata metadata,
ServerCallHandler<R, S> next
) {
log.info("Setting user context, metadata {}", metadata);
final String mdcData = String.format("[requestID=%s]", UUID.randomUUID().toString());
MDC.put(MDC_DATA_KEY, mdcData);
try {
return new WrappingListener<>(next.startCall(serverCall, metadata), mdcData);
} finally {
MDC.clear();
}
}
private static class WrappingListener<R>
extends ForwardingServerCallListener.SimpleForwardingServerCallListener<R> {
private final String mdcData;
public WrappingListener(ServerCall.Listener<R> delegate, String mdcData) {
super(delegate);
this.mdcData = mdcData;
}
#Override
public void onMessage(R message) {
MDC.put(MDC_DATA_KEY, mdcData);
try {
super.onMessage(message);
} finally {
MDC.clear();
}
}
#Override
public void onHalfClose() {
MDC.put(MDC_DATA_KEY, mdcData);
try {
super.onHalfClose();
} finally {
MDC.clear();
}
}
#Override
public void onCancel() {
MDC.put(MDC_DATA_KEY, mdcData);
try {
super.onCancel();
} finally {
MDC.clear();
}
}
#Override
public void onComplete() {
MDC.put(MDC_DATA_KEY, mdcData);
try {
super.onComplete();
} finally {
MDC.clear();
}
}
#Override
public void onReady() {
MDC.put(MDC_DATA_KEY, mdcData);
try {
super.onReady();
} finally {
MDC.clear();
}
}
}
}

Resources