Error with a JWT Token Reactive Authentication - spring

I am learning about reactive programming and spring-webflux and I am trying to implement a JWT Token Authorization, but I get a runtime error about the ResponseEntity:
open class JwtReactiveAuthenticationManager(
private val userDetailsService: ReactiveUserDetailsService,
private val passwordEncoder: PasswordEncoder
): ReactiveAuthenticationManager {
private val log: Logger = LoggerFactory.getLogger(this.javaClass)
override fun authenticate(authentication: Authentication): Mono<Authentication> {
return if (authentication.isAuthenticated) {
Mono.just(authentication)
} else Mono.just(authentication)
.switchIfEmpty(Mono.error(BadCredentialsException("Invalid Credentials")))
.cast(UsernamePasswordAuthenticationToken::class.java)
.flatMap { authenticationToken: UsernamePasswordAuthenticationToken -> authenticateToken(authenticationToken) }
.publishOn(Schedulers.parallel())
.onErrorResume { Mono.error(BadCredentialsException("Invalid Credentials")) }
.filter { u: UserDetails -> passwordEncoder.matches(authentication.credentials as String, u.password) }
.switchIfEmpty(Mono.error(BadCredentialsException("Invalid Credentials")))
.map { u: UserDetails -> UsernamePasswordAuthenticationToken(authentication.principal, authentication.credentials, u.authorities) }
}
private fun authenticateToken(authenticationToken: UsernamePasswordAuthenticationToken): Mono<UserDetails>? {
val username = authenticationToken.name
log.info("checking authentication for user $username")
if (username != null && SecurityContextHolder.getContext().authentication == null) {
log.info("authenticated user $username, setting security context")
return this.userDetailsService.findByUsername(username)
}
return null
}
}
Controller method invoked:
#GetMapping("")
override fun list(): Flux<ResponseEntity<Exercise>> {
return repo.findAll().map { o -> ResponseEntity(o, HttpStatus.OK) }
}
2020-01-09 17:54:15.790 ERROR 14680 --- [oundedElastic-1] a.w.r.e.AbstractErrorWebExceptionHandler : [e22e00ce] 500 Server Error for HTTP GET "/api/exercises"
java.lang.IllegalArgumentException: Only a single ResponseEntity supported
at org.springframework.util.Assert.isTrue(Assert.java:118) ~[spring-core-5.2.2.RELEASE.jar:5.2.2.RELEASE]
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
|_ checkpoint ? org.springframework.security.web.server.authentication.AuthenticationWebFilter [DefaultWebFilterChain]
|_ checkpoint ? org.springframework.security.web.server.authorization.AuthorizationWebFilter [DefaultWebFilterChain]
|_ checkpoint ? org.springframework.security.web.server.authentication.AuthenticationWebFilter [DefaultWebFilterChain]
|_ checkpoint ? org.springframework.security.web.server.authorization.ExceptionTranslationWebFilter [DefaultWebFilterChain]
|_ checkpoint ? org.springframework.security.web.server.savedrequest.ServerRequestCacheWebFilter [DefaultWebFilterChain]
|_ checkpoint ? org.springframework.security.web.server.context.SecurityContextServerWebExchangeWebFilter [DefaultWebFilterChain]
|_ checkpoint ? org.springframework.security.web.server.context.ReactorContextWebFilter [DefaultWebFilterChain]
|_ checkpoint ? org.springframework.security.web.server.header.HttpHeaderWriterWebFilter [DefaultWebFilterChain]
|_ checkpoint ? org.springframework.security.config.web.server.ServerHttpSecurity$ServerWebExchangeReactorContextWebFilter [DefaultWebFilterChain]
|_ checkpoint ? org.springframework.security.web.server.WebFilterChainProxy [DefaultWebFilterChain]
|_ checkpoint ? org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain]
|_ checkpoint ? HTTP GET "/api/exercises" [ExceptionHandlingWebHandler]
Stack trace:
at org.springframework.util.Assert.isTrue(Assert.java:118) ~[spring-core-5.2.2.RELEASE.jar:5.2.2.RELEASE]
at org.springframework.web.reactive.result.method.annotation.ResponseEntityResultHandler.handleResult(ResponseEntityResultHandler.java:121) ~[spring-webflux-5.2.2.RELEASE.jar:5.2.2.RELEASE]
at org.springframework.web.reactive.DispatcherHandler.handleResult(DispatcherHandler.java:169) ~[spring-webflux-5.2.2.RELEASE.jar:5.2.2.RELEASE]
at org.springframework.web.reactive.DispatcherHandler.lambda$handle$2(DispatcherHandler.java:147) ~[spring-webflux-5.2.2.RELEASE.jar:5.2.2.RELEASE]
at org.springframework.web.reactive.DispatcherHandler$$Lambda$1353.000000001B373A70.apply(Unknown Source) ~[na:na]
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:118) ~[reactor-core-3.3.1.RELEASE.jar:3.3.1.RELEASE]
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1630) ~[reactor-core-3.3.1.RELEASE.jar:3.3.1.RELEASE]
at reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:241) ~[reactor-core-3.3.1.RELEASE.jar:3.3.1.RELEASE]

you can't return Flux<ResponseEntity<Exercise>> as in multiple response entities.
change it to:
#GetMapping("")
override fun list(): Flux<Exercise> {
return repo.findAll()
}
if you wish to stream the data to the calling client, and it will probably go a lot better.

Related

syn request in webclient cannot use block for Mono in springboot

I am new to reactive and web client and i just doing a project in springboot microsevice.While when setting the ReactiveUserDetail Service which require the use of mono.I find that my synchronous request by webclient give an error.I suspect this is due to the "block" ,but it seems that i cannot eliminate it.
Error:
java.lang.IllegalStateException: block()/blockFirst()/blockLast() are blocking, which is not supported in thread reactor-http-epoll-3
at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:83) ~[reactor-core-3.5.1.jar:3.5.1]
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
*__checkpoint ⇢ org.springframework.security.web.server.authentication.AuthenticationWebFilter [DefaultWebFilterChain]
*__checkpoint ⇢ org.springframework.security.web.server.context.ReactorContextWebFilter [DefaultWebFilterChain]
*__checkpoint ⇢ org.springframework.security.web.server.header.HttpHeaderWriterWebFilter [DefaultWebFilterChain]
*__checkpoint ⇢ org.springframework.security.config.web.server.ServerHttpSecurity$ServerWebExchangeReactorContextWebFilter [DefaultWebFilterChain]
*__checkpoint ⇢ org.springframework.security.web.server.WebFilterChainProxy [DefaultWebFilterChain]
*__checkpoint ⇢ HTTP POST "/signin" [ExceptionHandlingWebHandler]
Original Stack Trace:
at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:83) ~[reactor-core-3.5.1.jar:3.5.1]
at reactor.core.publisher.Mono.block(Mono.java:1710) ~[reactor-core-3.5.1.jar:3.5.1]
My code:
#Service
public class SecurityUserService implements ReactiveUserDetailsService {
#Autowired
private WebClient.Builder webClientBuilder;
#Override
public Mono<UserDetails> findByUsername(String username) {
UserAuthdto result= webClientBuilder.baseUrl("http://USER").build().get() //make syn request
.uri(uriBuilder -> uriBuilder
.path("/User/AuthUser/{username}")
.build(username))
.retrieve()
.bodyToMono(UserAuthdto.class)
.block()
;
//map the userresult into securityuser
if(result!=null){
return Mono.just(new SecurityUser(result));
}
//no userfound
return null;
}
}
Entry point from the other microservice:
#GetMapping("/AuthUser/{username}")
public UserAuthdto getSecurityUser(#PathVariable String username){
return userCoreservice.getSecurityUser(username);
}
Web client bean:
#Bean
public WebClient webClient(){
return WebClient.builder().build();
}
Please help

"could not resolve view" with `#PreAuthorize` and `WebTestClient`

I'm getting this rather weird bug in a test. My controller has the following method:
#Operation(
summary = "Add a new quote",
operationId = "v1AddQuote",
description = "",
responses = [
ApiResponse(responseCode = "201", description = "Created")
]
)
#RequestMapping(
method = [RequestMethod.POST],
value = ["/quote"],
consumes = ["application/json"]
)
#PreAuthorize("hasRole('ADMIN')")
suspend fun v1AddQuote(quoteDto: QuoteDto): ResponseEntity<Unit> {
val entity = quoteRepository.insert(quoteMapper.dtoToEntity(quoteDto))
.awaitSingleOrNull() ?: return ResponseEntity.internalServerError().build()
return ResponseEntity.created(URI.create("/quote/${entity.id}")).build()
}
And I'm testing it with WebTestClient with:
webClient
.mutateWith(mockUser().roles("ADMIN"))
.post()
.uri("/quote")
.contentType(MediaType.APPLICATION_JSON)
.body(Mono.just(dto), QuoteDto::class.java)
.exchange()
.expectStatus()
.isCreated
.expectHeader()
.location("/quote/${entity.id}")
Surprisingly, this fails with:
java.lang.IllegalStateException: Could not resolve view with name 'quote'
Where does this even come from? Some details:
This doesn't happen when doing the same request in a deployed application, only in the test
Just removing #PreAuthorize solves it
The controller method actually runs, the error appears after it has run
Looking at the stack trace it makes me think this is some sort of redirection Spring Security does?
*__checkpoint ⇢ Handler jdk.proxy2.$Proxy126#v1AddQuote(QuoteDto, Continuation) [DispatcherHandler]
*__checkpoint ⇢ org.springframework.security.web.server.authorization.AuthorizationWebFilter [DefaultWebFilterChain]
*__checkpoint ⇢ org.springframework.security.web.server.authorization.ExceptionTranslationWebFilter [DefaultWebFilterChain]
*__checkpoint ⇢ org.springframework.security.web.server.savedrequest.ServerRequestCacheWebFilter [DefaultWebFilterChain]
*__checkpoint ⇢ org.springframework.security.web.server.context.SecurityContextServerWebExchangeWebFilter [DefaultWebFilterChain]
*__checkpoint ⇢ org.springframework.security.web.server.context.ReactorContextWebFilter [DefaultWebFilterChain]
*__checkpoint ⇢ org.springframework.security.web.server.header.HttpHeaderWriterWebFilter [DefaultWebFilterChain]
*__checkpoint ⇢ org.springframework.security.config.web.server.ServerHttpSecurity$ServerWebExchangeReactorContextWebFilter [DefaultWebFilterChain]
*__checkpoint ⇢ org.springframework.security.web.server.WebFilterChainProxy [DefaultWebFilterChain]
*__checkpoint ⇢ org.springframework.security.test.web.reactive.server.SecurityMockServerConfigurers$MutatorFilter [DefaultWebFilterChain]
*__checkpoint ⇢ org.springframework.security.test.web.reactive.server.SecurityMockServerConfigurers$SetupMutatorFilter [DefaultWebFilterChain]
*__checkpoint ⇢ HTTP POST "/quote" [ExceptionHandlingWebHandler]
In case it's necessary, my Spring Security configuration is:
#Configuration
#EnableWebFluxSecurity
#EnableReactiveMethodSecurity
class WebFluxSecurityConfig {
#Bean
fun userDetailsService(): ReactiveUserDetailsService {
val userDetails = User.withDefaultPasswordEncoder()
.username("admin")
.password("admin")
.roles("ADMIN")
.build()
return MapReactiveUserDetailsService(userDetails)
}
#Bean
fun springSecurityFilterChain(http: ServerHttpSecurity): SecurityWebFilterChain {
return http.invoke {
authorizeExchange {
authorize(anyExchange, permitAll)
}
csrf { disable() }
httpBasic { }
formLogin { disable() }
logout { disable() }
}
}
}
The problem was actually mostly unrelated to any of the info I gave on the post, and actually solved it almost by pure chance. The issue is that I was inheriting an OpenAPI-generated interface in my controller. I don't know why this gives such a weird error, but the solution is, apparently, to add this to your security #EnableReactiveMethodSecurity annotation:
#EnableReactiveMethodSecurity(proxyTargetClass = true)
Without the proxyTargetClass = true parameter, stuff breaks in weird ways.

How to migrate the GlobalMethodSecurityConfiguration to Reactive Spring?

I have overridden SecurityExpressionRoot in my project exposing a method verifying whether the current user has rights to a given resource.
Then I have overriden GlobalMethodSecurityComfiguration.createExpressionHandler() and then createSecurityExpressionRoot() returning the instance of the overriden SecurityExpressionRoot.
This has worked in a servlet scenario, unfortunately, this doesn't seem to work in a reactive scenario.
How do I convert the method security setup below to the reactive scenario?
In my tests I get the following error:
org.springframework.security.authentication.AuthenticationCredentialsNotFoundException: An Authentication object was not found in the SecurityContext
at org.springframework.security.access.intercept.AbstractSecurityInterceptor.credentialsNotFound(AbstractSecurityInterceptor.java:379) ~[spring-security-core-5.2.1.RELEASE.jar:5.2.1.RELEASE]
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
|_ checkpoint ⇢ org.springframework.security.web.server.authorization.AuthorizationWebFilter [DefaultWebFilterChain]
|_ checkpoint ⇢ org.springframework.security.web.server.authorization.ExceptionTranslationWebFilter [DefaultWebFilterChain]
|_ checkpoint ⇢ org.springframework.security.web.server.authentication.logout.LogoutWebFilter [DefaultWebFilterChain]
|_ checkpoint ⇢ org.springframework.security.web.server.savedrequest.ServerRequestCacheWebFilter [DefaultWebFilterChain]
|_ checkpoint ⇢ org.springframework.security.web.server.context.SecurityContextServerWebExchangeWebFilter [DefaultWebFilterChain]
|_ checkpoint ⇢ org.springframework.security.web.server.ui.LogoutPageGeneratingWebFilter [DefaultWebFilterChain]
|_ checkpoint ⇢ org.springframework.security.web.server.ui.LoginPageGeneratingWebFilter [DefaultWebFilterChain]
|_ checkpoint ⇢ org.springframework.security.web.server.authentication.AuthenticationWebFilter [DefaultWebFilterChain]
|_ checkpoint ⇢ org.springframework.security.web.server.authentication.AuthenticationWebFilter [DefaultWebFilterChain]
|_ checkpoint ⇢ org.springframework.security.web.server.context.ReactorContextWebFilter [DefaultWebFilterChain]
|_ checkpoint ⇢ org.springframework.security.web.server.csrf.CsrfWebFilter [DefaultWebFilterChain]
|_ checkpoint ⇢ org.springframework.security.web.server.header.HttpHeaderWriterWebFilter [DefaultWebFilterChain]
|_ checkpoint ⇢ org.springframework.security.config.web.server.ServerHttpSecurity$ServerWebExchangeReactorContextWebFilter [DefaultWebFilterChain]
|_ checkpoint ⇢ org.springframework.security.web.server.WebFilterChainProxy [DefaultWebFilterChain]
|_ checkpoint ⇢ org.springframework.security.test.web.reactive.server.SecurityMockServerConfigurers$MutatorFilter [DefaultWebFilterChain]
|_ checkpoint ⇢ HTTP GET "/things/1/jobs/1/log" [ExceptionHandlingWebHandler]
#Configuration
#EnableGlobalMethodSecurity(prePostEnabled = true)
#RequiredArgsConstructor
public class MethodSecurityConfig extends GlobalMethodSecurityConfiguration {
private final ThingsRepository thingsRepository;
#Override
protected MethodSecurityExpressionHandler createExpressionHandler() {
return new DefaultMethodSecurityExpressionHandler() {
#Override
protected MethodSecurityExpressionOperations createSecurityExpressionRoot(Authentication authentication, MethodInvocation invocation) {
DeployPermissionSecurityExpressionRoot root = new ThingsPermissionSecurityExpressionRoot(thingsRepository, authentication);
root.setThis(invocation.getThis());
root.setPermissionEvaluator(getPermissionEvaluator());
root.setTrustResolver(getTrustResolver());
root.setRoleHierarchy(getRoleHierarchy());
return root;
}
};
}
My security config:
#Bean
public SecurityWebFilterChain securityWebFilterChain(ServerHttpSecurity http) {
// #formatter:off
http
.csrf().disable()
.authorizeExchange()
.anyExchange().authenticated()
.and()
.oauth2ResourceServer()
.jwt(jwt ->{
jwt.jwtDecoder(jwtDecoder());
jwt.jwtAuthenticationConverter(customJwtAuthConverter());
})
.and()
.securityContextRepository(NoOpServerSecurityContextRepository.getInstance());
// #formatter:on
return http.build();
}
SecurityExpressionRoot implementation:
class ThingsPermissionSecurityExpressionRoot extends SecurityExpressionRoot implements MethodSecurityExpressionOperations {
private final ThingsRepository ThingsRepository;
private Object filterObject;
private Object returnObject;
private Object target;
ThingsPermissionSecurityExpressionRoot(ThingsRepository thingsRepository, Authentication authentication) {
super(authentication);
this.thingsRepository = thingsRepository;
}
public boolean hasThingsWritePrivilege(Long thingsId) {
Controller:
public class ThingsController {
private final JobPublisherProvider jobPublisherProvider;
#GetMapping("{thingsId}/jobs/{jobId}/log")
#PreAuthorize("hasThingsWritePrivilege(#thingsId)")
public Flux<DataBuffer> retrieveJobLog(#PathVariable String thingsId, #PathVariable int jobId) {
Controller test method
#Test
#WithMockUser(roles = {"ROLE_JAR_W"})
public void logValueProperlyRetrieved() {
It is not possible to define GlobalSecurityConfiguration in a reactive Spring application (current Spring Security version 5.5.2).
In order to have the same functionality, you need to replace methodSecurityExpressionHandler defined in ReactiveMethodSecurityConfiguration with your own method security expression handler.
To do that, you can extend DefaultMethodSecurityExpressionHandler and define it as #Primary bean.
For your case, it is going to be as following.
public class CustomMethodSecurityExpressionHandler extends DefaultMethodSecurityExpressionHandler {
#Override
protected MethodSecurityExpressionOperations createSecurityExpressionRoot(Authentication authentication, MethodInvocation invocation) {
DeployPermissionSecurityExpressionRoot root = new ThingsPermissionSecurityExpressionRoot(thingsRepository, authentication);
root.setThis(invocation.getThis());
root.setPermissionEvaluator(getPermissionEvaluator());
root.setTrustResolver(getTrustResolver());
root.setRoleHierarchy(getRoleHierarchy());
root.setDefaultRolePrefix(getDefaultRolePrefix());
return root;
}
}
#Bean
#Primary
public MethodSecurityExpressionHandler methodSecurityExpressionHandler() {
return new CustomMethodSecurityExpressionHandler();
}
References:
https://github.com/spring-projects/spring-security/issues/5046#issuecomment-427097710

onErrorResume() won't get called

I'm newbie to spring webflux and reactor I want to have a fallback mechanism when some specific exception occurs and based on my research onErrorResume method does that but it won't get called and I get 500 internal server error instead of my fallback triggered and preventing this error.
Note: I use spring webflux which means it made some changes in the normal behavior of the reactor project
public Mono<Date> getExpirationDateFromToken(String token) {
return getAllClaimsFromToken(token)
.onErrorResume(ExpiredJwtException.class, e -> Mono.just(e.getClaims()))
.map(Claims::getExpiration);
}
public Mono<Claims> getAllClaimsFromToken(String token) {
return Mono.just(Jwts.parserBuilder()
.setSigningKey(Base64.getEncoder().encodeToString(secret.getBytes()))
.build()
.parseClaimsJws(token)
.getBody());
}
and this is the stack trace
io.jsonwebtoken.ExpiredJwtException: JWT expired at 2020-03-22T08:17:15Z. Current time: 2020-03-22T08:50:56Z, a difference of 2021835 milliseconds. Allowed clock skew: 0 milliseconds.
at io.jsonwebtoken.impl.DefaultJwtParser.parse(DefaultJwtParser.java:439) ~[jjwt-impl-0.11.0.jar:0.11.0]
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
|_ checkpoint ⇢ Handler ir.siavash.customerservice.security.controller.AuthController#refreshToken(Mono) [DispatcherHandler]
|_ checkpoint ⇢ org.springframework.security.web.server.authorization.AuthorizationWebFilter [DefaultWebFilterChain]
|_ checkpoint ⇢ org.springframework.security.web.server.authorization.ExceptionTranslationWebFilter [DefaultWebFilterChain]
|_ checkpoint ⇢ org.springframework.security.web.server.authentication.logout.LogoutWebFilter [DefaultWebFilterChain]
|_ checkpoint ⇢ org.springframework.security.web.server.savedrequest.ServerRequestCacheWebFilter [DefaultWebFilterChain]
|_ checkpoint ⇢ org.springframework.security.web.server.context.SecurityContextServerWebExchangeWebFilter [DefaultWebFilterChain]
|_ checkpoint ⇢ org.springframework.security.web.server.context.ReactorContextWebFilter [DefaultWebFilterChain]
|_ checkpoint ⇢ org.springframework.security.web.server.header.HttpHeaderWriterWebFilter [DefaultWebFilterChain]
|_ checkpoint ⇢ org.springframework.security.config.web.server.ServerHttpSecurity$ServerWebExchangeReactorContextWebFilter [DefaultWebFilterChain]
|_ checkpoint ⇢ org.springframework.security.web.server.WebFilterChainProxy [DefaultWebFilterChain]
|_ checkpoint ⇢ org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain]
|_ checkpoint ⇢ HTTP POST "/auth/refreshToken" [ExceptionHandlingWebHandler]
Stack trace:
at io.jsonwebtoken.impl.DefaultJwtParser.parse(DefaultJwtParser.java:439) ~[jjwt-impl-0.11.0.jar:0.11.0]
at io.jsonwebtoken.impl.DefaultJwtParser.parse(DefaultJwtParser.java:541) ~[jjwt-impl-0.11.0.jar:0.11.0]
at io.jsonwebtoken.impl.DefaultJwtParser.parseClaimsJws(DefaultJwtParser.java:601) ~[jjwt-impl-0.11.0.jar:0.11.0]
at io.jsonwebtoken.impl.ImmutableJwtParser.parseClaimsJws(ImmutableJwtParser.java:173) ~[jjwt-impl-0.11.0.jar:0.11.0]
at ir.siavash.customerservice.security.JWTUtil.getAllClaimsFromToken(JWTUtil.java:42) ~[classes/:na]
Ok, On second look, it's clear. That's a common mistake. Reactive components could be difficult to learn, but you're not very far from the solution.
Mono.just takes a pre-computed value as parameter. Therefore, your getAllClaimsFromToken will create a Mono only after all code given to just is executed. That means that your error happens before any reactive component is available for error management (assembly time).
What you could do instead is using Mono#fromCallable to ask Reactor to compute Jwt token on demand, or creating Mono with just initial string, then mapping using your function.
Let's see second solution:
public Mono<Date> getExpirationDateFromToken(String token) {
return Mono.just(token)
.map(this::getAllClaimsFromToken)
.onErrorResume(ExpiredJwtException.class, e -> Mono.just(e.getClaims()))
.map(Claims::getExpiration);
}
public Claims getAllClaimsFromToken(String token) {
return Jwts.parserBuilder()
.setSigningKey(Base64.getEncoder().encodeToString(secret.getBytes()))
.build()
.parseClaimsJws(token)
.getBody();
}

configure spring.codec.max-in-memory-size When using ReactiveElasticsearchClient

I am using the ReactiveElasticsearchClient from spring-data-elasticsearch 3.2.3 with spring-boot 2.2.0. When upgrading to spring-boot 2.2.2 i have got org.springframework.core.io.buffer.DataBufferLimitException: Exceeded limit on max bytes to buffer : 262144.
It's indicated to fixe that to use spring.codec.max-in-memory-size but i still got the same exception.
Bellow the whole exception:
org.springframework.core.io.buffer.DataBufferLimitException: Exceeded limit on max bytes to buffer : 262144
at org.springframework.core.io.buffer.LimitedDataBufferList.raiseLimitException(LimitedDataBufferList.java:101)
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Assembly trace from producer [reactor.core.publisher.MonoCollect] :
reactor.core.publisher.Flux.collect(Flux.java:3273)
org.springframework.core.io.buffer.DataBufferUtils.join(DataBufferUtils.java:553)
Error has been observed at the following site(s):
|_ Flux.collect ⇢ at org.springframework.core.io.buffer.DataBufferUtils.join(DataBufferUtils.java:553)
|_ Mono.filter ⇢ at org.springframework.core.io.buffer.DataBufferUtils.join(DataBufferUtils.java:554)
|_ Mono.map ⇢ at org.springframework.core.io.buffer.DataBufferUtils.join(DataBufferUtils.java:555)
|_ Mono.map ⇢ at org.springframework.core.codec.AbstractDataBufferDecoder.decodeToMono(AbstractDataBufferDecoder.java:96)
|_ checkpoint ⇢ Body from POST http://localhost:9200/_bulk?timeout=1m [DefaultClientResponse]
|_ Mono.map ⇢ at org.springframework.data.elasticsearch.client.reactive.DefaultReactiveElasticsearchClient.readResponseBody(DefaultReactiveElasticsearchClient.java:669)
|_ Mono.doOnNext ⇢ at org.springframework.data.elasticsearch.client.reactive.DefaultReactiveElasticsearchClient.readResponseBody(DefaultReactiveElasticsearchClient.java:670)
|_ Mono.flatMap ⇢ at org.springframework.data.elasticsearch.client.reactive.DefaultReactiveElasticsearchClient.readResponseBody(DefaultReactiveElasticsearchClient.java:671)
|_ Mono.flatMapMany ⇢ at org.springframework.data.elasticsearch.client.reactive.DefaultReactiveElasticsearchClient.sendRequest(DefaultReactiveElasticsearchClient.java:591)
|_ Flux.publishNext ⇢ at org.springframework.data.elasticsearch.client.reactive.DefaultReactiveElasticsearchClient.bulk(DefaultReactiveElasticsearchClient.java:448)
|_ Flux.flatMap ⇢ at com.energisme.ds.reactive.aggregation.service.SensorAggregationService.save(SensorAggregationService.java:32)
|_ Flux.map ⇢ at com.energisme.ds.reactive.aggregation.service.SensorAggregationService.save(SensorAggregationService.java:33)
|_ Flux.reduce ⇢ at com.energisme.ds.reactive.aggregation.service.SensorAggregationService.save(SensorAggregationService.java:34)
|_ Mono.zip ⇢ at com.energisme.ds.reactive.aggregation.service.AggregateSensorFlowService.nonIndexDifferenceAggregateSensorData(AggregateSensorFlowService.java:178)
|_ Mono.map ⇢ at com.energisme.ds.reactive.aggregation.service.AggregateSensorFlowService.nonIndexDifferenceAggregateSensorData(AggregateSensorFlowService.java:179)
Stack trace:
at org.springframework.core.io.buffer.LimitedDataBufferList.raiseLimitException(LimitedDataBufferList.java:101)
at org.springframework.core.io.buffer.LimitedDataBufferList.updateCount(LimitedDataBufferList.java:94)
at org.springframework.core.io.buffer.LimitedDataBufferList.add(LimitedDataBufferList.java:59)
at reactor.core.publisher.MonoCollect$CollectSubscriber.onNext(MonoCollect.java:119)
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:121)
at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:203)
at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:203)
at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114)
at reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:218)
at reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:351)
at reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:348)
at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:571)
at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:89)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:326)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:313)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:427)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:281)
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:502)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407)
at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
Can anyone tell me what i am doing wrong or is that a bug?
Thank you
Using the plain reaction WebClient I ran into the same issue (going from 2.1.9 to 2.2.1.) I had no luck setting spring.codec.max-in-memory-size and later found a hint that this wasn't the way to go anyway:
… On the client side, the limit can be changed in WebClient.Builder.
(source, including dead link :-S )
I still haven't found out where WebClient.Builder gets the default 256K limit1. However, the following enabled me to raise the buffer size limit to 16M:
WebClient.builder()
.…
.exchangeStrategies(ExchangeStrategies.builder()
.codecs(configurer -> configurer
.defaultCodecs()
.maxInMemorySize(16 * 1024 * 1024))
.build())
.build();
So, it seems to me (without knowing the intricacies of spring-data-elasticsearch) that if you can somehow get your hands on the WebClient as returned from the WebClientProvider you should be able to mutate it to include the ExchangeStrategies from above.
Perhaps you can provide your own override of DefaultWebClientProvider along the lines of (absolutely untested!):
class MyDefaultWebClientProvider extends DefaultWebClientProvider {
#Override
public WebClient get(InetSocketAddress endpoint) {
return super.get(endpoint)
.mutate() // Obtain WebClient.Builder instance.
.exchangeStrategies(ExchangeStrategies.builder()
.codecs(configurer -> configurer
.defaultCodecs()
.maxInMemorySize(16 * 1024 * 1024))
.build())
.build();
}
}
YMMV.
UPDATE #1:
1) Now I found it. And it explains why setting spring.codec.max-in-memory-size has no effect; the property is hardcoded at 256K in the base class uses by all default codecs, cf. BaseDefaultCodecs.
A couple of days ago I implemented the possibility to customize the WebClient, check the corresponding Jira issue. This will be available in Spring Data Elasticsearch 3.2.4 and is already in the current master branch.
Configuration code looks like this:
#Configuration
public class ReactiveRestClientConfig extends AbstractReactiveElasticsearchConfiguration {
#Override
public ReactiveElasticsearchClient reactiveElasticsearchClient() {
final ClientConfiguration clientConfiguration = ClientConfiguration.builder() //
.connectedTo("localhost:9200") //
.withWebClientConfigurer(webClient -> {
ExchangeStrategies exchangeStrategies = ExchangeStrategies.builder()
.codecs(configurer -> configurer.defaultCodecs()
.maxInMemorySize(-1))
.build();
return webClient.mutate().exchangeStrategies(exchangeStrategies).build();
})
.build();
return ReactiveRestClients.create(clientConfiguration);
}
}
As of Spring Boot 2.3.0, there is now a dedicated configuration property for the Reactive Elasticsearch REST client.
You can use the following configuration property to set a specific memory limit for the client.
spring.data.elasticsearch.client.reactive.max-in-memory-size=
The already existing spring.codec.max-in-memory-size property is separate and only affects other WebClient instances in the application.
or:
final Consumer<ClientCodecConfigurer> consumer = configurer -> {
final ClientCodecConfigurer.ClientDefaultCodecs codecs = configurer.defaultCodecs();
codecs.maxInMemorySize(maxBufferMb * 1024 * 1024);
};
WebClient.builder().codecs(consumer).build();
In my case, using Spring boot 2.5.6, I had to use both to solve the issue
Created a configuration class;
#Configuration
public class WebfluxConfig implements WebFluxConfigurer {
#Override
public void configureHttpMessageCodecs(ServerCodecConfigurer configurer) {
configurer.defaultCodecs().maxInMemorySize(5000 * 1024);
}
#Bean("webClient")
public WebClient getSelfWebClient(WebClient.Builder builder) {
return builder.baseUrl("url").build();
}
}
in the .properties file;
spring.codec.max-in-memory-size=5MB
Class where I use the WebClient;
#Autowired
#Qualifier("webClient")
private WebClient webClient;
private void doSomething() {
String response = webClient.post()
.uri(uri)
.accept(MediaType.APPLICATION_JSON)
.contentType(MediaType.APPLICATION_JSON)
.bodyValue(requestJson)
.retrieve()
.bodyToMono(String.class).block();
}
.withWebClientConfigurer is deprecated. Had to use .withClientConfigurer, which worked for me. Below is the code -
.withClientConfigurer(
ReactiveRestClients.WebClientConfigurationCallback.from(webClient -> {
ExchangeStrategies exchangeStrategies = ExchangeStrategies.builder()
.codecs(configurer -> configurer.defaultCodecs()
.maxInMemorySize(-1))
.build();
return webClient.mutate().exchangeStrategies(exchangeStrategies).build();
}))
Referernce

Resources