I have the following code:
public Flux<Offer> getAllFilteredOffers(Map<String, String> searchParams) {
Flux<ProductProperties> productProperties = productPropertiesService.findProductPropertiesBySearchCriteria(searchParams);
Flux<Product> products = productService.findProductsByPropertyId(productProperties);
Flux<Product> productsByAvailability = productService.getAllProductsByAvailability(products, searchParams);
Flux<Offer> offers = offerRepository.findByPropertiesIds(productsByAvailability);
return offers;
This method:
productService.getAllProductsByAvailability(products, searchParams);
looks like:
public Flux<Product> getAllProductsByAvailability(Flux<Product> products,
Map<String, String> searchParams) {
How to pass List<Product> to getAllProductsByAvailability to keep non-blocking operations?
I've read that map is blocking and should be avoided.
Maybe something like that?
Flux
.just(productPropertiesService.findProductPropertiesBySearchCriteria(searchParams))
.flatMap(productProperties -> productService.findProductsByPropertyId(productProperties))
.flatMap(products -> productService.getAllProductsByAvailability(Flux.create(products)?????????, searchParams))
???
I'm not expert in Webflux, currently I'm trying to figure out how to handle problems like: I have Flux but in a second step I need to pull some data from the previous Flex<> object - keeping non-blocking stream.
Than you!
I don't know where you read about map, but if you look at the official documenation Webflux map operator there is nothing about blocking, it just uses synchronous function to each item.
Use this code:
productPropertiesService.findProductPropertiesBySearchCriteria(searchParams)
.flatMap(productProperties -> productService.findProductsByPropertyId(productProperties))
.collectList() (1)
.flatMapMany(products -> productService.getAllProductsByAvailability(Flux.fromIterable(products), searchParams)) (2)
1) collect all elements to List and convert to Mono>
2) create FLux from List and provide it as a parameter, flatMapMany transform Mono to Flux
Related
I'm new to Spring Reactive programming and I'm developing a REST endpoint that returns a Flux. For example:
#PostMapping
public Flux<MyResponse> processRequests(#RequestBody List<MyRequest> requests) {
return Flux.merge(Arrays.asList(dataSource.processRequest(requests.get(0)), dataSource2.processRequest(requests.get(0)))).parallel()
.runOn(Schedulers.elastic()).sequential();
}
Each data souce (dataSource and dataSource2) in the example code implements an interface that looks like this:
public interface MyResponseAdapter {
Flux<MyResponse> processRequest(MyRequest request);
}
This code works fine in that it returns the Flux as expected, but as you can see, the code only references the first element in the list of MyRequest. What I need to do is construct the Flux.merge for each element in the list of MyRequest. Can anyone point my in the right direction?
I think I've identified a simple solution:
List<Flux<MyResponse>> results = new ArrayList<>();
for (MyRequest myRequest : requests ) {
results.add(dataSource.processRequest(myRequest));
results.add(dataSource2.processRequest(myRequest));
}
return Flux.merge(results).parallel().runOn(Schedulers.elastic()).sequential();
I'm working on a Spring Boot (2.2) project using Kotlin, with CouchDB as (reactive) database, and in consequence, async DAO (either suspend functions, or functions returning a Flow). I'm trying to setup WebFlux in order to have async controllers too (again, I want to return Flows, not Flux). But I'm having troubles retrieving my security context from ReactiveSecurityContextHolder.
From what I've read, unlike SecurityContextHolder which is using ThreadLocal to store it, ReactiveSecurityContextHolder relies on the fact that Spring, while making a subscription to my reactive chain, also stored that context inside this chain, thus allowing me to call ReactiveSecurityContextHolder.getContext() from within the chain.
The problem is that I have to transform my Mono<SecurityContext> into a Flow at some point, which makes me loose my SecurityContext. So my question is: is there a way to have a Spring Boot controller returning a Flow while retrieving the security context from ReactiveSecurityContextHolder inside my logic? Basically, after simplification, it should look like this:
#GetMapping
fun getArticles(): Flow<String> {
return ReactiveSecurityContextHolder.getContext().flux().asFlow() // returns nothing
}
Note that if I return the Flux directly (skipping the .asFlow()), or add a .single() or .toList() in the end (hence using a suspend fun), then it works fine and my security context is returned, but again that's not what I want. I guess the solution would be to transfer the context from the Flux (initial reactive chain from ReactiveSecurityContextHolder) to the Flow, but it doesn't seem to be done by default.
Edit: here is a sample project showcasing the problem: https://github.com/Simon3/webflux-kotlin-sample
What you really try to achieve is accessing your ReactorContext from inside a Flow.
One way to do this is to relax the need for returning a Flow and return a Flux instead. This allows you to recover the ReactorContext and pass it to the Flow you are going to use to generate your data.
#ExperimentalCoroutinesApi
#GetMapping("/flow")
fun flow(): Flux<Map<String, String>> = Mono.subscriberContext().flatMapMany { reactorCtx ->
flow {
val ctx = coroutineContext[ReactorContext.Key]?.context?.get<Mono<SecurityContext>>(SecurityContext::class.java)?.asFlow()?.single()
emit(mapOf("user" to ((ctx?.authentication?.principal as? User)?.username ?: "<NONE>")))
}.flowOn(reactorCtx.asCoroutineContext()).asFlux()
}
In the case when you need to access the ReactorContext from a suspend method, you can simply get it back from the coroutineContext with no further artifice:
#ExperimentalCoroutinesApi
#GetMapping("/suspend")
suspend fun suspend(): Map<String,String> {
val ctx = coroutineContext[ReactorContext.Key]?.context?.get<Mono<SecurityContext>>(SecurityContext::class.java)?.asFlow()?.single()
return mapOf("user" to ((ctx?.authentication?.principal as? User)?.username ?: "<NONE>"))
}
I have a simple setup of server and client:
Flux.range(1, 5000)
.subscribeOn(Schedulers.parallel())
.flatMap(i -> WebClient.create()
.method(HttpMethod.POST)
.uri("http://localhost:8080/test")
.body(Mono.just(String.valueOf(i)), String.class)
.exchange())
.publishOn(Schedulers.parallel())
.subscribe(response ->
response.bodyToMono(String.class)
.publishOn(Schedulers.elastic())
.subscribe(body -> log.info("{}", body)));
here is the client:
#PostMapping
public Mono<String> test(#RequestBody Mono<String> body) {
return body.delayElement(Duration.ofSeconds(5));
}
Both things run on netty. Maybe someone has an idea what is causing this behavior?
This is not due to a WebClient limitation about connection pools, but this actually comes from a Reactor implementation details that you can change.
By default, Reactor operators such as flatMap have prefetch=32 (the number of elements we request before the end subscriber asks for those) and maxConcurrency=256 (the maximum number of elements processed concurrently by the operator).
You can use variants of Flux.flatMap(Function mapper, int concurrency, int prefetch) to change that behavior.
Your code snippet is using a mix of subscribeOn and publishOn; I'd say that given you're doing reactive I/O work with this code snippet, you shouldn't try to schedule work on an elastic/parallel scheduler. Removing those operators is best here.
I have a straightforward use case. This is to make a rest call, query mongo and then return an arbitrarily large stream of data back to the client, all with reactive streams type back pressure management.
This was quite easy to achieve using Spring WebFlux and Reactor. I am now trying to achieve the same goal using vert.x, as a comparison of ease of implementation.
Having found the vert.x mongo client to be lacking any support for managing back pressure, I am now attempting to use the WebFlux mongo client and then pump the data back through the vert.x HttpResponse, as shown in the following code:
public class MyMongoVerticle extends AbstractVerticle {
ReactiveMongoOperations operations;
public void start() throws Exception {
final Router router = Router.router(vertx);
router.route().handler(BodyHandler.create());
router.get("/myUrl").handler(ctx -> {
// WebFlux mongo operations returns a ReactiveStreams compatible entity
Flux<Document> mongoStream = operations.findAll(Document.class, "myCollection");
ReactiveReadStream rrs = ReactiveReadStream.readStream();
// rrs is ReactiveStream streams subscriber
mongoStream.subscribe(rrs);
// Pump pumps the rrs (ReactiveReadStream) to the HttpServerResponse (ReactiveWriteStream)
Pump pump = Pump.pump(rrs, ctx.response());
pump.start();
});
vertx.createHttpServer().requestHandler(router::accept).listen(8777);
}
}
The issue I have encountered is that the HttpServerResponse implements ReactiveWriteStream<Buffer> so is expecting a Buffer rather than a stream of Document's. The result is a ClassCaseException.
The question I have is how can I convert this stream of Documents into a into a ReactiveWriteStream<Buffer>? There may be another better way to do this, so I'm open to other suggestions on how to achieve this.
Pump won't work for you, as it doesn't support transformations currently. You'll have to implement pump by yourself. Luckily, this shouldn't be too hard:
Flux<Document> mongoStream = operations.findAll(Document.class, "myCollection");
ReactiveReadStream<Document> rrs = ReactiveReadStream.readStream();
mongoStream.subscribe(rrs);
HttpServerResponse outStream = ctx.response();
// Changes start here
rrs.handler(d -> {
if (outStream.writeQueueFull()) {
outStream.drainHandler((s) -> {
rrs.resume();
});
rrs.pause();
}
else {
outStream.write(d.toJson());
}
}).endHandler(h -> {
outStream.end();
});
Note that I wouldn't expect this to be more effective than "native" WebFlux implementation.
Also, JSON in this example will be mangled, as I don't wrap it in proper JSON Array
I'm trying to understand reactive part of spring 5. I have created simple rest endpoint for finding all entities using spring web-flux and spring data reactive (mongo) but don't see any way how to implement pagination.
Here is my simple example in Kotlin:
#GetMapping("/posts/")
fun getAllPosts() = postRepository.findAll()
Does it mean that reactive endpoint does not require pagination? Is some way to implement pagination from server side using this stack?
The reactive support in Spring Data does not provide means of a Page return type. Still, the Pageable parameter is supported in method signatures passing on limit and offset to the drivers and therefore the store itself, returning a Flux<T> that emits the range requested.
Flux<Person> findByFirstname(String firstname, Pageable pageable);
For more information please have a look at the current Reference Documentation for 2.0.RC2 and the Spring Data Examples.
Flux provides skip and take methods to get pagination support, and you also can use filter and sort to filter and sort the result. The filter and sort below is not a good example, but use skip and Pageable as 2nd parameter are no different.
The following codes work for me.
#GetMapping("")
public Flux<Post> all(
//#RequestParam(value = "q", required = false) String q,
#RequestParam(value = "page", defaultValue = "0") long page,
#RequestParam(value = "size", defaultValue = "10") long size) {
return this.postRepository.findAll()
//.filter(p -> Optional.ofNullable(q).map(key -> p.getTitle().contains(key) || p.getContent().contains(key)).orElse(true))//(replace this with query parameters)
.sort(comparing(Post::getCreatedDate).reversed())
.skip(page * size).take(size);
}
Update: The underlay drivers should be responsible for handling the result in the reactivestreams way.
And as you see in the answer from Christoph, if using a findByXXX method, Spring Data Mongo Reactive provides a variant to accept a pageable argument, but the findAll(reactive version) does not include such a variant, you have to do skip in the later operations if you really need the pagination feature. When switching to Flux instead of List, imagine the data in Flux as living water in the rivers or oil in the pipes, or the tweets in twitter.com.
I have tried to compare the queries using Pageale and not in the following case.
this.postRepository.findByTitleContains("title")
.skip(0)
.limitRequest(10)
.sort((o1, o2) -> o1.getTitle().compareTo(o2.getTitle()))
this.postRepository.findByTitleContains("title", PageRequest.of(0, 10, Sort.by(Sort.Direction.ASC, "title")))
When enabling logging for logging.level.org.springframework.data.mongodb.core.ReactiveMongoTemplate=DEBUG and found they print the same log for queries.
find using query: { "title" : { "$regularExpression" : { "pattern" : ".*title.*", "options" : ""}}} fields: Document{{title=1}} for class: class com.example.demo.Post in collection: post
//other logging...
find using query: { "title" : { "$regularExpression" : { "pattern" : ".*title.*", "options" : ""}}} fields: Document{{title=1}} for class: class com.example.demo.Post in collection: post
Keep in mind, all these operations should be DELEGATED to the underlay R2dbc drivers which implemented the reactive streams spec and performed on the DB side, NOT in the memory of your application side.
Check the example codes.
The early sample code I provided above maybe is not a good sample of filter and sort operations(MongoDB itself provides great regularexpression operations for it). But pagination in the reactive variant is not a good match with the concept in the reactive stream spec. When embracing Spring reactive stack, most of the time, we just move our work to a new collection of APIs. In my opinion, the realtime update and elastic response scene could be better match Reactive, eg. using it with SSE, Websocket, RSocket, application/stream+json(missing in the new Spring docs) protocols, etc
This is not efficient but it works for me while I look for another solution
Service
public Page<Level> getPage(int page, int size, Sort.Direction direction, String properties) {
var pageRequest = PageRequest.of(page, size, direction, properties);
var count = levelRepository.count().block();
var levels = levelRepository.findAllLevelsPaged(pageRequest).collectList().block();
return new PageImpl<>(Objects.requireNonNull(levels), pageRequest, Objects.requireNonNull(count));
}
Repo
#Repository
public interface LevelRepository extends ReactiveMongoRepository<Level, String> {
#Query("{ id: { $exists: true }}")
Flux<Level> findAllLevelsPaged(final Pageable page);
}
Ref example