Is there any way to implement pagination in spring webflux and spring data reactive - spring

I'm trying to understand reactive part of spring 5. I have created simple rest endpoint for finding all entities using spring web-flux and spring data reactive (mongo) but don't see any way how to implement pagination.
Here is my simple example in Kotlin:
#GetMapping("/posts/")
fun getAllPosts() = postRepository.findAll()
Does it mean that reactive endpoint does not require pagination? Is some way to implement pagination from server side using this stack?

The reactive support in Spring Data does not provide means of a Page return type. Still, the Pageable parameter is supported in method signatures passing on limit and offset to the drivers and therefore the store itself, returning a Flux<T> that emits the range requested.
Flux<Person> findByFirstname(String firstname, Pageable pageable);
For more information please have a look at the current Reference Documentation for 2.0.RC2 and the Spring Data Examples.

Flux provides skip and take methods to get pagination support, and you also can use filter and sort to filter and sort the result. The filter and sort below is not a good example, but use skip and Pageable as 2nd parameter are no different.
The following codes work for me.
#GetMapping("")
public Flux<Post> all(
//#RequestParam(value = "q", required = false) String q,
#RequestParam(value = "page", defaultValue = "0") long page,
#RequestParam(value = "size", defaultValue = "10") long size) {
return this.postRepository.findAll()
//.filter(p -> Optional.ofNullable(q).map(key -> p.getTitle().contains(key) || p.getContent().contains(key)).orElse(true))//(replace this with query parameters)
.sort(comparing(Post::getCreatedDate).reversed())
.skip(page * size).take(size);
}
Update: The underlay drivers should be responsible for handling the result in the reactivestreams way.
And as you see in the answer from Christoph, if using a findByXXX method, Spring Data Mongo Reactive provides a variant to accept a pageable argument, but the findAll(reactive version) does not include such a variant, you have to do skip in the later operations if you really need the pagination feature. When switching to Flux instead of List, imagine the data in Flux as living water in the rivers or oil in the pipes, or the tweets in twitter.com.
I have tried to compare the queries using Pageale and not in the following case.
this.postRepository.findByTitleContains("title")
.skip(0)
.limitRequest(10)
.sort((o1, o2) -> o1.getTitle().compareTo(o2.getTitle()))
this.postRepository.findByTitleContains("title", PageRequest.of(0, 10, Sort.by(Sort.Direction.ASC, "title")))
When enabling logging for logging.level.org.springframework.data.mongodb.core.ReactiveMongoTemplate=DEBUG and found they print the same log for queries.
find using query: { "title" : { "$regularExpression" : { "pattern" : ".*title.*", "options" : ""}}} fields: Document{{title=1}} for class: class com.example.demo.Post in collection: post
//other logging...
find using query: { "title" : { "$regularExpression" : { "pattern" : ".*title.*", "options" : ""}}} fields: Document{{title=1}} for class: class com.example.demo.Post in collection: post
Keep in mind, all these operations should be DELEGATED to the underlay R2dbc drivers which implemented the reactive streams spec and performed on the DB side, NOT in the memory of your application side.
Check the example codes.
The early sample code I provided above maybe is not a good sample of filter and sort operations(MongoDB itself provides great regularexpression operations for it). But pagination in the reactive variant is not a good match with the concept in the reactive stream spec. When embracing Spring reactive stack, most of the time, we just move our work to a new collection of APIs. In my opinion, the realtime update and elastic response scene could be better match Reactive, eg. using it with SSE, Websocket, RSocket, application/stream+json(missing in the new Spring docs) protocols, etc

This is not efficient but it works for me while I look for another solution
Service
public Page<Level> getPage(int page, int size, Sort.Direction direction, String properties) {
var pageRequest = PageRequest.of(page, size, direction, properties);
var count = levelRepository.count().block();
var levels = levelRepository.findAllLevelsPaged(pageRequest).collectList().block();
return new PageImpl<>(Objects.requireNonNull(levels), pageRequest, Objects.requireNonNull(count));
}
Repo
#Repository
public interface LevelRepository extends ReactiveMongoRepository<Level, String> {
#Query("{ id: { $exists: true }}")
Flux<Level> findAllLevelsPaged(final Pageable page);
}
Ref example

Related

Spring WebFlux + Kotlin Response Handling

I'm having some trouble wrapping my head around a supposedly simple RESTful WS response handling scenario when using Spring WebFlux in combination with Kotlin coroutines. Suppose we have a simple WS method in our REST controller that is supposed to return a possibly huge number (millions) of response "things":
#GetMapping
suspend fun findAllThings(): Flow<Thing> {
//Reactive DB query, return a flow of things
}
This works as one would expect: the result is streamed to the client as long as a streaming media type (e.g. "application/x-ndjson") is used. In more complex service calls that also accounts for the possibility of errors/warnings I would like to return a response object of the following form:
class Response<T> {
val errors: Flow<String>
val things: Flow<T>
}
The idea here being that a response either is successful (returning an empty error Flow and a Flow of things), or failed (errors contained in the corresponding Flow while the things Flow being empty). In blocking programming this is a quite common response idiom. My question now is how can I adapt this idiom to the reactive approach in Kotlin/Spring WebFlux?
I know its possible to just return the Response as described (or Mono<Response> for Java users), but this somewhat defeats the purpose of being reactive as the entire Mono has to exist in memory at serialization time. Is there any way to solve this? The only possible solution I can think of right now is a custom Spring Encoder that is smart enough to stream both errors or things (whatever is present).
How about returning Success/Error per Thing?
class Result<T> private constructor(val result: T?, val error: String?) {
constructor(data: T) : this(data, null)
constructor(error: String) : this(null, error)
val isError = error != null
}
#GetMapping
suspend fun findAllThings(): Flow<Result<Thing>> {
//Reactive DB query, return a flow of things
}

Spring Reactive Programming: How to create a dynamic list of Publishers as input to Flux.merge

I'm new to Spring Reactive programming and I'm developing a REST endpoint that returns a Flux. For example:
#PostMapping
public Flux<MyResponse> processRequests(#RequestBody List<MyRequest> requests) {
return Flux.merge(Arrays.asList(dataSource.processRequest(requests.get(0)), dataSource2.processRequest(requests.get(0)))).parallel()
.runOn(Schedulers.elastic()).sequential();
}
Each data souce (dataSource and dataSource2) in the example code implements an interface that looks like this:
public interface MyResponseAdapter {
Flux<MyResponse> processRequest(MyRequest request);
}
This code works fine in that it returns the Flux as expected, but as you can see, the code only references the first element in the list of MyRequest. What I need to do is construct the Flux.merge for each element in the list of MyRequest. Can anyone point my in the right direction?
I think I've identified a simple solution:
List<Flux<MyResponse>> results = new ArrayList<>();
for (MyRequest myRequest : requests ) {
results.add(dataSource.processRequest(myRequest));
results.add(dataSource2.processRequest(myRequest));
}
return Flux.merge(results).parallel().runOn(Schedulers.elastic()).sequential();

How to limit number of results in ReactiveMongoRepository

I am looking for a way to pass the limit to the mongo query in ReactiveCrudRepository
I tried adding "First2" to the method name but I'm still getting all the results.
What I'm really looking for is a way to pass the value of the 'limit' to the method, passing it in request as #RequestParam int limit
This is my code for the repository
public interface ReactiveUserRepository
extends ReactiveCrudRepository<User, String> {
#Query("{ 'roles': ?0 }")
Flux<User> findFirst2ByRole(String role);
}
And this is controller method:
#GetMapping(path = "/byrole", produces = "application/stream+json")
Flux<User> getByRole(#RequestParam String role) {
return users.findFirst2ByRole(role).doOnNext(next -> {
System.out.println("Next user=" + next.getAssocId());
}).switchIfEmpty(Mono.error(new ResponseStatusException(HttpStatus.NOT_FOUND, String.format("No users found with role=%s", role))));
}
limitRequest(long) on a Flux may also be worth to have a look at: https://projectreactor.io/docs/core/release/api/reactor/core/publisher/Flux.html#limitRequest-long-
it can be used as a stricter form of take(long)
As it does cap the request amount that goes to the upstream source. This may prevent the reactive MongoDB driver from requesting/loading a huge batch of data when only a limited number is needed.
When using limitRequest, make sure that the provided long is "> 0" (<0 makes no sense and =0 results in a never completing Flux)
try to use reactor method users.findFirst2ByRole(role).take(2)
as well you can use skip() if needed

Idiomatic way of verifying a reactive request before actually persisting to the database

I have an endpoint that accepts as well as returns a reactive type. What I'm trying to achieve is to somehow verify that the complete reactive request (that is actually an array of resources) is valid before persisting the changes to the database (read Full-Update of a ressource). The question is not so much concerned with how to actually verify the request but more with how to chain the steps together using which of springs reactive handler methods (map, flatMap and the likes) in the desired order which is basically:
verify correctness of request (the Ressource is properly annotated with JSR-303 annotations)
clear the current resource in case of valid request
persist new resources in the database after clearing the database
Let's assume the following scenario:
val service : ResourceService
#PostMapping("/resource/")
fun replaceResources(#Valid #RequestBody resources:
Flux<RessourceDto>): Flux<RessourceDto> {
var deleteWrapper = Mono.fromCallable {
service.deleteAllRessources()
}
deleteWrapper = deleteWrapper.subscribeOn(Schedulers.elastic())
return deleteWrapper.thenMany<RessourceDto> {
resources
.map(mapper::map) // map to model object
.flatMap(service::createResource)
.map(mapper::map) // map to dto object
.subscribeOn(Schedulers.parallel())
}
}
//alternative try
#PostMapping("/resourceAlternative/")
override fun replaceResourcesAlternative2(#RequestBody resources:
Flux<ResourceDto>): Flux<ResourceDto> {
return service.deleteAllResources()
.thenMany<ResourceDto> {
resources
.map(mapper::map)
.flatMap(service::createResource)
.map(mapper::map)
}
}
Whats the idiomatic way of doing this in a reactive fashion?

Is paging broken with spring data solr when using group fields?

I currently use the spring data solr library and implement its repository interfaces, I'm trying to add functionality to one of my custom queries that uses a Solr template with a SimpleQuery. it currently uses paging which appears to be working well, however, I want to use a Group field so sibling products are only counted once, at their first occurrence. I have set the group field on the query and it works well, however, it still seems to be using the un-grouped number of documents when constructing the page attributes.
is there a known work around for this?
the query syntax provides the following parameter for this purpose, but it would seem that Spring Data Solr isn’t taking advantage of it. &group.ngroups=true should return the number of groups in the result and thus give a correct page numbering.
any other info would be appreciated.
There are actually two ways to add this parameter.
Queries are converted to the solr format using QueryParsers, so it would be possible to register a modified one.
QueryParser modifiedParser = new DefaultQueryParser() {
#Override
protected void appendGroupByFields(SolrQuery solrQuery, List<Field> fields) {
super.appendGroupByFields(solrQuery, fields);
solrQuery.set(GroupParams.GROUP_TOTAL_COUNT, true);
}
};
solrTemplate.registerQueryParser(Query.class, modifiedParser);
Using a SolrCallback would be a less intrusive option:
final Query query = //...whatever query you have.
List<DomainType> result = solrTemplate.execute(new SolrCallback<List<DomainType>>() {
#Override
public List<DomainType> doInSolr(SolrServer solrServer) throws SolrServerException, IOException {
SolrQuery solrQuery = new QueryParsers().getForClass(query.getClass()).constructSolrQuery(query);
//add missing params
solrQuery.set(GroupParams.GROUP_TOTAL_COUNT, true);
return solrTemplate.convertQueryResponseToBeans(solrServer.query(solrQuery), DomainType.class);
}
});
Please feel free to open an issue.

Resources