WebClient 5.3+ exchange vs exchangeToMono. Extracting cookies and body together - spring-boot

I am currently trying to consume a whole request coming back via webClient. I need to be able to read the body and the cookies that come with it. However I am currently having issues doing both together in one call. Below I have two ways of grabbing the data I want. I am pretty new at using lambdas and webClient. I do realize that exchangeToMono() is the latest method call to return a Mono or Flux because of the possible memory leak found in exchange().
I want to say subscribing() to a return monoBody might help me but so far I have had no luck extracting the data out that way as well. Thank you all for your input.
Mono<String> monoBody = webBuilder.build()
.get()
.uri(baseURI + "?Level=&pageSize=50&pageNo=&name=&id=&authToken=" + sessionToken)
.accept(MediaType.ALL)
.header("authToken", sessionToken)
.header("x-Token", sessionToken)
.header("X-token-X", sessionToken)
.header("Ref", ref)
.exchangeToMono(clientResponse -> clientResponse.bodyToMono(String.class));
Mono<MultiValueMap<String,ResponseCookie>> monoMap = webBuilder.build()
.get()
.uri(baseURI + "?Level=&pageSize=50&pageNo=&unitName=&id=&authToken=" + sessionToken)
.accept(MediaType.ALL)
.header("authToken", sessionToken)
.header("x-Token", sessionToken)
.header("X-token-X", sessionToken)
.header("Ref", ref)
.exchangeToMono(clientResponse -> Mono.fromCallable(()-> clientResponse.cookies() ));

The key thing to remember about the Reactor API (used by WebClient) is that you are always composing operations into a 'plan' of what to do and then at some point (subscription) you execute the entire plan.
In your case you've found the two methods to get the information you want but with a Reactive stream you can only ever have one type at any point in the stream, so if you need two types of data, you can always compose them into a composite object.
record DataWithCookies(MultiValueMap<String,ResponseCookie> cookies, String body){}
...
Mono<DataWithCookies> dataWithCookies =
webBuilder.build()
.get()
.uri(baseURI + "?Level=&pageSize=50&pageNo=&unitName=&id=&authToken=" + sessionToken)
.accept(MediaType.ALL)
.header("authToken", sessionToken)
.header("x-Token", sessionToken)
.header("X-token-X", sessionToken)
.header("Ref", ref)
.exchangeToMono(response ->
response.bodyToMono(String.class)
.map(stringBody -> new DataWithCookies(
stringBody,
response.cookies())
)
);
I used the previewed JDK 16 feature "Records" for brevity, but you could use a simple class for DataWithCookies.
You now have a Mono that will yield the body along with the cookies. You might want to even extract that lambda into a private method for readability.
...
.exchangeToMono(this::extractBodyAndCookies);
...
private DataWithCookies extractBodyAndCookies(final ClientResponse response) {
return response.bodyToMono(String.class)
.map(stringBody -> new DataWithCookies(stringBody,
response.cookies())
);
}
Another option is to use Tuples.of(stringBody, response.cookies()) which avoids creating another type while still composing the two data.

Related

How to extract content from Mono<List<T>> in WebFlux to pass it down the call chain

I want to be able to extract the List<Payload> from the Mono<List<Payload>> to pass it to a downstream service for processing (or maybe return from the read(RequestParams params) method, instead of it returning void):
#PostMapping("/subset")
public void read(#RequestBody RequestParams params){
Mono<List<Payload>> result = reader.read(params.getDate(), params.getAssetClasses(), params.getFirmAccounts(), params.getUserId(), params.getPassword());
....
}
where reader.read(...) is a method on an autowired Spring service utilizing a webClient to get the data from external web service API:
public Mono<List<Payload>> read(String date, String assetClasses, String firmAccounts, String id, String password) {
Flux<Payload> nodes = client
.get()
.uri(uriBuilder -> uriBuilder
.path("/api/subset")
.queryParam("payloads", true)
.queryParam("date", date)
.queryParam("assetClasses", assetClasses)
.queryParam("firmAccounts", firmAccounts)
.build())
.headers(header -> header.setBasicAuth("abc123", "XXXXXXX"))
.retrieve()
.onStatus(HttpStatus::is4xxClientError, response -> {
System.out.println("4xx error");
return Mono.error(new RuntimeException("4xx"));
})
.onStatus(HttpStatus::is5xxServerError, response -> {
System.out.println("5xx error");
return Mono.error(new RuntimeException("5xx"));
})
.bodyToFlux(Payload.class);
Mono<List<Payload>> records = nodes
.collectList();
return records;
}
Doing a blocking result.block() is not allowed in WebFlux and throws an exception:
new IllegalStateException("block()/blockFirst()/blockLast() are blocking, which is not supported in thread ..." ;
What is a proper way to extract the contents of a Mono in WebFlux?
Is it some kind of a subscribe()? What would be the syntax?
Thank you in advance.
There is no "proper way" and that is the entire point. To get the value you need to block, and blocking is bad in webflux for many reasons (that I won't go into right now).
What you should do is to return the publisher all the way out to the calling client.
One of the things that many usually have a hard time understanding is that webflux works with a producer (Mono or Flux) and a subscriber.
Your entire service is also a producer, and the calling client can be seen as the subscriber.
Think of it as a long chain, that starts at the datasource, and ends up in the client showing the data.
A simple rule of thumb is that whomever is the final consumer of the data is the subscriber, everyone else is a producer.
So in your case, you just return the Mono<List<T> out to the calling client.
#PostMapping("/subset")
public Mono<List<Payload>> read(#RequestBody RequestParams params){
Mono<List<Payload>> result = reader.read(params.getDate(), params.getAssetClasses(), params.getFirmAccounts(), params.getUserId(), params.getPassword());
return result;
}
While the following does return the value of the Mono observable in the logs:
#PostMapping("/subset")
#ResponseBody
public Mono<ResponseEntity<List<Payload>>> read1(#RequestBody RequestParams params){
Mono<List<Payload>> result = reader.read(params.getDate(), params.getAssetClasses(), params.getFirmAccounts(), params.getUserId(), params.getPassword());
return result
.map(e -> new ResponseEntity<List<PayloadByStandardBasis>>(e, HttpStatus.OK));
}
the understanding I was seeking was a proper way to compose a chain of calls, with WebFlux, whereby a response from one of its operators/legs (materialized as as a result from a webclient call, producing a set of records, as above) could be passed downstream to another operator/leg to facilitate a side effect of saving those records in a DB, or something to that effect.
It would probably be a good idea to model each of those steps as a separate REST endpoint, and then have another endpoint for a composition operation which internally calls each independent endpoint in the right order, or would other design choices be more preferred?
That is ultimately the understanding I was looking for, so if anyone wants to share an example code as well as opinions to better implement the set of steps described above, I'm willing to accept the most comprehensive answer.
Thank you.

Complete WebClient asynchronous example with Spring WebFlux

I am new to Reactive programming paradigm, but recently I have decided to base a simple Http client on Spring WebClient, since the old sync RestTemplate is already under maintenance and might be deprecated in upoming releases.
So first I had a look at Spring documentation and, after that, I've searched the web for examples.
I must say that (only for the time being) I have consciously decided not to go through the Reactor lib documentation, so beyond the Publisher-Subscriber pattern, my knowledge about Mono's and Flux's is scarce. I focused instead on having something working.
My scenario is a simple POST to send a callback to a Server from which the client is only interested in response status code. No body is returned. So I finally came up with this code snippet that works:
private void notifyJobSuccess(final InternalJobData jobData) {
SuccessResult result = new SuccessResult();
result.setJobId(jobData.getJobId());
result.setStatus(Status.SUCCESS);
result.setInstanceId(jobData.getInstanceId());
log.info("Result to send back:" + System.lineSeparator() + "{}", result.toString());
this.webClient.post()
.uri(jobData.getCallbackUrl())
.body(Mono.just(result), ReplaySuccessResult.class)
.retrieve()
.onStatus(s -> s.equals(HttpStatus.OK), resp -> {
log.info("Expected CCDM response received with HttpStatus = {}", HttpStatus.OK);
return Mono.empty();
})
.onStatus(HttpStatus::is4xxClientError, resp -> {
log.error("CCDM response received with unexpected Client Error HttpStatus {}. "
+ "The POST request sent by EDA2 stub did not match CCDM OpenApi spec", resp.statusCode());
return Mono.empty();
})
.onStatus(HttpStatus::is5xxServerError, resp -> {
log.error("CCDM response received with unexpected Server Error HttpStatus {}", resp.statusCode());
return Mono.empty();
}).bodyToMono(Void.class).subscribe(Eda2StubHttpClient::handleResponseFromCcdm);
}
My poor understanding of how the reactive WebClient works starts with the call to subscribe. None of the tens of examples that I checked before coding my client included such a call, but the fact is that before I included that call, the Server was sitting forever waiting for the request.
Then I bumped into the mantra "Nothing happens until you subscribe". Knowing the pattern Plublisher-Subscriber I knew that, but I (wrongly) assumed that the subscription was handled by WebClient API, in any of the exchage, or bodyToMono methods... block() definitely must subscribe, because when you block it, the request gets out at once.
So my first question is: is this call to subscribe() really needed?
Second question is why the method StubHttpClient::handleResponse is never called back. For this, the only explanation that I find is that as the Mono returned is a Mono<Void>, because there is nothing in the response besides the status code, as it is never instantiated, the method is totally dummy... I could even replace it by just .subscribe(). Is this a correct assumption.
Last, is it too much to ask for a complete example of a a method receiving a body in a Mono that is later consumed? All examples I find just focus on getting the request out, but how the Mono or Flux is later consumed is now beyond my understanding... I know that I have to end up checking the Reactor doc sooner better than later, but I would appreciate a bit of help because I am having issues with Exceptions and errors handlin.
Thanks!
Some time has passed since I asked for help here. Now I'd like not to edit but to add an answer to my previous question, so that the answer remains clear and separate from he original question and comments.
So here goes a complete example.
CONTEXT: An application, acting as a client, that requests an Access Token from an OAuth2 Authorization server. The Access Token is requested asynchronously to avoid blocking the appliction's thread while the token request is processed at the other end and the response arrives.
First, this is a class that serves Access Token to its clients (method getAccessToken): if the Access Token is already initialized and it's valid, it returns the value stored; otherwise fetches a new one calling the internal method fetchAccessTokenAsync:
public class Oauth2ClientBroker {
private static final String OAUHT2_SRVR_TOKEN_PATH= "/auth/realms/oam/protocol/openid-connect/token";
private static final String GRANT_TYPE = "client_credentials";
#Qualifier("oAuth2Client")
private final WebClient oAuth2Client;
private final ConfigurationHolder CfgHolder;
#GuardedBy("this")
private String token = null;
#GuardedBy("this")
private Instant tokenExpireTime;
#GuardedBy("this")
private String tokenUrlEndPoint;
public void getAccessToken(final CompletableFuture<String> completableFuture) {
if (!isTokenInitialized() || isTokenExpired()) {
log.trace("Access Token not initialized or has exired: go fetch a new one...");
synchronized (this) {
this.token = null;
}
fetchAccessTokenAsync(completableFuture);
} else {
log.trace("Reusing Access Token (not expired)");
final String token;
synchronized (this) {
token = this.token;
}
completableFuture.complete(token);
}
}
...
}
Next, we will see that fetchAccessTokenAsync does:
private void fetchAccessTokenAsync(final CompletableFuture<String> tokenReceivedInFuture) {
Mono<String> accessTokenResponse = postAccessTokenRequest();
accessTokenResponse.subscribe(tr -> processResponseBodyInFuture(tr, tokenReceivedInFuture));
}
Two things happen here:
The method postAccessTokenRequest() builds a POST request and declares how the reponse will be consumed (when WebFlux makes it available once it is received), by using exchangeToMono:
private Mono postAccessTokenRequest() {
log.trace("Request Access Token for OAuth2 client {}", cfgHolder.getClientId());
final URI uri = URI.create(cfgHolder.getsecServiceHostAndPort().concat(OAUHT2_SRVR_TOKEN_PATH));
} else {
uri = URI.create(tokenUrlEndPoint);
}
}
log.debug("Access Token endpoint OAuth2 Authorization server: {}", uri.toString());
return oAuth2Client.post().uri(uri)
.body(BodyInserters.fromFormData("client_id", cfgHolder.getEdaClientId())
.with("client_secret", cfgHolder.getClientSecret())
.with("scope", cfgHolder.getClientScopes()).with("grant_type", GRANT_TYPE))
.exchangeToMono(resp -> {
if (resp.statusCode().equals(HttpStatus.OK)) {
log.info("Access Token successfully obtained");
return resp.bodyToMono(String.class);
} else if (resp.statusCode().equals(HttpStatus.BAD_REQUEST)) {
log.error("Bad request sent to Authorization Server!");
return resp.bodyToMono(String.class);
} else if (resp.statusCode().equals(HttpStatus.UNAUTHORIZED)) {
log.error("OAuth2 Credentials exchange with Authorization Server failed!");
return resp.bodyToMono(String.class);
} else if (resp.statusCode().is5xxServerError()) {
log.error("Authorization Server could not generate a token due to a server error");
return resp.bodyToMono(String.class);
} else {
log.error("Authorization Server returned an unexpected status code: {}",
resp.statusCode().toString());
return Mono.error(new Exception(
String.format("Authorization Server returned an unexpected status code: %s",
resp.statusCode().toString())));
}
}).onErrorResume(e -> {
log.error(
"Access Token could not be obtained. Process ends here");
return Mono.empty();
});
}
The exchangeToMono method does most of the magic here: tells WebFlux to return a Mono that will asynchronously receive a signal as soon as the response is received, wrapped in a ClientResponse, the parameter resp consumed in the lambda. But it is important to keep in mind that NO request has been sent out yet at this point; we are just passing in the Function that will take the ClientResponse when it arrives and will return a Mono<String> with the part of the body of our interest (the Access Token, as we will see).
Once the POST is built and the Mono returned, then the real thing starts when we subscribe to the Mono<String> returned before. As the Reacive mantra says: nothing happens until you subscribe or, in our case, the request is not actually sent until something attempts to read or wait for the response. There are other ways in WebClient fluent API to implicitly subscribe, but we have chosen here the explicit way of returing the Mono -which implements the reactor Publisher interface- and subscribe to it. Here we blocking the thread no more, releasing CPU for other stuff, probably more useful than just waiting for an answer.
So far, so good: we have sent out the request, released CPU, but where the processing will continue whenever the response comes? The subscribe() method takes as an argument a Consumer parameterized in our case with a String, being nothing less than the body of the response we are waiting for, wrapped in Mono. When the response comes, WebFlux will notify the event to our Mono, which will call the method processResponseBodyInFuture, where we finally receive the response body:
private void processResponseBodyInFuture(final String body, final CompletableFuture<String> tokenReceivedInFuture) {
DocumentContext jsonContext = JsonPath.parse(body);
try {
log.info("Access Token response received: {}", body);
final String aTkn = jsonContext.read("$.access_token");
log.trace("Access Token parsed: {}", aTkn);
final int expiresIn = jsonContext.read("$.expires_in");
synchronized (this) {
this.token = aTkn;
this.tokenExpireTime = Instant.now().plusSeconds(expiresIn);
}
log.trace("Signal Access Token request completion. Processing will continue calling client...");
tokenReceivedInFuture.complete(aTkn);
} catch (PathNotFoundException e) {
try {
log.error(e.getMessage());
log.info(String.format(
"Could not extract Access Token. The response returned corresponds to the error %s: %s",
jsonContext.read("$.error"), jsonContext.read("$.error_description")));
} catch (PathNotFoundException e2) {
log.error(e2.getMessage().concat(" - Unexpected json content received from OAuth2 Server"));
}
}
}
The invocation of this method happens as soon as the Mono is signalled about the reception of the response. So here we try to parse the json content with an Access Token and do something with it... In this case call complete() onto the CompletableFuture passed in by the caller of the initial method getAccessToken, that hopefully will know what to do with it. Our job is done here... Asynchronously!
Summary:
To summarize, these are the basic considerations to have your request sent out and the responses processed when you ise reactive WebClient:
Consider having a method in charge of preparing the request by means of the WebClient fluent API (to set http method, uri, headers and body). Remember: by doing this you are not sending any request yet.
Think on the strategy you will use to obtain the Publisher that will be receive the http client events (response or errors). retreive() is the most straight forward, but it has less power to manipulate the response than exchangeToMono.
Subscribe... or nothing will happen.
Many examples you will find around will cheat you: they claim to use WebClient for asyncrhony, but then they "forget" about subscribing to the Publisher and call block() instead. Well, while this makes things easier and they seem to work (you will see responses received and passed to your application), the thing is that this is not asynchronous anymore: your Mono (or Flux, whatever you use) will be blocking until the response arrives. No good.
Have a separate method (being the Consumer passed in the subscribe() method) where the response body is processed.

Handle multiple external services using project reactor

I am new to reactive programming and trying to simulate the below use case using project reactor but I see little bit difficult to pass the response from one service call to another dependent service. Any suggestions or references will be highly appreciated.
Response getDetails(Request inputRequest){
//Call two external services parallel based on the incoming request
Response1 = callExternalService1(inputRequest)
Response2 = callExternalService2(inputRequest)
//Call one external service based on the Response1 and Response2
Response3 = callExternalService3(Response1,Response2);
//Call three external service parallel based on the Response1, Response2 and Response3
Response4 = callExternalService4(Response1,Response2,Response3);
Response5 = callExternalService5(Response1,Response2,Response3);
Response6 = callExternalService6(Response1,Response2,Response3);
//Last service call
Response finalResponse= callLastExternalService(Response4,Response5,Response6);
return finalResponse;
}
I tried the below sample and it's working for one service call but not able to pass the response to other dependent service calls.
Updated answer:
Mono<Response> getDetails(Request inputRequest){
return Mono.just(inputRequest.getId())
.flatMap(id->{
DbResponse res = getDBCall(id).block();
if(res == null){
return Mono.error(new DBException(....));
}
return Mono.zip(callExternalService1(res),callExternalService2(inputRequest));
}).flatMap(response->{
Response extser1 = response.getT1();
Response extser2 = response.getT2();
//any exceptions?
return Mono.zip(Mono.just(extser1),Mono.just(extser2),callExternalService3();
}).flatMap(response->callExternalService4(response.getT1(),response.getT2(),response.getT3())
});
}
private Mono<DbResponse> getDBCall(String id) {
return Mono.fromCallable(()->dbservice.get(id))
.subscribeOn(Schedulers.boundedElastic());
}
Questions:
How to convert Mono<DbResponse> to DbResponse without using block
operation?
If any of the external service failed, how to build the
failure response inside the flatmap and return back?
If you have n calls and you want to move in steplock (that is, move ahead iff you have the response from all the calls), use zip. For instance:
Mono.zip(call1, call2)
.flatMap(tuple2 -> {
ResponseEntity<?> r1 = tuple2.getT1(); //response from call1
ResponseEntity<?> r2 = tuple2.getT2(); //response from call2
return Mono.zip(Mono.just(r1), Mono.just(r2), call3);
})
.flatMap(tuple3 -> {
//at this point, you have r1, r2, r3. tuple3.getT1() response from call 1
return Mono.zip(call4, call5, call6); //tuple3.getT2() response from call 2, tuple3.getT3() response from call3
})
.flatMap(tuple3 -> callLastService);
Note: If is more of a pseudo-code, won't compile right away
You can extend the above to answer your own question. Note that since call1 and call2 are independent, you can run them parallely using subscribeOn(Schedulers.boundedElastic())
Edit: Answering the two follow-up questions:
No need to subscribe using block() as flatMap subscribes to your inner streams eagerly. You can do something like:
Mono.just(inputRequest.getId())
.flatMap(a -> getDBCall(a).switchIfEmpty(Mono.defer(() -> Mono.error(..))))
Note: Mono.callable(..) returns an empty stream if the callable returns empty. That's why switchIfEmpty
You can use operators like onErrorResume to provide a fallback stream. See: The difference between onErrorResume and doOnError
if your services return Mono of Response (otherwise you have to transform them), you can make parllel calls using zip :
Mono.zip( callExternalService1( inputRequest ),
callExternalService2( inputRequest ) )
.flatMap( resp1AndResp2 -> this.callExternalService3( resp1AndResp2.getT1(),
resp1AndResp2.getT2() )
.flatMap( response3 -> Mono.zip( callExternalService4( resp1AndResp2.getT1(),
resp1AndResp2.getT2(),
response3 ),
callExternalService5( resp1AndResp2.getT1(),
resp1AndResp2.getT2(),
response3 ),
callExternalService6( resp1AndResp2.getT1(),
resp1AndResp2.getT2(),
response3 ) )
.flatMap( resp4AndResp5AndResp6 -> callLastExternalService( resp4AndResp5AndResp6.getT1(),
resp4AndResp5AndResp6.getT2(),
resp4AndResp5AndResp6.getT3() ) ) ) );

Multiple Mono and common subscriber

I am a newbie to reactive programming in Java. I plan to use spring-webclient instead of restclient as the latter is being decommissioned. I have a situation when I make several http post requests to different endpoints and the response structure is identical. With webclient code as below,
List<Mono<CommonResponse>> monolist = new ArrayList<>();
for(String endpoint : endpoints) {
Mono<CommonResponse> mono = webClient.post()
.uri(URI.create(endPoint))
.body(Mono.just(requestData), RequestData.class)
.retrieve()
.bodyToMono(CommonResponse.class);
monolist.add(mono);
}
I get a mono per request. As the response is common, I would like each mono to be subscribed a common method, but how can I distinguish the endpoints, assuming that the response data is not helping.
Can I pass additional arguments to the method while subscribing?
You can do this in following way. If you have many monos you can treat team as flux which actually means that you have many of Mono. Then you can subscribe all of them with single method. To pass to subscribing method some extra arguments like information about endpoint, you can create extra object with additional information.
Flux<ResponseWithEndpoint> commonResponseFlux = Flux.fromIterable(endpoints)
.flatMap(endpoint -> webClient.post()
.uri(URI.create(endpoint))
.body(Mono.just(requestData), RequestData.class)
.retrieve()
.bodyToMono(CommonResponse.class)
.map(response -> new ResponseWithEndpoint(response, endpoint)));
...
class ResponseWithEndpoint {
CommonResponse commonResponse;
String endpoint;
public ResponseWithEndpoint(CommonResponse commonResponse, String endpoint) {
this.commonResponse = commonResponse;
this.endpoint = endpoint;
}
}

Stream an object, Send request with completable future and assign result to the object

I have a list of Object Signers. For each of this signers I need to make a ReST request to get their signing URL. I am trying to do it with completable futures so all ReST requests can be send in parallel, then I need to set that URL in each of the signers, so this operation will not return new signers just update the ones that I am already iterating.
I have this code that is already working, but I think that could be improved.
List<Signer> signers=......
List<CompletableFuture> futures = signers.stream()
.map(signer -> CompletableFuture.completedFuture(signer))
.map(future -> future.thenCombine( CompletableFuture.supplyAsync(()-> signatureService.getSigningUrl(future.join().getSignerId())),
(signer, url) -> {
signer.setUrl(url);
return url;
}
)).collect(toList());
futures.stream()
.map(CompletableFuture::join)
.collect(toList());
Could I replace this
futures.stream()
.map(CompletableFuture::join)
.collect(toList());
With this?
futures.stream().forEach(CompletableFuture::join)
I don't like to return that because it was already used setting it in the signer. and don't like the second collect(toList()) because I am not trying to collect anything at that time.
What other implementation would you use?
No. futures.stream().forEach(CompletableFuture::join) returns void whereas futures.stream().map(CompletableFuture::join).collect(toList()); returns CompletableFuture<List<?>>.
They both are meant for different purposes. But both does one thing common(i.e, blocks the main thread till all the completablefutures are finished).
I would write your same code bit differently using CompletableFuture.allOf.
Stream<CompletableFuture<String>> streamFutures = signers.stream()
.map(signer -> CompletableFuture.completedFuture(signer))
.map(future -> future.thenCombine(CompletableFuture.supplyAsync(() -> signatureService.getSigningUrl(future.join().getSignerId())),
(signer, url) -> {
signer.setUrl(url);
return url;
}
));
CompletableFuture<String> [] futureArr = streamFutures.toArray(size -> new CompletableFuture[size]);
List<CompletableFuture<String>> futures = Arrays.asList(futureArr);
CompletableFuture<Void> allFuturesVoid = CompletableFuture.allOf(futureArr);
allFuturesVoid.join();
CompletableFuture<List<?>> allCompletableFuture = allFuturesVoid.thenApply(future -> futures.stream().map(completableFuture -> completableFuture.join()).collect(Collectors.toList()));
There is good tutorial here https://m-hewedy.blogspot.com/2017/02/completablefutureallof-that-doenst.html and here https://medium.com/#senanayake.kalpa/fantastic-completablefuture-allof-and-how-to-handle-errors-27e8a97144a0.

Resources