Spring Webclient : illegalargumentexception not enough variables to expand 'comment_count' - spring-boot

I am using spring webclient to make a Facebook graph api request with url containing {comment_count}
But, getting this exception
java.lang.IllegalArgumentException: Not enough variable values available to expand reactive spring
Code Snippet :
import org.springframework.stereotype.Component;
import org.springframework.web.reactive.function.client.WebClient;
import reactor.core.publisher.Mono;
#Component
public class Stackoverflow {
WebClient client = WebClient.create();
public Mono<Post> fetchPost(String url) {
// Url contains "comments{comment_count}"
return client.get().uri(url).retrieve()
.bodyToMono(Post.class);
}
}
I know the solution with resttemplate, But i need to use spring webclient.

You can create your URL using UriComponentsBuilder as follows
webClient.get().uri(getFacebookGraphURI(3)).retrieve().bodyToMono(Object.class);
private URI getFacebookGraphURI(int comments){
return UriComponentsBuilder.fromHttpUrl("https://graph.facebook.com")
.pathSegment("v3.2", "PAGE_ID", "posts").queryParam("fields", "comments{comment_count}")
.queryParam("access_token", "acacaaac").build(comments);
}
OR
int commentsCount = 3; webClient.get().uri(UriComponentsBuilder.fromHttpUrl("https://graph.facebook.com")
.pathSegment("v3.2", "PAGE_ID", "posts").queryParam("fields", "comments{comment_count}")
.queryParam("access_token", "acacaaac").build().toString(),commentsCount).retrieve().bodyToMono(Object.class);

The solution I use is to disable the encoding in DefaultUriBuilderFactory
val urlBuilderFactory = DefaultUriBuilderFactory("https://foo.bar.com").apply {
encodingMode = EncodingMode.NONE
}
val wc = wcb
.clone()
.uriBuilderFactory(urlBuilderFactory)
.build()
It's in Kotlin, in Java you just have to use DefaultUriBuilderFactory#setEncodingMode(EncodingMode) with NONE as parameter.
Due to this change of behavior, you have to encode your query params yourself. To do so, I use the
val query = URLEncoder.encode(query_as_string, StandardCharsets.UTF_8.toString())
And I can perform call like this:
wc.get()
.uri { it
.path(graphqlEndpoints)
.queryParam("variables", query)
.build()
}
.retrieve()
.bodyToFlux<String>()
// ...
For Java (as per comment by Anurag below) it will be something like this:
var uriBuilderFactory = new DefaultUriBuilderFactory(baseUrl);
uriBuilderFactory.setEncodingMode(DefaultUriBuilderFactory.EncodingMode.NONE);
var wc = WebClient.builder()
.baseUrl(baseUrl)
.uriBuilderFactory(uriBuilderFactory)
// ... other options
.build();
wc.get()
.uri(uriBuilder -> uriBuilder
.path(path)
// value still needs to be URL-encoded as needed (e.g., if value is a JSON
.queryParam("param", value))
.build())
.retrieve()
.bodyToMono(...);
As noted above, you will still need to URL-encode the parameter value. With this approach, you are merely avoiding "double URL encoding" by replacing the urlBuilderFactory.

Related

Extract Mono nonblocking response and store it in a variable and use it globally

In my project I have a requirement where I need to call a third party api authentic url to get the the access token. I need to set that access token in every subsequent request header .The access token has some lifetime and when the lifetime expired I need to regenerate the access token.
application.yml
I have hardcoded the client_id,client_secret,auth_url and grant_type .
AuthController.java
here I have created an endpoint to generate the access token.
**`AuthService.java`**
#Services
#Slf4j
public class AuthService{
#Autowired
private WebClient webClient;
static String accessToken="";
public Mono<SeekResponse> getAccessToken(AuthRequest authRequest) throws InvalidTokenException{
Mono<AuthResponse> authResponse=webClient.post()
.bodyValue(authRequest)
.accept(MediaType.APPLICATION_JSON)
.retrive()
.bodyToMono(AuthResponse.class);
authResponse.doOnNext(response->{
String value=response.getAccess_token();
accessToken=accessToken+value;
})
}
}
Although I have updated the "accessToken" value but it will return me null. I understand as I have made async call this value coming as null. I can't use blocking mechanism here.
Is there any other way to generate the access token and pass it as a header for the subsequent request for authentication. Or how can I use the accessToken value globally so that I can set those token value to my subsequent api request call.
I have tried with oAuth2 by following the below article:
https://medium.com/#asce4s/oauth2-with-spring-webclient-761d16f89cdd
But when I execute I am getting the below error :
"An expected CSRF token cannot found".
I'm also learning Webflux. Here's my thought. Please correct me if I'm wrong.
We are not going to rely on doOnNext() nor doOnSuccess() nor other similar method to try to work on an pre-defined variable accessToken (That's not a way to let Mono flow). What we should focus on is converting a mono to another mono, for example converting mono response to mono access token.
The way to do that is .flatmap()/.map()/.zipwith()/...
For example,
Mono<string> tokenMono = responseMono.flatmap(
// in the map or flatmap, we get the chance to operate on variables/objects.
resp -> {
string token = response.getAccess_token();
return Mono.just(token); // with Mono.just(), we are able to convert object to Mono again.
}
) // this example is not practical, as map() is better to do the same thing. flatmap with Mono.just() is meaningless here.
Mono<string> tokenMono2 = responseMono.map(
resp -> {
string token = response.getAccess_token();
return token;
}
)
Everything starting from Mono should be always Mono until subscribed or blocked. And they provide us ways to operate on those variables inside Mono<variables>. Those are map() flatmap(), zipwith(), etc.
https://stackoverflow.com/a/60105107/18412317
Referring to a point this author said, doOnNext() is for side effect such as logging.
It's hard to understand provided sample and implementation is not really reactive. The method returns Mono but at the same time throws InvalidTokenException or usage of onNext that is a so-called side-effect operation that should be used for logging, metrics, or other similar use cases.
The way you implement oauth flow for WebClient is to create filter, Client Filters.
Spring Security provides some boilerplates for common oauth flows. Check Spring Security OAuth2 for more details.
Here is an example of simple implementation of the client credential provider
private ServerOAuth2AuthorizedClientExchangeFilterFunction oauth(String clientRegistrationId, ClientConfig config) {
var clientRegistration = ClientRegistration
.withRegistrationId(clientRegistrationId)
.tokenUri(config.getAuthUrl() + "/token")
.clientId(config.getClientId())
.clientSecret(config.getClientSecret())
.authorizationGrantType(AuthorizationGrantType.CLIENT_CREDENTIALS)
.build();
var authRepository = new InMemoryReactiveClientRegistrationRepository(clientRegistration);
var authClientService = new InMemoryReactiveOAuth2AuthorizedClientService(authRepository);
var authClientManager = new AuthorizedClientServiceReactiveOAuth2AuthorizedClientManager(
authRepository, authClientService);
var oauth = new ServerOAuth2AuthorizedClientExchangeFilterFunction(authClientManager);
oauth.setDefaultClientRegistrationId(clientRegistrationId);
return oauth;
}
then you could use it in the WebClient
WebClient.builder()
.filter(oauth)
.build()
UPDATE
Here is an example of the alternative method without filters
AuthService
#Service
public class AuthService {
private final WebClient webClient;
public AuthService() {
this.webClient = WebClient.create("<url>/token");
}
public Mono<String> getAccessToken() {
return webClient.post()
.bodyValue()
.accept(MediaType.APPLICATION_JSON)
.retrieve()
.bodyToMono(AuthResponse.class)
.map(res -> res.getAccessToken());
}
}
ApiService
#Service
public class ApiService {
private final WebClient webClient;
private final Mono<String> requestToken;
public ApiService(AuthService authService) {
this.webClient = WebClient.create("<url>/api");
// cache for token expiration
this.requestToken = authService.getAccessToken().cache(Duration.ofMinutes(10));
}
public Mono<String> request() {
return requestToken
.flatMap(token ->
webClient.get()
.header(HttpHeaders.AUTHORIZATION, "Bearer " + token)
.accept(MediaType.APPLICATION_JSON)
.retrieve()
.bodyToMono(String.class)
);
}
}

How to return application/pdf through Mono in a Reactive way

I am currently using Spring WebFlux to try build an async end-point, which fetches a PDF from a third-party end-point via Web Client before returning the PDF back to our API consumer. However, I am struggling with returning a Mono<ResponseEntity> with content type application/pdf due to the below exception:
Resolved [org.springframework.http.converter.HttpMessageNotWritableException: No converter for [class reactor.core.publisher.MonoMapFuseable] with preset Content-Type 'application/pdf']
Here is controller implementation. My question is:
Is my implementation in the right direction, or would I need to create some sort of converter?
Does Mono<ResponseEntity> even support returning a PDF as a response body?
#RequestMapping(value="/get-pdf", method = RequestMethod.GET)
public Mono<ResponseEntity> getPDFAsync() {
String url = "http://some-end-point";
WebClient client = WebClient.create(url);
return client.get()
.accept(MediaType.APPLICATION_PDF)
.exchangeToMono(response ->
Mono.just(ResponseEntity.ok().contentType(MediaType.APPLICATION_PDF)
.body(response.bodyToMono(ByteArrayResource.class)
.map(byteArrayResource -> byteArrayResource.getByteArray())
)));
}
To download a file reactively, you could supply the file as a Flux<DataBuffer>, where DataBuffer is org.springframework.core.io.buffer.DataBuffer, like this:
// some shared buffer factory.
private final DataBufferFactory dataBufferFactory = new NettyDataBufferFactory(ByteBufAllocator.DEFAULT);
#RequestMapping(value = "/download",
method = RequestMethod.GET,
produces = {MediaType.APPLICATION_PDF_VALUE}
)
public Mono<ResponseEntity<Flux<DataBuffer>>> downloadDocument(
...
) {
return Mono.fromCallable(() -> {
return ResponseEntity.ok(
DataBufferUtils.read(
new File("somepdf.pdf").toPath(),
dataBufferFactory,
8096
))
});
}
Or more specifically, since you seem to be using the WebFlux WebClient, you can forward the response body flux directly to your own response, without having to buffer the complete response first:
#RequestMapping(value = "/download",
method = RequestMethod.GET,
produces = {MediaType.APPLICATION_PDF_VALUE}
)
public Mono<ResponseEntity<Flux<DataBuffer>>> downloadDocument(
...
) {
String url = "http://some-end-point";
WebClient client = WebClient.create(url);
return client.get()
.accept(MediaType.APPLICATION_PDF)
.exchange()
.map(response -> response.bodyToFlux(DataBuffer.class))
.map(ResponseEntity::ok);
}
Hint: I hope you are reusing the WebClient instance and not instantiating a new one on each request.
I have found the answer! In short, returning Mono<byte[]>, and add produces = {MediaType.APPLICATION_PDF_VALUE} to #RequestMapping works. See example below.
#RequestMapping(value="/get-pdf", produces = {MediaType.APPLICATION_PDF_VALUE}, method = RequestMethod.GET)
public Mono<byte[]> getPdf() {
String url = "some-end-point";
WebClient client = WebClient.create(url);
return client.get()
.accept(MediaType.APPLICATION_PDF)
.exchangeToMono(response -> response
.bodyToMono(ByteArrayResource.class))
.map(byteArrayResource -> byteArrayResource.getByteArray());
}

WebClient URI template variables from bodyValue?

I recently encountered some unexpected behaviour and was wondering if this was in fact intended functionality when using WebClient. With a client config as shown below, the uri variables in the template are being overridden with the fields from a POJO used as an arg to bodyValue.
class ExampleRequest {
private String namespace = "jar";
private String service = "zoo";
...
}
...
this.client = WebClient.builder()
.baseUrl("http://{service}.{namespace}.svc.cluster.local")
.defaultUriVariables(Map.of("service", "foo", "namespace", "bar"))
.build();
this.client.post()
.uri("/baz")
.bodyValue(new ExampleRequest())
.retrieve()
...
The above ends up calling out to http://zoo.jar.svc.cluster.local/baz, not http://foo.bar.svc.cluster.local/baz. I wasn't expecting the body of the message in this instance to be picked up as uri variables. :confused:
This seems strange to me, but if this is expected could point me to the source/docs of where this expansion is taking place..?
Could you please use standard HashMap implementation and recheck? I tested locally for me it is working as expected, I mean foo and bar chosen:
Map<String, String > params = new HashMap<>();
params.put("service", "foo");
params.put("namespace", "bar");
WebClient client = WebClient.builder()
.baseUrl("http://{service}.{namespace}.svc.cluster.local")
.defaultUriVariables(params)
.build();

Simple logging dump of WebClient request?

I'm trying to use Spring WebClient to make some basic REST API calls. I'm getting an error that the request is malformed, but I can't tell exactly why. Is there any way to easily log the contents of the request (really, just the request body)? Everything I find online is super complicated. Here's what I have:
LinkedMultiValueMap params = new LinkedMultiValueMap();
params.add("app_id", getOneSignalAppId());
params.add("included_segments", inSegment);
params.add("content_available", true);
params.add("contents", new LinkedMultiValueMap() {{
add("en", inTitle);
}});
BodyInserters.MultipartInserter inserter = BodyInserters.fromMultipartData(params);
WebClient client = WebClient.builder()
.baseUrl("https://onesignal.com")
.defaultHeader(HttpHeaders.AUTHORIZATION, "Basic " + getOneSignalKey())
.defaultHeader(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE)
.defaultHeader(HttpHeaders.ACCEPT, MediaType.APPLICATION_JSON_VALUE)
.build();
Mono<NotificationResponse> result = client
.post()
.uri("/api/v1/notifications")
.body(inserter)
.retrieve()
.bodyToMono(NotificationResponse.class);
I just want a string of the JSON that will be inserted into the request body.
You can create your own wrapper/proxy class around the JSON encoder (assuming you're using JSON) and intercept the serialized body before it is sent into the intertubes.
If your request is going to send JSON.
Specifically, you would extend the encodeValue method (or encodeValues in case of streaming data) of Jackson2JsonEncoder (the default encoder). Then you can do with that data what you wish, such as logging etc. And you could even do this conditionally based on environment/profile.
This custom logging-encoder can be specified when creating the WebClient, by providing it as a codec:
CustomBodyLoggingEncoder bodyLoggingEncoder = new CustomBodyLoggingEncoder();
WebClient.builder()
.codecs(clientDefaultCodecsConfigurer -> {
clientDefaultCodecsConfigurer.defaultCodecs().jackson2JsonEncoder(bodyLoggingEncoder);
clientDefaultCodecsConfigurer.defaultCodecs().jackson2JsonDecoder(new Jackson2JsonDecoder(new ObjectMapper(), MediaType.APPLICATION_JSON));
})
...
I made a blog post about this. You might be able to find the encoder for Multipart data and apply similar principles
For completeness, the encoder might look something like this:
public class CustomBodyLoggingEncoder extends Jackson2JsonEncoder {
#Override
public DataBuffer encodeValue(final Object value, final DataBufferFactory bufferFactory,
final ResolvableType valueType, #Nullable final MimeType mimeType, #Nullable final Map<String, Object> hints) {
// Encode/Serialize data to JSON
final DataBuffer data = super.encodeValue(value, bufferFactory, valueType, mimeType, hints);
// This is your code:
SomethingAmazing.doItWithThisData(extractBytes(data));
// Return the data as normal
return data;
}
private byte[] extractBytes(final DataBuffer data) {
final byte[] bytes = new byte[data.readableByteCount()];
data.read(bytes);
// We've copied the data above to our array, but must reset the buffer for actual usage
data.readPosition(0);
return bytes;
}
}
Hope that helps somehow!

Spring WebClient Post method Body

i'm trying to send a POST request with body data as described here: https://scrapyrt.readthedocs.io/en/stable/api.html#post.
Here's what i've tried to do but it gives me HTTP code 500
String uri = "http://localhost:3000";
WebClient webClient = WebClient.builder()
.baseUrl(uri)
.build();
LinkedMultiValueMap map = new LinkedMultiValueMap();
String q = "\"url\": \"https://blog.trendmicro.com/trendlabs-security-intelligence\",\"meta\":{\"latestDate\" : \"18-05-2020\"}}";
map.add("request", q);
map.add("spider_name", "blog");
BodyInserter<MultiValueMap<String, Object>, ClientHttpRequest> inserter2
= BodyInserters.fromMultipartData(map);
Mono<ItemsList> result = webClient.post()
.uri(uriBuilder -> uriBuilder
.path("/crawl.json")
.build())
.body(inserter2)
.retrieve()
.bodyToMono(ItemsList.class);
ItemsList tempItems = result.block();
Here's what i've tried to do but it gives me HTTP code 500
Most likely because you're sending the wrong data in a mixture of wrong formats with the wrong type:
You're using multipart form data, not JSON
You're then setting the request parameter as a JSON string (q)
The JSON string you're using in q isn't even valid (it's at least missing an opening curly brace) - and handwriting JSON is almost universally a bad idea, leverage a framework to do it for you instead.
Instead, the normal thing to do would be to create a POJO structure that maps to your request, so:
public class CrawlRequest {
private CrawlInnerRequest request;
#JsonProperty("spider_name")
private String spiderName;
//....add the getters / setters
}
public class CrawlInnerRequest {
private String url;
private String callback;
#JsonProperty("dont_filter")
private String dontFilter;
//....add the getters / setters
}
...then simply create a CrawlRequest, set the values as you wish, then in your post call use:
.body(BodyInserters.fromValue(crawlRequest))
This is a rather fundamental, basic part of using a WebClient. I'd suggest reading around more widely to give yourself a better understanding of the fundamentals, it will help tremendously in the long run.
For me following code worked:
public String wcPost(){
Map<String, String> bodyMap = new HashMap();
bodyMap.put("key1","value1");
WebClient client = WebClient.builder()
.baseUrl("domainURL")
.build();
String responseSpec = client.post()
.uri("URI")
.headers(h -> h.setBearerAuth("token if any"))
.body(BodyInserters.fromValue(bodyMap))
.exchange()
.flatMap(clientResponse -> {
if (clientResponse.statusCode().is5xxServerError()) {
clientResponse.body((clientHttpResponse, context) -> {
return clientHttpResponse.getBody();
});
return clientResponse.bodyToMono(String.class);
}
else
return clientResponse.bodyToMono(String.class);
})
.block();
return responseSpec;
}

Resources