In my Spring Webflux API gateway I am receiving a Flux from a microservice via REST:
public Flux<MyObject> getMyObjects(String id) {
Flux<MyObject> myObjects = webClient.get().uri(nextServerUrl + "/myobject" + issueId)
.accept(MediaType.APPLICATION_JSON)
.retrieve()
.bodyToFlux(MyObject.class);
return myObjects;
}
I have to rearrange the information received by the microservice in the API gateway for the response to the client. I tried to do it in two ways:
Use the Flux as far as possible:
private Rearranged createClientResponse(String id) {
Rearranged rearranged = new Rearranged();
Flux<MyObject> myObjects = myObjectService.getMyObjects(id);
rearranged.setMyObjects(myObjects);
myObjects.map(myObject -> {
rearranged.setInfo(myObject.getInfo());
//set more
return myObjects;
});
return rearranged;
}
public class Rearranged {
private Flux<MyObject> myObjects;
//more attributes
}
Result: Following empty object:
{
"information": null,
"myObjects": {
"scanAvailable": true,
"prefetch": -1
}
}
Block the Flux and work with synchronous objects
private Rearranged createClientResponse(String id) {
Rearranged rearranged = new Rearranged();
List<MyObject> myObjects = myObjectService.getMyObjects(id).collectList().block();
rearranged.setMyObjects(myObjects);
rearranged.setInfo(myObjects.get(0).getInfo());
return rearranged;
}
public class Rearranged {
private List<MyObject> myObjects;
//more attributes
}
Result: receiving the exception block()/blockFirst()/blockLast() are blocking which is not supported in thread
What would be the right way to achieve the possibility of rearranging the information from the microservice response to respond to the client?
How would I be able to block for the Flux to complete? I understand that a block is possible when I am returning a "synchronous" object (like I am doing but still getting the exception)?
First of all, your model should not countains reactive stream. Use plain object or list.
public class Rearranged {
private MyObject myObject;
}
Or
public class Rearranged {
private List<MyObject> myObjects;
}
If you block the thread, reactor threads will exhausted in a moments. If your getMyObjects method only receives one object (if not, look at the end of the comment), then you should handle it as a Mono.
Then in the createClientResponse, you have to return with Mono<Rearranged>
Now you can easily map from one Mono to another using the .map method.
private Mono<Rearranged> createClientResponse(String id) {
Mono<MyObject> myObjects = myObjectService.getMyObjects(id);
return myObjects.map(myObject -> {
retrun new Rearranged(myObject)
//create the proper object here
});
}
If you need more object, you can use the same method, for example, the collectList() collect the elements from the Flux<> into Mono<List<>>, then the same method can be accepted.
Related
I have a Spring boot application. End point A calls three different REST endpoints X, Y, Z. All the calls were using RestTemplate. I am trying to change from RestTemplate to Webclient. As a part of this I changed endpoint Y from RestTemplate to Webclient.
I had a blocking code. It was working as expected. But when I changed it to non-blocking using subscribe things are not working as expected.
With Blocking code
public class SomeImplClass {
#Autowired
private WebClient webClient;
public someReturnType someMethodName()
{
List myList = new ArrayList<>();
Mono<SomeResponse> result = this.webclient.post().uri(url).header(…).bodyValue(….).retrieve().bodyToMone(responseType);
someResponse = result.block(someDuration);
if(someResponse.getId().equals(“000”)
{
myList.addAll(this.somemethod(someResponse));
}else{
log.error(“some error”);
throw new SomeCustomException(“some error”)
}
return myList;
}
With Non Blocking Code
public class SomeImplClass {
#Autowired
private WebClient webClient;
public someReturnType someMethodName()
{
List myList = new ArrayList<>();
Mono<SomeResponse> result = this.webclient.post().uri(url).header(…).bodyValue(….).retrieve().bodyToMone(responseType);
result.subscribe(someResponse -> {
if(someResponse.getId().equals(“000”)
{
myList.addAll(this.somemethod(someResponse));
}
else{
log.error(“some error”);
throw new SomeCustomException(“some error”) //Not able to throw custom exception here.
}
});
return myList;
}
I am getting 2 issues
With non-blocking code the list which I am returning is empty. I guess return is called before subscribe consumes the data. How to resolve this? I tried result.doOnSuccess and doOnNext but both are not working. If I ad d Thread.sleep(5000) before return, everything is working as expected. How to achieve this without adding Thread.sleep.
I am able to throw RunTimeExceptions alone from subscribe. How to throw customeExceptions.
Hello I am new to Webflux I follow a tutorial for building reactive microservices. In my project I faced the following problem.
I want to create a crud api for the product service and the following is the Create method
#Override
public Product createProduct(Product product) {
Optional<ProductEntity> productEntity = Optional.ofNullable(repository.findByProductId(product.getProductId()).block());
productEntity.ifPresent((prod -> {
throw new InvalidInputException("Duplicate key, Product Id: " + product.getProductId());
}));
ProductEntity entity = mapper.apiToEntity(product);
Mono<Product> newProduct = repository.save(entity)
.log()
.map(mapper::entityToApi);
return newProduct.block();
}
The problem is that when I call this method from postman I get the error
"block()/blockFirst()/blockLast() are blocking, which is not supported in thread reactor-http-nio-3" but when I use a StreamListener this call works ok. The stream Listener gets events from a rabbit-mq channel
StreamListener
#EnableBinding(Sink.class)
public class MessageProcessor {
private final ProductService productService;
public MessageProcessor(ProductService productService) {
this.productService = productService;
}
#StreamListener(target = Sink.INPUT)
public void process(Event<Integer, Product> event) {
switch (event.getEventType()) {
case CREATE:
Product product = event.getData();
LOG.info("Create product with ID: {}", product.getProductId());
productService.createProduct(product);
break;
default:
String errorMessage = "Incorrect event type: " + event.getEventType() + ", expected a CREATE or DELETE event";
LOG.warn(errorMessage);
throw new EventProcessingException(errorMessage);
}
}
}
I Have two questions.
Why this works with The StreamListener and not with a simple request?
Is there a proper way in webflux to return the object of the Mono or we always have to return a Mono?
Your create method would want to look more like this and you would want to return a Mono<Product> from your controller rather than the object alone.
public Mono<Product> createProduct(Product product) {
return repository.findByProductId(product.getProductId())
.switchIfEmpty(Mono.just(mapper.apiToEntity(product)))
.flatMap(repository::save)
.map(mapper::entityToApi);
}
As #Thomas commented you are breaking some of the fundamentals of reactive coding and not getting the benefits by using block() and should read up on it more. For example the reactive mongo repository you are using will be returning a Mono which has its own methods for handling if it is empty without needing to use an Optional as shown above.
EDIT to map to error if entity already exists otherwise save
public Mono<Product> createProduct(Product product) {
return repository.findByProductId(product.getProductId())
.hasElement()
.filter(exists -> exists)
.flatMap(exists -> Mono.error(new Exception("my exception")))
.then(Mono.just(mapper.apiToEntity(product)))
.flatMap(repository::save)
.map(mapper::entityToApi);
}
Spring Boot Application:
a #RestController receives the following payload:
{
"cartoon": "The Little Mermaid",
"characterNames": ["Ariel", "Prince Eric", "Sebastian", "Flounder"]
}
I need to process it in the following way:
Get the unique Id for each character name: make an HTTP call to "cartoon-characters" microservice, that returns ids by names
Transform data received by the controller:
replace character names with appropriate ids that were received on the previous step from "cartoon-characters" microservice.
{
"cartoon": "The Little Mermaid",
"characterIds": [1, 2, 3, 4]
}
Send an HTTP POST request to "cartoon-db" microservice with transformed data.
Map the response from "cartoon-db" to the internal representation that is the controller return value.
The problem that I got:
I need to implement all these steps using the paradigm of Reactive Programming (non-blocking\async processing) with Spring WebFlux (Mono|Flux) and Spring Reactive WebClient - but I have zero experience with that stack, trying to read about it as much as I can, plus googling a lot but still, have a bunch of unanswered questions, for example:
Q1. I have already configured reactive webClient that sends a request to "cartoon-characters" microservice:
public Mono<Integer> getCartoonCharacterIdbyName(String characterName) {
return WebClient.builder().baseUrl("http://cartoon-characters").build()
.get()
.uri("/character/{characterName}", characterName)
.retrieve()
.bodyToMono(Integer.class);
}
As you may see, I have got a list of cartoon character names and for each of them I need to call getCartoonCharacterIdbyName(String name) method, I am not sure that the right option to call it in series, believe the right option: parallel execution.
Wrote the following method:
public List<Integer> getCartoonCharacterIds(List<String> names) {
Flux<Integer> flux = Flux.fromStream(names.stream())
.flatMap(this::getCartoonCharacterIdbyName);
return StreamSupport.stream(flux.toIterable().spliterator(), false)
.collect(Collectors.toList());
}
but I have doubts, that this code does parallel WebClient execution and also, code calls flux.toIterable() that block the thread, so with this implementation I lost non-blocking mechanism.
Are my assumptions correct?
How do I need to rewrite it to having parallelism and non-blocking?
Q2.
Is it technically possible to transform input data received by the controller (I mean replace names with ids) in reactive style: when we operate with Flux<Integer> characterIds, but not with the List<Integer> of characterIds?
Q3. Is it potentially possible to get not just transformed Data object, but Mono<> after step 2 that can be consumed by another WebClient in Step 3?
Actually it's a good question since understanding the WebFlux, or project reactor framework, when it comes to chaining micro-services requires a couple of steps.
The first is to realize that a WebClient should take a publisher in and return a publisher. Extrapolate this to 4 different method signatures to help with thinking.
Mono -> Mono
Flux -> Flux
Mono -> Flux
Flux -> Mono
For sure, in all cases, it is just Publisher->Publisher, but leave that until you understand things better. The first two are obvious, and you just use .map(...) to handle objects in the flow, but you need to learn how to handle the second two. As commented above, going from Flux->Mono could be done with .collectList(), or also with .reduce(...). Going from Mono->Flux seems to generally be done with .flatMapMany or .flatMapIterable or some variation of that. There are probably other techniques. You should never use .block() in any WebFlux code, and generally you will get a runtime error if you try to do so.
In your example you want to go to
(Mono->Flux)->(Flux->Flux)->(Flux->Flux)
As you said, you want
Mono->Flux->Flux
The second part is to understand about chaining Flows. You could do
p3(p2(p1(object)));
Which would chain p1->p2->p3, but I always found it more understandable to make a "Service Layer" instead.
o2 = p1(object);
o3 = p2(o2);
result = p3(o3);
This code is just much easier to read and maintain and, with some maturity, you come to understand the worth of that statement.
The only problem I had with your example was doing a Flux<String> with WebClient as a #RequestBody. Doesn't work. See WebClient bodyToFlux(String.class) for string list doesn't separate individual values. Other than that, it's a pretty straightforward application. You'll find when you debug it that it gets to the .subscribe(System.out::println) line before it gets to the Flux<Integer> ids = mapNamesToIds(fn) line. This is because the Flow is setup before it is executed. Takes a while to understand this but it is the point of the project reactor framework.
#SpringBootApplication
#RestController
#RequestMapping("/demo")
public class DemoApplication implements ApplicationRunner {
public static void main(String[] args) {
SpringApplication.run(DemoApplication.class, args);
}
Map<Integer, CartoonCharacter> characters;
#Override
public void run(ApplicationArguments args) throws Exception {
String[] names = new String[] {"Ariel", "Prince Eric", "Sebastian", "Flounder"};
characters = Arrays.asList( new CartoonCharacter[] {
new CartoonCharacter(names[0].hashCode(), names[0], "Mermaid"),
new CartoonCharacter(names[1].hashCode(), names[1], "Human"),
new CartoonCharacter(names[2].hashCode(), names[2], "Crustacean"),
new CartoonCharacter(names[3].hashCode(), names[3], "Fish")}
)
.stream().collect(Collectors.toMap(CartoonCharacter::getId, Function.identity()));
// TODO Auto-generated method stub
CartoonRequest cr = CartoonRequest.builder()
.cartoon("The Little Mermaid")
.characterNames(Arrays.asList(names))
.build();
thisLocalClient
.post()
.uri("cartoonDetails")
.body(Mono.just(cr), CartoonRequest.class)
.retrieve()
.bodyToFlux(CartoonCharacter.class)
.subscribe(System.out::println);
}
#Bean
WebClient localClient() {
return WebClient.create("http://localhost:8080/demo/");
}
#Autowired
WebClient thisLocalClient;
#PostMapping("cartoonDetails")
Flux<CartoonCharacter> getDetails(#RequestBody Mono<CartoonRequest> cartoonRequest) {
Flux<StringWrapper> fn = cartoonRequest.flatMapIterable(cr->cr.getCharacterNames().stream().map(StringWrapper::new).collect(Collectors.toList()));
Flux<Integer> ids = mapNamesToIds(fn);
Flux<CartoonCharacter> details = mapIdsToDetails(ids);
return details;
}
// Service Layer Methods
private Flux<Integer> mapNamesToIds(Flux<StringWrapper> names) {
return thisLocalClient
.post()
.uri("findIds")
.body(names, StringWrapper.class)
.retrieve()
.bodyToFlux(Integer.class);
}
private Flux<CartoonCharacter> mapIdsToDetails(Flux<Integer> ids) {
return thisLocalClient
.post()
.uri("findDetails")
.body(ids, Integer.class)
.retrieve()
.bodyToFlux(CartoonCharacter.class);
}
// Services
#PostMapping("findIds")
Flux<Integer> getIds(#RequestBody Flux<StringWrapper> names) {
return names.map(name->name.getString().hashCode());
}
#PostMapping("findDetails")
Flux<CartoonCharacter> getDetails(#RequestBody Flux<Integer> ids) {
return ids.map(characters::get);
}
}
Also:
#Data
#NoArgsConstructor
#AllArgsConstructor
#Builder
public class StringWrapper {
private String string;
}
#Data
#Builder
public class CartoonRequest {
private String cartoon;
private List<String> characterNames;
}
#Data
#Builder
#NoArgsConstructor
#AllArgsConstructor
public class CartoonCharacter {
Integer id;
String name;
String species;
}
I'm working with RxJava and Retrofit to consumes different endpoints. I'm working with a couple of microservices and all of them make use of RxJava and Retrofit to consume other services.
I don't have experience working with Observables so I'm checking some examples on internet to learn how to use it and also to create a couple of services on my own. I saw that method subscribe() is always used. Something like this:
#Setter
#Getter
private MovieDetail movieDetail;
public Observable<Movies> observe() {
allMoviesClientRetrofit
.getMovies()
.subscribeOn(Schedulers.io())
.observeOn(Schedulers.computation())
.subscribe(new Observer<Movies>() {
#Override
public void onCompleted() {
}
#Override
public void onError(Throwable throwable) {
}
#Override
public void onNext(Movies movies) {
allMovies = movies;
});
In the services of my work, I've searched everywhere and subscribe() is never used but everything is working correctly. How is that possible?
As you can see, in that example I need to return an Observable to keep my personal code aligned with I have in my work but if I use the subscribe() method, it returns a Subscription object and that doesn't work.
This is part of the code of my work where you can see that subscribe() is never called but it works
#GetMapping(
value = "/something",
produces = MediaType.APPLICATION_JSON_UTF8_VALUE
)
public Single<ResponseEntity<Something>> getSomething() {
return retrieveSomethingFactory
.observe()
.toSingle()
.map(something -> {
return ResponseEntity
.status(httpStatus)
.body(something);
});
class retrieveSomethingFactoryImpl implements retrieveSomethingFactory
#Override
public Observable<Something> observe() {
return Observable
.defer(() -> {
Observable<Something1> something1 = retrieveSomething1Factory
.call(link) // Retrofit call
.observe()
.subscribeOn(Schedulers.io())
.observeOn(Schedulers.computation());
Observable<Something2> something2 = retrieveSomething1Factory
.call(link) // Retrofit call
.observe()
.subscribeOn(Schedulers.io())
.observeOn(Schedulers.computation());
return Observable
.zip(something1, something2.toList(), (something1, something2) -> {
....
....
....
return something;
});
Thanks
The Observable returned from your retrieveSomethingFactory.observe() call seems to be a hot observable, meaning it will emit items irrespective of whether it is subscribed to or not. You can read a good article about cold and hot observables here.
I have one class that extends DeferredResults and extends Runnable as shown below
public class EventDeferredObject<T> extends DeferredResult<Boolean> implements Runnable {
private Long customerId;
private String email;
#Override
public void run() {
RestTemplate restTemplate=new RestTemplate();
EmailMessageDTO emailMessageDTO=new EmailMessageDTO("dineshshe#gmail.com", "Hi There");
Boolean result=restTemplate.postForObject("http://localhost:9080/asycn/sendEmail", emailMessageDTO, Boolean.class);
this.setResult(result);
}
//Constructor and getter and setters
}
Now I have controller that return the object of the above class,whenever new request comes to controller we check if that request is present in HashMap(That stores unprocessed request at that instance).If not present then we are creating object of EventDeferredObject class can store that in HashMap and call start() method on it.If this type request is already present then we will return that from HashMap.On completion on request we will delete that request from HashMap.
#RequestMapping(value="/sendVerificationDetails")
public class SendVerificationDetailsController {
private ConcurrentMap<String , EventDeferredObject<Boolean>> requestMap=new ConcurrentHashMap<String , EventDeferredObject<Boolean>>();
#RequestMapping(value="/sendEmail",method=RequestMethod.POST)
public EventDeferredObject<Boolean> sendEmail(#RequestBody EmailDTO emailDTO)
{
EventDeferredObject<Boolean> eventDeferredObject = null;
System.out.println("Size:"+requestMap.size());
if(!requestMap.containsKey(emailDTO.getEmail()))
{
eventDeferredObject=new EventDeferredObject<Boolean>(emailDTO.getCustomerId(), emailDTO.getEmail());
requestMap.put(emailDTO.getEmail(), eventDeferredObject);
Thread t1=new Thread(eventDeferredObject);
t1.start();
}
else
{
eventDeferredObject=requestMap.get(emailDTO.getEmail());
}
eventDeferredObject.onCompletion(new Runnable() {
#Override
public void run() {
if(requestMap.containsKey(emailDTO.getEmail()))
{
requestMap.remove(emailDTO.getEmail());
}
}
});
return eventDeferredObject;
}
}
Now this code works fine if there no identical request comes to that stored in HashMap. If we give number of different request at same time code works fine.
Well, I do not know if I understood correctly, but I think you might have race conditions in the code, for example here:
if(!requestMap.containsKey(emailDTO.getEmail()))
{
eventDeferredObject=new EventDeferredObject<Boolean>(emailDTO.getCustomerId(), emailDTO.getEmail());
requestMap.put(emailDTO.getEmail(), eventDeferredObject);
Thread t1=new Thread(eventDeferredObject);
t1.start();
}
else
{
eventDeferredObject=requestMap.get(emailDTO.getEmail());
}
think of a scenario in which you have two requests with the same key emailDTO.getEmail().
Request 1 checks if there is a key in the map, does not find it and puts it inside.
Request 2 comes some time later, checks if there is a key in the map, finds it, and
goes to fetch it; however just before that, the thread started by request 1 finishes and another thread, started by onComplete event, removes the key from the map. At this point,
requestMap.get(emailDTO.getEmail())
will return null, and as a result you will have a NullPointerException.
Now, this does look like a rare scenario, so I do not know if this is the problem you see.
I would try to modify the code as follows (I did not run it myself, so I might have errors):
public class EventDeferredObject<T> extends DeferredResult<Boolean> implements Runnable {
private Long customerId;
private String email;
private ConcurrentMap ourConcurrentMap;
#Override
public void run() {
...
this.setResult(result);
ourConcurrentMap.remove(this.email);
}
//Constructor and getter and setters
}
so the DeferredResult implementation has the responsibility to remove itself from the concurrent map. Moreover I do not use the onComplete to set a callback thread, as it seems to me an unnecessary complication. To avoid the race conditions I talked about before, one needs to combine somehow the verification of the presence of an entry with its fetching into one atomic operation; this is done by the putIfAbsent method of ConcurrentMap. Therefore I change the controller into
#RequestMapping(value="/sendVerificationDetails")
public class SendVerificationDetailsController {
private ConcurrentMap<String , EventDeferredObject<Boolean>> requestMap=new ConcurrentHashMap<String , EventDeferredObject<Boolean>>();
#RequestMapping(value="/sendEmail",method=RequestMethod.POST)
public EventDeferredObject<Boolean> sendEmail(#RequestBody EmailDTO emailDTO)
{
EventDeferredObject<Boolean> eventDeferredObject = new EventDeferredObject<Boolean>(emailDTO.getCustomerId(), emailDTO.getEmail(), requestMap);
EventDeferredObject<Boolean> oldEventDeferredObject = requestMap.putIfAbsent(emailDTO.getEmail(), eventDeferredObject );
if(oldEventDeferredObject == null)
{
//if no value was present before
Thread t1=new Thread(eventDeferredObject);
t1.start();
return eventDeferredObject;
}
else
{
return oldEventDeferredObject;
}
}
}
if this does not solve the problem you have, I hope that at least it might give some idea.