I have Controller that returns Flux of Strings - basically returns stream of data (Strings containing current time), Strings are delayed 1 second using on .delayElements(Duration.ofSeconds(1)) on Flux
Controller:
#GetMapping(value = "/currentTimeReactiveFlux", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
#ResponseBody
public Flux<String> exampleOfFluxDeleyedResponse() {
return Flux.range(1, 1000)
.delayElements(Duration.ofSeconds(1))
.map(integer -> "Current time (1) is " + OffsetDateTime.now())
.doOnNext(s -> log.info("S: " + s + " -> " + LocalDateTime.now().getSecond()))
.doOnCancel(() -> log.info("Cancelled"));
}
Is it possible to speed up this in Integration tests with WebClientTest. This is working slowly ( delays):
Controller integration test:
#Autowired
WebTestClient webTestClient;
#Test
void testSpeedUpStreamsOfFlux() {
FluxExchangeResult<String> result = webTestClient.get().uri("/currentTimeReactiveFlux")
.accept(MediaType.TEXT_EVENT_STREAM)
.exchange()
.expectStatus().isOk()
.returnResult(String.class);
Flux<String> eventFlux = result.getResponseBody();
StepVerifier.withVirtualTime(()-> eventFlux)
.expectNextCount(10)
.thenAwait(Duration.ofSeconds(10))
.thenCancel()
.verify();
}
Related
the purpose of this request - to get at least some hint/idea of how to do it -
new to java - so does the novice question -
I want to match my csv columns with output of get request
and tell me - matches all of it or no of columns --
#RunWith(SpringRunner.class)
#SpringBootTest(classes = TestApplication.class)
//#WebMvcTest(value = SwProductController.class)
#ActiveProfiles("dev") // In case you want to test a particular profile
#AutoConfigureMockMvc
public class SwProductImplTest
{
#Autowired
private MockMvc mockMvc;
#Test
public void testRetrieveDetails() throws Exception {
String filesss = "csv--file--path";
JSONArray expected = new JSONArray(Files.readAllLines(Paths.get(filesss))
.stream()
.map(s -> new yourJson(s.split(",")[0], s.split(",")[1]))
.collect(toList()));
RequestBuilder requestBuilder = MockMvcRequestBuilders
.get("/swproduct/list")
.accept(MediaType.APPLICATION_JSON);
MvcResult result = mockMvc.perform(requestBuilder).andReturn();
//System.out.println(result.getResponse());
JSONAssert.assertEquals(result.getResponse().getContentAsString(), expected, false);
}
private class yourJson {
String s;
String s1;
public yourJson(String s, String s1) {
this.s = s;
this.s1 = s1;
}
}
}
Output of get request looks like this -
get-request
csv file -
sw product,sw product module,technology
Product 1,Module 1,REGULAR
Product 1,Module 2,SPRING CLOUD
Product 2,Module 1,REGULAR
Product 2,Module 3,REGULAR
Both the data are the same but the only difference is that your output is in JSON format while the example you post is in tabular form. Both will work fine
I'm using a WebClient object to send Http Post request to a server.
It's sending a huge amount of requests quite rapidly (there is about 4000 messages in a QueueChannel). The problem is... it seems the server can't respond fast enough... so I'm getting a lot of server error 500 and connexion closed prematurely.
Is there a way to limit the number of request per seconds ? Or limit the number of threads it's using ?
EDIT :
The Message endpoint processe message in a QueueChannel :
#MessageEndpoint
public class CustomServiceActivator {
private static final Logger logger = LogManager.getLogger();
#Autowired
IHttpService httpService;
#ServiceActivator(
inputChannel = "outputFilterChannel",
outputChannel = "outputHttpServiceChannel",
poller = #Poller( fixedDelay = "1000" )
)
public void processMessage(Data data) {
httpService.push(data);
try {
Thread.sleep(20);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
The WebClient service class :
#Service
public class HttpService implements IHttpService {
private static final String URL = "http://www.blabla.com/log";
private static final Logger logger = LogManager.getLogger();
#Autowired
WebClient webClient;
#Override
public void push(Data data) {
String body = constructString(data);
Mono<ResponseEntity<Response>> res = webClient.post()
.uri(URL + getLogType(data))
.contentLength(body.length())
.contentType(MediaType.APPLICATION_JSON)
.syncBody(body)
.exchange()
.flatMap(response -> response.toEntity(Response.class));
res.subscribe(new Consumer<ResponseEntity<Response>>() { ... });
}
}
Resilience4j has excellent support for non-blocking rate limiting with Project Reactor.
Required dependencies (beside Spring WebFlux):
<dependency>
<groupId>io.github.resilience4j</groupId>
<artifactId>resilience4j-reactor</artifactId>
<version>1.6.1</version>
</dependency>
<dependency>
<groupId>io.github.resilience4j</groupId>
<artifactId>resilience4j-ratelimiter</artifactId>
<version>1.6.1</version>
</dependency>
Example:
import io.github.resilience4j.ratelimiter.RateLimiter;
import io.github.resilience4j.ratelimiter.RateLimiterConfig;
import io.github.resilience4j.reactor.ratelimiter.operator.RateLimiterOperator;
import org.springframework.web.reactive.function.client.WebClient;
import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;
import java.time.Duration;
import java.time.LocalDateTime;
import java.util.concurrent.atomic.AtomicInteger;
public class WebClientRateLimit
{
private static final AtomicInteger COUNTER = new AtomicInteger(0);
private final WebClient webClient;
private final RateLimiter rateLimiter;
public WebClientRateLimit()
{
this.webClient = WebClient.create();
// enables 3 requests every 5 seconds
this.rateLimiter = RateLimiter.of("my-rate-limiter",
RateLimiterConfig.custom()
.limitRefreshPeriod(Duration.ofSeconds(5))
.limitForPeriod(3)
.timeoutDuration(Duration.ofMinutes(1)) // max wait time for a request, if reached then error
.build());
}
public Mono<?> call()
{
return webClient.get()
.uri("https://jsonplaceholder.typicode.com/todos/1")
.retrieve()
.bodyToMono(String.class)
.doOnSubscribe(s -> System.out.println(COUNTER.incrementAndGet() + " - " + LocalDateTime.now()
+ " - call triggered"))
.transformDeferred(RateLimiterOperator.of(rateLimiter));
}
public static void main(String[] args)
{
WebClientRateLimit webClientRateLimit = new WebClientRateLimit();
long start = System.currentTimeMillis();
Flux.range(1, 16)
.flatMap(x -> webClientRateLimit.call())
.blockLast();
System.out.println("Elapsed time in seconds: " + (System.currentTimeMillis() - start) / 1000d);
}
}
Example output:
1 - 2020-11-30T15:44:01.575003200 - call triggered
2 - 2020-11-30T15:44:01.821134 - call triggered
3 - 2020-11-30T15:44:01.823133100 - call triggered
4 - 2020-11-30T15:44:04.462353900 - call triggered
5 - 2020-11-30T15:44:04.462353900 - call triggered
6 - 2020-11-30T15:44:04.470399200 - call triggered
7 - 2020-11-30T15:44:09.461199100 - call triggered
8 - 2020-11-30T15:44:09.463157 - call triggered
9 - 2020-11-30T15:44:09.463157 - call triggered
11 - 2020-11-30T15:44:14.461447700 - call triggered
10 - 2020-11-30T15:44:14.461447700 - call triggered
12 - 2020-11-30T15:44:14.461447700 - call triggered
13 - 2020-11-30T15:44:19.462098200 - call triggered
14 - 2020-11-30T15:44:19.462098200 - call triggered
15 - 2020-11-30T15:44:19.468059700 - call triggered
16 - 2020-11-30T15:44:24.462615 - call triggered
Elapsed time in seconds: 25.096
Docs: https://resilience4j.readme.io/docs/examples-1#decorate-mono-or-flux-with-a-ratelimiter
Question Limiting rate of requests with Reactor provides two answrers (one in comment)
zipWith another flux that acts as rate limiter
.zipWith(Flux.interval(Duration.of(1, ChronoUnit.SECONDS)))
just delay each web request
use delayElements function
edit: answer below is valid for blocking RestTemplate but do not really fit well into reactive pattern.
WebClient does not have ability to limit request, but you could easily add this feature using composition.
You may throttle your client externally using RateLimiter from Guava/
(https://google.github.io/guava/releases/19.0/api/docs/index.html?com/google/common/util/concurrent/RateLimiter.html)
In this tutorial http://www.baeldung.com/guava-rate-limiter you will find how to use Rate limiter in blocking way, or with timeouts.
I would decorate all calls that need to be throttled in separate class that
limits number of calls per second
performs actual web call using WebClient
I hope I'm not late for the party. Anyway, limiting the rate of the request is just one of the problem I faced a week ago as I was creating a crawler. Here are the issues:
I have to do a recursive, paginated sequential request. Pagination parameters are included in the API that I'm calling for.
Once a response is received, pause for 1 second before doing the next request.
For certain errors encountered, do a retry
On retry, pause for certain seconds
Here's the solution:
private Flux<HostListResponse> sequentialCrawl() {
AtomicLong pageNo = new AtomicLong(2);
// Solution for #1 - Flux.expand
return getHosts(1)
.doOnRequest(value -> LOGGER.info("Start crawling."))
.expand(hostListResponse -> {
final long totalPages = hostListResponse.getData().getTotalPages();
long currPageNo = pageNo.getAndIncrement();
if (currPageNo <= totalPages) {
LOGGER.info("Crawling page " + currPageNo + " of " + totalPages);
// Solution for #2
return Mono.just(1).delayElement(Duration.ofSeconds(1)).then(
getHosts(currPageNo)
);
}
return Flux.empty();
})
.doOnComplete(() -> LOGGER.info("End of crawling."));
}
private Mono<HostListResponse> getHosts(long pageNo) {
final String uri = hostListUrl + pageNo;
LOGGER.info("Crawling " + uri);
return webClient.get()
.uri(uri)
.exchange()
// Solution for #3
.retryWhen(companion -> companion
.zipWith(Flux.range(1, RETRY + 1), (error, index) -> {
String message = "Failed to crawl uri: " + error.getMessage();
if (index <= RETRY && (error instanceof RequestIntervalTooShortException
|| error instanceof ConnectTimeoutException
|| "Connection reset by peer".equals(error.getMessage())
)) {
LOGGER.info(message + ". Retries count: " + index);
return Tuples.of(error, index);
} else {
LOGGER.warn(message);
throw Exceptions.propagate(error); //terminate the source with the 4th `onError`
}
})
.map(tuple -> {
// Solution for #4
Throwable e = tuple.getT1();
int delaySeconds = tuple.getT2();
// TODO: Adjust these values according to your needs
if (e instanceof ConnectTimeoutException) {
delaySeconds = delaySeconds * 5;
} else if ("Connection reset by peer".equals(e.getMessage())) {
// The API that this app is calling will sometimes think that the requests are SPAM. So let's rest longer before retrying the request.
delaySeconds = delaySeconds * 10;
}
LOGGER.info("Will retry crawling after " + delaySeconds + " seconds to " + uri + ".");
return Mono.delay(Duration.ofSeconds(delaySeconds));
})
.doOnNext(s -> LOGGER.warn("Request is too short - " + uri + ". Retried at " + LocalDateTime.now()))
)
.flatMap(clientResponse -> clientResponse.toEntity(String.class))
.map(responseEntity -> {
HttpStatus statusCode = responseEntity.getStatusCode();
if (statusCode != HttpStatus.OK) {
Throwable exception;
// Convert json string to Java POJO
HostListResponse response = toHostListResponse(uri, statusCode, responseEntity.getBody());
// The API that I'm calling will return error code of 06 if request interval is too short
if (statusCode == HttpStatus.BAD_REQUEST && "06".equals(response.getError().getCode())) {
exception = new RequestIntervalTooShortException(uri);
} else {
exception = new IllegalStateException("Request to " + uri + " failed. Reason: " + responseEntity.getBody());
}
throw Exceptions.propagate(exception);
} else {
return toHostListResponse(uri, statusCode, responseEntity.getBody());
}
});
}
I use this to limit the number of active requests:
public DemoClass(WebClient.Builder webClientBuilder) {
AtomicInteger activeRequest = new AtomicInteger();
this.webClient = webClientBuilder
.baseUrl("http://httpbin.org/ip")
.filter(
(request, next) -> Mono.just(next)
.flatMap(a -> {
if (activeRequest.intValue() < 3) {
activeRequest.incrementAndGet();
return next.exchange(request)
.doOnNext(b -> activeRequest.decrementAndGet());
}
return Mono.error(new RuntimeException("Too many requests"));
})
.retryWhen(Retry.anyOf(RuntimeException.class)
.randomBackoff(Duration.ofMillis(300), Duration.ofMillis(1000))
.retryMax(50)
)
)
.build();
}
public Mono<String> call() {
return webClient.get()
.retrieve()
.bodyToMono(String.class);
}
We can customize ConnectionBuilder to rate limit the active connections on WebClient.
Need to add pendingAquiredMaxCount for number of waiting requests on queue as the default queue size is always 2 * maxConnections.
This rate limits the webclient to serve the requests at a time.
ConnectionProvider provider = ConnectionProvider.builder('builder').maxConnections(maxConnections).pendingAcquireMaxCount(maxPendingRequests).build()
TcpClient tcpClient = TcpClient
.create(provider)
WebClient client = WebClient.builder()
.baseUrl('url')
.clientConnector(new ReactorClientHttpConnector(HttpClient.from(tcpClient)))
I want to execute 3 calls simultaneously and process the results once they're all done.
I know this can be achieved using AsyncRestTemplate as it is mentioned here How to use AsyncRestTemplate to make multiple calls simultaneously?
However, AsyncRestTemplate is deprecated in favor of WebClient. I have to use Spring MVC in the project but interested if I can use a WebClient just to execute simultaneous calls. Can someone advise how this should be done properly with WebClient?
Assuming a WebClient wrapper (like in reference doc):
#Service
public class MyService {
private final WebClient webClient;
public MyService(WebClient.Builder webClientBuilder) {
this.webClient = webClientBuilder.baseUrl("http://example.org").build();
}
public Mono<Details> someRestCall(String name) {
return this.webClient.get().url("/{name}/details", name)
.retrieve().bodyToMono(Details.class);
}
}
..., you could invoke it asynchronously via:
// ...
#Autowired
MyService myService
// ...
Mono<Details> foo = myService.someRestCall("foo");
Mono<Details> bar = myService.someRestCall("bar");
Mono<Details> baz = myService.someRestCall("baz");
// ..and use the results (thx to: [2] & [3]!):
// Subscribes sequentially:
// System.out.println("=== Flux.concat(foo, bar, baz) ===");
// Flux.concat(foo, bar, baz).subscribe(System.out::print);
// System.out.println("\n=== combine the value of foo then bar then baz ===");
// foo.concatWith(bar).concatWith(baz).subscribe(System.out::print);
// ----------------------------------------------------------------------
// Subscribe eagerly (& simultaneously):
System.out.println("\n=== Flux.merge(foo, bar, baz) ===");
Flux.merge(foo, bar, baz).subscribe(System.out::print);
Mono javadoc
Flux javadoc
Spring WebClient reference doc
Spring Boot WebClient reference doc
Projectreactor reference doc
Which (reactive) operator to use!
Thanks, Welcome & Kind Regards,
You can make HTTP calls concurrently using simple RestTemplate and ExecutorService:
RestTemplate restTemplate = new RestTemplate();
ExecutorService executorService = Executors.newCachedThreadPool();
Future<String> firstCallFuture = executorService.submit(() -> restTemplate.getForObject("http://first-call-example.com", String.class));
Future<String> secondCallFuture = executorService.submit(() -> restTemplate.getForObject("http://second-call-example.com", String.class));
String firstResponse = firstCallFuture.get();
String secondResponse = secondCallFuture.get();
executorService.shutdown();
Or
Future<String> firstCallFuture = CompletableFuture.supplyAsync(() -> restTemplate.getForObject("http://first-call-example.com", String.class));
Future<String> secondCallFuture = CompletableFuture.supplyAsync(() -> restTemplate.getForObject("http://second-call-example.com", String.class));
String firstResponse = firstCallFuture.get();
String secondResponse = secondCallFuture.get();
You can use Spring reactive client WebClient to send parallel requests.
In this example,
public Mono<UserInfo> getUserInfo(User user) {
Mono<UserInfo> userInfoMono = getUserInfo(user.getId());
Mono<OrgInfo> organizationInfoMono = getOrgInfo(user.getOrgId());
return Mono.zip(userInfoMono, organizationInfoMono).map(tuple -> {
UserInfo userInfo = tuple.getT1();
userInfo.setOrganization(tuple.getT2());
return userInfo;
});
}
Here:
getUserInfo makes an HTTP call to get user info from another service and returns Mono
getOrgInfo method makes HTTP call to get organization info from another service and returns Mono
Mono.zip() waits all the results from all monos and merges into a new mono and returns it.
Then, call getOrgUserInfo().block() to get the final result.
Another way:
public Mono<Boolean> areVersionsOK(){
final Mono<Boolean> isPCFVersionOK = getPCFInfo2();
final Mono<Boolean> isBlueMixVersionOK = getBluemixInfo2();
return isPCFVersionOK.mergeWith(isBlueMixVersionOK)
.filter(aBoolean -> {
return aBoolean;
})
.collectList().map(booleans -> {
return booleans.size() == 2;
});
}
My goal here is to log time of a process without using xml files for configurations. By reading other posts I came up with enriching headers in the integration flow. This kinda works, but not for the right purpose. For every new started process it gives me a startTime when the application is launched (i.e. a constant). See below:
#Bean
public IntegrationFlow processFileFlow() {
return IntegrationFlows
.from(FILE_CHANNEL_PROCESSING)
.transform(fileToStringTransformer())
.enrichHeaders(h -> h.header("startTime", String.valueOf(System.currentTimeMillis())))
.handle(FILE_PROCESSOR, "processFile").get();
}
My goal is to properly log the process without using xml files like I said above but I don't manage to do this. I found an example and tried a solution with ChannelInterceptorAdapter like this:
#Component(value = "integrationLoggingInterceptor")
public class IntegrationLoggingInterceptor extends ChannelInterceptorAdapter {
private static final Logger log = LoggerFactory.getLogger(IntegrationLoggingInterceptor.class);
#Override
public void postSend(Message<?> message, MessageChannel channel, boolean sent) {
log.debug("Post Send - Channel " + channel.getClass());
log.debug("Post Send - Headers: " + message.getHeaders() + " Payload: " + message.getPayload() + " Message sent?: " + sent);
}
#Override
public Message<?> postReceive(Message<?> message, MessageChannel channel) {
try {
log.debug("Post Receive - Channel " + channel.getClass());
log.debug("Post Receive - Headers: " + message.getHeaders() + " Payload: " + message.getPayload());
} catch (Exception ex) {
log.error("Error in post receive : ", ex);
}
return message;
}
}
But I receive no logs at all. Any ideas?
The .enrichHeaders(h -> h.header("startTime", String.valueOf(System.currentTimeMillis()))) falls to this:
public <V> HeaderEnricherSpec header(String name, V value, Boolean overwrite) {
AbstractHeaderValueMessageProcessor<V> headerValueMessageProcessor =
new StaticHeaderValueMessageProcessor<>(value);
headerValueMessageProcessor.setOverwrite(overwrite);
return header(name, headerValueMessageProcessor);
}
Pay attention to the StaticHeaderValueMessageProcessor. So, what you show is really a constant.
If you need a value calculated for each message to process, you should consider to use Function-based variant:
.enrichHeaders(h ->
h.headerFunction("startTime",
m -> String.valueOf(System.currentTimeMillis())))
In a spring-boot 2.0 rest controller, I have created the following code which works as desired:
#ResponseBody
#GetMapping("/test3")
Mono<List<String>> test3(){
List<String> l1 = Arrays.asList("one","two","three");
List<String> l2 = Arrays.asList("four","five","six");
return Flux
.concat(Flux.fromIterable(l1),Flux.fromIterable(l2))
.collectList();
}
My problem comes from trying to do the same thing from an external datasource. I have created the following test case:
#ResponseBody
#GetMapping("/test4")
Flux<Object> test4(){
List<String> indecies = Arrays.asList("1","2");
return Flux.concat(
Flux.fromIterable(indecies)
.flatMap(k -> Flux.just(myRepository.getList(k))
.subscribeOn(Schedulers.parallel()),2
)
).collectList();
}
Where myRepository is the following:
#Repository
public class MyRepository {
List<String> l1 = Arrays.asList("one","two","three");
List<String> l2 = Arrays.asList("four","five","six");
Map<String, List<String>> pm = new HashMap<String, List<String>>();
MyRepository(){
pm.put("1", l1);
pm.put("2", l2);
}
List<String> getList(String key){
List<String> list = pm.get(key);
return list;
}
}
My code labeled test4 gives me the code hint error:
Type mismatch: cannot convert from Flux< List < String >> to Publisher < ?
extends Publisher < ? extends Object >>
So a few questions:
I thought that a Flux was a publisher? So why the error?
What am I doing wrong in test 4 so that it will output the same result as in test3?
The expected output is: [["one","two","three","four","five","six"]]
Using M. Deinum's comment, here is what works:
#ResponseBody
#GetMapping("/test6")
Mono<List<String>> test6(){
List<String> indecies = Arrays.asList("1","2");
return Flux.fromIterable(indecies)
.flatMap(k -> Flux.fromIterable(myRepository.getList(k)).subscribeOn(Schedulers.parallel()),2)
.collectList();
}