Microservice throws RejectedExecutionException during load testing. What could be the issue here? - spring-boot

I have developed a Spring Boot microservice which will be used very heavily (close to 10k hits/second). I have used Feign as my REST Client and used CompletableFutures to hit other services to get data in an async way.
In this one scenario, I am hitting another service for 4 different objects in one function using CompletableFutures.
My config is as follows :
Tomcat thread size :
server.tomcat.max-threads=250
Executor thread pool size :
private RejectedExecutionHandler rejectedExecutionHandler = new ThreadPoolExecutor.CallerRunsPolicy();
private ExecutorService executorService = new ThreadPoolExecutor(1000, 1000, 0L, TimeUnit.MILLISECONDS, new LinkedBlockingQueue<Runnable>(), rejectedExecutionHandler);
My code that I'm using to hit the external service :
CompletableFuture<Integer> future1 = CompletableFuture.supplyAsync(() ->
service1.function1(params) , executorService);
CompletableFuture<List<Object>> future2 = CompletableFuture.supplyAsync(() ->
service2.function2(params) , executorService);
CompletableFuture<Object> future3 = CompletableFuture.supplyAsync(() ->
service3.function3(params) , executorService);
CompletableFuture<Object> future4=CompletableFuture.supplyAsync(()->
service4.function4 (params) , executorService);
List<Object> response =Stream.of(future1, future2, future3, future4)
.parallel()
.map(CompletableFuture::join)
.collect(Collectors.toList());
These services are SpringBoot services which use Feign internally.
#Autowired
Service1(AppConfig config)
{
this.config=config;
this.currentService=connect(config);
}
#Bean
private Service1 connect(AppConfig config) {
Logger logger = new Logger.JavaLogger();
return Feign.builder()
.encoder(new GsonEncoder())
.decoder(new GsonDecoder())
.logger(logger)
.logLevel(feign.Logger.Level.FULL)
.target(Service1.class, config.getApiUrl());
}
On Load testing this Call using jMeter, which internally calls these 4 calls to the external service, I am able to achieve a rate of 200 hits/sec on 1 AWS instance. If I increase the load on 1 instance, I start getting RejectedExecutionException.
I tried the same with 10 AWS instances under a load balancer. Expectation was that it would scale linearly to 2000 hits/sec. However I could only achieve 1400 hits/sec. Any load increased after that caused the RejectedExecutionExceptions to come back.
What can be the solution here? Is there any tweak I can try here in the configurations?
Also, could Feign be the bottleneck over here? Should I try another client like Retrofit?

Related

Spring boot Reactive caching

In my application I am using spring webflux and I am using webclient to retrieve details from some 3rd party API. Now, I want to store the first time webClient response in some in memory cache so that for 2nd time I can have those response directly from the cache.
I am trying to use Spring boot in memory caching mechanism and also "caffine". But none is working as expected.
application.yml:
spring:
cache:
cache-names: employee
caffiene:
spec: maximumSize=200, expireAfterAccess=5m
EmployeeApplication.java:
#SpringBootApplication
#EnableCaching
public class EmployeeApplication{
public static void main(String[] args){
}
}
EmployeeController.java:
It has a rest endpoint employee/all which fetch all employee from the 3rd party Api.
EmployeeService.java:
#Service
#Slf4j
public class EmployeeService{
#Autowired
private WebClient webClient;
#Autowired
private CacheManager cacheManager;
#Cacheable("employee")
public Mono<List<Employee>> getAllEmployee(){
log.info("inside employee service {}");
return webClient.get()
.uri("/employees/")
.retrieve()
.bodyToMono(Employee.class);
}
}
Although I have configured the cache name , 2nd time when I hit the url it is calling the service method. What cache mechanism need to be used to cache Mono response? Please suggest.
There are several options to cache reactive publishers.
Use reactive cache API to cache Mono for the defined duration
employeeService.getAllEmployee()
.cache(Duration.ofMinutes(60))
.flatMap(employees -> {
// process data
})
Use external cache with Caffeine.
Caffeine supports async cache based on CompletableFuture that could be easily adapted to Reactive API.
AsyncLoadingCache<String, List<Employee>> cache = Caffeine.newBuilder()
.buildAsync((tenant, executor) ->
employeeService.getAllEmployee(tenant).toFuture()
);
Mono<List<Employee>> getEmployee(String tenant) {
return Mono.fromCompletionStage(clientCache.get(tenant));
}
Use external cache with Guava and CacheMono from reactor-extra. This option is more suitable if you need to cache results based on different input (e.g. multi tenant environment)
UPDATE: CacheMono has been deprecated since reactor-extra 3.4.7. Better use #2 Use external cache with Caffeine.
Here is an example for Guava but you could adapt it for CacheManager
Cache<String, List<Employee>> cache = CacheBuilder.newBuilder()
.expireAfterWrite(cacheTtl)
.build();
Mono<List<Employee>> getEmployee(String tenant) {
return CacheMono.lookup(key -> Mono.justOrEmpty(cache.getIfPresent(key)).map(Signal::next), tenant)
.onCacheMissResume(() -> employeeService.getAllEmployee(tenant))
.andWriteWith((key, signal) -> Mono.fromRunnable(() ->
Optional.ofNullable(signal.get())
.ifPresent(value -> cache.put(key, value))
)
);
}

Spring boot: Separate thread pool for specific endpoint

Given a microservice in Spring Boot, it offers 2 end-points to be consumed from 2 separate system.
One of this system is critical while the other one is not.
I would like to prevent the "not critical" one to consume (due to unexpected problems) all the threads (or many) of the HTTP thread pool, so I would like to configure separated thread pools for each one of these end-points.
Is that possible?
There are multiple ways to do this. Using DeferredResult is probably the easiest way:
#RestController
public class Controller {
private final Executor performancePool = Executors.newFixedThreadPool(128);
private final Executor normalPool = Executors.newFixedThreadPool(16);
#GetMapping("/performance")
DeferredResult<String> performanceEndPoint() {
DeferredResult<String> result = new DeferredResult<>();
performancePool.execute(() -> {
try {
Thread.sleep(5000); //A long running task
} catch (InterruptedException e) {
e.printStackTrace();
}
result.setResult("Executed in performance pool");
});
return result;
}
#GetMapping("/normal")
DeferredResult<String> normalEndPoint() {
DeferredResult<String> result = new DeferredResult<>();
normalPool.execute(() -> result.setResult("Executed in normal pool"));
return result;
}
}
You immediately release the Tomcat thread by returning a DeferredResult from a controller, allowing it to serve other requests. The actual response is written to the user when the .setResult method is called.
DeferredResult is one of the many ways you can perform asynchronous request processing in Spring. Check out this section of the docs to learn more about the other ways:
https://docs.spring.io/spring-framework/docs/current/reference/html/web.html#mvc-ann-async
Not sure you can prevent, but you can surely increase the thread pool capacity. By default, tomcat (if default server) can handler 200 simultaneous requests , you can increase that number
Check if this article helps
https://stackoverflow.com/questions/46893237/can-spring-boot-application-handle-multiple-requests-simultaneously#:~:text=Yes%2C%20Spring%20boot%20can%20handle,can%20handle%20200%20simultaneous%20requests.&text=However%2C%20you%20can%20override%20this,tomcat.

Spring Integration Framework - Slowness in registering flows dynamically

We are developing a Spring Boot (2.4.0) application that uses Spring Integration framework(5.4.1) to build SOAP Integration flows and register them dynamically. The time taken to register ‘IntegrationFlow’ with ‘FlowContext’ is increasing exponentially as the number of flows being registered increase.
Following is a quick snapshot of time taken to register flows:
5 flows – 500 ms
100 flows – 80 sec
300 flows – 300 sec
We see that first few flows are taking about 100ms to register, and as it reaches 300 it is taking up to 7 sec to register each flow. These flows are identical in nature (and they simply log an info message and return).
Any help to resolve this issue would be highly appreciated.
SoapFlowsAutoConfiguration.java (Auto Configuration class that registers Flows dynamically(manually))
#Bean
public UriEndpointMapping uriEndpointMapping(
ServerProperties serverProps,
WebServicesProperties webServiceProps,
IntegrationFlowContext flowContext,
FlowMetadataProvider flowMetadataProvider,
#ErrorChannel(Usage.SOAP) Optional<MessageChannel> errorChannel,
BeanFactory beanFactory) {
UriEndpointMapping uriEndpointMapping = new UriEndpointMapping();
uriEndpointMapping.setUsePath(true);
Map<String, Object> endpointMap = new HashMap<>();
flowMetadataProvider
.flowMetadatas()
.forEach(
metadata -> {
String contextPath = serverProps.getServlet().getContextPath();
String soapPath = webServiceProps.getPath();
String serviceId = metadata.id();
String serviceVersion = metadata.version();
String basePath = contextPath + soapPath;
String endpointPath = String.join("/", basePath, serviceId, serviceVersion);
SimpleWebServiceInboundGateway inboundGateway = new SimpleWebServiceInboundGateway();
errorChannel.ifPresent(inboundGateway::setErrorChannel);
endpointMap.put(endpointPath, inboundGateway);
IntegrationFlowFactory flowFactory = beanFactory.getBean(metadata.flowFactoryClass());
IntegrationFlow integrationFlow =
IntegrationFlows.from(inboundGateway).gateway(flowFactory.createFlow()).get();
flowContext.registration(integrationFlow).register();
});
uriEndpointMapping.setEndpointMap(endpointMap);
return uriEndpointMapping;
}
SoapFlow.java (Integration Flow)
#Autowired private SoapFlowResolver soapFlowResolver;
#Autowired private CoreFlow delegate;
#Override
public IntegrationFlow createFlow() {
IntegrationFlow a =
flow -> flow.gateway(soapFlowResolver.resolveSoapFlow(delegate.createFlow()));
return a;
}
SoapFlowResolver.java (Common class used by all integration flows to delegate request to a Coreflow that is responsible for business logic implementation)
public IntegrationFlow resolveSoapFlow(
IntegrationFlow coreFlow) {
return flow -> {
flow.gateway(coreFlow);
};
}
CoreFlow.java (Class that handles the business logic)
#Override
public IntegrationFlow createFlow() {
return flow -> flow.logAndReply("Reached CoreFlow");
}
You are crating too many beans, where each of them checks the rest if it wasn't created before. That's how you get increase with the start time when you add more and more flows dynamically.
What I see is an abuse of the dynamic flows purpose. Each time we decide to go this way we need to think twice if we definitely need to have the whole flow as a fresh instance. Again: the flow is not volatile object, it registers a bunch of beans in the application context which are going to stay there until you remove them. And they are singletons, so can be reused in any other places of your application.
Another concern that you don't count with the best feature of Spring Integration MessageChannel pattern implementation. You definitely can have some common flows in advance and connect your dynamic with those through channel between them. You probably just need to create dynamically a SimpleWebServiceInboundGateway and wire it with the channel for your target logic which is the same for all the flows and so on.

WebServiceTemplate performance issue with spring boot

I am consuming soap web services using WebServiceTemplate its working fine with good performance in spring3+ .
spring :- 3.2.4.RELEASE
spring-ws-core :- 2.1.4.RELEASE
spring-ws-support :- 2.1.4.RELEASE
spring-ws-security :-2.1.4.RELEASE
Class for calling soap service
SaajSoapMessageFactory messageFactory = new SaajSoapMessageFactory(MessageFactory.newInstance());
messageFactory.afterPropertiesSet();
WebServiceTemplate webServiceTemplate = new WebServiceTemplate(messageFactory);
Jaxb2Marshaller marshaller = new Jaxb2Marshaller();
marshaller.setContextPath("some package");
marshaller.afterPropertiesSet();
webServiceTemplate.setMarshaller(marshaller);
webServiceTemplate.setUnmarshaller(marshaller);
webServiceTemplate.afterPropertiesSet();
webServiceTemplate.setInterceptors(clientInterceptors);
webServiceTemplate.setMessageSender(webServiceMessageSenderWithAuth);
webServiceTemplate.setDefaultUri(url);
Output result= ((JAXBElement<Output >) webServiceTemplate.marshalSendAndReceive(jaxbRequest)).getValue();
Configuration File
#Configuration
public class WebServiceConfiguration {
#Autowired
private SaajSoapMessageFactory messageFactory;
#Autowired
private WebServiceMessageSenderWithAuth webServiceMessageSenderWithAuth;
#Bean
public Wss4jSecurityInterceptor getWss4jSecurityInterceptor(#Value("${WSDL.UserName}") String userName,
#Value("${WSDL.Password}") String password) {
Wss4jSecurityInterceptor wss4jSecurityInterceptor = new Wss4jSecurityInterceptor();
wss4jSecurityInterceptor.setSecurementActions("UsernameToken");
wss4jSecurityInterceptor.setSecurementPasswordType("PasswordText");
wss4jSecurityInterceptor.setSecurementUsername(userName);
wss4jSecurityInterceptor.setSecurementPassword(password);
return wss4jSecurityInterceptor;
}
#Bean
public SaajSoapMessageFactory getSaajSoapMessageFactory() {
return new SaajSoapMessageFactory();
}
#Bean
public ClientInterceptor[] clientInterceptors(Wss4jSecurityInterceptor wsSecurityInterceptor) {
return new ClientInterceptor[] { wsSecurityInterceptor };
}
}
Performance result
Timings -- around 500ms average time , max time :- 1 sec
Spring Boot 1.5.20.RELEASE and 2.2.2.RELEASE
With spring boot same code without any change takes around 4sec for first call and if continue hitting the same then takes around 2sec for
Performance result with spring boot
First Call :- 4 sec
Subsequent calls without interval (1-10 sec gap) :- 2 sec to 800 ms
Its keep on decreasing while keep hitting the same call again and again with less interval and goes down to spring mvc 3 like result but if tried again after some interval like 5 min then again follow the same pattern
If same tried after 5 mins then again the result is same for first and further calls.
Note:- With spring boot i have tried wss4j as well instead of wss4j2
Also tried AxiomSoapMessageFactory but no luck
i have tried connection keep alive etc etc but still no luck
Caching could be one of the factors for the results above
Caching is a mechanism to enhance the performance of a system. It is a temporary memory that lies between the application and the persistent database. Cache memory stores recently used data items in order to reduce the number of database hits as much as possible.
JVM warm-up effect
When a JVM based app is launched, the first requests it receives are generally significantly slower than the average response time. This warm-up effect is usually due to class loading and bytecode interpretation at startup.
For further optimizing the application, use the Hypersistence Optimizer, which allows you to get the most out of JPA and Spring boot by scanning your application configuration and mappings.
Running Hypersitence Optimizer is very easy, as you just have to pass the EntityManagerFactory instance to the HypersistenceOptimizer object constructor, and call the init method
I suppose you already did, but if you haven't, take a look at Faster StartUp and implement the fixes suggested there.
For disabling scanning with embedded tomcat, there is a suggestion in the comments here Tomcat JarScanning
Enable Asynchronous Calls in a SpringBootApplication
#EnableSync
#SpringBootApplication
public class App {
public static void main(String[] args) {
SpringApplication.run(App.class, args);
}
So the issue is not with code. I finally deployed it on jboss Wildfly and bang..
It just start performing really well without a single line change.
Now its taking around 300ms to 500ms. So the problem is with embedded tomcat and embedded jetty is not good

Spring webflux timeout with multiple clients

I have a service that interacts with a couple of other services. So I created separate webclients for them ( because of different basepaths). I had set timeouts for them individually based on https://docs.spring.io/spring/docs/5.1.6.RELEASE/spring-framework-reference/web-reactive.html#webflux-client-builder-reactor-timeout but that does not seem to be working effectively . For one of the services tried lowering the ReadTimeout to 2 seconds but the service doesn't seem to timeout ( The logs using logging.level.org.springframework.web.reactive=debug show that the request takes about 6-7 seconds to complete).
I am using spring5.1 and netty 0.8 , I am using blocking with the webclient though because we have not gone all in with webflux yet. I tried playing around with the timeouts for each of the calls a bit and it seems like some calls do respond to the timeout while others do not ( more details alongside code below)
How I initialize webclients -
#Bean
public WebClient serviceAWebClient(#Value("${serviceA.basepath}") String basePath,
#Value("${serviceA.connection.timeout}") int connectionTimeout,
#Value("${serviceA.read.timeout}") int readTimeout,
#Value("${serviceA.write.timeout}") int writeTimeout) {
return getWebClientWithTimeout(basePath, connectionTimeout, readTimeout, writeTimeout);
}
#Bean
public WebClient serviceBWebClient(#Value("${serviceB.basepath}") String basePath,
#Value("${serviceB.connection.timeout}") int connectionTimeout,
#Value("${serviceB.read.timeout}") int readTimeout,
#Value("${serviceB.write.timeout}") int writeTimeout) {
return getWebClientWithTimeout(basePath, connectionTimeout, readTimeout, writeTimeout);
}
#Bean
public WebClient serviceCWebClient(#Value("${serviceC.basepath}") String basePath,
#Value("${serviceC.connection.timeout}") int connectionTimeout,
#Value("${serviceC.read.timeout}") int readTimeout,
#Value("${serviceC.write.timeout}") int writeTimeout) {
return getWebClientWithTimeout(basePath, connectionTimeout, readTimeout, writeTimeout);
}
private WebClient getWebClientWithTimeout(String basePath,
int connectionTimeout,
int readTimeout,
int writeTimeout) {
TcpClient tcpClient = TcpClient.create()
.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectionTimeout)
.doOnConnected(connection ->
connection.addHandlerLast(new ReadTimeoutHandler(readTimeout))
.addHandlerLast(new WriteTimeoutHandler(writeTimeout)));
return WebClient.builder().baseUrl(basePath)
.clientConnector(new ReactorClientHttpConnector(HttpClient.from(tcpClient))).build();
How I am essentially using this (have wrapper classes for each webclient) -
Mono<ResponseA> serviceACallMono = ..;
Mono<ResponseB> serviceBCallMono = ..;
Mono.zip(serviceACallMono,serviceBCallMono,
(serviceAResponse, serviceBResponse) -> serviceC.getImportantData(serviceAResponse,serviceBResponse))
.flatMap(Function.identity)
.block();
So in the above, I noticed the following -
If I lower the serviceA ReadTimeout , I do get the timeout error.
If I lower the serviceB ReadTimeout , I do get the timeout error.
If I lower the serviceC ReadTimeout , it DOES NOT responds to lowering the ReadTimeout. It just keeps on working till it gets response.
So , am I missing something here ? I was under the impression these timeouts should work in all the scenarios. Please do let me know if I can add something more .
Edit : Update, so I sort of can reproduce the issue in a simpler manner.
So, for something like -
return serviceACallMono
.flatMap(notUsed -> serviceBCallMono);
The timeout of serviceACallMono is honored, but no matter how much you lower it for serviceB it doesn't timeout.
And if you just flip the order -
return serviceBCallMono
.flatMap(notUsed -> serviceACallMono);
Now the timeout for serviceB is honored but that for serviceA isn't.
I updated the service to return Mono as well while observing the behavior in this Edit.
Edit 2 :
This is essentially whats happening in ServiceC#getImportantData -
#Override
public Mono<ServiceCResponse> getImportantData(ServiceAResponse requestA,
ServiceBResponse requestB) {
return serviceCWebClient.post()
.uri(GET_IMPORTANT_DATA_PATH, requestB.getAccountId())
.body(BodyInserters.fromObject(formRequest(requestA)))
.retrieve()
.bodyToMono(ServiceC.class);
}
formRequest is a simple POJO transformation method.
I was using spring-boot starter parent to pull various spring dependencies. Making it got from version 2.1.2 to 2.1.4 seems to resolve the issue.

Resources