A simple #RestController is connected with a #MessagingGateway to an IntegrationFlow.
After a load test we saw within the tracing that we lose "a lot of time" before even starting the processing within the flow:
Tracing result
In this example we can see that over 90ms spend befor sending the message to the flow.
Did anyone have some idea what leads to this behavior?
As far as I understood the documentation, everything is handled in the sender thread and therefore no special worker threads are created.
We use the Restcontroller since we need to create the documentation with springdoc-openapi-ui
ExampleCode:
RestController
#RestController
public class DescriptionEndpoint {
HttpMessageGateway httpMessageGateway;
public Result findData(#Valid dataRequest dataRequest) {
final Map<String, Object> headerParams = new HashMap<>();
return httpMessageGateway.basicDataDescriptionFlow(dataRequest, headerParams);
}
}
Gateway
#MessagingGateway
public interface HttpMessageGateway {
#Gateway(requestChannel = "startDataFlow.input")
Result basicDataDescriptionFlow(#Payload dataRequest prDataRequest, #Headers Map<String, Object> map);
}
IntegrationFlow
public class ExampleFlow {
#Bean
public IntegrationFlow startDataFlow() {
return new FlowExtension()
.handle(someHandler1)
.handle(someHandler2)
.handle(someHandler3)
.get();
}
}
After adding some more traces I realized, that this timing issue is caused by my spring security configuration.
Unfortunatelly, i thought, the span is only representing the time after the start of findData(..). But it seems, the tracing starts already in the proxy methods and security chain.
After improving some implementation on our JWTToken filter, the spend times for these endpoints are OK.
Related
General problem description
Due to compatibility issues with the provided database I can not use the provided r2dbc driver for the database. The only possible option is using the standard jdbc driver but I have faced some issues getting transactions to work in the spring-weflux/ project reactor context.
Transactions with jdbc usually rely on the requirement of the connection being thread-local. In project reactor Flux/Mono it is not guaranteed that each flux execution is performed in the same thread. Even more i assume one of the major benefits of reactive programming is the ability to switch threads without having to worry about it. For this reason the standard spring jdbc TransactionManager can not be used and for r2dbc a ReactiveTransactionManager is implemented. As I am using jdbc in this case neither can I use the JdbcTransactionManager, nor is a ReactiveTransactionManager available.
First of all: Is there a simple solution to this Problem?
"Hacky" solution
I will now elaborate further on the steps I already took to solve this issue for me. My idea was implementing a custom ReactiveTransactionManager, which is based on the provided JdbcTransactionManager. My assumption was that it would be possible to wrap a transaction around a Mono/Flux this way. The issue is that I did not take into account the issue described above: It works currently only in a ThreadLocal context as the underlying JdbcTransactions still rely on it. Due to this the inner transactions are handled (commit,rollback) individually if the thread is changed in between.
The following class is the implementation of my custom transaction manager to be included in a reactive stream.
public class JdbcReactiveTransactionManager implements ReactiveTransactionManager {
// Jdbc or connection based transaction manager
private final DataSourceTransactionManager transactionManager;
// ReactiveTransaction delegates everything to TransactionStatus.
static class JdbcReactiveTransaction implements ReactiveTransaction {
public JdbcReactiveTransaction(TransactionStatus transactionStatus) {
this.transactionStatus = transactionStatus;
}
private final TransactionStatus transactionStatus;
public TransactionStatus getTransactionStatus() {
return transactionStatus;
}
// [...]
}
#Override
public #NonNull Mono<ReactiveTransaction> getReactiveTransaction(TransactionDefinition definition)
throws TransactionException {
return Mono.just(transactionManager.getTransaction(definition)).map(JdbcReactiveTransaction::new);
}
#Override
public #NonNull Mono<Void> commit(#NonNull ReactiveTransaction transaction) throws TransactionException {
if (transaction instanceof JdbcReactiveTransaction t) {
transactionManager.commit(t.getTransactionStatus());
return Mono.empty();
} else {
return Mono.error(new IllegalTransactionStateException("Illegal ReactiveTransaction type used"));
}
}
#Override
public #NonNull Mono<Void> rollback(#NonNull ReactiveTransaction transaction) throws TransactionException {
if (transaction instanceof JdbcReactiveTransaction t) {
transactionManager.rollback(t.getTransactionStatus());
return Mono.empty();
} else {
return Mono.error(new IllegalTransactionStateException("Illegal ReactiveTransaction type used"));
}
}
The implemented solution works in all scenarios where the tread does not change. But a fixed thread is not what one usually wants to archive using reactive approaches. Therefore the thread must be fixed using publishOn and subscribeOn. This is all very hacky and I myself consider this a good solution but I do not see a better alternative currently. As this is only required for one use case right now I can probably do but I would really like to find a better solution.
Pinning the Thread
The example below shows the situation that I need to use both: publishOn and subscribeOn to pin the thread. If I omit either on of these some statements wont be executed in the same thread. My current assumption is that Netty executes the parsing in a separate thread (or eventloop). Therefore the additional publishOn is required.
public Mono<ServerResponse> allocateFlows(ServerRequest request) {
final val single = Schedulers.newSingle("AllocationService-allocateFlows");
return request.bodyToMono(FlowsAllocation.class)
.publishOn(single) // Why do I need this although I execute subscribeOn later?
.flatMapMany(this::someProcessingLogic)
.concatMapDelayError(this::someOtherProcessingLogic)
.as(transactionalOperator::transactional)
.subscribeOn(single, false)
.then(ServerResponse.ok().build());
}
We are developing a Spring Boot (2.4.0) application that uses Spring Integration framework(5.4.1) to build SOAP Integration flows and register them dynamically. The time taken to register ‘IntegrationFlow’ with ‘FlowContext’ is increasing exponentially as the number of flows being registered increase.
Following is a quick snapshot of time taken to register flows:
5 flows – 500 ms
100 flows – 80 sec
300 flows – 300 sec
We see that first few flows are taking about 100ms to register, and as it reaches 300 it is taking up to 7 sec to register each flow. These flows are identical in nature (and they simply log an info message and return).
Any help to resolve this issue would be highly appreciated.
SoapFlowsAutoConfiguration.java (Auto Configuration class that registers Flows dynamically(manually))
#Bean
public UriEndpointMapping uriEndpointMapping(
ServerProperties serverProps,
WebServicesProperties webServiceProps,
IntegrationFlowContext flowContext,
FlowMetadataProvider flowMetadataProvider,
#ErrorChannel(Usage.SOAP) Optional<MessageChannel> errorChannel,
BeanFactory beanFactory) {
UriEndpointMapping uriEndpointMapping = new UriEndpointMapping();
uriEndpointMapping.setUsePath(true);
Map<String, Object> endpointMap = new HashMap<>();
flowMetadataProvider
.flowMetadatas()
.forEach(
metadata -> {
String contextPath = serverProps.getServlet().getContextPath();
String soapPath = webServiceProps.getPath();
String serviceId = metadata.id();
String serviceVersion = metadata.version();
String basePath = contextPath + soapPath;
String endpointPath = String.join("/", basePath, serviceId, serviceVersion);
SimpleWebServiceInboundGateway inboundGateway = new SimpleWebServiceInboundGateway();
errorChannel.ifPresent(inboundGateway::setErrorChannel);
endpointMap.put(endpointPath, inboundGateway);
IntegrationFlowFactory flowFactory = beanFactory.getBean(metadata.flowFactoryClass());
IntegrationFlow integrationFlow =
IntegrationFlows.from(inboundGateway).gateway(flowFactory.createFlow()).get();
flowContext.registration(integrationFlow).register();
});
uriEndpointMapping.setEndpointMap(endpointMap);
return uriEndpointMapping;
}
SoapFlow.java (Integration Flow)
#Autowired private SoapFlowResolver soapFlowResolver;
#Autowired private CoreFlow delegate;
#Override
public IntegrationFlow createFlow() {
IntegrationFlow a =
flow -> flow.gateway(soapFlowResolver.resolveSoapFlow(delegate.createFlow()));
return a;
}
SoapFlowResolver.java (Common class used by all integration flows to delegate request to a Coreflow that is responsible for business logic implementation)
public IntegrationFlow resolveSoapFlow(
IntegrationFlow coreFlow) {
return flow -> {
flow.gateway(coreFlow);
};
}
CoreFlow.java (Class that handles the business logic)
#Override
public IntegrationFlow createFlow() {
return flow -> flow.logAndReply("Reached CoreFlow");
}
You are crating too many beans, where each of them checks the rest if it wasn't created before. That's how you get increase with the start time when you add more and more flows dynamically.
What I see is an abuse of the dynamic flows purpose. Each time we decide to go this way we need to think twice if we definitely need to have the whole flow as a fresh instance. Again: the flow is not volatile object, it registers a bunch of beans in the application context which are going to stay there until you remove them. And they are singletons, so can be reused in any other places of your application.
Another concern that you don't count with the best feature of Spring Integration MessageChannel pattern implementation. You definitely can have some common flows in advance and connect your dynamic with those through channel between them. You probably just need to create dynamically a SimpleWebServiceInboundGateway and wire it with the channel for your target logic which is the same for all the flows and so on.
Spring Boot Application:
a #RestController receives the following payload:
{
"cartoon": "The Little Mermaid",
"characterNames": ["Ariel", "Prince Eric", "Sebastian", "Flounder"]
}
I need to process it in the following way:
Get the unique Id for each character name: make an HTTP call to "cartoon-characters" microservice, that returns ids by names
Transform data received by the controller:
replace character names with appropriate ids that were received on the previous step from "cartoon-characters" microservice.
{
"cartoon": "The Little Mermaid",
"characterIds": [1, 2, 3, 4]
}
Send an HTTP POST request to "cartoon-db" microservice with transformed data.
Map the response from "cartoon-db" to the internal representation that is the controller return value.
The problem that I got:
I need to implement all these steps using the paradigm of Reactive Programming (non-blocking\async processing) with Spring WebFlux (Mono|Flux) and Spring Reactive WebClient - but I have zero experience with that stack, trying to read about it as much as I can, plus googling a lot but still, have a bunch of unanswered questions, for example:
Q1. I have already configured reactive webClient that sends a request to "cartoon-characters" microservice:
public Mono<Integer> getCartoonCharacterIdbyName(String characterName) {
return WebClient.builder().baseUrl("http://cartoon-characters").build()
.get()
.uri("/character/{characterName}", characterName)
.retrieve()
.bodyToMono(Integer.class);
}
As you may see, I have got a list of cartoon character names and for each of them I need to call getCartoonCharacterIdbyName(String name) method, I am not sure that the right option to call it in series, believe the right option: parallel execution.
Wrote the following method:
public List<Integer> getCartoonCharacterIds(List<String> names) {
Flux<Integer> flux = Flux.fromStream(names.stream())
.flatMap(this::getCartoonCharacterIdbyName);
return StreamSupport.stream(flux.toIterable().spliterator(), false)
.collect(Collectors.toList());
}
but I have doubts, that this code does parallel WebClient execution and also, code calls flux.toIterable() that block the thread, so with this implementation I lost non-blocking mechanism.
Are my assumptions correct?
How do I need to rewrite it to having parallelism and non-blocking?
Q2.
Is it technically possible to transform input data received by the controller (I mean replace names with ids) in reactive style: when we operate with Flux<Integer> characterIds, but not with the List<Integer> of characterIds?
Q3. Is it potentially possible to get not just transformed Data object, but Mono<> after step 2 that can be consumed by another WebClient in Step 3?
Actually it's a good question since understanding the WebFlux, or project reactor framework, when it comes to chaining micro-services requires a couple of steps.
The first is to realize that a WebClient should take a publisher in and return a publisher. Extrapolate this to 4 different method signatures to help with thinking.
Mono -> Mono
Flux -> Flux
Mono -> Flux
Flux -> Mono
For sure, in all cases, it is just Publisher->Publisher, but leave that until you understand things better. The first two are obvious, and you just use .map(...) to handle objects in the flow, but you need to learn how to handle the second two. As commented above, going from Flux->Mono could be done with .collectList(), or also with .reduce(...). Going from Mono->Flux seems to generally be done with .flatMapMany or .flatMapIterable or some variation of that. There are probably other techniques. You should never use .block() in any WebFlux code, and generally you will get a runtime error if you try to do so.
In your example you want to go to
(Mono->Flux)->(Flux->Flux)->(Flux->Flux)
As you said, you want
Mono->Flux->Flux
The second part is to understand about chaining Flows. You could do
p3(p2(p1(object)));
Which would chain p1->p2->p3, but I always found it more understandable to make a "Service Layer" instead.
o2 = p1(object);
o3 = p2(o2);
result = p3(o3);
This code is just much easier to read and maintain and, with some maturity, you come to understand the worth of that statement.
The only problem I had with your example was doing a Flux<String> with WebClient as a #RequestBody. Doesn't work. See WebClient bodyToFlux(String.class) for string list doesn't separate individual values. Other than that, it's a pretty straightforward application. You'll find when you debug it that it gets to the .subscribe(System.out::println) line before it gets to the Flux<Integer> ids = mapNamesToIds(fn) line. This is because the Flow is setup before it is executed. Takes a while to understand this but it is the point of the project reactor framework.
#SpringBootApplication
#RestController
#RequestMapping("/demo")
public class DemoApplication implements ApplicationRunner {
public static void main(String[] args) {
SpringApplication.run(DemoApplication.class, args);
}
Map<Integer, CartoonCharacter> characters;
#Override
public void run(ApplicationArguments args) throws Exception {
String[] names = new String[] {"Ariel", "Prince Eric", "Sebastian", "Flounder"};
characters = Arrays.asList( new CartoonCharacter[] {
new CartoonCharacter(names[0].hashCode(), names[0], "Mermaid"),
new CartoonCharacter(names[1].hashCode(), names[1], "Human"),
new CartoonCharacter(names[2].hashCode(), names[2], "Crustacean"),
new CartoonCharacter(names[3].hashCode(), names[3], "Fish")}
)
.stream().collect(Collectors.toMap(CartoonCharacter::getId, Function.identity()));
// TODO Auto-generated method stub
CartoonRequest cr = CartoonRequest.builder()
.cartoon("The Little Mermaid")
.characterNames(Arrays.asList(names))
.build();
thisLocalClient
.post()
.uri("cartoonDetails")
.body(Mono.just(cr), CartoonRequest.class)
.retrieve()
.bodyToFlux(CartoonCharacter.class)
.subscribe(System.out::println);
}
#Bean
WebClient localClient() {
return WebClient.create("http://localhost:8080/demo/");
}
#Autowired
WebClient thisLocalClient;
#PostMapping("cartoonDetails")
Flux<CartoonCharacter> getDetails(#RequestBody Mono<CartoonRequest> cartoonRequest) {
Flux<StringWrapper> fn = cartoonRequest.flatMapIterable(cr->cr.getCharacterNames().stream().map(StringWrapper::new).collect(Collectors.toList()));
Flux<Integer> ids = mapNamesToIds(fn);
Flux<CartoonCharacter> details = mapIdsToDetails(ids);
return details;
}
// Service Layer Methods
private Flux<Integer> mapNamesToIds(Flux<StringWrapper> names) {
return thisLocalClient
.post()
.uri("findIds")
.body(names, StringWrapper.class)
.retrieve()
.bodyToFlux(Integer.class);
}
private Flux<CartoonCharacter> mapIdsToDetails(Flux<Integer> ids) {
return thisLocalClient
.post()
.uri("findDetails")
.body(ids, Integer.class)
.retrieve()
.bodyToFlux(CartoonCharacter.class);
}
// Services
#PostMapping("findIds")
Flux<Integer> getIds(#RequestBody Flux<StringWrapper> names) {
return names.map(name->name.getString().hashCode());
}
#PostMapping("findDetails")
Flux<CartoonCharacter> getDetails(#RequestBody Flux<Integer> ids) {
return ids.map(characters::get);
}
}
Also:
#Data
#NoArgsConstructor
#AllArgsConstructor
#Builder
public class StringWrapper {
private String string;
}
#Data
#Builder
public class CartoonRequest {
private String cartoon;
private List<String> characterNames;
}
#Data
#Builder
#NoArgsConstructor
#AllArgsConstructor
public class CartoonCharacter {
Integer id;
String name;
String species;
}
I have a CXF client configured in my Spring Boot app like so:
#Bean
public ConsumerSupportService consumerSupportService() {
JaxWsProxyFactoryBean jaxWsProxyFactoryBean = new JaxWsProxyFactoryBean();
jaxWsProxyFactoryBean.setServiceClass(ConsumerSupportService.class);
jaxWsProxyFactoryBean.setAddress("https://www.someservice.com/service?wsdl");
jaxWsProxyFactoryBean.setBindingId(SOAPBinding.SOAP12HTTP_BINDING);
WSAddressingFeature wsAddressingFeature = new WSAddressingFeature();
wsAddressingFeature.setAddressingRequired(true);
jaxWsProxyFactoryBean.getFeatures().add(wsAddressingFeature);
ConsumerSupportService service = (ConsumerSupportService) jaxWsProxyFactoryBean.create();
Client client = ClientProxy.getClient(service);
AddressingProperties addressingProperties = new AddressingProperties();
AttributedURIType to = new AttributedURIType();
to.setValue(applicationProperties.getWex().getServices().getConsumersupport().getTo());
addressingProperties.setTo(to);
AttributedURIType action = new AttributedURIType();
action.setValue("http://serviceaction/SearchConsumer");
addressingProperties.setAction(action);
client.getRequestContext().put("javax.xml.ws.addressing.context", addressingProperties);
setClientTimeout(client);
return service;
}
private void setClientTimeout(Client client) {
HTTPConduit conduit = (HTTPConduit) client.getConduit();
HTTPClientPolicy policy = new HTTPClientPolicy();
policy.setConnectionTimeout(applicationProperties.getWex().getServices().getClient().getConnectionTimeout());
policy.setReceiveTimeout(applicationProperties.getWex().getServices().getClient().getReceiveTimeout());
conduit.setClient(policy);
}
This same service bean is accessed by two different threads in the same application sequence. If I execute this particular sequence 10 times in a row, I will get a connection timeout from the service call at least 3 times. What I'm seeing is:
Caused by: java.io.IOException: Timed out waiting for response to operation {http://theservice.com}SearchConsumer.
at org.apache.cxf.endpoint.ClientImpl.waitResponse(ClientImpl.java:685) ~[cxf-core-3.2.0.jar:3.2.0]
at org.apache.cxf.endpoint.ClientImpl.processResult(ClientImpl.java:608) ~[cxf-core-3.2.0.jar:3.2.0]
If I change the sequence such that one of the threads does not call this service, then the error goes away. So, it seems like there's some sort of a race condition happening here. If I look at the logs in our proxy manager for this service, I can see that both of the service calls do return a response very quickly, but the second service call seems to get stuck somewhere in the code and never actually lets go of the connection until the timeout value is reached. I've been trying to track down the cause of this for quite a while, but have been unsuccessful.
I've read some mixed opinions as to whether or not CXF client proxies are thread-safe, but I was under the impression that they were. If this actually not the case, and I should be creating a new client proxy for each invocation, or use a pool of proxies?
Turns out that it is an issue with the proxy not being thread-safe. What I wound up doing was leveraging a solution kind of like one posted at the bottom of this post: Is this JAX-WS client call thread safe? - I created a pool for the proxies and I use that to access proxies from multiple threads in a thread-safe manner. This seems to work out pretty well.
public class JaxWSServiceProxyPool<T> extends GenericObjectPool<T> {
JaxWSServiceProxyPool(Supplier<T> factory, GenericObjectPoolConfig poolConfig) {
super(new BasePooledObjectFactory<T>() {
#Override
public T create() throws Exception {
return factory.get();
}
#Override
public PooledObject<T> wrap(T t) {
return new DefaultPooledObject<>(t);
}
}, poolConfig != null ? poolConfig : new GenericObjectPoolConfig());
}
}
I then created a simple "registry" class to keep references to various pools.
#Component
public class JaxWSServiceProxyPoolRegistry {
private static final Map<Class, JaxWSServiceProxyPool> registry = new HashMap<>();
public synchronized <T> void register(Class<T> serviceTypeClass, Supplier<T> factory, GenericObjectPoolConfig poolConfig) {
Assert.notNull(serviceTypeClass);
Assert.notNull(factory);
if (!registry.containsKey(serviceTypeClass)) {
registry.put(serviceTypeClass, new JaxWSServiceProxyPool<>(factory, poolConfig));
}
}
public <T> void register(Class<T> serviceTypeClass, Supplier<T> factory) {
register(serviceTypeClass, factory, null);
}
#SuppressWarnings("unchecked")
public <T> JaxWSServiceProxyPool<T> getServiceProxyPool(Class<T> serviceTypeClass) {
Assert.notNull(serviceTypeClass);
return registry.get(serviceTypeClass);
}
}
To use it, I did:
JaxWSServiceProxyPoolRegistry jaxWSServiceProxyPoolRegistry = new JaxWSServiceProxyPoolRegistry();
jaxWSServiceProxyPoolRegistry.register(ConsumerSupportService.class,
this::buildConsumerSupportServiceClient,
getConsumerSupportServicePoolConfig());
Where buildConsumerSupportServiceClient uses a JaxWsProxyFactoryBean to build up the client.
To retrieve an instance from the pool I inject my registry class and then do:
JaxWSServiceProxyPool<ConsumerSupportService> consumerSupportServiceJaxWSServiceProxyPool = jaxWSServiceProxyPoolRegistry.getServiceProxyPool(ConsumerSupportService.class);
And then borrow/return the object from/to the pool as necessary.
This seems to work well so far. I've executed some fairly heavy load tests against it and it's held up.
Using Springboot 1.5.x, Spring Cloud, and JAX-RS:
I could use a second pair of eyes since it is not clear to me whether the Spring configured, Javanica HystrixCommand works for all use cases or whether I may have an error in my code. Below is an approximation of what I'm doing, the code below will not actually compile.
From below WebService lives in a library with separate package path to the main application(s). Meanwhile MyWebService lives in the application that is in the same context path as the Springboot application. Also MyWebService is functional, no issues there. This just has to do with the visibility of HystrixCommand annotation in regards to Springboot based configuration.
At runtime, what I notice is that when a code like the one below runs, I do see "commandKey=A" in my response. This one I did not quite expect since it's still running while the data is obtained. And since we log the HystrixRequestLog, I also see this command key in my logs.
But all the other Command keys are not visible at all, regardless of where I place them in the file. If I remove CommandKey-A then no commands are visible whatsoever.
Thoughts?
// Example WebService that we use as a shared component for performing a backend call that is the same across different resources
#RequiredArgsConstructor
#Accessors(fluent = true)
#Setter
public abstract class WebService {
private final #Nonnull Supplier<X> backendFactory;
#Setter(AccessLevel.PACKAGE)
private #Nonnull Supplier<BackendComponent> backendComponentSupplier = () -> new BackendComponent();
#GET
#Produces("application/json")
#HystrixCommand(commandKey="A")
public Response mainCall() {
Object obj = new Object();
try {
otherCommandMethod();
} catch (Exception commandException) {
// do nothing (for this example)
}
// get the hystrix request information so that we can determine what was executed
Optional<Collection<HystrixInvokableInfo<?>>> executedCommands = hystrixExecutedCommands();
// set the hystrix data, viewable in the response
obj.setData("hystrix", executedCommands.orElse(Collections.emptyList()));
if(hasError(obj)) {
return Response.serverError()
.entity(obj)
.build();
}
return Response.ok()
.entity(healthObject)
.build();
}
#HystrixCommand(commandKey="B")
private void otherCommandMethod() {
backendComponentSupplier
.get()
.observe()
.toBlocking()
.subscribe();
}
Optional<Collection<HystrixInvokableInfo<?>>> hystrixExecutedCommands() {
Optional<HystrixRequestLog> hystrixRequest = Optional
.ofNullable(HystrixRequestLog.getCurrentRequest());
// get the hystrix executed commands
Optional<Collection<HystrixInvokableInfo<?>>> executedCommands = Optional.empty();
if (hystrixRequest.isPresent()) {
executedCommands = Optional.of(hystrixRequest.get()
.getAllExecutedCommands());
}
return executedCommands;
}
#Setter
#RequiredArgsConstructor
public class BackendComponent implements ObservableCommand<Void> {
#Override
#HystrixCommand(commandKey="Y")
public Observable<Void> observe() {
// make some backend call
return backendFactory.get()
.observe();
}
}
}
// then later this component gets configured in the specific applications with sample configuraiton that looks like this:
#SuppressWarnings({ "unchecked", "rawtypes" })
#Path("resource/somepath")
#Component
public class MyWebService extends WebService {
#Inject
public MyWebService(Supplier<X> backendSupplier) {
super((Supplier)backendSupplier);
}
}
There is an issue with mainCall() calling otherCommandMethod(). Methods with #HystrixCommand can not be called from within the same class.
As discussed in the answers to this question this is a limitation of Spring's AOP.