Use JSON transformer in Spring Integration - spring

I have a problem that seems unaddressed in any of the examples I can find.
My application reads an ActiveMQ topic of JSON messages. It will build a completely new outbound REST call based on this data. Note that this is not a "transformation". It is given "X" produce "Y" i.e. ServiceActivator.
My flows thus far are
public IntegrationFlow splitInputFlow() {
return IntegrationFlows.from("inboundJmsChannel")
.split()
.log(LoggingHandler.Level.DEBUG)
.route(Message.class, m -> m.getHeaders().get("x-bn-class").equals("Healthcheck.class") ? "healthcheckChannel" : "metricChannel")
.get();
}
public IntegrationFlow healthcheckFlow() {
return IntegrationFlows.from("healthcheckChannel")
.log(LoggingHandler.Level.DEBUG)
.transform(Transformers.fromJson(Healthcheck.class))
.handle("healthcheckActivator", "process")
.get();
}
There are dozens of examples on how to use spring transformers. I have even considered trying a MessageConverter. But I don't see why it would help and it doesn't seem to be the normal approach.
The main problem here is that Integration calls healthcheckActivator.process(String payload). The payload itself is the expected valid JSON string.
I am a little surprised it does not call healtcheckActivator.process(Message payload) but But that wouldn't help so it doesn't much matter.
The real question is why does it not call healtcheckActivator.process(Healthcheck healthcheck)?
Well actually I understand "why". It is because DSL generates an internal channel to tie the steps together and as far as I understand anything on a channel is a spring.messaging.Message.
I can easily instantiate my Healthcheck object once I get inside the SA. But that leaves the nagging question: What possible good is the entire transform step? If it always "serializes" the object back into a Message -- what's the point.
Like I said I think I'm missing something fundamental here.
EDIT
My new (and probably last) idea is maybe I'm publishing it wrong.
To publish it I am using
jmsTemplate.convertAndSend(topicName, healthcheck, messagePostProcessor -> {
messagePostProcessor.setJMSType("TextMessage");
messagePostProcessor.setStringProperty("x-bn-class", "Healthcheck.class");
messagePostProcessor.setStringProperty("x-bn-service-name", restEndpoint.getServiceName());
messagePostProcessor.setStringProperty("x-bn-service-endpoint-name", restEndpoint.getEndpointName());
messagePostProcessor.setLongProperty("x-bn-heathcheck-timestamp", queryDate);
messagePostProcessor.setStringProperty("x-bn-healthcheck-status", subsystemStatus.getStatus(subsystemStatus));
messagePostProcessor.setIntProperty("httpStatus", httpStatus.value());
return messagePostProcessor;
});
What arrives in the SI process(String payload) method is:
LoggingHandler - GenericMessage [payload={"healthcheckType":"LOCAL","outcome":"PASS","dependencyType":"DB","endpoint":"NODE TABLE","description":"Read from DB","durationSecs":0.025}, headers={x-bn-service-name=TG10-CS2, x-bn-service-endpoint-name=TG Q10-CS2 Ready Check, jms_destination=topic://HEALTH_MONITOR, _type=com.healthcheck.response.Healthcheck, x-bn-heathcheck-timestamp=1558356538000, priority=4, jms_timestamp=1558356544244, x-bn-healthcheck-status=SEV0, jms_redelivered=false, x-bn-class=Healthcheck.class, httpStatus=200, jms_type=TextMessage, id=b29ffea7-7128-c543-9a14-8bab450f0ac6, jms_messageId=ID:39479-1558356520091-1:2:1:1:1, timestamp=1558356544409}]
I hadn't noticed the _type parameter in the jms_destination header before. But before I started screwing around with this (because it didn't work) that is the correct class name for what the other team provided.
I have not implemented a JMS message converter. But the supplied SimpleMessageConverter seems that it should do exactly what I want.

Your understanding is correct; works fine for me, so something else is going on...
#SpringBootApplication
public class So56169938Application {
public static void main(String[] args) {
SpringApplication.run(So56169938Application.class, args);
}
#Bean
public IntegrationFlow flow() {
return IntegrationFlows.from(() -> "{\"foo\":\"bar\"}", e -> e.poller(Pollers.fixedDelay(5000)))
.transform(Transformers.fromJson(Foo.class))
.handle("myBean", "method")
.get();
}
#Bean
public MyBean myBean() {
return new MyBean();
}
public static class MyBean {
public void method(Foo foo) {
System.out.println(foo);
}
}
public static class Foo {
private String foo;
String getFoo() {
return this.foo;
}
void setFoo(String foo) {
this.foo = foo;
}
#Override
public String toString() {
return "Foo [foo=" + this.foo + "]";
}
}
}
and
Foo [foo=bar]
Foo [foo=bar]
Foo [foo=bar]
Foo [foo=bar]
Foo [foo=bar]
Foo [foo=bar]

Well, Spring Integration is a Messaging framework. It transfers messages from endpoint to endpoint via channels in between. That's already the target endpoint responsibility to deal with consumed message the proper way. The framework doesn't care about the payload. It is really a business part of the target application. That's how we can make framework components as generic as possible leaving the room for target business types for end-users.
Anyway the Framework provides some mechanisms to interact with payloads. We call it POJO method invocation. So, you provide some business with arbitrary contract, however following some Spring Integration rules: https://docs.spring.io/spring-integration/docs/current/reference/html/#service-activator.
So, according your description it is really a surprise that it doesn't work for healtcheckActivator.process(Healthcheck healthcheck). Your transform(Transformers.fromJson(Healthcheck.class)) should really produce a Message with Healthcheck object as a payload. The framework consults a method signature and tries to map a payload and/or headers to the method invocation arguments, having the whole message as a container for data to delegate to the method call.
From here it would be great to see your healtcheckActivator.process() method to determine why the transform(Transformers.fromJson(Healthcheck.class)) result cannon be mapped to that method arguments.

Related

Spring Integration Flow with #Restcontoller Timing issue

A simple #RestController is connected with a #MessagingGateway to an IntegrationFlow.
After a load test we saw within the tracing that we lose "a lot of time" before even starting the processing within the flow:
Tracing result
In this example we can see that over 90ms spend befor sending the message to the flow.
Did anyone have some idea what leads to this behavior?
As far as I understood the documentation, everything is handled in the sender thread and therefore no special worker threads are created.
We use the Restcontroller since we need to create the documentation with springdoc-openapi-ui
ExampleCode:
RestController
#RestController
public class DescriptionEndpoint {
HttpMessageGateway httpMessageGateway;
public Result findData(#Valid dataRequest dataRequest) {
final Map<String, Object> headerParams = new HashMap<>();
return httpMessageGateway.basicDataDescriptionFlow(dataRequest, headerParams);
}
}
Gateway
#MessagingGateway
public interface HttpMessageGateway {
#Gateway(requestChannel = "startDataFlow.input")
Result basicDataDescriptionFlow(#Payload dataRequest prDataRequest, #Headers Map<String, Object> map);
}
IntegrationFlow
public class ExampleFlow {
#Bean
public IntegrationFlow startDataFlow() {
return new FlowExtension()
.handle(someHandler1)
.handle(someHandler2)
.handle(someHandler3)
.get();
}
}
After adding some more traces I realized, that this timing issue is caused by my spring security configuration.
Unfortunatelly, i thought, the span is only representing the time after the start of findData(..). But it seems, the tracing starts already in the proxy methods and security chain.
After improving some implementation on our JWTToken filter, the spend times for these endpoints are OK.

Spring Clould Stream Resolving Input Channel dynamically based on Message

I need a way of resolving an Inbound Channel dynamically based on the type of the Incoming Message.
I am not looking for any header based solution which is already mentioned in this link
https://cloud.spring.io/spring-cloud-static/spring-cloud-stream/3.0.0.M1/spring-cloud-stream.html#_using_streamlistener_for_content_based_routing
The resolution has to happen based on the type of the message. If there is a custom binding that can be done at application startup to be able to do this, that should be ok; Please give me some samples on how this can be achieved.
There is no such support in Spring Cloud Stream.
The underlying Spring for Apache Kafka project does have support for such scenarios.
See #KafkaListener on a Class.
It requires the payload to have been deserialized by the Kafka deserializer; then the method called depends on the payload type.
It also supports a fallback "default" method.
#KafkaListener(id = "multi", topics = "myTopic")
static class MultiListenerBean {
#KafkaHandler
public void listen(String foo) {
...
}
#KafkaHandler
public void listen(Integer bar) {
...
}
#KafkaHandler(isDefault = true)
public void listenDefault(Object object) {
...
}
}

Reactive Programming: Spring WebFlux: How to build a chain of micro-service calls?

Spring Boot Application:
a #RestController receives the following payload:
{
"cartoon": "The Little Mermaid",
"characterNames": ["Ariel", "Prince Eric", "Sebastian", "Flounder"]
}
I need to process it in the following way:
Get the unique Id for each character name: make an HTTP call to "cartoon-characters" microservice, that returns ids by names
Transform data received by the controller:
replace character names with appropriate ids that were received on the previous step from "cartoon-characters" microservice.
{
"cartoon": "The Little Mermaid",
"characterIds": [1, 2, 3, 4]
}
Send an HTTP POST request to "cartoon-db" microservice with transformed data.
Map the response from "cartoon-db" to the internal representation that is the controller return value.
The problem that I got:
I need to implement all these steps using the paradigm of Reactive Programming (non-blocking\async processing) with Spring WebFlux (Mono|Flux) and Spring Reactive WebClient - but I have zero experience with that stack, trying to read about it as much as I can, plus googling a lot but still, have a bunch of unanswered questions, for example:
Q1. I have already configured reactive webClient that sends a request to "cartoon-characters" microservice:
public Mono<Integer> getCartoonCharacterIdbyName(String characterName) {
return WebClient.builder().baseUrl("http://cartoon-characters").build()
.get()
.uri("/character/{characterName}", characterName)
.retrieve()
.bodyToMono(Integer.class);
}
As you may see, I have got a list of cartoon character names and for each of them I need to call getCartoonCharacterIdbyName(String name) method, I am not sure that the right option to call it in series, believe the right option: parallel execution.
Wrote the following method:
public List<Integer> getCartoonCharacterIds(List<String> names) {
Flux<Integer> flux = Flux.fromStream(names.stream())
.flatMap(this::getCartoonCharacterIdbyName);
return StreamSupport.stream(flux.toIterable().spliterator(), false)
.collect(Collectors.toList());
}
but I have doubts, that this code does parallel WebClient execution and also, code calls flux.toIterable() that block the thread, so with this implementation I lost non-blocking mechanism.
Are my assumptions correct?
How do I need to rewrite it to having parallelism and non-blocking?
Q2.
Is it technically possible to transform input data received by the controller (I mean replace names with ids) in reactive style: when we operate with Flux<Integer> characterIds, but not with the List<Integer> of characterIds?
Q3. Is it potentially possible to get not just transformed Data object, but Mono<> after step 2 that can be consumed by another WebClient in Step 3?
Actually it's a good question since understanding the WebFlux, or project reactor framework, when it comes to chaining micro-services requires a couple of steps.
The first is to realize that a WebClient should take a publisher in and return a publisher. Extrapolate this to 4 different method signatures to help with thinking.
Mono -> Mono
Flux -> Flux
Mono -> Flux
Flux -> Mono
For sure, in all cases, it is just Publisher->Publisher, but leave that until you understand things better. The first two are obvious, and you just use .map(...) to handle objects in the flow, but you need to learn how to handle the second two. As commented above, going from Flux->Mono could be done with .collectList(), or also with .reduce(...). Going from Mono->Flux seems to generally be done with .flatMapMany or .flatMapIterable or some variation of that. There are probably other techniques. You should never use .block() in any WebFlux code, and generally you will get a runtime error if you try to do so.
In your example you want to go to
(Mono->Flux)->(Flux->Flux)->(Flux->Flux)
As you said, you want
Mono->Flux->Flux
The second part is to understand about chaining Flows. You could do
p3(p2(p1(object)));
Which would chain p1->p2->p3, but I always found it more understandable to make a "Service Layer" instead.
o2 = p1(object);
o3 = p2(o2);
result = p3(o3);
This code is just much easier to read and maintain and, with some maturity, you come to understand the worth of that statement.
The only problem I had with your example was doing a Flux<String> with WebClient as a #RequestBody. Doesn't work. See WebClient bodyToFlux(String.class) for string list doesn't separate individual values. Other than that, it's a pretty straightforward application. You'll find when you debug it that it gets to the .subscribe(System.out::println) line before it gets to the Flux<Integer> ids = mapNamesToIds(fn) line. This is because the Flow is setup before it is executed. Takes a while to understand this but it is the point of the project reactor framework.
#SpringBootApplication
#RestController
#RequestMapping("/demo")
public class DemoApplication implements ApplicationRunner {
public static void main(String[] args) {
SpringApplication.run(DemoApplication.class, args);
}
Map<Integer, CartoonCharacter> characters;
#Override
public void run(ApplicationArguments args) throws Exception {
String[] names = new String[] {"Ariel", "Prince Eric", "Sebastian", "Flounder"};
characters = Arrays.asList( new CartoonCharacter[] {
new CartoonCharacter(names[0].hashCode(), names[0], "Mermaid"),
new CartoonCharacter(names[1].hashCode(), names[1], "Human"),
new CartoonCharacter(names[2].hashCode(), names[2], "Crustacean"),
new CartoonCharacter(names[3].hashCode(), names[3], "Fish")}
)
.stream().collect(Collectors.toMap(CartoonCharacter::getId, Function.identity()));
// TODO Auto-generated method stub
CartoonRequest cr = CartoonRequest.builder()
.cartoon("The Little Mermaid")
.characterNames(Arrays.asList(names))
.build();
thisLocalClient
.post()
.uri("cartoonDetails")
.body(Mono.just(cr), CartoonRequest.class)
.retrieve()
.bodyToFlux(CartoonCharacter.class)
.subscribe(System.out::println);
}
#Bean
WebClient localClient() {
return WebClient.create("http://localhost:8080/demo/");
}
#Autowired
WebClient thisLocalClient;
#PostMapping("cartoonDetails")
Flux<CartoonCharacter> getDetails(#RequestBody Mono<CartoonRequest> cartoonRequest) {
Flux<StringWrapper> fn = cartoonRequest.flatMapIterable(cr->cr.getCharacterNames().stream().map(StringWrapper::new).collect(Collectors.toList()));
Flux<Integer> ids = mapNamesToIds(fn);
Flux<CartoonCharacter> details = mapIdsToDetails(ids);
return details;
}
// Service Layer Methods
private Flux<Integer> mapNamesToIds(Flux<StringWrapper> names) {
return thisLocalClient
.post()
.uri("findIds")
.body(names, StringWrapper.class)
.retrieve()
.bodyToFlux(Integer.class);
}
private Flux<CartoonCharacter> mapIdsToDetails(Flux<Integer> ids) {
return thisLocalClient
.post()
.uri("findDetails")
.body(ids, Integer.class)
.retrieve()
.bodyToFlux(CartoonCharacter.class);
}
// Services
#PostMapping("findIds")
Flux<Integer> getIds(#RequestBody Flux<StringWrapper> names) {
return names.map(name->name.getString().hashCode());
}
#PostMapping("findDetails")
Flux<CartoonCharacter> getDetails(#RequestBody Flux<Integer> ids) {
return ids.map(characters::get);
}
}
Also:
#Data
#NoArgsConstructor
#AllArgsConstructor
#Builder
public class StringWrapper {
private String string;
}
#Data
#Builder
public class CartoonRequest {
private String cartoon;
private List<String> characterNames;
}
#Data
#Builder
#NoArgsConstructor
#AllArgsConstructor
public class CartoonCharacter {
Integer id;
String name;
String species;
}

Spring Integration: Manual channel handling

What I want: Build a configurable library that
uses another library that has an internal routing and a subscribe method like: clientInstance.subscribe(endpoint, (endpoint, message) -> <handler>) , e.g. Paho MQTT library
later in my code I want to access the messages in a Flux.
My idea:
create MessageChannels like so:
integrationFlowContext
.registration(IntegrationFlows.from("message-channel:" + endpoint)).bridge().get())
.register()
forward to reactive publishers:
applicationContext.registerBean(
"publisher:" + endpoint,
Publisher.class,
() -> IntegrationFlows.from("message-channel:" + endpoint)).toReactivePublisher()
);
keep the message channels in a set or similar and implement the above handler: (endpoint, message) -> messageChannels.get(endpoint).send( <converter>(message))
later use (in a #PostConstruct method):
Flux
.from((Publihser<Message<?>>)applicationContext.getBean("publisher:" + enpoint))
.map(...)
.subscribe()
I doubt this to be the best way to do what I want. Feels like abusing spring integration. Any suggestions are welcome at this point.
In general however (at least in my tests) this seemed to be working. But when I run my application, I get errors like: "Caused by: org.springframework.messaging.core.DestinationResolutionException: no output-channel or replyChannel header available".
This is especially bad, since after this exception the publishers claim to not have a subscriber anymore. Thus, in a real application no messages are proceeded anymore.
I am not sure what this message means, but I can kind of reproduce it (but don't understand why):
#Test
public void channelTest() {
integrationFlowContext
.registration(
IntegrationFlows.from("any-channel").bridge().get()
)
.register();
registryUtil.registerBean(
"any-publisher",
Publisher.class,
() -> IntegrationFlows.from("any-channel").toReactivePublisher()
);
Flux
.from((Publisher<Message<?>>) applicationContext.getBean("any-publisher"))
.subscribe(System.out::println);
MessageChannel messageChannel = applicationContext.getBean("any-channel", MessageChannel.class);
try {
messageChannel.send(MessageBuilder.withPayload("test").build());
} catch (Throwable t) {
log.error("Error: ", t);
}
}
I of course read parts of the spring integration documentation, but don't quite get what happens behind the scenes. Thus, I feel like guessing possible error causes.
EDIT:
This, however works:
#TestConfiguration
static class Config {
GenericApplicationContext applicationContext;
Config(
GenericApplicationContext applicationContext,
IntegrationFlowContext integrationFlowContext
) {
this.applicationContext = applicationContext;
// optional here, but needed for some reason in my library,
// since I can't find the channel beans like I will do here,
// if I didn't register them like so:
//integrationFlowContext
// .registration(
// IntegrationFlows.from("any-channel").bridge().get())
// .register();
applicationContext.registerBean(
"any-publisher",
Publisher.class,
() -> IntegrationFlows.from("any-channel").toReactivePublisher()
);
}
#PostConstruct
void connect(){
Flux
.from((Publisher<Message<?>>) applicationContext.getBean("any-publisher"))
.subscribe(System.out::println);
}
}
#Autowired
ApplicationContext applicationContext;
#Autowired
IntegrationFlowContext integrationFlowContext;
#Test
#SneakyThrows
public void channel2Test() {
MessageChannel messageChannel = applicationContext.getBean("any-channel", MessageChannel.class);
try {
messageChannel.send(MessageBuilder.withPayload("test").build());
} catch (Throwable t) {
log.error("Error: ", t);
}
}
Thus apparently my issue above is realted to messages arriving "too early" .. I guess?!
No, your issue is related to round-robin dispatched on the DirectChannel for the any-channel bean name.
You define two IntegrationFlow instances starting with that channel and then you declare their own subscribers, but at runtime both of them are subscribed to the same any-channel instance. And that one comes with the round-robin balancer by default. So, one message goes to your Flux.from() subscriber, but another to that bridge() which doesn't know what to do with your message, so it tries to resolve a replyChannel header.
Therefore your solution just only with one IntegrationFlows.from("any-channel").toReactivePublisher() is correct. Although you could just do a FluxMessageChannel registration and use it from one side for regular messages sending and from other side as a reactive source for Flux.from().

Validating Spring Kafka payloads

I am trying to set up a service that has both a REST (POST) endpoint and a Kafka endpoint, both of which should take a JSON representation of the request object (let's call it Foo). I would want to make sure that the Foo object is valid (via JSR-303 or whatever). So Foo might look like:
public class Foo {
#Max(10)
private int bar;
// Getter and setter boilerplate
}
Setting up the REST endpoint is easy:
#PostMapping(value = "/", produces = MediaType.APPLICATION_JSON_VALUE)
public ResponseEntity<String> restEndpoint(#Valid #RequestBody Foo foo) {
// Do stuff here
}
and if I POST, { "bar": 9 } it processes the request, but if I post: { "bar": 99 } I get a BAD REQUEST. All good so far!
The Kafka endpoint is easy to create (along with adding a StringJsonMessageConverter() to my KafkaListenerContainerFactory so that I get JSON->Object conversion:
#KafkaListener(topics = "fooTopic")
public void kafkaEndpoint(#Valid #Payload Foo foo) {
// I shouldn't get here with an invalid object!!!
logger.debug("Successfully processed the object" + foo);
// But just to make sure, let's see if hand-validating it works
Validator validator = localValidatorFactoryBean.getValidator();
Set<ConstraintViolation<SlackMessage>> errors = validator.validate(foo);
if (errors.size() > 0) {
logger.debug("But there were validation errors!" + errors);
}
}
But no matter what I try, I can still pass invalid requests in and they process without error.
I've tried both #Valid and #Validated. I've tried adding a MethodValidationPostProcessor bean. I've tried adding a Validator to the KafkaListenerEndpointRegistrar (a la the EnableKafka javadoc):
#Configuration
public class MiscellaneousConfiguration implements KafkaListenerConfigurer {
private Logger logger = LoggerFactory.getLogger(this.getClass());
#Autowired
LocalValidatorFactoryBean validatorFactory;
#Override
public void configureKafkaListeners(KafkaListenerEndpointRegistrar registrar) {
logger.debug("Configuring " + registrar);
registrar.setMessageHandlerMethodFactory(kafkaHandlerMethodFactory());
}
#Bean
public MessageHandlerMethodFactory kafkaHandlerMethodFactory() {
DefaultMessageHandlerMethodFactory factory = new DefaultMessageHandlerMethodFactory();
factory.setValidator(validatorFactory);
return factory;
}
}
I've now spent a few days on this, and I'm running out of other ideas. Is this even possible (without writing validation into every one of my kakfa endpoints)?
Sorry for the delay; we are at SpringOne Platform this week.
The infrastructure currently does not pass a Validator into the payload argument resolver. Please open an issue on GitHub.
Spring kafka listener by default do not scan for #Valid for non Rest controller classes. For more details please refer this answer
https://stackoverflow.com/a/71859991/13898185

Resources