Does Spring Cloud Stream Kafka supports embedded headers? - spring

According to this topic:
Kafka Spring Integration: Headers not coming for kafka consumer -
this is no headers support for Kafka
But documentation says:
spring.cloud.stream.kafka.binder.headers
The list of custom headers that will be transported by the binder.
Default: empty.
I can't get it working with spring-cloud-stream-binder-kafka: 1.2.0.RELEASE
SENDING LOG:
MESSAGE (e23885fd-ffd9-42dc-ebe3-5a78467fee1f) SENT :
GenericMessage [payload=...,
headers={
content-type=application/json,
correlationId=51dd90b1-76e6-4b8d-b667-da25f214f383,
id=e23885fd-ffd9-42dc-ebe3-5a78467fee1f,
contentType=application/json,
timestamp=1497535771673
}]
RECEIVING LOG:
MESSAGE (448175f5-2b21-9a44-26b9-85f093b33f6b) RECEIVED BY HANDLER 1:
GenericMessage [payload=...,
headers={
kafka_offset=36,
id=448175f5-2b21-9a44-26b9-85f093b33f6b,
kafka_receivedPartitionId=0,
contentType=application/json;charset=UTF-8,
kafka_receivedTopic=new_patient, timestamp=1497535771715
}]
MESSAGE (448175f5-2b21-9a44-26b9-85f093b33f6b) RECEIVED BY HANDLER 2 :
GenericMessage [payload=...,
headers={
kafka_offset=36,
id=448175f5-2b21-9a44-26b9-85f093b33f6b,
kafka_receivedPartitionId=0,
contentType=application/json;charset=UTF-8,
kafka_receivedTopic=new_patient, timestamp=1497535771715
}]
I expect to see the same message id and get correlationId on receiving side.
application.properties:
spring.cloud.stream.kafka.binder.headers=correlationId
spring.cloud.stream.bindings.newTest.destination=new_test
spring.cloud.stream.bindings.newTestCreated.destination=new_test
spring.cloud.stream.default.consumer.headerMode=embeddedHeaders
spring.cloud.stream.default.producer.headerMode=embeddedHeaders
SENDING MESSAGE:
#Publisher(channel = "testChannel")
public Object newTest(Object param) {
...
return myObject;
}

Yes, it does: http://docs.spring.io/spring-cloud-stream/docs/Chelsea.SR2/reference/htmlsingle/index.html#_consumer_properties
headerMode
When set to raw, disables header parsing on input. Effective only for messaging middleware that does not support message headers natively and requires header embedding. Useful when inbound data is coming from outside Spring Cloud Stream applications.
Default: embeddedHeaders
But that is already Spring Cloud Stream story, not Spring Kafka per se.

Related

rSocket websocket postman testing mime types and endpoints

I am using spring-boot-starter-webflux and spring-boot-starter-rsocket version 2.7.1
The rSocket transport is set to websocket like this:
spring.rsocket.server.transport=websocket
spring.rsocket.server.mapping-path=/rsocket
# this setting has no effect when transport==WEBSOCKET
spring.rsocket.server.port=7000
There's a spring #Controller endpoint #MessageMapping setup for a simple string like:
#MessageMapping("test")
String test() {
Logs.Info("*** Received test ***");
return "tested";
}
I want to get a successful test done with Postman. Run the spring boot app locally and connect to ws://localhost:7000 using mime types
dataMimeType: 'application/json'
metadataMimeType: 'message/x.rsocket.routing.v0'
Like this:
The rsocket websocket connects, but I can't hit the endpoint test
With error 1005 No Status Received: Missing status code even though one was expected
On the server the error is
DEBUG [reactor-http-nio-2] debug: [c4e97d34-1, L:/127.0.0.1:7000 - R:/127.0.0.1:2051] Cancelling Websocket inbound. Closing Websocket
DEBUG [reactor-http-nio-2] debug: [c4e97d34, L:/127.0.0.1:7000 - R:/127.0.0.1:2051] Removed handler: PongHandler, pipeline: DefaultChannelPipeline{(wsencoder = io.netty.handler.codec.http.websocketx.WebSocket13FrameEncoder), (wsdecoder = io.netty.handler.codec.http.websocketx.WebSocket13FrameDecoder), (reactor.right.reactiveBridge = reactor.netty.channel.ChannelOperationsHandler)}
DEBUG [reactor-http-nio-2] debug: [c4e97d34, L:/127.0.0.1:7000 ! R:/127.0.0.1:2051] An outbound error could not be processed
java.nio.channels.ClosedChannelException
at reactor.core.publisher.MonoErrorSupplied.call(MonoErrorSupplied.java:61)
at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:228)
at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onComplete(MonoIgnoreThen.java:203)
at reactor.core.publisher.SinkEmptyMulticast$VoidInner.complete(SinkEmptyMulticast.java:238)
at reactor.core.publisher.SinkEmptyMulticast.tryEmitEmpty(SinkEmptyMulticast.java:70)
at reactor.core.publisher.SinkEmptySerialized.tryEmitEmpty(SinkEmptySerialized.java:46)
What's the incorrect setting in Postman?
The answer is don't use postman. Rsocket is a binary protocol, Even though based on Websocket, There are many tools test it.
use spring message write a unit test
#Autowired
private RSocketRequester rSocketRequester;
StepVerifier.create(rSocketRequester
.route("test")
.retrieveMono(String.class))
.expectNext("tested")
.verifyComplete();
RSocket Client CLI (RSC)
rsc --request --route=test --debug ws://localhost:7000/rsocket
Actually the following message was received:
{
"data":"test",
"metadata":4
}
Per screenshot
But now the error on the server side is:
DEBUG [reactor-http-nio-6] lambda$receive$0: receiving ->
Frame => Stream ID: 2064452128 Type: REQUEST_N Flags: 0b100000 Length: 42
RequestN: 539124833
Data:
DEBUG [reactor-http-nio-6] sendErrorAndClose: sending -> InvalidSetupException: SETUP or RESUME frame must be received before any others

Spring webflux websocket closed when receive message too fast

enter code herepublic Mono<Void> handle(#Nonnull WebSocketSession session) {
final WebSocketContext webSocketContext = new WebSocketContext(session);
Mono<Void> output = session.send(Flux.create(webSocketContext::setSink));
Mono<Void> input = session.receive()
.timeout(Duration.ofSeconds(adapterProperties.getSessionTimeout()))
.doOnSubscribe(subscription -> subscription.request(64))
.doOnNext(WebSocketMessage::retain)
.publishOn(Schedulers.boundedElastic())
.concatMap(msg -> {
// ....blocking operation
return Flux.empty();
}).then();
return Mono.zip(input, output).then();
When I use ws client to send information, because the message sending speed is too fast, when 2000 pieces of data are received, the connection is disconnected and there is no exception message,
After I slow down the sending speed of the message on the client side, there is no problem. How can I solve it?
Below is the Flux log information:
2022-07-14 17:19:40.295 adapter-iat [boundedElastic-5] INFO reactor.Flux.PublishOn.5 - | onNext(WebSocket TEXT message (13765 bytes))
2022-07-14 17:19:40.296 adapter-iat [boundedElastic-5] INFO reactor.Flux.PublishOn.5 - | onNext(WebSocket TEXT message (13765 bytes))
2022-07-14 17:19:40.300 adapter-iat [boundedElastic-5] INFO reactor.Flux.PublishOn.5 - | onComplete()????why

Conversion by Spring Cloud Stream of Message with pojo on consumer side returns object with null fields

Here is code snippets to demonstrate the problem.
On producer side:
ProductModelDto dto = new ProductModelDto(1L, "name", "desc", 100.5);
streamBridge.send(PRODUCTS_OUT, MessageBuilder.withPayload(dto).build());
On consumer side:
#Bean
public Consumer<Message<ProductModelDto>> productCreated() {
return message -> {
log.info("payload = {}", message.getPayload());
log.info("payload class = {}", message.getPayload().getClass().getName());
log.info("sourceData = {}", message.getHeaders().get("sourceData"));
};
}
Output:
payload = ProductModelDto(id=null, name=null, description=null, price=null)
payload class = ru.security.common.model.product.ProductModelDto
sourceData = (Body:'{"id":1,"name":"name","description":"desc","price":100.5}' MessageProperties [headers={}, timestamp=Tue Sep 07 11:13:02 MSK 2021, messageId=3840075f-1142-f94d-be37-7be950d73f54, contentType=application/json, contentLength=0, receivedDeliveryMode=PERSISTENT, priority=0, redelivered=false, receivedExchange=product-service.product-created-or-changed, receivedRoutingKey=product-service.product-created-or-changed, deliveryTag=1, consumerTag=amq.ctag-3i5ECkRTPs5_O5ZW5af-MA, consumerQueue=product-service.product-created-or-changed.some-group4])
I expect to receive payload which have been sent by producer. But resulted payload is ProductModelDto(id=null, name=null, description=null, price=null).
I know that spring automatically convert message to pojo if I use
Consumer<ProductModelDto> productCreated()
but I need
Consumer<Message<ProductModelDto>> productCreated()
to get Headers from message. Any suggestion where I miss some configuration?
I've created sample project and found out that this issue is reproduced in version 3.0.11.RELEASE of spring-cloud-starter-stream-rabbit. In versions 3.1.4 or 3.0.10.RELEASE i didn't see this problem. https://github.com/serjteplov/demo-gitter-scs1.git

How to send message to a specific subscription over spring websocket STOMP?

I am using STOMP over websockets with spring boot. Is there a possibility to send a message to a specific subscription? I subscribe to the STOMP endpoints using a STOMP header containing an id field according to the stomp documentation I want this id to be used to determine the clients who should receive a message, but spring seems not to use this id. I can not just use sendToUser because two clients can have the same user id e.g. if a user has two opened browser windows. Only one specific window should receive the message.
In the following example I have two connected clients which are using the same user, but different ids in the STOMP header.
Client1-ID: a32d66bf-03c7-47a4-aea0-e464c0727842
Client2-ID: b3673d33-1bf2-461e-8df3-35b7af07371b
In spring I have executed the following Kotlin code:
val subscriptions = userRegistry.findSubscriptions {
it.destination == "/user/topic/operations/$operationId/runs"
}
subscriptions.forEach{
println("subscription id: ${it.id}");
println("session id: ${it.session.id}");
println("user id ${it.session.user.name}");
}
The output:
subscription id: sub-7
session id: mcjpgn2i
user id 4a27ef88-25eb-4175-a872-f46e7b9d0564
subscription id: sub-7
session id: 0dxuvjgp
user id 4a27ef88-25eb-4175-a872-f46e7b9d0564
There is no sign of the id I have passed to the stomp header.
Is it possible to send a message to one specific subscription determined by the id I have passed with the header?
I got it working.
First of all, there was something wrong with my client setup. I have set the subscription id in the connection header like this:
this.stompClient.webSocketFactory = (): WebSocket => new SockJS("/ws");
this.stompClient.connectHeaders = { id: subscriptionId };
this.stompClient.activate();
But the subscription header has to be set in the subscription header:
this.stompClient.subscribe(this.commonEndpoint,
this.onMessageReceived.bind(this),
{ id: subScriptionId });
If I do this, spring is correctly using this id as the subscription id, instead of using some default like sub-7.
According to that thread I can send messages to specific sessions instead of user.
With the following code I can send a message to a specific subscription:
val subscriptions = userRegistry.findSubscriptions {
it.destination == "/user/topic/operations/$operationId/runs"
}
subscriptions.forEach {
if(it.id === mySubscriptionId){
val headerAccessor =
SimpMessageHeaderAccessor.create(SimpMessageType.MESSAGE)
headerAccessor.sessionId = it.session.id
headerAccessor.setLeaveMutable(true)
simpMessagingTemplate.convertAndSendToUser(it.session.id,
"/topic/operations/runs", messageResponseEntity,
headerAccessor.getMessageHeaders())
}
}
I'm able to send message to a specific subscription using SimpMessagingTemplate.
#Autowired
private final SimpMessagingTemplate messagingTemplate;
public void sendMessage(String simpUserId, String destination, String message) {
try {
messagingTemplate.convertAndSendToUser(simpUserId, destination, message);
} catch (Exception ex) {
LOG.error("Exception occurred while sending message [message={}]", ex.getMessage(), ex);
}
}
You may refer SimpMessagingTemplate.convertAndSendToUser

Using Publish Subscribe channel in Spring Integration

We have the below XML configuration for a rest service and Oracle Stored Proc. The use case is about handling Oracle SP errors.
When the Oracle SP throws error, we need to achieve 2 things:
we need to send a json response back to the client in the format
{"status": false,"message": "Oracle SP error payload"}
Send an email out.
The configuration is working as expected when Oracle SP throws error. But when the email send fails ( gave a wrong host), client is receiving error in the below format. The Oracle SP error message is not sent back.
---- Json message received by client when email send fails:------
{
"timestamp": 1501639940143,
"status": 500,
"error": "Internal Server Error",
"exception": "org.springframework.messaging.MessagingException",
"message": "failure occurred in error-handling flow; nested exception is org.springframework.messaging.MessageHandlingException: error occurred in message handler [org.springframework.integration.handler.MessageHandlerChain#0$child#2.handler]; nested exception is org.springframework.mail.MailSendException: Mail server connection failed; nested exception is javax.mail.MessagingException: Unknown SMTP host: xxx.com;..",
"path": "/init/5c82d1f4-e550-4bb1-be9c-8d0ddbc7f4401/NEW"
}
How do I send response in the expected json format - {"status": false,"message": Oracle SP error payload}?
Also how to handle the email send error.i.e may be how to store the email error in a DB table?
XML Configuration:
<int:channel id="spOutClobChannel"/>
<int:publish-subscribe-channel id="gatewayErrorChannel"/>
<int-http:inbound-gateway
request-channel="gatewayInitChannel"
error-channel="gatewayErrorChannel"
supported-methods="GET"
path="/init/{refId}/{refStatus}"
payload-expression="#pathVariables.refStatus">
<int-http:header name="UNIQUE_PROCESS_ID" expression="#pathVariables.refId"/>
</int-http:inbound-gateway>
<int-jdbc:stored-proc-outbound-gateway
id="sp-get-data"
data-source="dataSource"
request-channel="gatewayInitChannel"
reply-channel="spOutClobChannel".... >
...
...
</int-jdbc:stored-proc-outbound-gateway>
<int:transformer input-channel="spOutClobChannel" expression='{"status": true, "message": "RECEIVED"}'/>
<int:transformer input-channel="gatewayErrorChannel" expression='{"status": false,"message": payload.message}'/>
<int:transformer expression="payload.failedMessage.headers['UNIQUE_PROCESS_ID']"
input-channel="gatewayErrorChannel" output-channel="appEmailChannel"/>
<int:chain input-channel="appEmailChannel" >
<int-mail:header-enricher>
<int-mail:from value="${email.from}"/>
<int-mail:to value="${email.to}"/>
<int-mail:subject value="${email.subject}"/>
<int-mail:content-type value="text/html"/>
</int-mail:header-enricher>
<int-mail:outbound-channel-adapter host="${email.host}" />
</int:chain>
To avoid errors thrown from the appEmailChannel flow, plus get a gain do not wait for the successful email shipment, it would be better to add to your gatewayErrorChannel an executor configuration. This way all the subscribers to this publish-subscriber channel will be performed in parallel and in their own threads. Therefore any exception in those threads won't impact the reply to the <int-http:inbound-gateway> which you've done with the
<int:transformer input-channel="gatewayErrorChannel" expression='{"status": false,"message": payload.message}'/>
Also how to handle the email send error.i.e may be how to store the email error in a DB table?
With this goal you can use ExpressionEvaluatingRequestHandlerAdvice with its trapException = true and failureChannel properties.
This advice can be configured with the request-handler-advice-chain sub-element.
See Reference Manual and Sample on the matter.

Resources