I use Spring Cloud Stream 1.3.2.RELEASE to publish a String message to Kafka. When I consume the message using command line Kafka consumer or Spring Kafka #KafkaListener, a contentType header is always appended to the message body.
Question:
Is there any way to get rid of the embedded headers?
--
Spring Cloud Stream as producer
private void send() {
channel.test().send(MessageBuilder.withPayload("{\"foo\":\"bar\"}").build());
}
Command line Kafka consumer
$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test
�
contentType
"text/plain"{"foo":"bar"}
Spring Kafka as consumer
#KafkaListener(topics = "test")
public void receive(Message message){
log.info("Message payload received: {}", message.getPayload());
}
2018-05-16 07:12:05.241 INFO 19475 --- [ntainer#0-0-C-1] com.demo.service.Listener : Message payload received: �contentType"text/plain"{"foo":"bar"}
#KafkaListener(topics = "test")
public void receive(#Payload String message){
log.info("Message payload received: {}", message);
}
2018-05-16 07:16:14.313 INFO 19747 --- [ntainer#0-0-C-1] com.demo.service.Listener : Message payload received: �contentType"text/plain"{"foo":"bar"}
See headerMode binding property: https://docs.spring.io/spring-cloud-stream/docs/Ditmars.SR3/reference/htmlsingle/#_properties_for_use_of_spring_cloud_stream. You need to set it to raw for the destination you send messages.
Related
Please can someone help me to understand where is the probleme in this config:
Versions :
org.springframework.integration:spring-integration-mqtt:5.5.2
org.springframework.boot:spring-boot-starter:2.5.3
org.eclipse.paho:org.eclipse.paho.client.mqttv3:1.2.5
#Configuration
public class MqttConfig {
#Bean
public MqttPahoClientFactory mqttClientFactory() {
DefaultMqttPahoClientFactory factory = new DefaultMqttPahoClientFactory();
MqttConnectOptions options = new MqttConnectOptions();
options.setServerURIs(new String[] { "tcp://localhost:1883" });
return factory;
}
#Bean
public MqttPahoMessageDrivenChannelAdapter inboundAdapter(MqttPahoClientFactory clientFactory) {
return new MqttPahoMessageDrivenChannelAdapter("MyApp", clientFactory, "ReplyTopic");
}
#Bean
IntegrationFlow inboundFlow(MqttPahoMessageDrivenChannelAdapter inboundAdapter) {
return IntegrationFlows.from(inboundAdapter)
.bridge()
.channel("replyChannel")
.get();
}
#Bean
public MessageChannel replyChannel() {
return MessageChannels.publishSubscribe().get();;
}
#Bean
public MqttPahoMessageHandler outboundAdapter(MqttPahoClientFactory clientFactory) {
return new MqttPahoMessageHandler("MyApp", clientFactory);
}
#Bean
public IntegrationFlow outboundFlow(MqttPahoMessageHandler outboundAdapter) {
return IntegrationFlows.from("requestChannel")
.handle(outboundAdapter).get()
}
#MessagingGateway
public interface MyGateway {
#Gateway(requestChannel = "requestChannel", replyChannel = "replyChannel")
String send(String request, #Header(MqttHeaders.TOPIC) String requestTopic);
}
}
Client code
#RestController
public class MyController {
#Autowired
private MyGateway myGateway;
#GetMapping("/sendRequest")
public String sendRequest() {
var response = myGateway.send("Hello", "MyTopic");
return response;
}
}
Usage:
curl http://localhost:8080/sendRequest
manual response from the mqtt broker (HiveMQ)
docker exec -it hivemq mqtt pub -t ReplyTopic -m "World" --debug
CLIENT mqttClient-MQTT_5_0-9ecded84-8416-4baa-a8f3-d593c692bc65: acknowledged PUBLISH: 'World' for PUBLISH to Topic: ReplyTopic
But I dont know why i have this message on the Spring application output
2022-10-25 18:04:33.171 ERROR 17069 --- [T Call: MyApp] .m.i.MqttPahoMessageDrivenChannelAdapter : Unhandled exception for GenericMessage [payload=World, headers={mqtt_receivedRetained=false, mqtt_id=0, mqtt_duplicate=false, id=9dbd5e14-66ed-5dc8-6cea-6d04ef19c6cc, mqtt_receivedTopic=ReplyTopic, mqtt_receivedQos=0, timestamp=1666713873170}]
org.springframework.messaging.MessageHandlingException: error occurred in message handler [org.springframework.integration.handler.BridgeHandler#6f63903c]; nested exception is org.springframework.messaging.core.DestinationResolutionException: no output-channel or replyChannel header available
Please can someone explain why i have this ?
no output-channel or replyChannel header available
I think the problem you are facing is not related to your bridge() configuration.
This comes from the MessagingGatewaySupport and its replyMessageCorrelator feature which is activated by your replyChannel = "replyChannel".
The real problem that you are trying to do what is not possible with MQTT v3. There is just no headers transferring over MQTT broker to carry on a required for gateway initiator a correlation key - the TemporaryReplyChannel. See more in docs about gateway: https://docs.spring.io/spring-integration/docs/current/reference/html/messaging-endpoints.html#gateway.
In other words: independently of the replyChannel configuration on gateway, the replyChannel header must be present in the reply message. This is the way how gateway correlates requests with replies.
You have to look into an aggregator to send the request message in parallel and to preserve the mentioned TemporaryReplyChannel header. Then when you receive a reply (inboundAdapter) you send it to this aggregator. You need to ensure some correlation key from a request and reply payload, so they can match and fulfill group for reply to be sent back to the gateway.
See more info in docs: https://docs.spring.io/spring-integration/docs/current/reference/html/message-routing.html#aggregator
I am using the spring-cloud-starter-aws-messaging in my Spring Boot application and #sqslistener to consume SQS messages. my consumer weirdly stops getting messages out of nowhere and the ApproximateNumberOfMessagesVisible gradually increases triggering a CloudWatch alarm. I could see no error logs that are generated before it stops getting any more messages. Am I missing something?
#SqsListener(value = "${sqs.queue.url.indexSavedSetQueue}",
deletionPolicy = SqsMessageDeletionPolicy.ON_SUCCESS)
public void listenIndexSavedSetEvent(#NonNull String message) throws IOException {
log.info("Index saved set event received, message: {}", message);
IndexSavedSetPayloadDto indexSavedSetPayloadDto = objectMapper
.readValue(message, IndexSavedSetPayloadDto.class);
String setName = indexSavedSetPayloadDto.getSetName();
indexerService.indexSet(setName);
}
I'm Sending request to the Third party IBM MQ with space pattern, like below text (COBOL input file), but i'm getting reponse as a error in Reply queue like below but same input working fine spring jms, i'm facing this problem with apache camel
Sample Input:
GetProducerWithCommissionddRates
C26115
77104 99998
2010-01-01 011
Response:
printing data in controller:: EPPRD02007EAn invalid COBOL Date Format
was provided as input to the Producer Retrieval Service, Get Producer
With Commission Rates Operation: AsOfDate. Source:
(ProducerMediationModule)
ProducerMediationModule
Controller with camel:
#PostMapping("/request-reply")
public String requestReplyMapping(#RequestBody String inputReq) {
Exchange exchangeRequest = ExchangeBuilder.anExchange(camelContext).withBody(inputReq).build();
Exchange exchangeResponse = producer.send("direct:request-reply", exchangeRequest);
return exchangeResponse.getIn().getBody(String.class);
}
Camel Route:
#Component
public class RequestReplyRouter extends RouteBuilder {
#Override
public void configure() throws Exception {
from("direct:request-reply").
to("jms:REQUEST.Q1?ReplyTo=REPLY.Q1&exchangePattern=InOut")
.log("Request-reply Body is ${body} and header info is ${headers} ");
from("jms:REPLY.Q1")
.log("Received Body is ${body} and header info is ${headers} ");
}
}
JMS Controller Code(working) :
#PostMapping(value="request-reply")
public ResponseEntity<String> sendRequestAndReceiveReply (#RequestBody String prd2x4Request) { System.out.println("received prd2x4 request: "+prd2x4Request);
String reply = prd2x4Wrapper.sendReqAndGetMessageId(prd2x4Request);
return new ResponseEntity<String>(reply, HttpStatus.OK);
}
JMS MQ Calling (working) :
private String sendReqAndGetMessage(String prd2x4Request) {
String messageId = "";
MQQueue requestQueue = new MQQueue(RequestQueue);
Session session = jmsTemplate.getConnectionFactory().createConnection().createSession(); MessageProducer mp = session.createProducer(requestQueue);
TextMessage message = session.createTextMessage(prd2x4Request);
message.setStringProperty("JMS_IBM_Character_Set", "IBM037");
mp.send(message);
messageId = message.getJMSMessageID();
mp.close();
session.close();
return messageId;
}
I am experimenting with the Spring Webflux and Spring Integration to create a reactive stream (Flux) from a JMS queue.
I am attempting to create a reactive stream (Spring Webflux) from a JMS queue (IBM MQ using Spring Integration) for clients to get the JMS messages asynchronously. I believe that I have everything hooked up correctly as the messages are getting consumed by the reactive listener. However my reactive flux stream is not able to dsiplay those messages.
Any help would be appreciated.
Here is the code which I am using to make my JMS listener reactive :
UM Gateway
#Named
#Slf4j
public class UmGateway {
#Autowired
private ConnectionFactory connectionFactory;
#Autowired
private JmsTemplate jmsTemplate;
#Value("${um.mq.queueName}")
private String queueName;
#Bean
public Publisher<Message<MilestoneNotification>> jmsReactiveSource() {
return IntegrationFlows
.from(Jms.messageDrivenChannelAdapter(this.connectionFactory)
.destination(queueName))
.channel(MessageChannels.queue())
.log(Level.DEBUG)
.log()
.toReactivePublisher();
}
public Flux<MilestoneNotification> findAll() {
return Flux.from(jmsReactiveSource())
.map(Message::getPayload);
}
/**
* Method which sends Milestone Notifications to the UM Queue.
*/
public void send(final MilestoneNotification message) {
jmsTemplate.convertAndSend(queueName, message);
}
}
Controller
#RestController
#Slf4j
#RequiredArgsConstructor
#RequestMapping(ApiConstants.MILESTONE_UM)
public class MilestoneUmController {
#Autowired
private UmGateway umGateway;
#RequestMapping(value = "/message", method = RequestMethod.POST)
public ResponseEntity<Boolean> sendMessage(
final #RequestBody MilestoneNotification notification) {
umGateway.send(notification);
return new ResponseEntity<>(HttpStatus.OK);
}
#GetMapping(path = "/milestone-notification/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<MilestoneNotification> feed() {
return this.umGateway.findAll();
}
}
Here are the logs :
- 2020.02.06 13:53:04.900 [jmsReactiveSource.org.springframework.jms.listener.DefaultMessageListenerContainer#0-1] INFO o.s.i.h.LoggingHandler - GenericMessage [payload={"messageId":"MAHESH_007","NotificationTag":"MAHESH_007","messageTimeStamp":"2020-01-21T10:56:33Z","processMilestoneId":"MAHESH_007","processMilestoneName":"MAHESH_007","accountNumber":"12345","previousStatus":"In Progress","currentStatus":"complete","isNew":true}, headers={JMS_IBM_Character_Set=UTF-8, JMS_IBM_MsgType=8, jms_destination=queue:///NOTIFICATIONQUEUE, _type=com.jpmc.wss.portal.domain.um.MilestoneNotification, JMSXUserID=cibcfdid , JMS_IBM_Encoding=273, priority=4, jms_timestamp=1580997184878, JMSXAppID=jpmc.wss.portal.Application , JMS_IBM_PutApplType=28, JMS_IBM_Format=MQSTR , jms_redelivered=false, JMS_IBM_PutDate=20200206, JMSXDeliveryCount=1, JMS_IBM_PutTime=13530511, id=5d277be2-49f5-3e5d-8916-5793db3b76e7, jms_messageId=ID:414d51204e41544d31383820202020201d9f3b5e03738521, timestamp=1580997184900}]
- 2020.02.06 13:53:04.968 [qtp2132762784-23] DEBUG c.j.w.p.u.RequestLoggingInterceptor - Returning status code 200 for POST request to /common/dataservice/di/milestone/um/message with query=[null] and http-user=[null]
- 2020.02.06 13:53:53.521 [qtp2132762784-18] INFO c.j.w.p.u.RequestLoggingInterceptor - Received GET request to /common/dataservice/di/milestone/um/milestone-notification/stream with query=[null] and http-user=[null]
- 2020.02.06 13:54:09.070 [qtp2132762784-16] INFO c.j.w.p.u.RequestLoggingInterceptor - Received POST request to /common/dataservice/di/milestone/um/message with query=[null] and http-user=[null]
- 2020.02.06 13:54:09.541 [jmsReactiveSource.org.springframework.jms.listener.DefaultMessageListenerContainer#0-1] INFO o.s.i.h.LoggingHandler - GenericMessage [payload={"messageId":"MAHESH_007","diNotificationTag":"MAHESH_007","messageTimeStamp":"2020-01-21T10:56:33Z","processMilestoneId":"MAHESH_007","processMilestoneName":"MAHESH_007","accountNumber":"12345","previousStatus":"In Progress","currentStatus":"complete","isNew":true}, headers={JMS_IBM_Character_Set=UTF-8, JMS_IBM_MsgType=8, jms_destination=queue:///NOTIFICATIONQUEUE, _type=com.jpmc.wss.portal.domain.um.MilestoneNotification, JMSXUserID=cibcfdid , JMS_IBM_Encoding=273, priority=4, jms_timestamp=1580997249519, JMSXAppID=jpmc.wss.portal.Application , JMS_IBM_PutApplType=28, JMS_IBM_Format=MQSTR , jms_redelivered=false, JMS_IBM_PutDate=20200206, JMSXDeliveryCount=1, JMS_IBM_PutTime=13540975, id=5421898e-5ef6-1f9b-aaa6-81ebc7668f50, jms_messageId=ID:414d51204e41544d31383820202020201d9f3b5e08738521, timestamp=1580997249541}]
- 2020.02.06 13:54:09.593 [qtp2132762784-16] DEBUG c.j.w.p.u.RequestLoggingInterceptor - Returning status code 200 for POST request to /common/dataservice/di/milestone/um/message with query=[null] and http-user=[null]
Flux stream on browser
So, you can't just make an URL request from the browser when you need a Server Side Events.
You need some particular JavaScript API to use from your web page: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events
I could make it to test with a curl -N tool: cURL - Structuring request to validate server sent events
Or using a WebTestClient form Spring WebFlux:
Flux<String> stream =
this.webTestClient.get().uri("/stream")
.exchange()
.returnResult(String.class)
.getResponseBody();
StepVerifier
.create(stream)
.expectNext("m1", "m2", "m3")
.thenCancel()
.verify();
I am using Kafka Spring Integration for publishing and consuming messages using kafka. I see Payload is properly passed from producer to consumer, but the header information is getting overridden somewhere.
#ServiceActivator(inputChannel = "fromKafka")
public void processMessage(Message<?> message) throws InterruptedException,
ExecutionException {
try {
System.out.println("Headers :" + message.getHeaders().toString());
}
} catch (Exception e) {
e.printStackTrace();
}
}
I get following headers:
Headers :{timestamp=1440013920609, id=f8c645f7-677b-ec32-dad0-a7b79082ef81}
I am constructing the message at producer end like this:
Message<FeelDBMessage> message = MessageBuilder
.withPayload(samplePayloadObj)
.setHeader(KafkaHeaders.MESSAGE_KEY, "key")
.setHeader(KafkaHeaders.TOPIC, "sampleTopic").build();
// publish the message
publisher.publishMessage(message);
and below is the header info at producer:
headers={timestamp=1440013914085, id=c4159c1c-2c67-634b-ef8d-3fb026b1172e, kafka_messageKey=key, kafka_topic=sampleTopic}
Any idea why the Headers are overridden by a different value?
Just because by default Framework uses the immutable GenericMessage.
Any manipulation to the existing message (e.g. MessageBuilder.withPayload) will produce a new GenericMessage instance.
From other side Kafka doesn't support any headers abstraction like JMS or AMQP. That's why KafkaProducerMessageHandler just do this when it publishes a message to Kafka:
this.kafkaProducerContext.send(topic, partitionId, messageKey, message.getPayload());
As you see it doesn't send headers at all. So, other side (consumer) just deals with only message from the topic as a payload and some system options as headers like topic, partition, messageKey.
In two words: we don't transfer headers over Kafka because it doesn't support them.