I am using Kafka Spring Integration for publishing and consuming messages using kafka. I see Payload is properly passed from producer to consumer, but the header information is getting overridden somewhere.
#ServiceActivator(inputChannel = "fromKafka")
public void processMessage(Message<?> message) throws InterruptedException,
ExecutionException {
try {
System.out.println("Headers :" + message.getHeaders().toString());
}
} catch (Exception e) {
e.printStackTrace();
}
}
I get following headers:
Headers :{timestamp=1440013920609, id=f8c645f7-677b-ec32-dad0-a7b79082ef81}
I am constructing the message at producer end like this:
Message<FeelDBMessage> message = MessageBuilder
.withPayload(samplePayloadObj)
.setHeader(KafkaHeaders.MESSAGE_KEY, "key")
.setHeader(KafkaHeaders.TOPIC, "sampleTopic").build();
// publish the message
publisher.publishMessage(message);
and below is the header info at producer:
headers={timestamp=1440013914085, id=c4159c1c-2c67-634b-ef8d-3fb026b1172e, kafka_messageKey=key, kafka_topic=sampleTopic}
Any idea why the Headers are overridden by a different value?
Just because by default Framework uses the immutable GenericMessage.
Any manipulation to the existing message (e.g. MessageBuilder.withPayload) will produce a new GenericMessage instance.
From other side Kafka doesn't support any headers abstraction like JMS or AMQP. That's why KafkaProducerMessageHandler just do this when it publishes a message to Kafka:
this.kafkaProducerContext.send(topic, partitionId, messageKey, message.getPayload());
As you see it doesn't send headers at all. So, other side (consumer) just deals with only message from the topic as a payload and some system options as headers like topic, partition, messageKey.
In two words: we don't transfer headers over Kafka because it doesn't support them.
Related
I am using org.springframework.boot:spring-boot-starter-amqp:2.6.6 .
According to the documentation, I set up #RabbitListener - I use SimpleRabbitListenerContainerFactory and the configuration looks like this:
#Bean
public SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory(ObjectMapper om) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factory.setConnectionFactory(connectionFactory());
factory.setAcknowledgeMode(AcknowledgeMode.MANUAL);
factory.setConcurrentConsumers(rabbitProperties.getUpdater().getConcurrentConsumers());
factory.setMaxConcurrentConsumers(rabbitProperties.getUpdater().getMaxConcurrentConsumers());
factory.setMessageConverter(new Jackson2JsonMessageConverter(om));
factory.setAutoStartup(rabbitProperties.getUpdater().getAutoStartup());
factory.setDefaultRequeueRejected(false);
return factory;
}
The logic of the service is to receive messages from rabbitmq, contact an external service via the rest API (using rest template) and put some information into the database based on the results of the response (using spring data jpa). The service implemented it successfully, but during testing it ran into problems that if any exceptions occur during the work of those thrown up the stack, the message is not sent to the configured dlq, but simply hangs in the broker as unacked. Can you please tell me how you can tell spring amqp that if any error occurs, you need to redirect the message to dlq?
The listener itself looks something like this:
#RabbitListener(
queues = {"${rabbit.updater.consuming.queue.name}"},
containerFactory = "rabbitListenerContainerFactory"
)
#Override
public void listen(
#Valid #Payload MessageDTO message,
Channel channel,
#Header(AmqpHeaders.DELIVERY_TAG) Long deliveryTag
) {
log.debug(DebugMessagesConstants.RECEIVED_MESSAGE_FROM_QUEUE, message, deliveryTag);
messageUpdater.process(message);
channel.basicAck(deliveryTag, false);
log.debug(DebugMessagesConstants.PROCESSED_MESSAGE_FROM_QUEUE, message, deliveryTag);
}
In rabbit managment it look something like this:
enter image description here
and unacked will hang until the queue consuming application stops
See error handling documentation: https://docs.spring.io/spring-amqp/docs/current/reference/html/#annotation-error-handling.
So, you just don't do an AcknowledgeMode.MANUAL and rely on the Dead Letter Exchange configuration for those messages which are rejected in case of error.
Or try to use a this.channel.basicNack(deliveryTag, false, false) in case of messageUpdater.process(message); exception...
I am using the spring-cloud-starter-aws-messaging in my Spring Boot application and #sqslistener to consume SQS messages. my consumer weirdly stops getting messages out of nowhere and the ApproximateNumberOfMessagesVisible gradually increases triggering a CloudWatch alarm. I could see no error logs that are generated before it stops getting any more messages. Am I missing something?
#SqsListener(value = "${sqs.queue.url.indexSavedSetQueue}",
deletionPolicy = SqsMessageDeletionPolicy.ON_SUCCESS)
public void listenIndexSavedSetEvent(#NonNull String message) throws IOException {
log.info("Index saved set event received, message: {}", message);
IndexSavedSetPayloadDto indexSavedSetPayloadDto = objectMapper
.readValue(message, IndexSavedSetPayloadDto.class);
String setName = indexSavedSetPayloadDto.getSetName();
indexerService.indexSet(setName);
}
I want to create a simple IntegrationFlow with Spring integration, and I am having difficulties.
I want to create an integration flow that takes messages from a queue in Rabbit Mq and posts the messages to an endpoint Rest. I want to ack manually depending on the results of the post that I will make.
A typical behavior of the integration Flow would be like this:
I receive a message in the queue.
Spring detects it, takes the message and posts it in the Rest endpoint.
The end point responds with a 200 code.
Spring integration ack the message.
If the endpoint responds with an error code I want to be able to nack or retry.
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.APPLICATION_JSON);
RestTemplate restTemplate = new RestTemplate();
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer(connectionFactory);
container.setAcknowledgeMode(AcknowledgeMode.MANUAL);
container.setQueueNames(BOUTIQUE_QUEUE_NAME);
/* Get Message from RabbitMQ */
return IntegrationFlows.from(Amqp.inboundAdapter(container))
.handle(msg ->
{
String msgString = new String((byte[]) msg.getPayload(), StandardCharsets.UTF_8);
HttpEntity<String> requestBody = new HttpEntity<String>(msgString, headers);
restTemplate.postForObject(ENDPOINT_LOCAL_URL, requestBody, String.class);
System.out.println(msgString);
})
.get();
You don't need to use manual acknowledge mode for this use case; if he rest call returns normally, the container will ack the message; if an exception is thrown, the container will nack the message and it will be redelivered.
If you use manual acks, the Channel and deliveryTag are available in the AmqpHeaders.CHANNEL and AmqpHeaders.DELIVERY_TAG message headers and you can call basicAck or basicReject on the channel (you will have to add an error channel to the inbound adapter to handle errors.
I use Spring Cloud Stream 1.3.2.RELEASE to publish a String message to Kafka. When I consume the message using command line Kafka consumer or Spring Kafka #KafkaListener, a contentType header is always appended to the message body.
Question:
Is there any way to get rid of the embedded headers?
--
Spring Cloud Stream as producer
private void send() {
channel.test().send(MessageBuilder.withPayload("{\"foo\":\"bar\"}").build());
}
Command line Kafka consumer
$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test
�
contentType
"text/plain"{"foo":"bar"}
Spring Kafka as consumer
#KafkaListener(topics = "test")
public void receive(Message message){
log.info("Message payload received: {}", message.getPayload());
}
2018-05-16 07:12:05.241 INFO 19475 --- [ntainer#0-0-C-1] com.demo.service.Listener : Message payload received: �contentType"text/plain"{"foo":"bar"}
#KafkaListener(topics = "test")
public void receive(#Payload String message){
log.info("Message payload received: {}", message);
}
2018-05-16 07:16:14.313 INFO 19747 --- [ntainer#0-0-C-1] com.demo.service.Listener : Message payload received: �contentType"text/plain"{"foo":"bar"}
See headerMode binding property: https://docs.spring.io/spring-cloud-stream/docs/Ditmars.SR3/reference/htmlsingle/#_properties_for_use_of_spring_cloud_stream. You need to set it to raw for the destination you send messages.
I want to send a message to websocket subscribers of a specific record - when an action takes place in one of my service class.
I'm trying to read the Spring Websocket documentation but it's kind of ambiguous to the point of how to get all these things working together.
Here are my setup files (this is extending jHipster btw):
WebsocketConfiguration.java
#Override
public void configureMessageBroker(MessageBrokerRegistry config) {
config.enableStompBrokerRelay("/queue/", "/topic/", "/exchange/");
config.setApplicationDestinationPrefixes("/app");
config.setPathMatcher(new AntPathMatcher("."));
}
#Override
public void registerStompEndpoints(StompEndpointRegistry registry) {
registry.addEndpoint("/ws").withSockJS();
}
WebsocketSecurity.java
#Override
protected void configureInbound(MessageSecurityMetadataSourceRegistry messages) {
messages
// message types other than MESSAGE and SUBSCRIBE
.nullDestMatcher().authenticated()
// matches any destination that starts with /rooms/
.simpDestMatchers("/topic/tracker").hasAuthority(AuthoritiesConstants.ADMIN)
.simpDestMatchers("/topic/**").authenticated()
// (i.e. cannot send messages directly to /topic/, /queue/)
// (i.e. cannot subscribe to /topic/messages/* to get messages sent to
// /topic/messages-user<id>)
.simpTypeMatchers(SimpMessageType.MESSAGE, SimpMessageType.SUBSCRIBE).denyAll()
// catch all
.anyMessage().denyAll();
}
Controller class (attempt at implementing a simple broker I can test subscribing to from sockjs and recieving messages generated elsewhere in the application:
#MessageMapping("/ws")
#SendTo("/topic/sendactivity.{id}")
public void activity(#DestinationVariable string id, #Payload String message){
log.debug("Sending command center: "+message);
}
#RequestMapping(value = "/updateactivity", method = RequestMethod.PUT)
public ResponseEntity<Membership> updateMembership(
#RequestBody Membership membership) throws URISyntaxException {
// ...
String testString = "test";
messagingTemplate.convertAndSend("/topic/commandcenter"+membership.getId().toString(), testString);
// ...
}
When I put a breakpoint on the public void activity method, I don't get anything?
Sending a message to "/topic/commandcenterID" using the messaging template will send that message to the message broker, which will dispatch that message to clients subscribed to that topic. So it won't flow through your activity method.
When using #MessageMapping annotated methods, you're declaring those as application destinations. So sending a message to "/app/ws" should map to that method. Note that in that case I doubt it'll work since the destination variable you're expecting as a method argument is missing from the path definition in the #MessageMapping annotation.
Also, the #SendTo annotation in fact tells Spring that the value returned by the method should be converted to a message and sent to the given destination.
It seems you're mixing things up here, and I think you should:
read carefully the flow of messages in Spring STOMP support
look at a few example apps like the websocket portfolio and websocket chat