Spring cloud stream: Headers kafka_acknowledgment value is null - spring-boot

Question

I am trying to have the control over the offset commit on kafka topic according to the status of processing of message in the application. If message are successful, so offset can be committed. For this I am trying to get the Headers in my method if so that message can be acknowledge manually
spring:
cloud:
stream:
default:
contentType: application/json
default-binder: binder1-kafka
bindings:
myChannel:
binder: binder1-kafka
destination: my_topic
content-type: text/plain
consumer:
autoCommitOffset: false
outChannel:
binder: binder2-kafka
destination: my_topic
content-type: text/plain
consumer:
autoCommitOffset: false
binders:
#Connection config to different clusters
binder1-kafka:
type: kafka
defaultCandidate: true
environment:
spring:
cloud:
stream:
kafka:
binder:
brokers: some-url1:9092
binder2-kafka:
type: kafka
defaultCandidate: false
environment:
spring:
cloud:
stream:
kafka:
binder:
brokers: some-url2:9092
But When using the listener as
#StreamListener(target = IBrokerChannel.myChannel )
public void handlePayload(#Payload MyPayload payload, #Headers Map<String, Object> headers) {
Acknowledgment acknowledgment= (Acknowledgment) headers.get("kafka_acknowledgment"); // acknowledgment object is always null.
acknowledgment.acknowledge();
}
acknowledgment is always null. I am using kafka producer cli to send the message to the topic. spring-boot version is 1.5.10.RELEASE

Answer 1

You are missing kafka branch in your configuration property definition. It must be like this:
spring:
cloud:
stream:
default-binder: kafka
kafka:
bindings:
myChannel:
consumer:
autoCommitOffset: false
https://docs.spring.io/spring-cloud-stream/docs/Fishtown.BUILD-SNAPSHOT/reference/htmlsingle/#kafka-consumer-properties

Answer 2

Works fine for me...
#SpringBootApplication
#EnableBinding(Sink.class)
public class So51159949Application {
public static void main(String[] args) {
SpringApplication.run(So51159949Application.class, args);
}
#Bean
public ApplicationRunner runner(KafkaTemplate<byte[], byte[]> template) {
return args -> {
template.send("so51159949", "foo".getBytes());
};
}
#StreamListener(Sink.INPUT)
public void in(String in, #Headers MessageHeaders headers) {
System.out.println(in);
System.out.println(headers);
Acknowledgment ack = headers.get(KafkaHeaders.ACKNOWLEDGMENT, Acknowledgment.class);
ack.acknowledge();
}
}
and
spring:
cloud:
stream:
bindings:
input:
group: so51159949
destination: so51159949
kafka:
bindings:
input:
consumer:
auto-commit-offset: false
and
foo
{kafka_offset=2, kafka_consumer=org.apache.kafka.clients.consumer.KafkaConsumer#4ad39b5f, deliveryAttempt=1, kafka_timestampType=CREATE_TIME, kafka_receivedMessageKey=null, kafka_receivedPartitionId=0, kafka_receivedTopic=so51159949, kafka_receivedTimestamp=1530643662028, kafka_acknowledgment=Acknowledgment for ConsumerRecord(topic = so51159949, partition = 0, offset = 2, CreateTime = 1530643662028, serialized key size = -1, serialized value size = 3, headers = RecordHeaders(headers = [], isReadOnly = false), key = null, value = [B#700c9aa9), contentType=application/json}