Can MockProducer and MockConsumer be used with kafkaTemplate producer - spring-boot

I have a service which using kafkaTemplate to send the kafka message.
#Service
class KafkaProducer(#Autowired val kafkaTemplate: KafkaTemplate<String, String>) {
final fun sendMessage(msg: String) {
val abc = kafkaTemplate.send(AppConstants.TOPIC_NAME, "abc", msg)
abc.whenComplete {
result, ex ->
if (ex != null) {
print("ex occured")
print(ex.message)
} else {
print("sent successfully")
print(result.producerRecord.value())
}
}
}
}
For unit test, I am trying to mock this KafkaProducer.
Can i use MockProducer (import org.apache.kafka.clients.producer.MockProducer) for this?
#Test
fun verify_test() {
val mockProducer = MockProducer(true,StringSerializer(), StringSerializer())
val kafkaProducer = KafkaProducer(mockProducer)
}
On trying the above code, i am getting error bcoz KafkaProducer takes argument as KafkaTemplate object, and I am providing MockProducer.

Here is an example using a mock ProducerFactory...
#SpringJUnitConfig
class So74993413ApplicationTests {
#Test
void test(#Autowired KafkaTemplate<String, String> template,
#Autowired MockProducer<String, String> producer) throws Exception {
CompletableFuture<SendResult<String, String>> future = template.send("foo", "bar");
SendResult<String, String> sendResult = future.get();
// System.out.println(sendResult);
List<ProducerRecord<String, String>> history = producer.history();
assertThat(history).hasSize(1);
ProducerRecord<String, String> record = history.get(0);
assertThat(record.topic()).isEqualTo("foo");
assertThat(record.value()).isEqualTo("bar");
}
#Configuration
public static class Config {
#Bean
MockProducer<String, String> producer() {
return new MockProducer<>(true, new StringSerializer(), new StringSerializer());
}
#Bean
ProducerFactory<String, String> pf (MockProducer producer) {
return new ProducerFactory<>() {
#Override
public Producer<String, String> createProducer() {
return producer;
}
};
}
#Bean
KafkaTemplate<String, String> template(ProducerFactory<String, String> pf) {
return new KafkaTemplate<>(pf);
}
}
}

KafkaTemplate is not a type of Producer. Native Kafka classes cannot be used in place of Spring's - that'd be a reverse dependency.
You would have to mock the template instead since that's what you're actually using. By doing so, it'll bypass any KafkaProducer instance the template used, mock or not.

Related

Test onFailure of spring-kafka sending message

I try to test the onFailure case when I send a kafka message with producer but the onFailure method is never fire.
Here is my code where I send a message :
#Component
public class MessageSending {
#Autowired
Map<String, KafkaTemplate<String, String>> producerByCountry;
String topicName = "countryTopic";
public void sendMessage(String data) {
producerByCountry.get("countryName").send(topicName, data).addCallback(
onSuccess -> {},
onFailure -> log.error("failed")
);
}
}
Here is the test class but it's still a success case and I have no idea how I can test the failure case (I want to add some processing inside the onFailure block but I would like to first know how I can trigger onFailure by testing).
#EmbeddedKafka
#SpringBootTest
public class MessageSendingTest {
#MockBean
Map<Country, KafkaTemplate<String, String>> producerByCountry;
#Autowired
EmbeddedKafkaBroker embeddedKafka;
#Autowired
MessageSending messageSending;
#Test
void failTest(CapturedOutput capturedOutput) {
var props = KafkaTestUtils.producerProps(embeddedKafka);
var producerTemplate = new DefaultKafkaProducerFactory<String, String>(props);
var template = new KafkaTemplate<>(producerTemplate);
given(producerByCountry.get("USA"))).willReturn(template);
messageSending.sendMessage("data");
assertThat(capturedOutput).contains("failed");
}
}
I also tried the idea of this topic How to test Kafka OnFailure callback with Junit? by doing
doAnswer(invocationOnMock -> {
ListenableFutureCallback<SendResult<String, String>> listenableFutureCallback = invocationOnMock.getArgument(0);
KafkaProducerException value = new KafkaProducerException(new ProducerRecord<String, String>("myTopic", "myMessage"), "error", ex);
listenableFutureCallback.onFailure(value);
return null;
}).when(mock(ListenableFuture.class)).addCallback(any(ListenableFutureCallback.class));
But I got this mockito exception org.mockito.exceptions.misusing.UnnecessaryStubbingException due by when().addCallback
Can someone can help ?
Thanks.
You can use a mock template; see this answer for an example:
How to mock result from KafkaTemplate
EDIT
You can also mock the underlying Producer object - here is an example that is closer to your use case...
#SpringBootApplication
public class So75074961Application {
public static void main(String[] args) {
SpringApplication.run(So75074961Application.class, args);
}
#Bean
KafkaTemplate<String, String> france(ProducerFactory<String, String> pf) {
return new KafkaTemplate<>(pf, Map.of(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "france:9092"));
}
#Bean
KafkaTemplate<String, String> germany(ProducerFactory<String, String> pf) {
return new KafkaTemplate<>(pf, Map.of(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "germany:9092"));
}
}
#Component
class MessageSending {
private static final Logger log = LoggerFactory.getLogger(MessageSending.class);
#Autowired
Map<String, KafkaTemplate<String, String>> producerByCountry;
String topicName = "countryTopic";
public void sendMessage(String country, String data) {
producerByCountry.get(country).send(topicName, data).addCallback(
onSuccess -> log.info(onSuccess.getRecordMetadata().toString()),
onFailure -> log.error("failed: " + onFailure.getMessage()));
}
}
#SpringBootTest
#ExtendWith(OutputCaptureExtension.class)
class So75074961ApplicationTests {
#Test
void test(#Autowired MessageSending sending, CapturedOutput capture) {
ProducerFactory<String, String> pf = mock(ProducerFactory.class);
Producer<String, String> prod = mock(Producer.class);
given(pf.createProducer()).willReturn(prod);
willAnswer(inv -> {
Callback callback = inv.getArgument(1);
callback.onCompletion(null, new RuntimeException("test"));
return mock(Future.class);
}).given(prod).send(any(), any());
// inject the mock pf into "france" template
Map<?, ?> producers = KafkaTestUtils.getPropertyValue(sending, "producerByCountry", Map.class);
new DirectFieldAccessor(producers.get("france")).setPropertyValue("producerFactory", pf);
sending.sendMessage("france", "foo");
assertThat(capture)
.contains("failed: Failed to send; nested exception is java.lang.RuntimeException: test");
}
}
Use CompletableFuture instead of ListenableFuture for versions 3.0 or later.
public void sendMessage(String country, String data) {
producerByCountry.get(country).send(topicName, data).whenComplete(
(res, ex) -> {
if (ex == null) {
log.info(res.getRecordMetadata().toString());
}
else {
log.error("failed: " + ex.getMessage());
}
});
}
and
assertThat(capture)
.contains("failed: Failed to send");
(the latter because Spring Framework 6.0+ no longer merges nested exception messages; the top level exception is a KafkaProducerException, with the actual exception as its cause).

Spring Boot Kafka Configure DefaultErrorHandler?

I created a batch-consumer following the Spring Kafka docs:
#SpringBootApplication
public class ApplicationConsumer {
private static final Logger LOGGER = LoggerFactory.getLogger(ApplicationConsumer.class);
private static final String TOPIC = "foo";
public static void main(String[] args) {
ConfigurableApplicationContext context = SpringApplication.run(ApplicationConsumer.class, args);
}
#Bean
public RecordMessageConverter converter() {
return new JsonMessageConverter();
}
#Bean
public BatchMessagingMessageConverter batchConverter() {
return new BatchMessagingMessageConverter(converter());
}
#KafkaListener(topics = TOPIC)
public void listen(List<Name> ps) {
LOGGER.info("received name beans: {}", Arrays.toString(ps.toArray()));
}
}
I was able to successfully get the consumer running by defining the following additional configuration env variables, that Spring automatically picks up:
export SPRING_KAFKA_BOOTSTRAP-SERVERS=...
export SPRING_KAFKA_CONSUMER_GROUP-ID=...
So the above code works. But now I want to customize the default error handler to use exponential backoff. From the ref docs I tried adding the following to ApplicationConsumer class:
#Bean
public ConcurrentKafkaListenerContainerFactory<?, ?> kafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<String, Object> factory = new ConcurrentKafkaListenerContainerFactory<>();
factory.setCommonErrorHandler(new DefaultErrorHandler(new ExponentialBackOffWithMaxRetries(10)));
factory.setConsumerFactory(consumerFactory());
return factory;
}
#Bean
public ConsumerFactory<String, Object> consumerFactory() {
return new DefaultKafkaConsumerFactory<>(consumerConfigs());
}
#Bean
public Map<String, Object> consumerConfigs() {
Map<String, Object> props = new HashMap<>();
return props;
}
But now I get errors saying that it can't find some of the configuration. It looks like I'm stuck having to redefine all of the properties in consumerConfigs() that were already being automatically defined before. This includes everything from bootstrap server uris to the json-deserialization config.
Is there a good way to update my first version of the code to just override the default-error handler?
Just define the error handler as a #Bean and Boot will automatically wire it into its auto configured container factory.
EDIT
This works as expected for me:
#SpringBootApplication
public class So70884203Application {
public static void main(String[] args) {
SpringApplication.run(So70884203Application.class, args);
}
#Bean
DefaultErrorHandler eh() {
return new DefaultErrorHandler((rec, ex) -> {
System.out.println("Recovered: " + rec);
}, new FixedBackOff(0L, 0L));
}
#KafkaListener(id = "so70884203", topics = "so70884203")
void listen(String in) {
System.out.println(in);
throw new RuntimeException("test");
}
#Bean
NewTopic topic() {
return TopicBuilder.name("so70884203").partitions(1).replicas(1).build();
}
}
foo
Recovered: ConsumerRecord(topic = so70884203, partition = 0, leaderEpoch = 0, offset = 0, CreateTime = 1643316625291, serialized key size = -1, serialized value size = 3, headers = RecordHeaders(headers = [], isReadOnly = false), key = null, value = foo)

Can record processor be spring singleton bean?

I am using spring-kafka to implement the topology to convert lower-case to upper-case like this:
#Bean
public KStream<String, String> kStreamPromoToUppercase(StreamsBuilder builder) {
KStream<String, String> sourceStream = builder.stream(inputTopic, Consumed.with(Serdes.String(), Serdes.String()));
// A new processor object is created here per record
sourceStream.process(() -> new CapitalCaseProcessor());
...
}
The processor is not a spring singleton bean and is declared as follows:
public class CapitalCaseProcessor implements Processor<String, String> {
private ProcessorContext context;
#Override
public void init(ProcessorContext context) {
this.context = context;
}
#Override
public void process(String key, String value) {
context.headers().forEach(System.out::println);
}
The above processor is a stateful and holds the state of processor context.
Now, what would happen if we convert the stateful CapitalCaseProcessor to a spring singleton bean ?
#Component
public class CapitalCaseProcessor implements Processor<String, String> {
//Is the ProcessorContext going to have thread safety issue now?
private ProcessorContext context;
#Override
public void init(ProcessorContext context) {
this.context = context;
}
#Override
public void process(String key, String value) {
context.headers().forEach(System.out::println);
}
and try to inject it in the main topology as spring bean:
#Configuration
public class UppercaseTopologyProcessor {
#Autowired CapitalCaseProcessor capitalCaseProcessor;
#Bean
public KStream<String, String> kStreamPromoToUppercase(StreamsBuilder builder) {
KStream<String, String> sourceStream = builder.stream(inputTopic, Consumed.with(Serdes.String(), Serdes.String()));
// A singleton spring bean processor is now used for all the records
sourceStream.process(() -> capitalCaseProcessor);
...
}
Is it going to cause thread safety issue with the CapitalCaseProcessor now as it contains processorContext as a state?
Or is it better to declare it as a prototype bean like this as this?
#Configuration
public class UppercaseTopologyProcessor {
#Lookup
public CapitalCaseProcessor getCapitalCaseProcessor() {return null;}
#Bean
public KStream<String, String> kStreamPromoToUppercase(StreamsBuilder builder) {
KStream<String, String> sourceStream = builder.stream(inputTopic, Consumed.with(Serdes.String(), Serdes.String()));
// A singleton spring bean processor is now used for all the records
sourceStream.process(() -> getCapitalCaseProcessor());
...
}
Update: I essentially would like to know two things:
Should the processor instance be associated per stream record like AKKA actor model where actors are stateful and works per request or it can be a singleton object?
Is ProcessorContext thread safe?
I just ran a test and, the processor context is NOT thread-safe, what makes the stream thread-safe is you use a ProcessorSupplier (in your first example) to create a new processor instance for each record.
You must certainly not replace this with a Spring singleton.
Here is my test, using the MessagingTransformer provided by Spring for Apache Kafka:
#SpringBootApplication
#EnableKafkaStreams
public class So66200448Application {
private static final Logger log = LoggerFactory.getLogger(So66200448Application.class);
public static void main(String[] args) {
SpringApplication.run(So66200448Application.class, args);
}
#Bean
KStream<String, String> stream(StreamsBuilder sb) {
KStream<String, String> stream = sb.stream("so66200448");
stream.transform(() -> new MessagingTransformer(msg -> {
log.info(msg.toString());
log.info(new String(msg.getHeaders().get("foo", byte[].class)));
return msg;
}, new MessagingMessageConverter()) {
#Override
public KeyValue transform(Object key, Object value) {
try {
Thread.sleep(5000);
}
catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
return super.transform(key, value);
}
})
.to("so66200448out");
return stream;
}
#Bean
public NewTopic topic1() {
return TopicBuilder.name("so66200448").partitions(2).replicas(1).build();
}
#Bean
public NewTopic topic2() {
return TopicBuilder.name("so66200448out").partitions(2).replicas(1).build();
}
#Bean
public ApplicationRunner runner(KafkaTemplate<String, String> template) {
return args -> {
Headers headers = new RecordHeaders();
headers.add(new RecordHeader("foo", "bar".getBytes()));
ProducerRecord<String, String> record = new ProducerRecord<>("so66200448", 0, null, "foo", headers);
template.send(record);
headers.remove("foo");
headers.add(new RecordHeader("foo", "baz".getBytes()));
record = new ProducerRecord<>("so66200448", 1, null, "bar", headers);
template.send(record);
};
}
#KafkaListener(id = "so66200448out", topics = "so66200448out")
public void listen(String in) {
System.out.println(in);
}
}
spring.kafka.streams.application-id=so66200448
spring.kafka.streams.properties.num.stream.threads=2
spring.kafka.consumer.auto-offset-reset=earliest
2021-02-16 15:57:34.322 INFO 17133 --- [-StreamThread-1] com.example.demo.So66200448Application : bar
2021-02-16 15:57:34.322 INFO 17133 --- [-StreamThread-2] com.example.demo.So66200448Application : baz
Changing the supplier to return the same instance each time, definitely breaks it.
#Bean
KStream<String, String> stream(StreamsBuilder sb) {
KStream<String, String> stream = sb.stream("so66200448");
MessagingTransformer transformer = new MessagingTransformer(msg -> {
log.info(msg.toString());
log.info(new String(msg.getHeaders().get("foo", byte[].class)));
return msg;
}, new MessagingMessageConverter()) {
#Override
public KeyValue transform(Object key, Object value) {
try {
Thread.sleep(5000);
}
catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
return super.transform(key, value);
}
};
stream.transform(() -> transformer)
.to("so66200448out");
return stream;
}
2021-02-16 15:54:28.975 INFO 16406 --- [-StreamThread-1] com.example.demo.So66200448Application : baz
2021-02-16 15:54:28.975 INFO 16406 --- [-StreamThread-2] com.example.demo.So66200448Application : baz
So, streams relies on getting a new instance each time for thread-safety.

Spring boot JMS using different messages class

I'm using spring boot.
I want to use different models at both sender and receiver so that they don't depend to the same model (receiver doesn't need to add model of sender to classpath).
1) Should I do that?
2) And how can I do that?
Sender:
AccountEvent accountEvent = new com.dspc.account.domain.dto.AccountEvent(createdAccount.getId(), EventType.CREATED);
jmsMessagingTemplate.convertAndSend(new ActiveMQTopic("VirtualTopic.ACCOUNT-EVENT-TOPIC"), accountEvent);
Receiver:
#JmsListener(destination = "Consumer.AgentGenerator.VirtualTopic.ACCOUNT-EVENT-TOPIC")
public void receive(com.dspc.devicemgmt.domain.dto.AccountEvent accountEvent) {
System.out.println(accountEvent);
}
JMS config of both sender and receiver:
#Bean // Serialize message content to json using TextMessage
public MessageConverter jacksonJmsMessageConverter() {
MappingJackson2MessageConverter converter = new MappingJackson2MessageConverter();
converter.setTargetType(MessageType.TEXT);
converter.setTypeIdPropertyName("_type");
return converter;
}
Get exception when receiving message:
[com.dspc.account.domain.dto.AccountEvent]; nested exception is
java.lang.ClassNotFoundException:
com.dspc.account.domain.dto.AccountEvent
Note that there are 2 different packages:
- com.dspc.account.domain.dto.AccountEvent
- com.dspc.devicemgmt.domain.dto.AccountEvent
I'm thinking about creating a common message model. How do you think?
public class DspcCommonMessage {
private Map<String, String> properties;
private Optional<byte[]> payLoad = Optional.empty();
public Map<String, String> getProperties() {
return properties;
}
public void setProperties(Map<String, String> properties) {
this.properties = properties;
}
public Optional<byte[]> getPayLoad() {
return payLoad;
}
public void setPayLoad(Optional<byte[]> payLoad) {
this.payLoad = payLoad;
}
}
Sender and receiver:
public void publishMessage(com.dspc.account.domain.dto.AccountEvent accountEvent) {
ObjectMapper objectMapper = new ObjectMapper();
String messageAsString = objectMapper.writeValueAsString(accountEvent);
DspcCommonMessage dspcMessage = new DspcCommonMessage();
dspcMessage.setPayLoad(Optional.of(messageAsString.getBytes()));
jmsMessagingTemplate.convertAndSend(new ActiveMQTopic("VirtualTopic.ACCOUNT-EVENT-TOPIC"), dspcMessage);
}
#JmsListener(destination = "Consumer.AgentGenerator.VirtualTopic.ACCOUNT-EVENT-TOPIC")
public void receive(com.dspc.common.domain.DspcCommonMessage dspcCommonMessage) {
String jsonBody = new String(dspcCommonMessage.getPayload());
ObjectMapper objectMapper = new ObjectMapper();
com.dspc.devicemgmt.domain.dto.AccountEvent accountEvent = objectMapper.readValue(jsonBody,
com.dspc.devicemgmt.domain.dto.AccountEvent accountEvent.class);
System.out.println(accountEvent);
}

Simple embedded Kafka test example with spring boot

Edit FYI: working gitHub example
I was searching the internet and couldn't find a working and simple example of an embedded Kafka test.
My setup is:
Spring boot
Multiple #KafkaListener with different topics in one class
Embedded Kafka for test which is starting fine
Test with Kafkatemplate which is sending to topic but the
#KafkaListener methods are not receiving anything even after a huge sleep time
No warnings or errors are shown, only info spam from Kafka in logs
Please help me. There are mostly over configured or overengineered examples. I am sure it can be done simple.
Thanks, guys!
#Controller
public class KafkaController {
private static final Logger LOG = getLogger(KafkaController.class);
#KafkaListener(topics = "test.kafka.topic")
public void receiveDunningHead(final String payload) {
LOG.debug("Receiving event with payload [{}]", payload);
//I will do database stuff here which i could check in db for testing
}
}
private static String SENDER_TOPIC = "test.kafka.topic";
#ClassRule
public static KafkaEmbedded embeddedKafka = new KafkaEmbedded(1, true, SENDER_TOPIC);
#Test
public void testSend() throws InterruptedException, ExecutionException {
Map<String, Object> senderProps = KafkaTestUtils.producerProps(embeddedKafka);
KafkaProducer<Integer, String> producer = new KafkaProducer<>(senderProps);
producer.send(new ProducerRecord<>(SENDER_TOPIC, 0, 0, "message00")).get();
producer.send(new ProducerRecord<>(SENDER_TOPIC, 0, 1, "message01")).get();
producer.send(new ProducerRecord<>(SENDER_TOPIC, 1, 0, "message10")).get();
Thread.sleep(10000);
}
Embedded Kafka tests work for me with below configs,
Annotation on test class
#EnableKafka
#SpringBootTest(classes = {KafkaController.class}) // Specify #KafkaListener class if its not the same class, or not loaded with test config
#EmbeddedKafka(
partitions = 1,
controlledShutdown = false,
brokerProperties = {
"listeners=PLAINTEXT://localhost:3333",
"port=3333"
})
public class KafkaConsumerTest {
#Autowired
KafkaEmbedded kafkaEmbeded;
#Autowired
KafkaListenerEndpointRegistry kafkaListenerEndpointRegistry;
Before annotation for setup method
#Before
public void setUp() throws Exception {
for (MessageListenerContainer messageListenerContainer : kafkaListenerEndpointRegistry.getListenerContainers()) {
ContainerTestUtils.waitForAssignment(messageListenerContainer,
kafkaEmbeded.getPartitionsPerTopic());
}
}
Note: I am not using #ClassRule for creating embedded Kafka rather auto-wiring #Autowired embeddedKafka
#Test
public void testReceive() throws Exception {
kafkaTemplate.send(topic, data);
}
Hope this helps!
Edit: Test configuration class marked with #TestConfiguration
#TestConfiguration
public class TestConfig {
#Bean
public ProducerFactory<String, String> producerFactory() {
return new DefaultKafkaProducerFactory<>(KafkaTestUtils.producerProps(kafkaEmbedded));
}
#Bean
public KafkaTemplate<String, String> kafkaTemplate() {
KafkaTemplate<String, String> kafkaTemplate = new KafkaTemplate<>(producerFactory());
kafkaTemplate.setDefaultTopic(topic);
return kafkaTemplate;
}
Now #Test method will autowire KafkaTemplate and use is to send message
kafkaTemplate.send(topic, data);
Updated answer code block with above line
since the accepted answer doesn't compile or work for me. I find another solution based on https://blog.mimacom.com/testing-apache-kafka-with-spring-boot/ what I would like to share with you.
The dependency is 'spring-kafka-test' version: '2.2.7.RELEASE'
#RunWith(SpringRunner.class)
#EmbeddedKafka(partitions = 1, topics = { "testTopic" })
#SpringBootTest
public class SimpleKafkaTest {
private static final String TEST_TOPIC = "testTopic";
#Autowired
EmbeddedKafkaBroker embeddedKafkaBroker;
#Test
public void testReceivingKafkaEvents() {
Consumer<Integer, String> consumer = configureConsumer();
Producer<Integer, String> producer = configureProducer();
producer.send(new ProducerRecord<>(TEST_TOPIC, 123, "my-test-value"));
ConsumerRecord<Integer, String> singleRecord = KafkaTestUtils.getSingleRecord(consumer, TEST_TOPIC);
assertThat(singleRecord).isNotNull();
assertThat(singleRecord.key()).isEqualTo(123);
assertThat(singleRecord.value()).isEqualTo("my-test-value");
consumer.close();
producer.close();
}
private Consumer<Integer, String> configureConsumer() {
Map<String, Object> consumerProps = KafkaTestUtils.consumerProps("testGroup", "true", embeddedKafkaBroker);
consumerProps.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
Consumer<Integer, String> consumer = new DefaultKafkaConsumerFactory<Integer, String>(consumerProps)
.createConsumer();
consumer.subscribe(Collections.singleton(TEST_TOPIC));
return consumer;
}
private Producer<Integer, String> configureProducer() {
Map<String, Object> producerProps = new HashMap<>(KafkaTestUtils.producerProps(embeddedKafkaBroker));
return new DefaultKafkaProducerFactory<Integer, String>(producerProps).createProducer();
}
}
I solved the issue now
#BeforeClass
public static void setUpBeforeClass() {
System.setProperty("spring.kafka.bootstrap-servers", embeddedKafka.getBrokersAsString());
System.setProperty("spring.cloud.stream.kafka.binder.zkNodes", embeddedKafka.getZookeeperConnectionString());
}
while I was debugging, I saw that the embedded kaka server is taking a random port.
I couldn't find the configuration for it, so I am setting the kafka config same as the server. Looks still a bit ugly for me.
I would love to have just the #Mayur mentioned line
#EmbeddedKafka(partitions = 1, controlledShutdown = false, brokerProperties = {"listeners=PLAINTEXT://localhost:9092", "port=9092"})
but can't find the right dependency in the internet.
In integration testing, having fixed ports like 9092 is not recommended because multiple tests should have the flexibility to open their own ports from embedded instances. So, following implementation is something like that,
NB: this implementation is based on junit5(Jupiter:5.7.0) and spring-boot 2.3.4.RELEASE
TestClass:
#EnableKafka
#SpringBootTest(classes = {ConsumerTest.Config.class, Consumer.class})
#EmbeddedKafka(
partitions = 1,
controlledShutdown = false)
#TestInstance(TestInstance.Lifecycle.PER_CLASS)
public class ConsumerTest {
#Autowired
private EmbeddedKafkaBroker kafkaEmbedded;
#Autowired
private KafkaListenerEndpointRegistry kafkaListenerEndpointRegistry;
#BeforeAll
public void setUp() throws Exception {
for (final MessageListenerContainer messageListenerContainer : kafkaListenerEndpointRegistry.getListenerContainers()) {
ContainerTestUtils.waitForAssignment(messageListenerContainer,
kafkaEmbedded.getPartitionsPerTopic());
}
}
#Value("${topic.name}")
private String topicName;
#Autowired
private KafkaTemplate<String, Optional<Map<String, List<ImmutablePair<String, String>>>>> requestKafkaTemplate;
#Test
public void consume_success() {
requestKafkaTemplate.send(topicName, load);
}
#Configuration
#Import({
KafkaListenerConfig.class,
TopicConfig.class
})
public static class Config {
#Value(value = "${spring.kafka.bootstrap-servers}")
private String bootstrapAddress;
#Bean
public ProducerFactory<String, Optional<Map<String, List<ImmutablePair<String, String>>>>> requestProducerFactory() {
final Map<String, Object> configProps = new HashMap<>();
configProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapAddress);
configProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
configProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class);
return new DefaultKafkaProducerFactory<>(configProps);
}
#Bean
public KafkaTemplate<String, Optional<Map<String, List<ImmutablePair<String, String>>>>> requestKafkaTemplate() {
return new KafkaTemplate<>(requestProducerFactory());
}
}
}
Listener Class:
#Component
public class Consumer {
#KafkaListener(
topics = "${topic.name}",
containerFactory = "listenerContainerFactory"
)
#Override
public void listener(
final ConsumerRecord<String, Optional<Map<String, List<ImmutablePair<String, String>>>>> consumerRecord,
final #Payload Optional<Map<String, List<ImmutablePair<String, String>>>> payload
) {
}
}
Listner Config:
#Configuration
public class KafkaListenerConfig {
#Value(value = "${spring.kafka.bootstrap-servers}")
private String bootstrapAddress;
#Value(value = "${topic.name}")
private String resolvedTreeQueueName;
#Bean
public ConsumerFactory<String, Optional<Map<String, List<ImmutablePair<String, String>>>>> resolvedTreeConsumerFactory() {
final Map<String, Object> props = new HashMap<>();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapAddress);
props.put(ConsumerConfig.GROUP_ID_CONFIG, resolvedTreeQueueName);
return new DefaultKafkaConsumerFactory<>(props, new StringDeserializer(), new CustomDeserializer());
}
#Bean
public ConcurrentKafkaListenerContainerFactory<String, Optional<Map<String, List<ImmutablePair<String, String>>>>> resolvedTreeListenerContainerFactory() {
final ConcurrentKafkaListenerContainerFactory<String, Optional<Map<String, List<ImmutablePair<String, String>>>>> factory = new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(resolvedTreeConsumerFactory());
return factory;
}
}
TopicConfig:
#Configuration
public class TopicConfig {
#Value(value = "${spring.kafka.bootstrap-servers}")
private String bootstrapAddress;
#Value(value = "${topic.name}")
private String requestQueue;
#Bean
public KafkaAdmin kafkaAdmin() {
Map<String, Object> configs = new HashMap<>();
configs.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapAddress);
return new KafkaAdmin(configs);
}
#Bean
public NewTopic requestTopic() {
return new NewTopic(requestQueue, 1, (short) 1);
}
}
application.properties:
spring.kafka.bootstrap-servers=${spring.embedded.kafka.brokers}
This assignment is the most important assignment that would bind the embedded instance port to the KafkaTemplate and, KafkaListners.
Following the above implementation, you could open dynamic ports per test class and, it would be more convenient.

Resources