EmbeddedKafka AdminClient shuts down before Spring app starts for tests - spring-boot

Question

I'm trying to write integration tests for a Spring Kafka app (Spring Boot 2.0.6, Spring Kafka 2.1.10) and am seeing lots of instance of INFO org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x166e432ebec0001 type:create cxid:0x5e zxid:0x24 txntype:-1 reqpath:n/a Error Path:/brokers/topics/my-topic/partitions Error:KeeperErrorCode = NoNode for /brokers/topics/my-topic/partitions and various flavors of the path (/brokers, /brokers/topics, etc.) that show in the logs before the Spring app starts. The AdminClient then shuts down and this message is logged:
DEBUG org.apache.kafka.common.network.Selector - [SocketServer brokerId=0] Connection with /127.0.0.1 disconnected
java.io.EOFException: null
at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:124)
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:93)
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:235)
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:196)
at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:547)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:483)
at org.apache.kafka.common.network.Selector.poll(Selector.java:412)
at kafka.network.Processor.poll(SocketServer.scala:575)
at kafka.network.Processor.run(SocketServer.scala:492)
at java.lang.Thread.run(Thread.java:748)
I'm using the #ClassRule startup option in the test like so:
#ClassRule
#Shared
private KafkaEmbedded embeddedKafka = new KafkaEmbedded(1, true, 'my-topic')
, autowiring a KafkaTemplate, and setting the Spring properties for the connection based on the embedded Kafka values:
def setupSpec() {
System.setProperty('spring.kafka.bootstrap-servers', embeddedKafka.getBrokersAsString());
System.setProperty('spring.cloud.stream.kafka.binder.zkNodes', embeddedKafka.getZookeeperConnectionString());
}
Once the Spring app starts, I can see again instance of the user-level KeeperException messages: o.a.z.server.PrepRequestProcessor : Got user-level KeeperException when processing sessionid:0x166e445836d0001 type:setData cxid:0x6b zxid:0x2b txntype:-1 reqpath:n/a Error Path:/config/topics/__consumer_offsets Error:KeeperErrorCode = NoNode for /config/topics/__consumer_offsets.
Any idea where I'm going wrong here? I can provide other setup information and log messages but just took an educated guess on what may be most helpful initially.

Answer 1

I'm not familiar with Spock, but what I know that #KafkaListener method is invoked on its own thread, therefore you can't just assert it in the then: block directly.
You need to ensure somehow a blocking wait in your test-case.
I tried with the BlockingVariable against the real service not mock and I see in logs your println(message). But that BlockingVariable still doesn't work for me somehow:
#DirtiesContext
#SpringBootTest(classes = [KafkaIntTestApplication.class])
#ActiveProfiles('test')
class CustomListenerSpec extends Specification {
#ClassRule
#Shared
public KafkaEmbedded embeddedKafka = new KafkaEmbedded(1, false, 'my-topic')
#Autowired
private KafkaTemplate<String, String> template
#SpyBean
private SimpleService service
final def TOPIC_NAME = 'my-topic'
def setupSpec() {
System.setProperty('spring.kafka.bootstrapServers', embeddedKafka.getBrokersAsString());
}
def 'Sample test'() {
given:
def testMessagePayload = "Test message"
def message = MessageBuilder.withPayload(testMessagePayload).setHeader(KafkaHeaders.TOPIC, TOPIC_NAME).build()
def result = new BlockingVariable<Boolean>(5)
service.handleMessage(_) >> {
result.set(true)
}
when: 'We put a message on the topic'
template.send(message)
then: 'the service should be called'
result.get()
}
}
And logs are like this:
2018-11-05 13:38:51.089 INFO 8888 --- [ntainer#0-0-C-1] o.s.k.l.KafkaMessageListenerContainer : partitions assigned: [my-topic-0, my-topic-1]
Test message
BlockingVariable.get() timed out after 5,00 seconds
at spock.util.concurrent.BlockingVariable.get(BlockingVariable.java:113)
at com.example.CustomListenerSpec.Sample test(CustomListenerSpec.groovy:54)
2018-11-05 13:38:55.917 INFO 8888 --- [ main] s.c.a.AnnotationConfigApplicationContext : Closing org.springframework.context.annotation.AnnotationConfigApplicationContext#11ebb1b6: startup date [Mon Nov 05 13:38:49 EST 2018]; root of context hierarchy
Also I had to add this dependency:
testImplementation "org.hamcrest:hamcrest-core"
UPDATE
OK. There real problem that MockConfig was not visible for the test context configuration and that #Import(MockConfig.class) does the trick. Where #Primary also gives us additional signal what bean to pick up for the injection in the test class.

Answer 2

#ArtemBilan's response set me on the right path so thanks to him for chiming in, and I was able to figure it out after looking into other BlockingVariable articles and examples. I used BlockingVariable in a mock's response instead of as a callback. When the mock's response is invoked, make it set the value to true, and the then block just does result.get() and the test passes.
#DirtiesContext
#ActiveProfiles('test')
#SpringBootTest
#Import(MockConfig.class)
class CustomListenerSpec extends TestSpecBase {
#ClassRule
#Shared
private KafkaEmbedded embeddedKafka = new KafkaEmbedded(1, false, TOPIC_NAME)
#Autowired
private KafkaTemplate<String, String> template
#Autowired
private SimpleService service
final def TOPIC_NAME = 'my-topic'
def setupSpec() {
System.setProperty('spring.kafka.bootstrap-servers', embeddedKafka.getBrokersAsString());
}
def 'Sample test'() {
def testMessagePayload = "Test message"
def message = MessageBuilder.withPayload(testMessagePayload).setHeader(KafkaHeaders.TOPIC, TOPIC_NAME).build()
def result = new BlockingVariable<Boolean>(5)
service.handleMessage(_ as String) >> {
result.set(true)
}
when: 'We put a message on the topic'
template.send(message)
then: 'the service should be called'
result.get()
}
}