spring boot kafka failed with "Broker may not be available" while using testcontainers with kafka, zookeeper, schema registry - spring-boot

Question

I've run testcontainers like below in the test code.
#Testcontainers
public class TestEnvironmentSupport {
static String version = "5.4.0";
static DockerImageName kafkaImage = DockerImageName.parse("confluentinc/cp-server").withTag(version);
static DockerImageName zookeeperImage = DockerImageName.parse("confluentinc/cp-zookeeper").withTag(version);
static DockerImageName schemaRegistryImage = DockerImageName.parse("confluentinc/cp-schema-registry").withTag(version);
static Network network = Network.newNetwork();
#Container
static GenericContainer zookeeper = new GenericContainer<>(zookeeperImage)
.withNetwork(network)
.withCreateContainerCmdModifier(cmd -> cmd.withHostName("zookeeper"))
.withExposedPorts(2181)
.withEnv("ZOOKEEPER_CLIENT_PORT", "2181")
.withEnv("ZOOKEEPER_TICK_TIME", "2000");
#Container
static GenericContainer kafka = new GenericContainer<>(kafkaImage)
.withNetwork(network)
.withCreateContainerCmdModifier(cmd -> cmd.withHostName("kafka"))
.withExposedPorts(9092)
.dependsOn(zookeeper)
.withEnv("KAFKA_BROKER_ID", "1")
.withEnv("KAFKA_ZOOKEEPER_CONNECT", "zookeeper:2181")
.withEnv("KAFKA_LISTENER_SECURITY_PROTOCOL_MAP", "PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT")
.withEnv("KAFKA_ADVERTISED_LISTENERS", "PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092")
.withEnv("KAFKA_CONFLUENT_SCHEMA_REGISTRY_URL", "schema-registry:8081");
#Container
static GenericContainer schemaRegistry = new GenericContainer<>(schemaRegistryImage)
.withNetwork(network)
.withCreateContainerCmdModifier(cmd -> cmd.withHostName("schema-registry"))
.withExposedPorts(8081)
.dependsOn(zookeeper, kafka)
.withEnv("SCHEMA_REGISTRY_HOST_NAME", "schema-registry")
.withEnv("SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL", "zookeeper:2181");
#Test
void test() {
assertTrue(zookeeper.isRunning());
assertTrue(kafka.isRunning());
assertTrue(schemaRegistry.isRunning());
}
}
and it works totally fine.
But the problem occurred when I tried to run spring boot test with the above testcontainer configuration because testcontainer dynamically generates broker ports but NetworkClient continuously accessing the broker with localhost:9092 even though I dynamically override properties on #SpringBootTest code like below
#DynamicPropertySource
static void testcontainerProperties(final DynamicPropertyRegistry registry) {
var bootstrapServers = kafka.getHost() + ":" + kafka.getMappedPort(9092);
var schemaRegistryUrl = "http://" + schemaRegistry.getHost() + ":" + schemaRegistry.getMappedPort(8081);
registry.add("spring.cloud.stream.kafka.binder.brokers", () -> bootstrapServers);
registry.add("bootstrap.servers", () -> bootstrapServers);
registry.add("schema.registry.url", () -> schemaRegistryUrl);
registry.add("spring.cloud.stream.kafka.default.consumer.configuration.schema.registry.url", () -> schemaRegistryUrl);
}
Below is AdminClientConfig log on startup time, and it shows that bootstrap.servers = [localhost:56001] on which the port is dynamically binded by testcontainer.
2021-02-21 20:28:52.291 INFO 78241 --- [ Test worker] o.a.k.clients.admin.AdminClientConfig : AdminClientConfig values:
bootstrap.servers = [localhost:56013]
client.dns.lookup = use_all_dns_ips
Even though I set like these it keeps trying to connect to localhost:9092 like below.
2021-02-21 20:28:52.457 INFO 78241 --- [ Test worker] o.a.kafka.common.utils.AppInfoParser : Kafka startTimeMs: 1613906932454
2021-02-21 20:28:53.095 WARN 78241 --- [| adminclient-1] org.apache.kafka.clients.NetworkClient : [AdminClient clientId=adminclient-1] Connection to node 1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
2021-02-21 20:28:53.202 WARN 78241 --- [| adminclient-1] org.apache.kafka.clients.NetworkClient : [AdminClient clientId=adminclient-1] Connection to node 1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
2021-02-21 20:28:53.407 WARN 78241 --- [| adminclient-1] org.apache.kafka.clients.NetworkClient : [AdminClient clientId=adminclient-1] Connection to node 1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
And below is the result of docker ps while running spring boot test.
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e340d9e15fe4 confluentinc/cp-schema-registry:5.4.0 "/etc/confluent/dock…" 46 seconds ago Up 46 seconds 0.0.0.0:56014->8081/tcp optimistic_joliot
ad3bf06df4b3 confluentinc/cp-server:5.4.0 "/etc/confluent/dock…" 55 seconds ago Up 54 seconds 0.0.0.0:56013->9092/tcp infallible_brown
f7fa5f4ae23c confluentinc/cp-zookeeper:5.4.0 "/etc/confluent/dock…" About a minute ago Up 59 seconds 0.0.0.0:56012->2181/tcp, 0.0.0.0:56011->2888/tcp, 0.0.0.0:56010->3888/tcp agitated_leavitt
b1c036cdf00b testcontainers/ryuk:0.3.0 "/app" About a minute ago Up About a minute 0.0.0.0:56009->8080/tcp testcontainers-ryuk-68190eaa-8513-4dd8-ab67-175275f15a82
I tried run testcontainers with docker compose module but it has the same issue.
What am I doing wrong?
Please help.

Answer 1

It's trying to connect to localhost:9092 because you've tried to connect to the advertised PLAINTEXT_HOST port, and that's the address it'll return. You shouldn't need to advertise two listeners for tests, so try using kafka:29092 directly instead of calling the mapped port method. Also, unless you have a specific need for server-side schema validation, you only need confluentinc/cp-kafka image.
spring-kafka embedded broker should work your tests, too, so you wouldn't need testcontainers