java.lang.NoClassDefFoundError when running KafkaEmbedded for Unit test - spring

Question

I have a Spring-Kafka project running Spring Boot 2.0.2. In the unit test I am trying to start a KafkaEmbedded server. I get this error.
java.lang.NoClassDefFoundError: org/apache/kafka/common/security/auth/SecurityProtocol
at kafka.utils.TestUtils$.createBrokerConfig(TestUtils.scala:222)
at kafka.utils.TestUtils.createBrokerConfig(TestUtils.scala)
at org.springframework.kafka.test.rule.KafkaEmbedded.createBrokerProperties(KafkaEmbedded.java:278)
at org.springframework.kafka.test.rule.KafkaEmbedded.before(KafkaEmbedded.java:224)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:46)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:190)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.common.security.auth.SecurityProtocol
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 13 more
I am aware that this is because of the dependencies mis-match. But I have added the latest versions spring-kafka-test & kafka-clients. I am aware that this class "SecurityProtocol" is in the Kafka-clients jar but adding that dependency (old & new versions) still does not solve the problem.
The following is the dependencies in pom file.
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.0.2.RELEASE</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<java.version>1.8</java.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka-test</artifactId>
<version>2.1.6.RELEASE</version>
<scope>test</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients -->
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>1.0.0</version>
<scope>test</scope>
</dependency>
Here is the basic Unit test that uses the EmbeddedKafka.
#RunWith(SpringRunner.class)
#SpringBootTest
public class KafkaDemoApplicationTests {
#ClassRule
public static KafkaEmbedded embeddedKafka = new KafkaEmbedded(1, true,
Topology.DEMO_TOPIC);
#Test
public void contextLoads() {
}
}

Answer 1

You have the wrong kafka (scala) jars on the classpath.
java.lang.NoClassDefFoundError: org/apache/kafka/common/security/auth/SecurityProtocol
SecurityProtocol was moved from org.apache.kafka.common.protocol to org.apache.kafka.common.security.auth in kafka-clients 1.0.0.
So it looks like you have (somehow) got the older kafka_2.11 jar on the CP; spring-kafka-test should bring in the right version. Perhaps you have overrides in your pom that you are not showing?

Answer 2

The solution was to avoid specifying versions for all Kafka related dependencies in the Spring Boot project's pom.xml file. Like this -
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka-test</artifactId>
<scope>test</scope>
</dependency>

Answer 3

If you are using kafka-client (not spring kafka) , make sure the kafka-client version used in your pom, is the same version as used in the spring-kafka test dependency.
Here in the spring-kafka-test maven repo [https://mvnrepository.com/artifact/org.springframework.kafka/spring-kafka-test], it lists all the dependencies, including the kafka-client it uses.
So either update you kafka-client version or the spring-kafka-test version
to make sure clients are the same.