spring-kafka 2.0.x kafka .11 and spring-boot 2.0.0.RELEASE compatibility issues: java.lang.NoSuchMethodError - spring-boot

Question

When upgrading from spring-boot 2.0.0.Mx to 2.0.0.RELEASE it looks like there are incompatibilities between spring-boot, spring-kafka 2.0.x, and kafka .11. The compatibility issues are clearly defined between spring-kafka and the kafka version here. However when using the compatible versions mentioned and upgrading to spring-boot 2.0.0.RELEASE the following exception is thrown.
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2018-04-12 10:48:34.501 ERROR 13141 --- [ main] o.s.boot.SpringApplication : Application run failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'kafkaListenerContainerFactory' defined in class path resource [org/springframework/boot/autoconfigure/kafka/KafkaAnnotationDrivenConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory]: Factory method 'kafkaListenerContainerFactory' threw exception; nested exception is java.lang.BootstrapMethodError: java.lang.NoSuchMethodError: org.springframework.kafka.listener.config.ContainerProperties.setClientId(Ljava/lang/String;)V
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:587)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1250)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1099)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:545)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:502)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:312)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:310)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:758)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:868)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:549)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:140)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:752)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:388)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:327)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1246)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1234)
at Server.main(Server.kt:19)
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory]: Factory method 'kafkaListenerContainerFactory' threw exception; nested exception is java.lang.BootstrapMethodError: java.lang.NoSuchMethodError: org.springframework.kafka.listener.config.ContainerProperties.setClientId(Ljava/lang/String;)V
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:579)
... 18 common frames omitted
Caused by: java.lang.BootstrapMethodError: java.lang.NoSuchMethodError: org.springframework.kafka.listener.config.ContainerProperties.setClientId(Ljava/lang/String;)V
at org.springframework.boot.autoconfigure.kafka.ConcurrentKafkaListenerContainerFactoryConfigurer.configureContainer(ConcurrentKafkaListenerContainerFactoryConfigurer.java:99)
at org.springframework.boot.autoconfigure.kafka.ConcurrentKafkaListenerContainerFactoryConfigurer.configure(ConcurrentKafkaListenerContainerFactoryConfigurer.java:80)
at org.springframework.boot.autoconfigure.kafka.KafkaAnnotationDrivenConfiguration.kafkaListenerContainerFactory(KafkaAnnotationDrivenConfiguration.java:72)
at org.springframework.boot.autoconfigure.kafka.KafkaAnnotationDrivenConfiguration$$EnhancerBySpringCGLIB$$b9d721e8.CGLIB$kafkaListenerContainerFactory$1(<generated>)
at org.springframework.boot.autoconfigure.kafka.KafkaAnnotationDrivenConfiguration$$EnhancerBySpringCGLIB$$b9d721e8$$FastClassBySpringCGLIB$$25044a84.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:361)
at org.springframework.boot.autoconfigure.kafka.KafkaAnnotationDrivenConfiguration$$EnhancerBySpringCGLIB$$b9d721e8.kafkaListenerContainerFactory(<generated>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
... 19 common frames omitted
Caused by: java.lang.NoSuchMethodError: org.springframework.kafka.listener.config.ContainerProperties.setClientId(Ljava/lang/String;)V
at java.lang.invoke.MethodHandleNatives.resolve(Native Method)
at java.lang.invoke.MemberName$Factory.resolve(MemberName.java:975)
at java.lang.invoke.MemberName$Factory.resolveOrFail(MemberName.java:1000)
at java.lang.invoke.MethodHandles$Lookup.resolveOrFail(MethodHandles.java:1394)
at java.lang.invoke.MethodHandles$Lookup.linkMethodHandleConstant(MethodHandles.java:1750)
at java.lang.invoke.MethodHandleNatives.linkMethodHandleConstant(MethodHandleNatives.java:477)
... 32 common frames omitted
According to Artem Bilan in this random gitter thread, spring-boot 2 is only compatible with spring-kafka 2.1.x. I couldn't find any documentation stating this.
Given all of this these are my only options:
Upgrade to kafka 1.0 because spring-kafka 2.1 is only compatible with kafka 1.0.
Stick with an older version of spring-boot.
Configure kafka manually (I haven't proven this will work but would make sense).
Is there something in the works to allow compatibility between spring-boot 2 and older version of spring-kafka?

Answer 1

I do like this in POM to downgrade Spring Kafka:
<properties>
<spring-kafka.version>2.0.4.RELEASE</spring-kafka.version>
</properties>
Then I get the same NoSuchMethodError. From there I jumped to the ConcurrentKafkaListenerContainerFactoryConfigurer and decided to override it according KafkaAnnotationDrivenConfiguration:
#Bean
public ConcurrentKafkaListenerContainerFactoryConfigurer kafkaListenerContainerFactoryConfigurer(
KafkaProperties kafkaProperties,
ObjectProvider<RecordMessageConverter> messageConverterObjectProvider,
ObjectProvider<KafkaTemplate<Object, Object>> kafkaTemplateObjectProvider) {
RecordMessageConverter messageConverter = messageConverterObjectProvider.getIfUnique();
KafkaTemplate<Object, Object> kafkaTemplate = kafkaTemplateObjectProvider.getIfUnique();
return new ConcurrentKafkaListenerContainerFactoryConfigurer() {
#Override
public void configure(ConcurrentKafkaListenerContainerFactory<Object, Object> listenerFactory,
ConsumerFactory<Object, Object> consumerFactory) {
listenerFactory.setConsumerFactory(consumerFactory);
configureListenerFactory(listenerFactory);
configureContainer(listenerFactory.getContainerProperties());
}
private void configureListenerFactory(
ConcurrentKafkaListenerContainerFactory<Object, Object> factory) {
PropertyMapper map = PropertyMapper.get();
KafkaProperties.Listener properties = kafkaProperties.getListener();
map.from(properties::getConcurrency).whenNonNull().to(factory::setConcurrency);
map.from(() -> messageConverter).whenNonNull()
.to(factory::setMessageConverter);
map.from(() -> kafkaTemplate).whenNonNull().to(factory::setReplyTemplate);
map.from(properties::getType).whenEqualTo(KafkaProperties.Listener.Type.BATCH)
.toCall(() -> factory.setBatchListener(true));
}
private void configureContainer(ContainerProperties container) {
PropertyMapper map = PropertyMapper.get();
KafkaProperties.Listener properties = kafkaProperties.getListener();
map.from(properties::getAckMode).whenNonNull().to(container::setAckMode);
map.from(properties::getAckCount).whenNonNull().to(container::setAckCount);
map.from(properties::getAckTime).whenNonNull().as(Duration::toMillis)
.to(container::setAckTime);
map.from(properties::getPollTimeout).whenNonNull().as(Duration::toMillis)
.to(container::setPollTimeout);
map.from(properties::getNoPollThreshold).whenNonNull()
.to(container::setNoPollThreshold);
map.from(properties::getIdleEventInterval).whenNonNull().as(Duration::toMillis)
.to(container::setIdleEventInterval);
map.from(properties::getMonitorInterval).whenNonNull().as(Duration::getSeconds)
.as(Number::intValue).to(container::setMonitorInterval);
}
};
}
Right, that's mostly copy-paste, but at least I stop to get that exception and propagate as much properties as possible.