Issue with SASL mechanism in Kafka - spring-boot

Question

Use case -
Push data to Confluent cloud kafka from spring boot
Error -
java.lang.IllegalArgumentException: Could not find a 'KafkaClient' entry in the JAAS configuration. System property 'java.security.auth.login.config' is not set
I have gone through multiple discussions in SO and seems like I am not missing anything , below is my kafka config , I am still not sure why it's unable to create the client
import org.apache.kafka.clients.CommonClientConfigs;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.common.config.SaslConfigs;
//import org.apache.kafka.common.config.SaslConfigs;
import org.apache.kafka.common.security.plain.PlainLoginModule;
import org.apache.kafka.common.serialization.StringSerializer;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.core.DefaultKafkaProducerFactory;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.core.ProducerFactory;
import org.springframework.kafka.support.serializer.JsonSerializer;
import java.util.HashMap;
import java.util.Map;
#Configuration
public class KakfaConfiguration {
#Bean
public ProducerFactory<String, User> producerFactory() {
Map<String, Object> config = new HashMap<>();
config.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "pkc-l7pr2.ap-south-1.aws.confluent.cloud:9092");
config.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
config.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class);
config.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SASL_SSL");
config.put(SaslConfigs.SASL_MECHANISM, "PLAIN");
config.put("sasl.jaas.config", "org.apache.kafka.common.security.plain.PlainLoginModule required username='test' password='tes1/FRi/BNfpRhWwLlxb573NdgpQZ6SCTZ';");
return new DefaultKafkaProducerFactory<>(config);
}
#Bean
public KafkaTemplate<String, User> kafkaTemplate() {
return new KafkaTemplate<>(producerFactory());
}
}
I am providing jaas config in the properties so not sure why it's unable to pick up the value. Any help is highly appreciated
Seems like Jaas values are not being passed , not sure what's the reason -
acks = 1
batch.size = 16384
block.on.buffer.full = false
bootstrap.servers = [pkc-l7pr2.ap-south-1.aws.confluent.cloud:9092]
buffer.memory = 33554432
client.id =
compression.type = none
connections.max.idle.ms = 540000
interceptor.classes = null
key.serializer = class org.apache.kafka.common.serialization.StringSerializer
linger.ms = 0
max.block.ms = 60000
max.in.flight.requests.per.connection = 5
max.request.size = 1048576
metadata.fetch.timeout.ms = 60000
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.sample.window.ms = 30000
partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
receive.buffer.bytes = 32768
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retries = 0
retry.backoff.ms = 100
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.mechanism = PLAIN
security.protocol = SASL_PLAINTEXT
send.buffer.bytes = 131072
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
timeout.ms = 30000
value.serializer = class org.springframework.kafka.support.serializer.JsonSerializer
I am now able to resolve the issue after upgrading kafka client library to latest however in another program instead of adding the properties in config ,I am keeping in application.properties file and it's failing
Application.properties
spring.kafka.properties.sasl.mechanism=PLAIN
spring.kafka.properties.bootstrap.servers=pkc-l7pr2.ap-south-1.aws.confluent.cloud:9092
spring.kafka.properties.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='test' password='test/FRi/BNfpRhWwLlxb573NdgpQZ6SCTZ';
spring.kafka.properties.security.protocol=SASL_SSL
# Best practice for higher availability in Apache Kafka clients prior to 3.0
spring.kafka.properties.session.timeout.ms=45000
spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.IntegerSerializer
spring.kafka.producer.value-serializer=org.apache.kafka.common.serialization.StringSerializer
server.port=9099
Error
java.lang.NullPointerException: null
at io.pranav.kafkatest.Producer.lambda$1(KafkaTestApplication.java:61) ~[classes/:na]
at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:106) ~[reactor-core-3.4.16.jar:3.4.16]
at reactor.core.publisher.FluxZip$ZipCoordinator.drain(FluxZip.java:756) ~[reactor-core-3.4.16.jar:3.4.16]
at reactor.core.publisher.FluxZip$ZipInner.onNext(FluxZip.java:915) ~[reactor-core-3.4.16.jar:3.4.16]
at reactor.core.publisher.FluxInterval$IntervalRunnable.run(FluxInterval.java:125) ~[reactor-core-3.4.16.jar:3.4.16]
at reactor.core.scheduler.PeriodicWorkerTask.call(PeriodicWorkerTask.java:59) ~[reactor-core-3.4.16.jar:3.4.16]
at reactor.core.scheduler.PeriodicWorkerTask.run(PeriodicWorkerTask.java:73) ~[reactor-core-3.4.16.jar:3.4.16]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_291]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[na:1.8.0_291]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_291]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[na:1.8.0_291]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_291]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_291]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_291]
Suppressed: java.lang.Exception: #block terminated with an error
at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:99) ~[reactor-core-3.4.16.jar:3.4.16]
at reactor.core.publisher.Flux.blockFirst(Flux.java:2600) ~[reactor-core-3.4.16.jar:3.4.16]
at io.pranav.kafkatest.Producer.generate(KafkaTestApplication.java:61) ~[classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_291]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_291]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_291]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_291]
at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:344) ~[spring-context-5.3.18.jar:5.3.18]
at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:229) ~[spring-context-5.3.18.jar:5.3.18]
at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:166) ~[spring-context-5.3.18.jar:5.3.18]
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:176) ~[spring-context-5.3.18.jar:5.3.18]
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:169) ~[spring-context-5.3.18.jar:5.3.18]
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:143) ~[spring-context-5.3.18.jar:5.3.18]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:421) ~[spring-context-5.3.18.jar:5.3.18]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:378) ~[spring-context-5.3.18.jar:5.3.18]
at org.springframework.boot.context.event.EventPublishingRunListener.started(EventPublishingRunListener.java:108) ~[spring-boot-2.6.6.jar:2.6.6]
at org.springframework.boot.SpringApplicationRunListeners.lambda$started$5(SpringApplicationRunListeners.java:78) ~[spring-boot-2.6.6.jar:2.6.6]
at java.util.ArrayList.forEach(ArrayList.java:1259) ~[na:1.8.0_291]
at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:120) ~[spring-boot-2.6.6.jar:2.6.6]
at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:114) ~[spring-boot-2.6.6.jar:2.6.6]
at org.springframework.boot.SpringApplicationRunListeners.started(SpringApplicationRunListeners.java:78) ~[spring-boot-2.6.6.jar:2.6.6]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:309) [spring-boot-2.6.6.jar:2.6.6]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1312) [spring-boot-2.6.6.jar:2.6.6]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1301) [spring-boot-2.6.6.jar:2.6.6]
at io.pranav.kafkatest.KafkaTestApplication.main(KafkaTestApplication.java:25) [classes/:na]
Producer
class Producer{
private KafkaTemplate<Integer, String> template;
Faker faker;
#EventListener(ApplicationStartedEvent.class)
public void generate() {
faker = Faker.instance();
System.out.println(faker.hobbit().quote());
final Flux<Long> interval = Flux.interval(Duration.ofMillis(1_000));
System.out.println(interval);
System.out.println("Next Interval ..."+faker.random().nextInt(42));
final Flux<String> quotes = Flux.fromStream(Stream.generate(() -> faker.hobbit().quote()));
System.out.println("Quote is .." + quotes.toString());
Flux.zip(interval, quotes).map(it -> template.send("hobbit", 11, "hi") ).blockFirst();
System.out.println("Data pushed..");
}

Answer 1

sasl.jaas.conf was introduced in kafka-client 0.10.2. Can you check you have a recent kafka-clients release. The SASL_JAAS_CONFIG constant is located in the org.apache.kafka.common.config.SaslConfigs class