Spring Reactive kafka Receiver always overrides bootstrap server to localhost - spring-boot

I am trying to write Spring boot applications for a Reactive Kafka consumer using #EnableKafka and #KafkaListener annotations.
i have configured my kafka brokers are on different machine. when i give bootstrap-server to advertised host of kafka brokers, it is always overriding advertised host ip addresses to localhost. below are my code.
pom.xml file:-
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-webflux</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
</dependency>
<dependency>
<groupId>io.projectreactor.kafka</groupId>
<artifactId>reactor-kafka</artifactId>
<version>1.0.0.RELEASE</version>
</dependency>
config:-
#Configuration
#EnableKafka
public class AppConfig {
#Bean
Map consumerProps() {
Map<String, Object> props = new HashMap<>();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.12.12.24:9092,192.14.14.28:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "example-group");
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, true);
props.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, "100");
props.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, "15000");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,StringDeserializer.class);
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
return props;
}
#Bean
ReceiverOptions receiverOptions() {
ReceiverOptions receiverOptions = ReceiverOptions.create(consumerProps()).subscription(Arrays.asList("hellochange"));
return receiverOptions;
}
#Bean
public KafkaReceiver kafkaReceiver() {
return KafkaReceiver.create(receiverOptions());
}
}
Consumer:-
#Service
public class ChangeListener {
#Autowired
KafkaReceiver kafkaReceiver;
#KafkaListener(topics="hellochange",groupId="example-group")
public void receiver() {
kafkaReceiver.receive().subscribe(System.out::println);
}
}
Console:-
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [localhost:9092]
check.crcs = true
client.id =
connections.max.idle.ms = 540000
enable.auto.commit = true
2018-06-07 19:59:17.640 WARN 23536 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-1, groupId=example-group] Connection to node -1 could not be established. Broker may not be available.
I have validated without spring configurations in a simple consumer and non-reactive Spring kafka, for both, it is working fine. only Reactor kafka with EnableKafka and KafkaListener annotations i am getting this problem.
am i missing something/ doing wrong here ?
can we use EnableKafka and KafkaListener annotations with Reactor Kafka in Spring boot?
P.S. i understood, #EnableKafka and #KafkaListener are not reactive, if i remove spring-kafka from pom.xml, both the annotations are not available.
Like #EnableKafka and #KafkaListener for non-reactive kafka, is there any annotations are available to configure reactive kafka consumer with Spring boot application?

You cannot use KafkaListener annotations with Reactor Kafka; #KafkaListener is not reactive.

Related

Spring boot 3 and JMS configuration for Amazon SQS

The latest spring boot (i.e. 3.0.1) uses Spring JMS 6.0.3. And the latest Spring JMS that comes with the latest spring boot for ConnectionFactory uses jakarta.jms.ConnectionFactory, while the older JMS was using javax.jms.ConnectionFactory. This was my bean setup and configuration before:
#Configuration
#EnableJms
class JmsConfiguration {
#Autowired
private lateinit var sqsClient: SqsClient
#Bean
fun sqsConnectionFactory(): SQSConnectionFactory =
SQSConnectionFactory(ProviderConfiguration(), sqsClient)
#Bean
fun jmsListenerContainerFactory(): DefaultJmsListenerContainerFactory {
val factory = DefaultJmsListenerContainerFactory()
factory.setConnectionFactory(sqsConnectionFactory())
factory.setDestinationResolver(DynamicDestinationResolver())
factory.setSessionAcknowledgeMode(CLIENT_ACKNOWLEDGE)
return factory
}
#Bean
fun jmsTemplate(): JmsTemplate = JmsTemplate(sqsConnectionFactory())
}
However with the latest Spring boot, I cannot use SQSConnectionFactory, because it hasn't implemented its code based on jakarta rather javax.
Here are my SQS dependencies used:
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>sqs</artifactId>
<version>2.19.19</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>amazon-sqs-java-messaging-lib</artifactId>
<version>2.0.3</version>
</dependency>
So now I cannot set the connection factory.setConnectionFactory(sqsConnectionFactory()), because it complains sqsConnectionFactory() is not type of jakarta ConnectionFactory. Is there any solution?

Unit testing errors with #EmbeddedKafka and spring boot

I have IdentManagerApp with regular SpringBoot 2.6.8 and Kafka producers and consumers. Additionally I have custom Kafka configurations and failure scenarios handling with Retry/DLT topics. The respective factories creation need beans like :
ConcurrentKafkaListenerContainerFactory,
ConcurrentKafkaListenerContainerFactoryConfigurer configurer,
ConsumerFactory<Object, Object> kafkaConsumerFactoryOriginalToRetry,
KafkaTemplate<String, String> kafkaTemplate,
KafkaListenerEndpointRegistry registry,
etc
While all of that works smooth while running application, it doesnt go well while running Junits.
I keep getting errors like -
'org.springframework.boot.autoconfigure.kafka.ConcurrentKafkaListenerContainerFactoryConfigurer' that could not be found.
OR
Field registry in <package> required a bean of type 'org.springframework.kafka.config.KafkaListenerEndpointRegistry' that could not be found.
I fix one and other one pops up.
Is EmbeddedKafka compatible with exhaustive unit testing for springboot + Kafka ?
My simple test config looks like -
Main class :
#DirtiesContext
#EmbeddedKafka(partitions = 1, brokerProperties = { "listeners=PLAINTEXT://localhost:8888", "port=8888" })
#EnableKafkaCustomization
#SpringBootTest(classes = IdentManagerApp.class)
class OrchestratorAppTests {
#Test
void contextLoads() {
}
}
Ultra simple test class, doesnt even have Kafka related operations yet -
#ExtendWith(MockitoExtension.class)
#Import({
KafkaConfiguration.class
})
#ContextConfiguration
#WebMvcTest(controllers = IncomingRequestController.class)
public class IncomingRequestControllerTest { . . . . . }
Pom has below with Springboot's 2.6.8
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka-test</artifactId>
<scope>test</scope>
</dependency>

Spring boot/Cloud Eureka DiscoveryClient - No such host is known

I want to try out microservices using 2.5.4 spring boot and getting problems in discovery
Steps:
Created a RestTemplate with #LoadBalanced, and trying to call the service using "application name in the url"
I have the register and fetch registry true in properties (thought this should get available services?)
Error: No such host is known
I am trying
#Autowired
private DiscoveryClient discoveryClient;
and do
discoveryClient.getInstances("myappservice-name").forEach((ServiceInstance s) -> {
System.out.println(ToStringBuilder.reflectionToString(s));
});
But all examples tell to use an endpoint? or commandLineRunner. Both I'm looking for auto loading
https://spring.io/guides/gs/service-registration-and-discovery/
https://spring.io/blog/2015/01/20/microservice-registration-and-discovery-with-spring-cloud-and-netflix-s-eureka
Not ok to call the below for each app
#RequestMapping("/service-instances/{applicationName}") public
List<ServiceInstance> serviceInstancesByApplicationName(
How can I register automatically?
EDIT: More detailed question
(1) SERVICE App
bootstrap - spring.application.name=pluralsight-toll-service
application props
server.port=0
eureka.client.register-with-eureka=true
eureka.client.fetch-registry=true
eureka.instance.instance-id=${spring.application.name}:${random.int}
eureka.instance.hostname=localhost
management.endpoints.web.exposure.include=*
App
#SpringBootApplication
#EnableEurekaClient
public class PluralsightEurekaTollrateServiceApplication {
I see app registered in eureka server
(2) Client
bootstrap
spring.application.name=pluralsight-tollrate-billboard
eureka.client.register-with-eureka=true
eureka.client.fetch-registry=true
application.props
server.port=8081
eureka.client.register-with-eureka=true
eureka.client.fetch-registry=true
management.endpoints.web.exposure.include=*
App
#SpringBootApplication
#EnableEurekaClient
#EnableDiscoveryClient
public class PluralsightEurekaTollrateBillboardApplication {
Controller
#LoadBalanced
#Bean
public RestTemplate restTemplate(RestTemplateBuilder builder) {
return builder.build();
}
#Autowired
RestTemplate restTemplate;
#RequestMapping("/dashboard")
public String GetTollRate(#RequestParam int stationId, Model m) {
TollRate tr = restTemplate.getForObject("http://pluralsight-toll-service/tollrate/" + stationId, TollRate.class);
Error: pluralsight-toll-service host is unknown
How can call service from client using the name
Fixed the issue
Its all down to client project dependencies
<!-- <dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-netflix-eureka-client</artifactId>
<version>3.0.3</version>
</dependency> -->
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
</dependency>
The names are too close
spring-cloud-netflix-eureka-client is wrong
spring-cloud-starter-netflix-eureka-client is the right one
Because of the wrong one, I had to add eureka-client of com.netflix.eureka. Cleaned all and it works
(Also I was missing spring-cloud , as I was creating from eclipse. In future will use initializ only )

Spring Boot JMS No JmsTemplate bean available

I'm trying to send a message with JMS from my application.
I add in my pom
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-jms</artifactId>
</dependency>
<dependency>
<groupId>javax.jms</groupId>
<artifactId>jms</artifactId>
<version>1.1</version>
</dependency>
The spring getting started say
JmsTemplate and ConnectionFactory are created automatically by Spring Boot. In this case, the ActiveMQ broker runs embedded.
and in my batch writer
#Autowired
JmsTemplate jmsTemplate,
void writer(List<String> items) {
jmsTemplate.convertAndSend(items);
}
But the bean JmsTemplate is not found
No qualifying bean of type 'org.springframework.jms.core.JmsTemplate' available: expected at least 1 bean which qualifies as autowire candidate
I tried to add an message converter in the #configuration
#Bean
public MessageConverter jacksonJmsMessageConverter() {
MappingJackson2MessageConverter converter = new MappingJackson2MessageConverter();
converter.setTargetType(MessageType.TEXT);
converter.setTypeIdPropertyName("_type");
return converter;
}
I tried to add the #EnableJMS (even if it's just for listener...)
But it dosen't work...
I don't understand why, on tutorials it looks like easy...
To work we need to create a jmsTemplate bean
#Bean
public ConnectionFactory getConnectionFactory() {
TibjmsConnectionFactory connectionFactory = new TibjmsConnectionFactory(urlBrocker);
return connectionFactory;
}
#Bean
public JmsTemplate jmsTemplate() {
JmsTemplate template = new JmsTemplate();
template.setConnectionFactory(getConnectionFactory());
template.setPubSubDomain(false); // false for a Queue, true for a Topic
return template;
}

Spring Integration JMS Inbound Gateway reply channel has no subscriber

I have test with JMS Inbound Gateway in Spring Integration 5.1.3
But I got error as following:
Caused by: org.springframework.integration.MessageDispatchingException: Dispatcher has no subscribers
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:138) ~[spring-integration-core-5.1.3.RELEASE.jar:5.1.3.RELEASE]
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:105) ~[spring-integration-core-5.1.3.RELEASE.jar:5.1.3.RELEASE]
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:73) ~[spring-integration-core-5.1.3.RELEASE.jar:5.1.3.RELEASE]
POM:
<dependencies>
<dependency>
<groupId>org.springframework.integration</groupId>
<artifactId>spring-integration-jms</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-activemq</artifactId>
</dependency>
</dependencies>
I configure the Inbound Gateway as following:
#Bean
JmsInboundGateway jmsInboundGateway(
MessageChannel errorChannel,
ConnectionFactory connectionFactory,
DetailJmsProperties properties) {
final Listener listener = properties.getListener();
return Jms
.inboundGateway(connectionFactory)
.destination("request-queue")
.requestChannel("inputChannel")
.replyChannel("outputChannel")
.defaultReplyQueueName("response-queue")
.get();
}
And, the Service Activator:
#ServiceActivator(inputChannel = "inputChannel", outputChannel = "outputChannel")
public String process(String request) {
String response = null;
try {
LOGGER.info("Received message content: [{}]", request);
response = request + " was processed";
}
catch (Exception e) {
LOGGER.error("Error", e);
}
return response;
}
By the way, it works only if I remove outputChannel = "outputChannel" in Service Activator.
Is there any explanation for this issue, do I have any misunderstanding ?
You can't use DSL factories (Jms) like that, they are intended for use in a DSL flow
#Bean
IntegrationFLow flow()
return IntegrationFlows.from(jmsInboundGateway())
.handle("service", "process")
.get();
The DSL processing does all the wiring.
It works without the channel because a component without an output channel routes the reply to the replyChannel header.
If you don't want to use the DSL, you must wire up the inbound gateway directly as a bean instead of using the Jms factory.

Resources