I have been trying to integrate kafka within spring boot but doesn't seem to work. I am getting an exception while consuming the message. I think the publisher works fine but the consumer fails to deserialize the message.
Caused by: java.lang.ClassCastException: class com.example.schema.avro.Event cannot be cast to
class com.example.schema.avro.Event (com.example.schema.avro.Event is in unnamed module of loader
'app'; com.example.schema.avro.Event is in unnamed module of loader
org.springframework.boot.devtools.restart.classloader.RestartClassLoader #58da768e)
I searched for the exception and the solution mentioned was to specific.avro.reader: true but didn't seem to work for me.
This is my pom file
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-kafka</artifactId>
<version>3.2.4</version>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-schema</artifactId>
<version>2.2.1.RELEASE</version>
</dependency>
<dependency>
<groupId>io.confluent</groupId>
<artifactId>kafka-avro-serializer</artifactId>
<version>5.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
<version>1.11.0</version>
</dependency>
<build>
<plugins>
<plugin>
<groupId>org.apache.avro</groupId>
<artifactId>avro-maven-plugin</artifactId>
<version>1.8.2</version>
<executions>
<execution>
<id>schemas</id>
<phase>generate-sources</phase>
<goals>
<goal>schema</goal>
<goal>protocol</goal>
<goal>idl-protocol</goal>
</goals>
<configuration>
<sourceDirectory>${project.basedir}/src/main/resources/schema/</sourceDirectory>
<outputDirectory>${project.basedir}/target/generated-sources/avro/</outputDirectory>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<repositories>
<repository>
<id>confluent</id>
<url>https://packages.confluent.io/maven/</url>
</repository>
</repositories>
yaml file
spring:
main:
allow-bean-definition-overriding: true
kafka:
bootstrap-servers: localhost:9092
binder:
producer-properties:
key.serializer: io.confluent.kafka.serializers.KafkaAvroSerializer
value.serializer: io.confluent.kafka.serializers.KafkaAvroSerializer
schema.registry.url: http://localhost:8081
consumer-properties:
key.deserializer: io.confluent.kafka.serializers.KafkaAvroDeserializer
value.deserializer: io.confluent.kafka.serializers.KafkaAvroDeserializer
schema.registry.url: http://localhost:8081
specific.avro.reader: true
cloud:
stream:
schemaRegistryClient:
endpoint: http://localhost:8081
bindings:
event-out-0:
destination: event-details
contentType: application/*+avro
event-in-0:
destination: event-details
contentType: application/*+avro
function:
definition: event
And in the code to publish an event
public Event saveEvent(final EventDto eventDto) {
final boolean send = streamBridge.send("event-out-0", new com.example.schema.avro.Event(eventDto.eventId()));
log.info(String.valueOf(send));
}
Configuration
#Configuration
#Slf4j
public class Config {
#Bean
public SchemaRegistryClient schemaRegistryClient(#Value("${spring.cloud.stream.schemaRegistryClient.endpoint}") final String endPoint) {
final ConfluentSchemaRegistryClient client = new ConfluentSchemaRegistryClient();
client.setEndpoint(endPoint);
return client;
}
#Bean
public MessageConverter avroSchemaMessageConverterAnotherBean() {
return new AvroSchemaMessageConverter(new AvroSchemaServiceManagerImpl());
}
#Bean
public Function<EventDto, Event> eventOut() {
return e -> new Event(e.eventId());
}
#Bean
public Consumer<Event> event() {
return e -> log.info("event captured {}", e.getEventId().toString());
}
}
Related
I have a Spring Boot application I'm trying to migrate to use spring-jms dependency instead of spring-activemq.
I have a few tests written with Spock. I also want to use either spring-jms-server or artemis-junit module. These tests use a JmsMessagingTemplate to send a message on a multicast address (topic), the message is then processed by some #JmsListeners on the consumers queues of these topics.
It seems that the test fails sometimes because the message sent via a JmsMessagingTemplate fails to be acknowledged or never arrives at the listeners.
pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.7.2</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<groupId>be.fgov.minfin.esbsoa.ems.ccncsi</groupId>
<artifactId>audit</artifactId>
<version>3.10.0-SNAPSHOT</version>
<properties>
<java.version>1.8</java.version>
<common-lib.version>1.8.0-SNAPSHOT</common-lib.version>
<jacoco.version>0.8.5</jacoco.version>
<sonar.tests>src/test/groovy,src/test-integration/groovy</sonar.tests>
<spock.version>2.2-M3-groovy-3.0</spock.version>
<logbook-spring-boot-starter.version>2.14.0</logbook-spring-boot-starter.version>
<jacoco.version>0.8.5</jacoco.version>
<sonar-maven-plugin.version>3.9.1.2184</sonar-maven-plugin.version>
<maven-compiler-plugin.version>3.8.1</maven-compiler-plugin.version>
<maven-surefire-plugin.version>3.0.0-M5</maven-surefire-plugin.version>
<gmavenplus-plugin.version>1.13.1</gmavenplus-plugin.version>
</properties>
<dependencies>
<dependency>
<groupId>be.fgov.minfin.esbsoa.ems.ccncsi</groupId>
<artifactId>common-lib</artifactId>
<version>${common-lib.version}</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-artemis</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
<exclusions>
<exclusion>
<groupId>com.zaxxer</groupId>
<artifactId>HikariCP</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-validation</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-aop</artifactId>
</dependency>
<dependency>
<groupId>org.zalando</groupId>
<artifactId>logbook-spring-boot-starter</artifactId>
<version>${logbook-spring-boot-starter.version}</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
<dependency>
<groupId>org.apache.tomcat</groupId>
<artifactId>tomcat-jdbc</artifactId>
<version>10.0.23</version>
</dependency>
<dependency>
<groupId>com.ibm.db2</groupId>
<artifactId>jcc</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>artemis-junit</artifactId>
<version>2.21.0</version>
</dependency>
<!-- <!– https://mvnrepository.com/artifact/org.apache.activemq/artemis-server –>-->
<!-- <dependency>-->
<!-- <groupId>org.apache.activemq</groupId>-->
<!-- <artifactId>artemis-server</artifactId>-->
<!-- <version>2.21.0</version>-->
<!-- </dependency>-->
<dependency>
<groupId>org.spockframework</groupId>
<artifactId>spock-spring</artifactId>
<version>${spock.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
<scope>runtime</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>${maven-compiler-plugin.version}</version>
<configuration>
<source>${java.version}</source>
<target>${java.version}</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${maven-surefire-plugin.version}</version>
<configuration>
<enableAssertions>false</enableAssertions>
</configuration>
<executions>
<execution>
<id>unit-tests</id>
<phase>test</phase>
<goals>
<goal>test</goal>
</goals>
<configuration>
<includes>
<include>**/*Spec.java</include>
</includes>
<excludes>
<exclude>**/*ITSpec.java</exclude>
</excludes>
</configuration>
</execution>
<execution>
<id>integration-tests</id>
<phase>integration-test</phase>
<goals>
<goal>test</goal>
</goals>
<configuration>
<!-- Never skip running the tests when the integration-test phase is invoked -->
<skip>false</skip>
<includes>
<!-- Include integration tests within integration-test phase. -->
<include>**/*ITSpec.java</include>
</includes>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.codehaus.gmavenplus</groupId>
<artifactId>gmavenplus-plugin</artifactId>
<version>${gmavenplus-plugin.version}</version>
<configuration>
<testSources>
<testSource>
<directory>src/test/groovy</directory>
<includes>
<include>**/*.groovy</include>
</includes>
</testSource>
<testSource>
<directory>src/test-integration/groovy</directory>
<includes>
<include>**/*.groovy</include>
</includes>
</testSource>
</testSources>
</configuration>
<executions>
<execution>
<id>unit-test-compile</id>
<goals>
<goal>compileTests</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>${jacoco.version}</version>
<executions>
<execution>
<goals>
<goal>prepare-agent</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.sonarsource.scanner.maven</groupId>
<artifactId>sonar-maven-plugin</artifactId>
<version>${sonar-maven-plugin.version}</version>
</plugin>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<excludes>
<exclude>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
</exclude>
</excludes>
</configuration>
<executions>
<execution>
<goals>
<goal>repackage</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
In my specification, I use either
#Rule
public EmbeddedActiveMQResource server = new EmbeddedActiveMQResource();
or the artemis-server dependency to start a server.
I also tried to use sleep() and #Retry but tests seem to hang in the send() method even when they are retried. In the logs, I can see that the sent message is sometimes not acknowledged. If this is the case once, retry hangs in the same spot.
You can also see my attempt to get a "fresh" server by using : start(), setReceiveDelay() and stop() on the server in the setup() and cleanup() fixtures.
AuditMessageArtemisSpec.groovy
package be.fgov.minfin.esbsoa.ems.ccncsi.audit.service
import be.fgov.minfin.esbsoa.ems.ccncsi.audit.domain.AuditMessage
import be.fgov.minfin.esbsoa.ems.ccncsi.audit.domain.AuditMessageHistory
import be.fgov.minfin.esbsoa.ems.ccncsi.audit.repository.AuditMessageHistoryRepository
import be.fgov.minfin.esbsoa.ems.ccncsi.audit.repository.AuditMessageRepository
import be.fgov.minfin.esbsoa.ems.ccncsi.common.domain.Headers
import be.fgov.minfin.esbsoa.ems.ccncsi.common.service.JmsService
import com.fasterxml.jackson.databind.ObjectMapper
import groovy.util.logging.Slf4j
import org.apache.activemq.artemis.junit.EmbeddedActiveMQResource
import org.junit.Rule
import org.junit.runner.RunWith
import org.springframework.beans.factory.annotation.Autowired
import org.springframework.beans.factory.annotation.Value
import org.springframework.boot.test.context.SpringBootTest
import org.springframework.context.annotation.ComponentScan
import org.springframework.test.annotation.DirtiesContext
import org.springframework.test.context.ActiveProfiles
import org.springframework.test.context.junit4.SpringRunner
import spock.lang.Retry
import spock.lang.Specification
#RunWith(SpringRunner.class)
#SpringBootTest
#ActiveProfiles("artemis")
#ComponentScan("be.fgov.minfin.esbsoa.ems.ccncsi")
#Slf4j
class AuditMessageServiceArtemisSpec extends Specification{
static final SLEEP_TIME = 3000
#Rule
public EmbeddedActiveMQResource server = new EmbeddedActiveMQResource();
#Autowired
private JmsService jmsService;
#Autowired
private AuditMessageRepository repository
#Autowired
private AuditMessageHistoryRepository historyRepository
#Value("\${ccncsi.audit.topic.toEurope}")
private String europeDestination
#Value("\${ccncsi.audit.topic.commands}")
private String commandsDestination
private ObjectMapper jsonMapper = new ObjectMapper()
//def setup() {
// server.start()
// server.setDefaultReceiveTimeout(SLEEP_TIME)
//}
#Retry(delay = 10000)
def 'Send a valid message to Europe'() {
when:
def msg = send(europeDestination,"tests/message1.json")
then: "Verify if message is inserted in table AUDIT_MESSAGE"
sleep(SLEEP_TIME)
List<AuditMessage> messages = repository.findAll().collect()
messages.size() == 1
AuditMessage auditMessage = messages.get(0)
auditMessage.id != null
auditMessage.messageId == msg.headers.get(Headers.MESSAGE_ID)
auditMessage.reference == null
auditMessage.content != null
auditMessage.content.contentType == "text/plain"
new String(Base64.decoder.decode(auditMessage.content.content)) == "Here is the body of message1"
auditMessage.contentStored
auditMessage.messageTimestamp != null
auditMessage.creationDate != null
check(auditMessage, msg, Headers.MESSAGE_ID, Headers.APPLICATION, Headers.CONTENT_TYPE, Headers.COUNTRY_CODE, Headers.QUEUE_BASE_NAME, Headers.MESSAGE_TYPE)
and: "Check method findByMessageIdOrderByCreation"
List<AuditMessage> messagesByMessageId = repository.findByMessageIdOrderByCreationDate(auditMessage.getMessageId())
messagesByMessageId.size() == 1
messagesByMessageId.get(0).id == auditMessage.id
and: "Check if a status event is sent"
sleep(SLEEP_TIME)
List<AuditMessageHistory> histories = historyRepository.findAll().collect()
histories.size() == 1
}
#Retry(delay = 10000)
def 'Send a reply message from Europe - with correlationId'() {
when:
def msg = send(europeDestination,"tests/message2.json")
sleep(SLEEP_TIME)
then: "Verify if message is inserted in table AUDIT_MESSAGE"
List<AuditMessage> messagesByMessageId = repository.findByMessageIdOrderByCreationDate(msg.headers.get(Headers.MESSAGE_ID))
messagesByMessageId.size() == 1
messagesByMessageId.get(0).reference != null
}
#Retry(delay = 10000)
def 'Send message with body null'() {
when:
def msg = send(europeDestination,"tests/message5.json")
sleep(SLEEP_TIME)
then: "Verify if message is inserted in table AUDIT_MESSAGE and contentStored is false"
List<AuditMessage> messagesByMessageId = repository.findByMessageIdOrderByCreationDate(msg.headers.get(Headers.MESSAGE_ID))
messagesByMessageId.size() == 1
!messagesByMessageId.get(0).contentStored
}
#Retry(delay = 10000)
def 'Bad message3.json -> No inserts'() {
when:
def msg = send(europeDestination,"tests/message3.json")
sleep(SLEEP_TIME)
then: "Verify if message is not inserted in table AUDIT_MESSAGE"
List<AuditMessage> messagesByMessageId = repository.findByMessageIdOrderByCreationDate(msg.headers.get(Headers.MESSAGE_ID))
messagesByMessageId.size() == 0
List<AuditMessageHistory> messageHistories = repository.findByMessageIdOrderByCreationDate(msg.headers.get(Headers.MESSAGE_ID))
messageHistories.size() == 0
}
//def cleanup() {
// server.stop()
// sleep(SLEEP_TIME)
//}
def get(String path) {
URL url = this.getClass().getClassLoader().getResource(path)
return jsonMapper.readValue(url, AuditTestMessage.class)
}
def send(String destination, String path) {
log.error("sending message from file: {} to {}", path.toString(), destination)
AuditTestMessage msg = get(path)
jmsService.send(destination, msg.getHeaders(), msg.getBody())
return msg
}
def check(AuditMessage msg1, AuditTestMessage msg2, String... keys) {
for (String key : keys) {
if (msg2.getHeaders().get(key) != msg1.getHeaders().get(key)) {
log.error("Header ["+key+"] : "+ msg1.getHeaders().get(key) + " != " + msg2.getHeaders().get(key))
return false
}
}
return true
}
}
JMSConfig.java
Below you can see an attempt at setting subdomain to true on the JMSFactory as I'm sending the messages to a topic. This did not change any behavior.
package be.fgov.minfin.esbsoa.ems.ccncsi.audit.config;
import be.fgov.minfin.esbsoa.ems.ccncsi.common.service.JmsService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.autoconfigure.jms.DefaultJmsListenerContainerFactoryConfigurer;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.jms.annotation.EnableJms;
import org.springframework.jms.config.DefaultJmsListenerContainerFactory;
import org.springframework.jms.config.JmsListenerContainerFactory;
import org.springframework.jms.core.JmsMessagingTemplate;
import org.springframework.jms.support.destination.DynamicDestinationResolver;
import org.springframework.lang.Nullable;
import org.springframework.messaging.support.MessageBuilder;
import javax.jms.ConnectionFactory;
import javax.jms.Destination;
import javax.jms.JMSException;
import javax.jms.Session;
import java.util.Map;
#Configuration
#EnableJms
public class JMSConfig {
#Bean
public JmsListenerContainerFactory<?> myFactory(ConnectionFactory connectionFactory,
DefaultJmsListenerContainerFactoryConfigurer configurer) {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
// This provides all boot's default to this factory, including the message converter
configurer.configure(factory, connectionFactory);
//factory.setPubSubDomain(true);
// You could still override some of Boot's default if necessary.
return factory;
}
#Bean
JmsService jmsService(#Autowired JmsMessagingTemplate jmsMessagingTemplate) {
return new JmsService() {
#Override
public void send(String destination, Object payload) {
jmsMessagingTemplate.send(destination, MessageBuilder.withPayload(payload).build());
}
#Override
public void send(String destination, Map<String, Object> headers, Object payload) {
if (headers == null) send(destination, payload);
else jmsMessagingTemplate.send(destination, MessageBuilder.withPayload(payload).copyHeaders(headers).build());
}
};
}
}
MessageListener.java
package be.fgov.minfin.esbsoa.ems.ccncsi.audit.listener;
import be.fgov.minfin.esbsoa.ems.ccncsi.audit.domain.AuditMessage;
import be.fgov.minfin.esbsoa.ems.ccncsi.audit.service.AuditMessageHistoryService;
import be.fgov.minfin.esbsoa.ems.ccncsi.audit.service.AuditMessageMapper;
import be.fgov.minfin.esbsoa.ems.ccncsi.audit.service.AuditMessageService;
import be.fgov.minfin.esbsoa.ems.ccncsi.common.domain.DestinationType;
import be.fgov.minfin.esbsoa.ems.ccncsi.common.domain.command.Command;
import be.fgov.minfin.esbsoa.ems.ccncsi.common.domain.event.StatusEvent;
import be.fgov.minfin.esbsoa.ems.ccncsi.common.domain.event.StatusType;
import be.fgov.minfin.esbsoa.ems.ccncsi.common.service.JsonService;
import be.fgov.minfin.esbsoa.ems.ccncsi.common.service.StatusEventService;
import com.fasterxml.jackson.databind.JsonNode;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.jms.annotation.JmsListener;
import org.springframework.jms.core.JmsMessagingTemplate;
import org.springframework.jms.support.JmsHeaders;
import org.springframework.messaging.Message;
import org.springframework.stereotype.Component;
import javax.transaction.Transactional;
import java.io.IOException;
#Component
#Slf4j
public class MessageListener {
private final AuditMessageService service;
private final AuditMessageMapper mapper;
private final StatusEventService statusEventService;
private final JsonService jsonService;
private final AuditMessageHistoryService historyService;
private final JmsMessagingTemplate jmsMessagingTemplate;
#Value("${ccncsi.audit.queue.fromEuropeError:Consumer.ccncsiaudit.VirtualTopic.ccncsi.fromEurope.rme}")
private String fromEuropeError;
#Value("${ccncsi.audit.queue.toEuropeError:Consumer.ccncsiaudit.VirtualTopic.ccncsi.toEurope.rme}")
private String toEuropeError;
#Autowired
public MessageListener(AuditMessageService service, AuditMessageMapper mapper, StatusEventService statusEventService,
be.fgov.minfin.esbsoa.ems.ccncsi.common.service.JsonService jsonService, AuditMessageHistoryService historyService,
JmsMessagingTemplate jmsMessagingTemplate) {
this.mapper = mapper;
this.service = service;
this.statusEventService = statusEventService;
this.jsonService = jsonService;
this.historyService = historyService;
this.jmsMessagingTemplate = jmsMessagingTemplate;
}
#JmsListener(destination = "${ccncsi.audit.queue.toEurope:Consumer.ccncsiaudit.VirtualTopic.ccncsi.toEurope}", concurrency = "${ccncsi.audit.queue.concurrency:1-10}")
#Transactional(value = Transactional.TxType.REQUIRED)
public void onMessageToEurope(Message<?> message) {
save(message, DestinationType.TO_EUROPE);
}
#JmsListener(destination = "${ccncsi.audit.queue.fromEurope:Consumer.ccncsiaudit.VirtualTopic.ccncsi.fromEurope}", concurrency = "${ccncsi.audit.queue.concurrency:1-10}")
#Transactional(value = Transactional.TxType.REQUIRED)
public void onMessageFromEurope(Message<?> message) {
save(message, DestinationType.FROM_EUROPE);
}
private void save(Message<?> message, DestinationType destination) {
AuditMessage auditMessage = service.save(mapper.map(message, destination));
statusEventService.notify(message.getHeaders(), auditMessage.getId(), StatusType.AUDIT_STORE_SUCCESS);
}
#JmsListener(destination = "${ccncsi.audit.queue.events:Consumer.ccncsiaudit.VirtualTopic.ccncsi.events}", concurrency = "${ccncsi.audit.queue.concurrency:1-10}")
#Transactional(value = Transactional.TxType.REQUIRED)
public void onEventMessage(Message<?> event) throws IOException {
StatusEvent statusEvent = jsonService.fromString((String) event.getPayload(), StatusEvent.class);
historyService.save(statusEvent);
}
#JmsListener(destination = "${ccncsi.audit.queue.commands:Consumer.ccncsiaudit.VirtualTopic.ccncsi.commands}", concurrency = "${ccncsi.audit.queue.concurrency:1-10}")
#Transactional(value = Transactional.TxType.REQUIRED)
public void onCommandMessage(Message<?> command) throws Exception {
JsonNode jsonCommand = jsonService.fromString((String) command.getPayload());
service.execute((Command) jsonService.fromJSonNode(jsonCommand, Class.forName(jsonCommand.get("name").textValue())));
}
#JmsListener(destination = "DLQ.${ccncsi.audit.queue.toEurope:Consumer.ccncsiaudit.VirtualTopic.ccncsi.toEurope}")
#Transactional(value = Transactional.TxType.REQUIRED)
public void onErrorToEurope(Message<?> message) {
statusEventService.notify(message.getHeaders(), message.getHeaders().get(JmsHeaders.MESSAGE_ID), StatusType.AUDIT_STORE_FAILED);
jmsMessagingTemplate.send(toEuropeError, message);
}
#JmsListener(destination = "DLQ.${ccncsi.audit.queue.fromEurope:Consumer.ccncsiaudit.VirtualTopic.ccncsi.fromEurope}")
#Transactional(value = Transactional.TxType.REQUIRED)
public void onErrorFromEurope(Message<?> message) {
statusEventService.notify(message.getHeaders(), message.getHeaders().get(JmsHeaders.MESSAGE_ID), StatusType.AUDIT_STORE_FAILED);
jmsMessagingTemplate.send(fromEuropeError, message);
}
}
Below you can see that I modified the addresses to the artemis format. The specification sends messages to VirtualTopic.ccncsi.toEurope (topic) and are received on the consumer queues of this address.
application.yml (relevant section)
ccncsi:
common:
statusEventDestination: VirtualTopic.ccncsi.events
audit:
#Maximum size of content (in Kilobytes)
max-size-content-storage: 5000
topic:
toEurope: VirtualTopic.ccncsi.toEurope
fromEurope: VirtualTopic.ccncsi.fromEurope
commands: VirtualTopic.ccncsi.commands
queue:
toEurope: VirtualTopic.ccncsi.toEurope::Consumer.ccncsiaudit.VirtualTopic.ccncsi.toEurope
fromEurope: VirtualTopic.ccncsi.fromEurope::Consumer.ccncsiaudit.VirtualTopic.ccncsi.fromEurope
events: VirtualTopic.ccncsi.events::Consumer.ccncsiaudit.VirtualTopic.ccncsi.events
commands: VirtualTopic.ccncsi.commands::Consumer.ccncsiaudit.VirtualTopic.ccncsi.commands
concurrency: 1-30
application-junit.yml
spring:
jpa:
show-sql: true
hibernate:
ddl-auto: create
datasource:
url: jdbc:h2:mem:testdb;NON_KEYWORDS=KEY,VALUE;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE
username: sa
password: password
driverClassName: org.h2.Driver
jms:
pub-sub-domain: true
template:
default-destination: VirtualTopic.ccncsi.toEurope
# artemis:
# embedded:
# enabled: true
# persistent: true
# mode: embedded
# broker-url: vm://0?virtualTopicConsumerWildcards=Consumer.*.%3E%3B2
Can anyone tell me what I am doing wrong ?
I am getting No current JDBC Connection found. exception while using micronaut-data, i am referring to https://github.com/tdisanti/micronaut-single-ucp
micronaut version
<micronaut.version>2.0.0</micronaut.version>
<micronaut.data.version>1.1.1</micronaut.data.version>
logs:
2020-07-02 19:03:30.716 [nioEventLoopGroup-1-3] ERROR i.m.h.s.netty.RoutingInBoundHandler:1839- Unexpected error occurred: No current JDBC Connection found. Consider wrapping this call in transactional boundaries.
io.micronaut.transaction.jdbc.exceptions.CannotGetJdbcConnectionException: No current JDBC Connection found. Consider wrapping this call in transactional boundaries.
at io.micronaut.transaction.jdbc.DataSourceUtils.doGetConnection(DataSourceUtils.java:135)
at io.micronaut.transaction.jdbc.DataSourceUtils.getConnection(DataSourceUtils.java:93)
at io.micronaut.transaction.jdbc.DataSourceTransactionManager.getConnection(DataSourceTransactionManager.java:342)
at io.micronaut.transaction.jdbc.DataSourceTransactionManager.getConnection(DataSourceTransactionManager.java:88)
at io.micronaut.data.jdbc.operations.DefaultJdbcRepositoryOperations.prepareStatement(DefaultJdbcRepositoryOperations.java:1185)
java
#JdbcRepository(dialect = Dialect.ORACLE)
public class BooksDao {
private final JdbcOperations jdbcOperations;
#Inject
public BooksDao (JdbcOperations jdbcOperations) {
this.jdbcOperations = jdbcOperations;
}
#Transactional
public List<Book> get(){
...
}
}
yml properties
datasources:
default:
validationQuery: SELECT 1 FROM DUAL
driverClass: oracle.jdbc.driver.OracleDriver
url: jdbc:oracle:thin:<url>,cn=OracleContext,dc=uk,dc=csfb,dc=com
autoCommit: true
connectionTestQuery: SELECT 1 FROM DUAL
leakDetectionThreshold: 180000
poolName: hikariConnectionPoolName
maximumPoolSize: 10
minimumIdle: 5
data-source-properties:
truststore: '${client.truststore.path}'
truststore.password: '${client.truststore.password}'
truststore.type: JKS
keystore: '${client.keystore.path}'
keystore.password: '${client.keystore.password}'
keystore.type: PKCS12
dialect: ORACLE
DatasourceConfiguration.java
#Singleton
public class DatasourceConfiguration implements BeanCreatedEventListener<DatasourceConfiguration> {
#Override
public DatasourceConfiguration onCreated(BeanCreatedEvent<DatasourceConfiguration> event) {
LOGGER.info("Overriding DatasourceConfiguration Properties...");
DatasourceConfiguration bean = event.getBean();
Properties dataSourceProperties = bean.getDataSourceProperties();
Map<String, String> props = new HashMap<>();
props.put("javax.net.ssl.keyStore",dataSourceProperties.getProperty("keystore"));
props.put("javax.net.ssl.keyStorePassword",dataSourceProperties.getProperty("keystore.password"));
props.put("javax.net.ssl.keyStoreType",dataSourceProperties.getProperty("keystore.type"));
props.put("javax.net.ssl.trustStore",dataSourceProperties.getProperty("truststore"));
props.put("javax.net.ssl.trustStorePassword",dataSourceProperties.getProperty("truststore.password"));
props.put("javax.net.ssl.trustStoreType",dataSourceProperties.getProperty("truststore.type"));
props.put("oracle.net.authentication_services","(TCPS)");
props.put("oracle.net.ssl_client_authentication","TRUE");
props.put("ssl.keyManagerFactory.algorithm","SunX509");
props.put("oracle.net.ssl_cipher_suites","(TLS_RSA_WITH_AES_256_CBC_SHA)");
props.put("oracle.net.encryption_client","REJECTED");
props.put("oracle.net.crypto_checksum_client","REJECTED");
bean.setDataSourceProperties(props);
return bean;
}
}
The issue get resolved by adding micronaut-data-processor in annotationProcessorPaths of maven-compiler-plugin.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>${maven-compiler-plugin.version}</version>
<configuration>
<release>${jdk.version}</release>
<encoding>UTF-8</encoding>
<compilerArgs>
<arg>-parameters</arg>
</compilerArgs>
<annotationProcessorPaths>
<path>
<groupId>io.micronaut</groupId>
<artifactId>micronaut-inject-java</artifactId>
<version>${micronaut.version}</version>
</path>
<path>
<groupId>io.micronaut</groupId>
<artifactId>micronaut-validation</artifactId>
<version>${micronaut.version}</version>
</path>
<path>
<groupId>io.micronaut.configuration</groupId>
<artifactId>micronaut-openapi</artifactId>
<version>${micronaut.openapi.version}</version>
</path>
<path>
<groupId>io.micronaut.data</groupId>
<artifactId>micronaut-data-processor</artifactId>
<version>1.0.2</version>
</path>
</annotationProcessorPaths>
</configuration>
<executions>
<execution>
<id>test-compile</id>
<goals>
<goal>testCompile</goal>
</goals>
<configuration>
<compilerArgs>
<arg>-parameters</arg>
</compilerArgs>
<annotationProcessorPaths>
<path>
<groupId>io.micronaut</groupId>
<artifactId>micronaut-inject-java</artifactId>
<version>${micronaut.version}</version>
</path>
<path>
<groupId>io.micronaut</groupId>
<artifactId>micronaut-validation</artifactId>
<version>${micronaut.version}</version>
</path>
<path>
<groupId>io.micronaut.configuration</groupId>
<artifactId>micronaut-openapi</artifactId>
<version>${micronaut.openapi.version}</version>
</path>
<path>
<groupId>io.micronaut.data</groupId>
<artifactId>micronaut-data-processor</artifactId>
<version>1.0.2</version>
</path>
</annotationProcessorPaths>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
My Camel route app detects other app's published ampq message(publishes numbers), but, fails to handle, with "no type converter available" error. How can I resolve?.
"org.apache.camel.NoTypeConversionAvailableException: No type converter available to convert from type: java.lang.Integer to the required type: java.io.InputStream with value 79"
routebuilder class
package aaa.bbb.ccc.qscx;
import java.io.IOException;
import javax.ejb.Startup;
import javax.enterprise.context.ApplicationScoped;
import javax.inject.Inject;
import org.apache.camel.CamelContext;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.component.reactive.streams.api.CamelReactiveStreamsService;
import org.eclipse.microprofile.reactive.messaging.Incoming;
import org.reactivestreams.Subscriber;
#Startup
#ApplicationScoped
public class TheRoutes extends RouteBuilder {
#Inject
TheProcessor theProcessor;
#Inject
CamelContext ctx;
#Inject
CamelReactiveStreamsService crss;
#Override
public void configure() throws IOException, InterruptedException {
from("reactive-streams:in")
.process(theProcessor)
.log(".........from reactive-streams:in - body: ${body}");
}
#Incoming("prices")
public Subscriber<String> sink() {
return crss.subscriber("file:./target?fileName=values.txt&fileExist=append", String.class);
}
}
pom.xml
<?xml version="1.0"?>
<project xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<modelVersion>4.0.0</modelVersion>
<groupId>aaa.bbb.ccc </groupId>
<artifactId>qscx</artifactId>
<version>1.0</version>
<properties>
<compiler-plugin.version>3.8.1</compiler-plugin.version>
<maven.compiler.parameters>true</maven.compiler.parameters>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<quarkus-plugin.version>1.0.0.CR2</quarkus-plugin.version>
<quarkus.platform.artifact-id>quarkus-universe-bom</quarkus.platform.artifact-id>
<quarkus.platform.group-id>io.quarkus</quarkus.platform.group-id>
<quarkus.platform.version>1.0.0.CR2</quarkus.platform.version>
<surefire-plugin.version>2.22.1</surefire-plugin.version>
</properties>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>${quarkus.platform.group-id}</groupId>
<artifactId>${quarkus.platform.artifact-id}</artifactId>
<version>${quarkus.platform.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-junit5</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-resteasy</artifactId>
</dependency>
<dependency>
<groupId>io.rest-assured</groupId>
<artifactId>rest-assured</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-smallrye-reactive-messaging-amqp</artifactId>
</dependency>
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-smallrye-reactive-messaging</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel.quarkus</groupId>
<artifactId>camel-quarkus-core</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel.quarkus</groupId>
<artifactId>camel-quarkus-timer</artifactId>
</dependency>
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-artemis-jms</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel.quarkus</groupId>
<artifactId>camel-quarkus-bean</artifactId>
</dependency>
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-smallrye-reactive-streams-operators</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel.quarkus</groupId>
<artifactId>camel-quarkus-support-common</artifactId>
</dependency>
<dependency>
<groupId>io.smallrye.reactive</groupId>
<artifactId>smallrye-reactive-messaging-camel</artifactId>
<version>1.0.8</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<version>1.7.26</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-maven-plugin</artifactId>
<version>${quarkus-plugin.version}</version>
<executions>
<execution>
<goals>
<goal>build</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>${compiler-plugin.version}</version>
</plugin>
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>${surefire-plugin.version}</version>
<configuration>
<systemProperties>
<java.util.logging.manager>org.jboss.logmanager.LogManager</java.util.logging.manager>
</systemProperties>
</configuration>
</plugin>
</plugins>
</build>
<profiles>
<profile>
<id>native</id>
<activation>
<property>
<name>native</name>
</property>
</activation>
<build>
<plugins>
<plugin>
<artifactId>maven-failsafe-plugin</artifactId>
<version>${surefire-plugin.version}</version>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
<configuration>
<systemProperties>
<native.image.path>${project.build.directory}/${project.build.finalName}-runner</native.image.path>
</systemProperties>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<properties>
<quarkus.package.type>native</quarkus.package.type>
</properties>
</profile>
</profiles>
<name>qscx</name>
</project>
application.properties
amqp-username=quarkus
amqp-password=quarkus
mp.messaging.incoming.prices.address=prices
mp.messaging.incoming.prices.connector=smallrye-amqp
mp.messaging.incoming.prices.host=localhost
mp.messaging.incoming.prices.port=5672
mp.messaging.incoming.prices.username=quarkus
mp.messaging.incoming.prices.password=quarkus
mp.messaging.incoming.prices.broadcast=true
mp.messaging.incoming.prices.containerId=my-container-id
console stacktrace (excerpt)
2019-11-22 22:31:22,930 WARN [org.apa.cam.com.rea.str.ReactiveStreamsConsumer] (Camel (camel-1) thread #1 - reactive-streams://DCE85ACAC992C3A-0000000000000000) Error processing exchange. Exchange[DCE85ACAC992C3A-000000000000005F]. Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot store file: .\target\values.txt]: org.apache.camel.component.file.GenericFileOperationFailedException: Cannot store file: .\target\values.txt
at org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:376)
at org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:300)
at org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:164)
at org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:75)
at org.apache.camel.support.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:67)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:134)
at org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$RedeliveryState.run(RedeliveryErrorHandler.java:476)
at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:185)
at org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:59)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:87)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:228)
at org.apache.camel.component.reactive.streams.ReactiveStreamsConsumer.lambda$doSend$3(ReactiveStreamsConsumer.java:96)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.camel.InvalidPayloadException: No body available of type: java.io.InputStream but has value: 79 of type: java.lang.Integer on: Message[]. Caused by: No type converter available to convert from type: java.lang.Integer to the required type: java.io.InputStream with value 79. Exchange[DCE85ACAC992C3A-000000000000005F]. Caused by: [org.apache.camel.NoTypeConversionAvailableException - No type converter available to convert from type: java.lang.Integer to the required type: java.io.InputStream with value 79]
at org.apache.camel.support.MessageSupport.getMandatoryBody(MessageSupport.java:115)
at org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:355)
... 14 more
Caused by: org.apache.camel.NoTypeConversionAvailableException: No type converter available to convert from type: java.lang.Integer to the required type: java.io.InputStream with value 79
at org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:139)
at org.apache.camel.support.MessageSupport.getMandatoryBody(MessageSupport.java:113)
... 15 more
other notes
publishing app is based upon the Quark Ampq example:
https://github.com/quarkusio/quarkus-quickstarts/tree/master/amqp-quickstart/src/main/java/org/acme/quarkus/sample
technologies
java 8
quarkus
smallrye
camel
maven
It seems the subscriber does not currently execute the type conversion. It may be solved in a future release.
In the meantime, you need to enforce it in the route:
#Override
public void configure() throws IOException, InterruptedException {
from("direct:proc")
.convertBodyTo(String.class)
.to("file:./target?fileName=values.txt&fileExist=append");
}
#Incoming("prices")
public Subscriber<String> sink() {
return crss.subscriber("direct:proc", String.class);
}
I created two Spring Boot applications which both will be deployed in a Kubernetes cluster. One of those apps will act as a gateway and therefore uses Spring Cloud Gateway as a dependency. Also I want to integrate service discovery with Spring Cloud Kubernetes and that the gateway uses the service discovery to automatically generate corresponding routes. But when I expose the gateway application, which is running in an local Minikube cluster, and invoke the second app/service I get a 503 error with following message: Unable to find instance for ...-service
Currently I have installed following:
Minikube
VirtualBox
Docker Toolbox
I have created a Gradle project with two subprojects (gateway and another service). All projects will be build/deployed locally. The default Service Account has permission to read the Kubernetes API. After deployment of those services I expose the gateway service externally. In the gateway service I have some endpoints implemented, which
provide a list of all services in the cluster vie the DiscoveryClient.
on application layer invoke the other service based on the URI provided by the DiscoveryClient.
Everything seems to work but when I invoke the other service with URI/serviceId I get the 503 error...
Following Spring Cloud versions are used:
spring-cloud-starter-kubernetes 1.0.1.RELEASE
spring-cloud-starter-gateway 2.1.1.RELEASE
My demo app is available at https://github.com/nmaoez/spring-cloud-gateway-kubernetes and the README.md provides steps to get both services deployed in a local Minikube cluster. Also all available endpoints are shown. But the interessing part are the application.yaml and the application class of the gateway.
application.yaml:
spring:
application:
name: gateway-service
cloud:
gateway:
discovery:
locator:
enabled: true
lower-case-service-id: true
management:
endpoints:
web:
exposure:
include: '*'
GatewayApplication.java
#SpringBootApplication
#EnableDiscoveryClient
#RestController
public class GatewayApplication {
#Autowired
RestTemplate restTemplate;
#Autowired
private DiscoveryClient discoveryClient;
#GetMapping("/")
#ResponseBody
public String hello() {
return "GatewayApplication says hello!";
}
#GetMapping("/test")
#ResponseBody
public String invokeTestService() {
List<ServiceInstance> testServiceInstances = this.discoveryClient.getInstances("test-service");
return restTemplate.getForObject(testServiceInstances.get(0).getUri(), String.class);
}
#GetMapping("/services")
public List<String> services() {
return this.discoveryClient.getServices();
}
#GetMapping("/services/{serviceId}")
public List<ServiceInstance> servicesById(#PathVariable("serviceId") String serviceId) {
return this.discoveryClient.getInstances(serviceId);
}
#Bean
public RestTemplate restTemplate() {
return new RestTemplate();
}
public static void main(String[] args) {
SpringApplication.run(GatewayApplication.class, args);
}
}
I got it some how running after I overwrote the url-expression field in the gateway-service/application.yaml to
url-expression: "uri+'/'"
After that I got a correct response after I invoked gateway-uri/another-service/. But my wish is to not explicitly replace the default of lb://serviceid. How can I do that?
I expect that if I invoke another service in the cluster though the gateway, I get a 200 response and the correct answer based in the rest controller of the application.
You have to add the dependency to spring-cloud-starter-kubernetes-ribbon as well.
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-kubernetes-ribbon</artifactId>
<version>1.0.1.RELEASE</version>
</dependency>
Then it will work without any rewrites, just with the spring.cloud.gateway.discovery.locator.enabled: true
I just implemented an example application using Spring Cloud Gateway and Kubernetes, that works like a charm in Docker Desktop. And no extra nor funny configurations were needed.
If it may help this was my build.gradle:
plugins {
id 'org.springframework.boot' version '2.4.2'
id 'io.spring.dependency-management' version '1.0.11.RELEASE'
id 'java'
}
group = 'com.example'
version = '0.1.0-SNAPSHOT'
repositories {
mavenCentral()
maven { url 'https://repo.spring.io/milestone' }
}
ext {
set('springCloudVersion', "2020.0.0")
set('springCloudKubernetesVersion', '1.1.7.RELEASE')
set('springCloudVersion', '2020.0.0')
}
dependencies {
implementation 'org.springframework.boot:spring-boot-starter-actuator'
implementation 'org.springframework.cloud:spring-cloud-starter-gateway'
implementation 'org.springframework.cloud:spring-cloud-starter-sleuth'
testImplementation 'org.springframework.boot:spring-boot-starter-test'
implementation "org.springframework.cloud:spring-cloud-starter-kubernetes:$springCloudKubernetesVersion"
implementation "org.springframework.cloud:spring-cloud-starter-kubernetes-config:$springCloudKubernetesVersion"
implementation "org.springframework.cloud:spring-cloud-starter-kubernetes-ribbon:$springCloudKubernetesVersion"
implementation "org.springframework.cloud:spring-cloud-starter-kubernetes-loadbalancer:$springCloudKubernetesVersion"
}
dependencyManagement {
imports {
mavenBom "org.springframework.cloud:spring-cloud-dependencies:$springCloudVersion"
}
}
test {
useJUnitPlatform()
}
This is the configuration from the application.yaml:
spring:
cloud:
gateway:
discovery:
locator:
enabled: true
And finally, the DiscoveryClient is enabled in the app:
#SpringBootApplication
#EnableDiscoveryClient // So services can be discovered
public class GatewayApplication {
public static void main(String[] args) {
SpringApplication.run(GatewayApplication.class, args);
}
}
Note as Jakub Kubrynski said, you must include the Ribbon dependency.
Also beware routing only works when the gateway is deployed to the K8s cluster. Outside it, it can see the K8s services but cannot route to them since they are using IP addresses in the K8s network.
I'm able to setup spring cloud gateway with discovery of spring-cloud-kubernetes dependency version 1.1.10.RELASE and Spring-boot: 2.5.7 Spring cloud gateway: 3.0.4
Pom file looks like below:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.5.7</version>
<relativePath/>
</parent>
<groupId>com.csg.cro.rccs</groupId>
<artifactId>api-gateway</artifactId>
<version>${revision}</version>
<name>RCC-APIGateway</name>
<description>Enable Proxy and verify user token project for Risk 360</description>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<java.version>11</java.version>
<spring-cloud.version>2020.0.3</spring-cloud.version>
<revision>21.7.0-SNAPSHOT</revision>
</properties>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>${spring-cloud.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-gateway</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-kubernetes</artifactId>
<version>1.1.10.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-kubernetes-config</artifactId>
<version>1.1.10.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-kubernetes-ribbon</artifactId>
<version>1.1.10.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-kubernetes-loadbalancer</artifactId>
<version>1.1.10.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-webflux</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-security</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>flatten-maven-plugin</artifactId>
<version>1.1.0</version>
<configuration>
<updatePomFile>true</updatePomFile>
<flattenMode>resolveCiFriendliesOnly</flattenMode>
</configuration>
<executions>
<execution>
<id>flatten</id>
<phase>process-resources</phase>
<goals>
<goal>flatten</goal>
</goals>
</execution>
<execution>
<id>flatten.clean</id>
<phase>clean</phase>
<goals>
<goal>clean</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
My discovery config look like:
spring:
application.name: gateway
cloud:
gateway:
discovery:
locator:
enabled: true
url-expression: "'http://'+serviceId+':'+getPort()"
lower-case-service-id: true
I am creating a Spring Boot application, which will read configuration like DB properties from Consul. But I am not able to read the key value from Consul using my application. Following is, what I am trying to do.
**pom.xml**
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.tuturself</groupId>
<artifactId>spring-boot-consul</artifactId>
<version>1.0-SNAPSHOT</version>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>1.5.4.RELEASE</version>
<relativePath/>
</parent>
<properties>
<java.version>1.8</java.version>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<spring.retry.version>1.2.1.RELEASE</spring.retry.version>
<consul.version>1.1.2.RELEASE</consul.version>
<consul.discovery.version>1.1.2.RELEASE</consul.discovery.version>
<jackson.version>2.8.1</jackson.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-consul-all</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-consul-discovery</artifactId>
<version>${consul.discovery.version}</version>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-config</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-aop</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.retry</groupId>
<artifactId>spring-retry</artifactId>
<version>${spring.retry.version}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>${jackson.version}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<version>${jackson.version}</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.5.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>repackage</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-consul-dependencies</artifactId>
<version>1.2.1.RELEASE</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
</project>
And Following is my Main class:
#EnableRetry
#RefreshScope
#EnableDiscoveryClient
#SpringBootApplication
#ComponentScan("com.test.*")
public class SpringBootConsulApplication {
private static ConsulConfiguration consulConfiguration;
public static void main(String[] args) {
try {
String consulHost = System.getProperty("spring.cloud.consul.host");
System.out.println("consulHost ::" + consulHost);
String consulPort = System.getProperty("spring.cloud.consul.port");
System.out.println("consulPort ::" + consulPort);
String consulPrefix = System.getProperty("spring.cloud.consul.config.prefix");
System.out.println("consulPrefix ::" + consulPrefix);
new SpringApplicationBuilder(SpringBootConsulApplication.class).web(true).run(args);
} catch (Exception ex) {
ex.printStackTrace();
}
}
}
And I am reading the consul properties using the #Value annotation:
#Configuration
#EnableConfigurationProperties(PropertySourceBootstrapProperties.class)
public class ConsulConfiguration {
#Value("${cassandra.host}")
private String cassandraHost;
#Value("${cassandra.user}")
private String userName;
#Value("${cassandra.password}")
private String password;
}
I have my bootstrap.yml in resources folder:
spring:
cloud:
consul:
host: localhost
port: 8500
enabled: true
config:
enabled: true
prefix: config/application
defaultContext: apps
profileSeparator: '::'
application:
name: spring-boot-consul
Consul is up and running in my local system on localhost:8500 where I have the file config/application/spring-boot-consul.yml file;
spring:
application:
name: spring-boot-consul
cassandra:
host: 127.0.0.1:9042,127.0.0.2:9042
user: my_user
password: my_pass
pooling:
maxThread: 10
timeout: 50
keyspace:
name: test_keyspace
readConsistency: ONE
writeConsistency: ONE
When I am strating the application, it is showing not able to bind cassandra.host in my ConsulConfiguration class. Thus stopping the application. Any hints , What I am doing wrong here?
You can find a working example here.
Consul Configuration KV Store
You need to store your properties in the Consul KV store either from Consul UI or from the command line. The Consul Agent will not load your properties from the file system. To load the properties from the command line, you can use the following command once the Consul Agent is up and running. The YAML data can be read from a file by prefixing the file name with the # symbol.
./consul kv put config/application/data #spring-boot-consul.yml
where config/application/data is the key name.
If the data is successfully written in the KV, you should get the following response,
Success! Data written to: config/application/data
You can also fetch the properties from the KV by using the following command,
$ ./consul kv get config/application/data
cassandra:
host: 127.0.0.1:9042,127.0.0.2:9042
user: my_user
password: my_pass
You can also view the properties from the Consul Web UI,
Changes to bootstrap.yml
You need to modify your bootstrap.yml slightly. Here are the changes:
prefix value to config
defaultContext value to application
Added format to yaml
Added data-key by the name of data to fetch the YAML blob.
spring:
profiles: default
cloud:
consul:
host: localhost
port: 8500
config:
enabled: true
prefix: config
defaultContext: application
data-key: data
profileSeparator: '::'
format: yaml
application:
name: spring-boot-consul
Changes to ConsulConfiguration
#Configuration
#RefreshScope
public class ConsulConfiguration {
#Value("${cassandra.host}")
private String cassandraHost;
#Value("${cassandra.user}")
private String userName;
#Value("${cassandra.password}")
private String password;
#PostConstruct
public void postConstruct() {
// to validate if properties are loaded
System.out.println("** cassandra.host: " + cassandraHost);
System.out.println("** cassandra.user: " + userName);
System.out.println("** cassandra.password: " + password);
}
}
Changes to Application class,
#EnableRetry
#RefreshScope
#EnableDiscoveryClient
#EnableAutoConfiguration
#EnableConfigurationProperties
#SpringBootApplication
#ComponentScan("com.test.*")
public class SpringBootConsulApplication {
public static void main(String[] args) {
...
}
}