I'm using spring cloud stream versione 3.1.4, and this is my producer
#Component
public class Producer {
#Autowired
private StreamBridge streamBridge;
public void produce(int messageId, Object message) {
Message<Object> msg= MessageBuilder
.withPayload(message)
.setHeader("partitionKey", messageId)
.build();
streamBridge.send("outputchannel-out-0", msg);
}
#ServiceActivator(inputChannel = "errorchannel.errors")
public void errorHandler(ErrorMessage em) {
log.info("Error: {}", em);
}
}
Into application.yaml I set errorChannelEnabled
spring:
cloud:
stream:
bindings:
#Channel name
outputchannel-out-0:
destination: my-topic
contentType: application/json
producer:
partitionKeyExpression: headers['partitionKey']
partitionCount: 1
errorChannelEnabled: true
Now, If I change the produce() method in this way, in order to test the error handler
public void produce(int messageId, Object message) {
throw new RuntimeException("Producer error");
}
Nothing happens.
Error handler is not triggered.
I'm not sure that it's the right way to setup the error handler in spring cloud stream 3.1.4.
Can you help me?
errorchannel.errors does not exist.
There are two error channels errorChannel is the global error channel; the binding-specific error channel is named <destination>.<group>.errors. You don't currently have a group.
Related
I have one topic and Producer/Consumer:
Dependencies (Spring Initializr)
Producer (apache kafka)
Consumer (apache kafka stream, cloud stream)
Producer:
KafkaProducerConfig
#Configuration
public class KafkaProducerConfig {
#Bean
public KafkaTemplate<String, Person> kafkaTemplate(){
return new KafkaTemplate<>(producerFactory());
}
#Bean
public ProducerFactory<String, Person> producerFactory(){
Map<String, Object> configs = new HashMap<>();
configs.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "127.0.0.1:9092");
configs.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
configs.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class);
return new DefaultKafkaProducerFactory<>(configs);
}
}
Controller:
#RestController
public class KafkaProducerApplication {
private KafkaTemplate<String, Person> kafkaTemplate;
public KafkaProducerApplication(KafkaTemplate<String, Person> kafkaTemplate) {
this.kafkaTemplate = kafkaTemplate;
}
#GetMapping("/persons")
public Mono<List<Person>> findAll(){
var personList = Mono.just(Arrays.asList(new Person("Name1", 15),
new Person("Name2", 10)));
personList.subscribe(dataList -> kafkaTemplate.send("topic_test_spring", dataList.get(0)));
return personList;
}
}
It works correctly when accessing the endpoint and does not throw any exception in the IntelliJ console.
Consumer:
spring:
cloud:
stream:
function:
definition: personService
bindings:
personService-in-0:
destination: topic_test_spring
kafka:
bindings:
personService-in-0:
consumer:
configuration:
value:
deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
binders:
brokers:
- localhost:9091
- localhost:9092
kafka:
consumer:
properties:
spring:
json:
trusted:
packages: "*"
PersonKafkaConsumer
#Configuration
public class PersonKafkaConsumer {
#Bean
public Consumer<KStream<String, Person>> personService(){
return kstream -> kstream.foreach((key, person) -> {
System.out.println(person.getName());
});
}
}
Here I get the exception when run the project.
org.apache.kafka.streams.errors.StreamsException: Deserialization exception handler is set to fail upon a deserialization error. If you would rather have the streaming pipeline continue after a deserialization error, please set the default.deserialization.exception.handler appropriately. Caused by: java.lang.IllegalArgumentException: The class 'com.example.producer.model.Person' is not in the trusted packages: [java.util, java.lang, com.nttdata.bootcamp.yanki.model, com.nttdata.bootcamp.yanki.model.]. If you believe this class is safe to deserialize, please provide its name. If the serialization is only done by a trusted source, you can also enable trust all (). org.apache.kafka.streams.errors.StreamsException: Deserialization exception handler is set to fail upon a deserialization error. If you would rather have the streaming pipeline continue after a deserialization error, please set the default.deserialization.exception.handler appropriately.
The package indicated in the exception refers to the entity's package but in the producer. The producer's properties file has no configuration.
I'm very new with Kafka.
Using spring-boot kafka, I developed a publisher and a consumer using one Message object and manual ack. My code uses spring annotation. That's works perfectly.
Now, when I connect to production brokers, this one not send one Message but a list of message.
My listener method has the following signature:
#KafkaListener (topics="MessagesTopic", containerFactory="messageContainerfactory")
public void listen(#Payload Message message, Acknowledgment ack)
so I can acknowledge each Message. Good.
But now it's seems I must replace it with
#KafkaListener (topics="MessagesTopic", containerFactory="messageContainerfactory")
public void listen(#Payload List<Message> messages, Acknowledgment ack)
Even following the documentation it seems that I should use
#KafkaListener (topics="MessagesTopic", containerFactory="messageContainerfactory")
public void listen(#Payload List<Message> messages, Acknowledgment ack, Consumer<?,?> consumer)
Should I set batchmode to true ?
Now the question is : how can I acknowledge each message when this one has been completely handled?
Many many thanks for your help
Something like this one can help you either if you do want to manually commit offset.
If you do not want it then switch setAckMode to other value.
Here's this thing done the spring-way.
CoreAutoConfiguration class:
#Configuration
#Import({KafkaAutoConfiguration.class})
public class CoreAutoConfiguration {
#Bean("batchKafkaListenerContainerFactory")
public ConcurrentKafkaListenerContainerFactory<?, ?> batchKafkaListenerContainerFactory(ConcurrentKafkaListenerContainerFactoryConfigurer configurer, ConsumerFactory<Object, Object> kafkaConsumerFactory) {
ConcurrentKafkaListenerContainerFactory<Object, Object> factory = new ConcurrentKafkaListenerContainerFactory<>();
configurer.configure(factory, kafkaConsumerFactory);
factory.getContainerProperties().setAckMode(ContainerProperties.AckMode.MANUAL_IMMEDIATE);
factory.setBatchListener(true);
return factory;
}
}
Then there goes your Config class:
#Configuration
#Import({
CoreAutoConfiguration.class,
KafkaAutoConfiguration.class,
})
#EnableKafka
#EnableRetry
public class Config {
}
Finally the consumer:
#KafkaListener(
topics = "MessagesTopic",
containerFactory = "batchKafkaListenerContainerFactory"
)
public void dataReceived(#Payload List<String> payload) throws RuntimeException {
yourService.processIncomingData(payload);
}
And lastly, the properties:
spring.kafka.bootstrap-servers=localhost:9092
spring.kafka.consumer.group-id=helloworld
spring.kafka.listener.type=batch
spring.kafka.consumer.enable-auto-commit=false
# this is size of incoming list if broker has this many entries, can be lower eventually
spring.kafka.consumer.max-poll-records=100
spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.consumer.value-deserializer=org.apache.kafka.common.serialization.StringDeserializer
#Configuration
#EnableRabbit
public class RabbitConfiguration {
private static final String queueName = "3055";
private static final String topicExchangeName = queueName + "-exchange";
#Bean
Queue queue() {
return new Queue(queueName, false);
}
#Bean
TopicExchange exchange() {
return new TopicExchange(topicExchangeName);
}
#Bean
Binding binding(Queue queue, TopicExchange exchange) {
return BindingBuilder.bind(queue).to(exchange).with("foo.bar.#");
}
#Bean
RabbitTemplate rabbitTemplate(ConnectionFactory connectionFactory,
MessageConverter messageConverter) {
RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory);
rabbitTemplate.setMessageConverter(messageConverter);
return rabbitTemplate;
}
#Bean
MessageConverter messageConverter() {
return new Jackson2JsonMessageConverter();
}
}
Code above is my Spring Boot Project's RabbitMQ configuration class.
However, I cannot connect the RMQ server since below error pops up every time I try to connect.
Caused by: com.rabbitmq.client.AuthenticationFailureException: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.
The Server provider told me that I need to set the authentication mechanism to AMQPLAIN.
My question is that How can I set authentication mechanism to AMQPLAIN?
No matter how much I google, I couldn't figure out how.
I confirm to #Raja Anbazhagan. Check the RabbitMQ logs first. Supposedly your user credentials were guest/guest.
The easiest way to solve your problem could be to add those lines in your application.yml:
spring:
rabbitmq:
username: <user-name>
password: <user-password>
I am trying to use Spring integration to connect to JMS client , but i am getting :-
[WARN ] 2018-08-22 10:57:20.378 [DispatchThread: [com.ibm.mq.jmqi.remote.impl.RemoteSession[connectionId=414D514353414D5030303144202020206CF77A5B9E4A5E21]]] SimpleMessageListenerContainer - Execution of JMS message listener failed, and no ErrorHandler has been set.
org.springframework.messaging.MessageDeliveryException: Dispatcher has no subscribers for channel 'app-name:local:9010.inputChannel'.; nested exception is org.springframework.integration.MessageDispatchingException: Dispatcher has no subscribers, failedMessage=GenericMessage
and below is my spring integration configuration class
Any idea why i am getting this exception .
Many thanks in advance
The problem is exactly what the message says.
Dispatcher has no subscribers for channel 'app-name:local:9010.inputChannel'.
You have no subscriber on this bean
#Bean
public MessageChannel inputChannel() {
return new DirectChannel();
}
EDIT
#ServiceActivator(inputChannel = "inputChannel")
public void handle(String in) {
...
}
or
#Transformer(inputChannel = "inputChannel", outputChannel = "transformed")
public String handle(String in) {
retur in.toUpperCase();
}
#ServiceActivator(inputChannel = "transformed")
public void handle(String in) {
...
}
How can one ensure in Spring Cloud Stream Rabbit guaranteed delivery. My code below : -
class Source {
MessageChannel output;
Repository repo;
#Transactional
void publisher(Command command){
repo.save(command);
output.send(MessageBuilder
.withPayload(new Event()).build());
}
}
class Sink {
#StreamListener(Event.class)
void eventListener(Event event){
// method body
}
}
Any help is appreciated.
You can use the Rabbit consumer property spring.cloud.stream.rabbit.bindings.<channelName>.consumer.acknowledgeMode on how do you want to acknowledge. The acknowledgeMode comes from Spring AMQP and you can refer more documentation on this here