in previous solution it was possible to listen for messages using filter like:
#StreamListener(target = Sink.INPUT, condition = "headers['header_name'].equals('header_value')")
How can I achieve this in new approach with Consumers?
Related
I am trying to implement a flow control similar to https://cloud.google.com/pubsub/docs/pull#flow_control using spring cloud stream reactive client.
#Bean
ApplicationRunner reactiveSubscriber(PubSubReactiveFactory reactiveFactory, PubSubMessageConverter converter) {
return (args) -> {
reactiveFactory.poll("orders-subscription", 250L)
// Convert a JSON payload into an object
.map(msg -> converter.fromPubSubMessage(msg.getPubsubMessage(), Order.class))
.doOnNext(order -> proccessOrder(order))
// Mannually acknowledge the message
.doOnNext(AcknowledgeablePubsubMessage::ack);
.subscribe();
};
}
It seems that back-pressure is not possible to limit number of data flowing through the system. Since the processing can last for few second I am afraid to cause some memory problems with high number of incoming message (millions of incoming message per day).
What I would like to achieve is a constant processing of 100 messages in the flux. Anyone had success implement it ? Maybe reactive rabbitmq ?
resolved using spring amqp and prefetch count.
I am having a Application which connects to RabbitMQ through Spring Cloud Stream, which works prefectly.
For Integration test cases i am trying to use the sample - https://github.com/piomin/sample-message-driven-microservices/blob/master/account-service/src/test/java/pl/piomin/services/account/OrderReceiverTest.java
However, in my case my application sends back 3 messages in some time Interval. So if i put the below Lines, it fetches the messages, but if the there is a delay in getting the messages.
int i = 1;
while (i > 0) {
Message<String> received = (Message<String>) collector.forChannel(channels.statusMessage()).poll();
if (received != null) {
LOGGER.info("Order response received: {}", received.getPayload());
}
}
So Instead of my custom polling, is there any way i can wait and Poll for my messages, and stop when i get those ?
I want to get the pick Messages based on the Response Routing Key to different Channels. Is it possible ?
--> Example: If the routingKey is "InProcess" , it should go to Inprocess Method.
1) Your question is not at all clear, expand on it and explain exactly what you mean.
2) Routing keys are used within Rabbit to route to different queues, they are not used within the framework to route to channels or methods.
You can, however, use a condition on the #StreamListener (match on the headers['amqp_receivedRoutingKey]`), but it's better to route messages to different queues instead.
In my Spring Boot/Kafka project I have the following listener:
#KafkaListener(topics = "${kafka.topic.update}", containerFactory = "updateKafkaListenerContainerFactory")
public void onUpdateReceived(ConsumerRecord<String, Update> consumerRecord, Acknowledgment ack) {
// do some logic
ack.acknowledge();
}
Inside of the listener I need to check some condition according to my business logic and if it is not met - skip processing of this certain message and let Kafka know to redeliver this message one more time.
The reason I need this - according to the business logic of my application I need to avoid sending more than one post per second into the particular Telegram chat. This why I'd like to check the chatLastSent time in the Kafka listener and postpone message sending if needed(via message redelivery to this Kafka topic)
How to properly do it? Do I only need to not perform the ack.acknowledge(); this time or there is another, more proper way in order to achieve it?
Use the SeekToCurrentErrorHandler.
When you throw an exception, the container will invoke the error handler which will re-seek the unprocessed messages so they will be fetched again on the next poll.
You can use a RecordFilterStrategy.
See doc here : https://docs.spring.io/spring-kafka/docs/2.0.5.RELEASE/reference/html/_reference.html#_filtering_messages
I am currently using Akka-Camel for integrating my Akka based application with various message queues.
Now that I want to upgrade to the latest version of Akka I see that this integration is now deprecated.
I have tried using alpakka-jsm integration but am unable to see whether I can perform explicit ack to the message queues once I have processed the message. Does this functionality exist in this new component?
Alpakka 0.15 makes the acknowledgement mode in the JMS connector configurable. An example that's adapted from the linked documentation:
val jmsSource: Source[Message, NotUsed] = JmsSource(
JmsSourceSettings(connectionFactory)
.withQueue("myqueue")
.withAcknowledgeMode(AcknowledgeMode.ClientAcknowledge)
)
val result = jmsSource
.map {
case textMessage: TextMessage =>
val text = textMessage.getText
textMessage.acknowledge()
text
}
.runWith(Sink.seq)
From what I've read so-far, a JMS Topic is 1-to-Many and I wonder if its possible to support Many-to-Many using a topic. Consider a topic called "Reports" with multiple services spread out across an enterprise needing to publish scheduled reports. Having multiple publishers would alleviate the need to subscribe interested applications to a topic for each of the reporting services.
Note:
I'm going to use Spring and ActiveMQ in my solution.
#Mondain: yes, very much possible. A practical example would be live stock market price feed provided by multiple sources and those feed consumed by multiple channels.
Yes, you can create many TopicPublisher from your TopicSession, and many applications can connect the same Topic using TopicPublisher or TopicSubscriber.
You can do something like this, and call CreateMessageProducer to create a new instance of producer anywhere in your application.
public ActiveMqProducer(string activeMqServiceUrl)
{
_activeMqServiceUrl = activeMqServiceUrl;
IConnectionFactory factory = new ConnectionFactory(new Uri(_activeMqServiceUrl));
_activeMqConnection = factory.CreateConnection();
_activeMqSession = _activeMqConnection.CreateSession(AcknowledgementMode.Transactional);
_activeMqConnection.Start();
}
private IMessageProducer CreateMessageProducer(string mqTopicName)
{
ITopic destination = SessionUtil.GetTopic(_activeMqSession, mqTopicName);
var producer = _activeMqSession.CreateProducer(destination);
return producer;
}