aggregator spring cloud stream with timeout - spring

I want to make an application that receives messages, stores those messages in a list, and later with and schedule releases those messages every x amount of time.
I know spring cloud stream has an aggregator that already does this, but I think I need it to be done manually because I need to keep a unique message based upon a key and only replace the old message if it matches a specific condition ( I think of it as a Set aggregator with conditions)
what I have tried so far.
also in this link https://github.com/chalimbu/AggregatorQuestionStack
Processor.
import org.springframework.cloud.stream.annotation.EnableBinding
import org.springframework.cloud.stream.annotation.Input
import org.springframework.cloud.stream.annotation.Output
import org.springframework.cloud.stream.messaging.Processor
import org.springframework.scheduling.annotation.Scheduled
#EnableBinding(Processor::class)
class SetAggregatorProcessor(val storageService: StorageService) {
#Input
public fun inputMessage(input: Map<String,Any>){
storageService.messages.add(input)
}
#Output
#Scheduled(fixedDelay = 20000)
public fun produceOutput():List<Map<String,Any>>{
val message= storageService.messages
storageService.messages.clear()
return message;
}
}
Memory storage.
import org.springframework.stereotype.Service
#Service
class StorageService {
public var messages: MutableList<Map<String,Any>> = mutableListOf()
}
This code generates the following error when I start pushing messages.
Caused by: org.springframework.integration.MessageDispatchingException: Dispatcher has no subscribers
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:139) ~[spring-integration-core-5.5.8.jar:5.5.8]
The idea is to deploy this app as part of the spring cloud stream (dataflow) platform.
I prefer the declarative approach(over the functional approach), but if somebody knows how to do it with the reactor way, I could settle for it.
Thanks for any help or advice.

thanks to this example(https://github.com/spring-cloud/spring-cloud-stream-samples/blob/main/processor-samples/sensor-average-reactive-kafka/src/main/java/sample/sensor/average/SensorAverageProcessorApplication.java) I was able to figure something out using flux in case someone else needs it
#Configuration
class SetAggregatorProcessor : Function<Flux<Map<String, Any>>, Flux<MutableList<Map<String, Any>>>> {
override fun apply(data: Flux<Map<String, Any>>):Flux<MutableList<Map<String, Any>>> {
return data.window(Duration.ofSeconds(20)).flatMap { window: Flux<Map<String, Any>> ->
this.aggregateList(window)
}
}
private fun aggregateList(group: Flux<Map<String, Any>>): Mono<MutableList<Map<String, Any>>>? {
return group.reduce(
mutableListOf(),
BiFunction<MutableList<Map<String, Any>>, Map<String, Any>, MutableList<Map<String, Any>>> {
acumulator: MutableList<Map<String, Any>>, element: Map<String, Any> ->
acumulator.add(element)
acumulator
}
)
}
}
update https://github.com/chalimbu/AggregatorQuestionStack/tree/main/src/main/kotlin/com/project/co/SetAggregator

Related

Migrate from TopicProcessor

TopicProcessor has been removed since the 3.4+ reactor version and now I'm struggling how is the easiest way to replace it with a new API. I'd like to keep the implementation as reliable as possible, even if the performance will get worst. I don't want to struggle with loss events or an overflowing stream, just keep it stupid simple. I also need to buffer data, so I've used bufferTimeout for that and it worked perfectly with TopicProcessor.
How to achieve that with the new API?
I've tried to use Sinks but with a lot of pressure there was an overflow exception and the whole stream was terminated.
Below is my basic implementation before upgrade to Spring Boot 2.4.1.
package com.example.reactorplayground.config;
import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;
public class TopicProcessor<T> implements Processor<T> {
private final reactor.core.publisher.TopicProcessor<T> processor;
public TopicProcessor() {
this.processor = reactor.core.publisher.TopicProcessor.share("in-memory-topic-thread", 1);
}
#Override
public Mono<T> add(T event) {
return Mono.fromCallable(() -> {
processor.onNext(event);
return event;
});
}
#Override
public Flux<T> consumeWith() {
return processor;
}
}

Performing Aggregation of records and launch spring cloud task in single Processor in Spring cloud stream

I am trying to perform the following actions
Aggregating messages
Launching Spring Cloud Task
But not able to pass the aggregated message to the method launching Task. Below is the piece of code
#Autowired
private TaskProcessorProperties processorProperties;
#Autowired
Processor processor;
#Autowired
private AppConfiguration appConfiguration ;
#Transformer(inputChannel = MyProcessor.intermidiate, outputChannel = Processor.OUTPUT)
public Object setupRequest(String message) {
Map<String, String> properties = new HashMap<>();
if (StringUtils.hasText(this.processorProperties.getDataSourceUrl())) {
properties.put("spring_datasource_url", this.processorProperties.getDataSourceUrl());
}
if (StringUtils.hasText(this.processorProperties.getDataSourceDriverClassName())) {
properties.put("spring_datasource_driverClassName", this.processorProperties
.getDataSourceDriverClassName());
}
if (StringUtils.hasText(this.processorProperties.getDataSourceUserName())) {
properties.put("spring_datasource_username", this.processorProperties
.getDataSourceUserName());
}
if (StringUtils.hasText(this.processorProperties.getDataSourcePassword())) {
properties.put("spring_datasource_password", this.processorProperties
.getDataSourcePassword());
}
properties.put("payload", message);
TaskLaunchRequest request = new TaskLaunchRequest(
this.processorProperties.getUri(), null, properties, null,
this.processorProperties.getApplicationName());
System.out.println("inside task launcher **************************");
System.out.println(request.toString() +"**************************");
return new GenericMessage<>(request);
}
#ServiceActivator(inputChannel = Processor.INPUT,outputChannel = MyProcessor.intermidiate)
#Bean
public MessageHandler aggregator() {
AggregatingMessageHandler aggregatingMessageHandler =
new AggregatingMessageHandler(new DefaultAggregatingMessageGroupProcessor(),
new SimpleMessageStore(10));
AggregatorFactoryBean aggregatorFactoryBean = new AggregatorFactoryBean();
//aggregatorFactoryBean.setMessageStore();
//aggregatingMessageHandler.setOutputChannel(processor.output());
//aggregatorFactoryBean.setDiscardChannel(processor.output());
aggregatingMessageHandler.setSendPartialResultOnExpiry(true);
aggregatingMessageHandler.setSendTimeout(1000L);
aggregatingMessageHandler.setCorrelationStrategy(new ExpressionEvaluatingCorrelationStrategy("'FOO'"));
aggregatingMessageHandler.setReleaseStrategy(new MessageCountReleaseStrategy(3)); //ExpressionEvaluatingReleaseStrategy("size() == 5")
aggregatingMessageHandler.setExpireGroupsUponCompletion(true);
aggregatingMessageHandler.setGroupTimeoutExpression(new ValueExpression<>(3000L)); //size() ge 2 ? 5000 : -1
aggregatingMessageHandler.setExpireGroupsUponTimeout(true);
return aggregatingMessageHandler;
}
To pass the message between aggregator and task launcher method (setupRequest(String message)) , i am using a channel MyProcessor.intermidiate defined as below
import org.springframework.cloud.stream.annotation.Input;
import org.springframework.cloud.stream.annotation.Output;
import org.springframework.messaging.MessageChannel;
import org.springframework.messaging.SubscribableChannel;
import org.springframework.stereotype.Indexed;
public interface MyProcessor {
String intermidiate = "intermidiate";
#Output("intermidiate")
MessageChannel intermidiate();
}
Applicaion.properties used is below
aggregator.message-store-type=persistentMessageStore
spring.cloud.stream.bindings.input.destination=output
spring.cloud.stream.bindings.output.destination=input
Its not working , With the above mentioned approach .
In this class if i change the channel name from my defined channel MyProcessor.intermediate to Processor.input or Processor.output than any one of the things works (based on the channel name changed to Processor.*)
I want to aggregate the messages first and than want to launch task on aggragated messages in processor, which is not happening
See here:
public Object setupRequest(String message) {
So, you expect some string as a request payload.
Your AggregatorFactoryBean use a DefaultAggregatingMessageGroupProcessor, which does exactly this:
List<Object> payloads = new ArrayList<Object>(messages.size());
for (Message<?> message : messages) {
payloads.add(message.getPayload());
}
return payloads;
So, it is definitely not a String.
It is strange that you don't show what exception happens with your configuration, but I assume you need to change setupRequest() signature to expect a List of payloads or you need to provide some custom MessageGroupProcessor to build that String from the group of messages you have aggregated.

How to launch corutines in spring applications

I have an app that consumes some messages from a message Queue and processes them. The processing is implemented as suspending functions and there is a service that will publish the events to a Channel<Event>, I have another service that will basically do:
for (event in channel) {
eventProcessor.process(event)
}
The problem is that this is also a suspending function, and I am really not sure what's the proper way to launch it within the context of Spring.
My initial solution was to do the following:
#Bean
fun myProcessor(eventProcessor: EventProcessor, channel: Channel<Event>): Job {
GlobalScope.launch {
eventProcessor.startProcessing(channel)
}
}
But it seems somehow hacky, and I am not sure what's the proper way to do it.
Launching anything on a GlobalScope is a really bad idea. You loose all advantages of structured concurrency this way.
Instead, make your EventProcessor implement CoroutineScope.
This will force you to specify coroutineContext, so you can use Dispatchers.Default:
override val coroutineContext = Dispatchers.Default
So, the full example will look something like this:
#SpringBootApplication
class SpringKotlinCoroutinesApplication {
#Bean
fun myProcessor(eventProcessor: EventProcessor, channel: Channel<Event>): Job {
return eventProcessor.startProcessing(channel)
}
#Bean
fun p() = EventProcessor()
#Bean
fun c() = Channel<Event>()
}
fun main(args: Array<String>) {
runApplication<SpringKotlinCoroutinesApplication>(*args)
}
class Event
class EventProcessor : CoroutineScope {
override val coroutineContext = Dispatchers.Default
fun startProcessing(channel: Channel<Event>) = launch {
for (e in channel) {
println(e)
}
}
}

How to access JMS statistics in Spring Boot?

I understand that JMS has no statistics specs, so there is no standard way of reading things like "count of messages processed", "average time in queue" etc.
I'm looking at two approaches:
Access the ActiveMQ statistics directly
Maintain statistics in the JMS message consumer
With (1) I'm not finding examples how to get thos stats using Spring Boot. With (2), I'm wondering if the consumer itself needs to maintain the statistics, or if there's a better way.
Does anyone have any working examples?
For the record, I ended up implementing a broker-specific solution (here: ActiveMQ)
import org.springframework.stereotype.Component
import javax.jms.*
typealias QueueName = String
#Component
class BrokerFacade(private val connectionFactory: ConnectionFactory) {
private val statisticsBrokers = mutableMapOf<QueueName, StatisticsBrokerAccess>()
#Throws(JMSException::class)
fun getStatistics(queueName: QueueName): QueueStatistics? {
val brokerAccess = statisticsBrokers.getOrPut(queueName, { StatisticsBrokerAccess(queueName) })
return brokerAccess.getCurrentStatistics()?.let {
QueueStatistics(
queueName,
it.getLong("size"),
it.getLong("dequeueCount"),
it.getDouble("minEnqueueTime"),
it.getDouble("maxEnqueueTime"),
it.getDouble("averageEnqueueTime"),
it.getLong("memoryUsage"),
it.getLong("memoryPercentUsage")
)
}
}
inner class StatisticsBrokerAccess(queueName: QueueName) {
private val statisticsMessageConsumer: MessageConsumer;
private val statisticsMessageProducer: MessageProducer;
private val statisticsMessage: Message;
init {
val connection = connectionFactory.createConnection()
connection.start()
val session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE)
val statisticsReplyQueue = session.createTemporaryQueue()
statisticsMessageConsumer = session.createConsumer(statisticsReplyQueue)
val statisticsQueue = session.createQueue("ActiveMQ.Statistics.Destination.$queueName")
statisticsMessageProducer = session.createProducer(statisticsQueue)
statisticsMessage = session.createMessage()
statisticsMessage.setJMSReplyTo(statisticsReplyQueue)
}
fun getCurrentStatistics(): MapMessage? {
statisticsMessageProducer.send(statisticsMessage)
return statisticsMessageConsumer.receive(2000) as MapMessage?
}
}
}
QueueStatistics is a data class holding the statistic values.
If you are using JMS with Spring integration, its system management features provide statistics that are registered as micrometer metrics in the spring boot app.
https://docs.spring.io/spring-integration/docs/5.1.7.RELEASE/reference/html/#system-management-chapter
https://docs.spring.io/spring-boot/docs/2.1.7.RELEASE/reference/html/boot-features-integration.html
https://docs.spring.io/spring-boot/docs/2.1.7.RELEASE/reference/html/production-ready-metrics.html

Running cucumber-groovy features against a spring boot api

I've been attempting to get cucumber-groovy working with spring-boot, but it's not been going well. I get the error org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://localhost:8080/applicants": Connection refused; nested exception is java.net.ConnectException: Connection refused which seems to indicate that it's hitting the endpoint, but that the service isn't running.
I've read that I need to have a cucumber.xml file, but my project is not using any xml config, it's all annotations, so instead I've got this:
package support
import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.Configuration;
#Configuration
#ComponentScan(basePackages = "com.base.package")
public class CucumberConfiguration {}
I've added it to the World, but this seems to be the wrong way of doing things (i.e. I don't know how to add an annotation on groovy step defs).
package support
import com.thing.app.Application
import org.junit.runner.RunWith
import org.springframework.boot.test.SpringApplicationContextLoader
import org.springframework.boot.test.WebIntegrationTest
import org.springframework.test.context.ContextConfiguration
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner
import org.springframework.test.context.web.WebAppConfiguration
import static cucumber.api.groovy.Hooks.*
//#RunWith(SpringJUnit4ClassRunner)
//#ContextConfiguration(classes = Application, loader = SpringApplicationContextLoader)
//#WebAppConfiguration
//#WebIntegrationTest
#ContextConfiguration(classes = CucumberConfiguration)
public class AbstractTest {
}
World() {
new AbstractTest()
}
Before() {}
After() {}
I left in my other annotations to kind of show what I've done so far. None of it has worked.
I've also tried setting up an AbstractDefs class as seen here https://github.com/jakehschwartz/spring-boot-cucumber-example/tree/master/src/test/java/demo, but that also hasn't worked, mostly because I'm not using the cucumber-java style of things, but instead the cucumber-groovy style, which doesn't use step definition classes.
Edit: Just discovered I was doing things wrong by having an env.groovy, I'm used to the ruby cucumber, so I'm having trouble finding all the little problems. Still am having the same issue though, I don't know how to execute in a Spring context.
You can instantiate Spring test context with io.cucumber.spring.SpringFactory and register adapter in World to allow groovy script has access to Spring beans:
env.groovy:
#ContextConfiguration(classes = TestConfiguration, loader = SpringBootContextLoader)
class CucumberContextConfiguration {
}
//adapter bypassing World properties to Spring context
class SpringFactoryWorldAdapter {
private final SpringFactory factory;
SpringFactoryWorldAdapter(SpringFactory factory) {
this.factory = factory;
}
#Override
Object getProperty(String s) {
return factory.testContextManager.getContext().getBean(s);
}
}
def factory; //Keep state to prevent repeated context initialization
World { args ->
if (factory == null) {
factory = new SpringFactory()
factory.addClass(CucumberContextConfiguration)
factory.start()
}
new SpringFactoryWorldAdapter(factory)
}

Resources