I have an application which does a lot of data processing (in the order of ~1.3 million at a time) which happens in bursts. The application consumes data from a kafka topic.
I'm using a version 2.0.1 of spring-cloud-stream-starter-kafka to consume data.
My code is as follows:
Listener:
#Service
public class ListenerService {
#Autowired
private Application2<Foo> application;
#Override
#StreamListener(FooStreams.INPUT)
public void subscribe(#Payload Foo foo) {
application.sync(foo);
}
}
Streams:
public interface FooStreams {
String INPUT = "Foo";
#Input(value = INPUT)
SubscribableChannel subscribe();
}
In the main application, I've bound the stream to kafka like this:
#SpringBootApplication
#EnableBinding({FooStreams.class})
public class Application {
private static final Logger logger = LoggerFactory.getLogger(Application.class);
public static void main(String[] args) {
try {
SpringApplication.run(Application.class, args);
}
catch (Exception e) {
logger.error("Application failed to start");
}
}
}
Is there something I am missing? The issue is that I can see that the memory utilization spikes up during the time of data processing, which doesn't come down after the processing is done.
Related
I need to print/log/store the kafka partition and offset number in which my message is being processed.
How can I achieve that?
I am using StreamBridge to send the message from producer and also using functional spring kafka streams approach
Public delegateToSupplier(String id, Abc obj) {
Message<Abc> message = MessageBuilder.withPayload(obj).seHeaders(KafkaHeaders.MESSAGE_KEY, id.getBytes()).build();
streamBridge.send("out-topic", message);
}
The record metadata is available (asynchronously) via the metadata channel:
#SpringBootApplication
public class So66436499Application {
public static void main(String[] args) {
SpringApplication.run(So66436499Application.class, args);
}
#Autowired
StreamBridge bridge;
#Bean
public ApplicationRunner runner() {
return args -> {
this.bridge.send("myBinding", "test");
Thread.sleep(5000);
};
}
#ServiceActivator(inputChannel = "meta")
void meta(Message<?> sent) {
System.out.println("Sent: " + sent.getHeaders().get(KafkaHeaders.RECORD_METADATA, RecordMetadata.class));
}
}
spring.cloud.stream.bindings.myBinding.destination=foo
spring.cloud.stream.kafka.bindings.myBinding.producer.record-metadata-channel=meta
Sent: foo-0#5
I am looking for a way to stop consume messages with stream listener.
#StreamListener(MBinding.M_INPUT)
public void consumeMessage(Message<MerchantEvent> message) {
//handle when receive message
}
cloud:
stream:
bindings:
MInput:
destination: topicName
group: groupName
I have googled it but right now still have no idea how to stop consuming. Is there anyone who know it?
You can do it using the actuator (see Binding Visualization and Control). Or you can invoke the endpoint programmatically.
#SpringBootApplication
#EnableBinding(Sink.class)
public class So58795176Application {
public static void main(String[] args) {
SpringApplication.run(So58795176Application.class, args);
}
#StreamListener(Sink.INPUT)
public void listen(String in) {
System.out.println();
}
#Autowired
BindingsEndpoint endpoint;
#Bean
public ApplicationRunner runner() {
return args -> {
System.in.read();
endpoint.changeState("input", State.STOPPED);
System.in.read();
endpoint.changeState("input", State.STARTED);
};
}
}
With Spring Cloud Stream Kafka app, how can we ensure that the stream listener waits to process messages until some dependency tasks (reference data population, e.g.) are done? Below app fails to process messages because messages are delivered too early. How can we guarantee this kind of ordering within a Spring Boot App?
#Service
public class ApplicationStartupService implements ApplicationRunner {
private final FooReferenceDataService fooReferenceDataService;
#Override
public void run(ApplicationArguments args) throws Exception {
fooReferenceDataService.loadData();
}
}
#EnableBinding(MyBinding.class)
public class MyFooStreamProcessor {
#Autowired FooService fooService;
#StreamListener("my-input")
public void process(KStream<String, Foo> input) {
input.foreach((k,v)-> {
// !!! this fails to save
// messages are delivered too early before foo reference data got loaded into database
fooService.save(v);
});
}
}
spring-cloud-stream: 2.1.0.RELEASE
spring-boot: 2.1.2.RELEASE
I found this is not available in spring cloud stream as of May 15, 2018.
Kafka - Delay binding until complex service initialisation has completed
Do we have a plan/timeline when this is supported?
In the mean time, I achieved what I wanted by using #Ordered and ApplicationRunner. It's messy but works. Basically, stream listener will wait until other works are done.
#Service
#Order(1)
public class ApplicationStartupService implements ApplicationRunner {
private final FooReferenceDataService fooReferenceDataService;
#Override
public void run(ApplicationArguments args) throws Exception {
fooReferenceDataService.loadData();
}
}
#EnableBinding(MyBinding.class)
#Order(2)
public class MyFooStreamProcessor implements ApplicationRunner {
#Autowired FooService fooService;
private final AtomicBoolean ready = new AtomicBoolean(false);
#StreamListener("my-input")
public void process(KStream<String, Foo> input) {
input.foreach((k,v)-> {
while (ready.get() == false) {
try {
log.info("sleeping for other dependent components to finish initialization");
Thread.sleep(10000);
} catch (InterruptedException e) {
log.info("woke up");
}
}
fooService.save(v);
});
}
#Override
public void run(ApplicationArguments args) throws Exception {
ready.set(true);
}
}
I had implement CQRS+ES application using axon and spring-boot. I use separate query model and command model application. I use rabbitmq to publish event from command mode. It works correct. But tracking Processor implementation is not work in my application.
This is my query model
#SpringBootApplication
public class SeatQueryPart1Application {
public static void main(String[] args) {
SpringApplication.run(SeatQueryPart1Application.class, args);
}
#Bean
public SpringAMQPMessageSource statisticsQueue(Serializer serializer) {
return new SpringAMQPMessageSource(new DefaultAMQPMessageConverter(serializer)) {
#RabbitListener(exclusive = false, bindings = #QueueBinding(value = #Queue, exchange = #Exchange(value = "ExchangeTypesTests.FanoutExchange", type = ExchangeTypes.FANOUT), key = "orderRoutingKey"))
#Override
public void onMessage(Message arg0, Channel arg1) throws Exception {
super.onMessage(arg0, arg1);
}
};
}
#Autowired
public void conf(EventHandlingConfiguration configuration) {
configuration.registerTrackingProcessor("statistics");
}
}
this is a event handler class
#ProcessingGroup("statistics")
#Component
public class EventLoggingHandler {
private SeatReservationRepository seatReservationRepo;
public EventLoggingHandler(final SeatReservationRepository e) {
this.seatReservationRepo = e;
}
#EventHandler
protected void on(SeatResurvationCreateEvent event) {
Timestamp timestamp = new Timestamp(System.currentTimeMillis());
Seat seat=new Seat(event.getId(), event.getSeatId(), event.getDate(),timestamp ,true);
seatReservationRepo.save(seat);
}
}
this is yml configuration
axon:
eventhandling:
processors:
statistics.source: statisticsQueue
How can i do it correct. (Can anyone suggest tutorial or code sample)
The SpringAMQPMessageSource is a SubscribableMessageSource. This means you cannot use a tracking event processor to process messages. It is only compatible with a Subscribable Event Processor.
Removing configuration.registerTrackingProcessor("statistics"); and leaving it to the default (subscribing) should do the trick.
I'm working on a spring-boot service that performs sensitive payment processing, and would like to ensure that any shutdown to the app will be done without interrupting these transactions. Curious on how to best approach this in spring-boot.
I read about adding shutdown hooks to spring-boot, and I was thinking maybe to use a CountDownLatch on the class to check if the thread has completed processing - something like this:
#Service
public class PaymentService {
private CountDownLatch countDownLatch;
private void resetLatch() {
this.countDownLatch = new CountDownLatch(1);
}
public void processPayment() {
this.resetLatch();
// do multi-step processing
this.CountDownLatch.countDown();
}
public void shutdown() {
// blocks until latch is available
this.countDownLatch.await();
}
}
// ---
#SpringBootApplication
public class Application {
public static void main(String[] args) {
// init app and get context
ConfigurableApplicationContext context = SpringApplication.run(Application.class, args);
// retrieve bean needing special shutdown care
PaymentService paymentService = context.getBean(PaymentService.class);
Runtime.getRuntime().addShutdownHook(new Thread(paymentService::shutdown));
}
}
Constructive feedback is greatly appreciated - thanks.
I ended up using #PreDestroy annotation on the shutdown method:
#Service
public class PaymentService {
private CountDownLatch countDownLatch;
private synchronized void beginTransaction() {
this.countDownLatch = new CountDownLatch(1);
}
private synchronized void endTransaction() {
this.countDownLatch.countDown();
}
public void processPayment() {
try {
this.beginTransaction();
// - - - -
// do multi-step processing
// - - - -
} finally {
this.endTransaction();
}
}
#PreDestroy
public void shutdown() {
// blocks until latch is available
this.countDownLatch.await();
}
}