I’m implementing RestFul api using spring boot application. Based on the requester, I need to pick the topic and get the data from that topic.
Let say if I hit url requester1/data then I should get data from topic1 and if I hit the url requester2/data then I should get the data from topic2 and so on.
So how to handle these multiple topics in the spring boot app and get data from a particular topic when I get the request?
Well. you should inject a ConsumerFactory<?, ?> into your RESTFull service and use a KafkaConsumer API directly.
Any existing KafkaMessageListenerContainer architecture in the Spring Kafka is long-lived, event-driver implementation it can't be adjusted at runtime to perform consumption on demand. That's exactly for what KafkaConsumer exists.
Related
The messages created by the producer are all being consumed as expected.
The thing is, I need to create an endpoint to retrieve the latest messages from the consumer.
Is there a way to do it?
Like an on-demand consumer?
I found this SO post but is only to consume the last N records. I want to consume the latest without caring about the offsets.
Spring Kafka Consumer, rewind consumer offset to go back 'n' records
I'm working with Kotlin but if you have the answer in Java I don't mind either.
There are several ways to create listener containers dynamically; you can then start/stop them on demand. To get the records back into the controller, you'd need to use something like a blocking queue, or make the controller itself a MessageListener.
These answers show a couple of techniques for creating containers on demand:
How to dynamically create multiple consumers in Spring Kafka
Kafka Consumer in spring can I re-assign partitions programmatically?
A spring integration project pulls emails from Exchange Server using imap-idle-channel-adapter; it transforms the message; it invokes some SOAP webservices and persists data in DB using Spring Boot and JPA. All works fine.
This needs to be deployed in a four-weblogic-server cluster environment.
Could someone please help with some hints on what needs to be done? Is there any configuration needed?
As long as your logic is just like you show and there is no any more endpoints polling shared resource, your are good so far do nothing more. The mail API has built-in feature to mark messages in the box as read or at least seen, so other concurrent session won’t poll those messages again.
In a few words:
I'm trying to decide between using the default Spring for Apache Kafka stack, KafkaTemplate or the pair, ReactiveKafkaProducerTemplate and ReactiveKafkaConsumerTemplate for my Reactor based application.
Some more context:
In the company I work we're developing a high-disponibility application aiming to publish a set of requests directly to a Kafka Broker. Since this is an API centric application expecting to receive a few millions of requests per week, we decided to go with a stack based on the Project Reactor with Spring WebFlux and Kotlin.
After doing some digging I've discovered that the Spring for Apache Kafka has a simple wrapper designed around the Reactor Kafka implementation, but this wrapper lacks a lot of the functionalities present in the default KafkaTemplate mentioned before, things like: A Metrics Binder out of the box (for prometheus integration), associated factories, extensive documentation, Auto configuration, etc.
I'm trying to understand what I'm really giving up when using the default implementation in favor of the Reactive one. Am I giving up back pressure functionality? Am I sacrificing the Reactive Stack present in my application? Will this be a toll in the future? Does anyone has some experience in working with a Reactive Stack alongside a non-reactive solution?
I have, also, a few concerns regarding the DLT flow facilitated in the default implementation, things like the SeekToCurrentErrorHandler strategy
What benefits does spring Kafka template provide?
I have tried the existing Producer/Consumer API by Kafka. That is very simple to use, then why use Kafka template.
Kafka Template internally uses Kafka producer so you can directly use Kafka APIs. The benefit of using Kafka template is it provides different methods for sending message to Kafka topic, kind of added benefits you can see the API comparison between KafkaProducer and KafkaTemplate here:
https://kafka.apache.org/10/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html
https://docs.spring.io/spring-kafka/api/org/springframework/kafka/core/KafkaTemplate.html
You can see KafkaTemplate provide many additional ways of sending data to Kafka topics because of various send methods while some calls are the same as Kafka API and are simply forwarded from KafkaTemplate to KafkaProducer.
It's up to the developer what to use. If you feel like working with KafkaTemplate is easy as you don't have to create ProducerRecord a simple send method will do all the work for you.
At a high level, the benefit is that you can externalize your properties objects more easily and you can just focus on the record processing logic
Plus Spring is integrated with lots of other components.
Note: Other options still exist like Reactor Kafka, Alpakka, Apache Camel, Smallrye reactive messaging, Vert.x... But they all wrap the same Kafka API.
So, I'd say you're (marginally) trading efficiency for convinience
I am trying to use Spring Boot 1.5.2.RELEASE + Camel (Spring Boot starter) 2.19.2 to listen to ActiveMQ queue and then post the message to a rest endpoint URL (POST method) as its body. What would be the best possible way to achieve this?
I have gathered pieces of information and am trying to tie it all together but getting a bit confused.
Here is what I have gathered for Camel Rest DSL, I am not too sure if camel below is creating these rest services via this or is it just an already exposed endpoint, in my case it is an already exposed endpoint
rest("/basePath")
post("/someEndpoint").to("direct:restEndpoint")
Using the above is what I have gathered for ActiveMQ which I am not too sure is correct
from("activemq:queue:<queue_name>").to("direct:restEndpoint")
But again, I am not too sure how to listen to the ActiveMQ queue for new messages or is it something that Camel would do by default always? Additionally, I need to pass the message as a post body to my rest endpoint. I also saw some references to camel-http4 and camel-http as well and I am completely confused.
Any assistance would be greatly appreciated.
Some confusion is common when starting to use Camel, but your final solution will look something like:
from("activemq:queue:my-route")
.process(/* change the in/out messages if you need to */)
.to("http4://your-endpoint.com");
Don't try to simply copy/paste this code until it works. My Camel rule of thumb is: always read the component documentation and try playing with it using it in your software. In your case I suggest:
Read ActiveMQ component docs and try reading from ActiveMQ / writing to a Log;
Generate some input from a Timer and send to your Rest endpoint using HTTP4 Component;
Your first routes will take some time for simple things but you will get on flow quickly.