Is it possible to configure Kafka for working with two separate clusters in single spring boot application?
Use case: I have two clusters with replicas + zookeeper:
Cluster #1 bootstrap-servers: server1.example.com,server2.example.com,server3.example.com
Cluster #2 bootstrap-servers: target-server1.example.com,target-server2.example.com,target-server3.example.com
I need to consume the message from Cluster #1 then do some calculations based on that data and produce the results to Cluster #2 topic. Is there any way to configure Kafka in single Spring application to handle this approach?
Yes, but if you want to consume and produce with both, you have to manually configure the consumer and producer factory #Beans etc.
Boot can only auto configure one of each from properties.
But if you are simply consuming from one cluster and producing to another, it can be done with properties.
Use
spring.kafka.producer.bootstrap-servers=...
...
spring.kafka.consumer.bootstrap-servers=...
these will override the common properties.
Related
I'm experiencing a problem, I have an application in spring boot, this application consumes from one topic and produces in another topic.
the topic that the application consumes is on-premises, the topic that the application produces is on cloud aws.
is there a way to specify a bootstrap server and schema registry for each topic?
my application.properties has the following property:
spring.kafka.bootstrap-servers=localhost:32202 spring.kafka.properties.schema.registry.url=127.0.0.1:8082
the problem here is that these properties are for both consumer and producer.
I need to specify a bootstrap server for the consumer, another for the producer.
also specify a schema-registry for the consumer, another for the producer.
I don't know if this way is the best way to deal with this problem.
spring.kafka.consumer.bootstrap-servers=consumer-localhost:32202 spring.kafka.consumer.schema.registry.url=consumer-127.0.0.1:8082 spring.kafka.producer.bootstrap-servers=producer-localhost:10010 spring.kafka.producer.schema.registry.url=producer-127.0.0.1:9090
Thanks in advance!
See the Spring Boot documentation.
The properties supported by auto configuration are shown in the “Integration Properties” section of the Appendix. Note that, for the most part, these properties (hyphenated or camelCase) map directly to the Apache Kafka dotted properties. See the Apache Kafka documentation for details.
The first few of these properties apply to all components (producers, consumers, admins, and streams) but can be specified at the component level if you wish to use different values. Apache Kafka designates properties with an importance of HIGH, MEDIUM, or LOW. Spring Boot auto-configuration supports all HIGH importance properties, some selected MEDIUM and LOW properties, and any properties that do not have a default value.
Only a subset of the properties supported by Kafka are available directly through the KafkaProperties class. If you wish to configure the producer or consumer with additional properties that are not directly supported, use the following properties:
spring.kafka.properties[prop.one]=first
spring.kafka.admin.properties[prop.two]=second
spring.kafka.consumer.properties[prop.three]=third
spring.kafka.producer.properties[prop.four]=fourth
spring.kafka.streams.properties[prop.five]=fifth
So...
spring.kafka.consumer.bootstrap-servers=consumer-localhost:32202
spring.kafka.consumer.properties.schema.registry.url=consumer-127.0.0.1:8082
spring.kafka.producer.bootstrap-servers=producer-localhost:10010
spring.kafka.producer.properties.schema.registry.url=producer-127.0.0.1:9090
This sets the common prop.one Kafka property to first (applies to producers, consumers and admins), the prop.two admin property to second, the prop.three consumer property to third, the prop.four producer property to fourth and the prop.five streams property to fifth.
We have configured our ActiveMQ message broker as a Spring Boot project and there's another Spring Boot application (let's call it service-A) that has a listener configured to listen to some topics using #JmsListener annotation. It's a Spring Cloud microservice appilcation.
The problem:
It is possible that service-A can have multiple instances running.
If we have 2 instances running, then any message coming on topic gets listened to twice.
How can we avoid every instance listening to the topic?
We want to make sure that the topic is listened to only once no matte the number of service-A instances.
Is it possible to run the microservice in a cluster mode or something similar? I also checked out ActiveMQ virtual destinations but not too sure if that's the solution to the problem.
We have also thought of an approach where we can decide who's the leader node from the multiple instances, but that's the last resort and we are looking for a cleaner approach.
Any useful pointers, references are welcome.
What you really want is a shared topic subscription which was added in JMS 2. Unfortunately ActiveMQ 5.x doesn't support JMS 2. However, ActiveMQ Artemis does.
ActiveMQ Artemis is the next generation broker from ActiveMQ. It supports most of the same features as ActiveMQ 5.x (including full support for OpenWire clients) as well as many other features that 5.x doesn't support (e.g. JMS 2, shared-nothing high-availability using replication, last-value queues, ring queues, metrics plugins for integration with tools like Prometheus, duplicate message detection, etc.). Furthermore, ActiveMQ Artemis is built on a high-performance, non-blocking core which means scalability is much better as well.
I am writing a service with Spring and I am using Spring AMQP in order to connect to Rabbitmq.
I have two rabbitmq clusters, one is only for publishing messages(the messages are sent to the other cluster via the federation plugin) and the other cluster is for declaring queues that end users will consume from.
The nodes sit behind aws lb, each cluster has a lb.
I am using CachingConnectionFactory and RabbitTemplate,RabbitAdmin in my code and I want to have connections to all the nodes so I can use them.
For the cluster that will contain the queues I added to the config the queue-master-locator=random so new queues will be declared in all the nodes in the cluster even if my service does not have a connection to them.
With the cluster that publishes messages I have more of a problem because I need a direct connection in my service to each of the nodes so I will be able to separate the load between the nodes.
So my problem is, how do I create connections in my service to all the nodes in the cluster so they will all be used for declaring queues and sending messages?
Now, after I will have some sort of solution to this issue, the next issue will be what happens when a new node is added to the cluster? How can I create a connection to it and start using it as well?
I am using Rabbitmq - 3.7.9, Spring - 2.0.5, Spring AMQP - 2.0.5
Thanks alot!
There is currently no mechanism to do anything like that.
By default, Spring AMQP opens only one connection (optionally two, one for publishing, one for consuming).
Even when using CacheMode.CONNECTION, you'll get a new connection for each consumer (and connections will be created and cached on demand for producers), you won't get any control as to which node it connects to; that's a function of the LB.
The framework does provide the LocalizedQueueConnectionFactory which will try to consume from the node that hosts a queue, but it won't work with a load balancer in place.
In general, however, such optimization is rarely needed.
Are you trying to solve an actual problem you are experiencing now, or something that you perceive that might be a problem?
It is generally best not to perform premature optimization.
I have messages coming in from Kafka. So I am planning to write a listener and "onMessage". I want to process it and push it in to solr.
So my question is more architectural, like I have worked on web apps all my career, so in big data how to deploy the spring kafka listener, so I can process thousands of messages a second.
How do I make my spring code use multiple nodes to distribute the
load?
I am planning to write a SpringBoot application to run in
a tomcat container.
If you use the same group id for all instances, different partitions will be assigned to different consumers (instances of your application).
So, be sure that you specified enough partitions in the topic you are going to consume.
We are exploring Archaius for our microservices. We want to setup a Configuration Server as microservice and store configuration files of other microservices.
We have other microservices (Springboot based) say
a) Producer
b) Consumer
deployed in different environment/vm. We also deploy all these microservices as a cluster (ie., run multiple instances of Producer and Consumer) to support high availability.
Please let us know how to get dynamically changed values in Configuration Server to be available to other microservices (multiple Producer and Consumers).
Thanks