I am new to Kafka.
I have existing microservice with spring-boot, ribbon, eruka, and zuul.
If I use Kafka as the messaging platform between each microservice call, does kafka provide load balancer for microservice and I can get rid of ribbon ?
Please give me some suggestions.
Thanks
Kafka stores data in a distributed log and provides external clients for building a streaming platform. It is not a load balancer; but data is partitioned amongst servers so load is distributed as part of its custom TCP protocol.
Ribbon is a stateless service for spreading load over other services. I haven't used it, but it does not have an asynchronous, client, push-pull model to anything
You could use them together... A Kafka consumer would start an HTTP / RPC call to a Ribbon server
Related
We are developing an application using Spring Boot and Apache Camel that is reading a message from ActiveMQ Artemis, doing some transformation, and sending it to ActiveMQ Artemis. Our application is deployed as war file in on-premise JBoss EAP 7.2.0. Both the source and target applications are remote to our application and they are also deployed on JBoss EAP 7.2.0. The remote queues to which Camel is connecting are ActiveMQ Artemis which were created in JBoss and connecting using http-remoting protocol. These setup was working when there were only one node of each of the applications.
Now we are making the source and target applications 3 nodes each (i.e. they will be deployed in multiple JBoss servers). For accessing the front-end of the source and target applications we are configuring and accessing them through a load balancer.
Can we configure the load balancer to access the source and target brokers from the Camel layer? There will be 3 source and 3 target brokers. Or is clustering the brokers the only option in this case?
We are thinking of load balancing between the queues and not clustering. Suppose we have three queues q1, q2, and q3 with corresponding brokers b1, b2, and b3. I will configure the load balancer url in the Camel layer like http-remoting://<load-balancer-url>:<port> (much like we do while load balancing HTTP API requests). Any message coming in will hit the load balancer, and the load balancer will decide which queue to route the message to.
JMS connections are stateful. When a client creates a connection there is no indication of the queues to which it will send messages. The load-balancer will have to direct that client's connection to either b1, b2, or b3 and it will have no way to determine where it should go. A load-balancer working with messaging will almost certainly only be able to balance connections, not messages. It sounds like you want load-balancing at the message level instead. Perhaps you should look into something like Qpid Dispatch Router.
Messaging doesn't use HTTP so using an HTTP load balancer like you do with your HTTP API(s) won't work. It's easy for a load-balancer to inspect HTTP headers and route requests, especially since HTTP is stateless. However, messaging connections are stateful and the protocols are typically quite a bit more complex than HTTP. I don't know of any load-balancers that will work the way you are wanting for messaging.
You need your client not to use the topology, you can do this by using "setUseTopologyForLoadBalancing" on your AMQConnectionFactory. If you get the connection factory from EAP I think this is configurable on the connection factory since EAP 7.3.
I am trying to find a way to consume messages that being sent by a websocket to a kafka topic (the messages are sent by the websocket to the address 'ws://address:port/topic_name' and I want to add all of those messages to a kafka topic).
I read about kafka connect and tried to find a way to do it with it but it doesnt seem to work...
thanks in advance :)
There is no Kafka Connector to a socket in Confluent Platform.
I work in a team that use Kafka in production and our source is a socket, so your options are to use platforms that support this socket->Kafka producing, or write one by yourself.
About possible platforms, I think most of them will be overkill though you can utilize them for this problem, some options are:
1. NiFi or MiniFi for smaller loads, use PublishKafka Processor
2. StreamSets with Kafka Producer Destination
3. Apache Flume- not very recommended, this project is stops to evolve.
If you wish to write your own producer, you basically have to create a listener on this port, and produce the incoming messages to Kafka; if this is a web socket, just get the payload of the requests and produce them to Kafka.
Example Kafka Producer Code can be copied from tutorialspoint simple producer example*
Here are some open-source projects examples:
1. https://github.com/DataReply/kafka-connect-socket-source
2. https://github.com/kafka-socket/miniature_engine
3. https://github.com/dhanuka84/kafka-connect-tcp
4. https://github.com/krux/tcp-stream-kafka-producer
The idea of Kafka connect is that you have some sort of external integration that serves as storage. This can be SAP, Salesforce, RDBMS, MQ or anything else that has state. You websocket endpoint does not have data, you can not poll it it is someone else that is invoking it and there fore the data is transfered. Now if you know who is actualy holding the data than you can potentialy build a conector using this guide. https://docs.confluent.io/current/connect/devguide.html
For your particular case, the best you can do is either to use Kafka Producer API https://docs.confluent.io/current/clients/producer.html
and from your websocket enpoint use this producer to post a message to the topic, or even better if you are using spring you can use a higher level abstraction, that will be KafkaTemplate https://docs.spring.io/spring-kafka/reference/html/#sending-messages.
Full disclosure: I work for MigratoryData.
You can check out MigratoryData's solution for Kafka. MigratoryData is a scalable WebSocket server. The MigratoryData Source/Sink Connector for Kafka makes use of Kafka Connect API and can be used to stream data in real-time from Kafka to WebSocket clients and vice versa. The main advantage of the solution is it extends Kafka messaging to WebSocket clients while preserving Kafka's key features like guaranteed delivery, message ordering, etc.
We are using microservice architecture in our project. We deploy each service to a cluster by using Kubernetes. Services are developed by using Java programming language and Spring Boot framework.Three replicas exist for each service. Services communicate with each other using only events. RabbitMQ is used as a message queue. One of the services is used to send an email. The details of an email are provided by another service with an event. When a SendingEmail event is published by a service, three replicas of email service consume the event and the same email is sent three times.
How can I prevent that sending emails by other two services?
I think it depends on how you work with Rabbit MQ.
You can configure the rabbit mq with one queue for these events and make spring boot applications that represent the sending servers to be "Competing" Consumers.
If you configure it like this, only one replica will get an event at a time and only if it fails to process it the message will return to the queue and will become available to other consumers.
From what you've described all of them are getting the message, so it works like a pub-sub (which is also a possible way of work with rabbit mq, its just not good in this case).
I have 2 spring boot applications
On-premise teller application
Cloud based multi-tenant application that aggregates data from all teller applications
The teller application has to work offline(if connectivity is down) as well. What is the best approach to broadcast events from the teller application to the cloud. I would not prefer to implement code to persist events.
What is the best approach? ReST/message broker or some other approach. If using a message broker, will the spring cloud stream abstraction queue events when the connection to the broker is down and retry.
I would go with the message broker (rabbit, kafka) and spring-cloud-stream, since your use case was exactly what/how it was designed.
The microservcice (your app) is a consumer of the broker, that is: it can publish to and/or consume events from the broker. If the app is down the broker is up and potentially collecting events destined to the down app. Once app is up it consumes queued up events and so on.
I'll stop here given the general nature of your question but feel free to follow up with more details.
I want to know the major difference between VM and JMS component of Mule ESB. Can someone help me to know it.
As per Mule documentation, VM transport is for intra-JVM communication between Mule flows. So, that means when you use a VM in your flow, you can communicate between different flows in the application.
A flow containing VM inbound cannot be called externally from external application as thus the flow is equivalent to a private flow used within the application. By default uses in-memory queues.
Please go through the documentation :- https://docs.mulesoft.com/mule-user-guide/v/3.8/vm-transport-reference
On the other hand as per Mule documentation, JMS is an external host, allows communication between different components of a distributed application and JMS transport lets you easily send and receive messages to queues and topics for any message service which implements the JMS specification.
A flow, which has JMS inbound can be called from externally unlike VM. Documentation is here :- https://docs.mulesoft.com/mule-user-guide/v/3.8/jms-transport-reference
Within the application, if you send the control from one flow to another flow we use VM.VM can be used as both inbound and outbound.
Outside the application, for example, A application want to send something to B application(external application) there we use JMS.