I've seen brokers like RabbitMQ or Apache Pulsar provide a Websocket API to connect directly your browser to the broker.
AFAIK, I've not seen the same for Apache Kafka. You have to implement yourself an intermediary Websocket server.
Why Confluent, which owns Apache Kafka development, does not provide an out-of-the-box websocket API like Rabbitmq or Pulsar ?
Confluent does not own Apache Kafka development. Apache Kafka is a project owned by the Apache Software Foundation (ASF).
There are several examples of how to use WebSockets with Apache Kafka:
https://www.confluent.io/blog/data-stream-processing-with-kafka-streams-bitrock-and-confluent/
https://dev.to/victorgil/kafka-websockets-angular-event-driven-microservices-all-the-way-to-the-frontend-12aa
https://medium.com/swlh/angular-spring-boot-kafka-how-to-stream-realtime-data-the-reactive-way-510a0f1e5881
Your question seems somewhat a rhetorical one, but if you actually would like to see Websockets as part of Apache Kafka then the first step would be to raise a Kafka Improvement Proposal (KIP) for discussion in the community. There is a dev mailing list where you can get further guidance on the process on contributing code.
There is already an open source connecting Kafka with Websocket. https://github.com/b/kafka-websocket
If you have requirements connecting to Kafka from a browser, I suggest to consider to browserfiy this http://github.com/confluentinc/kafka-rest-node
There is a kafka connector to Ably hosted on the Confluent Hub. Ably is essentially a serverless WebSocket option (with pub/sub and message queues). You will also find a kafka rule on Ably's website.
Related
I am trying to find a way to consume messages that being sent by a websocket to a kafka topic (the messages are sent by the websocket to the address 'ws://address:port/topic_name' and I want to add all of those messages to a kafka topic).
I read about kafka connect and tried to find a way to do it with it but it doesnt seem to work...
thanks in advance :)
There is no Kafka Connector to a socket in Confluent Platform.
I work in a team that use Kafka in production and our source is a socket, so your options are to use platforms that support this socket->Kafka producing, or write one by yourself.
About possible platforms, I think most of them will be overkill though you can utilize them for this problem, some options are:
1. NiFi or MiniFi for smaller loads, use PublishKafka Processor
2. StreamSets with Kafka Producer Destination
3. Apache Flume- not very recommended, this project is stops to evolve.
If you wish to write your own producer, you basically have to create a listener on this port, and produce the incoming messages to Kafka; if this is a web socket, just get the payload of the requests and produce them to Kafka.
Example Kafka Producer Code can be copied from tutorialspoint simple producer example*
Here are some open-source projects examples:
1. https://github.com/DataReply/kafka-connect-socket-source
2. https://github.com/kafka-socket/miniature_engine
3. https://github.com/dhanuka84/kafka-connect-tcp
4. https://github.com/krux/tcp-stream-kafka-producer
The idea of Kafka connect is that you have some sort of external integration that serves as storage. This can be SAP, Salesforce, RDBMS, MQ or anything else that has state. You websocket endpoint does not have data, you can not poll it it is someone else that is invoking it and there fore the data is transfered. Now if you know who is actualy holding the data than you can potentialy build a conector using this guide. https://docs.confluent.io/current/connect/devguide.html
For your particular case, the best you can do is either to use Kafka Producer API https://docs.confluent.io/current/clients/producer.html
and from your websocket enpoint use this producer to post a message to the topic, or even better if you are using spring you can use a higher level abstraction, that will be KafkaTemplate https://docs.spring.io/spring-kafka/reference/html/#sending-messages.
Full disclosure: I work for MigratoryData.
You can check out MigratoryData's solution for Kafka. MigratoryData is a scalable WebSocket server. The MigratoryData Source/Sink Connector for Kafka makes use of Kafka Connect API and can be used to stream data in real-time from Kafka to WebSocket clients and vice versa. The main advantage of the solution is it extends Kafka messaging to WebSocket clients while preserving Kafka's key features like guaranteed delivery, message ordering, etc.
is there an bridge which can send data from a kafka topic to amqp(rabbitmq) or mqtt ?
I just find only the from mqtt/amqp to kafka but not backwards...
BR,
I believe the Stimzi supplies a Kafka AMQP 1.0 bridge that could be used for such a task. The project source is located on Github here so I'd suggest starting there and reading the documentation.
I have a 3rd party system pumping data into HornetQ using JMS. I need to replace HornetQ by Kafka but I cannot change the 3rd party system. What is the correct way to get the data into kafka.
I googled around and found JMS-Client and kafka connect. After reading both documentation I'm confused and not sure which one is the right one.
Has anyone any experience with this and can give me some hints on how to do this?
The right way is to use the JMS-Client because it's an implementation of the JMS API specification but with the Kafka wire-protocol. It means that you can use this client in your 3rd party system and using Kafka instead of HornetQ on the other side. It means that at least you need to add this dependency to the 3rd party system in order to use this JMS implementation for Kafka instead of the HornetQ one.
Use the Kafka JMS Client when you want to replace a JMS Broker with Apache Kafka
Use the Kafka JMS Connector when you want to integrate Kafka with a legacy JMS broker and send messages between the two different systems.
I was wondering could Apache Kafka communicate and send messages to JMS? Can I establish connection between them? For example, I'm using JMS in my system and it should send messages to the other system that uses Kafka
answering bit late, but if I understood correctly the requirement.
If the requirement is synchronous messaging from
client->JMS->Kafka --- > consumer
then following is not the solution, but if its ( and most likely) the async requirement like:
client->JMS | ----> Kafka ---> consumer
then, this would be related to KafkaConnect framework which is solving the problem of how to integrate different sources and sinks with Kafka.
http://docs.confluent.io/2.0.0/connect/
http://www.confluent.io/product/connectors
so what you need is a JMSSourceConnector.
Not directly. And the two are incomparable concepts. JMS is a vendor-neutral API specification of a messaging service.
While Kafka may be classified as a messaging service, it is not compatible with the JMS API, and to the best of my knowledge there is no trivial way of adapting JMS to fit Kafka's use cases without making significant compromises.
However, if your needs are simply to move messages between Kafka and a JMS-compliant broker, then this can easily be achieved by either writing a simple relay app that consumes from one and publishes onto another, or use something like Kafka Connect, which has pre-canned sinks for most data sources, including JMS brokers, databases, etc.
If the requirement is the reverse of the previous answer:
Kafka Producer -> Kafka Broker -> JMS Broker -> JMS Consumer
then you would need a KafkaConnect Sink like the following one from Data Mountaineer
http://docs.datamountaineer.com/en/latest/jms.html
I have tested AMQP Transport implementation for RabbitMQ (v3.3.5) from WSO2 ESB (v4.8.1) Proxies with a few minor issues, now everything works fine. For this, I followed this blog post (from 2013/03/26).
Now, I would like to try with WSO2 MessageStore & MessageProcessor with RabbitMQ, because I think this is the best approach to follow, but seems that is not any implementation for RabbitMQ and WSO2's Documentation Portal says nothing of how to implement it.
I know that RabbitMQ (v3.3.5) do not implement JMS1.1 and AMQP1.0 by default, It does need extra plugins. Then, What do I need to follow to implement MS/MP correctly?.
I have started reviewing the source code of the current implementation of org.apache.synapse.message.store.impl.jms.JmsStore and org.apache.synapse.message.processor.impl.forwarder.ScheduledMessageForwardingProcessor, but I need more guidance.
Someone has implement custom MessageStore/MessageProcessor?, Could you share your experience?.
Regards.
After digging in existing WSO2 JMS MessageStore implementation, I have created a new JMS-AMQP MessageStore for RabbitMQ. Using WSO2 MessageStore I can manage DeadLetter Channel and GuaranteedDelivery Patterns from WSO2 ESB.
The steps followed were:
1) To solve incompatibility issues between AMQP 0.9.1 implementation of RabbitMQ and JMS 1.1.
Right now does not exist any lib to connect to AMQP Broker using JMS, except the Apache Qpid client (0.30). It was a great help.
2) Improve existing code of WSO2 JMS MessageStore.
I added a new second JMS Destination, the first one was to consume and the another one was to produce messages. In this manner I got to connect to RabbitMQ using pre-configured Exchanges, Routing Keys and Queues.
I hope that is useful.
Regards.