Does Confluent by default provides this JMSSourceConnector for Kafka topic.
Or we need to write custom connector for this?
I dont see any documentation on Confluent page on this.
Currently Confluent doesn't provide source connector for JMS. Please find below link for number of connectors available in Kafka Connect.
http://www.confluent.io/product/connectors/
But developers can develop custom connectors for Kafka Connect. Please see the below URL for more information on how to write custom connectors for Kafka Connect.
http://docs.confluent.io/3.0.1/connect/devguide.html
Related
We are trying to use debezium kafka connector to capture postgres changelog. Based on the online tutorial, we need to start a debezium server, and send a POST http request to debezium api to deploy the connector. We want to capture the kafka connector configuration in a code repository, and have automated debezium based kafka connector deployment. What is debezium best practice for this kind of deployment automation?
I'd like to use Kakfa Connect to move JSON messages from Kafka to HDFS and then Impala, only using OpenSource libs.
I was trying to understand if I can use the Confluent Sink library for Kakfa Connect, without the need to use the entire Confluent distribution.
Are there are other and/or better options to achieve this?
The Kafka Connect HDFS 2 Sink is available under the Confluent Community Licence. It is a plugin for Apache Kafka; you do not have to run Confluent Platform to use it.
I've seen brokers like RabbitMQ or Apache Pulsar provide a Websocket API to connect directly your browser to the broker.
AFAIK, I've not seen the same for Apache Kafka. You have to implement yourself an intermediary Websocket server.
Why Confluent, which owns Apache Kafka development, does not provide an out-of-the-box websocket API like Rabbitmq or Pulsar ?
Confluent does not own Apache Kafka development. Apache Kafka is a project owned by the Apache Software Foundation (ASF).
There are several examples of how to use WebSockets with Apache Kafka:
https://www.confluent.io/blog/data-stream-processing-with-kafka-streams-bitrock-and-confluent/
https://dev.to/victorgil/kafka-websockets-angular-event-driven-microservices-all-the-way-to-the-frontend-12aa
https://medium.com/swlh/angular-spring-boot-kafka-how-to-stream-realtime-data-the-reactive-way-510a0f1e5881
Your question seems somewhat a rhetorical one, but if you actually would like to see Websockets as part of Apache Kafka then the first step would be to raise a Kafka Improvement Proposal (KIP) for discussion in the community. There is a dev mailing list where you can get further guidance on the process on contributing code.
There is already an open source connecting Kafka with Websocket. https://github.com/b/kafka-websocket
If you have requirements connecting to Kafka from a browser, I suggest to consider to browserfiy this http://github.com/confluentinc/kafka-rest-node
There is a kafka connector to Ably hosted on the Confluent Hub. Ably is essentially a serverless WebSocket option (with pub/sub and message queues). You will also find a kafka rule on Ably's website.
I would like to use an open source version of kafka-connect instead of the confluent one as it appears that confluent cli is not for production and only for dev. I would like to be able to listen to changes on mysql database on aws ec2. Can someone point me in the right direction.
Kafka Connect is part of Apache Kafka. Period. If you want to use Kafka Connect you can do so with any modern distribution of Apache Kafka.
You then need a connector plugin to use with Kafka Connect, specific to your source technology. For integrating with a database there are various considerations, and available for MySQL you specifically have:
Kafka Connect JDBC - see it in action here
Debezium - see it in action here
The Confluent CLI is just a tool for helping manage and deploy Confluent Platform on developer machines. Confluent Platform itself is widely used in production.
Question for Kafka experts:
Anyone knows how to get worker's config setting 'bootstrap.servers' either from SinkConnector or SinkTask? Or how to get Kafka cluster information from connector?
Thank you
AFAIK, the Connect API doesn't provide this information right now.
If you need this functionality, perhaps your best bet is to open a JIRA on the Apache Kafka project and explain the use-case (i.e. what are you planning on doing this information).