Question for Kafka experts:
Anyone knows how to get worker's config setting 'bootstrap.servers' either from SinkConnector or SinkTask? Or how to get Kafka cluster information from connector?
Thank you
AFAIK, the Connect API doesn't provide this information right now.
If you need this functionality, perhaps your best bet is to open a JIRA on the Apache Kafka project and explain the use-case (i.e. what are you planning on doing this information).
Related
In one of our spring boot applications I'm developing a retry mechanism. Below summarized what this new feature needs to do:
If the kafka cluster for whatever reason is not available the application should keep running because records should be inserted in the database.
When the kafka cluster is available again a producer should send all newly inserted records to a topic.
What I'm looking for is an event which tells me kafka is back up and running. So far I'm unable to find something like this. What I did find are forums telling this is not supported. I was wondering if somebody has experience how to implement this.
If you have any questions for me to clarify please let me know.
We are using spring-kafka 2.8.2
Kind regards,
Josip
I've seen brokers like RabbitMQ or Apache Pulsar provide a Websocket API to connect directly your browser to the broker.
AFAIK, I've not seen the same for Apache Kafka. You have to implement yourself an intermediary Websocket server.
Why Confluent, which owns Apache Kafka development, does not provide an out-of-the-box websocket API like Rabbitmq or Pulsar ?
Confluent does not own Apache Kafka development. Apache Kafka is a project owned by the Apache Software Foundation (ASF).
There are several examples of how to use WebSockets with Apache Kafka:
https://www.confluent.io/blog/data-stream-processing-with-kafka-streams-bitrock-and-confluent/
https://dev.to/victorgil/kafka-websockets-angular-event-driven-microservices-all-the-way-to-the-frontend-12aa
https://medium.com/swlh/angular-spring-boot-kafka-how-to-stream-realtime-data-the-reactive-way-510a0f1e5881
Your question seems somewhat a rhetorical one, but if you actually would like to see Websockets as part of Apache Kafka then the first step would be to raise a Kafka Improvement Proposal (KIP) for discussion in the community. There is a dev mailing list where you can get further guidance on the process on contributing code.
There is already an open source connecting Kafka with Websocket. https://github.com/b/kafka-websocket
If you have requirements connecting to Kafka from a browser, I suggest to consider to browserfiy this http://github.com/confluentinc/kafka-rest-node
There is a kafka connector to Ably hosted on the Confluent Hub. Ably is essentially a serverless WebSocket option (with pub/sub and message queues). You will also find a kafka rule on Ably's website.
I am trying to export data from Kafka to Oracle db. I've searched related questions and web but could not understand that we need a platform (confluent etc.. ) or not. I'd been read the link below but it's not clear enough.
https://docs.confluent.io/3.2.2/connect/connect-jdbc/docs/sink_connector.html
So, what we actually need to export data without 3rd party platform? Thanks in advance.
It's not clear what you mean by "third-party" here
What you linked to is Kafka Connect, which is Apache 2.0 Licensed and open source.
Kafka Connect is a plugin ecosystem, you install connectors individually, written by anyone, or write your own, just like any other Java dependency (i.e. a third-party)
The JDBC connector just happens to be maintained by Confluent. and you can configure the Confluent Hub CLI
to install within any Kafka Connect distribution (or use Kafka Connect Docker images from Confluent)
Alternatively, you use Apache Spark, Flink, Nifi, and many other Kafka Consumer libraries to read data and then start an Oracle transaction per record batch
Or you can explore non-JVM kafka libraries as well and use a language you're more familiar with doing Oracle operations with
Does Confluent by default provides this JMSSourceConnector for Kafka topic.
Or we need to write custom connector for this?
I dont see any documentation on Confluent page on this.
Currently Confluent doesn't provide source connector for JMS. Please find below link for number of connectors available in Kafka Connect.
http://www.confluent.io/product/connectors/
But developers can develop custom connectors for Kafka Connect. Please see the below URL for more information on how to write custom connectors for Kafka Connect.
http://docs.confluent.io/3.0.1/connect/devguide.html
I'm looking to consume from Kafka and save data into Hadoop and Elasticsearch.
I've seen 2 ways of doing this currently: using Filebeat to consume from Kafka and send it to ES and using Kafka-Connect framework. There is a Kafka-Connect-HDFS and Kafka-Connect-Elasticsearch module.
I'm not sure which one to use to send streaming data. Though I think that if I want at some point to take data from Kafka and place it into Cassandra I can use a Kafka-Connect module for that but no such feature exists for Filebeat.
Kafka Connect can handle streaming data and is a bit more flexible. If you are just going to elastic, Filebeat is a clean integration for log sources. However, if you are going from Kafka to a number of different sinks, Kafka Connect is probably what you want. I'd recommend checking out the connector hub to see some examples of open source connectors at your disposal currently http://www.confluent.io/product/connectors/