heroku data:connectors:create command only has "Tables to include" and "Columns to exclude" options. But I need debezium connector configured with transforms, converters, and predicates. Is there some way to specify a custom configuration file for connector? Or my best option is to not use a heroku-provided connector, but instead manually run a debezium container?
Related
We are trying to use debezium kafka connector to capture postgres changelog. Based on the online tutorial, we need to start a debezium server, and send a POST http request to debezium api to deploy the connector. We want to capture the kafka connector configuration in a code repository, and have automated debezium based kafka connector deployment. What is debezium best practice for this kind of deployment automation?
I am using a Debezium mysql connector, I want to replace the existing task with new task.
Tasks are all configured at the worker level.
You would have to send your configuration to the PUT <connector>/config endpoint.
I am trying to export data from Kafka to Oracle db. I've searched related questions and web but could not understand that we need a platform (confluent etc.. ) or not. I'd been read the link below but it's not clear enough.
https://docs.confluent.io/3.2.2/connect/connect-jdbc/docs/sink_connector.html
So, what we actually need to export data without 3rd party platform? Thanks in advance.
It's not clear what you mean by "third-party" here
What you linked to is Kafka Connect, which is Apache 2.0 Licensed and open source.
Kafka Connect is a plugin ecosystem, you install connectors individually, written by anyone, or write your own, just like any other Java dependency (i.e. a third-party)
The JDBC connector just happens to be maintained by Confluent. and you can configure the Confluent Hub CLI
to install within any Kafka Connect distribution (or use Kafka Connect Docker images from Confluent)
Alternatively, you use Apache Spark, Flink, Nifi, and many other Kafka Consumer libraries to read data and then start an Oracle transaction per record batch
Or you can explore non-JVM kafka libraries as well and use a language you're more familiar with doing Oracle operations with
I would like to use an open source version of kafka-connect instead of the confluent one as it appears that confluent cli is not for production and only for dev. I would like to be able to listen to changes on mysql database on aws ec2. Can someone point me in the right direction.
Kafka Connect is part of Apache Kafka. Period. If you want to use Kafka Connect you can do so with any modern distribution of Apache Kafka.
You then need a connector plugin to use with Kafka Connect, specific to your source technology. For integrating with a database there are various considerations, and available for MySQL you specifically have:
Kafka Connect JDBC - see it in action here
Debezium - see it in action here
The Confluent CLI is just a tool for helping manage and deploy Confluent Platform on developer machines. Confluent Platform itself is widely used in production.
I am trying to connect my MirroMaker Kafka 0.9 to the Kafka Brokers 0.10 (IBM Message Hub) without success. The links I have followed are the followings, but they are mostly for Kafka clients 0.10:
https://console.bluemix.net/docs/services/MessageHub/messagehub050.html#kafka_using https://console.bluemix.net/docs/services/MessageHub/messagehub063.html#kafka_connect
Do you know the steps for Kafka clients 0.9 and how to use the MessageHubLoginModule and the jaas creation?
UPDATE
After different tests, the solution works correctly.
In order to connect to the IBM message hub with a cloudera mirror maker you must set within cloudera manager the property Source Kafka Cluster's Security Protocol: source.security.protocol as PLAINTEXT and pass the following properties as Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for mirror_maker_consumers.properties:
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
ssl.protocol=TLSv1.2
ssl.enabled.protocols=TLSv1.2
ssl.endpoint.identification.algorithm=HTTPS
It worked for me.
First you should not be building a new Message Hub application using Kafka 0.9.
We've deprecated the custom login module 0.9 requires and our newer clusters won't support it. You should be using a Kafka client >= 0.10.2 as they properly support Sasl Plain authentication which is required by Message Hub. The newer Kafka clients offer many more features and are just better.
In case you're absolutely stuck with 0.9, you need:
The following properties set:
security.protocol=SASL_SSL
ssl.protocol=TLSv1.2
ssl.enabled.protocols=TLSv1.2
A JAAS file containing:
KafkaClient {
com.ibm.messagehub.login.MessageHubLoginModule required
serviceName="kafka"
username="<USERNAME>"
password="<PASSWORD>";
};
The custom login module JAR in the path:
The file is available on Github: https://github.com/ibm-messaging/message-hub-samples/blob/master/kafka-0.9/message-hub-login-library/messagehub.login-1.0.0.jar
The java.security.auth.login.config Java property set:
It needs to point to your JAAS file and can be either:
in the java command line using -Djava.security.auth.login.config=<PATH TO JAAS> or
programmatically using System.setProperty("java.security.auth.login.config", "<PATH TO JAAS>");