Can kafka connect create stream directly? - apache-kafka-streams

Question

I have a scenario where I need to import an entire DB in Kafka and create in DB term some views on those table that user can query after. My requirements is to rebuild the logical model via views out of the physical models (the tables).
Hence I am wondering about the step to do that.
My ideal would be that kafka Connect create the topics which corresponds to the tables, then right after that, for me to declaratively (using KSQL) to create the Views.
While what I describe here sounds feasible at first, I have an issue with the data the structure (schema) of the data within the topics. The problem it seems is that i might have to do an extra steps but wonder if it can be avoided or is actually necessary.
More specifically, Views usually represent join on table. I imagine that if i want to do join on table, I need to have Ktable or Kstream already created, which give the structure on which to do the joins. But if Kafka connect just create topics but no Ktable or Kstream, it seems that an extra steps need to happen that automatically make those topics availables as Ktable or Kstream. At which point, i can use KSQL to create the views that will represent the physical model.
1 - Hence the question, is there a way from Kafka connect to create Kstream or Ktable automatically ?
2 - Kafka connect as the notion of schema, how does that relate to the Kstream/KTable structure (schema) and format(json/avro/delimited) ?
3 - If Kafka connect can't create Kstream and KTable directly, can KSQL operate a join on the topics that Kafka connect create, directly ? Will it be able to interpret the structure of the data in those topics (i.e. kafka connect generated schema) and perform a join on it, and make the result available as a Kstream ?
4 - If all my assumption are wrong, can someone give me the step of what my problem would entail in term of KSQL/Kafka-stream/Kafka-connect ?

Answer 1

1 - Hence the question, is there a way from Kafka connect to create Kstream or Ktable automatically ?
No, you need to do so manually. But if you're using Avro then it's just a simple statement:
CREATE STREAM foo WITH (KAFKA_TOPIC='bar', VALUE_FORMAT='AVRO');
2 - Kafka connect as the notion of schema, how does that relate to the Kstream/KTable structure (schema) and format(json/avro/delimited) ?
KSQL Stream (or Table) = Kafka Topic plus Schema.
So you have a Kafka topic (loaded by Kafka Connect, for example), and you need a schema. The best thing is just use Avro when you produce the data (e.g. from Kafka Connect), because the schema then exists in the Schema Registry and KSQL can use it automagically.
If you want to use JSON or [shudder] Delimited then you have to provide the schema in KSQL when you declare the stream/table. Instead of the above statement you'd have something like
CREATE STREAM foo (COL1 INT, COL2 VARCHAR, COL3 INT, COL4 STRUCT<S1 INT,S2 VARCHAR>)
WITH (KAFKA_TOPIC='bar_json',VALUE_FORMAT='JSON');
3 - If Kafka connect can't create Kstream and KTable directly, can KSQL operate a join on the topics that Kafka connect create, directly ?
KSQL can join streams and tables, yes. A stream/table is just a Kafka topic, with a schema.
Will it be able to interpret the structure of the data in those topics (i.e. kafka connect generated schema) and perform a join on it, and make the result available as a Kstream ?
Yes. The schema is provided by Kafka Connect and if you're using Avro it 'just works'. If using JSON you need to manually enter the schema as shown above.
The output of a KSQL join is a Kafka topic, for example
CREATE STREAM A WITH (KAFKA_TOPIC='A', VALUE_FORMAT='AVRO');
CREATE TABLE B WITH (KAFKA_TOPIC='B', VALUE_FORMAT='AVRO', KEY='ID');
CREATE STREAM foobar AS
SELECT A.*, B.* FROM
A LEFT OUTER JOIN B ON A.ID = B.ID;
4 - If all my assumption are wrong, can someone give me the step of what my problem would entail in term of KSQL/Kafka-stream/Kafka-connect ?
I don't think your assumptions are wrong. Use Kafka Connect + KSQL, and use Avro :)
These references might help you further:
http://rmoff.dev/vienna19-ksql-intro
http://go.rmoff.net/devoxx18-build-streaming-pipeline