Elastic search sink connector need to read multiple topics nearly more than 1000 from kafka needs to write elastic search - apache-kafka-connect

Question

I am new to kafka connect.
I have a requirement, where it needs to read topics from kafka dynamically and needs to write to elastic search.
Is there any way to achieve it?
Is there any way to use like topics with patterns app*,test*(actual topic will apps-logging, app-location1, app-service)(test-app,test-app2)
Sample configuration which I used, it's writing index location.service-2020.06.11, If I want include other topics with wild card as mentioned above( I don't want write each topic name) how can I achieve it.
curl -X POST http://localhost:8083/connectors -H "Content-Type: application/json" -d '{
"name" : "kafka-elastic-test-01",
"config" : {
"connector.class" : "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
"connection.url" : "https://localhost:9200",
"connection.username": "admin",
"connection.password":"******",
"key.converter" : "org.apache.kafka.connect.json.JsonConverter",
"type.name" : "_doc",
"topics.regex": "location.service*",
"key.ignore" : "true",
"schema.ignore" : "true",
"transforms": "TimestampRouter",
"transforms.TimestampRouter.type": "org.apache.kafka.connect.transforms.TimestampRouter",
"transforms.TimestampRouter.topic.format": "${topic}-${timestamp}",
"transforms.TimestampRouter.timestamp.format": "yyyy.MM.dd"
}
}'
Edit on June 9 2020
Thanks for the reply #Iskuskov Alexander
I tried your suggestion, Here is the output. Any of your suggestions are welcome.
curl -X POST http://localhost:8083/connectors -H "Content-Type: application/json" -d '{
"name" : "kafka-elastic-test-03",
"config" : {
"connector.class" : "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
"connection.url" : "http://localhost:9200",
"connection.username": "admin",
"connection.password":"******",
"key.converter" : "org.apache.kafka.connect.json.JsonConverter",
"type.name" : "_doc",
"topics.regex": "(app|test).*",
"key.ignore" : "true",
"schema.ignore" : "true",
"transforms": "TimestampRouter",
"transforms.TimestampRouter.type": "org.apache.kafka.connect.transforms.TimestampRouter",
"transforms.TimestampRouter.topic.format": "${topic}-${timestamp}",
"transforms.TimestampRouter.timestamp.format": "yyyy.MM.dd"
}
}'
curl -w '\n' 'http://localhost:8083/connectors/kafka-elastic-test-03/status' {
"name": "kafka-elastic-test-03",
"connector": {
"state": "RUNNING",
"worker_id":"xxxx:8083"
},
"tasks": [{
"id":0,
"state":"FAILED",
"worker_id":"xxxx:8083",
"trace":
"org.apache.kafka.connect.errors.ConnectException: Tolerance exceeded in error handler\n\t
at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:178)\n\t
at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:104)\n\t
at org.apache.kafka.connect.runtime.WorkerSinkTask.convertAndTransformRecord(WorkerSinkTask.java:488)\n\t
at org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:465)\n\t
at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:321)\n\t
at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:224)\n\t
at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:192)\n\t
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:177)\n\t
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:227)\n\t
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\t
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\t
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\t
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\t
at java.base/java.lang.Thread.run(Thread.java:834)\n
Caused by: org.apache.kafka.connect.errors.DataException: Converting byte[] to Kafka Connect data failed due to serialization error: \n\t
at org.apache.kafka.connect.json.JsonConverter.toConnectData(JsonConverter.java:355)\n\t
at org.apache.kafka.connect.storage.Converter.toConnectData(Converter.java:86)\n\t
at org.apache.kafka.connect.runtime.WorkerSinkTask.lambda$convertAndTransformRecord$2(WorkerSinkTask.java:488)\n\t
at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndRetry(RetryWithToleranceOperator.java:128)\n\t
at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:162)\n\t
... 13 more\n
Caused by: org.apache.kafka.common.errors.SerializationException: com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (byte[])\"{\"; line: 1, column: 1])\n
at [Source: (byte[])\"{\"; line: 1, column: 3]\n
Caused by: com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (byte[])\"{\"; line: 1, column: 1])\n
at [Source: (byte[])\"{\"; line: 1, column: 3]\n\t
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportInvalidEOF(ParserMinimalBase.java:618)\n\t
at com.fasterxml.jackson.core.base.ParserBase._handleEOF(ParserBase.java:485)\n\t
at com.fasterxml.jackson.core.base.ParserBase._eofAsNextChar(ParserBase.java:497)\n\t
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._skipWSOrEnd(UTF8StreamJsonParser.java:2933)\n\t
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextFieldName(UTF8StreamJsonParser.java:964)\n\t
at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:246)\n\t
at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:68)\n\t
at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:15)\n\t
at com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4057)\n\t
at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2572)\n\t
at org.apache.kafka.connect.json.JsonDeserializer.deserialize(JsonDeserializer.java:58)\n\t
at org.apache.kafka.connect.json.JsonConverter.toConnectData(JsonConverter.java:353)\n\t
at org.apache.kafka.connect.storage.Converter.toConnectData(Converter.java:86)\n\t
at org.apache.kafka.connect.runtime.WorkerSinkTask.lambda$convertAndTransformRecord$2(WorkerSinkTask.java:488)\n\t
at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndRetry(RetryWithToleranceOperator.java:128)\n\t
at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:162)\n\t
at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:104)\n\t
at org.apache.kafka.connect.runtime.WorkerSinkTask.convertAndTransformRecord(WorkerSinkTask.java:488)\n\t
at org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:465)\n\t
at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:321)\n\t
at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:224)\n\t
at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:192)\n\t
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:177)\n\t
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:227)\n\t
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\t
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\t
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\t
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\t
at java.base/java.lang.Thread.run(Thread.java:834)\n"
}],
"type": "sink"
}

Answer 1

Yes, you can use topics.regex parameter of Kafka Connect Sink Configurations.
In your case it could be looks like: "topics.regex": "(app|test).*"