Kafka to Elasticsearch connector not able to start - elasticsearch

Question

Elasticsearch -> elasticsearch-7.6.0
I am trying to move data from Kafka topic to Elasticsearch by using Elasticsearch Sink connector.
But starting the connector is showing below error
2022-05-28 09:39:27,864] ERROR [SINK_ELASTIC_TEST_01|task-0] Failed to create mapping for index users.user_details3 with schema Schema{STRING} due to 'ElasticsearchStatusException[Elasticsearch exception [type=mapper_parsing_exception, reason=Root mapping definition has unsupported parameters: [type : text] [fields : {keyword={ignore_above=256, type=keyword}}]]]' after 6 attempt(s) (io.confluent.connect.elasticsearch.RetryUtil:164)
ElasticsearchStatusException[Elasticsearch exception [type=mapper_parsing_exception, reason=Root mapping definition has unsupported parameters: [type : text] [fields : {keyword={ignore_above=256, type=keyword}}]]]
at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:178)
at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2484)
at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2461)
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2184)
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154)
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118)
at org.elasticsearch.client.IndicesClient.putMapping(IndicesClient.java:440)
at io.confluent.connect.elasticsearch.ElasticsearchClient.lambda$createMapping$3(ElasticsearchClient.java:238)
at io.confluent.connect.elasticsearch.RetryUtil.callWithRetries(RetryUtil.java:158)
at io.confluent.connect.elasticsearch.RetryUtil.callWithRetries(RetryUtil.java:119)
at io.confluent.connect.elasticsearch.ElasticsearchClient.callWithRetries(ElasticsearchClient.java:426)
at io.confluent.connect.elasticsearch.ElasticsearchClient.createMapping(ElasticsearchClient.java:236)
at io.confluent.connect.elasticsearch.ElasticsearchSinkTask.checkMapping(ElasticsearchSinkTask.java:151)
at io.confluent.connect.elasticsearch.ElasticsearchSinkTask.tryWriteRecord(ElasticsearchSinkTask.java:294)
at io.confluent.connect.elasticsearch.ElasticsearchSinkTask.put(ElasticsearchSinkTask.java:118)
at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:584)
at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:334)
at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:235)
at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:204)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:200)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:255)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Suppressed: org.elasticsearch.client.ResponseException: method [PUT], host [http://localhost:9200], URI [/users.user_details3/_mapping?master_timeout=30s&timeout=30s], status line [HTTP/1.1 400 Bad Request]
Topic data
rowtime: 2022/05/27 13:46:48.136 Z, key: {"schema":{"type":"string","optional":false},"payload":"{"_id": {"_data": "826290D642000000022B022C0100296E5A100429BEBD4C4B7F4C0BA80B881299B5FF9246645F696400646290D642F23DB1FBE13668180004"}}"}, value: {"schema":{"type":"string","optional":false},"payload":"{"_id": {"_data": "826290D642000000022B022C0100296E5A100429BEBD4C4B7F4C0BA80B881299B5FF9246645F696400646290D642F23DB1FBE13668180004"}, "operationType": "insert", "clusterTime": {"$timestamp": {"t": 1653659202, "i": 2}}, "fullDocument": {"_id": {"$oid": "6290d642f23db1fbe1366818"}, "userid": 1.0, "name": "Gaurav"}, "ns": {"db": "users", "coll": "user_details3"}, "documentKey": {"_id": {"$oid": "6290d642f23db1fbe1366818"}}}"}, partition: 0
rowtime: 2022/05/27 13:47:06.142 Z, key: {"schema":{"type":"string","optional":false},"payload":"{"_id": {"_data": "826290D654000000012B022C0100296E5A100429BEBD4C4B7F4C0BA80B881299B5FF9246645F696400646290D654F23DB1FBE13668190004"}}"}, value: {"schema":{"type":"string","optional":false},"payload":"{"_id": {"_data": "826290D654000000012B022C0100296E5A100429BEBD4C4B7F4C0BA80B881299B5FF9246645F696400646290D654F23DB1FBE13668190004"}, "operationType": "insert", "clusterTime": {"$timestamp": {"t": 1653659220, "i": 1}}, "fullDocument": {"_id": {"$oid": "6290d654f23db1fbe1366819"}, "userid": 1.0, "name": "Gaurav"}, "ns": {"db": "users", "coll": "user_details3"}, "documentKey": {"_id": {"$oid": "6290d654f23db1fbe1366819"}}}"}, partition: 0
rowtime: 2022/05/27 13:47:24.149 Z, key: {"schema":{"type":"string","optional":false},"payload":"{"_id": {"_data": "826290D668000000012B022C0100296E5A100429BEBD4C4B7F4C0BA80B881299B5FF9246645F696400646290D668F23DB1FBE136681A0004"}}"}, value: {"schema":{"type":"string","optional":false},"payload":"{"_id": {"_data": "826290D668000000012B022C0100296E5A100429BEBD4C4B7F4C0BA80B881299B5FF9246645F696400646290D668F23DB1FBE136681A0004"}, "operationType": "insert", "clusterTime": {"$timestamp": {"t": 1653659240, "i": 1}}, "fullDocument": {"_id": {"$oid": "6290d668f23db1fbe136681a"}, "userid": 1.0, "name": "Gaurav"}, "ns": {"db": "users", "coll": "user_details3"}, "documentKey": {"_id": {"$oid": "6290d668f23db1fbe136681a"}}}"}, partition: 0
rowtime: 2022/05/27 13:48:00.156 Z, key: {"schema":{"type":"string","optional":false},"payload":"{"_id": {"_data": "826290D68A000000012B022C0100296E5A100429BEBD4C4B7F4C0BA80B881299B5FF9246645F696400646290D68AF23DB1FBE136681B0004"}}"}, value: {"schema":{"type":"string","optional":false},"payload":"{"_id": {"_data": "826290D68A000000012B022C0100296E5A100429BEBD4C4B7F4C0BA80B881299B5FF9246645F696400646290D68AF23DB1FBE136681B0004"}, "operationType": "insert", "clusterTime": {"$timestamp": {"t": 1653659274, "i": 1}}, "fullDocument": {"_id": {"$oid": "6290d68af23db1fbe136681b"}, "userid": 1.0, "name": "Gaurav"}, "ns": {"db": "users", "coll": "user_details3"}, "documentKey": {"_id": {"$oid": "6290d68af23db1fbe136681b"}}}"}, partition: 0
rowtime: 2022/05/27 13:50:00.182 Z, key: {"schema":{"type":"string","optional":false},"payload":"{"_id": {"_data": "826290D706000000012B022C0100296E5A100429BEBD4C4B7F4C0BA80B881299B5FF9246645F696400646290D706F23DB1FBE136681C0004"}}"}, value: {"schema":{"type":"string","optional":false},"payload":"{"_id": {"_data": "826290D706000000012B022C0100296E5A100429BEBD4C4B7F4C0BA80B881299B5FF9246645F696400646290D706F23DB1FBE136681C0004"}, "operationType": "insert", "clusterTime": {"$timestamp": {"t": 1653659398, "i": 1}}, "fullDocument": {"_id": {"$oid": "6290d706f23db1fbe136681c"}, "userid": 2.0, "name": "Gaurav"}, "ns": {"db": "users", "coll": "user_details3"}, "documentKey": {"_id": {"$oid": "6290d706f23db1fbe136681c"}}}"}, partition: 0
ElasticSearchSink Configuration
{
"name": "SINK_ELASTIC_TEST_01",
"config": {
"type.name": "kafka-connect",
"name": "SINK_ELASTIC_TEST_01",
"connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"errors.log.enable": "true",
"errors.log.include.messages": "true",
"topics": "users.user_details3",
"connection.url": "http://localhost:9200",
"connection.username": "",
"connection.password": "",
"key.ignore": "true",
"schema.ignore": "false"
}
}
Please see I am creating topic from MongoDB source connector , so on a high level I am trying to achieve this
MongoDB --> MongoDBSourceConnector--> Kafka --> ElasticSearchSinkConnector --> ElasticSearch
Please find the MongoDBSourceConnector Configuration for reference
{
"name": "source-mongodb-kafka-stream",
"config": {
"name": "source-mongodb-kafka-stream",
"connector.class": "com.mongodb.kafka.connect.MongoSourceConnector",
"tasks.max": "1",
"key.converter": "org.apache.kafka.connect.json.JsonConverter",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"connection.uri": "mongodb://localhost:27017",
"database": "users",
"collection": "",
"topic.prefix": ""
}
}
Update 06-03-2022 after OneCricketeer comments
I have created an index in Elasticsearch with the same name as topic
PUT /users_generic/_doc/1
{
"id":{
"_data":"826290D706000000012B022C0100296E5A100429BEBD4C4B7F4C0BA80B881299B5FF9246645F696400646290D706F23DB1FBE136681C0004"
},
"operationType":"insert",
"clusterTime":{
"$timestamp":{
"t":1653659398,
"i":1
}
},
"fullDocument":{
"id":{
"$oid":"6290d706f23db1fbe136681c"
},
"userid":2.0,
"name":"Gaurav"
},
"ns":{
"db":"users",
"coll":"user_details3"
},
"documentKey":{
"id":{
"$oid":"6290d706f23db1fbe136681c"
}
}
}
With above step , mapping is been created , now when I again executed the connector with same parameters as above , it throws me a different effort
2-06-03 12:39:16,691] ERROR [SINK_ELASTIC_TEST_01|task-0] Failed to execute bulk request due to 'org.elasticsearch.common.compress.NotXContentException: Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes' after 6 attempt(s) (io.confluent.connect.elasticsearch.RetryUtil:164)
org.elasticsearch.common.compress.NotXContentException: Compressor detection can only be called on some xcontent bytes