Kafka JDBC Sink Connector - Oracle - oracle

I am creating jdbc sink connector(confluent) to oracle, first time. Below is the connector config. It is not creating and loading the table, but offset is being increased as per log and no error thrown in the log. Please suggest what could be the issue.
Producing through java, 5 sample records. I have tried curl to check the status, it is running.
{
"name": "ora_sink_task",
"config": {
"connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
"connection.url": "jdbc:oracle:thin:#host:port/servicename",
"connection.user": "user",
"connection.password": "password",
"topics": "connecttest",
"tasks.max": "1",
"table.name.format": "member_cbdt_sink1",
"value.converter":"io.confluent.connect.avro.AvroConverter",
"value.converter.schema.registry.url":"http://localhost:8081",
"auto.create": "true"
}
}
Connector log:
[2019-09-09 00:34:23,832] INFO Checking Oracle dialect for existence
of table "member_cbdt_sink1"
(io.confluent.connect.jdbc.dialect.OracleDatabaseDialect:492)
[2019-09-09 00:34:23,834] INFO Using Oracle dialect table
"member_cbdt_sink1" absent
(io.confluent.connect.jdbc.dialect.OracleDatabaseDialect:500)
[2019-09-09 00:34:23,846] INFO Checking Oracle dialect for existence
of table "member_cbdt_sink1"
(io.confluent.connect.jdbc.dialect.OracleDatabaseDialect:492)
[2019-09-09 00:34:23,849] INFO Using Oracle dialect table
"member_cbdt_sink1" present
(io.confluent.connect.jdbc.dialect.OracleDatabaseDialect:500)
[2019-09-09 00:34:24,037] INFO Setting metadata for table
"member_cbdt_sink1" to Table{name='"member_cbdt_sink1"',
columns=[Column{'first_name', isPrimaryKey=false, allowsNull=false,
sqlType=CLOB}, Column{'height', isPrimaryKey=false, allowsNull=false,
sqlType=BINARY_FLOAT}, Column{'last_name', isPrimaryKey=false,
allowsNull=false, sqlType=CLOB}, Column{'age', isPrimaryKey=false,
allowsNull=false, sqlType=NUMBER}, Column{'automated_email',
isPrimaryKey=false, allowsNull=true, sqlType=NUMBER}, Column{'weight',
isPrimaryKey=false, allowsNull=false, sqlType=BINARY_FLOAT}]}
(io.confluent.connect.jdbc.util.TableDefinitions:65) [2019-09-09
00:35:13,775] INFO WorkerSinkTask{id=ora_sink_task-0} Committing
offsets asynchronously using sequence number 1:
{connecttest-0=OffsetAndMetadata{offset=55, metadata=''}}
(org.apache.kafka.connect.runtime.WorkerSinkTask:345) [2019-09-09
01:03:13,775] INFO WorkerSinkTask{id=ora_sink_task-0} Committing
offsets asynchronously using sequence number 29:
{connecttest-0=OffsetAndMetadata{offset=60, metadata=''}}
(org.apache.kafka.connect.runtime.WorkerSinkTask:345)

It worked by changing the value of table.name.format from lowercase to uppercase as Oracle looks for uppercase seems.

Related

Kafka HDFS Sink Connector error.[Top level type must be STRUCT..]

I'm testing the 2.7 version of Kafka with Kafka connect,
and i'm facing with problem that i don't understand.
I started distributed connector first with configuration like below.
bootstrap.servers=..:9092,...:9092, ...
group.id=kafka-connect-test
key.converter=org.apache.kafka.connect.converters.ByteArrayConverter
value.converter=org.apache.kafka.connect.converters.ByteArrayConverter
... some internal topic configuration
plugin.path=<plugin path>
This connector serviced with 8083 port.
And I want to write ORC format data with snappy codec at HDFS.
so i made new Kafka HDFS connector with REST API with json data like below.
and i don't use schema-registry.
curl -X POST <connector url:8083> \
-H Accept: application/json \
-H Content-Type: application/json \
-d
{
"name": "hdfs-sinkconnect-test",
"config": {
"connector.class": "io.confluent.connect.hdfs.HdfsSinkConnector",
"store.url": "hdfs:~",
"hadoop.conf.dir": "<my hadoop.conf dir>",
"hadoop.home": "<hadoop home dir>",
"tasks.max": "5",
"key.deserializer": "org.apache.kafka.common.serialization.StringDeserializer",
"value.deserializer": "org.apache.kafka.common.serialization.ByteArrayDeserializer",
"format.class": "io.confluent.connect.hdfs.orc.OrcFormat",
"flush.size": 1000,
"avro.codec": "snappy",
"topics": "<topic name>",
"topics.dir": "/tmp/connect-logs",
"partitioner.class": "io.confluent.connect.storage.partitioner.TimeBasedPartitioner",
"locale": "ko_KR",
"timezone": "Asia/Seoul",
"partition.duration.ms": "3600000",
"path.format": "'hour'=YYYYMMddHH/"
}
}
Then i have error message like this.
# connectDistributed.out
[2021-06-28 17:14:11,596] ERROR Exception on topic partition <topic name>-<partition number>: (io.confluent.connect.hdfs.TopicPartitionWriter:409)
org.apache.kafka.connect.errors.ConnectException: Top level type must be STRUCT but was bytes
at io.confluent.connect.hdfs.orc.OrcRecordWriterProvider$1.write(OrcRecordWriterProvider.java:98)
at io.confluent.connect.hdfs.TopicPartitionWriter.writeRecord(TopicPartitionWriter.java:742)
at io.confluent.connect.hdfs.TopicPartitionWriter.write(TopicPartitionWriter.java:385)
at io.confluent.connect.hdfs.DataWriter.write(DataWriter.java:333)
at io.confluent.connect.hdfs.HdfsSinkTask.put(HdfsSinkTask.java:126)
at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:586)
at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:329)
at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:232)
at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:201)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:185)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:234)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
I think this error message have something to do with Schema Information.
Is Schema Registry essential for Kafka Connector?
Any ideas or solutions to solve this error messsage? Thanks.
Writing ORC files requires a Struct type.
The options provided by Confluent include plain JSON, JSONSchema, Avro, or Protobuf. The only option that doesn't require the Registry is the plain JsonConverter
Note that key.deserializer and value.deserializer are not valid Connect properties. You need to refer to your key.converter and value.converter properties instead
If you're not willing to modify the converter, you can attempt to use a HoistField transformer to create a Struct, and this will create an ORC file with a schema of only one field

Camel aws-s3 Source Connector Error - How should the config be changed

I am working on defining a Camel S3 Source connector with our Confluent (5.5.1) installation. After creating the connector and checking status as "RUNNING", I upload a file to my S3 bucket. Even if I do ls for the bucket, it is empty, which indicates the file is processed and deleted. But, I do not see messages in the topic. I am basically following this example trying a simple 4 line file, but instead of standalone kafka, doing it on a confluent cluster.
This is my configuration
{
"name": "CamelAWSS3SourceConnector",
"connector.class": "org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector",
"bootstrap.servers": "broker1-dev:9092,broker2-dev:9092,broker3-dev:9092",
"sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"user\" password=\"password\";",
"security.protocol": "SASL_SSL",
"ssl.truststore.location": "/config/client.truststore.jks",
"ssl.truststore.password": "password",
"ssl.keystore.location": "/config/client.keystore.jks",
"ssl.keystore.password": "password",
"ssl.key.password": "password",
"errors.log.enable": "true",
"errors.log.include.messages": "true",
"errors.tolerance": "all",
"offset.flush.timeout.ms": "60000",
"offset.flush.interval.ms": "10000",
"max.request.size": "10485760",
"flush.size": "1",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter",
"camel.source.maxPollDuration": "10000",
"topics": "TEST-CAMEL-S3-SOURCE-POC",
"camel.source.path.bucketNameOrArn": "arn:aws:s3:::my-bucket",
"camel.component.aws-s3.region": "US_EAST_1",
"tasks.max": "1",
"camel.source.endpoint.useIAMCredentials": "true",
"camel.source.endpoint.autocloseBody": "true"
}
And I see these errors in the logs
[2020-12-23 09:05:01,876] ERROR WorkerSourceTask{id=CamelAWSS3SourceConnector-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:448)
[2020-12-23 09:05:01,876] ERROR WorkerSourceTask{id=CamelAWSS3SourceConnector-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:116)
[2020-12-23 09:20:58,685] DEBUG [Worker clientId=connect-1, groupId=connect-cluster] Received successful Heartbeat response (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1045)
[2020-12-23 09:20:58,688] DEBUG WorkerSourceTask{id=CamelAWSS3SourceConnector-0} Committing offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:111)
[2020-12-23 09:20:58,688] INFO WorkerSourceTask{id=CamelAWSS3SourceConnector-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:426)
[2020-12-23 09:20:58,688] INFO WorkerSourceTask{id=CamelAWSS3SourceConnector-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:443)
And if I do a curl request for the status of the connector, I get this error for the status
trace: org.apache.kafka.connect.errors.ConnectException: OffsetStorageWriter is already flushing
at org.apache.kafka.connect.storage.OffsetStorageWriter.beginFlush(OffsetStorageWriter.java:111)
at org.apache.kafka.connect.runtime.WorkerSourceTask.commitOffsets(WorkerSourceTask.java:438)
at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:257)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:184)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:234)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
I saw below solution in a couple of links, but that also didn't help. It suggested to add below keys to the config
"offset.flush.timeout.ms": "60000",
"offset.flush.interval.ms": "10000",
"max.request.size": "10485760",
Thank you
UPDATE
I cut the config to minimal, but still get the same error
{
"connector.class": "org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter",
"camel.source.maxPollDuration": "10000",
"topics": "TEST-S3-SOURCE-MINIMAL-POC",
"camel.source.path.bucketNameOrArn": "pruvpcaws003-np-use1-push-json-poc",
"camel.component.aws-s3.region": "US_EAST_1",
"tasks.max": "1",
"camel.source.endpoint.useIAMCredentials": "true",
"camel.source.endpoint.autocloseBody": "true"
}
Still get the same error
trace: org.apache.kafka.connect.errors.ConnectException: OffsetStorageWriter is already flushing
at org.apache.kafka.connect.storage.OffsetStorageWriter.beginFlush(OffsetStorageWriter.java:111)
at org.apache.kafka.connect.runtime.WorkerSourceTask.commitOffsets(WorkerSourceTask.java:438)
at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:257)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:184)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:234)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Not sure where else should I look too find the root cause

bulk data update from MSSQL vis kafka connector to elasticsearch with nested types is failing

MSSQL value : column name=prop --> value= 100 and column name=role --> value= [{"role":"actor"},{"role":"director"}]
NOTE: the column:role is saved in json format.
read from kafka topic :
{
"schema":{
"type":"struct",
"fields":[
{
"type":"int32",
"optional":false,
"field":"prop"
},
{
"type":"string",
"optional":true,
"field":"roles"
}
],
"optional":false
},
"payload":{ "prop":100, "roles":"[{"role":"actor"},{"role":"director"}]"}
failing with the reason :
Error was [{"type":"mapper_parsing_exception","reason":"object mapping for [roles] tried to parse field [roles] as object, but found a concrete value"}
Reason for failing is that the connector is not able to create schema as array for roles
The above input message is created by confluent JdbcSourceConnector and the sink connector used is confluent ElasticsearchSinkConnector
Configuration details :
sink config:
name=prop-test
connector.class=io.confluent.connect.elasticsearch.ElasticsearchSinkConnector
connection.url=<elasticseach url>
tasks.max=1
topics=test_prop
type.name=prop
#transforms=InsertKey, ExtractId
transforms.InsertKey.type=org.apache.kafka.connect.transforms.ValueToKey
transforms.InsertKey.fields=prop
transforms.ExtractId.type=org.apache.kafka.connect.transforms.ExtractField$Key
transforms.ExtractId.field=prop
Source config:
name=test_prop_source
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
tasks.max=1
connection.url=jdbc:sqlserver://*.*.*.*:1433;instance=databaseName=test;
connection.user=*****
connection.password=*****
query=EXEC <store proc>
mode=bulk
batch.max.rows=2000000
topic.prefix=test_prop
transforms=createKey,extractInt
transforms.createKey.type=org.apache.kafka.connect.transforms.ValueToKey
transforms.createKey.fields=prop
transforms.extractInt.type=org.apache.kafka.connect.transforms.ExtractField$Key
transforms.extractInt.field=prop
connect-standalone.properties :
bootstrap.servers=localhost:9092
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=true
value.converter.schemas.enable=true
need to understand how explicitly i can make the schema as ARRAY for roles and not a string.
The fact that the jdbc Source connector will always sees columnname as fieldname and columnvalue as value. This conversion from string to array is not possible with the existing jdbc source connector support, it has to be a custom transformation or custom plugin to enable it.
The best option available for getting the data transferred from the MSSQL and getting it inserted into the Elastic Search is by using logstash. It has rich filter plugins which can enable the data from MSSQL to flow in any desired format and to any desired JSON output environment ( logstash/kafka topic )
Flow : MSSQL --> logstash --> Kakfa Topic --> Kafka Elastic sink connector --> Elastic Search

Apache Kafka JDBC Connector - SerializationException: Unknown magic byte

We are trying to write back the values from a topic to a postgres database using the Confluent JDBC Sink Connector.
connector.class=io.confluent.connect.jdbc.JdbcSinkConnector
connection.password=xxx
tasks.max=1
topics=topic_name
auto.evolve=true
connection.user=confluent_rw
auto.create=true
connection.url=jdbc:postgresql://x.x.x.x:5432/Datawarehouse
value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url=http://localhost:8081
key.converter=io.confluent.connect.avro.AvroConverter
key.converter.schema.registry.url=http://localhost:8081
We can read the value in the console using:
kafka-avro-console-consumer --bootstrap-server localhost:9092 --topic topic_name
The schema exists and the value are correctly deserialize by kafka-avro-console-consumer because it gives no error but the connector gives those errors:
{
"name": "datawarehouse_sink",
"connector": {
"state": "RUNNING",
"worker_id": "x.x.x.x:8083"
},
"tasks": [
{
"id": 0,
"state": "FAILED",
"worker_id": "x.x.x.x:8083",
"trace": "org.apache.kafka.connect.errors.ConnectException: Tolerance exceeded in error handler\n\tat org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:178)\n\tat org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:104)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.convertAndTransformRecord(WorkerSinkTask.java:511)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:491)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:322)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:226)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:194)\n\tat org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)\n\tat org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: org.apache.kafka.connect.errors.DataException: f_machinestate_sink\n\tat io.confluent.connect.avro.AvroConverter.toConnectData(AvroConverter.java:103)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.lambda$convertAndTransformRecord$0(WorkerSinkTask.java:511)\n\tat org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndRetry(RetryWithToleranceOperator.java:128)\n\tat org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:162)\n\t... 13 more\nCaused by: org.apache.kafka.common.errors.SerializationException: Error deserializing Avro message for id -1\nCaused by: org.apache.kafka.common.errors.SerializationException: Unknown magic byte!\n"
}
],
"type": "sink"
}
The final error is :
org.apache.kafka.common.errors.SerializationException: Unknown magic byte!
The schema is registered in the schema registry.
Does the problem sit with the configuration file of the connector ?
The error org.apache.kafka.common.errors.SerializationException: Unknown magic byte! means that a message on the topic was not valid Avro and could not be deserialised. There are several reasons this could be:
Some messages are Avro, but others are not. If this is the case you can use the error handling capabilities in Kafka Connect to ignore the invalid messages using config like this:
"errors.tolerance": "all",
"errors.log.enable":true,
"errors.log.include.messages":true
The value is Avro but the key isn't. If this is the case then use the appropriate key.converter.
More reading: https://www.confluent.io/blog/kafka-connect-deep-dive-converters-serialization-explained/
This means that deserializer have checked the first 5 bytes of the message and found something unexpected. More about message packing with serializer here , check 'wire format' section. Just a guess that zero byte in message !=0

Kafka not Publishing Oracle Data

I have a Confluent on RHEL setup and am trying to read data from an Oracle 12C table/view (I tried both) and it is never creating messages at the consumer.
My suspicion is that that it has something to do with the data in the tables being loaded using a bulk loader and not unary inserts. I do have a unique incrementing id column in the data that I have specified, and the config loads and it shows my topic name as active/running.
Any ideas?
{
"name":"oracle_source_05",
"config": {
"connector.class":
"io.confluent.connect.jdbc.JdbcSourceConnector",
"key.converter": "io.confluent.connect.avro.AvroConverter",
"key.converter.schema.registry.url": "http://<host>:8081",
"value.converter": "io.confluent.connect.avro.AvroConverter",
"value.converter.schema.registry.url": "http://<host>:8081",
"connection.url": “<jdbc url>
"connection.user" : "<user>",
"connection.password" : "<pw>",
"table.whitelist": "<view name>",
"table.type" : "VIEW",
"mode": "incrementing",
"incrementing.column.name" : "<id column>",
"validate.non.null":"false",
"topic.prefix":"ORACLE-"
}
}
Log has this message:
[2018-04-17 10:59:19,965] DEBUG [Controller id=0] Topics not in preferred replica Map() (kafka.controller.KafkaController)
[2018-04-17 10:59:19,965] TRACE [Controller id=0] Leader imbalance ratio for broker 0 is 0.0 (kafka.controller.KafkaController)
server.log:
[2018-04-18 09:24:26,495] INFO Accepted socket connection from /127.0.0.1:39228 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2018-04-18 09:24:26,498] INFO Client attempting to establish new session at /127.0.0.1:39228 (org.apache.zookeeper.server.ZooKeeperServer)
[2018-04-18 09:24:26,499] INFO Established session 0x162d403daed0004 with negotiated timeout 30000 for client /127.0.0.1:39228 (org.apache.zookeeper.server.ZooKeeperServer)
[2018-04-18 09:24:26,516] INFO Processed session termination for sessionid: 0x162d403daed0004 (org.apache.zookeeper.server.PrepRequestProcessor)
[2018-04-18 09:24:26,517] INFO Closed socket connection for client /127.0.0.1:39228 which had sessionid 0x162d403daed0004 (org.apache.zookeeper.server.NIOServerCnxn)

Resources