Why does Kafka Connect treat timestamp columns differently? - jdbc

Question

I have a Kafka Connect configuration set up to pull data from DB2. I'm not using Avro, just the out-of-the-box json. Among the colums in the db are several timestamp columns, and when they are streamed, they come out like this:
"Process_start_ts": 1578600031762,
"Process_end_ts": 1579268248183,
"created_ts": 1579268247984,
"updated_ts": {
"long": 1579268248182
}
}
The last column is rendered with this sub-element, though the other 3 are not. (This will present problems for the consumer.)
The only thing I can see is that in the DB, that column alone has a default value of null.
Is there some way I can force this column to render in the message as the prior 3?

Answer 1

Try to flatten your message using Kafka Connect Transformations.
The configuration snippet below shows how to use Flatten to concatenate field names with the period . delimiter character (you have to add these lines to the connector config):
"transforms": "flatten",
"transforms.flatten.type": "org.apache.kafka.connect.transforms.Flatten$Value",
"transforms.flatten.delimiter": "."
As a result, your JSON message should look like this:
{
"Process_start_ts": 1578600031762,
"Process_end_ts": 1579268248183,
"created_ts": 1579268247984,
"updated_ts.long": 1579268248182
}
See example Flatten SMT for JSON.

Answer 2

I'm not sure created is any different than the first two. The value is still a long, only the key is all lower case. It's not clear how it would know what the default should be - are you using sure you're not using AvroConverter? If not, it's not clear what fields would have defaults
The updated time is nested like that, based on the Avro or structured JSON Kafka Connect specifications that say the type name is included as part of the record to explicitly denote the type of a nullable field