Nifi convert sql to json structured - apache-nifi

Question

I am trying to figure out a way to get data out of SQL and format it in a specific json format and i am having a hard time doing that in nifi.
Data that is in the table looks like this.
{
"location_id": "123456",
"name": "My Organization",
"address_1": "Address 1",
"address_2": "Suite 123",
"city": "My City",
"state": "FL",
"zip_code": "33333",
"description": "",
"longitude": "-2222.132131321332113",
"latitude": "111.21321321321321321",
"type": "data type"
}
And i want to convert it into a format like this.
{
"type": "FeatureCollection",
"features": [
{
"geometry": {
"type": "Point",
"coordinates": [
$.longitude,
$.latitude
]
},
"type": "Feature",
"properties": {
"name": $.name,
"phone": $.phone_number,
"address1": $.address_1,
"address2": $.address_2,
"city": $.city,
"state": $.state,
"zip": $.zip_code,
"type": $.type
}
}
]
}
This is what i have so far and by all means if i am doing this in a weird way let me know.
I was thinking i could split all of these into single record jsons format them in this format.
{
"geometry": {
"type": "Point",
"coordinates": [
$.longitude,
$.latitude
]
},
"type": "Feature",
"properties": {
"name": $.name,
"phone": $.phone_number,
"address1": $.address_1,
"address2": $.address_2,
"city": $.city,
"state": $.state,
"zip": $.zip_code,
"type": $.type
}
}
And then merge all of the records together and merge it back around this
{
"type": "FeatureCollection",
"features": [
]
}
I def feel like i am doing this weird, just not sure how to get it done haha.

Answer 1

Try ExecuteSQLRecord with a JsonRecordSetWriter instead of ExecuteSQL, this will allow you to output the rows as JSON objects without converting to/from Avro. If you don't have too many rows (that would cause an out-of-memory error), you can use JoltTransformJSON to do the whole transformation (without splitting the rows) with the following Chain spec:
[
{
"operation": "shift",
"spec": {
"#FeatureCollection": "type",
"*": {
"#Feature": "features[&1].type",
"name": "features[&1].properties.name",
"address_1": "features[&1].properties.address_1",
"address_2": "features[&1].properties.address_2",
"city": "features[&1].properties.city",
"state": "features[&1].properties.state",
"zip_code": "features[&1].properties.zip",
"type": "features[&1].properties.type",
"longitude": "features[&1].geometry.coordinates.longitude",
"latitude": "features[&1].geometry.coordinates.latitude"
}
}
}
]
If you do have too many rows, you can use SplitJson to split them into smaller chunks, then JoltTransformJSON (with the above spec) then MergeRecord to merge them back into one large array. To get them nested into the features field, you could use ReplaceText to "wrap" the array in the outer JSON object, but that too may cause an out-of-memory error.