How to load Avro array of records into Vertica ARRAY - vertica

Question

My avro files contain the following column:
{"name":"my_column","type":["null",{"type":"array","items":{"type":"record","name":"my_column","namespace":"v11","fields":[{"name":"my_column","type":["null","int"],"default":null}]}}],"default":null}
I loaded the data into Vertica and stored as VARBINARY. Example:
db=> select MapToString(my_column) from tab limit 1;
MapToString
------------------------------------------------------------------------------------------------------------------------------------------------------
{
"0.__name__": "my_column",
"0.my_column": "5",
"1.__name__": "my_column",
"1.my_column": "9"
}
(1 row)
The data can actually be simplified into ARRAY[INT]. (I.e. ARRAY[5,9]).
What is the correct way of performing this transformation?
Extend Vertica via UDTF or UDParser? Perform this transformation via SQL? Something else?
EDIT: I am going to check whether a scalar UDF can be embedded in the COPY command alongside the AVROPARSER, or wether it requires extra ETL.
Thank you!

Answer 1

Try materialising the column - well, you need to know what to expect in a flex table ...
I had this table first:
SELECT
__id__
, REGEXP_REPLACE(MAPTOSTRING(__raw__),'\s+',' ') AS rawasstring
FROM a;
-- out __id__ | rawasstring
-- out --------+--------------------------------------------------------------------------------------------------
-- out | { "0.__name__": "my_column", "0.my_column": "5", "1.__name__": "my_column", "1.my_column": "9" }
-- out (1 row)
Then, I just added a materialised column, like so:
ALTER TABLE a ADD int_array ARRAY[int,10]
DEFAULT ARRAY[
MAPLOOKUP(__raw__, '0.my_column')
, MAPLOOKUP(__raw__, '1.my_column')
, MAPLOOKUP(__raw__, '2.my_column')
, MAPLOOKUP(__raw__, '3.my_column')
, MAPLOOKUP(__raw__, '4.my_column')
, MAPLOOKUP(__raw__, '5.my_column')
, MAPLOOKUP(__raw__, '6.my_column')
, MAPLOOKUP(__raw__, '7.my_column')
, MAPLOOKUP(__raw__, '8.my_column')
, MAPLOOKUP(__raw__, '9.my_column')
]::ARRAY[INT,10];
Keys that don't exist in the Map are NULL and not really added to the array. And now I have the array:
SELECT
__id__::VARCHAR
, REGEXP_REPLACE(MAPTOSTRING(__raw__),'\s+',' ') AS rawasstring
, int_array
FROM a ;
-- out __id__ | rawasstring | int_array
-- out --------+--------------------------------------------------------------------------------------------------+-----------
-- out | { "0.__name__": "my_column", "0.my_column": "5", "1.__name__": "my_column", "1.my_column": "9" } | [5,9]
-- out (1 row)