Query/flatten a serialized protobuf stored as a string column in Clickhouse - protocol-buffers

Question

TLDR
I have a column of serialized protobufs in a table in Clickhouse, and I would like to flatten those protobufs into another table via a materialized view. How do I format the materialized view to do this?
Details
I have a RabbitMQ queue which serves messages to my Clickhouse server. The message outer.proto consists of a service name and a serialized protobuf message payload:
//outer.proto
syntax = "proto2"
message WrappedMessage {
required string svc_id = 1;
required bytes msg = 2;
}
This service name and payload are then stored in Clickhouse as such:
CREATE TABLE raw_records_rmq
(
svc_id String NOT NULL
, msg String NOT NULL
) ENGINE = RabbitMQ SETTINGS
--skip rabbitmq settings, this part works
rabbitmq_format='ProtobufSingle'
rabbitmq_schema = 'outer:WrappedMessage'
;
CREATE TABLE raw_records
(
svc_id String NOT NULL
, msg String NOT NULL
) ENGINE = MergeTree ORDER BY tuple()
;
CREATE MATERIALIZED VIEW raw_mv
TO raw_records AS
SELECT * FROM raw_records_rmq
This process works as expected, with the serialized message stored in raw_records.msg. The message is defined as such:
//inner.proto
syntax = "proto2"
message Person {
optional uint32 id = 1;
optional string name = 2;
optional bool arb_bool = 3;
}
I would now like to query the contents of the stored message; to simplify this, I create a destination table:
CREATE TABLE people
(
id UInt32
, name String
, arb_bool UInt8
) ENGINE = MergeTree ORDER BY tuple()
But this is where my success stops. My attempts so far have been to query the column as a subquery and then attempt to parse the results as protobufs using Clickhouse FORMAT and SETTINGS, as described in their documentation:
CREATE MATERIALIZED VIEW mv
TO people AS
SELECT * FROM (SELECT proto FROM raw_records)
FORMAT ProtobufSingle
SETTINGS format_schema='inner:Person'
However, this fails to unpack the protobuf message. Changing from a Materialized View to a standard View shows that Clickhouse is returning only the single column specified in the subquery with the entire protobuf message as each result.
Any advice on how to properly format this materialized view, or alternatives for processing protobufs-inside-protobufs would be greatly appreciated!

Answer 1

You can't convert a single column into multiple values.
You can use the protobuf format but it will be the entire protobuf message: https://clickhouse.com/docs/en/interfaces/formats/#protobuf