Modeling schema metadata without serializing into the protobuf message - protocol-buffers

Question

Does protobuf support embedding non functional metadata into the protobuf schema without affecting the message serialization/de-serialization? I am attempting to embed business information (ownership, contact info) into a large shared protobuf schema but do NOT want to impact functionality at all.
A structured comment or custom_option that does not get serialized would work. I would also like to parse the information from the .proto file for auditing purposes.
TIA
message Bar {
optional int32 a = 1 [(silent_options).owner = "team1", (silent_options).email = "team1#company.com"];
optional int32 b = 2;
}

Answer 1

To do what you want, you will need to use the extend keyword to create a custom FieldOption and then create a protoc plugin. I recommend using Go or C++ for custom plugins, their API is easier.
First we create a custom field option with extend. You can do so like this:
syntax = "proto3";
import "google/protobuf/descriptor.proto";
option go_package = "test.com/proto";
message SilentOptions {
string owner = 1; // maybe an enum if you know all teams
string email = 2;
}
extend google.protobuf.FieldOptions {
SilentOptions silent_options = 1000;
}
There are few things to note. The first one is that we are extending google.protobuf.FieldOptions which is available because we import descriptor.proto. Then, in my case my go module is called "test.com" and this file is in a folder called proto. So the go_package is "test.com/proto".
Once we have this, we need to create a protoc plugin. There are few tutorials on the web but here is a quick solution which is only working for your example:
package main
import (
"log"
"google.golang.org/protobuf/compiler/protogen"
"google.golang.org/protobuf/proto"
"google.golang.org/protobuf/types/descriptorpb"
"google.golang.org/protobuf/types/pluginpb"
silentOption "test.com/proto"
)
func main() {
protogen.Options{}.Run(func(gen *protogen.Plugin) error {
gen.SupportedFeatures = uint64(pluginpb.CodeGeneratorResponse_FEATURE_PROTO3_OPTIONAL)
for _, sourceFile := range gen.Files {
if !sourceFile.Generate {
continue
}
for _, msg := range sourceFile.Messages {
for _, field := range msg.Fields {
opt := field.Desc.Options().(*descriptorpb.FieldOptions)
if opt == nil { // no FieldOption
continue
}
ext1 := proto.GetExtension(opt, silentOption.E_SilentOptions)
if ext1 != nil { // no SilentOptions
log.Println(ext1)
}
}
}
}
return nil
})
}
For this to run you will need to go get -u google.golang.org/protobuf and run protoc on the silent_options.proto: protoc --go_out=. --go_opt=module=test.com proto/*.proto. For the protoc command to run you need to run the following commands: go get google.golang.org/protobuf/cmd/protoc-gen-go. This is getting the official plugin to generate go code from protobuf.
Finally, build the main.go with a command similar to go build -o protoc-gen-test (note that the name of the binary should start with protoc-gen and then is followed by you plugin name). Then, copy the generated binary to a place in your PATH. And with that you should be able to run: protoc --test_out=. test.proto where test.proto is:
syntax = "proto3";
import "proto/silent_options.proto"; // import the proto file for options
option go_package = "another_test.com"; // can be in a different package
message Bar {
optional int32 a = 1 [
(silent_options) = {
owner: "team1",
email: "team1#company.com"
}
];
optional int32 b = 2;
}
It should print something like:
owner:"team1" email:"team1#company.com"
Obviously this is quite an involved process but I would be happy to update this answer if anything is unclear or if you need more details on the code.
Edit
I forgot to mention that I chose the tag 1000 but there might already have other plugins have the same tag. If you plan to use other plugins later, you should be careful with that and check the tag already in use (protobuf documentation had a page on this, can't find it yet) otherwise you might have conflicts.
Edit 2
Here is the list of the extension tags that are already taken.