goTrying to generate golang pb.go file through protoc-gen-gogo. But it seems that there is a specific field 'uint64 sizeis always generated asSize_` with an unexpected _
The message is
message T {
uint64 size = 1;
}
=>
The definition in the pb.go is
type T struct {
Size_ ....
}
Thus my editor always pops an error like there no definition of Size_
My generated command is
protoc(v3) --gogo_out=. --gogo_opt=paths=source_relative *.proto
Underscores may be appended to field names that might collide in anyway with generated names by protoc-gen-go. Size() method is one of the essentials method created by the generator to get the size of the protobuf message. The same applies for keywords reversed by target language (Golang in this instance).
Related
am using a protobuf definition file to create golang classes
file.proto
message my_message {
<what type to use?> data = 1 [json_name = "data"]
}
generated.go
type MyMessage struct {
data any
}
I have checked out Struct , Value and Any, though none of them actually mark the type as “any” in generated classes.
The closes I get is Value type which can accomodate primitives as well as structs and lists.
As far as I'm aware the Go code generator doesn't have any circumstances where any/interface{} will be used for a generated field.
google.protobuf.Any is likely what you want here, but the field in the generated code will of type anypb.Any. If you absolutely require the any type be used, you'll unfortunately have to manually map to another struct.
In protobuf version 3 required and optional keywords first have been removed, since required often caused problems protobuf issue 2497.
Recently the 'optional' keyword has been reintroduced protobuf v3.15.0.
Is it possible to simply add the optional keyword to an existing message?
I.e. change
message Test {
int32 int32_value = 1;
string text_value = 2;
}
to
message Test {
optional int32 int32_value = 1;
optional string text_value = 2;
}
Or will this break the binary format?
non-optional primitive types in protobuf don't accept null-values and normally also map to non-nullable types like int in Java or C#.
But this doesn't mean, that the field is always included in the binary representation.
In fact, if a field contains the default value for the corresponding type the field is omitted in the binary representation.
Thus the following message
message Test {
int32 int32_value = 1;
string text_value = 2;
}
Test test = new Test();
byte[] buffer = test.ToByteArray();
gets serialized to buffer containing an empty byte[].
So missing fields default to default values without the use of optional.
If the optional keyword is changing the behaviour for missing fields in the binary format and for default values specified:
Missing fields indicate the field has not been specified and indicate null. Setting default values will not result in an empty byte[] but in the default values being serialized.
Thus changing a primitive field to optional won't break the format, but will change the semantics:
All fields of old messages that have been specified with the default value will be interpreted as null. Other values are not affected.
The same for optional being removed from a field:
The api won't break, but change semantics. Unspecified fields will then default to default values for the corresponding type.
Given the following proto message
message MyMsg {
string my_field = 1;
string your_field=2;
}
...the protoreflect package can be used to get a descriptor for each field
protoMessage := myMsg.ProtoReflect()
messageDescriptor := protoMessage.Descriptor() // protobuf type information
fieldDescriptors := messageDescriptor.Fields() // list of field declarations
Getting a field descriptor for a specific field is trivial
fieldDescriptor := fieldDescriptors.ByTextName("my_field") // describes a field
Can this be achieved without hard-coding the field name "my_field"? I guess it would be nice to use the generated code to refer to the field I'm interested in. Something like (not working code)
fieldDescriptor := fieldDescriptors.ByTextName(pb.MyMsg.MyField) // describes a field
This way, if the field name changes, it will be caught at compile time, or even from static analysis by an IDE.
The FieldDescriptors type has three methods for getting a field descriptor by name:
ByName(s Name)
ByJSONName(s string)
ByTextName(s string)
ByJSONName and ByTextName both require hard-coded field names (as strings), and by ByName accepts a Name which is type'd to a string. The upshot is, I don't see anything in the protoreflect package that points to a solution.
Context
Field masks are the recommended way to support partial resource updates. It's trivial to iterate over the masks provided in a field mask
protoMessage := myMsg.ProtoReflect()
messageDescriptor := protoMessage.Descriptor() // protobuf type information
fieldDescriptors := messageDescriptor.Fields() // list of field declarations
// iterate over the field paths in the field mask
for _, p := range mask.GetPaths() {
// find the field descriptor for the field path
fieldDescriptor := fieldDescriptors.ByTextName(p)
if fieldDescriptor == nil {
// field descriptor cannot be found for the field path
return
}
// great, the field path points to a field, let's use it
switch p {
case "my_field":
// the client wants to update MyMsg.my_field
case "your_field":
// the client wants to update MyMsg.your_field
}
}
The problem is, in order to actually update the correct field in MyMsg, it's necessary to hard-code the field name in the switch statement.
The interface has ByNumber too
I assume (I've not used the method) that you could give this 1 from your example.
And Get, of course so that you can enumerate all of the FieldDescriptors.
I think it's not unreasonable to want to reflect (!) the generated (struct) types and enumerate across them but it feels like turtles-all-the-way-down.
It may help solicit other answers if you could present the problem that you're unable to solve without the desired method?
Is it (practically) possible to change the type name of a protobuf message type (or enum) without breaking communications?
Obviously the using code would need to be adpated to re-compile. The question is if old clients that use the same structure, but the old names, would continue to work?
Example, base on the real file:
test.proto:
syntax = "proto3";
package test;
// ...
message TestMsgA {
message TestMsgB { // should be called TestMsgZZZ going forward
// ...
enum TestMsgBEnum { // should be called TestMsgZZZEnum going forward
// ...
}
TestMsgBEnum foo = 1;
// ...
}
repeated TestMsgB bar = 1;
// ...
}
Does the on-the-wire format of the protobuf payload change in any way if type or enum names are changed?
If you're talking about the binary format, then no: names don't matter and will not impact your ability to load data; For enums, only the integer value is stored in the payload. For fields, only the field-number is stored.
Obviously if you swap two names, confusion could happen, but: it should load as long as the structure matches.
If you're talking about the JSON format, then it may matter.
I am running into a bit of a problem with Protoc and my existing struct that contains nullable string fields.
The struct I am trying to serialize for transmission contains a bunch of fields that are nullable in json (so we can distinguish between null, "" and a set value).
type Message struct {
Path *string `json:"path"`
}
So if a user sends a empty json string {} the Path will be nil and not "", whereas {"path":""} is also valid and a different case from {"path": null}.
The proto3 declaration I came up with obviously looks like this (and is optional as required and optional got dropped from proto3:
syntax = "proto3";
message Message {
string Path = 1;
}
After running Protoc I end up with a struct that looks like this and all values are string and no way to declare them as *string:
type Message struct {
Path string `protobuf:"bytes,1,opt,name=Path,proto3" json:"Path,omitempty"`
}
Obviously I can't assign to this array from my existing struct. But even if I were to write the tedious mapping code with target.Path = *source.Path with the appropriate null pointer checks etc I'd loose the triple meaning of my source struct (nil, "", "value").
Any suggestions on how to proceed here or if there is an extension to the Go Protobuf to do this? Or how would one go about describing this proto declaration?
Proto3 returns the Zero Value even if a field isn't set. Currently there is no way to distinguish if a field has been set or not.
See Github issue #15.
Possible solutions:
Use proto2 instead of proto3.
Use gogoproto's nullable extension.
Use the google.protobuf.FieldMask extension, see Common Design Patterns
from the Google API Design Guide: Partial Responses and Output Fields.
https://stackoverflow.com/a/62566052
proto3 support optional with flag --experimental_allow_proto3_optional
In my case, I solved this problem with several packages:
https://github.com/gogo/protobuf
https://github.com/golang/protobuf
My proto file looks like this:
syntax = "proto3";
import "google/protobuf/wrappers.proto";
import "github.com/gogo/protobuf/gogoproto/gogo.proto";
message Message {
google.protobuf.StringValue path = 1 [(gogoproto.wktpointer) = true];
}
The command for generating the go code, which I used look like this:
protoc -I. -I%GOPATH%/src --gogofaster_out=plugins=grpc:. proto/*.proto
The generated go file looks like this:
type Message struct {
Path *string `protobuf:"bytes,1,opt,name=path,json=path,proto3,wktptr" json:"path,omitempty"`
}