I am running into a bit of a problem with Protoc and my existing struct that contains nullable string fields.
The struct I am trying to serialize for transmission contains a bunch of fields that are nullable in json (so we can distinguish between null, "" and a set value).
type Message struct {
Path *string `json:"path"`
}
So if a user sends a empty json string {} the Path will be nil and not "", whereas {"path":""} is also valid and a different case from {"path": null}.
The proto3 declaration I came up with obviously looks like this (and is optional as required and optional got dropped from proto3:
syntax = "proto3";
message Message {
string Path = 1;
}
After running Protoc I end up with a struct that looks like this and all values are string and no way to declare them as *string:
type Message struct {
Path string `protobuf:"bytes,1,opt,name=Path,proto3" json:"Path,omitempty"`
}
Obviously I can't assign to this array from my existing struct. But even if I were to write the tedious mapping code with target.Path = *source.Path with the appropriate null pointer checks etc I'd loose the triple meaning of my source struct (nil, "", "value").
Any suggestions on how to proceed here or if there is an extension to the Go Protobuf to do this? Or how would one go about describing this proto declaration?
Proto3 returns the Zero Value even if a field isn't set. Currently there is no way to distinguish if a field has been set or not.
See Github issue #15.
Possible solutions:
Use proto2 instead of proto3.
Use gogoproto's nullable extension.
Use the google.protobuf.FieldMask extension, see Common Design Patterns
from the Google API Design Guide: Partial Responses and Output Fields.
https://stackoverflow.com/a/62566052
proto3 support optional with flag --experimental_allow_proto3_optional
In my case, I solved this problem with several packages:
https://github.com/gogo/protobuf
https://github.com/golang/protobuf
My proto file looks like this:
syntax = "proto3";
import "google/protobuf/wrappers.proto";
import "github.com/gogo/protobuf/gogoproto/gogo.proto";
message Message {
google.protobuf.StringValue path = 1 [(gogoproto.wktpointer) = true];
}
The command for generating the go code, which I used look like this:
protoc -I. -I%GOPATH%/src --gogofaster_out=plugins=grpc:. proto/*.proto
The generated go file looks like this:
type Message struct {
Path *string `protobuf:"bytes,1,opt,name=path,json=path,proto3,wktptr" json:"path,omitempty"`
}
Related
am using a protobuf definition file to create golang classes
file.proto
message my_message {
<what type to use?> data = 1 [json_name = "data"]
}
generated.go
type MyMessage struct {
data any
}
I have checked out Struct , Value and Any, though none of them actually mark the type as “any” in generated classes.
The closes I get is Value type which can accomodate primitives as well as structs and lists.
As far as I'm aware the Go code generator doesn't have any circumstances where any/interface{} will be used for a generated field.
google.protobuf.Any is likely what you want here, but the field in the generated code will of type anypb.Any. If you absolutely require the any type be used, you'll unfortunately have to manually map to another struct.
My proto file looks something like:
message Ack {
bool required = 1;
optional string address = 2;
}
It looks like I am not able to name my variable as 'required' because its already been reserved by GRPC server system. Can anyone please guide me how to create a variable that has a name "required" instead of requiring a variable?
Please include more details in your question:
the command that you used
the error that resulted
I suspect the issue arises at the protocol buffer rather than gRPC layer and probably when you attempt to compile the protos using protoc.
required is a reserved word for protocol buffers version proto2 but not (currently?) in proto3.
Compilers and other tools fail to parse sources that include reserved words in unexpected locations. Even if you can get such sources to compile, it's not good practice to try to abuse this practice because it could result in unexpected behavior elsewhere.
You should consider using an alternative name:
required_
_required
is_required
...
Using the latest version of protoc release v21.2 and generated Golang sources, I'm able to compile proto2 and proto3 sources that include a field called required:
syntax = "proto2";
package foo;
option go_package = "github.com/stackoverflow/73038286/foo;foo";
message Ack {
required bool required = 1;
optional string address = 2;
}
And:
syntax = "proto3";
package foo;
option go_package = "github.com/DazWilkin/stackoverflow/73038286/foo;foo";
message Ack {
bool required = 1;
optional string address = 2;
}
goTrying to generate golang pb.go file through protoc-gen-gogo. But it seems that there is a specific field 'uint64 sizeis always generated asSize_` with an unexpected _
The message is
message T {
uint64 size = 1;
}
=>
The definition in the pb.go is
type T struct {
Size_ ....
}
Thus my editor always pops an error like there no definition of Size_
My generated command is
protoc(v3) --gogo_out=. --gogo_opt=paths=source_relative *.proto
Underscores may be appended to field names that might collide in anyway with generated names by protoc-gen-go. Size() method is one of the essentials method created by the generator to get the size of the protobuf message. The same applies for keywords reversed by target language (Golang in this instance).
I now tried, everything but cant get this simple thing to work.
I got the following test_file.hcl:
variable "value" {
test = "ok"
}
I want to parse it using the following code:
package hcl
import (
"github.com/hashicorp/hcl/v2/hclsimple"
)
type Config struct {
Variable string `hcl:"test"`
}
func HclToStruct(path string) (*Config, error) {
var config Config
return &config, hclsimple.DecodeFile(path, nil, &config)
But I receive:
test_file.hcl:1,1-1: Missing required argument; The argument "test" is required, but no definition was found., and 1 other diagnostic(s)
I checked with other projects using the same library but I cannot find my mistake .. I just dont know anymore. Can someone guide me into the right direction?
The Go struct type you wrote here doesn't correspond with the shape of the file you want to parse. Notice that the input file has a variable block with a test argument inside it, but the Go type you wrote has only the test argument. For that reason, the HCL parser is expecting to find a file with just the test argument at the top-level, like this:
test = "ok"
To parse this Terraform-like structure with hclsimple will require you to write two struct types: one to represent the top level body containing variable blocks and another to represent the content of each of the blocks. For example:
type Config struct {
Variables []*Variable `hcl:"variable,block"`
}
type Variable struct {
Test *string `hcl:"test"`
}
With that said, I'll note that the Terraform language uses some features of HCL that hclsimple can't support, or at least can't support with direct decoding like this. For example, the type argument inside variable blocks is a special sort of expression called a type constraint which hclsimple doesn't support directly, and so parsing it will require using the lower-level HCL API. (That's what Terraform is doing internally.)
Thanks to #Martin Atkins, I now have the following code that works:
type Config struct {
Variable []Variable `hcl:"variable,block"`
}
type Variable struct {
Value string `hcl:"name,label"`
Test string `hcl:"test"`
}
With that I can parse a variables.tf like:
variable "projectname" {
default = "cicd-template"
}
variable "ansibleuser" {
default = "centos"
}
I have a simple structure:
type MyWriter struct {
io.Writer
}
Which I then use in the following way:
writer = MyWriter{io.Stdout}
When running go vet this gives me a composite literal uses unkeyed fields.
In order to fix this would I have to turn io.Reader into a field in the MyWriter structure by adding a key?
type MyWriter struct {
w io.Writer
}
Is there any other way around this?
The only other answer I found on here suggests to disable the check altogether but I would rather not do that and find a proper solution.
Try this:
writer = MyWriter{Writer: io.Stdout}
Embedded structs have an implicit key of the type name itself without the package prefix (e.g. in this case, Writer).