I am not able to call a gRPC function due to type mismatch
my proto file :
message Analytics {
fields ...
}
message AnalyticsSet {
repeated Analytics analytics = 1;
}
service StatService {
rpc MyMethod(AnalyticsSet) returns (<something>) {}
}
Now, I need to call "MyMethod"
My current code :
type Analytics struct {
same fields as in proto : Analytics
}
analytics := make([]Analytics, 4)
// .. some modifications in analytics ...
_, err := c.MyMethod(context.Background(), analytics)
if err != nil {
log.Fatalf("error: %s", err)
}
in Proto file "AnalyticsSet" is the array of "Analytics"
and in Go code "analytics" is an array of type "Analytics"
but this is not enough to call "MyMethod", and I am facing type mismatch..
How should I modify the go code ?
You must use the Analytics struct generated from the proto file -- you cannot use your own type.
You can generate the required Go code using protoc with your .proto file. Here is an example with gRPC generation options set as well:
.
$ protoc --go_out=. --go-grpc_out=. --go_opt=paths=source_relative --go-grpc_opt=paths=source_relative analytics.proto
Your proto file should have the go_package option set to describe the Go import path for that your generated proto code belongs to. You will also need to install the go / go-grpc generator utilities required by protoc:
$ go install google.golang.org/protobuf/cmd/protoc-gen-go#latest
$ go install google.golang.org/grpc/cmd/protoc-gen-go-grpc#latest
More details can be found in:
https://developers.google.com/protocol-buffers/docs/gotutorial
https://developers.google.com/protocol-buffers/docs/reference/go-generated
https://grpc.io/docs/languages/go/quickstart/
Related
Does protobuf support embedding non functional metadata into the protobuf schema without affecting the message serialization/de-serialization? I am attempting to embed business information (ownership, contact info) into a large shared protobuf schema but do NOT want to impact functionality at all.
A structured comment or custom_option that does not get serialized would work. I would also like to parse the information from the .proto file for auditing purposes.
TIA
message Bar {
optional int32 a = 1 [(silent_options).owner = "team1", (silent_options).email = "team1#company.com"];
optional int32 b = 2;
}
To do what you want, you will need to use the extend keyword to create a custom FieldOption and then create a protoc plugin. I recommend using Go or C++ for custom plugins, their API is easier.
First we create a custom field option with extend. You can do so like this:
syntax = "proto3";
import "google/protobuf/descriptor.proto";
option go_package = "test.com/proto";
message SilentOptions {
string owner = 1; // maybe an enum if you know all teams
string email = 2;
}
extend google.protobuf.FieldOptions {
SilentOptions silent_options = 1000;
}
There are few things to note. The first one is that we are extending google.protobuf.FieldOptions which is available because we import descriptor.proto. Then, in my case my go module is called "test.com" and this file is in a folder called proto. So the go_package is "test.com/proto".
Once we have this, we need to create a protoc plugin. There are few tutorials on the web but here is a quick solution which is only working for your example:
package main
import (
"log"
"google.golang.org/protobuf/compiler/protogen"
"google.golang.org/protobuf/proto"
"google.golang.org/protobuf/types/descriptorpb"
"google.golang.org/protobuf/types/pluginpb"
silentOption "test.com/proto"
)
func main() {
protogen.Options{}.Run(func(gen *protogen.Plugin) error {
gen.SupportedFeatures = uint64(pluginpb.CodeGeneratorResponse_FEATURE_PROTO3_OPTIONAL)
for _, sourceFile := range gen.Files {
if !sourceFile.Generate {
continue
}
for _, msg := range sourceFile.Messages {
for _, field := range msg.Fields {
opt := field.Desc.Options().(*descriptorpb.FieldOptions)
if opt == nil { // no FieldOption
continue
}
ext1 := proto.GetExtension(opt, silentOption.E_SilentOptions)
if ext1 != nil { // no SilentOptions
log.Println(ext1)
}
}
}
}
return nil
})
}
For this to run you will need to go get -u google.golang.org/protobuf and run protoc on the silent_options.proto: protoc --go_out=. --go_opt=module=test.com proto/*.proto. For the protoc command to run you need to run the following commands: go get google.golang.org/protobuf/cmd/protoc-gen-go. This is getting the official plugin to generate go code from protobuf.
Finally, build the main.go with a command similar to go build -o protoc-gen-test (note that the name of the binary should start with protoc-gen and then is followed by you plugin name). Then, copy the generated binary to a place in your PATH. And with that you should be able to run: protoc --test_out=. test.proto where test.proto is:
syntax = "proto3";
import "proto/silent_options.proto"; // import the proto file for options
option go_package = "another_test.com"; // can be in a different package
message Bar {
optional int32 a = 1 [
(silent_options) = {
owner: "team1",
email: "team1#company.com"
}
];
optional int32 b = 2;
}
It should print something like:
owner:"team1" email:"team1#company.com"
Obviously this is quite an involved process but I would be happy to update this answer if anything is unclear or if you need more details on the code.
Edit
I forgot to mention that I chose the tag 1000 but there might already have other plugins have the same tag. If you plan to use other plugins later, you should be careful with that and check the tag already in use (protobuf documentation had a page on this, can't find it yet) otherwise you might have conflicts.
Edit 2
Here is the list of the extension tags that are already taken.
I am attempting to create named loggers automatically for HTTP handlers that I'm writing, where I am passed a function (pointer).
I'm using the code mentioned in this question to get the name of a function:
package utils
import (
"reflect"
"runtime"
)
func GetFunctionName(fn interface{}) string {
value := reflect.ValueOf(fn)
ptr := value.Pointer()
ffp := runtime.FuncForPC(ptr)
return ffp.Name()
}
I'm using this in my main function to try it out like so:
package main
import (
"github.com/naftulikay/golang-webapp/experiments/functionname/long"
"github.com/naftulikay/golang-webapp/experiments/functionname/long/nested/path"
"github.com/naftulikay/golang-webapp/experiments/functionname/utils"
"log"
)
type Empty struct{}
func main() {
a := long.HandlerA
b := path.HandlerB
c := path.HandlerC
log.Printf("long.HandlerA: %s", utils.GetFunctionName(a))
log.Printf("long.nested.path.HandlerB: %s", utils.GetFunctionName(b))
log.Printf("long.nested.path.HandlerC: %s", utils.GetFunctionName(c))
}
I see output like this:
github.com/naftulikay/golang-webapp/experiments/functionname/long.HandlerA
This is okay but I'd like an output such as long.HandlerA, long.nested.path.HandlerB, etc.
If I could get the Go module name (github.com/naftulikay/golang-webapp/experiments/functionname), I can then use strings.Replace to remove the module name to arrive at long/nested/path.HandlerB, then strings.Replace to replace / with . to finally get to my desired value, which is long.nested.path.HandlerB.
The first question is: can I do better than runtime.FuncForPC(reflect.ValueOf(fn).Pointer()) for getting the qualified path to a function?
If the answer is no, is there a way to get the current Go module name using runtime or reflect so that I can transform the output of runtime.FuncForPC into what I need?
Once again, I'm getting values like:
github.com/naftulikay/golang-webapp/experiments/functionname/long.HandlerA
github.com/naftulikay/golang-webapp/experiments/functionname/long/nested/path.HandlerB
github.com/naftulikay/golang-webapp/experiments/functionname/long/nested/path.HandlerC
And I'd like to get values like:
long.HandlerA
long.nested.path.HandlerB
long.nested.path.HandlerC
EDIT: It appears that Go does not have a runtime representation of modules, and that's okay, if I can do it at compile time that would be fine too. I've seen the codegen documentation and I'm having a hard time figuring out how to write my own custom codegen that can be used from go generate.
The module info is included in the executable binary, and can be acquired using the debug.ReadBuildInfo() function (the only requirement is that the executable must be built using module support, but this is the default in the current version, and likely the only in future versions).
BuildInfo.Path is the current module's path.
Let's say you have the following go.mod file:
module example.com/foo
Example reading the build info:
bi, ok := debug.ReadBuildInfo()
if !ok {
log.Printf("Failed to read build info")
return
}
fmt.Println(bi.Main.Path)
// or
fmt.Println(bi.Path)
This will output (try it on the Go Playground):
example.com/foo
example.com/foo
See related: Golang - How to display modules version from inside of code
If your goal is to just have the name of the module available in your program, and if you are okay with setting this value at link time, then you may use the -ldflags build option.
You can get the name of the module with go list -m from within the module directory.
You can place everything in a Makefile or in a shell script:
MOD_NAME=$(go list -m)
go build -ldflags="-X 'main.MODNAME=$MOD_NAME'" -o main ./...
With main.go looking like:
package main
import "fmt"
var MODNAME string
func main() {
fmt.Println(MODNAME) // example.com
}
With the mentioned "golang.org/x/mod/modfile" package, an example might look like:
package main
import (
"fmt"
"golang.org/x/mod/modfile"
_ "embed"
)
//go:embed go.mod
var gomod []byte
func main() {
f, err := modfile.Parse("go.mod", gomod, nil)
if err != nil {
panic(err)
}
fmt.Println(f.Module.Mod.Path) // example.com
}
However embedding the entire go.mod file in your use case seems overkill. Of course you could also open the file at runtime, but that means you have to deploy go.mod along with your executable. Setting the module name with -ldflags is more straightforward IMO.
I try to use REST over gRPC using google api annotations.
Unfortunately, I'm facing a protoc issue telling me annotations.proto is not exists or had errors.
I tried several fixes in vain.
I even tried to reinstall a complete stack in case I did something wrong.
I will detail you as much as possible the full lines and files I have set from my fresh install.
From a fresh go install VM I type these shell lines :
$ mkdir sources/golang
$ echo 'export GOPATH=$HOME/sources/golang' >> $HOME/.zshrc
$ source ~/.zshrc
$ cd sources/golang
$ mkdir src
$ cd src
$ export PATH=$PATH:$GOPATH/bin
$ go get -u google.golang.org/grpc
$ go get -u github.com/golang/protobuf/protoc-gen-go
$ go get -u github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway
$ go get -u github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger
$ mkdir -p test/proto/test
$ cd test/proto/test
$ vim test.proto
In My test.proto file, I wrote very basic lines :
syntax = "proto3";
package main;
import "google/api/annotations.proto";
service Tester {
rpc Test (Request) returns (Reply) {
option (google.api.http) = { get: "/v1/test" };
}
}
message Request {
string Message = 1;
}
message Reply {
string Message = 1;
}
Then
$ cd $GOPATH/src
$ vim main.go
In my main.go, very basic too :
package main
import (
tc "test/proto/test"
"context"
"fmt"
"log"
"net"
grpc "google.golang.org/grpc"
codes "google.golang.org/grpc/codes"
)
func main() {
err := StartServer("tcp", "127.0.0.1:50051")
if err != nil {
fmt.Printf("Error!! %s", err)
}
}
type Server struct {
tc.UnimplementedTesterServer
}
func (s *Server) Test(ctx context.Context, in *tc.Request) (*tc.Reply, error) {
return &tc.Reply{Message: ""}, nil
}
func StartServer(protocol string, port string) error {
lis, err := net.Listen(protocol, port)
if err != nil {
fmt.Printf("failed to listen: %v", err)
}
s := grpc.NewServer()
tc.RegisterTesterServer(s, &Server{})
error := s.Serve(lis)
if error != nil {
fmt.Printf("failed to serve: %v", error)
return error
}
return nil
}
Finally, I try to compile my proto files :
$ protoc --proto_path=.:$GOPATH/src --go_out=plugins=grpc:. proto/*/*.proto
And I systematically have the following error :
proto/test/test.proto:5:1: Import "github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis/google/api/annotations.proto" was not found or had errors.
When I see files get by go get ..., they are placed under $GOPATH/pkg/mod/
For example the googleapis' annotations.proto file is under : $GOPATH/pkg/mod/github.com/grpc-ecosystem/grpc-gateway#v1.12.1/third_party/googleapis/google/api/annotations.proto
Maybe is it the cause?
Mixing up your protobuf commands and your go commands is making this more complicated than it needs to be. Just focus on the protobuf to handle the error.
You are importing "google/api/annotations.proto"
Your proto_path is --proto_path=.:$GOPATH/src
So that means that when you execute protoc you should have the file located at ./google/api/annotations.proto or $GOPATH/src/google/api/annotations.proto
I faced the same problem and I was trying to automate a solution that would work on any Linux with as little dependencies as possible.
The official docs for grpc_gateway states:
You will need to provide the required third party protobuf files to the protoc compiler. They are included in this repo under the third_party/googleapis folder, and we recommend copying them into your protoc generation file structure. If you've structured your proto files according to something like the Buf style guide, you could copy the files into a top-level ./google folder.
In other words: download it somehow and make it work.
Also, since this quote is also a quote in the original source, it is not very visible which made me skip it the first 2 times I read it looking for a solution, which is bad.
So, as I argued on the comments of #SeanF answer, using go get is not a maintainable option since it saves the project in a folder whose name contains the latest version, which would cause headaches to maintain when the version changes.
So the best option is actually to clone the grpc-gateway project:
git clone https://github.com/grpc-ecosystem/grpc-gateway
protoc -I grpc-gateway/ \
-I grpc-gateway/third_party/googleapis \
--go_out=plugins=grpc:. proto/*/*.proto
I also wrote a gist with my solution which depends only on docker and bash and, thus, should be stable:
https://gist.github.com/VinGarcia/43dfa71b412c16b9365c224d3760af5e
There is a problem with it, where I am using go.mod but discarding it after every execution, which is not good. But it works well enough for now and it might be helpful to some people.
Regards
I have a file containing a slice of bytes of the following protomessage.
syntax = "proto3";
package main;
message Address {
string street = 1;
string country = 2;
string state = 3;
}
I have the message type described as follow:
func GetProtoDescriptor() (*descriptor.DescriptorProto, error) {
return &descriptor.DescriptorProto{
Name: proto.String("Address"),
Field: []*descriptor.FieldDescriptorProto{
&descriptor.FieldDescriptorProto{
Name: proto.String("street"),
JsonName: proto.String("street"),
Number: proto.Int(1),
Label: descriptor.FieldDescriptorProto_LABEL_OPTIONAL.Enum(),
Type: descriptor.FieldDescriptorProto_TYPE_STRING.Enum(),
},
&descriptor.FieldDescriptorProto{
Name: proto.String("state"),
JsonName: proto.String("state"),
Number: proto.Int(2),
Label: descriptor.FieldDescriptorProto_LABEL_OPTIONAL.Enum(),
Type: descriptor.FieldDescriptorProto_TYPE_STRING.Enum(),
},
&descriptor.FieldDescriptorProto{
Name: proto.String("country"),
JsonName: proto.String("country"),
Number: proto.Int(2),
Label: descriptor.FieldDescriptorProto_LABEL_OPTIONAL.Enum(),
Type: descriptor.FieldDescriptorProto_TYPE_STRING.Enum(),
},
},
}, nil
}
I would like to know how best I can use jhump/protoreflect to parse the content of the file using the message descriptor above.
Thank you for your help.
The typical way would be to compile the protocol to Go code using protoc:
protoc main.proto --go_out=.
This will generate main.pb.go, which will have a type called Address:
var addr Address
err := proto.Unmarshal(bytes, &addr)
If this for some reason is not do-able (e.g. you must do it dynamically, with a descriptor), there are options. (But it's a lot more complicated.)
Use a protoc-generated descriptor. You can have protoc export to a descriptor file (via the -o flag). If this is for RPC, have the server use server reflection; you can then use the github.com/jhump/protoreflect/grpcreflect package to download the server's descriptor (presumably generated by protoc).
If you must create the descriptor programmatically, instead of using protoc, I recommend using the github.com/jhump/protoreflect/desc/builder package to construct it (instead of trying to create raw protos by hand). For example, your raw protos are not sufficient since they have no parent FileDescriptorProto, which is the root of any descriptor hierarchy. That builder package can take care of details like that for you (e.g. it will synthesize a parent file descriptor if necessary).
In any event, the descriptor you want is a desc.Descriptor (from the github.com/jhump/protoreflect/desc package). The tips above (using other protoreflect sub-packages) will return desc.Descriptor instances. If you only have the raw descriptor protos (like in your example code), you'd want to first turn your *descriptor.FileDescriptorProto into a *desc.FileDescriptor using the desc#CreateFileDescriptor function.
If you are using protoc and its -o option to create a descriptor set file (don't forget --include_imports flag, too), you can load it and turn it into a *desc.FileDescriptor using the desc#CreateFileDescriptorFromSet function. Here's an example:
bytes, err := ioutil.ReadFile("protoset-from-protoc")
if err != nil {
panic(err)
}
var fileSet descriptor.FileDescriptorSet
if err := proto.Unmarshal(bytes, &fileSet); err != nil {
panic(err)
}
fd, err := desc.CreateFileDescriptorFromSet(&fileSet)
if err != nil {
panic(err)
}
// Now you have a *desc.FileDescriptor in `fd`
Once you have that right kind of descriptor, you can create a *dynamic.Message. You can then unmarshal into that using either the proto#Unmarshal function or using the dynamic message's Unmarshal method.
I have a main.go file that uses the proto files in pkg/models to Marshal and Unmarshal a proto struct like this:
// Convert to string
protoStr := proto.MarshalTextString(proto)
// Unmarshal string back to proto struct
var proto2 models.Stuff
err := proto.UnmarshalText(protoStr, &proto2)
The setup is here: https://github.com/chuyval/qqs/tree/master/q2
The project contains a vendor directory that only has the github.com/golang/protobuf repo checked out. (run glide install to create vendor if it doesnt exist)
The main.go program works fine when running go run main.go from inside the project.
When I move the main.go file one level up to the parent directory, and run the same command go run main.go at the parent level, it reports the following error:
line 2: unknown field name "value_list" in models.Stuff
When I delete the vendor directory in the project directory, and run go run main.go at the parent level, I get no error.
Why would having a vendor directory in the project repository make it error out?
Another thing to note, if I run the same main.go application inside of the dependent repo, it works every time (with or without the vendor repository).
Sample code:
package main
import (
"github.com/chuyval/qqs/q2/pkg/models"
"github.com/golang/protobuf/proto"
"log"
)
func main () {
stuff := createProtoStuff()
log.Printf("Stuff: %+v", stuff)
// Convert to string
stuffStr := proto.MarshalTextString(stuff)
// Unmarshal string back to proto struct
var stuff2 models.Stuff
err := proto.UnmarshalText(stuffStr, &stuff2)
if err != nil {
log.Printf("It didnt work. Error: %s", err.Error())
} else {
log.Printf("It worked. Proto: %+v", stuff2)
}
}
func createProtoStuff() *models.Stuff {
someValueList := []*models.SomeValue{&models.SomeValue{Id: "Some Value ID"}}
valueList := &models.SomeValueList{SomeValue: someValueList}
stuffValueList := &models.Stuff_ValueList{
ValueList: valueList,
}
stuff := &models.Stuff{
Id: "Stuff List Id",
Stuff: stuffValueList,
}
return stuff
}
Software Versions
glide version 0.13.1
go version go1.10.3 darwin/amd64
protoc version libprotoc 3.6.0