Hi I am trying to generate the simple protobuf file in Go language
syntax = "proto3";
package gen;
message EvtKeepAlive
{
string SvcName = 2;
}
In the header I see that the package uses two different proto go implementations, one from github.com and one from google.golang.org. As far as I understand the latter supersedes the former, so is this file generation valid?
// versions:
// protoc-gen-go v1.25.0-devel
// protoc v3.13.0
// source: common.proto
package gen
import (
proto "github.com/golang/protobuf/proto"
protoreflect "google.golang.org/protobuf/reflect/protoreflect"
protoimpl "google.golang.org/protobuf/runtime/protoimpl"
reflect "reflect"
sync "sync"
)
The file is valid; see the comments from dsnet in response to this issue:
The only reason the newly generated .pb.go files depend on the
deprecated proto package is to enforce a weak dependency on a
sufficiently new version of the legacy package. This is necessary
because not everyone is using Go modules such that the Go toolchain
would enforce this dependency constraint. I wasn't fond of adding it,
but I think it's probably necessary to keep at least for a few months.
Related
Handling errors in a gRPC service commonly requires both a status message and error codes. Both have two definitions:
Google APIs definition (googleapis/go-genproto) - the generated Go packages for common protocol buffer types, and the generated gRPC code for Google's gRPC APIs
gRPC definition (grpc/grpc-go) - the Go implementation of gRPC
The Go packages for both definitions of Status and Codes are:
Google APIs
Status: google.golang.org/genproto/googleapis/rpc/status
Codes: google.golang.org/genproto/googleapis/rpc/code
gRPC
Status: google.golang.org/grpc/status
Codes: google.golang.org/grpc/codes
Since I'm a client of my own gRPC service and not a client of an existing Google gRPC API, I want to use the gRPC definitions of Status and Code.
However, the gRPC proto file for Status is actually copied from Google APIs definition. See https://github.com/grpc/grpc/tree/master/src/proto/grpc/status. The go_package of status.proto is also unchanged, so both the Google API and gRPC definitions use the following Go package
option go_package = "google.golang.org/genproto/googleapis/rpc/status;status";
The upshot is the only way to use Status when defining an API is by importing it with
import "google/rpc/status.proto";
...and importing the language bindings in Go with
import (
"google.golang.org/genproto/googleapis/rpc/status"
)
// Go server code...
But as stated earlier, this is wrong since I'm not a client of a Google API, but rather my own gRPC service. Therefore the language bindings should be imported with
import (
"google.golang.org/grpc/status"
)
// Go server code...
As expected if I switch to importing the gRPC language bindings and try and return a Status message to the API client, I get a compile error
cannot use &(status.Status literal)
(value of type *"google.golang.org/grpc/internal/status".Status) as
*"google.golang.org/genproto/googleapis/rpc/status".Status value
This is caused by my .proto file using the Google API definition of Status while the server implementation (in Go) uses the gRPC definition.
The problem impacts error codes since Google APIs uses signed 32 bit integers (int32) whereas gRPC uses unsigned 32 bit integers (uint32).
Questions
Is my assertion that I should be using the gRPC definition of Status and Codes correct?
If my assertion is correct, how can I use the gRPC definition of Status when it's packaged for Google APIs?
We need to distinguish a few cases. Some of them are obvious, some are not.
Just returning Status from a gRPC handler
If your proto schema (.proto files) doesn't define messages that use Status or Code directly, then the gRPC handlers can satisfy the return type error simply with "google.golang.org/grpc/status".Error(), or Newf().Err(). And that's about it.
Example:
// implements SomeServiceServer unary RPC GetFoo
func (s *SomeService) GetFoo(ctx context.Context, req *grpc.FooRequest) (*grpc.FooResponse, error) {
// status is "google.golang.org/grpc/status"
return nil, status.Error(codes.Unimplemented, "coming soon")
Using Status in your .proto files
In this case, you are forced to use the googleapis implementation. As you already have seen, the status.proto Go package is defined as:
option go_package = "google.golang.org/genproto/googleapis/rpc/status;status";
So let's say you have the following .proto file, where the imported status.proto is just a copy-paste of the gRPC status.proto as per this question:
syntax = "proto3";
package test;
import "status.proto";
option go_package = ".;main";
message Foo {
string a = 1;
google.rpc.Status status = 2;
}
with directory structure as:
/protos
|_ status.proto
|_ test.proto
and you compile the above with:
cd protos && protoc -I=. --go_out=. test.proto
breathe ...then the generated Go code will have the following import
import (
status "google.golang.org/genproto/googleapis/rpc/status"
)
and you must satisfy that by go get google.golang.org/genproto.
So about your first question, you can only use Status from googleapis in proto files, because that's how status.proto declares its Go package.
Using generated googleapis Status in Go code
Since the imported Go package is from googleapis that is what you must use in your Go code in order to initialize such messages:
package main
import (
"fmt"
googleapis "google.golang.org/genproto/googleapis/rpc/status"
)
func main() {
foo := &Foo{
A: "foo",
Status: &googleapis.Status{
Code: int32(code.Code_OK),
Message: "all good",
},
}
// fmt.Println(foo)
}
Yes but I must use grpc-go Status in my Go code
You can't. protoc generates code with the packages described above. If you absolutely NEED to construct these Status fields using grpc-go, you can use Status.Proto:
package main
import (
"fmt"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
)
func main() {
foo := &Foo{
A: "foo",
Status: status.New(codes.OK, "all good").Proto(),
}
fmt.Println(foo)
}
Just for the record, the opposite is also possible with status.FromProto:
package main
import (
"fmt"
googleapis_codes "google.golang.org/genproto/googleapis/rpc/code"
googleapis "google.golang.org/genproto/googleapis/rpc/status"
"google.golang.org/grpc/status"
)
func main() {
gapisStatus := &googleapis.Status{
Code: int32(googleapis_codes.Code_OK),
Message: "all good",
}
grpcStatus := status.FromProto(gapisStatus)
fmt.Println(grpcStatus)
}
As a less well-behaved alternative, you can simply copy-paste the status.proto sources into your project and manually change the go_package:
option go_package = "google.golang.org/grpc/status;status";
This way protoc will generate the Go code with this import, and your own sources will be able to follow suit. Of course this means you now have your own fork of status.proto.
My current Directory is setup as follows:
-- /webserver
|
/ftp
|-> ftp.go
|-> go.mod
|->server.go
|->go.mod
Server.go page heading
package main
import (
[..]
"../webserver/ftp" // issue package
)
func main() {
[..]
ftp.test()
}
ftp.go page
package ftp
import "fmt"
func test() {
fmt.Println("YES!")
}
Here's the part I cant seem to understand in the console "PROMBLEMS" tab. The ftp.go file gets saved. Before I save the server.go file with "../webserver/ftp/" the console displays this:
When I save server.go it removes "./webserver/ftp", I went as far as adding it into the go src to make it "ftp" instead of "../webserver/ftp". I get the same result.
Not sure what I am doing wrong.
Basically what i want to accomplish is:
have a functions folder and have the functions in that folder (functions.go) to keep the server.go page as clean as possible.
You have several problems here.
ftp/go.mod should not exist. While it is possible to have multiple Go modules in a single repository, this is very advanced usage, and is rarely useful. Unless you absoultely know that you need this capability, and are already exremely comfortable with creating and publishing Go modules, don't do this.
Go does not support relative imports. import "../webserver/ftp" is an invalid import path*. Use the fully qualified import path, i.e. import "github.com/username/projectname/webserver/ftp"
The path to use here depends on the module name you've declared on the module line of your your top-level go.mod file.
You cannot import unexported values.
You've defined func test() string in ftp.go. This is unexported, by virtue of beginning with a lowercase letter. Only functions, variables, or other symbols which begin with a capital letter are exported in Go. Try func Test() string instead.
*Some will say "But I used import ".../foo" and it worked!". Yes, it can work under very specific circumstances. But its behavior is not reliable, so you should never rely on this, except in very specific testing types of situations.
I'm new to Protocol Buffers and gRPC stuff. Now I'm trying to build a client/server architecture with grpc + grpc-gateway in Go.
I tried to follow some examples but I always end up with the same problem.
After generating the code with protoc i run go build and I get this error:
proto/helloworld/hello_world.pb.gw.go:64:2: cannot use msg (type *HelloReply) as type protoreflect.ProtoMessage in return argument:
*HelloReply does not implement protoreflect.ProtoMessage (missing ProtoReflect method)
proto/helloworld/hello_world.pb.gw.go:98:2: cannot use msg (type *HelloReply) as type protoreflect.ProtoMessage in return argument:
*HelloReply does not implement protoreflect.ProtoMessage (missing ProtoReflect method)
This is go.mod:
module github.com/riccardopedrielli/grpc-gateway-test
go 1.15
require (
github.com/golang/protobuf v1.4.3
github.com/grpc-ecosystem/grpc-gateway/v2 v2.2.0
google.golang.org/genproto v0.0.0-20210207032614-bba0dbe2a9ea
google.golang.org/grpc v1.35.0
google.golang.org/protobuf v1.25.0
)
This is hello_world.proto:
syntax = "proto3";
package helloworld;
import "google/api/annotations.proto";
option go_package = "github.com/riccardopedrielli/grpc-gateway-test/proto/helloworld";
// Here is the overall greeting service definition where we define all our endpoints
service Greeter {
// Sends a greeting
rpc SayHello (HelloRequest) returns (HelloReply) {
option (google.api.http) = {
get: "/v1/example/echo/{name}"
};
}
}
// The request message containing the user's name
message HelloRequest {
string name = 1;
}
// The response message containing the greetings
message HelloReply {
string message = 1;
}
This is the link to the repository: https://github.com/riccardopedrielli/grpc-gateway-test
A difference I see between the generated go files is that they are importing different protobuf libraries.
The one generated by protoc-gen-go imports github.com/golang/protobuf/proto.
The one generated by protoc-gen-grpc-gateway imports google.golang.org/protobuf/proto.
Could this be the cause of the problem?
Still it's not clear to me which one should be used and how to force the same in both the generators.
I'm new to grpc and quite lost at this point, so I could have omitted some important informations. Any suggestion will be welcomed.
Thank you
Ok I solved the issue.
I had installed protoc via snap and the stable channel had version 3.11.4
Now I upgraded to 3.14.0 and everything is working well.
For generating the stubs, we can use either protoc or buf. protoc is the more classic generation experience used widely in the industry. Still, it has a pretty steep learning curve. buf is a newer tool built with user experience and speed in mind. It also offers linting and breaking change detection, and something protoc doesn’t provide.
You should check out the tutorial series on gRPC-Gateway, i.e., https://grpc-ecosystem.github.io/grpc-gateway/docs/tutorials/. Also, you can refer to my simple hello world program, which uses gRPC-Gateway, i.e., https://github.com/iamrajiv/helloworld-grpc-gateway.
I am using a golang package, say name pkgfoo; and the author explicitly said if we want to use a package subpkg under pkgfoo, I need to explicitly import subpkg. I don't understand the reason behind it. Isn't the subpkg automatically imported if I import the top pkg in Golang?
package main
import (
"myownpackage"
"github.com/usera/pkgfoo"
"github.com/usera/pkgfoo/subpkg"
)
func main() {
// Use functions in pkgfoo, and use functions in pkgfoo/subpkg
// ......
http.HandleFunc("/login", login)
err := http.ListenAndServe(":9090", nil) // setting listening port
if err != nil {
log.Fatal("ListenAndServe: ", err)
}
}
My question is whether I really need import "github.com/usera/pkgfoo/subpkg" statement.
Go packages are constructed from one or more source files, which are organized into folders in the file system. Source files of the same package must be in the same folder.
But in Go there is no "package hierarchy", the "subpackage" term simply refers to the folder of a package being a subfolder of another. We often organize packages to be in folders being subfolders of others because there is some connection between those packages (e.g. a package is used only by another being in the parent folder; or a package in a subfolder is a specific or more special implementation of the package in the parent folder; or simply just a logical grouping - see the end of the answer).
So whenever you do
import "github.com/usera/pkgfoo"
It only imports pkgfoo and no other packages sharing the same path as prefix. If you need github.com/usera/pkgfoo/subpkg too, you also need to explicitly import it:
import (
"github.com/usera/pkgfoo"
"github.com/usera/pkgfoo/subpkg"
}
The Go Blog: Package names on directories used in the standard library:
Directories. The standard library uses like directories crypto, container, encoding, and image to group packages for related protocols and algorithms. There is no actual relationship among the packages in one of these directories; a directory just provides a way to arrange the files. Any package can import any other package provided the import does not create a cycle.
I am using wgo for dependency management in Golang (although I think wgo has little to do with this), wgo has a folder structure like this
project/
.gocfg/
gopaths
vendor.json
vendor/
src/
github.com_or_whatever/
I have a library I coded myself which uses an nsq-go type in one of the exported methods:
func AddNsqSubscription(
topic, channel string,
handler nsq.Handler,
config *nsq.Config) error { }
The library is called messi and I import the nsq-go like so "messi/vendor/src/github.com/bitly/go-nsq"
The problem comes when I try to use this library in another project. For instance, in a project called scribe I have the following code (notice the imports):
import (
"scribe/vendor/src/github.com/bitly/go-nsq"
"scribe/vendor/src/messi"
)
//...
nsqHandler := nsq.HandlerFunc(func(message *nsq.Message) error {
msgHandler(MessiMessage{message})
return nil
})
return messi.AddNsqSubscription(destination, subdestination, nsqHandler, nsq.NewConfig())
When I go build the following error is returned:
cannot use nsqHandler (type "scribe/vendor/src/github.com/bitly/go-nsq".HandlerFunc) as type "messi/vendor/src/github.com/bitly/go-nsq".Handler in argument to messi.AddNsqSubscription:
"scribe/vendor/src/github.com/bitly/go-nsq".HandlerFunc does not implement "messi/vendor/src/github.com/bitly/go-nsq".Handler (wrong type for HandleMessage method)
have HandleMessage("scribe/vendor/src/github.com/bitly/go-nsq".Message) error
want HandleMessage("messi/vendor/src/github.com/bitly/go-nsq".Message) error
Why? I do not really know what is going on. The code go-nsq imported is exactly the same, yet golang wants that this code comes from the same folder?
What am I doing wrong?
Packages in Go are identified by full import path, not by name.
For example in the standard library there are two different packages with the same name template but different import paths: text/template and html/template.
You should make sure that go-nsq package is imported using the same path.