I am trying to generate Go code for some Protocol Buffers as well as a gRPC service.
In the past I have used https://github.com/golang/protobuf with a generate command that looked something like this:
//go:generate protoc --proto_path=. --go-out=. --go_opt=paths=source_relative <file>.proto
//go:generate protoc --proto_path=. --go_grpc_out=. --go_grpc_opt=paths=source_relative <file>.proto
This method worked just fine, but the repo has since been superceded by google.golang.org/protobuf and google.golang.org/grpc.
From what I've read, this was an intentional split to separate the protobuf and gRPC project release cycles.
Using the new repositories, I installed the tools like this:
go install google.golang.org/protobuf/cmd/protoc-gen-go#latest
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc#latest
Both tools are installed, and are visible on the path. The problem that I'm encountering is that protoc or protoc-gen-go is looking for the older tool - protoc-gen-go_grpc. Note the underscore.
I have been reading all the documentation and Github issues that I can find, and haven't yet been able to figure out how I am meant to use the new protoc-gen-go-grpc. Note the dash.
$ echo $GOPATH
/Users/<username>/dev/go
$ echo $GOBIN
/Users/<username>/dev/go/bin
$ which protoc
/opt/homebrew/bin/protoc
$ which protoc-gen-go
/Users/<username>/dev/go/bin/protoc-gen-go
$ which protoc-gen-go_grpc
protoc-gen-go_grpc not found
$ which protoc-gen-go-grpc
/Users/<username>/dev/go/bin/protoc-gen-go-grpc
My issue was so simple to fix, but darn annoying to find.
When running protoc, using the --go_grpc_out flag looks for protoc-gen-go_grpc.
When running protoc, using --go-grpc_out flag looks for protoc-gen-go-grpc.
It seems that instead of protoc having well defined, documented flags, and erroring when an incorrect flag is provided they're using the flags to determine which plugin to use. If you misspell the flag, it looks for the wrong plugin.
Related
go version: go version go1.14 linux/amd64
go.mod
module [redacted]
go 1.14
require (
github.com/golang/protobuf v1.4.0-rc.2
google.golang.org/grpc v1.27.1
google.golang.org/protobuf v1.20.0 // indirect
)
I am running the following command:
protoc -I ./src/pbdefs/protos/ --go-grpc_out=. src/pbdefs/protos/*.proto
to generate my GRPC output files from .proto files, with I am getting an error
protoc-gen-go-grpc: program not found or is not executable
Please specify a program using absolute path or make sure the program is available in your PATH system variable
--go-grpc_out: protoc-gen-go-grpc: Plugin failed with status code 1.
the missing plugin has been implemented at https://github.com/grpc/grpc-go.
command below should fix it
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc#latest
The Golang Protobuf has released a new version of Go protocol buffers which they are calling it as the APIv2.
Because APIv2 is not backwards compatible with APIv1, therefore we will need to adapt all our Golang code for the new API.
You can learn more about the new API here and here
Migration steps:
In order to run the new code generation will be necessary to install the following gRPC gen plugins:
go get -u google.golang.org/protobuf/cmd/protoc-gen-go
go install google.golang.org/protobuf/cmd/protoc-gen-go
go get -u google.golang.org/grpc/cmd/protoc-gen-go-grpc
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc
Then use the following command to generate the code.
# generate the messages
protoc --go_out="$GO_GEN_PATH" -I "$dependecies" "$proto"
# generate the services
protoc --go-grpc_out="$GO_GEN_PATH" -I "$dependecies" "$proto"
OK, just found out, as per https://github.com/protocolbuffers/protobuf-go/releases/tag/v1.20.0
The v1.20 protoc-gen-go does not support generating gRPC service definitions. In the future, gRPC service generation will be supported by a new protoc-gen-go-grpc plugin provided by the Go gRPC project.
The github.com/golang/protobuf version of protoc-gen-go continues to support gRPC and will continue to do so for the foreseeable future.
EDIT 29/05/2020:
Following an update from #Mark in the comments, according to the Tracking issue on github, protoc-gen-go-grpc has now been merged. However, according to the same issue:
Even with this merged, it is unreleased (v0.0) and subject to change. We do plan to add a requirement that the Unimplemented service implementation be included in all registered services, with an escape hatch in the form of a command-line arg as suggested by #neild.
If you haven't done, so you need to install the protoc-gen-go plugin like so:
go get github.com/golang/protobuf/protoc-gen-go
this will install the plugin (if your GOPATH is ${HOME}/go) to:
${HOME}/go/bin/protoc-gen-go
Then when running protoc, either update your path, or set it dynamically like so:
PATH="${PATH}:${HOME}/go/bin" protoc ...
For all who aren't much into the topic (like me) and still have trouble to figure out a working solution, here's a step-by-step approach:
apt install protobuf-compiler installs the compiler under apt install protobuf-compiler, available via protoc from then.
Install the old go generator plugin to be used by protoc: go get -u google.golang.org/protobuf/cmd/protoc-gen-go and go install google.golang.org/protobuf/cmd/protoc-gen-go. Also, make sure that the installed plugin can be found in $PATH or add it with export PATH=$PATH:$(go env GOPATH)/bin if needed.
To tell that plugin not only to generate the protobuf message type information but also the grcp methods, use a command like protoc --go_out=plugins=grpc:my/relative/output/path ./my_file.proto.
Looks kinda trivial once you've figured that out, but it's quite hard to figure that out if you aren't into that topic and only have scarce information about how the go files generator generator and the grcp extension are supposed to work together.
Share my useful bash command here:
go get -u google.golang.org/protobuf/cmd/protoc-gen-go
go install google.golang.org/protobuf/cmd/protoc-gen-go
go get -u google.golang.org/grpc/cmd/protoc-gen-go-grpc
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc
protoc --proto_path=./proto ./proto/*.proto --plugin=$(go env GOPATH)/bin/protoc-gen-go-grpc --go-grpc_out=./pb
protoc --proto_path=./proto ./proto/*.proto --plugin=$(go env GOPATH)/bin/protoc-gen-go --go_out=./pb
Update your PATH so that the protoc compiler can find the plugins:
export PATH="$PATH:$(go env GOPATH)/bin"
Please check the Go Environment variables by running the Go env command on the Terminal and make sure that the following values are set.
GOBIN=some folder location
GOPATH= some folder location
If these are looking good try installing the go plugin
go get -u google.golang.org/grpc
or run the following on the terminal
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc#latest
I was searching for a good answer and finally, it worked for me:
protoc --go-grpc_out=. file_path/file_name*.proto
i solve using this command :
protoc calculator/calculatorpb/calculator.proto --go-grpc_out=.
protoc -I=. --go_out=. calculator/calculatorpb/calculator.proto
to generate calculator_pb.go and calculator_grpc.go
syntax = "proto3";
package calculator;
option go_package = "calculator/calculatorpb";
message SumRequest {
int32 first_number = 1;
int32 second_number = 2;
}
message SumResponse {
int32 sum_result =1 ;
}
service CalculatorService{
rpc Sum(SumRequest) returns(SumResponse) {}
}
I had to install all these:
go install \
github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-grpc-gateway \
github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv2 \
google.golang.org/protobuf/cmd/protoc-gen-go \
google.golang.org/grpc/cmd/protoc-gen-go-grpc
Use go get to download the following packages:
$ go get github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-grpc-gateway
$ go get google.golang.org/protobuf/cmd/protoc-gen-go
$ go get google.golang.org/grpc/cmd/protoc-gen-go-grpc
This installs the protoc generator plugins we need to generate the stubs. Make sure to add $GOPATH/bin to your $PATH so that executables installed via go get are available on your $PATH.
Here's an example of what a protoc command might look like to generate Go stubs, assuming that you're at the root of your repository and you have your proto files in a directory called proto:
$ protoc -I ./proto \
--go_out ./proto --go_opt paths=source_relative \
--go-grpc_out ./proto --go-grpc_opt paths=source_relative \
./proto/helloworld/hello_world.proto
We use the go and go-grpc plugins to generate Go types and gRPC service definitions. We're outputting the generated files relative to the proto folder, and we're using the paths=source_relative option, which means that the generated files will appear in the same directory as the source .proto file.
You should check out the tutorial series on gRPC-Gateway, i.e., https://grpc-ecosystem.github.io/grpc-gateway/docs/tutorials/. Also, you can refer to my simple hello world program, which uses gRPC-Gateway, i.e., https://github.com/iamrajiv/helloworld-grpc-gateway.
You can also use https://github.com/storyicon/powerproto to install all protoc-related dependencies (including protoc-gen-go-grpc) with one click and for version control.
I was able to solve a problem in 2022 by finding an answer in a GitHub issue
https://github.com/golang/protobuf/issues/795
As #neild suggested,
I added both GOPATH and GOROOT to my PATH environment variable by using the command
'export PATH=$PATH:$HOME/go/bin' for GOPATH
'export PATH=$PATH:/usr/local/go/bin' for GOROOT
and it works for me
I'm trying to write some unit tests for a poc I'm doing in Golang / Kafka on a new M1 Mac. I'm using the official Golang Kafka libs from confluent:
"github.com/confluentinc/confluent-kafka-go/kafka"
Apparently, this package has a dependency on a librdkafka which is not built for M1 (yet?). For the build, there is a work around here, which goes something like this:
% brew install librdkafka openssl zstd
% PKG_CONFIG_PATH="/opt/homebrew/opt/openssl#3/lib/pkgconfig"
% go build -tags dynamic *yadda yadda yadda*
This is fine for build/run. Unfortunately, it doesn't seem to work for tests. In the link describing the workaround, using go test -tags dynamic ./... seems to work, but in my case the test run doesn't seem to read the exported PKG_CONFIG_PATH:
% go test -tags dynamic ./... -v
# pkg-config --cflags -- rdkafka
Package libcrypto was not found in the pkg-config search path.
Perhaps you should add the directory containing `libcrypto.pc'
to the PKG_CONFIG_PATH environment variable
Package 'libcrypto', required by 'rdkafka', not found
pkg-config: exit status 1
FAIL smartAC/shared [build failed]
Even though that env var is set, at least in my shell:
% echo $PKG_CONFIG_PATH
/opt/homebrew/opt/openssl#3/lib/pkgconfig
Is there some trick to get go test tool to see the env var?
Ok, never mind. I sorted this... in my ~/.zshrc I wasn't exporting the PKG_CONFIG_PATH, so I changed this:
PKG_CONFIG_PATH="/opt/homebrew/opt/openssl#3/lib/pkgconfig"
to this:
export PKG_CONFIG_PATH="/opt/homebrew/opt/openssl#3/lib/pkgconfig"
Which seems to work. Leaving the question up, just in case it might help some other noob like me :-).
We are using protoc-gen-go v2 (v2 is google.golang.org/protobuf, v1 is github.com/golang/protobuf)
When we are trying to compile our gRPC services, it tells us that plugins is not supported anymore and we should instead use --go-grpc_out:
$ protoc --go_out=plugins=grpc:. *.proto
--go_out: protoc-gen-go: plugins are not supported; use 'protoc --go-grpc_out=...' to generate gRPC
And when using --go-grpc_out, it is telling us we need the protoc-gen-go-rpc:
$ protoc --go-rpc_out=. *.proto
protoc-gen-go-rpc: program not found or is not executable
Please specify a program using absolute path or make sure the program is available in your PATH system variable
Where do you find, or how do you install protoc-gen-go-rpc?
Update: We also just found this post on Reddit for additional information
https://www.reddit.com/r/golang/comments/fe3a4k/documentation_on_getting_grpc_working_with_the/
I'm not sure about protoc-gen-go v2, because the latest protobuf tag I can see is v1.4.
Here is the example how we call protoc to generate source code for sevice called crab that has no external dependencies (e.g. no external protofile imports). The only binary dependency is protoc-gen-go
protoc -I $(pwd) -I /home/isaev/go/src \
$(pwd)/crab/error_codes.proto \
$(pwd)/crab/goproxy.proto \
$(pwd)/crab/crab.proto \
--go_out=plugins=grpc:/tmp/crab
go version: go version go1.14 linux/amd64
go.mod
module [redacted]
go 1.14
require (
github.com/golang/protobuf v1.4.0-rc.2
google.golang.org/grpc v1.27.1
google.golang.org/protobuf v1.20.0 // indirect
)
I am running the following command:
protoc -I ./src/pbdefs/protos/ --go-grpc_out=. src/pbdefs/protos/*.proto
to generate my GRPC output files from .proto files, with I am getting an error
protoc-gen-go-grpc: program not found or is not executable
Please specify a program using absolute path or make sure the program is available in your PATH system variable
--go-grpc_out: protoc-gen-go-grpc: Plugin failed with status code 1.
the missing plugin has been implemented at https://github.com/grpc/grpc-go.
command below should fix it
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc#latest
The Golang Protobuf has released a new version of Go protocol buffers which they are calling it as the APIv2.
Because APIv2 is not backwards compatible with APIv1, therefore we will need to adapt all our Golang code for the new API.
You can learn more about the new API here and here
Migration steps:
In order to run the new code generation will be necessary to install the following gRPC gen plugins:
go get -u google.golang.org/protobuf/cmd/protoc-gen-go
go install google.golang.org/protobuf/cmd/protoc-gen-go
go get -u google.golang.org/grpc/cmd/protoc-gen-go-grpc
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc
Then use the following command to generate the code.
# generate the messages
protoc --go_out="$GO_GEN_PATH" -I "$dependecies" "$proto"
# generate the services
protoc --go-grpc_out="$GO_GEN_PATH" -I "$dependecies" "$proto"
OK, just found out, as per https://github.com/protocolbuffers/protobuf-go/releases/tag/v1.20.0
The v1.20 protoc-gen-go does not support generating gRPC service definitions. In the future, gRPC service generation will be supported by a new protoc-gen-go-grpc plugin provided by the Go gRPC project.
The github.com/golang/protobuf version of protoc-gen-go continues to support gRPC and will continue to do so for the foreseeable future.
EDIT 29/05/2020:
Following an update from #Mark in the comments, according to the Tracking issue on github, protoc-gen-go-grpc has now been merged. However, according to the same issue:
Even with this merged, it is unreleased (v0.0) and subject to change. We do plan to add a requirement that the Unimplemented service implementation be included in all registered services, with an escape hatch in the form of a command-line arg as suggested by #neild.
If you haven't done, so you need to install the protoc-gen-go plugin like so:
go get github.com/golang/protobuf/protoc-gen-go
this will install the plugin (if your GOPATH is ${HOME}/go) to:
${HOME}/go/bin/protoc-gen-go
Then when running protoc, either update your path, or set it dynamically like so:
PATH="${PATH}:${HOME}/go/bin" protoc ...
For all who aren't much into the topic (like me) and still have trouble to figure out a working solution, here's a step-by-step approach:
apt install protobuf-compiler installs the compiler under apt install protobuf-compiler, available via protoc from then.
Install the old go generator plugin to be used by protoc: go get -u google.golang.org/protobuf/cmd/protoc-gen-go and go install google.golang.org/protobuf/cmd/protoc-gen-go. Also, make sure that the installed plugin can be found in $PATH or add it with export PATH=$PATH:$(go env GOPATH)/bin if needed.
To tell that plugin not only to generate the protobuf message type information but also the grcp methods, use a command like protoc --go_out=plugins=grpc:my/relative/output/path ./my_file.proto.
Looks kinda trivial once you've figured that out, but it's quite hard to figure that out if you aren't into that topic and only have scarce information about how the go files generator generator and the grcp extension are supposed to work together.
Share my useful bash command here:
go get -u google.golang.org/protobuf/cmd/protoc-gen-go
go install google.golang.org/protobuf/cmd/protoc-gen-go
go get -u google.golang.org/grpc/cmd/protoc-gen-go-grpc
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc
protoc --proto_path=./proto ./proto/*.proto --plugin=$(go env GOPATH)/bin/protoc-gen-go-grpc --go-grpc_out=./pb
protoc --proto_path=./proto ./proto/*.proto --plugin=$(go env GOPATH)/bin/protoc-gen-go --go_out=./pb
Update your PATH so that the protoc compiler can find the plugins:
export PATH="$PATH:$(go env GOPATH)/bin"
Please check the Go Environment variables by running the Go env command on the Terminal and make sure that the following values are set.
GOBIN=some folder location
GOPATH= some folder location
If these are looking good try installing the go plugin
go get -u google.golang.org/grpc
or run the following on the terminal
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc#latest
I was searching for a good answer and finally, it worked for me:
protoc --go-grpc_out=. file_path/file_name*.proto
i solve using this command :
protoc calculator/calculatorpb/calculator.proto --go-grpc_out=.
protoc -I=. --go_out=. calculator/calculatorpb/calculator.proto
to generate calculator_pb.go and calculator_grpc.go
syntax = "proto3";
package calculator;
option go_package = "calculator/calculatorpb";
message SumRequest {
int32 first_number = 1;
int32 second_number = 2;
}
message SumResponse {
int32 sum_result =1 ;
}
service CalculatorService{
rpc Sum(SumRequest) returns(SumResponse) {}
}
I had to install all these:
go install \
github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-grpc-gateway \
github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv2 \
google.golang.org/protobuf/cmd/protoc-gen-go \
google.golang.org/grpc/cmd/protoc-gen-go-grpc
Use go get to download the following packages:
$ go get github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-grpc-gateway
$ go get google.golang.org/protobuf/cmd/protoc-gen-go
$ go get google.golang.org/grpc/cmd/protoc-gen-go-grpc
This installs the protoc generator plugins we need to generate the stubs. Make sure to add $GOPATH/bin to your $PATH so that executables installed via go get are available on your $PATH.
Here's an example of what a protoc command might look like to generate Go stubs, assuming that you're at the root of your repository and you have your proto files in a directory called proto:
$ protoc -I ./proto \
--go_out ./proto --go_opt paths=source_relative \
--go-grpc_out ./proto --go-grpc_opt paths=source_relative \
./proto/helloworld/hello_world.proto
We use the go and go-grpc plugins to generate Go types and gRPC service definitions. We're outputting the generated files relative to the proto folder, and we're using the paths=source_relative option, which means that the generated files will appear in the same directory as the source .proto file.
You should check out the tutorial series on gRPC-Gateway, i.e., https://grpc-ecosystem.github.io/grpc-gateway/docs/tutorials/. Also, you can refer to my simple hello world program, which uses gRPC-Gateway, i.e., https://github.com/iamrajiv/helloworld-grpc-gateway.
You can also use https://github.com/storyicon/powerproto to install all protoc-related dependencies (including protoc-gen-go-grpc) with one click and for version control.
I was able to solve a problem in 2022 by finding an answer in a GitHub issue
https://github.com/golang/protobuf/issues/795
As #neild suggested,
I added both GOPATH and GOROOT to my PATH environment variable by using the command
'export PATH=$PATH:$HOME/go/bin' for GOPATH
'export PATH=$PATH:/usr/local/go/bin' for GOROOT
and it works for me
I've installed these packages:
google.golang.org/grpc
github.com/golang/protobuf/protoc-gen-go
and exported path like this:
export PATH=$PATH:/usr/local/go/bin
when I try compile proto file with protoc command, I see command not found error:
protoc --go_out=. helloworld/helloworld.proto
zsh: command not found: protoc
my project path is like: /home/my-username/go/src/github.com/my-username/helloworld
my go version: go1.12.5
and I use ubuntu 18.04
When I install it with binary file it works, but compiled go file does not contain some functions like: RegisterGreeterServer or NewGreeterClient
zsh: command not found: protoc indicates that protoc is not installed on your machine. To do so, you need to download binary from Official Releases, as you are on an ubuntu machine, I suggest you download protoc-3.7.1-linux-x86_64.zip (This is latest protoc at the time of writing this answer, you should check on the releases and download latest version)
You can download via browser or use following command:
wget "https://github.com/protocolbuffers/protobuf/releases/download/v3.7.1/protoc-3.7.1-linux-x86_64.zip" -O protoc-3.7.1-linux-x86_64.zip
Now unzip, you'll get two folders, "bin" and "include".
Copy bin/protoc to /usr/local/bin/protoc and include/google to /usr/local/include/google
This will properly install protoc on your machine.
To see if it got installed properly, try executing protoc command on your terminal. You should get something like following
If you still face any issues, please let me know.
Hope this helps!
Finally with help of Amit, I installed protoc. but when I compiled proto file with this command
protoc --go_out=. add/add.proto
go compiled file did not contain some functions like: RegisterGreeterServer or NewGreeterClient for instance.
by this reply I found out issue and added plugins=grpc, then tried this command and it worked:
protoc --go_out=plugins=grpc:. add/add.proto
I know its too late to discuss about it, but just in case it might be helpful for some one else, you can download protobuf for golang from github address Github Repo
and by navigating to {$LIB_PATH}/protobuf/protoc-gen-go and running "go build ." having a compiled binary from generator, and then add it to your path for feature usage