I'm trying to write some unit tests for a poc I'm doing in Golang / Kafka on a new M1 Mac. I'm using the official Golang Kafka libs from confluent:
"github.com/confluentinc/confluent-kafka-go/kafka"
Apparently, this package has a dependency on a librdkafka which is not built for M1 (yet?). For the build, there is a work around here, which goes something like this:
% brew install librdkafka openssl zstd
% PKG_CONFIG_PATH="/opt/homebrew/opt/openssl#3/lib/pkgconfig"
% go build -tags dynamic *yadda yadda yadda*
This is fine for build/run. Unfortunately, it doesn't seem to work for tests. In the link describing the workaround, using go test -tags dynamic ./... seems to work, but in my case the test run doesn't seem to read the exported PKG_CONFIG_PATH:
% go test -tags dynamic ./... -v
# pkg-config --cflags -- rdkafka
Package libcrypto was not found in the pkg-config search path.
Perhaps you should add the directory containing `libcrypto.pc'
to the PKG_CONFIG_PATH environment variable
Package 'libcrypto', required by 'rdkafka', not found
pkg-config: exit status 1
FAIL smartAC/shared [build failed]
Even though that env var is set, at least in my shell:
% echo $PKG_CONFIG_PATH
/opt/homebrew/opt/openssl#3/lib/pkgconfig
Is there some trick to get go test tool to see the env var?
Ok, never mind. I sorted this... in my ~/.zshrc I wasn't exporting the PKG_CONFIG_PATH, so I changed this:
PKG_CONFIG_PATH="/opt/homebrew/opt/openssl#3/lib/pkgconfig"
to this:
export PKG_CONFIG_PATH="/opt/homebrew/opt/openssl#3/lib/pkgconfig"
Which seems to work. Leaving the question up, just in case it might help some other noob like me :-).
Related
go version: go version go1.14 linux/amd64
go.mod
module [redacted]
go 1.14
require (
github.com/golang/protobuf v1.4.0-rc.2
google.golang.org/grpc v1.27.1
google.golang.org/protobuf v1.20.0 // indirect
)
I am running the following command:
protoc -I ./src/pbdefs/protos/ --go-grpc_out=. src/pbdefs/protos/*.proto
to generate my GRPC output files from .proto files, with I am getting an error
protoc-gen-go-grpc: program not found or is not executable
Please specify a program using absolute path or make sure the program is available in your PATH system variable
--go-grpc_out: protoc-gen-go-grpc: Plugin failed with status code 1.
the missing plugin has been implemented at https://github.com/grpc/grpc-go.
command below should fix it
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc#latest
The Golang Protobuf has released a new version of Go protocol buffers which they are calling it as the APIv2.
Because APIv2 is not backwards compatible with APIv1, therefore we will need to adapt all our Golang code for the new API.
You can learn more about the new API here and here
Migration steps:
In order to run the new code generation will be necessary to install the following gRPC gen plugins:
go get -u google.golang.org/protobuf/cmd/protoc-gen-go
go install google.golang.org/protobuf/cmd/protoc-gen-go
go get -u google.golang.org/grpc/cmd/protoc-gen-go-grpc
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc
Then use the following command to generate the code.
# generate the messages
protoc --go_out="$GO_GEN_PATH" -I "$dependecies" "$proto"
# generate the services
protoc --go-grpc_out="$GO_GEN_PATH" -I "$dependecies" "$proto"
OK, just found out, as per https://github.com/protocolbuffers/protobuf-go/releases/tag/v1.20.0
The v1.20 protoc-gen-go does not support generating gRPC service definitions. In the future, gRPC service generation will be supported by a new protoc-gen-go-grpc plugin provided by the Go gRPC project.
The github.com/golang/protobuf version of protoc-gen-go continues to support gRPC and will continue to do so for the foreseeable future.
EDIT 29/05/2020:
Following an update from #Mark in the comments, according to the Tracking issue on github, protoc-gen-go-grpc has now been merged. However, according to the same issue:
Even with this merged, it is unreleased (v0.0) and subject to change. We do plan to add a requirement that the Unimplemented service implementation be included in all registered services, with an escape hatch in the form of a command-line arg as suggested by #neild.
If you haven't done, so you need to install the protoc-gen-go plugin like so:
go get github.com/golang/protobuf/protoc-gen-go
this will install the plugin (if your GOPATH is ${HOME}/go) to:
${HOME}/go/bin/protoc-gen-go
Then when running protoc, either update your path, or set it dynamically like so:
PATH="${PATH}:${HOME}/go/bin" protoc ...
For all who aren't much into the topic (like me) and still have trouble to figure out a working solution, here's a step-by-step approach:
apt install protobuf-compiler installs the compiler under apt install protobuf-compiler, available via protoc from then.
Install the old go generator plugin to be used by protoc: go get -u google.golang.org/protobuf/cmd/protoc-gen-go and go install google.golang.org/protobuf/cmd/protoc-gen-go. Also, make sure that the installed plugin can be found in $PATH or add it with export PATH=$PATH:$(go env GOPATH)/bin if needed.
To tell that plugin not only to generate the protobuf message type information but also the grcp methods, use a command like protoc --go_out=plugins=grpc:my/relative/output/path ./my_file.proto.
Looks kinda trivial once you've figured that out, but it's quite hard to figure that out if you aren't into that topic and only have scarce information about how the go files generator generator and the grcp extension are supposed to work together.
Share my useful bash command here:
go get -u google.golang.org/protobuf/cmd/protoc-gen-go
go install google.golang.org/protobuf/cmd/protoc-gen-go
go get -u google.golang.org/grpc/cmd/protoc-gen-go-grpc
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc
protoc --proto_path=./proto ./proto/*.proto --plugin=$(go env GOPATH)/bin/protoc-gen-go-grpc --go-grpc_out=./pb
protoc --proto_path=./proto ./proto/*.proto --plugin=$(go env GOPATH)/bin/protoc-gen-go --go_out=./pb
Update your PATH so that the protoc compiler can find the plugins:
export PATH="$PATH:$(go env GOPATH)/bin"
Please check the Go Environment variables by running the Go env command on the Terminal and make sure that the following values are set.
GOBIN=some folder location
GOPATH= some folder location
If these are looking good try installing the go plugin
go get -u google.golang.org/grpc
or run the following on the terminal
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc#latest
I was searching for a good answer and finally, it worked for me:
protoc --go-grpc_out=. file_path/file_name*.proto
i solve using this command :
protoc calculator/calculatorpb/calculator.proto --go-grpc_out=.
protoc -I=. --go_out=. calculator/calculatorpb/calculator.proto
to generate calculator_pb.go and calculator_grpc.go
syntax = "proto3";
package calculator;
option go_package = "calculator/calculatorpb";
message SumRequest {
int32 first_number = 1;
int32 second_number = 2;
}
message SumResponse {
int32 sum_result =1 ;
}
service CalculatorService{
rpc Sum(SumRequest) returns(SumResponse) {}
}
I had to install all these:
go install \
github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-grpc-gateway \
github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv2 \
google.golang.org/protobuf/cmd/protoc-gen-go \
google.golang.org/grpc/cmd/protoc-gen-go-grpc
Use go get to download the following packages:
$ go get github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-grpc-gateway
$ go get google.golang.org/protobuf/cmd/protoc-gen-go
$ go get google.golang.org/grpc/cmd/protoc-gen-go-grpc
This installs the protoc generator plugins we need to generate the stubs. Make sure to add $GOPATH/bin to your $PATH so that executables installed via go get are available on your $PATH.
Here's an example of what a protoc command might look like to generate Go stubs, assuming that you're at the root of your repository and you have your proto files in a directory called proto:
$ protoc -I ./proto \
--go_out ./proto --go_opt paths=source_relative \
--go-grpc_out ./proto --go-grpc_opt paths=source_relative \
./proto/helloworld/hello_world.proto
We use the go and go-grpc plugins to generate Go types and gRPC service definitions. We're outputting the generated files relative to the proto folder, and we're using the paths=source_relative option, which means that the generated files will appear in the same directory as the source .proto file.
You should check out the tutorial series on gRPC-Gateway, i.e., https://grpc-ecosystem.github.io/grpc-gateway/docs/tutorials/. Also, you can refer to my simple hello world program, which uses gRPC-Gateway, i.e., https://github.com/iamrajiv/helloworld-grpc-gateway.
You can also use https://github.com/storyicon/powerproto to install all protoc-related dependencies (including protoc-gen-go-grpc) with one click and for version control.
I was able to solve a problem in 2022 by finding an answer in a GitHub issue
https://github.com/golang/protobuf/issues/795
As #neild suggested,
I added both GOPATH and GOROOT to my PATH environment variable by using the command
'export PATH=$PATH:$HOME/go/bin' for GOPATH
'export PATH=$PATH:/usr/local/go/bin' for GOROOT
and it works for me
i'm trying to install a library inside a package. However I don't understand where it compiles to.
Structure is like so:
package/cmd/library
I can install other executable targets fine with go install. My paths are set correctly. However now I want to build my shared library target and deploy it somewhere (this deployment step can be done manually). I'm running into two different issues.
Issue one, I can't seem to install it at all:
go install -buildmode=c-shared bpackage/cmd/library#latest
Returns with:
go install: no install location for directory /home/tpm/go/pkg/mod/package/cmd/library outside GOPATH
For more details see: 'go help gopath'
which tells me that it installs somewhere other than in my gopath, I'm just not sure where that might be.
Issue 2, using the -o flag doesn't work with go install, so I can't seem to alter the output location to place it inside the GOPATH (i did try setting the GOBIN to within my gopath, but since other commands work fine I don't think this should be causing any issue)
Quoting Ian from https://github.com/golang/go/issues/24253
Note that it doesn't make a great deal of sense to use go install -buildmode=c-shared. The expectation is that people will use go build -buildmode=c-shared -o foo.so. The only point of using -buildmode=c-shared is to use the shared library somewhere, and using go install without -o will put the shared library in a relatively unpredictable place.
I need to build a Go (1.13+ due to Modules) binary with Travis. The trick is, that I need to use CGO.
Thus, I have a language: cpp and Focal config. The default packaged version with Travis is Go 1.11, though.
So I tried installing Go 1.13:
using apt:
Setting up golang (2:1.13~1ubuntu2) ...
using gimme:
unset GOOS;
unset GOARCH;
export GOROOT='/home/travis/.gimme/versions/go1.13.1.linux.amd64';
export PATH="/home/travis/.gimme/versions/go1.13.1.linux.amd64/bin:${PATH}";
go version >&2;
export GIMME_ENV="/home/travis/.gimme/envs/go1.13.1.env"
When I try to call these specific versions I get errors:
compile: version "go1.11.1" does not match go tool version "go1.13.1"
This is probably due to PATH and other customizations in Travis-Go since:
$ which go
/home/travis/.gimme/versions/go1.11.1.linux.amd64/bin/go
It even fails when with the same error when trying to fix PATH via:
- export PATH="/home/travis/.gimme/versions/go1.13.1.linux.amd64/bin:${PATH}"
- go build
Does anyone know how to override the default Travis Go version or at least reverse Travis Go-changes so that apt Go could be used?
EDIT:
Added some more example output of the Travis instance.
It of course was a very obvious workaround.
Had to also e.g. set:
env:
global:
- GOROOT='/home/travis/.gimme/versions/go1.13.1.linux.amd64'
Still would appreciate any solution that makes this less hacky.
go version: go version go1.14 linux/amd64
go.mod
module [redacted]
go 1.14
require (
github.com/golang/protobuf v1.4.0-rc.2
google.golang.org/grpc v1.27.1
google.golang.org/protobuf v1.20.0 // indirect
)
I am running the following command:
protoc -I ./src/pbdefs/protos/ --go-grpc_out=. src/pbdefs/protos/*.proto
to generate my GRPC output files from .proto files, with I am getting an error
protoc-gen-go-grpc: program not found or is not executable
Please specify a program using absolute path or make sure the program is available in your PATH system variable
--go-grpc_out: protoc-gen-go-grpc: Plugin failed with status code 1.
the missing plugin has been implemented at https://github.com/grpc/grpc-go.
command below should fix it
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc#latest
The Golang Protobuf has released a new version of Go protocol buffers which they are calling it as the APIv2.
Because APIv2 is not backwards compatible with APIv1, therefore we will need to adapt all our Golang code for the new API.
You can learn more about the new API here and here
Migration steps:
In order to run the new code generation will be necessary to install the following gRPC gen plugins:
go get -u google.golang.org/protobuf/cmd/protoc-gen-go
go install google.golang.org/protobuf/cmd/protoc-gen-go
go get -u google.golang.org/grpc/cmd/protoc-gen-go-grpc
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc
Then use the following command to generate the code.
# generate the messages
protoc --go_out="$GO_GEN_PATH" -I "$dependecies" "$proto"
# generate the services
protoc --go-grpc_out="$GO_GEN_PATH" -I "$dependecies" "$proto"
OK, just found out, as per https://github.com/protocolbuffers/protobuf-go/releases/tag/v1.20.0
The v1.20 protoc-gen-go does not support generating gRPC service definitions. In the future, gRPC service generation will be supported by a new protoc-gen-go-grpc plugin provided by the Go gRPC project.
The github.com/golang/protobuf version of protoc-gen-go continues to support gRPC and will continue to do so for the foreseeable future.
EDIT 29/05/2020:
Following an update from #Mark in the comments, according to the Tracking issue on github, protoc-gen-go-grpc has now been merged. However, according to the same issue:
Even with this merged, it is unreleased (v0.0) and subject to change. We do plan to add a requirement that the Unimplemented service implementation be included in all registered services, with an escape hatch in the form of a command-line arg as suggested by #neild.
If you haven't done, so you need to install the protoc-gen-go plugin like so:
go get github.com/golang/protobuf/protoc-gen-go
this will install the plugin (if your GOPATH is ${HOME}/go) to:
${HOME}/go/bin/protoc-gen-go
Then when running protoc, either update your path, or set it dynamically like so:
PATH="${PATH}:${HOME}/go/bin" protoc ...
For all who aren't much into the topic (like me) and still have trouble to figure out a working solution, here's a step-by-step approach:
apt install protobuf-compiler installs the compiler under apt install protobuf-compiler, available via protoc from then.
Install the old go generator plugin to be used by protoc: go get -u google.golang.org/protobuf/cmd/protoc-gen-go and go install google.golang.org/protobuf/cmd/protoc-gen-go. Also, make sure that the installed plugin can be found in $PATH or add it with export PATH=$PATH:$(go env GOPATH)/bin if needed.
To tell that plugin not only to generate the protobuf message type information but also the grcp methods, use a command like protoc --go_out=plugins=grpc:my/relative/output/path ./my_file.proto.
Looks kinda trivial once you've figured that out, but it's quite hard to figure that out if you aren't into that topic and only have scarce information about how the go files generator generator and the grcp extension are supposed to work together.
Share my useful bash command here:
go get -u google.golang.org/protobuf/cmd/protoc-gen-go
go install google.golang.org/protobuf/cmd/protoc-gen-go
go get -u google.golang.org/grpc/cmd/protoc-gen-go-grpc
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc
protoc --proto_path=./proto ./proto/*.proto --plugin=$(go env GOPATH)/bin/protoc-gen-go-grpc --go-grpc_out=./pb
protoc --proto_path=./proto ./proto/*.proto --plugin=$(go env GOPATH)/bin/protoc-gen-go --go_out=./pb
Update your PATH so that the protoc compiler can find the plugins:
export PATH="$PATH:$(go env GOPATH)/bin"
Please check the Go Environment variables by running the Go env command on the Terminal and make sure that the following values are set.
GOBIN=some folder location
GOPATH= some folder location
If these are looking good try installing the go plugin
go get -u google.golang.org/grpc
or run the following on the terminal
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc#latest
I was searching for a good answer and finally, it worked for me:
protoc --go-grpc_out=. file_path/file_name*.proto
i solve using this command :
protoc calculator/calculatorpb/calculator.proto --go-grpc_out=.
protoc -I=. --go_out=. calculator/calculatorpb/calculator.proto
to generate calculator_pb.go and calculator_grpc.go
syntax = "proto3";
package calculator;
option go_package = "calculator/calculatorpb";
message SumRequest {
int32 first_number = 1;
int32 second_number = 2;
}
message SumResponse {
int32 sum_result =1 ;
}
service CalculatorService{
rpc Sum(SumRequest) returns(SumResponse) {}
}
I had to install all these:
go install \
github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-grpc-gateway \
github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv2 \
google.golang.org/protobuf/cmd/protoc-gen-go \
google.golang.org/grpc/cmd/protoc-gen-go-grpc
Use go get to download the following packages:
$ go get github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-grpc-gateway
$ go get google.golang.org/protobuf/cmd/protoc-gen-go
$ go get google.golang.org/grpc/cmd/protoc-gen-go-grpc
This installs the protoc generator plugins we need to generate the stubs. Make sure to add $GOPATH/bin to your $PATH so that executables installed via go get are available on your $PATH.
Here's an example of what a protoc command might look like to generate Go stubs, assuming that you're at the root of your repository and you have your proto files in a directory called proto:
$ protoc -I ./proto \
--go_out ./proto --go_opt paths=source_relative \
--go-grpc_out ./proto --go-grpc_opt paths=source_relative \
./proto/helloworld/hello_world.proto
We use the go and go-grpc plugins to generate Go types and gRPC service definitions. We're outputting the generated files relative to the proto folder, and we're using the paths=source_relative option, which means that the generated files will appear in the same directory as the source .proto file.
You should check out the tutorial series on gRPC-Gateway, i.e., https://grpc-ecosystem.github.io/grpc-gateway/docs/tutorials/. Also, you can refer to my simple hello world program, which uses gRPC-Gateway, i.e., https://github.com/iamrajiv/helloworld-grpc-gateway.
You can also use https://github.com/storyicon/powerproto to install all protoc-related dependencies (including protoc-gen-go-grpc) with one click and for version control.
I was able to solve a problem in 2022 by finding an answer in a GitHub issue
https://github.com/golang/protobuf/issues/795
As #neild suggested,
I added both GOPATH and GOROOT to my PATH environment variable by using the command
'export PATH=$PATH:$HOME/go/bin' for GOPATH
'export PATH=$PATH:/usr/local/go/bin' for GOROOT
and it works for me
If there a relatively simple way to make go + libxml2 + gokogiri work on windows?
I mean that I may be can install it (but at the moment I can not, stuck with Package libxml-2.0 was not found in the pkg-config search path), but then I need to provide my utilite to other people, who will never be able (or would wish ) to install lall libxml2 dependencies, modify PATH etc on windows...
It work flawless on Ubuntu...
I found this https://github.com/moovweb/gokogiri/issues/49 thats funny with installation of Gimp 2 (what?!), but I still cannot make it run with such error, I guess might be issue with PATH, but all PATH are set
$ go get github.com/moovweb/gokogiri
# github.com/moovweb/gokogiri/help
Documents\go\src\github.com\moovweb\gokogiri\help\help.go:6:25: fatal error: lib
xml/tree.h: No such file or directory
#include <libxml/tree.h>
^
compilation terminated.
# github.com/moovweb/gokogiri/xpath
Documents\go\src\github.com\moovweb\gokogiri\xpath\expression.go:4:26: fatal err
or: libxml/xpath.h: No such file or directory
#include <libxml/xpath.h>
^
compilation terminated.
You are struggling because it is hard to combine packages that were built by different people for different purposes and get your environment set up correctly. I think it is best to use MSYS2, an environment for Windows that provides a consistent set of packages for things like gcc, go, libxml2, and iconv. MSYS2 has a package manager (pacman) that helps you easily install them and keep them updated.
I don't do much programming with Go, but I am familiar with MSYS2 and it seems like I was able to get gokogiri installed using MSYS2. You should open MSYS2's "MinGW-w64 Win64 Shell" from the Start menu (mingw64_shell.bat), and try running these commands:
pacman -S mingw-w64-x86_64-{gcc,go,libxml2,iconv}
export GOROOT=/mingw64/
export GOPATH=/c/Users/David/Documents/goproj/
mkdir -p $GOPATH
go get github.com/moovweb/gokogiri
I think GOPATH should be set to the directory of your project. If you run into an error, it might be because some pacman package is required that I didn't list here.
The string mingw-w64-x86_64-{gcc,go,libxml2,iconv} gets expanded by Bash into the following list of packages:
mingw-w64-x86_64-gcc
mingw-w64-x86_64-go
mingw-w64-x86_64-libxml2
mingw-w64-x86_64-iconv
If you are actually using 32-bit Windows, replace x86_64 with i686 in the instructions above.
If you are curious, the scripts for building those packages are here: https://github.com/Alexpux/MINGW-packages
As a disclaimer, I haven't actually compiled any go programs in MSYS2, so there could be big problems I am unaware of.
Also, one of the main developers of MSYS2 (alexpux) said this in the #msys2 IRC chat on 2015-06-21:
We not build go for a long time.
This package in very WIP state
Also see
https://github.com/Alexpux/MINGW-packages/issues/421
So you might need to fix some issues with the MSYS2 Go package and recompile it yourself to really make this work. But you have the PKGBUILD script that was used to build it, so maybe that will be less hard than what you are trying to do right now, which involves compiling/collecting every dependency of gokogiri.
MSYS2 would make your other installation of go, libxml2, and iconv obsolete. You can delete those things once you get your MSYS2 environment working.
If you are using visual studio and want to add dependency to your project then just install it using NuGet Package Manager it's easiest method.
Install command: Install-Package libxml2