Referencing external protos like google/rpc/status.proto - protocol-buffers

I'm wondering how to properly reference external proto files. Say I've got a .proto file which references standard protobuf types such as Timestamp:
syntax = "proto3";
package api;
import "google/protobuf/timestamp.proto";
message ServerTimeResponse {
google.protobuf.Timestamp ts = 1;
}
Easy. Timestamp is automatically available when compiling.
Now I add an external
type, say google.rpc.Status:
syntax = "proto3";
package api;
import "google/protobuf/timestamp.proto";
import "google/rpc/status.proto";
message ServerTimeResponse {
google.protobuf.Timestamp ts = 1;
google.rpc.Status status = 2;
}
Of course we have to tell protoc how to find this file where it is via -I/--proto_path.
My question is this: What is the best practice for actually referencing this file, in particular to make version control happy? There appears not to be a go mod equivalent for protobufs. I've seen it copied verbatim into projects (such as in grpc-gateway) or just referenced from the local filesystem.

I think you sort of answered your own question here. I've done both successfully: manually copied the necessary files in verbatim (from https://github.com/googleapis/googleapis/tree/master/google and https://github.com/protocolbuffers/protobuf/tree/master/src/google/protobuf), and referenced local copies of the files.
If you want to do this and make version control happy, you could add these two repositories as git submodules inside your repository. Just make sure to pass the right locations to protoc using -I. E.g.:
cd $PROJECT_DIR
mkdir third_party && cd third_party
git submodule add https://github.com/googleapis/googleapis/tree/master/google
cd $PROJECT_DIR
<git commit the change>
protoc -I third_party/google <the rest of your protoc command>
As for referencing local copies of the files, and making sure they're present before attempting to build, you may find that adding something like the following to your Makefile will help (this is in a Go build environment):
go get -u github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway
go get -u github.com/golang/protobuf/protoc-gen-go
grpc_gateway_path=$(go list -m -f '{{.Dir}}' github.com/grpc-ecosystem/grpc-gateway)
googleapis_path="$grpc_gateway_path/third_party/googleapis"
protoc -I $googleapis_path --go_out=. <list of input files>

Related

Makefile change directory and call git function

I am using a very basic makefile so far and there is the need to extend it.
Since I only know the most basic stuff about makefiles, I am seeking for help here.
The very top of my makefile defines a few values I use for different builds etc.
The important part here is the EVALFILE. It should link to a binary file and is mostly manually set when calling make like make ... EVALFILE="example.bin"
CC = g++
SRC = *.cpp syzygy/tbprobe.c
LIBS = -pthread -Wl,--whole-archive -lpthread -Wl,--no-whole-archive
FOLDER = bin/
ROOT = ../
NAME = Koivisto
EVALFILE = none
EXE = $(ROOT)$(FOLDER)$(NAME)_$(MAJOR).$(MINOR)
MINOR = 0
MAJOR = 5
In case of no user-input for the EVALFILE, I like to get the file from the submodule in my git repository.
ifeq ($(EVALFILE),none)
pwd:=$(shell pwd); \
cd $(ROOT); \
git submodule update --init; \
cd $(pwd); \
EVALFILE := ../networks/default.net
endif
For this, I want to copy the pwd first to reset it later on. Then I change to the root of the repository, call my git submodule function to update the submodules. The submodule of interested here is ../networks/ which will contain a default.net which should be used as a binary eventually.
Unfortunately the code above seems to not work. When looking at the networks/ folder after I execute the make command, there seems to be no files inside of it. Furthermore the EVALFILEis not being set this way. If I move EVALFILE := ../networks/default.net to the top, it does get set but still nothing inside networks/.
I do assume I am not calling my cd, git.. calls correctly.
I am very happy if someone could help me out here.
You're doing something with the contents of $(EVALFILE). If it doesn't exist, you want to use a default file, and you might need to run a recipe to make that.
Assignments in the makefile aren't run if the variable's set on the command line.
So just do a plain assignment of the default value in the makefile, which you're expecting "is mostly manually set when calling make":
EVALFILE := ../networks/default.net
and supply a rule to make the default if it doesn't already exist:
../networks/default.net:
git -C .. submodule update --init
(with a tab subbed in for leading whitespace since markdown eats tabs).
Then in recipes that need the evalfile, pass it in as a dependency:
mytest: this that $(EVALFILE)
and everything works.

Undefined function despite being defined in imported .proto file

I have 2 .proto files in the same directory such that second.proto is dependent on first.proto
second.proto
import "first.proto"
enum ThingINeed {
...something
}
I have no problem running these commands:
$ protoc --go_out=generatedsources/first -I. first.proto
$ protoc --go_out=generatedsources/second -I. second.proto
so the directory structure looks something like
src
|-first.proto
|-second.proto
|-generatedsources
|-first
|-first.pb.go
|-second
|-second.pb.go
My problem is that when I run
$ cd generatedsources/second
$ go build second.pb.go
I recieve a "./second.pb.go: Undefined: ThingINeed" since second.pb.go uses ThingINeed from first.pb.go (seen in first.proto as well)
I notice that second.pb.go doesn't have an import . "generatedsources/first" line in it. When I put it in manually, it works just fine. But I mean, I'm not supposed to edit these .pb.go files, so was wondering how to fix this. I also would very rather not edit these .proto files.
Would appreciate any help!

How to use predifined protobuf type (i.e. "google/protobuf/timestamp.proto") with gRPC

I'm trying to use google/protobuf/timestamp.proto in with gRPC plugin and Go. This is how I run protoc:
protoc -I ./ ./*.proto --go_out=plugins=grpc:.
And this is my .proto:
#domain.proto
syntax = "proto3";
option java_multiple_files = true;
option java_package = "com.viant.xyz";
option java_outer_classname = "domain";
import "google/protobuf/timestamp.proto";
message Foo {
Timestamp modifiedTime = 1;
...
}
I'm seeing the following errors:
domain.proto: Import "google/protobuf/timestamp.proto" was not found or had errors.
domain.proto:44:5: "Timestamp" is not defined.
Am I missing something, or this is not yet supported?
Add /usr/local/include to include paths to use /usr/local/include/google/api/timestamp.proto:
protoc -I/usr/local/include -I. --go_out=plugins=grpc:. *.proto
As you can see in timestamp.proto, Timestamp exists in package google.protobuf, so you have to modify to use Timestamp like this:
message Foo {
google.protobuf.Timestamp modifiedTime = 1;
...
}
In my case problem was in my Fedora 29 setup.
# Install Protoc compiler. By default it is 3.5.0 version
sudo dnf -y install protoc
This was my bad setup. So i fixed it with following steps. Pay attention to grayed out command lines too.
# Uninstall old 3.5.0 version
sudo dnf remove protobuf
# Make sure you grab the latest version
curl -OL
https://github.com/protocolbuffers/protobuf/releases/download/v3.6.1/protoc-3.6.1-linux-x86_64.zip
# Unzip
unzip protoc-3.6.1-linux-x86_64.zip -d protoc3
# Move protoc to /usr/local/bin/
sudo mv protoc3/bin/* /usr/local/bin/
# Move protoc3/include to /usr/local/include/
sudo mv protoc3/include/* /usr/local/include/
# Optional: change owner
sudo chown $USER /usr/local/bin/protoc
sudo chown -R $USER /usr/local/include/google
After this I am able to use:
import "google/protobuf/timestamp.proto";
message Session {
google.protobuf.Timestamp create_time = 1;
}
It is not fully supported yet, but you can make it work by changing
message Foo {
google.protobuf.Timestamp modifiedTime = 1;
...
}
and by fixing generated file import
import google_protobuf "google/protobuf/timestamp.pb"
to
import google_protobuf "github.com/golang/protobuf/ptypes/timestamp"
After scratching my head for hours, I've found the issue.
My /usr/local/include directory doesn't have /google/protobuf files and without that once can't use predefined types. To solve this issue.
curl -OL https://github.com/google/protobuf/releases/download/v3.2.0/protoc-3.2.0-linux-x86_64.zip
unzip protoc-3.2.0-linux-x86_64.zip -d protoc3
sudo mv protoc3/bin/* /usr/local/bin/
sudo mv protoc3/include/* /usr/local/include/
Now you can simply use this command
protoc -I/usr/local/include -I. --go_out= {output_directory_path} {proto_file_path}
If you are facing this inside an alpine docker image, make sure you do an apk add protobuf-dev before generating your files using protoc.
I work around the problem by passing a Mgoogle/protobuf/timestamp.proto=github.com/golang/protobuf/ptypes/timestamp option to the Go grpc plugin.
In other words, I'm calling
protoc --go_out=plugins=grpc,Mgoogle/protobuf/timestamp.proto=github.com/golang/protobuf/ptypes/timestamp:outputdir input.proto
It's a bit of a hack. "Fortunately" I'm already using lots of Mprotofile=go/pkg/import/path parameters in my build setup, so it was easy to add.
In windows, clone the repository:protobuf.
And run the command
protoc -I=$SRC_DIR -I=$YOUR_CLONE_LOCATION/protobuf/src --go_out=$DST_DIR $SRC_DIR/$SRC_FILE

Using automake with files that may not be present in the source tree

I am transitioning to git-annex for storing data files in my git repo. With git-annex you can clone a repository and exclude the files in the annex until you specifically request them. So, Im trying to create a system in which users can build the src tree without the git-annexed data files being present during the make stage. Currently, we assume the files are present on the users filesystem, so the following snippet of the Makefile.am is sufficient:
foo_DATA=datafile1 datafile2 datafile3
EXTRA_DIST=$(foo_DATA)
However, when using the _DATA primary, an error is generated when the user attempts to make and any of the files in the list are not present. So my question is, is there a way to define the data files so that no error is generated when the data file is not present during make, but behaves exactly as the above code if the files are present during the make install step?
I have tried avoiding the _DATA primary altogether, doing something simple like the follow:
data_files=datafile1 datafile2 datafile3
install:
test -z $(foodir) || mkdir -p $(foodir)
install -c -m 644 $(data_files) $(foodir)
But that is rather hackey. Im hardcoding the "install" binary and the permission flags instead of getting them from the configure script. Ide rather just have automake not fail during the make step if the files were not present, but behave as expected in all other circumstances.
TLDR: Can I define a set of files using the _DATA primary that will not error out during make if the file is not present? But also installs exactly as the _DATA primary would if it were present?
Your current approach seems reasonable. I'd write it as an install-data-local hook, though:
data_files=datafile1 datafile2 datafile3
install-data-local:
test -z $(foodir) || $(MKDIR_P) $(foodir)
$(INSTALL_DATA) $(data_files) $(foodir)
MKDIR_P and INSTALL_DATA should be set up for you by automake.
If you'd rather not do this, the following may work in combination with your first Makefile.am snippet. I've not used git-annex before, so this may be wrong:
$(foo_DATA) :
if test ! -e $#; then git annex get $#; fi

Can I make gotest pass compiler flags?

I have just put together a Go package that is going to be a part in a fairly large system with a lot of shared packages. I was able to get it to compile by writing its Makefile such that the compiler is called with -I flags:
include $(GOROOT)/src/Make.inc
TARG=foobar
GOFILES=\
foobar.go\
foobar:
$(GC) -I$(CURDIR)/../intmath -I$(CURDIR)/../randnum foobar.go
include $(GOROOT)/src/Make.pkg
It compiles just fine, and being a good boy, I wrote a comprehensive set of tests. However, when I try to run the tests with gotest, I get a compile error:
$ gotest
rm -f _test/foobar.a
8g -o _gotest_.8 foobar.go foobar_test.go
foobar.go:4: can't find import: intmath
make: *** [_gotest_.8] Error 1
gotest: "C:\\msys\\bin\\sh.exe -c \"gomake\" \"testpackage\" \"GOTESTFILES=foobar_test.go\"" failed: exit status 2
So, the Go file itself will compile when I use the -I flags to tell it where to find the intmath and randnum packages, but gotest doesn't seem to use the Makefile.
Answering peterSO's question:
foobar.go's import section looks like this:
import (
"intmath"
"randnum"
"container/vector"
)
And the compile works fine as long as I have the -I flags going to the compiler. I have tried to use relative paths, like this:
import (
"../intmath"
"../randnum"
"container/vector"
)
but that just doesn't seem to work.
EDIT: answering further peterSO questions:
GOROOT is set to C:\Go the directory where I have all of the Go stuff -- aside from my source code -- installed. I was expecting the relative path to be relative to the directory in which the source file lives.
My source tree looks like this:
server/
foobar/
randnum/
intmath/
So, while I am open to a different, more Go-idiomatic directory structure, my instinct is to arrange them as peers.
Is there some way that I can nudge gotest into compiling foobar.go with the needed flags?
Create the Windows source code directory structure:
C:\server
C:\server\foobar
C:\server\intnum
For intnum.go:
package intnum
func IntNum() int {
return 42
}
Makefile:
include $(GOROOT)/src/Make.inc
TARG=server/intnum
GOFILES=\
intnum.go\
include $(GOROOT)/src/Make.pkg
Run:
$ cd c/server/intnum
$ make install
For foobar.go:
package foobar
import (
"math"
"server/intnum"
)
func FooBar() float64 {
return float64(intnum.IntNum()) * math.Pi
}
Makefile:
include $(GOROOT)/src/Make.inc
TARG=server/foobar
GOFILES=\
foobar.go\
include $(GOROOT)/src/Make.pkg
Run:
$ cd /c/server/foobar
$ make install
After the install, the intnum.a and foobar.a package files will be in the $GOROOT\pkg\windows_386\server (C:\Go\pkg\windows_386\server) directory`.

Resources