File does not reside within any path specified using proto_path - go

I am testing out importing .proto file from another directory.
$GOPATH/src/A/A.proto
syntax = "proto3";
package A;
message SomeMsg {
string msg = 2;
int64 id = 3;
}
$GOPATH/src/B/B.proto
syntax = "proto3";
package B; import "A/A.proto";
message Msg {
SomeMsg s = 1;
}
I'm doing this:
in folder A:
protoc A.proto --go_out=.
and then in folder B:
protoc B.proto --go_out=. --proto_path=$GOPATH/
But I will get this error:
B.proto: File does not reside within any path specified using --proto_path (or -I). You must specify a --proto_path which encompasses this file. Note that the proto_path must be an exact prefix of the .proto file names -- protoc is too dumb to figure out when two paths (e.g. absolute and relative) are equivalent (it's harder than you think).

Error seems clear enough to me, it is saying that you need to specify the exact directory that B.proto is in
protoc B.proto --go_out=. --proto_path=$GOPATH/src/B
or if you are in folder B already,
protoc B.proto --go_out=.

protoc B.proto --go_out=. --proto_path=$GOPATH/src/B --progo_path=. worked for me.
--progo_path=. may help you too.

Case1: The '..' cannot used in path of .proto file if all paths are in 'absolute format'.
Case2: Another word to explain that error. The relevant path and absolute path cannot use in mixed within -I and path to .proto file, since the 'prefix' means the STRING-PREFIX instead of 'a path can jump to by relevant-path in filesystem'.
=========
It seems that the .proto and -I its resided in should use both relative path or both, otherwise error occured.

In case of a reference to a proto file in a C# project, note that the path to the proto file is case sensitive. In my case a link to the proto file in a .net 4.6 project (VS2022 on Windows), looks like this. When I use uppercase characters in the path, the compiler gives the "File does not reside within any path specified using --proto_path" error, as mentioned. However, I can use relative paths.
<!-- all path characters must be lower case -->
<Protobuf Include="..\path.to.proto\protos\myfile.proto">
<Link>Protos\myfile.proto</Link>
</Protobuf>
Also note that you must change the csproj file manually to be sure that the xml-element is called 'Protobuf' and not 'None'! Of course, only applicable for visual studio project situations.

Related

protoc generated grpc files and gateway files end up in different paths?

I'm a bit baffled about this. File structure:
/Project
generate.sh
gateway/
cmd/
main.go
pkg/
gatewaypb/
proto/
service.proto
My generate.sh appears as such:
set -e
module="github.com/my-org/grpc-project"
// Find all dirs with .proto files in them
for proto in */proto; do
dir=$(echo "$proto"|cut -d"/" -f1)
echo "regenerating generated protobuf code for ${proto}"
protoc --proto_path .\
--go-grpc_out=. --go-grpc_opt=module=${module}\
--go_out=. --go_opt=module=${module}\
"${proto}"/*.proto
echo "creating reverse proxy protobuf code for ${proto}"
protoc -I . --grpc-gateway_out=.\
--grpc-gateway_opt logtostderr=true \
--grpc-gateway_opt paths=source_relative \
"${proto}"/*.proto
done
Correctly enough, the first protoc does as intended, while the second one does not. They output files in different paths when they are supposed to be in the same.
After running generate.sh, I am left with:
/Project
generate.sh
Service/
cmd/
main.go
pkg/
gatewaypb/
*gateway.pb.go
*gateway._grpc.pb.go
proto/
*gateway.pb.gw.go
service.proto
Why does the generated gateway file end up in the proto/ directory?
What do I need to give as a value to grpc-gateway-out for it to end up in /pkg/gatewaypb as the other generated grpc files?
My service.proto file:
syntax = "proto3";
package gateway.api.v1;
option go_package = "github.com/my-org/grpc-project/gateway/pkg/gatewaypb";
import "google/api/annotations.proto";
message StringMessage {
string value = 1;
}
service YourService {
rpc Echo(StringMessage) returns (StringMessage) {
option (google.api.http) = {
post: "/v1/example/echo"
body: "*"
};
}
}
As per the comments you were using the module option (--go_opt=module=${module} & --go-grpc_opt=module=${module}) when generating the protobuf & gRPC files. The option you were using when generating the gateway files (--grpc-gateway_opt paths=source_relative) uses a different algorithm to work out where to store the files; the fix being to use the module option - --grpc-gateway_opt module=${module}.
Here is what the docs have to say about the three options (these specific options are for protobuf generation; the other generators have the same functionality but replace go_ with go-grpc_/grpc-gateway_):
By default, the output file is placed in a directory named after the Go package's import path. For example, a file protos/foo.proto with the above go_package option results in a file named example.com/foo/bar/foo.pb.go.
If the --go_opt=module=$PREFIX flag is given to protoc, the specified directory prefix is removed from the output filename. For example, a file protos/foo.proto with the go_package option "example.com/foo/bar" and the flag --go_opt=module=example.com/foo results in a file named bar/foo.pb.go.
If the --go_opt=paths=source_relative flag is given to protoc, the output file is placed in the same relative directory as the input file. For example, the file protos/foo.proto results in a file named protos/foo.pb.go.

School me on Go packages, imports and protos

I'm mostly familiar with Python and a bazel build environment. I'm trying to rewrite a portion of code in Go and I'm struggling to get the proto imports to align properly.
In github.com/djhedges/exit_speed/gps.proto I have set go_package like so
option go_package = "github.com/djhedges/exit_speed/gps_go_proto";
I've added a reflector.proto which imports gps.proto and reuses some of the messages defined in there.
import "gps.proto";
I compile gps.proto with
protoc -I ./ --go_out=./ --go_opt=paths=source_relative --python_out ./ gps.proto
And compile reflector.proto with
LD_PRELOAD=/usr/lib/arm-linux-gnueabihf/libatomic.so.1 python3 -m grpc_tools.protoc -I ./ --go_out=./ --go_opt=paths=source_relative --python_out ./ --grpc_python_out ./ reflector.proto
And finally in exit_speed/reflector.go I try to import gps_go_proto with
import gpspb "github.com/djhedges/exit_speed/gps_go_proto"
Which errors with
go: finding github.com/djhedges/exit_speed/gps_go_proto latest
reflector.go:21:2: unknown import path "github.com/djhedges/exit_speed/gps_go_proto": cannot find module providing package github.com/djhedges/exit_speed/gps_go_proto
I would prefer to keep gps.proto in the root directory if possible because I have protos logged into data files and I believe moving this breaks the proto parsing. I suppose I could write a migration script.
I'm also confused as to how setup multiple protos with different go packages and imports. Do go packages have to have a dedicated folder?
If I'm understanding go packages correctly the package name must map to a directory. By changing --go_out=./ to --go_out=./gps_go_proto seems to solve my issue without having to move the proto files. It does mean I have some additional directories but that seems okay.
As the Go Specification import section says:
Import declarations
An import declaration states that the source file containing the declaration depends on functionality of the imported package (§Program initialization and execution) and enables access to exported identifiers of that package. The import names an identifier (PackageName) to be used for access and an ImportPath that specifies the package to be imported.
ImportDecl = "import" ( ImportSpec | "(" { ImportSpec ";" } ")" ) .
ImportSpec = [ "." | PackageName ] ImportPath .
ImportPath = string_lit .
(snipped here, and I broke some links; use the first link to get the text with the links). The string literal is the familiar:
import "github.com/..."
part; the PackageName, or a dot, are optional. Returning to the text:
The interpretation of the ImportPath is implementation-dependent but it is typically a substring of the full file name of the compiled package and may be relative to a repository of installed packages.
In your own answer, you add:
If I'm understanding go packages correctly the package name must map to a directory.
That's not necessarily so. It's up to the implementation to decide what the string literal means. In general, though, the Go build system will find a directory full of files, using this path. It may also use a version control system, such as Git, to fill a directory full of files, so that the compiler can find them.
Something that I found a bit surprising, but in the end, makes perfect sense (and is described in the spec), is this: the package name found in the package clause:
package somename
statement at the top of some imported file, is sometimes irrelevant. A directory full of files that implement package x will have them all say package x, of course. But it's the y identifier in the line:
import y "path/to/x"
in your package that supplies the name you use to access exported identifiers. The name you use here, with the import having y as the package name, is y, even if the package itself says package x. The package clause in package x is, in effect, just recommending the name x, so that you can write:
import "path/to/x"
if you'd like to refer to these as x.Thing.
There's no need for the string literal to mention x either: perhaps all the files are in something found via path/to/z, and you write:
import "path/to/z"
and then (still) write x.Thing because files in path/to/z/*.go recommend x as the package name.
How protoc uses this
Regardless of the above, the protobuf Go generators will print, in the generated name*.pb.go* Go code, an import directive. That import directive will read:
import "some/path/here"
You can specify the string literal using option go_package. The protobuf compiler doesn't need to interpret this itself. But, as the documentation notes, these name.pb.go files themselves will be dropped into a directory somewhere. They will also contain a package pkg line, of course.
The location of each output file depends on whether you use --go_opt=paths=source_relative. If you do, the go_package option string becomes irrelevant here.
The .pb.go file generator takes the go_package option string, strips off everything up to and including the last slash, and uses the remaining string for the package line.
The second point means that regardless of what the Go system itself does, how it finds these files, and how you write your own imports, the recommended package name in those files will be the last component of whatever was in the go_package option. So in a sense, all you really need is the final component. But if your Go implementation needs more than that, you can supply more.
The first point means that if you don't use --go_opt=paths=source_relative, the rest of the text in the go_package option matters: the Go generator will try to use that string to produce output files. So if your Go implementation needs more, and you do supply more, and it doesn't match what you want for output file names, you'll definitely need the --go-opt.

Problems running protoc with proto_path to generate Gocode

I am having a little bit of an issue to correctly understand the format of running protoc to generate Go source from proto files.
I have tried
protoc --proto_path=api/auth-service api/auth-service/proto/*.proto --go_out=pluigins=grpc:pb
and
protoc --proto_path=api/auth-service/proto/*.proto --go_out=pluigins=grpc:pb
The above always print
api/auth-service/proto/*.proto: warning: directory does not exist.
or
Missing input file.
eventually, after a lot of playing around it seems that thing works.
protoc --proto_path=api/auth-service api/auth-service/proto/*.proto --go_out=pluigins=grpc:pb
But I am very confused about the actual correct format (passing in the directory twice and the 2nd passing in *.proto)
I tried finding some syntax why it was working this way but I couldn't.
There was also passing -I instead of --proto_path.
You have to run with a named proto ( so replace the * with your proto filename, should fix it)

Is there any way to decompiler google protobuf binary file(.pb file) to .proto file

When I'm reversing an apk I got .pb file but not .proto file, is there any way to decompiler this file to .proto file or can I just generate java code from this .pb file ?
If (as per comments) the file you have is the compiled descriptor set, then you can use protoc to generate any language (that it usually supports) from this; simply use the --descriptor_set_in=FILES option at the command line to specify your file as input (in place of FILES), and use --java_out=OUT_DIR (or whatever) to indicate the output language and location.

protoc generate golang code from two proto files which belong to different package

I have two proto src file, in same folder, let's say:
the 1st one is foo.proto
syntax = "proto3";
package foo;
the second is bar.proto, which need import foo.proto
syntax = "proto3";
import "foo.proto";
package bar;
you can see they have different package name, when I use protoc to generate them one by one, (foo.proto as the 1st of cause), I have two golang file generated, but I have to put them into 2 directory (you can't put different package name file inside same directory, golang )
foo/foo.pb.go
bar/bar.pb.go
but inside bar.pb.go the import are using local import path, which is like
import foo "."
I'm tweeking several options which protoc provided, but didn't make this work, any suggestion?
I believe add option go_package should be the correct solution

Resources