I have spent hours trying to read blogs on how to use simple proto file in Golang. I generated the .pb.go files. All internet examples are littered with doing import from some random "github... urls for proto import. I am not able to find any example on how to import a simple proto file that exists in same dir as my .go file or diff directory. How do I use proto files from local file systems.
go build hello.go
hello.go:5:2: cannot find package "customer" in any of:
/usr/local/go/src/customer (from $GOROOT)
/Users/myhomedir/go/src/customer (from $GOPATH)
Contentes of hello.goin $SOME_DIR/customer
package main
import (
"fmt"
pb "customer"
)
func main() {
fmt.Println("hello test message\n")
}
Contents of customer.proto
syntax = "proto3";
package customer;
// The Customer service definition.
service Customer {
// Get all Customers with filter - A server-to-client streaming RPC.
rpc GetCustomers(CustomerFilter) returns (stream CustomerRequest) {}
// Create a new Customer - A simple RPC
rpc CreateCustomer (CustomerRequest) returns (CustomerResponse) {}
}
// Request message for creating a new customer
message CustomerRequest {
int32 id = 1; // Unique ID number for a Customer.
string name = 2;
string email = 3;
string phone= 4;
message Address {
string street = 1;
string city = 2;
string state = 3;
string zip = 4;
bool isShippingAddress = 5;
}
repeated Address addresses = 5;
}
message CustomerResponse {
int32 id = 1;
bool success = 2;
}
message CustomerFilter {
string keyword = 1;
}
Just put the Customer code in a directory and import it like you would a package
package main
import "$SOME_DIR/customer"
As some of the comments already mentioned, you have to import your customer file like any other go import - with its full url-like path. That is true even if your main.go is in the same directory as your dependency.
There is a specific way in which everyone should manage go code dependencies and you can read about it here:
How to write go code
It does feel weird to follow a common standard of how to organize your workspace (at least to me it did when I first started with go) - but since you run into errors like yours if you don't follow them; just try to do it.
A pretty good place to start (after you've read 'how to write go code') is actually the error message you got. It will change if you change your import statement/directory structure and it will help you to specify the full path to your dependency in your import ... statement.
Related
Does protobuf support embedding non functional metadata into the protobuf schema without affecting the message serialization/de-serialization? I am attempting to embed business information (ownership, contact info) into a large shared protobuf schema but do NOT want to impact functionality at all.
A structured comment or custom_option that does not get serialized would work. I would also like to parse the information from the .proto file for auditing purposes.
TIA
message Bar {
optional int32 a = 1 [(silent_options).owner = "team1", (silent_options).email = "team1#company.com"];
optional int32 b = 2;
}
To do what you want, you will need to use the extend keyword to create a custom FieldOption and then create a protoc plugin. I recommend using Go or C++ for custom plugins, their API is easier.
First we create a custom field option with extend. You can do so like this:
syntax = "proto3";
import "google/protobuf/descriptor.proto";
option go_package = "test.com/proto";
message SilentOptions {
string owner = 1; // maybe an enum if you know all teams
string email = 2;
}
extend google.protobuf.FieldOptions {
SilentOptions silent_options = 1000;
}
There are few things to note. The first one is that we are extending google.protobuf.FieldOptions which is available because we import descriptor.proto. Then, in my case my go module is called "test.com" and this file is in a folder called proto. So the go_package is "test.com/proto".
Once we have this, we need to create a protoc plugin. There are few tutorials on the web but here is a quick solution which is only working for your example:
package main
import (
"log"
"google.golang.org/protobuf/compiler/protogen"
"google.golang.org/protobuf/proto"
"google.golang.org/protobuf/types/descriptorpb"
"google.golang.org/protobuf/types/pluginpb"
silentOption "test.com/proto"
)
func main() {
protogen.Options{}.Run(func(gen *protogen.Plugin) error {
gen.SupportedFeatures = uint64(pluginpb.CodeGeneratorResponse_FEATURE_PROTO3_OPTIONAL)
for _, sourceFile := range gen.Files {
if !sourceFile.Generate {
continue
}
for _, msg := range sourceFile.Messages {
for _, field := range msg.Fields {
opt := field.Desc.Options().(*descriptorpb.FieldOptions)
if opt == nil { // no FieldOption
continue
}
ext1 := proto.GetExtension(opt, silentOption.E_SilentOptions)
if ext1 != nil { // no SilentOptions
log.Println(ext1)
}
}
}
}
return nil
})
}
For this to run you will need to go get -u google.golang.org/protobuf and run protoc on the silent_options.proto: protoc --go_out=. --go_opt=module=test.com proto/*.proto. For the protoc command to run you need to run the following commands: go get google.golang.org/protobuf/cmd/protoc-gen-go. This is getting the official plugin to generate go code from protobuf.
Finally, build the main.go with a command similar to go build -o protoc-gen-test (note that the name of the binary should start with protoc-gen and then is followed by you plugin name). Then, copy the generated binary to a place in your PATH. And with that you should be able to run: protoc --test_out=. test.proto where test.proto is:
syntax = "proto3";
import "proto/silent_options.proto"; // import the proto file for options
option go_package = "another_test.com"; // can be in a different package
message Bar {
optional int32 a = 1 [
(silent_options) = {
owner: "team1",
email: "team1#company.com"
}
];
optional int32 b = 2;
}
It should print something like:
owner:"team1" email:"team1#company.com"
Obviously this is quite an involved process but I would be happy to update this answer if anything is unclear or if you need more details on the code.
Edit
I forgot to mention that I chose the tag 1000 but there might already have other plugins have the same tag. If you plan to use other plugins later, you should be careful with that and check the tag already in use (protobuf documentation had a page on this, can't find it yet) otherwise you might have conflicts.
Edit 2
Here is the list of the extension tags that are already taken.
I'm working on a tutorial about gRPC. When I generated the .pb.go file, I'm getting some XXX_* type in my struct.
This is my consignment.proto file:
syntax = "proto3";
package go.micro.srv.consignment;
service ShippingService {
rpc CreateConsignment(Consignment) returns (Response) {}
}
message Consignment {
string id = 1;
string description = 2;
int32 weight = 3;
repeated Container containers = 4;
string vessel_id = 5;
}
message Container {
string id = 1;
string customer_id = 2;
string origin = 3;
string user_id = 4;
}
message Response {
bool created = 1;
Consignment consignment = 2;
}
This is the struct in the .pb.go file. Can someone tell me why are there 3 XXX types in my struct? Shouldn't the struct be reflecting what I'm defining in my proto?
type Consignment struct {
Id string `protobuf:"bytes,1,opt,name=id" json:"id,omitempty"`
Description string `protobuf:"bytes,2,opt,name=description" json:"description,omitempty"`
Weight int32 `protobuf:"varint,3,opt,name=weight" json:"weight,omitempty"`
Containers []*Container `protobuf:"bytes,4,rep,name=containers" json:"containers,omitempty"`
VesselId string `protobuf:"bytes,5,opt,name=vessel_id,json=vesselId" json:"vessel_id,omitempty"`
XXX_NoUnkeyedLiteral struct{} `json:"-"`
XXX_unrecognized []byte `json:"-"`
XXX_sizecache int32 `json:"-"`
}
The XXX_ types are used by the Protobuf library to store unknown fields. When you decode a proto, there may be additional fields in the serialized data that the library doesn't know what to do with. This can happen, for instance, when the reader and writer of the data are using different copies of the proto file. This is a feature to help with backwards compatibility between clients and serves built at different times.
Additionally, the XXX fields allow you to expose extensions, which were part of Proto2. They were removed in Proto3 in favor of Any, but the library still needs to support them.
As for what you should do with these? I would just leave them alone, and don't reference them. You don't need to set them, and you don't need to read them. The Go protobuf library will handle them for you.
If you want to specifically exclude them when generating, you can add this to your file
import "github.com/gogo/protobuf/gogoproto/gogo.proto";
option (gogoproto.goproto_unkeyed_all) = false;
option (gogoproto.goproto_unrecognized_all) = false;
option (gogoproto.goproto_sizecache_all) = false;
edit: Styling
The official protobuffer generator for golang still includes the XXX_ fields and the issue is still open and decisions are pending on wether to remove it ot not but you can use the gogo third party generator. I am using it for a long time without a problem. Just use the gogofaster_out
https://github.com/gogo/protobuf
Here are the section that discussed about exclude the XXX_ fields
More Speed and more generated code
Fields without pointers cause less time in the garbage collector. More
code generation results in more convenient methods.
Other binaries are also included:
protoc-gen-gogofast (same as gofast, but imports gogoprotobuf)
protoc-gen-gogofaster (same as gogofast, without XXX_unrecognized, less pointer fields)
protoc-gen-gogoslick (same as gogofaster, but with generated string, gostring and equal methods)
Installing any of these binaries is easy. Simply run:
go get github.com/gogo/protobuf/proto
go get github.com/gogo/protobuf/{binary}
go get github.com/gogo/protobuf/gogoproto
After installing the lib use
protoc --gogofaster_out=plugins=grpc
I try to use golang deal with this problem 557. Reverse Words in a String III
my code as below:
import "fmt"
import ss "strings"
func reverseWords(s string) string {
words := ss.Split(s," ");
res := "";
for i:=0; i < len(words);i++{
curWord := ss.Split(words[i],"");
for j:=len(curWord)-1; j >= 0;j--{
res += curWord[j];
}
if(i!=len(words)-1){
res += " ";
}
}
return res;
}
func main(){
s := "Let's take LeetCode contest";
fmt.Println(reverseWords(s));
}
Everything is ok in my pc, it can pass compile at least.
However, when I submit in leetcode it tell me :
Line 67: undefined: strings in strings.Trim
I google this error but get nothing relevant info. As a beginner in golang, I need help. Everything will be appreciated.
You're importing strings under an alias:
import ss "strings"
That means that everywhere in that file, instead of referring to strings you must refer to ss, for example:
words := ss.Split(s," ")
If you use the default import:
import "strings"
Then you can refer to it as strings as normal.
Note that the currently accepted answer is wrong about two things: you can absolutely use the alias as you have it, you just have to refer to the package with the aliased name. It will not cause any issues if you use the name you gave it. Second, you absolutely do need to import the strings package - with or without an alias, your choice - if you want to refer to it.
On a completely unrelated side note, you should strongly consider running go fmt on your code, as it does not follow Go coding standards; for example, standard Go code omits the vast majority of semicolons. The code will work regardless, but you'll have an easier time getting help from other Go developers if your code is formatted the way everyone else is used to seeing it.
If you import strings package with different name then it will cause issue as it is used by the wrapper code to run the function completely.
No need to import strings package again. It will be added.
Just use it directly.
func reverseWords(s string) string {
words := strings.Split(s," ");
res := "";
for i:=0; i < len(words);i++{
curWord := strings.Split(words[i],"");
for j:=len(curWord)-1; j >= 0;j--{
res += curWord[j];
}
if(i!=len(words)-1){
res += " ";
}
}
return res;
}
I am new to go and coming from a Ruby background. I am trying to understand code structuring in a world without classes and am probably making the mistake wanting to do it "the Ruby way" in Go.
I am trying to refactor my code to make it more modular / readable so I moved the loading of the configuration file to its own package. Good idea?
package configuration
import "github.com/BurntSushi/toml"
type Config struct {
Temperatures []struct {
Degrees int
Units string
}
}
func Load() Config {
var cnf Config
_, err := toml.DecodeFile("config", &cnf)
if err != nil {
panic(err)
}
return cnf
}
Now, in my main package:
package main
import "./configuration"
var conf Configuration = configuration.Load()
Gives undefined: Config. I understand why. I could copy the struct definition in the main package but that's not very DRY.
It's my understanding passing around structs like this is a bad practice as it makes your code harder to understand (now everyone needs to know about my Config struct).
Is hiding logic in a package like I am trying to do here a good idea in Go? If so, what's the "Go" way to pass this Config struct around?
In your main package you should specify
var conf configuration.Config = configuration.Load()
configuration refers to your imported package and Config is the exported struct (uppercase name) from that package. But you can also omit this, as the type can be inferred
var conf = configuration.Load()
As a side note: please don't use relative imports
in Go imports you always declare the full path of you package, dont use relative paths in imports, best example is that toml import import "github.com/BurntSushi/toml" that exist in:
GOPATH/src/github.com/BurntSushi/toml
GOPATH/pkg/_/github.com/BurntSushi/toml
Then build you package and main.go
package main
import "mypackage/configuration"
func main() {
// configuration contain all funcs & structs
var conf configuration.Config = configuration.Load()
}
Go it is not ruby.
Ref Packages: https://golang.org/doc/code.html
why don't you just import the configuration package and then do Go's variable declaration/instatiation shortcut? Maybe I'm missing something.
package main
import "mypackage/configuration"
func main() {
conf := configuration.Load()
}
How do I override variables in go from package ?
For example:
file1.go
package A
import "A/B"
var Arg1 = "Hello"
file2.go
package A/B
var Arg1 = "World"
How can I override arg1 in file1.go if arg1 exist in file2.go?
I'm not sure what you mean by "overriding" in this case. (Also let me assume file1.go is 'package a' and file2.go is 'package b')
If you mean to access Arg1, defined in package "b" inside/from within package "a", then eg.:
// In package "a"
x = Arg1 // references a.Arg1
y = b.Arg1 // references b.Arg1
However, nothing resembling overriding happens here. In package "a" both a's Arg1 and b's Arg1 are accessible as different entities; the later via a mandatory qualifier 'b'.
Are you trying to do something like this, where, for example, a specific location (USA), if present, overrides a general location (World)?
// file A/B.go
package B
var location = "USA"
func Location() string {
return location
}
// file A.go
package A
import "A/B"
var location = "World"
func Location() string {
loc := B.Location()
if len(loc) == 0 {
loc = location
}
return loc
}
// file main.go
package main
import (
"A"
"fmt"
)
func main() {
fmt.Println("Hello,", A.Location())
}
Output:
Hello, USA
If not, please provide a better and specific example of what you are trying to do.
You can't. Packages are self-contained.
If package A doesn't export arg1 (lowercase)
it is not visible -- and thus not set-able --
for an other package B.
BTW: "package A/B" looks pretty strange to me.
You could probably do it using a build flag :
go build -ldflags="-X 'package_path.variable_name=new_value'"
Couple of things to be aware of though :
In order to use ldflags, the value you want to change must exist and be a package level variable of type string. This variable can be either exported or unexported. The value cannot be a const or have its value set by the result of a function call
You'll find all the information needed in this excellent post from DO team