I just dived in Go programming using protobuf and I'm at the point where I need to validate data in a struct. I found govalidator, which seems to do the perfect job for what I need. It does validate structs based on the field tags, something like
type Contact struct {
firstName string `valid:"alpha,required"`
lastName string `valid:"alpha,required"`
email string `valid:"email,required"`
}
jdoe := &Contact{
firstName: "John",
lastName: "Doe",
email: "jdoe#mail.com"
}
ok, err = govalidator.ValidateStruct(jdoe)
And my protobuf definition would look like
message Contact {
string firstName = 1;
string lastName = 2;
string email = 3;
}
Now my question would be, is there a way to define the field tags in the proto message. From what I've seen in the generated go code, the compiler adds tags to the fields anyway, but could I "sneak" the ones that I need too? Also, I would imagine that unmarshalling could be one possible solution, but it somehow seems inefficient to me to unmarshal just to copy the field values to an equivalent struct which would have the necessary field tags.
Having the same structure for the data encapsulation and the input coming from the client was just a pure coincidence. As it has been suggested not only in the comments, but also by co-workers more experienced (than me) with protobuf I've just mapped (1:1 in this particular case) the fields from the structure generated by proto to the data encapsulation structure I have defined.
Related
I have a user message as such :
message User {
string uid = 1;
}
Protobuff generates a struct as such
type User struct {
Uid string
}
Is there a way to enforce destination case so that my user struct in Go would be like :
type User struct {
UID string
}
So far, I can use string uID in my proto definition but it feels a bit hacky.
This does not seem to be supported.
As listed in golang/protobuf issue 555
By convention, naming styles are different:
Protos uses CamelCase for messages, groups, enums, services, and methods
Protos uses snake_case for fields and oneof fields
Protos uses SCREAMING_SNAKE_CASE for enum values
Go uses CamelCase for everything
So for now, uID is the best workaround.
I am creating a proto3 schema that allow multiple/arbitrary data in incoming jsonObject. I would like to convert the incoming json object in one shot.
for example
{"key1":"value",
"key2": { //schema A}
}
I also want to support for schema B for key2 in different request.
{"key1":"value",
"key2": { //schema B}
}
I tried couple of different approach such as oneof but for oneof it needs different key names, since I am using the same key2, it does not work for me in this case.
and here is the schema.
message IncomingRequest {
string key1 = 1;
//google.protobuf.Any key2 = 2; --> not working
oneof message{
A payload = 2;
B payload = 3; --> duplicate key
}
}
Anyone know how to achieve this?
Two ways I can think of:
Message types for each request
If you know, based on the request (e.g. HTTP URL called), if it should be schema A or B, I recommend to create separate message types for each request. This might result in more proto types you have to define, but will be simple to use in the actual code you have to write to consume the payloads.
Struct type
If you really want/have to reuse the same message type you can use the Struct proto type to encode/decode any JSON structure.
message IncomingRequest {
string key1 = 1;
google.protobuf.Struct key2 = 2;
}
Although from the proto type definition it doesn't look like it does what you want, Protobuf decoders/encoders will handle this type in a special way to give you the behavior wanted.
The issue with this option is that what you gain in flexibility in the proto, you lose in expressiveness in the generated code, as you have to do a lot edge-case checking if a specific value/type is set.
So I have a question that I've been trying to figure out for a while and hopefully someone has some insight. Basically, I have two similar, but different structs in Go and I would like to convert between them.
Something like the following:
type A struct {
Name string
}
type B struct {
Name NameStruct
}
type NameStruct struct {
Firstname string
Lastname string
}
In this example,
B.Name == NameStruct{
Firstname: strings.Split(A.Name, " ")[0],
Lastname: strings.Split(A.Name, " ")[1],
}
Basically I would like to be able to define a mapping of fields from one struct to the other as well as a conversion function, so that I can do this without writing all of the code to do the conversion manually.
What I'm imagining is that there might be some sort of Go generator out there that could do something like this. I don't think it is possible using reflection because of the nested structs (although I'm open to suggestions).
Basically, does anyone know of a way to do this that is better than either writing conversion code manually or creating my own generator?
I am really stuck here with a seemingly trivial problem in Go:
I have a Golang microservice, which outputs data in json format. Let's say I have a simple struct with json tags for this result:
type Result struct {
Name string `json:"name"`
Age int `json:"age"`
}
In the code part where the data is actually pulled out of the database, I have a quite similar struct, like this:
type ResultBackend struct {
Name string `bson:"fullName"`
Age int `bson:"age"`
}
The struct fields are similar, except the different tags. I would like to keep things simple, and return just one struct from the backend service (ResultBackend), which can then be send as a JSON response, like this:
func process() Result {
var result ResultBackend
... do a MongoDB query here and store results in result variable ...
return result
}
This certainly would not work, because we have two different structs here.
Of course one solution would be to embed both tags in one struct like so:
type Result struct {
Name string `json:"name" bson:"fullName"`
Age int `json:"age bson:"age"`
}
and then use this struct in the main code and the "process" function.
This works, but this seems like "poisoning" the Result struct of the main code with the bson tags. What, for instance, if the backend result is a XML file? I'd have to add xml tags to the struct as well. Or maybe someday tags some very obsure database adapter.
This doesn't seem to be the cleanest approach in my eyes. I'd rather have a clean Result struct in the main code, and simply to a conversion from one struct to another.
Is there any easy way doing that, or do I really have to copy all the fields of a ResultBackend struct to a new Result struct and return that? Or I am trying to over-simplify my code here? :)
Cheers!
What I'd do is create a separate type for every "serialisation format" if you will. This approach has several advantages:
Your concerns are separated.
JSON un/marshalling doesn't interfere with BSON/XML, so you can add any kind of additional struct fields (like xml.Name for example).
You can actually create several such types for different APIs that need different parameters.
The only disadvantage I see is that it's more code and even more code to move data between those. Ultimately it's up to you and your application design. If you're sure that your JSON/BSON will stay the same, you can use one type. Otherwise, I'd recommend specialisation.
To those who want to add bson tags in golang struct, here is a tool which can be helpful. I have spent a whole day searching for this and I want others to save their valuable time.
You can convert the following struct
type Result struct {
Name string `json:"name"`
Age int `json:"age"`
}
into
type Result struct {
Name string `json:"name" bson:"fullName"`
Age int `json:"age bson:"age"`
}
With this tool, you can not only add bson tags but also
XML tags,
Define validation and scope
Format tag values
Apply transformation
skip unexported fields.
Go Modify Tags: https://github.com/fatih/gomodifytags
Cheers,
In the Go Language Specification, it mentions a brief overview of tags:
A field declaration may be followed by an optional string literal tag,
which becomes an attribute for all the fields in the corresponding
field declaration. The tags are made visible through a reflection
interface but are otherwise ignored.
// A struct corresponding to the TimeStamp protocol buffer.
// The tag strings define the protocol buffer field numbers.
struct {
microsec uint64 "field 1"
serverIP6 uint64 "field 2"
process string "field 3"
}
This is a very short explanation IMO, and I was wondering if anyone could provide me with what use these tags would be?
A tag for a field allows you to attach meta-information to the field which can be acquired using reflection. Usually it is used to provide transformation info on how a struct field is encoded to or decoded from another format (or stored/retrieved from a database), but you can use it to store whatever meta-info you want to, either intended for another package or for your own use.
As mentioned in the documentation of reflect.StructTag, by convention the value of a tag string is a space-separated list of key:"value" pairs, for example:
type User struct {
Name string `json:"name" xml:"name"`
}
The key usually denotes the package that the subsequent "value" is for, for example json keys are processed/used by the encoding/json package.
If multiple information is to be passed in the "value", usually it is specified by separating it with a comma (','), e.g.
Name string `json:"name,omitempty" xml:"name"`
Usually a dash value ('-') for the "value" means to exclude the field from the process (e.g. in case of json it means not to marshal or unmarshal that field).
Example of accessing your custom tags using reflection
We can use reflection (reflect package) to access the tag values of struct fields. Basically we need to acquire the Type of our struct, and then we can query fields e.g. with Type.Field(i int) or Type.FieldByName(name string). These methods return a value of StructField which describes / represents a struct field; and StructField.Tag is a value of type StructTag which describes / represents a tag value.
Previously we talked about "convention". This convention means that if you follow it, you may use the StructTag.Get(key string) method which parses the value of a tag and returns you the "value" of the key you specify. The convention is implemented / built into this Get() method. If you don't follow the convention, Get() will not be able to parse key:"value" pairs and find what you're looking for. That's also not a problem, but then you need to implement your own parsing logic.
Also there is StructTag.Lookup() (was added in Go 1.7) which is "like Get() but distinguishes the tag not containing the given key from the tag associating an empty string with the given key".
So let's see a simple example:
type User struct {
Name string `mytag:"MyName"`
Email string `mytag:"MyEmail"`
}
u := User{"Bob", "bob#mycompany.com"}
t := reflect.TypeOf(u)
for _, fieldName := range []string{"Name", "Email"} {
field, found := t.FieldByName(fieldName)
if !found {
continue
}
fmt.Printf("\nField: User.%s\n", fieldName)
fmt.Printf("\tWhole tag value : %q\n", field.Tag)
fmt.Printf("\tValue of 'mytag': %q\n", field.Tag.Get("mytag"))
}
Output (try it on the Go Playground):
Field: User.Name
Whole tag value : "mytag:\"MyName\""
Value of 'mytag': "MyName"
Field: User.Email
Whole tag value : "mytag:\"MyEmail\""
Value of 'mytag': "MyEmail"
GopherCon 2015 had a presentation about struct tags called:
The Many Faces of Struct Tags (slide) (and a video)
Here is a list of commonly used tag keys:
json - used by the encoding/json package, detailed at json.Marshal()
xml - used by the encoding/xml package, detailed at xml.Marshal()
bson - used by gobson, detailed at bson.Marshal(); also by the mongo-go driver, detailed at bson package doc
protobuf - used by github.com/golang/protobuf/proto, detailed in the package doc
yaml - used by the gopkg.in/yaml.v2 package, detailed at yaml.Marshal()
db - used by the github.com/jmoiron/sqlx package; also used by github.com/go-gorp/gorp package
orm - used by the github.com/astaxie/beego/orm package, detailed at Models – Beego ORM
gorm - used by gorm.io/gorm, examples can be found in their docs
valid - used by the github.com/asaskevich/govalidator package, examples can be found in the project page
datastore - used by appengine/datastore (Google App Engine platform, Datastore service), detailed at Properties
schema - used by github.com/gorilla/schema to fill a struct with HTML form values, detailed in the package doc
asn - used by the encoding/asn1 package, detailed at asn1.Marshal() and asn1.Unmarshal()
csv - used by the github.com/gocarina/gocsv package
env - used by the github.com/caarlos0/env package
Here is a really simple example of tags being used with the encoding/json package to control how fields are interpreted during encoding and decoding:
Try live: http://play.golang.org/p/BMeR8p1cKf
package main
import (
"fmt"
"encoding/json"
)
type Person struct {
FirstName string `json:"first_name"`
LastName string `json:"last_name"`
MiddleName string `json:"middle_name,omitempty"`
}
func main() {
json_string := `
{
"first_name": "John",
"last_name": "Smith"
}`
person := new(Person)
json.Unmarshal([]byte(json_string), person)
fmt.Println(person)
new_json, _ := json.Marshal(person)
fmt.Printf("%s\n", new_json)
}
// *Output*
// &{John Smith }
// {"first_name":"John","last_name":"Smith"}
The json package can look at the tags for the field and be told how to map json <=> struct field, and also extra options like whether it should ignore empty fields when serializing back to json.
Basically, any package can use reflection on the fields to look at tag values and act on those values. There is a little more info about them in the reflect package
http://golang.org/pkg/reflect/#StructTag :
By convention, tag strings are a concatenation of optionally
space-separated key:"value" pairs. Each key is a non-empty string
consisting of non-control characters other than space (U+0020 ' '),
quote (U+0022 '"'), and colon (U+003A ':'). Each value is quoted using
U+0022 '"' characters and Go string literal syntax.
It's some sort of specifications that specifies how packages treat with a field that is tagged.
for example:
type User struct {
FirstName string `json:"first_name"`
LastName string `json:"last_name"`
}
json tag informs json package that marshalled output of following user
u := User{
FirstName: "some first name",
LastName: "some last name",
}
would be like this:
{"first_name":"some first name","last_name":"some last name"}
other example is gorm package tags declares how database migrations must be done:
type User struct {
gorm.Model
Name string
Age sql.NullInt64
Birthday *time.Time
Email string `gorm:"type:varchar(100);unique_index"`
Role string `gorm:"size:255"` // set field size to 255
MemberNumber *string `gorm:"unique;not null"` // set member number to unique and not null
Num int `gorm:"AUTO_INCREMENT"` // set num to auto incrementable
Address string `gorm:"index:addr"` // create index with name `addr` for address
IgnoreMe int `gorm:"-"` // ignore this field
}
In this example for the field Email with gorm tag we declare that corresponding column in database for the field email must be of type varchar and 100 maximum length and it also must have unique index.
other example is binding tags that are used very mostly in gin package.
type Login struct {
User string `form:"user" json:"user" xml:"user" binding:"required"`
Password string `form:"password" json:"password" xml:"password" binding:"required"`
}
var json Login
if err := c.ShouldBindJSON(&json); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
return
}
the binding tag in this example gives hint to gin package that the data sent to API must have user and password fields cause these fields are tagged as required.
So generraly tags are data that packages require to know how should they treat with data of type different structs and best way to get familiar with the tags a package needs is READING A PACKAGE DOCUMENTATION COMPLETELY.