Dynamically create protocol buffer object - go

I'm working with a set of protocol buffers and I'm having trouble wrapping my head around what I need to do to instantiate them and it has to be calculated at runtime. I know I somehow need to use reflect to do this, but nothing seems to work. At compile time, I only know the first protocol buffer that I need to create. That buffer contains a field that tells me an integer id of the next one I need to create.
All of my buffers are defined in a package called kpb.
I find that first one like this:
info := &kpb.ArchiveInfo{}
err := proto.Unmarshal(<byte array>, info)
// error handling
messageType = *info.MessageType // 1
I have a map that defines the next buffers I need to call. That map is defined like this:
Note that all of the values of this map are proto.ProtoMessage objects, but using that seemed to cause more problems than it solved.
var registryMap = map[uint32]interface{}{
1: &kpb.KNDocumentArchive{},
...etc
}
So when I reference this map, I'm doing it like this:
var klass interface{}
//stuff
klass = registryMap[messageType]
fmt.Println(klass) // *kpb.KNDocumentArchive
But, what I can't figure out is how to instantiate a variable with the proper type to Unmarshal the payload I have.
I can get the type of the klass by doing this:
klassType := reflect.TypeOf(klass)
fmt.Println(klassType) // kpb.KNDocumentArchive - as expected
But, if I try to create a new variable with it, I get an error
payloadObj := new(klass)
// klassType (variable of type reflect.Type) is not a type
So even though the type is kpb.KNDocumentArchive like I expect, it's still somehow reflect.Type
When I used proto.ProtoMessage as the type for the map return, I could get past this part and have the variable instantiated, but I couldn't pass that to proto.Unmarshal because it, rightly, expects the type to be kpb.KNDocumentArchive
What am I doing wrong here?

I was able to figure this out. I needed to use proto built in features. The end result was updating my map to this:
var registryMap = map[uint32]string{
1: "KN.KNDocumentArchive",
}
And then the unmarshaling part was solved with this:
func GetProto(id uint32, messageBytes []byte) proto.Message {
klass := registryMap[id]
if klass == "" {
panic("Don't know how to parse Protobuf message type " + fmt.Sprint(id))
}
messageName := protoreflect.FullName(klass)
pbtype, err := protoregistry.GlobalTypes.FindMessageByName(messageName)
if err != nil {
panic(err)
}
msg := pbtype.New().Interface()
err = proto.Unmarshal(messageBytes, msg)
if err != nil {
panic(err)
}
return msg
}

Related

Generic Go code to retrieve multiple rows from BigQuery

I am writing some utils to retrieve multiple rows from BigQuery in a generic way using Go.
e.g.
type User struct {name string, surname string}
type Car struct {model string, platenumber string}
query1:="SELECT name, surname FROM UserTable"
query2:="SELECT model, platenumber FROM CarTable"
cars, _ := query2.GetResults()
users, _ := query1.GetResults()
OR
cars := []Car{}
query2.GetResults(cars) // and it would append to the slice
I am unsure about the signature of GetResults. I need somehow to pass the type to BigQuery library so it can retrieve the data and map it to the struct correctly. But at the same time I need to make it generic so it can be used for different types.
At the moment my GetResults looks like this: it doesn't work, the error is:
bigquery: cannot convert *interface {} to ValueLoader (need pointer to []Value, map[string]Value, or struct)[]
But I cannot pass directly the struct as I want to make it generic.
func (s *Query) GetResults() ([]interface{}, error) {
var result []interface{}
job, err := s.Run()
if err != nil {
s.log.Error(err, "error in running the query")
return nil, err
}
it, err := job.ReadData()
if err != nil {
s.log.Error(err, "error in reading the data")
return nil, err
}
var row interface{}
for {
err := it.Next(&row)
if err != nil {
fmt.Print(err)
break
}
result = append(result, row)
}
return result, nil
}
Is there another way to achieve that? Or is the good way not to create a method like that?
I've tried quite a lot of different things, with or without pointer, with or without array, by modifying the args, or returning a new list, nothing seem to work, and doing all of that feels a bit wrong regarding the nature "easy" of what I am trying to achieve.
I've also looked into doing the following
GetResults[T any]() ([]T, error)
But it's "excluded" as GetResults is part of an interface (and we can't define generic for a method of an interface). And I can't/don't want to define a type for all the interface, as it impacts other interfaces.

How to range over slice of a custom type

I'm trying to write in Go custom cache for Google DataStore (more precisely - a wrapper around one of existing cache libraries). At cache initialisation, it should accept any custom type of struct (with appropriately-defined datastore fields), which then would be the basis for all items stored. The idea is that cache can be created/initialised for various types which reflect the structure of a particular DataStore entry (CustomEntry)
Approach 1 - store reflect.Type and use it. Problem encountered - can't iterate over a slice of a custom type
type CustomEntry struct {
Data struct {
name string `datastore:"name,noindex"`
address []string `datastore:"address,noindex"`
} `datastore:"data,noindex"`
}
func (cache *MyCache) CacheData(dataQuery string, dataType reflect.Type) {
slice := reflect.MakeSlice(reflect.SliceOf(dataType), 10, 10)
if keys, err := DataStoreClient.GetAll(cache.ctx, datastore.NewQuery(dataQuery), &slice); err != nil {
//handle error
} else {
for i, dataEntry:= range slice {
// ERROR: Cannot range over 'slice' (type Value)
cache.Set(keys[i].Name, dataEntry)
}
}
//usage: Cache.CacheData("Person", reflect.TypeOf(CustomEntry{})
Approach 2 - accept an array of interfaces as arguments. Problem encountered = []CustomEntry is not []interface{}
func (cache *MyCache) CacheData(dataQuery string, dataType []interface{}) {
if keys, err := DataStoreClient.GetAll(cache.ctx, datastore.NewQuery(dataQuery), &dataType); err != nil {
//handle error
} else {
for i, dataEntry:= range slice {
// this seems to work fine
cache.Set(keys[i].Name, dataEntry)
}
}
//usage:
var dataType []CustomEntry
Cache.CacheData("Person", data)
// ERROR: Cannot use 'data' (type []CustomEntry) as type []interface{}
Any suggestions would be highly appreciated.
I have found a solution and thought it might be worth sharing in case anyone else has a similar problem.
The easiest way is to initiate a slice of structs which the DataStore is expected to receive, and then to pass a pointer to it as an argument (interface{}) into the desired function. DataStore, similarly to a few unmarshaling functions (I have tried with JSON package) will be able to successfully append the data to it.
Trying to dynamically create the slice within the function, given a certain Type, which would be then accepted by a function (such as DataStore client) might be quite difficult (I have not managed to find a way to do it). Similarly, passing a slice of interfaces (to allow for easy iteration) only complicates things.
Secondly, in order to iterate over the data (e.g. to store it in cache), it is necessary to:
(1) retrieve the underlying value of the interface (i.e. the pointer itself) - this can be achieved using reflect.ValueOf(pointerInterface),
(2) dereference the pointer so that we obtain access to the underlying, iterable slice of structs - this can be done by invoking .Elem(),
(3) iterate over the underlying slice using .Index(i) method (range will not accept an interface, even if the underlying type is iterable).
Naturally, adding a number of switch-case statements might be appropriate to ensure that any errors are caught rather than cause a runtime panic.
Hence the following code provides a working solution to the above problem:
In main:
var data []customEntry
c.CacheData("Person",&data)
And the function itself:
func (cache *MyCache) CacheData(dataQuery string, data interface{}) error {
if keys, err := DataStoreClient.GetAll(cache.ctx, datastore.NewQuery(dataQuery), data); err != nil {
return err
} else {
s := reflect.ValueOf(data).Elem()
for i := 0; i < s.Len(); i++ {
cache.Set(keys[i].Name, s.Index(i), 1)
}
}
}

Understanding GO variable assignment

I'm new to GO and trying to build functions with the aws-sdk I have something like this
input := &rds.CreateDBClusterSnapshotInput{
// removed for brevity
}
result, err := svc.CreateDBClusterSnapshot(input)
if err != nil {
// removed for brevity
}
input = &rds.ModifyDBClusterSnapshotAttributeInput{
// removed for brevity
}
When I try to build, I get this error
cannot use &rds.ModifyDBClusterSnapshotAttributeInput literal (type *rds.ModifyDBClusterSnapshotAttributeInput) as type *rds.CreateDBClusterSnapshotInput in assignment
What's wrong with my assignment?
As #Sergio Tulentsev pointed out, you are assigning ModifyDBClusterSnapshotAttributeInput type to the variable input, that is a CreateDBClusterSnapshotInput type.
There would be a few solutions to handle this problem, but the easiest way would be to make a method for each type struct that returns a compatible type for input like this;
func (createInput CreateDBClusterSnapshotInput) ReturnInput() {
return createInput.input // assuming that there are a input type your create
}
If you don't want to make a method with the same functionality for each struct, you can create a base type, make your two structs extend the type, and build a method for the base type.

Using empty interfaces in go

I am trying to understand the code that is used at my company. I am new to go lang, and I have already gone through the tutorial on their official website. However, I am having a hard time wrapping my head around empty interfaces, i.e. interface{}. From various sources online, I figured out that the empty interface can hold any type. But, I am having a hard time figuring out the codebase, especially some of the functions. I will not be posting the entire thing here, but just the minimal functions in which it has been used. Please bear with me!
Function (I am trying to understand):
func (this *RequestHandler) CreateAppHandler(rw http.ResponseWriter, r *http.Request) *foo.ResponseError {
var data *views.Data = &views.Data{Attributes: &domain.Application{}}
var request *views.Request = &views.Request{Data: data}
if err := json.NewDecoder(r.Body).Decode(request); err != nil {
logrus.Error(err)
return foo.NewResponsePropogateError(foo.STATUS_400, err)
}
requestApp := request.Data.Attributes.(*domain.Application)
requestApp.CreatedBy = user
Setting some context, RequestHandler is a struct defined in the same package as this code. domain and views are seperate packages. Application is a struct in the package domain. The following two structs are part of the package views:
type Data struct {
Id string `json:"id"`
Type string `json:"type"`
Attributes interface{} `json:"attributes"`
}
type Request struct {
Data *Data `json:"data"`
}
The following are part of the package json:
func NewDecoder(r io.Reader) *Decoder {
return &Decoder{r: r}
}
func (dec *Decoder) Decode(v interface{}) error {
if dec.err != nil {
return dec.err
}
if err := dec.tokenPrepareForDecode(); err != nil {
return err
}
if !dec.tokenValueAllowed() {
return &SyntaxError{msg: "not at beginning of value"}
}
// Read whole value into buffer.
n, err := dec.readValue()
if err != nil {
return err
}
dec.d.init(dec.buf[dec.scanp : dec.scanp+n])
dec.scanp += n
// Don't save err from unmarshal into dec.err:
// the connection is still usable since we read a complete JSON
// object from it before the error happened.
err = dec.d.unmarshal(v)
// fixup token streaming state
dec.tokenValueEnd()
return err
}
type Decoder struct {
r io.Reader
buf []byte
d decodeState
scanp int // start of unread data in buf
scan scanner
err error
tokenState int
tokenStack []int
}
Now, I understood that, in the struct Data in package views, Application is being set as a type for the empty interface. After that, a pointer to Request in the same package is created which points to the variable data.
I have the following doubts:
What exactly does this keyword mean in Go? What is the purpose of writing this * RequestHandler?
Initialization of a structure in Go can be done while assigning it to a variable by specifying the values of all it's members. However, here, for the struct Data, only the empty interface value is assigned and the values for the other two fields are not assigned?
What is the advantage of assigning the Application struct to an empty interface? Does it mean I can use the struct members using the interface variable directly?
Can someone help me figure out the meaning of this statement? json.NewDecoder(r.Body).Decode(request)?
While I know this is too much, but I am having a hard time figuring out the meaning of interfaces in Go. Please help!
this is not a keyword in go; any variable name can be used there. That is called the receiver. A function declared in that way must be called like thing.func(params), where "thing" is an expression of the type of the receiver. Within the function, the receiver is set to the value of thing.
A struct literal does not have to contain values for all the fields (or any of them). Any fields not explicitly set will have the zero value for their types.
As you said, an empty interface can take on a value of any type. To use a value of type interface{}, you would use type assertion or a type switch to determine the type of the value, or you could use reflection to use the value without having to have code for the specific type.
What specifically about that statement do you not understand? json is the name of a package in which the function NewDecoder is declared. That function is called, and then the Decode function (which is implemented by the type of the return value of NewDecoder) is called on that return value.
You may want to take a look at Effective Go and/or The Go Programming Language Specification for more information.

Need to construct interface/pointer using Reflect in Golang, not working

So, this works:
house := model.House {};
err = db.First(&house).Error;
However, this doesn't work:
var house model.House;
fetchFromDatabase := reflect.New(reflect.TypeOf(house)).Interface();
err = db.First(&fetchFromDatabase).Error;
... The database library gives the error:
unsupported destination, should be slice or struct
To me, that should be a struct, considering the "House" type is a struct. However, I'm still wrapping my head around Reflect ... can anyone help?
The library is complaining because the application is passing a *interface{} to the method. The value fetchFromDatabase is a pointer to a struct. Pass this value directly to the method:
var house model.House
fetchFromDatabase := reflect.New(reflect.TypeOf(house)).Interface()
err = db.First(fetchFromDatabase).Error

Resources