how to use protobuf.any in golang - go

I'm using grpc in my go project. Below is code:
example.proto:
syntax = "proto3";
message Example {
string message = 1;
google.protobuf.Any details = 2;
}
main.go
func logMessage (m string, d interface{}) {
message := & example.message{
message: m,
details: ??
}
log(&message)
}
But I'm not sure how to deal with the details(interface{}) field. I know I can use any type for interface, but not sure how to use it here. Anyone can help? Thanks

Since protobuf/ptypes is deprecated, it worth using anypb.UnmarshalTo
import (
"google.golang.org/protobuf/types/known/anypb"
"github.com/golang/protobuf/ptypes/any"
)
func Unmarshal(data *any.Any) (*YourMessage, err) {
var m YourMessage
err := anypb.UnmarshalTo(data, &m, proto.UnmarshalOptions{})
return &m,err
}

The protobuf/ptypes package has utilities to convert to/from arbitrary proto messages to any:
MarshalAny:
func MarshalAny(m proto.Message) (*anypb.Any, error)
MarshalAny marshals the given message m into an anypb.Any message.
UnmarshalAny:
func UnmarshalAny(any *anypb.Any, m proto.Message) error
UnmarshalAny unmarshals the encoded value contained in the anypb.Any message into the provided message m. It returns an error if the target message does not match the type in the Any message or if an unmarshal error occurs.
In your example, you would use something along the lines of:
func logMessage (m string, d proto.Message) {
details, err := ptypes.MarshalAny(d)
if err != nil {
panic(err)
}
message := & example.message{
message: m,
details: details
}
log(&message)
}

func pbany(v interface{}) (*anypb.Any, error) {
pv, ok := v.(proto.Message)
if !ok {
return &anypb.Any{}, fmt.Errorf("%v is not proto.Message", pv)
}
return anypb.New(pv)
use anypb.New api, In your code, pass d to pbany function

func interfaceToAny(v interface{}) (*anypb.Any, error) {
bytes, err := json.Marshal(v)
if err != nil {
println("error json.Marshal interfaceToAny")
return nil, err
}
m := api.Bytes{B: bytes}
return anypb.New(&m)
}
message Bytes {
bytes b = 1;
}

Related

Error while trying to fetch queryresult.KV object in JSON.Unmarshal

I am a little bit confused here and although I have searched a lot on this, something is clearly missing from my knowledge and I am asking your help.
I have created a Hyperledger Fabric Network and installed a chaincode in it. And I want to make a function that retrieves all the World State inputs about the Keys. I have done it already with the bytes.Buffer and it worked. But what I want to do is to do it with a struct.
So, I created the following struct that has only the key:
type WSKeys struct {
Key string `json: "key"`
Namespace string `json: "Namespace"`
}
And this is my code function:
func (s *SmartContract) getAllWsDataStruct(APIstub shim.ChaincodeStubInterface , args []string) sc.Response {
var keyArrayStr []WSKeys
resultsIterator, err := APIstub.GetQueryResult("{\"selector\":{\"_id\":{\"$ne\": null }} }")
if err != nil {
return shim.Error("Error occured when trying to fetch data: "+err.Error())
}
for resultsIterator.HasNext() {
// Get the next record
queryResponse, err := resultsIterator.Next()
if err != nil {
return shim.Error(err.Error())
}
fmt.Println(queryResponse)
var qry_key_json WSKeys
json.Unmarshal([]byte(queryResponse), &qry_key_json)
keyArray = append(keyArray, qry_key_json)
}
defer resultsIterator.Close()
all_bytes, _ := json.Marshal(keyArray)
fmt.Println(keyArray)
return shim.Success(all_bytes)
}
When executing the above I get the following error:
cannot convert queryResponse (type *queryresult.KV) to type []byte
I can get the results correctly if I, for example do this:
func (s *SmartContract) getAllWsDataStruct(APIstub shim.ChaincodeStubInterface , args []string) sc.Response {
var keyArray []string
resultsIterator, err := APIstub.GetQueryResult("{\"selector\":{\"_id\":{\"$ne\": null }} }")
if err != nil {
return shim.Error("Error occured when trying to fetch data: "+err.Error())
}
for resultsIterator.HasNext() {
// Get the next record
queryResponse, err := resultsIterator.Next()
if err != nil {
return shim.Error(err.Error())
}
fmt.Println(queryResponse)
keyArray = append(keyArray, queryResponse.Key)
}
defer resultsIterator.Close()
all_bytes, _ := json.Marshal(keyArray)
fmt.Println(keyArray)
return shim.Success(all_bytes)
}
But, why I get the above error when trying to add the queryResponse into a custom struct?
Do I need to add it to a struct that is only its type?
Please someone can explain what I am missing here?
The error statement is verbose enough to indicate, that your []byte conversion failed for the type queryResponse which, with a bit of lookup seems to be a struct type. In Go you cannot natively convert a struct instance to its constituent bytes without encoding using gob or other means.
Perhaps your intention was to use the Key record in the struct for un-marshalling
json.Unmarshal([]byte(queryResponse.Key), &qry_key_json)

Generic function with variable input/output types

Just playing with aws sdk for go. When listing resources of different types I tend to have alot of very similar functions like the two in the example bellow.
Is there a way to rewrite them as one generic function that will return a specific type depending on what is passed on as param?
Something like:
func generic(session, funcToCall, t, input) (interface{}, error) {}
currently I have to do this (functionality is the same just types change):
func getVolumes(s *session.Session) ([]*ec2.Volume, error) {
client := ec2.New(s)
t := []*ec2.Volume{}
input := ec2.DescribeVolumesInput{}
for {
result, err := client.DescribeVolumes(&input)
if err != nil {
return nil, err
}
t = append(t, result.Volumes...)
if result.NextToken != nil {
input.NextToken = result.NextToken
} else {
break
}
}
return t, nil
}
func getVpcs(s *session.Session) ([]*ec2.Vpc, error) {
client := ec2.New(s)
t := []*ec2.Vpc{}
input := ec2.DescribeVpcsInput{}
for {
result, err := client.DescribeVpcs(&input)
if err != nil {
return nil, err
}
t = append(t, result.Vpcs...)
if result.NextToken != nil {
input.NextToken = result.NextToken
} else {
break
}
}
return t, nil
}
Because you only deal with functions it is possible to use the reflect package to generate functions at runtime.
Using the object type (Volume, Vpc) it is possible to derive all subsequents information to provide a fully generic implementation that is really dry, at the extent at the being more complex and slower.
It is untested, you are welcome to help in testing and fixing it, but something like this should put you on the track
https://play.golang.org/p/mGjtYVG2OZS
The registry idea come from this answer https://stackoverflow.com/a/23031445/4466350
for reference the golang documentation of the reflect package is at https://golang.org/pkg/reflect/
package main
import (
"errors"
"fmt"
"reflect"
)
func main() {
fmt.Printf("%T\n", getter(Volume{}))
fmt.Printf("%T\n", getter(Vpc{}))
}
type DescribeVolumesInput struct{}
type DescribeVpcs struct{}
type Volume struct{}
type Vpc struct{}
type Session struct{}
type Client struct{}
func New(s *Session) Client { return Client{} }
var typeRegistry = make(map[string]reflect.Type)
func init() {
some := []interface{}{DescribeVolumesInput{}, DescribeVpcs{}}
for _, v := range some {
typeRegistry[fmt.Sprintf("%T", v)] = reflect.TypeOf(v)
}
}
var errV = errors.New("")
var errType = reflect.ValueOf(&errV).Elem().Type()
var zeroErr = reflect.Zero(reflect.TypeOf((*error)(nil)).Elem())
var nilErr = []reflect.Value{zeroErr}
func getter(of interface{}) interface{} {
outType := reflect.SliceOf(reflect.PtrTo(reflect.TypeOf(of)))
fnType := reflect.FuncOf([]reflect.Type{reflect.TypeOf(new(Session))}, []reflect.Type{outType, errType}, false)
fnBody := func(input []reflect.Value) []reflect.Value {
client := reflect.ValueOf(New).Call(input)[0]
t := reflect.MakeSlice(outType, 0, 0)
name := fmt.Sprintf("Describe%TsInput", of)
descInput := reflect.New(typeRegistry[name]).Elem()
mName := fmt.Sprintf("Describe%Ts", of)
meth := client.MethodByName(mName)
if !meth.IsValid() {
return []reflect.Value{
t,
reflect.ValueOf(fmt.Errorf("no such method %q", mName)),
}
}
for {
out := meth.Call([]reflect.Value{descInput.Addr()})
if len(out) > 0 {
errOut := out[len(out)-1]
if errOut.Type().Implements(errType) && errOut.IsNil() == false {
return []reflect.Value{t, errOut}
}
}
result := out[1]
fName := fmt.Sprintf("%Ts", of)
if x := result.FieldByName(fName); x.IsValid() {
t = reflect.AppendSlice(t, x)
} else {
return []reflect.Value{
t,
reflect.ValueOf(fmt.Errorf("field not found %q", fName)),
}
}
if x := result.FieldByName("NextToken"); x.IsValid() {
descInput.FieldByName("NextToken").Set(x)
} else {
break
}
}
return []reflect.Value{t, zeroErr}
}
fn := reflect.MakeFunc(fnType, fnBody)
return fn.Interface()
}
Proxying 3rd party API, is quite simple to implement with
go, here is how' it got implemented with endly e2e test runner AWS proxy
I would say that AWS API is perfect candidate for proxying, as long as reflection performance price is not an issue.
Some other 3rd party API like kubernetes
are much more challenging, but still quite easy to proxy with go, which is a combination of reflection and code generation:

How to handle Response JSON have custom field with out key?

Query Api and response a custom JSON, how to Unmarshal it. the sample JSON:
{"14AcKEr19gHJvgwQhK7sfFm6YJGmoZZoqu": {
"final_balance": 61914248289,
"n_tx": 3472,
"total_received": 3479994002972
}}
The key is a hex string. So how to handle it with golang convention, anyone can help me?
Below is my try test code:
c.OnResponse(func(r *colly.Response) {
jsonData := r.Body
fmt.Println(string(jsonData))
fmt.Println("==================")
//parse bitcoin json
jsonMap := make(map[string]interface{})
err := json.Unmarshal([]byte(jsonData), &jsonMap)
if err != nil {
panic(err)
}
fmt.Println(jsonMap)
dumpMap("", jsonMap)
})
func dumpMap(space string, m map[string]interface{}) {
for k, v := range m {
if mv, ok := v.(map[string]interface{}); ok {
fmt.Printf("{ \"%v\": \n", k)
dumpMap(space+"\t", mv)
fmt.Printf("}\n")
} else {
fmt.Printf("%v %v : %v\n", space, k, v)
}
}
}
and go run cmd/main.go, the console is print here:
{"14AcKEr19gHJvgwQhK7sfFm6YJGmoZZoqu": {
"final_balance": 75494521080,
"n_tx": 3493,
"total_received": 3493574275763
}}
==================
map[14AcKEr19gHJvgwQhK7sfFm6YJGmoZZoqu:map[n_tx:3493 total_received:3.493574275763e+12 final_balance:7.549452108e+10]]
{ "14AcKEr19gHJvgwQhK7sfFm6YJGmoZZoqu":
final_balance : 7.549452108e+10
n_tx : 3493
total_received : 3.493574275763e+12
}
Do I need customised unmarshal func to get string key? If I use 14AcKEr19gHJvgwQhK7sfFm6YJGmoZZoqu as key I can't easily to access. I just want to know how handle it.
you can unmarshal it into map, so you can get generated key as a key of map
https://play.golang.org/p/IfEjjvKakpu
package main
import (
"encoding/json"
"fmt"
"log"
)
var input = `{"14AcKEr19gHJvgwQhK7sfFm6YJGmoZZoqu": {
"final_balance": 61914248289,
"n_tx": 3472,
"total_received": 3479994002972
}}`
type object struct {
FinalBalance uint64 `json:"final_balance"`
NTX uint64 `json:"n_tx"`
TotalReceived uint64 `json:"total_received"`
}
func main() {
var result map[string]object;
err := json.Unmarshal([]byte(input), &result);
if err != nil {
log.Fatal(err)
}
fmt.Printf("result: %+v", result)
// result: map[14AcKEr19gHJvgwQhK7sfFm6YJGmoZZoqu:{FinalBalance:61914248289 NTX:3472 TotalReceived:3479994002972}]
}

Combine multiple error strings

I am new to golang, my application needs to return multiple errors in a loop, later requires to be combined and returned as a single error string. I am not able to use the string functions to combine the error messages. What methods can be use to combine these errors into a single error before returning ?
package main
import (
"fmt"
"strings"
)
func Servreturn() (err error) {
err1 = fmt.Errorf("Something else occured")
err2 = fmt.Errorf("Something else occured again")
// concatenate both the error
return err3
}
UPDATE for Go 1.13:
As of Go version 1.13, the language's errors package now supports error wrapping directly.
You can wrap an error by using the %w verb in fmt.Errorf:
err := errors.New("Original error")
err = fmt.Errorf("%w; Second error", err)
Use Unwrap to remove the last error added, and return what remains: previousErrors := errors.Unwrap(err)
Playground Example for errors.Unwrap
Two more functions, errors.Is and errors.As provide ways to check for and retrieve a specific type of error.
Playground Example for errors.As and errors.Is
Dave Cheney's excellent errors package (https://github.com/pkg/errors) include a Wrap function for this purpose:
package main
import "fmt"
import "github.com/pkg/errors"
func main() {
err := errors.New("error")
err = errors.Wrap(err, "open failed")
err = errors.Wrap(err, "read config failed")
fmt.Println(err) // "read config failed: open failed: error"
}
This also allows additional functionality, such as unpacking the cause of the error:
package main
import "fmt"
import "github.com/pkg/errors"
func main() {
err := errors.New("original error")
err = errors.Wrap(err, "now this")
fmt.Println(errors.Cause(err)) // "original error"
}
As well as the option to output a stack trace when specifying fmt.Printf("%+v\n", err).
You can find additional information about the package on his blog: here and here.
String functions don't work on errors because error is really an interface that implements the function Error() string.
You can use string functions on err1.Error() and err2.Error()
but not on the "err1" reference itself.
Some errors are structs, like the ones you get from database drivers.
So there's no natural way to use string functions on errors since they may not actually be strings underneath.
As for combining two errors:
Easy, just use fmt.Errorf again.
fmt.Errorf("Combined error: %v %v", err1, err2)
Alternatively:
errors.New(err1.Error() + err2.Error())
You could use the strings.Join() and append() function to acheive this slice.
example: golang playgorund
package main
import (
"fmt"
"strings"
"syscall"
)
func main() {
// create a slice for the errors
var errstrings []string
// first error
err1 := fmt.Errorf("First error:server error")
errstrings = append(errstrings, err1.Error())
// do something
err2 := fmt.Errorf("Second error:%s", syscall.ENOPKG.Error())
errstrings = append(errstrings, err2.Error())
// do something else
err3 := fmt.Errorf("Third error:%s", syscall.ENOTCONN.Error())
errstrings = append(errstrings, err3.Error())
// combine and print all the error
fmt.Println(fmt.Errorf(strings.Join(errstrings, "\n")))
}
This would output a single string which you can send back to the client.
First error:server1
Second error:Package not installed
Third error:Socket is not connected
hope this helps!
To expand on what #WillC had mentioned in a comment it is possible to define your own error type as error is an interface type. Any type that implements a Error() string function implements the error interface. Therefore, you could create a CollectionError which aggregates errors and returns a concatenated error string.
type ErrorCollector []error
func (c *ErrorCollector) Collect(e error) { *c = append(*c, e) }
func (c *ErrorCollector) Error() (err string) {
err = "Collected errors:\n"
for i, e := range *c {
err += fmt.Sprintf("\tError %d: %s\n", i, e.Error())
}
return err
}
This provides a collection function that appends a given error to a slice. Upon calling Error() string it iterates over the slice and creates a concatenated error string.
func main() {
collector := new(ErrorCollector)
for i := 0; i < 10; i++ {
collector.Collect(errors.New(fmt.Sprintf("%d Error", i)))
}
fmt.Println(collector)
}
There is a great golang.org blog post going over errors in more detail. A full example of the example is available on The Go Playground.
Uber has a multierr package for this use case:
return multierr.Combine(err1, err2)
People may be interested in https://github.com/hashicorp/go-multierror which describes itself as "A Go (golang) package for representing a list of errors as a single error.".
As of Go 1.20, we'll be able to wrap multiple errors using errors.Join.
See this proposal for more details.
First Option
You can print the errors; they will be separated by a newline character when you do that.
var (
ErrIncorrectUsername = errors.New("incorrect username")
ErrIncorrectPassword = errors.New("incorrect password")
)
func main() {
err := validate("ruster", "4321")
// You can print multi-line errors
// Each will be separated by a newline character (\n).
if err != nil {
fmt.Println(err)
// incorrect username
// incorrect password
}
}
func validate(username, password string) error {
var errs []error
// errors.Join the errors into a single error
if username != "gopher" {
errs = append(errs, ErrIncorrectUsername)
}
if password != "1234" {
errs = append(errs, ErrIncorrectPassword)
}
// Join returns a single `error`.
// Underlying, the error contains all the errors we add.
return errors.Join(errs...)
}
Second Option
errors.Join returns an error that contains each error you add. So you can use errors.Is and errors.As to check for individual errors for finer granularity.
// ...
func main() {
err := validate("ruster", "4321")
// You can detect each one:
if errors.Is(err, ErrIncorrectUsername) {
// handle the error here
}
// Or detect the other one:
if errors.Is(err, ErrIncorrectPassword) {
// handle the error here
}
}
func validate(username, password string) error {
// ...
}
Note: This naive validate example is here to convey the idea. Instead of chaining errors, think errors like a tree. Join allows you to do that when combined with other Joins.
Run both on Go Playground.
This seems to work well for me (space separated errors):
Put all your errors in a error slice/list/array, ie: var errors [] error
fmt.Sprintf("%s", errors)
var errors []error
errors = append(errors, fmt.Errorf("error 1"))
errors = append(errors, fmt.Errorf("error 2"))
errors = append(errors, fmt.Errorf("and yet another error"))
s := fmt.Sprintf("%s", errors)
fmt.Println(s)
Sometimes i need the way to detect if there some error in the chain. the standard way, provided with https://pkg.go.dev/errors is pretty convenient:
someErr:=errors.New("my error")
fmt.Errorf("can't process request: %w",someErr)
...
err:=f()
if errors.Is(err,someErr){...}
But it could be applied only in the case of detection of the last error in the chain. You can't wrap someErr1 and then someErr2 and then get true from both of checks: errors.Is(err,someErr1) and errors.Is(err,someErr2)
I solved this problem with following type:
func NewJoinedErrors(err1 error, err2 error) JoinedErrors {
return JoinedErrors{err1: err1, err2: err2}
}
type JoinedErrors struct {
err1 error
err2 error
}
func (e JoinedErrors) Error() string {
return fmt.Sprintf("%s: %s", e.err1, e.err2)
}
func (e JoinedErrors) Unwrap() error {
return e.err2
}
func (e JoinedErrors) Is(target error) bool {
return errors.Is(e.err1, target)
}
It uses the fact, that
An error is considered to match a target if it is equal to that target
or if it implements a method Is(error) bool such that Is(target)
returns true.
So you can join two errors and get positive result on both checks:
someErr1:=errors.New("my error 1")
someErr2:=errors.New("my error 2")
err:=NewJoinedErrors(someErr1, someErr2)
// this will be true because
// (e JoinedErrors) Is(target error)
// will return true
if errors.Is(err, someErr1){...}
// this will be true because
// (e JoinedErrors) Unwrap() error
// will return err2
if errors.Is(err, someErr2){...}
you can check it out here: https://play.golang.org/p/W7NGyfvr0v_N
func condenseErrors(errs []error) error {
switch len(errs) {
case 0:
return nil
case 1:
return errs[0]
}
err := errs[0]
for _, e := range errs[1:] {
err = errors.Wrap(err, e.Error())
}
return err
}
Use this function:
func JoinErrs(errs ...error) error {
var joinErrsR func(string, int, ...error) error
joinErrsR = func(soFar string, count int, errs ...error) error {
if len(errs) == 0 {
if count == 0 {
return nil
}
return fmt.Errorf(soFar)
}
current := errs[0]
next := errs[1:]
if current == nil {
return joinErrsR(soFar, count, next...)
}
count++
if count == 1 {
return joinErrsR(fmt.Sprintf("%s", current), count, next...)
} else if count == 2 {
return joinErrsR(fmt.Sprintf("1: %s\n2: %s", soFar, current), count, next...)
}
return joinErrsR(fmt.Sprintf("%s\n%d: %s", soFar, count, current), count, next...)
}
return joinErrsR("", 0, errs...)
}
It will give you nil when all errors are nil, will give you the same error when just one non-nil error, will give you numbered list of non-nil errors when multiple non-nil errors
Try it out here: https://play.golang.org/p/WttztCr-xHG

Idiomatic way to pass any type in Go (while keeping compile time type-checking?)

I am parsing a form and have written a number of functions func parseAndValidateX(val string) (T, err) where T is any type.
Now I would like to write a closure func catchError(T, Error) T, so that I can do something like:
errors []Error
func catchError(val T, err Error) T {
if err != nil {
//append err to errors
}
return val
}
data = MyStruct {
Age = catchError(parseAndValidateAge("5")) // Age is a int
DistanceFromHome = catchError(parseAndValidatePi("3.14")) // DistanceFromHome is a float
Location = catchError(parseAndValidatePi("3.14,2.0")) //Location is a custom Struct
}
if len(errors) > 0 {
// o, oh
}
Is this possible in Go? How can this be done easily/idiomatically?
Nope; you cannot do that since Go has no parametric polymorphism for user-defined functions. All you can do is take and return interface{} and add type assertions at the call sites.
Everything in Go is type-safe if you don't use the unsafe package, so you don't have to worry about that. A type assertion will fail at runtime instead of at compile-time, though.
If you are willing to violate DRY, though:
type errorList []Error
func (l *errorList) addIfNotNil(err Error) {
if err != nil {
*l = append(*l, err)
}
}
func (l *errorList) int(x int, err Error) int {
l.addIfNotNil(err)
return x
}
func (l *errorList) float32(x float32, err Error) float32 {
l.addIfNotNil(err)
return x
}
list := errorList([]Error{})
data := MyStruct{
Age: list.int(parseAndValidateAge("5")),
DistanceFromHome: list.float32(parseAndValidatePi("3.14")),
}

Resources