I'm building an http api and every one of my handlers returns JSON data, so I built a wrapper function that handles the JSON marshalling and http response (I've included the relevant section from the wrapper as well as one of the sample handlers below).
What is the best way to pass arbitrarily nested structs (the structs also contain arbitrary types/number of fields). Right now I've settled on a map with string keys and interface{} values. This works, but is this the most idiomatic go way to do this?
result := make(map[string]interface{})
customerList(httpRequest, &result)
j, err := json.Marshal(result)
if err != nil {
log.Println(err)
errs := `{"error": "json.Marshal failed"}`
w.Write([]byte(errs))
return
}
w.Write(j)
func customerList(req *http.Request, result *map[string]interface{}) {
data, err := database.RecentFiftyCustomers()
if err != nil {
(*result)["error"] = stringifyErr(err, "customerList()")
return
}
(*result)["customers"] = data//data is a slice of arbitrarily nested structs
}
If you do not know in advance what types, what structure and which nesting you get, there is no option but to decode it into something generic like map[string]interface{}. So nothing "idiomatic" or "non-idiomatic" here.
(Personally I'd try to somehow fix the structs and not have "arbitrary" nestings, and combinations.)
Related
I am writing some utils to retrieve multiple rows from BigQuery in a generic way using Go.
e.g.
type User struct {name string, surname string}
type Car struct {model string, platenumber string}
query1:="SELECT name, surname FROM UserTable"
query2:="SELECT model, platenumber FROM CarTable"
cars, _ := query2.GetResults()
users, _ := query1.GetResults()
OR
cars := []Car{}
query2.GetResults(cars) // and it would append to the slice
I am unsure about the signature of GetResults. I need somehow to pass the type to BigQuery library so it can retrieve the data and map it to the struct correctly. But at the same time I need to make it generic so it can be used for different types.
At the moment my GetResults looks like this: it doesn't work, the error is:
bigquery: cannot convert *interface {} to ValueLoader (need pointer to []Value, map[string]Value, or struct)[]
But I cannot pass directly the struct as I want to make it generic.
func (s *Query) GetResults() ([]interface{}, error) {
var result []interface{}
job, err := s.Run()
if err != nil {
s.log.Error(err, "error in running the query")
return nil, err
}
it, err := job.ReadData()
if err != nil {
s.log.Error(err, "error in reading the data")
return nil, err
}
var row interface{}
for {
err := it.Next(&row)
if err != nil {
fmt.Print(err)
break
}
result = append(result, row)
}
return result, nil
}
Is there another way to achieve that? Or is the good way not to create a method like that?
I've tried quite a lot of different things, with or without pointer, with or without array, by modifying the args, or returning a new list, nothing seem to work, and doing all of that feels a bit wrong regarding the nature "easy" of what I am trying to achieve.
I've also looked into doing the following
GetResults[T any]() ([]T, error)
But it's "excluded" as GetResults is part of an interface (and we can't define generic for a method of an interface). And I can't/don't want to define a type for all the interface, as it impacts other interfaces.
I would like to marshal in and out of x-www-form-urlencoding similar to how you can do it with json or xml. Is there an existing package to do this, or are there any documents on how to implement one myself if none exist?
gorilla/schema is popular and well maintained:
e.g.
func FormHandler(w http.RequestWriter, r *http.Request) {
err := r.ParseForm()
if err != nil {
// handle error
}
person := new(Person) // Person being a struct type
decoder := schema.NewDecoder()
err = decoder.Decode(person, r.Form)
if err != nil {
// handle error
}
}
goforms is also an alternative.
Update May 23rd 2015:
gorilla/schema is still my pick as one of the most-supported map-to-struct packages, with POST form values being a common use-case.
goji/param is also fairly solid and has many of the same features.
mholt/binding is a little more feature packed at the (IMO) expense of a slightly more complex API.
I've been using gorilla/schema for a couple of years now and haven't had any major issues with it. I use it in conjunction with vala for validating inputs (not nil, too short, too long, etc) before they hit the DB.
I just found https://github.com/ajg/form which is exactly what I was looking for. There is also https://github.com/gorilla/schema for strictly decoding and https://github.com/google/go-querystring for strictly encoding.
net/url seems to handle this just fine:
package main
import (
"fmt"
"net/url"
)
func main() {
{
m := url.Values{
"CR": {"\r"}, "LF": {"\n"},
}
s := m.Encode()
fmt.Println(s) // CR=%0D&LF=%0A
}
{
s := "CR=%0D&LF=%0A"
m, e := url.ParseQuery(s)
if e != nil {
panic(e)
}
fmt.Printf("%q\n", m) // map["CR":["\r"] "LF":["\n"]]
}
}
https://golang.org/pkg/net/url#ParseQuery
https://golang.org/pkg/net/url#Values.Encode
https://github.com/google/go-querystring is good, but doesn't support maps (and slices of maps).
I started https://github.com/drewlesueur/querystring for map support.
(It doesn't support structs yet, but pull requests welcome).
I have a Go project where I'm using Zap structured logging to log the contents of structs. That's how I initialise the logger:
zapLog, err := zap.NewProductionConfig().Build()
if err != nil {
panic(err)
}
Initially I started with my own structs with json tags and it all worked perfectly:
zapLog.Info("Event persisted", zap.Any("event", &event))
Result:
{"level":"info","ts":1626448680.69099,"caller":"persisters/log.go:56",
"msg":"Event persisted","event":{"sourceType":4, "sourceId":"some-source-id",
"type":"updated", "value":"{...}", "context":{"foo":"bar"}}}
I now switched to protobuf and I'm struggling to achieve the same result. Initially I just got the "reflected map" version, when using zap.Any():
zapLog.Info("Event persisted", zap.Any("event", &event))
{"level":"info","ts":1626448680.69099,"caller":"persisters/log.go:56",
"msg":"Event persisted","event":"sourceType:TYPE_X sourceId:\"some-source-id\",
type:\"updated\" value:{...}, context:<key: foo, value:bar>}
I tried marshalling the object with the jsonpb marshaller, which generated the correct output on itself, however, when I use it in zap.String(), the string is escaped, so I get an extra set of '\' in front of each quotation mark. Since there's processing of the logs at a later point, this causes problems there and hence I want to avoid it:
m := jsonpb.Marshaler{}
var buf bytes.Buffer
if err := m.Marshal(&buf, msg); err != nil {
// handle error
}
zapLog.Info("Event persisted", zap.ByteString("event", buf.Bytes()))
Result:
{"level":"info","ts":1626448680.69099,"caller":"persisters/log.go:56",
"msg":"Event persisted","event":"{\"sourceType\":\"TYPE_X\", \"sourceId\":\"some-source-id\",
\"type\":\"updated\", \"value\":\"{...}\", \"context\":{\"foo\":"bar\"}}"}
I then tried using zap.Reflect() instead of zap.Any() which was the closest thing I could get to what I need, except that enums are rendered as their numerical values (the initial solution did not have enums, so that didn't work in the pre-protobuf solution either):
zapLog.Info("Event persisted", zap.Reflect("event", &event))
Result:
{"level":"info","ts":1626448680.69099,"caller":"persisters/log.go:56",
"msg":"Event persisted","event":{"sourceType":4, "sourceId":"some-source-id",
"type":"updated", "value":"{...}", "context":{"foo":"bar"}}}
The only option I see so far is to write my own MarshalLogObject() function:
type ZapEvent struct {
event *Event
}
func (z *ZapEvent) MarshalLogObject(encoder zapcore.ObjectEncoder) error {
encoder.AddString("sourceType", z.event.SourceType.String()
// implement encoder for each attribute
}
func processEvent(e Event) {
...
zapLog.Info("Event persisted", zap.Object("event", &ZapEvent{event: &e}))
}
But since it's a complex struct, I would rather use a less error prone and maintenance heavy solution. Ideally, I would tell zap to use the jsonpb marshaller somehow, but I don't know if that's possible.
Use zap.Any with a json.RawMessage. You can convert directly the byte output of jsonpb.Marshaler:
foo := &pb.FooMsg{
Foo: "blah",
Bar: 1,
}
m := jsonpb.Marshaler{}
var buf bytes.Buffer
if err := m.Marshal(&buf, foo); err != nil {
// handle error
}
logger, _ := zap.NewDevelopment()
logger.Info("Event persisted", zap.Any("event", json.RawMessage(buf.Bytes())))
The bytes will be printed as:
Event persisted {"event": {"foo":"blah","bar":"1"}}`
I believe that's the easiest way, however I'm also aware of a package kazegusuri/go-proto-zap-marshaler (I'm not affiliated to it) that generates MarshalLogObject() implementations as a protoc plugin. You may want to take a look at that too.
I used another way to jsonify protos.
Since protos can be naturally marshaled, I just wrapped them in the strict-to-json marshaler.
And you can modify the internals to use protojson (newer jsonpb).
Unlike the marshaler in the previous solution, this one doesn't require ahead-of-logging processing.
type jsonObjectMarshaler struct {
obj any
}
func (j *jsonObjectMarshaler) MarshalJSON() ([]byte, error) {
bytes, err := json.Marshal(j.obj)
// bytes, err := protojson.Marshal(j.obj)
if err != nil {
return nil, fmt.Errorf("json marshaling failed: %w", err)
}
return bytes, nil
}
func ZapJsonable(key string, obj any) zap.Field {
return zap.Reflect(key, &jsonObjectMarshaler{obj: obj})
}
Then to use it, just
logger, _ := zap.NewDevelopment()
logger.Info("Event persisted", ZapJsonable("event", buf))
I'm trying to write in Go custom cache for Google DataStore (more precisely - a wrapper around one of existing cache libraries). At cache initialisation, it should accept any custom type of struct (with appropriately-defined datastore fields), which then would be the basis for all items stored. The idea is that cache can be created/initialised for various types which reflect the structure of a particular DataStore entry (CustomEntry)
Approach 1 - store reflect.Type and use it. Problem encountered - can't iterate over a slice of a custom type
type CustomEntry struct {
Data struct {
name string `datastore:"name,noindex"`
address []string `datastore:"address,noindex"`
} `datastore:"data,noindex"`
}
func (cache *MyCache) CacheData(dataQuery string, dataType reflect.Type) {
slice := reflect.MakeSlice(reflect.SliceOf(dataType), 10, 10)
if keys, err := DataStoreClient.GetAll(cache.ctx, datastore.NewQuery(dataQuery), &slice); err != nil {
//handle error
} else {
for i, dataEntry:= range slice {
// ERROR: Cannot range over 'slice' (type Value)
cache.Set(keys[i].Name, dataEntry)
}
}
//usage: Cache.CacheData("Person", reflect.TypeOf(CustomEntry{})
Approach 2 - accept an array of interfaces as arguments. Problem encountered = []CustomEntry is not []interface{}
func (cache *MyCache) CacheData(dataQuery string, dataType []interface{}) {
if keys, err := DataStoreClient.GetAll(cache.ctx, datastore.NewQuery(dataQuery), &dataType); err != nil {
//handle error
} else {
for i, dataEntry:= range slice {
// this seems to work fine
cache.Set(keys[i].Name, dataEntry)
}
}
//usage:
var dataType []CustomEntry
Cache.CacheData("Person", data)
// ERROR: Cannot use 'data' (type []CustomEntry) as type []interface{}
Any suggestions would be highly appreciated.
I have found a solution and thought it might be worth sharing in case anyone else has a similar problem.
The easiest way is to initiate a slice of structs which the DataStore is expected to receive, and then to pass a pointer to it as an argument (interface{}) into the desired function. DataStore, similarly to a few unmarshaling functions (I have tried with JSON package) will be able to successfully append the data to it.
Trying to dynamically create the slice within the function, given a certain Type, which would be then accepted by a function (such as DataStore client) might be quite difficult (I have not managed to find a way to do it). Similarly, passing a slice of interfaces (to allow for easy iteration) only complicates things.
Secondly, in order to iterate over the data (e.g. to store it in cache), it is necessary to:
(1) retrieve the underlying value of the interface (i.e. the pointer itself) - this can be achieved using reflect.ValueOf(pointerInterface),
(2) dereference the pointer so that we obtain access to the underlying, iterable slice of structs - this can be done by invoking .Elem(),
(3) iterate over the underlying slice using .Index(i) method (range will not accept an interface, even if the underlying type is iterable).
Naturally, adding a number of switch-case statements might be appropriate to ensure that any errors are caught rather than cause a runtime panic.
Hence the following code provides a working solution to the above problem:
In main:
var data []customEntry
c.CacheData("Person",&data)
And the function itself:
func (cache *MyCache) CacheData(dataQuery string, data interface{}) error {
if keys, err := DataStoreClient.GetAll(cache.ctx, datastore.NewQuery(dataQuery), data); err != nil {
return err
} else {
s := reflect.ValueOf(data).Elem()
for i := 0; i < s.Len(); i++ {
cache.Set(keys[i].Name, s.Index(i), 1)
}
}
}
With reference to this:
https://play.golang.org/p/0kYRHO5f7kE
If I have 20+ different fields, if one of the fields in the Struct is empty, don't update it. Only update the ones with values in them.
What's the best way forward? I've seen passing as variadic input to another function but how best can I do this elegantly?
you can use this library to convert your struct fields into map of interfaces (can be done by yourself using reflect from stdlib) then loop over it
pipe := redisClient.TxPipeline()
m := structs.Map(server)
for k, v := range m {
pipe.HMSet(username, k, v)
}
cmder, err := pipe.Exec()
if err != nil {
return nil, err
}
the driver for redis used is go-redis