Go: Implementing a ManyDecode for a "set" of individual results - go

I have implemented a very simple Decode method (using gob.Decoder for now) - this works well for single responses - it would even work well for slices, but I need to implement a DecodeMany method where it is able to decode a set of individual responses (not a slice).
Working Decode method:
var v MyType
_ = Decode(&v)
...
func Decode(v interface{}) error {
buf, _ := DoSomething() // func DoSomething() ([]byte, error)
// error handling omitted for brevity
return gob.NewDecoder(bytes.NewReader(buf)).Decode(v)
}
What I'm trying to do for a DecodeMany method is to deal with a response that isn't necessarily a slice:
var vv []MyType
_ = DecodeMany(&vv)
...
func DecodeMany(vv []interface{}) error {
for _, g := range DoSomething() { // func DoSomething() []struct{Buf []bytes}
// Use g.Buf as an individual "interface{}"
// want something like:
var v interface{} /* Somehow create instance of single vv type? */
_ = gob.NewDecoder(bytes.NewReader(g.Buf)).Decode(v)
vv = append(vv, v)
}
return
}
Besides not compiling the above also has the error of:
cannot use &vv (value of type *[]MyType) as type []interface{} in argument to DecodeMany

If you want to modify the passed slice, it must be a pointer, else you must return a new slice. Also if the function is declared to have a param of type []interface{}, you can only pass a value of type []interface{} and no other slice types... Unless you use generics...
This is a perfect example to start using generics introduced in Go 1.18.
Change DecodeMany() to be generic, having a T type parameter being the slice element type:
When taking a pointer
func DecodeMany[T any](vv *[]T) error {
for _, g := range DoSomething() {
var v T
if err := gob.NewDecoder(bytes.NewReader(g.Buf)).Decode(&v); err != nil {
return err
}
*vv = append(*vv, v)
}
return nil
}
Here's a simple app to test it:
type MyType struct {
S int64
}
func main() {
var vv []MyType
if err := DecodeMany(&vv); err != nil {
panic(err)
}
fmt.Println(vv)
}
func DoSomething() (result []struct{ Buf []byte }) {
for i := 3; i < 6; i++ {
buf := &bytes.Buffer{}
v := MyType{S: int64(i)}
if err := gob.NewEncoder(buf).Encode(v); err != nil {
panic(err)
}
result = append(result, struct{ Buf []byte }{buf.Bytes()})
}
return
}
This outputs (try it on the Go Playground):
[{3} {4} {5}]
When returning a slice
If you choose to return the slice, you don't have to pass anything, but you need to assign the result:
func DecodeMany[T any]() ([]T, error) {
var result []T
for _, g := range DoSomething() {
var v T
if err := gob.NewDecoder(bytes.NewReader(g.Buf)).Decode(&v); err != nil {
return result, err
}
result = append(result, v)
}
return result, nil
}
Using it:
vv, err := DecodeMany[MyType]()
if err != nil {
panic(err)
}
fmt.Println(vv)
Try this one on the Go Playground.

Related

Implement a struct-to-csv writer in Go

The following code attempt to implement a generic CSV writer for any simple struct. By "simple", I mean field value of the struct are of standard, simple types (int, string etc).
type (
CSV interface {
Header() []string
String([]string) (string, error)
}
CSVArray []CSV
)
func CSVOutput(w io.Writer, data CSVArray, cols []string) error {
if len(data) == 0 {
return nil
}
_, err := fmt.Fprintln(w, data[0].Header())
if err != nil {
return err
}
for _, d := range data {
str, err := d.String(cols)
if err != nil {
return err
}
_, err = fmt.Fprintln(w, str)
if err != nil {
return err
}
}
return nil
}
The problem is CSVOutput() does not actually work. e.g.:
var data []Employee //the Employee struct implements CSV interface
CSVOutput(w, data, nil)
Compilation failed: cannot use data (type []Employee) as type CSVArray in argument to CSVOutput
I understand that []CSV is not same as []Employee, as explained here, and many other resources available online.
That said, is it possible to rewrite the CSVOutput() function by using reflection:
func CSVOutput(w io.Writer, data interfac{}, cols []string) error {
sliceOfIntf = castToSlice(data) //how to do this?
if !implementedCSV(sliceOfIntf[0]) { //and how to do this?
return errors.New("not csv")
}
... ...
}
is it possible to rewrite the CSVOutput() function by using reflection
Yes
// if data is []Employee{...}, then you can do the following:
rv := reflect.ValueOf(data)
if rv.Kind() != reflect.Slice {
return fmt.Errorf("data is not slice")
}
if !rv.Type().Elem().Implements(reflect.TypeOf((*CSV)(nil)).Elem()) {
return fmt.Errorf("slice element does not implement CSV")
}
csvArr := make(CSVArray, rv.Len())
for i := 0; i < rv.Len(); i++ {
csvArr[i] = rv.Index(i).Interface().(CSV)
}
// now csvArr is CSVArray containing all the elements of data
https://go.dev/play/p/gcSOid533gx

Dynamic return type in a function

I have a function that is called from an other package. This function parses a text string received in the argument and returns a splice of structs. These structs are, in my case, 7 and all differ in some fields. I'm trying to return an interface{} type but I'd like to be able to assert the type on the receiving end as I need to do other operations on that struct.
So far I've got to this point (the function are meant to be in different packages):
func Process(input string) interface{} {
// ...
switch model {
case 1:
output := model1.Parse(input) // this function returns a []MOD1 type
return output
}
case 2:
output := model2.Parse(input) // this function returns a []MOD2 type
return output
}
// other cases with other models
}
func main() {
// ...
output := package.Process(input) // now output is of type interface{}
// I need the splice because I'll publish each element to a PubSub topic
for _, doc := range output {
obj, err := json.Marshal(doc)
if err != nil {
// err handling
}
err = publisher.Push(obj)
if err != nil {
// err handling
}
}
}
Now in main() I'd like output to be of type []MOD1, []MOD2, ..., or []MOD7.
I've tried the to use the switch t := output.(type) but t exists only in the scope of the switch case and it doesn't really solve my problem.
func main() {
output := package.Process(input)
switch t := output.(type) {
case []MOD1:
output = t // output is still of type interface{}
case []MOD2:
output = t
}
}
I need this because in the main func I have to do other operations on that structure and eventually Marshall it into a JSON. I've thought of marshalling the structs in the Process func and return the []byte but then I'd need to unmarshall in main without knowing the struct's type.
Change Process1 to return a []interface{}:
func Process(input string) []interface{} {
// ...
switch model {
case 1:
output := model1.Parse(input)
result := make([]interface{}, len(output))
for i := range result {
result[i] = output[i]
}
return result
}
case 2:
// other cases with other models
}
Work with the []interface{} in main:
output := package.Process(input)
for _, doc := range output {
obj, err := json.Marshal(doc)
if err != nil {
// err handling
}
err = publisher.Push(obj)
if err != nil {
// err handling
}
}

Transform struct to slice struct

I'm trying to select a struct by string input and then depending on the return JSON Object or Array, unmarshall the JSON. Is it correct to think of a way to reflect the struct to slice struct? if so how to do that with reflection?
Regards,
Peter
package main
import (
"bytes"
"encoding/json"
"fmt"
"log"
)
type NameStruct struct {
Name string
}
func main() {
jsonData := []byte(`[{"name":"james"},{"name":"steven"}]`)
returnModel := InitializeModel("NameStruct", jsonData)
fmt.Println(returnModel)
jsonData = []byte(`{"name":"james"}`)
returnModel = InitializeModel("NameStruct", jsonData)
fmt.Println(returnModel)
}
func getModelByName(modelType string) interface{} {
modelMap := make(map[string]interface{})
modelMap["NameStruct"] = new(NameStruct)
//don't want to do this
modelMap["arrNameStruct"] = new([]NameStruct)
return modelMap[modelType]
}
func InitializeModel(modelName string, jsonData []byte) interface{} {
switch IsArray(jsonData) {
case true:
// some conversion here, how?
returnModel := getModelByName("NameStruct")
if err := json.Unmarshal(jsonData, &returnModel); err != nil {
log.Println(err)
}
return returnModel
case false:
returnModel := getModelByName("NameStruct")
if err := json.Unmarshal(jsonData, &returnModel); err != nil {
log.Println(err)
}
return returnModel
}
return nil
}
func IsArray(jsonData []byte) bool {
return (bytes.HasPrefix(jsonData, []byte("["))) && (bytes.HasSuffix(jsonData, []byte("]")))
}
Expanding on my comment, you can create a Factory where pre-defined types are registered:
type Factory struct {
m map[string]reflect.Type
}
func (f *Factory) Register(v interface{}) {
vt := reflect.TypeOf(v)
n := vt.Name()
f.m[n] = vt
f.m["[]"+n] = reflect.SliceOf(vt) // implicitly register a slice of type too
}
these types can be looked up by name at runtime and initialized with JSON data:
func (f *Factory) Make(k string, bs []byte) (interface{}, error) {
vt, ok := f.m[k]
if !ok {
return nil, fmt.Errorf("type %q not registered", k)
}
pv := reflect.New(vt).Interface()
err := json.Unmarshal(bs, pv)
if err != nil {
return nil, err
}
return pv, nil
}
To use:
type Place struct {
City string `json:"city"`
}
factory.Register(Place{})
p, err := factory.Make("Place", []byte(`{"city":"NYC"}`))
fmt.Printf("%#v\n", p) // &main.Place{City:"NYC"}
Slices also work:
ps, err := factory.Make("[]Place", []byte(`[{"city":"NYC"},{"city":"Dublin"}]`))
fmt.Printf("%#v\n", p, p) // &[]main.Place{main.Place{City:"NYC"}, main.Place{City:"Dublin"}}
Playground: https://play.golang.org/p/qWEdwk-YUug

golang reflect into []interface{}

i wanna create a mini framework which takes a simple struct and creates a full crud out of it. i already started and the "findOne, update,create,delete" is working. not i have a problem to create a findAll method. to be more clear, i dont know how to use reflect to address my ptr to an array of struct.
Here a small example for the findOne function.
type company struct {
Id int
Name string
}
comp.InitModel(newDbConnection(), &comp)
In InitModel i can fill the pointer to the company with the following:
//m.caller = pointer to the ptr to comp (struct)
callerV := reflect.ValueOf(m.caller)
CallerField := callerV.Elem()
var values []interface{}
for _, e := range m.columns {
values = append(values, CallerField.FieldByName(e.name).Addr().Interface())
}
err := r.Scan(values...)
if err != nil {
return err
}
Now i wanna create a findAll method which would be called like this
var companies []company
comp.InitModel(newDbConnection(), &comp)
comp.FindAll(&companies) //in this is the db query and scan
fmt.Println(companies) //here should be the result
But i have a problem to get the reflect with a []interface working.
func (m *Model) FindAll(test []interface{}, c *Condition) error {
//get the sql statement from the struct
stmt := PrepairStmt(m, c)
rows, err := m.db.Query(stmt.selectParse(), c.arguments...)
if err != nil {
return err
}
defer rows.Close()
callerV := reflect.ValueOf(m.caller)
CallerField := callerV.Elem()
for rows.Next() {
var values []interface{}
for _, e := range m.columns {
values = append(values, CallerField.FieldByName(e.name).Addr().Interface())
}
err = rows.Scan(values...)
if err != nil {
return err
}
valuePtr := reflect.New(reflect.TypeOf(test).Elem())
test = reflect.Append(test,reflect.ValueOf(values))
}
return nil
}
Thats were my latest tries.
maybe someone can help me with this.
i would be really thankful
Use interface{} instead of []interface{} as the argument type:
func (m *Model) FindAll(result interface{}, c *Condition) error {
stmt := PrepairStmt(m, c)
rows, err := m.db.Query(stmt.selectParse(), c.arguments...)
if err != nil {
return err
}
defer rows.Close()
// resultv is the result slice
resultv := reflect.ValueOf(result).Elem()
// rowt is the struct type
rowt := resultv.Type().Elem()
// allocate a value for the row
rowv := reflect.New(rowt).Elem()
// collect values for scan
var values []interface{}
for _, e := range m.columns {
values = append(values, rowv.FieldByName(e.name).Addr().Interface())
}
for rows.Next() {
err = rows.Scan(values...)
if err != nil {
return err
}
// Append struct to result slice. Because the struct
// is copied in append, we can reuse the struct in
// this loop.
resultv.Set(reflect.Append(resultv, rowv))
}
return nil
}
Use it like this:
var companies []company
comp.InitModel(newDbConnection(), &comp)
comp.FindAll(&companies) //in this is the db query and scan

Using sqlx.Rows.StructScan for interface args

I need to use StructScan function for interface (pointer to struct).
But if I try to reflect value, I've got error, because reflect.New() returning reflect.Value type. How I can scan structure and store data into dest var?
// package 1
type Data struct {
id int `db:"id"`
caption string `db:"caption"`
}
func Func1 {
data := []Data{}
GetData(&data)
log.Println(data)
}
// package 2
func GetData(sql string, dest interface{}) {
rows, err := DBI.Queryx(sql)
if err == nil {
// reflect.Value
myData := reflect.New(reflect.TypeOf(dest).Elem().Elem())
for rows.Next() {
rows.StructScan(&myData) // Fail here
}
}
}
Solved
// package 2
func GetData(sql string, dest interface{}) {
arr := reflect.ValueOf(dest).Elem()
v := reflect.New(reflect.TypeOf(dest).Elem().Elem())
rows, err := DBI.Queryx(sql)
if err == nil {
if err = rows.StructScan(v.Interface()); err == nil {
arr.Set(reflect.Append(arr, v.Elem()))
}
}
}

Resources