Implement a struct-to-csv writer in Go - go

The following code attempt to implement a generic CSV writer for any simple struct. By "simple", I mean field value of the struct are of standard, simple types (int, string etc).
type (
CSV interface {
Header() []string
String([]string) (string, error)
}
CSVArray []CSV
)
func CSVOutput(w io.Writer, data CSVArray, cols []string) error {
if len(data) == 0 {
return nil
}
_, err := fmt.Fprintln(w, data[0].Header())
if err != nil {
return err
}
for _, d := range data {
str, err := d.String(cols)
if err != nil {
return err
}
_, err = fmt.Fprintln(w, str)
if err != nil {
return err
}
}
return nil
}
The problem is CSVOutput() does not actually work. e.g.:
var data []Employee //the Employee struct implements CSV interface
CSVOutput(w, data, nil)
Compilation failed: cannot use data (type []Employee) as type CSVArray in argument to CSVOutput
I understand that []CSV is not same as []Employee, as explained here, and many other resources available online.
That said, is it possible to rewrite the CSVOutput() function by using reflection:
func CSVOutput(w io.Writer, data interfac{}, cols []string) error {
sliceOfIntf = castToSlice(data) //how to do this?
if !implementedCSV(sliceOfIntf[0]) { //and how to do this?
return errors.New("not csv")
}
... ...
}

is it possible to rewrite the CSVOutput() function by using reflection
Yes
// if data is []Employee{...}, then you can do the following:
rv := reflect.ValueOf(data)
if rv.Kind() != reflect.Slice {
return fmt.Errorf("data is not slice")
}
if !rv.Type().Elem().Implements(reflect.TypeOf((*CSV)(nil)).Elem()) {
return fmt.Errorf("slice element does not implement CSV")
}
csvArr := make(CSVArray, rv.Len())
for i := 0; i < rv.Len(); i++ {
csvArr[i] = rv.Index(i).Interface().(CSV)
}
// now csvArr is CSVArray containing all the elements of data
https://go.dev/play/p/gcSOid533gx

Related

Go: Implementing a ManyDecode for a "set" of individual results

I have implemented a very simple Decode method (using gob.Decoder for now) - this works well for single responses - it would even work well for slices, but I need to implement a DecodeMany method where it is able to decode a set of individual responses (not a slice).
Working Decode method:
var v MyType
_ = Decode(&v)
...
func Decode(v interface{}) error {
buf, _ := DoSomething() // func DoSomething() ([]byte, error)
// error handling omitted for brevity
return gob.NewDecoder(bytes.NewReader(buf)).Decode(v)
}
What I'm trying to do for a DecodeMany method is to deal with a response that isn't necessarily a slice:
var vv []MyType
_ = DecodeMany(&vv)
...
func DecodeMany(vv []interface{}) error {
for _, g := range DoSomething() { // func DoSomething() []struct{Buf []bytes}
// Use g.Buf as an individual "interface{}"
// want something like:
var v interface{} /* Somehow create instance of single vv type? */
_ = gob.NewDecoder(bytes.NewReader(g.Buf)).Decode(v)
vv = append(vv, v)
}
return
}
Besides not compiling the above also has the error of:
cannot use &vv (value of type *[]MyType) as type []interface{} in argument to DecodeMany
If you want to modify the passed slice, it must be a pointer, else you must return a new slice. Also if the function is declared to have a param of type []interface{}, you can only pass a value of type []interface{} and no other slice types... Unless you use generics...
This is a perfect example to start using generics introduced in Go 1.18.
Change DecodeMany() to be generic, having a T type parameter being the slice element type:
When taking a pointer
func DecodeMany[T any](vv *[]T) error {
for _, g := range DoSomething() {
var v T
if err := gob.NewDecoder(bytes.NewReader(g.Buf)).Decode(&v); err != nil {
return err
}
*vv = append(*vv, v)
}
return nil
}
Here's a simple app to test it:
type MyType struct {
S int64
}
func main() {
var vv []MyType
if err := DecodeMany(&vv); err != nil {
panic(err)
}
fmt.Println(vv)
}
func DoSomething() (result []struct{ Buf []byte }) {
for i := 3; i < 6; i++ {
buf := &bytes.Buffer{}
v := MyType{S: int64(i)}
if err := gob.NewEncoder(buf).Encode(v); err != nil {
panic(err)
}
result = append(result, struct{ Buf []byte }{buf.Bytes()})
}
return
}
This outputs (try it on the Go Playground):
[{3} {4} {5}]
When returning a slice
If you choose to return the slice, you don't have to pass anything, but you need to assign the result:
func DecodeMany[T any]() ([]T, error) {
var result []T
for _, g := range DoSomething() {
var v T
if err := gob.NewDecoder(bytes.NewReader(g.Buf)).Decode(&v); err != nil {
return result, err
}
result = append(result, v)
}
return result, nil
}
Using it:
vv, err := DecodeMany[MyType]()
if err != nil {
panic(err)
}
fmt.Println(vv)
Try this one on the Go Playground.

How to set slice interface values with reflection

I would like to build a function that takes a generic pointer array and fill that list based on mongo results.
I don't know how to set the value I got from mongo into my pointer array. In the below attempt, program panics with following error : reflect.Set: value of type []interface {} is not assignable to type []Person
When I print total / documents found, it corresponds to what I am expecting. So I think question is about reflection.
func getListWithCount(ctx context.Context, receiver interface{}) (int, error) {
//my mongo query here
var mongoResp struct {
Total int `bson:"total"`
Documents interface{} `bson:"documents"`
}
if err := cursor.Decode(&mongoResp); err != nil {
return 0, err
}
receiverValue := reflect.ValueOf(receiver)
docs := []interface{}(mongoResp.Documents.(primitive.A))
receiverValue.Elem().Set(reflect.ValueOf(docs))
return mongoResp.Total, nil
}
type Person struct {
Name string `bson:"name"`
}
func main() {
var persons []Person
count, err := getListWithCount(context.Background(), &persons)
if err != nil {
log.Fatal(err)
}
fmt.Println(count)
fmt.Println(persons)
}
You should be able to decode first into bson.RawValue and then Unmarshal it into the receiver.
func getListWithCount(ctx context.Context, receiver interface{}) (int, error) {
//my mongo query here
var mongoResp struct {
Total int `bson:"total"`
Documents bson.RawValue `bson:"documents"`
}
if err := cursor.Decode(&mongoResp); err != nil {
return 0, err
}
if err := mongoResp.Documents.Unmarshal(receiver); err != nil {
return 0, err
}
return mongoResp.Total, nil
}
You can also implement it as a custom bson.Unmarshaler.
type MongoResp struct {
Total int `bson:"total"`
Documents interface{} `bson:"documents"`
}
func (r *MongoResp) UnmarshalBSON(data []byte) error {
var temp struct {
Total int `bson:"total"`
Documents bson.RawValue `bson:"documents"`
}
if err := bson.Unmarshal(data, &temp); err != nil {
return err
}
r.Total = temp.Total
return temp.Documents.Unmarshal(r.Documents)
}
With that you would use it in the function like so:
func getListWithCount(ctx context.Context, receiver interface{}) (int, error) {
//my mongo query here
mongoResp := MongoResp{Documents: receiver}
if err := cursor.Decode(&mongoResp); err != nil {
return 0, err
}
return mongoResp.Total, nil
}
Dynamically create a struct type that matches the queried document. See commentary below for details.
func getListWithCount(receiver interface{}) (int, error) {
dst := reflect.ValueOf(receiver).Elem()
// Your mongo query here
// Create a struct type that matches the document.
doct := reflect.StructOf([]reflect.StructField{
reflect.StructField{Name: "Total", Type: reflect.TypeOf(0), Tag: `bson:"total"`},
reflect.StructField{Name: "Documents", Type: dst.Type(), Tag: `bson:"documents"`},
})
// Decode to a value of the type.
docp := reflect.New(doct)
if err := cursor.Decode(docp.Interface()); err != nil {
return 0, err
}
docv := docp.Elem()
// Copy the Documents field to *receiver.
dst.Set(docv.Field(1))
// Return the total
return docv.Field(0).Interface().(int), nil
}
there is no need to use reflect here, you can decode it directly to your Person slices
func getPersons(ctx context.Context, coll *mongo.Collection, results interface{}) error {
cur, err := coll.Find(ctx, bson.D{})
if err != nil {
return err
}
err = cur.All(ctx, results)
if err != nil {
return err
}
return nil
}
and the len is the count of the results.
err = getPersons(ctx, coll, &persons)
require.NoError(t, err)
t.Logf("Got %d persons: %v", len(persons), persons)
see https://gist.github.com/xingyongtao/459f92490bdcbf7d5afe9f5d1ae6c04a

How to access map[string]interface {}?

How can i programmatically access the msg value "Design" in the go language structure shown below?
after subIssues[28].Fields.Unknowns["customfield_11801"] i dont find a language construct to access the data structure.
To convert into a struct and work with that, have a look at this repo: https://github.com/mitchellh/mapstructure.
And if you want to do it your self, something like this: (Adjust implementation to match your project)
func SetField(obj interface{}, name string, value interface{}) error {
structValue := reflect.ValueOf(obj).Elem()
structFieldValue := structValue.FieldByName(name)
if !structFieldValue.IsValid() {
return fmt.Errorf("No such field: %s in obj", name)
}
if !structFieldValue.CanSet() {
return fmt.Errorf("Cannot set %s field value", name)
}
structFieldType := structFieldValue.Type()
val := reflect.ValueOf(value)
if structFieldType != val.Type() {
return errors.New("Provided value type didn't match obj field type")
}
structFieldValue.Set(val)
return nil
}
type MyStruct struct {
Name string
Age int64
}
func (s *MyStruct) FillStruct(m map[string]interface{}) error {
for k, v := range m {
err := SetField(s, k, v)
if err != nil {
return err
}
}
return nil
}
func main() {
myData := make(map[string]interface{})
myData["Name"] = "Tony"
myData["Age"] = int64(23)
result := &MyStruct{}
err := result.FillStruct(myData)
if err != nil {
fmt.Println(err)
}
fmt.Println(result)
}

How to extend go-yaml to support custom tags

I have spent some time reading the code and docs of go-yaml, but I have not found any way to do this, except forking the project..
I want to extend the YAML unmarshaller so that it can accept a custom YAML tag (!include <file> in this case), which in turn would allow me to add support for including files. This is easily implemented with other YAML libraries, like in this answer.
Is there any way to accomplish this, using the public interface of the library (or another yaml library)?
Yes, this is possible (since v3). You can load the whole YAML file into a yaml.Node and then walk over the structure. The trick is that yaml.Node is an intermediate representation which you can only access if you define an unmarshaler.
For example:
package main
import (
"errors"
"fmt"
"io/ioutil"
"gopkg.in/yaml.v3"
)
// used for loading included files
type Fragment struct {
content *yaml.Node
}
func (f *Fragment) UnmarshalYAML(value *yaml.Node) error {
var err error
// process includes in fragments
f.content, err = resolveIncludes(value)
return err
}
type IncludeProcessor struct {
target interface{}
}
func (i *IncludeProcessor) UnmarshalYAML(value *yaml.Node) error {
resolved, err := resolveIncludes(value)
if err != nil {
return err
}
return resolved.Decode(i.target)
}
func resolveIncludes(node *yaml.Node) (*yaml.Node, error) {
if node.Tag == "!include" {
if node.Kind != yaml.ScalarNode {
return nil, errors.New("!include on a non-scalar node")
}
file, err := ioutil.ReadFile(node.Value)
if err != nil {
return nil, err
}
var f Fragment
err = yaml.Unmarshal(file, &f)
return f.content, err
}
if node.Kind == yaml.SequenceNode || node.Kind == yaml.MappingNode {
var err error
for i := range node.Content {
node.Content[i], err = resolveIncludes(node.Content[i])
if err != nil {
return nil, err
}
}
}
return node, nil
}
type MyStructure struct {
// this structure holds the values you want to load after processing
// includes, e.g.
Num int
}
func main() {
var s MyStructure
yaml.Unmarshal([]byte("!include foo.yaml"), &IncludeProcessor{&s})
fmt.Printf("Num: %v", s.Num)
}
Code prints Num: 42 when a file foo.yaml exists with the content num: 42.
Modified #flyx's original code a little to make it modular for adding custom resolvers.
package main
import (
"errors"
"fmt"
"io/ioutil"
"os"
"gopkg.in/yaml.v3"
)
var tagResolvers = make(map[string]func(*yaml.Node) (*yaml.Node, error))
type Fragment struct {
content *yaml.Node
}
func (f *Fragment) UnmarshalYAML(value *yaml.Node) error {
var err error
// process includes in fragments
f.content, err = resolveTags(value)
return err
}
type CustomTagProcessor struct {
target interface{}
}
func (i *CustomTagProcessor) UnmarshalYAML(value *yaml.Node) error {
resolved, err := resolveTags(value)
if err != nil {
return err
}
return resolved.Decode(i.target)
}
func resolveTags(node *yaml.Node) (*yaml.Node, error) {
for tag, fn := range tagResolvers {
if node.Tag == tag {
return fn(node)
}
}
if node.Kind == yaml.SequenceNode || node.Kind == yaml.MappingNode {
var err error
for i := range node.Content {
node.Content[i], err = resolveTags(node.Content[i])
if err != nil {
return nil, err
}
}
}
return node, nil
}
func resolveIncludes(node *yaml.Node) (*yaml.Node, error) {
if node.Kind != yaml.ScalarNode {
return nil, errors.New("!include on a non-scalar node")
}
file, err := ioutil.ReadFile(node.Value)
if err != nil {
return nil, err
}
var f Fragment
err = yaml.Unmarshal(file, &f)
return f.content, err
}
func resolveGetValueFromEnv(node *yaml.Node) (*yaml.Node, error) {
if node.Kind != yaml.ScalarNode {
return nil, errors.New("!getValueFromEnv on a non-scalar node")
}
value := os.Getenv(node.Value)
if value == "" {
return nil, fmt.Errorf("environment variable %v not set", node.Value)
}
var f Fragment
err := yaml.Unmarshal([]byte(value), &f)
return f.content, err
}
func AddResolvers(tag string, fn func(*yaml.Node) (*yaml.Node, error)) {
tagResolvers[tag] = fn
}
func main() {
// Register custom tag resolvers
AddResolvers("!include", resolveIncludes)
AddResolvers("!getValueFromEnv", resolveGetValueFromEnv)
type MyStructure struct {
// this structure holds the values you want to load after processing
// includes, e.g.
Num int
}
var s MyStructure
os.Setenv("FOO", `{"num": 42}`)
err := yaml.Unmarshal([]byte("!getValueFromEnv FOO"), &CustomTagProcessor{&s})
if err != nil {
panic("Error encountered during unmarshalling")
}
fmt.Printf("\nNum: %v", s.Num)
err = yaml.Unmarshal([]byte("!include foo.yaml"), &CustomTagProcessor{&s})
if err != nil {
panic("Error encountered during unmarshalling")
}
fmt.Printf("\nNum: %v", s.Num)
}

Using sqlx.Rows.StructScan for interface args

I need to use StructScan function for interface (pointer to struct).
But if I try to reflect value, I've got error, because reflect.New() returning reflect.Value type. How I can scan structure and store data into dest var?
// package 1
type Data struct {
id int `db:"id"`
caption string `db:"caption"`
}
func Func1 {
data := []Data{}
GetData(&data)
log.Println(data)
}
// package 2
func GetData(sql string, dest interface{}) {
rows, err := DBI.Queryx(sql)
if err == nil {
// reflect.Value
myData := reflect.New(reflect.TypeOf(dest).Elem().Elem())
for rows.Next() {
rows.StructScan(&myData) // Fail here
}
}
}
Solved
// package 2
func GetData(sql string, dest interface{}) {
arr := reflect.ValueOf(dest).Elem()
v := reflect.New(reflect.TypeOf(dest).Elem().Elem())
rows, err := DBI.Queryx(sql)
if err == nil {
if err = rows.StructScan(v.Interface()); err == nil {
arr.Set(reflect.Append(arr, v.Elem()))
}
}
}

Resources