I wanted to loop through a struct and modify fields value using reflection. How can I Set it?
func main() {
x := struct {
Foo string
Bar int
}{"foo", 2}
StructCheck(Checker, x)
}
func Checker(s interface{}) interface{} {
log.Println(s)
return s
}
func StructCheck(check func(interface{}) interface{}, x interface{}) interface{} {
v := reflect.ValueOf(x)
for i := 0; i < v.NumField(); i++ {
r := check(v.Field(i))
w := reflect.ValueOf(&r).Elem()
log.Println(w.Type(), w.CanSet())
// v.Field(i).Set(reflect.ValueOf(w))
}
return v
}
Running Set() causes panic and shows :reflect.Value.Set using unaddressable value
You must pass an addressable value to the function.
StructCheck(Checker, &x)
There's
Dereference the value in the StructCheck:
v := reflect.ValueOf(x).Elem() // Elem() gets value of ptr
There were some other issues. Here's the updated code:
func StructCheck(check func(interface{}) interface{}, x interface{}) {
v := reflect.ValueOf(x).Elem()
for i := 0; i < v.NumField(); i++ {
r := check(v.Field(i).Interface())
v.Field(i).Set(reflect.ValueOf(r))
}
}
Run it on the Playground.
Related
I want to take data from DB and write to excel
let's say I have a struct like:
type user struct {
ID int64
Name string
Age int
}
I can get a pointer to slice of user type form DB &[]user{}
but I want to convert that slice to a 2D slice of string [][]string{}
and here's my code try to do such job:
func toStrings(slice interface{}) [][]string {
switch reflect.TypeOf(slice).Elem().Kind() {
case reflect.Slice:
ret := [][]string{}
val := reflect.ValueOf(slice).Elem()
for i := 0; i < val.Len(); i++ {
tempSlice := []string{}
tempV := reflect.ValueOf(val.Index(i))
for j := 0; j < tempV.NumField(); j++ {
tempSlice = append(tempSlice, tempV.Field(j).String())
}
ret = append(ret, tempSlice)
}
return ret
}
return nil
}
But from the code above all I get is a slice like [<*reflect.rtype Value> <unsafe.Pointer Value> <reflect.flag Value>]
where I do it wrong?
my codes in golang playground
sorry, I found where I do it wrong, I got tempV wrong
func toStrings(slice interface{}) [][]string {
switch reflect.TypeOf(slice).Elem().Kind() {
case reflect.Slice:
ret := [][]string{}
val := reflect.ValueOf(slice).Elem()
for i := 0; i < val.Len(); i++ {
tempSlice := []string{}
// tempV should be:
tempV := val.Index(i)
// instead of reflect.ValueOf(val.Index(i))
for j := 0; j < tempV.NumField(); j++ {
tempSlice = append(tempSlice, tempV.Field(j).String())
}
ret = append(ret, tempSlice)
}
return ret
}
return nil
}
There are two problems in the code in the question. The first problem is the slice element is doubled wrapped by a a reflect.Value in the expression reflect.Value(val.Index(i)). Fix by removing the extra call to reflect.Value. The second problem is that the reflect.Value String method does not convert the underlying value to its string representation. Use fmt.Sprint (or one of its friends) to do that.
Try this:
func toStrings(slice interface{}) [][]string {
// Get reflect value for slice. Use Indirect to
// handle slice argument and pointer to slice
// argument.
v := reflect.Indirect(reflect.ValueOf(slice))
if v.Kind() != reflect.Slice {
return nil
}
var result [][]string
// For each element...
for i := 0; i < v.Len(); i++ {
// Get reflect value for slice element (a struct). Use
// Indirect to handle slice of struct and slice of
// pointer to struct.
e := reflect.Indirect(v.Index(i))
if e.Kind() != reflect.Struct {
return nil
}
// Convert fields to string and append.
var element []string
for i := 0; i < e.NumField(); i++ {
// Use fmt.Sprint to convert arbitrary Go value
// to a string.
element = append(element, fmt.Sprint(e.Field(i).Interface()))
}
result = append(result, element)
}
return result
}
Run it on the playground.
Maybe I have a simple way to resolve the problem, golang playground here
I used encoding/json to convert to json data, then convert it to map[string]interface{}.
func toStrings2(slice interface{}) [][]string {
jsonData, _ := json.Marshal(slice)
var out []map[string]interface{}
_ = json.Unmarshal(jsonData, &out)
var fields []string
if len(out) > 0 {
for k := range out[0] {
fields = append(fields, k)
}
}
var ret [][]string
for _, row := range out {
var r []string
for _, k := range fields {
r = append(r, fmt.Sprint(row[k]))
}
ret = append(ret, r)
}
return ret
}
Notice:
With the help of #CeriseLimón, I known that the code in this answer can't handle large values for User.ID.
I'm creating a utility package for my project.
Many of my string slices need a function to
1. remove duplicates
2. remove empty strings
I know 1 way to do this:1. Add a function for each case which accepts a string slice and returns a string slice
func removeEmpty(s []string) []string {
i := 0 // i points to next available pos
for _, v := range s {
if v != "" {
s[i] = v
i++
}
}
return s[:i]
}
func dedup(s []string) []string {
i := 0 // i points to next available pos
mp := map[string]bool{}
for _, v := range s {
if _, ok := mp[v]; !ok {
s[i] = v
mp[v] = true
i++
}
}
return s[:i]
}
when I apply these 2 functions to my slice, I can do:
mySlice := string[]{}
mySlice = dedup(removeEmpty(mySlice))
I want to make it somthing like:
mySlice = mySlice.dedup().removeEmpty()
or
mySlice.dedup().removeEmpty()
Maybe I can add custom method to slice[] ?
I tried writing it
func (s []string) removeEmpty() {
}
I'm getting complie error.
You can't define methods on []string, but you can define your own type based on []string and define methods on that:
type myStringSlice []string
func (s myStringSlice) removeEmpty() myStringSlice {
i := 0 // i points to next available pos
for _, v := range s {
if v != "" {
s[i] = v
i++
}
}
return s[:i]
}
func (s myStringSlice) dedup() myStringSlice {
i := 0 // i points to next available pos
mp := map[string]bool{}
for _, v := range s {
if _, ok := mp[v]; !ok {
s[i] = v
mp[v] = true
i++
}
}
return s[:i]
}
See https://play.golang.org/p/u1z_N3c_wPP.
As mentioned in the documentation:
You can only declare a method with a receiver whose type is defined in the same package as the method. You cannot declare a method with a receiver whose type is defined in another package.
So you have to declare a type on []string and then declare the method on your own type.
I'm curious why Go does't implicitly convert []T to []interface{} when it will implicitly convert T to interface{}. Is there something non-trivial about this conversion that I'm missing?
Example:
func foo([]interface{}) { /* do something */ }
func main() {
var a []string = []string{"hello", "world"}
foo(a)
}
go build complains
cannot use a (type []string) as type []interface {} in function argument
And if I try to do it explicitly, same thing: b := []interface{}(a) complains
cannot convert a (type []string) to type []interface {}
So every time I need to do this conversion (which seems to come up a lot), I've been doing something like this:
b = make([]interface{}, len(a), len(a))
for i := range a {
b[i] = a[i]
}
Is there a better way to do this, or standard library functions to help with these conversions? It seems kind of silly to write 4 extra lines of code every time I want to call a function that can take a list of e.g. ints or strings.
In Go, there is a general rule that syntax should not hide complex/costly operations.
Converting a string to an interface{} is done in O(1) time. Converting a []string to an interface{} is also done in O(1) time since a slice is still one value. However, converting a []string to an []interface{} is O(n) time because each element of the slice must be converted to an interface{}.
The one exception to this rule is converting strings. When converting a string to and from a []byte or a []rune, Go does O(n) work even though conversions are "syntax".
There is no standard library function that will do this conversion for you. Your best option though is just to use the lines of code you gave in your question:
b := make([]interface{}, len(a))
for i := range a {
b[i] = a[i]
}
Otherwise, you could make one with reflect, but it would be slower than the three line option. Example with reflection:
func InterfaceSlice(slice interface{}) []interface{} {
s := reflect.ValueOf(slice)
if s.Kind() != reflect.Slice {
panic("InterfaceSlice() given a non-slice type")
}
// Keep the distinction between nil and empty slice input
if s.IsNil() {
return nil
}
ret := make([]interface{}, s.Len())
for i:=0; i<s.Len(); i++ {
ret[i] = s.Index(i).Interface()
}
return ret
}
The thing you are missing is that T and interface{} which holds a value of T have different representations in memory so can't be trivially converted.
A variable of type T is just its value in memory. There is no associated type information (in Go every variable has a single type known at compile time not at run time). It is represented in memory like this:
value
An interface{} holding a variable of type T is represented in memory like this
pointer to type T
value
So coming back to your original question: why go does't implicitly convert []T to []interface{}?
Converting []T to []interface{} would involve creating a new slice of interface {} values which is a non-trivial operation since the in-memory layout is completely different.
Here is the official explanation: https://github.com/golang/go/wiki/InterfaceSlice
var dataSlice []int = foo()
var interfaceSlice []interface{} = make([]interface{}, len(dataSlice))
for i, d := range dataSlice {
interfaceSlice[i] = d
}
In Go 1.18 or later, use the following function to convert an arbitrary slice type to []interface{} or its alias any:
func ToSliceOfAny[T any](s []T) []any {
result := make([]any, len(s))
for i, v := range s {
result[i] = v
}
return result
}
The Go 1.18 generics feature does not eliminate the need to convert an arbitrary slice to []any. Here's an example of where the conversion is required: The application wants to query a database using the elements of a []string as the variadic query arguments declared as args ...any. The function in this answer allows the application to query the database in a convenient one-liner:
rows, err := db.Query(qs, ToSliceOfAny(stringArgs)...)
Try interface{} instead. To cast back as slice, try
func foo(bar interface{}) {
s := bar.([]string)
// ...
}
In case you need more shorting your code, you can creating new type for helper
type Strings []string
func (ss Strings) ToInterfaceSlice() []interface{} {
iface := make([]interface{}, len(ss))
for i := range ss {
iface[i] = ss[i]
}
return iface
}
then
a := []strings{"a", "b", "c", "d"}
sliceIFace := Strings(a).ToInterfaceSlice()
I was curious how much slower it is convert interface arrays via reflection vs. doing it inside a loop, as described in Stephen's answer. Here's a benchmark comparison of the two approaches:
benchmark iter time/iter bytes alloc allocs
--------- ---- --------- ----------- ------
BenchmarkLoopConversion-12 2285820 522.30 ns/op 400 B/op 11 allocs/op
BenchmarkReflectionConversion-12 1780002 669.00 ns/op 584 B/op 13 allocs/op
So using a loop is ~20% faster than doing it via reflection.
Here's my test code in case you'd like to verify if I did things correctly:
import (
"math/rand"
"reflect"
"testing"
"time"
)
func InterfaceSlice(slice interface{}) []interface{} {
s := reflect.ValueOf(slice)
if s.Kind() != reflect.Slice {
panic("InterfaceSlice() given a non-slice type")
}
// Keep the distinction between nil and empty slice input
if s.IsNil() {
return nil
}
ret := make([]interface{}, s.Len())
for i := 0; i < s.Len(); i++ {
ret[i] = s.Index(i).Interface()
}
return ret
}
type TestStruct struct {
name string
age int
}
var letters = []rune("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ")
func randSeq(n int) string {
b := make([]rune, n)
for i := range b {
b[i] = letters[rand.Intn(len(letters))]
}
return string(b)
}
func randTestStruct(lenArray int, lenMap int) map[int][]TestStruct {
randomStructMap := make(map[int][]TestStruct, lenMap)
for i := 0; i < lenMap; i++ {
var testStructs = make([]TestStruct, 0)
for k := 0; k < lenArray; k++ {
rand.Seed(time.Now().UnixNano())
randomString := randSeq(10)
randomInt := rand.Intn(100)
testStructs = append(testStructs, TestStruct{name: randomString, age: randomInt})
}
randomStructMap[i] = testStructs
}
return randomStructMap
}
func BenchmarkLoopConversion(b *testing.B) {
var testStructMap = randTestStruct(10, 100)
b.ResetTimer()
for i := 0; i < b.N; i++ {
obj := make([]interface{}, len(testStructMap[i%100]))
for k := range testStructMap[i%100] {
obj[k] = testStructMap[i%100][k]
}
}
}
func BenchmarkReflectionConversion(b *testing.B) {
var testStructMap = randTestStruct(10, 100)
b.ResetTimer()
for i := 0; i < b.N; i++ {
obj := make([]interface{}, len(testStructMap[i%100]))
obj = InterfaceSlice(testStructMap[i%100])
_ = obj
}
}
Though you can use a generic function to convert a slice to a slice of interface{}, it may be most appropriate and cheapest in terms of execution time to change foo to a generic function if possible.
For example:
func foo[T any](slice []T) { /* do something */ }
func main() {
var a []string = []string{"hello", "world"}
foo(a)
}
Now there is no conversion necessary at all.
Convert interface{} into any type.
Syntax:
result := interface.(datatype)
Example:
var employee interface{} = []string{"Jhon", "Arya"}
result := employee.([]string) //result type is []string.
I have the following code to double the slice.
func doubleSlice(s []int) []int {
t := make([]int, len(s), (cap(s) + 1) * 2 )
for i := range s {
t[i] = s[i]
}
return t
}
I want to make the func to double any type of slice. And I need to know the element type first.
func showInterfaceItem(s interface{}) interface{} {
if reflect.TypeOf(s).Kind() != reflect.Slice {
fmt.Println("The interface is not a slice.")
return
}
var t interface{}
newLen := reflect.ValueOf(s).Len()
newCap := (cap(reflect.ValueOf(s).Cap()) + 1) * 2
t = make([]reflect.TypeOf(s), newLen, newCap)
return t
}
The reflect.TypeOf(s) return the type of interface{}, not the type of element. How can I get the element type of slice interface?
You can use reflect.TypeOf(s).Elem()
to get the type of element of slice.
package main
import (
"fmt"
"reflect"
)
func doubleSlice(s interface{}) interface{} {
if reflect.TypeOf(s).Kind() != reflect.Slice {
fmt.Println("The interface is not a slice.")
return nil
}
v := reflect.ValueOf(s)
newLen := v.Len()
newCap := (v.Cap() + 1) * 2
typ := reflect.TypeOf(s).Elem()
t := reflect.MakeSlice(reflect.SliceOf(typ), newLen, newCap)
reflect.Copy(t, v)
return t.Interface()
}
func main() {
xs := doubleSlice([]string{"foo", "bar"}).([]string)
fmt.Println("data =", xs, "len =", len(xs), "cap =", cap(xs))
ys := doubleSlice([]int{3, 1, 4}).([]int)
fmt.Println("data =", ys, "len =", len(ys), "cap =", cap(ys))
}
The output will be:
data = [foo bar] len = 2 cap = 6
data = [3 1 4] len = 3 cap = 8
Check it in: Go Playground
This is doable in golang and takes me whole day to discover the pattern.
Firstly, we want to get a pointer of slice to make gorm happy, which is has type "*[]Obj". To achieve that in golang, we can create a make wrapper like so:
func makeWrapper(cap uint) interface{} {
arr:= make([]Sth, 0, cap)
return &arr
}
Notice that, we can't directly reference the maked value, which might be the book keeping data need to have a stack space to store.
//Not working example
func makeWrapper(cap uint) interface{} {
return &(make([]Sth, 0, cap))
}
And as the answer before, the reflect.MakeSlice(reflect.SliceOf(typ), 0, capacity).Interface() returns interface{[]Sth}. (the typ here is refer to reflect.TypeOf(Sth{}), which equiv to typ == reflect.TypeOf(v))
Thus we need to create a return object of *[]Sth and the value inside is a slice []Sth with capacity. After understanding the objective, we can have this code:
package main
import (
"reflect"
)
type Sth struct {
a, b string
}
func main() {
af:= createSlice(Sth{})
arr := makeWrapper(10).(*[]Sth)
println(reflect.TypeOf(arr).String())
// equiv to makeWrapper, but we do it via reflection
arr = af(10).(*[]Sth)
println(reflect.TypeOf(arr).String())
}
func makeWrapper(cap uint) interface{} {
arr:= make([]Sth, 0, cap)
return &arr
}
func createSlice(v interface{}) func(int) interface{} {
var typ reflect.Type
if reflect.ValueOf(v).Kind() == reflect.Ptr {
typ = reflect.ValueOf(v).Elem().Type()
} else if reflect.ValueOf(v).Kind() == reflect.Struct {
typ = reflect.TypeOf(v)
} else {
panic("only support instance of struct or pointer of that instance")
}
return func(capacity int) interface{}{
// create the outer object saves our slice
outerObj:=reflect.New(reflect.SliceOf(typ))
// create the slice and save it to return
outerObj.Elem().Set(reflect.MakeSlice(reflect.SliceOf(typ), 0, capacity))
// retrive the interface of outer object
return outerObj.Interface()
}
}
I'm curious why Go does't implicitly convert []T to []interface{} when it will implicitly convert T to interface{}. Is there something non-trivial about this conversion that I'm missing?
Example:
func foo([]interface{}) { /* do something */ }
func main() {
var a []string = []string{"hello", "world"}
foo(a)
}
go build complains
cannot use a (type []string) as type []interface {} in function argument
And if I try to do it explicitly, same thing: b := []interface{}(a) complains
cannot convert a (type []string) to type []interface {}
So every time I need to do this conversion (which seems to come up a lot), I've been doing something like this:
b = make([]interface{}, len(a), len(a))
for i := range a {
b[i] = a[i]
}
Is there a better way to do this, or standard library functions to help with these conversions? It seems kind of silly to write 4 extra lines of code every time I want to call a function that can take a list of e.g. ints or strings.
In Go, there is a general rule that syntax should not hide complex/costly operations.
Converting a string to an interface{} is done in O(1) time. Converting a []string to an interface{} is also done in O(1) time since a slice is still one value. However, converting a []string to an []interface{} is O(n) time because each element of the slice must be converted to an interface{}.
The one exception to this rule is converting strings. When converting a string to and from a []byte or a []rune, Go does O(n) work even though conversions are "syntax".
There is no standard library function that will do this conversion for you. Your best option though is just to use the lines of code you gave in your question:
b := make([]interface{}, len(a))
for i := range a {
b[i] = a[i]
}
Otherwise, you could make one with reflect, but it would be slower than the three line option. Example with reflection:
func InterfaceSlice(slice interface{}) []interface{} {
s := reflect.ValueOf(slice)
if s.Kind() != reflect.Slice {
panic("InterfaceSlice() given a non-slice type")
}
// Keep the distinction between nil and empty slice input
if s.IsNil() {
return nil
}
ret := make([]interface{}, s.Len())
for i:=0; i<s.Len(); i++ {
ret[i] = s.Index(i).Interface()
}
return ret
}
The thing you are missing is that T and interface{} which holds a value of T have different representations in memory so can't be trivially converted.
A variable of type T is just its value in memory. There is no associated type information (in Go every variable has a single type known at compile time not at run time). It is represented in memory like this:
value
An interface{} holding a variable of type T is represented in memory like this
pointer to type T
value
So coming back to your original question: why go does't implicitly convert []T to []interface{}?
Converting []T to []interface{} would involve creating a new slice of interface {} values which is a non-trivial operation since the in-memory layout is completely different.
Here is the official explanation: https://github.com/golang/go/wiki/InterfaceSlice
var dataSlice []int = foo()
var interfaceSlice []interface{} = make([]interface{}, len(dataSlice))
for i, d := range dataSlice {
interfaceSlice[i] = d
}
In Go 1.18 or later, use the following function to convert an arbitrary slice type to []interface{} or its alias any:
func ToSliceOfAny[T any](s []T) []any {
result := make([]any, len(s))
for i, v := range s {
result[i] = v
}
return result
}
The Go 1.18 generics feature does not eliminate the need to convert an arbitrary slice to []any. Here's an example of where the conversion is required: The application wants to query a database using the elements of a []string as the variadic query arguments declared as args ...any. The function in this answer allows the application to query the database in a convenient one-liner:
rows, err := db.Query(qs, ToSliceOfAny(stringArgs)...)
Try interface{} instead. To cast back as slice, try
func foo(bar interface{}) {
s := bar.([]string)
// ...
}
In case you need more shorting your code, you can creating new type for helper
type Strings []string
func (ss Strings) ToInterfaceSlice() []interface{} {
iface := make([]interface{}, len(ss))
for i := range ss {
iface[i] = ss[i]
}
return iface
}
then
a := []strings{"a", "b", "c", "d"}
sliceIFace := Strings(a).ToInterfaceSlice()
I was curious how much slower it is convert interface arrays via reflection vs. doing it inside a loop, as described in Stephen's answer. Here's a benchmark comparison of the two approaches:
benchmark iter time/iter bytes alloc allocs
--------- ---- --------- ----------- ------
BenchmarkLoopConversion-12 2285820 522.30 ns/op 400 B/op 11 allocs/op
BenchmarkReflectionConversion-12 1780002 669.00 ns/op 584 B/op 13 allocs/op
So using a loop is ~20% faster than doing it via reflection.
Here's my test code in case you'd like to verify if I did things correctly:
import (
"math/rand"
"reflect"
"testing"
"time"
)
func InterfaceSlice(slice interface{}) []interface{} {
s := reflect.ValueOf(slice)
if s.Kind() != reflect.Slice {
panic("InterfaceSlice() given a non-slice type")
}
// Keep the distinction between nil and empty slice input
if s.IsNil() {
return nil
}
ret := make([]interface{}, s.Len())
for i := 0; i < s.Len(); i++ {
ret[i] = s.Index(i).Interface()
}
return ret
}
type TestStruct struct {
name string
age int
}
var letters = []rune("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ")
func randSeq(n int) string {
b := make([]rune, n)
for i := range b {
b[i] = letters[rand.Intn(len(letters))]
}
return string(b)
}
func randTestStruct(lenArray int, lenMap int) map[int][]TestStruct {
randomStructMap := make(map[int][]TestStruct, lenMap)
for i := 0; i < lenMap; i++ {
var testStructs = make([]TestStruct, 0)
for k := 0; k < lenArray; k++ {
rand.Seed(time.Now().UnixNano())
randomString := randSeq(10)
randomInt := rand.Intn(100)
testStructs = append(testStructs, TestStruct{name: randomString, age: randomInt})
}
randomStructMap[i] = testStructs
}
return randomStructMap
}
func BenchmarkLoopConversion(b *testing.B) {
var testStructMap = randTestStruct(10, 100)
b.ResetTimer()
for i := 0; i < b.N; i++ {
obj := make([]interface{}, len(testStructMap[i%100]))
for k := range testStructMap[i%100] {
obj[k] = testStructMap[i%100][k]
}
}
}
func BenchmarkReflectionConversion(b *testing.B) {
var testStructMap = randTestStruct(10, 100)
b.ResetTimer()
for i := 0; i < b.N; i++ {
obj := make([]interface{}, len(testStructMap[i%100]))
obj = InterfaceSlice(testStructMap[i%100])
_ = obj
}
}
Though you can use a generic function to convert a slice to a slice of interface{}, it may be most appropriate and cheapest in terms of execution time to change foo to a generic function if possible.
For example:
func foo[T any](slice []T) { /* do something */ }
func main() {
var a []string = []string{"hello", "world"}
foo(a)
}
Now there is no conversion necessary at all.
Convert interface{} into any type.
Syntax:
result := interface.(datatype)
Example:
var employee interface{} = []string{"Jhon", "Arya"}
result := employee.([]string) //result type is []string.