I'm trying to achieve the following:
Use-case:
I have three structures, I need to compare 2 of those against one. (in the example described as: a & b need to be compared against full)
Reflection is used to loop over every field, retrieve the name of the field. And comparing the difference between a & full, b & full, storing the results in a shared structure.
If the field equals World, we know it's a slice struct:
I need to retrieve the first index of the Bar slice within the Foo structure.
Even though the variable is a slice, I know it will always have a length of 1 in this use-case.
When retrieved I need to loop over those fields, like what is happening in the previous if statement.
Example code:
type Foo struct {
Hello string
World []Bar
}
type Bar struct {
Fish string
}
type Result struct {
Field string
Correct_A bool
Distance_A int
Correct_B bool
Distance_B int
Result []Result
}
func compare_structs() {
var full, a, b Foo
// filling in all variables...
result := []Result{}
rfx_f := reflect.ValueOf(full)
rfx_a := reflect.ValueOf(a)
rfx_b := reflect.ValueOf(b)
type_result := rfx_f.Type()
for i := 0; i < rfx_f.NumField(); i++ {
tmp_res := Result{
Field: type_result.Field(i).Name,
}
if reflect.TypeOf(full).Field(i).Type.Kind() != reflect.Slice {
value := rfx_f.Field(i).Interface()
value_a := rfx_a.FieldByName(tmp_res.Field).Interface()
value_b := rfx_b.FieldByName(tmp_res.Field).Interface()
// functions to compare the values of this field
tmp_res.compare(value, value_a, value_b)
tmp_res.lev(value, value_a, value_b)
result = append(result, tmp_res)
} else if tmp_res.Field == "World" {
/*
I need to retrieve the first index of the Bar slice within the Foo structure.
Even though the variable is a slice, I know it will always have a length of 1 in this use-case.
When retrieved I need to loop over those fields, like what is happening in the previous if statement.
*/
}
}
}
You first need to get the field:
wordField:=rfx_f.Field(i)
which you know to be a slice, so you index it to get the first element
item:=wordField.Index(0)
This will panic if index is out of range.
Then you can iterate the fields:
for fieldIx:=0;fieldIx<item.NumField();fieldIx++ {
field:=item.Field(fieldIx)
}
Related
I know you can create a struct with a literal, listing the fields in order:
type Foo struct {
A string
B string
C string
}
foo := Foo{ "foo", "bar", "baz" }
Is there any way to do the same thing dynamically? I have an array of values (actually an array of arrays) and I want to assign them to an array of structs in field order, and there are rather more than three fields. Is there a way to say "assign this struct's fields from this array of values in order"? I really don't want to write a bunch of structArray[i].field1 = dataArray[i][0]; structArray[i].field2 = dataArray[i][1], etc.
My thoughts so far have been to use reflect, which seems overkillish, or maybe to create an array of field names and build json strings and unmarshal them. Any better ideas?
With reflection you can write a function like this:
func populate(dst any, src any) {
v := reflect.ValueOf(dst)
if v.Type().Kind() != reflect.Pointer {
panic("dst must be a pointer")
}
v = v.Elem()
if v.Type().Kind() != reflect.Struct {
panic("dst must be a pointer to struct")
}
w := reflect.ValueOf(src)
if w.Type().Kind() != reflect.Slice {
panic("src must be a slice")
}
for i := 0; i < v.NumField(); i++ {
// in case you need to support source slices of arbitrary types
value := w.Index(i)
if value.Type().Kind() == reflect.Interface {
value = value.Elem()
}
v.Field(i).Set(value)
}
}
You must make sure that dst is addressable, hence pass a pointer to Foo into populate; and that the i-th element in the source slice is actually assignable to the corresponding i-th field in the struct.
The code above is in a quite simplified form. You can add additional checks to it, e.g. with CanAddr or AssignableTo, if you think callers may misbehave.
Call it like:
func main() {
f := Foo{}
populate(&f, []string{"foo", "bar", "baz"})
fmt.Println(f) // {foo bar baz}
}
Here's a playground that also shows that you can pass a slice of []any as the source slice, in case the struct fields aren't all the same type: https://go.dev/play/p/G8qjDCt79C7
I have two nested structs like this:
type Block struct {
ID string
Contents []string
}
type Package struct {
Name string
Blocks []Block
}
Original package (p) does not change when I try to append a new Content in a specific block.
for _, b := range p.Blocks {
if b.ID == "B1" {
fmt.Println("Adding a new content")
b.Contents = append(b.Contents, "c3")
}
}
Example:
https://play.golang.org/p/5hm6RjPFk8o
This is happening because this line:
for _, b := range p.Blocks {
creates a copy of each element in the slice, and in this case this means creating a copy of each Block in the slice. So when you then make the changes in the loop body, you are making them to the copy of the Block, instead of to the Block in the slice.
If you instead use the index to get a pointer to each Block, e.g.
for i := range p.Blocks {
b := &p.Blocks[i]
// modify b ...
}
it works as expected:
https://play.golang.org/p/h_nXEX9oWRT
Alternatively, you can make the changes to the copy (as in your original code), and then copy the modified value back to the slice:
for i, b := range p.Blocks {
// modify b ...
p.Blocks[i] = b
}
https://go.dev/play/p/kVHTk-OTyC3
Even further, you could instead store pointers to Block in the slice (instead of the Block themselves), in which case your loop would be making a copy of the pointer, which is a valid way to access the Block the original pointers points to:
https://go.dev/play/p/I9-EyV_iCNS
When you are looping over a slice, each of the individual values retrieved from the slice is a copy of the corresponding element in the slice. So to modify the element in the slice, instead of the copy, you can access the element directly using the indexing expression. Or you can use pointers. Note that pointers are also copied but the copied pointer will point to the same address as the element in the slice and therefore can be used to directly modify the same data.
You can use indexing:
for i := range p.Blocks {
if p.Blocks[i].ID == "B1" {
fmt.Println("Adding a new content")
p.Blocks[i].Contents = append(p.Blocks[i].Contents, "c3")
}
}
https://play.golang.org/p/di175k18YQ9
Or you can use pointers:
type Block struct {
ID string
Contents []string
}
type Package struct {
Name string
Blocks []*Block
}
for _, b := range p.Blocks {
if b.ID == "B1" {
fmt.Println("Adding a new content")
b.Contents = append(b.Contents, "c3")
}
}
https://play.golang.org/p/1RjWlCZkhYv
I have a JSON file named test.json which contains:
[
{
"name" : "john",
"interests" : ["hockey", "jockey"]
},
{
"name" : "lima",
"interests" : ["eating", "poker"]
}
]
Now I have written a golang script which reads the JSON file to an slice of structs, and then upon a condition check, modifies a struct fields by iterating over the slice.
Here is what I've tried so far:
package main
import (
"log"
"strings"
"io/ioutil"
"encoding/json"
)
type subDB struct {
Name string `json:"name"`
Interests []string `json:"interests"`
}
var dbUpdate []subDB
func getJSON() {
// open the file
filename := "test.json"
val, err := ioutil.ReadFile(filename)
if err != nil {
log.Fatal(err)
}
err = json.Unmarshal(val, &dbUpdate)
}
func (v *subDB) Change(newresponse []string) {
v.Interests = newresponse
}
func updater(name string, newinterest string) {
// iterating over the slice of structs
for _, item := range dbUpdate {
// checking if name supplied matches to the current struct
if strings.Contains(item.Name, name) {
flag := false // declare a flag variable
// item.Interests is a slice, so we iterate over it
for _, intr := range item.Interests {
// check if newinterest is within any one of slice value
if strings.Contains(intr, newinterest) {
flag = true
break // if we find one, we terminate the loop
}
}
// if flag is false, then we change the Interests field
// of the current struct
if !flag {
// Interests holds a slice of strings
item.Change([]string{newinterest}) // passing a slice of string
}
}
}
}
func main() {
getJSON()
updater("lima", "jogging")
log.Printf("%+v\n", dbUpdate)
}
The output I'm getting is:
[{Name:john Interests:[hockey jockey]} {Name:lima Interests:[eating poker]}]
However I should be getting an output like:
[{Name:john Interests:[hockey jockey]} {Name:lima Interests:[jogging]}]
My understanding was that since Change() has a pointer passed, it should directly modify the field. Can anyone point me out what I'm doing wrong?
The problem
Let's cite what the language specification says on the for ... range loops:
A "for" statement with a "range" clause iterates through all entries
of an array, slice, string or map, or values received on a channel.
For each entry it assigns iteration values to corresponding iteration
variables if present and then executes the block.
So, in
for _, item := range dbUpdate { ... }
the whole statement forms a scope in which a variable named item is declared and it gets assigned a value of each element of dbUpdate, in turn, form the first to the last — as the statement performs its iterations.
All assignments in Go, always and everywhere do copy the value of the expression being assigned, into a variable receiving that value.
So, when you have
type subDB struct {
Name string `json:"name"`
Interests []string `json:"interests"`
}
var dbUpdate []subDB
you have a slice whose backing array contains a set of elements, each of which has type subDB.
Consequently, when for ... range iterates over your slice, on each iteration a shallow copy of the fields of a subDB value contained in the current slice element is done: the values of those fields are copied into the variable item.
We could re-write what happes as this:
for i := 0; i < len(dbUpdate); i++ {
var item subDB
item = dbUpdate[i]
...
}
As you can see, if you mutate item in the loop's body, the changes you do to it do not in any way affect the collection's element currently being iterated over.
The solutions
Broadly speaking, the solution is to become fully acquainted with the fact that Go is very simple in most of the stuff it implements, and so range is no magic to: the iteration variable is just a variable, and assignment to it is just an assignment.
As to solving the particular problem, there are multiple ways.
Refer to a collection element by its index
Do
for i := range dbUpdate {
dbUpdate[i].FieldName = value
}
A corollary to this is that sometimes, when the element is complex or you'd like to delegate its mutation to some function, you may take a pointer to it:
for i := range dbUpdate {
p := &dbUpdate[i]
mutateSubDB(p)
}
...
func mutateSubDB(p *subDB) {
p.SomeField = someValue
}
Keep pointers in the slice
If your slice were declated like
var dbUpdates []*subDB
…and you'd keep pointers to (usually heap-allocated) SubDB values,
the
for _, ptr := range dbUpdate { ... }
statement would naturally copy a pointer to a SubDB (anonymous) variable into ptr as the slice contains pointers and so the assignment copies a pointer.
Since all pointers containing the same address are pointing to the same value, mutating the target variable through the pointer kept in the iteration variable would mutate the same thing which is pointed to by the slice's element.
Which approach to select should usually depend on considerations other than thinking about how one would iterate over the elements — simply because once you understand why your code did not work, you do not have this problem anymore.
As usually: if your values are really big, consider keeping pointers to them.
If you values need to be referenced from multiple places at the same time, keep pointers to them. In other cases keep the values directly — this greatly improves CPU data cache locality (simply put, by the time you're about to access the next element its contents will most likely have been already fetched from the memory, which does not occur when the CPU has to chase a pointer to access some arbitrary memory location through it).
I would like to iterate over the fields of a struct after unmarshalling a JSON object into it and check for the fields whose value was not set (i.e. are empty).
I can get the value of each field and compare that to the reflect.Zero value for the corresponding type
json.Unmarshal([]byte(str), &res)
s := reflect.ValueOf(&res).Elem()
typeOfT := s.Type()
for i := 0; i < s.NumField(); i++ {
f := s.Field(i)
v := reflect.ValueOf(f.Interface())
if (reflect.DeepEqual(v.Interface(), reflect.Zero(v.Type()).Interface())) {
....
But the problem, of course, is that this will not work well for bool or int values.
If a bool field is set to false in the JSON or an int field is set to 0, they will be equal to the zero value of their type. The aforementioned check will consider the fields to be uninitialized, even though they actually have a value set.
I know one way around this is to use pointers, but I just don't see how that would be possible in this case as I'm working with reflect.Value types, not the actual struct.
As you've mentioned, you could use pointers.
The json package can handle unmarshalling values into pointers for you. You've not included the json payload you are trying to unmarshal, or the struct you are unmarshalling into, so I've made up an example.
// json
{
"foo": true,
"number_of_foos": 14
}
// go struct
type Foo struct {
Present bool `json:"foo"`
Num int `json:"number_of_foos"`
}
Here if the keys foo or number_of_foos is missing, then as you've correctly observed, the zero value (false/ 0) will be used. In general the best advice is to make use of the zero value. Create structures so that zero values of false are useful, rather than a pain.
This is not always possible, so changing the types of the fields in the Foo struct to be pointers will allow you to check the 3 cases you are after.
Present
Present and zero
Missing
here is the same struct with pointers:
// go struct
type Foo struct {
Present *bool `json:"foo"`
Num *int `json:"number_of_foos"`
}
Now you can check for presence of the value with fooStruct.Present != nil and if that condition holds, you can assume that the value in the field is the one you wanted.
There is no need to use the reflect package.
Another way of doing the same is by implementing json.Unmarshaler.
type MaybeInt struct {
Present bool
Value int
}
func (i *MaybeInt) UnmarshalJSON(bs []byte) error {
if e := json.Unmarshal(bs, &i.Value); e != nil {
return e
}
i.Present = true
return nil
}
You can then use MaybeInt in your top-level structure:
type Top struct {
N MaybeInt `json:"n"`
M MaybeInt `json:"m"`
}
func main() {
t := Top{}
if e := json.Unmarshal([]byte(` { "n": 4930 } `), &t); e != nil {
panic(e)
}
fmt.Println(t.N, t.M)
}
See it working on the playground
Try using the golang validator package. The validator package offers a required attribute that might do the required job for your need. The official documentation for required attribute states:
This validates that the value is not the data types default zero value. For numbers ensures value is not zero. For strings ensures value is not "". For slices, maps, pointers, interfaces, channels and functions ensures the value is not nil.
The example illustrating the same can be seen at: https://github.com/go-playground/validator/blob/v9/_examples/struct-level/main.go.
Hope this solves your requirement.
I have a struct that looks like this:
type guitaristT struct {
Surname string `required=true`
Year int64 `required=false`
American bool // example of missing tag
Rating float32 `required=true`
Styles []string `required=true,minsize=1`
}
I have an environment variable that looks like the following, and I'm using reflection to fill the struct based on the keys.
jimiEnvvar :="surname=Hendrix|year=1942|american=true|rating=9.99
|styles=blues|styles=rock|styles=psychedelic"
I'm able to set the string, int64, bool and float32 fields using reflection, but I'm stuck on how to append to the slice field Styles. For example, based on the above jimiEnvvar I would like the field jimi.Styles to have the values ["blues","rock", "psychedelic"].
I have the following (simplified) code:
result := guitaristT{}
// result.Styles = make([]string, 10) // XXX causes 10 empty strings at start
result.Styles = make([]string, 0) // EDIT: Alessandro's solution
...
v := reflect.ValueOf(&result).Elem()
...
field := v.FieldByName(key) // eg key = "styles"
...
switch field.Kind() {
case reflect.Slice:
// this is where I get stuck
//
// reflect.Append() has signature:
// func Append(s Value, x ...Value) Value
// so I convert my value to a reflect.Value
stringValue := reflect.ValueOf(value) // eg value = "blues"
// field = reflect.Append(field, stringValue) // XXX doesn't append
field.Set(reflect.Append(field, stringValue)) // EDIT: Alessandro's solution
EDIT:
A second part (which I solved) was filling a map in the struct. For example:
type guitaristT struct {
...
Styles []string `required=true,minsize=1`
Cities map[string]int
}
Where jimiEnvvar looks like:
jimiEnvvar += "|cities=New York^17|cities=Los Angeles^14"
I wrote in this manner:
case reflect.Map:
fmt.Println("keyAsString", keyAsString, "is Map, has value:", valueAsString)
mapKV := strings.Split(valueAsString, "^")
if len(mapKV) != 2 {
log.Fatalln("malformed map key/value:", mapKV)
}
mapK := mapKV[0]
mapV := mapKV[1]
thisMap := fieldAsValue.Interface().(map[string]int)
thisMap[mapK] = atoi(mapV)
thisMapAsValue := reflect.ValueOf(thisMap)
fieldAsValue.Set(thisMapAsValue)
The final result was:
main.guitaristT{
Surname: "Hendrix",
Year: 1942,
American: true,
Rating: 9.989999771118164,
Styles: {"blues", "rock", "psychedelic"},
Cities: {"London":11, "Bay Area":9, "New York":17, "Los Angeles":14},
}
If you're interested the full code is at https://github.com/soniah/reflect/blob/master/structs.go. Code is just some notes/exercises I'm writing.
EDIT2:
Chapter 11 of "Go in Practice" (Butcher and Farina) has detailed explanations of reflection, structs and tags.
You were not too far off. Just replace with
field.Set(reflect.Append(field, stringValue))
and you are done. Also, make sure you that you initialise the slice using
result.Styles = make([]string, 0)
or you will end up having 10 blank string at the top of the Styles array.
Hope this helps and good luck with your project.