I'm currently trying to get the size of a complex struct in Go.
I've read solutions that use reflect and unsafe, but neither of these help with structs that contain arrays or maps (or any other field that's a pointer to an underlying data structure).
Example:
type testStruct struct {
A int
B string
C struct{}
items map[string]string
}
How would I find out the correct byte size of the above if items contains a few values in it?
You can get very close to the amount of memory required by the structure and its content by using the package reflect. You need to iterate over the fields and obtain the size of each field. For example:
func getSize(v interface{}) int {
size := int(reflect.TypeOf(v).Size())
switch reflect.TypeOf(v).Kind() {
case reflect.Slice:
s := reflect.ValueOf(v)
for i := 0; i < s.Len(); i++ {
size += getSize(s.Index(i).Interface())
}
case reflect.Map:
s := reflect.ValueOf(v)
keys := s.MapKeys()
size += int(float64(len(keys)) * 10.79) // approximation from https://golang.org/src/runtime/hashmap.go
for i := range(keys) {
size += getSize(keys[i].Interface()) + getSize(s.MapIndex(keys[i]).Interface())
}
case reflect.String:
size += reflect.ValueOf(v).Len()
case reflect.Struct:
s := reflect.ValueOf(v)
for i := 0; i < s.NumField(); i++ {
if s.Field(i).CanInterface() {
size += getSize(s.Field(i).Interface())
}
}
}
return size
}
This obtains the size of v using reflect and then, for the supported types in this example (slices, maps, strings, and structs), it computes the memory required by the content stored in them. You would need to add here other types that you need to support.
There are a few details to work out:
Private fields are not counted.
For structs we are double-counting the basic types.
For number two, you can filter them out before doing the recursive call when handling structs, you can check the kinds in the documentation for the reflect package.
Related
I'm writing a simple program that takes in input from a form, populates an instance of a struct with the received data and the writes this received data to a file.
I'm a bit stuck at the moment with figuring out the best way to iterate over the populated struct and write its contents to the file.
The struct in question contains 3 different types of fields (ints, strings, []strings).
I can iterate over them but I am unable to get their actual type.
Inspecting my posted code below with print statements reveals that each of their types is coming back as structs rather than the aforementioned string, int etc.
The desired output format is be plain text.
For example:
field_1="value_1"
field_2=10
field_3=["a", "b", "c"]
Anyone have any ideas? Perhaps I'm going about this the wrong way entirely?
func (c *Config) writeConfigToFile(file *os.File) {
listVal := reflect.ValueOf(c)
element := listVal.Elem()
for i := 0; i < element.NumField(); i++ {
field := element.Field(i)
myType := reflect.TypeOf(field)
if myType.Kind() == reflect.Int {
file.Write(field.Bytes())
} else {
file.WriteString(field.String())
}
}
}
Instead of using the Bytes method on reflect.Value which does not work as you initially intended, you can use either the strconv package or the fmt to format you fields.
Here's an example using fmt:
var s string
switch fi.Kind() {
case reflect.String:
s = fmt.Sprintf("%q", fi.String())
case reflect.Int:
s = fmt.Sprintf("%d", fi.Int())
case reflect.Slice:
if fi.Type().Elem().Kind() != reflect.String {
continue
}
s = "["
for j := 0; j < fi.Len(); j++ {
s = fmt.Sprintf("%s%q, ", s, fi.Index(i).String())
}
s = strings.TrimRight(s, ", ") + "]"
default:
continue
}
sf := rv.Type().Field(i)
if _, err := fmt.Fprintf(file, "%s=%s\n", sf.Name, s); err!= nil {
panic(err)
}
Playground: https://play.golang.org/p/KQF3CicVzA
Why not use the built-in gob package to store your struct values?
I use it to store different structures, one per line, in files. During decoding, you can test the type conversion or provide a hint in a wrapper - whichever is faster for your given use case.
You'd treat each line as a buffer when Encoding and Decoding when reading back the line. You can even gzip/zlib/compress, encrypt/decrypt, etc the stream in real-time.
No point in re-inventing the wheel when you have a polished and armorall'd wheel already at your disposal.
I have a struct that I want to initialize with a slice of structs in golang, but I'm trying to figure out if there is a more efficient version of appending every newly generated struct to the slice:
package main
import (
"fmt"
"math/rand"
)
type LuckyNumber struct {
number int
}
type Person struct {
lucky_numbers []LuckyNumber
}
func main() {
count_of_lucky_nums := 10
// START OF SECTION I WANT TO OPTIMIZE
var tmp []LuckyNumber
for i := 0; i < count_of_lucky_nums; i++ {
tmp = append(tmp, LuckyNumber{rand.Intn(100)})
}
a := Person{tmp}
// END OF SECTION I WANT TO OPTIMIZE
fmt.Println(a)
}
You can use make() to allocate the slice in "full-size", and then use a for range to iterate over it and fill the numbers:
tmp := make([]LuckyNumber, 10)
for i := range tmp {
tmp[i].number = rand.Intn(100)
}
a := Person{tmp}
fmt.Println(a)
Try it on the Go Playground.
Note that inside the for I did not create new "instances" of the LuckyNumber struct, because the slice already contains them; because the slice is not a slice of pointers. So inside the for loop all we need to do is just use the struct value designated by the index expression tmp[i].
You can use make() the way icza proposes, you can also use it this way:
tmp := make([]LuckyNumber, 0, countOfLuckyNums)
for i := 0; i < countOfLuckyNums; i++ {
tmp = append(tmp, LuckyNumber{rand.Intn(100)})
}
a := Person{tmp}
fmt.Println(a)
This way, you don't have to allocate memory for tmp several times: you just do it once, when calling make. But, contrary to the version where you would call make([]LuckyNumber, countOfLuckyNums), here, tmp only contains initialized values, not uninitialized, zeroed values. Depending on your code, it might make a difference or not.
I have written the following:
func main() {
//inside main
fileInputBytes, err := ioutil.ReadFile("/tmp/test")
byteSize2 := len(fileInputBytes)
var inputFileByteSlice = fileInputBytes[0:]
var numberOfIndexes = math.Floor(float64(byteSize / indexingOffset))
for i := 1; i <= int(numberOfIndexes); i++ {
// adding i to the indexer insures that we use lookahed to ignore previously inserted indexing values
var v int = (i * indexingOffset) + i
Insert(&inputFileByteSlice, v+i, indexingByteValue)
fmt.Println(i)
}
}
//outside main
//variation of https://blog.golang.org/slices with pointers and such
func Insert(slice *[]byte, index int, value byte) {
// Grow the slice by one element.
(*slice) = (*slice)[0 : len(*slice)+1]
// Use copy to move the upper part of the slice out of the way and open a hole.
copy((*slice)[index+1:], (*slice)[index:])
// Store the new value.
(*slice)[index] = value
// Return the result.
}
The slice bounds out of range error is getting on my nerves. The length of the slice grows outside of the size and overflows, the reason I don't understand is that I thought the call to 'grow' the slice by one element(before copy) will dynamically allocate more space. Since that is not the case, can anyone offer me a better suggestion?
First of all, a slice is already a reference type. So you don't need to pass its pointer around if you are not going to change its capacity. So your main can be simplified as:
func main() {
fileInputBytes, err := ioutil.ReadFile("/tmp/test")
byteSize2 := len(fileInputBytes)
// No need to use pointer to slice. If you want a brand new slice
// that does not affect the original slice values, use copy()
inputFileByteArray := fileInputBytes
var numberOfIndexes = math.Floor(float64(byteSize / indexingOffset))
for i := 1; i <= int(numberOfIndexes); i++ {
var v int = (i * indexingOffset) + i
// Insert needs to return the newly updated slice reference
// which should be assigned in each iteration.
inputFileByteArray = Insert(inputFileByteArray, v+i, indexingByteValue)
fmt.Println(i)
}
}
Then, the Insert function can be simplified simply by using append along with copy and returning the newly created slice:
func Insert(slice []byte, index int, value byte) []byte {
if index >= len(slice) {
// add to the end of slice in case of index >= len(slice)
return append(slice, value)
}
tmp := make([]byte, len(slice[:index + 1]))
copy(tmp, slice[:index])
tmp[index] = value
return append(tmp, slice[index:]...)
}
This may not be the best implementation but it is simple enough. Example usage at: https://play.golang.org/p/Nuq4RX9XQD
Your function only works if the slice happens to have enough initial capacity. If you need more capacity, you can only "grow" the slice using the append function. You can still use the *[]byte pointer argument to modify the slice in place like so:
func Insert(slice *[]byte, index int, value byte) {
*slice = append(*slice, 0)
copy((*slice)[index+1:], (*slice)[index:])
(*slice)[index] = value
}
However, it's more customary to return a new slice value, and reassign it each time. This gives you a similar function signature to the builtin append.
func Insert(slice []byte, index int, value byte) []byte {
slice = append(slice, 0)
copy(slice[index+1:], slice[index:])
slice[index] = value
return slice
}
I am trying to create a generic function that can handle actions on slices in Go... for instance, append an item of any type to a slice of that same type. This is simply a generic purpose for a more complex solution, but overall the issue boils down to this example:
package main
type car struct {
make string
color string
}
type submarine struct {
name string
length int
}
func genericAppender(thingList interface{}, thing interface{}) []interface{} {
return append(thingList, thing)
}
func main() {
cars := make([]car, 0, 10)
cars[0] = car{make: "ford", color: "red"}
cars[1] = car{make: "chevy", color: "blue"}
subs := make([]submarine, 0, 10)
subs[0] = submarine{name: "sally", length: 100}
subs[1] = submarine{name: "matilda", length: 200}
newCar := car{make: "bmw", color: "white"}
genericAppender(&cars, newCar)
}
The code playground is at this location
The above errors as follows:
prog.go:14: first argument to append must be slice; have interface {}
After this change you're still getting a runtime error (index out of range) however the problem is that thingList is not of type []interface{} but rather interface{} so you can't append to it. Here's an updated version of your code on playground that does a type assertion to convert it to an []interface{} in line with the append. In reality you need to do that on a separate line and check for errors.
https://play.golang.org/p/YMed0VDZrv
So to put some code here;
func genericAppender(thingList interface{}, thing interface{}) []interface{} {
return append(thingList.([]interface{}), thing)
}
will solve the basic problem you're facing. As noted, you still get runtime errors when indexing into the slice. Also, you could change the argument to avoid this by making it;
func genericAppender(thingList []interface{}, thing interface{}) []interface{} {
return append(thingList, thing)
}
Here's a complete example of the second type; https://play.golang.org/p/dIuW_UG7XY
Note I also corrected the runtime error. When you use make with 3 args they are, in this order, type, length, capacity. This means the length of the array is 0 so when you try to assign to indexes 0 and 1 it was causing a panic for IndexOutoFRange. Instead I removed the middle argument so it's make([]interface{}, 10) meaning the length is initially set to 10 so you can assign to those indexes.
In the answer above if you do the following then it throws error. This is what the original question was about:
//genericAppender(subs, newCar). // Throws "cannot use subs (type []submarine) as type []interface {} in argument to genericAppender"
The trick is to convert your slice of specific type into a generic []interface{}.
func convertToGeneric(thingList interface{}) []interface{} {
input := reflect.ValueOf(thingList)
length := input.Len()
out := make([]interface{},length)
for i:=0 ;i < length; i++ {
out[i] = input.Index(i).Interface()
}
return out
}
This you can call the function like this:
genericAppender(convertToGeneric(subs), newCar)
You can check modified working code here: https://play.golang.org/p/0_Zmme3c8lT
With Go 1.19 (Q4 2022), no need for interface, or "convert your slice of specific type into a generic []interface{}"
CL 363434 comes with a new slices packages:
// Package slices defines various functions useful with slices of any type.
// Unless otherwise specified, these functions all apply to the elements
// of a slice at index 0 <= i < len(s).
package slices
import "constraints"
// Grow increases the slice's capacity, if necessary, to guarantee space for
// another n elements. After Grow(n), at least n elements can be appended
// to the slice without another allocation. If n is negative or too large to
// allocate the memory, Grow panics.
func Grow[S ~[]T, T any](s S, n int) S {
return append(s, make(S, n)...)[:len(s)]
}
// Equal reports whether two slices are equal: the same length and all
// elements equal. If the lengths are different, Equal returns false.
// Otherwise, the elements are compared in index order, and the
// comparison stops at the first unequal pair.
// Floating point NaNs are not considered equal.
func Equal[T comparable](s1, s2 []T) bool {
if len(s1) != len(s2) {
return false
}
for i, v1 := range s1 {
v2 := s2[i]
if v1 != v2 {
return false
}
}
return true
}
// ...
Ian Lance Taylor confirms in issue 45955:
This package is now available at golang.org/x/exp/slices.
Per this thread, it will not be put into standard library until the 1.19 release.
We may of course adjust it based on anything we learn about having it in x/exp.
Given a struct like so:
type B struct {
X string
Y string
}
type D struct {
B
Z string
}
I want to reflect on D and get to the fields X, Y, Z.
Intuitively, before attempting the solution, I was assuming I would be able to traverse the struct D and get all fields using reflection (X, Y, Z) and won't have to deal with B.
But as you can see, I only see the embedded struct B using reflection and not its fields.
http://play.golang.org/p/qZQD5GdTA8
Is there a way I can make B fully transparent when reflecting on D?
Why do I want this?
Imaging a common struct (B in the example here), that is used in multiple other structs by using embedding. Using reflection, the attempt is to copy D into another similar struct in a different package. The destination struct for copying will have all attributes flatly laid out (no embedding there). So there is a mismatch from the source to the destination (embedding vs no embedding) but all the attributes flatly laid out are the same. I don't want to create custom solutions for each struct.
The 'transparency' you expected is just syntactic sugar and has nothing to do with the data representation. If you want to have a function that flattens your data structure, you would have to write it by yourself.
For example (On play):
func DeepFields(iface interface{}) []reflect.Value {
fields := make([]reflect.Value, 0)
ifv := reflect.ValueOf(iface)
ift := reflect.TypeOf(iface)
for i := 0; i < ift.NumField(); i++ {
v := ifv.Field(i)
switch v.Kind() {
case reflect.Struct:
fields = append(fields, DeepFields(v.Interface())...)
default:
fields = append(fields, v)
}
}
return fields
}
Use the following code to collect all promoted field names as keys in map m:
func collectFieldNames(t reflect.Type, m map[string]struct{}) {
// Return if not struct or pointer to struct.
if t.Kind() == reflect.Ptr {
t = t.Elem()
}
if t.Kind() != reflect.Struct {
return
}
// Iterate through fields collecting names in map.
for i := 0; i < t.NumField(); i++ {
sf := t.Field(i)
m[sf.Name] = struct{}{}
// Recurse into anonymous fields.
if sf.Anonymous {
collectFieldNames(sf.Type, m)
}
}
}
Use it like this:
m := make(map[string]struct{})
collectFieldNames(reflect.TypeOf((*D)(nil)), m)
for name := range m {
fmt.Println(name)
}
Run it on the playground.
This program prints X, Y an Z as requested in the question, but also B because B is also a field name.
This function in this answer can be improved:
Don't blow up on recursive type definitions.
Do not include names repeated at the same level in the hierarchy.
The typeField function in encoding/json/encode.go handles both of these issues.