I have a slice of structs that looks like:
type KeyValue struct {
Key uint32
Value uint32
}
var mySlice []Keyvalue
This slice mySlice can have a variable length.
I would like to pass all elements of this slice into a function with this signature:
takeKeyValues(keyValue []uint32)
Now I could easily allocate a slice of []uint32, populate it with all the keys/values of the elements in mySlice and then pass that slice as an argument into takeKeyValues()...
But I'm trying to find a way of doing that without any heap allocations, by directly passing all the keys/values on the stack. Is there some syntax trick which allows me to do that?
There is no safe way to arbitrarily reinterpret the memory layout of the your data. It's up to you whether the performance gain is worth the use of an unsafe type conversion.
Since the fields of KeyValue are of equal size and the struct has no padding, you can convert the underlying array of KeyValue elements to an array of uint32.
takeKeyValues((*[1 << 30]uint32)(unsafe.Pointer(&s[0]))[:len(s)*2])
https://play.golang.org/p/Jjkv9pdFITu
As an alternative to JimB's answer, you can also use the reflect (reflect.SliceHeader) and unsafe (unsafe.Pointer) packages to achieve this.
https://play.golang.org/p/RLrMgoWgI7t
s := []KeyValue{{0, 100}, {1, 101}, {2, 102}, {3, 103}}
var data []uint32
sh := (*reflect.SliceHeader)(unsafe.Pointer(&data))
sh.Data = uintptr(unsafe.Pointer(&s[0]))
sh.Len = len(s) * 2
sh.Cap = sh.Len
fmt.Println(data)
I a bit puzzled by what you want. If the slice is dynamic, you have to grow it dynamically as you add elements to it (at the call site).
Go does not have alloca(3), but I may propose another approach:
Come up with an upper limit of the elements you'd need.
Declare variable of the array type of that size (not a slice).
"Patch" as many elements you want (starting at index 0) in that array.
"Slice" the array and pass the slice to the callee.
Slicing an array does not copy memory — it just takes the address of the array's first element.
Something like this:
func caller(actualSize int) {
const MaxLen = 42
if actualSize > MaxLen {
panic("actualSize > MaxLen")
}
var kv [MaxLen]KeyValue
for i := 0; i < actualSize; i++ {
kv[i].Key, kv[i].Value = 100, 200
}
callee(kv[:actualSize])
}
func callee(kv []KeyValue) {
// ...
}
But note that when the Go compiler sees var kv [MaxLen]KeyValue, it does not have to allocate it on the stack. Basically, I beleive the language spec does not tell anything about the stack vs heap.
Related
Let's say a Go 1.18 program has a quite heavy struct, for which copying is to be considered costly:
type MyStruct struct {
P string
// a lot of properties
}
Now let's define a function, taking a slice of such elements as input parameter, which goal is to update properties of each slice element:
func myFunc(sl []MyStruct) {
for i := range sl {
p := &sl[i] // <-- HERE
p.P = "bar"
// other properties mutations
}
}
At the <-- HERE mark, is the Golang compiler making a temporary copy of the slice element into the loop's scope, or is it taking the address of the slice element in-place?
The idea is to avoid copying the whole slice element.
A working example: https://go.dev/play/p/jHOC2DauyrQ?v=goprev
&sl[i] does not copy the slice element, it just evaluates to the address of the ith element.
Slice elements act as variables, and &x evaluates to the address of the x variable. Think about it: since &sl[i] is the address of the ith element, the address does not need nor use the struct value, why would it make a copy?
If your slices are so big that you're worried about the performance impact of (implicit) copies, you really should consider storing pointers in the slice in the first place, and that way you can make your loop and accessing elements much simpler without having to worry about copies:
func myFunc(sl []*MyStruct) {
for _, v := range sl {
v.P = "bar"
// other properties mutations
}
}
Also note that if your slice holds non-pointers, and you want to change a field of a slice element, indexing the slice and referring to the field also does not involve copying the struct element:
func myFunc(sl []MyStruct) {
for i := range sl {
sl[i].P = "bar"
// other properties mutations
}
}
Yes, this may be more verbose and may be less efficient if you have to modify multiple fields (but compilers may also recognize and optimize the evaluation of multiple sl[i] expressions).
var list = []func(*someType) error {
...
}
I am new to Go and I am trying to understand what does the syntax mean ?
Is the return of function an array?
This declares and initializes a variable list as a slice whose elements are functions with signature func(*someType) error.
Slices in Go are convenient mechanisms for representing sequences of data of a particular type. They have type []T for any element type T (but remember Go does not have generics). A slice is defined only by the type of the items it contains; its length is not part of its type definition and can change at runtime. (Arrays in Go, by contrast, are of fixed length - their type is [N]T for length N and element type T).
Under the surface, a slice consists of a backing array, a length of the current data and a capacity. The runtime manages the memory allocation of the array to accommodate all data in the slice.
func in go is a type like int,string...
So they are sample syntax:
var listInt := []int{1,2,3}
var listStr := []string{"1","2","3"}
var listFunc := []func(param anyType) anyType {
func(param anyType) anyType { ... return new(anyType) },
func(param anyType) anyType { ... return new(anyType) },
}
func (c *conn) setState(nc net.Conn, state ConnState) {
...
c.curState.Store(connStateInterface[state])
...
}
// connStateInterface is an array of the interface{} versions of
// ConnState values, so we can use them in atomic.Values later without
// paying the cost of shoving their integers in an interface{}.
var connStateInterface = [...]interface{}{
StateNew: StateNew,
StateActive: StateActive,
StateIdle: StateIdle,
StateHijacked: StateHijacked,
StateClosed: StateClosed,
}
I can't figure out the trick with connStateInterface, how exactly does it work?
There's a few things going on here...
The [...] declaration creates an actual array instead of a slice, so that indirection is removed. What's being declared here is an array of interface{} types... so you might wonder why the weird map-looking notation?
The StateXXX variables are simply constants declared further above, so they are ints... so the declaration is actually of the form index: value.
Here's a less obfuscated example of that using an array of ints:
var i = [...]int{4: 2, 2: 7}
This will allocate an array containing:
[0, 0, 7, 0, 2]
... note that index 2 has 7, index 4 has 2. Not a common way of declaring an array, but it's valid Go.
So going back to the original declaration, just take the example I gave above, and instead of int, make the array of type interface{}:
var i = [...]interface{}{4: 2, 2: 7}
And you'll get a similar array, but with nil interface values in place of zeroes.
Getting even closer to the original code, the StateXXX constants are just ints, only not literals like in my example.
So, what's the point of all this? Why all the obfuscation?
It's a performance hack. The function c.curState.Store() takes an argument of type interface{}. If you were to pass it an int, the compiled code would have to fumble about with converting the type on each call. A more clear (though obviously impractical) illustration of this might be:
var val interface{}
for i := 0; i < 1000000; i++ {
// the types are different, compiler has to fumble int vs. interface{}
val = i
// do something with val
}
Every time you do val = i a conversion between int and interface{} needs to happen. The code you posted avoids this by creating a static lookup table where all the values are already of type interface.
Therefore, this:
c.curState.Store(connStateInterface[state])
is more efficient than this:
c.curState.Store(state)
Since state would, in this case, need to undergo the int -> interface{} conversion. In the optimized code, state is merely an index looking up a value into an array, the result of which gets you an interface{}... so the int -> interface{} type conversion is avoided.
I'm not familiar with that code, but I'd imagine it's in a critical path and the nanoseconds or whatever savings shaved off likely makes a difference.
I have read on the golang blog: https://blog.golang.org/go-maps-in-action that:
var m map[string]int
Map types are reference types, like pointers or slices, and so the
value of m above is nil; it doesn't point to an initialized map. A nil
map behaves like an empty map when reading, but attempts to write to a
nil map will cause a runtime panic; don't do that. To initialize a
map, use the built in make function:
m = make(map[string]int)
The make function allocates and initializes a hash map data structure
and returns a map value that points to it.
I have a hard time understanding some parts of this:
What does var m map[string]int do?
Why do I need to write m = make(map[string]int) but not i = make(int)
What does var m map[string]int do?
It tells the compiler that m is a variable of type map[string]int, and assigns "The Zero Value" of the type map[string] int to m (that's why m is nil as nil is The Zero Value of any map).
Why do I need to write m = make(map[string]int) but not i = make(int)
You don't need to. You can create a initialized map also like this:
m = map[string]int{}
which does exactly the same.
The difference between maps and ints is: A nil map is perfectly fine. E.g. len() of a nil map works and is 0. The only thing you cannot do with a nil map is store key-value-pairs. If you want to do this you'll have to prepare/initialize the map. This preparation/initialization in Go is done through the builtin make (or by a literal map as shown above). This initialization process is not needed for ints. As there are no nil ints this initialization would be total noise.
Note that you do not initialize the variable m: The variable m is a map of strings to ints, initialized or not. Like i is a variable for ints. Now ints are directly usable while maps require one more step because the language works that way.
What does var m map[string]int do?
You can think about it like pointer with nil value, it does not point to anything yet but able to point to concrete value.
Why do I need to write m = make(map[string]int) but not i = make(int)
https://golang.org/doc/effective_go.html#allocation_make
Back to allocation. The built-in function make(T, args) serves a purpose different from new(T). It creates slices, maps, and channels only, and it returns an initialized (not zeroed) value of type T (not *T). The reason for the distinction is that these three types represent, under the covers, references to data structures that must be initialized before use. A slice, for example, is a three-item descriptor containing a pointer to the data (inside an array), the length, and the capacity, and until those items are initialized, the slice is nil. For slices, maps, and channels, make initializes the internal data structure and prepares the value for use. For instance,
make([]int, 10, 100)
allocates an array of 100 ints and then creates a slice structure with length 10 and a capacity of 100 pointing at the first 10 elements of the array. (When making a slice, the capacity can be omitted; see the section on slices for more information.) In contrast, new([]int) returns a pointer to a newly allocated, zeroed slice structure, that is, a pointer to a nil slice value.
These examples illustrate the difference between new and make.
var p *[]int = new([]int) // allocates slice structure; *p == nil; rarely useful
var v []int = make([]int, 100) // the slice v now refers to a new array of 100 ints
// Unnecessarily complex:
var p *[]int = new([]int)
*p = make([]int, 100, 100)
// Idiomatic:
v := make([]int, 100)
Remember that make applies only to maps, slices and channels and does not return a pointer. To obtain an explicit pointer allocate with new or take the address of a variable explicitly.
All words have the same length of 32 bits (4 bytes) or 64 bits (8 bytes),
depending on the processor and the operating system. They are identified by their memory address (represented as a hexadecimal number).
All variables of primitive types like int, float, bool, string ... are value types, they point directly to the values contained in the memory. Also composite types like arrays and structs are value types. When assigning with = the value of a value type to another variable: j = i, a copy of the original value i is made in memory.
More complex data which usually needs several words are treated as reference types. A reference type variable r1 contains the address (a number) of the memory location where the value of r1 is stored (or at least the 1st word of it):
For reference types when assigning r2 = r1, only the reference (the address) is copied and not the value!!. If the value of r1 is modified, all references of that value (like r1 and r2) will be reflected.
In Go pointers are reference types, as well as slices, maps and channels. The variables that are referenced are stored in the heap, which is garbage collected.
In the light of the above statements it's clear why the article states:
To initialize a map, use the built in make function.
The make function allocates and initializes a hash map data structure and returns a map value that points to it. This means you can write into it, compare to
var m map[string]int
which is readable, resulting a nil map, but an attempt to write to a nil map will cause a runtime panic. This is the reason why it's important to initialize the map with make.
m = make(map[string]int)
I have a struct array of type []Struct. When I range over it in the form:
for i, val := range mystructarray
I understand that val is a local variable which contains a copy of mystructarray[i]. Is there a better way of iterating through the addressess of mystructarray than this:
for i := range mystructarray{
valptr = &mystructarray[i]
}
?
There is no way to iterate while receiving a pointer to the contents of the slice (unless of course, it is a slice of pointers).
Your example is the best way:
for i := range mySlice {
x = &mySlice[i]
// do something with x
}
Remember however, if your structs aren't very large, and you don't need to operate on them via a pointer, it may be faster to copy the struct, and provide you with clearer code.