I'm writing a program which needs to call a number of C functions. These functions require pointers to structs, which are allocated by me. I know I can't simply convert and pass uintptrs because they hold no semantic value, thus the GC may take my structs away.
So, in order to keep GC from collecting my structs, I came up with this:
type Person struct {
thing uint32
other uint32
}
// Keeps pointers from being collected.
// Note this is a global variable.
var people map[*Person]struct{}
// Creates one or more structs.
func createPeople() (p1, p2 *Person) {
p1 = &Person{name: "John"}
people[p1] = struct{}{} // keep
p2 = &Person{name: "Mary"}
people[p2] = struct{}{} // keep
return
}
// Cleanup.
func deletePeople() {
for k := range people {
delete(people, k)
}
}
func main() {
people = make(map[*Person]struct{})
p1, p2 := createPeople()
// Lots of calls to C functions passing people pointers.
rawP1 := uintptr(unsafe.Pointer(p1))
ffiCall(rawP1)
deletePeople() // mandatory?
}
Will the solution above work?
Is there any risk of the structs being moved, thus messing up the pointers?
What if I don't call deletePeople()?
Related
I want to create a structure which will be accessible in other packages, but I don't want to allow modify this structure. In other languages this is archived by making all fields private and exposing only public getters.
Solution with getters works fine for all data types except slices and maps because returned slices and maps aren't copied by default so they can be modified. The only solution that I managed to figure out is to create new map/slice and assign all items in a loop but this introduce a lot of repetitive and ugly code, especially for large nested structures.
package main
import (
"fmt"
)
type OtherStruct struct {
prop string
}
type Struct struct {
prop map[string]OtherStruct
}
func (s Struct) Prop() map[string]OtherStruct {
return s.prop
}
func (s Struct) Prop2() map[string]*OtherStruct {
prop := make(map[string]*OtherStruct, 0)
for k := range s.prop {
v := s.prop[k]
prop[k] = &v
}
return prop
}
func main() {
var s Struct;
// Simple getter
s = Struct{make(map[string]OtherStruct, 0)}
p1 := s.Prop()
fmt.Println(s) // &{map[]}
p1["something"] = OtherStruct{"test"}
fmt.Println(s) // {map[something:{test}]}
// Getter which copies map
s = Struct{make(map[string]OtherStruct, 0)}
p2 := s.Prop2()
fmt.Println(s) // &{map[]}
p2["something"] = &OtherStruct{"test"}
fmt.Println(s) // &{map[]}
}
Is there any better way to encapsulate slices/maps in Go? Or maybe I shouldn't use encapsulation at all in Go and use different approach?
Returning slice or map values is idiomatic Go. The user of your package will know how those data structures work in Go.
In your example, the user of Struct should know that adding a new entry into the returned map would reflect for any other user of the same map.
// Simple getter
s = Struct{make(map[string]OtherStruct, 0)}
p1 := s.Prop()
fmt.Println(s) // &{map[]}
p1["something"] = OtherStruct{"test"}
fmt.Println(s) // {map[something:{test}]}
You should worry about such things only in case of concurrency. That is, when multiple goroutines are accessing, and possibly changing, the elements inside your slice or map.
I've had difficulty learning the basics of reflect, pointers and interface in go, so here's another entry level question I can't seem to figure out.
This code does what I want it to do - I'm using reflect to add another record to a slice that's typed as an interface.
package main
import (
"reflect"
"log"
)
type Person struct {
Name string
}
func Add(slice interface{}) {
s := reflect.ValueOf(slice).Elem()
// in my actual code, p is declared via the use of reflect.New([Type])
p := Person{Name:"Sam"}
s.Set(reflect.Append(s,reflect.ValueOf(p)))
}
func main() {
p := []Person{}
Add(&p)
log.Println(p)
}
If I changed the Add and main function to this, things don't work the way I want it to.
func Add(slice interface{}) {
s := reflect.ValueOf(&slice).Elem()
p := Person{Name:"Sam"}
s.Set(reflect.Append(reflect.ValueOf(slice),reflect.ValueOf(p)))
log.Println(s)
}
func main() {
p := []Person{}
Add(p)
log.Println(p)
}
That is, the log.Println(p) at the end doesn't show a slice with the record Sam in it like the way I had hoped. So my question is whether it's possible for me to have Add() receive a slice that is not a pointer, and for me to still write some code in Add() that will produce the outcome shown in my first scenario?
A lot of my recent questions dance around this kind of subject, so it's still taking me a while to figure out how to use the reflect package effectively.
No, it's not possible to append to a slice in a function without passing in a pointer to the slice. This isn't related to reflection, but to how variables are passed in to functions. Here's the same code, modified to not use reflection:
package main
import (
"log"
)
type Person struct {
Name string
}
func AddWithPtr(slicep interface{}) {
sp := slicep.(*[]Person)
// This modifies p1 itself, since *sp IS p1
*sp = append(*sp, Person{"Sam"})
}
func Add(slice interface{}) {
// s is now a copy of p2
s := slice.([]Person)
sp := &s
// This modifies a copy of p2 (i.e. s), not p2 itself
*sp = append(*sp, Person{"Sam"})
}
func main() {
p1 := []Person{}
// This passes a reference to p1
AddWithPtr(&p1)
log.Println("Add with pointer: ", p1)
p2 := []Person{}
// This passes a copy of p2
Add(p2)
log.Println("Add without pointer:", p2)
}
(Above, when it says 'copy' of the slice, it doesn't mean the copy of the underlying data - just the slice)
When you pass in a slice, the function effectively gets a new slice that refers to the same data as the original. Appending to the slice in the function increases the length of the new slice, but doesn't change the length of the original slice that was passed in. That's why the original slice remains unchanged.
I have a configuration that defines a number of instances (SomeConfigItems) which have a thing() created for each of them.
That thing is a struct returned by an included package, which contains, among other things, a Price (float64) and a nested struct. The nested struct maintains a map of trades.
The problem is that I am able to loop through the thing.Streams.Trades and see all trades happening in real time from my main()'s for{} loop. I am not able to see an updated thing.Price even though it is set in the Handler on occasion.
I am having a hard time understanding how the nested structs can contain data but not Price. I feel as though I am missing something with scoping, goroutines, or possibly pointers for instantiation of new objects.
Any help would be appreciated, I will continue reading in the meantime. I've reduced the code to what seems relevant.
main.go:
package main
import thing
var Things []thing.Handler
for _, name := range SomeConfigItems {
handler := thing.New(name)
Things = append(Things, handler)
}
for {
for _, t := range Things {
log.Info("Price: ", t.Price) // This is set to 0 every iteration, but I can actively data in thing.Streams.Trades
}
}
thing.go:
package thing
import streams
type Handler struct {
Name string
Price float64
Streams streams.Streams
}
func New(name string) (h Handler, err error) {
stream, err := streams.New(strings.ToLower(name))
h = Handler{
Name: name,
Price: "0.0"
Streams: stream,
}
go h.handler()
return h, err
}
func (bot *Handler) handler() {
var currentPrice float64
for {
currentPrice = external.GetPrice(bot.Name).Price // Validated that this returns a float64
bot.Price = currentPrice // Verified that this is updated immediately after in this context.
// Unable to see Price updated from outer context.
}
}
streams.go:
package streams
type Streams struct {
Trades
}
type State struct {
Price string `json:"p"`
Quantity string `json:"q"`
}
type Trades struct {
Trades map[float64]float64
TradeMutex sync.Mutex
Updates chan State
}
func New(name string) (s Streams, err error) {
p := newTradeStream(name)
s = Streams{
Trades: p,
}
return s, err
}
func newTradeStream(name string) (ts Trades) {
ts = Trades{}
ts.Trades = make(map[float64]float64, MaxDepth)
ts.Updates = make(chan State, 500)
// ... Other watchdog code
return ts
}
Note:
I am added some debug logging in multiple locations. From within the Bot Handler, the price was printed (successfully), then updated, and then printed (successfully) again -- Showing no gap in the setting of Price from within the handler() function.
When adding the same type of debugging to the main() for{} loop, I tried setting an incrementing counter and assigning the value of thing.Price -- Printing thing.Price on each loop results in 0, even if I set the price (and validate it gets set) in the same loop, it is back to 0 on the next iteration.
This behavior is why I think that I am missing something very fundamental.
In Go, arguments are passed to functions by value -- meaning what the function gets is a copy of the value, not a reference to the variable. The same is true of the function receiver, and also the return list.
It's not the most elegant description, but for the sake of explanation, let's call this the "function wall." If the value being passed one way or the other is a pointer, the function still gets a copy, but it's a copy of a memory address, and so the pointer can be used to change the value of the variable on the other side of the wall. If it is a reference type, which uses a pointer in the implementation of the type, then again a change to the thing being pointed to can cross that wall. But otherwise the change does not cross the wall, which is one reason so many Go functions are written to return values instead of just modifying values.
Here's a runnable example:
package main
import (
"fmt"
)
type Car struct {
Color string
}
func (c Car) Change() { // c was passed by value, it's a copy
c.Color = "Red"
}
func main() {
ride := Car{"Blue"}
ride.Change()
fmt.Println(ride.Color)
}
Prints "Blue"
But two small changes:
func (c *Car) Change() { // here
c.Color = "Red"
}
func main() {
ride := &Car{"Blue"} // and here
ride.Change()
fmt.Println(ride.Color)
}
And now it prints "Red". Struct is not a reference type. So if you want modifications to a struct to cross the wall without using the return list to do it, use a pointer. Of course this only applies to values being passed via argument, return list, or receiver; and not to variables that are in scope on both sides of the wall; or to modifying the underlying value behind a reference type.
See also "Pointers Versus Values" in Effective Go, and "Go Data Structures" by Russ Cox.
I am thinking, when I create a System struct, the builder system will cost much memory, but the result is simple, so if I return a address of result, will garbage know that it can collect the builder system memory?
How to test this?
I simulate the situation like this:
// Builder is used to build `System`, and it will cost much memory
type Builder struct {
aux [][]int
system *System
}
// System is the result of `Builder.build`, this is relatively simple
type System struct {
avg []float32
}
func NewSystem() *System {
builder := &Builder{system: &System{}}
builder.build()
return builder.system
}
func (builder *Builder) build() {
// mock initialize
maxCnt := 10000
builder.aux = make([][]int, maxCnt)
for i := range builder.aux {
builder.aux[i] = make([]int, maxCnt)
for j := range builder.aux[i] {
builder.aux[i][j] = rand.Intn(maxCnt)
}
}
builder.system.avg = make([]float32, len(builder.aux))
for i, col := range builder.aux {
var sum float32
for _, row := range col {
sum += float32(row)
}
builder.system.avg[i] = sum / float32(len(builder.aux))
}
}
func TestMem(t *testing.T) {
system := NewSystem()
// I want to know can garbage know that the memory cost by builder is able to be collected
fmt.Println("do many things with system")
fmt.Println(system.avg)
}
Yes, it will be garbage collected once there is enough memory pressure to trigger garbage collection (assuming it's even put on the heap; anything allocated to the stack doesn't need to be garbage collected, as the entire stack is deallocated when no longer in use). The garbage collector will deallocate anything with no remaining references. After your processing finishes, the only thing with a reference is the []float32. If that were instead a slice of structs, and those structs had a pointer back to the parent object, that would prevent the parent being collected.
What I understand about pointer method receiver and non-pointer method receiver is first one can be modified in the method and next one isn't.
So, following worked exactly as I expected.
type student struct {
name string
age int
}
func (s *student) update() {
s.name = "unknown"
s.age = 0
}
func main() {
s := student{"hongseok", 13}
fmt.Println(s)
s.update()
fmt.Println(s)
}
It prints hongseok/13 and unknown/0.
But, I want to replace whole s in update method at once with reassigning. So, I've just altered update method as bellow.
func (s *student) update() {
s = &student{"unknown", 0}
}
And it doesn't change s in main method and prints double hongseok/13.
func (s *student) update() {
*s = student{"unknown", 0}
}
Above change fix the problem.
I think there's no semantic difference. What am I missing?
In the first example:
func (s *student) update() {
s = &student{"unknown", 0}
}
You are assigning an entirely new "pointer value" to s, and the new *s points at a new student value. The variable s is scoped only to the method body, so there are no side effects after this returns.
In the second example
func (s *student) update() {
*s = student{"unknown", 0}
}
You are dereferencing s, and changing the value of *s to point to a new student value, or to put it differently, you are putting a new student value at the address where s points.
In this example you're changing the address that is stored in s to a different value;
func (s *student) update() {
s = &student{"unknown", 0}
}
While using a pointer is regarded as 'passing by reference' the reference itself is a value like any other that is pushed onto the call stack. When you return to main, the value of s is whatever it was in that scope. So to give something more concrete, you called main with s = 1 (calling the addresses 1 and 2 for simplicity), in the method you allocate a new student located at address 2 and you set s = 2, when you return that version of s is popped from the stack and the s in main points to 1 which is unchanged.
In this latter example;
func (s *student) update() {
*s = student{"unknown", 0}
}
You're dereferencing s and assigning a new object to that location, overwriting the existing memory. When you return the pointer in main is still pointing to the same location but you have different data at that location in memory. So in this example you're writing a new student instance to address 1 so when you return you see the new value in the calling scope.
I think, your main problem is that you don't understand well neither one of two concepts that appear in your question.
Let start with pointers. When you don't use pointers, assigning of value means create a simple copy of previous value. The new value is not bound any way with previous one. That means if you change the old value or new, it does not influence the second one. This is normal for primitive types (like ints, bool, string) and structures.
a := 1
b := a
a++ // a was changed, but value of b does not change at all
And now the pointers - it is something that points to some space into memory. For simplicity we create two pointers and both will point to same place.
package main
import (
"fmt"
)
func main() {
type student struct {
name string
age int
}
p1 := &student{"hongseok", 13} // this mean create a new student, but we want only address of student, not value
p2 := p1
(*p1).age = 15 // because p1 and p2 point to same memory (*p2).age also changed
fmt.Println("1.", p2.age) // 15
p2.age = 32 // in golang I could write (*p2).ago or p2.ago this is the same
fmt.Println("2.", p1.age) // 32
fmt.Println("3.",p1, "==", p2)
fmt.Printf("4. %p == %p\n", p1, p2)
// But now I force point p2 to new place
p2 = &student{"another student", 12}
fmt.Println("5.", p1, "!=", p2)
fmt.Printf("6. %p == %p\n", p1, p2)
p1.age = 14 // does it influce p2.age? no! Why? Because it is in different address
fmt.Println("7.", p1, "!=", p2)
// but could is somehow force that point to same address ? Of course
p2 = p1 // p2 will point to same place as p1 and old value of p2 will be freed by garbage collector
}
And don't be confused - pointer can also points named value.
a := 42 // means allocate memory for integer, and we give that memory name "a"
p := &a
*p++ // it change value of a
a = 5 // and of course this change value of *p
And now back to methods, that receiver is/is not a pointer. If receiver of method is pointer, that means you can change it value - the same way as I did few lines ago.
If method receiver is not a pointer, that means - before calling a method, it will be created a copy of struct and method will be called on that copy. You of course can change value of copy, but it does not influence original value.