Why does pointer assignment cause variable assignment to not always stick? - go

Pointer assignment of index is being consistently inconsistent within addData(..). I expect memory address is moving around as underlying array increases in size.
Behavior: I assign to variable A, then assign B = A*0.2, then assign y = sig(B), finally B = y. Sometimes on the next loop B == y || B == A*0.2. It is perfectly consistent across multiple executions.
I made a simpler and more full version of the code.
package main
import(
"fmt"
"math"
)
func main(){
//Structure setup
l := lots{}; l.addData(2); l.addData(2); l.addData(2); l.addData(2)
l.val[0].y[0] = 0.20700021
l.val[0].y[1] = 0.30003001
l.propagate()
}
type data struct{
y []float64
}
type pair struct {
one *data
two *data
}
// lots is the biggest part of the structure
// the problem seems to occure when this is introduced
type lots struct{
val []data
join []pair
}
// addData appends a data struct and a pair struct to
// the corresponding parts of lots struct
func (l *lots)addData(size int){
l.val = append(l.val, data{make([]float64, size)})
// should be skipped first call only
if(len(l.join) < len(l.val)-1){
fmt.Println("len of l.val: ", len(l.val))
l.join = append(l.join, pair{})
l.join[len(l.join)-1].one = &l.val[len(l.val)-2]
l.join[len(l.join)-1].two = &l.val[len(l.val)-1]
}
}
// propagate
func (l *lots)propagate(){
for _, v := range l.join{
v.travel()
}
}
// travel modifies values going from p.one -> p.two
func (p *pair) travel(){
fmt.Println("p.one.y: ", p.one.y)
p.mathy()
fmt.Println("p.two.y: ", p.two.y)
p.two.y = sigmoid(p.two.y)
fmt.Println("p.two.y: ", p.two.y)
}
func (p *pair) mathy(){
for i := range p.one.y {
p.two.y[i] = p.one.y[i] * p.one.y[i]
}
}
// sigmoid seems to be causing some problems.
// Works fine on it's own though
func sigmoid(x []float64)(y []float64){
y = make([]float64, len(x))
for i := range x{
y[i] = 1./(1.+math.Exp(-x[i]))
}
return
}
I expect the #'s of p.two.y: [#'s] to equal the following lines #'s of p.one.y: [#'s] all the time. The output I am getting is not consistently equal. Sometimes the p.one.y: [#'s] #'s are equal to the p.two.y: [#'s] from the line preceding line; that value was over written and then that value came back.
p.one.y: [0.20700021 0.30003001]
p.two.y: [0.04284908694004409 0.0900180069006001]/////// bad
p.two.y: [0.5107106330188076 0.5224893174114301] // overwritten
p.one.y: [0.04284908694004409 0.0900180069006001]/////// reappeared
p.two.y: [0.0018360442515954571 0.008103241566356488]
p.two.y: [0.5004590109339528 0.5020257993066767]//// overwritten
p.one.y: [0.5004590109339528 0.5020257993066767]//// good
p.two.y: [0.25045922162499035 0.25202990316950763]
p.two.y: [0.5622895277500193 0.5626760660176802]
I have tried to reduce function nesting, and it worked when I put everything into the Propagate() function and assigned directly to p.two.y[i] with the sigmoid function. (below)
// propagate
func (l *lots)propagate(){
for _, p := range l.join{
fmt.Println("p.one.y: ", p.one.y)
for i := range p.one.y {
p.two.y[i] = p.one.y[i] * p.one.y[i]
}
fmt.Println("p.two.y: ", p.two.y)
// using this extra variable causes the problem of inconsistent assignment
//y := make([]float64, len(p.two.y))
for i := range p.two.y{
//y[i] = 1./(1.+math.Exp(-p.two.y[i]))
p.two.y[i] = 1./(1.+math.Exp(-p.two.y[i]))
}
//p.two.y = y
fmt.Println("p.two.y: ", p.two.y)
}
}
This version provides good data, but takes away so much of the specialization I like.
p.one.y: [0.20700021 0.30003001]
p.two.y: [0.04284908694004409 0.0900180069006001]
p.two.y: [0.5107106330188076 0.5224893174114301]////
p.one.y: [0.5107106330188076 0.5224893174114301]//// Good
p.two.y: [0.2608253506784712 0.27299508680906215]
p.two.y: [0.564839170446528 0.5678280461350629]////
p.one.y: [0.564839170446528 0.5678280461350629]//// Good
p.two.y: [0.3190432884707219 0.3224286899775631]
p.two.y: [0.5790910765397528 0.5799160282084651]

The problem is with the pointer assignments as you build the slice you are trying to reference. The addresses keep changing.
func main(){
var lump []int
// A loop to build a slice of `int`'s from 0 size to 8 size
// and print each index address
for i:= 0; i < 8; i++{
lump = append(lump, int(i))
fmt.Printf("addr of lump[%v]: %p\n",i, &lump[i])
}
fmt.Println()
// A loop to look at the addresses of each index
for i := range lump{
fmt.Printf("addr of lump[%v]: %p\n",i, &lump[i])
}
}
Check out the addresses not being created in sequential memory locations.
//while building the slice
// notice the addresses making big jumps
addr of lump[0]: 0xc00000a0c8
addr of lump[1]: 0xc00000a0f8
addr of lump[2]: 0xc00000e3b0
addr of lump[3]: 0xc00000e3b8
addr of lump[4]: 0xc00000c2e0
addr of lump[5]: 0xc00000c2e8
addr of lump[6]: 0xc00000c2f0
addr of lump[7]: 0xc00000c2f8
//after building the slice
// notice all address being sequential
addr of lump[0]: 0xc00000c2c0
addr of lump[1]: 0xc00000c2c8
addr of lump[2]: 0xc00000c2d0
addr of lump[3]: 0xc00000c2d8
addr of lump[4]: 0xc00000c2e0
addr of lump[5]: 0xc00000c2e8
addr of lump[6]: 0xc00000c2f0
addr of lump[7]: 0xc00000c2f8
You could move to C/C++ where you could handle all the memory adjustments as the array increases in size. Or build one slice then the other.

Related

Pointing map to a new map inside a method looses effect after leaving method

This code, visible on the Go PlayGround:
package main
import "fmt"
type MyType map[int]int
func (x MyType) updateIt() {
newItem := make(MyType)
for i := 0; i < 3; i++ {
newItem[i] = i
}
x = newItem
fmt.Println(x)
}
func main() {
x := make(MyType)
x.updateIt()
fmt.Println(x)
}
Produces:
map[0:0 1:1 2:2]
map[]
While I expect:
map[0:0 1:1 2:2]
map[0:0 1:1 2:2]
Why?
My understanding is that: x.updateIt() takes x as an argument, then creates newItem and changes the pointer of x so that it points to newItem. So x points to the data of newItem. This seems to be indeed the case inside updateIt, as visible from the first print line. But when x is print after the method, the change is lost.
I'm new to Go and this is unexpected behaviour for me, could you please explain?
You should be using a pointer receiver:
func (x *MyType) updateIt() {
newItem := make(MyType)
for i := 0; i < 3; i++ {
newItem[i] = i
}
*x = newItem
fmt.Println(x)
}
Playground: https://play.golang.org/p/K82TTjHdDgg
Explanation
Arguments "in front" of a function (so-called receivers) behave more or less like ordinary arguments.
Imagine a function like this:
func updateIt(x MyType) {
newItem := make(MyType)
for i := 0; i < 3; i++ {
newItem[i] = i
}
x = newItem
fmt.Println(x)
}
Here, you pass x by value, not by reference (or as a pointer).
What happens is that x gets replaced inside the function, but keeps its original value outside of it. By passing a pointer you can replace the value to which x is pointing, thereby changing the value of x outside of the function.
See also "A Tour of Go, Pointers": https://tour.golang.org/moretypes/1
Inside the main function you create a new map and assign the pointer of that new map to x.
Then the value of that pointer is passed to the updateIt function.
Then inside updateIt you replace it with pointer to another map you create inside updateIt. Then it updates this new map instead of updating the original map created inside main
The most important thing to understand here is that your program creates two different maps.
So avoid creating two maps and update the same map passed to your updateIt function.
func (x MyType) updateIt() {
for i := 0; i < 3; i++ {
x[i] = i
}
fmt.Println(x)
}
https://play.golang.org/p/-MemfTv1uJV

Memory usage: nil interface{} vs struct{}

I'm trying to learn more regarding memory usage.
Doing some tests with interface{} and struct{} slices, I noticed that a slice of struct{} doesn't allocate any memory whereas a slice of interface{} does. It doesn't make so much sense to me, I'm actually expecting the same behavior (ie. both allocate nothing). Anyway I couldn't find any explanation regarding this particular case.
Could someone explain me why this happens?
package main
import (
"runtime"
"fmt"
)
func main() {
// Below is an example of using our PrintMemUsage() function
// Print our starting memory usage (should be around 0mb)
fmt.Println("Start")
PrintMemUsage()
fmt.Println("")
structContainer := make([]struct{}, 1000000)
for i := 0; i<1000000; i++ {
structContainer[i] = struct{}{}
}
fmt.Println("With 1kk struct{}")
PrintMemUsage()
fmt.Println("")
nilContainer := make([]interface{}, 1000000)
for i := 0; i<1000000; i++ {
nilContainer[i] = nil
}
fmt.Println("With 1kk nil interface{}")
PrintMemUsage()
fmt.Println("")
}
// PrintMemUsage outputs the current, total and OS memory being used. As well as the number
// of garage collection cycles completed.
func PrintMemUsage() {
var m runtime.MemStats
runtime.ReadMemStats(&m)
// For info on each, see: https://golang.org/pkg/runtime/#MemStats
fmt.Printf("Alloc = %v KiB", bToMb(m.Alloc))
fmt.Printf("\tTotalAlloc = %v KiB", bToMb(m.TotalAlloc))
fmt.Printf("\tSys = %v KiB", bToMb(m.Sys))
fmt.Printf("\tNumGC = %v\n", m.NumGC)
}
func bToMb(b uint64) uint64 {
return b / 1024
}
Playground link.
A variable of type interface{} can hold any value. E.g. it can hold the integer 8, it can hold the string value "hi", it can hold the struct value image.Point{X: 1, Y: 2} and pretty much everything else.
If you allocate a slice having interface{} as its element type, memory have to be allocated so that you can store any values in its elements. When using make() to allocate it, all its elements will get the zero value of the element type (which is nil for the interface{}), but memory still has to be allocated else you couldn't set elements later on.
On the other hand, the empty struct struct{} has no fields, it cannot hold any values (other than struct{}). When you allocate a slice having struct{} as its element type, memory does not need to be allocated because you won't be able to store anything in it that would require memory. So it's a simple and clever optimization not to allocate memory for such a type.
This is because an empty struct contains no value.
This is not very useful for arrays or slices. But it is useful for maps. A map without value is like a set. You can insert keys and test if they are present. The absence of value save space as you discovered.

Expanding a slice's size to prevent slice bounds out of range error

I have written the following:
func main() {
//inside main
fileInputBytes, err := ioutil.ReadFile("/tmp/test")
byteSize2 := len(fileInputBytes)
var inputFileByteSlice = fileInputBytes[0:]
var numberOfIndexes = math.Floor(float64(byteSize / indexingOffset))
for i := 1; i <= int(numberOfIndexes); i++ {
// adding i to the indexer insures that we use lookahed to ignore previously inserted indexing values
var v int = (i * indexingOffset) + i
Insert(&inputFileByteSlice, v+i, indexingByteValue)
fmt.Println(i)
}
}
//outside main
//variation of https://blog.golang.org/slices with pointers and such
func Insert(slice *[]byte, index int, value byte) {
// Grow the slice by one element.
(*slice) = (*slice)[0 : len(*slice)+1]
// Use copy to move the upper part of the slice out of the way and open a hole.
copy((*slice)[index+1:], (*slice)[index:])
// Store the new value.
(*slice)[index] = value
// Return the result.
}
The slice bounds out of range error is getting on my nerves. The length of the slice grows outside of the size and overflows, the reason I don't understand is that I thought the call to 'grow' the slice by one element(before copy) will dynamically allocate more space. Since that is not the case, can anyone offer me a better suggestion?
First of all, a slice is already a reference type. So you don't need to pass its pointer around if you are not going to change its capacity. So your main can be simplified as:
func main() {
fileInputBytes, err := ioutil.ReadFile("/tmp/test")
byteSize2 := len(fileInputBytes)
// No need to use pointer to slice. If you want a brand new slice
// that does not affect the original slice values, use copy()
inputFileByteArray := fileInputBytes
var numberOfIndexes = math.Floor(float64(byteSize / indexingOffset))
for i := 1; i <= int(numberOfIndexes); i++ {
var v int = (i * indexingOffset) + i
// Insert needs to return the newly updated slice reference
// which should be assigned in each iteration.
inputFileByteArray = Insert(inputFileByteArray, v+i, indexingByteValue)
fmt.Println(i)
}
}
Then, the Insert function can be simplified simply by using append along with copy and returning the newly created slice:
func Insert(slice []byte, index int, value byte) []byte {
if index >= len(slice) {
// add to the end of slice in case of index >= len(slice)
return append(slice, value)
}
tmp := make([]byte, len(slice[:index + 1]))
copy(tmp, slice[:index])
tmp[index] = value
return append(tmp, slice[index:]...)
}
This may not be the best implementation but it is simple enough. Example usage at: https://play.golang.org/p/Nuq4RX9XQD
Your function only works if the slice happens to have enough initial capacity. If you need more capacity, you can only "grow" the slice using the append function. You can still use the *[]byte pointer argument to modify the slice in place like so:
func Insert(slice *[]byte, index int, value byte) {
*slice = append(*slice, 0)
copy((*slice)[index+1:], (*slice)[index:])
(*slice)[index] = value
}
However, it's more customary to return a new slice value, and reassign it each time. This gives you a similar function signature to the builtin append.
func Insert(slice []byte, index int, value byte) []byte {
slice = append(slice, 0)
copy(slice[index+1:], slice[index:])
slice[index] = value
return slice
}

Read Random Memory Locations with Golang

Good evening,
I've been trying to build a golang application which scans values in memory but am struggling trying to understand how to address specific memory locations. I know that when accessing memory within the application you can use *variablename to deference and get the address location, but how would I provide an address location and print the value to the screen or grab the next allocated object of any size from RAM and print it's value?
Thanks in advance for any help you may be willing to share
I don't know how much useful this will be, but here is a sample code.
package main
import (
"fmt"
"unsafe"
)
func main() {
var i int = 1
fmt.Println("Address : ", &i, " Value : ", i)
var address *int
address = &i // getting the starting address
loc := (uintptr)(unsafe.Pointer(address))
p := unsafe.Pointer(loc)
// verification - it should print 1
var val int = *((* int)(p))
fmt.Println("Location : ", loc, " Val :",val) // it does print !!
// lets print 1000 bytes starting from address of variable i
// first memory location contains 1 as expected
printValueAtMemoryLocation(loc, 1000)
// now lets test for some arbitrary memory location
// not so random ! wanted to reduce the diff value also any arbitrary memory location you can't read !!
memoryToReach := 842350500000
loc = changeToInputLocation(loc, memoryToReach)
fmt.Println("Loc is now at : ", loc)
// lets print 1000 bytes starting from the memory location "memoryToReach"
printValueAtMemoryLocation(loc, 1000)
}
func changeToInputLocation(location uintptr, locationToreach int) uintptr {
var diff,i int
diff = locationToreach - int(location)
fmt.Println("We need to travel ", diff, " memory locations !")
if diff < 0 {
i= diff * -1
for i > 0 {
location--
i--
}
} else {
i= diff
for i > 0 {
location++
i--
}
}
return location
}
func printValueAtMemoryLocation(location uintptr, next int) {
var v byte
p := unsafe.Pointer(location)
fmt.Println("\n")
for i:=1; i<next; i++ {
p = unsafe.Pointer(location)
v = *((*byte)(p))
fmt.Print(v," ")
//fmt.Println("Loc : ", loc, " --- Val : ", v)
location++
}
fmt.Println("\n")
}
Using "unsafe" package is not a good idea, also you can not read any arbitrary location I believe.
For me when I tried some other random locations where, most probably, I didn't have read access, it threw me error like this:
unexpected fault address 0xc41ff8f780
fatal error: fault
[signal SIGBUS: bus error code=0x2 addr=0xc41ff8f780 pc=0x1093ec0]
But hopefully, it can be of some value to you.

creating generic functions for multi type arrays in Go

I am trying to create a generic function that can handle actions on slices in Go... for instance, append an item of any type to a slice of that same type. This is simply a generic purpose for a more complex solution, but overall the issue boils down to this example:
package main
type car struct {
make string
color string
}
type submarine struct {
name string
length int
}
func genericAppender(thingList interface{}, thing interface{}) []interface{} {
return append(thingList, thing)
}
func main() {
cars := make([]car, 0, 10)
cars[0] = car{make: "ford", color: "red"}
cars[1] = car{make: "chevy", color: "blue"}
subs := make([]submarine, 0, 10)
subs[0] = submarine{name: "sally", length: 100}
subs[1] = submarine{name: "matilda", length: 200}
newCar := car{make: "bmw", color: "white"}
genericAppender(&cars, newCar)
}
The code playground is at this location
The above errors as follows:
prog.go:14: first argument to append must be slice; have interface {}
After this change you're still getting a runtime error (index out of range) however the problem is that thingList is not of type []interface{} but rather interface{} so you can't append to it. Here's an updated version of your code on playground that does a type assertion to convert it to an []interface{} in line with the append. In reality you need to do that on a separate line and check for errors.
https://play.golang.org/p/YMed0VDZrv
So to put some code here;
func genericAppender(thingList interface{}, thing interface{}) []interface{} {
return append(thingList.([]interface{}), thing)
}
will solve the basic problem you're facing. As noted, you still get runtime errors when indexing into the slice. Also, you could change the argument to avoid this by making it;
func genericAppender(thingList []interface{}, thing interface{}) []interface{} {
return append(thingList, thing)
}
Here's a complete example of the second type; https://play.golang.org/p/dIuW_UG7XY
Note I also corrected the runtime error. When you use make with 3 args they are, in this order, type, length, capacity. This means the length of the array is 0 so when you try to assign to indexes 0 and 1 it was causing a panic for IndexOutoFRange. Instead I removed the middle argument so it's make([]interface{}, 10) meaning the length is initially set to 10 so you can assign to those indexes.
In the answer above if you do the following then it throws error. This is what the original question was about:
//genericAppender(subs, newCar). // Throws "cannot use subs (type []submarine) as type []interface {} in argument to genericAppender"
The trick is to convert your slice of specific type into a generic []interface{}.
func convertToGeneric(thingList interface{}) []interface{} {
input := reflect.ValueOf(thingList)
length := input.Len()
out := make([]interface{},length)
for i:=0 ;i < length; i++ {
out[i] = input.Index(i).Interface()
}
return out
}
This you can call the function like this:
genericAppender(convertToGeneric(subs), newCar)
You can check modified working code here: https://play.golang.org/p/0_Zmme3c8lT
With Go 1.19 (Q4 2022), no need for interface, or "convert your slice of specific type into a generic []interface{}"
CL 363434 comes with a new slices packages:
// Package slices defines various functions useful with slices of any type.
// Unless otherwise specified, these functions all apply to the elements
// of a slice at index 0 <= i < len(s).
package slices
import "constraints"
// Grow increases the slice's capacity, if necessary, to guarantee space for
// another n elements. After Grow(n), at least n elements can be appended
// to the slice without another allocation. If n is negative or too large to
// allocate the memory, Grow panics.
func Grow[S ~[]T, T any](s S, n int) S {
return append(s, make(S, n)...)[:len(s)]
}
// Equal reports whether two slices are equal: the same length and all
// elements equal. If the lengths are different, Equal returns false.
// Otherwise, the elements are compared in index order, and the
// comparison stops at the first unequal pair.
// Floating point NaNs are not considered equal.
func Equal[T comparable](s1, s2 []T) bool {
if len(s1) != len(s2) {
return false
}
for i, v1 := range s1 {
v2 := s2[i]
if v1 != v2 {
return false
}
}
return true
}
// ...
Ian Lance Taylor confirms in issue 45955:
This package is now available at golang.org/x/exp/slices.
Per this thread, it will not be put into standard library until the 1.19 release.
We may of course adjust it based on anything we learn about having it in x/exp.

Resources