I have the following code snippet which "go vet" complains about with the warning "possible misuse of reflect.SliceHeader". I can not find very much information about this warning other then this. After reading that it is not very clear to me what is needed to do this in a way that makes go vet happy - and without possible gc issues.
The goal of the snippet is to have a go function copy data to memory which is managed by an opaque C library. The Go function expects a []byte as a parameter.
func Callback(ptr unsafe.Pointer, buffer unsafe.Pointer, size C.longlong) C.longlong {
...
sh := &reflect.SliceHeader{
Data: uintptr(buffer),
Len: int(size),
Cap: int(size),
}
buf := *(*[]byte)(unsafe.Pointer(sh))
err := CopyToSlice(buf)
if err != nil {
log.Fatal("failed to copy to slice")
}
...
}
https://pkg.go.dev/unsafe#go1.19.4#Pointer
Pointer represents a pointer to an arbitrary type. There are four
special operations available for type Pointer that are not available
for other types:
A pointer value of any type can be converted to a Pointer.
A Pointer can be converted to a pointer value of any type.
A uintptr can be converted to a Pointer.
A Pointer can be converted to a uintptr.
Pointer therefore allows a program to defeat the type system and read
and write arbitrary memory. It should be used with extreme care.
The following patterns involving Pointer are valid. Code not using
these patterns is likely to be invalid today or to become invalid in
the future. Even the valid patterns below come with important caveats.
Running "go vet" can help find uses of Pointer that do not conform to
these patterns, but silence from "go vet" is not a guarantee that the
code is valid.
(6) Conversion of a reflect.SliceHeader or reflect.StringHeader Data
field to or from Pointer.
As in the previous case, the reflect data structures SliceHeader and
StringHeader declare the field Data as a uintptr to keep callers from
changing the result to an arbitrary type without first importing
"unsafe". However, this means that SliceHeader and StringHeader are
only valid when interpreting the content of an actual slice or string
value.
var s string
hdr := (*reflect.StringHeader)(unsafe.Pointer(&s)) // case 1
hdr.Data = uintptr(unsafe.Pointer(p)) // case 6 (this case)
hdr.Len = n
In this usage hdr.Data is really an alternate way to refer to the
underlying pointer in the string header, not a uintptr variable
itself.
In general, reflect.SliceHeader and reflect.StringHeader should be used only as *reflect.SliceHeader and *reflect.StringHeader pointing at actual slices or strings, never as plain structs. A program should not declare or allocate variables of these struct types.
// INVALID: a directly-declared header will not hold Data as a reference.
var hdr reflect.StringHeader
hdr.Data = uintptr(unsafe.Pointer(p))
hdr.Len = n
s := *(*string)(unsafe.Pointer(&hdr)) // p possibly already lost
It looks like JimB (from the comments) hinted upon the most correct answer, though he didn't post it as an answer and he didn't include an example. The following passes go vet, staticcheck, and golangci-lint - and doesn't segfault so I think it is the correct answer.
func Callback(ptr unsafe.Pointer, buffer unsafe.Pointer, size C.longlong) C.longlong {
...
buf := unsafe.Slice((*byte)(buffer), size)
err := CopyToSlice(buf)
if err != nil {
log.Fatal("failed to copy to slice")
}
...
}
Related
I'm interfacing with C code in Go using cgo, and I need to call a C function with a pointer to the underlying value in an Interface{} object. The value will be any of the atomic primitive types (not including complex64/complex128), or string.
I was hoping I'd be able to do something like this to get the address of ptr as an unsafe.Pointer:
unsafe.Pointer(reflect.ValueOf(ptr).UnsafeAddr())
But this results in a panic due to the value being unaddressable.
A similar question to this is Take address of value inside an interface, but this question is different, as in this case it is known that the value will always be one of the types specified above (which will be at most 64 bits), and I only need to give this value to a C function. Note that there are multiple C functions, and the one that will be called varies based off of a different unrelated parameter.
I also tried to solve this using a type switch statement, however I found myself unable to get the address of the values even after the type assertion was done. I was able to assign the values to temporary copies, then get the address of those copies, but I'd rather avoid making these copies if possible.
interface{} has own struct:
type eface struct {
typ *rtype
val unsafe.Pointer
}
You have no access to rtype directly or by linking, on the other hand, even though you'll copy whole rtype, it may be changed (deprecated) at future.
But thing is that you can replace pointer types with unsafe.Pointer (it may be anything else with same size, but pointer is much idiomatic, because each type has own pointer):
type eface struct {
typ, val unsafe.Pointer
}
So, now we can get value contained in eface:
func some_func(arg interface{}) {
passed_value := (*eface)(unsafe.Pointer(&arg)).val
*(*byte)(passed_value) = 'b'
}
some_var := byte('a')
fmt.Println(string(some_var)) // 'a'
some_func(some_var)
fmt.Println(string(some_var)) // 'a', it didn't changed, just because it was copied
some_func(&some_var)
fmt.Println(string(some_var)) // 'b'
You also might see some more usages at my repo:
https://github.com/LaevusDexter/fast-cast
Sorry for my poor English.
func Encode(i interface{}) ([]byte, error) {
buffer := bytes.NewBuffer(make([]byte, 0, 1024))
// size := unsafe.Sizeof(i)
size := reflect.TypeOf(i).Size()
fmt.Println(size)
ptr := unsafe.Pointer(&i)
startAddr := uintptr(ptr)
endAddr := startAddr + size
for i := startAddr; i < endAddr; i++ {
bytePtr := unsafe.Pointer(i)
b := *(*byte)(bytePtr)
buffer.WriteByte(b)
}
return buffer.Bytes(), nil
}
func TestEncode(t *testing.T) {
test := Test{10, "hello world"}
b, _ := Encode(test)
ptr := unsafe.Pointer(&b)
newTest := *(*Test)(ptr)
fmt.Println(newTest.X)
}
I am learning how to use golang unsafe and wrote this function for encoding any object. I meet with two problems, first, dose unsafe.Sizeof(obj) always return obj's pointer size? Why it different from reflect.TypeOf(obj).Size()? Second, I want to iterate the underlying bytes of obj and convert it back to obj in TestEncode function by unsafe.Pointer(), but the object's values all corrupt, why?
First, unsafe.Sizeof returns the bytes that needs to store the type. It is a little bit tricky, but it does not mean bytes that needs to store the data.
For example, a slice, as it is well known, stores 3 4-byte ints on a 32bit machine. One uintptr for memory address of the underlying array, and two int32 for len and cap. So no matter how long a slice is or what type it is of, a slice takes always 12 bytes on a 32 bit machine. Likely, a string uses 8 bytes: 1 uintptr for address and 1 int32 for len.
As for difference between reflect.TypeOf().Size, it is about interface. reflect.TypeOf looks into the interface and gets an concrete type, and reports bytes needed about the concrete type, while unsafe.Sizeof just returns 8 for an interface type: 2 uintptr for a pointer to the data and a pointer to the method lists.
Second part is quite clear now. For one, unsafe.Pointer is taking the address of the interface, instead of the concrete type. Two, in TestEncode, unsafe.Pointer is taking address to the 12-byte slice "header". There might be other errors, but with the two mentioned, they are meaningless to spot.
Note: I avoid talking about orders of the uintptr and int32 not only because I don't know, but also becuase they are not documented, unsafe, and implentation depended.
Note 2: Conclusion: Don't try to dump memory of a Go data.
Note 3: I change everything to 32 bit becuase playground is using it, so it is easier to check.
I know that converting from a []byte to a string, or vice versa, results in a copy of the underlying array being made. This makes sense to me, from the point of view of strings being immutable.
Then I read here that two optimisations get made by the compiler in specific cases:
"The first optimization avoids extra allocations when []byte keys are used to lookup entries in map[string] collections: m[string(key)]."
This makes sense because the conversion is only scoped to the square brackets, so no risk of mutating the string there.
"The second optimization avoids extra allocations in for range clauses where strings are converted to []byte: for i,v := range []byte(str) {...}."
This makes sense because once again - no way of mutating the string here.
Also mentioned is further optimisations on the todo list (not sure which todo list is being referred to), so my question is:
Does any other such (further) optimisations exist in Go 1.6 and if so, what are they?
[]byte to string
For []byte to string conversion, the compiler generates a call to the internal runtime.slicebytetostringtmp function (link to source) when it can prove
that the string form will be discarded before the calling goroutine
could possibly modify the original slice or synchronize with another
goroutine.
runtime.slicebytetostringtmp returns a string referring to the actual []byte bytes, so it does not allocate. The comment in the function says
// First such case is a m[string(k)] lookup where
// m is a string-keyed map and k is a []byte.
// Second such case is "<"+string(b)+">" concatenation where b is []byte.
// Third such case is string(b)=="foo" comparison where b is []byte.
In short, for a b []byte:
map lookup m[string(b)] does not allocate
"<"+string(b)+"> concatenation does not allocate
string(b)=="foo" comparison does not allocate
The second optimization was implemented on 22 Jan 2015, and it is in go1.6
The third optimization was implemented on 27 Jan 2015, and it is in go1.6
So, for example, in the following:
var bs []byte = []byte{104, 97, 108, 108, 111}
func main() {
x := string(bs) == "hello"
println(x)
}
the comparison does not cause allocations in go1.6.
String to []byte
Similarly, the runtime.stringtoslicebytetmp function (link to source) says:
// Return a slice referring to the actual string bytes.
// This is only for use by internal compiler optimizations
// that know that the slice won't be mutated.
// The only such case today is:
// for i, c := range []byte(str)
so i, c := range []byte(str) does not allocate, but you already knew that.
I'm trying to manipulate processes on Windows using Go language,
and I'm starting off by reading other process' memory by using ReadProcessMemory.
However, for most of the addresses I get Error: Only part of a ReadProcessMemory or WriteProcessMemory request was completed. error. Maybe my list of arguments is wrong, but I can't find out why.
Can anyone point out what I am doing wrong here?
package main
import (
"fmt"
)
import (
windows "golang.org/x/sys/windows"
)
func main() {
handle, _ := windows.OpenProcess(0x0010, false, 6100) // 0x0010 PROCESS_VM_READ, PID 6100
procReadProcessMemory := windows.MustLoadDLL("kernel32.dll").MustFindProc("ReadProcessMemory")
var data uint = 0
var length uint = 0
for i := 0; i < 0xffffffff; i += 2 {
fmt.Printf("0x%x\n", i)
// BOOL ReadProcessMemory(HANDLE hProcess, LPCVOID lpBaseAddress, LPVOID lpBuffer, DWORD nSize, LPDWORD lpNumberOfBytesRead)
ret, _, e := procReadProcessMemory.Call(uintptr(handle), uintptr(i), uintptr(data), 2, uintptr(length)) // read 2 bytes
if (ret == 0) {
fmt.Println(" Error:", e)
} else {
fmt.Println(" Length:", length)
fmt.Println(" Data:", data)
}
}
windows.CloseHandle(handle)
}
uintptr(data) is incorrect: it takes the value from data (0 of type uint) and converts that to unitptr type — yielding the same value converted to another type — producing, on x86, a null pointer.
Note that Go is not C, and you can't really play dirty games with pointers in it, or, rather, you can, but only through using the unsafe built-in package and its Pointer type which is like void* (pointing somewhere in a data memory block) in C.
What you need is something like
import "unsafe"
var (
data [2]byte
length uint32
)
ret, _, e := procReadProcessMemory.Call(uintptr(handle), uintptr(i),
uintptr(unsafe.Pointer(&data[0])),
2, uintptr(unsafe.Pointer(&length))) // read 2 bytes
Observe what was done here:
A variable of type "array of two bytes" is declared;
The address of the first element of this array is taken;
That address is type-converted to the type unsafe.Pointer;
The obtained value is then type-converted to uintptr.
The last two steps are needed because Go features garbage collection:
In Go, when you take an address of a value in memory and store it in a variable, the GC knows about this "implicit" pointer and the value which address was taken won't be garbage-collected even if it becomes unreachable with that value holding its address being the only reference left.
Even if you make that address value lose the type information it maintains — through type-converting it to unsafe.Pointer, the new value is still considered by GC and behaves like "normal" values containing addresses — as explained above.
By type-converting such value to uintptr you make GC stop considering it as a pointer. Hence this type is there only for FFI/interop.
In other words, in
var data [2]byte
a := &data[0]
p := unsafe.Pointer(a)
i := uintptr(p)
there are only three references to the value in data: that variable itself, a and p, but not i.
You should consider these rules when dealing with calling outside code because you should never ever pass around unitptr-typed values: they're only for marshaling data to the called functions and unmarshaling it back, and have to be used "on the spot" — in the same scope as the values they are type-converted from/to.
Also observe that in Go, you can't just take the address of a variable of an integer type and supply that address to a function which expects a pointer to a memory block of an appropriate size. You have to deal with byte arrays and after the data has been written by the called function, you need to explicitly convert it to a value of the type you need. That's why there's no "type casts" in Go but only "type conversions": you can't reinterpret the data type of a value through type-conversion, with the uintptr(unsafe.Pointer) (and back) being a notable exception for the purpose of FFI/interop, and even in this case you basically convert a pointer to a pointer, just transfer it through the GC boundary.
To "serialize" and "deserialize" a value of an integer type you might use the encoding/binary standard package or hand-roll no-brainer simple functions which do bitwise shifts and or-s and so on ;-)
2015-10-05, updated as per the suggestion of James Henstridge.
Note that after the function returns, and ret signalizes there's no error
you have to check the value of the length variable.
Usually when I see a field declared on a struct it's without a pointer or a dereferenced pointer symbol *, however in several code snippets where I've seen a database field in a struct it's with a pointer dereference as you see below. Why is that necessary?
type DB struct {
*bolt.DB
}
func Open(path string, mode os.FileMode) (*DB, error) {
db, err := bolt.Open(path, mode)
if err != nil {
return nil, err
}
return &DB{db}, nil
}
or a dereferenced pointer symbol *
That is the norm, for complex non-value type, in order to avoid making a copy.
See Golang book "Pointers" for example of struct with pointer(s) in them.
return &DB{db}
That returns a pointer to the newly created DB instance.
As noted in "Can you “pin” an object in memory with Go?":
Note that, unlike in C, it's perfectly OK to return the address of a local variable; the storage associated with the variable survives after the function returns
From "Pointer/Value Subtleties":
Go is also pass by value, but it has both pointers and value types. Pointers refer to a certain memory location, and allow you to mutate the data at that location
For more, see "Best practice “returning” structs in Go?"
Use pointers for big structs or structs you'll have to change, and otherwise use values, because getting things changed by surprise via a pointer is confusing.