I'm using golang to call a Dll function like char* fn(), the dll is not written by myself and I cannot change it. Here's my code:
package main
import (
"fmt"
"syscall"
"unsafe"
)
func main() {
dll := syscall.MustLoadDLL("my.dll")
fn := dll.MustFindProc("fn")
r, _, _ := fn.Call()
p := (*byte)(unsafe.Pointer(r))
// define a slice to fill with the p string
data := make([]byte, 0)
// loop until find '\0'
for *p != 0 {
data = append(data, *p) // append 1 byte
r += unsafe.Sizeof(byte(0)) // move r to next byte
p = (*byte)(unsafe.Pointer(r)) // get the byte value
}
name := string(data) // convert to Golang string
fmt.Println(name)
}
I have some questions:
Is there any better way of doing this? There're hundred of dll functions like this, I'll have to write the loop for all functions.
For very-long-string like 100k+ bytes, will append() cause performance issue?
Solved. the unsafe.Pointer(r) causes linter govet shows warning possible misuse of unsafe.Pointer, but the code runs fine, how to avoid this warning? Solution: This can be solved by adding -unsafeptr=false to govet command line, for vim-ale, add let g:ale_go_govet_options = '-unsafeptr=false'.
Casting uintptr as upointer is haram.
You must read the rules:
https://golang.org/pkg/unsafe/#Pointer
But there's hacky way, that shouldn't produce warning:
//go:linkname gostringn runtime.gostringn
func gostringn(p uintptr, l int) string
//go:linkname findnull runtime.findnull
//go:nosplit
func findnull(s uintptr) int
// ....
name := gostringn(r, findnull(r))
Functions takes pointer, but we link them from runtime as uintptr because they have same sizeof.
Might work in theory. But is also frowned upon.
Getting back to your code, as JimB said, you could do it one line with:
name := C.GoString((*C.char)(unsafe.Pointer(r)))
I got the following solution by tracking the os.Args of the go source code, But I am based on go1.17. If you are in another version, you can read the source code to solve it.
func UintPtrToString(r uintptr) string {
p := (*uint16)(unsafe.Pointer(r))
if p == nil {
return ""
}
n, end, add := 0, unsafe.Pointer(p), unsafe.Sizeof(*p)
for *(*uint16)(end) != 0 {
end = unsafe.Add(end, add)
n++
}
return string(utf16.Decode(unsafe.Slice(p, n)))
}
Related
I've found that function in the golang's source code and want to know whether it's truly a perfect hash function or not.
Is it the correct way to test that?
package main
import (
"fmt"
"strconv"
"unsafe"
)
//go:linkname strhash runtime.strhash
func strhash(p unsafe.Pointer, h uintptr) uintptr
const seed = 666
func main() {
m := make(map[uintptr]string)
for i := 0; i < 1000000000; i++ {
key := strconv.Itoa(i)
hash := strhash(unsafe.Pointer(&key), seed)
_, exist := m[hash]
if exist {
fmt.Println("collision")
break
}
m[hash] = key
}
fmt.Println("finish")
}
As far as I know/can tell, it is not. It uses the AES instructions to create the hash. You might want to check out something like https://github.com/cespare/mph.
I have this function to convert string to slice of bytes without copying
func StringToByteUnsafe(s string) []byte {
strh := (*reflect.StringHeader)(unsafe.Pointer(&s))
var sh reflect.SliceHeader
sh.Data = strh.Data
sh.Len = strh.Len
sh.Cap = strh.Len
return *(*[]byte)(unsafe.Pointer(&sh))
}
That works fine, but with very specific setup gives very strange behavior:
The setup is here: https://github.com/leviska/go-unsafe-gc/blob/main/pkg/pkg_test.go
What happens:
Create a byte slice
Convert it into temporary (rvalue) string and with unsafe convert it into byte slice again
Then, copy this slice (by reference)
Then, do something with the second slice inside goroutine
Print the pointers before and after
And I have this output on my linux mint laptop with go 1.16:
go test ./pkg -v -count=1
=== RUN TestSomething
0xc000046720 123 0xc000046720 123
0xc000076f20 123 0xc000046721 z
--- PASS: TestSomething (0.84s)
PASS
ok github.com/leviska/go-unsafe-gc/pkg 0.847s
So, the first slice magically changes its address, while the second isn't
If we remove the goroutine with runtime.GC() (and may be play with the code a little bit), we can get the both pointers to change the value (to the same one).
If we change the unsafe cast to just []byte() everything works without changing the addresses. Also, if we change it to the unsafe cast from here https://stackoverflow.com/a/66218124/5516391 everything works the same.
func StringToByteUnsafe(str string) []byte { // this works fine
var buf = *(*[]byte)(unsafe.Pointer(&str))
(*reflect.SliceHeader)(unsafe.Pointer(&buf)).Cap = len(str)
return buf
}
I run it with GOGC=off and got the same result. I run it with -race and got no errors.
If you run this as main package with main function, it seems to work correctly. Also if you remove the Convert function. My guess is that compiler optimizes stuff in this cases.
So, I have several questions about this:
What the hell is happening? Looks like a weird UB
Why and how go runtime magically changes the address of the variable?
Why in concurentless case it can change both addresses, while in concurrent can't?
What's the difference between this unsafe cast and the cast from stackoverflow answer? Why it does work?
Or is this just a compiler bug?
A copy of the full code from github, you need to put it in some package and run as test:
import (
"fmt"
"reflect"
"sync"
"testing"
"unsafe"
)
func StringToByteUnsafe(s string) []byte {
strh := (*reflect.StringHeader)(unsafe.Pointer(&s))
var sh reflect.SliceHeader
sh.Data = strh.Data
sh.Len = strh.Len
sh.Cap = strh.Len
return *(*[]byte)(unsafe.Pointer(&sh))
}
func Convert(s []byte) []byte {
return StringToByteUnsafe(string(s))
}
type T struct {
S []byte
}
func Copy(s []byte) T {
return T{S: s}
}
func Mid(a []byte, b []byte) []byte {
fmt.Printf("%p %s %p %s\n", a, a, b, b)
wg := sync.WaitGroup{}
wg.Add(1)
go func() {
b = b[1:2]
wg.Done()
}()
wg.Wait()
fmt.Printf("%p %s %p %s\n", a, a, b, b)
return b
}
func TestSomething(t *testing.T) {
str := "123"
a := Convert([]byte(str))
b := Copy(a)
Mid(a, b.S)
}
Answer from the github issue https://github.com/golang/go/issues/47247
The backing store of a is allocated on stack, because it does not
escape. And goroutine stacks can move dynamically. b, on the other
hand, escapes to heap, because it is passed to another goroutine. In
general, we don't assume the address of an object don't change.
This works as intended.
And my version is incorrect because
it uses reflect.SliceHeader as plain struct. You can run go vet on it,
and go vet will warn you.`
I've had difficulty learning the basics of reflect, pointers and interface in go, so here's another entry level question I can't seem to figure out.
This code does what I want it to do - I'm using reflect to add another record to a slice that's typed as an interface.
package main
import (
"reflect"
"log"
)
type Person struct {
Name string
}
func Add(slice interface{}) {
s := reflect.ValueOf(slice).Elem()
// in my actual code, p is declared via the use of reflect.New([Type])
p := Person{Name:"Sam"}
s.Set(reflect.Append(s,reflect.ValueOf(p)))
}
func main() {
p := []Person{}
Add(&p)
log.Println(p)
}
If I changed the Add and main function to this, things don't work the way I want it to.
func Add(slice interface{}) {
s := reflect.ValueOf(&slice).Elem()
p := Person{Name:"Sam"}
s.Set(reflect.Append(reflect.ValueOf(slice),reflect.ValueOf(p)))
log.Println(s)
}
func main() {
p := []Person{}
Add(p)
log.Println(p)
}
That is, the log.Println(p) at the end doesn't show a slice with the record Sam in it like the way I had hoped. So my question is whether it's possible for me to have Add() receive a slice that is not a pointer, and for me to still write some code in Add() that will produce the outcome shown in my first scenario?
A lot of my recent questions dance around this kind of subject, so it's still taking me a while to figure out how to use the reflect package effectively.
No, it's not possible to append to a slice in a function without passing in a pointer to the slice. This isn't related to reflection, but to how variables are passed in to functions. Here's the same code, modified to not use reflection:
package main
import (
"log"
)
type Person struct {
Name string
}
func AddWithPtr(slicep interface{}) {
sp := slicep.(*[]Person)
// This modifies p1 itself, since *sp IS p1
*sp = append(*sp, Person{"Sam"})
}
func Add(slice interface{}) {
// s is now a copy of p2
s := slice.([]Person)
sp := &s
// This modifies a copy of p2 (i.e. s), not p2 itself
*sp = append(*sp, Person{"Sam"})
}
func main() {
p1 := []Person{}
// This passes a reference to p1
AddWithPtr(&p1)
log.Println("Add with pointer: ", p1)
p2 := []Person{}
// This passes a copy of p2
Add(p2)
log.Println("Add without pointer:", p2)
}
(Above, when it says 'copy' of the slice, it doesn't mean the copy of the underlying data - just the slice)
When you pass in a slice, the function effectively gets a new slice that refers to the same data as the original. Appending to the slice in the function increases the length of the new slice, but doesn't change the length of the original slice that was passed in. That's why the original slice remains unchanged.
In go there are functions which return two values or more values, commonly one is an error. Suppose that I want to store the first return value into an already initialized variable, but I would like to initialize the variable to contain the error inline. Is there a way to do this?
For example, say I had this code
var a int
//This code doesn't compile because err doesn't exist
a, err = SomeFuncWithTwoReturnValues()
//This code doesn't compile either
a, err := SomeFuncWithTwoReturnValues()
I know you could do this, but I was hoping there was a way to do it all inline
var a int
var err error
a, err = SomeFuncWithTwoReturnValues()
or
a, err := SomeFuncWithTwoReturnValues()
EDIT: The code above actually compiles, so I looked back at my code to drill down more and have created a quick sample that actually replicates the problem (not just in my mind...).
package main
func myfunc() (int, int) {
return 1, 1
}
func main() {
a := make([]int, 1)
a[0], b := myfunc()
a[0] = b
}
Compiler says main.go|9| non-name a[0] on left side of :=. If I make it = instead of := though then b is never created. I get the feeling that there is not shorthand way to do it though.
As you've mentioned in the comments, you'll need to use the = operator in order to assign to a variable you've already declared. The := operator is used to simultaneously declare and assign a variable. The two are the same:
var x int
x = 5
//is the same as
x := 5
This solution will at least compile:
package main
func myfunc() (int, int) {
return 1, 1
}
func main() {
var b int
a := make([]int, 1)
a[0], b = myfunc()
a[0] = b
}
To answer your question, I don't think there is a way to simultaneously use an undeclared and a declared variable when returning multiple values. That would be trying to use two different operators simultaneously.
Edit: just saw your example from the code that compiles, so it appears you're already familiar with go's assignment operators. I'll leave the example up anyway.
Golang is not a very consistent language. This is a good example. At the beginning I was confused and it would be much simpler if they would always allow the := operator. The compiler is smart enough to detect already declared variables:
package main
import "fmt"
func testFunc() (int,error) {
return 42,fmt.Errorf("Test Error")
}
func main() {
number1,err := testFunc() // OK
number2,err := testFunc() // OK, even if err is already defined
number1,err = testFunc() // OK
// number1,err := testFunc() // ERROR: no new variables on left side of :=
fmt.Println(number1,number2,err)
}
Playground Link: https://play.golang.org/p/eZVB-kG6RtX
It's not consistent, because golang allows you to use := for already declared variables if you assign to them while also introducing a new variable. So the compiler can detect that variables already exists and skip their declaration. But the golang developers decided to allow that only if you introduce at least one new value. The last example shows that.
I ran into this situation like this:
package main
import "os"
func main() {
var cache struct { dir string }
// undefined: err
cache.dir, err = os.UserCacheDir()
// non-name cache.dir on left side of :=
cache.dir, err := os.UserCacheDir()
if err != nil {
panic(err)
}
println(cache.dir)
}
as you discovered, this issue does not have a clean solution. You can declare
an extra variable:
dir, err := os.UserCacheDir()
if err != nil {
panic(err)
}
cache := userCache{dir}
Or, while more verbose, you can declare the error beforehand. This can save
memory, as Go does not use a Rust ownership model:
var (
cache struct { dir string }
err error
)
cache.dir, err = os.UserCacheDir()
As mention in the spec, while using:=, if one of the variables is new, then the old one will just be assigned with the new data.
Unlike regular variable declarations, a short variable declaration may redeclare variables provided they were originally declared earlier in the same block (or the parameter lists if the block is the function body) with the same type, and at least one of the non-blank variables is new. As a consequence, redeclaration can only appear in a multi-variable short declaration. Redeclaration does not introduce a new variable; it just assigns a new value to the original.
field1, offset := nextField(str, 0)
field2, offset := nextField(str, offset) // redeclares offset
As mentioned by the other answers you cannot use assignment and declaration in the same return statement. You have to use either.
However I guess the main reason for your question is cleaning up the code so you don't have to declare an extra err variable above the method or function statement.
You can solve this in two ways:
Declare a global var err error variable and use it in the assignment:
var err error
func MyFunc(someInput string) {
var a int
a, err = someOtherFunction()
}
If your method or function returns an error you can use the declared return variable
func MyFunc(someInput string) (err error) {
var a int
a, err = someOtherFunction()
return
}
I mainly have the problem in methods when I want to assign something to a struct member, e.g.:
type MyStruct struct {
so string
}
func (m *MyStruct) SomeMethod() (err error) {
m.so, err = SomeFunction()
// handle error and continue or return it
return
}
in the main package i have:
var foo C.int
foo = 3
t := fastergo.Ctuner_new()
fastergo.Ctuner_register_parameter(t, &foo, 0, 100, 1)
in the fastergo package i have:
func Ctuner_register_parameter(tuner unsafe.Pointer, parameter *C.int, from C.int, to C.int, step C.int) C.int {
...
}
if i try to run it, i get:
demo.go:14[/tmp/go-build742221968/command-line-arguments/_obj/demo.cgo1.go:21]: cannot use &foo (type *_Ctype_int) as type *fastergo._Ctype_int in function argument
i am not really sure what go is trying to tell me here, but somehow i think it wants to tell me, that all C.int are not equal? why is this the case? how can i solve this / work around?
Since _Ctype_int doesn't begin with a Unicode upper case letter, the type is local to the package. Use Go types, except in the C wrapper package where you convert them to C types. The wrapper package should hide all the implementation details.
You don't provide sufficient information for us to create sample code which compiles and runs. Here's a rough outline of what I expected to see:
package main
import "tuner"
func main() {
var foo int
foo = 3
t := tuner.New()
t.RegisterParameter(&foo, 0, 100, 1)
}
.
package tuner
import (
"unsafe"
)
/*
#include "ctuner.h"
*/
import "C"
type Tuner struct {
ctuner uintptr
}
func New() *Tuner {
var t Tuner
t.ctuner = uintptr(unsafe.Pointer(C.ctuner_new()))
return &t
}
func (t *Tuner) RegisterParameter(parameter *int, from, to, step int) error {
var rv C.int
rv = C.ctuner_register_parameter(
(*C.ctuner)(unsafe.Pointer(t.ctuner)),
(*C.int)(unsafe.Pointer(parameter)),
C.int(from),
C.int(to),
C.int(step),
)
if rv != 0 {
// handle error
}
return nil
}
As explained by peterSO, you can't pass C.int between packages. However, you can pass pointers between packages by converting the pointer type. To do this, you would define a named type in the target package, import that type into the calling package and covert via unsafe.Pointer. There isn't any point in doing this with a single int.
However, it is helpful if you keep code to convert complex types in a package; for example an array of strings (or any sort of nested array).
The example below is for exporting a go function to be called in C, but this works in reverse, ie. if you want to call a C functions which a returns nested array.
package convert
import "C"
type PP_char **C.char
func From_c_to_go(arr_str PP_char, length int) []string {
// Some operation on the Ctype
var slice []string
for _, s := range unsafe.Slice(arr_str, length) {
if s == nil {
break
}
x := C.GoString(s)
slice = append(slice, x)
}
return slice
}
package main
import "C"
import "convert"
//export myFunc
func myFunc(arr_str **C.char, length int){
retyped_arr_str := convert.PP_char(unsafe.Pointer(arr_str))
slice := convert.From_c_to_go(retyped_arr_str, length)
// Do something with slice
}
You could instead decide to pass instance of unsafe.Pointer as an argument to the go function in the target package and perform the type conversion in that function.