Please consider this sample go code below:
package main
/*
#include <stdio.h>
#include <stdlib.h>
*/
import "C"
import (
"fmt"
"runtime"
"unsafe"
)
func main() {
// Convert Go string to C string using C.CString
cString := C.CString("Wold!")
fmt.Printf("C.CString type: %T\n", cString)
//C.free(unsafe.Pointer(cString)) // <-- this works, but I don't want to free it manually..
runtime.SetFinalizer(&cString, func(t *C.Char) {
C.free(unsafe.Pointer(t))
})
}
I am experimenting with cGo, and trying to free cString. When I try to free my variable cString using runtime.SetFinalizer I encounter:
$ go build a.go
# command-line-arguments
./a.go:22:41: could not determine kind of name for C.Char
Please point me the correct direction. Thanks!
When the cgo system is turning your wrapper into something the Go compiler understands, it has to translate each of the C types to a Go type for various purposes. It turns out this doesn't work for your case (this is the error you saw).
That's actually OK, because your code would never have worked the way you wanted in the first place. A runtime finalizer runs when Go's garbage collector is ready to release a Go object that occupies Go memory, but C.Cstring returns a pointer that is not Go memory. In particular, note the following quote from the cgo documentation:
// Go string to C string
// The C string is allocated in the C heap using malloc.
// It is the caller's responsibility to arrange for it to be
// freed, such as by calling C.free (be sure to include stdlib.h
// if C.free is needed).
func C.CString(string) *C.char
Since the returned string is on the "C heap" it will never be finalized by the Go garbage collector. Had your code compiled, it would have just been a no-op.
If you have a Go object whose lifetime parallels that of the C object, you could perhaps use that. Here's a made-up (but working) example:
package main
/*
#include <stdio.h>
#include <stdlib.h>
*/
import "C"
import (
"fmt"
"runtime"
"time"
"unsafe"
)
type S struct {
Foo int
ToFree unsafe.Pointer
}
func main() {
doit()
runtime.GC()
time.Sleep(10 * time.Millisecond) // ugly hack
}
func doit() {
cString := C.CString("Wold!")
fmt.Printf("C.CString type: %T\n", cString)
x := &S{Foo: 1, ToFree: unsafe.Pointer(cString)}
runtime.SetFinalizer(x, func(t *S) {
fmt.Println("freeing C string")
C.free(t.ToFree)
})
}
When the allocated object for x goes out of scope it becomes eligible for GC. The actual GC may never happen, so I forced one with runtime.GC() in main. This triggers the finalizer:
$ ./cfree_example
C.CString type: *main._Ctype_char
freeing C string
The "ugly hack" is in there because if main returns before the finalizer call has finished writing the freeing C string messages, it gets lost. In a real program you would not need this.
Related
I am using cgo to calling C method which accept struct pointer as below:
package main
/*
typedef struct Client{
struct real_client c;
} Client;
int doSomething(struct real_client *c ) {
....
}
*/
import "C"
import "fmt"
type Client struct {
client C.Client
}
func main() {
cl := Client{}
C.doSomething(&cl.client.c);
// no compile error
}
However, I get an error : cgo argument has Go pointer to Go pointer.
I am using go version go1.16.13.
Is that any way to do make it work ?
Is that any way to do make it work ?
You can set the environment variable GODEBUG to cgocheck=0.
See https://pkg.go.dev/cmd/cgo#hdr-Passing_pointers.
Keep in mind the mechanism exists to prevent C code accessing managed memory not pinned for the duration of the C function call. This can become relevant in the future when changes to Go's garbage collector pertaining memory relocation may be made.
I am confused regarding the passing of Go pointers (which, to my understanding, include all pointer types as well as unsafe.Pointer) to cgo. When calling C functions with cgo, I can only provide variables of types known on the C-side, or unsafe.Pointer if it matches with a void*-typed parameter in the C-function's signature. So when "Go pointers passed to C are pinned for lifetime of call", how does Go know that what I am passing is, in fact, a Go pointer, if I am ever forced to cast it to C.some_wide_enough_uint_type or C.some_c_pointer_type beforehand? The moment it is cast, isn't the information that it is a Go pointer lost, and I run risk of the GC changing the pointer? (I can see how freeing is prevented at least, when a pointer-type reference is retained on the Go-side)
We have a project with a fair amount of working cgo code, but zero confidence in its reliability. I would like to see an example of "here is how to do it correctly" which doesn't resort to circumventing Go's memory model by using C.malloc() or such, which most examples unfortunately do.
So regardless of what "pinning the pointer for lifetime of call" actually means, I see a problem either way:
If it means that Go will pin all pointers in the entire program, I see a race condition in the time interval between casting a Go pointer to a C-type and the cgo-call actually being invoked.
If it means that Go will pin only those Go pointers which are being passed, how does it know that they are Go pointers when, at the time of calling, they can only have a C-type?
I've been reading through Go issues for half the day and am starting to feel like I'm just missing something simple. Any pointers are appreciated.
EDIT: I will try to clarify the question by providing examples.
Consider this:
/*
#include <stdio.h>
void myCFunc(void* ptr) {
printf((char*)ptr);
}
*/
import "C"
import "unsafe"
func callMyCFunc() {
goPointer := []byte("abc123\n\x00")
C.myCFunc(unsafe.Pointer(&goPointer[0]))
}
Here, Go's unsafe.Pointer-type effortlessly translates into C's void*-type, so we are happy on the C-side of things, and we should be on the Go-side also: the pointer clearly points into Go-allocated memory, so it should be trivial for Go to figure out that it should pin this pointer during the call, despite it being an unsafe one. Is this the case? If it is, without further research, I would consider this to be the preferred way to pass Go pointers to cgo. Is it?
Then, consider this:
/*
#include <stdio.h>
void myCFunc(unsigned long long int stupidlyTypedPointerVariable) {
char* pointerToHopefullyStillTheSameMemory = (char*)stupidlyTypedPointerVariable;
printf(pointerToHopefullyStillTheSameMemory);
}
*/
import "C"
import "unsafe"
func callMyCFunc() {
goPointer := []byte("abc123\n\x00")
C.myCFunc(C.ulonglong(uintptr(unsafe.Pointer(&goPointer[0]))))
}
Here, I would expect that Go won't make any guesses on whether some C.ulonglong-typed variable actually means to contain the address of a Go pointer. But am I correct?
My confusion largely arises from the fact that it's not really possible to write some code to reliably test this with.
Finally, what about this:
/*
#include <stdio.h>
void cFuncOverWhichIHaveNoControl(char* ptr) {
printf(ptr);
}
*/
import "C"
import "unsafe"
func callMyCFunc() {
goPointer := []byte("abc123\n\x00")
C.cFuncOverWhichIHaveNoControl((*C.char)(unsafe.Pointer(&goPointer[0])))
}
If I am, for whatever reason, unable to change the signature of the C-function, I must cast to *C.char. Will Go still check if the value is a Go pointer, when it already is a C pointer-type?
Looking at the section on passing pointers in the current cgo documentation, (thanks to peterSO) we find that
the term Go pointer means a pointer to memory allocated by Go
as well as that
A pointer type may hold a Go pointer or a C pointer
Thus, using uintptr and other integer (read: non-pointer) types will lose us Go's guarantee of pinning the pointer.
A uintptr is an integer, not a reference. Converting a Pointer to a uintptr creates an integer value with no pointer semantics. Even if a uintptr holds the address of some object, the garbage collector will not update that uintptr's value if the object moves, nor will that uintptr keep the object from being reclaimed.
Source: https://golang.org/pkg/unsafe/#Pointer
Regarding C pointer types such as *char/*C.char, these are only safe when the pointed data does not itself contain pointers to other memory allocated by Go. This can actually be shown by trying to trigger Go's Cgo Debug mechanism, which disallows passing a Go pointer to (or into) a value which itself contains another Go pointer:
package main
import (
"fmt"
"unsafe"
/*
#include <stdio.h>
void cFuncChar(char* ptr) {
printf("%s\n", ptr);
}
void cFuncVoid(void* ptr) {
printf("%s\n", (char*)ptr);
}
*/
"C"
)
type MyStruct struct {
Distraction [2]byte
Dangerous *MyStruct
}
func main() {
bypassDetection()
triggerDetection()
}
func bypassDetection() {
fmt.Println("=== Bypass Detection ===")
ms := &MyStruct{[2]byte{'A', 0}, &MyStruct{[2]byte{0, 0}, nil}}
C.cFuncChar((*C.char)(unsafe.Pointer(ms)))
}
func triggerDetection() {
fmt.Println("=== Trigger Detection ===")
ms := &MyStruct{[2]byte{'B', 0}, &MyStruct{[2]byte{0, 0}, nil}}
C.cFuncVoid(unsafe.Pointer(ms))
}
This will print the following:
=== Bypass Detection ===
A
=== Trigger Detection ===
panic: runtime error: cgo argument has Go pointer to Go pointer
Using *C.char bypassed the detection. Only using unsafe.Pointer will detect Go pointer to Go pointer scenarios. Unfortunately, this means we will have to have an occasional nebulous void*-parameter in the C-function's signature.
Adding for clarity: Go may very well pin the value pointed by a *C.char or such, which is safe to pass; it just (reasonably) won't make an effort to find out whether it might be something else which could contain additional pointers into memory allocated by Go. Casting to unsafe.Pointer is actually safe; casting from it is what may be dangerous.
Here is my code:
helloworld.go:
package main
/*
#include <stdlib.h>
*/
import "C"
import "unsafe"
//export HelloWorld
func HelloWorld() *C.char {
cs := C.CString("Hello World!")
C.free(unsafe.Pointer(cs))
return cs
}
func main() {}
node-helloworld.cc:
#include "helloworld.h"
#include <node.h>
#include <string>
namespace demo {
using v8::FunctionCallbackInfo;
using v8::Isolate;
using v8::Local;
using v8::Object;
using v8::String;
using v8::Value;
void Method(const FunctionCallbackInfo<Value>& args) {
Isolate* isolate = args.GetIsolate();
args.GetReturnValue().Set(String::NewFromUtf8(isolate, HelloWorld()));
}
void init(Local<Object> exports) {
NODE_SET_METHOD(exports, "hello", Method);
}
NODE_MODULE(helloworld, init)
}
When I execute the code I get:
�Oc
or
��#
or
���
etc
It's actually random. I seem to be getting something different every time.
It might be that I am passing char array from HelloWorld() method.
What am I missing?
UPDATE
When I remove:
C.free(unsafe.Pointer(cs))
I get the good string. And not random characters.
But I need C.free to free the memory. It is recommended here: https://blog.golang.org/c-go-cgo
The call to C.CString returns a pointer to the start of the char
array, so before the function exits we convert it to an unsafe.Pointer
and release the memory allocation with C.free.
I am unsure how to do this.
The linked example frees the allocated memory because no other code needs it.
If your Go function needs to return some allocated memory so that it can be used by some C code then the Go function should not call C.free, instead the C code that uses that memory should be responsible for freeing it after it does not need it anymore.
Arbitrary example:
cgo/test/issue20910.go
cgo/test/issue20910.c
I am trying to interface with some C code from Go. Using cgo, this has been relatively straight-forward until I hit this (fairly common) case: needing to pass a pointer to a structure that itself contains a pointer to some data. I cannot seem to figure out how to do this from Go without resorting to putting the creation of the structure into the C code itself, which I'd prefer not to do. Here is a snippet that illustrates the problem:
package main
// typedef struct {
// int size;
// void *data;
// } info;
//
// void test(info *infoPtr) {
// // Do something here...
// }
import "C"
import "unsafe"
func main() {
var data uint8 = 5
info := &C.info{size: C.int(unsafe.Sizeof(data)), data: unsafe.Pointer(&data)}
C.test(info)
}
While this compiles fine, trying to run it results in:
panic: runtime error: cgo argument has Go pointer to Go pointer
In my case, the data being passed to the C call doesn't persist past the call (i.e. the C code in question digs into the structure, copies what it needs, then returns).
See "Passing pointers" section in cgo docs:
Go code may pass a Go pointer to C provided the Go memory to which it points does not contain any Go pointers.
And also:
These rules are checked dynamically at runtime. The checking is controlled by the cgocheck setting of the GODEBUG environment variable. The default setting is GODEBUG=cgocheck=1, which implements reasonably cheap dynamic checks. These checks may be disabled entirely using GODEBUG=cgocheck=0. Complete checking of pointer handling, at some cost in run time, is available via GODEBUG=cgocheck=2.
If you run the snippet you've provided with:
GODEBUG=cgocheck=0 go run snippet.go
Then there is no panic. However, the correct way to go is to use C.malloc (or obtain a "C pointer" from somewhere else):
package main
// #include <stdlib.h>
// typedef struct {
// int size;
// void *data;
// } info;
//
// void test(info *infoPtr) {
// // Do something here...
// }
import "C"
import "unsafe"
func main() {
var data uint8 = 5
cdata := C.malloc(C.size_t(unsafe.Sizeof(data)))
*(*C.char)(cdata) = C.char(data)
defer C.free(cdata)
info := &C.info{size: C.int(unsafe.Sizeof(data)), data: cdata}
C.test(info)
}
It works because while regular Go pointers are not allowed, C.malloc returns a "C pointer":
Go pointer means a pointer to memory allocated by Go (such as by using the & operator or calling the predefined new function) and the term C pointer means a pointer to memory allocated by C (such as by a call to C.malloc). Whether a pointer is a Go pointer or a C pointer is a dynamic property determined by how the memory was allocated.
Note that you need to include stdlib.h to use C.free.
I want to malloc large number of objects in to memory.(about 100 million objects) because the gc of golang is not effective enough,so i need to use c/c++ to malloc memory and use std::vector to hold objects.
this is my code,i want use std container in cgo:
package main
import (
"fmt"
)
/*
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <vector>
using namespace std;
void dosome(){
vector<int> ivec; // empty vector
for (vector<int>::size_type ix = 0; ix != 10; ++ix)
ivec[ix] = ix; // disaster: ivec has no elements
}
*/
// #cgo LDFLAGS: -lstdc++
import "C"
//import "fmt"
func main() {
C.dosome()
var input string
fmt.Scanln(&input)
}
and have error message below:
go run stddemo.go
# command-line-arguments
./stddemo.go:13:10: fatal error: 'vector' file not found
#include <vector>
^
1 error generated.
how can i set the include path or is there another idea?
While you can use C++ with CGo, you can't embed that code inside the .go file, since it ultimately gets built with a C compiler.
Instead, place your dosome function in a separate .cpp file in the same directory as the .go file, and declare your function to use C linkage. For example:
extern "C" {
void dosome() {
vector<int> ivec;
...
}
}
If you include a prototype for the function in the CGo comment in the .go file so you can call it from Go.
Since you have multiple files now, you can't use the go run foo.go shorthand any more (since it only compiles a single file). Instead, you will need to use go run package or go build package, where your code is located at $GOPATH/src/package.
Uhh I think your conclusions are a bit too fast. GC cost is driven by two things: The more garbage your program produces, the more the GC will have to run. Second: The more pointers there are to scan, the longer a single GC will take.
That is to say: as long as you put your 100 million things into a go slice and keep them there: the GC won't have to run much, because there's no garbage. And second: if your things don't contain pointers, GC time, should it still occur, will be fine.
So, my question is: do your things have pointers?
if you just want to call someone's golang code this is a quick ram inefficient way:
package main
import "C"
import "fmt"
import "unsafe"
func intArrayFromC (src unsafe.Pointer, sz int) []uint64 {
dest := make([]uint64, sz)
copy(dest, (*(*[1000000000]uint64)(unsafe.Pointer(src)))[:sz:sz])// big number dose not affect ram.
return dest
}
//export doPrint
func doPrint(src unsafe.Pointer, sz int){
var numbers []uint64 = intArrayFromC(src, sz);
for i := 0; i < len(numbers); i++ {
fmt.Printf("%d index: %d\n", numbers[i], i)
}
}
func main() {}
and the c++ code:
#include "print.h"
#include <string.h>
#include <vector>
int main() {
std::vector<GoUint64> numbers{99,44,11,00,2,33,44};
while (1) {
doPrint(numbers.data(), numbers.size());
}
return 0;
}