I want to write a small go application for processing videos and decided to use this library goav which is a FFmpeg binding for go. However, it does not support reading the video straightaway from memory. As a workaround, I decided to call the C function directly.
The signature of the function I want to call is as follows.
AVIOContext *avio_alloc_context(
unsigned char *buffer,
int buffer_size,
int write_flag,
void *opaque,
int (*read_packet)(void *opaque, uint8_t *buf, int buf_size),
int (*write_packet)(void *opaque, uint8_t *buf, int buf_size),
int64_t (*seek)(void *opaque, int64_t offset, int whence));
The go code I wrote to call this -
package video
// #include<libavformat/avio.h>
import "C"
import (
"bytes"
"github.com/giorgisio/goav/avformat"
"github.com/giorgisio/goav/avutil"
"unsafe"
)
func DecodeStream(data *bytes.Buffer) {
bufferSize := 8192
formatContext := avformat.AvformatAllocContext()
buffer := (*C.uchar)(avutil.AvMalloc(uintptr(bufferSize)))
ioContext := C.avio_alloc_context(
buffer,
C.int(bufferSize),
C.int(0),
unsafe.Pointer(data),
&[0]byte{},
&[0]byte{},
&[0]byte{},
) // <- error in this line
formatContext.SetPb((*avformat.AvIOContext)(unsafe.Pointer(ioContext)))
if formatContext.AvformatFindStreamInfo(nil) < 0 {
panic("Couldn't find stream info")
}
println("DURATION: ", formatContext.Duration())
}
However, I am getting this error runtime error: cgo argument has Go pointer to Go pointer.
I tried replacing &[0]byte{} with nil too. Can't seem to get away!
The runtime error is actually complaining about the unsafe.Pointer(data) argument. The bytes.Buffer type has internal (Go) pointers, and cgo doesn't allow passing pointers to C if those pointers point to Go objects that contain Go pointers themselves.
These rules are detailed in the cgo documentation here: https://golang.org/cmd/cgo/#hdr-Passing_pointers
The short summary is that the Go garbage collector needs to be aware when Go pointers are modified. The Go compilers make sure that compiled code appropriately informs the garbage collector, but C compilers don't know to do this. So to prevent GC problems, cgo simply doesn't allow passing pointers like this.
Related
I am using cgo to calling C method which accept struct pointer as below:
package main
/*
typedef struct Client{
struct real_client c;
} Client;
int doSomething(struct real_client *c ) {
....
}
*/
import "C"
import "fmt"
type Client struct {
client C.Client
}
func main() {
cl := Client{}
C.doSomething(&cl.client.c);
// no compile error
}
However, I get an error : cgo argument has Go pointer to Go pointer.
I am using go version go1.16.13.
Is that any way to do make it work ?
Is that any way to do make it work ?
You can set the environment variable GODEBUG to cgocheck=0.
See https://pkg.go.dev/cmd/cgo#hdr-Passing_pointers.
Keep in mind the mechanism exists to prevent C code accessing managed memory not pinned for the duration of the C function call. This can become relevant in the future when changes to Go's garbage collector pertaining memory relocation may be made.
I'm trying to call a C library from GO using cgo.
The C library has the following function:
int receive(void** data);
// I'd call it like that:
void* myptr; // myptr=null
int nbBytes = receive(&myptr);
if (nbBytes==0) { return }
// myptr has now an address to a valid buffer that contains nbBytes bytes.
// And then casting it with char* to access the data that can be anything. Example:
printf("%d", *(char*)myptr);
How can I call this receive() function from GO? Go doesn't allocate any memory, the memory address is returned via myptr and directly access from it.
receive() is a "no-copy" and writes the actual data's address into myptr. The data is then accessed using *(char*)myptr.
we can assume receive() allocates and frees the buffer, it's hidden from the lib's user
Ideally, I would read the data via []byte in go.
[Edit: you added a clarification that says this isn't just suggested. But we still don't know how the data are used afterward, from C code.]
There is not enough information in your question to answer it completely, because we don't know—the C language alone doesn't tell us—how this void ** is used. Your comments and additional code suggest (pretty strongly) that the way it's used is that receive fills in the pointer:
int receive(void **data) {
*data = <something>;
return <some value>;
}
where the angle-bracketed sections are unknown to us; to use this data from C code, we'd do just what you said:
void f(void) {
void *p;
int ret;
...
ret = receive(&p);
}
What we don't know, given this much and the (justifiable) assumptions, are:
What does the ret value indicate?
Is p always valid afterward?
How many bytes at *p are accessible?
For instance, would:
struct our_data dst;
memcpy(&dst, p, len);
be a valid way to get the bytes from p into the data-structure dst, and if so, where does the length len come from? Is it implied, e.g., do we know that because ret was not -1 (error) that p is valid and has sizeof(struct our_data) bytes available, and we need the memcpy just to make it properly aligned?
If we knew all of these things, we'd be able to do the job directly from Go:
var p unsafe.Pointer
var obj C.struct_our_data
ret := C.receive(&p)
C.memcpy(unsafe.Pointer(&obj), p, len) // or similar with copy()
although it might—depending on task etc—make more sense to just write a deserializer for the raw data, which we obtain as an array living in C memory; see, e.g., How to convert [1024]C.char to [1024]byte.
I am confused regarding the passing of Go pointers (which, to my understanding, include all pointer types as well as unsafe.Pointer) to cgo. When calling C functions with cgo, I can only provide variables of types known on the C-side, or unsafe.Pointer if it matches with a void*-typed parameter in the C-function's signature. So when "Go pointers passed to C are pinned for lifetime of call", how does Go know that what I am passing is, in fact, a Go pointer, if I am ever forced to cast it to C.some_wide_enough_uint_type or C.some_c_pointer_type beforehand? The moment it is cast, isn't the information that it is a Go pointer lost, and I run risk of the GC changing the pointer? (I can see how freeing is prevented at least, when a pointer-type reference is retained on the Go-side)
We have a project with a fair amount of working cgo code, but zero confidence in its reliability. I would like to see an example of "here is how to do it correctly" which doesn't resort to circumventing Go's memory model by using C.malloc() or such, which most examples unfortunately do.
So regardless of what "pinning the pointer for lifetime of call" actually means, I see a problem either way:
If it means that Go will pin all pointers in the entire program, I see a race condition in the time interval between casting a Go pointer to a C-type and the cgo-call actually being invoked.
If it means that Go will pin only those Go pointers which are being passed, how does it know that they are Go pointers when, at the time of calling, they can only have a C-type?
I've been reading through Go issues for half the day and am starting to feel like I'm just missing something simple. Any pointers are appreciated.
EDIT: I will try to clarify the question by providing examples.
Consider this:
/*
#include <stdio.h>
void myCFunc(void* ptr) {
printf((char*)ptr);
}
*/
import "C"
import "unsafe"
func callMyCFunc() {
goPointer := []byte("abc123\n\x00")
C.myCFunc(unsafe.Pointer(&goPointer[0]))
}
Here, Go's unsafe.Pointer-type effortlessly translates into C's void*-type, so we are happy on the C-side of things, and we should be on the Go-side also: the pointer clearly points into Go-allocated memory, so it should be trivial for Go to figure out that it should pin this pointer during the call, despite it being an unsafe one. Is this the case? If it is, without further research, I would consider this to be the preferred way to pass Go pointers to cgo. Is it?
Then, consider this:
/*
#include <stdio.h>
void myCFunc(unsigned long long int stupidlyTypedPointerVariable) {
char* pointerToHopefullyStillTheSameMemory = (char*)stupidlyTypedPointerVariable;
printf(pointerToHopefullyStillTheSameMemory);
}
*/
import "C"
import "unsafe"
func callMyCFunc() {
goPointer := []byte("abc123\n\x00")
C.myCFunc(C.ulonglong(uintptr(unsafe.Pointer(&goPointer[0]))))
}
Here, I would expect that Go won't make any guesses on whether some C.ulonglong-typed variable actually means to contain the address of a Go pointer. But am I correct?
My confusion largely arises from the fact that it's not really possible to write some code to reliably test this with.
Finally, what about this:
/*
#include <stdio.h>
void cFuncOverWhichIHaveNoControl(char* ptr) {
printf(ptr);
}
*/
import "C"
import "unsafe"
func callMyCFunc() {
goPointer := []byte("abc123\n\x00")
C.cFuncOverWhichIHaveNoControl((*C.char)(unsafe.Pointer(&goPointer[0])))
}
If I am, for whatever reason, unable to change the signature of the C-function, I must cast to *C.char. Will Go still check if the value is a Go pointer, when it already is a C pointer-type?
Looking at the section on passing pointers in the current cgo documentation, (thanks to peterSO) we find that
the term Go pointer means a pointer to memory allocated by Go
as well as that
A pointer type may hold a Go pointer or a C pointer
Thus, using uintptr and other integer (read: non-pointer) types will lose us Go's guarantee of pinning the pointer.
A uintptr is an integer, not a reference. Converting a Pointer to a uintptr creates an integer value with no pointer semantics. Even if a uintptr holds the address of some object, the garbage collector will not update that uintptr's value if the object moves, nor will that uintptr keep the object from being reclaimed.
Source: https://golang.org/pkg/unsafe/#Pointer
Regarding C pointer types such as *char/*C.char, these are only safe when the pointed data does not itself contain pointers to other memory allocated by Go. This can actually be shown by trying to trigger Go's Cgo Debug mechanism, which disallows passing a Go pointer to (or into) a value which itself contains another Go pointer:
package main
import (
"fmt"
"unsafe"
/*
#include <stdio.h>
void cFuncChar(char* ptr) {
printf("%s\n", ptr);
}
void cFuncVoid(void* ptr) {
printf("%s\n", (char*)ptr);
}
*/
"C"
)
type MyStruct struct {
Distraction [2]byte
Dangerous *MyStruct
}
func main() {
bypassDetection()
triggerDetection()
}
func bypassDetection() {
fmt.Println("=== Bypass Detection ===")
ms := &MyStruct{[2]byte{'A', 0}, &MyStruct{[2]byte{0, 0}, nil}}
C.cFuncChar((*C.char)(unsafe.Pointer(ms)))
}
func triggerDetection() {
fmt.Println("=== Trigger Detection ===")
ms := &MyStruct{[2]byte{'B', 0}, &MyStruct{[2]byte{0, 0}, nil}}
C.cFuncVoid(unsafe.Pointer(ms))
}
This will print the following:
=== Bypass Detection ===
A
=== Trigger Detection ===
panic: runtime error: cgo argument has Go pointer to Go pointer
Using *C.char bypassed the detection. Only using unsafe.Pointer will detect Go pointer to Go pointer scenarios. Unfortunately, this means we will have to have an occasional nebulous void*-parameter in the C-function's signature.
Adding for clarity: Go may very well pin the value pointed by a *C.char or such, which is safe to pass; it just (reasonably) won't make an effort to find out whether it might be something else which could contain additional pointers into memory allocated by Go. Casting to unsafe.Pointer is actually safe; casting from it is what may be dangerous.
From what research I have done there are no libraries out there to create and manage shared memory in Go. I thought it might be easy enough to pull of with some basic cgo code, and as it turns out it is. The concern that I would like to address before I move forward involves the interaction between the Go runtime and Go pointers to shared memory.
To demonstrate my concern, I created a simple Go struct
type Test struct {
X int32
Y float64
Z [5]int32
}
That I expose in two seperate Go programs, A and B along with some cgo code.
#include <sys/ipc.h>
#include <sys/shm.h>
#include <stdio.h>
#include <string.h>
int get_mem() {
key_t key = ftok("shared_mem",65);
return shmget(key,1024,0666|IPC_CREAT);
}
char * attach_mem(int id) {
return (char*) shmat(id,(void*)0,0);
}
void detach_mem(char * str) {
shmdt(str);
}
void destroy_mem(int id) {
shmctl(id,IPC_RMID,NULL);
}
void write(char * str, char * data, int len) {
memcpy(str, data, len);
}
Program A then creates a Test struct, writes it to shared memory, creates a *Test pointer called ptr to that memory, waits 10 seconds, then prints the value at ptr.
// Program A
func main() {
id := C.get_mem()
str := C.attach_mem(id)
dataLocal := Test{1, 2.0, [5]int32{3, 4, 5, 6, 7}}
C.write(str, (*C.char)(unsafe.Pointer(&dataLocal)), C.int(unsafe.Sizeof(dataLocal)))
ptr := (*Test)(unsafe.Pointer(str))
time.Sleep(10 * time.Second)
fmt.Println(ptr)
C.detach_mem(str)
C.destroy_mem(id)
}
In those 10 seconds, Program B starts, creates a *Test pointer to the shared memory, and modifies one of the fields.
// Program B
func main() {
id := C.get_mem()
str := C.attach_mem(id)
data := (*Test)(unsafe.Pointer(str))
data.X = 10
C.detach_mem(str)
}
Obviously this is not a practical IPC implementation, but it works. Program A prints
&{10 2 [3 4 5 6 7]}
My concern is that the Go runtime, namely the garbage collector, won't play nice with this implementation. Specifically, how will Go treat ptr in Program A? I know that the Go GC will move data in the heap around, but does that apply to data referenced by Go pointers that lie outside of the heap? Now that a Go pointer references this data, will it be garbage collected?
If neither of those issues apply, is this approach unsafe or otherwise considered to be bad style? If so what other alternatives are there?
I am trying to interface with some C code from Go. Using cgo, this has been relatively straight-forward until I hit this (fairly common) case: needing to pass a pointer to a structure that itself contains a pointer to some data. I cannot seem to figure out how to do this from Go without resorting to putting the creation of the structure into the C code itself, which I'd prefer not to do. Here is a snippet that illustrates the problem:
package main
// typedef struct {
// int size;
// void *data;
// } info;
//
// void test(info *infoPtr) {
// // Do something here...
// }
import "C"
import "unsafe"
func main() {
var data uint8 = 5
info := &C.info{size: C.int(unsafe.Sizeof(data)), data: unsafe.Pointer(&data)}
C.test(info)
}
While this compiles fine, trying to run it results in:
panic: runtime error: cgo argument has Go pointer to Go pointer
In my case, the data being passed to the C call doesn't persist past the call (i.e. the C code in question digs into the structure, copies what it needs, then returns).
See "Passing pointers" section in cgo docs:
Go code may pass a Go pointer to C provided the Go memory to which it points does not contain any Go pointers.
And also:
These rules are checked dynamically at runtime. The checking is controlled by the cgocheck setting of the GODEBUG environment variable. The default setting is GODEBUG=cgocheck=1, which implements reasonably cheap dynamic checks. These checks may be disabled entirely using GODEBUG=cgocheck=0. Complete checking of pointer handling, at some cost in run time, is available via GODEBUG=cgocheck=2.
If you run the snippet you've provided with:
GODEBUG=cgocheck=0 go run snippet.go
Then there is no panic. However, the correct way to go is to use C.malloc (or obtain a "C pointer" from somewhere else):
package main
// #include <stdlib.h>
// typedef struct {
// int size;
// void *data;
// } info;
//
// void test(info *infoPtr) {
// // Do something here...
// }
import "C"
import "unsafe"
func main() {
var data uint8 = 5
cdata := C.malloc(C.size_t(unsafe.Sizeof(data)))
*(*C.char)(cdata) = C.char(data)
defer C.free(cdata)
info := &C.info{size: C.int(unsafe.Sizeof(data)), data: cdata}
C.test(info)
}
It works because while regular Go pointers are not allowed, C.malloc returns a "C pointer":
Go pointer means a pointer to memory allocated by Go (such as by using the & operator or calling the predefined new function) and the term C pointer means a pointer to memory allocated by C (such as by a call to C.malloc). Whether a pointer is a Go pointer or a C pointer is a dynamic property determined by how the memory was allocated.
Note that you need to include stdlib.h to use C.free.