From what research I have done there are no libraries out there to create and manage shared memory in Go. I thought it might be easy enough to pull of with some basic cgo code, and as it turns out it is. The concern that I would like to address before I move forward involves the interaction between the Go runtime and Go pointers to shared memory.
To demonstrate my concern, I created a simple Go struct
type Test struct {
X int32
Y float64
Z [5]int32
}
That I expose in two seperate Go programs, A and B along with some cgo code.
#include <sys/ipc.h>
#include <sys/shm.h>
#include <stdio.h>
#include <string.h>
int get_mem() {
key_t key = ftok("shared_mem",65);
return shmget(key,1024,0666|IPC_CREAT);
}
char * attach_mem(int id) {
return (char*) shmat(id,(void*)0,0);
}
void detach_mem(char * str) {
shmdt(str);
}
void destroy_mem(int id) {
shmctl(id,IPC_RMID,NULL);
}
void write(char * str, char * data, int len) {
memcpy(str, data, len);
}
Program A then creates a Test struct, writes it to shared memory, creates a *Test pointer called ptr to that memory, waits 10 seconds, then prints the value at ptr.
// Program A
func main() {
id := C.get_mem()
str := C.attach_mem(id)
dataLocal := Test{1, 2.0, [5]int32{3, 4, 5, 6, 7}}
C.write(str, (*C.char)(unsafe.Pointer(&dataLocal)), C.int(unsafe.Sizeof(dataLocal)))
ptr := (*Test)(unsafe.Pointer(str))
time.Sleep(10 * time.Second)
fmt.Println(ptr)
C.detach_mem(str)
C.destroy_mem(id)
}
In those 10 seconds, Program B starts, creates a *Test pointer to the shared memory, and modifies one of the fields.
// Program B
func main() {
id := C.get_mem()
str := C.attach_mem(id)
data := (*Test)(unsafe.Pointer(str))
data.X = 10
C.detach_mem(str)
}
Obviously this is not a practical IPC implementation, but it works. Program A prints
&{10 2 [3 4 5 6 7]}
My concern is that the Go runtime, namely the garbage collector, won't play nice with this implementation. Specifically, how will Go treat ptr in Program A? I know that the Go GC will move data in the heap around, but does that apply to data referenced by Go pointers that lie outside of the heap? Now that a Go pointer references this data, will it be garbage collected?
If neither of those issues apply, is this approach unsafe or otherwise considered to be bad style? If so what other alternatives are there?
Related
I'm trying to call a C library from GO using cgo.
The C library has the following function:
int receive(void** data);
// I'd call it like that:
void* myptr; // myptr=null
int nbBytes = receive(&myptr);
if (nbBytes==0) { return }
// myptr has now an address to a valid buffer that contains nbBytes bytes.
// And then casting it with char* to access the data that can be anything. Example:
printf("%d", *(char*)myptr);
How can I call this receive() function from GO? Go doesn't allocate any memory, the memory address is returned via myptr and directly access from it.
receive() is a "no-copy" and writes the actual data's address into myptr. The data is then accessed using *(char*)myptr.
we can assume receive() allocates and frees the buffer, it's hidden from the lib's user
Ideally, I would read the data via []byte in go.
[Edit: you added a clarification that says this isn't just suggested. But we still don't know how the data are used afterward, from C code.]
There is not enough information in your question to answer it completely, because we don't know—the C language alone doesn't tell us—how this void ** is used. Your comments and additional code suggest (pretty strongly) that the way it's used is that receive fills in the pointer:
int receive(void **data) {
*data = <something>;
return <some value>;
}
where the angle-bracketed sections are unknown to us; to use this data from C code, we'd do just what you said:
void f(void) {
void *p;
int ret;
...
ret = receive(&p);
}
What we don't know, given this much and the (justifiable) assumptions, are:
What does the ret value indicate?
Is p always valid afterward?
How many bytes at *p are accessible?
For instance, would:
struct our_data dst;
memcpy(&dst, p, len);
be a valid way to get the bytes from p into the data-structure dst, and if so, where does the length len come from? Is it implied, e.g., do we know that because ret was not -1 (error) that p is valid and has sizeof(struct our_data) bytes available, and we need the memcpy just to make it properly aligned?
If we knew all of these things, we'd be able to do the job directly from Go:
var p unsafe.Pointer
var obj C.struct_our_data
ret := C.receive(&p)
C.memcpy(unsafe.Pointer(&obj), p, len) // or similar with copy()
although it might—depending on task etc—make more sense to just write a deserializer for the raw data, which we obtain as an array living in C memory; see, e.g., How to convert [1024]C.char to [1024]byte.
I want to write a small go application for processing videos and decided to use this library goav which is a FFmpeg binding for go. However, it does not support reading the video straightaway from memory. As a workaround, I decided to call the C function directly.
The signature of the function I want to call is as follows.
AVIOContext *avio_alloc_context(
unsigned char *buffer,
int buffer_size,
int write_flag,
void *opaque,
int (*read_packet)(void *opaque, uint8_t *buf, int buf_size),
int (*write_packet)(void *opaque, uint8_t *buf, int buf_size),
int64_t (*seek)(void *opaque, int64_t offset, int whence));
The go code I wrote to call this -
package video
// #include<libavformat/avio.h>
import "C"
import (
"bytes"
"github.com/giorgisio/goav/avformat"
"github.com/giorgisio/goav/avutil"
"unsafe"
)
func DecodeStream(data *bytes.Buffer) {
bufferSize := 8192
formatContext := avformat.AvformatAllocContext()
buffer := (*C.uchar)(avutil.AvMalloc(uintptr(bufferSize)))
ioContext := C.avio_alloc_context(
buffer,
C.int(bufferSize),
C.int(0),
unsafe.Pointer(data),
&[0]byte{},
&[0]byte{},
&[0]byte{},
) // <- error in this line
formatContext.SetPb((*avformat.AvIOContext)(unsafe.Pointer(ioContext)))
if formatContext.AvformatFindStreamInfo(nil) < 0 {
panic("Couldn't find stream info")
}
println("DURATION: ", formatContext.Duration())
}
However, I am getting this error runtime error: cgo argument has Go pointer to Go pointer.
I tried replacing &[0]byte{} with nil too. Can't seem to get away!
The runtime error is actually complaining about the unsafe.Pointer(data) argument. The bytes.Buffer type has internal (Go) pointers, and cgo doesn't allow passing pointers to C if those pointers point to Go objects that contain Go pointers themselves.
These rules are detailed in the cgo documentation here: https://golang.org/cmd/cgo/#hdr-Passing_pointers
The short summary is that the Go garbage collector needs to be aware when Go pointers are modified. The Go compilers make sure that compiled code appropriately informs the garbage collector, but C compilers don't know to do this. So to prevent GC problems, cgo simply doesn't allow passing pointers like this.
Here is my code:
helloworld.go:
package main
/*
#include <stdlib.h>
*/
import "C"
import "unsafe"
//export HelloWorld
func HelloWorld() *C.char {
cs := C.CString("Hello World!")
C.free(unsafe.Pointer(cs))
return cs
}
func main() {}
node-helloworld.cc:
#include "helloworld.h"
#include <node.h>
#include <string>
namespace demo {
using v8::FunctionCallbackInfo;
using v8::Isolate;
using v8::Local;
using v8::Object;
using v8::String;
using v8::Value;
void Method(const FunctionCallbackInfo<Value>& args) {
Isolate* isolate = args.GetIsolate();
args.GetReturnValue().Set(String::NewFromUtf8(isolate, HelloWorld()));
}
void init(Local<Object> exports) {
NODE_SET_METHOD(exports, "hello", Method);
}
NODE_MODULE(helloworld, init)
}
When I execute the code I get:
�Oc
or
��#
or
���
etc
It's actually random. I seem to be getting something different every time.
It might be that I am passing char array from HelloWorld() method.
What am I missing?
UPDATE
When I remove:
C.free(unsafe.Pointer(cs))
I get the good string. And not random characters.
But I need C.free to free the memory. It is recommended here: https://blog.golang.org/c-go-cgo
The call to C.CString returns a pointer to the start of the char
array, so before the function exits we convert it to an unsafe.Pointer
and release the memory allocation with C.free.
I am unsure how to do this.
The linked example frees the allocated memory because no other code needs it.
If your Go function needs to return some allocated memory so that it can be used by some C code then the Go function should not call C.free, instead the C code that uses that memory should be responsible for freeing it after it does not need it anymore.
Arbitrary example:
cgo/test/issue20910.go
cgo/test/issue20910.c
I am trying to interface with some C code from Go. Using cgo, this has been relatively straight-forward until I hit this (fairly common) case: needing to pass a pointer to a structure that itself contains a pointer to some data. I cannot seem to figure out how to do this from Go without resorting to putting the creation of the structure into the C code itself, which I'd prefer not to do. Here is a snippet that illustrates the problem:
package main
// typedef struct {
// int size;
// void *data;
// } info;
//
// void test(info *infoPtr) {
// // Do something here...
// }
import "C"
import "unsafe"
func main() {
var data uint8 = 5
info := &C.info{size: C.int(unsafe.Sizeof(data)), data: unsafe.Pointer(&data)}
C.test(info)
}
While this compiles fine, trying to run it results in:
panic: runtime error: cgo argument has Go pointer to Go pointer
In my case, the data being passed to the C call doesn't persist past the call (i.e. the C code in question digs into the structure, copies what it needs, then returns).
See "Passing pointers" section in cgo docs:
Go code may pass a Go pointer to C provided the Go memory to which it points does not contain any Go pointers.
And also:
These rules are checked dynamically at runtime. The checking is controlled by the cgocheck setting of the GODEBUG environment variable. The default setting is GODEBUG=cgocheck=1, which implements reasonably cheap dynamic checks. These checks may be disabled entirely using GODEBUG=cgocheck=0. Complete checking of pointer handling, at some cost in run time, is available via GODEBUG=cgocheck=2.
If you run the snippet you've provided with:
GODEBUG=cgocheck=0 go run snippet.go
Then there is no panic. However, the correct way to go is to use C.malloc (or obtain a "C pointer" from somewhere else):
package main
// #include <stdlib.h>
// typedef struct {
// int size;
// void *data;
// } info;
//
// void test(info *infoPtr) {
// // Do something here...
// }
import "C"
import "unsafe"
func main() {
var data uint8 = 5
cdata := C.malloc(C.size_t(unsafe.Sizeof(data)))
*(*C.char)(cdata) = C.char(data)
defer C.free(cdata)
info := &C.info{size: C.int(unsafe.Sizeof(data)), data: cdata}
C.test(info)
}
It works because while regular Go pointers are not allowed, C.malloc returns a "C pointer":
Go pointer means a pointer to memory allocated by Go (such as by using the & operator or calling the predefined new function) and the term C pointer means a pointer to memory allocated by C (such as by a call to C.malloc). Whether a pointer is a Go pointer or a C pointer is a dynamic property determined by how the memory was allocated.
Note that you need to include stdlib.h to use C.free.
Suppose I'm interacting with a third party C library from Go that maintains some complicated data structure. It allows me to iterate over the data structure using an API that looks something like this:
typedef void (*callback)(double x, void *params);
iterate(int64_t start_idx, int64_t end_idx, callback f, void *params);
With the idea being that iterate loops over every index between start_idx and end_idx and calls f on every element in that range. If you wanted to read the data out of the complicated data structure and into an array, you would write something like this:
typedef struct buffer {
double *data;
int64_t i;
} buffer;
void read_callback(double x, void *params) {
buffer *buf = (buffer*) params;
buf->data[i] = x;
buf->i++;
}
Now let's suppose that I wanted to wrap this API call in a Go function where the user passes the function a pre-allocated buffer. In Go 1.5, I might have done something like this:
func Read(startIdx, endIdx int64, data []float64) {
buf := &C.buffer{}
buf.data = unsafe.Pointer(&data[0])
C.iterate(C.int64_t(startIdx), C.int64_t(endIdx),
(C.callback)(C.read_callback), unsafe.Pointer(buf))
}
However, in Go 1.6 this is invalid. data is a Go-pointer and so is buf, meaning that the runtime panics to prevent a GC error. Allocating buf as a C-pointer is also not allowed. I think the intended way to handle something like this would be to allocate data as a C pointer, but I don't want to allocate a temporary array inside Read because these arrays are large enough that I can't hold two of them in memory at once (and even if I could, I wouldn't be able to deal with the heap fragmentation).
My current (very hacky) solutions are to either pass some of the data around as global variables or to pack i and the array length in the first element of array (and then to swap some data around at the end of the iteration). Is there a way for me to do this which is less terrible?