I am new to Go, and found it is OK to return the address of a local variable defined in a function. That is obviously not possible in C since local variable is in stack.
So I am just wondering why it is OK to do it in Go? In Go, the local variable is in heap? Will it affect performance since allocating heap memory is quite expensive than stack? Is it possible allocate local variable in stack in Go? Or actually is there stack memory in Go?
There's a very clear answer to that question in the FAQ:
How do I know whether a variable is allocated on the heap or the
stack?
From a correctness standpoint, you don't need to know. Each variable
in Go exists as long as there are references to it. The storage
location chosen by the implementation is irrelevant to the semantics
of the language.
The storage location does have an effect on writing efficient
programs. When possible, the Go compilers will allocate variables that
are local to a function in that function's stack frame. However, if
the compiler cannot prove that the variable is not referenced after
the function returns, then the compiler must allocate the variable on
the garbage-collected heap to avoid dangling pointer errors. Also, if
a local variable is very large, it might make more sense to store it
on the heap rather than the stack.
In the current compilers, if a variable has its address taken, that
variable is a candidate for allocation on the heap. However, a basic
escape analysis recognizes some cases when such variables will not
live past the return from the function and can reside on the stack.
TLDR: You shouldn't care. Go takes care of allocation for you.
Does gc Go (specifically go1.11) pre-allocates a chunk of memory and take from it for each allocation (like JVM), or it allocates every time a variable is created, and is it a kernel call (malloc)?
If it is one kernel call per allocation, that would make variable creation expensive. How can I force allocation on the stack/heap?
This is covered in various places, like the FAQ:
How do I know whether a variable is allocated on the heap or the stack?
From a correctness standpoint, you don't need to know. Each variable
in Go exists as long as there are references to it. The storage
location chosen by the implementation is irrelevant to the semantics
of the language.
The storage location does have an effect on writing efficient
programs. When possible, the Go compilers will allocate variables that
are local to a function in that function's stack frame. However, if
the compiler cannot prove that the variable is not referenced after
the function returns, then the compiler must allocate the variable on
the garbage-collected heap to avoid dangling pointer errors. Also, if
a local variable is very large, it might make more sense to store it
on the heap rather than the stack.
In the current compilers, if a variable has its address taken, that
variable is a candidate for allocation on the heap. However, a basic
escape analysis recognizes some cases when such variables will not
live past the return from the function and can reside on the stack.
Go's memory allocation is carefully optimized for its needs, for example with a custom malloc. I suspect you have a slightly different underlying question/problem that you're struggling with - it would be better to ask that instead. If this is just exploration/curiosity, you'll have to make your question much more specific.
I've written a password manager in Ocaml. In order to make it as secure as possible, I'd like to store a string (an encryption key) in memory in such a way that it can be overwritten. Since Ocaml is pass by value , and there's a garbage collector, this has proven difficult. I encrypt all buffers and variables I can, but I still need a "session key" stored to do this. To prevent this from being detected by automated key searching programs or put into swap, it's assembled from a bunch of random data in a buffer using a random increment. So really, all I need is a single variable that can be overwritten for the assembled key for a few seconds before it's passed into the Nocrypto library... Would a reference work for this?
According to this cornell "Refs and Arrays" page, refs are mutable and work similarly to pointers in C. That being said, I also found a stack overflow answer discussing Ocaml refs, in which the answer mentions "they act like pointers to new allocated memory". Does this mean every time, it just allocates a new thing in memory instead of actually mutating the stuff in memory? If so, you couldn't really "overwrite" a ref.
Other possible solutions I've come across are Bigarrays, and "custom blocks". I'm not entirely sure if "custom blocks" are actually allocated outside of the scope of garbage collection or not. They seem like they're used to access external C code. Are they copied around by the garbage collector? Could they be "overwritten?" There's also this idea of "opaque bytes" and opaque objects in memory. I'm having a pretty hard time wrapping my head around how this all fits together. A useful but confusing (to me) discussion of custom blocks in memory on stack overflow is here: Are custom blocks ever copied in memory? Answer says they can be moved around. Even so, could they be overwritten?
The last possible solution is to store it using a Cstruct like the Nocrypto library seems to do. They discuss it in this github issue: Secret material erasure The asker states:
"Granted, most key material is Cstruct.t, which is a Bigarray.Array1.t, which is allocated outside of the GC heap"
Is this even correct? If so, I can't seem to find a source file that actually does this. I'm pretty new to Ocaml and functional programming in general. If you're curious, my program is located on github here: ocaml-pass
TL;DR;
You shall not store any secret information in OCaml heap. Thus you must never copy your secret into any OCaml heap-allocated value, consequently, neither Bytes, nor Strings or Arrays could be used, even temporary.
Introduction to the OCaml Memory Model
OCaml values are uniformly represented as tagged machine words. The least significant bit of a word is used as a tag, that distinguishes between pointers (tag=0) and immediate values (tag=1). Thus a value has always a fixed size, and is a pointer or an immediate.
Immediate values store their data in the most significant part of the word, that is 31-bits in 32 bit systems, and 63 bits in 64-bit systems. Pointers store their data in blocks, that are located in a so-called OCaml Heap. The OCaml Heap is a set of blocks managed by the Garbage Collector (GC). A block is a chunk of data prefixed with a header. The header specifies the size of data, and some other meta information, used by the GC. Block can contain OCaml values (pointers or immediate values) or opaque data.
To summarize. All OCaml values are represented as machine words, that either store data directly in the word or are pointers to heap-allocated blocks. Each pointer points to one and only one block. Several pointers may point to the same block. Such values are considered physically equal. Some blocks are not pointed by any pointers. Such blocks are called dead and are reclaimed by the GC.
Introduction to the OCaml Garbage Collector
The GC manages blocks, by allocating, moving, and deallocating them. The GC itself uses an arena, that is either obtained from the C memory allocator (malloc) or directly from a kernel via the memmap syscall (depends on a particular system and runtime).
The GC is generational, that means that values are first allocated in a special region of a heap called minor heap. The minor heap is a contiguous region of memory of fixed size, represented in the runtime with three pointers: the pointer beg to the beginning of the minor heap, a pointer end to the end of the minor heap, and the pointer cur to the beginning of the free area of the minor heap. When a block is allocated, cur is increased by the size of the block. Then the block is initialized with data. When there is no more free space in the minor heap (i.e., then end - cur is less than the required block size), then a minor GC cycle is triggered. The GC analyzes all blocks stored in the Minor Heap and copies all blocks that are referenced by at least one pointer to the Major Heap. After that, the cur pointer is set to beg.
In the Major Heap, a block may also be copied several times during a process called compaction. The compactor may try to rearrange blocks in its arena in order to achieve more compact representation of the heap.
Security Consequences
As the OCaml GC is a moving GC, it may copy the heap-allocated data arbitrarily. Although it is called moving, it is still in fact just copying. I.e., when a block is moved from the minor heap to the major heap, it is in fact just bit-copied, and thus is duplicated. The block phantom in the minor heap may live for an arbitrary amount of time until it is overwritten by some newly allocated value. When an object is moved during the compaction, it is also copied, and may or may not be overwritten during the process. And, of course, it goes without saying, that once a block becomes dead, it still may survive in a heap for an arbitrary amount of time until reused by the GC.
That all means, that if a secret ends up in the OCaml heap, it will go wild, as the GC can replicate it multiple times in an arbitrary and rather unpredictable ways. Thus, we can only store secrets either in immediate values or in regions that are not controlled by the GC. As it was said before, all OCaml values that are pointers, always point to a block in the OCaml heap. A block may contain data directly, or it could contain a pointer itself, that will point outside the memory heap. The so-called custom blocks, may or may not store their information in the OCaml heap, it depeds on a particular representation of each custom block. For example, the Bigarray library provides custom blocks that store their payload outside of the OCaml heap. Thus a Bigarray is a custom block, that has two fields: a pointer and size. It is an opaque block, i.e., the GC will never treat these two values as OCaml values, and will never follow neither the size nor the pointer. The data pointed by a pointer is located outside of the OCaml heap, and is either allocated by malloc or by memmap (in fact, it could be arbitrary integer, and even point to stack, or static data, it doesn't really matter, as long as we treat bigarrays just as a ptr,len pair).
This all makes Bigarrays ideal for storing secrets. We can be sure, that they are not moved by the GC, we can overwrite them to prevent the information leakage once they are freed.
Further considerations
We should be careful, and never allow a secret to be copied into the OCaml heap from our safe place. That means, that even if our main storage is a safe bigarray the information will still leak if we will copy its contents to an OCaml string. Consequently, if we first read the information into OCaml string, and then copy it into bigarray, the information will still leak. Thus, any interface that uses OCaml heap-allocated values is unsafe and shall not be used. For example, we can't use OCaml channels to read or write secrets (we should rely on memory mapping or unbuffered IO provided by the Unix module). And again, whenever you get a string data type from a Bigarray, you get your data copied, with all the ramifications.
I would use a value of type bytes, essentially a mutable array of bytes:
# let buffer = Bytes.make 16 'x';;
val buffer : bytes = "xxxxxxxxxxxxxxxx"
# Bytes.set buffer 0 'T';;
- : unit = ()
# buffer;;
- : bytes = "Txxxxxxxxxxxxxxx"
# Bytes.fill buffer 0 16 ' ';;
- : unit = ()
# buffer;;
- : bytes = " "
You can overwrite with Bytes.fill after you're done.
I am new to Go, and found it is OK to return the address of a local variable defined in a function. That is obviously not possible in C since local variable is in stack.
So I am just wondering why it is OK to do it in Go? In Go, the local variable is in heap? Will it affect performance since allocating heap memory is quite expensive than stack? Is it possible allocate local variable in stack in Go? Or actually is there stack memory in Go?
There's a very clear answer to that question in the FAQ:
How do I know whether a variable is allocated on the heap or the
stack?
From a correctness standpoint, you don't need to know. Each variable
in Go exists as long as there are references to it. The storage
location chosen by the implementation is irrelevant to the semantics
of the language.
The storage location does have an effect on writing efficient
programs. When possible, the Go compilers will allocate variables that
are local to a function in that function's stack frame. However, if
the compiler cannot prove that the variable is not referenced after
the function returns, then the compiler must allocate the variable on
the garbage-collected heap to avoid dangling pointer errors. Also, if
a local variable is very large, it might make more sense to store it
on the heap rather than the stack.
In the current compilers, if a variable has its address taken, that
variable is a candidate for allocation on the heap. However, a basic
escape analysis recognizes some cases when such variables will not
live past the return from the function and can reside on the stack.
TLDR: You shouldn't care. Go takes care of allocation for you.
Let's say I have an allocation in memory containing a string, "ABCDEFG", but I only have a pointer to the 'E'. Is it possible, on win32, to free that block, given a pointer that is within the block, but not at the start? Any allocation method would work, but a Heap* function would be the path of least resistance.
If not a native solution, have there been any custom memory managers written which offer this feature?
EDIT: This isn't an excuse to be sloppy. I'm developing an automatic memory management system using 100% compile-time metadata. This odd requirement seems to be the only thing standing in the way of getting it working, and even then it's needed only for data types based on arrays (which are slicable).
It would be possible for the memory allocation routines in the runtime library to check a given memory address against the beginning and end of every allocated block. That search accomplished, it would be easy to release the block from the beginning.
Even with clever algorithms behind it, this would incur some kind of search with each memory deallocation. And why? Just to support erroneous programs too stupid to keep track of the beginning of the blocks of memory they allocated?
The standard C idiom thrives on treating blocks of allocated memory like arrays. The pointer returned from *alloc is a pointer to the beginning of an array, and the pointer can be used with subscripts to access any element of that array, subscripts starting at 0. This has worked well enough for 40 years that I can't think of a sensible reason to introduce a change here.
I suppose if you know what the malloc() guard blocks look like, you could write a function that backs up from the pointer you pass it until it finds a 'best guess' of the original memory address and then calls free(). Why not just keep a copy of the base pointer around?
If you use VirtualAlloc to allocate memory, you can use VirtualQuery to figure out which block a pointer belongs to. Once you have the base address, you can pass this to VirtualFree to free the entire block.