I need to create a string in Go that is 1048577 characters (1MB + 1 byte). The content of the string is totally unimportant. Is there a way to allocate this directly without concatenating or using buffers?
Also, it's worth noting that the value of string will not change. It's for a unit test to verify that strings that are too long will return an error.
Use strings.Builder to allocate a string without using extra buffers.
var b strings.Builder
b.Grow(1048577)
for i := 0; i < 1048577; i++ {
b.WriteByte(0)
}
s := b.String()
The call to the Grow method allocates a slice with capacity 1048577. The WriteByte calls fill the slice to capacity. The String() method uses unsafe to convert that slice to a string.
The cost of the loop can be reduced by writing chunks of N bytes at a time and filling single bytes at the end.
If you are not opposed to using the unsafe package, then use this:
p := make([]byte, 1048577)
s := *(*string)(unsafe.Pointer(&p))
If you are asking about how to do this with the simplest code, then use the following:
s := string(make([]byte, 1048577)
This approach does not meet the requirements set forth in the question. It uses an extra buffer instead of allocating the string directly.
I ended up using this:
string(make([]byte, 1048577))
https://play.golang.org/p/afPukPc1Esr
What is the easiest method to create an empty buffer of size n in Go using bytes.NewBuffer()?
Adding some additional info here. The quick way to create a new buffer is briefly mentioned at the end of the doc string:
b := new(bytes.Buffer)
or
b := &bytes.Buffer{}
The Buffer struct define includes a 64 byte internal bootstrap field that is initially used for small allocations. Once the default size is exceeded, a byte slice Buffer.buf is created and internally maintained.
As #leafbebop suggested we can pre-initalize the buf field of the Buffer struct using a new slice.
b := bytes.NewBuffer(make([]byte,0,N))
I also found another option to use the Grow() method:
b := new(bytes.Buffer)
b.Grow(n)
Also it's interesting to point out that the internal buf slice will grow at a rate of cap(buf)*2 + n. This means that if you've written 1MB into a buffer and then add 1 byte, your cap() will increase to 2097153 bytes.
I have read on the golang blog: https://blog.golang.org/go-maps-in-action that:
var m map[string]int
Map types are reference types, like pointers or slices, and so the
value of m above is nil; it doesn't point to an initialized map. A nil
map behaves like an empty map when reading, but attempts to write to a
nil map will cause a runtime panic; don't do that. To initialize a
map, use the built in make function:
m = make(map[string]int)
The make function allocates and initializes a hash map data structure
and returns a map value that points to it.
I have a hard time understanding some parts of this:
What does var m map[string]int do?
Why do I need to write m = make(map[string]int) but not i = make(int)
What does var m map[string]int do?
It tells the compiler that m is a variable of type map[string]int, and assigns "The Zero Value" of the type map[string] int to m (that's why m is nil as nil is The Zero Value of any map).
Why do I need to write m = make(map[string]int) but not i = make(int)
You don't need to. You can create a initialized map also like this:
m = map[string]int{}
which does exactly the same.
The difference between maps and ints is: A nil map is perfectly fine. E.g. len() of a nil map works and is 0. The only thing you cannot do with a nil map is store key-value-pairs. If you want to do this you'll have to prepare/initialize the map. This preparation/initialization in Go is done through the builtin make (or by a literal map as shown above). This initialization process is not needed for ints. As there are no nil ints this initialization would be total noise.
Note that you do not initialize the variable m: The variable m is a map of strings to ints, initialized or not. Like i is a variable for ints. Now ints are directly usable while maps require one more step because the language works that way.
What does var m map[string]int do?
You can think about it like pointer with nil value, it does not point to anything yet but able to point to concrete value.
Why do I need to write m = make(map[string]int) but not i = make(int)
https://golang.org/doc/effective_go.html#allocation_make
Back to allocation. The built-in function make(T, args) serves a purpose different from new(T). It creates slices, maps, and channels only, and it returns an initialized (not zeroed) value of type T (not *T). The reason for the distinction is that these three types represent, under the covers, references to data structures that must be initialized before use. A slice, for example, is a three-item descriptor containing a pointer to the data (inside an array), the length, and the capacity, and until those items are initialized, the slice is nil. For slices, maps, and channels, make initializes the internal data structure and prepares the value for use. For instance,
make([]int, 10, 100)
allocates an array of 100 ints and then creates a slice structure with length 10 and a capacity of 100 pointing at the first 10 elements of the array. (When making a slice, the capacity can be omitted; see the section on slices for more information.) In contrast, new([]int) returns a pointer to a newly allocated, zeroed slice structure, that is, a pointer to a nil slice value.
These examples illustrate the difference between new and make.
var p *[]int = new([]int) // allocates slice structure; *p == nil; rarely useful
var v []int = make([]int, 100) // the slice v now refers to a new array of 100 ints
// Unnecessarily complex:
var p *[]int = new([]int)
*p = make([]int, 100, 100)
// Idiomatic:
v := make([]int, 100)
Remember that make applies only to maps, slices and channels and does not return a pointer. To obtain an explicit pointer allocate with new or take the address of a variable explicitly.
All words have the same length of 32 bits (4 bytes) or 64 bits (8 bytes),
depending on the processor and the operating system. They are identified by their memory address (represented as a hexadecimal number).
All variables of primitive types like int, float, bool, string ... are value types, they point directly to the values contained in the memory. Also composite types like arrays and structs are value types. When assigning with = the value of a value type to another variable: j = i, a copy of the original value i is made in memory.
More complex data which usually needs several words are treated as reference types. A reference type variable r1 contains the address (a number) of the memory location where the value of r1 is stored (or at least the 1st word of it):
For reference types when assigning r2 = r1, only the reference (the address) is copied and not the value!!. If the value of r1 is modified, all references of that value (like r1 and r2) will be reflected.
In Go pointers are reference types, as well as slices, maps and channels. The variables that are referenced are stored in the heap, which is garbage collected.
In the light of the above statements it's clear why the article states:
To initialize a map, use the built in make function.
The make function allocates and initializes a hash map data structure and returns a map value that points to it. This means you can write into it, compare to
var m map[string]int
which is readable, resulting a nil map, but an attempt to write to a nil map will cause a runtime panic. This is the reason why it's important to initialize the map with make.
m = make(map[string]int)
I have this code:
func my_function(hash string) [16]byte {
b, _ := hex.DecodeString(hash)
return b // Compile error: fails since [16]byte != []byte
}
b will be of type []byte. I know that hash is of length 32. How can I make my code above work? Ie. can I somehow cast from a general-length byte array to a fixed-length byte array? I am not interested in allocating 16 new bytes and copying the data over.
There is no direct method to convert a slice to an array. You can however do a copy.
var ret [16]byte
copy(ret[:], b)
The standard library uses []byte and if you insist on using something else you will just have a lot more typing to do. I wrote a program using arrays for my md5 values and regretted it.
I'm pretty new to Go. I want to know the exact difference between arrays and slices so please forgive me if I'm too obsessive over the spec.
In documentation for package fmt, it says for Printf that %x can accept strings and slices of bytes. It said nothing about arrays of bytes. But in fact, if I put an array of bytes in it, it prints out fine too!
package main
import (
"fmt"
)
func main() {
var b [6]byte
for i := 0; i < 6; i++ {
b[i] = 'a'
}
fmt.Printf("%s\n", b) // "aaaaaa"
}
Can anyone please tell me why? AFAIK, bytes array and slices cannot implicitly convert to each other.
Thanks for reading my question!
You are correct that you can't implicitly convert slices and arrays.
A slice is a small data structure with 3 elements which are a pointer to some memory, the length of the memory, and the capacity of the memory.
However an array is just a block of memory.
Go passes everything by value in functions meaning that when you call Printf with the array, the whole of the memory of the array is passed to the function. When you call it with a slice, the whole of the slice structure is passed to Printf. However the slice structure contains a pointer to the underlying memory so this is very like passing a pointer to an array.
Printf uses lots of introspection to print its values. All the arguments to Printf are converted to interface{} types which means they are boxed up with a type in a small structure. Printf then examines these interface values, inspects the type and works out how to print the values. For most % types that Printf undestands it can take quite a few types and it attempts to print them in a user friendly way.
For example %x can take an int which is printed in hex, or a slice, or an array as you've discovered.
So in summary
array is contiguous memory
slice is a structure containing a pointer to contiguous memory
an interface value is a small structure containing a pointer to a type and the value
Printf takes interface{} values
It introspects the interface{} values to print lots of different things
I hope that explanation is helpful!