Related
I'm trying to implement a string shuffle function in Go that uses crypto/rand instead of math/rand. The Fisher-Yates Shuffle requires random integers so I've tried to implement that functionality, without having to use crypto/rand Int which relies on math/big. Below is the best I've come up with so far but is there a better method? The fact that I can't find existing examples leads me to wonder if there's a good reason why nobody does this!
package main
import "crypto/rand"
import "fmt"
import "encoding/binary"
func randomInt(max int) int {
var n uint16
binary.Read(rand.Reader, binary.LittleEndian, &n)
return int(n) % max
}
func shuffle(s *[]string) {
slice := *s
for i := range slice {
j := randomInt(i + 1)
slice[i], slice[j] = slice[j], slice[i]
}
*s = slice
}
func main() {
slice := []string{"a", "b", "c", "d", "e", "f", "h", "i", "j", "k"}
shuffle(&slice)
fmt.Println(slice)
}
Go's math/rand library has good facilities for producing random numerical primitives from a Source.
// A Source represents a source of uniformly-distributed
// pseudo-random int64 values in the range [0, 1<<63).
type Source interface {
Int63() int64
Seed(seed int64)
}
NewSource(seed int64) returns the builtin, deterministic PRNG, but New(source Source) will allow anything that satisfies the Source interface.
Here is an example of a Source that is backed by crypto/rand.
type CryptoRandSource struct{}
func NewCryptoRandSource() CryptoRandSource {
return CryptoRandSource{}
}
func (_ CryptoRandSource) Int63() int64 {
var b [8]byte
rand.Read(b[:])
// mask off sign bit to ensure positive number
return int64(binary.LittleEndian.Uint64(b[:]) & (1<<63 - 1))
}
func (_ CryptoRandSource) Seed(_ int64) {}
You can use it like this:
r := rand.New(NewCryptoRandSource())
for i := 0; i < 10; i++ {
fmt.Println(r.Int())
}
The math/rand library has a properly implemented Intn() method which ensures a uniform distribution.
func (r *Rand) Intn(n int) int {
if n <= 0 {
panic("invalid argument to Intn")
}
if n <= 1<<31-1 {
return int(r.Int31n(int32(n)))
}
return int(r.Int63n(int64(n)))
}
func (r *Rand) Int31n(n int32) int32 {
if n <= 0 {
panic("invalid argument to Int31n")
}
if n&(n-1) == 0 { // n is power of two, can mask
return r.Int31() & (n - 1)
}
max := int32((1 << 31) - 1 - (1<<31)%uint32(n))
v := r.Int31()
for v > max {
v = r.Int31()
}
return v % n
}
func (r *Rand) Int63n(n int64) int64 {
if n <= 0 {
panic("invalid argument to Int63n")
}
if n&(n-1) == 0 { // n is power of two, can mask
return r.Int63() & (n - 1)
}
max := int64((1 << 63) - 1 - (1<<63)%uint64(n))
v := r.Int63()
for v > max {
v = r.Int63()
}
return v % n
}
Cryptographic hash functions also can be wrapped as a Source for alternate means of randomness.
The numbers from n % max are not distributed uniformly. For example,
package main
import (
"fmt"
"math"
)
func main() {
max := 7
size := math.MaxUint8
count := make([]int, size)
for i := 0; i < size; i++ {
count[i%max]++
}
fmt.Println(count[:max])
}
Output:
[37 37 37 36 36 36 36]
Based on the comments received, I think I can improve on the example in my question by adding a uniformInt function, populating a uint32 instead of a uint16 and removing the pointer to the slice.
package main
import "crypto/rand"
import "fmt"
import "encoding/binary"
func randomInt() int {
var n uint32
binary.Read(rand.Reader, binary.LittleEndian, &n)
return int(n)
}
func uniformInt(max int) (r int) {
divisor := 4294967295 / max // Max Uint32
for {
r = randomInt() / divisor
if r <= max {
break
}
}
return
}
func shuffle(slice []string) {
for i := range slice {
j := uniformInt(i + 1)
slice[i], slice[j] = slice[j], slice[i]
}
}
func main() {
slice := []string{"a", "b", "c", "d", "e", "f", "h", "i", "j", "k"}
shuffle(slice)
fmt.Println(slice)
}
I have a function which receives a []byte but what I have is an int, what is the best way to go about this conversion ?
err = a.Write([]byte(myInt))
I guess I could go the long way and get it into a string and put that into bytes, but it sounds ugly and I guess there are better ways to do it.
I agree with Brainstorm's approach: assuming that you're passing a machine-friendly binary representation, use the encoding/binary library. The OP suggests that binary.Write() might have some overhead. Looking at the source for the implementation of Write(), I see that it does some runtime decisions for maximum flexibility.
func Write(w io.Writer, order ByteOrder, data interface{}) error {
// Fast path for basic types.
var b [8]byte
var bs []byte
switch v := data.(type) {
case *int8:
bs = b[:1]
b[0] = byte(*v)
case int8:
bs = b[:1]
b[0] = byte(v)
case *uint8:
bs = b[:1]
b[0] = *v
...
Right? Write() takes in a very generic data third argument, and that's imposing some overhead as the Go runtime then is forced into encoding type information. Since Write() is doing some runtime decisions here that you simply don't need in your situation, maybe you can just directly call the encoding functions and see if it performs better.
Something like this:
package main
import (
"encoding/binary"
"fmt"
)
func main() {
bs := make([]byte, 4)
binary.LittleEndian.PutUint32(bs, 31415926)
fmt.Println(bs)
}
Let us know how this performs.
Otherwise, if you're just trying to get an ASCII representation of the integer, you can get the string representation (probably with strconv.Itoa) and cast that string to the []byte type.
package main
import (
"fmt"
"strconv"
)
func main() {
bs := []byte(strconv.Itoa(31415926))
fmt.Println(bs)
}
Check out the "encoding/binary" package. Particularly the Read and Write functions:
binary.Write(a, binary.LittleEndian, myInt)
Sorry, this might be a bit late. But I think I found a better implementation on the go docs.
buf := new(bytes.Buffer)
var num uint16 = 1234
err := binary.Write(buf, binary.LittleEndian, num)
if err != nil {
fmt.Println("binary.Write failed:", err)
}
fmt.Printf("% x", buf.Bytes())
i thought int type has any method for getting int hash to bytes, but first i find math / big method for this
https://golang.org/pkg/math/big/
var f int = 52452356235; // int
var s = big.NewInt(int64(f)) // int to big Int
var b = s.Bytes() // big Int to bytes
// b - byte slise
var r = big.NewInt(0).SetBytes(b) // bytes to big Int
var i int = int(r.Int64()) // big Int to int
https://play.golang.org/p/VAKSGw8XNQq
However, this method uses an absolute value.
If you spend 1 byte more, you can transfer the sign
func IntToBytes(i int) []byte{
if i > 0 {
return append(big.NewInt(int64(i)).Bytes(), byte(1))
}
return append(big.NewInt(int64(i)).Bytes(), byte(0))
}
func BytesToInt(b []byte) int{
if b[len(b)-1]==0 {
return -int(big.NewInt(0).SetBytes(b[:len(b)-1]).Int64())
}
return int(big.NewInt(0).SetBytes(b[:len(b)-1]).Int64())
}
https://play.golang.org/p/mR5Sp5hu4jk
or new(https://play.golang.org/p/7ZAK4QL96FO)
(The package also provides functions for fill into an existing slice)
https://golang.org/pkg/math/big/#Int.FillBytes
Adding this option for dealing with basic uint8 to byte[] conversion
foo := 255 // 1 - 255
ufoo := uint16(foo)
far := []byte{0,0}
binary.LittleEndian.PutUint16(far, ufoo)
bar := int(far[0]) // back to int
fmt.Println("foo, far, bar : ",foo,far,bar)
output :
foo, far, bar : 255 [255 0] 255
Here is another option, based on the Go source code [1]:
package main
import (
"encoding/binary"
"fmt"
"math/bits"
)
func encodeUint(x uint64) []byte {
buf := make([]byte, 8)
binary.BigEndian.PutUint64(buf, x)
return buf[bits.LeadingZeros64(x) >> 3:]
}
func main() {
for x := 0; x <= 64; x += 8 {
buf := encodeUint(1<<x-1)
fmt.Println(buf)
}
}
Result:
[]
[255]
[255 255]
[255 255 255]
[255 255 255 255]
[255 255 255 255 255]
[255 255 255 255 255 255]
[255 255 255 255 255 255 255]
[255 255 255 255 255 255 255 255]
Much faster than math/big:
BenchmarkBig-12 28348621 40.62 ns/op
BenchmarkBit-12 731601145 1.641 ns/op
https://github.com/golang/go/blob/go1.16.5/src/encoding/gob/encode.go#L113-L117
You can try musgo_int. All you need to do is to cast your variable:
package main
import (
"github.com/ymz-ncnk/musgo_int"
)
func main() {
var myInt int = 1234
// from int to []byte
buf := make([]byte, musgo_int.Int(myInt).SizeMUS())
musgo_int.Int(myInt).MarshalMUS(buf)
// from []byte to int
_, err := (*musgo_int.Int)(&myInt).UnmarshalMUS(buf)
if err != nil {
panic(err)
}
}
Convert Integer to byte slice.
import (
"bytes"
"encoding/binary"
"log"
)
func IntToBytes(num int64) []byte {
buff := new(bytes.Buffer)
bigOrLittleEndian := binary.BigEndian
err := binary.Write(buff, bigOrLittleEndian, num)
if err != nil {
log.Panic(err)
}
return buff.Bytes()
}
Maybe the simple way is using protobuf, see the Protocol Buffer Basics: Go
define message like
message MyData {
int32 id = 1;
}
get more in Defining your protocol format
// Write
out, err := proto.Marshal(mydata)
read more in Writing a Message
Try math/big package to convert bytes array to int and to convert int to bytes array.
package main
import (
"fmt"
"math/big"
)
func main() {
// Convert int to []byte
var int_to_encode int64 = 65535
var bytes_array []byte = big.NewInt(int_to_encode).Bytes()
fmt.Println("bytes array", bytes_array)
// Convert []byte to int
var decoded_int int64 = new(big.Int).SetBytes(bytes_array).Int64()
fmt.Println("decoded int", decoded_int)
}
This is the most straight forward (and shortest (and safest) (and maybe most performant)) way:
buf.Bytes() is of type bytes slice.
var val uint32 = 42
buf := new(bytes.Buffer)
err := binary.Write(buf, binary.LittleEndian, val)
if err != nil {
fmt.Println("binary.Write failed:", err)
}
fmt.Printf("% x\n", buf.Bytes())
see also https://stackoverflow.com/a/74819602/589493
What's wrong with converting it to a string?
[]byte(fmt.Sprintf("%d", myint))
I'm using levigo, the leveldb bindings for Go. My keys are int64's and need to be kept sorted. By default, leveldb uses a bytewise comparator so I'm trying to use varint encoding.
func i2b(x int64) []byte {
b := make([]byte, binary.MaxVarintLen64)
n := binary.PutVarint(b, x)
return key[:n]
}
My keys are not being sorted correctly. I wrote the following as a test.
var prev int64 = 0
for i := int64(1); i < 1e5; i++ {
if bytes.Compare(i2b(i), i2b(prev)) <= 0 {
log.Fatalf("bytewise: %d > %d", b2i(prev), i)
}
prev = i
}
output: bytewise: 127 > 128
playground
I'm not sure where the problem is. Am I doing the encoding wrong? Is varint not the right encoding to use?
EDIT:
BigEndian fixed width encoding is bytewise comparable
func i2b(x int64) []byte {
b := make([]byte, 8)
binary.BigEndian.PutUint64(b, uint64(x))
return b
}
The varint encoding is not bytewise comparable* wrt to the order of the values it caries. One option how to write the ordering/collating function (cmp bellow) is for example:
package main
import (
"encoding/binary"
"log"
)
func i2b(x int64) []byte {
var b [binary.MaxVarintLen64]byte
return b[:binary.PutVarint(b[:], x)]
}
func cmp(a, b []byte) int64 {
x, n := binary.Varint(a)
if n < 0 {
log.Fatal(n)
}
y, n := binary.Varint(b)
if n < 0 {
log.Fatal(n)
}
return x - y
}
func main() {
var prev int64 = 0
for i := int64(1); i < 1e5; i++ {
if cmp(i2b(i), i2b(prev)) <= 0 {
log.Fatal("fail")
}
prev = i
}
}
Playground
(*) The reason is (also) the bit fiddling performed.
I need to read [100]byte to transfer a bunch of string data.
Because not all of the strings are precisely 100 characters long, the remaining part of the byte array is padded with 0s.
If I convert [100]byte to string by: string(byteArray[:]), the tailing 0s are displayed as ^#^#s.
In C, the string will terminate upon 0, so what's the best way to convert this byte array to string in Go?
Methods that read data into byte slices return the number of bytes read. You should save that number and then use it to create your string. If n is the number of bytes read, your code would look like this:
s := string(byteArray[:n])
To convert the full string, this can be used:
s := string(byteArray[:len(byteArray)])
This is equivalent to:
s := string(byteArray[:])
If for some reason you don't know n, you could use the bytes package to find it, assuming your input doesn't have a null character embedded in it.
n := bytes.Index(byteArray[:], []byte{0})
Or as icza pointed out, you can use the code below:
n := bytes.IndexByte(byteArray[:], 0)
Use:
s := string(byteArray[:])
Simplistic solution:
str := fmt.Sprintf("%s", byteArray)
I'm not sure how performant this is though.
For example,
package main
import "fmt"
func CToGoString(c []byte) string {
n := -1
for i, b := range c {
if b == 0 {
break
}
n = i
}
return string(c[:n+1])
}
func main() {
c := [100]byte{'a', 'b', 'c'}
fmt.Println("C: ", len(c), c[:4])
g := CToGoString(c[:])
fmt.Println("Go:", len(g), g)
}
Output:
C: 100 [97 98 99 0]
Go: 3 abc
The following code is looking for '\0', and under the assumptions of the question the array can be considered sorted since all non-'\0' precede all '\0'. This assumption won't hold if the array can contain '\0' within the data.
Find the location of the first zero-byte using a binary search, then slice.
You can find the zero-byte like this:
package main
import "fmt"
func FirstZero(b []byte) int {
min, max := 0, len(b)
for {
if min + 1 == max { return max }
mid := (min + max) / 2
if b[mid] == '\000' {
max = mid
} else {
min = mid
}
}
return len(b)
}
func main() {
b := []byte{1, 2, 3, 0, 0, 0}
fmt.Println(FirstZero(b))
}
It may be faster just to naively scan the byte array looking for the zero-byte, especially if most of your strings are short.
When you do not know the exact length of non-nil bytes in the array, you can trim it first:
string(bytes.Trim(arr, "\x00"))
Use this:
bytes.NewBuffer(byteArray).String()
Only use for performance tuning.
package main
import (
"fmt"
"reflect"
"unsafe"
)
func BytesToString(b []byte) string {
return *(*string)(unsafe.Pointer(&b))
}
func StringToBytes(s string) []byte {
return *(*[]byte)(unsafe.Pointer(&s))
}
func main() {
b := []byte{'b', 'y', 't', 'e'}
s := BytesToString(b)
fmt.Println(s)
b = StringToBytes(s)
fmt.Println(string(b))
}
Though not extremely performant, the only readable solution is:
// Split by separator and pick the first one.
// This has all the characters till null, excluding null itself.
retByteArray := bytes.Split(byteArray[:], []byte{0}) [0]
// OR
// If you want a true C-like string, including the null character
retByteArray := bytes.SplitAfter(byteArray[:], []byte{0}) [0]
A full example to have a C-style byte array:
package main
import (
"bytes"
"fmt"
)
func main() {
var byteArray = [6]byte{97,98,0,100,0,99}
cStyleString := bytes.SplitAfter(byteArray[:], []byte{0}) [0]
fmt.Println(cStyleString)
}
A full example to have a Go style string excluding the nulls:
package main
import (
"bytes"
"fmt"
)
func main() {
var byteArray = [6]byte{97, 98, 0, 100, 0, 99}
goStyleString := string(bytes.Split(byteArray[:], []byte{0}) [0])
fmt.Println(goStyleString)
}
This allocates a slice of slice of bytes. So keep an eye on performance if it is used heavily or repeatedly.
Use slices instead of arrays for reading. For example, io.Reader accepts a slice, not an array.
Use slicing instead of zero padding.
Example:
buf := make([]byte, 100)
n, err := myReader.Read(buf)
if n == 0 && err != nil {
log.Fatal(err)
}
consume(buf[:n]) // consume() will see an exact (not padded) slice of read data
Here is an option that removes the null bytes:
package main
import "golang.org/x/sys/windows"
func main() {
b := []byte{'M', 'a', 'r', 'c', 'h', 0}
s := windows.ByteSliceToString(b)
println(s == "March")
}
https://pkg.go.dev/golang.org/x/sys/unix#ByteSliceToString
https://pkg.go.dev/golang.org/x/sys/windows#ByteSliceToString
I want to assign string to bytes array:
var arr [20]byte
str := "abc"
for k, v := range []byte(str) {
arr[k] = byte(v)
}
Have another method?
Safe and simple:
[]byte("Here is a string....")
For converting from a string to a byte slice, string -> []byte:
[]byte(str)
For converting an array to a slice, [20]byte -> []byte:
arr[:]
For copying a string to an array, string -> [20]byte:
copy(arr[:], str)
Same as above, but explicitly converting the string to a slice first:
copy(arr[:], []byte(str))
The built-in copy function only copies to a slice, from a slice.
Arrays are "the underlying data", while slices are "a viewport into underlying data".
Using [:] makes an array qualify as a slice.
A string does not qualify as a slice that can be copied to, but it qualifies as a slice that can be copied from (strings are immutable).
If the string is too long, copy will only copy the part of the string that fits (and multi-byte runes may then be copied only partly, which will corrupt the last rune of the resulting string).
This code:
var arr [20]byte
copy(arr[:], "abc")
fmt.Printf("array: %v (%T)\n", arr, arr)
...gives the following output:
array: [97 98 99 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] ([20]uint8)
I also made it available at the Go Playground
For example,
package main
import "fmt"
func main() {
s := "abc"
var a [20]byte
copy(a[:], s)
fmt.Println("s:", []byte(s), "a:", a)
}
Output:
s: [97 98 99] a: [97 98 99 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
Piece of cake:
arr := []byte("That's all folks!!")
I think it's better..
package main
import "fmt"
func main() {
str := "abc"
mySlice := []byte(str)
fmt.Printf("%v -> '%s'",mySlice,mySlice )
}
Check here: http://play.golang.org/p/vpnAWHZZk7
Go, convert a string to a bytes slice
You need a fast way to convert a []string to []byte type. To use in situations such as storing text data into a random access file or other type of data manipulation that requires the input data to be in []byte type.
package main
func main() {
var s string
//...
b := []byte(s)
//...
}
which is useful when using ioutil.WriteFile, which accepts a bytes slice as its data parameter:
WriteFile func(filename string, data []byte, perm os.FileMode) error
Another example
package main
import (
"fmt"
"strings"
)
func main() {
stringSlice := []string{"hello", "world"}
stringByte := strings.Join(stringSlice, " ")
// Byte array value
fmt.Println([]byte(stringByte))
// Corresponding string value
fmt.Println(string([]byte(stringByte)))
}
Output:
[104 101 108 108 111 32 119 111 114 108 100] hello world
Please check the link playground
Besides the methods mentioned above, you can also do a trick as
s := "hello"
b := *(*[]byte)(unsafe.Pointer((*reflect.SliceHeader)(unsafe.Pointer(&s))))
Go Play: http://play.golang.org/p/xASsiSpQmC
You should never use this :-)
Ended up creating array specific methods to do this. Much like the encoding/binary package with specific methods for each int type. For example binary.BigEndian.PutUint16([]byte, uint16).
func byte16PutString(s string) [16]byte {
var a [16]byte
if len(s) > 16 {
copy(a[:], s)
} else {
copy(a[16-len(s):], s)
}
return a
}
var b [16]byte
b = byte16PutString("abc")
fmt.Printf("%v\n", b)
Output:
[0 0 0 0 0 0 0 0 0 0 0 0 0 97 98 99]
Notice how I wanted padding on the left, not the right.
http://play.golang.org/p/7tNumnJaiN
Arrays are values... slices are more like pointers. That is [n]type is not compatible with []type as they are fundamentally two different things. You can get a slice that points to an array by using arr[:] which returns a slice that has arr as it's backing storage.
One way to convert a slice of for example []byte to [20]byte is to actually allocate a [20]byte which you can do by using var [20]byte (as it's a value... no make needed) and then copy data into it:
buf := make([]byte, 10)
var arr [10]byte
copy(arr[:], buf)
Essentially what a lot of other answers get wrong is that []type is NOT an array.
[n]T and []T are completely different things!
When using reflect []T is not of kind Array but of kind Slice and [n]T is of kind Array.
You also can't use map[[]byte]T but you can use map[[n]byte]T.
This can sometimes be cumbersome because a lot of functions operate for example on []byte whereas some functions return [n]byte (most notably the hash functions in crypto/*).
A sha256 hash for example is [32]byte and not []byte so when beginners try to write it to a file for example:
sum := sha256.Sum256(data)
w.Write(sum)
they will get an error. The correct way of is to use
w.Write(sum[:])
However, what is it that you want? Just accessing the string bytewise? You can easily convert a string to []byte using:
bytes := []byte(str)
but this isn't an array, it's a slice. Also, byte != rune. In case you want to operate on "characters" you need to use rune... not byte.
If someone is looking for a quick consider use unsafe conversion between slices, you can refer to the following comparison.
package demo_test
import (
"testing"
"unsafe"
)
var testStr = "hello world"
var testBytes = []byte("hello world")
// Avoid copying the data.
func UnsafeStrToBytes(s string) []byte {
return *(*[]byte)(unsafe.Pointer(&s))
}
// Avoid copying the data.
func UnsafeBytesToStr(b []byte) string {
return *(*string)(unsafe.Pointer(&b))
}
func Benchmark_UnsafeStrToBytes(b *testing.B) {
for i := 0; i < b.N; i++ {
_ = UnsafeStrToBytes(testStr)
}
}
func Benchmark_SafeStrToBytes(b *testing.B) {
for i := 0; i < b.N; i++ {
_ = []byte(testStr)
}
}
func Benchmark_UnSafeBytesToStr(b *testing.B) {
for i := 0; i < b.N; i++ {
_ = UnsafeBytesToStr(testBytes)
}
}
func Benchmark_SafeBytesToStr(b *testing.B) {
for i := 0; i < b.N; i++ {
_ = string(testBytes)
}
}
go test -v -bench="^Benchmark" -run=none
output
cpu: Intel(R) Core(TM) i7-8565U CPU # 1.80GHz
Benchmark_UnsafeStrToBytes
Benchmark_UnsafeStrToBytes-8 1000000000 0.2465 ns/op
Benchmark_SafeStrToBytes
Benchmark_SafeStrToBytes-8 289119562 4.181 ns/op
Benchmark_UnSafeBytesToStr
Benchmark_UnSafeBytesToStr-8 1000000000 0.2530 ns/op
Benchmark_SafeBytesToStr
Benchmark_SafeBytesToStr-8 342842938 3.623 ns/op
PASS