How do you append a byte to a string in Go?
var ret string
var b byte
ret += b
invalid operation: ret += b (mismatched types string and byte)
In addition to ThunderCats answer.. you could initialize a bytes.Buffer from a string ... allowing you to continue appending bytes as you see fit:
buff := bytes.NewBufferString(ret)
// maybe buff.Grow(n) .. if you hit perf issues?
buff.WriteByte(b)
buff.WriteByte(b)
// ...
result := buff.String()
Here are a few options:
// append byte as slice
ret += string([]byte{b})
// append byte as rune
ret += string(rune(b))
// convert string to byte slice, append byte to slice, convert back to string
ret = string(append([]byte(ret), b))
Benchmark to see which one is best.
If you want to append more than one byte, then break the second option into multiple statements and append to the []byte:
buf := []byte(ret) // convert string to byte slice
buf = append(buf, b) // append byte to slice
buf = append(buf, b1) // append byte to slice
... etc
ret = string(buf) // convert back to string
If you want to append the rune r, then it's a little simpler:
ret += string(r)
Strings are immutable. The code above creates a new string that is a concatenation of the original string and a byte or rune.
It's a lot simpler than either of the other answers:
var ret string = "test"
var b byte = 'a'
ret += string(b)
// returns "testa"
That is, you can just cast an integer to a string and it will treat the integer as a rune (byte is an integer type). An then you can just concatenate the resulting string with +
Playground: https://play.golang.org/p/ktnUg70M-I
Another solution in Golang.
package main
import (
"fmt"
)
func main() {
byteArr := []byte{1,2,5,4}
stringStr := "Test"
output:= fmt.Sprintf("%v %v",byteArr,stringStr)
fmt.Println("Output: ",output)
}
Output
Output: [1 2 5 4] Test
You can also use strings.Builder:
package main
import "strings"
func main() {
var b strings.Builder
b.WriteByte('!')
println(b.String() == "!")
}
https://golang.org/pkg/strings#Builder.WriteByte
Related
For below code:
package main
import (
"fmt"
"unicode/utf8"
)
func main() {
// Declare a string with both chinese and english characters
s := "世界 means world"
// UTFMax is 4 -- up to 4 bytes per encoded rune
var buf [utf8.UTFMax]byte
fmt.Println(len(buf))
fmt.Println(cap(buf))
// Iterate over the string
for i, r := range s {
// Capture the number of bytes for this rune
rl := utf8.RuneLen(r)
fmt.Println("Rune lenght is:", rl)
// Calculate the slice offset for the bytes associated
// with this rune
si := i + rl
fmt.Println("Index is:", i)
fmt.Println("Slice offset:", si)
//Copy of rune from the string to buffer
copy(buf[:], s[i:si])
// Display the details
fmt.Printf("%2d: %q; codepoint: %#6x; encoded bytes: %#v\n", i, r, r, buf[:rl])
}
}
Built-in functions len & cap give the value for array type [4]byte
var buf [utf8.UTFMax]byte
fmt.Println(len(buf))
fmt.Println(cap(buf))
....
copy(buf[:], s[i:si])
Q1) Does buf[:] create a new slice header that points to array storage buf?
Q2) Is len(buf) performing implicit conversion of buf from array type to slice type?
Q3) Does s[i:si] create a new slice header that points to string s?
buf[:] is a slice whose cap and len are the length of the array, and the backing store is the array buf.
len(buf) gets the length of the array. It does not convert buf to a slice to do that. Length of buf is fixed at compile time.
s[i:si] creates a new string, not a new slice.
I'm trying to create an RSA Public Key from a Modulus and Exponent stored in a byte array. After some experimentation I've got the following:
func bytes_to_int(b []byte) (acc uint64) {
length := len(b)
if length % 4 != 0 {
extra := (4 - length % 4)
b = append([]byte(strings.Repeat("\000", extra)), b...)
length += extra
}
var block uint32
for i := 0; i < length; i += 4 {
block = binary.BigEndian.Uint32(b[i:i+4])
acc = (acc << 32) + uint64(block)
}
return
}
func main() {
fmt.Println(bytes_to_int(data[:128]))
fmt.Println(bytes_to_int(data[128:]))
}
This appears to work (although I'm not convinced there isn't a better way). My next step was to convert it to use math/big in order to handle larger numbers. I can see an Lsh function to do the << but can't figure out how to recursively add the Uint32(block) to the big.Int.
For reference, the Public Key I'm attempting to import is a Mixmaster Key stored in a keyring (pubring.mix):
http://www.mixmin.net/draft-sassaman-mixmaster-XX.html#key-format
http://pinger.mixmin.net/pubring.mix
You want Int.SetBytes to make a big.int from a slice of []byte.
func (z *Int) SetBytes(buf []byte) *Int
SetBytes interprets buf as the bytes of a big-endian unsigned integer, sets z to that value, and returns z.
This should be quite straightforward to use in your application since your keys are in big-endian format according to the doc you linked.
import "math/big"
z := new(big.Int)
z.SetBytes(byteSliceHere)
Like Nick mentioned, you could use SetBytes, keep in mind the input is in base64 so you have to decode that first.
Example:
func Base64ToInt(s string) (*big.Int, error) {
data, err := base64.StdEncoding.DecodeString(s)
if err != nil {
return nil, err
}
i := new(big.Int)
i.SetBytes(data)
return i, nil
}
I need to read [100]byte to transfer a bunch of string data.
Because not all of the strings are precisely 100 characters long, the remaining part of the byte array is padded with 0s.
If I convert [100]byte to string by: string(byteArray[:]), the tailing 0s are displayed as ^#^#s.
In C, the string will terminate upon 0, so what's the best way to convert this byte array to string in Go?
Methods that read data into byte slices return the number of bytes read. You should save that number and then use it to create your string. If n is the number of bytes read, your code would look like this:
s := string(byteArray[:n])
To convert the full string, this can be used:
s := string(byteArray[:len(byteArray)])
This is equivalent to:
s := string(byteArray[:])
If for some reason you don't know n, you could use the bytes package to find it, assuming your input doesn't have a null character embedded in it.
n := bytes.Index(byteArray[:], []byte{0})
Or as icza pointed out, you can use the code below:
n := bytes.IndexByte(byteArray[:], 0)
Use:
s := string(byteArray[:])
Simplistic solution:
str := fmt.Sprintf("%s", byteArray)
I'm not sure how performant this is though.
For example,
package main
import "fmt"
func CToGoString(c []byte) string {
n := -1
for i, b := range c {
if b == 0 {
break
}
n = i
}
return string(c[:n+1])
}
func main() {
c := [100]byte{'a', 'b', 'c'}
fmt.Println("C: ", len(c), c[:4])
g := CToGoString(c[:])
fmt.Println("Go:", len(g), g)
}
Output:
C: 100 [97 98 99 0]
Go: 3 abc
The following code is looking for '\0', and under the assumptions of the question the array can be considered sorted since all non-'\0' precede all '\0'. This assumption won't hold if the array can contain '\0' within the data.
Find the location of the first zero-byte using a binary search, then slice.
You can find the zero-byte like this:
package main
import "fmt"
func FirstZero(b []byte) int {
min, max := 0, len(b)
for {
if min + 1 == max { return max }
mid := (min + max) / 2
if b[mid] == '\000' {
max = mid
} else {
min = mid
}
}
return len(b)
}
func main() {
b := []byte{1, 2, 3, 0, 0, 0}
fmt.Println(FirstZero(b))
}
It may be faster just to naively scan the byte array looking for the zero-byte, especially if most of your strings are short.
When you do not know the exact length of non-nil bytes in the array, you can trim it first:
string(bytes.Trim(arr, "\x00"))
Use this:
bytes.NewBuffer(byteArray).String()
Only use for performance tuning.
package main
import (
"fmt"
"reflect"
"unsafe"
)
func BytesToString(b []byte) string {
return *(*string)(unsafe.Pointer(&b))
}
func StringToBytes(s string) []byte {
return *(*[]byte)(unsafe.Pointer(&s))
}
func main() {
b := []byte{'b', 'y', 't', 'e'}
s := BytesToString(b)
fmt.Println(s)
b = StringToBytes(s)
fmt.Println(string(b))
}
Though not extremely performant, the only readable solution is:
// Split by separator and pick the first one.
// This has all the characters till null, excluding null itself.
retByteArray := bytes.Split(byteArray[:], []byte{0}) [0]
// OR
// If you want a true C-like string, including the null character
retByteArray := bytes.SplitAfter(byteArray[:], []byte{0}) [0]
A full example to have a C-style byte array:
package main
import (
"bytes"
"fmt"
)
func main() {
var byteArray = [6]byte{97,98,0,100,0,99}
cStyleString := bytes.SplitAfter(byteArray[:], []byte{0}) [0]
fmt.Println(cStyleString)
}
A full example to have a Go style string excluding the nulls:
package main
import (
"bytes"
"fmt"
)
func main() {
var byteArray = [6]byte{97, 98, 0, 100, 0, 99}
goStyleString := string(bytes.Split(byteArray[:], []byte{0}) [0])
fmt.Println(goStyleString)
}
This allocates a slice of slice of bytes. So keep an eye on performance if it is used heavily or repeatedly.
Use slices instead of arrays for reading. For example, io.Reader accepts a slice, not an array.
Use slicing instead of zero padding.
Example:
buf := make([]byte, 100)
n, err := myReader.Read(buf)
if n == 0 && err != nil {
log.Fatal(err)
}
consume(buf[:n]) // consume() will see an exact (not padded) slice of read data
Here is an option that removes the null bytes:
package main
import "golang.org/x/sys/windows"
func main() {
b := []byte{'M', 'a', 'r', 'c', 'h', 0}
s := windows.ByteSliceToString(b)
println(s == "March")
}
https://pkg.go.dev/golang.org/x/sys/unix#ByteSliceToString
https://pkg.go.dev/golang.org/x/sys/windows#ByteSliceToString
I am looking to convert a string array to a byte array in GO so I can write it down to a disk. What is an optimal solution to encode and decode a string array ([]string) to a byte array ([]byte)?
I was thinking of iterating the string array twice, first one to get the actual size needed for the byte array and then a second one to write the length and actual string ([]byte(str)) for each element.
The solution must be able to convert it the other-way; from a []byte to a []string.
Lets ignore the fact that this is Go for a second. The first thing you need is a serialization format to marshal the []string into.
There are many option here. You could build your own or use a library. I am going to assume you don't want to build your own and jump to serialization formats go supports.
In all examples, data is the []string and fp is the file you are reading/writing to. Errors are being ignored, check the returns of functions to handle errors.
Gob
Gob is a go only binary format. It should be relatively space efficient as the number of strings increases.
enc := gob.NewEncoder(fp)
enc.Encode(data)
Reading is also simple
var data []string
dec := gob.NewDecoder(fp)
dec.Decode(&data)
Gob is simple and to the point. However, the format is only readable with other Go code.
Json
Next is json. Json is a format used just about everywhere. This format is just as easy to use.
enc := json.NewEncoder(fp)
enc.Encode(data)
And for reading:
var data []string
dec := json.NewDecoder(fp)
dec.Decode(&data)
XML
XML is another common format. However, it has pretty high overhead and not as easy to use. While you could just do the same you did for gob and json, proper xml requires a root tag. In this case, we are using the root tag "Strings" and each string is wrapped in an "S" tag.
type Strings struct {
S []string
}
enc := xml.NewEncoder(fp)
enc.Encode(Strings{data})
var x Strings
dec := xml.NewDecoder(fp)
dec.Decode(&x)
data := x.S
CSV
CSV is different from the others. You have two options, use one record with n rows or n records with 1 row. The following example uses n records. It would be boring if I used one record. It would look too much like the others. CSV can ONLY hold strings.
enc := csv.NewWriter(fp)
for _, v := range data {
enc.Write([]string{v})
}
enc.Flush()
To read:
var err error
var data string
dec := csv.NewReader(fp)
for err == nil { // reading ends when an error is reached (perhaps io.EOF)
var s []string
s, err = dec.Read()
if len(s) > 0 {
data = append(data, s[0])
}
}
Which format you use is a matter of preference. There are many other possible encodings that I have not mentioned. For example, there is an external library called bencode. I don't personally like bencode, but it works. It is the same encoding used by bittorrent metadata files.
If you want to make your own encoding, encoding/binary is a good place to start. That would allow you to make the most compact file possible, but I hardly thing it is worth the effort.
The gob package will do this for you http://godoc.org/encoding/gob
Example to play with http://play.golang.org/p/e0FEZm-qiS
same source code is below.
package main
import (
"bytes"
"encoding/gob"
"fmt"
)
func main() {
// store to byte array
strs := []string{"foo", "bar"}
buf := &bytes.Buffer{}
gob.NewEncoder(buf).Encode(strs)
bs := buf.Bytes()
fmt.Printf("%q", bs)
// Decode it back
strs2 := []string{}
gob.NewDecoder(buf).Decode(&strs2)
fmt.Printf("%v", strs2)
}
to convert []string to []byte
var str = []string{"str1","str2"}
var x = []byte{}
for i:=0; i<len(str); i++{
b := []byte(str[i])
for j:=0; j<len(b); j++{
x = append(x,b[j])
}
}
to convert []byte to string
str := ""
var x = []byte{'c','a','t'}
for i := 0; i < len(x); i++ {
str += string(x[i])
}
To illustrate the problem, convert []string to []byte and then convert []byte back to []string, here's a simple solution:
package main
import (
"encoding/binary"
"fmt"
)
const maxInt32 = 1<<(32-1) - 1
func writeLen(b []byte, l int) []byte {
if 0 > l || l > maxInt32 {
panic("writeLen: invalid length")
}
var lb [4]byte
binary.BigEndian.PutUint32(lb[:], uint32(l))
return append(b, lb[:]...)
}
func readLen(b []byte) ([]byte, int) {
if len(b) < 4 {
panic("readLen: invalid length")
}
l := binary.BigEndian.Uint32(b)
if l > maxInt32 {
panic("readLen: invalid length")
}
return b[4:], int(l)
}
func Decode(b []byte) []string {
b, ls := readLen(b)
s := make([]string, ls)
for i := range s {
b, ls = readLen(b)
s[i] = string(b[:ls])
b = b[ls:]
}
return s
}
func Encode(s []string) []byte {
var b []byte
b = writeLen(b, len(s))
for _, ss := range s {
b = writeLen(b, len(ss))
b = append(b, ss...)
}
return b
}
func codecEqual(s []string) bool {
return fmt.Sprint(s) == fmt.Sprint(Decode(Encode(s)))
}
func main() {
var s []string
fmt.Println("equal", codecEqual(s))
s = []string{"", "a", "bc"}
e := Encode(s)
d := Decode(e)
fmt.Println("s", len(s), s)
fmt.Println("e", len(e), e)
fmt.Println("d", len(d), d)
fmt.Println("equal", codecEqual(s))
}
Output:
equal true
s 3 [ a bc]
e 19 [0 0 0 3 0 0 0 0 0 0 0 1 97 0 0 0 2 98 99]
d 3 [ a bc]
equal true
I would suggest to use PutUvarint and Uvarint for storing/retrieving len(s) and using []byte(str) to pass str to some io.Writer. With a string length known from Uvarint, one can buf := make([]byte, n) and pass the buf to some io.Reader.
Prepend the whole thing with length of the string array and repeat the above for all of its items. Reading the whole thing back is again reading first the outer length and repeating n-times the item read.
You can do something like this:
var lines = []string
var ctx = []byte{}
for _, s := range lines {
ctx = append(ctx, []byte(s)...)
}
It can be done easily using strings package. First you need to convert the slice of string to a string.
func Join(elems []string, sep string) string
You need to pass the slice of strings and the separator you need to separate the elements in the string. (examples: space or comma)
Then you can easily convert the string to a slice of bytes by type conversion.
package main
import (
"fmt"
"strings"
)
func main() {
//Slice of Strings
sliceStr := []string{"a","b","c","d"}
fmt.Println(sliceStr) //prints [a b c d]
//Converting slice of String to String
str := strings.Join(sliceStr,"")
fmt.Println(str) // prints abcd
//Converting String to slice of Bytes
sliceByte := []byte(str) //prints [97 98 99 100]
fmt.Println(sliceByte)
//Converting slice of bytes a String
str2 := string(sliceByte)
fmt.Println(str2) // prints abcd
//Converting string to a slice of Strings
sliceStr2 := strings.Split(str2,"")
fmt.Println(sliceStr2) //prints [a b c d]
}
I want to assign string to bytes array:
var arr [20]byte
str := "abc"
for k, v := range []byte(str) {
arr[k] = byte(v)
}
Have another method?
Safe and simple:
[]byte("Here is a string....")
For converting from a string to a byte slice, string -> []byte:
[]byte(str)
For converting an array to a slice, [20]byte -> []byte:
arr[:]
For copying a string to an array, string -> [20]byte:
copy(arr[:], str)
Same as above, but explicitly converting the string to a slice first:
copy(arr[:], []byte(str))
The built-in copy function only copies to a slice, from a slice.
Arrays are "the underlying data", while slices are "a viewport into underlying data".
Using [:] makes an array qualify as a slice.
A string does not qualify as a slice that can be copied to, but it qualifies as a slice that can be copied from (strings are immutable).
If the string is too long, copy will only copy the part of the string that fits (and multi-byte runes may then be copied only partly, which will corrupt the last rune of the resulting string).
This code:
var arr [20]byte
copy(arr[:], "abc")
fmt.Printf("array: %v (%T)\n", arr, arr)
...gives the following output:
array: [97 98 99 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] ([20]uint8)
I also made it available at the Go Playground
For example,
package main
import "fmt"
func main() {
s := "abc"
var a [20]byte
copy(a[:], s)
fmt.Println("s:", []byte(s), "a:", a)
}
Output:
s: [97 98 99] a: [97 98 99 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
Piece of cake:
arr := []byte("That's all folks!!")
I think it's better..
package main
import "fmt"
func main() {
str := "abc"
mySlice := []byte(str)
fmt.Printf("%v -> '%s'",mySlice,mySlice )
}
Check here: http://play.golang.org/p/vpnAWHZZk7
Go, convert a string to a bytes slice
You need a fast way to convert a []string to []byte type. To use in situations such as storing text data into a random access file or other type of data manipulation that requires the input data to be in []byte type.
package main
func main() {
var s string
//...
b := []byte(s)
//...
}
which is useful when using ioutil.WriteFile, which accepts a bytes slice as its data parameter:
WriteFile func(filename string, data []byte, perm os.FileMode) error
Another example
package main
import (
"fmt"
"strings"
)
func main() {
stringSlice := []string{"hello", "world"}
stringByte := strings.Join(stringSlice, " ")
// Byte array value
fmt.Println([]byte(stringByte))
// Corresponding string value
fmt.Println(string([]byte(stringByte)))
}
Output:
[104 101 108 108 111 32 119 111 114 108 100] hello world
Please check the link playground
Besides the methods mentioned above, you can also do a trick as
s := "hello"
b := *(*[]byte)(unsafe.Pointer((*reflect.SliceHeader)(unsafe.Pointer(&s))))
Go Play: http://play.golang.org/p/xASsiSpQmC
You should never use this :-)
Ended up creating array specific methods to do this. Much like the encoding/binary package with specific methods for each int type. For example binary.BigEndian.PutUint16([]byte, uint16).
func byte16PutString(s string) [16]byte {
var a [16]byte
if len(s) > 16 {
copy(a[:], s)
} else {
copy(a[16-len(s):], s)
}
return a
}
var b [16]byte
b = byte16PutString("abc")
fmt.Printf("%v\n", b)
Output:
[0 0 0 0 0 0 0 0 0 0 0 0 0 97 98 99]
Notice how I wanted padding on the left, not the right.
http://play.golang.org/p/7tNumnJaiN
Arrays are values... slices are more like pointers. That is [n]type is not compatible with []type as they are fundamentally two different things. You can get a slice that points to an array by using arr[:] which returns a slice that has arr as it's backing storage.
One way to convert a slice of for example []byte to [20]byte is to actually allocate a [20]byte which you can do by using var [20]byte (as it's a value... no make needed) and then copy data into it:
buf := make([]byte, 10)
var arr [10]byte
copy(arr[:], buf)
Essentially what a lot of other answers get wrong is that []type is NOT an array.
[n]T and []T are completely different things!
When using reflect []T is not of kind Array but of kind Slice and [n]T is of kind Array.
You also can't use map[[]byte]T but you can use map[[n]byte]T.
This can sometimes be cumbersome because a lot of functions operate for example on []byte whereas some functions return [n]byte (most notably the hash functions in crypto/*).
A sha256 hash for example is [32]byte and not []byte so when beginners try to write it to a file for example:
sum := sha256.Sum256(data)
w.Write(sum)
they will get an error. The correct way of is to use
w.Write(sum[:])
However, what is it that you want? Just accessing the string bytewise? You can easily convert a string to []byte using:
bytes := []byte(str)
but this isn't an array, it's a slice. Also, byte != rune. In case you want to operate on "characters" you need to use rune... not byte.
If someone is looking for a quick consider use unsafe conversion between slices, you can refer to the following comparison.
package demo_test
import (
"testing"
"unsafe"
)
var testStr = "hello world"
var testBytes = []byte("hello world")
// Avoid copying the data.
func UnsafeStrToBytes(s string) []byte {
return *(*[]byte)(unsafe.Pointer(&s))
}
// Avoid copying the data.
func UnsafeBytesToStr(b []byte) string {
return *(*string)(unsafe.Pointer(&b))
}
func Benchmark_UnsafeStrToBytes(b *testing.B) {
for i := 0; i < b.N; i++ {
_ = UnsafeStrToBytes(testStr)
}
}
func Benchmark_SafeStrToBytes(b *testing.B) {
for i := 0; i < b.N; i++ {
_ = []byte(testStr)
}
}
func Benchmark_UnSafeBytesToStr(b *testing.B) {
for i := 0; i < b.N; i++ {
_ = UnsafeBytesToStr(testBytes)
}
}
func Benchmark_SafeBytesToStr(b *testing.B) {
for i := 0; i < b.N; i++ {
_ = string(testBytes)
}
}
go test -v -bench="^Benchmark" -run=none
output
cpu: Intel(R) Core(TM) i7-8565U CPU # 1.80GHz
Benchmark_UnsafeStrToBytes
Benchmark_UnsafeStrToBytes-8 1000000000 0.2465 ns/op
Benchmark_SafeStrToBytes
Benchmark_SafeStrToBytes-8 289119562 4.181 ns/op
Benchmark_UnSafeBytesToStr
Benchmark_UnSafeBytesToStr-8 1000000000 0.2530 ns/op
Benchmark_SafeBytesToStr
Benchmark_SafeBytesToStr-8 342842938 3.623 ns/op
PASS