Related
var b [88]byte
n, err := file.Read(b[:])
fmt.Printf("bytes read: %d Bytes: [% x]\n", n, b)
The above prints bytes in hexdecimal
I have a struct like this
type SomeStruct struct {
field1 []byte
field2 []byte
}
someStructInstance := SomeStruct{[249 190 180 217], [29 1 0 0]}
fmt.Println(someStructInstance)
=> {[249 190 180 217] [29 1 0 0]}
But ideally I would like it to print hexdecimal
=> {[f9 be b4 d9] [1d 01 00 00]}
How would I go about that?
I think you will just have to define your own String function on SomeStruct. Here is an example:
package main
import "fmt"
type SomeStruct struct {
field1 []byte
field2 []byte
}
func (s SomeStruct) String() string {
return fmt.Sprintf("{[% x] [% x]}", s.field1, s.field2)
}
func main() {
someStructInstance := SomeStruct{[]byte{249, 190, 180, 217}, []byte{29, 1, 0, 0}}
fmt.Println(someStructInstance)
}
See it in running on the Go Playground: http://play.golang.org/p/eYBa1n33a2
You could use reflection to inspect the struct and print any []bytes that it has.
package main
import (
"fmt"
"reflect"
)
type SomeStruct struct {
field1 []byte
field2 []byte
}
type OtherStruct struct {
intValue int
intSlice []int
byteSlice []byte
}
var typeOfBytes = reflect.TypeOf([]byte(nil))
func printSlicesHex(obj interface{}) {
value := reflect.ValueOf(obj)
typeOfObj := value.Type()
for i := 0; i < value.NumField(); i++ {
field := value.Field(i)
if field.Type() == typeOfBytes {
bytes := field.Bytes()
printBytes(typeOfObj.Field(i).Name, bytes)
}
}
}
func printBytes(name string, bytes []byte) {
fmt.Printf("%s: [% x]\n", name, bytes)
}
func main() {
someStructInstance := SomeStruct{[]byte{249, 190, 180, 217}, []byte{29, 1, 0, 0}}
fmt.Println("Printing []bytes in SomeStruct")
printSlicesHex(someStructInstance)
fmt.Println()
otherStruct := OtherStruct{0, []int{0, 1, 2}, []byte{0, 1, 2, 3}}
fmt.Println("Printing []bytes in OtherStruct")
printSlicesHex(otherStruct)
}
For each []byte, this example prints the name of the field and its data (in hex). You could improve on this by taking a custom function to do the printing, so you don't always have to print in hex.
Playground Link
Is there a built-in function to convert a uint to a slice of binary integers {0,1} ?
>> convert_to_binary(2)
[1, 0]
I am not aware of such a function, however you can use strconv.FormatUint for that purpose.
Example (on play):
func Bits(i uint64) []byte {
bits := []byte{}
for _, b := range strconv.FormatUint(i, 2) {
bits = append(bits, byte(b - rune('0')))
}
return bits
}
FormatUint will return the string representation of the given uint to a base, in this case 2, so we're encoding it in binary. So the returned string for i=2 looks like this: "10". In bytes this is [49 48] as 1 is 49 and 0 is 48 in ASCII and Unicode. So we just need to iterate over the string, subtracting 48 from each rune (unicode character) and converting it to a byte.
Here is another method:
package main
import (
"bytes"
"fmt"
"math/bits"
)
func unsigned(x uint) []byte {
b := make([]byte, bits.UintSize)
for i := range b {
if bits.LeadingZeros(x) == 0 {
b[i] = 1
}
x = bits.RotateLeft(x, 1)
}
return b
}
func trimUnsigned(x uint) []byte {
return bytes.TrimLeft(unsigned(x), string(0))
}
func main() {
b := trimUnsigned(2)
fmt.Println(b) // [1 0]
}
https://golang.org/pkg/math/bits#LeadingZeros
I have a function which receives a []byte but what I have is an int, what is the best way to go about this conversion ?
err = a.Write([]byte(myInt))
I guess I could go the long way and get it into a string and put that into bytes, but it sounds ugly and I guess there are better ways to do it.
I agree with Brainstorm's approach: assuming that you're passing a machine-friendly binary representation, use the encoding/binary library. The OP suggests that binary.Write() might have some overhead. Looking at the source for the implementation of Write(), I see that it does some runtime decisions for maximum flexibility.
func Write(w io.Writer, order ByteOrder, data interface{}) error {
// Fast path for basic types.
var b [8]byte
var bs []byte
switch v := data.(type) {
case *int8:
bs = b[:1]
b[0] = byte(*v)
case int8:
bs = b[:1]
b[0] = byte(v)
case *uint8:
bs = b[:1]
b[0] = *v
...
Right? Write() takes in a very generic data third argument, and that's imposing some overhead as the Go runtime then is forced into encoding type information. Since Write() is doing some runtime decisions here that you simply don't need in your situation, maybe you can just directly call the encoding functions and see if it performs better.
Something like this:
package main
import (
"encoding/binary"
"fmt"
)
func main() {
bs := make([]byte, 4)
binary.LittleEndian.PutUint32(bs, 31415926)
fmt.Println(bs)
}
Let us know how this performs.
Otherwise, if you're just trying to get an ASCII representation of the integer, you can get the string representation (probably with strconv.Itoa) and cast that string to the []byte type.
package main
import (
"fmt"
"strconv"
)
func main() {
bs := []byte(strconv.Itoa(31415926))
fmt.Println(bs)
}
Check out the "encoding/binary" package. Particularly the Read and Write functions:
binary.Write(a, binary.LittleEndian, myInt)
Sorry, this might be a bit late. But I think I found a better implementation on the go docs.
buf := new(bytes.Buffer)
var num uint16 = 1234
err := binary.Write(buf, binary.LittleEndian, num)
if err != nil {
fmt.Println("binary.Write failed:", err)
}
fmt.Printf("% x", buf.Bytes())
i thought int type has any method for getting int hash to bytes, but first i find math / big method for this
https://golang.org/pkg/math/big/
var f int = 52452356235; // int
var s = big.NewInt(int64(f)) // int to big Int
var b = s.Bytes() // big Int to bytes
// b - byte slise
var r = big.NewInt(0).SetBytes(b) // bytes to big Int
var i int = int(r.Int64()) // big Int to int
https://play.golang.org/p/VAKSGw8XNQq
However, this method uses an absolute value.
If you spend 1 byte more, you can transfer the sign
func IntToBytes(i int) []byte{
if i > 0 {
return append(big.NewInt(int64(i)).Bytes(), byte(1))
}
return append(big.NewInt(int64(i)).Bytes(), byte(0))
}
func BytesToInt(b []byte) int{
if b[len(b)-1]==0 {
return -int(big.NewInt(0).SetBytes(b[:len(b)-1]).Int64())
}
return int(big.NewInt(0).SetBytes(b[:len(b)-1]).Int64())
}
https://play.golang.org/p/mR5Sp5hu4jk
or new(https://play.golang.org/p/7ZAK4QL96FO)
(The package also provides functions for fill into an existing slice)
https://golang.org/pkg/math/big/#Int.FillBytes
Adding this option for dealing with basic uint8 to byte[] conversion
foo := 255 // 1 - 255
ufoo := uint16(foo)
far := []byte{0,0}
binary.LittleEndian.PutUint16(far, ufoo)
bar := int(far[0]) // back to int
fmt.Println("foo, far, bar : ",foo,far,bar)
output :
foo, far, bar : 255 [255 0] 255
Here is another option, based on the Go source code [1]:
package main
import (
"encoding/binary"
"fmt"
"math/bits"
)
func encodeUint(x uint64) []byte {
buf := make([]byte, 8)
binary.BigEndian.PutUint64(buf, x)
return buf[bits.LeadingZeros64(x) >> 3:]
}
func main() {
for x := 0; x <= 64; x += 8 {
buf := encodeUint(1<<x-1)
fmt.Println(buf)
}
}
Result:
[]
[255]
[255 255]
[255 255 255]
[255 255 255 255]
[255 255 255 255 255]
[255 255 255 255 255 255]
[255 255 255 255 255 255 255]
[255 255 255 255 255 255 255 255]
Much faster than math/big:
BenchmarkBig-12 28348621 40.62 ns/op
BenchmarkBit-12 731601145 1.641 ns/op
https://github.com/golang/go/blob/go1.16.5/src/encoding/gob/encode.go#L113-L117
You can try musgo_int. All you need to do is to cast your variable:
package main
import (
"github.com/ymz-ncnk/musgo_int"
)
func main() {
var myInt int = 1234
// from int to []byte
buf := make([]byte, musgo_int.Int(myInt).SizeMUS())
musgo_int.Int(myInt).MarshalMUS(buf)
// from []byte to int
_, err := (*musgo_int.Int)(&myInt).UnmarshalMUS(buf)
if err != nil {
panic(err)
}
}
Convert Integer to byte slice.
import (
"bytes"
"encoding/binary"
"log"
)
func IntToBytes(num int64) []byte {
buff := new(bytes.Buffer)
bigOrLittleEndian := binary.BigEndian
err := binary.Write(buff, bigOrLittleEndian, num)
if err != nil {
log.Panic(err)
}
return buff.Bytes()
}
Maybe the simple way is using protobuf, see the Protocol Buffer Basics: Go
define message like
message MyData {
int32 id = 1;
}
get more in Defining your protocol format
// Write
out, err := proto.Marshal(mydata)
read more in Writing a Message
Try math/big package to convert bytes array to int and to convert int to bytes array.
package main
import (
"fmt"
"math/big"
)
func main() {
// Convert int to []byte
var int_to_encode int64 = 65535
var bytes_array []byte = big.NewInt(int_to_encode).Bytes()
fmt.Println("bytes array", bytes_array)
// Convert []byte to int
var decoded_int int64 = new(big.Int).SetBytes(bytes_array).Int64()
fmt.Println("decoded int", decoded_int)
}
This is the most straight forward (and shortest (and safest) (and maybe most performant)) way:
buf.Bytes() is of type bytes slice.
var val uint32 = 42
buf := new(bytes.Buffer)
err := binary.Write(buf, binary.LittleEndian, val)
if err != nil {
fmt.Println("binary.Write failed:", err)
}
fmt.Printf("% x\n", buf.Bytes())
see also https://stackoverflow.com/a/74819602/589493
What's wrong with converting it to a string?
[]byte(fmt.Sprintf("%d", myint))
package main
import (
"bytes"
"encoding/binary"
"fmt"
)
func main() {
aa := uint(0xFFFFFFFF)
fmt.Println(aa)
byteNewbuf := []byte{0xFF, 0xFF, 0xFF, 0xFF}
buf := bytes.NewBuffer(byteNewbuf)
tt, _ := binary.ReadUvarint(buf)
fmt.Println(tt)
}
Need to convert 4 bytes array to uint32 but why the results are not same ?
go verion : beta 1.1
You can do this with one of the ByteOrder objects from the encoding/binary package. For instance:
package main
import (
"encoding/binary"
"fmt"
)
func main() {
aa := uint(0x7FFFFFFF)
fmt.Println(aa)
slice := []byte{0xFF, 0xFF, 0xFF, 0x7F}
tt := binary.LittleEndian.Uint32(slice)
fmt.Println(tt)
}
If your data is in big endian format, you can instead use the same methods on binary.BigEndian.
tt := uint32(buf[0])<<24 | uint32(buf[1])<<16 | uint32(buf[2]) <<8 |
uint32(buf[3])
for BE or
tt := uint32(buf[0]) | uint32(buf[1])<<8 | uint32(buf[2]) <<16 |
uint32(buf[3]) <<24
for LE.
[u]varint is a different kind of encoding (32 bit numbers can have as much as 5 bytes in the encoded form, 64 bit numbers up to 10).
No need to create a buffer for []byte. Use Varint or Uvarint directly on the byte slice instead.
You're throwing away the error returned by the function. The second result indicates how many bytes were read or if there was a problem. There is a problem while decoding 0xff, 0xff, 0xff, 0xff as an uvarint.
Here is how to use the encoding/binary package to do what you want. Note that you don't want to use any of the var functions as those do variable length encoding.
Playground version
package main
import (
"bytes"
"encoding/binary"
"fmt"
"log"
)
func main() {
aa := uint(0xFFFFFF0F)
fmt.Println(aa)
tt := uint32(0)
byteNewbuf := []byte{0x0F, 0xFF, 0xFF, 0xFF}
buf := bytes.NewBuffer(byteNewbuf)
err := binary.Read(buf, binary.LittleEndian, &tt)
if err != nil {
log.Fatalf("Decode failed: %s", err)
}
fmt.Println(tt)
}
Result is
4294967055
4294967055
Numeric types
byte alias for uint8
Since byte is an alias for uint8, your question, "Need to convert 4 bytes array to uint32", has already been answered:
How to convert [4]uint8 into uint32 in Go?
Package binary
[Uvarints and] Varints are a method of encoding integers using one
or more bytes; numbers with smaller absolute value take a smaller
number of bytes. For a specification, see
http://code.google.com/apis/protocolbuffers/docs/encoding.html.
Since Uvarints are a peculiar form of integer representation and storage, you should only use the ReadUvarint function on values that have been written with the Uvarint function.
For example,
package main
import (
"bytes"
"encoding/binary"
"fmt"
)
func main() {
buf := make([]byte, 10)
x := uint64(0xFFFFFFFF)
fmt.Printf("%2d %2d %v\n", x, len(buf), buf)
n := binary.PutUvarint(buf, x)
buf = buf[:n]
fmt.Printf("%2d %2d %v\n", x, len(buf), buf)
y, err := binary.ReadUvarint(bytes.NewBuffer(buf))
if err != nil {
fmt.Println(err)
return
}
fmt.Printf("%2d %2d %v\n", y, len(buf), buf)
}
Output:
4294967295 10 [0 0 0 0 0 0 0 0 0 0]
4294967295 5 [255 255 255 255 15]
4294967295 5 [255 255 255 255 15]
all. I'm encountering what seems to be a very strange problem. (It could be that it's far past when I should be asleep, and I'm overlooking something obvious.)
I have a []byte with length 8 as a result of some hex decoding. I need to produce a uint64 in order to use it. I have tried using binary.Uvarint(), from encoding/binary to do so, but it seems to only use the first byte in the array. Consider the following example.
package main
import (
"encoding/binary"
"fmt"
)
func main() {
array := []byte{0x00, 0x01, 0x08, 0x00, 0x08, 0x01, 0xab, 0x01}
num, _ := binary.Uvarint(array[0:8])
fmt.Printf("%v, %x\n", array, num)
}
Here it is on play.golang.org.
When that is run, it displays the num as 0, even though, in hex, it should be 000108000801ab01. Furthermore, if one catches the second value from binary.Uvarint(), it is the number of bytes read from the buffer, which, to my knowledge, should be 8, even though it is actually 1.
Am I interpreting this wrong? If so, what should I be using instead?
Thanks, you all. :)
You're decoding using a function whose use isn't the one you need :
Varints are a method of encoding integers using one or more bytes;
numbers with smaller absolute value take a smaller number of bytes.
For a specification, see
http://code.google.com/apis/protocolbuffers/docs/encoding.html.
It's not the standard encoding but a very specific, variable byte number, encoding. That's why it stops at the first byte whose value is less than 0x080.
As pointed by Stephen, binary.BigEndian and binary.LittleEndian provide useful functions to decode directly :
type ByteOrder interface {
Uint16([]byte) uint16
Uint32([]byte) uint32
Uint64([]byte) uint64
PutUint16([]byte, uint16)
PutUint32([]byte, uint32)
PutUint64([]byte, uint64)
String() string
}
So you may use
package main
import (
"encoding/binary"
"fmt"
)
func main() {
array := []byte{0x00, 0x01, 0x08, 0x00, 0x08, 0x01, 0xab, 0x01}
num := binary.LittleEndian.Uint64(array)
fmt.Printf("%v, %x", array, num)
}
or (if you want to check errors instead of panicking, thanks jimt for pointing this problem with the direct solution) :
package main
import (
"encoding/binary"
"bytes"
"fmt"
)
func main() {
array := []byte{0x00, 0x01, 0x08, 0x00, 0x08, 0x01, 0xab, 0x01}
var num uint64
err := binary.Read(bytes.NewBuffer(array[:]), binary.LittleEndian, &num)
fmt.Printf("%v, %x", array, num)
}
If don't care byte order, you can try this:
arr := [8]byte{1,2,3,4,5,6,7,8}
num := *(*uint64)(unsafe.Pointer(&arr[0]))
http://play.golang.org/p/aM2r40ANQC
If you look at the function for Uvarint you will see that it is not as straight a conversion as you expect.
To be honest, I haven't yet figured out what kind of byte format it expects (see edit).
But to write your own is close to trivial:
func Uvarint(buf []byte) (x uint64) {
for i, b := range buf {
x = x << 8 + uint64(b)
if i == 7 {
return
}
}
return
}
Edit
The byte format is none I am familiar.
It is a variable width encoding where the highest bit of each byte is a flag.
If set to 0, that byte is the last in the sequence.
If set to 1, the encoding should continue with the next byte.
Only the lower 7 bits of each byte are used to build the uint64 value. The first byte will set the lowest 7 bits of the uint64, the following byte bit 8-15, etc.