Decoding data from a byte slice to Uint32 - go

package main
import (
"bytes"
"encoding/binary"
"fmt"
)
func main() {
aa := uint(0xFFFFFFFF)
fmt.Println(aa)
byteNewbuf := []byte{0xFF, 0xFF, 0xFF, 0xFF}
buf := bytes.NewBuffer(byteNewbuf)
tt, _ := binary.ReadUvarint(buf)
fmt.Println(tt)
}
Need to convert 4 bytes array to uint32 but why the results are not same ?
go verion : beta 1.1

You can do this with one of the ByteOrder objects from the encoding/binary package. For instance:
package main
import (
"encoding/binary"
"fmt"
)
func main() {
aa := uint(0x7FFFFFFF)
fmt.Println(aa)
slice := []byte{0xFF, 0xFF, 0xFF, 0x7F}
tt := binary.LittleEndian.Uint32(slice)
fmt.Println(tt)
}
If your data is in big endian format, you can instead use the same methods on binary.BigEndian.

tt := uint32(buf[0])<<24 | uint32(buf[1])<<16 | uint32(buf[2]) <<8 |
uint32(buf[3])
for BE or
tt := uint32(buf[0]) | uint32(buf[1])<<8 | uint32(buf[2]) <<16 |
uint32(buf[3]) <<24
for LE.
[u]varint is a different kind of encoding (32 bit numbers can have as much as 5 bytes in the encoded form, 64 bit numbers up to 10).
No need to create a buffer for []byte. Use Varint or Uvarint directly on the byte slice instead.
You're throwing away the error returned by the function. The second result indicates how many bytes were read or if there was a problem. There is a problem while decoding 0xff, 0xff, 0xff, 0xff as an uvarint.

Here is how to use the encoding/binary package to do what you want. Note that you don't want to use any of the var functions as those do variable length encoding.
Playground version
package main
import (
"bytes"
"encoding/binary"
"fmt"
"log"
)
func main() {
aa := uint(0xFFFFFF0F)
fmt.Println(aa)
tt := uint32(0)
byteNewbuf := []byte{0x0F, 0xFF, 0xFF, 0xFF}
buf := bytes.NewBuffer(byteNewbuf)
err := binary.Read(buf, binary.LittleEndian, &tt)
if err != nil {
log.Fatalf("Decode failed: %s", err)
}
fmt.Println(tt)
}
Result is
4294967055
4294967055

Numeric types
byte alias for uint8
Since byte is an alias for uint8, your question, "Need to convert 4 bytes array to uint32", has already been answered:
How to convert [4]uint8 into uint32 in Go?
Package binary
[Uvarints and] Varints are a method of encoding integers using one
or more bytes; numbers with smaller absolute value take a smaller
number of bytes. For a specification, see
http://code.google.com/apis/protocolbuffers/docs/encoding.html.
Since Uvarints are a peculiar form of integer representation and storage, you should only use the ReadUvarint function on values that have been written with the Uvarint function.
For example,
package main
import (
"bytes"
"encoding/binary"
"fmt"
)
func main() {
buf := make([]byte, 10)
x := uint64(0xFFFFFFFF)
fmt.Printf("%2d %2d %v\n", x, len(buf), buf)
n := binary.PutUvarint(buf, x)
buf = buf[:n]
fmt.Printf("%2d %2d %v\n", x, len(buf), buf)
y, err := binary.ReadUvarint(bytes.NewBuffer(buf))
if err != nil {
fmt.Println(err)
return
}
fmt.Printf("%2d %2d %v\n", y, len(buf), buf)
}
Output:
4294967295 10 [0 0 0 0 0 0 0 0 0 0]
4294967295 5 [255 255 255 255 15]
4294967295 5 [255 255 255 255 15]

Related

ParseInt doesn't convert to the desired type

Here's my code:
package main
import (
"fmt"
"reflect"
"strconv"
)
func main() {
i, _ := strconv.ParseInt("10", 10, 8)
fmt.Println(reflect.TypeOf(i))
}
I expect i to be 8-bits long (the 3rd argument to strconv.ParseInt). It is int64, however (and the docs state that strconv.ParseInt will return int64).
What's the point of ParseInt if it always returns int64 (why not just use Atoi?)
Note this from the function's doc:
The bitSize argument specifies the integer type that the result must
fit into. Bit sizes 0, 8, 16, 32, and 64 correspond to int, int8,
int16, int32, and int64. For a bitSize below 0 or above 64 an error is
returned.
So it's guaranteed that you can convert your result to a byte with byte(i).
Go doesn't have generics yet, so having a single ParseInt that can accept pointers to multiple integer types is difficult. Instead the guarantee is done with the bitSize argument
Package strconv
import "strconv"
func ParseInt
func ParseInt(s string, base int, bitSize int) (i int64, err error)
ParseInt interprets a string s in the given base (0, 2 to 36) and bit
size (0 to 64) and returns the corresponding value i.
If base == 0, the base is implied by the string's prefix: base 16 for
"0x", base 8 for "0", and base 10 otherwise. For bases 1, below 0 or
above 36 an error is returned.
The bitSize argument specifies the integer type that the result must
fit into. Bit sizes 0, 8, 16, 32, and 64 correspond to int, int8,
int16, int32, and int64. For a bitSize below 0 or above 64 an error is
returned.
The errors that ParseInt returns have concrete type *NumError and
include err.Num = s. If s is empty or contains invalid digits, err.Err
= ErrSyntax and the returned value is 0; if the value corresponding to s cannot be represented by a signed integer of the given size, err.Err
= ErrRange and the returned value is the maximum magnitude integer of the appropriate bitSize and sign.
For example,
package main
import (
"fmt"
"strconv"
)
func main() {
i64, err := strconv.ParseInt("10", 10, 8)
if err != nil {
panic(err)
}
fmt.Printf("%[1]d %[1]T\n", i64)
i8 := int8(i64)
fmt.Printf("%[1]d %[1]T\n", i8)
}
Playground: https://play.golang.org/p/HSHtUnC7qql
Output:
10 int64
10 int8
In Go, we often use functions to hide implementation details.
For example,
package main
import (
"fmt"
"strconv"
)
func ParseInt8(s string, base int) (int8, error) {
i64, err := strconv.ParseInt(s, base, 8)
return int8(i64), err
}
func main() {
i8, err := ParseInt8("10", 10)
if err != nil {
panic(err)
}
fmt.Printf("%[1]d %[1]T\n", i8)
}
Playground: https://play.golang.org/p/HdA3O71U54z
Output:
10 int8
I think what you are really asking is what is the point of the 3rd parameter to ParseInt().
It's to save you having to check for overflow manually like this:
i, err := strconv.Atoi(intString)
if err != nil || i < -128 || i > 127 {
// handle error
}
i8 := int8(i)

Go binary encoding variant vs fixed slice length

I need to encode integer keys as byte slices for a KV database.
I want to make the encoding smaller and cut the zero padding.
I thought the variant encoding from the binary package would be the way to go.
But in both cases, variant and fixed, the byte slice length is the same.
Just different bits arrangement since first bit is used as a flag.
I assumed the variant encoding would cut the "extra fat". No.
package main
import (
"encoding/binary"
"fmt"
)
func main() {
x := 16
y := 106547
fmt.Println(x)
fmt.Println(y)
// Variant
bvx := make([]byte, 8)
bvy := make([]byte, 8)
xbts := binary.PutUvarint(bvx, uint64(x))
ybts := binary.PutUvarint(bvy, uint64(y))
fmt.Println("Variant bytes written x: ", xbts)
fmt.Println("Variant bytes written y: ", ybts)
fmt.Println(bvx)
fmt.Println(bvy)
fmt.Println("bvx length: ", len(bvx))
fmt.Println("bvy length: ", len(bvy))
// Fixed
bfx := make([]byte, 8)
bfy := make([]byte, 8)
binary.LittleEndian.PutUint64(bfx, uint64(x))
binary.LittleEndian.PutUint64(bfy, uint64(y))
fmt.Println(bfx)
fmt.Println(bfy)
fmt.Println("bfx length: ", len(bfx))
fmt.Println("bfy length: ", len(bfy))
}
My question is. Do I have to splice the byte slice manually with variant encoding to get rid of the extra bytes?
Since put PutUvariant returns the number of bytes written, I can just splice the byte slice.
Is this the right way to do it?
If not, what is the correct way to make the slices smaller?
Thanks
Package binary
import "encoding/binary"
func PutUvarint
func PutUvarint(buf []byte, x uint64) int
PutUvarint encodes a uint64 into buf and returns the number of bytes
written. If the buffer is too small, PutUvarint will panic.
Fix your code:
bvx := make([]byte, binary.MaxVarintLen64)
bvy := make([]byte, binary.MaxVarintLen64)
bvx = bvx[:binary.PutUvarint(bvx[:cap(bvx)], uint64(x))]
bvy = bvy[:binary.PutUvarint(bvy[:cap(bvy)], uint64(y))]
package main
import (
"encoding/binary"
"fmt"
)
func main() {
x := 16
y := 106547
fmt.Println(x)
fmt.Println(y)
// Variant
bvx := make([]byte, binary.MaxVarintLen64)
bvy := make([]byte, binary.MaxVarintLen64)
bvx = bvx[:binary.PutUvarint(bvx[:cap(bvx)], uint64(x))]
bvy = bvy[:binary.PutUvarint(bvy[:cap(bvy)], uint64(y))]
fmt.Println("Variant bytes written x: ", len(bvx))
fmt.Println("Variant bytes written y: ", len(bvy))
fmt.Println(bvx)
fmt.Println(bvy)
fmt.Println("bvx length: ", len(bvx))
fmt.Println("bvy length: ", len(bvy))
// Fixed
bfx := make([]byte, 8)
bfy := make([]byte, 8)
binary.LittleEndian.PutUint64(bfx, uint64(x))
binary.LittleEndian.PutUint64(bfy, uint64(y))
fmt.Println(bfx)
fmt.Println(bfy)
fmt.Println("bfx length: ", len(bfx))
fmt.Println("bfy length: ", len(bfy))
}
Playground: https://play.golang.org/p/XN46KafMY23
Output:
16
106547
Variant bytes written x: 1
Variant bytes written y: 3
[16]
[179 192 6]
bvx length: 1
bvy length: 3
[16 0 0 0 0 0 0 0]
[51 160 1 0 0 0 0 0]
bfx length: 8
bfy length: 8

How to add an ASCII value of a byte to an integer in golang?

I'd like to get a byte[] variables' asic value added to an integer. For example, I fisrt read all input in a byte[] buffer. And then I get out a number string "123" in it. And then I can assign it to an integer by ('1' - '0')*100 + ('2' - '0')*10 + '3' - '0'. But I can not assign integers with byte variables. How can I do that with any means? Thank you very much :)
In go, you can convert byte array to string with the string() coercion and then use strconv.Atoi on it. Presumably, you also want to use slice operations to isolate just the part of the input you want to convert.
package main
import (
"strconv"
"fmt"
)
func main() {
data := []byte { 0x20, 0x31, 0x32, 0x33, 0x20 } // Embedded number
// string(...) coercion yields a string from a byte buffer
// Number starts at char 1, ends before char 4
str := string(data[1:4])
i, err := strconv.Atoi(str)
if err != nil {
fmt.Printf("Error %v\n", err)
return
}
fmt.Printf("value %v\n", i)
}
Prints
value 123
And since go has nicely hygenic practicies, errors will be handled too.
If you need to read an integer from a stream of bytes, the fastest way would be just to scan it with fmt. Example:
n := 0
_, err := fmt.Scanf("%d", &n)
if err != nil {
log.Fatal(err)
}
fmt.Printf("you entered %d\n", n)

Go - Convert 2 byte array into a uint16 value

If I have a slice of bytes in Go, similar to this:
numBytes := []byte { 0xFF, 0x10 }
How would I convert it to it's uint16 value (0xFF10, 65296)?
you may use binary.BigEndian.Uint16(numBytes)
like this working sample code (with commented output):
package main
import (
"encoding/binary"
"fmt"
)
func main() {
numBytes := []byte{0xFF, 0x10}
u := binary.BigEndian.Uint16(numBytes)
fmt.Printf("%#X %[1]v\n", u) // 0XFF10 65296
}
and see inside binary.BigEndian.Uint16(b []byte):
func (bigEndian) Uint16(b []byte) uint16 {
_ = b[1] // bounds check hint to compiler; see golang.org/issue/14808
return uint16(b[1]) | uint16(b[0])<<8
}
I hope this helps.
To combine two bytes into uint16
x := uint16(numBytes[i])<<8 | uint16(numBytes[i+1])
where i is the starting position of the uint16. So if your array is always only two items it would be x := uint16(numBytes[0])<<8 | uint16(numBytes[1])
Firstly you have a slice not an array - an array has a fixed size and would be declared like this [2]byte.
If you just have a 2 bytes slice, I wouldn't do anything fancy, I'd just do
numBytes := []byte{0xFF, 0x10}
n := int(numBytes[0])<<8 + int(numBytes[1])
fmt.Printf("n =0x%04X = %d\n", n, n)
Playground
EDIT: Just noticed you wanted uint16 - replace int with that in the above!
You can use the following unsafe conversion:
*(*uint16)(unsafe.Pointer(&numBytes[0])

Convert an integer to a byte array

I have a function which receives a []byte but what I have is an int, what is the best way to go about this conversion ?
err = a.Write([]byte(myInt))
I guess I could go the long way and get it into a string and put that into bytes, but it sounds ugly and I guess there are better ways to do it.
I agree with Brainstorm's approach: assuming that you're passing a machine-friendly binary representation, use the encoding/binary library. The OP suggests that binary.Write() might have some overhead. Looking at the source for the implementation of Write(), I see that it does some runtime decisions for maximum flexibility.
func Write(w io.Writer, order ByteOrder, data interface{}) error {
// Fast path for basic types.
var b [8]byte
var bs []byte
switch v := data.(type) {
case *int8:
bs = b[:1]
b[0] = byte(*v)
case int8:
bs = b[:1]
b[0] = byte(v)
case *uint8:
bs = b[:1]
b[0] = *v
...
Right? Write() takes in a very generic data third argument, and that's imposing some overhead as the Go runtime then is forced into encoding type information. Since Write() is doing some runtime decisions here that you simply don't need in your situation, maybe you can just directly call the encoding functions and see if it performs better.
Something like this:
package main
import (
"encoding/binary"
"fmt"
)
func main() {
bs := make([]byte, 4)
binary.LittleEndian.PutUint32(bs, 31415926)
fmt.Println(bs)
}
Let us know how this performs.
Otherwise, if you're just trying to get an ASCII representation of the integer, you can get the string representation (probably with strconv.Itoa) and cast that string to the []byte type.
package main
import (
"fmt"
"strconv"
)
func main() {
bs := []byte(strconv.Itoa(31415926))
fmt.Println(bs)
}
Check out the "encoding/binary" package. Particularly the Read and Write functions:
binary.Write(a, binary.LittleEndian, myInt)
Sorry, this might be a bit late. But I think I found a better implementation on the go docs.
buf := new(bytes.Buffer)
var num uint16 = 1234
err := binary.Write(buf, binary.LittleEndian, num)
if err != nil {
fmt.Println("binary.Write failed:", err)
}
fmt.Printf("% x", buf.Bytes())
i thought int type has any method for getting int hash to bytes, but first i find math / big method for this
https://golang.org/pkg/math/big/
var f int = 52452356235; // int
var s = big.NewInt(int64(f)) // int to big Int
var b = s.Bytes() // big Int to bytes
// b - byte slise
var r = big.NewInt(0).SetBytes(b) // bytes to big Int
var i int = int(r.Int64()) // big Int to int
https://play.golang.org/p/VAKSGw8XNQq
However, this method uses an absolute value.
If you spend 1 byte more, you can transfer the sign
func IntToBytes(i int) []byte{
if i > 0 {
return append(big.NewInt(int64(i)).Bytes(), byte(1))
}
return append(big.NewInt(int64(i)).Bytes(), byte(0))
}
func BytesToInt(b []byte) int{
if b[len(b)-1]==0 {
return -int(big.NewInt(0).SetBytes(b[:len(b)-1]).Int64())
}
return int(big.NewInt(0).SetBytes(b[:len(b)-1]).Int64())
}
https://play.golang.org/p/mR5Sp5hu4jk
or new(https://play.golang.org/p/7ZAK4QL96FO)
(The package also provides functions for fill into an existing slice)
https://golang.org/pkg/math/big/#Int.FillBytes
Adding this option for dealing with basic uint8 to byte[] conversion
foo := 255 // 1 - 255
ufoo := uint16(foo)
far := []byte{0,0}
binary.LittleEndian.PutUint16(far, ufoo)
bar := int(far[0]) // back to int
fmt.Println("foo, far, bar : ",foo,far,bar)
output :
foo, far, bar : 255 [255 0] 255
Here is another option, based on the Go source code [1]:
package main
import (
"encoding/binary"
"fmt"
"math/bits"
)
func encodeUint(x uint64) []byte {
buf := make([]byte, 8)
binary.BigEndian.PutUint64(buf, x)
return buf[bits.LeadingZeros64(x) >> 3:]
}
func main() {
for x := 0; x <= 64; x += 8 {
buf := encodeUint(1<<x-1)
fmt.Println(buf)
}
}
Result:
[]
[255]
[255 255]
[255 255 255]
[255 255 255 255]
[255 255 255 255 255]
[255 255 255 255 255 255]
[255 255 255 255 255 255 255]
[255 255 255 255 255 255 255 255]
Much faster than math/big:
BenchmarkBig-12 28348621 40.62 ns/op
BenchmarkBit-12 731601145 1.641 ns/op
https://github.com/golang/go/blob/go1.16.5/src/encoding/gob/encode.go#L113-L117
You can try musgo_int. All you need to do is to cast your variable:
package main
import (
"github.com/ymz-ncnk/musgo_int"
)
func main() {
var myInt int = 1234
// from int to []byte
buf := make([]byte, musgo_int.Int(myInt).SizeMUS())
musgo_int.Int(myInt).MarshalMUS(buf)
// from []byte to int
_, err := (*musgo_int.Int)(&myInt).UnmarshalMUS(buf)
if err != nil {
panic(err)
}
}
Convert Integer to byte slice.
import (
"bytes"
"encoding/binary"
"log"
)
func IntToBytes(num int64) []byte {
buff := new(bytes.Buffer)
bigOrLittleEndian := binary.BigEndian
err := binary.Write(buff, bigOrLittleEndian, num)
if err != nil {
log.Panic(err)
}
return buff.Bytes()
}
Maybe the simple way is using protobuf, see the Protocol Buffer Basics: Go
define message like
message MyData {
int32 id = 1;
}
get more in Defining your protocol format
// Write
out, err := proto.Marshal(mydata)
read more in Writing a Message
Try math/big package to convert bytes array to int and to convert int to bytes array.
package main
import (
"fmt"
"math/big"
)
func main() {
// Convert int to []byte
var int_to_encode int64 = 65535
var bytes_array []byte = big.NewInt(int_to_encode).Bytes()
fmt.Println("bytes array", bytes_array)
// Convert []byte to int
var decoded_int int64 = new(big.Int).SetBytes(bytes_array).Int64()
fmt.Println("decoded int", decoded_int)
}
This is the most straight forward (and shortest (and safest) (and maybe most performant)) way:
buf.Bytes() is of type bytes slice.
var val uint32 = 42
buf := new(bytes.Buffer)
err := binary.Write(buf, binary.LittleEndian, val)
if err != nil {
fmt.Println("binary.Write failed:", err)
}
fmt.Printf("% x\n", buf.Bytes())
see also https://stackoverflow.com/a/74819602/589493
What's wrong with converting it to a string?
[]byte(fmt.Sprintf("%d", myint))

Resources