I am able to convert []byte into unsigned integers:
a := binary.LittleEndian.Uint16(sampleA)
b := binary.BigEndian.Uint32(sampleB)
This leverages the BigEndian and LittleEndian types within the Go package https://golang.org/src/encoding/binary/binary.go.
This provides Uint16() however there are no equivalent Int16() or Float32().
Any thoughts on why not? Also, how ought this be done?
Converting numeric types into a series of bytes ([]byte) and vice versa is about the endianness. How you interpret the result is entirely up to you.
All you need is to assemble a 16-bit, 32-bit or 64-bit value, once it's done, you can interpret the result as you want.
For example if you already have a uint16 value, to use it as a signed value, all you need is a type conversion because the memory layout of an uint16 and int16 is the same (converting from uint16 to int16 doesn't change the memory representation just the type):
a := binary.LittleEndian.Uint16(sampleA)
// If you need int16:
a2 := int16(a)
Similarly:
a := binary.LittleEndian.Uint64(sampleA)
// If you need int64:
a2 := int64(a)
The situation is a little more complicated with uint -> float conversion as using a simple type conversion would try to convert the numeric value and not just change the type (and thus would change the memory representation).
For converting unsigned integers to float types, you can use functions of the math package, namely math.Float32frombits() and math.Float64frombits(), and for the reverse direction (converting a float value to an unsigned integer) having the same memory layout: math.Float32bits() and math.Float64bits().
For example:
a := binary.LittleEndian.Uint64(sampleA)
// If you need a float64:
a2 := math.Float64frombits(a)
If you would look into the implementation of these functions from the math package, you can see that the memory value/layout is not manipulated, it is just "viewed" as a different type, by using the unsafe package. For example:
func Float32frombits(b uint32) float32 { return *(*float32)(unsafe.Pointer(&b)) }
As mentioned by Paul, the binary package provides Read() and Write() functions to do these conversions under the hood so you don't need to.
Showcasing using the same "pi" example (from the doc of binary.Read()):
b := []byte{0x18, 0x2d, 0x44, 0x54, 0xfb, 0x21, 0x09, 0x40}
// USING binary.Read()
var pi float64
buf := bytes.NewReader(b)
err := binary.Read(buf, binary.LittleEndian, &pi)
if err != nil {
fmt.Println("binary.Read failed:", err)
}
fmt.Println(pi)
// Using LittleEndian.Uint64() and math.Float64frombits()
a := binary.LittleEndian.Uint64(b)
a2 := math.Float64frombits(a)
fmt.Println(a2)
Output (try it on the Go Playground):
3.141592653589793
3.141592653589793
The ByteOrder type provides a low-level API for decoding binary values. To read float64 or other types, you can use binary.Read. There's an example on the godoc page for the binary package, which I've copied here:
var pi float64
b := []byte{0x18, 0x2d, 0x44, 0x54, 0xfb, 0x21, 0x09, 0x40}
buf := bytes.NewReader(b)
err := binary.Read(buf, binary.LittleEndian, &pi)
if err != nil {
fmt.Println("binary.Read failed:", err)
}
fmt.Print(pi)
There's no functions for decoding float16, because that's not a type in Go.
Related
This is strange but I cannot convert byte value into int.
Here is the code:
fmt.Println("numMsgsByte is:", numMsgsByte)
numMsgsStr := string(numMsgsByte)
fmt.Println("numMsgsStr is:", numMsgsStr)
numMsgs, err = strconv.Atoi(numMsgsStr)
if err != nil {
log.Println("error in msg conversion", err)
return 0
}
The terminal print out:
numMsgsByte is: [5]
numMsgsStr is:
counter.go:51: error in msg conversion strconv.Atoi: parsing "\x05": invalid syntax
What could be wrong here? How can I fix it?
numMsgByte is not a byte, it is a []byte, which contains 5 (not "5"). When you convert it to string using string(numMsgByte), you get a string "\x5".
What you need is: int(numMsgType[0])
The correct way to get an integer out of Redis using redigo would be to just use redis.Int in the first place, instead of using redis.Bytes and trying to convert the result to an int yourself.
In general though, to convert an arbitrary byte array to the integer it represents, you would use the encoding/binary package. You'll need to know some key details about the byte array though:
Does it represent a signed or unsigned value?
Of what width? 32 bit? 64 bit?
Of what byte order? Big-endian? Little-endian?
Are you sure it's represented as an integer, not e.g. a float, or a string representation of a number?
To quote the example from the docs:
b := []byte{0xe8, 0x03, 0xd0, 0x07}
x1 := binary.LittleEndian.Uint16(b[0:])
x2 := binary.LittleEndian.Uint16(b[2:])
fmt.Printf("%#04x %#04x\n", x1, x2)
I have a function that simply reads a file with ioutil.ReadFile(). The type returned is []byte, although the value itself can be represented as a float.
I am converting the []byte in this manner (where value is the []byte being returned from a function that reads a file):
var floatValue float64
fmt.Fscanf(bytes.NewReader(value), "%f", &floatValue)
Is this really the only way to extract/parse a valid float value from a []byte? There's a similar discussion but looks like it didn't really go anywhere.
You can easily use strconv.ParseFloat for this, just converting your []byte to a string first. This would surely have less overhead than creating a reader and scanning with a scanf-like function.
sb := []byte("3.1415")
s := string(sb)
f, err := strconv.ParseFloat(s, 64)
if err != nil {
panic("whoops!")
}
fmt.Printf("%f\n", f)
Output:
3.141500
I have a packet structure and I wish to serialize it into binary so I can send it through the wire.
There are many packet structures but I'll give the login packet as an example:
login struct {
seq uint8
id uint16
username [16]string
password [16]string
unknown1 [16]byte
}
I've read somewhere that you can't use binary.Write for non-fixed size structures. But I believe my structure is fixed size (correct me if I'm wrong, I could be very wrong).
Now using this code:
var buf bytes.Buffer
x := login{
seq: 2,
id: 1,
username: [16]string{"username"},
password: [16]string{"password"},
}
err := binary.Write(&buf, binary.LittleEndian, x)
if err != nil {
log.Fatalln(err)
}
Gives me the error: binary.Write: invalid type main.login
Now, is there a way to fix this? Is there an alternative approach? Much like how you can use structs in C and send it through the network.
You have two errors in your code. Firstly, your struct fields should be exported for encoding/binary to see it.
Secondly, [16]string means an array of 16 strings, not a 16-byte string. So your struct should look like this:
type login struct {
Seq uint8
ID uint16
Username [16]byte
Password [16]byte
Unknown1 [16]byte
}
Then it works: https://play.golang.org/p/Nq8fRqgkcp.
You can do that using the unsafe package.
type test struct {
a int
s string
}
v1 := test{
a: 5,
s: "sdaf",
}
fmt.Printf("v1: %#v\n", v1)
b := *(*[unsafe.Sizeof(v1)]byte)(unsafe.Pointer(&v1))
fmt.Printf("bytes: %#v\n", b)
v2 := *(*test)(unsafe.Pointer(&b))
fmt.Printf("v2: %#v\n", v2)
I'm looking for some solution to know whats the struct type of the hash. It is possible to do that without try an error method (casting to a specific type and see the cast is successfully)?
Please check the code:
import (
"bytes"
"encoding/binary"
"fmt"
"reflect"
)
type T struct {
A int64
B float64
}
type D struct {
A int64
B float64
C string
}
func main() {
// Create a struct and write it.
t := T{A: 0xEEFFEEFF, B: 3.14}
buf := &bytes.Buffer{}
err := binary.Write(buf, binary.BigEndian, t)
if err != nil {
panic(err)
}
fmt.Println(buf.Bytes())
out := getType(buf)
fmt.Println(out)
}
func getType(v interface{})(r string){
fmt.Println(reflect.TypeOf(v))
switch t := v.(type) {
case T:
return "Is type T"
case D:
return "Is type D"
default:
_ = t
return "unknown"
}
}
Since the encoding/binary package does not write out type information, it is not possible to tell what type was written / serialized.
And you're in a worse position that you might originally think: even trying to decode into a value of different type might succeed without errors, so there isn't even a reliable way to tell the type.
For example if you serialize a value of this type:
type T struct {
A int64
B float64
}
You can read it into a value of this type:
type T2 struct {
B float64
A int64
}
It will give no errors because the size of both structs is the same, but obviously you will get different numbers in the fields.
You are in a little better position if you use encoding/gob, as the gob package does transmit type information, and encoding a value of type T and then decoding it into a value of type T2 would work: order of fields does not matter, and extra or missing fields also do not cause trouble.
See this example:
// Create a struct and write it.
t := T{A: 0xEEFFEEFF, B: 3.14}
fmt.Println("Encoding:", t)
buf := &bytes.Buffer{}
fmt.Println(binary.Write(buf, binary.BigEndian, t))
fmt.Println(buf.Bytes())
fmt.Println(gob.NewEncoder(buf).Encode(t))
t2 := T2{}
fmt.Println(binary.Read(buf, binary.BigEndian, &t2))
fmt.Println(t2)
t2 = T2{}
fmt.Println(gob.NewDecoder(buf).Decode(&t2))
fmt.Println(t2)
Output (try it on the Go Playground):
Encoding: {4009750271 3.14}
<nil>
[0 0 0 0 238 255 238 255 64 9 30 184 81 235 133 31]
<nil>
<nil>
{1.9810798573e-314 4614253070214989087}
<nil>
{3.14 4009750271}
If you want to be able to detect the type before reading it, you have to take care of it yourself: you have to transmit type information (e.g. name of the type). Or even better, use a serialization method that already does this, for example Google's protocol buffers, and here is the Go implementation for it: github.com/golang/protobuf.
Does anybody know how to get memory size of the variable (int, string, []struct, etc) and print it? Is it possible?
var i int = 1
//I want to get something like this:
fmt.Println("Size of i is: %?", i)
//Also, it would be nice if I could store the value into a string
You can use the unsafe.Sizeof function for this.
It returns the size in bytes, occupied by the value you pass into it.
Here's a working example:
package main
import "fmt"
import "unsafe"
func main() {
a := int(123)
b := int64(123)
c := "foo"
d := struct {
FieldA float32
FieldB string
}{0, "bar"}
fmt.Printf("a: %T, %d\n", a, unsafe.Sizeof(a))
fmt.Printf("b: %T, %d\n", b, unsafe.Sizeof(b))
fmt.Printf("c: %T, %d\n", c, unsafe.Sizeof(c))
fmt.Printf("d: %T, %d\n", d, unsafe.Sizeof(d))
}
Take note that some platforms explicitly disallow the use of unsafe, because it is.. well, unsafe. This used to include AppEngine. Not sure if that is still the case today, but I imagine so.
As #Timur Fayzrakhmanov notes, reflect.TypeOf(variable).Size() will give you the same information. For the reflect package, the same restriction goes as for the unsafe package. I.e.: some platforms may not allow its use.
You can do it with either unsafe.Sizeof(), or reflect.Type.Size()
The size of a variable can be determined by using unsafe.Sizeof(a). The result will remain the same for a given type (i.e. int, int64, string, struct etc), irrespective of the value it holds. However, for type string, you may be interested in the size of the string that the variable references, and this is determined by using len(a) function on a given string. The following snippet illustrates that size of a variable of type string is always 8 but the length of a string that a variable references can vary:
package main
import "fmt"
import "unsafe"
func main() {
s1 := "foo"
s2 := "foobar"
fmt.Printf("s1 size: %T, %d\n", s1, unsafe.Sizeof(s1))
fmt.Printf("s2 size: %T, %d\n", s2, unsafe.Sizeof(s2))
fmt.Printf("s1 len: %T, %d\n", s1, len(s1))
fmt.Printf("s2 len: %T, %d\n", s2, len(s2))
}
Output:
s1 size: string, 8
s2 size: string, 8
s1 len: string, 3
s2 len: string, 6
The last part of your question is about assigning the length (i.e. an int value) to a string. This can be done by s := strconv.Itoa(i) where i is an int variable and the string returned by the function is assigned to s.
Note: the name of the converter function is Itoa, possibly a short form for Integer to ASCII. Most Golang programmers are likely to misread the function name as Iota.
I've written a package which calculates the actual memory size consumed by variable at runtime: https://github.com/DmitriyVTitov/size
It has single function, so basic usage is:
fmt.Println(size.Of(varName))
unsafe.Sizeof() is the correct solution.
var i int
var u uint
var up uintptr
fmt.Printf("i Type:%T Size:%d\n", i, unsafe.Sizeof(i))
fmt.Printf("u Type:%T Size:%d\n", u, unsafe.Sizeof(u))
fmt.Printf("up Type:%T Size:%d\n", up, unsafe.Sizeof(up))
The int, uint, and uintptr types are usually 32 bits wide on 32-bit systems and 64 bits wide on 64-bit systems. When you need an integer value you should use int unless you have a specific reason to use a sized or unsigned integer type.