I have a packet structure and I wish to serialize it into binary so I can send it through the wire.
There are many packet structures but I'll give the login packet as an example:
login struct {
seq uint8
id uint16
username [16]string
password [16]string
unknown1 [16]byte
}
I've read somewhere that you can't use binary.Write for non-fixed size structures. But I believe my structure is fixed size (correct me if I'm wrong, I could be very wrong).
Now using this code:
var buf bytes.Buffer
x := login{
seq: 2,
id: 1,
username: [16]string{"username"},
password: [16]string{"password"},
}
err := binary.Write(&buf, binary.LittleEndian, x)
if err != nil {
log.Fatalln(err)
}
Gives me the error: binary.Write: invalid type main.login
Now, is there a way to fix this? Is there an alternative approach? Much like how you can use structs in C and send it through the network.
You have two errors in your code. Firstly, your struct fields should be exported for encoding/binary to see it.
Secondly, [16]string means an array of 16 strings, not a 16-byte string. So your struct should look like this:
type login struct {
Seq uint8
ID uint16
Username [16]byte
Password [16]byte
Unknown1 [16]byte
}
Then it works: https://play.golang.org/p/Nq8fRqgkcp.
You can do that using the unsafe package.
type test struct {
a int
s string
}
v1 := test{
a: 5,
s: "sdaf",
}
fmt.Printf("v1: %#v\n", v1)
b := *(*[unsafe.Sizeof(v1)]byte)(unsafe.Pointer(&v1))
fmt.Printf("bytes: %#v\n", b)
v2 := *(*test)(unsafe.Pointer(&b))
fmt.Printf("v2: %#v\n", v2)
Related
I'd like to use amino marshal when I have private variables in the structure.
In test2 package, test.go :
type Lnt struct {
Neg bool
abs string // this variable
}
func NewLnt() Lnt {
return Lnt{
Neg: true,
abs: "abcdefef",
}
}
testing go file :
func TestAbc(t *testing.T) {
s := test2.NewLnt()
t.Log("s=", s)
cdc := amino.NewCodec()
b, err := cdc.MarshalBinary(s)
assert.Nil(t, err)
t.Log("b=",b)
var s2 test2.Lnt
err = cdc.UnmarshalBinary(b, &s2)
assert.Nil(t, err)
assert.Equal(t, s, s2)
t.Log("s2=", s2)
}
result :
encoding_test.go:39: s= {true abcdefef}
encoding_test.go:55:
Error Trace: encoding_test.go:55
Error: Not equal:
expected: test2.Lnt{Neg:true, abs:"abcdefef"}
actual : test2.Lnt{Neg:true, abs:""} // error
Diff:
--- Expected
+++ Actual
## -2,3 +2,3 ##
Neg: (bool) true,
- abs: (string) (len=8) "abcdefef"
+ abs: (string) ""
}
Test: TestAbc
encoding_test.go:57: s2= {true }
Private variable "abs" is lost..
Is it not supported, or is there another way to use it in this case?
The short answer is you can't.
What's happening here is you are marshalling all the exported values into the binary format, but the unexported values are not included because the marshaler doesn't have access to them.
The binary data is the unmarshalled into a new struct, and because the unexported field was not in the binary data, it's impossible for the struct to get initialised with that value. (also, it cannot have a value set because it's unexported).
You will need to export the struct field (make public) if you want this test to pass, or accept that that data is lost on marshalling.
I need some help with unmarshaling. I have this example code:
package main
import (
"encoding/json"
"fmt"
)
type Obj struct {
Id string `json:"id"`
Data []byte `json:"data"`
}
func main() {
byt := []byte(`{"id":"someID","data":["str1","str2"]}`)
var obj Obj
if err := json.Unmarshal(byt, &obj); err != nil {
panic(err)
}
fmt.Println(obj)
}
What I try to do here - convert bytes to the struct, where type of one field is []byte. The error I get:
panic: json: cannot unmarshal string into Go struct field Obj.data of
type uint8
That's probably because parser already sees that "data" field is already a slice and tries to represent "str1" as some char bytecode (type uint8?).
How do I store the whole data value as one bytes array? Because I want to unmarshal the value to the slice of strings later. I don't include a slice of strings into struct because this type can change (array of strings, int, string, etc), I wish this to be universal.
My first recommendation would be for you to just use []string instead of []byte if you know the input type is going to be an array of strings.
If data is going to be a JSON array with various types, then your best option is to use []interface{} instead - Go will happily unmarshal the JSON for you and you can perform checks at runtime to cast those into more specific typed variables on an as-needed basis.
If []byte really is what you want, use json.RawMessage, which is of type []byte, but also implements the methods for JSON parsing. I believe this may be what you want, as it will accept whatever ends up in data. Of course, you then have to manually parse Data to figure out just what actually IS in there.
One possible bonus is that this skips any heavy parsing because it just copies the bytes over. When you want to use this data for something, you use a []interface{}, then use a type switch to use individual values.
https://play.golang.org/p/og88qb_qtpSGJ
package main
import (
"encoding/json"
"fmt"
)
type Obj struct {
Id string `json:"id"`
Data json.RawMessage `json:"data"`
}
func main() {
byt := []byte(`{"id":"someID","data":["str1","str2", 1337, {"my": "obj", "id": 42}]}`)
var obj Obj
if err := json.Unmarshal(byt, &obj); err != nil {
panic(err)
}
fmt.Printf("%+v\n", obj)
fmt.Printf("Data: %s\n", obj.Data)
// use it
var d []interface{}
if err := json.Unmarshal(obj.Data, &d); err != nil {
panic(err)
}
fmt.Printf("%+v\n", d)
for _, v := range d {
// you need a type switch to deterine the type and be able to use most of these
switch real := v.(type) {
case string:
fmt.Println("I'm a string!", real)
case float64:
fmt.Println("I'm a number!", real)
default:
fmt.Printf("Unaccounted for: %+v\n", v)
}
}
}
Your question is:
convert bytes array to struct with a field of type []byte
But you do not have a bytearray but a string array. Your question is not the same as your example. So let answer your question, there are more solutions possible depending in how far you want to diverge from your original requirements.
One string can be converted to one byte-slice, two strings need first to be transformed to one string. So that is problem one. The second problem are the square-brackets in your json-string
This works fine, it implicitly converts the string in the json-string to a byte-slice:
byt := []byte(`{"id":"someID","data":"str1str2"}`)
var obj Obj
if err := json.Unmarshal(byt, &obj); err != nil {
panic(err)
}
fmt.Println(obj)
I'm new to golang and trying to figure out the correct way of casting a block of bytes to the correct struct. All structs start with two bytes that dictate the layout of the remaining bytes. In C I would point to the beginning of the block of memory and cast it as a simple struct that only contained those two bytes (X below) but here I get an invalid type assertion. I'm probably way off base here any help you be appreciated.
package main
import (
"fmt"
)
type A struct {
tag byte
ver byte
data1 int
data2 int
data3 int
}
type B struct {
tag byte
ver byte
data1 float32
}
type X struct {
tag byte
ver byte
}
func main() {
var a A
a.tag = 1
a.ver = 1
x := a.(X)
fmt.Printf("%d,%d", x.tag, x.ver)
}
Edit
In short I just want to create a custom method on type Foo that is the only reason why I want to perform the cast. If the solutions are very complex I will just create functions instead of methods I guess. I was just curios.
playground link
Here's my solution. It involves a few different tips:
Embed the shared struct in the individual structs.
Use encoding/binary package to load bytes into structs.
Fill header struct with first two bytes, then make a decision on which subtype to make and fill.
Always use fixed length int types for this kind of thing.
Your field names must be UpperCase to be fillable from encoding/binary
This is a pretty brittle way to manage marshalling.unmarshalling of data, but I'm sure you know that.
Here's my solution:
package main
import (
"bytes"
"encoding/binary"
"fmt"
"log"
)
type A struct {
X
Data1 int32
Data2 int32
Data3 int32
}
type B struct {
X
Data1 int32
}
type X struct {
Tag byte
Ver byte
}
func main() {
var err error
data := []byte{1, 1, 0, 0, 0, 42}
hdr := X{}
err = binary.Read(bytes.NewReader(data[:2]), binary.BigEndian, &hdr)
if err != nil {
log.Fatal(err)
}
fmt.Println(hdr.Tag, hdr.Ver)
if hdr.Tag == 1 {
b := B{}
err = binary.Read(bytes.NewReader(data), binary.BigEndian, &b)
if err != nil {
log.Fatal(err)
}
fmt.Println(b.Data1)
}
}
playground link
Go generally tries to discourage C-like memory fiddling as it leads to memory leaks, incorrect behavior, and security vulnerabilities unless extraordinary caution and testing are applied. That doesn't mean it's impossible though; in fact, the aptly-named unsafe.Pointer is exposed for exactly this purpose. Use it with caution.
I'm looking for some solution to know whats the struct type of the hash. It is possible to do that without try an error method (casting to a specific type and see the cast is successfully)?
Please check the code:
import (
"bytes"
"encoding/binary"
"fmt"
"reflect"
)
type T struct {
A int64
B float64
}
type D struct {
A int64
B float64
C string
}
func main() {
// Create a struct and write it.
t := T{A: 0xEEFFEEFF, B: 3.14}
buf := &bytes.Buffer{}
err := binary.Write(buf, binary.BigEndian, t)
if err != nil {
panic(err)
}
fmt.Println(buf.Bytes())
out := getType(buf)
fmt.Println(out)
}
func getType(v interface{})(r string){
fmt.Println(reflect.TypeOf(v))
switch t := v.(type) {
case T:
return "Is type T"
case D:
return "Is type D"
default:
_ = t
return "unknown"
}
}
Since the encoding/binary package does not write out type information, it is not possible to tell what type was written / serialized.
And you're in a worse position that you might originally think: even trying to decode into a value of different type might succeed without errors, so there isn't even a reliable way to tell the type.
For example if you serialize a value of this type:
type T struct {
A int64
B float64
}
You can read it into a value of this type:
type T2 struct {
B float64
A int64
}
It will give no errors because the size of both structs is the same, but obviously you will get different numbers in the fields.
You are in a little better position if you use encoding/gob, as the gob package does transmit type information, and encoding a value of type T and then decoding it into a value of type T2 would work: order of fields does not matter, and extra or missing fields also do not cause trouble.
See this example:
// Create a struct and write it.
t := T{A: 0xEEFFEEFF, B: 3.14}
fmt.Println("Encoding:", t)
buf := &bytes.Buffer{}
fmt.Println(binary.Write(buf, binary.BigEndian, t))
fmt.Println(buf.Bytes())
fmt.Println(gob.NewEncoder(buf).Encode(t))
t2 := T2{}
fmt.Println(binary.Read(buf, binary.BigEndian, &t2))
fmt.Println(t2)
t2 = T2{}
fmt.Println(gob.NewDecoder(buf).Decode(&t2))
fmt.Println(t2)
Output (try it on the Go Playground):
Encoding: {4009750271 3.14}
<nil>
[0 0 0 0 238 255 238 255 64 9 30 184 81 235 133 31]
<nil>
<nil>
{1.9810798573e-314 4614253070214989087}
<nil>
{3.14 4009750271}
If you want to be able to detect the type before reading it, you have to take care of it yourself: you have to transmit type information (e.g. name of the type). Or even better, use a serialization method that already does this, for example Google's protocol buffers, and here is the Go implementation for it: github.com/golang/protobuf.
I have the following JSON, and I want to parse it into array of class:
{
"1001": {"level":10, "monster-id": 1001, "skill-level": 1, "aimer-id": 301}
"1002": {"level":12, "monster-id": 1002, "skill-level": 1, "aimer-id": 302}
"1003": {"level":16, "monster-id": 1003, "skill-level": 2, "aimer-id": 303}
}
Here is what i am trying to do but failed:
type Monster struct {
MonsterId int32
Level int32
SkillLevel int32
AimerId int32
}
type MonsterCollection struct {
Pool map[string]Monster
}
func (mc *MonsterCollection) FromJson(jsonStr string) {
var data interface{}
b := []byte(jsonStr)
err := json.Unmarshal(b, &data)
if err != nil {
return
}
m := data.(map[string]interface{})
i := 0
for k, v := range m {
monster := new(Monster)
monster.Level = v["level"]
monster.MonsterId = v["monster-id"]
monster.SkillLevel = v["skill-level"]
monster.AimerId = v["aimer-id"]
mc.Pool[i] = monster
i++
}
}
The compiler complain about the v["level"]
<< invalid operation. index of type interface().
This code has many errors in it. To start with, the json isn't valid json. You are missing the commas in between key pairs in your top level object. I added the commas and pretty printed it for you:
{
"1001":{
"level":10,
"monster-id":1001,
"skill-level":1,
"aimer-id":301
},
"1002":{
"level":12,
"monster-id":1002,
"skill-level":1,
"aimer-id":302
},
"1003":{
"level":16,
"monster-id":1003,
"skill-level":2,
"aimer-id":303
}
}
Your next problem (the one you asked about) is that m := data.(map[string]interface{}) makes m a map[string]interface{}. That means when you index it such as the v in your range loop, the type is interface{}. You need to type assert it again with v.(map[string]interface{}) and then type assert each time you read from the map.
I also notice that you next attempt mc.Pool[i] = monster when i is an int and mc.Pool is a map[string]Monster. An int is not a valid key for that map.
Your data looks very rigid so I would make unmarshall do most of the work for you. Instead of providing it a map[string]interface{}, you can provide it a map[string]Monster.
Here is a quick example. As well as changing how the unmarshalling works, I also added an error return. The error return is useful for finding bugs. That error return is what told me you had invalid json.
type Monster struct {
MonsterId int32 `json:"monster-id"`
Level int32 `json:"level"`
SkillLevel int32 `json:"skill-level"`
AimerId int32 `json:"aimer-id"`
}
type MonsterCollection struct {
Pool map[string]Monster
}
func (mc *MonsterCollection) FromJson(jsonStr string) error {
var data = &mc.Pool
b := []byte(jsonStr)
return json.Unmarshal(b, data)
}
I posted a working example to goplay: http://play.golang.org/p/4EaasS2VLL
Slightly off to one side - you asked for an array of objects when you needed a map
If you need an array (actually a slice)
http://ioblocks.blogspot.com/2014/09/loading-arrayslice-of-objects-from-json.html