Why go json.Unmarshal auto convert interface{} to map - go

The program will receive many msg, msg has different struct "Data", so I define the Msg struct:
type Msg struct {
MsgType int
Data interface{}
}
type Data1 struct {
//msg type 1 Data struct
}
type Data2 struct {
//msg type 2 Data struct
}
func (msgStr string) {
msg := Msg{}
if err := json.Unmarshal([]byte(msgStr), &msg); err != nil {
//log err
}
switch msg.MsgType{
case 1:
//convert msg.Data to a type 1 struct
case 2:
//convert msg.Data to a type 2 struct
}
}
But print out the msg.Data, it is a map, not interface{}, so when I convert it to Data1 by msg.Data.(Data1), got an err.
So,
1. Why interface{} auto convert to map?
2. How to convert it to Data1 struct I want?
3. What is the best practices in this scenes.

1. Because it sees a JSON object, and, as documented, a JSON object becomes a map[string]interface{} when stored into an interface{} (this is the only type that can hold whatever is in a JSON object in generality).
2. Given your current situation, you could assign each field of the map to the appropriate field of a new Data1 or Data2.
3. The ideal way to handle this is to use json.RawMessage to defer the decoding of Data until you know what it is. This can be handled like so:
type Msg struct {
MsgType int
Data interface{}
}
func (m *Msg) UnmarshalJSON(b []byte) (err error) {
var tmp struct {
MsgType int
Data json.RawMessage
}
err = json.Unmarshal(b, &tmp)
if err != nil {
return
}
m.MsgType = tmp.MsgType
switch (tmp.MsgType) {
case 1:
data := Data1{}
err = json.Unmarshal(tmp.Data, &data)
if err != nil {
return
}
m.Data = data
case 2:
data := Data2{}
err = json.Unmarshal(tmp.Data, &data)
if err != nil {
return
}
m.Data = data
default:
return errors.New("invalid DataType")
}
return
}
And then you can call json.Unmarshal or json.Decode directly on a *Msg and its Data will be decoded as you want.

Related

Changing value of field in a go struct

I'm making an http request in golang to an external api. It gives a general response of {"error":[]string, "result":changing interface{}}. depending on the function that is making the request, the Result field changes. Since I know the structure of the Result field for each function I run, I want to be able to change the value of Result before unmarshalling to json. I've tried to do this with the following code:
func GetAssets(output *Resp, resultType interface{}) error {
return publicRequest("/Assets", output, resultType)
}
func publicRequest(endPoint string, output *Resp, resultType interface{}) error {
url := Rest_url + Pub_rest_url + endPoint //"https://api.kraken.com/0/public/Assets in this case
resp, err := http.Get(url)
if err != nil {
return err
}
defer resp.Body.Close()
output.Result = resultType
return json.NewDecoder(resp.Body).Decode(&output)
}
Here is how it's being ran in main
type Resp struct {
Error []string `json:"error"`
Result interface{} `json:"result"`
}
type AssetInfo struct {
Aclass string `json:"aclass"`
Altname string `json:"altname"`
Decimals int `json:"decimals"`
Display int `json:"display_decimals"`
}
func main() {
var result map[string]AssetInfo
jsonData := Resp{}
rest_api_client.GetAssets(&jsonData, result)
fmt.Println(jsonData)
}
The issue is that it doesn't unmarshal correctly. A map is created for each asset, but the data contained inside of each asset is also being stored inside of a map. I'm not sure if I explained this well, but here is the current response after unmarshalling to understand what I mean.
Here is the data type of Resp.Result: map[string]interface {}
{[] map[1INCH:map[aclass:currency altname:1INCH decimals:10 display_decimals:5] AAVE:map[aclass:currency altname:AAVE decimals:10 display_decimals:5] ACA:map[aclass:currency altname:ACA decimals:10 display_decimals:5] ADA:map[aclass:currency altname:ADA decimals:8 display_decimals:6]...}
The response type I'm looking for is map[string]AssetInfo. Hopefully it could be unmarshalled like this:
{[] map[1INCH:{currency 1INCH 10 5} AAVE:{currency AAVE 10 5} ACA:{currency ACA 10 5} ADA:{currency ADA 8 6} ADA.S:{currency ADA.S 8 6}...}
Any help? I'd rather keep the Resp struct as generic as possible and just change the value of the Result field (if this is even possible to do correctly) since I plan to have multiple functions that call different endpoints of the api, and they'll all have the same underlying response type of the Resp struct with different Result types
You can view a working example in the following repo:
https://github.com/alessiosavi/GoArbitrage/blob/e107af466852b1ed30c2413eb4401595f7412b4f/markets/kraken/kraken.go
Basically, I've defined the following structure:
type Tickers struct {
Error []interface{} `json:"error"`
Result map[string]Ticker `json:"result"`
}
type Ticker struct {
Aclass string `json:"aclass"`
Altname string `json:"altname"`
Decimals int `json:"decimals"`
DisplayDecimals int `json:"display_decimals"`
}
Than I execute the request in the following way:
const KRAKEN_TICKERS_URL string = `https://api.kraken.com/0/public/Assets`
type Kraken struct {
PairsNames []string `json:"pairs_name"`
Pairs map[string]datastructure.KrakenPair `json:"pairs"`
OrderBook map[string]datastructure.KrakenOrderBook `json:"orderbook"`
MakerFee float64 `json:"maker_fee"`
TakerFees float64 `json:"taker_fee"`
// FeePercent is delegated to save if the fee is in percent or in coin
FeePercent bool `json:"fee_percent"`
Tickers []string
}
// Init is delegated to initialize the maps for the kraken
func (k *Kraken) Init() {
k.Pairs = make(map[string]datastructure.KrakenPair)
k.OrderBook = make(map[string]datastructure.KrakenOrderBook)
k.SetFees()
}
// SetFees is delegated to initialize the fee type/amount for the given market
func (k *Kraken) SetFees() {
k.MakerFee = 0.16
k.TakerFees = 0.26
k.FeePercent = true
}
func (k *Kraken) GetTickers() error {
res := datastructure.Tickers{}
var err error
var request req.Request
var data []byte
var tickers []string
resp := request.SendRequest(KRAKEN_TICKERS_URL, "GET", nil, nil, false, 10*time.Second)
if resp.Error != nil {
zap.S().Debugw("Error during http request. Err: " + resp.Error.Error())
return resp.Error
}
if resp.StatusCode != 200 {
zap.S().Warnw("Received a non 200 status code: " + strconv.Itoa(resp.StatusCode))
return errors.New("NON_200_STATUS_CODE")
}
data = resp.Body
if err = json.Unmarshal(data, &res); err != nil {
zap.S().Warn("ERROR! :" + err.Error())
return err
}
zap.S().Infof("Data: %v", res.Result)
tickers = make([]string, len(res.Result))
i := 0
for key := range res.Result {
tickers[i] = res.Result[key].Altname
i++
}
k.Tickers = tickers
return nil
}

How to unmarshal JSOn with an array of different types

I have JSON like this that I need to parse into a golang type:
{
name: "something"
rules: [
{
"itemTypeBasedConditions": [["containsAny", ["first_match", "second_match"]]],
"validity": "INVALID"
}]
}
The problem is that each array of the array in itemTypeBasedConditions contains a mix of strings (always first element) and another array (second element), and I am not sure how to parse all of that into an object that I could then manipulate.
I got to:
type RulesFile struct {
Name string
Rules []RulesItem
}
type RulesItem struct {
itemTypeBasedConditions [][]interface{}
validity bool
}
And then I guess I have to convert elements one by one from interface{} to either string (containsAny) or an array of strings ("first_match", "second_match")
Is there a better way of approaching this JSON parsing?
I would do something like this, you can probably alter this to your needs.
package main
import (
"encoding/json"
"fmt"
"os"
"reflect"
)
type RulesFile struct {
Name string `json:"name"`
Rules []RulesItem `json:"rules"`
}
type RulesItem struct {
ItemTypeBasedConditions [][]Condition `json:"itemTypeBasedConditions"`
Validity bool `json:"validity"`
}
type Condition struct {
Value *string
Array *[]string
}
func (c Condition) String() string {
if c.Value != nil {
return *c.Value
}
return fmt.Sprintf("%v", *c.Array)
}
func (c *Condition) UnmarshalJSON(data []byte) error {
var y interface{}
err := json.Unmarshal(data, &y)
if err != nil {
return err
}
switch reflect.TypeOf(y).String() {
case "string":
val := fmt.Sprintf("%v", y)
c.Value = &val
return nil
case "[]interface {}":
temp := y.([]interface{})
a := make([]string, len(temp))
for i, v := range temp {
a[i] = fmt.Sprint(v)
}
c.Array = &a
return nil
}
return fmt.Errorf("cannot unmarshall into string or []string: %v", y)
}
var input string = `
{
"name": "something",
"rules": [
{
"itemTypeBasedConditions": [["containsAny",["first_match", "second_match"]]],
"validity": false
}
]
}`
func main() {
var ruleFile RulesFile
err := json.Unmarshal([]byte(input), &ruleFile)
if err != nil {
fmt.Println(err)
os.Exit(1)
}
fmt.Printf("%+v\n", ruleFile)
}
You can implement the json.Unmarshaler interface. Have that implementation first unmarshal the json into a slice of json.RawMessage, then, once you've done that, you can unmarshal the individual elements to their corresponding types.
type Cond struct {
Name string
Args []string
}
func (c *Cond) UnmarshalJSON(data []byte) error {
// unmarshal into a slice of raw json
var raw []json.RawMessage
if err := json.Unmarshal(data, &raw); err != nil {
return err
} else if len(raw) != 2 {
return errors.New("unsupported number of elements in condition")
}
// unmarshal the first raw json element into a string
if err := json.Unmarshal(raw[0], &c.Name); err != nil {
return err
}
// unmarshal the second raw json element into a slice of string
return json.Unmarshal(raw[1], &c.Args)
}
https://go.dev/play/p/-tbr73TvX0d

Unmarshal method for complex object with go reflections

I'm starting to writing more complex go code, and my object node it to convert a list from a JSON object to a map with a particular key. This operation helps me to speed up my algorithm. But I have a problem now, my container struct has several complex JSON and I'm not able to write a generic solution to achieve a generic solution. The only way that I have in mind is to use a big switch case, but I think this is not the right solution.
This is my code at the moment, where the statusChannel is a map in the code but it is a list in the JSON string
type MetricOne struct {
// Internal id to identify the metric
id int `json:"-"`
// Version of metrics format, it is used to migrate the
// JSON payload from previous version of plugin.
Version int `json:"version"`
// Name of the metrics
Name string `json:"metric_name"`
NodeId string `json:"node_id"`
Color string `json:"color"`
OSInfo *osInfo `json:"os_info"`
// timezone where the node is located
Timezone string `json:"timezone"`
// array of the up_time
UpTime []*status `json:"up_time"`
// map of informatonof channel information
ChannelsInfo map[string]*statusChannel `json:"channels_info"`
}
func (instance *MetricOne) MarshalJSON() ([]byte, error) {
jsonMap := make(map[string]interface{})
reflectType := reflect.TypeOf(*instance)
reflectValue := reflect.ValueOf(*instance)
nFiled := reflectValue.Type().NumField()
for i := 0; i < nFiled; i++ {
key := reflectType.Field(i)
valueFiled := reflectValue.Field(i)
jsonName := key.Tag.Get("json")
switch jsonName {
case "-":
// skip
continue
case "channels_info":
// TODO convert the map[string]*statusChannel in a list of statusChannel
statusChannels := make([]*statusChannel, 0)
for _, value := range valueFiled.Interface().(map[string]*statusChannel) {
statusChannels = append(statusChannels, value)
}
jsonMap[jsonName] = statusChannels
default:
jsonMap[jsonName] = valueFiled.Interface()
}
}
return json.Marshal(jsonMap)
}
func (instance *MetricOne) UnmarshalJSON(data []byte) error {
var jsonMap map[string]interface{}
err := json.Unmarshal(data, &jsonMap)
if err != nil {
log.GetInstance().Error(fmt.Sprintf("Error: %s", err))
return err
}
instance.Migrate(jsonMap)
reflectValue := reflect.ValueOf(instance)
reflectStruct := reflectValue.Elem()
// reflectType := reflectValue.Type()
for key, value := range jsonMap {
fieldName, err := utils.GetFieldName(key, "json", *instance)
if err != nil {
log.GetInstance().Info(fmt.Sprintf("Error: %s", err))
if strings.Contains(key, "dev_") {
log.GetInstance().Info("dev propriety skipped if missed")
continue
}
return err
}
field := reflectStruct.FieldByName(*fieldName)
fieldType := field.Type()
filedValue := field.Interface()
val := reflect.ValueOf(filedValue)
switch key {
case "channels_info":
statusChannelsMap := make(map[string]*statusChannel)
toArray := value.([]interface{})
for _, status := range toArray {
var statusType statusChannel
jsonVal, err := json.Marshal(status)
if err != nil {
return err
}
err = json.Unmarshal(jsonVal, &statusType)
if err != nil {
return err
}
statusChannelsMap[statusType.ChannelId] = &statusType
}
field.Set(reflect.ValueOf(statusChannelsMap))
default:
field.Set(val.Convert(fieldType))
}
}
return nil
}
And when I will decode the object I receive the following error:
➜ go-metrics-reported git:(dev) ✗ make check
go test -v ./...
? github.com/OpenLNMetrics/go-metrics-reported/cmd/go-metrics-reported [no test files]
? github.com/OpenLNMetrics/go-metrics-reported/init/persistence [no test files]
=== RUN TestJSONSerializzation
--- PASS: TestJSONSerializzation (0.00s)
=== RUN TestJSONDeserializzation
--- FAIL: TestJSONDeserializzation (0.00s)
panic: reflect.Value.Convert: value of type map[string]interface {} cannot be converted to type *plugin.osInfo [recovered]
panic: reflect.Value.Convert: value of type map[string]interface {} cannot be converted to type *plugin.osInfo
goroutine 7 [running]:
testing.tRunner.func1.1(0x61b440, 0xc0001d69a0)
/home/vincent/.gosdk/go/src/testing/testing.go:1072 +0x30d
testing.tRunner.func1(0xc000001e00)
/home/vincent/.gosdk/go/src/testing/testing.go:1075 +0x41a
panic(0x61b440, 0xc0001d69a0)
/home/vincent/.gosdk/go/src/runtime/panic.go:969 +0x1b9
reflect.Value.Convert(0x6283e0, 0xc0001bb1a0, 0x15, 0x6b93a0, 0x610dc0, 0x610dc0, 0xc00014cb40, 0x196)
/home/vincent/.gosdk/go/src/reflect/value.go:2447 +0x229
github.com/OpenLNMetrics/go-metrics-reported/internal/plugin.(*MetricOne).UnmarshalJSON(0xc00014cb00, 0xc0001d8000, 0x493, 0x500, 0x7f04d01453d8, 0xc00014cb00)
/home/vincent/Github/OpenLNMetrics/go-metrics-reported/internal/plugin/metrics_one.go:204 +0x5b3
encoding/json.(*decodeState).object(0xc00010be40, 0x657160, 0xc00014cb00, 0x16, 0xc00010be68, 0x7b)
/home/vincent/.gosdk/go/src/encoding/json/decode.go:609 +0x207c
encoding/json.(*decodeState).value(0xc00010be40, 0x657160, 0xc00014cb00, 0x16, 0xc000034698, 0x54ec19)
/home/vincent/.gosdk/go/src/encoding/json/decode.go:370 +0x6d
encoding/json.(*decodeState).unmarshal(0xc00010be40, 0x657160, 0xc00014cb00, 0xc00010be68, 0x0)
/home/vincent/.gosdk/go/src/encoding/json/decode.go:180 +0x1ea
encoding/json.Unmarshal(0xc0001d8000, 0x493, 0x500, 0x657160, 0xc00014cb00, 0x500, 0x48cba6)
/home/vincent/.gosdk/go/src/encoding/json/decode.go:107 +0x112
github.com/OpenLNMetrics/go-metrics-reported/internal/plugin.TestJSONDeserializzation(0xc000001e00)
/home/vincent/Github/OpenLNMetrics/go-metrics-reported/internal/plugin/metric_one_test.go:87 +0x95
testing.tRunner(0xc000001e00, 0x681000)
/home/vincent/.gosdk/go/src/testing/testing.go:1123 +0xef
created by testing.(*T).Run
/home/vincent/.gosdk/go/src/testing/testing.go:1168 +0x2b3
FAIL github.com/OpenLNMetrics/go-metrics-reported/internal/plugin 0.008s
? github.com/OpenLNMetrics/go-metrics-reported/pkg/db [no test files]
? github.com/OpenLNMetrics/go-metrics-reported/pkg/graphql [no test files]
? github.com/OpenLNMetrics/go-metrics-reported/pkg/log [no test files]
? github.com/OpenLNMetrics/go-metrics-reported/pkg/utils [no test files]
FAIL
make: *** [Makefile:15: check] Error 1
can someone explain how I can do this operation in a generic way?
https://play.golang.org/p/jzU_lHj1wk7
type MetricOne struct {
// ...
// Have this field be ignored.
ChannelsInfo map[string]*statusChannel `json:"-"`
}
func (m MetricOne) MarshalJSON() ([]byte, error) {
// Declare a new type using the definition of MetricOne,
// the result of this is that M will have the same structure
// as MetricOne but none of its methods (this avoids recursive
// calls to MarshalJSON).
//
// Also because M and MetricOne have the same structure you can
// easily convert between those two. e.g. M(MetricOne{}) and
// MetricOne(M{}) are valid expressions.
type M MetricOne
// Declare a new type that has a field of the "desired" type and
// also **embeds** the M type. Embedding promotes M's fields to T
// and encoding/json will marshal those fields unnested/flattened,
// i.e. at the same level as the channels_info field.
type T struct {
M
ChannelsInfo []*statusChannel `json:"channels_info"`
}
// move map elements to slice
channels := make([]*statusChannel, 0, len(m.ChannelsInfo))
for _, c := range m.ChannelsInfo {
channels = append(channels, c)
}
// Pass in an instance of the new type T to json.Marshal.
// For the embedded M field use a converted instance of the receiver.
// For the ChannelsInfo field use the channels slice.
return json.Marshal(T{
M: M(m),
ChannelsInfo: channels,
})
}
// Same as MarshalJSON but in reverse.
func (m *MetricOne) UnmarshalJSON(data []byte) error {
type M MetricOne
type T struct {
*M
ChannelsInfo []*statusChannel `json:"channels_info"`
}
t := T{M: (*M)(m)}
if err := json.Unmarshal(data, &t); err != nil {
panic(err)
}
m.ChannelsInfo = make(map[string]*statusChannel, len(t.ChannelsInfo))
for _, c := range t.ChannelsInfo {
m.ChannelsInfo[c.ChannelId] = c
}
return nil
}

Problem with Marshal/unMarshal when key of map is a struct

I defined a struct named Student and a map named score.
Data structure is shown below:
type Student struct {
CountryID int
RegionID int
Name string
}
stu := Student{111, 222, "Tom"}
score := make(map[Student]int64)
score[stu] = 100
i am using json.Marshal to marshal score into json, but i cannot use json.Unmarshal to unmarshal this json. Below is my code. i am using function GetMarshableObject to translate struct Student into string which is marshable.
Could anyone tell me how to deal with this json to unmarshal it back to map score.
package main
import (
"encoding/json"
"fmt"
"os"
"reflect"
)
type Student struct {
CountryID int
RegionID int
Name string
}
func GetMarshableObject(src interface{}) interface{} {
t := reflect.TypeOf(src)
v := reflect.ValueOf(src)
kind := t.Kind()
var result reflect.Value
switch kind {
case reflect.Map:
//Find the map layer count
layer := 0
cur := t.Elem()
for reflect.Map == cur.Kind() {
layer++
cur = cur.Elem()
}
result = reflect.MakeMap(reflect.MapOf(reflect.TypeOf("a"), cur))
for layer > 0 {
result = reflect.MakeMap(reflect.MapOf(reflect.TypeOf("a"), result.Type()))
layer--
}
keys := v.MapKeys()
for _, k := range keys {
value := reflect.ValueOf(GetMarshableObject(v.MapIndex(k).Interface()))
if value.Type() != result.Type().Elem() {
result = reflect.MakeMap(reflect.MapOf(reflect.TypeOf("a"), value.Type()))
}
result.SetMapIndex(reflect.ValueOf(fmt.Sprintf("%v", k)), reflect.ValueOf(GetMarshableObject(v.MapIndex(k).Interface())))
}
default:
result = v
}
return result.Interface()
}
func main() {
stu := Student{111, 222, "Tom"}
score := make(map[Student]int64)
score[stu] = 100
b, err := json.Marshal(GetMarshableObject(score))
if err != nil {
fmt.Println("error:", err)
}
os.Stdout.Write(b) //{"{111 222 Tom}":100}
scoreBak := make(map[Student]int64)
if err = json.Unmarshal(b, &scoreBak); nil != err {
fmt.Println("error: %v", err) // get error here: cannot unmarshal object into Go value of type map[main.Student]int64
}
}
From the docs:
The map's key type must either be a string, an integer type, or
implement encoding.TextMarshaler.
func (s Student) MarshalText() (text []byte, err error) {
type noMethod Student
return json.Marshal(noMethod(s))
}
func (s *Student) UnmarshalText(text []byte) error {
type noMethod Student
return json.Unmarshal(text, (*noMethod)(s))
}
As an example I'm using encoding/json to turn a Student value into a json object key, however that is not required and you can choose your own format.
https://play.golang.org/p/4BgZn4Y37Ww

Golang RPC encode custom function

I am trying to use github.com/dullgiulio/pingo and send my custom struct
type LuaPlugin struct {
Name string
List []PluginTable
}
type PluginTable struct {
Name string
F lua.LGFunction
}
// LoadPlugins walks over the plugin directory loading all exported plugins
func LoadPlugins() {
//
p := pingo.NewPlugin("tcp", "plugins/test")
// Actually start the plugin
p.Start()
// Remember to stop the plugin when done using it
defer p.Stop()
gob.Register(&LuaPlugin{})
gob.Register(&PluginTable{})
var resp *LuaPlugin
// Call a function from the object we created previously
if err := p.Call("MyPlugin.SayHello", "Go developer", &resp); err != nil {
log.Print(err)
} else {
log.Print(resp.List[0])
}
}
However I am always getting nil for the F field of ym struct. This is what I am sending on the client
// Create an object to be exported
type MyPlugin struct{}
// Exported method, with a RPC signature
func (p *MyPlugin) SayHello(name string, msg *util.LuaPlugin) error {
//
//
*msg = util.LuaPlugin{
Name: "test",
List: []util.PluginTable{
{
Name: "hey",
F: func(L *lua.LState) int {
log.Println(L.ToString(2))
return 0
},
},
},
}
return nil
}
Is it not possible to send custom data types over RPC?
I'm not familiar with the library however, you could try converting the struct to a byte slice before transport. Late reply, might help others....
Simple conversion: returns a struct as bytes
func StructToBytes(s interface{}) (converted []byte, err error) {
var buff bytes.Buffer
encoder := gob.NewEncoder(&buff)
if err = encoder.Encode(s); err != nil {
return
}
converted = buff.Bytes()
return
}
Decoder: returns a wrapper to decode the bytes into
func Decoder(rawBytes []byte) (decoder *gob.Decoder) {
reader := bytes.NewReader(rawBytes)
decoder = gob.NewDecoder(reader)
return
}
Example:
type MyStruct struct {
Name string
}
toEncode := MyStruct{"John Doe"}
// convert the struct to bytes
structBytes, err := StructToBytes(toEncode)
if err != nil {
panic(err)
}
//-----------
// send over RPC and decode on other side
//-----------
// create a new struct to decode into
var DecodeInto MyStruct
// pass the bytes to the decoder
decoder := Decoder(structBytes)
// Decode into the struct
decoder.Decode(&DecodeInto)
fmt.Println(DecodeInto.Name) // John Doe
Due to the use of the gob package you can swap types and typecast to a certain extent.
For more see: https://golang.org/pkg/encoding/gob/

Resources