Deserializing a BSON into a structure created by protobuf with a oneof property fails:
panic: no decoder found for test.isTest_Entry
This is probably because the oneof field is registered as a plain interface (from the generated code):
type Test struct {
// Types that are valid to be assigned to Entry:
// *Test_S1
// *Test_S2
Entry isTest_Entry `protobuf_oneof:"entry"`
XXX_NoUnkeyedLiteral struct{} `json:"-" bson:"-"`
XXX_unrecognized []byte `json:"-" bson:"-"`
XXX_sizecache int32 `json:"-" bson:"-"`
}
//...
type isTest_Entry interface {
isTest_Entry()
}
package main
import (
"fmt"
"go.mongodb.org/mongo-driver/bson"
"test"
)
func main() {
entry := test.Test{}
data, err := bson.Marshal(&entry)
if nil != err {
panic(err)
}
fmt.Println(data)
newMessage := &test.Test{}
err = bson.Unmarshal(data, newMessage)
if err != nil {
panic(err)
}
fmt.Println(newMessage.GetEntry())
}
Is there any tags I can add to make this work or is this a bug?
Related
I'm trying to unmarshal the following YAML data into Go structures.
The data is the in the following format:
fetchers:
- type: "aws"
config:
omega: "lul"
- type: "kubernetes"
config:
foo: "bar"
Based of the type field, I want to determine wether to unmarshal the config field into awsConfig or kubernetesConfig struct.
My current code looks like this (using "gopkg.in/yaml.v2"):
type kubernetesConfig struct {
foo string `yaml:"foo"`
}
type awsConfig struct {
omega string `yaml:"omega"`
}
var c struct {
Fetchers []struct {
Type string `yaml:"type"`
Config interface{} `yaml:"config"`
} `yaml:"fetchers"`
}
err := yaml.Unmarshal(data, &c)
if err != nil {
log.Fatal(err)
}
for _, val := range c.Fetchers {
switch val.Type {
case "kubernetes":
conf := val.Config.(kubernetesConfig)
fmt.Println(conf.foo)
case "aws":
conf := val.Config.(awsConfig)
fmt.Println(conf.omega)
default:
log.Fatalf("No matching type, was type %v", val.Type)
}
}
Code in playground: https://go.dev/play/p/klxOoHMCtnG
Currently it gets unmarshalled as map[interface {}]interface {}, which can't be converted to one of the structs above.
Error:
panic: interface conversion: interface {} is map[interface {}]interface {}, not main.awsConfig \
Do I have to implemented the Unmarshaler Interface of the YAML package with a custom UnmarshalYAML function to get this done?
Found the solution by implementing Unmarshaler Interface:
type Fetcher struct {
Type string `yaml:"type"`
Config interface{} `yaml:"config"`
}
// Interface compliance
var _ yaml.Unmarshaler = &Fetcher{}
func (f *Fetcher) UnmarshalYAML(unmarshal func(interface{}) error) error {
var t struct {
Type string `yaml:"type"`
}
err := unmarshal(&t)
if err != nil {
return err
}
f.Type = t.Type
switch t.Type {
case "kubernetes":
var c struct {
Config kubernetesConfig `yaml:"config"`
}
err := unmarshal(&c)
if err != nil {
return err
}
f.Config = c.Config
case "aws":
var c struct {
Config awsConfig `yaml:"config"`
}
err := unmarshal(&c)
if err != nil {
return err
}
f.Config = c.Config
}
return nil
}
This type of task - where you want to delay the unmarshaling - is very similar to how json.RawMessage works with examples like this.
The yaml package does not have a similar mechanism for RawMessage - but this technique can easily be replicated as outlined here:
type RawMessage struct {
unmarshal func(interface{}) error
}
func (msg *RawMessage) UnmarshalYAML(unmarshal func(interface{}) error) error {
msg.unmarshal = unmarshal
return nil
}
// call this method later - when we know what concrete type to use
func (msg *RawMessage) Unmarshal(v interace{}) error {
return msg.unmarshal(v)
}
So to leverage this in your case:
var fs struct {
Configs []struct {
Type string `yaml:"type"`
Config RawMessage `yaml:"config"` // delay unmarshaling
} `yaml:"fetchers"`
}
err = yaml.Unmarshal([]byte(data), &fs)
if err != nil {
return
}
and based on the config "Type" (aws or kubernetes), you can finally unmarshal the RawMessage into the correct concrete type:
aws := awsConfig{} // concrete type
err = c.Config.Unmarshal(&aws)
or:
k8s := kubernetesConfig{} // concrete type
err = c.Config.Unmarshal(&k8s)
Working example here: https://go.dev/play/p/wsykOXNWk3H
I am sending the data to the API like following:
{"after": {"amount": 811,"id":123,"status":"Hi"}, "key": [70]}
and i am getting following while printing :
map[after:map[amount:811 id:123 status:Hi ] key:[70]]
Is there any way to print individual field like this??
amount::800
id:123
status:Hi
The code:
package main
import (
"encoding/json"
"fmt"
"io"
"io/ioutil"
"log"
"net/http"
"strings"
)
var (
PORT = ":8080"
)
func main() {
fmt.Println("In Main")
http.HandleFunc("/", changedData)
http.ListenAndServe(PORT, nil)
}
type Data struct {
Id int64 `json:"id"`
Amount float64 `json:"amount"`
Status string `json:"status"`
}
type mark map[string]interface{}
func changedData(w http.ResponseWriter, r *http.Request) {
fmt.Println("Coming From API")
reqBody, _ := ioutil.ReadAll(r.Body)
fmt.Println("Data coming from API ", string(reqBody))
digit := json.NewDecoder(strings.NewReader(string(reqBody)))
for digit.More() {
var result mark
err := digit.Decode(&result)
if err != nil {
if err != io.EOF {
log.Fatal(err)
}
break
}
fmt.Println("final_data ", result)
}
}
Decode to a Go type that matches the structure of the JSON document. You declared a type for the "after" field. Wrap that type with a struct to match the document.
func changedData(w http.ResponseWriter, r *http.Request) {
var v struct{ After Data }
err := json.NewDecoder(r.Body).Decode(&v)
if err != nil {
http.Error(w, "bad request", 400)
return
}
fmt.Printf("final_data: %#v", v.After)
}
Playground example.
I think you can define a struct type if you know the JSON file format or if the JSON format is predefined. As far as I know that mostly using interface{} is a way when you don't know the JSON format or there is no predefined format of the JSON. If you define a struct type and use it while unmarshaling the JSON to struct, you can access the variables by typing like data.Id or data.Status.
Here's an example code:
package main
import (
"encoding/json"
"fmt"
)
type Data struct {
AfterData After `json:"after"`
Key []int `json:"key"`
}
type After struct {
Id int64 `json:"id"`
Amount float64 `json:"amount"`
Status string `json:"status"`
}
func main() {
j := []byte(`{"after": {"amount": 811,"id":123,"status":"Hi"}, "key": [70]}`)
var data *Data
err := json.Unmarshal(j, &data)
if err != nil {
fmt.Println(err.Error())
return
}
fmt.Println(data.AfterData)
fmt.Println(data.AfterData.Id)
fmt.Println(data.AfterData.Amount)
fmt.Println(data.AfterData.Status)
}
Output will be
{123 811 Hi}
123
811
Hi
Go Playground
I am using the REST proxy instance of Kafka for producing and consuming messages.Using the API to get new messages but I am not able to convert those messages to a struct model in Go. For example:
// Get records
req, err := http.NewRequest(http.MethodGet, fmt.Sprintf(FETCH_CONSUMER, URL, GROUP, CONSUMER), nil)
if err != nil {
panic(err)
}
req.Header.Add("Accept", CONTENT_TYPE)
respRecords, err := client.Do(req)
if err != nil {
panic(err)
}
defer respRecords.Body.Close()
fmt.Printf("Response %s\n", respRecords.Status)
fmt.Println(respRecords.Body)
recordsBodyResp := bufio.NewScanner(respRecords.Body)
for recordsBodyResp.Scan() {
fmt.Printf("<--Body %s\n", recordsBodyResp.Text())
}
The value returned is in the following format:
[{"topic":"backward","key":null,"value":{"AdoptionID":"abcd123","IPAddress":"8.8.8.8","Port":"80","Status":"requested"},"partition":0,"offset":7}]
Since it's an array of objects, I want to extract the value portion of the key "value" into a struct.
That is where I am stuck.
You can create a struct like:
type AutoGenerated []struct {
Topic string `json:"topic"`
Key interface{} `json:"key"`
Value Value `json:"value"`
Partition int `json:"partition"`
Offset int `json:"offset"`
}
type Value struct {
AdoptionID string `json:"AdoptionID"`
IPAddress string `json:"IPAddress"`
Port string `json:"Port"`
Status string `json:"Status"`
}
And Unmarshal in that struct.
See this sample code:
package main
import (
"fmt"
"encoding/json"
)
func main() {
type Value struct {
AdoptionID string `json:"AdoptionID"`
IPAddress string `json:"IPAddress"`
Port string `json:"Port"`
Status string `json:"Status"`
}
type AutoGenerated []struct {
Topic string `json:"topic"`
Key interface{} `json:"key"`
Value Value `json:"value"`
Partition int `json:"partition"`
Offset int `json:"offset"`
}
byt := []byte(`[{"topic":"backward","key":null,"value":{"AdoptionID":"abcd123","IPAddress":"8.8.8.8","Port":"80","Status":"requested"},"partition":0,"offset":7}]`)
var dat AutoGenerated
if err := json.Unmarshal(byt, &dat); err != nil {
panic(err)
}
fmt.Printf("%#v", dat)
}
In the below program I'm extracting some data from an API.
It outputs a rather complex data.
When I ioutil.ReadAll(resp.Body), the result is of type []uint8.
If I try to read the results, its just a random array of integers.
However, I'm able to read it if I convert it to string using string(diskinfo)
But I want to use this in a Struct and having trouble unmarshalling.
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"net/http"
"net/url"
"reflect"
)
type ApiResults struct {
results []struct {
statement_id int `json.statement_id`
series []struct {
name string `json.name`
tags struct {
host string `json.host`
}
columns []string `json.columns`
values []interface{} `json.values`
}
}
}
func main() {
my_url := "my_url"
my_qry := fmt.Sprintf("my_query")
resp, err := http.Get(my_url + url.QueryEscape(my_qry))
if err != nil {
fmt.Printf("ERROR: %v\n", err)
} else {
fmt.Println(reflect.TypeOf(resp))
diskinfo, _ := ioutil.ReadAll(resp.Body)
fmt.Println(reflect.TypeOf((diskinfo)))
fmt.Println(diskinfo)
fmt.Println(string(diskinfo))
diskinfo_string := string(diskinfo)
data := ApiResults{}
json.Unmarshal([]byte(diskinfo_string), &data)
//fmt.Printf("Values = %v\n", data.results.series.values)
//fmt.Printf("Server = %v\n", data.results.series.tags.host)
}
}
If I view the data as a string, I get this (formatted):
{"results":[
{"statement_id":0,
"series":[
{"name":"disk",
"tags":{"host":"myServer1"},
"columns":["time","disk_size"],
"values":[["2021-07-07T07:53:32.291490387Z",1044]]},
{"name":"disk",
"tags":{"host":"myServer2"},
"columns":["time","disk_size"],
"values":[["2021-07-07T07:53:32.291490387Z",1046]]}
]}
]}
I think my Apireturn struct is also structured incorrectly because the API results have info for multiple hosts.
But first, I doubt if the data has to be sent in a different format to the struct. Once I do this, I can probably try to figure out how to read from the Struct next.
The ioutil.ReadAll already provides you the data in the type byte[]. Therefore you can just call json.Unmarshal passing it as a parameter.
import (
"encoding/json"
"io/ioutil"
"net/http"
)
func toStruct(res *http.Response) (*ApiResults, error) {
body, err := ioutil.ReadAll(res.Body)
if err != nil {
return nil, err
}
defer res.Body.Close()
data := ApiResults{}
if err := json.Unmarshal(body, &data); err != nil {
return nil, err
}
return data, nil
}
There also seems to be an issue with your struct. The correct way to use struct tags is as follows. Plus, fields need to be exported for the json tag (used by json.Umarshal) to work – starting with uppercase will do it.
type ApiResults struct {
Results []struct {
StatementId int `json:"statement_id"`
Series []struct {
Name string `json:"name"`
Tags struct {
Host string `json:"host"`
} `json:"tags"`
Columns []string `json:"columns"`
Values []interface{} `json:"values"`
} `json:"series"`
} `json:"results"`
}
I get the following data:
{
"timestamp": "1526058949",
"bids": [
[
"7215.90",
"2.31930000"
],
[
"7215.77",
"1.00000000"
]
]
}
via websocket and I would like to unmarshall it into
type OrderBookItem struct {
Price string
Amount string
}
type OrderBookResult struct {
Timestamp string `json:"timestamp"`
Bids []OrderBookItem `json:"bids"`
Asks []OrderBookItem `json:"asks"`
}
Unmarshal it with:
s := e.Data.(string)
d := &OrderBookResult{}
err := json.Unmarshal([]byte(s), d)
if err == nil {
....
} else {
fmt.Println(err.Error())
}
But I keep getting the error:
json: cannot unmarshal string into Go struct field
OrderBookResult.bids of type feed.OrderBookItem
When I change the struct into
type OrderBookResult struct {
Timestamp string `json:"timestamp"`
Bids [][]string `json:"bids"`
Asks [][]string `json:"asks"`
}
it works. I would like them to be defined as float64 which is what they are. What do I have to change?
As the error says:
json: cannot unmarshal string into Go struct field
OrderBookResult.bids of type feed.OrderBookItem
We cannot convert OrderBookResult.bids which is a slice of string into OrderBookItem which is struct
Implement UnmarshalJSON interface to convert array into objects for price and amount of OrderBookItem struct. Like below
package main
import (
"fmt"
"encoding/json"
)
type OrderBookItem struct {
Price string
Amount string
}
func(item *OrderBookItem) UnmarshalJSON(data []byte)error{
var v []string
if err:= json.Unmarshal(data, &v);err!=nil{
fmt.Println(err)
return err
}
item.Price = v[0]
item.Amount = v[1]
return nil
}
type OrderBookResult struct {
Timestamp string `json:"timestamp"`
Bids []OrderBookItem `json:"bids"`
Asks []OrderBookItem `json:"asks"`
}
func main() {
var result OrderBookResult
jsonString := []byte(`{"timestamp": "1526058949", "bids": [["7215.90", "2.31930000"], ["7215.77", "1.00000000"]]}`)
if err := json.Unmarshal([]byte(jsonString), &result); err != nil{
fmt.Println(err)
}
fmt.Printf("%+v", result)
}
Playground working example
For more information read GoLang spec for Unmarshaler
You are treating your bids as a structure of two separate strings, when they are really a slice of strings in the JSON. If you change OrderBookItem to be
type OrderBookItem []string
which is how you have defined them in the second bit, which works.
To access the values you just have to do:
price := d.Bids[0]
amount := d.Bids[1]