I'm trying to dynamically create a slice of structs that I can marshal my data into, but doing it at runtime based on some unknown struct type. I'm doing this because I want to have one common bit of code that can unmarshal anything that implements a specific interface.
e.g. (pseudo code)
func unmarshalMyData(myInterface MyInterface, jsonData []byte) {
targetSlice := myInterface.EmptySlice()
json.Unmarshal(jsonData, &targetSlice)
}
I've tried several different alternatives. The one that seems the most promising is using the reflect package to construct the slice. Unfortunately, after unmarshalling, the dynamically created slice has a type of []interface{}. If I print out the type of the dynamically created slice before unmarshalling it prints []*main.myObj. Can anyone explain why? See playground link: https://play.golang.org/p/vvf1leuQeY
Is there some other way to dynamically create a slice of structs that unmarshals correctly?
I'm aware of json.RawMessage, but there are a few reasons I can't use it... there is no "type" field in my json. Also, the code where I am unmarshalling has no compile time knowledge of the struct that it is unmarshalling. It only knows that the struct implements a specific interface.
Some code that baffles me as to why dynamically created slice doesn't maintain its type after unmarshalling:
package main
import (
"encoding/json"
"fmt"
"reflect"
)
func main() {
input := `[{"myField":"one"},{"myField":"two"}]`
myObjSlice := []*myObj{}
fmt.Printf("Type of myObjSlice before unmarshalling %T\n", myObjSlice)
err := json.Unmarshal([]byte(input), &myObjSlice)
if err != nil {
panic(err)
}
fmt.Printf("Type of myObjSlice after unmarshalling %T\n", myObjSlice)
myConstructedObjSlice := reflect.MakeSlice(reflect.SliceOf(reflect.TypeOf(&myObj{})), 0, 0).Interface()
fmt.Printf("Type of myConstructedObjSlice before unmarshalling %T\n", myConstructedObjSlice)
err = json.Unmarshal([]byte(input), &myConstructedObjSlice)
if err != nil {
panic(err)
}
fmt.Printf("Type of myConstructedObjSlice after unmarshalling %T\n", myConstructedObjSlice)
}
type myObj struct {
myField string
}
The output:
Type of myObjSlice before unmarshalling []*main.myObj
Type of myObjSlice after unmarshalling []*main.myObj
Type of myConstructedObjSlice before unmarshalling []*main.myObj
Type of myConstructedObjSlice after unmarshalling []interface {}
You're building a []*myObj using reflect, but you're then passing an *interface{} to json.Unmarshal. The json package sees the target of the pointer as the type interface{}, and therefore uses its default types to unmarshal into. You need to create a pointer to the slice type you create, so the call to the Interface() method returns the exactly type you want to unmarshal into, which is *[]*myObj.
sliceType := reflect.SliceOf(reflect.TypeOf(&myObj{}))
slicePtr := reflect.New(sliceType)
err = json.Unmarshal([]byte(input), slicePtr.Interface())
Related
I am trying to use generics to convert a JSON object into a GRPC response that has an enum, i.e.:
type GRPCResponse {
str string
enu EnumType
}
type EnumType int32
const (
Type1 EnumType = 0
Type2 EnumType = 1
)
The function for unmarshalling looks like this:
func assertHTTPResponseOK[T any](t *testing.T, endpoint string) T {
body, err := GetResponse(endpoint)
var v T
err := json.Unmarshal(body, &v)
require.Nil(t, err)
return v
}
And the code calling it looks like this:
assertHTTPResponseOK[*GRPCResponse](t, "some-endpoint")
The JSON object in question looks like this:
{"str":"hello", "enu": "Type2"}
and I am getting an error along the lines of:
json: cannot unmarshal string into Go struct field GRPCResponse.enu of type EnumType
From similar questions, I have seen that the usual advice is to use jsonpb.Unmarshal or protojson.Unmarshal instead of the typical json.Unmarshal.
In changing the Unmarshal function, I also had to change T to be protoreflect.ProtoMessage. However, this prevents me from passing a pointer to v to Unmarshal, because it is a pointer to the interface, not the interface. Of course, I also cannot pass in a nil pointer (not take the address of v).
So my questions are:
Is there a way to have the pointer of this generic object satisfy the interface protoreflect.ProtoMessage?
Is there a better Unmarshalling function that would better fit my problem?
I ended up passing in the object I am unmarshalling into.
obj := new(GRPCResponse)
assertHTTPResponseOK[*GRPCResponse](t, ctx, "some-endpoint", obj)
func assertHTTPResponseOK[T protoreflect.ProtoMessage](t *testing.T, ctx context.Context, endpoint string, object T) {
body, err := GetResponse(endpoint)
require.Nil(t, err)
err = protojson.Unmarshal(body, object)
require.Nil(t, err)
}
Here's a generics-friendly proto unmarshaller that avoids passing the second type, at the cost of a reflective invoke to peek the type inside the pointer and call it's constructor.
var msg T // Constrained to proto.Message
// Peek the type inside T (as T= *SomeProtoMsgType)
msgType := reflect.TypeOf(msg).Elem()
// Make a new one, and throw it back into T
msg = reflect.New(msgType).Interface().(T)
errUnmarshal := proto.Unmarshal(body, msg)
I'm new to golang generics and have the following setup.
I've gathered loads of different kinds of reports.
Each report has enclosing fields
So I wrapped it in a ReportContainerImpl
I've used a type argument of [T Reportable] where the Reportable is defined as follows
type Reportable interface {
ExportDataPointReport | ImportDataPointReport | MissingDataPointReport | SensorThresoldReport
}
Each of the type in the type constraint is structs that is to be embedded in the container.
type ReportContainerImpl[T Reportable] struct {
LocationID string `json:"lid"`
Provider string `json:"pn"`
ReportType ReportType `json:"m"`
Body T `json:"body"`
}
I use a discriminator ReportType to determine the concrete type when Unmarshal.
type ReportType string
const (
ReportTypeExportDataPointReport ReportType = "ExportDataPointReport"
ReportTypeImportDataPointReport ReportType = "ImportDataPointReport"
ReportTypeMissingDataPointReport ReportType = "MissingDataPointReport"
ReportTypeSensorThresoldReport ReportType = "SensorThresoldReport"
)
Since go does not support type assertion for struct (only interfaces) it is not possible to cast the type when Unmarshal. Also go does not support pointer to the "raw" generic type. Hence, I've created a interface that the ReportContainerImpl implements.
type ReportContainer interface {
GetLocationID() string
GetProvider() string
GetReportType() ReportType
GetBody() interface{}
}
The problem I then get is that I cannot do type constrains on the return type in any form or shape and am back at "freetext semantics" on the GetBody() function to allow for type assertion when Unmarshal is done.
container, err := UnmarshalReportContainer(data)
if rep, ok := container.GetBody().(ExportDataPointReport); ok {
// Use the ReportContainerImpl[ExportDataPointReport] here...
}
Maybe I'm getting this wrong? - but however I do this, I always end up with somewhere needs a interface{} or to know the exact type before Unmarshal
Do you have a better suggestion how to solve this in a type (safer) way?
Cheers,
Mario :)
For completeness I add the UnmarshalReportContainer here
func UnmarshalReportContainer(data []byte) (ReportContainer, error) {
type Temp struct {
LocationID string `json:"lid"`
Provider string `json:"pn"`
ReportType ReportType `json:"m"`
Body *json.RawMessage `json:"body"`
}
var temp Temp
err := json.Unmarshal(data, &temp)
if err != nil {
return nil, err
}
switch temp.ReportType {
case ReportTypeExportDataPointReport:
var report ExportDataPointReport
err := json.Unmarshal(*temp.Body, &report)
return &ReportContainerImpl[ExportDataPointReport]{
LocationID: temp.LocationID,
Provider: temp.Provider,
ReportType: temp.ReportType,
Body: report,
}, err
// ...
}
}
but however I do this, I always end up with somewhere needs a interface{} or to know the exact type before Unmarshal
Precisely.
The concrete types needed to instantiate some generic type or function like ReportContainerImpl or UnmarshalReportContainer must be known at compile time, when you write the code. JSON unmarshalling instead occurs at run-time, when you have the byte slice populated with the actual data.
To unmarshal dynamic JSON based on some discriminatory value, you still need a switch.
Do you have a better suggestion how to solve this in a type (safer) way?
Just forgo parametric polymorphism. It's not a good fit here. Keep the code you have now with json.RawMessage, unmarshal the dynamic data conditionally in the switch and return the concrete structs that implement ReportContainer interface.
As a general solution — if, and only if, you can overcome this chicken-and-egg problem and make type parameters known at compile time, you can write a minimal generic unmarshal function like this:
func unmarshalAny[T any](bytes []byte) (*T, error) {
out := new(T)
if err := json.Unmarshal(bytes, out); err != nil {
return nil, err
}
return out, nil
}
This is only meant to illustrate the principle. Note that json.Unmarshal already accepts any type, so if your generic function actually does nothing except new(T) and return, like in my example, it is no different than "inlining" the entire thing as if unmarshalAny didn't exist.
v, err := unmarshalAny[SomeType](src)
functionally equivalent as
out := &SomeType{}
err := json.Unmarshal(bytes, out)
If you plan to put more logic in unmarshalAny, its usage may be warranted. Your mileage may vary; in general, don't use type parameters when it's not actually necessary.
I have a small Go program like this:
package main
import "encoding/json"
func main() {
bs := []byte(`{"items": ["foo", "bar"]}`)
data := Data{}
err := json.Unmarshal(bs, &data)
if err != nil {
panic(err)
}
}
type Data struct {
Items []Raw `json:"items"`
}
// If I do this, this program panics with "illegal base64 data at input byte 0"
type Raw json.RawMessage
// This works
type Raw = json.RawMessage
Why does json.Unmarshal work with type alias but not type definition? What does that error message mean?
This is a new type definition:
type Raw json.RawMessage
The Raw type is derived from json.RawMessage, but it is a new type without any of the methods defined for json.RawMessage. So, if json.RawMessage has a json unmarshaler method, Raw does not have that. Code such as following will not recognize a variable t of type Raw as a json.RawMessage:
if t, ok:=item.(json.RawMessage); ok {
...
}
This is a type alias:
type Raw = json.RawMessage
Here, Raw has all the methods json.RawMessage has, and type assertion to json.RawMessage will work, because Raw is simply another name for json.RawMessage.
When Unmarshalling data from an endpoint, I almost always have a struct with the data that I am want to get from my request that I unmarshal into. Well there are times where I want to see everything that is returning back to me but I am not sure what some of that data is so I don't know how to define it in my struct. Is there a way to have a struct that can just grab all data being unmarshalled without me having to specify it?
This is what I thought would work but that is not the case
resp, err := httpClient.Get("/api/stuff")
data, _ := ioutil.ReadAll(resp.Body)
var myStruct struct{}
json.Unmarshal(data, myStruct)
If you do not know the composition of a JSON object in advance, you can unmarshal into a map[string]interface{}.
var myMap map[string]interface{}
json.Unmarshal(data, &myMap)
See an example here.
If you don't know how to define your struct, then maybe you should use a map. It's a pretty good fit for unmarshalling JSON. Here is an example of how to do it without knowing exactly what data you are receiving:
func parseJSON(r *http.Request) (map[string]interface{}, error) {
var postData interface{}
decoder := json.NewDecoder(r.Body)
err := decoder.Decode(&postData)
if err != nil {
return nil, err
}
return postData.(map[string]interface{}), nil
}
Now you at least have a string name of each piece of data, which should give your application some idea of how to handle it.
I need to specify a type for decoding JSON data in a flexible manner, meaning the type needs to be specified at runtime.
Consider this snippet: http://play.golang.org/p/F-Jy4ufMPz
s := `{"b":[{"x":9},{"x":4}]}`
var a struct {
B []interface{}
}
err := json.Unmarshal([]byte(s), &a)
if err != nil {
panic(err)
}
fmt.Println(a)
Which will produce {[map[x:9] map[x:4]]}. I want to decode to an array of a specific (struct) type instead of []interface{}, without specifying it at compile time.
Is that somehow possible without creating the array up front? (the number of returned items is unknown)
The only way I can think of right now is to encode the returned maps again later, and decode them to the specified type, which would create unnecessary processing overhead.
If not specifying it at compile time, you still need to specify it somewhere.
If specified before the retrieval of the Json data, you can simply do a switch case, Unmarshalling it to your desired object.
If specified within the Json data, you can marshal the "flexible" part into a json.RawMessage to process it after you've decided what type of struct is suitable:
package main
import (
"encoding/json"
"fmt"
)
var s = `{"type":"structx", "data":{"x":9,"xstring":"This is structX"}}`
type JsonStruct struct {
Type string
Data json.RawMessage
}
type StructX struct {
X float64
Xstring string
}
type StructY struct {
Y bool
}
func main() {
var a *JsonStruct
err := json.Unmarshal([]byte(s), &a)
if err != nil {
panic(err)
}
switch a.Type {
case "structx":
// We Unmashal the RawMessage part into a StructX
var s *StructX
json.Unmarshal([]byte(a.Data), &s)
if err != nil {
panic(err)
}
fmt.Println(s)
case "structy":
// Do the same but for structY
}
}
Playground