How to use generics in Unmarshal (go 1.18) - go

I'm new to golang generics and have the following setup.
I've gathered loads of different kinds of reports.
Each report has enclosing fields
So I wrapped it in a ReportContainerImpl
I've used a type argument of [T Reportable] where the Reportable is defined as follows
type Reportable interface {
ExportDataPointReport | ImportDataPointReport | MissingDataPointReport | SensorThresoldReport
}
Each of the type in the type constraint is structs that is to be embedded in the container.
type ReportContainerImpl[T Reportable] struct {
LocationID string `json:"lid"`
Provider string `json:"pn"`
ReportType ReportType `json:"m"`
Body T `json:"body"`
}
I use a discriminator ReportType to determine the concrete type when Unmarshal.
type ReportType string
const (
ReportTypeExportDataPointReport ReportType = "ExportDataPointReport"
ReportTypeImportDataPointReport ReportType = "ImportDataPointReport"
ReportTypeMissingDataPointReport ReportType = "MissingDataPointReport"
ReportTypeSensorThresoldReport ReportType = "SensorThresoldReport"
)
Since go does not support type assertion for struct (only interfaces) it is not possible to cast the type when Unmarshal. Also go does not support pointer to the "raw" generic type. Hence, I've created a interface that the ReportContainerImpl implements.
type ReportContainer interface {
GetLocationID() string
GetProvider() string
GetReportType() ReportType
GetBody() interface{}
}
The problem I then get is that I cannot do type constrains on the return type in any form or shape and am back at "freetext semantics" on the GetBody() function to allow for type assertion when Unmarshal is done.
container, err := UnmarshalReportContainer(data)
if rep, ok := container.GetBody().(ExportDataPointReport); ok {
// Use the ReportContainerImpl[ExportDataPointReport] here...
}
Maybe I'm getting this wrong? - but however I do this, I always end up with somewhere needs a interface{} or to know the exact type before Unmarshal
Do you have a better suggestion how to solve this in a type (safer) way?
Cheers,
Mario :)
For completeness I add the UnmarshalReportContainer here
func UnmarshalReportContainer(data []byte) (ReportContainer, error) {
type Temp struct {
LocationID string `json:"lid"`
Provider string `json:"pn"`
ReportType ReportType `json:"m"`
Body *json.RawMessage `json:"body"`
}
var temp Temp
err := json.Unmarshal(data, &temp)
if err != nil {
return nil, err
}
switch temp.ReportType {
case ReportTypeExportDataPointReport:
var report ExportDataPointReport
err := json.Unmarshal(*temp.Body, &report)
return &ReportContainerImpl[ExportDataPointReport]{
LocationID: temp.LocationID,
Provider: temp.Provider,
ReportType: temp.ReportType,
Body: report,
}, err
// ...
}
}

but however I do this, I always end up with somewhere needs a interface{} or to know the exact type before Unmarshal
Precisely.
The concrete types needed to instantiate some generic type or function like ReportContainerImpl or UnmarshalReportContainer must be known at compile time, when you write the code. JSON unmarshalling instead occurs at run-time, when you have the byte slice populated with the actual data.
To unmarshal dynamic JSON based on some discriminatory value, you still need a switch.
Do you have a better suggestion how to solve this in a type (safer) way?
Just forgo parametric polymorphism. It's not a good fit here. Keep the code you have now with json.RawMessage, unmarshal the dynamic data conditionally in the switch and return the concrete structs that implement ReportContainer interface.
As a general solution — if, and only if, you can overcome this chicken-and-egg problem and make type parameters known at compile time, you can write a minimal generic unmarshal function like this:
func unmarshalAny[T any](bytes []byte) (*T, error) {
out := new(T)
if err := json.Unmarshal(bytes, out); err != nil {
return nil, err
}
return out, nil
}
This is only meant to illustrate the principle. Note that json.Unmarshal already accepts any type, so if your generic function actually does nothing except new(T) and return, like in my example, it is no different than "inlining" the entire thing as if unmarshalAny didn't exist.
v, err := unmarshalAny[SomeType](src)
functionally equivalent as
out := &SomeType{}
err := json.Unmarshal(bytes, out)
If you plan to put more logic in unmarshalAny, its usage may be warranted. Your mileage may vary; in general, don't use type parameters when it's not actually necessary.

Related

Returning a pointer to a struct from a function in Go

I have two public structs that contain different data, and a private intermediate struct containing either of the two public structs. I also have a function that unmarshalls the intermediate struct, determines which public struct it contains, and returns one of the two public structs.
The problem I'm facing is the return value of the last function. At it's simplest I thought I could return *struct{} but I keep getting a type mismatch in my IDE.
I apologize for posting more code than is probably necessary, but I'm trying to make it as close as possible to the code I'm working on.
package main
import (
"encoding/json"
"errors"
)
// These vars are some errors I'll use in the functions later on
var (
errInvalidBase64 = errors.New("invalid base64")
errInvalidStructType = errors.New("invalid struct type")
)
// Struct1 public struct
type Struct1 struct {
FName string `json:"first-name"`
LName string `json:"last-name"`
}
// Struct2 public struct
type Struct2 struct {
Date string `json:"date"`
Items []int `json:"items"`
}
// intermediateStruct private struct
// The Type field indicates which kind of struct Data contains (Struct1 or Struct2)
// The Data field contains either Struct1 or Struct2 which was previously marshalled into JSON
type intermediateStruct struct {
Type structType
Data []byte
}
// The following type and const are my understanding of an enum in Go
// structType is a private type for the type of struct intermediateStruct contains
type structType int
// These public constants are just to keep my hands out of providing values for the different struct types
const (
StructType1 structType = iota
StructType2
)
// unmarshalStruct1 unmarshalls JSON []byte into a new Struct1 and returns a pointer to that struct
func unmarshalStruct1(b []bytes) (*Struct1, error) {
newStruct1 := new(Struct1)
err := json.Unmarshal(b, newStruct1)
if err != nil {
return nil, errInvalidBase64
}
return newStruct1, nil
}
// unmarshalStruct2 unmarshalls JSON []byte into a new Struct2 and returns a pointer to that struct
func unmarshalStruct2(b []bytes) (*Struct2, error) {
newStruct2 := new(Struct2)
err := json.Unmarshal(b, newStruct2)
if err != nil {
return nil, errInvalidBase64
}
return newStruct2, nil
}
// receiveStruct accepts *intermediateStruct who's Data field contains either Struct1 or Struct2
// This function needs to return either *Struct1 or *Struct2 and an error
func receiveStruct(iStruct *intermediateStruct) (*struct{}, error) {
switch iStruct.Type {
case StructType1:
struct1, err := unmarshalStruct1(iStruct.Data)
if err != nil {
return nil, err
}
// The following line is where I'm getting the type mismatch
return struct1, nil
case StructType2:
struct2, err := unmarshalStruct2(iStruct.Data)
if err != nil {
return nil, err
}
// The following line is another type mismatch
return struct2, nil
default:
return nil, errInvalidStructType
}
}
I know there's a way to do what I'm trying to achieve - I just lack the experience/understanding to get there.
Thanks for any and all input!
Your unmarshallStruct function returns a pointer to type Struct1 or Struct2 (depending on which version of the function gets called). And therefore the variables struct1 and struct2 are pointers to types Struct1 and Struct2 respectively. Neither is a pointer to type struct (which is not a real Go type anyways I must add). A struct is a keyword which helps to declare types containing fields/attributes.
Depending on the use-cases you have in mind for the rest of your code, can instead try any of the below:
As mkopriva suggested, return a interface{} object, but you'd need to use type assertion to actually make sure of the object
Define an interface which both Struct1 and Struct2 implement, and return a pointer to this
Make separate functions which work with either Struct1 or Struct2. This is not necessarily as bad as it sounds as Go lets you pass functions in the same way you'd pass types (see example of the Less() function in sort package).

Issue with Unmarshalling GRPC Response To Generic Type

I am trying to use generics to convert a JSON object into a GRPC response that has an enum, i.e.:
type GRPCResponse {
str string
enu EnumType
}
type EnumType int32
const (
Type1 EnumType = 0
Type2 EnumType = 1
)
The function for unmarshalling looks like this:
func assertHTTPResponseOK[T any](t *testing.T, endpoint string) T {
body, err := GetResponse(endpoint)
var v T
err := json.Unmarshal(body, &v)
require.Nil(t, err)
return v
}
And the code calling it looks like this:
assertHTTPResponseOK[*GRPCResponse](t, "some-endpoint")
The JSON object in question looks like this:
{"str":"hello", "enu": "Type2"}
and I am getting an error along the lines of:
json: cannot unmarshal string into Go struct field GRPCResponse.enu of type EnumType
From similar questions, I have seen that the usual advice is to use jsonpb.Unmarshal or protojson.Unmarshal instead of the typical json.Unmarshal.
In changing the Unmarshal function, I also had to change T to be protoreflect.ProtoMessage. However, this prevents me from passing a pointer to v to Unmarshal, because it is a pointer to the interface, not the interface. Of course, I also cannot pass in a nil pointer (not take the address of v).
So my questions are:
Is there a way to have the pointer of this generic object satisfy the interface protoreflect.ProtoMessage?
Is there a better Unmarshalling function that would better fit my problem?
I ended up passing in the object I am unmarshalling into.
obj := new(GRPCResponse)
assertHTTPResponseOK[*GRPCResponse](t, ctx, "some-endpoint", obj)
func assertHTTPResponseOK[T protoreflect.ProtoMessage](t *testing.T, ctx context.Context, endpoint string, object T) {
body, err := GetResponse(endpoint)
require.Nil(t, err)
err = protojson.Unmarshal(body, object)
require.Nil(t, err)
}
Here's a generics-friendly proto unmarshaller that avoids passing the second type, at the cost of a reflective invoke to peek the type inside the pointer and call it's constructor.
var msg T // Constrained to proto.Message
// Peek the type inside T (as T= *SomeProtoMsgType)
msgType := reflect.TypeOf(msg).Elem()
// Make a new one, and throw it back into T
msg = reflect.New(msgType).Interface().(T)
errUnmarshal := proto.Unmarshal(body, msg)

Cast interface{} to struct

I have the following struct in one module:
type Event struct {
Name string `json:"name"`
Version string `json:"version"`
Payload interface{} `json:"payload"`
}
the goal is, that I can accept arbitrary structs as Payload and send the final structs of type Event as json-serialized strings using json.Marshal with some messaging-service.
However, when I try to json.Unmarshal in another project, I can't access the fields inside of Event.Payload, because Go obviously doesn't know about its final type.
I tried to use something like this:
type EventPayload struct{
Name string `json:"name"`
}
// ...
event := &events.Event{}
event.Payload = &EventPayload{}
if err := json.Unmarshal(msg.Data, event); err != nil {
return err
}
event.Payload.Name // Won't work: "type interface{} has no field or method Status"
however, Go still thinks, that event.Payload is an interface{}.
How can I tell Go the correct struct-type in this case?
You can do exactly what you're describing, you just need to use a type assertion to get the underlying type back. json.Unmarshal, provided an interface{} field with a concrete type prepopulated, will use that concrete type.
event := &Event{}
event.Payload = &EventPayload{}
msg := []byte(`{"Name": "foo", "Version": "1", "Payload": {"Name": "bar"}}`)
if err := json.Unmarshal(msg, event); err != nil {
panic(err)
}
pl := event.Payload.(*EventPayload)
fmt.Println(pl.Name)
Working example (slightly modified to run in playground): https://play.golang.org/p/IDXLKeMGw8_1
How can I tell Go the correct struct-type in this case?
You cannot. This is not how encoding/json.Unmarshal works. You either have to unmarshal into a proper type or something else or use json.Raw message and do a second unmarshaling or whatnot but they way you approached it simply doesn't work.

How do I use bson.SetBSON or bson.Raw?

Possibly related: How to use interface type as a model in mgo (Go)?
I have a struct like so:
type Game struct {
ID bson.ObjectId
Type string
Location string
Details interface{}
}
type FeudDetails struct {
...
}
type TriviaDetails struct {
...
}
type BingoDetails struct {
...
}
I want to use the Type field of Game to unserialize Details into a specific type (like an instance of FeudDetails or BingoDetails). It would still be an interface{} in Game, but then I could do this:
feudDetails, ok := game.Details.(FeudDetails)
if ok {
// we know this is a Feud game, and we have the details
feudDetails.Round++
}
The docs say that "it is possible to unmarshal or marshal values partially" using bson.Raw, but they don't provide any examples I've been able to find.
I've tried using:
func (game *Game) SetBSON(r bson.Raw) error {
err := r.Unserialize(game)
if err != nil {
return nil
}
games[game.Type].LoadDetails(game) // this calls a function based on the Type to
// create a concrete value for that game.
return nil
}
I get a (ahem) stack overflow here. I assume this is because r.Unserialize is recursively calling SetBSON.
My goal is to use standard unserializing on all fields except Details, and then be able to use game.Type to determine how to process Details. If I do something like this:
type GameDetails interface{}
type Game struct {
...
Details GameDetails
}
func (details *GameDetails) SetBSON(r bson.Raw) error {
// game isn't defined
games[game.Type].LoadDetails(r, details)
}
then how can I access the outer Type field to know which game type to unserialize it to?
I'll also accept the answer "you're doing it all wrong, and a better pattern in Go is XYZ..."
EDIT: I also tried deserializing normally, and then casting the interface{} version of Details using game.Details.(FeudDetails), but the conversion failed. I guess I can't do it that way because the underlying type after unserialization is not a FeudDetails but rather probably map[string]interface{}.
EDIT 2: I thought I'd be clever and pre-populate an object with the right types when retrieving from the database (game := Game{Details: FeudDetails: {}} prior to calling db...One(&game)) but my trickery did not work:
DEBU[Mar 31 22:19:09.442] Caching show gid=5e814448ef5b9858b7ff4e57
TRAC[Mar 31 22:19:09.442] Before database call dtype=main.FeudDetails
TRAC[Mar 31 22:19:09.446] After database call dtype=bson.M
Ignore Details during (un)marshalling
Change the definition of Game so bson doesn't try to do anything with the Details field:
type Game struct {
...
Details interface{} `json:"details" bson:"-"`
}
Unmarshal Details manually
func (game *Game) SetBSON(r bson.Raw) error {
// Unmarshall everything except Details
type tempGame Game
err := r.Unmarshal((*tempGame)(game)) // this is necessary to prevent recursion
if err != nil {
return err
}
// Get the raw data for Details
var d struct {
Details bson.Raw
}
if err := r.Unmarshal(&d); err != nil {
return err
}
gameType, ok := games[game.Type]
if ok {
// Use individual processing based on game Type
game.Details, err = gameType.LoadDetails(d.Details)
if err != nil {
return err
}
// fmt.Sprintf("%T", game.Details) => main.FeudDetails
}
return nil
}
Marshal Details manually
As far as I can tell, the only way to get bson to include Details after we told it to omit it in the original struct, is to create a whole other structure, copy the data over one by one, and use that type in GetBSON. It seems like there must be a better way.

Am I using renamed types correctly in Golang?

I have to deal with huge integers in Golang that come in from a Swagger-defined REST API. Since Swagger needs a Validate(strfmt.Registry), I define my custom type like this:
// BigInt is a big.Int, but includes a Validate() method for swagger
// Once created, it can be used just like a big.Int.
type BigInt struct {
*big.Int
}
Since it needs to be transformed to and from JSON, I define some JSON Marshaling interface:
// UnmarshalJSON implements encoding/json/RawMessage.UnmarshalJSON
func (b *BigInt) UnmarshalJSON(data []byte) error {
err := json.Unmarshal(data, &b.Int)
if err != nil {
return err
}
return nil
}
// MarshalJSON calls json.Marshal() on the BigInt.Int field.
func (b *BigInt) MarshalJSON() ([]byte, error) {
if b == nil {
return []byte("null"), nil
}
return json.Marshal(b.Int)
}
Now I realized that my custom type doesn't actually behave exactly like big.Int. In order to compare two BigInts:
example := BigInt{Int: &big.Int{}}
other := BigInt{Int: &big.Int{}}
example.Cmp(other.Int)
I cannot do
example.Cmp(other)
which is much cleaner. And creating the BigInt is also a terrible experience, which I have to wrap in a function like this:
// NewBigInt creates a BigInt with its Int struct field
func NewBigInt() (i *BigInt) {
return &BigInt{Int: &big.Int{}}
}
Is this really how I'm supposed to do things?
Why can't golang treat big.Int just like its other built in types like int64/uint64/float64?
Is this really how I'm supposed to do things?
It's a way to do it, but it's not what I'd call a "renamed" type; it's a struct containing a single field. You could also do (like, for example, time.Duration):
type BigInt *big.Int
And apply methods to it. This would allow you to seamlessly convert between *big.Int and your type.
Why can't golang treat big.Int just like its other built in types like int64/uint64/float64?
Because unlike those types, big.Int isn't a built-in type; you can tell because it's big.Int, i.e., it's defined in a package, not in the language.

Resources