I have the following struct in one module:
type Event struct {
Name string `json:"name"`
Version string `json:"version"`
Payload interface{} `json:"payload"`
}
the goal is, that I can accept arbitrary structs as Payload and send the final structs of type Event as json-serialized strings using json.Marshal with some messaging-service.
However, when I try to json.Unmarshal in another project, I can't access the fields inside of Event.Payload, because Go obviously doesn't know about its final type.
I tried to use something like this:
type EventPayload struct{
Name string `json:"name"`
}
// ...
event := &events.Event{}
event.Payload = &EventPayload{}
if err := json.Unmarshal(msg.Data, event); err != nil {
return err
}
event.Payload.Name // Won't work: "type interface{} has no field or method Status"
however, Go still thinks, that event.Payload is an interface{}.
How can I tell Go the correct struct-type in this case?
You can do exactly what you're describing, you just need to use a type assertion to get the underlying type back. json.Unmarshal, provided an interface{} field with a concrete type prepopulated, will use that concrete type.
event := &Event{}
event.Payload = &EventPayload{}
msg := []byte(`{"Name": "foo", "Version": "1", "Payload": {"Name": "bar"}}`)
if err := json.Unmarshal(msg, event); err != nil {
panic(err)
}
pl := event.Payload.(*EventPayload)
fmt.Println(pl.Name)
Working example (slightly modified to run in playground): https://play.golang.org/p/IDXLKeMGw8_1
How can I tell Go the correct struct-type in this case?
You cannot. This is not how encoding/json.Unmarshal works. You either have to unmarshal into a proper type or something else or use json.Raw message and do a second unmarshaling or whatnot but they way you approached it simply doesn't work.
Related
I am trying to use generics to convert a JSON object into a GRPC response that has an enum, i.e.:
type GRPCResponse {
str string
enu EnumType
}
type EnumType int32
const (
Type1 EnumType = 0
Type2 EnumType = 1
)
The function for unmarshalling looks like this:
func assertHTTPResponseOK[T any](t *testing.T, endpoint string) T {
body, err := GetResponse(endpoint)
var v T
err := json.Unmarshal(body, &v)
require.Nil(t, err)
return v
}
And the code calling it looks like this:
assertHTTPResponseOK[*GRPCResponse](t, "some-endpoint")
The JSON object in question looks like this:
{"str":"hello", "enu": "Type2"}
and I am getting an error along the lines of:
json: cannot unmarshal string into Go struct field GRPCResponse.enu of type EnumType
From similar questions, I have seen that the usual advice is to use jsonpb.Unmarshal or protojson.Unmarshal instead of the typical json.Unmarshal.
In changing the Unmarshal function, I also had to change T to be protoreflect.ProtoMessage. However, this prevents me from passing a pointer to v to Unmarshal, because it is a pointer to the interface, not the interface. Of course, I also cannot pass in a nil pointer (not take the address of v).
So my questions are:
Is there a way to have the pointer of this generic object satisfy the interface protoreflect.ProtoMessage?
Is there a better Unmarshalling function that would better fit my problem?
I ended up passing in the object I am unmarshalling into.
obj := new(GRPCResponse)
assertHTTPResponseOK[*GRPCResponse](t, ctx, "some-endpoint", obj)
func assertHTTPResponseOK[T protoreflect.ProtoMessage](t *testing.T, ctx context.Context, endpoint string, object T) {
body, err := GetResponse(endpoint)
require.Nil(t, err)
err = protojson.Unmarshal(body, object)
require.Nil(t, err)
}
Here's a generics-friendly proto unmarshaller that avoids passing the second type, at the cost of a reflective invoke to peek the type inside the pointer and call it's constructor.
var msg T // Constrained to proto.Message
// Peek the type inside T (as T= *SomeProtoMsgType)
msgType := reflect.TypeOf(msg).Elem()
// Make a new one, and throw it back into T
msg = reflect.New(msgType).Interface().(T)
errUnmarshal := proto.Unmarshal(body, msg)
I'm new to golang generics and have the following setup.
I've gathered loads of different kinds of reports.
Each report has enclosing fields
So I wrapped it in a ReportContainerImpl
I've used a type argument of [T Reportable] where the Reportable is defined as follows
type Reportable interface {
ExportDataPointReport | ImportDataPointReport | MissingDataPointReport | SensorThresoldReport
}
Each of the type in the type constraint is structs that is to be embedded in the container.
type ReportContainerImpl[T Reportable] struct {
LocationID string `json:"lid"`
Provider string `json:"pn"`
ReportType ReportType `json:"m"`
Body T `json:"body"`
}
I use a discriminator ReportType to determine the concrete type when Unmarshal.
type ReportType string
const (
ReportTypeExportDataPointReport ReportType = "ExportDataPointReport"
ReportTypeImportDataPointReport ReportType = "ImportDataPointReport"
ReportTypeMissingDataPointReport ReportType = "MissingDataPointReport"
ReportTypeSensorThresoldReport ReportType = "SensorThresoldReport"
)
Since go does not support type assertion for struct (only interfaces) it is not possible to cast the type when Unmarshal. Also go does not support pointer to the "raw" generic type. Hence, I've created a interface that the ReportContainerImpl implements.
type ReportContainer interface {
GetLocationID() string
GetProvider() string
GetReportType() ReportType
GetBody() interface{}
}
The problem I then get is that I cannot do type constrains on the return type in any form or shape and am back at "freetext semantics" on the GetBody() function to allow for type assertion when Unmarshal is done.
container, err := UnmarshalReportContainer(data)
if rep, ok := container.GetBody().(ExportDataPointReport); ok {
// Use the ReportContainerImpl[ExportDataPointReport] here...
}
Maybe I'm getting this wrong? - but however I do this, I always end up with somewhere needs a interface{} or to know the exact type before Unmarshal
Do you have a better suggestion how to solve this in a type (safer) way?
Cheers,
Mario :)
For completeness I add the UnmarshalReportContainer here
func UnmarshalReportContainer(data []byte) (ReportContainer, error) {
type Temp struct {
LocationID string `json:"lid"`
Provider string `json:"pn"`
ReportType ReportType `json:"m"`
Body *json.RawMessage `json:"body"`
}
var temp Temp
err := json.Unmarshal(data, &temp)
if err != nil {
return nil, err
}
switch temp.ReportType {
case ReportTypeExportDataPointReport:
var report ExportDataPointReport
err := json.Unmarshal(*temp.Body, &report)
return &ReportContainerImpl[ExportDataPointReport]{
LocationID: temp.LocationID,
Provider: temp.Provider,
ReportType: temp.ReportType,
Body: report,
}, err
// ...
}
}
but however I do this, I always end up with somewhere needs a interface{} or to know the exact type before Unmarshal
Precisely.
The concrete types needed to instantiate some generic type or function like ReportContainerImpl or UnmarshalReportContainer must be known at compile time, when you write the code. JSON unmarshalling instead occurs at run-time, when you have the byte slice populated with the actual data.
To unmarshal dynamic JSON based on some discriminatory value, you still need a switch.
Do you have a better suggestion how to solve this in a type (safer) way?
Just forgo parametric polymorphism. It's not a good fit here. Keep the code you have now with json.RawMessage, unmarshal the dynamic data conditionally in the switch and return the concrete structs that implement ReportContainer interface.
As a general solution — if, and only if, you can overcome this chicken-and-egg problem and make type parameters known at compile time, you can write a minimal generic unmarshal function like this:
func unmarshalAny[T any](bytes []byte) (*T, error) {
out := new(T)
if err := json.Unmarshal(bytes, out); err != nil {
return nil, err
}
return out, nil
}
This is only meant to illustrate the principle. Note that json.Unmarshal already accepts any type, so if your generic function actually does nothing except new(T) and return, like in my example, it is no different than "inlining" the entire thing as if unmarshalAny didn't exist.
v, err := unmarshalAny[SomeType](src)
functionally equivalent as
out := &SomeType{}
err := json.Unmarshal(bytes, out)
If you plan to put more logic in unmarshalAny, its usage may be warranted. Your mileage may vary; in general, don't use type parameters when it's not actually necessary.
Possibly related: How to use interface type as a model in mgo (Go)?
I have a struct like so:
type Game struct {
ID bson.ObjectId
Type string
Location string
Details interface{}
}
type FeudDetails struct {
...
}
type TriviaDetails struct {
...
}
type BingoDetails struct {
...
}
I want to use the Type field of Game to unserialize Details into a specific type (like an instance of FeudDetails or BingoDetails). It would still be an interface{} in Game, but then I could do this:
feudDetails, ok := game.Details.(FeudDetails)
if ok {
// we know this is a Feud game, and we have the details
feudDetails.Round++
}
The docs say that "it is possible to unmarshal or marshal values partially" using bson.Raw, but they don't provide any examples I've been able to find.
I've tried using:
func (game *Game) SetBSON(r bson.Raw) error {
err := r.Unserialize(game)
if err != nil {
return nil
}
games[game.Type].LoadDetails(game) // this calls a function based on the Type to
// create a concrete value for that game.
return nil
}
I get a (ahem) stack overflow here. I assume this is because r.Unserialize is recursively calling SetBSON.
My goal is to use standard unserializing on all fields except Details, and then be able to use game.Type to determine how to process Details. If I do something like this:
type GameDetails interface{}
type Game struct {
...
Details GameDetails
}
func (details *GameDetails) SetBSON(r bson.Raw) error {
// game isn't defined
games[game.Type].LoadDetails(r, details)
}
then how can I access the outer Type field to know which game type to unserialize it to?
I'll also accept the answer "you're doing it all wrong, and a better pattern in Go is XYZ..."
EDIT: I also tried deserializing normally, and then casting the interface{} version of Details using game.Details.(FeudDetails), but the conversion failed. I guess I can't do it that way because the underlying type after unserialization is not a FeudDetails but rather probably map[string]interface{}.
EDIT 2: I thought I'd be clever and pre-populate an object with the right types when retrieving from the database (game := Game{Details: FeudDetails: {}} prior to calling db...One(&game)) but my trickery did not work:
DEBU[Mar 31 22:19:09.442] Caching show gid=5e814448ef5b9858b7ff4e57
TRAC[Mar 31 22:19:09.442] Before database call dtype=main.FeudDetails
TRAC[Mar 31 22:19:09.446] After database call dtype=bson.M
Ignore Details during (un)marshalling
Change the definition of Game so bson doesn't try to do anything with the Details field:
type Game struct {
...
Details interface{} `json:"details" bson:"-"`
}
Unmarshal Details manually
func (game *Game) SetBSON(r bson.Raw) error {
// Unmarshall everything except Details
type tempGame Game
err := r.Unmarshal((*tempGame)(game)) // this is necessary to prevent recursion
if err != nil {
return err
}
// Get the raw data for Details
var d struct {
Details bson.Raw
}
if err := r.Unmarshal(&d); err != nil {
return err
}
gameType, ok := games[game.Type]
if ok {
// Use individual processing based on game Type
game.Details, err = gameType.LoadDetails(d.Details)
if err != nil {
return err
}
// fmt.Sprintf("%T", game.Details) => main.FeudDetails
}
return nil
}
Marshal Details manually
As far as I can tell, the only way to get bson to include Details after we told it to omit it in the original struct, is to create a whole other structure, copy the data over one by one, and use that type in GetBSON. It seems like there must be a better way.
All I looking to do is to create an array of struct Response from a json encoded file.
the file that contains json data looks like this.
cat init.txt
{"events": [{"action":"cpr","id":69,"sha1":"abc","cpr":"cpr_data0"},{"action":"cpr","id":85,"sha1":"def","cpr":"cpr_data1"}]}
The way I have gone about approaching this is
I created a response of type map[string][]Response
.. decoded the JSON from the file
.. created a responseStruct of type []Response
But somehow when I inspect the Value they all look 0 or empty
map[events:[{ 0 } { 0 }]
What is wrong with the approach mentioned above.
type Response struct {
action string `json:"action"`
id int64 `json:"id"`
sha1 string `json:"sha1"`
cpr string `json:"cpr"`
}
func main() {
file, err := os.Open("init.txt")
if err != nil {
fmt.Println(err)
os.Exit(1)
}
var response map[string][]Response
err = json.NewDecoder(file).Decode(&response)
if err != nil {
fmt.Println(err)
os.Exit(1)
}
var responseArray []Response
responseArray = response["events"]
for _, responseStruct := range responseArray {
fmt.Println("id =", responseStruct.id)
fmt.Println("action =", responseStruct.action)
fmt.Println("sha1 = ", responseStruct.sha1)
fmt.Println("cpr =", responseStruct.cpr)
fmt.Println("==========")
}
fmt.Println(response)
}
Well If I modify the struct to look like this it works
type Response struct {
Action string `json:"action"`
ID int64 `json:"id"`
Sha1 string `json:"sha1"`
Cpr string `json:"cpr"`
}
So my question is this how the stuff would work, can't I get the above code to work in the way it is?
Go has the notion of public and private fields in objects, and the only distinction is whether they start with an initial capital letter or not. This applies to not just code modules but also the reflect package and things that use it. So in your initial version
type Response struct {
action string `json:"action"`
...
}
nothing outside your source package, not even the "encoding/json" module, can see the private fields, so it can't fill them in. Changing these to public fields by capitalizing Action and the other field names makes them visible to the JSON decoder.
Lowercase struct elements in golang are private, Hence json decoder (which is an external package) can't access them.
Its able to create the struct object but not able to set values. They appear zero because they 0 is default value.
I am writing a go client for the Flowdock API. Their API for Message has a number of attributes two of which are Event and Content
When Event = message then Content is a string. When Event = comment Content is a JSON object.
I want to defer unmarhsalling Content until it is needed. To do this I map RawContent in my struct and define a Content() method on Message to return the correct object.
Here is the code to illustrate:
package main
import (
"fmt"
"encoding/json"
)
// Message represents a Flowdock chat message.
type Message struct {
Event *string `json:"event,omitempty"`
RawContent *json.RawMessage `json:"content,omitempty"`
}
func (m *Message) Content() (content interface{}) {
// when m.Event is a message the RawContent is a string
// real code handles unmarshaling other types (removed for this example)
return string(*m.RawContent)
}
func main() {
var message = &Message{}
rawJSON := `{"event": "message", "content": "Howdy-Doo #Jackie #awesome"}`
if err := json.Unmarshal([]byte(rawJSON), &message); err != nil {
panic(err.Error())
}
event := "message"
rawMessage := json.RawMessage("Howdy-Doo #Jackie #awesome")
want := &Message{
Event: &event,
RawContent: &rawMessage,
}
fmt.Println(message.Content(), want.Content())
}
The result of running this is: http://play.golang.org/p/eds_AA6Aay
"Howdy-Doo #Jackie #awesome" Howdy-Doo #Jackie #awesome
Note: message.Content() and want.Content() are different!
At first I did not expect the quotes to be included in message but it makes sense because of how the JSON is parsed. It is just a slice of the whole rawJSON string.
So my questions are:
Should I just strip off the quotes?
If so what is the simplest way in Go to strip? maybe this: http://play.golang.org/p/kn8XKOqF0z lines(6, 19)?
Is this even the right approach to handling JSON that can have different types for the same attribute?
Here is a more complete example showing how I handle a JSON RawContent: http://play.golang.org/p/vrBJ5RYcql
Question 1:
No, you should json.Unmarshal the content also when it only contains a string. Apart from the quotes, JSON strings may also contain backslash-escaped control characters.
Question 2:
Because of the answer in Question 1, do the following for the "message" case:
var s string
if err := json.Unmarshal([]byte(*m.RawContent), &s); err != nil {
panic(err)
}
return s
Question 3:
It is a good approach to Unmarshal the event type string and use RawMessage to store the rest of the JSON until you've evaluated the type and know what kind of structure the content has.
You might want to consider also having a specific type also for just simple string contents, eg.:
type MessageContent string
This way you are free to implement methods on the type, allowing the Content method to return some other interface than just the empty interface{}.
Note:
Beware that, if you also json.Unmarshal the message string as I suggested, your Playground example will panic when trying to Unmarshal your non-quoted want.RawContent string, because it is not valid JSON.