Golang abstracted Interface Slice conversion - go

I have a list of objects (olievere/Elastic SearchResult.Hits to be exact). Each of these has a json.RawMessage object and I'm looking to create an abstractable method that takes in an interface slice of any struct, Unmarshal's each individual hits' json.RawMessage into said struct, and appends it to the passed in []interface.
This func is not supposed to have any logic or insight into the desired business layer struct, and the DB Call is interfaced pretty heavily, and as such has no visibility into the Elastic package mentioned above. Example of what I was attempting to do...
foo.go
import (bar, package)
type TestStruct struct {
Slice []*package.Struct // package.Struct has a value of Source which is a
// json.RawMessage
}
func GetData() bar.Test {
return &TestStruct{*package.GetData()}
}
func (result TestStruct) UnmarshalStruct(v []interface{}) {
for _, singleStruct := range result.Slice {
append(json.Unmarshal(singleStruct, &v))
}
Second File
bar.go
type Handler interface {
GetData() Test
}
type Test interface {
UnmarshalStruct
}
type OtherType struct {
foo string
bar string
}
func RetrieveData() []OtherType {
handler := New(Handler)
test := handler.GetData()
var typeSlice []OtherType
test.UnmarshalStruct(&typeSlice)
}
I'm looking to hand something of type []OtherType, or any other new struct I decide to create, to UnmarshalStruct, and have it return to me that same struct, just full of data
As an example, I have two different types of data I'll be searching for from Elastic. I will be receiving a list of ONE of the following two objects.
{ 'foo': '',
'id':
}
And in a different index
{ 'bar': '',
'baz': '',
'eee': ''
}
Each of these will naturally require two different structs.
However, I desire a single method to be able to decode either of these lists. I'll be given below, and using the same function I want to be able to convert this to a bar struct, and the other type to a foo struct.
{ 'source': [
{ 'bar': '',
'baz': '',
'eee': ''
},
{ 'bar': '',
'baz': '',
'eee': ''
},
{ 'bar': '',
'baz': '',
'eee': ''
}
]
}

There's really no way to do what you want without reflection. I would personally structure this differently, so that you unmarshal into more generic types, like a map[string]string, or as #ThunderCat shows, get rid of the intermediary state and unamrshal directly into the correct types. But it can be done...
(I moved the json.RawMessage directly into TestStruct to get rid of one level of indirection and make the example more clear)
type TestStruct struct {
Slice []json.RawMessage
}
func (t TestStruct) UnmarshalStruct(v interface{}) error {
// get the a Value for the underlying slice
slice := reflect.ValueOf(v).Elem()
// make sure we have adequate capacity
slice.Set(reflect.MakeSlice(slice.Type(), len(t.Slice), len(t.Slice)))
for i, val := range t.Slice {
err := json.Unmarshal(val, slice.Index(i).Addr().Interface())
if err != nil {
return err
}
}
return nil
}
You can then call it like so
var others []OtherType
err := ts.UnmarshalStruct(&others)
if err != nil {
log.Fatal(err)
}
http://play.golang.org/p/dgly2OOXDG

If I understand correctly, you want to unmarshal data to slices of two types:
type A struct {
Foo string `json:"foo"`
ID string `json:"id"`
}
type B struct {
Bar string `json:"bar"`
Baz string `json:"baz"`
Eee string `json:"eee"`
}
from a SearchHit Source.
The JSON package can do most of the work for you:
func executeQuery(q Query, v interface{}) error {
// Get a SearchHit. I am making this up.
// I have no idea how the package works.
searchHit, err := getHit(q)
if err != nil {
return err
}
// This is the important part. Convert the raw message to
// a slice of bytes and decode to the caller's slice.
return json.Unmarshal([]byte(*searchHit.Source), v)
}
You can call this function to decode to a slice of the types or a slice of pointers to the types.
// Slice of type
var s1 []TypeA
if err := executeQuery(q1, &s1); err != nil {
// handle error
}
// Slice of pointer to type
var s2 []*TypeB
if err := error(q2, &s2); err != nil {
// handle error
}
I know that this is not a direct answer to the question, but this is how this scenario is typically handled.

I don't believe this is easy to do. In the Raw Message Example in the godocs they use a value within the json, "Space" in their example, to determine which struct type to unmarshal into.
For this to work, the function would have to have some way of getting every struct that has been defined ever for the program, and then it would have to examine each json object and compare it to each struct using reflection to figure out which type it most likely is. And what if there are multiple structs that "could be it"? Then conflict resolution complicates things.
In short, I don't think you can do this.

Related

Merging two different structs and marshalling them - Golang

Let's say I have the first struct as
type Person struct {
Name string `json:"person_name"`
Age int `json:"person_age"`
Data map[string]interface{} `json:"data"`
}
and I am trying to marshal an array of the above struct
Things work well till here and a sample response I receive is
[
{
"person_name":"name",
"person_age":12,"data":{}
},
{
"person_name":"name2",
"person_age":12,"data":{}
}
]
Now, I need to append another struct over here and the final response should like
[
{
"person_name":"name",
"person_age":12,"data":{}
},
{
"person_name":"name2",
"person_age":12,"data":{}
},
{
"newData":"value"
}
]
So can someone help on this and how i can achieve this ?
I tried by creating an []interface{} and then iterating over person to append each data, but the issue in this approach is that it makes the Data as null if in case it's an empty string.
I would need it be an empty map only.
Let me prefix this by saying this looks to me very much like you might be dealing with an X-Y problem. I can't really think of many valid use-cases where one would end up with a defined data-type that has to somehow be marshalled alongside a completely different, potentially arbitrary/freeform data structure. It's possible, though, and this is how you could do it:
So you just want to append a completely different struct to the data-set, then marshal it and return the result as JSON? You'll need to create a new slice for that:
personData := []Person{} // person 1 and 2 here
more := map[string]string{ // or some other struct
"newdata": "value",
}
allData := make([]any, 0, len(personData) + 1) // the +1 is for the more, set cap to however many objects you need to marshal
for _, p := range personData {
allData = append(allData, p) // copy over to this slice, because []Person is not compatible with []any
}
allData = append(allData, more)
bJSON, err := json.Marshal(allData)
if err != nil {
// handle
}
fmt.Println(string(bJSON))
Essentially, because you're trying to marshal a slice containing multiple different types, you have to add all objects to a slice of type any (short for interface{}) before marshalling it all in one go
Cleaner approaches
There are much, much cleaner approaches that allow you to unmarshal the data, too, assuming the different data-types involved are known beforehand. Consider using a wrapper type like so:
type Person struct {} // the one you have
type NewData {
NewData string `json:"newdata"`
}
type MixedData struct {
*Person
*NewData
}
In this MixedData type, both Person and NewData are embedded, so MixedData will essentially act as a merged version of all embedded types (fields with the same name should be overridden at this level). With this type, you can marshal and unmarshal the JSON accordingly:
allData := []MixedData{
{
Person: &person1,
},
{
Person: &person2,
},
{
NewData: &newData,
},
}
Similarly, when you have a JSON []byte input, you can unmarshal it same as you would any other type:
data := []MixedData{}
if err := json.Unmarshal(&data, in); err != nil {
// handle
}
fmt.Printf("%#v\n", data) // it'll be there
It pays to add some functions/getters to the MixedData type, though:
func (m MixedData) IsPerson() bool { return m.Person != nil }
func (m MixedData) Person() *Person {
if m.Person == nil {
return nil
}
cpy := *m.Person // create a copy to avoid shared pointers
return &cpy // return pointer to the copy
}
Do the same for all embedded types and this works like a charm.
As mentioned before, should your embedded types contain fields with the same name, then you should override them in the MixedData type. Say you have a Person and Address type, and both have an ID field:
type MixedData struct {
ID string `json:"id"`
*Person
*Address
}
This will set the ID value on the MixedData type, and all other (non-shared) fields on the corresponding embedded struct. You can then use the getters to set the ID where needed, or use a custom unmarshaller, but I'll leave that to you to implement

Returning a pointer to a struct from a function in Go

I have two public structs that contain different data, and a private intermediate struct containing either of the two public structs. I also have a function that unmarshalls the intermediate struct, determines which public struct it contains, and returns one of the two public structs.
The problem I'm facing is the return value of the last function. At it's simplest I thought I could return *struct{} but I keep getting a type mismatch in my IDE.
I apologize for posting more code than is probably necessary, but I'm trying to make it as close as possible to the code I'm working on.
package main
import (
"encoding/json"
"errors"
)
// These vars are some errors I'll use in the functions later on
var (
errInvalidBase64 = errors.New("invalid base64")
errInvalidStructType = errors.New("invalid struct type")
)
// Struct1 public struct
type Struct1 struct {
FName string `json:"first-name"`
LName string `json:"last-name"`
}
// Struct2 public struct
type Struct2 struct {
Date string `json:"date"`
Items []int `json:"items"`
}
// intermediateStruct private struct
// The Type field indicates which kind of struct Data contains (Struct1 or Struct2)
// The Data field contains either Struct1 or Struct2 which was previously marshalled into JSON
type intermediateStruct struct {
Type structType
Data []byte
}
// The following type and const are my understanding of an enum in Go
// structType is a private type for the type of struct intermediateStruct contains
type structType int
// These public constants are just to keep my hands out of providing values for the different struct types
const (
StructType1 structType = iota
StructType2
)
// unmarshalStruct1 unmarshalls JSON []byte into a new Struct1 and returns a pointer to that struct
func unmarshalStruct1(b []bytes) (*Struct1, error) {
newStruct1 := new(Struct1)
err := json.Unmarshal(b, newStruct1)
if err != nil {
return nil, errInvalidBase64
}
return newStruct1, nil
}
// unmarshalStruct2 unmarshalls JSON []byte into a new Struct2 and returns a pointer to that struct
func unmarshalStruct2(b []bytes) (*Struct2, error) {
newStruct2 := new(Struct2)
err := json.Unmarshal(b, newStruct2)
if err != nil {
return nil, errInvalidBase64
}
return newStruct2, nil
}
// receiveStruct accepts *intermediateStruct who's Data field contains either Struct1 or Struct2
// This function needs to return either *Struct1 or *Struct2 and an error
func receiveStruct(iStruct *intermediateStruct) (*struct{}, error) {
switch iStruct.Type {
case StructType1:
struct1, err := unmarshalStruct1(iStruct.Data)
if err != nil {
return nil, err
}
// The following line is where I'm getting the type mismatch
return struct1, nil
case StructType2:
struct2, err := unmarshalStruct2(iStruct.Data)
if err != nil {
return nil, err
}
// The following line is another type mismatch
return struct2, nil
default:
return nil, errInvalidStructType
}
}
I know there's a way to do what I'm trying to achieve - I just lack the experience/understanding to get there.
Thanks for any and all input!
Your unmarshallStruct function returns a pointer to type Struct1 or Struct2 (depending on which version of the function gets called). And therefore the variables struct1 and struct2 are pointers to types Struct1 and Struct2 respectively. Neither is a pointer to type struct (which is not a real Go type anyways I must add). A struct is a keyword which helps to declare types containing fields/attributes.
Depending on the use-cases you have in mind for the rest of your code, can instead try any of the below:
As mkopriva suggested, return a interface{} object, but you'd need to use type assertion to actually make sure of the object
Define an interface which both Struct1 and Struct2 implement, and return a pointer to this
Make separate functions which work with either Struct1 or Struct2. This is not necessarily as bad as it sounds as Go lets you pass functions in the same way you'd pass types (see example of the Less() function in sort package).

What is the equivalence of Type Name Handling with encoding/json in Go?

Short Description in Prose
I have a situation where I want to unmarshal JSON data into an array of structs (either Foo or Bar and many more) that all implement a common interface MyInterface. Also all of the eligble struct types that implement the interface have a common field which I named Discrimininator in the example below.
The Discriminator¹ allows to bi-uniquely find the correct struct type for each value of Discriminator.
The Problem and Error Message
But during unmarshling the code does not "know" which is the correct "target" type. The unmarshaling fails.
cannot unmarshal object into Go value of type main.MyInterface
MWE in
https://play.golang.org/p/Dw1hSgUezLH
package main
import (
"encoding/json"
"fmt"
)
type MyInterface interface {
// some other business logic methods go here!
GetDiscriminator() string // GetDiscriminator returns something like a key that is unique per struct type implementing the interface
}
type BaseStruct struct {
Discriminator string // will always be "Foo" for all Foos, will always be "Bar" for all Bars
}
type Foo struct {
BaseStruct
// actual fields of the struct don't matter. it's just important that they're different from Bar
FooField string
}
func (foo *Foo) GetDiscriminator() string {
return foo.Discriminator
}
type Bar struct {
BaseStruct
// actual fields of the struct don't matter. it's just important that they're different from Foo
BarField int
}
func (bar *Bar) GetDiscriminator() string {
return bar.Discriminator
}
// Foo and Bar both implement the interface.
// Foo and Bars are always distinguishible if we check the value of Discriminator
func main() {
list := []MyInterface{
&Bar{
BaseStruct: BaseStruct{Discriminator: "Bar"},
BarField: 42,
},
&Foo{
BaseStruct: BaseStruct{Discriminator: "Foo"},
FooField: "hello",
},
}
jsonBytes, _ := json.Marshal(list)
jsonString := string(jsonBytes)
fmt.Println(jsonString)
// [{"Discriminator":"Bar","BarField":42},{"Discriminator":"Foo","FooField":"hello"}]
var unmarshaledList []MyInterface
err := json.Unmarshal(jsonBytes, &unmarshaledList)
if err != nil {
// Unmarshaling failed: json: cannot unmarshal object into Go value of type main.MyInterface
fmt.Printf("Unmarshaling failed: %v", err)
}
}
In other languages
TypeNameHandling as known from .NET
In Newtonsoft, a popular .NET JSON Framework, this is solved by a something called "TypeNameHandling" or can be solved with a custom JsonConverter . The framework will add something like a magic "$type" key on root level to the serialized/marshaled JSON which is then used to determine the original type on deserialization/unmarshaling.
Polymorphism in ORMs
¹A similar situation occurs under the term "polymorphism" in ORMs when instances of multiple types having the same base are saved in the same table. One typically introduces a discriminator column, hence the name in above example.
You can implement a custom json.Unmarshaler. For that you'll need to use a named slice type instead of the unnamed []MyInterface.
Within the custom unmarshaler implementation you can unmarshal the JSON array into a slice where each element of the slice is a json.RawMessage representing the corresponding JSON object. After that you can iterate over the slice of raw messages. In the loop unmarshal from each raw message only the Discriminator field, then use the Discriminator field's value to determine what the correct type is into which the full raw message can be unmarshaled, finally unmarshal the full message and add the result to the receiver.
type MyInterfaceSlice []MyInterface
func (s *MyInterfaceSlice) UnmarshalJSON(data []byte) error {
array := []json.RawMessage{}
if err := json.Unmarshal(data, &array); err != nil {
return err
}
*s = make(MyInterfaceSlice, len(array))
for i := range array {
base := BaseStruct{}
data := []byte(array[i])
if err := json.Unmarshal(data, &base); err != nil {
return err
}
var elem MyInterface
switch base.Discriminator {
case "Foo":
elem = new(Foo)
case "Bar":
elem = new(Bar)
}
if elem == nil {
panic("whoops")
}
if err := json.Unmarshal(data, elem); err != nil {
return err
}
(*s)[i] = elem
}
return nil
}
https://play.golang.org/p/mXiZrF392aV

How do I use bson.SetBSON or bson.Raw?

Possibly related: How to use interface type as a model in mgo (Go)?
I have a struct like so:
type Game struct {
ID bson.ObjectId
Type string
Location string
Details interface{}
}
type FeudDetails struct {
...
}
type TriviaDetails struct {
...
}
type BingoDetails struct {
...
}
I want to use the Type field of Game to unserialize Details into a specific type (like an instance of FeudDetails or BingoDetails). It would still be an interface{} in Game, but then I could do this:
feudDetails, ok := game.Details.(FeudDetails)
if ok {
// we know this is a Feud game, and we have the details
feudDetails.Round++
}
The docs say that "it is possible to unmarshal or marshal values partially" using bson.Raw, but they don't provide any examples I've been able to find.
I've tried using:
func (game *Game) SetBSON(r bson.Raw) error {
err := r.Unserialize(game)
if err != nil {
return nil
}
games[game.Type].LoadDetails(game) // this calls a function based on the Type to
// create a concrete value for that game.
return nil
}
I get a (ahem) stack overflow here. I assume this is because r.Unserialize is recursively calling SetBSON.
My goal is to use standard unserializing on all fields except Details, and then be able to use game.Type to determine how to process Details. If I do something like this:
type GameDetails interface{}
type Game struct {
...
Details GameDetails
}
func (details *GameDetails) SetBSON(r bson.Raw) error {
// game isn't defined
games[game.Type].LoadDetails(r, details)
}
then how can I access the outer Type field to know which game type to unserialize it to?
I'll also accept the answer "you're doing it all wrong, and a better pattern in Go is XYZ..."
EDIT: I also tried deserializing normally, and then casting the interface{} version of Details using game.Details.(FeudDetails), but the conversion failed. I guess I can't do it that way because the underlying type after unserialization is not a FeudDetails but rather probably map[string]interface{}.
EDIT 2: I thought I'd be clever and pre-populate an object with the right types when retrieving from the database (game := Game{Details: FeudDetails: {}} prior to calling db...One(&game)) but my trickery did not work:
DEBU[Mar 31 22:19:09.442] Caching show gid=5e814448ef5b9858b7ff4e57
TRAC[Mar 31 22:19:09.442] Before database call dtype=main.FeudDetails
TRAC[Mar 31 22:19:09.446] After database call dtype=bson.M
Ignore Details during (un)marshalling
Change the definition of Game so bson doesn't try to do anything with the Details field:
type Game struct {
...
Details interface{} `json:"details" bson:"-"`
}
Unmarshal Details manually
func (game *Game) SetBSON(r bson.Raw) error {
// Unmarshall everything except Details
type tempGame Game
err := r.Unmarshal((*tempGame)(game)) // this is necessary to prevent recursion
if err != nil {
return err
}
// Get the raw data for Details
var d struct {
Details bson.Raw
}
if err := r.Unmarshal(&d); err != nil {
return err
}
gameType, ok := games[game.Type]
if ok {
// Use individual processing based on game Type
game.Details, err = gameType.LoadDetails(d.Details)
if err != nil {
return err
}
// fmt.Sprintf("%T", game.Details) => main.FeudDetails
}
return nil
}
Marshal Details manually
As far as I can tell, the only way to get bson to include Details after we told it to omit it in the original struct, is to create a whole other structure, copy the data over one by one, and use that type in GetBSON. It seems like there must be a better way.

JSON decode with flexible type

I need to specify a type for decoding JSON data in a flexible manner, meaning the type needs to be specified at runtime.
Consider this snippet: http://play.golang.org/p/F-Jy4ufMPz
s := `{"b":[{"x":9},{"x":4}]}`
var a struct {
B []interface{}
}
err := json.Unmarshal([]byte(s), &a)
if err != nil {
panic(err)
}
fmt.Println(a)
Which will produce {[map[x:9] map[x:4]]}. I want to decode to an array of a specific (struct) type instead of []interface{}, without specifying it at compile time.
Is that somehow possible without creating the array up front? (the number of returned items is unknown)
The only way I can think of right now is to encode the returned maps again later, and decode them to the specified type, which would create unnecessary processing overhead.
If not specifying it at compile time, you still need to specify it somewhere.
If specified before the retrieval of the Json data, you can simply do a switch case, Unmarshalling it to your desired object.
If specified within the Json data, you can marshal the "flexible" part into a json.RawMessage to process it after you've decided what type of struct is suitable:
package main
import (
"encoding/json"
"fmt"
)
var s = `{"type":"structx", "data":{"x":9,"xstring":"This is structX"}}`
type JsonStruct struct {
Type string
Data json.RawMessage
}
type StructX struct {
X float64
Xstring string
}
type StructY struct {
Y bool
}
func main() {
var a *JsonStruct
err := json.Unmarshal([]byte(s), &a)
if err != nil {
panic(err)
}
switch a.Type {
case "structx":
// We Unmashal the RawMessage part into a StructX
var s *StructX
json.Unmarshal([]byte(a.Data), &s)
if err != nil {
panic(err)
}
fmt.Println(s)
case "structy":
// Do the same but for structY
}
}
Playground

Resources