Let's say I have the first struct as
type Person struct {
Name string `json:"person_name"`
Age int `json:"person_age"`
Data map[string]interface{} `json:"data"`
}
and I am trying to marshal an array of the above struct
Things work well till here and a sample response I receive is
[
{
"person_name":"name",
"person_age":12,"data":{}
},
{
"person_name":"name2",
"person_age":12,"data":{}
}
]
Now, I need to append another struct over here and the final response should like
[
{
"person_name":"name",
"person_age":12,"data":{}
},
{
"person_name":"name2",
"person_age":12,"data":{}
},
{
"newData":"value"
}
]
So can someone help on this and how i can achieve this ?
I tried by creating an []interface{} and then iterating over person to append each data, but the issue in this approach is that it makes the Data as null if in case it's an empty string.
I would need it be an empty map only.
Let me prefix this by saying this looks to me very much like you might be dealing with an X-Y problem. I can't really think of many valid use-cases where one would end up with a defined data-type that has to somehow be marshalled alongside a completely different, potentially arbitrary/freeform data structure. It's possible, though, and this is how you could do it:
So you just want to append a completely different struct to the data-set, then marshal it and return the result as JSON? You'll need to create a new slice for that:
personData := []Person{} // person 1 and 2 here
more := map[string]string{ // or some other struct
"newdata": "value",
}
allData := make([]any, 0, len(personData) + 1) // the +1 is for the more, set cap to however many objects you need to marshal
for _, p := range personData {
allData = append(allData, p) // copy over to this slice, because []Person is not compatible with []any
}
allData = append(allData, more)
bJSON, err := json.Marshal(allData)
if err != nil {
// handle
}
fmt.Println(string(bJSON))
Essentially, because you're trying to marshal a slice containing multiple different types, you have to add all objects to a slice of type any (short for interface{}) before marshalling it all in one go
Cleaner approaches
There are much, much cleaner approaches that allow you to unmarshal the data, too, assuming the different data-types involved are known beforehand. Consider using a wrapper type like so:
type Person struct {} // the one you have
type NewData {
NewData string `json:"newdata"`
}
type MixedData struct {
*Person
*NewData
}
In this MixedData type, both Person and NewData are embedded, so MixedData will essentially act as a merged version of all embedded types (fields with the same name should be overridden at this level). With this type, you can marshal and unmarshal the JSON accordingly:
allData := []MixedData{
{
Person: &person1,
},
{
Person: &person2,
},
{
NewData: &newData,
},
}
Similarly, when you have a JSON []byte input, you can unmarshal it same as you would any other type:
data := []MixedData{}
if err := json.Unmarshal(&data, in); err != nil {
// handle
}
fmt.Printf("%#v\n", data) // it'll be there
It pays to add some functions/getters to the MixedData type, though:
func (m MixedData) IsPerson() bool { return m.Person != nil }
func (m MixedData) Person() *Person {
if m.Person == nil {
return nil
}
cpy := *m.Person // create a copy to avoid shared pointers
return &cpy // return pointer to the copy
}
Do the same for all embedded types and this works like a charm.
As mentioned before, should your embedded types contain fields with the same name, then you should override them in the MixedData type. Say you have a Person and Address type, and both have an ID field:
type MixedData struct {
ID string `json:"id"`
*Person
*Address
}
This will set the ID value on the MixedData type, and all other (non-shared) fields on the corresponding embedded struct. You can then use the getters to set the ID where needed, or use a custom unmarshaller, but I'll leave that to you to implement
Related
Short Description in Prose
I have a situation where I want to unmarshal JSON data into an array of structs (either Foo or Bar and many more) that all implement a common interface MyInterface. Also all of the eligble struct types that implement the interface have a common field which I named Discrimininator in the example below.
The Discriminator¹ allows to bi-uniquely find the correct struct type for each value of Discriminator.
The Problem and Error Message
But during unmarshling the code does not "know" which is the correct "target" type. The unmarshaling fails.
cannot unmarshal object into Go value of type main.MyInterface
MWE in
https://play.golang.org/p/Dw1hSgUezLH
package main
import (
"encoding/json"
"fmt"
)
type MyInterface interface {
// some other business logic methods go here!
GetDiscriminator() string // GetDiscriminator returns something like a key that is unique per struct type implementing the interface
}
type BaseStruct struct {
Discriminator string // will always be "Foo" for all Foos, will always be "Bar" for all Bars
}
type Foo struct {
BaseStruct
// actual fields of the struct don't matter. it's just important that they're different from Bar
FooField string
}
func (foo *Foo) GetDiscriminator() string {
return foo.Discriminator
}
type Bar struct {
BaseStruct
// actual fields of the struct don't matter. it's just important that they're different from Foo
BarField int
}
func (bar *Bar) GetDiscriminator() string {
return bar.Discriminator
}
// Foo and Bar both implement the interface.
// Foo and Bars are always distinguishible if we check the value of Discriminator
func main() {
list := []MyInterface{
&Bar{
BaseStruct: BaseStruct{Discriminator: "Bar"},
BarField: 42,
},
&Foo{
BaseStruct: BaseStruct{Discriminator: "Foo"},
FooField: "hello",
},
}
jsonBytes, _ := json.Marshal(list)
jsonString := string(jsonBytes)
fmt.Println(jsonString)
// [{"Discriminator":"Bar","BarField":42},{"Discriminator":"Foo","FooField":"hello"}]
var unmarshaledList []MyInterface
err := json.Unmarshal(jsonBytes, &unmarshaledList)
if err != nil {
// Unmarshaling failed: json: cannot unmarshal object into Go value of type main.MyInterface
fmt.Printf("Unmarshaling failed: %v", err)
}
}
In other languages
TypeNameHandling as known from .NET
In Newtonsoft, a popular .NET JSON Framework, this is solved by a something called "TypeNameHandling" or can be solved with a custom JsonConverter . The framework will add something like a magic "$type" key on root level to the serialized/marshaled JSON which is then used to determine the original type on deserialization/unmarshaling.
Polymorphism in ORMs
¹A similar situation occurs under the term "polymorphism" in ORMs when instances of multiple types having the same base are saved in the same table. One typically introduces a discriminator column, hence the name in above example.
You can implement a custom json.Unmarshaler. For that you'll need to use a named slice type instead of the unnamed []MyInterface.
Within the custom unmarshaler implementation you can unmarshal the JSON array into a slice where each element of the slice is a json.RawMessage representing the corresponding JSON object. After that you can iterate over the slice of raw messages. In the loop unmarshal from each raw message only the Discriminator field, then use the Discriminator field's value to determine what the correct type is into which the full raw message can be unmarshaled, finally unmarshal the full message and add the result to the receiver.
type MyInterfaceSlice []MyInterface
func (s *MyInterfaceSlice) UnmarshalJSON(data []byte) error {
array := []json.RawMessage{}
if err := json.Unmarshal(data, &array); err != nil {
return err
}
*s = make(MyInterfaceSlice, len(array))
for i := range array {
base := BaseStruct{}
data := []byte(array[i])
if err := json.Unmarshal(data, &base); err != nil {
return err
}
var elem MyInterface
switch base.Discriminator {
case "Foo":
elem = new(Foo)
case "Bar":
elem = new(Bar)
}
if elem == nil {
panic("whoops")
}
if err := json.Unmarshal(data, elem); err != nil {
return err
}
(*s)[i] = elem
}
return nil
}
https://play.golang.org/p/mXiZrF392aV
I've read around and rightfully so, Golang doesn't allow you to assign the data from one struct to another if the types are different. However, in my case, I have a need to make the packet size of the data as light as possible (very weighty inbound JSON property names, and very large json requests) but when fetching, we only need short codes for the properties.
// FrequencyCondition ...
type FrequencyCondition struct {
Occurrence string `json:"occurrence,omitempty"`
IntervalUnit string `json:"interval_unit,omitempty"`
IntervalType string `json:"interval_type,omitempty"`
Comparison string `json:"comparison,omitempty"`
Identifier string `json:"identifier,omitempty"`
ValueText []string `json:"value_text,omitempty"`
ValueInt []int `json:"value_int,omitempty"`
ValueTimestamp []string `json:"value_timestamp,omitempty"`
ValueDate []string `json:"value_date,omitempty"`
ValueDecimal []string `json:"value_decimal,omitempty"`
}
// FrequencyConditionMinified ...
type FrequencyConditionMinified struct {
Occurrence string `json:"o,omitempty"`
IntervalUnit string `json:"iu,omitempty"`
IntervalType string `json:"it,omitempty"`
Comparison string `json:"c,omitempty"`
Identifier string `json:"i,omitempty"`
ValueText []string `json:"vt,omitempty"`
ValueInt []int `json:"vi,omitempty"`
ValueTimestamp []string `json:"vts,omitempty"`
ValueDate []string `json:"vdt,omitempty"`
ValueDecimal []string `json:"vdc,omitempty"`
}
// Condition ...
type Condition struct {
ConditionType string `json:"condition_type"`
Conditions []*Condition `json:"conditions,omitempty"`
NextConditionOperator Conditional `json:"next_condition_operator,omitempty"`
Input *InputCondition `json:"input,omitempty"`
Event *EventCondition `json:"event,omitempty"`
Frequency *FrequencyCondition `json:"frequency,omitempty"`
}
// MinifiedCondition ...
type MinifiedCondition struct {
ConditionType string `json:"t"`
Conditions []MinifiedCondition `json:"c,omitempty"`
NextConditionOperator ConditionalMinified `json:"nco,omitempty"`
Input InputConditionMinified `json:"i,omitempty"`
Event EventConditionMinified `json:"e,omitempty"`
Frequency FrequencyConditionMinified `json:"f,omitempty"`
}
type Response struct {
Conditions []*Condition `json:"conditions"`
}
type ResponseMinified struct {
Conditions []*ConditionalMinified `json:"c"`
}
Whilst this seems silly, when there are thousands of these being returned tens of thousands of times a day, we save quite a lot of bandwidth from the DB server > App.
func ParseJson(data string) {
var response Response
err := json.Unmarshal([]byte(data), &response)
if err != nil {
fmt.Println(err)
return
}
var minifiedResponse = ResponseMinified(response) <--- Error here
Notice the only difference is at the very top level of the structs Response/Condition - under that, it's all OK. But the internal types of those Minified structs end up referencing types that are identical.
Is there any smart, or better way of converting than iterating through each struct and manually assigning the properties, then forming the struct back together again?
Is it possible in Go to create a struct type programmatically (i.e. not in the compiled source code)?
We have a particular use case where a type will be created via user-defined metadata (so the schema/types are not known in advance)
and will vary for every customer. We would then need to auto-generate REST services for those and persist them in a NoSQL backend.
We would also need to define different dynamic validators per field (e.g. mandatory, regex, max/min size, max/min value, a reference to another type instance, etc.)
I was wondering if something similar is possible in the Go?
Edit 1:
For example
From frontend in JSON
For customer 1:
{
"id":1,
"patientid":1,
"name":"test1",
"height":"160",
"weight":"70",
"temp":"98"
}
For customer 2:
{
"id":2,
"patientid":1,
"name":"test1",
"height":"160",
"weight":"70"
}
For customer 3
may be different new fields will add
Backend
// For One customer need to have these fields
type Vitalsigns struct {
ID int64 `datastore:"-"`
PatientID int64 `json:"patientid,omitempty"`
Name string `json:"name,omitempty"`
Height string `json:"height,omitempty"`
Weight string `json:"weight,omitempty"`
Temp string `json:"temp,omitempty"`
}
// Another need to have these fields
type Vitalsigns struct {
ID int64 `datastore:"-"`
PatientID int64 `json:"patientid,omitempty"`
Name string `json:"name,omitempty"`
Height string `json:"height,omitempty"`
Weight string `json:"weight,omitempty"`
}
//CreateVitalsignsHandler is to create vitals for a patient
func CreateVitalsignsHandler(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
//Creating the Vitalsigns
kinVitalsigns := &Vitalsigns{}
ctx := appengine.NewContext(r)
if err := json.NewDecoder(r.Body).Decode(kinVitalsigns); err != nil {
RespondErr(w, r, http.StatusInternalServerError, err.Error())
return
}
//Getting namespace
namespace := ps.ByName("namespace")
ctx, err := appengine.Namespace(ctx, namespace)
if err != nil {
log.Infof(ctx, "Namespace error from CreateVitalsignsHandler")
RespondErr(w, r, http.StatusInternalServerError, err.Error())
return
}
//Geting the patientID
pID, err := strconv.Atoi(ps.ByName("id"))
if err != nil {
log.Infof(ctx, "Srting to Int64 conversion error from CreateVitalsignsHandler")
RespondErr(w, r, http.StatusInternalServerError, err.Error())
return
}
patientID := int64(pID)
kinVitalsigns.PatientID = patientID
//Generating the key
vitalsignsKey := datastore.NewIncompleteKey(ctx, "Vitalsigns", nil)
//Inserting the data to the datastore
vk, err := datastore.Put(ctx, vitalsignsKey, kinVitalsigns)
if err != nil {
log.Infof(ctx, "Entity creation was failed from CreateVitalsignsHandler")
RespondErr(w, r, http.StatusInternalServerError, err.Error())
return
}
kinVitalsigns.ID = vk.IntID()
message := "Vitalsigns created successfully!! "
Respond(w, r, http.StatusOK, SuccessResponse{kinVitalsigns.ID, 0, "", message})
return
}
Edit: Your edit reveals you want to handle dynamic objects to put / retrieve from Google Datastore. For this it is completely unnecessary to create struct types at runtime, you may just use a dynamic map presented in this answer: How can I have dynamic properties in go on the google app engine datastore.
Original answer follows.
Note that if the types are known at compile time, best / most efficient is to create the types and compile them, so everything will be "static". You may create the types manually, or you may use go generate to automate the process.
Also note that you may not necessarily need struct types to model dynamic objects, many times maps may be sufficient.
If types are not known at compile time, and struct types are a must, read on.
Yes, it's possible to create "dynamic" struct types at runtime using Go's reflection, specifically with the reflect.StructOf() function.
Let's see a simple example, creating a struct type at runtime that has a Name string and an Age int field:
t := reflect.StructOf([]reflect.StructField{
{
Name: "Name",
Type: reflect.TypeOf(""), // string
},
{
Name: "Age",
Type: reflect.TypeOf(0), // int
},
})
fmt.Println(t)
v := reflect.New(t)
fmt.Printf("%+v\n", v)
v.Elem().FieldByName("Name").Set(reflect.ValueOf("Bob"))
v.Elem().FieldByName("Age").Set(reflect.ValueOf(12))
fmt.Printf("%+v\n", v)
This outputs (try it on the Go Playground):
struct { Name string; Age int }
&{Name: Age:0}
&{Name:Bob Age:12}
If you want to define validation rules, you may use a 3rd party lib for that, for example github.com/go-validator/validator. This package uses struct tags to specify validation rules, struct tags which you may also specify using reflection.
For example, if you want to specify that the Name must be at least 3 characters and 40 at most, and it may only contain letters of the English alphabet, and valid range for Age is 6..100 (both inclusive), this is how it would look like:
t := reflect.StructOf([]reflect.StructField{
{
Name: "Name",
Type: reflect.TypeOf(""), // string
Tag: reflect.StructTag(`validate:"min=3,max=40,regexp=^[a-zA-Z]*$"`),
},
{
Name: "Age",
Type: reflect.TypeOf(0), // int
Tag: reflect.StructTag(`validate:"min=6,max=100"`),
},
})
Printing this type would output (wrapped by me) (try it on the Go Playground):
struct { Name string "validate:\"min=3,max=40,regexp=^[a-zA-Z]*$\"";
Age int "validate:\"min=6,max=100\"" }
Once you create an instance of this struct, you can validate it using the validator.Validate() function, e.g.:
v := reflect.New(t)
v.Elem().FieldByName("Name").Set(reflect.ValueOf("Bob"))
v.Elem().FieldByName("Age").Set(reflect.ValueOf(12))
if errs := validator.Validate(v.Elem().Interface()); errs != nil {
// values not valid, deal with errors here
}
I have a list of objects (olievere/Elastic SearchResult.Hits to be exact). Each of these has a json.RawMessage object and I'm looking to create an abstractable method that takes in an interface slice of any struct, Unmarshal's each individual hits' json.RawMessage into said struct, and appends it to the passed in []interface.
This func is not supposed to have any logic or insight into the desired business layer struct, and the DB Call is interfaced pretty heavily, and as such has no visibility into the Elastic package mentioned above. Example of what I was attempting to do...
foo.go
import (bar, package)
type TestStruct struct {
Slice []*package.Struct // package.Struct has a value of Source which is a
// json.RawMessage
}
func GetData() bar.Test {
return &TestStruct{*package.GetData()}
}
func (result TestStruct) UnmarshalStruct(v []interface{}) {
for _, singleStruct := range result.Slice {
append(json.Unmarshal(singleStruct, &v))
}
Second File
bar.go
type Handler interface {
GetData() Test
}
type Test interface {
UnmarshalStruct
}
type OtherType struct {
foo string
bar string
}
func RetrieveData() []OtherType {
handler := New(Handler)
test := handler.GetData()
var typeSlice []OtherType
test.UnmarshalStruct(&typeSlice)
}
I'm looking to hand something of type []OtherType, or any other new struct I decide to create, to UnmarshalStruct, and have it return to me that same struct, just full of data
As an example, I have two different types of data I'll be searching for from Elastic. I will be receiving a list of ONE of the following two objects.
{ 'foo': '',
'id':
}
And in a different index
{ 'bar': '',
'baz': '',
'eee': ''
}
Each of these will naturally require two different structs.
However, I desire a single method to be able to decode either of these lists. I'll be given below, and using the same function I want to be able to convert this to a bar struct, and the other type to a foo struct.
{ 'source': [
{ 'bar': '',
'baz': '',
'eee': ''
},
{ 'bar': '',
'baz': '',
'eee': ''
},
{ 'bar': '',
'baz': '',
'eee': ''
}
]
}
There's really no way to do what you want without reflection. I would personally structure this differently, so that you unmarshal into more generic types, like a map[string]string, or as #ThunderCat shows, get rid of the intermediary state and unamrshal directly into the correct types. But it can be done...
(I moved the json.RawMessage directly into TestStruct to get rid of one level of indirection and make the example more clear)
type TestStruct struct {
Slice []json.RawMessage
}
func (t TestStruct) UnmarshalStruct(v interface{}) error {
// get the a Value for the underlying slice
slice := reflect.ValueOf(v).Elem()
// make sure we have adequate capacity
slice.Set(reflect.MakeSlice(slice.Type(), len(t.Slice), len(t.Slice)))
for i, val := range t.Slice {
err := json.Unmarshal(val, slice.Index(i).Addr().Interface())
if err != nil {
return err
}
}
return nil
}
You can then call it like so
var others []OtherType
err := ts.UnmarshalStruct(&others)
if err != nil {
log.Fatal(err)
}
http://play.golang.org/p/dgly2OOXDG
If I understand correctly, you want to unmarshal data to slices of two types:
type A struct {
Foo string `json:"foo"`
ID string `json:"id"`
}
type B struct {
Bar string `json:"bar"`
Baz string `json:"baz"`
Eee string `json:"eee"`
}
from a SearchHit Source.
The JSON package can do most of the work for you:
func executeQuery(q Query, v interface{}) error {
// Get a SearchHit. I am making this up.
// I have no idea how the package works.
searchHit, err := getHit(q)
if err != nil {
return err
}
// This is the important part. Convert the raw message to
// a slice of bytes and decode to the caller's slice.
return json.Unmarshal([]byte(*searchHit.Source), v)
}
You can call this function to decode to a slice of the types or a slice of pointers to the types.
// Slice of type
var s1 []TypeA
if err := executeQuery(q1, &s1); err != nil {
// handle error
}
// Slice of pointer to type
var s2 []*TypeB
if err := error(q2, &s2); err != nil {
// handle error
}
I know that this is not a direct answer to the question, but this is how this scenario is typically handled.
I don't believe this is easy to do. In the Raw Message Example in the godocs they use a value within the json, "Space" in their example, to determine which struct type to unmarshal into.
For this to work, the function would have to have some way of getting every struct that has been defined ever for the program, and then it would have to examine each json object and compare it to each struct using reflection to figure out which type it most likely is. And what if there are multiple structs that "could be it"? Then conflict resolution complicates things.
In short, I don't think you can do this.
In Go, http form data (e.g. from a POST or PUT request) can be accessed as a map of the form map[string][]string. I'm having a hard time converting this to structs in a generalizable way.
For example, I want to load a map like:
m := map[string][]string {
"Age": []string{"20"},
"Name": []string{"John Smith"},
}
Into a model like:
type Person struct {
Age int
Name string
}
So I'm trying to write a function with the signature LoadModel(obj interface{}, m map[string][]string) []error that will load the form data into an interface{} that I can type cast back to a Person. Using reflection so that I can use it on any struct type with any fields, not just a Person, and so that I can convert the string from the http data to an int, boolean, etc as necessary.
Using the answer to this question in golang, using reflect, how do you set the value of a struct field? I can set the value of a person using reflect, e.g.:
p := Person{25, "John"}
reflect.ValueOf(&p).Elem().Field(1).SetString("Dave")
But then I'd have to copy the load function for every type of struct I have. When I try it for an interface{} it doesn't work.
pi := (interface{})(p)
reflect.ValueOf(&pi).Elem().Field(1).SetString("Dave")
// panic: reflect: call of reflect.Value.Field on interface Value
How can I do this in the general case? Or even better, is there a more idiomatic Go way to accomplish what I'm trying to do?
You need to make switches for the general case, and load the different field types accordingly. This is basic part.
It gets harder when you have slices in the struct (then you have to load them up to the number of elements in the form field), or you have nested structs.
I have written a package that does this. Please see:
http://www.gorillatoolkit.org/pkg/schema
For fun, I tried it out. Note that I cheated a little bit (see comments), but you should get the picture. There is usually a cost to use reflection vs statically typed assignments (like nemo's answer), so be sure to weigh that in your decision (I haven't benchmarked it though).
Also, obvious disclaimer, I haven't tested all edge cases, etc, etc. Don't just copy paste this in production code :)
So here goes:
package main
import (
"fmt"
"reflect"
"strconv"
)
type Person struct {
Age int
Name string
Salary float64
}
// I cheated a little bit, made the map's value a string instead of a slice.
// Could've used just the index 0 instead, or fill an array of structs (obj).
// Either way, this shows the reflection steps.
//
// Note: no error returned from this example, I just log to stdout. Could definitely
// return an array of errors, and should catch a panic since this is possible
// with the reflect package.
func LoadModel(obj interface{}, m map[string]string) {
defer func() {
if e := recover(); e != nil {
fmt.Printf("Panic! %v\n", e)
}
}()
val := reflect.ValueOf(obj)
if val.Kind() == reflect.Ptr {
val = val.Elem()
}
// Loop over map, try to match the key to a field
for k, v := range m {
if f := val.FieldByName(k); f.IsValid() {
// Is it assignable?
if f.CanSet() {
// Assign the map's value to this field, converting to the right data type.
switch f.Type().Kind() {
// Only a few kinds, just to show the basic idea...
case reflect.Int:
if i, e := strconv.ParseInt(v, 0, 0); e == nil {
f.SetInt(i)
} else {
fmt.Printf("Could not set int value of %s: %s\n", k, e)
}
case reflect.Float64:
if fl, e := strconv.ParseFloat(v, 0); e == nil {
f.SetFloat(fl)
} else {
fmt.Printf("Could not set float64 value of %s: %s\n", k, e)
}
case reflect.String:
f.SetString(v)
default:
fmt.Printf("Unsupported format %v for field %s\n", f.Type().Kind(), k)
}
} else {
fmt.Printf("Key '%s' cannot be set\n", k)
}
} else {
// Key does not map to a field in obj
fmt.Printf("Key '%s' does not have a corresponding field in obj %+v\n", k, obj)
}
}
}
func main() {
m := map[string]string{
"Age": "36",
"Name": "Johnny",
"Salary": "1400.33",
"Ignored": "True",
}
p := new(Person)
LoadModel(p, m)
fmt.Printf("After LoadModel: Person=%+v\n", p)
}
I'd propose to use a specific interface instead of interface{} in your LoadModel
which your type has to implement in order to be loaded.
For example:
type Loadable interface{
LoadValue(name string, value []string)
}
func LoadModel(loadable Loadable, data map[string][]string) {
for key, value := range data {
loadable.LoadValue(key, value)
}
}
And your Person implements Loadable by implementing LoadModel like this:
type Person struct {
Age int
Name string
}
func (p *Person) LoadValue(name string, value []string) {
switch name {
case "Age":
p.Age, err = strconv.Atoi(value[0])
// etc.
}
}
This is the way, the encoding/binary package or the encoding/json package work, for example.