Unmarshal a YAML to a struct with unexpected fields in Go - go

I encountered an issue while trying to unmarshal a struct with an unexported field using github.com/go-yaml/yaml. The struct looks like:
type Example struct {
ExportedField string `yaml:"exported-field"`
OneMoreExported string `yaml:"one-more-exported"`
unexportedField map[string]*AnotherExample `yaml:"unexported-field"`
}
type AnotherExample struct {
Name string `yaml:"name"`
}
And I'd like to unmarshal such YAML as
exported-field: lorem ipsum
one-more-exported: dolor set
unexported-field:
something:
name: anything
What I tried is a custom unmarshaler:
func (example *Example) UnmarshalYAML(unmarshal func(interface{}) error) error {
type Alias Example
tmp := struct {
UnexportedField map[string]*AnotherExample `yaml:"unexported-field"`
*Alias
}{
Alias: (*Alias)(example),
}
if err := unmarshal(&tmp); err != nil {
return err
}
if tmp.UnexportedField != nil {
example.unexportedField = tmp.UnexportedField
}
example.CopyJobNonNil(Example(*tmp.Alias)) // Copies all the non-nil fields from the passed Example instance
return nil
}
tmp after calling unmarshal() doesn't contain any fields but unexportedField — other fields seems to be omitted.
The reproduced issue on Go Playground (although it isn't working due to the dependencies): https://play.golang.org/p/XZg7tEPGXna

Because most Go unmarshalling packages (including the encoding/* packages) use the reflect package to get at struct fields, and reflect can't access unexported struct fields, the unmarshaler can't parse into unexported fields.
That said, there is still a way to do it. You can unmarshall the YAML into an an unexported type with public fields, which can then be embedded into an exported type. Getters and setters in the same package can just use the embedded struct.
For example:
// Define the types...
type innerData struct {
ExportedField string
unexportedField string
}
type Data struct {
innerData
}
// and then...
d := Data{}
DoSomeUnmarshalling(yamlbytes, &d.innerData)
// and then you can use getters/setters/whatever on `Data`
func (d *Data) GetUnexported() string {
return d.innerData.unexportedField;
}
(warning: completely untested)
See JSON and dealing with unexported fields for reference.

Related

XML unmarshalling in Golang

I want to unmarshall the below xml payload to struct
<linearPackagePublish>
<linearPackage>
<name>ABC</name>
<packagedServiceReference>
<availabilityWindowEnd>2329-12-31 23:59:59</availabilityWindowEnd>
<availabilityWindowStart>2007-11-14 11:40:00</availabilityWindowStart>
<packagedServiceId>1111111111</packagedServiceId>
</packagedServiceReference>
<partnerPackageId>XXXXXXX</partnerPackageId>
</linearPackage>
<partnerId>XXXXXX</partnerId>
<wantLinearPublishResult>true</wantLinearPublishResult>
</linearPackagePublish>
I want to add transactionId just before tag like this:
<linearPackagePublish>
<linearPackage>
<name>ABC</name>
<packagedServiceReference>
<availabilityWindowEnd>2329-12-31 23:59:59</availabilityWindowEnd>
<availabilityWindowStart>2007-11-14 11:40:00</availabilityWindowStart>
<packagedServiceId>1111111111</packagedServiceId>
</packagedServiceReference>
<partnerPackageId>XXXXXXX</partnerPackageId>
</linearPackage>
<partnerId>XXXXXX</partnerId>
<transactionId>111111111111</transactionId>
<wantLinearPublishResult>true</wantLinearPublishResult>
</linearPackagePublish>
For this purpose I was trying to unmarshall the above xml to below struct:
type linearPackagePublish struct {
LinearPackage string `xml:"linearPackage"`
MsoPartnerID string `xml:"partnerId"`
TransactionID string `xml:"transactionId,omitempty"`
WantLinearPublishResult bool `xml:"wantLinearPublishResult,omitempty"`
}
But the thing is I don’t want to unmarshall the linearPackage element that is why I put LinearPackage as string in struct
I can also do it via regex but it will be more error prone if something change in the xml.
Is there any way that we can make some nested xml element as a string??
Edited to provide more detail. Including a working demo on Go Playground
Rather than trying to use a string you can define a custom type, with its own marshal and unmarshal functions defined. That way you can have direct control over that element.
You can see the custom datatype holds an array of xml.Tokens. This array is populated during unmarshalling and then written to the encoder during marshalling.
type skipUnmarshal struct {
data []xml.Token
}
func (s skipUnmarshal) MarshalXML(e *xml.Encoder, start xml.StartElement) error {
for i := 0; i < len(s.data); i++ {
e.EncodeToken(s.data[i])
}
return nil
}
func (s *skipUnmarshal) UnmarshalXML(d *xml.Decoder, start xml.StartElement) error {
for {
t, err := d.Token()
if err != nil {
break // or alternatively return the error
}
s.data = append(s.data, t)
}
return nil
}

Returning a pointer to a struct from a function in Go

I have two public structs that contain different data, and a private intermediate struct containing either of the two public structs. I also have a function that unmarshalls the intermediate struct, determines which public struct it contains, and returns one of the two public structs.
The problem I'm facing is the return value of the last function. At it's simplest I thought I could return *struct{} but I keep getting a type mismatch in my IDE.
I apologize for posting more code than is probably necessary, but I'm trying to make it as close as possible to the code I'm working on.
package main
import (
"encoding/json"
"errors"
)
// These vars are some errors I'll use in the functions later on
var (
errInvalidBase64 = errors.New("invalid base64")
errInvalidStructType = errors.New("invalid struct type")
)
// Struct1 public struct
type Struct1 struct {
FName string `json:"first-name"`
LName string `json:"last-name"`
}
// Struct2 public struct
type Struct2 struct {
Date string `json:"date"`
Items []int `json:"items"`
}
// intermediateStruct private struct
// The Type field indicates which kind of struct Data contains (Struct1 or Struct2)
// The Data field contains either Struct1 or Struct2 which was previously marshalled into JSON
type intermediateStruct struct {
Type structType
Data []byte
}
// The following type and const are my understanding of an enum in Go
// structType is a private type for the type of struct intermediateStruct contains
type structType int
// These public constants are just to keep my hands out of providing values for the different struct types
const (
StructType1 structType = iota
StructType2
)
// unmarshalStruct1 unmarshalls JSON []byte into a new Struct1 and returns a pointer to that struct
func unmarshalStruct1(b []bytes) (*Struct1, error) {
newStruct1 := new(Struct1)
err := json.Unmarshal(b, newStruct1)
if err != nil {
return nil, errInvalidBase64
}
return newStruct1, nil
}
// unmarshalStruct2 unmarshalls JSON []byte into a new Struct2 and returns a pointer to that struct
func unmarshalStruct2(b []bytes) (*Struct2, error) {
newStruct2 := new(Struct2)
err := json.Unmarshal(b, newStruct2)
if err != nil {
return nil, errInvalidBase64
}
return newStruct2, nil
}
// receiveStruct accepts *intermediateStruct who's Data field contains either Struct1 or Struct2
// This function needs to return either *Struct1 or *Struct2 and an error
func receiveStruct(iStruct *intermediateStruct) (*struct{}, error) {
switch iStruct.Type {
case StructType1:
struct1, err := unmarshalStruct1(iStruct.Data)
if err != nil {
return nil, err
}
// The following line is where I'm getting the type mismatch
return struct1, nil
case StructType2:
struct2, err := unmarshalStruct2(iStruct.Data)
if err != nil {
return nil, err
}
// The following line is another type mismatch
return struct2, nil
default:
return nil, errInvalidStructType
}
}
I know there's a way to do what I'm trying to achieve - I just lack the experience/understanding to get there.
Thanks for any and all input!
Your unmarshallStruct function returns a pointer to type Struct1 or Struct2 (depending on which version of the function gets called). And therefore the variables struct1 and struct2 are pointers to types Struct1 and Struct2 respectively. Neither is a pointer to type struct (which is not a real Go type anyways I must add). A struct is a keyword which helps to declare types containing fields/attributes.
Depending on the use-cases you have in mind for the rest of your code, can instead try any of the below:
As mkopriva suggested, return a interface{} object, but you'd need to use type assertion to actually make sure of the object
Define an interface which both Struct1 and Struct2 implement, and return a pointer to this
Make separate functions which work with either Struct1 or Struct2. This is not necessarily as bad as it sounds as Go lets you pass functions in the same way you'd pass types (see example of the Less() function in sort package).

What is the equivalence of Type Name Handling with encoding/json in Go?

Short Description in Prose
I have a situation where I want to unmarshal JSON data into an array of structs (either Foo or Bar and many more) that all implement a common interface MyInterface. Also all of the eligble struct types that implement the interface have a common field which I named Discrimininator in the example below.
The Discriminator¹ allows to bi-uniquely find the correct struct type for each value of Discriminator.
The Problem and Error Message
But during unmarshling the code does not "know" which is the correct "target" type. The unmarshaling fails.
cannot unmarshal object into Go value of type main.MyInterface
MWE in
https://play.golang.org/p/Dw1hSgUezLH
package main
import (
"encoding/json"
"fmt"
)
type MyInterface interface {
// some other business logic methods go here!
GetDiscriminator() string // GetDiscriminator returns something like a key that is unique per struct type implementing the interface
}
type BaseStruct struct {
Discriminator string // will always be "Foo" for all Foos, will always be "Bar" for all Bars
}
type Foo struct {
BaseStruct
// actual fields of the struct don't matter. it's just important that they're different from Bar
FooField string
}
func (foo *Foo) GetDiscriminator() string {
return foo.Discriminator
}
type Bar struct {
BaseStruct
// actual fields of the struct don't matter. it's just important that they're different from Foo
BarField int
}
func (bar *Bar) GetDiscriminator() string {
return bar.Discriminator
}
// Foo and Bar both implement the interface.
// Foo and Bars are always distinguishible if we check the value of Discriminator
func main() {
list := []MyInterface{
&Bar{
BaseStruct: BaseStruct{Discriminator: "Bar"},
BarField: 42,
},
&Foo{
BaseStruct: BaseStruct{Discriminator: "Foo"},
FooField: "hello",
},
}
jsonBytes, _ := json.Marshal(list)
jsonString := string(jsonBytes)
fmt.Println(jsonString)
// [{"Discriminator":"Bar","BarField":42},{"Discriminator":"Foo","FooField":"hello"}]
var unmarshaledList []MyInterface
err := json.Unmarshal(jsonBytes, &unmarshaledList)
if err != nil {
// Unmarshaling failed: json: cannot unmarshal object into Go value of type main.MyInterface
fmt.Printf("Unmarshaling failed: %v", err)
}
}
In other languages
TypeNameHandling as known from .NET
In Newtonsoft, a popular .NET JSON Framework, this is solved by a something called "TypeNameHandling" or can be solved with a custom JsonConverter . The framework will add something like a magic "$type" key on root level to the serialized/marshaled JSON which is then used to determine the original type on deserialization/unmarshaling.
Polymorphism in ORMs
¹A similar situation occurs under the term "polymorphism" in ORMs when instances of multiple types having the same base are saved in the same table. One typically introduces a discriminator column, hence the name in above example.
You can implement a custom json.Unmarshaler. For that you'll need to use a named slice type instead of the unnamed []MyInterface.
Within the custom unmarshaler implementation you can unmarshal the JSON array into a slice where each element of the slice is a json.RawMessage representing the corresponding JSON object. After that you can iterate over the slice of raw messages. In the loop unmarshal from each raw message only the Discriminator field, then use the Discriminator field's value to determine what the correct type is into which the full raw message can be unmarshaled, finally unmarshal the full message and add the result to the receiver.
type MyInterfaceSlice []MyInterface
func (s *MyInterfaceSlice) UnmarshalJSON(data []byte) error {
array := []json.RawMessage{}
if err := json.Unmarshal(data, &array); err != nil {
return err
}
*s = make(MyInterfaceSlice, len(array))
for i := range array {
base := BaseStruct{}
data := []byte(array[i])
if err := json.Unmarshal(data, &base); err != nil {
return err
}
var elem MyInterface
switch base.Discriminator {
case "Foo":
elem = new(Foo)
case "Bar":
elem = new(Bar)
}
if elem == nil {
panic("whoops")
}
if err := json.Unmarshal(data, elem); err != nil {
return err
}
(*s)[i] = elem
}
return nil
}
https://play.golang.org/p/mXiZrF392aV

How to fill a struct that has lots of nested structs in Go

What is the best way to fill a struct that has lots of nested structs inside it?
I made a struct to generate a json schema file from it that looks like this:
type Schema struct {
Schema string `default:"http://json-schema.org/draft-04/schema#"`
Title string `default:"Test Schema"`
Type string `default:"object"`
AdditionalProperties bool `default:false`
Properties struct {
Core struct {
Type string
AdditionalProperties bool
Properties struct{}
}
Work struct {
Type string
AdditionalProperties bool
Properties struct{}
}
}
}
At first, I wanted to put the default data in tags and fill the struct from that, but reflect package doesn't look inside the nested structs.
Here is what I did using reflect:
t := reflect.TypeOf(Schema{})
for i := 0; i < t.NumField(); i++ {
field := t.Field(i)
tag := field.Tag.Get("default")
}
The best way to do this is with a constructor method. This will be more readable, and much faster, than using tags plus reflection. Something like:
func NewSchema() *Schema {
return &Schema{
Schema: "http://json-schema.org/draft-04/schema#",
...
}
}

Missing Xml Tag in Golang

I am new to golang I am trying to make a xml in which i am using nested tags
For that my code is
type MyXml struct {
XMLName xml.Name `xml:"myXml"`
Id int `xml:"id,attr"`
NewXml
}
type NewXml struct {
XMLName xml.Name `xml:"newXml"`
OneMoreXml
}
type OneMoreXml struct {
Msg interface{} `xml:"oneMore"`
}
type Child struct {
Param1 string `xml:"Param1"`
}
func main() {
baseXml := &Child{Param1: "Param1"}
retXml := GetXml(baseXml)
fmt.Println("my xml is", retXml)
}
func MarshallXml(reqString interface{}) (newXml string) {
xmlBody, err := xml.Marshal(reqString)
if err != nil {
fmt.Printf("error: %v\n", err)
}
newXml = string(xmlBody)
//fmt.Println(newXml)
return
}
func GetXml(baseXml interface{}) (finalXml string) {
startXml := new(MyXml)
startXml.Id = 1
startXml.Msg = baseXml
finalXml = MarshallXml(startXml)
return
}
but in my output xml Tag newXml is missing. I have tried it in various ways but some tag is always missing. I guess i am not understanding struct tag properly. So What i am doing wrong in above code and which basic concept of golang struct i am missing
I had a look at the xml package doc and they say that "an anonymous struct field is handled as if the fields of its value were part of the outer struct". In your case, all fields thus get serialized as if they were part of MyXml.
NewXml does not have any field (you just give it a name but there is nothing else) so nothing gets serialized for it.
If you add a new field to it, you can see that it is serialized.
type NewXml struct {
XMLName xml.Name `xml:"newXml"`
Test int
OneMoreXml
}
http://play.golang.org/p/vibSeQHTCr

Resources