I'm following this tutorial and got to the part on serializing/marshaling Go structs into a protocol buffer. My struct has a map and I can't find any documentation on how to handle marshaling a map.
In the following I want to serialize Fields map[string]string:
Go struct:
type Note struct {
ID NoteID
Fields map[string]string
}
protobuf schema:
package internal;
message Note {
optional int64 ID = 1;
optional map<string, string> Fields = 2;
}
Go marshal:
func MarshalNote(n *remember.Note) ([]byte, error) {
return proto.Marshal(&Note{
ID: proto.Int64(int64(n.ID))
Fields: proto.???
})
}
I have no idea what to do for the last line and anything I search for talks about mapping a field to a protobuf scheme, and not about mapping a map to a protobuf scheme.
protobuf is a well defined serialization format and one of the benefits of use it is that generates all the data structures for you(on your favorite language) just using the protobuf schema
Related
I have a yaml file of a certain configuration which is read by the go program to build a struct object.
The struct itself looks like this
type YamlConfig struct {
Attributes map[string]struct {
Label string `yaml:"label"`
Type string `yaml:"type"`
Presence bool `yaml:"presence"`
Uniqueness bool `yaml:"uniqueness"`
Strip bool `yaml:"strip"`
Values []string `yaml:"values"`
Default string `yaml:"default"`
Multi bool `yaml:"multi"`
Searchable bool `yaml:"searchable"`
Pattern struct {
Value string `yaml:"value"`
Message string `yaml:"message"`
} `yaml:"pattern"`
Length struct {
Min int `yaml:"min"`
Max int `yaml:"max"`
} `yaml:"length"`
} `yaml:"attributes"`
}
I have that map of Attributes that can be anything from "name" to "whatever", that should represent and db table columns with their types.
My question is - can I somehow take that object, which is quite dynamic and may not include the data for all the attributes' properties and convert it somehow into a usable ORM model with Gorm or something?
Should I always define a model struct or is it possible to build structs dynamically?
I think you are already using yaml.Unmarshal() for parsing your yaml config, right?
When you unmarshall yaml into a struct, you can use empty interface instead of complete yaml struct
var config map[string]interface{}
yaml.Unrmarshal(configFile, &config)
fmt.Println(config["label"])
I need to marshal into this JSON format:
{"messageProtocolHandshake":[{"handshakeType":"announceMax"},{"version":[{"major":1},{"minor":0}]}]}
Problem is matching the handshakeType. My struct is
type MessageProtocolHandshake struct {
HandshakeType HandshakeType `json:"handshakeType"`
Version []Version `json:"version"`
}
type HandshakeType struct {
HandshakeType string
}
Marshaling can be done using slice of interface:
func (h MessageProtocolHandshake) MarshalJSON() ([]byte, error) {
res := make([]interface{}, 3)
res[0] = struct {
HandshakeType string `json:"handshakeType"`
}{h.HandshakeType.HandshakeType}
res[1] = struct {
Version []Version `json:"version"`
}{h.Version}
return json.Marshal(res)
}
Using a simple marshaler/unmarshaler takes away the surrounding curly brackets from the handshakeType, so that doesn't work:
{"messageProtocolHandshake":[{"handshakeType":"announceMax","version":[{"major":1,"minor":0}],"formats":[{"format":"JSON-UTF8"}]}]}
Seems as if Go applies some heuristic in that case on the retuned byte array (undocumented?).
Is there a more elegant way of omitting the structs outer field name?
--
UPDATE To summarise the answers: key is to think about different structs for marshalling and unmarshalling if nothing else works, potentially a using a 3rd presentation for working internally with the data.
When custom (Un)Marshalers come into play remember that promoted fields inherit their methods and hence influence parent structs.
The JSON that you specified has a different model from that of your struct.
There are a few approaches to aligning these: Change the specification of the JSON data to match your structs, change the structs to match the specification of the JSON, or create a new struct that is only used for marshaling.
I omit the last example, because it's very similar to the second method.
Changing the specification of the JSON
The following model stays the same:
type MessageProtocolHandshake struct {
HandshakeType HandshakeType `json:"handshakeType"`
Version []Version `json:"version"`
}
type HandshakeType struct {
HandshakeType string
}
The JSON for this would be:
{"handshakeType":{"HandshakeType":""},"version":[]}
You did not specify the Version type so I don't know how one would change the JSON for that.
Changing the structs
The following JSON stays the same:
{"messageProtocolHandshake":[{"handshakeType":"announceMax"},{"version":[{"major":1},{"minor":0}]}]}
The structs for this would be:
type Model struct {
MessageProtocolHandshake []interface{} `json:"messageProtocolHandshake"`
}
type HandshakeType struct {
HandshakeType string `json:"handshakeType"`
}
type Versions struct {
Version []Version `json:"version"`
}
type Version struct {
Major *int `json:"major,omitempty"`
Minor *int `json:"minor,omitempty"`
}
Unmarshaling would not be trivial.
https://play.golang.org/p/89WUhcMFM0B
As is obvious from the results, the models you are using are not good. If there's a way to change all of this, I would recommend starting from scratch, using the data that is necessary and creating the JSON specification from the structs.
I recommend reading up on JSON: https://www.json.org/json-en.html
Also, I recommend this introduction to Go and JSON: https://blog.golang.org/json
We have tool which need to read YAML file with specific structure. When we got the YAML file we need to know if
Check if the YAML file is valid according to some guideline - semantic check
Where is the syntax error if any
For example this is example of the validation that we need to address
_version: {required: true}
id: {required: true, pattern: '/^[A-Za_\-\.]+$/'}
release-version: {required: true}
type:
builds:
type:seq
sequence:
-type:map
mapping:
name:{required: true, unique: true, pattern: '/^[A-Za-z0-3_\-\.]+$/'}
params:
type: map
mapping: { =: {type: any} }
Mapping is a key value object
seq can have multiple builds
type any is and key value
We use this open source to parse the yaml https://github.com/go-yaml/yaml
One idea (which is good) is to convert to json like following to do it by converting the file to json and validate it which have library to support it, any example in my context will be very helpful https://github.com/xeipuuv/gojsonschema
But not sure how I handle
Type map
Type seq
Here is what you could try.
Model a struct after the shape of the expected yaml data:
type Config struct {
Version struct {
Required bool
}
ID struct {
Required bool
Pattern string
}
ReleaseVersion struct {
Required bool
}
Type interface{}
Builds struct {
Type []interface{} `yaml:"type"`
Sequence struct {
Type string
}
Mapping struct {
Name map[string]interface{}
Params struct {
Type string `yaml:"type"`
Mapping struct {
To map[string]string `yaml:"="`
}
}
} `yaml:"mapping"`
}
}
The yaml flag yaml:"somefield" is added to label the field name of the yaml the data we're interested in.
Also many fields with unknown/undetermined type can be declared as empty interface (interface{}) or if you want to "enforce" that the underlying form is a key-value object you can either declare it as a map[string]interface{} or another struct.
We then unmarshal the yaml data into the struct:
cfg := Config{}
err := yaml.Unmarshal([]byte(data), &cfg)
if err != nil {
log.Fatalf("error: %v", err)
}
Since we have modeled fields as either anonymous structs or maps, we can test if a particular field has a "key-value" value by checking its equality to nil.
// Mapping is a key value object
if (Mapping != nil) {
// Mapping is a key-value object, since it's not nil.
}
// type any is and key value
// Mapping.To is declared with map[string]string type
// so if it's not nil we can say there's a map there.
if (Mapping.To != nil) {
// Mapping.To is a map
}
In marshaling/unmarshaling, maps and structs are pretty interchangeable. The benefit of a struct is you can predefine the field's names ahead of time while unmarshaling to a map it won't be clear to you what the keys are.
You can make go-yaml work with jsonschema. See this issue: https://github.com/santhosh-tekuri/jsonschema/issues/5
In short:
Create a custom yaml parser that produces compatible output types, as per this issue.
Parse the yaml into an interface{} using that custom parser
Validate with jsonschema.ValidateInterface.
(once yaml.v3 has been released, the custom unmarshaller should be able to be replaced with a configuration option)
I was originally using the accepted answer's approach of parsing into a struct and then writing code to manually validate that the struct met my spec. This quickly got very ugly - the above approach allows for a clean separate spec and reliable validation of it.
I have the following response struct that I want to use as a base wrapper for responding to API calls users send to me.
type Response struct {
Data ??? `json:"data,omitempty"`
Time int64 `json:"time,omitempty"`
Message string `json:"message,omitempty"`
}
The type of the Data field is varying and could be a map[string]*CustomStruct1 map[string*CustomStruct2 or an []CustomStruct3.
What is the best way to attack this kind of problem?
One option is to simply treat "Data" as the interface{} (any) type, instead of using your custom structs, and handle the resulting values based on inspection of what actually got unmarshaled. Of course, once you've inspected the data to determine what type it should be you could convert it into the appropriate strong type after the fact.
type Response struct {
Data interface{} `json:"data,omitempty"`
Time int64 `json:"time,omitempty"`
Message string `json:"message,omitempty"`
}
Another option is to embed the "Response" struct into specialized structs that look for your custom types and unmarshal into the appropriate one, assuming you know which one you've got ahead of time:
type BaseResponse struct {
Time int64 `json:"time,omitempty"`
Message string `json:"message,omitempty"`
}
type Response1 struct {
BaseResponse
Data map[string]*CustomStruct1 `json:"data"`
}
type Response2 struct {
BaseResponse
Data map[string]*CustomStruct2 `json:"data"`
}
// etc...
Ultimately, the unmarshaler cannot pick a varying type based on the document that gets unmarshaled, it only deserializes JSON values into structures either defined explicitly by you or into generic ones.
You could try to use reflection, but it wouldn't be very idiomatic.
I have a collection of documents that contain an array of a custom interface type that I have. Example below. What do I need to do to Unmarshal the bson from mongo so I can eventually return a JSON response?
type Document struct {
Props here....
NestedDocuments customInterface
}
What do I need to do to map the nested interfaces to the right structs?
I think it is evident that an interface cannot be instantiated, therefore the bson runtime does not know which struct has to be used to Unmarshal that object. Additionally, your customInterface type should be exported (i.e. with capital "C") otherwise it won't be accessible from the bson runtime.
I suspect that using an interface implies that the NestedDocuments array may contain different types, all implementing customInterface.
If that's the case, I am afraid that you will have to do some changes:
Firstly, NestedDocument need to be a struct holding your document plus some information to help the decoder understand what's the underpinning type. Something like:
type Document struct {
Props here....
Nested []NestedDocument
}
type NestedDocument struct {
Kind string
Payload bson.Raw
}
// Document provides
func (d NestedDocument) Document() (CustomInterface, error) {
switch d.Kind {
case "TypeA":
// Here I am safely assuming that TypeA implements CustomInterface
result := &TypeA{}
err := d.Payload.Unmarshal(result)
if err != nil {
return nil, err
}
return result, nil
// ... other cases and default
}
}
In this way, the bson runtime will decode the whole Document but leave the payload as a []byte.
Once you have decoded the main Document, you can use the NestedDocument.Document() function to get a concrete representation of your struct.
One last thing; when you persist your Document, make sure that Payload.Kind is set to 3, which represents an embedded document. See the BSON specifications for more details on this.
Hope it is all clear and good luck with your project.