I have a yaml file of a certain configuration which is read by the go program to build a struct object.
The struct itself looks like this
type YamlConfig struct {
Attributes map[string]struct {
Label string `yaml:"label"`
Type string `yaml:"type"`
Presence bool `yaml:"presence"`
Uniqueness bool `yaml:"uniqueness"`
Strip bool `yaml:"strip"`
Values []string `yaml:"values"`
Default string `yaml:"default"`
Multi bool `yaml:"multi"`
Searchable bool `yaml:"searchable"`
Pattern struct {
Value string `yaml:"value"`
Message string `yaml:"message"`
} `yaml:"pattern"`
Length struct {
Min int `yaml:"min"`
Max int `yaml:"max"`
} `yaml:"length"`
} `yaml:"attributes"`
}
I have that map of Attributes that can be anything from "name" to "whatever", that should represent and db table columns with their types.
My question is - can I somehow take that object, which is quite dynamic and may not include the data for all the attributes' properties and convert it somehow into a usable ORM model with Gorm or something?
Should I always define a model struct or is it possible to build structs dynamically?
I think you are already using yaml.Unmarshal() for parsing your yaml config, right?
When you unmarshall yaml into a struct, you can use empty interface instead of complete yaml struct
var config map[string]interface{}
yaml.Unrmarshal(configFile, &config)
fmt.Println(config["label"])
Related
I need to marshal into this JSON format:
{"messageProtocolHandshake":[{"handshakeType":"announceMax"},{"version":[{"major":1},{"minor":0}]}]}
Problem is matching the handshakeType. My struct is
type MessageProtocolHandshake struct {
HandshakeType HandshakeType `json:"handshakeType"`
Version []Version `json:"version"`
}
type HandshakeType struct {
HandshakeType string
}
Marshaling can be done using slice of interface:
func (h MessageProtocolHandshake) MarshalJSON() ([]byte, error) {
res := make([]interface{}, 3)
res[0] = struct {
HandshakeType string `json:"handshakeType"`
}{h.HandshakeType.HandshakeType}
res[1] = struct {
Version []Version `json:"version"`
}{h.Version}
return json.Marshal(res)
}
Using a simple marshaler/unmarshaler takes away the surrounding curly brackets from the handshakeType, so that doesn't work:
{"messageProtocolHandshake":[{"handshakeType":"announceMax","version":[{"major":1,"minor":0}],"formats":[{"format":"JSON-UTF8"}]}]}
Seems as if Go applies some heuristic in that case on the retuned byte array (undocumented?).
Is there a more elegant way of omitting the structs outer field name?
--
UPDATE To summarise the answers: key is to think about different structs for marshalling and unmarshalling if nothing else works, potentially a using a 3rd presentation for working internally with the data.
When custom (Un)Marshalers come into play remember that promoted fields inherit their methods and hence influence parent structs.
The JSON that you specified has a different model from that of your struct.
There are a few approaches to aligning these: Change the specification of the JSON data to match your structs, change the structs to match the specification of the JSON, or create a new struct that is only used for marshaling.
I omit the last example, because it's very similar to the second method.
Changing the specification of the JSON
The following model stays the same:
type MessageProtocolHandshake struct {
HandshakeType HandshakeType `json:"handshakeType"`
Version []Version `json:"version"`
}
type HandshakeType struct {
HandshakeType string
}
The JSON for this would be:
{"handshakeType":{"HandshakeType":""},"version":[]}
You did not specify the Version type so I don't know how one would change the JSON for that.
Changing the structs
The following JSON stays the same:
{"messageProtocolHandshake":[{"handshakeType":"announceMax"},{"version":[{"major":1},{"minor":0}]}]}
The structs for this would be:
type Model struct {
MessageProtocolHandshake []interface{} `json:"messageProtocolHandshake"`
}
type HandshakeType struct {
HandshakeType string `json:"handshakeType"`
}
type Versions struct {
Version []Version `json:"version"`
}
type Version struct {
Major *int `json:"major,omitempty"`
Minor *int `json:"minor,omitempty"`
}
Unmarshaling would not be trivial.
https://play.golang.org/p/89WUhcMFM0B
As is obvious from the results, the models you are using are not good. If there's a way to change all of this, I would recommend starting from scratch, using the data that is necessary and creating the JSON specification from the structs.
I recommend reading up on JSON: https://www.json.org/json-en.html
Also, I recommend this introduction to Go and JSON: https://blog.golang.org/json
I'm trying to parse some xml documents in Go. I need to define a few structs for this purpose, and my struct tags depend on a certain condition.
Imagine the following code (even though I know it won't work)
if someCondition {
type MyType struct {
// some common fields
Date []string `xml:"value"`
}
} else {
type MyType struct {
// some common fields
Date []string `xml:"anotherValue"`
}
}
var t MyType
// do the unmarshalling ...
The problem is that these two structs have lots of fields in common. The only difference is in one of the fields and I want to prevent duplication. How can I solve this problem?
You use different types to unmarshal. Basically, you write the unmarshaling code twice and either run the first version or the second. There is no dynamic solution to this.
The simplest is probably to handle all possible fields and do some post-processing.
For example:
type MyType struct {
DateField1 []string `xml:"value"`
DateField2 []string `xml:"anotherValue"`
}
// After parsing, you have two options:
// Option 1: re-assign one field onto another:
if !someCondition {
parsed.DateField1 = parsed.DateField2
parsed.DateField2 = nil
}
// Option 2: use the above as an intermediate struct, the final being:
type MyFinalType struct {
Date []string `xml:"value"`
}
if someCondition {
final.Date = parsed.DateField1
} else {
final.Date = parsed.DateField2
}
Note: if the messages are sufficiently different, you probably want completely different types for parsing. The post-processing can generate the final struct from either.
As already indicated, you must duplicate the field. The question is where the duplication should exist.
If it's just a single field of many, one option is to use embedding, and field shadowing:
type MyType struct {
Date []string `xml:"value"`
// many other fields
}
Then when Date uses the other field name:
type MyOtherType struct {
MyType // Embed the original type for all other fields
Date []string `xml:"anotherValue"`
}
Then after unmarshaling of MyOtherType, it's easy to move the Date value into the original struct:
type data MyOtherType
err := json.Unmarshal(..., &data)
data.MyType.Date = data.Date
return data.MyType // will of MyType, and fully populated
Note that this only works for unmarshaling. If you need to also marshal this data, a similar trick can be used, but the mechanics around it must be essentially reversed.
We have tool which need to read YAML file with specific structure. When we got the YAML file we need to know if
Check if the YAML file is valid according to some guideline - semantic check
Where is the syntax error if any
For example this is example of the validation that we need to address
_version: {required: true}
id: {required: true, pattern: '/^[A-Za_\-\.]+$/'}
release-version: {required: true}
type:
builds:
type:seq
sequence:
-type:map
mapping:
name:{required: true, unique: true, pattern: '/^[A-Za-z0-3_\-\.]+$/'}
params:
type: map
mapping: { =: {type: any} }
Mapping is a key value object
seq can have multiple builds
type any is and key value
We use this open source to parse the yaml https://github.com/go-yaml/yaml
One idea (which is good) is to convert to json like following to do it by converting the file to json and validate it which have library to support it, any example in my context will be very helpful https://github.com/xeipuuv/gojsonschema
But not sure how I handle
Type map
Type seq
Here is what you could try.
Model a struct after the shape of the expected yaml data:
type Config struct {
Version struct {
Required bool
}
ID struct {
Required bool
Pattern string
}
ReleaseVersion struct {
Required bool
}
Type interface{}
Builds struct {
Type []interface{} `yaml:"type"`
Sequence struct {
Type string
}
Mapping struct {
Name map[string]interface{}
Params struct {
Type string `yaml:"type"`
Mapping struct {
To map[string]string `yaml:"="`
}
}
} `yaml:"mapping"`
}
}
The yaml flag yaml:"somefield" is added to label the field name of the yaml the data we're interested in.
Also many fields with unknown/undetermined type can be declared as empty interface (interface{}) or if you want to "enforce" that the underlying form is a key-value object you can either declare it as a map[string]interface{} or another struct.
We then unmarshal the yaml data into the struct:
cfg := Config{}
err := yaml.Unmarshal([]byte(data), &cfg)
if err != nil {
log.Fatalf("error: %v", err)
}
Since we have modeled fields as either anonymous structs or maps, we can test if a particular field has a "key-value" value by checking its equality to nil.
// Mapping is a key value object
if (Mapping != nil) {
// Mapping is a key-value object, since it's not nil.
}
// type any is and key value
// Mapping.To is declared with map[string]string type
// so if it's not nil we can say there's a map there.
if (Mapping.To != nil) {
// Mapping.To is a map
}
In marshaling/unmarshaling, maps and structs are pretty interchangeable. The benefit of a struct is you can predefine the field's names ahead of time while unmarshaling to a map it won't be clear to you what the keys are.
You can make go-yaml work with jsonschema. See this issue: https://github.com/santhosh-tekuri/jsonschema/issues/5
In short:
Create a custom yaml parser that produces compatible output types, as per this issue.
Parse the yaml into an interface{} using that custom parser
Validate with jsonschema.ValidateInterface.
(once yaml.v3 has been released, the custom unmarshaller should be able to be replaced with a configuration option)
I was originally using the accepted answer's approach of parsing into a struct and then writing code to manually validate that the struct met my spec. This quickly got very ugly - the above approach allows for a clean separate spec and reliable validation of it.
I have a nested struct
type Posts struct {
Id int
CreatedAt time.Time
Data struct {
Title string
UserName string
}
}
I want to create a Data object but var innerData Posts.Data doesn't seem to work. I don't want to create separate Data struct because I'm planning to avoid name collusions by having multiple structs which will have different inner structs named Data.
You can't. Posts.Data isn't the name of a type. The only thing you could do is var innerData struct{ ... }, which I'm sure you don't want to because then you would have repetition and need to make changes in two places. Otherwise, you have to give it a name. That name doesn't have to be Data, it can be PostData or something to make it unique.
I don't understand the significance of struct tags. I have been looking them, and noticed that they can be used with reflect package. But I don't know any practical uses of them.
type TagType struct { // tags
field1 bool “An important answer”
field2 string “The name of the thing”
field3 int “How much there are”
}
The use of tags strongly depends on how your struct is used.
A typical use is to add specifications or constraints for persistence or serialisation.
For example, when using the JSON parser/encoder, tags are used to specify how the struct will be read from JSON or written in JSON, when the default encoding scheme (i.e. the name of the field) isn't to be used.
Here are a few examples from the json package documentation :
// Field is ignored by this package.
Field int `json:"-"`
// Field appears in JSON as key "myName".
Field int `json:"myName"`
// Field appears in JSON as key "myName" and
// the field is omitted from the object if its value is empty,
// as defined above.
Field int `json:"myName,omitempty"`
// Field appears in JSON as key "Field" (the default), but
// the field is skipped if empty.
// Note the leading comma.
Field int `json:",omitempty"`
Example of use is json encoding/decoding in encoding/json:
type TagType struct {
field1 bool `json:"fieldName"`
field2 string `json:",omitempty"`
}
More details in documentation: encoding/json
You can also use XML struct tags as shown below
type SearchResult struct {
Title string `xml:"title,attr"`
Author string `xml:"author,attr"`
Year string `xml:"hyr,attr"`
ID string `xml:"owi,attr"`
}