Unmarshaling YAML into different struct based off YAML field - go

I'm trying to unmarshal the following YAML data into Go structures.
The data is the in the following format:
fetchers:
- type: "aws"
config:
omega: "lul"
- type: "kubernetes"
config:
foo: "bar"
Based of the type field, I want to determine wether to unmarshal the config field into awsConfig or kubernetesConfig struct.
My current code looks like this (using "gopkg.in/yaml.v2"):
type kubernetesConfig struct {
foo string `yaml:"foo"`
}
type awsConfig struct {
omega string `yaml:"omega"`
}
var c struct {
Fetchers []struct {
Type string `yaml:"type"`
Config interface{} `yaml:"config"`
} `yaml:"fetchers"`
}
err := yaml.Unmarshal(data, &c)
if err != nil {
log.Fatal(err)
}
for _, val := range c.Fetchers {
switch val.Type {
case "kubernetes":
conf := val.Config.(kubernetesConfig)
fmt.Println(conf.foo)
case "aws":
conf := val.Config.(awsConfig)
fmt.Println(conf.omega)
default:
log.Fatalf("No matching type, was type %v", val.Type)
}
}
Code in playground: https://go.dev/play/p/klxOoHMCtnG
Currently it gets unmarshalled as map[interface {}]interface {}, which can't be converted to one of the structs above.
Error:
panic: interface conversion: interface {} is map[interface {}]interface {}, not main.awsConfig \
Do I have to implemented the Unmarshaler Interface of the YAML package with a custom UnmarshalYAML function to get this done?

Found the solution by implementing Unmarshaler Interface:
type Fetcher struct {
Type string `yaml:"type"`
Config interface{} `yaml:"config"`
}
// Interface compliance
var _ yaml.Unmarshaler = &Fetcher{}
func (f *Fetcher) UnmarshalYAML(unmarshal func(interface{}) error) error {
var t struct {
Type string `yaml:"type"`
}
err := unmarshal(&t)
if err != nil {
return err
}
f.Type = t.Type
switch t.Type {
case "kubernetes":
var c struct {
Config kubernetesConfig `yaml:"config"`
}
err := unmarshal(&c)
if err != nil {
return err
}
f.Config = c.Config
case "aws":
var c struct {
Config awsConfig `yaml:"config"`
}
err := unmarshal(&c)
if err != nil {
return err
}
f.Config = c.Config
}
return nil
}

This type of task - where you want to delay the unmarshaling - is very similar to how json.RawMessage works with examples like this.
The yaml package does not have a similar mechanism for RawMessage - but this technique can easily be replicated as outlined here:
type RawMessage struct {
unmarshal func(interface{}) error
}
func (msg *RawMessage) UnmarshalYAML(unmarshal func(interface{}) error) error {
msg.unmarshal = unmarshal
return nil
}
// call this method later - when we know what concrete type to use
func (msg *RawMessage) Unmarshal(v interace{}) error {
return msg.unmarshal(v)
}
So to leverage this in your case:
var fs struct {
Configs []struct {
Type string `yaml:"type"`
Config RawMessage `yaml:"config"` // delay unmarshaling
} `yaml:"fetchers"`
}
err = yaml.Unmarshal([]byte(data), &fs)
if err != nil {
return
}
and based on the config "Type" (aws or kubernetes), you can finally unmarshal the RawMessage into the correct concrete type:
aws := awsConfig{} // concrete type
err = c.Config.Unmarshal(&aws)
or:
k8s := kubernetesConfig{} // concrete type
err = c.Config.Unmarshal(&k8s)
Working example here: https://go.dev/play/p/wsykOXNWk3H

Related

Unmarshal YAML into complex object which may be either struct or string

Trying to unmarshal YAML into complex object such as map[string]map[interface{}]string.
The problem is that I want to be able to differentiate an interface{} part between string and Source which is a struct.
type Source struct {
ID string `yaml:"id"`
Name string `yaml:"name"`
LogoID string `yaml:"logoId"`
URL string `yaml:"url"`
}
type UNFT struct {
ItemMeta map[string]map[interface{}]string `yaml:"item_meta"`
// could be
// ItemMeta map[string]map[string]string `yaml:"item_meta"`
// or
// ItemMeta map[string]map[Source]string `yaml:"item_meta"`
}
Obviously YAML does not know how to unmarshal into Source struct so I have to implement Unmarshaler interface:
type Unmarshaler interface {
UnmarshalYAML(value *Node) error
}
But I don't quite understand the big picture of unmarshaling process. In general I assume that I have to manually traverse *yaml.Node and call func UnmarshalYAML(value *Node) error on every node.
package main
import (
"fmt"
"gopkg.in/yaml.v3"
)
type Source struct {
ID string `json:"id"`
Name string `json:"name"`
LogoID string `json:"logoId"`
URL string `json:"url"`
}
var data = `
unf:
item_meta:
source:
!struct
? id: "data-watch"
name: "DataWatch"
logoid: "data-watch"
url: "https"
: "product_any('SS')"
public_usage:
"": "source_any('SDF')"
"provider": "source_any('ANO')"`
type UNFT struct {
ItemMeta map[string]map[interface{}]string `yaml:"item_meta"`
}
type MetaConverterConfigT struct {
UNFT UNFT `yaml:"unf"`
}
func main() {
cfg := MetaConverterConfigT{}
err := yaml.Unmarshal([]byte(data), &cfg)
if err != nil {
fmt.Println("%w", err)
}
fmt.Println(cfg)
}
func (s *UNFT) UnmarshalYAML(n *yaml.Node) error {
var cfg map[string]map[interface{}]string
if err := n.Decode(&cfg); err != nil {
fmt.Println("%w", err)
}
return nil
}
Go playground
type MetaKey struct {
String string
Source Source
}
func (k *MetaKey) UnmarshalYAML(n *yaml.Node) error {
if n.Tag == "!!str" {
return n.Decode(&k.String)
}
if n.Tag == "!!map" {
return n.Decode(&k.Source)
}
return fmt.Errorf("unsupported MetaKey type")
}
// ...
type UNFT struct {
ItemMeta map[string]map[MetaKey]string `yaml:"item_meta"`
}
https://go.dev/play/p/Nhtab4l-ANT
If you need the map type to remain as is, i.e. without adding the custom key type, then you can implement the unmarshaler on UNFT as well and just do a re-mapping with any:
type UNFT struct {
ItemMeta map[string]map[any]string `yaml:"item_meta"`
}
func (u *UNFT) UnmarshalYAML(n *yaml.Node) error {
var obj struct {
ItemMeta map[string]map[MetaKey]string `yaml:"item_meta"`
}
if err := n.Decode(&obj); err != nil {
return err
}
u.ItemMeta = make(map[string]map[any]string, len(obj.ItemMeta))
for k, v := range obj.ItemMeta {
m := make(map[any]string, len(v))
for k, v := range v {
if k.Source != (Source{}) {
m[k.Source] = v
} else {
m[k.String] = v
}
}
u.ItemMeta[k] = m
}
return nil
}
https://go.dev/play/p/uwboGKf3qnD

Unmarshal yaml map dict key to struct property

I really searched a while here, but didn't found an adequate answer:
I am trying to unmarshall yaml dict keys onto a property of a struct rather than the key of a map.
Given this yaml
commands:
php:
service: php
bin: /bin/php
node:
service: node
bin: /bin/node
I am able to unmarshall this into a struct like this:
type Config struct {
Commands map[string]struct {
Service string
Bin string
}
}
But how am I able to unmarshall it into a struct like this:
type Config struct {
Commands []struct {
Name string // <-- this should be key from the yaml (i.e. php or node)
Service string
Bin string
}
}
Thx in advance for the help
You can write a custom unmarshaler, like this (on Go playground):
package main
import (
"fmt"
"gopkg.in/yaml.v3"
)
var input []byte = []byte(`
commands:
php:
service: php
bin: /bin/php
node:
service: node
bin: /bin/node
`)
type Command struct {
Service string
Bin string
}
type NamedCommand struct {
Command
Name string
}
type NamedCommands []NamedCommand
type Config struct {
Commands NamedCommands
}
func (p *NamedCommands) UnmarshalYAML(value *yaml.Node) error {
if value.Kind != yaml.MappingNode {
return fmt.Errorf("`commands` must contain YAML mapping, has %v", value.Kind)
}
*p = make([]NamedCommand, len(value.Content)/2)
for i := 0; i < len(value.Content); i += 2 {
var res = &(*p)[i/2]
if err := value.Content[i].Decode(&res.Name); err != nil {
return err
}
if err := value.Content[i+1].Decode(&res.Command); err != nil {
return err
}
}
return nil
}
func main() {
var f Config
var err error
if err = yaml.Unmarshal(input, &f); err != nil {
panic(err)
}
for _, cmd := range f.Commands {
fmt.Printf("%+v\n", cmd)
}
}
I have split the command data into Command and NamedCommand to make the code simpler, since you can just call Decode giving the embedded Command struct for the values. If everything was in the same struct, you'd need to manually map keys to struct fields.

Go lang type assertion

I am trying to do dependency injection in golang with applying dependency inversion principle, so I have the following service
package account
import (
types "zaClouds/modules/account/domain/types"
"zaClouds/modules/shared"
)
type IPlanDomainService interface {
GetUsagePlanById(string) *shared.Result[types.UsagePlan]
}
type PlanDomainService struct {
usagePlanService types.IUsagePlanService
}
func (planDomainService *PlanDomainService) GetUsagePlanById(id string) *shared.Result[types.UsagePlan] {
result := &shared.Result[types.UsagePlan]{}
usagePlanResult := planDomainService.usagePlanService.GetPlanById(id)
if usagePlanResult.Err != nil {
result.Err = usagePlanResult.Err
return result
}
result.Data = usagePlanResult.Data
return result
}
func PlanDomainServiceFactory(usagePlanService types.IUsagePlanService) IPlanDomainService {
return &PlanDomainService{usagePlanService: usagePlanService}
}
as you can see, it accepts another service with type IUsagePlanService
and here is the interface for it
package account
import (
"zaClouds/modules/shared"
"github.com/shopspring/decimal"
)
type UsagePlan struct {
ID string
Title string
Description interface{}
PlanID string
Price decimal.Decimal
Duration int
Features map[string]map[string]string
}
type IUsagePlanService interface {
GetPlanById(string) *shared.Result[UsagePlan]
}
and here is the way I am injecting this service to domain service
func DiInit(usagePlanService interface{}) domainServices.IPlanDomainService {
domainServices.PlanDomainServiceFactory(types.IUsagePlanService(usagePlanService))
return domainServices.PlanDomainServiceFactory(usagePlanService.(types.IUsagePlanService))
}
as you can see, I am trying to do a type assertion but it doesn't work, and gives me the following error:
panic: interface conversion: *usagePlan.UsagePlanRepository is not account.IUsagePlanService: missing method GetPlanById
Edit
Here is the actual implementation for usagePlanService
type IUsagePlanRepository interface {
createClient(string) *http.Request
GetPlanById(string) *shared.Result[usagePlanRepoModels.UsagePlan]
}
type UsagePlanRepository struct {
plansEndpoint string
httpClient *http.Client
}
func (r *UsagePlanRepository) GetPlanById(id string) *shared.Result[usagePlanRepoModels.UsagePlan] {
result := &shared.Result[usagePlanRepoModels.UsagePlan]{}
req := r.createClient(id)
resp, err := r.httpClient.Do(req)
if err != nil {
log.Println("failed to load plan details \n[ERROR]", err)
result.Err = err
return result
}
defer func() {
bodyError := resp.Body.Close()
if bodyError != nil {
result.Err = bodyError
}
}()
if result.Err != nil {
return result
}
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
utils.Logger.Error("failed to load plan details \n[ERROR]", err, nil)
result.Err = err
return result
}
if resp.StatusCode >= 400 {
result.Err = errors.New(string(body))
utils.Logger.Info("getPlanById", string(body))
}
getUsagePlanResponse, foundError := usagePlanRepoModels.CreateGetUsagePlanResponse(body)
if foundError != nil {
result.Err = foundError
return result
}
result.Data = *getUsagePlanResponse
return result
}
When using an interface, you need to define all functions that you will use with the same name and signature as the implementation.
The error message you got indicates that the implementation and the interface are different.
The implementation is not shown in your question, but you defined the function for your interface like this: GetPlanById(string) *shared.Result[UsagePlan]. Any deviation from it will result in error. One common mistake is with the pointers. Adding or removing the * to the return type will incur in error if it differs from the original.
Edit:
Your interface should look like this:
type IUsagePlanService interface {
GetPlanById(id string) *shared.Result[usagePlanRepoModels.UsagePlan]
}
If your function is returning a private type, and you can change that, you should. If you cant change it, than you should create a function that wraps the function you are trying to abstract with the interface.

Strange behaviour when Unmarshalling into struct in Go

I'm developing a tool that can be implemented to simplify the process of creating simple CRUD operations/endpoints. Since my endpoints don't know what kind of struct they'll be receiving, I've created an interface that users can implement, and return an empty object to be filled.
type ItemFactory interface {
GenerateEmptyItem() interface{}
}
And the users would implement something like:
type Test struct {
TestString string `json:"testString"`
TestInt int `json:"testInt"`
TestBool bool `json:"testBool"`
}
func (t Test) GenerateEmptyItem() interface{} {
return Test{}
}
When the Test object gets created, its type is "Test", even though the func returned an interface{}. However, as soon as I try to unmarshal some json of the same format into it, it strips it of its type, and becomes of type "map[string]interface {}".
item := h.ItemFactory.GenerateEmptyItem()
//Prints "Test"
fmt.Printf("%T\n", item)
fmt.Println(reflect.TypeOf(item))
err := ConvertRequestBodyIntoObject(r, &item)
if err != nil {...}
//Prints "map[string]interface {}"
fmt.Printf("%T\n", item)
Func that unmarshalls item:
func ConvertRequestBodyIntoObject(request *http.Request, object interface{}) error {
body, err := ioutil.ReadAll(request.Body)
if err != nil {
return err
}
// Unmarshal body into request object
err = json.Unmarshal(body, object)
if err != nil {
return err
}
return nil
}
Any suggestions as to why this happens, or how I can work around it?
Thanks
Your question lacks an example showing this behavior so I'm just guessing this is what is happening.
func Generate() interface{} {
return Test{}
}
func GeneratePointer() interface{} {
return &Test{}
}
func main() {
vi := Generate()
json.Unmarshal([]byte(`{}`), &vi)
fmt.Printf("Generate Type: %T\n", vi)
vp := GeneratePointer()
json.Unmarshal([]byte(`{}`), vp)
fmt.Printf("GenerateP Type: %T\n", vp)
}
Which outputs:
Generate Type: map[string]interface {}
GenerateP Type: *main.Test
I suggest you return a pointer in GenerateEmptyItem() instead of the actual struct value which is demonstrated in the GenerateP() example.
Playground Example

How to parse each HCL dictionary item by golang?

I have tried to parse HCL config using golang, but it's not working.
type cfg_dict struct {
name string `hcl:",key"`
type string `hcl:"type"`
}
type hcl_config struct {
config_items cfg_dict `hcl:"config"`
}
func main() {
hcl_example = `config "cfg1" {
type = "string"
}`
hcl_opts := &hcl_config{}
hcl_tree, err := hcl.Parse(hcl_example)
if err != nil {
fmt.Println(err)
os.Exit(1)
}
if err := hcl.DecodeObject(&hcl_opts, hcl_tree); err != nil {
fmt.Println(err)
os.Exit(1)
}
fmt.Println(hcl_opts)
}
When I tried to run this test code after built, it shows empty value.
&{[]}
Is there any problem what I have to fix?
Fields on the struct you are attempting to unmarshal from HCL need to be exported. To export the fields make the first character in the field name upper case.
type cfg_dict struct {
Name string `hcl:",key"`
Type string `hcl:"type"`
}
type hcl_config struct {
Config_items cfg_dict `hcl:"config"`
}

Resources