I have tried to parse HCL config using golang, but it's not working.
type cfg_dict struct {
name string `hcl:",key"`
type string `hcl:"type"`
}
type hcl_config struct {
config_items cfg_dict `hcl:"config"`
}
func main() {
hcl_example = `config "cfg1" {
type = "string"
}`
hcl_opts := &hcl_config{}
hcl_tree, err := hcl.Parse(hcl_example)
if err != nil {
fmt.Println(err)
os.Exit(1)
}
if err := hcl.DecodeObject(&hcl_opts, hcl_tree); err != nil {
fmt.Println(err)
os.Exit(1)
}
fmt.Println(hcl_opts)
}
When I tried to run this test code after built, it shows empty value.
&{[]}
Is there any problem what I have to fix?
Fields on the struct you are attempting to unmarshal from HCL need to be exported. To export the fields make the first character in the field name upper case.
type cfg_dict struct {
Name string `hcl:",key"`
Type string `hcl:"type"`
}
type hcl_config struct {
Config_items cfg_dict `hcl:"config"`
}
Related
I really searched a while here, but didn't found an adequate answer:
I am trying to unmarshall yaml dict keys onto a property of a struct rather than the key of a map.
Given this yaml
commands:
php:
service: php
bin: /bin/php
node:
service: node
bin: /bin/node
I am able to unmarshall this into a struct like this:
type Config struct {
Commands map[string]struct {
Service string
Bin string
}
}
But how am I able to unmarshall it into a struct like this:
type Config struct {
Commands []struct {
Name string // <-- this should be key from the yaml (i.e. php or node)
Service string
Bin string
}
}
Thx in advance for the help
You can write a custom unmarshaler, like this (on Go playground):
package main
import (
"fmt"
"gopkg.in/yaml.v3"
)
var input []byte = []byte(`
commands:
php:
service: php
bin: /bin/php
node:
service: node
bin: /bin/node
`)
type Command struct {
Service string
Bin string
}
type NamedCommand struct {
Command
Name string
}
type NamedCommands []NamedCommand
type Config struct {
Commands NamedCommands
}
func (p *NamedCommands) UnmarshalYAML(value *yaml.Node) error {
if value.Kind != yaml.MappingNode {
return fmt.Errorf("`commands` must contain YAML mapping, has %v", value.Kind)
}
*p = make([]NamedCommand, len(value.Content)/2)
for i := 0; i < len(value.Content); i += 2 {
var res = &(*p)[i/2]
if err := value.Content[i].Decode(&res.Name); err != nil {
return err
}
if err := value.Content[i+1].Decode(&res.Command); err != nil {
return err
}
}
return nil
}
func main() {
var f Config
var err error
if err = yaml.Unmarshal(input, &f); err != nil {
panic(err)
}
for _, cmd := range f.Commands {
fmt.Printf("%+v\n", cmd)
}
}
I have split the command data into Command and NamedCommand to make the code simpler, since you can just call Decode giving the embedded Command struct for the values. If everything was in the same struct, you'd need to manually map keys to struct fields.
I'm trying to unmarshal the following YAML data into Go structures.
The data is the in the following format:
fetchers:
- type: "aws"
config:
omega: "lul"
- type: "kubernetes"
config:
foo: "bar"
Based of the type field, I want to determine wether to unmarshal the config field into awsConfig or kubernetesConfig struct.
My current code looks like this (using "gopkg.in/yaml.v2"):
type kubernetesConfig struct {
foo string `yaml:"foo"`
}
type awsConfig struct {
omega string `yaml:"omega"`
}
var c struct {
Fetchers []struct {
Type string `yaml:"type"`
Config interface{} `yaml:"config"`
} `yaml:"fetchers"`
}
err := yaml.Unmarshal(data, &c)
if err != nil {
log.Fatal(err)
}
for _, val := range c.Fetchers {
switch val.Type {
case "kubernetes":
conf := val.Config.(kubernetesConfig)
fmt.Println(conf.foo)
case "aws":
conf := val.Config.(awsConfig)
fmt.Println(conf.omega)
default:
log.Fatalf("No matching type, was type %v", val.Type)
}
}
Code in playground: https://go.dev/play/p/klxOoHMCtnG
Currently it gets unmarshalled as map[interface {}]interface {}, which can't be converted to one of the structs above.
Error:
panic: interface conversion: interface {} is map[interface {}]interface {}, not main.awsConfig \
Do I have to implemented the Unmarshaler Interface of the YAML package with a custom UnmarshalYAML function to get this done?
Found the solution by implementing Unmarshaler Interface:
type Fetcher struct {
Type string `yaml:"type"`
Config interface{} `yaml:"config"`
}
// Interface compliance
var _ yaml.Unmarshaler = &Fetcher{}
func (f *Fetcher) UnmarshalYAML(unmarshal func(interface{}) error) error {
var t struct {
Type string `yaml:"type"`
}
err := unmarshal(&t)
if err != nil {
return err
}
f.Type = t.Type
switch t.Type {
case "kubernetes":
var c struct {
Config kubernetesConfig `yaml:"config"`
}
err := unmarshal(&c)
if err != nil {
return err
}
f.Config = c.Config
case "aws":
var c struct {
Config awsConfig `yaml:"config"`
}
err := unmarshal(&c)
if err != nil {
return err
}
f.Config = c.Config
}
return nil
}
This type of task - where you want to delay the unmarshaling - is very similar to how json.RawMessage works with examples like this.
The yaml package does not have a similar mechanism for RawMessage - but this technique can easily be replicated as outlined here:
type RawMessage struct {
unmarshal func(interface{}) error
}
func (msg *RawMessage) UnmarshalYAML(unmarshal func(interface{}) error) error {
msg.unmarshal = unmarshal
return nil
}
// call this method later - when we know what concrete type to use
func (msg *RawMessage) Unmarshal(v interace{}) error {
return msg.unmarshal(v)
}
So to leverage this in your case:
var fs struct {
Configs []struct {
Type string `yaml:"type"`
Config RawMessage `yaml:"config"` // delay unmarshaling
} `yaml:"fetchers"`
}
err = yaml.Unmarshal([]byte(data), &fs)
if err != nil {
return
}
and based on the config "Type" (aws or kubernetes), you can finally unmarshal the RawMessage into the correct concrete type:
aws := awsConfig{} // concrete type
err = c.Config.Unmarshal(&aws)
or:
k8s := kubernetesConfig{} // concrete type
err = c.Config.Unmarshal(&k8s)
Working example here: https://go.dev/play/p/wsykOXNWk3H
I have a problem with binding my request, because there are a lot of parameters, so I used struct containing param.
package api
import (
"github.com/labstack/echo/v4"
"net/http"
"trains-api/domain/models"
"trains-api/domain/services"
)
type reqCreate struct {
RequestNotifi models.ResquestCreateNotifi
}
func CreateNotification (c echo.Context) error {
req := reqCreate{}
if err := c.Bind(req); err != nil {
return c.JSON(http.StatusNotFound, err)
}
}
package models
type RequestCreateNotifi struct {
Name_param1 string `db:"Name_param1"`
Name_param2 string `db:"Name_param2"`
....
Name_param_n string `db:"Name_paramN"`
}
error at if err := c.Bind(req); err != nil
r = {interface {} | string } "reflect: Elem of invalid type"
You need to set the JSON equivalent of each field in the model like so:
package models
type RequestCreateNotifi struct {
Name_param1 string `json:"name_param1" db:"Name_param1"`
Name_param2 string `json:"name_param2" db:"Name_param2"`
....
Name_param_n string `json:"name_param_n" db:"Name_param n"`
}
This json field specifies how the field is represented in the request so it can bind it to the correct value.
You need to add the pointer
req := reqCreate{}
if err := c.Bind(&req); err != nil {
return c.JSON(http.StatusNotFound, err)
}
Unfortunately you can't bind automatically query parameter using Post methode for security reasons according to issue#1670, the way to do it is using echo.QueryParamsBinder
type Query struct {
Param1 string `query:"param1"`
Param2 string `query:"param2"`
}
...
query := new(Query)
err := echo.QueryParamsBinder(ctx).String("param1", &query.Param1).String("param2", &query.Param2).BindError()
...
I have a struct defined for my Yaml file like so:
type Service struct {
ServiceName string `yaml:"service_name"`
PipelineType PipelineType `yaml:"pipeline_type"`
}
In the file, this struct comes in as a string:
service_name: service
pipeline_type: app
My type is defined like this:
// PipelineType Pipeline Types
type PipelineType struct {
Value string
}
var (
AppPipeline = PipelineType{"app"}
...
)
Because of the type declaration, I'm getting the following error (expected):
line 4: cannot unmarshal !!str `app` into main.PipelineType
Is there a way to tell GoYAML, or create some form of consturctor to convert the value using string(PipelineType) or something similar?
Alternatively, is there a "GoYAML" friendly way to do this?
try this:
type Service struct {
ServiceName string `yaml:"service_name"`
PipelineType PipelineType `yaml:"pipeline_type"`
}
func (s *Service) myYml() *Service {
yamlFile, err := ioutil.ReadFile("service.yaml")
if err != nil {
log.Println(err.Error())
}
err = yaml.Unmarshal(yamlFile, s)
if err != nil {
log.Println(err.Error())
}
return s
}
get it:
var s Service
s.myYml()
Trying to Unmarshal a hcl config file to a struct, using viper, this error is returned: 1 error(s) decoding:\n\n* 'NATS' expected a map, got 'slice'. What is missing?
The code:
func lab() {
var c conf
// config file
viper.SetConfigName("draft")
viper.AddConfigPath(".")
viper.SetConfigType("hcl")
if err := viper.ReadInConfig(); err != nil {
log.Error(err)
return
}
log.Info(viper.Get("NATS")) // gives [map[port:10041 username:cl1 password:__Psw__4433__ http_port:10044]]
if err := viper.Unmarshal(&c); err != nil {
log.Error(err)
return
}
log.Infow("got conf", "conf", c)
}
type conf struct {
NATS struct {
HTTPPort int
Port int
Username string
Password string
}
}
And the config file (draft.hcl inside current directory):
NATS {
HTTPPort = 10044
Port = 10041
Username = "cl1"
Password = "__Psw__4433__"
}
Edit
Have checked this struct with hcl package and it gets marshaled/unmarshalled correctly. Also this works correctly with yaml and viper.
There is a difference between these two where log.Info(viper.Get("NATS")) is called. While the hcl version returns a slice of maps, the yaml version returns a map: map[password:__psw__4433__ httpport:10044 port:10041 username:cl1].
Your conf struct is not matching the HCL. When converted to json the HCL looks like below
{
"NATS": [
{
"HTTPPort": 10044,
"Password": "__Psw__4433__",
"Port": 10041,
"Username": "cl1"
}
]
}
So the Conf Struct should look like this
type Conf struct {
NATS []struct{
HTTPPort int
Port int
Username string
Password string
}
}
Modified code
package main
import (
"log"
"github.com/spf13/viper"
"fmt"
)
type Conf struct {
NATS []struct{
HTTPPort int
Port int
Username string
Password string
}
}
func main() {
var c Conf
// config file
viper.SetConfigName("draft")
viper.AddConfigPath(".")
viper.SetConfigType("hcl")
if err := viper.ReadInConfig(); err != nil {
log.Fatal(err)
}
fmt.Println(viper.Get("NATS")) // gives [map[port:10041 username:cl1 password:__Psw__4433__ http_port:10044]]
if err := viper.Unmarshal(&c); err != nil {
log.Fatal(err)
}
fmt.Println(c.NATS[0].Username)
}
I know this question is more than two years old now, but I came across the same issue recently.
I'm using viper to be able to load different configuration files into a Go struct, allowing configuration in JSON, YAML, TOML, HCL, just pick your favourite :)
HCL file format does wrap a map into a slice because it allows redefining a section like:
section = {
key1 = "value"
}
section = {
key2 = "value"
}
which is something that is not supported by the other formats.
And here's how I fixed it:
My solution implies each new block will override any previous definition of the same key, and keep all the others. You can do some merging magic but I didn't need to.
You need to make a hook to convert a slice of maps into a map:
// sliceOfMapsToMapHookFunc merges a slice of maps to a map
func sliceOfMapsToMapHookFunc() mapstructure.DecodeHookFunc {
return func(from reflect.Type, to reflect.Type, data interface{}) (interface{}, error) {
if from.Kind() == reflect.Slice && from.Elem().Kind() == reflect.Map && (to.Kind() == reflect.Struct || to.Kind() == reflect.Map) {
source, ok := data.([]map[string]interface{})
if !ok {
return data, nil
}
if len(source) == 0 {
return data, nil
}
if len(source) == 1 {
return source[0], nil
}
// flatten the slice into one map
convert := make(map[string]interface{})
for _, mapItem := range source {
for key, value := range mapItem {
convert[key] = value
}
}
return convert, nil
}
return data, nil
}
}
then you need to create a DecodeHook:
configOption := viper.DecodeHook(mapstructure.ComposeDecodeHookFunc(
sliceOfMapsToMapHookFunc(),
mapstructure.StringToTimeDurationHookFunc(),
mapstructure.StringToSliceHookFunc(","),
))
the two other hooks are the default ones so you might want to keep them
then you pass the option to the Unmarshal method
viper.Unmarshal(&c, configOption)
With this method you don't need a slice around your structs or your maps. Also that makes it compatible with the other configuration file formats