Yaml file to struct parsing (translation) in Golang - go

I'm trying to build a translation feature for my webapp. There are multiple packages in my app. Each package(directory) contains a translation folder and yaml files inside. But I have a problem with parsing and assign it to messages.
en.yaml
msgLogin : "You've login successfully"
msgProducts:
0: "You don't have any product."
1: "You have %d product."
2: "You have %d products."
errLogin: "Wrong password or username"
my code:
type TranslationEntry struct {
Key struct {
Value interface{}
}
}
func parseTranslations(dir string) {
files, _ := ioutil.ReadDir(dir)
for _, f := range files {
yamlFile, _ := ioutil.ReadFile(dir + "/" + f.Name())
var data translation
if err := yaml.Unmarshal(yamlFile, &data); err != nil {
return nil, err
}
lang := strings.Split(f.Name(), ".")[0]
switch msg := data.Key.Value.(type) {
case string:
message.SetString(language.Make(lang), cast.ToString(data.Key), cast.ToString(data.Key.Value))
case map[int]string:
message.Set(language.Make(lang), cast.ToString(data.Key),
plural.Selectf(1, "%d",
"=0", cast.ToString(data.Key.Value["0"]),
"=1", cast.ToString(data.Key.Value["1"]),
"=2", cast.ToString(data.Key.Value["2"]),
))
}
translations[lang] = &dictionary{Data: data}
}
}
I'm totally lost about how to design my struct or parse yaml file.
Thank you in advanced

If you're using the YAML library I think you're using (https://godoc.org/gopkg.in/yaml.v2), to make a Golang struct which can use Unmarshal to map from the YAML file in your post you can do this:
type TranslationEntry struct {
MsgLogin string `yaml:"msgLogin"`
MsgProducts map[int]string `yaml:"msgProducts"`
ErrLogin string `yaml:"errLogin"`
}
The things inside the `` after the field declarations are called tags. They're the way field names are usually specified when mapping between some datatype and a Golang struct (in my case usually I map a struct to JSON, but I've also done YAML). If you're using the same YAML parser I mentioned above, this is how it works.
Basically the text inside the double quotes is the YAML key to which your struct field will be mapped. In the above code the only difference between the YAML key name and the struct field name is capitalization, but here is an example using totally different names:
type ExampleStruct struct {
MyAbcField string `yaml:"abc"`
}
This will set the value of MyAbcField to "my data" when using Unmarshal with ExampleStruct and the following YAML:
abc: "my data"
This allows you to design a Golang struct which matches however you decide to structure your YAML.
Here's my above code in Go Playground: https://play.golang.org/p/Q9FvNsw-BOx
Now, if you are unable to fix a structure for your YAML files, you can also parse into nested maps. You can do this by passing a variable of type interface{} (empty interface) to Unmarshal instead of a struct. However, this requires a lot of boilerplate because you will need to do type assertions to access your data. Thus I recommend using a fixed structure instead unless you absolutely can't avoid it.
Here's an example where I parse the YAML you posted and then get the msgLogin field:
var data interface{}
if err := yaml.Unmarshal([]byte(yamlFile), &data); err != nil {
// handle error
}
// a type assertion that data is a map is needed in order to get keys or iterate
topLevel, ok := data.(map[interface{}]interface{})
if !ok {
// handle error
}
fmt.Println(topLevel["msgLogin"])
And here's the Go Playground of my struct example changed to use parsing into a nested map instead: https://play.golang.org/p/ERBjClSazkz

Related

Trouble unmarshaling YAML structure

I have what I think is a very straightforward YAML structure I'm attempting to write to and read from a file.
appName:
version: 1.2.3.4
md5_checksum: 987654321
And I'm really struggling to understand nested structs and how they relate to yaml marshaling.
At this point I have the following:
type Application struct {
Name string `yaml:"application"`
Version string `yaml:"version"`
Checksum int `yaml:"md5_checksum"`
}
yamlData := Application{"MyProgram", "1.2.3.4", 34235234123}
y, err := yaml.Marshal(yamlData)
if err != nil {
log.Fatal("Yaml marshal failed")
}
err = ioutil.WriteFile("applications.yaml", y, 0644)
if err != nil {
log.Fatal("File write failed")
}
This gets me close, but the file reads as follows:
application: MyProgram
version: 1.2.3.4
md5_checksum: 34235234123
That looks nice, but this file will be populated by many applications. I've tried a few types of nested structs, but I get hung up on how to supply values to them when finally building them.
You can define Application struct as
type AppVersion struct {
Version string `yaml:"version"`
Checksum int `yaml:"md5_checksum"`
}
type Application struct {
Name AppVersion `yaml:"appName"`
}
Then call AppVersion as
yamlData := Application{Name: AppVersion{Version: "1.2.3.4", Checksum: 34235234123}}
I've solved my own problem with help from KibGzr.
No structs necessary just for the marshaling steps.
yamlData := map[string]map[string]string{
appName: map[string]string{"version": appVersion, "md5_checksum": checksumString},
}
I've created a nested map rather than a struct. This allows me just initialize the map when I've collected all my values rather than trying to work around struct field names. My output is now:
appName:
md5_checksum: checksumString
version: appVersion
Thanks for all the help!

interface{} to []string

I'm trying to parse some YAML file into go struct, but the file itself can not be treated as ordinary go structure: some of values may be either string or map[string][]string.
What I have tried is to do custom unmarshal func:
func (domain *Domain) UnmarshalYAML(unmarshal func(interface{}) error) error {
fmt.Println("Parsing domain")
var hostUrl interface{}
unmarshal(&hostUrl)
fmt.Println(reflect.TypeOf(hostUrl))
switch hostUrl.(type) {
case string:
domain.Host = hostUrl.(string)
case map[interface {}]interface {}:
fmt.Println("got map")
v := reflect.ValueOf(hostUrl)
fmt.Println(v.MapKeys()[0])
for _, host := range v.MapKeys() {
domain.Host = host.Interface().(string)
fmt.Println(v.MapIndex(host))
//HERE I NEED TO DO SMTH LIKE THIS:
//domain.Urls = v.MapIndex(host).([]string)
}
default:
return errors.New("invalid config file, cant parse domains")
}
return nil
}
My domain structure looks like this:
type Domain struct {
Host string
Urls []string
}
But this code causes an error:
invalid type assertion: v.MapIndex(host).([]string) (non-interface type reflect.Value on left)
So my question may sound like "how to convert {}interface to []string?" or it may become more complex: "How to parse YAML file into go struct if some key can be either simple string or map[string][]string?"
UPDATE:
Responding to #mkopriva:
fmt.Println(v.MapIndex(host))
for url := range v.MapIndex(host).Interface().([] interface{}) {
fmt.Println(url)
}
Didnt help me, as now it just prints some integers (0), but there should be a string. Converting it to an array of strings throws another error:
panic: interface conversion: interface {} is []interface {}, not []string
Thanks to #mkopriva and the snippet from the sandbox. The reason of integer appearing during iterating over v.MapIndex(host).Interface().([] interface{}) is that range returns two values: index and corresponding to that index value. I was only catching the first one. It is why I wasn't able to cast it to string.
Working loop:
for _, url := range v.MapIndex(host).Interface().([] interface{}) {
fmt.Println(url.(string))
domain.Urls = append(domain.Urls,url.(string) )
}

Golang get string representation of specific struct field name

I really want a way to print the string representation of a field name in go. It has several use cases, but here is an example:
lets say I have a struct
type Test struct {
Field string `bson:"Field" json:"field"`
OtherField int `bson:"OtherField" json:"otherField"`
}
and, for example, I want to do a mongo find:
collection.Find(bson.M{"OtherField": someValue})
I don't like that I have to put the string "OtherField" in there. It seems brittle and easy to either misstype or have the struct change and then my query fails without me knowing it.
Is there any way to get the string "OtherField" without having to either declare a const or something like that? I know I can use reflection to a get a list of field names from a struct, but I'd really like to do something along the lines of
fieldName := nameOf(Test{}.OtherField)
collection.Find(bson.M{fieldName: someValue})
is there any way to do this in Go?? C# 6 has the built in nameof, but digging through reflection I can't find any way to do this in Go.
I don't really think there is. You may be able to load a set of types via reflection and generate a set of constants for the field names. So:
type Test struct {
Field string `bson:"Field" json:"field"`
OtherField int `bson:"OtherField" json:"otherField"`
}
Could generate something like:
var TestFields = struct{
Field string
OtherField string
}{"Field","OtherField"}
and you could use TestFields.Field as a constant.
Unfortunately, I don't know of any existing tool that does anything like that. Would be fairly simple to do, and wire up to go generate though.
EDIT:
How I'd generate it:
Make a package that accepts an array of reflect.Type or interface{} and spits out a code file.
Make a generate.go somewhere in my repo with main function:
func main(){
var text = mygenerator.Gen(Test{}, OtherStruct{}, ...)
// write text to constants.go or something
}
Add //go:generate go run scripts/generate.go to my main app and run go generate
Here is a function that will return a []string with the struct field names. I think it comes in the order they are defined.
WARNING: Reordering the fields in the struct definition will change the order in which they appear
https://play.golang.org/p/dNATzNn47S
package main
import (
"fmt"
"strings"
"regexp"
)
type Test struct {
Field string `bson:"Field" json:"field"`
OtherField int `bson:"OtherField" json:"otherField"`
}
func main() {
fields, err := GetFieldNames(Test{})
if err != nil {
fmt.Println(err)
return
}
fmt.Println(fields)
}
func GetFieldNames(i interface{}) ([]string, error) {
// regular expression to find the unquoted json
reg := regexp.MustCompile(`(\s*?{\s*?|\s*?,\s*?)(['"])?(?P<Field>[a-zA-Z0-9]+)(['"])?:`)
// print struct in almost json form (fields unquoted)
raw := fmt.Sprintf("%#v", i)
// remove the struct name so string begins with "{"
fjs := raw[strings.Index(raw,"{"):]
// find and grab submatch 3
matches := reg.FindAllStringSubmatch(fjs,-1)
// collect
fields := []string{}
for _, v := range matches {
if len(v) >= 3 && v[3] != "" {
fields = append(fields, v[3])
}
}
return fields, nil
}

Using empty interfaces in go

I am trying to understand the code that is used at my company. I am new to go lang, and I have already gone through the tutorial on their official website. However, I am having a hard time wrapping my head around empty interfaces, i.e. interface{}. From various sources online, I figured out that the empty interface can hold any type. But, I am having a hard time figuring out the codebase, especially some of the functions. I will not be posting the entire thing here, but just the minimal functions in which it has been used. Please bear with me!
Function (I am trying to understand):
func (this *RequestHandler) CreateAppHandler(rw http.ResponseWriter, r *http.Request) *foo.ResponseError {
var data *views.Data = &views.Data{Attributes: &domain.Application{}}
var request *views.Request = &views.Request{Data: data}
if err := json.NewDecoder(r.Body).Decode(request); err != nil {
logrus.Error(err)
return foo.NewResponsePropogateError(foo.STATUS_400, err)
}
requestApp := request.Data.Attributes.(*domain.Application)
requestApp.CreatedBy = user
Setting some context, RequestHandler is a struct defined in the same package as this code. domain and views are seperate packages. Application is a struct in the package domain. The following two structs are part of the package views:
type Data struct {
Id string `json:"id"`
Type string `json:"type"`
Attributes interface{} `json:"attributes"`
}
type Request struct {
Data *Data `json:"data"`
}
The following are part of the package json:
func NewDecoder(r io.Reader) *Decoder {
return &Decoder{r: r}
}
func (dec *Decoder) Decode(v interface{}) error {
if dec.err != nil {
return dec.err
}
if err := dec.tokenPrepareForDecode(); err != nil {
return err
}
if !dec.tokenValueAllowed() {
return &SyntaxError{msg: "not at beginning of value"}
}
// Read whole value into buffer.
n, err := dec.readValue()
if err != nil {
return err
}
dec.d.init(dec.buf[dec.scanp : dec.scanp+n])
dec.scanp += n
// Don't save err from unmarshal into dec.err:
// the connection is still usable since we read a complete JSON
// object from it before the error happened.
err = dec.d.unmarshal(v)
// fixup token streaming state
dec.tokenValueEnd()
return err
}
type Decoder struct {
r io.Reader
buf []byte
d decodeState
scanp int // start of unread data in buf
scan scanner
err error
tokenState int
tokenStack []int
}
Now, I understood that, in the struct Data in package views, Application is being set as a type for the empty interface. After that, a pointer to Request in the same package is created which points to the variable data.
I have the following doubts:
What exactly does this keyword mean in Go? What is the purpose of writing this * RequestHandler?
Initialization of a structure in Go can be done while assigning it to a variable by specifying the values of all it's members. However, here, for the struct Data, only the empty interface value is assigned and the values for the other two fields are not assigned?
What is the advantage of assigning the Application struct to an empty interface? Does it mean I can use the struct members using the interface variable directly?
Can someone help me figure out the meaning of this statement? json.NewDecoder(r.Body).Decode(request)?
While I know this is too much, but I am having a hard time figuring out the meaning of interfaces in Go. Please help!
this is not a keyword in go; any variable name can be used there. That is called the receiver. A function declared in that way must be called like thing.func(params), where "thing" is an expression of the type of the receiver. Within the function, the receiver is set to the value of thing.
A struct literal does not have to contain values for all the fields (or any of them). Any fields not explicitly set will have the zero value for their types.
As you said, an empty interface can take on a value of any type. To use a value of type interface{}, you would use type assertion or a type switch to determine the type of the value, or you could use reflection to use the value without having to have code for the specific type.
What specifically about that statement do you not understand? json is the name of a package in which the function NewDecoder is declared. That function is called, and then the Decode function (which is implemented by the type of the return value of NewDecoder) is called on that return value.
You may want to take a look at Effective Go and/or The Go Programming Language Specification for more information.

Exported and Unexported fields in Go Language

I have a function in Go, the return value of which I hope to encode using gob. The return value is a struct pointer. However even though I do understand what exported variables are, I am not quite sure how to get it working.
Here is what my function is like
fun loadXYZ(root *structABC) *structABC{
const once = "stateData.bin"
rd, errr := ioutil.ReadFile(once)
if errr!=nil{
//Do some computation and store in "root"
buf := &bytes.Buffer{}
errr = gob.NewEncoder(buf).Encode(root)
if errr != nil {
panic(errr)
}
errr = ioutil.WriteFile(once, buf.Bytes(), 0666)
if errr != nil {
panic(errr)
}
return root
}
var d *structABC
errr = gob.NewDecoder(bytes.NewReader(rd)).Decode(&d)
if errr != nil {
panic(errr)
}
return d
}
This is the error I get
panic: gob: type main.stateNode has no exported fields
I know why the error is occurring. But can someone help me solve it?
In go, fields and variables that start with an Uppercase letter are "Exported", and are visible to other packages. Fields that start with a lowercase letter are "unexported", and are only visible inside their own package.
The encoding/gob package depends on reflection to encode values, and can only see exported struct fields.
In order to make things encodable, capitalize the first letter of each field name in your stateNode struct that you want to be saved.
Exported field it's a filed which name started with capitalized char like:
type stateNode struct {
ImExported string // Exported
butImNot string // unexported
}

Resources