Go YAML parsing: mandatory fields - go

Summary:
I need to parse data in YAML format into golang struct. It there a way (library, attributes) to make some of the fields mandatory, i.e. to make Unmarshal function return the error in case if some field doesn't exist?
Example what is wanted:
Unmarshal function in this code should raise an error because input data doesn't contain 'b' field.
package main
import (
"fmt"
"gopkg.in/yaml.v2"
)
type TestStruct struct {
FieldA string `yaml:"a"`
FieldB string `yaml:"b"`
}
func main() {
input := []byte(`{a: 1}`)
var output TestStruct
_ = yaml.Unmarshal(input, &output)
}

You can use this library's HasZero method to check wether there are any missing values in a struct. This will return true or false depending wether the struct is completely filled or not. Please see the playground example to get an idea.
But if you specifically need to tell what field is missing, you need to check wether the value is nil like in the example below.
package main
import (
"fmt"
"errors"
"gopkg.in/yaml.v2"
)
type TestStruct struct {
FieldA string `yaml:"a"`
FieldB string `yaml:"b"`
}
func main() {
input := []byte(`{a: 1}`)
var output TestStruct
if err := output.ParseFromFile(input); err != nil {
fmt.Println(err)
}
fmt.Println(output)
}
func (output *TestStruct) ParseFromFile(data []byte) error {
if err := yaml.Unmarshal(data, output); err != nil {
return err
}
if output.FieldA == "" {
return errors.New("Blank Field A")
}
if output.FieldB == "" {
return errors.New("Blank Field B")
}
return nil
}
Playground example if you need to specifically return an error

Related

Reading and Unmarshalling API results in Golang

In the below program I'm extracting some data from an API.
It outputs a rather complex data.
When I ioutil.ReadAll(resp.Body), the result is of type []uint8.
If I try to read the results, its just a random array of integers.
However, I'm able to read it if I convert it to string using string(diskinfo)
But I want to use this in a Struct and having trouble unmarshalling.
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"net/http"
"net/url"
"reflect"
)
type ApiResults struct {
results []struct {
statement_id int `json.statement_id`
series []struct {
name string `json.name`
tags struct {
host string `json.host`
}
columns []string `json.columns`
values []interface{} `json.values`
}
}
}
func main() {
my_url := "my_url"
my_qry := fmt.Sprintf("my_query")
resp, err := http.Get(my_url + url.QueryEscape(my_qry))
if err != nil {
fmt.Printf("ERROR: %v\n", err)
} else {
fmt.Println(reflect.TypeOf(resp))
diskinfo, _ := ioutil.ReadAll(resp.Body)
fmt.Println(reflect.TypeOf((diskinfo)))
fmt.Println(diskinfo)
fmt.Println(string(diskinfo))
diskinfo_string := string(diskinfo)
data := ApiResults{}
json.Unmarshal([]byte(diskinfo_string), &data)
//fmt.Printf("Values = %v\n", data.results.series.values)
//fmt.Printf("Server = %v\n", data.results.series.tags.host)
}
}
If I view the data as a string, I get this (formatted):
{"results":[
{"statement_id":0,
"series":[
{"name":"disk",
"tags":{"host":"myServer1"},
"columns":["time","disk_size"],
"values":[["2021-07-07T07:53:32.291490387Z",1044]]},
{"name":"disk",
"tags":{"host":"myServer2"},
"columns":["time","disk_size"],
"values":[["2021-07-07T07:53:32.291490387Z",1046]]}
]}
]}
I think my Apireturn struct is also structured incorrectly because the API results have info for multiple hosts.
But first, I doubt if the data has to be sent in a different format to the struct. Once I do this, I can probably try to figure out how to read from the Struct next.
The ioutil.ReadAll already provides you the data in the type byte[]. Therefore you can just call json.Unmarshal passing it as a parameter.
import (
"encoding/json"
"io/ioutil"
"net/http"
)
func toStruct(res *http.Response) (*ApiResults, error) {
body, err := ioutil.ReadAll(res.Body)
if err != nil {
return nil, err
}
defer res.Body.Close()
data := ApiResults{}
if err := json.Unmarshal(body, &data); err != nil {
return nil, err
}
return data, nil
}
There also seems to be an issue with your struct. The correct way to use struct tags is as follows. Plus, fields need to be exported for the json tag (used by json.Umarshal) to work – starting with uppercase will do it.
type ApiResults struct {
Results []struct {
StatementId int `json:"statement_id"`
Series []struct {
Name string `json:"name"`
Tags struct {
Host string `json:"host"`
} `json:"tags"`
Columns []string `json:"columns"`
Values []interface{} `json:"values"`
} `json:"series"`
} `json:"results"`
}

Inputs on correct usage of binding:"required" in Golang struct

I have a struct (with 1 field, for simplification purpose here) for which data comes from another function.
Further, for the struct field, I have put binding:"required", as I need it to be non-empty/present.
Code:
package main
import (
"fmt"
"encoding/json"
)
type Config struct {
Name string `yaml:"name" json:"name" binding:"required"`
}
func main() {
var myConfig Config
var rawConfig = []byte(`{}`) // Empty, Nothing getting passed.
err := json.Unmarshal(rawConfig, &myConfig)
if err != nil {
panic(err)
}
fmt.Printf("Name = %s\n", myConfig.Name)
}
When I run this, even if Name value has not been passed in, it passes.
Output:
Name =
Go playground Code link : https://play.golang.org/p/zA6nij9vTvY
I want to achieve the following:
As I expect the 'Name' field to be present, what do I do to make it fail, if not present. Is unmarshalling not expected to use binding=required tag and fail ?
Else, whats the best approach to figure out if a required field is not present so that we fail ?
Or is the crude way of iterating on fields, and figuring out if empty, the only way ?
You might do this in hard way by iterating over struct fields using reflect, get the tags and parse them which is not advised unless you need some customizations.
Or you might like use available validator packages:
https://github.com/dealancer/validate
package main
import (
"encoding/json"
"fmt"
"gopkg.in/dealancer/validate.v2"
)
type Config struct {
Name string `yaml:"name" json:"name" validate:"empty=false"`
}
func main() {
var myConfig Config
var rawConfig = []byte(`{}`) // Empty, Nothing getting passed.
err := json.Unmarshal(rawConfig, &myConfig)
if err != nil {
panic(err)
}
err = validate.Validate(myConfig)
if err != nil {
panic(err)
}
fmt.Printf("Name = %s\n", myConfig.Name)
}

How to omit empty json fields using json.decoder

I try to understand why both functions return the same output.
As far as I understood, the point of omit empty is to not add that key to the result struct.
I wrote this example, I was expecting the first output not to have the "Empty" key, but for some reason its value still shows as 0.
package main
import (
"encoding/json"
"fmt"
"strings"
)
type agentOmitEmpty struct {
Alias string `json:"Alias,omitempty"`
Skilled bool `json:"Skilled,omitempty"`
FinID int32 `json:"FinId,omitempty"`
Empty int `json:"Empty,omitempty"`
}
type agent struct {
Alias string `json:"Alias"`
Skilled bool `json:"Skilled"`
FinID int32 `json:"FinId"`
Empty int `json:"Empty"`
}
func main() {
jsonString := `{
"Alias":"Robert",
"Skilled":true,
"FinId":12345
}`
fmt.Printf("output with omit emtpy: %v\n", withEmpty(strings.NewReader(jsonString)))
// output with omit emtpy: {Robert true 12345 0}
fmt.Printf("output regular: %v\n", withoutEmpty(strings.NewReader(jsonString)))
// output without omit: {Robert true 12345 0}
}
func withEmpty(r *strings.Reader) agentOmitEmpty {
dec := json.NewDecoder(r)
body := agentOmitEmpty{}
err := dec.Decode(&body)
if err != nil {
panic(err)
}
return body
}
func withoutEmpty(r *strings.Reader) agent {
dec := json.NewDecoder(r)
body := agent{}
err := dec.Decode(&body)
if err != nil {
panic(err)
}
return body
}
You need to define Empty as *int so it will be replaced with nil when there is no value. Then it will not be saved in the database.

Print struct field tags while unmarshalling JSON content for a field?

In Go, Is it possible to get the tags from a struct field while I'm unmarshaling JSON content to it? Here's my failed attempt at doing so:
package main
import (
"log"
"encoding/json"
)
type Person struct {
ProfileName AltField `json:"profile_name"`
}
type AltField struct {
Val string
}
func (af *AltField) UnmarshalJSON(b []byte) error {
log.Println("Show tags")
//log.Println(af.Tag) // I want to see `json:"profile_name"`
if e := json.Unmarshal(b,&af.Val); e != nil {
return e
}
return nil
}
func main() {
p := Person{}
_ = json.Unmarshal([]byte(`{"profile_name":"Af"}`),&p)
}
I commented out the line log.Println(af.Tag) because it causes compilation errors. If I can get a handle on the tags from the Person struct, that will allow me to develop some other conditional logic.
Is this possible?
Use reflection to get the value of struct field tag. The reflect package provides functions to work with tags including to get the value of tag
package main
import (
"log"
"encoding/json"
"reflect"
)
type Person struct {
ProfileName AltField `json:"profile_name"`
}
type AltField struct {
Val string `json:"val"`
}
func (af *AltField) UnmarshalJSON(b []byte) error {
field, ok := reflect.TypeOf(*af).FieldByName("Val")
if !ok {
panic("Field not found")
}
log.Println(string(field.Tag))
if e := json.Unmarshal(b,&af.Val); e != nil {
return e
}
return nil
}
func main() {
p := Person{}
_ = json.Unmarshal([]byte(`{"profile_name":"Af"}`),&p)
}
You can only get the value of those field tags which has them. The struct field reflect object should be created for fetching the tags of its fields.
Working Code on Playground

how to access deeply nested json keys and values

I'm writing a websocket client in Go. I'm receiving the following JSON from the server:
{"args":[{"time":"2013-05-21 16:57:17"}],"name":"send:time"}
I'm trying to access the time parameter, but just can't grasp how to reach deep into an interface type:
package main;
import "encoding/json"
import "log"
func main() {
msg := `{"args":[{"time":"2013-05-21 16:56:16", "tzs":[{"name":"GMT"}]}],"name":"send:time"}`
u := map[string]interface{}{}
err := json.Unmarshal([]byte(msg), &u)
if err != nil {
panic(err)
}
args := u["args"]
log.Println( args[0]["time"] ) // invalid notation...
}
Which obviously errors, since the notation is not right:
invalid operation: args[0] (index of type interface {})
I just can't find a way to dig into the map to grab deeply nested keys and values.
Once I can get over grabbing dynamic values, I'd like to declare these messages. How would I write a type struct to represent such complex data structs?
You may like to consider the package github.com/bitly/go-simplejson
See the doc: http://godoc.org/github.com/bitly/go-simplejson
Example:
time, err := json.Get("args").GetIndex(0).String("time")
if err != nil {
panic(err)
}
log.Println(time)
The interface{} part of the map[string]interface{} you decode into will match the type of that field. So in this case:
args.([]interface{})[0].(map[string]interface{})["time"].(string)
should return "2013-05-21 16:56:16"
However, if you know the structure of the JSON, you should try defining a struct that matches that structure and unmarshal into that. Ex:
type Time struct {
Time time.Time `json:"time"`
Timezone []TZStruct `json:"tzs"` // obv. you need to define TZStruct as well
Name string `json:"name"`
}
type TimeResponse struct {
Args []Time `json:"args"`
}
var t TimeResponse
json.Unmarshal(msg, &t)
That may not be perfect, but should give you the idea
I'm extremely new to Golang coming from Python, and have always struggled with encode/decoding json. I found gjson at https://github.com/tidwall/gjson, and it helped me immensely:
package main
import "github.com/tidwall/gjson"
func main() {
msg := (`{"args":[{"time":"2013-05-21 16:56:16", "tzs":[{"name":"GMT"}]}],"name":"send:time"}`)
value := gjson.Get(msg, "args.#.time")
println(value.String())
}
-----------------------
["2013-05-21 16:56:16"]
Additionally, I noticed the comment of how to convert into Struct
package main
import (
"encoding/json"
"fmt"
)
type msgFormat struct {
Time string `json:"time"`
Tzs msgFormatTzs `json:"tzs"`
Name string `json:"name"`
}
type msgFormatTzs struct {
TzsName string `json:"name"`
}
func main() {
msg := (`{"args":[{"time":"2013-05-21 16:56:16", "tzs":[{"name":"GMT"}]}],"name":"send:time"}`)
r, err := json.Marshal(msgFormatTzs{msg})
if err != nil {
panic(err)
}
fmt.Printf("%v", r)
}
Try on Go playground

Resources