Given the following YAML:
array.test: ["val1", "val2", "val3"]
I Unmarshal it using gopkg.in/yaml.v2 into a map[string]interface{}. Then I get a single key whose value is an array of 3 values.
When I then Marshal it again to YAML, the resulting YAML looks like this:
array.test:
- val1
- val2
- val3
The array was actually marshaled as a sequence instead of an array.
This is the entire GoLang code:
func main(){
data := `array.test: ["val1", "val2", "val3"]`
conf := make(map[string]interface{})
yaml.Unmarshal([]byte(data), conf)
data2, _ := yaml.Marshal(conf)
fmt.Printf("%s\n", string(data2))
}
How can I overcome this issue?
This one helped me in the same case as you.
package main
import (
"fmt"
"log"
"gopkg.in/yaml.v2"
)
var data = `
a: Easy!
b:
c: 2
d.test: ["d1", "d2"]
`
// Note: struct fields must be public in order for unmarshal to
// correctly populate the data.
type T struct {
A string
B struct {
RenamedC int `yaml:"c"`
DTest []string `yaml:"d.test,flow"`
}
}
func main() {
// if we use struct containing yaml encoding for yaml formated string
t := T{}
err := yaml.Unmarshal([]byte(data), &t)
if err != nil {
log.Fatalf("error: %v", err)
}
fmt.Printf("--- t after unmarshal:\n%v\n\n", t)
d, err := yaml.Marshal(&t)
if err != nil {
log.Fatalf("error: %v", err)
}
fmt.Printf("--- t after marshal:\n%s\n\n", string(d))
}
Ref: https://github.com/go-yaml/yaml
Use flow in struct field tag format, to indicate you desire this behavior. But, of course, this requires unmarshaling to a struct, not to a map.
Flow tag allows you to choose the representation of an array in yaml
package main
import (
"fmt"
"gopkg.in/yaml.v2"
)
type Conf struct {
Test []string `yaml:"array.test,flow"`
}
func main(){
data := `array.test: ["val1", "val2", "val3"]`
var conf Conf
yaml.Unmarshal([]byte(data), &conf)
data2, _ := yaml.Marshal(conf)
fmt.Printf("%s\n", string(data2))
}
Related
In the below program I'm extracting some data from an API.
It outputs a rather complex data.
When I ioutil.ReadAll(resp.Body), the result is of type []uint8.
If I try to read the results, its just a random array of integers.
However, I'm able to read it if I convert it to string using string(diskinfo)
But I want to use this in a Struct and having trouble unmarshalling.
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"net/http"
"net/url"
"reflect"
)
type ApiResults struct {
results []struct {
statement_id int `json.statement_id`
series []struct {
name string `json.name`
tags struct {
host string `json.host`
}
columns []string `json.columns`
values []interface{} `json.values`
}
}
}
func main() {
my_url := "my_url"
my_qry := fmt.Sprintf("my_query")
resp, err := http.Get(my_url + url.QueryEscape(my_qry))
if err != nil {
fmt.Printf("ERROR: %v\n", err)
} else {
fmt.Println(reflect.TypeOf(resp))
diskinfo, _ := ioutil.ReadAll(resp.Body)
fmt.Println(reflect.TypeOf((diskinfo)))
fmt.Println(diskinfo)
fmt.Println(string(diskinfo))
diskinfo_string := string(diskinfo)
data := ApiResults{}
json.Unmarshal([]byte(diskinfo_string), &data)
//fmt.Printf("Values = %v\n", data.results.series.values)
//fmt.Printf("Server = %v\n", data.results.series.tags.host)
}
}
If I view the data as a string, I get this (formatted):
{"results":[
{"statement_id":0,
"series":[
{"name":"disk",
"tags":{"host":"myServer1"},
"columns":["time","disk_size"],
"values":[["2021-07-07T07:53:32.291490387Z",1044]]},
{"name":"disk",
"tags":{"host":"myServer2"},
"columns":["time","disk_size"],
"values":[["2021-07-07T07:53:32.291490387Z",1046]]}
]}
]}
I think my Apireturn struct is also structured incorrectly because the API results have info for multiple hosts.
But first, I doubt if the data has to be sent in a different format to the struct. Once I do this, I can probably try to figure out how to read from the Struct next.
The ioutil.ReadAll already provides you the data in the type byte[]. Therefore you can just call json.Unmarshal passing it as a parameter.
import (
"encoding/json"
"io/ioutil"
"net/http"
)
func toStruct(res *http.Response) (*ApiResults, error) {
body, err := ioutil.ReadAll(res.Body)
if err != nil {
return nil, err
}
defer res.Body.Close()
data := ApiResults{}
if err := json.Unmarshal(body, &data); err != nil {
return nil, err
}
return data, nil
}
There also seems to be an issue with your struct. The correct way to use struct tags is as follows. Plus, fields need to be exported for the json tag (used by json.Umarshal) to work – starting with uppercase will do it.
type ApiResults struct {
Results []struct {
StatementId int `json:"statement_id"`
Series []struct {
Name string `json:"name"`
Tags struct {
Host string `json:"host"`
} `json:"tags"`
Columns []string `json:"columns"`
Values []interface{} `json:"values"`
} `json:"series"`
} `json:"results"`
}
I try to understand why both functions return the same output.
As far as I understood, the point of omit empty is to not add that key to the result struct.
I wrote this example, I was expecting the first output not to have the "Empty" key, but for some reason its value still shows as 0.
package main
import (
"encoding/json"
"fmt"
"strings"
)
type agentOmitEmpty struct {
Alias string `json:"Alias,omitempty"`
Skilled bool `json:"Skilled,omitempty"`
FinID int32 `json:"FinId,omitempty"`
Empty int `json:"Empty,omitempty"`
}
type agent struct {
Alias string `json:"Alias"`
Skilled bool `json:"Skilled"`
FinID int32 `json:"FinId"`
Empty int `json:"Empty"`
}
func main() {
jsonString := `{
"Alias":"Robert",
"Skilled":true,
"FinId":12345
}`
fmt.Printf("output with omit emtpy: %v\n", withEmpty(strings.NewReader(jsonString)))
// output with omit emtpy: {Robert true 12345 0}
fmt.Printf("output regular: %v\n", withoutEmpty(strings.NewReader(jsonString)))
// output without omit: {Robert true 12345 0}
}
func withEmpty(r *strings.Reader) agentOmitEmpty {
dec := json.NewDecoder(r)
body := agentOmitEmpty{}
err := dec.Decode(&body)
if err != nil {
panic(err)
}
return body
}
func withoutEmpty(r *strings.Reader) agent {
dec := json.NewDecoder(r)
body := agent{}
err := dec.Decode(&body)
if err != nil {
panic(err)
}
return body
}
You need to define Empty as *int so it will be replaced with nil when there is no value. Then it will not be saved in the database.
How do I write the code below to get a string from my nested yaml struct?
Here is my yaml:
element:
- one:
url: http://test
nested: 123
- two:
url: http://test
nested: 123
weather:
- test:
zipcode: 12345
- ca:
zipcode: 90210
Here is example code
viper.SetConfigName("main_config")
viper.AddConfigPath(".")
err := viper.ReadInConfig()
if err != nil {
panic(err)
}
testvar := viper.GetString("element.one.url")
My problem:
I get a blank string when I print this. According to the docs, this is how you get a nested element. I suspect its not working because the elements are lists. Do I need to do a struct? I am not sure how to make one, especially if it needs to be nested.
You can unmarshal a nested configuration file.
main.go
package main
import (
"fmt"
"github.com/spf13/viper"
)
type NestedURL struct {
URL string `mapstructure:"url"`
Nested int `mapstructure:"nested"`
}
type ZipCode struct {
Zipcode string `mapstructure:"zipcode"`
}
type Config struct {
Element [] map[string]NestedURL `mapstructure:"element"`
Weather [] map[string]ZipCode `mapstructure:"weather"`
}
func main() {
viper.SetConfigName("config")
viper.AddConfigPath(".")
if err := viper.ReadInConfig(); err != nil {
return
}
var config Config
if err := viper.Unmarshal(&config); err != nil {
fmt.Println(err)
return
}
fmt.Println(config)
}
config.yml
element:
- one:
url: http://test
nested: 123
- two:
url: http://test
nested: 123
weather:
- test:
zipcode: 12345
- ca:
zipcode: 90210
There are different Get methods available in viper library and your YML structure is of type []map[string]string, so to parse your YML configuration file you have to use viper.Get method. So you have to parse your file something like this..
Note: You can use struct as well to un-marshal the data. Please refer this post mapping-nested-config-yaml-to-struct
package main
import (
"fmt"
"github.com/spf13/viper"
)
func main() {
viper.SetConfigName("config")
viper.AddConfigPath(".")
err := viper.ReadInConfig()
if err != nil {
panic(err)
}
testvar := viper.Get("element")
fmt.Println(testvar)
elementsMap := testvar.([]interface{})
for k, vmap := range elementsMap {
fmt.Print("Key: ", k)
fmt.Println(" Value: ", vmap)
eachElementsMap := vmap.(map[interface{}]interface{})
for k, vEachValMap := range eachElementsMap {
fmt.Printf("%v: %v \n", k, vEachValMap)
vEachValDataMap := vEachValMap.(map[interface{}]interface{})
for k, v := range vEachValDataMap {
fmt.Printf("%v: %v \n", k, v)
}
}
}
}
// Output:
/*
Key: 0 Value: map[one:map[url:http://test nested:123]]
one: map[url:http://test nested:123]
url: http://test
nested: 123
Key: 1 Value: map[two:map[url:http://test nested:123]]
two: map[url:http://test nested:123]
url: http://test
nested: 123
*/
You can use Unmarshal or UnmarshalKey to parse all or part of your data and fill a struct. It is very similar to unmarshaling a json.
In your case, code will be like this:
package main
import (
"fmt"
"github.com/spf13/viper"
)
// here we define schema of data, just like what we might do when we parse json
type Element struct {
Url string `mapstructure:"url"`
Nested int `mapstructure:"nested"`
}
func main() {
viper.SetConfigName("config")
viper.AddConfigPath(".")
err := viper.ReadInConfig()
if err != nil {
panic(err)
}
// data in `element` key is a map of string to Element. We define a variable to store data into it.
elementParsed := make(map[string]*Element)
// read the key `element` in the yaml file, and parse it's data and put it in `elementParsed` variable
err = viper.UnmarshalKey("element", &elementParsed)
if err != nil {
panic(err)
}
fmt.Println(elementParsed["one"].Url) // will print: http://test
fmt.Println(elementParsed["one"].Nested) // will print: 123
}
I am trying to use WordNik () to get random words for the dictionary for this script: https://github.com/jmagrippis/password
WordNik is outputting:
[{"id":7936915,"word":"Tanganyikan"},{"id":27180,"word":"cartographic"},{"id":48094,"word":"deterministic"},{"id":1485119,"word":"higher-risk"},{"id":120986,"word":"juristic"},{"id":1830806,"word":"magnetorheological"},{"id":320495,"word":"quelled"},{"id":324610,"word":"remoter"},{"id":215158,"word":"telemetric"},{"id":225207,"word":"uninquisitive"}]
Here is my code:
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"log"
"net/http"
"time"
"github.com/jmagrippis/password"
)
type Words struct {
id []int64
word []string
}
type GetWordsResponse struct {
WordList []Words
}
func getWords(speech string) (*GetWordsResponse, error) {
url := fmt.Sprintf("http://api.wordnik.com/v4/words.json/randomWords?hasDictionaryDef=false&includePartOfSpeech=%s&minCorpusCount=0&maxCorpusCount=-1&minDictionaryCount=1&maxDictionaryCount=-1&minLength=5&maxLength=-1&limit=10&api_key=api_key", speech)
res, err := http.Get(url)
if err != nil {
panic(err.Error())
}
body, err := ioutil.ReadAll(res.Body)
if err != nil {
panic(err.Error())
}
var s = new(GetWordsResponse)
var arr []string
_ = json.Unmarshal([]byte(body), &arr)
log.Printf("Unmarshaled: %v", arr)
return s, err
}
func main() {
dictionary := &password.Dictionary{
Adjectives: []string{"beautiful", "homely", "magical", "posh", "excellent"},
Subjects: []string{"mermaids", "unicorns", "lions", "piranhas"},
Verbs: []string{"love", "fancy", "eat", "bring", "fear", "aggravate"},
Adverbs: []string{"cuddling", "slapping", "shouting", "jumping"},
Objects: []string{"teddy-bears", "diamonds", "buckets", "boxes"},
}
generator := password.NewGenerator(dictionary, time.Now().UnixNano())
pass := generator.Generate()
fmt.Printf("%s", pass)
getWords("Verb")
}
As you can see, what I am trying to do is use WordNik API to request Adverbs, nouns, etc and then make a dictionary based off those words to generate a password. I am horrible with arrays and handling data.
As pointed out in the comments you need to export the fields so they can be unmarshaled
The encoding/json package relies on reflection and since it is in another package, it cannot access unexported fields. ( in go, fields, methods or functions that start with a small letter are unexported whereas with capital letters they are exported )
Then your example json does not contain the WordList at all, so what you want is unmarshaling directly into the array of Words. Also a words object only consists of Id and Word and not of arrays themselves.
type Words struct {
Id int64
Word string
}
func main() {
....
var words []Words
// you don't need to read the whole body first, you can decode in the same turn like this
err := json.NewDecoder(req.Body).Decode(&words)
if nil != err {
log.Fatal(err)
}
...
}
Another very important thing is that you should not ignore errors. This would have helped you to debug the issue. ( What I mean is _ = json.Unmarshal)
As for beginning with go, you could implement a simple test to see if your code works as intended.
https://play.golang.org/p/nuz9uXdka5S < check this working example for reference.
I am currently unable to unmarshall the data correctly from a map into a structure. Following is a code snippet (Brief Code at playground):
Request you to kindly provide the reason of getting default values on unmarshlling back the data.
package main
import (
"fmt"
"encoding/json"
"os"
)
func main() {
fmt.Println("Hello, playground")
type PDPOffer struct {
cart_value int `json:"cart_value"`
discount_amount_default int `json:"discount_amount_default"`
max_discount string `json:"max_discount"`
}
a:= map[string]interface{} {
"cart_value" : 1,
"max_discount" : 2,
}
var pdf PDPOffer
b, err := json.Marshal(a)
if err != nil {
fmt.Println("error:", err)
}
os.Stdout.Write(b)//working
err1 := json.Unmarshal(b, &pdf)
if err1 != nil {
fmt.Println("error:", err)
}
fmt.Printf("%+v", pdf)//displaying just the defualt values????????
}
json.Marshal and json.Unmarshal can only work on exported struct fields. Your fields aren't exported and aren't visible to the json code.
The reason for your unmarshal not working is that you need to expose fields of the struct and for doing that you need to start field name with Capital letters. Something has below:
type PDPOffer struct {
Cart_value int `json:"cart_value"`
Discount_amount_default int `json:"discount_amount_default"`
Max_discount string `json:"max_discount"`
}
Also, you were trying to marshal an int value into a string for "max_discount", you need to store it as a string in your map that you're marshaling:
a := map[string]interface{}{
"cart_value": 1,
"max_discount": "2",
}
The error handling had a bug checking for err1 != nil then printing err which was hiding the message error: json: cannot unmarshal number into Go value of type string
Working example with all the fixes: http://play.golang.org/p/L8VC-531nS