Unmarshal GO YAML to either a Map or a String - go

I'm trying to unmarshal YAML entries that can be either a string or a list of key: value strings (a map as per Go). I cannot figure out how to get this done sadly. I know I can write my own unmarshaller but that seems to only work with structs.
I have the first part working:
package main
import (
"log"
"gopkg.in/yaml.v2"
)
type Data struct {
Entry []Entry `yaml:"entries"`
}
type Entry map[string]string
var dat string = `
entries:
- keya1: val1
keya2: val2
- keyb1: val1
keyb2: val2
- val3`
func main() {
out := Data{}
if err := yaml.Unmarshal([]byte(dat), &out); err != nil {
log.Fatal(err)
}
log.Printf("%+v", out)
}
But the - val3 entry causes an error now, obviously. How can I get it to recognise both lists and single string entries?
Thank you

This is just a followup to the excellent #Benjamin Kadish answer above, but here's a somewhat more complete version and this uses yaml.v3, which makes it just a bit more obvious. Note that the type of the unmarshalled items is map[string]interface{} instead of map[interface{}]interface{} in yaml v3.
package main
import (
"gopkg.in/yaml.v3"
"log"
)
type Data struct {
Entry []Entry `yaml:"entries"`
}
type Entry interface {}
var dat string = `
entries:
- keya1: val1
keya2: val2
- keyb1: val1
keyb2: val2
- val3`
func main() {
out := Data{}
if err := yaml.Unmarshal([]byte(dat), &out); err != nil {
log.Fatal(err)
}
for _, entry := range out.Entry {
switch i := entry.(type) {
case string:
log.Printf("i is a string: %+v\n", i)
case map[string]interface{}:
log.Printf("i is a map.")
for k,v := range i {
log.Printf("%s=%v\n",k,v)
}
default:
log.Printf("Type i=%s", i)
}
}
}

This has been answered in various ways before, but long story short it is easy unmarshall into an interface and then deal with both cases
type Entry interface{}
for _, entry := range out.Entry {
switch i := entry.(type) {
case string:
log.Printf("i is a string %+v\n", i)
case map[interface{}]interface{}:
log.Printf("i is a map %+v\n", i)
}
}

As, go is static type language, you can't leave val3 only a list item if you convert it to defined struct. It should be key value pair. E.g. (val3: "") (if you want it empty.)
Here is modified code
package main
import (
"log"
"gopkg.in/yaml.v2"
)
type Data struct {
Entry []Entry `yaml:"entries"`
}
type Entry map[string]string
var dat string = `
entries:
- keya1: val1
keya2: val2
- keyb1: val1
keyb2: val2
- val3: ""`
func main() {
out := Data{}
if err := yaml.Unmarshal([]byte(dat), &out); err != nil {
log.Fatal(err)
}
log.Printf("%+v", out)
}
And output:
2018/02/02 01:07:36 {Entry:[map[keya1:val1 keya2:val2] map[keyb2:val2 keyb1:val1] map[val3:]]}

Related

Golang comparing two yaml files and updating them

Im a newbie in golang. I am trying to compare two yaml files and update the 2nd file's value if there is any new value in 1st yaml for that particular key.
So the files are of format: These are sample yaml files. Real yaml files have much more nested complicated maps with different datatypes for each key.
1st yaml:
name: john
city: washington
2nd yaml:
name: peter
city: washington
Final result for 2nd yaml file should be:
name: john
city: washington
Tried creating a map string interface for both yaml files using unmarshal. But having trouble how to compare both maps. Was trying to loop over each key of map and search for that key in 2nd yaml map. If key exists update the value in 2nd yaml map. But i am not able to implement that. Any suggestions/better ideas?
Edit: Updated code
package main
import (
"fmt"
"io/ioutil"
"log"
"github.com/imdario/mergo"
"gopkg.in/yaml.v3"
)
func main() {
yfile, err := ioutil.ReadFile("C:/Users/212764682/lifecycle/userconfig.yaml")
if err != nil {
log.Fatal(err)
}
data := make(map[string]interface{})
err2 := yaml.Unmarshal(yfile, &data)
if err2 != nil {
log.Fatal(err2)
} else {
yfile1, err3 := ioutil.ReadFile("C:/Users/212764682/lifecycle/conf.yaml")
yfile2, err4 := ioutil.ReadFile("C:/Users/212764682/lifecycle/prof.yaml")
if err3 != nil && err4 != nil {
log.Fatal(err3)
log.Fatal(err4)
} else {
dat := make(map[string]interface{})
dat2 := make(map[string]interface{})
err5 := yaml.Unmarshal(yfile1, &dat)
err6 := yaml.Unmarshal(yfile2, &dat2)
_ = err5
_ = err6
for key1, element1 := range data {
for key2, element2 := range dat {
if key1 == key2 {
if element1 == element2 {
} else {
element2 = element1
}
} else {
dat[key1] = data[key1]
}
}
}
}
}
}
So im want to compare each key of data with dat. If that key exists in dat, check for value in data. If value different in dat, update with value of data in dat for that key. Also, if any key of data dosent exist in dat, then append that key in dat. But not able to implement it correctly.
You can try to compare the map and then update it if the key exists. Here's some example using your case.
package main
import (
"fmt"
"io/ioutil"
"log"
"gopkg.in/yaml.v3"
)
func main() {
// read file
yfile1, err := ioutil.ReadFile("file1.yaml")
if err != nil {
log.Fatal(err)
return
}
yfile2, err := ioutil.ReadFile("file2.yaml")
if err != nil {
log.Fatal(err)
return
}
// unmarshal ymal
data1 := make(map[string]interface{})
if err := yaml.Unmarshal(yfile1, &data1); err != nil {
log.Fatal(err)
return
}
data2 := make(map[string]interface{})
if err := yaml.Unmarshal(yfile2, &data2); err != nil {
log.Fatal(err)
return
}
// from this we can iterate the key in data2 then check whether it exists in data1
// if so then we can update the value in data2
// iterate key in data2
for key2 := range data2 {
// check whether key2 exists in data1
if val1, ok := data1[key2]; ok {
// update the value of key2 in data2
data2[key2] = val1
}
}
fmt.Printf("data2: %v", data2)
// output:
// data2: map[city:washington name:john]
// you can write the data2 into ymal
newYfile2, err := yaml.Marshal(&data2)
if err != nil {
log.Fatal(err)
return
}
// write to file
if err = ioutil.WriteFile("new_file2.yaml", newYfile2, 0644); err != nil {
log.Fatal(err)
return
}
}
Inside new_file2.yaml will be like this:
city: washington
name: john
One thing that you need to take a not is that map in Go doesn't maintain the order (AFAIK Go doesn't have built-in OrderedMap type per 7 May 2022) so the order key in the new file will be random
Additional note: for error handling, you better handle it right away (right after you got the error). Here's a good article about it Error handling and Go
There is a maps package, since 1.18 I believe. If you don't care about new keys being added to the destination map you can use its copy function.
func Copy[M ~map[K]V, K comparable, V any](dst, src M)
Copy copies all key/value pairs in src adding them to dst. When a key in src is already present in dst, the value in dst will be overwritten by the value associated with the key in src.
The source code of that function is very simple:
func Copy[M ~map[K]V, K comparable, V any](dst, src M) {
for k, v := range src {
dst[k] = v
}
}
You could also do it yourself. The below code is from the helm source code. Unlike the copy function above, it works recursivly:
func mergeMaps(a, b map[string]interface{}) map[string]interface{} {
out := make(map[string]interface{}, len(a))
for k, v := range a {
out[k] = v
}
for k, v := range b {
if v, ok := v.(map[string]interface{}); ok {
if bv, ok := out[k]; ok {
if bv, ok := bv.(map[string]interface{}); ok {
out[k] = mergeMaps(bv, v)
continue
}
}
}
out[k] = v
}
return out
}

How to parse a JSON string returned from scanner.Text() [duplicate]

Objects like the below can be parsed quite easily using the encoding/json package.
[
{"something":"foo"},
{"something-else":"bar"}
]
The trouble I am facing is when there are multiple dicts returned from the server like this :
{"something":"foo"}
{"something-else":"bar"}
This can't be parsed using the code below.
correct_format := strings.Replace(string(resp_body), "}{", "},{", -1)
json_output := "[" + correct_format + "]"
I am trying to parse Common Crawl data (see example).
How can I do this?
Assuming your input is really a series of valid JSON documents, use a json.Decoder to decode them:
package main
import (
"encoding/json"
"fmt"
"io"
"log"
"strings"
)
var input = `
{"foo": "bar"}
{"foo": "baz"}
`
type Doc struct {
Foo string
}
func main() {
dec := json.NewDecoder(strings.NewReader(input))
for {
var doc Doc
err := dec.Decode(&doc)
if err == io.EOF {
// all done
break
}
if err != nil {
log.Fatal(err)
}
fmt.Printf("%+v\n", doc)
}
}
Playground: https://play.golang.org/p/ANx8MoMC0yq
If your input really is what you've shown in the question, that's not JSON and you have to write your own parser.
Seems like each line is its own json object.
You may get away with the following code which will structure this output into correct json:
package main
import (
"fmt"
"strings"
)
func main() {
base := `{"trolo":"lolo"}
{"trolo2":"lolo2"}`
delimited := strings.Replace(base, "\n", ",", -1)
final := "[" + delimited + "]"
fmt.Println(final)
}
You should be able to use encoding/json library on final now.
Another option would be to parse each incoming line, line by line, and then add each one to a collection in code (ie a slice) Go provides a line scanner for this.
yourCollection := []yourObject{}
scanner := bufio.NewScanner(YOUR_SOURCE)
for scanner.Scan() {
obj, err := PARSE_JSON_INTO_yourObject(scanner.Text())
if err != nil {
// something
}
yourCollection = append(yourCollection, obj)
}
if err := scanner.Err(); err != nil {
fmt.Fprintln(os.Stderr, "reading standard input:", err)
}
You can read the ndjson from the file row by row and parse it then apply the logical operations on it. In the below sample instead of reading from the file, I have used an Array of JSON string.
import (
"encoding/json"
"fmt"
)
type NestedObject struct {
D string
E string
}
type OuterObject struct {
A string
B string
C []NestedObject
}
func main() {
myJsonString := []string{`{"A":"1","B":"2","C":[{"D":"100","E":"10"}]}`, `{"A":"11","B":"21","C":[{"D":"1001","E":"101"}]}`}
for index, each := range myJsonString {
fmt.Printf("Index value [%d] is [%v]\n", index, each)
var obj OuterObject
json.Unmarshal([]byte(each), &obj)
fmt.Printf("a: %v, b: %v, c: %v", obj.A, obj.B, obj.C)
fmt.Println()
}
}
Output:
Index value [0] is [{"A":"1","B":"2","C":[{"D":"100","E":"10"}]}]
a: 1, b: 2, c: [{100 10}]
Index value [1] is [{"A":"11","B":"21","C":[{"D":"1001","E":"101"}]}]
a: 11, b: 21, c: [{1001 101}]
Try it on golang play

Copying field/value from src to dest object

I am trying to copy the fields from one struct value to another, where they have the same field definitions. I have this program:
package main
import (
"log"
"reflect"
)
func setExistingFields(src interface{}, dst interface{}) {
fields := reflect.TypeOf(src)
values := reflect.ValueOf(src)
num := fields.NumField()
s := reflect.ValueOf(src).Elem()
d := reflect.ValueOf(dst).Elem()
for i := 0; i < num; i++ {
field := fields.Field(i)
value := values.Field(i)
fsrc := s.FieldByName(field.Name)
fdest := d.FieldByName(field.Name)
if fdest.IsValid() && fsrc.IsValid() {
if fdest.CanSet() && fsrc.CanSet() {
fdest.Set(value)
}
}
}
}
// and then we main:
func main() {
src := struct {
Foo string
Bar string
}{
"dog",
"pony",
}
dest := struct{ Foo string; Bar string }{}
setExistingFields(&src, &dest)
log.Println("dest.Foo", dest.Foo)
}
I run that, but I get an error:
reflect: NumField of non-struct type
I can't figure out what that's about.
Here's a playground link:
https://play.golang.org/p/TsHTfAaeKhc
Try this out:
func setExistingFields(src interface{}, dst interface{}) {
srcFields := reflect.TypeOf(src).Elem()
srcValues := reflect.ValueOf(src).Elem()
dstValues := reflect.ValueOf(dst).Elem()
for i := 0; i < srcFields.NumField(); i++ {
srcField := srcFields.Field(i)
srcValue := srcValues.Field(i)
dstValue := dstValues.FieldByName(srcField.Name)
if dstValue.IsValid() {
if dstValue.CanSet() {
dstValue.Set(srcValue)
}
}
}
}
Note that you need to do additional checking if src field value is assignable to dst field type.
Edit: The reason why you are getting that error is because fields at that point is a pointer to a struct. You need to get the actual struct value by using Elem().
This won't work: A struct always gets its "schema" (eg. its fields) during compile time... You cannot add more fields during runtime.
I don't see what your exact use case is, but consider something like map[string]string or even map[string]interface{} to be able to "extend" the content/fields of the thing you are passing around...

How to convert string from interface to []string in golang?

I'm parsing a JSON object which contains an array of strings :
var ii interface{}
json := "{\"aString\": [\"aaa_111\", \"bbb_222\"], \"whatever\":\"ccc\"}"
err := json.Unmarshal([]byte(json), &ii)
if err != nil {
log.Fatal(err)
}
data := ii.(map[string]interface{})
fmt.Println(data["aString"]) // outputs: ["aaa_111" "bbb_222"]
I tried to convert data["aString"] to []string to be able to loop over it, but it fails :
test := []string(data["aString"]).([]string)
fmt.Println(test) // panic -> interface conversion:
// interface is string, not []string
How can I convert data["aString"] ?
edit:
I didn't express myself properly. If I print data, I have such map :
map[aString:["BBB-222","AAA-111"] whatever:ccc]
I want to loop over aString (to manipule each array entry). But I can't find how, because aString is type interface {} :
for i, v := range aString { // <-- fails
// ...
fmt.Println(i, v)
}
That's why I want to convert aString. I don't want to convert a string which looks like an array to an array.
I recommend you move away from this implementation in general. Your json may vary but you can easily use objects and avoid all this type unsafe nonsense.
Anyway, that conversion doesn't work because the types inside the slice are not string, they're also interface{}. You have to iterate the collection then do a type assertion on each item like so:
aInterface := data["aString"].([]interface{})
aString := make([]string, len(aInterface))
for i, v := range aInterface {
aString[i] = v.(string)
}
Is it what you need?
package main
import (
"fmt"
"encoding/json"
)
func main() {
js := "{\"aString\": [\"aaa_111\", \"bbb_222\"], \"whatever\":\"ccc\"}"
a := make(map[string]interface{})
json.Unmarshal([]byte(js), &a)
for _, v := range a["aString"].([]interface{}) {
str := v.(string)
fmt.Println(str)
}
}
Check on Go Playground
For another approach, you can use a struct instead:
package main
import (
"encoding/json"
"fmt"
)
func main() {
s := []byte(`{"aString": ["aaa_111", "bbb_222"], "whatever":"ccc"}`)
var t struct {
Astring []string
Whatever string
}
json.Unmarshal(s, &t)
fmt.Printf("%+v\n", t) // {Astring:[aaa_111 bbb_222] Whatever:ccc}
}

Get a simple string representation of a struct field’s type

Using Go’s ast package, I am looping over a struct’s field list like so:
type Thing struct {
Field1 string
Field2 []int
Field3 map[byte]float64
}
// typ is a *ast.StructType representing the above
for _, fld := range typ.Fields.List {
// get fld.Type as string
}
…and would like to get a simple string representation of fld.Type, as it appears in the source code, e.g. []int or map[byte]float64.
The ast package field type Type property is an Expr, so I’ve found myself getting off into the weeds using type switches and handling every type specifically – when my only goal is to get out the plain string to the right of each field name, which seems like it should be simpler.
Is there a simple way?
There are two things you could be getting at here, one is the type of an expression as would ultimately be resolved during compilation and the other is the code which would determine that type.
Digging through the docs, I don't believe the first is at all available. You can get at the later, however, by using End() and Pos() on Node.
Quick example program:
package main
import (
"fmt"
"go/ast"
"go/parser"
"go/token"
)
func main() {
src := `
package foo
type Thing struct {
Field1 string
Field2 []int
Field3 map[byte]float64
}`
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, "", src, 0)
if err != nil {
panic(err)
}
// hard coding looking these up
typeDecl := f.Decls[0].(*ast.GenDecl)
structDecl := typeDecl.Specs[0].(*ast.TypeSpec).Type.(*ast.StructType)
fields := structDecl.Fields.List
for _, field := range fields {
typeExpr := field.Type
start := typeExpr.Pos() - 1
end := typeExpr.End() - 1
// grab it in source
typeInSource := src[start:end]
fmt.Println(typeInSource)
}
}
This prints:
string
[]int
map[byte]float64
I through this together in the golang playground, if you want to mess with it.
You can use go/types ExprString
This works with complicated types like []string, []map[string]string, etc.
import (
...
"go/types"
...
)
...
// typ is a *ast.StructType representing the above
for _, fld := range typ.Fields.List {
...
typeExpr := fld.Type
typeString := types.ExprString(typeExpr)
...
}
https://golang.org/src/go/types/exprstring.go
This is exactly what Fprint in the go/printer package is for. It takes any AST node as an argument and writes its string representation to a io.Writer.
You can use it in your example as follows:
package main
import (
"bytes"
"fmt"
"go/ast"
"go/parser"
"go/printer"
"go/token"
"log"
)
func main() {
src := `
package foo
type Thing struct {
Field1 string
Field2 []int
Field3 map[byte]float64
}`
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, "", src, 0)
if err != nil {
panic(err)
}
typeDecl := f.Decls[0].(*ast.GenDecl)
structDecl := typeDecl.Specs[0].(*ast.TypeSpec).Type.(*ast.StructType)
for i, fld := range structDecl.Fields.List {
// get fld.Type as string
var typeNameBuf bytes.Buffer
err := printer.Fprint(&typeNameBuf, fset, fld.Type)
if err != nil {
log.Fatalf("failed printing %s", err)
}
fmt.Printf("field %d has type %q\n", i, typeNameBuf.String())
}
}
Output:
field 0 has type "string"
field 1 has type "[]int"
field 2 has type "map[byte]float64"
Try it in playground: https://play.golang.org/p/cyrCLt_JEzQ
I found a way to do this without using the original source code as a reference for simple members (not slices, arrays or structs):
for _, field := range fields {
switch field.Type.(type) {
case *ast.Ident:
stype := field.Type.(*ast.Ident).Name // The type as a string
tag = ""
if field.Tag != nil {
tag = field.Tag.Value //the tag as a string
}
name := field.Names[0].Name //name as a string
...
For the non-simple members you just need another case statement (IE: case *ast.ArrayType:).

Resources