Get a value from a known key in a dynamic nested YAML - go

I'm pretty new to Golang. I have a YAML file with dynamic keys but only one is known, and it's not necessarily the first one (after config). There is no other key at the level of config.
The yaml file :
config:
foo:bar: baz
bar:foo: baz
abs:getit: myvalue
I want to retrieve myvalue from the nested key config:abs:getit. This nested key name will never change, it will always be config:abs:getit. All other keys can be whatever, we don't care, with different types of content (arrays, int, strings, array of array).
What is the best way to recover the value ?
I worked with yaml package, but I have to fix every field in a struct to unmarshall it, but I don't know how many nested keys there can be so I cannot write a struct which works all the time.
I worked with a map, but I can figure out which map I have to use, because if I can have a field with 6 nested keys or 3 nested keys with array in it before the value I'm searching and it will fails.
I am pretty lost with those kind of things in a dynamic context.
Ideally, I want to do a cat myFile.yaml | yq '.config."abs:getit"', but in Golang...
Any ideas and best practices to do that ?

You can do:
func main() {
var obj struct {
Config struct {
AbsGetit string `yaml:"abs:getit"`
} `yaml:"config"`
}
err := yaml.Unmarshal(data, &obj)
if err != nil {
panic(err)
}
fmt.Printf("%q\n", obj.Config.AbsGetit)
}
https://go.dev/play/p/KJ_lzZxaZBy

I think you need this code. Just put the correct path to your "myFile.yaml" file. In main() function, you will see two different examples of how to use the code according to your needs.
getConfVal finds a node of a YAML tree with an arbitrary sequence in N-dimensional depth. If the node does not exist, the value will be nil.
myFile.yaml
config:
foo:bar: baz
bar:foo: baz
abs:getit: myvalue
foo:
bar: "conf-foo-bar"
bar:
foo: "conf-bar-foo"
abs:
getit: "conf-abs-getit"
one:
two:
three:
four:
five: 5
five2: [4, 7]
package main
import (
"fmt"
"os"
yaml "gopkg.in/yaml.v3"
)
func main() {
if err := readTConf("./path/to/myFile.yaml", &cfg); err != nil {
fmt.Printf("Read YAML Conf: %v\n", err)
return
}
// e.g.
getConfVal(cfg, []string{"foo:bar"})
// getConfVal(cfg, []string{"foo", "bar"})
// getConfVal(cfg, []string{"four", "five"})
// getConfVal(cfg, []string{"four", "five2"})
fmt.Printf("\nThis is the result you are looking for. (%v)\n", needleRes)
}
var needleRes interface{}
var cfg map[string]interface{}
func getConfVal(o map[string]interface{}, ns []string) (map[string]interface{}, bool) {
nsCnt := len(ns)
for kn, vn := range ns {
for ko, vo := range o {
if fmt.Sprintf("%T", vo) == "map[string]interface {}" {
res, ok := getConfVal(vo.(map[string]interface{}), ns)
if ok {
return res, true
break
}
}
if fmt.Sprintf("%T", vo) == "string" {
if ko == vn {
if kn+1 == nsCnt {
needleRes = vo
return map[string]interface{}{}, true
}
}
}
}
}
return map[string]interface{}{}, false
}
func readTConf(f string, c *map[string]interface{}) error {
yamlFile, err := os.ReadFile(f)
if err != nil {
return err
}
if err := yaml.Unmarshal([]byte(yamlFile), &c); err != nil {
return err
}
return nil
}

Thank you for your precise answer. I'm sorry but there is an error in the question, and I apologize for the mistake.
It's not a single flow scalar Yaml but a map, since there is space before the value :
config:
foo:bar: baz
bar:foo: baz
abs:getit: myvalue
The code above logically returns a conversion error like this :
panic: yaml: unmarshal errors:
line 2: cannot unmarshal !!map into string
My whole code is here. The file I read is a Pulumi config Yaml, which will be different for all projects, except for one common key ("abs:getit:"), only the value is different.
The original question file has been modified. Really sorry for that...

Related

Map seems to drop values in recursion

I've been working on a problem and I figured I would demonstrate it using a pokemon setup. I am reading from a file, parsing the file and creating objects/structs from them. This normally isn't a problem except now I need to implement interface like inheriting of traits. I don't want there to be duplicate skills in there so I figured I could use a map to replicate a set data structure. However it seems that in the transitive phase of my recursive parsePokemonFile function (see the implementsComponent case), I appear to be losing values in my map.
I am using the inputs like such:
4 files
Ratatta:
name=Ratatta
skills=Tackle:normal,Scratch:normal
Bulbosaur:
name=Bulbosaur
implements=Ratatta
skills=VineWhip:leaf
Oddish:
name=Oddish
implements=Ratatatt
skills=Acid:poison
Venosaur:
name=Venosaur
implements=bulbosaur,oddish
I'm expecting the output for the following code to be something like
Begin!
{Venosaur [{VineWhip leaf} {Acid poison} {Tackle normal} {Scratch normal}]}
but instead I get
Begin!
{Venosaur [{VineWhip leaf} {Acid poison}]}
What am I doing wrong? Could it be a logic error? Or am I making an assumption about the map holding values that I shouldn't?
package main
import (
"bufio"
"fmt"
"os"
"strings"
)
// In order to create a set of pokemon abilities and for ease of creation and lack of space being taken up
// We create an interfacer capability that imports the skills and attacks from pokemon of their previous evolution
// This reduces the amount of typing of skills we have to do.
// Algorithm is simple. Look for the name "implements=x" and then add x into set.
// Unfortunately it appears that the set is dropping values on transitive implements interfaces
func main() {
fmt.Println("Begin!")
dex, err := parsePokemonFile("Venosaur")
if err != nil {
fmt.Printf("Got error: %v\n", err)
}
fmt.Printf("%v\n", dex)
}
type pokemon struct {
Name string
Skills []skill
}
type skill struct {
SkillName string
Type string
}
func parsePokemonFile(filename string) (pokemon, error) {
file, err := os.Open(filename)
if err != nil {
return pokemon{}, err
}
defer file.Close()
scanner := bufio.NewScanner(file)
var builtPokemon pokemon
for scanner.Scan() {
component, returned := parseLine(scanner.Text())
switch component {
case nameComponent:
builtPokemon.Name = returned
case skillsComponent:
skillsStrings := strings.Split(returned, ",")
var skillsArr []skill
// split skills and add them into pokemon skillset
for _, skillStr := range skillsStrings {
skillPair := strings.Split(skillStr, ":")
skillsArr = append(skillsArr, skill{SkillName: skillPair[0], Type: skillPair[1]})
}
builtPokemon.Skills = append(builtPokemon.Skills, skillsArr...)
case implementsComponent:
implementsArr := strings.Split(returned, ",")
// create set to remove duplicates
skillsSet := make(map[*skill]bool)
for _, val := range implementsArr {
// recursively call the pokemon files and get full pokemon
implementedPokemon, err := parsePokemonFile(val)
if err != nil {
return pokemon{}, err
}
// sieve out the skills into a set
for _, skill := range implementedPokemon.Skills {
skillsSet[&skill] = true
}
}
// append final set into the currently being built pokemon
for x := range skillsSet {
builtPokemon.Skills = append(builtPokemon.Skills, *x)
}
}
}
return builtPokemon, nil
}
type component int
// components to denote where to put our strings when it comes time to assemble what we've parsed
const (
nameComponent component = iota
implementsComponent
skillsComponent
)
func parseLine(line string) (component, string) {
arr := strings.Split(line, "=")
switch arr[0] {
case "name":
return nameComponent, arr[1]
case "implements":
return implementsComponent, arr[1]
case "skills":
return skillsComponent, arr[1]
default:
panic("Invalid field found")
}
}
This has nothing to do with Golang maps dropping any values.
The problem is that you are using a map of skill pointers and not skills. Two pointers to the same skill content can be different.
skillsSet := make(map[*skill]bool)
If you change this to map[skill]bool, this should work. You may try it out!

how to insert an array in sqlite?

I have struct like:
type Foo struct {
bars []string
}
Since sqlite3 doesn't have array data type supported, can we store []string as string and while retrieving return as slice of string? Was trying to implement like below, but getting error because of type mismatch. What need to be done here?
Edit: I have changed the code and look like working
type strArray []string
func (strarr StrArray) Value() (driver.Value, error) {
if strarr != nil {
resarr := strings.Join(strarr, "")
return resarr, nil
}
return nil, nil
}
Complementary to database/sql/driver.Valuer you need also to implement database/sql.Scanner for reading your type from the database.
When you think of how to implement it, it's obvious that in Valuer you should Join your slice with some delimiter character/string (not occurring in the data of course) to be able to Split it back when retrieving.
Assuming that such delimiter would be ; (my wild guess), the code for reading would look like:
func (a *strArray) Scan(value interface{}) error {
if value == nil {
return nil // case when value from the db was NULL
}
s, ok := value.(string)
if !ok {
return fmt.Errorf("failed to cast value to string: %v", value)
}
*a = strings.Split(s, ";")
return nil
}
For writing, you'd need to use strings.Join(strarr, ";") in Valuer implementation.
Other less-trivial implementation would require marshaling your slice and encoding the resulting bytes as string somehow (base32/64? json?). In any case you need to not loose the information what are distinct slice elements when saving them to the database.

how to access nested Json key values in Golang

Team,
new to Programming.
I have data available after unmarshaling the Json as shown below, which has nested Key values. flat key values I am able to access, how do I access nested key values.
Here is the byte slice data shown below after unmarshaling —>
tables:[map[name:basic__snatpool_members] map[name:net__snatpool_members] map[name:optimizations__hosts] map[columnNames:[name] name:pool__hosts rows:[map[row:[ry.hj.com]]]] traffic_group:/Common/traffic-group-1
Flat key values I am able to access by using the following code
p.TrafficGroup = m[“traffic_group”].(string)
here is the complete function
func dataToIapp(name string, d *schema.ResourceData) bigip.Iapp {
var p bigip.Iapp
var obj interface{}
jsonblob := []byte(d.Get("jsonfile").(string))
err := json.Unmarshal(jsonblob, &obj)
if err != nil {
fmt.Println("error", err)
}
m := obj.(map[string]interface{}) // Important: to access property
p.Name = m[“name”].(string)
p.Partition = m[“partition”].(string)
p.InheritedDevicegroup = m[“inherited_devicegroup”].(string)
}
Note: This may not work with your JSON structure. I inferred what it would be based on your question but without the actual structure, I cannot guarantee this to work without modification.
If you want to access them in a map, you need to assert that the interface pulled from the first map is actually a map. So you would need to do this:
tmp := m["tables"]
tables, ok := tmp.(map[string]string)
if !ok {
//error handling here
}
r.Name = tables["name"].(string)
But instead of accessing the unmarshaled JSON as a map[string]interface{}, why don't you create structs that match your JSON output?
type JSONRoot struct {
Name string `json:"name"`
Partition string `json:"partition"`
InheritedDevicegroup string `json:"inherited_devicegroup"`
Tables map[string]string `json:"tables"` //Ideally, this would be a map of structs
}
Then in your code:
func dataToIapp(name string, d *schema.ResourceData) bigip.Iapp {
var p bigip.Iapp
var obj &JSONRoot{}
jsonblob := []byte(d.Get("jsonfile").(string))
err := json.Unmarshal(jsonblob, &obj)
if err != nil {
fmt.Println("error", err)
}
p.Name = obj.Name
p.Partition = obj.Partition
p.InheritedDevicegroup = obj.InheritedDevicegroup
p.Name = obj.Tables["name"]
}
JSON objects are unmarshaled into map[string]interface{}, JSON arrays into []interface{}, same applies for nested objects/arrays.
So for example if a key/index maps to a nested object you need to type assert the value to map[string]interface{} and if the key/index maps to an array of objects you first need to assert the value to []interface{} and then each element to map[string]interface{}.
e.g. (for brevity this code is not guarding against panic)
tables := obj.(map[string]interface{})["tables"]
table1 := tables.([]interface{})[0]
name := table1.(map[string]interface{})["name"]
namestr := name.(string)
However, if it's the case that the json you are parsing is not dynamic but instead has a specific structure you should define a struct type that mirrors that structure and unmarshal the json into that.
All you have to do is repeatedly accessing the map via type-switching or assertion:
for _, table := range m["tables"] {
switch val := table {
case string:
fmt.Println("table is string")
case int:
fmt.Println("table is integer")
// This is your case, since JSON is unmarshaled to type []interface{} and map[string]interface{}
case []interface{}:
fmt.Println("table is a slice of interface{}")
for _, tb := range value {
if m, ok := tb.(map[string]interface{}); ok {
// Now it's accessible
fmt.Println(m["name"])
}
}
default:
fmt.Println("unknown type")
}
}
You might want to handle errors better than this.
To read more, check out my writing from a while ago https://medium.com/code-zen/dynamically-creating-instances-from-key-value-pair-map-and-json-in-go-feef83ab9db2.

Is there a way to write generic code to find out whether a slice contains specific element in Go?

I want to know is there a generic way to write code to judge whether a slice contains an element, I find it will frequently useful since there is a lot of logic to fist judge whether specific elem is already in a slice and then decide what to do next. But there seemed not a built-in method for that(For God's sake, why?)
I try to use interface{} to do that like:
func sliceContains(slice []interface{}, elem interface{}) bool {
for _, item := range slice {
if item == elem {
return true
}
}
return false
}
I thought interface{} is sort of like Object of Java, but apparently, I was wrong. Should I write this every time meet with a new struct of slice? Isn't there a generic way to do this?
You can do it with reflect, but it will be MUCH SLOWER than a non-generic equivalent function:
func Contains(slice, elem interface{}) bool {
sv := reflect.ValueOf(slice)
// Check that slice is actually a slice/array.
// you might want to return an error here
if sv.Kind() != reflect.Slice && sv.Kind() != reflect.Array {
return false
}
// iterate the slice
for i := 0; i < sv.Len(); i++ {
// compare elem to the current slice element
if elem == sv.Index(i).Interface() {
return true
}
}
// nothing found
return false
}
func main(){
si := []int {3, 4, 5, 10, 11}
ss := []string {"hello", "world", "foo", "bar"}
fmt.Println(Contains(si, 3))
fmt.Println(Contains(si, 100))
fmt.Println(Contains(ss, "hello"))
fmt.Println(Contains(ss, "baz"))
}
How much slower? about x50-x60 slower:
Benchmarking against a non generic function of the form:
func ContainsNonGeneic(slice []int, elem int) bool {
for _, i := range slice {
if i == elem {
return true
}
}
return false
}
I'm getting:
Generic: N=100000, running time: 73.023214ms 730.23214 ns/op
Non Generic: N=100000, running time: 1.315262ms 13.15262 ns/op
You can make it using the reflect package like that:
func In(s, e interface{}) bool {
slice, elem := reflect.ValueOf(s), reflect.ValueOf(e)
for i := 0; i < slice.Len(); i++ {
if reflect.DeepEqual(slice.Index(i).Interface(), elem.Interface()) {
return true
}
}
return false
}
Playground examples: http://play.golang.org/p/TQrmwIk6B4
Alternatively, you can:
define an interface and make your slices implement it
use maps instead of slices
just write a simple for loop
What way to choose depends on the problem you are solving.
I'm not sure what your specific context is, but you'll probably want to use a map to check if something already exists.
package main
import "fmt"
type PublicClassObjectBuilderFactoryStructure struct {
Tee string
Hee string
}
func main() {
// Empty structs occupy zero bytes.
mymap := map[interface{}]struct{}{}
one := PublicClassObjectBuilderFactoryStructure{Tee: "hi", Hee: "hey"}
two := PublicClassObjectBuilderFactoryStructure{Tee: "hola", Hee: "oye"}
three := PublicClassObjectBuilderFactoryStructure{Tee: "hi", Hee: "again"}
mymap[one] = struct{}{}
mymap[two] = struct{}{}
// The underscore is ignoring the value, which is an empty struct.
if _, exists := mymap[one]; exists {
fmt.Println("one exists")
}
if _, exists := mymap[two]; exists {
fmt.Println("two exists")
}
if _, exists := mymap[three]; exists {
fmt.Println("three exists")
}
}
Another advantage of using maps instead of a slice is that there is a built-in delete function for maps. https://play.golang.org/p/dmSyyryyS8
If you want a rather different solution, you might try the code-generator approach offered by tools such as Gen. Gen writes source code for each concrete class you want to hold in a slice, so it supports type-safe slices that let you search for the first match of an element.
(Gen also offers a few other kinds of collection and allows you to write your own.)

How to use .Key in go text/template usage

I don't understand the document paragraph in http://golang.org/pkg/text/template/
- The name of a key of the data, which must be a map, preceded
by a period, such as
.Key
The result is the map element value indexed by the key.
Key invocations may be chained and combined with fields to any
depth:
.Field1.Key1.Field2.Key2
Although the key must be an alphanumeric identifier, unlike with
field names they do not need to start with an upper case letter.
Keys can also be evaluated on variables, including chaining:
$x.key1.key2
Here is my test code, but failed. code: http://play.golang.org/p/lbLJ4yoL2T.
var season = map[int]string{1: "spring", 2: "summer",
3: "autumn", 4: "winter"}
func main() {
const greeting = `Welcome, {{.Key}}`
t := template.Must(template.New("greet").Parse(greeting))
err := t.Execute(os.Stdout, season)
if err != nil {
fmt.Println(err)
}
}
Output
Welcome, template: greet:1:11: executing "greet" at <.Key>: can't evaluate field Key in type map[int]string
I'd assume "Key" is the key (as in key/value) of the map. Also, the map keys can't be an int to use it in the template like that. So instead of {{.Key}} try {{.a}} as shown in this fork of your playground:
var season = map[string]string{"a": "spring", "b": "summer",
"c": "autumn", "d": "winter"}
func main() {
const greeting = `Welcome, {{.a}}`
t := template.Must(template.New("greet").Parse(greeting))
err := t.Execute(os.Stdout, season)
if err != nil {
fmt.Println(err)
}
}
Output:
Welcome, spring

Resources