Map seems to drop values in recursion - go

I've been working on a problem and I figured I would demonstrate it using a pokemon setup. I am reading from a file, parsing the file and creating objects/structs from them. This normally isn't a problem except now I need to implement interface like inheriting of traits. I don't want there to be duplicate skills in there so I figured I could use a map to replicate a set data structure. However it seems that in the transitive phase of my recursive parsePokemonFile function (see the implementsComponent case), I appear to be losing values in my map.
I am using the inputs like such:
4 files
Ratatta:
name=Ratatta
skills=Tackle:normal,Scratch:normal
Bulbosaur:
name=Bulbosaur
implements=Ratatta
skills=VineWhip:leaf
Oddish:
name=Oddish
implements=Ratatatt
skills=Acid:poison
Venosaur:
name=Venosaur
implements=bulbosaur,oddish
I'm expecting the output for the following code to be something like
Begin!
{Venosaur [{VineWhip leaf} {Acid poison} {Tackle normal} {Scratch normal}]}
but instead I get
Begin!
{Venosaur [{VineWhip leaf} {Acid poison}]}
What am I doing wrong? Could it be a logic error? Or am I making an assumption about the map holding values that I shouldn't?
package main
import (
"bufio"
"fmt"
"os"
"strings"
)
// In order to create a set of pokemon abilities and for ease of creation and lack of space being taken up
// We create an interfacer capability that imports the skills and attacks from pokemon of their previous evolution
// This reduces the amount of typing of skills we have to do.
// Algorithm is simple. Look for the name "implements=x" and then add x into set.
// Unfortunately it appears that the set is dropping values on transitive implements interfaces
func main() {
fmt.Println("Begin!")
dex, err := parsePokemonFile("Venosaur")
if err != nil {
fmt.Printf("Got error: %v\n", err)
}
fmt.Printf("%v\n", dex)
}
type pokemon struct {
Name string
Skills []skill
}
type skill struct {
SkillName string
Type string
}
func parsePokemonFile(filename string) (pokemon, error) {
file, err := os.Open(filename)
if err != nil {
return pokemon{}, err
}
defer file.Close()
scanner := bufio.NewScanner(file)
var builtPokemon pokemon
for scanner.Scan() {
component, returned := parseLine(scanner.Text())
switch component {
case nameComponent:
builtPokemon.Name = returned
case skillsComponent:
skillsStrings := strings.Split(returned, ",")
var skillsArr []skill
// split skills and add them into pokemon skillset
for _, skillStr := range skillsStrings {
skillPair := strings.Split(skillStr, ":")
skillsArr = append(skillsArr, skill{SkillName: skillPair[0], Type: skillPair[1]})
}
builtPokemon.Skills = append(builtPokemon.Skills, skillsArr...)
case implementsComponent:
implementsArr := strings.Split(returned, ",")
// create set to remove duplicates
skillsSet := make(map[*skill]bool)
for _, val := range implementsArr {
// recursively call the pokemon files and get full pokemon
implementedPokemon, err := parsePokemonFile(val)
if err != nil {
return pokemon{}, err
}
// sieve out the skills into a set
for _, skill := range implementedPokemon.Skills {
skillsSet[&skill] = true
}
}
// append final set into the currently being built pokemon
for x := range skillsSet {
builtPokemon.Skills = append(builtPokemon.Skills, *x)
}
}
}
return builtPokemon, nil
}
type component int
// components to denote where to put our strings when it comes time to assemble what we've parsed
const (
nameComponent component = iota
implementsComponent
skillsComponent
)
func parseLine(line string) (component, string) {
arr := strings.Split(line, "=")
switch arr[0] {
case "name":
return nameComponent, arr[1]
case "implements":
return implementsComponent, arr[1]
case "skills":
return skillsComponent, arr[1]
default:
panic("Invalid field found")
}
}

This has nothing to do with Golang maps dropping any values.
The problem is that you are using a map of skill pointers and not skills. Two pointers to the same skill content can be different.
skillsSet := make(map[*skill]bool)
If you change this to map[skill]bool, this should work. You may try it out!

Related

Get a value from a known key in a dynamic nested YAML

I'm pretty new to Golang. I have a YAML file with dynamic keys but only one is known, and it's not necessarily the first one (after config). There is no other key at the level of config.
The yaml file :
config:
foo:bar: baz
bar:foo: baz
abs:getit: myvalue
I want to retrieve myvalue from the nested key config:abs:getit. This nested key name will never change, it will always be config:abs:getit. All other keys can be whatever, we don't care, with different types of content (arrays, int, strings, array of array).
What is the best way to recover the value ?
I worked with yaml package, but I have to fix every field in a struct to unmarshall it, but I don't know how many nested keys there can be so I cannot write a struct which works all the time.
I worked with a map, but I can figure out which map I have to use, because if I can have a field with 6 nested keys or 3 nested keys with array in it before the value I'm searching and it will fails.
I am pretty lost with those kind of things in a dynamic context.
Ideally, I want to do a cat myFile.yaml | yq '.config."abs:getit"', but in Golang...
Any ideas and best practices to do that ?
You can do:
func main() {
var obj struct {
Config struct {
AbsGetit string `yaml:"abs:getit"`
} `yaml:"config"`
}
err := yaml.Unmarshal(data, &obj)
if err != nil {
panic(err)
}
fmt.Printf("%q\n", obj.Config.AbsGetit)
}
https://go.dev/play/p/KJ_lzZxaZBy
I think you need this code. Just put the correct path to your "myFile.yaml" file. In main() function, you will see two different examples of how to use the code according to your needs.
getConfVal finds a node of a YAML tree with an arbitrary sequence in N-dimensional depth. If the node does not exist, the value will be nil.
myFile.yaml
config:
foo:bar: baz
bar:foo: baz
abs:getit: myvalue
foo:
bar: "conf-foo-bar"
bar:
foo: "conf-bar-foo"
abs:
getit: "conf-abs-getit"
one:
two:
three:
four:
five: 5
five2: [4, 7]
package main
import (
"fmt"
"os"
yaml "gopkg.in/yaml.v3"
)
func main() {
if err := readTConf("./path/to/myFile.yaml", &cfg); err != nil {
fmt.Printf("Read YAML Conf: %v\n", err)
return
}
// e.g.
getConfVal(cfg, []string{"foo:bar"})
// getConfVal(cfg, []string{"foo", "bar"})
// getConfVal(cfg, []string{"four", "five"})
// getConfVal(cfg, []string{"four", "five2"})
fmt.Printf("\nThis is the result you are looking for. (%v)\n", needleRes)
}
var needleRes interface{}
var cfg map[string]interface{}
func getConfVal(o map[string]interface{}, ns []string) (map[string]interface{}, bool) {
nsCnt := len(ns)
for kn, vn := range ns {
for ko, vo := range o {
if fmt.Sprintf("%T", vo) == "map[string]interface {}" {
res, ok := getConfVal(vo.(map[string]interface{}), ns)
if ok {
return res, true
break
}
}
if fmt.Sprintf("%T", vo) == "string" {
if ko == vn {
if kn+1 == nsCnt {
needleRes = vo
return map[string]interface{}{}, true
}
}
}
}
}
return map[string]interface{}{}, false
}
func readTConf(f string, c *map[string]interface{}) error {
yamlFile, err := os.ReadFile(f)
if err != nil {
return err
}
if err := yaml.Unmarshal([]byte(yamlFile), &c); err != nil {
return err
}
return nil
}
Thank you for your precise answer. I'm sorry but there is an error in the question, and I apologize for the mistake.
It's not a single flow scalar Yaml but a map, since there is space before the value :
config:
foo:bar: baz
bar:foo: baz
abs:getit: myvalue
The code above logically returns a conversion error like this :
panic: yaml: unmarshal errors:
line 2: cannot unmarshal !!map into string
My whole code is here. The file I read is a Pulumi config Yaml, which will be different for all projects, except for one common key ("abs:getit:"), only the value is different.
The original question file has been modified. Really sorry for that...

How to get columns data from golang apache-arrow?

I am using apache-arrow/go to read parquet data.
I can parse the data to table by using apach-arrow.
reader, err := ipc.NewReader(buf, ipc.WithAllocator(alloc))
if err != nil {
log.Println(err.Error())
return nil
}
defer reader.Release()
records := make([]array.Record, 0)
for reader.Next() {
rec := reader.Record()
rec.Retain()
defer rec.Release()
records = append(records, rec)
}
table := array.NewTableFromRecords(reader.Schema(), records)
Here, i can get the column info from table.Colunmn(index), such as:
for i, _ := range table.Schema().Fields() {
a := table.Column(i)
log.Println(a)
}
But the Column struct is defined as
type Column struct {
field arrow.Field
data *Chunked
}
and the println result is like
["WARN" "WARN" "WARN" "WARN" "WARN" "WARN" "WARN" "WARN" "WARN" "WARN"]
However, this is not a string or slice. Is there anyway that i can get the data of each column with string type or []interface{} ?
Update:
I find that i can use reflect to get the element from col.
log.Println(col.(*array.Int64).Value(0))
But i am not sure if this is the recommended way to use it.
When working with Arrow data, there's a couple concepts to understand:
Array: Metadata + contiguous buffers of data
Record Batch: A schema + a collection of Arrays that are all the same length.
Chunked Array: A group of Arrays of varying lengths but all the same data type. This allows you to treat multiple Arrays as one single column of data without having to copy them all into a contiguous buffer.
Column: Is just a Field + a Chunked Array
Table: A collection of Columns allowing you to treat multiple non-contiguous arrays as a single large table without having to copy them all into contiguous buffers.
In your case, you're reading multiple record batches (groups of contiguous Arrays) and treating them as a single large table. There's a few different ways you can work with the data:
One way is to use a TableReader:
tr := array.NewTableReader(tbl, 5)
defer tr.Release()
for tr.Next() {
rec := tr.Record()
for i, col := range rec.Columns() {
// do something with the Array
}
}
Another way would be to interact with the columns directly as you were in your example:
for i := 0; i < table.NumCols(); i++ {
col := table.Column(i)
for _, chunk := range col.Data().Chunks() {
// do something with chunk (an arrow.Array)
}
}
Either way, you eventually have an arrow.Array to deal with, which is an interface containing one of the typed Array types. At this point you are going to have to switch on something, you could type switch on the type of the Array itself:
switch arr := col.(type) {
case *array.Int64:
// do stuff with arr
case *array.Int32:
// do stuff with arr
case *array.String:
// do stuff with arr
...
}
Alternately, you could type switch on the data type:
switch col.DataType().ID() {
case arrow.INT64:
// type assertion needed col.(*array.Int64)
case arrow.INT32:
// type assertion needed col.(*array.Int32)
...
}
For getting the data out of the array, primitive types which are stored contiguously tend to have a *Values method which will return a slice of the type. For example array.Int64 has Int64Values() which returns []int64. Otherwise, all of the types have .Value(int) methods which return the value at a particular index as you showed in your example.
Hope this helps!
Make sure you use v9
(import "github.com/apache/arrow/go/v9/arrow") because it have implemented json.Marshaller (from go-json)
Use "github.com/goccy/go-json" for Marshaler (because of this)
Then you can use TableReader to Marshal it then Unmarshal with type []any
In your example maybe look like this:
import (
"github.com/apache/arrow/go/v9/arrow"
"github.com/apache/arrow/go/v9/arrow/array"
"github.com/apache/arrow/go/v9/arrow/memory"
"github.com/goccy/go-json"
)
...
tr := array.NewTableReader(tabel, 6)
defer tr.Release()
// fmt.Printf("tbl.NumRows() = %+v\n", tbl.NumRows())
// fmt.Printf("tbl.NumColumn = %+v\n", tbl.NumCols())
// keySlice is for sorting same as data source
keySlice := make([]string, 0, tabel.NumCols())
res := make(map[string][]any, 0)
var key string
for tr.Next() {
rec := tr.Record()
for i, col := range rec.Columns() {
key = rec.ColumnName(i)
if res[key] == nil {
res[key] = make([]any, 0)
keySlice = append(keySlice, key)
}
var tmp []any
b2, err := json.Marshal(col)
if err != nil {
panic(err)
}
err = json.Unmarshal(b2, &tmp)
if err != nil {
panic(err)
}
// fmt.Printf("key = %s\n", key)
// fmt.Printf("tmp = %+v\n", tmp)
res[key] = append(res[key], tmp...)
}
}
fmt.Println("res", res)

Using "dynamic" key to extract value from map [duplicate]

This question already has answers here:
Access struct property by name
(5 answers)
Golang dynamic access to a struct property
(2 answers)
How to access to a struct parameter value from a variable in Golang
(1 answer)
Closed 9 months ago.
Came from javascript background, and just started with Golang. I am learning all the new terms in Golang, and creating new question because I cannot find the answer I need (probably due to lack of knowledge of terms to search for)
I created a custom type, created an array of types, and I want to create a function where I can retrieve all the values of a specific key, and return an array of all the values (brands in this example)
type Car struct {
brand string
units int
}
....
var cars []Car
var singleCar Car
//So i have a loop here and inside the for-loop, i create many single cars
singleCar = Car {
brand: "Mercedes",
units: 20
}
//and i append the singleCar into cars
cars = append(cars, singleCar)
Now what I want to do is to create a function that I can retrieve all the brands, and I tried doing the following. I intend to have key as a dynamic value, so I can search by specific key, e.g. brand, model, capacity etc.
func getUniqueByKey(v []Car, key string) []string {
var combined []string
for i := range v {
combined = append(combined, v[i][key])
//this line returns error -
//invalid operation: cannot index v[i] (map index expression of type Car)compilerNonIndexableOperand
}
return combined
//This is suppose to return ["Mercedes", "Honda", "Ferrari"]
}
The above function is suppose to work if i use getUniqueByKey(cars, "brand") where in this example, brand is the key. But I do not know the syntaxes so it's returning error.
Seems like you're trying to get a property using a slice accessor, which doesn't work in Go. You'd need to write a function for each property. Here's an example with the brands:
func getUniqueBrands(v []Car) []string {
var combined []string
tempMap := make(map[string]bool)
for _, c := range v {
if _, p := tempMap[c.brand]; !p {
tempMap[c.brand] = true
combined = append(combined, c.brand)
}
}
return combined
}
Also, note the for loop being used to get the value of Car here. Go's range can be used to iterate over just indices or both indices and values. The index is discarded by assigning to _.
I would recommend re-using this code with an added switch-case block to get the result you want. If you need to return multiple types, use interface{} and type assertion.
Maybe you could marshal your struct into json data then convert it to a map. Example code:
package main
import (
"encoding/json"
"fmt"
)
type RandomStruct struct {
FieldA string
FieldB int
FieldC string
RandomFieldD bool
RandomFieldE interface{}
}
func main() {
fieldName := "FieldC"
randomStruct := RandomStruct{
FieldA: "a",
FieldB: 5,
FieldC: "c",
RandomFieldD: false,
RandomFieldE: map[string]string{"innerFieldA": "??"},
}
randomStructs := make([]RandomStruct, 0)
randomStructs = append(randomStructs, randomStruct, randomStruct, randomStruct)
res := FetchRandomFieldAndConcat(randomStructs, fieldName)
fmt.Println(res)
}
func FetchRandomFieldAndConcat(randomStructs []RandomStruct, fieldName string) []interface{} {
res := make([]interface{}, 0)
for _, randomStruct := range randomStructs {
jsonData, _ := json.Marshal(randomStruct)
jsonMap := make(map[string]interface{})
err := json.Unmarshal(jsonData, &jsonMap)
if err != nil {
fmt.Println(err)
// panic(err)
}
value, exists := jsonMap[fieldName]
if exists {
res = append(res, value)
}
}
return res
}

Examining fields of a struct by reference (via static analysis)

I'm trying to write a parser for golang code to examine the fields of a referenced struct. For example, given:
type Hello struct {
id int64
}
func Test(ref Hello) {}
I would like to be able to statically analyze this code and go from the args of Test and inspect Hello's fields.
I'm currently using the analysis package. I know how to inspect the struct definition itself in the ast, and also how to parse the function's args for its types. But is there a way to go from reference to parsing the struct? What if the struct is defined in a different file?
If you're doing static analysis and you'd like to better understand how the packages go/ast, go/types, etc. work together then you should definitely check out Alan Donovan's go types document.
You can use the golang.org/x/tools/go/packages package to get the syntax tree and the type info. There may be better, less involved, approaches to achieve the same but this one's the one I'm familiar with.
To get the go/types representation of Hello you can do the following:
func main() {
cfg := new(packages.Config)
cfg.Mode = packages.NeedSyntax | packages.NeedTypes | packages.NeedTypesInfo
cfg.Fset = token.NewFileSet()
// "." specifies the current directory.
// You should replace it with a pattern that
// will match the package you want to analyse.
pkgs, err := packages.Load(cfg, ".")
if err != nil {
panic(err)
}
for _, pkg := range pkgs {
// Loop over the list of files in the package.
for _, syn := range pkg.Syntax {
// Loop over the top-level declarations in the file.
for _, dec := range syn.Decls {
// Look for the func declaration
// of your Help function.
fd, ok := dec.(*ast.FuncDecl)
if !ok || fd.Name.Name != "Test" {
continue
}
// Get the expression node that
// represents the identifier of
// the parameter's type i.e. Hello.
p := fd.Type.Params.List[0].Type
// NOTE: if the type is not a named
// package-local type, e.g. a pointer,
// a slice, or an imported type, then
// you'll have have to "dig deeper"
// to get to the *ast.Ident.
id, ok := p.(*ast.Ident)
if !ok {
continue
}
// With the packages.NeedTypesInfo mode set
// the package will also include the result
// of the complete type-check of the package's
// syntax trees.
//
// The TypeInfo.Types field maps ast expressions
// to their types, this allows you to get the type
// information using the identifier.
typ := pkg.TypesInfo.Types[id]
named := typ.Type.(*types.Named)
fmt.Println(named) // Hello's *types.Named
fmt.Println(named.Underlying().(*types.Struct)) // Hello's *types.Struct
}
}
}
}
To get the go/ast representation of the Hello type's definition you can do the following:
func main() {
// You'll need to repeat the steps above
// to load the packages as well as finding
// the *types.Named instance which will be
// used to determine the position of the
// type's definition ast.
pos := named.Obj().Pos() // the source position of the type's name
for _, pkg := range pkgs {
// Loop over the files in the package.
for _, syn := range pkg.Syntax {
// Use the position to determine whether
// or not the type is declared in this
// file, if not then go to the next one.
if syn.Pos() >= pos || pos >= syn.End() {
continue
}
// Loop over the top-level declarations in the file.
for _, dec := range syn.Decls {
// If the declaration is something
// other than a type declaration then
// continue to the next one.
gd, ok := dec.(*ast.GenDecl)
if !ok || gd.Tok != token.TYPE {
continue
}
// Loop over the specs in the declaration.
for _, spec := range gd.Specs {
// Look for the type spec whose name matches
// the name of the *types.Named instance.
ts, ok := spec.(*ast.TypeSpec)
if !ok || ts.Name.Name != named.Obj().Name() {
continue
}
fmt.Println(ts) // Hello's *ast.TypeSpec
fmt.Println(ts.Type.(*ast.StructType)) // Hello's *ast.StructType
}
}
}
}
}

Filtering a slice of structs based on a different slice in Golang

In the example code below, I have a few users in manySimpleUsers that I would like to remove from manyFullUsers based on the Username.
If I do it with a nested couple of for... range loops, there will be many cycles required to filter all of the elements, especially when there are large numbers of elements in both Slices.
What is the best way to achieve this in Go?
package main
import "fmt"
func main() {
fmt.Println("Hello, playground")
type FullUser struct {
UserName string
UserEmail string
}
manyFullUsers := []FullUser{{"foo", "foo#jawohl.com"},
{"bar", "bar#jawohl.com"},
{"baz", "baz#jawohl.com"}}
type SimpleUser struct {
UserName string
}
manySimpleUsers := []SimpleUser{{"foo"}, {"bar"}}
fmt.Println(manyFullUsers)
fmt.Println(manySimpleUsers)
}
Create a map then use it to filter.
func filterByUserName(fu []FullUser, su []SimpleUser) (out []FullUser) {
f := make(map[string]struct{}, len(su))
for _, u := range su {
f[u.UserName] = struct{}{}
}
for _, u := range fu {
if _, ok := f[u.UserName]; ok {
out = append(out, u)
}
}
return
}
playground
you can use Filter() and Exist() from https://github.com/ledongthuc/goterators that I created to reuse aggregate & transform functions.
filteredFullUsers := goterators.Filter(manyFullUsers, func(item FullUser) bool {
return !goterators.Exist(manySimpleUsers, SimpleUser{item.UserName})
})
You also can build a map from manySimpleUsers that will be helpful to optimize searching username.
This lib requires the Go 1.18 to use that support generic + dynamic type you want to use with.

Resources