Is there a golang package that is equivalent to Python's ChainMap? - go

Python's ChainMap enables you to chain together several maps such that a lookup searches all of the maps in sequence until a matching key is found. Here is one of many articles about it:
https://dzone.com/articles/python-201-what-is-a-chainmap
And the official documentation:
https://docs.python.org/3/library/collections.html
Does anyone know of any existing equivalent package written in Go? I've so far been unable to find one, and would like to avoid re-inventing the wheel if one does exist.

There's not package but it's fairly simple to implement something similar:
Let's do it with a map[string]interface{}
package main
import (
"reflect"
"testing"
)
type ChainMap struct {
Map map[string]interface{}
maps []map[string]interface{}
}
func NewChainMap(maps ...map[string]interface{}) ChainMap {
if len(maps) == 0 {
return ChainMap{
Map: make(map[string]interface{}, 0),
maps: maps,
}
}
r := make(map[string]interface{}, len(maps[0]))
for i := len(maps) - 1; i >= 0; i-- {
m := maps[i]
for k, v := range m {
r[k] = v
}
}
return ChainMap{
Map: r,
maps: maps,
}
}
func (c ChainMap) Parents() ChainMap {
if len(c.maps) < 2 {
return c
}
return NewChainMap(c.maps[1:]...)
}
A small test:
func TestChainMap(t *testing.T) {
var m = NewChainMap(
map[string]interface{}{
"foo": "bar",
},
map[string]interface{}{
"foo": "baz",
"hello": "world",
},
map[string]interface{}{
"foo": "baw",
},
)
if !reflect.DeepEqual(
m.Map,
map[string]interface{}{
"foo": "bar",
"hello": "world",
},
) {
t.Fail()
}
if !reflect.DeepEqual(
m.Parents().Map,
map[string]interface{}{
"foo": "baz",
"hello": "world",
},
) {
t.Fail()
}
}

I eventually found an implementation buried in the bigkevmcd/go-configparser package: https://github.com/bigkevmcd/go-configparser/blob/master/chainmap/chainmap.go
The upside of this one is that it uses the original maps for storage, so does the right thing if one of them is updated later; this makes it work more like the original python version. The downside is that the caller needs to call Get() and Len() functions rather than simply using the object as a normal map.
package chainmap
type Dict map[string]string
type ChainMap struct {
maps []Dict
}
func New(dicts ...Dict) *ChainMap {
chainMap := &ChainMap{
maps: make([]Dict, 0),
}
for _, dict := range dicts {
chainMap.maps = append(chainMap.maps, dict)
}
return chainMap
}
func (c *ChainMap) Len() int {
return len(c.maps)
}
func (c *ChainMap) Get(key string) string {
var value string
for _, dict := range c.maps {
if result, present := dict[key]; present {
value = result
}
}
return value
}

Related

Basic + Slice + Map Type Compatible Generics?

Is there a way to create a generic function that can adjust its operation when passed a map or a slice type vs a basic type?
Goal
Create a slice reading function generator with a flexible return type:
func ValueReader[T <probably something fancy>](i int) func ([]ProtoConvertable) T {
return func (row []ProtoConvertable) T {
return ...
}
}
row := []ProtoConvertable{
&Data[int]{Value: 333},
&ListData{Values: []ProtoConvertable{
&Data[string]{Value: "hello"},
&Data[string]{Value: "world"},
}},
&MapData{Values: map[ProtoConvertable]ProtoConvertable{
&Data[int]{Value: 22}: &Data[string]{Value: "world"},
&Data[int]{Value: 11}: &Data[string]{Value: "hello"},
}},
}
dataReader := ValueReader[int](0) // A function that converts the first element to an int
listDataReader := ValueReader[[]string](1) // A function that converts the second element to a slice
mapDataReader := ValueReader[map[int]string](2) // A function that converts the third element to a map
data := dataReader(row) // 333
listData := listDataReader(row) // []string{"hello", "world"}
mapData := mapDataReader(row) // map[int]string{11: "hello", 22: "world"}
Types
type ValueType interface {
int | string
}
type ProtoConvertable interface {
ToProto() *pb.GenericMessage
}
type Data[T ValueType] struct {
Value T
}
func (d *Data) ToProto() *pb.GenericMessage{
...
}
type ListData struct {
Values []ProtoConvertable
}
func (d *ListData) ToProto() *pb.GenericMessage {
...
}
type MapData struct {
Values map[ProtoConvertable]ProtoConvertable
}
func (d *MapData) ToProto() *pb.GenericMessage {
...
}
Current Solution
func ValueReader[T ValueType](i int) func([]ProtoConvertable) T {
return func(row []ProtoConvertable) T {
return row[i].(*Data[T]).Value
}
}
func ListValueReader[T ValueType](i int) func([]ProtoConvertable) []T {
return func(row []ProtoConvertable) []T {
vs := row[i].(*ListData).Values
res := make([]T, len(vs))
for i, v := range vs {
res[i] = v.(*Data[T]).Value
}
return res
}
}
func MapValueReader[K ValueType, V ValueType](i int) func([]ProtoConvertable) map[K]V {
return func(row []ProtoConvertable) map[K]V {
vs := row[i].(*MapData).Values
res := make(map[K]V, len(vs))
for k, v := range vs {
res[k.(*Data[K]).Value] = v.(*Data[V]).Value
}
return res
}
}
dataReader := ValueReader[int](0)
listDataReader := ListValueReader[string](1)
mapDataReader := MapValueReader[int, string](2)
Note: all of this code is an untested simplification of a more complicated library. It might need some tweaking to get to actually work.
The <probably something fancy> doesn't exist.
The main issue is that you want to model a type parameter that matches a base value and two composite types, one of which is a map type where you want to capture both K and V.
Even if it existed, the body of ValueReader would be a type-switch on T to return each specialized reader function, so your existing solution that involves a small amount of code duplication seems just a better strategy overall.
My advice is to use generics when the operations on the different concrete types of T are really identical. You can read more at: https://go.dev/blog/when-generics

Avoid duplicating identical code for string map and slice

While writing two methods (one for a slice and one for a string map) I realized the implementation for both methods is identical and the only thing that changes is the prototype of the functions.
I'm trying to avoid repeating it and originally I thought of the following (see the FIXME part):
package main
import (
"fmt"
"strings"
)
type SomethingList []*Something
type SomethingMap map[string]*Something
type Something struct {
ID string
Type int
}
func (sl SomethingList) GetIDsString() string {
return getIDsString(sl)
}
func (sm SomethingMap) GetIDsString() string {
return getIDsString(sm)
}
func getIDsString(elements interface{}) string {
var ids []string
// FIXME: Yes, I know I can't iterate over an interface
for element = range elements {
ids = append(ids, element.ID)
}
return strings.Join(ids, ",")
}
func main() {
s1 := Something{ID: "ABC", Type: 1}
s2 := Something{ID: "DEF", Type: 1}
sl := SomethingList{&s1, &s2}
sm := SomethingMap{s1.ID: &s1, s2.ID: &s2}
fmt.Println(sl.GetIDsString())
fmt.Println(sm.GetIDsString())
}
The important part is the function getIDsString which basically takes the ID field of the struct and concatenates it's content across all the members of the slice or map.
I realize now after reading a bit about how interfaces work (yes, I'm quite a newbie in Go as is probably obvious already :-)) that this is not going to work, as Go is statically typed and I can't simply change the types on runtime. Also the interfaces don't allow me to iterate. I've been able to get close using a loop that iterates using the Len() method on the reflect.ValueOf and Index() to access each element. But Index() doesn't work on the string map.
What would be the most idiomatic way of solving this without duplicating much code?
Thanks!
In general repetition of small part of the code in golang is quite common. But in case you have a large amount of duplicate code, you can have that logic in one structure, and ad hoc transform the second structure to the first one to invoke that logic:
package main
import (
"fmt"
"strings"
)
type (
SomethingList []*Something
SomethingMap map[string]*Something
Something struct {
ID string
Type int
}
)
func (sl SomethingList) GetIDsString() string {
ids := make([]string, len(sl))
for i := range sl {
ids[i] = sl[i].ID
}
return strings.Join(ids, ",")
}
func (sm SomethingMap) GetIDsString() string {
l := make(SomethingList, len(sm))
i := 0
for key := range sm {
l[i] = sm[key]
i++
}
return l.GetIDsString()
}
func main() {
s1 := Something{ID: "ABC", Type: 1}
s2 := Something{ID: "DEF", Type: 1}
sl := SomethingList{&s1, &s2}
sm := SomethingMap{s1.ID: &s1, s2.ID: &s2}
fmt.Println(sl.GetIDsString())
fmt.Println(sm.GetIDsString())
}
Alternatively you could decouple the creation of IDsString from the structure itself in a following way.
package main
import (
"fmt"
"strings"
)
type (
SomethingList []*Something
SomethingMap map[string]*Something
Something struct {
ID string
Type int
}
somethingIterator interface {
ForEach(func(value Something))
}
)
func (sl SomethingList) ForEach(f func(value Something)) {
for i := range sl {
f(*sl[i])
}
}
func (sm SomethingMap) ForEach(f func(value Something)) {
for key := range sm {
f(*sm[key])
}
}
func GetIDsString(iterator somethingIterator) string {
var ids []string
iterator.ForEach(func(value Something) {
// Some sophisticated logic is here.
ids = append(ids, value.ID)
})
return strings.Join(ids, ",")
}
func main() {
s1 := Something{ID: "ABC", Type: 1}
s2 := Something{ID: "DEF", Type: 1}
sl := SomethingList{&s1, &s2}
sm := SomethingMap{s1.ID: &s1, s2.ID: &s2}
fmt.Println(GetIDsString(sl))
fmt.Println(GetIDsString(sm))
}
Second approach allows to avoid extra intermediate structure creation which could be beneficial for big list/map.

Can a variable be used as a placeholder for a function call?

I am writing a program which has several structs and functions to handle these structs differently. I am having a generic function which calls the required function based on the inputs. Is there a generic way to use the returned value from getStruct()?
package main
var X func(s []string) A
var Y func(s []string) B
type A struct {
Name string
Place string
}
type B struct {
Name string
Place string
Value string
}
func newA(s []string) A {
a := A{
Name: s[0],
Place: s[1],
}
return a
}
func newB(s []string) B {
a := B{
Name: s[0],
Place: s[1],
Value: s[2],
}
return a
}
func getStruct(t string) interface{} {
switch {
case t == "A":
return X
case t == "B":
return Y
default:
return //someStruct
}
}
func main() {
buildNewStruct := getStruct("A") //Lets assume "A" here is got as an argument
var strSlice = []string{"Bob", "US"}
buildNewStruct(strSlice) //How to do this operation?
//I am hoping to use buildNewStruct(strSlice) to dynamically call
//either of newA(strSlice) or newB(strSlice) function
}
I have tried looking at this and this the later is not exactly the same as my question.
Since I am new to go, I am not sure if something like this is possible.
you can use the reflect package to set the struct properties to the equivalent index positioned value from an []interface{} slice.
package main
import (
"fmt"
"log"
"reflect"
)
func main() {
var a A
err := decode(&a, []interface{}{"Name", "Place"})
log.Println(err)
log.Println(a)
}
func decode(dst interface{}, values []interface{}) error {
rvptr := reflect.ValueOf(dst)
if rvptr.Kind() != reflect.Ptr {
return fmt.Errorf("value must be ptr")
}
rv := rvptr.Elem()
if rv.NumField() < len(values) {
return fmt.Errorf("too many values")
}
if rv.NumField() > len(values) {
return fmt.Errorf("too few values")
}
rvalues := reflect.ValueOf(values)
for i := range values {
f := rv.FieldByIndex([]int{i})
f.Set(rvalues.Index(i).Elem())
}
return nil
}
type A struct {
Name string
Place string
}
type B struct {
Name string
Place string
Value string
}
prints
$ go run main.go
2019/11/21 17:00:17 <nil>
2019/11/21 17:00:17 {Name Place}
The problem is the return type for the function.
func newA(in []string) interface{} {...}
func newB(in []string) interface{} {...}
func getStruct(name string) func([]string) interface{} {
switch name {
case "A": return newA
case "B": return newB
}
return nil
}
func main() {
buildNewStruct := getStruct("A")
var strSlice = []string{"Bob", "US"}
str:=buildNewStruct(strSlice)
if a, ok:=str.(A); ok {
...
}
}
With this approach, even though you saved some code by calling a unified buildNewStruct(), you have to use type assertions to figure out what is returned from that function, so this may not make a lot of sense. It depends on your exact use case though.

How do I use reflect.FieldByName when the value is of kind reflect.Ptr

I have a project function which returns a slice containing the field values by name of each struct or map in an input slice. I am having trouble with case where the input slice contains pointers to structs. I have setup a recursive function to operate on the value, but need to know how to convert from kind reflect.Ptr to the underlying reflect.Struct. How is this done? Any other design recommendations are appreciated. I am still a bit new to Go.
Here is the code:
func project(in []interface{}, property string) []interface{} {
var result []interface{}
var appendValue func(list []interface{}, el interface{})
appendValue = func(list []interface{}, el interface{}) {
v := reflect.ValueOf(el)
kind := v.Kind()
if kind == reflect.Ptr {
// How do I get the struct behind this ptr?
// appendValue(list, el)
} else if kind == reflect.Struct {
result = append(result, v.FieldByName(property).Interface())
} else if kind == reflect.Map {
result = append(result, el.(map[string]interface{})[property])
} else {
panic("Value must be a struct or map")
}
}
for _, el := range in {
appendValue(result, el)
}
return result
}
... and the test cases:
func Test_project(t *testing.T) {
cases := map[string]struct {
input []interface{}
property string
expected []interface{}
}{
"simple-map": {
[]interface{}{
map[string]interface{}{
"a": "a1",
},
},
"a",
[]interface{}{"a1"},
},
"simple-struct": {
[]interface{}{
simpleStruct{
A: "a1",
},
},
"A",
[]interface{}{"a1"},
},
// THIS ONE FAILS
"simple-struct-ptr": {
[]interface{}{
&simpleStruct{
A: "a1",
},
},
"A",
[]interface{}{"a1"},
},
}
for k, v := range cases {
t.Run(k, func(t *testing.T) {
got := project(v.input, v.property)
if !reflect.DeepEqual(got, v.expected) {
t.Fatalf("Expected %+v, got %+v", v.expected, got)
}
})
}
}
Use Elem() to go from a reflect.Ptr to the value it points to.

Contains method for a slice

Is there anything similar to a slice.contains(object) method in Go without having to do a search through each element in a slice?
Mostafa has already pointed out that such a method is trivial to write, and mkb gave you a hint to use the binary search from the sort package. But if you are going to do a lot of such contains checks, you might also consider using a map instead.
It's trivial to check if a specific map key exists by using the value, ok := yourmap[key] idiom. Since you aren't interested in the value, you might also create a map[string]struct{} for example. Using an empty struct{} here has the advantage that it doesn't require any additional space and Go's internal map type is optimized for that kind of values. Therefore, map[string] struct{} is a popular choice for sets in the Go world.
No, such method does not exist, but is trivial to write:
func contains(s []int, e int) bool {
for _, a := range s {
if a == e {
return true
}
}
return false
}
You can use a map if that lookup is an important part of your code, but maps have cost too.
Starting with Go 1.18, you can use the slices package – specifically the generic Contains function:
https://pkg.go.dev/golang.org/x/exp/slices#Contains.
go get golang.org/x/exp/slices
import "golang.org/x/exp/slices"
things := []string{"foo", "bar", "baz"}
slices.Contains(things, "foo") // true
Note that since this is outside the stdlib as an experimental package, it is not bound to the Go 1 Compatibility Promise™ and may change before being formally added to the stdlib.
With Go 1.18+ we could use generics.
func Contains[T comparable](s []T, e T) bool {
for _, v := range s {
if v == e {
return true
}
}
return false
}
The sort package provides the building blocks if your slice is sorted or you are willing to sort it.
input := []string{"bird", "apple", "ocean", "fork", "anchor"}
sort.Strings(input)
fmt.Println(contains(input, "apple")) // true
fmt.Println(contains(input, "grow")) // false
...
func contains(s []string, searchterm string) bool {
i := sort.SearchStrings(s, searchterm)
return i < len(s) && s[i] == searchterm
}
SearchString promises to return the index to insert x if x is not present (it could be len(a)), so a check of that reveals whether the string is contained the sorted slice.
Instead of using a slice, map may be a better solution.
simple example:
package main
import "fmt"
func contains(slice []string, item string) bool {
set := make(map[string]struct{}, len(slice))
for _, s := range slice {
set[s] = struct{}{}
}
_, ok := set[item]
return ok
}
func main() {
s := []string{"a", "b"}
s1 := "a"
fmt.Println(contains(s, s1))
}
http://play.golang.org/p/CEG6cu4JTf
If the slice is sorted, there is a binary search implemented in the sort package.
func Contain(target interface{}, list interface{}) (bool, int) {
if reflect.TypeOf(list).Kind() == reflect.Slice || reflect.TypeOf(list).Kind() == reflect.Array {
listvalue := reflect.ValueOf(list)
for i := 0; i < listvalue.Len(); i++ {
if target == listvalue.Index(i).Interface() {
return true, i
}
}
}
if reflect.TypeOf(target).Kind() == reflect.String && reflect.TypeOf(list).Kind() == reflect.String {
return strings.Contains(list.(string), target.(string)), strings.Index(list.(string), target.(string))
}
return false, -1
}
I think map[x]bool is more useful than map[x]struct{}.
Indexing the map for an item that isn't present will return false. so instead of _, ok := m[X], you can just say m[X].
This makes it easy to nest inclusion tests in expressions.
You can use the reflect package to iterate over an interface whose concrete type is a slice:
func HasElem(s interface{}, elem interface{}) bool {
arrV := reflect.ValueOf(s)
if arrV.Kind() == reflect.Slice {
for i := 0; i < arrV.Len(); i++ {
// XXX - panics if slice element points to an unexported struct field
// see https://golang.org/pkg/reflect/#Value.Interface
if arrV.Index(i).Interface() == elem {
return true
}
}
}
return false
}
https://play.golang.org/p/jL5UD7yCNq
Not sure generics are needed here. You just need a contract for your desired behavior. Doing the following is no more than what you would have to do in other languages if you wanted your own objects to behave themselves in collections, by overriding Equals() and GetHashCode() for instance.
type Identifiable interface{
GetIdentity() string
}
func IsIdentical(this Identifiable, that Identifiable) bool{
return (&this == &that) || (this.GetIdentity() == that.GetIdentity())
}
func contains(s []Identifiable, e Identifiable) bool {
for _, a := range s {
if IsIdentical(a,e) {
return true
}
}
return false
}
If it is not feasable to use a map for finding items based on a key, you can consider the goderive tool. Goderive generates a type specific implementation of a contains method, making your code both readable and efficient.
Example;
type Foo struct {
Field1 string
Field2 int
}
func Test(m Foo) bool {
var allItems []Foo
return deriveContainsFoo(allItems, m)
}
To generate the deriveContainsFoo method:
Install goderive with go get -u github.com/awalterschulze/goderive
Run goderive ./... in your workspace folder
This method will be generated for deriveContains:
func deriveContainsFoo(list []Foo, item Foo) bool {
for _, v := range list {
if v == item {
return true
}
}
return false
}
Goderive has support for quite some other useful helper methods to apply a functional programming style in go.
The go style:
func Contains(n int, match func(i int) bool) bool {
for i := 0; i < n; i++ {
if match(i) {
return true
}
}
return false
}
s := []string{"a", "b", "c", "o"}
// test if s contains "o"
ok := Contains(len(s), func(i int) bool {
return s[i] == "o"
})
If you have a byte slice, you can use bytes package:
package main
import "bytes"
func contains(b []byte, sub byte) bool {
return bytes.Contains(b, []byte{sub})
}
func main() {
b := contains([]byte{10, 11, 12, 13, 14}, 13)
println(b)
}
Or suffixarray package:
package main
import "index/suffixarray"
func contains(b []byte, sub byte) bool {
return suffixarray.New(b).Lookup([]byte{sub}, 1) != nil
}
func main() {
b := contains([]byte{10, 11, 12, 13, 14}, 13)
println(b)
}
If you have an int slice, you can use intsets package:
package main
import "golang.org/x/tools/container/intsets"
func main() {
var s intsets.Sparse
for n := 10; n < 20; n++ {
s.Insert(n)
}
b := s.Has(16)
println(b)
}
https://golang.org/pkg/bytes
https://golang.org/pkg/index/suffixarray
https://pkg.go.dev/golang.org/x/tools/container/intsets
I created the following Contains function using reflect package.
This function can be used for various types like int32 or struct etc.
// Contains returns true if an element is present in a slice
func Contains(list interface{}, elem interface{}) bool {
listV := reflect.ValueOf(list)
if listV.Kind() == reflect.Slice {
for i := 0; i < listV.Len(); i++ {
item := listV.Index(i).Interface()
target := reflect.ValueOf(elem).Convert(reflect.TypeOf(item)).Interface()
if ok := reflect.DeepEqual(item, target); ok {
return true
}
}
}
return false
}
Usage of contains function is below
// slice of int32
containsInt32 := Contains([]int32{1, 2, 3, 4, 5}, 3)
fmt.Println("contains int32:", containsInt32)
// slice of float64
containsFloat64 := Contains([]float64{1.1, 2.2, 3.3, 4.4, 5.5}, 4.4)
fmt.Println("contains float64:", containsFloat64)
// slice of struct
type item struct {
ID string
Name string
}
list := []item{
item{
ID: "1",
Name: "test1",
},
item{
ID: "2",
Name: "test2",
},
item{
ID: "3",
Name: "test3",
},
}
target := item{
ID: "2",
Name: "test2",
}
containsStruct := Contains(list, target)
fmt.Println("contains struct:", containsStruct)
// Output:
// contains int32: true
// contains float64: true
// contains struct: true
Please see here for more details:
https://github.com/glassonion1/xgo/blob/main/contains.go
There are several packages that can help, but this one seems promising:
https://github.com/wesovilabs/koazee
var numbers = []int{1, 5, 4, 3, 2, 7, 1, 8, 2, 3}
contains, _ := stream.Contains(7)
fmt.Printf("stream.Contains(7): %v\n", contains)
It might be considered a bit 'hacky' but depending the size and contents of the slice, you can join the slice together and do a string search.
For example you have a slice containing single word values (e.g. "yes", "no", "maybe"). These results are appended to a slice. If you want to check if this slice contains any "maybe" results, you may use
exSlice := ["yes", "no", "yes", "maybe"]
if strings.Contains(strings.Join(exSlice, ","), "maybe") {
fmt.Println("We have a maybe!")
}
How suitable this is really depends on the size of the slice and length of its members. There may be performance or suitability issues for large slices or long values, but for smaller slices of finite size and simple values it is a valid one-liner to achieve the desired result.

Resources