I'm trying to deserialize an AWS struct in go so that I can pass in parameters and output values based on those parameters, for example;
func DisplayResults(conf *config.Configuration,
regionalData []*ec2.DescribeInstancesOutput) {
log.Debug("Displaying results")
log.Debug("Table view [%v]", conf.Display)
for _, rv := range regionalData {
for _, reservation := range rv.Reservations {
for _, instance := range reservation.Instances {
var i map[string]interface{}
json.Unmarshal(instance, i)
}
}
}
}
I've also tried using:
"github.com/fatih/structs"
i := structs.Map(instance)
log.Print("%v", i)
however, I think that I need to somehow derefence the struct because my output looks like this:
RamdiskId:<nil> SriovNetSupport:<nil> VpcId:0xc420321ba0 State:map[Code:0xc42031ea08 Name:0xc420321890] VirtualizationType:0xc420321b90 CapacityReservationId:<nil> ClientToken:0xc420321560 HibernationOptions:map[Configured:0xc42031e96b] IamInstanceProfile:map[Arn:0xc4203215a0 Id:0xc4203215b0] ImageId:0xc4203215c0
I've also tried this:
func DisplayResults(conf *config.Configuration, regionalData []*ec2.DescribeInstancesOutput, parameters []string) {
log.Debug("Displaying results")
log.Debug("Table view [%v]", conf.Display)
for _, rv := range regionalData {
for _, reservation := range rv.Reservations {
for _, instance := range reservation.Instances {
y := deref(instance)
log.Print("%v", y["InstanceId"])
}
}
}
}
func deref(instance *ec2.Instance) ec2.Instance {
var i ec2.Instance
x := &i
*x = *instance
return i
}
Related
I know how to do in javascript, it will be something similar to:
The Go function will receive a function as parameter, I wanna get function as string to build a map that then I'll save in some database.
package main
import (
"fmt"
)
func describe(i interface{}) string {
return fmt.Sprintf("%v", i)
}
func dummyReducer(int) int {
return 1
}
func Accumulator(reducer func(int, int) int, init int) func(int) int {
input := map[string]interface{}{
"func_name": "accumulatorLoger",
"func_data": map[string]interface{}{
"reducer": string(describe(reducer)),
"init": init,
},
}
// {
// func_data: { init: 10, reducer: '0x64b880' },
// func_name: 'accumulatorLoger'
// }
// TODO: next: save the input data in the database
fmt.Println(input)
return dummyReducer
}
If you want the body you need the source. That means that your program will need access to the Go file in which the function you want the body of was declared.
To get a function's file, you can use (*runtime.Func).FileLine. And you'll also need the name of the function later, so use the runtime information to get that too:
func getFuncInfo(f interface{}) (name, file string) {
pc := reflect.ValueOf(f).Pointer()
fn := runtime.FuncForPC(pc)
file, _ = fn.FileLine(pc)
return fn.Name(), file
}
The name of the function may be package-qualified, if so you should clean it up:
if i := strings.LastIndexByte(name, '.'); i >= 0 {
name = name[i+1:]
}
https://play.golang.org/p/63zwvOh1qzE
Once you have the file, and your program has access to it, you can parse it with go/parser.ParseFile, retrieve the function's AST, and then print the body with go/printer.Fprint:
func getFuncAST(funcname, filename string) (*ast.FuncDecl, *token.FileSet) {
fs := token.NewFileSet()
file, err := parser.ParseFile(fs, filename, "", 0)
if err != nil {
panic(err)
}
for _, d := range file.Decls {
if f, ok := d.(*ast.FuncDecl); ok && f.Name.Name == funcname {
return f, fs
}
}
panic("function not found")
}
func getFuncBodyString(f *ast.FuncDecl, fs *token.FileSet) string {
var buf bytes.Buffer
if err := printer.Fprint(&buf, fs, f.Body); err != nil {
panic(err)
}
return buf.String()
}
https://play.golang.org/p/QDMSMhwrf39
I have a map of the format:
map[string]map[string]int
In this main map, I want to do something like putIfAbsent("key", new HashMap<>() as we have in Java. What is a clean and shorthand way to do it in Go?
You can do:
var val map[string]int
val, exists := m[key]
if !exists {
val = make(map[string]int)
m[key] = val
}
If you don't need the val in the code coming below this:
if _,exists := m[key]; !exists {
m[key]=make(map[string]int)
}
If you don't intend to use the value right away, here you go...
m := make(map[string]map[string]int)
if _, ok := m["unknown"]; !ok {
m["unknown"] = make(map[string]int)
}
Below is a suggestion for improvement:
To keep things clean and easy to understand, you can define your own types. For example, if your data is a mapping of "cities to persons to age", I would do it like this:
type Person map[string]int
type City map[string]Person
m := make(City)
if _, ok := m["Dhaka"]; !ok {
m["Dhaka"] = make(Person)
}
func main() {
var testMap map[int]interface{}
testMap = make(map[int]interface{})
var addMap map[int]string
addMap = make(map[int]string)
addMap[1] = "999"
addMap[2] = "888"
Add(testMap, 111, addMap)
for key, val := range testMap {
fmt.Println(key)
for key2, val2 := range val.(map[int]string) {
fmt.Println(key2, val2)
}
}
}
func Add(_testMap map[int]interface{}, _key int, _val map[int]string) {
_, exist := _testMap[_key] // _ -> value
if exist == false {
//addmap
_testMap[_key] = _val
} else {
//whatever wanna to do
}
}
I'm using an AWS Lambda function written in Go with Dynamo stream but I don't see any way where I can marshall the old & new image to my struct because it's returning the image as map[string]DynamoDBAttributeValue.
I can check individual key and then assign it to my struct one by one, but is there an direct way to Marshall directly?
func HandleRequest(ctx context.Context, event events.DynamoDBEvent) {
for _, record := range event.Records {
var newStruct models.MyStruct // want to marshall newImage in this struct
logger.Debugf("New: %#v", record.Change.NewImage)
}
}
UPDATE:
Here is the custom DynamoDBEvent that I'm using now:
type DynamoDBEvent struct {
Records []DynamoDBEventRecord `json:"Records"`
}
type DynamoDBEventRecord struct {
AWSRegion string `json:"awsRegion"`
Change DynamoDBStreamRecord `json:"dynamodb"`
EventID string `json:"eventID"`
EventName string `json:"eventName"`
EventSource string `json:"eventSource"`
EventVersion string `json:"eventVersion"`
EventSourceArn string `json:"eventSourceARN"`
UserIdentity *events.DynamoDBUserIdentity `json:"userIdentity,omitempty"`
}
type DynamoDBStreamRecord struct {
ApproximateCreationDateTime events.SecondsEpochTime `json:"ApproximateCreationDateTime,omitempty"`
Keys map[string]*dynamodb.AttributeValue `json:"Keys,omitempty"`
NewImage map[string]*dynamodb.AttributeValue `json:"NewImage,omitempty"`
OldImage map[string]*dynamodb.AttributeValue `json:"OldImage,omitempty"`
SequenceNumber string `json:"SequenceNumber"`
SizeBytes int64 `json:"SizeBytes"`
StreamViewType string `json:"StreamViewType"`
}
Unfortunately there is no elegant way yet, sot you have to check each key/value pairs and assign them to your struct.
PS: As small sugar you can use func FromDynamoDBMap from this package and work with map[string]interface{} not with map[string]events.DynamoDBAttributeValue which is way easier.
I have created a solution for the problem. This was crucial to my implementation since my implementation uses custom keys.
Unfortunately, the Number type has to be cast to Float64 to make sure that it is not lost.
This will result in a completely usable struct against the changeset. You may need a map of your models and a known field
Usage
newResult, _ := UnmarshalStreamImage(record.Change.NewImage)
newMarshal, _ := attributevalue.MarshalMap(newResult)
model := SomeModel{}
err := attributevalue.UnmarshalMap(newMarshal, &model)
Function Code
// UnmarshalStreamImage converts events.DynamoDBAttributeValue to struct
func UnmarshalStreamImage(attribute map[string]events.DynamoDBAttributeValue) (map[string]interface{}, error) {
baseAttrMap := make(map[string]interface{})
for k, v := range attribute {
baseAttrMap[k] = extractVal(v)
}
return baseAttrMap, nil
}
func extractVal(v events.DynamoDBAttributeValue) interface{} {
var val interface{}
switch v.DataType() {
case events.DataTypeString:
val = v.String()
case events.DataTypeNumber:
val, _ = v.Float()
case events.DataTypeBinary:
val = v.Binary()
case events.DataTypeBoolean:
val = v.Boolean()
case events.DataTypeNull:
val = nil
case events.DataTypeList:
list := make([]interface{}, len(v.List()))
for _, item := range v.List() {
list = append(list, extractVal(item))
}
val = list
case events.DataTypeMap:
mapAttr := make(map[string]interface{}, len(v.Map()))
for k, v := range v.Map() {
mapAttr[k] = extractVal(v)
}
val = mapAttr
case events.DataTypeBinarySet:
set := make([][]byte, len(v.BinarySet()))
for _, item := range v.BinarySet() {
set = append(set, item)
}
val = set
case events.DataTypeNumberSet:
set := make([]string, len(v.NumberSet()))
for _, item := range v.NumberSet() {
set = append(set, item)
}
val = set
case events.DataTypeStringSet:
set := make([]string, len(v.StringSet()))
for _, item := range v.StringSet() {
set = append(set, item)
}
val = set
}
return val
}```
Just playing with aws sdk for go. When listing resources of different types I tend to have alot of very similar functions like the two in the example bellow.
Is there a way to rewrite them as one generic function that will return a specific type depending on what is passed on as param?
Something like:
func generic(session, funcToCall, t, input) (interface{}, error) {}
currently I have to do this (functionality is the same just types change):
func getVolumes(s *session.Session) ([]*ec2.Volume, error) {
client := ec2.New(s)
t := []*ec2.Volume{}
input := ec2.DescribeVolumesInput{}
for {
result, err := client.DescribeVolumes(&input)
if err != nil {
return nil, err
}
t = append(t, result.Volumes...)
if result.NextToken != nil {
input.NextToken = result.NextToken
} else {
break
}
}
return t, nil
}
func getVpcs(s *session.Session) ([]*ec2.Vpc, error) {
client := ec2.New(s)
t := []*ec2.Vpc{}
input := ec2.DescribeVpcsInput{}
for {
result, err := client.DescribeVpcs(&input)
if err != nil {
return nil, err
}
t = append(t, result.Vpcs...)
if result.NextToken != nil {
input.NextToken = result.NextToken
} else {
break
}
}
return t, nil
}
Because you only deal with functions it is possible to use the reflect package to generate functions at runtime.
Using the object type (Volume, Vpc) it is possible to derive all subsequents information to provide a fully generic implementation that is really dry, at the extent at the being more complex and slower.
It is untested, you are welcome to help in testing and fixing it, but something like this should put you on the track
https://play.golang.org/p/mGjtYVG2OZS
The registry idea come from this answer https://stackoverflow.com/a/23031445/4466350
for reference the golang documentation of the reflect package is at https://golang.org/pkg/reflect/
package main
import (
"errors"
"fmt"
"reflect"
)
func main() {
fmt.Printf("%T\n", getter(Volume{}))
fmt.Printf("%T\n", getter(Vpc{}))
}
type DescribeVolumesInput struct{}
type DescribeVpcs struct{}
type Volume struct{}
type Vpc struct{}
type Session struct{}
type Client struct{}
func New(s *Session) Client { return Client{} }
var typeRegistry = make(map[string]reflect.Type)
func init() {
some := []interface{}{DescribeVolumesInput{}, DescribeVpcs{}}
for _, v := range some {
typeRegistry[fmt.Sprintf("%T", v)] = reflect.TypeOf(v)
}
}
var errV = errors.New("")
var errType = reflect.ValueOf(&errV).Elem().Type()
var zeroErr = reflect.Zero(reflect.TypeOf((*error)(nil)).Elem())
var nilErr = []reflect.Value{zeroErr}
func getter(of interface{}) interface{} {
outType := reflect.SliceOf(reflect.PtrTo(reflect.TypeOf(of)))
fnType := reflect.FuncOf([]reflect.Type{reflect.TypeOf(new(Session))}, []reflect.Type{outType, errType}, false)
fnBody := func(input []reflect.Value) []reflect.Value {
client := reflect.ValueOf(New).Call(input)[0]
t := reflect.MakeSlice(outType, 0, 0)
name := fmt.Sprintf("Describe%TsInput", of)
descInput := reflect.New(typeRegistry[name]).Elem()
mName := fmt.Sprintf("Describe%Ts", of)
meth := client.MethodByName(mName)
if !meth.IsValid() {
return []reflect.Value{
t,
reflect.ValueOf(fmt.Errorf("no such method %q", mName)),
}
}
for {
out := meth.Call([]reflect.Value{descInput.Addr()})
if len(out) > 0 {
errOut := out[len(out)-1]
if errOut.Type().Implements(errType) && errOut.IsNil() == false {
return []reflect.Value{t, errOut}
}
}
result := out[1]
fName := fmt.Sprintf("%Ts", of)
if x := result.FieldByName(fName); x.IsValid() {
t = reflect.AppendSlice(t, x)
} else {
return []reflect.Value{
t,
reflect.ValueOf(fmt.Errorf("field not found %q", fName)),
}
}
if x := result.FieldByName("NextToken"); x.IsValid() {
descInput.FieldByName("NextToken").Set(x)
} else {
break
}
}
return []reflect.Value{t, zeroErr}
}
fn := reflect.MakeFunc(fnType, fnBody)
return fn.Interface()
}
Proxying 3rd party API, is quite simple to implement with
go, here is how' it got implemented with endly e2e test runner AWS proxy
I would say that AWS API is perfect candidate for proxying, as long as reflection performance price is not an issue.
Some other 3rd party API like kubernetes
are much more challenging, but still quite easy to proxy with go, which is a combination of reflection and code generation:
I have two structs:
type A struct {
BankCode string `json:"bankCode"`
BankName string `json:"bankName"`
}
And:
type B struct {
A
extra string `json:" extra"`
}
And two slices:
listsA []A and listsB []B
I want to get bankCodes from listA and listB. bankcodes only contains bankcodes. It is a []string
It will be so easy as using two function.
func getBankCodes(data []A) []string {
res := make([]string, len(data))
for i := 0; i < len(data); i++ {
res[i] = data[i].BankCode
}
return res
}
func getBankCodes(data []B) []string {
res := make([]string, len(data))
for i := 0; i < len(data); i++ {
res[i] = data[i].BankCode
}
return res
}
How to use one common function ?
Well the clean solution would be to use an interface, since go doesn't support classic inheritance, so something like []parentclass can't work. Interfaces however can only describe functions not a common field, so you have to implement a Getter (essentially).
// GetBankCoder provides a function that gives the BankCode
type GetBankCoder interface {
getBankCode() string
}
// implement GetBankCoder for A (and indirectly for B)
func (a A) getBankCode() string {
return a.BankCode
}
and make your getBankCodes work on that interface type, notice the parameter of the function as well as the statement inside the loop:
func getBankCodes(data []GetBankCoder) []string { // <-- changed
res := make([]string, len(data))
for i := 0; i < len(data); i++ {
res[i] = data[i].getBankCode() // <-- changed
}
return res
}
There are other solutions where the function parameter is of interface{} type and then reflection is used to assure you can actually do .BankCode, but I don't like those, as they are not adding more clarity either.
... However, I couldn't get the golang playground to make this work correctly without putting it into a []GetBankCoder var first, before giving it to the function.
banks := make([]GetBankCoder, 0)
banks = append(banks, A{ BankCode: "ABC", BankName: "ABC Bank"})
getBankCodes(banks)
You may use one common function like so:
func BankCodes(data interface{}) []string {
if reflect.TypeOf(data).Kind() != reflect.Slice {
panic("err: data is not slice")
}
slice := reflect.Indirect(reflect.ValueOf(data))
res := make([]string, slice.Len())
for i := 0; i < slice.Len(); i++ {
a := slice.Index(i).Interface().(BankCoder)
res[i] = a.Bankcode()
}
return res
}
Code (try on The Go Playground):
package main
import (
"fmt"
"reflect"
)
func main() {
bs := []B{B{A{"BC1", "BN"}, "e"}, B{A{"BC2", "BN"}, "e"}}
strs := BankCodes(bs)
fmt.Println(strs)
as := []A{A{"AC1", "BN"}, A{"AC2", "BN"}}
strs2 := BankCodes(as)
fmt.Println(strs2)
}
func BankCodes(data interface{}) []string {
if reflect.TypeOf(data).Kind() != reflect.Slice {
panic("err: data is not slice")
}
slice := reflect.Indirect(reflect.ValueOf(data))
res := make([]string, slice.Len())
for i := 0; i < slice.Len(); i++ {
a := slice.Index(i).Interface().(BankCoder)
res[i] = a.Bankcode()
}
return res
}
type A struct {
BankCode string `json:"bankCode"`
BankName string `json:"bankName"`
}
type B struct {
A
extra string `json:" extra"`
}
type BankCoder interface {
Bankcode() string
}
func (a A) Bankcode() string {
return a.BankCode
}