I want to scan AWS DynamoDB table and then pull only a certain value. Here is my code:
package main
import (
"fmt"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/dynamodb"
)
func main() {
svc := dynamodb.New(session.New(), &aws.Config{Region: aws.String("us-west-2")})
params := &dynamodb.ScanInput{
TableName: aws.String("my_Dynamo_table_name"),
Limit: aws.Int64(2),
}
resp, err := svc.Scan(params)
if err != nil {
fmt.Println(err.Error())
return
}
fmt.Println(resp)
}
and The output is:
{
Count: 2,
Items: [{
update_time: {
N: "1466495096"
},
create_time: {
N: "1465655549"
}
},{
update_time: {
N: "1466503947"
},
create_time: {
N: "1466503947"
}
}],
LastEvaluatedKey: {
Prim_key: {
S: "1234567890"
}
},
ScannedCount: 2
}
Now, I want to retrieve the update_time value for all elements in above output. Here are my attempts:
for _, value := range resp.Items {
fmt.Println(value["create_time"]["N"])
}
and
for _, value := range resp.Items {
fmt.Println(value.create_time.N)
}
and
for _, value := range resp.Items {
fmt.Println(*value.create_time.N)
}
All above attempts error out with /var/tmp/dynamo.go:37: invalid operation: error.
I am from perl/python background and recently started learning golang.
How to retrieve nested map/array values in this case. Also, any reading references would be of great help. My google search did not reveal anything relevant.
The value of resp above is of the type *ScanOutput, which has Items type as []map[string]*AttributeValue.
To access update_time, you can try:
updateTimes := make([]string, 0)
// Items is a slice of map of type map[string]*AttributeValue
for _, m := range resp.Items {
// m is of type map[string]*AttributeValue
timeStrPtr := *m["update_time"].N
updateTimes = append(updateTimes, *timeStrPtr)
}
updateTimes should now contains all the "update_time" values as strings.
More details here.
You should use the dynamodbattribute package. it's cheaper, safer, and more readable.
Following your example:
type Row struct {
CreateTime int `dynamodbav:"create_time"`
UpdateTime int `dynamodbav:"update_time"`
}
// ...
rows := make([]*Row, len(resp.Items))
if err := dynamodbattribute.Unmarshal(resp.Items, &rows); err != nil {
// handle the error
}
// access the data
for _, row := range rows {
fmt.Println(row.CreateTime, row.UpdateTime)
}
Related
I want to validate my input map of array with govalidator.ValidateMap.
Please can someone suggest for Sample mapTemplate for map of array.
Please find below the code snippet.
Thanks in Advance
package main
import (
"fmt"
"github.com/asaskevich/govalidator"
)
func main() {
var mapTemplate = map[string]interface{}{
"name": "required,alpha",
"categories": []interface{}{",alpha"}, //error: map validator has to be either map[string]interface{} or string; got []interface {}
}
var inputMap = map[string]interface{}{
"name": "Prabhu",
"categories": []interface{}{"category1", "category2"},
}
result, err := govalidator.ValidateMap(inputMap, mapTemplate)
if err != nil {
fmt.Println("error: " + err.Error())
}
fmt.Printf("result : %v\n", result)
for _, v := range inputMap["categories"].([]interface{}) {
fmt.Printf("category : %v\n", v)
}
}
It seems validation of slices has not yet been implemented. There is no check in the What to contribute list for slices/arrays.
You can however use the function ValidateArray to iterate over a slice and validate its members.
govalidator.ValidateArray(inputMap["categories"], func(val interface{}, i int) bool {
valStr, ok := val.(string)
if !ok {
return false
}
return govalidator.IsAlpha(valStr)
})
I have a case where I'm I have to support multiple versions. Each one has different data so I create 2 structs. Based on the version I will return 1 of the structs. Once I identify which struct, I then would request the data and Unmarshal into the struct. However, Since that struct satisfies an interface, I dont think the unmarshal is working correctly. I always get the zero value for the sctruct
package main
import (
"encoding/json"
"fmt"
)
// Ten300 ...
type Ten300 struct {
Map string `json:"map"`
Enabled string `json:"enabled"`
}
// Ten400 ...
type Ten400 struct {
Block1 int `json:"block_1"`
Block2 int `json:"block_2"`
}
// NET ...
type NET struct {
CMD Commands
S TenIft
}
// Commands ...
type Commands struct {
Get string
}
// TenIft ...
type TenIft interface {
Get(string) error
}
// Get ...
func (n *Ten300) Get(module string) error {
fmt.Println("Ten300", "Get()")
return nil
}
// Get ...
func (n *Ten400) Get(module string) error {
fmt.Println("Ten400", "Get()")
n.Block2 = 100
return nil
}
// TenGen easy to read fun type
type TenGen func() NET
// VersionSetup is a map of possible version
var VersionSetup = map[string]TenGen{
"3.0.0": func() NET {
return NET{
CMD: Commands{
Get: "getConfig",
},
S: &Ten300{},
}
},
"4.0.0": func() NET {
return NET{
CMD: Commands{
Get: "getSettings",
},
S: &Ten400{},
}
},
}
const input300 = `{
"map": "one",
"enabled": "two"
}`
const input400 = `{
"block_1": 1,
"block_2": 2
}`
// Setup ...
func (n *NET) Setup() error {
// This Switch is just to use hard coded data
var input string
switch n.S.(type) {
case *Ten400:
input = input400
case *Ten300:
input = input300
}
err := json.Unmarshal([]byte(input), n.S)
if err != nil {
fmt.Println("returning err:", err)
return err
}
fmt.Printf("n.s type: %T\nn.s value: %+v\n", n.S, n.S)
n.S.Get("xxx")
return nil
}
func main() {
version := "3.0.0"
if f, ok := VersionSetup[version]; ok {
net := f()
err := net.Setup()
if err != nil {
panic(err)
}
}
version = "4.0.0"
if f, ok := VersionSetup[version]; ok {
net := f()
err := net.Setup()
if err != nil {
panic(err)
}
}
}
Added go-playground ... this seems to work, but is this the best solution?
I'm trying to deserialize an AWS struct in go so that I can pass in parameters and output values based on those parameters, for example;
func DisplayResults(conf *config.Configuration,
regionalData []*ec2.DescribeInstancesOutput) {
log.Debug("Displaying results")
log.Debug("Table view [%v]", conf.Display)
for _, rv := range regionalData {
for _, reservation := range rv.Reservations {
for _, instance := range reservation.Instances {
var i map[string]interface{}
json.Unmarshal(instance, i)
}
}
}
}
I've also tried using:
"github.com/fatih/structs"
i := structs.Map(instance)
log.Print("%v", i)
however, I think that I need to somehow derefence the struct because my output looks like this:
RamdiskId:<nil> SriovNetSupport:<nil> VpcId:0xc420321ba0 State:map[Code:0xc42031ea08 Name:0xc420321890] VirtualizationType:0xc420321b90 CapacityReservationId:<nil> ClientToken:0xc420321560 HibernationOptions:map[Configured:0xc42031e96b] IamInstanceProfile:map[Arn:0xc4203215a0 Id:0xc4203215b0] ImageId:0xc4203215c0
I've also tried this:
func DisplayResults(conf *config.Configuration, regionalData []*ec2.DescribeInstancesOutput, parameters []string) {
log.Debug("Displaying results")
log.Debug("Table view [%v]", conf.Display)
for _, rv := range regionalData {
for _, reservation := range rv.Reservations {
for _, instance := range reservation.Instances {
y := deref(instance)
log.Print("%v", y["InstanceId"])
}
}
}
}
func deref(instance *ec2.Instance) ec2.Instance {
var i ec2.Instance
x := &i
*x = *instance
return i
}
Query Api and response a custom JSON, how to Unmarshal it. the sample JSON:
{"14AcKEr19gHJvgwQhK7sfFm6YJGmoZZoqu": {
"final_balance": 61914248289,
"n_tx": 3472,
"total_received": 3479994002972
}}
The key is a hex string. So how to handle it with golang convention, anyone can help me?
Below is my try test code:
c.OnResponse(func(r *colly.Response) {
jsonData := r.Body
fmt.Println(string(jsonData))
fmt.Println("==================")
//parse bitcoin json
jsonMap := make(map[string]interface{})
err := json.Unmarshal([]byte(jsonData), &jsonMap)
if err != nil {
panic(err)
}
fmt.Println(jsonMap)
dumpMap("", jsonMap)
})
func dumpMap(space string, m map[string]interface{}) {
for k, v := range m {
if mv, ok := v.(map[string]interface{}); ok {
fmt.Printf("{ \"%v\": \n", k)
dumpMap(space+"\t", mv)
fmt.Printf("}\n")
} else {
fmt.Printf("%v %v : %v\n", space, k, v)
}
}
}
and go run cmd/main.go, the console is print here:
{"14AcKEr19gHJvgwQhK7sfFm6YJGmoZZoqu": {
"final_balance": 75494521080,
"n_tx": 3493,
"total_received": 3493574275763
}}
==================
map[14AcKEr19gHJvgwQhK7sfFm6YJGmoZZoqu:map[n_tx:3493 total_received:3.493574275763e+12 final_balance:7.549452108e+10]]
{ "14AcKEr19gHJvgwQhK7sfFm6YJGmoZZoqu":
final_balance : 7.549452108e+10
n_tx : 3493
total_received : 3.493574275763e+12
}
Do I need customised unmarshal func to get string key? If I use 14AcKEr19gHJvgwQhK7sfFm6YJGmoZZoqu as key I can't easily to access. I just want to know how handle it.
you can unmarshal it into map, so you can get generated key as a key of map
https://play.golang.org/p/IfEjjvKakpu
package main
import (
"encoding/json"
"fmt"
"log"
)
var input = `{"14AcKEr19gHJvgwQhK7sfFm6YJGmoZZoqu": {
"final_balance": 61914248289,
"n_tx": 3472,
"total_received": 3479994002972
}}`
type object struct {
FinalBalance uint64 `json:"final_balance"`
NTX uint64 `json:"n_tx"`
TotalReceived uint64 `json:"total_received"`
}
func main() {
var result map[string]object;
err := json.Unmarshal([]byte(input), &result);
if err != nil {
log.Fatal(err)
}
fmt.Printf("result: %+v", result)
// result: map[14AcKEr19gHJvgwQhK7sfFm6YJGmoZZoqu:{FinalBalance:61914248289 NTX:3472 TotalReceived:3479994002972}]
}
I'm trying to set-up an AWS-lambda using aws-sdk-go that is triggered whenever a new user is added to a certain dynamodb table.
Everything is working just fine but I can't find a way to unmarshal a map map[string]DynamoDBAttributeValue like:
{
"name": {
"S" : "John"
},
"residence_address": {
"M": {
"address": {
"S": "some place"
}
}
}
}
To a given struct, for instance, a User struct. Here is shown an example of unsmarhaling a map[string]*dynamodb.AttributeValue into a given interface, but I can't find a way to do the same thing with map[string]DynamoDBAttributeValue even though these types seem to fit the same purposes.
map[string]DynamoDBAttributeValue is returned by a events.DynamoDBEvents from package github.com/aws/aws-lambda-go/events. This is my code:
package handler
import (
"context"
"github.com/aws/aws-lambda-go/events"
"github.com/aws/aws-sdk-go/service/dynamodb/dynamodbattribute"
"github.com/aws/aws-sdk-go/service/dynamodb"
)
func HandleDynamoDBRequest(ctx context.Context, e events.DynamoDBEvent) {
for _, record := range e.Records {
if record.EventName == "INSERT" {
// User Struct
var dynamoUser model.DynamoDBUser
// Of course this can't be done for incompatible types
_ := dynamodbattribute.UnmarshalMap(record.Change.NewImage, &dynamoUser)
}
}
}
Of course, I can marshal record.Change.NewImage to JSON and unmarshal it back to a given struct, but then, I would have to manually initialize dynamoUser attributes starting from the latter ones.
Or I could even write a function that parses map[string]DynamoDBAttributeValue to map[string]*dynamodb.AttributeValue like:
func getAttributeValueMapFromDynamoDBStreamRecord(e events.DynamoDBStreamRecord) map[string]*dynamodb.AttributeValue {
image := e.NewImage
m := make(map[string]*dynamodb.AttributeValue)
for k, v := range image {
if v.DataType() == events.DataTypeString {
s := v.String()
m[k] = &dynamodb.AttributeValue{
S : &s,
}
}
if v.DataType() == events.DataTypeBoolean {
b := v.Boolean()
m[k] = &dynamodb.AttributeValue{
BOOL : &b,
}
}
// . . .
if v.DataType() == events.DataTypeMap {
// ?
}
}
return m
}
And then simply use dynamodbattribute.UnmarshalMap, but on events.DataTypeMap it would be quite a tricky process.
Is there a way through which I can unmarshal a DynamoDB record coming from a events.DynamoDBEvent into a struct with a similar method shown for map[string]*dynamodb.AttributeValue?
I tried the function you provided, and I met some problems with events.DataTypeList, so I managed to write the following function that does the trick:
// UnmarshalStreamImage converts events.DynamoDBAttributeValue to struct
func UnmarshalStreamImage(attribute map[string]events.DynamoDBAttributeValue, out interface{}) error {
dbAttrMap := make(map[string]*dynamodb.AttributeValue)
for k, v := range attribute {
var dbAttr dynamodb.AttributeValue
bytes, marshalErr := v.MarshalJSON(); if marshalErr != nil {
return marshalErr
}
json.Unmarshal(bytes, &dbAttr)
dbAttrMap[k] = &dbAttr
}
return dynamodbattribute.UnmarshalMap(dbAttrMap, out)
}
I was frustrated that the type of NewImage from the record wasn't map[string]*dynamodb.AttributeValue so I could use the dynamodbattribute package.
The JSON representation of events.DynamoDBAttributeValue seems to be the same as the JSON represenation of dynamodb.AttributeValue.
So I tried creating my own DynamoDBEvent type and changed the type of OldImage and NewImage, so it would be marshalled into map[string]*dynamodb.AttributeValue instead of map[string]events.DynamoDBAttributeValue
It is a little bit ugly but it works for me.
package main
import (
"github.com/aws/aws-lambda-go/events"
"github.com/aws/aws-lambda-go/lambda"
"github.com/aws/aws-sdk-go/service/dynamodb"
"github.com/aws/aws-sdk-go/service/dynamodb/dynamodbattribute"
"fmt"
)
func main() {
lambda.Start(lambdaHandler)
}
// changed type of event from: events.DynamoDBEvent to DynamoDBEvent (see below)
func lambdaHandler(event DynamoDBEvent) error {
for _, record := range event.Records {
change := record.Change
newImage := change.NewImage // now of type: map[string]*dynamodb.AttributeValue
var item IdOnly
err := dynamodbattribute.UnmarshalMap(newImage, &item)
if err != nil {
return err
}
fmt.Println(item.Id)
}
return nil
}
type IdOnly struct {
Id string `json:"id"`
}
type DynamoDBEvent struct {
Records []DynamoDBEventRecord `json:"Records"`
}
type DynamoDBEventRecord struct {
AWSRegion string `json:"awsRegion"`
Change DynamoDBStreamRecord `json:"dynamodb"`
EventID string `json:"eventID"`
EventName string `json:"eventName"`
EventSource string `json:"eventSource"`
EventVersion string `json:"eventVersion"`
EventSourceArn string `json:"eventSourceARN"`
UserIdentity *events.DynamoDBUserIdentity `json:"userIdentity,omitempty"`
}
type DynamoDBStreamRecord struct {
ApproximateCreationDateTime events.SecondsEpochTime `json:"ApproximateCreationDateTime,omitempty"`
// changed to map[string]*dynamodb.AttributeValue
Keys map[string]*dynamodb.AttributeValue `json:"Keys,omitempty"`
// changed to map[string]*dynamodb.AttributeValue
NewImage map[string]*dynamodb.AttributeValue `json:"NewImage,omitempty"`
// changed to map[string]*dynamodb.AttributeValue
OldImage map[string]*dynamodb.AttributeValue `json:"OldImage,omitempty"`
SequenceNumber string `json:"SequenceNumber"`
SizeBytes int64 `json:"SizeBytes"`
StreamViewType string `json:"StreamViewType"`
}
I have found the same problem and the solution is to perform a simple conversion of types. This is possible because in the end the type received by lambda events events.DynamoDBAttributeValue and the type used by the SDK V2 of AWS DynamoDB types.AttributeValue are the same. Next I show you the conversion code.
package aws_lambda
import (
"github.com/aws/aws-lambda-go/events"
"github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue"
"github.com/aws/aws-sdk-go-v2/service/dynamodb/types"
)
func UnmarshalDynamoEventsMap(
record map[string]events.DynamoDBAttributeValue, out interface{}) error {
asTypesMap := DynamoDbEventsMapToTypesMap(record)
err := attributevalue.UnmarshalMap(asTypesMap, out)
if err != nil {
return err
}
return nil
}
func DynamoDbEventsMapToTypesMap(
record map[string]events.DynamoDBAttributeValue) map[string]types.AttributeValue {
resultMap := make(map[string]types.AttributeValue)
for key, rec := range record {
resultMap[key] = DynamoDbEventsToTypes(rec)
}
return resultMap
}
// DynamoDbEventsToTypes relates the dynamo event received by AWS Lambda with the data type that is
// used in the Amazon SDK V2 to deal with DynamoDB data.
// This function is necessary because Amazon does not provide any kind of solution to make this
// relationship between types of data.
func DynamoDbEventsToTypes(record events.DynamoDBAttributeValue) types.AttributeValue {
var val types.AttributeValue
switch record.DataType() {
case events.DataTypeBinary:
val = &types.AttributeValueMemberB{
Value: record.Binary(),
}
case events.DataTypeBinarySet:
val = &types.AttributeValueMemberBS{
Value: record.BinarySet(),
}
case events.DataTypeBoolean:
val = &types.AttributeValueMemberBOOL{
Value: record.Boolean(),
}
case events.DataTypeList:
var items []types.AttributeValue
for _, value := range record.List() {
items = append(items, DynamoDbEventsToTypes(value))
}
val = &types.AttributeValueMemberL{
Value: items,
}
case events.DataTypeMap:
items := make(map[string]types.AttributeValue)
for k, v := range record.Map() {
items[k] = DynamoDbEventsToTypes(v)
}
val = &types.AttributeValueMemberM{
Value: items,
}
case events.DataTypeNull:
val = nil
case events.DataTypeNumber:
val = &types.AttributeValueMemberN{
Value: record.Number(),
}
case events.DataTypeNumberSet:
val = &types.AttributeValueMemberNS{
Value: record.NumberSet(),
}
case events.DataTypeString:
val = &types.AttributeValueMemberS{
Value: record.String(),
}
case events.DataTypeStringSet:
val = &types.AttributeValueMemberSS{
Value: record.StringSet(),
}
}
return val
}
There is a package that allows conversion from events.DynamoDBAttributeValue to dynamodb.AttributeValue
https://pkg.go.dev/github.com/aereal/go-dynamodb-attribute-conversions/v2
From there one can unmarshal AttributeValue into struct
func Unmarshal(attribute map[string]events.DynamoDBAttributeValue, out interface{}) error {
av := ddbconversions.AttributeValueMapFrom(attribute)
return attributevalue.UnmarshalMap(av, out)
}