dynamodbattribute.MarshalMap returns empty map - go

I'm attempting to write a really simple Go function to insert an entry into a DynamoDB table.
I'm following the tutorial provided on the AWS Documentation site, but for some reason, the function dynamodbattribute.MarshalMap is returning an empty map.
The code compiles and runs, but breaks when attempting to insert the record as it can't find the required keys in the map. A little println action shows that the map is empty, even though the struct it's created from isn't.
Any help is appreciated!
package main
import (
"context"
"fmt"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-lambda-go/lambda"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/dynamodb"
"github.com/aws/aws-sdk-go/service/dynamodb/dynamodbattribute"
)
// Item Struct for Fruit Item
type Item struct {
fruitID int
fruitName string
}
/*HandleRequest handles the lambda request.*/
func HandleRequest(ctx context.Context) (string, error) {
// Initialize a session that the SDK will use to load
// credentials from the shared credentials file ~/.aws/credentials
// and region from the shared configuration file ~/.aws/config.
sess := session.Must(session.NewSessionWithOptions(session.Options{
SharedConfigState: session.SharedConfigEnable,
}))
// Create DynamoDB client
svc := dynamodb.New(sess)
item := Item{
fruitID: 2,
fruitName: "Orange",
}
fmt.Println(item)
av, err := dynamodbattribute.MarshalMap(item)
if err != nil {
fmt.Println("Got error marshalling new fruit item: ")
fmt.Println(err.Error())
return "", err
}
fmt.Println("av: ", av)
fmt.Println("ln: ", len(av))
fmt.Println("KEY VALUE PAIRS::")
for key, value := range av {
fmt.Println("Key:", key, "Value:", value)
}
tableName := "fruits"
input := &dynamodb.PutItemInput{
Item: av,
TableName: aws.String(tableName),
}
_, err = svc.PutItem(input)
if err != nil {
fmt.Println("Got error inserting new fruit item: ")
fmt.Println(err.Error())
return "", err
}
return "SUCCESS!", nil
}
func main() {
lambda.Start(HandleRequest)
}
Log output from Lambda
START RequestId: 82f120d9-6bba-40d9-9204-136d491dbb88 Version: $LATEST
{2 Orange}
av: map[]
ln: 0
KEY VALUE PAIRS::
Got error inserting new fruit item:
ValidationException: One or more parameter values were invalid: Missing the key fruitId in the item
status code: 400, request id: QUJG8R8EHPO65O1JFVI98PRQ1VVV4KQNSO5AEMVJF66Q9ASUAAJG
ValidationException: One or more parameter values were invalid: Missing the key fruitId in the item
status code: 400, request id: QUJG8R8EHPO65O1JFVI98PRQ1VVV4KQNSO5AEMVJF66Q9ASUAAJG: requestError
null
END RequestId: 82f120d9-6bba-40d9-9204-136d491dbb88

The fields in struct starts with Capital letter is only exported and can be used in outside of this package.
Can you try with this struct and see if that works ?
type Item struct {
FruitID int
FruitName string
}

Related

How to fix error with Query dynamodb request?

In DynamoDB I Have a table that contains:
- email (primary key)
- password (attribute)
- rname (attribute)
I'm using V1 of the AWS Go SDK, to implement to perform a query using just the primary key to my database:
My struct to unMarshal to is:
type Item struct {
Email string `json:"email"`
Password string `json:"password"`
Rname string `json:"rname"`
}
and the code:
result, err := client.Query(&dynamodb.QueryInput{
TableName: aws.String("accountsTable"),
KeyConditions: map[string]*dynamodb.Condition{
"email": {
ComparisonOperator: aws.String("EQ"),
AttributeValueList: []*dynamodb.AttributeValue{
{
S: aws.String(email),
},
},
},
},
})
if err != nil {
return false, err
}
item := []Item{}
err = dynamodbattribute.UnmarshalListOfMaps(result.Items, &item)
if err != nil {
return false, err
}
However, I get the issue that the key is invalid. I check the key in the database and it matches the one i print out to the console too.
Not sure how to get round this issue as example's i've seen seem to work for their's and look the same.
Any help in fixing this issue would be appreciated thanks :)
You need to set the values of password and rname to omitempty so that it's not set to empty values as they are not keys they should not be included on a Query as it throws an invalid key exception:
type Item struct {
Email string `json:"email" dynamodbav:"email,omitempty"`
Password string `json:"password" dynamodbav:"password,omitempty"`
Rname string `json:"rname" dynamodbav:"rname,omitempty"`
}
Update
I believe the issue is due to the fact you try to marshall the entire response in a single command, however, iterating works for me. (I do not use Go).
package main
import (
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/dynamodb"
"github.com/aws/aws-sdk-go/service/dynamodb/dynamodbattribute"
"fmt"
)
func main() {
// Create Session
sess := session.Must(session.NewSessionWithOptions(session.Options{
SharedConfigState: session.SharedConfigEnable,
}))
// Create DynamoDB Client with Logging
client := dynamodb.New(sess, aws.NewConfig())
type Item struct {
Email string `dynamodbav: "email"`
Password string `dynamodbav: "password,omitempty"`
Rname string `dynamodbav: "rname,omitempty"`
}
result, err := client.Query(&dynamodb.QueryInput{
TableName: aws.String("accountsTable"),
KeyConditions: map[string]*dynamodb.Condition{
"email": {
ComparisonOperator: aws.String("EQ"),
AttributeValueList: []*dynamodb.AttributeValue{
{
S: aws.String("lhnng#amazon.com"),
},
},
},
},
})
if err != nil {
fmt.Println("Query API call failed:")
fmt.Println((err.Error()))
}
for _, i := range result.Items {
item := Item{}
err = dynamodbattribute.UnmarshalMap(i, &item)
if err != nil {
fmt.Println("Got error unmarshalling: %s", err)
}
fmt.Println("Email: ", item.Email)
fmt.Println()
}
}
Moreover, as you use a single key of email, it means there is at most 1 item with the same email address, meaning you should use GetItem rather than Query:
package main
import (
"fmt"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/dynamodb"
"github.com/aws/aws-sdk-go/service/dynamodb/dynamodbattribute"
)
func main() {
// Item to Get
type Item struct {
Email string `dynamodbav: "email"`
Password string `dynamodbav: "password,omitempty"`
Rname string `dynamodbav: "rname,omitempty"`
}
// Create Session
sess := session.Must(session.NewSessionWithOptions(session.Options{
SharedConfigState: session.SharedConfigEnable,
}))
// Create DynamoDB Client
client := dynamodb.New(sess, aws.NewConfig())
// Get Item
result, err := client.GetItem(&dynamodb.GetItemInput{
TableName: aws.String("accountsTable"),
Key: map[string]*dynamodb.AttributeValue{
"email": {
S: aws.String("lhnng#amazon.com"),
},
},
})
// Catch Error
if err != nil {
fmt.Println("GetItem API call failed:")
fmt.Println((err.Error()))
}
item := Item{}
// Unmarhsall
err = dynamodbattribute.UnmarshalMap(result.Item, &item)
if err != nil {
panic(fmt.Sprintf("Failed to unmarshal Record, %v", err))
}
// If Item Returns Empty
if item.Email == "" {
fmt.Println("Could not find Item")
return
}
// Print Result
fmt.Println("Found item:")
fmt.Println("Email: ", item.Email)
}

Add root element to existing Json in Go lang

I'm trying to add string "Employee" to my existing JSON response. Also, we need to be able to generate this version of json based on an user condition. Only if the user condition is met, I need to generate second version of json with string "Employee" added. If not the first version without string "Employee" should be generated. How can I achieve it with out updating the existing struct and how can I check this with if clause to check for the condition and then generate json based on it?
Below is my existing json response in go
[
{
"EmpId":{
"String":"ABCD",
"Valid":true
},
"Department":{
"Float64":0,
"Valid":true
}
}
]
How can I get my json response like below with out changing existing struct based on input parameter?
{
"Employee":[
{
"EmpId":{
"String":"ABCD",
"Valid":true
},
"Department":{
"Float64":0,
"Valid":true
}
}
]
}
Below is my code:
Step 1: model folder
type EmployeeWithRoot struct {
Employee []Employee
}
type Employee struct {
EmpNbr sql.NullString `json:"EmpNbr"`
DateofJoin sql.NullString `json:"DateofJoin"`
DeptId sql.NullString `json:"DeptId"`
DeptName sql.NullString `json:"DeptName"`
}
Step 2: code folder
func GetEmpDetails(logRequestId string, logNestedLevel int, EmpNbr string, DateofJoin string) ([]model.EmployeeWithRoot, error) {
logFunctionFunctionName := "code.GetEmpDetails"
logStartTime := time.Now()
logNestedLevel++
defer configurations.TimeTrack(logFunctionFunctionName, logRequestId, logStartTime, logNestedLevel)
rows, err := db.Query(utils.SELECT_OF_EMP_AGGR, EmpNbr, DateofJoin, DeptId, DeptName)
if err != nil {
return nil, err
}
defer rows.Close()
var e []model.EmployeeWithRoot
for rows.Next() {
var x model.EmployeeWithRoot
err := rows.Scan(&x.Employee.EmpNbr, &x.Employee.DateofJoin, &x.Employee.DeptId,&x.Employee.DeptName)
if err != nil {
return nil, err
}
e = append(e, x)
}
err = rows.Err()
if err != nil {
return nil, err
}
return e, nil
}
STEP 3: API folder
Employee, err := code.GetEmpDetails(logRequestId, logNestedLevel, EmpNbr, DateofJoin)
if err != nil {
log.Panic(err)
}
marshalDataForRequestContentType(logRequestId, logNestedLevel, w, r, Employee)
I'm getting the below error.
x.Employee.EmpNbr undefined (type []model.Employee has no field or method EmpNbr)
x.Employee.DateofJoin undefined (type []model.Employee has no field or method DateofJoin)enter code here
x.Employee.DeptId undefined (type []model.Employee has no field or method DeptId)
x.Employee.DeptName undefined (type []model.Employee has no field or method DeptName)
Considering you're just wrapping it it an outer object, I don't see any reason you'd need to change the existing struct, just wrap it in a new one. I'll have to make some guesses/assumptions here since you've only shown the JSON and not the Go code that produces it, but assuming your existing JSON is produced by marshaling something like var response []Employee, the desired JSON could be produced in your condition by marshaling instead:
json.Marshal(struct{Employee []Employee}{response})
Working example: https://go.dev/play/p/vwDvxnQ96G_2
Use string concatenation:
func addRoot(json string) string {
return `{ "Employee":` + json + `}`
}
Run an example on the GoLang playground.
Here's the code if you are working with []byte instead of string:
func addRoot(json []byte) []byte {
const prefix = `{ "Employee":`
const suffix = `}`
result := make([]byte, 0, len(prefix)+len(json)+len(suffix))
return append(append(append(result, prefix...), json...), suffix...)
}
Run this example on the GoLang playground.
If you have some JSON in a byte slice ([]byte) then you can just add the outer element directly - e.g. (playground):
existingJSON := []byte(`[
{
"EmpId":{
"String":"ABCD",
"Valid":true
},
"Department":{
"Float64":0,
"Valid":true
}
}
]`)
b := bytes.NewBufferString(`{"Employee":`)
b.Write(existingJSON)
b.WriteString(`}`)
fmt.Println(string(b.Bytes()))
If this is not what you are looking for please add further details to your question (ideally your attempt as a minimal, reproducible, example)

How to Delete Multiple Items from a DynamoDB Table in Go

I've a DynamoDB table that contains items like this:
type AuthEntry struct {
UserID string `dynamodbav:"userId"`
Token string `dynamodbav:"token"`
CreatedOn time.Time `dynamodbav:"createdOn"`
}
I need to delete all the AuthEntry items older than 5 minutes (CreatedOn < now - 5 mins) and without a token (Token is empty). It is clear to me how to remove one item at a time... but I'm wondering how to delete multiple items in one shot. Thank u very much.
I was looking for an example like the one here below... and I hope it helps other newbie like me. For instance, first I use Scan to retrieve the expired entries, and then I run BatchWriteItemInput to actually delete them.
import (
"context"
"time"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue"
"github.com/aws/aws-sdk-go-v2/service/dynamodb"
"github.com/aws/aws-sdk-go-v2/service/dynamodb/types"
)
var tableName = "USER_AUTH"
...
type AuthRepository struct {
ctx context.Context
svc *dynamodb.Client
}
...
func NewAuthRepository(ctx context.Context) (*AuthRepository, error) {
cfg, err := config.LoadDefaultConfig(ctx)
if err != nil {
return nil, err
}
return &AuthRepository{ctx, dynamodb.NewFromConfig(cfg)}, nil
}
...
func (r *AuthRepository) Collect(maxAge int) (int32, error) {
t := time.Now().Add(time.Duration(maxAge*-1) * time.Millisecond).UTC()
params := &dynamodb.ScanInput{
TableName: aws.String(tableName),
ProjectionExpression: aws.String("userId"),
ExpressionAttributeValues: map[string]types.AttributeValue{
"t": &types.AttributeValueMemberS{*aws.String(t.String())},
},
FilterExpression: aws.String("createdOn < :t"),
}
result, err := r.svc.Scan(r.ctx, params)
if err != nil {
return 0, err
}
wr := make([]types.WriteRequest, result.Count)
for _, v := range result.Items {
authEntry := &AuthEntry{}
if err := attributevalue.UnmarshalMap(v, &authEntry); err != nil {
return 0, err
}
wr = append(wr, types.WriteRequest{
DeleteRequest: &types.DeleteRequest{
Key: map[string]types.AttributeValue{
"userId": &types.AttributeValueMemberS{*aws.String(authEntry.UserID)},
},
}})
}
input := &dynamodb.BatchWriteItemInput{
RequestItems: map[string][]types.WriteRequest{
tableName: wr,
},
}
_, err = r.svc.BatchWriteItem(r.ctx, input)
return result.Count, nil
}
When it comes to deletion, you have a few options.
deleteItem - Deletes a single item in a table by primary key.
batchWriteItem - The BatchWriteItem operation puts or deletes multiple items in one or more tables. A single call to BatchWriteItem can write up to 16 MB of data, which can comprise as many as 25 put or delete requests
TimeToLive - You can utilize DynamoDBs Time To Live (TTL) feature to delete items you no longer need. Keep in mind that TTL only marks your items for deletion and actual deletion could take up to 48 hours.
I'm not sure which items in your table are part of the primary key, so it's difficult to give you an example. However, this operation is the preferred method to delete multiple items at a time.

Golang - create an object of the same type as passed

I'm trying to build a generic function which will parse input (in JSON) into a specified structure. The structure may vary at run-time, based on parameters which are passed to the function. I'm currently trying to achieve this by passing an object of the right type and using reflect.New() to create a new output object of the same type.
I'm then parsing the JSON into this object, and scanning the fields.
If I create the object and specify the type in code, everything works. If I pass an object and try to create a replica, I get an "invalid indirect" error a few steps down (see code).
import (
"fmt"
"reflect"
"encoding/json"
"strings"
)
type Test struct {
FirstName *string `json:"FirstName"`
LastName *string `json:"LastName"`
}
func genericParser(incomingData *strings.Reader, inputStructure interface{}) (interface{}, error) {
//******* Use the line below and things work *******
//parsedInput := new(Test)
//******* Use vvv the line below and things don't work *******
parsedInput := reflect.New(reflect.TypeOf(inputStructure))
decoder := json.NewDecoder(incomingData)
err := decoder.Decode(&parsedInput)
if err != nil {
//parsing error
return nil, err
}
//******* This is the line that generates the error "invalid indirect of parsedInput (type reflect.Value)" *******
contentValues := reflect.ValueOf(*parsedInput)
for i := 0; i < contentValues.NumField(); i++ {
//do stuff with each field
fmt.Printf("Field name was: %s\n", reflect.TypeOf(parsedInput).Elem().Field(i).Name)
}
return parsedInput, nil
}
func main() {
inputData := strings.NewReader("{\"FirstName\":\"John\", \"LastName\":\"Smith\"}")
exampleObject := new(Test)
processedData, err := genericParser(inputData, exampleObject)
if err != nil {
fmt.Println("Parsing error")
} else {
fmt.Printf("Success: %v", processedData)
}
}
If I can't create a replica of the object, then a way of updating / returning the one supplied would be feasible. The key thing is that this function must be completely agnostic to the different structures available.
reflect.New isn't a direct analog to new, as it can't return a specific type, it only can return a reflect.Value. This means that you are attempting to unmarshal into a *reflect.Value, which obviously isn't going to work (even if it did, your code would have passed in **Type, which isn't what you want either).
Use parsedInput.Interface() to get the underlying value after creating the new value to unmarshal into. You then don't need to reflect on the same value a second time, as that would be a reflect.Value of a reflect.Value, which again isn't going to do anything useful.
Finally, you need to use parsedInput.Interface() before you return, otherwise you are returning the reflect.Value rather than the value of the input type.
For example:
func genericParser(incomingData io.Reader, inputStructure interface{}) (interface{}, error) {
parsedInput := reflect.New(reflect.TypeOf(inputStructure).Elem())
decoder := json.NewDecoder(incomingData)
err := decoder.Decode(parsedInput.Interface())
if err != nil {
return nil, err
}
for i := 0; i < parsedInput.Elem().NumField(); i++ {
fmt.Printf("Field name was: %s\n", parsedInput.Type().Elem().Field(i).Name)
}
return parsedInput.Interface(), nil
}
https://play.golang.org/p/CzDrj6sgQNt

How can I call dbmap.Insert(interface{}) from another function?

I have a bunch of very similar structs (A and B in the example) whose instances I want to handle in some function ( f() in the example) and then insert them into my database. I figured I could handle that with the empty interface somehow, but it seems this is not the solution as I get the error:
i: &{{6 2019-04-03 15:11:37.822100431 +0200 CEST m=+0.001291882} 7} *main.A
2019/04/03 15:11:37 Insert i no table found for type:
exit status 1
I tried to create some minimal but executable example:
package main
import (
"database/sql"
"fmt"
_ "github.com/mattn/go-sqlite3"
gorp "gopkg.in/gorp.v2"
"log"
"time"
)
type Meta struct {
Id int
CreatedAt time.Time
}
type A struct {
Meta
Value int
}
type B struct {
Meta
Value string
}
var dbmap *gorp.DbMap
func f(i interface{}) {
fmt.Printf("i: %v %T\n", i, i)
err := dbmap.Insert(&i)
checkErr(err, "Insert i")
}
func main() {
Init()
a := A{Meta: Meta{CreatedAt: time.Now()}, Value: 7}
b := B{Meta: Meta{CreatedAt: time.Now()}, Value: "seven"}
err := dbmap.Insert(&a) // works
checkErr(err, "Insert a")
err = dbmap.Insert(&b) // works
checkErr(err, "Insert b")
f(&a) // fails
}
func Init() {
db, err := sql.Open("sqlite3", "/tmp/post_db.bin")
checkErr(err, "sql.Open failed")
dbmap = &gorp.DbMap{Db: db, Dialect: gorp.SqliteDialect{}}
dbmap.AddTableWithName(A{}, "As").SetKeys(true, "Id")
dbmap.AddTableWithName(B{}, "Bs").SetKeys(true, "Id")
err = dbmap.CreateTablesIfNotExists()
checkErr(err, "Couldn't create tables")
}
func checkErr(err error, msg string) {
if err != nil {
log.Fatalln(msg, err)
}
}
What's the right way to do that? In C++ I'd simply use templates ;)
If you are calling you func like f(&a). You should call inside func f just dbmap.Insert(i), because your value is already a pointer. So your func will look like
func f(i interface{}) {
fmt.Printf("i: %v %T\n", i, i)
err := dbmap.Insert(i)
checkErr(err, "Insert i")
}

Resources