I am trying to retrieve a column from dynamodb which is saved as map[string][]string (map with key as string and values as a list of string)
example:
{
{ "string1": ["1", "2"]},
{ "string2": ["3", "4"]},
}
I am getting the data but I am not able to unmarshal it and store it in the struct
type Record struct {
listOfMap map[string][]string
}
scanInput := &dynamodb.ScanInput{
TableName: aws.String("someTable"),
ProjectionExpression: aws.String("desiredColumnToRetrieve"),
}
dynamodbOutput, err := svc.dynamodb.Scan(scanInput)
if err != nil {
log.Errorf("dynamodb.Scan() error - %s", err)
}
fmt.Println(dynamodbOutput)
records := []Record{}
err = dynamodbattribute.UnmarshalListOfMaps(dynamodbOutput.Items, &records)
if err != nil {
log.Errorf("dynamodbattribute.UnmarshalListOfMaps() error - %s", err)
}
fmt.Println(records)
dynamodb output (this output is from the above print statements):
{
Count: 2,
Items: [{
desiredColumnToRetrieve: {
M: {
string1: {
L: [{
S: "1"
},{
S: "2"
}]
}
}
}
},{
desiredColumnToRetrieve: {
M: {
string2: {
L: [{
S: "3"
},{
S: "4"
}]
}
}
}
}],
ScannedCount: 2
}
{[map[] map[]}
The data is not getting saved to the struct/map
Probably it would not work, you need to iterate over items first, like:
for _, i := range dynamodbOutput.Items {
record := Record{}
err = dynamodbattribute.UnmarshalMap(i, &record)
And in fact, your listOfMaps is regular map. Why do you call so? It's rather mapOfLists
By the way doing full scans on NoSQL tables isn't the best practice, if you are going to do always, you'd better to review partitioning sorting keys and whole data schema. Cheers.
Related
I want to write a generic function func GetVal(map[interface{}]interface{}, key interface{}) interface{}. This will take a map and a key to search for and return either the value or nil.
The map can have any data type and can go to any level of nesting. For example,
var s1 = "S1"
var s2 = "S2"
var s3 = s1 + "==" + s2 + "==S3"
var s4 = s3 + "==S4"
var nestedMap = map[interface{}]interface{}{
"data": map[interface{}]interface{}{
"TEST_KEY": "1234353453",
"metadata": map[interface{}]interface{}{
"created_time": "2022-08-06",
},
"custom_metadata": map[interface{}][]interface{}{
"destroyed": []interface{}{
&s1,
map[string]interface{}{
"auth": []interface{}{
"val3", "val4", "val45",
},
},
},
},
&s2: &[]*string{
&s1, &s2,
},
&s1: &[]int{
10, 20, 233,
},
123: &s3,
},
s3: []interface{}{
[]interface{}{
map[string]*string{
s4: &s4,
},
},
},
}
Expected return values
GetVal(nestedMap, "metadata") should return {"created_time": "2022-08-06"}
GetVal(nestedMap, "destroyed") should return
{ &s1,
map[string]interface{}{
"auth": []interface{}{
"val3", "val4", "val45",
},
},
}
Is there a way to do it without an external library?
This question looks similar to
Accessing Nested Map of Type map[string]interface{} in Golang but in my case the fields are not limited or always same
The question is kind of cryptic because the example is overcomplicated. If you want to get knowledge regarding the recurrent functions, you should start with something simpler like:
var nestedMap = map[string]any{
"k1": "v1",
"k2": map[string]any{
"nestedK1": "nestedV1",
"nestedK2": "nestedV2",
"nestedK3": map[string]any{
"superNestedK1" : "FOUND!!!",
},
},}
Otherwise, an explanation will be hard.
Then you can work on functions like:
func GetVal(data map[string]any, key string) (result any, found bool) {
for k, v := range data {
if k == key {
return v, true
} else {
switch v.(type) {
case map[string]any:
if result, found = GetVal(v.(map[string]any), key); found {
return
}
}
}
}
return nil, false}
Later you can think about adding support for fancy stuff like map[interface{}][]interface{}
However, if you really need such complicated structure, I am not sure if the whole design of application is ok.
Maybe you should also think about adding searching for a full path inside the map k2.nestedK3.superNestedK1. It will remove ambiguity.
// Supports getting value by path: i.e. k2.nestedK3.superNestedK1
// Thanks Lukasz Szymik for his answer, which inspired me to implement this functionality based on his code.
func GetValueByPathFromMap(data map[string]any, key string, passedKey string) (result any, found bool) {
keyAndPath := strings.SplitN(key, ".", 2)
currentKey := keyAndPath[0]
if passedKey != "" {
passedKey = passedKey + "." + currentKey
} else {
passedKey = currentKey
}
if _, isKeyExistInData := data[currentKey]; !isKeyExistInData {
logrus.Warnf("[W] key path { %s } not found", passedKey)
return
} else {
if len(keyAndPath) > 1 {
remainingPath := keyAndPath[1]
switch data[currentKey].(type) {
case map[string]any:
if result, found = GetValueByPathFromMap(data[currentKey].(map[string]any), remainingPath, passedKey); found {
return
}
}
} else {
return data[currentKey], true
}
}
return nil, false
}
I am trying to write a table-driven test to test, say, if two orders passed in to a function are the same,
where Order could something like
type Order struct {
OrderId string
OrderType string
}
Now right now, my test looks like:
func TestCheckIfSameOrder(t *testing.T) {
currOrder := Order{
OrderId: "1",
OrderType: "SALE"
}
oldOrder := Order{
OrderId: "1",
OrderType: "SALE"
}
tests := []struct {
name string
curr Order
old Order
want bool
}{
{
name: "Same",
curr: currOrder,
old: oldOrder,
want: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := checkIfSameOrder(tt.curr, tt.old)
if got != tt.want {
t.Errorf("want %v: got %v", tt.want, got)
}
})
}
}
func checkIfSameOrder(currOrder Order, oldOrder Order) bool {
if currOrder.OrderId != oldOrder.OrderId {
return false
}
if currOrder.OrderType != oldOrder.OrderType {
return false
}
return true
}
What I want to do is to add a second test, where I change the OrderId on the currOrder or something and so that my tests slice looks like
tests := []struct {
name string
curr Order
old Order
want bool
}{
{
name: "Same",
curr: currOrder,
old: oldOrder,
want: true,
},
{
name: "Different OrderId",
curr: currOrder, <-- where this is the original currOrder with changed OrderId
old: oldOrder,
want: false,
},
}
Seems to me like I can't use a simple []struct and use something where I pass in a function, but I can't seem to find how to do that anywhere. I'd appreciate if anybody could point me in the right direction. Thanks!
If only the OrderId is different from each test, you can just pass the order id and construct oldOrder and currOrder based on that inside loop.
Sharing global variable that you mutate throughout processes has caused confusion all the time. Better initialize new variable for each test.
func TestCheckIfSameOrder(t *testing.T) {
tests := []struct {
name string
currId string
oldId string
want bool
}{
{
name: "Same",
currId: "1",
oldId: "1",
want: true,
},
{
name: "Different",
currId: "1",
oldId: "2",
want: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
curr := Order{
OrderId: tt.currId,
OrderType: "SALE"
}
old := Order{
OrderId: tt.oldId,
OrderType: "SALE"
}
got := checkIfSameOrder(curr, old)
if got != tt.want {
t.Errorf("want %v: got %v", tt.want, got)
}
})
}
}
I want to save a slice of structs in Google Cloud Datastore (Firestore in Datastore mode).
Take this Phonebook and Contact for example.
type Contact struct {
Key *datastore.Key `json:"id" datastore:"__key__"`
Email string `json:"email" datastore:",noindex"`
Name string `json:"name" datastore:",noindex"`
}
type Phonebook struct {
Contacts []Contact
Title string
}
Saving and loading this struct is no problem as the Datastore library takes care of it.
Due to the presence of some complex properties in my actual code, I need to implement PropertyLoadSaver methods.
Saving the Title property is straightforward. But I have problems storing the slice of Contact structs.
I tried using the SaveStruct method:
func (pb *Phonebook) Save() ([]datastore.Property, error) {
ps := []datastore.Property{
{
Name: "Title",
Value: pb.Title,
NoIndex: true,
},
}
ctt, err := datastore.SaveStruct(pb.Contacts)
if err != nil {
return nil, err
}
ps = append(ps, datastore.Property{
Name: "Contacts",
Value: ctt,
NoIndex: true,
})
return ps, nil
}
This code compiles but doesn't work.
The error message is datastore: invalid entity type
Making a slice of Property explicitly also does not work:
func (pb *Phonebook) Save() ([]datastore.Property, error) {
ps := []datastore.Property{
{
Name: "Title",
Value: pb.Title,
NoIndex: true,
},
}
cttProps := datastore.Property{
Name: "Contacts",
NoIndex: true,
}
if len(pb.Contacts) > 0 {
props := make([]interface{}, 0, len(pb.Contacts))
for _, contact := range pb.Contacts {
ctt, err := datastore.SaveStruct(contact)
if err != nil {
return nil, err
}
props = append(props, ctt)
}
cttProps.Value = props
}
ps = append(ps, cttProps)
return ps, nil
}
Making a slice of Entity does not work either:
func (pb *Phonebook) Save() ([]datastore.Property, error) {
ps := []datastore.Property{
{
Name: "Title",
Value: pb.Title,
NoIndex: true,
},
}
cttProps := datastore.Property{
Name: "Contacts",
NoIndex: true,
}
if len(pb.Contacts) > 0 {
values := make([]datastore.Entity, len(pb.Contacts))
props := make([]interface{}, 0, len(pb.Contacts))
for _, contact := range pb.Contacts {
ctt, err := datastore.SaveStruct(contact)
if err != nil {
return nil, err
}
values = append(values, datastore.Entity{
Properties: ctt,
})
}
for _, v := range values {
props = append(props, v)
}
cttProps.Value = props
}
ps = append(ps, cttProps)
return ps, nil
}
Both yielded the same error datastore: invalid entity type
Finally I resorted to using JSON. The slice of Contact is converted into a JSON array.
func (pb *Phonebook) Save() ([]datastore.Property, error) {
ps := []datastore.Property{
{
Name: "Title",
Value: pb.Title,
NoIndex: true,
},
}
var values []byte
if len(pb.Contacts) > 0 {
js, err := json.Marshal(pb.Contacts)
if err != nil {
return nil, err
}
values = js
}
ps = append(ps, datastore.Property{
Name: "Contacts",
Value: values,
NoIndex: true,
})
return ps, nil
}
Isn't there a better way of doing this other than using JSON?
I found this document and it mentions src must be a struct pointer.
The only reason you seem to customize the saving of PhoneBook seems to be to avoid saving the Contacts slice if there are no contacts. If so, you can just define your PhoneBook as follows and directly use SaveStruct on the PhoneBook object.
type Phonebook struct {
Contacts []Contact `datastore:"Contacts,noindex,omitempty"`
Title string `datastore:"Title,noindex"`
}
I have the following json data coming through an API. I want to unmarshal this data into a different way of structure as it is defined below. How can I do it in an elegant way?
{
"_meta": {
"count": 2,
"total": 2
},
"0": {
"key": "key0",
"name": "name0"
},
"1": {
"key": "key1",
"name": "name1"
},
"2": {
"key": "key2",
"name": "name2"
}
// It goes on..
}
type Data struct {
Meta Meta `json:"_meta,omitempty"`
Elements []Element
}
type Element struct {
Key string
Name string
}
type Meta struct{
Count int
Total int
}
This can be quite tricky because you have a json object that holds everything. So i went with the approach of unmarshalling to map of string to *json.RawMessage and then fixing the struct from there.
To do that you will be using a custom Unmarshaler and the benefit of it is that you delay the actual parsing of the inner messages until you need them.
So for example if your meta field was wrong or the numbers it said didn't match the length of the map-1 you could exit prematurely.
package main
import (
"encoding/json"
"fmt"
)
type jdata map[string]*json.RawMessage
type data struct {
Meta Meta
Elements []Element
}
//Element is a key val assoc
type Element struct {
Key string
Name string
}
//Meta holds counts and total of elems
type Meta struct {
Count int
Total int
}
var datain = []byte(`
{
"_meta": {
"count": 2,
"total": 2
},
"0": {
"key": "key0",
"name": "name0"
},
"1": {
"key": "key1",
"name": "name1"
},
"2": {
"key": "key2",
"name": "name2"
}
}`)
func (d *data) UnmarshalJSON(buf []byte) (err error) {
var (
meta *json.RawMessage
ok bool
)
jdata := new(jdata)
if err = json.Unmarshal(buf, jdata); err != nil {
return
}
if meta, ok = (*jdata)["_meta"]; !ok {
return fmt.Errorf("_meta field not found in JSON")
}
if err = json.Unmarshal(*meta, &d.Meta); err != nil {
return
}
for k, v := range *jdata {
if k == "_meta" {
continue
}
elem := &Element{}
if err = json.Unmarshal(*v, elem); err != nil {
return err
}
d.Elements = append(d.Elements, *elem)
}
return nil
}
func main() {
data := &data{}
if err := data.UnmarshalJSON(datain); err != nil {
panic(err)
}
fmt.Printf("decoded:%v\n", data)
}
I want to scan AWS DynamoDB table and then pull only a certain value. Here is my code:
package main
import (
"fmt"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/dynamodb"
)
func main() {
svc := dynamodb.New(session.New(), &aws.Config{Region: aws.String("us-west-2")})
params := &dynamodb.ScanInput{
TableName: aws.String("my_Dynamo_table_name"),
Limit: aws.Int64(2),
}
resp, err := svc.Scan(params)
if err != nil {
fmt.Println(err.Error())
return
}
fmt.Println(resp)
}
and The output is:
{
Count: 2,
Items: [{
update_time: {
N: "1466495096"
},
create_time: {
N: "1465655549"
}
},{
update_time: {
N: "1466503947"
},
create_time: {
N: "1466503947"
}
}],
LastEvaluatedKey: {
Prim_key: {
S: "1234567890"
}
},
ScannedCount: 2
}
Now, I want to retrieve the update_time value for all elements in above output. Here are my attempts:
for _, value := range resp.Items {
fmt.Println(value["create_time"]["N"])
}
and
for _, value := range resp.Items {
fmt.Println(value.create_time.N)
}
and
for _, value := range resp.Items {
fmt.Println(*value.create_time.N)
}
All above attempts error out with /var/tmp/dynamo.go:37: invalid operation: error.
I am from perl/python background and recently started learning golang.
How to retrieve nested map/array values in this case. Also, any reading references would be of great help. My google search did not reveal anything relevant.
The value of resp above is of the type *ScanOutput, which has Items type as []map[string]*AttributeValue.
To access update_time, you can try:
updateTimes := make([]string, 0)
// Items is a slice of map of type map[string]*AttributeValue
for _, m := range resp.Items {
// m is of type map[string]*AttributeValue
timeStrPtr := *m["update_time"].N
updateTimes = append(updateTimes, *timeStrPtr)
}
updateTimes should now contains all the "update_time" values as strings.
More details here.
You should use the dynamodbattribute package. it's cheaper, safer, and more readable.
Following your example:
type Row struct {
CreateTime int `dynamodbav:"create_time"`
UpdateTime int `dynamodbav:"update_time"`
}
// ...
rows := make([]*Row, len(resp.Items))
if err := dynamodbattribute.Unmarshal(resp.Items, &rows); err != nil {
// handle the error
}
// access the data
for _, row := range rows {
fmt.Println(row.CreateTime, row.UpdateTime)
}